Hide robots.txt file

Results 1 to 6 of 6

Thread: Hide robots.txt file

  1. #1
    Join Date
    Dec 1969
    Posts
    661

    Default Hide robots.txt file

    Is there a way to make it so when you go to:<BR><BR>www.mysite.com/robots.txt you do not actually see the robots.txt file<BR><BR>This has always seemed weird. OK world, these are all the really important folders we DON&#039;T want you to see.<BR><BR>hmm,<BR><BR>Paul<BR>

  2. #2
    Join Date
    Dec 1969
    Posts
    11,334

    Default No

    Why would you? How else would the bot find it?<BR><BR>If a file is in your www area, anyone can get to it.

  3. #3
    Join Date
    Dec 1969
    Posts
    661

    Default RE: No

    Of course. But often one would do a dis-allow for thimgs that are sensitive or one does not want to get into search engines. These things may also be protected via log ins. It just seems wierd that you can go to a site and see all these folders.<BR><BR>I have noticed that GE.com and yahoo do not have a robots.txt<BR><BR>You get a 404<BR><BR>why would they not have a robots.txt<BR><BR>

  4. #4
    Join Date
    Dec 1969
    Posts
    11,334

    Default Because they let the bots do

    whatever they want in there.<BR><BR>AFAIK, the bots don&#039;t go into protected areas... they typically don&#039;t follow those types of links (you are talking NT protection, no?)<BR><BR>And even if they do follow &#039;em, any ol&#039; user can&#039;t log in.<BR><BR>Remeber about this bot stuff... there are only a handful of &#039;good&#039; bots out there... the &#039;evil&#039; ones don&#039;t even bother to look at robots.txt and index away anyways, so.....


  5. #5
    Join Date
    Dec 1969
    Posts
    661

    Default Road Signs to Poke Around

    so,<BR><BR>I want to poke around a Web Server. I go to: www.mysite.com/robots.txt and get started. Why have a robots.txt file in the first place? If you secure stuff with SQL or what ever all one is doing with the robots.txt is showing the world your "hand".<BR><BR>I see no reason to have the robots.txt<BR><BR>

  6. #6
    Join Date
    Dec 1969
    Posts
    11,334

    Default Deep link preventer, mainly

    That&#039;s our reason... dunno about others.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •