Catching Spiders?

Results 1 to 3 of 3

Thread: Catching Spiders?

  1. #1
    jut Guest

    Default Catching Spiders?

    Hello everyone,<BR> Does anyone know if it is possible to detect a spider entering a site and redirecting to another page accordingly using asp?<BR><BR>Many thanks in advance!<BR>Justin.

  2. #2
    Join Date
    Dec 1969

    Default RE: Catching Spiders?

    Well, the only thing you&#039ve got in ASP that could tell you that the client&#039s a spider is Request.ServerVariables("HTTP_USER_AGENT"). Unfortunately there are loads of spiders out there, each with a different user agent name. You could try assuming that any user agent that isn&#039t a well-known browser is a spider, but there are a heck of a lot of browsers out there too.<BR><BR>Why do you want to do this? If you want to control how spiders index your site you&#039d be better off reading up on the use of the robots.txt file.<BR><BR>Dunc

  3. #3
    Join Date
    Dec 1969

    Default RE: Catching Spiders?

    yes. spiders provide a User_agent string just like any browser - trouble is they&#039re not that easy to find out. they also don&#039t respond to redirection by &#060;meta&#062;, response.redirect or javascript.<BR><BR>if you&#039re using ASP 3.0 you could server.execute based on if it&#039s a spider or not, otherwise, you&#039re left with using robots.txt (lots of links available at a search engine near you)<BR><BR>jason

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts