Question about search engine bots

Results 1 to 2 of 2

Thread: Question about search engine bots

  1. #1
    Join Date
    Dec 1969

    Default Question about search engine bots

    Hi,<BR><BR>I have been struggling for quite some time now to get Google to see my web site. I finally realized that b/c I only allow standards-compliant browsers (all others are redirected to a browser-upgrade page, similar to the way works) Google is only ever seeing the Upgrade page.<BR><BR>I&#039;m doing my simple browser check in the global.asax file using .NET&#039;s BrowserCapabilities object. Is there a way that I can tell if the browser is a bot, and if so, let it pass through so that it can crawl my site?<BR><BR>Thanks<BR>-Steve

  2. #2
    Join Date
    Dec 1969

    Default RE: Question about search engine bots

    I personally can&#039;t think of anyway to do that unless you played around with HTTP_USER_AGENT searching through it and if it contains "bot" then allow type thing, this won&#039;t let them all through though, you need to add various strings for the different engines a few that I get through:<BR><BR>HTTP_USER_AGENT=HenryTheMiragoRobo t (<BR>HTTP_USER_AGENT=Mozilla/5.0 (compatible; Yahoo! Slurp;<BR>HTTP_USER_AGENT=msnbot/0.11 (+<BR>HTTP_USER_AGENT=psbot/0.1 (+<BR>HTTP_USER_AGENT=ia_archiver<BR>HTTP_U SER_AGENT=Googlebot/2.1 (+<BR><BR>Thats just a few that&#039;ve come through in the past few days, I can try and find more for you if you like?<BR><BR>Hope that helps<BR><BR>Tim

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts