Search Engine Spider

Results 1 to 3 of 3

Thread: Search Engine Spider

  1. #1
    Martin Guest

    Default Search Engine Spider

    Hi, I would like my search engine to spider html pages. It would be usefull for it to see if the page is working, and to get the description, title and keywords from Meta tags. How can I do that? Thank you, this will help me alot.

  2. #2
    Join Date
    Dec 1969

    Default RE: Search Engine Spider

    Your question is a bit vague. You have developed a search engine to crawl the web and now you want to develop a spider? It would be useful to see if what pages are working? Yours? Perhaps reposting with more detail would be helpful.

  3. #3
    Martin Guest

    Default RE: Search Engine Spider

    Sorry,<BR>What I want to do is "something" that I called a spider (which isn&#039;t really a spider) that checks if the urls listed in a database are working (if the URL brings to 404 Error "The page cannot be found", the spider would delete this url... If the url/page is working (that brings to the web site), the spider would update the information of the database, so get the description of the url, the title and the keywords (taken from the meta tags). This "spider" would be usefull for all these things. Sorry if I&#039;m not that clear. All I want it to do is to parse meta tags from urls listed in a database. I would like it to detect "404, Page cannot be found" urls and auto delete them.<BR>Thank you!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts