Numerous sites, in particular search engines, use spidering as a means of providing up-to-date data. Web crawlers are primarily used to create a copy of all the visited pages for later on processing by a search engine that will index the downloaded pages to give fast searches. Crawlers can as well be used for automating maintenance tasks on a website, like checking links or validating HTML code. In addition, crawlers can be used to collect detailed types of information from Web pages, for example, harvesting e-mail addresses (usually for spam cases).
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment