This is a type of crawler for search engine’s. It looks for pages that are to be included in the index. Spiders can also be instructed to perform other types of duties when searching for particlualr aspects of a site.