spider

The Web-walking part of a search engine that collects pages for indexing in
the search engine's database. Also called a bot. The best-known  is Scooter,
the web-walker for the Alta Vista search engine.

