Crawler (also Spider, Robot or Bot):
- Component of a search engine that gathers listings by automatically trolling the Web and following links to webpages. It makes copies of the webpages found and stores them in the search engine’s index.
The faster the crawlers work, the faster the pages are indexed and show up in search engine results pages.
____
Glossary Definition From ABC’s of Online Marketing by Alexandra Wharton, Issue 22, Revenue Magazine
