← Back to glossary definition
Web scraping and extraction
Web crawler
Also called: Spider, bot
A program that discovers pages by following links from a set of starting URLs and fetching each page it finds.
In practice
Crawlers decide what to fetch; extractors decide what to keep. Most data programs need both.
at import.io
Import.io extractors are built by point-and-click, AI assistance or engineers, with rendering, pagination and schema detection built in.
Web data extraction →