Description
Creates web crawlers that collect structured data from pages, links, and APIs. Developers use it to automate scraping jobs, follow sites, parse responses, and store extracted information for later processing.
It is a Python framework with terminal tooling. Scraping can violate site rules or privacy expectations, so check robots.txt, terms of service, rate limits, and legal requirements before collecting data.