Description
Web pages can be crawled from the command line with a Unix-style tool that fits shell pipelines. It is useful for developers and administrators checking links, collecting URLs, or feeding later processing steps.
Crawlers can create unwanted traffic and collect sensitive content. Respect robots rules, rate limits, authentication boundaries, and privacy before scanning a site.