Description
Crawler exclusion rules can be parsed and matched through a C++ robots.txt library. It is development support, not a standalone app, for search, crawling, and web indexing software.
FICHA · AUR
robotstxt
Google's robots.txt parser and matcher as a C++ library (compliant to C++11).
- robots.txt C++ library
- LIBRARY
- Web
- PARSER
- Dependency only
official+codex · reviewed · Jun 4, 2026 description in
en Permissions
Permissions not analysed for this source yet.