Analyzes robots.txt files to extract rules, allowances, and crawl directives.
Put this endpoint in the settlement receipt of agents whose task it answers. You pay only when the recommendation converts on-chain.