njh [at] despammed.com writes:
> If you spider several sites and one of them has a broken robots.txt file you
> can't tell which one since the warning doesn't tell you.
This will be better in 5.801. I've applied a variation of Bill
Moseley's suggested patch for the same problem.
> Around line 73 of RobotRules.pm
> change:
> warn "RobotRules: Disallow without preceding User-agent\n";
> to
> # njh [at] bandsman.co.uk: added $netloc
> warn "RobotRules: $netloc Disallow without preceding User-agent\n";
