Proposal to integrate the robots_txt package.

[vxern]

Hi, author of the robots_txt package here. I thought that, since linkcheck currently features its own implementation of a parser for robots.txt, it would be a good idea to instead depend on something more complete, tested and documented.

[filiph]

Thanks for the suggestion, @vxern! I'm afraid I don't have the time to implement this. Are you suggesting you'd implement this as well?

Also, cc @parlough for his opinion.

[vxern]

Hi, @filiph. Sure, I wouldn't mind. Let's wait for parlough's opinion first, though. :)

[parlough]

Sounds good to me, but it's a bit hard to determine for sure without seeing what the end result would look like.

I'd be happy to take a look if you open a PR :)