241–250 of 261 posts
Earlier quoted context omitted.
It's forward confirming reverse DNS. I assumed that everyone does that by default.
What everyone does by default doesn't matter really here, it's that an IP owner/user can literally set the reverse to any arbitrary domain regardless if the actual domain has a record for that IP. What matters is both match, thats all I meant
I’m working on a centralized platform[1] to help web crawlers be polite by default by respecting robots.txt, 429s, etc, and sharing a platform-wide TTL cache just for crawlers. The goal is to reduce global bot traffic by providing a convenient option to crawler authors that makes their bots play nice with the open web. [1] https://crawlspace.dev
> respecting robots.txt What agent name should we put in robots.txt to deny your crawler without using a wildcard? I can't see that documented anywhere.
> I'm working on a proof of work reverse proxy to protect my server from bots in the future. Honestly I think this might end up being the mid-term solution. For legitimate traffic it's not too onerous, and recognized users can easily have bypasses. For bulk traffic it's extremely costly, and can be scaled to make it more costly as abuse happens. Hashcash is a near-ideal corporate-bot combat system, and layers nicely…
I'm gonna publish more about this tomorrow. I didn't expect this to blow up so much.
Earlier quoted context omitted.
> respecting robots.txt What agent name should we put in robots.txt to deny your crawler without using a wildcard? I can't see that documented anywhere.
Thanks for the feedback, it’s mentioned in the platform FAQ but I should make it more prominent in the docs. The UA will always be prefixed with the string `Crawlspace`. May I ask why you’d want to block it, even if it crawls respectfully?
User-agent: Crawlspace
Disallow: /
The main audience for the product seems to be AI companies, and some people just aren't interested in feeding that beast. Lots of sites block Common Crawl even though their bot is usually polite.
Earlier quoted context omitted.
Thanks for the feedback, it’s mentioned in the platform FAQ but I should make it more prominent in the docs. The UA will always be prefixed with the string `Crawlspace`. May I ask why you’d want to block it, even if it crawls respectfully?
The bot having "Crawlspace" in its UA doesn't necessarily mean it honors "Crawlspace" directives in robots.txt. Would it bail out if it saw this robots.txt? User-agent: Crawlspace Disallow: / > May I ask why you’d want to block it, even if it crawls respectfully? The main audience for the product seems to be AI companies, and some people just aren't interested in feeding that beast. Lots of sites block Common Crawl e…
Yes, it should — I use the library below, and it should split at the slash character, treating it as a prefix match, per spec.
https://github.com/samclarke/robots-parser/blob/master/Robot...
Earlier quoted context omitted.
I'm gonna publish more about this tomorrow. I didn't expect this to blow up so much.
Thank you for the issue-awareness-building and the tool! Hopefully not too much hn-drama went your way, I always enjoy your content :)
Earlier quoted context omitted.
I had to deal with some bot activities that used huge address space, and I tried something very similar, when condition confirming bot was detected I banned that IP for 24h but due to amount of IPs involved this did not have any impact on about if traffic my suggestion is to look very closely on headers that you receive (varnishlog in very nice of this and of you stare long enough at then you might stop something tha…
If you just block the connection, you send a signal that you are blocking it, and they will change it. You need to impose cost per every connection through QoS buckets. If they rotate IPs, ban by ASN, have a page with some randomized pseudo looking content in the source (not static), and explain that the traffic allocated to this ASN has exceed normal user limits and has been rate limited (to a crawl). Have graduated…
https://www.routeviews.org/routeviews/>
That also provides the associated AS, enabling blocking at that level as well, if warranted.
Earlier quoted context omitted.
Even some non-profit ignore it now, Internet Archive stopped respecting it years ago: https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
As I recall, this is outdated information. Internet Archive does respect robots.txt and will remove a site from its archive based on robots.txt. I have done this a few years after your linked blog post to get an inconsequential site removed from archive.org.
https://blog.archive.org/?s=robots.txt>
Earlier quoted context omitted.
Terms of use contract violation?
Robots.txt is completely irrelevant. TOU/TOS are also irrelevant unless you restrict access to only those who have agreed to terms.
https://www.imperva.com/legal/website-terms-of-use/>
Many, many, many hits for this or similar language:
https://duckduckgo.com/?q=%22By+accessing+this+Site%2C+you+a...>
Mind: just because it's written doesn't mean it's enforceable, but to argue that what you've just denied isn't a widely-used premise of online contracts of adhesion fails the simplest empirical test.
Earlier quoted context omitted.
What everyone does by default doesn't matter really here, it's that an IP owner/user can literally set the reverse to any arbitrary domain regardless if the actual domain has a record for that IP. What matters is both match, thats all I meant
xena said "forward confirming reverse" twice which means rdns and then resolving that forward to confirm it matches.
I'd still be surprised if an Amazon domain resolved to a residential IP