191–200 of 261 posts
I’m working on a centralized platform[1] to help web crawlers be polite by default by respecting robots.txt, 429s, etc, and sharing a platform-wide TTL cache just for crawlers. The goal is to reduce global bot traffic by providing a convenient option to crawler authors that makes their bots play nice with the open web. [1] https://crawlspace.dev
It’s like the friggin tobacco companies or something. Is anyone being the “good guys” on this?
map $http_user_agent $bottype {
default "";
"~.*Amazonbot.*" "amazon";
"~.*ImagesiftBot.*" "imagesift";
"~.*Googlebot.*" "google";
"~.*ClaudeBot.*" "claude";
"~.*gptbot.*" "gpt";
"~.*semrush.*" "semrush";
"~.*mj12.*" "mj12";
"~.*Bytespider.*" "bytedance";
"~.*facebook.*" "facebook";
}
limit_req_zone $bottype zone=bots:10m rate=6r/m;
limit_req zone=bots burst=10 nodelay;
limit_req_status 429;
Note that if SEO is a goal, this does make you vulnerable to blackhat SEO by someone faking a UA of a search engine you care about and eating their 6 req/minute quota with fake bots. You could treat Google differently.
This approach won't solve for the case where the UA is dishonest and pretends to be a browser - that's an especially hard problem if they have a large pool of residential IPs and emulate / are headless browsers, but that's a whole different problem that needs different solutions.
Earlier quoted context omitted.
On what legal basis?
In the UK, the Computer Misuse Act applies if: * There is knowledge that the intended access was unauthorised * There is an intention to secure access to any program or data held in a computer I imagine US law has similar definitions of unauthorized access? `robots.txt` is the universal standard for defining what is unauthorised access for bots. No programmer could argue they aren't aware of this, and ignoring it, fo…
The last part basically means the robots.txt file can be circumstantial evidence of intent, but there needs to be other factors at the heart of the case.
If you use nginx to front it, consider something like this in the `http` block of your config: map $http_user_agent $bottype { default ""; "~.*Amazonbot.*" "amazon"; "~.*ImagesiftBot.*" "imagesift"; "~.*Googlebot.*" "google"; "~.*ClaudeBot.*" "claude"; "~.*gptbot.*" "gpt"; "~.*semrush.*" "semrush"; "~.*mj12.*" "mj12"; "~.*Bytespider.*" "bytedance"; "~.*facebook.*" "facebook"; } limit_req_zone $bottype zone=bots:10m r…
You can ingest this IP list periodically and set rules based on those IPs instead. Makes you not prone to the blackhat SEO tactic you mentioned. In fact, you could completely block GoogleBot UA strings that don’t match the IPs, without harming SEO, since those UA strings are being spoofed ;)
Honestly I think this might end up being the mid-term solution.
For legitimate traffic it's not too onerous, and recognized users can easily have bypasses. For bulk traffic it's extremely costly, and can be scaled to make it more costly as abuse happens. Hashcash is a near-ideal corporate-bot combat system, and layers nicely with other techniques too.
Earlier quoted context omitted.
I had to deal with some bot activities that used huge address space, and I tried something very similar, when condition confirming bot was detected I banned that IP for 24h but due to amount of IPs involved this did not have any impact on about if traffic my suggestion is to look very closely on headers that you receive (varnishlog in very nice of this and of you stare long enough at then you might stop something tha…
My favorite example of this was how folks fingerprinted the active probes of the Great Firewall of China. It has a large pool of IP addresses to work with (i.e. all ISPs in China), but the TCP timestamps were shared across a small number of probing machines: "The figure shows that although the probers use thousands of source IP addresses, they cannot be fully independent, because they share a small number of TCP time…
Earlier quoted context omitted.
When I was writing a crawler for my search engine (now offline), I found almost no crawler library actually compliant with the real world. So I ended up going to a lot of effort to write one that complied with Amazon and Google's rather complicated nested robots files, including respecting the cool off periods as requested. ... And then found their own crawlers can't parse their own manifests.
Could you link the source of your crawler library?
It complied, but it was absolutely not fast or efficient. I aimed at compliance first, good code second, but never got to the second because of more human-oriented issues that killed the project.
Earlier quoted context omitted.
Why work hard… Train a model to recognize the AI bots!
This isn't a problem domain that models are capable of solving. Ultimately in two party communications, computers are mostly constrained by determinism, and the resulting halting/undecidability problems (in core computer science). All AI Models are really bad at solving stochastic types of problems. They can approximate generally only to a point after which it falls off. Temporal consistency im time series data is al…