Web Bot Auth • Hacker News Board

Re: Web Bot Auth

#71

Post by immibis » Fri, Aug 29, 2025, 9:23 PM UTC

acdha wrote: post #70

immibis wrote: post #65

Earlier quoted context omitted.

You could just, like, not make a website that takes several seconds to handle each request. I let bots hit Gitea 2-3 times per second on a $10/month VPS, and the only actual problem was that it doesn't seem to ever delete zip snapshots, filling up the disk when enough snapshot links are clicked. So I disabled that feature by setting the snapshots folder read-only. There were no other problems. I mention Gitea because…

Sure, I’ve been doing that since the 90s. I still pay for hardware and egress, and it turns out that everything has limits for the amount of traffic it can handle which bots can easily saturate. I’ve had sites which were mostly Varnish serving cached content at wire speed go down because they saturated the upstream.

I hope 2-3 requests per second is not that limit, or you're fucked.

Re: Web Bot Auth

#72

Post by estearum » Fri, Aug 29, 2025, 9:27 PM UTC

skeezyboy wrote: post #68

estearum wrote: post #67

Earlier quoted context omitted.

Yeah, I don't find this to be a compelling argument at all. That's an argument against all authentication anywhere.

> That's an argument against all authentication anywhere. its a problem isnt it

No

Re: Web Bot Auth

#73

Post by acdha » Fri, Aug 29, 2025, 9:51 PM UTC

immibis wrote: post #71

acdha wrote: post #70

Earlier quoted context omitted.

Sure, I’ve been doing that since the 90s. I still pay for hardware and egress, and it turns out that everything has limits for the amount of traffic it can handle which bots can easily saturate. I’ve had sites which were mostly Varnish serving cached content at wire speed go down because they saturated the upstream.

I hope 2-3 requests per second is not that limit, or you're fucked.

It’s not, but you’re off by 3+ orders of magnitude on the traffic volume and ignoring the cost of serving non-trivial responses.

Re: Web Bot Auth

#74

Post by vntok » Fri, Aug 29, 2025, 10:57 PM UTC

immibis wrote: post #71

acdha wrote: post #70

Earlier quoted context omitted.

Sure, I’ve been doing that since the 90s. I still pay for hardware and egress, and it turns out that everything has limits for the amount of traffic it can handle which bots can easily saturate. I’ve had sites which were mostly Varnish serving cached content at wire speed go down because they saturated the upstream.

I hope 2-3 requests per second is not that limit, or you're fucked.

It is on a simple WordPress install with the top 4 most used plugins, when you don't have a Caching Reverse Proxy like Cloudflare to filter bad traffic and serve fully cached pages from POP nodes located near the visitors.

The alternative, of course, is to set up a caching system server-side (like Redis), which most people who set up their WordPress blog don't have the first idea how to do in a secure way.

Re: Web Bot Auth

#75

Post by bobbiechen » Sat, Aug 30, 2025, 2:23 AM UTC

palmfacehn wrote: post #69

>3. Register your bot and key directory Register with CF is the specific part I object to. Of all of the numerous hazards here centralizing the registration with CF is most clearly problematic . This part of the spec could have easily been an additional header linking to key data.

Good news - this part of the spec is literally an additional header linking to key data: https://datatracker.ietf.org/doc/html/draft-meunier-http-mes... .

Cloudflare is doing this registration as part of their "verified" program, which gives special treatment to bots/agents who go through the process. That's a Cloudflare-specific feature, not part of the spec.

Re: Web Bot Auth

#76

Post by palmfacehn » Sat, Aug 30, 2025, 4:29 AM UTC

bobbiechen wrote: post #75

palmfacehn wrote: post #69

>3. Register your bot and key directory Register with CF is the specific part I object to. Of all of the numerous hazards here centralizing the registration with CF is most clearly problematic . This part of the spec could have easily been an additional header linking to key data.

Good news - this part of the spec is literally an additional header linking to key data: https://datatracker.ietf.org/doc/html/draft-meunier-http-mes... . Cloudflare is doing this registration as part of their "verified" program, which gives special treatment to bots/agents who go through the process. That's a Cloudflare-specific feature, not part of the spec.

Nice catch. Even if we like the standard, the specific implementation can have problems. MSIE, extend and embrace are similar examples.

Re: Web Bot Auth

#77

Post by immibis » Sat, Aug 30, 2025, 8:43 AM UTC

bgwalter wrote: post #35

Cloudflare is playing both sides: grok.com is served by Cloudflare.

Famously, a lot of DDoS-for-hire providers were served by Cloudflare. CF has a strong interest in making sure you can always get DDoSed.