Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The most-seen UI on the internet? Redesigning turnstile and challenge pages

blog.cloudflare.com

Post reply on HN
Print view View original

71–77 of 77 posts

Re: The most-seen UI on the internet? Redesigning turnstile and challenge pages

#71

Post by PaulDavisThe1st »

neoromantique wrote: post #32
PaulDavisThe1st wrote: post #6

Earlier quoted context omitted.

One of the things that a lot of LLM scrapers are fetching are git repositories. They could just use git clone to fetch everything at once. But instead, they fetch them commit by commit. That's about as static as you can get, and it is absolutely NOT a non-issue.

that's a pretty niche issue, but fairly easy to solve. Prebuild statically the most common commits (last XX) and heavily rate limit deeper ones

1. that doesn't appear to match the fetching patterns of the scrapers at all

2. 1M independent IPs hitting random commits from across a 25 year history is not, in fact, "easy to solve". It is addressable, but not easy ...

3. why should I have to do anything at all to deal with these scrapers? why is the onus not on them to do the right thing?

Re: The most-seen UI on the internet? Redesigning turnstile and challenge pages

#72

Post by 1vuio0pswjnm7 »

"Our Turnstile widget and Challenge Pages are served 7.67 billion times every single day. That's not a typo. Billions. This might just be the most-seen user interface on the Internet."

Or it might not

The majority of the traffic on the internet is from so-called "bots"

If a "bot" hits this "interface" does that count as being "seen"

The web's failing, its inability (unwillingness) to accept non-interactive use (no good for advertising), is Cloudflare's success

A strange thing to celebrate. MITM'ing the majority of the web for "security". Could there be a better way

Another source of amusement is the "You've been blocked" Cloudflare page showing the user's IP address and suggesting contacting the site operator might solve the problem

The truth is that sending an acceptable user-agent header value solves the problem

"You" are not being blocked (Cloudflare does not who "you" are), your IP address is not being blocked, the _request_ you sent was blocked because of crude heuristics

If a site operator wants a certain header value (why) then it should publish the list of acceptable values

Send another request with an acceptable header value and the requests succeeds. It appears "you" are not blocked, same IP address, same living, breathing, thinking person sending the request

Re: The most-seen UI on the internet? Redesigning turnstile and challenge pages

#73

Post by noplacelikehome »

sebzim4500 wrote: post #66
noplacelikehome wrote: post #65

Earlier quoted context omitted.

Quite likely fingerprinting detection, which is remaining firmly enabled.

How does that work technically? Presumably a fork of firefox is almost indistinguishable from firefox from Cloudflare's perspective?

[dead]

Re: The most-seen UI on the internet? Redesigning turnstile and challenge pages

#74

Post by andrepd »

holden_nelson wrote: post #59
andrepd wrote: post #42

Earlier quoted context omitted.

That's why I say most AI content isn't just slop—it's fundamentally about deception. It's about tricking someone into believing that a text was written by a human, or that a photo or video is a true recording of a real event. Like this, its purpose is to fly under the radar unless your figurative ears are pricked up and primed to detect the telltale signs. Fuck this shit.

Can’t tell if the “it’s not X — it’s Y” as your first sentence is intentional irony or not lol

You're absolutely right!

Re: The most-seen UI on the internet? Redesigning turnstile and challenge pages

#75

Post by tempest_ »

noplacelikehome wrote: post #63
tempest_ wrote: post #3

Earlier quoted context omitted.

As bad as cloudflare is there is a reason people use it. If you try and run a site that has content that LLMs want or expensive calls that require a lot of compute and can exhaust resources if they are over used the attack is relentless. It can be a full time job trying to stop people who are dedicated to scrapping the shit out of your site. Even CF doesnt even really stop it any more. The agent run browsers seem to…

Granted, but there are open source alternatives that don’t have the same obsession with meaningless digital signatures. Turnstile is just a terrible product.

What are the open source options? Turnstile is a replacement for Recaptcha after google moved it from a free product to a paid one.

The main advantage of Turnstile is that is benefits from CFs ubiquity to help judge legitimate vs illegitimate requests.

I would love to know what other options are available in this space aside from Turnstile, Recaptcha and HCaptcha.

Re: The most-seen UI on the internet? Redesigning turnstile and challenge pages

#76

Post by noplacelikehome »

tempest_ wrote: post #75
noplacelikehome wrote: post #63

Earlier quoted context omitted.

Granted, but there are open source alternatives that don’t have the same obsession with meaningless digital signatures. Turnstile is just a terrible product.

What are the open source options? Turnstile is a replacement for Recaptcha after google moved it from a free product to a paid one. The main advantage of Turnstile is that is benefits from CFs ubiquity to help judge legitimate vs illegitimate requests. I would love to know what other options are available in this space aside from Turnstile, Recaptcha and HCaptcha.

Anubis is the new hotness, specifically billing itself as an "AI firewall". If you've had an animé waifu check you're human you've even used it.

Re: The most-seen UI on the internet? Redesigning turnstile and challenge pages

#77

Post by neoromantique »

PaulDavisThe1st wrote: post #71
neoromantique wrote: post #32

Earlier quoted context omitted.

that's a pretty niche issue, but fairly easy to solve. Prebuild statically the most common commits (last XX) and heavily rate limit deeper ones

1. that doesn't appear to match the fetching patterns of the scrapers at all 2. 1M independent IPs hitting random commits from across a 25 year history is not, in fact, "easy to solve". It is addressable, but not easy ... 3. why should I have to do anything at all to deal with these scrapers? why is the onus not on them to do the right thing?

I did not imply that it does, I meant to have a budget allocated for 'unauthenticated deep history queries', when it's over it's over and you only handle dynamic fetching for authorized users until cooldown.

Is it pretty? No, but it also is a pretty niche thing overall (git repo storage).

Post reply on HN