Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare’s CAPTCHA replacement with FIDO2/WebAuthn is a bad idea

herrjemand.medium.com

Post reply on HN
Print view View original

51–60 of 294 posts

Re: Cloudflare’s CAPTCHA replacement with FIDO2/WebAuthn is a bad idea

#51

Post by PaulHoule »

jgrahamc wrote: post #24
navanchauhan wrote: post #23

Earlier quoted context omitted.

Crawling a Cloudflare powered website is basically impossible without needing to do some bodges as to how to crawl it. How can you expect someone to crawl a bunch of websites if they are actively blocked from accessing it? Now, you might say users can whitelist bots in their robots.txt file but then again will the person creating the engine individually ask companies to allow them to crawl? Also, slightly unrelated b…

If you are building a search engine and getting blocked you can always contact me and I'll make sure that the teams that work on bot detection and DDoS are aware. We would like to know because we should not be blocking a legit crawler like this.

What makes a web crawler "legit?"

When I had a site that had millions of pages, I found that sites like Baidu would crawl my site as often, if not more often than Google.

I already felt the relationship with Google was parasitic, but I looked through my logs and never found a single hit that came from Baidu and many of the other search engines that would overload my site.

I was looking at a substantial part of the site running costs going to supporting web crawlers that were not doing anything (1) to help me, or (2) to help end users (if they don't want to send Chinese users to an English-speaking web site, why crawl the site?)

So like it or not I am inclined to only allow Google and Bing in the robots.txt because Google is the only site that sends a significant amount of traffic and because Bing sends some, and Google needs some competition.

There are web crawler behaviors that are annoying: harvesting email addresses, overloading your site, etc. But how do you know who is doing something wrong with the data and who is just collecting it do do nothing with it? (Probably 95% of web crawling ex. Google.)

Re: Cloudflare’s CAPTCHA replacement with FIDO2/WebAuthn is a bad idea

#52

Post by SXX »

PaulHoule wrote: post #45
SXX wrote: post #29

Earlier quoted context omitted.

For instance there is no way for distributed search engines to work with CloudFlare. No, "contact me and we'll help" is not always a solution.

I've never been able to "reach a human" at Google, Facebook and other web giants and I'm skeptical that you can at a place like Cloudflare. In fact, I'd be really astonished it was possible, because otherwise their business isn't scalable.

I personally love CloudFlare and (just like with e.g DigitalOcean) I always found a way to contact human there. Unfortunately it's doesn't fix fundamental issue of how they make internet much more centralized and easy to MiTM or apply censorship.

Re: Cloudflare’s CAPTCHA replacement with FIDO2/WebAuthn is a bad idea

#53

Post by joepie91_ »

jgrahamc wrote: post #42
oefrha wrote: post #38

Earlier quoted context omitted.

Say I’m interested in building a small scale domain-specific search engine and only just started development. There’s no prototype yet and may never be. In this situation, how do you determine it’s a legit crawler? And what about crawlers with even more limited scopes (targeting only a handful of sites) that they can’t possibly be called search engines? Are they ever considered legit?

Be a good netizen? Respect robots.txt. Don't lie in your User-Agent. Don't crawl at a ridiculous rate. All those are a good starting point.

Do you not realize the more fundamental problem with you, as a company, essentially being the one who gatekeeps crawler access to the web?

Re: Cloudflare’s CAPTCHA replacement with FIDO2/WebAuthn is a bad idea

#54

Post by wbkang »

lsaferite wrote: post #15

Privacy seems like a bad argument considering CF already has the technical ability to easily track you across all of the sites they front if they so desire.

With each domain u2f generates a different key (conceptually) so this should be harder to track potentiality.

Re: Cloudflare’s CAPTCHA replacement with FIDO2/WebAuthn is a bad idea

#55

Post by y7 »

dboreham wrote: post #47
y7 wrote: post #39

Earlier quoted context omitted.

Yup, you can't. Keys are perfectly trackable by Cloudflare, but they promise they won't do this. Edit: I was wrong. Cloudflare claims they could track people, but it would require tracking via cookies. [1] The hardware security keys have an "attestation key pair" that is shared among all units in one production batch (which contains at least 100K units). [2] 1: https://blog.cloudflare.com/introducing-cryptographic-at…

No, they can't do this. It's the U2F key vendor that promises not to release a device-unique key to someone like CF.

Thanks, I stand corrected.

Re: Cloudflare’s CAPTCHA replacement with FIDO2/WebAuthn is a bad idea

#56

Post by clukic »

Cloudflare is the professional wall builder you hire to protect your garden.

Tech monopolies have always had a vested interest in locking up user data, dictating the policies, and enforcing their own ownership rights. It used to be that only the largest and most sophisticated companies had the resources to shield that data, but Cloudflare changed all that. Walls are now trivial to set up, and virtually unbreachable, and that has forever changed the character of the internet by enforcing monopolistic policies with such technical precision that they're virtually impossible to overcome.

Re: Cloudflare’s CAPTCHA replacement with FIDO2/WebAuthn is a bad idea

#57

Post by mwcampbell »

There is no perfect solution, but I'm in favor of anything that's a net improvement in accessibility for disabled people, even if it's not ideal in some other way. So I'm disappointed to see this solution being shot down before it even gets deployed on a large scale.

Re: Cloudflare’s CAPTCHA replacement with FIDO2/WebAuthn is a bad idea

#58

Post by acdha »

PaulHoule wrote: post #51
jgrahamc wrote: post #24

Earlier quoted context omitted.

If you are building a search engine and getting blocked you can always contact me and I'll make sure that the teams that work on bot detection and DDoS are aware. We would like to know because we should not be blocking a legit crawler like this.

What makes a web crawler "legit?" When I had a site that had millions of pages, I found that sites like Baidu would crawl my site as often, if not more often than Google. I already felt the relationship with Google was parasitic, but I looked through my logs and never found a single hit that came from Baidu and many of the other search engines that would overload my site. I was looking at a substantial part of the si…

> So like it or not I am inclined to only allow Google and Bing in the robots.txt because Google is the only site that sends a significant amount of traffic and because Bing sends some, and Google needs some competition.

This sounds like you're onto a reasonable “legit” factor: does the crawler honor robots.txt? Baidu would be legit because they don't lie about their identity and if you put a rule in your robots.txt file they'll honor it.

Re: Cloudflare’s CAPTCHA replacement with FIDO2/WebAuthn is a bad idea

#60

Post by FriedrichN »

I hate this new web where you're automatically assumed to be some malicious actor only because you don't accept cookies and strange third party code and then have to jump through hoops to show that you're not some evil bot. To be honest, if a website immediately throws some Cloudflare anti-DDoS thing in my face I don't even bother anymore.
Post reply on HN