Live data from Hacker News

Cloudflare's new marketplace lets websites charge AI bots for scraping

techcrunch.com

211–220 of 280 posts

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#211
post #171
post #169

Earlier quoted context omitted.

Cloudflare isn't unilaterally inserting themselves between the website and you. They're contracted by the website owner to provide website security, just like how ticketmaster is contracted by the venue owner to provide ticketing. I don't see what the difference is.

"Security" in the real world doesn't get to profile people. Profiling is Cloudflare's entire business model.

What do you think club bouncers are doing?

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#212
post #209
post #198

Earlier quoted context omitted.

>Define "bots" in a way computers can understand. How is having a specific definition relevant to this conversation? An approximate definition of "a human using a browser to visit a site" probably suffices, without having to get into weird edge cases like "but what if they programmed lynx to visit your site at 3am when they're asleep?". >Regular users that cloudflare (profiles) accuses of being bots. God help you if…

> How is having a specific definition relevant to this conversation? Because it's a computer that automatically does it. That's the entire problem here. Humans are not in the loop, except collecting the paychecks. > An approximate definition of "a human using a browser to visit a site" probably suffices Humans are not doing the blocking. "Approximate" is not good enough when, for example, I need to go to a coffee sho…

>Because it's a computer that automatically does it. That's the entire problem here. Humans are not in the loop, except collecting the paychecks.

I don't see how "Humans are not in the loop" is a relevant factor for whether something is a "criminal enterprise" or not. Humans are often not in the loop in approving loans/credit cards either. That doesn't make equifax a "criminal enterprise" for blocking you from getting a loan because you can't pass a credit check. Even in jurisdictions with laws against automated decision making by computers, you can only seek human redress in specific circumstances (eg. when applying for credit), not for whether a website blocked you for being a suspected bot or not

>I need to go to a coffee shop and use an entirely different computer to trick cloudflare into letting me order parts on digikey. And I must repeat that my work computer is doing absolutely nothing interesting. My job and livelihood depend on this.

1. At least looking at the response headers, digikey.com is served by akamai, not cloudflare

2. I can visit the site just fine on commercial VPN providers. Maybe there's something extra sus about your connection/browser, but I find it hard to believe that you have to resort to getting a separate computer and making a 10 minute trek to visit a site

3. like it or not, neither cloudflare nor digikey has any obligation to serve you. They can deny you service for any reason they want, except for a very small list of exceptions (eg. race or disability). "browser/configuration looks weird" is an entirely valid reason, and them denying you service on that basis doesn't mean cloudflare is running a "protection racket".

>What about an edge case like 'using your bone stock phone to visit a site once'?

that's clearly not an edge case

>What about all the poor suckers that installed an app that loaded legal software designed specifically to use their phone's connection for scraping a la brightdata? Residential proxies are big business.

That's a false negative, not a false positive. Maybe the site operator has a right of action against cloudflare for not doing their job against such actors, but you have no standing when you're blocked and they're not.

>Yes. Their entire business model is "we have a magic crystal ball that only stops 'the wrong people'™ from your website".

And do they actually claim 100% accuracy?

>They quite literally don't have that agency.

They can go with another anti-bot vendor. Competitors such as imperva or ddos-guard use similar techniques because it's the state of the art when it comes to bot detection.

>This goes back to "define bot". There are zero websites that would want to block me from making purchases from them and yet that is exactly the result in the end. I had to change vendors for a five figure order because I was up against a deadline and couldn't get around the cloudflare block from my office, and the vendor had closed for the night so I couldn't call them and bypass the whole mess.

>Afterwards we spent nearly a week trying to figure out how to let me buy from them again and they were willing to keep going back and forth with CF on my behalf but I was over it and not going to spend any more time. Now I'm using the non-CF vendor to their disappointment. So much for agency.

I'm sorry this happened to you, but any anti-fraud/bot system is going to have false negatives and false positives. For every privacy conscious person that's making a legitimate purchase using TOR browser and delivering to a different shipping address, there's 10 other fraudsters with the same profile trying to scam the site. This is an extreme example, but neither the business or cloudflare has any obligation to serve you.

>Good for you? I have a bone-stock computer on its own connection just to try to work around this BS and yet I still sometimes get an infinite loop where the checkbox never goes away.

What OS/browser (and versions of both) are you using?

>When I have my VPN to our euro office on I am 100% unable to access CF sites whatsoever. Been that way for as long as I can remember.

sounds like their residential proxy detection (that you were asking about earlier) is working as intended then :^)

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#215
post #161

Earlier quoted context omitted.

It really is a fantastic scam. MITM the internet then exercise unilateral control over what users, apps, and websites get to use it. Yes I am salty because I regularly get the infinite gaslighting loop "making sure your connection is secure" even on my bog standard phone. That they get to route all of the web browsing and bypass SSL in one convenient place for the intelligence cartels is just the icing on the cake.

No one is forced to use cloudflare for their site. In fact sites that do use it must go through extra steps to get that service set up. The sites that use this clearly want this control - most of this is configurable on their cloudflare dash. The fact that you blame Cloudflare rather than the sites that sign up (and often pay) for these features actually helps cloudflare - no site owner wanting some security wants to…

> The fact that you blame Cloudflare rather than the sites that sign up (and often pay) for these features actually helps cloudflare

Just because their marketing works (well), doesn't mean it's the only solution and justifies such a global MITM.

> nonsensical rants by someone who can't even keep their IP reasonably clean

Says who? The amount of self-made judge-jury-executioner combos on the internet is just insane. Why should we _like_ one more in the mix?

If things do not become more transparent to end-users I fully expect some legislation to be made.

Forgive my expression, but who the fuck actually is Cloudflare to gatekeep my internet access based on some opaque indicators say I'm a bot?

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#216
post #214

Just use some residential proxy network and slam your target. They can't detect you.

Cloudflare has probably noticed those proxy networks are quite expensive.

Sometimes get hit by the captcha but captcha solvers are cheap (0.3 cents a captcha).

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#217
post #121
post #107

Earlier quoted context omitted.

I’ve just taken to blocking entire swaths of cloud services IP networks. I don’t care what the intentions are, my personal sites don’t get the infinite bandwidth to put up with a thousands of poorly written spiders.

Is there a public list of those address blocks, which you'd recommend?

Set up a honeypot, or more like a booby trap, and boldly ban all IPs that access it.

Then you can consider banning OVH, DO, AWS, GCP, Oracle, China, Russia.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#218

This seems like a gimmick. Isn't preventing crawling a sisyphean task? The only real difference this will make is further entrenching big players who have already crawled a ton of data. And if this feature comes at the cost of false positives and overbearing captchas, it will start to affect users.

My website contains millions of pages. It's not hard to notice the difference between a bot (or network) that wants to access all pages and a regular user.

Oh you will not notice. The pages can easily be spread out between residential IPs using headless browsers (masked as real ones), unless you really pay attention you won't see the ones that want to hide.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#219
post #212
post #209

Earlier quoted context omitted.

> How is having a specific definition relevant to this conversation? Because it's a computer that automatically does it. That's the entire problem here. Humans are not in the loop, except collecting the paychecks. > An approximate definition of "a human using a browser to visit a site" probably suffices Humans are not doing the blocking. "Approximate" is not good enough when, for example, I need to go to a coffee sho…

>Because it's a computer that automatically does it. That's the entire problem here. Humans are not in the loop, except collecting the paychecks. I don't see how "Humans are not in the loop" is a relevant factor for whether something is a "criminal enterprise" or not. Humans are often not in the loop in approving loans/credit cards either. That doesn't make equifax a "criminal enterprise" for blocking you from gettin…

> At least looking at the response headers, digikey.com is served by akamai, not cloudflare

I edited them out because they were only one of many problem sites.

> Maybe there's something extra sus about your connection/browser, but I find it hard to believe that you have to resort to getting a separate computer and making a 10 minute trek to visit a site

Maybe half a decade ago someone had malware from my IP. Maybe my router's mac address was used by some botnet software somewhere. Maybe I'm on the same subnet as some other assholes.

> 3. like it or not, neither cloudflare nor digikey has any obligation to serve you. They can deny you service for any reason they want

The vendor in question (this one was not digikey) very explicitly wanted me as a customer.

> them denying you service on that basis doesn't mean cloudflare is running a "protection racket".

Them charging to correct their mistake is.

> that's clearly not an edge case

That's my point. I know for sure that vanilla android on t-mobile periodically gets the infinite loop in this area of my city. It usually goes away within a week but there's no rhyme or reason.

> What OS/browser (and versions of both) are you using?

I have seen it on linux windows and android.

> sounds like their residential proxy detection (that you were asking about earlier) is working as intended then :^)

I don't understand this. They have a normal ISP in a business district?

ETA: I have less issues on my home computer, which browser extension'd up, ironically enough.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#220

Earlier quoted context omitted.

No one is forced to use cloudflare for their site. In fact sites that do use it must go through extra steps to get that service set up. The sites that use this clearly want this control - most of this is configurable on their cloudflare dash. The fact that you blame Cloudflare rather than the sites that sign up (and often pay) for these features actually helps cloudflare - no site owner wanting some security wants to…

> The fact that you blame Cloudflare rather than the sites that sign up (and often pay) for these features actually helps cloudflare Just because their marketing works (well), doesn't mean it's the only solution and justifies such a global MITM. > nonsensical rants by someone who can't even keep their IP reasonably clean Says who? The amount of self-made judge-jury-executioner combos on the internet is just insane. W…

> Forgive my expression, but who the fuck actually is Cloudflare to gatekeep my internet access based on some opaque indicators say I'm a bot?

Cloudflare is in no way gatekeeping your internet access. Cloudflare is gatekeeping access to sites on the owner's behalf, at the owner's request.

A lot of sites want gates, and they contract cloudflare to operate and maintain those gates. If it wasn't cloudflare it would be some other company, or done in-house. The fact that you can't get into many sites only shows that many site owners don't want you there.

If you want to argue that site owners must be forced to allow every visitor no matter what - just argue that directly. Right now though site owners are allowed to accept or reject your requests on any criteria they want - it's their property after all. Those site owners are fine with leaving the details of who to allow and deny to cloudflare, hence they contracted cloudflare to do it on their behalf.

> Says who? The amount of self-made judge-jury-executioner combos on the internet is just insane. Why should we _like_ one more in the mix?

Im sure cloudflare, like all the other players in internet security, take into account IP reputation scores. It's a common and fairly effective tool.

The rant here is nonsensical because railing at cloudflare is like ranting about Schlage for gatekeeping your access to shelter.... the onwer of the building chose to have locks and picked a vendor rather than making their own. Much like cloudflare.... Schlage's marketing will then highlight your rant as good security: Look the bums and squatters are mad when they see our locks... do you really want to trust another vendor.

Another reason it's nonsensical is this:

> justifies such a global MITM.

It only does MITM on sites that sign up for cloudflare. It's not global - any site that isn't behind cloudflare is not MITMed. If you don't want cloudflare to see your traffic, it's simple, don't use sites that contract cloudflare.

Post reply on HN