Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare's new marketplace lets websites charge AI bots for scraping

techcrunch.com

Post reply on HN
Print view View original

231–240 of 280 posts

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#231

Post by IX-103 »

theyeenzbeanz wrote: post #98
Aachen wrote: post #83

Earlier quoted context omitted.

> gatekeep access to those who pay and those who don't, and that applied whether they are bots or people. I'm already constantly being classified as bot. Just today: To check if something is included in a subscription that we already pay for, I opened some product page on the Microsoft website this morning. Full-page error: "We are currently experiencing high demand. Please try again later." It's static content but i…

Lately I’ve been noticing captchas have been increasingly difficult day by day on Firefox. Checking the box use to go through without issue, but now it’s been starting to pop up challenges with the boxes that fade after clicking. Just like your experience, chrome has no hiccups on the same machine.

Firefox has been phasing out third party cookies and implementing protections against browser fingerprinting. Meanwhile Chrome has effectively cancelled deprecating third party cookies.

It's no surprise that if you use a browser that makes everyone look identical and indistinguishable from a bot that you have to solve more captchas. Welcome to the private web future you've always asked for...

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#232

Post by theamk »

neilv wrote: post #11

Cloudflare found a new variation on their traditional service of protecting from abusers. This time, Cloudflare has formed a "marketplace" for the abuse from which they're protecting you, partnering with the abusers. And requiring you to use Cloudflare's service, or the abusers will just keep abusing you, without even a token payment. I'd need to ask the lawyer how close this is to technically being a protection rack…

doesn't seem this way?

> Website owners can block all web scrapers using AI Audit, or let certain web scrapers through if they have deals or find their scraping beneficial.

You don't have to make any deals, or participate in the marketplace, "block all" is right there.

And if you are not using Cloudflare, you are going to be abused. This is a sad fact, but I have no idea why you are blaming Cloudflare and not AI companies.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#233

Post by selcuka »

diggan wrote: post #130
Aachen wrote: post #102

Earlier quoted context omitted.

Those "keep clicking until we stop fading in more results" challenges mean they're fairly confident you're a bot and this is the highest difficulty level to prove your lack of guilt. I get these only when using a browser that isn't already full of advertising cookies (edit: which, to be clear, I hope is still considered an acceptable state to have your browser in)

> Those "keep clicking until we stop fading in more results" challenges mean they're fairly confident you're a bot Those ones are the fucking worst. I've noticed that if I try to succeed in these captchas too quickly, it'll just say "Sorry, try again" even when every click was correct, so instead, I've started going in slow motion and faking "misclicking" which makes it much more likely to accept me as human. I canno…

> I cannot stand the idea that I have to pretend to be slower than I am, in order for a computer to not think I'm a computer.

It is not only about detecting if you are a computer or not. They intentionally waste your time (regardless of whether you are a human or computer) to make it unfeasible to scrape millions of pages. The actual "detection" part is relatively less important.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#234

Post by MathMonkeyMan »

epc wrote: post #107
flaburgan wrote: post #81

I was recently speaking with people from OpenFoodFacts and OpenStreetMap, and I guess Wikipedia as the same issue. They are under constantly DDoS by bots which are scraping everything, even if the full dataset can be downloaded for free with a single HTTP request. They said this useless traffic was a huge cost for them. This is not about copyright, just about bots being stupid and people behind them not caring at all…

I’ve just taken to blocking entire swaths of cloud services IP networks. I don’t care what the intentions are, my personal sites don’t get the infinite bandwidth to put up with a thousands of poorly written spiders.

I use a VPN when bittorrent is running, and I've found that several websites outright block me "for security reasons." They like to show me my IP address, too, like a great secret has been revealed and the SWAT team is on their way.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#235

Post by ed_mercer »

Avamander wrote: post #218
spiderfarmer wrote: post #49

Earlier quoted context omitted.

My website contains millions of pages. It's not hard to notice the difference between a bot (or network) that wants to access all pages and a regular user.

Oh you will not notice. The pages can easily be spread out between residential IPs using headless browsers (masked as real ones), unless you really pay attention you won't see the ones that want to hide.

How many scrapers are sophisticated enough to go this far though? Most of them are probably of bad quality and can be detected.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#237

Post by marcus_holmes »

> If you don’t compensate creators one way or another, then they stop creating, and that’s the bit which has to get solved

I'm not sure this is true. Maybe they stop creating commercial stuff for sale, and go do something else for money, but generally creative people don't stop creating just because they can't get paid for it.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#238

Post by jart »

sophacles wrote: post #220
Avamander wrote: post #215

Earlier quoted context omitted.

> The fact that you blame Cloudflare rather than the sites that sign up (and often pay) for these features actually helps cloudflare Just because their marketing works (well), doesn't mean it's the only solution and justifies such a global MITM. > nonsensical rants by someone who can't even keep their IP reasonably clean Says who? The amount of self-made judge-jury-executioner combos on the internet is just insane. W…

> Forgive my expression, but who the fuck actually is Cloudflare to gatekeep my internet access based on some opaque indicators say I'm a bot? Cloudflare is in no way gatekeeping your internet access. Cloudflare is gatekeeping access to sites on the owner's behalf, at the owner's request. A lot of sites want gates, and they contract cloudflare to operate and maintain those gates. If it wasn't cloudflare it would be s…

It's not even a very good padlock. Using Cloudflare makes you powerless to stop level 4 DDOS attacks, because Cloudflare isn't very good at preventing hackers from abusing their service as a means of amplifying them. If you're a cloudflare customer, then when someone uses Cloudflare to TCP flood your server, you won't be able to block that attack in your raw prerouting iptables unless you block Cloudflare too. Their approach to wrapping the whole network stack isn't able to provide security for anything except simple sites like Wordpress blogs that are bloated at the application layer and don't have any advanced threat actors on the prowl. Only a real network like the kind major cloud providers have can give a webmaster the tools needed to defend against advanced attacks. The rest of Cloudflare's services are pretty good though.

Re: Cloudflare's new marketplace lets websites charge AI bots for scraping

#240

Post by ComputerGuru »

Aachen wrote: post #116
ajsnigrutin wrote: post #113

Earlier quoted context omitted.

Aw man, you haven't seen the 'captchas' of arkose labs yet... those are a pain (twitter used to have them some time ago).

Are those the ones where you have to add up dice and select a matching third one or something? The ones GitHub used for registration, say, ~9 months ago? You're right! I forgot about those. A colleague and I tried to complete it independently but literally could not. One run would take multiple minutes and on the second try I was more diligent (taking even longer) and certain I did all the math correctly, but registr…

Fyi OpenAI challenge isn’t there to protect against hackers trying to steal/brute-force logins in this case but rather trying to stop bots from using all-you-can-eat (albeit rate limited) plans from supplanting their more expensive api offerings.
Post reply on HN