Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

Post reply on HN
Print view View original

141–150 of 453 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#141

Post by criley2 »

hk__2 wrote: post #117
jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

Scenario one: you use software to connect to their server and download a webpage. You are a user.

Scenario two: you use software to connect to their server and download a webpage. You are a "bot".

Make it make sense

Re: A year of fighting scrapers on my 1.5 million-page website

#142

Post by gruez »

Galanwe wrote: post #49
drum55 wrote: post #31

Earlier quoted context omitted.

Which is trivially bypassed by an actual implementation of the proof of work in non-javascript, rendering it absolutely useless. The website is approximately 3800x times slower than native code, and hundreds of thousands of times slower than the CUDA kernel claude wrote. The "proof of work" is just non existent at that point, they're solved in milliseconds for what would take the browser version 10 minutes or more, i…

The point of PoW access is not that its hard to bypass, it's that you cannot bypass it at scale.

>it's that you cannot bypass it at scale.

Define "scale". For any reasonable wait that you're willing to impose on your users, any PoW scheme heavily favors attackers. They have unlimited time and can be scraping even while they're asleep. Your visitors on the other hand don't have that luxury. You might argue that's not the point and it's only to stop dumb scrapers that are effectively ddosing your site, but if it's just dumb scrapers, you could've stopped them less onerous measures like tls or javascript fingerprinting.

Re: A year of fighting scrapers on my 1.5 million-page website

#143

Post by esseph »

inigyou wrote: post #59
rglover wrote: post #17

Earlier quoted context omitted.

All of my boxes are cheap VPS. Highly recommend people throw Cloudflare in front of their stuff. I switched all of my load balancers over to there and all of that bot crap went away. That combined with a proper UFW setup keeps the weather clear for me.

Highly do not recommend centralising the internet.

Build a better service or better technology.

If you can't, well then... We're stuck.

Re: A year of fighting scrapers on my 1.5 million-page website

#144

Post by skinfaxi »

hk__2 wrote: post #117
jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

I assume you are unfamiliar with the concept of user agents? Otherwise your browser would count as a bot user no?

And if not, what if it was a custom browser and not Chrome/Firefox/Edge?

Re: A year of fighting scrapers on my 1.5 million-page website

#145

Post by [deleted] »

RattlesnakeJake wrote: post #32
johnorourke wrote: post #22

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.

[deleted]

Re: A year of fighting scrapers on my 1.5 million-page website

#146

Post by paytonjjones »

aeddZX_0 wrote: post #116
jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

How do you monetise bot traffic?

https://blog.cloudflare.com/introducing-pay-per-crawl/

Re: A year of fighting scrapers on my 1.5 million-page website

#147

Post by elorant »

hk__2 wrote: post #117
jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

What if I’m a web alerts company? I crawl your content but my clients are all end users who actually see your content at your site.

Re: A year of fighting scrapers on my 1.5 million-page website

#148

Post by tonyhart7 »

jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

why you blaming cloudflare that try to solve botting issue and not the Botters ???

you literally can turn off cloudflare and use your own solution

Re: A year of fighting scrapers on my 1.5 million-page website

#149

Post by selckin »

criley2 wrote: post #141
hk__2 wrote: post #117

Earlier quoted context omitted.

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

Scenario one: you use software to connect to their server and download a webpage. You are a user. Scenario two: you use software to connect to their server and download a webpage. You are a "bot". Make it make sense

because they maintain the websites for social status, if the user never sees the specific website or knows they used it, you can't gain any social status

Re: A year of fighting scrapers on my 1.5 million-page website

#150

Post by Terretta »

hk__2 wrote: post #117
jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

you know what a browser is called by the web site? check the header that tells the version.

USER AGENT

not user, an agent on behalf of the user.

the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT.

for a glimpse at one possible futureverse, check out what the home assistant community is up to, how they assemble then update the ambient information displays on their walls. their USER AGENTS are doing what most HN-style "hackers" dreamed of reading scifi as kids.

the alternative is all your in home information owned by corpos when the info should be from user agents not corpo feeds. if you want to vote this idea down, you might be a corpo. :-)

Post reply on HN