Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

Post reply on HN
Print view View original

261–270 of 450 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#261

Post by baby_souffle »

hluska wrote: post #206
matsemann wrote: post #171

Earlier quoted context omitted.

I don't disagree, but there is a sliding scale here. For instance, I wanted to buy a piece of equipment the other day from a local company for a specific usecase. I wanted to find a specific price/weight/specs ratio, and asked an llm to loop through the 20 or so items, fetch their page and calculate and present some values for each. This then led to me going and buying the one I found. So the llm was mainly just an e…

I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you. They clearly don’t mind so what’s the issue? You still used a bot.

> I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you.

The point is that not all bots are bad. Assuming as much / implementing policies to that effect won't block all bots but it _will_ block the portion of bots that represent users considering giving you money.

I have a series of bots that monitor various eCom sites to monitor prices over time for big-ticket items I am considering as well as staples/groceries and everything in between. I have this little scrape/ingest pipeline because there's no other way to obtain this data... not even an API that I can pay for access to.

One of the large appliance sellers that I have in the scrape queue has gotten _hyper_ aggressive with bot detection to the point where even my personal head-full chrome instance doesn't always get to load the page. Guess who I will never buy that ~$2000 appliance from.

Re: A year of fighting scrapers on my 1.5 million-page website

#262

Post by cuu508 »

D2OQZG8l5BI1S06 wrote: post #244
johnorourke wrote: post #22

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

Because of this wasteful crap the internet is so slow nowadays... Try opening gcc bug tracker on your phone: https://gcc.gnu.org/bugzilla/

The Anubis challenge took ~5 seconds.

What do you propose instead?

Re: A year of fighting scrapers on my 1.5 million-page website

#263

Post by bakugo »

xena wrote: post #226
bakugo wrote: post #182

Earlier quoted context omitted.

It's open source, just ask your AI agent to change the logo.

Just keep in mind that changing the logo is a great way to move yourself down the priority list for bug reports and support.

Someone willing to take the 2 minutes to ask Claude to change the logo can probably also ask it to fix any bugs they find, or add any new features they might want.

Re: A year of fighting scrapers on my 1.5 million-page website

#264

Post by giantrobot »

aprdm wrote: post #237
sdellis wrote: post #219

Earlier quoted context omitted.

Because AI cannot be trusted as a reliable source of information. Information is more trustworthy when it comes from from primary sources on the open web.

Why not ? Can google be trusted ? Or facebook ? How do people have been accessing the internet for the last decade you reckon and how's this any different ?

> Can google be trusted ? Or facebook ?

No. Neither can be trusted as far as you can throw them. They're both incredibly invasive data brokers. Their customer facing products are just vehicles to show ads and Hoover up more PII and behavioral data on everyone.

Re: A year of fighting scrapers on my 1.5 million-page website

#265

Post by duskdozer »

throwaway219450 wrote: post #53
conartist6 wrote: post #29

Earlier quoted context omitted.

That's the thing: even if you don't use AI, if most of what you read is AI, soon this is what you'll sound like, just because it's so much of the training data your own brain's model has. Using it slowly sucks the uniqueness out of you.

Using AI to write blogs doesn’t bother me in principle, but I hate the prose that gets left in. Even if it’s not AI writing, being human doesn’t give you a pass for writing like a self-help guru. This sort of grammar is straight out of Claude: > A human on a VPN sees one CAPTCHA and passes, but a headless browser fleet sees a wall. I do wish we’d stop complaining about em dashes though, that’s lazy criticism. The sen…

Well, I definitely never saw nearly as many emdashes a few years ago. I didn't even realize it wasn't just a typeface quirk as I'd used hyphens a lot. I don't even know how to type an emdash on my devices and would have to copy/paste it from somewhere.

But yes your other points are absolutely right

Re: A year of fighting scrapers on my 1.5 million-page website

#266

Post by esseph »

inigyou wrote: post #213
esseph wrote: post #143

Earlier quoted context omitted.

Build a better service or better technology. If you can't, well then... We're stuck.

Until proven otherwise in your specific case, the better technology is just hosting directly on a VPS without Cloudflare. And if your 5$ VPS is maxed out, while serving any less than 10 requests per second, then you need to optimize your software before considering an upgrade.

Okay but the bots are loose and you have a bandwidth cap on most VPSs.

Because you have no CDN, all the bots are coming directly to you.

Because that's no CDN or distributed WAF, they're hammering your box.

Because your IP is directly exposed, you're also showing up on Shodan.

Your server may have enough CPU and RAM to handle the load, but does your monthly bandwidth allocation with your provider?

Many are finding previous bandwidth limits (starting around 10TB/month) are no longer enough.

Re: A year of fighting scrapers on my 1.5 million-page website

#267

Post by antonvs »

inigyou wrote: post #47
drum55 wrote: post #31

Earlier quoted context omitted.

Which is trivially bypassed by an actual implementation of the proof of work in non-javascript, rendering it absolutely useless. The website is approximately 3800x times slower than native code, and hundreds of thousands of times slower than the CUDA kernel claude wrote. The "proof of work" is just non existent at that point, they're solved in milliseconds for what would take the browser version 10 minutes or more, i…

But the people you're defending against don't do that. They also don't load CSS but for some reason the security theater PoW won the mindshare.

Security theater is what gets the economic rewards.

Re: A year of fighting scrapers on my 1.5 million-page website

#268

Post by IsTom »

moralestapia wrote: post #113
jwr wrote: post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

>That is not the open web that I would like to see. Cloudflare is opt-in so I don't see that being an issue (yet).

Internet walled gardens are all opt-in and still they've made the web a worse place.

Re: A year of fighting scrapers on my 1.5 million-page website

#269

Post by sethops1 »

mmh0000 wrote: post #254
ballooney wrote: post #169

Earlier quoted context omitted.

This is such a grim thing to read.

I disagree. I personally consider these my biggest problems with the Web: - Bias, specifically commercial bias - Webpage formatting: every website looks different, hides the information I want in different places. Sure, it "looks pretty", but I don't want pretty; I want info - Scams/SEO/etc... The LLMs are very good at reading from multiple sources, parsing, and presenting only the data in a consistent format. There…

And if you're lucky, it won't include rocks as an ingredient.

Re: A year of fighting scrapers on my 1.5 million-page website

#270

Post by baby_souffle »

deathbob wrote: post #258
Terretta wrote: post #150

Earlier quoted context omitted.

you know what a browser is called by the web site? check the header that tells the version. USER AGENT not user, an agent on behalf of the user. the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT. for a glimpse at one possible…

> for a glimpse at one possible futureverse, check out what the home assistant community is up to, how they assemble then update the ambient information displays on their walls. their USER AGENTS are doing what most HN-style "hackers" dreamed of reading scifi as kids. I would love some pointers to read up on this, what are some examples of this? Genuinely, not being a troll or ironic

> I would love some pointers to read up on this, what are some examples of this? Genuinely, not being a troll or ironic

There's a "look what I built!" thread every few days on the /r/homeassistant sub and the home assistant forums, too.

The data sources vary but it's usually a mix of devices that the user has locally and some remote data sources like weather/public transit schedules APIs or RSS and in some cases also utility rate info all integrated into a SPOG that is then displayed on some ambient screen (usually a hacked tablet or other cheap ARM device)

Post reply on HN