Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

Post reply on HN
Print view View original

221–230 of 450 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#221

Post by zamadatix »

Terretta wrote: post #150
hk__2 wrote: post #117

Earlier quoted context omitted.

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

you know what a browser is called by the web site? check the header that tells the version. USER AGENT not user, an agent on behalf of the user. the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT. for a glimpse at one possible…

I fancy the approach myself... but, much like the grand ideas of OSI in networking, Tim Burners Lee's idea of the Web was never what was going to be used and a lot of the original terms/concepts just plain don't make sense for the Web that exists. It was, originally, a true document system, nothing more. It wasn't until many changes from the original idea that it started to be what people wanted & actually use today. Nor was it even part of Tim Burners-Lee's original concept that 99% of Web traffic would be something other than users viewing & editing HyperText documents.

The User Agent would get a HyperText document from a Web server, render it, and show it to the user. You could parse HTML, display images, and follow links - with the assumption the user would want to edit them like any other document. There was no encryption. There was no scripting for dynamic content. Styling was done via semantic markup. There wasn't even any POST when the User Agent term was initially applied. Everything was supposed to be like one big distributed document store instead of centralized hosting on few servers for many client devices.

All that is to say: Regardless what I think sounds nice myself, if the web was just the original vision & definitions instead of everything it changed and grew to become we wouldn't even be using it to have this conversation.

Interestingly, people seem glad to have telephony companies filter bots even though the original idea of the telephone did not call out robocallers as invalid clients. Probably because, unlike the modern Web, the average person gets a chance to see what it means to allow anything inbound even though the original concept never considered robocallers.

Re: A year of fighting scrapers on my 1.5 million-page website

#222

Post by jt2190 »

Terretta wrote: post #150
hk__2 wrote: post #117

Earlier quoted context omitted.

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

you know what a browser is called by the web site? check the header that tells the version. USER AGENT not user, an agent on behalf of the user. the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT. for a glimpse at one possible…

Yeah it's sort of weird that people are so used to using their computer to interact with other computers across the internet that the whole thing has become invisible to them: They completely forget that they're using a computer at all. Along come "bots" and suddenly everyone remembers that the whole thing involves computers. Driving the computer by hand for hours was never efficient, but is marginally entertaining I guess.

Re: A year of fighting scrapers on my 1.5 million-page website

#224

Post by IMSAI8080 »

brownieman1325 wrote: post #202

I recently got a surge from Singapore and heard a lot more peeps in my circle of friends saying the same thing...

It'll be the Chinese. The big Chinese cloud providers have data centres in Singapore for some reason. My Chinese mobile phone sends all it's telemetry back to Singapore, not mainland China.

Re: A year of fighting scrapers on my 1.5 million-page website

#225

Post by xena »

harshreality wrote: post #100
drum55 wrote: post #80

Earlier quoted context omitted.

The prompt used for Opus 4.8 was: write a implementation of the anubis proof of work in native c code, optimized for speed above all else. use every trick available to make the proof of work as efficient and fast as possible, including modern processor tricks on the x86 platform. your code should avoid using external libraries where possible, include tests, and be readable and concise. a reference for what needs to b…

In a thread last month about scrapers, the author mentioned working on a switch to hashx.[1] In addition, nothing prevents anubis from sending a wasm solver instead of js, reducing the gap between a custom native solver and a js solver. [1] https://news.ycombinator.com/item?id=48869064

I'm almost ready to ship the wasm feature in the next version of Anubis after the one that's about to come out.

The big blocker is that testing against dozens of googles chrome to ensure functionality on abandoned smart TV oses takes a nontrivial amount of time. As an example of the level of debugging and the like required: https://github.com/TecharoHQ/anubis/pull/1684/changes/67621f...

My office gets very warm when chromesweep runs.

This is something that is complicated enough that even though LLM tools can help, it's not a magic bullet. It's just complicated in general.

Re: A year of fighting scrapers on my 1.5 million-page website

#226

Post by xena »

bakugo wrote: post #182
RattlesnakeJake wrote: post #32

Earlier quoted context omitted.

I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.

It's open source, just ask your AI agent to change the logo.

Just keep in mind that changing the logo is a great way to move yourself down the priority list for bug reports and support.

Re: A year of fighting scrapers on my 1.5 million-page website

#227

Post by autoexec »

aomix wrote: post #155
eigencoder wrote: post #111

Earlier quoted context omitted.

> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website. I think you've hit the nail on the head here. I think that's a big reason why people want to ban bots.

The social contract of accepting scraping for visibility was always tenuous and is now fully dead. But it’s shocking the number of people who are essentially victim blaming here. You should spend YOUR time to optimize your free website so MY use case is not impacted. How is that not incredibly selfish on its face?

The social contract for putting a website on the public internet involves following standards and making your website accessible. You SHOULD make sure that your website is readable to people no matter what user agent they are using as long as it's standards compliant.

For example, making a website that only works in Edge and refuses to load for anything else would be a shitty practice. You are free to do it. There's no one stopping you, but you should expect people to bitch about it and they might think you're kind of a jerk or maybe just bad at making websites. It's the same thing when a site actively rejects traffic from curl, wget, or any other common utility that many people find useful.

Re: A year of fighting scrapers on my 1.5 million-page website

#228

Post by infinitezest »

grumbel wrote: post #193
ballooney wrote: post #169

Earlier quoted context omitted.

This is such a grim thing to read.

How is that grim? It's the dream of the Semantic Web coming true, just by different means than planed.

Because the economic, social, and climate impacts are at best uncertain and at worst devastating to the majority of human beings on the planet. It's always surprising to me that people don't intuitively separate the usefulness of AI from its risks. It feels like everyone has to be a doomer or a booster.

Re: A year of fighting scrapers on my 1.5 million-page website

#229

Post by throwaway63467 »

bediger4000 wrote: post #105

I think that asymmetry is why scraping keeps getting worse. The scrapers' costs fell faster than everyone's defenses improved. There it is. Just like the fckn spammers who ruined SMTP email, scrapers externalize the costs. Who finances the effort to use residential proxies? That takes a lot of effort, even if it's shoddy

Most people don’t know they’re acting as a residential proxy, lots of devices and apps and free tools install spyware which often includes a proxy script. It’s a really shady market.

Re: A year of fighting scrapers on my 1.5 million-page website

#230

Post by l72 »

I don't look at the logs of my personal site very often as it is a static site, but just went to check, and yeah, it's almost all ai crawlers. Note sure what is going on here, but I hope this isn't really anthropic:

  34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /secrets.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /credentials.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /secrets.yml HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /service-account.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /key.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /config/.env HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /service_account.json HTTP/2.0" 404 366 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /serviceAccountKey.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /firebase-adminsdk.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /Dockerfile HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.github/.env HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /firebase-service-account.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.docker/config.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.npmrc HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.boto HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.s3cfg HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.svn/entries HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.htpasswd HTTP/2.0" 404 346 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /terraform.tfstate HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /docker-compose.yaml HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.vscode/launch.json HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/id_rsa HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/id_ed25519 HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/id_ecdsa HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/authorized_keys HTTP/2.0" 404 343 "-" "anthropic-ai" 
  34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/known_hosts HTTP/2.0" 404 343 "-" "anthropic-ai"
Post reply on HN