Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

401–410 of 449 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#401

Earlier quoted context omitted.

No, the first case I used their website in a way that gave them a sale, which is why they have it. Why is it a straw man case when it's reality? Don't argue in bad faith, please. Express your reasoning, like I did, don't just dismiss me because you disagree..

You are assuming that because you are giving the operator money that you are entitled to use their website however you please -- that's not how it works. If you enter a brick & mortar establishment and break their rules, even as a paying customer, you might risk e.g. being kicked out and banned, depending on the behavior. This is not unique to e-commerce To be fair, I do not generally support wholesale banning of scr…

I think he's saying that his use of the site, via the LLM, is consistent with what the site owner-operator likely desires.

To give a concrete example: I am a physician and have a site for my private clinic. I am not only unopposed to being scraped by bots/LLMs, I prefer it. The only value I gain from users being on my site is referral to my practice, and that same information being surfaced by search engines and LLMs just means I'm getting in front of more potential patients.

Re: A year of fighting scrapers on my 1.5 million-page website

#403

I just checked Cloudflare for SignalBloom ( https://www.signalbloom.ai , which I own and operate). Over the last 72 hours, Claude-searchbot [1] alone fetched ~205,000 pages. Sent exactly 1 referral. There is a lot of free financial data on the site, hoping for real users to benefit from it. It is hard to not feel a little cheated out that Claude gets to claim "Found it!" to its users without me getting no credits or…

FYI, your proof is hosted on a NSFW page. You should have mentioned that.

Re: A year of fighting scrapers on my 1.5 million-page website

#404

I just checked Cloudflare for SignalBloom ( https://www.signalbloom.ai , which I own and operate). Over the last 72 hours, Claude-searchbot [1] alone fetched ~205,000 pages. Sent exactly 1 referral. There is a lot of free financial data on the site, hoping for real users to benefit from it. It is hard to not feel a little cheated out that Claude gets to claim "Found it!" to its users without me getting no credits or…

FYI, your proof is hosted on a NSFW page. You should have mentioned that.

Reason #13843 for using an ad blocker.

Re: A year of fighting scrapers on my 1.5 million-page website

#405
post #11

> There's a real conflict here. I want Google and Bing and DuckDuckGo to crawl my site and send me new readers. But I don't want everyone else strip-mining it. Kinda sounds like we're missing a peer to peer network here. Instead of downloading the same data over and over again we can just download it once and then share it. Wouldn't that be better for everyone involved? It would also function as a distributed WayBack…

Have you checked out IPFS?

Re: A year of fighting scrapers on my 1.5 million-page website

#406
post #101

The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see. A second side effect of a knee-jerk reaction to bots…

> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website

Yes, that's the social contract. Bots are not a part of it

Re: A year of fighting scrapers on my 1.5 million-page website

#407
I had similar problems with my free movie festival ratings database, so eventually I had to get rid of the dynamic lookup, and I only dump now static sites, hosted on GitHub Pages. I could not fight the AI bots for free, the hosting CGI constantly ran out of memory.

Re: A year of fighting scrapers on my 1.5 million-page website

#408

Earlier quoted context omitted.

> I would love some pointers to read up on this, what are some examples of this? Genuinely, not being a troll or ironic There's a "look what I built!" thread every few days on the /r/homeassistant sub and the home assistant forums, too. The data sources vary but it's usually a mix of devices that the user has locally and some remote data sources like weather/public transit schedules APIs or RSS and in some cases also…

It's an interesting example. People did very similar things during the "desktop shell modification" scene in the early 2000s, by far the most common thing people modified their desktop shell to do was display a bunch of charts and graphs of semi-random information, usually about their local computer. It was useless in 99.9% of the cases. You don't need or want to have constant access to how much free ram your compute…

I had to write a gnome shell extension to see my various LLM token budgets. Green, orange, red for Claude, OpenAI, GitHub Business, Kimi, DeepSeek, and such. Very useful.

Re: A year of fighting scrapers on my 1.5 million-page website

#409
This will resonate with anyone who operates a public-facing website and doesn't work at a big tech company. In the past few years I have also been down this exact same road. I'm desperately trying to avoid resorting to Cloudflare, but I'm running out of time and patience to keep tweaking nginx and firewall rules every few weeks.

I had initial success with https://git.gammaspectra.live/git/go-away as a more powerful and more reasonable self-hosted alternative to Cloudflare than Anubis, but it seems to have gone unmaintained. There is also https://github.com/dgl/haphash if you run HAProxy, though I have not tried it.

On that note, does anyone know what happened to Ted Unangst aka tedu? He was a prolific OpenBSD developer and blogger, and he had developed one of his own simple solutions https://humungus.tedunangst.com/r/anticrawl, but all his web properties seem to have gone away recently...?

Re: A year of fighting scrapers on my 1.5 million-page website

#410

Something suspiciously absent from this article is addressing whether some of these are in fact human visitors, but humans issuing chatgpt or similar queries instead of going directly. Is it really a bot if it's in response to a human asking for some aggregate information about charities, triggering a web search and then following the result links to get details for the human? Well, clearly yes it is, but it's a very…

> Running a site this way What do you mean "this way". What other way is there to run the site?

For the age of the Internet, running a website metered is a pretty new phenomenon.

Or at least this directly metered. 20 years ago maybe you would pay for your monthly 95th percentile bandwidth use. Not saying that was great either, but it was certainly more managable than what's common now.

Post reply on HN