Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

Post reply on HN
Print view View original

251–260 of 450 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#251

Post by xena »

D2OQZG8l5BI1S06 wrote: post #244
johnorourke wrote: post #22

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

Because of this wasteful crap the internet is so slow nowadays... Try opening gcc bug tracker on your phone: https://gcc.gnu.org/bugzilla/

The GCC bug tracker uses the meta-refresh challenge, which does not require JavaScript. Due to the fact that the server makes sure the client has waited at least 75% as long as it should, the HTML has to add one second to the meta-refresh wait. Patches welcome. Meta refresh granularity is in single digit seconds.

Re: A year of fighting scrapers on my 1.5 million-page website

#252

Post by bakugo »

autoexec wrote: post #227
aomix wrote: post #155

Earlier quoted context omitted.

The social contract of accepting scraping for visibility was always tenuous and is now fully dead. But it’s shocking the number of people who are essentially victim blaming here. You should spend YOUR time to optimize your free website so MY use case is not impacted. How is that not incredibly selfish on its face?

The social contract for putting a website on the public internet involves following standards and making your website accessible. You SHOULD make sure that your website is readable to people no matter what user agent they are using as long as it's standards compliant. For example, making a website that only works in Edge and refuses to load for anything else would be a shitty practice. You are free to do it. There's…

The social contract for putting a website on the public internet was built on the assumption that browsers would be used to access those websites, and that most browsers would generally behave in a similar way (click on page, display page, pretty simple).

LLMs are not browsers, nor are they people, so I don't see why they should be part of this equation at all. They do not display the page for a real person to read, they merely ingest its contents and then spit out something completely different.

Re: A year of fighting scrapers on my 1.5 million-page website

#253

Post by everybodyknows »

speak_plainly wrote: post #72
storus wrote: post #9

Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.

Cloudflare offers a free plan that's fantastic. The free plan gives you effectively unlimited DNS/CDN traffic for a normal site, while the main practical cap is 100,000 Worker invocations per day, with 10 ms CPU per Worker request and a 100 MB request body limit. (If that means anything to you). The next tier up from free is $25/month or $240 per year. https://www.cloudflare.com/plans/ https://www.cloudflare.com/plan…

Does the free tier support geo-blocks? I'm experimenting with it -- running only a "hello, world" site thus far -- but haven't come across any offer of geo-block capability.

Re: A year of fighting scrapers on my 1.5 million-page website

#254

Post by mmh0000 »

ballooney wrote: post #169
ako wrote: post #138

Earlier quoted context omitted.

The best way to read the information on the internet today is via a LLM. Just like you don't process raw data in a data warehouse or datalake by looking at tables, you use a SQL or BI tool, to process all the information on the internet you need to tool to digest it for you. For many, today that tool is a LLM chat interface or agent.

This is such a grim thing to read.

I disagree. I personally consider these my biggest problems with the Web:

  - Bias, specifically commercial bias
  - Webpage formatting: every website looks different, hides the information I want in different places. Sure, it "looks pretty", but I don't want pretty; I want info
  - Scams/SEO/etc...
The LLMs are very good at reading from multiple sources, parsing, and presenting only the data in a consistent format. There are many examples of this, but if you want a good one to try for yourself:

Google "How to make ham fried rice"; you'll get 10,000 articles, most pretty good recipes. But they're all different; most of them are just bait for ads. And most of them, the 10-line recipe is hidden between 50 useless paragraphs about how serving food is life's most important goal.

Now, ask an LLM to search for it, find the best combination, and list only the recipes. You get a perfect, 10-line recipe that doesn't waste your time.

Re: A year of fighting scrapers on my 1.5 million-page website

#256

Post by thomashabets2 »

Something suspiciously absent from this article is addressing whether some of these are in fact human visitors, but humans issuing chatgpt or similar queries instead of going directly.

Is it really a bot if it's in response to a human asking for some aggregate information about charities, triggering a web search and then following the result links to get details for the human? Well, clearly yes it is, but it's a very different proposition from this article's implication that "they have no throttling on their scrapers"[1].

> Challenge 46 datacenter ASNs. Humans don't browse from AWS.

People who have workstations in the cloud do.

> The bots use 99% of the bill and I pay 100% of it.

Running a site this way is always a wallet-DDoS risk.

[1] though yes, by far most will be pure automation with no human in the loop. It's an assumption on my part, but feels like a safe one.

Re: A year of fighting scrapers on my 1.5 million-page website

#257

Post by duskdozer »

thorsson12 wrote: post #203

The experience of browsing the web has really suffered lately. The mandatory 3-4 second "verifying that you're a human" block from Cloudflare seem to show up on more and more websites. Seems like a questionable choice from Cloudflare to teach everyone to associate Cloudflare's logo with high latency...

A 3-4 second cloudflare wait seems good to me at this point. I had to just block any cloudflare requests because some pages that used it would just rev up a cpu core indefinitely and hang the browser.

Re: A year of fighting scrapers on my 1.5 million-page website

#258

Post by deathbob »

Terretta wrote: post #150
hk__2 wrote: post #117

Earlier quoted context omitted.

> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. No; in this case you are not a user, you are a bot user.

you know what a browser is called by the web site? check the header that tells the version. USER AGENT not user, an agent on behalf of the user. the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT. for a glimpse at one possible…

> for a glimpse at one possible futureverse, check out what the home assistant community is up to, how they assemble then update the ambient information displays on their walls. their USER AGENTS are doing what most HN-style "hackers" dreamed of reading scifi as kids.

I would love some pointers to read up on this, what are some examples of this? Genuinely, not being a troll or ironic

Re: A year of fighting scrapers on my 1.5 million-page website

#259

Post by ivanjermakov »

RattlesnakeJake wrote: post #32
johnorourke wrote: post #22

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.

Pay to change logo??? Just fork Anubis and remove one div... It's not SaaS first software, you're expected to deploy it on your web server.

Re: A year of fighting scrapers on my 1.5 million-page website

#260

Post by moralestapia »

skinfaxi wrote: post #158
moralestapia wrote: post #113

Earlier quoted context omitted.

>That is not the open web that I would like to see. Cloudflare is opt-in so I don't see that being an issue (yet).

Being opt-in doesn't negate the fact that it is closing the web.

I agree but it's a bit nuanced.

If you can still buy a domain, publish a site, and other people read it as usual, then the web is still open imo.

But I can see a lot of negative network effects if/when Cloudflare gets to control 60%+ of web traffic.

Post reply on HN