Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A year of fighting scrapers on my 1.5 million-page website

patronview.com

Post reply on HN
Print view View original

31–40 of 451 posts

Re: A year of fighting scrapers on my 1.5 million-page website

#31

Post by drum55 »

johnorourke wrote: post #22

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

Which is trivially bypassed by an actual implementation of the proof of work in non-javascript, rendering it absolutely useless. The website is approximately 3800x times slower than native code, and hundreds of thousands of times slower than the CUDA kernel claude wrote. The "proof of work" is just non existent at that point, they're solved in milliseconds for what would take the browser version 10 minutes or more, it's security by obscurity being dressed up as something more.

  pow_server  http://127.0.0.1:8080   backend avx512-x16
  ──────────────────────────────────────────────────────────
  uptime   00:03:12
  solver   ● BUSY  difficulty 9, 0.3s
  queue    [####################............] 5/8   peak 12
  ──────────────────────────────────────────────────────────
  accepted 1240        solved 1180
  503 shed 48      504 timeout 2      4xx/5xx 10
  ──────────────────────────────────────────────────────────
  last     difficulty 5  nonce 645376  in 9 ms  (101.6MH/s, avx512-x16)
  hashes   3.90GH total   avg 65.3MH/s
  Ctrl-C to stop
Claude even made a nice little API server for it after implementing midstate compression, AVX multi way hashing, and a CUDA kernel. This doesn't stop the literal LLM it's trying to block from solving the challenges, it's really annoying that everybody is using it and claiming that it's something that's usable in the real world as a result of it using proof of work. It's obscure, and obscure is fine so long as nobody is pretending that it is secure.

Re: A year of fighting scrapers on my 1.5 million-page website

#32

Post by RattlesnakeJake »

johnorourke wrote: post #22

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.

Re: A year of fighting scrapers on my 1.5 million-page website

#33

Post by pixl97 »

mysterydip wrote: post #27
behole wrote: post #13

Earlier quoted context omitted.

HN's blanket AI allegations are almost as annoying and tired as the thing they are rallying against. I think you are AI.

That’s exactly what an AI prompted to reply to AI accusations would say!

I'm not a bot, you're a bot!

So this is how AI wins, humans kill each other off because we might be bots and the bots inherit the earth.

Re: A year of fighting scrapers on my 1.5 million-page website

#34

Post by Bender »

Seems about right. I rotated my logs this morning. Most humans go to access.log and most bots go to botpoop.log. This is the line count:

     2 access.log [1]
    40 botpoop.log [2]
2 is really 1 since a human will grab the CSS file. Most bots do not bother with the style-sheet so that's a 40:1 bots to humans. I could cut that down by blocking data-centers but then I inadvertently block a lot of VPN's which I really don't need to do for a static compressed blog served from ram. The bots just get a TCP Reset but it's still fun to log and study them. The most interesting one I've seen recently is ReadYou which may be a reader but it appears to be much more, possibly acting as a cell phone distributed bot collecting data for a centralized site.

[1] - https://nochan.net/logs/access.log

[2] - https://nochan.net/logs/botpoop.log

Re: A year of fighting scrapers on my 1.5 million-page website

#36

Post by ashton314 »

storus wrote: post #9

Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.

I'm on a $4/mo droplet on Digital Ocean and my static site has been just fine. I'm using Caddy and it seems to handle the load like a champ. My site is very lightweight though, so ymmv.

Re: A year of fighting scrapers on my 1.5 million-page website

#38

Post by gfaster »

RattlesnakeJake wrote: post #32
johnorourke wrote: post #22

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.

I think that's partly why they do it? If you care about that, you should pay?

Re: A year of fighting scrapers on my 1.5 million-page website

#39

Post by Venn1 »

I'm blocking the Amazon search crawler, anything coming from Googleusercontent, and limiting AI crawlers to search rather than allowing AI assistants. The residential proxy waves are something to behold, but Cloudflare does an okay job catching those in the AI labyrinth. Still, it's all a bit silly, and I can't imagine what large sites deal with when I'm tangoing with this much nonsense on a small tech blog.

Re: A year of fighting scrapers on my 1.5 million-page website

#40

Post by inigyou »

RattlesnakeJake wrote: post #32
johnorourke wrote: post #22

Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software. [1] https://anubis.techaro.lol/

I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.

They're not stopping you. They're asking you politely not to.
Post reply on HN