Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

471–480 of 688 posts

Re: Creepy Crawlies

#471

Post by FridgeSeal »

lxgr wrote: post #130
nneonneo wrote: post #59

Earlier quoted context omitted.

I disagree. The kernel finds it effective - 66% of scrapers are turned away directly. The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site. If each new site they hit forces…

> 66% of scrapers are turned away directly. Until they discover this neat trick [1] and solve challenges orders of magnitudes more efficiently than legitimate users. The game theory of Anubis is not sound. It makes fundamentally less sense than Captchas, and even those have been on the way out for a while. [1] https://news.ycombinator.com/item?id=49500040

It doesn’t have to be fully sound, it’s there to provide _some_ degree of interim protection.

Evidently, it makes _enough_ of a difference for the people running it. Criticising it is just letting the perfect be the enemy of the good.

The alternative to Anubis isn’t “something better that isn’t being picked”, it’s “completely unprotected”, and the people criticising it for being imperfect could helpfully move the conversation forward by actively suggesting a concrete alternative.

Re: Creepy Crawlies

#472

Post by phyzome »

DrJThomasHusk wrote: post #368

When a hapless user visits my site well muahahahahah Sorry, just the thought of it But when they do… boy do I have a trap waiting for them. My wife calls me The Genius. I’m the guy she calls when her battery dies or when her instagram breaks like when it shows that random guy in her DMs, stupid bugs LOL I digress. Alas, when a user lands on my page. My page wants to know exactly 2 things: 1. Why are you here and who…

I don't see any relevant reference online to "Pharoah’s Fortune".

Re: Creepy Crawlies

#473

Post by zbentley »

CGamesPlay wrote: post #454
zbentley wrote: post #443

Earlier quoted context omitted.

That'd fail for three reasons. First, scrapers would start running JS. Whether they're running chromium-in-a-box or something more clever doesn't matter. Compute is effectively free for them, and even shitty WebOS set-top boxes can probably run a stripped-down headless browser with a JavaScript engine. Second, a full clone in browser memory is massive for something like the Linux kernel. Lots of browsers and pro-JS f…

1. Preventing access to open-source code by scrapers is not the goal behind this idea. The goal is to reduce the server load taken by the scrapers. 2. Why would you need a full clone to access /blob/cee9395acd8043be0644b25c34bfa86623f2b935/block/badblocks.c?

I don't think the show-commit or file-at-revision routes are what's causing the bot load.

Rather, the article describes the routes that compute views (e.g. show the change history across several commits) into history as being the issue. If we take that example, a JS client would have to fetch N individual commits/routes/objects and compute the requested view, but first it'd have to fetch the indexes/logs to determine what commits exist within e.g. a specified range.

I suspect that'd require more work on the frontend than "just run WASM-built git" ... unless the proposal is for it to fetch all requested objects lazily, in which case I think you'd be surprised how many files are read by git when answering a question like "show me the diff by user XYZ in file ABC on branch QRS between date 1 and date 2". That starts to get expensive to pull in the browser, and the bandwidth costs might start to hurt even if the backend now only had to serve cacheable dumb blobs.

Re: Creepy Crawlies

#474

Post by __MatrixMan__ »

inigyou wrote: post #263
__MatrixMan__ wrote: post #199

This is a fundamental flaw in the web. Since we treat a server's name as authoritative, anybody maintaining a replica needs to repeatedly hit that server to know if their remote version is up to date. If we trusted digital signatures on content instead of server names, we could have a model where a single bit of server load propagates to millions of interested parties. As it is the server must do something distinct f…

Or poll a merkle tree root hash, like, um, git

What I'm after is a way to get away from polling by server name. With a sufficiently interested audience I ought to be able to be offline except when I need to update the merkle root, meanwhile users can just serve the content to each other.

Re: Creepy Crawlies

#475

Post by jay_kyburz »

sgsjchs wrote: post #402
jay_kyburz wrote: post #336

Earlier quoted context omitted.

why is this not the answer? then you can also rate limit each cookie as well.

it'll load the cookie, make one request, move to a different ip, load the cookie, make one request, move to a different ip, ...

If you are discovering urls you have to wait for a previous request to finish. The rate limit should work. Requests without a cookie wait 2 seconds. Request with cookies can only make human scale number of requests per second? (1?)

If you have a thousands of IP addresses, and you know all the urls you want to request in advance, you can just request them all simultaneously I guess.

The next more advanced version is that URLs are unique to your cookie.

Users can't share urls anymore, but it might be a tradeoff worth making. Unique urls for each user.

You could probably still make this work, if you share your url with another user, they get the page, but heavily rate limited like a regular no cookie request. (a cookie url mismatch gets the rate limited version of the page)

Re: Creepy Crawlies

#476

Post by rvz »

tptacek wrote: post #424
rvz wrote: post #391

Earlier quoted context omitted.

Same here. But of course and as expected; 5 hours later, the GP comment offers no proper solutions for this "better computer science" they speak of. It shows they really don't care and they are just rage-baiting readers in their weekly grumpiness without any suggestions. Unless they come up with said "solution", they can completely ignored in their complaints.

I had a fairly long conversation with Xe privately about this and didn't feel the need to repeat it here, especially since (as the search box will tell you) I've already talked about this before. But, do go on.

We are not asking you to repeat the full conversation here. We just want the actual solution that was discussed.

So could you give a short elaboration on what was this "better computer science" solution that you came up with in your discussion?

Re: Creepy Crawlies

#477

Post by ygouzerh »

easton wrote: post #37

Side note: why are shallow clones evil? I always thought they were cheaper, but I guess that’s really just for my disk space. (since the server has to compute what blobs to give you instead of just “everything”?)

It's the point that surprised me the most! We always used shallow clones, to speed the CI, I didn't knew that it got that much impact server side!

Re: Creepy Crawlies

#478

Post by eloisius »

robotmay wrote: post #169

I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…

How do you reliably detect the bots? Instead of slowban, I would feed it something abhorrent like an archive of 4chan posts.

Re: Creepy Crawlies

#479

Post by stickfigure »

oasisbob wrote: post #185
stickfigure wrote: post #167

How about just stop offering a html interface to the code? This doesn't seem like a critical service. Let people clone the repo normally. If someone else wants to run a public HTML service, let them deal with the bots. If you really want to offer a web interface, put it behind login. You can apply enough restrictions (captcha, super slow rate limit for new accounts) that it isn't cost effective to generate zillions o…

I think you underestimate the difficulty in effectively gating user registration. If the anti-bot efforts in general don't work for other site pages, they won't work for signup functionality either.

I've been on the other side of this kind of thing (hero rather than villain, though I'm sure someone out there disagrees).

You can fairly arbitrarily increase the difficulty of user registration, far beyond what users will tolerate for viewing an individual page. You can exploit this; a bot needs to make many accounts for the activity desired, and it's not hard to make "generate account" more expensive than it's worth for the amount of activity they get from each account.

If it costs your attacker a penny to solve the captcha to make an account, and they can only get 100 pages out of an account, you win.

Re: Creepy Crawlies

#480

Post by nozzlegear »

eru wrote: post #444
pbronez wrote: post #193

Earlier quoted context omitted.

It’s more economical at a compute level, but not at the developer level. The moment you start customizing your crawler to use protocol X for site Y your scale story collapses.

> It’s more economical at a compute level, but not at the developer level. Developers and compute are interchangeable now.

Who's proompting the machine to do it differently without a developer there to ask the right questions?
Post reply on HN