Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

291–300 of 684 posts

Re: Creepy Crawlies

#291

Post by oowa »

mik1998 wrote: post #17
chuckadams wrote: post #8

Given the nature of git, wouldn't all that HTML be highly cacheable? I get that's not free either, but it's got to be a lot less intensive than having cgit generate it every single time.

The cache gets evicted between the times the old commits are revisited.

using the current solution and a cache pattern doesnt solve it, but it is an "affordable" solvable problem. but first we need to ignore OP said its ok for now...have a hackathon to solve for this. OP says it's not a problem for him right now but if we extrapolate what he's talking about it's definitely a problem aaaaaaand It's totally solvable, Even with all of the crazy combinations he's talking about it's still solvable. And it's already been solved using patterns we see in streaming services. This is completely hackathonable. but why do we even need to bother with this? The slurpers are the cause of this, and they can cause this problem because of Murphy's Law. well you can only account for Murphy's Law with good architecture or something like that or whatever. Ha ha hackathon.

Re: Creepy Crawlies

#292

Post by yorwba »

tptacek wrote: post #67
nneonneo wrote: post #59

Earlier quoted context omitted.

I disagree. The kernel finds it effective - 66% of scrapers are turned away directly. The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site. If each new site they hit forces…

It's weird to believe data center based, Internet-scale scraping operations will be less able to allocate compute to proof-of-work challenges than individual users. This is design problem with things like Anubis: proof-of-work depends on a cost asymmetry between attacker and defender. But in scraping, both legitimate users and scrapers get the same value out of a transaction.

For PoW to work, you don't need scrapers to have less compute than users, you only need their compute advantage to be less than the load-generation advantage they would otherwise have. And unlike legitimate users, scrapers aren't carefully picking the URLs they request, so the average value they get out of it is probably rather low. They have to make it up in volume.

Re: Creepy Crawlies

#293

Post by oowa »

kijin wrote: post #21
teo_zero wrote: post #10

Earlier quoted context omitted.

But there are "cubic bazillions" of possible URLs that are being requested. Even if they boil down to "only" some millions actual commits, their rendered HTMLs are all different.

Yes, this is very difficult to solve for sites with many URL variations, like git repos and heavily threaded forums. Your cache is always full, but the hit ratio is abysmal.

no its only difficult for you. and only at this moment... any minute now you will see the way. btw, the OP is just saying "its OK for now". And the OP is just telling us: this is what has been happening... maybe its difficult for OP also, but they didnt say that. they just said their current setup cant handle it. old tech...have a hackathon to solve for this. OP says it's not a problem for him right now but if we extrapolate what he's talking about it's definitely a problem aaaaaaand It's totally solvable, Even with all of the crazy combinations he's talking about it's still solvable. And it's already been solved using patterns we see in streaming services. This is completely hackathonable. but why do we even need to bother with this? The slurpers are the cause of this, and they can cause this problem because of Murphy's Law. well you can only account for Murphy's Law with good architecture or something like that or whatever. Ha ha hackathon.

Re: Creepy Crawlies

#294

Post by inigyou »

TZubiri wrote: post #104

Same problem we've been having for ages. Using shared ip banlists is the best solution so far, like cloudflare. Sure maybe they hit your server for 5 seconds and then desist, but they'll attack someone else, and they'll eventually rotate. I'm not sure if Anubis has a feature for centralized banlists, but I'm assuming since it's OS and privacy oriented, there isn't. There's a tradeoff between privacy and abuse, you wa…

Cloudflare doesn't block bots.

Re: Creepy Crawlies

#295

Post by inigyou »

okanat wrote: post #218
leoqa wrote: post #122

It seems clear to me we are moving towards a world where you will have to perform device attestation to access the internet. The spam/abuse is too great and accelerating.

What prevents people from obtaining or buying such devices and automating them? Using TVs as proxies is just one example of that. People will be willing to give their ID cards away too, if you pay them or beat them enough.

In fact they already do this. Buying 100 android phones and chargers is cheaper than reverse engineering whatever you're trying to automate - or was, before AI.

Re: Creepy Crawlies

#296

Post by inigyou »

jwilk wrote: post #222
grep_it wrote: post #123

Earlier quoted context omitted.

Did you read the article? “[…]because we can generate 1.2 METRIC BAJILLION valid URLs just for a single fork of linux.git.”

From the HN guidelines https://news.ycombinator.com/newsguidelines.html >: > Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".

What if it's really obvious they didn't read the article?

Re: Creepy Crawlies

#297

Post by inigyou »

jopsen wrote: post #136
znnajdla wrote: post #58

Put a CDN in front and let them absorb the load? Seriously, this is static content, which is so cheap to serve it should be free.

They allow you to diff commits, which is an awesome feature. But if bots a crawling diffs between all possible commits it's crazy. CDN will do nothing, because it's new urls each time. You can maybe find a CDN provider that block bots.

No CDN blocks bots. Some claim to. None do.

Re: Creepy Crawlies

#298

Post by alkonaut »

Arubis wrote: post #165
alkonaut wrote: post #161

Proof-of-humanity can’t come soon enough. We’re talking about privacy-preserving proof of age, but as we see here the real utility of such a system will be proof of humanity.

How do you define humanity? How do you ensure it includes every human? How do you ensure it doesn’t include every non-human? I’m not even asking about computation or algorithms. I straight up don’t think you can make a definition that isn’t a tautology or an approximation. Both of which are useful, but neither of which can fit a _proof_.

Every place on earth has some legal definition of who is human. The system I’m thinking of isn’t a technical/captcha one, it’s a human curated list of humans. Just an electronic ID. Those already exist but the challenge is making them (acceptably) privacy-preserving.

I want to take my existing national digital ID and use it online basically. BUT I don’t want the websites to know it’s me. Just that I’m human (or perhaps over a certain age). And I don’t want the ID issuer to know what site/service asked whether I’m a human or I’m 18 etc.

Re: Creepy Crawlies

#299

Post by oowa »

teo_zero wrote: post #10
Zariel wrote: post #9

Earlier quoted context omitted.

That was my first thought, varnish (vinyl these days) in front of the website should dramatically reduce this as the git repo should be practically static for most of the content.

But there are "cubic bazillions" of possible URLs that are being requested. Even if they boil down to "only" some millions actual commits, their rendered HTMLs are all different.

have a hackathon to solve for this. OP says it's not a problem for him right now but if we extrapolate what he's talking about it's definitely a problem aaaaaaand It's totally solvable, Even with all of the crazy combinations he's talking about it's still solvable. And it's already been solved using patterns we see in streaming services. This is completely hackathonable. but why do we even need to bother with this? The slurpers are the cause of this, and they can cause this problem because of Murphy's Law. well you can only account for Murphy's Law with good architecture or something like that or whatever. Ha ha hackathon.

Re: Creepy Crawlies

#300

Post by inigyou »

TeMPOraL wrote: post #267
inigyou wrote: post #238

Earlier quoted context omitted.

Which part of computer science is about getting DDoSed by billionaires who are above the law?

Where on Earth do people get the belief that: - It's the SOTA companies doing it? - Scrapers are doing it for training data ? Those are two assumptions I see in posts and threads around Anubis, that are taken at faith, and never once substantiated.

What's your alternative hypothesis?
Post reply on HN