Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

441–450 of 694 posts

Re: Creepy Crawlies

#441

Post by asddubs »

mzajc wrote: post #201

> Why is git.kernel.org “interesting” to crawlers I think the post underestimates just how little thought and effort is put into these bots. I also run a cgit instance with far less interesting projects, and am not spared from the deluge of HTTP requests. The explanation I could come up with is that they try to crawl all links regardless of how much sense it makes or how much load it causes. cgit being cgit, this mea…

I run a non source code service and I also get these, also getting hit by literally millions of residential IPs being cycled after 1 - 2 hits. Ingesting thousands of copies of near identical pages. The bots seem to be programmed in an extremely simple manner, using non-obfuscated javascript to write the href attribute seems to stop them from seeing links, where previously they would endlessly crawl generated links. Unfortunately it also means noscript users get shafted.

Re: Creepy Crawlies

#442

Post by CGamesPlay »

Git forges seem especially prone to this: tons of information, highly valuable to scrapers, rendered through several different lenses, gives a combinatoric explosion of URLs. Obviously scrapers could just be less stupid and clone the repo, but it's not happening.

It feels like turning these frontends into JS-only viewers would resolve this, for the most part. The JS clones the repo in memory and renders whatever lens the requestor wants, and the server becomes a dumb object storage that uses less resources. The anti-JS folks are free to clone the repo still, and view whatever lens they want, so that minuscule slice of the legitimate requests is still served, albeit with a degraded experience.

Re: Creepy Crawlies

#443

Post by zbentley »

CGamesPlay wrote: post #442

Git forges seem especially prone to this: tons of information, highly valuable to scrapers, rendered through several different lenses, gives a combinatoric explosion of URLs. Obviously scrapers could just be less stupid and clone the repo, but it's not happening. It feels like turning these frontends into JS-only viewers would resolve this, for the most part. The JS clones the repo in memory and renders whatever lens…

That'd fail for three reasons.

First, scrapers would start running JS. Whether they're running chromium-in-a-box or something more clever doesn't matter. Compute is effectively free for them, and even shitty WebOS set-top boxes can probably run a stripped-down headless browser with a JavaScript engine.

Second, a full clone in browser memory is massive for something like the Linux kernel. Lots of browsers and pro-JS folks wouldn't have the resources to run that.

Third, assuming that scrapers are willing to play the JS game, suddenly you have a massively increased rate of full clones happening. Even if serving raw git is cheap for your backend, the bandwidth the elevated clone count drives is not.

Re: Creepy Crawlies

#444

Post by eru »

pbronez wrote: post #193
paytonjjones wrote: post #157

Earlier quoted context omitted.

In this case, because there's a vastly more efficient economic path for the adversaries (cloning). They're not trying to engage in an arms race, they're trying to channel a racing river into its natural course.

It’s more economical at a compute level, but not at the developer level. The moment you start customizing your crawler to use protocol X for site Y your scale story collapses.

> It’s more economical at a compute level, but not at the developer level.

Developers and compute are interchangeable now.

Re: Creepy Crawlies

#445

Post by latentsea »

gizmodo59 wrote: post #342
semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

Side note.. Fable just rejected this. GLM 5.3 did without questioning me. 5.6 sol did it beautifully.

It's kinda interesting to see simultaneously the 'holy shit' response to the OpenAI / HuggingFace incident, and then the griping about Fable's controls regarding this.

Re: Creepy Crawlies

#446

Post by rpcope1 »

igor47 wrote: post #62
sethops1 wrote: post #33

Earlier quoted context omitted.

In a few years the VC money will dry up and this gross overspend on slurping data will end.

Visions of vast data centers surrounded by fields of browning grass, in which aging, rusting, formerly extremely expensive hardware is spending billions of compute cycles looking at anime catgirls

Sounds like an even shittier version of the Lorax. :(

Re: Creepy Crawlies

#447

Post by eru »

chr15m wrote: post #426
stjo wrote: post #386

Earlier quoted context omitted.

> I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. Lots of work has been done to do just that by monero’s proof of work algorithm. They achieved ASIC and GPU resistance by making you generate entire programs (with quite a bit of RAM usage) and finding one that produces a desired output. Then you submit the seed used for the program (a serie…

Monero is fine with making people wait, that's literally the whole basis of mining blocks in "crypto currency". It's tuned to the time required. Anubis wants "slow for bots but fast for humans" which is not possible if somebody has a farm of devices that costs them basically zero.

> [...] which is not possible if somebody has a farm of devices that costs them basically zero.

Opportunity costs are real.

Re: Creepy Crawlies

#448

Post by rstuart4133 »

inigyou wrote: post #197
semiquaver wrote: post #126

Earlier quoted context omitted.

OK, but that doesn't change anything. You have a large pool of IPs, each of which only needs to expend a handful of extra milliseconds of work to get unlimited access to the protected resource. Even if you had to solve a L6 challenge for every request it's faster than the total RTT time of most servers. In other words not a meaningful barrier. And L6 is already a level which severely interferes with human usage of a…

The way they're internally implemented doesn't allow pinning an IP. They buy a rotating proxy service from a vendor, and don't get to choose their source IP.

It's not hard to test. Go to a page that demands PoW, change your IP and see what happens. I just did it. Spoiler: kernel.org asks for a new PoW.

If the source IP was an issue, you could do it other ways: for example, make the cookie rotate on every access, and insist there is a single stream of accesses.

Re: Creepy Crawlies

#449

Post by afdbcreid »

inigyou wrote: post #302
ekidd wrote: post #285

Earlier quoted context omitted.

Largely because they're residential botnets in places like Brazil (a real example from one of my sites that was crawled to near-destruction). Someone could probably do something about this, but it's out of reach for individual site owners.

If you block Brazil, they'll find an alternative, maybe then you can sue them.

But they operate from neither. You can at most sue the one renting them IP addresses. Which will do basically nothing.

(Also, blocking a whole country is likely not what you do, but you probably know that).

Re: Creepy Crawlies

#450

Post by zbentley »

johneth wrote: post #143
NooneAtAll3 wrote: post #57

from what I see there are 2 solutions: 1) ban TV-proxy-as-a-service - straight up go to every representative there is and start pushing and lobbying and everything to stop spammers from distributing over non-computer devices, especially legally 2) make old commits more expensive to access than new ones. Legit users are not going to access those much, so they can pay the time. I assume diverse (unpredictable?) difficu…

If there was a way of detecting the proxy SDKs, it would be a simple matter of banning the device. Or at least naming and shaming the apps that use them.

I'm sure there are ways of detecting residential proxy clients that are reasonably effective. I'm equally sure that the people in possession of those techniques (thinking CDN providers, big social networks with public links, newsrooms) are unlikely to share their secret sauce in public. Doing that loses a competitive edge in cost-of-hosting, and as soon as you go public the residential proxy makers can adapt.
Post reply on HN