Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

241–250 of 665 posts

Re: Creepy Crawlies

#241

Post by inigyou »

Demiurge wrote: post #119
andai wrote: post #30

Earlier quoted context omitted.

> And I think there is no reason to act like this race really has to be at such a breakneck speed. I would promote this idea to all of my competitors. Nah mate, you don't have to ask her out right now. You can wait until next week ;) Anyway, silver linings, looks like we're finally going to get widely adopted infra for microtransactions. https://web.archive.org/web/20030202042510/http://www.openp2...

I’m not a participant in this race. Are the AI companies worried about anything but their valuations? I don’t care about their valuations, but I do care about the risks that they are creating for the economy, society, and the technological advancement, at large. Micro transactions [in this case] are a great idea, these crawlers need to be taxed and made to pay for the unaccounted external costs. Furthermore, we need…

[to prove you are not a crawler and read this comment, please deposit $2.00]

Re: Creepy Crawlies

#242

Post by inigyou »

anal_reactor wrote: post #84
Demiurge wrote: post #26

I maintain a formerly popular gaming website, and it used to have hundreds of legitimate requests per second. The load would be especially high during popular event times. So, it’s always been running on a dedicated server. It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and d…

> Overall, it seems like the LLM are really straining the internet economy, the openness of it I feel like it's the same issue that public resources are facing. Step 1: public resource is freely available Step 2: asshole shows up, drains the resource completely just because they can Step 3: either resource is taken away, or it has massive security behind it

Alternative step 3: arrest the asshole.

We don't even need a new law. What they're doing is already a felony, and there's a process already in place to go to a court and request unmasking of 10000 IP addresses.

Re: Creepy Crawlies

#243

Post by TZubiri »

1oooqooq wrote: post #148
TZubiri wrote: post #111

Earlier quoted context omitted.

There's a winner takes all (monopoly) dynamic, if everyone uses your product, you can share the ip banlists between customers.

See, this is you falling for their marketing. It makes sense doesn't it? If you own all the pipes, you can effectively block bad actors. But in reality what they do is to just block everyone who blocks advertising tracking and profiling tech (such as canvas fingerprint etc)... and because all vendors use cf, they all block those same clients, and they think there's some winner takes all magic happening with finely tu…

If I recall by browsing the docs, there's a lot of fine grained client control over the blocking params. Which makes sense since the main product of cf is bot blocking.

So you can definitely block by IP, by fingerprint, both or neither

Re: Creepy Crawlies

#244

Post by inigyou »

Demiurge wrote: post #151
1oooqooq wrote: post #148

Earlier quoted context omitted.

See, this is you falling for their marketing. It makes sense doesn't it? If you own all the pipes, you can effectively block bad actors. But in reality what they do is to just block everyone who blocks advertising tracking and profiling tech (such as canvas fingerprint etc)... and because all vendors use cf, they all block those same clients, and they think there's some winner takes all magic happening with finely tu…

What would you suggest as an alternative? Block the bots yourself ?

Yes. Any number of methods work. From requiring a cookie to requiring CSS to be loaded to a PoW to a JavaScript redirect. Most scraper bots are really dumb. If you do it cookie based, you can also put a poison link on every page that invalidates their cookie and starts returning poison on every page after that, so they're damned if they do and damned if they don't.

Re: Creepy Crawlies

#245

Post by inigyou »

Waterluvian wrote: post #75

> It was immediately extremely effective — the bots just gave up. For a few months, it was bliss: bots were blocked at the perimeter and gave up, moving on to easier targets; the users were mildly annoyed but tolerated it, and the Anubis stack was easy enough to deploy everywhere. As a tech person who works with tech people, I have become extremely sensitive to this kind of bias. Is this solution actually better? Is…

Both groups exist. Scraper DDoS is likely to burn 100% of your CPU on git diffs if you host git. But static file sites are unlikely to notice it.

Re: Creepy Crawlies

#246

Post by TZubiri »

lxgr wrote: post #121
TZubiri wrote: post #106

Earlier quoted context omitted.

I thought we learned this with Litecoin, it works only as a novelty factor, nothing fundamental. When the reward to abuse crosses a certain threshold, a determined party will come along and completely dominate it.

I mean, it works just fine for PoW-based cryptocurrencies (with "why are we doing this if there are much better solutions to the same problem" values of "fine") because there the game theory works out, which is "solve hashes to gain a probabilistic lottery ticket for a shared prize pool" having a higher EV than "solve hashes for the probabilistic chance of double spending some money, but if you don't pull it off you…

I mention litecoin specifically, it was developed right after BTC ASICS started dominating, and the LTC goal was for hardness to reward consumer hardware rather than ASICS, it was briefly successful until LTC ASICS were developed.

Re: Creepy Crawlies

#247

Post by inigyou »

IshKebab wrote: post #85
MithrilTuxedo wrote: post #68

Earlier quoted context omitted.

Has anyone considered having Anubis perform more valuable hashing? Like, maybe you can't stop the LLM bots, but you can use them as one-off Bitcoin pool mining pool participants. You have to assume that making them find hash values with N leading zeroes has led to finding hash values with more than N leading zeroes. Maybe run a Bitcoin node under there and let each visitor take a couple swings for you with their pick…

Yeah that has been around for many years. Usually by sketchy download sites. It's not really going to help though because the scrapers using residential proxies aren't burning their own compute.

Yes they are. The proxy is just a proxy. All processing is central.

Re: Creepy Crawlies

#248

Post by inigyou »

marginalia_nu wrote: post #114
sethops1 wrote: post #33

Earlier quoted context omitted.

In a few years the VC money will dry up and this gross overspend on slurping data will end.

It's very likely the last few years of bot behavior is the consequence of the residential proxy business booming. This is indirectly due to AI company crawling, but the fact that they are as cheap and available as they are changes the incentives for anyone using them toward reckless and unsustainable request behavior, as there is no risk of burning your IPs, and very small chances of seeing any consequences of essent…

And the residential proxy business was created by Cloudflare, who was created by us using Cloudflare.

I've been on all three sides (user of RPs, getting paid to run an RP, and trying to block RPs from my site). Residential proxy service is nice. You can scrape anything, even with the dumbest curl command, and only get a Cloudflare block maybe 15% of the time, in which case you just try again. That's less often than I get a cloudflare block from using a privacy browser from a non-proxy address. Cloudflare does not stop bots, it stops humans.

Re: Creepy Crawlies

#249

Post by calvinmorrison »

I love to see the 'leet kernel hackers and maintainers' struggling with basic volume. Each page load should cost you near nil. Us lowly PHP developers have been caching shit for close to twenty years.

Learn how to cache your application and your cpu usage should be almost zero. In fact basically any read should cost nothing in comparison to writes.

Re: Creepy Crawlies

#250

Post by robotmay »

embedding-shape wrote: post #179
robotmay wrote: post #169

I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…

> I have an admin dashboard with a little leaderboard for which ones get the most stuffed You can't tease us like that without showing screenshots.

Haha here's a few snippets (will auto expire in a week, future readers):

https://ibb.co/mCgXPZPg https://ibb.co/qFxVRQGW https://ibb.co/9m6xV1xg

I only just added the token wasting measure so I'm really curious to see what happens if another bot loads over 400 pages.

Post reply on HN