Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

611–620 of 711 posts

Re: Creepy Crawlies

#611

Post by gavinsyancey »

wongarsu wrote: post #100
semiquaver wrote: post #89

> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…

Isn't the issue here the three order of magnitude difference between the (presumably optimized) JS implementation and the optimized C kernel on your browser? If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution. Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them Fo…

The point of Anubis isn't to prevent bots from accessing a site, but rather to prevent dumb bots from accidentally DOSing a site.

If someone adds an Anubis solver to their spider, but then also makes it respect cache tags and stops crawling random old commits through a git-web interface, that's probably still mission accomplished.

Re: Creepy Crawlies

#612

Post by huhtenberg »

robotmay wrote: post #169

I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources. Currently I trick bad scrapers in…

If you know the IP, it's easy to tar scraper's requests.

However the article mentions that now they are being scraped 4-5 requests at a time from random IPs that are then never reused. It still probably possible to detect scrapers if they start by requesting obscure pages, but in general it's a much more difficult case.

Re: Creepy Crawlies

#613

Post by __MatrixMan__ »

inigyou wrote: post #517
__MatrixMan__ wrote: post #474

Earlier quoted context omitted.

What I'm after is a way to get away from polling by server name. With a sufficiently interested audience I ought to be able to be offline except when I need to update the merkle root, meanwhile users can just serve the content to each other.

Do you know Zooko's triangle?

Yes. I don't think human meaningful identifiers are important. We can make do with local pet names, like how Unison works.

We hardly even use the globally consistent names we have. Between humans its always "hey did you check out that website I sent you?" And never "hey did you check out https://foo.bar.biz"

Sharing and endorsement can just be part of the protocol. My reference to the site may as well be in terms of which of my friends think its trustworthy so I'm better enabled to make that determination myself (some of my friends have better taste than others).

Knowing that Alice.brownierecipe.2 and Bob.favoritebrownies refer to the same underlying document with an unfriendly global name is not so gnarly of a UX problem that we need to tolerate single points of failure to avoid it.

Re: Creepy Crawlies

#614

Post by Demiurge »

desterothx wrote: post #595
Demiurge wrote: post #303

Earlier quoted context omitted.

I think that to prove I am not a crawler I probably just need to supply a Safari user agent string. The crawlers on my sites actually self identify for the most part. The malicious ones are banned when they try to post spam. Anyway, what’s wrong with making 2 cents? :)

The troubling part for me is that a system like this could also just as easily be pawned onto the humans (I realize it technically already is with the subscription model internet, but there are still plenty of open sites). A feature like this seems to me will lead to an internet where no one can browse anything without having a wallet connected and paying for each attempted access. Exactly what the internet is not su…

I don’t see how this is different from having Stripe easily available to any web developer. I want this feature, but it doesn’t mean I’m going to add it to every website I build. I think of it as Stripe pay wall at infrastructure level rather than a JavaScript popup. If we look at NYT, WP, many already do this. But HuffPost and Reddit rely on different models.

I don’t think this will make the internet less open, I think the internet is already pretty much less open. Who blogs these days for free? They rely on medium or substack, and have no choice, they submit to these platforms rules about the articles they write, even though they’re pretty generous.

Now, if the reward system was as open as say, Bitcoin, I think there would be a lot more independent and different bloggers and journalists, and independent platforms.

Re: Creepy Crawlies

#615

Post by incognito124 »

Demiurge wrote: post #26

I maintain a formerly popular gaming website, and it used to have hundreds of legitimate requests per second. The load would be especially high during popular event times. So, it’s always been running on a dedicated server. It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and d…

bzflag

Re: Creepy Crawlies

#616

Post by desterothx »

ynniv wrote: post #74

this is an increasingly common situation. it goes something like: - i have a free, niche resource - it becomes too popular - i make it more efficient - now it's really popular, and people are "abusing" it - let's make them proof-of-work - ... and proof-of-work harder - but now "legitimate" users can't use it - ??? the core problem is that the average person uses a mobile device where work is expensive, and the "attac…

I mean generally what you said is true. But who is actually going to look at the linux kernel from their phone (probably just the maintainers in extreme circumstances)

Re: Creepy Crawlies

#617

Post by Backslasher »

forrestthewoods wrote: post #220

> Shallow clones are awful. Run your own damn mirror if you're going to do something nasty like that. TIL shallow clones are expensive. That's wild to me. It's supposed to be cheaper!

I thought they were expensive compared to fetches from established repos. TIL they're also expensive compared to full clones.

Re: Creepy Crawlies

#618

Post by schobi »

I'm puzzled as there seems to be a clear pattern on how a human user would look like vs a bot.

High bot likely hood if: If a session jumps to a different ip. If the session jumps IP after just a few requests. If a new blank session starts with a deep link. Maybe those are cases where some POW is better justified?

Assumption: the rendered HTML might be viewed by a legitimate developer, even via a deep link from outside. But rarely from a wget script without a session cookie .

The other nice idea from the comments - is this rendering effort something that can be pushed to the user? Instead of pointless POW work, can you offload the expensive rendering to the user side?

But certainly, this is just an armchair comment and the kernel guys certainly have tried everything in this arms race...

Re: Creepy Crawlies

#619

Post by desterothx »

ksimukka wrote: post #576

Remember SETI? Maybe we should serve the bots a problem worth solving and benefit both parties. They spend some energy/tokens on a problem and we pay them with content. If only I had a bot problem, this would be interesting to explore.

The trouble is still differentiating between real users and agents. Sure this could be useful (forcing agents to spend a certain amount of compute towards something actually useful), however real users that get stuck with it would be stuck for an even greater length of time, because the compute required to actually do something useful is beyond the reach of most consumer devices. An rtx pro 6000 costs .76$ per hour for renting, so quite a lot of compute needs to be done for it to be considered "worth it" for the website

Re: Creepy Crawlies

#620

Post by throwawayffffas »

8474_s wrote: post #596

The solution is allowing the convenient web interface only for trusted members, like e.g. 4 year old account can view all pages(with easy anubis setting) but guest/bots/everyone else has to git clone the thing and do it on their backend. IIRC old forums also limited 'content for registered/trusted/moderators/etc' content views decades ago, turns out this crap saves gigatons of traffic.

That excludes new legitimate users, just put up a very cheap paywall for the web interface, 1 dollar for lifetime access, that way each request is tied to a user and can be rate limited. While remaining essentially free.
Post reply on HN