Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creepy Crawlies

people.kernel.org

Post reply on HN
Print view View original

711–720 of 724 posts

Re: Creepy Crawlies

#711

Post by dspillett »

semiquaver wrote: post #631
dspillett wrote: post #429

Earlier quoted context omitted.

And yet you have not addressed the point that many report that tricks like Anubis work. If they are so stupid an idea that they could never work, why do they seem to having the desired effect?

I never said the approach was so stupid it could never work. I just shared how it’s annoying and how I worked around my annoyance. An even stupider approach would work exactly as well or better, like a form saying “type the letter y in this box to continue”. The only benefit is as a road bump that makes the site in any way custom. The moment anyone you are defending against so much as looks at the mechanics of solvin…

> I never said the approach was so stupid it could never work.

In that case I have read too deeply between the lines/tone. Sorry for the somewhat accusatory assumption of your opinion's details.

Re: Creepy Crawlies

#712

Post by sgsjchs »

inigyou wrote: post #703
sgsjchs wrote: post #691

Earlier quoted context omitted.

Maybe I misunderstand what you mean by "load the cookie". What is it if not Set-Cookie header with redirect status? That doesn't seem particularly special to me.

Set-Cookie header, 200 status and a JavaScript redirect? Many ways. Set-Cookie on pages that load quickly and check it on pages that load slowly?

What happens when a user without the cookie GETs a slow page? Does he get redirected to fast one to and back?

Re: Creepy Crawlies

#713

Post by account42 »

leohart wrote: post #698
whstl wrote: post #594

Earlier quoted context omitted.

My team runs a quite popular website, #1 or #2 in the market depending on the region. Several million visits per day. Around June/July we got a 10x boost out of nowhere, and it started affecting performance for users, increased hosting costs, and random bursts would bring the website down. We spent some time trying out solutions, from Cloudflare and Anubis to AWS, but it ended up affecting real users, and we got comp…

I am surprised POW Anubis doesn't work for you. Are these bots bypassing POW somehow?

POW never worked. The only bots its stops are the simple ones that don't support java script or cookies at all.

Re: Creepy Crawlies

#714

Post by trashb »

nxobject wrote: post #585
trashb wrote: post #575

Yea these AI bots are getting out of hand. The article mentions that some requested pages are less likely to be legit traffic (old commits) and more likely to have bot activity. Perhaps they could increase difficulty on those pages for the proof of work. Keeping the "current" at a lower difficulty allows most normal users to use the pages as normal, while penalizing the bots. One other way I have been thinking of is…

> One other way I have been thinking of is just delay the delivery of the pages, either limit bandwidth or just wait for a bit until you deliver the page. For one user a (lets say max)3s delay on some pages is not a huge deal, however at scale that adds up and means the client can't gather other pages in the meantime Or some kind of vintage “set up a request in a form and press a submit form, and we’ll pretend to tak…

However you can implement it. There are several options but probably you'll want to do it server side since I believe most bots don't run javascript. If you search there are ways to implement something similar in nginx using modules or buildins.

Re: Creepy Crawlies

#715

Post by leoqa »

inigyou wrote: post #509
leoqa wrote: post #385

Earlier quoted context omitted.

.. and we ban those device ids and move on. Your capital is lost.

Evidence shows otherwise. There are people making lots of money from these device farms. Right now. This isn't hypothetical.

I literally work on this. The problem is that enforcing device attestation impacts DAU so platforms don’t want to do it but it’s effective. Once the internet is unusable, device attestation will be required to access services. Ad driven businesses will start to see demand from ad buyers for non-bot traffic etc.

Re: Creepy Crawlies

#716

Post by golem14 »

cobbzilla wrote: post #710
golem14 wrote: post #705

Earlier quoted context omitted.

why not require an account for html access and otherwise just serve say a pre-cached complete repository requiring minimal work?

Because I’m lazy. It has HTTP basic auth. If anyone wants access they can message me directly.

Yeah, I meant for kernel.org

It's hugely inconvenient for users not having an account and want to do more than clone an entire repo via an API of course.

Re: Creepy Crawlies

#717

Post by lxgr »

done_lurking wrote: post #660
lxgr wrote: post #533

Earlier quoted context omitted.

Sure, but then you’re essentially just doing versions of the old “what is 2+2” text captchas with extra computational steps, so why not save yourself and your users those?

As gp said, it does provide some protection. If it didn't, nobody would use it and there would be nothing to discuss. Making snarky comparisons doesn't do anything.

People can remain irrationally convinced about the efficacy of non-solutions to their problems for quite a while, in my experience.

And I really wasn't trying to be snarky; people had "what is i + j" type of captchas for a long time and I still see them every once in a while!

Re: Creepy Crawlies

#718

Post by parineum »

inigyou wrote: post #704
parineum wrote: post #695

Earlier quoted context omitted.

The people doing the work aren't paying for AI, they're getting paid by AI. The people who are having the costs of users not paying "their fair share" (this phrase has officially jumped the shark) are the investors who are subsidizing these companies.

I'm getting paid for AI to scrape my stuff? That's news to me. How do I claim it?

Do you get paid when people "scrape your stuff"?

Re: Creepy Crawlies

#719

Post by Dylan16807 »

latentsea wrote: post #708
Dylan16807 wrote: post #506

Earlier quoted context omitted.

"It should write the code I tell it to in an interactive session. Also when running autonomously it shouldn't decide to hack into systems." I don't see much connection between that problem and these controls.

What's the difference when you prompt it versus when it prompts itself?

It shouldn't prompt itself to go outside its original scope. That would be a good place for controls.

But also I didn't say it should hack when I prompt it. For the purpose of this discussion that could go either way. To avoid the griping it just needs to write the code I want when I prompt it.

Re: Creepy Crawlies

#720

Post by NavinF »

p_j_w wrote: post #395
NavinF wrote: post #327

Earlier quoted context omitted.

turns out the overlap between "people who can't configure their webserver to serve at wire speed" and "people who can get law enforcement to take them seriously" is the empty set

Didn’t read the article, huh?

Oh I did. They're complaining about literally 14 cores across all of git.kernel.org. Definitely not in the "people who can get law enforcement to take them seriously" group.
Post reply on HN