Live data from Hacker News

The bot situation on the internet is worse than you could imagine

gladeart.com

151–160 of 179 posts

Re: The bot situation on the internet is worse than you could imagine

#151
My blog gets this degree of scraping too. I have some 5 million requests over the same period as they say they got 7 million over and I barely noticed before I put Cloudflare in front to cache and now I don’t notice at all. I have the Cloudflare AI stuff turned off and mostly use it through the tunnel so I don’t have to expose my local IP.

Is this actually a problem? Most of my requests claim to be Amazonbot but someone showed me they weren’t and I’ve forgotten how.

Re: The bot situation on the internet is worse than you could imagine

#152
post #136

Earlier quoted context omitted.

I dont understand what you mean. Are you saying that in C you only calculate the first few digits of the hash? That's not how sha256 works. Edit: oh i think you mean in c the string comparison short curcuits. I would expect the same to be true in javascript too. Its true in most languages. Maybe you are just worried about general language overhead, which is a fair point. Is the anubis check even using multiple thread…

The language matters, but your original guess was actually correct, you can do tricks with sha256 where you only end up calculating a fraction of the total double hash in order to get a pass or fail. Modern bitcoin miners do a double sha256 hash and increment in just a little bit more than a single hash of work. The input is 80 bytes, which is two compression rounds of 64 bytes in sha256, only the data in the second…

Oh good point. It looks like anubis is using 64 byte random plus a nonce. Unless i'm missing something it seems like using 56 bytes random plus 8 byte nonce would be a better design or maybe hmac'ing the nonce & data together.

But then again using sha256 doesn't make sense at all.

Re: The bot situation on the internet is worse than you could imagine

#153
post #57

So the elephant in the room: How much of HN is bot generated? Those who know have every incentive not to share and those who don't have no way to figure it out. At this point i have to assume that every new account is a bot

That's a good point ... it's trivial to have an agent post something onto HN on your behalf, so even old accounts are not immune. It's just the nature of things now, until we get better technology to assign some sort of "ALIVENESS" attestation to folks without revealing identity too much.

Re: The bot situation on the internet is worse than you could imagine

#154

Earlier quoted context omitted.

The Anubis difficulty setting is (or was) so high that nobody could visit the site without leaving it open for minutes or hours.

>Anubis was enabled in the tar pit at difficulty 1 (lowest setting) when requests were pouring in 24/7 >difficulty 1 (lowest setting) literally in the comment you're responding to

The actual site did not have it at difficulty 1.

Literally discussed throughout the whole comment section.

Re: The bot situation on the internet is worse than you could imagine

#156

Earlier quoted context omitted.

Don’t worry, man, once AGI is here you’ll get your allowance (or whatever the hyperscalers plan is).

You’ll enjoy painting or some other art even if you aren’t interested in the arts. That’s what I’ve seen written about it.

Or they'll let you starve to death which is way way easier and way faster for "them"

Re: The bot situation on the internet is worse than you could imagine

#157

Earlier quoted context omitted.

The problem here is that pretty much every part of modern life has been government and capitalism riding in the same cart - from cities installing electric power stations 100 years ago, to roads and inventions like the transistor and internet itself was government and private capital working towards common goals. The issue is we want “good” government and “good” corporate behaviour but not the bad. And knowing the di…

> The problem here is that pretty much every part of modern life has been government and capitalism riding in the same cart - from cities installing electric power stations 100 years ago, to roads and inventions like the transistor and internet itself was government and private capital working towards common goals. And here is your argument’s fatal flaw: mistaking public works , that benefits the commons and society…

Just a sec - you said: “”” Remember: when government and capitalism rides in the same cart, it is called corporatism, and is the basis of Fascism. ”””

The implication being when gov and big business get together bad things do happen. My point is they can happen yes, but also same players can make good things happen - it depends on the players (and regulatory and reporting and voting and and and )

Government tracking can be good - tracking covid cases good, tracking criminals planning robberies good. It it can also be bad.

This is imo why this is such hard problem - there is no clear answers only something we want sensible courts to (quickly) decide upon based on good laws.

Re: The bot situation on the internet is worse than you could imagine

#158

Knew it was getting bad, but Meta's facebookexternalhit bot changed their behavior recently. In addition to pulling responses with huge amplification (40x, at least, for posting a single Facebook post to an empty audience), it's sending us traffic with fbclids in the mix. No idea why. They're also sending tons of masked traffic from their ASN (and EC2), with a fully deceptive UserAgent. The weirdest part though is th…

> it's sending us traffic with fbclids in the mix. No idea why. The click IDs are likely to make the traffic look more like a human who has clicked a link rather than a bot? That way it gets past simple filters that explicitly let such requests in before bothering to check that the source address of the request seems to be a DC rather than a residential IP. > citing the competitive environment All the companies are c…

> All the companies are competing to be the biggest inconvenience to everyone else while scraping as much stuff as they can.

So the invisible hand is just a middle finger?

Re: The bot situation on the internet is worse than you could imagine

#159
post #129

Earlier quoted context omitted.

Thanks for sharing the perspective here. I think a lot of folks on HN have rightly said that a lot of the problems with the modern internet are due to the ad-supported business model. I don't think you were ever going to move away from it voluntarily -- too many people support it, even if they grumble about it. But maybe (and likely for worse) LLMs will finally kill this model.

I would love for the ad-supported model to die. I hate ads, and I hate having to serve ads. We get some subscription users but nowhere near enough to cover costs. Unfortunately, what I think will happen - and indeed already is - is that the AI companies themselves will replace much of the WWW. Sites like the one I am talking about will cease to exist. AI companies, once they can no longer scrape (steal) the data will…

Evidently, users and customers like not having to sift through hundreds or thousands of independent websites.

Re: The bot situation on the internet is worse than you could imagine

#160

Earlier quoted context omitted.

> it's sending us traffic with fbclids in the mix. No idea why. The click IDs are likely to make the traffic look more like a human who has clicked a link rather than a bot? That way it gets past simple filters that explicitly let such requests in before bothering to check that the source address of the request seems to be a DC rather than a residential IP. > citing the competitive environment All the companies are c…

> All the companies are competing to be the biggest inconvenience to everyone else while scraping as much stuff as they can. So the invisible hand is just a middle finger?

[dead]
Post reply on HN