Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let us serve you, but don't bring us down

blog.archive.org

Post reply on HN
Print view View original

241–250 of 255 posts

Re: Let us serve you, but don't bring us down

#241

Post by nektro »

thayne wrote: post #208
crazygringo wrote: post #196

Is there an open-source rate limiter that works well for sites large and small? It just strikes me as surprising that sites are still dealing with problems like this in 2023. The idea that a site is still manually having to identify a set of IP addresses and block them with human intervention seems absolutely archaic by this point. And I don't think different sites have particularly different needs here... basic patt…

It isn't that hard to set up naive rate limiting per ip address. It's a few lines in haproxy, and there is documentation on how to do it. There are a couple of problems that make it more complicated though. The first is that with NATs, you can have a lot of users behind a single IP address, which can result in legitimate requests getting blocked. The second is that, while it can help against a DoS, it doesn't help th…

ipv6 would solve these issues because they remove the need for NATs

Re: Let us serve you, but don't bring us down

#242

Post by crazydoggers »

rodgerd wrote: post #232
crazydoggers wrote: post #91

Earlier quoted context omitted.

I just wanted to add, while I don’t agree with the assessment that copyright holders need to be protected from something like archive.org, i don’t think the parent comment deserved to be flagged, so I vouched for it. I think the question was raised in sincerity, and I think it offers a point of discussion for those not familiar with the issues. I don’t think it’s helpful to flag things people disagree with, as long a…

> i don’t think the parent comment deserved to be flagged, so I vouched for it. Since archive.org have now updated that the problem scraper is now evading countermeasures and bringing them down repeatedly, and has been identified as an "AI" company, it could end up being an existential risk: if companies start using archive.org as a large-scale commercial IP theft proxy, they will likely face even more legal challeng…

Yes. The whole AI space using questionable data sources for training is just a tempest in a teapot about to explode.

Re: Let us serve you, but don't bring us down

#243

Post by johnklos »

It's a shame that AWS can be used to do this and they'll either do nothing or take their sweet time doing anything, and half the time they reply saying that the abuse is no longer happening from the reported IPs (because they've already moved to new IPs).

It's infuriating.

Re: Let us serve you, but don't bring us down

#244

Post by remram »

jlokier wrote: post #240
remram wrote: post #222

Earlier quoted context omitted.

If you are willing to spend money, there is no problem keeping content up with existing technology. Like always, blockchain solutions miss the problem entirely.

It would be nice not to shoot the messenger. The problem described by maksumur is a real problem people encounter often - torrents for less popular large files that people only occasionally want, but do want or need, have a habit of disappearing, or being non-functional when you do find a seed. Popularity plays too strong a role. As far as I know the proof-of-data-availability folks (whether using a blockchain or not…

No, that still misses the mark entirely. We are in a thread about archive.org's server being hit too hard, someone proposed they should offer distributed downloads via torrent, and someone commented asking for "a way to keep them alive regardless of interest".

You are proposing a system based on financial rewards for hosting. Who pays those rewards, for those files in which there is no interest? If archive.org is to pay for it, we are back to square one. They are already very good at hosting content, within the limits of the resources they have. If people with an interest in downloading the files pay for it, no go, files with no interest go away. If you propose that people currently abusing the free service of archive.org, to the point of bringing it down, would pay a fee per download, you must be joking.

Re: Let us serve you, but don't bring us down

#245

Post by scotty79 »

hombre_fatal wrote: post #239
scotty79 wrote: post #225

Earlier quoted context omitted.

For honest users it's not noticeable in normal conditions. But the DDOSer will DDOS themselves. They might make your service slow down for the users when they attack but not because your server gets swamped just because you choose to increase the difficulty of the challenge while you are attacked. DDOSers will notice because they won't be able to make requests as fast. At least not the ones that are costly for you. I…

I think the part you're not addressing is the fact that DDOSers aren't using their own hardware and more importantly: their request still goes through at the end so what do they care if it took 2000ms longer? I actually do like your puzzle challenge idea from an obfuscation standpoint of making an API less attractive for clients that aren't your own, though. A challenge system + an annoying format like Protobufs inst…

> their request still goes through at the end so what do they care if it took 2000ms longer?

It means that on this hardware they can make one request per two seconds not thousands per second.

And it affects all of the hardware under the control of an attacker, so his attack becomes thousands times less dangerous.

> Biggest problem with DDOS though is that if the volume is even reaching your application server, you're probably hosed.

That's why I'm suggesting that challenge generation and validation can be done on separate machines. So your application servers can be safe.

Obfuscation is a valid point too.

I thought about it for a bit, made some experiments. Now I think the challenge should be completely random but the server needs to keep track of recently issued challenges and solved challenges to prevent reusing of already calculated solutions. I think bloom filters would be perfect for that because some small percentage of false negatives doesn't matter.

Re: Let us serve you, but don't bring us down

#246

Post by Terretta »

marginalia_nu wrote: post #120
alentred wrote: post #103

Just yesterday there was a comment here on HN [1] about https://jsonip.com , which is essentially supported by a single person (all operational costs included) and gets abused in a somewhat similar manner. I am not even sure what to think: do the folks not understand what they do, or are they just bluntly ignorant of it? [1] https://news.ycombinator.com/item?id=36092417

Yeah I'm getting this shit too with Marginalia Search. I'm getting about 2-2.5M queries per day that are definitely from bots that would 100% sink my server if they somehow go through the bot mitigation. It peaks at hundreds of search queries per second. To be clear these are search queries, and search queries typically trigger disk reads of about ~10-20 Mb. I get about 20,000 queries per day that may be human.

You could update the line:

> The search engine is currently serving about 36 queries/minute.

To:

> The search engine is currently serving about 36 real queries/minute, and deflecting 1806 bot queries/minute (please don't).

Re: Let us serve you, but don't bring us down

#247

Post by otikik »

coldpie wrote: post #237
otikik wrote: post #235

Earlier quoted context omitted.

I would block it, if I was the maintainer. As the linked post mentions, automated requests can "ramp up" at any moment, risking server stability. By preemptively blocking automated parsing (on a resource which primary usage is individual requests, not mass ones) I would avoid future problems for myself. Let them contact us via support if they really need an exception. In general I would rate limit by IP anything conn…

So the answer is it's just speculation and has never been seen in the wild :)

Sure, you can interpret my answer like that, if that makes you happy.

Re: Let us serve you, but don't bring us down

#248

Post by pablosr11 »

coldpie wrote: post #238
tastysandwich wrote: post #227

Earlier quoted context omitted.

I can't say how the website I'm scraping would respond if I just went full throttle. But it's also just a matter of courtesy anyway not to make ten thousands requests per second. Funny story - at work we once had a huge spike in requests from a single IP. We all crowded around, thinking it was some malicious hacker from France. How exciting - we're now interesting enough to warrant a DoS! Turns out another team in th…

I was asking about randomizing the interval between requests.

tends to be around not repeating same patterns.

Re: Let us serve you, but don't bring us down

#249

Post by jlokier »

remram wrote: post #244
jlokier wrote: post #240

Earlier quoted context omitted.

It would be nice not to shoot the messenger. The problem described by maksumur is a real problem people encounter often - torrents for less popular large files that people only occasionally want, but do want or need, have a habit of disappearing, or being non-functional when you do find a seed. Popularity plays too strong a role. As far as I know the proof-of-data-availability folks (whether using a blockchain or not…

No, that still misses the mark entirely. We are in a thread about archive.org's server being hit too hard, someone proposed they should offer distributed downloads via torrent, and someone commented asking for "a way to keep them alive regardless of interest". You are proposing a system based on financial rewards for hosting. Who pays those rewards, for those files in which there is no interest? If archive.org is to…

> You are proposing a system based on financial rewards for hosting.

No, I'm not. You are incorrectly assuming that blockchains are necessarily financial or that cryptoeconomic incentive structure involves net pay to someone.

> If you propose that people currently abusing the free service of archive.org, to the point of bringing it down, would pay a fee per download, you must be joking.

I'm not proposing that.

People "pay" for hosting by participating in some amount of upload to offset their download, in order to be granted higher download rates. That is the same principle as BitTorrent has used since its inception: Upload is measured and download is traded for upload to ensure users choose to upload for a while.

The difference is that information is split and diffused in a different way, which ensure that some amount of upload bandwidth and temporary storage is available for less popular data as long as there are people participating in the network, mostly when they are downloading something more popular and providing some upload in exchange. The network power law helps by ensuring the long tail of less popular content needs relatively little "extra" bandwidth so it is not onerous on the users who, most of the time, are downloading and storing popular data.

No money needs to be involved.

Probably some financial things will emerge much like paid Torrent sites do at the moment for enhanced access which some people prefer. You can't prevent them, but they are not required for the network's operation nor required to access it.

Re: Let us serve you, but don't bring us down

#250

Post by xenotize »

coldpie wrote: post #237
otikik wrote: post #235

Earlier quoted context omitted.

I would block it, if I was the maintainer. As the linked post mentions, automated requests can "ramp up" at any moment, risking server stability. By preemptively blocking automated parsing (on a resource which primary usage is individual requests, not mass ones) I would avoid future problems for myself. Let them contact us via support if they really need an exception. In general I would rate limit by IP anything conn…

So the answer is it's just speculation and has never been seen in the wild :)

Kind of a snarky response. Obviously this has been seen in the wild. If you created an intrusion detection system to look for suspicious requests, I think one occurring over and over and at a regular interval would clearly be seen as malicious and not a genuine user.
Post reply on HN