Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We survived 10k requests/second: Switching to signed asset URLs in an emergency

hardcover.app

Post reply on HN
Print view View original

141–150 of 176 posts

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#141

Post by cuu508 »

BenjiWiebe wrote: post #115
blibble wrote: post #39

Earlier quoted context omitted.

yeah, as a crotchety old unix guy, 10k requests a second was a benchmark 30 years ago on an actual server today a raspberry pi 5 can do 50k/s with TLS no sweat

Can you give me an example of how to do 50k/s with TLS on an rpi? Also what do you use to measure that? I've tried a little with httpd (apache) on an older desktop I use as my home server and got terrible results. I can't remember but it might have been single digit or low double digit rps.

Based on these benchmark numbers, 50k/s seems plausible (not sure about the no sweat bit though ;-)): https://www.wolfssl.com/wolfssl-on-pi5-benchmarks/

Also found this bit:

> But second, the new Broadcom SOC finally supports the ARM cryptography extensions, which make it 45x faster at AES, for instance. With TLS almost everywhere, this keeps crypto performance from becoming the bottleneck. Nice.

(https://hackaday.com/2023/09/28/a-raspberry-pi-5-is-better-t...)

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#142

Post by nirui »

In addition to "signing" the URL, you may also require users to login to view the original image, and serve visitors a compressed version. This could give you the benefit of gaining users (good for VC) while respecting the guests, as well as protecting your investments.

Back in the old days where everyone operates their own server, another thing you could do is to just setup a per-IP traffic throttling with iptables (`-m recent` or `-m hashlimit`). Just something to consider in case one day you might grow tired of Google Cloud Storage too ;)

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#143

Post by nirui »

lionkor wrote: post #130
intelVISA wrote: post #119

10k/s... is that a lot? Computers are insanely fast nowadays..!

No. It's not a lot. 20-30k req/s is easy for serving simple, small files. If you have beefy machine (say, a $50 hetzner rootserver), you get a few TB of storage and unlimited or cheap bandwidth. 8-16 cores can easily(!!!) push this kind of data without even heating up, not sure wtf OP is doing. Well, I know what OP is doing - they fell for the idea that the cloud is more scalable. The issue with this is that the clou…

I'm not a fan of cloud too, but I have to admit that the networks of these big cloud providers built is just better than self-hosted ones. When they say they'll distribute your file globally, they mean it, as long as you pay of course.

But I would rather say, cloud is not for everyone. Especially in the case mentioned in the article. Think this: do you really REALLY need to distribute enlarged images globally at top speed? I bet most people just don't.

Same thing goes for "scaling", it's true the cloud can do that very well, but do you really need it that bad?

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#144

Post by BonoboIO »

rcarmo wrote: post #138

I had to do a similar thing a decade ago when someone started scraping my site by brute force. At the time I was using CoralCDN already, but my server was getting hammered, so I just started serving up assets with hashed URLs and changing the key every 24h--their scraper was dumb enough to not start again from scratch. I ended up using the exact same code for sharding, and later to move to a static site with Azure St…

It would be funny to give the scraper some „funny“ pictures :D

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#145

Post by jeffhuys »

hansvm wrote: post #125

What I just read is that for the cost of a single 16TB hard drive, they were able to rent a hard drive for 7 hours to stream 16TB, and they still had to devote meaningful engineering resources to avoid the cost overrun. Does anybody here have a success story where AWS was either much cheaper to operate or to develop for (ideally both) than the normal alternatives?

They don't use AWS, by the way. This was GCP.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#146

Post by jeffhuys »

ksnsnsj wrote: post #58
ponytech wrote: post #44

Earlier quoted context omitted.

I rent a bare metal server for $50/month with unlimited bandwith...

There is no such thing as unlimited bandwidth. What I'm aware of are services which do not charge extra for egress but severely limit your egress bandwidth (like 10 Gbit peak, 100 Mbit avg) And limiting egress bandwidth is better is better done in the service per client than by the hoster for your system

https://getdeploying.com/reference/data-egress

Check this out. You _almost_ use the most expensive service.

I think you should expand your awareness. Hetzner for instance doesn't mention anywhere that they throttle your 10gbit uplink, but they limit to 20TB/month, with ~1EUR for every TB over. Seems like you wouldn't even have noticed what you described in your article.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#147

Post by jeroenhd »

twothamendment wrote: post #116

We recently had a bot from Taiwan downloading all of our images, over and over and over - similar to the author. By the time we noticed they had downloaded them many times over and showed no signs of stopping! Bots these days are our of control and have lost their mind!

I recently found out that Bytedance was scraping a website of mine over and over again. I don't care about their stupid AI crawler scanning my cheapo server, but they were hitting the same files from different IP addresses, all from the same /56 China Telecom subnet.

I added a firewall rule to block the subnet and that seems to have worked. Earlier attempts involving robots.txt failed and my logs still got spammed by all the HTTPS requests when I blocked the bots in Nginx.

I don't understand how you could write a scraper like that and not notice that you're downloading the same files over and over again.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#148

Post by austin-cheney »

10k requests per second has historically been a lower challenge to overcome than 10k concurrent sessions on a single box. 10k concurrent sessions was the historic design goal for standing up Node.js 15 years ago.

For everything high traffic and/or concurrency related my go to solution is dedicated sockets. Sockets are inherently session-oriented which makes everything related to security and routing more simple. If there is something about a request you don’t like then just destroy the socket. If you believe there is a DOS flood attack then keep the socket open and discard its messaging. If there are too many simultaneous sockets then jitter traffic processing via load balancer as resources become available.

Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency

#149

Post by jeroenhd »

lionkor wrote: post #130
intelVISA wrote: post #119

10k/s... is that a lot? Computers are insanely fast nowadays..!

No. It's not a lot. 20-30k req/s is easy for serving simple, small files. If you have beefy machine (say, a $50 hetzner rootserver), you get a few TB of storage and unlimited or cheap bandwidth. 8-16 cores can easily(!!!) push this kind of data without even heating up, not sure wtf OP is doing. Well, I know what OP is doing - they fell for the idea that the cloud is more scalable. The issue with this is that the clou…

Based on the names of the endpoints, I get the idea that they're altering the image files on the fly (and probably caching processed files) based on the URL. I've seen this quite often on blogs and such. Serving files shouldn't take much CPU power, but resizing images can get quite expensive, especially if you want to achieve lower egress fees by using better compression methods.

Still, you need to deal with bad scrapers. Plus, this scraper downloaded at a consistent 650mbps, taking up half the unlimited Hetzner pipe by itself; if you'd go for a 10gbps Hetzner machine, you suddenly start paying egress fees once you hit 20TB of traffic. Even then, if you go the cheapo Hetzner server route, you probably still want at least some kind of CDN to keep latency down. Add to that costs of backups and synchronising failovers, and you may end up with more traffic than you'd expect.

I think going bare metal would save more than the signed URLs would, at least until the ten thousanth customer, but not everyone is proficient in maintaining servers. A lot of cloud projects I see are coming from programmers who don't want to/don't know how to maintain a Linux server and just want to run their code. If you're in that category, taking time off to learn server maintenance or hiring a sysadmin can easily be a lot more expensive than paying the extortionate rates cloud providers demand.

Post reply on HN