Beauty of cloud :) This could be easily served by a $100/month DO droplet with 0 worries about $.
We survived 10k requests/second: Switching to signed asset URLs in an emergency
11–20 of 176 posts
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#12Have you considered putting cloudflare or similar CDN with unlimited egress in front of your bucket? Reading your blogpost I don't fully get how the current signing implementation can halt massive downloads, or the "attacker"(?) would just adapt their methods to get the signed URLs first and then proceed to download what they are after anyway?
Yup. The only mitigation here is that there is a limit to how many different asset URLs they will be able to generate, but if they want to be malicious they can download the same file over and over again and still make you rack up a huge bill.
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#13Beauty of cloud :) This could be easily served by a $100/month DO droplet with 0 worries about $.
Does DO have free bandwidth? If not how exactly does that solve the problem?
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#14Have you considered putting cloudflare or similar CDN with unlimited egress in front of your bucket? Reading your blogpost I don't fully get how the current signing implementation can halt massive downloads, or the "attacker"(?) would just adapt their methods to get the signed URLs first and then proceed to download what they are after anyway?
Honestly, I would just move to R2 and save on egress fees even without the CDN. Runaway egress bills are no fun.
I saved myself thousands $/mo moving to R2.
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#15I'm always surprised to read how much money companies are willing to spend on things that can be done for essentially nothing. I had a look at the site - why does this need to run on a major cloud provider at all? Why use VERY expensive cloud storage at 9 cents per gigabyte? Why use very expensive image conversion at $50/month when you can run sharp on a Linux server? I shouldn't be surprised - the world is all in on…
How much does it cost to have an ISP let you do that? What are the barriers generally?
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#16Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#17Beauty of cloud :) This could be easily served by a $100/month DO droplet with 0 worries about $.
Not on DO. ~$100 a month droplet gets you about 5TB of transfer out. They pulled 15TB in 7 hours. That's ~1,440,000 (16330) on overage or about $15k extra.
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#18Did this guy just write a blog post about how he completely rewrote a functional feature to save $800?
In all seriousness, the devil is in the details around this kind of stuff, but I do worry that doing something not even clever, but just nonstandard, introduces a larger maintenance effort than necessary.
Interesting problem, and an interesting solution, but I'd probably rather just throw money at it until it gets to a scale that merits further bot prevention measures.
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#19I'm always surprised to read how much money companies are willing to spend on things that can be done for essentially nothing. I had a look at the site - why does this need to run on a major cloud provider at all? Why use VERY expensive cloud storage at 9 cents per gigabyte? Why use very expensive image conversion at $50/month when you can run sharp on a Linux server? I shouldn't be surprised - the world is all in on…
How much does it cost to have an ISP let you do that? What are the barriers generally?
Let you do what? What barriers do you see?
Re: We survived 10k requests/second: Switching to signed asset URLs in an emergency
#20Have you considered putting cloudflare or similar CDN with unlimited egress in front of your bucket? Reading your blogpost I don't fully get how the current signing implementation can halt massive downloads, or the "attacker"(?) would just adapt their methods to get the signed URLs first and then proceed to download what they are after anyway?
10k req/s would potentially crash the ruby proxy server halting the image serving.
Cloudflare is the way to go. I generally serve heavy files, e.g. videos, from a Cloudflare bucket to avoid expensive bills from primary host.