Live data from Hacker News

AWS S3: Sometimes you should press the $100k button

cyclic.sh

211–220 of 240 posts

Re: AWS S3: Sometimes you should press the $100k button

#211

Earlier quoted context omitted.

Awesome. So I should spend more money to protect myself from flaws in Amazon's billing model with a service that I don't need for static file serving. This kind of "blame the user" thinking is why I avoid the cloud for my own use, and can't recommend it for most customers unless they have a specific reason.

> blame the user Not sure how this is blame the user. If you are setting up a bare metal server for a client and they don't ask you for (say) DDOS protection, will you still set up a DDOS protection protocol for them? I would think not since most people would try to match what a client asks for and maybe throw in some freebies. If after that, they get hit by DDOS, the onus is on them to have told you to plan ahead fo…

There are two kinds of outcomes from a DDoS:

1. an outage, which in reality is just an inconvenience, not the end of the world, unlike what most IT people seem to think.

2. a bill that can bankrupt you, which may as well be the end of the world for many people or small businesses. It can be literally "game over".

A bare metal box doesn't need protection from the 2nd risk. Its costs are fixed, irrespective of the amount of traffic attempting to hit it. A 100 Mbps link can't put out more than 100 Mbps, so even if you're charged by the terabyte of egress, there's a cost ceiling integrated into the hardware itself.

The cloud generally has no such limits, or much, much higher ones than is typically desirable.

Okay, here's another random example your WAF will not protect you from: cloud-hosted DNS.

The bare metal scenario is a box sitting on the end of a 1 Gbps Ethernet link. If attacked by some crazy UDP DNS flood attack, it could probably saturate that pipe and send out... 1 Gbps. On a fixed-cost-link plan this costs $0.00 additional money. You might have an outage, or merely a brown-out, but you won't see a cent added to your next bill.

On Azure's DNS Zones service, there's no "1 Gbps" pipe to rate limit them. They have infrastructure deployed globally, typically with 100 Gbps links. In practice, the DNS server probably only gets about 10 Gbps per region, but there's many regions. At 100 bytes per packet, you could be looking at a billion requests per second billed to your account, at an eyewatering $200/s or $720K/hour. Ouch!

Now, Azure will probably forgive that bill because it's clearly an attack.

But what if it isn't clearly an attack? Application Insights by design puts the Instrumentation Key into client-side JavaScript. It charges $3/GB on ingress! It's trivial to charge someone thousands or tens of thousands of dollars before they notice, and then they'd have a hard time convincing support that the traffic wasn't legitimate.

I can send a terabyte out for cents, each of which would cost some poor fool $3,000.

Good luck plugging every such hole, monitoring every alert (there's literally tens of thousands of metrics to alert on), and keeping up with every spike in billing that's a day late reporting on costs that can ramp up to thousands of dollars per minute.

Re: AWS S3: Sometimes you should press the $100k button

#212

Earlier quoted context omitted.

> blame the user Not sure how this is blame the user. If you are setting up a bare metal server for a client and they don't ask you for (say) DDOS protection, will you still set up a DDOS protection protocol for them? I would think not since most people would try to match what a client asks for and maybe throw in some freebies. If after that, they get hit by DDOS, the onus is on them to have told you to plan ahead fo…

There are two kinds of outcomes from a DDoS: 1. an outage, which in reality is just an inconvenience, not the end of the world, unlike what most IT people seem to think. 2. a bill that can bankrupt you, which may as well be the end of the world for many people or small businesses. It can be literally "game over". A bare metal box doesn't need protection from the 2nd risk. Its costs are fixed, irrespective of the amou…

An outage is a pretty big reputational risk imho - if I'm a startup offering a SaaS and being on Hackernews' front page hugs my site to death, I'm probably losing some potential conversions and have to answer questions about the solidity and longevity of my infra. It's an inconvenience in the moment but erodes trust over time. Unless you're Twitter and your fail whales become memes, being hugged to death is a bad thing.

I agree that the downside of scaling is the risk of running up huge bills. But the safety net to prevent the run-up is literally a monthly flat fee - $5+ $1/WAF rule. Also, you don't have to monitor every alert - just the common ones. If I had to build a comparable alerting system on bare metal, I'd go crazy.

To me, the flexibility of the cloud is worth the trade off.

> WAF will not protect you from...UDP..

Don't think WAF is the tool to protect against UDP layer attacks. Shield (which is available standard) already handles this.

To be clear, I have not had to deal with DDOS attacks. We once had to deal with was someone repeatedly downloading a 1mb gif from our website which led to big egress fees. WAF's rate based rules but an end to that nonsense.

Re: AWS S3: Sometimes you should press the $100k button

#213

Earlier quoted context omitted.

How is encryption compliant? I’ve implemented GDPR data infrastructures twice now, and as far as I’m aware, the only way to be compliant with encryption is when you throw the decryption key away.

As mentioned, encrypt something and throw a way the key, often called "crypto shredding".

That doesn't sound like something jeff_vader was talking about, since "deleting any old data was rejected" and this is definitely a way of deleting stuff.

Re: AWS S3: Sometimes you should press the $100k button

#214
post #45

sigh . My team is facing all these issues. Drowning in data. Crazy S3 bill spikes. And not just S3 - Azure, GCP, Alibaba, etc since we are a multi-cloud product. Earlier, we couldn't even figure out lifecycle policies to expire objects since naturally every PM had a different opinion on the data lifecycle. So it was old-fashioned cleanup jobs that were scheduled and triggered when a byzantine set of conditions were m…

Are you multi-cloud because your customers need you to be multi-cloud?

Yes, geographically diverse customers who prefer different cloud platforms.

Re: AWS S3: Sometimes you should press the $100k button

#215
post #198

Earlier quoted context omitted.

> I then went for a compromise solution asking if we could stitch the small objects together after a period of time so they would be eligible for things like infrequent access or glacier but, alas, "dev time is expensive you know" so N figure s3 bills continue as far as I know. This hits home so hard that it hurts. In my case is not S3 but compute bills but the core concept is the same.

It is also because devops is shoved down devs throats while claiming that it is easier and better. So now many of developers don’t want to spend time rewriting their code for something that is supposed to reduce their workload not increase it.

Yeah, I totally agree, and I do this from an Ops perspective. 6-7 years ago I was really fearing my job would disappear because "everything will be automated in the cloud and owned by developers who write business code". Turns out it just transformed a bit but there is still plenty of not-strictly-related-to-business code to be written and maintained that developers mainly don't care about.

Re: AWS S3: Sometimes you should press the $100k button

#216
post #84

Earlier quoted context omitted.

I wonder sometimes if it would help if we collectively watched more anti-hoarding shows, in order to see how the consultants convince their customers they can get rid of stuff.

humans started their first 300k years as nomads – storing was just impossible and decrufing happened by itself when moving along. So maybe that's why we're not good at it yet.

Also hoarding digital data is far easier than real. I wish I could have grep on real space.

Re: AWS S3: Sometimes you should press the $100k button

#217
post #166

Just avoid the cloud. You get a Ceph storage with the performance of Amazon S3 at the price point of Amazon S3 Glacier in any Datacenter worldwide deployed if you want. There are companies that help you doing this. Feel free to ask if you need help.

You have to properly administrate those servers else you'll lose all your files and everything will be inaccessible.

Administrating CEPH is unfortunately hard.

Re: AWS S3: Sometimes you should press the $100k button

#218

Earlier quoted context omitted.

Ahh I see, and that way you can quickly “remove” a whole lot of data by just removing the key, which makes for cheap operations, and/or more flexible workflow (you can periodically compact the database and remove entries for which you have no key). Is my understanding correct?

yes, but also its that a lot of the data these days ends up in pseudo-append-only stores (like s3/glacier, or many data warehouse products) where deletes/updates to old data are extremely expensive. Or just having to scan petabytes of cold stored data looking for a particular users records. Throwing away the key is instant and "free".

Interesting... this raises soooo many questions.

How are "crypto-shredding" actions propagated to the access patterns/layer?

I assume that there is an encrypted partition/cluster/shard key (in addition to similarly encrypted rows/fields) that is invalidated during the shredding causing any predicate matching on these ids to evaluate to false.

---

Now that I've typed this out, i realize that by electing to encrypt individual fields, all and any predicate matching will evaluate to false and has nothing to do with partitioning, sharding, or clustering...

I guess it would also be pretty awesome since you could invalidated entire sets of data by "shredding" grouping ids that are being used as partition/cluster/shard keys.

Now I realize that this implies that you shouldn't encrypt each and every fields of related data the same way (grouping ids), otherwise you're potentially going to end up with unique keys/ids for common attributes across sets of data... potentially rendering clustering/sharding/partition useless (cardinality too great).

While "defragging" or "rebalancing" this increasingly "sparse", old data would be expensive, surely there has to come a point where the storage costs start to exceed that of interaction costs for specific subsets of your prefixes. For instance, partitions that consist entirely of data that has had all of its respective encryption keys shredded.

---

Illuminating comment that has set my mind into overdrive... Fascinating stuff!

Re: AWS S3: Sometimes you should press the $100k button

#219

Earlier quoted context omitted.

now this is a spin i havent heard before.

As a sysadmin I really wish you had. SO MANY problems have come to my desk because some dude 3 years ago did not consider retention or rotation and now I have to figure out what to do with a 4TB .txt that is apparently important.

Have you come up with a process, or an idea for a process to ensure this doesn't happen?

For instance when they create a provisioning request, are you able to set an extremely low threshold? When they say that won't do, the cost increases and their able to see/understand and start to care about the actual lifecycles of what they're creating?

Surely there is a way to project and monitor the cost of their resources over time, and deliver them an invoice on a regular basis? In other words something like a cost attribution model? That way when the bills start to increase dramatically overtime, pinpointing the heavy hitters becomes trivial, and when they come knocking on your door to "do something about it" you can just say "go talk to Bob".

I don't mean to sound like I'm trivializing the problem (honestly I can relate as I've gone through it myself), but I'd love to hear how anyone else has dealt with this issue effectively.

Re: AWS S3: Sometimes you should press the $100k button

#220
post #45

sigh . My team is facing all these issues. Drowning in data. Crazy S3 bill spikes. And not just S3 - Azure, GCP, Alibaba, etc since we are a multi-cloud product. Earlier, we couldn't even figure out lifecycle policies to expire objects since naturally every PM had a different opinion on the data lifecycle. So it was old-fashioned cleanup jobs that were scheduled and triggered when a byzantine set of conditions were m…

I host stuff on AWS, but I am pretty sure that hosting on my own server or a server a IT service provider maintains is much cheaper.

Did you include maintenance, patching and machine upgrades? Cause likely it’s not.
Post reply on HN