Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS S3: Sometimes you should press the $100k button

cyclic.sh

Post reply on HN
Print view View original

101–110 of 240 posts

Re: AWS S3: Sometimes you should press the $100k button

#101

Post by ectopod »

Sophira wrote: post #52
jwalton wrote: post #22

Your website renders as a big empty blue page in Firefox unless I disable tracking protection (and in my case, since I have noscript, I have to enable javascript for "website-files.com", a domain that sounds totally legit).

The problem is that the DIV that contains the main text has the attribute 'style="opacity:0"'. Presumably, this is something that the JavaScript turns off. A lot of sites like to do things like this for some reason. I haven't figured out why. I like to use Stylus to mitigate these if I can, rather than enabling JavaScript.

A lot of these sites (including this one) do work in reader view.

Re: AWS S3: Sometimes you should press the $100k button

#102

Post by ec109685 »

ebingdom wrote: post #26
korostelevm wrote: post #20

Earlier quoted context omitted.

AWS does the optimizations over time based on access patterns for the data. Should have made that clearer in the article. The problem becomes unusual burst load - usually from infrequent analytics jobs. The indexing cant respond fast enough.

Thanks for the clarification. But now I'm confused about the limits: > 3,500 PUT/COPY/POST/DELETE requests per second per prefix > 5,500 GET/HEAD requests per second per prefix Most of those APIs don't even take a delimiter. So for these limits, does the prefix get inferred based on whatever delimiter you've used for previous list requests? What if you've used multiple delimiters in the past? Basically what I'm tryin…

Delimiter isn’t used for writes, only list operations.

S3 simply looks at the common string prefixes in your object names and uses that to internally shard objects, so you can achieve a multiple of those request limits.

aaa122348

aaa484585

bbb484858

bbb474827

Would have same performance as:

aaa/122348

aaa/484585

bbb/484858

bbb/474827

Re: AWS S3: Sometimes you should press the $100k button

#103

Post by snowwrestler »

charcircuit wrote: post #24

Can someone explain what happened in the end? From my understanding nothing happened (they deprioritizod the story for fixing it) and they are still blowing through the cloud budget.

They didn’t resolve the issue.

There’s an important moment in the story, where they realize the fix will incur a one-time fee of $100,000. No one in engineering can sign off on that amount, and no one wants to try to explain it to non-technical execs.

They don’t explain why. But it’s probably because they expect a negative response like “how could you let this happen?!” or “I’m not going to pay that, find another way to fix it.”

In a lot of organizations it’s easier to live with a steadily growing recurring cost than a one-time fee… even if the total of the steady growth ends up much larger than the one-time fee!

It’s not necessarily pathological. Future costs will be paid from future revenue; whereas a big fee has to be paid from cash on-hand now.

But sometimes the calculation is not even attempted because of internal culture. When the decision is “keep your head down” instead of “what’s the best financial strategy,” that could hint at even bigger potential issues down the road.

Re: AWS S3: Sometimes you should press the $100k button

#104

Post by Uehreka »

pphysch wrote: post #99
pontifier wrote: post #93

DON'T PRESS THAT BUTTON. The egress and early deletion fees on those "cheaper options" killed a company that I had to step in and save.

On a related note, suppose the Fed raises rates to mitigate inflation and indirectly kills thousands of zombie companies, including many SaaS renting the cloud. What happens to their data? Does the cloud unilaterally evict/delete it, or does it get handled like an asset -- auctioned off, etc?

> does it get handled like an asset -- auctioned off, etc?

Who would buy that? I guess if this happened enough then people would start "data salvager" companies that specialize in going through data they have no schema for looking for a way to sell something of it to someone else. I have to imagine the margins in a business like that would be abysmal, and all the while you'd be in a pretty dark place ethically going through data that users never wanted you to have in the first place.

Of course, all these questions are moot because if this happened the GDPR would nuke the cloud provider from orbit.

Re: AWS S3: Sometimes you should press the $100k button

#105

Post by twistedpair »

cj wrote: post #5

Off topic: for people with a "million billion" objects, does the S3 console just completely freeze up for you? I have some large buckets that I'm unable to even interact with via the GUI. I've always wondered if my account is in some weird state or if performance is that bad for everyone. (This is a bucket with maybe 500 million objects, under a hundred terabytes)

Honestly this is when most folks move to using their own dashboards, metrics, and tooling. The AWS GUIs were designed for small to moderate use cases.

You don't peer into a bucket with a billion objects and ask for a complete listing, or accounting of bytes. There are tools and APIs for that.

That's what I do with my thousands of buckets and billions of files (dashboards).

Re: AWS S3: Sometimes you should press the $100k button

#106

Post by koolba »

theshrike79 wrote: post #81
CydeWeys wrote: post #55

Earlier quoted context omitted.

The data protection regulations really are so freeing, huh. It's amazing to be able to delete all this stuff without worrying about having to keep it forever.

Yep, having everything disappear at 2 months max is a life-saver. That "absolutely essential thing" isn't essential any more when there is a possible GDPR/CCPA violation with a significant fine just around the corner.

Just make sure you actually test your backups. Two months of unusable backups are just as useful as no backups.

Re: AWS S3: Sometimes you should press the $100k button

#107

Post by Aeolun »

pontifier wrote: post #93

DON'T PRESS THAT BUTTON. The egress and early deletion fees on those "cheaper options" killed a company that I had to step in and save.

If they were already paying 100k per month for their storage, I doubt the additional 100k would severely impact their business.

Proven by the fact that they happily went on to pay the bill for the next 6 months.

Re: AWS S3: Sometimes you should press the $100k button

#108

Post by twistedpair »

ebingdom wrote: post #16

I'm confused about prefixes and sharding: > The files are stored on a physical drive somewhere and indexed someplace else by the entire string app/events/ - called the prefix. The / character is really just a rendered delimiter. You can actually specify whatever you want to be the delimiter for list/scan apis. > Anyway, under the hood, these prefixes are used to shard and partition data in S3 buckets across whatever…

This is where GCP's GCS (Google Cloud Storage) shines.

You don't need to mess with prefixing all your files. They auto level the cluster for you [1].

[1] https://cloud.google.com/storage/docs/request-rate#redistrib...

Re: AWS S3: Sometimes you should press the $100k button

#109

Post by alisonkisk »

hvs wrote: post #75
whimsicalism wrote: post #68

Earlier quoted context omitted.

now this is a spin i havent heard before.

You haven't heard it because it's not spin, it's from an engineer's point of view. That's not the view you hear in the news when it comes to these things.

Eh, Retention and Deletion are both pain for devs. Not having to care is the happy state.

Re: AWS S3: Sometimes you should press the $100k button

#110

Post by stingraycharles »

jeff_vader wrote: post #76
CydeWeys wrote: post #55

Earlier quoted context omitted.

The data protection regulations really are so freeing, huh. It's amazing to be able to delete all this stuff without worrying about having to keep it forever.

In case of my previous employer it led to incredibly complicated encryption system. It took couple years to maybe implement in 10% of the system. Deleting any old data was rejected.

How is encryption compliant? I’ve implemented GDPR data infrastructures twice now, and as far as I’m aware, the only way to be compliant with encryption is when you throw the decryption key away.
Post reply on HN