Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS S3: Sometimes you should press the $100k button

cyclic.sh

Post reply on HN
Print view View original

221–230 of 240 posts

Re: AWS S3: Sometimes you should press the $100k button

#221

Post by tekknik »

marcosdumay wrote: post #135
dasil003 wrote: post #86

Earlier quoted context omitted.

I'm not sure any sizable group is banging their head against a wall. Yes, AWS is complex. Yes, AWS has cost foot guns. These are natural outcomes of removing friction from scaling. Sure we could start with something simpler, but as you may have noticed, even the more basic hosting providers like DigitalOcean and Linode have been adding S3-compatible object storage because of its proven utility. In terms of making som…

> These are natural outcomes of removing friction from scaling. Yes, and making scaling frictionless brings a very tiny bit of value for everybody, but a huge amount of value for the cloud operator. Any bit of friction would completely remove that problem. Also, focusing on scaling before efficiency benefits nobody but the cloud provider.

AWS was originally built to run amazon workloads. When building software at amazon scale absolutely is one of the first things you think about.

Re: AWS S3: Sometimes you should press the $100k button

#222

Post by chrisjc »

rmbyrro wrote: post #83
asim wrote: post #60

The AWS horror stories never cease to amaze me. It's like we're banging our heads against the wall expecting a different outcome each time. What's more frustrating, the AWS zealots are quite happy to tell you how you're doing it wrong. It's the users fault for misusing the service. The reality is, AWS was built for a specific purpose and demographic of user. It's now complexity and scale makes it unusable for newer d…

What do you see missing or not well explained in AWS documentation that newer devs wouldn't understand? I started using S3 early in my career and didn't see this problem. I always thought in data retention during design phase. My opinion is that lazy, careless or under time pressure developers will not, and then will get bitten. But it would happen to any tool. Maybe a different problem, but they'll always get bitten…

Forget newer devs for a moment... I've had years of experience with S3 and sounds like the author of the article has too. Despite my years of experience in programming/DBs/etc, I'm definitely not an amazing developer.

But I learned a whole lot of new things from this article that I didn't understand from reading the AWS documentation, let alone think I had to even concern myself with some of these issues. Spotty warnings about transitional request charges?

Anyway, kudos to you for always thinking about (and i hope actualizing) retention during policies the design phase. However, while I certainly think devs bare some of this responsibility, I'm sure they're usual met with all of the usual excuses and kicking the can down the road line of reasoning from PM/PO/etc that lead to these kinds of nightmares in the beginning... Then again, it will probably be another developer or system admins' nightmare when it becomes an issue.

Even as an experience engineer, I still struggle setting the retention policy at the beginning of a new design... I'd love to hear any advice you have about how manage this incredibly important aspect?

Re: AWS S3: Sometimes you should press the $100k button

#223

Post by tekknik »

jiggawatts wrote: post #211
saimiam wrote: post #209

Earlier quoted context omitted.

> blame the user Not sure how this is blame the user. If you are setting up a bare metal server for a client and they don't ask you for (say) DDOS protection, will you still set up a DDOS protection protocol for them? I would think not since most people would try to match what a client asks for and maybe throw in some freebies. If after that, they get hit by DDOS, the onus is on them to have told you to plan ahead fo…

There are two kinds of outcomes from a DDoS: 1. an outage, which in reality is just an inconvenience, not the end of the world, unlike what most IT people seem to think. 2. a bill that can bankrupt you, which may as well be the end of the world for many people or small businesses. It can be literally "game over". A bare metal box doesn't need protection from the 2nd risk. Its costs are fixed, irrespective of the amou…

So let’s say they implement cost limits, how does this work? When you reach the limits does it delete all your resources? Idle them then force AWS to pay the bill by keeping those resources out of rotation for others? Someone has to pay for those resources while they’re used and most don’t want their database tables dropped because they got popular and hit their spend limit.

Re: AWS S3: Sometimes you should press the $100k button

#224

Post by tekknik »

prewett wrote: post #186
deanCommie wrote: post #139

Earlier quoted context omitted.

HackerNews loves to criticize the cloud. It always reminds me of this infamous Dropbox comment: https://news.ycombinator.com/item?id=9224 The cloud abstracts SO MUCH complexity from the user. The fact that people are then gleefully taking these "simple" services and overloading them with way too much data, and way too much complexity on top is not a failure of the underlying primitives, but a success. Without these c…

A surprise massive bill can be worse than no solution at all, in my opinion. And if the easy path leads to massive-bill lock-in, that's also not very helpful. It's not like people didn't know how to run servers and remote storage before AWS showed up. Before AWS showed up at least your data-center costs were pretty predictable: you managed the servers yourself, so whatever the salaries summed to, that was it. It's no…

But you did waste money through capacity planning. As you either have the exact quantity of servers needed, or you have some sitting there idle. Then there’s swapping out older/failing hardware. In fact your DC calculations are rather difficult to get accurate.

Re: AWS S3: Sometimes you should press the $100k button

#225

Post by tekknik »

gmiller123456 wrote: post #196
rizkeyz wrote: post #167

I did the back-of-the-envelope math once. You get a Petabyte of storage today for $60K/year if you buy the hardware (retail disks, server, energy). It actually fits into the corner of a room. What do you get for $60K in AWS S3? Maybe a PB for 3 months (w/o egress). If you replace all your hardware every year, the cloud is 4x more expensive. If you manage to use your getto-cloud for 5 year, you are 20x cheaper than Am…

I don't get what's going on with on-line storage. You can walk in Best Buy and get a few Tb hard drive for well under $100. Yet every cloud service wants to charge you several times that per year for just 1Tb. I understand drives fail, there's operating cost, and some need extremely low latency. But there seems to be a huge disparity between what a hard drive costs, and what it costs to make it available on the Inter…

There’s a difference between a consumer drive and a server drive. Plop that $100 drive in and you may be back in a week or so replacing it.

Re: AWS S3: Sometimes you should press the $100k button

#226

Post by gmiller123456 »

tekknik wrote: post #225
gmiller123456 wrote: post #196

Earlier quoted context omitted.

I don't get what's going on with on-line storage. You can walk in Best Buy and get a few Tb hard drive for well under $100. Yet every cloud service wants to charge you several times that per year for just 1Tb. I understand drives fail, there's operating cost, and some need extremely low latency. But there seems to be a huge disparity between what a hard drive costs, and what it costs to make it available on the Inter…

There’s a difference between a consumer drive and a server drive. Plop that $100 drive in and you may be back in a week or so replacing it.

Why would you think a drive automatically looses lifespan just because the computer it's in is referred to as a server? Many of my desktop hard drives see more activity that some of my website HDs.

Re: AWS S3: Sometimes you should press the $100k button

#227

Post by tekknik »

gmiller123456 wrote: post #226
tekknik wrote: post #225

Earlier quoted context omitted.

There’s a difference between a consumer drive and a server drive. Plop that $100 drive in and you may be back in a week or so replacing it.

Why would you think a drive automatically looses lifespan just because the computer it's in is referred to as a server? Many of my desktop hard drives see more activity that some of my website HDs.

Because a server has more traffic than a consumer machine. You’re mistaken if you think your home server has the same traffic as an S3 storage node.

Read more about the differences in drives here: https://blog.storagecraft.com/consumer-vs-enterprise-hard-dr...

Re: AWS S3: Sometimes you should press the $100k button

#228

Post by chrisjc »

ignoramous wrote: post #114
asim wrote: post #60

The AWS horror stories never cease to amaze me. It's like we're banging our heads against the wall expecting a different outcome each time. What's more frustrating, the AWS zealots are quite happy to tell you how you're doing it wrong. It's the users fault for misusing the service. The reality is, AWS was built for a specific purpose and demographic of user. It's now complexity and scale makes it unusable for newer d…

> It's the users fault for misusing the service. I believe, AWS' usage -based billing make for long-tail surprises because its users are designing systems exactly as one would expect them to. For example, S3 is never meant for a bazillion small objects which Kinesis Firehose makes it easy to deliver to it. In such cases, dismal retrieval performance aside [0], the cost to list/delete dominate abnormally. We spin up a…

Perhaps I don't fully understand the nuances of what you're trying to do, but...

> S3 is never meant for a bazillion small objects which Kinesis Firehose makes it easy to deliver to it

Are you saying Firehose increases the likelihood of creating the "small file problem"?

If so, isn't this exactly what Firehose tries to prevent? Sure, you can set all the thresholds low and unnecessarily generate lots of small files, but you can tune those thresholds to maximize record/file size and attain a reasonable latency. If there's a daily batch job to make this data useful, then who cares about latency?

Also, why would you run a daily batch job to coalesce all these files into parquet files instead of letting Firehose just do that for you. It can also do a certain amount of partitioning if it's required.

Re: AWS S3: Sometimes you should press the $100k button

#229

Post by chrisjc »

xyzzy_plugh wrote: post #34
ebingdom wrote: post #16

I'm confused about prefixes and sharding: > The files are stored on a physical drive somewhere and indexed someplace else by the entire string app/events/ - called the prefix. The / character is really just a rendered delimiter. You can actually specify whatever you want to be the delimiter for list/scan apis. > Anyway, under the hood, these prefixes are used to shard and partition data in S3 buckets across whatever…

The prefix isn't delimited, it's an arbitrary length based on access patterns. A fictitious example which is close to reality: In parallel, you write a million objects each to: tomato/red/... tomato/green/... tomatoes/colors/... The shortest prefixes that evenly divides writes are thus tomato/r tomato/g tomatoes If you had an existing access pattern of evenly writing to tomatoes/colors/... bananas/... The shortest pr…

> So suddenly writing 3 million objects that begin with a t would cause an uneven load or hotspot on the backing shards.

makes sense

> The system realizes your new access pattern and determines new prefixes and moves data around to accommodate what it thinks your needs are.

What does "determines new prefixes" mean? Obviously AWS isn't going to come up with new prefixes and change object names.

So does AWS maintain prefix-surrogates (prefix sub-string(0,?) references) and those are what actually gets shuffled around to handle the new unbalanced workload? Sort of like resharding?

Moreover, since it's really prefix-surrogates being used, the recommendation of randomizing prefixes can be replace with randomizing prefix-surrogates and delegated to AWS, removing the prior responsibility from the customer. Hence the 2018 announcement https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3...

Re: AWS S3: Sometimes you should press the $100k button

#230

Post by gmiller123456 »

tekknik wrote: post #227
gmiller123456 wrote: post #226

Earlier quoted context omitted.

Why would you think a drive automatically looses lifespan just because the computer it's in is referred to as a server? Many of my desktop hard drives see more activity that some of my website HDs.

Because a server has more traffic than a consumer machine. You’re mistaken if you think your home server has the same traffic as an S3 storage node. Read more about the differences in drives here: https://blog.storagecraft.com/consumer-vs-enterprise-hard-dr...

Again, you're making assumptions about how busy a server has to be. I have websites that don't get 10 visitors a day.
Post reply on HN