Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS S3: Sometimes you should press the $100k button

cyclic.sh

Post reply on HN
Print view View original

231–240 of 240 posts

Re: AWS S3: Sometimes you should press the $100k button

#231

Post by jabroni_salad »

chrisjc wrote: post #219
jabroni_salad wrote: post #92

Earlier quoted context omitted.

As a sysadmin I really wish you had. SO MANY problems have come to my desk because some dude 3 years ago did not consider retention or rotation and now I have to figure out what to do with a 4TB .txt that is apparently important.

Have you come up with a process, or an idea for a process to ensure this doesn't happen? For instance when they create a provisioning request, are you able to set an extremely low threshold? When they say that won't do, the cost increases and their able to see/understand and start to care about the actual lifecycles of what they're creating? Surely there is a way to project and monitor the cost of their resources ove…

It comes down to monitoring, alerting, and followup. In other words, "good ops", which is lacking almost everywhere. Unfortunately that is always a moving target, with added complexity being that we're an external service provider and have limited authority in the client environment. Also, the sorts of companies that outsource their ops will also be willing to change providers multiple times, so it's often like trying to live in a library that has seen many generations of librarians each with their own ideas for how things ought to be organized.

Re: AWS S3: Sometimes you should press the $100k button

#232

Post by wizwit999 »

scapecast wrote: post #156
twistedpair wrote: post #105

Earlier quoted context omitted.

Honestly this is when most folks move to using their own dashboards, metrics, and tooling. The AWS GUIs were designed for small to moderate use cases. You don't peer into a bucket with a billion objects and ask for a complete listing, or accounting of bytes. There are tools and APIs for that. That's what I do with my thousands of buckets and billions of files (dashboards).

It's also the reason why some AWS product teams have started acquiring IDE- or CLI-type of start-ups. They don't want to be boxed in by the constraints of the AWS Console - which is run by a central team. For example, the Redshift team bought DataRow. Disclosure, co-founder here, we're building one of those CLIs. We started as an internal project at D2iQ (my co-founder Lukas commented further up), with tooling to col…

Product teams don't do acquisitions. And thats not why that acquisition happened.

Re: AWS S3: Sometimes you should press the $100k button

#233

Post by mro_name »

travisgriggs wrote: post #137
hinkley wrote: post #118

Earlier quoted context omitted.

Being a renter definitely kept me lighter for a long time. When you have to box things up over and over you find that the physical and mental energy around keeping it aren’t adding up. I wonder if migrating from cloud to cloud would simulate this experience.

Two space garbage collector in action right there. Maybe all things software need a "move it or lose it" impetus. Features in apps, old data, you name it. If you've gotta keep transferring/translating it, it would definitely pare things down.

maybe reuse is inferior to re-implement. Moderately re-inventing wheels may be beneficial. What may be a threshold?

Re: AWS S3: Sometimes you should press the $100k button

#234

Post by mritzmann »

gmiller123456 wrote: post #230
tekknik wrote: post #227

Earlier quoted context omitted.

Because a server has more traffic than a consumer machine. You’re mistaken if you think your home server has the same traffic as an S3 storage node. Read more about the differences in drives here: https://blog.storagecraft.com/consumer-vs-enterprise-hard-dr...

Again, you're making assumptions about how busy a server has to be. I have websites that don't get 10 visitors a day.

And your website, with 10 visits per day, needs several TB of data?

Re: AWS S3: Sometimes you should press the $100k button

#235

Post by gmiller123456 »

mritzmann wrote: post #234
gmiller123456 wrote: post #230

Earlier quoted context omitted.

Again, you're making assumptions about how busy a server has to be. I have websites that don't get 10 visitors a day.

And your website, with 10 visits per day, needs several TB of data?

Need? No. Could use reasonably priced storage? Yes. I think you've lost track of the point I was trying to make.

Re: AWS S3: Sometimes you should press the $100k button

#237

Post by Too »

amelius wrote: post #7
Tehchops wrote: post #3

We’ve got data in S3 buckets not nearly at that scale and managing them, god forbid trying a mass delete, is absolute tedium.

Mass delete also takes an eternity on my Linux desktop machine. The filesystem is hierarchical, but the delete operation still needs to visit all the leaves.

There are some tricks on Linux. For example using mv into a trash dir instead of rm. I’ve also seen some successful use of rsync that does real deletion many times faster than rm -rf, not sure why but guessing some parallelism is involved.

Google for this problem. There are surprisingly many creative ideas, many which also surprisingly are a lot better than the built in rm command.

Re: AWS S3: Sometimes you should press the $100k button

#238

Post by tekknik »

gmiller123456 wrote: post #235
mritzmann wrote: post #234

Earlier quoted context omitted.

And your website, with 10 visits per day, needs several TB of data?

Need? No. Could use reasonably priced storage? Yes. I think you've lost track of the point I was trying to make.

What exactly is your point? AWS should use consumer drives?

Re: AWS S3: Sometimes you should press the $100k button

#239

Post by rmbyrro »

chrisjc wrote: post #222
rmbyrro wrote: post #83

Earlier quoted context omitted.

What do you see missing or not well explained in AWS documentation that newer devs wouldn't understand? I started using S3 early in my career and didn't see this problem. I always thought in data retention during design phase. My opinion is that lazy, careless or under time pressure developers will not, and then will get bitten. But it would happen to any tool. Maybe a different problem, but they'll always get bitten…

Forget newer devs for a moment... I've had years of experience with S3 and sounds like the author of the article has too. Despite my years of experience in programming/DBs/etc, I'm definitely not an amazing developer. But I learned a whole lot of new things from this article that I didn't understand from reading the AWS documentation, let alone think I had to even concern myself with some of these issues. Spotty warn…

In my previous experiences, it really boils down to the unit economics.

If a given process generates $1 in revenue over a year, and it takes pennies for AWS services, that's a good sign your design is not going to break the company's pockets down the road.

In some cases, it's not easy to narrow the unit economics so much, which adds uncertainty to your premises, and there might be market fluctuations that change the unit economics in the future. I try to anticipate which areas are most likely to change and think of a trade-off in terms of short term speed and flexibility to change later, if needed. Almost always they're a trade off.

Re: AWS S3: Sometimes you should press the $100k button

#240

Post by ignoramous »

chrisjc wrote: post #228
ignoramous wrote: post #114

Earlier quoted context omitted.

> It's the users fault for misusing the service. I believe, AWS' usage -based billing make for long-tail surprises because its users are designing systems exactly as one would expect them to. For example, S3 is never meant for a bazillion small objects which Kinesis Firehose makes it easy to deliver to it. In such cases, dismal retrieval performance aside [0], the cost to list/delete dominate abnormally. We spin up a…

Perhaps I don't fully understand the nuances of what you're trying to do, but... > S3 is never meant for a bazillion small objects which Kinesis Firehose makes it easy to deliver to it Are you saying Firehose increases the likelihood of creating the "small file problem"? If so, isn't this exactly what Firehose tries to prevent? Sure, you can set all the thresholds low and unnecessarily generate lots of small files, b…

> Are you saying Firehose increases the likelihood of creating the "small file problem"?

Firehose makes it easy to do so (when the thresholds are too low, as you point out). That is, it'd happily chug along and do what you ask of it to. Sometimes, these problems only manifest in the long run (kind of like a frog in boiling water).

> Also, why would you run a daily batch job to coalesce all these files into parquet files instead of letting Firehose just do that for you.

Firehose recommends that the output be at least 64M to 128M for parquet files... we don't have anywhere near that much amount of data to yeet out of Firehose, especially because data is partitioned per-user (and a single user doesn't generate anywhere near that much data, and so we're left with the current setup). And so: It was either to let Firehose batch the data up in larger parquets (and run the partitioning job offline), or employ its partitioning magic online (and run the merge job offline, on-demand). We chose the latter for cost efficiency given our workloads.

Post reply on HN