Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS S3: Sometimes you should press the $100k button

cyclic.sh

Post reply on HN
Print view View original

11–20 of 240 posts

Re: AWS S3: Sometimes you should press the $100k button

#11

Post by sokoloff »

amelius wrote: post #7
Tehchops wrote: post #3

We’ve got data in S3 buckets not nearly at that scale and managing them, god forbid trying a mass delete, is absolute tedium.

Mass delete also takes an eternity on my Linux desktop machine. The filesystem is hierarchical, but the delete operation still needs to visit all the leaves.

Is S3 actually hierarchical? I always took the mental model that the S3 object namespace within a bucket was flat and the treatment of ‘/‘ as different was only a convenient fiction presented in the tooling, which is consistent with the claim in this article.

Re: AWS S3: Sometimes you should press the $100k button

#13

Post by liveoneggs »

wodenokoto wrote: post #2

I've never been in this situation, but I do wish you could query files with more advanced filters on these blob storage services. - But why SageMaker? - Why do some orgs choose to put almost everything in 1 buckets?

1 athena?

2 some jobs make a lot of data

Re: AWS S3: Sometimes you should press the $100k button

#14

Post by cle »

sokoloff wrote: post #11
amelius wrote: post #7

Earlier quoted context omitted.

Mass delete also takes an eternity on my Linux desktop machine. The filesystem is hierarchical, but the delete operation still needs to visit all the leaves.

Is S3 actually hierarchical? I always took the mental model that the S3 object namespace within a bucket was flat and the treatment of ‘/‘ as different was only a convenient fiction presented in the tooling, which is consistent with the claim in this article.

This is mostly correct, with the additional feature that S3 can efficiently list objects by "key prefix" which helps preserve the illusion.

Re: AWS S3: Sometimes you should press the $100k button

#15

Post by base698 »

cj wrote: post #5

Off topic: for people with a "million billion" objects, does the S3 console just completely freeze up for you? I have some large buckets that I'm unable to even interact with via the GUI. I've always wondered if my account is in some weird state or if performance is that bad for everyone. (This is a bucket with maybe 500 million objects, under a hundred terabytes)

Yes, and sometimes even listing can take days.

I worked somewhere that a person decided using Twitter Firehose was a good idea for S3. Keyed by tweet per file.

Ended up figuring out a way to get them in batches and condense. Ended up costing about $800 per hour to fix coupled with lifecycle changes they mentioned.

Re: AWS S3: Sometimes you should press the $100k button

#16

Post by ebingdom »

I'm confused about prefixes and sharding:

> The files are stored on a physical drive somewhere and indexed someplace else by the entire string app/events/ - called the prefix. The / character is really just a rendered delimiter. You can actually specify whatever you want to be the delimiter for list/scan apis.

> Anyway, under the hood, these prefixes are used to shard and partition data in S3 buckets across whatever wires and metal boxes in physical data centers. This is important because prefix design impacts performance in large scale high volume read and write applications.

If the delimiter is not set at bucket creation time, but rather can be specified whenever you do a list query, how can the prefix be used to influence where objects are physically stored? Doesn't the prefix depend on what delimiter you use? How can the sharding logic know what the prefix is if it doesn't know the delimiter in advance?

For example, if I have a path like `app/events/login-123123.json`, how does S3 know the prefix is `app/events/` without knowing that I'm going to use `/` as the delimiter?

Re: AWS S3: Sometimes you should press the $100k button

#17

Post by liveoneggs »

I have caused billing spikes like this before those little warnings were invented and it was always a dark day. They are really a life saver.

Lifecycle rules are also welcome. Writing them yourself was always a pain and tended to be expensive with list operations eating up that api calls bill.

----

Once I supported an app that dumped small objects into s3 and begged the dev team to store the small objects in oracle as BLOBS to be concatenated into normal-sized s3 bjects after a reasonable timeout where no new small objects would reasonably be created. They refused (of course) and the bills for managing a bucket with millions and millions of tiny objects were just what you expect.

I then went for a compromise solution asking if we could stitch the small objects together after a period of time so they would be eligible for things like infrequent access or glacier but, alas, "dev time is expensive you know" so N figure s3 bills continue as far as I know.

Re: AWS S3: Sometimes you should press the $100k button

#19

Post by throwaway984393 »

solatic wrote: post #12

TL-DR: Object stores are not databases. Don't treat them like one.

Try telling that to developers; they love using S3 as both a database and a filesystem. It's gotten to the point where we need a training for new devs to tell them what not to do in the cloud.

Re: AWS S3: Sometimes you should press the $100k button

#20

Post by korostelevm »

ebingdom wrote: post #16

I'm confused about prefixes and sharding: > The files are stored on a physical drive somewhere and indexed someplace else by the entire string app/events/ - called the prefix. The / character is really just a rendered delimiter. You can actually specify whatever you want to be the delimiter for list/scan apis. > Anyway, under the hood, these prefixes are used to shard and partition data in S3 buckets across whatever…

AWS does the optimizations over time based on access patterns for the data. Should have made that clearer in the article.

The problem becomes unusual burst load - usually from infrequent analytics jobs. The indexing cant respond fast enough.

Post reply on HN