Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS S3: Sometimes you should press the $100k button

cyclic.sh

Post reply on HN
Print view View original

31–40 of 240 posts

Re: AWS S3: Sometimes you should press the $100k button

#31

Post by ebingdom »

inopinatus wrote: post #25
ebingdom wrote: post #16

I'm confused about prefixes and sharding: > The files are stored on a physical drive somewhere and indexed someplace else by the entire string app/events/ - called the prefix. The / character is really just a rendered delimiter. You can actually specify whatever you want to be the delimiter for list/scan apis. > Anyway, under the hood, these prefixes are used to shard and partition data in S3 buckets across whatever…

There’s no delimiter. There is only the appearance of a delimiter, to appease folks who think S3 is a filesystem, and fool them into thinking they’re looking at folders. The object name is the entire label, and every character is equally significant for storage. When listing objects, a prefix filters the list. That’s all. However, S3 also uses substrings to partition the bucket for scale. Since they’re anchored at th…

So if I have a bunch of objects whose names are hashes like 2df6ad6ca44d06566cffde51155e82ad0947c736 that I expect to access randomly, is there any performance benefit to introducing artificial delimiters like 2d/f6/ad6ca44d06566cffde51155e82ad0947c736? I've seen this used in some places.

Re: AWS S3: Sometimes you should press the $100k button

#32

Post by elcomet »

ebingdom wrote: post #31
inopinatus wrote: post #25

Earlier quoted context omitted.

There’s no delimiter. There is only the appearance of a delimiter, to appease folks who think S3 is a filesystem, and fool them into thinking they’re looking at folders. The object name is the entire label, and every character is equally significant for storage. When listing objects, a prefix filters the list. That’s all. However, S3 also uses substrings to partition the bucket for scale. Since they’re anchored at th…

So if I have a bunch of objects whose names are hashes like 2df6ad6ca44d06566cffde51155e82ad0947c736 that I expect to access randomly, is there any performance benefit to introducing artificial delimiters like 2d/f6/ad6ca44d06566cffde51155e82ad0947c736? I've seen this used in some places.

No difference other than readability. And amazon may distribute your application with another prefix anyway, like "2d/f6/ad6c"

Re: AWS S3: Sometimes you should press the $100k button

#33

Post by musingsole »

charcircuit wrote: post #23
tyingq wrote: post #9

Earlier quoted context omitted.

>Why do some orgs choose to put almost everything in 1 buckets? The article seems to be making the case it's because the delimiter makes it seem like there's a real hierarchy. So the ramifications of /bucket/1 /bucket/2 versus /bucket1/ /bucket2/ aren't well known until it's too late.

>So the ramifications of /bucket/1 /bucket/2 versus /bucket1/ /bucket2/ aren't well known until it's too late. What's the difference?

In the choice between a single bucket with hierarchical paths versus multiple buckets, there's a long list of nuances between either strategy.

For the purposes of this article, you can probably have more intuitive, sensible lifecycle policies across multiple buckets than you can trying to set policies on specific paths within a single bucket. Something like "ShortLifeBucket" and "LongLifeBucket" would allow you to have items with similar prefixes (something like a "{bucket}/anApplication/file1.csv" in each bucket) that then have different lifecycle policies

Re: AWS S3: Sometimes you should press the $100k button

#34

Post by xyzzy_plugh »

ebingdom wrote: post #16

I'm confused about prefixes and sharding: > The files are stored on a physical drive somewhere and indexed someplace else by the entire string app/events/ - called the prefix. The / character is really just a rendered delimiter. You can actually specify whatever you want to be the delimiter for list/scan apis. > Anyway, under the hood, these prefixes are used to shard and partition data in S3 buckets across whatever…

The prefix isn't delimited, it's an arbitrary length based on access patterns.

A fictitious example which is close to reality:

In parallel, you write a million objects each to:

   tomato/red/...
   tomato/green/...
   tomatoes/colors/...
The shortest prefixes that evenly divides writes are thus

   tomato/r
   tomato/g
   tomatoes
If you had an existing access pattern of evenly writing to

   tomatoes/colors/...
   bananas/...
The shortest prefixes would be

   t
   b
So suddenly writing 3 million objects that begin with a t would cause an uneven load or hotspot on the backing shards. The system realizes your new access pattern and determines new prefixes and moves data around to accommodate what it thinks your needs are.

--

The delimiter is just a wildcard option. The system is just a key value store, essentially. Specifying a delimiter tells the system to transform delimiters at the end of a list query like

   my/path/
into a pattern match like

   my/path/[^/]+/?

Re: AWS S3: Sometimes you should press the $100k button

#35

Post by hakube »

cj wrote: post #5

Off topic: for people with a "million billion" objects, does the S3 console just completely freeze up for you? I have some large buckets that I'm unable to even interact with via the GUI. I've always wondered if my account is in some weird state or if performance is that bad for everyone. (This is a bucket with maybe 500 million objects, under a hundred terabytes)

I have millions (about 16m PDF and text files) of objects and it's completely freezing

Re: AWS S3: Sometimes you should press the $100k button

#36

Post by charcircuit »

inopinatus wrote: post #25
ebingdom wrote: post #16

I'm confused about prefixes and sharding: > The files are stored on a physical drive somewhere and indexed someplace else by the entire string app/events/ - called the prefix. The / character is really just a rendered delimiter. You can actually specify whatever you want to be the delimiter for list/scan apis. > Anyway, under the hood, these prefixes are used to shard and partition data in S3 buckets across whatever…

There’s no delimiter. There is only the appearance of a delimiter, to appease folks who think S3 is a filesystem, and fool them into thinking they’re looking at folders. The object name is the entire label, and every character is equally significant for storage. When listing objects, a prefix filters the list. That’s all. However, S3 also uses substrings to partition the bucket for scale. Since they’re anchored at th…

>There’s no delimiter.

What's the delimiter parameter for then?

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObje...

Re: AWS S3: Sometimes you should press the $100k button

#37

Post by mst »

jwalton wrote: post #22

Your website renders as a big empty blue page in Firefox unless I disable tracking protection (and in my case, since I have noscript, I have to enable javascript for "website-files.com", a domain that sounds totally legit).

I have tracking protection and ublock origin both enabled and it rendered fine (FF on Win10).

(presented as a data point for any poor soul trying to replicate your problem)

Re: AWS S3: Sometimes you should press the $100k button

#38

Post by inopinatus »

charcircuit wrote: post #36
inopinatus wrote: post #25

Earlier quoted context omitted.

There’s no delimiter. There is only the appearance of a delimiter, to appease folks who think S3 is a filesystem, and fool them into thinking they’re looking at folders. The object name is the entire label, and every character is equally significant for storage. When listing objects, a prefix filters the list. That’s all. However, S3 also uses substrings to partition the bucket for scale. Since they’re anchored at th…

>There’s no delimiter. What's the delimiter parameter for then? https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObje...

To help you fool yourself. It affects how object list results are presented in the api response.

Re: AWS S3: Sometimes you should press the $100k button

#39

Post by sokoloff »

cle wrote: post #14
sokoloff wrote: post #11

Earlier quoted context omitted.

Is S3 actually hierarchical? I always took the mental model that the S3 object namespace within a bucket was flat and the treatment of ‘/‘ as different was only a convenient fiction presented in the tooling, which is consistent with the claim in this article.

This is mostly correct, with the additional feature that S3 can efficiently list objects by "key prefix" which helps preserve the illusion.

Followup question: Is there something special about the PRE notations in the example output below? I can list objects by any textual prefix, but I can't tell if the PRE (what we think of as folders) is more efficient than just the substring prefix.

Full bucket list, then two text prefix, then an (empty) folder list

  sokoloff@ Downloads % aws s3 ls s3://foo-asdf            
                             PRE bar-folder/
                             PRE baz-folder/
  2022-02-17 09:25:38          0 bar-file-1.txt
  2022-02-17 09:25:42          0 bar-file-2.txt
  2022-02-17 09:25:57          0 baz-file-1.txt
  2022-02-17 09:25:49          0 baz-file-2.txt
  sokoloff@ Downloads % aws s3 ls s3://foo-asdf/ba
                             PRE bar-folder/
                             PRE baz-folder/
  2022-02-17 09:25:38          0 bar-file-1.txt
  2022-02-17 09:25:42          0 bar-file-2.txt
  2022-02-17 09:25:57          0 baz-file-1.txt
  2022-02-17 09:25:49          0 baz-file-2.txt
  sokoloff@ Downloads % aws s3 ls s3://foo-asdf/bar
                             PRE bar-folder/
  2022-02-17 09:25:38          0 bar-file-1.txt
  2022-02-17 09:25:42          0 bar-file-2.txt
  sokoloff@ Downloads % aws s3 ls s3://foo-asdf/bar-folder
                             PRE bar-folder/

Re: AWS S3: Sometimes you should press the $100k button

#40

Post by sharken »

liveoneggs wrote: post #17

I have caused billing spikes like this before those little warnings were invented and it was always a dark day. They are really a life saver. Lifecycle rules are also welcome. Writing them yourself was always a pain and tended to be expensive with list operations eating up that api calls bill. ---- Once I supported an app that dumped small objects into s3 and begged the dev team to store the small objects in oracle a…

I suppose it's not just dev time on the line, but also the risk of doing the change that is thought to be too high.

If I ever get to be a manager I'd go for an idea such as yours. Though I suspect too many managers are too far removed from the technical aspect of things and don't listen nearly enough.

Post reply on HN