Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building and operating a pretty big storage system called S3

allthingsdistributed.com

Post reply on HN
Print view View original

91–100 of 169 posts

Re: Building and operating a pretty big storage system called S3

#91

Post by polynomial »

jandrewrogers wrote: post #81
mabbo wrote: post #35

Earlier quoted context omitted.

Even at a billion requests per second, 128 bit UUIDs shouldn't collide for something like a billion years. And that's if you're going completely random and not taking care to try to reduce collisions.

There have been many cases of UUIDv4 collisions because an RNG wasn’t as random as expected, due to broken RNG or developer error. It is one of those cases where practice is not as reliable as theory, and it is banned in some places as a consequence. It depends on how paranoid you need to be.

Facts.

Re: Building and operating a pretty big storage system called S3

#92

Post by yashap »

dylan604 wrote: post #75
Narciss wrote: post #58

"As a really senior engineer in the company, of course I have strong opinions and I absolutely have a technical agenda. But If I interact with engineers by just trying to dispense ideas, it’s really hard for any of us to be successful. It’s a lot harder to get invested in an idea that you don’t own. So, when I work with teams, I’ve kind of taken the strategy that my best ideas are the ones that other people have inst…

There's a saying that I'm often told, and I'm sure we've all heard it at some point "don't bring me problems, bring me solutions". It's such a shit comment to make. I interpret it as if they are saying "You plebe! I don't have time for your issues. I can't get promoted from your work if you only bring problems." Being able to solve the problem is being able to understand the problem and admit it exists first.

Depends how it’s used. If it’s used in an org where major, high impact problems are ignored, as a way to just say “ignore all problems”, then yeah, it’s a shit comment.

However, if it’s used to legitimately say “don’t just complain, fix”, then I think it’s a positive. An organization where everyone is constantly negative and complaining about every little issue, but not working to implement improvements/fixes, is essentially a failed company. Successful companies are full of people who actively fix the high impact problems, while also being realists, who can accept that the low impact problems aren’t worth the effort to fix, and aren’t worth endlessly complaining about.

Re: Building and operating a pretty big storage system called S3

#93

Post by tsxxst »

forrestthewoods wrote: post #64
Narciss wrote: post #58

"As a really senior engineer in the company, of course I have strong opinions and I absolutely have a technical agenda. But If I interact with engineers by just trying to dispense ideas, it’s really hard for any of us to be successful. It’s a lot harder to get invested in an idea that you don’t own. So, when I work with teams, I’ve kind of taken the strategy that my best ideas are the ones that other people have inst…

That section really stood out to be as well. If Andy Warfield is reading, and I bet he is, I have a question. When developing a problem how valuable is it to sketch possible solutions? If you articulate the problem that probably springs to mind a few possible solutions. Is it worth sharing those possible solutions to help kickstart the gears for potential owners? Or is it better to focus only on the problem and let t…

For the "very senior IC", I'd recommend https://staffeng.com

Re: Building and operating a pretty big storage system called S3

#94

Post by 0cf8612b2e1e »

jl6 wrote: post #18

Great to see Amazon employees being allowed to talk openly about how S3 works behind the scenes. I would love to hear more about how Glacier works. As far as I know, they have never revealed what the underlying storage medium is, leading to a lot of wild speculation (tape? offline HDDs? custom HDDs?).

Are there any public details on how Azure or GCP do archival storage?

Re: Building and operating a pretty big storage system called S3

#95

Post by withinboredom »

epistasis wrote: post #71
dekhn wrote: post #68

Earlier quoted context omitted.

I didn't make my statement out of ignorance. Most of these applications depend on OS optimizations that have been made over the decades; multithreaded readers, readahead, and caching are critically important to read performance. In principle, a remote storage system could be as fast as a local disk. This includes random access. after all, the storage system is just a bunch of drives attached to machines connected by…

Right, most people that try to really optimize these things do not have access to the parallelism tools thay Google has built, and end up doing their own ad-hoc sharding schemes. Things that can be built by 1-3 people over the course of a few weeks tk solve ann immediate scaling problem. And of course BAM itself dates back to before standardized serialization formats were brought out of Google. Even with potential op…

Of course it’s slower. Your using https to do something that’s meant to be raw binary. The overhead is killing you. Something like iscsci, is pretty quick compared to https as a storage protocol.

Re: Building and operating a pretty big storage system called S3

#96

Post by jdwithit »

ignoramous wrote: post #4
Twirrim wrote: post #2

> That’s a bit error rate of 1 in 10^15 requests. In the real world, we see that blade of grass get missed pretty frequently – and it’s actually something we need to account for in S3. One of the things I remember from my time at AWS was conversations about how 1 in a billion events end up being a daily occurrence when you're operating at S3 scale. Things that you'd normally mark off as so wildly improbable it's not…

James Hamilton, AWS' chief architect, wrote about this phenomena in 2017: At scale, rare events aren't rare ; https://news.ycombinator.com/item?id=14038044

James' posts are always a treat. It's so rare to encounter such plain, straightforward content from someone with a title and responsibilities like his. Without layers of marketing sugar over everything. Dude just wants to post about the cool shit he did on his GeoCities-tier website and I love it.

Re: Building and operating a pretty big storage system called S3

#97

Post by jdwithit »

jl6 wrote: post #18

Great to see Amazon employees being allowed to talk openly about how S3 works behind the scenes. I would love to hear more about how Glacier works. As far as I know, they have never revealed what the underlying storage medium is, leading to a lot of wild speculation (tape? offline HDDs? custom HDDs?).

It's honestly super impressive that it's never leaked. All it takes is one engineer getting drunk and spouting off. In much higher stakes, a soldier in Massachusetts is about to go to jail for a long time for leaking national security intel on Discord to look cool to his gamer buddies. I would have expected details on Glacier to come out by now.

Re: Building and operating a pretty big storage system called S3

#98

Post by jdwithit »

bombcar wrote: post #65
anyoneamous wrote: post #37

Earlier quoted context omitted.

My impression is that the ambiguity gives them freedom to implement in different ways across different regions and over time. The original Glacier was very clearly tape, but given the instant retrieval capabilities the newer S3-Glacier tiers are most likely just low-margin HDDs, maybe with some dynamic powering on and off of drives/servers.

Glacier is just run on S3 with some sleep statements added.

I am going to choose to believe this

Re: Building and operating a pretty big storage system called S3

#99

Post by epistasis »

withinboredom wrote: post #95
epistasis wrote: post #71

Earlier quoted context omitted.

Right, most people that try to really optimize these things do not have access to the parallelism tools thay Google has built, and end up doing their own ad-hoc sharding schemes. Things that can be built by 1-3 people over the course of a few weeks tk solve ann immediate scaling problem. And of course BAM itself dates back to before standardized serialization formats were brought out of Google. Even with potential op…

Of course it’s slower. Your using https to do something that’s meant to be raw binary. The overhead is killing you. Something like iscsci, is pretty quick compared to https as a storage protocol.

Which is exactly why using standard Unix/POSIX files makes sense as a universal interface for genomics programs that are run in highly heterogeneous environments across the world, even if it leads to software engineers wishing that their internal custom data storage systems were used instead.

If random access is needed in a cloud environment, use either that local ephemeral SSD, or a cloud block device which is probably just an iSCSI implementation underneath, or at least a close equivalent.

Operating fleets of compute and IO in cloud environments means that POSIX semantics generally work really well for genomics.

Folks that have reimplemented basic genomics algorithms on top of protocol buffers standardized serialization still store them as BLOBs, and have not delivered benefits that can be realized in publicly available compute environments.

Re: Building and operating a pretty big storage system called S3

#100

Post by fomine3 »

jl6 wrote: post #18

Great to see Amazon employees being allowed to talk openly about how S3 works behind the scenes. I would love to hear more about how Glacier works. As far as I know, they have never revealed what the underlying storage medium is, leading to a lot of wild speculation (tape? offline HDDs? custom HDDs?).

I don't expect high salary engineers leak it, but random contractor at datacenter or supplier would eventually leak if they use special storage device other than HDD/SSD. Since we don't see any leaks, I suspect that it's based on HDD, with very long IO waitlist.
Post reply on HN