Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building and operating a pretty big storage system called S3

allthingsdistributed.com

Post reply on HN
Print view View original

71–80 of 169 posts

Re: Building and operating a pretty big storage system called S3

#71

Post by epistasis »

dekhn wrote: post #68
epistasis wrote: post #66

Earlier quoted context omitted.

Most of these tools treat the "local file" as a stream which can be a pipe to a network stream from the object store. The files that are not streamed and need random access are often better on a local ephemeral SSDs or in RAM after a fetch of the, say, 50GB hash table, or whatever it is. At least, that's my experience: streams and in-RAM pre-processed DBs are >99% of file IO.

I didn't make my statement out of ignorance. Most of these applications depend on OS optimizations that have been made over the decades; multithreaded readers, readahead, and caching are critically important to read performance. In principle, a remote storage system could be as fast as a local disk. This includes random access. after all, the storage system is just a bunch of drives attached to machines connected by…

Right, most people that try to really optimize these things do not have access to the parallelism tools thay Google has built, and end up doing their own ad-hoc sharding schemes. Things that can be built by 1-3 people over the course of a few weeks tk solve ann immediate scaling problem. And of course BAM itself dates back to before standardized serialization formats were brought out of Google.

Even with potential optimizations, initiating a seek on GCS or S3 is far far slower than on a local SSD, so even if Google exposes fast cross-network seeks on objects inside an internal object store system, it is not readily accessible to the plebes like me and 99.9% of genomicists that use cloud systems or their own hardware.

Re: Building and operating a pretty big storage system called S3

#72

Post by mannyv »

What most people don't realize is that the magic isn't in handling the system itself; the magic is making authorization appear to be zero-cost.

In distributed systems authorization is incredibly difficult. At the scale of AWS it might as well be magic. AWS has a rich permissions model with changes to authorization bubbling through the infrastructure at sub-millisecond speed - while handling probably trillions of requests.

This and logging/accounting for billing are the two magic pieces of AWS that I'd love to see an article about.

Note that S3 does AA differently than other services, because the permissions are on the resource. I suspect that's for speed?

Re: Building and operating a pretty big storage system called S3

#73

Post by seized »

dekhn wrote: post #70
seized wrote: post #63

Earlier quoted context omitted.

The latency is higher so the key is parallelism... Which means you need more cores/hardware/VMs/pick your poison. New but same problem...

Is single-job performance the only criterion? Or can you just run a bunch of different jobs at the same time (genomics has many embarassingly parallel problems, often per-sample) and use the higher aggregate storage bandwidth of your object store to get "more work done in unit time".

I did the latter, across usually about 4000+ CPUs. That got me a peak of 15 or so GB/sec read from one GCS bucket, writing to another.

But yeah if it's not something that can be paralleled then it sucks.

Re: Building and operating a pretty big storage system called S3

#74

Post by skybrian »

rtpg wrote: post #67
Narciss wrote: post #58

"As a really senior engineer in the company, of course I have strong opinions and I absolutely have a technical agenda. But If I interact with engineers by just trying to dispense ideas, it’s really hard for any of us to be successful. It’s a lot harder to get invested in an idea that you don’t own. So, when I work with teams, I’ve kind of taken the strategy that my best ideas are the ones that other people have inst…

I don't mean this to be cynical, but I do think that it's worth acknowledging that describing the problem is also, in itself, a tool to guide people towards a solution they want. After all, people often disagree about what "the problem" even is! Fortunately not every problem is like this. But if you look at, say, discussions around Python's "packaging problem" (and find people in fact describing like 6 different prob…

At a toy scale, using ChatGPT's Code Interpreter to do some programming for fun can be an exercise in getting what you want from an inconsistent worker by changing the problem definition (prompt engineering).

This is sort of like:

* writing an exam question so the person taking the exam is likely to get the answer you want

* guiding someone in a code interview that isn't going so well, without giving away the answer

* being in the back seat while pair programming, except you're not allowed to take a turn at the keyboard

Re: Building and operating a pretty big storage system called S3

#75

Post by dylan604 »

Narciss wrote: post #58

"As a really senior engineer in the company, of course I have strong opinions and I absolutely have a technical agenda. But If I interact with engineers by just trying to dispense ideas, it’s really hard for any of us to be successful. It’s a lot harder to get invested in an idea that you don’t own. So, when I work with teams, I’ve kind of taken the strategy that my best ideas are the ones that other people have inst…

There's a saying that I'm often told, and I'm sure we've all heard it at some point "don't bring me problems, bring me solutions". It's such a shit comment to make.

I interpret it as if they are saying "You plebe! I don't have time for your issues. I can't get promoted from your work if you only bring problems."

Being able to solve the problem is being able to understand the problem and admit it exists first.

Re: Building and operating a pretty big storage system called S3

#76

Post by niscocity35 »

Narciss wrote: post #58

"As a really senior engineer in the company, of course I have strong opinions and I absolutely have a technical agenda. But If I interact with engineers by just trying to dispense ideas, it’s really hard for any of us to be successful. It’s a lot harder to get invested in an idea that you don’t own. So, when I work with teams, I’ve kind of taken the strategy that my best ideas are the ones that other people have inst…

this only works if your team are made up of smart competent people.

Re: Building and operating a pretty big storage system called S3

#77

Post by [deleted] »

jl6 wrote: post #18

Great to see Amazon employees being allowed to talk openly about how S3 works behind the scenes. I would love to hear more about how Glacier works. As far as I know, they have never revealed what the underlying storage medium is, leading to a lot of wild speculation (tape? offline HDDs? custom HDDs?).

[deleted]

Re: Building and operating a pretty big storage system called S3

#78

Post by ChainOfFools »

Narciss wrote: post #58

"As a really senior engineer in the company, of course I have strong opinions and I absolutely have a technical agenda. But If I interact with engineers by just trying to dispense ideas, it’s really hard for any of us to be successful. It’s a lot harder to get invested in an idea that you don’t own. So, when I work with teams, I’ve kind of taken the strategy that my best ideas are the ones that other people have inst…

I strongly agree with this perspective but I wish it could be generalized into techniques that work in everyday life, where there isn't already this established ranking of expertise that focuses attention on what is being said and not whether you have the clout or the authority to say it.

Because absent preestablied perceived authority or expertise, which is the context that most day to day problems surface within, holding forth and hogging the entire two-way discussion channel with your long detailed and carefully articulated description of the problem is going to make you sound like someone who wants to do all the talking and none of the work, or the kind of person who doesn't want to share in finding a solution together with others.

Re: Building and operating a pretty big storage system called S3

#79

Post by gooseyman »

This is a fantastic point on ownership that those “placing” it on others can often miss.

“Ownership carries a lot of responsibility, but it also carries a lot of trust – because to let an individual or a team own a service, you have to give them the leeway to make their own decisions about how they are going to deliver it.”

Re: Building and operating a pretty big storage system called S3

#80

Post by anderspitman »

ent101 wrote: post #53
apitman wrote: post #11

The things we could build if S3 specified a simple OAuth2-based protocol for delegating read/write access. The world needs an HTTP-based protocol for apps to access data on the user's behalf. Google Drive is the closest to this but it only has a single provider and other issues[0]. I'm sad remoteStorage never caught on. I really hope Solid does well but it feels too complex to me. My own take on the problem is https:…

We're building this at https://puter.com

You mean you're implementing something like this to be used by puter.com?
Post reply on HN