Live data from Hacker News

Building and operating a pretty big storage system called S3

allthingsdistributed.com

151–160 of 169 posts

Re: Building and operating a pretty big storage system called S3

#151

S3 is a truly amazing piece of technology. It offers peace of mind (well, almost), zero operations, and practically unlimited bandwidth for at least analytics workload. Indeed, it's so good that there has not been much progress in building an open-source alternative to S3. There seems not much activity in the Hadoop community. I have yet heard any company who uses RADOS on Ceph to handle PBs of data for analytics wor…

> There seems not much activity in the Hadoop community There is apache ozone https://ozone.apache.org/

Yeah, Ozone looks interesting. I was just not sure who used it at scale other than a Japanese startup. The community engagement seems much lower than other communities, though.

Re: Building and operating a pretty big storage system called S3

#152
post #58

"As a really senior engineer in the company, of course I have strong opinions and I absolutely have a technical agenda. But If I interact with engineers by just trying to dispense ideas, it’s really hard for any of us to be successful. It’s a lot harder to get invested in an idea that you don’t own. So, when I work with teams, I’ve kind of taken the strategy that my best ideas are the ones that other people have inst…

That section really stood out to be as well. If Andy Warfield is reading, and I bet he is, I have a question. When developing a problem how valuable is it to sketch possible solutions? If you articulate the problem that probably springs to mind a few possible solutions. Is it worth sharing those possible solutions to help kickstart the gears for potential owners? Or is it better to focus only on the problem and let t…

Here's a really quick story on how i accidentally worked out this strategy by getting it wrong first. When I started at Amazon and was trying to convince the team that we should do certain things, I did what I'd always been trained to do: I wrote down the problem and then sketched a solution to it. Then I'd start floating the doc around to try to get folks excited about it. And invariably, they'd do what they were trained to do, which was to have a critical response to the proposed solution. They'd argue that I was solving it the wrong way, and I'd be in a spot where we'd have a conversation where I was defending a position. But this was the last thing I wanted — I was trying to get everyone excited about fixing a problem, but I slowly realized that when I approached it this way, I was just getting feedback on my proposed solution.

So I started doing an experiment where I'd write that same doc, including the ideas i had on the shape of the work we should do, but then I'd delete my solution before sharing it. To your question: I'd still totally write my solution ideas down. Partially because I can't help myself and honestly it was a helpful way to think things through. But when I deleted it and shared a doc with just a problem statement, I'd get feedback on the problem statement. It's pretty obvious, but it was also a pretty surprising result: all of a sudden i was in conversations where we were all on the same side of the table. Feedback was either refining the problem (which was awesome) or proposing solutions. And when the person reading your problem statement starts trying to solve it, it's really cool... because they totally start getting invested and the conversations are great.

Like everything, none of this is actually either/or. There are points in between, like including a sketch of the shape of a solution, or properties that a solution would have to have. But the overall thing of separating the problem and the end state of where you want to get to, from the solution and the plan on how to get there is a pretty effective tool from a sharing ownership perspective.

Re: Building and operating a pretty big storage system called S3

#153

Earlier quoted context omitted.

Which is exactly why using standard Unix/POSIX files makes sense as a universal interface for genomics programs that are run in highly heterogeneous environments across the world, even if it leads to software engineers wishing that their internal custom data storage systems were used instead. If random access is needed in a cloud environment, use either that local ephemeral SSD, or a cloud block device which is proba…

I’m still waiting for people to realize nvme doesn’t have to be local in newer kernels. That’s when the real fun will begin.

I used to work on datacenter NVMe products. I wrote the tests which validated them (mostly functional not performance). I left that company before things got hot with fabrics, but I really want to see that stuff succeed. It looked really cool.

Re: Building and operating a pretty big storage system called S3

#154
> It’s all one thing, and you can’t really think about it just as software. It’s > software, hardware, and people, and it’s always growing and constantly evolving.

This is a lesson a lot of software people haven't yet learned. Bad UI, bad operational experiences, insufficient logging to resolve issues, un-fixable code because it's too complicated, and so on. But they use git.

The other term of art for this concept is "system engineering", in the aerospace sense. There are a lot of good texts and courses.

One example: Wesson: System Analysis Design and Development, Wiley, 2005. ISBN-10 0-471-39333-9

Re: Building and operating a pretty big storage system called S3

#155
post #67
post #58

"As a really senior engineer in the company, of course I have strong opinions and I absolutely have a technical agenda. But If I interact with engineers by just trying to dispense ideas, it’s really hard for any of us to be successful. It’s a lot harder to get invested in an idea that you don’t own. So, when I work with teams, I’ve kind of taken the strategy that my best ideas are the ones that other people have inst…

I don't mean this to be cynical, but I do think that it's worth acknowledging that describing the problem is also, in itself, a tool to guide people towards a solution they want. After all, people often disagree about what "the problem" even is! Fortunately not every problem is like this. But if you look at, say, discussions around Python's "packaging problem" (and find people in fact describing like 6 different prob…

I don't think it's cynical, I think it's the point. Describing the problem is not easy, and to your point, is sometimes controversial.

One advantage of focusing on describing the problem is that it naturally lets you have an impact on what you believe to be the important parts of the solution.

Re: Building and operating a pretty big storage system called S3

#156

Earlier quoted context omitted.

That section really stood out to be as well. If Andy Warfield is reading, and I bet he is, I have a question. When developing a problem how valuable is it to sketch possible solutions? If you articulate the problem that probably springs to mind a few possible solutions. Is it worth sharing those possible solutions to help kickstart the gears for potential owners? Or is it better to focus only on the problem and let t…

Here's a really quick story on how i accidentally worked out this strategy by getting it wrong first. When I started at Amazon and was trying to convince the team that we should do certain things, I did what I'd always been trained to do: I wrote down the problem and then sketched a solution to it. Then I'd start floating the doc around to try to get folks excited about it. And invariably, they'd do what they were tr…

That’s helpful. Thank you!

Re: Building and operating a pretty big storage system called S3

#157
post #123

Earlier quoted context omitted.

Personally I'd love working in that kind of environment. That one in a billion hole still itches at me. There's also a slightly-perverse little voice in my head ready with popcorn in case I'm lucky enough to watch the ensuing fallout from the first major crypto hash collision :-).

That probability is significantly lower than one in a billion. One in a billion would be if keys were ~30 bits. Luckily it isn't.

The one in a billion was in reference to storage related stats described in the article. Not private crypto keys.

Re: Building and operating a pretty big storage system called S3

#158
post #67

Earlier quoted context omitted.

I don't mean this to be cynical, but I do think that it's worth acknowledging that describing the problem is also, in itself, a tool to guide people towards a solution they want. After all, people often disagree about what "the problem" even is! Fortunately not every problem is like this. But if you look at, say, discussions around Python's "packaging problem" (and find people in fact describing like 6 different prob…

I don't think it's cynical, I think it's the point. Describing the problem is not easy, and to your point, is sometimes controversial. One advantage of focusing on describing the problem is that it naturally lets you have an impact on what you believe to be the important parts of the solution.

I just want to acknowledge that describing the problem is part of picking the solution, and it's not really _that much_ of a "I'm making the most neutral action and letting other people actually choose the solution".

Honestly the "real" hands off thing is letting somebody else also describe the problem and then probing it. But that might lead to a bit too much of an existential crisis for some people. And hey, if something works it works

Re: Building and operating a pretty big storage system called S3

#159
post #158

Earlier quoted context omitted.

I don't think it's cynical, I think it's the point. Describing the problem is not easy, and to your point, is sometimes controversial. One advantage of focusing on describing the problem is that it naturally lets you have an impact on what you believe to be the important parts of the solution.

I just want to acknowledge that describing the problem is part of picking the solution, and it's not really _that much_ of a "I'm making the most neutral action and letting other people actually choose the solution". Honestly the "real" hands off thing is letting somebody else also describe the problem and then probing it. But that might lead to a bit too much of an existential crisis for some people. And hey, if som…

For sure, it’s only partly hands off. But he is an engineer after all, he should be doing something outside of just managing.

Re: Building and operating a pretty big storage system called S3

#160
post #35

Earlier quoted context omitted.

Even at a billion requests per second, 128 bit UUIDs shouldn't collide for something like a billion years. And that's if you're going completely random and not taking care to try to reduce collisions.

There have been many cases of UUIDv4 collisions because an RNG wasn’t as random as expected, due to broken RNG or developer error. It is one of those cases where practice is not as reliable as theory, and it is banned in some places as a consequence. It depends on how paranoid you need to be.

NIST standards on RNG are not as random as expected?

Or do you mean certain folks intentionally chose substandard implementations for some reason?

Post reply on HN