S3 is a truly amazing piece of technology. It offers peace of mind (well, almost), zero operations, and practically unlimited bandwidth for at least analytics workload. Indeed, it's so good that there has not been much progress in building an open-source alternative to S3. There seems not much activity in the Hadoop community. I have yet heard any company who uses RADOS on Ceph to handle PBs of data for analytics wor…
> There seems not much activity in the Hadoop community There is apache ozone https://ozone.apache.org/
Building and operating a pretty big storage system called S3
151–160 of 169 posts
Re: Building and operating a pretty big storage system called S3
#152"As a really senior engineer in the company, of course I have strong opinions and I absolutely have a technical agenda. But If I interact with engineers by just trying to dispense ideas, it’s really hard for any of us to be successful. It’s a lot harder to get invested in an idea that you don’t own. So, when I work with teams, I’ve kind of taken the strategy that my best ideas are the ones that other people have inst…
That section really stood out to be as well. If Andy Warfield is reading, and I bet he is, I have a question. When developing a problem how valuable is it to sketch possible solutions? If you articulate the problem that probably springs to mind a few possible solutions. Is it worth sharing those possible solutions to help kickstart the gears for potential owners? Or is it better to focus only on the problem and let t…
So I started doing an experiment where I'd write that same doc, including the ideas i had on the shape of the work we should do, but then I'd delete my solution before sharing it. To your question: I'd still totally write my solution ideas down. Partially because I can't help myself and honestly it was a helpful way to think things through. But when I deleted it and shared a doc with just a problem statement, I'd get feedback on the problem statement. It's pretty obvious, but it was also a pretty surprising result: all of a sudden i was in conversations where we were all on the same side of the table. Feedback was either refining the problem (which was awesome) or proposing solutions. And when the person reading your problem statement starts trying to solve it, it's really cool... because they totally start getting invested and the conversations are great.
Like everything, none of this is actually either/or. There are points in between, like including a sketch of the shape of a solution, or properties that a solution would have to have. But the overall thing of separating the problem and the end state of where you want to get to, from the solution and the plan on how to get there is a pretty effective tool from a sharing ownership perspective.
Re: Building and operating a pretty big storage system called S3
#153Earlier quoted context omitted.
Which is exactly why using standard Unix/POSIX files makes sense as a universal interface for genomics programs that are run in highly heterogeneous environments across the world, even if it leads to software engineers wishing that their internal custom data storage systems were used instead. If random access is needed in a cloud environment, use either that local ephemeral SSD, or a cloud block device which is proba…
I’m still waiting for people to realize nvme doesn’t have to be local in newer kernels. That’s when the real fun will begin.
Re: Building and operating a pretty big storage system called S3
#154This is a lesson a lot of software people haven't yet learned. Bad UI, bad operational experiences, insufficient logging to resolve issues, un-fixable code because it's too complicated, and so on. But they use git.
The other term of art for this concept is "system engineering", in the aerospace sense. There are a lot of good texts and courses.
One example: Wesson: System Analysis Design and Development, Wiley, 2005. ISBN-10 0-471-39333-9
Re: Building and operating a pretty big storage system called S3
#155"As a really senior engineer in the company, of course I have strong opinions and I absolutely have a technical agenda. But If I interact with engineers by just trying to dispense ideas, it’s really hard for any of us to be successful. It’s a lot harder to get invested in an idea that you don’t own. So, when I work with teams, I’ve kind of taken the strategy that my best ideas are the ones that other people have inst…
I don't mean this to be cynical, but I do think that it's worth acknowledging that describing the problem is also, in itself, a tool to guide people towards a solution they want. After all, people often disagree about what "the problem" even is! Fortunately not every problem is like this. But if you look at, say, discussions around Python's "packaging problem" (and find people in fact describing like 6 different prob…
One advantage of focusing on describing the problem is that it naturally lets you have an impact on what you believe to be the important parts of the solution.
Re: Building and operating a pretty big storage system called S3
#156Earlier quoted context omitted.
That section really stood out to be as well. If Andy Warfield is reading, and I bet he is, I have a question. When developing a problem how valuable is it to sketch possible solutions? If you articulate the problem that probably springs to mind a few possible solutions. Is it worth sharing those possible solutions to help kickstart the gears for potential owners? Or is it better to focus only on the problem and let t…
Here's a really quick story on how i accidentally worked out this strategy by getting it wrong first. When I started at Amazon and was trying to convince the team that we should do certain things, I did what I'd always been trained to do: I wrote down the problem and then sketched a solution to it. Then I'd start floating the doc around to try to get folks excited about it. And invariably, they'd do what they were tr…
Re: Building and operating a pretty big storage system called S3
#157Earlier quoted context omitted.
Personally I'd love working in that kind of environment. That one in a billion hole still itches at me. There's also a slightly-perverse little voice in my head ready with popcorn in case I'm lucky enough to watch the ensuing fallout from the first major crypto hash collision :-).
That probability is significantly lower than one in a billion. One in a billion would be if keys were ~30 bits. Luckily it isn't.
Re: Building and operating a pretty big storage system called S3
#158Earlier quoted context omitted.
I don't mean this to be cynical, but I do think that it's worth acknowledging that describing the problem is also, in itself, a tool to guide people towards a solution they want. After all, people often disagree about what "the problem" even is! Fortunately not every problem is like this. But if you look at, say, discussions around Python's "packaging problem" (and find people in fact describing like 6 different prob…
I don't think it's cynical, I think it's the point. Describing the problem is not easy, and to your point, is sometimes controversial. One advantage of focusing on describing the problem is that it naturally lets you have an impact on what you believe to be the important parts of the solution.
Honestly the "real" hands off thing is letting somebody else also describe the problem and then probing it. But that might lead to a bit too much of an existential crisis for some people. And hey, if something works it works
Re: Building and operating a pretty big storage system called S3
#159Earlier quoted context omitted.
I don't think it's cynical, I think it's the point. Describing the problem is not easy, and to your point, is sometimes controversial. One advantage of focusing on describing the problem is that it naturally lets you have an impact on what you believe to be the important parts of the solution.
I just want to acknowledge that describing the problem is part of picking the solution, and it's not really _that much_ of a "I'm making the most neutral action and letting other people actually choose the solution". Honestly the "real" hands off thing is letting somebody else also describe the problem and then probing it. But that might lead to a bit too much of an existential crisis for some people. And hey, if som…
Re: Building and operating a pretty big storage system called S3
#160Earlier quoted context omitted.
Even at a billion requests per second, 128 bit UUIDs shouldn't collide for something like a billion years. And that's if you're going completely random and not taking care to try to reduce collisions.
There have been many cases of UUIDv4 collisions because an RNG wasn’t as random as expected, due to broken RNG or developer error. It is one of those cases where practice is not as reliable as theory, and it is banned in some places as a consequence. It depends on how paranoid you need to be.
Or do you mean certain folks intentionally chose substandard implementations for some reason?