Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building and operating a pretty big storage system called S3

allthingsdistributed.com

Post reply on HN
Print view View original

41–50 of 169 posts

Re: Building and operating a pretty big storage system called S3

#41

Post by CobrastanJorji »

baq wrote: post #13

> What’s interesting here, when you look at the highest-level block diagram of S3’s technical design, is the fact that AWS tends to ship its org chart. This is a phrase that’s often used in a pretty disparaging way, but in this case it’s absolutely fascinating. I’d go even further: at this scale, it is essential and required to develop these kind of projects with any sort of velocity. Large organizations ship their c…

I'll take the metaphor one step further. The architecture will, over time, inevitably change to resemble its org chart, therefore it is the job of a sufficiently senior technical lead to organize the teams in such a way that the correct architecture emerges.

Re: Building and operating a pretty big storage system called S3

#42

Post by inopinatus »

Twirrim wrote: post #28
inopinatus wrote: post #21

Earlier quoted context omitted.

Never officially stated, but frequent leaks from insiders confirm that Glacier is based on Very Large Arrays of Wax Phonograph Records (VLAWPR) technology.

We came up with that idea in Glacier during the run up to April one year (2014, I think?), half jokingly suggested it as an April Fool's Day Joke, but Amazon quite reasonably decided against doing such jokes. One of the tag line ideas we had was "8 out of 10 customers say they prefer the feel of their data after it is restored"

The real problem is the lack of Star Wars references.

Re: Building and operating a pretty big storage system called S3

#43

Post by CobrastanJorji »

mcapodici wrote: post #26

S3 is more than storage. It is a standard. I like how you can get S3 compatible (usually with some small caveats) storage from a few places. I am not sure how open the standards is, and if you have to pay Amazon to say you are "S3 compatible" but it is pretty cool. Examples: iDrive has E2, Digital Ocean has Object Storage, Cloudflare has R2, Vultr has Object Storage, Backblaze has B2

Google's GCS as well, and I haven't used Microsoft, but it'd be weird if they didn't also have an "S3 compatible" option.

Edit: I looked it up and apparently no, Azure does not have one :-/

Re: Building and operating a pretty big storage system called S3

#44

Post by ttymck »

Spivak wrote: post #39
apitman wrote: post #11

The things we could build if S3 specified a simple OAuth2-based protocol for delegating read/write access. The world needs an HTTP-based protocol for apps to access data on the user's behalf. Google Drive is the closest to this but it only has a single provider and other issues[0]. I'm sad remoteStorage never caught on. I really hope Solid does well but it feels too complex to me. My own take on the problem is https:…

Such a system would be amazing. It would really force companies whose products are UIs on top of S3 to compete hard because adversarial interoperability would be an ever present threat from your competitors. It really is such a shame that all the projects that tried/are trying to create data sovereignty for users became weird crypto.

I agree with both halves of your comment, but I realized I can't identify the connection between S3 oauth and data sovereignty. Could you elaborate?

Re: Building and operating a pretty big storage system called S3

#45

Post by nijave »

acdha wrote: post #14
deathanatos wrote: post #9

> Now, let’s go back to that first hard drive, the IBM RAMAC from 1956. Here are some specs on that thing: > Storage Capacity: 3.75 MB > Cost: ~$9,200/terabyte Those specs can't possibly be correct. If you multiply the cost by the storage, the cost of the drive works out to 3¢. This site[1] states, > It stored about 2,000 bits of data per square inch and had a purchase price of about $10,000 per megabyte So perhaps t…

https://en.m.wikipedia.org/wiki/IBM_305_RAMAC has the likely source of the error: 30M bits (using the 6 data bits but not parity), but it rented for $3k per month so you didn’t have a set cost the same as buying a physical drive outright - very close to S3’s model, though.

I think this is still IBMs license model (at least a few years ago). It was explained to me you basically license a certain amount of compute even though the hardware is in your data center and you pay overages if you exceed your licensed throughput.

Since you license a fixed amount, there were projects at the company looking at running batch/non time sensitive jobs on the mainframe since it was effectively free off peak (I guess power cost was trivially compared to licensing).

Re: Building and operating a pretty big storage system called S3

#47

Post by capableweb »

CobrastanJorji wrote: post #41
baq wrote: post #13

> What’s interesting here, when you look at the highest-level block diagram of S3’s technical design, is the fact that AWS tends to ship its org chart. This is a phrase that’s often used in a pretty disparaging way, but in this case it’s absolutely fascinating. I’d go even further: at this scale, it is essential and required to develop these kind of projects with any sort of velocity. Large organizations ship their c…

I'll take the metaphor one step further. The architecture will, over time, inevitably change to resemble its org chart, therefore it is the job of a sufficiently senior technical lead to organize the teams in such a way that the correct architecture emerges.

Also known as "Conway's law"

> Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure

https://en.wikipedia.org/wiki/Conway%27s_law

Re: Building and operating a pretty big storage system called S3

#48

Post by simonw »

apitman wrote: post #11

The things we could build if S3 specified a simple OAuth2-based protocol for delegating read/write access. The world needs an HTTP-based protocol for apps to access data on the user's behalf. Google Drive is the closest to this but it only has a single provider and other issues[0]. I'm sad remoteStorage never caught on. I really hope Solid does well but it feels too complex to me. My own take on the problem is https:…

Absolutely this. I would LOVE to be able to build apps that store people's data in their own S3 bucket, billed to their own account.

Doing that right now is monumentally difficult. I built an entire CLI app just for solving the "issue AWS credentials that can only access this specific bucket" problem, but I really don't want to have to talk my users through installing and running something like that: https://s3-credentials.readthedocs.io/en/stable/

Re: Building and operating a pretty big storage system called S3

#49

Post by CobrastanJorji »

capableweb wrote: post #47
CobrastanJorji wrote: post #41

Earlier quoted context omitted.

I'll take the metaphor one step further. The architecture will, over time, inevitably change to resemble its org chart, therefore it is the job of a sufficiently senior technical lead to organize the teams in such a way that the correct architecture emerges.

Also known as "Conway's law" > Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure https://en.wikipedia.org/wiki/Conway%27s_law

Right. Conway's Law describes the property that the architecture will grow to resemble the org chart. I'm suggesting that you can productively apply that principal to produce good software by shaping the org chart.

If Conway's Law is phrenology, the "science" of determining someone's personality by measuring their skull's dimensions, I'm suggesting Terry Prachett's retrophrenology, the process of hitting someone with a hammer very precisely to make them a better person.

Re: Building and operating a pretty big storage system called S3

#50

Post by donavanm »

CobrastanJorji wrote: post #40
epistasis wrote: post #7

Working in genomics, I've dealt with lots of petabyte data stores over the past decade. Having used AWS S3, GCP GCS, and a raft of storage systems for collocated hardware (Ceph, Gluster, and an HP system whose name I have blocked from my memory), I have no small amount of appreciation for the effort that goes into operating these sorts of systems. And the benefits of sharing disk IOPs with untold numbers of other cus…

It also explains some of the cost model for cloud storage. The best possible customer, from a cloud storage perspective, stores a whole lot of data but reads almost none of it. That's kind of like renting hard drives, except if you only fill some of each hard drive with the "cold" data, you can still use the hard drive's full I/O capacity to handle the hot work. So, if you very carefully balance what sort of data is…

You get similar properties/challenges in lots of multi consumer storage scenarios. I learned lots of similar lessons working on CDNs when it comes to object distribution and access rates.

If youre interested go search for some of the published work from "Coho Data", they had some great usenix presentations IIRC. This was the previous company Andy Warfield was at and they had an emphasis on effective tracking & prediction of IO workloads across very large datasets.

Post reply on HN