Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building and operating a pretty big storage system called S3

allthingsdistributed.com

Post reply on HN
Print view View original

101–110 of 169 posts

Re: Building and operating a pretty big storage system called S3

#101

Post by jdwithit »

Twirrim wrote: post #28
inopinatus wrote: post #21

Earlier quoted context omitted.

Never officially stated, but frequent leaks from insiders confirm that Glacier is based on Very Large Arrays of Wax Phonograph Records (VLAWPR) technology.

We came up with that idea in Glacier during the run up to April one year (2014, I think?), half jokingly suggested it as an April Fool's Day Joke, but Amazon quite reasonably decided against doing such jokes. One of the tag line ideas we had was "8 out of 10 customers say they prefer the feel of their data after it is restored"

This would have been incredible. But I guess I get the angle of not wanting to risk pissing off the audiophile CTO paying you 10 figures per month. Cause he can TOTALLY hear the difference listening to Dark Side of the Moon on vinyl via Monster Cables.

Re: Building and operating a pretty big storage system called S3

#102

Post by withinboredom »

epistasis wrote: post #99
withinboredom wrote: post #95

Earlier quoted context omitted.

Of course it’s slower. Your using https to do something that’s meant to be raw binary. The overhead is killing you. Something like iscsci, is pretty quick compared to https as a storage protocol.

Which is exactly why using standard Unix/POSIX files makes sense as a universal interface for genomics programs that are run in highly heterogeneous environments across the world, even if it leads to software engineers wishing that their internal custom data storage systems were used instead. If random access is needed in a cloud environment, use either that local ephemeral SSD, or a cloud block device which is proba…

I’m still waiting for people to realize nvme doesn’t have to be local in newer kernels. That’s when the real fun will begin.

Re: Building and operating a pretty big storage system called S3

#103

Post by anderspitman »

epistasis wrote: post #71
dekhn wrote: post #68

Earlier quoted context omitted.

I didn't make my statement out of ignorance. Most of these applications depend on OS optimizations that have been made over the decades; multithreaded readers, readahead, and caching are critically important to read performance. In principle, a remote storage system could be as fast as a local disk. This includes random access. after all, the storage system is just a bunch of drives attached to machines connected by…

Right, most people that try to really optimize these things do not have access to the parallelism tools thay Google has built, and end up doing their own ad-hoc sharding schemes. Things that can be built by 1-3 people over the course of a few weeks tk solve ann immediate scaling problem. And of course BAM itself dates back to before standardized serialization formats were brought out of Google. Even with potential op…

You might be interested in our paper that just got published in Bioinformatics today as chance would have it: https://academic.oup.com/bioinformatics/advance-article-abst...

Re: Building and operating a pretty big storage system called S3

#104

Post by jamesblonde »

Twirrim wrote: post #27
ilyt wrote: post #12

Earlier quoted context omitted.

I think Ceph hit similar problems and they had to add more robust checksumming to the system, as relying on just tcp checksums for integrity for example was no longer enough

Yes, I remember tcp checksumming coming up as not sufficient at one stage. Even saw S3 deal with a real head-scratcher of a non-impacting event that came down to a single NIC in a single machine corrupting the tcp checksum under very specific circumstances.

HDFS never relied on only network checksums. Blocks should be checksummed and validated at clients - a reliable end-to-end guarantee.

Re: Building and operating a pretty big storage system called S3

#105

Post by g9yuayon »

S3 is a truly amazing piece of technology. It offers peace of mind (well, almost), zero operations, and practically unlimited bandwidth for at least analytics workload. Indeed, it's so good that there has not been much progress in building an open-source alternative to S3. There seems not much activity in the Hadoop community. I have yet heard any company who uses RADOS on Ceph to handle PBs of data for analytics workload. MinIO made its name recently, but its license is restrictive and its community is quite small compared to that of Hadoop of its hay days.

Re: Building and operating a pretty big storage system called S3

#106

Post by baz00 »

ldjkfkdsjnv wrote: post #20
Twirrim wrote: post #2

> That’s a bit error rate of 1 in 10^15 requests. In the real world, we see that blade of grass get missed pretty frequently – and it’s actually something we need to account for in S3. One of the things I remember from my time at AWS was conversations about how 1 in a billion events end up being a daily occurrence when you're operating at S3 scale. Things that you'd normally mark off as so wildly improbable it's not…

Also worked at Amazon, saw some issues with major well known open source libraries that broke in places nobody would ever expect.

We get this on a much lower scale. We have to maintain many forks because no one is responsive on taking patches.

Re: Building and operating a pretty big storage system called S3

#107

Post by jamesblonde »

apitman wrote: post #11

The things we could build if S3 specified a simple OAuth2-based protocol for delegating read/write access. The world needs an HTTP-based protocol for apps to access data on the user's behalf. Google Drive is the closest to this but it only has a single provider and other issues[0]. I'm sad remoteStorage never caught on. I really hope Solid does well but it feels too complex to me. My own take on the problem is https:…

Most apps, however, assume POSIX-like data access. I would love to see a client-side minimally dependent library that mounts a local directory that is actually the user's S3 bucket.

Re: Building and operating a pretty big storage system called S3

#108

Post by Sparkyte »

g9yuayon wrote: post #105

S3 is a truly amazing piece of technology. It offers peace of mind (well, almost), zero operations, and practically unlimited bandwidth for at least analytics workload. Indeed, it's so good that there has not been much progress in building an open-source alternative to S3. There seems not much activity in the Hadoop community. I have yet heard any company who uses RADOS on Ceph to handle PBs of data for analytics wor…

There was a time when S3 was getting resilient. Today it is excellent. Pepridge Farms remembers.

Re: Building and operating a pretty big storage system called S3

#109

Post by epistasis »

anderspitman wrote: post #103
epistasis wrote: post #71

Earlier quoted context omitted.

Right, most people that try to really optimize these things do not have access to the parallelism tools thay Google has built, and end up doing their own ad-hoc sharding schemes. Things that can be built by 1-3 people over the course of a few weeks tk solve ann immediate scaling problem. And of course BAM itself dates back to before standardized serialization formats were brought out of Google. Even with potential op…

You might be interested in our paper that just got published in Bioinformatics today as chance would have it: https://academic.oup.com/bioinformatics/advance-article-abst...

Thanks for sharing, I'll check it out. It would be interesting to see if it would help with these pretty astonishing Open Omics results that recently came out:

https://community.intel.com/t5/Blogs/Tech-Innovation/Artific...

Re: Building and operating a pretty big storage system called S3

#110

Post by Sai_ »

dsalzman wrote: post #24

> Imagine a hard drive head as a 747 flying over a grassy field at 75 miles per hour. The air gap between the bottom of the plane and the top of the grass is two sheets of paper. Now, if we measure bits on the disk as blades of grass, the track width would be 4.6 blades of grass wide and the bit length would be one blade of grass. As the plane flew over the grass it would count blades of grass and only miss one blade…

The standing joke is that Americans love strange units of measure but this is one is so outre that it deserves an award.
Post reply on HN