Live data from Hacker News

Torus: A distributed storage system by CoreOS

coreos.com

61–70 of 187 posts

Re: Torus: A distributed storage system by CoreOS

#61

It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "…

> unaware of how difficult it is to even just reliably get bits to and from stable storage

It's not difficult, it's actually impossible. And any system relying on the reliability of that would be broken by design.

> let alone string that into a distributed system that must make CAP tradeoffs

No. Distributed systems and "CAP tradeoffs" actually exist to solve the problem above with predictable certainty.

Re: Torus: A distributed storage system by CoreOS

#62
post #31
post #19

Earlier quoted context omitted.

> I'm still hoping for an open-source competitor for elasticsearch I thought Elasticsearch was open-sourced, and the GitHub repo says Apache v2. What am I missing here?

Nothing. It is open-source. But it needs an open-source competitor. There's too little happening at the open-source side of search.

Huh? It has a major competitor: Solr. I'm not sure what you mean unless you object because they both use lucene.

https://lucene.apache.org/solr/

Re: Torus: A distributed storage system by CoreOS

#63
Good... Good... Let the hate flow through you.

An open source OS company just blogged about a new open source project that they are putting resources into. They did not release a commercial product that competes with any existing storage solution.

How exactly would one expect a new project to be announced? Tell me more about how far away from done you think they are.

Sorry if that's a bit sarcastic, but seriously wouldn't it be nice if these comments where more along the lines of: "Interesting new project, here's some issues we've run into vis a vis reliable storage that you should look out for..."

Re: Torus: A distributed storage system by CoreOS

#64
post #43
post #41

Earlier quoted context omitted.

Actually, persistent storage is fairly hard in itself. Look at what ZFS does to ensure data integrity in the face of phantom writes, dropped writes, bad controllers, and other implicit, non-fatal failures.

ZFS still also have the issue of having to perform well. You have a point, but ZFS is still trivial compared to a proper distributed filesystem, and you could achieve the same reliability much easier than ZFS if you sacrificed the performance.

The ClusterHQ guys behind the FlockerHQ found this out the hard way [0]. Initially Flocker was meant to provide a container data migration tool on top of ZFS, now it is a front-end to more established storage systems like Cinder,vSan,EBS,GPD and so on.

[0] https://docs.clusterhq.com/en/latest/faq/#what-happened-to-z...

Re: Torus: A distributed storage system by CoreOS

#65

It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "…

I totally agree with you. I also liked how they said the motivation is to make Google infrastructure for everyone else. How did Google do this? They basically imitated and improved on clustered filesystems developed in HPC. There were a lot of lessons to learn for emerging cloud market in all tooling done in HPC. Some was FOSS. Whereas, many companies seem to be doing the opposite in their work on these cloud filesys…

While I strongly agree with bcantrill, I have to disagree with your HPC origins model in the most strenuous fashion.

The HPC world can be thought of as an alternative universe, one where dinosaurs came to some sort of lazy sentience instead us mammals. HPC RAS (reliability, availability, serviceability) requirements and demands are profoundly different and profoundly lower from what enterprise or even SMB (small & medium sized businesses) would find acceptable. There are or were interesting ideas in GPFS but that was long ago, far away, and sterile besides - not that it stops IBM squeezing revenue from locked-in customers.

Some people in HPC do realize their insular state, but mostly there are hopes for things to be done better that were done better earlier (e.g. "declustered RAID") in the "real" world. Their ongoing hype is for yet-another-meta-filesystem built with feet of straw upon a pile of manure.

The likes of Google infrastructure, are built instead on academic storage R&D, which has been a great source of ideas and interesting test implementations - log structured file systems, distributed hash tables, etc. There is some small overlap, for example in the work of Garth Gibson, but it's not a joint evolution at all.

Re: Torus: A distributed storage system by CoreOS

#67

It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "…

I totally agree with you. I also liked how they said the motivation is to make Google infrastructure for everyone else. How did Google do this? They basically imitated and improved on clustered filesystems developed in HPC. There were a lot of lessons to learn for emerging cloud market in all tooling done in HPC. Some was FOSS. Whereas, many companies seem to be doing the opposite in their work on these cloud filesys…

Yes, that's absolutely been my experience -- and even then, when it comes to the data path, you will likely find new failure modes in "tried and true" as you push it harder and longer and with the bar being set at absolute perfection. I have learned this painful lesson twice: first, with Fishworks at Sun when we turned ZFS into a storage appliance -- and we learned the painful difference between something that seems to work all of the time and something that actually works all of the time. (2009 was a really tough year.[1]) And ZFS was fundamentally sound (and certainly had been running for years in production) before we pushed it into the broad enterprise storage substrate: the bugs that we found weren't ones of durability, but rather of deeply pathological performance. (As I was fond of saying at the time, we never lost anyone's data -- but we has some data take some very, very long vacations.) I shudder to think about building a data path on much less proven components than ZFS circa 2008, let alone building a data path seemingly in total ignorance of the mechanics -- let alone challenges -- of writing to persistent storage.

The second time I learned the painful lessons of storage was with Manta.[2] Here again, we built on ZFS and reliable, proven "tried and true" technologies like PostgreSQL and Zookeeper. And here again, we learned about really nasty, surprising failure modes at the margins.[3] These failure modes haven't led to data loss -- but when someone's data is unavailable, that is of little solace. In this regard, the data path -- the world of persistent state -- is a different world in terms of expectations for quality. That most of our domain thinks in terms of stateless apps is probably a good thing: state is a very hard thing to get right, and, in your words, tried and true absolutely beats novel and new. All of this is what makes Torus's ignorance of what comes before it so exasperating; one gets the sense that if they understood how thorny this problem actually is, they would be trying much harder to use the proven open source components out there rather than attempt to sloppily (if cheerfully) reinvent them.

[1] http://dtrace.org/blogs/bmc/2010/03/10/turning-the-corner/

[2] https://github.com/joyent/manta

[3] https://www.joyent.com/blog/manta-postmortem-7-27-2015

Re: Torus: A distributed storage system by CoreOS

#68

Earlier quoted context omitted.

Ceph, OpenStack Swift, GlusterFS, OrangeFS, Lustre. https://en.wikipedia.org/wiki/Clustered_file_system#Distribu... https://en.wikipedia.org/wiki/Object_storage

If Lustre is the answer, you're probably asking the wrong question. Unless that question involves short life-span, massively parallel swap-outs from one weapons simulation to another. I write this not as storage religion (of which there's far too much), but to warn away those who haven't experienced the many kinds of data (and stomach lining) loss that come with being a Lustre admin.

https://www.csc.fi/web/blog/post/-/blogs/the-largest-unplann...

Re: Torus: A distributed storage system by CoreOS

#69
post #61

It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "…

> unaware of how difficult it is to even just reliably get bits to and from stable storage It's not difficult, it's actually impossible. And any system relying on the reliability of that would be broken by design. > let alone string that into a distributed system that must make CAP tradeoffs No. Distributed systems and "CAP tradeoffs" actually exist to solve the problem above with predictable certainty.

It seems you're disagreeing with yourself.

Re: Torus: A distributed storage system by CoreOS

#70

Earlier quoted context omitted.

I totally agree with you. I also liked how they said the motivation is to make Google infrastructure for everyone else. How did Google do this? They basically imitated and improved on clustered filesystems developed in HPC. There were a lot of lessons to learn for emerging cloud market in all tooling done in HPC. Some was FOSS. Whereas, many companies seem to be doing the opposite in their work on these cloud filesys…

While I strongly agree with bcantrill, I have to disagree with your HPC origins model in the most strenuous fashion. The HPC world can be thought of as an alternative universe, one where dinosaurs came to some sort of lazy sentience instead us mammals. HPC RAS (reliability, availability, serviceability) requirements and demands are profoundly different and profoundly lower from what enterprise or even SMB (small & me…

HPC as an alternative universe is a fine metaphor. Yet, I don't see how there's no comparison given HPC already had more RAS than I could get out of any Linux box for years. IBM SP2, SGI, and some Beowulf-style clusters had high-performance I/O, single-system image, distributed filesystems, and fault-tolerance where you wanted. That sounds a lot like what cloud market aimed for. On top of it, you had grid and metacomputing groups that spread it out across multiple sites with ease of use for developers.

They developed in parallel and separately but there's lots of similarities. Many startups in HPC were taking HPC stuff to meet cloud-style requirements for years in terms of management, reliability, and performance. That it wasn't up to the standards of businesses is countered by how many bought those NUMA machines and clusters for mission-critical apps. A huge chunk of work in Windows and Linux in 90's and early 2000's went to trying to match their metrics in order to replace them on the cheap with x86 servers and Ethernet. So, yes, HPC was making stuff to meet tough, business needs that commercial servers and regular stacks couldn't compare to. That's despite number-crunching supercomputers being their main market.

Post reply on HN