Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Torus: A distributed storage system by CoreOS

coreos.com

Post reply on HN
Print view View original

71–80 of 187 posts

Re: Torus: A distributed storage system by CoreOS

#71

Post by pinewurst »

nickpsecurity wrote: post #70
pinewurst wrote: post #65

Earlier quoted context omitted.

While I strongly agree with bcantrill, I have to disagree with your HPC origins model in the most strenuous fashion. The HPC world can be thought of as an alternative universe, one where dinosaurs came to some sort of lazy sentience instead us mammals. HPC RAS (reliability, availability, serviceability) requirements and demands are profoundly different and profoundly lower from what enterprise or even SMB (small & me…

HPC as an alternative universe is a fine metaphor. Yet, I don't see how there's no comparison given HPC already had more RAS than I could get out of any Linux box for years. IBM SP2, SGI, and some Beowulf-style clusters had high-performance I/O, single-system image, distributed filesystems, and fault-tolerance where you wanted. That sounds a lot like what cloud market aimed for. On top of it, you had grid and metacom…

Maybe things have fallen back drastically then or we're using different definitions of HPC (it's the likes of weather or bombs for me).

Supercomputer system MTBF numbers aren't high at all. Supercomputer storage is fault tolerant only by fortuitous accident and often requires week-long outages with destroyed data for software upgrades. These systems are built with the likes of dubious metafilesystems (I'm talking to you, Lustre) sitting on top of shaky file systems on top of junk-bin HW RAID systems almost guaranteed to corrupt your data over time.

I think your overall statement is valuable with the global search/replace of "enterprise" for "HPC" - airline reservations or credit card processing. That's all I'm saying - HPC people (at least currently) are massively overrated as developers and as admins. Maybe it's all just a plan to extract energy from Seymour Cray's pulsar-like rotating remains?

Re: Torus: A distributed storage system by CoreOS

#72

Post by zzzcpan »

halayli wrote: post #69
zzzcpan wrote: post #61

Earlier quoted context omitted.

> unaware of how difficult it is to even just reliably get bits to and from stable storage It's not difficult, it's actually impossible. And any system relying on the reliability of that would be broken by design. > let alone string that into a distributed system that must make CAP tradeoffs No. Distributed systems and "CAP tradeoffs" actually exist to solve the problem above with predictable certainty.

It seems you're disagreeing with yourself.

Care to point out where? I can explain if something seems too ambiguous.

It is impossible to reliably store something in your local storage. It is possible to achieve some probability of data retention and availability in a distributed system.

Re: Torus: A distributed storage system by CoreOS

#73

Post by nickpsecurity »

pinewurst wrote: post #71
nickpsecurity wrote: post #70

Earlier quoted context omitted.

HPC as an alternative universe is a fine metaphor. Yet, I don't see how there's no comparison given HPC already had more RAS than I could get out of any Linux box for years. IBM SP2, SGI, and some Beowulf-style clusters had high-performance I/O, single-system image, distributed filesystems, and fault-tolerance where you wanted. That sounds a lot like what cloud market aimed for. On top of it, you had grid and metacom…

Maybe things have fallen back drastically then or we're using different definitions of HPC (it's the likes of weather or bombs for me). Supercomputer system MTBF numbers aren't high at all. Supercomputer storage is fault tolerant only by fortuitous accident and often requires week-long outages with destroyed data for software upgrades. These systems are built with the likes of dubious metafilesystems (I'm talking to…

" Supercomputer storage is fault tolerant only by fortuitous accident and often requires week-long outages with destroyed data for software upgrades."

Have you tried Ceph or Sector/Sphere? Lustre is known to be crap while Ceph gets a lot of praise and Sector/Sphere has potential for reliability with good performance. I think you may just be stuck with tools that suck. I'll admit it was about 10+ years ago when I was knowledgeable of this area. It could've all gone downhill since.

"That's all I'm saying - HPC people (at least currently) are massively overrated as developers and as admins. "

I'll agree with that. It's one of reasons field built so much tooling to make up for it. ;)

Re: Torus: A distributed storage system by CoreOS

#74

Post by sysexit »

schmichael wrote: post #54
sysexit wrote: post #30

I love CoreOS and they've done some super impressive engineering. But really, a new storage system? Rewriting in Go and using etcd for central state management makes things easier, but this is still a hard problem. Some things that need to be solved sooner or later: data replication so that N faults of X entities are protected against (X can be disks, enclosures, racks, data centers, regions, ..), recovery from faile…

> But really, a new storage system? I think the post outlines quite well why they chose to build a new distributed storage layer. From a product perspective it makes a lot of sense for them to have an offering in this space. Also, the fact that it's a hard problem is what makes it a compelling product offering. You can't make much money solving easy problems.

> Also, the fact that it's a hard problem is what makes it a compelling product offering. You can't make much money solving easy problems.

If they can solve the hard problem, yes. Which I believe is unlikely.

Re: Torus: A distributed storage system by CoreOS

#75

Post by halayli »

bcantrill wrote: post #50

It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "…

Looking at dtrace, fishwork, zfs and Solaris (imo was the best OS technologically speaking) it's interesting to see how strong and innovative its engineering team was while the business was just going down. How did that happen? How can the engineering team be that productive and functional while the business vision was so lacking?

Re: Torus: A distributed storage system by CoreOS

#76

Post by zxcvcxz »

It's threads like these where you start to wonder just how many marketing teams are arguing with each other in the comments. I take the FUD with a grain of salt: there is a lot of financial incentive to create FUD, where there is financial incentive and little to no regulations a market will naturally arise. It's going to get worse as the bubble pops and companies become more and more desperate.

Re: Torus: A distributed storage system by CoreOS

#77

Post by objectivefs »

gsmethells wrote: post #28
zsmith928 wrote: post #8

Earlier quoted context omitted.

what is the existing alternative for storage in distributed container environments?

Ceph, OpenStack Swift, GlusterFS, OrangeFS, Lustre. https://en.wikipedia.org/wiki/Clustered_file_system#Distribu... https://en.wikipedia.org/wiki/Object_storage

ObjectiveFS[1] is another option if you have access to an object store (e.g. S3 or GCS) from your containers.

[1] https://objectivefs.com

Re: Torus: A distributed storage system by CoreOS

#78

Post by lobster_johnson »

paulsutter wrote: post #46
philips wrote: post #2

The CoreOS team is excited to make this initial release and start collaborating with folks that want to tackle distributed storage. tl;dr this is a new OSS distributed storage project that is written in Go and backed by etcd for consistency. The first project built on top of Torus is a network block device that can be mounted into containers for persistent storage. It also includes integrations out of the box for Kub…

What's the current largest-scale deployment?

It's described as a prototype, so I'm pretty sure there's no deployment at scale at all.

Re: Torus: A distributed storage system by CoreOS

#79

Post by notacoward »

bcantrill wrote: post #50

It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "…

Agree 100% on this, Bryan. Their claims about how existing storage solutions are a poor fit for this use case are completely false. As too often happens, they address only high-margin commercial systems and ignore the fact that open-source solutions are out there as well. (Disclaimer: I'm a Gluster developer). Then they leave both files and objects as "exercises for the reader" which shows a total lack of understanding for the problem or respect for people already working on it. Their announcement is clearly more about staking a claim to the "storage for containers" space, before the serious players get there, than it is about realistic expectations for how the project might grow. Particularly galling are their comments about the difficulty of building a community for something like this, when such communities already exist and their announcement actually harms those communities.

Until now, I've thought quite highly of the CoreOS team. Now, not so much. They're playing a "freeze the competition" game instead of trying to compete on merit.

Re: Torus: A distributed storage system by CoreOS

#80

Post by bcantrill »

halayli wrote: post #75
bcantrill wrote: post #50

It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "…

Looking at dtrace, fishwork, zfs and Solaris (imo was the best OS technologically speaking) it's interesting to see how strong and innovative its engineering team was while the business was just going down. How did that happen? How can the engineering team be that productive and functional while the business vision was so lacking?

Even when commercially misguided, Sun always had terrific engineering talent -- and my farewell to the company captures some of that.[1] In terms of why did the company fail, the short answer is probably that SPARC was disrupted by x86, and by the time the company figured that out, it was too late to recover.[2]

[1] http://dtrace.org/blogs/bmc/2010/07/25/good-bye-sun/

[2] Longer answer: https://news.ycombinator.com/item?id=2287033

Post reply on HN