Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Torus: A distributed storage system by CoreOS

coreos.com

Post reply on HN
Print view View original

41–50 of 187 posts

Re: Torus: A distributed storage system by CoreOS

#41

Post by dap »

vidarh wrote: post #36
ams6110 wrote: post #34

Earlier quoted context omitted.

It's like any time we take a step forward in one area we have to reinvent the last 50 years of computing to support it. Persistent storage is a "hard problem" Really?

Persistent storage isn't a hard problem. Distributed, well performing, scalable and consistent storage is a hard problem.

Actually, persistent storage is fairly hard in itself. Look at what ZFS does to ensure data integrity in the face of phantom writes, dropped writes, bad controllers, and other implicit, non-fatal failures.

Re: Torus: A distributed storage system by CoreOS

#42

Post by vidarh »

ideal0227 wrote: post #39
vidarh wrote: post #35

Earlier quoted context omitted.

Rkt had good reasoning behind it, and I think it's reasonable to assume that a lot of the recent changes to Docker came because they forced Docker's hand. For Etcd I'm of two minds. I like its simplicity. On the other hand, I have had far more problems with Etcd than I've ever had with e.g. Consul (though I dislike Consuls "and the kitchen sink" approach), up to and including data loss and more than once having to re…

Have you ever thought the etcd issues was just because how you operate it? Or maybe you were running an old version of it? Have you reported to upstream? If there is a very common issue that you can meet frequently, it should have been fixed or you should have reported it.

I'm quite sure the etcd issues was "just because" of how we operated it. But the point is that I have had these issues with Etcd, using the default CoreOS configuration of Etcd, and I've not had these issues with Gluster despite having Gluster volumes running many times as long.

Gluster had many issues too when it was a young project, but they were fixed many years ago. I don't doubt that Etcd too will become solid enough for me to trust, but looking at part of the docs [1] we still have "gems" such as "Permanent Loss of Quorum Requires New Cluster" (you don't lose your data to that one, but it's painful nevertheless).

And yes, I've run into that one. Not "permanent" as in "we could never recover the machines in question", but "permanent" as in "we're not going to be able to get this sorted fast enough, so let's fail everything over". Everything else we run can handle that scenario easily. We know it means consistency issues, but if you run into that situation we have taken the decision that they are acceptable and will be resolved later if necessary. The last thing I need in a scenario like that is for some critical component to just refuse to run without requiring lots of manual intervention.

All we'd like would be to be able to say "ok, drop all the other members and pretend all is well, and yes we know what we're asking". There are a number of gotcha's like that with Etcd that sounds find until they bite you in the ass.

They'll resolve these things eventually, but storage is the very last place I'd be willing to take risks with it. So until they have years of track record of running flawlessly, with less hassle than Gluster or Ceph, and substantial advantages, I'll see it as the dangerous choice.

[1] https://coreos.com/etcd/docs/latest/runtime-reconf-design.ht...

Re: Torus: A distributed storage system by CoreOS

#43

Post by vidarh »

dap wrote: post #41
vidarh wrote: post #36

Earlier quoted context omitted.

Persistent storage isn't a hard problem. Distributed, well performing, scalable and consistent storage is a hard problem.

Actually, persistent storage is fairly hard in itself. Look at what ZFS does to ensure data integrity in the face of phantom writes, dropped writes, bad controllers, and other implicit, non-fatal failures.

ZFS still also have the issue of having to perform well. You have a point, but ZFS is still trivial compared to a proper distributed filesystem, and you could achieve the same reliability much easier than ZFS if you sacrificed the performance.

Re: Torus: A distributed storage system by CoreOS

#44

Post by philips »

dap wrote: post #37

> At its core, Torus is a library with an interface that appears as a traditional file, allowing for storage manipulation through well-understood basic file operations. Coordinated and checkpointed through etcd’s consensus process, this distributed file can be exposed to user applications in multiple ways. Today, Torus supports exposing this file as block-oriented storage via a Network Block Device (NBD). We also exp…

It isn't a POSIX filesystem API. It is like a single file or a big distributed tape.

Re: Torus: A distributed storage system by CoreOS

#45

Post by dap »

philips wrote: post #44
dap wrote: post #37

> At its core, Torus is a library with an interface that appears as a traditional file, allowing for storage manipulation through well-understood basic file operations. Coordinated and checkpointed through etcd’s consensus process, this distributed file can be exposed to user applications in multiple ways. Today, Torus supports exposing this file as block-oriented storage via a Network Block Device (NBD). We also exp…

It isn't a POSIX filesystem API. It is like a single file or a big distributed tape.

Virtualized, network block devices have all the same problems I described -- even worse, because the abstraction coneys even less about what an application is trying to do.

Re: Torus: A distributed storage system by CoreOS

#46

Post by paulsutter »

philips wrote: post #2

The CoreOS team is excited to make this initial release and start collaborating with folks that want to tackle distributed storage. tl;dr this is a new OSS distributed storage project that is written in Go and backed by etcd for consistency. The first project built on top of Torus is a network block device that can be mounted into containers for persistent storage. It also includes integrations out of the box for Kub…

What's the current largest-scale deployment?

Re: Torus: A distributed storage system by CoreOS

#47

Post by dap »

vidarh wrote: post #43
dap wrote: post #41

Earlier quoted context omitted.

Actually, persistent storage is fairly hard in itself. Look at what ZFS does to ensure data integrity in the face of phantom writes, dropped writes, bad controllers, and other implicit, non-fatal failures.

ZFS still also have the issue of having to perform well. You have a point, but ZFS is still trivial compared to a proper distributed filesystem, and you could achieve the same reliability much easier than ZFS if you sacrificed the performance.

Absolutely -- I didn't mean to imply that ZFS even comes close to solving the distributed parts of the problem, but rather that a distributed storage system does have to address the problems of putting bits on disk.

Re: Torus: A distributed storage system by CoreOS

#48

Post by fintler »

vidarh wrote: post #29
philips wrote: post #10

Earlier quoted context omitted.

A few major things we wanted to accomplish: Backed by etcd. Today etcd is a well tested and widely used consistent store with users like Kubernetes, Flannel (now Canal), Fleet, SkyDNS and many others. Building something like etcd requires tons of testing and etcd is becoming the solid go to for this category of distributed problems. Easy to work on code base. Building an OSS community around a complex technology is r…

I've managed to lose data with Etcd, and have regularly had issues with membership issues requiring maintenance. Meanwhile I've had Glusterfs volumes remain available for 5-6 years without maintenance at all. To me at least, having it "backed by etcd" is a big red flag, not a feature.

  I've managed to lose data with Etcd
How did you manage to lose data?

Re: Torus: A distributed storage system by CoreOS

#49

Post by ibotty »

dap wrote: post #45
philips wrote: post #44

Earlier quoted context omitted.

It isn't a POSIX filesystem API. It is like a single file or a big distributed tape.

Virtualized, network block devices have all the same problems I described -- even worse, because the abstraction coneys even less about what an application is trying to do.

No. The difference is, that with network block devices you are usually only allow accessing the block device once. That's an easier problem than mapping POSIX file system semantics!

Re: Torus: A distributed storage system by CoreOS

#50

Post by bcantrill »

It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "the design includes the ability to support [...] Reed-Solomon error correction in the near future" -- and the fact that such an empty system hails itself as pioneering an unsolved problem in storage is galling in its ignorance of prior work (much of it open source).

Take it from someone who has been involved in both highly durable local filesystems[1] and highly available object storage systems[2][3]: this is such a hard, nasty problem with so many dark, hidden and dire failure modes, that it takes years of running in production to get these systems to the level of reliability and operability that the data path demands. Given that (according to the repo, though not the breathless blog entry) its creators "do not recommend its use in production", Torus is -- in the famous words of Wolfgang Pauli -- not even wrong.

[1] http://dtrace.org/blogs/bmc/2008/11/10/fishworks-now-it-can-...

[2] http://dtrace.org/blogs/bmc/2013/06/25/manta-from-revelation...

[3] http://dtrace.org/blogs/dap/2013/07/03/fault-tolerance-in-ma...

Post reply on HN