Torus: A distributed storage system by CoreOS • Hacker News Board

Re: Torus: A distributed storage system by CoreOS

#121

Post by pinewurst » Thu, Jun 02, 2016, 3:01 AM UTC

ymse wrote: post #120

pinewurst wrote: post #104

Earlier quoted context omitted.

At this point, this is really an apples-to-oranges comparison. Lustre, as truly awful as it is, is a POSIX filesystem (or close enough for (literally) government work). Redhat/Ceph only at the end of April, announced that POSIX functionality was ready for production. Personally, that's not when I'd choose to deploy production storage. Ceph object and nominally block have much more time in production. If you need POSI…

A proprietary (and solid) alternative to Lustre would be GPFS, which also has a long track record in HPC (and other markets in which IBM thrives). As someone who completely shares your Lustre sentiment, I can't fathom why Intel keeps pouring resources into it.

GPFS has an amazing number of features, offers high performance and, given a certain fiddliness of configuration and administration, is reliable and performant. It can even sit on top of block storage that itself manages with advanced software RAID and volume management.

The problem (surprise!) is IBM. It's mature software, which means 21st Century Desperate IBM sees it as a cash cow - aggressively squeezing customers - and as something they can let their senior, expensive developers move on from - or lay them off in favor of "rightsourcing". You can certainly trust your data to it (unlike Lustre), but it'll be very expensive, especially on an ongoing basis, and the support team isn't going to know more than you by then. Also expect surprise visits from IBM licensing ninja squads looking for violations of the complex terms, which they will find.

As for Lustre, it brings to mind Oliver Wendell Holmes, Jr's, "Three generations of imbeciles are enough". I've been at least peripherally involved with it since 1999, with LLNL trying to strong-arm storage vendors into support. Someone should write a book following 16 years of the tangled Lustre trail from LLNL/CMU/CFS -> Sun -> Oracle -> WhamCloud -> OpenSFS -> ClusterStor -> Xyratex -> Seagate -> Intel (and probably ISIS too).

The answer to your question IHMO, is that Intel just isn't that smart. They're basically a PR firm with a good fab in the basement. What do they know about storage or so many other things? People don't remember when they tried to corner the web serving market back during the 1st Internet boom. They fail a lot, but until now had enough of a cash torrent coming that it didn't matter. They still do, of course, but there are inklings of an ebb.

Re: Torus: A distributed storage system by CoreOS

#122

Post by mrmondo » Thu, Jun 02, 2016, 3:33 AM UTC

Watch out, next we'll be seeing CoreOS writing their own encryption protocols. As someone that's designed and built storage clusters from the ground up - take it from me: storage is not as easy as it seems if you care about your data and performance at scale. The number of people I've seen using CoreOS and then moving away from it is quite alarming, I feel like CoreOS will become the Ubuntu of the container world.

Re: Torus: A distributed storage system by CoreOS

#123

Post by alexandre_m » Thu, Jun 02, 2016, 3:53 AM UTC

jsmthrowaway wrote: post #102

I love that I know who this is at CoreOS by the choice of username. Pick better throwaways if you're going to blow up a thread like this. (ideal0227 upthread is also a CoreOS employee and one of the etcd developers.) On topic, yes. It is quite trivial to lose data with etcd and pretty much everybody I know who runs it has experienced problems. Try backing it up and restoring. etcd is compsci great yet operationally b…

Cockroachdb is built on top of etcd. So far seems to work very well for their product.

I think this whole thread is a bit harsh and unfair to CoreOs and their solutions.

Re: Torus: A distributed storage system by CoreOS

#124

Post by gyuho » Thu, Jun 02, 2016, 4:19 AM UTC

alexandre_m wrote: post #123

jsmthrowaway wrote: post #102

I love that I know who this is at CoreOS by the choice of username. Pick better throwaways if you're going to blow up a thread like this. (ideal0227 upthread is also a CoreOS employee and one of the etcd developers.) On topic, yes. It is quite trivial to lose data with etcd and pretty much everybody I know who runs it has experienced problems. Try backing it up and restoring. etcd is compsci great yet operationally b…

Cockroachdb is built on top of etcd. So far seems to work very well for their product. I think this whole thread is a bit harsh and unfair to CoreOs and their solutions.

Disclaimer: I work on etcd at CoreOS.

CockroachDB is built on top of etcd/raft package, not etcd itself.

Re: Torus: A distributed storage system by CoreOS

#125

Post by vidarh » Thu, Jun 02, 2016, 4:35 AM UTC

alexandre_m wrote: post #123

jsmthrowaway wrote: post #102

I love that I know who this is at CoreOS by the choice of username. Pick better throwaways if you're going to blow up a thread like this. (ideal0227 upthread is also a CoreOS employee and one of the etcd developers.) On topic, yes. It is quite trivial to lose data with etcd and pretty much everybody I know who runs it has experienced problems. Try backing it up and restoring. etcd is compsci great yet operationally b…

Cockroachdb is built on top of etcd. So far seems to work very well for their product. I think this whole thread is a bit harsh and unfair to CoreOs and their solutions.

I think that reflects the complexity of a storage product more than CoreOS itself. I use CoreOS. I like it. I love the update mechanism, and the tight focus on running as much as possible in containers.

But I also have spent enough time using it to run across certain warts, and Etcd clusters refusing to start without manual intervention etc. have been a frequent enough problem that when they go after something incredibly hard such as distributed storage, and it's relying on Etcd, I see that as a scary combination.

Re: Torus: A distributed storage system by CoreOS

#126

Post by grogers » Thu, Jun 02, 2016, 4:37 AM UTC

notacoward wrote: post #98

barakm wrote: post #16

Earlier quoted context omitted.

It's in the sync() path, but not in the I/O path.

Why is it in the sync path? That seems unnecessary, as well as bad for both performance and correctness (another thing in any path means another potential for partial failure).

This is not a great analogy since their first application is block storage, but think about a journaling file system. Typically file data is not journaled, but metadata is. By having a consistent view of the metadata, the entire filesystem (as far as you can interact with it) is consistent. That consistent journal is the same primitive that etcd is providing.

Re: Torus: A distributed storage system by CoreOS

#127

Post by stevelandis » Thu, Jun 02, 2016, 5:06 AM UTC

joshuak wrote: post #63

Good... Good... Let the hate flow through you. An open source OS company just blogged about a new open source project that they are putting resources into. They did not release a commercial product that competes with any existing storage solution. How exactly would one expect a new project to be announced? Tell me more about how far away from done you think they are. Sorry if that's a bit sarcastic, but seriously wou…

IMO it is because this can derail the adoption of containers. I think an immature project like this announced so prematurely does more harm than good. And I appreciate people that have experience with this bringing up the real issues.

Re: Torus: A distributed storage system by CoreOS

#128

Post by stevelandis » Thu, Jun 02, 2016, 5:08 AM UTC

philips wrote: post #2

The CoreOS team is excited to make this initial release and start collaborating with folks that want to tackle distributed storage. tl;dr this is a new OSS distributed storage project that is written in Go and backed by etcd for consistency. The first project built on top of Torus is a network block device that can be mounted into containers for persistent storage. It also includes integrations out of the box for Kub…

Brandon, a few months ago I met you at a meetup and you did not think this problem needed to be solved, and containers were for ephemeral apps... just an AMA request, what changed hearts?

Re: Torus: A distributed storage system by CoreOS

#129

Post by stevelandis » Thu, Jun 02, 2016, 5:11 AM UTC

fintler wrote: post #48

vidarh wrote: post #29

Earlier quoted context omitted.

I've managed to lose data with Etcd, and have regularly had issues with membership issues requiring maintenance. Meanwhile I've had Glusterfs volumes remain available for 5-6 years without maintenance at all. To me at least, having it "backed by etcd" is a big red flag, not a feature.

I've managed to lose data with Etcd How did you manage to lose data?

Use etcd in production and you will lose data. Bounce nodes, simulate power loss and you will lose data. Have you actually used it?

Re: Torus: A distributed storage system by CoreOS

#130

Post by stevelandiss » Thu, Jun 02, 2016, 5:23 AM UTC

philips wrote: post #66

koolba wrote: post #14

So is this the building blocks for an in-house EBS?

Yes, that is the idea. Torus block is like EBS. Torus "library" could be used by other applications like an append only log, object store, etc.

How do you say that with conviction? I looked at the code and it has no bearings of being able to build a fault tolerant elastic block storage system.