Torus: A distributed storage system by CoreOS • Hacker News Board

Re: Torus: A distributed storage system by CoreOS

#141

Post by wazoox » Thu, Jun 02, 2016, 11:39 AM UTC

drauh wrote: post #93

nickpsecurity wrote: post #73

Earlier quoted context omitted.

" Supercomputer storage is fault tolerant only by fortuitous accident and often requires week-long outages with destroyed data for software upgrades." Have you tried Ceph or Sector/Sphere? Lustre is known to be crap while Ceph gets a lot of praise and Sector/Sphere has potential for reliability with good performance. I think you may just be stuck with tools that suck. I'll admit it was about 10+ years ago when I was…

Re CEPH vs Lustre: what's performance like? I've seen anecdotes quoting 3 GBps over Infiniband for Lustre. (I'm curious because I run an HPC installation with Lustre and NFS over XFS, and trying to think of the future. MBTF doesn't matter as much as raw speed while it actually runs.)

In the HPC crowd, I'm quite familiar with OrangeFS (aka PVFS2) which recently entered the standard kernel. I had a PVFS 2.7 cluster running for many years, 24/7 with decent reliability (it crashed a few times, but never lost data).

It works with RDMA, has a POSIX layer, and is roughly equivalent to Lustre in performance in my tests, but 1° is very easy to setup (compared to Lustre) 2° has NFS actually working.

Re: Torus: A distributed storage system by CoreOS

#142

Post by notacoward » Thu, Jun 02, 2016, 12:00 PM UTC

grogers wrote: post #126

notacoward wrote: post #98

Earlier quoted context omitted.

Why is it in the sync path? That seems unnecessary, as well as bad for both performance and correctness (another thing in any path means another potential for partial failure).

This is not a great analogy since their first application is block storage, but think about a journaling file system. Typically file data is not journaled, but metadata is. By having a consistent view of the metadata, the entire filesystem (as far as you can interact with it) is consistent. That consistent journal is the same primitive that etcd is providing.

I don't think the two cases are analogous. First, block devices don't have any metadata to speak of. Second, filesystems have journals to "cover up" for the main-store operations being asynchronous and/or non-atomic. However, sync() is completely synchronous by definition - hence the name - so such "covering up" would be superfluous. There must be some metadata that's being written at etcd because it needs to be read from there later, but there's nothing in block-device semantics to require any such thing.

Thinking about it further, I think I can guess at what's going on here. The key observation is that there's no sync() at the block device level. It's a filesystem operation; block devices don't see it. Sure, there are queue flags and FUA and such, but those are different (and I'm not sure any of those exist in NBD). Where is this sync() path? I'm guessing it's internal on the data servers, to deal with data that's being buffered there. With both replication and erasure coding, correct recovery requires exact knowledge of what has been fully written where, and that's the kind of metadata I suspect is being put in etcd. There's not even necessarily anything wrong with it, unless updating that information only on sync() means that supposedly durable writes since the last (unpredictable to the client) sync could be lost on failure. I hope that's not the case.

Maybe I'll find time, in the midst of my work on an actual production-level distributed filesystem, to look at the code and see if my guess is correct.

Re: Torus: A distributed storage system by CoreOS

#143

Post by tveita » Thu, Jun 02, 2016, 12:02 PM UTC

sysexit wrote: post #30

I love CoreOS and they've done some super impressive engineering. But really, a new storage system? Rewriting in Go and using etcd for central state management makes things easier, but this is still a hard problem. Some things that need to be solved sooner or later: data replication so that N faults of X entities are protected against (X can be disks, enclosures, racks, data centers, regions, ..), recovery from faile…

I've been trying to find performance evaluations of distributed file systems, and in most tests I've seen Ceph is a lot slower than alternatives like GlusterFS.

When you say Ceph is "state of the art in software defined storage", are you including performance, or are there advantages in features or reliability where you think it outclasses the competition?

Re: Torus: A distributed storage system by CoreOS

#144

Post by betawaffle » Thu, Jun 02, 2016, 12:42 PM UTC

nwmcsween wrote: post #56

So a userspace fs, I'm guessing this will use fuse to actually expose a POSIX fs? Small sync writes will absolutely kill performance and will require all sorts of hacks like glusterfs has had to implement due to the amount of context switches. CoreOS really should have added whatever needed to ceph, filesystems are not something you just hack together overnight.

We had a POSIX interface early on (via FUSE), but decided to expose a a block storage interface first instead. This is not a "filesystem", it's a storage abstraction, and we've spent more than 6 months on it. Seems like a lot of folks are quickly jumping to conclusions, which is expected from "the internet", but I would have hoped for better from HN.

Re: Torus: A distributed storage system by CoreOS

#145

Post by curtisptrsn » Thu, Jun 02, 2016, 1:45 PM UTC

bcantrill wrote: post #50

It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "…

Just how many things is CoreOS trying to do? Last I counted, they want to

a) Build a distributed OS

b) Build a distributed scheduler (Fleet)

c) Build a distributed key value system (etcd)

d) Build a new container engine (Rocket)

e) Build a network fabric (Flannel)

f) Now embark on building a brand new distributed storage system.

Holy cow that's some goal list. Sounds like something my kids would make up for their Christmas wish list.

I really don't get how their board and investors let them get away with such a childish imagination.

Each one of those is a company effort on its own.

Re: Torus: A distributed storage system by CoreOS

#146

Post by merb » Thu, Jun 02, 2016, 1:57 PM UTC

notacoward wrote: post #88

mevile wrote: post #84

Earlier quoted context omitted.

How is them developing their own technology not competing on merit? Did they steal the technology? Did they claim that anyone ought to use this in production? Why all the negativity? Yeah never mind let's just discourage everyone from trying to build new technology. Nobody is forcing you to use this. If it's not for you, move on. Going out on a rant about what you think their intentions are is ridiculous. Really you…

> They're not writing blog posts or comments on HN, they're writing code. Actually, the problem is that 90% of the code still remains to be written, while others (including me) have already done so . They've addressed only the very simplest part of the problem, not even far enough to show any performance comparisons, in a manner strongly reminiscent of Sheepdog (belying your "own approach" claim). That's a poor basis…

Actually running just:

    torusctl init
    torusctl -C $ETCD_IP:2379 init
   ./torusd --etcd 127.0.0.1:2379 --peer-address http://127.0.0.1:40000 --data-dir /tmp/torus1 --size 20GiB

to have a near production ready system is way easier than setting up glusterfs (even as a demo) and ceph. A distributed system doesn't need to be complicated.

Re: Torus: A distributed storage system by CoreOS

#147

Post by betawaffle » Thu, Jun 02, 2016, 2:00 PM UTC

notacoward wrote: post #142

grogers wrote: post #126

Earlier quoted context omitted.

This is not a great analogy since their first application is block storage, but think about a journaling file system. Typically file data is not journaled, but metadata is. By having a consistent view of the metadata, the entire filesystem (as far as you can interact with it) is consistent. That consistent journal is the same primitive that etcd is providing.

I don't think the two cases are analogous. First, block devices don't have any metadata to speak of. Second, filesystems have journals to "cover up" for the main-store operations being asynchronous and/or non-atomic. However, sync() is completely synchronous by definition - hence the name - so such "covering up" would be superfluous. There must be some metadata that's being written at etcd because it needs to be read…

Block devices (and NBD specifically) absolutely have a notion of sync(). We use sync() as the unit of write visibility. All writes up until a sync are effectively anonymous until a sync().

Re: Torus: A distributed storage system by CoreOS

#148

Post by nickpsecurity » Thu, Jun 02, 2016, 2:24 PM UTC

pinewurst wrote: post #121

ymse wrote: post #120

Earlier quoted context omitted.

A proprietary (and solid) alternative to Lustre would be GPFS, which also has a long track record in HPC (and other markets in which IBM thrives). As someone who completely shares your Lustre sentiment, I can't fathom why Intel keeps pouring resources into it.

GPFS has an amazing number of features, offers high performance and, given a certain fiddliness of configuration and administration, is reliable and performant. It can even sit on top of block storage that itself manages with advanced software RAID and volume management. The problem (surprise!) is IBM. It's mature software, which means 21st Century Desperate IBM sees it as a cash cow - aggressively squeezing customer…

Yeah, yeah, GPFS was one of them that inspired my HPC and cloud comparison. It, combined with management software, got one to about 80-90% of what they needed for cloud filesystems. It was badass back when I read about it being deployed in ASC Purple. I didn't know it turned into some stagnating, fascist crap with IBM. Sad outcome for such great technology.

Typical IBM, though. (shakes head)

Re: Torus: A distributed storage system by CoreOS

#149

Post by nickpsecurity » Thu, Jun 02, 2016, 2:27 PM UTC

nl wrote: post #136

nickpsecurity wrote: post #92

Earlier quoted context omitted.

That was a pretty humble and good read. I don't think I'd have seen the autovacuuming issue coming. Actually, this quote is a perfect example of how subtle and ridiculous these issues can be: "During the event, one of the shard databases had all queries on our primary table blocked by a three-way interaction between the data path queries that wanted shared locks, a "transaction wraparound" autovacuum that held a shar…

Anyone who's worked with Postgres at scale would guess autovacuum. Postgres doesn't have many weaknesses, but most of them relate to autovacuum.

"Anyone who's worked with Postgres at scale would guess autovacuum."

Well, there's knowing it's autovacuum-related then there's the specific way it's causing a failure. First part was obvious. The rest took work.

"Postgres doesn't have many weaknesses, but most of them relate to autovacuum."

Sounds like that statement should be on a bug submission or something. They probably need to replace that with something better.

Re: Torus: A distributed storage system by CoreOS

#150

Post by zzzcpan » Thu, Jun 02, 2016, 2:38 PM UTC

nickpsecurity wrote: post #113

jbooth wrote: post #90

Earlier quoted context omitted.

What triggered me was just throwing out "reed solomon" when talking about random writes. How does that work? We'll read from 5 places to complete your write?

My impression was that they heard reed solomon was used in robust systems like they are describing. They intend to use it in theirs. It will therefore be just as robust. Similar to how some firms describe their security after adding "256-bit, military-grade AES." ;)

They operate on blocks and can implement Reed-Solomon with no issues. Random writes do not matter with the architecture like this. The tricky part would be latency and performance during periods of growth.