Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Torus: A distributed storage system by CoreOS

coreos.com

Post reply on HN
Print view View original

151–160 of 187 posts

Re: Torus: A distributed storage system by CoreOS

#151

Post by notacoward »

betawaffle wrote: post #147
notacoward wrote: post #142

Earlier quoted context omitted.

I don't think the two cases are analogous. First, block devices don't have any metadata to speak of. Second, filesystems have journals to "cover up" for the main-store operations being asynchronous and/or non-atomic. However, sync() is completely synchronous by definition - hence the name - so such "covering up" would be superfluous. There must be some metadata that's being written at etcd because it needs to be read…

Block devices (and NBD specifically) absolutely have a notion of sync(). We use sync() as the unit of write visibility. All writes up until a sync are effectively anonymous until a sync().

Please go look at the man page for sync(2), which is also the page for syncfs(2). It is explicitly a filesystem-level operation. Obviously, this will cause data to be flushed from the filesystem down to lower layers. Obviously, you can "sync" virtual (e.g. NBD or loopback) block devices by syncing the filesystems that contain their backing stores, but that's not the same thing. No filesystem, no sync(2). For block devices with no file-based backing store, sync(2) is inapplicable. Also, sync(2) is latency-inducing overkill if you're trying to ensure durability for anything less than all filesystems attached to a machine. More often, fsync(2) on the backing files is what you should be using.

> All writes up until a sync are effectively anonymous until a sync().

"Anonymous" means nothing in this context. Do you mean non-durable? First, please use the correct term. Second, if that is what you mean then you're probably doing it wrong. File writes are allowed to be asynchronous (unless O_SYNC and friends). Block device writes are expected to be synchronous, or at least to preserve order. This is exactly the kind of thing that needs to be thoroughly thought out before code is even written, and that thinking should be spelled out somewhere for people to help make sure all the nasty corner cases are covered. Your cart is way before your horse.

Re: Torus: A distributed storage system by CoreOS

#152

Post by anarazel »

nickpsecurity wrote: post #149
nl wrote: post #136

Earlier quoted context omitted.

Anyone who's worked with Postgres at scale would guess autovacuum. Postgres doesn't have many weaknesses, but most of them relate to autovacuum.

"Anyone who's worked with Postgres at scale would guess autovacuum." Well, there's knowing it's autovacuum-related then there's the specific way it's causing a failure. First part was obvious. The rest took work. "Postgres doesn't have many weaknesses, but most of them relate to autovacuum." Sounds like that statement should be on a bug submission or something. They probably need to replace that with something better…

It's known to the postgres developers, and we are working on it. This specific issue (anti-wraparound vacuums being a lot more expensive) should be fixed in the upcoming 9.6.

Re: Torus: A distributed storage system by CoreOS

#153

Post by vox_mollis »

zxcvcxz wrote: post #76

It's threads like these where you start to wonder just how many marketing teams are arguing with each other in the comments. I take the FUD with a grain of salt: there is a lot of financial incentive to create FUD, where there is financial incentive and little to no regulations a market will naturally arise. It's going to get worse as the bubble pops and companies become more and more desperate.

Indeed. The upvoting of bcantrill seems to be devoid of understanding that he has been waging a personal vendetta against CoreOS for some time now, and his comments should be understood in that context.

Re: Torus: A distributed storage system by CoreOS

#154

Post by zzzcpan »

betawaffle wrote: post #144
nwmcsween wrote: post #56

So a userspace fs, I'm guessing this will use fuse to actually expose a POSIX fs? Small sync writes will absolutely kill performance and will require all sorts of hacks like glusterfs has had to implement due to the amount of context switches. CoreOS really should have added whatever needed to ceph, filesystems are not something you just hack together overnight.

We had a POSIX interface early on (via FUSE), but decided to expose a a block storage interface first instead. This is not a "filesystem", it's a storage abstraction, and we've spent more than 6 months on it. Seems like a lot of folks are quickly jumping to conclusions, which is expected from "the internet", but I would have hoped for better from HN.

> but I would have hoped for better from HN

Don't worry about it. Some guys just mistake their experience with traditional storage systems as meaningful here.

Re: Torus: A distributed storage system by CoreOS

#155

Post by nickpsecurity »

anarazel wrote: post #152
nickpsecurity wrote: post #149

Earlier quoted context omitted.

"Anyone who's worked with Postgres at scale would guess autovacuum." Well, there's knowing it's autovacuum-related then there's the specific way it's causing a failure. First part was obvious. The rest took work. "Postgres doesn't have many weaknesses, but most of them relate to autovacuum." Sounds like that statement should be on a bug submission or something. They probably need to replace that with something better…

It's known to the postgres developers, and we are working on it. This specific issue (anti-wraparound vacuums being a lot more expensive) should be fixed in the upcoming 9.6.

Awesome! I already push Postgres and praise its team for the quality focus. Just extra evidence in your favor. :)

Re: Torus: A distributed storage system by CoreOS

#156

Post by notacoward »

merb wrote: post #146
notacoward wrote: post #88

Earlier quoted context omitted.

> They're not writing blog posts or comments on HN, they're writing code. Actually, the problem is that 90% of the code still remains to be written, while others (including me) have already done so . They've addressed only the very simplest part of the problem, not even far enough to show any performance comparisons, in a manner strongly reminiscent of Sheepdog (belying your "own approach" claim). That's a poor basis…

Actually running just: torusctl init torusctl -C $ETCD_IP:2379 init ./torusd --etcd 127.0.0.1:2379 --peer-address http://127.0.0.1:40000 --data-dir /tmp/torus1 --size 20GiB to have a near production ready system is way easier than setting up glusterfs (even as a demo) and ceph. A distributed system doesn't need to be complicated.

Actually that's almost exactly the same steps as for GlusterFS.

  > gluster peer probe ...
  > gluster volume create ...
  > gluster volume start ...
But that's not even the point. You're right that the interface to a distributed storage system doesn't need to be complicated, but the implementation inevitably must be to handle the myriad error conditions that will be thrown at it. Correctness is even more important for storage than for other areas in computing, and something that only implements the "happy path" for the simplest data model or semantics is barely even a beginning. The distance between "seems to work" and "can be counted on to work" is far greater for this type of system than for most others. I think it's important to understand and communicate that, so that people don't develop unrealistic expectations. That way lies nothing but heartbreak, not least for the developers themselves. It's far better for everyone to set and meet modest goals than to make extravagant promises that can't be kept.

Re: Torus: A distributed storage system by CoreOS

#157

Post by js4all »

elsonrodriguez wrote: post #115
js4all wrote: post #87

Despite all the negative sentiment here, I am super excited about this. I use CoreOS heavily and really like how everything just works. Running Kubernetes on it, is the first cluster solution for me that works without configuration orgies and is robust against machine outages. Torus seems to be the missing piece. For now we use local volumes with sidecar containers for r/o storage and nfs volumes for r/w storage. All…

There's lots of work being done in Ceph-Docker to make ceph easier to configure. On top of that, there's work going on to get Ceph easily deployable on Kubernetes. https://github.com/cornelius-keller/ceph-docker/tree/kuberne...

Many thanks, that looks doable. I'll give it a try.

Re: Torus: A distributed storage system by CoreOS

#158

Post by jbooth »

zzzcpan wrote: post #150
nickpsecurity wrote: post #113

Earlier quoted context omitted.

My impression was that they heard reed solomon was used in robust systems like they are describing. They intend to use it in theirs. It will therefore be just as robust. Similar to how some firms describe their security after adding "256-bit, military-grade AES." ;)

They operate on blocks and can implement Reed-Solomon with no issues. Random writes do not matter with the architecture like this. The tricky part would be latency and performance during periods of growth.

Yeah, but they have to read several data/parity blocks, and then rewrite all parity blocks plus one data block, for any write to a given block.

This creates big difficulties for both consistency and performance, and fixes for consistency make the performance worse (and vice versa).

Google's filesystem could use reed-solomon because they're append-only, making consistency a non-issue and performance can be fixed by buffering on the client side.

Re: Torus: A distributed storage system by CoreOS

#159

Post by betawaffle »

jbooth wrote: post #158
zzzcpan wrote: post #150

Earlier quoted context omitted.

They operate on blocks and can implement Reed-Solomon with no issues. Random writes do not matter with the architecture like this. The tricky part would be latency and performance during periods of growth.

Yeah, but they have to read several data/parity blocks, and then rewrite all parity blocks plus one data block, for any write to a given block. This creates big difficulties for both consistency and performance, and fixes for consistency make the performance worse (and vice versa). Google's filesystem could use reed-solomon because they're append-only, making consistency a non-issue and performance can be fixed by bu…

Torus is append-only too. We also plan to support something more like what Facebook's paper describes, where they have extra parity (xor) to support more efficient local repair.

Re: Torus: A distributed storage system by CoreOS

#160

Post by jsmthrowaway »

curtisptrsn wrote: post #145
bcantrill wrote: post #50

It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "…

Just how many things is CoreOS trying to do? Last I counted, they want to a) Build a distributed OS b) Build a distributed scheduler (Fleet) c) Build a distributed key value system (etcd) d) Build a new container engine (Rocket) e) Build a network fabric (Flannel) f) Now embark on building a brand new distributed storage system. Holy cow that's some goal list. Sounds like something my kids would make up for their Chr…

Fleet died once Kubernetes became a thing, even though they serve entirely different use cases.
Post reply on HN