Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Torus: A distributed storage system by CoreOS

coreos.com

Post reply on HN
Print view View original

181–187 of 187 posts

Re: Torus: A distributed storage system by CoreOS

#181

Post by nwmcsween »

betawaffle wrote: post #144
nwmcsween wrote: post #56

So a userspace fs, I'm guessing this will use fuse to actually expose a POSIX fs? Small sync writes will absolutely kill performance and will require all sorts of hacks like glusterfs has had to implement due to the amount of context switches. CoreOS really should have added whatever needed to ceph, filesystems are not something you just hack together overnight.

We had a POSIX interface early on (via FUSE), but decided to expose a a block storage interface first instead. This is not a "filesystem", it's a storage abstraction, and we've spent more than 6 months on it. Seems like a lot of folks are quickly jumping to conclusions, which is expected from "the internet", but I would have hoped for better from HN.

6 months is overnight in terms of a filesystem, there are literally millions of dollars of work that was done with glusterfs and probably a magnitude less man hours. I understand coreos wants.to innovate but what does torus bring to the table besides negatives and NIH?

Re: Torus: A distributed storage system by CoreOS

#183

Post by frozenice »

Funny, just this week I was looking for an easy solution to cluster storage for containers.

I found SXCluster and posted it here https://news.ycombinator.com/item?id=11812566 - not directly geared towards containers, but very easy to setup (no need for etcd and the like).

Re: Torus: A distributed storage system by CoreOS

#184

Post by betawaffle »

gregsfortytwo wrote: post #178
zzzcpan wrote: post #177

Earlier quoted context omitted.

They accept data loss, yes, and haven't figured out consistency yet. But given that it's an early prototype I would expect them to evolve and change things a lot in a couple of years, possibly throwing away raft and etcd. Good work on Ceph, by the way. I've been following your work since it was a PhD if I remember correctly.

"haven't figured out consistency yet" is the issue. Seemingly very small decisions have huge impacts that you don't expect, and the state of the art is far enough along that you don't get better than the existing solutions except by exploiting newly-identified workload characteristics (or acceptable ways of losing data, loosening consistency, etc based on the workload) that you've planned out to make use of ahead of…

These are very good points, but also probably much more constructive as GitHub issues, where they can be either answered or addressed. In the meantime, hopefully I can talk to a few of these:

>haven't figured out consistency yet I don't recall this being the case, but I'm not the authority on the matter.

>I presume Torus is targeting (high IO efficiency, low latency) The best-possible performing storage solution is _definitely_ not our primary goal, though we'll take it where we can get it. The most important goals for the project are ease of use, ease of management, flexibility, and correctness when the use-case desires it. Please note that the block device interface is only one of many planned. The underlying abstraction was designed (and will be improved) to support other situations.

>Using etcd for allocation would be a reasonable choice, but putting it in the persistence path is now. Right now a database in your container would require two separate write streams to do an fsync In Torus today (with the block device interface specifically), and with the caveat that I'm not the authority, so I may be slightly wrong, calling sync(), fsync(), and friends result in what I think you would consider an "allocation". Writes happen against a snapshot of the file (in this block storage case, the block volume is the "file"), and then a sync() makes those changes visible as the "current" version. Syncs hit etcd, writes do not.

I would really encourage you to submit feedback like this in GitHub issues. The project is still in _very_ early stages, and legitimate feedback can actually make a difference.

Re: Torus: A distributed storage system by CoreOS

#185

Post by jbooth »

betawaffle wrote: post #173
jbooth wrote: post #167

Earlier quoted context omitted.

Ok, so that plus a little MVCC can make you consistent, but you've still got the read-many-to-write-one thing from the perspective of your block device interface, right? And block devices, if I'm remembering right, don't leave you any room to buffer pending writes.

Torus implements a kind of MVCC, yes. As for read-many-to-write-one, I assume you're talking about Reed-Solomon or similar erasure coding? There have been some papers written about ways to reduce that, a good one is from Facebook: https://code.facebook.com/posts/536638663113101/saving-capac... . And that's just one option. Also, this is all speculative since we have yet to implement erasure coding.

Don't know if you'll see this, but:

If only one host at a time has access to a given virtual block device, there are some opportunities to buffer outgoing writes with a write-through cache. That might be the way to go if you explore erasure coding in the future.

Re: Torus: A distributed storage system by CoreOS

#187

Post by electrum »

stevelandis wrote: post #116
stevelandiss wrote: post #114

Earlier quoted context omitted.

If I could also add that NBD is extremely notorious in Linux. I am a block storage developer and NBD has been one of the main reasons why so many openstack storage products (like formation data and others) have really struggled. There are many known Linux kernel issues with NBD, for example, if an NBD provider (user space daemon) exits for any reason, the kernel panics. Here is an example of a long outstanding bug th…

Here is an example script that will panic anyone using NBD to server a block device (replace qemu-nbd with the nbd export provided by torus) qemu-img create -f qcow2 f.img 1G mkfs.ext4 f.img modprobe nbd || true qemu-nbd -c /dev/nbd0 f.img mount /dev/nbd0 k killall -KILL qemu-nbd sleep 1 ls k

Above script with correct formatting:

  qemu-img create -f qcow2 f.img 1G
  mkfs.ext4 f.img
  modprobe nbd || true
  qemu-nbd -c /dev/nbd0 f.img
  mount /dev/nbd0 k
  killall -KILL qemu-nbd
  sleep 1
  ls k
Post reply on HN