Torus: A distributed storage system by CoreOS • Hacker News Board

Re: Torus: A distributed storage system by CoreOS

#111

Post by nickpsecurity » Thu, Jun 02, 2016, 1:22 AM UTC

notacoward wrote: post #107

nickpsecurity wrote: post #73

Earlier quoted context omitted.

" Supercomputer storage is fault tolerant only by fortuitous accident and often requires week-long outages with destroyed data for software upgrades." Have you tried Ceph or Sector/Sphere? Lustre is known to be crap while Ceph gets a lot of praise and Sector/Sphere has potential for reliability with good performance. I think you may just be stuck with tools that suck. I'll admit it was about 10+ years ago when I was…

What does "potential for reliability" even mean? Even Lustre, which you malign and I have maligned even more, has potential for reliability if they just fix a few hundred egregious design flaws.

I haven't had a chance to run it. I also don't have clear data on it's userbase. I just know it's been used in supercomputing centers for a while doing large jobs fast over fast WAN's. So, potentially reliable if I see more users telling me it is in various situations.

Re: Torus: A distributed storage system by CoreOS

#112

Post by drauh » Thu, Jun 02, 2016, 1:27 AM UTC

pinewurst wrote: post #104

drauh wrote: post #93

Earlier quoted context omitted.

Re CEPH vs Lustre: what's performance like? I've seen anecdotes quoting 3 GBps over Infiniband for Lustre. (I'm curious because I run an HPC installation with Lustre and NFS over XFS, and trying to think of the future. MBTF doesn't matter as much as raw speed while it actually runs.)

At this point, this is really an apples-to-oranges comparison. Lustre, as truly awful as it is, is a POSIX filesystem (or close enough for (literally) government work). Redhat/Ceph only at the end of April, announced that POSIX functionality was ready for production. Personally, that's not when I'd choose to deploy production storage. Ceph object and nominally block have much more time in production. If you need POSI…

Cheers!

Re: Torus: A distributed storage system by CoreOS

#113

Post by nickpsecurity » Thu, Jun 02, 2016, 1:27 AM UTC

jbooth wrote: post #90

bcantrill wrote: post #67

Earlier quoted context omitted.

Yes, that's absolutely been my experience -- and even then, when it comes to the data path, you will likely find new failure modes in "tried and true" as you push it harder and longer and with the bar being set at absolute perfection. I have learned this painful lesson twice: first, with Fishworks at Sun when we turned ZFS into a storage appliance -- and we learned the painful difference between something that seems…

What triggered me was just throwing out "reed solomon" when talking about random writes. How does that work? We'll read from 5 places to complete your write?

My impression was that they heard reed solomon was used in robust systems like they are describing. They intend to use it in theirs. It will therefore be just as robust. Similar to how some firms describe their security after adding "256-bit, military-grade AES." ;)

Re: Torus: A distributed storage system by CoreOS

#114

Post by stevelandiss » Thu, Jun 02, 2016, 1:32 AM UTC

bcantrill wrote: post #50

It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "…

If I could also add that NBD is extremely notorious in Linux. I am a block storage developer and NBD has been one of the main reasons why so many openstack storage products (like formation data and others) have really struggled. There are many known Linux kernel issues with NBD, for example, if an NBD provider (user space daemon) exits for any reason, the kernel panics. Here is an example of a long outstanding bug that has plagued the NBD community and openstack community for years (https://www.mail-archive.com/nbd-general@lists.sourceforge.n...). It won't get addressed any time soon. I also looked at the repo and it looks like a very simplistic approach to a hard problem. BTW I have used etcd and if this is anything like etcd, I'd really be worried. etcd snapshots bring an entire cluster down for a while.

Re: Torus: A distributed storage system by CoreOS

#115

Post by elsonrodriguez » Thu, Jun 02, 2016, 1:33 AM UTC

js4all wrote: post #87

Despite all the negative sentiment here, I am super excited about this. I use CoreOS heavily and really like how everything just works. Running Kubernetes on it, is the first cluster solution for me that works without configuration orgies and is robust against machine outages. Torus seems to be the missing piece. For now we use local volumes with sidecar containers for r/o storage and nfs volumes for r/w storage. All…

There's lots of work being done in Ceph-Docker to make ceph easier to configure.

On top of that, there's work going on to get Ceph easily deployable on Kubernetes.

https://github.com/cornelius-keller/ceph-docker/tree/kuberne...

Re: Torus: A distributed storage system by CoreOS

#116

Post by stevelandis » Thu, Jun 02, 2016, 1:38 AM UTC

stevelandiss wrote: post #114

bcantrill wrote: post #50

It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "…

If I could also add that NBD is extremely notorious in Linux. I am a block storage developer and NBD has been one of the main reasons why so many openstack storage products (like formation data and others) have really struggled. There are many known Linux kernel issues with NBD, for example, if an NBD provider (user space daemon) exits for any reason, the kernel panics. Here is an example of a long outstanding bug th…

Here is an example script that will panic anyone using NBD to server a block device (replace qemu-nbd with the nbd export provided by torus)

qemu-img create -f qcow2 f.img 1G mkfs.ext4 f.img modprobe nbd || true qemu-nbd -c /dev/nbd0 f.img mount /dev/nbd0 k killall -KILL qemu-nbd sleep 1 ls k

Re: Torus: A distributed storage system by CoreOS

#117

Post by vidarh » Thu, Jun 02, 2016, 2:10 AM UTC

SoxNSandalz wrote: post #81

vidarh wrote: post #42

Earlier quoted context omitted.

I'm quite sure the etcd issues was "just because" of how we operated it. But the point is that I have had these issues with Etcd, using the default CoreOS configuration of Etcd, and I've not had these issues with Gluster despite having Gluster volumes running many times as long. Gluster had many issues too when it was a young project, but they were fixed many years ago. I don't doubt that Etcd too will become solid e…

Just to be clear: your complaint is etcd won't let you force the cluster into an inconsistent state to mask your poor infrastructure decisions and therefore etcd is a "dangerous choice"?

The "poor infrastructure decision" would be to assume that because there's a network split it is automatically going to be unsafe to, for example, keep scheduling containers in each data centre, even the minority, without knowing whether or not constraints have been put in place to ensure this is safe to do.

How many separate data centres do you want me to host in? Two is certainly insufficient, as if the data centre with the majority of Etcd nodes now falls of the network, you're SOL.

Three? Think you can't have two fall of face of the earth the first time? Been there. Redundant connections doesn't always help if there are major external connectivity issues and you don't have multi-million dollar budgets for connectivity alone.

Most small to mid size companies do not have the budget to run a system that is sufficiently redundant to be able to guarantee that they won't sooner or later be in a situation where either 1) the most viable partition is the minority, or 2) the Etcd cluster is centralised enough that too large parts of the cluster can loose access to it. When fighting large scale failures is the last time where you want to have to fight your tools to be able to convince them that you know what you're doing.

For those kind of situations, Etcd's lack of support for more fine-grained control of consistency or for intentionally breaking apart the cluster and letting them continue to operate independently when you know what you're doing basically makes it unsuitable for running multi-data centre clusters for anything important.

Which basically means any tools that assumes a single underlying Etcd cluster is appropriate as storage, for, say, a cluster scheduler (cough) becomes an inappropriate tool for those kind of setups.

Re: Torus: A distributed storage system by CoreOS

#118

Post by vidarh » Thu, Jun 02, 2016, 2:15 AM UTC

fintler wrote: post #48

vidarh wrote: post #29

Earlier quoted context omitted.

I've managed to lose data with Etcd, and have regularly had issues with membership issues requiring maintenance. Meanwhile I've had Glusterfs volumes remain available for 5-6 years without maintenance at all. To me at least, having it "backed by etcd" is a big red flag, not a feature.

I've managed to lose data with Etcd How did you manage to lose data?

I don't remember the details, and to be clear this was not with the most recent version of Etcd at all - it was quite a while ago. I think and hope that whatever the problem was is no longer an issue. It has certainly gotten a lot better. My point is mainly that it is way too young to be something to trust important data to. We didn't either - what we lost was a cluster configuration that we could recreate from a combination of backups and redoing a handful of operations, but at the time it was scary to see, and prompted us to be very careful about what we put into Etcd going forward.

Re: Torus: A distributed storage system by CoreOS

#119

Post by abrookewood » Thu, Jun 02, 2016, 2:16 AM UTC

joshuak wrote: post #63

Good... Good... Let the hate flow through you. An open source OS company just blogged about a new open source project that they are putting resources into. They did not release a commercial product that competes with any existing storage solution. How exactly would one expect a new project to be announced? Tell me more about how far away from done you think they are. Sorry if that's a bit sarcastic, but seriously wou…

Yes, I have to agree. There's a little too much negativity for my liking, especially from people who have a vested interest in downplaying the efforts of others.

Re: Torus: A distributed storage system by CoreOS

#120

Post by ymse » Thu, Jun 02, 2016, 2:29 AM UTC

pinewurst wrote: post #104

drauh wrote: post #93

Earlier quoted context omitted.

Re CEPH vs Lustre: what's performance like? I've seen anecdotes quoting 3 GBps over Infiniband for Lustre. (I'm curious because I run an HPC installation with Lustre and NFS over XFS, and trying to think of the future. MBTF doesn't matter as much as raw speed while it actually runs.)

At this point, this is really an apples-to-oranges comparison. Lustre, as truly awful as it is, is a POSIX filesystem (or close enough for (literally) government work). Redhat/Ceph only at the end of April, announced that POSIX functionality was ready for production. Personally, that's not when I'd choose to deploy production storage. Ceph object and nominally block have much more time in production. If you need POSI…

A proprietary (and solid) alternative to Lustre would be GPFS, which also has a long track record in HPC (and other markets in which IBM thrives).

As someone who completely shares your Lustre sentiment, I can't fathom why Intel keeps pouring resources into it.