Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Torus: A distributed storage system by CoreOS

coreos.com

Post reply on HN
Print view View original

101–110 of 187 posts

Re: Torus: A distributed storage system by CoreOS

#101

Post by agentultra »

nickpsecurity wrote: post #92
bcantrill wrote: post #67

Earlier quoted context omitted.

Yes, that's absolutely been my experience -- and even then, when it comes to the data path, you will likely find new failure modes in "tried and true" as you push it harder and longer and with the bar being set at absolute perfection. I have learned this painful lesson twice: first, with Fishworks at Sun when we turned ZFS into a storage appliance -- and we learned the painful difference between something that seems…

That was a pretty humble and good read. I don't think I'd have seen the autovacuuming issue coming. Actually, this quote is a perfect example of how subtle and ridiculous these issues can be: "During the event, one of the shard databases had all queries on our primary table blocked by a three-way interaction between the data path queries that wanted shared locks, a "transaction wraparound" autovacuum that held a shar…

> Note: Amazon's TLA+ reports said they model-checkers for finding bugs that didn't show up until 30+ steps in the protocols. An unlikely set of steps that actually was likely in production per logs. Reading such things, I have no hope that code review or unit tests will save my ass or my stack if I try to clean-slate Google or Amazon infrastructure. Not even gonna try haha.

For those unfamiliar with the reference, there was an eye-opening report from Amazon engineers who'd used formal methods to find bugs in the design of S3 and other systems several years ago [0]. I highly recommend reading it and then watching as many of Leslie Lamports talks on TLA+ and system specifications as possible.

[0] http://research.microsoft.com/en-us/um/people/lamport/tla/fo...

Re: Torus: A distributed storage system by CoreOS

#102

Post by jsmthrowaway »

I love that I know who this is at CoreOS by the choice of username. Pick better throwaways if you're going to blow up a thread like this. (ideal0227 upthread is also a CoreOS employee and one of the etcd developers.)

On topic, yes. It is quite trivial to lose data with etcd and pretty much everybody I know who runs it has experienced problems. Try backing it up and restoring. etcd is compsci great yet operationally burdensome; it is extremely difficult to operate, tends to make assumptions and explain to you how to operate your fleet, and the developers are not very receptive to these and other operational concerns. This stems from CoreOS culture, to be quite clear -- CoreOS seeks to eliminate operations as a discipline and replace it with software, and tends to disregard or devalue operational concerns as a result.

Launching down the storage path with Go software and disregarding several decades of operational experience in the industry on how to do distributed storage correctly (not to mention the "looking down" on etcd-related infrastructure decisions from this anonymous employee) should indicate to you that I'm not making this up, despite how it may appear.

90% of folks I know on etcd, I'd estimate, have (a) reverted to Zookeeper or (b) moved on to Consul. It is the single piece of software that is holding Kubernetes back, too, and it's fairly obvious from roadmap direction that Kubernetes is now the etcd customer. Plan accordingly.

Re: Torus: A distributed storage system by CoreOS

#103

Post by brazzledazzle »

amelius wrote: post #31
runako wrote: post #19

Earlier quoted context omitted.

> I'm still hoping for an open-source competitor for elasticsearch I thought Elasticsearch was open-sourced, and the GitHub repo says Apache v2. What am I missing here?

Nothing. It is open-source. But it needs an open-source competitor. There's too little happening at the open-source side of search.

What about Solr? They're both based on Lucene but I imagine it qualifies.

Re: Torus: A distributed storage system by CoreOS

#104

Post by pinewurst »

drauh wrote: post #93
nickpsecurity wrote: post #73

Earlier quoted context omitted.

" Supercomputer storage is fault tolerant only by fortuitous accident and often requires week-long outages with destroyed data for software upgrades." Have you tried Ceph or Sector/Sphere? Lustre is known to be crap while Ceph gets a lot of praise and Sector/Sphere has potential for reliability with good performance. I think you may just be stuck with tools that suck. I'll admit it was about 10+ years ago when I was…

Re CEPH vs Lustre: what's performance like? I've seen anecdotes quoting 3 GBps over Infiniband for Lustre. (I'm curious because I run an HPC installation with Lustre and NFS over XFS, and trying to think of the future. MBTF doesn't matter as much as raw speed while it actually runs.)

At this point, this is really an apples-to-oranges comparison.

Lustre, as truly awful as it is, is a POSIX filesystem (or close enough for (literally) government work).

Redhat/Ceph only at the end of April, announced that POSIX functionality was ready for production. Personally, that's not when I'd choose to deploy production storage. Ceph object and nominally block have much more time in production.

If you need POSIX, trusting Ceph at this point is an issue unless, as you say, MTBF isn't a concern. You might want to try BeeGFS, a similar logical model but much simpler to implement, performance up to a very high level, and a record of reliable HPC deployments (as oxymoronic as that sounds).

If you can do with object then certainly exorcise Lustre from your environment in favor of Ceph (or try Scality if non-OS isn't an issue). Lustre's only useful as a jobs program anyway - keeping people occupied who'd otherwise be bodging up real software.

Re: Torus: A distributed storage system by CoreOS

#105

Post by halayli »

jacques_chester wrote: post #99
halayli wrote: post #86

Earlier quoted context omitted.

I've always been very impressed by Sun's talent, the team was . And thank you for dtrace! :) If only Linux community / leaders would be less arrogant and adopt technologies from Solaris/BSD that are order of magnitude better (kqueue, netgraph and more) instead of coming up with new ways to screw up. Sun should be resurrected now given that risc is leading the way and build everything on top of ARM! :). if only.

Linux can't be blamed for the decision to place Sun's technologies under the CDDL.

there's no license for kqueue's interface or netgraph, and many other great interfaces. but they decide to create square wheels and not leverage from errors that others have made before them.

Re: Torus: A distributed storage system by CoreOS

#106

Post by nickpsecurity »

pinewurst wrote: post #104
drauh wrote: post #93

Earlier quoted context omitted.

Re CEPH vs Lustre: what's performance like? I've seen anecdotes quoting 3 GBps over Infiniband for Lustre. (I'm curious because I run an HPC installation with Lustre and NFS over XFS, and trying to think of the future. MBTF doesn't matter as much as raw speed while it actually runs.)

At this point, this is really an apples-to-oranges comparison. Lustre, as truly awful as it is, is a POSIX filesystem (or close enough for (literally) government work). Redhat/Ceph only at the end of April, announced that POSIX functionality was ready for production. Personally, that's not when I'd choose to deploy production storage. Ceph object and nominally block have much more time in production. If you need POSI…

Sounds like good recommendations based on what research I've done in these things. I forgot BeeGFS but it was in my bookmarks. Must be good in some way. ;)

Re: Torus: A distributed storage system by CoreOS

#107

Post by notacoward »

nickpsecurity wrote: post #73
pinewurst wrote: post #71

Earlier quoted context omitted.

Maybe things have fallen back drastically then or we're using different definitions of HPC (it's the likes of weather or bombs for me). Supercomputer system MTBF numbers aren't high at all. Supercomputer storage is fault tolerant only by fortuitous accident and often requires week-long outages with destroyed data for software upgrades. These systems are built with the likes of dubious metafilesystems (I'm talking to…

" Supercomputer storage is fault tolerant only by fortuitous accident and often requires week-long outages with destroyed data for software upgrades." Have you tried Ceph or Sector/Sphere? Lustre is known to be crap while Ceph gets a lot of praise and Sector/Sphere has potential for reliability with good performance. I think you may just be stuck with tools that suck. I'll admit it was about 10+ years ago when I was…

What does "potential for reliability" even mean? Even Lustre, which you malign and I have maligned even more, has potential for reliability if they just fix a few hundred egregious design flaws.

Re: Torus: A distributed storage system by CoreOS

#108

Post by ymse »

nickpsecurity wrote: post #95
drauh wrote: post #93

Earlier quoted context omitted.

Re CEPH vs Lustre: what's performance like? I've seen anecdotes quoting 3 GBps over Infiniband for Lustre. (I'm curious because I run an HPC installation with Lustre and NFS over XFS, and trying to think of the future. MBTF doesn't matter as much as raw speed while it actually runs.)

Found these for CEPH that indicates there's fast deployments for Infiniband. Don't have more data as I've been out of HPC a while. https://www.mellanox.com/related-docs/solutions/ppt_ceph_mel... http://www.snia.org/sites/default/files/JohnKim_CephWithHigh... Also, look at Sector/Sphere which was made by UDT designer for distributed, parallel workloads for supercomputing. It has significant advantages over Hadoop. It'…

Native RDMA support for Ceph is still a ways off. The current implementation requires disabling Cephx authentication, which is a no-go in any environment where you can't completely trust every client (e.g. "cloud", where most current deployments/users live). It also hasn't seen much development since the initial proof-of-concept (still highly experimental).

That said, IPoIB should work just fine, and the main bottleneck currently is (Ethernet) latency. I'm running a couple of 1,6PB clusters (432 * 4TB) and can only get 20-60MBps on a single client with a 4kB block size, but got bored of benchmarking after saturating 5 concurrent 10Gb clients with a 4MB block size.

I do expect the RDMA situation to improve substantially over the next year or so, even if authentication will still be unsupported. The latter generally isn't a problem in HPC where stuff like GPFS lives (where you also have to trust every client). And they clearly want that market now that CephFS is finally deemed production ready.

Re: Torus: A distributed storage system by CoreOS

#110

Post by ymse »

Inflatablewoman wrote: post #5
CSDude wrote: post #3

Seems really nice. Would love to see it how it compares to Ceph, we were considering Ceph for our container storage. For starters, it seems much more simpler to deploy.

In my old firm we managed to get ceph running on coreos using ceph in docker. The storage was exposed using a s3 gateway. https://github.com/ceph/ceph-docker

Docker supports native RBD since version 1.8. Which should be both faster and more reliable than using the S3 gateway (though that has other benefits and becoming pretty good too).
Post reply on HN