Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Torus: A distributed storage system by CoreOS

coreos.com

Post reply on HN
Print view View original

91–100 of 187 posts

Re: Torus: A distributed storage system by CoreOS

#91

Post by lisivka »

js4all wrote: post #87

Despite all the negative sentiment here, I am super excited about this. I use CoreOS heavily and really like how everything just works. Running Kubernetes on it, is the first cluster solution for me that works without configuration orgies and is robust against machine outages. Torus seems to be the missing piece. For now we use local volumes with sidecar containers for r/o storage and nfs volumes for r/w storage. All…

Gluster does not need heavy configuration: just add peers and create and start volume - 3 commands. Not harder than lvm. Of course, gluster has tons of options to fine tune cluster for various kinds of loads, which is not bad, but confusing for newbies. But if you are newbie, why you need to change defaults?

Re: Torus: A distributed storage system by CoreOS

#92

Post by nickpsecurity »

bcantrill wrote: post #67
nickpsecurity wrote: post #53

Earlier quoted context omitted.

I totally agree with you. I also liked how they said the motivation is to make Google infrastructure for everyone else. How did Google do this? They basically imitated and improved on clustered filesystems developed in HPC. There were a lot of lessons to learn for emerging cloud market in all tooling done in HPC. Some was FOSS. Whereas, many companies seem to be doing the opposite in their work on these cloud filesys…

Yes, that's absolutely been my experience -- and even then, when it comes to the data path, you will likely find new failure modes in "tried and true" as you push it harder and longer and with the bar being set at absolute perfection. I have learned this painful lesson twice: first, with Fishworks at Sun when we turned ZFS into a storage appliance -- and we learned the painful difference between something that seems…

That was a pretty humble and good read. I don't think I'd have seen the autovacuuming issue coming. Actually, this quote is a perfect example of how subtle and ridiculous these issues can be:

"During the event, one of the shard databases had all queries on our primary table blocked by a three-way interaction between the data path queries that wanted shared locks, a "transaction wraparound" autovacuum that held a shared lock and ran for several hours, and an errant query that wanted an exclusive table lock."

That's with well-documented, well-debugged components doing the kinds of things they're expected to do. Still downed by a series of just three interactions creating a corner case. Three out of a probably ridiculous number over a large amount of time. Any system redoing and debugging components plus dealing with these interaction issues will fare far worse. Hence, both of our recommendations to avoid that risk.

Note: Amazon's TLA+ reports said they model-checkers for finding bugs that didn't show up until 30+ steps in the protocols. An unlikely set of steps that actually was likely in production per logs. Reading such things, I have no hope that code review or unit tests will save my ass or my stack if I try to clean-slate Google or Amazon infrastructure. Not even gonna try haha.

Re: Torus: A distributed storage system by CoreOS

#93

Post by drauh »

nickpsecurity wrote: post #73
pinewurst wrote: post #71

Earlier quoted context omitted.

Maybe things have fallen back drastically then or we're using different definitions of HPC (it's the likes of weather or bombs for me). Supercomputer system MTBF numbers aren't high at all. Supercomputer storage is fault tolerant only by fortuitous accident and often requires week-long outages with destroyed data for software upgrades. These systems are built with the likes of dubious metafilesystems (I'm talking to…

" Supercomputer storage is fault tolerant only by fortuitous accident and often requires week-long outages with destroyed data for software upgrades." Have you tried Ceph or Sector/Sphere? Lustre is known to be crap while Ceph gets a lot of praise and Sector/Sphere has potential for reliability with good performance. I think you may just be stuck with tools that suck. I'll admit it was about 10+ years ago when I was…

Re CEPH vs Lustre: what's performance like? I've seen anecdotes quoting 3 GBps over Infiniband for Lustre.

(I'm curious because I run an HPC installation with Lustre and NFS over XFS, and trying to think of the future. MBTF doesn't matter as much as raw speed while it actually runs.)

Re: Torus: A distributed storage system by CoreOS

#94

Post by js4all »

lisivka wrote: post #91
js4all wrote: post #87

Despite all the negative sentiment here, I am super excited about this. I use CoreOS heavily and really like how everything just works. Running Kubernetes on it, is the first cluster solution for me that works without configuration orgies and is robust against machine outages. Torus seems to be the missing piece. For now we use local volumes with sidecar containers for r/o storage and nfs volumes for r/w storage. All…

Gluster does not need heavy configuration: just add peers and create and start volume - 3 commands. Not harder than lvm. Of course, gluster has tons of options to fine tune cluster for various kinds of loads, which is not bad, but confusing for newbies. But if you are newbie, why you need to change defaults?

That's interesting. When I was looking at it, it seemed overwhelming. Do you have some pointers for the basics?

Re: Torus: A distributed storage system by CoreOS

#95

Post by nickpsecurity »

drauh wrote: post #93
nickpsecurity wrote: post #73

Earlier quoted context omitted.

" Supercomputer storage is fault tolerant only by fortuitous accident and often requires week-long outages with destroyed data for software upgrades." Have you tried Ceph or Sector/Sphere? Lustre is known to be crap while Ceph gets a lot of praise and Sector/Sphere has potential for reliability with good performance. I think you may just be stuck with tools that suck. I'll admit it was about 10+ years ago when I was…

Re CEPH vs Lustre: what's performance like? I've seen anecdotes quoting 3 GBps over Infiniband for Lustre. (I'm curious because I run an HPC installation with Lustre and NFS over XFS, and trying to think of the future. MBTF doesn't matter as much as raw speed while it actually runs.)

Found these for CEPH that indicates there's fast deployments for Infiniband. Don't have more data as I've been out of HPC a while.

https://www.mellanox.com/related-docs/solutions/ppt_ceph_mel...

http://www.snia.org/sites/default/files/JohnKim_CephWithHigh...

Also, look at Sector/Sphere which was made by UDT designer for distributed, parallel workloads for supercomputing. It has significant advantages over Hadoop. It's used with high-performance links to share data between supercomputing centers.

http://sector.sourceforge.net/index.html

http://sector.sourceforge.net/pub/Sector%20vs%20Hadoop%20-%2...

Re: Torus: A distributed storage system by CoreOS

#97

Post by cyphar »

bcantrill wrote: post #50

It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "…

Just as a separate question: why are you so bitter about btrfs? ;)

Re: Torus: A distributed storage system by CoreOS

#98

Post by notacoward »

barakm wrote: post #16
xfactor973 wrote: post #15

Earlier quoted context omitted.

It sounds like etcd is in the I/O path. Is that correct?

It's in the sync() path, but not in the I/O path.

Why is it in the sync path? That seems unnecessary, as well as bad for both performance and correctness (another thing in any path means another potential for partial failure).

Re: Torus: A distributed storage system by CoreOS

#99

Post by jacques_chester »

halayli wrote: post #86
bcantrill wrote: post #80

Earlier quoted context omitted.

Even when commercially misguided, Sun always had terrific engineering talent -- and my farewell to the company captures some of that.[1] In terms of why did the company fail, the short answer is probably that SPARC was disrupted by x86, and by the time the company figured that out, it was too late to recover.[2] [1] http://dtrace.org/blogs/bmc/2010/07/25/good-bye-sun/ [2] Longer answer: https://news.ycombinator.com/i…

I've always been very impressed by Sun's talent, the team was . And thank you for dtrace! :) If only Linux community / leaders would be less arrogant and adopt technologies from Solaris/BSD that are order of magnitude better (kqueue, netgraph and more) instead of coming up with new ways to screw up. Sun should be resurrected now given that risc is leading the way and build everything on top of ARM! :). if only.

Linux can't be blamed for the decision to place Sun's technologies under the CDDL.

Re: Torus: A distributed storage system by CoreOS

#100

Post by notacoward »

mevile wrote: post #84
notacoward wrote: post #79

Earlier quoted context omitted.

Agree 100% on this, Bryan. Their claims about how existing storage solutions are a poor fit for this use case are completely false. As too often happens, they address only high-margin commercial systems and ignore the fact that open-source solutions are out there as well. (Disclaimer: I'm a Gluster developer). Then they leave both files and objects as "exercises for the reader" which shows a total lack of understandi…

How is them developing their own technology not competing on merit? Did they steal the technology? Did they claim that anyone ought to use this in production? Why all the negativity? Yeah never mind let's just discourage everyone from trying to build new technology. Nobody is forcing you to use this. If it's not for you, move on. Going out on a rant about what you think their intentions are is ridiculous. Really you…

Also, they are writing comments on HN. As are you. What exactly is your point?
Post reply on HN