Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Torus: A distributed storage system by CoreOS

coreos.com

Post reply on HN
Print view View original

81–90 of 187 posts

Re: Torus: A distributed storage system by CoreOS

#81

Post by SoxNSandalz »

vidarh wrote: post #42
ideal0227 wrote: post #39

Earlier quoted context omitted.

Have you ever thought the etcd issues was just because how you operate it? Or maybe you were running an old version of it? Have you reported to upstream? If there is a very common issue that you can meet frequently, it should have been fixed or you should have reported it.

I'm quite sure the etcd issues was "just because" of how we operated it. But the point is that I have had these issues with Etcd, using the default CoreOS configuration of Etcd, and I've not had these issues with Gluster despite having Gluster volumes running many times as long. Gluster had many issues too when it was a young project, but they were fixed many years ago. I don't doubt that Etcd too will become solid e…

Just to be clear: your complaint is etcd won't let you force the cluster into an inconsistent state to mask your poor infrastructure decisions and therefore etcd is a "dangerous choice"?

Re: Torus: A distributed storage system by CoreOS

#82

Post by subway »

philips wrote: post #26
agentultra wrote: post #21

Where's the formal specification of the system? If we want these distributed systems to be reliable how can we be sure our algorithms and processes work if we do not first model them? I'd be happy to work on a specification with the community.

We haven't gotten that far down the path. We would love the help. I believe there are some issues related to documenting the architecture and failure domains for the v0.1.1 release: https://github.com/coreos/torus/milestones/v0.1.1 This is one of those classic "release too early or release too late" sort of things that got cut in preference to get early feedback and community participation.

Coming from the company that raised hell about Docker's lack of a spec, this is an interesting approach.

Re: Torus: A distributed storage system by CoreOS

#83

Post by wnoise »

zzzcpan wrote: post #52
dap wrote: post #41

Earlier quoted context omitted.

Actually, persistent storage is fairly hard in itself. Look at what ZFS does to ensure data integrity in the face of phantom writes, dropped writes, bad controllers, and other implicit, non-fatal failures.

> Actually, persistent storage is fairly hard in itself Don't we have distributed data storages precisely because it's impossible to guaranty persistence locally? It's kind of a way to not bother trying to solve the impossible, but to achieve some guarantees on a different level.

That's one use case, but far more common is reducing latency and increasing bandwidth for distributed computing acting on a shared set of data.

Re: Torus: A distributed storage system by CoreOS

#84

Post by mevile »

notacoward wrote: post #79
bcantrill wrote: post #50

It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "…

Agree 100% on this, Bryan. Their claims about how existing storage solutions are a poor fit for this use case are completely false. As too often happens, they address only high-margin commercial systems and ignore the fact that open-source solutions are out there as well. (Disclaimer: I'm a Gluster developer). Then they leave both files and objects as "exercises for the reader" which shows a total lack of understandi…

How is them developing their own technology not competing on merit? Did they steal the technology? Did they claim that anyone ought to use this in production? Why all the negativity? Yeah never mind let's just discourage everyone from trying to build new technology. Nobody is forcing you to use this. If it's not for you, move on. Going out on a rant about what you think their intentions are is ridiculous.

Really you come off as defensive. I don't understand how CoreOS has disrespected anyone by coming up with their own approach to the problem.

They're not writing blog posts or comments on HN, they're writing code.

Re: Torus: A distributed storage system by CoreOS

#85

Post by merb »

bcantrill wrote: post #50

It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "…

Actually the biggest problem is actual either Full Posix or Mutable Storage. If there is just one way in and never a way out, it's actually way easier. So practically if you don't run your database on shared storage etc and manage it "the old way". you could have something reliable by a 'never delete'.

However CAS and immutable file systems aren't that common.

Re: Torus: A distributed storage system by CoreOS

#86

Post by halayli »

bcantrill wrote: post #80
halayli wrote: post #75

Earlier quoted context omitted.

Looking at dtrace, fishwork, zfs and Solaris (imo was the best OS technologically speaking) it's interesting to see how strong and innovative its engineering team was while the business was just going down. How did that happen? How can the engineering team be that productive and functional while the business vision was so lacking?

Even when commercially misguided, Sun always had terrific engineering talent -- and my farewell to the company captures some of that.[1] In terms of why did the company fail, the short answer is probably that SPARC was disrupted by x86, and by the time the company figured that out, it was too late to recover.[2] [1] http://dtrace.org/blogs/bmc/2010/07/25/good-bye-sun/ [2] Longer answer: https://news.ycombinator.com/i…

I've always been very impressed by Sun's talent, the team was . And thank you for dtrace! :) If only Linux community / leaders would be less arrogant and adopt technologies from Solaris/BSD that are order of magnitude better (kqueue, netgraph and more) instead of coming up with new ways to screw up.

Sun should be resurrected now given that risc is leading the way and build everything on top of ARM! :). if only.

Re: Torus: A distributed storage system by CoreOS

#87

Post by js4all »

Despite all the negative sentiment here, I am super excited about this. I use CoreOS heavily and really like how everything just works. Running Kubernetes on it, is the first cluster solution for me that works without configuration orgies and is robust against machine outages. Torus seems to be the missing piece. For now we use local volumes with sidecar containers for r/o storage and nfs volumes for r/w storage.

All other solutions are not practical. GCE and EBS are only single mount. iSCSI is unsupported in the cloud. Leaving only Ceph and Glusterfs, both mentioned here, but needing heavy configuration.

Re: Torus: A distributed storage system by CoreOS

#88

Post by notacoward »

mevile wrote: post #84
notacoward wrote: post #79

Earlier quoted context omitted.

Agree 100% on this, Bryan. Their claims about how existing storage solutions are a poor fit for this use case are completely false. As too often happens, they address only high-margin commercial systems and ignore the fact that open-source solutions are out there as well. (Disclaimer: I'm a Gluster developer). Then they leave both files and objects as "exercises for the reader" which shows a total lack of understandi…

How is them developing their own technology not competing on merit? Did they steal the technology? Did they claim that anyone ought to use this in production? Why all the negativity? Yeah never mind let's just discourage everyone from trying to build new technology. Nobody is forcing you to use this. If it's not for you, move on. Going out on a rant about what you think their intentions are is ridiculous. Really you…

> They're not writing blog posts or comments on HN, they're writing code.

Actually, the problem is that 90% of the code still remains to be written, while others (including me) have already done so. They've addressed only the very simplest part of the problem, not even far enough to show any performance comparisons, in a manner strongly reminiscent of Sheepdog (belying your "own approach" claim). That's a poor basis from which to promise so much. It's like writing an interpreter for a simple programming language and claiming it'll be a full optimizing C++ compiler soon. Just a few little pieces remaining, right?

It's perfectly fine for them to start their own project and have high hopes for it. The more the merrier. However, I have little patience for people who blur the lines between what's there and what might hypothetically exist some time in the future. That's far too often used to stifle real innovation that's occurring elsewhere. Maybe it's more common in storage than whatever your specialty is, but it's a well known part of the playbook. It's important to be crystal clear about what's real vs. what's seriously thought out vs. what's total blue-sky. Users and fellow developers deserve nothing less.

Re: Torus: A distributed storage system by CoreOS

#89

Post by Perceptes »

__jal wrote: post #32
bogomipz wrote: post #6

Earlier quoted context omitted.

I kind of see this as a general trend where CoreOS is concerned they reimplement everything in house, even when open source solution might already exist - rkt, etcd, now this.

Yeah, I've been smelling NIH for quite a while. And it generally seems that, when they say "composable modular tools", one should read "includes dependencies on the rest of our stack". This isn't intended as harshness. It is just that every time a tool from COS seems interesting enough to look in to, it turns out that I wouldn't be investing in a tool, it would be investing in an ecosystem that duplicates a ton of wh…

Kubernetes is a big exception to this, it seems. Before Kubernetes was announced, CoreOS was actively developing fleet. After Kubernetes, fleet development was largely halted and is now just a "distributed init system for low-level cluster components," e.g. a way to bootstrap Kubernetes itself. I asked the lead developer of fleet if they would have created it if Kubernetes had existed at the time, and he said no. Kubernetes has turned into a big part of the CoreOS pitch, and they've contributed a lot to the project. Giving up a project you started in house because you recognized that another organization's project fit the bill better is very far from NIH.

Re: Torus: A distributed storage system by CoreOS

#90

Post by jbooth »

bcantrill wrote: post #67
nickpsecurity wrote: post #53

Earlier quoted context omitted.

I totally agree with you. I also liked how they said the motivation is to make Google infrastructure for everyone else. How did Google do this? They basically imitated and improved on clustered filesystems developed in HPC. There were a lot of lessons to learn for emerging cloud market in all tooling done in HPC. Some was FOSS. Whereas, many companies seem to be doing the opposite in their work on these cloud filesys…

Yes, that's absolutely been my experience -- and even then, when it comes to the data path, you will likely find new failure modes in "tried and true" as you push it harder and longer and with the bar being set at absolute perfection. I have learned this painful lesson twice: first, with Fishworks at Sun when we turned ZFS into a storage appliance -- and we learned the painful difference between something that seems…

What triggered me was just throwing out "reed solomon" when talking about random writes. How does that work? We'll read from 5 places to complete your write?
Post reply on HN