This reminds me of the SheepDog storage architecture (https://sheepdog.github.io/sheepdog/)
SheepDog provides safe distributed block devices and uses corosync or Zookeeper instead of etcd.
Torus: A distributed storage system by CoreOS
131–140 of 187 posts
Re: Torus: A distributed storage system by CoreOS
#132Seems really nice. Would love to see it how it compares to Ceph, we were considering Ceph for our container storage. For starters, it seems much more simpler to deploy.
I work at a company where object storage is something they have worked on for many years to get it stable. You can download it for free (it's Open Source)! Open vStorage is very easy to deploy through Ansible! Try it :)! https://www.openvstorage.com/ https://github.com/openvstorage
Re: Torus: A distributed storage system by CoreOS
#133Good... Good... Let the hate flow through you. An open source OS company just blogged about a new open source project that they are putting resources into. They did not release a commercial product that competes with any existing storage solution. How exactly would one expect a new project to be announced? Tell me more about how far away from done you think they are. Sorry if that's a bit sarcastic, but seriously wou…
I for one find it very positive, a storage engine implemented in Go, for what I could gather in a quick Github glance.
For me, this is surely systems work.
Re: Torus: A distributed storage system by CoreOS
#134It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "…
Their one saving grace might be that NBD is such a simple protocol. By not trying to make it a full-fledged FS, they might actually have hit on a tractable problem.
Re: Torus: A distributed storage system by CoreOS
#135Earlier quoted context omitted.
I kind of see this as a general trend where CoreOS is concerned they reimplement everything in house, even when open source solution might already exist - rkt, etcd, now this.
what is the existing alternative for storage in distributed container environments?
Have a look at Open vStorage (http://www.openvstorage.com). Some highlights: open-source, core is battle proven for more than 7 years, performance, scales across datacenters , unlimited snapshots, ...
(disclaimer: I'm an engineer there...)
Re: Torus: A distributed storage system by CoreOS
#136Earlier quoted context omitted.
Yes, that's absolutely been my experience -- and even then, when it comes to the data path, you will likely find new failure modes in "tried and true" as you push it harder and longer and with the bar being set at absolute perfection. I have learned this painful lesson twice: first, with Fishworks at Sun when we turned ZFS into a storage appliance -- and we learned the painful difference between something that seems…
That was a pretty humble and good read. I don't think I'd have seen the autovacuuming issue coming. Actually, this quote is a perfect example of how subtle and ridiculous these issues can be: "During the event, one of the shard databases had all queries on our primary table blocked by a three-way interaction between the data path queries that wanted shared locks, a "transaction wraparound" autovacuum that held a shar…
Anyone who's worked with Postgres at scale would guess autovacuum.
Postgres doesn't have many weaknesses, but most of them relate to autovacuum.
Re: Torus: A distributed storage system by CoreOS
#137It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "…
If I could also add that NBD is extremely notorious in Linux. I am a block storage developer and NBD has been one of the main reasons why so many openstack storage products (like formation data and others) have really struggled. There are many known Linux kernel issues with NBD, for example, if an NBD provider (user space daemon) exits for any reason, the kernel panics. Here is an example of a long outstanding bug th…
But: that's not a kernel panic, just a BUG() stack trace from kernel which doesn't halt the system.
Re: Torus: A distributed storage system by CoreOS
#138Good... Good... Let the hate flow through you. An open source OS company just blogged about a new open source project that they are putting resources into. They did not release a commercial product that competes with any existing storage solution. How exactly would one expect a new project to be announced? Tell me more about how far away from done you think they are. Sorry if that's a bit sarcastic, but seriously wou…
I absolutely agree, all those negative comments sound like rants from grumpy old guys. I have worked in HPC distributed storage and the problems the CoreOS people are trying to solve is a valid and present one: deploying flexible distributed storage today is an absolute pain.
Plus all the arguments about local storage failure being hard are moot because this is a distributed solution and the hardship of local storage reliability can be abstracted away almost completely.
Re: Torus: A distributed storage system by CoreOS
#139Watch out, next we'll be seeing CoreOS writing their own encryption protocols. As someone that's designed and built storage clusters from the ground up - take it from me: storage is not as easy as it seems if you care about your data and performance at scale. The number of people I've seen using CoreOS and then moving away from it is quite alarming, I feel like CoreOS will become the Ubuntu of the container world.
+1 for the assertion about the destiny of CoreOS.
Re: Torus: A distributed storage system by CoreOS
#140Earlier quoted context omitted.
Linux can't be blamed for the decision to place Sun's technologies under the CDDL.
there's no license for kqueue's interface or netgraph, and many other great interfaces. but they decide to create square wheels and not leverage from errors that others have made before them.
You're right.
I was thinking mostly of the eternal buzzkill that ZFS is only usable through indirect means.