Congrats on the relaase! How does this compare to GlusterFS (loved it, was free - I used it before they got acquired by RedHat) and Isilon (expensive but fast and reliable). Is there an upper limit on number of nodes(?) or total size?
Torus: A distributed storage system by CoreOS
51–60 of 187 posts
Re: Torus: A distributed storage system by CoreOS
#52Earlier quoted context omitted.
Persistent storage isn't a hard problem. Distributed, well performing, scalable and consistent storage is a hard problem.
Actually, persistent storage is fairly hard in itself. Look at what ZFS does to ensure data integrity in the face of phantom writes, dropped writes, bad controllers, and other implicit, non-fatal failures.
> Actually, persistent storage is fairly hard in itself
Don't we have distributed data storages precisely because it's impossible to guaranty persistence locally? It's kind of a way to not bother trying to solve the impossible, but to achieve some guarantees on a different level.
Re: Torus: A distributed storage system by CoreOS
#53It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "…
I totally agree with you. I also liked how they said the motivation is to make Google infrastructure for everyone else. How did Google do this? They basically imitated and improved on clustered filesystems developed in HPC. There were a lot of lessons to learn for emerging cloud market in all tooling done in HPC. Some was FOSS.
Whereas, many companies seem to be doing the opposite in their work on these cloud filesystems. They don't build on proven, already-OSS components that have been battle-tested for a long time. They lack features and wisdom from prior deployments. They duplicate effort. They are also prone to using popular components, languages, whatever with implicitly-higher risk due to fact they were never built for fault-tolerant systems. Actually, many of them often assume something will watch and help them out in time of failure.
If it's high-assurance or fault-tolerance, my manta is "tried and true beats novel and new." Just repurpose what's known to work while improving its capabilities and code quality. Knocks out risks you know about plus others you don't since they were never documents. Has that been your experience, too? Should be a maxim in IT given how often problem plays out.
Re: Torus: A distributed storage system by CoreOS
#54I love CoreOS and they've done some super impressive engineering. But really, a new storage system? Rewriting in Go and using etcd for central state management makes things easier, but this is still a hard problem. Some things that need to be solved sooner or later: data replication so that N faults of X entities are protected against (X can be disks, enclosures, racks, data centers, regions, ..), recovery from faile…
> But really, a new storage system?
I think the post outlines quite well why they chose to build a new distributed storage layer. From a product perspective it makes a lot of sense for them to have an offering in this space. Also, the fact that it's a hard problem is what makes it a compelling product offering. You can't make much money solving easy problems.
Re: Torus: A distributed storage system by CoreOS
#55I love CoreOS and they've done some super impressive engineering. But really, a new storage system? Rewriting in Go and using etcd for central state management makes things easier, but this is still a hard problem. Some things that need to be solved sooner or later: data replication so that N faults of X entities are protected against (X can be disks, enclosures, racks, data centers, regions, ..), recovery from faile…
It's like any time we take a step forward in one area we have to reinvent the last 50 years of computing to support it. Persistent storage is a "hard problem" Really?
Persistent storage on today's filesystems and complex hardware is a hard problem. All kinds of failures can happen during any write. Some are obvious with some silently corrupting data. There's been decades of work on approaches to dealing with this with a variety of tradeoffs. Picking the right one for a widely-deployed, portable, distributed app is tricky by itself.
They're aiming to do a lot more than that. ;)
Re: Torus: A distributed storage system by CoreOS
#56So a userspace fs, I'm guessing this will use fuse to actually expose a POSIX fs? Small sync writes will absolutely kill performance and will require all sorts of hacks like glusterfs has had to implement due to the amount of context switches. CoreOS really should have added whatever needed to ceph, filesystems are not something you just hack together overnight.
Re: Torus: A distributed storage system by CoreOS
#57I'm always in awe and grateful for the teams like CoreOS that tackle these tough problems.
But when it comes to container storage I'm confused.
The 12 factor app methodologies work very well for every service I have written and supported.
Processes/containers get ephemeral storage. When you need persistence you delegate to a stateful service, generally Postgres and S3.
Id much rather write my apps around these simple and well understood constraints then depend on magic file systems.
Re: Torus: A distributed storage system by CoreOS
#58I'm always in awe and grateful for the teams like CoreOS that tackle these tough problems. But when it comes to container storage I'm confused. The 12 factor app methodologies work very well for every service I have written and supported. Processes/containers get ephemeral storage. When you need persistence you delegate to a stateful service, generally Postgres and S3. Id much rather write my apps around these simple…
The average 12 factor app doesn't really need this, but as you mentioned, sometimes you delegate to Postgres, or S3, and today those are available as hosted products via Amazon, Google, Heroku, etc.
However, when your running in your own datacenter, you don't have the luxury of using Amazon's hosted products, so Torus exists to help provide some building blocks to build your own S3, or potentially even RDS.
Re: Torus: A distributed storage system by CoreOS
#59Earlier quoted context omitted.
A few major things we wanted to accomplish: Backed by etcd. Today etcd is a well tested and widely used consistent store with users like Kubernetes, Flannel (now Canal), Fleet, SkyDNS and many others. Building something like etcd requires tons of testing and etcd is becoming the solid go to for this category of distributed problems. Easy to work on code base. Building an OSS community around a complex technology is r…
I've managed to lose data with Etcd, and have regularly had issues with membership issues requiring maintenance. Meanwhile I've had Glusterfs volumes remain available for 5-6 years without maintenance at all. To me at least, having it "backed by etcd" is a big red flag, not a feature.
I agree. I was also thinking of Ceph when I wrote my reply to Cantrill. Esp Ceph on XFS given maturity of both plus enormous effort that's gone into XFS. Sector/Sphere with UDT is also kick-ass. It took both of these a long time to get to where they were fighting all kinds of unforseen issues. They also tried to build on proven components that themselves were battletested for years.
So, this company is trading away stuff like that for custom components and Etcd? And for a component focusing on integrity and availability? Huh?
Re: Torus: A distributed storage system by CoreOS
#60Earlier quoted context omitted.
what is the existing alternative for storage in distributed container environments?
Ceph, OpenStack Swift, GlusterFS, OrangeFS, Lustre. https://en.wikipedia.org/wiki/Clustered_file_system#Distribu... https://en.wikipedia.org/wiki/Object_storage
Add this to your list:
http://sector.sourceforge.net/
I swore I added it to the Wikipedia links. I'll have to do it again, I guess.