I kind of wonder if there will ever be a kubernetes operator built for Ceph (not rook ontop of Ceph). Besides it being a bit of a PITA to maintain, Ceph is about as good as exists regarding OSS distributed object storage currently. If they could kill some of the operational overhead via an operator that did much of it, they might have a serious winner on their hands. Note that I'm just referring to the radosgw bits for the S3 style storage API, not the posix filesystem bits.
Torus development has been stopped at CoreOS
31–40 of 59 posts
Re: Torus development has been stopped at CoreOS
#32I have karma to burn on this, so here goes: I worked for several years in VFX/HPC. 30k+ cpus and 15pbs of storage. Firstly with storage its very rare that people want actual block storage (unless you are hosting VMs, but thats so 2007.....) Yes, I know, openstack, but that's just fucking horrific, seriously just use netboot and be done with it. I've seen people do it inside new clustereing systems, but its really not…
I find your offhand dismissal of clustered filesystems, on which literally every supercomputer relies, to be a little strange. They might not have worked well for you, but "googleable" is not the bar that is generally set for the HPC problem space.
Re: Torus development has been stopped at CoreOS
#33The next question that needs to be answered at CoreOS: "Why, exactly, are we maintaining our own Linux distro when the Go binaries that we're writing can mostly ignore userspace?"
For one, CoreOS auto-updates smartly, so you can install and forget.
Re: Torus development has been stopped at CoreOS
#34Dangit, I trust the CoreOS team more/better than a lot of people in the space. Torus would have been so useful. At the other end of the spectrum though, maybe this is reasonable? As a developer, my first thoughts for "I want my own S3" is not etcd (strong consistency) but projects like https://github.com/minio/minio , or even using eventually consistent SQLite replication / synchronization tools https://github.com/gu…
At a glance, Torus was intended to be a distributed file system. As I understand it, distributed file systems are easier than distributed block systems, but harder than distributed blob systems.
A blob system is all-or-nothing. You create or replace the entire blob at once. This makes bookkeeping and replication much easier for the implementer.
A filesystem supports much richer semantics, including the ability to seek parts of files and modify small regions of files. You need a lot more mechanics to maintain consistency across a network.
A block store is difficult because you're trying to work at very high speed on very small units of state wooshing back and forth willy-nilly. You don't get to rely on any of the higher semantics provided by a filesystem or blobstore, since you're pretending to be a magical harddrive.
I am often wrong in these matters, as an interested outsider, so I'd be happy to receive correction.
Re: Torus development has been stopped at CoreOS
#35The next question that needs to be answered at CoreOS: "Why, exactly, are we maintaining our own Linux distro when the Go binaries that we're writing can mostly ignore userspace?"
For one, CoreOS auto-updates smartly, so you can install and forget.
There were Linux distros that could upgrade across releases twenty(!) years ago. There is so much opportunity in infrastructure right now, so it strikes me as weird that CoreOS took a bunch of VC money to go off on an extended Linux-From-Scratch-Adventure.
Re: Torus development has been stopped at CoreOS
#36I have karma to burn on this, so here goes: I worked for several years in VFX/HPC. 30k+ cpus and 15pbs of storage. Firstly with storage its very rare that people want actual block storage (unless you are hosting VMs, but thats so 2007.....) Yes, I know, openstack, but that's just fucking horrific, seriously just use netboot and be done with it. I've seen people do it inside new clustereing systems, but its really not…
I find your offhand dismissal of clustered filesystems, on which literally every supercomputer relies, to be a little strange. They might not have worked well for you, but "googleable" is not the bar that is generally set for the HPC problem space.
Let me expand:
What I was angling at is unpartitioned clustered filesystems are fragile. Dirty nodes cause lots of problems. Storage area networks which then interface via another means is much easier to look after, for little/no performance hit. (it can be a lot faster because there is less coordination/chatter)
Would I use Lustre for long term storage, with 100% uptime requirement? no. WOuld I use it as a linearly scalable scratch space? yes.
Would I use GPFS (or whatever its been rebranded to) instead? yes, probably. If I was hosting lost of small(Would I wire all my clients directly into the same clustered namespace? not if I could avoid it.
The issue is this, in HPC, or any other multi-node scheduled based system(basically mainframes but without the documentation, or the error checking) nodes die in new and interesting ways, If you have shared memory then its surprising how well interesting problems propagate.
In most HPC senarios, you can, if you are desperate stop and restart from a known good point. If you are serving public things, you don't get that option. So its in your best interest to partition.
Now, as for the googable bar, Unless everyone running the cluster is intimately familiar with the filesystem, including the interesting ways its fucked with VFS, How the metadata server handles stale locks, sudden bursts in lost clients or what ever, you need google. Even if you are a master, you might forget.
Re: Torus development has been stopped at CoreOS
#37This is good news - CoreOS needs to focus on what's most important to their core business to be successful.
Being chock full of bright, relatively young and enthusiastic engineers drunk on the Golang kool-aid, there's a very real risk of getting distracted by reimplementing everything under the sun in their favorite shiny new language.
Even if Torus is a good idea, CoreOS has to prioritize, commit, and execute. They can't afford too many diversions. This is a competitive space, their opportunity window and runway are both limited, as usual.
Re: Torus development has been stopped at CoreOS
#38I have karma to burn on this, so here goes: I worked for several years in VFX/HPC. 30k+ cpus and 15pbs of storage. Firstly with storage its very rare that people want actual block storage (unless you are hosting VMs, but thats so 2007.....) Yes, I know, openstack, but that's just fucking horrific, seriously just use netboot and be done with it. I've seen people do it inside new clustereing systems, but its really not…
Why isn't a fourth option being explored: local storage with async replication? Seems like it'd be fairly simple and fast, and no worse than non-clustered NFS regarding data integrity.
I'm just talking from ignorance, so am I missing something?
Re: Torus development has been stopped at CoreOS
#39I have karma to burn on this, so here goes: I worked for several years in VFX/HPC. 30k+ cpus and 15pbs of storage. Firstly with storage its very rare that people want actual block storage (unless you are hosting VMs, but thats so 2007.....) Yes, I know, openstack, but that's just fucking horrific, seriously just use netboot and be done with it. I've seen people do it inside new clustereing systems, but its really not…
Why isn't a fourth option being explored: local storage with async replication? Seems like it'd be fairly simple and fast, and no worse than non-clustered NFS regarding data integrity. I'm just talking from ignorance, so am I missing something?
I believe this is exactly what's needed..
Re: Torus development has been stopped at CoreOS
#40I have karma to burn on this, so here goes: I worked for several years in VFX/HPC. 30k+ cpus and 15pbs of storage. Firstly with storage its very rare that people want actual block storage (unless you are hosting VMs, but thats so 2007.....) Yes, I know, openstack, but that's just fucking horrific, seriously just use netboot and be done with it. I've seen people do it inside new clustereing systems, but its really not…
> Firstly with storage its very rare that people want actual block storage (unless you are hosting VMs, but thats so 2007.....)
What do you mean by this? A 1:1 remote mapping to a physical block?