Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Torus development has been stopped at CoreOS

github.com

Post reply on HN
Print view View original

51–59 of 59 posts

Re: Torus development has been stopped at CoreOS

#51

Post by politician »

hagbarddenstore wrote: post #42
el_isma wrote: post #33

Earlier quoted context omitted.

For one, CoreOS auto-updates smartly, so you can install and forget.

Ah.... Hahah... Hahhahahahhahahahahahahahah. No. During the 1 year I ran CoreOS in production, updates were turned off, because they caused all sorts of issues. They only reliable way of doing updates in CoreOS is to replace the machine and reconfiguring it. But then you need to automate joining etcd, which itself is a major pain in the ass.

You don't deserve the downvotes. When Docker arbitrarily changes something important and pushes those changes to Docker Hub, CoreOS is dragged along for the ride.

We had our auto-updating servers move to Docker 1.10 over a weekend. Of course, this brought down our CI/CD process because that version of Docker changed something important. Our staging environment was totally horked, but our production environment survived due to an unexplained reboot lock. We were lucky.

Turn off auto updates.

Re: Torus development has been stopped at CoreOS

#52

Post by jacques_chester »

fh973 wrote: post #47
jacques_chester wrote: post #34

Earlier quoted context omitted.

At a glance, Torus was intended to be a distributed file system. As I understand it, distributed file systems are easier than distributed block systems, but harder than distributed blob systems. A blob system is all-or-nothing. You create or replace the entire blob at once. This makes bookkeeping and replication much easier for the implementer. A filesystem supports much richer semantics, including the ability to see…

Working on a distributed file system (Quobyte), I'd say the order of complexity is file > block > object. Why? Because a drop-in replacement for a local file system also needs to be good at high-performance concurrent block IO, otherwise you wouldn't be able to run applications like databases.

So am I wrong in thinking FS semantics can give some hints on how to optimise activity?

That is: a pure block store must treat every operation with identical care, as it has no insight into the higher-level meaning. But a filesystem can, for example, give consideration to whether it's a write, read, create or appending op; where in the file it's happening; the order of operations; metadata vs file contents and so on.

Again, I am only distantly familiar with any of this.

Re: Torus development has been stopped at CoreOS

#53

Post by zzzcpan »

jacques_chester wrote: post #52
fh973 wrote: post #47

Earlier quoted context omitted.

Working on a distributed file system (Quobyte), I'd say the order of complexity is file > block > object. Why? Because a drop-in replacement for a local file system also needs to be good at high-performance concurrent block IO, otherwise you wouldn't be able to run applications like databases.

So am I wrong in thinking FS semantics can give some hints on how to optimise activity? That is: a pure block store must treat every operation with identical care, as it has no insight into the higher-level meaning. But a filesystem can, for example, give consideration to whether it's a write, read, create or appending op; where in the file it's happening; the order of operations; metadata vs file contents and so on.…

"block store must treat every operation with identical care"

Yes, and because of that it's easier to do a block storage, than a filesystem. But sure, filesystems have enough semantics to potentially scale across multiple data centers and probably be performant enough for many apps to tolerate those latencies. However, the level of complexity would be similar to that of Google's Spanner.

Re: Torus development has been stopped at CoreOS

#54

Post by hunter_n »

SEJeff wrote: post #31

I kind of wonder if there will ever be a kubernetes operator built for Ceph (not rook ontop of Ceph). Besides it being a bit of a PITA to maintain, Ceph is about as good as exists regarding OSS distributed object storage currently. If they could kill some of the operational overhead via an operator that did much of it, they might have a serious winner on their hands. Note that I'm just referring to the radosgw bits f…

There is some discussion on this in the ceph-docker project - https://github.com/ceph/ceph-docker/issues/472

Interestingly there is an unannounced project by CoreOS for a storage Operator that will handle Ceph, Gluster, etc. I'm sure we'll hear more about that now that Torus has been retired.

Re: Torus development has been stopped at CoreOS

#55

Post by umamukkara »

fh973 wrote: post #48
umamukkara wrote: post #44

Earlier quoted context omitted.

Disclosure: I work for OpenEBS project Torus was intending to write distributed block storge that is container native. Metadata management using key value (KV / etcd) method is increases the complexity and not new. Ceph tried it. OpenEBS uses a novel approch, linux sparse files for managing the blocks of a volume. Fork of Rancher longhorn. The issue of managing the large scale distributed block storage metadata is so…

What I don't get around the new efforts around container-native storage: if you decide to build a new container-native storage system, why would you aim for block storage and not file storage? Block storage is not exactly a great fit for containers as you can't access its file systems from multiple hosts and fail-over is a hassle (forced remount, fsck).

Container native storage has two aspects. One, the storage that container uses, (Like Docker uses Device Mapper, Overlay2 etc.) and two, the persistent storage that the applications inside containers need.

Both of them need to be truly container native. Portworx is attempting LCFS to provide a container native storage for containers itself https://github.com/portworx/lcfs. So, you are right. You would need a container-native storage (file system for Docker) for running containers.

OpenEBS is targeting providing containerized storage (persistent+block storage) for applications in containers. OpenEBS builds a storage volume as a container and presents the volume-container as part of the K8s POD. This way the the storage persistence problems are resolved by K8S orchestration ingelligence written for application PODs.

Ofcouse, OpenEBS containers will use LCFS when it is ready.

Re: Torus development has been stopped at CoreOS

#56

Post by wmf »

fh973 wrote: post #48
umamukkara wrote: post #44

Earlier quoted context omitted.

Disclosure: I work for OpenEBS project Torus was intending to write distributed block storge that is container native. Metadata management using key value (KV / etcd) method is increases the complexity and not new. Ceph tried it. OpenEBS uses a novel approch, linux sparse files for managing the blocks of a volume. Fork of Rancher longhorn. The issue of managing the large scale distributed block storage metadata is so…

What I don't get around the new efforts around container-native storage: if you decide to build a new container-native storage system, why would you aim for block storage and not file storage? Block storage is not exactly a great fit for containers as you can't access its file systems from multiple hosts and fail-over is a hassle (forced remount, fsck).

File systems are much harder than block storage due to the large amount of metadata and the consistency/durability requirements for metadata. So a lot of systems just build block storage and format ext4 on top or put NFS+ZFS on top (but that's not very distributed).

Re: Torus development has been stopped at CoreOS

#57

Post by SEJeff »

hunter_n wrote: post #54
SEJeff wrote: post #31

I kind of wonder if there will ever be a kubernetes operator built for Ceph (not rook ontop of Ceph). Besides it being a bit of a PITA to maintain, Ceph is about as good as exists regarding OSS distributed object storage currently. If they could kill some of the operational overhead via an operator that did much of it, they might have a serious winner on their hands. Note that I'm just referring to the radosgw bits f…

There is some discussion on this in the ceph-docker project - https://github.com/ceph/ceph-docker/issues/472 Interestingly there is an unannounced project by CoreOS for a storage Operator that will handle Ceph, Gluster, etc. I'm sure we'll hear more about that now that Torus has been retired.

Source for the unannounced project, or just overheard in person from someone? I don't see anything obvious on their github but it could be private.

Re: Torus development has been stopped at CoreOS

#58

Post by hunter_n »

SEJeff wrote: post #57
hunter_n wrote: post #54

Earlier quoted context omitted.

There is some discussion on this in the ceph-docker project - https://github.com/ceph/ceph-docker/issues/472 Interestingly there is an unannounced project by CoreOS for a storage Operator that will handle Ceph, Gluster, etc. I'm sure we'll hear more about that now that Torus has been retired.

Source for the unannounced project, or just overheard in person from someone? I don't see anything obvious on their github but it could be private.

Some more info here: https://docs.google.com/document/d/1Nm3ZQXtojd7Ruw8gQ-8xNo0v...

Re: Torus development has been stopped at CoreOS

#59

Post by jacques_chester »

zzzcpan wrote: post #53
jacques_chester wrote: post #52

Earlier quoted context omitted.

So am I wrong in thinking FS semantics can give some hints on how to optimise activity? That is: a pure block store must treat every operation with identical care, as it has no insight into the higher-level meaning. But a filesystem can, for example, give consideration to whether it's a write, read, create or appending op; where in the file it's happening; the order of operations; metadata vs file contents and so on.…

"block store must treat every operation with identical care" Yes, and because of that it's easier to do a block storage, than a filesystem. But sure, filesystems have enough semantics to potentially scale across multiple data centers and probably be performant enough for many apps to tolerate those latencies. However, the level of complexity would be similar to that of Google's Spanner.

Thankyou.
Post reply on HN