Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Torus development has been stopped at CoreOS

github.com

Post reply on HN
Print view View original

41–50 of 59 posts

Re: Torus development has been stopped at CoreOS

#41

Post by KaiserPro »

icebraining wrote: post #38
KaiserPro wrote: post #30

I have karma to burn on this, so here goes: I worked for several years in VFX/HPC. 30k+ cpus and 15pbs of storage. Firstly with storage its very rare that people want actual block storage (unless you are hosting VMs, but thats so 2007.....) Yes, I know, openstack, but that's just fucking horrific, seriously just use netboot and be done with it. I've seen people do it inside new clustereing systems, but its really not…

Why isn't a fourth option being explored: local storage with async replication? Seems like it'd be fairly simple and fast, and no worse than non-clustered NFS regarding data integrity. I'm just talking from ignorance, so am I missing something?

Oh yes, sorry I assumed that.

The simplest storage, is a bunch of dumb servers (well beefy dumb servers) with some application aware scripts to move/copy the dataset.

A place I worked at had a wrapper around rsync that would split up the directory and spawn multiple rsyncs to do a parallel copy.

The Directory structure was effectively copy on write, so backup to the nearline was <15 minutes

Re: Torus development has been stopped at CoreOS

#42

Post by hagbarddenstore »

el_isma wrote: post #33
alrs wrote: post #23

The next question that needs to be answered at CoreOS: "Why, exactly, are we maintaining our own Linux distro when the Go binaries that we're writing can mostly ignore userspace?"

For one, CoreOS auto-updates smartly, so you can install and forget.

Ah.... Hahah... Hahhahahahhahahahahahahahah.

No.

During the 1 year I ran CoreOS in production, updates were turned off, because they caused all sorts of issues.

They only reliable way of doing updates in CoreOS is to replace the machine and reconfiguring it. But then you need to automate joining etcd, which itself is a major pain in the ass.

Re: Torus development has been stopped at CoreOS

#43

Post by icebraining »

KaiserPro wrote: post #41
icebraining wrote: post #38

Earlier quoted context omitted.

Why isn't a fourth option being explored: local storage with async replication? Seems like it'd be fairly simple and fast, and no worse than non-clustered NFS regarding data integrity. I'm just talking from ignorance, so am I missing something?

Oh yes, sorry I assumed that. The simplest storage, is a bunch of dumb servers (well beefy dumb servers) with some application aware scripts to move/copy the dataset. A place I worked at had a wrapper around rsync that would split up the directory and spawn multiple rsyncs to do a parallel copy. The Directory structure was effectively copy on write, so backup to the nearline was <15 minutes

I wonder if you could have something better, closer to streaming replication of databases. A few weeks ago I found zrep, which sounds more like what I had in mind: http://www.bolthole.com/solaris/zrep/

Re: Torus development has been stopped at CoreOS

#44

Post by umamukkara »

jacques_chester wrote: post #34
marknadal wrote: post #29

Dangit, I trust the CoreOS team more/better than a lot of people in the space. Torus would have been so useful. At the other end of the spectrum though, maybe this is reasonable? As a developer, my first thoughts for "I want my own S3" is not etcd (strong consistency) but projects like https://github.com/minio/minio , or even using eventually consistent SQLite replication / synchronization tools https://github.com/gu…

At a glance, Torus was intended to be a distributed file system. As I understand it, distributed file systems are easier than distributed block systems, but harder than distributed blob systems. A blob system is all-or-nothing. You create or replace the entire blob at once. This makes bookkeeping and replication much easier for the implementer. A filesystem supports much richer semantics, including the ability to see…

Disclosure: I work for OpenEBS project

Torus was intending to write distributed block storge that is container native. Metadata management using key value (KV / etcd) method is increases the complexity and not new. Ceph tried it.

OpenEBS uses a novel approch, linux sparse files for managing the blocks of a volume. Fork of Rancher longhorn. The issue of managing the large scale distributed block storage metadata is solved easily throught he management of the files (not blocks).

https://blog.openebs.io/torus-from-coreos-steps-aside-as-clo...

Re: Torus development has been stopped at CoreOS

#46

Post by epowell2017 »

wmf wrote: post #25
alrs wrote: post #22

I understand they don't hand out Internet points for this sort of thing anymore: https://news.ycombinator.com/item?id=11816821

Note that CoreOS has not given up on the concept of distributed storage; they just gave up on writing their own. So they haven't proved you right. I realize reliable block/file storage isn't "cloud native" but legacy apps require it and they are willing to spend billions to have it.

What does "cloud native" mean here? To me it suggests purpose built much as Torus was /is - though the OpenEBS engineers are asserting there are really only two "container native" storage solutions going, their open source project and PortWorx. https://medium.com/@kiranmova/persistent-storage-for-contain...

Re: Torus development has been stopped at CoreOS

#47

Post by fh973 »

jacques_chester wrote: post #34
marknadal wrote: post #29

Dangit, I trust the CoreOS team more/better than a lot of people in the space. Torus would have been so useful. At the other end of the spectrum though, maybe this is reasonable? As a developer, my first thoughts for "I want my own S3" is not etcd (strong consistency) but projects like https://github.com/minio/minio , or even using eventually consistent SQLite replication / synchronization tools https://github.com/gu…

At a glance, Torus was intended to be a distributed file system. As I understand it, distributed file systems are easier than distributed block systems, but harder than distributed blob systems. A blob system is all-or-nothing. You create or replace the entire blob at once. This makes bookkeeping and replication much easier for the implementer. A filesystem supports much richer semantics, including the ability to see…

Working on a distributed file system (Quobyte), I'd say the order of complexity is file > block > object. Why? Because a drop-in replacement for a local file system also needs to be good at high-performance concurrent block IO, otherwise you wouldn't be able to run applications like databases.

Re: Torus development has been stopped at CoreOS

#48

Post by fh973 »

umamukkara wrote: post #44
jacques_chester wrote: post #34

Earlier quoted context omitted.

At a glance, Torus was intended to be a distributed file system. As I understand it, distributed file systems are easier than distributed block systems, but harder than distributed blob systems. A blob system is all-or-nothing. You create or replace the entire blob at once. This makes bookkeeping and replication much easier for the implementer. A filesystem supports much richer semantics, including the ability to see…

Disclosure: I work for OpenEBS project Torus was intending to write distributed block storge that is container native. Metadata management using key value (KV / etcd) method is increases the complexity and not new. Ceph tried it. OpenEBS uses a novel approch, linux sparse files for managing the blocks of a volume. Fork of Rancher longhorn. The issue of managing the large scale distributed block storage metadata is so…

What I don't get around the new efforts around container-native storage: if you decide to build a new container-native storage system, why would you aim for block storage and not file storage?

Block storage is not exactly a great fit for containers as you can't access its file systems from multiple hosts and fail-over is a hassle (forced remount, fsck).

Re: Torus development has been stopped at CoreOS

#49

Post by fh973 »

parenthephobia wrote: post #16
chuhnk wrote: post #3

"But we didn't achieve the development velocity over the 8 months that we had hoped for when we started out, and as such we didn't achieve the depth of community engagement we had hoped for either." Open source is tough, even as a successful VC funded company. Gotta give credit to CoreOS though, rather than beating a dead horse they're acknowledging there's little external interest and their time would be better spen…

> Everyone is already struggling to understand the complexities of this new ecosystem and how the pieces all fit together I believe Google's (and Amazon's) secret sauce is having large sysops and devops teams filled with subject matter experts - and the software's original developers. Efforts like Torus are trying to build zero-to-low-maintenance turn-key solutions, whilst Google is happy to have tens of full-time di…

For Google at least the SRE teams for infrastructure components are not large, neither in total nor relative to the huge infrastructure that they are managing.

You could call this operational scalability, it is made possible by decoupling service quality from individual pieces of hardware. The key ingredient are redundancy across relatively large failure domains and auto mated handling of all foreseeable events. Everything is built around this concept, compute with a fault-tolerant container scheduler, storage on a fault-tolerant file system, checksuming everywhere,...

Another enabler is probably keeping things simple at the component level.

Re: Torus development has been stopped at CoreOS

#50

Post by wmf »

epowell2017 wrote: post #46
wmf wrote: post #25

Earlier quoted context omitted.

Note that CoreOS has not given up on the concept of distributed storage; they just gave up on writing their own. So they haven't proved you right. I realize reliable block/file storage isn't "cloud native" but legacy apps require it and they are willing to spend billions to have it.

What does "cloud native" mean here? To me it suggests purpose built much as Torus was /is - though the OpenEBS engineers are asserting there are really only two "container native" storage solutions going, their open source project and PortWorx. https://medium.com/@kiranmova/persistent-storage-for-contain...

As with all buzzwords, it means what people want it to mean. Many people say that cloud-native apps should use only ephemeral storage (probably because they don't provide reliable storage).
Post reply on HN