Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Is Storage on Kubernetes So Hard?

softwareengineeringdaily.com

Post reply on HN
Print view View original

21–30 of 92 posts

Re: Why Is Storage on Kubernetes So Hard?

#21

Post by hardwaresofton »

Kubernetes has a few good storage solutions that are built on well-known technology...

https://rook.io (built on ceph)

https://www.openebs.io (built on Jiva/cStor)

The distributed storage problem is difficult on it's own, this isn't a kubernetes specific issue. IMO Kubernetes is improving on the state of the art by dealing with both the distributed storage problem and dynamic provisioning, and that's why it might seem somewhat flustered.

What isn't shown here is how effortless it feels when you do have a solution like rook or openEBS in place -- we've never seen ergonomics like this before for deploying applications. Also, the CSI (Container Storage Interface) that they're building and refining as they go will be extremely valuable to the community going forward.

BTW, if you want to do things in a static provisioning sense, support for local volumes (and hostPaths) have been around for a very long time -- just use those and handle your storage how you would have handled it before kubernetes existed.

Shameless plug I've also written about this on a fair number of occasions:

https://vadosware.io/post/kicking-the-tires-on-openebs-for-c...

https://vadosware.io/post/disassembling-raid-on-hetzner-with...

I've gone from a hostPath -> Rook (Ceph) -> hostPath (undoing RAID had issues) -> OpenEBS, and now I have easy to spin up, dynamic storage & resiliency on my tiny kubernetes cluster.

All of this being capable as someone who is not a sysadmin by trade, should not be understated. The bar is being lowered -- I learned enough about ceph to be dangerous, enough about openebs to be dangerous, and got resiliency and ease of use/integration from kubernetes.

Re: Why Is Storage on Kubernetes So Hard?

#23

Post by 0xEFF »

lostmyoldone wrote: post #18
wora wrote: post #3

I saw many comments about stateful workloads. I am not sure it is a necessary issue for cloud environment. Within a zone or a cluster, the latency is about 1ms, which is faster than most hard disks. The network bandwidth is on par with disk throughput. What we really need is a faster database and a faster object storage that can match the network performance (1ms and 10Gbps), then all workloads can be stateless. If o…

From a database perspective, 1ms to disk is an eternity. A good disk subsystem had less write latency than that in the early 90’s.

I don’t think that’s true. I recall ~5ms seek times being top of the line.

Re: Why Is Storage on Kubernetes So Hard?

#24

Post by parasubvert »

Storage itself is a complicated problem domain, one that is easily dismissed until you actually have to deal with it in gory detail.

K8s grew organically here, and thus the seams show. It’s getting better every release, particularly being able able to dynamically provision and schedule storage to pods in a “zone aware” manner, which has been tricky.

The deeper issue issue is we are spoiled for choice on storage engines. Using Ceph for random access r/w low latency block storage I wouldn’t wish on my worst enemy, for example. But it’s hard to distinguish hard numbers for comparative purposes.

Re: Why Is Storage on Kubernetes So Hard?

#25

Post by SomeHacker44 »

jrockway wrote: post #7

I don't really think this is a Kubernetes-specific problem. If you have a million machines, want your database to run on one of them that is selected by some upstream orchestrator, and want the physical SSDs with that data on it to be in the same machine, you're going to have to do some work. But at the same time, you have to realize that you are doing this to get that tiny last bit of performance (most likely that 9…

For databases, local storage beats SAN storage massively. I can get better performance and IOPS from an Intel NUC with a decent PCIe SSD than I can get from an AWS RDS instance that costs as much per month as the NUC did to buy. If I optimize a proper rack mount server for database I can get an insane amount and storage and performance compared to even a few months of RDS. Even a pair of 40 Gbps SAN links would not compare.

Re: Why Is Storage on Kubernetes So Hard?

#26

Post by latch »

notacoward wrote: post #14

Here's the root of the problem. > Static provisioning also goes against the mindset of Kubernetes Then the mindset of Kubernetes is wrong. Or at least incomplete. Persistent data is an essential part of computing. In some ways it's the most important part. You could swap out every compute element in your system and be running exactly the way you were very quickly. Now try it with your storage elements. Oops, screwed.…

The tail wagging the dog that is the last 5 years of devops advancement. Constraints imposed on software development to achieve reproducibility.

The most obvious example of this is the need to prematurely scale horizontally and deal with ephemeraliy (even compute).

Re: Why Is Storage on Kubernetes So Hard?

#27

Post by SomeHacker44 »

devereaux wrote: post #11
jrockway wrote: post #7

I don't really think this is a Kubernetes-specific problem. If you have a million machines, want your database to run on one of them that is selected by some upstream orchestrator, and want the physical SSDs with that data on it to be in the same machine, you're going to have to do some work. But at the same time, you have to realize that you are doing this to get that tiny last bit of performance (most likely that 9…

I agree that 90% of people do not need that, but there is a scale between 1 machine and 1 million machines. It is not just IOPS but latency. Some of my production servers are connected one-to-one by crossover cable, in clusters with as many extra nics as needed (say 3 machines: 2 extra nics per machine) just to shave that little extra overhead of going through a router as yes, it matters in some applications!

Try using fiber then. I was surprised and pleased to learn that my 10G fiber has something like a third or quarter of the latency of my 10G copper in my new house.

Re: Why Is Storage on Kubernetes So Hard?

#28

Post by candiodari »

nemothekid wrote: post #17
notacoward wrote: post #14

Here's the root of the problem. > Static provisioning also goes against the mindset of Kubernetes Then the mindset of Kubernetes is wrong. Or at least incomplete. Persistent data is an essential part of computing. In some ways it's the most important part. You could swap out every compute element in your system and be running exactly the way you were very quickly. Now try it with your storage elements. Oops, screwed.…

AFAICT, the problem is have a separate, generic, distributed storage layer that is also performant is a solution very few have (like Google), and the rest of us have tried to hack things together with StatefulSets, GlusterFS, NFS, Persistent Volumes, etc. The Borg/Omega model kind of assumes you have a Google-like storage tier interconnected with 10GbE links that is automatically replicated and accessible from potent…

Unless I totally misread the "colossus" papers, Google does not in fact have an "accessible from anywhere" storage layer. GFS/Colossus is a per-cluster filesystem capable of read and append only. It is very much NOT generic and only supports custom applications.

Presumably that means that on every Google machine there is a directory that is shared for the entire cluster BUT:

1) It is per-cluster, not global (given Google's cluster sizes one imagine's that's not that much of a limitation, but then again, it seems like that is a clear limit)

2) you can only create or append to files (or delete them I guess) (and this is therefore non-posix, and does not support things like mysql or postgres)

3) this is very different from what GlusterFS, NFS, persistent volumes, etc provide. Therefore disks on google cloud are presumably very much not just files on this GFS/colossus thing.

4b) it was single master at least until 2004. Maybe until 2010. Apparently that can work.

https://cloud.google.com/files/storage_architecture_and_chal...

https://static.googleusercontent.com/media/research.google.c...

Re: Why Is Storage on Kubernetes So Hard?

#29

Post by YZF »

SomeHacker44 wrote: post #25
jrockway wrote: post #7

I don't really think this is a Kubernetes-specific problem. If you have a million machines, want your database to run on one of them that is selected by some upstream orchestrator, and want the physical SSDs with that data on it to be in the same machine, you're going to have to do some work. But at the same time, you have to realize that you are doing this to get that tiny last bit of performance (most likely that 9…

For databases, local storage beats SAN storage massively. I can get better performance and IOPS from an Intel NUC with a decent PCIe SSD than I can get from an AWS RDS instance that costs as much per month as the NUC did to buy. If I optimize a proper rack mount server for database I can get an insane amount and storage and performance compared to even a few months of RDS. Even a pair of 40 Gbps SAN links would not c…

I'm not sure this is necessarily true. 2 x 40Gbps is bandwidth which is typically not the limiting factor. If it was you can go with higher bandwidth like FC. RDS is a database service, not SAN. Look at SAN boxes from storage companies, you can get things like 60 SSDs in a single box. To match the IOPs of that you'd need a lot of servers with local storage. I think in general dis-aggregating compute and storage is the optimal approach, whether some particular solution is better/price effective is a different question. Having all your drives in a box means they're easier to share, easier to service, easier to replace the servers as well ... lots of wins there.

Re: Why Is Storage on Kubernetes So Hard?

#30

Post by philsnow »

candiodari wrote: post #28
nemothekid wrote: post #17

Earlier quoted context omitted.

AFAICT, the problem is have a separate, generic, distributed storage layer that is also performant is a solution very few have (like Google), and the rest of us have tried to hack things together with StatefulSets, GlusterFS, NFS, Persistent Volumes, etc. The Borg/Omega model kind of assumes you have a Google-like storage tier interconnected with 10GbE links that is automatically replicated and accessible from potent…

Unless I totally misread the "colossus" papers, Google does not in fact have an "accessible from anywhere" storage layer. GFS/Colossus is a per-cluster filesystem capable of read and append only. It is very much NOT generic and only supports custom applications. Presumably that means that on every Google machine there is a directory that is shared for the entire cluster BUT: 1) It is per-cluster, not global (given Go…

> GFS/Colossus is a per-cluster filesystem capable of read and append only. It is very much NOT generic and only supports custom applications.

For the most part, google only has custom applications. Just about everything is written in-house, and takes advantage of things like being able to open a file from the local disk file just as easily as opening one in Colossus or their equivalent of Zookeeper.

Post reply on HN