Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Is Storage on Kubernetes So Hard?

softwareengineeringdaily.com

Post reply on HN
Print view View original

11–20 of 92 posts

Re: Why Is Storage on Kubernetes So Hard?

#11

Post by devereaux »

jrockway wrote: post #7

I don't really think this is a Kubernetes-specific problem. If you have a million machines, want your database to run on one of them that is selected by some upstream orchestrator, and want the physical SSDs with that data on it to be in the same machine, you're going to have to do some work. But at the same time, you have to realize that you are doing this to get that tiny last bit of performance (most likely that 9…

I agree that 90% of people do not need that, but there is a scale between 1 machine and 1 million machines.

It is not just IOPS but latency. Some of my production servers are connected one-to-one by crossover cable, in clusters with as many extra nics as needed (say 3 machines: 2 extra nics per machine) just to shave that little extra overhead of going through a router as yes, it matters in some applications!

Re: Why Is Storage on Kubernetes So Hard?

#12

Post by rzzzt »

ohiovr wrote: post #2

I could not figure out cluster storage with docker swarm at least. With containers springing up and going away on multiple computers I could not figure out where the volumes were supposed to “live” The only thing that at least seemed possible without additional software was nfs mounts but I was thinking it does not help the high availability cause to see the one and only storage node go down. Nor could I see any bene…

Clustered filesystems like Ceph or GlusterFS might help with the reliability part, but latency and bandwidth still remain a problem.

The other thing I was quite interested in at one point was flocker, a volume manager which is responsible for replicating/migrating data along with the container it is connected to. The company shut down quite a while ago but code is still available: https://github.com/ClusterHQ/flocker

Re: Why Is Storage on Kubernetes So Hard?

#13

Post by halbritt »

I think the premise of this article is misleading. It's true that creating a storage subsystem in Kubernetes is exceedingly difficult, however using Kubernetes for stateful applications isn't any more difficult than using instances with persistent disk in any cloud provider.

For applications deployed in cloud infrastructure, most folks are using persistent disk simply for the ease of management. Some folks end up going to local disk for scale and performance reasons, at which point they end up having exactly the same problem one would have trying to do the same with Kubernetes.

Re: Why Is Storage on Kubernetes So Hard?

#14

Post by notacoward »

Here's the root of the problem.

> Static provisioning also goes against the mindset of Kubernetes

Then the mindset of Kubernetes is wrong. Or at least incomplete. Persistent data is an essential part of computing. In some ways it's the most important part. You could swap out every compute element in your system and be running exactly the way you were very quickly. Now try it with your storage elements. Oops, screwed. The data is the identity of your system.

The problem is that storage is not trivially relocatable like compute is, and yet every single orchestration system I've seen seems to assume otherwise. The people who write them develop models to represent the easy case, then come back to the harder one as an afterthought. A car that's really a boat with wheels bolted on isn't going to have great handling, but that's pretty much where we are with storage in Kubernetes.

Re: Why Is Storage on Kubernetes So Hard?

#15

Post by linkmotif »

Kind of feels like the author doesn’t know kubernetes? Stateful sets make all this very easy. I only have a user-end knowledge of k8s from GCE and it seems like k8s makes storage very easy.

EDIT K8s makes storage very easy. No “seems.”

Re: Why Is Storage on Kubernetes So Hard?

#17

Post by nemothekid »

notacoward wrote: post #14

Here's the root of the problem. > Static provisioning also goes against the mindset of Kubernetes Then the mindset of Kubernetes is wrong. Or at least incomplete. Persistent data is an essential part of computing. In some ways it's the most important part. You could swap out every compute element in your system and be running exactly the way you were very quickly. Now try it with your storage elements. Oops, screwed.…

AFAICT, the problem is have a separate, generic, distributed storage layer that is also performant is a solution very few have (like Google), and the rest of us have tried to hack things together with StatefulSets, GlusterFS, NFS, Persistent Volumes, etc.

The Borg/Omega model kind of assumes you have a Google-like storage tier interconnected with 10GbE links that is automatically replicated and accessible from potentially anywhere. Once you have that, then storage on k8s is "easy".

Re: Why Is Storage on Kubernetes So Hard?

#18

Post by lostmyoldone »

wora wrote: post #3

I saw many comments about stateful workloads. I am not sure it is a necessary issue for cloud environment. Within a zone or a cluster, the latency is about 1ms, which is faster than most hard disks. The network bandwidth is on par with disk throughput. What we really need is a faster database and a faster object storage that can match the network performance (1ms and 10Gbps), then all workloads can be stateless. If o…

From a database perspective, 1ms to disk is an eternity.

A good disk subsystem had less write latency than that in the early 90’s.

Re: Why Is Storage on Kubernetes So Hard?

#19

Post by rconti »

ohiovr wrote: post #2

I could not figure out cluster storage with docker swarm at least. With containers springing up and going away on multiple computers I could not figure out where the volumes were supposed to “live” The only thing that at least seemed possible without additional software was nfs mounts but I was thinking it does not help the high availability cause to see the one and only storage node go down. Nor could I see any bene…

I'm just afraid all of these volumes and resources I'm creating will be 100x easier to 'lose' and have total sprawl than it was in VM world, which already had enormous sprawl issues.

Re: Why Is Storage on Kubernetes So Hard?

#20

Post by LukeShu »

ohiovr wrote: post #8
outworlder wrote: post #4

Earlier quoted context omitted.

You could use multiple network interfaces, one for storage, the other for the rest. Not sure if this is in a datacenter or not. In a datacenter, you could use something like 3-PAR to have network attached storage. But storage is hard. This is one of the advantages of using cloud providers, they have this part figured out for you. AWS's EBS volumes are network-attached storage. Then there's another layer of abstractio…

I am Leary of Amazon because of the potential for sudden ruinous expenses. I looked over digital ocean and they say they are working on kubernetes but I don’t see how block storage is going to make it work. I’m just developing this on my home server for now. I’m just trying to learn the basics. Seems like cluster computing isn’t something that can be roll your own like ordinary docker can be. Thanks for the lead thou…

> I looked over digital ocean and they say they are working on kubernetes

FYI, Digital Ocean Kubernetes opened to everyone on December 11th.

Post reply on HN