Why Is Storage on Kubernetes So Hard?
softwareengineeringdaily.com
Why Is Storage on Kubernetes So Hard?
1–10 of 92 posts
Re: Why Is Storage on Kubernetes So Hard?
#2I could not figure out cluster storage with docker swarm at least. With containers springing up and going away on multiple computers I could not figure out where the volumes were supposed to “live”
The only thing that at least seemed possible without additional software was nfs mounts but I was thinking it does not help the high availability cause to see the one and only storage node go down. Nor could I see any benefits to a cluster that used the same network for file access as external network requests. Say I wanted to build a horizontally scalable video sharing site I don’t see how I could test the real world performance of it before it went live.
Re: Why Is Storage on Kubernetes So Hard?
#3I saw many comments about stateful workloads. I am not sure it is a necessary issue for cloud environment.
Within a zone or a cluster, the latency is about 1ms, which is faster than most hard disks. The network bandwidth is on par with disk throughput. What we really need is a faster database and a faster object storage that can match the network performance (1ms and 10Gbps), then all workloads can be stateless.
If one uses a VM on GCP, the VM has no local storage besides the small local SSDs. Practically even the VM is stateless besides some cache.
Re: Why Is Storage on Kubernetes So Hard?
#4I could not figure out cluster storage with docker swarm at least. With containers springing up and going away on multiple computers I could not figure out where the volumes were supposed to “live” The only thing that at least seemed possible without additional software was nfs mounts but I was thinking it does not help the high availability cause to see the one and only storage node go down. Nor could I see any bene…
You could use multiple network interfaces, one for storage, the other for the rest.
Not sure if this is in a datacenter or not. In a datacenter, you could use something like 3-PAR to have network attached storage.
But storage is hard. This is one of the advantages of using cloud providers, they have this part figured out for you. AWS's EBS volumes are network-attached storage.
Then there's another layer of abstraction when you are using containers. Ok, so you have this volume accessible from the network now, in whatever form. How do you attach it to running containers? That's what this article is about, mostly. Not with the underlying storage mechanisms.
Re: Why Is Storage on Kubernetes So Hard?
#5Not sure I follow, the op is complaining that a tool is not the 100% replacement for an sro/pe? No shit...
Re: Why Is Storage on Kubernetes So Hard?
#6I saw many comments about stateful workloads. I am not sure it is a necessary issue for cloud environment. Within a zone or a cluster, the latency is about 1ms, which is faster than most hard disks. The network bandwidth is on par with disk throughput. What we really need is a faster database and a faster object storage that can match the network performance (1ms and 10Gbps), then all workloads can be stateless. If o…
> The network bandwidth is on par with disk throughput
Yes, and most storage you have access to, in cloud environments, is network attached. GCP disks, AWS EBS volumes, etc. All network and outside the hypervisors. You may have some local storage, but that's ephemeral, by design.
However, since we are talking about Kubernetes: not only VMs are ephemeral, but your containers are ephemeral too! And they can move around. So now you (or rather, K8s) need to figure out which worker node has the pod, and which storage is assigned to it, and then attach/detach accordingly.
This is what persistent volumes and persistent volume claims give you. They actually work fine already for StatefulSets.
Now, if you are in a cloud environment you should look into the possibility of using the hosted database offerings. If you can (even at a price premium), that's a great deal of complexity you are going to avoid.
Re: Why Is Storage on Kubernetes So Hard?
#7I don't really think this is a Kubernetes-specific problem. If you have a million machines, want your database to run on one of them that is selected by some upstream orchestrator, and want the physical SSDs with that data on it to be in the same machine, you're going to have to do some work. But at the same time, you have to realize that you are doing this to get that tiny last bit of performance (most likely that 99.9%-ile latency) and that that last 0.1% is always the most expensive. Sometimes you need it, and I get that, but it's not a problem that everyone has.
Most applications are not so IOPS limited that they depend on the difference between PCIE request latency and going out over the network to get stuff that's actually stored on a nearby rack. And in that case what Kubernetes offers (with the help of cloud providers) is fine. You make a storage class. You make a persistent volume claim. Your pod mounts that. Not all that hard. If the performance isn't good enough, though, then you have to build something yourself.
I am used to a completely different model that we had at Google. You could not get durable storage in your job allocation. Everything went through some other controller that did not give you a block device or even POSIX semantics, and you designed your app around that. If you needed more IOPS you talked to more backends and duplicated more data.
Meanwhile in the public cloud world, you get to have a physical block device with an ext4 filesystem that can magically appear in any of your 5 availability zones, provisioned on the type of disk you specify with a guaranteed number of IOPS. It's honestly pretty good for 90% or even 99% of the things people are using disks for. (In my last production environment I actually ran stuff like InfluxDB against EFS, the fully-managed POSIX filesystem that Amazon provides. It did fine.)
Re: Why Is Storage on Kubernetes So Hard?
#8I could not figure out cluster storage with docker swarm at least. With containers springing up and going away on multiple computers I could not figure out where the volumes were supposed to “live” The only thing that at least seemed possible without additional software was nfs mounts but I was thinking it does not help the high availability cause to see the one and only storage node go down. Nor could I see any bene…
You could use multiple network interfaces, one for storage, the other for the rest. Not sure if this is in a datacenter or not. In a datacenter, you could use something like 3-PAR to have network attached storage. But storage is hard. This is one of the advantages of using cloud providers, they have this part figured out for you. AWS's EBS volumes are network-attached storage. Then there's another layer of abstractio…
I am Leary of Amazon because of the potential for sudden ruinous expenses. I looked over digital ocean and they say they are working on kubernetes but I don’t see how block storage is going to make it work. I’m just developing this on my home server for now. I’m just trying to learn the basics. Seems like cluster computing isn’t something that can be roll your own like ordinary docker can be.
Thanks for the lead though.
Re: Why Is Storage on Kubernetes So Hard?
#9I saw many comments about stateful workloads. I am not sure it is a necessary issue for cloud environment. Within a zone or a cluster, the latency is about 1ms, which is faster than most hard disks. The network bandwidth is on par with disk throughput. What we really need is a faster database and a faster object storage that can match the network performance (1ms and 10Gbps), then all workloads can be stateless. If o…
With stateless services, forwarding requests to underlying storage and serialization can dominate resource consumption. After all, some services will do little besides fetch the right content and transform the data somehow.
Addressing this requires caching data in memory while making sure those caches are also disjoint so that you fully utilize your cluster memory. This has driven Google (and others) to make some services semi stateful and build dynamic sharing infrastructure to make this easier [1].
Re: Why Is Storage on Kubernetes So Hard?
#10Earlier quoted context omitted.
You could use multiple network interfaces, one for storage, the other for the rest. Not sure if this is in a datacenter or not. In a datacenter, you could use something like 3-PAR to have network attached storage. But storage is hard. This is one of the advantages of using cloud providers, they have this part figured out for you. AWS's EBS volumes are network-attached storage. Then there's another layer of abstractio…
I am Leary of Amazon because of the potential for sudden ruinous expenses. I looked over digital ocean and they say they are working on kubernetes but I don’t see how block storage is going to make it work. I’m just developing this on my home server for now. I’m just trying to learn the basics. Seems like cluster computing isn’t something that can be roll your own like ordinary docker can be. Thanks for the lead thou…
You can get a decent-sized cluster running free for a bit with Kubernetes Engine in Google Cloud, using the initial credits they give you. The cluster will shut off when the credits do expire.