Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Is Storage on Kubernetes So Hard?

softwareengineeringdaily.com

Post reply on HN
Print view View original

61–70 of 92 posts

Re: Why Is Storage on Kubernetes So Hard?

#61

Post by traek »

hinkley wrote: post #56
jrockway wrote: post #7

I don't really think this is a Kubernetes-specific problem. If you have a million machines, want your database to run on one of them that is selected by some upstream orchestrator, and want the physical SSDs with that data on it to be in the same machine, you're going to have to do some work. But at the same time, you have to realize that you are doing this to get that tiny last bit of performance (most likely that 9…

> and that that last 0.1% is always the most expensive This is one of the things that gets me. If you get the last .1% that can ever be gotten, what do you do for an encore? When your userbase grows another 5% or 10% or 20%, it won't be enough. You'd be better off trying to figure out ways to give your users something they want that doesn't require most exotic thing that can be procured. It's expensive to begin with,…

Nobody’s getting “the last .1% that can ever be gotten”, so it’s a moot point. Google and Facebook are still growing. There’s always room for more improvements.

Re: Why Is Storage on Kubernetes So Hard?

#62

Post by hinkley »

traek wrote: post #61
hinkley wrote: post #56

Earlier quoted context omitted.

> and that that last 0.1% is always the most expensive This is one of the things that gets me. If you get the last .1% that can ever be gotten, what do you do for an encore? When your userbase grows another 5% or 10% or 20%, it won't be enough. You'd be better off trying to figure out ways to give your users something they want that doesn't require most exotic thing that can be procured. It's expensive to begin with,…

Nobody’s getting “the last .1% that can ever be gotten”, so it’s a moot point. Google and Facebook are still growing. There’s always room for more improvements.

And they are both aggressively using horizontal scalability. What’s your point?

Re: Why Is Storage on Kubernetes So Hard?

#63

Post by manigandham »

a11595 wrote: post #52
SomeHacker44 wrote: post #25

Earlier quoted context omitted.

For databases, local storage beats SAN storage massively. I can get better performance and IOPS from an Intel NUC with a decent PCIe SSD than I can get from an AWS RDS instance that costs as much per month as the NUC did to buy. If I optimize a proper rack mount server for database I can get an insane amount and storage and performance compared to even a few months of RDS. Even a pair of 40 Gbps SAN links would not c…

Don't know where you are getting this from, but it's literally not how every enterprise-level company runs. You have something like and EMC Symmetrix all flash full of optane or nvme flash drives - hundreds of them. your data is striped across all of them. they're connected to something like a cisco mds. usually you have 4 paths to each LUN, and they're usually 32Gb FC. you can certainly have as many paths as you wan…

Leave out the last sentence next time, that attitude overshadows your entire comment and isn't a good fit for HN.

Re: Why Is Storage on Kubernetes So Hard?

#64

Post by jordanbeiber »

a11595 wrote: post #52
SomeHacker44 wrote: post #25

Earlier quoted context omitted.

For databases, local storage beats SAN storage massively. I can get better performance and IOPS from an Intel NUC with a decent PCIe SSD than I can get from an AWS RDS instance that costs as much per month as the NUC did to buy. If I optimize a proper rack mount server for database I can get an insane amount and storage and performance compared to even a few months of RDS. Even a pair of 40 Gbps SAN links would not c…

Don't know where you are getting this from, but it's literally not how every enterprise-level company runs. You have something like and EMC Symmetrix all flash full of optane or nvme flash drives - hundreds of them. your data is striped across all of them. they're connected to something like a cisco mds. usually you have 4 paths to each LUN, and they're usually 32Gb FC. you can certainly have as many paths as you wan…

The standing joke amongst my co-workers is that ”enterprise” in this context is the tier you reach when you have gotten ripped of enough by third party vendors.

You usual run of the mill enterprise use most of this horsepower to heat air and have large clusters of oracle db’s power even more expensive SAP modules.

There’s a reason no serious iaas provider build infra this way. I have one example, actually an old customer to the vendor that used to employ me, that had probably well over 1k of tennants all with VM’s and DB’s backed by humongous SAN (replicated ofc!).

Not nice when corrupt data was mirrored due to a software bug in said SAN system. This went to a national level, really, due to the nature of many of the customers.

Sure you can build a reasonable architecture using the products and vendors mentioned, but, it doesn’t scale and it really locks you in.

I’ve seen and worked this stuff at numerous enterprises.

This only makes it my experience and one datapoint, so please bear that in mind! =D

Re: Why Is Storage on Kubernetes So Hard?

#65

Post by KaiserPro »

SomeHacker44 wrote: post #25
jrockway wrote: post #7

I don't really think this is a Kubernetes-specific problem. If you have a million machines, want your database to run on one of them that is selected by some upstream orchestrator, and want the physical SSDs with that data on it to be in the same machine, you're going to have to do some work. But at the same time, you have to realize that you are doing this to get that tiny last bit of performance (most likely that 9…

For databases, local storage beats SAN storage massively. I can get better performance and IOPS from an Intel NUC with a decent PCIe SSD than I can get from an AWS RDS instance that costs as much per month as the NUC did to buy. If I optimize a proper rack mount server for database I can get an insane amount and storage and performance compared to even a few months of RDS. Even a pair of 40 Gbps SAN links would not c…

There are a few issues here.

EBS is a compromise, its allows your data to run anywhere in a region, on any machine. That means lots of hops and lots of interconnects.

the latest SAS stuff runs at 12 gigs(most likely 4 lanes per cable, and dual linked too.), _but_ thats dedicated for local traffic. The performance difference between having a SAS disk inside a box, or in the next rack is negligible. A decent SAN that exports over a 56gig connect (inifiniband et al) will be >> than a local pcie in terms of iops and sustained bandwith (at the expense of latency.)

Crucially, in somewhere that runs it's own datacenter, the storage is modelled for a specific workload. In VFX land we had 32 60 disk file servers, each capable of saturating two 10 gig ethernet links. But it'd be appalling for mixed VM hosting.

EBS is a hedged bet, it had a _boatload_ of caching to make any sort of performance. It had a huge amount of QoS to stop selfish loads stealing all the IOPs.

To get the best performance from EBS you have to have large volumes (5Tb+) and large latest generation instances (c5, r5, m5) 4 or 8/9 xlarge ec2 instances. This costs money to run.

Re: Why Is Storage on Kubernetes So Hard?

#66

Post by manigandham »

SomeHacker44 wrote: post #25
jrockway wrote: post #7

I don't really think this is a Kubernetes-specific problem. If you have a million machines, want your database to run on one of them that is selected by some upstream orchestrator, and want the physical SSDs with that data on it to be in the same machine, you're going to have to do some work. But at the same time, you have to realize that you are doing this to get that tiny last bit of performance (most likely that 9…

For databases, local storage beats SAN storage massively. I can get better performance and IOPS from an Intel NUC with a decent PCIe SSD than I can get from an AWS RDS instance that costs as much per month as the NUC did to buy. If I optimize a proper rack mount server for database I can get an insane amount and storage and performance compared to even a few months of RDS. Even a pair of 40 Gbps SAN links would not c…

There's a wide range of performance with storage systems but why are you comparing that to RDS? That's a managed database service that just runs on EC2 instances with EBS volumes, and EBS is designed for affordable and scalable persistence instead of extreme performance.

EBS is not your typical SAN, which usually offer much more throughput, IOPS, and reliability in exchange for more complexity and latency, however you probably won't even notice that latency if your SAN system is close enough and using highend links.

Re: Why Is Storage on Kubernetes So Hard?

#67

Post by KaiserPro »

A few observations here

o managing state is hard. Kubernetes makes it look simple because we have been moving the state from apps to databases/messaging queues o storing data durably at scale, with speed and consistently is a CAP problem o distributed block storage is slow, complex and resource hungry o distributed storage with metadata almost always has a metadata speed problem (again CAP) o managing your own high performance storage system is bankruptingly hard

Ceph/rook is almost certainly not the answer. Ceph has been pushed with openstack, which is a horridly mess of complexity. Ceph is slow hard to mange and eats resources. if you are running k8s on real steel, then use a SAN. if you're on AWS/google use the block primitives provided.

Firstly, there is no general storage backend that is a good fit for all workloads. Some things need low latency (either direct attached, or local SAN) some can cope with single instances of small EBS volumes. Some need shared block sorage, some need shared posix.

With AWS, there is no share block storage publicly available. Mapping volumes to random containers is pretty simple. Failing that there is EFS, which despite terrible metadata performance, kicks out a boat load of data. If you app can store all its state in one large file, this might be for you.

Google has the same, although I've not tried their new EFS/NFS service.

Re: Why Is Storage on Kubernetes So Hard?

#68

Post by pkphilip »

The old fashioned approach of using a database server in a co-hosted setup with a locally hosted storage on SSD with another couple of servers in active or standby type of replication is the best option for the db side.

The rest of the stateless compute cores can run on kubernetes.

Re: Why Is Storage on Kubernetes So Hard?

#69

Post by fh973 »

While storage generally not a simple problem, Kubernetes' design makes it even harder. Why Kubernetes went this way is not entirely clear to me. One of the reasons could be its owners' focus on cloud infrastructures and tutorial-level developer use cases, where you want to show how to dynamically create and consume a file system backed by GCE persistent disks or EBS. Also getting a secure and usable Kubernetes setup running is currently a lucrative business model.

But as the OP writes, the standard case it should support is secure access to existing storage, and Kubernetes fails there on all levels. Simple things should be simple, hard things should be possible.

In its API design, it hides storage under abstractions like PVC, PV, StorageClass that most users have not seen before in their career (where do they come from btw?) and which do not re-use general system architecture concepts. Worse, these abstractions do not have exact definitions as their semantics can be massaged with access control rules.

Consider the most simple case: mounting an existing file systems and authenticate with a Kubernetes secret. You should be able to do this in 3-4 lines in a pod defnition. Instead you need to create several yaml files, whose content is strongly dependent on how your Kubernetes was set up, so no general tutorial will help you (but others are, see above).

And this is just for getting the basics going. Secure access / user authentication is still unsolved after two years (https://www.quobyte.com/blog/2017/03/17/the-state-of-secure-...), and does not seem to be high on the agenda neither in Kubernetes nor CSI.

There are other basics missing (like file systems do not have necessarily a "size"), but let's not go into detail there.

Re: Why Is Storage on Kubernetes So Hard?

#70

Post by KaiserPro »

KaiserPro wrote: post #65
SomeHacker44 wrote: post #25

Earlier quoted context omitted.

For databases, local storage beats SAN storage massively. I can get better performance and IOPS from an Intel NUC with a decent PCIe SSD than I can get from an AWS RDS instance that costs as much per month as the NUC did to buy. If I optimize a proper rack mount server for database I can get an insane amount and storage and performance compared to even a few months of RDS. Even a pair of 40 Gbps SAN links would not c…

There are a few issues here. EBS is a compromise, its allows your data to run anywhere in a region, on any machine. That means lots of hops and lots of interconnects. the latest SAS stuff runs at 12 gigs(most likely 4 lanes per cable, and dual linked too.), _but_ thats dedicated for local traffic. The performance difference between having a SAS disk inside a box, or in the next rack is negligible. A decent SAN that e…

> its allows your data to run anywhere in a region

that should read AZ

Post reply on HN