Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Is Storage on Kubernetes So Hard?

softwareengineeringdaily.com

Post reply on HN
Print view View original

41–50 of 92 posts

Re: Why Is Storage on Kubernetes So Hard?

#41

Post by justinclift »

justinclift wrote: post #33
SomeHacker44 wrote: post #25

Earlier quoted context omitted.

For databases, local storage beats SAN storage massively. I can get better performance and IOPS from an Intel NUC with a decent PCIe SSD than I can get from an AWS RDS instance that costs as much per month as the NUC did to buy. If I optimize a proper rack mount server for database I can get an insane amount and storage and performance compared to even a few months of RDS. Even a pair of 40 Gbps SAN links would not c…

> local storage beats SAN storage massively Only if your (storage) network is slow (eg 1/10/25 GbE). With a decent network (eg modern infiniband, 40+GbE, etc) for the storage, the latency and throughput to the storage shouldn't make a difference. For example (years ago), I used to set up SSD arrays - SATA at the time, as M.2 wasn't a thing - and have them served over a 20Gbs Infiniband network to hosts in the same da…

For reference, the SCST project was the software used (back-in-the-day) for high performance block storage over infiniband:

http://scst.sourceforge.net

Looks like it's still an active project too, as there's a new release listed from November 2018.

Re: Why Is Storage on Kubernetes So Hard?

#42

Post by justinclift »

hardwaresofton wrote: post #21

Kubernetes has a few good storage solutions that are built on well-known technology... https://rook.io (built on ceph) https://www.openebs.io (built on Jiva/cStor) The distributed storage problem is difficult on it's own, this isn't a kubernetes specific issue. IMO Kubernetes is improving on the state of the art by dealing with both the distributed storage problem and dynamic provisioning, and that's why it might see…

OpenEBS looked interesting, but it turns out they're spammers. :(

Starred the GitHub repo... and a few minutes later received email spam from them to my personal email (it's in my GitHub profile). :(

Completely lost interest in their project at that point.

Probably best to skip it, as rewarding spammers doesn't lead to good things. :(

Re: Why Is Storage on Kubernetes So Hard?

#43

Post by kthejoker2 »

Netapp has Kubernetes-as-a-Service that includes their own persistent volume. As a consultant I had it foisted on me for a project, but it worked pretty well - I never had to worry about storage - it just worked - so I guess I never realized it was "hard"? Definitely something I'm glad I let someone else manage.

https://cloud.netapp.com/kubernetes-service

Re: Why Is Storage on Kubernetes So Hard?

#44

Post by snuxoll »

SomeHacker44 wrote: post #27
devereaux wrote: post #11

Earlier quoted context omitted.

I agree that 90% of people do not need that, but there is a scale between 1 machine and 1 million machines. It is not just IOPS but latency. Some of my production servers are connected one-to-one by crossover cable, in clusters with as many extra nics as needed (say 3 machines: 2 extra nics per machine) just to shave that little extra overhead of going through a router as yes, it matters in some applications!

Try using fiber then. I was surprised and pleased to learn that my 10G fiber has something like a third or quarter of the latency of my 10G copper in my new house.

You mean use (X)SFP(+), not “use fiber”. The SFP+ copper twinax cables I use in my lab at home have lower latency than the majority of MMF/SMF transceivers on the market.

Re: Why Is Storage on Kubernetes So Hard?

#45

Post by steelframe »

candiodari wrote: post #28
nemothekid wrote: post #17

Earlier quoted context omitted.

AFAICT, the problem is have a separate, generic, distributed storage layer that is also performant is a solution very few have (like Google), and the rest of us have tried to hack things together with StatefulSets, GlusterFS, NFS, Persistent Volumes, etc. The Borg/Omega model kind of assumes you have a Google-like storage tier interconnected with 10GbE links that is automatically replicated and accessible from potent…

Unless I totally misread the "colossus" papers, Google does not in fact have an "accessible from anywhere" storage layer. GFS/Colossus is a per-cluster filesystem capable of read and append only. It is very much NOT generic and only supports custom applications. Presumably that means that on every Google machine there is a directory that is shared for the entire cluster BUT: 1) It is per-cluster, not global (given Go…

Would regional Persistent Disks be acceptable?

https://cloud.google.com/compute/docs/disks/#repds

Global might mean you're accessing blocks stored in Europe from a container running in Australia. If your workload is doing that, you might want to consider network costs and latency, and then reconsider whether you really want to architect things that way.

Re: Why Is Storage on Kubernetes So Hard?

#46

Post by zbentley »

latch wrote: post #38
YZF wrote: post #29

Earlier quoted context omitted.

I'm not sure this is necessarily true. 2 x 40Gbps is bandwidth which is typically not the limiting factor. If it was you can go with higher bandwidth like FC. RDS is a database service, not SAN. Look at SAN boxes from storage companies, you can get things like 60 SSDs in a single box. To match the IOPs of that you'd need a lot of servers with local storage. I think in general dis-aggregating compute and storage is th…

If you're doing it locally, you'd ideally use RAID with a battery backed unit and your fsync essentially end up going to memory.

Until a battery relearn takes your app down in the middle of the night. I don't miss those.

Re: Why Is Storage on Kubernetes So Hard?

#47

Post by paulddraper »

latch wrote: post #35
dilyevsky wrote: post #34

Earlier quoted context omitted.

If you’re ever planing on running at scale (and I assume most startups/projects strive for it), then there’s no point (other than maybe a rough poc) at which sharding is “premature” because later on it most likely will be prohibitively expensive.

Picking sharding specifically is weird given the many problem spaces where it won't work. Also, whatever unnecessary scaling solution you come up with today for the problems you have today is not likely to be applicable to the problems you have years down the road; features will have changed, apps will be rewritten. I think people underestimate how much a handful of properly specd servers can achieve. My best experie…

> I think people underestimate how much a handful of properly specd servers can achieve.

Absolutely.

> Quick search

Even AWS (i.e. the very pricey one) has an x1e.2xlarge with 4.5 physical (9 hyperthreaded) Xenon E7 8880 v3 processors, 244GB DDR4, 240GB SSD, 25 Gbps network for $1200/month paid monthly or $700/month paid annually.

And that scales linearly 16x.

You've really got to be something amazing (maybe a ton of video?) to scale past what a single commodity box can handle.

Re: Why Is Storage on Kubernetes So Hard?

#48

Post by sharms »

latch wrote: post #38
YZF wrote: post #29

Earlier quoted context omitted.

I'm not sure this is necessarily true. 2 x 40Gbps is bandwidth which is typically not the limiting factor. If it was you can go with higher bandwidth like FC. RDS is a database service, not SAN. Look at SAN boxes from storage companies, you can get things like 60 SSDs in a single box. To match the IOPs of that you'd need a lot of servers with local storage. I think in general dis-aggregating compute and storage is th…

If you're doing it locally, you'd ideally use RAID with a battery backed unit and your fsync essentially end up going to memory.

The servers I buy now have large capacitors which don't fail like batteries, and you can then have the server on a UPS

Re: Why Is Storage on Kubernetes So Hard?

#49

Post by hardwaresofton »

justinclift wrote: post #42
hardwaresofton wrote: post #21

Kubernetes has a few good storage solutions that are built on well-known technology... https://rook.io (built on ceph) https://www.openebs.io (built on Jiva/cStor) The distributed storage problem is difficult on it's own, this isn't a kubernetes specific issue. IMO Kubernetes is improving on the state of the art by dealing with both the distributed storage problem and dynamic provisioning, and that's why it might see…

OpenEBS looked interesting, but it turns out they're spammers. :( Starred the GitHub repo... and a few minutes later received email spam from them to my personal email (it's in my GitHub profile). :( Completely lost interest in their project at that point. Probably best to skip it, as rewarding spammers doesn't lead to good things. :(

I had no idea they did that, just made an issue about it[0].

Also, I wouldn't be so quick to write them off -- their solution is based on Container Attached Storage (CAS), and is the only relatively mature solution so far I've seen (I haven't seen any others that do CAS) that sort of take the Ceph model and turn it inside out -- pods talk to volumes over iSCSI via "controller" pods, and writes are replicated amongst these controller pods (controller pods have anti-affinity to ensure they end up on separate machines).

I have yet to do any performance testing on ceph vs openebs but I can tell you it was easier to wrap my head around than Ceph (though of course ceph is a pretty robust system), and way easier to debug/trace through the system.

[0]: https://github.com/openebs/openebs/issues/2345

Re: Why Is Storage on Kubernetes So Hard?

#50

Post by wora »

lostmyoldone wrote: post #18
wora wrote: post #3

I saw many comments about stateful workloads. I am not sure it is a necessary issue for cloud environment. Within a zone or a cluster, the latency is about 1ms, which is faster than most hard disks. The network bandwidth is on par with disk throughput. What we really need is a faster database and a faster object storage that can match the network performance (1ms and 10Gbps), then all workloads can be stateless. If o…

From a database perspective, 1ms to disk is an eternity. A good disk subsystem had less write latency than that in the early 90’s.

Write to disk has no practical latency because of write buffer, either local file system or remote database. Flush to disk would be slow unless you use SSD.

On the other hand, a single machine has limited reliability. If one wants to have high availability, they needs to dual write to another machine, which also has network latency.

Post reply on HN