Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Is Storage on Kubernetes So Hard?

softwareengineeringdaily.com

Post reply on HN
Print view View original

71–80 of 92 posts

Re: Why Is Storage on Kubernetes So Hard?

#71

Post by huffmsa » 

  Why is it so hard to hammer
  a nail with a screwdriver?
Because stateful storage isn't the problem the system was developed to solve. The author is conflating what K8s is (a stateless container orchestrator) with what he wants it to be (a full service devops guy).

Now, if you simply must have stateful storage, I've had a pretty good time with PVCs pointing to Plain Jane NFS volumes combined with node tainting and performing local replication of high need data on the pods as needed.

If your pods are scaling up and down or dying so quickly that this seems untenable, you have other fish to fry.

Re: Why Is Storage on Kubernetes So Hard?

#72

Post by pas »

hinkley wrote: post #62
traek wrote: post #61

Earlier quoted context omitted.

Nobody’s getting “the last .1% that can ever be gotten”, so it’s a moot point. Google and Facebook are still growing. There’s always room for more improvements.

And they are both aggressively using horizontal scalability. What’s your point?

That these kind of micro optimizations are worth it. They reduce bulk and tail latency, and probably make the system more load-bearing. Of course, since now the system depends on a more "lucky" allocation of resources, it has more failure cases, thus a tighter operational envelope.

Re: Why Is Storage on Kubernetes So Hard?

#73

Post by pas »

KaiserPro wrote: post #65
SomeHacker44 wrote: post #25

Earlier quoted context omitted.

For databases, local storage beats SAN storage massively. I can get better performance and IOPS from an Intel NUC with a decent PCIe SSD than I can get from an AWS RDS instance that costs as much per month as the NUC did to buy. If I optimize a proper rack mount server for database I can get an insane amount and storage and performance compared to even a few months of RDS. Even a pair of 40 Gbps SAN links would not c…

There are a few issues here. EBS is a compromise, its allows your data to run anywhere in a region, on any machine. That means lots of hops and lots of interconnects. the latest SAS stuff runs at 12 gigs(most likely 4 lanes per cable, and dual linked too.), _but_ thats dedicated for local traffic. The performance difference between having a SAS disk inside a box, or in the next rack is negligible. A decent SAN that e…

This sort of portability is usually handled by the DB routing layer in cloud native ("webscale") database deployments. eg in Cassandra.

of course there are a lot of systems where one MySQL DB instance is needed and sufficient for the forseeable future. so that's a very static resource allocation one VM on a hypervisor with local disk, (and with backups every day), makes a lot of smaller sites very happy, and if they get the local provisioned performance instead of the EBS variant they will be happier for longer.

of course, EBS is a lot more fault tolerant.

Re: Why Is Storage on Kubernetes So Hard?

#74

Post by jjirsa »

ThePhysicist wrote: post #60
jjirsa wrote: post #59

Earlier quoted context omitted.

A few million DAU is not at scale. A few million concurrent active users might be, but really, “at scale” has no meaning other than “we got to the point where shit gets hard”. And that point is different for every app. I remember watching a big xbox/ps3 game launch crash and burn when 16 of the biggest MySQL servers we could buy couldn’t keep up. That was two jobs ago and embarrassing (and probably expensive). That w…

Really curious what kind of problem requires 100.000s of DB servers, and what database system is actually capable of such massive horizontal scaling? Are you able to provide any specifics? I imagine that typically a single instance can serve at least 100-1000 concurrent users, meaning you have about 10 to 100.000.000 concurrent users on the system? Since you mentioned gaming I assume it's a massive multiplayer-game l…

Gaming was two jobs ago. It’s apache Cassandra, and I work for a large tech company.

Re: Why Is Storage on Kubernetes So Hard?

#75

Post by nemothekid »

candiodari wrote: post #28
nemothekid wrote: post #17

Earlier quoted context omitted.

AFAICT, the problem is have a separate, generic, distributed storage layer that is also performant is a solution very few have (like Google), and the rest of us have tried to hack things together with StatefulSets, GlusterFS, NFS, Persistent Volumes, etc. The Borg/Omega model kind of assumes you have a Google-like storage tier interconnected with 10GbE links that is automatically replicated and accessible from potent…

Unless I totally misread the "colossus" papers, Google does not in fact have an "accessible from anywhere" storage layer. GFS/Colossus is a per-cluster filesystem capable of read and append only. It is very much NOT generic and only supports custom applications. Presumably that means that on every Google machine there is a directory that is shared for the entire cluster BUT: 1) It is per-cluster, not global (given Go…

You are entirely correct, Colossus isn't POSIX. However, the solution Google has, is since they control all the software, essentially everything they write acts as if Colossus is their storage layer.

This "solves" the persistent storage layer in a way Kubernetes cannot. Inside Google you might deploy your "Spanner" database container which knows how to interface with Colossus and doesn't require any special setup or colocation (below the cluster level). You cannot deploy MySQL on K8s and expect the same.

Re: Why Is Storage on Kubernetes So Hard?

#76

Post by hinkley »

pas wrote: post #72
hinkley wrote: post #62

Earlier quoted context omitted.

And they are both aggressively using horizontal scalability. What’s your point?

That these kind of micro optimizations are worth it. They reduce bulk and tail latency, and probably make the system more load-bearing. Of course, since now the system depends on a more "lucky" allocation of resources, it has more failure cases, thus a tighter operational envelope.

You keep implying that Facebook and Google are making these micro-optimizations with no support. I don't think you can support them, which means your premise is flawed and your conclusions are wrong.

They have ridiculous numbers of servers. Rather than fiddling to reduce server count, they're improving their ability to scale out. It's cheaper and it's repeatable. Doing crazy things like building custom hardware and power distribution buses (Facebook) to reduce heat in the data centers.

Re: Why Is Storage on Kubernetes So Hard?

#77

Post by tw04 »

SomeHacker44 wrote: post #25
jrockway wrote: post #7

I don't really think this is a Kubernetes-specific problem. If you have a million machines, want your database to run on one of them that is selected by some upstream orchestrator, and want the physical SSDs with that data on it to be in the same machine, you're going to have to do some work. But at the same time, you have to realize that you are doing this to get that tiny last bit of performance (most likely that 9…

For databases, local storage beats SAN storage massively. I can get better performance and IOPS from an Intel NUC with a decent PCIe SSD than I can get from an AWS RDS instance that costs as much per month as the NUC did to buy. If I optimize a proper rack mount server for database I can get an insane amount and storage and performance compared to even a few months of RDS. Even a pair of 40 Gbps SAN links would not c…

That has almost nothing to do with SAN and everything to do with Amazon's implementation. A nuc will never give you the resiliency or performance of a properly sized SAN.

Furthermore, you'll never get close to saturating a 40gbps link with a database workload, your limitation is iops, not throughput, and you don't have anywhere near enough cpu to max out those links in a loaded server much less a nuc.

Re: Why Is Storage on Kubernetes So Hard?

#78

Post by SomeHacker44 »

snuxoll wrote: post #44
SomeHacker44 wrote: post #27

Earlier quoted context omitted.

Try using fiber then. I was surprised and pleased to learn that my 10G fiber has something like a third or quarter of the latency of my 10G copper in my new house.

You mean use (X)SFP(+), not “use fiber”. The SFP+ copper twinax cables I use in my lab at home have lower latency than the majority of MMF/SMF transceivers on the market.

Yes, thank you for the clarification/nomenclature. My SFP+ on MMF (multi-mode fiber) has significantly lower latency than the copper 10GbaseT I use. I haven't measured the direct-connect copper (coax? twinax?) SFP+ latency, though.

I would be curious if the SFP for 1G using fiber also has lower latency than copper 1000baseT.

Re: Why Is Storage on Kubernetes So Hard?

#79

Post by a11595 »

manigandham wrote: post #63
a11595 wrote: post #52

Earlier quoted context omitted.

Don't know where you are getting this from, but it's literally not how every enterprise-level company runs. You have something like and EMC Symmetrix all flash full of optane or nvme flash drives - hundreds of them. your data is striped across all of them. they're connected to something like a cisco mds. usually you have 4 paths to each LUN, and they're usually 32Gb FC. you can certainly have as many paths as you wan…

Leave out the last sentence next time, that attitude overshadows your entire comment and isn't a good fit for HN.

Not a chance buddy. I've been here for a while, and this place has become reddit, with mods from /r/incels. I have two accounts. When I see someone with zero knowledge of subject matter, like OP, authoritatively and dismissively spew complete bs, I reply like I do.

People with a clear agenda spewing fake crap gets upvoted here now. Disagree with a mod - your comments are shadow removed, and possibly your account is shadow banned, with no warning, so the insecure mod can feel better.

I give actual valuable information from 20 years of experience at almost all the fortune100 firms. Snarkness? Yes. Much better and funnier than the guy I replied to.

I know you disagree. That's specifically why I include sentences like the last - to draw attention to the issue. comment karma has become useless here. Stuff is greyed out, stuff is shadow deleted - good things, filtered by idiots. I don't want my content filtered by insecure idiots.

Re: Why Is Storage on Kubernetes So Hard?

#80

Post by a11595 »

jordanbeiber wrote: post #64
a11595 wrote: post #52

Earlier quoted context omitted.

Don't know where you are getting this from, but it's literally not how every enterprise-level company runs. You have something like and EMC Symmetrix all flash full of optane or nvme flash drives - hundreds of them. your data is striped across all of them. they're connected to something like a cisco mds. usually you have 4 paths to each LUN, and they're usually 32Gb FC. you can certainly have as many paths as you wan…

The standing joke amongst my co-workers is that ”enterprise” in this context is the tier you reach when you have gotten ripped of enough by third party vendors. You usual run of the mill enterprise use most of this horsepower to heat air and have large clusters of oracle db’s power even more expensive SAP modules. There’s a reason no serious iaas provider build infra this way. I have one example, actually an old cust…

they don't build it this way because cloud is cheap and slow. I have a UCS farm with 2000VMs, connected to 3PB of usable all-flash across 2 arrays, replicated active/active to another array, second-hopped to cloud. That's one site. About 100k users across 2 domains - 1k? I said enterprise. A SAN is not scalable but internal disk is? You need to look up what a SAN is.

The reason there is a standing joke with your coworkers is because you don't work on important things that run the world. When 1 minute of downtime costs you over $1mil, your "ripped off" Oracle costs, your SAN costs, etc, are lost in the rounding errors. I can have a SAN have hundreds of arrays, across multiple datacenters, and dynamically grow and shrink my storage needs. It is the definition of scalable. I can have one VM farm vmotion to another VM farm 30 miles away transparently, which will sit on a different array attached to the SAN. What happens when the storage needs of your server outgrow the drives you can shove in there? There are servers and databases a petabyte in size, pushing a million IOPS. They're in charge of money. If there is corruption on the SAN, well it infrequently happens, just like with servers and memory. Twice in my vast experience. You can roll back all writes on the replication software, and usually keep an undo journal a few days long, and something like hourly snapshots. This also protects against cryptoviruses and other types of corruption.

This is why people like you work at small companies, fiddling with your cute little projects, while the world moves forward with you on the sidelines. I've been doing this for 20+ years, and have been at most fortune100 companies, in 80 countries. But yeah, your opinion, while being dismissive, is cute.

The CIO is the guy in charge of getting this stuff at enterprises. It's clear you don't understand the impact of design. I am guessing you are not at an architect level. There's a reason for that, and a reason other people - the ones you put down in your post, are making the big decisions. People like you would cost the company millions of dollars in loss per year.

Post reply on HN