Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Is Storage on Kubernetes So Hard?

softwareengineeringdaily.com

Post reply on HN
Print view View original

51–60 of 92 posts

Re: Why Is Storage on Kubernetes So Hard?

#51

Post by wora »

jrockway wrote: post #7

I don't really think this is a Kubernetes-specific problem. If you have a million machines, want your database to run on one of them that is selected by some upstream orchestrator, and want the physical SSDs with that data on it to be in the same machine, you're going to have to do some work. But at the same time, you have to realize that you are doing this to get that tiny last bit of performance (most likely that 9…

A large data center has 20-30k machines. Amazon disclosed that back in 2014. Most likely you want to partition the machine into some chunks, so each application or each database only use some of them. So you don't need to talk to many machines at once.

In general, a distributed database can be really fast. DynamoDB claim to be 3ms (AWS reinvent 2014). The current technology can probably achieve close to 1ms, which is equivalent to memcache. With such performance, you don't really need local storage.

You can get much faster IO if you use many local SSDs. The downside is utilization. It is very rare a single machine has a workload that fully utilize local disk. You end up over provision greatly fleet-wise to get high performance. A managed database over a network is more likely to utilize disk/SSD throughput.

Re: Why Is Storage on Kubernetes So Hard?

#52

Post by a11595 »

SomeHacker44 wrote: post #25
jrockway wrote: post #7

I don't really think this is a Kubernetes-specific problem. If you have a million machines, want your database to run on one of them that is selected by some upstream orchestrator, and want the physical SSDs with that data on it to be in the same machine, you're going to have to do some work. But at the same time, you have to realize that you are doing this to get that tiny last bit of performance (most likely that 9…

For databases, local storage beats SAN storage massively. I can get better performance and IOPS from an Intel NUC with a decent PCIe SSD than I can get from an AWS RDS instance that costs as much per month as the NUC did to buy. If I optimize a proper rack mount server for database I can get an insane amount and storage and performance compared to even a few months of RDS. Even a pair of 40 Gbps SAN links would not c…

Don't know where you are getting this from, but it's literally not how every enterprise-level company runs.

You have something like and EMC Symmetrix all flash full of optane or nvme flash drives - hundreds of them. your data is striped across all of them. they're connected to something like a cisco mds. usually you have 4 paths to each LUN, and they're usually 32Gb FC. you can certainly have as many paths as you want if you need more bandwidth. The array also has 10-20TB of cache to front-end the flash. your response time is about 100-200 microseconds.

I don't know how much ram or disk you're going to shove into your server, but good luck running a bunch of half-petabyte databases in it, and expanding it for growth.

this is literally how 90% the world runs. your little theories are cute though.

Re: Why Is Storage on Kubernetes So Hard?

#53

Post by justinclift »

hardwaresofton wrote: post #49
justinclift wrote: post #42

Earlier quoted context omitted.

OpenEBS looked interesting, but it turns out they're spammers. :( Starred the GitHub repo... and a few minutes later received email spam from them to my personal email (it's in my GitHub profile). :( Completely lost interest in their project at that point. Probably best to skip it, as rewarding spammers doesn't lead to good things. :(

I had no idea they did that, just made an issue about it[0]. Also, I wouldn't be so quick to write them off -- their solution is based on Container Attached Storage (CAS), and is the only relatively mature solution so far I've seen (I haven't seen any others that do CAS) that sort of take the Ceph model and turn it inside out -- pods talk to volumes over iSCSI via "controller" pods, and writes are replicated amongst…

Thanks for doing that. I have personal strong aversion to spammers. To the point where their product doesn't matter, I'll just use/help a competitor instead.

Reaching out to the occasional person who stars a project, if there's some strong overlap of stuff then maybe sure. An automatic spam approach though... that's not on. That makes starring projects a "dangerous" thing for end users, as they'd have to be open to emails for every one. :/

And yeah, it did look useful up until this point. Lets see what their response to the GH issue is like. :)

Re: Why Is Storage on Kubernetes So Hard?

#54

Post by traek »

justinclift wrote: post #53
hardwaresofton wrote: post #49

Earlier quoted context omitted.

I had no idea they did that, just made an issue about it[0]. Also, I wouldn't be so quick to write them off -- their solution is based on Container Attached Storage (CAS), and is the only relatively mature solution so far I've seen (I haven't seen any others that do CAS) that sort of take the Ceph model and turn it inside out -- pods talk to volumes over iSCSI via "controller" pods, and writes are replicated amongst…

Thanks for doing that. I have personal strong aversion to spammers. To the point where their product doesn't matter, I'll just use/help a competitor instead. Reaching out to the occasional person who stars a project, if there's some strong overlap of stuff then maybe sure. An automatic spam approach though... that's not on. That makes starring projects a "dangerous" thing for end users, as they'd have to be open to e…

They just replied saying they disabled the emails.

Re: Why Is Storage on Kubernetes So Hard?

#55

Post by umamukkara »

justinclift wrote: post #53
hardwaresofton wrote: post #49

Earlier quoted context omitted.

I had no idea they did that, just made an issue about it[0]. Also, I wouldn't be so quick to write them off -- their solution is based on Container Attached Storage (CAS), and is the only relatively mature solution so far I've seen (I haven't seen any others that do CAS) that sort of take the Ceph model and turn it inside out -- pods talk to volumes over iSCSI via "controller" pods, and writes are replicated amongst…

Thanks for doing that. I have personal strong aversion to spammers. To the point where their product doesn't matter, I'll just use/help a competitor instead. Reaching out to the occasional person who stars a project, if there's some strong overlap of stuff then maybe sure. An automatic spam approach though... that's not on. That makes starring projects a "dangerous" thing for end users, as they'd have to be open to e…

Thank you for the feedback. Feedback like this will help us understand the best practices in community building. We have disabled the email trigger on starring. Thanks again for this open feedback.

Re: Why Is Storage on Kubernetes So Hard?

#56

Post by hinkley »

jrockway wrote: post #7

I don't really think this is a Kubernetes-specific problem. If you have a million machines, want your database to run on one of them that is selected by some upstream orchestrator, and want the physical SSDs with that data on it to be in the same machine, you're going to have to do some work. But at the same time, you have to realize that you are doing this to get that tiny last bit of performance (most likely that 9…

> and that that last 0.1% is always the most expensive

This is one of the things that gets me. If you get the last .1% that can ever be gotten, what do you do for an encore?

When your userbase grows another 5% or 10% or 20%, it won't be enough. You'd be better off trying to figure out ways to give your users something they want that doesn't require most exotic thing that can be procured. It's expensive to begin with, and it's the end of the road. You don't want to be at the end of a road trying to figure out what to do next.

Re: Why Is Storage on Kubernetes So Hard?

#57

Post by justinclift »

umamukkara wrote: post #55
justinclift wrote: post #53

Earlier quoted context omitted.

Thanks for doing that. I have personal strong aversion to spammers. To the point where their product doesn't matter, I'll just use/help a competitor instead. Reaching out to the occasional person who stars a project, if there's some strong overlap of stuff then maybe sure. An automatic spam approach though... that's not on. That makes starring projects a "dangerous" thing for end users, as they'd have to be open to e…

Thank you for the feedback. Feedback like this will help us understand the best practices in community building. We have disabled the email trigger on starring. Thanks again for this open feedback.

No worries Uma. The project looks interesting, and it's even written in Go, so I'll probably take it for a spin in the next few weeks. :)

Re: Why Is Storage on Kubernetes So Hard?

#58

Post by nwmcsween »

jrockway wrote: post #7

I don't really think this is a Kubernetes-specific problem. If you have a million machines, want your database to run on one of them that is selected by some upstream orchestrator, and want the physical SSDs with that data on it to be in the same machine, you're going to have to do some work. But at the same time, you have to realize that you are doing this to get that tiny last bit of performance (most likely that 9…

Its more of an issue with the OS, *nix is showing its age as a useful OS as we are moving away from a static and simple system (PC's, servers, etc) to more distributed computing.

Re: Why Is Storage on Kubernetes So Hard?

#59

Post by jjirsa »

sbov wrote: post #40
dilyevsky wrote: post #34

Earlier quoted context omitted.

If you’re ever planing on running at scale (and I assume most startups/projects strive for it), then there’s no point (other than maybe a rough poc) at which sharding is “premature” because later on it most likely will be prohibitively expensive.

What do you mean by at scale? At what scale? In the past, I've supported a few million daily active users on a single MySQL database. Use cases obviously vary, your numbers may not line up. But I'm pretty sure there are lot of startups that would be fine starting with an architecture that can scale to 2 million users per day.

A few million DAU is not at scale. A few million concurrent active users might be, but really, “at scale” has no meaning other than “we got to the point where shit gets hard”. And that point is different for every app.

I remember watching a big xbox/ps3 game launch crash and burn when 16 of the biggest MySQL servers we could buy couldn’t keep up. That was two jobs ago and embarrassing (and probably expensive). That was “at scale” for us.

Last job we started at about 60 m4 ex2 instanced and were well into the thousands when I left. I suspect they’re approaching 10k instances now. And they’re pre-IPO startup, and I think that was at scale.

Current job measures in the hundreds of thousands of database instances, and I only count one specific database engine. Probably counts as at scale.

Re: Why Is Storage on Kubernetes So Hard?

#60

Post by ThePhysicist »

jjirsa wrote: post #59
sbov wrote: post #40

Earlier quoted context omitted.

What do you mean by at scale? At what scale? In the past, I've supported a few million daily active users on a single MySQL database. Use cases obviously vary, your numbers may not line up. But I'm pretty sure there are lot of startups that would be fine starting with an architecture that can scale to 2 million users per day.

A few million DAU is not at scale. A few million concurrent active users might be, but really, “at scale” has no meaning other than “we got to the point where shit gets hard”. And that point is different for every app. I remember watching a big xbox/ps3 game launch crash and burn when 16 of the biggest MySQL servers we could buy couldn’t keep up. That was two jobs ago and embarrassing (and probably expensive). That w…

Really curious what kind of problem requires 100.000s of DB servers, and what database system is actually capable of such massive horizontal scaling? Are you able to provide any specifics? I imagine that typically a single instance can serve at least 100-1000 concurrent users, meaning you have about 10 to 100.000.000 concurrent users on the system? Since you mentioned gaming I assume it's a massive multiplayer-game like Fortnite?
Post reply on HN