Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Is Storage on Kubernetes So Hard?

softwareengineeringdaily.com

Post reply on HN
Print view View original

81–90 of 92 posts

Re: Why Is Storage on Kubernetes So Hard?

#81

Post by jordanbeiber »

a11595 wrote: post #80
jordanbeiber wrote: post #64

Earlier quoted context omitted.

The standing joke amongst my co-workers is that ”enterprise” in this context is the tier you reach when you have gotten ripped of enough by third party vendors. You usual run of the mill enterprise use most of this horsepower to heat air and have large clusters of oracle db’s power even more expensive SAP modules. There’s a reason no serious iaas provider build infra this way. I have one example, actually an old cust…

they don't build it this way because cloud is cheap and slow. I have a UCS farm with 2000VMs, connected to 3PB of usable all-flash across 2 arrays, replicated active/active to another array, second-hopped to cloud. That's one site. About 100k users across 2 domains - 1k? I said enterprise. A SAN is not scalable but internal disk is? You need to look up what a SAN is. The reason there is a standing joke with your cowo…

I said tenants, not users. Tiny little detail that you overlooked there. =) Last company I worked for had a €150.000.000 tech budget and about ~90k employees.

Again, if you read what I said you'll notice that I pointed out that you actually can build decent architectures using enterprisey stuff, it will however never scale horizontally in a resonable way. What happens when you have filled all your shelves in the array box? Ah, you'll need a new $250K box...

Mentioning vmotion and all... yeah so... VSAN et al... brrr... I'll put my trust in the open source world rather than shoddy software from 3rd party vendors trying to appease to the latest fad. Hyperconverge my *ss! =)

KISS is what rules but vendors are busy feeding channels and partners with impossible to penetrate acronyms leaving you in a mess sooner or later.

Focus from these vendors is to keep integration at a minimum, which means API's are usually crappy and achieving a reasonable level of automation is often a chore.

Now there is probably a little percentage of "enterprises" that uses tech in a sane way, but my bet is that the CIO you mention is a blockchain expert, as well as putting AI at the top of strategic actions to "implement" this year.

He probably have commissioned a pre-study from Accenture that outlines these strategic imperatives.

This is of course a little rant, but also true for the 90% of the 90% mentioned.

Don't be defensive and close minded. That's what's leaving most enterprises in the dust. There's just a lot of inertia inherent within certain businesses that will let them continue to burn through cash on pointless tech for yet a while.

Re: Why Is Storage on Kubernetes So Hard?

#82

Post by jordanbeiber »

a11595 wrote: post #79
manigandham wrote: post #63

Earlier quoted context omitted.

Leave out the last sentence next time, that attitude overshadows your entire comment and isn't a good fit for HN.

Not a chance buddy. I've been here for a while, and this place has become reddit, with mods from /r/incels. I have two accounts. When I see someone with zero knowledge of subject matter, like OP, authoritatively and dismissively spew complete bs, I reply like I do. People with a clear agenda spewing fake crap gets upvoted here now. Disagree with a mod - your comments are shadow removed, and possibly your account is s…

Hey budz, many around here have 20+ years of experience.

I've fought all my career to get "enterprise" folks to understand that spending $100s of mill on hardware vendors and 3rd party software is madness, if what you are doing is actually mission critical.

Money should be spend on acquiring talent and adapting to actual business needs.

The business you support could not care less about your shiny toys, they want to transform, and probably yesteryear!

A digression here, from one of your other posts: if your business applications rely on live migration of vm's, you're doing it wrong. You should take this up with your CIO or even better the CTO, but he's probably busy looking at Gartner magic quadrants and contemplating the dire need of a blockchain.

Keep an open mind, and keep it simple. Best advice ever in this crazy business!

Re: Why Is Storage on Kubernetes So Hard?

#83

Post by pas »

hinkley wrote: post #76
pas wrote: post #72

Earlier quoted context omitted.

That these kind of micro optimizations are worth it. They reduce bulk and tail latency, and probably make the system more load-bearing. Of course, since now the system depends on a more "lucky" allocation of resources, it has more failure cases, thus a tighter operational envelope.

You keep implying that Facebook and Google are making these micro-optimizations with no support. I don't think you can support them, which means your premise is flawed and your conclusions are wrong. They have ridiculous numbers of servers. Rather than fiddling to reduce server count, they're improving their ability to scale out. It's cheaper and it's repeatable. Doing crazy things like building custom hardware and p…

Google does a lot of micro-optimizations.

There are services where latency is important, they launch multiple requests to different replicas but to optimize this when one of the replicas is serving the response it sends a request cancel to the other one.

Horizontal scaling and vertical scaling is both very important.

Both G and FB pours many engineering hours into maximizing per server "ROI". Just think about how much they work on scheduling work/tasks so they can do more with the same number of servers.

Re: Why Is Storage on Kubernetes So Hard?

#84

Post by a11595 »

jordanbeiber wrote: post #81
a11595 wrote: post #80

Earlier quoted context omitted.

they don't build it this way because cloud is cheap and slow. I have a UCS farm with 2000VMs, connected to 3PB of usable all-flash across 2 arrays, replicated active/active to another array, second-hopped to cloud. That's one site. About 100k users across 2 domains - 1k? I said enterprise. A SAN is not scalable but internal disk is? You need to look up what a SAN is. The reason there is a standing joke with your cowo…

I said tenants, not users. Tiny little detail that you overlooked there. =) Last company I worked for had a €150.000.000 tech budget and about ~90k employees. Again, if you read what I said you'll notice that I pointed out that you actually can build decent architectures using enterprisey stuff, it will however never scale horizontally in a resonable way. What happens when you have filled all your shelves in the arra…

again, cute. $250K. very cute. Try $5mil for a box, at least, after a 50% vendor discount. VSAN has nothing to do with SAN, and no one on a SAN uses VSAN. You are again showing your lack of enterprise experience, yet you are strongly trashing what you don't understand. A server sees many arrays, over a SAN. That is called horizontal scaling. You can't do that with internal disk.

Re: Why Is Storage on Kubernetes So Hard?

#85

Post by a11595 »

jordanbeiber wrote: post #82
a11595 wrote: post #79

Earlier quoted context omitted.

Not a chance buddy. I've been here for a while, and this place has become reddit, with mods from /r/incels. I have two accounts. When I see someone with zero knowledge of subject matter, like OP, authoritatively and dismissively spew complete bs, I reply like I do. People with a clear agenda spewing fake crap gets upvoted here now. Disagree with a mod - your comments are shadow removed, and possibly your account is s…

Hey budz, many around here have 20+ years of experience. I've fought all my career to get "enterprise" folks to understand that spending $100s of mill on hardware vendors and 3rd party software is madness, if what you are doing is actually mission critical. Money should be spend on acquiring talent and adapting to actual business needs. The business you support could not care less about your shiny toys, they want to…

plenty of talent, and for stability and performance all that talent spends millions of dollars on top of the line proven solutions. Applications rely on live migrations for load balancing a farm, and it happens automatically in the background around the clock to rebalance farms. Hardware maintenance is another use case. So is a bucket of water. So is change control - I want to load new HBA firmware? I'll evacuate the ESX to somewhere else first.

no one contemplates blockchain, but for someone CIO level, gartner quadrants present a quick high-level summary of where the industry is going. and $100mil+ for hardware is completely normal to run things that process billions of dollars. The US treasury department is a great example. Mastercard is another.

Re: Why Is Storage on Kubernetes So Hard?

#86

Post by jordanbeiber »

a11595 wrote: post #84
jordanbeiber wrote: post #81

Earlier quoted context omitted.

I said tenants, not users. Tiny little detail that you overlooked there. =) Last company I worked for had a €150.000.000 tech budget and about ~90k employees. Again, if you read what I said you'll notice that I pointed out that you actually can build decent architectures using enterprisey stuff, it will however never scale horizontally in a resonable way. What happens when you have filled all your shelves in the arra…

again, cute. $250K. very cute. Try $5mil for a box, at least, after a 50% vendor discount. VSAN has nothing to do with SAN, and no one on a SAN uses VSAN. You are again showing your lack of enterprise experience, yet you are strongly trashing what you don't understand. A server sees many arrays, over a SAN. That is called horizontal scaling. You can't do that with internal disk.

Oh, 250 was for the box, no disks. 75% discount. :)

Keep it old-school!

Vsan btw is vmwares attempt to actually achieve _horizontal_ scalability.

I just don’t trust vmware with anything except for that bytecode vm. It used to rock, but time went ahead, and even as they IPO’d I thought they would end up dead in the water eventually. I had just been introduced to the wonderful zones in Solaris and it just made hypervisor vm’s seem silly.

And here we are with cgroups and friends...

Have a look at ceph if you’re interested: http://docs.ceph.com/docs/master/architecture/

Anyway, I’m out.

If your horizon extends to what vendors are prepared to sell you, take that 50 discount and run with it!

One final thought: is that disk not ”internal” to the array? This is what makes blockstorage notoriously difficult to scale horizontally. You’ll need very clever software!

GL&HF

Re: Why Is Storage on Kubernetes So Hard?

#87

Post by jordanbeiber »

a11595 wrote: post #85
jordanbeiber wrote: post #82

Earlier quoted context omitted.

Hey budz, many around here have 20+ years of experience. I've fought all my career to get "enterprise" folks to understand that spending $100s of mill on hardware vendors and 3rd party software is madness, if what you are doing is actually mission critical. Money should be spend on acquiring talent and adapting to actual business needs. The business you support could not care less about your shiny toys, they want to…

plenty of talent, and for stability and performance all that talent spends millions of dollars on top of the line proven solutions. Applications rely on live migrations for load balancing a farm, and it happens automatically in the background around the clock to rebalance farms. Hardware maintenance is another use case. So is a bucket of water. So is change control - I want to load new HBA firmware? I'll evacuate the…

What you describe is said talent doing it wrong.

But if it floats your boat — awesome!

Take care.

Re: Why Is Storage on Kubernetes So Hard?

#88

Post by a11595 »

jordanbeiber wrote: post #86
a11595 wrote: post #84

Earlier quoted context omitted.

again, cute. $250K. very cute. Try $5mil for a box, at least, after a 50% vendor discount. VSAN has nothing to do with SAN, and no one on a SAN uses VSAN. You are again showing your lack of enterprise experience, yet you are strongly trashing what you don't understand. A server sees many arrays, over a SAN. That is called horizontal scaling. You can't do that with internal disk.

Oh, 250 was for the box, no disks. 75% discount. :) Keep it old-school! Vsan btw is vmwares attempt to actually achieve _horizontal_ scalability. I just don’t trust vmware with anything except for that bytecode vm. It used to rock, but time went ahead, and even as they IPO’d I thought they would end up dead in the water eventually. I had just been introduced to the wonderful zones in Solaris and it just made hypervis…

yeah, I know what VSAN is. It's you who does not if you think you run VSAN on top of a SAN-connected cluster. Yes, the disk is internal to the array. A server can see a hundred arrays on the same HBAs. It doesn't care what array the storage comes from. I am positive at this point you know nothing about what a SAN is.

What "box" - the 42U Rack? I don't think so. The rack is always free. You then have DAs connected to the disk, and FAs connected to the SAN, which are on a pair of directors. You literally cannot get those w/o disk.

You don't know what a SAN is, you've never seen an itemized quote for an array. Thanks for your link. It's like sending a hooked on phonics link to an English professor. Cute. Keep it cute! People like you are the reason people like me get paid a lot.

Re: Why Is Storage on Kubernetes So Hard?

#89

Post by jordanbeiber »

a11595 wrote: post #88
jordanbeiber wrote: post #86

Earlier quoted context omitted.

Oh, 250 was for the box, no disks. 75% discount. :) Keep it old-school! Vsan btw is vmwares attempt to actually achieve _horizontal_ scalability. I just don’t trust vmware with anything except for that bytecode vm. It used to rock, but time went ahead, and even as they IPO’d I thought they would end up dead in the water eventually. I had just been introduced to the wonderful zones in Solaris and it just made hypervis…

yeah, I know what VSAN is. It's you who does not if you think you run VSAN on top of a SAN-connected cluster. Yes, the disk is internal to the array. A server can see a hundred arrays on the same HBAs. It doesn't care what array the storage comes from. I am positive at this point you know nothing about what a SAN is. What "box" - the 42U Rack? I don't think so. The rack is always free. You then have DAs connected to…

For free! Good one. Cause’ that’s whats really happening... right? With a straight face?

I was certain I had nothing more for you, but this is too much fun!

You clearly don’t grasp the difference between vertical and horizontal scalability which means you have never been subjected to a bunch of scenarios requiring the latter.

Dinosaurs taking the p*ss are the reason many enterprises opt to off-shore and out-source.

Re: Why Is Storage on Kubernetes So Hard?

#90

Post by a11595 »

jordanbeiber wrote: post #89
a11595 wrote: post #88

Earlier quoted context omitted.

yeah, I know what VSAN is. It's you who does not if you think you run VSAN on top of a SAN-connected cluster. Yes, the disk is internal to the array. A server can see a hundred arrays on the same HBAs. It doesn't care what array the storage comes from. I am positive at this point you know nothing about what a SAN is. What "box" - the 42U Rack? I don't think so. The rack is always free. You then have DAs connected to…

For free! Good one. Cause’ that’s whats really happening... right? With a straight face? I was certain I had nothing more for you, but this is too much fun! You clearly don’t grasp the difference between vertical and horizontal scalability which means you have never been subjected to a bunch of scenarios requiring the latter. Dinosaurs taking the p*ss are the reason many enterprises opt to off-shore and out-source.

yes idiot. I have worked for several vendors and sold this stuff. I have also been on the customer side buying this stuff. You're paying 250k for an empty 42U that you call "box" - you literally are lying.

there are, literally, zero enterprises off-shoring their hard IO hitting datacenters. In fact, having been in 80 different countries for these enterprises, they usually have many datacenters all over the world.

VSAN is for hyperconverged systems only. racks with a thousand 1u nodes that all have disk, connected to a fat ethernet backplane. Things on a SAN are not hyperconverged - they are on a SAN. VSAN is for tier 2 stuff that goes on hyperconverged - like web servers and DMZ things. The fastest processing is done on solid databases like Oracle or UDB on clusters of large servers, connected to a SAN. VSAN is even positioned by sales for tier 2 from all major vendors.

You literally picked up some technical words you heard around the office, googled a few things, and now consider yourself and expert so you give authoritative opinion on here about things you have never worked with. I bet you are deskside support or a code monkey, and have never architected a solution. When someone gives you a budget of 20mil and says you can average 1min of application unavailability per year or it impacts billions in company's bottom line and your whole team gets fired, do let me know. I'm sure your suggestion would be to cluster together a bunch of old dell laptops over wifi.

Post reply on HN