Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rolling your own servers with Kubernetes

gravitational.com

Post reply on HN
Print view View original

101–110 of 209 posts

Re: Rolling your own servers with Kubernetes

#101

Post by noahdesu »

the-rc wrote: post #36
q3k wrote: post #12

Earlier quoted context omitted.

I highly recommend Rook [1], which is based on Ceph, provides PersistentVolumes to k8s workloads and is also running on k8s itself. [1] - https://rook.io/

I worked on Colossus at Google and Ceph is the closest thing out there. Gregory Farnum gave a great talk about it at the Open Source Summit 2017. I heard from someone at a large company, though, that it's not getting a lot of love from Red Hat nowadays, even if you're a large paying customer. Now I'm curious.

I'm working on Rook+Ceph at Red Hat. Rook 1.0 was just released last week which added support for the very latest Nautilus release of Ceph.

Re: Rolling your own servers with Kubernetes

#102

Post by lima »

turk73 wrote: post #47
lima wrote: post #9

Earlier quoted context omitted.

In many environments, you don't actually need distributed storage: - Application servers are stateless. - All state lives in stateful databases. - Those database handle high availability at the application layer by replication data, so you can use local storage and deploy them using StatefulSets. Has the extra benefit of avoiding the network storage latency penalty.

But StatefulSets require storage...someplace. We're dealing with that exact issue right now. Our K8s cluster is not playing nice with a NetApp NAS. We are exploring other options such as local iscsi storage volumes ($$$) or an external state store db (slow).

The point is that with native HA, it doesn't have to be distributed storage. You can use locally attached SSDs or NVMe drives, which are straight forward to manage and fast.

Re: Rolling your own servers with Kubernetes

#103

Post by ericd »

chocolatebunny wrote: post #76
segmondy wrote: post #59

I use to enjoy having a cheap desktop under my table or in a closet serving traffic to people across the internet. Computers have gotten faster, software has gotten better, network has gotten faster, things have gotten cheaper. Sadly instead of seeing more of these, most of these are now being outsourced to cloud providers, we have bought and drank the kool-aid that they can do it better and cheap. Which is not true,…

I don't think cost is the benefit of cloud. For tech companies the benefit is scale (you won't know how popular a product is until you deploy). For non-tech companies the benefit is not having to deal with things that aren't related to your core business. I think you can argue that tech is always related to your core business and you only need a few servers to handle a million simultaneous users but those kinds of ar…

Yeah, you don't know how big you'll need to scale, but I think that's a bit of a boogeyman for a lot of companies. Servers have a relatively massive amount of cores and memory these days, along with screaming fast IO, and you can handle quite a lot of web traffic with only a few dual socket servers if you don't bog them down with incredibly inefficient code.

Re: Rolling your own servers with Kubernetes

#104

Post by shaklee3 »

Nullabillity wrote: post #55
kelseyhightower wrote: post #35

Earlier quoted context omitted.

All of the following assumes you plan on running more than a single instance of your monolithic application. If that's not the case, then ignore Kubernetes, and be glad you don't have the problems it was designed to solve. If you consider what it takes to manage the end-to-end lifecycle of a single application, monolith or micro-service, you need a solution for the following items: deployments, application configurat…

> Just like I would not recommend standing up OpenStack from the ground up in order to deploy your monolithic application across a set of virtual machines, I don't recommend rolling your own Kubernetes cluster either. You should strongly consider leveraging a fully managed Kubernetes offering such as Google Kubernetes Engine, Digital Ocean's Managed Kubernetes, or Azure Kubernetes Service. The rest seems reasonable,…

FYI: the person you are responding to is the author of Kubernetes The Hard Way [1], which is effectively a tutorial of learning how all the K8s pieces work together. He also co-authored the first book on it [2]. He's also a Google employee, but I would trust his opinion more than others just because he's probably seen more use cases than anyone else.

[1] https://github.com/kelseyhightower/kubernetes-the-hard-way

[2] https://www.amazon.com/Kubernetes-Running-Dive-Future-Infras...

Re: Rolling your own servers with Kubernetes

#105

Post by imtringued »

geezerjay wrote: post #51
SteveNuts wrote: post #38

Earlier quoted context omitted.

Meanwhile the market and mindshare has moved on to the next shiny new tech.

Kubernetes has been here for a while. I'm sure we can agree that it ceased to be a candidate for a meteoric fad for a while now.

I've probably wasted more than three weeks installing kubernetes on 3 nodes. I've tried Racher, RKE, kube-admin, kubespray and doing everything myself. I always failed and the way it failed was completely opaque and I didn't feel like I could even understand what was going wrong.

Re: Rolling your own servers with Kubernetes

#106

Post by RussianCow »

rhizome wrote: post #60
icebraining wrote: post #56

Earlier quoted context omitted.

It's kinda annoying how the valid criticism of cloud solutions is getting drowned out by criticism on strawmen. Who exactly is saying that no one can afford their own infra anymore? What some people say is that cloud is usually cheaper than running your own infra. They may be wrong (I'm doubtful myself), but it's a wholly different argument.

It usually takes the form of "ugh then you have to hire dedicated people to manage it."

Which is a valid argument, isn't it? Engineers are expensive! Whether they're more expensive than AWS depends entirely on what you're doing, but the answer is almost always "yes" at a small enough scale.

Re: Rolling your own servers with Kubernetes

#107

Post by hn_throwaway_99 »

gambler wrote: post #91
jasode wrote: post #80

Earlier quoted context omitted.

>The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to afford their own infrastructure. The decision framework the executives use isn't just the "hardware+software" -- it's the whole "IT organization" . In other words, it's not "in-house cpu" vs "Amazon's cpu". It's in-house I…

>Most companies are not like Facebook or Dropbox that can maintain an internal IT organization at a high level equivalent to AWS. Let's try this with different phrasing. In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. Forward 10-13 years. We have cheaper hardware. We have extra 10+ years of development in open-source software. And yet that list…

I worked at a couple of those self-hosted companies in the 200Xs. For me, as a developer, it was painful and way, way, way worse than AWS or GCP. Planning for big expected spikes of traffic was extremely painful, and once the cloud vendors started adding new services we felt continually behind the times.

I can only think it was "interesting" if you've forgotten how difficult and expensive managing your own infrastructure can be.

Re: Rolling your own servers with Kubernetes

#108

Post by user5994461 »

jasode wrote: post #80
gambler wrote: post #53

>Should you roll your own servers? If you are not certain, the answer is most likely “no”. The staggering growth of AWS happened for a reason. Funny how for many decades companies and people were running their own servers. The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to…

>The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to afford their own infrastructure. The decision framework the executives use isn't just the "hardware+software" -- it's the whole "IT organization" . In other words, it's not "in-house cpu" vs "Amazon's cpu". It's in-house I…

>>> the IT bureaucracy tells them that it will take 2 weeks

You're not in touch with reality. Try 2 months at a bare minimum.

Re: Rolling your own servers with Kubernetes

#109

Post by ericd »

linuxdude314 wrote: post #39
a012 wrote: post #4

You can colo your server and roll your own k8s clusters. But you can't afford the luxury of high throughput networking, EBS, etc., and don't forget HA options (Multi-AZ) for your clusters.

Untrue. I purchased a 48 port 10GbE Arista 7148 the other month on eBay for $320.

Nice! OT, but have you seen any reasonably priced RJ45 10GBe out there? Preferably without screaming fans :-)

Re: Rolling your own servers with Kubernetes

#110

Post by user5994461 »

hotsauceror wrote: post #100
rhinoceraptor wrote: post #69

Earlier quoted context omitted.

The hardware isn’t the expensive part, it’s the people to run it that is.

Using publicly available costing data from last November, I see that an m4.xlarge RDS reserved instance of SQL Server Enterprise Edition with multi-AZ failover is $39,000 / year. If I have 4 DBAs at a cost of $150K / yr salary + benefits, that's the equivalent of around 15 smallish, managed SQL Server instances. My current SQL footprint is on the order of 600 instances, which works out to be a bit over $23M a year. I…

How much is the license for SQL enterprise nowadays? still 5k per node or does Microsoft charge per socket now?
Post reply on HN