Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Convergence to Kubernetes

medium.com

Post reply on HN
Print view View original

41–50 of 124 posts

Re: Convergence to Kubernetes

#41

Post by rehemiau »

thawkins wrote: post #35
hinkley wrote: post #17

Earlier quoted context omitted.

> having a single artifact and reproducible builds where a developer can run the docker image locally This is the key thing here. As a team grows it’s easy to get various kinds of learned helplessness. Docker, for its faults, is mostly simple enough that you can expect/insist that the team use it. Which means fewer kinds of surprises at deployment time.

We enforced that by giving our devs linux workstations with no sudo rights (iso27001 requirement), but they have access to the docker daemon, and MUST install everything they need in docker images, its massively increased tools sharing between teams, and forced people to learn docker. We have some devs starting to use kubernetes locally using minikube .

Sorry for being off-topic but do you have any links or tips on how to achieve this? My understand was that adding a user to the 'docker' group gives him 'sudo'-equivalent rights.

Re: Convergence to Kubernetes

#42

Post by lowbloodsugar »

hardwaresofton wrote: post #30

Can anyone who used Docker Swarm/Mesos/Nomad and then switched to Kubernetes comment on anything that was done better by Swarm/Mesos/Nomad? I invested in Kubernetes early and always meant to give the others a try (so I could at least know the differences), but never got a chance to.

Mesos is a two layer system. That is what is better about it. Unfortunately, the most common second layer for reliable service execution is Marathon, and it caused us no end of trouble.

Re: Convergence to Kubernetes

#43

Post by chairmanwow »

tonyarkles wrote: post #28
accnumnplus1 wrote: post #20

Earlier quoted context omitted.

30 teams' services fit on 2-3 servers? We don't have anywhere near 30 teams, but at my last count end of last year there were ~250 repos, and everyday I get auto-emailed telling me I've been subscribed to a new repo. No way could we have only 2-3 servers.

I'm perpetually curious about what kind of resource usage you'd expect given those 250 repos, and which runtime you're deploying. I've recently been playing with Go and looking at existing JVM-based services (Spring Boot + starters + our microservices). We've seen rewrites of small services change from ~500MB of RAM to ~10MB of RAM. 250 Go-based services would probably fit very comfortably on a small cluster of cheap…

Can you go into detail about what things changed in the rewrites that enabled this reduction in RAM usage?

Re: Convergence to Kubernetes

#44

Post by antod »

ajross wrote: post #32
accnumnplus1 wrote: post #20

Earlier quoted context omitted.

30 teams' services fit on 2-3 servers? We don't have anywhere near 30 teams, but at my last count end of last year there were ~250 repos, and everyday I get auto-emailed telling me I've been subscribed to a new repo. No way could we have only 2-3 servers.

> 30 teams' services fit on 2-3 servers? Why would system load scale with the number of engineers? Yeah. 30 "teams" worth of work can totally fit on one system. The sum total of all software engineering ever done before, I dunno, 1979 can fit on one box. Really, that's my point. People are far too enthused with the "feeling" of working on a "big" project and not being sufficiently reasonable or conservative about the…

You can't see other reasons for running more servers than just system load?

We are a bit smaller than uSwitch (no Kubernetes though), but we maybe a thousand VMs spread a couple of dozen physical servers and some cloud stuff.

We have multiple environments across multiple sites for each of multiple enterprise customers with multiple server roles. There are no single points of failure in any server role and contractural/legal/privacy requirements to keep data (and staff access) separate between different 'zones'. There are integration points with 3rd party partners, logging systems, monitoring systems, bastion hosts, internal CI build clusters, internal business tools etc etc.

Beyond a certain size and complexity, you need to separate stuff out to stop people treading all over each other. You need to be able to contain breakage by ideally running a single service on each VM. None of this duplication was due to people wanting a big project - it grew over time out of necessity even with conservative 'enterprisey' attitudes to technology.

Re: Convergence to Kubernetes

#45

Post by tannhaeuser »

manigandham wrote: post #22
ajross wrote: post #15

> We have close to 30 teams that run some or all of their workloads on our clusters. Approximately 70% of all HTTP traffic we serve is generated from applications within our Kubernetes clusters. Sounds big. But then per wikipedia: > uSwitch.com [...] allows consumers to compare prices for a range of energy, personal finance, insurance and communications services. And: > On 30 April 2015, the property website firm Zoo…

Scale isn't everything. Removing "hand-managed" from the entire process is an even bigger benefit. It's really not that complicated, it's clustering software that lets you encapsulate your applications into containers and just write simple declarative YAML files while it takes care of actually running it as specified, and keeping it running regardless of what happens to the hardware. Given all the managed offerings n…

> simple declarative YAML files

If the config files and underlying infrastructure are changing all the time and have varying degrees of documentation, that's just shifting know-how from established tools to the newest fad, especially when k8s know-how is extremely scarce/expensive, and will leave you in a trial-and-error situation with unclear diagnostics if anything goes wrong due to the sheer complexity. Automation is of course not limited to k8s at all. I've witnessed moving perfectly running Terraform-like setups to more expensive k8s setups by admins just so that k8s appears on their resume. There are valid reasons to use k8s, mesos or openshift in big shops, but at this point k8s is just oversold and overhyped IMHO, and not a good match for startups ("our k8s guy comes next tuesday"). Especially when it only allows Docker containers, which is another "political" marchitecture solution with awkward technical constraints to compensate rather than a solution based on merit IMHO.

Re: Convergence to Kubernetes

#46

Post by halbritt »

hardwaresofton wrote: post #30

Can anyone who used Docker Swarm/Mesos/Nomad and then switched to Kubernetes comment on anything that was done better by Swarm/Mesos/Nomad? I invested in Kubernetes early and always meant to give the others a try (so I could at least know the differences), but never got a chance to.

Couple things: Mesos had a more sophisticated scheduler with better isolation for disk IO. It could also run arbitrary binaries in cgroups. I.e. a JVM based application without having to run it in a docker container.

Uber took advantage of this by running thousands of nodes of Cassandra in DC/OS.

Re: Convergence to Kubernetes

#47

Post by pageald »

hardwaresofton wrote: post #30

Can anyone who used Docker Swarm/Mesos/Nomad and then switched to Kubernetes comment on anything that was done better by Swarm/Mesos/Nomad? I invested in Kubernetes early and always meant to give the others a try (so I could at least know the differences), but never got a chance to.

Thoughts on Nomad after admittedly minimal dabbling: I pushed for using Nomad at my job without success. Managerial perceptions of Kubernetes as "the consensus" is a self-fulfilling prophecy. Nobody wants to pick a technology with a fraction of the buy-in of Kubernetes or be forced into paying boku bucks for an enterprise contract. Hashicorp's insistence on releasing premium closed-source features doesn't work when its biggest competitor is fully free and open.

I found it dead simple to get up and running with Nomad, but it is (perhaps intentionally) missing a lot of features of Kubernetes. For instance, if you want load balancing and auto-scaling, you need to rig it up yourself. In K8s, you set up a service and horizontal pod autoscaling and you're done.

Re: Convergence to Kubernetes

#48

Post by snuxoll »

oso2k wrote: post #27
ordinaryperson wrote: post #2

Dumb question from someone who doesn't use Kubernetes (or Docker) in production: don't routine security updates mean you're constantly rebuilding and redeploying these images? And if so, how is that more efficient than just using Puppet / Chef / Ansible and a 'real' server?

Full-disclosure: I’m a Consulting Architect for Red Hat focused on OpenShift. No. In Kubernetes and OpenShift you can control whether builds and deployments are automatic or manual, & which events triggers them [0][1]. Combined with fact that each application’s config is an independent object, this allows an admin to host hundreds of apps on a single cluster node. Usually you’re using IaC practice and storing each ap…

OpenShift has an answer to what GP was asking, but build triggers are not it.

Security vulnerabilities in images is the concern, OpenShift handles this through ImageStream('s) - parent images are tracked in the integrated registry and when one is updated all dependent images are updated.

Good example, the dotnet image in our OpenShift cluster was updated late last week - all of our .Net Core projects were automatically rebuilt with the latest image with no intervention from the developers. It doesn't handle your application INSIDE the container, or if you build the image yourself via Dockerfile so you'll still need some dependency scanning tools and/or release notifications to keep your own stuff up-to-date and secure.

To be honest ImageStream is one of the best value-add features OpenShift has over vanilla Kubernetes, I don't have to worry if developers are keeping their images up-to-date when some vulnerability in a CentOS package or application runtime gets patched.

Re: Convergence to Kubernetes

#49

Post by polskibus »

marcc wrote: post #10
polskibus wrote: post #6

Is there a recommended way to handle database migrations in kubernetes? Is there a best practice or a tool for that?

We've been packaging migrations into a container and just shipping it as a Kubernetes Job. It executes once, and if doesn't success (non-zero exit code), Kubernetes will reschedule it.

So if you want to migrate a particular env from schema version 3 to version 6, do you deploying 3 different init containers manually, or is there a single migration container that always contains all migrations and it by itself understands from which level to which level should the migration be run? Schema migration failure is a stop the world a scenario for our entire stack.

Re: Convergence to Kubernetes

#50

Post by gnur »

hardwaresofton wrote: post #30

Can anyone who used Docker Swarm/Mesos/Nomad and then switched to Kubernetes comment on anything that was done better by Swarm/Mesos/Nomad? I invested in Kubernetes early and always meant to give the others a try (so I could at least know the differences), but never got a chance to.

Having used both DC/OS (mesos & marathon) and kubernetes in production.

The one thing DC/OS had absolutely nailed was the web interface and bootstrap. The DC/OS interface is brilliant as it allows you to explore all the possibilities and actually have a overview of what is happening.

Also, it was a lot easier to reason about because everything is contained in a single marathon job. No need to split everything up in deployments / services / ingresses. A single JSON file is all you need for DC/OS. Less to think about.

The downside of all this is that DC/OS feels like a solution for a theoretical problem, while kubernetes is the solution to practical problems.

Post reply on HN