We're Leaving Kubernetes • Hacker News Board

Re: We're Leaving Kubernetes

#251

Post by jen20 » Tue, Nov 05, 2024, 1:07 PM UTC

rtpg wrote: post #224

pmarreck wrote: post #190

Earlier quoted context omitted.

> installing Nix seems pretty invasive I'm writing a test to check whether a tool I'm writing can work without Nix (it works with it perfectly, but I want it to also work without it because there are a lot of folks like you, and like me about 3 years ago, who still think they'd rather struggle with manually installing the right glibc that goes with the right python dependency installed with the right pip and venv ver…

Nix on MacOS makes a bunch of build users and has a bad habit of touching files that then get touched by MacOS updates, leading to a system that works... then doesn't work.

The only such case I’ve seen was the early betas of Sequoia requiring an extra flag to use higher UIDs for build users.

Re: We're Leaving Kubernetes

#252

Post by baq » Tue, Nov 05, 2024, 1:24 PM UTC

maxrecursion wrote: post #130

0xbadcafebee wrote: post #57

Earlier quoted context omitted.

Most teams/products I have been involved in, the stack always grows to the point that a dev can no longer test it on their own machine, regardless of how big the machine is. And having a different development machine than production leads to completely predictable and unavoidable problems. Devs need to create the software tooling to make remote dev less painful. I mean, they're devs... making software is kind of thei…

"Most teams/products I have been involved in, the stack always grows to the point that a dev can no longer test it on their own machine" Isn't this problem solved by CICD? When the developer is ready to test, they make a commit, and the pipeline deploys the code to a dev/test environment. That's how my teams have been doing it.

When you want to run the whole test suite, yes.

When you're developing and only need to touch 0.1% of the product and 0.001% of the code, that's a total and complete waste of time.

Re: We're Leaving Kubernetes

#253

Post by baq » Tue, Nov 05, 2024, 1:29 PM UTC

d_sem wrote: post #144

jrockway wrote: post #137

Earlier quoted context omitted.

This turns a 1 hour task into a 1 day task. Fast feedback cycles are critical to software development. I don't quite understand how people get into the situation where their work can't fit on their workstation. I've worked on huge projects at huge tech companies, and I could run everything on my workstation. I've worked at startups where the CI situation was passing 5% of the time and required 3 hours to run, that yo…

It's likely you haven't come across these use cases in your professional career, but I assure you its very common. My entire career has only seen projects where you need dozen to hundreds of CPU's in order to have a short feedback loop to verify the system works. I saw this in simple algorithms in automotive, to Advanced Driver Assistance Systems and machine learning applications. When you are working on a software p…

There's a lot of folks in startups who think 100 devs is a large org and can't comprehend the scale at which '100% tests pass' stops being a build blocker. I've migrated from such an org to a late stage startup and 'tests must pass' even if fifty engineers are blocked with their PRs and the release train is fully halted. 'But our pipelines must be green' no they don't, at least not all of them.

Re: We're Leaving Kubernetes

#254

Post by 0xCMP » Tue, Nov 05, 2024, 1:36 PM UTC

wlll wrote: post #247

czhu12 wrote: post #148

Earlier quoted context omitted.

We tried this approach at a former company with ~600 engineers at the time. Trying to boot the full service on a single machine required every single developer in the company installing ~50ish microservices on their machine, for things to work correctly. Became totally intractable. I guess one can grumble about bad architecture all day but this had to be solved. we had to move to remote development environments which…

That's ~12 people per microservice. I understand that it might be hard to boot and coordinate all that on each developers machine, and that local docker images/vms/whatever might not suit you, but is it possible that this is a problem of your companies own creation?

It's very likely not their decisions that lead to this, but their responsibility to improve velocity.

Imagine the goal is to fix the problems (e.g. make it possible to run less of the services or something like that): How do you do that without first running all the services, making the proper changes, and then testing those changes? You need to be able to run all the services in that interim period.

So, wouldn't it be nice if there were a solution for this in-general? And, maybe, it would lead to better conditions later on. But in the meantime there is really no way around the existing design/decisions/etc. You simply have to deal with that reality and engineer around it.

Re: We're Leaving Kubernetes

#255

Post by brigandish » Tue, Nov 05, 2024, 1:40 PM UTC

Kye wrote: post #249

gchamonlive wrote: post #238

Earlier quoted context omitted.

You need an option to enable only the services they need to develop locally and automatically configure them to talk to the other services hosted elsewhere.

What ever happened to make? It seems like so much of what developers do now just duplicates what could be done with a single makefile that pokes around, figures out what you need, and makes a script to set it up.

Make is horrible though. It works, but it’s horrible. Hardly surprising people have looked for alternatives, it’s just a shame that most of those are horrible too.

Re: We're Leaving Kubernetes

#256

Post by wlll » Tue, Nov 05, 2024, 1:41 PM UTC

0xCMP wrote: post #254

wlll wrote: post #247

Earlier quoted context omitted.

That's ~12 people per microservice. I understand that it might be hard to boot and coordinate all that on each developers machine, and that local docker images/vms/whatever might not suit you, but is it possible that this is a problem of your companies own creation?

It's very likely not their decisions that lead to this, but their responsibility to improve velocity. Imagine the goal is to fix the problems (e.g. make it possible to run less of the services or something like that): How do you do that without first running all the services, making the proper changes, and then testing those changes? You need to be able to run all the services in that interim period. So, wouldn't it…

> It's very likely not their decisions that lead to this

Yeah, I get that, I was deliberate about the phrasing of "your company" rather than just "your".

Obviously we don't know anything about the parent commenters company and situation, perhaps 12 people per microservice genuinely is the right solution for them, but it seems like it would be better not to get into this situation in the first place, though once there you obviously have to tackle the problem as it presents itself.

Re: We're Leaving Kubernetes

#257

Post by jt2190 » Tue, Nov 05, 2024, 1:44 PM UTC

horsawlarway wrote: post #85

jt2190 wrote: post #78

Earlier quoted context omitted.

I’m not sure we should leap from: > I have seen several attempts to move dev environments to a remote host. They invariably suck. To “therefore they will always suck and have no benefits and nobody should ever use them ever”. Apologies for the hyperbole but I’m making a point that comments like these tend to shut down interesting explorations of the state of the art of remote computing and what the pros/cons are. Edi…

But they don't suck because of lack of effort - they suck because there are real physical constraints. Ex - even on a VERY good connection, RTT on the network is going to exceed your frame latency for a computer sitting in front of you (before we even get into the latency of the actual frame rendering of that remote computer). There's just not a solution for "make the light go faster". Then we get into the issues the…

> even on a VERY good connection, RTT on the network is going to exceed your frame latency for a computer sitting in front of you (before we even get into the latency of the actual frame rendering of that remote computer). There's just not a solution for "make the light go faster".

Are you imagining the implementation as some kind of Remote Desktop setup, where no software runs on the local machine (except the Remote Desktop, of course)? This is not the state of the art for using remote developer machines: Typically some editor/IDE components run locally, for example.

> Shared resources are unpredictable.

Then don’t share them! We should do something akin to physically moving your local computer into the data center next to the servers.

> Is my code running slowly right now because I just introduced an issue, or is it because I'm sharing an env and my neighbor just ate 99% of the CPU/IO, or my network provider has picked a different route and my latency just went up 500ms?

If the software you’re developing is going to run in a shared environment then it’s better you experience these issues while developing otherwise you’re asking for a lot of “works on my machine” problems.

> And that's before we even touch the "My machine is down/unreachable

This seems less about remote and more about stability/software design?

> I don't know why and I have no visibility into resolving the issue…

This seems more about observability/software design?

> We're not talking about machines running production data.

The source code itself is something that a company should protect for multiple reasons, not the least of which is preventing an attacker from reading your source to find exploits. There are also various legal and compliance reasons for limiting the distribution of source as much as possible.

> What I do want is a fast system that I control so that I can actually tweak it as needed to develop and debug the system

I don’t understand why this is impossible on a remote machine. Can you elaborate?

> it is almost impossible to give a developer "the least access needed" to do development locally because if you know what that access was you wouldn't be developing still.

I’m sure we can imagine all sorts of setups, from free-for-all root access to so locked down it’s impossible to do work. The sweet spot is typically “you can sudo within reason but we’re logging your activity.”

Re: We're Leaving Kubernetes

#258

Post by consteval » Tue, Nov 05, 2024, 1:48 PM UTC

Cthulhu_ wrote: post #244

themanmaran wrote: post #165

Earlier quoted context omitted.

I bet it's less of a RAM issue, and more of an orchestration problem. Making sure you have the latest version of every microservice and it's configuration. "Oh it's not running locally, you need to also run service_18_v2.js, and include the right env variables"

If you have to have the latest version etc of every microservice, it's a distributed monolith.

Right, the crux here is bad architects are building distributed systems that are essentially monoliths. It's the worst of both worlds. You get none of the guarantees and visibility of a monolith, and you get all the friction of a distributed system.

Re: We're Leaving Kubernetes

#259

Post by peeters » Tue, Nov 05, 2024, 1:55 PM UTC

horsawlarway wrote: post #33

Personally - just let the developer own the machine they use for development. If you really need consistency for the environment - Let them own the machine, and then give them a stable base VM image, and pay for decent virtualization tooling that they run... on their own machine. I have seen several attempts to move dev environments to a remote host. They invariably suck. Yes - that means you need to pay for decent h…

> The only real downside is data control (ie - the company has less control over how a developer manages assets like source code).

I've worked in a remote, secured development environment and it sucked, but to their credit the company did it for exactly this reason - control over the source. But bear in mind that source control is a two-way street.

Losing proprietary source can be harmful (especially in compiled languages where the source might carry much more information than the distributable). But they were mostly worried about the opposite way...that something malicious gets INTO the source which could pose an existential threat. You'd be correct to say "well that should be the domain of source control, peer review etc", but in this case the company assessed the risk high enough to do both.

Re: We're Leaving Kubernetes

#260

Post by santiagobasulto » Tue, Nov 05, 2024, 1:55 PM UTC

bittermandel wrote: post #208

concerndc1tizen wrote: post #81

Sounds more to me like they need a new CTO. And that they're desperate to tell customers that they've fixed their problems. Kubernetes is absolutely the wrong tool for this use case, and I argue that this should be obvious to someone in a CTO-level position, or their immediate advisors. Kubernetes excels as a microservices platform, running reasonably trustworthy workloads. The key features of Kubernetes are rollout…

I consider Kubernetes to be an excellent framework to build these kinds of applications. The difference here is Gitpod being stateful, which is notoriously hard on Kubernetes, though easier now than ever before! To clarify on one of your points, Kubernetes itself has nothing to do with actually setting the security boundaries. It only providers a schema to describe resources and policies, and then an underlying syste…

> The difference here is Gitpod being stateful, which is notoriously hard on Kubernetes, though easier now than ever before!

We've always had issues with stateful kubernetes setups. Can you share what makes it easier today than before? Genuinely interested.