Live data from Hacker News

We're Leaving Kubernetes

gitpod.io

201–210 of 348 posts

Re: We're Leaving Kubernetes

#201

Earlier quoted context omitted.

I bet it's less of a RAM issue, and more of an orchestration problem. Making sure you have the latest version of every microservice and it's configuration. "Oh it's not running locally, you need to also run service_18_v2.js, and include the right env variables"

No, it was definitely a ram issue in this case. The laptops had 16GB of ram, the maximum available at the time. With the java VM overhead that ran things straight into swap and then some. Running the dev environments remotely (or rewriting in Go) were the options being considered before the whole project was canned and people redistributed to other things.

Given some Kubernetes workloads, even writing everything in Assembly might not help.

Re: We're Leaving Kubernetes

#202
post #73

Earlier quoted context omitted.

I have used remote dev machines just fine, but my workflow vastly differs from many of my coworkers: terminal-only spacemacs + tmux + mosh. I have a lot of CLI and TUI tools, and I do not use VScode at all. The main GUI app I run is a browser, and that runs locally. I have worked on developing VMs for other developers that rely on a local IDE such. The main sticking point is syncing and schlepping source code (someth…

I am hardly a dev but occasionally have had to do some or some scripting or web stuff and have really loved VSCode and using the remote SSH support to basically feel like I’m coding locally. Does that not work for your devs?

UNIX, and other competing timesharing systems of the time, have always been remote first, with Windows catching up with Citrix, followed by RDP, and nowadays finally headless as well.

Nowadays Web frontends and SSH/cloud shell, have replaced what used to be X Windows / telnet / rsh, but the underlying workflows aren't much different than running an IDE / emacs /vi / joe /... from a UNIX development server in a 1990's office.

Re: We're Leaving Kubernetes

#203

Earlier quoted context omitted.

From my perspective, installing Nix seems pretty invasive. I can understand if someone doesn't want to mess with their system "unnecessarily" especially if the tool and it's workings are foreign. And I can't really remember the last time I had issues with non-deterministic dependencies either. Dependency versions are locked. Maybe I'm missing something?

> installing Nix seems pretty invasive I'm writing a test to check whether a tool I'm writing can work without Nix (it works with it perfectly, but I want it to also work without it because there are a lot of folks like you, and like me about 3 years ago, who still think they'd rather struggle with manually installing the right glibc that goes with the right python dependency installed with the right pip and venv ver…

Thanks for providing some concrete examples and explaining how it works. Sounds like a reasonable setup to automate. I'm actually running NixOS at home on a few hosts (but not on my desktop), so Nix isn't entirely foreign to me, but I haven't (yet?) used it with other distros or for developing as my needs have been simpler. In your case, I probably would have chosen the recommended path.

In my current work project, we use Windows and .NET with some libs and tools. Nothing too complicated, but automating it would be nice. I could probably push for it a bit, but I'm not familiar with automating Windows environments, since I mainly use Linux at home.

Re: We're Leaving Kubernetes

#204
post #120

Earlier quoted context omitted.

> the stack always grows to the point that a dev can no longer test it on their own machine So the solution here is to not have that kind of "stack". I mean, if it's all so big and complex that it can't be run on a laptop then you almost certainly got a lot of problems regardless. What typically happens is tons of interconnected services without clear abstractions or interfaces, and no one really understands this spa…

> So the solution here is to not have that kind of "stack". Reminds me of my favorites debugging technique. It's super fast: Don't write any bugs!

What a boring trite reply. All of this is analogous to badly written spaghetti code. And yes, you can absolutely avoid all of this if you know what you're doing.

Re: We're Leaving Kubernetes

#205

The original k8s paper mentioned that the only use case was a low latency and a high latency workflow combination and the resource allocation is based on that. The generic idea is that you can easily move low latency work between nodes and there are no serios repercussions when a high latency job fails. Based on this information, it is hard to justify to even consider k8s for the problem that gitpod has.

Thanks for reading the paper!

For those who are interested:

https://static.googleusercontent.com/media/research.google.c...

I am not sure what differences k8s has compare to Borg. At the concept level these are pretty comparable.

Re: We're Leaving Kubernetes

#206
post #3

> This is not a story of whether or not to use Kubernetes for production workloads that’s a whole separate conversation. As is the topic of how to build a comprehensive soup-to-nuts developer experience for shipping applications on Kubernetes. > This is the story of how (not) to build development environments in the cloud. I'd like to request that the comment thread not turn into a bunch of generic k8s complaints. Th…

Can someone clarify if they mean development environments, or if they're talking about a service that they sell that's related to development environments. Because I don't understand most of the article if it's the former. How are things like performance are a concern for internal development environments? And why are so many things stateful - ideally there should be some kind of configuration/secret management solut…

It's for running their commercial products, which are stateful and long-lived developer environments.

Re: We're Leaving Kubernetes

#207
post #180

Earlier quoted context omitted.

> From my perspective, installing Nix seems pretty invasive. How so? With what other software does Nix interfere?

From the provided link: > Nix requires a broad set of changes to your system, from creating new users to installing and running a daemon to creating a root volume and beyond

Ok, I see what you mean now.

What the post is trying to do there is motivate the creation of a new installer, including to the existing Nix community. The snippet you've highlighted is essentially correct, but I still wouldn't characterize Nix as particularly invasive.

The only that Nix strictly needs is to be plugged into your shell. That's it. It doesn't need deep or special hooks into a system just to function.

But including the daemon enables sandboxing for builds that Nix performs, which improves both the security and isolation of those builds, and it also lets Nix be shared nicely between unprivileged users on multiuser systems. For those reasons, daemonful installs are the default and with them come the system users.

(Adding system users is pretty much bog standard stuff for Unix system software, since the main kind of security boundary designed into that system is boundaries between users. Indeed, that's exactly what that's used for with Nix, too.)

The two things I described above comprise the totality of what is required to enable all of Nix's functionality. Everything else that the Determinate Nix installer does as of now is to work around or avoid macOS quirks, and is totally unnecessary for using Nix on any other OS.

The 'root volume' stuff is the result of a collision between the historical and conventional location of the Nix store at `/nix` and Apple's later imposition of a read-only root partition. So Nix installers do a little Apple-specific dance that creates a kind of filesystem volume that doesn't take up any real space or involve any physical partitioning of the disk when they run on macOS.

The other thing this installer does is build in an attempt to self-repair the damage that Apple inflicts upon Nix's sole real requirement by having macOS unconditionally clobber the shell config files under /etc during major macOS updates.

That's it. That's an exhaustive list of all the things a Nix installer does and why. It's not particularly tricky, or hard to remember or figure out. It's not even hard to undo manually— before the Determinate Nix installer existed, I sometimes uninstalled Nix by hand while manually testing the macOS bootstrap scripts for my dotfiles. It was annoying to do, and the uninstallation functionality of the Determinate Nix installer is extremely reliable and convenient and nice. But anyone who knows what `$PATH` is and has ever run `man` before could completely uninstall Nix even if some joker walked over to their machine and deleted the uninstaller.

At the same time, none of the changes Nix installers make on your system affect the behavior of outside programs at all, except by exposing what you choose to install via Nix through standard Unix environment variables like PATH.

Lacking things like kernel components, automatic self-updates, or the requirement for privileged APIs (e.g., on macOS, the endpoint security APIs and accessibility APIs), Nix is not only far less invasive than any endpoint security software, monitoring software, or MDM software you are likely to run on a work machine, but I'd argue tons of common desktop software like Zoom, Discord, DisplayLink and tons of popular macOS powertools like Amphetamine, SteerMouse, SoundSource, etc.

Plus the uninstall procedure with the DetSys installer and its forks is totally conventional and leaves nothing behind: run uninstaller, thing gone.

Nix on macOS is admittedly not an installer-free, drag-and-drop app bundle like some lovely applications get to be. But at most workplaces it's not likely to crack the top 10 most invasive applications installed on the average developer machine, either. Nix installers are just very up front about the things they do set up.

All that said, there are reasonable people who find having a daemon at all offensive. People who are deeply committed to minimalism or simplicity might prefer a single-user install or to use some other tool. But I think for most people, Nix is imo more than fine in terms of invasiveness.

Re: We're Leaving Kubernetes

#208

Sounds more to me like they need a new CTO. And that they're desperate to tell customers that they've fixed their problems. Kubernetes is absolutely the wrong tool for this use case, and I argue that this should be obvious to someone in a CTO-level position, or their immediate advisors. Kubernetes excels as a microservices platform, running reasonably trustworthy workloads. The key features of Kubernetes are rollout…

I consider Kubernetes to be an excellent framework to build these kinds of applications. The difference here is Gitpod being stateful, which is notoriously hard on Kubernetes, though easier now than ever before!

To clarify on one of your points, Kubernetes itself has nothing to do with actually setting the security boundaries. It only providers a schema to describe resources and policies, and then an underlying system (perhaps Cilium for networking, or Kata Containers for micro VMs) can ensure that the resources created actually follow those schemas and policies.

For example, Neon have built https://github.com/neondatabase/autoscaling which manages Neon Instances with Kubernetes by running them with QEMU instead. This allows them to do live migrations and resource (de)allocation while the service is running, without having to replace Kubernetes. These workloads are, as far as I understand it, stateless.

Re: We're Leaving Kubernetes

#209
post #202

Earlier quoted context omitted.

I am hardly a dev but occasionally have had to do some or some scripting or web stuff and have really loved VSCode and using the remote SSH support to basically feel like I’m coding locally. Does that not work for your devs?

UNIX, and other competing timesharing systems of the time, have always been remote first, with Windows catching up with Citrix, followed by RDP, and nowadays finally headless as well. Nowadays Web frontends and SSH/cloud shell, have replaced what used to be X Windows / telnet / rsh, but the underlying workflows aren't much different than running an IDE / emacs /vi / joe /... from a UNIX development server in a 1990's…

The funniest (?) thing to me about all this: we're still hoping, if we do things right, to replicate the technology (terminals) from 50 years ago.

I honestly don't understand why nobody has simply invented some software to solve this problem, after 50 years.

Re: We're Leaving Kubernetes

#210
The debate in the comments about whether you should run locally is fascinating.

To the people saying ultra modern hardware could handle it: worth remembering the companies on question started on this path X years ago with Y set of technologies and Z set of experiences.

Because it made sense for Google in 2012 or whatever doesn't necessarily mean they would choose it again --or not-- given a do over (but there's basically no way back).

Post reply on HN