181–190 of 348 posts
Sounds more to me like they need a new CTO. And that they're desperate to tell customers that they've fixed their problems. Kubernetes is absolutely the wrong tool for this use case, and I argue that this should be obvious to someone in a CTO-level position, or their immediate advisors. Kubernetes excels as a microservices platform, running reasonably trustworthy workloads. The key features of Kubernetes are rollout…
I feel like anyone who was building a CI solution to sell to others and chose kubernetes didn't really understand the problem. You're running hot pods for crypto miners and against people who really want to see the rest of the code that box has ever seen. You should be isolating with something purpose built like firecracker, and do your own dispatch & shred for security.
Earlier quoted context omitted.
In my last role as a director of engineering at a startup, I found that a project `flake.nix` file (coupled with simply asking people to use https://determinate.systems/posts/determinate-nix-installer/ to install Nix) led to the fastest "new-hire-to-able-to-contribute" time of anything I've seen. Unfortunately, after a few hires (hand-picked by me), this is what happened: 1) People didn't want to learn Nix, neither d…
I think this, or something of equal complexity, is probably the right choice. I have spent a lot of time helping people with their dev environments, and the same problems keep coming up; "no, you need this version of kubectl", "no, you need this version of jq", "no, the Makefile expects THIS version of The Silver Searcher". A mass of shell scripts and random utilities was a consistent drag on the entire team and ever…
Wherever there's such overlap, those dependencies are already shared. Static linking in such a situation means more disk usage, not less.
Packages in Nixpkgs have large closure sizes for entirely other reasons, like not splitting packages as aggressively as they could be split, or enabling/including most optional dependencies by default. Distros like Alpine typically lean the other way for their defaults.
It's true that if you're willing to manually mangle them, static binaries are nice because you can very easily strip all docs and examples or even executables that you don't need, and still know your executable will have the libs it's linked against. In one place at work I actually do this with Nixpkgs— there's a pkgsStatic that includes only statically compiled packages. I pull just the tiny parts of some package I need out and copy them onto a blank OCI image because it was the path of least resistance.
But Nix also has some really nice tools for inspecting dependency graphs to figure out why large packages are getting pulled in. nix-tree is my favorite, but there's also the older nix-du that gives the same info via graphviz instead of the terminal, and the built-in `nix why-depends`.
-----
Edited to add: wait, are you saying you used some other base distro to create Docker images where some things were supplied by Nix and others came from the base distro? If so, yeah, Nix is going to bring all the dependencies along, all the way down to libc or whatever. That's required for the kind of hermeticity that is its goal.
Mixed images like that are always going to be larger. But you also don't need a base distro at all with Nix. You can use one of the existing Nix libraries for Docker/OCI stuff to generate a complete image from scratch, or just copy your Nix packages' dependency closure onto an empty image with a FROM SCRATCH Dockerfile.
If you can't do that, you can do various things to try to slim things down but it's best to just Nixify whatever other packages you're using so you don't need a base distro. (And if you're trying to save space, Nix itself doesn't need to be in your Docker images either, which can also cut out some deps.)
Earlier quoted context omitted.
From my perspective, installing Nix seems pretty invasive. I can understand if someone doesn't want to mess with their system "unnecessarily" especially if the tool and it's workings are foreign. And I can't really remember the last time I had issues with non-deterministic dependencies either. Dependency versions are locked. Maybe I'm missing something?
> From my perspective, installing Nix seems pretty invasive. How so? With what other software does Nix interfere?
> Nix requires a broad set of changes to your system, from creating new users to installing and running a daemon to creating a root volume and beyond
Earlier quoted context omitted.
Why would you say that performance is bad on public cloud infrastructure?
There are things that public cloud is great for. Cost efficiency at high performance is not it. For Gitpod, performance is critical to their product offering, because any latency in a dev environment is terrible UX. Example: What performance do you get out of your NVMe disks? Because these days you can build storage that delivers 100-200 GB/s. https://www.graidtech.com/wp-content/uploads/2023/04/Results... I bet few…
Earlier quoted context omitted.
In my last role as a director of engineering at a startup, I found that a project `flake.nix` file (coupled with simply asking people to use https://determinate.systems/posts/determinate-nix-installer/ to install Nix) led to the fastest "new-hire-to-able-to-contribute" time of anything I've seen. Unfortunately, after a few hires (hand-picked by me), this is what happened: 1) People didn't want to learn Nix, neither d…
> time is possibly too short (especially if you have kids) to learn new things that aren't simple, even if better Having a kid has drastically altered my ability to learn new things outside of work, simply due to lack of time. I never could have imagined how big of an impact having a kid would be, its crazy! The worst thing is when you actually manage to carve out some time to do some learning or experimentation with…
yeah, I could have written this verbatim. Either I was not warned enough, or I did not pay enough attention/heed whatever I was warned of. I don't have a large family, so I've basically had ZERO kid experience since I was a kid... yikes... almost 50 years ago LOL. What worries me though is that it's kind of been an assumption at this job that you DO spend some off-duty time learning/tinkering. And I enjoyed it!
> The worst thing is when you actually manage to carve out some time to do some learning or experimentation with a new tool, library, etc only to find out that it sucks
I got briefly excited about the V language to maybe use for little utility scripts and maybe even as a first teaching language for my kid, then realized that when you scratch the surface of it it's basically kind of ugly underneath. (Example- The "this should never happen" error was literally most of the errors, lol.) It looks like something with a lot of great ideas but slipshod not-deeply-thought-out implementation. And the final nail in the coffin was all the evidence that the language creator simply bans anyone with valid criticism- I'm a free-speech near-absolutist so that one was the killer for me.
One of a few examples of what you're referring to. The thing is, before kids, we could afford to waste that time. Now we cannot. :/
Personally - just let the developer own the machine they use for development. If you really need consistency for the environment - Let them own the machine, and then give them a stable base VM image, and pay for decent virtualization tooling that they run... on their own machine. I have seen several attempts to move dev environments to a remote host. They invariably suck. Yes - that means you need to pay for decent h…
We tried this approach at a former company with ~600 engineers at the time. Trying to boot the full service on a single machine required every single developer in the company installing ~50ish microservices on their machine, for things to work correctly. Became totally intractable. I guess one can grumble about bad architecture all day but this had to be solved. we had to move to remote development environments which…
This is certainly one of the critical mistakes you did.
No developer needs to launch half of the company's services to work on a local deployment. That's crazy, and awfully short-sighted.
The only services a developer ever needs to launch locally are the ones that are being changed. Anything else they can consume straight out of a non-prod development environment. That's what non-prod environments are for. You launch your local service locally, you consume whatever you need to consume straight from a cloud environment, you test the contract with a local test set, and you deploy the service. That's it.
> I guess one can grumble about bad architecture all day but this had to be solved.
Yes, it needs to be solved. You need to launch your service locally while consuming dependencies deployed to any cloud environment. That's not a company problem. That's a problem plaguing that particular service, and one which is trivial to solve.
> Both FAANG companies I’ve worked at had remote dev environments that were built in house.
All FANG companies I personally know had indeed remote dev environments. They also had their own custom tool sets to deploy services locally, either in isolation or consuming dependencies deployed to the cloud.
This is not a FANG cargo cult problem. This is a problem you created for yourself out of short-sightedness and for thinking you're too smart for your own good. Newbies know very well they need to launch one service instance alone because that's what they are changing. Veterans know that too well. Why on earth would anyone believe it's reasonable to launch 50 services to do anything at all? Just launch the one service you're working on. That's it. If you believe something prevents you from doing that, that's the problem you need to fix. Simple. Crazy.
Earlier quoted context omitted.
From my perspective, installing Nix seems pretty invasive. I can understand if someone doesn't want to mess with their system "unnecessarily" especially if the tool and it's workings are foreign. And I can't really remember the last time I had issues with non-deterministic dependencies either. Dependency versions are locked. Maybe I'm missing something?
Typically when you start a new dev job the company will provide you with a pre-provisioned laptop that has their security stuff setup and maybe dev tools already installed, eg source code, compilers, VMs, Nix, and a supported editor, so it's not exactly a personal machine that they're messing with.
Earlier quoted context omitted.
I bet it's less of a RAM issue, and more of an orchestration problem. Making sure you have the latest version of every microservice and it's configuration. "Oh it's not running locally, you need to also run service_18_v2.js, and include the right env variables"
No, it was definitely a ram issue in this case. The laptops had 16GB of ram, the maximum available at the time. With the java VM overhead that ran things straight into swap and then some. Running the dev environments remotely (or rewriting in Go) were the options being considered before the whole project was canned and people redistributed to other things.
Earlier quoted context omitted.
In my last role as a director of engineering at a startup, I found that a project `flake.nix` file (coupled with simply asking people to use https://determinate.systems/posts/determinate-nix-installer/ to install Nix) led to the fastest "new-hire-to-able-to-contribute" time of anything I've seen. Unfortunately, after a few hires (hand-picked by me), this is what happened: 1) People didn't want to learn Nix, neither d…
From my perspective, installing Nix seems pretty invasive. I can understand if someone doesn't want to mess with their system "unnecessarily" especially if the tool and it's workings are foreign. And I can't really remember the last time I had issues with non-deterministic dependencies either. Dependency versions are locked. Maybe I'm missing something?
I'm writing a test to check whether a tool I'm writing can work without Nix (it works with it perfectly, but I want it to also work without it because there are a lot of folks like you, and like me about 3 years ago, who still think they'd rather struggle with manually installing the right glibc that goes with the right python dependency installed with the right pip and venv versions, to the right location, that goes with the right python version that makes Whisper models work (literally the thing I'm currently working on), instead of just running `nix develop` and getting a coffee and then done.
And all I have to do to simulate "no Nix" is to remove all the nix paths from PATH (I suppose I could purge it from the linker paths as well, now that I think about it). But that's it.
What Nix does is put its entire repo into a separate part of your hard drive owned by root, and create a few build users for security reasons. That's (to me) not particularly "invasive," but YMMV (and if you use the Determinate Nix installer, it's even more trivial to uninstall than the official way). Also, when you run `nix develop`, the environment changes it does to make everything "just work" (like PATH changes etc) are only valid for that terminal session. Again, this is the least intrusive thing possible while also providing the guarantees it does, and is also (more or less) guaranteed to work.
The Nix whitepaper is pretty readable and not that long. I recommend it to understand why it's important and useful: https://edolstra.github.io/pubs/nspfssd-lisa2004-final.pdf
There is also Guix, which is like Nix but uses Guile (a Scheme dialect) as its scripting language all the way down to the bare metal (literally, the boot loader is written in it, I believe, as soon as the interpreter is loaded somehow). Their strategy seems to be to let Nix take the lead and make all the mistakes and then implement the way that seems to work the best, in its own ecosystem/tooling: https://guix.gnu.org/ But they have a lot fewer packages than Nix does.
Both of these let you define an entire machine with a single configuration file that is far more guaranteed to work than running a Dockerfile.