Up until recently, I was the steward of a large, distributed, and profitable app that used no containers. The infrastructure was managed with packages and puppet, and it worked well.
Ask HN: Who operates at scale without containers?
121–130 of 446 posts
Re: Ask HN: Who operates at scale without containers?
#122Many years ago I worked at a place that deployed thousands of bare metal servers. We were effectively running our own cloud before the cloud became a thing.
The way it worked was simple. We created our own Red Hat variant using a custom Anaconda script. We then used PXE boot. During OS install this script would call back to our central server to provision itself. You can do that a few ways. If I recall, we baked in a minimal set of packages into the ISO to get us up and then downloaded a provision script that was more frequently updated to finish the job.
This is still a fine way of handling horizontal SaaS type scaling, where you do a sort of white labeling of your service with one customer per VM. Swap Postgres/MySQL for SQLite on each node and everything is just stupidly simple.
Re: Ask HN: Who operates at scale without containers?
#123My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…
I asked the question without context because I didn't want to fire the thread off in the wrong direction but I suppose in a comment chain it's fine. For what it's worth we do use containers and K8s heavily at my current job. I know that there are quite a few people opposed to the state of containers and the technologies revolving around them. I don't think the arguments they present are bad. [Attacking the premise](…
> So essentially I'm interesting in knowing how people find ways to reproduce the value these technologies offer, what they instead rely on, which things they leave on the table
It's interesting that you're not interested in what they get by not using containers...
Re: Ask HN: Who operates at scale without containers?
#124depends on what kind of scale.
I used to make online games and our gameservers at launch were in the order of 100,000 physical CPU cores and about 640TiB of RAM spread across the world.
But we did this: On windows, with a homegrown stack, before kubernetes was a thing (or when it was becoming a thing).
With the advent of cloud we wrote a predictive autoscaler too. That was fun.
I don't work there anymore, and they moved to Linux, but they're hiring: https://www.massive.se/
You can learn a lot from a technical interview ;)
Re: Ask HN: Who operates at scale without containers?
#125My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…
> Containers only give reproducible deployments, not builds Could someone elaborate on this please? Doesn't it depend entirely on your stack how reproducible your build is? Say I have a Python app with its OS level packages installed into a (base) image and its Python dependencies specified in a Pipfile, doesn't that make it pretty reproducible? Is the weak spot here any OS dependencies being installed as part of the…
This is something that even the Docker core team is not entirely clear on (either they understand it and can't explain it, or they don't understand it).
The RUN command is Turing complete, therefore cannot be idempotent. Nearly every usable docker image and base image include the RUN command, often explicitly to do things that are not repeatable, like "fetch the latest packages from the repository".
This is all before you get to application code which may be doing the same non-repeatable actions with gems or crates or maven or node modules, or calling a service and storing the result in your build. Getting repeatable docker images usually is a matter of first getting your build system to be repeatable, and then expanding it to include the docker images. And often times standing up a private repository so you have something like a reliable snapshot of the public repository, one you can go back to if someone plays games with the published versions of things.
Re: Ask HN: Who operates at scale without containers?
#126My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…
> What works for us is to do the simplest thing that works, then iterate. The older I get, the more often I'm reminded that this un-sexy approach is really the best way to go. When I was younger, I always thought the old guys pushing boring solutions just didn't want to learn new things. Now I'm starting to realize that after several decades of experience, they simply got burned enough times to learn a thing or two h…
Or it could be that you just became old and don't want to learn new things anymore ))
Re: Ask HN: Who operates at scale without containers?
#127My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…
I asked the question without context because I didn't want to fire the thread off in the wrong direction but I suppose in a comment chain it's fine. For what it's worth we do use containers and K8s heavily at my current job. I know that there are quite a few people opposed to the state of containers and the technologies revolving around them. I don't think the arguments they present are bad. [Attacking the premise](…
What you're asking for is an essay on comparative devops architectures, with a focus on k8s alternatives. I think what you'll find is a lot of ad hoc persistent systems that tend to drift over time in unpredictable ways, and take on the feel of a public lobby if you're being generous, a public restroom if you're not. So what you're asking is really a sample of these ad hoc approaches. What I think you'll find are a few inspired gems, and about 1000 bad implementations of K8s, Docker, Salt, Terraform and all the rest.
The case that always interested me is the service that can really fit on one machine. Machines are truly gigantic in terms of memory and disk. Heck, I worked at BrickLink which ran on two beefy machines that serviced 300k active users (SQL Server is actually quite good, it turns out). (A single beefy server is a really great place to start iterating on application design! PostgreSQL + Spring Boot (+ React) is a pretty sweet stack, for example. By keeping it simple, there are so many tools you just don't need anymore, because their purpose is to recombine things you never chose to split. I can't imagine why you'd need more than 16 cores, 128G of RAM and 10T of storage to prove out an ordinary SaaS. That is a ferocious amount of resources.)
Re: Ask HN: Who operates at scale without containers?
#128Earlier quoted context omitted.
is Nix a hurdle when onboarding engineers? do only a few people need to know the language? I've wanted to use Nix and home-manager for personal stuff but the learning curve seems big.
It's a hurdle to be able to write Nix stuff, but to consume it is pretty easy. Shameless plug here: https://www.reactivated.io/documentation/why-nix/
I just tried Nix yesterday for the first time in order to play with reactivated. Was surprised by how simple it was to get it running. Made me think that perhaps containers are not as indispensable as I thought.
And reactivated is awesome, by the way. I just worry about how brittle it will be as both Django and React evolve.
Re: Ask HN: Who operates at scale without containers?
#129My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…
> That's the easy part! The hard part is managing databases. Ding ding ding ding ding. "But what about disk?" (so, relatedly, databases) is the hard part. Balancing performance (network disks suuuuuck) and flexibility ("just copy the disk image to another machine" is fine when it's a few GB—less useful when it's lots of GB and you might need to migrate it to another city and also you'd rather not have much downtime)…
Sometimes I think K8s is largely pushed for Amazon/Google/Microsoft to sell the disk that goes along with it.
Re: Ask HN: Who operates at scale without containers?
#130My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…
The thing is, you spent probably a lot of time on things that were granted elsewhere and while the rest of the world is improving on those tech you keep your home grown solution that is harder and harder to maintain. Plus the knowledge that is not transferable. > Containers only give reproducible deployments, not builds, so they would be a step down. This is not true, if you use a docker image A with specific version…
I also don’t agree with the concept, and I’ve had problems about with docket using a specific OS version but pulling in some minor software version which broke, so I think even if being slightly true, that statement represent something that might be solved by specifying minor versions for the packages you depend on, which is a level of effort compatible with the one you need on nix to specify packages hashes, so you can have both with docker, a strict and non strict approach, while I guess nix only supports strict