Can't say I'm working at scale but one of my product is latency sensitive and we stripped docker because it was slowing down each request.
I never really got to the bottom of it, someone linked me a bug in the interaction of docker / linux kernel (now fixed) which could have caused it, but I don't have time to waste chasing docker performance.
Ours is a fairly simple setup: one postgres db per machine, one python app per machine on $cheapVPSProvider; number of instances goes up and down based on traffic (basically cloning one of the machines); a load balancer in front; data gets updated once per day and replicated; auth / subscription status data is stored in redis
Can you elaborate more? From what I've generally heard, Nix has been a good foundation for a lot of companies.
As far as I know the final effect is great - getting reproducible, statically linked programs. However, learning Nix / NixOS is quite difficult - it requires a specific set of skills (functional programming, shell, Nix building blocks which you can learn from existing Nix code), the documentation is lacking, error messages are cryptic, debugging support is bad.
Yeah Nix is why "choose boring" is a bit of a misnomer.
The truth is tautologically unhelpful: choose good technologies.
Most people have trouble separating snake oil and fads from good things, so old/boring is a safe heuristic.
Nix is not boring, but it is honest. It's hard because it doesn't take half measures. It's not an easy learning curve gaslighting to you to a dead end.
But taste in choosing good technologies is hard to teach, especially when people's minds are warped by a shitty baseline.
We currently run a Java Monolith that is build on every push to master on Gitlab pipeline, when successful the fat JAR is copied to S3 (including all resources) and then a config file is pushed to S3 to ask DEV servers to run it.
The machines are pure EC2 instances with the service registered in systemd with auto-restart after 5s. A simple shell script downloads the config file, the fat JAR and runs it. We detect the environment and machine we're at using the EC2 meta- and user-data, which is set when the EC2 instance is launched. All of this is basically simple plain batch script using JQ and other standard tools.
It is a little more complicated, because we have graceful restart by first asking a specific instance in each cluster to update itself and when it reports that is is updated, the next instance will be asked to update.
All the EC2 instances in a cluster are behind a simple Global Accelerator and are in multiple regions. The service has an API that is invoked by Gitlab with a token to ask for graceful restart, then it will report back to GA that it is unhealthy for some time, so it can finish pending requests (close sockets) and then it simply terminates, the rest is back up to bash script and systemd.
To deploy or redeploy to an environment is as well a simply click in Gitlab UI and especially important for On-Call, so you can see which version was deployed to which environment when by whom. Additionally some JIRA tickets are automatically created.
Eventually this only needs Gitlab, EC2 instances, Global Accelerator and Bash scripts, while allow us to be multi-region and have stateful connections, we can even ask clients to directly connect to specific instances or ask GA to redirect certain ports to specific instances. Basically GA is our load balancer, router, edge location. It is stable, fast and easy with the smallest amount of pieces involved. We can remove individual instances, update one instances in a specific region for testing to a new version and so on.
Booking.com - at least until I left in 2018, not sure about now. Their code base is (still) mostly in Perl (5), running on uWSGI bare instances, installed on KVMs deployed on a self-hosted infrastructure.
Working in Perl in 2022, the horror, especially because most likely it's 10y/o perl version.
Booking has competent perl ops people - including people who contribute to the language core and toolchain - and keeps very much up to date.
Sane application scale perl is pretty much a different language to old school scripting perl - the two just happen to share a parser and a VM.
(not everything about Booking's code is necessarily sane, when you've got a codebase that's been running that long making heaps of money there's inevitably stuff people wish they could replace, but your 'most likely' simply isn't descriptive of most companies running OO perl application stacks in 2022 - the ones that wrote crap perl mostly already blamed perl and are now writing hopefully slightly less crap something else)
Hey former Guardian employee here. The Guardian has hundreds of servers running, pretty much all EC2 instances. EC2 images are baked and derived from official images, similarly to the way you bake a docker image. We built tools before docker became the de facto standard, so we could easily keep the EC2 images up to date. We integrated pretty well with AWS so that the basic constructs of autoscaling and load balancer…
> Now for a company that starts today I don't think I'd recommend that
> What works for us is to do the simplest thing that works, then iterate. The older I get, the more often I'm reminded that this un-sexy approach is really the best way to go. When I was younger, I always thought the old guys pushing boring solutions just didn't want to learn new things. Now I'm starting to realize that after several decades of experience, they simply got burned enough times to learn a thing or two h…
Or it could be that you just became old and don't want to learn new things anymore ))
Some old things really are better, or at least not worse and already familiar.
Docker is heavier (and more dangerous) because of dockerd, the management and api daemon that runs as root. Actual process isolation is handled by cgroup controls which are already built into the kernel and have been for years. You can apply them to any process, not just docker ones. However, Docker is essentially dead; the future is CRI-O or something similar which has no daemon and runs as an unprivileged user. And…
All the so-called "docker killers" are essentially unfinished products. They don't compare 1:1 to docker in feature set and even if they run as rootless, they still are vulnerable to namespace exploits in the Linux kernel. Though docker runs as root, it's still well protected out-of-the-box for the average user and is a very mature technology.
Are you from 2018? Everyone running OpenShift is using CRI-O and that footprint is not small. We made the switch in our EKS and vanilla k8s clusters in 2021. Docker has now even made their API OCI-compliant in order to not be left behind. And the point is that most people don't want a docker feature-for-feature running in prod. The attack surface is simply too large. I don't need an API server running as root on all my container hosts.
Use docker on your laptop, sure. Its time in prod is over.
Don't know if they still use it (I suspect so!) but at least as of 2015 Amazon was using a homebrewed deployment service called Apollo, which could spin up a VM from an internally developed Linux image then populate it with all the software and dependencies needed for a single service. It later inspired AWS CodeDeploy which does the same thing. I remember it being pretty irritating to use, though, since it wasn't par…
I've always thought of Apollo environments as containers before kernel features for containers existed. With enough environment variables and wrapper scripts taking the name of real binaries to populate stuff like LD_LIBRARY_PATH, Apollo makes a private environment that is only _slightly_ contaminated by the host.
Apollo environments try to solve the same problem, but they're definitely not containers - one cannot depend on any fixed paths. I still bear scars from wrestling with various packages' autotools / glib / Python path dependencies trying and hoping desperately to find poorly / undocumented environment variable overrides to get them to let go of a static, build-time specified path and play nicely with Apollo's environment path shell-game.
I agree. I've seen containerization mentioned by many vendors in the last year. It adds another layer of questions to vet like "do they possibly know what they're doing?"
Do you see shipping in containers as a sign that they do know what they're doing, or as a sign that they don't?
The key thing is that it's a sign that you're effectively going to be outsourcing a complete userland to them, which means you'll be much more dependent on the vendor for security updates to anything their code depends on as well as their code itself.
Whether or not this is a good idea depends on your situation and their level of competence.
As far as I know the final effect is great - getting reproducible, statically linked programs. However, learning Nix / NixOS is quite difficult - it requires a specific set of skills (functional programming, shell, Nix building blocks which you can learn from existing Nix code), the documentation is lacking, error messages are cryptic, debugging support is bad.
Yeah Nix is why "choose boring" is a bit of a misnomer. The truth is tautologically unhelpful: choose good technologies. Most people have trouble separating snake oil and fads from good things, so old/boring is a safe heuristic. Nix is not boring, but it is honest. It's hard because it doesn't take half measures. It's not an easy learning curve gaslighting to you to a dead end. But taste in choosing good technologies…
> It's not an easy learning curve gaslighting to you to a dead end.
This is a really good way of describing a certain type of technology. I'm stealing this line.