StackOverflow is known to operate their own servers on their own datacenters and as far I recently read, they are still tuning with no container tech.
https://nickcraver.com/blog/2016/05/03/stack-overflow-how-we...
Ask HN: Who operates at scale without containers?
171–180 of 446 posts
Re: Ask HN: Who operates at scale without containers?
#172Depends what you mean by 'container runtime' or 'container orchestration tool'... For example, Google's Borg absolutely uses Linux namespacing for its workloads, and these workloads get scheduled automatically on arbitrary nodes, but this doesn't feel at all like Docker/OCI containers (ie., no whole-filesystem image, no private IP address to bind to, no UID 0, no control over passwd...). Instead, it feels much closer…
> no whole-filesystem image
At least in the past, almost all jobs ran in their own private filesystem - it was stitched together in userspace via bind mounts rather than having the kernel do it with an overlayfs extracted from layer tar files (since overlayfs didn't exist back then), but the result was fairly similar.
Most jobs didn't actually request any customization so they ended up with a filesystem that looked a lot like the node's filesystem but with most of it mounted read-only. But e.g. for a while anything running Java needed to include in their job definition an overlay that updated glibc to an appropriate version since the stock Google redhat image was really old.
Re: Ask HN: Who operates at scale without containers?
#173Stack Overflow/Stack Exchange maybe?
Re: Ask HN: Who operates at scale without containers?
#174Earlier quoted context omitted.
Docker is far heavier - the overhead is the flexibility and process isolation you get. I imagine that's really useful for certain types of workloads (e.g. an ETL pipeline), but is crazy inefficient for something single purpose like a web app.
Docker is heavier (and more dangerous) because of dockerd, the management and api daemon that runs as root. Actual process isolation is handled by cgroup controls which are already built into the kernel and have been for years. You can apply them to any process, not just docker ones. However, Docker is essentially dead; the future is CRI-O or something similar which has no daemon and runs as an unprivileged user. And…
All the so-called "docker killers" are essentially unfinished products. They don't compare 1:1 to docker in feature set and even if they run as rootless, they still are vulnerable to namespace exploits in the Linux kernel. Though docker runs as root, it's still well protected out-of-the-box for the average user and is a very mature technology.
Re: Ask HN: Who operates at scale without containers?
#175Earlier quoted context omitted.
> Chose boring technology Doesn't fit in this case, I'd say Nix is still cutting edge. It may also be simple, but it's not easy (in the Rich Hickey sense).
Can you elaborate more? From what I've generally heard, Nix has been a good foundation for a lot of companies.
As far as I know the final effect is great - getting reproducible, statically linked programs.
However, learning Nix / NixOS is quite difficult - it requires a specific set of skills (functional programming, shell, Nix building blocks which you can learn from existing Nix code), the documentation is lacking, error messages are cryptic, debugging support is bad.
Re: Ask HN: Who operates at scale without containers?
#176My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…
> What works for us is to do the simplest thing that works, then iterate. The older I get, the more often I'm reminded that this un-sexy approach is really the best way to go. When I was younger, I always thought the old guys pushing boring solutions just didn't want to learn new things. Now I'm starting to realize that after several decades of experience, they simply got burned enough times to learn a thing or two h…
Jonathan blow had a great take on this that really spoke to me. I can’t do it justice, but paraphrasing as best I can:
Get the simple to understand, basic thing working and push the harder refactor / abstraction until later. Leave the fancy stuff for a future engineer who better understands the problem.
That future engineer is you with more experience with the actual problem.
Re: Ask HN: Who operates at scale without containers?
#177Earlier quoted context omitted.
> Containers only give reproducible deployments, not builds Could someone elaborate on this please? Doesn't it depend entirely on your stack how reproducible your build is? Say I have a Python app with its OS level packages installed into a (base) image and its Python dependencies specified in a Pipfile, doesn't that make it pretty reproducible? Is the weak spot here any OS dependencies being installed as part of the…
This is something that even the Docker core team is not entirely clear on (either they understand it and can't explain it, or they don't understand it). The RUN command is Turing complete, therefore cannot be idempotent. Nearly every usable docker image and base image include the RUN command, often explicitly to do things that are not repeatable, like "fetch the latest packages from the repository". This is all befor…
pedantic nit-pick that doesn't detract from your main point: Turing completeness doesn't imply that it can never be idempotent. In fact we'd expect that a particular Turing machine given a particular tape should reliably produce the same output, and similarly any Turing-complete program given the exact same input should produce the exact same output. Turing machines are single-threaded, non-networked machines with no concept of time.
The main reason the RUN statement isn't idempotent, in my opinion, is that it can reach out to the network and e.g. fetch the latest packages from a repository, like you mention. (Other things like the clock, race conditions in multi-threaded programs, etc. can also cause RUN statements to do different things on multiple runs, but I'd argue those are relatively uncommon and unimportant.)
Re: Ask HN: Who operates at scale without containers?
#178Earlier quoted context omitted.
> What works for us is to do the simplest thing that works, then iterate. The older I get, the more often I'm reminded that this un-sexy approach is really the best way to go. When I was younger, I always thought the old guys pushing boring solutions just didn't want to learn new things. Now I'm starting to realize that after several decades of experience, they simply got burned enough times to learn a thing or two h…
Jonathan blow had a great take on this that really spoke to me. I can’t do it justice, but paraphrasing as best I can: Get the simple to understand, basic thing working and push the harder refactor / abstraction until later. Leave the fancy stuff for a future engineer who better understands the problem. That future engineer is you with more experience with the actual problem.
The way I see it, I can either prematurely use the complex solution where I don't fully understand the problem or the solution OR I use the simple solution and learn why the complex solution is needed and more importantly what parameters and specific ways I can best leverage the complex solution for this specific problem.
Re: Ask HN: Who operates at scale without containers?
#179when I was at Jackbox we ran the multiplayer servers without containers and handled hundreds of thousands of simultaneous websocket connections (a few thousand per node). The servers were statically compiled Go binaries that took care of their own isolation at the process level and didn't write to disk, they were just running as systemd services. Game servers are inherently stateful, they're more like databases than…
Interesting -- how did you handle redeploys? Given that game servers are stateful (so you'd want to drain servers at their own pace instead of force them down at a specific time), it seems like redeploying a server without machinery to do things like dynamically allocate ports/service discovery for an upstream load balancer would be tricky.
> it seems like redeploying a server without machinery to do things like dynamically allocate ports/service discovery for an upstream load balancer would be tricky
like most things with running servers, it's not that hard, there's just an industry dedicated to making people think it's hard (the tech industry).
Every game has a room code to identify the game, and a websocket open handshake has a URL in it. Every room listens on a unix domain socket and nginx just routes the websocket to the correct domain socket based on the URL.
Re: Ask HN: Who operates at scale without containers?
#180My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…
Any hints in how use nix with DBs?
I have a semi-docker setup where I stay with the base OS + PostgreSQL and the rest is docker. But that means that upgrade the OS/DB is not that simple.
I tried before dockerize PG, but still the OS need management anyway and was harder to use diagnostics/psql/backups with docker....