Live data from Hacker News

Ask HN: Who operates at scale without containers?

news.ycombinator.com

51–60 of 446 posts

Re: Ask HN: Who operates at scale without containers?

#51
I worked at WhatsApp, prior to moving to Facebook infra, we had some jails for specific things, but mostly ran without containers.

Stack looked like:

FreeBSD on bare metal servers (host service provided a base image, our shell script would fetch source, apply patches, install a small handful of dependencies, make world, manage system users, etc)

OTP/BEAM (Erlang) installed via rsync etc from build machine

Application code rsynced and started via Makefile scripts

Not a whole lot else. Lighttpd and php for www. Jails for stud (a tls terminator, popular fork is called hitch) and ffmpeg (until end to end encrypted media made server transcoding unpossible).

No virtualized servers (I ran a freebsd vm on my laptop for dev work, though).

When WA moved to Facebook infra, it made sense to use their deployment methodology for the base system (Linux containers), for organizational reasons. There was no consideration for which methodology was technically superior; both are sufficient, but running a very different methodology inside a system that was designed for everyone to use one methodology is a recipie for operational headaches and difficulty getting things diagnosed and fixed as it's so tempting to jump to the conclusion that any problem found on a different setup is because of the difference and not a latent problem. We had enough differences without requiring a different OS.

Re: Ask HN: Who operates at scale without containers?

#52
post #32

I work for one of the largest enterprise productivity companies out there. We have tens of thousands of self-managed EC2 VMs, no containers. We use Chef and Terraform for the most part.

Are you maintaining AMIs or using a tool like Ansible? For instance, how do you apply security patches to 10,000 instances?

Re: Ask HN: Who operates at scale without containers?

#53

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

> Containers only give reproducible deployments, not builds

Could someone elaborate on this please? Doesn't it depend entirely on your stack how reproducible your build is? Say I have a Python app with its OS level packages installed into a (base) image and its Python dependencies specified in a Pipfile, doesn't that make it pretty reproducible?

Is the weak spot here any OS dependencies being installed as part of the continuous build?

Re: Ask HN: Who operates at scale without containers?

#54

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

> Containers only give reproducible deployments, not builds Could someone elaborate on this please? Doesn't it depend entirely on your stack how reproducible your build is? Say I have a Python app with its OS level packages installed into a (base) image and its Python dependencies specified in a Pipfile, doesn't that make it pretty reproducible? Is the weak spot here any OS dependencies being installed as part of the…

I think that's what they're saying, that containers are orthogonal to reproducible builds.

Re: Ask HN: Who operates at scale without containers?

#55
We use ansible on bare metal (no VM) to manage about 200 servers in our basement. We use PXE booting to manage the images. We use a customized arch linux image and we have a few scripts to select what feature we'd like. It's "old school" but it's been working fine for nearly 20 years (we used plain scripts before ansible, so we always used the "agentless" approach). Our networking stack uses OpenBSD.

Re: Ask HN: Who operates at scale without containers?

#56
post #20
post #3

Back in 2016 at least, Stack overflow was container free https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... No idea how much has changed since then

That's the first thing that came to mind as well, and Stack Overflow definitely operates at scale, but OP asked for people doing distributed systems. SO is famously monolithic, famously operating from just a handful of machines.

I think this was changed a bit since moving to dot net core ( not sure).

Re: Ask HN: Who operates at scale without containers?

#57
post #51

I worked at WhatsApp, prior to moving to Facebook infra, we had some jails for specific things, but mostly ran without containers. Stack looked like: FreeBSD on bare metal servers (host service provided a base image, our shell script would fetch source, apply patches, install a small handful of dependencies, make world, manage system users, etc) OTP/BEAM (Erlang) installed via rsync etc from build machine Application…

That's a very cool deployment method to just use rsync like that! Simple and very composable.

And now I feel self-conscious about my pile of AWS and Docker!

Re: Ask HN: Who operates at scale without containers?

#58
Is your question "How do I write software without tons of dependencies?" or "How do I ship software that has tons of dependencies but without shipping the dependencies with them?" or "How do I ship software with tons of dependencies without using containers or VMs?"

Re: Ask HN: Who operates at scale without containers?

#59
when I was at Jackbox we ran the multiplayer servers without containers and handled hundreds of thousands of simultaneous websocket connections (a few thousand per node). The servers were statically compiled Go binaries that took care of their own isolation at the process level and didn't write to disk, they were just running as systemd services. Game servers are inherently stateful, they're more like databases than web application layer servers. For large audience games I wrote a peering protocol and implemented a handful of CRDT types to replicate the state, so it was a hand-rolled distributed system. Most things were handled with chef, systemd, terraform, and aws autoscaling groups.

Re: Ask HN: Who operates at scale without containers?

#60
post #32

I work for one of the largest enterprise productivity companies out there. We have tens of thousands of self-managed EC2 VMs, no containers. We use Chef and Terraform for the most part.

Are you maintaining AMIs or using a tool like Ansible? For instance, how do you apply security patches to 10,000 instances?

I'm not too involved with the ops side, but I believe Chef handles it.
Post reply on HN