Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who operates at scale without containers?

news.ycombinator.com

Post reply on HN
Print view View original

211–220 of 446 posts

Re: Ask HN: Who operates at scale without containers?

#211

Post by _-david-_ »

gen220 wrote: post #97
tailspin2019 wrote: post #17

Earlier quoted context omitted.

I’m in the process of moving to exactly this approach. I’ve been trying to pick the right Linux distro to base my images on. Ubuntu Server is the low effort route but a bit big to redeploy constantly. I’ve also been looking at the possibility of using Alpine Linux which feels like a better fit but a bit more tweaking needed for compatibility across cloud providers. Unikernels are also interesting but I think that mig…

Google has used debian as their base. Netflix uses a BSD flavor (I forget which) as their CDN cache. FB used CentOS, not sure what they use today since CentOS is EOL'd. Debian (and, formerly, CentOS) is a good standard: it occupies a sweet spot between ubuntu server and alpine, in the sense that it's batteries-included and very well-supported (apt/yum), but not particularly bloated. I use debian for all my personal s…

>FB used CentOS, not sure what they use today since CentOS is EOL'd.

They are using CentOS Stream now.

Re: Ask HN: Who operates at scale without containers?

#212

Post by tptacek »

Ironically, here at Fly.io, we run containers (in single-use VMs) for our customers, but none of our own infrastructure is containerized --- though some of our customer-facing stuff, like the API server, is.

We have a big fleet of machines, mostly in two roles (smaller traffic-routing "edge" hosts that don't run customer VMs, and chonky "worker" hosts that do). All these hosts run `fly-proxy`, a Rust CDN-style proxy server we wrote, and `attache`, a Consul-to-sqlite mirroring server we built in Go. The workers also run our orchestration code, all in Go, and Firecracker (which is Rust). Workers and WireGuard gateways run a Go DNS server we wrote that syncs with Consul. All these machines are linked together in a WireGuard mesh managed in part by Consul.

The servers all link to our logging and metrics stack with Vector and Telegraf; our core metrics stack is another role of chonky machines running VictoriaMetrics.

We build our code with a Buildkite-based CI system and deploy with a mixture of per-project `ctl` scripts and `fcm`, our in-house Ansible-like. Built software generally gets staged on S3 and pulled by those tools.

Happy to answer any questions you have. I think we fit the bill of what you're asking about, even though if you read the label on our offering you'd get the opposite impression.

Re: Ask HN: Who operates at scale without containers?

#213

Post by tailspin2019 »

gen220 wrote: post #186
tailspin2019 wrote: post #142

Earlier quoted context omitted.

Useful insight, thanks. What attracts you to BSD over Debian if you were to go that route at some point?

No problem! If you search around for "Debian vs BSD", you'll find more exhaustive explanations [1], but it mostly reduces to the fact that BSD is more coherent & organized than GNU/Linux; it's more feature-rich in the domains that sysadmins and hackers appreciate, but feature-sparse in the domains "normal users" appreciate. Depending on your needs these facts can be advantages or disadvantages. I tend to gravitate to…

Very interesting. I knew very little about BSD but your and mst’s comments have convinced me that I need to take a closer look.

Thanks for the link - I’m taking a look at that now!

Re: Ask HN: Who operates at scale without containers?

#214

Post by weego »

alex_duf wrote: post #45

Hey former Guardian employee here. The Guardian has hundreds of servers running, pretty much all EC2 instances. EC2 images are baked and derived from official images, similarly to the way you bake a docker image. We built tools before docker became the de facto standard, so we could easily keep the EC2 images up to date. We integrated pretty well with AWS so that the basic constructs of autoscaling and load balancer…

Also ex employee. Riff Raff is absolutely still an excellent mod for build and deploy. At the time I was there it the initial stack build via handwritten cloudformation script that was the friction and pain point.

Re: Ask HN: Who operates at scale without containers?

#215

Post by lamontcg »

motoboi wrote: post #80
freeqaz wrote: post #57

Earlier quoted context omitted.

That's a very cool deployment method to just use rsync like that! Simple and very composable. And now I feel self-conscious about my pile of AWS and Docker!

Containers are just tar.gz files, you know? The whole layers thing it’s just an optimization. You can actually very simply run those tar.gz files without docker involved, just cgroups. But then you’ll have to write some daemon scripts to start, stop, restart, etc Follow this path and soon you’ll have a (worst) custom docker. Try to create a network out of those containers and soon a (worst) SDN network appears. Try t…

"worst" might not really be all that bad.

K8s may be the "worst" orchestrator for you, and you may not actually need all or any of that complexity/functionality.

But starting with containers viewed as an RPM/tarball with some extra sauce (union filesystem and cgroups/jail) is a way better mental model than the whole "immutable infrastructure" meme to me.

Re: Ask HN: Who operates at scale without containers?

#216

Post by lamontcg »

cheeze wrote: post #193
whateveracct wrote: post #94

Earlier quoted context omitted.

Apollo reminded me more of Nix than containers. The wrapper scripts are super Nix-y :)

That's what it was. VM with a barebones deployment system that had a ton of hooks in it. Really really smart idea that IMO helped Amazon in the 2010s immensely. While everyone else was figuring out k8s and whatnot, Amazon had a good system with CI in place for years. I wonder how it's fared over time. Amazon was never known for internal tooling in many other places. I hope Apollo is still running strong today.

Anyone remember disco and third-party packages, guam and cmf?

Re: Ask HN: Who operates at scale without containers?

#217

Post by zemo »

pipe_connector wrote: post #196
zemo wrote: post #179

Earlier quoted context omitted.

> it seems like redeploying a server without machinery to do things like dynamically allocate ports/service discovery for an upstream load balancer would be tricky like most things with running servers, it's not that hard, there's just an industry dedicated to making people think it's hard (the tech industry). Every game has a room code to identify the game, and a websocket open handshake has a URL in it. Every room…

How do you handle deploying a new version of nginx without forcibly closing connections?

you don't. That requires new nodes, so you have to drain nodes and replace them with new nodes that have the upgrade installed before adding them to the pool that hosts games. That process sucked but how often are you upgrading nginx?

Re: Ask HN: Who operates at scale without containers?

#218

Post by jka »

smilliken wrote: post #134
barbazoo wrote: post #53

Earlier quoted context omitted.

> Containers only give reproducible deployments, not builds Could someone elaborate on this please? Doesn't it depend entirely on your stack how reproducible your build is? Say I have a Python app with its OS level packages installed into a (base) image and its Python dependencies specified in a Pipfile, doesn't that make it pretty reproducible? Is the weak spot here any OS dependencies being installed as part of the…

You already got a few good answers, but I'll echo them: you can do reproducible builds in containers, and nothing's stopping you from using nix inside containers. But you're at the mercy of all the different package managers that people will end up using (apt, npm, pip, make, curl, etc). So your system is only as good as the worst one. I inherited a dozen or so docker containers a while back that I tried to maintain.…

Do your developers run the same Nix packages that you deploy to production?

(that sounds like a energy-level-transition in developer productivity and debugging capability, if so)

Re: Ask HN: Who operates at scale without containers?

#219

Post by smilliken »

smilliken wrote: post #104
ixxie wrote: post #63

Earlier quoted context omitted.

I would love to hear more about your architecture and deployment... do your services run as NixOS modules? Are you using NixOps or Morph or something else? How does your world look like without K8s and Containers?

For the first 6 years of using Nix, it was depoyed on Ubuntu. We recently migrated to NixOS. NixOS is fantastic for other reasons that are similar but separate from Nix as a package manager/build system. It's easier to incrementally switch to Nix first then NixOS later. We don't use systemd services for our code. We only use NixOS as an operating system, not for application layer. Our code works just the same on any…

> Are you using NixOps or Morph or something else? How does your world look like without K8s and Containers?

Not using NixOps or Morph. I recall considering a few others as well. In each case, I wasn't able to understand what they were doing that I couldn't do with a simple script. Instead, there's a python program that does an rsync, installs some configuration files, and runs the nix build. It deploys to any linux OS, but it does some stuff differently for NixOS, and the differences account for about 100 lines while increasing scope (managing filesystems, kernel versions, kernel modules, os users, etc). A deploy to all hosts takes about a minute because it runs in parallel. Deploys are zero downtime, including restarting a bunch of web apps.

The NixOS configuration itself is another ~200 lines, plus a bit per host for constants like hostname and ip addresses. It's really neat being able to define an entire operating system install in a single configuration file that gets type checked.

Re: Ask HN: Who operates at scale without containers?

#220

Post by mijoharas »

camtarn wrote: post #41

Don't know if they still use it (I suspect so!) but at least as of 2015 Amazon was using a homebrewed deployment service called Apollo, which could spin up a VM from an internally developed Linux image then populate it with all the software and dependencies needed for a single service. It later inspired AWS CodeDeploy which does the same thing. I remember it being pretty irritating to use, though, since it wasn't par…

Not too much of an update, but they were still using it in 2017.
Post reply on HN