Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who operates at scale without containers?

news.ycombinator.com

Post reply on HN
Print view View original

311–320 of 446 posts

Re: Ask HN: Who operates at scale without containers?

#311

Post by tptacek »

po_ta_toes wrote: post #293
tptacek wrote: post #212

Ironically, here at Fly.io, we run containers (in single-use VMs) for our customers, but none of our own infrastructure is containerized --- though some of our customer-facing stuff, like the API server, is. We have a big fleet of machines, mostly in two roles (smaller traffic-routing "edge" hosts that don't run customer VMs, and chonky "worker" hosts that do). All these hosts run `fly-proxy`, a Rust CDN-style proxy…

Hi there, Very interesting read! I work for a large news org, The team I'm in primarily uses elixir, which I know the people at fly.io love too! Why did you decide not to containerise your own infrastructure? We use some 'chonky' ec2s but are thinking about using containers. Given that the BEAM has quite a large footprint, do you think it still a good candidate for containers, or would that introduces too much overhe…

We would containerize everything if we could! But our infrastructure components usually live outside the security boundaries our container interface sets up. The proxy has to be able to talk to every app, not just apps in its own organization, and it needs direct access to other infrastructure components. The orchestration code needs to be able to launch Firecracker VMs --- it can't itself be a Firecracker VM.

We can design around all these constraints! We just haven't gotten around to it yet. I think there's a general consensus that hiding things inside Firecrackers is a good design, and we'll do it where we can.

Elixir runs great inside Firecrackers. The idea behind Firecracker is that it introduces a minimal load, by dint of being ruthlessly simple. When you run an Elixir app on an EC2 instance, you're running hypervised as well (and probably by a hypervisor that's less efficient --- not a knock, Firecracker is Amazon code too).

The thing that makes this doable at Fly.io is that we run our own hardware, so we're not trying to nest VMs inside VMs. Of course, that's what EC2 is doing too. So whatever your intuitions are about things that run well on vanilla EC2, they should carry over to us.

Re: Ask HN: Who operates at scale without containers?

#312

Post by kazinator »

smilliken wrote: post #225
jka wrote: post #218

Earlier quoted context omitted.

Do your developers run the same Nix packages that you deploy to production? (that sounds like a energy-level-transition in developer productivity and debugging capability, if so)

Yeah, the environment is bit-for-bit identical in dev and prod. Any difference is an opportunity for bugs. OK, there's one concession, there's an env var that indicates if it's a dev and prod environment. We try to use it sparingly. Useful for stuff like not reporting exceptions that originate in a dev environment. Basically, there's a default.nix file in the repo, and you run nix-shell and it builds and launches you…

Does that mean you turn off security-related randomizations in everything, like address space randomization and hash table randomization?

Re: Ask HN: Who operates at scale without containers?

#313

Post by topspin »

benatkin wrote: post #114
topspin wrote: post #8

Earlier quoted context omitted.

The commercial, licensed software I've dealt with in the past year has all been containerized.

OCI images don't necessarily mean lxc/docker style containers. Many are run directly on MicroVMs. https://fly.io/blog/docker-without-docker/ Ditto Google Cloud Run.

The three products I have in mind were are all conventional 'lxc/docker' containers. Two are provided with docker-compose scripts. Two are available as either containerized or traditional products. One is container only.

Re: Ask HN: Who operates at scale without containers?

#314

Post by [deleted] »

smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

[deleted]

Re: Ask HN: Who operates at scale without containers?

#315

Post by jcadam »

bogomipz wrote: post #68
jcadam wrote: post #61

Is using CUDA inside a container still a massive PITA?

Can you elaborate on what makes containerized CUDA so difficult?

My last project where we needed CUDA, we ended running our GPU workloads directly on a dedicated machine loaded up with GPUs. The rest of the system was containerized. Though, we were just a small team of software engineers working on an R&D project - we didn't have any k8s wizards on our team.

Re: Ask HN: Who operates at scale without containers?

#316

Post by spion »

treis wrote: post #247
hinkley wrote: post #125

Earlier quoted context omitted.

This is something that even the Docker core team is not entirely clear on (either they understand it and can't explain it, or they don't understand it). The RUN command is Turing complete, therefore cannot be idempotent. Nearly every usable docker image and base image include the RUN command, often explicitly to do things that are not repeatable, like "fetch the latest packages from the repository". This is all befor…

>Getting repeatable docker images But there's no real need for repeatable build docker images. You copy the image and run it where ever you need to. The entire point of Docker is to not have to repeat the build.

Until there is a critical vulnerability in one of the components present on that image (system packages or application packages).

Re: Ask HN: Who operates at scale without containers?

#318

Post by gabrielgrant »

senorsmile wrote: post #297
zthrowaway wrote: post #138

Earlier quoted context omitted.

People still think stateful things are impossible on k8s but Stateful sets and persistent volumes solves a lot of this. You should be relying on out of the box DB replication to make sure data is available in multiple areas. This is no different on other platforms.

Kelsey Hightower just had a good interview where he disagrees with you: https://changelog.com/shipit/44 Edit: somewhat* disagrees with you. It's a good listen.

tl;dr?

Re: Ask HN: Who operates at scale without containers?

#319

Post by qwertywert_ »

richardwhiuk wrote: post #233
p_l wrote: post #221

Earlier quoted context omitted.

One of the biggest benefits of k8s for me, back in 2016 when I first used it in prod, was that it threw away all the extra features of Docker and implemented them directly by itself - better. Writing was already in the wall that docker will face stern competition that doesn't have all of its accidental complexity (rktnetes and hypernetes were a thing already)

Kubernetes used to (tediously) pass everything through to Docker, but since 1.20, that's resolved, and it now uses containerd.

I thought its moving to CRI-o as well instead of containerd, or is k8s just not using docker, and containerd still supported in the future?

Re: Ask HN: Who operates at scale without containers?

#320

Post by oneplane »

One of my customers used to deploy on "Application Servers" which were essentially just VMs with CentOS and Tomcat. You push a WAR to an HTTP endpoint and that's that. The servers themselves were just images made with Packer and Ansible (two staged; first get a public image, update and preconfigure it, then the second stage templates Tomcat). While running ~1K nodes that way works, the problem wasn't "can it do the job" but "can we find enough people with knowledge and will this scale with more people". The answer was no and migration to first Mesos and then Kubernetes was done.

For developers, not much changed in the end (push code, receive rollout progress), technically, traffic is still the same too (external -> Load Balancer -> Application), but the glue is much more 'standard' making it way easier to support, buy support, and hire people that already know how it works.

Our "Big Data" is just a fat cluster of machines with Apache Spark, no containers. Technically we could add a container in there somewhere, but there is no benefit, so we just build server images that do exactly the same thing. I think that containers only make sense if you need to run multiple things per node, or if you want things that you can re-use (knowledge, existing software, local vs. remote environments). For everything else it doesn't really matter, a static binary does the same thing. That said, the people finishing software engineering school hardly know how to compile their down code, let alone link it statically.

While containers can solve a technical problem, they are mostly solving organisational and knowledge problems. Applies to microservices in some scenarios as well.

Post reply on HN