Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who operates at scale without containers?

news.ycombinator.com

Post reply on HN
Print view View original

181–190 of 446 posts

Re: Ask HN: Who operates at scale without containers?

#181

Post by chousuke »

xmprt wrote: post #178
0x20cowboy wrote: post #176

Earlier quoted context omitted.

Jonathan blow had a great take on this that really spoke to me. I can’t do it justice, but paraphrasing as best I can: Get the simple to understand, basic thing working and push the harder refactor / abstraction until later. Leave the fancy stuff for a future engineer who better understands the problem. That future engineer is you with more experience with the actual problem.

The way I see it, I can either prematurely use the complex solution where I don't fully understand the problem or the solution OR I use the simple solution and learn why the complex solution is needed and more importantly what parameters and specific ways I can best leverage the complex solution for this specific problem.

Simple solutions are also easier to refactor into more complex solutions when the complexity becomes necessary. Going the other way is much harder.

In simple systems, you often have the option of simply throwing away entire components and remaking them because the system is simple and the dependencies between components are still clear.

Re: Ask HN: Who operates at scale without containers?

#182

Post by freemint »

Almost every HPC center. Tech stack: Linux (RHEL-like); MPI as middle ware for distributed communication through vendor specific communication hardware (also called interconnect); shared high performance network filesystem usually setup on login node, scheduler like SLURM, IBM Spectrum LSF Suites or others to launch jobs from login node which accessed via SSH. This setup scales to tens of thousands of machines.

Re: Ask HN: Who operates at scale without containers?

#183

Post by silviogutierrez »

olifante wrote: post #128
silviogutierrez wrote: post #50

Earlier quoted context omitted.

It's a hurdle to be able to write Nix stuff, but to consume it is pretty easy. Shameless plug here: https://www.reactivated.io/documentation/why-nix/

I just tried Nix yesterday for the first time in order to play with reactivated. Was surprised by how simple it was to get it running. Made me think that perhaps containers are not as indispensable as I thought. And reactivated is awesome, by the way. I just worry about how brittle it will be as both Django and React evolve.

Thanks! Hopefully not brittle at all. Specially since both React and Django are very stable projects, the latter even more than the former.

Re: Ask HN: Who operates at scale without containers?

#184

Post by mst »

gen220 wrote: post #97
tailspin2019 wrote: post #17

Earlier quoted context omitted.

I’m in the process of moving to exactly this approach. I’ve been trying to pick the right Linux distro to base my images on. Ubuntu Server is the low effort route but a bit big to redeploy constantly. I’ve also been looking at the possibility of using Alpine Linux which feels like a better fit but a bit more tweaking needed for compatibility across cloud providers. Unikernels are also interesting but I think that mig…

Google has used debian as their base. Netflix uses a BSD flavor (I forget which) as their CDN cache. FB used CentOS, not sure what they use today since CentOS is EOL'd. Debian (and, formerly, CentOS) is a good standard: it occupies a sweet spot between ubuntu server and alpine, in the sense that it's batteries-included and very well-supported (apt/yum), but not particularly bloated. I use debian for all my personal s…

Google's debian-based distroless is fascinating: https://github.com/GoogleContainerTools/distroless

Re: Ask HN: Who operates at scale without containers?

#185

Post by 0xbadcafebee »

I mean, sure, any statically compiled application can be deployed at scale without any dependencies at all. If you don't consider the application's 1GB worth of compiled-in SDKs and libraries to be bundled dependencies. :)

Back in the day we used to continuously deploy to thousands of servers without VMs or containers. Probably the largest-traffic sports site of the 2000s. But the application was mostly mod_perl, so dependencies still had to be managed, and it was intermittently a tire fire. There was no abstraction to run the applications immutably or idempotently. We actually acquired a company that had built a whole immutable build/deploy system around RPMs, so things became more predictable, but still host OS limitations would cause random bugs and the management interface was buggy. Re-bootstrapping hosts to shift load in the middle of peak traffic is a huge pain.

Containers would have been great. We could've finally ditched Cfengine2 and most of the weird custom host-level configuration magic and just ship applications without fear that something that worked in dev would break in prod due to a configuration or build issue. We also could have changed load patterns nearly instantly, which you can't do with VMs as you have to spin up new VMs to have a new host OS (not that we had VMs, what a luxury!)

Re: Ask HN: Who operates at scale without containers?

#186

Post by gen220 »

tailspin2019 wrote: post #142
gen220 wrote: post #97

Earlier quoted context omitted.

Google has used debian as their base. Netflix uses a BSD flavor (I forget which) as their CDN cache. FB used CentOS, not sure what they use today since CentOS is EOL'd. Debian (and, formerly, CentOS) is a good standard: it occupies a sweet spot between ubuntu server and alpine, in the sense that it's batteries-included and very well-supported (apt/yum), but not particularly bloated. I use debian for all my personal s…

Useful insight, thanks. What attracts you to BSD over Debian if you were to go that route at some point?

No problem! If you search around for "Debian vs BSD", you'll find more exhaustive explanations [1], but it mostly reduces to the fact that BSD is more coherent & organized than GNU/Linux; it's more feature-rich in the domains that sysadmins and hackers appreciate, but feature-sparse in the domains "normal users" appreciate. Depending on your needs these facts can be advantages or disadvantages.

I tend to gravitate towards projects like BSD, since they align with my principles, but you do pay a cost in terms of compatibility with common software.

For example, Firefox does not treat BSD as a first class build target, so it's up to community members to build and deploy Firefox binaries, and report feedback on bugs that break BSD installations.

If access to the most up-to-date compiled versions of popular software packages is a big sticking point, BSD-land may not be the right choice. But if you're living mostly in `vim`, `bash` and `man`, and are willing to roll up the sleeves, BSD feels cozy.

It's maybe a 20% correct analogy (don't read too deeply into it, or you'll draw incorrect conclusions), but C++ is to GNU/Linux, as D/Rust/Zig is to BSD.

[1]: https://unixsheikh.com/articles/technical-reasons-to-choose-...

Re: Ask HN: Who operates at scale without containers?

#187

Post by nunez »

smilliken wrote: post #134
barbazoo wrote: post #53

Earlier quoted context omitted.

> Containers only give reproducible deployments, not builds Could someone elaborate on this please? Doesn't it depend entirely on your stack how reproducible your build is? Say I have a Python app with its OS level packages installed into a (base) image and its Python dependencies specified in a Pipfile, doesn't that make it pretty reproducible? Is the weak spot here any OS dependencies being installed as part of the…

You already got a few good answers, but I'll echo them: you can do reproducible builds in containers, and nothing's stopping you from using nix inside containers. But you're at the mercy of all the different package managers that people will end up using (apt, npm, pip, make, curl, etc). So your system is only as good as the worst one. I inherited a dozen or so docker containers a while back that I tried to maintain.…

I don't know Nix and can't comment on that, but in my experience, when I've inherited containers that couldn't build, this was usually due to its image orphaned from their parent Dockerfiles (i.e. someone wrote a Dockerfile, pushed an image from said Dockerfile, but never committed the Dockerfile anywhere, so now the image is orphaned and unreproducable) or due to the container being mutated after being brought up with `docker exec` or similar.

Assuming that the container's Dockerfile is persisted somewhere in source control, the base image used by that Dockerfile is tagged with a version whose upstream hasn't changed, and that the container isn't modified from the image that Dockerfile produced, you get extremely reproducable builds with extremely explicit dependencies therein.

That said, I definitely see the faults in all of this (the base image version is mutable, and the Dockerfile schema doesn't allow you to verify that an image is what you'd expect with a checksum or something like that, containers can be mutated after startup, containers running as root is still a huge problem, etc), but this is definitely a step up from running apps in VMs. Now that I'm typing this out, I'm surprised that buildpacks or Chef's Habitat didn't take off; they solve a lot of these problems while providing similar reproducability and isolation guarantees.

Re: Ask HN: Who operates at scale without containers?

#188

Post by gen220 »

mst wrote: post #184
gen220 wrote: post #97

Earlier quoted context omitted.

Google has used debian as their base. Netflix uses a BSD flavor (I forget which) as their CDN cache. FB used CentOS, not sure what they use today since CentOS is EOL'd. Debian (and, formerly, CentOS) is a good standard: it occupies a sweet spot between ubuntu server and alpine, in the sense that it's batteries-included and very well-supported (apt/yum), but not particularly bloated. I use debian for all my personal s…

Google's debian-based distroless is fascinating: https://github.com/GoogleContainerTools/distroless

See also https://marc.merlins.org/linux/talks/ProdNG-LinuxCon2013/Pro... which has some interesting history and context from that time.

In paper form (more gory details): https://www.usenix.org/system/files/conference/lisa13/lisa13...

Re: Ask HN: Who operates at scale without containers?

#189

Post by smilliken »

Thaxll wrote: post #84
smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

The thing is, you spent probably a lot of time on things that were granted elsewhere and while the rest of the world is improving on those tech you keep your home grown solution that is harder and harder to maintain. Plus the knowledge that is not transferable. > Containers only give reproducible deployments, not builds, so they would be a step down. This is not true, if you use a docker image A with specific version…

Dan Luu wrote an essay on this topic recently: https://danluu.com/nothing-works/

He ponders why it is that big websites inevitably have kernel developers. Way out of their domain of expertise, right? If you adopt a technology, you're responsible for it.

When Kubernetes inevitably has an issue that is a blocker for us, I don't have confidence in my ability to fix it. When an internal python or shell program has an issue that is a blocker for us, I change it.

PostgreSQL is used by probably millions of people, but we've had to patch it on multiple occasions and run our fork in production. Nobody wants to do that, but sometimes you have to.

The point is, you can't just say "oh, we use kubernetes so we don't have to think about it". No. You added it to your stack, and now you're responsible for it. Pick technologies that you're able to support if they're abandoned, unresponsive to your feature requests and bug reports, or not interested in your use case. Pick open source, obviously.

This is another reason I like Nix. It's a one-line change to add a patch file to a build to fix an issue. So I can contribute a fix upstream to some project, and then I don't have to wait for their release process, I can use the patch immediately and let go of it whenever the upstream eventually integrates it. It lowers the cost of being responsible for third party software that we depend on.

Re: Ask HN: Who operates at scale without containers?

#190

Post by camtarn »

_16k4 wrote: post #77
camtarn wrote: post #41

Don't know if they still use it (I suspect so!) but at least as of 2015 Amazon was using a homebrewed deployment service called Apollo, which could spin up a VM from an internally developed Linux image then populate it with all the software and dependencies needed for a single service. It later inspired AWS CodeDeploy which does the same thing. I remember it being pretty irritating to use, though, since it wasn't par…

I've always thought of Apollo environments as containers before kernel features for containers existed. With enough environment variables and wrapper scripts taking the name of real binaries to populate stuff like LD_LIBRARY_PATH, Apollo makes a private environment that is only _slightly_ contaminated by the host.

Ooh, I'd forgotten about the wrapper scripts.

And yeah, the other thing that made it work, I guess, was having the machine image be very minimal, very tightly controlled, and very infrequently changed - so you didn't have to worry about things changing all the time due to the upstream distro.

Post reply on HN