Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who operates at scale without containers?

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 446 posts

Re: Ask HN: Who operates at scale without containers?

#91

Post by Osiris »

thinkingkong wrote: post #67
barbazoo wrote: post #53

Earlier quoted context omitted.

> Containers only give reproducible deployments, not builds Could someone elaborate on this please? Doesn't it depend entirely on your stack how reproducible your build is? Say I have a Python app with its OS level packages installed into a (base) image and its Python dependencies specified in a Pipfile, doesn't that make it pretty reproducible? Is the weak spot here any OS dependencies being installed as part of the…

The dependencies can change out from underneath you transparently unless you pin everything all the way down the stack. Upstream docker images for example are an easy to understand vector of change. The deb packages can all change minor versions between runs, the npm packages (for example) can change their contents without making a version bump. Theres tons of implicit trust with all these tools, build wise.

npm packages can change without a version change?

Can you explain this?

npm doesn't allow you to delete any published versions (you can only deprecate them). You aren't allowed to publish a version that's already been published.

Even when there have been malicious packages published the solution has been to publish newer versions of the package with the old code. There's no way to delete the malicious package (maybe npm internally can do it?).

Re: Ask HN: Who operates at scale without containers?

#93

Post by smilliken »

higeorge13 wrote: post #38
smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

It sounds interesting. You mentioned big data; what is your tech stack other than postgres?

Python, PostgreSQL, Nix, Linux. Javascript and Typescript for frontend. Everything else is effectively libraries. After 300k lines of Python, with the occasional C extension, we haven't found any limit.

Re: Ask HN: Who operates at scale without containers?

#94

Post by whateveracct »

_16k4 wrote: post #77
camtarn wrote: post #41

Don't know if they still use it (I suspect so!) but at least as of 2015 Amazon was using a homebrewed deployment service called Apollo, which could spin up a VM from an internally developed Linux image then populate it with all the software and dependencies needed for a single service. It later inspired AWS CodeDeploy which does the same thing. I remember it being pretty irritating to use, though, since it wasn't par…

I've always thought of Apollo environments as containers before kernel features for containers existed. With enough environment variables and wrapper scripts taking the name of real binaries to populate stuff like LD_LIBRARY_PATH, Apollo makes a private environment that is only _slightly_ contaminated by the host.

Apollo reminded me more of Nix than containers. The wrapper scripts are super Nix-y :)

Re: Ask HN: Who operates at scale without containers?

#95

Post by cj »

Thaxll wrote: post #84
smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

The thing is, you spent probably a lot of time on things that were granted elsewhere and while the rest of the world is improving on those tech you keep your home grown solution that is harder and harder to maintain. Plus the knowledge that is not transferable. > Containers only give reproducible deployments, not builds, so they would be a step down. This is not true, if you use a docker image A with specific version…

> the rest of the world is improving on those tech

Does that matter if the current stack works just fine?

Imagine someone who built a calculator app with jQuery javascript 10 years ago, and all it does is add and subtract numbers. You could spend time porting it to Ember, and then migrating to Angular, and then porting it to React, and then porting it to React with SSR and hooks.

If the calculator app worked with 15 year old code, you can either leave it be, or try to keep up with latest tech trends and continuously refactor and port the code.

I think there are a lot of "calculator app" type components of many systems that, at a certain point, can be considered "done, complete" without ever needing a rebuild or rewrite. Even if they were built on technologies that might now be considered antiquated.

Re: Ask HN: Who operates at scale without containers?

#96

Post by Melatonic »

nailer wrote: post #72

Probably most people? Newer apps often use MicroVMs, older apps often use Xen VMs. Containers aren't the only containment mechanism and some implementations are known as complex time sinkholes.

Surprising amount of places are still just running VM's - especially hybrid cloud or on "on prem" type stuff. If what you have already works and your staff is trained for that type of environment then I personally do not think it makes sense to switch unless there is a big advantage.

Re: Ask HN: Who operates at scale without containers?

#97

Post by gen220 »

tailspin2019 wrote: post #17
Nextgrid wrote: post #2

I've been at a company where they weren't (yet) using containers nor K8S. The build process would just create VM images with the required binaries in there and then deploy that to an autoscaling group. It worked well and if you only ever intend to run a single service per machine then is the right solution.

I’m in the process of moving to exactly this approach. I’ve been trying to pick the right Linux distro to base my images on. Ubuntu Server is the low effort route but a bit big to redeploy constantly. I’ve also been looking at the possibility of using Alpine Linux which feels like a better fit but a bit more tweaking needed for compatibility across cloud providers. Unikernels are also interesting but I think that mig…

Google has used debian as their base. Netflix uses a BSD flavor (I forget which) as their CDN cache. FB used CentOS, not sure what they use today since CentOS is EOL'd.

Debian (and, formerly, CentOS) is a good standard: it occupies a sweet spot between ubuntu server and alpine, in the sense that it's batteries-included and very well-supported (apt/yum), but not particularly bloated.

I use debian for all my personal stuff. The BSD flavors (https://en.wikipedia.org/wiki/Comparison_of_BSD_operating_sy...) are tempting, though. Perhaps one weekend!

For other distros, if you're working on non-x86, or have uncommon dependencies, you should be prepared to pull in and build your dependencies from scratch, which is not much fun to maintain.

Re: Ask HN: Who operates at scale without containers?

#98

Post by david38 »

smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

Ehhh, I run a high performance database system in Kubernetes and it works great. It’s distributed, uses EBS volumes. That’s about as opposite of stateless as it gets.

Re: Ask HN: Who operates at scale without containers?

#99

Post by bluefirebrand »

16bytes wrote: post #20
efficax wrote: post #3

Back in 2016 at least, Stack overflow was container free https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... No idea how much has changed since then

That's the first thing that came to mind as well, and Stack Overflow definitely operates at scale, but OP asked for people doing distributed systems. SO is famously monolithic, famously operating from just a handful of machines.

If anything, Stack Overflow should be taken as an example that large scale operations can be run a lot more simply than many of us are making them.

Re: Ask HN: Who operates at scale without containers?

#100

Post by tombert »

I don't think this will get me in trouble; the iTunes side of Apple, for a long time (they might have changed semi-recently), was using a home-spun, container-free distribution platform called "Carnival". It was running JVMs straight on Linux boxes, and as far as I know, was not using any kind of containers.

There was talk of moving to k8s before I left, so it's possible this is no longer true.

Post reply on HN