Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who operates at scale without containers?

news.ycombinator.com

Post reply on HN
Print view View original

131–140 of 446 posts

Re: Ask HN: Who operates at scale without containers?

#131

Post by jandrewrogers »

Thaxll wrote: post #84
smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

The thing is, you spent probably a lot of time on things that were granted elsewhere and while the rest of the world is improving on those tech you keep your home grown solution that is harder and harder to maintain. Plus the knowledge that is not transferable. > Containers only give reproducible deployments, not builds, so they would be a step down. This is not true, if you use a docker image A with specific version…

Sure, Kubernetes can run a database, but not efficiently. Companies with intensive data infrastructure frequently operate at scale without containers, either VMs or bare metal. The larger the data volume, the less likely they are to use containers because efficiency is more important. It is also simpler to manage this kind of thing outside containers, frankly, since you are running a single process per server.

People have been building this infrastructure since long before containers, thousands of servers in single operational clusters. Highly automated data infrastructure without containers is pretty simple and straightforward if this is your business.

Re: Ask HN: Who operates at scale without containers?

#132

Post by ryanjkirk »

danielrhodes wrote: post #66
maxk42 wrote: post #26

Back in 2010 I built and operated MySpace' analytics system on 14 EC2 instances. Handled 30 billion writes per day. Later I was involved in ESPN's streaming service which handled several million concurrent connections with VMs but no containers. More recently I ran an Alexa top 2k website (45 million visitors per month) off of a single container-free EC2 insurance. Then I spent two years working for a streaming compa…

Docker is far heavier - the overhead is the flexibility and process isolation you get. I imagine that's really useful for certain types of workloads (e.g. an ETL pipeline), but is crazy inefficient for something single purpose like a web app.

Docker is heavier (and more dangerous) because of dockerd, the management and api daemon that runs as root. Actual process isolation is handled by cgroup controls which are already built into the kernel and have been for years. You can apply them to any process, not just docker ones.

However, Docker is essentially dead; the future is CRI-O or something similar which has no daemon and runs as an unprivileged user. And you still get the flexibility and process isolation, but with more security.

Re: Ask HN: Who operates at scale without containers?

#133

Post by anonymousDan »

javajosh wrote: post #127
disintegore wrote: post #62

Earlier quoted context omitted.

I asked the question without context because I didn't want to fire the thread off in the wrong direction but I suppose in a comment chain it's fine. For what it's worth we do use containers and K8s heavily at my current job. I know that there are quite a few people opposed to the state of containers and the technologies revolving around them. I don't think the arguments they present are bad. [Attacking the premise](…

What you're asking for is an essay on comparative devops architectures, with a focus on k8s alternatives. I think what you'll find is a lot of ad hoc persistent systems that tend to drift over time in unpredictable ways, and take on the feel of a public lobby if you're being generous, a public restroom if you're not. So what you're asking is really a sample of these ad hoc approaches. What I think you'll find are a f…

Availability?

Re: Ask HN: Who operates at scale without containers?

#134

Post by smilliken »

barbazoo wrote: post #53
smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

> Containers only give reproducible deployments, not builds Could someone elaborate on this please? Doesn't it depend entirely on your stack how reproducible your build is? Say I have a Python app with its OS level packages installed into a (base) image and its Python dependencies specified in a Pipfile, doesn't that make it pretty reproducible? Is the weak spot here any OS dependencies being installed as part of the…

You already got a few good answers, but I'll echo them: you can do reproducible builds in containers, and nothing's stopping you from using nix inside containers. But you're at the mercy of all the different package managers that people will end up using (apt, npm, pip, make, curl, etc). So your system is only as good as the worst one.

I inherited a dozen or so docker containers a while back that I tried to maintain. Literally none of them would build— they all required going down the rabbit hole of troubleshooting some build error of some transitive dependency. So most of them never got updated and the problem got worse over time until they were abandoned.

The reason Nix is different is because it was a radically ambitious idea to penetrate deep into how all software is built, and fix issues however many layers down the stack it needed to. They set out to boil the ocean, and somehow succeeded. Containers give up, and paper over the problems by adding another layer of complexity on top. Nix is also complex, but it solves a much larger problem, and it goes much deeper to address root causes of issues.

Re: Ask HN: Who operates at scale without containers?

#136

Post by goodpoint »

usaphp wrote: post #126
tharne wrote: post #89

Earlier quoted context omitted.

> What works for us is to do the simplest thing that works, then iterate. The older I get, the more often I'm reminded that this un-sexy approach is really the best way to go. When I was younger, I always thought the old guys pushing boring solutions just didn't want to learn new things. Now I'm starting to realize that after several decades of experience, they simply got burned enough times to learn a thing or two h…

Or it could be that you just became old and don't want to learn new things anymore ))

Such "new things" are just a big bunch of unnecessary complexity.

Re: Ask HN: Who operates at scale without containers?

#137

Post by tetha »

david38 wrote: post #98
smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

Ehhh, I run a high performance database system in Kubernetes and it works great. It’s distributed, uses EBS volumes. That’s about as opposite of stateless as it gets.

Do note that you have offloaded a good chunk of state management into EBS volumes via CSI. Attaching the CSI volumes is one thing, running the disks hosting the volumes is another thing.

Re: Ask HN: Who operates at scale without containers?

#138

Post by zthrowaway »

brimble wrote: post #44
smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

> That's the easy part! The hard part is managing databases. Ding ding ding ding ding. "But what about disk?" (so, relatedly, databases) is the hard part. Balancing performance (network disks suuuuuck) and flexibility ("just copy the disk image to another machine" is fine when it's a few GB—less useful when it's lots of GB and you might need to migrate it to another city and also you'd rather not have much downtime)…

People still think stateful things are impossible on k8s but Stateful sets and persistent volumes solves a lot of this. You should be relying on out of the box DB replication to make sure data is available in multiple areas. This is no different on other platforms.

Re: Ask HN: Who operates at scale without containers?

#139

Post by anonymousDan »

jandrewrogers wrote: post #131
Thaxll wrote: post #84

Earlier quoted context omitted.

The thing is, you spent probably a lot of time on things that were granted elsewhere and while the rest of the world is improving on those tech you keep your home grown solution that is harder and harder to maintain. Plus the knowledge that is not transferable. > Containers only give reproducible deployments, not builds, so they would be a step down. This is not true, if you use a docker image A with specific version…

Sure, Kubernetes can run a database, but not efficiently. Companies with intensive data infrastructure frequently operate at scale without containers, either VMs or bare metal. The larger the data volume, the less likely they are to use containers because efficiency is more important. It is also simpler to manage this kind of thing outside containers, frankly, since you are running a single process per server. People…

Can you point to any resources on the perf implications of docker for databases?

Re: Ask HN: Who operates at scale without containers?

#140

Post by nancarrow »

bogomipz wrote: post #68
jcadam wrote: post #61

Is using CUDA inside a container still a massive PITA?

Can you elaborate on what makes containerized CUDA so difficult?

the CUDA library versions are very coupled to the hosts' CUDA driver versions and the container system itself (Docker) needs special code to link the two
Post reply on HN