Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who operates at scale without containers?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 446 posts

Re: Ask HN: Who operates at scale without containers?

#81

Post by smilliken »

claytonjy wrote: post #47
smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

is Nix a hurdle when onboarding engineers? do only a few people need to know the language? I've wanted to use Nix and home-manager for personal stuff but the learning curve seems big.

In my case, adopting Nix was a response to having a poor onboarding process for new engineers. It was always fully automated, but it wasn't reliable before nix. So somebody would join the team, and it was embarrassing because the first day would be them troubleshooting a complex build process to get it to work on their machine. Not a great first impression.

So I adopted Nix "in anger" and now new machines always build successfully on the first try.

For me it was easy to set up. 80% done on the first day. It helped that I understood the conceptual model and benefits first. There's a lot of directions you can take it, I'd recommend getting something really simple working then iterating. Don't try to build the Sistine Chapel with all the different things integrated on day one.

It's hard to overstate how well that decision has worked out for me. Computers used to stress me out a lot more because of dependencies and broken builds. Now I have a much healthier relationship with computers.

Re: Ask HN: Who operates at scale without containers?

#82

Post by yabones »

kuon wrote: post #55

We use ansible on bare metal (no VM) to manage about 200 servers in our basement. We use PXE booting to manage the images. We use a customized arch linux image and we have a few scripts to select what feature we'd like. It's "old school" but it's been working fine for nearly 20 years (we used plain scripts before ansible, so we always used the "agentless" approach). Our networking stack uses OpenBSD.

That sounds really interesting. If you have a write-up about how it's built, the decisions that went into it, and the problems you've had to solve, I would absolutely read it!

Re: Ask HN: Who operates at scale without containers?

#83

Post by hungryforcodes »

wanderr wrote: post #28

Grooveshark didn't use any of that. We were very careful about avoiding dependencies where possible and keeping our backend code clean and performant. We supported about 45M MAU at our biggest, with only a handful of physical servers. I'm not aware of any blog posts we made detailing any of this, though. And if you're not familiar with the saga, Grooveshark went under for legal, not technical reasons. The backend API…

Groove shark was the best.

Re: Ask HN: Who operates at scale without containers?

#84

Post by Thaxll »

smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

The thing is, you spent probably a lot of time on things that were granted elsewhere and while the rest of the world is improving on those tech you keep your home grown solution that is harder and harder to maintain. Plus the knowledge that is not transferable.

> Containers only give reproducible deployments, not builds, so they would be a step down.

This is not true, if you use a docker image A with specific version to build then the result is reproducible.

That's what most people should be doing, you pin versions in your build image that's it.

> Kubernetes has mostly focused on stateless applications so far. That's the easy part

Kubernetes can run database, and stateless application is not a solved problem, I mean how does your services get restarted on your system, is it something that you had to re-create as well?

Re: Ask HN: Who operates at scale without containers?

#85

Post by Melatonic »

wsostt wrote: post #36
JonChesterfield wrote: post #21

Earlier quoted context omitted.

Do you see shipping in containers as a sign that they do know what they're doing, or as a sign that they don't?

Neither by default. I look to be convinced that they've implemented containers because it made sense for their technical architecture or strategy. I have a vendor who pitched their new "cloud-native" re-platforming project and it really spooked me. It's data management-type tool that was migrating from a traditional on-prem client/server architecture to an AWS-hosted Angular interface with a MongoDB backend. I got th…

To be fair they may have just realized that MongoDB is sort of meh and decided to change to something better

Re: Ask HN: Who operates at scale without containers?

#86

Post by Melatonic »

bogomipz wrote: post #68
jcadam wrote: post #61

Is using CUDA inside a container still a massive PITA?

Can you elaborate on what makes containerized CUDA so difficult?

I would guess the fact that its a highly proprietary design that is designed to specifically scale to massive amounts of parallel threads on specific physical hardware devices

Re: Ask HN: Who operates at scale without containers?

#88

Post by frellus »

Great talk here from HCA Healthcare about their architecture and tech stack using Elixir OTP, Riak:

https://www.youtube.com/watch?v=cVQUPvmmaxQ

TL;DR - Elixir/OTP for fault tolerance and process supervision, Riak as a KV store (no other database) and an interesting process-actor model for patients I found delightful. Bonus: hot code patching, zero downtime

Re: Ask HN: Who operates at scale without containers?

#89

Post by tharne »

smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

> What works for us is to do the simplest thing that works, then iterate.

The older I get, the more often I'm reminded that this un-sexy approach is really the best way to go.

When I was younger, I always thought the old guys pushing boring solutions just didn't want to learn new things. Now I'm starting to realize that after several decades of experience, they simply got burned enough times to learn a thing or two had developed much better BS-detectors than 20-something me.

Re: Ask HN: Who operates at scale without containers?

#90

Post by onebot »

We use freebsd jails and a lightweight in house orchestration tool written in Rust. We are running hundreds of Ryzen machines with 64 cores. Our costs compared to running equivalent on Amazon is so much less. We estimate our costs are about 6x lower than AWS and we have far better performance in terms of networking, CPU, and disk write speed.

Jails has been a pleasure to work with! We even dynamically scale up and down resources as needed.

We use bare metal machines on Interserver. But there a quite a few good data centers worth considering.

Post reply on HN