Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who operates at scale without containers?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 446 posts

Re: Ask HN: Who operates at scale without containers?

#41

Post by camtarn »

Don't know if they still use it (I suspect so!) but at least as of 2015 Amazon was using a homebrewed deployment service called Apollo, which could spin up a VM from an internally developed Linux image then populate it with all the software and dependencies needed for a single service. It later inspired AWS CodeDeploy which does the same thing.

I remember it being pretty irritating to use, though, since it wasn't particularly easy to get Apollo to deploy to a desktop machine in the same way it would in production, and of course you couldn't isolate yourself from the desktop's installed dependencies in the same way. I'm using Docker nowadays and it definitely feels a lot smoother.

This is a nice writeup: https://www.allthingsdistributed.com/2014/11/apollo-amazon-d...

Re: Ask HN: Who operates at scale without containers?

#42

Post by turkeywelder »

wanderr wrote: post #28

Grooveshark didn't use any of that. We were very careful about avoiding dependencies where possible and keeping our backend code clean and performant. We supported about 45M MAU at our biggest, with only a handful of physical servers. I'm not aware of any blog posts we made detailing any of this, though. And if you're not familiar with the saga, Grooveshark went under for legal, not technical reasons. The backend API…

I miss Grooveshark so much - Licensing issues aside it was one of the best UIs for music ever. I'd love to hear more stories about the backend

Re: Ask HN: Who operates at scale without containers?

#43

Post by Aachen »

wanderr wrote: post #28

Grooveshark didn't use any of that. We were very careful about avoiding dependencies where possible and keeping our backend code clean and performant. We supported about 45M MAU at our biggest, with only a handful of physical servers. I'm not aware of any blog posts we made detailing any of this, though. And if you're not familiar with the saga, Grooveshark went under for legal, not technical reasons. The backend API…

Man I miss Grooveshark still today. Spotify is okay but still a step down. Needing billion-dollar licensing schemes to even get started makes this such a hard market to actually get into and provide a competitively superior experience.

Re: Ask HN: Who operates at scale without containers?

#44

Post by brimble »

smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

> That's the easy part! The hard part is managing databases.

Ding ding ding ding ding.

"But what about disk?" (so, relatedly, databases) is the hard part. Balancing performance (network disks suuuuuck) and flexibility ("just copy the disk image to another machine" is fine when it's a few GB—less useful when it's lots of GB and you might need to migrate it to another city and also you'd rather not have much downtime) is what's tricky. Spinning up new workers, reliable- and repeatable-enough deployment with rollbacks and such, that's easy, until it touches the disk (or the DB schema, that's its own whole messy dance to get right)

Re: Ask HN: Who operates at scale without containers?

#45

Post by alex_duf »

Hey former Guardian employee here.

The Guardian has hundreds of servers running, pretty much all EC2 instances. EC2 images are baked and derived from official images, similarly to the way you bake a docker image.

We built tools before docker became the de facto standard, so we could easily keep the EC2 images up to date. We integrated pretty well with AWS so that the basic constructs of autoscaling and load balancer were well understood by everyone.

The stack is mostly JVM based so the benefits of running docker locally weren't really significant. We've evaluated moving to a docker solution a few times and always reached the conclusion that the cost of doing so wouldn't be worth the benefits.

Now for a company that starts today I don't think I'd recommend that, it just so happen that The Guardian invested early on the right tooling so that's pretty much an exception.

Re: Ask HN: Who operates at scale without containers?

#46

Post by Negitivefrags »

We don’t use containers with a pretty large deployment. (1k or so bare metal servers)

If you statically link all your binaries then your deployment system can be rsync a directory.

The only dependency is that the Linux kernel is new enough. Other than that the packages we deploy will run on literally any setup.

Re: Ask HN: Who operates at scale without containers?

#47

Post by claytonjy »

smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

is Nix a hurdle when onboarding engineers? do only a few people need to know the language? I've wanted to use Nix and home-manager for personal stuff but the learning curve seems big.

Re: Ask HN: Who operates at scale without containers?

#48

Post by brimble »

wanderr wrote: post #28

Grooveshark didn't use any of that. We were very careful about avoiding dependencies where possible and keeping our backend code clean and performant. We supported about 45M MAU at our biggest, with only a handful of physical servers. I'm not aware of any blog posts we made detailing any of this, though. And if you're not familiar with the saga, Grooveshark went under for legal, not technical reasons. The backend API…

> That said, I'm using Docker for my current side project. Even if it never runs at scale, I just don't want to have to muck around with system administration, not to mention how nice it is to have dev and prod be identical.

This is why I use docker, at work and for my own stuff. No longer having to give a shit whether the hosting server is LTS or latest-release is wonderful. I barely even have to care which distro it is. Much faster and easier than doing something similar with scripted-configuration VMs, plus the hit to performance is much lower.

Re: Ask HN: Who operates at scale without containers?

#49

Post by claytonjy »

Negitivefrags wrote: post #46

We don’t use containers with a pretty large deployment. (1k or so bare metal servers) If you statically link all your binaries then your deployment system can be rsync a directory. The only dependency is that the Linux kernel is new enough. Other than that the packages we deploy will run on literally any setup.

what languages do you use in this way? this sounds a lot easier and more natural for C than python

Re: Ask HN: Who operates at scale without containers?

#50

Post by silviogutierrez »

claytonjy wrote: post #47
smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

is Nix a hurdle when onboarding engineers? do only a few people need to know the language? I've wanted to use Nix and home-manager for personal stuff but the learning curve seems big.

It's a hurdle to be able to write Nix stuff, but to consume it is pretty easy.

Shameless plug here: https://www.reactivated.io/documentation/why-nix/

Post reply on HN