Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who operates at scale without containers?

news.ycombinator.com

Post reply on HN
Print view View original

261–270 of 446 posts

Re: Ask HN: Who operates at scale without containers?

#261

Post by hawk_ »

zthrowaway wrote: post #138
brimble wrote: post #44

Earlier quoted context omitted.

> That's the easy part! The hard part is managing databases. Ding ding ding ding ding. "But what about disk?" (so, relatedly, databases) is the hard part. Balancing performance (network disks suuuuuck) and flexibility ("just copy the disk image to another machine" is fine when it's a few GB—less useful when it's lots of GB and you might need to migrate it to another city and also you'd rather not have much downtime)…

People still think stateful things are impossible on k8s but Stateful sets and persistent volumes solves a lot of this. You should be relying on out of the box DB replication to make sure data is available in multiple areas. This is no different on other platforms.

Can you elaborate on "multiple areas"? Do you mean each node inside a db "cluster" should run in a different area? And how does one achieve that?

Re: Ask HN: Who operates at scale without containers?

#262

Post by treesknees »

motoboi wrote: post #80
freeqaz wrote: post #57

Earlier quoted context omitted.

That's a very cool deployment method to just use rsync like that! Simple and very composable. And now I feel self-conscious about my pile of AWS and Docker!

Containers are just tar.gz files, you know? The whole layers thing it’s just an optimization. You can actually very simply run those tar.gz files without docker involved, just cgroups. But then you’ll have to write some daemon scripts to start, stop, restart, etc Follow this path and soon you’ll have a (worst) custom docker. Try to create a network out of those containers and soon a (worst) SDN network appears. Try t…

This is essentially what fly.io does, they unpack the container image into a micro-VM and execute it directly. Calling it worse is subjective, you certainly need to account for things you would never worry about by using traditional Docker. But for some companies it ends up working much better than Docker.

https://fly.io/blog/docker-without-docker/

Re: Ask HN: Who operates at scale without containers?

#264

Post by pineconewarrior »

wanderr wrote: post #28

Grooveshark didn't use any of that. We were very careful about avoiding dependencies where possible and keeping our backend code clean and performant. We supported about 45M MAU at our biggest, with only a handful of physical servers. I'm not aware of any blog posts we made detailing any of this, though. And if you're not familiar with the saga, Grooveshark went under for legal, not technical reasons. The backend API…

I loved Grooveshark! thanks for your work

Re: Ask HN: Who operates at scale without containers?

#265

Post by armcat »

Not sure if this counts, but for more than a decade I was at a telecom vendor, working with radio base stations (3G, 4G and 5G). That (to me), is probably one of the most distributed systems on the planet - we worked across several million nodes around the globe. I've been out of the loop for a bit, but I know they now have vRAN, Cloud RAN, etc (basically certain soft-real time functions pulled out of base stations and deployed as VMs or containers). But back then, there was no virtualization being used.

The tech stack was as follows: hardware was either PowerPC or ARM based System-on-Chip variants; we initially used our own in-house real-time OS, but later switched to a just-enough Linux distro; management functions were implemented either in IBM's "real-time" JVM (J9), or in Erlang; radio control plane (basically messages used to authenticate you, setup the connection and establish radio bearers, i.e. "tunnels" for payload) was written in C++. Hard real-time functions (actual scheduling of radio channel elements, digital signal processing, etc) were written in C and assembly.

Really cool thing - we even deployed a xgboost ML model on these (used for fast frequency reselection - reduced your time in low coverage) - the model was written in C++ (no Python runtime was allowed), and it was completely self-supervised, closed-look (it would update/finetune its parameters during off-peak periods, typically at night).

Back then, we were always self-critical of ourselves, but looking back at it, it was an incredibly performant and robust system. We accounted for every CPU cycle and byte - at one point I was able to do a walkthrough (from-memory) of every single memory allocation during a particular procedure (e.g. a call setup). We could upgrade thousands of these nodes in one maintenance window, with a few secs of downtime. The build system we always complained about, but looking back at it, you could compile and package everything in a matter of minutes.

Anyway, I think it was a good example of what you can accomplish with good engineering.

Re: Ask HN: Who operates at scale without containers?

#266

Post by dolni »

zokier wrote: post #254
dolni wrote: post #6

Earlier quoted context omitted.

At my workplace we use Docker to run services, but there is no container orchestration like Kubernetes. An AMI bakes in some provisioning logic and the container image. Autoscaling does the rest. Even without orchestration, I argue containers are useful. They abstract the operating system from the application and allow you to manage each independently. Much more easily than you'd be able to otherwise, anyway. Plus yo…

> An AMI bakes in some provisioning logic and the container image That does sound bit weird setup to me; with something like Packer building an AMI is almost as easy as building a Docker image with Dockerfiles so the benefits of using Docker seem quite slim?

Nah, Docker still has lots of benefits. You can run a Docker image on a developer system, using docker-compose, and it could even be the exact same one running in prod.

That takes a TON of variables out of the equation.

Re: Ask HN: Who operates at scale without containers?

#268

Post by Thaxll »

cj wrote: post #95
Thaxll wrote: post #84

Earlier quoted context omitted.

The thing is, you spent probably a lot of time on things that were granted elsewhere and while the rest of the world is improving on those tech you keep your home grown solution that is harder and harder to maintain. Plus the knowledge that is not transferable. > Containers only give reproducible deployments, not builds, so they would be a step down. This is not true, if you use a docker image A with specific version…

> the rest of the world is improving on those tech Does that matter if the current stack works just fine? Imagine someone who built a calculator app with jQuery javascript 10 years ago, and all it does is add and subtract numbers. You could spend time porting it to Ember, and then migrating to Angular, and then porting it to React, and then porting it to React with SSR and hooks. If the calculator app worked with 15…

It works fine until the core people that know that stack leave your compagny and no one wants to touch it. Then the new people comes in a pich a solution that is widely used.

Re: Ask HN: Who operates at scale without containers?

#269

Post by hinkley »

treis wrote: post #247
hinkley wrote: post #125

Earlier quoted context omitted.

This is something that even the Docker core team is not entirely clear on (either they understand it and can't explain it, or they don't understand it). The RUN command is Turing complete, therefore cannot be idempotent. Nearly every usable docker image and base image include the RUN command, often explicitly to do things that are not repeatable, like "fetch the latest packages from the repository". This is all befor…

>Getting repeatable docker images But there's no real need for repeatable build docker images. You copy the image and run it where ever you need to. The entire point of Docker is to not have to repeat the build.

If you could explain that to the Docker team you would be performing a great humanitarian service.

They for many years close issues and reject PRs based on the assertion that Dockerfile output needs to be repeatable (specifically, that the requested feature or PR makes it not repeatable). It’s not, it can’t be without a time machine, and if it was I believe you’d find that ace icing traction would have been more difficult, even impossible.

Re: Ask HN: Who operates at scale without containers?

#270

Post by nhoughto »

cyberge99 wrote: post #11

I believe tools like nomad and consul shine here. Using nomad as a job scheduler and deployer allows you to use various modules for jobs: java, shell, ec2, apps (and containers). I use it in my homelab and it’s great. That said, I don’t use it professionally. I think Cloudflare is running this stack alongside firecracker for some amazing edge stuff.

Recent convo with Cloudflare they said they are mostly container-less, debian packages installed and running on the the host, which is impressive at their scale and complexity. They did seem to think that this approach was hitting its limits and what the future might look like when I spoke to ppl there recently (end 2021), assumption being the future could be containers.
Post reply on HN