Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who operates at scale without containers?

news.ycombinator.com

Post reply on HN
Print view View original

321–330 of 446 posts

Re: Ask HN: Who operates at scale without containers?

#321

Post by dylan604 »

hkt wrote: post #304
rattlesnakedave wrote: post #226

Earlier quoted context omitted.

This certainly seems more common. Every time I’ve had to refactor 20 year old cruft it’s been because of engineers with the launch-and-iterate mentality 20 years ago, that stopped caring about the “iterate” part once they gained enough job security.

Blaming the engineers rather than the businesses seems.. odd. It isn't like the people writing that code had control over what the company set as priorities. If anything, what you experienced seems to be a reflection of corporate short termism.

Quite right. The devs probably had a bunch of "TODO: This can be refactored" through out the code or at least mentally stored, yet the pointy haired bosses decided it was good enough and re-assigned the devs to other tasks.

Re: Ask HN: Who operates at scale without containers?

#322

Post by benatkin »

topspin wrote: post #313
benatkin wrote: post #114

Earlier quoted context omitted.

OCI images don't necessarily mean lxc/docker style containers. Many are run directly on MicroVMs. https://fly.io/blog/docker-without-docker/ Ditto Google Cloud Run.

The three products I have in mind were are all conventional 'lxc/docker' containers. Two are provided with docker-compose scripts. Two are available as either containerized or traditional products. One is container only.

Normally docker-compose should be portable. If they have OCI images they should be runnable on MicroVMs (no lxc container) with fly.io, Google Cloud Run, https://katacontainers.io/ and others.

The point I'm trying to make is that if you're aware of different ways to run an OCI image, you can run things in virtually the same way as the old way of having things packaged as a VM image (AMIs on EC2).

It does prevent you from setting up your own OS distribution and integrating the app directly with it, but so do AMIs.

Re: Ask HN: Who operates at scale without containers?

#323

Post by zeckalpha »

mijoharas wrote: post #220
camtarn wrote: post #41

Don't know if they still use it (I suspect so!) but at least as of 2015 Amazon was using a homebrewed deployment service called Apollo, which could spin up a VM from an internally developed Linux image then populate it with all the software and dependencies needed for a single service. It later inspired AWS CodeDeploy which does the same thing. I remember it being pretty irritating to use, though, since it wasn't par…

Not too much of an update, but they were still using it in 2017.

And as of 2021

Re: Ask HN: Who operates at scale without containers?

#324

Post by nijave »

Not sure if they still do, but Github used to run a decent amount of stuff on baremetal https://github.blog/2015-12-01-githubs-metal-cloud/

Afaik Dropbox Magic Pocket is bare metal (they have an internal baremetal provisioning system) https://dropbox.tech/infrastructure/inside-the-magic-pocket

Fastly's edge network I'm pretty sure is baremetal.

JP Morgan Chase has a significant amount of non-containerized workloads including one of the largest IBM mainframe setups in the world (can't find a source but internally they claimed they were IBM's biggest mainframe customer). They run high availability apps using some parallel sysplex setup where they can failover an app at the hardware level (https://en.wikipedia.org/wiki/IBM_Parallel_Sysplex#Geographi...). Their largest apps were usually either DB2 based on the mainframe or some J2E setup on distributed/x86 with Oracle databases (Websphere, Tomcat, WebLogic). They also still had a pretty big HP Nonstop setup afaik (they're so big they pretty had at least one of every Big Enterprise Thing)

For Java apps, you basically just build a war/jar/ear and publish it to a Java repo where it gets deployed to servers. The handful of things I know about all had shell or Perl scripts and operations teams to manage deployment. It's effectively the same thing as a container but all Java. Some of those stacks like WebSphere run as a compute cluster that does similar things to container orchestrators like deploy management, config management, scheduling

>without relying on the specific practice of shipping software with heaps of dependencies All that stuff still had heaps of dependencies be it internal or external.

Re: Ask HN: Who operates at scale without containers?

#325

Post by pdx6 »

When I worked at Okta, they used EC2 without containers for their primary monolith workload. Scaling would be manual via ElB. The latest hack shows they are using NLB now, but I suspect it is still EC2 only otherwise.

To distribute database load, a cell or “block” of instances would be filled until the largest database was at 80% write capacity or so, then a new account and group was spun up for newer customers.

Re: Ask HN: Who operates at scale without containers?

#326

Post by nijave »

zthrowaway wrote: post #138
brimble wrote: post #44

Earlier quoted context omitted.

> That's the easy part! The hard part is managing databases. Ding ding ding ding ding. "But what about disk?" (so, relatedly, databases) is the hard part. Balancing performance (network disks suuuuuck) and flexibility ("just copy the disk image to another machine" is fine when it's a few GB—less useful when it's lots of GB and you might need to migrate it to another city and also you'd rather not have much downtime)…

People still think stateful things are impossible on k8s but Stateful sets and persistent volumes solves a lot of this. You should be relying on out of the box DB replication to make sure data is available in multiple areas. This is no different on other platforms.

Putting a 50TB+ database that enables eye watering revenue on k8s is a hard sell for a lot of businesses--especially when they have non-containerized solutions that work.

Simple topologies and NoSQL databases (or databases they can handle replication/partitioning/node failures automatically) are pretty easy to stick in StatefulSets.

There's crazy things like this https://blogs.oracle.com/mysql/post/circular-replication-in-... that would be fairly difficult to run in a StatefulSet.

In addition, the patch and lifecycle cadence of k8s is pretty quick so rebooting SQL databases with 100ks TPS more than 1-2 times a year tends to be problematic

Re: Ask HN: Who operates at scale without containers?

#327

Post by bruth »

gabrielgrant wrote: post #318
senorsmile wrote: post #297

Earlier quoted context omitted.

Kelsey Hightower just had a good interview where he disagrees with you: https://changelog.com/shipit/44 Edit: somewhat* disagrees with you. It's a good listen.

tl;dr?

If you have a low-volume, non-performant, non-critical database, k8s is fine. If you need it to perform and/or need built-in ops (managed backups or replication), use a managed service.

k8s _can_ do stateful, but if a managed service exists for this workload, use it. It is not about _can_ I run it on k8s, it is a _should_ question. Is it worth the cumulative effort required to achieve the same degree of quality.

Re: Ask HN: Who operates at scale without containers?

#328

Post by illfit »

cpach wrote: post #241
tptacek wrote: post #212

Ironically, here at Fly.io, we run containers (in single-use VMs) for our customers, but none of our own infrastructure is containerized --- though some of our customer-facing stuff, like the API server, is. We have a big fleet of machines, mostly in two roles (smaller traffic-routing "edge" hosts that don't run customer VMs, and chonky "worker" hosts that do). All these hosts run `fly-proxy`, a Rust CDN-style proxy…

Intriguing! This makes me curious: How does one learn to design and build systems like this…? Also: How do you folks at Fly decide what parts to use “as is” and what parts to build from scratch? Do you have any specific process for making those choices?

Google uses "Non-Abstract Large System Design (NALSD)" https://sre.google/workbook/non-abstract-design/ for this style of design.

The emphasis on a concrete design with concrete numbers can help identify the main scaling and reliability limitations, and put a cost on these. "Design X costs $A/year for Y scheduled fly.io tasks".

To build such a design relies on knowing fundamentals such as the performance characteristics of CPU/disk/network. "How many disks would it take to serve 50k QPS at 20ms, each time performing 1k of random disk I/O."

Knowing this helps identify where in your stack you want flexibility, and why you'd want it.

"We log 100MB/s spread across 200 machines, which can be done with vanilla mature Elastic search"

"We need to be able to perform container routing at 0.5ms overhead, and updates need to be atomic. eBPF can do this but existing solutions are immature. Since this is also our core competency, let's do this ourselves."

Re: Ask HN: Who operates at scale without containers?

#329

Post by ironmagma »

tharne wrote: post #89
smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

> What works for us is to do the simplest thing that works, then iterate. The older I get, the more often I'm reminded that this un-sexy approach is really the best way to go. When I was younger, I always thought the old guys pushing boring solutions just didn't want to learn new things. Now I'm starting to realize that after several decades of experience, they simply got burned enough times to learn a thing or two h…

The thing is, “simple” is not trivial to discern. What looks simple to a beginner may look quite hard to maintain to an experienced dev. “GOTO” being the obvious example.

Re: Ask HN: Who operates at scale without containers?

#330

Post by ironmagma »

hkt wrote: post #304
rattlesnakedave wrote: post #226

Earlier quoted context omitted.

This certainly seems more common. Every time I’ve had to refactor 20 year old cruft it’s been because of engineers with the launch-and-iterate mentality 20 years ago, that stopped caring about the “iterate” part once they gained enough job security.

Blaming the engineers rather than the businesses seems.. odd. It isn't like the people writing that code had control over what the company set as priorities. If anything, what you experienced seems to be a reflection of corporate short termism.

That dichotomy falls apart because engineers are part of the business. If the business wants to do something but the engineers won’t do it, it won’t happen.
Post reply on HN