Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who operates at scale without containers?

news.ycombinator.com

Post reply on HN
Print view View original

241–250 of 446 posts

Re: Ask HN: Who operates at scale without containers?

#241

Post by cpach »

tptacek wrote: post #212

Ironically, here at Fly.io, we run containers (in single-use VMs) for our customers, but none of our own infrastructure is containerized --- though some of our customer-facing stuff, like the API server, is. We have a big fleet of machines, mostly in two roles (smaller traffic-routing "edge" hosts that don't run customer VMs, and chonky "worker" hosts that do). All these hosts run `fly-proxy`, a Rust CDN-style proxy…

Intriguing!

This makes me curious: How does one learn to design and build systems like this…?

Also: How do you folks at Fly decide what parts to use “as is” and what parts to build from scratch? Do you have any specific process for making those choices?

Re: Ask HN: Who operates at scale without containers?

#242

Post by p_l »

richardwhiuk wrote: post #233
p_l wrote: post #221

Earlier quoted context omitted.

One of the biggest benefits of k8s for me, back in 2016 when I first used it in prod, was that it threw away all the extra features of Docker and implemented them directly by itself - better. Writing was already in the wall that docker will face stern competition that doesn't have all of its accidental complexity (rktnetes and hypernetes were a thing already)

Kubernetes used to (tediously) pass everything through to Docker, but since 1.20, that's resolved, and it now uses containerd.

Not everything - for a bunch of things, the actual setup increasingly happened outside docker then docker was just informed how to access it, bypassing all the higher level logic in Docker.

1.20 is when docker mode got deprecated, IIRC, but many of us were already happily running in containerd for some time.

Re: Ask HN: Who operates at scale without containers?

#243

Post by Art9681 »

zomglings wrote: post #155
Thaxll wrote: post #84

Earlier quoted context omitted.

The thing is, you spent probably a lot of time on things that were granted elsewhere and while the rest of the world is improving on those tech you keep your home grown solution that is harder and harder to maintain. Plus the knowledge that is not transferable. > Containers only give reproducible deployments, not builds, so they would be a step down. This is not true, if you use a docker image A with specific version…

The OP is talking about reproducible builds . A docker image is a build artifact . If I gave you a Dockerfile for a node.js service that installed its dependencies using npm, and the service used non-trivial dependencies, and I gave you this file 10 years after it was first written, chances are you would not be able to build a new image using that Dockerfile. This would be a problem for you if you had to make some sm…

There are other container build tools such as Kaniko that have solved the reproducible builds issue right? If we're operating at scale, it probably means we are using some flavor of Kubernetes and the Docker runtime is no longer relevant. Would Redhat OpenShift's source-to-image (S2I) build process solve the reproducible builds requirement? This space moves quickly and the assumptions we had last week may no longer be relevant.

For example Crunchy Postgres offers an enterprise supported Kubernetes Operator that leverages stateful sets. Yet I am reading in these comments that it is an unsolved issue.

Re: Ask HN: Who operates at scale without containers?

#244

Post by mountainriver »

Ericson2314 wrote: post #202
mateuszf wrote: post #175

Earlier quoted context omitted.

As far as I know the final effect is great - getting reproducible, statically linked programs. However, learning Nix / NixOS is quite difficult - it requires a specific set of skills (functional programming, shell, Nix building blocks which you can learn from existing Nix code), the documentation is lacking, error messages are cryptic, debugging support is bad.

Yeah Nix is why "choose boring" is a bit of a misnomer. The truth is tautologically unhelpful: choose good technologies. Most people have trouble separating snake oil and fads from good things, so old/boring is a safe heuristic. Nix is not boring, but it is honest. It's hard because it doesn't take half measures. It's not an easy learning curve gaslighting to you to a dead end. But taste in choosing good technologies…

Thank you, I’ve always hated the Boring Technology idiom. The right tool doesn’t have to be boring, but it certainly can be

Re: Ask HN: Who operates at scale without containers?

#245

Post by tomc1985 »

usaphp wrote: post #126
tharne wrote: post #89

Earlier quoted context omitted.

> What works for us is to do the simplest thing that works, then iterate. The older I get, the more often I'm reminded that this un-sexy approach is really the best way to go. When I was younger, I always thought the old guys pushing boring solutions just didn't want to learn new things. Now I'm starting to realize that after several decades of experience, they simply got burned enough times to learn a thing or two h…

Or it could be that you just became old and don't want to learn new things anymore ))

Such ageism.

Re: Ask HN: Who operates at scale without containers?

#246

Post by mountainriver »

zthrowaway wrote: post #138
brimble wrote: post #44

Earlier quoted context omitted.

> That's the easy part! The hard part is managing databases. Ding ding ding ding ding. "But what about disk?" (so, relatedly, databases) is the hard part. Balancing performance (network disks suuuuuck) and flexibility ("just copy the disk image to another machine" is fine when it's a few GB—less useful when it's lots of GB and you might need to migrate it to another city and also you'd rather not have much downtime)…

People still think stateful things are impossible on k8s but Stateful sets and persistent volumes solves a lot of this. You should be relying on out of the box DB replication to make sure data is available in multiple areas. This is no different on other platforms.

Yes you can run DBs on kube now, much of people thinking this isn’t good comes from years back when it wasn’t

Re: Ask HN: Who operates at scale without containers?

#247

Post by treis »

hinkley wrote: post #125
barbazoo wrote: post #53

Earlier quoted context omitted.

> Containers only give reproducible deployments, not builds Could someone elaborate on this please? Doesn't it depend entirely on your stack how reproducible your build is? Say I have a Python app with its OS level packages installed into a (base) image and its Python dependencies specified in a Pipfile, doesn't that make it pretty reproducible? Is the weak spot here any OS dependencies being installed as part of the…

This is something that even the Docker core team is not entirely clear on (either they understand it and can't explain it, or they don't understand it). The RUN command is Turing complete, therefore cannot be idempotent. Nearly every usable docker image and base image include the RUN command, often explicitly to do things that are not repeatable, like "fetch the latest packages from the repository". This is all befor…

>Getting repeatable docker images

But there's no real need for repeatable build docker images. You copy the image and run it where ever you need to. The entire point of Docker is to not have to repeat the build.

Re: Ask HN: Who operates at scale without containers?

#248

Post by tomc1985 »

rattlesnakedave wrote: post #229
goodpoint wrote: post #136

Earlier quoted context omitted.

Such "new things" are just a big bunch of unnecessary complexity.

Or is cope for a loss of neuroplasticity? Has to be evaluated case by case. “Experience” doesn’t count for much when evaluating new technical tooling. Landscape shifts far too much.

The landscape doesn't need to shift nearly as often as it does. But we are an industry obsessed with hiring magpie developers as cheaply as we can, and those magpie developers demand cool merit badges to put on their resumes, and here we are.

"Everything old is new again" does not BEGIN to do tech justice. Compared to nearly every other profession on the planet, tech knowledge churn is like 10x as fast.

Re: Ask HN: Who operates at scale without containers?

#249

Post by charcircuit »

Ericson2314 wrote: post #202
mateuszf wrote: post #175

Earlier quoted context omitted.

As far as I know the final effect is great - getting reproducible, statically linked programs. However, learning Nix / NixOS is quite difficult - it requires a specific set of skills (functional programming, shell, Nix building blocks which you can learn from existing Nix code), the documentation is lacking, error messages are cryptic, debugging support is bad.

Yeah Nix is why "choose boring" is a bit of a misnomer. The truth is tautologically unhelpful: choose good technologies. Most people have trouble separating snake oil and fads from good things, so old/boring is a safe heuristic. Nix is not boring, but it is honest. It's hard because it doesn't take half measures. It's not an easy learning curve gaslighting to you to a dead end. But taste in choosing good technologies…

Nix isn't a good option for developing rails applications. The nix ecosystem doesn't have good support for ruby meaning you will waste a lot of time and run into a lot of issues getting stuff to work. That's not boring. An example of being boring would be using Nix for something like C where the ecosystem around it is already built up.

I was really pro Nix at the time I took on a rails project and it caused many problems for my team due to Nix just not having good support for my use case. If I could go back in time I would definitely choose something else.

Re: Ask HN: Who operates at scale without containers?

#250

Post by tptacek »

cpach wrote: post #241
tptacek wrote: post #212

Ironically, here at Fly.io, we run containers (in single-use VMs) for our customers, but none of our own infrastructure is containerized --- though some of our customer-facing stuff, like the API server, is. We have a big fleet of machines, mostly in two roles (smaller traffic-routing "edge" hosts that don't run customer VMs, and chonky "worker" hosts that do). All these hosts run `fly-proxy`, a Rust CDN-style proxy…

Intriguing! This makes me curious: How does one learn to design and build systems like this…? Also: How do you folks at Fly decide what parts to use “as is” and what parts to build from scratch? Do you have any specific process for making those choices?

We had to build the orchestration stuff (it was originally a Nomad driver, but has outgrown that) because the tooling to run OCI containers as Firecracker VMs didn't exist in a deployable form when we started doing this stuff.

Most of the big CDNs seem to start with an existing traffic server like Nginx, Varnish, or ATS. One way to look at what we did with our "CDN" layer is that rather than building on top of something like Nginx, we built on top of Tokio and Hyper and its whole ecosystem. We have more control this way, and our routing needs are fussy.

By comparison, we use VictoriaMetrics and ElasticSearch (I don't know about "as-is" --- lots of tooling! --- but we don't muck with the cores of these packages), because our needs are straightforwardly addressed by what's already there.

Lots of companies doing stuff similar to what we're doing have elaborate SDN and "service mesh" layers that they built. We get away with the Linux kernel networking stack and a couple hundred lines of eBPF.

We definitely don't have a specific process for this stuff; it's much more an intuition, and is more about our constraints as a startup than about a coherent worldview.

Post reply on HN