Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who operates at scale without containers?

news.ycombinator.com

Post reply on HN
Print view View original

281–290 of 446 posts

Re: Ask HN: Who operates at scale without containers?

#281

Post by Ericson2314 »

tharne wrote: post #210
Ericson2314 wrote: post #202

Earlier quoted context omitted.

Yeah Nix is why "choose boring" is a bit of a misnomer. The truth is tautologically unhelpful: choose good technologies. Most people have trouble separating snake oil and fads from good things, so old/boring is a safe heuristic. Nix is not boring, but it is honest. It's hard because it doesn't take half measures. It's not an easy learning curve gaslighting to you to a dead end. But taste in choosing good technologies…

> It's not an easy learning curve gaslighting to you to a dead end. This is a really good way of describing a certain type of technology. I'm stealing this line.

Please do! In this current era of VCs loving developer tool businesses, we are seeing a whole lot of this!

Re: Ask HN: Who operates at scale without containers?

#282

Post by JohnBooty »

anonymousDan wrote: post #133
javajosh wrote: post #127

Earlier quoted context omitted.

What you're asking for is an essay on comparative devops architectures, with a focus on k8s alternatives. I think what you'll find is a lot of ad hoc persistent systems that tend to drift over time in unpredictable ways, and take on the feel of a public lobby if you're being generous, a public restroom if you're not. So what you're asking is really a sample of these ad hoc approaches. What I think you'll find are a f…

Availability?

Devops is not my area of expertise to put it mildly, but "a single big-ass server" scenarios are often pretty well covered by "a second big-ass server configured as a mirror/hot-spare", right?

Depends on how many nines of uptime you need, of course, and other such things.

(Obviously, there are reams of cases where this isn't sufficient)

Re: Ask HN: Who operates at scale without containers?

#283

Post by JohnBooty »

rootusrootus wrote: post #69
smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

> But I'd so often dive in to understand the details and be disillusioned about how much complexity there is for relatively little benefit. I feel similarly about many cloud services in e.g. AWS. I get where it's sometimes handy, but the management overhead can get pretty insane. I used to hear people say "just do X in AWS and ta-da you're done!" only to find out that in many cases it isn't actually a net improvement…

Yeah, I feel this way a lot.

In recent years I've done work for companies where it feels like the end result of a lot of very expensive AWS infra is... more expensive, and not really superior to things we had ten years ago when hosting on-prem.

There's a level of scale for which distributed AWS/Azure is necessary, but I'm not sure how many shops need that.

The devops guy at a previous gig thought I was some kind of naive simp for insisting we didn't need a cluster of Amazon's version of Redis. We were storing like, a megabyte of data in there. That's not a typo.

It's not that he was an idiot. Smart guy actually. Not sure if he was just padding his resume with experience ("sure, I've admin'd Redis clusters!") or if he was locked into some kind of default thinking that wouldn't even allow him to consider some kind of minimal setup.

Re: Ask HN: Who operates at scale without containers?

#284

Post by jshen »

I’m not fully sure I understand your question, particularly the part about scale. By scale do you mean a large organization (I.e. thousands of developers with highly heterogeneous tech stacks), or by scale do you mean a high volume of traffic?

If you mean high volume, I’ve seen people do it with serverless stacks like Google app engine or AWS lambda and some managed databases.

If you mean large organization, then I haven’t heard of anyone doing it without containers or VMs. It’s not that alternatives don’t exist, it’s that it’s nearly impossibly to get everything onto an alternative and containers are extremely flexible. For example, large companies often buy other companies and it’s prohibitively expensive to rebuild it all relative to the cost of putting it in a container and leveraging the build/deploy pipelines that they have already. Same thing with ancient legacy tech, far easier to containerize it than rebuild it.

Re: Ask HN: Who operates at scale without containers?

#285

Post by time0ut »

I manage a mid-size fleet of EC2s across 4 AWS regions with several auto scaling groups per region. The VMs themselves are general purpose and the image is built with our standard set of dependencies. All of our applications are homogeneous as far as stack, so the process for managing the VM ecosystem is pretty simple.

The various application binaries are deployed through a small set of SSM documents (with an automation layer built on Jenkins in front and using S3 to shuffle things around). The design predates me, but I've made small improvements to it. It works fine and its cheap as far as AWS stuff goes.

We are slowly migrating applications to ECS Fargate which I have yet to form a strong opinion on. Docker is a nice experience even if the leanest artifacts are still chunkier than a naked application.

For reference, I have built some larger and some smaller stuff using k8s and even docker+machine once.

May not be at the scale you are thinking, but it serves maybe 100k users spread across 10k customers working in 30 something countries.

Re: Ask HN: Who operates at scale without containers?

#286

Post by charcircuit »

dqv wrote: post #271
charcircuit wrote: post #249

Earlier quoted context omitted.

Nix isn't a good option for developing rails applications. The nix ecosystem doesn't have good support for ruby meaning you will waste a lot of time and run into a lot of issues getting stuff to work. That's not boring. An example of being boring would be using Nix for something like C where the ecosystem around it is already built up. I was really pro Nix at the time I took on a rails project and it caused many prob…

What was the problem you were having with Ruby (on Rails) and Nix? It wasn't ergonomic for me, but I was able to get things to work.

I think one of the main issues was depending on a custom gem. And then something about sandboxing, and outdated things being cached (maybe this was just a rails thing though). Also setting up rails project was convoluted. It was just obvious that there wasn't much time devoted towards that use case of Nix.

The rails support was more of just to handle packaging existing software as opposed to being good for a development environment.

Re: Ask HN: Who operates at scale without containers?

#287

Post by abecedarius »

claytonjy wrote: post #47
smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

is Nix a hurdle when onboarding engineers? do only a few people need to know the language? I've wanted to use Nix and home-manager for personal stuff but the learning curve seems big.

Just my experience with Nix, two or three years back: I went to the Nix website for the install instructions, saw something like "curl | sh", said "what? no way" and used the alternative non-recommended out-of-date instructions, which turned into a full-day rabbit hole until it would build. A year later I went through this again for another machine, and this time just gave up.

So I would say curl|sh is probably the way to go, which I guess means I'd only want to install it into a container. But I do really like the idea of Nix.

Re: Ask HN: Who operates at scale without containers?

#288

Post by treis »

hinkley wrote: post #269
treis wrote: post #247

Earlier quoted context omitted.

>Getting repeatable docker images But there's no real need for repeatable build docker images. You copy the image and run it where ever you need to. The entire point of Docker is to not have to repeat the build.

If you could explain that to the Docker team you would be performing a great humanitarian service. They for many years close issues and reject PRs based on the assertion that Dockerfile output needs to be repeatable (specifically, that the requested feature or PR makes it not repeatable). It’s not, it can’t be without a time machine, and if it was I believe you’d find that ace icing traction would have been more diff…

Do you have an example?

Re: Ask HN: Who operates at scale without containers?

#289

Post by musicale »

1. Containers != Docker.

2. If all you need are resource limits, processes in cgroups work pretty well.

3. Networking adds complexity - avoid network namespaces if you can, and use use an abstraction layer so the application doesn't have to worry about things like wire formats, encryption, TCP connections, IP addresses, and port numbers.

Re: Ask HN: Who operates at scale without containers?

#290

Post by gnufx »

freemint wrote: post #182

Almost every HPC center. Tech stack: Linux (RHEL-like); MPI as middle ware for distributed communication through vendor specific communication hardware (also called interconnect); shared high performance network filesystem usually setup on login node, scheduler like SLURM, IBM Spectrum LSF Suites or others to launch jobs from login node which accessed via SSH. This setup scales to tens of thousands of machines.

I'd hope the filesystem isn't provided from the login node, but otherwise, yes. That said, they may still run jobs with unprivileged -- I hope -- container images. There are some that seem to feel the need to make life more complicated, but then you have N problems compared with stateless nodes off a networked root running basically native binaries.
Post reply on HN