Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who operates at scale without containers?

news.ycombinator.com

Post reply on HN
Print view View original

371–380 of 446 posts

Re: Ask HN: Who operates at scale without containers?

#371

Post by charcircuit »

debaserab2 wrote: post #367
charcircuit wrote: post #286

Earlier quoted context omitted.

I think one of the main issues was depending on a custom gem. And then something about sandboxing, and outdated things being cached (maybe this was just a rails thing though). Also setting up rails project was convoluted. It was just obvious that there wasn't much time devoted towards that use case of Nix. The rails support was more of just to handle packaging existing software as opposed to being good for a developm…

What do you mean exactly? At the end of the day, rails is just a collection of ruby gems. If nix has first class support for ruby and Gemfiles in theory there shouldn't be a ton of problems. I've not used Nix so I legitimately don't know (but I am quite interested in it)

I don't remember exactly the problem since it was a couple years ago. There was a lot of issues in regards to failing to run / build rails which regarded doing special nix stuff to fix. It wasn't possible to include a custom gem from a separate directory. I had to disable some sort of sandboxing to prevent Nix from complaining. Nix kept getting in my way and as my team members were not familiar with Nix they couldn't fix it themselves. Sadly, I ruined their opinion of Nix by showing them it with this project.

Re: Ask HN: Who operates at scale without containers?

#372

Post by pojzon »

SQueeeeeL wrote: post #103
disintegore wrote: post #62

Earlier quoted context omitted.

I asked the question without context because I didn't want to fire the thread off in the wrong direction but I suppose in a comment chain it's fine. For what it's worth we do use containers and K8s heavily at my current job. I know that there are quite a few people opposed to the state of containers and the technologies revolving around them. I don't think the arguments they present are bad. [Attacking the premise](…

>So essentially I'm interesting in knowing how people find ways to reproduce the value these technologies offer, what they instead rely on, which things they leave on the table Decades of work typically. That or just standard parallel deployments, most things Docker is good at would take the average developer many unnecessary hours to reproduce.

Currently there is a maltitude of solutions available on the market that make containers and k8s just a complex and hard to maintain choice.

It all boils down to “what bullshit was my management sold on”.

Promises of “easy to use, easy to replace, cloud-agnostic, doesnt require domain knowledge” etc etc etc.

Its all bullshit. At some point/scale you need experts in the field that cash in a month more than average dev per year.

Re: Ask HN: Who operates at scale without containers?

#373

Post by DeathArrow »

javajosh wrote: post #127
disintegore wrote: post #62

Earlier quoted context omitted.

I asked the question without context because I didn't want to fire the thread off in the wrong direction but I suppose in a comment chain it's fine. For what it's worth we do use containers and K8s heavily at my current job. I know that there are quite a few people opposed to the state of containers and the technologies revolving around them. I don't think the arguments they present are bad. [Attacking the premise](…

What you're asking for is an essay on comparative devops architectures, with a focus on k8s alternatives. I think what you'll find is a lot of ad hoc persistent systems that tend to drift over time in unpredictable ways, and take on the feel of a public lobby if you're being generous, a public restroom if you're not. So what you're asking is really a sample of these ad hoc approaches. What I think you'll find are a f…

>I can't imagine why you'd need more than 16 cores, 128G of RAM and 10T of storage to prove out an ordinary SaaS. That is a ferocious amount of resources.)

In my case because I was working for a checkout solution for a Cash & Carry chain, with 1000 stores in 25 countries, each with tens of tills, self service points, queue busting tills who also had to serve the web shop. And that solution had to work also when there was no Internet access.

Re: Ask HN: Who operates at scale without containers?

#374

Post by pojzon »

thinkingkong wrote: post #67
barbazoo wrote: post #53

Earlier quoted context omitted.

> Containers only give reproducible deployments, not builds Could someone elaborate on this please? Doesn't it depend entirely on your stack how reproducible your build is? Say I have a Python app with its OS level packages installed into a (base) image and its Python dependencies specified in a Pipfile, doesn't that make it pretty reproducible? Is the weak spot here any OS dependencies being installed as part of the…

The dependencies can change out from underneath you transparently unless you pin everything all the way down the stack. Upstream docker images for example are an easy to understand vector of change. The deb packages can all change minor versions between runs, the npm packages (for example) can change their contents without making a version bump. Theres tons of implicit trust with all these tools, build wise.

Tbh I hope ppl do pin stuff to specific versions and have their own repositories of packages because you dont want to have external dependencies during builds that can fail.

DevOps 101 more or less..

Re: Ask HN: Who operates at scale without containers?

#375

Post by freemint »

gnufx wrote: post #290
freemint wrote: post #182

Almost every HPC center. Tech stack: Linux (RHEL-like); MPI as middle ware for distributed communication through vendor specific communication hardware (also called interconnect); shared high performance network filesystem usually setup on login node, scheduler like SLURM, IBM Spectrum LSF Suites or others to launch jobs from login node which accessed via SSH. This setup scales to tens of thousands of machines.

I'd hope the filesystem isn't provided from the login node, but otherwise, yes. That said, they may still run jobs with unprivileged -- I hope -- container images. There are some that seem to feel the need to make life more complicated, but then you have N problems compared with stateless nodes off a networked root running basically native binaries.

I never seen a HPC center where use of containers was encouraged. It is just running binaries from user directories on disk under different users.

Attempts to introduce containers in a way which doesn't stand in the way are treated as revolutionary. Such as the Fuzzball scheduler (I found no good written source on a quick skim so have a talk) https://youtu.be/Pbmxq3dg35E

Re: Ask HN: Who operates at scale without containers?

#376

Post by pojzon »

deathanatos wrote: post #277
smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

> Kubernetes has mostly focused on stateless applications so far. That's the easy part! The hard part is managing databases. Kubernetes can absolutely host stateful applications. A StatefulSet is the app-level construct pretty much intended for exactly that use case. My company runs a distributed database on top of Kubernetes. (We have another app that uses StatefulSets to maintain a cache across restarts, so that it…

For anyone looking for confirmations look for Zalando’s Postgres Operator. They host 100++ db clusters using that also for production workloads.

Re: Ask HN: Who operates at scale without containers?

#377

Post by DeathArrow »

nijave wrote: post #326
zthrowaway wrote: post #138

Earlier quoted context omitted.

People still think stateful things are impossible on k8s but Stateful sets and persistent volumes solves a lot of this. You should be relying on out of the box DB replication to make sure data is available in multiple areas. This is no different on other platforms.

Putting a 50TB+ database that enables eye watering revenue on k8s is a hard sell for a lot of businesses--especially when they have non-containerized solutions that work. Simple topologies and NoSQL databases (or databases they can handle replication/partitioning/node failures automatically) are pretty easy to stick in StatefulSets. There's crazy things like this https://blogs.oracle.com/mysql/post/circular-replicati…

>Putting a 50TB+ database that enables eye watering revenue on k8s

But you can run the software in k8s while using an external data store for data. With microservices you can also use small databases for each and push the data you want to persist like finished transactions and invoices to external systems.

No reason to have only one giant database to do everything in. At my last place of work we moved from a monolithic software using a giant database to microservices and many small databases, some of them NoSQL.

Re: Ask HN: Who operates at scale without containers?

#379

Post by jesterson »

CTO at e-commerce. Operating full-chain of e-commerce from customer to supplier, including backends for planning, support, marketing, operation etc. Thousands of orders daily, millions of visitors on webfront.

Not a single container. Don't see any possible use or benefits of it whatsoever. It's trendy, cool, but if you just want to get things done, avoid mess in your infrastructure and avoid accumulation of tech debt it's probably not the tool to go with.

Re: Ask HN: Who operates at scale without containers?

#380

Post by ComradePhil »

smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

If you really believed what you pretend to, you would be programming in .NET on Windows computers and running your apps on Windows... or even Java.

Or take that problem to another level and use golang.

Instead, you use python and try to fix the problems with another immature solution: Nix, which has uniform builds... but not really in certain cases... and it is the same on every Linux... except not really so you kind of have to use NixOS.

Post reply on HN