Ask HN: Who operates at scale without containers? • Hacker News Board

Re: Ask HN: Who operates at scale without containers?

#421

Post by exikyut » Wed, Mar 23, 2022, 6:13 PM UTC

senorsmile wrote: post #296

JohnBooty wrote: post #282

Earlier quoted context omitted.

Devops is not my area of expertise to put it mildly, but "a single big-ass server" scenarios are often pretty well covered by "a second big-ass server configured as a mirror/hot-spare", right? Depends on how many nines of uptime you need, of course, and other such things. (Obviously, there are reams of cases where this isn't sufficient)

In 2009 I had a pair of (what I thought at the time) very cleverly architected VM servers with complete redundancy between them. Either one could be pulled and there'd maybe be a few seconds of data loss at the worst. One day lightning struck the building and even managed to "jump" the grounded pair of UPS's and fried all hard disks in both nodes. Thankfully I had both onsite and offsite backups; but it took a couple…

Two questions:

1. How would power conditioners and lightning arresters handle these sorts of situations?

2. Was this using VMWare fault tolerant VMs, just out of curiosity? (It continually live-migrates one VM to a hot spare over a 10G link so either can disappear and the VM continues to run.) Or was this a bespoke application architecture implementing effectively the same thing?

Re: Ask HN: Who operates at scale without containers?

#422

Post by freedomben » Wed, Mar 23, 2022, 6:41 PM UTC

tptacek wrote: post #418

freedomben wrote: post #412

Earlier quoted context omitted.

What are you using for the host OS on your bare metal?

Linux. :)

heh, could you share if it's more of an rpm flavor or more of a deb flavor? :-) (or some third option. no false dichotomies intended. actually it would be badass if you're running Arch :-D)

Re: Ask HN: Who operates at scale without containers?

#423

Post by tomc1985 » Wed, Mar 23, 2022, 7:12 PM UTC

goodpoint wrote: post #415

iamstupidsimple wrote: post #397

Earlier quoted context omitted.

The vast majority of computer science lectures from 20 years ago are still relevant. A ton has changed in terms of frameworks but the fundamentals are mostly the same.

20?! More like 60 or 70, especially if around distributed systems and Dijkstra's work in general. [1] It's amazing how some people are familiar with every framework of the month and don't understand how a computer works... [1] https://en.wikipedia.org/wiki/List_of_important_publications...

Maybe I was in the wrong subsector of this industry but there was very little theory work in the jobs I have held. A lot of it was just reusing the same design patterns over and over again, and then scaling that out somehow.

I did not mean my comment to say that seminal works in CS theory are useless, just that most of the books published in this field cover more practical matters and they tend to age very quickly. Like I have an e-bookshelf full of Packt freebies back when they did that, and those books aren't really that useful now unless I want to start on an old version of something.

Re: Ask HN: Who operates at scale without containers?

#424

Post by Ericson2314 » Wed, Mar 23, 2022, 7:24 PM UTC

debaserab2 wrote: post #367

charcircuit wrote: post #286

Earlier quoted context omitted.

I think one of the main issues was depending on a custom gem. And then something about sandboxing, and outdated things being cached (maybe this was just a rails thing though). Also setting up rails project was convoluted. It was just obvious that there wasn't much time devoted towards that use case of Nix. The rails support was more of just to handle packaging existing software as opposed to being good for a developm…

What do you mean exactly? At the end of the day, rails is just a collection of ruby gems. If nix has first class support for ruby and Gemfiles in theory there shouldn't be a ton of problems. I've not used Nix so I legitimately don't know (but I am quite interested in it)

There are no fundamentals hurdles, but because we are currently prevented from properly dogfooding these tools in Nixpkgs, they tend to be unpolished in various ways, they are just used to by a few random projects in relative isolation.

Re: Ask HN: Who operates at scale without containers?

#425

Post by fnordpiglet » Wed, Mar 23, 2022, 8:00 PM UTC

smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

I think containers and k8s start to make sense when you need to impose logical isolation on your flat topology networks, often for security but also for preventing everything becoming overly complex in their dependencies. But this IMO isn’t often a concern until your enterprise is very large and complex with some very crufty dependencies that you realize eventually you can never practically demise because you can’t even fully determine what all depends on it at any given time.

That said, I would prefer managing that stuff through SDNs rather than trying to make user space overly complex software replication of commodity capabilities. Or, just use a cloud provider to start managing the isolation for the parts that benefit for it. What really baffles me is the use of k8s in aws. Why?

BTW - congrats on effective use of Nix. I use it extensively at home for my home services. It’s got some weirdness to it for sure but it’s a heck of a lot better than almost anything I’ve ever seen to date. I can’t wait to see what evolves out of it.

Re: Ask HN: Who operates at scale without containers?

#426

Post by wanderr » Wed, Mar 23, 2022, 11:07 PM UTC

mhitza wrote: post #37

wanderr wrote: post #28

Grooveshark didn't use any of that. We were very careful about avoiding dependencies where possible and keeping our backend code clean and performant. We supported about 45M MAU at our biggest, with only a handful of physical servers. I'm not aware of any blog posts we made detailing any of this, though. And if you're not familiar with the saga, Grooveshark went under for legal, not technical reasons. The backend API…

What a great service. I'd be curious if you could go into details how the radio feature worked back then, because I found myself receiving worse suggestions when I used similar features in Spotify/Google Play Music.

Oh man, I should write a blog post about that, as I built that feature myself. It was meant to be a stopgap until we could get some real matchine learning in there, but nothing else we tried did as well. First, for efficiency all recommendations were artist to artist,nl not song to song. That works well for a lot of genres but is pretty bad for others. We started with a free DB of artist similarities, I don't remember where we got that from, maybe musicbrainz? We built a shitty internal interface for adding and removing links between artists and adjusting the weights of those links and then made it available to all employees to mess with. As you might imagine just about everyone there was passionate about music so it didn't take long to crowdsource a huge catalog of quality recommendations and then for really obscure stuff we would fall back to the open db. So the actual algorithm would look at your seeds - artists you put in the queue before turning on radio or artists with songs that you liked while radio was on, pull the top n linked artists for each of your seed artists, and do some weighted shuffling. It would also make sure to space out artists so you don't hear the same one too often etc. Then for genre radio we just secretly selected a bunch of artists we felt were representative of the genre and used those as the seeds. Oh yeah and if you disliked a song we'd prevent that artist from playing for the rest of your session. We also would look at anomalies like popular artists with not many recommendations, or artists that, when used as seeds, lead to shorter listening sessions (implying that the recommendations need to be cleaned up). Most attempts to replace this with something smarter ran into 2 problems: 1. Popular stuff is popular, so it looks like a good recommendation for anything, and 2. ML is hard and takes a lot of time, which we never had enough of

Re: Ask HN: Who operates at scale without containers?

#427

Post by p10_user » Wed, Mar 23, 2022, 11:23 PM UTC

hinkley wrote: post #340

p10_user wrote: post #222

Earlier quoted context omitted.

Nice post. I wish I was reading this 6 years ago, but at least I quickly learned to be humble about the quality of my code.

I'm not quite sure how to interpret what you've said exactly. Some of what I said is a performance piece. If I insist that I am human and thus make mistakes, then it makes space both for others to make mistakes and for us to have a frank conversation of how, in our more lucid moments, we can do things that fight/counteract our basic nature. It's morally equivalent to putting a plastic cover over the Big Red Button so…

Everything you are talking about seems centered around cultivating humility. You are absolutely right. It’s ultimately the ability to allow yourself to make mistakes. Being wrong is no fun. Best to get it over with, accept it and work forward, rather than to bring yourself into some delusion.

Re: Ask HN: Who operates at scale without containers?

#428

Post by qwertywert_ » Thu, Mar 24, 2022, 1:10 AM UTC

p_l wrote: post #393

qwertywert_ wrote: post #319

Earlier quoted context omitted.

I thought its moving to CRI-o as well instead of containerd, or is k8s just not using docker, and containerd still supported in the future?

CRI-O is the target and containerd is the most common runtime implementing it at the moment.

Are you sure? This isn't my subject area but CRI-O looks like an alternative to containerd and implements the OCI compliant runtime like containerd does. And then there is a 3rd which is docker engine which is the one being dropped.

Re: Ask HN: Who operates at scale without containers?

#429

Post by tptacek » Thu, Mar 24, 2022, 2:06 AM UTC

freedomben wrote: post #422

tptacek wrote: post #418

Earlier quoted context omitted.

Linux. :)

heh, could you share if it's more of an rpm flavor or more of a deb flavor? :-) (or some third option. no false dichotomies intended. actually it would be badass if you're running Arch :-D)

At this moment I will go as far as to say that it is not a badass flavor of Linux.

:)

Re: Ask HN: Who operates at scale without containers?

#430

Post by jandrewrogers » Thu, Mar 24, 2022, 6:59 AM UTC

anonymousDan wrote: post #139

jandrewrogers wrote: post #131

Earlier quoted context omitted.

Sure, Kubernetes can run a database, but not efficiently. Companies with intensive data infrastructure frequently operate at scale without containers, either VMs or bare metal. The larger the data volume, the less likely they are to use containers because efficiency is more important. It is also simpler to manage this kind of thing outside containers, frankly, since you are running a single process per server. People…

Can you point to any resources on the perf implications of docker for databases?

High-performance database engines take complete control of their resources, I/O, and scheduling, completely bypassing the OS kernel. Linux is explicitly designed to allow this. This architecture enables integer factor improvements in throughput on the same hardware, which is why you would design your software this way. Older, slower database engine designs (e.g. Postgres) don't take explicit control of resources in this way and will work reasonably well in containers.

The original Linux container design was never intended with this type of software in mind, and tacitly reduces the control a process has over its resources. Invariants required for performance are violated, and therefore performance suffers relative to bare metal (or conditionally VMs). The v2 container implementation, which isn't widely available yet, recognizes these problems and attempts to remedy some of these poor behaviors of the v1 container implementation.

If your database does not assume strict control of its underlying hardware resources then this does not affect you. However, this kind of strict resource control is idiomatic in all the highest performing databases architectures, to great effect, so it is the most performant databases that are most adversely affected by containerization.

FWIW, virtual machines used to have similar terrible performance for high-performance databases for similar reasons. Now there are mechanisms for the database to effectively bypass the VM similar to the kernel, and the loss of performance is minimal. Container environments typically don't offer similar bypass mechanisms.