Live data from Hacker News

Ask HN: Who operates at scale without containers?

news.ycombinator.com

191–200 of 446 posts

Re: Ask HN: Who operates at scale without containers?

#191
I worked at (massive live-streaming website) and my ops team operated tens of thousands of bare-metal machines. Not to say we didn't have an enormous amount of containerized infrastructure in AWS, but we had both.

When the company was younger containerized networking had latency and throughput issues, especially when you were trying to squeeze every bit of traffic you can from a white-box bare-metal server, i.e. bonding together 10Gb or 40Gb network interfaces. The other thing is that the orchestration engines like K8s simply had maturity issues when not using Cloud Load Balancers.

As for the implementation details, I've worked at lots of companies doing metal and they look a lot alike. PXE and TFTP, something like Chef, Puppet, Ansible (But at a certain scale you have to transcend those tools and come up with better patterns), you need services to manage IPMI or console servers, power strips, etc., you need a team of folks to rack and stack things, you need inventory, you need network engineers, and so on. At a certain scale you can simply push code around with SSH and a build system, at a scale beyond that you need to come up with some special sauce like P2P asset distribution or an internal CDN. At the pinnacle of bare-metal, you'd ideally have a very evolved control-plane, a slimmed down OS that runs a single static binary, and a stateless application. It takes a lot of work to get there.

Of course, getting servers to run some code is scratching the surface. Service discovery, network architecture, security, etc., are all things that require specialized skill sets and infrastructure. You also need to build and maintain every "glue" service you get from a cloud provider, you need to run file servers, you need to run repositories, you need to run and manage your own databases, and so on. Sometimes you can hybridize those with cloud services but that opens up yet another can of worms and teams of people who need to answer questions like.. what if The Cloud(tm) goes down? What if there's some kind of split brain scenario? What if there's a networking issue? How does service discovery work if half of the services disappear? etc. etc.

Re: Ask HN: Who operates at scale without containers?

#192
post #97

Earlier quoted context omitted.

Google has used debian as their base. Netflix uses a BSD flavor (I forget which) as their CDN cache. FB used CentOS, not sure what they use today since CentOS is EOL'd. Debian (and, formerly, CentOS) is a good standard: it occupies a sweet spot between ubuntu server and alpine, in the sense that it's batteries-included and very well-supported (apt/yum), but not particularly bloated. I use debian for all my personal s…

Useful insight, thanks. What attracts you to BSD over Debian if you were to go that route at some point?

Having an operating system where the kernel and core userland are developed together is a qualitatively different experience to one that's been stitched together from external projects. I'm not really sure how to describe the difference in experience but the "this is an integrated whole" feel is really quite pleasant as a sysadmin and as a user.

Note that my personal infrastructure is a mixture of Debian and FreeBSD and I dearly love both - if you forced me to pick one of the two to keep I'd be horribly torn.

Re: Ask HN: Who operates at scale without containers?

#193
post #77

Earlier quoted context omitted.

I've always thought of Apollo environments as containers before kernel features for containers existed. With enough environment variables and wrapper scripts taking the name of real binaries to populate stuff like LD_LIBRARY_PATH, Apollo makes a private environment that is only _slightly_ contaminated by the host.

Apollo reminded me more of Nix than containers. The wrapper scripts are super Nix-y :)

That's what it was. VM with a barebones deployment system that had a ton of hooks in it.

Really really smart idea that IMO helped Amazon in the 2010s immensely. While everyone else was figuring out k8s and whatnot, Amazon had a good system with CI in place for years.

I wonder how it's fared over time. Amazon was never known for internal tooling in many other places. I hope Apollo is still running strong today.

Re: Ask HN: Who operates at scale without containers?

#194
post #178

Earlier quoted context omitted.

Jonathan blow had a great take on this that really spoke to me. I can’t do it justice, but paraphrasing as best I can: Get the simple to understand, basic thing working and push the harder refactor / abstraction until later. Leave the fancy stuff for a future engineer who better understands the problem. That future engineer is you with more experience with the actual problem.

The way I see it, I can either prematurely use the complex solution where I don't fully understand the problem or the solution OR I use the simple solution and learn why the complex solution is needed and more importantly what parameters and specific ways I can best leverage the complex solution for this specific problem.

Sometimes the simple solution works well enough for as long as you need a solution, too. Often times, it just works, or the problem changes, or the company changes.

Re: Ask HN: Who operates at scale without containers?

#195

Many years ago I worked at a place that deployed thousands of bare metal servers. We were effectively running our own cloud before the cloud became a thing. The way it worked was simple. We created our own Red Hat variant using a custom Anaconda script. We then used PXE boot. During OS install this script would call back to our central server to provision itself. You can do that a few ways. If I recall, we baked in a…

Did you work for Tellme too?

Re: Ask HN: Who operates at scale without containers?

#196
post #179

Earlier quoted context omitted.

Interesting -- how did you handle redeploys? Given that game servers are stateful (so you'd want to drain servers at their own pace instead of force them down at a specific time), it seems like redeploying a server without machinery to do things like dynamically allocate ports/service discovery for an upstream load balancer would be tricky.

> it seems like redeploying a server without machinery to do things like dynamically allocate ports/service discovery for an upstream load balancer would be tricky like most things with running servers, it's not that hard, there's just an industry dedicated to making people think it's hard (the tech industry). Every game has a room code to identify the game, and a websocket open handshake has a URL in it. Every room…

How do you handle deploying a new version of nginx without forcibly closing connections?

Re: Ask HN: Who operates at scale without containers?

#197

Place I worked until recently had (and probably still has) the majority of the site running on bare metal. Java stack, home grown RPC framework, home grown release system that boiled down to a whole lot of rsync and ssh commands by one controller script which knew how to do things in parallel. Configuration was through files which lived in source control. Our servers were hand packed, which would have sucked except w…

Moving stuff around is always aggravating but I do suspect that for a lot of organisations the time spent swearing at that ends up being lower over time than the time spent swearing at k8s until it reliably does it for you.

(this is an observation, not a recommendation, different teams and workloads will result in different preferences for how to pick your trade-offs)

Re: Ask HN: Who operates at scale without containers?

#199
I worked one place that did so. Our traffic was sharded over a couple dozen sites, but combined we were at US top-100 scale.

Every site was on a very large bare metal box (sites were grouped together when possible, IIRC only one required it’s own dedicated machine). Each box was a special snowflake.

The DBs were on separate hardware.

When I left they were starting to embrace containerized microservices.

Re: Ask HN: Who operates at scale without containers?

#200
I've done relatively large scale projects without containers.

In one case, we were running something like 80% of all auto dealer websites on two bare metal web servers and one bare metal SQL Server db, with a fail-over hot replica. Quite a bit of traffic, especially on holiday weekends, and we never came close to maxing out the machines. This was in 2007, on fairly modest hardware.

I used to write fulfillment systems for Verizon, we handled about 30,000 orders and 10,000ish returns per day, with pretty complex RMA and advance-replacement logic, ILEC integration and billing in Business Basic on NCR Unix, with complex pick/pack/ship rules and validation. Again, that was a single bare metal db server, SQL Server and a web server with SOAP/XML/WSDL services (this was in early 2000's, on laughable hardware by today's standard).

I was part of writing a healthcare claims processing system that did about 1TB per day of data processing and storage, on a single bare metal SQL Server instance and OLAP cubes for analytics.

I've also been involved in projects that took the opposite approach, Kubernetes, Kafka, CQRS, etc... in order to do "massive scale" and the result was that they struggled to process a few thousand health care messages per day. Obviously the devil is in the details of implementation, but I wasn't particularly impressed with the "modern" tech stack. So many layers of abstraction, each has a performance and operational cost.

These days I mostly use Node and Postgres, so I haven't had a lot of need for containers. npm install is a pretty simple mechanism for dependencies, I try to keep the stack minimal and lean. With the current cloud offerings of hundreds of VCPUs, hundreds of gigs of memory and petabytes of storage, it's difficult for me to envision a scenario where vertical scale wouldn't meet the needs of any conceivable use case.

This works for me, partly because I'm a fair hand at sysadmin stuff on linux and prefer maintaining a well-tuned "pet" over a bunch of ephemeral and difficult to debug "cattle".

Post reply on HN