Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who operates at scale without containers?

news.ycombinator.com

Post reply on HN
Print view View original

331–340 of 446 posts

Re: Ask HN: Who operates at scale without containers?

#331

Post by JohnBooty »

senorsmile wrote: post #296
JohnBooty wrote: post #282

Earlier quoted context omitted.

Devops is not my area of expertise to put it mildly, but "a single big-ass server" scenarios are often pretty well covered by "a second big-ass server configured as a mirror/hot-spare", right? Depends on how many nines of uptime you need, of course, and other such things. (Obviously, there are reams of cases where this isn't sufficient)

In 2009 I had a pair of (what I thought at the time) very cleverly architected VM servers with complete redundancy between them. Either one could be pulled and there'd maybe be a few seconds of data loss at the worst. One day lightning struck the building and even managed to "jump" the grounded pair of UPS's and fried all hard disks in both nodes. Thankfully I had both onsite and offsite backups; but it took a couple…

Yeah, there definitely needs to be an offsite spare for sure.

Re: Ask HN: Who operates at scale without containers?

#332

Post by pclmulqdq »

maccolgan wrote: post #303
senorsmile wrote: post #296

Earlier quoted context omitted.

In 2009 I had a pair of (what I thought at the time) very cleverly architected VM servers with complete redundancy between them. Either one could be pulled and there'd maybe be a few seconds of data loss at the worst. One day lightning struck the building and even managed to "jump" the grounded pair of UPS's and fried all hard disks in both nodes. Thankfully I had both onsite and offsite backups; but it took a couple…

I think it's always better to have the mirror be located in a separate datacenter and then connect both via a SDN.

Why not both?

Re: Ask HN: Who operates at scale without containers?

#334

Post by voidfunc »

JohnBooty wrote: post #283
rootusrootus wrote: post #69

Earlier quoted context omitted.

> But I'd so often dive in to understand the details and be disillusioned about how much complexity there is for relatively little benefit. I feel similarly about many cloud services in e.g. AWS. I get where it's sometimes handy, but the management overhead can get pretty insane. I used to hear people say "just do X in AWS and ta-da you're done!" only to find out that in many cases it isn't actually a net improvement…

Yeah, I feel this way a lot. In recent years I've done work for companies where it feels like the end result of a lot of very expensive AWS infra is... more expensive, and not really superior to things we had ten years ago when hosting on-prem. There's a level of scale for which distributed AWS/Azure is necessary, but I'm not sure how many shops need that. The devops guy at a previous gig thought I was some kind of n…

The default setup trap thing resonates. Also a lot of DevOps folks I have met are former sysadmins with unflexing mindsets about the way things should be done. It all just culminates as cargo cult “best practices” that bloat your infra.

Re: Ask HN: Who operates at scale without containers?

#335

Post by annoyingnoob »

kuon wrote: post #55

We use ansible on bare metal (no VM) to manage about 200 servers in our basement. We use PXE booting to manage the images. We use a customized arch linux image and we have a few scripts to select what feature we'd like. It's "old school" but it's been working fine for nearly 20 years (we used plain scripts before ansible, so we always used the "agentless" approach). Our networking stack uses OpenBSD.

I've used pretty much the same model to easily manage hundreds of servers. PXE boot, install an image, run ansible scripts to configure the specifics.

Re: Ask HN: Who operates at scale without containers?

#337

Post by mustache_kimono »

disintegore wrote: post #62
smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

I asked the question without context because I didn't want to fire the thread off in the wrong direction but I suppose in a comment chain it's fine. For what it's worth we do use containers and K8s heavily at my current job. I know that there are quite a few people opposed to the state of containers and the technologies revolving around them. I don't think the arguments they present are bad. [Attacking the premise](…

> Attacking the premise isn't particularly hard.

The linked Drew Devault seems to have a glib response for quite a few things. His devs shouldn't build software for users take is just as infuriating. You just want to say "Yeah, I'd rather not, but unfortunately it kinda goes along with writing software. Send 1000 happy users to me to build my software on every platform, because I'd be glad to let them. Otherwise, what is your alternative?"

Re: Ask HN: Who operates at scale without containers?

#338

Post by anonymoushn »

dknecht wrote: post #309
cyberge99 wrote: post #11

I believe tools like nomad and consul shine here. Using nomad as a job scheduler and deployer allows you to use various modules for jobs: java, shell, ec2, apps (and containers). I use it in my homelab and it’s great. That said, I don’t use it professionally. I think Cloudflare is running this stack alongside firecracker for some amazing edge stuff.

That is correct. We run baremetal and all process run on all boxes. The one exception is we put our remote browsing product in gvisor which does use containers.

there was a push to have non-edge services run in k8s. Was that reversed?

Re: Ask HN: Who operates at scale without containers?

#339

Post by stickyricky »

maxk42 wrote: post #26

Back in 2010 I built and operated MySpace' analytics system on 14 EC2 instances. Handled 30 billion writes per day. Later I was involved in ESPN's streaming service which handled several million concurrent connections with VMs but no containers. More recently I ran an Alexa top 2k website (45 million visitors per month) off of a single container-free EC2 insurance. Then I spent two years working for a streaming compa…

> Handled 30 billion writes per day.

Writes to what?

Re: Ask HN: Who operates at scale without containers?

#340

Post by hinkley »

p10_user wrote: post #222
hinkley wrote: post #141

Earlier quoted context omitted.

The people who are invested in their home-grown solutions are usually pretty bad at doing tech support for them. They enjoy writing code, and it doesn't take long before you've created enough surface area that you couldn't possibly keep up with requests even if you wanted to. Which they don't, because nothing has convinced me of the crappiness of some of my code more thoroughly than watching other people try and fail…

Nice post. I wish I was reading this 6 years ago, but at least I quickly learned to be humble about the quality of my code.

I'm not quite sure how to interpret what you've said exactly.

Some of what I said is a performance piece. If I insist that I am human and thus make mistakes, then it makes space both for others to make mistakes and for us to have a frank conversation of how, in our more lucid moments, we can do things that fight/counteract our basic nature.

It's morally equivalent to putting a plastic cover over the Big Red Button so that pushing it becomes a conscious and considered act. Deciding to push the button is step 1, moving the cover is step 2, and step 1.5 or 2.5 is realizing this is in fact not what you want to do after all, instead of a momentary bit of inattention, bad judgement, or the Imp of the Perverse.. It's not that I won't make mistakes, but that (hopefully) I a make them less often. But making them less often, I'm entrusted with bigger potential for mistakes, and so my mistakes tend to cause more harm. In a Precautionary Principle sense, we end up roughly within the same order of magnitude. When I make 5x fewer mistakes I'm allowed to be responsible for problems that are twice as risky.

If you want candor about mistakes you have to start by giving people permission to make them (or at least, to make new ones).

Post reply on HN