Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who operates at scale without containers?

news.ycombinator.com

Post reply on HN
Print view View original

291–300 of 446 posts

Re: Ask HN: Who operates at scale without containers?

#291

Post by geoduck14 »

smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

Well, I was about to ask if you were looking for a job - but then I saw you were the founder and CTO. So... are you hiring?

Re: Ask HN: Who operates at scale without containers?

#292

Post by KerrAvon »

nhoughto wrote: post #270
cyberge99 wrote: post #11

I believe tools like nomad and consul shine here. Using nomad as a job scheduler and deployer allows you to use various modules for jobs: java, shell, ec2, apps (and containers). I use it in my homelab and it’s great. That said, I don’t use it professionally. I think Cloudflare is running this stack alongside firecracker for some amazing edge stuff.

Recent convo with Cloudflare they said they are mostly container-less, debian packages installed and running on the the host, which is impressive at their scale and complexity. They did seem to think that this approach was hitting its limits and what the future might look like when I spoke to ppl there recently (end 2021), assumption being the future could be containers.

Scaling limits or technical ones? If scaling limits, should still be pretty solid for homelab usage.

Re: Ask HN: Who operates at scale without containers?

#293

Post by po_ta_toes »

tptacek wrote: post #212

Ironically, here at Fly.io, we run containers (in single-use VMs) for our customers, but none of our own infrastructure is containerized --- though some of our customer-facing stuff, like the API server, is. We have a big fleet of machines, mostly in two roles (smaller traffic-routing "edge" hosts that don't run customer VMs, and chonky "worker" hosts that do). All these hosts run `fly-proxy`, a Rust CDN-style proxy…

Hi there,

Very interesting read!

I work for a large news org, The team I'm in primarily uses elixir, which I know the people at fly.io love too!

Why did you decide not to containerise your own infrastructure?

We use some 'chonky' ec2s but are thinking about using containers.

Given that the BEAM has quite a large footprint, do you think it still a good candidate for containers, or would that introduces too much overhead?

Re: Ask HN: Who operates at scale without containers?

#294

Post by api »

tharne wrote: post #89
smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

> What works for us is to do the simplest thing that works, then iterate. The older I get, the more often I'm reminded that this un-sexy approach is really the best way to go. When I was younger, I always thought the old guys pushing boring solutions just didn't want to learn new things. Now I'm starting to realize that after several decades of experience, they simply got burned enough times to learn a thing or two h…

https://www.ariel.com.au/jokes/The_Evolution_of_a_Programmer...

Re: Ask HN: Who operates at scale without containers?

#295

Post by VectorLock »

popotamonga wrote: post #33

Containers yes but nothing else. Running 1B valuation with manually going to instances and docker pull xxx && docker-compose down && docker-compose up -d. EC2 created by hand. No issues.

"1B valuation" could be 1 server or 10,000.

Re: Ask HN: Who operates at scale without containers?

#296

Post by senorsmile »

JohnBooty wrote: post #282
anonymousDan wrote: post #133

Earlier quoted context omitted.

Availability?

Devops is not my area of expertise to put it mildly, but "a single big-ass server" scenarios are often pretty well covered by "a second big-ass server configured as a mirror/hot-spare", right? Depends on how many nines of uptime you need, of course, and other such things. (Obviously, there are reams of cases where this isn't sufficient)

In 2009 I had a pair of (what I thought at the time) very cleverly architected VM servers with complete redundancy between them. Either one could be pulled and there'd maybe be a few seconds of data loss at the worst. One day lightning struck the building and even managed to "jump" the grounded pair of UPS's and fried all hard disks in both nodes.

Thankfully I had both onsite and offsite backups; but it took a couple of weeks to order the new equipment, install it and set everything up again.

Now I "devops" all the things and always have the ability to spin up all the services and restore from backups within minutes. This is the new standard in my opinion.

Re: Ask HN: Who operates at scale without containers?

#297

Post by senorsmile »

zthrowaway wrote: post #138
brimble wrote: post #44

Earlier quoted context omitted.

> That's the easy part! The hard part is managing databases. Ding ding ding ding ding. "But what about disk?" (so, relatedly, databases) is the hard part. Balancing performance (network disks suuuuuck) and flexibility ("just copy the disk image to another machine" is fine when it's a few GB—less useful when it's lots of GB and you might need to migrate it to another city and also you'd rather not have much downtime)…

People still think stateful things are impossible on k8s but Stateful sets and persistent volumes solves a lot of this. You should be relying on out of the box DB replication to make sure data is available in multiple areas. This is no different on other platforms.

Kelsey Hightower just had a good interview where he disagrees with you: https://changelog.com/shipit/44

Edit: somewhat* disagrees with you. It's a good listen.

Re: Ask HN: Who operates at scale without containers?

#298

Post by asiachick »

tharne wrote: post #89
smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

> What works for us is to do the simplest thing that works, then iterate. The older I get, the more often I'm reminded that this un-sexy approach is really the best way to go. When I was younger, I always thought the old guys pushing boring solutions just didn't want to learn new things. Now I'm starting to realize that after several decades of experience, they simply got burned enough times to learn a thing or two h…

Hmmm, me, I want batteries included. Why I don't I have continuous deployment? Because that would require work doing something I don't have time to do. Why don't I have a staging server? Because more work. Why don't I have backups. Again, more work. Why don't I have metrics so I can see where my bottlenecks are? More work. Why don't I have logging? More work. Why don't I have a way to do updates without taking the servers down? No time, more work, too complicated.

Sure I can write a server in 5 lines of code but adding all that stuff is many many many person years of work and plus I'd have to go through all the learning pains of doing it wrong, crashing things, losing data, before I'd add it correctly.

I want some existing solution that domain experts already created that do things right.

It has nothing to do with "sexy/unsexy". It's unsexy to use UnrealEngine. It's sexy to write your own. But unsexy ships while sexy is spending time recreating stuff that already exists, unsexy is moving on to the actual content and shipping.

Re: Ask HN: Who operates at scale without containers?

#299

Post by jka »

smilliken wrote: post #225
jka wrote: post #218

Earlier quoted context omitted.

Do your developers run the same Nix packages that you deploy to production? (that sounds like a energy-level-transition in developer productivity and debugging capability, if so)

Yeah, the environment is bit-for-bit identical in dev and prod. Any difference is an opportunity for bugs. OK, there's one concession, there's an env var that indicates if it's a dev and prod environment. We try to use it sparingly. Useful for stuff like not reporting exceptions that originate in a dev environment. Basically, there's a default.nix file in the repo, and you run nix-shell and it builds and launches you…

That sounds extremely impressive, thank you. I hope to find some time to try out Nix soon (appending it to the list of technologies to learn...).

Re: Ask HN: Who operates at scale without containers?

#300

Post by throwbigdata »

JohnBooty wrote: post #283
rootusrootus wrote: post #69

Earlier quoted context omitted.

> But I'd so often dive in to understand the details and be disillusioned about how much complexity there is for relatively little benefit. I feel similarly about many cloud services in e.g. AWS. I get where it's sometimes handy, but the management overhead can get pretty insane. I used to hear people say "just do X in AWS and ta-da you're done!" only to find out that in many cases it isn't actually a net improvement…

Yeah, I feel this way a lot. In recent years I've done work for companies where it feels like the end result of a lot of very expensive AWS infra is... more expensive, and not really superior to things we had ten years ago when hosting on-prem. There's a level of scale for which distributed AWS/Azure is necessary, but I'm not sure how many shops need that. The devops guy at a previous gig thought I was some kind of n…

You are correct
Post reply on HN