Live data from Hacker News

Kubernetes on Hetzner: cutting my infra bill by 75%

bilbof.com

201–210 of 229 posts

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#201
post #147
post #136

Earlier quoted context omitted.

> our country shouldn't care about emissions when China is the biggest emitter To be fair, until China does something about their emissions, the rest of us are just pissing in the ocean.

Eh, per capita China has lower emissions than the US whilst manufacturing and exporting significantly more. Everything is intertwined and tightly coupled, such simple statements are rarely accurate.

US need to sort their shit out as well.

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#202
post #200
post #150

Earlier quoted context omitted.

In my experience noone bothers unless they are using GPUs or they are already at 100k/mo. I do think 100k/mo is the tipping point actually, that is $1.2M/yr. It costs around $400k/yr in engineering salaries to reasonably support a sophisticated bare metal deployment (though such people can generally do that AND provide a lot of value elsewhere in the business, so really it's actual cost is lower than this) and about…

If you want to do HA kubernetes, you need oncalls and at least 10 engineers to get a stable rotation. If you do that in Europe you have to pay them during standby hours. 400k/year seems very low to me.

You really don't need all 10 people on-call to know k8s to that level. They just need to know enough as to when to wake someone else up.

Everywhere I have worked where we have run clusters in the 100s to 1000s of nodes we have rarely had a team larger than 4-5 of true k8s folks and even then it's been a split between folks that are very hardware provisioning/network/etc focused and more higher level k8s folk which also take on a large portion of CI/CD work also.

At smaller scale (in the $1M/yr ballpark) I have done all the k8s bare metal ops myself along with all CI/CD and been responsible for a ton of the backend programming too. This is feasible because with distros like Talos etc it doesn't take a lot of manpower once it's setup and upgrades aren't too painful at small scale if you aren't running stateful services.

So tbh no, you just need ideally 2 folks at around ~200k/yr each that are competent and have done it before. The rest of the folks on the on-call rotation are just the rest of your engineers (and if you are at $1m/yr cloud spend you have more than 10 of those).

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#203
post #121

Earlier quoted context omitted.

2% of total worldwide electricity use in 2022 is a shit load of electricity and emissions. Your argument is the same as those who argue "our country shouldn't care about emissions when China is the biggest emitter". If you dive into a detailed breakdown of emissions you'll find that it's a complex hierarchy of categories. You can't just fix "all of transport" or treat it like a "low hanging fruit", just look at how m…

> Your argument is the same as those who argue "our country shouldn't care about emissions when China is the biggest emitter". China and the US are in the same order of magnitude in emissions. So NO that's absolutely not the argument I am making. > Any energy use that's measurable in whole percentage points of global emissions needs addressing But it isn't! That's my point. Electricity use is about 20% of total energ…

> Yes so let's focus on that instead of data centers. Data centers are not the problem!

This is like saying I shouldn’t care about pollution from the local auto painting shop because there are strip mines somewhere else. Yes, it’s not the top priority but that doesn’t mean that we shouldn’t be trying to reduce pollution just as we are for every large producer, and with both LLMs and cryptocurrency having potential demand outstripping the existing supply we have every reason to expect continued growth in emissions at a time when we need decline.

Rather than taking this so personally, consider that people on HN talk about it because our choices actually matter here. Very few of us affect heavy industrial policy but all of us can think about how much our applications need to run.

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#204
We at Syself.com also have great experiences with Kubernetes on Hetzner. We built a platform on top of Cluster API and brought a managed Kubernetes experience to Hetzner. Now we have self-healing, automated updates and 100% reproductibility, with full bare metal support.

> Hetzner volumes are, in my experience, too slow for a production database.

That's true, though. To solve that we developed a way to persist the local storage of bare metal servers across reprovisionings. This way it's both faster and cheaper. Now we are adding an automated database deployment layer on top of it.

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#205

I used to do my own car maintenance, because I wanted to save money, and it was fun. It turned out it was more complex than I thought, things slowly fell apart or broke. I spent a good deal of time "re-fixing" things. Spent probably thousands on tools over the years, partly replacing the cheap stuff that broke or rusted quickly. My cars were often up on blocks. But I learned a lot of great lessons. The biggest one? S…

Depends on one: how interested/motivated are you, now and down the line; and two: how likely is your dependency on a third party going to screw you over in the long run.

My opinion, from the viewpoint of a consultant often involved in Kubernetes, is to get initial help and a persistent help line, but get somebody internally interested enough to ride along and learn.

Consultants and experts in general can save you from a lot of bad up-front decisions and banging your head against the wall for months. It's not trivial to learn your way around technologies or ecosystems, including common dark corners and pitfalls, in a reasonable amount of time while also having to focus on your core business. Accept help but learn to fish and to make a fire.

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#206

Earlier quoted context omitted.

Ceph is a bastard to run. Its expensive, slow and just not really ready. Yes I know people use it, but compared to a fully grown up system (ie lustre[don't its raid 0 in prod] or GPFS [great but expensive]) its just a massive time sync. You are much better off having a bunch of smaller file systems exported over NFS make sure that you have block level replication. Single address space filesystems are ok and convenien…

I agree that it is hideously complicated (to anyone saying “just use Rook,” I’ll counter that if you haven’t read through Ceph’s docs in full, you’re deluding yourself that you know how to run it), but given that CERN uses it at massive scale, I think it’s definitely prod-ready.

Oh it probably is prod ready, I just wouldn't use it unless I had to (ie I had the staff to look after it and no money to buy something better)

whether is a good fit for general purpose storage of stuff at a small scale is harder question. Its not easy to get good performance at small scale, and to get good performance requires a larger than you'd like number of storage nodes.

Yes it has inline FEC, (https://www.ibm.com/docs/en/storage-ceph/7?topic=components-...) but its lots of layers to get to a file system.

Personally I'd have a redundant array of storage nodes and be done with it. Its easier to debug a single server than 3 layers of ceph weirdness.

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#207
post #146

Earlier quoted context omitted.

> As a consultancy company, you want to sell that. As a customer, I don't see how that's worth it at all. Well I do rather agree, but as a consultancy I'm biased. But let's do some math. Say it's 4 months (because who has uninterrupted time), a senior rate of $1000/day. 20 days a month, so 80 days, is an $80k outlay. That's assuming you can get the skills (because AWS et al like to hire these kinds of engineers). Say…

this assumes there are no devops/consulting cost to setup something with AWS. My experience is that "the aws way of doing XYZ" is almost as complicated as doing it the non-AWS-way. On top of that: the non-AWS-way is much more portable across hosting providers, so you decrease your business risks considerably.

Well, it at least assumes the cost to setup the AWS way is sunken. What is a given for anybody that may hire them.

But if you are starting from scratch instead of looking for someone to help you migrate, then yeah, the AWS way has probably higher setup costs than making it portable.

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#208

Earlier quoted context omitted.

> I believe that Hetzner data centers in Europe (Germany, Finland) are powered by green energy, but not the locations in US. Green lignite.

While fans of nuclear energy like to meme about the German power grid, Hetzner is — in so far as anyone with a grid connection can be — powered by 100% green wind and hydro energy. You can see the paperwork here: - https://cdn.hetzner.com/assets/Uploads/oekostrom-zertifikat-... - https://cdn.hetzner.com/assets/Oomi-sertifikaatti-tuuli+vesi...

This does not apply to Hetzner US data centers, as far as I know. That's just for Germany and Finland.

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#209
post #151
post #136

Earlier quoted context omitted.

> our country shouldn't care about emissions when China is the biggest emitter To be fair, until China does something about their emissions, the rest of us are just pissing in the ocean.

China is actively working on reducing their emissions (they're building tons of nuclear and renewables, and have long term plans for both), and a lot of theirs are to manufacture stuff the whole world uses.

> manufacture stuff the whole world uses

I would argue that the world wouldnt use as much stuff if China stopped manufacturing it

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#210
post #196

Earlier quoted context omitted.

We all start somewhere. :) Hetzner can be a good fit for many small companies. But let’s also be honest, if you’re THAT bootstrapped, you probably have no business running kubernetes to begin with. If the company has a short runway, it doesn’t make sense to work on a complex architecture from the start. Focus on shipping something and getting revenue.

Depends entirely on what is your starting skillset. K8s is my tool of choice when I am that boostrapped, because a single server with k3s thrown on it will cost me maybe 80 EUR a month and hold all environments plus CI/CD plus various self-hosted business components (with backup sent over to separate provider, just in case), and I'll be free to build my project instead of messing with server setup or worry about clou…

that’s a good point. +1
Post reply on HN