Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes on Hetzner: cutting my infra bill by 75%

bilbof.com

Post reply on HN
Print view View original

51–60 of 229 posts

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#51

Post by thwarted »

SoftTalker wrote: post #22
slillibri wrote: post #12

When I worked in web hosting (more than 10 years ago), we would constantly be blackholeing Hetzner IPs due to bad behavior. Same with every other budget/cheap vm provider. For us, it had nothing to do with geo databases, just behavior. You get what you pay for, and all that.

Yep I had the same problem years ago when I tried to use Mailgun's free tier. Not picking on them, I loved the features of their product but the free tier IPs had a horrble reputation and mail just would not get accepted especially by hotmail or yahoo. Any free hosting service will be overwhelmed by spammers and fraudsters. Cheap services the same but less so, and the more expensive they are the less they will be use…

Tragedy of the Commons Ruins Everything Around Me.

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#52

Post by jorams »

pdpi wrote: post #33
mhuffman wrote: post #17

Earlier quoted context omitted.

For dedicated they say this: >All root servers have a dedicated 1 GBit uplink by default and with it unlimited traffic. >Inclusive monthly traffic for servers with 10G uplink is 20TB. There is no bandwidth limitation. We will charge € 1/TB for overusage. So it sounds like it depends. I have used them for (I'm guessing) 20 years and have never had a network problem with them or a surprise charge. Of course I mostly wo…

1 / 8 * 3600 * 24 * 30 = 324000 so that 1GBit/s server could conceivably get 324TB of traffic per month "for free". It obviously won't, but even a tenth of data is more than the data included with the 10G link.

They do have a fair use policy on the 1GBit uplink. I know of one report[1] of someone using over 250TB per month getting an email telling them to reduce their traffic usage.

The 10GBit uplink is something you need to explicitly request, and presumably it is more limited because if you go through the trouble of requesting it, you likely intend to saturate it fairly consistently, and that server's traffic usage is much more likely to be an outlier.

[1]: https://lowendtalk.com/discussion/180504/hetzner-traffic-use...

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#53

Post by karussell »

BillFranklin wrote: post #28
Scotrix wrote: post #24

Very nicely written article. I’m also running a k8s cluster but on bare metal and qemu-kvms for the base load. Wonder why you would chose VMs instead of bare metal if you looking for cost optimisation (additional overhead maybe?), could you share more about this or did I miss it?

Thank you! The cloud servers are sufficiently cheap for us that we could afford the extra flexibility we get from them. Hetzner can move around VMs without us noticing but in contrast they are rebooting a number of metal machines for maintenance now and for the last little while, which would have been disruptive especially during the migration. I might have another look next year at metal but I’m happy with the cloud…

Note, they usually do not reboot or touch your servers. But yes, the current maintenance of their metal routers (rare, like once every 2 years) requires you to juggle a bit with different machines in different datacenters.

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#54

Post by thenaturalist »

ArtTimeInvestor wrote: post #36

Can anybody speak to the pros and cons of Hetzner vs OVH? There ain't many large European cloud companies, and I would like to understand how they differentiate. Ionos is another European one. Currently, it looks like their cloud business is stagnating, though.

I'd say stay clear of Ionos.

Bonkers first experience in the last two weeks.

Graphical "Data center designer", no ability to open multiple tabs, instead always rerouting to the main landing page.

Attached 3 IGWs to a box, all public IPs, GUI shows "no active firewall rules".

IGW 1: 100% packet loss over 1 minute.

IGW 2: 85% packet loss over 1 minute.

IGW3: 95% packet loss over 1 minute.

Turns out "no active Firewall rules" just wasn't the case and explicit whitelisting is absolutely required.

But wait, there's more!

Created a hosted PostgreSQL instance, assigned a private subnet for creation.

SSH into my server, ping the URL of the created Postgres instance: The DB's IP is outside the CIDR range of the assigned subnet and unreachable.

What?

Deleted the instance, created another one, exact same settings. Worked this time around.

Support quality also varies extremely.

Out of 3 encounters, I had a competent person once.

Other two straight out said they have no idea what's going on.

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#55

Post by jgalt212 »

ronsor wrote: post #41
hipadev23 wrote: post #19

Be careful with Hetzner, they null routed my game server on launch day due to false positives from their abuse system, and then took 3 days for their support team to re-enable traffic. By that point I had already moved to a different provider of course.

Reading comments from the past few days makes it seem like dealing with Hetzner is a pain (and as far as I can tell, they aren't really that cheaper than the competitors).

> they aren't really that cheaper than the competitors

This is demonstrably false.

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#56

Post by tutfbhuf »

I have experience running Kubernetes clusters on Hetzner dedicated servers, as well as working with a range of fully or highly managed services like Aurora, S3, and ECS Fargate.

From my experience, the cloud bill on Hetzner can sometimes be as low as 20% of an equivalent AWS bill. However, this cost advantage comes with significant trade-offs.

On Kubernetes with Hetzner, we managed a Ceph cluster using NVMe storage, MariaDB operators, Cilium for networking, and ArgoCD for deploying Helm charts. We had to handle Kubernetes cluster updates ourselves, which included facing a complete cluster failure at one point. We also encountered various bugs in both Kubernetes and Ceph, many of which were documented in GitHub issues and Ceph trackers. The list of tasks to manage and monitor was endless. Depending on the number of workloads and the overall complexity of the environment, maintaining such a setup can quickly become a full-time job for a DevOps team.

In contrast, using AWS or other major cloud providers allows for a more hands-off setup. With managed services, maintenance often requires significantly less effort, reducing the operational burden on your team.

In essence, with AWS, your DevOps workload is reduced by a significant factor, while on Hetzner, your cloud bill is significantly lower.

Determining which option is more cost-effective requires a thorough TCO (Total Cost of Ownership) analysis. While Hetzner may seem cheaper upfront, the additional hours required for DevOps work can offset those savings.

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#57

Post by jonas21 »

This is an interesting writeup, but I feel like it's missing a description of the cluster and the workload that's running on it.

How many nodes are there, how much traffic does it receive, what are the uptime and latency requirements?

And what's the absolute cost savings? Saving 75% of $100K/mo is very different from saving 75% of $100/mo.

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#58

Post by Keyframe »

slillibri wrote: post #12

When I worked in web hosting (more than 10 years ago), we would constantly be blackholeing Hetzner IPs due to bad behavior. Same with every other budget/cheap vm provider. For us, it had nothing to do with geo databases, just behavior. You get what you pay for, and all that.

depending on the prices, maybe a valid strategy would be to have servers at hetzner and then tunnel ingress/egress somewhere more prominent. Maybe adding the network traffic to the calculation still makes financial sense?

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#59

Post by maestrae »

Volundr wrote: post #10

I haven't used it personally, but https://github.com/kube-hetzner/terraform-hcloud-kube-hetzne... looks amazing as a way to setup and manage kubernetes on Hetzner. At the moment I'm on Oracle free tier, but I keep thinking about switching to it to get off... Well Oracle.

i recently read an article about running k8s on the oracle free tier and was looking to try it. i'm curious, are there any specific pain points that are making you think of switching?

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#60

Post by danpalmer »

hipadev23 wrote: post #19

Be careful with Hetzner, they null routed my game server on launch day due to false positives from their abuse system, and then took 3 days for their support team to re-enable traffic. By that point I had already moved to a different provider of course.

Digital Ocean did this to my previous company. They said we’d been the target of a DOS attack (no evidence we could see). They re-enabled the traffic, then did it again the next day, and then again. When we asked them to stop doing that they said we should use Cloudflare to prevent DOS attacks… all the box did was store backups that we transferred over SSH. Nothing that could go behind Cloudflare, no web server running, literally only one port open.
Post reply on HN