Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes on Hetzner: cutting my infra bill by 75%

bilbof.com

Post reply on HN
Print view View original

181–190 of 229 posts

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#181

Post by supriyo-biswas »

tuukkah wrote: post #179
supriyo-biswas wrote: post #175

Earlier quoted context omitted.

Everyone claims it’s a spelling and grammar check, but it’s the OP trying to spread “we tried running self-managed clusters on Hetzner and it only saved us 20% while being a chore in terms of upkeep” into a full essay that causes all that annoying filler. You’d assume people would use tools to deliver a better and well composed message; whereas most people try to use LLMs to decompress their text into an inefficient…

You do realize it wasn't "saved us 20%" but "Hetzner can sometimes be as low as 20% of an equivalent AWS bill" ie saved 80%?

Ah, apologies.

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#182

Post by 0xbadcafebee »

I used to do my own car maintenance, because I wanted to save money, and it was fun. It turned out it was more complex than I thought, things slowly fell apart or broke. I spent a good deal of time "re-fixing" things. Spent probably thousands on tools over the years, partly replacing the cheap stuff that broke or rusted quickly. My cars were often up on blocks. But I learned a lot of great lessons. The biggest one? Some things are not worth DIYing; pay a mechanic or lease your car, especially if you depend on it for your livelihood.

Even something as simple as an oil change, really isn't worth doing yourself. First you buy the tools (oil drip pan, filter wrench, funnel, creeper). Then you set aside the time to use them, find your dingy work clothes. You go to the store and buy new oil and a filter. You go home and change the oil. Then that day or another day you go to a store that will take your used oil. Versus 20 minutes at an auto mechanic, for about $15 more than the cost of the oil and filter.

Kubernetes is an entire car (and a complex one). It's really not worth doing the maintenance yourself, I promise you. Unless you're just doing it for fun.

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#184

Post by mfld »

KaiserPro wrote: post #165
tutfbhuf wrote: post #56

I have experience running Kubernetes clusters on Hetzner dedicated servers, as well as working with a range of fully or highly managed services like Aurora, S3, and ECS Fargate. From my experience, the cloud bill on Hetzner can sometimes be as low as 20% of an equivalent AWS bill. However, this cost advantage comes with significant trade-offs. On Kubernetes with Hetzner, we managed a Ceph cluster using NVMe storage,…

Ceph is a bastard to run. Its expensive, slow and just not really ready. Yes I know people use it, but compared to a fully grown up system (ie lustre[don't its raid 0 in prod] or GPFS [great but expensive]) its just a massive time sync. You are much better off having a bunch of smaller file systems exported over NFS make sure that you have block level replication. Single address space filesystems are ok and convenien…

I saw that Hetzner is beta testing ceph-based object storage. This could make the setup much easier. Anyone tested this already?

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#185

Post by bc569a80a344f9c »

adamcharnock wrote: post #171
bambambazooka wrote: post #167

Earlier quoted context omitted.

> 5) If you want network redundancy you can create a 1G vSwitch (VLAN) on the 1G ports for internal use. Give each server a loopback IP, then use BGP to distribute routes (bird). Are you willing to share example config for that part?

I don't have one I can share publicly, but if you send me an email I'll see what I can do :-) Email is in my profile. You'll need a bit of baseline networking knowledge.

Should note that if you don't have enough networking knowledge, this is an excellent way to build a gun to shoot yourself in the foot with. If you misconfigure BGP or don't take basic precautions such as sanity filters on in- and outbound routes, you can easily do something silly like overwrite each server's default route, taking down all your services.

It's not rocket science, but it is complex, and building something complex you don't fully understand for production services can be a very bad idea.

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#186

Post by DonHopkins »

theshrike79 wrote: post #176
simtel20 wrote: post #173

Earlier quoted context omitted.

Why would you want to restrict contributions from people with relevant experience and willingness to share, just because the author ran a spelling and grammar check?

Unless the spelling and grammar is HORRENDOUS people won't really care. Bad English is the words most used language, we all deal with it every day. Just using your browser's built-in proofreader is enough in 99.9% of the cases. Using ChatGPT to rewrite your ideas will make them feel formulaic (LLMs have a style and people exposed to them will spot it instantly, like a code smell) and usually needlessly verbose.

You can tell it's AI when it refuses to take a side and equivocally considers issues first on one hand and then the other hand, but can't get the number of fingers right.

Or as ChatGPT would put it:

Precise grammar and spelling are undeniably important, but minor imperfections in English rarely obstruct communication. As the most widely used language in the world, English is highly flexible, and most people navigate small errors without issue. For the majority of cases, a browser’s built-in proofreader is entirely sufficient.

On one hand, tools like ChatGPT can be valuable for refining text and ensuring clarity. On the other hand, frequent reliance on such tools can result in writing that feels formulaic, especially to those familiar with AI-generated styles. Balancing the benefits of polished phrasing with the authenticity of your own voice is often the most effective approach.

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#187

Post by 0xbadcafebee »

acac10 wrote: post #159

// Taking another slant at the discussion: Why kubernetes? Thank you for sharing your experience. I also have my 3 personal servers with Hetzner, plus a couple VM instances in Scaleways (French outfit). Disclaimer: I’m a Googler, was SRE for ~10 years for GMail, identity, social, apps (gsuites nowadays) and more, managed hundreds of jobs in Borg, one of the 3 founders of the current dev+devops internal platform (and…

Same reason they'll make 10 different microservices for a single product that isn't even 5K LoC. People chase trends because they don't know any better. K8s is a really big trend.

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#188

Post by p_l »

acac10 wrote: post #159

// Taking another slant at the discussion: Why kubernetes? Thank you for sharing your experience. I also have my 3 personal servers with Hetzner, plus a couple VM instances in Scaleways (French outfit). Disclaimer: I’m a Googler, was SRE for ~10 years for GMail, identity, social, apps (gsuites nowadays) and more, managed hundreds of jobs in Borg, one of the 3 founders of the current dev+devops internal platform (and…

I'll put it this way:

Every time I try to use Nomad, or any of the other "simpler" solutions, I hit a wall - there turns out to be a critical feature that is not available, and which if I want to retrofit into them, will be a hacky one-off that is badly integrated into API.

Additionally, I don't get US-style budgets or wages - this means that cloud prices which target such budgets are horrifyingly expensive to me, to the point that kubernetes pays itself off at the scale of single server

Yes, single server. The more I make it fit the proper kubernetes mold, the cheaper it gets, even. If I need to extend something, the CustomResourceDefinition system makes it easy to use a sensible common API.

Was there a cost to learning it? Yes, but honestly not so bad. And with things like k3s deploying small clusters on bare metal became trivial.

And I can easily wrap kubernetes API into something simpler for developers to use - create paved paths that reduce the amount of what they have to know, provide, and that will enforce certain deployment standards. At lowest cost I have encountered in my life, funnily enough.

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#189

Post by p_l »

0xbadcafebee wrote: post #182

I used to do my own car maintenance, because I wanted to save money, and it was fun. It turned out it was more complex than I thought, things slowly fell apart or broke. I spent a good deal of time "re-fixing" things. Spent probably thousands on tools over the years, partly replacing the cheap stuff that broke or rusted quickly. My cars were often up on blocks. But I learned a lot of great lessons. The biggest one? S…

Unless you find out that calling AA for everything and going to big name mechanics garage every time costs you significant portion of your budget.

A lot of it is finding balance between what to do yourself, what to outsource, and it's not as easy or clean as some people here like to claim.

Re: Kubernetes on Hetzner: cutting my infra bill by 75%

#190

Post by p_l »

mountainriver wrote: post #80
tutfbhuf wrote: post #56

I have experience running Kubernetes clusters on Hetzner dedicated servers, as well as working with a range of fully or highly managed services like Aurora, S3, and ECS Fargate. From my experience, the cloud bill on Hetzner can sometimes be as low as 20% of an equivalent AWS bill. However, this cost advantage comes with significant trade-offs. On Kubernetes with Hetzner, we managed a Ceph cluster using NVMe storage,…

Manually updating k8s clusters is a huge tradeoff. I can’t imagine doing that to save a couple bucks unless I was desperate

The "couple bucks" in my experience were difference between viable business and bankrupt one - including time spent on maintaining k8s!
Post reply on HN