Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

"Hetzner decided to cancel our account and terminate all servers"

mastodon.social

Post reply on HN
Print view View original

341–350 of 401 posts

Re: "Hetzner decided to cancel our account and terminate all servers"

#341

Post by attendant3446 »

Tainnor wrote: post #175
Cumpiler69 wrote: post #102

Earlier quoted context omitted.

>Add that to the fact that any clause is worthless if it includes something a reasonable person wouldn't expect. The problem is you always need to sue to get justice for that which means paying for lawyers and consuming time and money plus stress.

That's true in probably every jurisdiction, though? At least in Germany you can often get free legal advice for many things (Verbraucherschutz, Mietrechtsberatung etc.) and there's insurance you can buy that covers your legal fees in case you lose. And legal fees in Germany are typically not exorbitant. (Also in some cases, it's the other way around. If your landlord wants to increase the rent it's on them to sue you…

Legal advice and reality in Germany are 2 different things. The truth is that dealing with any kind of legal situation in Germany is a huge headache and all you get in the end is to prove you are right and get what should be yours anyway, without any additional compensation for your trouble. And many companies use this to abuse the system. The landlord can steal a small part of your deposit, you can only sue. But nobody's going to go through this hell for, say, €100, so the landlord gets to keep €100. Of course you can sue, but it will cost you a lot more than 100€ (even with insurance there is usually a deductible of 300€+) and it will take at least a year. And pretty much everything works this way.

Re: "Hetzner decided to cancel our account and terminate all servers"

#342

Post by attendant3446 »

fabian2k wrote: post #104
Cumpiler69 wrote: post #76

Earlier quoted context omitted.

>German companies are awful at customer service. True also from my experience. I've noted several potential reasons why that is from my time in Germany. Government provided customer protection laws are quite lax and disputes tricky to win and don't represent a big enough deterrent for the scammers when they're just a slap on the wrist and therefore part of the cost of doing business. Sure, you can get sued and you lo…

There is a huge amount of protection for renters, a lot of things are simply illegal to put into the rental agreement and are automatically void. I really have no idea what you're talking about here.

Yes, there are laws that nullify certain clauses in tenancy agreements, but enforcing them is another story.

Re: "Hetzner decided to cancel our account and terminate all servers"

#344

Post by apelapan »

antonvs wrote: post #331
hmmm-i-wonder wrote: post #329

Earlier quoted context omitted.

Having gone from managing several thousand physical to virtual/cloud instances, there are certainly major differences and the company has to structure its approach accordingly (IMO). On premise in my opinion needs a dedicated team managing hardware and leverage solutions to provide that as VM's/Containers/etc to teams. Another team focused on OS level security and base image, then your dev teams can effectively focus…

> On premise in my opinion needs a dedicated team managing hardware and leverage solutions to provide that as VM's/Containers/etc to teams. Another team focused on OS level security and base image, then your dev teams can effectively focus on their app and leverage the automated tools provided by the hardware and OS teams. Exactly. At which point, you’re essentially reinventing a cloud, usually not very well. If you…

If you are in the cloud, you are going to need a team that understands cloud networking, storage, deployment, security etc. You will need enough people to maintain support rotations and survive normal churn.

It seems like many people/organizations belived that they would be rid of the whole "operations problem" once they shifted all their workloads from on-prem to cloud. They believed that they paid a full team for running cables and replacing broken fans/hard drives/PSU:s, when that aspect of on-prem is a tiny (but non-zero) amount of work.

Re: "Hetzner decided to cancel our account and terminate all servers"

#345

Post by tetha »

ambicapter wrote: post #142
diggan wrote: post #22

Earlier quoted context omitted.

Another great question is "When did you last try to restore from a backup?" which usually is answered with "It's the built-in tooling, why would we assume it's broken?" or similar. Then fast-forward some months/years, and they try to restore from backups only to realize the backups never actually backed up what they cared about.

One thing I've never figured out is, what is the difference between backups and replication? And, does restoring from backups always mean losing more _recent_ data than replication?

A simple way to remember, I think by Devops Borat: Redundancy/Replication fix hardware problems. Backups fix stupid human problems.

> And, does restoring from backups always mean losing more _recent_ data than replication?

This depends on the archiving technology and what you're archiving.

Our file and object stores take one full backup every day. This means, we could lose up to 24 hours of data changes on these stores if something happens within these 24 hours. If this is acceptable or not depends on the RPO - the recovery point objective, or the "maximum acceptable data loss". However, especially for documents, 24 hours can be acceptable, because users and customers do tend to have files they uploaded to the system around for a few days. Especially if you have a chance to identify the lost documents.

Both on MySQL with the InnoDB driver, as well as on postgres, you can use PITR backup solutions - point in time recovery. With this, pgbackrest or e.g. xtrabackup store a full backup of the database usually once a day at our place, and then keep archiving the WAL / transaction logs of the system. And we, in turn, archive snapshots of these into the longterm archiving once a day.

If we need a restore, we'd first restore a pgbackrest or xtrabackup state from the long term archiving onto a system. And then we can use the PITR recovery mechanisms to restore at a specific point in time.

Technically, we could precisely recover down to the last transaction before the disastrous transaction to minimize data loss. In fact, I've done so one or two times after some database migrations went haywire. That involved scrolling through transaction logs with a viewer to identify when the migration tool starts running, noting down the transaction ID of the transaction tool starting it's check and then restoring to the transaction before. Very cool tbh.

This is important for an RDBMS, because the data in the relational database tends to be much more volatile than the data in a file or object store. With a filestore, users upload a file and then move it to their recycling bin or their "done" folder on the local system and can easily drag it back out tomorrow. With the database, the user spent 30 minutes to an hour writing up some text or a comment and expects it to be saved and sound once they hit "Reply". Losing this kinda data creates a lot more work & effort for our customers, because then they have to figure out what state the data is in and what to redo. This may also cause their business processes to run haywire and... it's not great.

Re: "Hetzner decided to cancel our account and terminate all servers"

#346

Post by rodgerd »

cyberax wrote: post #257
rodgerd wrote: post #243

Earlier quoted context omitted.

I have spent most of my career in newspaper publishing and banking. A newspaper that doesn't publish for a few days might recover. A bank that drops off the Swift network for days isn't a bank any more.

Russia and Iran would like to have a few words with you. Banks regularly close for multiple days for bank holidays. Unscheduled downtime is a somewhat different story, though. Luckily, the traditional SWIFT/banking infrastructure is so negligible these days, my phone can host a classic banking infrastructure for an entire small country.

This is absolutely peak "confidently incorrect"; it's hilarious, but completely expected on this site.

Re: "Hetzner decided to cancel our account and terminate all servers"

#347

Post by PeterZaitsev »

I have no idea if Hetzner actions are justified and surely they can do better to ensure there is better customer communication. The moral of the story though is - if you're using commodity hosting and Open Source software it is relatively easy for you to find another home... assuming you had good offsite backups of course

If you're locked on some proprietary services like with AWS it is much bigger issue.

Re: "Hetzner decided to cancel our account and terminate all servers"

#348

Post by movedx »

hmmm-i-wonder wrote: post #328
movedx wrote: post #249

Earlier quoted context omitted.

The biggest issue I see here is the misguided assumption that Cloud is just automatically and unilaterally better than on-premise or professionally managed, hosted hardware. This isn't true in most cases. There are so many providers, and therefore examples, of physical tin being accessible in under a minute with cost:hardware ratios that blow Cloud out if the sky (pun! ha!) OVH have a server for USD $95/month (with n…

It was during the wave of "Moving costs from capex to opex give C levels more flexibility" movement after the initial 'cloud is better' wave. In retrospect it seems like another of their badly thought out reactions to a situation they caused by short term thinking, in this case the issues caused by trying to reduce headcount on teams supporting legacy and new physical locations while increasing the pace of new locati…

Good summary.

I think this all boils down to a knee-jerk reaction culture that doesn't think about the second or third degree consequences and/pr beyond the next 2-3 years.

Re: "Hetzner decided to cancel our account and terminate all servers"

#349

Post by movedx »

hmmm-i-wonder wrote: post #329
movedx wrote: post #316

Earlier quoted context omitted.

Precisely. Operating systems aren’t hard. They’re so easy and well established it’s crazy not to use them directly, and even though I’m not the world’s biggest Docker fan, Compose is kind of awesome to be honest. Deploying software and maintaining and OS is simple in this day and age.

Having gone from managing several thousand physical to virtual/cloud instances, there are certainly major differences and the company has to structure its approach accordingly (IMO). On premise in my opinion needs a dedicated team managing hardware and leverage solutions to provide that as VM's/Containers/etc to teams. Another team focused on OS level security and base image, then your dev teams can effectively focus…

> On premise in my opinion needs a dedicated team managing hardware and leverage solutions to provide that as VM's/Containers/etc to teams.

You're assuming that "On premise" equates to "inside our building, in racks we've installed, using power and networking we have to manage." You're correct if that's the case for your business, but my argument is based around the idea that you can use _managed_ hosting providers of physical hardware that'll be either next door to you, in the same city, or close to your users (i.e, you're a business in Germany but your customer base is in London, so you host the servers using a London based provider.)

The idea that you have to manage hardware is greatly diminished when you consider the availability of managed providers that are dirt cheap.

Re: "Hetzner decided to cancel our account and terminate all servers"

#350

Post by movedx »

antonvs wrote: post #331
hmmm-i-wonder wrote: post #329

Earlier quoted context omitted.

Having gone from managing several thousand physical to virtual/cloud instances, there are certainly major differences and the company has to structure its approach accordingly (IMO). On premise in my opinion needs a dedicated team managing hardware and leverage solutions to provide that as VM's/Containers/etc to teams. Another team focused on OS level security and base image, then your dev teams can effectively focus…

> On premise in my opinion needs a dedicated team managing hardware and leverage solutions to provide that as VM's/Containers/etc to teams. Another team focused on OS level security and base image, then your dev teams can effectively focus on their app and leverage the automated tools provided by the hardware and OS teams. Exactly. At which point, you’re essentially reinventing a cloud, usually not very well. If you…

I don't believe a lot of this is required.

OS level security? So, "apt update && apt upgrade", then? I mean, what else are you doing, writing patches for the kernel? Checking every line of code that runs? Are you aware of how effective SELinux and systemd containers are? Just a simple firewall at the OS level? Maybe even just using Tailscale (or the open source Headscale) to introduce zero trust access capabilities.

There's a Terraform provider for Proxmox, which is an excellent hypervisor. Making a template takes less than an hour with configuration.

You do need an Ops person for sure, but an entire _team_?

Post reply on HN