201–210 of 401 posts
Earlier quoted context omitted.
https://woem.men/notes/9r86xd69cu89052m Also shoutout to Cloudflare for showing off what a diverse company they are in this one /s
https://woem.men/notes/9r5bwnci8x2204it “Actually, they’re 1000 years old”
Whenever I ask a CTO if they have a backup (or plan-B) they say we're on AWS, we backup there and they will never go down as a company. And then I ask them what they do when their account gets shut (e.g. because they are selling something bad on Amazon and have the same phone number as the company account?) Or the instance some years ago where GCP closed because someone had wrongly classified image on their drive? Yo…
Another great question is "When did you last try to restore from a backup?" which usually is answered with "It's the built-in tooling, why would we assume it's broken?" or similar. Then fast-forward some months/years, and they try to restore from backups only to realize the backups never actually backed up what they cared about.
As with code deployment, it's not so scary when it's something you do so frequently that it's just a little script you run.
Hetzner froze my account because I owed them 0.02 EURO's. It was not possible to pay it with a VISA credit or VISA debit, nor Amex card. They required me to wire transfer the money. However my bank does not allow the wiring of a 0.02 EURO amount, as the amount is too low. Out of pure spite I built my own data center.
How did you build your own data center? Is it in your house, or renting a place somewhere? How much did it cost? I would be curious about any details you can share.
Of course it wouldn't work for all cases but I find it beats having a vps somewhere that can be taken down for no reason at all.
Whenever I ask a CTO if they have a backup (or plan-B) they say we're on AWS, we backup there and they will never go down as a company. And then I ask them what they do when their account gets shut (e.g. because they are selling something bad on Amazon and have the same phone number as the company account?) Or the instance some years ago where GCP closed because someone had wrongly classified image on their drive? Yo…
Earlier quoted context omitted.
They've gone the route of multiple AWS accounts in my company to avoid the issue they introduced with horrible planning. First they wanted us out of on-premise, and told us costs wouldn't matter. Then they wanted us to be 'cloud agnostic', but when given deadlines changed to 'get it working in AWS ASAP, doesn't matter the tech debt' Now they're freaking out about AWS costs, and we're back to juggling 'cloud agnostic'…
I would wager it’s not uncommon. But also, the execs are the ones making the business risk decisions. Just make sure they have the correct info to make those decisions, the. Your responsibility is done.
Not the first time this is happening: - Ask HN: Hetzner banned me with no explanation. What can I do? (https://news.ycombinator.com/item?id=32318524) - Hetzner didn't even provide a detailed info on why they deactivated my account (https://news.ycombinator.com/item?id=40781617)
> Dear Mr David Allison
> After reviewing your updated customer information, we have decided to deactivate your account because of some concerns we have regarding this information. Therefore, we have cancelled all your existing products and orders with us.
> Best regards
> Your Hetzner Online Team
Earlier quoted context omitted.
I agree about data backups but replicating your setup in another cloud provider is: 1) Expensive 2) Not straightforward, e.g. is there a 1:1 setup in another cloud for your system? 3) Likely to go untested and be useless when you need it most
The best part of cloud providers is that short-term VMs are relatively cheap to deploy. You don't need a full active-active failover setup, you just need to design your infrastructure in a cloud-agnostic way and test the deployment scripts a few times a year. The most expensive part is going to be maintaining an up-to-date offsite data backup. Running a few VMs for a handful of hours is basically free.
But that's one helluva "just", and also means that you can't use the platform-specific features that make life easier. In practice that's probably way more expensive than spinning some testing VPSes up and down.
Earlier quoted context omitted.
Agreed, if you haven't tested your backups recently (daily, automatic best), you don't have backups. Several of my clients (CTO Coaching) had problems in the past because they restored backups and where finding they were not complete (for various reasons).
daily test restore is infeasible for anything but toy projects. You should periodically test your restore procedures, but its incredibly costly and time consuming for sizeable platforms. Its just not that easy to restore a 10+TB backup for example, and thats a _tiny_ backup size for a b2c product. they can easily go into the hundreds of TB, depending on your platform. and i might add: i vividly remember gitlabs artic…
If your restores are too time consuming to test regularly, they sure as shit aren't going to be useful in a disaster.