Live data from Hacker News

Our production servers were suspended by Google Cloud

onvoard.com

111–120 of 189 posts

Re: Our production servers were suspended by Google Cloud

#111

I hate how the internet age has conditioned customers to expect less. There is no way a non-internet business would cut off a client of 4-years without actually talking (phone) to them.

It's all relative to how much you spend and how much other customers spend.

The first rule of production on GCP is to get off credit card billing, and that has been pretty well-known for a while.

Re: Our production servers were suspended by Google Cloud

#113

I am not a fan of "The Cloud" for ongoing production. I understand the case for it but I never liked the risk of someone else, with a different agenda and incentives than me, running my critical infrastructure. The Cloud does alleviate some risks with hosting your own servers, but it introduces new risks. This reminds me of a slogan I saw many years ago: There is no cloud, it's just somebody else's computer. That som…

I sort of tend to agree with you, but the counter-point is that there is always going to be some critical piece of the infrastructure that you are dependent upon that someone else controls. If you colocate, it's the data centre. If you host on-prem, it's the physical telco infrastructure. There are no shortages of incompetent people and institutions that will screw you. It's typically incompetence, not malice. But yo…

I've done large colo deployments. When you're provisioning your own hardware it's cloud-init, ansible, K8s, whatever. Nothing even knows you're in colos operated by different providers in different regions. Nothing knows or cares you're using five different upstream internet providers (whether your own - complicated or IP blends from the colo).

Have a falling out or big issue? No reputable colo provider is going to hold your hardware hostage (it's the physical world and legal and reputational issues heat up quickly). They'll be happy to ship it out (get rid of you) to another provider, who will be even happier to re-rack. Meanwhile if your architecture is even remotely well designed (load balancer, multiple facilities) you're never down and it's not even that much of a "fire drill". With the capabilities of modern hardware this is often a single 1U redundant HW machine.

All of this, by the way, is a fraction of the cost of cloud solution (a $25k bonkers 1U is ~$700 a month with a three year lease and 1U hosting + bandwidth + power is ~$200 a month). So for $2k a month (and it's actually predictable) you're up in two facilities and have capabilities an order of magnitude greater than a big cloud at equivalent spend.

Companies screaming on twitter and profusely apologizing to their customers due to the most recent "name a cloud" outage demonstrate that multi-cloud (and even multi-region) is extremely difficult if not impossible to do in practice.

Re: Our production servers were suspended by Google Cloud

#114

Please move to Hetzner. If you miss payments even they might pull the plug, but at least they have humans who respond to emails and can get you back online quickly once you pay.

As much as I love Hetzner and can recommend their legal department, they are very risk averse especially with new customers and might not even let you open an account.

Any sort of weirdness from your server will also result in a very harsh email and a threat to suspend your services in way less than 10 days. I had the pleasure to experience their grumpy network admins when IPFS decided that node discovery included dialing 900+ RFC1918 IPs.

Re: Our production servers were suspended by Google Cloud

#115

A nice big Threadripper in the corner of your office is all you need, I can almost certainly guarantee it. There are very very few businesses with less than 1000 employees that can't be run on one modern high end server.

What happens when the power goes out? Internet goes out? Someone spills a cup of coffee on the server? Landlord needs to do some work and drywall dust flies around everywhere? It's not that most CRUD apps could run on a single machine, it's that they probably shouldn't. Customers want redundancy, automated failover, etc. Datacenters are much better at offering these things than your office park.

Someone spills a cup of coffee on the server?

Are people spilling coffee on your servers at work?

Re: Our production servers were suspended by Google Cloud

#116
post #65

Earlier quoted context omitted.

I disagree. I’ve hosted on AWS for ages. They don’t do this. We didn’t pay the bill for two months once because the owner of the card left. No problems. This was a small spender outfit. They could have thrown us on the street as we were a rounding error. Now we’re a big spender and we still forgot to pay them sometimes (sorry Bezos) But some of your point is worth considering. Design for portability should the worst…

How did Russian companies mitigate the effects of cloud services hosting being blocked by sanctions

Mostly by migrating to other countries.

Re: Our production servers were suspended by Google Cloud

#118

Make sure your decision to migrate to AWS is not an emotional one. I had my own feud with AWS over a $60,000 bill and an enormous amount of hours spent by my team fighting crypto hackers after they took over one of our accounts and use very advanced scripts to create hundreds of large instances. AWS has become very complicated to manage and at the same time the attacks to their infrastructure have increased exponenti…

I find the author's stance of "let's move to the competition" hilarious for this reason.

I was trying some VPN services at the time, and logged in to an AWS account with the VPN turned on, after which AWS suspended resources in the account. This took a two days to review and fix, though the only saving grace was that I did not have a paid offering running there.

When pressed as to why they'd flag my account, they'd would evade the question as to why my account was suspended, though I'm pretty sure it's the VPN. (And before you ask, no, my account was not compromised, as I had to review 90 days' Cloudtrail logs before AWS would budge on the issue).

Re: Our production servers were suspended by Google Cloud

#119
post #8

"I missed an email that had important in the title because I didn't think it was important, now my servers are suspended and it's all Google's fault". Makes sense, I'm sure you'll fare better with Amazon.

It's all about expectations bound to your scale. I worked at a place that spend 7 digits with AWS a year. We missed invoices a lot, mostly because the person who could sign them off was a single point of failure. We never got suspended though, just confused emails from our account manager.

Re: Our production servers were suspended by Google Cloud

#120

Reminder: if your level of interaction with a service is putting your credit card in, the service is likely to interact with you at the level of automated emails, and also likely to trust you as much as someone who filled out a web form. If it's business critical, talk to a human. All the cloud providers have account managers just waiting to talk to you. If this sort of thing happens and you have an account manager,…

How do you get such a contract at the big providers?

I work in gcp as account manager for enterprise, you won’t believe I chase deal for as low as 2k per month, tough time, every dollar counts
Post reply on HN