I remember when Google was the best thing because they were saving us from MS.
Now they behave like this. I doubt even MS would have done a thing like this back in the day when they were the baddies.
Our production servers were suspended by Google Cloud
81–90 of 189 posts
Re: Our production servers were suspended by Google Cloud
#82"I missed an email that had important in the title because I didn't think it was important, now my servers are suspended and it's all Google's fault". Makes sense, I'm sure you'll fare better with Amazon.
I’d normally agree — if you ignore an important email, the consequences are on you — but the Google verification process is very hostile and Google do constantly send emails about “important” but actually innocuous things which trains users to consider these emails as unimportant. The OP certainly has some culpability but having bad experience of this process myself multiple times, it does feel as if Google designed…
>Google do constantly send emails about “important” but actually innocuous things which trains users to consider these emails as unimportant.
Haha yea. Like the post notes, the Data Processor notifications have gotten so irritating that I had to setup a filter to junk them immediately.
GDPR caused spam being foisted on the rest of us not in the EU.
Re: Our production servers were suspended by Google Cloud
#83"I missed an email that had important in the title because I didn't think it was important, now my servers are suspended and it's all Google's fault". Makes sense, I'm sure you'll fare better with Amazon.
> "I missed an email that had important in the title because I didn't think it was important, now my servers are suspended and it's all Google's fault". Makes sense, I'm sure you'll fare better with Amazon.
No, but Google pretending that email is reliable way to inform customers about such important thing is.
Re: Our production servers were suspended by Google Cloud
#84I am not a fan of "The Cloud" for ongoing production. I understand the case for it but I never liked the risk of someone else, with a different agenda and incentives than me, running my critical infrastructure. The Cloud does alleviate some risks with hosting your own servers, but it introduces new risks. This reminds me of a slogan I saw many years ago: There is no cloud, it's just somebody else's computer. That som…
I sort of tend to agree with you, but the counter-point is that there is always going to be some critical piece of the infrastructure that you are dependent upon that someone else controls. If you colocate, it's the data centre. If you host on-prem, it's the physical telco infrastructure.
There are no shortages of incompetent people and institutions that will screw you. It's typically incompetence, not malice. But you will run into malicious actors as well.
The reason I sort of agree with you is that, being someone who is very comfortable with bare metal solutions and came from the 90s mentality of writing portable code, I have started to see "The Cloud" as a return to mainframe programming. There is nothing wrong with that, but you are programming for "a machine" that often leads to code that is exceedingly hard to port. "Cloud Agnosticism" is the industry buzzword for not locking yourself into one vendor, but you need to make that decision on day one and set engineering policies enforcing a cloud services abstraction layer to be maintained. Most startups in the process of scaling are not going to see the immediate business value in incurring that cost. (though, with more horror stories like this one maybe that will change... but I doubt it).
Re: Our production servers were suspended by Google Cloud
#85Make sure your decision to migrate to AWS is not an emotional one. I had my own feud with AWS over a $60,000 bill and an enormous amount of hours spent by my team fighting crypto hackers after they took over one of our accounts and use very advanced scripts to create hundreds of large instances. AWS has become very complicated to manage and at the same time the attacks to their infrastructure have increased exponenti…
[dead]
Re: Our production servers were suspended by Google Cloud
#86Reminder: if your level of interaction with a service is putting your credit card in, the service is likely to interact with you at the level of automated emails, and also likely to trust you as much as someone who filled out a web form.
If it's business critical, talk to a human. All the cloud providers have account managers just waiting to talk to you. If this sort of thing happens and you have an account manager, they call you!
Even better, get a contract that prevents them from cutting you off like this, or sets an SLA for getting back online. Again, all the cloud providers will offer this, even for relatively low spends.
Re: Our production servers were suspended by Google Cloud
#87A nice big Threadripper in the corner of your office is all you need, I can almost certainly guarantee it. There are very very few businesses with less than 1000 employees that can't be run on one modern high end server.
What happens when the power goes out? Internet goes out? Someone spills a cup of coffee on the server? Landlord needs to do some work and drywall dust flies around everywhere? It's not that most CRUD apps could run on a single machine, it's that they probably shouldn't. Customers want redundancy, automated failover, etc. Datacenters are much better at offering these things than your office park.
You invest in 1-2 UPSes and a 4/5G mobile modem data plan that kicks in when needed. If you really want to be sure, get a diesel generator also. Will take some time to setup but it is worth it.
- No one can see your code
- No one can shut you down
Re: Our production servers were suspended by Google Cloud
#88It seems like Google has burned up nearly all the goodwill it used to have years ago. They now have a nearly broken search engine that needs the term “Reddit” appended to get anything useful, and you routinely hear horror stories about them acting badly without any regard to users. I hope they get disrupted and replaced by bing chat and others asap. The only thing that will possibly teach them a lesson is if their bu…
These are memes that are popular on HN, but don't reflect reality outside of the tech bubble. There are of course elements of truth to them, hence why they're so pervasive, but they are not representative of reality.
Re: Our production servers were suspended by Google Cloud
#89Earlier quoted context omitted.
> The industry has moved beyond depending on a single cloud computing vendor. Given how much of the industry struggles when an AWS region goes down, this seems... unlikely.
How about “the industry should have moved beyond depending on a single cloud computing vendor.”?
Our company tried that and was huge waste of time. Resulted in us wasting a lot of time trying to make our stuff "generic" so we could switch off of AWS if we ever wanted to. Eventually 1+ years later someone finally realized how much time we were wasting designing for some hypothetical event that they finally told us to stop doing that and fully utilize AWS directly.
Re: Our production servers were suspended by Google Cloud
#90I hate how the internet age has conditioned customers to expect less. There is no way a non-internet business would cut off a client of 4-years without actually talking (phone) to them.
Same with uptime.
I come from a healthcare and telephony background where "five nines" (roughly five minutes and 32 seconds of cumulative downtime in a year) is actually still a thing. I'm absolutely amazed at the number of companies that just accept each region/service/control plane/networking/etc outage in the big clouds. First you wait an extended period of time for them to even acknowledge the issue (rarely even updating their status dashboard) and then you stand around powerless waiting for your business to come back online. Time and time again.
Yes, multiple region deployments are encouraged (still not even five nines) but when AWS and others give you the ability to create really convoluted architectures (which they love and encourage - it's sticky) using at least a half dozen of their products (IAM, VPC, RDS, S3, etc) for the simplest solution that make it virtually impossible to deploy, debug, and maintain multi-region without a dedicated "AWS Architect" role. Downstream outages for solutions/companies building in clouds because of region or service issues demonstrate just how difficult this approach is in practice.
Then you sit back and wait for the almost inevitable "where did this $50k invoice come from", "we ran through our monthly budget overnight", "Oh they don't like us anymore and are closing our account", "What's the spend going to be this month - no one knows" and other scenarios discussed in this thread.
Can't figure out how to wrangle costs? Sick of the outages? Good luck getting your cloud Rube Goldberg machine migrated out (especially because by now you've hired based on cloud of choice).
Depending on a variety of factors any of these scenarios can kill a company.
This isn't normal. This is regressive. It is unacceptable.