Hi everyone, I'm the customer in this case. Wanted to let everyone know that this sort of abrupt suspension for production apps has been a recurring issue that has happened time and again with Google Clown Platform. Similar horror stories: https://news.ycombinator.com/item?id=32547912 https://news.ycombinator.com/item?id=33737577 ----- [Update 1] Yes I agree I have some fault for the initial suspension, but that's no…
Our production servers were suspended by Google Cloud
91–100 of 189 posts
Re: Our production servers were suspended by Google Cloud
#92Make sure your decision to migrate to AWS is not an emotional one. I had my own feud with AWS over a $60,000 bill and an enormous amount of hours spent by my team fighting crypto hackers after they took over one of our accounts and use very advanced scripts to create hundreds of large instances. AWS has become very complicated to manage and at the same time the attacks to their infrastructure have increased exponenti…
Did you get to a root cause of how the “crypto hackers” “took over” your account, or of what the “very advanced” scripts did?
Re: Our production servers were suspended by Google Cloud
#93Make sure your decision to migrate to AWS is not an emotional one. I had my own feud with AWS over a $60,000 bill and an enormous amount of hours spent by my team fighting crypto hackers after they took over one of our accounts and use very advanced scripts to create hundreds of large instances. AWS has become very complicated to manage and at the same time the attacks to their infrastructure have increased exponenti…
> You simply can't count on the baseline configuration of your account and instances provided by AMS, you need advanced knowledge on how to secure your resources beyond basic stuff like MFA and monitoring. What advanced knowledge are we talking about here?
Re: Our production servers were suspended by Google Cloud
#94Reminder: if your level of interaction with a service is putting your credit card in, the service is likely to interact with you at the level of automated emails, and also likely to trust you as much as someone who filled out a web form. If it's business critical, talk to a human. All the cloud providers have account managers just waiting to talk to you. If this sort of thing happens and you have an account manager,…
Re: Our production servers were suspended by Google Cloud
#95Hi everyone, I'm the customer in this case. Wanted to let everyone know that this sort of abrupt suspension for production apps has been a recurring issue that has happened time and again with Google Clown Platform. Similar horror stories: https://news.ycombinator.com/item?id=32547912 https://news.ycombinator.com/item?id=33737577 ----- [Update 1] Yes I agree I have some fault for the initial suspension, but that's no…
Why do you use a hosting solution that you yourself call "Google Clown" ?
Re: Our production servers were suspended by Google Cloud
#96It seems like Google has burned up nearly all the goodwill it used to have years ago. They now have a nearly broken search engine that needs the term “Reddit” appended to get anything useful, and you routinely hear horror stories about them acting badly without any regard to users. I hope they get disrupted and replaced by bing chat and others asap. The only thing that will possibly teach them a lesson is if their bu…
These are memes that are popular on HN, but don't reflect reality outside of the tech bubble. There are of course elements of truth to them, hence why they're so pervasive, but they are not representative of reality.
Re: Our production servers were suspended by Google Cloud
#97“Moving Forward. Starting today, I will be making plans to move everything to AWS.” I laughed when I read this because it’s presenting moving to AWS as a completely different future. The industry has moved beyond depending on a single cloud computing vendor. It’s negligent to put all your eggs in one basket. I would say don’t put your business in the hands of a single cloud vendor. Either exit the cloud or go multi c…
> The industry has moved beyond depending on a single cloud computing vendor. Given how much of the industry struggles when an AWS region goes down, this seems... unlikely.
Don't host production stuff in there. And if you must, have a backup somewhere.
Re: Our production servers were suspended by Google Cloud
#98Hi everyone, I'm the customer in this case. Wanted to let everyone know that this sort of abrupt suspension for production apps has been a recurring issue that has happened time and again with Google Clown Platform. Similar horror stories: https://news.ycombinator.com/item?id=32547912 https://news.ycombinator.com/item?id=33737577 ----- [Update 1] Yes I agree I have some fault for the initial suspension, but that's no…
Why do you use a hosting solution that you yourself call "Google Clown" ?
Re: Our production servers were suspended by Google Cloud
#99Reminder: if your level of interaction with a service is putting your credit card in, the service is likely to interact with you at the level of automated emails, and also likely to trust you as much as someone who filled out a web form. If it's business critical, talk to a human. All the cloud providers have account managers just waiting to talk to you. If this sort of thing happens and you have an account manager,…
How do you get such a contract at the big providers?
Re: Our production servers were suspended by Google Cloud
#100Make sure your decision to migrate to AWS is not an emotional one. I had my own feud with AWS over a $60,000 bill and an enormous amount of hours spent by my team fighting crypto hackers after they took over one of our accounts and use very advanced scripts to create hundreds of large instances. AWS has become very complicated to manage and at the same time the attacks to their infrastructure have increased exponenti…
Did you get to a root cause of how the “crypto hackers” “took over” your account, or of what the “very advanced” scripts did?
The attack resembled a very aggressive, clever virus so the people behind it definitely had very advance knowledge of AWS.
They could enable regions we had disabled and recreate roles that were deleted in a matter of minutes, again MFA on, no keys.
AWS kept giving us conflicting instructions and refused to disable our account unless we stop the attack first.
We have been customers of AWS since 2014.
I believe during that time the Shared Responsibility Agreement did not exist, so AWS demanded we signed it now for them to even consider taking a look.
That's when I reached out to the Attorney General of my state who sent a letter to AWS and got things moving.
We were contacted by another team (two months later) in AWS that seemed much more knowledgeable and they gave us the scripts that eventually stopped the attack.
Our case is not that rare, if you do some research you'll fine plenty of stories like ours.