Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Our production servers were suspended by Google Cloud

onvoard.com

Post reply on HN
Print view View original

141–150 of 189 posts

Re: Our production servers were suspended by Google Cloud

#141

Post by tasuki »

Mizoguchi wrote: post #100
jen20 wrote: post #70

Earlier quoted context omitted.

Did you get to a root cause of how the “crypto hackers” “took over” your account, or of what the “very advanced” scripts did?

We had basic security like MFA set up and we never shared keys, so we don't understand until this day how they managed to take over. The attack resembled a very aggressive, clever virus so the people behind it definitely had very advance knowledge of AWS. They could enable regions we had disabled and recreate roles that were deleted in a matter of minutes, again MFA on, no keys. AWS kept giving us conflicting instruc…

> The attack resembled a very aggressive, clever virus so the people behind it definitely had very advance knowledge of AWS.

In what way did the attack resemble a very aggressive, clever virus?

Do you mean that it happened fast? How do you know it wasn't something simple like your credentials leaking?

Re: Our production servers were suspended by Google Cloud

#144

Post by kkielhofner »

dman wrote: post #132
kkielhofner wrote: post #113

Earlier quoted context omitted.

I've done large colo deployments. When you're provisioning your own hardware it's cloud-init, ansible, K8s, whatever. Nothing even knows you're in colos operated by different providers in different regions. Nothing knows or cares you're using five different upstream internet providers (whether your own - complicated or IP blends from the colo). Have a falling out or big issue? No reputable colo provider is going to h…

Could I contact you to pick your brains about this? I spend much more than $2k for dev environments at my startup and a setup like this would work beautifully for us.

Sure! Contact info in profile.

Re: Our production servers were suspended by Google Cloud

#145

Post by Mizoguchi »

oliwarner wrote: post #77

It's a shame the industry can't agree on a basic SLA: - Solicit a hard limit when you open an account. People can only afford so much, don't bankrupt them. - As soon as we see your account heading for a 10,000% increase on last month, we'll send a SMS, GIVE YOU A PHONE CALL, even send a letter and confirm everything is okay. - If you're about to trigger a suspension, we'll send you an automated SMS, and if you're a p…

Absolutely agree.

AWS is pretty much a line of credit, unlimited credit actually.

When you open a credit card you are usually given a limit, say 10K.

If there is a (pretty obvious) suspicious activity on your account it gets disabled, the bank just won't let your card be used indefinitely to buy yachts and planes.

And when you contact the bank they won't ask you to go find the bad guys and stop them from using your card before they can help you.

That would make absolutely no sense but that's exactly what AWS did in our case.

Re: Our production servers were suspended by Google Cloud

#146

Post by unxdfa »

uppe wrote: post #106
orangepurple wrote: post #74

Earlier quoted context omitted.

How did Russian companies mitigate the effects of cloud services hosting being blocked by sanctions

[flagged]

They are a society manipulated heavily by propaganda, run by rich elites and pushed into an unproductive war by a political dictatorship supported by the elites. Perhaps everyone should have stopped doing business with the USA for the last few decades for the same reason.

You can creatively frame a problem any way you choose.

All countries have a problem with media, elites and politicians. We should aim to solve those problems, not cut a large chunk of the planet off. They are people too.

Re: Our production servers were suspended by Google Cloud

#148

Post by Mizoguchi »

tasuki wrote: post #138
Mizoguchi wrote: post #68

Make sure your decision to migrate to AWS is not an emotional one. I had my own feud with AWS over a $60,000 bill and an enormous amount of hours spent by my team fighting crypto hackers after they took over one of our accounts and use very advanced scripts to create hundreds of large instances. AWS has become very complicated to manage and at the same time the attacks to their infrastructure have increased exponenti…

> very advanced scripts to create hundreds of large instances. What was very advanced about those scripts? As an outsider, I'd think that perhaps a very simple script might be enough to create hundreds of large instances.

Sure, but that was only one part of the attack and not the clever one. The ability to bypass security, even after following AWS support instructions is what made it so difficult to fight, to the point that only after AWS security team took over the case it did not stop. The problem is AWS security doesn't interact with customers directly (as far as I know), you have to go through tech support and these guys just don't have that level of knowledge.

Re: Our production servers were suspended by Google Cloud

#149

Post by ChatGTP »

ashwagary wrote: post #133
pirsquare wrote: post #7

Hi everyone, I'm the customer in this case. Wanted to let everyone know that this sort of abrupt suspension for production apps has been a recurring issue that has happened time and again with Google Clown Platform. Similar horror stories: https://news.ycombinator.com/item?id=32547912 https://news.ycombinator.com/item?id=33737577 ----- [Update 1] Yes I agree I have some fault for the initial suspension, but that's no…

I wonder if there is a cheap existing solution to use multiple cloud hosts at once and minimize this risk instead of moving all eggs to the AWS basket.

Well if you deploy to Kubernetes it standardizes thee APIs you're using to deploy with quite a lot, then you can move around easier.

Re: Our production servers were suspended by Google Cloud

#150

Post by ChatGTP »

eigenvalue wrote: post #64

It seems like Google has burned up nearly all the goodwill it used to have years ago. They now have a nearly broken search engine that needs the term “Reddit” appended to get anything useful, and you routinely hear horror stories about them acting badly without any regard to users. I hope they get disrupted and replaced by bing chat and others asap. The only thing that will possibly teach them a lesson is if their bu…

It's funny you're appending Reddit to your searches too. I noticed this is very common, ha.
Post reply on HN