Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Our production servers were suspended by Google Cloud

onvoard.com

Post reply on HN
Print view View original

161–170 of 189 posts

Re: Our production servers were suspended by Google Cloud

#161

Post by kjs3 »

Mizoguchi wrote: post #68

Make sure your decision to migrate to AWS is not an emotional one. I had my own feud with AWS over a $60,000 bill and an enormous amount of hours spent by my team fighting crypto hackers after they took over one of our accounts and use very advanced scripts to create hundreds of large instances. AWS has become very complicated to manage and at the same time the attacks to their infrastructure have increased exponenti…

If your company doesn't have a dedicated team to stay on top of AWS you'll be vulnerable to attacks like the one I experienced.

This has been my experience, 5+ years into our 'cloud journey'. For all those people we didn't have to have on board because cloud magically made them redundant, we've had to add almost as many people to ride herd on our relationship with the cloud providers.

TANSTAAFL

Re: Our production servers were suspended by Google Cloud

#162

Post by permo-w »

ceejayoz wrote: post #39
permo-w wrote: post #14

yes this is fairly unreasonable behaviour by Google, but also, come on man. if this is your business and livelihood, then you should be at the very least scanning over all emails from your main service provider, never mind ones clearly marked as “important”

I have nothing running on GCP and still have hundreds of "important", "alert", etc. emails from their platform. Tax changes in Belarus! Advisory Notifications will become Generally Available! Spike in user reported spam! Same reason I've almost thrown out important communications from my credit card company; they send so much "important!" mail that's actually an offer to sign up for paid credit card monitoring it sta…

I personally have 2 cloud compute instances running, one with a specially requested GPU, and going back 3 months I have no emails from them marked "important", and only two that haven't directly resulted from my actions, each an update - clearly marked "[Update]" - to some element of their legal infrastructure which I suspect they're required to inform of

have you actually checked this or are you just assuming it's the case?

Re: Our production servers were suspended by Google Cloud

#163

Post by dustymcp »

scarby2 wrote: post #131
NineStarPoint wrote: post #104

Earlier quoted context omitted.

It's for this reason that Google's ad business isn't really in danger at the moment, yes. People are going to continue to use google search and gmail and chrome. But having this reputation in the tech bubble negatively effects things like google cloud that tech people get to make decisions on. This is one of the reasons that google cloud is so far behind AWS and Azure in adoption.

The only reason it might be behind azure is how aggressively Microsoft market azure to existing customers and bundle it with other products. As a standalone product I'm not sure anyone would use it.

Azure has been nice to work with i have been comped multiple times for our own mistakes support is great im really not sure i get the hate..

Re: Our production servers were suspended by Google Cloud

#164

Post by oliwarner »

sofixa wrote: post #159
oliwarner wrote: post #77

It's a shame the industry can't agree on a basic SLA: - Solicit a hard limit when you open an account. People can only afford so much, don't bankrupt them. - As soon as we see your account heading for a 10,000% increase on last month, we'll send a SMS, GIVE YOU A PHONE CALL, even send a letter and confirm everything is okay. - If you're about to trigger a suspension, we'll send you an automated SMS, and if you're a p…

> - Solicit a hard limit when you open an account. People can only afford so much, don't bankrupt them. As is said every time this gets proposed, this is not only wildly impractical to implement, but also dangerous from an availability. Why is it impractical? You are billed for active things (running databases, network traffic) but also passive ones such as storage. What do you want AWS to do when you go over the har…

Admittedly, I'm no trillion dollar cloud host —and I don't know your perspective here— but I'd expect that a trillion dollar cloud host could look at my billing data at an hourly resolution and fairly accurately predict that that'll look like at the end of the month.

They don't need to cut me off right away (unless I'm already outside my budget) but for most bitten people, getting a call the day after making a colossal mistake will save everybody a lot of money and angst. Neither of us needs to negotiate and I don't have to sell Grandma.

It's been a few years but the last time I tried AWS's Budgets, it took a hundred clicks or so per service to just get an email out of it. Emails get missed, life gets in the way, etc. The cost to a trillion dollar cloud host to call me up is far less than it would cost you to call me as they off-shore it all.

And I'm all for people actively choosing to play without limits. I just think a lot of people a) don't respect that from the off, and b) aren't notified in a timely fashion.

Re: Our production servers were suspended by Google Cloud

#165

Post by kevincox »

nova22033 wrote: post #112

> 1) Google Cloud dropped us an email 10 days back that I missed out You missed a KYC e-mail and they locked your account. If you miss a "Important information about your bank account", what do you think your bank is going to do?

I agree that this was partly the OPs fault, but I think that Google did the very least.

1. Send more than one email. Three would probably be reasonable for an account shutdown action maybe at 10, 4 and 2 days.

2. Use a better subject. That subject sounds like they are giving you information. At least something like "Required Important Information about your Bank Account". But something even better would probably be like "Must Provide Information about your Bank Account to avoid Account Closure" and maybe some escalating severity in the later warnings.

3. Do they have a phone number on file? Give that a call if there is no response to the original email in 3 business days. It could even be an automated message. "Hello, this is Google Cloud. We have sent you an email about required action on your account. Please read the information in that email and act appropriately otherwise your account will be closed. Remember that all Google Cloud emails come from @google.com addresses."

4. A big banner in the Cloud Console. After 4 business days show it to all users in the account.

We are talking account shutdown of a paying customer's likely production services. They should take a bit of care.

Re: Our production servers were suspended by Google Cloud

#166

Post by sofixa »

kkielhofner wrote: post #113
gspencley wrote: post #84

Earlier quoted context omitted.

I sort of tend to agree with you, but the counter-point is that there is always going to be some critical piece of the infrastructure that you are dependent upon that someone else controls. If you colocate, it's the data centre. If you host on-prem, it's the physical telco infrastructure. There are no shortages of incompetent people and institutions that will screw you. It's typically incompetence, not malice. But yo…

I've done large colo deployments. When you're provisioning your own hardware it's cloud-init, ansible, K8s, whatever. Nothing even knows you're in colos operated by different providers in different regions. Nothing knows or cares you're using five different upstream internet providers (whether your own - complicated or IP blends from the colo). Have a falling out or big issue? No reputable colo provider is going to h…

> All of this, by the way, is a fraction of the cost of cloud solution (a $25k bonkers 1U is ~$700 a month with a three year lease and 1U hosting + bandwidth + power is ~$200 a month). So for $2k a month (and it's actually predictable) you're up in two facilities and have capabilities an order of magnitude greater than a big cloud at equivalent spend.

You mean you have raw power an order of magnitude greater than a big cloud at equivalent spend. Capabilities are all DIY - if you want a database, object storage, replicated storage between your two locations, message queue, etc. etc. you have to set it all (and think of the backups, patching, monitoring, etc.) on your own. That's a non-insignificant costs that will easily dwarf that $2k. Source: I've done this at an MSP/MHP, handling a decent chunk of the infra side.

Re: Our production servers were suspended by Google Cloud

#167

Post by b1c1jones »

Mizoguchi wrote: post #68

Make sure your decision to migrate to AWS is not an emotional one. I had my own feud with AWS over a $60,000 bill and an enormous amount of hours spent by my team fighting crypto hackers after they took over one of our accounts and use very advanced scripts to create hundreds of large instances. AWS has become very complicated to manage and at the same time the attacks to their infrastructure have increased exponenti…

I had a similar experience with AWS. They warned us of a crypto hijacking of one of our servers. They weren't any help on internally battling it. They were hands off on anything within the server, and didn't have any advice on how to address it. I had to manually battle it. Their backup and device swapping services were pretty handy though.

Re: Our production servers were suspended by Google Cloud

#168

Post by philliphaydon »

ragnarsson wrote: post #71
Mizoguchi wrote: post #68

Make sure your decision to migrate to AWS is not an emotional one. I had my own feud with AWS over a $60,000 bill and an enormous amount of hours spent by my team fighting crypto hackers after they took over one of our accounts and use very advanced scripts to create hundreds of large instances. AWS has become very complicated to manage and at the same time the attacks to their infrastructure have increased exponenti…

> You simply can't count on the baseline configuration of your account and instances provided by AMS, you need advanced knowledge on how to secure your resources beyond basic stuff like MFA and monitoring. What advanced knowledge are we talking about here?

[dead]

Re: Our production servers were suspended by Google Cloud

#169

Post by ROTMetro »

igortg wrote: post #96
danpalmer wrote: post #88

Earlier quoted context omitted.

These are memes that are popular on HN, but don't reflect reality outside of the tech bubble. There are of course elements of truth to them, hence why they're so pervasive, but they are not representative of reality.

Partially agree, but ... the tech bubble is pretty good on dictate trends and set standards that the general public will be in 2-3 years in advance.

Is that still true? I think a group of tech influencers existed in the past but I don't see any evidence that continues to be the case. I guess they could all exist in discords I don't have site into. Previously you could see trends migrate through tech communities and out into the world. I don't really see that now.

Re: Our production servers were suspended by Google Cloud

#170

Post by jen20 »

thr717272 wrote: post #160
jen20 wrote: post #157

Earlier quoted context omitted.

It’s to quote specific words which are pertinent to the threat level implied, rather than the entire post. Extraordinary claims require extraordinary evidence.

Common sense is a thing and what common sense can tell us absolutely isn't an extraordinary claim. Edit: By the way, what do you mean by threat level here? And FWIW, here I could have used quotes without being rude. Also: if you read that sentence again you'll probably find that it wasn't just the use of quotes but the sentence that as a whole that made it stand out as rude.

A baseline AWS account with MFA will not be taken over. That would require a credential leak or an IAM misconfiguration permitting access from some other account.

Suggesting it is an extraordinary claim, and would suggest an extraordinary threat to all AWS accounts. This seems very unlikely.

Post reply on HN