Our production servers were suspended by Google Cloud • Hacker News Board

Re: Our production servers were suspended by Google Cloud

#171

Post by gspencley » Mon, Mar 13, 2023, 5:18 PM UTC

sofixa wrote: post #166

kkielhofner wrote: post #113

Earlier quoted context omitted.

I've done large colo deployments. When you're provisioning your own hardware it's cloud-init, ansible, K8s, whatever. Nothing even knows you're in colos operated by different providers in different regions. Nothing knows or cares you're using five different upstream internet providers (whether your own - complicated or IP blends from the colo). Have a falling out or big issue? No reputable colo provider is going to h…

> All of this, by the way, is a fraction of the cost of cloud solution (a $25k bonkers 1U is ~$700 a month with a three year lease and 1U hosting + bandwidth + power is ~$200 a month). So for $2k a month (and it's actually predictable) you're up in two facilities and have capabilities an order of magnitude greater than a big cloud at equivalent spend. You mean you have raw power an order of magnitude greater than a b…

Yeah your mileage will vary greatly depending on your requirements. A couple of years ago I owned a very high-traffic, media-oriented website. Bandwidth costs were my highest cost, not compute or storage. I explored moving to something cloud-based and the cost increase just on bandwidth per GB would have been enough to put me out of business. Yes this was including a paid Cloudflare / CDN account sitting in front of my servers.

I considered a "hybrid" model where I would keep bare metal servers for distributing media but move compute and persistence to "the cloud" but just couldn't really justify it on cost alone. The replication and auto-scaling would have been nice, and were the major selling points, but wasn't a high enough ROI for me.

Re: Our production servers were suspended by Google Cloud

#172

Post by GrinningFool » Mon, Mar 13, 2023, 5:54 PM UTC

ar9av wrote: post #12

iamben wrote: post #6

Emails are a difficult thing, aren't they? Was the 10 day countdown the first time Google mentioned "we'll suspend the account unless..."? Generally on these things - and unless it's an absolute emergency - you'd expect lots of prior warning and at least a couple of "YOUR SERVICES ARE DUE TO BE SUSPENDED" emails in the 48 hours before. That said, I've rightly called my bank out for sending an email with the subject l…

It's frustrating when emails don't categorize as critical and we end up missing something important. And don't even get me started on those misleading subject lines that turn out to be nothing but a sales pitch. It's like we have to play detective just to figure out which emails are worth our attention. It would be great if companies could be more transparent and upfront about the urgency of their messages.

Every message is urgent to the company that sent it.

Re: Our production servers were suspended by Google Cloud

#173

Post by zamnos » Mon, Mar 13, 2023, 5:59 PM UTC

GrinningFool wrote: post #172

ar9av wrote: post #12

Earlier quoted context omitted.

It's frustrating when emails don't categorize as critical and we end up missing something important. And don't even get me started on those misleading subject lines that turn out to be nothing but a sales pitch. It's like we have to play detective just to figure out which emails are worth our attention. It would be great if companies could be more transparent and upfront about the urgency of their messages.

Every message is urgent to the company that sent it.

Get a better vendor? They should at least have separate categories for marketing vs "important" account information, vs actually important account information (that last category can't be unsubscribed from).

Re: Our production servers were suspended by Google Cloud

#174

Post by rurp » Mon, Mar 13, 2023, 7:19 PM UTC

philihp wrote: post #140

Mizoguchi wrote: post #68

Make sure your decision to migrate to AWS is not an emotional one. I had my own feud with AWS over a $60,000 bill and an enormous amount of hours spent by my team fighting crypto hackers after they took over one of our accounts and use very advanced scripts to create hundreds of large instances. AWS has become very complicated to manage and at the same time the attacks to their infrastructure have increased exponenti…

AWS understands lifetime value of the user, not just the account. I've had mistakes ON MY PERSONAL ACCOUNT that were in no way connected to my work. Times when I was just experimenting. They refunded it in full. I've also heard from STUDENTS who may not ever get into infra work that they're still reasonable and lenient. You would be absolute mad to use any other platform. AWS gets it right on the customer service fro…

A friend of mine had a similar experience when he was just learning. He accidentally racked up a large bill (for a student), contacted support, and they forgave the charges.

That's not to say that Amazon is being altruistic here, I think it just makes good business sense. Foregoing a few thousand dollars for even a tiny chance that a student's future self decides to use AWS for a multi-million spend project down the line is worth it.

Re: Our production servers were suspended by Google Cloud

#175

Post by albertopv » Mon, Mar 13, 2023, 7:44 PM UTC

nova22033 wrote: post #112

> 1) Google Cloud dropped us an email 10 days back that I missed out You missed a KYC e-mail and they locked your account. If you miss a "Important information about your bank account", what do you think your bank is going to do?

I would try to put me in customer shoes before shutting down their business because of missing paperwork. Even a single missed payment shouldn't be enough. Assume that counterpart may e didn't mean to do wrong, it almost always works.

Re: Our production servers were suspended by Google Cloud

#176

Post by LeBit » Mon, Mar 13, 2023, 9:02 PM UTC

ChatGTP wrote: post #149

ashwagary wrote: post #133

Earlier quoted context omitted.

I wonder if there is a cheap existing solution to use multiple cloud hosts at once and minimize this risk instead of moving all eggs to the AWS basket.

Well if you deploy to Kubernetes it standardizes thee APIs you're using to deploy with quite a lot, then you can move around easier.

I've been trying very hard to sell that idea to my team.

But no, they insist on using every product the different cloud providers offers. This, IMO, is a recipe to lock yourself to vendors and make maintenance and interoperability a nightmare.

It feels like when devs felt they had to use all the gang of four design patterns.

Re: Our production servers were suspended by Google Cloud

#177

Post by kkielhofner » Mon, Mar 13, 2023, 11:56 PM UTC

sofixa wrote: post #166

kkielhofner wrote: post #113

Earlier quoted context omitted.

I've done large colo deployments. When you're provisioning your own hardware it's cloud-init, ansible, K8s, whatever. Nothing even knows you're in colos operated by different providers in different regions. Nothing knows or cares you're using five different upstream internet providers (whether your own - complicated or IP blends from the colo). Have a falling out or big issue? No reputable colo provider is going to h…

> All of this, by the way, is a fraction of the cost of cloud solution (a $25k bonkers 1U is ~$700 a month with a three year lease and 1U hosting + bandwidth + power is ~$200 a month). So for $2k a month (and it's actually predictable) you're up in two facilities and have capabilities an order of magnitude greater than a big cloud at equivalent spend. You mean you have raw power an order of magnitude greater than a b…

When you turn $20k/mo of cloud spend into $2k/mo that’s $216k/yr - $648k over three years.

Postgres, minio, rabbit, whatever absolutely does not even come close for initial implementation and ongoing support. If you spend more than a tenth of that you’re doing it wrong.

Source: CTO who’s done all these things.

Re: Our production servers were suspended by Google Cloud

#178

Post by harry8 » Tue, Mar 14, 2023, 1:45 AM UTC

shadowgovt wrote: post #156

harry8 wrote: post #72

Earlier quoted context omitted.

Tell them to take a hike until they come back with a warrant or at least a reasonable justifiable suspicion not dragnet surveillance. You know like google is a goddamn american company? What is this "Oh it's ok, we're just going along with the government's presumption of guilt to justify yet another power grab so that's fine." TBF Google are big enough to display some spine if they'd "prefer not to" I'm pretty solid…

> Tell them to take a hike until they come back with a warrant or at least a reasonable justifiable suspicion not dragnet surveillance The Patriot Act doesn't work that way. Congress passed laws placing some reasonable minimum requirements on financial institutions that they have some sense of the actual identity of the people they are doing business with. Those are part of the "You're incorporated and do business in…

You talk like google pay less in lobbying cash than an individual citizen.

Google could refuse and take that s&^t to the supreme court while funding the opponent of every politician who took them on for it. That's what they do to ensure they're not getting broken up like a monopoly should be, after all. But yeah the latter is their profit, the former is merely the right thing to do. They're motivated to enhance their profit at the expense of the country, which is what out sized market power /is/ and to ignore something where they might improve the country because there's no cash in it for them.

You kind of might want to revisit this part: "if you're an individual US citizen, you have more power to change it than Google does." Because it's clearly ridiculous on every level and comically so.

Did the law really change since these guys signed up with google or did google "overlook it" to sign them, then pull the switch later without any change in law between the two dates?

Re: Our production servers were suspended by Google Cloud

#179

Post by greenyoda » Tue, Mar 14, 2023, 2:18 AM UTC

nova22033 wrote: post #117

If your servers were suspended, how is your website functional? https://console.onvoard.com/register/

In the second paragraph of the article: "I managed to get our servers back online few hours ago with a workaround by adding a new billing account."

Re: Our production servers were suspended by Google Cloud

#180

Post by shadowgovt » Tue, Mar 14, 2023, 2:59 AM UTC

harry8 wrote: post #178

shadowgovt wrote: post #156

Earlier quoted context omitted.

> Tell them to take a hike until they come back with a warrant or at least a reasonable justifiable suspicion not dragnet surveillance The Patriot Act doesn't work that way. Congress passed laws placing some reasonable minimum requirements on financial institutions that they have some sense of the actual identity of the people they are doing business with. Those are part of the "You're incorporated and do business in…

You talk like google pay less in lobbying cash than an individual citizen. Google could refuse and take that s&^t to the supreme court while funding the opponent of every politician who took them on for it. That's what they do to ensure they're not getting broken up like a monopoly should be, after all. But yeah the latter is their profit, the former is merely the right thing to do. They're motivated to enhance their…

A) SCOTUS would tell them to file the damn paperwork. Patriot Act has been challenged several times; it's well-crafted to be in compliance with the Constitution.

B) Google already has a PAC but PACs aren't votes, though the cynics would claim otherwise. If people want to repeal any part of the Patriot Act, they need to make it an issue. The kind of issue that shows up on the news.

C) It's hard to get purchase on changing this because know-your-customer is... Kind of a good requirement actually? Anonymity in business enables all manner of lawbreaking; sunlight is the best disinfectant, etc.