Tell HN: Google Cloud suspended our production projects at 1am on Saturday • Hacker News Board

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#381

Post by stuff4ben » Mon, Aug 22, 2022, 1:58 PM UTC

staticassertion wrote: post #326

hazbo wrote: post #124

A few years ago I was working at a startup and we had just started moving some stuff over from AWS to GCP. Things were going pretty good with GCP until one day they pulled the plug on everything with no explanation. It turns out that our company credit card details had been fraudulently used without our knowledge. The criminal had decided to use the card to buy Google Ad Words or something like that - this is the sam…

I hear way too many stories like this about Google products. Whether it's Youtube, AdWords, GCP, or anything else - you get blackholed by them and there's zero recourse. I just can't trust Google for something like running production services.

"can't trust Google"

There, fix that for you. If it's mission-critical, there are better customer-focused solutions out there than Google. Despite their massive brain power, they just can't seem to get dealing with users correctly.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#382

Post by davidspiess » Mon, Aug 22, 2022, 2:05 PM UTC

markbnj wrote: post #362

b112 wrote: post #157

Earlier quoted context omitted.

Google is just horrible, when it comes to any form of support, paid or not. They make paypal look helpful by comparison. I really can't understand why anyone would use any of their services. Support counts. It's one of the first things I look at, when evaluating any external partner. What does it cost, how long does it take for it to happen, etc? Whatever anyone else says in this thread, that's what you need to deter…

> Google is just horrible, when it comes to any form of support, paid or not. They make paypal look helpful by comparison. We have been running production on Google Cloud for six years. We are not a large customer: our spend is in the low five figures. We pay for a middle-tier support plan and support has been excellent during this entire period. We very rarely need to take advantage of it, since the underlying syste…

Exactly our experience too. We are extremely happy with the stability and quality of their services.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#383

Post by softawre » Mon, Aug 22, 2022, 2:05 PM UTC

designium wrote: post #341

mattbee wrote: post #130

This is why my old hosting company changed its auto disconnection code to work at 11am on a Monday morning, never at a weekend. That way it's the bill-payers that notice and get in touch, not the poor tech on the out-of-hours rota. Also, it only firewalled the customer's public IPs from the Internet - it didn't stop servers. That way when the account is settled there was often nothing for the techs to do. Also it was…

Which hosting company are you referring to?

From his profile:

> Cofounder and former MD of Bytemark, a British hosting company.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#384

Post by acka » Mon, Aug 22, 2022, 2:06 PM UTC

intellectronica wrote: post #245

paskozdilar wrote: post #202

Earlier quoted context omitted.

> These mistakes are relatively easy to prevent. Assuming one knows about them. How would one learn about them, without making the same mistakes themselves?

Most of it will actually be documented by your provider, and is often pretty obvious. For example, if you're paying with card, have at least one extra as backup payment method (which most providers allow and recommend). For extra safety: make one of these cards a pre-paid debit card loaded with enough money to pay for at least one month. The one thing you don't want to happen is to miss a payment, which could happen…

intellectronica:

> For example, if you're paying with card, have at least one extra as backup payment method (which most providers allow and recommend).

Please don't generalize. There are many countries where credit cards aren't common, and applying for just one (and getting it approved) can already involve a lot of work and take lots of time. Such as: the bank and the national credit authority needing to do a thorough credit check, you having to supply proof of identity and stable income, the bank having to register your new line of credit with the national credit authority, etcetera. Applying for multiple CC's, if at all possoble, only adds more trouble: your registered total amount of credit will rise (even if you haven't used any of it yet) which can limit your options for any other loans like a mortgage, a car loan, a subscription, and so on.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#385

Post by lights0123 » Mon, Aug 22, 2022, 2:07 PM UTC

jquery wrote: post #223

jeroenhd wrote: post #55

Earlier quoted context omitted.

Backblaze is very cheap and has several "don't overload our system" clauses in their terms and conditions. They're cheap and reliable, that generally comes at the risk of early termination. I don't think you can have a cheap, reliable, unlimited service; pick two at most.

Indeed. If a customer tries to backup a petabyte of data on the base plan (because it’s “unlimited”) and they get the boot, they really shouldn’t be that surprised. Wouldn’t really compare it to OP’s situation, where they got an assurance in writing they were in good standing, and didn’t appear to be blocked for violating ToS.

"Token" hints at everything-is-metered B2, not their unlimited backup plan.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#386

Post by luka-birsa » Mon, Aug 22, 2022, 2:09 PM UTC

A suggestion for engineers that lack business experience. As soon as your project grows beyond the non-hobby scale and starts making real money, invest some time to reach out to company providing the services that keep it alive and find an account manager that will take care of you.

We're a Google customer and of course we had our fair share of issues (btw: it's the same with Azure, based on our experience), and we always escalate to our account manager. That usually starts turning the wheels much faster.

Also, do not be afraid to ask your account manager for additional discounts. Everybody in sales has some wiggle room to get you a better deal. With Google their account manager actually suggested that we work through a reseller and we're getting % off the list prices.

Dealing with Google, Amazon, etc.. is not like dealing with code. It's not only transactional and you need to invest some time in the relationship even if you're the customer.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#387

Post by skybrian » Mon, Aug 22, 2022, 2:09 PM UTC

christophilus wrote: post #309

alphaomegacode wrote: post #262

For a company that operates at such massive scale for their own services, Google has some glaring oversights when it comes to enterprise services. I have a friend who volunteers as a webmaster for a charity org and they kept getting notices about their impending shut down of G Suite and the need to move to Google Workspace. The problem seemed to be a credit card and yet, when my friend was showing me on his laptop, t…

Was Angular 1.x an official Google thing? I always thought it was just a Google employee’s project.

I doubt it was ever an official Google product because open source packages generally aren't. But there was a team at Google working on it when Angular 1.x was big.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#388

Post by powerhour » Mon, Aug 22, 2022, 2:09 PM UTC

rglullis wrote: post #128

riedel wrote: post #116

Earlier quoted context omitted.

I fully agree that automakers should be held responsible. However, I am feeling a bit of negative sentiment here. What does 'properly' mean? There has been court cases that have been decided against them. Sure there is always more. But this sounds a bit like whatabotism. Also there is a high probability of defeat devices in Citroen, Dacia, Fiat, Ford and Honda cars as well. It is however true that the EU create a hug…

"Properly" means that the automakers should pay a fine per vehicle sold that was producing more emissions than allowed, and that fine should be so high as to effectively get them into bankruptcy. It should also put in jail anyone that was aware of the violation and did not report to the authorities. Everyone , from the CEO to the test engineer who knew about the cheating devices. Anything less than that and all you a…

I'm mostly with you in spirit but I'd stop at big fines and jailing the executives. Most of the employees are probably fine and someone better can take the helm without too much disruption to their work. Jailing executives might be enough on its own but the fine is useful as well, to pay compensation if nothing else.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#389

Post by asojfdowgh » Mon, Aug 22, 2022, 2:11 PM UTC

mhoad wrote: post #350

mcv wrote: post #315

Earlier quoted context omitted.

I don't think you can. My takeaway from this discussion is definitely that GCP is only suitable for toy projects, and not for serious production stuff that requires continuity.

This is a fundamentally unserious point unless you want to add more qualifiers. I’m pointing to this as evidence https://cloud.google.com/customers I feel like a lot of the problems that people run into with GCP are avoided with two main components. Number one you should look seriously at the “enterprise” designation for different products and services if you are looking for long term stability and guarantees about t…

or just cluster over several other providers and make sure to test failover

the main problem with azure/aws/gcp is that their parent companies will track everything you and others do, and then close every related thing because they have enough market share to not give a fuck about you.

There are more than enough blog posts of enterprise customers running into random issues with the big thing, buying "big three, promise edition" isn't much of an assurance.

I'd suspect most of those customers have rapid migration plans written up. For a reason.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#390

Post by scrollaway » Mon, Aug 22, 2022, 2:11 PM UTC

Nebasuke wrote: post #284

black_puppydog wrote: post #266

Earlier quoted context omitted.

> wait until the car is basically stopped you mean like an email literally saying they acknowledge the situation and won't suspend your account?

I think an email from support is more like "confirmation that they checked whether the car has seen me". I'm not sure how successfully you've dealt with support with companies, but I have almost never been able to rely on the promises of a support agent, as they are simply not the ones actually making the decisions (in this case on whether your account gets suspended or not).

A support agent is supposed to speak on behalf of the company. If they say "your account will not be suspended for this issue", and the company then suspends the account for that issue, that should be grounds to sue.