Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Google Cloud suspended our production projects at 1am on Saturday

news.ycombinator.com

Post reply on HN
Print view View original

491–500 of 531 posts

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#491

Post by mhoad »

asojfdowgh wrote: post #389
mhoad wrote: post #350

Earlier quoted context omitted.

This is a fundamentally unserious point unless you want to add more qualifiers. I’m pointing to this as evidence https://cloud.google.com/customers I feel like a lot of the problems that people run into with GCP are avoided with two main components. Number one you should look seriously at the “enterprise” designation for different products and services if you are looking for long term stability and guarantees about t…

or just cluster over several other providers and make sure to test failover the main problem with azure/aws/gcp is that their parent companies will track everything you and others do, and then close every related thing because they have enough market share to not give a fuck about you. There are more than enough blog posts of enterprise customers running into random issues with the big thing, buying "big three, promi…

I feel like you’ve just ignored everything I mentioned, made up a bunch of hand wavy stuff and ended up with what might be actively the worst idea (multi-cloud) as a solution.

Not sure what to say in response here…

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#492

Post by Gibbon1 »

jnsaff2 wrote: post #396
luka-birsa wrote: post #386

A suggestion for engineers that lack business experience. As soon as your project grows beyond the non-hobby scale and starts making real money, invest some time to reach out to company providing the services that keep it alive and find an account manager that will take care of you. We're a Google customer and of course we had our fair share of issues (btw: it's the same with Azure, based on our experience), and we a…

I found the Azure reps to be completely useless for any of the actual problems. I worked for couple of really big enterprises, one of which had such a high commitment that Azure sent two engineers to sit with the teams working on their cloud. Highest level Enterprise support .. and yet for actual non-obvious problems it took them a month to reply with: can't help you with this, you must be doing it wrong. This was te…

You comment reminded me of what my dad said about service contracts from IBM back in the 1970's. They promise they'll get a tech on site in 24 hours. They don't promise he'll know how to do anything.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#493

Post by locutous »

londons_explore wrote: post #99
KingOfCoders wrote: post #29

Also: If you're on AWS/GCP and only backup to S3/GCS, you're doing it wrong. All your backups to setup a new system (even if it takes time) need to be with a different provider.

Backups to another provider are very costly for most users due to egress bandwidth pricing. Most backup systems can't do efficient incremental diffs either - so you end up sending your companies entire database every 12/24 hours, burning through a lot of bandwidth.

I use borg backup for this very use case.

Not free, certainly. But it's a system requirement there to be able to rebuild the system in a non AWS environment.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#494

Post by ufmace »

rglullis wrote: post #175
sumedh wrote: post #168

Earlier quoted context omitted.

That post sounds like war on drugs, on paper it makes sense but does not work in real life. Do you really think an industry like oil refinery can be managed with a limit of 150 people?

The point is not to limit the size of the industry. It is to limit the size of a company . Think of a oil refinery as a monolith, and that such a mandate would enforce that you'd have to build the whole system a set of independent microservices. The industry as a whole would still employ a large number of people, but now they would be forced to coordinate through specific interfaces (the smaller business units) inste…

I know you meant a general idea, but for oil refineries in particular, and probably a bunch of others, this sounds disastrously bad from a safety perspective. A lot of accidents that are tremendously costly in terms of human lives and health, equipment destroyed, and operational uptime eventually trace back to poor communication between groups. There needs to be some top-level department with stop-work authority over everything to coordinate who's doing what maintenance when, equipment replacements and switchovers, and the overall plan for what's being produced when. I can't help but think that if you intentionally made inter-department communication worse, it would result in a huge increase in the number of disasters.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#495

Post by Gibbon1 »

kyriakos wrote: post #425
mattbee wrote: post #130

This is why my old hosting company changed its auto disconnection code to work at 11am on a Monday morning, never at a weekend. That way it's the bill-payers that notice and get in touch, not the poor tech on the out-of-hours rota. Also, it only firewalled the customer's public IPs from the Internet - it didn't stop servers. That way when the account is settled there was often nothing for the techs to do. Also it was…

That's how hetzner handles late payments and only does this after 2 warning emails.

In the normal BnB world anyone with a account gets a phone call before they get cut off.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#496

Post by salmonlogs »

I used to work at Google Cloud, I left about 2 years ago so my advice might be slightly outdated so hopefully it helps someone.

I highly recommend anyone running production workloads to get in touch with sales and meet your account manager, and customer engineer.

You also need to swap to invoice billing and sign the offline payment terms. Credit card payments leave you in a MUCH higher risk category at higher risk of suspension.

You can ask them to put a flag on your accounts so that you won't get suspended in the event of billing issues. Iirc it gives you something like 180 days and blocks any of the automated systems.

The downside is you could get a crazy huge bill if you have an account compromised or some rogue IaC code. But you can manage that with quotas mostly. And with some good negotiation you can get a rogue bill written off. YMMV.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#497

Post by kyrra »

Googler, opinions are my own.

I know nothing about this case, but I have a few thoughts about this story.

1) The "full story" is actually rather sparse on exact details. The author quoted just a few words of "Your Projects are at risk of suspension", but doesn't provide more details about what was actually in the email. Did something else potentially cause this shutoff, not just billing? (I agree, based on his wording that this likely was a billing issue, but it would be good to know the full story).

2) The OP has not replied to any comments on here.

3) If the OP is reading this, they should go through their front-line support people at GCP saying this story is about you, so they can help you look into it. (it'll likely get the PR people to know who you are). If there was really a screw up where you paid and were still shut off for billing non-payment, I bet that would lead to a larger postmortem internally.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#498

Post by faizshah »

ravenstine wrote: post #434
vidarh wrote: post #237

Earlier quoted context omitted.

The pendulum will swing over cost, not over outages like these, because the cost challenges are real, while these outages are rare. It won't go all back to on prem because cloud providers will slash their margins once they struggle to maintain growth, and doing so they'll diminish the advantages of on-prem/hybrid setups for a lot of people, and so we'll find some equilibrium or other. I'd expect more emphasis on hybr…

Which is funny because reduced cost was one of the major selling points of "cloud" way back when. Most companies aren't actually at the scale where they need cloud. Cloud comes with the benefit of minimal to no need for ops/devops, high uptime (when nothing goes wrong lol), integrated DDoS protections, APIs your junior devs can string together, and the illusion of infinite scaling. In reality, it turns out you do nee…

On demand pricing is also a key benefit of cloud. Instead of needing to make a big initial spend on hosting infra you can pay only for what you need.

The other big benefit is auto scaling, in 2012 lots of companies would launch and be unable to meet traffic from a big push from reddit or HN or others. Today it’s rare to see a new product’s site go down due to a “hug of death.”

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#499

Post by pdimitar »

exikyut wrote: post #413
patchtopic wrote: post #402

Earlier quoted context omitted.

after Azure sunk my employers project with no recourse or discussion allowed, I always advise everyone I meet to run far away from Azure if they value a reliable service.

"No discussion allowed" sounds impressive, almost like legal impetus was involved. What parts of the story can be shared?

Pretty sure your parent poster just meant "they cut you off and you can't do absolutely anything about it".

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#500

Post by omniglottal »

theshrike79 wrote: post #428
fortran77 wrote: post #365

Earlier quoted context omitted.

But the person you're replying to didn't use ad words! Someone stole his CC number and used adwords.

Alphabet doesn't care, as far as they are concerned, someone messed with their Holy Money Machine and doesn't deserve to exist.

Again, the "doesn't deserve to exist" refers clearly to a different entity than the paying customer who was punished for zero action of their own.
Post reply on HN