Live data from Hacker News

Tell HN: Google Cloud suspended our production projects at 1am on Saturday

news.ycombinator.com

431–440 of 531 posts

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#431
post #6

I wonder what it would take to bring a culture change to Google so they can finally start having some empathy for customers and stop the casual cruelty. A new CEO?

I doubt that can happen organically. The Google is ground zero for ego in Silicon Valley. When ego is inflated, empathy must deflate to make room.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#432

A suggestion for engineers that lack business experience. As soon as your project grows beyond the non-hobby scale and starts making real money, invest some time to reach out to company providing the services that keep it alive and find an account manager that will take care of you. We're a Google customer and of course we had our fair share of issues (btw: it's the same with Azure, based on our experience), and we a…

TOTALLY.

-

I had amazing AWS reps, that I had a good personal relationship. Request your rep to visit your office, and have lunch.

Seriously.

The best reps I had from AWS moved to GCP, and while I couldnt move my loads from AWS to GCP, I still had a good relationship with them. One of the best things an account rep can do is understand your business, and, in my case, actually pushed for changes in the system to our benefit based on how we were using the system.

They have evolved a lot since that occurred, but a good rep is going to seriously push for your success. (They want you to be successful and increase your use! :-) and thats a good ting)

So, make requests for your reps to meet with you face to face.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#433

The same is true of DigitalOcean, fwiw. Shutting down machines used in production with no notice, outside working hours. I don’t know why these companies don’t care - they’re massively shooting themselves in their feet. Incidents like this turn customers off for life.

Years ago, I used a cheap droplet on DO as an IRC bouncer to hide my home IP address. Someone came into a channel I was an op in and started spamming racial slurs, so I did what a normal op does and banned them.

They proceeded to DDoS my droplet.

Only a couple minutes later, DO responded by disconnecting my droplet from the Internet for 3 hours, supposedly to protect other customers. Thanks, DO, you let the DDoSer win. My droplet was handling the attack just fine, it just made my IRC connection a little laggy.

As a $5/month customer, I was just small potatoes, for sure. But I had ideas for products in the future, and I decided that I would not use DO for them, knowing that some skiddie could just take it offline for 3 hours by just packet flooding it for a couple minutes.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#434
post #237

Seeing all these horror stories, I can't help but wonder when people are going to switch back to on prem, or at least some hybrid. It's pretty apparent we can't trust cloud providers...

The pendulum will swing over cost, not over outages like these, because the cost challenges are real, while these outages are rare. It won't go all back to on prem because cloud providers will slash their margins once they struggle to maintain growth, and doing so they'll diminish the advantages of on-prem/hybrid setups for a lot of people, and so we'll find some equilibrium or other. I'd expect more emphasis on hybr…

Which is funny because reduced cost was one of the major selling points of "cloud" way back when.

Most companies aren't actually at the scale where they need cloud. Cloud comes with the benefit of minimal to no need for ops/devops, high uptime (when nothing goes wrong lol), integrated DDoS protections, APIs your junior devs can string together, and the illusion of infinite scaling.

In reality, it turns out you do need someone to effectively do ops, you don't have 99.999% uptime, your don't need most or any of those APIs, and you didn't need that promise of scaling after all.

With DDoS protection, on-prem hosting or independent dedicated hosting might be more economical and practical.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#435

For a company that operates at such massive scale for their own services, Google has some glaring oversights when it comes to enterprise services. I have a friend who volunteers as a webmaster for a charity org and they kept getting notices about their impending shut down of G Suite and the need to move to Google Workspace. The problem seemed to be a credit card and yet, when my friend was showing me on his laptop, t…

Was Angular 1.x an official Google thing? I always thought it was just a Google employee’s project.

I'm not sure if it was ever an official thing but I remember accessing their pages and speaking to their reps at conferences regarding Angular.

For example, if you look at the Angular 1.2 docs, look at the bottom of the page here and you'll see Google included their copyright on those docs https://code.angularjs.org/1.2.27/docs/guide/ie

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#436
post #124

A few years ago I was working at a startup and we had just started moving some stuff over from AWS to GCP. Things were going pretty good with GCP until one day they pulled the plug on everything with no explanation. It turns out that our company credit card details had been fraudulently used without our knowledge. The criminal had decided to use the card to buy Google Ad Words or something like that - this is the sam…

Never mess with Ad Words, that makes the Alphabet Beast angry every time. I've heard the exact same Ad Words story a few times now, someone used their CC or had at one point in the distant past used with Ad Words and the system flagged it. Everything gone with no warning, no explanation and no way to complain. Even getting to the front page of HN didn't work =)

Someone needs to pipe ad-words into DALLE and see what the ad-words-as-prompts-as-an-AI-service pumps out. Im out of credits at the moment, else I'd do it.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#437
post #409

Murfie died because of Google cloud costs. It's a bit of a long story, but from what I was able to piece together, a Google cloud service provider recommended some changes to the way the storage was done to save money. Mostly, because of those changes they got hit by a surprise $30,000 bill for data egress. It was basically the last straw for a company that wasn't profitable due mostly to cloud costs. It killed the c…

Does the customer data have a proper home yet or is it still sitting in the broken-into shipping containers?

According to the owner's twitter, he's been repeatedly getting robbed.

https://twitter.com/pontifier

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#438

A suggestion for engineers that lack business experience. As soon as your project grows beyond the non-hobby scale and starts making real money, invest some time to reach out to company providing the services that keep it alive and find an account manager that will take care of you. We're a Google customer and of course we had our fair share of issues (btw: it's the same with Azure, based on our experience), and we a…

Ironically enough, the last time I was "Courted" by google cloud to move some of our infra there (after it went public that we had raised a round), they were up front about how their "account management sucks" and that we'd have a better "customer experience" if we went through a reseller.

That...always sticks out to me, but, when dealing with the Google Education folks, there were some things that even resellers couldn't do, and I had to email people directly (via LinkedIn stalking) only to have my issue disappear a few days later with no reply.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#439
post #283

Earlier quoted context omitted.

Google tech is the best. Google invented k8s, their k8s cloud is the best in the world. There's nothing similar to GKE Autopilot. Similar things could be said about some other services. People want the best cloud and they hope that those issues won't affect them.

K8S runs well on all major cloud providers. You can run it on premise without licensing costs quite well (for example k3s, rancher, …). Or fully managed with Open Shift. AWS, Azure and GC are all on the same level. Any of them gives you „the best“ technology.

https://www.reddit.com/r/kubernetes/comments/wuboxb/best_kub...

This is recent topic and you can see that so many people prefer GKE. That's not my opinion.

Sure, any cloud works good enough.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#440

Earlier quoted context omitted.

after Azure sunk my employers project with no recourse or discussion allowed, I always advise everyone I meet to run far away from Azure if they value a reliable service.

"No discussion allowed" sounds impressive, almost like legal impetus was involved. What parts of the story can be shared?

I think when the parent commend said "no discussion allowed", it was in the context of "no recourse", not that they got some sort of a pseudo-gag-order.

I.e., basically meaning that Azure refused having any further discussions between then and the parent user, as attempts to resolve that specific case. Not that they dont't allow the parent comment user to talk about it with others on HN (or elsewhere).

Post reply on HN