Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Shouldn't Google need to give a public statement about Railway incident?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 116 posts

Re: Ask HN: Shouldn't Google need to give a public statement about Railway incident?

#51

Post by esseph »

r_lee wrote: post #15
whh wrote: post #8

Earlier quoted context omitted.

I've directly asked our account manager about it. It's pretty scary that we don't know what automated mechanisms could just cut us off.

I really think Google underestimates the damage they've done to their reputation with these. These incidents are rare, but they're common enough where you can't trust them reliably anymore, and if that's the case, why would you pick them over other vendors? it's not like they're the only provider, or even the #1

> why would you pick them over other vendors?

Their security track record is pretty exceptional compared to the other two.

Re: Ask HN: Shouldn't Google need to give a public statement about Railway incident?

#52

Post by zarzavat »

GCP is basically just a toy. If you're hosting something important there you should probably not do that. It works as a Plan B, but by no means should Google be your Plan A.

I don't know what happened in this case, there's a chance it wasn't Google's fault but it doesn't matter, Google already lost all benefit of the doubt long ago.

Re: Ask HN: Shouldn't Google need to give a public statement about Railway incident?

#53

Post by ceejayoz »

r_lee wrote: post #42
ceejayoz wrote: post #36

Earlier quoted context omitted.

> if it wasn't something specific to their setup They're a web host; it could be any number of plausible mundane things that triggered automated action. This is a big recurring problem for any shared hosting provider.

a huge account like theirs should not be subject to automated actions like that. an entire gcp project deleted along with its persistent disks. how does that make any sense? nobody thought to call them or anything

> a huge account like theirs should not be subject to automated actions like that.

No matter how damaging the behavior?

> an entire gcp project deleted along with its persistent disks.

Railway doesn't say that - "persistent disks inaccessible", followed by "persistent disks restored to ready state". It was a suspension, not a wipe.

Re: Ask HN: Shouldn't Google need to give a public statement about Railway incident?

#54

Post by noahlt »

ZiiS wrote: post #17

I really don't see how they can. The business and usage details of their clients are confidential. We have their word that ToS where violated, I don't really think they should say more. This needs to go to arbitration.

Is there a statement somewhere from Google saying the ToS was violated? I hadn't seen that and can't find one right now.

Re: Ask HN: Shouldn't Google need to give a public statement about Railway incident?

#55

Post by Salgat »

zarzavat wrote: post #52

GCP is basically just a toy. If you're hosting something important there you should probably not do that. It works as a Plan B, but by no means should Google be your Plan A. I don't know what happened in this case, there's a chance it wasn't Google's fault but it doesn't matter, Google already lost all benefit of the doubt long ago.

Yeah this is the main argument I have for AWS. Is it the best tech-wise? Arguably not, but they have real support you can contact on a moment's notice, they'll even reach out ahead of time if they suspect something is seriously wrong.

Re: Ask HN: Shouldn't Google need to give a public statement about Railway incident?

#56

Post by pugworthy »

> Google Cloud placed Railway’s production account into a suspended status incorrectly, as part of an automated action. This action extended to many accounts within Google Cloud. As this was a platform-wide action, there was no proactive outreach to individual customers prior to the restriction.

I'm interpreting that bit from Railway's blog to mean it wasn't just them that was impacted.

Re: Ask HN: Shouldn't Google need to give a public statement about Railway incident?

#57

Post by antasvara »

graemep wrote: post #31
r_lee wrote: post #11

I think so. I'm a GCP user and I'm afraid of hosting workloads there now. I've heard too many nightmare stories, and I thought Google would be proper and thus not be infested with these kind of problems that cheaper providers are known for. Maybe AWS is the only player in town now? I don't know. Google doesn't instill confidence with these incidents, same with those cases of insurmountable bills caused by simple mist…

> I thought Google would be proper and thus not be infested with these kind of problems that cheaper providers are known for. These sorts of things have happened before with Google and the other expensive providers. Are cheaper providers known for doing this? I would have thought they would be less lively to, as they are smaller and therefore every customer is relatively more important to them, and they are therefore…

> Are cheaper providers known for doing this? I would have thought they would be less lively to, as they are smaller and therefore every customer is relatively more important to them, and they are therefore more likely to check before turning services off.

In my experience, the reason they're cheaper is because they offer fewer features (cut down on ongoing expenses) and because they aggressively enforce TOS (the margins are thinner, so you're less able to afford people using more resources than allocated).

Re: Ask HN: Shouldn't Google need to give a public statement about Railway incident?

#58

Post by danesparza »

tomComb wrote: post #35

I can't believe that readers of HN actually think that that is how it does or should work. Google/GCP can only make very general statements and in this case we want more than that. They need to tell Railway and Railway needs to tell us, or Railway can tell us that Google is refusing to tell them. Either way, we need to hear about this from Railway.

They did:

"We take full responsibility for the architectural decisions that allowed a single upstream provider action to cascade into a platform-wide outage, and detail below what happened, how we recovered, and the changes we are making to prevent this from happening again."

My guess is they will be switching away from Google Cloud. Because anything else would be nuts.

Re: Ask HN: Shouldn't Google need to give a public statement about Railway incident?

#59

Post by oneplane »

dogscatstrees wrote: post #22
r_lee wrote: post #11

I think so. I'm a GCP user and I'm afraid of hosting workloads there now. I've heard too many nightmare stories, and I thought Google would be proper and thus not be infested with these kind of problems that cheaper providers are known for. Maybe AWS is the only player in town now? I don't know. Google doesn't instill confidence with these incidents, same with those cases of insurmountable bills caused by simple mist…

What about Azure?

I'd rather go back to bare metal than use Azure.

Re: Ask HN: Shouldn't Google need to give a public statement about Railway incident?

#60

Post by continueops_com »

This makes me so mad, Google is not the first i've seen doing this (AWS, Upwork, Twilio) - and the cheek to say precisely nothing about why. If you're a buyer of cloud services right, i bet old-school on-prem is looking as tempting as an actress to a teenager rn.

Every regulated firm running on GCP is going to spend Monday explaining to their board how their resilience plan accounts for a hyperscaler that operates this opaquely. The compliance paperwork is the easy bit - the honest answer is we trusted that a hyperscaler would behave like a utility and they didn't.

So yes - they owe a statement. The whole point of paying hyperscaler prices is the assumption you won't wake up suspended with no explanation.

Post reply on HN