181–190 of 279 posts
Google has a culture problem. This is not something that can change easily nor will it change when it’s not recognized as being an issue within their organization. Between my peer c-suites, the conversation is that GCP cannot even be in the consideration set until such a time as a several-year period has elapsed without this kind of incident.
Google, the new Microsoft!
Amazon played AWS from day 1 as if they were the runner-up (and in a sense they were), and while it does look like it's day 2 there, they are not letting the momentum down Microsoft might have technical warts but commercially they are strong and Azure is a lot of times bundled with other services and you know you can get someone on the phone if needed Google has... ?
> Google has... ?
At least that's my understanding from discussing with people praising GCP.
Let's say you want a big cloud provider, but you don't want Azure because of Microslop's old and recent history, and you don't want AWS because it's the default cloud provider.
You're left with GCP. And many people are stuck in the 00's, and still believe Google is the cool kid crushing the boring old corporations.
The interesting and yet-to-be-explained part is why google flagged the account? Put all the timestamps you want in the post mortem about what you observed, but you haven't addressed the root cause. The "this doesn't make sense" part of the story likely has a real explanation that nobody wants to reveal yet.
The moment GCP shut off without any forewarning, its done deal, no need to ask any further questions.
"Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for secondary/failover." That's pretty clear. Google can no longer be trusted as a B2B service provider.
Railway don't have a great reputation for building scalable systems (effects of vibe coding?). It's worth waiting for Google's response before jumping to conclusions. They can move to Azure/AWS/own datacenter, but there's a good chance this will repeat in a few months.
If you're picking them instead of the underlying cloud provider, but you want all the knows and dials the underlying provider has, you've made the wrong choice.
Earlier quoted context omitted.
I don't think you're typically told why for these things, and it's mostly automated from what I can tell. The automated systems make mistakes but more importantly they're completely opaque. Nobody, not even Google, knows how they work exactly.
Google should know why a human accepted the automated suggestion, or if and why there wasn't any human oversight in the first place.
This should be a warning to anyone running GCP. They suspend accounts left right and centre without even thinking about what they're doing. It seems like they use Gemini 3.1 Pro to run their production decisions. TK has a history of absolutely destroying the culture of the place like in OCI and has done something similar in GCP from what I've heard. GCP and Google are completely different entities with how they work.…
And this is Railway, a big enough name to top the HN main page and presumably find someone from Google to intervene at some point. I would have zero recourse if it was some little product that I built.
May 19, 22:10 UTC - Our automated monitoring detected API health check failures and paged our on-calls, who started investigating the issue. May 19, 22:11 UTC - Dashboard returning 503 errors. Users unable to log in. May 19, 22:19 UTC - Root cause identified: Google Cloud Platform has suspended Railway's production account. May 19, 22:22 UTC - P0 ticket filed with Google Cloud. Railway's GCP account manager engaged directly. May 19, 22:29 UTC - Incident declared. May 19, 22:29 UTC - GCP account access restored. All compute instances remained stopped and persistent disks inaccessible.
Earlier quoted context omitted.
What company?
In May 2024, Google Cloud Platform (GCP) accidentally deleted the private cloud account and all backups belonging to UniSuper, an Australian pension fund managing over $125 billion.
Earlier quoted context omitted.
Yeah, people loose their business because a kid is logged in on their iPad, gets their google account suspended, and google knows it's the same household as the parent, and everything gets shut down
> google knows it's the same household as the parent, Nearly all these linkages are due to people sharing recovery email addresses and phone numbers. Don't do that.
This should be a warning to anyone running GCP. They suspend accounts left right and centre without even thinking about what they're doing. It seems like they use Gemini 3.1 Pro to run their production decisions. TK has a history of absolutely destroying the culture of the place like in OCI and has done something similar in GCP from what I've heard. GCP and Google are completely different entities with how they work.…