There goes a 9
Incident Report: Railway Blocked by Google Cloud [resolved]
251–260 of 381 posts
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#252Does anyone know how this even happens inside the walls of google? Is it an automated process? How is such a (presumably) high revenue account just magically blocked without human intervention? I'm quite perplexed.
Yeah, compared to the AWS experience:
I had a toy Free Tier account that managed to overstep a limit one month and rack up $0.0038 in charges.
AWS hounded me about it for an entire year before finally putting the account on hold. Then kept at it for months more before finally deleting it.
It’s pike the paperboy from Better off Dead, if he were to continue delivering newspapers while hounding you for his two dollars.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#253Earlier quoted context omitted.
The instant cascading worldwide deletion upon closing or deleting a subscription sounds like a recipe for disaster. Why not mark it for deletion and delete say... a day or a week later?
Either mark-for-delete has the same impact as deleting in terms of shooting all the Cloud resources associated with the subscription, at which point the outage still happens but maybe the recovery is smoother or you've just delayed the inevitable by a week because no one will look at it unless there is actual impact.
You just turn it all off. So yes, the disruption is the same but restoral is much smoother. Much easier said than done - that has be baked into every service and there would certainly be a cost from it that would have to be passed along to everyone.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#254Lesson learned: don't rely on a single hyperscaler, even (or especially) as a startup.
I just... I don't really understand why startups even use AWS, GC, or any other cloud hosted software? Hetzner, etc. Are all extremely cheap, and honestly scale so well... Code nowadays is cheaper for configs, and having full control over your compute is... liberating.
A few years ago, when I was kinda active in the startup scene in my area, you have people selling access to cloud credits with penny-on-the-dollar price. The credits are given out liberally to big-corps, organization by AWS/GCP, through workshops, webinars, events. All in the hope of roping the departments into building MVPs, demos on AWS/GCP, but people also find a way to cheat on that system and make some quick bucks.
I know a startup of my acquaintances that have been running on AWS for 5 years straight without paying a single dollar to AWS. When the credits almost run out, they started to migrate their data over to another account with credit. That happened twice already.
It helps to have a portable, replicable IaC config. But also this is sustainable because they are a pretty small struggling shop. You will probably not be able to do this if you are trying to maintain more than 3 nines for an enterprise client.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#255Earlier quoted context omitted.
>That’s assuming your vendor was pentesting AWS systems. If you meant you hired a vendor to pentest your own systems on AWS, that’s of course a totally different matter. Sorry for being unclear, the vendor was attacking our organization only, and any other company was expressly forbidden in the contract. As I recall it was a fake SSO sign-in page to collect credentials that they would try and social engineer our empl…
At a minimum you should contact AWS before you launch a phishing page as a test that targets AWS customers.
I understood it as a phishing page imitating their own system, targeting their own employees. Nothing related to AWS, except for being hosted there.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#256Earlier quoted context omitted.
The instant cascading worldwide deletion upon closing or deleting a subscription sounds like a recipe for disaster. Why not mark it for deletion and delete say... a day or a week later?
It’s a good question. That said unless there are compliance or fallback concerns i would prefer a service that burns my data on departure.
No, that's the naive view
Because in case of a compromise/unauthorized access that's exactly what you don't want to happen
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#257Everyone is eager to point a finger at Google, but I've been a user of Railway for a while now, and I've seen enough nonsense to want to hear what GCP has to say about this before drawing any conclusions. Let's just say Railway has had problems like this before, and the way their team handles them does not inspire any confidence. Regardless of how it happened, for me, this is the straw that broke the camel's back.
Could you point us to any specific past instances? I'd be interested to read about them.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#258Earlier quoted context omitted.
On the other hand i can’t remember when there was a serious outage on GCP, unlike AWS/Azure who seem to go down catastrophically a couple of times per year.
Perhaps you don't notice GCP outages because so few companies rely on them?
Spotify, Ebay, Paypal, Apple, Walmart, Uber are huge users. Lots of other big named companies are big users that I don't think are public.
Then there's Anthropic...huge user.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#259When you signup for Railway, they have uncommon way of making sure you have read and understood their T&C regarding abuse of their systems, including crypto mining, etc.
My guess is that many are abusing their free tier, causing them trouble with their service providers.
I take no joy in seeing Railway take a hit like this, even as a competitor, but free compute attracts all sorts of strange users. We've been there and decided early on to avoid free compute even it costs us our top of the funnel.
Re: Incident Report: Railway Blocked by Google Cloud [resolved]
#260It has been 0 days since GCP has taken down a startup (again). You see this at least once a year. Never heard of this from AWS or Azure. In all seriousness, this is why we don't use them. They have the most ergonomic cloud of the big three, then absolutely murder it by having this kind of reputation.
AWS has throttled our service so badly that we couldn't operate. I was thinking of writing a blog post about how they stalled our growth for a month but it seems moot