Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Railway (PaaS) global outage

status.railway.com

Post reply on HN
Print view View original

51–60 of 71 posts

Re: Railway (PaaS) global outage

#51

Post by czhu12 »

Onavo wrote: post #44
czhu12 wrote: post #24

Earlier quoted context omitted.

Been building an open source version of railway at https://canine.sh . Offers all the same features without the potential of a vendor lock-in / price gouging.

The docs seem to be non existent. Is the canine yaml documented? You want docs like this: https://coolify.io/docs/applications/ci-cd/github/setup-app https://coolify.io/docs/applications/build-packs/dockerfile https://coolify.io/docs/applications/build-packs/overview Plenty of screenshots and exact step by step instructions. Throwing an "example git repo" with no documentation won't get you any users. Put your shoes…

Yeah working through documentation still. The goal isn’t so much to replace coolify. Mostly born out of my last start up that ran a $20M business, 15 engineers, with about 300-1000qps at peak, with fairly complex query patterns.

I think the single VPS model is just too hard to get working right at that scale.

I think north flank / enterprise applications, would be a better comparison of what canine is trying to do, rather than coolify / indie hackers. The goal is not take away kubernetes, but to simplify it massively for 90% of use cases but still give full k8s api for any more advanced features

Re: Railway (PaaS) global outage

#52

Post by caseyohara »

lbrito wrote: post #50
colesantiago wrote: post #8

What is this 'railway'? I am assuming that a domain like railway.com should be about trains. Why does every tech company have to name themselves as a one word .com website and what they do is unrelated and vague to their own name? Does every tech company think they are Apple and have to register every word in the dictionary and redefine it as a technology company? Really bad name for a company.

A lot of companies have been doing that for a long time Lotus Jaguar Caterpillar Shell its a human thing

Shell was originally very literal though. They sold seashells.

> The "Shell" Transport and Trading Company (the quotation marks were part of the legal name) was a British company, founded in 1897 by Marcus Samuel, 1st Viscount Bearsted, and his brother Samuel Samuel. Their father had owned an antique company in Houndsditch, London, which expanded in 1833 to import and sell seashells, after which the company "Shell" took its name.

https://en.wikipedia.org/wiki/Shell_plc

Re: Railway (PaaS) global outage

#53

Post by port3000 »

justjake wrote: post #33

Hello! Railway founder here We'll have a post mortem for this one as we always write post mortems for anything that affects users Our initial investigation reveals this affects Apologies from myself + the Team. Any amount of downtime is completely unacceptable You may monitor this incident here: https://status.railway.com/cmli5y9xt056zsdts5ngslbmp

Second complete outage on railway in 2 months for us (there was also a total outage on December 16th), and many issues with stuck builds and other minor issues in the months before that.

Looking to move. It's a bit of hassle to setup coolify and Hetzner but I have lost all trust.

Re: Railway (PaaS) global outage

#54

Post by iJohnDoe »

justjake wrote: post #33

Hello! Railway founder here We'll have a post mortem for this one as we always write post mortems for anything that affects users Our initial investigation reveals this affects Apologies from myself + the Team. Any amount of downtime is completely unacceptable You may monitor this incident here: https://status.railway.com/cmli5y9xt056zsdts5ngslbmp

Many questions on their forum are similar to our situation. People wondering if they should restart their containers to get things working again. Worried about if they should do anything, risk losing data if they do anything, or just give everything more time.

Re: Railway (PaaS) global outage

#55

Post by linhns »

zachrip wrote: post #16
searls wrote: post #12

lol, just yesterday a friend asked me if he should move his business to Railway from Heroku. Welp.

I've been using railway a while now, and I've basically never paid them but I would. It's even better than heroku. Super easy to use.

Their 5$ monthly has been far more than enough for me to host my demos.

Re: Railway (PaaS) global outage

#56

Post by vintagedave »

justjake wrote: post #48
vintagedave wrote: post #38

Earlier quoted context omitted.

Hi Jake. Appreciate your presence here on HN. This affected a seemingly random set of services across three of my accounts (pro and hobby, depending on if this is for work or just myself.) That ranges from Wordpress to static site hosting to a custom Python server. All of the deployments showed as Online, even after receiving a SIGTERM. While 3% is 'good', that's an awfully wide range of things across multiple accoun…

Totally! People who see the impact will likely see more impacted than say, 3% of their services. Not all disruption created equal. We rolled out a change to update our fraud model, and that uses workload fingerprinting Since, in all likelyhood, your projects are similarly structured, there will be more impacted workloads if the shape of your workloads was in the "false positive" set Will have more information soon bu…

> We rolled out a change to update our fraud model, and that uses workload fingerprinting

> Since, in all likelyhood, your projects are similarly structured...

Thanks for the info. For what it's worth and to inform your retrospective, this included:

* A Wordpress frontend, with just a few posts, minimal traffic -- but one that had been posted to LinkedIn yesterday

* A Docusaurus-generated static site. Completely static.

* A Python server where workload would show OpenAI API usage, with consistent behavioural patterns for at least two months (and, I am strongly skeptical would have different patterns to any hosted service that calls OpenAI.)

These all seem pretty different to me. Some that _are_ similarly structured (eg a second Python OpenAI-using server) were not killed.

Some things come to mind for your post-mortem:

* If 3% of your services were affected, does that match your expected fraud rate? That is an awful lot of customers to take down in one go, and you'd want to be very accurate in your modeling. I can't see how you'd plan to kill that many without false positives and negative media.

* I'm speaking only for myself but I cannot understand what these three services have in common, nor how at least 2/3 of them (Wordpress, static HTML) could seem anything other than completely normal.

* How or why were customers not notified? I have used services before where if something seemed dodgy they would proactively reach out and say 'tell us if it's legit or in 24 hours it will be shut down' or for something truly bad, eg massive CPU usage affecting other services, they'd kill it right away but would _tell you_. Invisible SIGTERMS to random containers we find out about the hard way seems the exact opposite of sensible handling of supposedly questionable clients.

Re: Railway (PaaS) global outage

#57

Post by dabbz »

mstank wrote: post #21
huydotnet wrote: post #14

Joke about train line aside, I think Railway fits right in the spot that Heroku left. They have a nice UI, support deploy any kind of backend-involved apps as long as it can be built into a docker container. While many PaaS out there seems to prioritize frontend only apps. And they have a free plan, so people can just quickly deploy some POC before decide if it's good to move on. Anyone know if there is any other Paa…

VPS + Dokploy gives you just as much functionality with an additional performance boost. Hostinger has great prices and a one-click setup. Good for dozens of small projects.

+1 for dokploy, it's very flexible and allows me to setup my sites how I need. Especially as it concerns to the way I setup a static landing page, then /app goes to the react app. And /auth goes to a separate auth service, etc.

Re: Railway (PaaS) global outage

#58

Post by nhubbard »

esseph wrote: post #18
Gravityloss wrote: post #3

Earlier quoted context omitted.

Indeed! Remote bricking of trains is perhaps a thing: https://www.thedrive.com/news/hackers-beat-anti-repair-softw...

It's also mandated by Congress in the US, it's called PTC. (Remote control)

This wasn't PTC. It was repair lockouts instituted by the manufacturer of the trains based on a GPS geofencing beacon.

Re: Railway (PaaS) global outage

#59

Post by ndneighbor »

iJohnDoe wrote: post #34

Affected by the outage since about 6:15 AM PT this morning. We're still down as of 9:00 AM PT. Our existing containers were in a failure state and are now are in a partial failure state. Containers are running, but underlying storage/database is offline. Many questions on their forum are similar to our situation. People wondering if they should restart their containers to get things working again. Worried about if th…

(Angelo from Railway here)

Heard. Being transparent, usually the delay on ack is us trying to determine and correlate the issue. We have a post mortem going out but we note that first report was in our system 10 minutes before it was acked, to which the platform team was trying to see which layer the impact was at.

That said, this is maybe concern #1 of the support team. Where we want the delta between report and customer outage detected to be as small as possible. The way it usually works is that we have the platform alarms and pages go first, and then the platform engineer usually will page a support eng. to run communications.

Usually the priority is to have the platform engineer focus on triaging the issue and then offload the workload to our support team so that we can accurately state what is going on. We have a new comms clustering system that rolling out so that if we get 5 reports with the similar content, it pages up to the support team as well. (We will roll this out after we communicated with affected customers first.)

Re: Railway (PaaS) global outage

#60

Post by ndneighbor »

vintagedave wrote: post #25

Multiple services are receiving SIGTERM or shutdown signals. See dozens of support messages here: https://station.railway.com/questions/services-down-799f7bc1 Here's a sample log entry: > 2026-02-11T14:35:11.916787622Z [err] 2026/02/11 14:35:03 [notice] 1#1: signal 15 (SIGTERM) received, exiting I've had about one third of my Railway services affected. I had no notification from Railway, and logging in showed each af…

Hey there Dave, Angelo from Railway here-

First off, super duper sorry. It's sometimes a good/bad thing if I can remember someones handle. ...and I specifically remember the support thread where we did have an outage before your demo :| - the number one goal for us is to deliver a great product. Number two is that we should never embarrass a user, outages do exactly that.

We just wrapped up the post mortem and that'll be published soon where it explains why the dashboard was reporting the state of the application incorrectly and would be more than happy to credit you for the impact to keep your business. That said, totally understand if two is way too much impact for your services.

Post reply on HN