Incident March 30th, 2026 – Accidental CDN Caching • Hacker News Board

Re: Incident March 30th, 2026 – Accidental CDN Caching

#11

Post by antics » Tue, Mar 31, 2026, 4:01 AM UTC

slopinthebag wrote: post #9

codechicago277 wrote: post #8

Earlier quoted context omitted.

The problem is that these visible errors make us wonder what other errors in the post are less visible. Fixing them doesn’t fix the process that led to them.

I'm pretty sure it's AI. https://x.com/JustJake/status/2007730898192744751 I wouldn't be surprised if most of Railway's infra is running on Claude at this point.

The CEO says it's not: https://x.com/JustJake/status/2038799619640250864

A lot of people are confident in enough in their ability to spot AI infra that they are willing to dismiss a firsthand source on this, and I admit I have no idea why. There isn't any upside to making this claim, and anyway, I assure you that people need no help at all from AI to make these kinds of mistakes.

Re: Incident March 30th, 2026 – Accidental CDN Caching

#12

Post by muragekibicho » Tue, Mar 31, 2026, 4:06 AM UTC

Does Stripe use Railway? The dashboard was down today and this is the only incident report I've encountered and the timeline matches Stripe's downtime.

Re: Incident March 30th, 2026 – Accidental CDN Caching

#13

Post by rileymichael » Tue, Mar 31, 2026, 4:13 AM UTC

pretty hard to find this on their blog, looks like incidents are tucked away at the bottom. an issue of this size deserve a higher spot.

(also looks like two versions of the 'postmortem' are published at https://blog.railway.com/engineering)

Re: Incident March 30th, 2026 – Accidental CDN Caching

#14

Post by stingraycharles » Tue, Mar 31, 2026, 4:16 AM UTC

slopinthebag wrote: post #9

codechicago277 wrote: post #8

Earlier quoted context omitted.

The problem is that these visible errors make us wonder what other errors in the post are less visible. Fixing them doesn’t fix the process that led to them.

I'm pretty sure it's AI. https://x.com/JustJake/status/2007730898192744751 I wouldn't be surprised if most of Railway's infra is running on Claude at this point.

It's fine they use AI, it's not fine they don't proofread things.

Re: Incident March 30th, 2026 – Accidental CDN Caching

#15

Post by kay_o » Tue, Mar 31, 2026, 4:50 AM UTC

sublinear wrote: post #3

I'm curious if having unique URLs per user session would mitigate this. I think that's already best practice in most API designs anyway?

Probably.

No, it isn't. Ive not seen this in an API ever and only in webapps ?phpsessid= back in childhood

Re: Incident March 30th, 2026 – Accidental CDN Caching

#16

Post by heyethan » Tue, Mar 31, 2026, 5:47 AM UTC

[flagged]

Re: Incident March 30th, 2026 – Accidental CDN Caching

#17

Post by slopinthebag » Tue, Mar 31, 2026, 5:49 AM UTC

antics wrote: post #11

slopinthebag wrote: post #9

Earlier quoted context omitted.

I'm pretty sure it's AI. https://x.com/JustJake/status/2007730898192744751 I wouldn't be surprised if most of Railway's infra is running on Claude at this point.

The CEO says it's not: https://x.com/JustJake/status/2038799619640250864 A lot of people are confident in enough in their ability to spot AI infra that they are willing to dismiss a firsthand source on this, and I admit I have no idea why. There isn't any upside to making this claim, and anyway, I assure you that people need no help at all from AI to make these kinds of mistakes.

Their reply doesn't make much sense, they're supposedly soc2 compliant. How are they compliant but letting a single engineer push out a change like that?

I'm sure Claude didn't literally ship the feature itself with no oversight, but I also find it hard to believe that their approach to adopting AI didn't factor in at all. Even just like, the mental overhead of moving faster and adopting AI code with less stringent review leading to an increase in codebase complexity could cause it. Couple that with an AI hallucinating an answer to the engineer who shipped this change, I'm not sure why people are so quick to discount this as a potential source of the issue. Surely none of us want our infra to become less secure and reliable, and so part of preventing that from happening is being honest about the challenges of integrating AI into our development processes.

Re: Incident March 30th, 2026 – Accidental CDN Caching

#18

Post by algolint » Tue, Mar 31, 2026, 6:22 AM UTC

[flagged]

Re: Incident March 30th, 2026 – Accidental CDN Caching

#19

Post by antics » Tue, Mar 31, 2026, 6:43 AM UTC

slopinthebag wrote: post #17

antics wrote: post #11

Earlier quoted context omitted.

The CEO says it's not: https://x.com/JustJake/status/2038799619640250864 A lot of people are confident in enough in their ability to spot AI infra that they are willing to dismiss a firsthand source on this, and I admit I have no idea why. There isn't any upside to making this claim, and anyway, I assure you that people need no help at all from AI to make these kinds of mistakes.

Their reply doesn't make much sense, they're supposedly soc2 compliant. How are they compliant but letting a single engineer push out a change like that? I'm sure Claude didn't literally ship the feature itself with no oversight, but I also find it hard to believe that their approach to adopting AI didn't factor in at all. Even just like, the mental overhead of moving faster and adopting AI code with less stringent r…

> I'm not sure why people are so quick to discount [AI] as a potential source of the issue.

Because (per the link above) the CEO said that (1) it was their fault, and (2) it had nothing to do with AI.

I understand that on this forum statements like this are inevitably greeted with some amount of skepticism, but right now I'm seeing no particular reason to disbelieve Jake, and the reason that "if they did use AI they'd deny it" should frankly not be considered good enough to fly around here. Like probably everyone in this comment section I'm open to evidence that they used AI to slop-incident themselves, but until we can reach that standard let's please calm down and focus on what we actually know to be true.

Re: Incident March 30th, 2026 – Accidental CDN Caching

#20

Post by slopinthebag » Tue, Mar 31, 2026, 7:36 AM UTC

antics wrote: post #19

slopinthebag wrote: post #17

Earlier quoted context omitted.

Their reply doesn't make much sense, they're supposedly soc2 compliant. How are they compliant but letting a single engineer push out a change like that? I'm sure Claude didn't literally ship the feature itself with no oversight, but I also find it hard to believe that their approach to adopting AI didn't factor in at all. Even just like, the mental overhead of moving faster and adopting AI code with less stringent r…

> I'm not sure why people are so quick to discount [AI] as a potential source of the issue. Because (per the link above) the CEO said that (1) it was their fault, and (2) it had nothing to do with AI. I understand that on this forum statements like this are inevitably greeted with some amount of skepticism, but right now I'm seeing no particular reason to disbelieve Jake, and the reason that "if they did use AI they'…

Come on man, their CEO is a massive vibe coding proponent and his company spent $300,000 on Claude this month. But yeah, I'm sure Claude had nothing to do with any of it. I bet they don't use it to write any code.

https://xcancel.com/JustJake/status/2030063630709096483#m