Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io outage – resolved

status.flyio.net

Post reply on HN
Print view View original

241–250 of 287 posts

Re: Fly.io outage – resolved

#241

Post by tptacek »

pajeetz wrote: post #235
zackify wrote: post #198

Earlier quoted context omitted.

The reliability is very very bad. It was really insane that 2 times in the past few months the main dashboard was down as I’m demoing something. Not to mention the deploy outages and almost daily some random thing was unavailable or delayed. I had to leave a few months ago after the price raises and how many times my boss saw some issue in the project I had with them. They also deprecated and removed their sqlite bac…

theres just so many anecdotes/nightmare stories from people using fly.io here much more than the ones linked by GP expect to see more of these "post-mortem apologies" from fly.io in the future because it won't be the last

You're right. It won't. Nobody could claim otherwise.

Re: Fly.io outage – resolved

#242

Post by cxr »

pc86 wrote: post #213
huijzer wrote: post #170

Earlier quoted context omitted.

For experiments and hobby projects the value proposition is amazing. Where else can you spin up an independent instance for $1.94 per month?* *Note this is for an instance with only 256MB RAM ( https://fly.io/docs/about/pricing/ ), but it's definitely possible to run non-trivial projects on that. Rust-based web servers like Rocket require only about 10MB RAM. Basic PHP servers should also fit from what I can find.

Maybe if you're limiting yourself to AWS-wrapper cloud companies. What good is a $2/mo cloud instance if it's down multiple times a month? Just get a $5/mo VPS instead if you're really concerned about a few dollars a month.

> What good is a $2/mo cloud instance if it's down multiple times a month?

The perverse irony is that the most common reason cited by cloud providers for not letting people set a hard cap on charges is an insistence that surely the last thing you want in the world is for your service to be taken offline, even if it does means avoiding a $1k–$100k bill at the end of the month.

Re: Fly.io outage – resolved

#243

Post by tptacek »

pajeetz wrote: post #233
cryptos wrote: post #162

Fly.io seems to be a bit of a mixed bag: https://news.ycombinator.com/item?id=41917436 https://news.ycombinator.com/item?id=35044516 https://news.ycombinator.com/item?id=34742946 https://news.ycombinator.com/item?id=34229751 If a cloud platform doesn't really provide reliability, I'd say it's probably not worth it. You could better just rent a (virtual) server and save the cloud tax.

fly.io has a very bad reputation for reliability there doesn't seem to be any damage control beyond hackernews and even here the consensus seems to be "dont run anything mission critical on fly.io or expect data redundancy" in fact, you can almost get the same thing fly.io does by running firecracker on your own bare metal servers and cheaper too. I'm afraid the public sentiment towards fly.io has been tainted for go…

This is the second place you've offered this sentiment. Was it your expectation that we were going to hit some point, sometime in the near future, where we weren't going to have deployment-blocking outages? I'd like to better understand your premise. If it's "I can get more reliability by deploying on a hyperscaler cloud", who ever told you otherwise?

Re: Fly.io outage – resolved

#244

Post by tptacek »

paxys wrote: post #116
xyst wrote: post #91

Recurring pattern I notice is outages tend to occur the week of major holidays in US. - MS 365/Teams/Exchange had a blip in the morning - Fly.io with complete outage - then a handful of sites and services impacted due to those outages Usually advocate against “change freezes” but I think a change freeze around major holidays makes sense. Give all teams a recharge/pause/whatever. Don’t put too much pressure on the B-s…

Bad code rarely causes outages at this scale. The culprit is always configuration changes. Sure you can try and reduce those as well during the holiday season, but what if a certificate has to be renewed? What if a critical security patch needs to be applied? What if a set of servers need to be reprovisioned? What if a hard disk is running out of space? You cannot plan your way out of operational challenges, regardle…

We'll have a postmortem in next week's infra log update, but here it was a particularly ambitious customer app pushing our state sync service into a corner case; it's one we knew about, but the solution (federating regional state sharing clusters rather than running one globally) is taking time to roll out.

Re: Fly.io outage – resolved

#245

Post by tptacek »

benreesman wrote: post #82
redslazer wrote: post #43

Earlier quoted context omitted.

The tech is impressive and the pricing is attractive which is why we use them. I just wish there was less black magic.

I don’t always agree with @tptacek on social/political issues, and I don’t always agree with @xe on the direction of Nix, but these are legends on the technical side of things. And they’re trying to build an equitable relationship between the user of cloud services and the provider, not fund a private space program. If I were in the market for cloud services I’d highly prize a long-term relationship on mutual benefit…

I'm several steps removed from day-to-day engineering at this point; the team working on this is much better than I am. It's just a very hard problem; biting it off is something you can certainly blame me for, though.

(Also: not a legend, just loud.)

Re: Fly.io outage – resolved

#246

Post by tptacek »

foldr wrote: post #173
benreesman wrote: post #82

Earlier quoted context omitted.

I don’t always agree with @tptacek on social/political issues, and I don’t always agree with @xe on the direction of Nix, but these are legends on the technical side of things. And they’re trying to build an equitable relationship between the user of cloud services and the provider, not fund a private space program. If I were in the market for cloud services I’d highly prize a long-term relationship on mutual benefit…

I suspect that making a cloud service provider run reliably requires tons of grunt work more than it requires technical heroism from a small number of highly talented individuals.

Yes.

Re: Fly.io outage – resolved

#247

Post by tptacek »

Aeolun wrote: post #163
veggieWHITES wrote: post #16

I was considering these guys the other day until I saw their pricing page: https://fly.io/pricing/ (There's not a single price on there, why even create the page?)

OMG, that's hilarious. I use them, and I know what my prices are, but I'd never noticed that the page called pricing doesn't actually have any.

We've always had public pricing; you can't do a metered cloud provider without a rate sheet. But it's been part of our product documentation, rather than the front page of the website, until recently; there's a whole saga behind it, which gets into whether we offer "plans" or not, how support works, all that jazz, all of which kept us from putting together a marketing pricing page.

Re: Fly.io outage – resolved

#248

Post by justjake »

ignoramous wrote: post #193
neya wrote: post #151

Personal experience between Fly.io and Railway.com - Railway wins for me hands down. I have used both and the Railways support is stellar too, in comparison. Fly.io never responded to my query about data deletion till date. Despite emailing on their support email. I have had my Railway app online till date without any major downtimes too. I recommend anyone looking for a decent replacement to try them.

Fly builds on their own hardware. Is Railway doing the same? If not, that'd explain some of why Railway has relatively less number of outages (they're engineering fewer things). I understand that end-users want reliability (and Fly gets a bad rep despite pretty significant investment on this front in the past 2 years), but such outages aren't exclusive to one provider & not the other. Building cloud infra is no one's…

We run on Google Cloud, AWS, and our own hardware now since middle of this year :)

https://railway.com/changelog/2024-09-20-railway-metal-beta#...

Re: Fly.io outage – resolved

#249

Post by justjake »

andai wrote: post #159
neya wrote: post #151

Personal experience between Fly.io and Railway.com - Railway wins for me hands down. I have used both and the Railways support is stellar too, in comparison. Fly.io never responded to my query about data deletion till date. Despite emailing on their support email. I have had my Railway app online till date without any major downtimes too. I recommend anyone looking for a decent replacement to try them.

I've used Railway control panel maybe a total of 10 times in my life and half the time it was having weird issues. Control panel UI not loading or not working, actions failing, deploys randomly failing... I love the idea but in practice it's not something I'd want to use for anything serious.

While we've always aimed for great reliability on compute, the dashboard reliability wasn't very good at the start of the year.

We ack'd this and then pretty heavily to making it stellar, so if you're still having issues please let us know (that should not be the case)

Best, Jake from Railway

Re: Fly.io outage – resolved

#250

Post by justjake »

punkpeye wrote: post #152
neya wrote: post #151

Personal experience between Fly.io and Railway.com - Railway wins for me hands down. I have used both and the Railways support is stellar too, in comparison. Fly.io never responded to my query about data deletion till date. Despite emailing on their support email. I have had my Railway app online till date without any major downtimes too. I recommend anyone looking for a decent replacement to try them.

How does it compare in terms of price?

We actually only charge you for what you use. As a result people often see 30%+ savings when moving stuff over from other providers (especially Heroku)

https://railway.com/pricing

Post reply on HN