Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io outage – resolved

status.flyio.net

Post reply on HN
Print view View original

111–120 of 287 posts

Re: Fly.io outage – resolved

#111

Post by fragmede »

SOLAR_FIELDS wrote: post #104
fragmede wrote: post #64

Earlier quoted context omitted.

Who's giving GitHub a pass on shitting the bed? They go down often enough that if you don't have an internal git server setup for your CICD to hit, that's on you.

My point is made by your very post - getting off GitHub onto alternatives is not seriously discussed as an option - instead it’s “well, why didn’t you prepare better to deal with your vendor’s poor ops story”

I wasn't going to bring up being on an internally hosted gitlab instead of github, but that would be the "not giving them a pass" part.

Re: Fly.io outage – resolved

#112

Post by [deleted] »

fragmede wrote: post #111
SOLAR_FIELDS wrote: post #104

Earlier quoted context omitted.

My point is made by your very post - getting off GitHub onto alternatives is not seriously discussed as an option - instead it’s “well, why didn’t you prepare better to deal with your vendor’s poor ops story”

I wasn't going to bring up being on an internally hosted gitlab instead of github, but that would be the "not giving them a pass" part.

[deleted]

Re: Fly.io outage – resolved

#113

Post by vrosas »

littlestymaar wrote: post #100
cj wrote: post #58

Earlier quoted context omitted.

I think what a lot of people fail to understand is that there are certain categories of apps that simply “can never go down” Examples include basically any PaaS, IaaS, or any company that provides a mission-critical service to another company (B2B SaaS). If you run a basic B2C CRUD app, maybe it’s not a big deal if you service goes down for 5 minutes. Unfortunately there are quite a few categories of companies where…

If your app cannot go down ever, then you cannot use a cloud provider either (because even AWS and Azure do fail sometime, just look up for “Azur down” on HN). But the truth is everybody can afford some level of outage, simply because nobody has the budget to provision an infra that can never fail.

I’ve seen a team try and be truly “multi-cloud” but then ended up with this Frankenstein architecture where instead of being able to weather one cloud going down, their app would die if _any_ cloud had an issue. It was also surprisingly hard to convince people it doesn’t matter how many globally distributed clusters you have if all your data is in us-east.

Re: Fly.io outage – resolved

#114

Post by ignoramous »

ploxiln wrote: post #107
xyst wrote: post #91

Recurring pattern I notice is outages tend to occur the week of major holidays in US. - MS 365/Teams/Exchange had a blip in the morning - Fly.io with complete outage - then a handful of sites and services impacted due to those outages Usually advocate against “change freezes” but I think a change freeze around major holidays makes sense. Give all teams a recharge/pause/whatever. Don’t put too much pressure on the B-s…

I think you can't avoid the fact that these holiday weeks are different from regular weeks. If you "change freeze" then you also freeze out the little fixes and perf tuning that usually happens across these systems, because they're not "critical". And then inevitably it turns out that there's a special marketing/product push, with special pricing logic that needs new code, and new UI widgets, causing a huge traffic/l…

Some shops conduct game days as the freeze approaches.

https://wa.aws.amazon.com/wellarchitected/2020-07-02T19-33-2... / https://archive.md/uaJlR

Re: Fly.io outage – resolved

#115

Post by __turbobrew__ »

akshayshah wrote: post #56

The series of outages early in 2023 also had some Corrosion-related pain: https://community.fly.io/t/reliability-its-not-great/11253

Seems like rolling their own datastore turned out to be a bad bet.

Im not super familiar with their constraints but scylladb can do eventual consistency and is generally quite flexible. CouchDB is also an option for multi-leader replication.

Re: Fly.io outage – resolved

#116

Post by paxys »

xyst wrote: post #91

Recurring pattern I notice is outages tend to occur the week of major holidays in US. - MS 365/Teams/Exchange had a blip in the morning - Fly.io with complete outage - then a handful of sites and services impacted due to those outages Usually advocate against “change freezes” but I think a change freeze around major holidays makes sense. Give all teams a recharge/pause/whatever. Don’t put too much pressure on the B-s…

Bad code rarely causes outages at this scale. The culprit is always configuration changes.

Sure you can try and reduce those as well during the holiday season, but what if a certificate has to be renewed? What if a critical security patch needs to be applied? What if a set of servers need to be reprovisioned? What if a hard disk is running out of space?

You cannot plan your way out of operational challenges, regardless of what time of year it is.

Re: Fly.io outage – resolved

#117

Post by verelo »

benreesman wrote: post #82
redslazer wrote: post #43

Earlier quoted context omitted.

The tech is impressive and the pricing is attractive which is why we use them. I just wish there was less black magic.

I don’t always agree with @tptacek on social/political issues, and I don’t always agree with @xe on the direction of Nix, but these are legends on the technical side of things. And they’re trying to build an equitable relationship between the user of cloud services and the provider, not fund a private space program. If I were in the market for cloud services I’d highly prize a long-term relationship on mutual benefit…

I want to believe, but in the meantime they’re killing the product I’ve been working hard to build trust with my own customers though. There is a limit to my idealism, and it’s well and truly in the past.

Re: Fly.io outage – resolved

#118

Post by [deleted] »

ojame wrote: post #55
SOLAR_FIELDS wrote: post #38

Earlier quoted context omitted.

I get this but I think if people can give GitHub a pass for shitting the bed every two weeks maybe Fly should get a bit of goodwill here. I am not affiliated with Fly at all but I do think that people should temper their expectations when even mega corp can’t get it right I guess the secret is to be the incumbent with no suitable replacement. Then you can be complete garbage in terms of reliability and everyone will…

The biggest difference is GitHub in your infrastructure is (nearly always) internal. Fly in your infrastructure is external. Users generally don't see when you have issues with GitHub, but they do generally see when you have issues with Fly. That's the core difference.

[deleted]

Re: Fly.io outage – resolved

#119

Post by sanswork »

nomilk wrote: post #49
benhoyt wrote: post #8

My fly.io-hosted website went down for 5 minutes (6 hours ago), but then came right back up, and has been up ever since. I use a free monitoring service that checks it every 5 minutes, so it's possible it missed another short bit of downtime. But fly.io has been pretty reliable overall for me!

Would be fascinated to see your data over a period of months. Application up time is flakey, but what was worse were fly deploys failing for no clear reason. Sometimes layers would just hang and eventually fail for no particular reason; I'd run the same command an hour or two later without any changes and it would just work as expected. I'd love to make a monitoring service to deploy a basic app (i.e. run the fly dep…

My downtimes from fly are pretty rare but generally global when they happen, in this outage we had no downtime but couldn't deploy for a few hours. I have issues with deploying about once per quarter(deploy most days across a few apps)

Re: Fly.io outage – resolved

#120

Post by [deleted] »

paxys wrote: post #116
xyst wrote: post #91

Recurring pattern I notice is outages tend to occur the week of major holidays in US. - MS 365/Teams/Exchange had a blip in the morning - Fly.io with complete outage - then a handful of sites and services impacted due to those outages Usually advocate against “change freezes” but I think a change freeze around major holidays makes sense. Give all teams a recharge/pause/whatever. Don’t put too much pressure on the B-s…

Bad code rarely causes outages at this scale. The culprit is always configuration changes. Sure you can try and reduce those as well during the holiday season, but what if a certificate has to be renewed? What if a critical security patch needs to be applied? What if a set of servers need to be reprovisioned? What if a hard disk is running out of space? You cannot plan your way out of operational challenges, regardle…

[deleted]
Post reply on HN