Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io outage – resolved

status.flyio.net

Post reply on HN
Print view View original

131–140 of 287 posts

Re: Fly.io outage – resolved

#131

Post by jart »

ignoramous wrote: post #122
jart wrote: post #92

fly.io publishes their post-mortems here: https://fly.io/infra-log/ The last post-mortem they wrote is very interesting and full of details. Basically back in 2016 the heart or keystone component of fly.io production infrastructure was called consul, which is a highly secure TLS server that tracks shared state and it requires that both the server certificate and the client certificate be authenticated. Since it was c…

On that Consul outage, Fly Infra concludes, "The moral of the story is, no more half-measures." On their careers page [1], the Fly team goes, "We're not big believers in tech debt." As an outsider, reads like a cacophony of contradictions? [1] https://fly.io/docs/hiring/working/#we-re-ruthless-about-doi...

No one actually lives up to their principles, but it's still important that we have them.

If you actually do live up to yours, then you need to adopt better principles.

Re: Fly.io outage – resolved

#132

Post by subarctic »

vrosas wrote: post #106
xyst wrote: post #91

Recurring pattern I notice is outages tend to occur the week of major holidays in US. - MS 365/Teams/Exchange had a blip in the morning - Fly.io with complete outage - then a handful of sites and services impacted due to those outages Usually advocate against “change freezes” but I think a change freeze around major holidays makes sense. Give all teams a recharge/pause/whatever. Don’t put too much pressure on the B-s…

Then you just get devs rushing out changes before the freeze…

As a developer I don't see why I would rush out a change before the freeze when I could just wait until after. Maybe a stakeholder that really wants it would press for it to get out but personally I'd rather wait until after so I'm not fixing a bug during my holiday.

Re: Fly.io outage – resolved

#133

Post by xena »

benreesman wrote: post #130
xena wrote: post #126

Earlier quoted context omitted.

Xe here. As a sibling comment said, I didn't survive layoffs. If you're looking for someone like me, I'm on the market!

Hiring people is above my pay grade, but I can vouch to my lords and masters and anyone else who cares what I think that a legend is up for grabs. b7r6@b7r6.net

I'd email but I'm about to pass out in bed. Please see https://xeiaso.net/contact/ in case I don't get back to you in the morning.

Re: Fly.io outage – resolved

#134

Post by bobsyourbuncle »

paxys wrote: post #116
xyst wrote: post #91

Recurring pattern I notice is outages tend to occur the week of major holidays in US. - MS 365/Teams/Exchange had a blip in the morning - Fly.io with complete outage - then a handful of sites and services impacted due to those outages Usually advocate against “change freezes” but I think a change freeze around major holidays makes sense. Give all teams a recharge/pause/whatever. Don’t put too much pressure on the B-s…

Bad code rarely causes outages at this scale. The culprit is always configuration changes. Sure you can try and reduce those as well during the holiday season, but what if a certificate has to be renewed? What if a critical security patch needs to be applied? What if a set of servers need to be reprovisioned? What if a hard disk is running out of space? You cannot plan your way out of operational challenges, regardle…

This is a good observation. Do you have any resources I can read up on to make this safer?

Re: Fly.io outage – resolved

#135

Post by vrosas »

subarctic wrote: post #132
vrosas wrote: post #106

Earlier quoted context omitted.

Then you just get devs rushing out changes before the freeze…

As a developer I don't see why I would rush out a change before the freeze when I could just wait until after. Maybe a stakeholder that really wants it would press for it to get out but personally I'd rather wait until after so I'm not fixing a bug during my holiday.

Congrats on not working for the product team I work for

Re: Fly.io outage – resolved

#136

Post by fragmede »

aaomidi wrote: post #105
xyst wrote: post #91

Recurring pattern I notice is outages tend to occur the week of major holidays in US. - MS 365/Teams/Exchange had a blip in the morning - Fly.io with complete outage - then a handful of sites and services impacted due to those outages Usually advocate against “change freezes” but I think a change freeze around major holidays makes sense. Give all teams a recharge/pause/whatever. Don’t put too much pressure on the B-s…

What do "Freezes" mean? Like, do you stop renewing your certificates? Do you stop taking in security updates for your software? Sure maybe "unnecessary" changes, but the line gets very gray very fast.

Certs shouldn't still be done by hand that this point; if another heartbleed comes out in the next 7 days then the risk can be examined, escalated, and the CISO can overrule the freeze. If it's a patch for remote root via Bluetooth drivers on a server that has no Bluetooth hardware, it's gonna wait.

you're right that there's a grey line, but crossing that line involves waking up several people and the on call person makes a judgement call. if it's not important enough to wake up several people over, then things stay frozen.

Re: Fly.io outage – resolved

#137

Post by dizhn »

ilrwbwrkhv wrote: post #11
shubhamjain wrote: post #7

This is probably 5th or 6th major outage from Fly.io that I have personally seen. Pretty sure there were many others and some just went unnoticed. I recommended the service to a friend, and within two days he faced two outages. Fly.io seriously needs to get it together. Why it hasn’t happened yet is a mystery to me. They have a good product but stability needs to be an absolute top for a hosting service. Everything e…

Does anyone use them beyond the free tier? Same with Vercel for example.

Which company? GitHub? As far as I know fly.io does not have a free tier.

Re: Fly.io outage – resolved

#138

Post by nomilk »

sanswork wrote: post #119
nomilk wrote: post #49

Earlier quoted context omitted.

Would be fascinated to see your data over a period of months. Application up time is flakey, but what was worse were fly deploys failing for no clear reason. Sometimes layers would just hang and eventually fail for no particular reason; I'd run the same command an hour or two later without any changes and it would just work as expected. I'd love to make a monitoring service to deploy a basic app (i.e. run the fly dep…

My downtimes from fly are pretty rare but generally global when they happen, in this outage we had no downtime but couldn't deploy for a few hours. I have issues with deploying about once per quarter(deploy most days across a few apps)

If that’s the case I suspect fly is getting a lot more reliable. I stopped using them about a year ago so haven’t kept up on their reliability since. Glad to hear, it’s good for a competitive market to have many providers, and fly might have issues but hopefully has a bright future

Re: Fly.io outage – resolved

#139

Post by punkpeye »

stoicjumbotron wrote: post #95
punkpeye wrote: post #45

Earlier quoted context omitted.

They are fundamentally different. If Cloudflare provided a way to host docker containers with volumes though, that would be game over for so many paas platforms.

Can't wait: https://blog.cloudflare.com/container-platform-preview/

wow, this will be huge

Re: Fly.io outage – resolved

#140

Post by jimmyl02 »

paxys wrote: post #116
xyst wrote: post #91

Recurring pattern I notice is outages tend to occur the week of major holidays in US. - MS 365/Teams/Exchange had a blip in the morning - Fly.io with complete outage - then a handful of sites and services impacted due to those outages Usually advocate against “change freezes” but I think a change freeze around major holidays makes sense. Give all teams a recharge/pause/whatever. Don’t put too much pressure on the B-s…

Bad code rarely causes outages at this scale. The culprit is always configuration changes. Sure you can try and reduce those as well during the holiday season, but what if a certificate has to be renewed? What if a critical security patch needs to be applied? What if a set of servers need to be reprovisioned? What if a hard disk is running out of space? You cannot plan your way out of operational challenges, regardle…

I think a good way of looking at it is risk. Is the change (whether it is code or configuration, etc.) worth the risk it brings on.

For example if it's a small feature then it probably makes sense to wait and keep things stable. But, if it's something that itself causes larger imminent danger like security patches / hard disk space constraints, then it's worth taking on the risk of change to mitigate the risk of not doing it.

At the end of the day no system is perfect and it ends up being judgement calls but I think viewing it as a risk tradeoff is helpful to understand.

Post reply on HN