Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io outage – resolved

status.flyio.net

Post reply on HN
Print view View original

101–110 of 287 posts

Re: Fly.io outage – resolved

#101

Post by throwaway984393 »

benreesman wrote: post #82
redslazer wrote: post #43

Earlier quoted context omitted.

The tech is impressive and the pricing is attractive which is why we use them. I just wish there was less black magic.

I don’t always agree with @tptacek on social/political issues, and I don’t always agree with @xe on the direction of Nix, but these are legends on the technical side of things. And they’re trying to build an equitable relationship between the user of cloud services and the provider, not fund a private space program. If I were in the market for cloud services I’d highly prize a long-term relationship on mutual benefit…

[dead]

Re: Fly.io outage – resolved

#102

Post by littlestymaar »

dumah wrote: post #57
MaxfordAndSons wrote: post #10

Kinda funny that they've named their global state store "Corrosion"... not really a word I'd associate with stability and persistence.

I take your point but corrosion-resistant metals such as Aluminum, Titanium, Weathering Steel and Stainless Steel don’t avoid corrosion entirely but form a thin and extremely stable corrosion layer (under the right conditions).

Gold and platinum really are corrosion resistant though (but have questionable mechanical properties…)

Re: Fly.io outage – resolved

#103

Post by toast0 »

ixaxaar wrote: post #89
dilyevsky wrote: post #79

Earlier quoted context omitted.

> Ultimately, I found the answer in their tech docs where it was spelled out clearly that an fly instance is hardwired to one physical server and thus cannot fail over in case that server dies. Majority of EC2 instance types did not have live migration until very recently. Some probably still don't (they don't really spell out how and when it's supposed to work). It is also not free - there's a noticeable brown-out w…

Can you shed some more light on this "browning out" phenomenon?

Here's the GCP doc [1]. Other live migration products are similar.

Generally, you have worse performance while in the preparing to move state, an actual pause, then worse performance as the move finishes up. Depending on the networking setup, some inbound packets may be lost or delayed.

[1] https://cloud.google.com/compute/docs/instances/live-migrati...

Re: Fly.io outage – resolved

#104

Post by SOLAR_FIELDS »

fragmede wrote: post #64
SOLAR_FIELDS wrote: post #38

Earlier quoted context omitted.

I get this but I think if people can give GitHub a pass for shitting the bed every two weeks maybe Fly should get a bit of goodwill here. I am not affiliated with Fly at all but I do think that people should temper their expectations when even mega corp can’t get it right I guess the secret is to be the incumbent with no suitable replacement. Then you can be complete garbage in terms of reliability and everyone will…

Who's giving GitHub a pass on shitting the bed? They go down often enough that if you don't have an internal git server setup for your CICD to hit, that's on you.

My point is made by your very post - getting off GitHub onto alternatives is not seriously discussed as an option - instead it’s “well, why didn’t you prepare better to deal with your vendor’s poor ops story”

Re: Fly.io outage – resolved

#105

Post by aaomidi »

xyst wrote: post #91

Recurring pattern I notice is outages tend to occur the week of major holidays in US. - MS 365/Teams/Exchange had a blip in the morning - Fly.io with complete outage - then a handful of sites and services impacted due to those outages Usually advocate against “change freezes” but I think a change freeze around major holidays makes sense. Give all teams a recharge/pause/whatever. Don’t put too much pressure on the B-s…

What do "Freezes" mean? Like, do you stop renewing your certificates? Do you stop taking in security updates for your software?

Sure maybe "unnecessary" changes, but the line gets very gray very fast.

Re: Fly.io outage – resolved

#106

Post by vrosas »

xyst wrote: post #91

Recurring pattern I notice is outages tend to occur the week of major holidays in US. - MS 365/Teams/Exchange had a blip in the morning - Fly.io with complete outage - then a handful of sites and services impacted due to those outages Usually advocate against “change freezes” but I think a change freeze around major holidays makes sense. Give all teams a recharge/pause/whatever. Don’t put too much pressure on the B-s…

Then you just get devs rushing out changes before the freeze…

Re: Fly.io outage – resolved

#107

Post by ploxiln »

xyst wrote: post #91

Recurring pattern I notice is outages tend to occur the week of major holidays in US. - MS 365/Teams/Exchange had a blip in the morning - Fly.io with complete outage - then a handful of sites and services impacted due to those outages Usually advocate against “change freezes” but I think a change freeze around major holidays makes sense. Give all teams a recharge/pause/whatever. Don’t put too much pressure on the B-s…

I think you can't avoid the fact that these holiday weeks are different from regular weeks. If you "change freeze" then you also freeze out the little fixes and perf tuning that usually happens across these systems, because they're not "critical".

And then inevitably it turns out that there's a special marketing/product push, with special pricing logic that needs new code, and new UI widgets, causing a huge traffic/load surge, and it needs to go out NOW during the freeze, and this is revenue, so it is critical to the business leaders. Most of eng, and all of infra, didn't know about it, because the product team was cramming until the last minute, and it was kinda secret. So it turns out you can freeze the high-quality little fixes, but you can't really freeze the flaky brand-new features ...

It's just a struggle, and I still advise to forget the freeze, and try to be reasonable and not rush things (before, during, or after the freeze).

Re: Fly.io outage – resolved

#108

Post by vrosas »

aaomidi wrote: post #105
xyst wrote: post #91

Recurring pattern I notice is outages tend to occur the week of major holidays in US. - MS 365/Teams/Exchange had a blip in the morning - Fly.io with complete outage - then a handful of sites and services impacted due to those outages Usually advocate against “change freezes” but I think a change freeze around major holidays makes sense. Give all teams a recharge/pause/whatever. Don’t put too much pressure on the B-s…

What do "Freezes" mean? Like, do you stop renewing your certificates? Do you stop taking in security updates for your software? Sure maybe "unnecessary" changes, but the line gets very gray very fast.

No unnecessary code deployments.

Re: Fly.io outage – resolved

#109

Post by toast0 »

cj wrote: post #58
kachapopopow wrote: post #40

Earlier quoted context omitted.

It's still 99.99+% SLA? Would you really pay 100% more for <0.01% more uptime?

I think what a lot of people fail to understand is that there are certain categories of apps that simply “can never go down” Examples include basically any PaaS, IaaS, or any company that provides a mission-critical service to another company (B2B SaaS). If you run a basic B2C CRUD app, maybe it’s not a big deal if you service goes down for 5 minutes. Unfortunately there are quite a few categories of companies where…

> I think what a lot of people fail to understand is that there are certain categories of apps that simply “can never go down”

I refuse to believe that this category still exists, when I need to keep my county's alternate number for 911 in my address book, because CenturyLink had a 6 hour outage in 2014 and a two day outage in 2018. If the phone company can't manage to keep 911 running anymore, I'd be very surprised what does have zero downtime over a ten year period.

Personally, nine nines is too hard, so I shoot for eight eights.

Post reply on HN