Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io outage – resolved

status.flyio.net

Post reply on HN
Print view View original

251–260 of 287 posts

Re: Fly.io outage – resolved

#251

Post by benreesman »

tptacek wrote: post #245
benreesman wrote: post #82

Earlier quoted context omitted.

I don’t always agree with @tptacek on social/political issues, and I don’t always agree with @xe on the direction of Nix, but these are legends on the technical side of things. And they’re trying to build an equitable relationship between the user of cloud services and the provider, not fund a private space program. If I were in the market for cloud services I’d highly prize a long-term relationship on mutual benefit…

I'm several steps removed from day-to-day engineering at this point; the team working on this is much better than I am. It's just a very hard problem; biting it off is something you can certainly blame me for, though. (Also: not a legend, just loud.)

I may be the minority on this view, but I think that it's possible to be both a recognized expert aka legend and loud ("visible" might be a kinder word).

When you talk technology, I listen, and I doubt I'm alone in that. Keep up the good work with fly.io!

Re: Fly.io outage – resolved

#252

Post by Aeolun »

tptacek wrote: post #247
Aeolun wrote: post #163

Earlier quoted context omitted.

OMG, that's hilarious. I use them, and I know what my prices are, but I'd never noticed that the page called pricing doesn't actually have any.

We've always had public pricing; you can't do a metered cloud provider without a rate sheet. But it's been part of our product documentation, rather than the front page of the website, until recently; there's a whole saga behind it, which gets into whether we offer "plans" or not, how support works, all that jazz, all of which kept us from putting together a marketing pricing page.

Yeah, I’m not trying to say you didn’t. After all, I wouldn’t have signed up just to find out the price. I just never noticed it wasn’t actually on the pricing page.

Re: Fly.io outage – resolved

#253

Post by tptacek »

Aeolun wrote: post #252
tptacek wrote: post #247

Earlier quoted context omitted.

We've always had public pricing; you can't do a metered cloud provider without a rate sheet. But it's been part of our product documentation, rather than the front page of the website, until recently; there's a whole saga behind it, which gets into whether we offer "plans" or not, how support works, all that jazz, all of which kept us from putting together a marketing pricing page.

Yeah, I’m not trying to say you didn’t. After all, I wouldn’t have signed up just to find out the price. I just never noticed it wasn’t actually on the pricing page.

I'm overexplainey, because (looks around at whole thread). These aren't fun!

Anyways we've been dunking on ourselves for not having a proper pricing page longer than anyone else could have. :)

Re: Fly.io outage – resolved

#254

Post by infecto »

sofixa wrote: post #239
infecto wrote: post #204

Earlier quoted context omitted.

I have asked this multiple times but is anyone really using edge compute and getting value out of it? I am certain there are cases but I have not seen any of them written up before.

Depends on what you mean by edge compute, but you probably are. 5G towers are a ton of compute on the edge to secure and protect the traffic passing through them. Or if by edge you mean having stuff close to your consumers, every non trivial operation does that.

How is it not obvious based on the thread at hand, fly.io.

And no not every nontrivial operation does it to the extreme of an envisioned fly.io deployment.

Re: Fly.io outage – resolved

#255

Post by tptacek »

rozenmd wrote: post #155
benhoyt wrote: post #8

My fly.io-hosted website went down for 5 minutes (6 hours ago), but then came right back up, and has been up ever since. I use a free monitoring service that checks it every 5 minutes, so it's possible it missed another short bit of downtime. But fly.io has been pretty reliable overall for me!

I externally monitor fly.io and it's docs here: https://flyio.onlineornot.com/ Looks like it lasted 16 minutes for them.

It wasn't a request routing outage; apps running on Fly.io didn't stop running. It was a deployments outage. For reasons passing understanding (I am reliably informed I'm wrong to complain about this), our website is the same Elixir app as our dashboard, and the dashboard got redeployed at one point. Our website being down is not the same as the whole service being down, though I guess there's a truth-in-advertising poetry to it being down when deployments are busted.

Re: Fly.io outage – resolved

#256

Post by tptacek »

ignoramous wrote: post #122
jart wrote: post #92

fly.io publishes their post-mortems here: https://fly.io/infra-log/ The last post-mortem they wrote is very interesting and full of details. Basically back in 2016 the heart or keystone component of fly.io production infrastructure was called consul, which is a highly secure TLS server that tracks shared state and it requires that both the server certificate and the client certificate be authenticated. Since it was c…

On that Consul outage, Fly Infra concludes, "The moral of the story is, no more half-measures." On their careers page [1], the Fly team goes, "We're not big believers in tech debt." As an outsider, reads like a cacophony of contradictions? [1] https://fly.io/docs/hiring/working/#we-re-ruthless-about-doi...

All the other comments aside: these aren't even contradictory statements. We really do have no-tech-debt rules, and they generally have not been responsible for our outages. Consul wasn't tech debt; it was a carefully made decision (that I happen to disagree with and enjoy thinking about Michael Ehrmantrout shooting in the face).

We're just people, working on building a thing.

https://www.youtube.com/watch?v=ghNJxYP5Ses

Also: stop calling yourself an "outsider". You follow us as closely as anybody. :)

Re: Fly.io outage – resolved

#257

Post by tptacek »

whilenot-dev wrote: post #154
jart wrote: post #131

Earlier quoted context omitted.

No one actually lives up to their principles, but it's still important that we have them. If you actually do live up to yours, then you need to adopt better principles.

Any principle in itself isn't without critique, agree, but it's still the choice being made to pick this specific principle that tells the whole story. There are so many principles to pick from and the tech dept pick follows up with a "We have a 3-month “no refactoring” rule for new hires. This isn’t everyone’s preferred work style! We try to be up front about stuff." , which sounds a bit like an additional perform o…

It is absolutely not a "perform or else" rule. Why are you reading so far into this? We really do have a rule about tech-debt changes, and it's a useful insight into why you might or might not want to work here, which is why we bring it up, despite the possibility it might alienate people; we'd like to be as honest as we can be. Worrying about people reading hustle-culture bullshit into stuff like this is a reason not to be transparent, which sucks.

Re: Fly.io outage – resolved

#258

Post by tptacek »

cryptos wrote: post #162

Fly.io seems to be a bit of a mixed bag: https://news.ycombinator.com/item?id=41917436 https://news.ycombinator.com/item?id=35044516 https://news.ycombinator.com/item?id=34742946 https://news.ycombinator.com/item?id=34229751 If a cloud platform doesn't really provide reliability, I'd say it's probably not worth it. You could better just rent a (virtual) server and save the cloud tax.

This is a completely sane way to look at the world and we won't push back on it at all. We're building something extraordinarily difficult, and we're a relatively new company, and we don't have even a fraction of the resources the hyperscalers do, or, in the cases of AWS, GCP, and OCI, had at the time they started. If you're minmaxing for reliability --- which is an absolutely sane way to play --- we're not going to tell you you'd do worse in 2024 UE1.

If it helps: all sorts of things can and do go wrong, but the most likely form of disruption you're likely to see here are periods of times when deployments don't work. This outage was a deployments/orchestration outage. We had a total request routing outage several months back, owing to a Rust concurrency landmine we stepped on, but those are very rare.

(Deployment and state-update outages are a big deal, and if you deploy to diverse groups of Fly Machines constantly, as we encourage you to do, that being one of the big features of the platform, they can impact your availability. I'm not downplaying them.)

Re: Fly.io outage – resolved

#259

Post by pajeetz »

tptacek wrote: post #243
pajeetz wrote: post #233

Earlier quoted context omitted.

fly.io has a very bad reputation for reliability there doesn't seem to be any damage control beyond hackernews and even here the consensus seems to be "dont run anything mission critical on fly.io or expect data redundancy" in fact, you can almost get the same thing fly.io does by running firecracker on your own bare metal servers and cheaper too. I'm afraid the public sentiment towards fly.io has been tainted for go…

This is the second place you've offered this sentiment. Was it your expectation that we were going to hit some point, sometime in the near future, where we weren't going to have deployment-blocking outages? I'd like to better understand your premise. If it's "I can get more reliability by deploying on a hyperscaler cloud", who ever told you otherwise?

I see so you think its good business practice to basically say "expect more downtimes in the future who cares about your entire business going down for several hours more than once a year.

Gotcha. I'll be sure to pass on the good word.

Re: Fly.io outage – resolved

#260

Post by pajeetz »

tptacek wrote: post #241
pajeetz wrote: post #235

Earlier quoted context omitted.

theres just so many anecdotes/nightmare stories from people using fly.io here much more than the ones linked by GP expect to see more of these "post-mortem apologies" from fly.io in the future because it won't be the last

You're right. It won't. Nobody could claim otherwise.

"Expect more downtime in the near future, btw please host your business critical applications with our cloud offering" did i read that correctly?
Post reply on HN