Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Postgres cluster down for 3 days, no word from them about it

webcache.googleusercontent.com

Post reply on HN
Print view View original

371–380 of 493 posts

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#371

Post by tptacek »

gowthamgts12 wrote: post #172
michaeldwan wrote: post #93

Earlier quoted context omitted.

I don't know why the app-not-working category effectively delists threads, but until we find out, I just removed it so this thread is public again.

may be it's to avoid search engines to not scrape these threads?

My understanding is that it was causing support problems, because people were Googling for solutions to problems with their apps (because of the Heroku diaspora, we have a lot of first-time Docker users), finding old stale threads on our forum that looked related, and then reviving them.

I think we can just `noindex` the category instead of making it private?

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#372

Post by seti0Cha »

throwawaaarrgh wrote: post #338

There's a lot of bullshit in this HN thread, but here's the important takeaway: - it seems their staff were working on the issue before customers noticed it. - once paid support was emailed, it took many hours for them to respond. - it took about 20 hours for an update from them on the downed host. - they weren't updating their users that were affected about the downed host or ways to recover. - the status page was b…

Not a great summary from my perspective. Here's what I got out of it:

- Their free tier support depended on noticing message board activity and they didn't.

- Those experiencing outages were seeing the result of deploying in a non-HA configuration. Opinions differ as to whether they were properly aware that they were in that state.

- They had an unusually long outage for one particular server.

- Those points combined resulted in many people experiencing an unexplained prolonged outage.

- Their dashboard shows only regional and service outages, not individual servers being down. People did not realize this and so assumed it was a lie.

- Some silliness with Discourse tags caused people to think they were trying to hide the problems.

In short, bad luck, some bad procedures from a customer management POV, possibly some bad documentation resulted in a lot of smoke but not a lot of fire.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#374

Post by ctvo »

stephenr wrote: post #244
throwaway220033 wrote: post #69

The worst thing about Fly is, when something goes wrong, it's not just one thing, there's bunch of things broken at the same time and their status page will show everything green. Their typical response is either silence or so casual ("oh this is what happens we deploy on friday"). The product looks amazing but it's just a nice package around the most unreliable hosting service I've ever used. You can't just keep bre…

> when something goes wrong, it's not just one thing, there's bunch of things broken at the same time and their status page will show everything green How dare they use AWS' patented approach to having a service outage.

I wouldn’t put AWS and Fly in the same sentence. AWS is magnitudes more reliable, with better support.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#375

Post by fauigerzigerk »

okhuman wrote: post #343
fauigerzigerk wrote: post #217

Earlier quoted context omitted.

Do they offer authentication/authorization? This is the one thing I need in every app and don't want to do myself.

Would you consider a project like https://github.com/authcompanion/authcompanion2 for the authentication side? Missing anything?

No I would not.

I don't like self hosting anything that requires its own process. And if I did decide to self host I would choose a more mature project.

This is a very young one man project delegating the heavy lifting to another one man project. And it doesn't appear to support social logins.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#377

Post by stephenr »

ctvo wrote: post #374
stephenr wrote: post #244

Earlier quoted context omitted.

> when something goes wrong, it's not just one thing, there's bunch of things broken at the same time and their status page will show everything green How dare they use AWS' patented approach to having a service outage.

I wouldn’t put AWS and Fly in the same sentence. AWS is magnitudes more reliable, with better support.

I didn't mention how often they fail.

I merely mentioned two characteristics of how they fail, that are spectacularly shit.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#378

Post by malablaster »

throwawaaarrgh wrote: post #338

There's a lot of bullshit in this HN thread, but here's the important takeaway: - it seems their staff were working on the issue before customers noticed it. - once paid support was emailed, it took many hours for them to respond. - it took about 20 hours for an update from them on the downed host. - they weren't updating their users that were affected about the downed host or ways to recover. - the status page was b…

> there’s a lot of bullshit

…proceeds to make a bunch of non-factual statements.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#379

Post by tptacek »

seti0Cha wrote: post #372
throwawaaarrgh wrote: post #338

There's a lot of bullshit in this HN thread, but here's the important takeaway: - it seems their staff were working on the issue before customers noticed it. - once paid support was emailed, it took many hours for them to respond. - it took about 20 hours for an update from them on the downed host. - they weren't updating their users that were affected about the downed host or ways to recover. - the status page was b…

Not a great summary from my perspective. Here's what I got out of it: - Their free tier support depended on noticing message board activity and they didn't. - Those experiencing outages were seeing the result of deploying in a non-HA configuration. Opinions differ as to whether they were properly aware that they were in that state. - They had an unusually long outage for one particular server. - Those points combined…

Apologies for repeating myself, but:

You get to a certain number of servers and the probability on any one day that some server somewhere is going to hiccup and bounce gets pretty high. That's what happened here: a single host in Sydney, one of many, had a problem.

When we have an incident with a single host, we update a notification channel for people with instances on that host. They are a tiny sliver of all our users, but of course that's cold comfort for them; they're experiencing an outage! That's what happened here: we did the single-host notification thing for users with apps on that Sydney host.

Normally, when we have a single-host incident, the host is back online pretty quickly. Minutes, maybe double-digit minutes if something gnarly happened. About once every 18 months or so, something worse than gnarly happens to a server (they're computers, we're not magic, all the bad things that happen to computers happen to us too). That's what happened here: we had an extended single-host outage, one that lasted over 12 hours.

(Specifically, if you're interested: somehow a containerd boltdb on that host got corrupted, so when the machine bounced, containerd refused to come back online. We use containerd as a cache for OCI container images backing flyd; if containerd goes down, no new machines can start on the host. It took a member of our team, also a containerd maintainer, several hours to do battlefield surgery on that boltdb to bring the host back up.)

Now, as you can see from the fact that we were at the top of HN all night, there is a difference between a 5 minute single-host incident and a 12-hour single-host outage. Our runbook for single-host problems is tuned for the former. 12-hour single-host outages are pretty rare, and we probably want to put them on the global status page (I'm choosing my words carefully because we have an infra team and infra management and I'm not on it, and I don't want to speak for them or, worse, make commitments for them, all I can say is I get where people are coming with this one).

Post reply on HN