Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Postgres cluster down for 3 days, no word from them about it

webcache.googleusercontent.com

Post reply on HN
Print view View original

441–450 of 493 posts

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#441

Post by itake »

tinco wrote: post #423
CSSer wrote: post #383

Earlier quoted context omitted.

I'm surprised by your risk tolerance. If I had any cloud service at this level in my stack go down for three days, I'd start shopping for an alternative. This exceeds the level of acceptability for me for even non-HA requirements. After all, if I can't trust them for this, why would I ever consider giving them my HA business? Just based on napkin math for us, this could've been a potential loss of nearly half a milli…

I think you're not exposed enough to the reality of hardware. There was no need for the host to come back online at all. I think it was a mistake of Fly.io to even attempt to do it. Just say tell the customer the host was lost and offer them a new one (with a freshly zeroed volume attached). You rent a machine, it breaks, you get a new one. If they're sad that they lost their data, it's their fault for running on a s…

Is this the posture of other hosting providers? If not, it seems other hosting providers offer better quality of service.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#442

Post by seti0Cha »

tptacek wrote: post #435
seti0Cha wrote: post #411

Earlier quoted context omitted.

I agree, articles tend not to get read by those who need them most. A warning from the CLI and a banner on the app management page with a link to a detailed explanation would seem like a good approach. edit: sibling post shows there is such a message on the CLI. The only other thing I can think of is an "Are you sure you want to do this?" prompt, but in the end you can't reach everybody.

There is an "Are you sure want to do this?" prompt!

Make them type the phrase "I'm OK with downtimes of arbitrary length"!

I kid, seems like you guys did what you could.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#443

Post by tinco »

itake wrote: post #441
tinco wrote: post #423

Earlier quoted context omitted.

I think you're not exposed enough to the reality of hardware. There was no need for the host to come back online at all. I think it was a mistake of Fly.io to even attempt to do it. Just say tell the customer the host was lost and offer them a new one (with a freshly zeroed volume attached). You rent a machine, it breaks, you get a new one. If they're sad that they lost their data, it's their fault for running on a s…

Is this the posture of other hosting providers? If not, it seems other hosting providers offer better quality of service.

I would think so, it's honestly strange to think about. The idea of having the node come back after it broke is a bit ridiculous to me. A node breaks, you delete it from your interface and provision a new one, the idea of even waiting 5 minutes for it to come up is strange. This whole conversation seems detached from how the cloud is supposed to and has operated in the past decade.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#444

Post by Jupe »

makoz wrote: post #440
Jupe wrote: post #405

Earlier quoted context omitted.

Ouch? The bad news is that I'd be out of a job if I chose your service in this instance. 47 hours is two full days. For an entire cluster to be down for that long is just unacceptable. Rebuilding a cluster from the last-known-good backup should not take that long, unless there are PBs of data involved; dividing such large data stores into separate clusters/instances seems warranted. Solution archs should steer custom…

Disclaimer work in AWS. > Rebuilding a cluster from the last-known-good backup should not take that long It's not even clear if that's the right thing to do as a service provider. Let's say you host a database on some database service, and the entire host is lost. I don't think you want the service provider to restore automatically from the last backup because it makes assumptions about what data loss you're tolerant…

Restoring from backup doesn't mean you actually have to use it - just prepare it in case you need it. Since this can take time, starting such a restore early would be an insurance policy, if needed. If there are snapshots to apply after the last-known-good backup, all the better.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#445

Post by Jupe »

TheDong wrote: post #412
Jupe wrote: post #405

Earlier quoted context omitted.

Ouch? The bad news is that I'd be out of a job if I chose your service in this instance. 47 hours is two full days. For an entire cluster to be down for that long is just unacceptable. Rebuilding a cluster from the last-known-good backup should not take that long, unless there are PBs of data involved; dividing such large data stores into separate clusters/instances seems warranted. Solution archs should steer custom…

> The bad news is that I'd be out of a job if I chose your service in this instance. 47 hours is two full days. There was one physical server down. That's it. They even brought it back. I've had AWS delete more instances, including all local NVMe store data, than I can count on my hands. Just in the last year. Those instances didn't experience 47 hours downtime, they experienced infinite downtime, gone forever. I gue…

Since the post said "cluster", I assumed it was a set of instances with replicas and the like.

I've never experienced AWS killing nodes forever; at least not DB instances.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#446

Post by pinkcan »

tptacek wrote: post #368
pinkcan wrote: post #157

Earlier quoted context omitted.

what's up with the status page?

There's a global status page, and then there's a local update for people with instances on an affected host --- past some threshold of hosts, the probability of having an issue on some random host gets pretty high just because math. The local status thing happened for people with instances on that machine. Ordinarily, a single-host incident takes a couple minutes to resolve, and, ordinarily, when it's resolved, every…

thanks for clearing that up

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#447

Post by pinkcan »

ctvo wrote: post #387
fuzztester wrote: post #162

Earlier quoted context omitted.

That's a common phrase, not to be taken literally. It just means one single person (at the vendor) who you can complain to, or raise an issue with.

Been in the industry a long time. It’s not a common phrase. It’s weirdly violent. At most “someone to yell at”. A throat to choke? What the fuck.

not a native speaker, but have been reading and writing english long enough to pick up the meaning immediately

anyway, had the same though when typing my sibling comment, felt so disgusted reading that

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#448

Post by okhuman »

fauigerzigerk wrote: post #375
okhuman wrote: post #343

Earlier quoted context omitted.

Would you consider a project like https://github.com/authcompanion/authcompanion2 for the authentication side? Missing anything?

No I would not. I don't like self hosting anything that requires its own process. And if I did decide to self host I would choose a more mature project. This is a very young one man project delegating the heavy lifting to another one man project. And it doesn't appear to support social logins.

thanks for the feedback.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#449

Post by yencabulator »

yjftsjthsd-h wrote: post #416
TheDong wrote: post #412

Earlier quoted context omitted.

> The bad news is that I'd be out of a job if I chose your service in this instance. 47 hours is two full days. There was one physical server down. That's it. They even brought it back. I've had AWS delete more instances, including all local NVMe store data, than I can count on my hands. Just in the last year. Those instances didn't experience 47 hours downtime, they experienced infinite downtime, gone forever. I gue…

When does AWS delete instances? Migrate, sure, and yes, local storage is supposed to be treated as disposable for that reason, but AFAIK only spot instances should be able to be destroyed outright.

The underlying problem is that Fly doesn't provide non-local, less-eager-to-disappear, volumes.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#450

Post by yencabulator »

tptacek wrote: post #371
gowthamgts12 wrote: post #172

Earlier quoted context omitted.

may be it's to avoid search engines to not scrape these threads?

My understanding is that it was causing support problems, because people were Googling for solutions to problems with their apps (because of the Heroku diaspora, we have a lot of first-time Docker users), finding old stale threads on our forum that looked related, and then reviving them. I think we can just `noindex` the category instead of making it private?

So the tagged posts were intentionally hidden, then.
Post reply on HN