Why is this company always on HN frontpage - ironically for their bad services? Normally, poor service from a provider isn't grounds for such attention - but seems like Fly.io has not done anything great. They still continue to get love from the developer community who "wants them to succeed". I'm puzzled as to why? Because of some blog posts?
Fly.io Postgres cluster down for 3 days, no word from them about it
391–400 of 493 posts
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#392Earlier quoted context omitted.
I'm just a person on Hacker News that happens to be at Fly.io; as I've said before, it's probably reasonable to think of me as an HN person first, and a Fly.io person second. My tone is my tone, and has been for the many years I've participated in this community. I got back from an evening out, saw that we were on the front page, poked around a little to find out what the hell was going on, and did my best to add som…
> If you're reading my comments on HN as some kind of official response from the company, you've misconstrued them. For what it’s worth, this is the reason most companies eventually restrict their employees from making statements about the company; It doesn’t matter if you thought it was clear that is was unofficial, any statement from an employee in a position of power (such as someone with access to the control pan…
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#393Earlier quoted context omitted.
historically, I've used Vultr, but I don't see anyone talking about it—I'm curious if anyone else has thoughts on them? (I've been happy, but then again my usage has been exceedingly basic)
I've used Vultr for several years (hobby projects) with no issues. My favorite feature is having a BGP session from my VM, which is unusual among cloud providers. I have an AS and am able to advertise my own IPs from multiple Vultr instances (anycast).
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#394Earlier quoted context omitted.
I adore DO. They’re seriously underrated. I love how they’ll just give you a server and say here, have at it. No abstractions, no fancy crap, just get out of my way and let me do my thing.
I went to DO's site due to your comment and I don't see anywhere where I can just get a server. Do you mean a VPS/Droplet? (I'm looking under Products and Solutions.)
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#395I actually have been advocating against them for a while here on HN ( https://news.ycombinator.com/item?id=31394179 ) for the same reason. They had my account on some sort of shadow ban with no communication whatsoever after asking them to delete my account from their systems. I emailed them and to date never even got a response. I have moved everything over to Railway app and back to Google Cloud Run ever since.
> they never bothered to reply and put me in some kind of shadow ban from re-registering with my email. So did you manage to delete your account then attempt to re-register using the same email address you deleted the account with? Why would a company shadow ban you for asking an innocuous question?
If you are literally overwhelmed with crises, it becomes appealing to make problems go away in this manner. Not saying they are, but this thread is suggesting that.
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#396Earlier quoted context omitted.
Not a great summary from my perspective. Here's what I got out of it: - Their free tier support depended on noticing message board activity and they didn't. - Those experiencing outages were seeing the result of deploying in a non-HA configuration. Opinions differ as to whether they were properly aware that they were in that state. - They had an unusually long outage for one particular server. - Those points combined…
Apologies for repeating myself, but: You get to a certain number of servers and the probability on any one day that some server somewhere is going to hiccup and bounce gets pretty high. That's what happened here: a single host in Sydney, one of many, had a problem. When we have an incident with a single host, we update a notification channel for people with instances on that host. They are a tiny sliver of all our us…
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#397There's a lot of bullshit in this HN thread, but here's the important takeaway: - it seems their staff were working on the issue before customers noticed it. - once paid support was emailed, it took many hours for them to respond. - it took about 20 hours for an update from them on the downed host. - they weren't updating their users that were affected about the downed host or ways to recover. - the status page was b…
To clarify, we communicated this incident to the personalized status page [1] of all affected customers within 30 minutes of this single host going down, and resolved the incident on the status page once it was resolved ~47h later. Here's the timeline (UTC):
- 2023-07-17 16:19 - host goes down
- 2023-07-17 16:49 - issue posted to personalized status page
- 2023-07-19 15:00 - host is fixed
- 2023-07-19 15:17 - issue marked resolved on status page
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#398Earlier quoted context omitted.
I'm using Digital Ocean App platform, which does pretty much everything for me. It's very simple to use. I can run my app as a single developer without caring about infrastructure for 99% of the time.
Do they offer authentication/authorization? This is the one thing I need in every app and don't want to do myself.
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#399Earlier quoted context omitted.
But HN is not a customer service forum ?
It's often used as an escalation point when people can't get support from certain companies (most notably, Google). If an employee lurks in here and sees your post, they might contact the right people to fix your issue. Smaller companies also do a lot of PR damage control and constantly monitor HN for threads complaining about their services. You're not wrong but that's how it works.
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#400Earlier quoted context omitted.
Why are your customers exposed to this? This sounds like a tough problem that I'm sympathetic to for you personally, but it sounds like there's no failover or appropriate redundancy in place to rollover to while you work to fix the problem. edit: I hope this comment doesn't sound accusatory. At the end of the day I want everyone to succeed. I hope there's a silver lining to this in the post-mortem.
The way to not be exposed to this is to run an HA configuration with more than one instance. If you're running an app on Fly.io without local durable storage, then it's easy to fail over to another server. But durable storage on Fly.io is attached NVMe storage. By far the most common way people use durable storage on Fly.io is with Postgres databases. If you're doing that on Fly.io, we automatically manage failover a…
If this sounds ludicrous, then I think I probably don't understand who Fly.io wants to be and that's okay. If I don't understand, however, you may want to take a look at your image and messaging to potentially recalibrate what kind of customers you're attracting.