Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Postgres cluster down for 3 days, no word from them about it

webcache.googleusercontent.com

Post reply on HN
Print view View original

111–120 of 493 posts

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#111

Post by teraflop »

michaeldwan wrote: post #105
teraflop wrote: post #92

Earlier quoted context omitted.

Yes, from the Google-cached version, it appears that the thread previously didn't have the app-not-working tag; it was only tagged with "rails". Not going to try and guess why or when that tag change happened. Personally, I'm less concerned with this particular thread than with the apparent decision to systematically hide all potentially-negative threads from search engines.

That category was added after one of our support folks replied, likely for tracking. I don't know why it's private. They may not even know this category is private. Hiding negative shit wasn't a deliberate decision... we're aware of google cache and we don't need to give HN another reason to dunk on us.

> That category was added after one of our support folks replied

FYI, this doesn't appear to be strictly accurate. The OP commented at 23:52 UTC saying that the thread had been made private, and the reply from "Sam-Fly" was not posted until 02:36 UTC.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#112

Post by thyrox »

You know what's interesting? It feels like history is repeating itself with Fly.io, just like it did back when I first encountered Heroku. Back in the day, I was super excited about Fly.io – it had that same fresh, exciting vibe that Heroku had when it burst onto the scene.

I remember being blown away by Fly.io's simplicity and how easy it was to use. It was like hosting made simple, and I couldn't help but think, "This is it, this is the one!"

But, as time went on, I noticed little signs of trouble. Downtimes became more frequent, and my deployments, which were once snappy and seamless, turned into agonizingly slow affairs. It was like déjà vu from the time when Heroku's greatness started to wane.

It's disheartening to see Fly.io go down a similar path. As more people flocked to the platform, it seems like its performance began to suffer – just like what happened with Heroku. The more popular it got, the less reliable it seemed to become.

Scrolling through Hacker News, I can't help but feel a sense of disappointment. Others are expressing their frustration too, and it's like we're all reliving that moment when Heroku lost its charm and became a hassle.

I have to admit; it worries me. It's like a cautionary tale of how even the most promising platforms can fall from grace. It's the reality of the fast-paced tech world, but it's tough to accept.

So yeah, here I am, hoping against hope that Fly.io can somehow break free from this cycle and find its footing before it becomes as useless as Heroku was at its lowest point.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#113

Post by okeuro49 »

neya wrote: post #89

I actually have been advocating against them for a while here on HN ( https://news.ycombinator.com/item?id=31394179 ) for the same reason. They had my account on some sort of shadow ban with no communication whatsoever after asking them to delete my account from their systems. I emailed them and to date never even got a response. I have moved everything over to Railway app and back to Google Cloud Run ever since.

> they never bothered to reply and put me in some kind of shadow ban from re-registering with my email.

So did you manage to delete your account then attempt to re-register using the same email address you deleted the account with?

Why would a company shadow ban you for asking an innocuous question?

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#114

Post by subarctic »

tptacek wrote: post #99
subarctic wrote: post #98

It's really disappointing that they made this forum thread private, apparently in response to this HN thread blowing up. This is the first negative HN thread I've seen about them, it's not even really that bad because this kind of downtime is expected, and they can't get to every forum post, and their response that someone posted here is totally reasonable in my opinion. So why is the link to the thread 404ing and wh…

(a) Not even close to the first negative HN thread about us. (b) We definitely didn't make the thread private in response to HN. (c) It should be public again.

I saw your other comment, glad to see this wasn't intentional as the optics were pretty bad

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#115

Post by tptacek »

xupybd wrote: post #106
tptacek wrote: post #80

Y'all, this is going to be deeply unsatisfying, but it's what I can report personally: I have no earthly clue why this thread on our community site is unlisted. We're looking at the admin UI for it right now, and there's like, a little lock next to do the story, but the "unlist story" option is still there for us to click. The best I can say is: I'm reasonably sure there wasn't some top-down edict to hide this thread…

From this my take away is that I could get fired for picking Fly.io for work. Not because there was an outage but because days could pass before getting support. What assurances could you give the community here that the support would be better next time?

This is our public site, for people who don't have support plans with us.

It's difficult for me to say more about what happened here and how you might have handled it, because I don't know what happened with this SYD host, because it's 1AM and the people who worked on it are, I assume, asleep. When I know more, I'll do my best to get you a postmortem.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#116

Post by justsid »

danielvaughn wrote: post #56
steve_adams_86 wrote: post #31

Earlier quoted context omitted.

I left digitalocean for fly because some of their tooling was excellent. I was pretty excited. I’m back on digitalocean now. I’m not unhappy about it, they’re very solid. I don’t love some things about their services, but overall I’d highly recommend them to other developers. I gave up on fly because I’d spontaneously be unable to automate deployments due to limited resources. Or I’d have previously happy deployments…

I adore DO. They’re seriously underrated. I love how they’ll just give you a server and say here, have at it. No abstractions, no fancy crap, just get out of my way and let me do my thing.

I wish I could say the same. My ISP and DO have absolutely terrible peering, unfortunately a lot of our internal stuff is hosted there. It’s always fun to git push/pull with 40kb/s on a gigabit connection.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#117

Post by justinclift »

skullone wrote: post #77
justinclift wrote: post #57

Earlier quoted context omitted.

Maybe there needs to be a better "burn in" test setup for their new hardware, just to catch mistakes in the build prep and/or catch bad hardware?

Not that nothing will fail - but some manufacturers have just really good fault management, monitoring, alerting, etc. And even the simplest shit like SNMP with a few custom MIBs from the vendor (which theres some that do it better). Facilities and vendors that lend a good hand with remote hands is also nice, if you remote management infrastructure should fail. But out of band, full featured management cards with all…

Good point. :)

I'm still wondering about their hardware acceptance/qualification though, prior to it being deployed. ;)

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#118

Post by fmajid »

nerpderp82 wrote: post #72
pier25 wrote: post #46

Earlier quoted context omitted.

So you didn't have a HA setup with multiple machines and volumes?

Is that even possible on Fly?

He may have been talking about Fly themselves. Certainly having only a single machine to serve a wealthy metropolis of 8 million people seems like amateur hour.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#119

Post by michaeldwan »

teraflop wrote: post #111
michaeldwan wrote: post #105

Earlier quoted context omitted.

That category was added after one of our support folks replied, likely for tracking. I don't know why it's private. They may not even know this category is private. Hiding negative shit wasn't a deliberate decision... we're aware of google cache and we don't need to give HN another reason to dunk on us.

> That category was added after one of our support folks replied FYI, this doesn't appear to be strictly accurate. The OP commented at 23:52 UTC saying that the thread had been made private, and the reply from "Sam-Fly" was not posted until 02:36 UTC.

My point was that the app-not-working category is used in conjunction with support/our team getting involved. I assume this is what Sam meant by "flagged it internally", which was followed by investigation, then a post. I don't see how the timestamps uncover something nefarious.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#120

Post by x86hacker1010 »

NicoJuicy wrote: post #91
steve_adams_86 wrote: post #31

Earlier quoted context omitted.

I left digitalocean for fly because some of their tooling was excellent. I was pretty excited. I’m back on digitalocean now. I’m not unhappy about it, they’re very solid. I don’t love some things about their services, but overall I’d highly recommend them to other developers. I gave up on fly because I’d spontaneously be unable to automate deployments due to limited resources. Or I’d have previously happy deployments…

I moved from DO to Hetzner ( cheaper), I am happy about it.

Same here
Post reply on HN