You unfortunately get what you pay for. AWS is more expensive than God, but I'll be damned if you can't have a throat to choke in less than 10 minutes whenever something like this happens.
> a throat to choke
yikes
Fly.io Postgres cluster down for 3 days, no word from them about it
81–90 of 493 posts
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#82Y'all, this is going to be deeply unsatisfying, but it's what I can report personally: I have no earthly clue why this thread on our community site is unlisted. We're looking at the admin UI for it right now, and there's like, a little lock next to do the story, but the "unlist story" option is still there for us to click. The best I can say is: I'm reasonably sure there wasn't some top-down edict to hide this thread…
> I have no earthly clue why this thread on our community site is unlisted.
Maybe it's hosted in the SYD region
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#83Y'all, this is going to be deeply unsatisfying, but it's what I can report personally: I have no earthly clue why this thread on our community site is unlisted. We're looking at the admin UI for it right now, and there's like, a little lock next to do the story, but the "unlist story" option is still there for us to click. The best I can say is: I'm reasonably sure there wasn't some top-down edict to hide this thread…
There's also a lock icon next to the "App not working" category in the header, which I took to mean that that entire category is hidden from logged-out users (which experimentally seems to be the case).
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#84Y'all, this is going to be deeply unsatisfying, but it's what I can report personally: I have no earthly clue why this thread on our community site is unlisted. We're looking at the admin UI for it right now, and there's like, a little lock next to do the story, but the "unlist story" option is still there for us to click. The best I can say is: I'm reasonably sure there wasn't some top-down edict to hide this thread…
There's also a lock icon next to the "App not working" category in the header, which I took to mean that that entire category is hidden from logged-out users (which experimentally seems to be the case).
I have the impression from this thread that this thread was public (as in, would work if you just linked to it from something like HN) earlier, and now it isn't?
Obviously, deliberately hiding a negative story on our Discourse is a little like deleting a bad tweet; it's just going to guarantee someone captures and boosts it. We have a lot of flaws! But not knowing how the Internet works probably isn't one of them. No idea what's going on here, still trying to work it out.
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#85Earlier quoted context omitted.
Maybe there needs to be a better "burn in" test setup for their new hardware, just to catch mistakes in the build prep and/or catch bad hardware?
Not that nothing will fail - but some manufacturers have just really good fault management, monitoring, alerting, etc. And even the simplest shit like SNMP with a few custom MIBs from the vendor (which theres some that do it better). Facilities and vendors that lend a good hand with remote hands is also nice, if you remote management infrastructure should fail. But out of band, full featured management cards with all…
Could you expand your answer to list vendors which you would recommend?
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#86For those of you with postgres apps, you can avoid this pretty much universally with cockroach db (they have a serverless version they host) It takes basically no work moving from postgres, even postgres dump works.
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#87Wondering if for small/bootstrapped projects there's any alternative people suggest? Fly has a nice UX and accessible prices, but it's unstable at best. I use the big clouds at work, but for personal they are $$$. Also I want to keep devops tending asymptotically to zero.
I use Dokku on top of Hetzner for my hobby projects - hosting is super cheap, for a little extra I can add a mounted volume for storage, and if the project outgrows a single server I can always just break out of Dokku and use some Docker containers behind a load balancer. If you are outside of Europe, Digital Ocean or Linode may work better for you.
[deleted]
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#88You unfortunately get what you pay for. AWS is more expensive than God, but I'll be damned if you can't have a throat to choke in less than 10 minutes whenever something like this happens.
I had one situation where a Hetzner dedi didn't come back up on a reboot. Their dedis are cheap, this one is like $40ish/mo? Opened a ticket and support had it back up again within about 10 minutes, turned out to be a failed CPU fan which caused an overheat condition and made it so the system wouldn't complete the boot. They swapped the fan and it came up. It's the only failure I've had in years of dealing with them…
https://www.youtube.com/watch?v=5eo8nz_niiM
Datacenters in my country usually had some rooms with tower servers 20 years ago here, well my first colo was for the tower server I brought in the large backpack:-). But density requirements, cold/hot aisles etc. prevailed and towers are generally considered inefficient for the datacenter purposes.
And then you have Hetzner datacenter that probably all people running DCs I know would ridicule, but they would not be able to respond to fan replacement at the same time. I wonder how many rack server chassis are recycled each year because the manufacturer just won't let you reuse them with new motherboard, power supply due to new shape, design, ports placement etc.
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#89I actually have been advocating against them for a while here on HN (https://news.ycombinator.com/item?id=31394179) for the same reason.
They had my account on some sort of shadow ban with no communication whatsoever after asking them to delete my account from their systems. I emailed them and to date never even got a response. I have moved everything over to Railway app and back to Google Cloud Run ever since.
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#90I really want to love Fly.io. It's super easy to get setup and use, but to be honest I don't think anyone should be building mission critical applications on their service. I ended up migrating everything over to AWS (which I reallllly didn't want to do) because: * Frequent machines not working, random outages, builds not working * Support wasn't responsive, didn't read my questions (kept asking same questions over a…
I've had the same experience, unfortunately.
The Fly dashboard reported everything was A-ok, but requests would time out. I had to manually dig into the fly logs to see that their proxy couldn't reach the server, and there was nothing I could do to fix it.
This went on for hours, until I made an issue on their forums. They never replied or gave any indication they read the thread, but it somehow magically got fixed not long after.
I really want them to succeed, but this utter lack of communication and helpless feeling of not being able to do anything has cured me from fly.io for now.