This seems to be a pattern at fly but I guess they have the apology ready.
But yeah I cannot complain too much, I pay nothing so I got the appropriate support.
Fly.io Postgres cluster down for 3 days, no word from them about it
221–230 of 493 posts
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#222Honesty would be more impressed if marking the thread private was intentional to take focus off their falling asleep at the wheel. At least that would have indicated some competent leadership.
Yeah, I think the worst behavior to observe is when people make hard decisions and then start to have doubts about others' judgement and try to distance from the decision. That's even worse than the hard decision itself.
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#223Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#224They’ve been working on Fly for years now and seems like they haven’t been able to turn it into a reliable service or profitable business (making assumptions about the second part here), and the overall general sentiment seems to be to avoid it for anything but the most toy applications. I note that the team was also unable to get their recruiting business off the ground either and shuttered it.
My assumption based on the creator’s very online hacker news commentary is that they seem to be at least smart in tech. So what’s the lesson here for the rest of us who may want to start a business? Is this a “shots on goal” thing and we’re just seeing these failures more publicly than most so it biases the perception, or is there some je ne sais quoi missing that we could learn from? No offense intended by my post, but I would be very keen to learn whether there’s some X Factor missing from an otherwise ostensibly smart team’s repeated failure that we could learn from.
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#225Earlier quoted context omitted.
I moved from DO to Hetzner ( cheaper), I am happy about it.
Does anyone know how Hetzner pricing is half of DO yet is profitable, while DO is loss making with 6% operating margin?
Efficiency. They get much more processing power per kw/h of energy than everybode else
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#226Earlier quoted context omitted.
For what it’s worth, I left Fly because of this crap. At first my Fly machine web app had intermittent connection issues to a new production PG machine. Then my PG machine died. Hard. I lost all data. A restart didn’t work - it could not recover. I restored an older backup over at RDS and couldn’t be happier I left.
Fly is in my “try later book” from a year or two ago. I remember it was hard to deploy anything due to downtime so gave up. Sad that stuff like this still happens. You shouldn’t need to multi region a postgres yourself - they should have at least 2 data centre redundancy for the region and it just works. Hope they get some magic sauce to become better at this.
> Hope they get some magic sauce to become better at this.
When I saw them describe their multiregion SQL replication architecture I thought "what crazy person thought this wouldn't eventually open up a spider's nest of distributed systems errors?"
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#227Earlier quoted context omitted.
Does anyone know how Hetzner pricing is half of DO yet is profitable, while DO is loss making with 6% operating margin?
Me and my partner have paid a visit to their datacenter in Nüremberg. The answer is efficiency. They get more processing power than the other providers for the energy they have to put in
What do they do that makes them more efficient?
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#228Y'all, this is going to be deeply unsatisfying, but it's what I can report personally: I have no earthly clue why this thread on our community site is unlisted. We're looking at the admin UI for it right now, and there's like, a little lock next to do the story, but the "unlist story" option is still there for us to click. The best I can say is: I'm reasonably sure there wasn't some top-down edict to hide this thread…
It looks like being authentic is valued over anything else at Fly. I can’t explain how a company responds this immaturely to incidents like these.
Eh, I like it. It's refreshing to see a company representative communicate like an actual human being instead of the usual meaningless corporate robot-speak.
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#229Earlier quoted context omitted.
I moved from DO to Hetzner ( cheaper), I am happy about it.
Does anyone know how Hetzner pricing is half of DO yet is profitable, while DO is loss making with 6% operating margin?
They run their own data centres and have for a while. There is a pretty big industry for that sort of thing as an alternative to “the cloud” here in Europe.
We used to use nianet to house our hardware in Denmark. Basically these companies does hardware renting and they also do hardware renting with more steps which is where you rent rack space but own the hardware. They provide the place for the hardware and they also have multiple locations so that you have both backup and redundancy, and while it doesn’t scale globally in 20 years I’ve literally never worked on anything that needed to beyond having some buffer caches for clients logging in on their vacations or something like that.
What Hetzner seems to be doing with the DO styled hosting, and this is just a guess, is that they are one or the many EU companies preparing for the big EU exodus from the non-EU cloud. Which is frankly a solid bet these days where both AWS and Azure are increasing prices and are becoming more and more unusable because of EU legislation. Part of this is privacy which Microsoft and Amazon are great with in terms of compliance, but part of it is also national security. I work in an investment bank that builds solar plants, since finance and energy are both critical sectors we risk being told that half of the finance/energy companies in the world can’t use Microsoft because the EU seems it as a single point of failure if our entire energy sector relies on Azure. Which is sort of reasonable right? But what this means for us is that we can’t vendor lock-in, not really, because we need to have up-to-date exit strategies for how we plan on being fully operation a month after leaving Azure. Which is easy when you just containerise everything and run it in VMs or similar, and really annoying if you go full in on things like AKS. Which doesn’t help our Azure costs.
Anyway, right now we are planning on leaving Azure because of cost. Not today, not next week but sometime in the next 5-10 years and a lot of these EU cloud alternatives that actually operate the hardware instead of renting it are likely going to be a very realistic alternative. And that is the private sector, I spend time in the EU public sector which is a massive amount of money and I’m guessing it’ll leave both AWS and Azure by 2050. Some of these EU cloud initiatives is going to explode when that happens, and right now, hetzner is one of the best bets.
To get back to your question, DO rents server space. I have no idea where they’d rent it in Germany but they could potentially be renting it from Hetzner.
Re: Fly.io Postgres cluster down for 3 days, no word from them about it
#230Earlier quoted context omitted.
I moved from DO to Hetzner ( cheaper), I am happy about it.
I use both and am very satisfied, especially by Hetzner.
Do they have Terraform providers? And managed Postgres? Besides from the ability to just host a Docker container, that is all I need.