Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Postgres cluster down for 3 days, no word from them about it

webcache.googleusercontent.com

Post reply on HN
Print view View original

151–160 of 493 posts

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#151

Post by tinco »

xupybd wrote: post #106
tptacek wrote: post #80

Y'all, this is going to be deeply unsatisfying, but it's what I can report personally: I have no earthly clue why this thread on our community site is unlisted. We're looking at the admin UI for it right now, and there's like, a little lock next to do the story, but the "unlist story" option is still there for us to click. The best I can say is: I'm reasonably sure there wasn't some top-down edict to hide this thread…

From this my take away is that I could get fired for picking Fly.io for work. Not because there was an outage but because days could pass before getting support. What assurances could you give the community here that the support would be better next time?

Try filing a bug with any of the big three cloud vendors when you're on their free plan. It's really not different, the thing that is going to get you fired is not realizing you're not paying a couple hundred bucks per month for premium service on the infrastructure that is mission critical to your company.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#152

Post by remus »

quickthrower2 wrote: post #143
spiderice wrote: post #14

There is now a response to the support thread from Fly[1]: > Hi Folks, > Just wanted to provide some more details on what happened here, both with the thread and the host issue. > The radio silence in this thread wasn’t intentional, and I’m sorry if it seemed that way. While we check the forum regularly, sometimes topics get missed. Unfortunately this thread one slipped by us until today, when someone saw it and flag…

> We strongly recommend running multiple instances to mitigate the impact of single-host failures like this. Make it impossible not to do so, and make it frictionless then.

That would presumably cost more money which is not a trade off every user would want to make.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#153

Post by pinkcan »

system2 wrote: post #137
wg0 wrote: post #132

Not a sarcastic or rhetorical question - how come the three big A clouds or even smaller ones (Hetzner,my favorite) are mostly so stable (give or take some outages) and anyone knows their internal engineering, architecture and practices to keep systems that much stable?

IMHO it is their approach. I use Hetzner and OVH (and their other variants for lower budget clients) for our EU clients. They do not use buzz words like "deploy app server", "cloud clusters", "turbo charge this app". They are simply providing VPS and similarly configured droplets. They are also established and don't want to mess around with very modern experimental infrastructures. Same goes for Digital Ocean. No buz…

DO, OVH, and Hetzner are more stable because they don't use buzzwords?

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#155

Post by fredrikholm »

thyrox wrote: post #112

You know what's interesting? It feels like history is repeating itself with Fly.io, just like it did back when I first encountered Heroku. Back in the day, I was super excited about Fly.io – it had that same fresh, exciting vibe that Heroku had when it burst onto the scene. I remember being blown away by Fly.io's simplicity and how easy it was to use. It was like hosting made simple, and I couldn't help but think, "T…

I feel the same way.

Do you think its related to scale? As in, once a company has enough paying customers to become profitable/investable, it has also accrued enough issues to where it starts feeling fresh and exciting like you said, and gradually becomes like the older competitor it once wanted to replace?

This is my experience at least. Once the company goes from a few pizzas to "we've booked a venue", entropy creeps in and adages like Conway's/Brook's law become increasingly evident.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#156

Post by stevefan1999 »

YetAnotherNick wrote: post #150
NicoJuicy wrote: post #91

Earlier quoted context omitted.

I moved from DO to Hetzner ( cheaper), I am happy about it.

Does anyone know how Hetzner pricing is half of DO yet is profitable, while DO is loss making with 6% operating margin?

Simple, Hetzner mainly operates on Germany, the people are mostly Germans, and they automate the stuff to a point a small team could manage it well even if not remotely, so they have less cost on human resources.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#157

Post by pinkcan »

tptacek wrote: post #99
subarctic wrote: post #98

It's really disappointing that they made this forum thread private, apparently in response to this HN thread blowing up. This is the first negative HN thread I've seen about them, it's not even really that bad because this kind of downtime is expected, and they can't get to every forum post, and their response that someone posted here is totally reasonable in my opinion. So why is the link to the thread 404ing and wh…

(a) Not even close to the first negative HN thread about us. (b) We definitely didn't make the thread private in response to HN. (c) It should be public again.

what's up with the status page?

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#159

Post by mythz »

NicoJuicy wrote: post #91
steve_adams_86 wrote: post #31

Earlier quoted context omitted.

I left digitalocean for fly because some of their tooling was excellent. I was pretty excited. I’m back on digitalocean now. I’m not unhappy about it, they’re very solid. I don’t love some things about their services, but overall I’d highly recommend them to other developers. I gave up on fly because I’d spontaneously be unable to automate deployments due to limited resources. Or I’d have previously happy deployments…

I moved from DO to Hetzner ( cheaper), I am happy about it.

Same, been enjoying Hetzner's great value for 10 years, and now Hetzner Cloud for 2 years.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#160

Post by pritambarhate »

Here the even bigger red flag is that Fly doesn't have a (automated?) way to quickly move workload from a faulty server to a good server. Especially when containers (and orchestrators) have abstracted away the concept of data volumes which can be attached and detached. (Yes, it needs a lot of serious technical investment to provide this and I think it's one of the reasons storage is expensive on the big 3 clouds.) If you are offering data persistence services then you absolutely need this capability.

I think there is an expectation mismatch between what Fly wants to offer and what the market wants from it. Fly wanted to innovate on offering the ability to the devs to be able run their apps from multiple data centers. But without a proper data persistence service, the ability to run apps from multiple data centers is not useful to a vast majority of people.

I think Fly is trying to solve the persistence issue with their SQLite replication, but that means the vast majority of the devs will have to change the way they develop applications to suit Fly platform.

I think Fly needs to choose between what it wants to become. A reliable and affordable Heroku replacement, which is a decent sized market or offer an opinionated way of developing apps which offer best performance to users all around the world.

But opinionated ways of doing things is a double edged sword. (Rails and Spring Boot are highly successful because of their opinionated defaults.) App Engine is an interesting case study in the app hosting domain. It was way ahead of the time and prescribed you a way of developing apps which allowed the apps to scale to very high traffic. But people didn't want to change the way they develop to adapt to it.

Post reply on HN