Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Postgres cluster down for 3 days, no word from them about it

webcache.googleusercontent.com

Post reply on HN
Print view View original

251–260 of 493 posts

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#251

Post by tinco »

mst wrote: post #213
xupybd wrote: post #165

Earlier quoted context omitted.

Funny story, when I started my current role I researched our hosting provider. I couldn't find the matching invoices in the accounting system. So I called the vendor, a local company. They'd not set our account up correctly, billing was not enabled. Since then we've been billed. I'm glad we sorted it but it wasn't a good look to start my role by increasing our spending.

I feel like starting your role by discovering a crucial service wasn't being paid for and therefore was at risk of suddenly going away should be a pretty positive thing. However 'should' is pretty load bearing there and actual results are probably heavily dependent on management culture and the current state of office politics.

We had a customer once that our automatic billing system tried to reach for 3 months about failing credit card charges (<$5k/mo). Our system stopped the service.. I'm pretty sure their subsequent outage cost their customers millions. Lessons about what it means to have (and be) enterprise customers were learned. Unfortunately the lady who was ignoring our e-mails in her inbox got fired.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#252

Post by Takennickname »

Dylan16807 wrote: post #104
Fire-Dragon-DoL wrote: post #49

Earlier quoted context omitted.

I don't have to use an object store, but it makes the cost of setting up a server more expensive if I use the filesystem, if I delete the instance, the data is gone. A volume kinda offset this, but it's way less portable and accessible only by one instance at a time The peace of mind of managed is nice, all I have to think about is running the app, without having to deal with making sure db and files don't get lost

At that level I think I'd just put the images in the database.

In the git repo even

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#253

Post by d_k_f »

kristiandupont wrote: post #230
tacker2000 wrote: post #171

Earlier quoted context omitted.

I use both and am very satisfied, especially by Hetzner.

Do they have Terraform providers? And managed Postgres? Besides from the ability to just host a Docker container, that is all I need.

Yes and (unfortunately) no. Terraform providers are here [1] with the official documentation at [2]. Managed databases are not available, though. I think they have some sort of database offering if you select their web hosting options, but you can't just get a managed Postgres instance yourself.

[1] https://registry.terraform.io/providers/hetznercloud/hcloud/... [2] https://community.hetzner.com/tutorials/howto-hcloud-terrafo...

EDIT: For what it's worth, I have had good experiences with app servers hosted on Hetzner Cloud and managed Postgres provided by ElephantSQL (https://www.elephantsql.com/) for Germany-based apps.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#254

Post by preseinger »

TekMol wrote: post #233
danjac wrote: post #218

Earlier quoted context omitted.

Not sure what fly.io offers vs Heroku or others (I have played with it some time ago but not used for anything serious), but for an equivalent I'd be looking for automated load balancer setup with SSL, easy scaling up so I can go from 1 to 2 or however many web services (with UI or CLI), simple deployment configuration with a Procfile (or whatever) and managed PostgreSQL/MySQL/Redis including backup/restore when need…

And what kind of project do you run which needs up/down scaling and load balancing? In my experience, for a simple PHP web application, the smallest VMs already can handle a thousand concurrent users, which amounts to something like a million monthly users.

and what does your experience tell you about applications which are not written in PHP, and which need to handle more than 1000 concurrent users?

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#256

Post by throwaway290 »

xupybd wrote: post #106
tptacek wrote: post #80

Y'all, this is going to be deeply unsatisfying, but it's what I can report personally: I have no earthly clue why this thread on our community site is unlisted. We're looking at the admin UI for it right now, and there's like, a little lock next to do the story, but the "unlist story" option is still there for us to click. The best I can say is: I'm reasonably sure there wasn't some top-down edict to hide this thread…

From this my take away is that I could get fired for picking Fly.io for work. Not because there was an outage but because days could pass before getting support. What assurances could you give the community here that the support would be better next time?

You can/should get fired for picking any plan without proper support guarantees for something serious, regardless of provider.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#257

Post by heeton »

arrowsmith wrote: post #231
asaddhamani wrote: post #60

Earlier quoted context omitted.

Curious to know, have you tried Render? What is the successor to Heroku in your eyes?

If you're deploying an Elixir/Phoenix app, then Gigalixir has worked really well for me. It's expensive, but then so is Heroku.

What’s their reliability been like?

Am I right in thinking the platform got bought a little while ago, and it’s being run by a relatively small outfit?

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#258

Post by heartbreak »

subarctic wrote: post #146
throwaway220033 wrote: post #107

Earlier quoted context omitted.

It looks like being authentic is valued over anything else at Fly. I can’t explain how a company responds this immaturely to incidents like these.

If you're talking about the comment you're replying to, tbh I found it was way more relatable than a more "professional" PR-speak response. Maybe you were talking about something else

Unfortunately PR-speak exists for a reason.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#259

Post by ThePhysicist »

Instances going down happens sporadically on Hetzner Cloud as well, but often by the time I see the e-mail alert that some instance is unreachable I log into the dashboard to find that it has been restarted or migrated to another host already. I've been running a production system there for more than 4 years now and had zero provider-related downtime (as I have some redundancy for most instances). In terms of features they move way slower than Fly.io and it took them years adding stuff like virtual networking, but everything they add works rock-solid. I guess there are just very different engineering cultures when it comes to building cloud infrastructure provider, and I have to say I prefer the "take your time and do it right" approach.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#260

Post by marcinzm »

smallerfish wrote: post #245
benjaminwootton wrote: post #184

Earlier quoted context omitted.

Should losing a single host machine be a big deal nowadays? Instance failure is a fact of life. Even if customers are only running one instance, I would expect the whole thing to rebalance in an automated way especially with fly.io being so container centric. It also sounds like this is some managed Postgres service rather than users running only one instance of their container, so it’s even more reasonable to expect…

If you lose a single instance on RDS and you don't have replication set up, you'll also have downtime. (Maybe not with Aurora?) And +1 to the sibling comment; Fly makes it very clear that single instance postgres isn't HA, and talks about what you need to do architecturally to maintain uptime.

Downtime but limited downtime since the data is stored with redundantly across multiple machines in the same AZ. So unless the AZ goes down (which is a different failure than what happened here) you can restart the DB on a different instance pretty quickly and I'm guessing AWS will do it automatically for you.

edit: Remove triple as not certain about level of redundancy

Post reply on HN