Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Postgres cluster down for 3 days, no word from them about it

webcache.googleusercontent.com

Post reply on HN
Print view View original

281–290 of 493 posts

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#281

Post by andy_ppp »

How many days work is it to build a deployment of an Elixir app with Pulumi, Github Actions and AWS?

As someone not incredibly experienced with devops, I always wonder what is best with databases? Should they be provisioned in Pulumi or do I just manually create them in RDS?

Secrets Manager seems like a bit of a pain point as does IAM which I think I just about understand until I get lost! Giving everything access to ingress and egress also seems a bit overly complex/powerful.

Probably the time to get something working is dramatically shorter than it once was with ChatGPT to help.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#283

Post by yard2010 »

tptacek wrote: post #109
throwaway220033 wrote: post #107

Earlier quoted context omitted.

It looks like being authentic is valued over anything else at Fly. I can’t explain how a company responds this immaturely to incidents like these.

We're just people. We don't have the part of the company that keeps us from communicating like people in public. Maybe we'll grow it someday.

Please don't.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#284

Post by yard2010 »

throwaway290 wrote: post #256
xupybd wrote: post #106

Earlier quoted context omitted.

From this my take away is that I could get fired for picking Fly.io for work. Not because there was an outage but because days could pass before getting support. What assurances could you give the community here that the support would be better next time?

You can/should get fired for picking any plan without proper support guarantees for something serious, regardless of provider.

People here said they have specifically paid for a higher support tier and got no responses.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#285

Post by dpeckett »

dpeckett wrote: post #248
devjab wrote: post #229

Earlier quoted context omitted.

They run their own data centres and have for a while. There is a pretty big industry for that sort of thing as an alternative to “the cloud” here in Europe. We used to use nianet to house our hardware in Denmark. Basically these companies does hardware renting and they also do hardware renting with more steps which is where you rent rack space but own the hardware. They provide the place for the hardware and they als…

Couldn't agree more, I think Hetzner is probably Europe's best bet on a hyperscaler. One of the more telling indicators IMO is their growing market share outside of the EU/DACH. To add on to the comments about Hetzner building their own custom hardware, they also custom built their own software stack. They rejected the hype that was OpenStack and worked diligently on their own hypervisor platform (that they are incre…

For anyone interested in Kubernetes on Hetzner, there's a really interesting CAPI provider being actively developed:

https://github.com/syself/cluster-api-provider-hetzner

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#286

Post by nik736 »

ThePhysicist wrote: post #259

Instances going down happens sporadically on Hetzner Cloud as well, but often by the time I see the e-mail alert that some instance is unreachable I log into the dashboard to find that it has been restarted or migrated to another host already. I've been running a production system there for more than 4 years now and had zero provider-related downtime (as I have some redundancy for most instances). In terms of feature…

Hetzner has a great price/performance ratio, but they are not rock-solid. Speaking of the private network... look at their forum where people complain about downtimes for their "vSwitch" every other week, sometimes it doesn't show up on the status page because it happens on the weekend (lol).

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#287

Post by jinzo »

ThePhysicist wrote: post #259

Instances going down happens sporadically on Hetzner Cloud as well, but often by the time I see the e-mail alert that some instance is unreachable I log into the dashboard to find that it has been restarted or migrated to another host already. I've been running a production system there for more than 4 years now and had zero provider-related downtime (as I have some redundancy for most instances). In terms of feature…

I'm running some instances on Hetzner Cloud, the oldest is ~5 years old, only recently had 2hr or so downtime, other than that - without any problems. And we are talking the cheap ones.

I did have a problem with their dedicated server almost immediately after spinning it up. Noticed that NVMe is broken, and support went like:

- 16:28 -> I contacted them

- 16:36 -> Their first response

- 16:44 -> I sent them SMART data

- 16:48 -> They acknowledged that the NVMe needs replacing and asked me if I consent to that (and loosing of the data that was not already lost -> but running RAID so no problems there)

- 16:52 -> I agreed

- 17:30 -> NVMe was replaced and server booted

I don't have too much experience with hosting providers on that level, but that was freaking impressive response time from them. So a happy camper as well :D

EDIT: Formatting

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#288

Post by almost »

I tried Fly.io when looking to move away from Heroku. Some really cool stuff, I love their focus on multi-region apps. But it just felt like too many under documented things and edge cases and the support didn't seem like it would be there for me when I really needed. I ended up going with NorthFlank as my Heroku-replacement, they've had the odd hicup (mostly related to me being the first customer on their us east region) but communication and support has always been incredible. Really happy I choose them.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#289

Post by gtirloni »

revskill wrote: post #181
bongobingo1 wrote: post #25

Earlier quoted context omitted.

Seems like the OP should have made a HN thread in the first place instead of posting to community.stri^H^H^H^Hfly.io

But HN is not a customer service forum ?

It's often used as an escalation point when people can't get support from certain companies (most notably, Google). If an employee lurks in here and sees your post, they might contact the right people to fix your issue.

Smaller companies also do a lot of PR damage control and constantly monitor HN for threads complaining about their services.

You're not wrong but that's how it works.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#290

Post by truetraveller »

marcinzm wrote: post #260
smallerfish wrote: post #245

Earlier quoted context omitted.

If you lose a single instance on RDS and you don't have replication set up, you'll also have downtime. (Maybe not with Aurora?) And +1 to the sibling comment; Fly makes it very clear that single instance postgres isn't HA, and talks about what you need to do architecturally to maintain uptime.

Downtime but limited downtime since the data is stored with redundantly across multiple machines in the same AZ. So unless the AZ goes down (which is a different failure than what happened here) you can restart the DB on a different instance pretty quickly and I'm guessing AWS will do it automatically for you. edit: Remove triple as not certain about level of redundancy

I don't believe their RDS / EBS has 3x redundancy. With SSD, that would be super costly for them. But if that's correct, that would be incredible.
Post reply on HN