Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Postgres cluster down for 3 days, no word from them about it

webcache.googleusercontent.com

Post reply on HN
Print view View original

51–60 of 493 posts

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#51

Post by eropple »

erulabs wrote: post #16
orangepurple wrote: post #10

Earlier quoted context omitted.

AWS support replies back to your messages when they feel like it. Their support is just as shady but they have better uptime for sure

FWIW, our aws enterprise support reps are available 24/7 and usually respond within a few minutes. But again, you get what you pay for.

I was working for a pretty big early AWS customer--one that had realized that for the low low price of all your money you could make DynamoDB scale to some truly massive numbers--and one time when we were having trouble around noon Eastern, a colleague called up our TAM. As he told it, the TAM sounded half-asleep, so my colleague asked if everything was alright.

"I'm in Hawaii on my honeymoon and my backup missed your call, so it escalated."

I probably wouldn't have answered the phone. Granted, that's why I don't do that job. But I have always had a real appreciation for the good TAMs ever since.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#53

Post by [deleted] »

rendaw wrote: post #47
pech0rin wrote: post #28

I really want to love Fly.io. It's super easy to get setup and use, but to be honest I don't think anyone should be building mission critical applications on their service. I ended up migrating everything over to AWS (which I reallllly didn't want to do) because: * Frequent machines not working, random outages, builds not working * Support wasn't responsive, didn't read my questions (kept asking same questions over a…

Half the critical info for using their services is buried in some thread in the forum (posted by an employee). How bad is their documentation pipeline that they can't with similar effort get that same info in the documentation? Requests to put stuff in the docs go ignored. The answer to _any_ usage related forum question should be: 1. It's in the documentation (maybe I just added it) 2. If you're left with any confus…

[deleted]

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#54

Post by cschmatzler »

windexh8er wrote: post #30
marban wrote: post #8

They're less humble in communicating other things https://fly.io/blog/we-raised-a-bunch-of-money/

I tried Fly once, but, at the end of the day it seemed way too expensive for what it was and the completeness of the vision. And then I started to see the complaints in random corners of the Internet. I don't read their blog regularly but I always thought they had great content. But not after reading this. The irony: "What people actually wanted to talk about, though? Databases." ...but apparently not when they are t…

Their blog is great, because they invested heavily in perception from the outside. Coming from the Elixir world, them hiring Chris McCord (creator of Phoenix) and sponsoring a ton of open source projects slapping on their logo, seemed great at first, but when it comes to actually deploying stuff to production and day 2 operations (monitoring is so much more difficult than it should be, and troubleshooting tools are lacking) they are way behind. I can imagine them getting lots of hobby projects on board due to free tier and day 1 impression, but that won’t win over enterprises.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#56

Post by danielvaughn »

steve_adams_86 wrote: post #31
mrcwinn wrote: post #18

Earlier quoted context omitted.

For what it’s worth, I left Fly because of this crap. At first my Fly machine web app had intermittent connection issues to a new production PG machine. Then my PG machine died. Hard. I lost all data. A restart didn’t work - it could not recover. I restored an older backup over at RDS and couldn’t be happier I left.

I left digitalocean for fly because some of their tooling was excellent. I was pretty excited. I’m back on digitalocean now. I’m not unhappy about it, they’re very solid. I don’t love some things about their services, but overall I’d highly recommend them to other developers. I gave up on fly because I’d spontaneously be unable to automate deployments due to limited resources. Or I’d have previously happy deployments…

I adore DO. They’re seriously underrated. I love how they’ll just give you a server and say here, have at it. No abstractions, no fancy crap, just get out of my way and let me do my thing.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#57

Post by justinclift »

skullone wrote: post #22
jssjr wrote: post #15

Earlier quoted context omitted.

I appreciate the honest feedback. We could have done better communicating about the problem. We've been marking single host failures in the dashboard for affected users and using our status page to reflect things like platform and regional issues, but there's clearly a spot in the middle where the status we're communicating and actual user experience don't line up. We've been adding a ton more hardware lately to stay…

What hardware are you buying? Across tens of thousands of physical nodes in my environment, only a few would have "fatal" enough problems that required manual intervention per year. Yes we had hundreds of drives die a year, some ECC ram would exceed error thresholds, but downtime on any given node was rare (aside from patching, but we'd just live migrate KVM instances around as needed.

Maybe there needs to be a better "burn in" test setup for their new hardware, just to catch mistakes in the build prep and/or catch bad hardware?

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#58

Post by justinclift »

eropple wrote: post #51
erulabs wrote: post #16

Earlier quoted context omitted.

FWIW, our aws enterprise support reps are available 24/7 and usually respond within a few minutes. But again, you get what you pay for.

I was working for a pretty big early AWS customer--one that had realized that for the low low price of all your money you could make DynamoDB scale to some truly massive numbers--and one time when we were having trouble around noon Eastern, a colleague called up our TAM. As he told it, the TAM sounded half-asleep, so my colleague asked if everything was alright. "I'm in Hawaii on my honeymoon and my backup missed you…

Wonder if that marriage lasted though? ;)

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#59

Post by justinclift »

aledalgrande wrote: post #41
q7xvh97o2pDhNrh wrote: post #34

Earlier quoted context omitted.

Maybe just pick up 3 chonky EC2 boxes, set up iptables on each of them, have each one run a containerized version of your code that gets built and deployed from CI every time you push to Github, slap an ALB in front of it all, and call it a day? And if you need state, then spin up a little RDS with your favorite SQL flavor of choice? The CI deploy script could even bake in little health-checks so you can do rolling d…

The system you describe is quite the monthly bill, off the top of my head.

You can do the same thing using Hetzner dedicated hosts fairly cheaply:

https://www.hetzner.com/dedicated-rootserver/matrix-ax

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#60

Post by asaddhamani »

pech0rin wrote: post #28

I really want to love Fly.io. It's super easy to get setup and use, but to be honest I don't think anyone should be building mission critical applications on their service. I ended up migrating everything over to AWS (which I reallllly didn't want to do) because: * Frequent machines not working, random outages, builds not working * Support wasn't responsive, didn't read my questions (kept asking same questions over a…

Curious to know, have you tried Render? What is the successor to Heroku in your eyes?
Post reply on HN