Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Postgres cluster down for 3 days, no word from them about it

webcache.googleusercontent.com

Post reply on HN
Print view View original

91–100 of 493 posts

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#91

Post by NicoJuicy »

steve_adams_86 wrote: post #31
mrcwinn wrote: post #18

Earlier quoted context omitted.

For what it’s worth, I left Fly because of this crap. At first my Fly machine web app had intermittent connection issues to a new production PG machine. Then my PG machine died. Hard. I lost all data. A restart didn’t work - it could not recover. I restored an older backup over at RDS and couldn’t be happier I left.

I left digitalocean for fly because some of their tooling was excellent. I was pretty excited. I’m back on digitalocean now. I’m not unhappy about it, they’re very solid. I don’t love some things about their services, but overall I’d highly recommend them to other developers. I gave up on fly because I’d spontaneously be unable to automate deployments due to limited resources. Or I’d have previously happy deployments…

I moved from DO to Hetzner ( cheaper), I am happy about it.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#92

Post by teraflop »

tptacek wrote: post #84
teraflop wrote: post #83

Earlier quoted context omitted.

There's also a lock icon next to the "App not working" category in the header, which I took to mean that that entire category is hidden from logged-out users (which experimentally seems to be the case).

I have the impression from this thread that this thread was public (as in, would work if you just linked to it from something like HN) earlier, and now it isn't? Obviously, deliberately hiding a negative story on our Discourse is a little like deleting a bad tweet; it's just going to guarantee someone captures and boosts it. We have a lot of flaws! But not knowing how the Internet works probably isn't one of them. No…

Yes, from the Google-cached version, it appears that the thread previously didn't have the app-not-working tag; it was only tagged with "rails".

Not going to try and guess why or when that tag change happened. Personally, I'm less concerned with this particular thread than with the apparent decision to systematically hide all potentially-negative threads from search engines.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#93

Post by michaeldwan »

tptacek wrote: post #80

Y'all, this is going to be deeply unsatisfying, but it's what I can report personally: I have no earthly clue why this thread on our community site is unlisted. We're looking at the admin UI for it right now, and there's like, a little lock next to do the story, but the "unlist story" option is still there for us to click. The best I can say is: I'm reasonably sure there wasn't some top-down edict to hide this thread…

I don't know why the app-not-working category effectively delists threads, but until we find out, I just removed it so this thread is public again.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#94

Post by q7xvh97o2pDhNrh »

aledalgrande wrote: post #41
q7xvh97o2pDhNrh wrote: post #34

Earlier quoted context omitted.

Maybe just pick up 3 chonky EC2 boxes, set up iptables on each of them, have each one run a containerized version of your code that gets built and deployed from CI every time you push to Github, slap an ALB in front of it all, and call it a day? And if you need state, then spin up a little RDS with your favorite SQL flavor of choice? The CI deploy script could even bake in little health-checks so you can do rolling d…

The system you describe is quite the monthly bill, off the top of my head.

I admit I didn't run the numbers before posting that. But you got me curious, so I went ahead and did it now...

Render.com looks like [1] their "$0 + compute costs" plan would work out to:

  ∙ $25/mo for a single "Web Services" box of 1 CPU and 2GB RAM
  ∙ $20/mo for a single "PostgreSQL" box of 1 GB RAM, 1 CPU, and 16GB SSD
  ∙ TOTAL: $45/mo, and you're assuming they'll magically give you zero-downtime
Those are grim numbers, performance-wise, but let's use them as the standard and see what it'd cost in the scrappy AWS architecture I threw together in a few minutes:

  ∙ $12.10/mo for a single t4g.small box, which is actually 2 vCPU and 2GB RAM [2]
  ∙ 3x redundancy on that brings you up to $36.30/mo for compute
  ∙ $16.20/mo for an ALB [3]
  ∙ $11.52/mo for a single db.t4g.micro PostgreSQL box, plus $1.84/mo for the equivalent 16GB of storage [4]
  ∙ TOTAL: $65.86/mo for substantially more CPU, redundancy, and control, or...
  ∙ TOTAL: $41.66/mo for substantially more CPU and control over your infra, if you're willing to drop the redundancy
So it looks like it's pretty comparable in terms of raw dollars.

I'll admit there's a little more "devops" overhead with the AWS setup. Though I think it's not as big of a deal as people make it out to be — it's basically an afternoon of Terraforming, and you'd probably spend an equal or greater amount of time digging through Render's docs to understand their bespoke platform anyway.

(Also, once you contemplate bulk pricing for the underlying commodities, it's easy to see how companies like Render make a healthy margin, even on their low-end offerings.)

Anyway, I guess I've nerd-sniped myself, so I'd better stop here. But that was a fun analysis!

[1] https://render.com/pricing#compute

[2] https://aws.amazon.com/ec2/pricing/on-demand/

[3] https://aws.amazon.com/elasticloadbalancing/pricing/

[4] https://aws.amazon.com/rds/postgresql/pricing/?pg=pr&loc=3

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#95

Post by tptacek »

sho wrote: post #82
tptacek wrote: post #80

Y'all, this is going to be deeply unsatisfying, but it's what I can report personally: I have no earthly clue why this thread on our community site is unlisted. We're looking at the admin UI for it right now, and there's like, a little lock next to do the story, but the "unlist story" option is still there for us to click. The best I can say is: I'm reasonably sure there wasn't some top-down edict to hide this thread…

> I have no earthly clue why this thread on our community site is unlisted. Maybe it's hosted in the SYD region

It's hosted by Discourse.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#96

Post by unmole »

CameronNemo wrote: post #75
unmole wrote: post #27

Earlier quoted context omitted.

> use S3 or run minio Is using Cloudflare R2 not an option?

Backblaze even has an s3 api these days.

Backblaze B2 is much older than Cloudflare's offering. But from what I remember, they didn't have any presence outside North America.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#97

Post by NBJack »

tptacek wrote: post #95
sho wrote: post #82

Earlier quoted context omitted.

> I have no earthly clue why this thread on our community site is unlisted. Maybe it's hosted in the SYD region

It's hosted by Discourse.

Another good reason to avoid that platform like the plague.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#98

Post by subarctic »

It's really disappointing that they made this forum thread private, apparently in response to this HN thread blowing up. This is the first negative HN thread I've seen about them, it's not even really that bad because this kind of downtime is expected, and they can't get to every forum post, and their response that someone posted here is totally reasonable in my opinion.

So why is the link to the thread 404ing and why does this post have to link to google webcache of it? I've grown to like fly.io and use them for my side projects now, and this just isn't sometime they would do. Going through some minor cognitive dissonance right now :/

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#99

Post by tptacek »

subarctic wrote: post #98

It's really disappointing that they made this forum thread private, apparently in response to this HN thread blowing up. This is the first negative HN thread I've seen about them, it's not even really that bad because this kind of downtime is expected, and they can't get to every forum post, and their response that someone posted here is totally reasonable in my opinion. So why is the link to the thread 404ing and wh…

(a) Not even close to the first negative HN thread about us.

(b) We definitely didn't make the thread private in response to HN.

(c) It should be public again.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#100

Post by tetha »

jdkfoo wrote: post #29
mrcwinn wrote: post #13

Earlier quoted context omitted.

Yes, this. It's tough when you've already played your "we messed up but we're making it right" card, and then you continue to not have it right.

Hosting service that cannot get basics right after a decade plus of solving these problems as an industry. Are we even trying or just repeating ourselves because we don’t know what else to do? How can the entire industry keep making the same basic errors? “Let’s keep it simp… ohh nope we invented a Turing complete language and customer service is terri… wait do we have customer service?” I get the world turning again…

The thing is, running a good SaaS service requires quite a bit of staff and hard operational skills and a lot of manpower. You know, the kinda stuff people always call useless, zero-value add, blockers and entirely to automate.

Sure, we have most of the day-to-day grunt work for our applications automated. But good operations is just more. It's more about maintaining control over your infrastructure at one hand, and making sure your customers feel informed and safe about their data and systems. This is hard and takes lots of experience to do well, as well as manpower.

And yes, that's entirely a soft skill. You end up with questions such as: Should we elevate this issue to an outage on the status page? To a degree you'd be scaring other customers. "Oh no, yellow status page. Something terrible must happen!". At the same time you're communicating to the affected customers just how serious you're taking their issues. "It's a thing on the status page after an initial misjudgement - sorry for that." We have many discussions ilke that during degradations and outages.

Post reply on HN