Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Postgres cluster down for 3 days, no word from them about it

webcache.googleusercontent.com

Post reply on HN
Print view View original

471–480 of 493 posts

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#471

Post by steve_adams_86 »

dbingham wrote: post #327
danielvaughn wrote: post #56

Earlier quoted context omitted.

I adore DO. They’re seriously underrated. I love how they’ll just give you a server and say here, have at it. No abstractions, no fancy crap, just get out of my way and let me do my thing.

I love DO for projects where I don't need control. For my side project, I eventually migrated to AWS after running into a lot of issues with DO. Things like they don't give you the postgres root user on their managed postgres. And I ran into issues trying to capture the deployments in code. Their terraform providers are pretty good, but still leave something to be desired. For all its many warts, I'm much happier bac…

Those are good things to know. I’ve been wondering about their managed databases recently, so I’ll keep that in mind.

I’m nowhere near as picky as you are, but maybe I’ll need to be at some point. As it is I mostly just build stuff and send it to the internet. If it builds and it does what I expected, I’m pretty happy! I don’t often need anything too special.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#472

Post by steve_adams_86 »

graypegg wrote: post #354
steve_adams_86 wrote: post #31

Earlier quoted context omitted.

I left digitalocean for fly because some of their tooling was excellent. I was pretty excited. I’m back on digitalocean now. I’m not unhappy about it, they’re very solid. I don’t love some things about their services, but overall I’d highly recommend them to other developers. I gave up on fly because I’d spontaneously be unable to automate deployments due to limited resources. Or I’d have previously happy deployments…

DO actually does have a free tier! If you use their “app platform” (their equivalent to fly/heroku/render/etc) you can host 3 “static” apps for free. So if you have a Hugo/Jekyll blog or something, it’ll set up a whole little CD system for it for free.

You’re totally right. I kind of forgot about this, in part because I’m over their free limit. I think their static sites are still dirt cheap once you hit that limit, though. I find their pricing totally reasonable for what I need.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#473

Post by weird-eye-issue »

weird-eye-issue wrote: post #463
say_it_as_it_is wrote: post #380

Earlier quoted context omitted.

Are you a paying customer to any hosting cloud provider?

We are on Cloudflare Enterprise. We also pay over $20k/mo to OpenAI and are on their Enterprise network too. So, yes.

We also pay for Supabase

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#474

Post by Dylan16807 »

tinco wrote: post #423
CSSer wrote: post #383

Earlier quoted context omitted.

I'm surprised by your risk tolerance. If I had any cloud service at this level in my stack go down for three days, I'd start shopping for an alternative. This exceeds the level of acceptability for me for even non-HA requirements. After all, if I can't trust them for this, why would I ever consider giving them my HA business? Just based on napkin math for us, this could've been a potential loss of nearly half a milli…

I think you're not exposed enough to the reality of hardware. There was no need for the host to come back online at all. I think it was a mistake of Fly.io to even attempt to do it. Just say tell the customer the host was lost and offer them a new one (with a freshly zeroed volume attached). You rent a machine, it breaks, you get a new one. If they're sad that they lost their data, it's their fault for running on a s…

They're not responsible for extreme data recovery, but (almost?) all of the customer data volumes on that server were completely intact. They damn well should be responsible for getting that data back to their customers, whether or not they get the server going again.

If you run off a single drive, and the drive dies, any resulting data loss is your fault. But not if something else dies.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#475

Post by abwizz »

ushakov wrote: post #350
abwizz wrote: post #247

Earlier quoted context omitted.

i'll guess they pick optimized components for it. like the longtime workhorse was a high performance skylake desktop cpu w/o ecc ram

The secret is in the cooling system. They have individual cooling systems for each server. Less heat = longer sustained loads

pardon my ingorance but i cannot quite see how cooling individual machines vs. the hole rack or row makes a difference in total heat production per machine

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#476

Post by bongobingo1 »

Fire-Dragon-DoL wrote: post #459
Dylan16807 wrote: post #104

Earlier quoted context omitted.

At that level I think I'd just put the images in the database.

That's an option, but I want to keep things simple and the assumption is usually "filesystem" but weirdly most libraries assume S3 usage. I don't think I've seen native support for db-stored images in any of the libraries I use, which is sad but a reality.

Ya cant just throw em at a blob type field/column?

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#477

Post by markonen »

Dylan16807 wrote: post #474
tinco wrote: post #423

Earlier quoted context omitted.

I think you're not exposed enough to the reality of hardware. There was no need for the host to come back online at all. I think it was a mistake of Fly.io to even attempt to do it. Just say tell the customer the host was lost and offer them a new one (with a freshly zeroed volume attached). You rent a machine, it breaks, you get a new one. If they're sad that they lost their data, it's their fault for running on a s…

They're not responsible for extreme data recovery, but (almost?) all of the customer data volumes on that server were completely intact. They damn well should be responsible for getting that data back to their customers, whether or not they get the server going again. If you run off a single drive, and the drive dies, any resulting data loss is your fault. But not if something else dies.

I'm absolutely 100% certain that AWS (for example) wouldn't do that for you with the instance types that feature direct attached storage.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#478

Post by justinclift »

skullone wrote: post #438
justinclift wrote: post #200

Earlier quoted context omitted.

Have you come across Fujitsu PRIMERGY servers before? https://www.fujitsu.com/global/products/computing/servers/pr... I used to use them a few years ago in a local data centre, and they were pretty good back then. They don't seem to be widely known about though.

Have not - looks nice though. Around here, you'll mostly only encounter the Dell/Supermicro/HP/Lenovo. I actually find Dell to have acheived the lowest "friction" for deployments. You can get device manifests before the gear even ships, including MAC addresses, serials, out of band NIC MAC, etc. We pre-stage our configurations based on this, have everything ready to go (rack location/RU, switch ports, PDUs, DHCP/DNS)…

> You can get device manifests before the gear even ships, including MAC addresses, serials, out of band NIC MAC, etc.

That does sound pretty useful.

So for yourselves, you rack them then run hardware qualification tests?

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#479

Post by steve_adams_86 »

steve_adams_86 wrote: post #468
robertlagrant wrote: post #214

Earlier quoted context omitted.

> I need a service that’ll let me just build my goddamn project. This was a small static site built by Node, so nothing spectacular happening (Cloudflare|GitHub|GitLab) Pages should do you nicely!

I actually use digitalocean and it’s pretty solid for static sites (they’re free, I think). It’s also convenient because that’s where pretty much all of my stuff lives these days. I used to put piles of stuff on GitHub pages though! I have some great memories of learning how awesome static sites could be, and how cool it was that they’d deploy just by pushing your repository. That seemed like magic back then.

Free up until 3 static sites, that’s what it was (on the app platform)

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#480

Post by apple4ever »

danielvaughn wrote: post #56
steve_adams_86 wrote: post #31

Earlier quoted context omitted.

I left digitalocean for fly because some of their tooling was excellent. I was pretty excited. I’m back on digitalocean now. I’m not unhappy about it, they’re very solid. I don’t love some things about their services, but overall I’d highly recommend them to other developers. I gave up on fly because I’d spontaneously be unable to automate deployments due to limited resources. Or I’d have previously happy deployments…

I adore DO. They’re seriously underrated. I love how they’ll just give you a server and say here, have at it. No abstractions, no fancy crap, just get out of my way and let me do my thing.

I really love DO except for one thing - you can't run your own firewall/router there (like opnSense). Really hard to link systems together.
Post reply on HN