Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Postgres cluster down for 3 days, no word from them about it

webcache.googleusercontent.com

Post reply on HN
Print view View original

451–460 of 493 posts

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#451

Post by camgunz »

marcinzm wrote: post #311
tptacek wrote: post #80

Y'all, this is going to be deeply unsatisfying, but it's what I can report personally: I have no earthly clue why this thread on our community site is unlisted. We're looking at the admin UI for it right now, and there's like, a little lock next to do the story, but the "unlist story" option is still there for us to click. The best I can say is: I'm reasonably sure there wasn't some top-down edict to hide this thread…

Honest advice, probably to Kurt rather than you, is you need better processes, accountability and (probably) communication in your company. The tone of your reply (and other communications from fly.io) is reflective of the lack of those things given the public sentiment regarding fly.io. At 60+ employees and so many issues that tone goes from humanly endearing to indicative of a non-scaling business. Other replies in…

For an opposing viewpoint: I don't want HN to become the place where corporate comms comes to bullshit us. I want engineers who work there to talk to us as peers, which seems like what's happening here. I get candor and humility (and playfulness, sure) from Fly's tone, which I appreciate.

I get stuff like this is frustrating. But I bet Fly staff are pretty frustrated too.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#453

Post by CSSer »

tinco wrote: post #423
CSSer wrote: post #383

Earlier quoted context omitted.

I'm surprised by your risk tolerance. If I had any cloud service at this level in my stack go down for three days, I'd start shopping for an alternative. This exceeds the level of acceptability for me for even non-HA requirements. After all, if I can't trust them for this, why would I ever consider giving them my HA business? Just based on napkin math for us, this could've been a potential loss of nearly half a milli…

I think you're not exposed enough to the reality of hardware. There was no need for the host to come back online at all. I think it was a mistake of Fly.io to even attempt to do it. Just say tell the customer the host was lost and offer them a new one (with a freshly zeroed volume attached). You rent a machine, it breaks, you get a new one. If they're sad that they lost their data, it's their fault for running on a s…

In hindsight I wish I could edit because my above comment was pretty trigger happy and focused overly focused on the amount of downtime. It was colored by some existing preconceptions I had about Fly, and I'm honestly surprised it continues to be upvoted. When I made this comment I hadn't yet learned some of the bits you mentioned here at the end from another thread. Anyway, I tend to agree overall. I actually suggested Fly even reconsider offering this configuration given that they refer to it as a "single-node cluster", which is an oxymoron.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#454

Post by CSSer »

mrkurt wrote: post #418
CSSer wrote: post #383

Earlier quoted context omitted.

I'm surprised by your risk tolerance. If I had any cloud service at this level in my stack go down for three days, I'd start shopping for an alternative. This exceeds the level of acceptability for me for even non-HA requirements. After all, if I can't trust them for this, why would I ever consider giving them my HA business? Just based on napkin math for us, this could've been a potential loss of nearly half a milli…

You're saying a single server failure is going to to cost your business half a million dollars? This was a server with local NVMe storage. The simplest thing to do would have been to just get rid of it, but we have quite a few free users with data they care about running on single node Postgres (because it's cheaper). It seemed like a better idea to recover this thing.

No, it wouldn't, at least not given the contextual details of this situation because we wouldn't do that. Honestly there are parts of my above comment that hold but I admit in the moment that it was a bit impulsive of me because I hadn't yet learned all of the details necessary to make that judgment call. That number is right under slightly different circumstances if you're asking, but it sounds like you were trying to prove a point. If that's true, you succeeded. I learned a bit later that what they were calling a cluster was a single server and that's just... yeah.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#455

Post by CSSer »

TheDong wrote: post #404
CSSer wrote: post #400

Earlier quoted context omitted.

I see. Have you considered eliminating this configuration from your offering? It sounds like the terminology could confuse people, and it may be the case that they're assuming that a host isn't really what it is (a single host). This kind of thing is difficult for those seeking to build managed services, because I think people expect you to provide offerings that can't harm them when the cause is related to the servi…

Plenty of people would rather take downtime than pay for redundancy, for example for a test database. AWS RDS lets you spin up a RDS instance that costs 3x less and regularly has downtime (the 'single-az' one), quite similar to this. Anyone who's used servers before knows "A single instance" is the same as "sometimes you might have downtime". Computers aren't magic, everyone from heroku (you must have multiple dynos…

I don't disagree. I was latching onto the idea that people are running single-node "clusters". Whatever it is, it isn't a cluster.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#456

Post by kunley »

tptacek wrote: post #413
kunley wrote: post #406

Earlier quoted context omitted.

> somehow a containerd boltdb on that host got corrupted, so when the machine bounced, containerd refused to come back online. We use containerd as a cache Hey, even if I can feel sympathetic for the course of unfortunate events, it's hard to not to comment: if you're using a cache, you should invalidate it on failure!

It's a read-through cache. This wasn't a cache invalidation issue. It's a systems-level state corruption problem that just happened to break a system used primarily as a cache.

What I meant is that if the compromised host was unable to use broken boltdb cache, the cache should be zeroed and repopulated. Was it really hours of such cache rebuild vs hours of trying to fix the boltdb?

Btw I am happy I got only small amounts of data in any of bolt databases...

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#457

Post by Ensorceled »

api wrote: post #420
Ensorceled wrote: post #357

Earlier quoted context omitted.

Who fully sets up a significant project locally? I used Heroku for a project mostly because my team didn't have skill set to set this up and I wasn't going to do it. As far as I know they are still on Heroku (with a smattering of AWS services) for that same reason: just works and cheaper than doing it yourself.

> Who fully sets up a significant project locally? Who doesn't? I couldn't imagine having to push to some cloud agent and wait a random amount of time every time I want to test something. With it local I can just save, maybe rebuild or have it auto-rebuild if necessary, and test, then repeat. On a fast machine this can be a few seconds or instantaneous. Maybe the niche I'm missing here is very "green" developers who…

Really? Redundant databases? Redundant redis servers? Caching? All locally?

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#458

Post by Fire-Dragon-DoL »

Takennickname wrote: post #252
Dylan16807 wrote: post #104

Earlier quoted context omitted.

At that level I think I'd just put the images in the database.

In the git repo even

The images are supplied by the users, so that wouldn't be an option

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#459

Post by Fire-Dragon-DoL »

Dylan16807 wrote: post #104
Fire-Dragon-DoL wrote: post #49

Earlier quoted context omitted.

I don't have to use an object store, but it makes the cost of setting up a server more expensive if I use the filesystem, if I delete the instance, the data is gone. A volume kinda offset this, but it's way less portable and accessible only by one instance at a time The peace of mind of managed is nice, all I have to think about is running the app, without having to deal with making sure db and files don't get lost

At that level I think I'd just put the images in the database.

That's an option, but I want to keep things simple and the assumption is usually "filesystem" but weirdly most libraries assume S3 usage. I don't think I've seen native support for db-stored images in any of the libraries I use, which is sad but a reality.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#460

Post by tptacek »

kunley wrote: post #456
tptacek wrote: post #413

Earlier quoted context omitted.

It's a read-through cache. This wasn't a cache invalidation issue. It's a systems-level state corruption problem that just happened to break a system used primarily as a cache.

What I meant is that if the compromised host was unable to use broken boltdb cache, the cache should be zeroed and repopulated. Was it really hours of such cache rebuild vs hours of trying to fix the boltdb? Btw I am happy I got only small amounts of data in any of bolt databases...

This isn't a boltdb we designed. It's just containerd. I am probably not doing the outage justice, because "blitz and repopulate" is a time-honored strategy here.
Post reply on HN