Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly.io Postgres cluster down for 3 days, no word from them about it

webcache.googleusercontent.com

Post reply on HN
Print view View original

331–340 of 493 posts

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#331

Post by azemetre »

MuffinFlavored wrote: post #319
oefrha wrote: post #296

Earlier quoted context omitted.

I was confused why support for platform failure relies on a forum where employees may or may not check. After checking docs[1], apparently you have to be on a paid plan (at least $29/mo) to access email support, so you may not have it even you’re paying for resources. I won’t be using it for side projects where I’m okay with paying $5-10/mo but don’t want to have three day outages. [1] https://fly.io/docs/about/suppo…

Forewarning: I am not being critical of fly.io nor their free support whatsoever when I say this. From a technical perspective, could they have "been better" from a technical perspective? I see their name a lot on HN so I know they are doing really cool + advanced things and this is probably some super small edge case that slipped through the cracks. Could they have added some message / do we as the HN community feel…

They may not owe anyone anything but over time these types of issues can cause a large reputation hit.

If I was just searching online or trying to find out what various communities think about Fly.io and see several threads about major outages with poor communications, do you think I will use their services? It would be an immediate pass.

It takes a long time to build a reputation, and you can lose it instantly.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#332

Post by wongarsu »

EspressoGPT wrote: post #313
YetAnotherNick wrote: post #150

Earlier quoted context omitted.

Does anyone know how Hetzner pricing is half of DO yet is profitable, while DO is loss making with 6% operating margin?

Overstaffed, overinflated and inefficient Silicon Valley startup vs. organically-grown, well-adjusted, efficient German company.

Not to mention a German company that has price sensitivity in their DNA. Their first servers were just regular consumer tower PCs to drastically cut hardware costs. Now many years later it's a highly optimized mix of consumer, server and inhouse parts (e.g. they use their own racking system instead of 19", and the datacenters are built to make use of convection for a lot of the cooling). They also offer regular Dell servers for those that want them, but at 2x-4x the price of their homegrown boxes.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#333

Post by ryanrussell »

michaeldwan wrote: post #163
throwawayfly wrote: post #131

Earlier quoted context omitted.

Ok as long as we’re getting conspiratorial, something similar I observed has bugged me. About a year ago fly awarded a few people in the forums, I think it was 3, the “aeronaut” badge. Basically just pointless bling for a “routinely very helpful” person or somesuch. Still, I can imagine it was cool to get it. No, it wasn’t me. One person I saw with it absolutely deserved it: this person is, to this day, always hoppin…

Get out of here with this nonsense. We tell people when we’re a bad option all the time. Do you really think we have a desire (or time) to punish somebody for doing the same? Also, here’s the long forgotten badge, still with 3 people… https://community.fly.io/badges/107/aeronaut

Why is anyone on HN "dunking" on Fly.IO of all companies?

Michael - Don't take the bait.

As someone who has zero affiliation with Fly.IO other than a few PR's to their OSS(I don't even know Michael), I greatly appreciate the contributions they have given back to the community.

There are a lot of great hosting companies. Fly.IO stands out due to their revolutionary architecture and contributions back to the OSS community. I wish more companies operated like this.

It's understandable some are upset about an outage. But Fly is doing really interesting and game-changing things, not copying a traditional vmware, cpanel or k8s route.

Just as a reminder to what this company has offered back to everyone.

SQLite: Ben Johnson's OSS work around SQLite stands out. Fly.IO and his work have really made sqlite a contender. - https://fly.io/blog/all-in-on-sqlite-litestream/ - https://fly.io/blog/introducing-litefs/ - https://github.com/superfly/litefs - https://github.com/benbjohnson/litestream - https://fly.io/blog/sqlite-internals-wal/ - https://fly.io/blog/wal-mode-in-litefs/

Who really considered sqlite as a production option before Fly and Ben? Not me.

Firecracker: Firecracker is amazing, but difficult to debug when something bad happens. There aren't a ton of people in devops who would share what they have. If you've ever used Firecracker, you've really been helped a lot by the various guides they have provided back to the community like these: - https://fly.io/docs/reference/architecture/ - https://fly.io/blog/fly-machines/ - https://fly.io/blog/sandboxing-and-workload-isolation/

Their architecture is beautiful and revolutionary. They're probably the first or second ones to find a lot of the new edge cases as they grow.

It's a lot harder to be the first one over the wall than it is to copy. They've literally given the average developer a blueprint to build scalable businesses that compete with their own.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#335

Post by danielvaughn »

vasco wrote: post #126
danielvaughn wrote: post #56

Earlier quoted context omitted.

I adore DO. They’re seriously underrated. I love how they’ll just give you a server and say here, have at it. No abstractions, no fancy crap, just get out of my way and let me do my thing.

Same! I've had my first server there for 10 years now. They added a lot of stuff in the meantime, they have AWS-like things you can do. But in terms of launching a VM that just works, they are a great choice.

Yeah I hadn't seen those newer features until recently, the one-click deployments are super cool.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#336

Post by quickthrower2 »

TekMol wrote: post #198
danjac wrote: post #182

Earlier quoted context omitted.

There's a sweet spot of early startup or side project where you don't have the time, budget or people to manually set up and maintain servers on your own or deal with the complexity and cost of Kubernetes or AWS, especially when your focus is on building the product and acquiring customers. Heroku (before its inevitable enshittification under Salesforce) was great for this use case. Sure you will outgrow it at some p…

What exactly does Fly.io give you? You already know how to set up your project locally. Why not just do the same setup on any cloud VM and boom it is online?

Probably saves you a good hour of "sudo apt gets" and "vim /etc/nginx/nginx.conf" etc.

Having used various PaaS services that take this "pain" away from you, I sort of think the tradeoff isn't worth it. For $5/m DO will give you a backed up server. Add $15 for postgres that is a good deal.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#337

Post by danielvaughn »

dbingham wrote: post #327
danielvaughn wrote: post #56

Earlier quoted context omitted.

I adore DO. They’re seriously underrated. I love how they’ll just give you a server and say here, have at it. No abstractions, no fancy crap, just get out of my way and let me do my thing.

I love DO for projects where I don't need control. For my side project, I eventually migrated to AWS after running into a lot of issues with DO. Things like they don't give you the postgres root user on their managed postgres. And I ran into issues trying to capture the deployments in code. Their terraform providers are pretty good, but still leave something to be desired. For all its many warts, I'm much happier bac…

That's interesting, because granular control is why I enjoy DO, although I'm thinking about it from the server perspective. They set up a machine, give me root access, and that's literally it. I set up my own ssh keys, firewalls, and there's no additional abstraction that I have to learn. I might just be reminiscing because right now I'm on a team where we're writing terraform/helm/k8s in GCP and it makes me want to cry myself to sleep each night lol.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#338

Post by throwawaaarrgh »

There's a lot of bullshit in this HN thread, but here's the important takeaway:

- it seems their staff were working on the issue before customers noticed it.

- once paid support was emailed, it took many hours for them to respond.

- it took about 20 hours for an update from them on the downed host.

- they weren't updating their users that were affected about the downed host or ways to recover.

- the status page was bullshit - just said everything was green even though they told customers in their own dashboard they had emergency maintenance going on.

I get that due to the nature of their plans and architecture, downtime like this is guaranteed and normal. But communication this poor is going to lose you customers. Be like other providers, who spam me with emails whenever a host I'm on even feels ticklish. Then at least I can go do something for my own apps immediately.

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#339

Post by mewmew07 »

yard2010 wrote: post #280
tptacek wrote: post #80

Y'all, this is going to be deeply unsatisfying, but it's what I can report personally: I have no earthly clue why this thread on our community site is unlisted. We're looking at the admin UI for it right now, and there's like, a little lock next to do the story, but the "unlist story" option is still there for us to click. The best I can say is: I'm reasonably sure there wasn't some top-down edict to hide this thread…

[flagged]

you have no idea wtf you writing about; it's been a few hours now and it's become clear that someone tagged the post as 'app-not-working, which made the post got 'private' and only available for logged-in users. it's also become apparent that the linked post in on a community forum for users without a support plan.

the dramatic tone and accusations in your reply are not warranted anymore

Re: Fly.io Postgres cluster down for 3 days, no word from them about it

#340

Post by arrowsmith »

heartbreak wrote: post #258
subarctic wrote: post #146

Earlier quoted context omitted.

If you're talking about the comment you're replying to, tbh I found it was way more relatable than a more "professional" PR-speak response. Maybe you were talking about something else

Unfortunately PR-speak exists for a reason.

But is it a good reason?
Post reply on HN