Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reliability: It’s not great

community.fly.io

Post reply on HN
Print view View original

321–330 of 476 posts

Re: Reliability: It’s not great

#321

Post by snapetom »

zamnos wrote: post #160
snapetom wrote: post #100

Earlier quoted context omitted.

This is probably therapy, but your message and fly.io's post resonates a lot with what I'm going through. I took a product owner role about 6 months ago, my first, with a company that has turned out to be just a mired mess, and a product universally hated both internally and externally. Long story short, it's completely over-engineered by a bunch of intellectual engineers with no focus, no discipline, and no oversigh…

Can you help me in a detailed sense - what did you tell customers? did you literally say there's product is "completely over-engineered by a bunch of intellectual engineers with no focus, no discipline, and no oversight"? That seems a little over-honest to me but of course I wasn't there.

It depends on the audience.

First off, it helps I've been 15+ years as an engineer, 5 as an engineering manager, and throughout have the community contributions in the field on my resume. I instantly spotted the problems when I was given an architecture diagram on day 1 and discussed what I would do differently. All that gives credibility.

If it's internal audience, I am brutally honest. The organization needs to know this wasn't happenstance and bad luck that put us where we are now. It was a deliberate series of bad decisions based on a poor engineering and product culture. Now, for better or for worse, we are tasked with paying the debt.

There's a certain class of customers that are sister companies under the same parent. I'm honest with them, too, but go on the offensive. They have abused my team and our company in the past, and unfortunately, we have let them. I am more than happy to fire back and go toe to toe with bad behavior, and at the same time working to fix critical support issues.

For external customers, I've had remarkably good response in listening to their complaints. I am honest in discussing, in deep engineering detail, how the new product will address their problems, where issues might still be, and development timeline. I like to think the credibility portion comes into play here. In the past, customers were just told, "We'll look at it" and "We'll fix it" but nothing was ever planned.

Re: Reliability: It’s not great

#323

Post by TheDong »

KETpXDDzR wrote: post #322

That sounds like a typical problem of unnecessary complexity to me. I wonder how many over engineered (web) applications could run as a single, efficient process on a single machine.

Do you know what fly.io does?

Like, it cannot possibly run on a single machine by definition because the product they're building is running customer code on edge compute nodes distributed across the globe.

The main selling point of their service is that it's _not_ a single machine.

Re: Reliability: It’s not great

#324

Post by throwawaaarrgh »

I've been doing reliability stuff for near two decades. The one thing I am sure of is there is no way to just engineer your way to reliability. That is to say, no person, no matter how smart, can just invent some whizbang engineering thing and suddenly you have reliability.

Reliability is a thing that grows, like a plant. You start out with a new system or piece of software. It's fragile, small, weak. It is threatened by competing things and literal bugs and weather and the soil it's grown in and more. It needs constant care. Over time it grows stronger, and can eventually fend for itself pretty well. Sometimes you get lucky and it just grows fine by itself. And sometimes 50 different things conspire to kill it. But you have to be there monitoring it, finding the problems, learning how to prevent them. Every garden is a little different.

It doesn't matter what a company like Fly does technology wise. It takes time and care and churning. Eventually they will be reliable. But the initial process takes a while. And every new piece of tech they throw in is another plant in the garden.

So the good news is, they can become really reliable. But the bad news is, it doesn't come fast, and the more new plants they put in the ground, the more concerns there are to address before the garden is self sustaining.

Re: Reliability: It’s not great

#326

Post by doodlesdev »

josegonzalez wrote: post #255
doodlesdev wrote: post #212

Earlier quoted context omitted.

How do Coolify and Dokku compare? I've been aware of Dokku for a long time already, however I've never been confident enough to rely on these interfaces to deploy applications, specially because of their business model to keep things going. I'll have to try them both eventually though, I absolutely hate PaaS honestly, the prices are all just too high, but the convenience is really nice when managing a multitude of se…

Dokku Maintainer here. I don't really have a business model. I do take donations from Open Collective (and Github Sponsors, which funnels to OC) and there is Dokku Pro, but those don't collect anywhere near the funds I'd need to stop my dayjob (at least now. Maybe someday?). My business model is that code releasing is something I'm pretty passionate about. Dokku isn't even originally my project (Jeff Lindsay started…

Oh that's really cool to know, wasn't expecting the Dokku maintainer to read my comment LOL. From what I had looked at some time ago I though the project was run with the profits from Dokku pro, although upon further thought I understand that that's probably not enough to keep someone working full-time on it LOL.

I checked the repo and yeah, it checks out, Dokku _is_ pretty manageable with a decently small codebase. Having a low bus factor is really important for me. I'll check it out soon, and hopefully leave a donation to help you keep the project going too :)

Re: Reliability: It’s not great

#327

Post by dastbe »

tptacek wrote: post #311
dastbe wrote: post #267

Earlier quoted context omitted.

this doesn't paint a full picture of your options, as there's nothing that stops you from having zonal/regional consensus and then replication across regions/long-range topologies for global distribution. to be pithy about it, going full-bore gossip protocol is like going full-bore blockchain: solves a problem, introduces a lot of much more painful problems, and would've been solved much more neatly with a little bit…

I don't disagree that there are opportunities to introduce topology. I do disagree that there are opportunities to benefit from distributed consensus. If a server in ORD is down, it doesn't matter what some server in SJC says it's hosting; all the ORD instances of all the apps on that server are down. If that same ORD server is up, it doesn't matter what any server says it's running; it's authoritative for what it's…

> I do disagree that there are opportunities to benefit from distributed consensus

there's some benefits to static stability and grey failure, but sure, whatever. the important bit is to have clear paths of aggregation and dissemination in your system.

that being said

> it doesn't matter what some server in SJC says it's hosting

it kind of does matter doesn't it? assuming that server in SJC is your forwarding proxy that does your global loadbalancing, what that server is aware of is highly relevant to what global actions you can take safely.

Re: Reliability: It’s not great

#328

Post by mrkurt »

throwawaaarrgh wrote: post #324

I've been doing reliability stuff for near two decades. The one thing I am sure of is there is no way to just engineer your way to reliability. That is to say, no person, no matter how smart, can just invent some whizbang engineering thing and suddenly you have reliability. Reliability is a thing that grows, like a plant. You start out with a new system or piece of software. It's fragile, small, weak. It is threatene…

This is an excellent description.

Re: Reliability: It’s not great

#329

Post by alex_sf »

ricardobeat wrote: post #302
alex_sf wrote: post #298

Earlier quoted context omitted.

Not exactly related to the OP, but: I think I speak for a large number of folks when I say that we don't care. The EU keeps passing all sorts of absurd laws that require dedicated auditors to comply with. It's just not going to happen. If they decide to actively enforce these things, they'll just isolate themselves from the rest of the world.

As an EEUU resident, we also don't care. We can survive without youtube and instagram and the whole surveillance industry. Some of the laws place a heavy burden on giant tech companies, but for good reason.

They place a burden on everyone. A burden that's going to create a two-tier internet where service is immediately refused to EU citizens by every provider except the giant tech companies that can afford to comply.

Re: Reliability: It’s not great

#330

Post by tptacek »

dastbe wrote: post #327
tptacek wrote: post #311

Earlier quoted context omitted.

I don't disagree that there are opportunities to introduce topology. I do disagree that there are opportunities to benefit from distributed consensus. If a server in ORD is down, it doesn't matter what some server in SJC says it's hosting; all the ORD instances of all the apps on that server are down. If that same ORD server is up, it doesn't matter what any server says it's running; it's authoritative for what it's…

> I do disagree that there are opportunities to benefit from distributed consensus there's some benefits to static stability and grey failure, but sure, whatever. the important bit is to have clear paths of aggregation and dissemination in your system. that being said > it doesn't matter what some server in SJC says it's hosting it kind of does matter doesn't it? assuming that server in SJC is your forwarding proxy t…

My point is just that there isn't a consensus algorithm that needs to get run to know which of the two proposals to accept.
Post reply on HN