Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reliability: It’s not great

community.fly.io

Post reply on HN
Print view View original

421–430 of 476 posts

Re: Reliability: It’s not great

#421

Post by JeremyNT »

mixmastamyk wrote: post #250
otterley wrote: post #231

Earlier quoted context omitted.

Well, why did you do that? If you’d asked them whether this was a supported configuration or intended purpose, they’d have said no; and anyone who had experience deploying Consul at large scale would have told you the same. There is truly no compression algorithm for experience.

I don't think he personally designed the first implementation. But in any case, understanding of complex topics comes in waves. Many times I've had to read all the docs then use a system for several months before the epiphany hits me.

This is especially true for scaling. A solution that works great for your current deployment may be completely unworkable for 2x your current deployment.

You just won't know until you fall off the cliff. The armchair quarterback can opine that you should have just hired experts in XYZ domains from the start to design robust systems that can scale to arbitrary sizes, but most orgs don't need to scale to arbitrary sizes so this is highly likely to be wasted effort.

Re: Reliability: It’s not great

#422

Post by jen20 »

unxdfa wrote: post #365
throwawaaarrgh wrote: post #324

I've been doing reliability stuff for near two decades. The one thing I am sure of is there is no way to just engineer your way to reliability. That is to say, no person, no matter how smart, can just invent some whizbang engineering thing and suddenly you have reliability. Reliability is a thing that grows, like a plant. You start out with a new system or piece of software. It's fragile, small, weak. It is threatene…

You can make sensible assumptions that result in engineering gains though. Step around the problems not through them. For example I have learned that the first step to reliability is removing as many hashicorp products from your stack as possible though. Appears I am not the only one.

If you’ve been using them in ways clearly explicitly called out as not per the design goals, then sure, removing any piece of technology will help you. I’m guessing that is not your actual problem though.

Re: Reliability: It’s not great

#423

Post by tonnydourado »

Damn. I don't know, and I guess almost no one can know, how much of this is genuine honesty and how much is calculated messaging, but I barely care. It's refreshing enough that I kinda wanna give the thing a try. Which, ironically, might make their problems just a tiny little bit worse, since I likely won't be a paying customer XD

Re: Reliability: It’s not great

#424

Post by otterley »

JeremyNT wrote: post #421
mixmastamyk wrote: post #250

Earlier quoted context omitted.

I don't think he personally designed the first implementation. But in any case, understanding of complex topics comes in waves. Many times I've had to read all the docs then use a system for several months before the epiphany hits me.

This is especially true for scaling. A solution that works great for your current deployment may be completely unworkable for 2x your current deployment. You just won't know until you fall off the cliff. The armchair quarterback can opine that you should have just hired experts in XYZ domains from the start to design robust systems that can scale to arbitrary sizes, but most orgs don't need to scale to arbitrary size…

While I largely agree with you, this isn’t one of those cases. If Fly wasn’t supposed to scale in due course to this size, it probably wouldn’t have been funded. If your business model is predicated on you scaling, yes, you should hire appropriately in anticipation of that.

Besides, I’m not even necessarily talking about hiring here - even consulting would have been sufficient to avoid this catastrophe.

Re: Reliability: It’s not great

#425

Post by amluto »

re-thc wrote: post #405
amluto wrote: post #171

Earlier quoted context omitted.

Egress pricing, for one. fly.io charges an outrageous 2 cents/GB. Google is over 4x that. At fly.io rates, 1Gbps average over a month is $6400/mo. Google is tiered and you’re looking at over $10k/mo. For comparison, a cheap managed switch that can handle 1Gbps costs about $100, maybe a bit more if you want a nice one. A nice router is more. You can rent an entire rack , including power, cooling, and an unmetered 1Gbp…

It's potentially a lot more for the big clouds. Anything in the network path has a charge - load balancers, NAT gateways, etc.

The cost of egress plus a gateway or two is fairly close to the cost of burning a DVD twenty years ago. And it appears to actually be cheaper to burn DVDs and mail them today than to send data from a major cloud.

This becomes very relevant for things like archiving data. If you generate data outside of a major cloud, you can pay a major cloud a very reasonable fee to archive it for you. But if you ever download your archive, it will cost you about half the price of buying an external disk to store it on.

(To be fair, object storage is rather more reliable than a single crappy external drive. But if you access the data more than once, maybe you should have a colo or on-prem copy too.)

Re: Reliability: It’s not great

#426

Post by stratosmacker »

mrkurt wrote: post #307
say_it_as_it_is wrote: post #271

Earlier quoted context omitted.

Can you please explain what the Vault related failure is about? Is this about timing out services failing to start within an acceptable time range?

Yeah, basically that. One of the servers in our Vault cluster failed and prevented Vault agents from receiving secrets. For Nomad apps, this showed up as "allocation failures" and failed deploys. Machine based apps took an abnormally long time to start and caused other issues.

Doesn't Vault self-promote in the case of single node failure?

I noticed you mention Vault lives in the US, I'm sure you've already heard of this pattern, but Vault (Enterprise) supports [multi-region clusters for performance and DR](https://developer.hashicorp.com/vault/tutorials/day-one-raft...)

Re: Reliability: It’s not great

#427

Post by bennetthi »

throwawaaarrgh wrote: post #324

I've been doing reliability stuff for near two decades. The one thing I am sure of is there is no way to just engineer your way to reliability. That is to say, no person, no matter how smart, can just invent some whizbang engineering thing and suddenly you have reliability. Reliability is a thing that grows, like a plant. You start out with a new system or piece of software. It's fragile, small, weak. It is threatene…

Nicely said. I remember AWS outages (S3, EBS, and RDS) in the early 2010s when their products were younger. But given time to improve each has become more and more resilient.

Re: Reliability: It’s not great

#428

Post by amw-zero »

samwillis wrote: post #18

Fundamentally I think some of the problems come down to the difference between what Fly set out to build and what the market currently want. Fly (to my understanding) at its core is about edge compute. That is where they started and what the team are most excited about developing. It's a brilliant idea, they have the skills and expertise. They are going to be successful at it. However, at the same time the market is…

Why does there need to be a successor to Heroku?

Re: Reliability: It’s not great

#430

Post by jlokier »

ignoramous wrote: post #173
deivid wrote: post #154

I'm a bit sour reading this. I've always liked fly and particularly the engineering blog, so much so that a couple of months ago I decided to apply for an infra position, to work on some of these very topics. Sadly after 4~5 rounds of interviews (including a workday) they just ghosted me.

> Don't feel too bad nor take it personal. They probably have a lot of applicants, and are looking to grow their team by hiring someone with very specific skills. > I also applied a few months ago while I was in the middle of my job search. For one, I couldn’t really answer their "favorite syscall question" because I’ve never dealt with syscalls :) so maybe I just wasn't a good fit. Surely, everyone's favourite sysca…

For what it's worth, exit() is not a syscall, but _exit() often is.
Post reply on HN