Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly Kubernetes

fly.io

Post reply on HN
Print view View original

91–100 of 180 posts

Re: Fly Kubernetes

#91

Post by kasey_junk »

verdverm wrote: post #30
tptacek wrote: post #25

Earlier quoted context omitted.

We're not a K8s vendor. We're a lower-level platform than that. If all you care about is K8s, and no part of the rest of our platform is interesting to you --- the global distribution and Anycast, the fly-proxy features, the Machines API --- we're not a natural fit for what you're doing. We were surprised at how FKS turned out, which is part of why we decided to launch it as a feature and all of why we wrote it up th…

> We're not a K8s vendor It might now be more accurate to say "You were not a k8s vendor", but now you are based on > If K8s is important for your project, and that’s all that’s been holding you back from trying out Fly.io, we’ve spent the past several months building something for you. If it's fundamentally different, maybe you shouldn't call it Kubernetes, perhaps a Kubernetes API compatible alternative? fwiw/conte…

Are any of the cloud provided managed K8s offerings just K8s under the covers? I’ve always assumed all of them shim the K8s api onto other more bespoke orchestration systems.

Re: Fly Kubernetes

#92

Post by JCharante »

rozenmd wrote: post #90
nathancahill wrote: post #81

Man, I just wish they'd work on stability. Fly.io is an amazing offering. But it's so buggy, it's almost more headache than it's worth trying to build PaaS-flavored software on it. Even the Fly docs are "buggy" since they mostly transitioned to v2 Machines but the docs are still a mix of Nomad and Machines. There's so much power on the platform with Flycast, LiteFS and other clever ways to work with containers. If it…

I agree - I find if you pick the "mainstream" regions like IAD you get close to 100% uptime, like what you see from my 3rd-party status page here: https://flyio.onlineornot.com/ Once you start deploying in SIN/CDG etc you start to get really weird instability (and this is on v2 machines).

One of fly's main features is global distribution so it's kinda silly if you have to avoid SIN and CDG

Re: Fly Kubernetes

#93

Post by whalesalad »

ekidd wrote: post #87
whalesalad wrote: post #79

Earlier quoted context omitted.

Kubernetes is really epic and powerful if you actually take the time to understand it from first principles. Unfortunately people don't do this, and individuals without good networking/devops experience roll something half-baked out with a terrible deployment process, a mess of helm charts, etc... and it ends up being hated by everyone. At FarmLogs (yc 12) we had a pretty righteous gitops (homegrown) kube platform ru…

I have used a number of orchestrator platforms in production, including AWS ECS (Docker + custom orchestration), EKS (Kubernetes) and Google Cloud's Kubernetes. I've also used Chef, custom RPM packages and classic Unix startup scripts. And I'm probably forgetting some. And honestly? Kubernetes can be really great. Especially if you: - Read enough to understand the split between pods/replication controllers/deployment…

There is absolutely (still!!!!) room in the market for a PaaS built on top of Kube that is actually good, and is a hybrid of Heroku+Kube. More convention, less all the rope to hang yourself with, while still enabling advanced use cases with full control.

Even docker compose is too annoying for what most people need.

Most of our use cases are so simple. I need a (black box) container that exposes a TCP port. I want N of them. I want them load-balanced with a friendly name. I want them load balanced behind a friendly name with auth in front of it. I want centralized logs, based on these names. I want centralized stats, based on these names (not xyz-pod-232903209284390-dev but xyz aggregated). I want auto deploys based on a github repo. I want my releases tagged with the git short hash.

When someone cracks the nut of heroku+kube they will become the next billionares. This is why I think it is wise to try and enter this space, as Fly is doing.

The new stuff Microsoft has announced (named Radius, not to be confused with RADIUS network auth protocol) is the closes thing to this sort of solution I have been imagining for years: https://azure.microsoft.com/en-us/blog/the-microsoft-azure-i...

I want N http/tcp/php/python/node/insert-tool-here API or web servers. I want a best-practices RabbitMQ deployment, a best-practices Redis deployment, and a best-practices PostgreSQL deployment. For Redis, I do not care about state, just make it work. For Rabbit, I want 3+ nodes for HA. For PSQL, I do care about state so please use and do backups.

This is the same shit we have been reinventing for years over and over again and the recipes are all the same now. Radius is the best (at least on paper) attempt at unifying all this stuff.

I've been beating this drum for over 10 years now though, I keep thinking someone will figure it out and no one has. Maybe time to bite the bullet and just do the thing.

Re: Fly Kubernetes

#94

Post by tptacek »

alpb wrote: post #88

I kind of miss the point of this. So if I'm reading this right, fly.io practically only exposes the Pods API, but Kubernetes is really much more than that. I'm not very familiar with any serious company that directly uses Pods API to launch containers, so if their reimplementation of Pods API is just a shim, and they're not going to be able to implement ever-growing set of features in Kubernetes Pod lifecycle/configu…

This is a good comment. More like this!

Right now, the immediate things you'd get out of using FKS are:

* The declarative K8s style of defining an app deployment, and some of the K8s mechanics for reconciling that declaration to what's actually running. We did most of this stuff before when we were backed on Nomad, but less of it now with Fly Machines. If you missed having a centralized orchestrator, here's one.

* Some compatibility with K8s tooling (we spin up a cluster, spit out a kubeconfig file, and you can just go to town with kubectl or whatever).

This is absolutely not going to let you do everything you can possibly do with K8s! Maybe we'll beef it up over time. Maybe not many people will use it, because people who want K8s want the entire K8s Cinematic Universe, and we'll keep it simple.

Mostly: we wrote about it because it was interesting, is all that's happening here.

I think you asked a super good question, and "I don't know, you might be right" is our genuine answer. Are there big things this is missing for you? (Especially if they're low-hanging fruit). I can (sort of) predict how likely we are to do them near term.

Re: Fly Kubernetes

#95

Post by benpacker »

btown wrote: post #80
benpacker wrote: post #8

Am I understanding correctly that because they map a “Pod” to a “Fly Machine”, there’s no intermediate “Node” concept? If so, this is very attractive. When using GKS, we had to do a lot of work to get our Node utilization (the percent of resources we had reserve on a VM actually occupied by pods) to be higher than 50%. Curios what happens when you run “kubectl get nodes” - does it lie to you, or call each region one…

GKE Autopilot is an attractive option here if you don't want to worry about node utilization and provisioning. Effectively you have an on-demand infinitely-sized k8s cluster that scales up and down as you need new pods. Some caveats, but it's an incredible onramp if you're coming from a Heroku or similar PaaS and don't want to worry about the infrastructure side of things: Github Actions building images and deploying…

Unfortunately last I checked the compute pricing for GKE autopilot was almost double, so if you can beat 50% utilization, you might as well just keep the under-utilized Node around.

Re: Fly Kubernetes

#96

Post by benpacker »

chologrande wrote: post #77
benpacker wrote: post #8

Am I understanding correctly that because they map a “Pod” to a “Fly Machine”, there’s no intermediate “Node” concept? If so, this is very attractive. When using GKS, we had to do a lot of work to get our Node utilization (the percent of resources we had reserve on a VM actually occupied by pods) to be higher than 50%. Curios what happens when you run “kubectl get nodes” - does it lie to you, or call each region one…

> Had to do a lot of work to get node utilization ... higher than 50% How is this the schedulers fault? Is this not just your resource requests being wildly off? Mapping directly to a "fly machine" just means your "fly machine" utilization will be low

I think there’s a slight misunderstanding - I’m referring to how much of a Node is being used by the Pods running on it, not how much of each Pod’s compute is being used by the software inside it.

Even if my Pods were perfectly sized, a large percent of the VMs running the Pod was underutilized because the Pods were poorly distributed across the Nodes

Re: Fly Kubernetes

#97

Post by loloquwowndueo »

JCharante wrote: post #92
rozenmd wrote: post #90

Earlier quoted context omitted.

I agree - I find if you pick the "mainstream" regions like IAD you get close to 100% uptime, like what you see from my 3rd-party status page here: https://flyio.onlineornot.com/ Once you start deploying in SIN/CDG etc you start to get really weird instability (and this is on v2 machines).

One of fly's main features is global distribution so it's kinda silly if you have to avoid SIN and CDG

You can use hkg and ams instead :)

Re: Fly Kubernetes

#98

Post by whalesalad »

tptacek wrote: post #84
jeromegn wrote: post #83

Earlier quoted context omitted.

We don't use k8s and you don't have to either. This is for current and future users who absolutely want k8s. We are a compute provider after all and making it easy to host a great variety of apps is good for our users.

We really didn't expect so many people to read this like "Fly is going all K8s"! It's interesting.

There are a substantial number of frankly ignorant people on HN who see K8S and run for the hills just because it is something they genuinely do not comprehend or were burned by an inappropriate deployment. The irony is that you arent switching to K8s, you are simply offering it as a compatibility layer, and still seeing flak. Case in point.

Re: Fly Kubernetes

#99

Post by benpacker »

arccy wrote: post #21
benpacker wrote: post #8

Am I understanding correctly that because they map a “Pod” to a “Fly Machine”, there’s no intermediate “Node” concept? If so, this is very attractive. When using GKS, we had to do a lot of work to get our Node utilization (the percent of resources we had reserve on a VM actually occupied by pods) to be higher than 50%. Curios what happens when you run “kubectl get nodes” - does it lie to you, or call each region one…

is GKS some amalgamation of GKE and EKS

Typo haha - I meant GKE. Fixed now.

Re: Fly Kubernetes

#100

Post by loloquwowndueo »

thowrjasdf32432 wrote: post #89

Great writeup! Love reading about orchestration, especially distributed. > When you create a cluster, we run K3s and the Virtual Kubelet on a single Fly Machine. Why a single machine? Is it because this single fly machine is itself orchestrated by your control plane (Nomad)? > ...we built our own Rust-based TLS-terminating Anycast proxy (and designed a WireGuard/IPv6-based private network system based on eBPF). But t…

The control plane is not nomad anymore : https://community.fly.io/t/the-death-of-nomad/16220
Post reply on HN