Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly Kubernetes

fly.io

Post reply on HN
Print view View original

161–170 of 180 posts

Re: Fly Kubernetes

#161

Post by harpratap »

philsnow wrote: post #111
kuhsaft wrote: post #73

Earlier quoted context omitted.

I think the biggest hurdle would be networking between the pods since they will be running on different cloud providers.

I've seen some people using wireguard for intra-cluster networking so that all their nodes can run pretty much anywhere.

Wouldn't the network cost be absurd in such case? Not only the pod-to-pod communication cost skyrocket, all the heartbeats, health checks, metrics, daemonsets pinging each other will probably end up costing more than the CPU and Memory

Re: Fly Kubernetes

#162

Post by harpratap »

benpacker wrote: post #96
chologrande wrote: post #77

Earlier quoted context omitted.

> Had to do a lot of work to get node utilization ... higher than 50% How is this the schedulers fault? Is this not just your resource requests being wildly off? Mapping directly to a "fly machine" just means your "fly machine" utilization will be low

I think there’s a slight misunderstanding - I’m referring to how much of a Node is being used by the Pods running on it, not how much of each Pod’s compute is being used by the software inside it. Even if my Pods were perfectly sized, a large percent of the VMs running the Pod was underutilized because the Pods were poorly distributed across the Nodes

Is that really a problem in Cloud environments where you would typically use a Cluster Autoscaler? GKE has "optimize-utilization" profile or you could use a descheduler to binpack your nodes better

Re: Fly Kubernetes

#163

Post by verdverm »

kasey_junk wrote: post #159
verdverm wrote: post #157

Earlier quoted context omitted.

Maybe you are reading it differently than me, but when they refer to the same binaries, as if I managed it myself, as being the pieces they use, it definitely seems like the open source project being used. > The control plane is the unified endpoint for your cluster. You interact with the control plane through Kubernetes API calls. The control plane runs the Kubernetes API server process (kube-apiserver) to handle AP…

We are well into the reeds of what doesn’t matter, only in that fly has given us a very under the covers look at their implementation that is hard to find with other alternatives. But as someone who has run K8s in other contexts I find the following to be pretty circumspect (not in a way that causes me concern, I’m a happy gke user) > GKE Autopilot manages the entire underlying infrastructure of clusters, including t…

> I’m a happy gke user && Which I’m happy for them to keep on their side of the street.

100% agreement

iirc, there was a time where I thought they were doing some consolidation things with how they run the control plane, then at some point my cluster updates had a warning related to control plane unavailability during an update, this was on a single node cluster

I get what you are saying though, there's probably some magic going on somewhere, but after many years on GKE, I don't really think about it.

Re: Fly Kubernetes

#164

Post by xgbi »

I have so many questions, it is a very good article!

My most important one is this: can I build a distributed k8s cluster with this?

I mean having fly machines in Europe, US and Asia acting as a solid k8s cluster and letting the kube scheduler do its job?

If yes then it is better than what the current cloud offerings are, with their region-based implementation.

My second question is obviously how is the storage handled when my workload migrates from the US to Europe: so I still profit from NVME speeds? Is it replicated synchronously?

Last but not least: does it support RWM semantics?

If all the answers are yes, kudos, you just solved many folk’s problems.

Stellar article, as usual.

Re: Fly Kubernetes

#165

Post by antonvs »

spankalee wrote: post #144
btown wrote: post #80

Earlier quoted context omitted.

GKE Autopilot is an attractive option here if you don't want to worry about node utilization and provisioning. Effectively you have an on-demand infinitely-sized k8s cluster that scales up and down as you need new pods. Some caveats, but it's an incredible onramp if you're coming from a Heroku or similar PaaS and don't want to worry about the infrastructure side of things: Github Actions building images and deploying…

Why would you use GKE Autopilot over Cloud Run?

Cloud Run is great if you just need to deploy a few services and expose their endpoints, and don't have a particularly complex backend service architecture.

But with more complex architectures, you'll end up implementing a sort of GKE-like layer over Cloud Run, at which point GKE would probably make more sense.

GKE lets you shell into containers, run all different kinds of workloads (e.g. no need for a separate "Cloud Tasks" system), supports stateful workloads, provides a standardized language for defining and deploying resources of all kinds (the k8s resource definition language), and as such integrates with standard gitops deployment systems such as ArgoCD.

Re: Fly Kubernetes

#167

Post by remram »

tptacek wrote: post #134
kuhsaft wrote: post #132

Earlier quoted context omitted.

Right, but what is the point of FKS then? It’s no longer Kubernetes if it doesn’t support a core behavior of Kubernetes. If you only support deploying single containers with single processes on FKS, then you might as well use flyctl. It’s a solvable issue of course. The virtual-kubelet implementation would need to create a Machine running a container runtime image that would then run a the pod containers to match the…

The article discusses what you get by using K8s alongside Fly.io. If you want to bin-pack containers onto Fly Machines, you can of course just boot up your own K8s cluster here; that has always been an option.

The problem is not the need to bin-pack, the problem is completeness. Sidecars and multiple containers are used for logging, backups, etc. Not to mention that if you grab a manifest or chart for an app, it is going to have pods with multiple containers (whether that's strictly needed or not), and those won't work on fly.io.

This is a critical feature available in every Kubernetes offering, and as such people rely on it. Trying to say that maybe you can do without is missing the forest for the trees.

Like saying that your C compiler doesn't need arrays because it has pointers: sure, maybe, but now good luck compiling any existing code. Maybe don't call it a C compiler if no C program will work on it unmodified.

Re: Fly Kubernetes

#168

Post by ricketycricket »

loloquwowndueo wrote: post #97
JCharante wrote: post #92

Earlier quoted context omitted.

One of fly's main features is global distribution so it's kinda silly if you have to avoid SIN and CDG

You can use hkg and ams instead :)

I haven't been able to bring up a clustered Elixir server in hkg without experiencing netsplits every 5-10 minutes. ewr, ord, and cdg have been totally reliable.

Re: Fly Kubernetes

#169

Post by nojvek »

motoboi wrote: post #9

There is a very high price to pay when going with your own scheduling solution: you have to compete with the resources google and others are throwing at the problem. Also, there is the market for talent, which is non-existent for fly.io technology if it's not open source (I see what you did here, Google): you'll have to teach people how your solution works internally and congratulations, now you have a global pool of…

What if it’s really not that complicated, and by adding more people you make it more complex. So complex that you need even more people to maintain that complexity?

I love fly.io for rethinking some of the problems.

Re: Fly Kubernetes

#170

Post by tptacek »

remram wrote: post #167
tptacek wrote: post #134

Earlier quoted context omitted.

The article discusses what you get by using K8s alongside Fly.io. If you want to bin-pack containers onto Fly Machines, you can of course just boot up your own K8s cluster here; that has always been an option.

The problem is not the need to bin-pack, the problem is completeness. Sidecars and multiple containers are used for logging, backups, etc. Not to mention that if you grab a manifest or chart for an app, it is going to have pods with multiple containers (whether that's strictly needed or not), and those won't work on fly.io. This is a critical feature available in every Kubernetes offering, and as such people rely on…

Again: Fly Machines are just Linux VMs, and you have root on them.
Post reply on HN