Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly Kubernetes

fly.io

Post reply on HN
Print view View original

41–50 of 180 posts

Re: Fly Kubernetes

#41

Post by tick_tock_tick »

netshade wrote: post #31

I am a current Fly customer (personal and work), and have been happy with the service. Will likely be trying this out. That said, the marketing tone of this final part of the blog: > More to come! We’re itching to see just how many different ways this bet might pay off. Or: we’ll perish in flames! Either way, it’ll be fun to watch. is like nails on the chalkboard for me.

Why? If you're using something like Fly you should 100% always have a fallback plan ready. You are gambling using smaller players to get cheaper services or some other benefit the big players don't offer in exchange for the very real possibility of a random day they announce 30 days til they permanently shutdown with zero migration path.

I don't think it's in poor taste to acknowledge exactly what everyone should understand and be prepared for.

Re: Fly Kubernetes

#42

Post by lkjadflkj4 »

netshade wrote: post #31

I am a current Fly customer (personal and work), and have been happy with the service. Will likely be trying this out. That said, the marketing tone of this final part of the blog: > More to come! We’re itching to see just how many different ways this bet might pay off. Or: we’ll perish in flames! Either way, it’ll be fun to watch. is like nails on the chalkboard for me.

Some people have a cheeky sense of humor. Counterpoint: I'm ok with it.

Re: Fly Kubernetes

#43

Post by zifnab06 »

therein wrote: post #32
zifnab06 wrote: post #17

Earlier quoted context omitted.

fwiw - our network folks _should_ have fixed this a few weeks ago. Some of the outbound IPs were incorrectly tagged in some of the geoip databases as being in the US when they were not.

I remember asking about this about half a year ago. Back then I was told since Fly can route these IPs to wherever they want in their infrastructure by making simple configuration changes, the line is blurred anyway but GeoIP providers just take the shortcut and resolve to where the company is registered. I'll try again if the situation is different now.

Inbound IPs use anycast - the IP we give you routes to the nearest fly region and then hits a wireguard tunnel to get to the correct region.

Outbound IPs are tied to the individual host your machine is running on (which is located in a single region). These are tied to a single region, and in some cases, these were resolving to the wrong region in a few of the GeoIP databases. We hopefully fixed this part.

Re: Fly Kubernetes

#44

Post by kuhsaft »

benpacker wrote: post #8

Am I understanding correctly that because they map a “Pod” to a “Fly Machine”, there’s no intermediate “Node” concept? If so, this is very attractive. When using GKS, we had to do a lot of work to get our Node utilization (the percent of resources we had reserve on a VM actually occupied by pods) to be higher than 50%. Curios what happens when you run “kubectl get nodes” - does it lie to you, or call each region one…

The node would be a virtual-kubelet. You can check out the virtual-kubelet GitHub repo for more info.

Interestingly, there are already multiple providers of virtual-kubelet. For example, Azure AKS has virtual nodes where pods are Azure Container Instances. There’s even a Nomad provider.

> So that’s what we do. When you create a cluster, we run K3s and the Virtual Kubelet on a single Fly Machine.

So probably a cluster per region. You could theoretically spin up multiple virtual-kubelets though and configure each one as a specific region.

> Because of kine, K3s can manage multiple servers, but also gracefully runs on a single server, without distributed state.

This would mean the control-plane would be on a single-server without high-availability? Although, I suppose there really isn’t any state stored since they are just proxying requests to the Fly Machine API. But still, if the machine went down your kubectl commands wouldn’t work.

Re: Fly Kubernetes

#45

Post by [deleted] »

netshade wrote: post #31

I am a current Fly customer (personal and work), and have been happy with the service. Will likely be trying this out. That said, the marketing tone of this final part of the blog: > More to come! We’re itching to see just how many different ways this bet might pay off. Or: we’ll perish in flames! Either way, it’ll be fun to watch. is like nails on the chalkboard for me.

[deleted]

Re: Fly Kubernetes

#46

Post by roozbeh18 »

netshade wrote: post #31

I am a current Fly customer (personal and work), and have been happy with the service. Will likely be trying this out. That said, the marketing tone of this final part of the blog: > More to come! We’re itching to see just how many different ways this bet might pay off. Or: we’ll perish in flames! Either way, it’ll be fun to watch. is like nails on the chalkboard for me.

I made the same bet with cloudways which is now owned by digitalocean. they filled a gap for me, and I was ok if they decided to close shop; I am glad it didn't go that direction, and they are part of a bigger company that also was once a small company, but they are now publicly traded. you make your bets...

Re: Fly Kubernetes

#47

Post by tptacek »

netshade wrote: post #31

I am a current Fly customer (personal and work), and have been happy with the service. Will likely be trying this out. That said, the marketing tone of this final part of the blog: > More to come! We’re itching to see just how many different ways this bet might pay off. Or: we’ll perish in flames! Either way, it’ll be fun to watch. is like nails on the chalkboard for me.

Interesting! We're mostly not kidding about that. We launched in 2020 with a scheduler that looks a lot like how K8s works†. We ran into scaling issues. Instead of scaling a globally coordinated "eye in the sky" scheduler, like Nomad and K8s offers, we relaxed a constraint ("when you ask to run a job, we'll move heaven and earth to put it somewhere") and wound up with a totally different scheduling model (a market-based system that bids on resources, where requests to place jobs are all effectively fill-or-kill limit orders).

This was a bet. We're bullish about this bet! Even without K8s, having core scheduling be "less reliable" but with a simpler, more responsive interface puts us in a position to do some of the "move heaven and earth" work that K8s and Nomad do in simpler components (like: we can write Elixir code to drive the scheduler).

But it might not pay off! That's what makes it a bet.

† (see: comments on this thread asking why overengineered and wrote out own version of stuff; the expectation that you'd run a platform like Fly.io on standard K8s or Nomad is pretty strong!).

Re: Fly Kubernetes

#48

Post by Dowwie »

Wouldn't it have cost less to enhance the Nomad scheduler rather than move to, and enhance, Kubernetes?

This aside, Fly is in a position to build its own alternative to K8s and Nomad from scratch, so maybe it will?

Re: Fly Kubernetes

#49

Post by tptacek »

tootie wrote: post #29

This is impressive, but also seems to fly in the face of their raison d'etre. I don't even bother with k8s on AWS because it's too complex for even a mid-size operation. Isn't the point of PaaS to obscure complexity?

We're not replacing Fly.io and the Fly Machines API and the Fly Launch stuff in `flyctl` with FKS. FKS is just there for people who want a K8s interface. If you're not interested in K8s at all, you shouldn't touch FKS.

Re: Fly Kubernetes

#50

Post by netshade »

tick_tock_tick wrote: post #41
netshade wrote: post #31

I am a current Fly customer (personal and work), and have been happy with the service. Will likely be trying this out. That said, the marketing tone of this final part of the blog: > More to come! We’re itching to see just how many different ways this bet might pay off. Or: we’ll perish in flames! Either way, it’ll be fun to watch. is like nails on the chalkboard for me.

Why? If you're using something like Fly you should 100% always have a fallback plan ready. You are gambling using smaller players to get cheaper services or some other benefit the big players don't offer in exchange for the very real possibility of a random day they announce 30 days til they permanently shutdown with zero migration path. I don't think it's in poor taste to acknowledge exactly what everyone should und…

You can communicate those ideas specifically without hiding it beneath the veneer of relatability. The entire post started with this bit:

> But, come on: you never took us too seriously about K8s, right? K8s is hard for us to use, but that doesn’t mean it’s not a great fit for what you’re building. We’ve been clear about that all along, right? Sure we have!

which already starts the post in a bad space for the reader. I have cognitive whiplash from what is intended. "We DON'T like Kubernets UNTIL WE DO but then WE MIGHT NOT IN THE FUTURE". Clear meaning is far more appreciated.

Post reply on HN