Live data from Hacker News

Fly Kubernetes

fly.io

11–20 of 180 posts

Re: Fly Kubernetes

#11
> To keep things simple, we used Nomad, and instead of K8s CNIs, we built our own Rust-based TLS-terminating Anycast proxy (and designed a WireGuard/IPv6-based private network system based on eBPF).

That is quite the opposite of “simple”. That is in fact, overly complex and engineered.

Re: Fly Kubernetes

#12

Am I understanding correctly that because they map a “Pod” to a “Fly Machine”, there’s no intermediate “Node” concept? If so, this is very attractive. When using GKS, we had to do a lot of work to get our Node utilization (the percent of resources we had reserve on a VM actually occupied by pods) to be higher than 50%. Curios what happens when you run “kubectl get nodes” - does it lie to you, or call each region one…

> we had to do a lot of work to get our Node utilization ... over 50%

Same, a while back you had to install cluster-autoscaler and set it to aggressive mode. GKE has this option now on setup, though I think anyone who's had to do this stuff knows that just using a cluster-autoscaler is never enough. I don't see this being different for any cluster and is more a consequence of your workloads and how they are partitioned (if not partitioning, you'll have real trouble getting high utilization)

Re: Fly Kubernetes

#13

> To keep things simple, we used Nomad, and instead of K8s CNIs, we built our own Rust-based TLS-terminating Anycast proxy (and designed a WireGuard/IPv6-based private network system based on eBPF). That is quite the opposite of “simple”. That is in fact, overly complex and engineered.

What part of it is overly complex and engineered? Maybe you're right, but it's hard to respond without a better idea of what you think our problem domain was.

Re: Fly Kubernetes

#14

> To keep things simple, we used Nomad, and instead of K8s CNIs, we built our own Rust-based TLS-terminating Anycast proxy (and designed a WireGuard/IPv6-based private network system based on eBPF). That is quite the opposite of “simple”. That is in fact, overly complex and engineered.

[deleted]

Re: Fly Kubernetes

#15
post #10
post #2

If they are reluctant and only do it because they have to, are they really the right vendor for managed k8s? What about them makes for a good trade-off when considering the many other vendors?

If their reluctance were based on valid reasons that they handled in a unique way - might be good. In theory.

k8s has become a standard api and platform for running apps, having a * on it makes the implementation an outlier from the standard, not normally considered a good thing because you have to be aware of the nuanced differences.

Re: Fly Kubernetes

#16

> To keep things simple, we used Nomad, and instead of K8s CNIs, we built our own Rust-based TLS-terminating Anycast proxy (and designed a WireGuard/IPv6-based private network system based on eBPF). That is quite the opposite of “simple”. That is in fact, overly complex and engineered.

How do you know their own Anycast proxy isn't simpler than K8s CNIs? Building something yourself isn't necessarily overly complex or over engineered. Sometimes building a simple thing yourself is the way to simplicity when the only available options already built are very heavy/overkill or complex

Re: Fly Kubernetes

#17
post #7
post #4

And fly becomes the standard cloud provider like everyone else. I think this transition is only natural. It's hard to be a big business without catering to the needs of larger companies and that is the operation of many services, not individual apps.

I used Fly for some projects, I really like it. But once again, for many of my projects, I still need my outbound IPs to resolve to a specific country. I can't have them all resolve to Chicago, US in undeterministic ways. I would be willing to pay an additional cost for this but even with reserved IPs, I am given IPs that are labelled as Chicago, US IPs by GeoIP providers even for non US regions.

fwiw - our network folks _should_ have fixed this a few weeks ago. Some of the outbound IPs were incorrectly tagged in some of the geoip databases as being in the US when they were not.

Re: Fly Kubernetes

#18

> To keep things simple, we used Nomad, and instead of K8s CNIs, we built our own Rust-based TLS-terminating Anycast proxy (and designed a WireGuard/IPv6-based private network system based on eBPF). That is quite the opposite of “simple”. That is in fact, overly complex and engineered.

This is all very common platform/infrastructure stuff for any PAAS. Even more-so as multi-tenant k8s (and nics, and nvmeOF, etc) isn't exactly one of the most supported or talked about things. Lots of secret sauce everywhere, but they have to do it in a lot of scenarios.

Re: Fly Kubernetes

#19
post #2

If they are reluctant and only do it because they have to, are they really the right vendor for managed k8s? What about them makes for a good trade-off when considering the many other vendors?

Maybe a got fit for someone who is reluctant to use Kubernetes but has to for whatever reason.

Re: Fly Kubernetes

#20

Am I understanding correctly that because they map a “Pod” to a “Fly Machine”, there’s no intermediate “Node” concept? If so, this is very attractive. When using GKS, we had to do a lot of work to get our Node utilization (the percent of resources we had reserve on a VM actually occupied by pods) to be higher than 50%. Curios what happens when you run “kubectl get nodes” - does it lie to you, or call each region one…

I wonder how it copes with things like anti-affinity rules, where you don't want two things running on the same physical / virtual server for resilience reasons.
Post reply on HN