Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fly Kubernetes

fly.io

Post reply on HN
Print view View original

141–150 of 180 posts

Re: Fly Kubernetes

#141

Post by kasey_junk »

verdverm wrote: post #140
kasey_junk wrote: post #124

Earlier quoted context omitted.

I’ve looked extensively at the documentation for gke autopilot (a system I use extensively) and haven’t found any documentation on how they orchestrate those clusters under the covers. I’ve always assumed it was a plugin to borg, not a different fleet orchestrator. Not to be overtly needy, but do you have a link to docs that contradicts that?

- https://cloud.google.com/kubernetes-engine/docs/concepts/clu... - https://cloud.google.com/kubernetes-engine/docs/concepts/kub... Each of these has more peer pages that are pretty good too

You’ll note those documents are extremely careful in describing the control plane architecture to not promise you are running a stock k8s install. Which is why I’ve always assumed otherwise.

But I’ll trust the sibling comment which suggests the only bespoke component in gke is storage well enough to leave it alone.

Re: Fly Kubernetes

#142

Post by javaunsafe2019 »

kuhsaft wrote: post #127
javaunsafe2019 wrote: post #125

Earlier quoted context omitted.

Why should you do this - sounds like an antipattern to me

It’s used widely in the Kubernetes world and is known as sidecars [0]. [0] https://kubernetes.io/blog/2023/08/25/native-sidecar-contain...

Widely? I lived in this helm, kubernetes, pulumi world the past 4 years and we followed the simple rule of one service/container per pod. Why add complexity where it’s not needed. Like running a dB a and a service in the same docker container - a no go for me and many.

Re: Fly Kubernetes

#143

Post by verdverm »

0xbadcafebee wrote: post #139
verdverm wrote: post #26

Earlier quoted context omitted.

You're basically describing Kubernetes and why it has become so popular

K8s is the opposite. It's proprietary, insular, not compatible with anything else, tightly coupled, not layered, not backwards compatible, etc. It has network services, logging, auth, etc, but so does literally every other system in the world, that doesn't make them all identical. K8s is popular because it's free, has a lot of bells and whistles, and was made by Google. Otherwise nobody would use it. It's basically a…

> K8s is the opposite. It's proprietary, insular, not compatible with anything else

Not accurate, k8s is open source and every major tech company is developing or using it. There are a large number of companies building on top of it as well.

- https://github.com/kubernetes (open source)

- https://landscape.cncf.io/ (huge ecosystem)

- https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1 (count of contributions by company)

> It's basically a larger, slightly less crappy Jenkins.

What? This is not even remotely accurate. Where did you come to this opinion?

K8s replaced mesos/marathon, which was the dominant open source orchestration system at the time. Jenkins is a CI system and can run on k8s. There is also JenkinsX that is trying to be the yaml driven, k8s native platform, but I think it missed the mark. There are better k8s native CI/CD systems like the Argo projects

Re: Fly Kubernetes

#144

Post by spankalee »

btown wrote: post #80
benpacker wrote: post #8

Am I understanding correctly that because they map a “Pod” to a “Fly Machine”, there’s no intermediate “Node” concept? If so, this is very attractive. When using GKS, we had to do a lot of work to get our Node utilization (the percent of resources we had reserve on a VM actually occupied by pods) to be higher than 50%. Curios what happens when you run “kubectl get nodes” - does it lie to you, or call each region one…

GKE Autopilot is an attractive option here if you don't want to worry about node utilization and provisioning. Effectively you have an on-demand infinitely-sized k8s cluster that scales up and down as you need new pods. Some caveats, but it's an incredible onramp if you're coming from a Heroku or similar PaaS and don't want to worry about the infrastructure side of things: Github Actions building images and deploying…

Why would you use GKE Autopilot over Cloud Run?

Re: Fly Kubernetes

#145

Post by paxys »

nailer wrote: post #113
paxys wrote: post #19

Earlier quoted context omitted.

Maybe a got fit for someone who is reluctant to use Kubernetes but has to for whatever reason.

If someone isn't a cloud provider they should be reluctant to use Kubernetes.

Plenty of companies/teams qualify as "cloud providers" even though all the usage is internal.

Re: Fly Kubernetes

#146

Post by qeternity »

javaunsafe2019 wrote: post #142
kuhsaft wrote: post #127

Earlier quoted context omitted.

It’s used widely in the Kubernetes world and is known as sidecars [0]. [0] https://kubernetes.io/blog/2023/08/25/native-sidecar-contain...

Widely? I lived in this helm, kubernetes, pulumi world the past 4 years and we followed the simple rule of one service/container per pod. Why add complexity where it’s not needed. Like running a dB a and a service in the same docker container - a no go for me and many.

Nobody would suggest this. But what about a metrics scraper for your db pod? That’s where sidecars come in.

Re: Fly Kubernetes

#147

Post by GauntletWizard »

kuhsaft wrote: post #127
javaunsafe2019 wrote: post #125

Earlier quoted context omitted.

Why should you do this - sounds like an antipattern to me

It’s used widely in the Kubernetes world and is known as sidecars [0]. [0] https://kubernetes.io/blog/2023/08/25/native-sidecar-contain...

Like helm, it's a widely used anti-pattern

Re: Fly Kubernetes

#148

Post by kuhsaft »

javaunsafe2019 wrote: post #142
kuhsaft wrote: post #127

Earlier quoted context omitted.

It’s used widely in the Kubernetes world and is known as sidecars [0]. [0] https://kubernetes.io/blog/2023/08/25/native-sidecar-contain...

Widely? I lived in this helm, kubernetes, pulumi world the past 4 years and we followed the simple rule of one service/container per pod. Why add complexity where it’s not needed. Like running a dB a and a service in the same docker container - a no go for me and many.

The thing is you might not be adding the sidecars yourself.

Kubernetes has a resource called MutatingAdmissionWebhook that allows mutations of the Pod object before creation.

I think the most common use case is for service meshes. Your Deployment might not have any sidecars, but the service mesh controller will automatically add a network proxy container to your pod via the admission webhook.

Another use case would be OpenTelemetry or some other observability service sidecar injection for auto-instrumentation.

Re: Fly Kubernetes

#149

Post by therein »

zifnab06 wrote: post #43
therein wrote: post #32

Earlier quoted context omitted.

I remember asking about this about half a year ago. Back then I was told since Fly can route these IPs to wherever they want in their infrastructure by making simple configuration changes, the line is blurred anyway but GeoIP providers just take the shortcut and resolve to where the company is registered. I'll try again if the situation is different now.

Inbound IPs use anycast - the IP we give you routes to the nearest fly region and then hits a wireguard tunnel to get to the correct region. Outbound IPs are tied to the individual host your machine is running on (which is located in a single region). These are tied to a single region, and in some cases, these were resolving to the wrong region in a few of the GeoIP databases. We hopefully fixed this part.

Thank you, I will try again today. It would unblock deploying my usecase on Fly.

Re: Fly Kubernetes

#150

Post by dangoodmanUT »

This is really exciting, but there are a few things they will certainly have to work through:

*Services:*

Kubernetes expects DNS records like {pod}.default.svc.cluster.local. In order to achieve this they will have to have some custom DNS records on the "pod" (fly machine) to resolve this with their metadata. Not impossible, but something that has to be take into account.

*StatefulSets:*

This has 2 major obstacles:

The first is dealing with disk. k8s expects that it can move disks to different logical pods when they lose them (e.g. mapping EBS to an EC2 node). The problem here is that fly has a fundamentally different model. It means that it either has to decide not to schedule a pod because it can't get the machine that the disk lives on, or not guarantee that the disk is the same. While this does exist as a setting currently, the former is a serious issue.

The second major issue is again with DNS. StatefulSets have ordinal pod names (e.g. {ss-name}-{0..n}.default.sv.cluster.local). While this can be achieved with their machine metadata and custom DNS on the machine, it means that it either has to run a local DNS server to "translate" DNS records to the fly nomenclature, or have to constantly update local services on machines to tell them about new records. Both will incur some penalty.

Post reply on HN