If you want "k8s but less" I highly recommend Nomad from Hashicorp and the Hashi-stack in general. It's so damn good.
MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT
121–130 of 138 posts
Re: MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT
#122Earlier quoted context omitted.
The classic "it depends" answer applies here, along two axes: container assembly, and container execution. If you mean "docker" the binary used to build container images, you don't even need that right now -- there are multiple projects that will build container images without involving docker or dockerd. If you mean "dockerd" the container management engine that one controls via "docker" to start and stop containers…
Do you think Docker will disappear and everybody will some kind of "single binary kubernetes" on their machine + A tool to build the images.
Unlikely, although I for sure don't trust dockerd for my actual clusters anymore. I still use docker, and its dind friend, to build images because the whole world uses Dockerfile and the compatibility story between those alternative tools and the sea of Dockerfiles out there is ... tricky
I would love for something with more sanity to catch on, because Dockerfile as a format has terrible DX, but "wishes horses etc etc."
Re: MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT
#123Earlier quoted context omitted.
Edge DC "dropping packets for several minutes"? Where have you been suffering from this?
Real case - shops have "terminals" installed on site, various physical locations. Some of them have networking glitches 1-2 times a month on average, usually lasting a couple of minutes. I don't want to have to restart the whole thing on each site every time it happens. I'd like a deployment/orchestration system that can work in such scenarios, showing a node as unreachable but then back online when it gets network b…
> showing a node as unreachable but then back online when it gets network back.
Isn't that exactly what happens with K8s worker nodes? They will show as "not ready" but will be back once connectivity is restored.
EDIT: Just saw that the intention is to have some nodes in a DC and some nodes in the edge and the intention is to have a single K8s cluster spanning both locations with unreliable network in between. No idea how badly the cluster would react to this.
Re: MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT
#124I thought Kubernetes is not great for environments with poor network connectivity, which is quite common when dealing with Edge and IoT scenarios. Has that changed?
> I thought Kubernetes is not great for environments with poor network connectivity,
No, it's ok. What you don't want to have is:
* Poor connectivity between K8s masters and ETCD. The backing store needs to be reliable or things don't work right. If it's an IOT scenario, it's possible you won't have multiple k8s master nodes anyway. If you can place etcd and k8s master in the same machine, you are fine.
You need to have a not horrible connection between masters and workers. If connectivity gets disrupted for a long time and nodes start going NotReady then, depending on how your cluster and workloads are configured, K8s may start shuffling things around to work around the (perceived) node failure(which is normally a very good thing). If this happens too often and for too long time it can be disruptive to your workloads. If it's sporadic, it can be a good thing to have K8s route around the failure.
So, if that is your scenario, then you will need to adjust. But keep in mind that no matter what you do, if network is really bad, you would have to mitigate the effects regardless, Kubernetes or not. I can only really see a problem if a) network is terrible and b) your workloads are mostly computing in nature and don't rely on the network (or they communicate in bursts). Otherwise, a network failure means you can't reach your applications anyway...
Re: MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT
#125Earlier quoted context omitted.
Do you mean connectivity to the outside, or inside the cluster? The examples of Kubernetes and similar things in such scenarios I've seen usually had stable connectivity between nodes. E.g. an edge scenario would be one tiny well-connected cluster per location, remote-controlled over the (bad) external link through the API.
I meant intra-cluster communication between nodes, when some nodes are on the Edge, some are inside the datacenter. The Edge may have pretty good overall connection to DC, but have to work with intermittent connectivity problems like dropping packets for several minutes, etc., without going crazy.
> I meant intra-cluster communication between nodes, when some nodes are on the Edge, some are inside the datacenter.
Don't do this. Have two K8s clusters. Even if the network were reliable you might still have issues spanning the overlay network geographically.
If you _really_ need to manage them as a unit for whatever reason, federate them(keeping in mind that federation is still not GA). But keep each control plane local.
Then setup the data flows as if K8s wasn't in the picture at all.
Re: MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT
#126Earlier quoted context omitted.
Canonical is just one of several players in this area. There's also k3s and k0s, for example. I suppose it might depend on what you count as "edge", but we're using kubernetes to distribute a complex product to customers onprem. The product has multiple databases, services, transient processes, scheduled jobs, and machine learning. It needs to be able to run on a single machine or a cluster depending on customer requ…
on-prem is not edge IMO. Edge is something small, far away from datacenters, close to customers with limited compute and storage capacity. Did I get this wrong? For example, SQLite advertise itself as "database on edge".
"Edge" is kind of a relative term. E.g. relative to a public cloud, anything not in that cloud could be considered edge. Or, if a company's systems are in a shared datacenter somewhere, then the systems in their actual offices might be considered "edge".
Just as an example (not saying it's authoritative):
> "Edge computing is often referred to as 'on-premise.'"
-- https://dzone.com/articles/demystifying-the-edge-vs-cloud-computing
Btw since you mention SQLite, the k3s system uses SQLite instead of the default etcd used by Kubernetes, for the reason you mention. These systems really are intended to support true edge scenarios. K3s is distributed as a single 40 MB binary, and you can run it on non-PC edge hardware.
For anyone who runs a system that involves multiple containers on a single machine, it can be worth looking at systems like k3s as an alternative to e.g. Docker Compose. There's not much downside other than some learning curve, and it gives you a wealth of capabilities that you otherwise tend to end up hacking together with scripts or whatever.
Re: MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT
#127Looking forward to this being decoupled from snapd eventually. Until then, not the fientist hope I'd touch this when any alternative exists where snapd can be avoided
I feel like snapd must be hindering the uptake of microK8s, surely? Canonical is pushing it hard, but snapd is a hard no.
Every time I see it mentioned, I check to see if they ditched snapd yet, alas, today is not the that day.
Re: MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT
#128How is persistent storage handled on microk8s (or k3s)?
If you want more than just local disk (directory/path on the node the pods currently reside on) you will have to deploy your own storage provider to handle PVCs (on k3s at least). According to Microk8s docs, it's the same there.
OpenEBS is a plug-in too which opens up lots of options
Re: MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT
#129I moved from k3s to microk8s for local development. I gave up on k3s because I needed calico CNI and it was a pain to set up, on microk8s it's just `microk8s enable calico`. I also found k3s a bit too opinionated with the default Traefik ingress and service-lb.
Traefik is probably the best Ingress out there capability wise for now I think. I've written a bit on it[0] before, but IMO that choice is a good one. Even used it to do some fun external-in SSH connections[1]. I also use it to run a multi-tenant email setup (haraka @ the edge + maddy). It's not like NGINX can't run SMTP expose other ports, but Traefik is easier to manage -- CRDs instead of a ConfigMap update. [0]: h…
Is Traefik difficult to initially configure? I use the default ingress-nginx controller on my home lab setup (separate from the “official”, non default & non-free nginx-ingress, absolutely terrible name choices) and it seems to be ok for smaller use cases. It’s not without its idiosyncrasies though.
At previous employment with large scale clusters (thousands of nodes) Traefik seemed to be heavily preferred by the SRE’s in my org.
Re: MicroK8s – Low-ops, minimal Kubernetes, for cloud, clusters, Edge and IoT
#130Earlier quoted context omitted.
Traefik is probably the best Ingress out there capability wise for now I think. I've written a bit on it[0] before, but IMO that choice is a good one. Even used it to do some fun external-in SSH connections[1]. I also use it to run a multi-tenant email setup (haraka @ the edge + maddy). It's not like NGINX can't run SMTP expose other ports, but Traefik is easier to manage -- CRDs instead of a ConfigMap update. [0]: h…
Is Traefik difficult to initially configure? I use the default ingress-nginx controller on my home lab setup (separate from the “official”, non default & non-free nginx-ingress, absolutely terrible name choices) and it seems to be ok for smaller use cases. It’s not without its idiosyncrasies though. At previous employment with large scale clusters (thousands of nodes) Traefik seemed to be heavily preferred by the SRE…
I personally think it's not that bad -- the documentation is a bit overwhelming because there are so many ways to configure it (some people are running with only docker, others using the CRDs, some people are using the built-in Ingress support, some using the new Gateway stuff). As with most things, if you read and mostly digest the documentation, you won't feel lost when it comes to setting it up -- I haven't run into any corners that were too hard to figure out or inconsistent (which is even worse).
You could get Traefik working inside your existing cluster actually by just letting NGINX route to it and seeing how easy it is to use that way -- though that may be more difficult than just spinning up a cluster on a brand new machine (or locally) and feeling your way around.
I can say that Traefik's dashboard makes it much easier to debug while it's running as it gives you a fantastic amount of feedback, Prometheus built in, etc.