31–36 of 36 posts
Are there any cloud providers providing remote disks without replications? It looks such needs are popular for deploying databases in which replications are maintained by the databases themselves.
That's effectively what EBS is, no?
I would recommend against running stateful apps in kubernetes. It's not really ready for it. Big problems include routing (it works fine for http requests, but not for DBs, message brokers, etc) and just the pain of setting up stateful sets. If you don't believe me, take it from someone who should know what they're talking about: https://twitter.com/kelseyhightower/status/96341350830081229...
We run stateful apps on Kubernetes. There are obvious rough areas (the lack of persistent volume resizing, for example, which is scheduled for 1.11), but overall, it's great. What a lot of naysayers leave out, or choose to ignore, is that the challenges running stateful apps on Kubernetes mirror those of running stateful apps anywhere. If you run Postgres on a VM, for example, you're completely reliant on that VM sta…
1. While "we already rely on VM staying up", with k8s we reply on both VM staying up and kubernetes infra on top of that VM staying UP 2. Maintaining a complex stateful system on k8s _requires_ having and maintaining an operator for that system. 3. You reduce your options when it comes to tweaking systems, e.g. local SSDs on GCP are available in SCSI and NVMe flavors, while GKE supports only SCSI; harder to perform fine-tuning and other tasks on the underlying VMs that would have been trivial with Chef or similar. 4. Enterprise systems like Splunk explicitly mention that their support does not cover Splunk clusters running on kubernetes. 5. As mentioned, you can't even resize a disk without going through dance of operations that would take days or weeks when you're working something like Kafka at scale. 6. Some stateful services like Zookeeper require stable identities and this is far from perfect on kubernetes. 7. More complex traffic routing that involves additional fees because to achieve (6) you sometimes need to expose things publicly.
That's just from the top of my head.
Disclaimer: We run 10+ stateful services on Kubernetes.
Has anyone looked at Service Fabric (Microsoft tech) for things like this? That has offered stateful services for years now. I'm pretty sure it runs on Linux, and I've seen that it's Docker compatible. I know it's kinda in the same space as K8s but I don't really know the details. Would SF be able to do something like this in a similar (or better?) way?
As zapita said, Service Fabric now handles containers but I think it is just because containers became trendy and FOMO kicked in.
Where Service Fabric is decades ahead of the container orchestration solutions is as a framework to build truly stateful services, meaning the state is entirely managed by your code through SF, not externalized in a remote disk, Redis, some DB, etc...
It offers high level primitives like reliable collections [0], as well as very low level primitives like a replicated log to implement custom replication between replicas [1]. I feel that publicly this is not advertised enough and it is unfortunate because it is a key differentiator for Service Fabric that the competitors won't have for a while, if ever because it is a completely opposite approach: containers are all about isolation, being self-contained and plateform independent while SF stateful services are deeply integrated with Service Fabric.
[0] https://docs.microsoft.com/en-us/azure/service-fabric/servic...
[1] https://docs.microsoft.com/en-us/dotnet/api/system.fabric.fa...
For example if you ran CDB on a baremetal cluster of 3 nodes with 30TB of raw capacity, 15TB is lost to RAID10, 10TB is lost to running a replicated database such as cockroach DB, leaving you with 5TB effective capacity which is a 1/6 dilution of your initial capacity.
If you ran cockroach DB on a replicated network volume, with a replication factor of three, it gets worse. If you bought 30 TB of disks, you'd lose 20 TB to volume replication, ~6.67TB to CDB replication leaving you with 3.3TB of effective capacity or a 1/9 dilution. If those disks were configured with RAID your effective capacity would drop to a 1/18 dilution.
You could achieve a 1/3 dilution which is the effective minimum for a replicated database if you didn't configure RAID, but you increase the impact of disk failure, in that it would take much much longer to recover a cluster.
Earlier quoted context omitted.
Note that none of those things are impossible on kubernetes, k8s just doesn't offer them by default (which is good IMO). There are projects that help you run databases in kubernetes and also make backups of many things hosted: - Automatic CephFS for your cluster -> https://rook.io/docs/rook/master/ - Backups for cluster resources and volumes -> https://github.com/heptio/ark - Spin up dynamic postgres clusters -> http…
Well with k8s 1.10+ it's also possible to use statefulsets and local volume, so with affinity it's possible to just use k8s as an orchestration system where you "install" your database and keep it up to date with k8s. of course if a node goes down you need to failover, etc. but patroni/zalando postgres works really well with statefulsets and local volume. (as long as a single node is still running, which should alway…
That's what I was doing until rook came around. If you're running in something like AWS (or even if you're not), you can also do something like attach an EBS volume (to a local host) and do that. Or, you can set up a plain ISCSI drive (or get one from your provider, even basic providers these days might offer storage that way) and use that.
There have been a LOT of choices for a long time.
Earlier quoted context omitted.
Well with k8s 1.10+ it's also possible to use statefulsets and local volume, so with affinity it's possible to just use k8s as an orchestration system where you "install" your database and keep it up to date with k8s. of course if a node goes down you need to failover, etc. but patroni/zalando postgres works really well with statefulsets and local volume. (as long as a single node is still running, which should alway…
I want to note that this has actually been possible since like k8s 1.7, you can just start a DB with node affinity and use hostPath volumes. That's what I was doing until rook came around. If you're running in something like AWS (or even if you're not), you can also do something like attach an EBS volume (to a local host) and do that. Or, you can set up a plain ISCSI drive (or get one from your provider, even basic p…
my "bare metal" provider doesn't ;)