171–180 of 242 posts
Earlier quoted context omitted.
You seem confuse Borg and borgcfg. The evaluation rules are merely a borgcfg artifact. Disclaimer: I maintain borgcfg.
I found https://jsonnet.org/ to fix several issues with the Borg configuration language. https://github.com/ksonnet/kubecfg is an attempt to reboot the borgcfg experience with k8s + jsonnet
Earlier quoted context omitted.
If you have some time to read "how Google works" you would be surprised by how long the company ran on NFS. I assume there are lots of workloads running on Borg to this day on top of NFS. If that isn't enough for you you should have a look in the client list of Isilon and see which kind of work they do, in case you ever attend a SIGGRAPH most of what you see is built on top of NFS, so, essentially, all of the compute…
Sorry I don’t have to read it because i was borg sre for 6 years and i know how (the server part of) it works. You assume wrong. I know there are a lot of companies that try to put some lipstick on nfs pig and call it reliable/scalable/etc. so long their clients don’t actually try to run it at scale or don’t complain too publicly when they try and can’t, they are able to get away with it.
Supporting all of a Fortune 500’s business operations is very reasonable to call ‘scale’ in the normal world.
Your comment is a like a billionaire claiming that somebody that managed to hit 30 million isn’t rich.
I just went through all of the post-mortems for my own company's purposes of evaluating Kubernetes. I've been running Kubernetes clusters for about a year and a half and have run into a few of these, but here's what I found striking: * About half of the post-mortems involve issues with AWS load balancers (mostly ELB, one with ALB) * Two of the post-mortems involve running control plane components dependent on consens…
Earlier quoted context omitted.
It took me a while to get comfortable in Borg (and in general that your binary can take hundredths of verbosely written command-line arguments (coming from gamedev, I was in a bit of shock state for a while)... But then got used to it - still I felt I could never fully internalize the evaluation rules - but the other tooling (diffing) really helped in that respect. One thing I've really appreciated, was how one could…
You seem confuse Borg and borgcfg. The evaluation rules are merely a borgcfg artifact. Disclaimer: I maintain borgcfg.
Earlier quoted context omitted.
Digital Ocean's K8S offering is out of beta now: https://www.digitalocean.com/products/kubernetes/
It’s good, but the storage layer has some bugs. For example, if you create a pvc, then resize the volume according to their docs, the new size doesn’t reflect in k8s. Also you can create pv’s manually and it won’t show up in the dashboard.
When you say dashboard, which dashboard do you mean? Happy to look at it. Thanks again for the feedback.
I just went through all of the post-mortems for my own company's purposes of evaluating Kubernetes. I've been running Kubernetes clusters for about a year and a half and have run into a few of these, but here's what I found striking: * About half of the post-mortems involve issues with AWS load balancers (mostly ELB, one with ALB) * Two of the post-mortems involve running control plane components dependent on consens…
Nice observation, I haven't done statistics on the linked postportems myself yet. Please note that your observation might also be due to the fact that AWS has a far larger market share and did not provide managed Kubernetes until recently (so people roll their own). We can therefore assume that any random sample of Kubernetes postmortems would be biased towards seeing more incidents with Kubernetes on AWS (compared t…
Earlier quoted context omitted.
Migrated my very small cluster from GKE to DigitalOcean's K8s a few weeks ago. I was using 3 nodes on GKE with 1 core & 3.75GB RAM per node, and the cost was around 100 $ per month including load balancer for the cheapest region, `us-central1-a`. Now, on DigitalOcean, I have 3 nodes with 1 core & 2GB RAM per node. The cost is exactly 40$ including load balancer. I am a pretty basic user, I have started using k8s on t…
The problem with that is that I can almost guarantee that it would still be cheaper and easier to manage if you just leveraged whatever cloud provider's managed service was there to run your stuff.
First, the biggest problem I see is the huge vendor lock-in you accept with the PaaS offerings such as AWS EBS or GCP App Engine. When you commit to one of these platforms, it is really hard to get out of it; it requires engineering effort to move to another provider and feature parity between the providers for your application to be supported. Plus, you get to learn platform-specific stuff which has no standards across providers. Plus, it is usually slow and bloated; have you ever tried deploying something to EBS? It takes at least five minutes without any meaningful information about what is going on or if your deployment succeeded.
Second, the tooling you get is usually very small compared to what Kubernetes ecosystem has. Each and every platform ask you to use their own tools, but there is a high possibility that the tools don't fit your usecase, or you may need to modify your workflow. With a solution like k8s, you only need to support the standard, which is k8s itself roughly, and you are free to use whatever tooling you want.
Third, done right, Kubernetes allows you to move to another provider very easily without changing a single line of code in your Kubernetes definitions or your application. You define the desired state of your cluster, you check in all these stuff into your VCS, and since k8s forces you to do these stuff from the beginning, at the end you usually have a nice, reproducible system that is more or less cloud agnostic. You have logging, horizontal scalability, isolation, easy deployments, easy rollbacks and all that stuff. I have migrated from GKE to DO's Kubernetes offering without changing a single line in my Kubernetes definitions or my application. Of course, my usecase is very very very small compared to most of people around here, but that was my experience.
FWIW, I think Kubernetes is still a good learning to understand about current state of infrastructure, deployments and the ideal state we all try to achieve. Whether or not a business should depend on it is a whole another topic.
I am a developer and I find k8s frustrating. To me, its documentation is confusing and scattered among too many places (best example: overlay networks). I have read multiple books and gazillions of articles and yet I have the feeling that I am lacking the bigger picture. I was able to set it up successfully a couple of times, with more or less time required. Last time, I gave up after four days because I realized tha…
Have you used other google products? I find their documentation routinely incomprehensible and difficult.
Earlier quoted context omitted.
> app developers or platform operators Definitely not the former. The YAML-based configuration is not a pleasant app deployment experience. Companies end up needing to do some sort of auto-generation for it to make it sane for app devs. App developers want experiences similar to heroku. They want to git push and have applications safely roll out without downtime or configuration.
I’m a little behind on Cloud Native adoption so I gotta ask - what’s preventing a “git push to heroku” from being the norm here? Are we using the wrong abstractions? Or are the abstractions still too low? ...why is Heroku/buildpack not running away with it?
Earlier quoted context omitted.
82gb/s (assuming you mean gigabit) is _per-node_ throughput at Google (or FB, or I assume Amazon/Microsoft -- they all use 100GbE networks now). 300K IOPS is probably per-node, too, at this point. :-)
Having a 100gbps nic in a node isn’t the same thing as doing storage at that speed in an HA cluster. Also, don’t confuse 100 gbe networks where spine links are 100 but the node links are only bonded 10s (much more common at $fang).
It's a necessity really. All storage at Google has been remote and distributed for at least the past decade. That puts serious demands on network throughput if you want your CPUs to actually do work and not just sit there and wait for data.
Here's some detail as of 2012: https://storage.googleapis.com/pub-tools-public-publication-.... Note that host speed is 40Gbps. And here's FB talking about migrating from 40Gbps to 100Gbps in 2016: https://code.fb.com/data-center-engineering/introducing-back...