Connecting Kubernetes services with linkerd • Hacker News Board

Re: Connecting Kubernetes services with linkerd

#21

Post by mwpmaybe » Mon, Apr 10, 2017, 10:41 PM UTC

cookiecaper wrote: post #17

mwpmaybe wrote: post #12

Earlier quoted context omitted.

> If you want Ruby, the habit is to go find someone's "pre-configured" alpine-ruby image (for instance) I think the habit for most folks (and the best practice) is to use one of the "official" Ruby images[0]. Similar official repositories exist for many server daemons, language runtimes, and command-line tools[1]. > My understanding is that most public image repositories are unmoderated. To the best of my understandi…

>I think the habit for most folks (and the best practice) is to use one of the "official" Ruby images. Ah, I didn't realize there was a blessed subset of repositories for which the Docker team has independently verified the identity. Are these signed with some sort of crypto and known-good keys, similar to package signing keys attached to a package repo, or are we taking Docker's word for it? Is there a way to filter…

> Are these signed with some sort of crypto and known-good keys, similar to package signing keys attached to a package repo, or are we taking Docker's word for it?

I'm not sure. You can find more information about them here[0].

> Is there a way to filter to "official" repositories only, or tell Docker to only fetch from a whitelist of repositories (so that private ones can be included)?

I'm not sure about this either. It looks like Rancher and OpenShift support whitelisting registries, so this may have to be handled at a higher level. I believe you can always tell whether something being pulled from Docker Hub is official because it won't have a leading path component indicating the contributor (e.g. mwpmaybe/ruby:2.4-alpine is a non-official image whereas ruby:2.4-alpine is official).

> the habit of casually importing images is still rampant

This is just my two cents, but that seems like a policy problem, not a technology problem. Enforce a rule for your project/team/organization that containers have to use an official image, or a private image based on an official image. I do agree that a simple per-registry namespace whitelist would be a nice feature to have in dockerd.

> a) still derived from some other upstream Dockerfile (must this file also be validated by Docker to qualify as an "official repo"?)

I believe official images must be based on other official images.

> b) a good example of the "brilliance" of the Dockerfile format

I don't think it was ever meant to be brilliant, just simple. I'm definitely looking forward to the next generation of Dockerfile formats that ameliorate this issue and many others.

> Is this "transparency" only the case on "official" images? As far as I can see Docker Hub just stores the pushed binary blob and not the Dockerfile required to build it, so if you're not picking your image base carefully, there may not be a way to verify its contents (short of introspecting the image).

I believe so. This is a complaint I have with Docker Hub. I personally feel that the Dockerfiles used to create public images should always be published along with the images.

0. https://docs.docker.com/docker-hub/official_repos/

Re: Connecting Kubernetes services with linkerd

#22

Post by cookiecaper » Mon, Apr 10, 2017, 10:50 PM UTC

mwpmaybe wrote: post #21

cookiecaper wrote: post #17

Earlier quoted context omitted.

>I think the habit for most folks (and the best practice) is to use one of the "official" Ruby images. Ah, I didn't realize there was a blessed subset of repositories for which the Docker team has independently verified the identity. Are these signed with some sort of crypto and known-good keys, similar to package signing keys attached to a package repo, or are we taking Docker's word for it? Is there a way to filter…

> Are these signed with some sort of crypto and known-good keys, similar to package signing keys attached to a package repo, or are we taking Docker's word for it? I'm not sure. You can find more information about them here[0]. > Is there a way to filter to "official" repositories only, or tell Docker to only fetch from a whitelist of repositories (so that private ones can be included)? I'm not sure about this either…

>This is just my two cents, but that seems like a policy problem, not a technology problem.

Software that makes dangerous things too easy is a technology problem. This is why Linux distros require signed packages by default.

Furthermore, one of Docker's popular appeals is "never get stuck in dependency hell again: just install someone's container!" You see more and more projects saying "The best way to install this is just to use our Docker container." The culture is definitely not leading to a security-minded perspective.

Compare with something like Ansible Playbooks, where the commands executed are always visible.

>I don't think it was ever meant to be brilliant, just simple.

Requiring long continuations for reasonable image sizes is not simple. Without the continuations, final image sizes regularly bloat to be larger than a typical distro ISO. On the other hand, exchanging the cached layers for size nullifies some of Docker's claimed value, specifically wrt atomicity and build time.

Re: Connecting Kubernetes services with linkerd

#23

Post by xnxn » Mon, Apr 10, 2017, 10:52 PM UTC

cookiecaper wrote: post #17

mwpmaybe wrote: post #12

Earlier quoted context omitted.

> If you want Ruby, the habit is to go find someone's "pre-configured" alpine-ruby image (for instance) I think the habit for most folks (and the best practice) is to use one of the "official" Ruby images[0]. Similar official repositories exist for many server daemons, language runtimes, and command-line tools[1]. > My understanding is that most public image repositories are unmoderated. To the best of my understandi…

>I think the habit for most folks (and the best practice) is to use one of the "official" Ruby images. Ah, I didn't realize there was a blessed subset of repositories for which the Docker team has independently verified the identity. Are these signed with some sort of crypto and known-good keys, similar to package signing keys attached to a package repo, or are we taking Docker's word for it? Is there a way to filter…

> Is there a way to filter to "official" repositories only, or tell Docker to only fetch from a whitelist of repositories (so that private ones can be included)?

In Kubernetes land, you can use the ImagePolicyWebhook admission controller. https://kubernetes.io/docs/admin/admission-controllers/#imag...

I agree that Dockerfiles are awful.

Re: Connecting Kubernetes services with linkerd

#24

Post by mwpmaybe » Mon, Apr 10, 2017, 11:01 PM UTC

cookiecaper wrote: post #22

mwpmaybe wrote: post #21

Earlier quoted context omitted.

> Are these signed with some sort of crypto and known-good keys, similar to package signing keys attached to a package repo, or are we taking Docker's word for it? I'm not sure. You can find more information about them here[0]. > Is there a way to filter to "official" repositories only, or tell Docker to only fetch from a whitelist of repositories (so that private ones can be included)? I'm not sure about this either…

>This is just my two cents, but that seems like a policy problem, not a technology problem. Software that makes dangerous things too easy is a technology problem. This is why Linux distros require signed packages by default. Furthermore, one of Docker's popular appeals is "never get stuck in dependency hell again: just install someone's container!" You see more and more projects saying "The best way to install this i…

> Requiring long continuations for reasonable image sizes is not simple.

We're just arguing semantics at this point. I mean "simple" like Assembler is simple: rudimentary instructions requiring a certain amount of understanding and sophistication to wield correctly. Not "simple" as in easy, intuitive, and footgun-free.

Re: Connecting Kubernetes services with linkerd

#25

Post by jimktrains2 » Mon, Apr 10, 2017, 11:48 PM UTC

alpb wrote: post #15

jimktrains2 wrote: post #5

> Facilities that linkerd provides to assist with this include service discovery, load balancing, encryption, tracing and logging, handling retries, expiration and timeouts, back-offs, dynamic routing, and metrics. Why not use SRV records and TLS? Retries, expiration, timeouts, and back-offs can all be placed into the networking library, and are, commonly. Also, regarding https://linkerd.io/ , what's with landing pag…

No programming language's HTTP/RPC stack supports SRV records. They are just not a thing. Changing your microservices to know about SRV is not a great solution either. Using something like linkerd frees your application from concerns like how discovery and transport is done. SRV is also not compatible with most components out there. e.g. you can't tell memcached/redis to resolve SRV records. You should take a look at…

Multiple A records are also fine. Also more supported as you pointed out.

Unless you're constantly talking to a service, your stats will get old quickly. It could also cause everyone to target a specific host creating problems.

I'm still not sure what service discovery is required beyond knowing the name of a service.

Linkerd doesn't free me from anything. I still need to know how it's done, how it's configured, and how to fix it when it breaks.

Re: Connecting Kubernetes services with linkerd

#26

Post by jimktrains2 » Mon, Apr 10, 2017, 11:49 PM UTC

cookiecaper wrote: post #9

jimktrains2 wrote: post #5

> Facilities that linkerd provides to assist with this include service discovery, load balancing, encryption, tracing and logging, handling retries, expiration and timeouts, back-offs, dynamic routing, and metrics. Why not use SRV records and TLS? Retries, expiration, timeouts, and back-offs can all be placed into the networking library, and are, commonly. Also, regarding https://linkerd.io/ , what's with landing pag…

Disclaimer: I'm still researching linkerd and haven't tried to use it. One component we're interested in is its "automatic" tracing. It claims to be able to annotate requests with Zipkin stats without requiring any instrumentation at the software level. You are correct that other solutions exist for service discovery, but that's an in-demand feature right now as well, and afaik the main reason that people use linkerd…

I'm personally a big BSD jails fan. The idea of containerization is nice, but people often forget that it requires tooling and care, just like anything. And isn't a magic bullet

Re: Connecting Kubernetes services with linkerd

#27

Post by stock_toaster » Tue, Apr 11, 2017, 12:02 AM UTC

Artemis2 wrote: post #16

Lyft has a tech similar to linkerd called Envoy: https://github.com/lyft/envoy . It is a bit lower level than linkerd (based on APIs rather than already integrated with common services) and written in C++. I'm working on an adapter which should offer a better integration with Kubernetes. The end goal is to have envoy be able to replace our Ingress controllers. Not having to run the JVM on every machine is also nice.

  > Not having to run the JVM on every machine is also nice.

Personally, Envoy compiling to a binary makes it much more likely that I would experiment with it than with linkerd. I have hesitated trying out linkerd because we don't run anything JVM based in any of our stack, and we aren't currently excited about introducing any.

I am definitely going to take a look at Envoy. Thanks for referencing it!

Re: Connecting Kubernetes services with linkerd

#28

Post by TheIronYuppie » Tue, Apr 11, 2017, 12:23 AM UTC

cookiecaper wrote: post #20

rubiquity wrote: post #7

Earlier quoted context omitted.

Yes, it's insane. I can only attribute it to that I don't work on problem of any scale (amount of apps, servers, deployments per day, etc.) and all of these people do. Who really wants to spend their time and effort running a Kubernetes master and nodes, 3+ Consul/etcd nodes, and now some fancy load balancers? The ratio of companies that would benefit from that to the number of companies working on products in the sp…

I would not take the claims of scale at face value. People make this claim to boost their own egos, and the reality is that many of the people who do this would be more than fine on a handful of servers. They just want to feel important. My frank take is that k8s was developed at Google by the seat of their pants (and yes, I know it's their third-gen orchestration platform), they talked about it with outsiders here a…

Just FYI, over 50% of contributions come from !Google, and that's part of the reason that so many people (CoreOS, Redhat, Apprenda, Deis, thousands of internal folks) have rebased on it.

I saw some of your other comments - I'd love to understand better what we could do to either a) make it feel like not an internal tool and b) if you didn't use Kubernetes or containers, how you'd prefer to run distributed workloads.

TBC, I totally agree with you - if you're running 1-2 nodes, and don't care about downtime, you should NEVER use Kubernetes or any orchestrator. Once you get to 3+, however, I can't imagine using anything else.

Disclosure: I work at Google on Kubernetes

Re: Connecting Kubernetes services with linkerd

#29

Post by 220 » Tue, Apr 11, 2017, 12:30 AM UTC

jimktrains2 wrote: post #5

> Facilities that linkerd provides to assist with this include service discovery, load balancing, encryption, tracing and logging, handling retries, expiration and timeouts, back-offs, dynamic routing, and metrics. Why not use SRV records and TLS? Retries, expiration, timeouts, and back-offs can all be placed into the networking library, and are, commonly. Also, regarding https://linkerd.io/ , what's with landing pag…

I don't think SRV records are the right answer; your networking layer should be k8s aware and issue a watch command on Endpoints; it'll be updated immediately-ish when the servers changes. This is similar to how finagle's zookeeper server set is supposed to work.

What linkerd buys you is you don't have to write this type k8s-aware, zipkin logging library for every language you're running in production. But I think it's straddling a very narrow section; small users shouldn't care about this and just rely on the round robin, zipkin is a PITA to run anyway. Large users will probably want to write their own libraries (zipkin is a lot better if you put traces in your process)

Re: Connecting Kubernetes services with linkerd

#30

Post by lobster_johnson » Tue, Apr 11, 2017, 12:58 AM UTC

cookiecaper wrote: post #20

rubiquity wrote: post #7

Earlier quoted context omitted.

Yes, it's insane. I can only attribute it to that I don't work on problem of any scale (amount of apps, servers, deployments per day, etc.) and all of these people do. Who really wants to spend their time and effort running a Kubernetes master and nodes, 3+ Consul/etcd nodes, and now some fancy load balancers? The ratio of companies that would benefit from that to the number of companies working on products in the sp…

I would not take the claims of scale at face value. People make this claim to boost their own egos, and the reality is that many of the people who do this would be more than fine on a handful of servers. They just want to feel important. My frank take is that k8s was developed at Google by the seat of their pants (and yes, I know it's their third-gen orchestration platform), they talked about it with outsiders here a…

As a Kubernetes user, I don't understand the "vibe of an internal tool" comment. It feels nothing like that to me.

Kubernetes is big and complex, to be sure, but to me its state of being unfinished is not because it's "internal" anything, but because — since Google couldn't just open-source Borg and Omega, which presumably have a decade of Google-specific cruft that wouldn't translate to a reusable tool — it had to be built from scratch, and is only slowly accruing the various features needed to approximate the maturity of a full-blown orchestration system of the kind that Google uses already.

In other words, Kubernetes works great and all, but it's also immature in many ways: Ingress is painful, secrets management not secure enough; we're only now getting stateful sets, affinities, federation, role-based security, etc.; we're still waiting for things like scheduled jobs and custom metrics support in the autoscaler; Kubelet is still an awkward SPOF; etc. But those shortcomings have nothing to do with it being an internal tool, but rather because there's just a lot of work left to be done.