Live data from Hacker News

Service mesh use cases (2020)

lucperkins.dev

41–50 of 50 posts

Re: Service mesh use cases (2020)

#41
post #12

I just wrote something extremely similar, but it's only internal right now. I personally find that the service mesh value-prop is hard to justify for a serverless stack (mostly Cloud Run, but AWS Lambda too probably), and in situations where your services are mostly all in the same language and you can bake the features into libraries that are much easier to import. Observability is a great example of this. In server…

> I personally find that the service mesh value-prop is hard to justify for a serverless stack (mostly Cloud Run, but AWS Lambda too probably), and in situations where your services are mostly all in the same language and you can bake the features into libraries that are much easier to import.

If you’re running server less you already have 90% of what you’d get from a service mesh.

I will tell you that having seen what happens in big companies, baking distributed concerns into libraries always ends in disaster long after you’re gone.

When you have a piece of code deployed in 200 separate apps, every change requires tons of project management.

Re: Service mesh use cases (2020)

#42
Service meshes make it easier to roll out advanced load management/reliability features such as prioritized load shedding, which would otherwise need to be implemented within each language/framework.

For instance, Aperture[0] open-source flow control system is built on service meshes.

[0]: https://github.com/fluxninja/aperture

[1]: https://docs.fluxninja.com

Re: Service mesh use cases (2020)

#43

Earlier quoted context omitted.

It's a "big company" thing. In my opinion, the best way to add mTLS to your stack is to just adjust your application code to verify the certificate on the other end of the connection. But if the "dev team" has the mandate "add features X, Y, and Z", and the "devops team" has the mandate "implement mTLS by the end of Q1", you can see why "bolt on a bunch of sidecars" becomes the selected solution. The two teams don't…

I very much disagree the sentiment that adding mTLS is just “verifying the certificate on the other end of the connection”. You ignore the process of distribution and rotation of certificates which is non-trivial to implement application side.

Can’t each service just have a job that calls the Let’s Encrypt api once a day to get a new cert?

Re: Service mesh use cases (2020)

#44

Earlier quoted context omitted.

Right - by why not use something like an API gateway then?

API gateways are primarily used for HTTP traffic coming from clients external to your backend services eg. an iOS device (hence the term 'gateway' vs. 'mesh'). I don't think they support thrift or grpc (at least aws doesn't, not sure about other providers). https://aws.amazon.com/api-gateway/

Google cloud supports grpc on their api gateway: https://cloud.google.com/api-gateway/docs/grpc-overview

Re: Service mesh use cases (2020)

#45

Earlier quoted context omitted.

It's a "big company" thing. In my opinion, the best way to add mTLS to your stack is to just adjust your application code to verify the certificate on the other end of the connection. But if the "dev team" has the mandate "add features X, Y, and Z", and the "devops team" has the mandate "implement mTLS by the end of Q1", you can see why "bolt on a bunch of sidecars" becomes the selected solution. The two teams don't…

I very much disagree the sentiment that adding mTLS is just “verifying the certificate on the other end of the connection”. You ignore the process of distribution and rotation of certificates which is non-trivial to implement application side.

I honestly thought about covering a few ideas in the post, but decided it was off topic. The service meshes do include some rudimentary key generation and distribution code, which is nice to not have to build yourself. The simplest thing, if you're deployed in k8s or similar, is cert-manager + a CA + code that reloads keys when the secret is updated (pretty easy to write). This has downsides (good luck when your CA expires!) but it is easy and does keep itself functional. Cloud providers also have a service like this, which protects the root key with their own IAM (and presumably dedicated hardware); it's definitely a route you'll want to look into.

What's missing are a bunch of things you probably want to check before issuing keys; was the release approved, was all the code reviewed before release, is the code reading the foo-service key actually foo-service? That involves some input from your orchestration layer; i.e. an admission controller that checks all these things against your policies, and only then injects a key that the application can read. (Picking up rotated keys becomes more difficult, but this might be a good thing. "If you don't re-deploy your code for 90 days, it stops being able to talk to other services" doesn't seem like the worst policy I can think of in a world where Dependabot opens up 8 PRs a day against your project.)

This all has the downside that it doesn't really prevent untrusted applications from ruining the security; a dump_keys endpoint that prints the secret key to a log, nefarious code checked into source control but approved (perhaps due to a compromised developer workstation), etc. Fixing those problems is well outside the scope of a service mesh, but something you have to have a plan for. CircleCI didn't! Now you read 3 blog posts a day about how they got hacked.

Anyway, not sure where I was going with this, but application teams need to consider their threat model and protect against it. Security isn't a checkbox that can be checked by someone that didn't write the code. Sure, you can get all sorts of certifications this way that look nice on your marketing page, but the certifications really only cover "did they do the bare minimum to look kind of competent if it was 10 years ago". If you have sophisticated adversaries, you're going to need a sophisticated security team.

Re: Service mesh use cases (2020)

#46
post #17

Earlier quoted context omitted.

>slightly easier As a company grows sooner or later most of these features become pretty desirable from an operations perspective. Feature developers likely don't and shouldn't need to care. It probably starts with things like Auth and basic load balancing. As the company grows to dozens of teams and services then you'll start feeling pain around service discovery and wish you didn't need to implement yet another cus…

Some of the functionality you mentioned above is possible without a service mesh.

All of the functionality of kubernetes can be implemented independently. It’s still a useful set of abstractions, therefore/because it’s understood by a large portion of the industry.

Re: Service mesh use cases (2020)

#47
post #30

Earlier quoted context omitted.

Right - by why not use something like an API gateway then?

That can work, but it means you simply outsourced the problem to AWS. It's not a bad idea per se, but it means your service needs to talk, in some way, http. You could use the service mesh thing from AWS, along with cognito jwts, for authenticatetion and authorization

You can easily self host your own proxy. I bet API gateway is just Nginx, Traefik or HAProxy under the hood anyway.

Re: Service mesh use cases (2020)

#48
post #8

I’ve only read about Service Mesh, my impression was that it seems to add an awful lot of processes and complexity just to make developer’s lives slightly easier. Maybe I’m wrong but it almost feels like busy work for DevOps. Is my first impression wrong? Is this the right way to architect systems in some use cases, and if so what are they?

Many of the use cases described in the post are solved by service meshes. So, in my opinion, the questions are introspective: - “Do I have enough context to know what problem those solutions are solving, and to at least appreciate the problem space to understand why someone may solve it like this?” - “Do I have or perceive those problem to impact my infrastructure/applications?” - “Does the solution offered by the us…

I like how you frame the questions. How many times people pick a technology without answering them? Even having some knowledge in them.

I am wondering does Nomad/Consul continue to scale after some level?

Re: Service mesh use cases (2020)

#49
post #48
post #8

Earlier quoted context omitted.

Many of the use cases described in the post are solved by service meshes. So, in my opinion, the questions are introspective: - “Do I have enough context to know what problem those solutions are solving, and to at least appreciate the problem space to understand why someone may solve it like this?” - “Do I have or perceive those problem to impact my infrastructure/applications?” - “Does the solution offered by the us…

I like how you frame the questions. How many times people pick a technology without answering them? Even having some knowledge in them. I am wondering does Nomad/Consul continue to scale after some level?

I don't know about Consul, but Nomad has been scaled to 2,000,000 containers on >6000 hosts

https://www.hashicorp.com/c2m

Post reply on HN