11–20 of 33 posts
n00b question, I always see service meshes used in the context of containers and mostly with kube. Would they work with more monolithic/traditional n-tier architecture deployed directly on host OS as well? Or maybe put another way, are there likely to be pain points that don't exist in containerized architectures?
Earlier quoted context omitted.
There is magic, (caveat: I work here), but Instana ( https://instana.com ) will instrument most major languages and frameworks auto-magically. As in, I don't have to declare a dependency, change a configuration, or anything - our agent finds processes running on the system and bootstraps the libraries while running and monkey patches a huge number of standard libraries and frameworks with no restarts. (don't believe…
That sounds horrific for whoever is going to be supporting that system... The last thing I would want in a production environment is to have some 3rd party software monkey-patching the code at runtime. What happens when: - a bug only occurs (due to timing or some other extremely subtle issue) when this monkey-patching is applied. - there's a bug in the monkey-patching itself (sounds like a fun debugging session!) - a…
It's OK that you're not, but I hope you can agree that engineering observability isn't cheap nor easy - and if you're using standard libraries, frameworks, and tooling (and not going way off the rails) we have observed that, for the most part, our agent works as intended.
We always recommend our customers run the agent in their test and integration environments, but you are correct, there are always risks involved. Other than the automation how is this any different then putting a New Relic jar into your Java app, or including a Datadog library? We simply figured out how to do it automatically at runtime.
Zero instrumentation visibility with the service mesh, but the demo app is instrumented. I’ve seen this point being sold everywhere for service mesh, but the vanilla tracing data given by Istio or others is not that useful by itself. There is no magic, you need to instrument your code
E.g. Linkerd gives you service "golden metrics" (success rate, latency distribution, request volumes) without any app changes. It can draw the service topology too, since it's observing everything in realtime. https://linkerd.io/2/features/telemetry/
Earlier quoted context omitted.
There is magic, (caveat: I work here), but Instana ( https://instana.com ) will instrument most major languages and frameworks auto-magically. As in, I don't have to declare a dependency, change a configuration, or anything - our agent finds processes running on the system and bootstraps the libraries while running and monkey patches a huge number of standard libraries and frameworks with no restarts. (don't believe…
That sounds horrific for whoever is going to be supporting that system... The last thing I would want in a production environment is to have some 3rd party software monkey-patching the code at runtime. What happens when: - a bug only occurs (due to timing or some other extremely subtle issue) when this monkey-patching is applied. - there's a bug in the monkey-patching itself (sounds like a fun debugging session!) - a…
Monkey-patching third party software will totally void the warranty on it. I've been involved in cases like this before and if there's any kind of weird bug that's conceivably related to the monkey-patching, it's hard to get help until you disable it.
n00b question, I always see service meshes used in the context of containers and mostly with kube. Would they work with more monolithic/traditional n-tier architecture deployed directly on host OS as well? Or maybe put another way, are there likely to be pain points that don't exist in containerized architectures?
Meshes are a lot more than just sidecar proxying -- they are what make sidecar proxying manageable, and they add a lot of other features like authentication, network policies, various other traffic control policies, service discovery, etc. They are an attempt to do for service-to-service communication what Kubernetes has done for container deployment -- make it abstract and declarative, with configurations that are independent from the underlying implementation.
The underlying implementation that works right now is the Kubernetes API and etcd, and alternate implementations need to be provided for those features to work well outside of Kubernetes. I think it will happen sometime in the next few years.
Can we please stop the buzzword train?
Earlier quoted context omitted.
That sounds horrific for whoever is going to be supporting that system... The last thing I would want in a production environment is to have some 3rd party software monkey-patching the code at runtime. What happens when: - a bug only occurs (due to timing or some other extremely subtle issue) when this monkey-patching is applied. - there's a bug in the monkey-patching itself (sounds like a fun debugging session!) - a…
We have hundreds of customers (and thousands of engineers) who are willing to make the trade-off. It's OK that you're not, but I hope you can agree that engineering observability isn't cheap nor easy - and if you're using standard libraries, frameworks, and tooling (and not going way off the rails) we have observed that, for the most part, our agent works as intended. We always recommend our customers run the agent i…
Testing with the agent would certainly help, but then you lose some of the "ease of use" benefits as I expect you would have to run a mini cluster in CI in order to run your agent?
There are few important difference between this and a "normal" dependency:
- Even if the application is fully tested with your agent, it could be something as simple as turning your agent off that could break things.
Hypothetical scenario: multiple instances of the application are running with your agent enabled. Someone decides to turn off monitoring for some reason - nothing bad happens and they go home at the end of the day. Later on, some instances are restarted, or the cluster is re-scaled. Now you have half your cluster on a different code-base and your serialisation breaks because you were doing something silly like using pickle or a java object stream.
- The examples I mentioned in my previous comment would not happen with a normal dependency, because the version of that dependency would already be managed through standard means. If I were to go an look at the code, I would be able to see the actual code that is running, and the exact versions of all dependencies used.
I am still praying that some day soon AWS will announce that they are joining Opencensus (along with Google, MS, Datadog, Prometheus)[1] in the hopes that we can move towards standard tooling for observability. They also seriously need to give CloudWatch a UI/UX overhaul. 1. https://opencensus.io/introduction/#partners-contributors
OpenCensus seems like it's really a Google-only project. OpenTracing and OpenMetrics appear to have more community and vendor engagement. E.G. Datadog is basing their newer tracing libraries on OpenTracing, and Prometheus devs are behind OpenMetrics.
OpenTracing and OpenMetrics are more like API specs with libraries left to others to implement, and they're never really used standalone for them to be separate projects. The best option for the industry would be to fold OT and OM into OC and make a single stack, and hopefully include structured logging as well.
Earlier quoted context omitted.
There is magic, (caveat: I work here), but Instana ( https://instana.com ) will instrument most major languages and frameworks auto-magically. As in, I don't have to declare a dependency, change a configuration, or anything - our agent finds processes running on the system and bootstraps the libraries while running and monkey patches a huge number of standard libraries and frameworks with no restarts. (don't believe…
That sounds horrific for whoever is going to be supporting that system... The last thing I would want in a production environment is to have some 3rd party software monkey-patching the code at runtime. What happens when: - a bug only occurs (due to timing or some other extremely subtle issue) when this monkey-patching is applied. - there's a bug in the monkey-patching itself (sounds like a fun debugging session!) - a…
At a certain level of scale you're running code you didn't write, anyway (some mix of open-source code and code from previous team members) and having exact source with exciting surprises you've never seen before isn't going to save you from getting woken up at 4 AM. Though it might make it easier to fix the problem.
I would interested in anyones feedback of embracing and rolling out a service mesh/Istio in a non-GCP environment. I apologize if this is a naive question but how come this wasn't included as part of the Kubernetes project given that it has the same Google origins?