OTel isn’t going well
matduggan.com
OTel isn’t going well
1–10 of 123 posts
Re: OTel isn’t going well
#2(TFA author is not me.)
Re: OTel isn’t going well
#3I've never found instrumentation to be a huge issue. Sure it takes more effort but you get a lot more value once you understand _business_ events.
Re: OTel isn’t going well
#4The alternative is vendor lockin, $$$, and spotty support for complex environments with zero chance of ever getting 100% coverage.
At least with Open Telemetry, anyone can write an OTLP "source" using free, open specifications, and it'll "just work" with dozens of third-party "sinks". That's huge!
Sure, there's a lot of experimental tags on semantic conventions, but at the end of the day, that's not that critical. It's just data: most sinks don't "interpret" these tags, they just display them as-is, so changes aren't breaking changes.
Re: OTel isn’t going well
#5The alternative is vendor lockin, $$$, and spotty support for complex environments with zero chance of ever getting 100% coverage. At least with Open Telemetry, anyone can write an OTLP "source" using free, open specifications, and it'll "just work" with dozens of third-party "sinks". That's huge! Sure, there's a lot of experimental tags on semantic conventions, but at the end of the day, that's not that critical. It…
The alternative is Prometheus (which is freaking great) and Jaegar (which is freaking great), each alone. This is better, because Otel is trying to put two distinct things (monitoring and metrics, distributed tracing) into one package, because they know how to use neither.
Neither Prometheus metrics nor Jaeger traces are magic bullets. Neither of them are complicated, either, and in fact the fact that they're not complicated is their greatest strength. You can and should understand every facet of what they entail. You should build the (very small) shims that they need for your company's framework every time. It's not hard. It's not hard because it's not complicated. The fact that it's not complicated seems to break people's brains. They are accurate because they're simple and they're easy to work with because they're simple, and OTel is neither.
Re: OTel isn’t going well
#6The alternative is vendor lockin, $$$, and spotty support for complex environments with zero chance of ever getting 100% coverage. At least with Open Telemetry, anyone can write an OTLP "source" using free, open specifications, and it'll "just work" with dozens of third-party "sinks". That's huge! Sure, there's a lot of experimental tags on semantic conventions, but at the end of the day, that's not that critical. It…
The alternative is Prometheus (which is freaking great) and Jaegar (which is freaking great), each alone . This is better, because Otel is trying to put two distinct things (monitoring and metrics, distributed tracing) into one package, because they know how to use neither. Neither Prometheus metrics nor Jaeger traces are magic bullets. Neither of them are complicated , either, and in fact the fact that they're not c…
I’ve built custom Prometheus metrics very easily and had node exporter pick up the .prom files. Python and bash scripts reading and translating.
Node exporter runs on my Prometheus server next to Blackbox Exporter. Blackbox Exporter handles TLS expiry metrics.
Re: OTel isn’t going well
#7OTel is so frustrating. If it wasn't shaping to be the clear winner in the space, I wouldn't complain about it as much. But today:
1. Every major vendor is still in some weird alpha/beta support for OTel even after all this time.
2. The performance hit is substantial and makes you question what the point of performance instrumentation is if you need twice as much compute/RAM to run the same workload now.
3. Serverless runtimes pay a heavy penalty for cold starts with OTel.
4. You're basically forced to run both gateway collectors and edge collectors for any realistic usage.
5. You still need to configure destination exporters in unique ways. This leaves you questioning what the value of OTel was.
6. Vendors that go beyond the scope of what OTel covers still need their own bespoke instrumentation. What was the point of any of this then?
Re: OTel isn’t going well
#8I wish otel was never there. It is badly designed abstraction and due to otel the code gets very very messy and bad.
Re: OTel isn’t going well
#9What always puzzles me about OpenTelemetry is that tracing, metrics and logs are all designed independently. I wish there was a way I could just annotate my code base once, and let the ultimate decision to expose something as a metric/log/trace be dynamic at runtime.
For example, if I look at a graph in monitoring dashboard and see something suspicious, I’d like to say: “The next time something like this occurs again, please save me a trace.” I should be able to just do that with a single mouse click.
I remember them releasing the tracing spec/SDKs and saying “now let’s move on to metrics/logs.” That never sat right with me.
Re: OTel isn’t going well
#10What always puzzles me about OpenTelemetry is that tracing, metrics and logs are all designed independently. I wish there was a way I could just annotate my code base once, and let the ultimate decision to expose something as a metric/log/trace be dynamic at runtime. For example, if I look at a graph in monitoring dashboard and see something suspicious, I’d like to say: “The next time something like this occurs again…
One strategy do to do that is to trace everything by default and select what to sample later, e.g. https://grafana.com/docs/grafana-cloud/observe-and-act/adapt...