I had the impression, logs and metrics are a pre-observability thing.
I've never heard the term "pre-observability", what does that mean?
Migrating to OpenTelemetry
11–20 of 76 posts
Re: Migrating to OpenTelemetry
#12Earlier quoted context omitted.
Author here, thanks for the question! The current split developed from the personal preferences of the engineers who initially set up our observability systems, based on what they had used (and liked) at previous jobs. We're definitely open to doing more consolidation in the future, especially if we can save money by doing that, but from a usability standpoint we've been pretty happy with Honeycomb for traces and Dat…
> from the personal preferences of the engineers https://www.honeycomb.io/pricing https://www.datadoghq.com/pricing/ Am I wrong to say... having 2 is "expensive"? Maybe not if 50% of your stuff is going to Honeycomb and 50% going to DataDog. Could you save money/complexity (less places to look for things) having just DataDog or just Honeycomb?
Right now, there isn't much duplication of what we're sending to each vendor, so I don't think we'd save a ton by consolidating, at least based on list prices. We could maybe negotiate better prices based on higher volumes, but I'm not sure if Airplane is spending enough at this point to get massive discounts there.
Another potential benefit would definitely be reduced complexity and better integration for the engineering team. So, for instance, you could look at a log and then more easily navigate to the UI for the associated trace. Currently, we do this by putting Honeycomb URLs in our Datadog log events, which works but isn't quite as seamless. But, given that our team is pretty small at this point and that we're not spending a ton of our time on performance optimizations, we don't feel an urgent need to consolidate (yet).
Re: Migrating to OpenTelemetry
#13I made this switch very recently. For our Java apps it was as simple as loading the otel agent in place of the Datadog SDK, basically "-javaagent:/opt/otel/opentelemetry-javaagent.jar" in our args. The collector (which processes and ships metrics) can be installed in K8S through Helm or an operator, and we just added a variable to our charts so the agent can be pointed at the collector. The collector speaks OTLP whic…
We tried this about a year and a half ago and ended up going somewhat backwards into DD entrenchment, because they've decided that anything not an official DD metric (that is, collected by their agent typically) is custom and then becomes substantially more expensive. We wanted a nice migration path from any vendor to any other vendor but they have a fairly effective strategy for making gradual migrations more expens…
This has been a concern for me too. But the agent is just a statsd receiver with some extra magic, so this seems like a thing that could be solved with the collector sending traffic to an agent rather than the HTTP APIs?
I looked at the OTel DD stuff and did not see any support for this, fwiw, maybe it doesn't work b/c the agent expects more context from the pod (e.g. app and label?)
Re: Migrating to OpenTelemetry
#14Re: Migrating to OpenTelemetry
#15Interesting read. What did you find easier about using GCP's log tooling for your internal system logs, rather than the OTel collector?
Re: Migrating to OpenTelemetry
#16> Moreover, we encountered some rough edges in the metrics-related functionality of the Go SDK referenced above. Ultimately, we had to write a conversion layer on top of the OTel metrics API that allowed for simple, Prometheus-like counters, gauges, and histograms.
Have encountered this a lot from teams attempting to use the metrics SDK.
Are you open to comment on specifics here and also what kind of shim you had to put in front of the SDK? It would be great to continue to retrieve feedback so that we can as a community have a good idea of what remains before it's possible to use the SDK for real world production use cases in anger. Just wiring up the setup in your app used to be fairly painful but that has gotten somewhat better over the last 12-24 months, I'd love to also hear what is currently causing compatibility issues w/ the metric types themselves using the SDK which requires a shim and what the shim is doing to achieve compatibility.
Re: Migrating to OpenTelemetry
#17I made this switch very recently. For our Java apps it was as simple as loading the otel agent in place of the Datadog SDK, basically "-javaagent:/opt/otel/opentelemetry-javaagent.jar" in our args. The collector (which processes and ships metrics) can be installed in K8S through Helm or an operator, and we just added a variable to our charts so the agent can be pointed at the collector. The collector speaks OTLP whic…
We tried this about a year and a half ago and ended up going somewhat backwards into DD entrenchment, because they've decided that anything not an official DD metric (that is, collected by their agent typically) is custom and then becomes substantially more expensive. We wanted a nice migration path from any vendor to any other vendor but they have a fairly effective strategy for making gradual migrations more expens…
> somewhat backwards into DD entrenchment, because they've decided that anything not an official DD metric (that is, collected by their agent typically) is custom and then becomes substantially more expensive.
It a vendor pulled shit like this on me. That’s when I would counsel them. Of course most big orgs would rather not do the leg work to actually become portable, migrate off vendor. So of course they will just pay the bill.
Vendors love the custom shit they build because they know once it’s infiltrated the stack then it’s basically like gangrene (have to cut off the appendage to save the host)
Re: Migrating to OpenTelemetry
#18Earlier quoted context omitted.
We tried this about a year and a half ago and ended up going somewhat backwards into DD entrenchment, because they've decided that anything not an official DD metric (that is, collected by their agent typically) is custom and then becomes substantially more expensive. We wanted a nice migration path from any vendor to any other vendor but they have a fairly effective strategy for making gradual migrations more expens…
This has been a concern for me too. But the agent is just a statsd receiver with some extra magic, so this seems like a thing that could be solved with the collector sending traffic to an agent rather than the HTTP APIs? I looked at the OTel DD stuff and did not see any support for this, fwiw, maybe it doesn't work b/c the agent expects more context from the pod (e.g. app and label?)
Yeah, the DD agent and the otel-collector DD exporter actually use the same code paths for the most part. The relevant difference tends to be in metrics, where the official path involves the DD agent doing collection directly, for example, collecting redis metrics by giving the agent your redis database hostname and creds. It can then pack those into the specific shape that DD knows about and they get sent with the right name, values, etc so that DD calls them regular metrics.
If you instead went the more flexible route of using many of the de-facto standard prometheus exporters like the one for redis, or built-in prometheus metrics from something like istio, and forward those to your agent or configure your agent to poll those prometheus metrics, it won't do any reshaping (which I can see the arguments for, kinda, knowing a bit about their backend) and they just end up in the DD backend as custom metrics, and charge you at $0.10/mo per 100 time series. If you've used prometheus before for any realistic deployments with enrichment etc, you can probably see this gets expensive ridiculously fast.
What I wish they'd do instead is have some form of adapter from those de facto standards, so I can still collect metrics 99% my own way, in a portable fashion, and then add DD as my backend without ending up as custom everything, costing significantly more.
Re: Migrating to OpenTelemetry
#19> The data collected from these streams is sent to several vendors including Datadog (for application logs and metrics), Honeycomb (for traces), and Google Cloud Logging (for infrastructure logs).
It sounds like they were in a place that a lot of companies are in where they don't have a single pane of glass for observability. One of if not the main benefit I've gotten out of Datadog is having everything in Datadog so that it's all connected and I can easily jump from a trace to logs for instance.
One of the terrible mistakes I see companies make with this tooling is fragmenting like this. Everyone has their own personal preference for tool and ultimately the collective experience is significantly worse than the sum of its parts.
Re: Migrating to OpenTelemetry
#20Earlier quoted context omitted.
> from the personal preferences of the engineers https://www.honeycomb.io/pricing https://www.datadoghq.com/pricing/ Am I wrong to say... having 2 is "expensive"? Maybe not if 50% of your stuff is going to Honeycomb and 50% going to DataDog. Could you save money/complexity (less places to look for things) having just DataDog or just Honeycomb?
Right now, there isn't much duplication of what we're sending to each vendor, so I don't think we'd save a ton by consolidating, at least based on list prices. We could maybe negotiate better prices based on higher volumes, but I'm not sure if Airplane is spending enough at this point to get massive discounts there. Another potential benefit would definitely be reduced complexity and better integration for the engine…
When you say DataDog for everything else (as in not traces), besides logs, what else do you mean?