91–100 of 107 posts
We switched to OpenTelemetry agent from DataDog agent. Out of all commercial providers, Datadog is the worst one to show traces properly. Although it captures mostly the same traces, all of them have io.opentelemetry.something on UI making it very hard to read. I’ve tried Honeycomb, NewRelic, Lightstep.
Earlier quoted context omitted.
Not sure I entirely follow what's going on here. Is there some context behind this that's useful to know?
OpenTelemetry basically allows for vendor neutral instrumentation for your metrics, logs, and traces. It makes it easy to switch to a cheaper or better service with minimal lock-in, which is obviously bad for DataDog and other companies business model
Meh. The best way to keep somebody on your product is to make it easy for them to get off your product.
Earlier quoted context omitted.
When we switched off of AppSignal, since our instrumentation as deeply intertwined with our code, it took a while to change that over to Otel. But at least now, being Otel, theoretically if we want to change it to a different vendor, we can. If we were to change off of DD, and there is an otel collector that can accept DD APM spans, then we can switch now, refactor later. In practice, DD has a lot going for it that I…
Why sample otel spans and miss out on the important ones?
Even if we were self-hosting, there's a cost to ingesting and storing every single span.
And even if we are able to pay for ingesting 100%, not everything is practical to be ingested 100%. Our most common request type (heartbeat) generate a span payload size that is a multiple of the original request. We're using Elixir in production, and those can absorb a tremendous amount of traffic, saturating the entire CPU capacity of the hardware if we let it. The agents are not capable of keeping up.
Earlier quoted context omitted.
I have seen it around, but didn't know it was open source. If you guys are working on correlation, then I will definitely be watching this. Consistent link urls that can be shared is also useful. Hopefully someone else will contribute the notebooks feature. Those are very useful. Something that DD is not careful about, is being able to consistently use UTC for all time labels in all graphs (and maybe a quick way to c…
Yeah, we are actively working on correlation. We have metrics, traces and logs in a single app - so we should be able to provide a seamless correlation. Thanks for the point about Notebooks, we have not thought in detail on how people use that. Is it primarily to collaborate between team members when an incident happens or even when there is no incident, and you are analysing stuff
- Incidents, collecting different metrics and showing them next to each other, with comments
- Longer-term reliability debugging. They can form a kind of ad-hoc dashboard. These are usually issues that degrade performance, don't have immediate or wide-spread customer impact, and are things we are not immediately able to detect
- Related, performance tuning. Sometimes, the key metric is unknown. We want to explore it, and then make changes to infra, and then see if that moved the needle
- Sometimes, the ad-hoc widgets are useful enough to export to a dashboard
- I can take any widget anywhere else and import it into a notebook, or start a new notebook out of it.
The notebooks are similar to the dashboard, just that, the layout engine only allows a linear notebook layout instead of a grid. There are already text widgets, though the button to add that is easier to access. Other than the comments, it's basically a dashboard with the UI changed so that it feels like a notebook.
Keep in mind too, all dashboard and notebooks modify timestamps and other states in the browser URL, so it is easy for me to copy-paste those into Slack so that other people can see what I am seeing.
Earlier quoted context omitted.
I have seen it around, but didn't know it was open source. If you guys are working on correlation, then I will definitely be watching this. Consistent link urls that can be shared is also useful. Hopefully someone else will contribute the notebooks feature. Those are very useful. Something that DD is not careful about, is being able to consistently use UTC for all time labels in all graphs (and maybe a quick way to c…
We use Datadog at the medium sized unicorn I work at. People often don’t understand how important correlating timestamps is. I wish Datadog was just opinionated as hell and said everything you see is UTC by default. Or even less dramatic than that, literally just slap the time zone on whatever it is that you’re viewing. It’s so insanely critical when you’re viewing logs on a prod outage that people immediately see an…
When we talk to stakeholders and customer-facing folks though, tend to convert it to local time.
Earlier quoted context omitted.
I won't speculate on the top-level strategies of companies competing with another, so take this as a grain of salt. From my perspective (maintainer, employed by a vendor), all of us who work for these different companies collaborate very well together. We all recognize that it's both technically tractable and fundamentally user-friendly to make instrumentation be a common standard that anyone can use to point at any…
Everything you've just said is compatible with what I've said. The vendors are concerned about being left out at the "source", and are seeking to differentiate with their proprietary "sink", usually closed-source SaaS solutions. I'm not even arguing that this is bad, it's just how markets work, and it's currently beneficial to developers in general, including both open-source developers and the type working in a cubi…
Datadog is stupidly expensive for what it provides. Don't recommend and I hope something open kills it soon.
More vendor lock-in shenanigans. Data dog has always been a proprietary POS. I don’t know why people use it, APM traces? How long before Grafana has these capabilities in OSS? So annoying seeing a company like DD who cannot innovate at all, trying to lock in the average company.
You see the same thing around integrations, everyone used to have to roll their own proprietary chunks of code that in the end were all querying mostly the same data points back from servers and API's. Now everyone just prefers to wait for the Prometheus exporter and they adopt that instead.