Look into envoy/istio - e.g. these introduce side-processes (sidecars) where your process talks to, and these create the traces for you at some perf cost. There are proxies for some of the existing services - like mysql -
https://www.envoyproxy.io/docs/envoy/latest/configuration/li...I haven't used it myself, I've used census, and now looking into OpenTelemetry (though from the least finished version - C++). Had mixed success with it in the past, but trying again. Also not looking at all into side-cars, etc. - We compile all our internal tools, so adding this inside is where I'm getting into.
I've had several times (while at Google), being asked by an SRE that I would call on issues, and they would request to bump the sample tracing from minimal defaults (was it 1 in 100,000 or million - forget) sometimes to 1:1 - for say 30 seconds. This way they'll receive on their end (in their systems that we use) flags to sample too, and at the end get full logs.
Usually the whole trace is visible in few minutes. There were few UI's (nothign like zipkin/jaeged/others outside) - some of them with very "imgui"-like hacky (in good sense) view - like programmer art all over (which I loved - it was much more condensed than standard zipkin/jaeger).
You could've marked something as important, and it'll retain for longer period - otherwise - poof - soon gone. Also it would collect info sometimes directly from the machine it was in (rather than wait to populate).
Obviously, I don't know the details - I was just an user, or more like - allowing (when oncall) trace sampling to be bumped by the SRE - so they would get more info. It's what hooked me actually, because how else would one get everything end-to-end.
Surprisingly it's also useful for single apps, where you have threads (or concurrency tasks, like with TBB/ConCRT) doing nested parallel_for's or spawning jobs, and you want to get idea what's going on. The only tricky bit is how to get your "context" propagated from one thread to another (also not readily done).
It's one thing that the "golan" got right with their context for example.
So it's really awesome, but probably really hard to get right the first few times.