Does this run in production or is it used for testing? We wrote a distributed testing system ( https://github.com/gundb/panic-server ), so I'm trying to understand if integrating Jaeger would be helpful. If Jaeger is run on a production stack, I'd be curious to understand how that works (are there any tech talks on it yet?). If it is designed to run for tests, that makes sense, but then does it depend upon another distributed testing tool? If so, I'd love to see that tool. Glad to see it be Open Sourced!!!
Jaeger – A Distributed Tracing System
11–20 of 66 posts
Re: Jaeger – A Distributed Tracing System
#12Does this run in production or is it used for testing? We wrote a distributed testing system ( https://github.com/gundb/panic-server ), so I'm trying to understand if integrating Jaeger would be helpful. If Jaeger is run on a production stack, I'd be curious to understand how that works (are there any tech talks on it yet?). If it is designed to run for tests, that makes sense, but then does it depend upon another di…
It's a production tracing tool, but you could run it in a testing environment too.
Re: Jaeger – A Distributed Tracing System
#13Neat! I was looking into tracing solutions for our k8s cluster the other day and was going to look into setting up Zipkin. Now I'll this to my list of tools to evaluate. I found this blog post by uber informative https://eng.uber.com/distributed-tracing/ so maybe there is no need to even setting up Zipkin and just start with Jaegar?
Most folks will choose either Zipkin or Jaeger, but both are OpenTracing-compatible distributed tracing systems. You might find the Cloud Native Landscape useful for thinking about the options: https://github.com/cncf/landscape/blob/master/README.md
Disclosure: I’m the executive director of CNCF, which just adopted Jaeger 2 weeks ago, and I’m an author of the landscape.
Re: Jaeger – A Distributed Tracing System
#14I'm disappointed that all of the open-source tracing systems have adopted the Dapper [0] model. It's understandable why: it's extremely simple to implement, as it handles scaling challenges by doing client-side sampling.
A bit of background about how Dapper-style distributed tracing works. Things typically start with an RPC call of some kind (typically from an external source like a public load balancer). At that point, you must decide whether to trace this request or not, which is typically done as a random sample (say, 1% of requests). At that point, the request gets assigned a _trace id_, a random identifier for that request.
The trace id is stored in some request context and propagated to each subsequent service. Each service, meanwhile, divides up its request processing flow into a series of "spans" which represent some piece of computation. For example, a span cover an RPC call or a DB query. Spans are identified by a random _span id_. Once a request has been sampled, all spans for that request are sent to a central span collector where they're stored for later querying.
This model is simple but very limited. It's often hard to know whether a trace is interesting at the outset, hence the reliance on random sampling. For example, you might want to understand why your 99p latency is high, but if you're just sampling 1% the 99p requests will only be 0.01% of your sample.
More generally, interesting events (like errors or slow requests) tend to be rare, and sampling a random, small percent of requests is unlikely to turn up the interesting cases.
A better model, as implemented by lightstep [1] (and an in-progress distributed tracer I've been working on) is to collect all spans. Even with very high request volume it's reasonable to store all spans for at least a few minutes. Doing so opens up all sorts of interesting possibilities, because you can start tracing a request at any point during that window. For example, you might want to trace all requests that have errors in them. Or all requests that take longer than a certain time. Or get a google sample of requests across different latency buckets. Or requests that violate some application invariants you've defined.
Ultimately, though, distributed tracing is so helpful for understanding complex distributed systems and webs of microservices, and it's exciting to see more open-source competition for zipkin.
[0] Dapper is Google's distributed tracing system. The paper (https://research.google.com/pubs/pub36356.html) kicked off a lot of interest in distributed tracing in the broader community. [1] http://lightstep.com/
Re: Jaeger – A Distributed Tracing System
#15Is there a pros/cons comparison to Zipkin? Browsed the docs and couldn't find anything.
Re: Jaeger – A Distributed Tracing System
#16What's the performance hit for using Jaeger in production? Are there benchmarks?
Re: Jaeger – A Distributed Tracing System
#17I'm disappointed that all of the open-source tracing systems have adopted the Dapper [0] model. It's understandable why: it's extremely simple to implement, as it handles scaling challenges by doing client-side sampling. A bit of background about how Dapper-style distributed tracing works. Things typically start with an RPC call of some kind (typically from an external source like a public load balancer). At that poi…
That sounds pretty expensive. With dapper the trace span annotations do nothing if the request isn’t being traced. If it is being traced you might have significant costs, along the lines of sprintf(“%.03f”, ...) or other very cpu-intensive activity. This is OK when you trace one per million but when you trace everything you have to think about the cost. This could lead to either just using more CPU than you really wanted, or to discourage trace annotations. Either would be bad.
Re: Jaeger – A Distributed Tracing System
#18I'm disappointed that all of the open-source tracing systems have adopted the Dapper [0] model. It's understandable why: it's extremely simple to implement, as it handles scaling challenges by doing client-side sampling. A bit of background about how Dapper-style distributed tracing works. Things typically start with an RPC call of some kind (typically from an external source like a public load balancer). At that poi…
LightStep sounds interesting, but it seems to be closed source?
That being the case, it's not hard to see why people are going with the (existing) OSS solutions. :/
UPDATE - Found this on GitHub, is this the whole thing?
https://github.com/lightstep/lightstep-tracer-go
If so, pointing people towards it from the .com website might help get people trying it out, as the .com website makes it seem non-OSS. :)
Re: Jaeger – A Distributed Tracing System
#19Neat! I was looking into tracing solutions for our k8s cluster the other day and was going to look into setting up Zipkin. Now I'll this to my list of tools to evaluate. I found this blog post by uber informative https://eng.uber.com/distributed-tracing/ so maybe there is no need to even setting up Zipkin and just start with Jaegar?
Most folks will choose either Zipkin or Jaeger, but both are OpenTracing-compatible distributed tracing systems. You might find the Cloud Native Landscape useful for thinking about the options: https://github.com/cncf/landscape/blob/master/README.md Disclosure: I’m the executive director of CNCF, which just adopted Jaeger 2 weeks ago, and I’m an author of the landscape.
I have used that landscape document as a very informative point of reference for the last couple of months. Thank you for creating it.
Re: Jaeger – A Distributed Tracing System
#20I'm disappointed that all of the open-source tracing systems have adopted the Dapper [0] model. It's understandable why: it's extremely simple to implement, as it handles scaling challenges by doing client-side sampling. A bit of background about how Dapper-style distributed tracing works. Things typically start with an RPC call of some kind (typically from an external source like a public load balancer). At that poi…
That sounds pretty expensive. With dapper the trace span annotations do nothing if the request isn’t being traced. If it is being traced you might have significant costs, along the lines of sprintf(“%.03f”, ...) or other very cpu-intensive activity. This is OK when you trace one per million but when you trace everything you have to think about the cost. This could lead to either just using more CPU than you really wa…
Seems like you could store the raw arguments to printf and then only process them later if you decide you want the trace?