11–20 of 72 posts
One great use of agentic coding is being able to add and very granular tracing instrumentation to help with these sort of optimizations.
Also a great way to make sure that your app spends most of its time in observability overhead. For example even the latency histogram that the OP mentions is wildly expensive.
All of the significant server applications I have encountered in the industry have suffered from the same problem, which surprised their authors but seemed obvious to me: the application was spending the majority of its CPU time doing meta-work like entering and leaving epoll, stealing work from itself, etc. There are principles for writing Tokio servers and these are good points in the OP but I think they are little…
I can't say I'm surprised when I see the 100+ function stack traces that Axum built on Tokio produces. Before you say Axum is "holding it wrong" the project lives under the tokio-rs GitHub org.
Earlier quoted context omitted.
Also a great way to make sure that your app spends most of its time in observability overhead. For example even the latency histogram that the OP mentions is wildly expensive.
That just sounds like bad tracing implementations. A good tracing implementation should be able to drive gigabytes per second of trace logs to memory. If you are generating it slow enough to allow actual offload then you should be in the 1—10% range even if you are saturating your offload. You should, of course, upper bound this overhead by switching to a full time travel debugging solution, thus tracing everything,…
1: https://llvm.org/docs/XRay.html ... is there even a Rust analog to this?
Earlier quoted context omitted.
That just sounds like bad tracing implementations. A good tracing implementation should be able to drive gigabytes per second of trace logs to memory. If you are generating it slow enough to allow actual offload then you should be in the 1—10% range even if you are saturating your offload. You should, of course, upper bound this overhead by switching to a full time travel debugging solution, thus tracing everything,…
I'm just reporting from the trenches here. I think you are suggesting that everyone is aware of and capable of using state-of-the-art (from 20 years ago) tracing schemes like XRay[1], when in reality they are not. Most projects would be well-served by any basic profiler but even profiling is apparently for wizards, because I've seen a lot of projects that will resort to manually annotating functions with OTel trace s…
It's worth pointing out though that just tracing function calls isn't good enough for the kinds of stackless coroutines that run in async Rust tasks. You need a way of mapping between the async tasks and the compiler emitted traces.
afaik, C/C++ have the same problem.
Earlier quoted context omitted.
That just sounds like bad tracing implementations. A good tracing implementation should be able to drive gigabytes per second of trace logs to memory. If you are generating it slow enough to allow actual offload then you should be in the 1—10% range even if you are saturating your offload. You should, of course, upper bound this overhead by switching to a full time travel debugging solution, thus tracing everything,…
I'm just reporting from the trenches here. I think you are suggesting that everyone is aware of and capable of using state-of-the-art (from 20 years ago) tracing schemes like XRay[1], when in reality they are not. Most projects would be well-served by any basic profiler but even profiling is apparently for wizards, because I've seen a lot of projects that will resort to manually annotating functions with OTel trace s…
I am by no means an expert, but I've recently improved performance for some code and used tracy. They have rust bindings as well. It's pretty cool and it seems to be low overhead. Wonder if I can couple it with something like xray? Tracy is more the tracing library + tracing interpretations/aquisition tool.
Edit: apparently rust already supports xray natively on the nightly.
When you're at a point of tuning Tokio, consider taking a look at ef_vi/DPDK + SPDK