Leaving Mozilla

I've used it, and I've been using rr. gdb's reverse debugging was useful to me a few times, in extremely limited situations.

I often do my initial runs under rr, just in case I might want to debug, because it's useful most of the time and it's not enough overhead to worry about.

gdb's reverse debugging works after you narrow the range of interest down to a very small region, because otherwise its buffer will repeatedly overflow and it'll take ages.

I'd push back on the 'complicated' part. The dtrace language is heavily based on awk, which every modern *NIX ships with, and that every software engineer and sysadmin have at least a passing familiarity with. The main difference between the two is whereas awk is meant to operate over a stream of semi-structured lines, dtrace operates over a stream of probe firings. The high-level structures of the scripts, though, are the same:

    /pattern/ { statements }

In a awk, when a line matches a the patter, the corresponding statements fire. In dtrace, when a probe matches a pattern, the corresponding statements fire. Integers, strings, arrays, and all those other sorts of features of the language are the same (though there are some special functions introduced by dtrace to help you collect and visualize aggregations).

So from a language perspective, if you know awk, you mostly know dtrace. All that's left to do is learn the pattern language. If you want to do that effectively, you have to know not just the syntax of the patterns, but useful probes to listen for, and useful ways to use the information that's gleaned.

This involves knowing things about operating systems. Most software engineers don't know much about operating systems so they think the tool is stupid.

I think you've accidentally set up a straw man because mentions of low level tools don't correlate to use. In fact "trending" popularity is how a lot of bad technical decisions are made, cue crappy database flavor of the month. The hammer is a pretty useful tool but builders don't blog du jour about how elegant combining the strike face and nail puller on one implement are. None the less, millions are used every day.

DTrace is used extensively in FreeBSD development these days.

It's being used to bring up new generations of Systems Programmers: http://teachbsd.org/

My team uses DTrace to develop TCP, which ensures open standards and a copyfree implementation that ends up in Apple products, Juniper, NetApp, and who knows where else. Of course, also Netflix and the large CDN I work for which run at least 1/3 of North American Internet volume.

We also use DTrace to fix the reference copyfree Intel drivers, which may in a small way pay back Illumos for the awesome tech they share. I think those end up in some popular embedded microkernels too.

I grin thinking about people that emphasize "GNU/Linux", but I will empathize with FSF for a moment to show that low level software is not particularly exciting to most. GCC and GDB are hugely influential tools for the whole open source ecosystem, especially the high growth during the 80s and 90s that led to "Enterprise" acceptance. It's hard to imagine where everyone else would be, including the BSDs and many chip companies, were it not for the GNU toolchain. Nonetheless, most people just begrudgingly use these and hope they don't give them any trouble because of course they are usually trying to do something else.

> Let me put it this way. In the last say five years, I can't think of a single article I've come across where someone describes DTrace making their life better (except one I wrote in 2011: http://blog.reverberate.org/2011/05/using-dtrace-on-os-x-to-...), or a single time anyone I know or work with has even mentioned DTrace.

Then you haven't been looking:

http://dtrace.org/blogs/brendan/2012/08/09/10-performance-wi...

http://dtrace.org/blogs/brendan/2014/02/11/another-10-perfor...

https://www.joyent.com/blog/node-js-in-production-runtime-lo...

https://twitter.com/b0rk/status/681536144362848257

https://speakerdeck.com/sartak/dtrace-war-stories

https://www.bignerdranch.com/blog/hooked-on-dtrace-part-1/

https://medium.com/@jlouis666/erlang-dirty-scheduler-overhea...

Seriously, it wasn't that hard to find this stuff. I found the last one by searching HN: https://hn.algolia.com/?q=dtrace

Like others who replied to you, I use DTrace every day. Very often, it solves in seconds what would have taken hours without it. Frequently, it's helped solve problems I'd never have been able to understand without it. My colleagues use it all the time as well. Sometimes, I write about the general techniques (I wrote the "runtime log snooping" blog post above), and most of our bug reports in recent years are public, but we don't write about individual bugs all that often. (Seems like you wouldn't have run into them if we had, though.)

I don't blog, and therefore have never written about the multiple times DTrace helped me and saved me a lot of time.

It also made my job fun again as a sysadmin.

Here are couple of examples I remember, from my days at a large financial institution.

We're planning a samba upgrade on a server with ~700 clients. Non regression tests show the performance of copying large files has degraded by 30% Dtrace, aggregating on smbd stacks, helped us figure out the root cause in half an hour. This was reported and eventually properly patched upstream.

We started to dtrace most large batch jobs or apps, and found lots of easy performance fixes. In house apps launching thousands of 'grep", "cut", etc.. with a LD_LIBRARY_PATH set to search over 50 directories OVER NFS. haha, libc's not there, not there.. ! ohhh there it is.

Identifying stupid behavior in vendor apps for which we had no source. (but could complain about and sometimes get a fix)

NIS groups (yeah Im old) slowing down NFS a lot, etc well, I guess I used it with profit at least once a week.

You could probably do some of what DTrace does with other tools, but: it was safe in production, it had little overhead, and it was fun :-)

Ah! I also used it to teach our interns a little about the OS internals. fbt probes with flowindent, watch them go "ahhhhhh that's what happens!".

My current job is all about linux and I wish everyday for something half as good. So yeah, "it made my life better" is an understatement.