Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Everything Is Broken: Shipping Rust-Minidump at Mozilla

hacks.mozilla.org

Post reply on HN
Print view View original

21–30 of 67 posts

Re: Everything Is Broken: Shipping Rust-Minidump at Mozilla

#21

Post by js2 »

Thank you for this work!

I've been involved with minidumps in one way or another since around 2010. Was at a startup at the time that had a browser based on Chromium and we needed crash reporting for our own app. So I wrote a pretty simply backend that received minidumps, ran them through the breakpad processor and shoved the output into Splunk. That was our crash-reporting system.

Circa 2013 the company gets acquired by Yahoo which at the time was using Crittercism for its mobile apps but Yahoo wasn't happy with it. Somehow I was now the mobile app crash reporting expert at the company though so I built a whole new in-house crash reporting solution.

For iOS I wrote an SDK around PLCrashReporter because unwinding stacks on the client works out way better on iOS than dealing with a minidump.

For Android I had to deal with both JVM (er, Dalvik, er ART) stack traces, easy enough, but also native code crashes. For the latter I used breakpad's crash handler and minidumps. But it turns out that minidumps from Android devices are almost useless for two reasons:

1) If the crashes originate in managed code or calls into managed code you can't trace back through the managed code frames from a minidump. Especially if you don't have frame pointers.

2) You basically cannot get the symbols for all the different flavors of Android. Without symbols any stack trace that breakpad reconstructs is pretty useless.

Eventually I abandoned minidumps on Android and instead unwinding on the phone using corkscrew, wait no, libbacktrace, wait no, libunwind. But that still doesn't give useful stack traces very often. In the end, I ended up capturing logcat output when restarting after a crash which actually tends to have the most useful stack traces.

Which is all to say, both Apple and Google make it really hard for a mobile app to find out why it crashed. Both Android and iOS create a crash report for any app which crashes, but the app can't access those. So we're all shipping apps with third-party crash handlers built-in that try to capture a stack or minidump in-process and make sense of it later.

Re: Everything Is Broken: Shipping Rust-Minidump at Mozilla

#24

Post by _cbsz »

marcosdumay wrote: post #15
singhrac wrote: post #14

Earlier quoted context omitted.

> you can't assume anything is valid at the point you use the data. Just to double check my understanding: are you talking about raw pointers (i.e. void*) being common in C++ and not in Rust? You're right that I was using ADT a bit loosely; to be honest the main value add for me has been the first class data-holding enums/sum types. C++ has std::variant, but the syntax support in Rust feels nicer.

C++ has a series of issues. You can't trust pointers have a value, or that the value is valid, you can't trust that your enums have a value inside their interval, or in fact you can't trust that any value from any type is inside its interval at all. You also can't really trust that your values have the correct size. We choose some of those to ignore, otherwise we wouldn't be able to program at all, but C++ gives you…

> you can't trust that your enums have a value inside their interval

If you don't set the underlying type, assigning a value that doesn't match an enumerator via `static_cast` is undefined behavior. See https://en.cppreference.com/w/cpp/language/enum . (Doing weird pointer casting things is also undefined behavior per the strict aliasing rule, though, come to think of it, I'm not sure whether memcpying an out-of-range value into an enum through the "reinterpret_cast to `char*`" loophole is undefined behavior.)

Re: Everything Is Broken: Shipping Rust-Minidump at Mozilla

#25

Post by tclancy »

j3s wrote: post #20

What a fun read! :3 I really like your writing style. Deploying stuff to production is always so nerve-wracking, I related to that very hard. I recently developed a golang alternative to an old erlang-ruby-hodgepodge, and when it worked in production I found myself constantly not believing that nothing went wrong.

Ha, weeks and months of thinking, "Please just work" and then it does and it's always a shock.

Re: Everything Is Broken: Shipping Rust-Minidump at Mozilla

#26

Post by nemetroid »

_cbsz wrote: post #24
marcosdumay wrote: post #15

Earlier quoted context omitted.

C++ has a series of issues. You can't trust pointers have a value, or that the value is valid, you can't trust that your enums have a value inside their interval, or in fact you can't trust that any value from any type is inside its interval at all. You also can't really trust that your values have the correct size. We choose some of those to ignore, otherwise we wouldn't be able to program at all, but C++ gives you…

> you can't trust that your enums have a value inside their interval If you don't set the underlying type, assigning a value that doesn't match an enumerator via `static_cast` is undefined behavior. See https://en.cppreference.com/w/cpp/language/enum . (Doing weird pointer casting things is also undefined behavior per the strict aliasing rule, though, come to think of it, I'm not sure whether memcpying an out-of-rang…

I’m assuming you are referring to this part:

> If the underlying type is not fixed and the source value is out of range, the behavior is undefined.

Note the fine print about the meaning of ”out of range”:

> (The source value, as converted to the enumeration's underlying type if floating-point, is in range if it would fit in the smallest bit field large enough to hold all enumerators of the target enumeration.)

So this is not undefined:

  enum E { A = 0, B = 1, C = 2 };
  E valid = static_cast(3);

Re: Everything Is Broken: Shipping Rust-Minidump at Mozilla

#27

Post by Gankra »

ComputerGuru wrote: post #23

A better/more technical article on the same tech, from Mozilla's collaborators on this project: https://jake-shadle.github.io/crash-reporting/

That article is about the client-side (generating the minidump for a crashed process) to this article's server-side (processing/analyzing the minidump).

Re: Everything Is Broken: Shipping Rust-Minidump at Mozilla

#28

Post by gpm »

rockdoe wrote: post #16
Gankra wrote: post #6

Earlier quoted context omitted.

No we removed many random crashes that the C++ code had. You cannot "simply" discard a crash report if something is slightly off because then you would discard most crash reports. And most debuginfo too. You can't expect "thing that runs when a process may have just experienced memory corruption" and "all builds of your application for all eternity" and "every toolchain you ever built your program with for all eterni…

Reminds me of "Your program shouldn't have bugs in it isn't an acceptable position to take for a debugger", from the rr folks. Unfortunately I can't find the source of the quote any more, but it stuck in my mind.

Speaking of the rr folk, they also had the fascinating point that you can reliably generate a "stack trace" by figuring out which `call` instructions were executed with what values (also other jump instructions I suppose), instead of walking the stack. Thereby skipping the whole "parsing the stack is insanely difficult and unreliable" issue.

Re: Everything Is Broken: Shipping Rust-Minidump at Mozilla

#29

Post by glandium »

gpm wrote: post #28
rockdoe wrote: post #16

Earlier quoted context omitted.

Reminds me of "Your program shouldn't have bugs in it isn't an acceptable position to take for a debugger", from the rr folks. Unfortunately I can't find the source of the quote any more, but it stuck in my mind.

Speaking of the rr folk, they also had the fascinating point that you can reliably generate a "stack trace" by figuring out which `call` instructions were executed with what values (also other jump instructions I suppose), instead of walking the stack. Thereby skipping the whole "parsing the stack is insanely difficult and unreliable" issue.

FWIW, that's from pernosco, not rr.

Re: Everything Is Broken: Shipping Rust-Minidump at Mozilla

#30

Post by gpm »

glandium wrote: post #29
gpm wrote: post #28

Earlier quoted context omitted.

Speaking of the rr folk, they also had the fascinating point that you can reliably generate a "stack trace" by figuring out which `call` instructions were executed with what values (also other jump instructions I suppose), instead of walking the stack. Thereby skipping the whole "parsing the stack is insanely difficult and unreliable" issue.

FWIW, that's from pernosco, not rr.

I think it's the same people?
Post reply on HN