Reverse Debugging at Scale
engineering.fb.com
Reverse Debugging at Scale
1–10 of 22 posts
Re: Reverse Debugging at Scale
#2It is fairly surprising to me that FB would pay a roughly 5% throughput penalty to get this.
Re: Reverse Debugging at Scale
#3It is fairly surprising to me that FB would pay a roughly 5% throughput penalty to get this.
It is not that surprising. Facebook is complex ecosystem, and therefore experiences lots of outages/problems.
Looking at their status page (https://developers.facebook.com/status/dashboard/), they seem to have a problems every week. These are only the public ones!
I guess they figured they are losing more money due to these problems than the additional 5% they have to spend on infrastructure.
Re: Reverse Debugging at Scale
#4It is fairly surprising to me that FB would pay a roughly 5% throughput penalty to get this.
In a car, you get the best speed when you press the pedal to the metal and close your eyes. Yet we pay performance penalty by driving it, instead.
Re: Reverse Debugging at Scale
#5It is fairly surprising to me that FB would pay a roughly 5% throughput penalty to get this.
It is not that surprising. Facebook is complex ecosystem, and therefore experiences lots of outages/problems. Looking at their status page ( https://developers.facebook.com/status/dashboard/ ), they seem to have a problems every week. These are only the public ones! I guess they figured they are losing more money due to these problems than the additional 5% they have to spend on infrastructure.
I can see why an org would choose to do this, but the number is still frightening. At Google, we were held to a limit of at most 0.01% cost for sampling LBR events. 5% for debug-ability just seems really high.
Re: Reverse Debugging at Scale
#6It is fairly surprising to me that FB would pay a roughly 5% throughput penalty to get this.
Where does the 5% number come from? I didn't see it in the article or on the linked page for Intel Processor Trace.
Re: Reverse Debugging at Scale
#7From what I can tell, they are just using standard instruction trace rather than a full trace, so they can only inspect execution history rather than full data history that most other time travel debugging solutions provide. The advantages of their approach of just using the standard hardware instruction trace functionality is that it functions even on shared-memory multithreaded applications at "full" speed unlike most other time travel debugging solutions. The disadvantages being that it requires higher storage bandwidth, Intel does not seem to support data trace, and even if it did support data trace would require significantly more storage bandwidth (something like 10x-100x).
Re: Reverse Debugging at Scale
#8It is fairly surprising to me that FB would pay a roughly 5% throughput penalty to get this.
In a car, you get the best speed when you press the pedal to the metal and close your eyes. Yet we pay performance penalty by driving it, instead.
Car analogy is not applicable - think more instead of lowering speed of the entire high-speed roadway by 5mph (e.g. by road material/quality) - has a qualitative difference at that scale.
Re: Reverse Debugging at Scale
#9It is fairly surprising to me that FB would pay a roughly 5% throughput penalty to get this.
Where does the 5% number come from? I didn't see it in the article or on the linked page for Intel Processor Trace.
Pulled from my experience. How much stuff needs to be logged depends on the structure of the program, and different programs are more or less sensitive to having some of their memory store bandwidth stolen.
Re: Reverse Debugging at Scale
#10[deleted]