The "magic" is very simple and understandable if you read the docs; I fairly quickly gained an understanding of it that allowed me to dig into the odd performance issue or bug without much trouble
1) Observables are properties on objects; when their getter is called, it becomes "tracked" (subscribed to) by the currently-running tracked function. When their setter is called, it publishes to all subscribers. This happens through JS proxies for plain objects, and class decorators for class instances.
2) A tracked function is anything that must be re-evaluated when an observable it depends on is modified. This includes React render functions, computed functions, and any other side-effects you've created via autorun(), reaction(), etc. Though the idea is to avoid side-effects, so 90% of the time it's just render functions and computed functions. In these cases it amounts to a cache-clear on a pure function: "the return value will be different, so re-evaluate it and use the new return value". A computed function is an observable property getter that's also a tracked function (it doesn't have a corresponding setter; instead, it publishes when it's published to).
3) Tracking works by a) marking global state before a tracked function is run, b) making note of any observables that report back during that time, c) clearing the global state (and subscribing to the observables) when the tracked function completes. Because this is done temporally based on before/after the function has executed, any nested function calls get treated exactly the same as the top-level function call. There's no special magic necessary for that.
That's about all there is to it. Everything else in the library is just another permutation on these concepts.