I don't think it's the scale, but the specialization. When what you need is unspecialized manual labor, favors work great because those kinds of favors are fungible. It starts to fall apart when you need specialized work. I.e. I'm a farmer and I need to see a doctor. The doctor doesn't want to see me, though, because 134 other farmers already owe him/her favors and he/she doesn't know what to do with the favors he/she already has.
The doctor needs a way to convert all those favors they don't need into favors they do need. You could trade them, but then you need an exchange rate. There's 400 farmers in the town and only one electrician; is a favor from the electrician worth the same as a favor from a farmer? At a certain point, it makes sense to just exchange stores of value rather than try to manage favors like assets.
> but we are still trading the same thing
I don't really think they are. Favors are closer to an asset. $5 is $5, regardless of the holder. Favors have bearer-specific value, such that one might be near-worthless to me but priceless to you.
> For example, I use currency to track and buy favors from some laborer in China I have never met.
To me, this demonstrates the above premise. That laborer in China would rightly value your favor at near-zero. You're too far away for them to meaningfully cash in on that favor. Your favor behaves like an asset that they have no use for. Money is distinct in that it allows you to crystalize the labor that would have been done into a physical form, and allows you to do it ahead of time. No more worrying about fungibility, and no more worrying about whether the other party will actually honor the favor.