How easy is it for somebody to get your PGP key off a yubikey if they stole it? In particular since physical devices can be fuzzed, etc, it might be worse than an encrypted keychain on device for some people?
The Yubikey NEO used the NXP a700x microcontroller family the last I checked (NEO-n might as well but I'm not sure). You can read about the security features of the chip at NXP's website[1] to get a sense for what they're designed to defend against, but despite the very real possibility of a successful key retrieval attack on the card, in general it would be MUCH easier for an attacker to obtain your on-disk keychain…
Bottom line: It depends largely on who stole it and for how long - however, if someone knew enough to target stealing your hardware token/smart card, you are under targeted attack and your key's probably toast: revoke it.
I would also gently caution that the elliptic curve routines provided by NXP should not be used over special prime fields (elliptic curves that are fast in software, i.e. just about everything except Brainpool) as they abuse the large (RSA) multiplier whose side-channel blinding was not designed for non-random fields. (Unless they've rewritten it since I last saw it?)