Can caching (Google) fonts, in itself, be used as some sort of security exploit?
I mean, sure, a site can time whether or not you've already downloaded a certain font before, but that's only useful to determine whether or not you've visited any of the sites with that font. Which is only usable if your sensitive site is also using a custom, nowhere-else-deployed font. That doesn't seem to be that painful security wise, compared to, say, telling websites which browser you're running.
Regardless - I support less reliance on custom fonts and a slimmer web overall.