Determining the impact of library bugs can be really hard. For example, some functionality might be so broken that it's simply impossible to use correctly, so a buffer overflow on top does not make a difference. Or a buffer overflow vulnerability triggers consistently during system boot, so that you never get to the login prompt. These can hardly be considered vulnerabilities. On the other hand, we have clear buffer management bugs, where we must expect that there is application code out there which specifies tight buffer bounds and requires that the library stays within that buffer. (Rust should help here, at least out-of-bounds accesses should turn into panics.) But most bugs are unfortunately somewhere in the middle. Tools like Debian Code Search can provide some evidence. But in the end, it's more subjective than I'd like it to be.
On the extreme end, we have compiler soundness bugs. I'm a bit of worried that I'm hitting any of those when I'm tweaking the types until the compiler no longer complains. Beyond the basics, I really don't have a grasp of Rust's type system rules. But I suspect they very difficult to hit by accident, and even if I do, the code must be miscompiled in meaningful, but difficult-to-notice way. All that seems rather unlikely, which is why these bugs aren't treated as vulnerabilities.
I really think we need some corrective factor (such as potential implication impact) when determining whether toolchain bugs are vulnerabilities.