I'm an audio nerd; I went to school for it, and I'm back to working professionally in the field (after a 15 year hiatus where I did it as a hobby). Serious audio problems do still exist on Linux systems, and it is a source of huge annoyance for me that my system still causes my (very expensive) studio monitors to pop when switching between some sound sources in some situations. But, these are mostly to do with the sound server and switching sources rather than with glitches in actual audio.
I still can't use Linux as my primary OS for audiovisual work because there are still issues with getting many channels of low latency, high resolution, audio playing and recording reliably.
But, the issues you describe as they apply to this task are mostly a long solved problem, at least for very simple tasks, like reliably playing 16 bit, 44.1 kHz, stereo audio. Honestly, there's never been a time that I can remember when I couldn't reliably play a stereo 16 bit, 44.1k, file back on a Linux system...going back to 1995, or so, when I first started using Linux. The problems come when you demand a little more of the system.
Once again, errors of the level that exist today (which are completely inaudible to even trained ears in most cases), in terms of latency, jitter, dropouts, etc. are so tiny and insignificant that they would be impossible for the current hardware to replicate. The hardware is so much slower than the input signal that noise from the audio signal will be lost in the much larger noise of the hardware. You can see the hardware noise in the printed objects. It's vastly larger than anything the computer output is going to screw up, and it'll be many generations before that stops being true (and, by then, maybe Linux will finally have its act together on audio).
Windows, BTW, finally does have its act together on audio, and has for five+ years. I can reliably play a couple dozen 96k 24 bit tracks while recording more on slightly high end laptop. Obviously Mac is also solid on this front.