I thought is hard to have low latency audio in Linux with stuff like PulseAudio.
I switched to BSD and got much better latency on the same hardware
But I think that was the alsa -> BSD oss driver swap, which is possible on linux too. I had just never tried it, since I assumed linux was best
(I'm only sharing this cus i think its interesting, and may help you get good latency)