There's low power inference from Intel (Movidius) and Google (Coral Edge TPU). Nvidia doesn't really have anything below the Jetson Nano. I think there are a smattering of other low power cores out there (also dedicated chips in phones). TPUs are used on the high performance end and there are also companies like Graphcore who do insane things in silicon. Also niche HPC products like Intel Knight's Landing (Xeon Phi) which is designed for heterogeneous compute.
There isn't a huge amount of competition in the consumer/midrange sector. Nvidia has almost total market domination here. Really we just need a credible cross platform solution that could open up gpgpu on AMD. I'm surprised Apple isn't pushing this more, as they heavily use ML on-device and to actually train anything you need Nvidia hardware (eg try buying a Macbook for local deep learning training using only apple approved bits, it's hard!). Maybe they'll bring out their own training silicon at some point.
Also you need to make a distinction between training and inference hardware. Nvidia absolutely dominate model training, but inference is comparably simpler to implement and there is more competition there - often you don't even need dedicated hardware beyond a cpu.