This is an area that's dear to me, I'm the cofounder of Vectara and have been working with embedding-based semantic search, aka neural search or neural IR, since 2017.
To whether Google uses semantic search, the answer is yes, very heavily [1][2]. Not only that, but they have led, and continue to lead, much of the pioneering research in NLP and neural IR for the past decade [3][4][5].
Technical challenges lie along a few primary dimensions. The first has been search quality, because, while early neural systems like Google Talk to Books [5][6] demonstrated the potential of these techniques, benchmarks like BEIR [7], released a few years later, in 2020, showed that the best keyword retrieval algorithms still outperformed neural techniques in general settings.
The landscape since then has shifted very rapidly: In 2022, for the first time, neural search methods outperformed BM25 on BEIR. This includes late interaction [8], sparse encoding [9], and, most challengingly, dense encoding [10] systems.
The second technical challenge is scalability. After decades of infrastructure optimization, keyword systems scale well to very large corpora, while semantic systems struggle to achieve the same scale. The k-d tree approach presented in the article, for example, while good for experimentation, would be difficult to productionize, as-is, in a large-scale system.
However, research into scaling dense vector retrieval has received a lot of focus recently [11], so I'm confident this will change.
I'll close by saying your observation about being stuck with keyword search in a lot of apps is accurate, but I expect that to change soon. It's becoming easier to embed neural models everywhere, and I think that distilled models in the 5-50mb size range can feasibly power semantic search everywhere you press Ctrl-F today.
[1] https://blog.google/products/search/search-language-understa...
[2] https://blog.google/products/search/introducing-mum/
[2] https://arxiv.org/abs/1706.03762
[3] https://arxiv.org/abs/1810.04805
[4] https://arxiv.org/abs/1907.04307
[5] https://books.google.com/talktobooks/
[6] https://ai.googleblog.com/2018/04/introducing-semantic-exper...
[7] https://arxiv.org/abs/2104.08663
[8] https://arxiv.org/abs/2112.01488
[9] https://arxiv.org/abs/2109.10086
[10] https://arxiv.org/pdf/2112.09118.pdf
[11] https://www.microsoft.com/en-us/research/uploads/prod/2021/1...