Great take. I worked on Confluence for a few years and have a bit of insight.
Search has been an area of focus on and off for the most part of the last 15 years. It actually has gotten a lot better and Atlassian has an entire team focused on improving the search experience across their suite of products (they started with Confluence). And from what I hear, they are focusing on all the right things.
To your point, no search system can be a good fit for every possible use-case. Confluence has a number of different use-cases, but let's just pick "documentation" and "intranet" as an example here.
Intranets are, to a large degree, about keeping up with what's new in a company. Therefore recent content is likely more relevant than older content.
When used for documentation, recency doesn't matter at all. If a document was written 2 years ago, but the content is still accurate, it's just as relevant as it was on day one.
That means no single relevance configuration will work well for all use-cases. Leveraging ML is essential. But even a single ML model across an entire Confluence instance is not going to work as different spaces are used for different use-cases. What's really required here is to build different models for different spaces to create a tailored relevancy for each space. It's not an easy problem to solve, but I'm confident they will get there with time.
Seeing the challenges with Search at Atlassian, despite having a large, dedicated team of engineers working on the problem, is what motivated me to join http://sajari.com. We've been doing a lot of work on reinforcement learning and Neural Search. Our focus right now is on public content websites and e-commerce, but eventually we will get around to enable products like Confluence to create a great search experience without the need for an entire team. Search is a hard problem, but there is so much opportunity to improve the experiences that are available today. Exciting times.