The problem is that website are commodities. The future for original high quality websites is that they should not allow search engines to crawl their content and slowly become a brand which people trust. Stop relying on search for traffic. Be an app, websites for marketing only. Alternative new search engines are needed that allow for embedding/iframe/integration of global web search results in any 3rd party website…
My understanding is that .com (commerce) prevailed and people who have websites want to be fully public, visible and searchable.
On the other hand deep web and dark web exist but most of the people who use it and operate there do it because of shady business.
I don't entirely understand your search engine idea but if internet would start all over again I think the right model for search engine would be to have website owners submit their websites to the search engines along with metadata so searching experience becomes rich.
Metadata would be similar to something like Resource Description Framework https://www.dlib.org/dlib/may98/miller/05miller.html