Earlier quoted context omitted.
> Still need deep pockets but it’s now increasingly easier to be much better at search than Google. That's the big problem here. All the new search engines rely on the infrastructure of the old ones. It would take very little collusion between Google and Microsoft to crush something like Kagi. And the moment any new competitor gets big, they'll do it. What's needed is an anti-trust breakup. There's simply too big a c…
Agree, intervention is needed. Any new entrant sort-of has to prove itself also as crawling sites can be hard, e.g. Cloudflare which fronts an appreciable portion of the web. Also large sites like Reddit you need to crawl them at 100s of pages a second to keep up to date with content, not something they're generally going to appreciate without a recognisable benefit to them.
It's very hard, getting harder as websites grow more hostile to crawlers. (With more and more putting up login walls to keep poorly written AI crawlers out.)
But the real problem is that tech is culturally unused to dealing with monopolists "on even ground". The wider tech community operates on the implied worldview that they can take on anyone by just asking daddy A16Z for a duovigintillion dollars. For 25 nearly-continuous years, you could just outspend to get into even the most hostile markets.
That economic climate is just ... gone now. No more free money. You can't just buy your way to the top anymore, the business has to financially work.
And most importantly: You can't outspend the incumbents anymore, but they still can outspend you.
