What’s the incentive for websites to let Kagi and others indexing content if llms in search show relevant informations right away? Wouldn’t something like perplexity ai making more sense then? Or perhaps better application of llms to search
If you publish to make information known, then your incentive is that it helps spread that information to people who may not otherwise visit your site. If you are trying to make money off of search engine traffic then you might not like this much at all. I think most people would rather not be pointed to those sites in the first place, so it’s a win-win if they block crawlers.
Imagine for a second a world where instead of publishing directly to your own website (home), you publish into the LLM's knowledge base. Nobody comes to you for information, they come to the LLM. Nobody cares about you, or the work you put in. Your labour is just a means to their end. To some extent, you could argue that Wikipedia works in this way, but it really doesn't. The work you put in is reproduced verbatim, and the work is collaborative. You get the joy of seeing your writing being used to help other people in a very direct sort of way, as opposed to being aggregated, misinterpreted and generally warped by a non-intelligent LLM.
In other words, you cannot possibly expect others to want to work in a sweat shop, toiling away to provide you with instant gratification. We must leave room for human expression.