Blocking LLMs from your website cuts you off from next-generation search

Another one not having a clue about what “consent” means. Next?

And the freeness cuts both ways — if I could, I'd happily open my content to Mistral and all the other totally-free/open-source-releasing LLM companies' scrapers. But I can't; they're going into big corpuses or scraped directly by the commercial actors with funds to scrape the whole kit & kaboodle.

> LLMs are not search engines, and I'm not gaining any followers or customers in any meaningful way because an LLM indexes my site.

Counterpoint: my wife owns an accounting firm and publishes a lot of highly valuable informational content on their website's blog. Stuff like sales tax policies and rates in certain states, accounting/payroll best practices articles, etc. I guess you could call it "content marketing".

Lately they have been getting highly qualified leads coming from LLMs that cite her website's content when answering questions like "What is the sales tax nexus policy in California?". Users presumably follow the citation and then engage with the website, eventually becoming a very warm lead.

So LLMs are obviously not search engines in the conventional sense, but it doesn't mean they are not useful at generating valuable traffic to your marketing website.

> most LLMs have an agentic web-search component that will actively generate links

I guess that’s the problem - search being only a component.

Is the possible search traffic worth having your content become part of an LLM’s training set and possibly used elsewhere?

I guess the answer depends on the content and the website’s business model.

I'm somewhat torn on this one.

As an amateur blogger, I would not like LLMs to "steal" my content, display the users the needed pieces they are looking for, while leaving me with zero visitors. The reason I write is to convey a particular message, which the meaning of gets lost, or worse communicated wrongly, due to LLMs.

As an online business owner, I do see both ChatGPT and Perplexity as referrers to my business, meaning that potential customers ask LLM a question/service recommendation, and LLM is directing them to my service, and I would not like to lose this vertical of organic customer acquisition.

---

On a completely different note, medium should die as a platform, together with substack. The amount of intrusive popups, "install our app" bars, and paywalls is just insane. Bloggers, especially technically savvy ones, should be able to host their own blog.