AI assisted search-based research works now
simonwillison.net
AI assisted search-based research works now
1–10 of 156 posts
Re: AI assisted search-based research works now
#2I think it's important to keep tabs on things that LLM systems fail at (or don't do well enough on) and try to notice when their performance rises above that bar.
Gemini 2.5 Pro and o3/o4-mini seem to have crossed a threshold for a bunch of things (at least for me) in the last few weeks.
Tasteful, effective use of the search tool for o3/o4-mini is one of those. Being able to "reason" effectively over long context inputs (particularly useful for understanding and debugging larger volumes of code) is another.
Re: AI assisted search-based research works now
#3My main observation here is
1. Technically it might be possible to search the Internet, but it might not surface correct and/or useful information.
2. High-value information that would make a research report valuable is rarely public nor free. This holds especially true in capital-intensive or regulated industries.
Re: AI assisted search-based research works now
#4It's a relevant question about the economic model for the web. On one hand, the replacement of search with a LLM-based approach threatens the existing, advertising-based model. On the other hand, the advertising model has produced so much harm: literally irreparable damage to attention spans, outrage-driven "engagement", and the general enshittification of the internet to mention just a few. I find it a bit hard to imagine whatever succeeds it will be worse for us collectively.
My question is, how to reproduce this level of functionality locally, in a "home lab" type setting. I fully expect the various AI companies to follow the exact same business model as any other VC-funded tech outfit: free service (you're the product) -> paid service (you're still the product) -> paid service with advertising baked in (now you're unabashedly the product).
I fear that with LLM-based offerings, the advertising will be increasingly inseparable, and eventually undetectable, from the actual useful information we seek. I'd like to get a "clean" capsule of the world's compendium of knowledge with this amazing ability to self-reason, before it's truly corrupted.
Re: AI assisted search-based research works now
#5I feel like the benefit which AI gives us programmers is limited. They can be extremely advanced, accelerative and helpful assistants, but we're limited to just that: architecting and developing software.
Biologists, mathematicians, physicists, philosophers and the like seem to have an open-ended benefit from the research which AI is now starting to enable. I kind of envy them.
Unless one moves into AI research?
Re: AI assisted search-based research works now
#6I feel like the benefit which AI gives us programmers is limited. They can be extremely advanced, accelerative and helpful assistants, but we're limited to just that: architecting and developing software. Biologists, mathematicians, physicists, philosophers and the like seem to have an open-ended benefit from the research which AI is now starting to enable. I kind of envy them. Unless one moves into AI research?
I don't think AI is trustworthy or accurate enough to be valuable for anyone trying to do real science
That doesn't mean they won't try though. I think the replication crisis has illustrated how many researchers actually care about correctness versus just publishing papers
Re: AI assisted search-based research works now
#7My main observation here is 1. Technically it might be possible to search the Internet, but it might not surface correct and/or useful information. 2. High-value information that would make a research report valuable is rarely public nor free. This holds especially true in capital-intensive or regulated industries.
I fully expect one of the AI-related business models going forward to be charging subscriptions for LLM search tool access to those kinds of archives.
ChatGPT plus an extra $30/month for search access to a specific archive would make sense to me.
Re: AI assisted search-based research works now
#8It's a relevant question about the economic model for the web. On one hand, the replacement of search with a LLM-based approach threatens the existing, advertising-based model. On the other hand, the advertising model has produced so much harm: literally irreparable damage to attention spans, outrage-driven "engagement", and the general enshittification of the internet to mention just a few. I find it a bit hard to i…
You need a copy of r1 and enough ram to run it, and a web searching tool, or a rag database with your personal data store.
Re: AI assisted search-based research works now
#9The article doesn’t mention Kagi: The Assistant, a search-powered LLM frontend that came out of closed beta around the beginning of the year, and got included in all paid plans since yesterday.
It really is a game changer when the search engine
I find that an AI performing multiple searches on variations of keywords, and aggregating the top results across keywords is more extensive than most people, myself included, would do.
I had luck once asking what its search queries were. It usually provides the references.
Re: AI assisted search-based research works now
#10I feel like the benefit which AI gives us programmers is limited. They can be extremely advanced, accelerative and helpful assistants, but we're limited to just that: architecting and developing software. Biologists, mathematicians, physicists, philosophers and the like seem to have an open-ended benefit from the research which AI is now starting to enable. I kind of envy them. Unless one moves into AI research?
I don't think AI is trustworthy or accurate enough to be valuable for anyone trying to do real science That doesn't mean they won't try though. I think the replication crisis has illustrated how many researchers actually care about correctness versus just publishing papers
If you're a skilled researcher I expect you should be able to get great results out of unreliable AI assistants already.
Scientists are meant to be good at verifying and double-checking results - similar to how journalists have to learn to derive the truth from unreliable sources.
These are skills that turn out to be crucial when working with LLMs.