I guess it really comes down to how much faith you're willing to put into the answers LLMs are putting in front of you.
If you trust them blindly, then the vast experiment improvements are obvious and apparent.
If you don't, or if you're the kind of person that likes to research your sources, LLMs are a speed bump.
> because you can't have the vocabulary to search for it directly unless you already know the answer. Indirect questions such as "framework options to do x and y in z situation in this language". The best you could hope for pre-LLM was to find forum posts asking the same or a vaguely similar question and comparing a lot of options, finding out you picked a dud after spending an hour on it because it's fundamentally incompatible due to reasons, searching again, etc.
I disagree with this. Stack Overflow (pre-moderation insanity) forums and the like was and is great at finding answers to questions like this. Reddit threads were also useful for this sort of discussion. Much learning was had while reading through the comments on my way to the answer. Sometimes, doing that refined or re-aligned what I was looking for, as is common when doing research.
Again, it comes back to faith in LLMs. Sure, I can ask an LLM to give me a comprehensive overview of web serving frameworks for $LANGUAGE. It's up to the user to determine how valid the information being put in front of them is.
LLMs can also be extremely confidently incorrect. Example: I used an LLM recently in "research" mode to outline how a solution I sell stacks up to the next biggest competitor in pricing structures. It gave me a lot of (too much) information in a readily-digestible format, including, surprisingly, the "agreed-upon" price of the competitor's products per SKU.
Pricing for enterprise sales contracts is very dark arts, so I went to the sources attached to the result to confirm those numbers. Lo and behold, the prices I was given were nowhere to be found in any of those articles.
Meanwhile, I used a search engine manually to see if I could find a leaked price book using "filetype:pdf" operators, mostly for grins. Found it in 15 seconds. That still wasn't applicable to what I was looking for, as it was for an industry different from mine, but it was there.
I could have told the LLM to deep search PDFs (despite telling it to "ultrathink"), but at that point, again, what is the point of using LLMs if I can do the work myself?