I can provide a data point for what the article calls pseudo productivity: I extensively use LLMs as semantic search engines or expert systems (but not as agents). Recently I asked one how to consume a Google Pub/Sub topic using Python (context: I come from an C++/Java/JS background with some Python knowledge). The LLM gave me a good intro and some code. As it usually happens, I had a few follow up questions/clarific…
This is a very common effect and I don't want to be defending LLMs here. But I've seen the same study with CLIs - people using them feel more productive but take longer than using GUIs. What I want to say is that it's very situational and it's likely good to focus on the average. Using LLMs as docs are bad when good docs exist, but if you aren't sure if they do, it's a gamble. A much better approach would be to have…
I hope that this isn't the case for me poking around in vim, using ctags etc. But sadly it may be true.