Where's the shovelware? Why AI coding claims don't add up
mikelovesrobots.substack.com
Where's the shovelware? Why AI coding claims don't add up
1–10 of 498 posts
Re: Where's the shovelware? Why AI coding claims don't add up
#2This makes some sense. We have CEOs saying they're not hiring developers because AI makes their existing ones 10X more productive. If that productivity enhancement was real, wouldn't they be trying to hire all the developers? If you're getting 10X the productivity for the same investment, wouldn't you pour cash into that engine like crazy?
Perhaps these graphs show that management is indeed so finely tuned that they've managed to apply the AI revolution to keep productivity exactly flat while reducing expenses.
Re: Where's the shovelware? Why AI coding claims don't add up
#3Maybe if the number of "Show HNs" has gone up that might be a data point.
Re: Where's the shovelware? Why AI coding claims don't add up
#4While I like the self reflection from this article, I don't think his methodology adds up (pun intended). First there are two main axis where LLMs can make you more productive: speed & code quality. I think everyone is obsessed about the first one, but its less relevant.
My personal hypothesis is that when using LLMs, you are only faster if you would be doing things like boilerplate code. For the rest, LLMs don't really make you faster but can make your code quality higher, which means better implementation and caching bugs earlier. I am a big fan of giving the diff of a commit to an LLM that has a file MCP so he can search for files in the repo and having it point any mistakes I have made.
Re: Where's the shovelware? Why AI coding claims don't add up
#5I think the author misses a few points
* METR was at best a flawed study. Repo-familiarity and tool-unfamiliarity being the biggest points of critique, but far from the only one
* they assume that all code gets shipped as a product. Meanwhile, AI code has (at least in my field of view) led to a proliferation of useful-but-never-shipped one-off tools. Random dashboards to visualize complex queries, scripts to drive refactors, or just sheer joy like "I want to generate an SVG of my vacation trip and consume 15 data sources and give it a certain look".
* Their own self-experiment is not exactly statistically sound :)
That does leave the fact that we aren't seeing AI shovelware. I'm still convinced that's because commercially viable software is beyond the AI complexity horizon, not because AI isn't an extremely useful tool
Re: Where's the shovelware? Why AI coding claims don't add up
#6Good article, gave me some points I hadn't considered before. I know there are some AI generated games out there, but maybe the same people were using asset flips before?
I'd also be curious how the numbers look for AI generated videos/images, because social media and youtube seem absolutely flooded with the stuff. Maybe it's because the output doesn't have to "function" like code does?
Grammatical nit: The phrase is "neck and neck", like where two race horses are very close in progress
Re: Where's the shovelware? Why AI coding claims don't add up
#7There is actually a lot of AI shovelware on Steam. Sort by newest releases and you'll see stuff like a developer releasing 10 puzzle games in one day.
I have the same experience as OP, I use AI every day including coding agents, I like it, it's useful. But it's not transformative to my core work.
I think this comes down to the type of work you're doing. I think the issue is that most software engineering isn't in fields amenable to shovelware.
Most of us either work in areas where the coding is intensely brownfield. AI is great but not doubling anyone's productivity. Or, in areas where the productivity bottlenecks are nowhere near the code.
Re: Where's the shovelware? Why AI coding claims don't add up
#8While I like the self reflection from this article, I don't think his methodology adds up (pun intended). First there are two main axis where LLMs can make you more productive: speed & code quality. I think everyone is obsessed about the first one, but its less relevant. My personal hypothesis is that when using LLMs, you are only faster if you would be doing things like boilerplate code. For the rest, LLMs don't rea…
I don't know what world you're living in, but quality code isn't a forte of ai
Re: Where's the shovelware? Why AI coding claims don't add up
#9Multiple things can be true at the same time:
1. LLMs do not increase general developer productivity by 10x across the board for general purpose tasks selected at random.
2. LLMs dramatically increases productivity for a limited subset of tasks
3. LLMs can be automated to do busy work and although they may take longer in terms of clock time than a human, the work is effectively done in the background.
LLMs can get me up to speed on new APIs and libraries far faster than I can myself, a gigantic speedup. If I need to write a small bit of glue code in a language I do not know, LLMs not only save me time, but they make it so I don't have to learn something that I'll likely never use again.
Fixing up existing large code bases? Productivity is at best a wash.
Setting up a scaffolding for a new website? LLMs are amazing at it.
Writing mocks for classes? LLMs know the details of using mock libraries really well and can get it done far faster than I can, especially since writing complex mocks is something I do a couple times a year and completely forget how to do in-between the rare times I am doing it.
Navigating a new code base? LLMs are ~70% great at this. If you've ever opened up an over-engineered WTF project, just finding where HTTP routes are defined at can be a problem. "Yo, Claude, where are the route endpoints in this project defined at? Where do the dependency injected functions for auth live?"
Right tool, right job. Stop using a hammer on nails.
Re: Where's the shovelware? Why AI coding claims don't add up
#10While I agree with the points he’s raising let me play devils advocate.
There’s a lot more code being written now that’s not counted in these statistics. A friend of mine vibe coded a writing tool for himself entirely using Gemini canvas.
I regularly vibe code little analyses or scripts in ChatGPT which would have required writing code earlier.
None of these are counted in these statistics.
And yes AI isn’t quite good enough to super charge app creation end to end. Claude has only been good for a few months. That’s hardly enough time for adoption !
This would be like analysing the impact of languages like Perl or Python on software 3 months after their release.