Where's the shovelware? Why AI coding claims don't add up

Chatgpt.

What if someone came out with a study saying they had a tool to make the fastest doctors and lawyers? You'd say that doesn't even make sense, what kinds of doctors doing what kinds of work?

AI coding isn't some across the board, helps everyone do anything kind of tool.

Maybe sometime soon we'll stop strawmanning this.

- Built a windows desktop app that scans local folders for videos and automatically transcribes the audio, summarises the content into a structured JSON format based on screenshots and subtitles, and automatically categorises each video. I used it on my PC to scan a couple of TB of videos. Has a relatively nice interface for browsing videos and searching and stores everything locally in SQLite. Did this in C# & Avalonia - which I've never used before. AI wrote about 75% of the code (about 28k LOC now).

- Built a custom throw-away migration tool to export a customers data from one CRM to import into another. Windows app with basic interface.

- Developed an AI process for updating a webform system that uses XML to update the form structure. This one felt like magic and I initially didn't think it would work, but it only took a minute to try. Some background - years ago I built a custom webform/checklist app for a customer. They update the forms very rarely so we never built an interface for making updates but we did write 2 stored procs to update forms - one outputs the current form as XML and another takes the same XML and runs updates across multiple tables to create a new version of the form. For changes, the customer sends me a spreadsheet with all the current form questions in one column and their changes in another. It's normally just wording changes so I go through and manually update the XML and import it, but this time they had a lot of changes - removing questions, adding new ones, combining others. They had a column with the label changes and another with a description of what they wanted (i.e. "New Question", "Update label", "Combine this with q1, q2 and q3", "remove this question"). The form has about 100 questions and the XML file is about 2500 lines long and defines each form field, section layout, conditional logic, grid display, task creation based on incorrect answers etc, so it's time consuming to make a lot of little changes like this. With no expectation of it working, I took a screenshot of the spreadsheet and the exported XML file and prompted the LLM to modify the XML based on the instructions in the spreadsheet and some basic guidelines. It did it close to perfect, even fixing the spelling mistakes the customer had missed while writing their new questions.

- Along with using it on a daily basis across multiple projects.

I've seen the stat that says developers "...thought AI was making them 20% faster, but it was actually making them 19% slower". Maybe I'm hoodwinking myself somehow, but it's been transformative for me in multiple ways.

I think the article's premise is basically correct - if we had a 10x explosion of productivity where is the evidence? I would think some is potentially hidden in corporate / internal apps but despite everyone at my current employer using these tools we don't seem to be going any faster.

I will admit that my initial thoughts on Copilot were that "yes this is faster" but that was back when I was only using it for rote / boilerplate work. I've not had a lot of success trying to get it to do higher level work and that's also the experience of my co-workers.

I can certainly see why a particular subset of programmers find the tools particularly compelling, if their job was producing boilerplate then AI is perfect.