Preliminary data from a longitudinal AI impact study

> Model's are no longer revolutionary, they are evolutionary, and the evolution and per model-version difference is narrowing each release.

We've definitely culled some low hanging fruit, but I think there's still a lot of room for improvements that could lead to step changes in capabilities. I think we're only scratching the surface of looped language models, thinking in latent space, and multimodality.

And even if the per-model differences are narrowing, even single digit improvements in performance metrics could yield outsized effects in applicability and productivity. Consider services that guarantee one 9 of reliability vs. five 9s. In absolute terms that change is a trivia difference, but the increased reliability allows use in way, way more domains.

I wrote a short bit on a similar topic the other day[^a]. Just because something is faster or even measurably better, that doesn't translate to end productivity.

1. You might be speeding up something that is inherently not productive (the "faster horses" trope). I see companies using AI to generate performance reviews. Same company using AI to summarize all the new performance stuff they're getting. All that's happening is amplified busywork (there is real work in there, but questionable if it's improved).

2. Some things are zero sum. If you're not using AI for marketing you might fall behind. So you adopt these tools, but attention/etc are limited. There is no net gain, just competition.

3. You might speed one part up (typing code), but then other parts of your pipeline quickly become constraints. It might be a long time before we're able to adapt the end-to-end process. This is amplified by coding tools being three strides ahead.

4. Then there are actual productivity improvements. One of these PRs could have been "translate this to German". That could be one PR but a whole step-change for the business.

So much of what is happening falls in buckets 1+2+3. I don't think we've really got into the meat of 4 yet.

a: https://jonathannen.com/ai-productivity/

If you think a department or individual working 10% faster makes a company 10% more productive, you’re almost certainly wrong.

Productivity only improves if the change increases revenue or reduces costs. And that rarely happens unless you improve the actual bottleneck of the organization.

To understand why, I recommend the book The Goal: A Process of Ongoing Improvement by Eliyahu M. Goldratt and Jeff Cox.

The cost of doing something wrong still is high. Even if bad code is produced instantaneously, its detrimental effect on production remains the same. Yes, yes, what fell on the floor and was picked up in five seconds is still considered fine to eat! Does not apply to eggs though. Customer trust is usually such an egg.

Writing code has become much faster. Writing correct and reliable code has become somehow faster, but not nearly as much. Understanding what code to write has barely become faster.

The more novel is the code you're writing, the smaller are gains from AI writing it.

As someone who spends an hour or two a week using AI tools, it's a time sink. I spend (almost) all my LLM time keeping on top of changes in my company's preferred tooling, updating software, updating authentication mechanisms, etc.

If I spent twice the time with these tools, most of the additional time invested would be "profit". So maybe there's something to these arguments that "it will only get better".

OTOH, we also see this with business investments of all types. "We're spending all our revenue on growth, if we wanted profits, we could slow investment at any time, and be immediately profitable!"