I genuinely don't understand why some people are still bullish about LLMs

Ive had the same experience as you, and also rather recently. I had to learn two lessons: first, what I could trust it with (as with Wikipedia when it was new), and second, what makes sense to ask it (as with YouTube when it was new). Once I got that down, it is one fabulous tool to have on my belt, among many other tools.

Thing is, the LLMs that I use are all freeware, and they run on my gaming PC. Two to six tokens per second are alright honestly. I have enough other things to take care of in the meantime. Other tools to work with.

I don't see the billion dollar business. And even if that existed, the means of production would be firmly in the hands of the people, as long as they play video games. So, have we all tripled our salaries?

If we haven't, is that because knowledge work is a limited space that we are competing in, and LLMs are an equalizer because we all have them? Because I was taught that knowledge work was infinite. And the new tool should allow us to create more, and better, and more thoroughly. And that should get us all paid better.

Right?

I see a difference between seeing them as valuable in their current state vs being "bullish about LLMs" in the stock market sense.

The big problem with being bullish in the stock market sense is that OpenAI isn't selling the LLMs that currently exist to their investors, they're selling AGI. Their pitch to investors is more or less this:

> If we accomplish our goal we (and you) will have infinite money. So the expected value of any investment in our technology is infinite dollars. No, you don't need to ask what the odds are of us accomplishing our goal, because any percent times infinity is infinity.

Since OpenAI and all the founders riding on their coat tails are selling AGI, you see a natural backlash against LLMs that points out that they are not AGI and show no signs of asymptotically approaching AGI—they're asymptotically approaching something that will be amazing and transformative in ways that are not immediately clear, but what is clear to those who are watching closely is that they're not approaching Altman's promises.

The AI bubble will burst, and it's going to be painful. I agree with the author that that is inevitable, and it's shocking how few people see it. But also, we're getting a lot of cool tech out of it and plenty of it is being released into the open and heavily commoditized, so that's great!

Could it be that people who are too bullish are being deceived by smoke and mirrors?

I think the disconnect is that people who produce serious, researched and referenced work tend to believe that most work is like that. It is not. The majority of content created by humans is not referenced, it's not deeply researched, and a lot of it isn't even read, at least not closely. It sends a message just by existing in a particular place at a particular time. And its creation gainfully employs millions of people, which of course costs millions of dollars. That's why, warts and all, people are bullish on LLMs.

I think there is a big problem of expectations. People are told that it is great for software development, so they try to use it on big existing software projects, and it sucks.

Usually that's because of context: LLMs are not very good at understanding a very large amount of context, but if you don't give LLMs enough context, they can't magically figure it out on their own. This relegates AI to only really being useful for pretty self-contained examples where the amount of code is small, and you can provide all the context it needs to do its job in a relatively small amount of text (few thousand words or lines of code at most).

That's why I think LLMs are only useful right now in real-world software development for things like one-off functions, new prototypes, writing small scripts, or automating lots of manual changes you have to do. For example, I love using o3-mini-high to take existing tests that I have and modifying them to make a new test case. Often this involves lots of tiny changes that are annoying to write, and o3-mini-high can make those changes pretty reliably. You just give it a TODO list of changes, and it goes ahead and does it. But I'm not asking these models how to implement a new feature in our codebase.

I think this is why a lot of software developers have a bad view of AI. It's just not very good at the core software development work right now, but it's good enough at prototypes to make people freak out about how software development is going to be replaced.

That's not to mention that often when people first try to use LLMs for coding, they don't give the LLMs enough context or instructions to do well. Sometimes I will spend 2-3 minutes writing a prompt, but I often see other people putting the bare minimum effort into it, and then being surprised when it doesn't work very well.