On being wrong about AI

Transformer architecture happened, that is key for neural networks to build up context dependent reasoning, we wouldn't have gotten to where we are today without it.

Without it scaling neural networks doesn't add much value, since without being able to separate out contexts they fail to develop a large amount of separate skills within the same model.

So yes, a very important key was found, it wasn't just scaling that got us here.

In the 1980s I was lucky enough to stumble into the opportunity of serving on a DARPA neural networks tools panel for a year and getting lucky applying a simple backprop model to a show-stopper problem for a bomb detector my company designed and built for the FAA. Bonus time!

Decades went by, and then deep learning really changed my view of AI and technology (I managed a deep learning team at Capital One, and it was a thrill to see DL applications there).

Now I am witnessing the same sort of transformations driven by attention based LLMs. It is interesting how differently people now view pros/cons/dangers/this-shit-will-save-the-world possibilities of AI. Short anecdote: last week my wife and I had friends for dinner. Our friends, a husband and wife team who have decades of success writing and producing content for movies and now streaming. I gave them a fairly deep demo of what ChatGPT Pro could do for fleshing out story ideas, generating images for movie posters and other content, etc. The husband was thrilled at the possibility of now being able to produce some of his previously tabled projects inexpensively. His wife had the opposite view, that this would cause many Hollywood jobs to be lost, and other valid worries. Normally our friends seem aligned in their views, but not in this instance.