What we know about LLMs
willthompson.name
What we know about LLMs
1–10 of 173 posts
Re: What we know about LLMs
#2You had me at "Crypto VCs & ”builders” making a hard left into AI"
Re: What we know about LLMs
#3You had me at "Crypto VCs & ”builders” making a hard left into AI"
Oh really now lol
Re: What we know about LLMs
#4You had me at "Crypto VCs & ”builders” making a hard left into AI"
As a matter of fact, there’s even more developers making a hard left into AI who have never touched crypto.
The interesting follow up question is: what will they actually spend time on? Training new models? Copy pasting front ends on ChatGPT? Fine tuning models?
I think many of them will be scared by how much of a hard science ML is vs just spinning up old CRUD apps
Re: What we know about LLMs
#5You had me at "Crypto VCs & ”builders” making a hard left into AI"
As a matter of fact, there’s even more developers making a hard left into AI who have never touched crypto. The interesting follow up question is: what will they actually spend time on? Training new models? Copy pasting front ends on ChatGPT? Fine tuning models? I think many of them will be scared by how much of a hard science ML is vs just spinning up old CRUD apps
Have you used an llm as an application developer? The hard tech for generative ai will be commodified. The products built on top will not.
Re: What we know about LLMs
#6How do you do science on LLMs? I would imagine that is super important, given their broad impact on the social fabric. But they're non-deterministic, very expensive to train, and subjective. I understand we have some benchmarks for roughly understanding a model's competence. But is there any work in the area of understanding, through repeatable experiments, why LLMs behave how they do?
Do we care?
Re: What we know about LLMs
#7Great summary.
I’ve been reading a pop neuroscience book called Incognito (2011).
In it, the author talks about how the brain is a group of competing sub-brains of many forms, and the brain might have several ways of doing the same thing (e.g. recognizing an object). The author also posited that the lack of AI progress back then was due to the fact that there are no constantly competing sub-brains. Our brains are always adjusting and trying new scenarios.
I was struck by how similar these brain observations were to recent developments in AI and LLMs.
The book is full of cool stories, even if some of them are now recognized as non-reproducible. I recommend!
Re: What we know about LLMs
#8Earlier quoted context omitted.
As a matter of fact, there’s even more developers making a hard left into AI who have never touched crypto. The interesting follow up question is: what will they actually spend time on? Training new models? Copy pasting front ends on ChatGPT? Fine tuning models? I think many of them will be scared by how much of a hard science ML is vs just spinning up old CRUD apps
Have you used an llm as an application developer? The hard tech for generative ai will be commodified. The products built on top will not.
As an enthusiast delving in for the pure pleasure of it I see a future ahead where as a programmer, I will have an array of options to use LLM's in hybrid systems. I'm not looking for an AGI, rather a mixture of experts that I can remix as I see fit, each one tuned for a subset of 'intelligence' that can be wielded with relative precision as part of a larger system that combines traditional programming with the new abilities offered by LLMS's and their kin. Certainly an interesting time to dip back in, there are aspects of the domain that mirror my experiences with the early web and for those I am grateful.
Re: What we know about LLMs
#9You had me at "Crypto VCs & ”builders” making a hard left into AI"
As a matter of fact, there’s even more developers making a hard left into AI who have never touched crypto. The interesting follow up question is: what will they actually spend time on? Training new models? Copy pasting front ends on ChatGPT? Fine tuning models? I think many of them will be scared by how much of a hard science ML is vs just spinning up old CRUD apps
> Training new models?
There is value in applying old techniques to new problems. Training a model to, I don't know, recognize snake species might help save snake bite victims lives.
(This is an example I came up with in 5 seconds, please don't take it seriously)
But there's also the whole "sell the shovel" aspect; it can be hard to train models. It can be hard to interpret the quality of the results. How do I know version 2 of the model is better than version 1? How do I even get labeled photos of snakes and not-snakes?
I suspect solving some of those problems are where some of the real gold is buried.
Re: What we know about LLMs
#10Given a set of instructions, an instruction fine-tuned/aligned LLM is able (conditional on size and training quality) to reason through a set of steps to produce a desired output.
This is plainly wrong. The model's growing size makes it better at guessing the outcome of a reasoning task, but little to no actual reasoning is performed.
It's trivial to prove this as well, as LLMs will still fail miserably at (larger) math problems that even basic computer algebra systems will handle with ease.