21–30 of 181 posts
Earlier quoted context omitted.
Link? Sounds fun and reminds me of this tweet [0] [0] https://x.com/jaschasd/status/1756930242965606582
Nah someone down voted it. And yes, it looks like that + 20 others that are animated.
Earlier quoted context omitted.
Nah someone down voted it. And yes, it looks like that + 20 others that are animated.
Downvotes == empty boats. If "Empty Boat parable" doesn't ring a bell, Google it...
I finally understand this! Why did every other video make it so confusing!
Because: 1. good communication requires an intelligence that most people sadly lack 2. because the type of people who are smart enough to invent transformers have zero incentive to make them easily understandable. most documents are written by authors subconsciously desperate to mentally flex on their peers.
Often, the disseminating medium is a one-sided, like a video or a blog post, which doesn't help, either. A conversational interaction would help the expert sense why someone outside the domain find the subject confusing ("ah, I see what you mean"...), discuss common pitfalls ("you might think it's like this... but no, it's more like this...") etc.
Hold on, every predicted token is only a function of the previous token? I must have something wrong. This would mean that within the embedding of "was", which is of length 12,228 in this example. Is it really possible that this space is so rich as to have a single point in it encapsulate a whole novel?
each prediction is based on a weighted context of all previous tokens, not just the immediately preceding one.
Hold on, every predicted token is only a function of the previous token? I must have something wrong. This would mean that within the embedding of "was", which is of length 12,228 in this example. Is it really possible that this space is so rich as to have a single point in it encapsulate a whole novel?
(is my understanding)
I finally understand this! Why did every other video make it so confusing!
In my experience, with very few notable exceptions (e.g. Feynmann), researchers are the worst when it comes to clearly explaining to others what they're doing.
I'm at the point where I'm starting believe that pedagogy and research generally are mutually exclusive skills.
One great way to improve on this bottleneck is a new-ish idea called Ring Attention. This is a good article explaining it:
https://learnandburn.ai/p/how-to-build-a-10m-token-context
(I edited that article.)