Out of curiosity, I wondered if you could break a tokenizer by introducing weird characters not mapped to an id.
But apparently, they either just emit a [UNK] token or translate the unrecognized character into raw UTF-8 bytes.
How LLMs work
191–200 of 293 posts
Re: How LLMs work
#192Back when ChatGPT came out, I was so shocked by how _good_ it was for an “AI” product that I simply had to know how it worked. Over the next month I ended up drawing out a block diagram on a whiteboard I have in my office, with the math involved next to each step in the blackboard. I’d puzzle about each step along the way, and the triumph of completing the drawing was also that of this sense of deep understanding. I…
For anyone who is curious about the first paragraph here, this is actually a great video overview of how LLM works and the tokenization part. Tangentially related: This part always seemed fuzzy to me, especially when dealing with data scientists and how they talk about how 'ML' looks at problems. I had this issue when working at a SIEM vendor where they kept going on about use case development having to be designed a…
> this is actually a great video overview of how LLM works and the tokenization part
Did you mean to link to the video? I would be interested.
Re: How LLMs work
#193Nice article but chain of thought is what makes frontier LLMs smart, not really the token loop
Is chain of thought same as test time compute?
Re: How LLMs work
#194Back when ChatGPT came out, I was so shocked by how _good_ it was for an “AI” product that I simply had to know how it worked. Over the next month I ended up drawing out a block diagram on a whiteboard I have in my office, with the math involved next to each step in the blackboard. I’d puzzle about each step along the way, and the triumph of completing the drawing was also that of this sense of deep understanding. I…
What hopes/paths does a mere CS bachelor (not deep into stats/maths), and mid level dev (native mobile only; 10-15 years exp.), have about not only understanding it (maybe not fully) but getting possibly into this as a career? Not expecting churning out models and AI systems from the first weeks/months but entry/employment into this field? (If I can be honest, and I am not being disparaging about anything lest it mig…
Im also a mere mortal, and after putting a few years into it IMO I’d say people make it much more complicated than it actually is. I failed most of my math courses for lack of interest, but found passion later with the aforementioned SLAM stuff. I have no doubt you or any other programmer could learn this stuff, especially since you can ask ChatGPT clarifying questions.
I have no idea about careers at this point, I’m still doing fancy IT work as my day job I and look away from the future with dread. I also haven’t been looking for new roles on the open job market, so who knows maybe there’s multimillion pay packages for anyone who can articulate how attention works in an interview.
Re: How LLMs work
#195Back when ChatGPT came out, I was so shocked by how _good_ it was for an “AI” product that I simply had to know how it worked. Over the next month I ended up drawing out a block diagram on a whiteboard I have in my office, with the math involved next to each step in the blackboard. I’d puzzle about each step along the way, and the triumph of completing the drawing was also that of this sense of deep understanding. I…
Since you spent a month digging into this, can you recommend any materials/projects to look into to get a decent grasp of how they work?
I’d recommend my method of just drawing out the block diagram and drawing out + digging into the math at each step! I’m the kind of person who needs to take time to ask lots of questions before stuff clicks, and if you are too I strongly recommend it.
I picked it up from trying to teach myself that SLAM stuff. The papers are very short, but highly information dense and at the time there was no ChatGPT to help me. I got through them by just creeping my way through the math with a whiteboard, and something about drawing it out and having it there in my office made it all click. Trying to watch piecemeal lectures on YouTube or grind through foundational books like MVG just didn’t work for me, I used them instead as references for my drawings.
Same happened when I tried learning this GPT stuff. karpathy’s videos were out at the time, but I couldn’t really stay focused on them or connect the math with the code. Most other descriptions I could find were focused on getting you to use their inference library or harness. Assembling the picture together on my whiteboard by focusing on drawing out the block diagram continues to be my personal favorite method for deep understanding of complex systems.
Re: How LLMs work
#196Earlier quoted context omitted.
Those are all just optimizations. We still don’t really know why they work, we just know how to build them.
Hm, I wonder if it's more that we're shocked such a simple thing (relatively speaking) can work so well .
It was precisely that for me! Another commenter captures it well; “the bitter lesson” indeed.
Re: How LLMs work
#197Back when ChatGPT came out, I was so shocked by how _good_ it was for an “AI” product that I simply had to know how it worked. Over the next month I ended up drawing out a block diagram on a whiteboard I have in my office, with the math involved next to each step in the blackboard. I’d puzzle about each step along the way, and the triumph of completing the drawing was also that of this sense of deep understanding. I…
Don't forget the stolen data from books and papers. You'll never get anything intelligent without using the stolen data they had access to.
WTF?
Re: How LLMs work
#198The part about positional encoding is not correct. > The intuition: instead of adding position info to each token’s vector, RoPE rotates the vector by an angle that depends on its position You can't rotate the token's entire vector (or all three vectors, whatever is being implied is unclear). You rotate each token's Query and Key vectors only, so dot product can be used to tell how far apart the tokens are when compa…
Could you restate this another way: I don't follow.
Re: How LLMs work
#199Earlier quoted context omitted.
> I think you need to ask what you actually want to do with the AI. What about improving the efficiency of token consumption, etc., basically opportunities for improving cost/performance? I keep thinking there has to be a better way to share context with models than dumping entire gigantic skill files of raw text or otherwise into them - I'm betting there's a bunch of low-hanging fruit there.
There may be some low hanging fruit, but they're not available to people without deep understanding of how the math works. Well paid people already spend a lot of time thinking about this.
i am not sure acctually of the math is acctually that complicated/important. the math around neural networks is calculus/chain rule etc and for model comparison/validation one needs statistics. the required math for e.g. understand transformers is quite accessible.
Re: How LLMs work
#200What is the minimum Nvidia graphics card required to train a 1B model from scratch??