This is probably a consequence of the training data being fully lowercase:
You> hello Guppy> hi. did you bring micro pellets.
You> HELLO Guppy> i don't know what it means but it's mine.
Show HN: I built a tiny LLM to demystify how language models work
31–40 of 174 posts
Re: Show HN: I built a tiny LLM to demystify how language models work
#32Why are there so many dead comments from new accounts?
Re: Show HN: I built a tiny LLM to demystify how language models work
#33Why are there so many dead comments from new accounts?
They all seem to be slop comments.
Re: Show HN: I built a tiny LLM to demystify how language models work
#34[flagged]
Re: Show HN: I built a tiny LLM to demystify how language models work
#35I love these kinds of educational implementations. I want to really praise the (unintentional?) nod to Nagel, by limiting capabilities to representation of a fish, the user is immediately able to understand the constraints. It can only talk like a fish cause it’s very simple Especially compared to public models, thats a really simple correspondence to grok intuitively (small LLM > only as verbose as a fish, larger LL…
> the user is immediately able to understand the constraints Nagel's point was quite literally the opposite [1] of this, though. We can't understand what it must "be like to be a bat" because their mental model is so fundamentally different than ours. So using all the human language tokens in the world can't get us to truly understand what it's like to be a bat, or a guppy, or whatever. In fact, Nagel's point is argu…
IMO we're a step before that: We don't even have a real fish involved, we have a character that is fictionally a fish.
In LLM-discussions, obviously-fictional characters can be useful for this, like if someone builds a "Chat with Count Dracula" app. To truly believe that a typical "AI" is some entity that "wants to be helpful" is just as mistaken as believing the same architecture creates an entity that "feels the dark thirst for the blood of the living."
Or, in this case, that it really enjoys food-pellets.
Re: Show HN: I built a tiny LLM to demystify how language models work
#36Is this a reference from the Bobiverse?
Re: Show HN: I built a tiny LLM to demystify how language models work
#37I love these kinds of educational implementations. I want to really praise the (unintentional?) nod to Nagel, by limiting capabilities to representation of a fish, the user is immediately able to understand the constraints. It can only talk like a fish cause it’s very simple Especially compared to public models, thats a really simple correspondence to grok intuitively (small LLM > only as verbose as a fish, larger LL…
> the user is immediately able to understand the constraints Nagel's point was quite literally the opposite [1] of this, though. We can't understand what it must "be like to be a bat" because their mental model is so fundamentally different than ours. So using all the human language tokens in the world can't get us to truly understand what it's like to be a bat, or a guppy, or whatever. In fact, Nagel's point is argu…
Id highly disagree with that. Were all living in the same shared universe, and underlying every intelligence must be precisely an understanding of events happening in this space-time.
Re: Show HN: I built a tiny LLM to demystify how language models work
#38This is probably a consequence of the training data being fully lowercase: You> hello Guppy> hi. did you bring micro pellets. You> HELLO Guppy> i don't know what it means but it's mine.
Great find! It appears uppercase tokens are completely unknonw to the tokenizer.
But the character still comes through in response :)
Re: Show HN: I built a tiny LLM to demystify how language models work
#39I could fork it and create TrumpLM. Not a big leap, I suppose.
Re: Show HN: I built a tiny LLM to demystify how language models work
#40[flagged]