Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I built a tiny LLM to demystify how language models work

github.com

Post reply on HN
Print view View original

111–120 of 174 posts

Re: Show HN: I built a tiny LLM to demystify how language models work

#111

Post by thomasfl »

Is there some documentation for this? The code is probably the simplest (Not So) Large Language Model implementation possible, but it is not straight forward to understand for developers not familiar with multi-head attention, ReLU FFN, LayerNorm and learned positional embeddings.

This projects shares similarities with Minix. Minix is still used at universities as an educational tool for teaching operating system design. Minix is the operating system that taught Linus Torvalds how to design (monolithic) operating systems. Similarly having students adding capabilities to GuppyLM is a good way to learn LLM design.

Re: Show HN: I built a tiny LLM to demystify how language models work

#114

Post by BrokenCogs »

tantalor wrote: post #108
BrokenCogs wrote: post #104

Earlier quoted context omitted.

Who cares how it compares, it's not a product it's a cool project

Even cool projects can learn from others. Maybe they missed something that could benefit the project, or made some interesting technical choice that gives a different result. For the readers/learners, it's useful to understand the differences so we know what details matter, and which are just stylistic choices. This isn't art; it's science & engineering.

But it isn't the OP's responsibility to compare their project to all other projects. The GP could themselves perform the comparison and post their thoughts instead of asking an open ended question.

Re: Show HN: I built a tiny LLM to demystify how language models work

#117

Post by achenatx »

thomasfl wrote: post #111

Is there some documentation for this? The code is probably the simplest (Not So) Large Language Model implementation possible, but it is not straight forward to understand for developers not familiar with multi-head attention, ReLU FFN, LayerNorm and learned positional embeddings. This projects shares similarities with Minix. Minix is still used at universities as an educational tool for teaching operating system des…

give the code to an LLM and have a discussion about it.

Re: Show HN: I built a tiny LLM to demystify how language models work

#118

Post by tatrions »

bblb wrote: post #53

Could it be possible to train LLM only through the chat messages without any other data or input? If Guppy doesn't know regular expressions yet, could I teach it to it just by conversation? It's a fish so it wouldn't probably understand much about my blabbing, but would be interesting to give it a try. Or is there some hard architectural limit in the current LLM's, that the training needs to be done offline and with…

[flagged]

Re: Show HN: I built a tiny LLM to demystify how language models work

#120

Post by tantalor »

BrokenCogs wrote: post #114
tantalor wrote: post #108

Earlier quoted context omitted.

Even cool projects can learn from others. Maybe they missed something that could benefit the project, or made some interesting technical choice that gives a different result. For the readers/learners, it's useful to understand the differences so we know what details matter, and which are just stylistic choices. This isn't art; it's science & engineering.

But it isn't the OP's responsibility to compare their project to all other projects. The GP could themselves perform the comparison and post their thoughts instead of asking an open ended question.

100% agree, I didn't mean to imply that OP is responsible for that, or that the (lack of) comparison detracts in any way from the work.
Post reply on HN