Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM Visualization

bbycroft.net

Post reply on HN
Print view View original

21–30 of 138 posts

Re: LLM Visualization

#21

Post by dannyw »

cmiller1 wrote: post #13
Arson9416 wrote: post #11

Earlier quoted context omitted.

Isn't it amazing that a random person on the internet can produce free educational content that trumps university courses? With all the resources and expertise that universities have, why do they get shown up all the time? Do they just not know how to educate?

Have you considered the considerably greater breadth of content required for a full course, as well as the other responsibilities of the people teaching them such as testing, public speaking, etc.

It probably would be massively beneficial to society and progress if teaching professors could spend more time and attention on teaching.

Re: LLM Visualization

#22

Post by 29athrowaway »

I big kudos to the author of this.

Not only has the visualization, but it's interactive, has explanations for each item, has excellent performance and is open source: https://github.com/bbycroft/llm-viz/blob/main/src/llm

Another interesting visualization related thing: https://github.com/shap/shap

Re: LLM Visualization

#24

Post by tiffanyg »

syntaxers wrote: post #15
Arson9416 wrote: post #11

Earlier quoted context omitted.

Isn't it amazing that a random person on the internet can produce free educational content that trumps university courses? With all the resources and expertise that universities have, why do they get shown up all the time? Do they just not know how to educate?

It's an incentives problem. At research universities, promotion is contingent on research output, and teaching is often seen as a distraction. At so-called teaching universities, promotion (or even survival) is mainly contingent on throughput, and not measures of pedagogical outcome. If you are a teaching faculty at a university, it is against your own interests to invest time to develop novel teaching materials. The…

Unfortunately, this is a problem throughout our various economies, at this point.

Professionalization of management with its easy over-reliance on the simplification of the quantitative - of "metrics" - along with the scales (size) this allows and manner in which fundamental issues get obscured tends to produce these types of results. This is, of course, well known in business schools and efforts are generally made to ensure graduates are aware of some of the downsides of "the quantitative." Unsurprisingly, over time, there is a kind of "forcing" that tends to drive these systems towards the results like you describe.

It's usually the case that imposition of metrics, optimization, etc. - "mathematical methods" - is quite beneficial at first, but once systems are improved in sensible ways based on insights gained through this, less desirable behavior begins to occur. Multiple factors including basic human psychology factor into this ... which I think is getting beyond the scope of what's reasonable to include in this comment.

Re: LLM Visualization

#25

Post by baq »

Could as well be titled 'dissecting magic into matmuls and dot products for dummies'. Great stuff. Went away even more amazed that LLMs work as well as they do.

Re: LLM Visualization

#26

Post by namocat »

warkanlock wrote: post #10

This is an excellent tool to realize how an LLM actually works from the ground up! For those reading it and going through each step, if by chance you get stuck on why 48 elements are in the first array, please refer to the model.py on minGPT [1] It's an architectural decision that it will be great to mention in the article since people without too much context might lose it [1] https://github.com/karpathy/minGPT/blob…

Yes, thank you - It was unexplained, so I got stuck on "Why 48?", thinking I'd missing something right out of the gate.

Re: LLM Visualization

#27

Post by [deleted] »

reqo wrote: post #4

I feel like visualizations like this are what is missing from univeristy curricula. Now imagine a professor going trough each animation describing exactly what is happening, I am pretty sure students would get a much more in-depth understanding!

[deleted]

Re: LLM Visualization

#28

Post by [deleted] »

tsunamifury wrote: post #18
visarga wrote: post #16

Earlier quoted context omitted.

The deeper you go, the higher the order. It's what attention does at each layer, makes information circulate.

Thanks, I assumed that was the case, but they didn't make that explicit. Then the question is, is the world simulation running at the highest order attention layer or is it an emergent property of the interaction cycle between the attention layers.

[deleted]

Re: LLM Visualization

#30

Post by hmate9 »

This is a phenomenal visualisation. I wish I saw this when I was trying to wrap my head around transformers a while ago. This would have made it so much easier.
Post reply on HN