Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microgpt

karpathy.github.io

Post reply on HN
Print view View original

191–200 of 354 posts

Re: Microgpt

#191

Post by danielEM »

krige wrote: post #170
Dansvidania wrote: post #110

Earlier quoted context omitted.

If you see gaining fine motor control, understanding pictographic language […] as a prerequisite to driving a car, then yes, all of them are

That's an exaggeration. Nobody is trained to read STOP signs for 16 years, a few months top. And Waymo doesn't need to coordinate a four-limbed, 20-digited, one-headed body to operate a car.

Well, I also think that there is a lot that we process 'in background' and learn on beforehand in order to learn how to drive and then drive. I think the most 'fair' would be to figure out absolute lowest age of kids that would allow them to perform well on streets behind steering wheel.

Re: Microgpt

#192

Post by maipen »

znnajdla wrote: post #95

Super useful exercise. My gut tells me that someone will soon figure out how to build micro-LLMs for specialized tasks that have real-world value, and then training LLMs won’t just be for billion dollar companies. Imagine, for example, a hyper-focused model for a specific programming framework (e.g. Laravel, Django, NextJS) trained only on open-source repositories and documentation and carefully optimized with a spec…

That would only produce a model that you can ask questions to.

Re: Microgpt

#193

Post by otabdeveloper4 »

ricericerice wrote: post #189
mootothemax wrote: post #180

Earlier quoted context omitted.

> > BERT isn’t a SLM Huh? BERT is literally a language model that's small and uses attention. Astute readers will note what’s been missed here. Fascinating, really. Your confidently-statement yet factually void comments I’d have previously put down to one of the classic programmer mindsets. Nowadays though - where do I see that kind of thing most often? Curious.

After some research, I think I understand what you're getting at here - BERT being a model for encoding text but not architecturally feasible to generate text with it, which "LLMs" (the lack of definition here is resulting in you two talking past eachother), maybe more accurately referred to as GPTs, can do. Also the irony of your comment when it in itself was confidently stated yet void of any content was not missed…

BERT is one example of a language model that solved specific language tasks very well and that existed before LLM's.

We had very good language models for decades. The problem was they needed to be trained, which LLM's mostly don't. You can solve a language model problem now with just some system prompt manipulation.

(And honestly typing in system prompts by hand feels like a task that should definitely be automated. I'm waiting for "soft prompting" be become a thing so we can come full circle and just feed the LLM with an example set.)

Re: Microgpt

#195

Post by krisoft »

mootothemax wrote: post #180
otabdeveloper4 wrote: post #176

Earlier quoted context omitted.

> BERT isn’t a SLM Huh? BERT is literally a language model that's small and uses attention. And we had good language models before BERT too. They were a royal bitch to train properly, though. Nowadays you can get the same with just 30 minutes of prompt engineering.

> > BERT isn’t a SLM Huh? BERT is literally a language model that's small and uses attention. Astute readers will note what’s been missed here. Fascinating, really. Your confidently-statement yet factually void comments I’d have previously put down to one of the classic programmer mindsets. Nowadays though - where do I see that kind of thing most often? Curious.

> Astute readers will note what’s been missed here.

I’m not astute enough to see what was missed here. Could you explain?

Re: Microgpt

#196

Post by regularfry »

nurettin wrote: post #149
xdennis wrote: post #116

Earlier quoted context omitted.

> Train an LLM on all human knowledge up to 1905 and see if it comes up with General Relativity. It won’t. AGI just means human level intelligence. I couldn't come up with General Relativity. That doesn't mean I don't have general intelligence. I don't understand why people are moving the goalposts.

> AGI just means human level intelligence. It seems more like people haven't decided on what the goal post is. If AGI is just another human, that's pretty underwhelming. That's why people are imagining something that surpasses humans by heaps and bounds in terms of reasoning, leading to wondrous new discoveries.

"Just another human" would be outright astonishing if it landed.

Re: Microgpt

#197

Post by geon »

with wrote: post #103

"everything else is just efficiency" is a nice line but the efficiency is the hard part. the core of a search engine is also trivial, rank documents by relevance. google's moat was making it work at scale. same applies here.

I think the hard part is improving on the basic concept.

The current top of the line models are extremely overfitted and produce so much nonsense they are useless for anything but the most simple tasks.

This architecture was an interesting experiment, but is not the future.

Re: Microgpt

#198

Post by pmarreck »

hei-lima wrote: post #177
subset wrote: post #57

I had good fun transliterating it to Rust as a learning experience ( https://github.com/stochastical/microgpt-rs ). The trickiest part was working out how to represent the autograd graph data structure with Rust types. I'm finalising some small tweaks to make it run in the browser via WebAssmebly and then compile it up for my blog :) Andrej's code is really quite poetic, I love how much it packs into such a concise p…

Great work! Might do it too in some other language...

Zig, here.

Anything but Python

Re: Microgpt

#199

Post by ruszki »

WithinReason wrote: post #138
ruszki wrote: post #125

> [p for mat in state_dict.values() for row in mat for p in row] I'm so happy without seeing Python list comprehensions nowadays. I don't know why they couldn't go with something like this: [state_dict.values() for mat for row for p] or in more difficult cases [state_dict.values() for mat to mat*2 for row for p to p/2] I know, I know, different times, but still.

I would have gone for: [for p in row in mat in state_dict.values()]

That’s also an option. The left to right flow is better for the sake of autocomplete and comprehension: when you start to read your right to left version, you don’t know what is p, then row, then mat. With left to right, this problem doesn’t exist.

One for sure, both are superior to the garbled mess of Python’s.

Of course if the programming language would be in a right to left natural language, then these are reversed.

Re: Microgpt

#200

Post by joefourier »

ricericerice wrote: post #189
mootothemax wrote: post #180

Earlier quoted context omitted.

> > BERT isn’t a SLM Huh? BERT is literally a language model that's small and uses attention. Astute readers will note what’s been missed here. Fascinating, really. Your confidently-statement yet factually void comments I’d have previously put down to one of the classic programmer mindsets. Nowadays though - where do I see that kind of thing most often? Curious.

After some research, I think I understand what you're getting at here - BERT being a model for encoding text but not architecturally feasible to generate text with it, which "LLMs" (the lack of definition here is resulting in you two talking past eachother), maybe more accurately referred to as GPTs, can do. Also the irony of your comment when it in itself was confidently stated yet void of any content was not missed…

You can actually generate surprisingly coherent text with minimal finetuning of BERT, by reinterpreting it as a diffusion model: https://nathan.rs/posts/roberta-diffusion/

I don’t see a useful definition of LLM that doesn’t include BERT, especially given its historical importance. 340M parameters is only “small” in the sense that a baby whale is small.

Post reply on HN