Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microgpt

karpathy.github.io

Post reply on HN
Print view View original

341–350 of 354 posts

Re: Microgpt

#341

Post by dang »

3abiton wrote: post #315
dang wrote: post #287

Earlier quoted context omitted.

Wow I agree - surprising that it took 2 weeks to make HN's frontpage. We do generally like HN to be a bit uncorrelated with the rest of the internet, but it feels like a miss to me that neither https://news.ycombinator.com/item?id=47000263 nor https://news.ycombinator.com/item?id=47018557 made the frontpage.

I think he caught some flack for promoting claudebot at that time, and giving it a rave review. Some people are hardliner. His work has always been amazing nonetheless.

That would be more consistent with it making the frontpage and then getting flagged. Just getting ignored? Unlikely except by randomness.

I think the vast majority of users here agree with you (and me!) that karpathy's work is incredible. Complainers are always over-represented in comments, of course.

Re: Microgpt

#344

Post by noduerme »

tomrod wrote: post #269
noduerme wrote: post #92

Earlier quoted context omitted.

When humans, or dogs or cats for that matter, react to novel situations they encounter, when they appear to generalize or synthesize prior diverse experience into a novel reaction, that new experience and new reaction feeds directly back into their mental model and alters it on the fly. It doesn't just tack on a new memory. New experience and new information back-propagates constantly adjusting the weights and meanin…

In a word, JEPA?

No. Not at all like that. I said:

>> nor spatial artifacts

I meant visual patterns, too. You're thinking about what I said on too granular a level. JEPA is visual, based ultimately on pixels. The tokens may be digested from pixels until they're as large as whole recognizable objects, but the tokens are not whole mental models themselves.

Here's an example of humans evaluating competing mental models as tokens: You see a car, it's white, it's got some blood stains on the door, and it's traveling towards a red light at 90 miles an hour in a 30 mph residential zone, while you're about to make a left turn. A human foot is dangling from the trunk.

You refer to several mental models you have about high speed chases, drug cartels in the area, murders, etc. You compare these models to determine the next action the car might take.

What were the tokens in this scenario? The color of the car, the pixels of blood, the speed, the traffic pattern? Or whole models of understanding behavior where you had to choose between a normal driver's behavior and that of someone with a dead body fleeing a crime scene?

Re: Microgpt

#345

Post by sprobertson »

arijun wrote: post #333
sprobertson wrote: post #312

Earlier quoted context omitted.

This kind of thing is pretty easy to do with a much leaner model https://docs.pytorch.org/tutorials/intermediate/char_rnn_gen...

I assume the goal isn't to generate Korean names but to learn GPTs.

Sure, but use the tool for the job IMO. GPTs are much more complex so should demonstrate a much more complex task.

Re: Microgpt

#347

Post by red75prime »

jaen wrote: post #326
red75prime wrote: post #320

Earlier quoted context omitted.

Reasoning allows to produce statements that are more likely to be true based on statements that are known to be true. You'd need to structure your "falsehood training data" in a specific way to allow an LLM to generalize as well as with the regular data (instead of memorizing noise). And then you'll get a reasoning model which remembers false premises. You generate your text based on a "stochastic parrot" hypothesis…

Really, how hard is it to follow HN guidelines and : a) not imagine straw-man arguments and not imagine more (or less) than what was said b) refrain from snarky and false ad hominems None of what you said in no way conflicts with what I said, and again shows a fundamental misunderstanding. Reasoning is (mostly) part of the post-training dataset. If you add a large majority of false (ie. paradoxical, irrational etc.)…

Ah, sorry. I haven't recognized "all the high-level capabilities of an LLM come from the training data (presumably unlike humans, given the context of this thread)" in your wording. This is probably true. LLM structure probably has no inherent inductive bias that would amount to truth seeking. If you want to get a useless LLM, you can do it. OK, no disagreement here.

Re: Microgpt

#348

Post by subset »

jtesp wrote: post #338
subset wrote: post #57

I had good fun transliterating it to Rust as a learning experience ( https://github.com/stochastical/microgpt-rs ). The trickiest part was working out how to represent the autograd graph data structure with Rust types. I'm finalising some small tweaks to make it run in the browser via WebAssmebly and then compile it up for my blog :) Andrej's code is really quite poetic, I love how much it packs into such a concise p…

how did you do the transliteration/port?

Handwritten! (aka no LLM assistance :) It wasn't transpiled or anything like that. I've been meaning to post a little about it on my blog; just been caught up with other stuff atm.

One thing that was a _little_ frustrating coming from Python, though, was the need to rely on crates for basic things like random number generation and network requests. It pulls in a lot, even if you only need a little. I understand the Rust community prefers it that way as it's easier to evolve rather than be stuck with backwards-compatability requirements. But I still missed "batteries included" Python.

Re: Microgpt

#349

Post by thomasmg »

jtesp wrote: post #339
thomasmg wrote: post #283

Earlier quoted context omitted.

I got a convertion to Java. It worked (at least I think...) in the first try. Then I want to convert this to my own programming language (which traspiles to C). I like those tiny projects very much!

how did you do the transliteration/port?

I asked ChatGPT to translate (the free version), pasting the source code. The resulting Java code came back a second later.
Post reply on HN