Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microgpt

karpathy.github.io

Post reply on HN
Print view View original

321–330 of 354 posts

Re: Microgpt

#321

Post by mym1990 »

O4epegb wrote: post #289
growingswe wrote: post #98

Great stuff! I wrote an interactive blogpost that walks through the code and visualizes it: https://growingswe.com/blog/microgpt

> By the end of training, the model produces names like "kamon", "karai", "anna", and "anton". None of them are copies from the dataset. All 4 are in the dataset, btw

This is likely because the blog is AI generated and keys off this point from Karpathy: "As a preview, by the end of the script our model will generate (“hallucinate”!) new, plausible-sounding names.", so the LLM just repackaged that into something that is obviously wrong, which is kind of ironic.

Re: Microgpt

#322

Post by hu3 »

WithinReason wrote: post #139
verma7 wrote: post #91

I wrote a C++ translation of it: https://github.com/verma7/microgpt/blob/main/microgpt.cc 2x the number of lines of code (~400L), 10x the speed The hard part was figuring out how to represent the Value class in C++ (ended up using shared_ptrs).

I made an explicit reverse pass (no autodiff), it was 8x faster in Python

I made an explicit double-reverse pass (no code!), it was 80x faster in my head!

Re: Microgpt

#323

Post by bear3r »

WithinReason wrote: post #139
verma7 wrote: post #91

I wrote a C++ translation of it: https://github.com/verma7/microgpt/blob/main/microgpt.cc 2x the number of lines of code (~400L), 10x the speed The hard part was figuring out how to represent the Value class in C++ (ended up using shared_ptrs).

I made an explicit reverse pass (no autodiff), it was 8x faster in Python

tradeoff worth naming: you avoid the autodiff graph overhead (hence the speedup), but any architecture change means rewriting every gradient by hand. fine for a pedagogical project, but that's exactly why autodiff exists.

Re: Microgpt

#324

Post by big-chungus4 »

love2read wrote: post #309
teleforce wrote: post #94

Someone has modified microgpt to build a tiny GPT that generates Korean first names, and created a web page that visualizes the entire process [1]. Users can interactively explore the microgpt pipeline end to end, from tokenization until inference. [1] English GPT lab: https://ko-microgpt.vercel.app/

By "modified" this person of course means that they swapped out the list of X0,000 names from English to Korean names. That is seemingly the only change. The attached website is a fully ai-generated "visualization" based on the original blog post with little added.

It's a good website and probably AI generated with some insane expensive model that us mere mortals are too poor to afford, thus it has a value

Re: Microgpt

#325

Post by lynx97 »

Question: Can this be modified to score a "document"? I'd basically like to pass it a name, and get a score (0..1) on how realistic themodel "thinks" the document is? This would be extremely helpful for a project of mine.

Re: Microgpt

#326

Post by jaen »

red75prime wrote: post #320
jaen wrote: post #234

Earlier quoted context omitted.

Uh, to explain what? You probably read something into what I said while I was being very literal. If you train an LLM on mostly false statements, it will generate both known and novel falsehoods. Same for truth. An LLM has no intrinsic concept of true or false, everything is a function of the training set. It just generates statements similar to what it has seen and higher-dimensional analogies of those .

Reasoning allows to produce statements that are more likely to be true based on statements that are known to be true. You'd need to structure your "falsehood training data" in a specific way to allow an LLM to generalize as well as with the regular data (instead of memorizing noise). And then you'll get a reasoning model which remembers false premises. You generate your text based on a "stochastic parrot" hypothesis…

Really, how hard is it to follow HN guidelines and :

a) not imagine straw-man arguments and not imagine more (or less) than what was said

b) refrain from snarky and false ad hominems

None of what you said in no way conflicts with what I said, and again shows a fundamental misunderstanding.

Reasoning is (mostly) part of the post-training dataset. If you add a large majority of false (ie. paradoxical, irrational etc.) reasoning traces to those, you'll get a model that successfully replicates the false reasoning of humans. If you mix it in with true reasoning traces, I imagine you'll get infinite loop behaviour as the reasoning trace oscillates between the true and the false.

The original premise that truth is purely a function of the training dataset still stands... I'm not even sure what people are arguing here, as that seems quite trivially obvious?

Re: Microgpt

#327

Post by Otterly99 »

geokon wrote: post #141

> What’s the deal with “hallucinations”? The model generates tokens by sampling from a probability distribution. It has no concept of truth, it only knows what sequences are statistically plausible given the training data. Extremely naiive question.. but could LLM output be tagged with some kind of confidence score? Like if I'm asking an LLM some question does it have an internal metric for how confident it is in its…

There is this paper that proposed data compression as a way to judge the ability of a LLM to "understand" things correctly, training on older texts and trying to predict more recent articles:

https://ar5iv.labs.arxiv.org/html//2402.00861

Re: Microgpt

#328

Post by Otterly99 »

krisoft wrote: post #195
mootothemax wrote: post #180

Earlier quoted context omitted.

> > BERT isn’t a SLM Huh? BERT is literally a language model that's small and uses attention. Astute readers will note what’s been missed here. Fascinating, really. Your confidently-statement yet factually void comments I’d have previously put down to one of the classic programmer mindsets. Nowadays though - where do I see that kind of thing most often? Curious.

> Astute readers will note what’s been missed here. I’m not astute enough to see what was missed here. Could you explain?

If I'm not mistaken, BERT is a classifier (enters text, outputs labels) so it is not a "Language model", as it cannot be used for text generation.

Re: Microgpt

#330

Post by sahildeepreel »

teleforce wrote: post #94

Someone has modified microgpt to build a tiny GPT that generates Korean first names, and created a web page that visualizes the entire process [1]. Users can interactively explore the microgpt pipeline end to end, from tokenization until inference. [1] English GPT lab: https://ko-microgpt.vercel.app/

so impressive!
Post reply on HN