Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NanoChat – The best ChatGPT that $100 can buy

github.com

Post reply on HN
Print view View original

121–130 of 326 posts

Re: NanoChat – The best ChatGPT that $100 can buy

#121

Post by gyomu »

tehnub wrote: post #107

Interesting exchange on the use of AI coding tools: curious how much did you write the code by hand of it? Karpathy: Good question, it's basically entirely hand-written (with tab autocomplete). I tried to use claude/codex agents a few times but they just didn't work well enough at all and net unhelpful, possibly the repo is too far off the data distribution. https://x.com/karpathy/status/1977758204139331904

> the repo is too far off the data distribution

ah, this explains why these models have been useless to me this whole time. everything i do is just too far off the data distribution!

Re: NanoChat – The best ChatGPT that $100 can buy

#122

Post by simonw »

oblio wrote: post #120

I wonder, if something like this were trained on Wikipedia, could it become a reliable local Wikipedia search engine, basically?

I don't think so. Training on documents is not a great way of building a search engine for those for the information in those documents, because the training process mixes all of that information together in ways that detach the individual words from the source documents they came from.

As usual, if you want an LLM to be able to help search a corpus of text the best way to achieve that is to teach it how to use a search tool against that text.

Re: NanoChat – The best ChatGPT that $100 can buy

#123

Post by arkmm »

dabockster wrote: post #110

The title is extremely misleading - you have to rent time on an H100 cluster to get it to work. It is not on-device, and thus not truly $100. I was really excited, too, until I looked through the readme files and the code.

What's misleading about that? You rent $100 of time on an H100 to train the model.

Re: NanoChat – The best ChatGPT that $100 can buy

#124

Post by typpilol »

Lerc wrote: post #101
sammyd56 wrote: post #58

I'm doing a training run right now (started 20min ago). You can follow it at https://api.wandb.ai/links/sjd333-none/dsv4zkij Will share the resulting model once ready (4 hours from now) for anyone to test inference.

The comment beside the first chart >Our main measure of progress. Bits per byte is, per Karpathy, "a much better measure than just the typical cross-entropy loss, because it further normalizes the loss on each token by the number of bytes of that token, making the metric tokenizer-invariant". Is so blindingly obvious, that I'm ashamed to think that I didn't think do it when trialing my own tokenizer approach on tinys…

Why hasn't anyone made a tokenizer that's 1 character per token. Is it because it requires an insane amount of compute?

Or would the loss of efficiency make it dumber then modern tokenizers?

Re: NanoChat – The best ChatGPT that $100 can buy

#125

Post by dude250711 »

tehnub wrote: post #107

Interesting exchange on the use of AI coding tools: curious how much did you write the code by hand of it? Karpathy: Good question, it's basically entirely hand-written (with tab autocomplete). I tried to use claude/codex agents a few times but they just didn't work well enough at all and net unhelpful, possibly the repo is too far off the data distribution. https://x.com/karpathy/status/1977758204139331904

How convenient! You know, my code is somewhat far off the data distribution too.

Re: NanoChat – The best ChatGPT that $100 can buy

#126

Post by CamperBob2 »

contingencies wrote: post #86
CamperBob2 wrote: post #80

Earlier quoted context omitted.

If you can't afford $100 or learn how to train it locally with more time and less money, then this isn't something you should be focusing on at all.

It is amusing to note the dichotomy between the clearly compassionate, empathetic and altruistic perspective displayed here and the comically overstated framing of helping humanity.

(Shrug) Other sites beckon.

Re: NanoChat – The best ChatGPT that $100 can buy

#127

Post by danielmarkbruce »

sieve wrote: post #97
danielmarkbruce wrote: post #93

Earlier quoted context omitted.

Unwilling to deal with pytorch? You couldn't possibly hobble yourself anymore if you tried.

If you want to train/sample large models, then use what the rest of the industry uses. My use case is different. I want something that I can run quickly on one GPU without worrying about whether it is supported or not. I am interested in convenience, not in squeezing out the last bit of performance from a card.

You wildly misunderstand pytorch.

Re: NanoChat – The best ChatGPT that $100 can buy

#128

Post by SeanAnderson »

tehnub wrote: post #107

Interesting exchange on the use of AI coding tools: curious how much did you write the code by hand of it? Karpathy: Good question, it's basically entirely hand-written (with tab autocomplete). I tried to use claude/codex agents a few times but they just didn't work well enough at all and net unhelpful, possibly the repo is too far off the data distribution. https://x.com/karpathy/status/1977758204139331904

This makes sense, right? It's a relatively novel thing to be writing. I don't find it to be a damning remark like other comments here seem to be concluding.

If anything, the fact that Karpathy reached towards Claude/Codex in an attempt to gain value is indicative that, in previous coding efforts, those tools were helpful to him.

Re: NanoChat – The best ChatGPT that $100 can buy

#129

Post by SeanAnderson »

Lerc wrote: post #101
sammyd56 wrote: post #58

I'm doing a training run right now (started 20min ago). You can follow it at https://api.wandb.ai/links/sjd333-none/dsv4zkij Will share the resulting model once ready (4 hours from now) for anyone to test inference.

The comment beside the first chart >Our main measure of progress. Bits per byte is, per Karpathy, "a much better measure than just the typical cross-entropy loss, because it further normalizes the loss on each token by the number of bytes of that token, making the metric tokenizer-invariant". Is so blindingly obvious, that I'm ashamed to think that I didn't think do it when trialing my own tokenizer approach on tinys…

ELI5 for anyone else (I had to have this explained to me):

When you train a language model, it tries to predict the next token.

We measure how good it is at that using loss aka how surprised it was by the real answer.

Different models might use different token lengths. So, if you describe loss relative to tokens then you can't easily compare the performance of two models that use different token lengths.

So, compare loss to bytes of text data instead.

Post reply on HN