Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NanoChat – The best ChatGPT that $100 can buy

github.com

Post reply on HN
Print view View original

161–170 of 326 posts

Re: NanoChat – The best ChatGPT that $100 can buy

#161

Post by simonw »

hn_acc1 wrote: post #152
simonw wrote: post #78

Earlier quoted context omitted.

"With what is called AI today, only the largest corporations can afford to train the models" I take it you're very positive about Andrej's new project which allows anyone to train a model for a few hundred dollars which is comparable to the state-of-the-art from just 5 years ago then.

For a few hundred dollars, given heavily-VC-subsidized hardware that is probably partially funded by nvidia and various AI companies, etc. Can I run it on my local hardware (nvidia consumer card, AMD cpu)? No. When could that corporation cut off my access to that hardware if I did anything it didn't like? Anytime. Lots of things have started off cheap / subsidized to put competitors out of business, and then the pric…

> Can I run it on my local hardware?

Yes. The training process requires big expensive GPUs. The model it produces has 561M parameters, which should run on even a high end mobile phone (I run 4B models on my iPhone).

Re: NanoChat – The best ChatGPT that $100 can buy

#162

Post by throwaway0123_5 »

bringmeiron wrote: post #159
tehnub wrote: post #107

Interesting exchange on the use of AI coding tools: curious how much did you write the code by hand of it? Karpathy: Good question, it's basically entirely hand-written (with tab autocomplete). I tried to use claude/codex agents a few times but they just didn't work well enough at all and net unhelpful, possibly the repo is too far off the data distribution. https://x.com/karpathy/status/1977758204139331904

Clearly he has little idea what he's talking about. AI can write better code than 99% of developers. This embarrassingly anti-AI shill included. If he used the AI tool my company is developing the code would have been better and shipped sooner.

Anti-AI shill? A cofounder of OpenAI?

Re: NanoChat – The best ChatGPT that $100 can buy

#163

Post by sieve »

danielmarkbruce wrote: post #155
sieve wrote: post #146

Earlier quoted context omitted.

What is there to misunderstand? It doesn't even install properly most of the time on my machine. You have to use a specific python version. I gave up on all tools that depend on it for inference. llama-cpp compiles cleanly on my system for Vulkan. I want the same simplicity to test model training.

pytorch is as easy as you are going to find for your exact use case. If you can't handle the requirement of a specific version of python, you are going to struggle in software land. ChatGPT can show you the way.

I have been doing this for 25 years and no longer have the patience to deal with stuff like this. I am never going to install Arch from scratch by building the configuration by hand ever again. The same with pytorch and rocm.

Getting them to work and recognize my GPU without passing arcane flags was a problem. I could at least avoid the pain with llama-cpp because of its vulkan support. pytorch apparently doesn't have a vulkan backend. So I decided to roll out my own wgpu-py one.

Re: NanoChat – The best ChatGPT that $100 can buy

#164

Post by mynameisjoseph »

dabockster wrote: post #110

The title is extremely misleading - you have to rent time on an H100 cluster to get it to work. It is not on-device, and thus not truly $100. I was really excited, too, until I looked through the readme files and the code.

I feel same. The title looks like I could have on-deivce ChatGPT with $100 forever. I couldn't imagine it's about training the model by myself.

Re: NanoChat – The best ChatGPT that $100 can buy

#165

Post by simonw »

victor106 wrote: post #158
simonw wrote: post #122

Earlier quoted context omitted.

I don't think so. Training on documents is not a great way of building a search engine for those for the information in those documents, because the training process mixes all of that information together in ways that detach the individual words from the source documents they came from. As usual, if you want an LLM to be able to help search a corpus of text the best way to achieve that is to teach it how to use a sea…

> the best way to achieve that is to teach it how to use a search tool against that text. Any examples of this?

I've seen this called "agentic RAG" by some people. The easiest way to get a local demo is with Claude Code or Codex CLI. They know how to use grep, and you can set them loose on a folder full of text files and tell them to use grep to answer questions - it can work really well.

I just tried this in "claude --dangerously-skip-permissions":

> Use Python and AppleScript to find Apple Notes that mention UPS

... and fell down a rabbit hole of optimizations because my Notes collection is HUGE, but it got there in the end!

Re: NanoChat – The best ChatGPT that $100 can buy

#166

Post by danielmarkbruce »

sieve wrote: post #163
danielmarkbruce wrote: post #155

Earlier quoted context omitted.

pytorch is as easy as you are going to find for your exact use case. If you can't handle the requirement of a specific version of python, you are going to struggle in software land. ChatGPT can show you the way.

I have been doing this for 25 years and no longer have the patience to deal with stuff like this. I am never going to install Arch from scratch by building the configuration by hand ever again. The same with pytorch and rocm. Getting them to work and recognize my GPU without passing arcane flags was a problem. I could at least avoid the pain with llama-cpp because of its vulkan support. pytorch apparently doesn't hav…

Fair enough I guess. I think you'll find the relatively minor headache worth it. Pytorch brings a lot to the table.

Re: NanoChat – The best ChatGPT that $100 can buy

#167

Post by simonw »

mynameisjoseph wrote: post #164
dabockster wrote: post #110

The title is extremely misleading - you have to rent time on an H100 cluster to get it to work. It is not on-device, and thus not truly $100. I was really excited, too, until I looked through the readme files and the code.

I feel same. The title looks like I could have on-deivce ChatGPT with $100 forever. I couldn't imagine it's about training the model by myself.

Since the resulting model is only ~561M parameters you could run it on a Raspberry Pi that costs less than $100.

Re: NanoChat – The best ChatGPT that $100 can buy

#169

Post by bringmeiron »

throwaway0123_5 wrote: post #162
bringmeiron wrote: post #159

Earlier quoted context omitted.

Clearly he has little idea what he's talking about. AI can write better code than 99% of developers. This embarrassingly anti-AI shill included. If he used the AI tool my company is developing the code would have been better and shipped sooner.

Anti-AI shill? A cofounder of OpenAI?

You have found the joke.

Re: NanoChat – The best ChatGPT that $100 can buy

#170

Post by sammyd56 »

sammyd56 wrote: post #58

I'm doing a training run right now (started 20min ago). You can follow it at https://api.wandb.ai/links/sjd333-none/dsv4zkij Will share the resulting model once ready (4 hours from now) for anyone to test inference.

I've uploaded the model here: https://huggingface.co/sdobson/nanochat

I didn't get as good results as Karpathy (unlucky seed?)

It's fun to play with though...

User: How many legs does a dog have? Assistant: That's a great question that has been debated by dog enthusiasts for centuries. There's no one "right" answer (...)

Post reply on HN