241–250 of 326 posts
I'm doing a training run right now (started 20min ago). You can follow it at https://api.wandb.ai/links/sjd333-none/dsv4zkij Will share the resulting model once ready (4 hours from now) for anyone to test inference.
The measures that drop exponentially like val/bpb and train/loss you should put the x-axis in log-scale. That will better show you if it's converged
Earlier quoted context omitted.
Everything is unless your app is a React todolist or leatcode questions.
HN's cynicism towards AI coding (and everything else ever) is exhausting. Karpathy would probably cringe reading this.
Earlier quoted context omitted.
It will work great with 40GB GPU, probably a bit less than twice slower. These are micro models of a few B param at most and fit easily during both training and inference.
How low can this go? Can this run on a 5090 card (32GiB)?
Earlier quoted context omitted.
I got your model working on CPU on macOS by having Claude Code hack away furiously for a while. Here's a script that should work for anyone: https://gist.github.com/simonw/912623bf00d6c13cc0211508969a1... You can run it like this: cd /tmp git clone https://huggingface.co/sdobson/nanochat uv run https://gist.githubusercontent.com/simonw/912623bf00d6c13cc0211508969a100a/raw/80f79c6a6f1e1b5d4485368ef3ddafa5ce853131/gene…
For anyone curious this is the error when running uv sync on macos, > uv sync Resolved 88 packages in 3ms error: Distribution `torch==2.8.0+cu128 @ registry+ https://download.pytorch.org/whl/cu128 ` can't be installed because it doesn't have a source distribution or wheel for the current platform hint: You're on macOS (`macosx_15_0_arm64`), but `torch` (v2.8.0+cu128) only has wheels for the following platforms: `many…
Earlier quoted context omitted.
The original context of this sub-thread was Karpathy saying how AI coding tools were pretty useless for him when working on this particular project.
Indeed. And only Karpathy is entitled to say that AI tools produce wrong code for him. And he's only entitled to say it for this project only. If anyone else says this, "the skepticism is exhausting", and their experience is completely irrelevant.
The grievance attitude seems to exist in both directions and is actually what is exhausting.
Earlier quoted context omitted.
Back in the 90s you could drag and drop a vb6 applet in Microsoft word. Somehow we’ve regressed.. Edit: for the young, wysiwyg (what you see is what you get) was common for all sorts of languages from c++ to Delphi to html. You could draw up anything you wanted. Many had native bindings to data sources of all kinds. My favourite was actually HyperCard because I learned it in grade school.
Wysiwyg kind of fell apart once we had to stop assuming everyone had an 800x600 or 1024x768 screen, because what you saw was no longer what others got.
Earlier quoted context omitted.
> the repo is too far off the data distribution ah, this explains why these models have been useless to me this whole time. everything i do is just too far off the data distribution!
I work on this typed lua language in lua, and sometimes use llms to help fix internal analyzer stuff, which works 30% of the time for complex, and sometimes not at all, but helps me find a solution in the end. However when I ask an llm to generate my typed lua code, with examples and all, on how the syntax is supposed to be, it mostly gets it wrong. my syntax for tables/objects is: local x: {foo = boolean} but an llm…
Earlier quoted context omitted.
Everything is unless your app is a React todolist or leatcode questions.
people say this like it's a criticism, but damn is it ever nice to start writing a simple crud form and just have copilot autocomplete the whole thing for me.
Interesting exchange on the use of AI coding tools: curious how much did you write the code by hand of it? Karpathy: Good question, it's basically entirely hand-written (with tab autocomplete). I tried to use claude/codex agents a few times but they just didn't work well enough at all and net unhelpful, possibly the repo is too far off the data distribution. https://x.com/karpathy/status/1977758204139331904