91–100 of 326 posts
Nice synergy here, the lineage is: Karpathy's nano-GPT -> Keller Jordan's modded-nanoGPT (a speedrun of training nanoGPT) -> NanoChat
modded-nanoGPT [1] is a great project, well worth checking out, it's all about massively speeding up the training of a small GPT model.
Notably it uses the author's Muon optimizer [2], rather than AdamW, (for the linear layers).
Nice! His Shakespeare generator was one of the first projects I tried after ollama. The goal was to understand what LLMs were about. I have been on an LLM binge this last week or so trying to build a from-scratch training and inference system with two back ends: - CPU (backed by JAX) - GPU (backed by wgpu-py). This is critical for me as I am unwilling to deal with the nonsense that is rocm/pytorch. Vulkan works for m…
Earlier quoted context omitted.
You could do that indeed, but the performance would be abysmal. For this kind of use-case, it would be a LOT better to use a small pre-trained model and either fine-tune it on your materials, or use some kind of RAG workflow (possibly both).
> it would be a LOT better to use a small pre-trained model and either fine-tune it on your materials, or use some kind of RAG workflow (possibly both). I noticed NewRelic has a chat feature that does this sort of thing, it's scoped very narrowly down to their website and analytics DSL language, and generates charts/data from their db. I've always wondered how they did that (specifically in terms of set up the traini…
The most likely way of building that would be to equip it with a "search_docs" tool that lets it look up relevant information for your query. No need to train an extra model at all if you do that.
This is an LLM trained using a $100 budget to RENT access to graphics cards. It's not about what you could do BUYING hardware for $100.
Earlier quoted context omitted.
There's a Stephen Boyd quote that's something like "if your optimization problem is too computationally expensive, just go on vacation to Greece for a few weeks and by the time you get back, computers might be fast enough to solve it." With LLMs there's sort of an equivalent situation with cost: how mindblowing would it be able to train this kind of LLM at all even just 4 years ago? And today you can get a kindergart…
> today you can get a kindergartener level chat model for about $100. Not hard to imagine the same model costing $10 of compute in a few years. No, it's extremely hard to imagine since I used one of Karpathy's own models to have a basic chat bot like six years ago. Yes, it spoke nonsense; so did my GPT-2 fine tune four years ago and so does this. And so does ChatGPT Improvement is linear at best. I still think it's a…
Nice! His Shakespeare generator was one of the first projects I tried after ollama. The goal was to understand what LLMs were about. I have been on an LLM binge this last week or so trying to build a from-scratch training and inference system with two back ends: - CPU (backed by JAX) - GPU (backed by wgpu-py). This is critical for me as I am unwilling to deal with the nonsense that is rocm/pytorch. Vulkan works for m…
Unwilling to deal with pytorch? You couldn't possibly hobble yourself anymore if you tried.
My use case is different. I want something that I can run quickly on one GPU without worrying about whether it is supported or not.
I am interested in convenience, not in squeezing out the last bit of performance from a card.
> nanochat is also inspired by modded-nanoGPT Nice synergy here, the lineage is: Karpathy's nano-GPT -> Keller Jordan's modded-nanoGPT (a speedrun of training nanoGPT) -> NanoChat modded-nanoGPT [1] is a great project, well worth checking out, it's all about massively speeding up the training of a small GPT model. Notably it uses the author's Muon optimizer [2], rather than AdamW, (for the linear layers). [1] https:/…