141–150 of 173 posts
I'm not sure how many people understand how much of a badass move this is. Andrej is helping apple and Facebook and more importantly the open source movement while also being paid really well by OpenAI(MSFT) But they are not going to push him out because he will go directly to Tesla or xai.
Earlier quoted context omitted.
Create a computer game about a small island with 100 people, with each person being politically aware, with llama2.c being their brain. Then you can simulate politics for a thousand years and see what happens. For instance.
Neat idea. Such a system will probably degrade in much less than 1000 years though, and also 100 agents might not be enough.
Never seen the word “inference” used as a verb.
Random thought: right now an LLM returns a probabilities distribution, an RNG sampler picks one and apoends it to the output, then the sequence repeats; but can the RNG instead pick N tokens that approximate the distribution, ask LLM to generate N new distributions, combine them somehow, then pick another set of N tokens from the combined dustribution?
Earlier quoted context omitted.
I did use a tweaked nanoGPT to pretrain a 12M model on TinyStories (2Gbytes produced by GPT4), and results are pretty amazing. I've adapted it a bit on Wikipedia then, and it looks like a solid bullshit generator, much smarter than any smoothed n-gram model, and significantly smaller. My bet small LLMs will be predominant in multiple areas. My next goal is to reduce 7B llama2 to 10-100M without making it much dumber.
>My next goal is to reduce 7B llama2 to 10-100M without making it much dumber. That is going to be hard as the 7B model was trained on 2T tokens. Maybe if you heavily restrict the range in which the model should operate.
2. Better than tokens is to train on probability distributions (distillation) and trees of probability distributions
Here's a Rust version in case anyone's curious what it would look like. It also clocks 106 tokens/second in release mode. https://github.com/garrisonhess/llama2.c/blob/517a1a3e487f31...
As often with Rust, someone transliterates something that already exists just because they can, without providing any benefit at all. Sometimes it even results in fragmenting the community efforts to improve the project.
Yay fun to see it make its way to HN :) It turns out that my original checkpoint runs _way_ faster than I expected (100 tok/s) on MacBook Air M1 with -O3 when compiling, so I am now training a bigger 44M model, which should still running interactively. Maybe the 7B Llama model is within reach... :thinking_emoji:
Do you think it's possible also to create a trainer in pure C, instead of using python?
Here's a Rust version in case anyone's curious what it would look like. It also clocks 106 tokens/second in release mode. https://github.com/garrisonhess/llama2.c/blob/517a1a3e487f31...
As often with Rust, someone transliterates something that already exists just because they can, without providing any benefit at all. Sometimes it even results in fragmenting the community efforts to improve the project.
Yay fun to see it make its way to HN :) It turns out that my original checkpoint runs _way_ faster than I expected (100 tok/s) on MacBook Air M1 with -O3 when compiling, so I am now training a bigger 44M model, which should still running interactively. Maybe the 7B Llama model is within reach... :thinking_emoji:
I did use a tweaked nanoGPT to pretrain a 12M model on TinyStories (2Gbytes produced by GPT4), and results are pretty amazing. I've adapted it a bit on Wikipedia then, and it looks like a solid bullshit generator, much smarter than any smoothed n-gram model, and significantly smaller. My bet small LLMs will be predominant in multiple areas. My next goal is to reduce 7B llama2 to 10-100M without making it much dumber.
Earlier quoted context omitted.
As often with Rust, someone transliterates something that already exists just because they can, without providing any benefit at all. Sometimes it even results in fragmenting the community efforts to improve the project.
Looks like you spoke too soon, I'm clocking 340+ tokens per second with my improved Rust implementation, compared to 106 with the original C. That being said, I didn't share this for any reason other than to share ideas and promote learning. Cheers
https://twitter.com/karpathy/status/1683301419716313089?s=20