RustGPT: A pure-Rust transformer LLM built from scratch
1–10 of 186 posts
Re: RustGPT: A pure-Rust transformer LLM built from scratch
#2> ndarray = "0.16.1" rand = "0.9.0" rand_distr = "0.5.0"
Looking good!
Re: RustGPT: A pure-Rust transformer LLM built from scratch
#3Absolutely love how readable the entire project is
Re: RustGPT: A pure-Rust transformer LLM built from scratch
#4> ndarray = "0.16.1" rand = "0.9.0" rand_distr = "0.5.0" Looking good!
I was slightly curious: cargo tree
llm v0.1.0 (RustGPT)
├── ndarray v0.16.1
│ ├── matrixmultiply v0.3.9
│ │ └── rawpointer v0.2.1
│ │ [build-dependencies]
│ │ └── autocfg v1.4.0
│ ├── num-complex v0.4.6
│ │ └── num-traits v0.2.19
│ │ └── libm v0.2.15
│ │ [build-dependencies]
│ │ └── autocfg v1.4.0
│ ├── num-integer v0.1.46
│ │ └── num-traits v0.2.19 ()
│ ├── num-traits v0.2.19 ()
│ └── rawpointer v0.2.1
├── rand v0.9.0
│ ├── rand_chacha v0.9.0
│ │ ├── ppv-lite86 v0.2.20
│ │ │ └── zerocopy v0.7.35
│ │ │ ├── byteorder v1.5.0
│ │ │ └── zerocopy-derive v0.7.35 (proc-macro)
│ │ │ ├── proc-macro2 v1.0.94
│ │ │ │ └── unicode-ident v1.0.18
│ │ │ ├── quote v1.0.39
│ │ │ │ └── proc-macro2 v1.0.94 ()
│ │ │ └── syn v2.0.99
│ │ │ ├── proc-macro2 v1.0.94 ()
│ │ │ ├── quote v1.0.39 ()
│ │ │ └── unicode-ident v1.0.18
│ │ └── rand_core v0.9.3
│ │ └── getrandom v0.3.1
│ │ ├── cfg-if v1.0.0
│ │ └── libc v0.2.170
│ ├── rand_core v0.9.3 ()
│ └── zerocopy v0.8.23
└── rand_distr v0.5.1
├── num-traits v0.2.19 ()
└── rand v0.9.0 ()
yep, still looks relatively good.
Re: RustGPT: A pure-Rust transformer LLM built from scratch
#5Absolutely love how readable the entire project is
Never knew Rust could be that readable. Makes me think other Rust engineers are stuck in a masochistic ego driven contest, which would explain everything else I've encountered about the Rust community and recruiting on that side.
Re: RustGPT: A pure-Rust transformer LLM built from scratch
#6I’m curious where you got your training data? I will look myself, but saw this and thought I’d ask. I have a CPU-first, no-backprop architecture that works very well on classification datasets. It can do single‑example incremental updates which might be useful for continuous learning. I made a toy demo to train on tiny.txt and it can predict next characters, but I’ve never tried to make an LLM before. I think my architecture might work well as an on-device assistant or for on-premises needs, but I want to work with it more before I embarrass myself. Any open-source LLM training datasets you would recommend?
Re: RustGPT: A pure-Rust transformer LLM built from scratch
#7This looks rather similar to when I asked an AI to implement a basic xor problem solver I guess fundementally there's really only a very limited amount of ways to implement this.
Re: RustGPT: A pure-Rust transformer LLM built from scratch
#8I’m curious where you got your training data? I will look myself, but saw this and thought I’d ask. I have a CPU-first, no-backprop architecture that works very well on classification datasets. It can do single‑example incremental updates which might be useful for continuous learning. I made a toy demo to train on tiny.txt and it can predict next characters, but I’ve never tried to make an LLM before. I think my arch…
huggingface has plenty of openai and antrophic user to assistant chains, beware there are dragons (hallucinations), but good enough for instruction training. I actually recommend distilling kimi k2 instead for instruction following capabilities.
Re: RustGPT: A pure-Rust transformer LLM built from scratch
#9> ndarray = "0.16.1" rand = "0.9.0" rand_distr = "0.5.0" Looking good!
is this satire or does I must know context behind this comment???
Re: RustGPT: A pure-Rust transformer LLM built from scratch
#10I’m curious where you got your training data? I will look myself, but saw this and thought I’d ask. I have a CPU-first, no-backprop architecture that works very well on classification datasets. It can do single‑example incremental updates which might be useful for continuous learning. I made a toy demo to train on tiny.txt and it can predict next characters, but I’ve never tried to make an LLM before. I think my arch…