Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RustGPT: A pure-Rust transformer LLM built from scratch

github.com

Post reply on HN
Print view View original

141–150 of 186 posts

Re: RustGPT: A pure-Rust transformer LLM built from scratch

#141

Post by stonemetal12 »

WhereIsTheTruth wrote: post #95
DiabloD3 wrote: post #49

Earlier quoted context omitted.

uv is great, but I think the real fix is just abandoning Python. The culture that language maintains is rather hostile to maintainable development, easier to just switch to Rust and just write better code by default.

abandoning Python for Rust in AI would cripple the field, not rescue it the disease is the cargo cult addiction (which Rust is full of) to micro libraries, not the language that carries 90% of all peer reviewed papers, datasets, and models published in the last decade every major breakthrough, from AlphaFold to Stable Diffusion, ships with a Python reference implementation because that is the language researchers can…

Nah, it would be like going from chemistry to chemical engineering. Doing chemical reactions in the lab by hand is great for learning but you aren't going to run a fleet of cars on hand made gas. Getting ML out of the lab and into production needs that same mental conversion from CS to SE.

Re: RustGPT: A pure-Rust transformer LLM built from scratch

#142

Post by yobbo »

tripplyons wrote: post #136
yobbo wrote: post #128

Very nice! Next thing to add would be numerical gradient testing.

Is that where you approximate a partial derivative as a difference in loss over a small difference in a single parameter's value? Seems like a great way to verify results, but it has the same downsides as forward mode automatic differentiation since it works in a pretty similar fashion.

Yes, the purpose is to verify the gradient computations which are typically incorrect on the first try for things like self-attention and softmax. It is very slow.

It is not necessary for auto-differentiation, but this project does not use that.

Re: RustGPT: A pure-Rust transformer LLM built from scratch

#143

Post by virtualritz »

Galanwe wrote: post #52
untrimmed wrote: post #16

As someone who has spent days wrestling with Python dependency hell just to get a model running, a simple cargo run feels like a dream. But I'm wondering, what was the most painful part of NOT having a framework? I'm betting my coffee money it was debugging the backpropagation logic.

> spent days wrestling with Python dependency hell I mean I would understand that comment in 2010, but in 2025 it's grossly ridiculous.

So in 2025, in Python, if I depend on two packages. A and B, and they both depend on different, API-incompatible or behavior-incompatible (or both) versions of C, that won't be an issue?

That's not my experience and e.g. uv hasn't helped me with that. I believe this is an issue with Python itself?

If parent was saying something "grossly ridiculous" I must be doing something wrong too. And I'm happy to hear what as that would lower the pain of using Python.

I.e. this was assumably true three years ago:

https://stackoverflow.com/questions/70828570/what-if-two-pyt...

Re: RustGPT: A pure-Rust transformer LLM built from scratch

#144

Post by famouswaffles »

pjmlp wrote: post #135
airza wrote: post #50

Earlier quoted context omitted.

There's not really another game in town if you want to do fast ML development :/

PyTorch also supports C++ and Java, Tensorflow also does C++ and Java, Apple AI is exposing ML libraries via Swift, Microsoft is exposing their AI stuff via .NET and Java as well, then there is Julia and Mojo is coming along. It is happening.

TensorFlow is a C++ library with a python wrapping, yet nobody (obviously exaggeration) actually uses tensorflow (or torch) in C++ for ML R&D.

It's like people just don't get it. The ML ecosystem in python didn't just spring from the ether. People wanted to interface in python badly, that's why you have all these libraries with substantial code in another language yet development didn't just shift to that language.

If python was fast enough, most would be fine to ditch the C++ backends and have everything in python, but the reverse isn't true. The C++ interface exists, and no-one is using it.

Re: RustGPT: A pure-Rust transformer LLM built from scratch

#145

Post by virtualritz »

taminka wrote: post #75
Sl1mb0 wrote: post #71

Earlier quoted context omitted.

> dependency inclusion _should not_ be a main criterion for evaluating how good a build system is That's just like, your opinion, man.

i mean, unless you have some absolute divine truths, that's kind of the best i have :shrug

There are no truths but your opinion in this case runs counter of what 35 years developing software have taught me.

Obviously, I may be an outlier. Some crank who's just smitten by the proposal of spending his time writing code instead of trying to get a dependency (and its sub-dependencies and their sub-dependencies) to build at all (e.g. C/C++) or to have the right version that works with ALL the code that depends on it (e.g. Python).

I.e. I use cargo foremost (by a large margin) for that reason.

Re: RustGPT: A pure-Rust transformer LLM built from scratch

#146

Post by tcfhgj »

leoh wrote: post #132
ericdotlee wrote: post #92

This is incredibly cool, but I wonder when more of the AI ecosystem will move past python tooling into something more... performant? Very interesting to already see rust based inference frameworks as well.

"Python" is perfectly performant for AI and this demonstrates a deep lack of understanding. Virtually every library in python used for AI delegates to lower-level code written in C++.

well, not all the time, e.g. orchestration and handling between multiple libraries

Re: RustGPT: A pure-Rust transformer LLM built from scratch

#147

Post by taminka »

virtualritz wrote: post #145
taminka wrote: post #75

Earlier quoted context omitted.

i mean, unless you have some absolute divine truths, that's kind of the best i have :shrug

There are no truths but your opinion in this case runs counter of what 35 years developing software have taught me. Obviously, I may be an outlier. Some crank who's just smitten by the proposal of spending his time writing code instead of trying to get a dependency (and its sub-dependencies and their sub-dependencies) to build at all (e.g. C/C++) or to have the right version that works with ALL the code that depends…

in my original comment i specifically mentioned that C (and C++) situation is also too extreme and not optimal...

Re: RustGPT: A pure-Rust transformer LLM built from scratch

#148

Post by thomask1995 »

abricq wrote: post #21

This is great ! Congratulations. I really like your project, especially I like how easily it is to peak at. Do you plan on moving forward with this project ? I seem to understand that all the training is done on the CPU, and that you have next steps regarding optimizing that. Do you consider GPU accelerations ? Also, do you have any benchmarks on known hardware ? Eg, how long would it take to train on a macbook lates…

HI! OG Author here.

Honestly, I don't know.

This was purely a toy project/thought experiment to challenge myself to learn exactly how these LLMs worked.

It was super cool to see the loss go down and it actually "train".

This is SUPER far from a the real deal. Maybe it could be cool to see how far a fully in memory LLM running on CPU can go.

Re: RustGPT: A pure-Rust transformer LLM built from scratch

#149

Post by airza »

adastra22 wrote: post #110
airza wrote: post #82

Earlier quoted context omitted.

Okay. Humor me. I want to write a transformer-based classifier for a project. I am accustomed to the pytorch and tensorflow libraries. What is the equivalent using C?

You do know that tensorflow was written in C++ and the Python API bolted on top?

I am. Are you suggesting that as an alternative to the python bindings i should use C to invoke the C++ ABI for tensorflow?

Re: RustGPT: A pure-Rust transformer LLM built from scratch

#150

Post by steveklabnik »

0xffff2 wrote: post #138
Diggsey wrote: post #119

Earlier quoted context omitted.

It doesn't link two versions of `rand-core`. That's not even possible with rust (you can only link two semver-incompatible versions of the same crate). And dependency specifications in Rust don't work like that - unless you explicitly override it, all dependencies are semver constraints, so "0.9.0" will happily match "0.9.3".

So there's no difference at all between "0", "0.9" and "0.9.3" in cargo.toml (Since semver says only major version numbers are breaking)? As a decently experienced Rust developer, that's deeply surprising to me. What if devs don't do a good job of versioning and there is a real incompatibility between 0.9.3 and 0.9.4? Surely there's some way to actually require an exact version?

Note that in the output, there is rand 0.9.0, and two instances of rand_core 0.9.3. You may have thought it selected two versions because you missed the _core there.

> So there's no difference at all between "0", "0.9" and "0.9.3" in cargo.toml

No, there is a difference, in particular, they all specify different minimum bounds.

The trick is that these are using the ^ operator to match, which means that the version "0.9.3" will satisfy all of those constraints, and so Cargo will select 0.9.3 (the latest version at the time I write this comment) as the one version to satisfy all of them.

Cargo will only select multiple versions when it's not compatible, that is, if you had something like "1.0.0" and "0.9.0".

> Surely there's some way to actually require an exact version?

Yes, you'd have to use `=`, like `=0.9.3`. This is heavily discouraged because it would lead to a proliferation of duplication in dependency versions, which aren't necessarily unless you are trying to avoid some sort of specific bugfix. This is sometimes done in applications, but basically should never be done in libraries.

Post reply on HN