sexy self is a cool naming convention that will possibly replace foobar
https://github.com/wangyi-fudan/wyGPT/blob/main/cpu.cpp#L140
WyGPT: Minimal mature GPT model in C++
11–18 of 18 posts
Re: WyGPT: Minimal mature GPT model in C++
#12This project comes from the author of wyhash, which is now the default hashing algorithm in Go, Zig etc.
What's lacking is more information about the project. I was able to compile it and just started training on GPU, but I have no idea what's going on, when it will end, and what the characteristics are in general. Currently the only output I get is:
./train book-war-and-peace.txt
book-war-and-peace.txt 3.05397 1
para 1679360
32 1.3654 206.1985
Re: WyGPT: Minimal mature GPT model in C++
#13This project comes from the author of wyhash, which is now the default hashing algorithm in Go, Zig etc. What's lacking is more information about the project. I was able to compile it and just started training on GPU, but I have no idea what's going on, when it will end, and what the characteristics are in general. Currently the only output I get is: ./train book-war-and-peace.txt book-war-and-peace.txt 3.05397 1 par…
2.5 years of effort but the author certainly didn't spend even 2.5 mins writing code for more descriptive outputs. Meanwhile I'm also training in order to find how well it works but I only have a small GPU. I would appreciate if someone could provide a trained model.
Re: WyGPT: Minimal mature GPT model in C++
#14Very neat.
Re: WyGPT: Minimal mature GPT model in C++
#15Re: WyGPT: Minimal mature GPT model in C++
#16This project comes from the author of wyhash, which is now the default hashing algorithm in Go, Zig etc. What's lacking is more information about the project. I was able to compile it and just started training on GPU, but I have no idea what's going on, when it will end, and what the characteristics are in general. Currently the only output I get is: ./train book-war-and-peace.txt book-war-and-peace.txt 3.05397 1 par…
2.5 years of effort but the author certainly didn't spend even 2.5 mins writing code for more descriptive outputs. Meanwhile I'm also training in order to find how well it works but I only have a small GPU. I would appreciate if someone could provide a trained model.
So it's a character-based model (makes sense, because there's no tokenizer to be found).
The model file is 3.3 MB
Output after 3 hours of training (loss = 0.4160):
hello, the sounds of the rain assistants. In provations
which had been banitz had taken step in, the spirit of the crowd, and
the right winot to scrap on , Valuex, one of marks othhhanled me in a
most powerful power. The other Augun had been in
commands among that guests, the regiments were placed by imists, and
science of irritation, but or thoughts that he would never cease to point
before meant in the losses were discussing, and at once she had lived
in the church.
[0] https://www.reddit.com/r/MachineLearning/comments/o2u2cm/pwy...
Re: WyGPT: Minimal mature GPT model in C++
#17I love every ambitious and tiny codebase, but the formatting looks very uncommon and for me personally - hard to read. Is there a tool that produces it?
Re: WyGPT: Minimal mature GPT model in C++
#18The code is wizardry. It looks more confusing than obfuscated JavaScript