21–30 of 31 posts
This is really cool! Had a question on the MoE: in kimmi-K2/model.py, the router does torch.topk(..., k=self.num_experts) while every other MoE uses k=self.top_k. The ctor's top_k=8 is never stored either.. Is that intentional dense routing or should that be self.top_k?
hey, Building these from scratch in pure PyTorch is honestly the best way to deeply understand the paper details. something better than simply implementing a traditional Transformer or GPT-2，As an individual maintainer, will be able to keep up with future model updates?
I was yesterday years old when I learned that those open weight models need custom code to run. Somehow I expected inference engines are generic LLM runtimes that can execute any weight. So, to get this right. Someone trains a model. They release the weights and a reference implementation of the model architecture. Then a provider has to host this model either by running inference via the reference implementation, an…
In fact, this was close to be true until last year: almost every open model except DeepSeek had a very similar architecture that was pretty close to the GPT-2 one with very few variations on top (and sometimes an MoE architecture, which itself was a few year old at that point).
But a year ago there's been a cambrian explosion, first in attention mechanism but also in a bunch of other directions, mostly coming from China, and now there's a very massive diversity today's space.
I was yesterday years old when I learned that those open weight models need custom code to run. Somehow I expected inference engines are generic LLM runtimes that can execute any weight. So, to get this right. Someone trains a model. They release the weights and a reference implementation of the model architecture. Then a provider has to host this model either by running inference via the reference implementation, an…
FYI Mistral at launch just dropped the weights without any model architecture mentioned. Most of the OSS models follow the same architecture which is Llama +- a few things, so it wasn't too hard for people to make it work.
It used to be the case until last year, but now almost every Chinese model come with their own linear attention mechanism.
I was yesterday years old when I learned that those open weight models need custom code to run. Somehow I expected inference engines are generic LLM runtimes that can execute any weight. So, to get this right. Someone trains a model. They release the weights and a reference implementation of the model architecture. Then a provider has to host this model either by running inference via the reference implementation, an…
> Somehow I expected inference engines are generic LLM runtimes that can execute any weight. In fact, this was close to be true until last year: almost every open model except DeepSeek had a very similar architecture that was pretty close to the GPT-2 one with very few variations on top (and sometimes an MoE architecture, which itself was a few year old at that point). But a year ago there's been a cambrian explosion…
This is really cool, great way to reinforce our understanding of model architectures! But how is the author confirming that these model architecture implementations are correct though? I don't see any details in the README.md.
Hey anuj This is excellent for understanding. I'm having some trouble to get into understanding - pytorch is for me the RL which is used as gym/training. There I can chose ppo, dnq and other agents to perform some predefined actions in a predefined gym/world. The repo you are showing - I really have problems to get it into RL understanding of mine. What's the gym? What are the agents. Can it be used to train that mod…
Not the poster, but maybe I can help. Your comment is a little unclear, so it‘s hard to parse your exact question. But it seems you are conflating 3 things, PyTorch, RL and Gym/Training (?). – PyTorch is a framework which lets you define neural network models. – RL is a collection of methods to train neural networks (change the network parameters to improve its performance). – An RL-Gym is a framework to apply the ne…
I think my problem is with the imagination and knowledge transfer:
As i remember, in the year ~2010 +/- 2-3y, machine learning became (again) POI for technology. I remember convolutional networks, deepQ, Genetic algorithms, etc being in the press. I remember Michael Schmidt, a biology student at that time (~roundabout~), being as "the influential data scientist of the year" at forbe's cover. MS did symbolic regression combined with genetic algorithm and founded nutopian that was developing this propriatary "eureqa" algorithm.
At that time, PyTorch was "created" by Facebook as an answer to google's tensorflow. I remember a guy trained on PyTorch a model that could play Nintendo's Mario. I wanted to understand how to do it, but it was toooo much for my brain. What i remember:
- one needs a gym that stands for the world the agent "lives in". The gym/world is data, like the input of a screenshot, or a gym/world for Stock Trading might have historical courses, prices etc.. also rewards, penalties are defined in the gym.
- the agent living in this gym/world have "actions" - left,right,buy, sell, ...
- RL training is then to let that agent "move through world/gym and perform some of its and agent's actions, where the outcome is rewarded or not. agent learns through rewards.."
Gym and RL training can be PyTorch or TensorFlow. In my understanding that gym is only a definition of what data is avaible, shape of data, etc. Then, the agent needs to be defined too - i remember Stable Baselines 3..
and then the RL training is just agent does x -> gym checks for outcome -> reward/penalty & adaption of weights -> inference -> repeat
This is cleary imaginable for me, even its not fully correct - there is an agent, there is a world, there is method to act&compare/data manipulation of weights - there is PyTorch that offers the tools to build a world, agent and to do the weight manipulations ..
OP have implementations of different Archs in PyTorch and my knowledge implodes. I cant imagine what is the "world/gym" in here? What is the data and actionables , whats the agent and whats its role/actions - and how would I use this implementations??
can I clone the repo, pick the one arch i want - and, when i would feed in data, after training, i will get the model that i can use for inference???
I definitely have a knot in my brain because of this!
Thank you for taking your time and trying to understand what my previous comment meant :) Are there some usefull sources for seeing how the learning is actually done (despite the tokenization) - and the skills training?? how are models trained to find the loophole that allows them to break out and reach their objectives "illegaly" - thats bugs my head since weeks!
I was yesterday years old when I learned that those open weight models need custom code to run. Somehow I expected inference engines are generic LLM runtimes that can execute any weight. So, to get this right. Someone trains a model. They release the weights and a reference implementation of the model architecture. Then a provider has to host this model either by running inference via the reference implementation, an…
Imagine what could happen if other open source projects had this level of engagement and expertise at hand and eager to contribute... instead of arguing over politics and making yet another fork of something.
I was yesterday years old when I learned that those open weight models need custom code to run. Somehow I expected inference engines are generic LLM runtimes that can execute any weight. So, to get this right. Someone trains a model. They release the weights and a reference implementation of the model architecture. Then a provider has to host this model either by running inference via the reference implementation, an…
There was a post here the other day highlighting this by showing the benchmark perf of different I defence providers, it's a fantastic area to cheap out in, because you can never really tell if a model is 75% good or 83% good on some specific benchmark when you use it to build your own stuff