Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenArch – PyTorch implementations of modern LLM architectures

github.com

Post reply on HN
Print view View original

11–20 of 30 posts

Re: OpenArch – PyTorch implementations of modern LLM architectures

#11

Post by anuj0456 »

ipieter wrote: post #7
k__ wrote: post #6

I was yesterday years old when I learned that those open weight models need custom code to run. Somehow I expected inference engines are generic LLM runtimes that can execute any weight. So, to get this right. Someone trains a model. They release the weights and a reference implementation of the model architecture. Then a provider has to host this model either by running inference via the reference implementation, an…

The implementation of a language model is usually good enough for running _a single forward pass_ through the model, but to host it via an inference engine you typically need to convert a few operations. For instance the MLP can be easily split across GPUs (tensor parallelism) and MoEs also have a way of parallelizing. Most of it is pretty standard, since not that many different layers and primitives are used in LLM…

yes. this is just raw implementation of the model arch as described in papers. for complete model training with back propogation we need training pipeline with optmizer and loss calculation.

Re: OpenArch – PyTorch implementations of modern LLM architectures

#13

Post by ismailmaj »

k__ wrote: post #6

I was yesterday years old when I learned that those open weight models need custom code to run. Somehow I expected inference engines are generic LLM runtimes that can execute any weight. So, to get this right. Someone trains a model. They release the weights and a reference implementation of the model architecture. Then a provider has to host this model either by running inference via the reference implementation, an…

FYI Mistral at launch just dropped the weights without any model architecture mentioned.

Most of the OSS models follow the same architecture which is Llama +- a few things, so it wasn't too hard for people to make it work.

Re: OpenArch – PyTorch implementations of modern LLM architectures

#14

Post by anuj0456 »

ismailmaj wrote: post #13
k__ wrote: post #6

I was yesterday years old when I learned that those open weight models need custom code to run. Somehow I expected inference engines are generic LLM runtimes that can execute any weight. So, to get this right. Someone trains a model. They release the weights and a reference implementation of the model architecture. Then a provider has to host this model either by running inference via the reference implementation, an…

FYI Mistral at launch just dropped the weights without any model architecture mentioned. Most of the OSS models follow the same architecture which is Llama +- a few things, so it wasn't too hard for people to make it work.

yes, most of them are similar. but implementation of GQA, MLA, mHC, Sliding Window changes the implementation drastically because of which the overall model effeciency changes.

Re: OpenArch – PyTorch implementations of modern LLM architectures

#15

Post by dguest »

k__ wrote: post #6

I was yesterday years old when I learned that those open weight models need custom code to run. Somehow I expected inference engines are generic LLM runtimes that can execute any weight. So, to get this right. Someone trains a model. They release the weights and a reference implementation of the model architecture. Then a provider has to host this model either by running inference via the reference implementation, an…

Makes me feel better as a guy who tries to get O(1000) times smaller models working for scientific applications. Writing a backend that works for all the models people train is quite a chore, so the common refrain is "why don't we just do what industry does?" (he answer is that "industry" has billions of dollars). You also get "no one uses X backend any more" to which the reply is also "yes, but they have billions of dollars and a team of software engineers".

Re: OpenArch – PyTorch implementations of modern LLM architectures

#16

Post by anuj0456 »

philipportner wrote: post #9
k__ wrote: post #6

I was yesterday years old when I learned that those open weight models need custom code to run. Somehow I expected inference engines are generic LLM runtimes that can execute any weight. So, to get this right. Someone trains a model. They release the weights and a reference implementation of the model architecture. Then a provider has to host this model either by running inference via the reference implementation, an…

Yes. https://inferencex.semianalysis.com provides some comparisons wrt. certain cost metrics. Some commonly used ones are vLLM, TensorRT-LLM, and SGLang. These three at least are open source, and all come with an Apache 2.0 license. Some providers also have to implement their own engines, e.g., Cerebras has their own inference serving stack for their wafer-scale chips, as does Google for their TPUs (XLA compiler).

Thanks for sharing

Re: OpenArch – PyTorch implementations of modern LLM architectures

#17

Post by tweedler290 »

This is really cool!

Had a question on the MoE: in kimmi-K2/model.py, the router does torch.topk(..., k=self.num_experts) while every other MoE uses k=self.top_k. The ctor's top_k=8 is never stored either.. Is that intentional dense routing or should that be self.top_k?

Re: OpenArch – PyTorch implementations of modern LLM architectures

#19

Post by anuj0456 »

tweedler290 wrote: post #17

This is really cool! Had a question on the MoE: in kimmi-K2/model.py, the router does torch.topk(..., k=self.num_experts) while every other MoE uses k=self.top_k. The ctor's top_k=8 is never stored either.. Is that intentional dense routing or should that be self.top_k?

thanks for pointing out. its a bug
Post reply on HN