Viewing profile — joennlae

what a find

How can I make sure that each github runner uses exactly one cpu core?

Trainable Llama-like transformer (with backpropagation) in numpy only (~600 lines) https://github.com/joennlae/tensorli

+1. Does someone know how to do that?

Author here: Let me try to give an overview as I saw some questions repeating itself. * This accelerator is for an Edge/Inference case, so there is no training on this chip. * We i…

Thank you for the feedback :-) We have to be careful with the comparisons we make. The TPUv3 is a training and datacenter chip and not an Edge/Inference chip. They optimise for a d…

Author here: Thank you for the feedback :-) A lot of the work regarding the comparison with „simple“ approximate matrix multiplication has been done in the preceding paper: https:/…

That is true. I went for a simple implementation of the layer norm and included it in the tensorli definition. But it would have been better to define it as a moduli for clarity.

This would be interesting to consider. But at the moment, nothing is optimized, so many things must be tackled first (especially in the backwards path, for example, buffering) to j…

They are still applying: https://tmsearch.uspto.gov/bin/showfield?f=doc&state=4805:wl...

The author here: I absolutely agree with you. I went for a bit more catchy title.

Llama1 --> 1.0T Llama2 --> 2.0T Mistral --> ?? They do not publish how many tokens it is pre-trained on, additionally to sharing no info on datasets used (except for fine-tuning). …

You will probably notice it also when trying to push a new branch: `error: failed to push some refs to ` when using --set-upstream