Viewing profile — owl_brawl

For those who haven't read it, Rich Caruana's thesis on multi-task learning is beautifully written (the cited 1998 paper here). It's amazing to see how far the field has come, and,…

I would love answers to these questions too, particularly on the vocab size

Hi alekandreev, Any reason you decided to go with a token vocabulary size of 256k? Smaller vocab/vector sizes like most models in this size seem to be using (~16-32k) are much easi…

The "RAG" part (over the training set) is by far the smallest contribution to the performance gains reported (see the ablation study in section 5.2). I don't think the model is act…