Viewing profile — ajtejankar

Exactly! I don't think Skills is a new algorithm but it's definitely a new paradigm of organizing your prompt. Essentially, dynamic context assembling with stuff crossing user boun…

I have been independently thinking about a lot of this for some time now. So this is so exciting for me. Concretizing _skills_ allows, as you said, a common pattern for people to r…

IMO LoRAs are no different from context tokens. In fact, before LoRAs tuned prompt vectors were a popular adapter architecture. Conceptually, the only difference is that prompt ada…

Yup! I fully agree. It also taps into the ability of LLMs to write code given good prompts. All you need is for the LLM to recognize that it needs something, fetch it into the cont…

With so many code sandbox providers coming out I would go further than you say that this is almost a non-problem.

There is an 'old' paper https://arxiv.org/abs/1701.06538 but I believe there needs to be renewed effort in this direction.

Fixing the experts for a layer might not work since all experts fire almost with equal probability. There are small variations by topic but they are consistent enough to be capture…

The plots show 2 dimensional projection of the 8 dimensional feature vector of each paragraph. So, x and y axis are linear combination of 8 different experts. Ideally, all of this …

The base model has 32 layers and there is a single linear layer for language modeling (going from embeddings to the vocabulary) that gets applied at the very end.