131–137 of 137 posts
The paper puts out an interesting hypothesis that these LLM-derived transformer layers have the ability to "refine" any set of learned tokens, even in different modalities. I wonder if what you're seeing here is related?
Earlier quoted context omitted.
im kind of wondering like what the ceiling would be on reasoning for something like the 1.5T models with the repeating technique, but they would take a long time to download. i think if you have them already it would take maybe an hour or so to check against a swath of prompts. whats the reasoningest open model at the moment? my guess is that large models trained on large corpuses there is just some ceiling of "reaso…
I stick with models I can run on VRAM, but DeepSeek Speciale have the best reasoning capabilities of the models I can actually run ( https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Speciale ). What hardware can you access? I have Deepseek etc, but inferencing on DDR5 would take about 2-3 weeks for a simple scan. I think this works best with dense models, but it also seems ok with MoE. @everyone: Can someone hook me…
but yeah on demand would be a lot of ssd churn so id just do it for testing or getting some hidden state vectors.
Earlier quoted context omitted.
Isn’t that what LoRA does ?
LoRAs are better at steering models to produce correct answers from their data set than imparting new knowledge.
>Overall, our findings position LoRA as the complementary axis of memory alongside RAG and ICL, offering distinct advantages.
That's really interesting. Makes me immediately ask two questions: 1. Should we be training models like this from the start? It seems that a model trained with layer loops would be able to take advantage of it better than rearranging the layers of a naive model. 2. Should we even be using a fixed number of layers? If models are this tolerant to their inner layers being meddled with, then it doesn't make sense to run…
Earlier quoted context omitted.
I think the main challenge with combining layers of different would models be their differing embedding sizes and potentially different vocabularies. Even between two models of identical architecture, they may have landed on quite different internal representations if the training data recipe was substantially different. But it would be fun to experiment with.
Even with the same embedding sizes and vocabularies, there’s nothing that forces the meaning of dimension 1 of model 1 to mean the same thing as dimension 1 of model 2 — there are lots of ways to permute the dimensions of a model without changing its output, so whatever dimension 1 means the first time you train a model is just as likely to end up as dimension 2 the second time you train is as it is to be consistent…