Mistral-8x7B-Chat

Post by lannisterstark » Mon, Dec 11, 2023, 5:15 AM UTC

appplication wrote: post #69

lannisterstark wrote: post #5

Man this LLM stuff gets released faster than I can keep up. Is there a centralized list somewhere that tests "use this for x purpose, use that for y?"

Honestly none of them are particularly incredible compared to neither. At this point everything is a prototype, there’s no sense in building knowledge specific to them unless you’re also building LLM products. It’s just distributed ADHD at this point. LLMs are new and cool and each new release will be significantly better than the last, but as with any emerging tech, we’re on an exponential curve so there’s no sense…

Post by mirekrusin » Mon, Dec 11, 2023, 6:59 AM UTC

jakderrida wrote: post #21

mirekrusin wrote: post #18

Earlier quoted context omitted.

LLMs work on tokens where characters are hidden away. They'd have to be explicitly trained on spelling each token out into single letter tokens and as they are bad at information symmetry - from single letter tokens back onto tokens as well. I don't think anybody does this so they're left with what's in training data only. Otherwise they don't have chance to reconstruct this information as tokens could map to any equ…

I thought so, too. But then I asked it to define fake words that were portmanteaus I made up. Believe me, my understanding of BERT and discriminant models aligned perfectly with what you're saying. But testing out the theory that it can break down and make meaning of fake words with accurate depictions of what words I'm combining proved me wrong. Generative models must work differently than you and I thought.

Post by tarruda » Mon, Dec 11, 2023, 9:07 AM UTC

tarruda wrote: post #74

antirez wrote: post #2

llama.cpp support needed ASAP, not just quantized, but I hope it will be albe to run in a way that keeps the same two experts for minimum N tokens generated and re-valuate the expert to select only then, so that it will be possible to do the inference in low ram doing something similar to "paging".

> so that it will be possible to do the inference in low ram doing something similar to "paging". I was also hoping for that, but my initial research suggests that you need to fit everything into VRAM (or RAM when doing llama.cpp CPU inference). The upside is that it should be as fast as running a single 7B model, which on llama.cpp should be in the ballpark of 10 tok/sec when 4-bit quantized. For local usage, I'd sa…