91–100 of 252 posts
What's the best way to run this on my Macbook Pro? I've tried LMStudio, but I'm not a fan of the interface compared to OpenAI's. The lack of automatic regeneration every time I edit my input, like on ChatGPT, is quite frustrating. I also gave Ollama a shot, but using the CLI is less convenient. Ideally, I'd like something that allows me to edit my settings quite granularly, similar to what I can do in OpenLM, with th…
Earlier quoted context omitted.
This is a bit of a misnomer. Each expert is a sub network that specializes in sub understanding we can't possibly track. During training a routing network is punished if it does not evenly distribute training tokens to the correct experts. This prevents any one or two networks from becoming the primary networks. The result of this is that each token has essentially even probability of being routed to one of the sub m…
Why do we expect this to perform better? Couldn’t a regular network converge on this structure anyways?
"64K tokens context window" I do wish they had managed to extend it to at least 128K to match the capabilities of GPT-4 Turbo Maybe this limit will become a joke when looking back? Can you imagine reaching a trillion tokens context window in the future, as Sam speculated on Lex's podcast?
I just find it hilarious how approximately 100% of models beat all other models on benchmarks.
Earlier quoted context omitted.
This is a bit of a misnomer. Each expert is a sub network that specializes in sub understanding we can't possibly track. During training a routing network is punished if it does not evenly distribute training tokens to the correct experts. This prevents any one or two networks from becoming the primary networks. The result of this is that each token has essentially even probability of being routed to one of the sub m…
Any idea why everyone seems to be using 8 experts? (Or was GPT-4 using 16?) Did we just try different numbers and found 8 was the optimum?
What's the best way to run this on my Macbook Pro? I've tried LMStudio, but I'm not a fan of the interface compared to OpenAI's. The lack of automatic regeneration every time I edit my input, like on ChatGPT, is quite frustrating. I also gave Ollama a shot, but using the CLI is less convenient. Ideally, I'd like something that allows me to edit my settings quite granularly, similar to what I can do in OpenLM, with th…
Ollama with WebUI https://github.com/open-webui/open-webui
I'm really excited about this model. Just need someone to quantize it to ~3 bits so it'll run on a 64GB MacBook Pro. I've gotten a lot of use from the 8x7b model. Paired with llamafile and it's just so good.
Can you explain your use case? I tried to get into offline llms, on my machine and even android but without discrete graphics, its a slow hog so I didnt enjoy it but suppose I buy one, what then ?
It has been a huge force multiplier for me and most importantly of all, it removes the dread of not knowing where to start and the dread of sending your inner monologue to someone's stupid cloud.
If you're curious: https://github.com/noman-land/transcript.fish/ though this doesn't include any Mixtral stuff because I don't use it programmatically (yet). I soon hope to use it to answer questions about the episodes like who the special guest is and whatnot, which is something I do manually right now.
Earlier quoted context omitted.
You can't upgrade it? Edit: I haven't owned a laptop for years, probably could have surmised they'd be more user hostile nowadays.
Everything is soldered in these days. It's complete garbage. And most of the other vendors just copy Apple so even things like Lenovo have the same problems. The current state of laptops is such trash
People need to vote with their wallet, and not buy stuff that goes against their principles.
Does anyone have a good layman's explanation of the "Mixture-of-Experts" concept? I think I understand the idea of having "sub-experts", but how do you decide what each specialization is during training? Or is that not how it works at all?
This is a bit of a misnomer. Each expert is a sub network that specializes in sub understanding we can't possibly track. During training a routing network is punished if it does not evenly distribute training tokens to the correct experts. This prevents any one or two networks from becoming the primary networks. The result of this is that each token has essentially even probability of being routed to one of the sub m…
Or would that make each expert too small to be useful? TinyLlama is 1B and it's almost useful! I guess 8x1B would be Mixture of TinyLLaMAs...