101–110 of 252 posts
Is this the best permissively licensed model out there?
So far it is Command R+. Let's see how this will fare on Chatbot Arena after a few weeks of use.
Most people would not consider Command R+ to count as the "best permissively licensed model" since CC-BY-NC is not usually considered "permissively licensed" – the "NC" part means "non-commercial use only"
Earlier quoted context omitted.
This is a bit of a misnomer. Each expert is a sub network that specializes in sub understanding we can't possibly track. During training a routing network is punished if it does not evenly distribute training tokens to the correct experts. This prevents any one or two networks from becoming the primary networks. The result of this is that each token has essentially even probability of being routed to one of the sub m…
I heard MoE reduces inference costs. Is that true? Don't all the sub networks need to be kept in RAM the whole time? Or is the idea that it only needs to run compute on a small part of the total network, so it runs faster? (So you complete more requests per minute on same hardware.) Edit: Apparently each part of the network is on a separate device. Fascinating! That would also explain why the routing network is train…
Earlier quoted context omitted.
This is a bit of a misnomer. Each expert is a sub network that specializes in sub understanding we can't possibly track. During training a routing network is punished if it does not evenly distribute training tokens to the correct experts. This prevents any one or two networks from becoming the primary networks. The result of this is that each token has essentially even probability of being routed to one of the sub m…
Has anyone tried MoE at smaller scales? e.g. a 7B model that's made of a bunch of smaller ones? I guess that would be 8x1B. Or would that make each expert too small to be useful? TinyLlama is 1B and it's almost useful! I guess 8x1B would be Mixture of TinyLLaMAs...
I'm considering switching my function calling requests from OpenAI's API to Mistral. Are they using similar formats? What's the easiest way to use Mistral? Is it by using Huggingface?
easiest is probably with ollama [0]. I think the ollama API is OpenAI compatible. [0] https://ollama.com/
Earlier quoted context omitted.
This is a bit of a misnomer. Each expert is a sub network that specializes in sub understanding we can't possibly track. During training a routing network is punished if it does not evenly distribute training tokens to the correct experts. This prevents any one or two networks from becoming the primary networks. The result of this is that each token has essentially even probability of being routed to one of the sub m…
Would it be analogous to say instead of having a single Von Neumann who is a polymath, we’re posing the question to a pool of people who are good at their own thing, and one of them gets picked to answer?
Human brains seem to do something similar, inasmuch as blood flow (and hence energy use) per region varies depending on the current problem.
Earlier quoted context omitted.
Can you explain your use case? I tried to get into offline llms, on my machine and even android but without discrete graphics, its a slow hog so I didnt enjoy it but suppose I buy one, what then ?
I run Mistral-7B on an old laptop. It's not very fast and it's not very good, but it's just good enough to be useful. My use case is that I'm more productive working with a LLM but being online is a constant temptation and distraction. Most of the time I'll reach for offline docs to verify. So the LLM just points me in the right direction. I also miss Google offline, so I'm working on a search engine. I thought I cou…
Earlier quoted context omitted.
Sure, in the same way SaaS companies are just thin wrappers on top of databases and the open web.
You will find that a disproportionately large amount of work and innovation in an AI product is in the backing model (GPT, Mixtral, etc.). While there's a huge amount of work in databases and the open web, SaaS products typically add a lot more than a thin API layer and a shiny website (well some do but you know what I mean)
Does anyone have a good layman's explanation of the "Mixture-of-Experts" concept? I think I understand the idea of having "sub-experts", but how do you decide what each specialization is during training? Or is that not how it works at all?
During inference, instead of a single MLP in each transformer layer, MoEs have `n` MLPs and a single layer "gate" in each transformer layer. In the forward pass, softmax of the gate's output is used to pick the top `k` (where k is https://github.com/huggingface/transformers/blob/main/src/tr...
It feels absolutely amazing to build an AI startup right now. It's as if your product automatically becomes cheaper, more reliable, and more scalable with each new major model release. - We first struggled with limited context windows [solved] - We had issues with consistent JSON ouput [solved] - We had rate limiting and performance issues for the large 3rd party models [solved] - Hosting our own OSS models for small…
We are going to quickly reach the point where most of these AI startups (which do nothing but provide thin wrappers on top of public LLMs) aren't going to be needed at all. The differentiation will need to come from the value of the end product put in front of customers, not the AI backend.
Or you can continue selling shovels. Still lots of expensive labeling services out there, to stay in the image-recognition parallel