41–50 of 252 posts
Does anyone have a good layman's explanation of the "Mixture-of-Experts" concept? I think I understand the idea of having "sub-experts", but how do you decide what each specialization is during training? Or is that not how it works at all?
- We first struggled with limited context windows [solved]
- We had issues with consistent JSON ouput [solved]
- We had rate limiting and performance issues for the large 3rd party models [solved]
- Hosting our own OSS models for small and medium complex tasks was a pain [solved]
Obivously every startup still needs to build up defensibility and focus on differentiating with everything “non-AI”.
Pricing? Found it: https://mistral.ai/technology/#pricing It'd useful to add a link to the blog post. While it's an open model, most will only be able to use it via the API.
It's open source, you can just download and run it for free on your own hardware.
Does anyone have a good layman's explanation of the "Mixture-of-Experts" concept? I think I understand the idea of having "sub-experts", but how do you decide what each specialization is during training? Or is that not how it works at all?
During training a routing network is punished if it does not evenly distribute training tokens to the correct experts. This prevents any one or two networks from becoming the primary networks.
The result of this is that each token has essentially even probability of being routed to one of the sub models, with the underlying logic of why that model is an expert for that token being beyond our understanding or description.
Keep on doing this great work.
Edit: been using the previous version, seems like this one is even better?
These LLMs are making RAM great again. Wish I had invested in the extra 32GB for my mac laptop.
Edit: I haven't owned a laptop for years, probably could have surmised they'd be more user hostile nowadays.
Pricing? Found it: https://mistral.ai/technology/#pricing It'd useful to add a link to the blog post. While it's an open model, most will only be able to use it via the API.
That looks expensive compared to what groq was offering: https://wow.groq.com/
Earlier quoted context omitted.
not that massive, we're talking six figures. There was a blogpost about this a while back on the startpage of HN.
for finetuning or parameter training from scratch?
It feels absolutely amazing to build an AI startup right now. It's as if your product automatically becomes cheaper, more reliable, and more scalable with each new major model release. - We first struggled with limited context windows [solved] - We had issues with consistent JSON ouput [solved] - We had rate limiting and performance issues for the large 3rd party models [solved] - Hosting our own OSS models for small…