Isn't equating active parameters with cost a little unfair since you still need full memory for all the inactive parameters?
Fewer parameters at inference time makes a massive difference in cost for batch jobs, assuming vram usage is the same
Mixtral 8x22B
181–190 of 252 posts
Re: Mixtral 8x22B
#182It feels absolutely amazing to build an AI startup right now. It's as if your product automatically becomes cheaper, more reliable, and more scalable with each new major model release. - We first struggled with limited context windows [solved] - We had issues with consistent JSON ouput [solved] - We had rate limiting and performance issues for the large 3rd party models [solved] - Hosting our own OSS models for small…
> It's as if your product automatically becomes cheaper, more reliable, and more scalable with each new major model release.
and so do your competitor's products.
Re: Mixtral 8x22B
#183How does this compare to ChatGPT4?
Re: Mixtral 8x22B
#184I just find it hilarious how approximately 100% of models beat all other models on benchmarks.
Benchmarks are often weird because of what a benchmark inherently needs to be.
If you compare LLMs by asking them to tell you how to catch dragonflies - the free text chat answer you get will be impossible to objectively evaluate.
Whereas if you propose four ways to catch dragonflies and ask each model to choose option A, B, C or D (or check the relative probability the model assigns to those four output logits) the result is easy to objectively evaluate - you just check if it chose the one right answer.
Hence a lot of the most famous benchmarks are multiple-choice questions - even though 99.9% of LLM usage doesn't involve answering multiple-choice questions.
Re: Mixtral 8x22B
#185Earlier quoted context omitted.
It would make sense, wouldn't it? Just as we've seen rising fuel efficiency, safety, dependability, etc. over the lifecycle of a particular car model. The different teams are learning from each other and pushing boundaries; there's virtually no reason for any of the teams to release a model or product that is somehow inferior to a prior one (unless it had some secondary attribute such as requiring lower end hardware)…
That's a valid theory, a priori, but if you actually follow up you'll find that the vast majority of these benchmark results don't end up matching anyone's subjective experience with the models. The churn at the top is not nearly as fast as the press releases make it out to be.
Subjective experience is not a benchmark that you can measure success against. Also, of course new models are better on some set of benchmarks. Why would someone bother releasing a "new" model that is inferior to old ones? (Aside from attributes like more preferable licensing).
This is completely normal, the opposite would be strange.
Re: Mixtral 8x22B
#186Seems that Perplexity Labs already offers a free demo of it. https://labs.perplexity.ai/
That's the old/regular model. This post is about the new "instruct" model.
Re: Mixtral 8x22B
#187I'm really excited about this model. Just need someone to quantize it to ~3 bits so it'll run on a 64GB MacBook Pro. I've gotten a lot of use from the 8x7b model. Paired with llamafile and it's just so good.
Shopping for a new mbp. Do you think going with more ram would be wise?
Re: Mixtral 8x22B
#188"64K tokens context window" I do wish they had managed to extend it to at least 128K to match the capabilities of GPT-4 Turbo Maybe this limit will become a joke when looking back? Can you imagine reaching a trillion tokens context window in the future, as Sam speculated on Lex's podcast?
How useful is such a large input window when most of the middle isn't really used? I'm thinking mostly about coding. But wheb putting even say 20k tokens into the input, a good chunk doesn't seem to be "remembered" or used for the output
Re: Mixtral 8x22B
#189Earlier quoted context omitted.
Ollama with WebUI https://github.com/open-webui/open-webui
Not sure why your comment was downvoted. ^ is absolutely the right answer. Open WebUI is functionally identical to the ChatGPT interface. You can even use it with the OpenAI APIs to have your own pay per use GPT 4. I did this.
Hey can you guys elaborate how this works? I'm looking at the Ollama section in their docs and it talks about load balancing? I don't understand what that means in this context.
Re: Mixtral 8x22B
#190Earlier quoted context omitted.
FWIW, the 128k context window for GPT-4 is only for input. I believe the output content is still only 4k.
How does that make any sense on a decoder-only architecture?
It's not about the model. The model can output more - it's about the API.
A better phrasing would be that they don't allow you to output more than 4k tokens per message.
Same with Anthropic and Claude, sadly.