Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mixtral 8x22B

mistral.ai

Post reply on HN
Print view View original

231–240 of 252 posts

Re: Mixtral 8x22B

#231

Post by Zuiii »

jmorgan wrote: post #131
byteknight wrote: post #113

First test I tried to run a random taxation question through it Output: https://gist.github.com/IAmStoxe/7fb224225ff13b1902b6d172467... Within the first paragraph, it outputs: > GET AN ESSAY WRITTEN FOR YOU FROM AS LOW AS $13/PAGE Thought that was hilarious.

The `mixtral:8x22b` tag still points to the text completion model – instruct is on the way, sorry! Update: mixtral:8x22b now points to the instruct model: ollama pull mixtral:8x22b ollama run mixtral:8x22b

Wait. Isn't it a breaking change to change the underlying model like this? Wouldn't people start running into consistency issues in production? (given ollama appears to be oriented towards backend use)

Re: Mixtral 8x22B

#232

Post by PhilippGille »

neillyons wrote: post #141
hubraumhugo wrote: post #42

It feels absolutely amazing to build an AI startup right now. It's as if your product automatically becomes cheaper, more reliable, and more scalable with each new major model release. - We first struggled with limited context windows [solved] - We had issues with consistent JSON ouput [solved] - We had rate limiting and performance issues for the large 3rd party models [solved] - Hosting our own OSS models for small…

> We had issues with consistent JSON ouput [solved] It says the JSON output is constrained via their platform (on la Plateforme). Does that mean JSON output is only available in the hosted version? Are there any small models that can be self hosted that output valid JSON.

> Are there any small models that can be self hosted that output valid JSON.

Yes, for example this one is optimized for function calling and JSON output: https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B

Re: Mixtral 8x22B

#233

Post by ljhskyso »

pseudosavant wrote: post #157
mdrzn wrote: post #14

"64K tokens context window" I do wish they had managed to extend it to at least 128K to match the capabilities of GPT-4 Turbo Maybe this limit will become a joke when looking back? Can you imagine reaching a trillion tokens context window in the future, as Sam speculated on Lex's podcast?

FWIW, the 128k context window for GPT-4 is only for input. I believe the output content is still only 4k.

you can always put the unfinished output as the input to continue forever until reaching the full 128k context window

Re: Mixtral 8x22B

#234

Post by squigz »

moffkalast wrote: post #173
kiney wrote: post #17

Earlier quoted context omitted.

not that massive, we're talking six figures. There was a blogpost about this a while back on the startpage of HN.

6 figures are a massive pile of cash.

It's... really not, considering the audience here. Even less massive if 2-3 engineers get together to do it.

Re: Mixtral 8x22B

#235

Post by orra »

Zuiii wrote: post #231
jmorgan wrote: post #131

Earlier quoted context omitted.

The `mixtral:8x22b` tag still points to the text completion model – instruct is on the way, sorry! Update: mixtral:8x22b now points to the instruct model: ollama pull mixtral:8x22b ollama run mixtral:8x22b

Wait. Isn't it a breaking change to change the underlying model like this? Wouldn't people start running into consistency issues in production? (given ollama appears to be oriented towards backend use)

Sure, in theory. But if you move so fast that you already are running the base 8x22B model from last week, you can easily fix this.

I've long thought that if you want reproducibility and reliability, you need to pin your deps.

So, IMO, the change is very much worth it to reduce confusion going forward.

Re: Mixtral 8x22B

#236

Post by moffkalast »

squigz wrote: post #234
moffkalast wrote: post #173

Earlier quoted context omitted.

6 figures are a massive pile of cash.

It's... really not, considering the audience here. Even less massive if 2-3 engineers get together to do it.

It is considering what you get for it, and it's not lower end six figures and most likely seven. The JetMoe team released their training cost estimate and it took them $100k to train what's effectively a 2.2B model for 1.25 T tokens. Compare that to the still tiny Mistral 7B which is 3x larger and was trained on 4x more data you get a figure more around $1.7M. These are the absolute smallest production-viable LLMs.

For something like Mixtral 8X22B with 40B active params you'd looking at the $10M range, and if something gets screwed up during training you can be left with a dud and nothing to show for it, like LLama-2-33B. It's like buying millions worth of lootboxes and hoping something good drops.

Re: Mixtral 8x22B

#237

Post by rbut »

labs.perplexity.ai now has mixtral-8x22b-instruct.

I asked it what it's knowledge cutoff was, and it said 2021-09.

Anyone know why it's trained on such old data?

Re: Mixtral 8x22B

#240

Post by bavell »

sleepingreset wrote: post #53
hubraumhugo wrote: post #42

It feels absolutely amazing to build an AI startup right now. It's as if your product automatically becomes cheaper, more reliable, and more scalable with each new major model release. - We first struggled with limited context windows [solved] - We had issues with consistent JSON ouput [solved] - We had rate limiting and performance issues for the large 3rd party models [solved] - Hosting our own OSS models for small…

If you don't mind, I'm trying to experiment w/ local models more. Just now getting into messing w/ these but I'm struggling to come up w/ good use cases. Would you happen to know of any cool OSS model projects that might be good inspiration for a side project? Wondering what most people use these local models for

I'm experimenting with using them to help me make a mod for a game (Vic3). It has a lot of config files and I'm using AI to help generate the data structures and parsers/serializers.

It's coming along but very hit or miss with the small models I'm using (all my poor 6750XT 12GB can manage).

I could be doing something wrong though, even with GPT-4 im struggling a bit - it's very lazy and doesn't want to fully populate the data structure fields (the game file objects can haze dozens/hundredss of fields). I'm probably just not using the right incantation/magic phrase/voodoo?

Post reply on HN