Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mixtral 8x22B

mistral.ai

Post reply on HN
Print view View original

241–250 of 252 posts

Re: Mixtral 8x22B

#241

Post by zone411 »

J_Shelby_J wrote: post #227
zone411 wrote: post #203

It ranks between Mistral Small and Mistral Medium on my NYT Connections benchmark and is indeed better than Command R Plus and Qwen 1.5 Chat 72B, which were the top two open weights models. Grok 1.0 is not an instruct model, so it cannot be compared fairly.

Can you share the details about the benchmark?

Uses an archive of 267 NYT Connections puzzles (try them yourself if unfamiliar). Three different 0-shot prompts, words in both lowercase and uppercase. One attempt per puzzle. Partial credit is awarded if not all lines are solved correctly. Top humans get near 100.

Most other benchmarks don't clearly show the difference between the top models and the rest. This may be because they are older and have been over-optimized or perhaps because they are just easier.

Re: Mixtral 8x22B

#242

Post by zone411 »

arnaudsm wrote: post #211
zone411 wrote: post #202

Earlier quoted context omitted.

LMSYS leaderboard is just one benchmark (that I think is fundamentally flawed). GPT-4 is clearly better.

Which alterative benchmarks do you recommend?

I don't have a perfect solution, except for the obvious answer that the best we can do is a combination of multiple benchmarks. It's harder now than ever because you also want to test long contexts, and older benchmarks are over-optimized. I think it would be best to overweigh newer benchmarks and underweigh benchmarks on which objectively dumb models like Claude 3 Haiku score well. And I have a big problem with HumanEval, which is Python-only and has only 164 problems but is used as the catch-all for coding abilities.

Re: Mixtral 8x22B

#243

Post by astrodust »

MacsHeadroom wrote: post #164
astrodust wrote: post #93

Earlier quoted context omitted.

$1500 each, plus the server they go in, plus plus plus plus.

Sure, but it's still a lot less than 8 h100s. ~$8k for an LLM server with 128GB of VRAM vs like $250k+ for 8 H100s.

$8K sounds like a lot more than "free".

Re: Mixtral 8x22B

#244

Post by PeterStuer »

Taking a moment to thank Mistral for coming through with the open release. In a just world Mistral and Meta would be the ones being decorated with the 'Open AI' medal. For now in AI, if it has 'Open' in the name, it isn't.

Re: Mixtral 8x22B

#245

Post by Sohcahtoa82 »

mysteria wrote: post #139
orost wrote: post #119

Earlier quoted context omitted.

That's not the model this post is about. You used the base model, not trained for tasks. (The instruct model is probably not on ollama yet.)

Yeah this is exactly what happens when you ask a base model a question. It'll just attempt to continue what you already wrote based off its training set, so if you say have it continue a story you've written it may wrap up the story and then ask you to subscribe for part 2, followed by a bunch of social media comments with reviews.

It can be fun, though, to prompt a text completion with something like "I'm thinking about" and just seeing what random thing it completes it with.

Re: Mixtral 8x22B

#246

Post by m463 »

PeterStuer wrote: post #244

Taking a moment to thank Mistral for coming through with the open release. In a just world Mistral and Meta would be the ones being decorated with the 'Open AI' medal. For now in AI, if it has 'Open' in the name, it isn't.

Reminds me of open source vs free software/GPL years ago, when the internet was taking off.

folks like ESR were weak with their definition of open source and it wasn't as good.

stallman said:

... the obvious meaning for the expression “open source software” is “You can look at the source code.” This is a much weaker criterion than free software ...

https://www.gnu.org/philosophy/free-software-for-freedom.htm...

https://www.gnu.org/philosophy/open-source-misses-the-point....

Re: Mixtral 8x22B

#247

Post by Sohcahtoa82 »

kristopolous wrote: post #62
Workaccount2 wrote: post #46

Earlier quoted context omitted.

You can't upgrade it? Edit: I haven't owned a laptop for years, probably could have surmised they'd be more user hostile nowadays.

Everything is soldered in these days. It's complete garbage. And most of the other vendors just copy Apple so even things like Lenovo have the same problems. The current state of laptops is such trash

> And most of the other vendors just copy Apple

I don't think it's so much "copying Apple" as much as it's learning that they can do things like solder in the RAM to cut costs and discovering that most people won't care and will still buy it.

Re: Mixtral 8x22B

#248

Post by sleepingreset »

ajcp wrote: post #222
sleepingreset wrote: post #53

Earlier quoted context omitted.

If you don't mind, I'm trying to experiment w/ local models more. Just now getting into messing w/ these but I'm struggling to come up w/ good use cases. Would you happen to know of any cool OSS model projects that might be good inspiration for a side project? Wondering what most people use these local models for

I've got a couple I've done and it's been really enjoyable. I think the real value in using local models is exposing them to personal/unique information that only you have, thus getting novel and unique outcomes that no public model could provide. 1. Project 1 - Self Knowledge - Download/extract all of my emails and populate into a vector database, like Chroma[0] - For each prompt do a search of the vector store and…

thank you tons. this is a great start and I'm gonna use these for learning :)

u rock

Re: Mixtral 8x22B

#249

Post by ajcp »

sleepingreset wrote: post #248
ajcp wrote: post #222

Earlier quoted context omitted.

I've got a couple I've done and it's been really enjoyable. I think the real value in using local models is exposing them to personal/unique information that only you have, thus getting novel and unique outcomes that no public model could provide. 1. Project 1 - Self Knowledge - Download/extract all of my emails and populate into a vector database, like Chroma[0] - For each prompt do a search of the vector store and…

thank you tons. this is a great start and I'm gonna use these for learning :) u rock

Pleasure is all mine :D Have fun and please reach out if you get stuck!

Re: Mixtral 8x22B

#250

Post by octobus2021 »

kristopolous wrote: post #62
Workaccount2 wrote: post #46

Earlier quoted context omitted.

You can't upgrade it? Edit: I haven't owned a laptop for years, probably could have surmised they'd be more user hostile nowadays.

Everything is soldered in these days. It's complete garbage. And most of the other vendors just copy Apple so even things like Lenovo have the same problems. The current state of laptops is such trash

> Everything is soldered in these days.

[...insert sound of old timers laughing at newbies who declared T480/T490 overhyped for no reason because "no one care about upgradability anymore"...]

:)

Post reply on HN