Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Snowflake Arctic Instruct (128x3B MoE), largest open source model

replicate.com

Post reply on HN
Print view View original

81–90 of 224 posts

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#81

Post by modeless »

cs702 wrote: post #2

Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…

These projects all started a long time ago, I expect, and they're all finishing now. Now that there are so many models, people will hopefully change focus from training new duplicate language models to exploring more interesting things. Multimodal, memory, reasoning.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#82

Post by coder543 »

kaibee wrote: post #80
coder543 wrote: post #36

Earlier quoted context omitted.

It has 480B parameters total, apparently. You would only need 512GB of RAM if you were running at 8-bit. It could probably fit into 256GB at 4-bit, and 4-bit quantization is broadly accepted as a good trade-off these days. Still... that's a lot of memory. EDIT: This[0] confirms 240GB at 4-bit. [0]: https://github.com/ggerganov/llama.cpp/issues/6877#issue-226...

I know quantizing larger models seems to be more forgiving but I’m wondering if that applies less to these extreme-MoE models. It seems to be that it should be more like quantizing a 3B model.

4-bit is fine for models of all sizes, in my experience.

The only reason I personally don’t quantize tiny models very much is because I don’t have to, not because the accuracy gains from running at 8-bit or fp16 are that great. I tried out 4-bit Phi-3 yesterday, and it was just fine.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#83

Post by grahamgooch »

mlsu wrote: post #35
cs702 wrote: post #2

Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…

Far fewer than 600,000 of those are pretrained. Most are finetuned which is much easier. You can finetune a 7B model on gamer cards. There is basically the big guys that everyone's heard of (google, meta, microsoft/openAI, and anthropic) and then a handful of smaller players who are training foundation models mostly so that they can prove to VCs that they are capable of doing so -- to acquire more funding/access to c…

600k?

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#84

Post by PaulHoule »

nerpderp82 wrote: post #68
PaulHoule wrote: post #24

It got the right answer for "Who is Tim Bray?" but it got "Who is Worsel the Dragon?" wrong.

Looks like they aren't targeting DRGN24 as one of their benchmark suites.

I love getting into arguments with LLMs over whether Worsel is an eastern dragon (in my imagination) or a western dragon (like the bad lensman anime.)

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#85

Post by jrm4 »

cs702 wrote: post #2

Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…

This seems to me to be the simple story of "capitalism, having learned from the past, undertands that free/open source is actually advantageous for the little guys."

Which is to say, "everyone" knows that this stuff has a lot of potential. Everyone is also used to what often happens in tech, which is outrageous winner-take-all scale effects. Everyone ALSO knows that there's almost certainly little MARGINAL difference between what the big guys will be able to do and and what the little guys can do on their own ESPECIALLY if they essentially 'pool their knowledge.'

So, I suppose it's the whole industry collectively and subconsciously preventing e.g. OpenAI/ChatGPT becoming the Microsoft of AI.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#86

Post by klysm »

isoprophlex wrote: post #48
klysm wrote: post #38

Earlier quoted context omitted.

Why on earth is trading onion futures illegal in the us

I looked it up, the story is pretty hilarious. https://en.m.wikipedia.org/wiki/Onion_Futures_Act

Wow I’m surprised the reaction was to ban futures on just onions specifically due to some market manipulation occurring. Surely this kind of manipulation wasn’t restricted to just onions? It seems incredibly short sighted

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#87

Post by bee_rider »

01HNNWZ0MV43FF wrote: post #46
bee_rider wrote: post #40

Earlier quoted context omitted.

I wonder what is greater, the CO2 produced by training AI models, the CO2 produced by researchers flying around to talk about AI models, or the CO2 produced by private jets funded by AI investments.

Institute a carbon tax and I'm sure we'll find out soon enough

For sure; I didn’t realize sensible systemic reforms were on the table.

I’m not sure if any of these things would be the first on the chopping block if a carbon tax were implemented, but it is worth a shot.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#88

Post by mritchie712 »

adrien-treuille wrote: post #79
mritchie712 wrote: post #28

llama3 narrowly beats arctic at SQL generation (80.2 vs 79.0) and Mixtral 8x22B scored 79.2. You'd think SQL would be the one thing they'd be sure to smoke other models on. 0 - https://www.snowflake.com/blog/arctic-open-efficient-foundat...

Actually, Snowflake doesn’t use Arctic for SQL codegen internally. They use a different model chained with mistral-large… and they do smoke the competition. https://medium.com/snowflake/1-1-3-how-snowflake-and-mistral...

smoke? it's the same as gpt4

https://medium.com/snowflake/1-1-3-how-snowflake-and-mistral...

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#89

Post by blackeyeblitzar »

dantheman wrote: post #30
blackeyeblitzar wrote: post #4

Let’s stop using terms like open source falsely. The model isn’t open source, it is open weights. It’s good that the license for the weights is Apache, but for this model to be “truly open” they must release training data and source code under an OSI approved license. Otherwise it’s just misleading marketing. So far it seems like Snowflake will release some blog posts and “cookbooks”, whatever that means, but not act…

Why would they need to release the training data? that's nonsense.

Open source for traditional software means that you can see how the software works and reproduce the executable by compiling the software from source code. For LLMs, reproducing the model means reproducing the weights. And to do that you need the training source code AND the training data. There are already other great models that do this (see my comment at https://news.ycombinator.com/item?id=40147298).

I get that there may be some training data that is proprietary and cannot be released. But in those scenarios, it would still be good to know what the data is, how it was curated or filtered (this greatly affects LLM performance), how it is weighted relative to other training data, and so forth. But a significant portion of data used to train models is not proprietary and in those cases they can simply link to that data elsewhere or release it themselves, which is what others have done.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#90

Post by N0b8ez »

kaibee wrote: post #75
N0b8ez wrote: post #70

Earlier quoted context omitted.

How will the LLMs be used for this? They can't solve captchas, and they're not smart enough to navigate the internet by themselves. All they do is generate text.

Transformers can definitely solve captchas. Not sure why you think otherwise.

So captchas are obsolete now?
Post reply on HN