Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Snowflake Arctic Instruct (128x3B MoE), largest open source model

replicate.com

Post reply on HN
Print view View original

31–40 of 224 posts

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#31

Post by oceanplexian »

cornholio wrote: post #14
cs702 wrote: post #2

Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…

The model seems to be " build something fast, get users, engagement, and venture capital, hope you can grow fast enough to still be around after the Great AI cull ". > offers over 0.6 million different pretrained open models. One estimate I saw was that training GPT3 released 500 tons of CO2 back in 2020. Out of those 600k models, at least hundreds are of a comparable complexity. I can only hope building large models…

Flights from the Western USA to Hawaii are ~2 million tons a year, at least in 2017, wouldn’t be surprised if that number doubled.

500t to train a model at least seems like a more productive use of carbon than spending a few days on the beach. So I don’t think the carbon use of training models is that extreme.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#32

Post by mritchie712 »

mike_hearn wrote: post #23
ur-whale wrote: post #17

However big it may be, it still hallucinates very, very badly. I just asked it an economics question and asked it to cite its sources. All the links provided as sources were complete BS. Color me unimpressed.

It's intended for SQL generation and similar with cheap fine tuning and inference, not answering general knowledge questions. Their blog post is pretty clear about that. If you just want a chatbot this isn't the model for you. If you want to let non-SQL trained people ask questions of your data, it might be really useful.

It's worse at SQL generation than llama3 according to their own post.

https://www.snowflake.com/blog/arctic-open-efficient-foundat...

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#34

Post by margalabargala »

ReptileMan wrote: post #26
cornholio wrote: post #14

Earlier quoted context omitted.

The model seems to be " build something fast, get users, engagement, and venture capital, hope you can grow fast enough to still be around after the Great AI cull ". > offers over 0.6 million different pretrained open models. One estimate I saw was that training GPT3 released 500 tons of CO2 back in 2020. Out of those 600k models, at least hundreds are of a comparable complexity. I can only hope building large models…

>One estimate I saw was that training GPT3 released 500 tons of CO2 back in 2020 So absolute nothing in the grand scheme of things?

That's the amount that would be released by burning 50,000 gallons of gas, which is about that ten typical cars will burn throughout their entire lifespan.

Done once, I agree, that's very little.

But if each of those 600,000 other models used that much (or even a tenth that much), then that now becomes impactful.

Releasing 500 tons of CO2 600,000 times over would amount to about 1% of all human global annual emissions.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#35

Post by mlsu »

cs702 wrote: post #2

Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…

Far fewer than 600,000 of those are pretrained. Most are finetuned which is much easier. You can finetune a 7B model on gamer cards.

There is basically the big guys that everyone's heard of (google, meta, microsoft/openAI, and anthropic) and then a handful of smaller players who are training foundation models mostly so that they can prove to VCs that they are capable of doing so -- to acquire more funding/access to compute so that they may eventually dethrone openAI and take a piece of the multi-billion dollar "enterprise AI" market for themselves.

Below that, there is a frothing ocean of mostly 7B finetunes created mostly by individuals who want to jailbreak base models for... reasons, plus the occasional research group.

The most oddball one I have seen is the databricks LLM which seems to have been an exercise of pure marketing. Those I suspect will disappear when the bubble deflates a bit.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#36

Post by coder543 »

chessgecko wrote: post #3

This is the sparsest model thats been put out in a while (maybe ever, kinda forget the shapes of googles old sparse models). This probably wont be a great tradeoff for chat servers, but could be good for local stuff if you have 512GB of ram with your cpu.

It has 480B parameters total, apparently. You would only need 512GB of RAM if you were running at 8-bit. It could probably fit into 256GB at 4-bit, and 4-bit quantization is broadly accepted as a good trade-off these days. Still... that's a lot of memory.

EDIT: This[0] confirms 240GB at 4-bit.

[0]: https://github.com/ggerganov/llama.cpp/issues/6877#issue-226...

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#37

Post by sp332 »

mritchie712 wrote: post #28

llama3 narrowly beats arctic at SQL generation (80.2 vs 79.0) and Mixtral 8x22B scored 79.2. You'd think SQL would be the one thing they'd be sure to smoke other models on. 0 - https://www.snowflake.com/blog/arctic-open-efficient-foundat...

Yeah but that's a 70B model. You can see on the Inference Efficiency chart that it takes more than 3x as much compute to run it compared to this one.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#39

Post by TrueDuality »

cs702 wrote: post #2

Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…

Most of those are fine tuned variants of open base models and shouldn't be included in the "every tech company" thing you're trying to communicate. Most of those are researcher or engineers learning how to work with these models, or are training them on specific data sets to improve their effectiveness in a particular task.

These fine tunes are not a huge amount of compute, most of them are doing these trainings on a single personal machine over a day or so of effort, NOT the six+ months across a massive cluster it takes to make a good base model.

That isn't wasted effort either. We need to know how to use these tools effectively, they're not going away. It's a very reductionist and inaccurate view of the world you're peddling in that comment.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#40

Post by bee_rider »

cornholio wrote: post #14
cs702 wrote: post #2

Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…

The model seems to be " build something fast, get users, engagement, and venture capital, hope you can grow fast enough to still be around after the Great AI cull ". > offers over 0.6 million different pretrained open models. One estimate I saw was that training GPT3 released 500 tons of CO2 back in 2020. Out of those 600k models, at least hundreds are of a comparable complexity. I can only hope building large models…

I wonder what is greater, the CO2 produced by training AI models, the CO2 produced by researchers flying around to talk about AI models, or the CO2 produced by private jets funded by AI investments.
Post reply on HN