Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Snowflake Arctic Instruct (128x3B MoE), largest open source model

replicate.com

Post reply on HN
Print view View original

71–80 of 224 posts

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#71

Post by blackeyeblitzar »

WhitneyLand wrote: post #21
blackeyeblitzar wrote: post #4

Let’s stop using terms like open source falsely. The model isn’t open source, it is open weights. It’s good that the license for the weights is Apache, but for this model to be “truly open” they must release training data and source code under an OSI approved license. Otherwise it’s just misleading marketing. So far it seems like Snowflake will release some blog posts and “cookbooks”, whatever that means, but not act…

What’s the problem? This is what it says on their repo home page. ————— Truly Open: Apache 2.0 license provides ungated access to weights and code. In addition, we are also open sourcing all of our data recipes and research insights.

The source code they're talking about is not the training code. The only thing I saw released was their inference code and weights. You can verify this by visiting the following:

https://github.com/Snowflake-Labs/snowflake-arctic/tree/main

https://huggingface.co/Snowflake/snowflake-arctic-base

https://huggingface.co/Snowflake/snowflake-arctic-instruct

To put it another way, when they share the weights for the model, that's like sharing the compiled output for some software - like releasing an executable instead of the source code that can produce the executable. They aren't sharing the things you need to produce the weights (the training code, training data, any preprocessing code, etc). Without those inputs you actually cannot even audit or verify how the model works. The team making the model might bias the model in all sorts of ways without your knowledge.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#72

Post by blackeyeblitzar »

jerrygenser wrote: post #22
blackeyeblitzar wrote: post #4

Let’s stop using terms like open source falsely. The model isn’t open source, it is open weights. It’s good that the license for the weights is Apache, but for this model to be “truly open” they must release training data and source code under an OSI approved license. Otherwise it’s just misleading marketing. So far it seems like Snowflake will release some blog posts and “cookbooks”, whatever that means, but not act…

> they must release training data and source code under an OSI approved license The source code is also apache 2.0

Snowflake has only released the inference code - meaning the code you need to "run" the model. So if you take the weights they have released (which is the model that is a result of training), you can host the weights and inference code, and feed prompts to it, to get answers. But you don't have the actual source code you need to produce the weights in the first place.

As an example of what open source actually means for LLMs, you can look at what AI2 does with their OLMo model (https://allenai.org/olmo), where each model that they release comes with:

> Full training data used for these models, including code that produces the training data, from AI2’s Dolma, and WIMBD for analyzing pretraining data.

> Full model weights, training code, training logs, training metrics in the form of Weights & Biases logs, and inference code.

> 500+ checkpoints per model, from every 1000 steps during the training process, available as revisions on HuggingFace.

> Evaluation code under the umbrella of AI2’s Catwalk and Paloma.

> Fine-tuning code and adapted models (with Open Instruct)

> All code, weights, and intermediate checkpoints are released under the Apache 2.0 License.

OLMo is what "truly open" is, while the rest is openwashing and marketing.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#74

Post by kaibee »

cornholio wrote: post #69
oceanplexian wrote: post #31

Earlier quoted context omitted.

Flights from the Western USA to Hawaii are ~2 million tons a year, at least in 2017, wouldn’t be surprised if that number doubled. 500t to train a model at least seems like a more productive use of carbon than spending a few days on the beach. So I don’t think the carbon use of training models is that extreme.

GPT3 was a 175 bln parameters model. All the big boys are now doing trillions of parameters without a substantial chip efficiency increase. So we are talking about thousands of tons of carbon per model, repeated every year or two or however fast they become obsolete. To that we need to add embedded carbon in the entire hardware stack and datacenter, it quickly adds up. If it's just a handfull of companies doing it, f…

AI models don’t care if the electricity comes from renewable sources. Renewables are cheaper than fossil fuels at this point and getting cheaper still. I feel a lot better about a world where we consume 10x the energy but it comes from renewables than one where we only consume 2x but the lack of demand limits investment in renewables.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#75

Post by kaibee »

N0b8ez wrote: post #70
rvnx wrote: post #9

Earlier quoted context omitted.

Even before that, elections are coming end of the year, chat bots are great for telling whom to vote for. 2020's elections costed 15B USD in total, so we can't afford to lose (we are the good guys, right ?)

How will the LLMs be used for this? They can't solve captchas, and they're not smart enough to navigate the internet by themselves. All they do is generate text.

Transformers can definitely solve captchas. Not sure why you think otherwise.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#76

Post by bugbuddy »

a13n wrote: post #64
cs702 wrote: post #2

Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…

Seems like capitalism is doing its thing here. The potential future revenue from having the best model is presumably in the trillions.

L0L Trillions ROFL

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#77

Post by refulgentis »

coder543 wrote: post #36
chessgecko wrote: post #3

This is the sparsest model thats been put out in a while (maybe ever, kinda forget the shapes of googles old sparse models). This probably wont be a great tradeoff for chat servers, but could be good for local stuff if you have 512GB of ram with your cpu.

It has 480B parameters total, apparently. You would only need 512GB of RAM if you were running at 8-bit. It could probably fit into 256GB at 4-bit, and 4-bit quantization is broadly accepted as a good trade-off these days. Still... that's a lot of memory. EDIT: This[0] confirms 240GB at 4-bit. [0]: https://github.com/ggerganov/llama.cpp/issues/6877#issue-226...

Yeah, and usually GPU RAM, unless you enjoy waiting for a minute for filling the context :(

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#78

Post by kaibee »

msp26 wrote: post #52
sp332 wrote: post #37

Earlier quoted context omitted.

Yeah but that's a 70B model. You can see on the Inference Efficiency chart that it takes more than 3x as much compute to run it compared to this one.

Most people are vram constrained not compute constrained.

Cloud providers aren’t though.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#79

Post by adrien-treuille »

mritchie712 wrote: post #28

llama3 narrowly beats arctic at SQL generation (80.2 vs 79.0) and Mixtral 8x22B scored 79.2. You'd think SQL would be the one thing they'd be sure to smoke other models on. 0 - https://www.snowflake.com/blog/arctic-open-efficient-foundat...

Actually, Snowflake doesn’t use Arctic for SQL codegen internally. They use a different model chained with mistral-large… and they do smoke the competition. https://medium.com/snowflake/1-1-3-how-snowflake-and-mistral...

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#80

Post by kaibee »

coder543 wrote: post #36
chessgecko wrote: post #3

This is the sparsest model thats been put out in a while (maybe ever, kinda forget the shapes of googles old sparse models). This probably wont be a great tradeoff for chat servers, but could be good for local stuff if you have 512GB of ram with your cpu.

It has 480B parameters total, apparently. You would only need 512GB of RAM if you were running at 8-bit. It could probably fit into 256GB at 4-bit, and 4-bit quantization is broadly accepted as a good trade-off these days. Still... that's a lot of memory. EDIT: This[0] confirms 240GB at 4-bit. [0]: https://github.com/ggerganov/llama.cpp/issues/6877#issue-226...

I know quantizing larger models seems to be more forgiving but I’m wondering if that applies less to these extreme-MoE models. It seems to be that it should be more like quantizing a 3B model.
Post reply on HN