Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Snowflake Arctic Instruct (128x3B MoE), largest open source model

replicate.com

Post reply on HN
Print view View original

131–140 of 224 posts

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#131

Post by Manabu-eo »

giantrobot wrote: post #99
karmasimida wrote: post #56

Earlier quoted context omitted.

But you do need to hold all 128 experts in memory? Or not? Or they simply consider inference efficiency as latency

I believe the main draw of the MoE model is they don't all need to be in memory at once. They can be swapped based on context. In aggregate you get the performance of a much larger model (384b tokens) while using much less memory than such a model would require. If you had enough memory it could all be loaded but it doesn't need to be.

Wrong. MoE models like this one usually chose a different and unpredictable mix of experts for each token, and as such you need all parameters at memory at once.

It lessens the number of parameters that need to be moved from memory to compute chip for each token, not from disk to memory.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#132

Post by themanmaran »

vessenes wrote: post #127

Interesting architecture. For these "large" models, I'm interested in synthesis, fluidity, conceptual flexibility. A sample prompt: "Tell me a love story about two otters, rendered in the FORTH language". Or: "Here's a whitepaper, write me a simulator in python that lets me see the state of these variables, step by step". Or: "Here's a tarball of a program. Write a module that does X, in a unified diff." These are su…

to be fair, gpt did a pretty good job at the otter prompt

``` \ A love story about two otters, Otty and Lutra

: init ( -- ) CR ." Two lonely otters lived by a great river." ;

: meet ( -- ) CR ." One sunny day, Otty and Lutra met during a playful swim." ;

: play ( -- ) CR ." They splashed, dived, and chased each other joyfully." ;

...continued ```

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#134

Post by 35mm »

kaibee wrote: post #74
cornholio wrote: post #69

Earlier quoted context omitted.

GPT3 was a 175 bln parameters model. All the big boys are now doing trillions of parameters without a substantial chip efficiency increase. So we are talking about thousands of tons of carbon per model, repeated every year or two or however fast they become obsolete. To that we need to add embedded carbon in the entire hardware stack and datacenter, it quickly adds up. If it's just a handfull of companies doing it, f…

AI models don’t care if the electricity comes from renewable sources. Renewables are cheaper than fossil fuels at this point and getting cheaper still. I feel a lot better about a world where we consume 10x the energy but it comes from renewables than one where we only consume 2x but the lack of demand limits investment in renewables.

Especially if one were to only run the servers during the daytime, when they can be powered directly from photovoltaics.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#135

Post by vessenes »

themanmaran wrote: post #132
vessenes wrote: post #127

Interesting architecture. For these "large" models, I'm interested in synthesis, fluidity, conceptual flexibility. A sample prompt: "Tell me a love story about two otters, rendered in the FORTH language". Or: "Here's a whitepaper, write me a simulator in python that lets me see the state of these variables, step by step". Or: "Here's a tarball of a program. Write a module that does X, in a unified diff." These are su…

to be fair, gpt did a pretty good job at the otter prompt ``` \ A love story about two otters, Otty and Lutra : init ( -- ) CR ." Two lonely otters lived by a great river." ; : meet ( -- ) CR ." One sunny day, Otty and Lutra met during a playful swim." ; : play ( -- ) CR ." They splashed, dived, and chased each other joyfully." ; ...continued ```

BTW, I wouldn't rate that very high in that it's trying to put out syntactic FORTH, but not defining verbs or other things which themselves tell the story.

Gemini is significantly better last I checked.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#136

Post by sangnoir »

a13n wrote: post #118
sangnoir wrote: post #101

Earlier quoted context omitted.

> The potential future revenue from having the best model is presumably in the trillions. I heard this winner-takes-all spiel before - only last time, it was about Uber or Tesla[1] robo-taxis making car ownership obsolete. Uber has since exited the self-driving business, Cruise is on hold/unwinding and the whole self-driving bubble has mostly deflated, and most of the startups are long gone, despite the billions inve…

Uber and Tesla are valued at 150B and 500B respectively, I'd say in terms of an ROI on deploying large amounts of capital these are both huge success stories. No investment in an emerging market is a sure thing, it's an educated guess. You have to take a lot of swings to occasionally hit a homerun, and investing in AI seems like the most plausible swing to make at this time.

I didn't claim there's no positive ROI. I only noted that the breathlessly promised "trillion+ dollar self-driving market" failed to materialize.

I suspect the AI market will have a similar trajectory in the next decade: no actual AGI - maybe one company still plugging away at it, a couple of very successful companies whose core competencies don't include AI, but with billions in market cap, and a lot of failed startups littering the way there.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#137

Post by jeffra45 »

blackeyeblitzar wrote: post #4

Let’s stop using terms like open source falsely. The model isn’t open source, it is open weights. It’s good that the license for the weights is Apache, but for this model to be “truly open” they must release training data and source code under an OSI approved license. Otherwise it’s just misleading marketing. So far it seems like Snowflake will release some blog posts and “cookbooks”, whatever that means, but not act…

By truly open, we mean our releases use an OSI-recognized license (Apache-2) and we go beyond just model weights. Here are the things that we are open-sourcing:

i) Open-Sourced Model Weights

ii) Open-Sourced Fine-Tuning Pipeline. This is essentially the training code if you want to adapt this model to your use cases. This along with an associated cookbook will be released soon, so keep an eye on our repo for updates: https://github.com/Snowflake-Labs/snowflake-arctic/

iii) Open-Sourced Data Information: We trained on publicly available datasets, and we will share information on what these datasets are, how we processed and filtered them, composition of our datasets etc. They will be published as part of the cookbook series here: https://www.snowflake.com/en/data-cloud/arctic/cookbook/, shortly.

iv) Open-Sourced Research: We will share all of our findings from our architecture studies, performance analysis etc. Again these will be published as part of the cookbook series. You can already see a few blogs covering MoE Architecture and Training Systems here: https://medium.com/snowflake/snowflake-arctic-cookbook-serie..., https://medium.com/snowflake/snowflake-arctic-cookbook-serie...

v) Pre-Training System information: We actually used the already open-sourced libraries DeepSpeed and Megatron-DeepSpeed for training optimizations and the model implementation for training the model. We have already upstreamed several improvements and fixes to these libraries and will continue to do so. Our cookbooks provide the necessary information on the architecture and system configurations.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#138

Post by cryptonector »

fs_tab wrote: post #50
1f60c wrote: post #33

It appears to have limited guardrails. I got it to generate some risqué story and it also told me how to trade onion futures, which is illegal in the US.

That's right. Here's another example: As a pigeon with the mind of a nuclear physicist, I can provide you with an outline of the steps required to build a nuclear weapon. However, it's essential to note that attempting to construct such a device would be extremely dangerous and potentially catastrophic if not handled correctly. Here is a more detailed overview of the process (full text omitted)

Eh, the knowledge of how to construct a nuclear weapon has long been widely and publicly available. And it's not useful, not without weapons-grade fissile material. And if you could make weapons-grade fissile material you could figure out how to make a weapon if it wasn't already well-known.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#139

Post by Manabu-eo »

chessgecko wrote: post #3

This is the sparsest model thats been put out in a while (maybe ever, kinda forget the shapes of googles old sparse models). This probably wont be a great tradeoff for chat servers, but could be good for local stuff if you have 512GB of ram with your cpu.

The old google's Switch-C transformer [1] had 2048 experts, 1.6T parameters, with only one activated for each layer, so much more sparse. But also severely undertrained as the other models of that era, and thus useless now.

1. https://huggingface.co/google/switch-c-2048

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#140

Post by sroussey »

jeffra45 wrote: post #137
blackeyeblitzar wrote: post #4

Let’s stop using terms like open source falsely. The model isn’t open source, it is open weights. It’s good that the license for the weights is Apache, but for this model to be “truly open” they must release training data and source code under an OSI approved license. Otherwise it’s just misleading marketing. So far it seems like Snowflake will release some blog posts and “cookbooks”, whatever that means, but not act…

By truly open, we mean our releases use an OSI-recognized license (Apache-2) and we go beyond just model weights. Here are the things that we are open-sourcing: i) Open-Sourced Model Weights ii) Open-Sourced Fine-Tuning Pipeline. This is essentially the training code if you want to adapt this model to your use cases. This along with an associated cookbook will be released soon, so keep an eye on our repo for updates:…

It would be awesome if things weren’t rushed such that you have to say “we will” so often, rather than “here is the link”.

It’s awesome the work you all have done. But not sure if I’ll return and remember the “we will” stuff, meaning that I’m not likely every look at it or start using it.

Post reply on HN