Snowflake Arctic Instruct (128x3B MoE), largest open source model
1–10 of 224 posts
Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model
#2Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that:
https://www.snowflake.com/blog/arctic-open-efficient-foundat...
But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is overwhelming. As of right now, Huggingface is hosting over 600,000 different pretrained open models.
Lots of money has been forever burned training or finetuning all those open models. Even more money has been forever burned training or finetuning all the models that have not been publicly released. It's like a giant bonfire, with Nvidia supplying most of the (very expensive) chopped wood.
Who's going to recoup all that investment? When? How? What's the rationale for releasing all these models to the public? Do all these tech companies know something we don't? Why are they doing this?
---
EDIT: Changed "0.6 million" to "600,000," which seems clearer. Added "or finetuning".
Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model
#3This is the sparsest model thats been put out in a while (maybe ever, kinda forget the shapes of googles old sparse models). This probably wont be a great tradeoff for chat servers, but could be good for local stuff if you have 512GB of ram with your cpu.
Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model
#4Let’s stop using terms like open source falsely. The model isn’t open source, it is open weights. It’s good that the license for the weights is Apache, but for this model to be “truly open” they must release training data and source code under an OSI approved license. Otherwise it’s just misleading marketing. So far it seems like Snowflake will release some blog posts and “cookbooks”, whatever that means, but not actual training source code. Only the inference code is open source here, which is uninteresting.
Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model
#5Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…
> Who's going to recoup all that investment? When? How? What's the long-term strategy AI of all these tech companies? Do they know something we don't?
The first droid armies will rapidly recoup the cost when the final wars for world domination begin…
Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model
#6I suppose more and smaller experts would also help reduce over-fitting?
Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model
#7Abnormally large. I don't see the cost/performance numbers going well for this one.
Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model
#8Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…
Why 0.6 million and not +600k ?
Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model
#9Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…
> Who's going to recoup all that investment? When? How? What's the long-term strategy AI of all these tech companies? Do they know something we don't? The first droid armies will rapidly recoup the cost when the final wars for world domination begin…
Even before that, elections are coming end of the year, chat bots are great for telling whom to vote for.
2020's elections costed 15B USD in total, so we can't afford to lose (we are the good guys, right ?)
Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model
#10Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…
> over 0.6 million
What a peculiar way to say: 600,000