Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Snowflake Arctic Instruct (128x3B MoE), largest open source model

replicate.com

Post reply on HN
Print view View original

181–190 of 224 posts

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#181

Post by genewitch »

jasongill wrote: post #104
salomonk_mur wrote: post #96

Earlier quoted context omitted.

Yeah, I wouldn't say the benefits were marginal at all. CPUs went from dozens of MHz in the 90's to over 4 GHz nowadays.

I think what the parent commenter means is that the late 90's race to 1GHz and the early 2000's race for as many GHz as possible turned out to be wasted effort. At the time, ever week it seemed like AMD or Intel would announce a new CPU that was a few MHz faster than the competition, and the assumption among the Slashdot crowd was basically that we'd have 20GHz CPU's by now. Instead, there was a plateau in terms of C…

it's been well known (i'd heard it numerous times) that the maximum clock speed of x86 is somewhere Once single core single threaded CPUs hit ~4GHz the new "frontier" was the core 2 duo, then the core 2 quad, and now we have desktop chips with 16c/32t (and beyond).

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#182

Post by LordDragonfang »

temuze wrote: post #151
cs702 wrote: post #2

Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…

In the short-term, these kinds of investments can hype up a stock and create a small bump. However, in the long-term, as the hype dies down, so will the stock prices. At the end of the day, I think it will be a transfer of wealth from shareholders to Nvidia and power companies.

I just wish that AMD (and, pie in the sky, Intel) had gotten their shit together enough that these flaming dumptrucks full of money would have actually resulted in a competitive GPU market.

Honestly, Zuckerburg (seemingly the only CEO willing to actually invest in an open AI ecosystem for the obvious benefits it brings them) should just invest a few million into hiring a few real firmware hackers to port all the ML CUDA code into an agnostic layer that AMD can build to.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#183

Post by qeternity »

giantrobot wrote: post #99
karmasimida wrote: post #56

Earlier quoted context omitted.

But you do need to hold all 128 experts in memory? Or not? Or they simply consider inference efficiency as latency

I believe the main draw of the MoE model is they don't all need to be in memory at once. They can be swapped based on context. In aggregate you get the performance of a much larger model (384b tokens) while using much less memory than such a model would require. If you had enough memory it could all be loaded but it doesn't need to be.

"Expert" in MoE has no bearing on what you might think of as a human expert.

It's not like there is one expert that is proficient at science, and one that is proficient in history.

For a given inference request, you're likely to activate all the experts at various points. But for each individual forward pass (e.g. each token), you are only activating a few.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#184

Post by LordDragonfang »

paxys wrote: post #176
isoprophlex wrote: post #48

Earlier quoted context omitted.

I looked it up, the story is pretty hilarious. https://en.m.wikipedia.org/wiki/Onion_Futures_Act

> The Onion Futures Act is a United States law banning the trading of futures contracts on onions as well as "motion picture box office receipts" Wut

To use a metaphor more appropriate to this site, US legal system is the smelliest, most hack-and-bodge-filled legacy codebase most people will ever interact with.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#185

Post by Zambyte »

ko27 wrote: post #157
Zambyte wrote: post #150

Earlier quoted context omitted.

Training data is a build dependency for the weights. You cannot realistically get the same weights without the same training data.

Developer's mindset, knowledge and tooling is also a build dependency for any open source code. You can not realistically get the same code without it.

> you can not realistically get the same code without it

You mean the same source code? Because... I agree. That's why it's important for the source to be open. Both in the context of software and language models.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#186

Post by hnthrowaway9812 »

mlsu wrote: post #35
cs702 wrote: post #2

Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…

Far fewer than 600,000 of those are pretrained. Most are finetuned which is much easier. You can finetune a 7B model on gamer cards. There is basically the big guys that everyone's heard of (google, meta, microsoft/openAI, and anthropic) and then a handful of smaller players who are training foundation models mostly so that they can prove to VCs that they are capable of doing so -- to acquire more funding/access to c…

You've nerdsniped me so hard that I had to make an account.

There are DOZENS of orgs releasing foundational models, not "a handful."

Salesforce, EleuthierAI, NVIDIA, Amazon, Stanford, RedPajama, Cohere, Mistral, MosaicML, Yandex, Huawei StabilityLM, ...

https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp...

It's completely bonkers and a huge waste of resources. Most of them will see barely any use at all.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#187

Post by hnthrowaway9812 »

DowagerDave wrote: post #110
cs702 wrote: post #2

Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…

Snowflake has a pretty good story in this space: "Your data is already in our cloud, so governance and use is a solved problem. Now use our AI (and burn credits)". This is a huge pain-point if you're thinking about ML with your (probably private) data. It's less clear if this entices companies to move INTO Snowflake IMO And streamlit, if you're as old as me, looks an awful lot like a MS-Access application for today.…

Snowflake could have the same story by hosting Llama 3 which is probably more efficient/better.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#188

Post by hnthrowaway9812 »

LordDragonfang wrote: post #182
temuze wrote: post #151

Earlier quoted context omitted.

In the short-term, these kinds of investments can hype up a stock and create a small bump. However, in the long-term, as the hype dies down, so will the stock prices. At the end of the day, I think it will be a transfer of wealth from shareholders to Nvidia and power companies.

I just wish that AMD (and, pie in the sky, Intel) had gotten their shit together enough that these flaming dumptrucks full of money would have actually resulted in a competitive GPU market. Honestly, Zuckerburg (seemingly the only CEO willing to actually invest in an open AI ecosystem for the obvious benefits it brings them) should just invest a few million into hiring a few real firmware hackers to port all the ML C…

Groq seems to be well positioned to give Nvidia a run for their money, actually.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#189

Post by hnthrowaway9812 »

a13n wrote: post #64
cs702 wrote: post #2

Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…

Seems like capitalism is doing its thing here. The potential future revenue from having the best model is presumably in the trillions.

It doesn't seem like that's true at all.

If the "best model" only stays the best for a few months and if, during those few months, the second best model is near indistinguishable, then it will be extremely hard to extract trillions of dollars.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#190

Post by hnthrowaway9812 »

bfirsh wrote: post #19

If you want to have a conversation with it, here's a full chat app: https://arctic.streamlit.app/ Official blog post: https://www.snowflake.com/blog/arctic-open-efficient-foundat... Weights: https://huggingface.co/Snowflake/snowflake-arctic-instruct

It claims to have a knowledge cut-off of 2021. Not sure if its hallucinating or its true.

But when I asked it about the best LLMs it suggested GPT-3, Bert and T5!

Post reply on HN