Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Snowflake Arctic Instruct (128x3B MoE), largest open source model

replicate.com

Post reply on HN
Print view View original

111–120 of 224 posts

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#111

Post by rajhans »

karmasimida wrote: post #56
sp332 wrote: post #37

Earlier quoted context omitted.

Yeah but that's a 70B model. You can see on the Inference Efficiency chart that it takes more than 3x as much compute to run it compared to this one.

But you do need to hold all 128 experts in memory? Or not? Or they simply consider inference efficiency as latency

Arctic dev here. Yes keeping all experts in memory is the recommendation here and understandably that is a barrier to some. But once you have 1 H100 node or two (gpu middle-class I guess...?), then a few things to note: 1. FP6/FP8 inference is pretty good. How to on a single node: https://github.com/Snowflake-Labs/snowflake-arctic/tree/main... (vllm support coming soon!) 2. Small number of activated parameters shine in batch inference case for cloud providers.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#112

Post by ru552 »

tosh wrote: post #16
ru552 wrote: post #7

Abnormally large. I don't see the cost/performance numbers going well for this one.

It is both cost efficient in training (+ future fine-tuning) as well as inference compared to most other current models. Can you elaborate?

the unquantized model is almost 1tb in size and the benchmarks provided by Snowflake shows performance in the middle of the pack compared to other recent releases.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#113

Post by xcdzvyn »

rajhans wrote: post #109
1f60c wrote: post #33

It appears to have limited guardrails. I got it to generate some risqué story and it also told me how to trade onion futures, which is illegal in the US.

One of the modelers working on Arctic. We have done no alignment training whatsoever.

Thank you.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#114

Post by ankit219 »

analyte123 wrote: post #67
cs702 wrote: post #2

Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…

At a bare minimum, training and releasing a model like this builds critical skills in their engineering workforce that can't really be done any other way for now. It also requires compilation of a training dataset, which is not only another critical human skill, but also potentially a secret sauce if it turns out to give your model specific behaviors or skills. A big one is that it shows investors, partners, and futu…

These bigger companies are releasing open source models for publicity. For Databricks and Snowflake, both want enterprise customers, and want to show they can handle swathes of data and orchestration jobs, what better way to show that than by training a model. The pretraining part is done on a GPU but everything before that is managed on the Snowflake infra or Databricks. Databricks' website does focus heavily on this.[1]

I am speculating here, they would use their own OSS models to create a proprietary version which does one thing well. Answering questions for customers based on their own data. It's not an easy problem to solve as it seemed initially given enterprises need high reliability. Need models which are good at tool use, and can be grounded well. They could have done it on an oss model, but only now we have Llama-3 which is trained to make tool use easy. (Tool use as in function calling and use of stuff like OpenAI's code interpreter)

[1]: https://www.databricks.com/product/data-intelligence-platfor...

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#116

Post by HDThoreaun »

klysm wrote: post #38
1f60c wrote: post #33

It appears to have limited guardrails. I got it to generate some risqué story and it also told me how to trade onion futures, which is illegal in the US.

Why on earth is trading onion futures illegal in the us

someone cornered the onion market and instead of of prosecuting them the government decided to just make the whole thing illegal.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#117

Post by ko27 »

Zambyte wrote: post #60
ko27 wrote: post #29

Earlier quoted context omitted.

You have a weird definition of open source. OS software developers don't release the books they have read or the tools they've used to write code. This is fully 100% OSI compliant source code with an approved license (Apache 2.0). You are not entitled to anything more than this.

They don't have a weird definition of open source. I recently outlined a LLM chat that I think clearly outlines this: https://news.ycombinator.com/item?id=40035688

A bunch of code was autocompleted or generated by IDEs, are open source developers supposed to release the source code of that IDE to be OSI compliant?

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#118

Post by a13n »

sangnoir wrote: post #101
a13n wrote: post #64

Earlier quoted context omitted.

Seems like capitalism is doing its thing here. The potential future revenue from having the best model is presumably in the trillions.

> The potential future revenue from having the best model is presumably in the trillions. I heard this winner-takes-all spiel before - only last time, it was about Uber or Tesla[1] robo-taxis making car ownership obsolete. Uber has since exited the self-driving business, Cruise is on hold/unwinding and the whole self-driving bubble has mostly deflated, and most of the startups are long gone, despite the billions inve…

Uber and Tesla are valued at 150B and 500B respectively, I'd say in terms of an ROI on deploying large amounts of capital these are both huge success stories.

No investment in an emerging market is a sure thing, it's an educated guess. You have to take a lot of swings to occasionally hit a homerun, and investing in AI seems like the most plausible swing to make at this time.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#119

Post by heisgone »

jasongill wrote: post #104
salomonk_mur wrote: post #96

Earlier quoted context omitted.

Yeah, I wouldn't say the benefits were marginal at all. CPUs went from dozens of MHz in the 90's to over 4 GHz nowadays.

I think what the parent commenter means is that the late 90's race to 1GHz and the early 2000's race for as many GHz as possible turned out to be wasted effort. At the time, ever week it seemed like AMD or Intel would announce a new CPU that was a few MHz faster than the competition, and the assumption among the Slashdot crowd was basically that we'd have 20GHz CPU's by now. Instead, there was a plateau in terms of C…

It wasn't irrational at the time as it was much harder to harnest parallelism then.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#120

Post by ericd »

cornholio wrote: post #69
oceanplexian wrote: post #31

Earlier quoted context omitted.

Flights from the Western USA to Hawaii are ~2 million tons a year, at least in 2017, wouldn’t be surprised if that number doubled. 500t to train a model at least seems like a more productive use of carbon than spending a few days on the beach. So I don’t think the carbon use of training models is that extreme.

GPT3 was a 175 bln parameters model. All the big boys are now doing trillions of parameters without a substantial chip efficiency increase. So we are talking about thousands of tons of carbon per model, repeated every year or two or however fast they become obsolete. To that we need to add embedded carbon in the entire hardware stack and datacenter, it quickly adds up. If it's just a handfull of companies doing it, f…

The average American family is responsible for something like 50 tons per year. The carbon of one family for a decade is nothing compared to the benefits. The carbon of 1000 families for a decade is also approximately nothing compared to the benefits. It's just not relevant in the scheme of our economy.

There aren't that many base models, and finetunes take very little energy to perform.

Post reply on HN