Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Snowflake Arctic Instruct (128x3B MoE), largest open source model

replicate.com

Post reply on HN
Print view View original

141–150 of 224 posts

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#141

Post by mlyle »

35mm wrote: post #134
kaibee wrote: post #74

Earlier quoted context omitted.

AI models don’t care if the electricity comes from renewable sources. Renewables are cheaper than fossil fuels at this point and getting cheaper still. I feel a lot better about a world where we consume 10x the energy but it comes from renewables than one where we only consume 2x but the lack of demand limits investment in renewables.

Especially if one were to only run the servers during the daytime, when they can be powered directly from photovoltaics.

Which isn't going to happen, because you want to amortize these cards over 24 hours per day, not just when the renewables are shining or blowing.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#142

Post by mlyle »

TeMPOraL wrote: post #108
bee_rider wrote: post #87

Earlier quoted context omitted.

For sure; I didn’t realize sensible systemic reforms were on the table. I’m not sure if any of these things would be the first on the chopping block if a carbon tax were implemented, but it is worth a shot.

They're probably above the median on the scale of actually useful human activities; there's a lot of stuff carbon tax would eat first.

Yup, but even for the useful stuff, a greater price of carbon-intensive energy would change some about how you consider doing it.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#143

Post by CharlesW »

ru552 wrote: post #121
CharlesW wrote: post #103

Earlier quoted context omitted.

To be fair, that's comparing their 17B model with the 70B Llama 3 model.

To stay fair, their "17B" model sits at 964GB on your disk and the 70B Llama 3 model sits at 141GB. unquantized GB numbers for both

Sorry, it sounds like you know a lot more than I do about this, and I'd appreciate it if you'd connect the dots. Is your comment a dig at either Snowflake or Llama? Where are you finding the unquantized size of Llama 3 70B? Isn't it extremely rare to do inference with large unquantized models?

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#144

Post by mlyle »

margalabargala wrote: post #126
renewiltord wrote: post #59

Earlier quoted context omitted.

500 tons is like a few flights between SF and NYC dude. And those 600k models are mostly fine-tunes. If running your 4090 at home is too much then we're going to have to get rid of the gamers. This CO2 objection is such an innumerate objection. Just making 100 cars already is more than making one of these LLMs from scratch. A finetune is so cheap in comparison. In fact, I bet if you asked most LLM companies they'd gl…

There are two groups here. One sees the high impact of the large model, and the growth of model training, and is concerned with how much that could increase in coming years. The other group assumes the first group is complaining about right now, and thinks they're being ridiculous. This whole thing reminds me of ten years ago when people were pointing out energy waste as a downside of bitcoin. "It's so little! Electr…

> If AI model training follows a similar curve, then it's reasonable to be concerned.

Yes, but one can at least still imagine scenarios where AI training being 0.5% of electricity use could still be a net win.

(I hope we're more efficient than that; but if we're training models that end up helping a little with humanity's great problems, using 1/200th of our electricity for it could be worth it).

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#145

Post by ko27 »

Zambyte wrote: post #123
ko27 wrote: post #117

Earlier quoted context omitted.

A bunch of code was autocompleted or generated by IDEs, are open source developers supposed to release the source code of that IDE to be OSI compliant?

Is the IDE a primary input for building the program? Is the IDE a build dependency? Probably not. Certainly not based on the situation you described. The LLM equivalent here would be programmatically generating synthetic input or cleaning input for training. You don't need the tools used to generate or clean the data in order to train the model, and thus they can be propriety in the context of an open source model, s…

> Is the IDE a primary input for building the program? Is the IDE a build dependency?

No, the same way training is not a build dependency for the weights source code. You can literally compile and run them without any training data.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#146

Post by ponector »

cryptonector wrote: post #138
fs_tab wrote: post #50

Earlier quoted context omitted.

That's right. Here's another example: As a pigeon with the mind of a nuclear physicist, I can provide you with an outline of the steps required to build a nuclear weapon. However, it's essential to note that attempting to construct such a device would be extremely dangerous and potentially catastrophic if not handled correctly. Here is a more detailed overview of the process (full text omitted)

Eh, the knowledge of how to construct a nuclear weapon has long been widely and publicly available. And it's not useful, not without weapons-grade fissile material. And if you could make weapons-grade fissile material you could figure out how to make a weapon if it wasn't already well-known.

And yet, chatgpt is censored to not hallucinate about it.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#147

Post by vineyardmike »

theturtletalks wrote: post #54
mlsu wrote: post #35

Earlier quoted context omitted.

Far fewer than 600,000 of those are pretrained. Most are finetuned which is much easier. You can finetune a 7B model on gamer cards. There is basically the big guys that everyone's heard of (google, meta, microsoft/openAI, and anthropic) and then a handful of smaller players who are training foundation models mostly so that they can prove to VCs that they are capable of doing so -- to acquire more funding/access to c…

Yep, seems like every company is taking a longshot on a AI project. Even companies like Databricks (MosaicML) and Vercel (v0 and ai.sdk) are seeing if they can take a piece of this every growing pie. Snowflake and the like are training and releasing new models because they intend to integrate the AI into their existing product down the line. Why not use and fine-tune an existing model? Their in-grown model maybe bett…

> Why not use and fine-tune an existing model?

Not all of them have permissive licenses for whatever the companies may want (or their clients want). Kind of a funny situation where everyone would benefit, but no one wants to burn their money for the greater good.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#148

Post by kiratp »

rajhans wrote: post #111
karmasimida wrote: post #56

Earlier quoted context omitted.

But you do need to hold all 128 experts in memory? Or not? Or they simply consider inference efficiency as latency

Arctic dev here. Yes keeping all experts in memory is the recommendation here and understandably that is a barrier to some. But once you have 1 H100 node or two (gpu middle-class I guess...?), then a few things to note: 1. FP6/FP8 inference is pretty good. How to on a single node: https://github.com/Snowflake-Labs/snowflake-arctic/tree/main... (vllm support coming soon!) 2. Small number of activated parameters shine…

> 2. Small number of activated parameters shine in batch inference case for cloud providers

Could you elaborate more please? Batch inference activates pretty much all the experts since token in every sequence in a batch could hit a different expert. So at Bs=128 you’re not really getting a sparsity win.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#149

Post by _flux »

cs702 wrote: post #2

Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…

And huggingface is hosting (randomly assuming 8-64 GB per model) 5..40 PB of models for free? That's generous of them. Or can the models share data? Ollama seems to have some ability to do that.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#150

Post by Zambyte »

ko27 wrote: post #145
Zambyte wrote: post #123

Earlier quoted context omitted.

Is the IDE a primary input for building the program? Is the IDE a build dependency? Probably not. Certainly not based on the situation you described. The LLM equivalent here would be programmatically generating synthetic input or cleaning input for training. You don't need the tools used to generate or clean the data in order to train the model, and thus they can be propriety in the context of an open source model, s…

> Is the IDE a primary input for building the program? Is the IDE a build dependency? No, the same way training is not a build dependency for the weights source code. You can literally compile and run them without any training data.

Training data is a build dependency for the weights. You cannot realistically get the same weights without the same training data.
Post reply on HN