Snowflake Arctic Instruct (128x3B MoE), largest open source model • Hacker News Board

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#211

Post by shadowgovt » Thu, Apr 25, 2024, 5:51 PM UTC

cornholio wrote: post #206

kaibee wrote: post #74

Earlier quoted context omitted.

AI models don’t care if the electricity comes from renewable sources. Renewables are cheaper than fossil fuels at this point and getting cheaper still. I feel a lot better about a world where we consume 10x the energy but it comes from renewables than one where we only consume 2x but the lack of demand limits investment in renewables.

This is a dangerous fantasy. Everything we know about the de-carbonation of the grid suggests that conservation is a key strategy for the next decades. There is no credible scenario towards 100% renewables. Storage is insanely expensive and green load smoothing capacity such as hydro and biomass is naturally limited. So a substantial part of the production when renewables drop will be handled by natural gas, which se…

Sounds like we'll have to adjust the price of non-renewables to reflect total cost, not just extraction, transportation, and generation cost.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#212

Post by adrianN » Thu, Apr 25, 2024, 6:04 PM UTC

kaibee wrote: post #74

cornholio wrote: post #69

Earlier quoted context omitted.

GPT3 was a 175 bln parameters model. All the big boys are now doing trillions of parameters without a substantial chip efficiency increase. So we are talking about thousands of tons of carbon per model, repeated every year or two or however fast they become obsolete. To that we need to add embedded carbon in the entire hardware stack and datacenter, it quickly adds up. If it's just a handfull of companies doing it, f…

AI models don’t care if the electricity comes from renewable sources. Renewables are cheaper than fossil fuels at this point and getting cheaper still. I feel a lot better about a world where we consume 10x the energy but it comes from renewables than one where we only consume 2x but the lack of demand limits investment in renewables.

We currently don't live in a world where renewable energy is available in excess.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#213

Post by Manabu-eo » Thu, Apr 25, 2024, 6:29 PM UTC

fsiefken wrote: post #154

ru552 wrote: post #121

Earlier quoted context omitted.

To stay fair, their "17B" model sits at 964GB on your disk and the 70B Llama 3 model sits at 141GB. unquantized GB numbers for both

to stay fairer, the required extra disk space for snowflake-arctic is cheaper then the required extra ram memory for llama3

For decent performance, you need to keep all the parameters on memory for both. Well, with a raid-0 of two PCIe 5 SSDs (or 4 PCIe 4) you might get 1 t/s loading experts from disk on snowflake-artic... but that is slooow.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#214

Post by newshackr » Thu, Apr 25, 2024, 6:35 PM UTC

dudus wrote: post #168

theturtletalks wrote: post #54

Earlier quoted context omitted.

Yep, seems like every company is taking a longshot on a AI project. Even companies like Databricks (MosaicML) and Vercel (v0 and ai.sdk) are seeing if they can take a piece of this every growing pie. Snowflake and the like are training and releasing new models because they intend to integrate the AI into their existing product down the line. Why not use and fine-tune an existing model? Their in-grown model maybe bett…

Their biggest competitor release a model. They must follow suit.

It isn't like they could have started this after the release and been done by now.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#215

Post by RhodesianHunter » Thu, Apr 25, 2024, 8:28 PM UTC

andy99 wrote: post #192

hnthrowaway9812 wrote: post #186

Earlier quoted context omitted.

You've nerdsniped me so hard that I had to make an account. There are DOZENS of orgs releasing foundational models, not "a handful." Salesforce, EleuthierAI, NVIDIA, Amazon, Stanford, RedPajama, Cohere, Mistral, MosaicML, Yandex, Huawei StabilityLM, ... https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp... It's completely bonkers and a huge waste of resources. Most of them will see barely any use at all.

Competition isn't a waste of resources, it's the best mechanism we have to ensure quality. Furthermore, I'm happy to be in a golden age with lots of orgs trying things and many options. It's going to suck once the market eventually consolidates us and we have to take whatever enshittified thing the ologopolists feed us.

Gods if this isn't exactly how it'll turn out.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#216

Post by jrm4 » Thu, Apr 25, 2024, 10:12 PM UTC

squigz wrote: post #166

jrm4 wrote: post #85

Earlier quoted context omitted.

This seems to me to be the simple story of "capitalism, having learned from the past, undertands that free/open source is actually advantageous for the little guys." Which is to say, "everyone" knows that this stuff has a lot of potential. Everyone is also used to what often happens in tech, which is outrageous winner-take-all scale effects. Everyone ALSO knows that there's almost certainly little MARGINAL difference…

> This seems to me to be the simple story of "capitalism, having learned from the past, undertands that free/open source is actually advantageous for the little guys." This seems rather generous.

Yeah, I don't mean it to sound that generous, as in "capitalism likes little guys."

More like "little guys, or even literally any 'guy' that isn't dominant in this space -- which tends toward dominance -- have learned, perhaps counterintuitively, that free/open source is best for their own greedy interests."

:)

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#217

Post by BPA0 » Fri, Apr 26, 2024, 8:53 AM UTC

richardw wrote: post #169

cs702 wrote: post #2

Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…

It diminishes the story that Databricks is the default route to privately trained models on your own data. Databricks jumped on the LLM bandwagon really quickly to good effect. Now every enterprise must at least consider Snowflake, and especially their existing clients who need to defend decisions to board members. It also means they build large scale rails necessary to use Snowflake for training and can market such…

I don't think that you understand Databricks. Databricks gives you the tools to train, tune or build RAG models. Snowflake doesn't.

Having said that, I'm a big fan of Llama-3 at the moment.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#218

Post by tzekid » Fri, Apr 26, 2024, 9:13 AM UTC

rajhans wrote: post #109

1f60c wrote: post #33

It appears to have limited guardrails. I got it to generate some risqué story and it also told me how to trade onion futures, which is illegal in the US.

One of the modelers working on Arctic. We have done no alignment training whatsoever.

That's actually awesome tbh.

Wonder what effect alignment training will have on the output quality

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#219

Post by oceanplexian » Fri, Apr 26, 2024, 3:33 PM UTC

cornholio wrote: post #206

kaibee wrote: post #74

Earlier quoted context omitted.

AI models don’t care if the electricity comes from renewable sources. Renewables are cheaper than fossil fuels at this point and getting cheaper still. I feel a lot better about a world where we consume 10x the energy but it comes from renewables than one where we only consume 2x but the lack of demand limits investment in renewables.

This is a dangerous fantasy. Everything we know about the de-carbonation of the grid suggests that conservation is a key strategy for the next decades. There is no credible scenario towards 100% renewables. Storage is insanely expensive and green load smoothing capacity such as hydro and biomass is naturally limited. So a substantial part of the production when renewables drop will be handled by natural gas, which se…

Sure there is, Nuclear is zero emission and renewable and powers 70% of the French electric grid. Uranium in the oceans is thought to regenerate itself but even if that turned out to not be true, should be good for at least a few hundred thousand years. It would require a massive infrastructure investment so now would be a good time to get started.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#220

Post by margalabargala » Fri, Apr 26, 2024, 4:50 PM UTC

ctoth wrote: post #170

margalabargala wrote: post #163

Earlier quoted context omitted.

> many of these large training companies, such as Microsoft, have committed to being net negative on carbon by 2030 Are you claiming that by 2030, the majority of AI will be trained in a carbon-neutral-or-better environment? If not, then my point stands. If so, I think that's an unrealistic claim. I'm willing to put my money where my mouth is. I'll bet you $1000 that by the year 2030, fewer than half of (major, trail…

I'm willing to take this bet, if we can figure out what the heck "major" trained-from-scratch models are and if we can figure out some objective source for tracking. Right now I believe I am on the path to easily win given that both the major upcoming models, (GPT-5 and Claude 4?) are training in large companies actively working on reducing their carbon output (Microsoft and Amazon data centers) Mistral appears to be…

> if we can figure out what the heck "major" trained-from-scratch models are and if we can figure out some objective source for tracking

Hmm. Yeah, we'll need to hammer out a solid definition. Further complicating things are models that may not be publicly available and are internally used by companies, though those may not be trained from scratch.

I would be fine with your suggestion to frame it in terms of percent power generation, though it might be hard to disentangle training costs from usage costs from that number. I would argue that including usage energy cleanliness is in the "spirit" of the bet but I'm happy to try to disentangle it as I originally proposed training-only.

> Something else to consider is the algorithmic improvements between now and 2030. From Yann LeCunn: Training LLaMA 13B emits 24 times less greenhouse gases than training GPT-3 175B yet performs better on benchmarks.

This is an excellent point, and definitely works in your favor. Really I'm on the good side of this bet. I win either way :) either my charity makes money, or I'm pleasantly surprised by climate impacts.

I've also not used longbets before. I would think we want to hammer out exact terms here before we set something up there?