Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Snowflake Arctic Instruct (128x3B MoE), largest open source model

replicate.com

Post reply on HN
Print view View original

201–210 of 224 posts

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#201

Post by karmasimida »

kristianp wrote: post #197
rajhans wrote: post #111

Earlier quoted context omitted.

Arctic dev here. Yes keeping all experts in memory is the recommendation here and understandably that is a barrier to some. But once you have 1 H100 node or two (gpu middle-class I guess...?), then a few things to note: 1. FP6/FP8 inference is pretty good. How to on a single node: https://github.com/Snowflake-Labs/snowflake-arctic/tree/main... (vllm support coming soon!) 2. Small number of activated parameters shine…

1 H100 is only 80GB of HBM. I guess you mean a server with 4xH100 is 1 node?

this is essentially 400b params. With FP8, comparing to Grok'3 320B model, which requires 320GB VRam in int4, I think what the OP meant is actually 8 H100.

Which is ... a lot to say the least.

And all optimization is for latency, not throughput, because with 8 H100, you can easily hosted 4 replicas of 70B.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#202

Post by klysm »

cryptonector wrote: post #193
klysm wrote: post #86

Earlier quoted context omitted.

Wow I’m surprised the reaction was to ban futures on just onions specifically due to some market manipulation occurring. Surely this kind of manipulation wasn’t restricted to just onions? It seems incredibly short sighted

It's bloody stupid is what it is. Knee-jerking like that is really terrible.

Knee-jerk is definitely the right term for this kind of legislation. I’m confused about how it has persisted. Do onion producers not want to be able to hedge out some price exposure?

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#203

Post by hnfong »

hnthrowaway9812 wrote: post #186
mlsu wrote: post #35

Earlier quoted context omitted.

Far fewer than 600,000 of those are pretrained. Most are finetuned which is much easier. You can finetune a 7B model on gamer cards. There is basically the big guys that everyone's heard of (google, meta, microsoft/openAI, and anthropic) and then a handful of smaller players who are training foundation models mostly so that they can prove to VCs that they are capable of doing so -- to acquire more funding/access to c…

You've nerdsniped me so hard that I had to make an account. There are DOZENS of orgs releasing foundational models, not "a handful." Salesforce, EleuthierAI, NVIDIA, Amazon, Stanford, RedPajama, Cohere, Mistral, MosaicML, Yandex, Huawei StabilityLM, ... https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp... It's completely bonkers and a huge waste of resources. Most of them will see barely any use at all.

Very nice! This list is super convenient for LLM “connoisseurs”(?) like me.

Did you have a script to generate it or was it manually done?

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#204

Post by kristianp »

karmasimida wrote: post #201
kristianp wrote: post #197

Earlier quoted context omitted.

1 H100 is only 80GB of HBM. I guess you mean a server with 4xH100 is 1 node?

this is essentially 400b params. With FP8, comparing to Grok'3 320B model, which requires 320GB VRam in int4, I think what the OP meant is actually 8 H100. Which is ... a lot to say the least. And all optimization is for latency, not throughput, because with 8 H100, you can easily hosted 4 replicas of 70B.

Thanks for the correction, there are indeed 8x nodes. https://developer.nvidia.com/blog/introducing-nvidia-hgx-h10...

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#205

Post by lhl »

hnfong wrote: post #203
hnthrowaway9812 wrote: post #186

Earlier quoted context omitted.

You've nerdsniped me so hard that I had to make an account. There are DOZENS of orgs releasing foundational models, not "a handful." Salesforce, EleuthierAI, NVIDIA, Amazon, Stanford, RedPajama, Cohere, Mistral, MosaicML, Yandex, Huawei StabilityLM, ... https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp... It's completely bonkers and a huge waste of resources. Most of them will see barely any use at all.

Very nice! This list is super convenient for LLM “connoisseurs”(?) like me. Did you have a script to generate it or was it manually done?

Just spotted this link. Just to clarify, I (not the original poster, although everyone's welcome to share this link, it's a public doc) maintain this list (and the rest of the sheet) manually. While I keep the foundation models that I'm interested in fairly up to date, obviously there are too many fine-tunes/datasets to track now. I started this when LLaMA was first released and I was getting myself up to speed on the LLM landscape.

A group at the CRFM maintains a bigger list of models (their goal is stated for cataloguing foundation models, but it looks like they have some tunes mixed in these days): https://crfm.stanford.edu/ecosystem-graphs/

This site also seems to keep track of models, with more closed/announced models that I don't bother to track: https://lifearchitect.ai/models-table/

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#206

Post by cornholio »

kaibee wrote: post #74
cornholio wrote: post #69

Earlier quoted context omitted.

GPT3 was a 175 bln parameters model. All the big boys are now doing trillions of parameters without a substantial chip efficiency increase. So we are talking about thousands of tons of carbon per model, repeated every year or two or however fast they become obsolete. To that we need to add embedded carbon in the entire hardware stack and datacenter, it quickly adds up. If it's just a handfull of companies doing it, f…

AI models don’t care if the electricity comes from renewable sources. Renewables are cheaper than fossil fuels at this point and getting cheaper still. I feel a lot better about a world where we consume 10x the energy but it comes from renewables than one where we only consume 2x but the lack of demand limits investment in renewables.

This is a dangerous fantasy. Everything we know about the de-carbonation of the grid suggests that conservation is a key strategy for the next decades. There is no credible scenario towards 100% renewables. Storage is insanely expensive and green load smoothing capacity such as hydro and biomass is naturally limited. So a substantial part of the production when renewables drop will be handled by natural gas, which seem to have equivalent emissions similar to coal when you factor in the lost methane, fracked methane in particular.

In addition, even 100% renewable would be attainable, that would still require massive infrastructure investment, resource use and associated emissions, since most of the corresponding industries, such as concrete and steel production, aluminum and copper ore mining and refining etc. are very far from net zero and will stay that way for decades.

To throw into this planet-sized bonfire a large uninterruptible consumer, whose standby capital depreciation on things like state of the art datacenters far exceeds the cost most industries are willing to pay for energy, all predicated on the idea that "demand spurs renewable investments", is frankly idiotic.

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#207

Post by hnfong »

lhl wrote: post #205
hnfong wrote: post #203

Earlier quoted context omitted.

Very nice! This list is super convenient for LLM “connoisseurs”(?) like me. Did you have a script to generate it or was it manually done?

Just spotted this link. Just to clarify, I (not the original poster, although everyone's welcome to share this link, it's a public doc) maintain this list (and the rest of the sheet) manually. While I keep the foundation models that I'm interested in fairly up to date, obviously there are too many fine-tunes/datasets to track now. I started this when LLaMA was first released and I was getting myself up to speed on th…

Very useful info. Thank you!

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#208

Post by throwaway120487 »

hnthrowaway9812 wrote: post #187
DowagerDave wrote: post #110

Earlier quoted context omitted.

Snowflake has a pretty good story in this space: "Your data is already in our cloud, so governance and use is a solved problem. Now use our AI (and burn credits)". This is a huge pain-point if you're thinking about ML with your (probably private) data. It's less clear if this entices companies to move INTO Snowflake IMO And streamlit, if you're as old as me, looks an awful lot like a MS-Access application for today.…

Snowflake could have the same story by hosting Llama 3 which is probably more efficient/better.

Snowflake hosts a couple models: https://docs.snowflake.com/en/user-guide/snowflake-cortex/ll...

Re: Snowflake Arctic Instruct (128x3B MoE), largest open source model

#210

Post by dantheman »

Zambyte wrote: post #61
dantheman wrote: post #30

Earlier quoted context omitted.

Why would they need to release the training data? that's nonsense.

Because the training data is the source of the model. This thread may illuminate it for you: https://news.ycombinator.com/item?id=40035688 Most models that are described as "open source" are actually open weight, because their source is not open.

It's still open source and can be used; Just like open source refers to code, not all the design documents, discussion, plans, etc.
Post reply on HN