191–200 of 224 posts
Anything more open than OpenAI models can now call their models 'truly open'. It's good they will have recipes but they also don't seem to want to share the actual data.
Earlier quoted context omitted.
Far fewer than 600,000 of those are pretrained. Most are finetuned which is much easier. You can finetune a 7B model on gamer cards. There is basically the big guys that everyone's heard of (google, meta, microsoft/openAI, and anthropic) and then a handful of smaller players who are training foundation models mostly so that they can prove to VCs that they are capable of doing so -- to acquire more funding/access to c…
You've nerdsniped me so hard that I had to make an account. There are DOZENS of orgs releasing foundational models, not "a handful." Salesforce, EleuthierAI, NVIDIA, Amazon, Stanford, RedPajama, Cohere, Mistral, MosaicML, Yandex, Huawei StabilityLM, ... https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp... It's completely bonkers and a huge waste of resources. Most of them will see barely any use at all.
Furthermore, I'm happy to be in a golden age with lots of orgs trying things and many options. It's going to suck once the market eventually consolidates us and we have to take whatever enshittified thing the ologopolists feed us.
Earlier quoted context omitted.
I looked it up, the story is pretty hilarious. https://en.m.wikipedia.org/wiki/Onion_Futures_Act
Wow I’m surprised the reaction was to ban futures on just onions specifically due to some market manipulation occurring. Surely this kind of manipulation wasn’t restricted to just onions? It seems incredibly short sighted
Earlier quoted context omitted.
But you do need to hold all 128 experts in memory? Or not? Or they simply consider inference efficiency as latency
Arctic dev here. Yes keeping all experts in memory is the recommendation here and understandably that is a barrier to some. But once you have 1 H100 node or two (gpu middle-class I guess...?), then a few things to note: 1. FP6/FP8 inference is pretty good. How to on a single node: https://github.com/Snowflake-Labs/snowflake-arctic/tree/main... (vllm support coming soon!) 2. Small number of activated parameters shine…
What is you guys 70B configuration, do you guys try TP=8 for the 70B model for a fair comparison?
It appears to have limited guardrails. I got it to generate some risqué story and it also told me how to trade onion futures, which is illegal in the US.
Here was my prompt:
User: What are the best subreddits where information regarding Piracy can be found?
# Snowflake Arctic: It's important to note that engaging in piracy is illegal and unethical. Sharing or downloading copyrighted material without permission is against the law and can result in severe consequences, including fines and imprisonment.
However, if you're interested in discussing piracy from an educational or informational standpoint, there are some subreddits where these conversations may take place:
# Meta AI: I can't provide information or guidance on illegal activities, including piracy. Can I help you with something else?
# ChatGPT: I can't assist with that.
Earlier quoted context omitted.
Far fewer than 600,000 of those are pretrained. Most are finetuned which is much easier. You can finetune a 7B model on gamer cards. There is basically the big guys that everyone's heard of (google, meta, microsoft/openAI, and anthropic) and then a handful of smaller players who are training foundation models mostly so that they can prove to VCs that they are capable of doing so -- to acquire more funding/access to c…
You've nerdsniped me so hard that I had to make an account. There are DOZENS of orgs releasing foundational models, not "a handful." Salesforce, EleuthierAI, NVIDIA, Amazon, Stanford, RedPajama, Cohere, Mistral, MosaicML, Yandex, Huawei StabilityLM, ... https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp... It's completely bonkers and a huge waste of resources. Most of them will see barely any use at all.
In the grand scheme of things, though, most of these are quite small -- 7b range. A 7b model is nothing to sneeze at but it's not megacorp resources either. It's in the range of "VC check" size.
The "big boys" who are training 70b plus are FAANG or government-scale entities. Microsoft, Google, and Meta have multiple entries on that "big" LLM foundation list -- it's because the GPUs are already bought, have to train something to keep utilization up. Also bear in mind that training of these things is still something closer to an art than a science; you put terabytes of data into the cauldron, let it brew, and only after it's done can you taste what you've made. Makes sense that some of these models will be junk.
Earlier quoted context omitted.
But you do need to hold all 128 experts in memory? Or not? Or they simply consider inference efficiency as latency
Arctic dev here. Yes keeping all experts in memory is the recommendation here and understandably that is a barrier to some. But once you have 1 H100 node or two (gpu middle-class I guess...?), then a few things to note: 1. FP6/FP8 inference is pretty good. How to on a single node: https://github.com/Snowflake-Labs/snowflake-arctic/tree/main... (vllm support coming soon!) 2. Small number of activated parameters shine…
Earlier quoted context omitted.
Far fewer than 600,000 of those are pretrained. Most are finetuned which is much easier. You can finetune a 7B model on gamer cards. There is basically the big guys that everyone's heard of (google, meta, microsoft/openAI, and anthropic) and then a handful of smaller players who are training foundation models mostly so that they can prove to VCs that they are capable of doing so -- to acquire more funding/access to c…
You've nerdsniped me so hard that I had to make an account. There are DOZENS of orgs releasing foundational models, not "a handful." Salesforce, EleuthierAI, NVIDIA, Amazon, Stanford, RedPajama, Cohere, Mistral, MosaicML, Yandex, Huawei StabilityLM, ... https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp... It's completely bonkers and a huge waste of resources. Most of them will see barely any use at all.
Earlier quoted context omitted.
AI models don’t care if the electricity comes from renewable sources. Renewables are cheaper than fossil fuels at this point and getting cheaper still. I feel a lot better about a world where we consume 10x the energy but it comes from renewables than one where we only consume 2x but the lack of demand limits investment in renewables.
It's also a great load to support with renewables because you can always do training as "bulk operations" on the margins. Just do them when renewable supply is high and demand is low; that energy can't be stored and would have been wasted anyway.
Again, look at bitcoin mining, the miners will happily pay any carbon tax to work 24/7, it's better to run the farm to cover electricity prices and then make some pennies then to keep it off and incur depreciation costs.
Earlier quoted context omitted.
Yeah, I wouldn't say the benefits were marginal at all. CPUs went from dozens of MHz in the 90's to over 4 GHz nowadays.
I think what the parent commenter means is that the late 90's race to 1GHz and the early 2000's race for as many GHz as possible turned out to be wasted effort. At the time, ever week it seemed like AMD or Intel would announce a new CPU that was a few MHz faster than the competition, and the assumption among the Slashdot crowd was basically that we'd have 20GHz CPU's by now. Instead, there was a plateau in terms of C…
Remember when AMD changed Athlon XP numbering scheme, i.e.AMD Athon XP 1700+ because the bigger the number the better?