121–130 of 224 posts
Earlier quoted context omitted.
It's worse at SQL generation than llama3 according to their own post. https://www.snowflake.com/blog/arctic-open-efficient-foundat...
To be fair, that's comparing their 17B model with the 70B Llama 3 model.
It appears to have limited guardrails. I got it to generate some risqué story and it also told me how to trade onion futures, which is illegal in the US.
How is the model supposed to know what country it is in?
Earlier quoted context omitted.
They don't have a weird definition of open source. I recently outlined a LLM chat that I think clearly outlines this: https://news.ycombinator.com/item?id=40035688
A bunch of code was autocompleted or generated by IDEs, are open source developers supposed to release the source code of that IDE to be OSI compliant?
The LLM equivalent here would be programmatically generating synthetic input or cleaning input for training. You don't need the tools used to generate or clean the data in order to train the model, and thus they can be propriety in the context of an open source model, so long as the source for the model is open (the training data).
Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…
Hype and jumping on the bandwagon are perfectly good reasons for a business. There's no business without risk. This is the cost of doing business which you want to explore greenfield projects.
Earlier quoted context omitted.
Why would they need to release the training data? that's nonsense.
Open source for traditional software means that you can see how the software works and reproduce the executable by compiling the software from source code. For LLMs, reproducing the model means reproducing the weights. And to do that you need the training source code AND the training data. There are already other great models that do this (see my comment at https://news.ycombinator.com/item?id=40147298 ). I get that…
I'd rather also have the data for sure! But in terms of what useful things I can do with it, weights are closer to source code than they are to a binary blob.
Earlier quoted context omitted.
That's the amount that would be released by burning 50,000 gallons of gas, which is about that ten typical cars will burn throughout their entire lifespan. Done once, I agree, that's very little. But if each of those 600,000 other models used that much (or even a tenth that much), then that now becomes impactful. Releasing 500 tons of CO2 600,000 times over would amount to about 1% of all human global annual emission…
500 tons is like a few flights between SF and NYC dude. And those 600k models are mostly fine-tunes. If running your 4090 at home is too much then we're going to have to get rid of the gamers. This CO2 objection is such an innumerate objection. Just making 100 cars already is more than making one of these LLMs from scratch. A finetune is so cheap in comparison. In fact, I bet if you asked most LLM companies they'd gl…
One sees the high impact of the large model, and the growth of model training, and is concerned with how much that could increase in coming years.
The other group assumes the first group is complaining about right now, and thinks they're being ridiculous.
This whole thing reminds me of ten years ago when people were pointing out energy waste as a downside of bitcoin. "It's so little! Electricity prices will prevent it from ever becoming significant!" was the response that it was met with, just like people are saying in this thread.
In 2023, crypto mining accounted for about 0.5% of humanity's electricity consumption. If AI model training follows a similar curve, then it's reasonable to be concerned.
A sample prompt: "Tell me a love story about two otters, rendered in the FORTH language".
Or: "Here's a whitepaper, write me a simulator in python that lets me see the state of these variables, step by step".
Or: "Here's a tarball of a program. Write a module that does X, in a unified diff."
These are super hard tasks for any LLM I have access to, BTW. Good for testing current edges of capacity.
Arctic does not do great on these, unfortunately. It's not willing to make 'the leap' to be creative in FORTH where creativity = storytelling, and tries to redirect me to either getting a story about otters, or telling me things about FORTH.
Google made a big deal about emergent sophistication in models as they grew in parameter size with the original PaLM paper, and I wonder if these horizontally-scaled MOE of many small models are somehow architecturally limited. The model weights here, 480B, are sized close to the original PaLM model (540B if I recall).
Anyway, more and varied architectures are always welcome! I'd be interested to hear from the Snowflake folks if they think the architecture has additional capacity with more training, or if they think it could improve on recall tasks, but not 'sophistication' type tasks.
Earlier quoted context omitted.
Yeah but that's a 70B model. You can see on the Inference Efficiency chart that it takes more than 3x as much compute to run it compared to this one.
Most people are vram constrained not compute constrained.
At those scales, most people become bandwidth and compute constrained using CPU inference instead of multiple GPUs. In those cases, an MOE with a low number of active parameters is the fastest.
Earlier quoted context omitted.
But you do need to hold all 128 experts in memory? Or not? Or they simply consider inference efficiency as latency
I believe the main draw of the MoE model is they don't all need to be in memory at once. They can be swapped based on context. In aggregate you get the performance of a much larger model (384b tokens) while using much less memory than such a model would require. If you had enough memory it could all be loaded but it doesn't need to be.
Earlier quoted context omitted.
GPT3 was a 175 bln parameters model. All the big boys are now doing trillions of parameters without a substantial chip efficiency increase. So we are talking about thousands of tons of carbon per model, repeated every year or two or however fast they become obsolete. To that we need to add embedded carbon in the entire hardware stack and datacenter, it quickly adds up. If it's just a handfull of companies doing it, f…
AI models don’t care if the electricity comes from renewable sources. Renewables are cheaper than fossil fuels at this point and getting cheaper still. I feel a lot better about a world where we consume 10x the energy but it comes from renewables than one where we only consume 2x but the lack of demand limits investment in renewables.
Just do them when renewable supply is high and demand is low; that energy can't be stored and would have been wasted anyway.