161–170 of 224 posts
So many models fail at basic reasoning. > What weighs more: a pound of feathers or a great british pound? > A pound of feathers and a Great British Pound weigh the same, which is one pound. It works if you add "Think step by step," though.
Earlier quoted context omitted.
There are two groups here. One sees the high impact of the large model, and the growth of model training, and is concerned with how much that could increase in coming years. The other group assumes the first group is complaining about right now, and thinks they're being ridiculous. This whole thing reminds me of ten years ago when people were pointing out energy waste as a downside of bitcoin. "It's so little! Electr…
> If AI model training follows a similar curve, then it's reasonable to be concerned. Yes, but one can at least still imagine scenarios where AI training being 0.5% of electricity use could still be a net win. (I hope we're more efficient than that; but if we're training models that end up helping a little with humanity's great problems, using 1/200th of our electricity for it could be worth it).
It does not seem well-poised to yield novel advancements in unrelated-to-AI fields, yet. Possibly genetics. But things like solving global warming, there is not any sort of path towards that for anything we're currently creating.
It's not clear to me that spending 0.5% of electricity generation to put a solid chunk of the lower-middle-class out of work is worth it.
Earlier quoted context omitted.
There are two groups here. One sees the high impact of the large model, and the growth of model training, and is concerned with how much that could increase in coming years. The other group assumes the first group is complaining about right now, and thinks they're being ridiculous. This whole thing reminds me of ten years ago when people were pointing out energy waste as a downside of bitcoin. "It's so little! Electr…
> The other group assumes the first group is complaining about right now, and thinks they're being ridiculous. Except this is obviously not the case, as "the other group" is aware that many of these large training companies, such as Microsoft, have committed to being net negative on carbon by 2030, and are actively making progress with this whereas the other group seems to be motivated by flailing for anything they c…
Are you claiming that by 2030, the majority of AI will be trained in a carbon-neutral-or-better environment?
If not, then my point stands.
If so, I think that's an unrealistic claim. I'm willing to put my money where my mouth is. I'll bet you $1000 that by the year 2030, fewer than half of (major, trailed-from-scratch) models are trained in a carbon-neutral-or-better environment. Money goes to charity of the winner's choice.
Earlier quoted context omitted.
> If AI model training follows a similar curve, then it's reasonable to be concerned. Yes, but one can at least still imagine scenarios where AI training being 0.5% of electricity use could still be a net win. (I hope we're more efficient than that; but if we're training models that end up helping a little with humanity's great problems, using 1/200th of our electricity for it could be worth it).
The current crop of generative AIs seems well-poised to take over a significant amount of low-skill human labor. It does not seem well-poised to yield novel advancements in unrelated-to-AI fields, yet. Possibly genetics. But things like solving global warming, there is not any sort of path towards that for anything we're currently creating. It's not clear to me that spending 0.5% of electricity generation to put a so…
Can LLMs help us save energy? It doesn't seem to be such a ridiculous idea to me.
And can they be an effort multiplier for others working on harder problems? Likely-- I am a high-skill worker and I routinely have lower-skill tasks that I can delegate to LLMs more easily than I could either do myself or delegate to other humans. (And, now and then, they're helpful for brainstorming in my chosen fields).
I had a big manual to write communicating how to use something I've built. Giving GPT-4 some bulleted lists and a sample of my writing got about 2/3rds of it done. (I had to throw a fraction away, and make some small correctness edits). It took much less of my time than working with a doc writer usually does and probably yielded a better result. In turn, I'm back to my high-value tasks sooner.
That is, LLMs may help attacking the great problems directly, or they may help us dedicate more effort to the great problems. (Or they may do nothing or may screw us all up in other ways).
Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…
Far fewer than 600,000 of those are pretrained. Most are finetuned which is much easier. You can finetune a 7B model on gamer cards. There is basically the big guys that everyone's heard of (google, meta, microsoft/openAI, and anthropic) and then a handful of smaller players who are training foundation models mostly so that they can prove to VCs that they are capable of doing so -- to acquire more funding/access to c…
Interesting you'd say that in a discussion on Snowflake's LLM, no less. As someone who has a good opinion of Databricks, genuinely curious what made you arrive at such a damning conclusion.
Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…
This seems to me to be the simple story of "capitalism, having learned from the past, undertands that free/open source is actually advantageous for the little guys." Which is to say, "everyone" knows that this stuff has a lot of potential. Everyone is also used to what often happens in tech, which is outrageous winner-take-all scale effects. Everyone ALSO knows that there's almost certainly little MARGINAL difference…
This seems rather generous.
Earlier quoted context omitted.
The current crop of generative AIs seems well-poised to take over a significant amount of low-skill human labor. It does not seem well-poised to yield novel advancements in unrelated-to-AI fields, yet. Possibly genetics. But things like solving global warming, there is not any sort of path towards that for anything we're currently creating. It's not clear to me that spending 0.5% of electricity generation to put a so…
There was an important "if" there in what I said. That's why I didn't say that it was the case. Though, no matter what, LLMs are doing more useful work than looking for hash collisions. Can LLMs help us save energy? It doesn't seem to be such a ridiculous idea to me. And can they be an effort multiplier for others working on harder problems? Likely-- I am a high-skill worker and I routinely have lower-skill tasks tha…
The trouble I have is, what determines whether AI grow to 0.5% (or whatever %) of our electricity usage is not whether the AI is a net good for humanity even considering power use. It's going to be determined by whether the AI is a net benefit for the bank account of the people with the means to make AI.
We can just as easily have a situation where AI grows to 0.5% electricity usage, is economically viable for those in control of it, while having a net negative impact for the rest of society.
As a parent said, a carbon tax would address a lot of this and would be great for a lot of reasons.
Earlier quoted context omitted.
Far fewer than 600,000 of those are pretrained. Most are finetuned which is much easier. You can finetune a 7B model on gamer cards. There is basically the big guys that everyone's heard of (google, meta, microsoft/openAI, and anthropic) and then a handful of smaller players who are training foundation models mostly so that they can prove to VCs that they are capable of doing so -- to acquire more funding/access to c…
Yep, seems like every company is taking a longshot on a AI project. Even companies like Databricks (MosaicML) and Vercel (v0 and ai.sdk) are seeing if they can take a piece of this every growing pie. Snowflake and the like are training and releasing new models because they intend to integrate the AI into their existing product down the line. Why not use and fine-tune an existing model? Their in-grown model maybe bett…
Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the most interesting aspect about this, for me, is that every tech company seems to be coming out with a free open model claiming to be better than the others at this thing or that thing. The number of choices is…
It also means they build large scale rails necessary to use Snowflake for training and can market such at every release.
Earlier quoted context omitted.
> The other group assumes the first group is complaining about right now, and thinks they're being ridiculous. Except this is obviously not the case, as "the other group" is aware that many of these large training companies, such as Microsoft, have committed to being net negative on carbon by 2030, and are actively making progress with this whereas the other group seems to be motivated by flailing for anything they c…
> many of these large training companies, such as Microsoft, have committed to being net negative on carbon by 2030 Are you claiming that by 2030, the majority of AI will be trained in a carbon-neutral-or-better environment? If not, then my point stands. If so, I think that's an unrealistic claim. I'm willing to put my money where my mouth is. I'll bet you $1000 that by the year 2030, fewer than half of (major, trail…
Mistral appears to be using the Leonardo supercomputer, which doesn't seem to have direct numbers available, but I did find this quote upon its launch in 2022:
> One of the most powerful supercomputers in the world – and definitely Europe’s largest – was recently unveiled in Bologna, Italy. Powerful machine Leonardo (which aptly means “lion-hearted”, and is also the name of the famous Italian artist, engineer and scientist Leonardo da Vinci) is a €120 million system that promises to utilise artificial intelligence to undertake “unprecedented research”, according to the European Commission. Plus, the system is sustainably-focused, and equipped with tools to enable a dynamical adjustment of power consumption. It also uses a water-cooling system for increased energy efficiency.
You might have a greater chance to win the bet if we think about all models trained in 2030, not just flagship/cutting-edge models, as it's likely that all the GPUs which are frantically being purchased now will be depreciated and sold to hackers by the truckload here in 4-5 years, the same way some of us collect old servers from 2018ish now. But even that is a hard calculation to make--do we count old H100s running at home but on solar power as sustainable? Will the new hardware running in sustainable datacenters continue to vastly outpace the old depreciated?
For cutting-edge models which almost by definition require huge compute infrastructure, a majority of them will be carbon neutral by 2030.
A better way to frame this bet might be to consider it in percentages of total energy generation? It might be easier to actually get that number in 2030. Like Dirty AI takes 3% of total generation and clean AI 3.5%?
Something else to consider is the algorithmic improvements between now and 2030. From Yann LeCunn: Training LLaMA 13B emits 24 times less greenhouse gases than training GPT-3 175B yet performs better on benchmarks.
I haven't done longbets before, but I think that's what we're supposed to use for stuff like this? :) My email is in my profile.
One more thing to consider before we commit is that the current global share of renewable energy is something close to 29%. You should probably factor in overall renewable growth by 2030, if >50% of energy is renewable by then, I win by default but that doesn't exactly seem sporting.