We'll see GPT-5 in a few months and that will be vastly more useful information to update your sense of whether the current approach will continue to work than anyone's speculation today.
Rodney Brooks on limitations of generative AI
31–40 of 202 posts
Re: Rodney Brooks on limitations of generative AI
#32Earlier quoted context omitted.
I’m still waiting for SalesForce to integrate an LLM into Slack so I can ask it business logic and decisions long lost. Still waiting for Microsoft to integrate an LLM into outlook so I can get a summary of a 20 email long chain I just got CCed into. I don’t think the iPod comparison is a valid one. People only have so much time to listen to music. Past a certain point, no one has enough good music they like to put i…
> However, the more data you feed into an LLM, the smarter it should be in the response. Is it that way? For example if it lacks a certain reasoning capability, then more data may not change that. So far LLMs lack useful ideas of truth, it will easily generate untrue statements. We see lots of hacks how to control that, with unconvincing results.
Llama-1, 1T tokens, dumb as a box of rocks
Llama-2, 2T tokens, smarter than a box of rocks
Mistral-7B, 8T tokens, way smarter than llama-2
Llama-3, 15T tokens, smarter than anything a few times its size
Gemma-2, 13T synthetic tokens, slightly better than llama-3
(for the same approximate parameter size)
I think it roughly tracks that moar data = moar betterer.
Re: Rodney Brooks on limitations of generative AI
#33After using Copilot that is pretty bad at guessing what I exactly want to do, but still occasionally right on the money and often pretty close: AI is not really AI and it won't kill us all, but the realization is that a lot of work is just repetitive and really not that clever at all. If I think about all the work I did in my life it follows the same pattern: a new way of doing things comes along, then you start figuring out how to do it and how to use it, and once you're there you rinse and repeat. The real value in the work will be increasingly in why is it useful for people using it, although it probably was like this always, the geeks just didn't pay attention to it.
Sorry for not commenting on the article directly.
Re: Rodney Brooks on limitations of generative AI
#34To me, this reads like a very reasonable take. He suggests to limit the scope of the AI problem, add manual overrides in case there are unexpected situations, and he (rightly, in my opinion) predicts that the business case for exponentially scaling LLM models isn't there. With that context, I like his iPod example. Apple probably could have made a 3TB iPod to stick to Moore's law for another few years, but after they…
I’m still waiting for SalesForce to integrate an LLM into Slack so I can ask it business logic and decisions long lost. Still waiting for Microsoft to integrate an LLM into outlook so I can get a summary of a 20 email long chain I just got CCed into. I don’t think the iPod comparison is a valid one. People only have so much time to listen to music. Past a certain point, no one has enough good music they like to put i…
> Still waiting for Microsoft to integrate an LLM into outlook so I can get a summary of a 20 email long chain I just got CCed into.
Still waiting Microsoft to add a email search to Outlook that isn’t complete garbage. Ideally with a decent UI and presentation of results that isn’t complete garbage.
…why are we hoping that AI will make these products better, when they’re not using conventional methods appropriately, and have been enshittified to shit.
Re: Rodney Brooks on limitations of generative AI
#35The iPod analogy is a poor one. Instead of 160TB music players, we got general computers (iPhone) with effectively unlimited storage (wireless Internet).
I don’t need to store all my music on my device. I can have it beamed directly to my ears on-demand.
Re: Rodney Brooks on limitations of generative AI
#36To me, this reads like a very reasonable take. He suggests to limit the scope of the AI problem, add manual overrides in case there are unexpected situations, and he (rightly, in my opinion) predicts that the business case for exponentially scaling LLM models isn't there. With that context, I like his iPod example. Apple probably could have made a 3TB iPod to stick to Moore's law for another few years, but after they…
I’m still waiting for SalesForce to integrate an LLM into Slack so I can ask it business logic and decisions long lost. Still waiting for Microsoft to integrate an LLM into outlook so I can get a summary of a 20 email long chain I just got CCed into. I don’t think the iPod comparison is a valid one. People only have so much time to listen to music. Past a certain point, no one has enough good music they like to put i…
Why should the response be better just because there is "more data"?
Should I be adding extra random tokens to my prompts to make the LLM "smarter"?
Re: Rodney Brooks on limitations of generative AI
#37Earlier quoted context omitted.
> However, the more data you feed into an LLM, the smarter it should be in the response. Is it that way? For example if it lacks a certain reasoning capability, then more data may not change that. So far LLMs lack useful ideas of truth, it will easily generate untrue statements. We see lots of hacks how to control that, with unconvincing results.
Llama-1, 1T tokens, dumb as a box of rocks Llama-2, 2T tokens, smarter than a box of rocks Mistral-7B, 8T tokens, way smarter than llama-2 Llama-3, 15T tokens, smarter than anything a few times its size Gemma-2, 13T synthetic tokens, slightly better than llama-3 (for the same approximate parameter size) I think it roughly tracks that moar data = moar betterer.
but the OP was talking about the size of the context window, not the size of the training corpus
Re: Rodney Brooks on limitations of generative AI
#38Earlier quoted context omitted.
> However, the more data you feed into an LLM, the smarter it should be in the response. Is it that way? For example if it lacks a certain reasoning capability, then more data may not change that. So far LLMs lack useful ideas of truth, it will easily generate untrue statements. We see lots of hacks how to control that, with unconvincing results.
Llama-1, 1T tokens, dumb as a box of rocks Llama-2, 2T tokens, smarter than a box of rocks Mistral-7B, 8T tokens, way smarter than llama-2 Llama-3, 15T tokens, smarter than anything a few times its size Gemma-2, 13T synthetic tokens, slightly better than llama-3 (for the same approximate parameter size) I think it roughly tracks that moar data = moar betterer.
> not smart
> slightly smarter
> way smarter
> Last one, "slightly smarter"
So, the the usual s-curve, that has an exponential phase, then topping out?
Re: Rodney Brooks on limitations of generative AI
#39The iPod analogy is a poor one. Instead of 160TB music players, we got general computers (iPhone) with effectively unlimited storage (wireless Internet). I don’t need to store all my music on my device. I can have it beamed directly to my ears on-demand.
That's the point though, an iPhone is not just a bigger iPod - scaling by itself isn't enough.
Re: Rodney Brooks on limitations of generative AI
#40To me, this reads like a very reasonable take. He suggests to limit the scope of the AI problem, add manual overrides in case there are unexpected situations, and he (rightly, in my opinion) predicts that the business case for exponentially scaling LLM models isn't there. With that context, I like his iPod example. Apple probably could have made a 3TB iPod to stick to Moore's law for another few years, but after they…
Is there a reason why memory was used and not compute power as an example? I don't understand how cherry picking random examples from past explain future of AI. If he think business needs does not exist he should explain how he arrived at that conclusion instead of a random iPod example.