Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rodney Brooks on limitations of generative AI

techcrunch.com

Post reply on HN
Print view View original

41–50 of 202 posts

Re: Rodney Brooks on limitations of generative AI

#41

Post by roenxi »

albert_e wrote: post #23
roenxi wrote: post #9

> He uses the iPod as an example. For a few iterations, it did in fact double in storage size from 10 all the way to 160GB. If it had continued on that trajectory, he figured out we would have an iPod with 160TB of storage by 2017, but of course we didn’t. I think Brooks' opinions will age poorly, but if anyone doesn't already know all the arguments for that they aren't interested in learning them now. This quote see…

> we would have an iPod with 160TB of storage by 2017, but of course we didn’t. 160 Tera Bytes to store what? I probably would never listen to more than a few GBs of high fidelity music in my life time. Why would anyone keep investing in exponential growth or even linear growth beyond a point of utility and economic sense. This can be extrapolated to miniaturization also ... why is a personal computer not smaller tha…

Arguing from a lack of personal imagination is not a strong position. It is the people with ideas who are responsible for finding uses for resources. They've succeeded every other time they were given an excess of storage space; I'm still not sure how I manage to use up all the TB of storage I've bought over the years.

Maybe we store a local copy of a personal universe for you to test ideas out in, I dunno. There'll be something.

> I probably would never listen to more than a few GBs of high fidelity music in my life time.

Well from that I'd predict that uses would be found other than music. My "music" folder has made it up to 50GB because I've taken to storing a few movies in it. But games can quickly add up to TB of media if the space is available.

Re: Rodney Brooks on limitations of generative AI

#42

Post by DonHopkins »

bubblyworld wrote: post #10
DonHopkins wrote: post #6

Earlier quoted context omitted.

Rodney Brooks is the Godfather of Out of Control Theory. FAST, CHEAP AND OUT OF CONTROL: A ROBOT INVASION OF THE SOLAR SYSTEM: https://people.csail.mit.edu/brooks/papers/fast-cheap.pdf

"Out-of-Control Theory" is a great name, love it =) thanks for the link, although I guess the idea never really took off.

It certainly did take off, it's called "Subsumption Architecture", and Rodney Brooks started iRobot, who created the Roomba, which is based on those ideas.

https://en.wikipedia.org/wiki/Subsumption_architecture

Subsumption architecture is a reactive robotic architecture heavily associated with behavior-based robotics which was very popular in the 1980s and 90s. The term was introduced by Rodney Brooks and colleagues in 1986.[1][2][3] Subsumption has been widely influential in autonomous robotics and elsewhere in real-time AI.

https://en.wikipedia.org/wiki/IRobot

iRobot Corporation is an American technology company that designs and builds consumer robots. It was founded in 1990 by three members of MIT's Artificial Intelligence Lab, who designed robots for space exploration and military defense.[2] The company's products include a range of autonomous home vacuum cleaners (Roomba), floor moppers (Braava), and other autonomous cleaning devices.[3]

Re: Rodney Brooks on limitations of generative AI

#43

Post by richrichie »

aurareturn wrote: post #22
richrichie wrote: post #20

Earlier quoted context omitted.

> the more data you feed into an LLM, the smarter it should be in the response This is not obvious though.

It’s in theory. The more information you have, the better the decision in theory.

Not quite. There are bounds on capacity of learning machines.

https://en.wikipedia.org/wiki/Vapnik%E2%80%93Chervonenkis_di...

Re: Rodney Brooks on limitations of generative AI

#44

Post by moffkalast »

SideburnsOfDoom wrote: post #38
moffkalast wrote: post #32

Earlier quoted context omitted.

Llama-1, 1T tokens, dumb as a box of rocks Llama-2, 2T tokens, smarter than a box of rocks Mistral-7B, 8T tokens, way smarter than llama-2 Llama-3, 15T tokens, smarter than anything a few times its size Gemma-2, 13T synthetic tokens, slightly better than llama-3 (for the same approximate parameter size) I think it roughly tracks that moar data = moar betterer.

> not smart > slightly smarter > way smarter > Last one, "slightly smarter" So, the the usual s-curve, that has an exponential phase, then topping out?

Pretty much, yep. There was definitely a more significant jump there in the middle where 7B models went from being a complete waste of time to actually useful. Then going from being able to craft a sensible response to 80% of questions to 90% is a much smaller apparent increase but takes a lot more compute to achieve as per the pareto principle.

Re: Rodney Brooks on limitations of generative AI

#45

Post by sigmoid10 »

eysgshsvsvsv wrote: post #40
fxtentacle wrote: post #13

To me, this reads like a very reasonable take. He suggests to limit the scope of the AI problem, add manual overrides in case there are unexpected situations, and he (rightly, in my opinion) predicts that the business case for exponentially scaling LLM models isn't there. With that context, I like his iPod example. Apple probably could have made a 3TB iPod to stick to Moore's law for another few years, but after they…

Is there a reason why memory was used and not compute power as an example? I don't understand how cherry picking random examples from past explain future of AI. If he think business needs does not exist he should explain how he arrived at that conclusion instead of a random iPod example.

This. The scaling of compute has vastly different applications than the scaling of memory. Shows once again that people who are experts in a related field aren't necessarily the best to comment on trendy topics. If e.g. an aeroplane expert critiques Spacex's starship, you should be equally vary, even though they might have some overlap. The only reason this is in the media at all is because negative sentiment to hype generates many clicks. That's why you see these topics every day instead of Rubik's cube players criticising the latest version of Mikado.

Re: Rodney Brooks on limitations of generative AI

#46

Post by IshKebab »

globalnode wrote: post #12

i dont know much about machine learning but what i think i know is that its getting an outcome based on averages of witnessed data/events. so how's it going to come up with anything novel? or outside of normal?

It's predicting a function. You train it on known inputs (and potentially corresponding known outputs). You get a novel output by feeding it a novel input.

For example asking an LLM a question that nobody has even asked it before. The degree to which it does a good job on those questions is called "generalisation".

Re: Rodney Brooks on limitations of generative AI

#47

Post by bubblyworld »

DonHopkins wrote: post #42
bubblyworld wrote: post #10

Earlier quoted context omitted.

"Out-of-Control Theory" is a great name, love it =) thanks for the link, although I guess the idea never really took off.

It certainly did take off, it's called "Subsumption Architecture", and Rodney Brooks started iRobot, who created the Roomba, which is based on those ideas. https://en.wikipedia.org/wiki/Subsumption_architecture Subsumption architecture is a reactive robotic architecture heavily associated with behavior-based robotics which was very popular in the 1980s and 90s. The term was introduced by Rodney Brooks and colleagues…

Okay, interesting, but I meant nobody actually ended up sending thousands of little robots to other planets. No doubt the research led to some nice things.

Edit: the direct sensory-action coupling idea makes sense from a control perspective (fast interaction loops can compensate for chaotic dynamics in the environment), but we know these days that brains don't work that way, for instance. I wonder how that perspective has changed in robotics since the 90s, do you know?

Re: Rodney Brooks on limitations of generative AI

#48

Post by moffkalast »

cerved wrote: post #37
moffkalast wrote: post #32

Earlier quoted context omitted.

Llama-1, 1T tokens, dumb as a box of rocks Llama-2, 2T tokens, smarter than a box of rocks Mistral-7B, 8T tokens, way smarter than llama-2 Llama-3, 15T tokens, smarter than anything a few times its size Gemma-2, 13T synthetic tokens, slightly better than llama-3 (for the same approximate parameter size) I think it roughly tracks that moar data = moar betterer.

but the OP was talking about the size of the context window, not the size of the training corpus

Hmm right, I read that wrong. Still, interesting data I think.

Re: Rodney Brooks on limitations of generative AI

#49

Post by oska »

bubblyworld wrote: post #4
portaouflop wrote: post #3

Let’s call it machine learning, the AI term is just so far from what it actually is

He's talking about control theory, and other kinds of optimisation systems too. I think AI is a fine blanket term for all of that stuff.

There will never be, and can never be, "artificial intelligence". (The creation of consciousness is impossible.)

It's a fun/interesting device in science fiction, just like the concept of golems (animated beings) are in folk tales. But it's complete nonsense to talk about it as a possibility in the real world so yes, the label of 'machine learning' is a far, far better label to use for this powerful and interesting domain.

Re: Rodney Brooks on limitations of generative AI

#50

Post by bubblyworld »

oska wrote: post #49
bubblyworld wrote: post #4

Earlier quoted context omitted.

He's talking about control theory, and other kinds of optimisation systems too. I think AI is a fine blanket term for all of that stuff.

There will never be, and can never be, "artificial intelligence". (The creation of consciousness is impossible.) It's a fun/interesting device in science fiction , just like the concept of golems (animated beings) are in folk tales. But it's complete nonsense to talk about it as a possibility in the real world so yes, the label of 'machine learning' is a far, far better label to use for this powerful and interesting…

I'll happily engage in specifics if you provide an argument for your position. Here's mine (which is ironically self-defeating but has a grain of truth): single-sentence theories about reality are probably wrong.
Post reply on HN