Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will scaling work?

dwarkeshpatel.com

Post reply on HN
Print view View original

211–220 of 289 posts

Re: Will scaling work?

#211

Post by Negitivefrags »

laurencerowe wrote: post #206
Negitivefrags wrote: post #166

Earlier quoted context omitted.

I remember long ago reading an argument that information technology has not actually increased productivity. I really wish I could find a source for this now, but I just can't seem to find it anywhere on the internet. Here it is anyway: The administration of the Tax Service uses 4% of the total tax revenue it generates. This percentage has stayed relatively fixed over time. If IT really improved productivity, wouldn'…

> The administration of the Tax Service uses 4% of the total tax revenue it generates. This percentage has stayed relatively fixed over time. The tax administration is far more efficient than that. The IRS has 79K workers out of a total workforce of 158M, or 1/2000 workers. Federal taxes are about 19% GDP (28% of GDP including state and local taxes.) The IRS costs $14.3B to run and collects 19% of $25.46T = $4,800B o…

It's possible that I misremembered 0.4% as 4%. As I said, it was long time ago.

Re: Will scaling work?

#212

Post by FergusArgyll »

I love Dwarkesh, his podcast is phenomenal.

But every article/post of this kind immediately begs the question; What is AGI? I have yet to hear even a decent standard.

It always seems like I'm reading Greek philosophers struggling with things they have almost no understanding of and just throwing out the wildest theories.

Honestly, it raised my opinion of them seeing how hard it is to reason about things which we have no grasp of.

Re: Will scaling work?

#213

Post by simonebrunozzi »

YetAnotherNick wrote: post #8

> ‘5 OOMs off’ I think Google, Microsoft and facebook could easily have 5 OOM data than the entire public web combined if we just count text. Majority of people don't have any content on public web except for personal photos. A minority has few public social media posts and it is rare for people to write blog or research paper etc. And almost everyone has some content written in mail or docs or messaging.

Try the NSA. We speak a lot on the phone, and we did so for the past 30 years.

Re: Will scaling work?

#215

Post by LunicLynx »

Here is an idea. Maybe the most optimized neural network is the brain. Computation to energy consumption ratio. So essentially the way doing this in silicon is just pointless.

There must be a reason we can do so much while consuming so little, and then again struggling with other tasks.

What is the success if we build a machine that consumes just heaps of energy and then is as bad in maths as us?

Re: Will scaling work?

#216

Post by gnaritas99 »

crowbahr wrote: post #76
visarga wrote: post #69

Earlier quoted context omitted.

There is no magic in the brain. There is no magic in LLMs. There is just new experience we gain by interacting with the environment and society. And there is the trove of past experience encoded in our books. We got smart by collecting experience, in other words, from outside. The magic in the brain was not in the brain, but everywhere else. What is experience? We are in state S, and take action A, and observe feedba…

To say there's no magic in the brain drastically *minimizes the complexity of the brain. Your brain is several orders of magnitude more complex than even the largest LLM. GPT4 has 1 trillion parameters? Big deal. Your brain has 1 quadrillion synapses, constantly shifting. Beyond that the synapses are analog messages, not binary. Each synapse is approximately like 1000 transistors based on the granularity of messaging…

[dead]

Re: Will scaling work?

#217

Post by marcosdumay »

Negitivefrags wrote: post #166
bee_rider wrote: post #60

Earlier quoted context omitted.

The internet did change things pretty dramatically. Productivity at information communication tasks just isn’t the entire economy. I think we are massively more productive. Some of the biggest new companies are ad companies (Google, Facebook), or spend a ton of their time designing devices that can’t be modified by their users (Apple, Microsoft). Even old fashioned companies like tractor and train companies have time…

I remember long ago reading an argument that information technology has not actually increased productivity. I really wish I could find a source for this now, but I just can't seem to find it anywhere on the internet. Here it is anyway: The administration of the Tax Service uses 4% of the total tax revenue it generates. This percentage has stayed relatively fixed over time. If IT really improved productivity, wouldn'…

> If IT really improved productivity, wouldn't you expect that that number would decrease

A large change on an otherwise stagnant activity... I would expect it to increase quickly.

If that number is stable for that long, it means it's defined by some factor that doesn't depend on its performance.

Re: Will scaling work?

#218

Post by elktown »

lern_too_spel wrote: post #207
elktown wrote: post #197

Earlier quoted context omitted.

Even with added quotes you can't stop reading it literally? The total lack of critical thinking due to confirmation bias is just as embarrassing.

Visarga used the word literally and stated in broad strokes what the brain does (process input, update state, emit outputs) without the details of how it does so. You're the one who misinterpreted that as meaning we already know exactly how the brain works.

No, not at all. The only thing I did was to react on how ridiculous that oversimplification is and how such a thing can only come about due to an embarrassing amount of hubris currently going around our field with relation to "AI". It's a hand-wavy "Eh, how hard can it be?" comment to rationalize ML being a pathway to AGI.

Re: Will scaling work?

#219

Post by lern_too_spel »

elktown wrote: post #218
lern_too_spel wrote: post #207

Earlier quoted context omitted.

Visarga used the word literally and stated in broad strokes what the brain does (process input, update state, emit outputs) without the details of how it does so. You're the one who misinterpreted that as meaning we already know exactly how the brain works.

No, not at all. The only thing I did was to react on how ridiculous that oversimplification is and how such a thing can only come about due to an embarrassing amount of hubris currently going around our field with relation to "AI". It's a hand-wavy "Eh, how hard can it be?" comment to rationalize ML being a pathway to AGI.

There is nothing oversimplified in that description at all. Here is an equivalent description from a neuroscience textbook:

"Neuroscience is the study of the nervous system, the collection of nerve cells that interpret all sorts of information which allows the body to coordinate activity in response to the environment."

https://openbooks.lib.msu.edu/introneuroscience1/chapter/wha...

This is even simpler than the RL agent description of the brain that visagra provided earlier, ignoring that the brain must have some internal state instead of being a pure function from each input to each response. Would you say that neuroscientists have even more hubris?

Re: Will scaling work?

#220

Post by sweezyjeezy »

LunicLynx wrote: post #215

Here is an idea. Maybe the most optimized neural network is the brain. Computation to energy consumption ratio. So essentially the way doing this in silicon is just pointless. There must be a reason we can do so much while consuming so little, and then again struggling with other tasks. What is the success if we build a machine that consumes just heaps of energy and then is as bad in maths as us?

There's a couple of false dichotomies here

- to say that because we're "more optimised" we must be the most optimised. Our brains are optimised well for certain things, sure, but computers are far more efficient at e.g. crunching numbers than we are

- to say that there's no success in a machine that can't currently beat us at math - this year has already proven that false

Post reply on HN