Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will scaling work?

dwarkeshpatel.com

Post reply on HN
Print view View original

201–210 of 289 posts

Re: Will scaling work?

#201

Post by jodrellblank »

timacles wrote: post #99
hutzlibu wrote: post #94

Earlier quoted context omitted.

"Consciousness requires instincts in order to prioritize the endless streams of information. " What if "instinct" is also just (pretrained) model weight? The human brain is very complex and far from understood and definitely does NOT work like a LLM. But it likely shares some core concepts. Neuronal networks were inspired by brain synapses after all.

> What if "instinct" is also just (pretrained) model weight? Sure - then it will take the same amount of energy to train as our reptilian and higher brains took. That means trillions of real life experiences over millions of years.

Not at all, it took life hundreds of millions of years to develop brains that could work with language, and took us tens of thousands of years to develop languages and writing and universal literacy. Now computers can print it, visually read it, speech-to-text transcribe it, write/create/generate it coherently, text-to-speech output it, translate between languages, rewrite in different styles, explain other writings, and that only took - well, roughly one human lifetime since computers became a thing.

Re: Will scaling work?

#202

Post by SgtBastard »

Negitivefrags wrote: post #166
bee_rider wrote: post #60

Earlier quoted context omitted.

The internet did change things pretty dramatically. Productivity at information communication tasks just isn’t the entire economy. I think we are massively more productive. Some of the biggest new companies are ad companies (Google, Facebook), or spend a ton of their time designing devices that can’t be modified by their users (Apple, Microsoft). Even old fashioned companies like tractor and train companies have time…

I remember long ago reading an argument that information technology has not actually increased productivity. I really wish I could find a source for this now, but I just can't seem to find it anywhere on the internet. Here it is anyway: The administration of the Tax Service uses 4% of the total tax revenue it generates. This percentage has stayed relatively fixed over time. If IT really improved productivity, wouldn'…

>Either.

A third option: technology investments improved the efficiency of the previous tax base, which allowed the expansion of the tax base - through additional enforcement activity, increasing the tax base in absolute terms but also returning the overhead to its historical norms.

Without tracking the size of the tax base in inflation-adjusted terms, hard to account for.

(The cynically, you’re probably right re: useless bureaucratic expansion)

Re: Will scaling work?

#203

Post by dasil003 »

berniedurfee wrote: post #187
joe_the_user wrote: post #167

Earlier quoted context omitted.

I'm not sure you realize this, but that is literally what this article was written to explore! Yeah but it's "exploration" answers all the reasonable objections by just extrapolating vague "smartness" (EDITED [1]). "LLMs seem smart, more data will make 'em smarter..." If apparent intelligence were the only measure of where things are going, we could be certain GPT-5 or whatever would reach AGI. But I don't many peopl…

I’ve also heard it said that “apparent” intelligence is good enough to be called “real” intelligence if it’s indistinguishable from the real thing. That’s where I have a strong feeling that we’re missing the true meaning of intelligence, reasoning and consciousness. As you said, we may very well be a couple iterations away from a chatbot that is truly indistinguishable from a human, but I still strongly assert that e…

I agree with you, but I think it's more of a philosophical topic (ie. Chinese Room argument) than something that technicians working on raw LLM capabilities usually care to engage in. For them, the Turing Test and utility in applications are the most important thing.

Personally, I don't think we can construct an equivalent intelligence to a human out of silicon. That's not say AGI is unachievable or that it can't surpass human intelligence and be superficially undistinguishable from a human, but it will always be different and alien in some way. I believe our intelligence is fundamentally closer to other earth animals descended from common genetic ancestors than it can be to an artificial intelligence. As the creators of AI, we can and will paper over these differences enough to Get The Job Done™, but the uncanny valley will always be there if you know where to look.

Re: Will scaling work?

#204

Post by ketzo »

herval wrote: post #171
ketzo wrote: post #152

Earlier quoted context omitted.

I mean, very broad strokes, but I can see GP’s point. - people eat plants and animals - people pay money for goods and services - there are countries, sometimes they fight, sometimes they work together - men and women come together to create children, and often raise those children together etc, etc, etc The “bones” of what make up a capital-S Society are pretty much the same. None of these things had to stay the sam…

I mean, has _any_ change in _human history_ impacted those considerably? This argument is like saying we live the same way the cavemen did...

Some people claim AGI will. If you believe in the heights of “singularity” talk, we should expect some pretty fundamental changes to the basics of our lives.

Not sure how much stock I put in that, though.

Re: Will scaling work?

#205

Post by dnissley »

herval wrote: post #90
slibhb wrote: post #49

The best analogy for LLMs (up to and including AGI) is the internet + google search. Imagine explaining the internet/google to someone in 1950. That person might say "Oh my god, everything will change! Instantaneous, cheap communication! The world's information available at light speed! Science will accelerate, productivity will explode!" And yet, 70 years later, things have certainly changed, but we're living in the…

> And yet, 70 years later, things have certainly changed, but we're living in the same world with the same general patterns and limitations. With LLMs I expect something similar. Not a singularity, just a new, better tool that, yes, changes things, increases productivity, but leaves human societies more or less the same. by what criteria do you see the world as the same today vs 70 years ago?

Look around. The most significant change is that there are a lot more screens, and a lot more "cheap stuff" (consumer electronics, food, clothes, entertainment, plastic anything, etc).

Things "behind the scenes" have perhaps changed a lot -- e.g. financialization, more competitive markets, explosion of communication options, which are the driving force behind those visible changes.

Re: Will scaling work?

#206

Post by laurencerowe »

Negitivefrags wrote: post #166
bee_rider wrote: post #60

Earlier quoted context omitted.

The internet did change things pretty dramatically. Productivity at information communication tasks just isn’t the entire economy. I think we are massively more productive. Some of the biggest new companies are ad companies (Google, Facebook), or spend a ton of their time designing devices that can’t be modified by their users (Apple, Microsoft). Even old fashioned companies like tractor and train companies have time…

I remember long ago reading an argument that information technology has not actually increased productivity. I really wish I could find a source for this now, but I just can't seem to find it anywhere on the internet. Here it is anyway: The administration of the Tax Service uses 4% of the total tax revenue it generates. This percentage has stayed relatively fixed over time. If IT really improved productivity, wouldn'…

> The administration of the Tax Service uses 4% of the total tax revenue it generates. This percentage has stayed relatively fixed over time.

The tax administration is far more efficient than that. The IRS has 79K workers out of a total workforce of 158M, or 1/2000 workers. Federal taxes are about 19% GDP (28% of GDP including state and local taxes.) The IRS costs $14.3B to run and collects 19% of $25.46T = $4,800B or 0.3% of collected taxes.

> If IT really improved productivity, wouldn't you expect that that number would decrease, since Tax Administration is presumably an area that we should expect to see great gains from computerisation?

Those gains are so great that tax administration has been computerised for more than half a century now.

We could certainly save an awful lot more in tax preparation for the economy as a whole if we sent out pre-filled tax forms (the IRS already has all the information required for most people) like other countries do but the tax preparation companies have made a lot of contributions to politicians to prevent this.

Re: Will scaling work?

#207

Post by lern_too_spel »

elktown wrote: post #197
lern_too_spel wrote: post #194

Earlier quoted context omitted.

Just because we don't understand it doesn't mean it's magic. That's the whole point of science. > It's _our field_ that are the alchemist mystics, rambling about AGI/Philosopher's Stone in ever increasingly unhinged ways, These "ramblings" are what scientists call hypotheses. The people making these hypotheses have even proposed how to test them.

Even with added quotes you can't stop reading it literally? The total lack of critical thinking due to confirmation bias is just as embarrassing.

Visarga used the word literally and stated in broad strokes what the brain does (process input, update state, emit outputs) without the details of how it does so. You're the one who misinterpreted that as meaning we already know exactly how the brain works.

Re: Will scaling work?

#208

Post by stevenhuang »

timacles wrote: post #89
lumost wrote: post #43

Earlier quoted context omitted.

An LLM is simply a model which given a sequence, predicts the rest of the sequence. You can accurately describe any AGI or reasoning problem as an open domain sequence modeling problem. It is not an unreasonable hypothesis that brains evolved to solve a similar sequence modeling problem.

> It is not an unreasonable hypothesis that brains evolved to solve a similar sequence modeling problem. The real world is random, requires making decisions on incomplete information in situations that have never happened before. The real world is not a sequence of tokens. Consciousness requires instincts in order to prioritize the endless streams of information. One thing people dont want to accept about any AI is t…

The real world is informational. If the world is truly random and devoid of information, you wouldn't exist.

Re: Will scaling work?

#209

Post by az226 »

beardedwizard wrote: post #16
az226 wrote: post #11

Earlier quoted context omitted.

LLMs are closer to discoveries on the spectrum than inventions. Nobody predicted or planned the many emergent capabilities we’ve seen. Almost like magic. Now is a period of moving along the axis to invention with many intentional design, architecture, and feature development alongside testing and evaluation. We are far from done with LLMs, plenty of room for many more discoveries, lots to explore. It’s definitely a p…

> It’s definitely a precursor to AGI. What are you basing this claim on? There is no intelligence in an LLM, only humans fooled by randomness.

Precursor as in it will help in synthetic data generation, testing, etc. at scale that gives us more powerful models. It is a necessary intermediate step on our path to AGI.

Re: Will scaling work?

#210

Post by az226 »

nsagent wrote: post #21
az226 wrote: post #11

Earlier quoted context omitted.

LLMs are closer to discoveries on the spectrum than inventions. Nobody predicted or planned the many emergent capabilities we’ve seen. Almost like magic. Now is a period of moving along the axis to invention with many intentional design, architecture, and feature development alongside testing and evaluation. We are far from done with LLMs, plenty of room for many more discoveries, lots to explore. It’s definitely a p…

You might want to reconsider your stance on emergent abilities in LLMs considering the NeurIPS 2023 best paper winner is titled: "Are Emergent Abilities of Large Language Models a Mirage?" https://arxiv.org/abs/2304.15004 https://blog.neurips.cc/2023/12/11/announcing-the-neurips-20...

Models trained mostly in English and trained only to respond in English can respond in another language. That ability is not a mirage. It’s very very real.
Post reply on HN