Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will scaling work?

dwarkeshpatel.com

Post reply on HN
Print view View original

171–180 of 289 posts

Re: Will scaling work?

#171

Post by herval »

ketzo wrote: post #152
herval wrote: post #90

Earlier quoted context omitted.

> And yet, 70 years later, things have certainly changed, but we're living in the same world with the same general patterns and limitations. With LLMs I expect something similar. Not a singularity, just a new, better tool that, yes, changes things, increases productivity, but leaves human societies more or less the same. by what criteria do you see the world as the same today vs 70 years ago?

I mean, very broad strokes, but I can see GP’s point. - people eat plants and animals - people pay money for goods and services - there are countries, sometimes they fight, sometimes they work together - men and women come together to create children, and often raise those children together etc, etc, etc The “bones” of what make up a capital-S Society are pretty much the same. None of these things had to stay the sam…

I mean, has _any_ change in _human history_ impacted those considerably? This argument is like saying we live the same way the cavemen did...

Re: Will scaling work?

#172

Post by joe_the_user »

skippyboxedhero wrote: post #116
espadrine wrote: post #46

Earlier quoted context omitted.

Demis Hassabis of Deepmind echoes a similar sentiment[0]: > I still think there are missing things with the current systems. […] I regard it a bit like the Industrial Revolution where there was all these amazing new ideas about energy and power and so on, but it was fueled by the fact that there were dead dinosaurs, and coal and oil just lying in the ground. Imagine how much harder the Industrial Revolution would hav…

His view of the Industrial Revolution is completely wrong. Societies pre-IR had multiple periods where energy usage increased significantly, some of them based specifically around coal. No IR. Early IR was largely based around the usage of water power, not coal. IR was pure innovation, people being able to imagine and create the impossible, it was going straight to nuclear already. Ironically, someone who is an innov…

Societies pre-IR had multiple periods where energy usage increased significantly, some of them based specifically around coal. No IR.

That's a straight up misstatement of the parent argument - the parent argued that coal was necessary, not that coal sufficient. True or not, the argument isn't refuted by the IR starting with water power either.

And pairing this with "anti-woke" jabs is discourse-diminishing stuff. The theory that petroleum was a key ingredient of the IR is much older than that (I don't even agree with it but it's better than "pure innovation" fluff).

Re: Will scaling work?

#173

Post by nyrikki »

sgt101 wrote: post #123

>Here’s one of the many astounding finds in Microsoft Research’s Sparks of AGI paper. They found that GPT-4 could write the LaTex code to draw a unicorn. a lot of people have tried to replicate this, I have tried. It's very hard to get GPT-4 to draw a unicorn, also asking it to draw an upside down unicorn is even harder.

Stocastic parrots are going to stochasticate.

The author also cited a few human assisted efforts as ML only.

The fact that the author also is surprised that GPT is better at falsifying user input while it struggles at new ideas demonstrates the fact that those who are hyping LLMs as getting us closer to strong AI don't know or ate ignoring the know limitations of problems like automated theorem solving.

I think generative AI is powerful and useful. But the AGI is near camp is starting to make it a hard sell because the general public is discovering the limits and people are trying to force it into inappropriate domains.

Over parameterization and double decent is great at expanding what it can do, but I haven't seen anything that justifies the AGI hype yet.

Re: Will scaling work?

#174

Post by lern_too_spel »

bob1029 wrote: post #170

I think the "self-play" path is where the scary-powerful AI solutions will emerge. This implies persistence of state and logic that lives external to the LLM. The language model is just one tool. AGI/ASI/whatever will be a system of tools, of which the LLM might be the least complicated one to worry about. In my view, domain modeling, managing state, knowing when to transition between states, techniques for final dec…

It's not necessary for the author's purpose of providing more data. We're only training on one kind of input so far, text, from which these models have built some understanding of the world. Humans train on more inputs, and the data to provide those inputs for training a model is readily available, in far larger quantities than individual human brains consume. Data is not the issue.

Re: Will scaling work?

#175

Post by YetAnotherNick »

saulpw wrote: post #163
YetAnotherNick wrote: post #107

Earlier quoted context omitted.

I meant 100,000x. At least for everyone I know, they have 100,000x data in mail/messaging/docs/notes/meeting etc. than their blog or any public site they own. Hell I would even say that if you just have all the meetings of zoom, it will be few order of magnitude higher than the entire public web.

If I have 1MB on my blog, 100,000x would be 100GB. Just, no. OOMs are not to be trifled with.

How many people have blogs? How many people sent any message or created a google docs? The answer could easily be 10,000x times of people having blog. Also I was just counting text content as I mentioned.

For reference, there are 175,000 authors in medium compared to billions using whatsapp or gmail or difference of around 50,000.

Re: Will scaling work?

#176

Post by lossolo »

__MatrixMan__ wrote: post #135
lossolo wrote: post #118

Earlier quoted context omitted.

> We do have systems that reason. Prolog comes to mind. It's a niche tool, used in isolated cases by relatively few people. I think that the other candidates are similar: proof assistants, physics simulators, computational chemistry and biology workflows, CAD, etc. I think OP meant other definition of reason, because by your definition calculator can also reason. These are tools created by humans, that help them to r…

If an expert system is not reasoning, and a statistical apparatus like an LLM is not reasoning, then I think the only definition that remains is the rather antiquated one which defines reason as that capability which makes humans unique and separates us from animals. I don't think it's likely to be a helpful one in this case.

Reasoning, in the context of artificial intelligence and cognitive sciences, can be seen as the process of drawing inferences or making decisions based on available information. This doesn't make machines like calculators or LLMs equivalent to human reasoning, but it does suggest they engage in some form of reasoning.

Expert systems, for instance, use a set of if-then rules derived from human expertise to make decisions in specific domains. This is a form of deductive reasoning, albeit limited and highly structured. They don't 'understand' in a human sense but operate within a framework of logic provided by humans.

LLMs, on the other hand, use statistical methods to generate responses based on patterns learned from vast amounts of data. This isn't reasoning in the traditional philosophical sense, but it's a kind of probabilistic reasoning. They can infer, locally generalize, and even 'extrapolate' to some extent within the bounds of their training data. However, this is not the same as human extrapolation, which often involves creativity and a deep understanding of context.

Re: Will scaling work?

#177

Post by lern_too_spel »

elktown wrote: post #106
visarga wrote: post #69

Earlier quoted context omitted.

There is no magic in the brain. There is no magic in LLMs. There is just new experience we gain by interacting with the environment and society. And there is the trove of past experience encoded in our books. We got smart by collecting experience, in other words, from outside. The magic in the brain was not in the brain, but everywhere else. What is experience? We are in state S, and take action A, and observe feedba…

> There is no magic in the brain. The amount of hubris we have in our field is deeply embarrassing. Imagine a neuroscientists reading that. The thought makes me blush.

Neuroscientists would agree with GP, otherwise they would be neuromystics instead of neuroscientists. There is no magic. It's all physical processes that we can eventually understand.

Re: Will scaling work?

#178

Post by FirmwareBurner »

Negitivefrags wrote: post #166
bee_rider wrote: post #60

Earlier quoted context omitted.

The internet did change things pretty dramatically. Productivity at information communication tasks just isn’t the entire economy. I think we are massively more productive. Some of the biggest new companies are ad companies (Google, Facebook), or spend a ton of their time designing devices that can’t be modified by their users (Apple, Microsoft). Even old fashioned companies like tractor and train companies have time…

I remember long ago reading an argument that information technology has not actually increased productivity. I really wish I could find a source for this now, but I just can't seem to find it anywhere on the internet. Here it is anyway: The administration of the Tax Service uses 4% of the total tax revenue it generates. This percentage has stayed relatively fixed over time. If IT really improved productivity, wouldn'…

>Either it doesn't do so directly, or it does do so directly, but all the efficiency gains are immediately consumed by more useless beurocracy.

That's how government digitalization has functioned in my country. It hasn't improved things, it just moved all the paper hassle to a digital hassle now where I need to go to Reddit to find out how to use it right and then do a back and forth to get it right. Same with the new digitalization of medical activities, a lot of doctors I know say it actually slows them down instead of making them more productive as they say they're now drowning in even more bureaucracy.

So depending on how you design and use your IT systems, they can improve things for you if done well, but they cal also slow you down if done poorly. And they're more often done poorly than great because the people in charge of ordering and buying them (governments, managers, execs, bean counters, etc) are not the same people who have to use them every day (doctors, taxpayers, clerks, employees in the trenches, etc).

I kind of feel the same way about the Slack "revolution". It hasn't made me more productive compared to the days when I was using IBM Lotus Sametime. Come to think of it, Slack and Teams, and all these IM apps designed around constant group chatting instead of 1-1, is actually making me less productive since it's full of SO .... MUCH ... NOISE, that I need to go out of my way to turn off or tune out in order to get any work done.

The famous F1 aerodinamic engineer, Arain Newey, doesn't even use computers, he has his secretary print out his emails every day which he reads at home and replies through his secretary the next day, and draws everything by hand on the drafting board and has the people below him draw them in CAD and send him the printed simulation results through his secretary, and guess what, his cars have been world class winning designs. So more IT and more sync communication, doesn't necessarily mean more results.

Re: Will scaling work?

#179

Post by beebmam »

slibhb wrote: post #49

The best analogy for LLMs (up to and including AGI) is the internet + google search. Imagine explaining the internet/google to someone in 1950. That person might say "Oh my god, everything will change! Instantaneous, cheap communication! The world's information available at light speed! Science will accelerate, productivity will explode!" And yet, 70 years later, things have certainly changed, but we're living in the…

> leaves human societies more or less the same

My mom, who is 70 years old, regularly tells me how profoundly transformative the internet has been for society.

Re: Will scaling work?

#180

Post by cortic »

Jensson wrote: post #134
cortic wrote: post #114

Earlier quoted context omitted.

My first answer was a bit hasty, let me try again; We are clearly a product of our past experience (in LLMs this is called our datasets). If you go back to the beginning of our experiences, there is little identity, consciousness, or ability to reason. These things are learned indirectly, (in LLMs this is called an emergent property). We don't learn indiscriminately, evolved instinct, social pressure and culture guid…

Genes encode a ton of behaviors, you can't just ignore that. Tabula rasa doesn't exist among humans. > If you go back to the beginning of our experiences, there is little identity, consciousness, or ability to reason. That is because babies brains aren't properly developed. There is nothing preventing a fully conscious being from being born, you see that among animals etc. A newborn foal is a fully functional animal…

>Genes encode a ton of behaviors, you can't just ignore that.

I'm not ignoring that, I'm just saying that in LLMs we call these things weights. And i don't want to downplay the importance of weights, its probably a significant difference between us and other hominids.

But even if you considered some behaviors to be more akin to the server or interface or preprocess in LLMs it still wouldn't detract from the fact that the vast majority of the things that make us autonomous logical sentient beings come about through a process that is very similar to the core workings of LLMs. I'm also not saying that all animal brains function like LLMs, though that's an interesting thought to consider.

Post reply on HN