Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will scaling work?

dwarkeshpatel.com

Post reply on HN
Print view View original

131–140 of 289 posts

Re: Will scaling work?

#131

Post by nextworddev »

cavisne wrote: post #129

If the size of the internet is really a bottleneck it seems Google is in quite a strong position. Assuming they have effectively a log of the internet, rather than counting the current state of the internet as usable data we should be thinking about the list of diffs that make up the internet. Maybe this ends up like Millenium Management where a key differentiator is having access to deleted datasets.

True that said market structure changes so rapidly that old datasets aren’t that useful for most strategies

Re: Will scaling work?

#132

Post by diggan »

nextworddev wrote: post #130

I am in the believer camp for simple reasons: 1) we haven’t even scratched the surface of government led investments into AI, 2) AI itself could probably discover better architectures than transformers (willing to bet heavily on this)

> AI itself could probably discover better architectures than transformers (willing to bet heavily on this)

Is there any existing cases of LLMs coming up with novel, useful and namely better architectures? Either related to AI/ML itself or any other field.

Re: Will scaling work?

#133

Post by stupidcar »

bee_rider wrote: post #108
HarHarVeryFunny wrote: post #96

Earlier quoted context omitted.

> The internet did change things pretty dramatically. For sure - I grew up in the mid-late 70s having to walk to the library to research stuff for homework, parents having to use the yellow-pages to find things, etc. Maybe smartphones are more of a game changer than desk-bound internet though - a global communication device in your pocket that'll give you driving directions, etc, etc. BUT ... does the world really FE…

I don’t know really, I was a kid in the 90’s. This is a bit far from the economic aspect, but the world currently seemed to be utterly suffused with a looming sense of dread, I think because we have, or know other people have, news notifications in their pockets telling us all about how bad things are. I don’t remember that feeling from the 90’s, but then, I was a kid. And of course before that there was the constant…

People who experienced a stable childhood seem to have a natural tendency to view the period they grew up in as, if not a golden age, then a safer, simpler time. Which makes sense: You’re too young to be aware of much of the complexity of the world, and your parents provide most of your essential needs and shield you from a lot of bad stuff.

That’s not to say all eras are the same. Clearly there’s better and worse times to be alive, but it’s hard to be objective about our childhoods.

Re: Will scaling work?

#134

Post by Jensson »

cortic wrote: post #114
enieslobby wrote: post #38

Earlier quoted context omitted.

Why do you think humans are basically evolved LLMs? Honest question, would love to read more about this viewpoint.

My first answer was a bit hasty, let me try again; We are clearly a product of our past experience (in LLMs this is called our datasets). If you go back to the beginning of our experiences, there is little identity, consciousness, or ability to reason. These things are learned indirectly, (in LLMs this is called an emergent property). We don't learn indiscriminately, evolved instinct, social pressure and culture guid…

Genes encode a ton of behaviors, you can't just ignore that. Tabula rasa doesn't exist among humans.

> If you go back to the beginning of our experiences, there is little identity, consciousness, or ability to reason.

That is because babies brains aren't properly developed. There is nothing preventing a fully conscious being from being born, you see that among animals etc. A newborn foal is a fully functional animal for example. Genes encode the ability to move around, identify objects, follow other beings, collision avoidance etc.

Re: Will scaling work?

#135

Post by __MatrixMan__ »

lossolo wrote: post #118
__MatrixMan__ wrote: post #71

Earlier quoted context omitted.

We do have systems that reason. Prolog comes to mind. It's a niche tool, used in isolated cases by relatively few people. I think that the other candidates are similar: proof assistants, physics simulators, computational chemistry and biology workflows, CAD, etc. When we get to the point where LLMs are able to invoke these tools for a user, even if that user has no knowledge of them, and are able to translate the res…

> We do have systems that reason. Prolog comes to mind. It's a niche tool, used in isolated cases by relatively few people. I think that the other candidates are similar: proof assistants, physics simulators, computational chemistry and biology workflows, CAD, etc. I think OP meant other definition of reason, because by your definition calculator can also reason. These are tools created by humans, that help them to r…

If an expert system is not reasoning, and a statistical apparatus like an LLM is not reasoning, then I think the only definition that remains is the rather antiquated one which defines reason as that capability which makes humans unique and separates us from animals.

I don't think it's likely to be a helpful one in this case.

Re: Will scaling work?

#136

Post by Jensson »

auggierose wrote: post #14
hokeone wrote: post #4

>Furthermore, the fact that LLMs seem to need such a stupendous amount of data to get such mediocre reasoning indicates that they simply are not generalizing. If these models can’t get anywhere close to human level performance with the data a human would see in 20,000 years, we should entertain the possibility that 2,000,000,000 years worth of data will be also be insufficient. There’s no amount of jet fuel you can a…

And yet, we reached the moon, and I would say airplanes were a necessary step on the way, even if only for psychological reasons. For airplanes we had at least an example in nature, birds. But I am not aware of any animal that travelled from earth to the moon on its own, except us.

What if LLMs are hot air balloons of flight, or kites of flight? Kites and hot air balloons didn't really lead to getting to the moon, they are a very different tangent.

Re: Will scaling work?

#137

Post by __MatrixMan__ »

cchance wrote: post #93
__MatrixMan__ wrote: post #71

Earlier quoted context omitted.

We do have systems that reason. Prolog comes to mind. It's a niche tool, used in isolated cases by relatively few people. I think that the other candidates are similar: proof assistants, physics simulators, computational chemistry and biology workflows, CAD, etc. When we get to the point where LLMs are able to invoke these tools for a user, even if that user has no knowledge of them, and are able to translate the res…

Ya i feel like this issue is people think an LLM will someday "wake up" no, LLM's will just be multimodal and developed to use tools, and a software ecosystem around it will end up using the LLM to reason how to execute, basically the LLM will be the internal monologue of whatever the AGI looks like.

Agreed. I think it's more likely that we'll reach a point where their complexity is so great that no single person can usefully reason about their outputs in relation to their structure.

Not so much a them waking up as an us falling asleep.

Re: Will scaling work?

#138

Post by HarHarVeryFunny »

stupidcar wrote: post #133
bee_rider wrote: post #108

Earlier quoted context omitted.

I don’t know really, I was a kid in the 90’s. This is a bit far from the economic aspect, but the world currently seemed to be utterly suffused with a looming sense of dread, I think because we have, or know other people have, news notifications in their pockets telling us all about how bad things are. I don’t remember that feeling from the 90’s, but then, I was a kid. And of course before that there was the constant…

People who experienced a stable childhood seem to have a natural tendency to view the period they grew up in as, if not a golden age, then a safer, simpler time. Which makes sense: You’re too young to be aware of much of the complexity of the world, and your parents provide most of your essential needs and shield you from a lot of bad stuff. That’s not to say all eras are the same. Clearly there’s better and worse ti…

That's certainly all true, and not just parents shielding you from bad stuff, but the bad stuff just not appearing on the TV or in the newspaper the way it will today on TV or internet. If it was going on then nobody was aware of it, and maybe not a bad thing. Is my life really better for reading about some teenage cartel hitman making human "stew" etc ?

But I do think that perhaps the 70's was a somewhat more decent time than today. Lines have been crossed and levels of violence normalized that it seems really didn't exist back then, or certainly were not as widespread. e.g. I grew up with the IRA constantly in the news - often bombings in the UK as well as violence in Northern Ireland. But, by today's standard the IRA's terrorism was almost quaint and gentlemanly ... they'd plant a bomb, but then call it into the police and/or media so that people could be evacuated - they still created terror/disruption which realistically probably did help them achieve their goals, but without the level of ultra violence and complete disregard for human life that we see today, such as ISIS beheadings posted on FaceBook or Twitter that some people happily watch and forward to their friends, or the 9/11 attack which was really inconceivable beforehand.

Re: Will scaling work?

#139

Post by [deleted] »

HarHarVeryFunny wrote: post #92
slibhb wrote: post #49

The best analogy for LLMs (up to and including AGI) is the internet + google search. Imagine explaining the internet/google to someone in 1950. That person might say "Oh my god, everything will change! Instantaneous, cheap communication! The world's information available at light speed! Science will accelerate, productivity will explode!" And yet, 70 years later, things have certainly changed, but we're living in the…

I think AGI can change the world once it gets way beyond human level both in terms of types of beyond-human "senses" and pattern matching/prediction (i.e. intelligence), but we are nowhere near that yet. On their current trajectory LLMs are just expert systems that will let certain types of simple job be automated. A potential productivity amplifier similar to having a personal assistant that you can assign tasks too…

[deleted]

Re: Will scaling work?

#140

Post by jeremyjh »

ctoth wrote: post #98
berniedurfee wrote: post #31

I think there’s a huge assumption here that more LLM will lead to AGI. Nothing I’ve seen or learned about LLMs leads me to believe that LLMs are in fact a pathway to AGI. LLMs trained on more data with more efficient algorithms will make for more interesting tools built with LLMs, but I don’t see this technology as a foundation for AGI. LLMs don’t “reason” in any sense of the word that I understand and I think the ab…

> I think there’s a huge assumption here that more LLM will lead to AGI. I'm not sure you realize this, but that is literally what this article was written to explore! I feel like you just autocompleted what you believe about large language models in this thread, rather than engaging with the article. Engagement might look like "I hold the skeptic position because of X, Y, and Z, but I see that the other position has…

I feel like an LLM would do a much better job than GP.
Post reply on HN