Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why is it taken for granted that LLM models will keep improving?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 68 posts

Re: Ask HN: Why is it taken for granted that LLM models will keep improving?

#31

Post by fatherzine »

pinkgolem wrote: post #18
fragmede wrote: post #6

Because a lot of smart people are spending a lot of time, money, and effort on this. It's as simple as that. We could go into all sorts of details, like how increase in GPU capabilities will improve training capabilities, both in size and speed, or how GPU(/TPU) capabilities will improve, or how better techniques will make training on the same data set result in better models, or where other improvements will make be…

I mean a lot of highly payed/intelligent people worked in crypto/fusion/quantum computing, still all of those topics are evolving rather gradually.

a more apt analogy is Moore's law

Re: Ask HN: Why is it taken for granted that LLM models will keep improving?

#32

Post by fatherzine »

haltist wrote: post #3

The main assumption of techno-optimism is that a large enough computer can do anything people can do and it can do it better. The goal of techno-optimism is to create a mechanical god that will rule the planet and scaling LLMs is a stepping stone to that goal. I, of course, already know how to do all this for a mere $80B.

"create a mechanical god that will rule the planet" -- on what basis people call this 'optimism'?!

Re: Ask HN: Why is it taken for granted that LLM models will keep improving?

#33

Post by karaterobot »

Not an expert, but have wondered the same thing. From what I've read, it comes down to optimism and extrapolation from current trends. Both of these have problems of course, but what else can you do? My working hypothesis is that we'll reach a practical limit on the quality of what we can get from the current class of models, and to extend beyond that would require a new approach, rather than just more data and more horsepower. The new breakthrough would have to be as significant as the last, but would be more likely to happen in a short time span because there is so much more activity in AI research now than even 5 years ago. Again, I'm a dummy about this stuff, not claiming more than that.

Re: Ask HN: Why is it taken for granted that LLM models will keep improving?

#34

Post by fatherzine »

ActorNightly wrote: post #11

I dunno if LLMs will get better, but ML in general is a task of compression, and there is definitely a whole bunch of human knowledge and history that neural nets can compress. Its not unfeasable in the future to have a box at home that you can ask a fairly complicated question, like "how do I build a flying car", and it will have the ability to - tell you step by step instructions of what you need to order - write a…

unbounded possibilities! imagine, and hear me out, asking the box: 'how do I build a box that can answer fairly complicated questions', and getting the output in an automatized way all the way from atoms, ready to be plugged into the power grid.

Re: Ask HN: Why is it taken for granted that LLM models will keep improving?

#35

Post by jrm4 »

Your skepticism is, I think, very well founded -- especially with such unclear definitions of "improvement."

I think I have a corollary type idea: Why are LLM's not perhaps like "Linux," something than never really needs to be REWRITTEN from scratch, merely added to or improved on? In other words, isn't it fair to think that LoRA's are the really important thing to pay attention to?

(And perhaps, like Google Fuschia or whatever, new LLMs might just be mostly a waste of time from an innovators POV?)

Re: Ask HN: Why is it taken for granted that LLM models will keep improving?

#36

Post by ortusdux »

I enjoyed Tom Scott's YT video monologue about this. To summarize, he postulates that most major innovations follow a sigmoid growth curve, wherein they ramp up, explode, and then level off. The question then becomes, where are we on this curve? He concludes that we will probably only know in hindsight.

https://www.youtube.com/watch?v=jPhJbKBuNnA

Re: Ask HN: Why is it taken for granted that LLM models will keep improving?

#37

Post by quickthrower2 »

joecool1029 wrote: post #27
imranq wrote: post #15

LLMs are comprised of just three elements Data Compute Algorithms All three are just scratching the surface of what is possible. Data: What has been scraped off the internet is just Compute: While 3nm processes are approaching an atomic limit (0.21nm for Si), there is still room to explore more densely packed transistors or other materials like Gallium Nitride or optical computing. Not only that but there is a lot of…

For data though, as LLM's generate more output, over time wouldn't they be expected to mess themselves with their own generated data? Wouldn't that be the wall we'll hit? Think of how shitted up Google Search is with generated garbage, I'm imagining we're already in the 'golden age' where we were able to train on good datasets before it gets 'polluted' with LLM generated data that may not be accurate, and it just con…

We need models that need less language data to train. Babies learn to talk on way less data than the entire internet. We need something closer to human experience. Kids have a feel for what is bullshit before they have consumed the entire internet :-).

I think feeding the internet into a LLM will be seen as the mainframe days of AI.

Re: Ask HN: Why is it taken for granted that LLM models will keep improving?

#38

Post by [deleted] »

discreteevent wrote: post #24
imranq wrote: post #15

LLMs are comprised of just three elements Data Compute Algorithms All three are just scratching the surface of what is possible. Data: What has been scraped off the internet is just Compute: While 3nm processes are approaching an atomic limit (0.21nm for Si), there is still room to explore more densely packed transistors or other materials like Gallium Nitride or optical computing. Not only that but there is a lot of…

> LLMs are comprised of just three elements > Data > Compute > Algorithms Not to be facetious but so is all other software. LLMs appear to scale in correlation to the first two but it's not clear what the correlation is and that's the basis of the question being asked.

[deleted]

Re: Ask HN: Why is it taken for granted that LLM models will keep improving?

#39

Post by fatherzine »

afjeafaj848 wrote: post #13

I think part of it is that some people say a person is 20 petaflops of compute So if we have that much compute power already why can't we just configure it in the right way to match a human brain? I'm not sure I totally buy that logic though, since I would think the architecture/efficiency of a brain is way different from a computer

what if we already did and all that's missing is the skeletomuscular system?
Post reply on HN