Things we learned about LLMs in 2024 • Hacker News Board

Re: Things we learned about LLMs in 2024

#41

Post by Der_Einzige » Tue, Dec 31, 2024, 7:27 PM UTC

vunderba wrote: post #11

fosterfriends wrote: post #5

My fav part of the writeup at the end: """ LLMs need better criticism # A lot of people absolutely hate this stuff. In some of the spaces I hang out (Mastodon, Bluesky, Lobste.rs, even Hacker News on occasion) even suggesting that “LLMs are useful” can be enough to kick off a huge fight. I like people who are skeptical of this stuff. The hype has been deafening for more than two years now, and there are enormous quan…

I agree, but I think my biggest issue with LLMs (and a lot of GenAI) is that they act as a massive accelerator for the WORST (and unfortunately most common) type of human - the lazy one. The signal-to-noise ratio just goes completely out of control. https://journal.everypixel.com/ai-image-statistics

Sorry but the "lazy is bad" crowd is ludditism in another form, and it's telling that a whole lot of very smart people were passionate defenders of being lazy!

https://en.wikipedia.org/wiki/The_Human_Use_of_Human_Beings

https://en.wikipedia.org/wiki/Inventing_the_Future:_Postcapi...

https://en.wikipedia.org/wiki/The_Right_to_Be_Lazy

https://en.wikipedia.org/wiki/In_Praise_of_Idleness_and_Othe... (That's Bertrand Russell)

https://en.wikipedia.org/wiki/The_Abolition_of_Work

https://en.wikipedia.org/wiki/The_Society_of_the_Spectacle

https://en.wikipedia.org/wiki/Bonjour_paresse

AI systems are literally the most amazing technology on earth for this exact reason. I am so glad that it is destroying the minds of time thieves world-wide!

Re: Things we learned about LLMs in 2024

#42

Post by [deleted] » Tue, Dec 31, 2024, 7:28 PM UTC

datadrivenangel wrote: post #33

shihab wrote: post #20

Earlier quoted context omitted.

The last OpenAI valuation I read about was 157 billion. I am struggling to understand what justifies this. To me, it feels like OpenAI is at best few months ahead of competitors in some areas. But even if I am underestimating the advantage, it's few years instead of few months, why does it matter? It's not like AI companies are going to enjoy the first-mover advantage internet giants had over the competition.

It's justified if AGI is possible. If AGI is possible, then the entire human economy stops making sense as far as money goes, and 'owning' part of OpenAI gives you power. That is of course, assuming AGI is possible and exponential, and that marketshare goes to a single entity instead of a set of entities. Lots of big assumptions. Seems like we're heading towards a slow-lackluster singularity though.

[deleted]

Re: Things we learned about LLMs in 2024

#43

Post by simonw » Tue, Dec 31, 2024, 7:28 PM UTC

Der_Einzige wrote: post #26

RE: Slop: Having Slop generations from an LLM is a choice. There are so many tricks to make models genuinely creative just at the sampler level alone. https://github.com/sam-paech/antislop-sampler https://openreview.net/forum?id=FBkpCyujtS

It doesn't matter how good the generated text is: it is still slop if the recipient didn't request it and no human has reviewed it.

Re: Things we learned about LLMs in 2024

#44

Post by philipkglass » Tue, Dec 31, 2024, 7:29 PM UTC

datadrivenangel wrote: post #33

shihab wrote: post #20

Earlier quoted context omitted.

The last OpenAI valuation I read about was 157 billion. I am struggling to understand what justifies this. To me, it feels like OpenAI is at best few months ahead of competitors in some areas. But even if I am underestimating the advantage, it's few years instead of few months, why does it matter? It's not like AI companies are going to enjoy the first-mover advantage internet giants had over the competition.

It's justified if AGI is possible. If AGI is possible, then the entire human economy stops making sense as far as money goes, and 'owning' part of OpenAI gives you power. That is of course, assuming AGI is possible and exponential, and that marketshare goes to a single entity instead of a set of entities. Lots of big assumptions. Seems like we're heading towards a slow-lackluster singularity though.

If AGI is possible, then the entire human economy stops making sense as far as money goes, and 'owning' part of OpenAI gives you power.

That's if AGI is possible and not easily replicated. If AGI can be copied and/or re-developed like other software then the value of owning OpenAI stock is more like owning stock in copper producers or other commodity sector companies. (It might even be a poorer investment. Even AGI can't create copper atoms, so owners of real physical resources could be in a better position in a post-human-labor world.)

Re: Things we learned about LLMs in 2024

#45

Post by throwup238 » Tue, Dec 31, 2024, 7:29 PM UTC

comte7092 wrote: post #14

agentultra wrote: post #2

Don’t forget that 2024 was also a record year for new methane power plant projects. Some 200 new projects in the US alone and I’d wager most of them are funded directly by big tech for AI data centres. https://www.bnnbloomberg.ca/investing/2024/09/16/ai-boom-is-... This is definitely extending the runway of O&G at a crisis point in the climate disaster when we’re supposed to be reducing and shutting down these power…

Energy generation methods aren’t fungible. Methane is favored in many cases because they can be quickly ramped up and down to handle momentary peaks in demand or spotty supply generated from renewables. Without knowing more details about those projects it is difficult to make the claim that these plants have anything to do with increased demand due to LLMs, though if anything, they’d just add to base load demands and…

Methane is also worth burning to lessen the GHG impact since we produce so much of it as a byproduct of both resource extraction and waste disposal anyway.

Re: Things we learned about LLMs in 2024

#46

Post by asqueella » Tue, Dec 31, 2024, 7:29 PM UTC

throwpoaster wrote: post #39

shihab wrote: post #20

Earlier quoted context omitted.

The last OpenAI valuation I read about was 157 billion. I am struggling to understand what justifies this. To me, it feels like OpenAI is at best few months ahead of competitors in some areas. But even if I am underestimating the advantage, it's few years instead of few months, why does it matter? It's not like AI companies are going to enjoy the first-mover advantage internet giants had over the competition.

157 billion implies about a 1% chance at dominating a 1.5 trillion market. Seems reasonable.

10%, no?

Re: Things we learned about LLMs in 2024

#47

Post by Terr_ » Tue, Dec 31, 2024, 7:30 PM UTC

datadrivenangel wrote: post #33

shihab wrote: post #20

Earlier quoted context omitted.

The last OpenAI valuation I read about was 157 billion. I am struggling to understand what justifies this. To me, it feels like OpenAI is at best few months ahead of competitors in some areas. But even if I am underestimating the advantage, it's few years instead of few months, why does it matter? It's not like AI companies are going to enjoy the first-mover advantage internet giants had over the competition.

It's justified if AGI is possible. If AGI is possible, then the entire human economy stops making sense as far as money goes, and 'owning' part of OpenAI gives you power. That is of course, assuming AGI is possible and exponential, and that marketshare goes to a single entity instead of a set of entities. Lots of big assumptions. Seems like we're heading towards a slow-lackluster singularity though.

One strata in that assumption-heap to call out explicitly: Assuming LLMs are an enabling route to AGI and not a dead-end or supplemental feature.

Re: Things we learned about LLMs in 2024

#48

Post by api » Tue, Dec 31, 2024, 7:31 PM UTC

datadrivenangel wrote: post #33

shihab wrote: post #20

Earlier quoted context omitted.

The last OpenAI valuation I read about was 157 billion. I am struggling to understand what justifies this. To me, it feels like OpenAI is at best few months ahead of competitors in some areas. But even if I am underestimating the advantage, it's few years instead of few months, why does it matter? It's not like AI companies are going to enjoy the first-mover advantage internet giants had over the competition.

It's justified if AGI is possible. If AGI is possible, then the entire human economy stops making sense as far as money goes, and 'owning' part of OpenAI gives you power. That is of course, assuming AGI is possible and exponential, and that marketshare goes to a single entity instead of a set of entities. Lots of big assumptions. Seems like we're heading towards a slow-lackluster singularity though.

If AGI is possible then that too becomes a commodity and we experience a massive round of deflation in the cost of everything not intrinsically rare. Land, food, rare materials, energy, and anything requiring human labor is expensive and everything else is almost free.

I don't see how OpenAI wouldn't crash and burn here. Given the history of models it would be at most a year before you'd have open AGI, then the horse is out of the barn and the horse begins to self-improve. Pretty soon the horse is a unicorn, then it's a Satyr, and so on.

(I am a near-term AGI skeptic BTW, but I could be wrong.)

OpenAI's valuation is a mixture of hype speculation and the "golden boy" cult around Sam Altman. In the latter sense it's similar to the golden boy cults around Elon Musk and (politically) Donald Trump. To some extent these cults work because they are self-fulfilling feedback loops: these people raise tons of capital (economic or political) because everyone knows they're going to raise tons of capital so they raise tons of capital.

Re: Things we learned about LLMs in 2024

#49

Post by Der_Einzige » Tue, Dec 31, 2024, 7:31 PM UTC

simonw wrote: post #43

Der_Einzige wrote: post #26

RE: Slop: Having Slop generations from an LLM is a choice. There are so many tricks to make models genuinely creative just at the sampler level alone. https://github.com/sam-paech/antislop-sampler https://openreview.net/forum?id=FBkpCyujtS

It doesn't matter how good the generated text is: it is still slop if the recipient didn't request it and no human has reviewed it.

By that definition machine to machine communication that happens "organically" (like how humans do it, where they sometimes strike up conversations unprompted with each other) is "slop".

You're not seeing how the future of the world will develop.

Re: Things we learned about LLMs in 2024

#50

Post by simonw » Tue, Dec 31, 2024, 7:31 PM UTC

henning wrote: post #37

Spookily good at writing code? LLMs frequently hallucinate broken nonsense shit when I use them. Recognize what they do well (generate simple code in popular languages) while acknowledging where they are weak (non-trivial algorithms, any novel code situation the LLM hasn't seen before, less popular languages).

Did you try learning HOW to get good code out of them?

As with all things LLM there's a whole lot of undocumented and under appreciated depth to getting decent results.

Code hallucinations are also the least damaging type of hallucinations, because you get fact checking for free: if you run the code and get an error you know there's a problem.

A lot of the time I find pasting that error message back into the LLM gets me a revision that fixes the problem.