Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What we know about LLMs

willthompson.name

Post reply on HN
Print view View original

131–140 of 173 posts

Re: What we know about LLMs

#131

Post by Legend2440 »

mjr00 wrote: post #50
killernap wrote: post #13

ChatGPT was announced November, 2022 - 8 months ago. Time flies. Question for HN: Where are we in the hype cycle on this? We can run shitty clones slowly on Raspberry Pi's and your phone. The educational implementations demonstrate the basics in under a thousand lines of brisk C. Great. At some point you have to wonder... well, so what? Not one killer app has emerged. I for one am eager to be all hip and open minded…

> Not one killer app has emerged. I for one am eager to be all hip and open minded and pretend like I use LLMs all the time for everything and they are "the future" but novelty aside it seems like so far we have a demented clippy and some sophomoric arguments about alignment and wrong think. In my mind I divide LLM usage into two categories, creation and ingestion. Creation is largely a parlor trick that blew the min…

Image generators (midjourney, etc) are half LLM and still doing very impressive creation.

The big thing holding them back is legal/copyright concerns, but I expect that will be worked out eventually.

Re: What we know about LLMs

#132

Post by Legend2440 »

Jevon23 wrote: post #57
killernap wrote: post #13

ChatGPT was announced November, 2022 - 8 months ago. Time flies. Question for HN: Where are we in the hype cycle on this? We can run shitty clones slowly on Raspberry Pi's and your phone. The educational implementations demonstrate the basics in under a thousand lines of brisk C. Great. At some point you have to wonder... well, so what? Not one killer app has emerged. I for one am eager to be all hip and open minded…

>Not one killer app has emerged. Surely the “killer app” is ChatGPT itself? ChatGPT has already put some copywriters and journalists out of work, or at least reduced their hours. The app is quite literally “killing” something, i.e. people’s jobs. For those people, it’s not just empty hype. It’s very real. Certainly it’s already more real than anything having to do with blockchain/crypto.

I'm dubious. The few news websites that started publishing LLM articles (CNET, etc) were already circling the drain. They'd probably have fired their journalists anyway because they're on the edge of bankruptcy.

Re: What we know about LLMs

#133

Post by unethical_ban »

killernap wrote: post #13

ChatGPT was announced November, 2022 - 8 months ago. Time flies. Question for HN: Where are we in the hype cycle on this? We can run shitty clones slowly on Raspberry Pi's and your phone. The educational implementations demonstrate the basics in under a thousand lines of brisk C. Great. At some point you have to wonder... well, so what? Not one killer app has emerged. I for one am eager to be all hip and open minded…

I can't tell if this is satire or not. It is so... Well, to be polite, sounds so much like an uninformed stock trader, that I find it hard to believe this isn't some sort of meta commentary on hacker News conversations.

There are plenty examples of where the technology can eventually lead in terms of entertainment, impact on society and news, knowledge work, and so on. It doesn't have to happen immediately. But to handwave The myriad articles about the subject away and just say " I don't believe any of it, what else you got" is a bit annoying.

Re: What we know about LLMs

#134

Post by mdp2021 »

robblbobbl wrote: post #66
killernap wrote: post #13

ChatGPT was announced November, 2022 - 8 months ago. Time flies. Question for HN: Where are we in the hype cycle on this? We can run shitty clones slowly on Raspberry Pi's and your phone. The educational implementations demonstrate the basics in under a thousand lines of brisk C. Great. At some point you have to wonder... well, so what? Not one killer app has emerged. I for one am eager to be all hip and open minded…

This got way out of hand as of by now and isn't about serving humanity as a whole anymore in big parts (!). This is some actors with the money and hardware trying to build their AI dream castles up on the shoulders of the rest and even don't care what the implications of their actions are. Money is regulating this business and is taking more away from us all in the long term than it pays in the short. I'm kinda glad…

> not in for a discussion

?

Re: What we know about LLMs

#135

Post by myownpetard »

iainctduncan wrote: post #98
killernap wrote: post #13

ChatGPT was announced November, 2022 - 8 months ago. Time flies. Question for HN: Where are we in the hype cycle on this? We can run shitty clones slowly on Raspberry Pi's and your phone. The educational implementations demonstrate the basics in under a thousand lines of brisk C. Great. At some point you have to wonder... well, so what? Not one killer app has emerged. I for one am eager to be all hip and open minded…

I work in tech diligence so I look at companies in detail. I have seen a couple where good machine learning is going to make a massive difference (whether it will keep them ahead of everyone is a separate question). I think it really boils down to: "Is this a problem where an answer that is mostly right and sometimes wrong is still a great value proposition?" This is what people don't get. If sometimes the answer is…

Interesting final point. It's like the business equivalent of NP-Complete problems, difficult to compute but easy to verify.

Can you give any examples of those types of problems you've encountered?

Re: What we know about LLMs

#136

Post by mdp2021 »

Mertax wrote: post #72
killernap wrote: post #13

ChatGPT was announced November, 2022 - 8 months ago. Time flies. Question for HN: Where are we in the hype cycle on this? We can run shitty clones slowly on Raspberry Pi's and your phone. The educational implementations demonstrate the basics in under a thousand lines of brisk C. Great. At some point you have to wonder... well, so what? Not one killer app has emerged. I for one am eager to be all hip and open minded…

I wonder if language translation will be one of the "killer apps". Especially if it can be done real-time and according to the context/level of the audience/listener. Even within the same language, translation from a more technical/expert level to a simplified summary helps education/communication/knowledge transfer significantly.

> summary

That is an intellectual exercise, it requires understanding, and I have not yet seen an LLM implementation that does it properly. If you know one...

What I have seen are outputs that can give the illusion of a properly done job, if the user were willingly (or not) blind to quality.

So: non intellectual translation, we already had tools; intellectually valid one, then we would have much higher opportunities than translation.

Re: What we know about LLMs

#137

Post by jstarfish »

haswell wrote: post #52
killernap wrote: post #38

Earlier quoted context omitted.

It is great that everyone you know uses it but the traffic to ChatGPT is decreasing and has been for over two months now. If pointing this fact out makes me demented consider that perhaps you are emotionally invested in this new toy/brand. I guess we can wait and see what kind of usage trends will emerge long term. My anecdotal evidence (which is not worth much, same as yours) is that many normies tried it a few time…

> the traffic to ChatGPT is decreasing and has been for over two months now This seems entirely unsurprising, and isn’t by itself enough to support your general thesis. Interacting with these LLMs was extremely novel for most people when the tech first dropped, and those earlier months were the peak of the viral growth/expansion into public awareness. As the novelty dies down, it’s not surprising that there would be…

> Early on, I had all sorts of ridiculous conversations just to see what would happen. [...] That transition points to this being the opposite of a toy - after the fun dies down, the real work begins.

The "intelligence" behind it is too unpredictable to be reliable for work, and using it for fun is about as amusing as emailing HR.

Re: What we know about LLMs

#138

Post by jebarker »

adamcblodgett wrote: post #109
DebtDeflation wrote: post #108

Earlier quoted context omitted.

The killer app for large enterprises is Q&A against the corporate knowledgebase(s). Big companies have an insane amount of tribal knowledge locked away in documents sitting on Sharepoint, on Box, on file servers, etc. Best case scenario, their employees can do keyword search against a subset of those documents. Chunk those docs, run them through an embedding process, store the embeddings in a vector store, let employ…

So the killer app for LLMs and AI in general is...a librarian?

Yes! Never in my career have I seen an organization do a good job of organizing institutional knowledge and making it easily available to employees. It'd be a huge benefit to many organizations to be able to ask questions of the collective text holdings.

Re: What we know about LLMs

#139

Post by FishInTheWater »

famouswaffles wrote: post #117
FishInTheWater wrote: post #95

Earlier quoted context omitted.

Humans provide increasingly wrong answers as questions get more complex too. Human this, Human that. LLMs aren't humans. "My model is crap but the human brain isn't very good at this either" is irrelevant when we have machines that are not only very good at these tasks but almost perfect at them. Humans make such mistakes precisely because they are not perfect reasoning machines. To compare LLMs to humans is not only…

>Human this, Human that. LLMs aren't humans. You said you trivially proved something and made up nonsensical lines of reasoning to justify it. If your "proof" can't port to Humans then it's not proof. You are just rambling. >Humans make such mistakes precisely because they are not perfect reasoning machines. Nobody is calling LLMs perfect reasoning machines. Your "point" was that they don't reason at all which of non…

If your "proof" can't port to Humans then it's not proof

Learn to take a hint. I'm not going to argue this on human terms because you're playing a dumb um-akshually game.

Computer reasoning systems can solve vastly more complex problems perfectly. Expert mathematicians can solve vastly more complex problems with only minimally increased errors. The ability of LLMs to solve reasoning problems completely disintegrates when the problems get more complex.

Trying to argue that LLMs are alike humans because of you can put these three into the buckets of "No mistakes" and "Some mistakes" is ridiculous.

Nobody is calling LLMs perfect reasoning machines.

Yes.

You said humans make mistakes, my point here is, humans make mistakes precisely because they stop doing reasoning and start doing blind pattern matching estimation of the answer.

The idea that you must make no mistake reasoning before you can be considered to be reasoning has no ground.

Reading comprehension.

I did not say no mistakes. I said that the failure pattern follows that of estimated guesses; Rapidly increasing errors as the size of the problem increases.

Whereas with computer reasoning, the rate of errors does not increase at all. And with (expert) humans the rate only goes up a little.

Did you even bother looking at the link?

You are missing the point.

I am not referring to literally English or any other language. I'm referring to the structure of language problems, which is vastly simpler than any moderately complex math or programming problem.

To more explicitly spell out the reason for my unimpressed-ness: They trained a pattern-repeating-machine and found that it will repeat some of their patterns, some of which were patterns trained on.

This does not demonstrate the ability to reason abstractly about new models, so I do not care.

Re: What we know about LLMs

#140

Post by jsmith45 »

hallqv wrote: post #127
jawerty wrote: post #63

Earlier quoted context omitted.

So I'm not doing RLHF that's how LLama is pre-trained. It's in the loss/optimization phase in their training I believe. For the finetuning i'm using LoRA to freeze most of the layers for parameter optimization. Using PEFT from huggingface

RLHF is not part of LLaMa pretraining, or pretraning of any other models for that matter. RLHF comes after pretraining. https://twitter.com/Jeande_d/status/1661833563069620247/phot...

Seems like a classic case of a term of art overlapping with normal English terminology.

Knowing that you will be doing further training on a provided model (even "just" extensive fine-tuning), one would want to distinguish the training done before you get your hands on it, from the training you do. An obvious word for that previous training is pre-training, which unfortunately conflicts with a term of art.

Post reply on HN