Things we learned about LLMs in 2024 • Hacker News Board

Re: Things we learned about LLMs in 2024

#251

Post by simonw » Tue, Dec 31, 2024, 10:05 PM UTC

o11c wrote: post #243

lumost wrote: post #214

Earlier quoted context omitted.

When I started my career in 2010, google was a semi-serious skill. All of the little things that we know how to do now such as ignoring certain sites, lingering on others, and iteratively refining our search queries were not universally known at the time. Experienced engineers often relied on encyclopedic knowledge of their environment or by "reading the manual". In my experience, LLM tools are the same, you ask for…

The problems with that skill is that: * Most existing LLM interfaces are very bad at editing history, instead focusing entirely on appending to history. You can sort of ignore this for one-shot, and this can be properly fixed with additional custom tools, but ... * By the time you refine your input enough to patch over all the errors in the LLM's output for your sensible input, you're bigger than the LLM can actually…

Yeah, a key thing to understand about LLMs is that managing the context is everything. You need to know when to wipe the slate by starting a new chat session and then pasting across a subset of the previous conversation.

A lot of my most complex LLM interactions take place across multiple sessions - and in some cases I'll even move the project from Claude 3.5 Sonnet to OpenAI o1 (or vice versa) to help get out of a rut.

It's infuriatingly difficult to explain why I decide to do that though!

Re: Things we learned about LLMs in 2024

#252

Post by ac29 » Tue, Dec 31, 2024, 10:06 PM UTC

torginus wrote: post #177

datadrivenangel wrote: post #33

Earlier quoted context omitted.

It's justified if AGI is possible. If AGI is possible, then the entire human economy stops making sense as far as money goes, and 'owning' part of OpenAI gives you power. That is of course, assuming AGI is possible and exponential, and that marketshare goes to a single entity instead of a set of entities. Lots of big assumptions. Seems like we're heading towards a slow-lackluster singularity though.

I was thinking about how the economy has been actively makes less sense and gets divorced more and more from reality year after year, AI or not. It's the simple fact that the ability of assets to generate wealth has far outstripped the abiliy of individuals to earn money by working. Somehow real estate has become so expensive everywhere that owning a shitty apartment is impossible for the vast majority. When the worl…

> Somehow real estate has become so expensive everywhere that owning a shitty apartment is impossible for the vast majority.

Approximately 2/3s of homes in the US are owner occupied.

Re: Things we learned about LLMs in 2024

#253

Post by [deleted] » Tue, Dec 31, 2024, 10:09 PM UTC

duped wrote: post #202

antirez wrote: post #54

About "people still thinking LLMs are quite useless", I still believe that the problem is that most people are exposed to ChatGPT 4o that at this point for my use case (programming / design partner) is basically a useless toy. And I guess that in tech many folks try LLMs for the same use cases. Try Claude Sonnet 3.5 (not Haiku!) and tell me if, while still flawed, is not helpful. But there is more: a key thing with L…

> Try Claude Sonnet 3.5 (not Haiku!) and tell me if, while still flawed, is not helpful. It's not as helpful as Google was ten years ago. It's more helpful than Google today, because Google search has slowly been corrupted by garbage SEO and other LLM spam, including their own suggestions.

[deleted]

Re: Things we learned about LLMs in 2024

#254

Post by jdlshore » Tue, Dec 31, 2024, 10:10 PM UTC

th0ma5 wrote: post #223

More dishonest magical thinking. I wish this guy would learn how systems work and stop flooding the field with mystical nonsense unless he really is trying to make people think LLMs are worthless, then I guess he should be honest about it instead of subversive.

I read the article and thought it was well done and level-headed. What exactly did you think was mystical or magical thinking?

Re: Things we learned about LLMs in 2024

#255

Post by pixl97 » Tue, Dec 31, 2024, 10:12 PM UTC

ninth_ant wrote: post #99

antirez wrote: post #54

About "people still thinking LLMs are quite useless", I still believe that the problem is that most people are exposed to ChatGPT 4o that at this point for my use case (programming / design partner) is basically a useless toy. And I guess that in tech many folks try LLMs for the same use cases. Try Claude Sonnet 3.5 (not Haiku!) and tell me if, while still flawed, is not helpful. But there is more: a key thing with L…

I think a lot of the confusion is in how we approach LLMs. Perhaps stemming from the over-broad term “AI”. There are certain classes of problems that LLMs are good at. Accurately regurgitating all accumulated world knowledge ever is not one, so don’t ask a language model to diagnose your medical condition or choose a political candidate. But do ask them to perform suitable tasks for a language model! Every day by aut…

>don’t ask a language model to diagnose your medical condition

Honestly they are very decent at it if you give them accurate information in which to make the diagnosis. The typical problem people have is being unable to feed accurate information to the model. They'll cut out parts they don't want to think about or not put full test results in for consideration.

Re: Things we learned about LLMs in 2024

#256

Post by simonw » Tue, Dec 31, 2024, 10:12 PM UTC

calebm wrote: post #248

I love your breadth-first approach of having an outline at the top.

I wrote custom software for that! https://tools.simonwillison.net/render-markdown - If you paste in some Markdown with ## section headings in it the output will start with a list of links to those headings.

Re: Things we learned about LLMs in 2024

#257

Post by FooBarWidget » Tue, Dec 31, 2024, 10:14 PM UTC

antirez wrote: post #54

About "people still thinking LLMs are quite useless", I still believe that the problem is that most people are exposed to ChatGPT 4o that at this point for my use case (programming / design partner) is basically a useless toy. And I guess that in tech many folks try LLMs for the same use cases. Try Claude Sonnet 3.5 (not Haiku!) and tell me if, while still flawed, is not helpful. But there is more: a key thing with L…

Why do people have such narrow views on what makes LLMs useful? I use them for basically everything.

My son throwing an irrational tantrum at the amusement park and I can't figure out why he's like that (he won't tell me or he doesn't know himself either) or what I should do? I feed Claude all the facts of what happened that day and ask for advice. Even if I don't agree with the advice, at the very least the analysis helps me understand/hypothesize what's going on with him. Sure beats having to wait until Monday to call up professionals. And in my experience, those professionals don't do a better job of giving me advice than Claude does.

It's weekend, my wife is sick, the general practitioner is closed, the emergency weekend line has 35 people in the queue, and I want some quick half-assed medical guidance that while I know might not be 100% reliable, is still better than nothing for the next 2 hours? Feed all the symptoms and facts to Claude/ChatGPT and it does an okay job a lot of the time.

I've been visiting Traditional Chinese Medicine (TCM) practitioner for a week now and my symptoms are indeed reducing. But TCM paradigm and concepts are so different from western medicine paradigms and concepts that I can't understand the doctor's explanation at all. Again, Claude does a reasonable job of explaining to me what's going on or why it works from a western medicine point of view.

Want to write a novel? Brainstorm ideas with GPT-4o.

I had a debate with a friend's child over the correct spelling of a Dutch word ("instabiel" vs "onstabiel"). Google results were not very clear. ChatGPT explained it clearly.

Just where is this "useless" idea coming from? Do people not have a life outside of coding?

Re: Things we learned about LLMs in 2024

#258

Post by ninth_ant » Tue, Dec 31, 2024, 10:17 PM UTC

rrix2 wrote: post #114

ninth_ant wrote: post #99

Earlier quoted context omitted.

I think a lot of the confusion is in how we approach LLMs. Perhaps stemming from the over-broad term “AI”. There are certain classes of problems that LLMs are good at. Accurately regurgitating all accumulated world knowledge ever is not one, so don’t ask a language model to diagnose your medical condition or choose a political candidate. But do ask them to perform suitable tasks for a language model! Every day by aut…

> Every day by automation I feed in the hourly weather forecast my home ollama server and it builds me a nice readable concise weather report. not to dissuade you from a thing you find useful but are you aware that the national weather service produces an Area Forecast Discussion product in each local NWS office daily or more often that accomplishes this with human meteorologists and clickable jargon glossary? https:…

Doesn’t dissuade me at all, that’s a really neat service. I’m not American though, and even if my own country had a similar service I still enjoying tuning the results to focus on what I’m interested in. And it was just an example of the kinds of computer-human interfaces that are newly possible from this technology.

Anytime you have data and want it explained in a casual way — and it’s not mission critical to be extremely precise — LLMs are going to be a good option to consider.

More useful AGI-like behaviours may be enabled by combining LLMs with other technologies down the line, but we shouldn’t try to pretend that LLMs can do everything nor are they useless.

Re: Things we learned about LLMs in 2024

#259

Post by krapp » Tue, Dec 31, 2024, 10:20 PM UTC

FooBarWidget wrote: post #257

antirez wrote: post #54

About "people still thinking LLMs are quite useless", I still believe that the problem is that most people are exposed to ChatGPT 4o that at this point for my use case (programming / design partner) is basically a useless toy. And I guess that in tech many folks try LLMs for the same use cases. Try Claude Sonnet 3.5 (not Haiku!) and tell me if, while still flawed, is not helpful. But there is more: a key thing with L…

Why do people have such narrow views on what makes LLMs useful? I use them for basically everything. My son throwing an irrational tantrum at the amusement park and I can't figure out why he's like that (he won't tell me or he doesn't know himself either) or what I should do? I feed Claude all the facts of what happened that day and ask for advice. Even if I don't agree with the advice, at the very least the analysis…

Yes people have lives outside of coding, but most people are able to manage without having AI software intercede in as much of their lives as possible.

It seems like you trust AI more than people and prefer it to direct human interaction. That seems to be satisfying a need for you that most people don't have.

Re: Things we learned about LLMs in 2024

#260

Post by cloverich » Tue, Dec 31, 2024, 10:20 PM UTC

airstrike wrote: post #113

throwpoaster wrote: post #39

Earlier quoted context omitted.

157 billion implies about a 1% chance at dominating a 1.5 trillion market. Seems reasonable.

that's 10% and who's to say that market is worth 1.5 trillion to begin with

Market cap of apple, google, facebook.