Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things we learned about LLMs in 2024

simonwillison.net

Post reply on HN
Print view View original

271–280 of 615 posts

Re: Things we learned about LLMs in 2024

#271

Post by FooBarWidget »

krapp wrote: post #259
FooBarWidget wrote: post #257

Earlier quoted context omitted.

Why do people have such narrow views on what makes LLMs useful? I use them for basically everything. My son throwing an irrational tantrum at the amusement park and I can't figure out why he's like that (he won't tell me or he doesn't know himself either) or what I should do? I feed Claude all the facts of what happened that day and ask for advice. Even if I don't agree with the advice, at the very least the analysis…

Yes people have lives outside of coding, but most people are able to manage without having AI software intercede in as much of their lives as possible. It seems like you trust AI more than people and prefer it to direct human interaction. That seems to be satisfying a need for you that most people don't have.

When I'm done working, chased my children to properly finish their dinner, helped my son with homework, and putting them to bed, it's already 9+ PM — the only time of the day when I have free time. Just which human besides my wife can I talk to at that point? What if she doesn't have a clue either? All the professionals are only open when I'm working. A lot of the issues happen during the weekend, when professionals are closed. I don't want to disturb friends during the evening, and it's not like they have the expertise I need anyway.

LLMs are infinitely patient, don't think I am dumb for asking certain things, consider all the information I feed them, are available whenever I need them, have a wide range of expertise, and are dirt cheap compared to professionals.

That they might hallucinate is not a blocker most of the time. If the information I require is critical, I can always double check with my own research or with professionals (in which case the LLM has already primed me with a basic mental model so that I can ask quick, short, targeted questions, which saves the both of us time, and me money). For everything else (such as my curiocity on why TCM works, or the correct spelling of a word), LLMs are good enough.

Re: Things we learned about LLMs in 2024

#272

Post by karmakaze »

antirez wrote: post #54

About "people still thinking LLMs are quite useless", I still believe that the problem is that most people are exposed to ChatGPT 4o that at this point for my use case (programming / design partner) is basically a useless toy. And I guess that in tech many folks try LLMs for the same use cases. Try Claude Sonnet 3.5 (not Haiku!) and tell me if, while still flawed, is not helpful. But there is more: a key thing with L…

We're at the "computers play chess badly" stage. Then we'll hit the Deep Thought (1988) and Deep Blue (1995-1997) stages, but still saying that solving Go won't happen for 50+ years and that humans will continue to be better than computers.

The date/time that divides my world into before/after is AlphaGo v Lee Sedol game 3 (2016). From that time forward, I don't dismiss out of hand speculations of how soon we can have intelligent machines. Ray Kurzweil date of 2045 is as good as any (and better than most) for an estimate. Like Moore's (and related) Laws, it's not about how but the historical pace of advancements crossing a fairly static point of human capability.

Application coding, requires much less intelligence than playing Go at these high levels. The main differences are concise representation and clear final outcome scoring. LLMs deal quite well with the fuzziness of human communications. There may be a few more pegs to place but when seems predictably unknown.

Re: Things we learned about LLMs in 2024

#273

Post by grimgrin »

simonw wrote: post #251
o11c wrote: post #243

Earlier quoted context omitted.

The problems with that skill is that: * Most existing LLM interfaces are very bad at editing history, instead focusing entirely on appending to history. You can sort of ignore this for one-shot, and this can be properly fixed with additional custom tools, but ... * By the time you refine your input enough to patch over all the errors in the LLM's output for your sensible input, you're bigger than the LLM can actually…

Yeah, a key thing to understand about LLMs is that managing the context is everything . You need to know when to wipe the slate by starting a new chat session and then pasting across a subset of the previous conversation. A lot of my most complex LLM interactions take place across multiple sessions - and in some cases I'll even move the project from Claude 3.5 Sonnet to OpenAI o1 (or vice versa) to help get out of a…

I bought in early to typingmind, a great web based frontend. Good for editing context, and switching from say gemini to claude. This is a very normal flow for me, and whatever tool you use should enable this

also nice to interact with an LLM in vim, as the context is the buffer

obviously simon’s llm tool rules. I’ve wrapped it for vim

Re: Things we learned about LLMs in 2024

#274

Post by dartos »

simonw wrote: post #251
o11c wrote: post #243

Earlier quoted context omitted.

The problems with that skill is that: * Most existing LLM interfaces are very bad at editing history, instead focusing entirely on appending to history. You can sort of ignore this for one-shot, and this can be properly fixed with additional custom tools, but ... * By the time you refine your input enough to patch over all the errors in the LLM's output for your sensible input, you're bigger than the LLM can actually…

Yeah, a key thing to understand about LLMs is that managing the context is everything . You need to know when to wipe the slate by starting a new chat session and then pasting across a subset of the previous conversation. A lot of my most complex LLM interactions take place across multiple sessions - and in some cases I'll even move the project from Claude 3.5 Sonnet to OpenAI o1 (or vice versa) to help get out of a…

What kinds of things do you with these LLMs?

I feel like I’m good at understanding context. I’ve been working in AI startups over the last 2 years. Currently at an AI search startup.

Managing context for info retrieval is the name of the game.

But for my personal use as a developer, they’ve caused me much headache.

Answers that are subtly wrong in such a way that it took me a week to realize my initial assumption based on the LLM response was totally bunk.

This happened twice. With the yjs library, it gave me half incorrect information that led me to misimplementing the sync protocol. Granted it’s a fairly new library.

And again with the web history api. It said that the history stack only exists until a page reload. The examples it gave me ran as it described, but that isn’t how the history api works.

I lost a week of time because of that assumption.

I’ve been hesitant to dive back in since then. I ask questions every now and again, but I jump off much faster now if I even think it may be wrong.

Re: Things we learned about LLMs in 2024

#275

Post by HarHarVeryFunny »

mvkel wrote: post #235
antirez wrote: post #54

About "people still thinking LLMs are quite useless", I still believe that the problem is that most people are exposed to ChatGPT 4o that at this point for my use case (programming / design partner) is basically a useless toy. And I guess that in tech many folks try LLMs for the same use cases. Try Claude Sonnet 3.5 (not Haiku!) and tell me if, while still flawed, is not helpful. But there is more: a key thing with L…

I'm surprised at the description that it's "useless" as a programming / design partner. Even if it doesn't make "elegant" code (whatever that means), it's the difference between an app existing at all, or not. I built and shipped a Swift app to the App Store, currently generating $10,200 in MRR, exclusively using LLMs. I wouldn't describe myself as a programmer, and didn't plan to ever build an app, mostly because in…

Did you need a Mac for that, or is it possible to use Linux to develop a Swift app targeting iOS?

Would you mind sharing which app you released?

Re: Things we learned about LLMs in 2024

#276

Post by tootie »

hdjjhhvvhga wrote: post #69
antirez wrote: post #54

About "people still thinking LLMs are quite useless", I still believe that the problem is that most people are exposed to ChatGPT 4o that at this point for my use case (programming / design partner) is basically a useless toy. And I guess that in tech many folks try LLMs for the same use cases. Try Claude Sonnet 3.5 (not Haiku!) and tell me if, while still flawed, is not helpful. But there is more: a key thing with L…

While Claude Sonnet is superior than 4o for most my use cases, there are still occasionally some specific tasks where it performs slightly better.

Like what? Claude has become my go-to, but I find that it's wrong enough often enough that I really can't trust it for anything. If it says something I have to go dig through it's citations very carefully.

Re: Things we learned about LLMs in 2024

#277

Post by d0mine »

antirez wrote: post #75
hdjjhhvvhga wrote: post #69

Earlier quoted context omitted.

While Claude Sonnet is superior than 4o for most my use cases, there are still occasionally some specific tasks where it performs slightly better.

Probably. But statistically to work with 4o is a lose of time for me. LLMs is like an investment: you write the prompts, you "work" with them. If the LLM is too weak, this is a lose of time. You need to have a return on the investment that is positive. With ChatGPT 4o / o1 most of the times for me the investment of time has almost zero return. Before Claude Sonnet 3.5 I already had a ChatGPT PRO account but never use…

why "lose of time" instead of "loss of time" Is it a typo or fingerprinting?

Re: Things we learned about LLMs in 2024

#278

Post by dartos »

swalsh wrote: post #241
dartos wrote: post #198

> There’s a flipside to this too: a lot of better informed people have sworn off LLMs entirely because they can’t see how anyone could benefit from a tool with so many flaws. The key skill in getting the most out of LLMs is learning to work with tech that is both inherently unreliable and incredibly powerful at the same time. This is a decidedly non-obvious skill to acquire! I wish the author qualified this more. How…

It's amazing this is still an opinion in 2025. I now ask devs how they use AI as part of their workflows when I interview. It's a standard skill I expect my guys to have.

I feel bad for your team.

Let people work how they want. I wouldn’t not hire someone on the basis of them not using a language server.

The creator of the Odin language famously doesn’t use one. He’s says that he, specifically, is faster without one.

Re: Things we learned about LLMs in 2024

#279

Post by karmakaze »

jsheard wrote: post #190
com2kid wrote: post #93

Earlier quoted context omitted.

Apple uses HBM, basically RAM on the same die as the CPU. It has a lot more memory bandwidth than typically PC dram, but still less than many GPUs. (Although the highest end macs have bandwidth that is in the same ballpark as GPUs)

Apple does not use HBM, they use LPDDR. The way they use it is similar in principle to HBM (on-package, very wide bus) but it's not the same thing.

Right so Apple uses high-bandwidth memory, but not HBM.

Re: Things we learned about LLMs in 2024

#280

Post by krapp »

claar wrote: post #268
krapp wrote: post #259

Earlier quoted context omitted.

Yes people have lives outside of coding, but most people are able to manage without having AI software intercede in as much of their lives as possible. It seems like you trust AI more than people and prefer it to direct human interaction. That seems to be satisfying a need for you that most people don't have.

Why do you postulate that "most people don't have" this need? I also use AI non-stop throughout my day for similar uses. This feels identical to when I was an early "smart phone" user w/my palm pilot. People would condescend saying they didn't understand why I was "on it all the time". A decade or two later, I'm the one trying to get others to put down their phones during meetings. My take? Those who aren't using AI…

>Those who aren't using AI continually currently are simply later adopters of AI. Give it a few years - or at most a decade - and the idea of NOT asking 100+ AI queries per day (or per hour) will seem positively quaint.

I don't think you're wrong, I just think a future in which it's all but physically and socially impossible to have a single thought or communication not mediated by software is fucking terrifying.

Post reply on HN