Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things we learned about LLMs in 2024

simonwillison.net

Post reply on HN
Print view View original

431–440 of 615 posts

Re: Things we learned about LLMs in 2024

#431

Post by throwpoaster »

airstrike wrote: post #113
throwpoaster wrote: post #39

Earlier quoted context omitted.

157 billion implies about a 1% chance at dominating a 1.5 trillion market. Seems reasonable.

that's 10% and who's to say that market is worth 1.5 trillion to begin with

There’s a risk term I’m not including and the comparable is the size of the American economy ($27 trillion).

So take the entire economy and ask the question: what does AI not impact? Net that out and assume there’s pricing efficiencies, then build in a risk buffer.

1.5t to 15t seems right.

Re: Things we learned about LLMs in 2024

#432

Post by raincole »

tokai wrote: post #320
FooBarWidget wrote: post #257

Earlier quoted context omitted.

Why do people have such narrow views on what makes LLMs useful? I use them for basically everything. My son throwing an irrational tantrum at the amusement park and I can't figure out why he's like that (he won't tell me or he doesn't know himself either) or what I should do? I feed Claude all the facts of what happened that day and ask for advice. Even if I don't agree with the advice, at the very least the analysis…

This reads like satire to me. Scarry that it isn't.

I too read it as satire at first, but after thinking twice I think it's a quite reasonable take. I've added "utilize LLM more in my daily life outside programming" to my new year resolution.

Re: Things we learned about LLMs in 2024

#433

Post by ripped_britches »

dash2 wrote: post #209

Look, when are these models going to not just talk to me, but do stuff for me? If they're so clever, why can't I tell one to buy chocolates and send them to my wife? Meanwhile, they can allegedly solve frontier maths problems. What's the holdup to models that go online and perform simple tasks?

Same reason that a powerful graphing calculator can’t teach a math class. “Unhobbling” needs to occur. This means a lot of things but includes modalities, reliability, persistence, alignment, etc.

Re: Things we learned about LLMs in 2024

#435

Post by [deleted] »

simonw wrote: post #231
dartos wrote: post #198

> There’s a flipside to this too: a lot of better informed people have sworn off LLMs entirely because they can’t see how anyone could benefit from a tool with so many flaws. The key skill in getting the most out of LLMs is learning to work with tech that is both inherently unreliable and incredibly powerful at the same time. This is a decidedly non-obvious skill to acquire! I wish the author qualified this more. How…

One of the things I find most frustrating about LLMs is how resistant they are to teaching other people how to use them! I'd love to figure this out. I've written more about them than most people at this point, and my goal has always been to help people learn what they can and cannot do - but distilling that down to a concise set of lessons continues to defeat me. The only way to really get to grips with them is to u…

[deleted]

Re: Things we learned about LLMs in 2024

#436

Post by swalsh »

dartos wrote: post #278
swalsh wrote: post #241

Earlier quoted context omitted.

It's amazing this is still an opinion in 2025. I now ask devs how they use AI as part of their workflows when I interview. It's a standard skill I expect my guys to have.

I feel bad for your team. Let people work how they want. I wouldn’t not hire someone on the basis of them not using a language server. The creator of the Odin language famously doesn’t use one. He’s says that he, specifically, is faster without one.

My expectations around productivity are going to assume you're using AI. That means stuff that might have taken a few days, i'm going to expect in a few hours or less. It's not unreasonable, i've seen over and over agian that kind of speed up. I have a lot less approval to hire people than I used to... so it's really important to me that I can extract that level of productivity out of my team.

If you're "working the way you want to" ie still handrolling all your code, you're going to find my expectations unrealistic, and that is certainly not fair to you.

Re: Things we learned about LLMs in 2024

#437

Post by swalsh »

paxys wrote: post #337
theptip wrote: post #294

Earlier quoted context omitted.

No, it’s reasonable. If your team uses Git then it’s a valid question to establish if someone has only worked with Perforce. They didn’t say how heavily they weight the question. (All that said I expect that, soon, experience with the appropriate LLM tooling will be as important as having experience with the language your system is implemented in.)

You hire people based on their fundamental knowledge and the ability to learn, not skills in arbitrary tools and frameworks which come and go every other day. If someone has used Perforce they will be able to get perfectly comfortable with Git by the end of their first week. So not knowing Git is an idiotic reason to reject a skilled developer. Same with programming languages, and just about every other aspect of sof…

I don't really test any specific tools or frameworks, what i'm using has changed twice just in the last year. More so, I just want to hear that the candidate has some knowledge of what the current models can do well, what they can't do, and how they're integrating it. Whether you're copying pasting code or using something like cursor is not what i'm concerned about.

Re: Things we learned about LLMs in 2024

#438

Post by Sn0wCoder »

atombender wrote: post #322
antirez wrote: post #54

About "people still thinking LLMs are quite useless", I still believe that the problem is that most people are exposed to ChatGPT 4o that at this point for my use case (programming / design partner) is basically a useless toy. And I guess that in tech many folks try LLMs for the same use cases. Try Claude Sonnet 3.5 (not Haiku!) and tell me if, while still flawed, is not helpful. But there is more: a key thing with L…

Is there a way to use this in Jetbrains IDEs? (I've not been impressed with their AI Assistant.) There are a few plugins, but from the reviews they all seem kind of mediocre.

I use IntelliJ as my main coding tool but also use VSCode and Sublime text. If you have access to local LLMs or have an API key for some the Continue Plugin (basically Cursor but can use in IntelliJ) is the Best of the Best for IntelliJ (IMO). I have a box running some local models including Phind and StarCoder (plus some small embeddings) and have been super happy with the end product. The next up is Google Gemini Code Assist has been the best of the IntelliJ (non-configured) AI tools I have tried. There are better ones out there but IMO not for IntelliJ. It's still free for a few more weeks and I have been using it since the free release, fun to use. Can pre-prompt, say you are an expert XXX, please be funny, fill in the rest of your regular prompts. The Co-Pilot I use for work is very limited and will only answer coding questions. I tried to tell it that it was my coding buddy, and its name was Phil and told me it cannot have a personality or be funny. I believe the paid personal Co-Pilot allows you to choose which LLM it uses (I cannot confirm). The Phind VSCode plugin works really well. Also, the Phind coding models are on par with some of the other big ones and free if you have a subscription (or run locally). Sublime is around to open those GIG+ files as VSCode chocks and not worth the RAM of opening another IntelliJ.

Each task / programming language / query requires trying different LLM models and novel ways of prompting. If it's not work-related (or work pays for the one you use) sending as much of the code as relevant also helps the answers be more useful.

Most of the people I meet that say LLMs are not useful have only tried one (flavor / plugin), do not know how to pre-prompt or prompt, and do not give the tools a chance. Try one or two things, say yep, it's not good and give up.

Still hard for me to admit that Prompt Engineering is a profession, but it's the same as Google Fu. Once you learn it you can become an LLM Ninja!

I do not believe LLMs are coming for my job (just yet) but do believe they are going to be able to replace some people, are useful and those that do not use them will be at a disadvantage.

Re: Things we learned about LLMs in 2024

#439

Post by swalsh »

BeetleB wrote: post #244
swalsh wrote: post #241

Earlier quoted context omitted.

It's amazing this is still an opinion in 2025. I now ask devs how they use AI as part of their workflows when I interview. It's a standard skill I expect my guys to have.

Just curious, but what AI related skills do you expect them to have?

I ask what their current workflow is, how they check and verify things, what their approach to prompting is etc. I'm looking to see that they've developed basic skills, have a reasonable mental model of what models can do well, what they currently can't do, and have an approach to be productive using the tools.

Re: Things we learned about LLMs in 2024

#440

Post by simonw »

whatshisface wrote: post #434

Can someone please just tell me what model and workflow is so productive? I've seen so many allusions to the concept of skills for LLM use but no explanations of what they are.

The best LLM for code right now, in my opinion, is still Claude 3.5 Sonnet.

The big challenge is figuring out how to use it. I usually like working at the function level: I figure out the exact function signature I want in Python or JavaScript and then get Claude to implement it for me.

Claude Artifacts are neat too: Claude can build a full HTML+JavaScript UI, and then iterate on it. I use this for interactive UI prototypes and building small tools.

I've published a whole lot of notes on this stuff here: https://simonwillison.net/tags/ai-assisted-programming/

Post reply on HN