Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things we learned about LLMs in 2024

simonwillison.net

Post reply on HN
Print view View original

391–400 of 615 posts

Re: Things we learned about LLMs in 2024

#391

Post by BeetleB »

dartos wrote: post #327
BeetleB wrote: post #242

Earlier quoted context omitted.

I think most tech folks struggle with it because they treat LLMs as computer programs, and their experience is that SW should be extremely reliable - imagine using a calculator that was wrong 5% of the time - no one would accept that! Instead, think of an LLM as the equivalent of giving a human a menial task. You know that they're not 100% reliable, and so you give them only tasks that you can quickly verify and corr…

> Don't use LLMs where accuracy is paramount. Then why do people keep pushing it for code related tasks? Accuracy and precision is paramount with code. It needs to express exactly what needs to be done and how.

> Then why do people keep pushing it for code related tasks?

They don't. You are likely experiencing selection bias. My guess is you work in SW, and so it makes sense that you're the target of those campaigns. The bulk of ChatGPT subscribers are not doing SW, and no one is bugging them to use it for code related tasks.

Re: Things we learned about LLMs in 2024

#392

Post by blibble »

zachrip wrote: post #23
ToucanLoucan wrote: post #6

Earlier quoted context omitted.

It would be really cool if big tech could find a new hyperscaler model that didn't also require offsetting the goals of green energy projects worldwide. Between LLM and crypto you'd swear they're trying to find the most energy-wasteful tech possible.

It seems odd to put crypto and LLMs in the same boat in this regard - I might be wrong but are there any crypto projects that actually provide value? I'm sure there are ones that do folding or something but among the big ones?

crypto has real uses, most of them illegal

meanwhile "AI" is used to produce infinity+1 pictures of shrimp jesus and more spam than we've ever known before

and if we're really lucky, it will put us all out of work

Re: Things we learned about LLMs in 2024

#393

Post by [deleted] »

antirez wrote: post #54

About "people still thinking LLMs are quite useless", I still believe that the problem is that most people are exposed to ChatGPT 4o that at this point for my use case (programming / design partner) is basically a useless toy. And I guess that in tech many folks try LLMs for the same use cases. Try Claude Sonnet 3.5 (not Haiku!) and tell me if, while still flawed, is not helpful. But there is more: a key thing with L…

[deleted]

Re: Things we learned about LLMs in 2024

#394

Post by sarchertech »

cj wrote: post #380
duggan wrote: post #309

Earlier quoted context omitted.

antirez is clearly going to be “Staff-plus” for almost any definition. Can you clarify what you mean?

(Not original commenter) “Staff” engineer is typically one of the most senior and highest paid engineer titles in very large tech company. “Staff plus” is implying they are the best of the best.

Staff plus just means staff or higher. Staff, senior staff, principal, mega ultra principal etc…

Re: Things we learned about LLMs in 2024

#395

Post by YeGoblynQueenne »

antirez wrote: post #348
theptip wrote: post #291

Earlier quoted context omitted.

The context here is super-important - the commenter is the author of Redis. So, a super-experienced and productive low-level programmer. It’s not surprising that Staff-plus experts find LLMs much less useful. Though I’d be interested if this was an opinion on “help me write this gnarly C algorithm” or “help me to be productive in ” as I find a big productivity increase from the latter.

Quick example. I was implementing dot product between two quantized vectors that have two different min/max quantization ranges (later I changed the implementation to just centered range quantization, thanks to Claude and what I'm writing in this comment). I wanted to still have the math with the integers and adjust for the ranges at the end. Claude was able to mathematically scompose the operations as multiplication…

>> scompose the operations

I wonder whether that is some specialised terminology I'm not familiar with - or it just means to decompose the operations (but with an Italian s- for negation)?

Re: Things we learned about LLMs in 2024

#396

Post by Tostino »

dartos wrote: post #274
simonw wrote: post #251

Earlier quoted context omitted.

Yeah, a key thing to understand about LLMs is that managing the context is everything . You need to know when to wipe the slate by starting a new chat session and then pasting across a subset of the previous conversation. A lot of my most complex LLM interactions take place across multiple sessions - and in some cases I'll even move the project from Claude 3.5 Sonnet to OpenAI o1 (or vice versa) to help get out of a…

What kinds of things do you with these LLMs? I feel like I’m good at understanding context. I’ve been working in AI startups over the last 2 years. Currently at an AI search startup. Managing context for info retrieval is the name of the game. But for my personal use as a developer, they’ve caused me much headache. Answers that are subtly wrong in such a way that it took me a week to realize my initial assumption bas…

Not OP, but I've just gotten really used to verifying implementation details. Yup, those subtle ones really suck. It's pretty much just up to intuition if something in the response (or your followups) rings the `not quite right` bell for you.

Re: Things we learned about LLMs in 2024

#397

Post by Legend2440 »

JCM9 wrote: post #352

Nice overview. The challenge ahead for “AI” companies is that it appears there’s really no technical moat here. Someone comes out with something amazing and new and within months (if not weeks or days) it’s quickly copied. That environment where everything quickly becomes a commodity is a recipe for many/most companies in this space to quickly get washed out as it becomes economically unviable to play in such an envi…

That may be a challenge for AI companies but that doesn't sound like a problem to me. Commodities are great for consumers.

Re: Things we learned about LLMs in 2024

#398

Post by Kiro »

henning wrote: post #124
simonw wrote: post #50

Earlier quoted context omitted.

Did you try learning HOW to get good code out of them? As with all things LLM there's a whole lot of undocumented and under appreciated depth to getting decent results. Code hallucinations are also the least damaging type of hallucinations, because you get fact checking for free: if you run the code and get an error you know there's a problem. A lot of the time I find pasting that error message back into the LLM gets…

Like all AI simps, your blanket response to pointing out flaws is to tell me to do more prompt engineering and then dismiss the issue entirely. In the time it takes me to coax the model to do the thing I was told it knows how to do, I could just do the task myself. Your examples of LLM code generation are simple, easy to specify, self-contained applications that are not representative of software you can actually bui…

Hilarious that you're trying to gaslight us into "recognizing" your own incorrect assumptions as facts. You've lost all credibility.

Re: Things we learned about LLMs in 2024

#399

Post by dartos »

simonw wrote: post #390
dartos wrote: post #274

Earlier quoted context omitted.

What kinds of things do you with these LLMs? I feel like I’m good at understanding context. I’ve been working in AI startups over the last 2 years. Currently at an AI search startup. Managing context for info retrieval is the name of the game. But for my personal use as a developer, they’ve caused me much headache. Answers that are subtly wrong in such a way that it took me a week to realize my initial assumption bas…

Here are a bunch of things I use LLMs for relating to code. - Running micro-benchmarks (using Python in Code Interpreter) - if I have a question about which of two approaches is faster I often use this pattern: https://simonwillison.net/2023/Apr/12/code-interpreter/ - Building small ad-hoc one-off tools. Many of the examples in https://simonwillison.net/2024/Oct/21/claude-artifacts/ fit that bill, and I have a bunch…

Thank you!

I’ll read through these and try again in the new year.

Re: Things we learned about LLMs in 2024

#400

Post by antirez »

YeGoblynQueenne wrote: post #395
antirez wrote: post #348

Earlier quoted context omitted.

Quick example. I was implementing dot product between two quantized vectors that have two different min/max quantization ranges (later I changed the implementation to just centered range quantization, thanks to Claude and what I'm writing in this comment). I wanted to still have the math with the integers and adjust for the ranges at the end. Claude was able to mathematically scompose the operations as multiplication…

>> scompose the operations I wonder whether that is some specialised terminology I'm not familiar with - or it just means to decompose the operations (but with an Italian s- for negation)?

Decompose indeed :)
Post reply on HN