Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things we learned about LLMs in 2024

simonwillison.net

Post reply on HN
Print view View original

161–170 of 615 posts

Re: Things we learned about LLMs in 2024

#161

Post by simonw »

lolinder wrote: post #141
simonw wrote: post #112

Earlier quoted context omitted.

Yeah, the other skill you need to develop to make the most of AI-assisted programming is really good manual QA.

Have you found that to be a good trade-off for large-scale projects? Where I'm at right now with LLMs is that I find them to be very helpful for greenfield personal projects. Eliminating the blank canvas problem is huge for my productivity on side projects, and they excel at getting projects scaffolded and off the ground. But as one of the lead engineers working on a million+ line, 10+ year-old codebase, I've yet to…

I use them successfully on larger project all the time.

"Here's some example JavaScript code that sends an email through the SendGrid REST API. Write me a python function for sending an email that accepts an email address, subject, path to a Jinja template and a dictionary of template context. It should return true or false for if the email was sent without errors, and log any error messages to stderr"

That prompt is equally effective for a project that's 500 lines or 5,000,000 lines of code.

I also use them for code spelunking - you can pipe quite a lot of code into Gemini and ask questions like "which modules handle incoming API request validation?" - that's why I built https://github.com/simonw/files-to-prompt

Re: Things we learned about LLMs in 2024

#162

Post by phantompeace »

henning wrote: post #124
simonw wrote: post #50

Earlier quoted context omitted.

Did you try learning HOW to get good code out of them? As with all things LLM there's a whole lot of undocumented and under appreciated depth to getting decent results. Code hallucinations are also the least damaging type of hallucinations, because you get fact checking for free: if you run the code and get an error you know there's a problem. A lot of the time I find pasting that error message back into the LLM gets…

Like all AI simps, your blanket response to pointing out flaws is to tell me to do more prompt engineering and then dismiss the issue entirely. In the time it takes me to coax the model to do the thing I was told it knows how to do, I could just do the task myself. Your examples of LLM code generation are simple, easy to specify, self-contained applications that are not representative of software you can actually bui…

Like all stubborn anti-AI know-it-alls, you sound like you’ve tried a couple of times to do something and have decided to label all LLMs with the same brush.

What models have you tried, and what are you trying to do with them? Give us an example prompt too so we can see how you’re coaxing it so we can rule out skill issue.

And a big strength LLMs have is summarizing things - I’d like to see you summarize the latest 10 arxiv papers relating to prompt engineering and produce a report geared towards non-techies. And do this every 30 mins please. Also produce social media threads with that info. Is this a task you could do yourself, better than LLMs?

Re: Things we learned about LLMs in 2024

#163

Post by Jensson »

api wrote: post #148
com2kid wrote: post #104

Earlier quoted context omitted.

There are some really good books about wars between cultures that have AGI and it always comes down to math - whoever can get their hands on more compute faster wins.

This is also a strong argument for immigration, particularly high-skill immigration. In the absence of synthetic AGI whoever imports the most human AGI wins.

Which suggests that total AGI compute doesn't matter that much, as India isn't the world leader the amount of human compute they posses would suggest then.

What matters is how you use the AGI, not how much you have, with wrong or bad or limiting regulations it will not lead anywhere.

Re: Things we learned about LLMs in 2024

#164

Post by mooreds »

dxbydt wrote: post #92
antirez wrote: post #54

About "people still thinking LLMs are quite useless", I still believe that the problem is that most people are exposed to ChatGPT 4o that at this point for my use case (programming / design partner) is basically a useless toy. And I guess that in tech many folks try LLMs for the same use cases. Try Claude Sonnet 3.5 (not Haiku!) and tell me if, while still flawed, is not helpful. But there is more: a key thing with L…

> best LLMs are able to accelerate you https://www2.math.upenn.edu/~ghrist/preprints/LAEF.pdf - this math textbook was written in just 55 days! Paraphrasing the acknowledgements - ...Begun November 4, 2024, published December 28, 2024. ...assisted by Claude 3.5 sonnet, trained on my previous books... ...puzzles co-created by the author and Claude ...GPT-4o and -o1 were useful in latex configurations...doing proof-rea…

I started a book about CIAM (customer identity and access management) using Claude to help outline a chapter. I'd edit and refine the outline to make sure it covered everything.

Then I'd have Claude create text. I'd then edit/refine each chapter's text.

Wow, was it unpleasant. It was kinda cool to see all the words put together, but editing the output was a slog.

It's bad enough editing your own writing, but for some reason this was even worse.

Re: Things we learned about LLMs in 2024

#165

Post by Peacefulz »

scary-size wrote: post #138
lumost wrote: post #119

Earlier quoted context omitted.

They are still in the lead, and I'd be willing to bet that they have 10x the DAU on chat.com/chatgpt.com than all other providers combined. Barring massive innovation on small sub 10B models - we are all likely to need remote inference from large server farms for the foreseeable future. Even in the case that local inference is possible - it's unlikely it will be desirable from a power perspective in the next 3 years.…

Not sure about 10x DAUs. Google flicked the switch on Gemini and it surfaced in pretty much every GSuite app over night.

Requiring that Gemini take over the job that Google Assistant did when installing the Gemini APK really rubbed me the wrong way. I get it. I just don't like that it was required for use.

Re: Things we learned about LLMs in 2024

#166

Post by fullstackchris »

whatshisface wrote: post #82
philipkglass wrote: post #44

Earlier quoted context omitted.

If AGI is possible, then the entire human economy stops making sense as far as money goes, and 'owning' part of OpenAI gives you power. That's if AGI is possible and not easily replicated . If AGI can be copied and/or re-developed like other software then the value of owning OpenAI stock is more like owning stock in copper producers or other commodity sector companies. (It might even be a poorer investment. Even AGI…

This belief comes from confusing the singularity (every atom on Earth is converted into a giant image of Sam Altman) with AGI (a store employee navigates a confrontation with an unruly customer, then goes home and wins at Super Mario).

Exactly. I continually fail to see how "the entire human economy ends" overnight with another human like agent out there - especially if its confined to a server in the first place - it can't even "go home" :)

Re: Things we learned about LLMs in 2024

#167

Post by belter »

hdjjhhvvhga wrote: post #72
nntwozz wrote: post #65

I think John Gruber summed it up nicely: https://daringfireball.net/2024/12/openai_unimaginable OpenAI’s board now stating “We once again need to raise more capital than we’d imagined” less than three months after raising another $6.6 billion at a valuation of $157 billion sounds alarmingly like a Ponzi scheme — an argument akin to “Trust us, we can maintain our lead, and all it will take is a never-ending stream of…

What is funny is that their "lead" is just because of inertia - they were the first to make an LLM publicly available. But they are no longer leaders so their attempts at getting more and more money only prove Altman's skills at convincing people to give him money.

Their best hope now is to hire John Carmack :-)

Re: Things we learned about LLMs in 2024

#168

Post by brokencode »

scary-size wrote: post #138
lumost wrote: post #119

Earlier quoted context omitted.

They are still in the lead, and I'd be willing to bet that they have 10x the DAU on chat.com/chatgpt.com than all other providers combined. Barring massive innovation on small sub 10B models - we are all likely to need remote inference from large server farms for the foreseeable future. Even in the case that local inference is possible - it's unlikely it will be desirable from a power perspective in the next 3 years.…

Not sure about 10x DAUs. Google flicked the switch on Gemini and it surfaced in pretty much every GSuite app over night.

Same with Microsoft and all their Copilots, which are built on OpenAI. Not to mention all the other companies using OpenAI since it’s still the best.

Re: Things we learned about LLMs in 2024

#169

Post by aetherson »

jsheard wrote: post #151
antirez wrote: post #54

About "people still thinking LLMs are quite useless", I still believe that the problem is that most people are exposed to ChatGPT 4o that at this point for my use case (programming / design partner) is basically a useless toy. And I guess that in tech many folks try LLMs for the same use cases. Try Claude Sonnet 3.5 (not Haiku!) and tell me if, while still flawed, is not helpful. But there is more: a key thing with L…

I swear these goalposts keep getting moved, I remember being told that GPT3.5 is a useless toy but the paid GPT4 is lifechanging, and now that GPT4 is free I'm told that it's a useless toy but paid o1 or paid Sonnet are lifechanging. Looking forward to o1 and Sonnet becoming useless toys, unlike the lifechanging o3.

You will also be dismayed to hear that a 2011 iPhone is no longer state-of-the-art, and indeed can't run most modern apps.

Re: Things we learned about LLMs in 2024

#170

Post by xnx »

viccis wrote: post #95

I didn't realize "agent" designs were that ambiguously defined. Every AI engineer I've talked to uses it to mean a design that combines several separate LLM prompts (or even models) to solve problems in multiple stages.

This sounds like ensemble chain of thought.
Post reply on HN