Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things we learned about LLMs in 2024

simonwillison.net

Post reply on HN
Print view View original

491–500 of 615 posts

Re: Things we learned about LLMs in 2024

#491

Post by isoprophlex »

antirez wrote: post #54

About "people still thinking LLMs are quite useless", I still believe that the problem is that most people are exposed to ChatGPT 4o that at this point for my use case (programming / design partner) is basically a useless toy. And I guess that in tech many folks try LLMs for the same use cases. Try Claude Sonnet 3.5 (not Haiku!) and tell me if, while still flawed, is not helpful. But there is more: a key thing with L…

Super interesting that my experience mirrors exactly what you are writing... except for me finding Claude to be almost useless (often misunderstands me, gives answers that are plain wrong) and 4o to be a very helpful, if not somewhat dull, jack-of-all trades in helping me be a cruise control for the mind.

I could only ever really jam with 4o.

Makes me wonder if there's personal communication preferences at play here.

Re: Things we learned about LLMs in 2024

#492

Post by janstice »

fullstackchris wrote: post #176
k2xl wrote: post #38

Something not mentioned is AI generated music. Suno's development this year is impressive. Unclear what this will mean for music artists over next few years.

Very clear; I like buying music produced by people who play instruments.

I’m even happy to listen to generative music, so long as it’s orchestrated (haha) by musicians using musical taste to make musical decisions, rather than a pastiche of the worst derivative house you’ve ever heard by a rando with no intent.

Re: Things we learned about LLMs in 2024

#493

Post by carom »

simonw wrote: post #142
viccis wrote: post #95

I didn't realize "agent" designs were that ambiguously defined. Every AI engineer I've talked to uses it to mean a design that combines several separate LLM prompts (or even models) to solve problems in multiple stages.

I'll add that one to the list. Surprisingly it doesn't closely match most of the 211 definitions I've collected already! The closest in that collection is "A division of responsibilities between LLMs that results in some sort of flow?" - https://lite.datasette.io/?json=https://gist.github.com/simo...

I am surprised as well, as it only takes a few hundred lines of code to implement them. [1]

    Agents are an abstraction that creates well defined roles for an LLM or LLMs to act within.
It's like object oriented programming for prompts.

1. https://github.com/openai/swarm/tree/main

Re: Things we learned about LLMs in 2024

#494

Post by jahnu »

csomar wrote: post #453
mvdtnz wrote: post #408

Earlier quoted context omitted.

> I built and shipped a Swift app to the App Store, currently generating $10,200 in MRR, exclusively using LLMs. My experience is that people who claim they build worthwhile software "exclusively" using LLMs are lying. I don't know you and I don't know if you are lying, but I would be willing to bet my paycheck you are.

They are also usually selling another AI-wrapper. I don't know the parent poster either but if your LLM product is generating $10k/month, your moat is really weak and you'll probably shut the f* up because your only moat is obscurity. Why risk that?

We shouldn’t assume the app created the customer base anew or solves a novel problem. Maybe this one does, we don’t know. But, what if the app is just an app version of a existing website store?

As an example I could imagine a clothing brand wanting an app that customers can install instead of using their phone browser. $10k/month in that context isn’t as surprising or impressive.

Re: Things we learned about LLMs in 2024

#495

Post by manmal »

cj wrote: post #380
duggan wrote: post #309

Earlier quoted context omitted.

antirez is clearly going to be “Staff-plus” for almost any definition. Can you clarify what you mean?

(Not original commenter) “Staff” engineer is typically one of the most senior and highest paid engineer titles in very large tech company. “Staff plus” is implying they are the best of the best.

I’ve seen your comment below, but you did specify big tech as context in this parent comment, no? Or is „very large tech company“ not FAANG?

Google has Staff at L6, and their ladder goes up to L11. Apple‘s Staff pendant is ICT5, which is below ICT6 and Distinguished. Amazon has E7-E9 above Staff, if you count E6 as Staff. Netflix very recently departed from their flat hierarchy and even they have Principal above Staff.

Re: Things we learned about LLMs in 2024

#496

Post by latexr »

nashadelic wrote: post #469
beoberha wrote: post #449

Earlier quoted context omitted.

LLMs being able to detect bugs in my own code is absolutely mind blowing to me. These things are “just” predicting the next token, but somehow are able to take in code that has never been written before and somehow understand it and find what’s wrong with it. I think I’m more amazed by them because I know how they work. They shouldn’t be able to do this, but the fact that they can is absolutely jaw dropping science f…

I actually get annoyed when experienced folks say this isn't AGI, its next word predict and not human-like intelligence. But we don't know how human intelligence works. Is it also just a matrix of neuron weights? Maybe it ends up looking like humans are also just next-word/thought predictors. Maybe that is what AGI will be.

> I actually get annoyed when experienced folks say this isn't AGI, its next word predict and not human-like intelligence. But we don't know how human intelligence works.

I’m pretty sure you’re committing a logical fallacy there. Like someone in antiquity claiming “I get annoyed when experienced folks say thunderstorms aren’t the gods getting angry, it’s nature and physical phenomena. But we don’t know how the weather works”. Your lack of understanding in one area does not give you the authority to make a claim in another.

Re: Things we learned about LLMs in 2024

#497

Post by weMadeThat »

antirez wrote: post #54

About "people still thinking LLMs are quite useless", I still believe that the problem is that most people are exposed to ChatGPT 4o that at this point for my use case (programming / design partner) is basically a useless toy. And I guess that in tech many folks try LLMs for the same use cases. Try Claude Sonnet 3.5 (not Haiku!) and tell me if, while still flawed, is not helpful. But there is more: a key thing with L…

> They work great to explore what is at the borders of your knowledge.

But not at exploring what is at the border of knowledge itself. And by converging on the conventional, LLMs actually lead you away from anything that actually extends.

> doing boring tasks for which you can provide perfect guidance

That's true but you never need an LLM for that. There are wonderful scripts written by wonderful people and provided for free almost all the time and for those who search in the right places. LLM companies benefit/profit of these without providing anything in return.

They are worse than people who grab FOSS and turn it into overpriced and aggressively marketed business models and services or people who threaten and sue FOSS for being better and free alternatives to their bloated and often "illegally telemetric" services.

> able to accelerate you

True, but you leave too much for data brokers and companies like Meta to abuse and exploit in the future. All that additional "interactional data" will do so much worse to humanity than all those previous data sets did in elections, for example, or pretty much all consumer markets. They will mostly accelerate all these dimwitted Fortune 5000 companies that have sabotaged consumers into way too much dumb shit - way more than is reasonable or "ok". And educated, wealthy and or tech-savvy people won't be able to avoid/evade any of that. Especially when it's paired with meds, drugs, foods, biases, fallacies, priming and so on and all the knowledge we will gain on bio-chemical pathways and human liability to sabotage.

They are great for coders, of course, everyone can be an army of clone-warriors with auto-complete on steroids now and nobody can tell you what to do with all that time that you now have and all that money, which, thanks to all of us but mostly our ancestors, is the default. The problem is the resulting hyper-amplified, augmented financial imbalance. It's gonna fuck our species if all the technical people don't restore some of that balance, and everybody knows what that means and what must be done.

Re: Things we learned about LLMs in 2024

#498

Post by andrewaylett »

antirez wrote: post #63
wslh wrote: post #57

Earlier quoted context omitted.

Right, in simpler terms: The measure of LLMs success is how effectively they help you achieve your goal faster.

Exactly, and right now the LLMs acceleration effect is a tool , not "give me the final solution". Even people that can't code, using LLMs to build applications from scratch, still have this tool mindset. This is why they can use them effectively: they don't stop at the first failed solution; they provide hints to the LLM, test the code, try to figure what's the problem (also with the LLM help), and so forth. It's a m…

> people that can't code

These people may not be Software Engineers, but they are coding.

Re: Things we learned about LLMs in 2024

#499

Post by EagnaIonat »

mewpmewp2 wrote: post #489
miki123211 wrote: post #484

Earlier quoted context omitted.

> But there is more: a key thing with LLMs is that their ability to help, as a tool, changes vastly based on your communication ability. The prompt is the king to make those models 10x better than they are with the lazy one-liner question. People keep saying this, and there are use cases for which this is definitely the case, but I find the opposite to be just as true in some circumstances. I'm surprised at how good…

Using the correct keywords like you did is part of communication though. Good communication with LLMs is the least keywords used to make it deducible for LLM what you exactly want.

> Good communication with LLMs is the least keywords used to make it deducible for LLM what you exactly want.

I am not sure that is the case, at least with a large number of LLMs. CO-STAR and TIDD-EC are much about structure and explanation than brevity.

Re: Things we learned about LLMs in 2024

#500

Post by danielbln »

raincole wrote: post #432
tokai wrote: post #320

Earlier quoted context omitted.

This reads like satire to me. Scarry that it isn't.

I too read it as satire at first, but after thinking twice I think it's a quite reasonable take. I've added "utilize LLM more in my daily life outside programming" to my new year resolution.

I had the flu at the beginning of December, with high fever, the whole nine yards. Keeping a running log with Claude in which I shared temperature readings, medications etc. has been so useful. If nothing else it's the world's most sophisticated rubber duck / secretary, but that's quite useful in many daily life situations on its own. Caveats apply etc.
Post reply on HN