Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things we learned about LLMs in 2024

simonwillison.net

Post reply on HN
Print view View original

451–460 of 615 posts

Re: Things we learned about LLMs in 2024

#451

Post by FooBarWidget »

jiggawatts wrote: post #422
fragmede wrote: post #413

Earlier quoted context omitted.

To be fair though, humanity doesn't know how some medicines work at a fundamental level either. The method of action for Tylenol, lithium, and metformin, among others isn't fully understood.

True, but modern "western"[1] medicine is not about the specific chemicals used, or even knowing how they exactly work at a chemical level, but the process for identifying what does and what does not work. It's an "evidence based" science with with experiments designed to counter known biases such as the placebo effect. Much of what we consider modern medicine was developed before we were entirely sure that atoms act…

"Western medicine" (which is exactly what it is called in China, to contrast with TCM) is shorthand for "practices invented in the west". That these methods chase universal truths, or are practiced world-wide, do not make them "non-west" in terms of origin.

The Chinese call the practice of truth seeking, in a more broader sense (outside of medicine) just "science".

"Western" medicine is also not merely the practice of seeking universal medical truth. It is also a collection of paradigms that have been developed in its long history. Like all paradigms, there are limits and drawbacks: phenomena that do not fit well. Truth seeking tends to be done on established paradigms rather than completely new ones.

The "western" prefix is helpful in contrasting it with TCM, which has a completely different paradigm. Many Chinese, myself included, have the experience that there are all sorts of ailments that are not meaningfully solved by "western" medicine practitioners, but are meaningfully solved by TCM practitioners.

Re: Things we learned about LLMs in 2024

#452

Post by hanikesn »

airstrike wrote: post #101
antirez wrote: post #75

Earlier quoted context omitted.

Probably. But statistically to work with 4o is a lose of time for me. LLMs is like an investment: you write the prompts, you "work" with them. If the LLM is too weak, this is a lose of time. You need to have a return on the investment that is positive. With ChatGPT 4o / o1 most of the times for me the investment of time has almost zero return. Before Claude Sonnet 3.5 I already had a ChatGPT PRO account but never use…

This mirrors my experience 100%. I'm not even sure why I still pay for OpenAI at this point. Claude 3.5 is just incredibly superior. And I totally agree on the point about dropping in context and asking very specific questions. I've had Claude pinpoint a bug in a 2k LOC module that I was struggling to find the cause for. After wasting a lot of time on it on my own, I thought "what the heck, maybe Claude can figure it…

I'm curious. Can you go into more detail what kind of bug it found?

Re: Things we learned about LLMs in 2024

#453

Post by csomar »

mvdtnz wrote: post #408
mvkel wrote: post #235

Earlier quoted context omitted.

I'm surprised at the description that it's "useless" as a programming / design partner. Even if it doesn't make "elegant" code (whatever that means), it's the difference between an app existing at all, or not. I built and shipped a Swift app to the App Store, currently generating $10,200 in MRR, exclusively using LLMs. I wouldn't describe myself as a programmer, and didn't plan to ever build an app, mostly because in…

> I built and shipped a Swift app to the App Store, currently generating $10,200 in MRR, exclusively using LLMs. My experience is that people who claim they build worthwhile software "exclusively" using LLMs are lying. I don't know you and I don't know if you are lying, but I would be willing to bet my paycheck you are.

They are also usually selling another AI-wrapper. I don't know the parent poster either but if your LLM product is generating $10k/month, your moat is really weak and you'll probably shut the f* up because your only moat is obscurity. Why risk that?

Re: Things we learned about LLMs in 2024

#454

Post by Terr_ »

Teever wrote: post #319
whatshisface wrote: post #82

Earlier quoted context omitted.

This belief comes from confusing the singularity (every atom on Earth is converted into a giant image of Sam Altman) with AGI (a store employee navigates a confrontation with an unruly customer, then goes home and wins at Super Mario).

But what if that AGI can fit inside a humanoid robot and that robot is capable of self replication even if it means digging the sand out of the ground to make silicon with a spade?

We already have humanoid intelligeces that self-assemble and power from common materials, as a colony of incredibly advanced nanobots.

Re: Things we learned about LLMs in 2024

#455

Post by csomar »

duped wrote: post #264
ChicagoDave wrote: post #233

Earlier quoted context omitted.

Claude Sonnet 3.5 can write whole React applications with proper contextual clues and some minor iterations. Google has never coded for you. I’ve written two large applications and about a dozen smaller ones using Claude as an assistant. I’m a terrible front-end developer and almost none of that work was possible without Claude. The API and AWS deployment were sped up tremendously. I’ve created unit tests and I’ve re…

I've never really used Claude for writing code, becuase I'm not really bottlenecked by that problem. I have used it quite a bit for asking questions about what code to write and it's almost always wrong (usually in subtle ways that would trick someone with little experience). Maybe it was overtrained on react sources, but for me it's pretty useless. The big annoyance for me is it just makes up APIs that don't exist.…

They are mostly useful for front-end/React because front-end shouldn't been code in the first place. They can do the UX but not the state management. Honestly, as someone who sucks and dread UX building (and having to frequently adjust my divs/components), they are a life saver when you are doing very conventional things. That is things you can find 100s of examples of but will take you hours to glue together.

Re: Things we learned about LLMs in 2024

#456

Post by Terr_ »

datadrivenangel wrote: post #200
fullstackchris wrote: post #174

Earlier quoted context omitted.

Not OP but yes. Electron size vs band gap, computing costs (in terms of electricity) any other raw materials needed for that energy, etc... sigh... its physics, always physics... what fundamental property of physics do you think would let a vertical take off in intelligence occur?

If you look at the rate of mathematical operations conducted, we're already going hard vertical. Physics and material limitations will slow that eventually as we reach a marginal return on converting the planet to computer chips, but we're in the singularity as proxy measured by mathematical operations.

> If you look at the rate of mathematical operations conducted, we're already going hard vertical.

Not if you remember to count all the computations being done by the quintillions of nanobots across the world known as "human cells."

That's not only inside cells, and not just neurons either. For example, your thyroid is busy brute-forcing the impossibly large space of antibody combinations, and putting every candidate cell-release through a very rigorous set of acceptance tests.

Re: Things we learned about LLMs in 2024

#457

Post by LtWorf »

ninth_ant wrote: post #258
rrix2 wrote: post #114

Earlier quoted context omitted.

> Every day by automation I feed in the hourly weather forecast my home ollama server and it builds me a nice readable concise weather report. not to dissuade you from a thing you find useful but are you aware that the national weather service produces an Area Forecast Discussion product in each local NWS office daily or more often that accomplishes this with human meteorologists and clickable jargon glossary? https:…

Doesn’t dissuade me at all, that’s a really neat service. I’m not American though, and even if my own country had a similar service I still enjoying tuning the results to focus on what I’m interested in. And it was just an example of the kinds of computer-human interfaces that are newly possible from this technology. Anytime you have data and want it explained in a casual way — and it’s not mission critical to be ext…

The best forecast available on the internet is norwegian.

Re: Things we learned about LLMs in 2024

#458

Post by LtWorf »

pixl97 wrote: post #255
ninth_ant wrote: post #99

Earlier quoted context omitted.

I think a lot of the confusion is in how we approach LLMs. Perhaps stemming from the over-broad term “AI”. There are certain classes of problems that LLMs are good at. Accurately regurgitating all accumulated world knowledge ever is not one, so don’t ask a language model to diagnose your medical condition or choose a political candidate. But do ask them to perform suitable tasks for a language model! Every day by aut…

>don’t ask a language model to diagnose your medical condition Honestly they are very decent at it if you give them accurate information in which to make the diagnosis. The typical problem people have is being unable to feed accurate information to the model. They'll cut out parts they don't want to think about or not put full test results in for consideration.

Yes so basically bias it into what you think it should reply in the question and it will magically somehow give the reply you wanted! Very useful :D

Re: Things we learned about LLMs in 2024

#459

Post by djhn »

bjt wrote: post #269
simonw wrote: post #231

Earlier quoted context omitted.

One of the things I find most frustrating about LLMs is how resistant they are to teaching other people how to use them! I'd love to figure this out. I've written more about them than most people at this point, and my goal has always been to help people learn what they can and cannot do - but distilling that down to a concise set of lessons continues to defeat me. The only way to really get to grips with them is to u…

Thank you for doing this work, though. My first stab at trying ChatGPT last year was asking it to write some Rust code to do audio processing. It was not a happy experience. I stepped back and didn't play with LLMs at all for a while after that. Reading your posts has helped me keep tabs on the state of the art and decide to jump back in (though with different/easier problems this time).

To be fair I think that is a hard task even for a human expert, in the sense that there isn’t much prior art.

Re: Things we learned about LLMs in 2024

#460

Post by madmask »

layer8 wrote: post #347
simonw wrote: post #308

Earlier quoted context omitted.

That was one of the themes of my article: LLMs are power-user tools, mis-sold as "easy to use". To get great results out of them you need to invest a whole lot of under-documented and under-appreciated effort. https://simonwillison.net/2024/Dec/31/llms-in-2024/#llms-som...

It’s not just that you need to be a power user (I certainly am), you also need to be fine with nondeterminism and typing a lot of prose, instead of doing everything with keyboard shortcuts and CLI commands, with reproducible outcomes. It’s a different mode of operation and interaction, requiring a different predisposition to some degree.

Exactly! I don’t like talking or writing or explaining.

My mind generally uses language as little as possible, I have no inner monologue running in the background.

Greatly prefer something deterministic to random bs popping up without the ability of recognizing it.

I don’t like llms but sometimes use them as autocomplete or to generate words, like a template for a letter or boilerplate scripts, never for actual information (à la google).

Post reply on HN