Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things we learned about LLMs in 2024

simonwillison.net

Post reply on HN
Print view View original

531–540 of 615 posts

Re: Things we learned about LLMs in 2024

#531

Post by vasco »

antirez wrote: post #54

About "people still thinking LLMs are quite useless", I still believe that the problem is that most people are exposed to ChatGPT 4o that at this point for my use case (programming / design partner) is basically a useless toy. And I guess that in tech many folks try LLMs for the same use cases. Try Claude Sonnet 3.5 (not Haiku!) and tell me if, while still flawed, is not helpful. But there is more: a key thing with L…

Most people consider their own brain useless and don't use it, so it's not strange that they do the same with AI. How many people just refuse to learn how to parallel park, a new language, calculus or even basic arithmetic, "because they aren't good at it".

Re: Things we learned about LLMs in 2024

#532

Post by lowercased »

refulgentis wrote: post #230
lowercased wrote: post #219

Earlier quoted context omitted.

hugging-quants/llama-3.2-1b-instruct-q8_0-gguf - 100-150 tok/sec second-state/llama-2-7b-chat-gguf net me around ~35 tok/sec lmstudio-community/granite-3.1.-8b-instruct-GGUF - ~50 tok/sec MBP M3 Max, 64g. - $3k

I'm not sure if you're pointing out any / all of these: #1. It is possible to get an arbitrarily fast tokens/second number, given you can pick model size. #2. Llama 1B is roughly GPT-4. #3. Given Llama 1B runs at 100 tokens/sec, and given performance at a given model size has continued to improve over the past 2 years, we can assume there will eventually be a GPT-4 quality model at 1B. On my end: #1. Agreed. #2. Vehe…

I specifically missed the GPT4 part of "up to 10 token/sec out of a kinda sorta GPT-4". Was just looking at token/sec.

Re: Things we learned about LLMs in 2024

#533

Post by mvkel »

archagon wrote: post #465
mvkel wrote: post #235

Earlier quoted context omitted.

I'm surprised at the description that it's "useless" as a programming / design partner. Even if it doesn't make "elegant" code (whatever that means), it's the difference between an app existing at all, or not. I built and shipped a Swift app to the App Store, currently generating $10,200 in MRR, exclusively using LLMs. I wouldn't describe myself as a programmer, and didn't plan to ever build an app, mostly because in…

Off topic, but I'm a bit confused. Your iOS apps as listed on your website are CarPrep and Brocly, neither of which appear to have notable review activity or buzz in the media. If the app you're referring to is one of these, the more interesting question (to me) is: how on Earth are you generating $10,200 MRR from it? Or is there another app that I'm missing? (In my experience as an app developer, getting any tractio…

Those are just my silly personal projects, not businesses. The business I mentioned above is in the recruiting agency space, B2B SaaS. The app itself is not the thing being purchased per se, the point was it was built using LLMs.

$10K MRR isn't much; we're still validating PMF. We're carefully selecting paid customers at this point, not open for wide release, hence my vagueness. Just wanted to illustrate that building robust apps that have value are possible today.

Re: Things we learned about LLMs in 2024

#534

Post by mvkel »

mvdtnz wrote: post #408
mvkel wrote: post #235

Earlier quoted context omitted.

I'm surprised at the description that it's "useless" as a programming / design partner. Even if it doesn't make "elegant" code (whatever that means), it's the difference between an app existing at all, or not. I built and shipped a Swift app to the App Store, currently generating $10,200 in MRR, exclusively using LLMs. I wouldn't describe myself as a programmer, and didn't plan to ever build an app, mostly because in…

> I built and shipped a Swift app to the App Store, currently generating $10,200 in MRR, exclusively using LLMs. My experience is that people who claim they build worthwhile software "exclusively" using LLMs are lying. I don't know you and I don't know if you are lying, but I would be willing to bet my paycheck you are.

I mean, I'm pretty upfront on my personal site that I've built successful companies in the past. Not sure why I would lie about this one, especially when I'm admitting that I'm not doing the work :)

See comment above for more context.

Re: Things we learned about LLMs in 2024

#535

Post by mvkel »

csomar wrote: post #453
mvdtnz wrote: post #408

Earlier quoted context omitted.

> I built and shipped a Swift app to the App Store, currently generating $10,200 in MRR, exclusively using LLMs. My experience is that people who claim they build worthwhile software "exclusively" using LLMs are lying. I don't know you and I don't know if you are lying, but I would be willing to bet my paycheck you are.

They are also usually selling another AI-wrapper. I don't know the parent poster either but if your LLM product is generating $10k/month, your moat is really weak and you'll probably shut the f* up because your only moat is obscurity. Why risk that?

I hope someday that people will understand that you can use AI to build "boring" non-AI apps.

Re: Things we learned about LLMs in 2024

#536

Post by mvkel »

oblio wrote: post #446
mvkel wrote: post #235

Earlier quoted context omitted.

I'm surprised at the description that it's "useless" as a programming / design partner. Even if it doesn't make "elegant" code (whatever that means), it's the difference between an app existing at all, or not. I built and shipped a Swift app to the App Store, currently generating $10,200 in MRR, exclusively using LLMs. I wouldn't describe myself as a programmer, and didn't plan to ever build an app, mostly because in…

> I built and shipped a Swift app to the App Store, currently generating $10,200 in MRR, exclusively using LLMs. That's great, but professional programmers are afraid of the future maintenance burden.

"maintenance burden" is introduced when a non-original programmer starts contributing to a repo, regardless of how objectively maintainable the code is.

Re: Things we learned about LLMs in 2024

#537

Post by airstrike »

hanikesn wrote: post #452
airstrike wrote: post #101

Earlier quoted context omitted.

This mirrors my experience 100%. I'm not even sure why I still pay for OpenAI at this point. Claude 3.5 is just incredibly superior. And I totally agree on the point about dropping in context and asking very specific questions. I've had Claude pinpoint a bug in a 2k LOC module that I was struggling to find the cause for. After wasting a lot of time on it on my own, I thought "what the heck, maybe Claude can figure it…

I'm curious. Can you go into more detail what kind of bug it found?

I was writing a custom widget for iced (the Rust GUI library) and I was getting a panic due to some fancy logic I was trying to do. I guess the shortest description I can say is that it was a combination of what appeared to be a caching issue at first, but the real cause turned out to be some method shadowing where I was using a struct's method where I meant to use the trait's method.

I had made the specific operation generic (moving it out of the struct and into a trait) but forgot to delete it from the struct, so I was calling the incorrect function. Claude pinpointed the cache issue immediately when I just dumped two files into the context and asked it:

    somewhere in my codebase I'm triggering a perform() on the editor but the next call on highlight() panics because `Line layout should be cached`

    what am I missing? do I need to do something after perform() to re-cache the layout?
at first that seemed to fix the issue, but other errors persisted. so we kept debugging together until we found the root cause. either way I knew where to look thanks to its assistance

Re: Things we learned about LLMs in 2024

#538

Post by codr7 »

paxys wrote: post #337
theptip wrote: post #294

Earlier quoted context omitted.

No, it’s reasonable. If your team uses Git then it’s a valid question to establish if someone has only worked with Perforce. They didn’t say how heavily they weight the question. (All that said I expect that, soon, experience with the appropriate LLM tooling will be as important as having experience with the language your system is implemented in.)

You hire people based on their fundamental knowledge and the ability to learn, not skills in arbitrary tools and frameworks which come and go every other day. If someone has used Perforce they will be able to get perfectly comfortable with Git by the end of their first week. So not knowing Git is an idiotic reason to reject a skilled developer. Same with programming languages, and just about every other aspect of sof…

Yeah, but it's oh so easy to test for, and oh so nice to have plenty of boxes checked to cover your ass if the hire goes wrong.

Re: Things we learned about LLMs in 2024

#539

Post by theanonymousone »

simonw wrote: post #518
theanonymousone wrote: post #479

Earlier quoted context omitted.

> There is still Gemini free tier which is ofc basically impossible to beat Is it free free? The last time I checked there was a daily request limit, still generous but limiting for some use cases. Isn't it still the case?

Providing an unlimited free tier would be a terrible business decision for them.

Of course. My point is, probably a super cheap LLM that does not cut you off after 1500th API request of the day is preferred over the free model that does so, at least for certain use cases.

Re: Things we learned about LLMs in 2024

#540

Post by ninth_ant »

pixl97 wrote: post #255
ninth_ant wrote: post #99

Earlier quoted context omitted.

I think a lot of the confusion is in how we approach LLMs. Perhaps stemming from the over-broad term “AI”. There are certain classes of problems that LLMs are good at. Accurately regurgitating all accumulated world knowledge ever is not one, so don’t ask a language model to diagnose your medical condition or choose a political candidate. But do ask them to perform suitable tasks for a language model! Every day by aut…

>don’t ask a language model to diagnose your medical condition Honestly they are very decent at it if you give them accurate information in which to make the diagnosis. The typical problem people have is being unable to feed accurate information to the model. They'll cut out parts they don't want to think about or not put full test results in for consideration.

If the LLM is trained on accurate medical data and you provide accurate symptoms data, then the LLM can be a useful tool to output the information in a human-readable way.

This is not a diagnosis. Any reasonably capable person can read webmd and apply the symptoms listed and compare them to what the patient describes. This is widely regarded as dangerous because the input data as well as the patient data are limited in ways that can be medically relevant.

So even if you can use it as a good substitute for browsing webmd, it’s still not a substitute for seeing a medical professional. And for the foreseeable future it will not be.

Post reply on HN