Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things we learned about LLMs in 2024

simonwillison.net

Post reply on HN
Print view View original

221–230 of 615 posts

Re: Things we learned about LLMs in 2024

#221

Post by zekica »

dartos wrote: post #191
jacobgkau wrote: post #173

Earlier quoted context omitted.

Doesn't it hold true for investment in AI (or potentially any other industry that experiences a boom) in general? Anyone who bought in at the ground floor is now rich. Anyone who buys in now is incentivized to try and keep getting more people to buy in so their investment will give a return regardless of if actual value is being created.

If effect, kind of. The money being invested does not go directly to investors. It goes to the cost of R&D, which in turn increases the value of openai shares, then the early investors can sell those shares to realize those gains. The difference between that and a ponzi is that the investment creates value which is reflected in the share price. No value is created in a Ponzi scheme. The actual dollar worth of the val…

Only a part of the value is created in OpenAI's stock valuation. Most of it is still a ponzi-like scheme.

Re: Things we learned about LLMs in 2024

#222

Post by dekhn »

karolist wrote: post #186
minimaxir wrote: post #96

Earlier quoted context omitted.

LLMs run on the GPU, and the unified memory of Apple silicon means that the 64 GB can be used by the GPU. Consumer GPUs top out at 24 GB VRAM.

llama.cpp can run LLMs on CPU. iGPU can also use system memory, the novel thing is not that, it's that the LLM inference is mostly memory bandwidth bound and memory bandwidth of a custom built PC with really fast DDR5 RAM is around 100GB/s, nVidia consumer GPUs at the top end are around 1TB/s, with mid range GPUs at around half that. M1 Max has 400GB/s, M1 Ultra is 800GB/s, but you can have Apple Silicon Macs with up…

It'd be really nice to have good memory bandwidth usage metrics collected from a wide range of devices while doing inference.

For example, how close does it get to the peak, and what's the median bandwidth during inference? And is that bandwidth, rather than some other clever optimization elsewhere, actually providing the Mac's performance?

Personally, I don't develop HPC stuff on a laptop - I am much more interested in what a modern PC with Intel or AMD and nvidia can do, when maxxed out. But it's certainly interesting to see that some of Apple's arch decisions have worked out well for local LLMs.

Re: Things we learned about LLMs in 2024

#223

Post by th0ma5 »

More dishonest magical thinking. I wish this guy would learn how systems work and stop flooding the field with mystical nonsense unless he really is trying to make people think LLMs are worthless, then I guess he should be honest about it instead of subversive.

Re: Things we learned about LLMs in 2024

#224

Post by orsenthil »

One of the best written summary of LLMs for the year 2024.

We all have silently started to realize Slops, hopefully we can recognize them more easily and prevent them.

Test Driven Development (Integration Tests or functional tests specifically) for Prompt Driven Development seems like the way to go.

Thank you, Simon.

Re: Things we learned about LLMs in 2024

#225

Post by simonw »

th0ma5 wrote: post #223

More dishonest magical thinking. I wish this guy would learn how systems work and stop flooding the field with mystical nonsense unless he really is trying to make people think LLMs are worthless, then I guess he should be honest about it instead of subversive.

Which bit was dishonest magical thinking?

In case you're interested, here's a summarized list (thanks, Claude) of the negative/critical things I said about LLMs and the companies that build them in this post: https://gist.github.com/simonw/73f47184879de4c39469fe38dbf35...

Re: Things we learned about LLMs in 2024

#226

Post by agentultra »

simonw wrote: post #143
uludag wrote: post #98

Earlier quoted context omitted.

But according to the author, apparently bringing this up isn't helpful criticism. I'm curious what peoples thoughts are of what the future of LLMs would be like if we severely overshoot our carbon goals. How bad would thinks have to get for people to stop caring about this technology?

It's helpful criticism as part of the conversation. What frustrates me is when people go "LLMs are burning the planet!" and leave it at that.

It is a rather contrasting opinion that the trade-offs to have AI aren’t worth the value they bring.

The growth in this technology isn’t outpacing car pollution and O&G extraction… yet, but the growth rate has been enough in recent years to put it on the radar of industries to watch out for.

I hope the compute efficiencies are rapid and more than commensurate with the rate of growth so that we can make progress on our climate targets.

However it seems unlikely to me.

It’s been a year of progress for the tech… but also a lot of setbacks for the rest of the world. I’m fairly certain we don’t need AGI to tell us how to cope with the climate crisis; we already have the answer for that.

Although if the industry does continue to grow and the efficiency gains aren’t enough… will society/investors be willing to scale back growth in order to meet climate targets (assuming that AI becomes a large enough segment of global emissions to warrant reductions)?

Interesting times for the field.

Re: Things we learned about LLMs in 2024

#227

Post by DavidSJ »

cactusfrog wrote: post #145
nntwozz wrote: post #65

I think John Gruber summed it up nicely: https://daringfireball.net/2024/12/openai_unimaginable OpenAI’s board now stating “We once again need to raise more capital than we’d imagined” less than three months after raising another $6.6 billion at a valuation of $157 billion sounds alarmingly like a Ponzi scheme — an argument akin to “Trust us, we can maintain our lead, and all it will take is a never-ending stream of…

Every waste of money is not a Ponzi scheme.

> Every waste of money is not a Ponzi scheme.

Using this as an opportunity to grind an axe (not your fault, cactusfrog!): I find it clearer when people write "not every X is a Y" than "every X is not a Y", which could be (and would be, literally) interpreted to mean the same thing as "no X is a Y".

Re: Things we learned about LLMs in 2024

#228

Post by th0ma5 »

henning wrote: post #37

Spookily good at writing code? LLMs frequently hallucinate broken nonsense shit when I use them. Recognize what they do well (generate simple code in popular languages) while acknowledging where they are weak (non-trivial algorithms, any novel code situation the LLM hasn't seen before, less popular languages).

Simon gets one thing working for one task and assumes everyone can do the same for everything. That's the trick is that he has no idea how the failures happen or how to maintain actual working systems.

Re: Things we learned about LLMs in 2024

#230

Post by refulgentis »

lowercased wrote: post #219
datadrivenangel wrote: post #122

Earlier quoted context omitted.

Which models?

hugging-quants/llama-3.2-1b-instruct-q8_0-gguf - 100-150 tok/sec second-state/llama-2-7b-chat-gguf net me around ~35 tok/sec lmstudio-community/granite-3.1.-8b-instruct-GGUF - ~50 tok/sec MBP M3 Max, 64g. - $3k

I'm not sure if you're pointing out any / all of these:

#1. It is possible to get an arbitrarily fast tokens/second number, given you can pick model size.

#2. Llama 1B is roughly GPT-4.

#3. Given Llama 1B runs at 100 tokens/sec, and given performance at a given model size has continued to improve over the past 2 years, we can assume there will eventually be a GPT-4 quality model at 1B.

On my end:

#1. Agreed.

#2. Vehemently disagree.

#3. TL;DR: I don't expect that, at least, the trend line isn't steep enough for me to expect that in the next decade.

Post reply on HN