Ask HN: Why is it taken for granted that LLM models will keep improving?

Its not unfeasable in the future to have a box at home that you can ask a fairly complicated question, like "how do I build a flying car", and it will have the ability to

- tell you step by step instructions of what you need to order

- write and run code to simulate certain things

- analyze you work from video streams and provide feedback

- possibly even have a robotic arm with attachments that can do some work.

So if we have that much compute power already why can't we just configure it in the right way to match a human brain?

I'm not sure I totally buy that logic though, since I would think the architecture/efficiency of a brain is way different from a computer

Data

Compute

Algorithms

All three are just scratching the surface of what is possible.

Data: What has been scraped off the internet is just Compute: While 3nm processes are approaching an atomic limit (0.21nm for Si), there is still room to explore more densely packed transistors or other materials like Gallium Nitride or optical computing. Not only that but there is a lot of room in hardware architecture to allow more parallelism and 3-D stacked transistors.

Algorithms: The transformer and other attention mechanisms have several sub-optimal components to them like how arbitrary the Transformer is in terms of design decisions, and quadratic time complexity for attention. There also seems to be a large space of LLM augmentations like RLHF for instruction following and improvements in factuality and other mechanisms.

And these ideas are just from my own limited experience. So I think its fair to say that LLMs have plenty of room to improve.

No one has hit a model/dataset size where the curves break down, and they're fairly smooth. Usually simple models that accurately predict performance work pretty well nearby existing performance, so I expect trillion or 10-trillion parameter models to be on the same curve.

What we haven't seen yet (that I'm aware of) is whether the specializations to existing models (LoRa, RLHF, different attention methods, etc.) follow similar scaling laws, since most of the efforts have been focused on achieving similar performance on smaller/sparser models and not investing the large amounts of money into huge experiments. It will be interesting to see what Deepmind Gemini reveals.