Ask HN: Why is it taken for granted that LLM models will keep improving?

It appears as if improving data corpus quality and size and improving processing capacity are still driving performance gains. I have no idea of the functional relationship, and its likely not a Moore's law kind of thing, although that would be an underlying driver of available capacity to saturation.

> it won’t just plateau in performance at some point?

I suspect that we've already seen the shape of the curve: a 1B parameter model can index a book; a 4B model can converse, but a 14B model can be a little more eloquent. Beyond that no real gains will be seen.

The "technology advancement" phase has already happened mostly, but the greater understanding of theory, that would discourage foolish investments hasn't propagated yet. So there's probably at least another full year of hype cycle before the next buzzword is brought out to start hoovering up excess investment funds.

Because a lot of smart people are spending a lot of time, money, and effort on this. It's as simple as that. We could go into all sorts of details, like how increase in GPU capabilities will improve training capabilities, both in size and speed, or how GPU(/TPU) capabilities will improve, or how better techniques will make training on the same data set result in better models, or where other improvements will make better use of existing models or make them better or where we're seeing additions to training data sets and how that will improve models using existing techniques. But it really all boils down to a lot of smart people, some with a lot of money, that are personally invested (with time and money) in making them better.

That doesn't mean there isn't possibly a plateau somewhere but it's somewhere way off in the distance.

I think I can do it far, far cheaper than that... and I've been talking about it, in public, for over a decade.[1] I could easily be wrong, of course. It really all depends on how much power a 4x4 LUT and 4 bit latch Latch Leak, and how much energy it takes to clock data through them for a cycle, and how fast they can be cycled. If the number are good, this thing will be amazingly cool. I can't find good numbers anywhere.

If you've made chips with latches and LUTS, any performance data you can share, no matter how old, would be helpful

It's an idea that's been bouncing around in my head since reading George Gilder's call to waste transistors. Imagine the worst possible FPGA, no routing hardware, and slow it down even more with a latch on every single LUT. Optimize it slightly, by making cells with 4 bits in, 4 bits out (64 bits of programming per cell), with the cells clocked in 2 phases, like the colors of a chess board. This means that each white cell has static inputs from the black cells.... and is thus fully deterministic, and easy to reason about. The complement happens on the other phase. Together, it becomes turning complete.

The thing is, it does computing with NO delays between compute and memory. All the memory is effectively transferred to compute every clock cycle. The latency sucks because you'll take n/2 cycles to get data across an N*N grid. However, you'll get an answer every clock cycle after that.

Imagine a million GPT-4 tokens/second.... not related to each other, of course, but parallel streams, interleaved as the data streams across the chips.

Imagine a bad cell.... you can detect it, and route around it. Yields don't have to be 100%.

The extreme downside is that tools for programming this thing don't exist. VHDL, etc... aren't appropriate. I'm going to have to build them. I've been stuck in analysis paralysis, but I've decided to try to get Advent of Code done using my bitgrid simulator. I hope to be done before it starts again next December. ;-)

[1] https://bitgrid.blogspot.com/

[2] https://github.com/mikewarot/Bitgrid

It will plateau at best. But crowd is never smart, yet can scream.