Many in the AI field think the bigger-is-better approach is running out of road

I also heard this. I unfortunately forget which study it was but yes, their paper spoke of likely diminishing returns at least at around 400-500B parameters for current LLM's. The recent news of GPT-4 running on 8x 220B LLM's (which doesn't equal a 8*220B size) fits that range and it's also questionable how much further we can push LLM's further by introducing multiple models like this, because this too eventually introduces problems due to granularity and picking the right model if I understood this correctly (from an earlier Hacker News discussion). (sorry for altogether no sources lol)

Non paywall: https://archive.ph/XwWTi

Imo, it is true that the current architecture is hitting the limit. We need a breakthrough on the scale of the transistor to get past this problem. We know it is possible though. Every single human is proof that high performance AI can be run with less energy than a laptop. We just need a dedicated architecture for their working mechanisms the same way a transistor is the embodiment of 1 and 0.

Unfortunately, in terms of understanding intelligence and how they work, I don't think we have made any significant advance in the last few decades. Maybe with better tools at probing how the LLM work, we can get some new insights.

Data requirements are overstated - you can train on longer and longer sequences and I am pretty sure most organizations are still using the “show the model the data only once“ approach which is just wasteful.

Compute challenges are more real, but we are seeing for the first time huge amounts of global capital being allocated to solve specifically these problems, so I am curious what fruit that will bear in a few years.

I mean already the stuff that some of these low level people are doing is absolutely nuts. Tim Dettmers work training with only 4 bits means only 16 possible values per weight and still getting great results.

It seems consistent with the behavior we see when using GPT4 in chat mode. Every once in a while it will change its answer as it’s generating it, as though it’s switched which model it favors to produce the response. GPT3.5 doesn’t do that.