Amazing that you can just shove a ton of multimodal data into a big transformer and get a really good multimodal model. I wonder where things will top out. For many years a lot of people (including me) were saying "you can't just take existing architectures, scale them up, feed them a lot of data, and expect something mpressive", but here we are.
Now I’ve reached the opposite SWAG hypothesis: given a sufficiently general optimization problem, the more one scales training/inference compute, the more impossible it is to prevent intelligence.
That seems disappointingly prosaic. Shouldn’t there be more to it? Human ego creates anthropocentric bias toward thinking we’re special. We aren’t. At best we’re lucky. And nature doesn’t care about our biases. She repeatedly disabuses us from our “special” places — at the center of the universe, the solar system, the tree of life, and now the spectrum of general intelligence.
This changed my perspective on the LLM “statistical parrot”/“no True Scotsman” critics. Their conviction without evidence (i.e faith) that SotA models don’t “really” reason comes from insecurity. It’s a loud reaction to egos popping. That’s a trauma that I can sympathize with.