Snowflake Arctic Instruct (128x3B MoE), largest open source model

It's likely not the model size that's bigger, but the training corpus (see 15T for llama3). I doubt anyone has a model with “trillions” of parameters right now, one trillion maybe as rumored for GPT-4, but even for GPT-4 I'm skeptical about the rumors given the inference cost for super large models and the fact that the biggest lesson we got since llama is that training corpus size alone is enough for performance increase, at a reduce inference cost.

Edit: that doesn't change your underlying argument though: no matter if it's the parameter count that increases while staying at “Chinchilla optimal” level of training, or the training time that increases, there's still a massive increase in training power spent.

Total aside, but I appreciate your arxiv submissions here. Just because they don't hit the front page, doesn't mean they are seen.

Derivations of well known problems trip these models up big time.

   Which bullet lands first, a bullet shot horizontally or one shot towards the ground

   The bullet shot horizontally will land first, assuming both bullets are fired from the same height and with the same initial velocity. This is because the bullet shot horizontally has a horizontal velocity component that causes it to travel forward while it falls under gravity. The bullet shot towards the ground only has the force of gravity acting on it, causing it to accelerate downwards. Therefore, the bullet shot horizontally will hit the ground later than the one shot towards the ground

Even if you ask it to think step by step, it gets confused:

   Bullet shot towards the ground: When a bullet is shot towards the ground, it has both an initial horizontal and vertical velocity components. However, since we're only interested in comparing the time it takes for each bullet to reach the ground, we can focus on their vertical velocities and ignore their horizontal velocities.

   Let's denote:

   ( v_y ) as the initial vertical velocity of the second bullet (which is zero in this case)

   Using the same equation of motion in the vertical direction: [ y = h - \frac{1}{2}g(t')^2 ] Since ( v_y = 0 ), we have: [ t' = \sqrt{\frac{2h}{g}} ]