Liquid Foundation Models: Our First Series of Generative AI Models

What you are referring to is called "temperature" with respect to LLMs, and it is a parameter that can be tweaked at inference time. Google's AI Overview gives a pretty good summary IMO:

> The temperature parameter in a large language model (LLM) controls the amount of randomness in the model's output. It's a numerical value that influences the model's choices, and is often set between 0 and 1.

> The temperature setting affects the quality of the output by balancing exploration and exploitation:

> Low temperature

> The model is more likely to exploit patterns it's already learned, resulting in more deterministic and conservative outputs. This is ideal for tasks that require accuracy, such as data analysis, technical writing, and summarizing research papers.

> High temperature

> The model is encouraged to explore, resulting in more varied and creative outputs. This is ideal for creative tasks, such as brainstorming, artistic content, and storytelling. However, adjusting the temperature too high can result in non-sensical outputs.

They point out in the paper stats around where effectiveness drops off hard. It's at 32k at most everywhere now. Some models still at 4 to 8k. Getting to 200k in a smaller model is an open research area, as far as I'm aware. Ideas so far are pretty broad ranging, including using fourier transforms to try and capture cyclicality in inputs (mamba et al), RWKV (which I do not fully understand, but claims very long input context support), larger range of possible tokenizations shortening input length (llama 3), ..

There's always risk!

But when you ask a model to rely on just the input data, you are (mostly) trying to tap into its reasoning, not knowledge side. Obviously what's kind of magical is that some knowledge will be needed for reasoning, and you have it. But LLMs small and large are pretty good at doing the in-context stuff. It is precisely what they're trained on, and in fact it was kind of a surprise how well they seemed to generalize outside of this task in the first place.

Every time there's a tech hype cycle, trust some academics to come out of the woodwork, raise some absurd amount of money and sit out the resulting crash, only to be acquired/acqui-hired by some company to on-shore talent in an emerging area, rest and vest, then become VCs/partners. Plenty of examples:

1. Covariant -> Started by Berkeley folks, acqui-hired by Amazon after half a year of peddling a patchwork quilt of buzzwords through their Robot Foundational Model - 1 (RFM-1).

2. Drive.ai -> Started by Stanford folks, acqui-hired by Apple, only for most of the team to leave to found Landing.ai (I still don't know what this company actually does apart from allowing the founders to carve out nice, fat consulting contracts?).

3. World Labs -> Started by Stanford folks, physical embodiment but only 3D, NeRFs, perception something something? Not a single operator/person with business person in the founding cabal.

4. Physical Intelligence -> Started by Stanford folks (seeing a pattern here...), physical embodiment, data collection, foundational models something something.

5. Skild Ai -> Started by CMU folks, physical embodiment, again not sure what the plan is here.

6. Argo.AI -> Started by CMU folks, self-driving group that didn't go anywhere. Eventually shuttered and absorbed into Ford/Volkswagen.

7. Aurora.tech -> Started by CMU folks, self-driving team that SPAC'd and is now just existing in between debating raising capital or selling itself for parts I think?

8. Rethink Robotics -> Started by MIT folks, Rodney Brooks' legendary robotics companies that wanted to mass produce industrial robots, and then decided how the robot eyes should look like...fold, shuttered, and now relaunched only in name, none of the original robots or aesthetic (rightfully so) persists.

So many examples like this...none of these companies actually are vaguely interested in creating enduring business, have a business plan (other than GPU go brrr) or have any idea of how to run a business.

I don’t think we’ve yet come to the point where, how an LLM end to end goes from prompt to output is blatantly obvious.

LLMs operate with language at multiple levels of abstraction and tokens are not the only way to have character level knowledge.

For example, nothing excludes pre-training data from directly or indirectly encoding such knowledge.

And of course LLMs famously have emergent properties, for which there’s not yet a precise mechanism to illuminate the results.

We’re dealing with very complex systems that are still relatively poorly understood, and I believe the pool of concepts understood to the point of being blatantly obvious is still small.