21–30 of 79 posts
Earlier quoted context omitted.
https://www.forbes.com/sites/antoniopequenoiv/2026/04/30/elo...
Sounds fair, since Sam can run a for profit NGO
It kind of confirms a hypothesis I have that the next phase of AI development will be about getting smaller (in terms of model size and compute), because smaller is more capital efficient for training (allowing faster iteration and more iteration cycles for a given amount of capital), allows for denser inference (more inference for a given amount of compute hardware), and allows for more edge inference applications.…
Continual learning tends to imply individualized models, else there is no data privacy (the secrets learned on the job at your company now being available to your competition), which really turns the current AI business model of a single centralized model served to everyone on it's head. If every customer has a different model that essentially means the end of batch processing with the same weights loaded into the GPU.
The direction this suggests is a move away from centrally served common models to locally served individual ones, which generally requires them to be smaller, even if some larger companies may be willing to invest in beefier hardware.
I think this is at least in part why the AI companies are trying NOT to implement true continual learning and see if they can instead finesse it by implementing continual compacted(?) memorization instead, since then it's "just" additional context that needs to be recalled and fed into every request, not weights that need feeding into the GPU. I don't think memorization is any substitute for learning, especially learning of practiced skills, but since it's far easier to implement, and non-disruptive to the cloud-based API business model, this is what we will see first.
The recent news of NVIDIA' investment in Sutskever's SSI has a tiny hint of this also, talking about SSI advising on NVIDIA's future architectural direction - apparently pushing it in a different direction than current (cloud-based, pre-trained) models. NVIDIA may be quite happy to see a move towards local models.
K3 is natively trained to mxfp4, if they cannot get a hold of Blackwell chip, it is meaningless. Hopper does not do native 4-bit floating math. Either they have Blackwell with native 4-bit floating math, or they use have Chinese domestic NPU that support mxfp4 natively. The article’s statement does not make sense.
Personally, I think 20K nvidias is a stop gap solution because they really don't have the capacity to serve their models to earn any money right now.
Interesting if true - that Moonshot can train a ~3T SOTA model on only 20K NVIDIA GPUs, while others like Musk (who freely admits to distilling OpenAI's models) struggle to build a competitive 1T model (Grok 4.5) with massively more compute (Colossus-1 100-250K GPUs, Colossus-2 500K+ GPUs). I guess at least partly a reflection of all the optimizations in the Kimi 3 architecture. In the recent leaked DeepSeek investor…
Interesting if true - that Moonshot can train a ~3T SOTA model on only 20K NVIDIA GPUs, while others like Musk (who freely admits to distilling OpenAI's models) struggle to build a competitive 1T model (Grok 4.5) with massively more compute (Colossus-1 100-250K GPUs, Colossus-2 500K+ GPUs). I guess at least partly a reflection of all the optimizations in the Kimi 3 architecture. In the recent leaked DeepSeek investor…
Are you sure they are using all of their compute on training? Didn't they rent out a ton to other AI companies?
K3 is natively trained to mxfp4, if they cannot get a hold of Blackwell chip, it is meaningless. Hopper does not do native 4-bit floating math. Either they have Blackwell with native 4-bit floating math, or they use have Chinese domestic NPU that support mxfp4 natively. The article’s statement does not make sense.
From the article:
A person familiar with Moonshot’s procurement strategy confirmed that the company does indeed have a channel for accessing Blackwell processors via Southeast Asia. They didn’t specify whether this was a rental channel...
Interesting if true - that Moonshot can train a ~3T SOTA model on only 20K NVIDIA GPUs, while others like Musk (who freely admits to distilling OpenAI's models) struggle to build a competitive 1T model (Grok 4.5) with massively more compute (Colossus-1 100-250K GPUs, Colossus-2 500K+ GPUs). I guess at least partly a reflection of all the optimizations in the Kimi 3 architecture. In the recent leaked DeepSeek investor…
> Musk (who freely admits to distilling OpenAI's models) struggle to build a competitive 1T model (Grok 4.5) with massively more compute (Colossus-1 100-250K GPUs, Colossus-2 500K+ GPUs) Are you sure they are using all of their compute on training? Didn't they rent out a ton to other AI companies?
It kind of confirms a hypothesis I have that the next phase of AI development will be about getting smaller (in terms of model size and compute), because smaller is more capital efficient for training (allowing faster iteration and more iteration cycles for a given amount of capital), allows for denser inference (more inference for a given amount of compute hardware), and allows for more edge inference applications.…
Yes, continual learning is currently a hot topic since this is what will allow an "AI intern or new employee" to learn on the job rather than be stuck on groundhog day, or pre-trained for every eventuality - needing to anticipate all the quirks and proprietary knowledge of every customer! Continual learning tends to imply individualized models, else there is no data privacy (the secrets learned on the job at your com…