Someone (anyone!) get David Sacks on the horn and tell him to read this.
Who's afraid of Chinese models?
171–180 of 965 posts
Re: Who's afraid of Chinese models?
#172According to openAI's own @deanwball: Even OpenAI isn't buying this distillation talk: https://xcancel.com/deanwball/status/2078133895766114412#m
here's the original X link : https://x.com/deanwball/status/2078133895766114412#m
Re: Who's afraid of Chinese models?
#173> I expect the inference market to grow much faster than training costs
This was my assumption as well. It's also generally true of 'traditional' deep learning models that inference cost is expensive compared to training.
But the cost per token for inference has been very quickly dropping. I don't recall where, but I recall about ~50x down from GPT3, even as model complexity has increased. Even with agentic systems, there are lots of optimization opportunities. I'm less assured about claims like this.
Re: Who's afraid of Chinese models?
#174Earlier quoted context omitted.
I'm arguing we can't trust retail prices because the marginal pricing isn't meaningfully connected to it anyway. But if we have to look at what we think margins might look like, DeepSeek continues to host v4 Flash at the existing price despite competitors beating it in price ( https://openrouter.ai/deepseek/deepseek-v4-flash ), so there's at least one example of a Chinese lab charging a predetermined price despite co…
>DeepSeek continues to host v4 Flash at the existing price despite competitors beating it in price ( https://openrouter.ai/deepseek/deepseek-v4-flash ), their competitors are discounted at around 33%, so it's safe to say that's the margin, maybe less if their competitors have worse caching or quantization. Meanwhile claude code/codex resellers selling tokens for 90% off API price, presumably by reselling usage from f…
The 33% discounted competitors have no non-retention policy
Re: Who's afraid of Chinese models?
#175Earlier quoted context omitted.
[flagged]
I’m 100% certain that China won’t be sending any goons to my front door.
This is true, but there is another foreign country that can send people to your door. What's to stop China from eventually buying that type of influence over our govt officials?
Re: Who's afraid of Chinese models?
#176Earlier quoted context omitted.
The (quite excellent) article discusses several of your points. If you haven't read it, I recommend it. - Commodity market profitability is determined by marginal cost of production. LLMs have marginal cost; traditional software does not. - Models are not free. Downloading them is free. Running them is not. This has manufacturing economics, not software economics; the idea that they are "free" is an economic category…
I think there are some really interesting thought there, but I’d challenge some of this: > Models are not free. Downloading them is free. Running them is not. This has manufacturing economics, not software economics; the idea that they are "free" is an economic category error as it relates to their actual use I think a large part of manufacturing economics is illiquid overhead and the cost of expertise to set up and…
Amazon is not "meaningfully in inference"? Bedrock seems to have a ton of enterprise customers, some of which would never trust the AI labs themselves with their data but will trust Amazon.
Re: Who's afraid of Chinese models?
#177Commenting wholesale on some folks who are asking for hard evidence. I cannot provide that either but can contribute some empirical data.
I have been working on a project with about a dozen generation tasks, each of which comes with a fixed token budget. The nature of this system requires that most tasks be completed by distinct model families.
As a result, I tested ~50 models across as many model families as I could gather, frontier and open weight, API (gateway and direct) and self-hosted. Evaluation was based on a set of cosine similarity validations that was repeated across ~50 different embedding models.
Interestingly, frontier models did worse on the tasks than open weight models. However, when it came to costs, the picture was reversed: frontier models were much, much more token-efficient. In fact, almost no open-weight model was able to meet the initial token budget, while almost all frontier models did. Moreover, open weight models struggled massively with reasoning, in terms of latency and token consumption.
I also found that the latest models did not perform better than older models. And any a priori benchmarking data was utterly useless.
So, I ended up using a set of open weight models without reasoning, as it turned out reasoning as well as frontier negatively correlated with the tasks. However, before I knew this, I had spent a lot of time running each available reasoning level for each model.
Lastly, as an aside, when it came to embedding models, size (dims as well as model size) did not correlate with quality, once a hurdle figure (~2k dims) was met. In fact, sweet spot was 3-5K, and for my (text-based) set of tasks, dense models tended to outperform MoE ones.
Re: Who's afraid of Chinese models?
#178> distillation: why exactly is it bad? After all, what are large language models but the distillation of all of the knowledge on the open Internet, scraped by the frontier labs and distilled into the models that are themselves being distilled? Who is exactly being wronged here? ... The U.S. should pass a law that (1) makes explicit that collecting data for training models is fair use, and (2) bars terms of service th…
Why would reading copyrighted material ever be an issue anyway? Wouldn't copyright law only apply to what you create and publish using the model? Training on every comic book should already be perfectly legal, as long as you accessed them legally, right? But publishing your own Batman comic using that training is copyright infringement.
What I'm saying is, doesn't the law already cover 1?
Re: Who's afraid of Chinese models?
#179"Anthropic and OpenAI likely have among the lowest costs per unit of frontier-quality intelligence" That's a big claim that his whole thesis rests on but is largely not backed up. Where are the apples-to-apples tokens-to-answer benchmarks that he's using - doesn't look like there are any, just a handwavy implication that US models are more token efficient, which they may be. But how is there so little effort in estab…
But.. if you are running Chinese model in the US, what difference does it make? Isn't the whole "scare" (khm khm) with Kimis is that now I don't need Claude, cause I can run Kimi on my own hardware in my own datacenter and it's maybe not as good as Claude July edition but it's is as good as Claude January edition.
It doesn't need to be as good. You can route to the appropriate model and save so much money.
I have sonnet do the thinking, deepseek does all the tasks. I've massively reduced costs with this approach.
Re: Who's afraid of Chinese models?
#180Earlier quoted context omitted.
I think there are some really interesting thought there, but I’d challenge some of this: > Models are not free. Downloading them is free. Running them is not. This has manufacturing economics, not software economics; the idea that they are "free" is an economic category error as it relates to their actual use I think a large part of manufacturing economics is illiquid overhead and the cost of expertise to set up and…
Amazon is not "meaningfully in inference"? Bedrock seems to have a ton of enterprise customers, some of which would never trust the AI labs themselves with their data but will trust Amazon.
Relative to their other compute or other players inference, not as significantly. Though yes, they certainly have some share.