Earlier quoted context omitted.
> LLMs and foundation models are fungible and easy. The top couple LLMs are extraordinarily expensive - will get dramatically more expensive yet - and are one of the most challenging products that have been created in all of human history. If what you're claiming were true, it wouldn't cost so much for Meta and OpenAI to do their models, and it wouldn't take trillion dollar corporations as sponsors to make it all wor…
> The top couple LLMs are extraordinarily expensive - will get dramatically more expensive yet - and are one of the most challenging products that have been created in all of human history. I disagree. The more we learn about LLMs the more it appears that they're not as difficult to build as it initially seemed. You need a lot of GPUs and electricity, so you need money, but the core ideas: dump in a ton of pre-traini…
And not to discount the other important advances like RLHF, but the reason everyone talks about the big model companies as having "no moat" is because it's not really a secret of how to recreate these models. That is basically the complete opposite of, say, other companies that really do build "the most challenging products that have been created in all of human history." E.g. nobody has really been able to recreate TSMC's success, which requires not only billions but a highly educated, trained, and specialized workforce.