Earlier quoted context omitted.
Publishing weights? Meh. Publishing code and data would lead to abolishing copyright.
Copyright is just not prepared for AI. Training with copyrighted material could become "officialy legal" under copyleft terms, at least when the amount of training material exceeds a certain threshold.
They are called large language _models_ for a reason. They are just statistical models of existing work.
If anybody seriously thought they were intelligent, they'd be arguing for giving them personhood and we'd see massive protests akin to pro-life vs pro-choice. People (well, ML companies) only use the word "intelligence" as long as it suits their marketing purposes.