> Labs have already used up internet-scale data
Not really, but a lot of what isn't used isn't very good.
More important is synthetic data. Use a teacher model with RAG with a huge reference library to write synthetic transcripts of idealized behavior for the model. Use models to judge and correct these transcripts. Train on the good ones. Use bad traces to train the model to correct its own errors (e.g. don't train it to produce a bad transcript but if it finds itself in the middle of one train it to self correct).
Similarly, for tasks that can be closed loop evaluated -- e.g. running computer software and programming, unlimited amounts of novel training data can be generated... including for highly original tasks: e.g. run publications in any domain through a model prompted to look for programming problems suggested by the material. Then write/judge/improve transcripts of solving those novel problems.
I expect in the future smaller models won't be directly trained on any internet data at all-- but entirely on simulations of idealized expected behavior from the model under construction. Raw internet data in that case would show up in prompts, but never in the target output (except of course for prompts that are asking it to copy the input).