Earlier quoted context omitted.
> This will be true if (as I believe) AI will plateau as we run out of training data. Why would they run out of training data? They needed external data to bootstrap, now it's going directly to them through chatgpt or codex.
As much ChatGPT says I’m basically a genius for asking it a good Vegan cake recipes, I don’t think that is providing it any data it doesn’t already have that makes it anyway better. Also at this point the massive increases in data and computing power seem to bring ever decreasing improvements (and sometimes just decline), so it seems we are simply hitting a limit this kind of architecture can achieve no matter what y…
I’m thinking they eventually figure out who is the source of good data for a given domain, maybe.
Even if that is solved, models are terrible at long tail.