How is ChatGPT's behavior changing over time?

With those GPT4 architecture infos leaked a few weeks ago, my personal theory is that they use mixture-of-experts routing for scaling the system. For instance, they could say: from 0 to 50% system load use a mixture of top-6 models for inference, from 60 to 80% use top-4 models, above 80% use top-2 models. This could make ChatGPT look dumber or smarter depending on the time of the day and system usage. Naturally, the variance of inference quality would spread and people would experience regressions in quality with some probability.

They could also use these parameters over time to become more cost efficient, in general, or reserve GPUs (i.e. allocating more expert networks) for some higher-margin use-cases.

And even the lecturers acquiesced when they found that a lecture on the sea was none the less stimulating when compiled out of other lectures that had already been delivered on the same subject. “Beware of first-hand ideas!” exclaimed one of the most advanced of them. “First-hand ideas do not really exist. They are but the physical impressions produced by love and fear, and on this gross foundation who could erect a philosophy? Let your ideas be second-hand, and if possible tenth-hand, for then they will be far removed from that disturbing element — direct observation. Do not learn anything about this subject of mine — the French Revolution. Learn instead what I think that Enicharmon thought Urizen thought Gutch thought Ho-Yung thought Chi-Bo-Sing thought Lafcadio Hearn thought Carlyle thought Mirabeau said about the French Revolution. Through the medium of these ten great minds, the blood that was shed at Paris and the windows that were broken at Versailles will be clarified to an idea which you may employ most profitably in your daily lives. But be sure that the intermediates are many and varied, for in history one authority exists to counteract another. Urizen must counteract the scepticism of Ho-Yung and Enicharmon, I must myself counteract the impetuosity of Gutch. You who listen to me are in a better position to judge about the French Revolution than I am. Your descendants will be even in a better position than you, for they will learn what you think I think, and yet another intermediate will be added to the chain. And in time” — his voice rose — “there will come a generation that had got beyond facts, beyond impressions, a generation absolutely colourless, a generation ‘seraphically free From taint of personality,’ which will see the French Revolution not as it happened, nor as they would like it to have happened, but as it would have happened, had it taken place in the days of the Machine.”

E M Forster, "The Machine Stops", 1909

> never going to have that kind of awareness

Be careful with that goalpost, it might make sudden movements.

It's important to read company marketing statements as if you were a lawyer. We haven't made GPT-4 dumber != We haven't made ChatGPT(4) dumber.

Personal hypothesis is that they have made a few changes to ChatGPT recently - possibly quantization, and almost certainly some tweaks to make it give shorter/less detailed answers.

But by the nature of a probibalistic tool being run by a secretive company, it's hard to say for sure. Maybe I and everyone else complaining have just started to get unlucky answers.

Just yesterday I gave ChatGPT a summarization task and it performed horribly. I even tried multiple times and got the identical answer. Then I gave the identical prompt to gpt-3.5-turbo via the API and I immediately got the expected good answer.