There have been many wonderful ideas and concepts for education systems over the centuries. They have/had different, sometimes contradictory philosophies. Most fall short of their values and ambitions, regardless of what these are.
IMO, the pressures leading to degradation are all somehow linked to universalization:
(a) Resource constraints. Student/teacher ratios. The availability of good teachers, at scale. A great teacher is the ultimate lever. But great teachers in every class, with enough time and energy to invest in every student... very hard to achieve at national scale.
(b) Voluntary, self-motivated students who want to learn vs checked-out teenagers that just want to pass the exam with minimum effort... it's a massive difference. It's the difference between a world class gymnastics club and the PE class from an 80s teen movie. Even if half the class is highly motivated, it can't be like the gymnastics club when half the class is there involuntarily.
The visionary, optimistic concepts are usually focused on what students can achieve when motivated and willing. Universal, mandatory education rarely achieves this attitude.
(c) The bureaucracy required for scale. Decisions about teaching methods, standardized testing and whatnot... these can be performing terribly for years and decades before getting dropped. A department starts judging schools or teachers by standardized tests... and then a whole generation falls into a stale "teaching to the test" paradigm that disillusions both teacher and student.
"Why are we doing this" - because we have to.