Oh, this is a fun subject.
That the system file contained the "entire OS" is a bit of a misconception: the bulk of the OS primitives are stored in ROM, and the system file mostly contained diffs between the OS version stored in each machine's ROM and the current OS. Eventually, the practice of storing all these diffs in the system file stopped with 7.1.2, and they began releasing the diffs as seperate files called "enablers" (before then, the introduction of any new model required a new version of the OS to be released to contain that model's diffs). It also contained a number of resources that are too big to go in ROM, such as sounds, fonts, and "desk accessories".
Let's talk desk accessories, because they're cool and they exemplify the most interesting properties of classic Mac OS. Originally, Mac OS was purely a single-tasking OS. But Apple knew that people would be very frustrated if they couldn't run small apps, like a calculator, over the active application. They got around this with an applet system they called "desk accessories", or DAs for short. DAs were technically coded as device drivers, and they were stored in the system file as resources of type DRVR, just like any other device driver. The system file populated the Apple Menu with a list of installed DAs, and when you selected one, it would patch the DA's code directly into the running application's memory, giving the illusion of multitasking.
Fonts, too, were stored in the system file as resources of type FONT. Apple honestly never expected third parties to write their own DAs or fonts, which is why they opted for such a hardcoded solution. But people did so anyway, and they wrote particularly ugly hacks to do it. So Apple caved and wrote a utility called the Font/DA Mover, which allowed you to install FONT and DRVR resources from other files into the system file. It also allowed you to do the opposite, and extract those resources out of the system file and into new files (IIRC, the system file had a limit of how many resources of each type it could have, so power users had to have some way to swap things out). Apple also introduced the word "suitcase" to refer to files that consisted entirely of FONT or DRVR resources and existed solely as holding pens for fonts and DAs that weren't installed into the system file.
System sounds, too, were stored in the system file as resources of type SND (actually "SND ", as names had to be four characters), but there was so little interest in custom system sounds at the time that Apple never made a utility for it (third parties provided them, though).
A lot of this changed in System 7. First off, System 7 came with built-in coöperative multitasking, so DAs were no longer necessary. DAs were evicted from the system file, and Apple implemented a clever hack so DA suitcases could be directly executed as if they were programs, by causing them to be patched into the Finder's memory space when double-clicked in the Finder or run from the Apple Menu. Accordingly, the Apple Menu was no longer generated by the system file but reflected the contents of a folder under the System Folder called "Apple Menu Items", which typically consisted of aliases (another new System 7 feature: aliases were kinda like symlinks, but smarter). Second, the Font/DA Mover was obsoleted, and custom sounds endorsed, by giving the Finder the ability to browse the FONT and SND resources of the system file as if the system file was a folder, allowing the user to drag and drop font suitcases and sounds files in and out of the system file. Apple also redefined the word "suitcase" to now mean "any file whose resource fork can be browsed in the Finder like it was a folder": the system file was now listed in the Finder as having a file type of "suitcase", and the contents of font suitcases could now be browsed as if they were folders (users could now drag and drop individual FONT resources between suitcases, and individual font resources within a suitcase could be double-clicked to show a preview of the font). DA files were no longer described as "suitcases", as they were simply displayed to the user as if they were executable programs. The DA API was also deprecated, and Apple began telling developers to just write regular programs instead.
System 7.1 changed this further, by pulling fonts out of the system file, leaving only system sounds. Instead, the System Folder now contained a directory named "Fonts", which was filled with font suitcases. IIRC, at some point, they also extended the "suitcase" functionality of the system file to allow keyboard layouts to be dragged and dropped in and out of the system file, but I don't remember too much about that.
The technical details of Classic Mac OS were really interesting.