I did some of the initial mmu support for A/UX while at UniSoft, so lived a fair amount of the history, at least for version 0 of A/UX.
For those not familiar with the times, this was still at the outset of the Unix wars - Berkeley vs AT&T vs everyone else.
UniSoft was a porting house based in Berkeley that specialized in putting Unix on almost anything. For example, the first Unix implementations for Sun and SGI were Version 7 ports done by UniSoft. The business model problem was that support costs, Time-to-market, and vendor customization rapidly pushed high volume customers into doing all their Unix work in-house. Apple followed the same path with A/UX, pulling the entire project in-house after UniSoft delivered the first version.
UniPlus was a blend of software from AT&T (System V release 2 and later SVR3) and BSD (TCP/IP, sendmail, bind and other utilities) and eventually Sun (NFS). This was viable because AT&T had a semi-official position that UUCP was networking. That was the business wing of AT&T, not Bell Labs. The internal fights at AT&T are items of legend, but basically Bell Labs stepped back and kept to its research charter (with OS work turning into Plan9). AT&T corporate kept adding legal and technical stupid to Unix until eventually the only option was SysVR4 and the unified field theory with Sun.
The Mac-II target for UniPlus used the Motorola 68851 MMU, a table walking highly configurable system. It was a stock item that UniPlus supported, but Apple wanted quite a bit of customization. 4K pages, Nubus memory, and MacOS address space support.
4k pages was a mostly trivial tweak from the 8k baseline, which had been selected at UniSoft for TLB efficiency. Apple wanted 4k because they had a smaller memory footprint and wanted to get better memory utilization. This was a good decision - I tested a 2k page size and it was even snappier for the small memory size, but lost on the TLB issues for larger memory and Apple didn't want the pagesize to be determined at boot time.
The NuBus memory was a discontiguous physical address space that wasn't initialized by the system. It was also an extra 2 clocks away compared to main memory. I dealt with the memory map, and added initialization hooks so that the memory could be found and used early, and built up a test system with two expansion cards and 20 MBytes of memory. Slower memory was pretty bad normally, but the 68020 I-cache and large register file made it benchmark OK. Unfortunately, sometimes the system stack was placed on NuBus memory and the impact on interrupt and system call latency was horrible. I set it up so that by default any external memory was dedicated to the IO buffer cache. Apple wasn't happy with this, but they accepted it since there was a driver boot-time flag to force a big pool of slow memory.
The MacOS address space was the most interesting. The old Mac systems were so memory starved that they took advantage of the 32 bit memory and 24 address lines on the 68000/68010 to store stuff in the high order byte. I can't remember if it was general storage or metadata - I seem to have blocked out the usage details. I put together a design that would alias the high order bytes by creating 256 overlapping top level segments that could share the rest of the page tables so that physical memory wouldn't be exhausted with page tables to cover the 32-bit address space.
Apple decided not to do this - they wanted a 100% user mode implementation. I was irritated at the time since they were choosing to sacrifice memory protection and security, but in retrospect I was only a couple of years out of school and I am not sure I fully appreciated the side effects of aliasing addresses that way. Still, it would have been fun to implement.
