It's weird to say this, but BD (including 4K UHD BD) is actually really old tech. Sony was talking about quad-layer discs back before they deleted the mandatory disc caddy from the draft BD spec! OK, practically speaking nobody had their hands on a quad-layer disc until BDXL in the mid-2010s, but even then that's still old.
As for why nobody's made a higher-capacity disc... well, they did. It was even an industry standard. You just never heard about it because it was exclusively intended to be a replacement for tape libraries. I guess rolling out this tech to consumers was just too impractically expensive?
You can absolutely do resale rights with solid-state media, too. On the other hand, the Switch library is littered with games that require downloading an update in order to play. Switch 2 went further and had games that shipped as a pure license key with no data storage. The underlying economics of game distribution are actually really unfavorable to any amount of overhead. Hell, the reason why physical games even still exist AT ALL is because we can press BDXLs for pennies.
Going back to the stagnation in optical media, the read performance hit a wall a while back, too. You basically can't stream anything off these discs anymore. Hell, some Internet connections might actually be faster than an install from optical media! So that's not really the advantage people think anymore either.
The resale ability is basically the only reason to keep physical media around, though - and I'm surprised we haven't seen a renewed attempt to kill physical. I mean, with movies, most stores have already removed their BD sections; you basically can only buy those online or at some Barnes & Noble stores.