There's never been a better time to consider writing your own media server solution from scratch.
I really don't like the idea of transcoding on demand. It has always created a janky experience on the client. I've been using a custom solution that runs everything through ffmpeg normalization as part of the media upload phase. Whenever a client wants to stream something we give them raw byte ranges directly from the block storage provider. In practice, this feels like flipping the channels on an analog tv. Every video starts and seeks nearly instantly.
I've long since given up on the dream of maintaining lossless master copies of everything just in case. Every practical use of that 50 gb copy of Home Alone would have you transcoding it. Most clients would choke on that bit rate. When would you actually be able to use all of the available information? Why not just discard what you'll never use? I realize it feels offensive to kill information like this but you can do a much better job with the end experience if you do.
You can get a modern chatbot to produce this experience in a single prompt. The code can run on anything slightly more powerful than a potato because you aren't transcoding at playback time anymore. A raspberry pi would be more than enough to serve a few clients on a LAN. If you remain humble in your list of demands (a simple directory browser that plays videos linked within), you can more easily achieve total autonomy.