Is multimedia programming inherently hard?

I spent 100+ hours developing a system that performs fairly simple functionality: capture screen/connected webcams on motion detection, add overlays, and save it as .mp4. The system grew up to be a mess, so I rewrote it from scratch again spending 60+ hours, this time with functionality to capture sources in parallel. It uses Windows Media Foundation API, written in 7000 lines of C++ code.

The thing is, it doesn't work and has nasty bugs on slower systems, it crashes once in a while, sometimes has audio/video synchronization issues, etc. - the bugs are mostly very hard to reproduce. Prior to this, I considered myself to be a good programmer but it has almost made me reevaluate my decision whether I should pursue programming anymore (I'm a CS student), not to mention the toll it took on my self-worth and mental health.

The question is, what's the reason after spending so much time, the end result is mostly unusable: quirkiness of Windows API, C++, or just my incompetence as a programmer? Or is multimedia programming by nature tricky?

P.S. Please ignore my username and new account as I'm too embarrassed to ask from my real account that gives away my identity.

You are not your code.

You show admirable qualities that good programmers have. For example -- boldness to take on a challenging project, tenacity to pursue a rewrite, and drive to pursue improvements such as parallelism. After all, if you only pursue easy projects then you are unlikely to improve your skills!

There are definitely challenges involved with multimedia apps. They demand good performance, and they require you to rely on third-party libraries and drivers.

I assume you are decent at debugging to already have gotten as far as you have. So, I would continue to take guesses at what the problem might be then test whether you are correct. Based on your description, perhaps the CPU cannot always keep up with the framerate. You could test this (for example) by seeing if rendering a lower-resolution output would alleviate some of the bugs.

If none of your guesses seem to work, pare your code down to make it as simple as possible (e.g. a basic webcam app without overlays). Once you get it reliably working, then slowly re-add your features. Worst case, you can always post-process the video after streaming it to disk, or have a tool such as FFmpeg add the overlay.

Good luck!

It sounds like you are talking about real-time data capture, analysis, visualization, encoding, and archiving. If so, what you describe is not just multimedia. It is a real-world hybrid of multiple specialities, and definitely becomes more difficult because you have to address all these aspects at once to meet your original goal.

If you had started with pre-recorded videos, you could imagine having pursued the challenges incrementally. Process a video and output some frame-by-frame analysis metadata. Process the analysis results and visualize them in the context of the source video imagery. Re-encode the results to a good storage format. Explore options for quality, correctness, performance, and resource requirements.

But, if you need to do this on live streams and run for indefinite periods of time, you suddenly need to be much more aware of computational delays, working memory sizes, buffering of input and output data, etc. You need it to work consistently and predictably while handling the full workload and any potential variability in other elements of the system. How much can the incoming video stream vary in bandwidth? How much can decoding, analysis, visualization, and encoding speeds vary with changes in the video content? How reliably can the input and output IO paths support the bandwidth?

And, as you mentioned, it becomes challenging to then squeeze this into a slower system than it was originally designed and tested on. Ultimately, you have to design for failure: think about how you want it to perform if it cannot satisfy the full objective in the time and resources available. Drop frames? Degrade stream quality? Drop entire low-priority streams...?

Multimedia is tricky but, from the way you describe it, you've come across the more general problem that writing concurrent software in general, and parallel software in particular, can be a real sod.

What you're likely to be experiencing are bugs due to race conditions. These are notoriously hard to debug.

It's one of the reasons why there's so much fuss around languages like Rust and Go. They have builtin features that try to make these types of issue less prevalent. I appreciate this doesn't help you directly but should make you feel a bit better; there are whole engineering teams working to avoid the types of issues that you're experiencing.

In terms of how to help you move forward, that's quite hard in this format. Sorry.