Unfortunately, there's little curation of interesting debugging case studies or "debugmes". You can try to scavenge bug trackers, but they aren't optimized for this, just a bag of itches users want to scratch. Neither are war stories very fruitful, since rarely do you get to hunt down the corresponding bugfix commits, which are essential to not miss on the details that made those bugs tricky. But then, you already have the solution in front of you.
Instead, I've found much more deliberate practice and therefore value from reverse engineering: just like in debugging, you want to understand the underlying logic through program analysis.
You can pick security CTFs, crackmes, malware samples, proprietary software with bugs you want to fix, or functionalities you want to add, or even games where you want to find some hidden content, extract some resources, or modify some behaviors. In all that you will find something of your interest, and apply approaches that I think translate well to software development:
* Differential tracing: Want to know how some action reflects in the codebase? Take k instruction traces where you do everything except that action, then take a trace where you do that action, find out what are the unique differences in that last trace. Want to know what data gets written? Same approach with memory dumps and breakpoints. How do different inputs affect these changes? You will learn to be methodic and throughout in what you test and log.
* Recognizing patterns: Sometimes you don't have symbols for your functions, are you able to identify printf at a glance, or will you waste time following the logic of the function? Do you see relative offsets being used and recognize accesses to an array? With source code all this happens on a more macro view, such as algorithms or design patterns hidden in all those coupled functions and classes. But the micro view also applies: figure out what constants relate to specific functionalities, and you can grep your way to relevant functions or documentation.
* Avoiding boilerplate: This follows from the previous point, since you want to recognize the flow of data through the codebase, in order to have some call hierarchy to follow, otherwise it's easy to waste time on functionally that is irrelevant to you. Start with how data enters the application: stdin, files, database connections, http endpoints... Tests, examples, or client apps will also help here.
Oh and don't worry about learning some assembly language, just make that investment, since that's the straightforward, well defined, predictable part.