I've been trying to learn how to make games in my spare time on-and-off for 20 years (wow time does fly). Here's my postmortem to serve as a warning for younger versions of me.
My first mistake was underestimating how much math one needs to know. Spoiler: it's all math - mostly linear algebra. That's why any graphics / physics / gamedev book has at least one math chapter. You do need it for everything - from coloring something to making things move on the screen.
My second mistake was not making games. One would think that it would be an obvious thing to do, but it's so easy to underestimate the amount of time it takes to understand graphics programming (in retrospect, just the size and weight of the "Computer Graphics - Principles and Practice" book should have been a gigantic red flag). I ended up writing game engine code instead of game code. Being able to draw triangles on the screen is just not as satisfying for me as making cars chase each other and it also doesn't move the needle on getting a game out the door. In practice this translates to maybe reading "Programming Game AI" instead of "Real-Time Rendering" (given limited spare hours) or actually finishing "Nature of Code" instead of reading "Game Engine Architecture".
Arguably my biggest mistake was being stubborn, but not stubborn enough to persevere through all the crap. Early on I didn't want to use an engine because I generally like to understand things from first principles and I didn't understand how deep the rabbit hole went. It turns out that the first principles in game graphics are closer to a collection of progressively more horrible hacks on how to achieve realism in real time without ray tracing, because that's too slow, and how to patch all the side effects of that reality (don't drop a book on implementing shadows on your feet). People with PhDs describe these techniques in scientific looking papers, which then get discussed at large conferences, referenced in books, etc and normalized. Want to learn a new technique? Read the (dense) paper. Other people decide then to take the best hacks, accelerate them in hardware and expose them as APIs. Decades of optimizations and hacks built on top of each other pushed the entry bar higher and higher, to the point where the amount of complexity one needs to wrap their head around is just not realistic for a beginner, unless they are stubborn enough and diligent enough to persevere through all of it. I should have paid that $100 to use the Torque engine, instead of buying the OpenGL Reference Manual, OpenGL Superbible and the OpenGL Shading Manual.
Lastly I fell into the trap of buying the books, but not working on games, aka "all the gear but no f*cking idea". My shelf is filled with highly regarded books that I read and some I gave up on (because they were over my head). I released 0 games. That said it's a good way to discover gems like "The Ray Tracing Challenge" book. Learning new things is fun and addictive.
I came to the realization that I would have really benefited from having a mentor, a well defined and structured curriculum, and being a bit more humble about my own ability to jump into this field, but, most importantly, I should have just made games - crappy looking, never-going-to-make-money, but fun games. That's why learned to code in the first place, after all.
These days I am using Godot and am finding it easy to use, especially after spending all this time fiddling in the graphics world. However, having children greatly reduced the amount of time I can dedicate to this hobby.
It was rather painful and cathartic to type this out.
P.S. I tell myself I'll find time to read "Physically Based Rendering" (another book that would make a whole in the floor if dropped), "Realtime Collision Detection" (more fun math!) and the newest edition of "Real Time Rendering" (its website is a nice resource to find other graphics books to purchase and never read).