I studied Mechanical Engineering but gradually ended up as a full fledged software engineer from one who wrote ME related programs. So I have sort of always been in the midst of mid level maths and physics, but, as it happens, I lost it all except for some fundamental concepts. I had a similar epiphany as yours in my early thirties and this is what I did and it helped me greatly.
First of all, you have to realize that you are learning these stuff only for the sake of learning, as an intellectual challenge, rather than making a career out of it. So, you need not follow a pattern that is made for late teen students attending university. That includes not needing to religiously solve problems in textbooks, especially the numerical ones. If you have to, solve the conceptual ones. Further, you wouldn't even need textbooks.
What you do need, however, is a perspective. Unlike university students whose primary aim is to pass the exams that matter, graduate and get a job, for which they learn by rote, what you need is to understand the reason why learning certain topics is essential. You need to learn about the topics first in order to appreciate whatever you will encounter later. For this purpose, well regarded popular science books should be your first choice. Any worthy textbook will also give a bit of perspective in the preface, early chapters or the appendix, but it can never compete with popsci books which are designed for the purpose of elucidating academically complex stuff to an otherwise educated public. Good popsci books provide enough stimulation to your mind. They prime you for further, formal exploration of the topic.
Often times what happens is that highly technical topics such as those you mentioned look attractive from the outside but get painfully dry, boring and difficult once you pick up a text book and start studying from it. What I realized is that for many topics what I really crave is a deep understanding as an educated person as opposed to deep academic knowledge. For such topics popsci books regarded as 'hard' are better than proper textbooks. These days, with the availability of opencoursewares, you can simply watch a few lectures before deciding if its worth your time delving deep into the topics with a textbook. The textbooks are often listed in the course content, and we all know where we can find the appropriate pdf versions.
These books worked for me. They are in my home library and I try to flip through some of them once a year.
For maths, books like Prime Obsession by John Derbyshire, e The story of a Number by Eli Moar, Number by Tobias Dantzig, The Unknown Quantity by John Derbyshire provide good orientation for number theory, analysis & calculus and algebra. The second book might be tough to finish as its really really deep for a thin book. You might also look at books like God Created the Integers by Hawking, The Calculus Gallery by William Dunham that curate interesting and historically important results, where original works are often reproduced.
For physics, my goto book (which I have never finished) would be The Road To Reality by Roger Penrose. This self contained 1000+ page monster builds up to advanced physics in a methodical fashion from scratch. The first half deals with all the necessary maths from elementary to advanced and the second half covers physics. It has chapters with names such as calculus on manifolds, hypercomplex numbers, the entangled quantum world, gravity's role in quantum state reduction etc which sound pretty deep and complex for a recreational learner. Beyond that, you can also read books such as The Evolution of Physics by Einstein himself, Thirty Years that Shook Physics by George Gamow to get a historical perspective on the development of advanced physics. For a midway academic treatment, you can read Feynman's lecture on Physics volumes.
For discrete maths, books like The Algorithm Design by Skiena and Algorithm Design by Kleinberg and Tardos should serve you well. They don't push you into a maze of mathematics like Knuth books do, or a maze of source code like CLRS or Sedgewick books do, but show you the practical side of things with limited code and pseudocode. That said, Knuth's The Art of Computer Programming 4A Combinatorial Algorithms Part 1 will just blow your mind.