Rather than simply give you a list of resources or textbooks, I’d like to give you a broad “map” of the various domains of mathematics, this way you understand what you’re working towards. I’d also like to recommend how you can maximally optimize your self-study, as someone who mostly self-learned enough mathematics to be active in research. I think this meta-direction is just as important as the resources you choose to learn from.
Mathematics, in my opinion, can be divided in two very major ways ways as concerns pedagogy. First, most mathematics computation-based or proof-based. Math research in general is about proving things, and most “serious” math books after a certain level are almost exclusively about proving properties instead of calculating results. On the other hand, most applied mathematics is computationally inclined, and uses methods derived from research. Here is a simple example: I can ask you to calculate the square root of 2 or I can ask you to prove that it’s irrational.
It’s important for you to know what you want. Do you, for example, want to achieve theoretical mastery of linear algebra that subsumes e.g. solving linear equations, or do you just want to be able to execute the computational methods proficiently? As you get into higher mathematics the line here blurs, but different resources may still emphasize one approach or the other.
Now let’s talk about the domains of mathematics. Broadly speaking, we can divide them into algebra and analysis. More accurately, we can divide their methods into algebraic or analytic. Algebra is concerned with mathematical structures and their properties, like fields, groups, rings, vector spaces, etc. Analysis is concerned with functions, surfaces and continuity. I like to say that in algebra, it’s difficult to identify what you’re studying and whether it’s worth studying it, but once you do there is a lot of machinery that’s relatively straightforward to prove. On the other hand, in analysis it’s easy to find things worth studying, but difficult to prove interesting things about them. For example, if you can prove that what you’re studying satisfies all the conditions of a field, you immediately can prove many other things about it. On the other hand, the toolbox of analysis is widely applicable to many things, but it often seems like you’re trying a hodge podge of techniques, and the proofs can look kind of magical at first. For a concrete example, try to prove that 1 + 2 + 3 + ... + n = n(n + 1)/2.
Now let’s take a tour of mathematics at the undergraduate level. In theoretical (but not necessarily pedagogical) order we have: set theory, calculus, analysis, topology and probability theory on the analytic side; and set theory, linear algebra and abstract algebra on the algebraic side. Analysis can be further subdivided into real analysis, complex analysis, functional analysis, harmonic analysis, Fourier analysis, as you move from foundational material to specialized material. Similarly abstract algebra divides into group theory, ring theory, finite fields, Galois theory, etc. Probability breaks down into discrete versus continuous random variables, measure theory, statistics (on the applied side), etc.
Here is my concrete advice regarding learning the material. First, internalize the idea that mathematics is “not a spectator sport.” You learn it by doing it, not just by reading it. Every time you’re sitting down with a textbook, attempt every exercise in good faith, and take an author’s lack of a proof as an invitation to prove it yourself. The first time you read a chapter, read it briskly, skipping over what you don’t know to get to the end of the chapter. Let that material percolate in your mind a bit, even though you won’t understand much of it. Then read the chapter again, but slowly. Write down every definition, theorem and proof. Try to prove each theorem yourself before reading the author’s proof. For anything unclear, search for different examples of that concept or for different proofs of that theorem. Then attempt at least half of the exercises at the end of the chapter. You will struggle a lot, and you will be demotivated a lot. It will feel frustrating and you will be humbled continually. But I can promise you that if you keep challenging yourself this way you will continue to improve. It’s not enough to find the right textbooks or the right resources, you need to study them the right way - in an active, focused way.
That brings me to my second piece of advice. There are many good books and resources for any given topic. Different people respond more favorably to different types of exposition. Sometimes you’ll receive a book suggestion and realize it’s not for you - that’s fine! It might still be a good book. For example, I rather like Rudin’s Principles of Mathematical Analysis, but please don’t try to learn from it without a teacher! For any given topic, find four or five strong suggestions, preferably all at your level of capability at the time. Then read the preface and the first 10 pages of the first chapter in each book. Look at the table of contents to understand not only the coverage of topics, but the pedagogical arrangement of topics. Proceed with the book you have the strongest affinity for, and use other books when the author is unclear.
Finally, now I will give you textbook suggestions:
1. Set Theory: Naive Set Theory, Halmos.
2. Calculus: Single Variable Calculus, Stewart; Multivariable Calculus, Stewart; Calculus, Spivak.
3. Linear Algebra: Linear Algebra and Its Applications, Strang; Linear Algebra Done Right, Axler; Linear Algebra, Hoffman & Kunz; Finite Dimensional Vector Spaces, Halmos.
4. Analysis: Analysis I, Tao; Analysis II, Tao; Understanding Analysis, Abbott; Principles of Mathematical Analysis, Rudin.
5. Abstract Algebra: A Book of Abstract Algebra, Pinter; Abstract Algebra, Dummit & Foote; Algebra, Artin; Algebra, Hungerford; Algebra, MacLane & Birkhoff; Algebra: Chapter 0, Aluffi.
Start with that, and once you've gained sufficient mathematical maturity look for more targeted and specialized resources. I also recommend that you read Concrete Mathematics by Graham, Knuth, Patashnik; and Mathematics: Its Content, Methods and Meaning by Kolmogorov, Aleksandrov and Lavrent'ev. These two, especially the latter, are good for covering a variety of mathematics at once. They are good for both learning and mathematical "culture."
I can't stress this enough: it's important that you really optimize the way you're studying and what your goals are, instead of trying to collect as many book recommendations as possible.