No one cannot possibly know what happening in your mind, to find what is wrong with it, based on this short description. But I have a guess.
Mostly people do not try to remember this things, they do it the other way. For example, I'm learning what is two's complement. I'm not trying to remember math representation, this representation is not something to rememeber, it is just one more way to explain what happening. It is a way to clarify things, in case when textual explanation was not enough. What I'm trying to do, is to find a way how should I think, to reinvent two's complement when needed. I'm trying to understand what original inventor of two's complement was thinking: why he did it this way, not the other? And I'm trying to find a visual representation of two's complement.
It is the best way to deal with math and technology. Even if you learn social science, you can learn how some author thought, instead of remembering his books by a heart. If you learnt how to think like others, then you would need not to remember their thoughts, you would be able to generate their thoughts yourself. It reminds me: https://hackernoon.com/how-not-to-memorise-mathematics-98fef...
My history teacher in the school claimed, that he didn't remembered any historical dates except two or three. He managed to calculate all the other dates, using these as a reference. I never learned to do that, but I believe that it is the same thing, as I do with math and CS. He learned to think like history, and to generate dates like history did, instead of remembering them.
Your memory is not like computer's one, your memory is generative, your mind is a compression algorithm, that can represent data by smallest possible about of remembered bits. As a funny consequence, your memory can generate ideas, that no one have found before, and sometimes you cannot tell is the idea you found is novel, or you read it somewhere. It is because generating new ideas and remembering old ones actually the same process in a mind.
(With two's complement, I confess, it is easier for me to remember, because if is modular arithmetic, it is ring of integers modulo 2^n, I dealt with them a lot while studying math.)