First, if you have a lot of baggage about numbers, realize math is not "numbers" anymore than speaking English is "the alphabet."
Second, look for "normal" books filled with words instead of numbers that talk about math and read those. When I was homeschooling my oldest son, I spent about four years picking up books of that sort. I got a lot of them off the clearance table for as little as a dollar. After I finally got my husband on board with my plan, I handed the stack of books to my son and said "Here, read these." For the next few months, there were no tests, no quizes, no written work of any kind. I periodically asked if he was still reading them. He periodically gave me a book back that he had finished. After six weeks, he began bouncing into my presence joyously announcing "I finally understand why you can't divide by zero! I knew it was a rule, but I never understood why!"
Third, look for fun ways to explore math. There are entertaining math books like "The Number Devil" that are a fun read and will likely introduce you to new concepts if you do not have a strong math background. I also bought games for my sons to play instead of doing arithmetic drill.
Last, don't let anyone intimidate you with jargon or with their failure to explain it effectively. (Write them off as "stupid" and move on. :-p) My oldest is still incredibly bitter about learning that "x" in algebra is basically the same thing as the empty space in basic arithmetic, it is just easier to move it around and lets you have more than one "empty space" by giving it a name. So that let's you have two or more mystery numbers instead of just one.
After I explained to him that X was the same as the empty space in more familiar problems, he realized he had been doing algebra in his head for years while playing video games in order to infer, for example, how many hit points of damage a particular attack did so he would know how many hits it took to kill something. He is still bitter at the system convincing him algebra was hard and something he could not do when he had been doing it competently for years.
Also, consider jumping in at the top end and working backwards. I read about half of "A tour of the calculus" with my son to help win him over and convince him math is your friend. Think of it this way: Just because you can't spell doesn't mean you won't fall in love with writing poetry. Once you are thrilling to the pursuit of writing a good poem, you will have motive to look up the correct spelling and spelling might start sticking.
Last, if you went through the standard algebra-geometry-trig track and have baggage about that, consider pursuing a statistics track. There are some very approachable stats books for laymen, such as "How to lie with statistics," and it shouldn't trip your "oh, god, I hate math and am bad at it" switch if you haven't previously been exposed to stats. Statistics is very serious math, yet many people do not realize that. I think the first chapter or two of "The cartoon guide to statistics" is the rough equivalent of my entire college intro to stats class. It is a deep subject, yet we hear statistics every day in the news, so it can feel a lot more approachable and familiar and relevant for some people.