One of the benefits of writing tests is that helps a programmer avoid writing code that is hard to test (and by implication hard to measure correctness). And one of the goals...or maybe the only goal is to use tests to improve the code. Based on your description, a first order approach might be to write (and/or refactor) your code into a testable form. And by 'testable' I mean what your current skill set allows you to test.
If you've listened to 'uncle' Bob Martin, you've probably seen him advocate writing tests first (if you haven't listened to him and you're interested in testing, he has an experienced view whether or not you wind up agreeing with him). Writing tests before the code means that the code will either be testable or it will be wrong (it might be testable and still wrong but there's a chance that the tests will show that).
Anyway, I suppose my point is that if I don't know how to test my code, the problem is probably the quality of my code not my knowledge of testing. Perhaps complex testing techniques are a code smell for complicated programs. Or not.