I'm going to say the unthinkable here:
In my Information Retrieval class, I got numpy/scipy set up and went about implementing homework assignments with it.
However, no matter how much I tried to push as much as possible down into the matrix libraries implemented in C/C++, the surrounding Python code slowed everything down. I was having trouble getting everything to finish in time to hand in my homework by the deadline.
I talked to a classmate who was using Java, and not having any speed problems at all. The night before it was due, I rewrote the whole thing in Java and got it to finish running (I handed in a day late, but at least I had something to hand in.)
I'm sure there are tricks to make things faster in Python. (For example, I later figured out a method I was calling was running all Python code, and if I had called a different method, it would have dropped directly into the fast C code.) But with Java, I didn't have to think about performance. It was just fast.
Java almost certainly has a library for anything you might possibly want to do. "But," I hear you say, "that means I have to write my program in...Java! shudder"
And I empathize with you. Which is why now I'm doing a lot of experimenting with Clojure. Fast as Java, because it compiles to the JVM (as long as you follow as few guidelines.) Access to any Java library with no extra effort on your part. (One of my favorite moments on one of Rich Hickey's Clojure video is where he shows a macro that makes Java calls requiring FEWER parentheses than Java. He was pretty excited about that.) I found a Java open source matrix library that, while not nearly as pretty as Python, got the job done.
So, that's my totally radical recommendation. Clojure + whatever Java libraries you need to get your work done.