Hey everybody, OP here. Thanks for the great feedback! We're really happy that so many people have checked this out. One thing that I want to mention: our service was built favoring Precision over Recall; we reasoned that we'd rather have a low number of false positives and make sure that when we do report a detection, that it actually is one. Thus, our service may occasionally miss instances. I'm going to implement…
You might want to let the user decide if it is more important to have a false positive or a false negative. For some applications a false alarm is a minor nuisance but a false negative is catastrophic, but for some applications it is flipped. In the past I have let the end user define the balance (i.e. "a false negative is 10X as bad as a false positive") and the decision results were scaled by their decision rule. It's not always easy to do as many machine learning algorithms are nonlinear but at least you can cast a wider net of potential customers.
