This kind of research always interests me and at first glance I had a concern with this paper given that they mention:
> The dataset is based on a simple index file provided in the
Python Package Index [76]. In total, 224,651 packages were
listed in the index at the time of retrieving it.
For separate research reasons I've recently had cause to download the index as well and current versions are 315,000+ packages. The reference [76] indicates they retrieved it on March 28, 2020.
Initially I had thought that almost 100k packages in a little over a year had to be incorrect given that the first archived index from 2018 had around 170k packages listed (meaning 170k -> 225k in 2 years).
This increase probably just highlights just how much Python has just exploded in popularity.
However, it does cast some doubt on the effectiveness of this kind of research on the basis that there are a lot of new packages and likely noise in that dataset.
A follow-up and perhaps more useful bit of research would be to do this same analysis with the top downloaded packages visible via the published stats[1] and then perform evaluations as to whether Bandit was actually identifying vulnerabilities. I have no doubt that of the 197,726 packages they actually scanned there was a lot of noise. Also, if a package has fewer than some cutoff of downloads in the past month (perhaps 10, 100?) or is newer than a certain date it may make sense to exclude it.
The authors mention the accuracy of static analysis tools being a potential problem but the fact that no spot checking was done to see if it was even remotely correct is a bit of a problem given the conclusion. I admittedly skimmed over certain sections but I didn't notice any discussion of the "confidence" metric that Bandit uses and this is a huge problem. Bandit only ever reported low confidence injections. Consider that for a moment: the static analysis tool reported no other type of vulnerability that was low confidence, and that includes a break-out for XSS. Every other category was Medium- or High-confidence only.
Having worked extensively with a variety of static analysis tools they vary in quality by language and detection capabilities but are generally very poor measures of application security and are often rife with false positives. The underlying premise of the paper's conclusion is that Bandit is trustworthy enough of a tool to merit the conclusion that "security issues are common in PyPI packages."
Having some experience with it in the past, I disagree with that foundational assumption. Bandit is good at finding certain classes of issues but is overall not something to rely upon for anything more than sanity-check catching egregious types of problems. For example, I'd rate it quite highly on detecting the use of the "generally avoid this" functions. A good regex could also detect these. I would not trust the XSS findings, though no doubt some are correct.
[1]: https://packaging.python.org/guides/analyzing-pypi-package-d...