Live data from Hacker News

Malicious software libraries found in PyPI posing as well known libraries

nbu.gov.sk

191–200 of 254 posts

Re: Malicious software libraries found in PyPI posing as well known libraries

#191
post #161

Earlier quoted context omitted.

This is a scary attack. One partial mitigation is to use a firewall (e.g., Amazon VPC network ACLs) to restrict outbound network traffic to a small number of known addresses like well-known repos. I can't think of a good reason why code in any well-behaved application should be allowed to make random outbound network calls. I think it's also on app developers to rethink the culture of randomly grabbing packages to bu…

> One partial mitigation is to use a firewall (e.g., Amazon VPC network ACLs) to restrict outbound network traffic to a small number of known addresses like well-known repos. That breaks down very quickly with the combination of public CDNs and TLS. I suppose you could do SNI based firewalling, but that is bit ugly, and afaik you can't do that easily with common firewalls (like netfilter).

As I mentioned above some of this goes back to application design. Ideally if you are layered correctly the application components of the system that own data should not talk to the outside world except in very narrowly circumscribed ways.

Unfortunately most real world systems that have internet access are created in far from ideal conditions.

Re: Malicious software libraries found in PyPI posing as well known libraries

#192
post #180

Earlier quoted context omitted.

>both the pypi devs and the python security team decided to ignore it. Without trying to downplay the seriousness, I think that's a less-than-charitable take on the bind they felt they were in. It also sounds like there's a crack team of dedicated devs sitting around waiting for something to work on.

When I read things like, https://caremad.io/posts/2013/07/packaging-signing-not-holy-... I get the impression they don't care about security at all. They seem like children plugging their ears and shouting "nah nah nah" while putting repo users at risk. They've obviously done nothing since that post was made four years ago. In contrast, Maven central requires signing. Unsurprisingly, Maven central doesn't have typosq…

> In contrast, Maven central requires signing.

forgive my ignorance and my lack of 56 minutes to watch the entire youtube video, but who are the identities behind these signatures? The blog post you reference discusses the problem both of users signing their own packages (anyone can make a signature and any malicious package author can point people at a maliciously-owned signature as well) as well as having a central key (used by organizations with employees and known contributors, does not scale to pypi's model).

I'm also ignorant of a vulnerability scanner for Python (haven't looked). Does such a tool exist and have you proposed it as part of pypi's infrastructure? I am sure they'd be interested in that.

I'm not sure how the license file of a product impacts the issue of it being malware or not.

Took a look at https://www.sonatype.com/ and it appears to be a closed-source, commercial product - it appears to have a database of vulnerabilities in some format, but it appears to use hashes of some kind. I'm not sure how that would work against arbitrary Python source code, but again, I am ignorant. I would encourage you to write a comprehensive rebuttal to the blog post you refer towards.

Re: Malicious software libraries found in PyPI posing as well known libraries

#193
post #146

Earlier quoted context omitted.

pip list --format=legacy | cut -d' ' -f1 | xargs egrep '^(acqusition|apidev-coop|bzip|crypt|django-server|pwd|setup-tools|telnet|urlib3|urllib)$'

When running that command, I get output like this: grep: alabaster: No such file or directory grep: appdirs: No such file or directory grep: arandr: No such file or directory for dozens and dozens of packages. Are those errors benign?

No, jastr's command is wrong.

Re: Malicious software libraries found in PyPI posing as well known libraries

#194
post #84

Ok, here's some ugly backstory on this: This problem has been known for a while, yet both the pypi devs and the python security team decided to ignore it. Last year someone wrote his thesis describing python typosquatting and standard library name squatting: http://incolumitas.com/2016/06/08/typosquatting-package-mana... However after that the packages used in this thesis - the most successful one being urllib2 - wer…

[deleted]

Re: Malicious software libraries found in PyPI posing as well known libraries

#195
post #193

Earlier quoted context omitted.

When running that command, I get output like this: grep: alabaster: No such file or directory grep: appdirs: No such file or directory grep: arandr: No such file or directory for dozens and dozens of packages. Are those errors benign?

No, jastr's command is wrong.

Whoops, remove the xargs

Re: Malicious software libraries found in PyPI posing as well known libraries

#196
post #84

Ok, here's some ugly backstory on this: This problem has been known for a while, yet both the pypi devs and the python security team decided to ignore it. Last year someone wrote his thesis describing python typosquatting and standard library name squatting: http://incolumitas.com/2016/06/08/typosquatting-package-mana... However after that the packages used in this thesis - the most successful one being urllib2 - wer…

> Being total jerks, we have a pingback in the setup.py of all packages

Ugh. Yes, you are being jerks. The ethical way to collect statistics would be to ask the victim to click the pingback link.

Re: Malicious software libraries found in PyPI posing as well known libraries

#197

Earlier quoted context omitted.

One thing to consider if you're going to provide a service like this: What happens if a vulnerability nevertheless sneaks through? The whoever did the vetting could conceivably get sued. So then they might want to take out insurance or try to protect themselves from lawsuits in some other way -- all of which is likely to make such a service even more expensive.

It has to be constrained to something reasonable. You can't guarantee the software is safe, but you can guarantee it is published by someone who is who they say they are, similar to EV certificates for domains. You can also refuse to publish packages with intentionally-confusing names.

"you can guarantee it is published by someone who is who they say they are"

Can you? Positively identifying people seems a pretty tricky and easily screwed up business.

ID's can be forged, and a web of trust requires, well, trust.

I guess such a service could say something like "we got this person's ID (and/or address)" or "here's this key's web of trust", and that would probably be a bit better than what we have today (which is virtually nothing), but it would still be a far cry from "guaranteeing it is published by someone who is who they say they are".

Re: Malicious software libraries found in PyPI posing as well known libraries

#198
post #180

Earlier quoted context omitted.

>both the pypi devs and the python security team decided to ignore it. Without trying to downplay the seriousness, I think that's a less-than-charitable take on the bind they felt they were in. It also sounds like there's a crack team of dedicated devs sitting around waiting for something to work on.

When I read things like, https://caremad.io/posts/2013/07/packaging-signing-not-holy-... I get the impression they don't care about security at all. They seem like children plugging their ears and shouting "nah nah nah" while putting repo users at risk. They've obviously done nothing since that post was made four years ago. In contrast, Maven central requires signing. Unsurprisingly, Maven central doesn't have typosq…

Package signing doesn't achieve anything without a trust model behind it, which is exactly what that post states. Too many people go "we need to add some crypto to this thing!" without developing a threat model and that ends up making the crypto pointless wankery to act as a security blanket without actually solving any problems.

Maven Central, to my knowledge, does not have typo squatting problems because Sonatype has a manual review process for all new projects. It has absolutely nothing to do with the fact that they allow projects to upload PGP signatures and it could not have anything to do with that, because PGP does not provide any mechanism to prevent that.

For example, there may be `urllib3` which is a valid project that must be signed by key X. We'll ignore how a tool like pip would find out that key X is the right key (although this is actually the most important part of a package signing solution) and just grant that we've solved that problem. Someone then comes and registers another project, `urlib3` which must be signed by key Y. The attack that is being described here is that a user would erroneously say ``pip install urlib3`` when they meant to type ``pip install urllib3`` and pip would then fetch that and download the package and install it. I think it is pretty obvious that signing doesn't help here, because pip doesn't know that the user really wanted urllib3 and not urlib3, so it can only determine that urlib3 is supposed to be signed by key Y (which of course, the hypothetical malicious person controlling urlib3 would have), fetch the package and verify it's signature.

There is only one tried and true method for preventing across the board this kind of human introduced error collision (aka typo squatting), and that is manual review of all new projects. The problem with manual review then becomes one scale. There are as of this time of writing 117,226 unique projects on PyPI with an average growth of around 100 new projects a day. In addition there are zero full time developers or operations or support people working on PyPI. There is one part time paid person (me), plus my unpaid time, plus one other part time unpaid developer/ops person who do the vast bulk of the work. There is simply not enough available bandwidth to process 100 new projects every day and to validate them for typo squatting/confusion possibilities.

Beyond that, there are a number of possible heuristic based approaches that can try to reduce the chance of this from happening such as using levinstein distance, unicode confusables, attempting develop "reputation", etc. Most of these are either so broad as to catch a lot of projects which are not typo squatting but are real, actual different things or are so narrow as to be trivially defeated. That's not to say they aren't worthwhile or there isn't an idea that would make sense but focusing on that has not been a priority for a largely volunteer based organization because there are lower hanging fruit that are more impactful , because at the end of the day without a manual review system individual end users are still ultimately responsible for ensuring they're asking for the correct thing (and even beyond that, they're responsible for ensuring that the thing they're asking be installed is something that satisfies their own security constraints).

Security is achieved by layering multiple secure systems on top of each other, not by randomly rubbing crypto on things because it makes you feel good to have crypto involved.

Re: Malicious software libraries found in PyPI posing as well known libraries

#199

Earlier quoted context omitted.

Interesting if you think that npm/Rubygems/PyPI are leaving a load of money on the table, why do you think they haven't introduced those services so far...

ISTM we're just talking about running an alternate, more restrictive registry? npm etc. don't have to play any part in that. This service could be offered by anyone: IBM could do it.

Indeed IBM or anyone else could do this, but they're not, which implies a lack of demand.

Re: Malicious software libraries found in PyPI posing as well known libraries

#200
post #180

Earlier quoted context omitted.

When I read things like, https://caremad.io/posts/2013/07/packaging-signing-not-holy-... I get the impression they don't care about security at all. They seem like children plugging their ears and shouting "nah nah nah" while putting repo users at risk. They've obviously done nothing since that post was made four years ago. In contrast, Maven central requires signing. Unsurprisingly, Maven central doesn't have typosq…

Package signing doesn't achieve anything without a trust model behind it, which is exactly what that post states. Too many people go "we need to add some crypto to this thing!" without developing a threat model and that ends up making the crypto pointless wankery to act as a security blanket without actually solving any problems. Maven Central, to my knowledge, does not have typo squatting problems because Sonatype h…

>For example, there may be `urllib3` which is a valid project that must be signed by key X. Someone then comes and registers another project, `urlib3` which must be signed by key Y.

Key X is on the company approved key list, key y is not. Your argument just fell apart.

>The problem with manual review then becomes one scale. There are as of this time of writing 117,226 unique projects on PyPI with an average growth of around 100 new projects a day.

You're not dealing with projects. You're dealing with keys. It's not one key per project. It's one key per contributor. This has the added bonus that if a contributor goes rogue, you can revoke the one key and all the suspect projects are are invalidated at once.

>There is one part time paid person (me), plus my unpaid time, plus one other part time unpaid developer/ops person who do the vast bulk of the work.

Sonatype has turned this into a rather nice business. It's not a volunteer project for them. You expect me to believe it's impossible despite solid examples to the contrary?

>at the end of the day without a manual review system individual end users are still ultimately responsible for ensuring they're asking for the correct thing

Blaming the victims.

>Security is achieved by layering multiple secure systems on top of each other, not by randomly rubbing crypto on things because it makes you feel good to have crypto involved.

It's also not achieved by doing absolutely nothing at all.

Post reply on HN