Malicious software libraries found in PyPI posing as well known libraries • Hacker News Board

Re: Malicious software libraries found in PyPI posing as well known libraries

#181

Post by anthony_barker » Fri, Sep 15, 2017, 6:32 PM UTC

hathawsh wrote: post #176

jsty wrote: post #169

Earlier quoted context omitted.

> I can't think of a good reason why code in any well-behaved application should be allowed to make random outbound network calls If your application implements webhooks, then there's a valid use case? I'm in complete agreement however that you should default to deny and open up as required.

For security, you should call outbound webhooks using a server dedicated to that purpose, where the main server and webhook caller are connected through an async message passing system such as RabbitMQ. That way you can maintain strict firewall rules on the main server.

Aws Sqs write only queue for external facing servers is what we do. What worries me is not servers... It's admin/Dev machines...

Re: Malicious software libraries found in PyPI posing as well known libraries

#182

Post by peterwwillis » Fri, Sep 15, 2017, 6:33 PM UTC

zzzcpan wrote: post #149

peterwwillis wrote: post #142

Earlier quoted context omitted.

Of course you can do this - all packages are basically just wrappers around upstream code. But you still need someone to maintain the wrapper, and they have to check every new code release to see if there's something in the wrapper that has to change. And there are multiple distros. There's no getting away from maintainers with traditional linux distros. Code package management is different. The author writes their s…

To be clear, what I'm suggesting is to generate those wrappers automatically, instead of maintaining them manually. A script can visit a release page daily, parse it and check for updates. If there is a new upstream release, it can generate a wrapper and let the build system do the rest, produce binaries, test them, etc. When things break, the code needs to be fixed, but it's definitely very far from every release. A…

That's basically how packages are maintained today, they just don't have as much automation and there's a lot less packages as a result. If you made one package for every software update and had to review each one you'd spend a lot more time reviewing.

Trust isn't an issue in reviewed/maintained repos because you have eyeballs on everything. When anyone can just ship an app/library and release it automatically you get these malicious software issues.

Re: Malicious software libraries found in PyPI posing as well known libraries

#183

Post by katastic » Fri, Sep 15, 2017, 6:35 PM UTC

zokier wrote: post #161

hodgesrm wrote: post #151

Earlier quoted context omitted.

This is a scary attack. One partial mitigation is to use a firewall (e.g., Amazon VPC network ACLs) to restrict outbound network traffic to a small number of known addresses like well-known repos. I can't think of a good reason why code in any well-behaved application should be allowed to make random outbound network calls. I think it's also on app developers to rethink the culture of randomly grabbing packages to bu…

> One partial mitigation is to use a firewall (e.g., Amazon VPC network ACLs) to restrict outbound network traffic to a small number of known addresses like well-known repos. That breaks down very quickly with the combination of public CDNs and TLS. I suppose you could do SNI based firewalling, but that is bit ugly, and afaik you can't do that easily with common firewalls (like netfilter).

This. If it was that easy we'd already be doing it on all our desktops and servers.

Re: Malicious software libraries found in PyPI posing as well known libraries

#184

Post by pmoriarty » Fri, Sep 15, 2017, 6:41 PM UTC

fovc wrote: post #165

raesene6 wrote: post #30

This isn't, in any way, a new problem. I did a presentation on this topic for OWASP AppSecEU 2015 ( https://www.youtube.com/watch?v=Wn190b4EJWk&list=PLpr-xdpM8w... ) and when doing the research for that I encountered cases of repo. attacks and compromise. IME the problem will continue unless the customers (e.g. companies making use of the libraries hosted) are willing to pay more for a service with higher levels of a…

If someone here wants to build a business around this, count me in for NPM (high willingness to pay) or PyPi (lower WTP). Here's an idea: make it similar to Kickstarter, where customers can commit a certain amount of funds towards a specific package. If the package doesn't "tilt" in a certain amount of time money goes back. Otherwise you vet a point release and add it to your repo. you could offer subscriptions to ke…

One thing to consider if you're going to provide a service like this:

What happens if a vulnerability nevertheless sneaks through?

The whoever did the vetting could conceivably get sued. So then they might want to take out insurance or try to protect themselves from lawsuits in some other way -- all of which is likely to make such a service even more expensive.

Re: Malicious software libraries found in PyPI posing as well known libraries

#185

Post by tclancy » Fri, Sep 15, 2017, 6:45 PM UTC

danjoc wrote: post #180

tclancy wrote: post #174

Earlier quoted context omitted.

>both the pypi devs and the python security team decided to ignore it. Without trying to downplay the seriousness, I think that's a less-than-charitable take on the bind they felt they were in. It also sounds like there's a crack team of dedicated devs sitting around waiting for something to work on.

When I read things like, https://caremad.io/posts/2013/07/packaging-signing-not-holy-... I get the impression they don't care about security at all. They seem like children plugging their ears and shouting "nah nah nah" while putting repo users at risk. They've obviously done nothing since that post was made four years ago. In contrast, Maven central requires signing. Unsurprisingly, Maven central doesn't have typosq…

>I get the impression they don't care about security at all. They seem like children

I had the absolute pleasure of working with dstufft for 6 or 7 months years ago and learned a ton from his breadth of knowledge and willingness to teach (and I had more than a decade of experience at the time). I can appreciate being unhappy with the situation and not agreeing with the approach, but personal attacks and assuming ill intent don't help. It is, of course, possible the developers actually do pay attention to the details and just see the issue better than you do.

I read that link at the time and I do wonder how you would address the issues he raises. The concern seems not so much not wanting to apply security but actually thinking about the underlying issues. Which is worse: an open but dangerous Pypi or one with a bunch of security theater around keys and hashes and other stuff everyone pays lip service to but never actually checks?

Re: Malicious software libraries found in PyPI posing as well known libraries

#186

Post by pmoriarty » Fri, Sep 15, 2017, 6:48 PM UTC

jastr wrote: post #146

rantanplan wrote: post #3

The regex they have for identifying fake/harmful packages is wrong. `pip list –format=legacy | egrep '^(acqusition|apidev-coop|bzip|crypt|django-server|pwd|setup-tools|telnet|urlib3|urllib) '` This incorrectly lists `urllib3` or the `cryptography` package for example, which are perfectly valid packages. [UPDATE] Read "tobltobs" comment below. I incorrectly removed a trailing space from the regex.

pip list --format=legacy | cut -d' ' -f1 | xargs egrep '^(acqusition|apidev-coop|bzip|crypt|django-server|pwd|setup-tools|telnet|urlib3|urllib)$'

When running that command, I get output like this:

  grep: alabaster: No such file or directory
  grep: appdirs: No such file or directory
  grep: arandr: No such file or directory

for dozens and dozens of packages. Are those errors benign?

Re: Malicious software libraries found in PyPI posing as well known libraries

#187

Post by cookiecaper » Fri, Sep 15, 2017, 6:52 PM UTC

danjoc wrote: post #180

tclancy wrote: post #174

Earlier quoted context omitted.

>both the pypi devs and the python security team decided to ignore it. Without trying to downplay the seriousness, I think that's a less-than-charitable take on the bind they felt they were in. It also sounds like there's a crack team of dedicated devs sitting around waiting for something to work on.

When I read things like, https://caremad.io/posts/2013/07/packaging-signing-not-holy-... I get the impression they don't care about security at all. They seem like children plugging their ears and shouting "nah nah nah" while putting repo users at risk. They've obviously done nothing since that post was made four years ago. In contrast, Maven central requires signing. Unsurprisingly, Maven central doesn't have typosq…

Typosquatting and package signatures are separate issues. Package signing only prevents typosquatting insofar as either the user or some intermediate layer resolves the typo to the intended package. If someone was going to this effort, they'd probably go to the effort of double-checking the package name before installation anyway.

PyPi needs moderators to sit in the middle and remove anything that is obviously malicious, whether the packages are signed or not. Bad guys can sign packages just as easily as good guys.

Software should also be used to correct likely typos, perhaps including checking against a blacklist of known-bad package hashes, before the package is installed.

Yes, these approaches are imperfect, but they are better than doing nothing. "Perfect is the enemy of good".

Re: Malicious software libraries found in PyPI posing as well known libraries

#188

Post by cookiecaper » Fri, Sep 15, 2017, 6:59 PM UTC

disconnected wrote: post #61

kasabali wrote: post #53

Yet another attack vector that doesn't exist at all in Linux distributions but invented by language package managers, sadly. They solved the issue 2 decades ago by heavily vetting packages before accepting them into repositories. Users are allowed to add and use packages from 3rd party repositories. Maybe solution to this is creating curated repositories based on publicly open ones and using them by default (and requ…

> Yet another attack vector that doesn't exist at all in Linux distributions but invented by language package managers, sadly. https://www.schneier.com/blog/archives/2008/05/random_number... A.K.A., the Debian openssl Fiasco. Just one example of distros fucking up the packages from upstream and causing major havoc.

This is a totally different issue. Distro package management isn't perfect, but you don't have to worry about a random malicious individual squatting on "opensssl" and including compromised code. That's a whole different ball game than a bug inadvertently introduced during a backport.

Re: Malicious software libraries found in PyPI posing as well known libraries

#189

Post by cookiecaper » Fri, Sep 15, 2017, 7:01 PM UTC

pmoriarty wrote: post #184

fovc wrote: post #165

Earlier quoted context omitted.

If someone here wants to build a business around this, count me in for NPM (high willingness to pay) or PyPi (lower WTP). Here's an idea: make it similar to Kickstarter, where customers can commit a certain amount of funds towards a specific package. If the package doesn't "tilt" in a certain amount of time money goes back. Otherwise you vet a point release and add it to your repo. you could offer subscriptions to ke…

One thing to consider if you're going to provide a service like this: What happens if a vulnerability nevertheless sneaks through? The whoever did the vetting could conceivably get sued. So then they might want to take out insurance or try to protect themselves from lawsuits in some other way -- all of which is likely to make such a service even more expensive.

It has to be constrained to something reasonable. You can't guarantee the software is safe, but you can guarantee it is published by someone who is who they say they are, similar to EV certificates for domains. You can also refuse to publish packages with intentionally-confusing names.

Re: Malicious software libraries found in PyPI posing as well known libraries

#190

Post by [deleted] » Fri, Sep 15, 2017, 7:03 PM UTC

chupapuma wrote: post #152

edraferi wrote: post #123

Earlier quoted context omitted.

Man. You're right - that's a mess.

And just to be super clear, this is the code. Hard from scary or obfuscated: html = urllib_request.urlopen( "https://www.pytosquatting.org/pingback/pypi/{}/".format(package_name) ) raise Exception( "This is a bogus package that should not be installed\n\n" "Please read https://www.pytosquatting.org" )

[deleted]