Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Malicious software libraries found in PyPI posing as well known libraries

nbu.gov.sk

Post reply on HN
Print view View original

211–220 of 254 posts

Re: Malicious software libraries found in PyPI posing as well known libraries

#211

Post by teilo »

zokier wrote: post #164
teilo wrote: post #156

I think we need a system to prevent this instead of the wild-west that PyPi has become. For example: Developer signatures that are checked against a community rating. If someone does `pip install` pip would look up the developer signature of the package and check a community rating that would verify this is a developer who has offered legit packages in the past. It's not foolproof, but it would go a long way towards…

You could add the ability for well known members to vet newbie developers, maybe by signing their key. And now you have re-invented web of trust.

I was thinking more than the ratings would handle this, rather than having to sign keys. Ratings by well-respected and vetted members would have more weight.

Re: Malicious software libraries found in PyPI posing as well known libraries

#212

Post by teilo »

wongarsu wrote: post #170
teilo wrote: post #156

I think we need a system to prevent this instead of the wild-west that PyPi has become. For example: Developer signatures that are checked against a community rating. If someone does `pip install` pip would look up the developer signature of the package and check a community rating that would verify this is a developer who has offered legit packages in the past. It's not foolproof, but it would go a long way towards…

That sounds easy to defeat. Make some mundane, but legit packages (maybe on of those "$X but without the pointless complexity"-packages), gain trust, once trust is reached start uploading typo-squatting packages. Knowing today's internet, programmers from cheap-labour nations (India & Co.) would soon start offering "trusted PyPi accounts" for sale on hacker forums.

Yes, but keys can also be revoked, providing a way to mitigate this.

Re: Malicious software libraries found in PyPI posing as well known libraries

#213

Post by takluyver »

danjoc wrote: post #203
geofft wrote: post #201

Earlier quoted context omitted.

> You're not dealing with projects. You're dealing with keys. It's not one key per project. It's one key per contributor. My rough guess is that for the Python community, these are roughly proportional; there are a lot of different people maintaining approximately one library each, not a small number of people (or companies) maintaining large parts of the ecosystem. There's nothing directly like org.apache for Python…

And yet in every typo squatting case, there's multiple projects leading back to a single contributor. It's almost like noticing a little known nobody sneaking to the front of the Pareto distribution would be a huge red flag.

That sounds like you're relying on blacklisting the 'bad guy' key that's uploading all the malicious packages. Any half-way competent bad guy will generate a new key for each package (or try to steal keys with some good reputation), so it won't work.

Re: Malicious software libraries found in PyPI posing as well known libraries

#214

Post by danjoc »

donaldstufft wrote: post #208
danjoc wrote: post #200

Earlier quoted context omitted.

>For example, there may be `urllib3` which is a valid project that must be signed by key X. Someone then comes and registers another project, `urlib3` which must be signed by key Y. Key X is on the company approved key list, key y is not. Your argument just fell apart. >The problem with manual review then becomes one scale. There are as of this time of writing 117,226 unique projects on PyPI with an average growth of…

> Key X is on the company approved key list, key y is not. Your argument just fell apart. A minuscule amount of people are going to bother to do something like approve keys. Security for the minority can already be achieved by those companies mandating their developers use DevPI and mirroring trusted projects from PyPI to DevPI (or similar system). Complicating the system further for something that, for practical pur…

>Complicating the system further for something that, for practical purposes, does not improve the security of the vast bulk of people is not a trade off we're willing to make.

This is the weakest argument. Are Python devs somehow dumber than Java devs? Are they dumber than Android devs? Are they dumber than iOS devs? Everyone knows how to sign a dependency/app/project except python devs? I don't believe that. I honestly think that's the most insulting aspect of this argument.

The rest of this post seems to have turned to hand waving and personal attacks, so I won't bother responding to that. I'm just glad I got to share this perspective with you. Once you cool down, I hope you look harder at the problem. All I care about is improved security. I'm not here for the imaginary internet points.

Re: Malicious software libraries found in PyPI posing as well known libraries

#215

Post by takluyver »

eeZah7Ux wrote: post #107
IshKebab wrote: post #100

Earlier quoted context omitted.

The Linux distribution approach to package management ("we'll package everything ourselves!") simply doesn't scale.

That's a feature. You want a set of vetted and curated packages you can trust, and you want to receive security updates on them.

But developers also want a way to get software without getting it blessed by Debian and waiting months/years for a distro release. That's why repositories like PyPI exist and are in very widespread use.

Distro repositories are a great example of 'secure for ideal users'. They give you security if you can put up with a small selection of software and older versions. In practice, we end up working around distro repositories by installing stuff with pip, or PPAs, or downloaded from websites.

Re: Malicious software libraries found in PyPI posing as well known libraries

#216

Post by takluyver »

geekamongus wrote: post #158

Why is there no indication of any of this on the python.org website or any of their social media accounts? I checked: https://pypi.python.org/pypi https://www.python.org/blogs/ http://planetpython.org/ https://pypi.python.org/security https://twitter.com/pythoninsider https://plus.google.com/+Python https://www.facebook.com/pythonlang?fref=ts https://twitter.com/ThePSF

I guess that's because it's not a surprise. This has come up before, and it's basically unavoidable with the way PyPI is designed to work: if you see an unclaimed name, you can put whatever you want there.

Re: Malicious software libraries found in PyPI posing as well known libraries

#217

Post by mschuster91 »

IgorPartola wrote: post #102
mschuster91 wrote: post #93

Earlier quoted context omitted.

> npm alone has over 500,000 packages ( http://www.modulecounts.com/ ) so even a one time review isn't going to happen. But at least the modules with the most downloads (webpack, react, or stuff like left-pad) could be vetted, and especially npm could implement a 2-or-more person model - basically, everyone with publish access can upload a new artifact, but to actually have it distributed to endusers, a second person…

That's the thing. I worry less about popular packages. I can check that Django's GitHub repo links to PyPI and vice versa. But a random package to parse DSN's? I don't know it from Adam. I want to use it, and lots of others do too, but not everyone is going to review it. Maybe just a button on the package that says "I found insecure code!" Would be good.

> That's the thing. I worry less about popular packages. I can check that Django's GitHub repo links to PyPI and vice versa.

I worry about the most popular, and there the small and next-to-unmaintained. Just think back to the left-pad desaster that broke builds all over the world and imagine it was not a deleted package but an update containing malware. I assume there are lots of such "hidden gems" where the maintainer has gone away... the consequences of hacking just one improperly secured account are severe.

Re: Malicious software libraries found in PyPI posing as well known libraries

#218

Post by donaldstufft »

danjoc wrote: post #214
donaldstufft wrote: post #208

Earlier quoted context omitted.

> Key X is on the company approved key list, key y is not. Your argument just fell apart. A minuscule amount of people are going to bother to do something like approve keys. Security for the minority can already be achieved by those companies mandating their developers use DevPI and mirroring trusted projects from PyPI to DevPI (or similar system). Complicating the system further for something that, for practical pur…

>Complicating the system further for something that, for practical purposes, does not improve the security of the vast bulk of people is not a trade off we're willing to make. This is the weakest argument. Are Python devs somehow dumber than Java devs? Are they dumber than Android devs? Are they dumber than iOS devs? Everyone knows how to sign a dependency/app/project except python devs? I don't believe that. I hones…

> This is the weakest argument. Are Python devs somehow dumber than Java devs? Are they dumber than Android devs? Are they dumber than iOS devs? Everyone knows how to sign a dependency/app/project except python devs? I don't believe that. I honestly think that's the most insulting aspect of this argument.

Nope, I think they're perfectly capable of signing things. I also think it's silly to ask them to do that when the proposed system hasn't been designed to provide any benefit. Properly designing that system is hard, and 99% of people who go "just use PGP!" or "just use X" have spent exactly zero amount of time doing that. Particularly when the proposed solution doesn't actually solve the problem at hand (though it does solve other problems if it's correctly designed).

Ultimately your "suggestions" are nothing new, they're the same generic, cargo culting, suggestions that folks who haven't looked really hard at the problem tend to make.

Re: Malicious software libraries found in PyPI posing as well known libraries

#219

Post by takluyver »

elcapitan wrote: post #99

Would it be possible to have a general package manager (like apt) as reusable base for the individual language specific package managers? I know that npm and pip and gem etc all do some additional stuff, but at the core they all do the same (pull packages from repo, do some postinstall, resolve dependencies, maybe in some cases even check if the package is legit). So we could implement and check that once and then ju…

The packaging technology is not the issue here, it's about the repository. If you made an apt repo where anyone could claim an unused name and start uploading packages, you'd have exactly the same issue.

Re: Malicious software libraries found in PyPI posing as well known libraries

#220

Post by takluyver »

danjoc wrote: post #210
geofft wrote: post #207

Earlier quoted context omitted.

By "every typo squatting case" do you just mean researchers demonstrating the viability of the attack against various systems? A system that successfully defends against researchers but not against actual genuine attackers would be worse than useless. If I actually wanted to pull off an attack without anyone noticing for as long as possible, I'd just target a single package whose maintainer is on vacation. I think th…

Your arguments and exaggerated claims are silly. I can point to Maven Central all day long. They're doing it right. They don't have these problems. You know who does this sort of thing? Politicians. They can't just look at a working system, single payer for instance, and copy it. No, they have to make silly arguments about why it will never work, despite a concrete, working example, right in front of their own eyes.

Donald already pointed out that the key difference in Maven Central is a manual review process, not package signing.

If Python introduced manual review of new packages, it would either need a massive amount of resources that no-one is offering to provide, or it would immediately be a huge bottleneck on people making new packages, which the community doesn't want to do.

Post reply on HN