Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Malicious software libraries found in PyPI posing as well known libraries

nbu.gov.sk

Post reply on HN
Print view View original

221–230 of 254 posts

Re: Malicious software libraries found in PyPI posing as well known libraries

#221

Post by kasabali »

zzzcpan wrote: post #65
Sir_Cmpwn wrote: post #29

I think a more Linux-like approach to package repos is better - a curated package repository run by volunteers in maintainership roles. Then you have a human being verifying the upstream and keeping malware out, and get more consistency across packages as a bonus. If you want your package added it's as simple as sending an email and provides a new avenue for people to contribute to the success of the ecosystem as pac…

I don't think there are any maintainers that verify upstream code, they only manage packages and updates. Which is actually safer to do without maintainers, completely automatically, as it will eliminate a huge attack surface introduced by a maintainer.

> I don't think there are any maintainers that verify upstream code, they only manage packages and updates.

Even that minimal amount of work is enough to prevent such a ridiculous attack like typo squatting.

Re: Malicious software libraries found in PyPI posing as well known libraries

#222

Post by xapata »

cookiecaper wrote: post #187
danjoc wrote: post #180

Earlier quoted context omitted.

When I read things like, https://caremad.io/posts/2013/07/packaging-signing-not-holy-... I get the impression they don't care about security at all. They seem like children plugging their ears and shouting "nah nah nah" while putting repo users at risk. They've obviously done nothing since that post was made four years ago. In contrast, Maven central requires signing. Unsurprisingly, Maven central doesn't have typosq…

Typosquatting and package signatures are separate issues. Package signing only prevents typosquatting insofar as either the user or some intermediate layer resolves the typo to the intended package. If someone was going to this effort, they'd probably go to the effort of double-checking the package name before installation anyway. PyPi needs moderators to sit in the middle and remove anything that is obviously malici…

> needs moderators

Like Anaconda and Enthought? And countless internal departments? Or are you suggesting folks donate to the PSF and they hire a team?

Re: Malicious software libraries found in PyPI posing as well known libraries

#223

Post by jefurii »

tekromancr wrote: post #159
ams6110 wrote: post #148

Earlier quoted context omitted.

Doubly so since when installed on a server very often it will be done as "sudo pip install ....." (/s/pip/other-package-manager/ as needed)

I almost never see this. Even on systems that are only running a single python project, I only ever see folks use virtualenv. The only time I ever see things installed with sudo is when the package is being installed in a docker container.

> virtualenv

Running "pip install" as a user that has access only to the virtualenv directory is sounding like a good strategy.

Re: Malicious software libraries found in PyPI posing as well known libraries

#224

Post by xapata »

cdnsteve wrote: post #66
raesene6 wrote: post #48

Earlier quoted context omitted.

Interesting if you think that npm/Rubygems/PyPI are leaving a load of money on the table, why do you think they haven't introduced those services so far...

Because their mission isn't to generate income like a traditional business. But if the income went back to the foundations, like Python Foundation, I think that would make sense.

Anaconda gives a healthy amount to open source, either by donations to foundations like NumFOCUS or paying salaries of contributors. Is that what you're looking for?

Re: Malicious software libraries found in PyPI posing as well known libraries

#225

Post by drdaeman »

nyrikki wrote: post #154
drdaeman wrote: post #46

Earlier quoted context omitted.

Maybe you're right, but I see one possible downside that is quite important. I have encountered the case "the package has an important bugfix but is not yet published on PyPI" way more than once or twice. With the intermediate maintainers, that's going to get worse. I believe namespaces and signatures are the way to go. With a special privileged namespace for the curated widely known packages (e.g. SciPy or Django) -…

PyPI should also run a build to audit behavior which would be fairly easy to implement. A submitted package would just fail if it access the network or privileged files during compile unless unique needs are called out in an spec file. I do wish that `--user` was the default for pip. It is also a pity that trivial Debian bugs like this block adoption of non sudo pip installs weren't ignored. https://bugs.debian.org/c…

1. This would require PyPI to provide computing resources to build the packages (for all OSes, if the package contains native code). And then, malicious packages would just detect the build environment and avoid exhibiting the unwanted behavior. I think I read that it's not unusual for malware to detect being ran in a VM and doesn't do anything suspicious to complicate detection and analysis.

2. I don't think `pip install --user` adds any significant security. A little bit - sure, but not much. A trivial injection into ~/.*shrc or ~/.profile (I don't think anyone would notice the file was changed until it's too late) would result in full system compromise on the next login and sudo invocation. Same goes if you have ~/bin or ~/.local/bin (or anything user-writeable) in $PATH.

And even with non-root access, malicious software can do a lot of undesirable things (e.g. send spam or steal user data).

---

I believe, signature-based trust (with mandatory code signing) is the way to go. On the first `pip install` ever ask "The package not-expect (1.2.3) is signed by The Spanish Inquisition (key: ...) and was audited by The Python Developers (key: ...). Have you verified the keys and do you trust a) this vendor, b) this auditor or c) both?", then it gets recorded in ~/.pip (for this machine) and ./requirements.txt or ./setup.{py,cfg} (for distribution) so future installations don't ask anything.

To get non-interactive mode (for CI or something), on must either pass something like --insecure-skip-signature-checks (so they mean it) or pre-supply all the trusted keys.

(Not ideal, of course - just a quick idea. Surely, it has a lot of rough edges to polish.)

Re: Malicious software libraries found in PyPI posing as well known libraries

#226

Post by haypo »

geekamongus wrote: post #158

Why is there no indication of any of this on the python.org website or any of their social media accounts? I checked: https://pypi.python.org/pypi https://www.python.org/blogs/ http://planetpython.org/ https://pypi.python.org/security https://twitter.com/pythoninsider https://plus.google.com/+Python https://www.facebook.com/pythonlang?fref=ts https://twitter.com/ThePSF

While there is no public announcement from the PSF yet, I sent an email to the python-dev mailing list at least to announce the issue but also try to discuss how to mitigate/prevent it.

https://mail.python.org/pipermail/python-dev/2017-September/...

Honestly, I am impressed that the information gone so quick! The National Security Authority of Slovakia contacted the PSRT 10 days ago. All packages were removed 1h10 after we got their email. We were discussing how to communicate about this issue, while they published an advisory. A few hours after the advisory was published, I saw the information on IRC, Twitter, LWN, etc. I didn't expect that the advisory would be published so quickly. FYI last week there was also a CPython sprint attended by more than 20 Python core developers. We were busy on discussing Python enhancements.

Re: Malicious software libraries found in PyPI posing as well known libraries

#227

Post by danjoc »

takluyver wrote: post #220
danjoc wrote: post #210

Earlier quoted context omitted.

Your arguments and exaggerated claims are silly. I can point to Maven Central all day long. They're doing it right. They don't have these problems. You know who does this sort of thing? Politicians. They can't just look at a working system, single payer for instance, and copy it. No, they have to make silly arguments about why it will never work, despite a concrete, working example, right in front of their own eyes.

Donald already pointed out that the key difference in Maven Central is a manual review process, not package signing. If Python introduced manual review of new packages, it would either need a massive amount of resources that no-one is offering to provide, or it would immediately be a huge bottleneck on people making new packages, which the community doesn't want to do.

>the key difference in Maven Central is a manual review process

Lipstick on the pig, still covered in mud.

The key difference is the regular occurrence of malware finding its way into PyPi and NPM due to the lack of multilayered security on those repos.

You guys keep trying to prop up the straw man that ONLY package signing is needed. It's not. It's a start. Nobody is making that argument but you. You not only repeatedly beat that dead horse, but you carry it to the illogical extreme that package signing is somehow harmful. Not only do you see no value in that layer of security, but you actively resist any talk or attempts at implementing it.

Meanwhile, your repo is infested with hackers and malware. Big surprise.

Re: Malicious software libraries found in PyPI posing as well known libraries

#228

Post by zzzeek »

danjoc wrote: post #206
zzzeek wrote: post #204

Earlier quoted context omitted.

> You're not dealing with projects. You're dealing with keys. It's not one key per project. It's one key per contributor. This has the added bonus that if a contributor goes rogue, you can revoke the one key and all the suspect projects are are invalidated at once. you cannot locate said rogue contributor without regularly manually reviewing 117,226 packages.

>you cannot locate said rogue contributor without regularly manually reviewing 117,226 packages. Herd immunity. Someone is out there reviewing it. Most users won't need to lift a finger beyond verifying signatures.

If 100k packages are already security audited by the community.. Then what's the issue? They send them to dstufft, he takes them down (which of course does not actually happen because nobody is auditing most packages). As mentioned elsewhere, most contributors to pypi have only one package so the notion of "find one rogue package == dozens of untrustworthy packages removed in one swoop" doesn't really exist (esp because a rogue agent would be making one account / key per package just to avoid this kind of detection!)

Re: Malicious software libraries found in PyPI posing as well known libraries

#229

Post by danjoc »

zzzeek wrote: post #192
danjoc wrote: post #180

Earlier quoted context omitted.

When I read things like, https://caremad.io/posts/2013/07/packaging-signing-not-holy-... I get the impression they don't care about security at all. They seem like children plugging their ears and shouting "nah nah nah" while putting repo users at risk. They've obviously done nothing since that post was made four years ago. In contrast, Maven central requires signing. Unsurprisingly, Maven central doesn't have typosq…

> In contrast, Maven central requires signing. forgive my ignorance and my lack of 56 minutes to watch the entire youtube video, but who are the identities behind these signatures? The blog post you reference discusses the problem both of users signing their own packages (anyone can make a signature and any malicious package author can point people at a maliciously-owned signature as well) as well as having a central…

>who are the identities behind these signatures?

https://maven.apache.org/guides/mini/guide-central-repositor...

"we require you to provide PGP signatures for all your artifacts (all files except checksums), and distribute your public key to a key server like http://pgp.mit.edu."

>anyone can make a signature

The article flip flops on this.

any hacker can do it

it's too much burden for developers

>any malicious package author can point people at a maliciously-owned signature as well

Anyone can also verify ownership of the key before accepting packages signed by it. This is something professionals do. This is something institutions do. This is something three letter agencies do.

>Does such a tool exist

There are CVEs for python. One could even scan a repository using those. Java has prebuilt tools for this. OWASP has the dependency-check plugin for Maven. Nexus uses the same information in their repository health checks.

>have you proposed it as part of pypi's infrastructure? I am sure they'd be interested in that.

Why would I? Given their response to signed packages, I would expect a response along the lines of "Too much burden. Too hard. Not perfect. Not worth it. Security theater. Go away. Ur dumb."

>I'm not sure how the license file of a product impacts the issue of it being malware or not.

It's one of those nice features of good repository management. Do python packages even list licenses? I mean, I assume they would, but then, they actively resist implementing other basic things which I would just assume they could do.

Licenses change over time. Some enterprises treat GPL like a virus. Knowing ReactJS changes from Apache to BSD + Patents in a new version is as important to someone in the business as knowing if a package is compromised.

>it appears to be a closed-source, commercial product

Nexus OSS is open source, Nexus Professional is commercially licensed. The later has a few nice features the former does not. Both can manage PyPi, NPM, Ruby, Docker, Maven, and Nuget repos to name a few.

https://www.sonatype.com/nexus-repository-oss

>I would encourage you to write a comprehensive rebuttal to the blog post you refer towards

It's easier to fool people than to convince them that they have been fooled. -- Mark Twain

Re: Malicious software libraries found in PyPI posing as well known libraries

#230

Post by nyrikki »

drdaeman wrote: post #225
nyrikki wrote: post #154

Earlier quoted context omitted.

PyPI should also run a build to audit behavior which would be fairly easy to implement. A submitted package would just fail if it access the network or privileged files during compile unless unique needs are called out in an spec file. I do wish that `--user` was the default for pip. It is also a pity that trivial Debian bugs like this block adoption of non sudo pip installs weren't ignored. https://bugs.debian.org/c…

1. This would require PyPI to provide computing resources to build the packages (for all OSes, if the package contains native code). And then, malicious packages would just detect the build environment and avoid exhibiting the unwanted behavior. I think I read that it's not unusual for malware to detect being ran in a VM and doesn't do anything suspicious to complicate detection and analysis. 2. I don't think `pip in…

On point 1) yes so they would have to compile, that is not a huge barrier and it would also result in improved quality of modules just as CI/CD does.

On 2) even if you ignore the much larger attack surface due to running ALL installs as the root user, consider the one-shot opportunities by disabling the protections of capabilities(7) # capsh --print -- -c 'pip list > /dev/null' Current: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read+ep

And if --user was the norm, it would be trivial to write an apparmor/selinux policy to protect files like ~/.*shrc or ~/.profile

It is just the basic principle of least privilege. Heck if irssi can bother with an apparmor profile, the maintainers of pip or the package should be able to.

Post reply on HN