Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Malicious software libraries found in PyPI posing as well known libraries

nbu.gov.sk

Post reply on HN
Print view View original

241–250 of 254 posts

Re: Malicious software libraries found in PyPI posing as well known libraries

#241

Post by eeZah7Ux »

Aissen wrote: post #121
eeZah7Ux wrote: post #116

Earlier quoted context omitted.

...which is even less secure.

May I ask why ? If anything, it's more secure, since you know exactly who's publishing what. Yes, it might put a higher burden on the publisher if they don't host on github/gitlab, etc. But it strips the "magic" part and makes sure the dev knows where the code is coming from.

With packages identified by full URLs it's more likely to make a typo, or a misremember a part of the URL, or search for it on a search engine and pick a fork instead of the right one, or paste one from stack overflow or other forums that is plain false or even look legitimate due to unicode tricks. Also DNS MiTM/hijack can be used to inject a backdoor. Or the expiration of a legitimate domain.

Re: Malicious software libraries found in PyPI posing as well known libraries

#242

Post by ex_amazon_sde »

larrykwg wrote: post #237
takluyver wrote: post #215

Earlier quoted context omitted.

But developers also want a way to get software without getting it blessed by Debian and waiting months/years for a distro release. That's why repositories like PyPI exist and are in very widespread use. Distro repositories are a great example of 'secure for ideal users'. They give you security if you can put up with a small selection of software and older versions. In practice, we end up working around distro reposit…

I don't understand why they even try, Debian stable seem to have an almost arbitrary selection of outdated ruby and python libraries, at this point that hardly seems worth the effort. Sure I get the idea, but it obviously doesn't work in practice. Their security methodology also seems heavily flawed, to backport security fixes to older versions is neither scalable nor particularly reliable. I sincerely doubt that Deb…

> Debian stable seem to have an almost arbitrary selection of outdated ruby and python libraries

Yet Amazon and other big tech companies have a very similar process of packaging open source software for internal use and relying on "outdated" libraries.

> I sincerely doubt that Debian can provide adequate security to its almost 50,000 packages.

There's a security tracker where you can see how quickly packages are assigned CVEs and patched - sometimes even before the upstream patch is ready.

Re: Malicious software libraries found in PyPI posing as well known libraries

#243

Post by krapp »

eeZah7Ux wrote: post #241
Aissen wrote: post #121

Earlier quoted context omitted.

May I ask why ? If anything, it's more secure, since you know exactly who's publishing what. Yes, it might put a higher burden on the publisher if they don't host on github/gitlab, etc. But it strips the "magic" part and makes sure the dev knows where the code is coming from.

With packages identified by full URLs it's more likely to make a typo, or a misremember a part of the URL, or search for it on a search engine and pick a fork instead of the right one, or paste one from stack overflow or other forums that is plain false or even look legitimate due to unicode tricks. Also DNS MiTM/hijack can be used to inject a backdoor. Or the expiration of a legitimate domain.

You can also make a typo when all you need is a package name - as long as human beings have to type thigns out, that's going to be a problem. On the other hand, with a URL, you can actually inspect the code directly and (if it's hosted on Github or somewhere similar) see whether it's starred, forked or has any issues. It's not a case of URLs being less secure, it's just a tradeoff that pushes some of the security work to the community itself, rather than a dedicated staff of curators.

And since most package managers eventually resolve packages to a URL somewhere, the issues you mention are probably present in other package managers, albeit hidden behind abstractions.

Re: Malicious software libraries found in PyPI posing as well known libraries

#244

Post by xapata »

danjoc wrote: post #227
takluyver wrote: post #220

Earlier quoted context omitted.

Donald already pointed out that the key difference in Maven Central is a manual review process, not package signing. If Python introduced manual review of new packages, it would either need a massive amount of resources that no-one is offering to provide, or it would immediately be a huge bottleneck on people making new packages, which the community doesn't want to do.

>the key difference in Maven Central is a manual review process Lipstick on the pig, still covered in mud. The key difference is the regular occurrence of malware finding its way into PyPi and NPM due to the lack of multilayered security on those repos. You guys keep trying to prop up the straw man that ONLY package signing is needed. It's not. It's a start. Nobody is making that argument but you. You not only repeat…

Hyperbole and insults. Now you're just trolling. If I see a cockroach in the kitchen, I kill it and spray. I don't rip out the walls or move house.

Re: Malicious software libraries found in PyPI posing as well known libraries

#245

Post by eeZah7Ux »

krapp wrote: post #243
eeZah7Ux wrote: post #241

Earlier quoted context omitted.

With packages identified by full URLs it's more likely to make a typo, or a misremember a part of the URL, or search for it on a search engine and pick a fork instead of the right one, or paste one from stack overflow or other forums that is plain false or even look legitimate due to unicode tricks. Also DNS MiTM/hijack can be used to inject a backdoor. Or the expiration of a legitimate domain.

You can also make a typo when all you need is a package name - as long as human beings have to type thigns out, that's going to be a problem. On the other hand, with a URL, you can actually inspect the code directly and (if it's hosted on Github or somewhere similar) see whether it's starred, forked or has any issues. It's not a case of URLs being less secure, it's just a tradeoff that pushes some of the security wor…

> You can also make a typo when all you need is a package name

"packagename" instead of a full URL is quite a difference. And you are not addressing the other risks.

> On the other hand, with a URL, you can actually inspect the code directly

You can do that with most package managers as they show you the upstream URL.

Expecting every developer and every system engineer to verify every package and every dependency they install is not "just a tradeoff". It's simply impossible.

> since most package managers eventually resolve packages to a URL somewhere, the issues you mention are probably present in other package managers

Some check for the SSL certificate, some use package signing (e.g. APT). Also if the pypi domain expires everybody will know, unlike a random library.

Re: Malicious software libraries found in PyPI posing as well known libraries

#246

Post by geekamongus »

takluyver wrote: post #216
geekamongus wrote: post #158

Why is there no indication of any of this on the python.org website or any of their social media accounts? I checked: https://pypi.python.org/pypi https://www.python.org/blogs/ http://planetpython.org/ https://pypi.python.org/security https://twitter.com/pythoninsider https://plus.google.com/+Python https://www.facebook.com/pythonlang?fref=ts https://twitter.com/ThePSF

I guess that's because it's not a surprise. This has come up before, and it's basically unavoidable with the way PyPI is designed to work: if you see an unclaimed name, you can put whatever you want there.

I see your point, but I don't think that it needs to be a surprise to be announced or made well-known. No one is surprised that Microsoft releases loads of patches every Patch Tuesday, but they still publicize it and make it well known and easy for people to find out about.

Re: Malicious software libraries found in PyPI posing as well known libraries

#247

Post by closeparen »

hodgesrm wrote: post #151
hannob wrote: post #84

Ok, here's some ugly backstory on this: This problem has been known for a while, yet both the pypi devs and the python security team decided to ignore it. Last year someone wrote his thesis describing python typosquatting and standard library name squatting: http://incolumitas.com/2016/06/08/typosquatting-package-mana... However after that the packages used in this thesis - the most successful one being urllib2 - wer…

This is a scary attack. One partial mitigation is to use a firewall (e.g., Amazon VPC network ACLs) to restrict outbound network traffic to a small number of known addresses like well-known repos. I can't think of a good reason why code in any well-behaved application should be allowed to make random outbound network calls. I think it's also on app developers to rethink the culture of randomly grabbing packages to bu…

>well-known repos

I think it's clear at this point that you should be using internal mirrors of both programming language and OS package repos, so there is no need for build or production machines (other than those responsible for syncing the mirrors) to have outbound internet access at all.

Re: Malicious software libraries found in PyPI posing as well known libraries

#248

Post by gkya »

eeZah7Ux wrote: post #240
concede_pluto wrote: post #178

Earlier quoted context omitted.

There's nothing especially awful about left-pad being its own package, the disaster was because a huge number of developers were betting on npm to somehow be highly available (despite being donated by its admins at no cost and with no committed SLA) rather than vendoring their deps.

Vendoring thousands of tiny libs is even worse. Trusting many lesser known, tiny libs is more risky than few, big well known ones. Also, they are not vetted and there are much more opportunities for an attacker to sneak in a backdoored lib on the edge of the dependency graph. Finally, due to vendoring there's no way to receive timely drop-in security fixes for all dependencies from a trusted source.

One can both vendor and use the package manager to fetch updates. Just add the node-modules directory to your VCS.

The thing with node is that AFAIK it requires you to have libraries for what in most languages would be in the standard library. Maybe someone should start a "stdnode" project where the most popular / successful libraries for generic tasks are integrated into a dependable, maintained de-facto standard library, with an eye on quality and sanity, and community / Joyent funding.

Re: Malicious software libraries found in PyPI posing as well known libraries

#249

Post by danjoc »

donaldstufft wrote: post #209
danjoc wrote: post #206

Earlier quoted context omitted.

>you cannot locate said rogue contributor without regularly manually reviewing 117,226 packages. Herd immunity. Someone is out there reviewing it. Most users won't need to lift a finger beyond verifying signatures.

> Herd immunity. Someone is out there reviewing it. More likely everyone assumes someone else is reviewing it, and nobody actually does.

We definitely do where I work. I'm really, really surprised to hear they do not where you work. A multibillion dollar company like Amazon, who runs half the internet with AWS, does not verify dependencies? Wow. That's breathtaking.

Re: Malicious software libraries found in PyPI posing as well known libraries

#250

Post by krapp »

eeZah7Ux wrote: post #245
krapp wrote: post #243

Earlier quoted context omitted.

You can also make a typo when all you need is a package name - as long as human beings have to type thigns out, that's going to be a problem. On the other hand, with a URL, you can actually inspect the code directly and (if it's hosted on Github or somewhere similar) see whether it's starred, forked or has any issues. It's not a case of URLs being less secure, it's just a tradeoff that pushes some of the security wor…

> You can also make a typo when all you need is a package name "packagename" instead of a full URL is quite a difference. And you are not addressing the other risks. > On the other hand, with a URL, you can actually inspect the code directly You can do that with most package managers as they show you the upstream URL. Expecting every developer and every system engineer to verify every package and every dependency the…

>"packagename" instead of a full URL is quite a difference. And you are not addressing the other risks.

It is more characters, and therefore easier to misspell, but a URL also gives you a domain and probably a namespace for the developer, each of which can act as indicators of trustworthiness and help disambiguate packages with the same or similar names.

If you can't double check your spelling for a package name or you just pick the first Google result, or paste from SO, then you deserve what you get. Domain hijacking, MITM, Unicode shenanigans and such are real risks, but not of URLs as package identifiers per se, so much as risks of distributing packages over the internet, which most if not all do anyway.

>You can do that with most package managers as they show you the upstream URL.

But if you don't have to deal with the URL, chances are you won't, and it's less likely you'll bother to follow it. I'm arguing that, if URLs are dangerous because of their length, then package names alone are dangerous because of their abstraction. I know that I can probably trust including "https://github.com/symfony/symfony" but "symfony" or even "symfony/symfony" alone tells me nothing useful.

>Expecting every developer and every system engineer to verify every package and every dependency they install is not "just a tradeoff". It's simply impossible.

True, but Linus' Law is still basically the security model that's supposed to underpin open source software, even it it's proven not to scale as well as assumed. Someone, somewhere has to know the code is safe, and that's either you or someone you trust, or (as is likely the case with most developers) someone you just assume exists.

>Some check for the SSL certificate, some use package signing (e.g. APT). Also if the pypi domain expires everybody will know, unlike a random library.

There's no reason a package manager using URLs can't also require package servers (which, let's face it, are probably going to be Github and Bitbucket in almost all cases) or maintainers to do something similar. Or at the very least put out warnings the way browsers do about invalid or untrusted certificates or unknown domains. You would lose the freedom of the "wild west" model in its purest form but still not be tied down to a single source of authority.

Post reply on HN