Devs unknowingly use “malicious” modules snuck into official Python repository • Hacker News Board

Re: Devs unknowingly use “malicious” modules snuck into official Python repository

#61

Post by zero_one_one » Mon, Sep 18, 2017, 9:17 AM UTC

yeukhon wrote: post #34

zizek23 wrote: post #17

The casual culture of pulling in hundred of dependencies, and mushrooming language specific package managers is ridiculously insecure and has to go. There is no way for anyone to know what all this code is doing, there is little way to verify updates and its simply untenable. If some developers like this sort of unsafe practice it should be strictly limited to their machines and in no way make it across in any form a…

Everything we do in life relies on a trust system more or less. Reproducible build only guarantees integrity, not the trustworthiness of the code. > There is no way for anyone to know what all this code is doing, there is little way to verify updates and its simply untenable. Because few people have time to read source code. Take Django as an example. Big community, lots of contributors. Can we say we should trust th…

Not only reading source code - how many people disassemble their compiled code and check for any unwanted side effects the compiler may have generated? For any non-trivial code base, this is not an option, hence where the implicit trust in the tools lies, and also the potential for exploitation

Re: Devs unknowingly use “malicious” modules snuck into official Python repository

#62

Post by zizek23 » Mon, Sep 18, 2017, 9:29 AM UTC

yeukhon wrote: post #34

zizek23 wrote: post #17

The casual culture of pulling in hundred of dependencies, and mushrooming language specific package managers is ridiculously insecure and has to go. There is no way for anyone to know what all this code is doing, there is little way to verify updates and its simply untenable. If some developers like this sort of unsafe practice it should be strictly limited to their machines and in no way make it across in any form a…

Everything we do in life relies on a trust system more or less. Reproducible build only guarantees integrity, not the trustworthiness of the code. > There is no way for anyone to know what all this code is doing, there is little way to verify updates and its simply untenable. Because few people have time to read source code. Take Django as an example. Big community, lots of contributors. Can we say we should trust th…

Only the developers know why they need 150 deps, there is no way for end users or deployment to verify this chain or stay on top of this.

They simply need to deploy the app and keep it updated safely. If developers and languages have not thought of this basic step then please defer to the distribution package manager which has.

These kind of frauds won't pass the average distribution package managers scrutiny and CVEs will have quick updates that are tested to work.

Contrast this with users scrambling to update affected apps and their individual local libraries which in turn may have their own specific deps which may not have been updated and thus will fail because of version inconsistencies. End result. Millions of man hours wasted because of clearly bad engineering practices.

Re: Devs unknowingly use “malicious” modules snuck into official Python repository

#63

Post by xorcist » Mon, Sep 18, 2017, 9:31 AM UTC

yeukhon wrote: post #34

zizek23 wrote: post #17

The casual culture of pulling in hundred of dependencies, and mushrooming language specific package managers is ridiculously insecure and has to go. There is no way for anyone to know what all this code is doing, there is little way to verify updates and its simply untenable. If some developers like this sort of unsafe practice it should be strictly limited to their machines and in no way make it across in any form a…

Everything we do in life relies on a trust system more or less. Reproducible build only guarantees integrity, not the trustworthiness of the code. > There is no way for anyone to know what all this code is doing, there is little way to verify updates and its simply untenable. Because few people have time to read source code. Take Django as an example. Big community, lots of contributors. Can we say we should trust th…

> someone apparently made an authorized changeset to BitKeeper

I suggest you read the link you mentioned. It was not an authorized change in BK, it was not an authorized change at all. It was only in a third-party CVS mirror, and it was found exactly when someone asked why there was a changeset in the CVS mirror that wasn't in BK.

And while I don't doubt that there are enough unreadable code to hide a backdoor in, certainly anywhere in the massive driver tree, this backdoor is a fairly obvious one. I don't think you could hide an assignment as a comparison through code review. It's such a common error to make that it really stands out.

> Are you going to read JRE code to make sure no backdoor?

That is a highly misleading question.

While I will trust the JRE developers without checking every single change, they are a diverse bunch enough that the fact that they are checking each other's work goes a long way.

It's very much the same situation as with the kernel and the compiler. The difference is only one of magnitude.

The difference between that and nodejs or ruby where any one can upload anything, anonymously and completely unchecked, is enormous.

Re: Devs unknowingly use “malicious” modules snuck into official Python repository

#64

Post by namelost » Mon, Sep 18, 2017, 9:55 AM UTC

zizek23 wrote: post #23

reificator wrote: post #20

Earlier quoted context omitted.

NPM is my primary reason for not using NodeJS. I installed a specific package in an empty project and got over 690 dependencies. For running `npm install {package-name}`. What's worse is that I skimmed the tree to check for anything particularly heinous, but there was nothing that stood out as unneeded. With such a tiny stdlib, especially out of the browser environment, there's not really a better alternative than to…

A package manager must at a minimum address code security, dependency hell and deployment. A json parser that wgets urls into a folder is not a package manager. These things must be thought of upfront and cannot be left vague and open to adhoc practices that leak to end users and deployment creating insecurity. For instance a package should be a package, it should not be a simple class or function that in turn pulls…

Who says? npm is proof that package managers need to do none of these things to be successful, and indeed the traditional design of a package manager as a constraint solver is self-defeating, because it puts a perverse pressure on libraries to have as few dependencies as possible and to avoid using the package manager to resolve their dependencies.

The reason npm has teething problems is because it's one of the first package managers to actually reliably address the problem of libraries depending on other libraries. Almost incredibly, nobody bothered to solve this problem before npm. i.e. npm addresses the case of library X depending on Z v2.0 and library Y depending on Z v3.0 without falling flat on its face.

Re: Devs unknowingly use “malicious” modules snuck into official Python repository

#65

Post by gjtorikian » Mon, Sep 18, 2017, 9:59 AM UTC

zizek23 wrote: post #17

The casual culture of pulling in hundred of dependencies, and mushrooming language specific package managers is ridiculously insecure and has to go. There is no way for anyone to know what all this code is doing, there is little way to verify updates and its simply untenable. If some developers like this sort of unsafe practice it should be strictly limited to their machines and in no way make it across in any form a…

> Many have suffered and do not even bother with Ruby apps anymore.

These sort of grandiose statements have of course existed on the Internet for decades, but with the rise of demagogues like Trump, we see that such statements can be readily believed en masse without a second thought.

Could you provide data or numbers explaining that Ruby's package management system is a factor in new apps not being built in that language anymore?

I don't mean to necessarily add politics in here, but hastily throwing out intimidating messages--"Ruby's package manager is broken and now no one is using it; Node and others will of course head down the same path"--helps no one. First, it harms the morale of new programmers learning these languages who may be lead to believe they are wasting their time. Second, it frankly is rude to the groups of individuals who do work on the package managers themselves; if you have a better idea, build or sponser one.

The burden of proof is on you to provide these statistics, so until then, there's also the third result: it makes you look like a crank, not a professional who has considered the pros and cons of different package management systems.

Re: Devs unknowingly use “malicious” modules snuck into official Python repository

#66

Post by jpttsn » Mon, Sep 18, 2017, 10:12 AM UTC

gjtorikian wrote: post #65

zizek23 wrote: post #17

The casual culture of pulling in hundred of dependencies, and mushrooming language specific package managers is ridiculously insecure and has to go. There is no way for anyone to know what all this code is doing, there is little way to verify updates and its simply untenable. If some developers like this sort of unsafe practice it should be strictly limited to their machines and in no way make it across in any form a…

> Many have suffered and do not even bother with Ruby apps anymore. These sort of grandiose statements have of course existed on the Internet for decades, but with the rise of demagogues like Trump, we see that such statements can be readily believed en masse without a second thought. Could you provide data or numbers explaining that Ruby's package management system is a factor in new apps not being built in that lan…

Don't you think bringing Trump into this is an example of the same things you admonish GP for?

Re: Devs unknowingly use “malicious” modules snuck into official Python repository

#67

Post by ubernostrum » Mon, Sep 18, 2017, 10:19 AM UTC

seanp2k2 wrote: post #54

onion2k wrote: post #36

Earlier quoted context omitted.

Debian has ~3000 packages, and every single package had an identified maintainer... As of yesterday npmjs had 516,132 packages, which was an increase of 373 since the day before. Debian's 3000 packages is a couple of weeks of npmjs activity. Even if you stripped out the unnecessary, abandoned, or duplicate packages you're still looking at a something significantly different to Debian. For what it's worth I think a we…

The culture around what should be in a package is vastly different. Node has left-pad. Debian has stuff like Apache. Sure, those are radical examples, but the barrier to entry for Debian is pretty high (as is the standard for quality) vs npm where anyone can put whatever out there. How do things like left-pad even come to be widespread dependencies? Does the node development process involve a lot of "gee I wonder if…

JavaScript has things like left-pad because it doesn't have a standard library of modules for common tasks like other languages do, and because some of its built-in types don't have the same rich set of operations as other languages.

Re: Devs unknowingly use “malicious” modules snuck into official Python repository

#68

Post by chii » Mon, Sep 18, 2017, 10:26 AM UTC

namelost wrote: post #64

zizek23 wrote: post #23

Earlier quoted context omitted.

A package manager must at a minimum address code security, dependency hell and deployment. A json parser that wgets urls into a folder is not a package manager. These things must be thought of upfront and cannot be left vague and open to adhoc practices that leak to end users and deployment creating insecurity. For instance a package should be a package, it should not be a simple class or function that in turn pulls…

Who says? npm is proof that package managers need to do none of these things to be successful, and indeed the traditional design of a package manager as a constraint solver is self-defeating, because it puts a perverse pressure on libraries to have as few dependencies as possible and to avoid using the package manager to resolve their dependencies . The reason npm has teething problems is because it's one of the firs…

How does npm solve the transitive dependencies problem?

Re: Devs unknowingly use “malicious” modules snuck into official Python repository

#69

Post by falsedan » Mon, Sep 18, 2017, 10:27 AM UTC

coldtea wrote: post #56

viraptor wrote: post #22

Earlier quoted context omitted.

You mean that every time you write a new project you write your own request parser, your own server, your own web framework, your own database access libraries, etc. from scratch? Packages taken over include misspelling of urllib. What exactly do you propose as an alternative here? I'm all for limiting bloat, but your rant here seems completely inappropriate for the issue bring described.

> You mean that every time you write a new project you write your own request parser, your own server, your own web framework, your own database access libraries, etc. from scratch? No, they mean that we need a better method than trusting some random popular GitHub / PyPy / npm whatever storage and delivery mechanism. Strong core language libraries ("batteries included"), that come with your distribution of the langu…

> Strong core language libraries ("batteries included"), that come with your distribution of the language/compiler and are well maintained, used by millions, and signed would be a good start to at least have 80% of dependencies being actually dependable.

The trick is avoiding another urllib: shipped as a stdlib, and passed over for libraries with better interfaces.

perl has the idea of core and dual-life modules: when a new version is released, it ships with some additional modules. Some of these modules live in the perl repository & are called 'core modules'. Upgrading them requires a new version of perl to be release. Some of these modules are forked & published to CPAN & are called 'dual-life' modules. Upgrading them requires installing them from CPAN, and new versions of perl can provide a well-tested newer version.

All the modules used to be core-only, and over the years any which weren't heavily tied to the internals of the perl interpreter (like opcode deparsers etc.) have moved to be dual-life instead. This avoids problems like interface issues in the standard library by allowing a new version to be immediately released (with some overhead for developers who need to upgrade it for deployments), be tested in the real world, updated, and then included in the next language release.

Re: Devs unknowingly use “malicious” modules snuck into official Python repository

#70

Post by 7jewndjwq » Mon, Sep 18, 2017, 10:33 AM UTC

zizek23 wrote: post #17

The casual culture of pulling in hundred of dependencies, and mushrooming language specific package managers is ridiculously insecure and has to go. There is no way for anyone to know what all this code is doing, there is little way to verify updates and its simply untenable. If some developers like this sort of unsafe practice it should be strictly limited to their machines and in no way make it across in any form a…

> Users should simply boycott such user hostile developers and languages that encourage this kind of insecurity.

We are talking about something that started as a catalog: package index.

You are an entitled loser who is too dumb or lazy to maintain a list of checksums. Please don't use Python.