Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devs unknowingly use “malicious” modules snuck into official Python repository

arstechnica.com

Post reply on HN
Print view View original

21–30 of 119 posts

Re: Devs unknowingly use “malicious” modules snuck into official Python repository

#21

Post by flukus »

zizek23 wrote: post #17

The casual culture of pulling in hundred of dependencies, and mushrooming language specific package managers is ridiculously insecure and has to go. There is no way for anyone to know what all this code is doing, there is little way to verify updates and its simply untenable. If some developers like this sort of unsafe practice it should be strictly limited to their machines and in no way make it across in any form a…

On top of this, updating dependencies is becoming just as important as patching the OS itself, except without the culture of providing non-breaking updates. Often you're left with the choice of having to either upgrade to the latest version and deal with breaking changes or leave the insecure versions in place.

This compounds the dislike companies already have for updating software, updating packages isn't something that's ever planned and budgeted for. A standard enterprise app will be using tonnes of outdated and potentially insecure packages, I've come across some that are a decade out of date with no pain-free update path.

And now it seems that even "systems languages" are heading down this path.

Re: Devs unknowingly use “malicious” modules snuck into official Python repository

#22

Post by viraptor »

zizek23 wrote: post #17

The casual culture of pulling in hundred of dependencies, and mushrooming language specific package managers is ridiculously insecure and has to go. There is no way for anyone to know what all this code is doing, there is little way to verify updates and its simply untenable. If some developers like this sort of unsafe practice it should be strictly limited to their machines and in no way make it across in any form a…

You mean that every time you write a new project you write your own request parser, your own server, your own web framework, your own database access libraries, etc. from scratch?

Packages taken over include misspelling of urllib. What exactly do you propose as an alternative here?

I'm all for limiting bloat, but your rant here seems completely inappropriate for the issue bring described.

Re: Devs unknowingly use “malicious” modules snuck into official Python repository

#23

Post by zizek23 »

reificator wrote: post #20
zizek23 wrote: post #17

The casual culture of pulling in hundred of dependencies, and mushrooming language specific package managers is ridiculously insecure and has to go. There is no way for anyone to know what all this code is doing, there is little way to verify updates and its simply untenable. If some developers like this sort of unsafe practice it should be strictly limited to their machines and in no way make it across in any form a…

NPM is my primary reason for not using NodeJS. I installed a specific package in an empty project and got over 690 dependencies. For running `npm install {package-name}`. What's worse is that I skimmed the tree to check for anything particularly heinous, but there was nothing that stood out as unneeded. With such a tiny stdlib, especially out of the browser environment, there's not really a better alternative than to…

A package manager must at a minimum address code security, dependency hell and deployment. A json parser that wgets urls into a folder is not a package manager.

These things must be thought of upfront and cannot be left vague and open to adhoc practices that leak to end users and deployment creating insecurity.

For instance a package should be a package, it should not be a simple class or function that in turn pulls in a hundred dependencies. 10 packages like this can pull in 600 packages all with conflicting versions. This is exactly dependency hell that creates insecure practice as no one can verify the chain and sheer number of packages.

Language developers cannot just sit back and let this happen. Eventually you will pay the price for this kind of shoddy engineering. This should be the minimum required from any responsible language.

And best then if you don't use the system package manager, do not mix up both to create a frankenstein language that multiplies complexity for everyone.

The biggest thing is stop being faddish. Recognize upfront that there are always developers and groups jockeying for influence and creating dependencies on their apps and platforms. Discourage them from hoisting their self serving bad ideas on the ecosystem and multiplying complexity for everyone. Ultimately the language and ecosystem have to develop consensus on robust engineering practices or a wildland which will eventually lose users.

Re: Devs unknowingly use “malicious” modules snuck into official Python repository

#24

Post by flukus »

viraptor wrote: post #22
zizek23 wrote: post #17

The casual culture of pulling in hundred of dependencies, and mushrooming language specific package managers is ridiculously insecure and has to go. There is no way for anyone to know what all this code is doing, there is little way to verify updates and its simply untenable. If some developers like this sort of unsafe practice it should be strictly limited to their machines and in no way make it across in any form a…

You mean that every time you write a new project you write your own request parser, your own server, your own web framework, your own database access libraries, etc. from scratch? Packages taken over include misspelling of urllib. What exactly do you propose as an alternative here? I'm all for limiting bloat, but your rant here seems completely inappropriate for the issue bring described.

> You mean that every time you write a new project you write your own request parser, your own server, your own web framework, your own database access libraries, etc. from scratch?

They made it pretty clear that they get them from a distro repository. Most are much better about managing breaking changes and offering years of security patches (at least to core packages).

> Packages taken over include misspelling of urllib. What exactly do you propose as an alternative here?

Stricter guidelines and auditing should have caught this.

Re: Devs unknowingly use “malicious” modules snuck into official Python repository

#25

Post by zizek23 »

viraptor wrote: post #22
zizek23 wrote: post #17

The casual culture of pulling in hundred of dependencies, and mushrooming language specific package managers is ridiculously insecure and has to go. There is no way for anyone to know what all this code is doing, there is little way to verify updates and its simply untenable. If some developers like this sort of unsafe practice it should be strictly limited to their machines and in no way make it across in any form a…

You mean that every time you write a new project you write your own request parser, your own server, your own web framework, your own database access libraries, etc. from scratch? Packages taken over include misspelling of urllib. What exactly do you propose as an alternative here? I'm all for limiting bloat, but your rant here seems completely inappropriate for the issue bring described.

You are fighting a strawman. Either use the system package manager which has already thought about and solved these issues or if you must use a 'home brew' package manager, design a secure one. And if you can't do that limit the circus to developer machines.

Re: Devs unknowingly use “malicious” modules snuck into official Python repository

#26

Post by xroche »

viraptor wrote: post #22
zizek23 wrote: post #17

The casual culture of pulling in hundred of dependencies, and mushrooming language specific package managers is ridiculously insecure and has to go. There is no way for anyone to know what all this code is doing, there is little way to verify updates and its simply untenable. If some developers like this sort of unsafe practice it should be strictly limited to their machines and in no way make it across in any form a…

You mean that every time you write a new project you write your own request parser, your own server, your own web framework, your own database access libraries, etc. from scratch? Packages taken over include misspelling of urllib. What exactly do you propose as an alternative here? I'm all for limiting bloat, but your rant here seems completely inappropriate for the issue bring described.

> You mean that every time you write a new project you write your own request parser, your own server, your own web framework, your own database access libraries, etc. from scratch?

No, but downloading random code is just insane.

As a comparison, Debian has ~3000 packages, and every single package had an identified maintainer, with its own GPG key, validated in face-to-face meeting with an ID card by three people, an identified upstream, etc. Each maintainer is physically identified, and has passed a number of technical validation steps, explained his motivation etc.

There is also a dedicated security team that can be contacted 24/7.

The system is not perfect, but it provides a good level of security. And this is a project only made by volunteers.

Re: Devs unknowingly use “malicious” modules snuck into official Python repository

#27

Post by viraptor »

zizek23 wrote: post #25
viraptor wrote: post #22

Earlier quoted context omitted.

You mean that every time you write a new project you write your own request parser, your own server, your own web framework, your own database access libraries, etc. from scratch? Packages taken over include misspelling of urllib. What exactly do you propose as an alternative here? I'm all for limiting bloat, but your rant here seems completely inappropriate for the issue bring described.

You are fighting a strawman. Either use the system package manager which has already thought about and solved these issues or if you must use a 'home brew' package manager, design a secure one. And if you can't do that limit the circus to developer machines.

I've not seen a non-trivial project which doesn't require at least one package outside of the repo, or an updated version. And then you either have to run your own repo and packaging process. (Been there, done that, not fun) Or mix sources. (noooooooope) Or just switch to pypi completely.

Even if you can work with that, you need to sort out differences between your destination system, test system, and all of your developers machines (who most likely run Mac, not your server Linux flavour). Sure, there's still docker, vagrant, etc. But this means slapping more and more layers just to make the same packages from the same source available.

And unless you want a "works on my machine" environment, you can't just leave developers to do whatever and use a different system in production. A home brew, limited system will just result in shadow it, which will use pypi unless your solution is much better and easier to use (and allows fresh versions to be imported)

I've been doing this stuff for years in big projects. It's not easy.

Re: Devs unknowingly use “malicious” modules snuck into official Python repository

#28

Post by grrowl »

reificator wrote: post #20
zizek23 wrote: post #17

The casual culture of pulling in hundred of dependencies, and mushrooming language specific package managers is ridiculously insecure and has to go. There is no way for anyone to know what all this code is doing, there is little way to verify updates and its simply untenable. If some developers like this sort of unsafe practice it should be strictly limited to their machines and in no way make it across in any form a…

NPM is my primary reason for not using NodeJS. I installed a specific package in an empty project and got over 690 dependencies. For running `npm install {package-name}`. What's worse is that I skimmed the tree to check for anything particularly heinous, but there was nothing that stood out as unneeded. With such a tiny stdlib, especially out of the browser environment, there's not really a better alternative than to…

You specifically chose a large dependency with many sub-dependencies, so yes, that will happen. There's also the risk of installing an outdated, unmaintained dependency.

Do a little research — check the package's npm page, assess whether it's too light or too heavy for your use-case. Check its github page to assess whether it's currently maintained (and how important that is for your use-case). If you're unsure, look at similar packages and/or peruse the source code.

It only takes a few minutes and you'll have much greater confidence because you know you picked the correct multi-byte-string-length-calculating dependency for your use case, not the naive implementation which is 100x slower (for example).

Re: Devs unknowingly use “malicious” modules snuck into official Python repository

#29

Post by mrmondo »

xroche wrote: post #26
viraptor wrote: post #22

Earlier quoted context omitted.

You mean that every time you write a new project you write your own request parser, your own server, your own web framework, your own database access libraries, etc. from scratch? Packages taken over include misspelling of urllib. What exactly do you propose as an alternative here? I'm all for limiting bloat, but your rant here seems completely inappropriate for the issue bring described.

> You mean that every time you write a new project you write your own request parser, your own server, your own web framework, your own database access libraries, etc. from scratch? No, but downloading random code is just insane. As a comparison, Debian has ~3000 packages, and every single package had an identified maintainer, with its own GPG key, validated in face-to-face meeting with an ID card by three people, an…

Debian security are very good at their job indeed, however so many of the packages are so old they’re not usable, furthermore Debian doesn’t enable SELinux by default and has a number of policies missing which significantly weakens the average deployment.

Re: Devs unknowingly use “malicious” modules snuck into official Python repository

#30

Post by cassowary »

mrmondo wrote: post #29
xroche wrote: post #26

Earlier quoted context omitted.

> You mean that every time you write a new project you write your own request parser, your own server, your own web framework, your own database access libraries, etc. from scratch? No, but downloading random code is just insane. As a comparison, Debian has ~3000 packages, and every single package had an identified maintainer, with its own GPG key, validated in face-to-face meeting with an ID card by three people, an…

Debian security are very good at their job indeed, however so many of the packages are so old they’re not usable, furthermore Debian doesn’t enable SELinux by default and has a number of policies missing which significantly weakens the average deployment.

Those comments are all irrelevant to the OP's point. Debian software is "old" by intention; it is part of the spec that it shouldn't be a moving target.

As for Debian making some policy decisions that you disagree with, it's very different for Debian to make decisions than for NPM to make decisions. There's just no comparison.

Post reply on HN