Here is the solution to this Debian/Gentoo/ packager's dilemma. Packagers realize that their numbers are small and they can't keep up fixing all the modern big software (BS) projects that go against their philosophy. They define the core of the OS that they can keep managing in this traditional way (kernel, basic userland, basic desktop, basic server services, basic libraries, security support for all that). "But peo…
I think that there is a good reason to keep those projects in the main distribution: it makes them more robust and prevents the build systems and bootstrapping process from becoming a massive pile of technical debt. For example, in order to get clojure 10 running on Gentoo I had to go dig through the git history to figure out where spec-alpha and core-specs-alpha and clojure itself did not form a circular dependency…
The modern packager’s security nightmare
41–50 of 282 posts
Re: The modern packager’s security nightmare
#42Over the last several weeks I was working on an essay about this exact problem, including the connection between static linking and bundling. This one is so well done that I probably won't even publish it. But I'll add this, for people who may not immediately see why this is important. I think that the real danger these new technologies represent is not inherently bad technology, but the possibility of ecosystem dama…
I disagree, it's proven to be inadequate for modern software development and that's why these new languages/ecosystems are springing up. The least reliable way to package and distribute software is by relying on traditional package managers.
> Do what you can to give their applications the ability to easily split dependencies and support system provided libraries
This is unrealistic. I do not trust system provided libraries to function with my applications because I've been burned so many times in the past.
> how can I make this program as easy for maintainers to distribute as possible
By statically linking everything as much as possible and shipping everything else in a self contained bundle with a launcher that overrides any symbols that might inadvertently be pulled in from the system.
The universe I'd like to live in is where the only use case for dynamic linking are OS vendor APIs and cryptographically secure functions like TLS. My dream package manager would whitelist those system librarie and forbid distribution for any bundle that does contain the shared objects with the symbols it needs.
Re: The modern packager’s security nightmare
#43This will be a somewhat intemperate response, because as a developer of a significant library I found this quite irritating. If you publish a Python library without pinned dependencies, your code is broken. It happens to work today, but there will come a day when the artifact you have published no longer works. It's only a matter of time. The command the user had run before, like "pip install spacy==2.3.5" will no lo…
I can see some instances in which this expectation is important, and others where it is likely not or else certainly less important than the security implications.
For the extremes, I see research using spaCy has a very strong interest in reproducibility and the impact of any security issues would likely be minimal on the whole simply due to the relatively few people likely to run into them.
On the other extreme, say some low-level dependency is somehow so compromised simply running the code will end up with the user ransomware'd after just-long-enough that this whole scenario is marginally plausible. Then say spaCy gets incorporated into some other project that goes up the chain a ways and ultimately ends up in LibreOffice. If all of these projects have pinned dependencies, there is now no way to quickly or reasonably create a safe LibreOffice update. It would require a rather large number of people to sequentially update their dependencies, and publish the new version, so that the next project up the chain can do the same. LibreOffice would remain compromised or at best unavailable until the whole chain finished, or else somebody found a way to remove the offending dependency without breaking LibreOffice.
I'm not sure how to best reconcile these two competing interests. I think it seems clear that both are important. Even more than that, a particular library might sit on both extremes simultaneously depending on how it is used.
The only solution - though a totally unrealistic and terrible one - that comes to mind is to write all code such that all dependencies can be removed without additional work and all dependent features would be automatically disabled. With a standardized listing of these feature-dependency pairs you could even develop more fine-grained workarounds for removal of any feature from any dependency.
The sheer scale of possible configurations this would create is utterly horrifying.
At any rate, your utter rejection of the article's point seems excessively extreme and even ultimately user-hostile. I can understand your point of view, particularly given the library you develop, however I think you should probably give some more thought to indirect users - ie users of programs that (perhaps ultimately) use spaCy. I don't know that it makes sense to practically change how you do anything, but I don't think the other viewpoint is as utterly wrongheaded as you seem to think.
Re: The modern packager’s security nightmare
#44Earlier quoted context omitted.
The way iOS and Fuchsia are dealing with the problem is to completely lockdown the operating system with a tight permissions system. An app can be compromised but the damage is limited. Perhaps it is time for servers to move to a similar model.
We should call this newly invented and wholly original concept a "container". The software gets "contained". It just might work. /s
Re: The modern packager’s security nightmare
#45Here is the solution to this Debian/Gentoo/ packager's dilemma. Packagers realize that their numbers are small and they can't keep up fixing all the modern big software (BS) projects that go against their philosophy. They define the core of the OS that they can keep managing in this traditional way (kernel, basic userland, basic desktop, basic server services, basic libraries, security support for all that). "But peo…
Re: The modern packager’s security nightmare
#46This will be a somewhat intemperate response, because as a developer of a significant library I found this quite irritating. If you publish a Python library without pinned dependencies, your code is broken. It happens to work today, but there will come a day when the artifact you have published no longer works. It's only a matter of time. The command the user had run before, like "pip install spacy==2.3.5" will no lo…
Pinning dependencies in applications/binaries/end-products is clearly the right choice, but it’s much fuzzier for libraries.
Re: The modern packager’s security nightmare
#47Over the last several weeks I was working on an essay about this exact problem, including the connection between static linking and bundling. This one is so well done that I probably won't even publish it. But I'll add this, for people who may not immediately see why this is important. I think that the real danger these new technologies represent is not inherently bad technology, but the possibility of ecosystem dama…
> The distribution / maintainer / package manager approach has proven to be an extremely reliable way to get trustworthy software. Many of us love it and want to see it stick around. I disagree, it's proven to be inadequate for modern software development and that's why these new languages/ecosystems are springing up. The least reliable way to package and distribute software is by relying on traditional package manag…
Well, that's exactly why the OP (and my essay) are "anti" modern software development in many ways. The view is that we're moving away from the traditional open source ecosystem and methods of software development with these new technologies, which (to be clear) are good technologies, but were created mostly to solve problems that some large corporations have, not to solve the problems that the open source ecosystem has.
> The least reliable way to package and distribute software is by relying on traditional package managers.
Not sure what you mean by this, but it's entirely untrue in my experience. Anything I install with a package manager just works, 100% of the time. Stuff I try to get any other way is a shitshow, and the lack of "quality control" provided by maintainers speaks for itself. I mean, just look at Android apps or the Chrome extension store. Heaven forbid we go back to the days of curl | bash off someone's website.
> By statically linking everything as much as possible and shipping everything else in a self contained bundle with a launcher that overrides any symbols that might inadvertently be pulled in from the system.
I know you know this, but just to be clear, that's not a solution to the problem of "making things easier for maintainers to distribute", that's cutting maintainers out of the loop. The whole point of my focus on ecosystems is that this is something that I, as a user, don't want to happen.
Re: The modern packager’s security nightmare
#48>> Why do people pin dependencies? The primary reason is that they don’t want dependency updates to suddenly break their packages for end users, or to have their CI results suddenly broken by third-party changes. Or because we dont want accidental or malicious security vulnerabilities to get automatically incorporated into the software. This stuff works both ways. You dont automatically incorporate fixes, nor new pro…
The vast, vast majority of updates fix security issues. It's like not vaccinating in case you're one of the million people that has an allergic reaction. Supply chain attacks are rare, not the norm. We hear about such things (and only rarely at that) because it's exceptional enough to make the news.
Re: The modern packager’s security nightmare
#49Over the last several weeks I was working on an essay about this exact problem, including the connection between static linking and bundling. This one is so well done that I probably won't even publish it. But I'll add this, for people who may not immediately see why this is important. I think that the real danger these new technologies represent is not inherently bad technology, but the possibility of ecosystem dama…
> The distribution / maintainer / package manager approach has proven to be an extremely reliable way to get trustworthy software. Many of us love it and want to see it stick around. I disagree, it's proven to be inadequate for modern software development and that's why these new languages/ecosystems are springing up. The least reliable way to package and distribute software is by relying on traditional package manag…
This idea seems to be predicated on a belief that OS vendor APIs and cryptographic libraries are the only attack surface for serious user-affecting software exploits.
Re: The modern packager’s security nightmare
#50Until then, I'll keep doing it, because it saves me time and money.
I don't really buy that dynamic linking all the things is such a boon to security. But I'll link to this: https://drewdevault.com/dynlib
> Not including libc, the only libraries which had "critical" or "high" severity vulnerabilities in 2019 which affected over 100 binaries on my system were dbus, gnutls, cairo, libssh2, and curl. 265 binaries were affected by the rest.
> The total download cost to upgrade all binaries on my system which were affected by CVEs in 2019 is 3.8 GiB. This is reduced to 1.0 GiB if you eliminate glibc.
> It is also unknown if any of these vulnerabilities would have been introduced after the last build date for a given statically linked binary; if so that binary would not need to be updated. Many vulnerabilities are also limited to a specific code path or use-case, and binaries which do not invoke that code path in their dependencies will not be affected. A process to ascertain this information in the wake of a vulnerability could be automated.