Debian Janitor: 60k Lintian Issues Automatically Fixed

To be honest, this sounds more like a >/dev/null solution to me. The underlying issue is that debian packages are out of date, and that the repos (ppas) containing them are heavily unmaintained up to the point were decades old libraries are required even when upstream has moved on.

Why not go with the approach to try and fix it upstream where it belongs instead of maintaining a set of fixers downstream - which will likely not work anymore in the forseeable future?

I mean, if you start building an auto-fixer pipeline for an auto-linted error... then is that error actually valid in the first place? Probably not.

The reason why a lot of developers these days have moved to Arch's/AUR's ecosystem is not the amount of available packages (they probably are around the same) - It's integration of packaging with upstream.

Every time a package upstream is out-of-date or not working, you'll likely find an Arch user trying to fix it with a pull request in the git repos.

The debian janitor should've tried to automate pull requests rather than trying to automate self-hosted fixes downstream without letting upstream even know that their packages are wrongly formatted.

But that's just my two cents, maintained a lot of PPAs in the past for both Debian and Ubuntu and have moved on to Arch/Manjaro, because I ain't got no time for that anymore.

It is already as up the stream as it goes, bruv. Debian is the well from which Debian packages spring.

Though a lot of these errors are in the package metadata, Debian doesn't make it easy for itself as the build system and packaging format is very complex. There are lots of different build system variants, relying on weird poorly-documented helper scripts which have strange interactions. In my opinion, there's just too much magic in the debhelper system to understand how to fix problems. You also have the long, complex and constantly changing list of rules which need to be applied for each update. They are very strict about noting down the copyright of all the files, for example (which I can understand to some extent).

I think the biggest issue is the social problem of the long and arduous process to become a trusted Debian Developer. Although there's now an easier Debian Maintainer option for restricted access, it's still not easy to become one. Other people have to go through a developer to update or add packages. If you can't find a responsive developer, then package changes can sit for months or years in a broken state before they are applied (as I have found out). The whole packaging process is a frustrating exercise in bureaucracy compared to other non-Debian distributions (e.g. Fedora), which is no fun to be involved in for most people.

> I mean, if you start building an auto-fixer pipeline for an auto-linted error... then is that error actually valid in the first place? Probably not.

Just because you can automate it does not mean that the task is worthless.

React provides auto-migration scripts to newer versions, Python had the 2to3 utility, the Linux kernel has patches auto-generated with Coccinelle, editors have "rename this variable and all references" functions. Starting from a certain code base size there are changes that are worthwhile and best tackled with automation.

Hi, I've been working with Jelmer on the janitor and related infrastructure for over a year now and worked with him writing this blog post.

As others have pointed out, this is used for Debian metadata, so upstreaming doesn't really apply. While "priority extra" might seem trivial, there are other much more sophisticated examples of what the janitor can do. For example, applying multiarch fixes. These fixes are a lot more complicated, so we didn't want to show their code and have to explain everything that's going on during the blog post, as the blog post was already long enough. This was just to give people an idea that it's possible to make large scale, high quality fixes relatively easily and how they might go about it if they also have a fix they want to propose.

We've been using the lintian metadata fixes as a proving ground to learn how to make safe, high quality changes at scale. Fixing one package is trivial, doing even a mediocre job of fixing 60,000 packages is much much harder. Nobody wants crappy automated junk in the repository.

There are also other easy to overlook, massive non-technical problems in this space, like trying to convince people that the Janitor is doing worthwhile work and people should embrace it. Significantly more thought and effort has gone into making sure people are delighted (or at least not too annoyed) by the janitor than the actual changes themselves. There's things like making sure you can correctly identify and preserve a large number of idiosyncratic formatting styles. This is open source software, and everyone has their own distinct opinions as to what is "best". You also don't want to overwhelm people with fixes they're not interested in. We've already got sketches of blog posts coming up that discuss some of these issues and how the janitor handles them.

The janitor can do things other than just lintian fixes, such as multiarch improvements and merge in new upstream releases and even upstream snapshots. These aren't enabled yet, as there are still being polished up to the standards Debian demands. For some of them (like merging new upstreams) there a lot of very difficult non-technical questions that still need answering. (How do you make sure that a compromised upstream developer submitting a backdoor into a common library doesn't have that code automatically pushed out to everyone?)

The goal of the janitor is to help Debian move forward (more) rapidly, to increase the overall quality of packages in the archive, and free up the humans so they can concentrate on things that require human judgement (eg writing high quality descriptions).

If the janitor is successful, then you shouldn't have to spend much time working on packages. All of the drudge work that takes human time should be automated, leaving only the things that require actual human discretion. This helps alleviate your "I ain't got no time for that". We're not there yet, but we're working towards it.

We have discussed being able to apply fixes further upstream. It's an even harder problem to get right, and again, most of the complexity is in the non-technical side. So far, we've not tackled this, but we have deeply considered it. For the technical sided problems, before we can do any upstream fixes, we need a reliable way of being able to find the upstream repositories, we need to be working on the latest upstream snapshot, we need to know how to build the package, etc. These are all problems the Janitor is already working on solving cleanly and elegantly first.

Don't worry, we've got some awesome plans for the Janitor. Lintian fixes are just the very first step.