Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Debian Janitor: 60k Lintian Issues Automatically Fixed

jelmer.uk

Post reply on HN
Print view View original

11–20 of 21 posts

Re: Debian Janitor: 60k Lintian Issues Automatically Fixed

#11

Post by debiandev »

xioxox wrote: post #6
cookiengineer wrote: post #2

To be honest, this sounds more like a >/dev/null solution to me. The underlying issue is that debian packages are out of date, and that the repos (ppas) containing them are heavily unmaintained up to the point were decades old libraries are required even when upstream has moved on. Why not go with the approach to try and fix it upstream where it belongs instead of maintaining a set of fixers downstream - which will l…

Though a lot of these errors are in the package metadata, Debian doesn't make it easy for itself as the build system and packaging format is very complex. There are lots of different build system variants, relying on weird poorly-documented helper scripts which have strange interactions. In my opinion, there's just too much magic in the debhelper system to understand how to fix problems. You also have the long, compl…

> weird poorly-documented helper scripts which have strange interactions

I've never found a packaging tool without an extensive and clear manpage with examples.

> I think the biggest issue is the social problem of the long and arduous process to become a trusted Debian Developer. Although there's now an easier Debian Maintainer option for restricted access, it's still not easy to become one. Other people have to go through a developer to update or add packages.

There is an arduous process to become a senior engineer in a FAANG, or a tenured professor, or an airline pilot.

That's why a lot of people trust Debian.

Re: Debian Janitor: 60k Lintian Issues Automatically Fixed

#12

Post by jelmer »

cookiengineer wrote: post #2

To be honest, this sounds more like a >/dev/null solution to me. The underlying issue is that debian packages are out of date, and that the repos (ppas) containing them are heavily unmaintained up to the point were decades old libraries are required even when upstream has moved on. Why not go with the approach to try and fix it upstream where it belongs instead of maintaining a set of fixers downstream - which will l…

This is fixing issues in the packaging, not the upstream part of the package. I have plans to also have it fix issues in upstreams, but as you say - those will go into the upstream repositories.

It processes the main Debian archive, not PPAs. The bot makes changes to the Git repository with the packaging - developers would get a pull request with the changes after doing verification, and they can just click "Merge" on the GitLab UI. That seems pretty similar to what you're saying would happen in arch.

For a broader description of the design, see https://jelmer.uk/debian-janitor.html

Re: Debian Janitor: 60k Lintian Issues Automatically Fixed

#13

Post by Isomer »

Greek0 wrote: post #8
cookiengineer wrote: post #2

To be honest, this sounds more like a >/dev/null solution to me. The underlying issue is that debian packages are out of date, and that the repos (ppas) containing them are heavily unmaintained up to the point were decades old libraries are required even when upstream has moved on. Why not go with the approach to try and fix it upstream where it belongs instead of maintaining a set of fixers downstream - which will l…

> I mean, if you start building an auto-fixer pipeline for an auto-linted error... then is that error actually valid in the first place? Probably not. Just because you can automate it does not mean that the task is worthless. React provides auto-migration scripts to newer versions, Python had the 2to3 utility, the Linux kernel has patches auto-generated with Coccinelle, editors have "rename this variable and all refe…

This is an excellent point.

One change the Janitor can try to make automatically is upgrading debhelper version. Newer debhelper versions include support for compiling code with additional security measures. So we end up with more secure packages.

If ~everyone migrates off an older debhelper version, then the debhelper developers don't need to keep maintaining that code and can delete it, leading to a more maintainable code base, less complex documentation that says "X, unless you're on an older version, when it's Y" and so on.

The Janitor is pretty good at trying something like upgrading debhelper, then doing a build, and running the package tests, performing a package diff to make sure nothing unexpected happened then proposing a merge. Doing this by hand is slow and laborious. If debhelper always tried to autoupgrade to the newest version when it was run, it would be frustratingly slow.

And once the merge proposal is accepted, then it's clear which packages need human attention, and which were able to be cleanly migrated by automation. We use this a lot in the janitor. Run a fixer over all eligible packages, and see which ones succeed. Then investigate a handful of packages that failed, see if there is a common pattern that can be extracted and fixed. Then we reapply the fixer to all the remaining packages, see how many were fixed, and repeat the process until there are only a very few remaining.

Re: Debian Janitor: 60k Lintian Issues Automatically Fixed

#14

Post by Isomer »

xioxox wrote: post #6
cookiengineer wrote: post #2

To be honest, this sounds more like a >/dev/null solution to me. The underlying issue is that debian packages are out of date, and that the repos (ppas) containing them are heavily unmaintained up to the point were decades old libraries are required even when upstream has moved on. Why not go with the approach to try and fix it upstream where it belongs instead of maintaining a set of fixers downstream - which will l…

Though a lot of these errors are in the package metadata, Debian doesn't make it easy for itself as the build system and packaging format is very complex. There are lots of different build system variants, relying on weird poorly-documented helper scripts which have strange interactions. In my opinion, there's just too much magic in the debhelper system to understand how to fix problems. You also have the long, compl…

The janitor is trying to help make packaging less toilsome:

- The janitor is trying to automatically migrate people to newer versions of debhelper, reducing the number of different build system variants.

- The janitor attempts to perform validation of all the long, complex, changing rules for you, so if you meet their requirements then it will propose an upgrade. If the janitor can automatically bring you into compliance with the rules, it will, otherwise it'll leave you upgraded to the step that requires a human to intervene. You can manually upgrade it one step, and then the janitor will loop back around and do the rest for you.

- The janitor will also fix suboptimal, or unnecessary packaging changes and improve them for you too, hopefully leaving you with a much smaller, simpler packaging system that's easier to reason about.

Re: Debian Janitor: 60k Lintian Issues Automatically Fixed

#15

Post by bhaak »

debiandev wrote: post #11
xioxox wrote: post #6

Earlier quoted context omitted.

Though a lot of these errors are in the package metadata, Debian doesn't make it easy for itself as the build system and packaging format is very complex. There are lots of different build system variants, relying on weird poorly-documented helper scripts which have strange interactions. In my opinion, there's just too much magic in the debhelper system to understand how to fix problems. You also have the long, compl…

> weird poorly-documented helper scripts which have strange interactions I've never found a packaging tool without an extensive and clear manpage with examples. > I think the biggest issue is the social problem of the long and arduous process to become a trusted Debian Developer. Although there's now an easier Debian Maintainer option for restricted access, it's still not easy to become one. Other people have to go t…

> I've never found a packaging tool without an extensive and clear manpage with examples.

That might be fine if you already know which packaging tool to start with. Debian has several and it's not clear upfront which one is the preferred one. But even then I'd say that there is something like too much information.

And the process is complex. If I want to package a standard autoconf/cmake program, I have to do a lot of command typing. If you compare that to Gentoo, Arch or even Homebrew packages where you essentially edit one file and one or two command invocations, that's just cumbersome.

I recently tried to find out what I would have to do to for a NMU and I just couldn't find any entrance point that made sense to me. It felt like trying to get into a cabal and all the information is encoded in arcane Latin.

Re: Debian Janitor: 60k Lintian Issues Automatically Fixed

#16

Post by OnlyOneCannolo »

renewiltord wrote: post #4
cookiengineer wrote: post #2

To be honest, this sounds more like a >/dev/null solution to me. The underlying issue is that debian packages are out of date, and that the repos (ppas) containing them are heavily unmaintained up to the point were decades old libraries are required even when upstream has moved on. Why not go with the approach to try and fix it upstream where it belongs instead of maintaining a set of fixers downstream - which will l…

It is already as up the stream as it goes, bruv. Debian is the well from which Debian packages spring.

I think "upstream" here means the software repo itself, not the Debian packaging repo.

Re: Debian Janitor: 60k Lintian Issues Automatically Fixed

#17

Post by jelmer »

bhaak wrote: post #15
debiandev wrote: post #11

Earlier quoted context omitted.

> weird poorly-documented helper scripts which have strange interactions I've never found a packaging tool without an extensive and clear manpage with examples. > I think the biggest issue is the social problem of the long and arduous process to become a trusted Debian Developer. Although there's now an easier Debian Maintainer option for restricted access, it's still not easy to become one. Other people have to go t…

> I've never found a packaging tool without an extensive and clear manpage with examples. That might be fine if you already know which packaging tool to start with. Debian has several and it's not clear upfront which one is the preferred one. But even then I'd say that there is something like too much information. And the process is complex. If I want to package a standard autoconf/cmake program, I have to do a lot o…

The process is indeed complex, but there are efforts to standardize on a single simple way to do packaging. Unfortunately with an archive of 30k source packages, that is a slow process.

A great resource is https://trends.debian.net/, which tracks some of the different ways of doing packaging, as well as the progress on convergence.

With the new style debhelper that we're converging on and a straightforward package, creating a new package should not have to involve a lot of typing - although it's still split across multiple files.

Re: Debian Janitor: 60k Lintian Issues Automatically Fixed

#18

Post by beefhash »

xioxox wrote: post #6
cookiengineer wrote: post #2

To be honest, this sounds more like a >/dev/null solution to me. The underlying issue is that debian packages are out of date, and that the repos (ppas) containing them are heavily unmaintained up to the point were decades old libraries are required even when upstream has moved on. Why not go with the approach to try and fix it upstream where it belongs instead of maintaining a set of fixers downstream - which will l…

Though a lot of these errors are in the package metadata, Debian doesn't make it easy for itself as the build system and packaging format is very complex. There are lots of different build system variants, relying on weird poorly-documented helper scripts which have strange interactions. In my opinion, there's just too much magic in the debhelper system to understand how to fix problems. You also have the long, compl…

> You also have the long, complex and constantly changing list of rules which need to be applied for each update. They are very strict about noting down the copyright of all the files, for example (which I can understand to some extent).

Tracking licensing information very strictly makes sense.

Other than that, however, the Debian packaging process is something only a lawyer could love.

Re: Debian Janitor: 60k Lintian Issues Automatically Fixed

#19

Post by VLM »

xioxox wrote: post #6
cookiengineer wrote: post #2

To be honest, this sounds more like a >/dev/null solution to me. The underlying issue is that debian packages are out of date, and that the repos (ppas) containing them are heavily unmaintained up to the point were decades old libraries are required even when upstream has moved on. Why not go with the approach to try and fix it upstream where it belongs instead of maintaining a set of fixers downstream - which will l…

Though a lot of these errors are in the package metadata, Debian doesn't make it easy for itself as the build system and packaging format is very complex. There are lots of different build system variants, relying on weird poorly-documented helper scripts which have strange interactions. In my opinion, there's just too much magic in the debhelper system to understand how to fix problems. You also have the long, compl…

If a process doesn't work, adding lots of people to it will not help.

A major part of the problem is scalability. "In the old days" the automated system that eats signed binary packages and outputs a usable apt-get accessible archive, couldn't do much to any individual package if we wanted it to complete in time. Times change and people seem to want more testing and automation, toss the binary packages and rebuild all archs from scratch, more automated tests, etc.

In the long run you're basically asking why the archive automation scripts accept incoming packages with "debian-changelog-has-wrong-day-of-week" instead of kicking those package updates back. Adding more people isn't going to fix it, adding a large system working around the process is kinda helpful, but really the archive automation scripts should just not accept packages with "trailing-whitespace" any more than they'd accept packages with invalid GPG signatures. This will result in more processing time for updates, but this can be clouded and parallelized and is fixable in 2020, even if simpler archive processing made engineering sense in 1995.

Post reply on HN