21–30 of 46 posts
Earlier quoted context omitted.
> the admin interface, which obviously, we don't explicitly test This... is not obvious. It broke a part of your software which you care about. You care, it caused a problem, so it should be tested.
You routinely write unit tests for dependent packages in your apps?
Same feelings. I like the idea but in practice I don’t trust it. Or rather, I don’t trust package maintainers to adhere to semver. I prefer to manually go through dependencies updating one at a time and reading the change logs. I usually do this in batch. Peace of mind is worth more than the hour saved every week or two. I do really like the tool that flags security issues with packages though.
Dependabot creates the PRs and I review the changelog / commits to make sure I'm not introducing a bug or security issue.
Earlier quoted context omitted.
> the admin interface, which obviously, we don't explicitly test This... is not obvious. It broke a part of your software which you care about. You care, it caused a problem, so it should be tested.
You routinely write unit tests for dependent packages in your apps?
With regards to writing tests of the Django admin, you need to do it if your site is customizing and/or depending upon it.
Earlier quoted context omitted.
You routinely write unit tests for dependent packages in your apps?
But once you've imported that package it's part of _your_ codebase.
So between choosing to write everything themselves (and getting nothing done), writing tests against dependencies (and getting little done due to the overhead), or claiming that external dependencies should have tests of their own, many will pick the latter.
Then again, in a world where create-react-app results in 180 MB of dependencies and about 1500 modules (probably different numbers now, using some older ones from my blog post), auditing security is an uphill battle, not even talking about actual testing.
The situation in the back end development, isn't that much better either, to be honest, because once you look into the complexity of any framework like Spring, Laravel, Django, Rails etc., it becomes apparent that creating a fully featured framework like that is a huge undertaking.
That said, you should at least test the bits where the external dependency is integrated with your codebase.
1) For more important projects I have high test coverage (over 95%) and integration tests. In this setting, I have dependabot set to check once a week and the PRs get automatically merged if everything works. I have never had a problem with this setup, if the update breaks something, the tests fail and I get to fix it manually.
2) For less important projects without CD, I usually merge dev dependencies and minor versions by hand if there isn't something obviously wrong in the CI. Yeah, there's a risk it breaks something, but I can fix that before a release - obviously these are projects that only get worked on occasionally. Major versions I usually check manually before merging.
I think the ultimate answer is good testing as part of CI. Either you trust your tests and then I think it's ok to auto-merge, or you don't and then all bets are of...
> What I've noticed in practice however, is that occasionally, this process will allow an upgrade to a dependency that will pass the automated build and test step, but introduce the wildest runtime error into the application. Usually at the time when we aim to deliver something. Sounds like dependabot is very useful for uncovering insufficient test coverage or missing integration tests :)
That would be a shallow reading, however. The last two major runtime issues wer actually one that broke the test runner and ignored a number of tests. And another runtime error was a Python Django specific sub dependency that broke the admin interface, which obviously, we don't explicitly test. On the other hand, very recently, we had to abort a release, because of an outdated dependency that Dependabot DID actually…
That will then happen whether you use dependabot or another approach. If you don't test this manually and don't have enough test coverage, you will fail there eventually.
Dependabot helps you fail here early when you have N packages that changed and it's easy to figure out what happened, rather than wait until you're upgrading 10*N packages together and have to isolate things manually.
Dependabot has to be used because of the threats. However everyone automating and moving to latest is also a threat. SolarWinds/VMWare/USGov hack [3] was all related to CI builds and automated "trust", ended up infecting tens of thousands of systems that thought they were secure with SOC2. SOC2 ends up making enterprises "trust" many third parties. What happens when dependabot is an attack vector as well...
The log4j/Log4Shell [4] issue shows how long exploits can go on without detection or automated fixes. Node is filled with dependency issues and that is just the known exploits besides all the "telemetry". [5]
Any third party or dependency is a potential attack vector, and dependency saturation is adding lots of tedium to shipping. So much time goes to just updating libs it is a bit of a tragic comedy.
[1] https://en.wikipedia.org/wiki/DLL_Hell
[2] https://en.wikipedia.org/wiki/Dependency_hell
[3] https://en.wikipedia.org/wiki/2020_United_States_federal_gov...
[4] https://en.wikipedia.org/wiki/Log4Shell
[5] https://en.wikipedia.org/wiki/Npm_(software)#Notable_breakag...
This still lets me know when an update to ANY internal or external dependency will break the build, so that I can take a look.
My tool only works with Nix Flakes, so it updates dependencies of Nix packages, but you often have other language-specific package managment tools wrapped inside of Nix (e.g. nix calling cargo) and I think it would be a cool extension to add a configuarable language-specific shell command where the results of running that command are added to that single PR as well. That could document the exact update workflow a dev should go through while also regularly exercising it.
https://github.com/serokell/update-daemon is a better tool than mine that does the same thing and I have been thinking about adding the feature there. I'd be interested in discussing that idea further, maybe I'll open an issue for that.