Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you work with Dependabot?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 46 posts

Re: Ask HN: How do you work with Dependabot?

#11

Post by damienwebdev »

dynamite-ready wrote: post #9
dignati wrote: post #3

> What I've noticed in practice however, is that occasionally, this process will allow an upgrade to a dependency that will pass the automated build and test step, but introduce the wildest runtime error into the application. Usually at the time when we aim to deliver something. Sounds like dependabot is very useful for uncovering insufficient test coverage or missing integration tests :)

That would be a shallow reading, however. The last two major runtime issues wer actually one that broke the test runner and ignored a number of tests. And another runtime error was a Python Django specific sub dependency that broke the admin interface, which obviously, we don't explicitly test. On the other hand, very recently, we had to abort a release, because of an outdated dependency that Dependabot DID actually…

> Python Django specific sub dependency, that broke the admin interface, which obviously, we don't explicitly test.

There's your problem.

Re: Ask HN: How do you work with Dependabot?

#12

Post by cr3ative »

dynamite-ready wrote: post #9
dignati wrote: post #3

> What I've noticed in practice however, is that occasionally, this process will allow an upgrade to a dependency that will pass the automated build and test step, but introduce the wildest runtime error into the application. Usually at the time when we aim to deliver something. Sounds like dependabot is very useful for uncovering insufficient test coverage or missing integration tests :)

That would be a shallow reading, however. The last two major runtime issues wer actually one that broke the test runner and ignored a number of tests. And another runtime error was a Python Django specific sub dependency that broke the admin interface, which obviously, we don't explicitly test. On the other hand, very recently, we had to abort a release, because of an outdated dependency that Dependabot DID actually…

> the admin interface, which obviously, we don't explicitly test

This... is not obvious. It broke a part of your software which you care about. You care, it caused a problem, so it should be tested.

Re: Ask HN: How do you work with Dependabot?

#14

Post by Narretz »

dynamite-ready wrote: post #9
dignati wrote: post #3

> What I've noticed in practice however, is that occasionally, this process will allow an upgrade to a dependency that will pass the automated build and test step, but introduce the wildest runtime error into the application. Usually at the time when we aim to deliver something. Sounds like dependabot is very useful for uncovering insufficient test coverage or missing integration tests :)

That would be a shallow reading, however. The last two major runtime issues wer actually one that broke the test runner and ignored a number of tests. And another runtime error was a Python Django specific sub dependency that broke the admin interface, which obviously, we don't explicitly test. On the other hand, very recently, we had to abort a release, because of an outdated dependency that Dependabot DID actually…

Was it an update to the test runner or test specific packages that broke the test runner? I would ignore infrastructure/testability/tooling packages in dependabot and do them manually to prevent these errors.

Re: Ask HN: How do you work with Dependabot?

#15

Post by dignati »

dynamite-ready wrote: post #9
dignati wrote: post #3

> What I've noticed in practice however, is that occasionally, this process will allow an upgrade to a dependency that will pass the automated build and test step, but introduce the wildest runtime error into the application. Usually at the time when we aim to deliver something. Sounds like dependabot is very useful for uncovering insufficient test coverage or missing integration tests :)

That would be a shallow reading, however. The last two major runtime issues wer actually one that broke the test runner and ignored a number of tests. And another runtime error was a Python Django specific sub dependency that broke the admin interface, which obviously, we don't explicitly test. On the other hand, very recently, we had to abort a release, because of an outdated dependency that Dependabot DID actually…

> one that broke the test runner, and ignored a number of tests

That's unfortunate! For the project I'm working on, we've "solved" that by showing the number of test and the difference to the number of tests that ran on main.

FWIW, at previous jobs, upgrading Java dependencies was a major pain because they were all outdated and the latest versions introduced too many breaking changes for us. At my current job, we pretty much instantly merge all PRs from dependabot because we trust our CI. Upgrades rarely introduce problems and if they do, they are easy to fix.

Re: Ask HN: How do you work with Dependabot?

#16

Post by ncphillips »

Same feelings. I like the idea but in practice I don’t trust it.

Or rather, I don’t trust package maintainers to adhere to semver. I prefer to manually go through dependencies updating one at a time and reading the change logs. I usually do this in batch. Peace of mind is worth more than the hour saved every week or two.

I do really like the tool that flags security issues with packages though.

Re: Ask HN: How do you work with Dependabot?

#18

Post by codelion »

To address this issue we designed a static analysis that can check if the upgrade is likely to break the application. Here are some details of the work- “Effective Static Checking of Library Updates” https://dl.acm.org/doi/abs/10.1145/3236024.3275535

When using the analysis a PR for upgrading the dependencies would look like this - https://github.com/tmroberts56/java-maven/pull/3

Re: Ask HN: How do you work with Dependabot?

#19

Post by dynamite-ready »

cr3ative wrote: post #12
dynamite-ready wrote: post #9

Earlier quoted context omitted.

That would be a shallow reading, however. The last two major runtime issues wer actually one that broke the test runner and ignored a number of tests. And another runtime error was a Python Django specific sub dependency that broke the admin interface, which obviously, we don't explicitly test. On the other hand, very recently, we had to abort a release, because of an outdated dependency that Dependabot DID actually…

> the admin interface, which obviously, we don't explicitly test This... is not obvious. It broke a part of your software which you care about. You care, it caused a problem, so it should be tested.

You routinely write unit tests for dependent packages in your apps?

Re: Ask HN: How do you work with Dependabot?

#20

Post by kristiandupont »

My strategy is to blindly merge developer dependencies like linter tools etc., with the rationale that if CI succeeds, the dependency didn't break anything. Obviously, that is not completely guaranteed but the risk feels small enough that it's acceptable.

Any update to production dependencies I will want to test manually unless it happens to be a part of the code where I feel exceptionally confident in our test coverage.

Post reply on HN