Ask HN: How do you work with Dependabot?

#1

Post by dynamite-ready » Fri, Aug 12, 2022, 9:47 AM UTC

I like the idea of Dependabot. A to that actively tracks down dependency updates can be useful. Where I work, we have a daily CI job that creates a PR for each new dependency and runs a build in both our UI (JavaScript) and API (Python) projects. If the build passes, "Happy Days", we can merge the PR, and the app is all the more secure and effective for it.

What I've noticed in practice however, is that occasionally, this process will allow an upgrade to a dependency that will pass the automated build and test step, but introduce the wildest runtime error into the application. Usually at the time when we aim to deliver something.

Dependency 'spam' is also a very real issue - https://news.ycombinator.com/item?id=27929596 - the daily deluge of often insignificant updates is a trudge to deal with, especially when coupled with the risk of these sly runtime errors.

Dependabot is a great idea, but no-one appears to have anything bad, or practical, to say against it. But it does clearly have flaws.

I don't think I'd want to switch the bot off, but I would be interested in hearing how other people get on with the tool.

Thanks. :]

Re: Ask HN: How do you work with Dependabot?

#2

Post by Raed667 » Fri, Aug 12, 2022, 10:02 AM UTC

At a previous job, we loved the idea of depandabot but in practice it didn't match the way we work or review PRs. And as you said, just because a test passed doesn't mean that the update is 100% safe.

So instead, we identified our critical dependencies, then included dependencies update task to the list of tech-debt tasks we handle every week manually.

Re: Ask HN: How do you work with Dependabot?

#3

Post by dignati » Fri, Aug 12, 2022, 10:10 AM UTC

> What I've noticed in practice however, is that occasionally, this process will allow an upgrade to a dependency that will pass the automated build and test step, but introduce the wildest runtime error into the application. Usually at the time when we aim to deliver something.

Sounds like dependabot is very useful for uncovering insufficient test coverage or missing integration tests :)

Re: Ask HN: How do you work with Dependabot?

#4

Post by [deleted] » Fri, Aug 12, 2022, 10:12 AM UTC

[deleted]

Re: Ask HN: How do you work with Dependabot?

#5

Post by blibble » Fri, Aug 12, 2022, 10:13 AM UTC

I ignore everything it says, for me it's just noise

Re: Ask HN: How do you work with Dependabot?

#6

Post by dynamite-ready » Fri, Aug 12, 2022, 10:13 AM UTC

Raed667 wrote: post #2

At a previous job, we loved the idea of depandabot but in practice it didn't match the way we work or review PRs. And as you said, just because a test passed doesn't mean that the update is 100% safe. So instead, we identified our critical dependencies, then included dependencies update task to the list of tech-debt tasks we handle every week manually.

Yes. This is exactly what I would prefer to do. My current plan is to round up all the PR requests, at the start of each new sprint (every two weeks), and make it the first development task in the sprint.

The problem though, is that so far, I can't point to any literature online, to support this idea.

I would still like to keep Dependabot, because the diagnostic step it performs is useful. But introducing new dependency upgrades daily, even minor upgrades, seems like a recipe for trouble. Minor upgrades are just as likely to introduce a vulnerability, as they are to patch them, after all.

Re: Ask HN: How do you work with Dependabot?

#7

Post by eurasiantiger » Fri, Aug 12, 2022, 10:16 AM UTC

Someone knowledgeable of the codebase reviews changelogs of individual packages. They merge all simple cases and flag any breaking changes and anything that affects features used in the codebase. These can then be tested/fixed by others not so deeply familiar with the codebase, which fuels knowledge transfer.

Re: Ask HN: How do you work with Dependabot?

#8

Post by Raed667 » Fri, Aug 12, 2022, 10:18 AM UTC

dynamite-ready wrote: post #6

Raed667 wrote: post #2

At a previous job, we loved the idea of depandabot but in practice it didn't match the way we work or review PRs. And as you said, just because a test passed doesn't mean that the update is 100% safe. So instead, we identified our critical dependencies, then included dependencies update task to the list of tech-debt tasks we handle every week manually.

Yes. This is exactly what I would prefer to do. My current plan is to round up all the PR requests, at the start of each new sprint (every two weeks), and make it the first development task in the sprint. The problem though, is that so far, I can't point to any literature online, to support this idea. I would still like to keep Dependabot, because the diagnostic step it performs is useful. But introducing new depende…

We ended up developing a (naive) internal tool that allowed to see how many versions (and days) behind each dependency is, sorted by how much we care about each package.

This gave us a quick dashboard to checkout before running `yarn upgrade-interactive --latest`

Re: Ask HN: How do you work with Dependabot?

#9

Post by dynamite-ready » Fri, Aug 12, 2022, 10:21 AM UTC

dignati wrote: post #3

> What I've noticed in practice however, is that occasionally, this process will allow an upgrade to a dependency that will pass the automated build and test step, but introduce the wildest runtime error into the application. Usually at the time when we aim to deliver something. Sounds like dependabot is very useful for uncovering insufficient test coverage or missing integration tests :)

That would be a shallow reading, however. The last two major runtime issues wer actually one that broke the test runner and ignored a number of tests. And another runtime error was a Python Django specific sub dependency that broke the admin interface, which obviously, we don't explicitly test.

On the other hand, very recently, we had to abort a release, because of an outdated dependency that Dependabot DID actually raise.

Which is why I don't want to throw the baby out with the bathwater, as one or two people have suggested.

But I can say that I think that the reality of working with Dependabot, is not very well reflected in popular online articles.

Re: Ask HN: How do you work with Dependabot?

#10

Post by cpswan » Fri, Aug 12, 2022, 10:22 AM UTC

My main gripe is that Dependabot can end up raising multiple PRs for the same dependency bump in the same repo (especially with Dockerfiles). I really wish I could tell it to do rollups e.g. `@dependabot rollup #1234 #1235 #1236` or something like that.

To save having to do multiple rounds of merge PR, rebase next PR, wait for CI... I end up doing my own rollup PRs by merging the various Dependabot branches. At least Dependabot is smart enough to close all of the original PRs when the rollup is merged.