Live data from Hacker News

Just say no to :latest

platformers.dev

11–20 of 135 posts

Re: Just say no to :latest

#11

This is always a balance. The moment you pin to a specific version you need to have a process in place to ensure you regularly upgrade to avoid introducing vulnerabilities in your production system. Throughout my career I have seen many cases where certain software still runs on ancient versions as the team originally maintaining it is no longer around (e.g. reorganisations or lay-offs). It is always hard to convince…

Doesn’t the same logic apply to using :latest? If you are always pulling the latest, unvetted, code, shouldn’t you also have a vulnerability management process in place?

You need vulnerability & risk management for any dependency, full stop.

Re: Just say no to :latest

#12
post #3
post #2

# GOOD: image: "nginx:1.21.6" nothing "good" about it, literally no different from ":latest". only full hash reference.

Sure version tags could also move, but by convention they do not. Unlike the latest tag, which by convention does move a lot.

By convention npm packages are not deleted or hijacked.

Re: Just say no to :latest

#13

This is always a balance. The moment you pin to a specific version you need to have a process in place to ensure you regularly upgrade to avoid introducing vulnerabilities in your production system. Throughout my career I have seen many cases where certain software still runs on ancient versions as the team originally maintaining it is no longer around (e.g. reorganisations or lay-offs). It is always hard to convince…

Doesn’t the same logic apply to using :latest? If you are always pulling the latest, unvetted, code, shouldn’t you also have a vulnerability management process in place? You need vulnerability & risk management for any dependency, full stop.

Ideally you only use Docker official images,or their equivalent to avoid using unvetted code.

It is always a trade off, however it is far more likely that a hacker will use a ten year old well exploited CVE, rather than a recent one

Re: Just say no to :latest

#14

This is always a balance. The moment you pin to a specific version you need to have a process in place to ensure you regularly upgrade to avoid introducing vulnerabilities in your production system. Throughout my career I have seen many cases where certain software still runs on ancient versions as the team originally maintaining it is no longer around (e.g. reorganisations or lay-offs). It is always hard to convince…

In that case, it's better to just pin the major version of the container:

    FROM python:3
will work as long as Python3 doesn't make any backwards incompatible changes (note: Python3 occasionally deprecates then removes a feature that was part of the official API[0], so Python3 is not technically semver-correct).

If, however, one wants automatic updates WITH reproducible builds, then a CI/CD pipeline that automatically updates the FROM line in a Dockerfile on every upstream release is an only solution.

[0] https://docs.python.org/3.10/whatsnew/3.10.html#removed

Re: Just say no to :latest

#15

This is always a balance. The moment you pin to a specific version you need to have a process in place to ensure you regularly upgrade to avoid introducing vulnerabilities in your production system. Throughout my career I have seen many cases where certain software still runs on ancient versions as the team originally maintaining it is no longer around (e.g. reorganisations or lay-offs). It is always hard to convince…

It's better to only change one thing at a time, not to get your dependencies updated as a surprise when you're trying to update something else.

Besides, if the dependencies aren't pinned in version control, how can you answer a query like "On this date last year, what version were we running?"

Re: Just say no to :latest

#17

This is always a balance. The moment you pin to a specific version you need to have a process in place to ensure you regularly upgrade to avoid introducing vulnerabilities in your production system. Throughout my career I have seen many cases where certain software still runs on ancient versions as the team originally maintaining it is no longer around (e.g. reorganisations or lay-offs). It is always hard to convince…

I'd say that there are two things that you should have in place regardless of which approach you take:

  - having tests (or even manually test the system with scenarios, if for whatever reason you cannot automate) in place to catch things breaking, before shipping any changes
  - having security scanning in place, be it for containers, your dependencies, or anything else; ideally for everything
Then, things should get a bit more clear:

  - you should be able to spot the publicly known vulnerabilities and adequately evaluate their impact to decide what must be done
  - when you update versions of your dependencies, you should then also be able to see whether anything breaks before shipping
Admittedly, all of that will only be viable if you have the buy-in from the people involved, since otherwise you'll get "blamed" for things breaking as a result of your initiative for avoiding shipping vulnerable software, or you'll find it difficult to justify to people why you're spending so much time updating dependencies and doing refactoring.

Not only that, but some systems out there are not really easily testable, e.g. those that have a large amount of functionality within the DB so your tests might as well end up mocking either too much of the system or the wrong parts of the system (i'm yet to see anyone mocking the low level queries that go to the DB and back, e.g. setting and validating the individual parameters within query abstractions, as opposed to just the data that's returned) - in many cases it simply won't be viable to test everything.

Another thing is that in practice semver tends to lie to us and even minor updates or patches can sometimes have breaking changes within them, something that i wrote about in my blog article "Never update anything": https://blog.kronis.dev/articles/never-update-anything (albeit there i also touch upon the fact that software should largely be more stable and have fewer breaking updates in the first place, or even less new features in "stable" branches)

I guess my argument is that there's a lot of complexity to be tackled here and that people should invest more time and effort into handling updates, refactoring and even testing, than they do now: i've seen teams where people all agree that tests are important, yet nobody wants to write any because if they tried, they'd have to mock large parts of the system OR try to do integration tests and handle the fact that nobody has invested the time and effort into bringing up reproducible environments and services, e.g. a new automatically migrated DB instance for the tests OR the fact that they'll need to do tests in a shared DB instance and clean up afterwards. Environments like that are just a downwards spiral that's bound to produce brittle software.

So what's my practical advice?

Use pinned versions, preferably starting with the latest and most boring one that you can get away with (e.g. LTS). Have a process in place for figuring out when you need to update, set aside time for doing just that (even if manually), assign someone to do this and someone else to make sure it's been done. Have testing and serious validations be a part of this process, to make sure that there's no breakage that'd slip past - full regression testing, all your unit tests, integration tests, feature tests etc. How often should you do it? Depends on the importance of the system - i've seen it done quarterly, i've seen it done monthly, some with plenty of resources out there could probably do it weekly or more often. Additionally, be able to do this ASAP when critical vulnerabilities become apparent, e.g. log4shell.

Of course, it's the ultimate example of useful but exceedingly boring work that people don't really want to do, so i've no doubts about running into unmaintained software in the future.

Re: Just say no to :latest

#18
Use latest during development, but push to production using image in your private Docker registry which has proper names and tags.

git push -> Docker build (bonus points for building only if needed [for example only if Dockerfile, Jenkinsfile or requirements.txt has changed], otherwise use latest from Artifactory) -> run all automated tests -> if pass, push the Docker image to Artifactory with reasonable name and tag.

When doing a release, push the image from Artifactory to production.

This way you don't have to have a process to update all the things in Dockerfile on reqular basis, but you still only push to production the actual binaries that are really tested and proven to be ok. And can re-use the actual binary when doing a hotfix of something similar.

Re: Just say no to :latest

#19

Same with pip/Python. Recucing dependencies to a bare minimum helps long term survival of software.

It also helps with reducing dependency distribution issues - I've written single-file Python programs using just the stdlib, sometimes of a really old Python3 version. It's really nice to just `scp` the script to a bunch of remote devices and have everything work out-of-the-box.

Of course, PyInstaller can solve dependency distribution issues - but I've sometimes ran into packages that don't play well with PyInstaller (uwsgi, for example, because it's basically a C program distributed using pip, so PyInstaller can't figure out how to bundle it).

Re: Just say no to :latest

#20

This is always a balance. The moment you pin to a specific version you need to have a process in place to ensure you regularly upgrade to avoid introducing vulnerabilities in your production system. Throughout my career I have seen many cases where certain software still runs on ancient versions as the team originally maintaining it is no longer around (e.g. reorganisations or lay-offs). It is always hard to convince…

In that case, it's better to just pin the major version of the container: FROM python:3 will work as long as Python3 doesn't make any backwards incompatible changes (note: Python3 occasionally deprecates then removes a feature that was part of the official API[0], so Python3 is not technically semver-correct). If, however, one wants automatic updates WITH reproducible builds, then a CI/CD pipeline that automatically…

this works great until a library decides it only likes python 3.8 or 3.9 or 3.10 and won't work with 3.10.
Post reply on HN