Ask HN: How do you manage your *nix binary package updates? (first post. eek)

#1

Post by baconhigh » Sat, Jun 25, 2011, 11:37 AM UTC

We run clusters of machines and whenever there's an update via USN / DSA or whatever I end up manually patching each cluster with cluster-ssh.

This is less than ideal, but seems to work.

What do you do?

Note; I'm talking about binary packages distributed by your OS : apt upgrades / rpms.. not config files (Hi, puppet/chef), or deprec for capistrano style stuff.

Re: Ask HN: How do you manage your *nix binary package updates? (first post. eek)

#2

Post by bastiat » Sat, Jun 25, 2011, 12:12 PM UTC

Corporate repository to which official packages migrate upon certification. This usually takes about 24 hours.

Re: Ask HN: How do you manage your *nix binary package updates? (first post. eek)

#3

Post by aspitzer » Sat, Jun 25, 2011, 1:13 PM UTC

I used to manage around 300 servers myself. The only way it was possible was to have a completely stripped OS. All apps we used I installed under:

/apps// example: /apps/perl/5.8.12 then I would symlink /apps/perl/5.8.12 to /apps/perl/current

The profiles on the machine would add /apps/*/current/bin to the path. This allowed upgrades and roll backs just by changing the symlink to the one I wanted to be current. This also allowed me to push out versions of software ahead of time, and they just change the link when we were ready to use it.

Each machine would rsync /apps from a master distro nightly and of course I could force it with a for i in `cat hosts.list`...

Re: Ask HN: How do you manage your *nix binary package updates? (first post. eek)

#4

Post by bretthoerner » Sat, Jun 25, 2011, 1:26 PM UTC

Isn't this what people run their own apt mirrors for?

Re: Ask HN: How do you manage your *nix binary package updates? (first post. eek)

#5

Post by bcl » Sat, Jun 25, 2011, 1:31 PM UTC

I run a squid caching proxy and direct all my (Fedora or RHEL) systems to use it along with a non-mirror list repository that is fairly close to me. I'm not totally sure what your question is though.

If you want to tightly control the packages that get updated then there isn't much to be done other than manually managing your own repo. Although if it is this big of a concern you should probably be running a distribution that is less volatile than Ubuntu. Say RHEL, CentOS, or Debian.

Re: Ask HN: How do you manage your *nix binary package updates? (first post. eek)

#6

Post by rjh29 » Sat, Jun 25, 2011, 1:36 PM UTC

I've heard good things about MCollective for distributed package updates:

http://www.puppetlabs.com/mcollective

Re: Ask HN: How do you manage your *nix binary package updates? (first post. eek)

#7

Post by BCM43 » Sat, Jun 25, 2011, 1:48 PM UTC

bretthoerner wrote: post #4

Isn't this what people run their own apt mirrors for?

Yep. At my work we have a apt repository, and push packages to in, then issue an apt-get update on all the machines.

Re: Ask HN: How do you manage your *nix binary package updates? (first post. eek)

#8

Post by dkarl » Sat, Jun 25, 2011, 1:48 PM UTC

We manage our own dists. We package our own software as .debs, so everything gets managed the same way. All security updates, release deployments, rollbacks, etc., are managed with apt-get. I don't exactly understand pinning, but it's also important to how we manage packages.

Depending on the release and the kind of server we're deploying to, we may do them all in one night or in batches of a few hundred over a week. All our boxes install security updates regularly (because we promptly add security updates to our dists.)

Re: Ask HN: How do you manage your *nix binary package updates? (first post. eek)

#9

Post by rdtsc » Sat, Jun 25, 2011, 2:03 PM UTC

Create a custom repository of your packages and point all the machines in your cluster to it? If it is CentOS or RHEL just add a new repo to /etc/yum.repos.d. If you are upgrading system repos then just set the priority of your repo to be higher. Of course this implies that you have successfully rpm-fied your packages. We did that to all our packages and configuration.

Re: Ask HN: How do you manage your *nix binary package updates? (first post. eek)

#10

Post by [deleted] » Sat, Jun 25, 2011, 2:16 PM UTC

dkarl wrote: post #8

We manage our own dists. We package our own software as .debs, so everything gets managed the same way. All security updates, release deployments, rollbacks, etc., are managed with apt-get. I don't exactly understand pinning, but it's also important to how we manage packages. Depending on the release and the kind of server we're deploying to, we may do them all in one night or in batches of a few hundred over a week.…

[deleted]