YGingras^.net

All of a sudden, I find myself editing lots of config files on systems on systems maintained by lots of other people. Most of the time, a text editor is all what you need but there are times when is doesn't cut. When I become a living implementation of bin/patch, I seriously consider putting /etc under a revision control system.

Why is it good to have /etc under a revision control system? We often have to perform a change on related but not quite similar servers. Furthermore, it happens that the solution is not straight forward to come up to. If we know that it's doing to require a lot of tweaking, we can always backup the original file and generate a diff once we have it working. This sucks for many reasons: it won't take into accounts changes involving multiple files, it's hard to merge into a single operation multiple edits, and you end up with way too many backups: foo.cfg, foo.cfg.bak, foo.cfg.old, foo.cfg.old1, foo.cfg.2007-11-18, and foo.cfg.051211. Yuck!

If you have the opportunity of sitting near a pack of sysadmins, you will probably hear once in a while one of them exclaim something like "What the fuck! Since when is MaxClient set to 500 on server 42?" A great advantage of config files over binary registry is that you can add comments; in practice comments are rarely used for anything but commenting out the old setting. And there are all those graphical configuration tools that will happily fill config files with white noise.

Can a revision control system really help? You bet! Distributed revision control systems like Git and Mercurial do not need a server; they store the history locally on disk and this is critical for us because we wouldn't want to move all those sensitive settings to a central location. With a revision control system, you can review your changes before you commit and you need a comment in order to commit. It's easy to see when a certain file was edited, by either a human or a tool, and it's trivial to selectively undo a modification far back in time. Also, all the modern revision control systems can export a set of modifications into a logical diff that can be applied to another system, even if the files are not completely similar.

This is great but we have to keep in mind that those tools were designed to track programming projects; there are a few design choices that make them tricky to use on /etc. Firstly, the commit log is tightly tied to the Posix user of the commiter. Of course, most edits on /etc are done as root so we need a way to override that if we want the commit log to tell us who did what. Secondly, /etc is hard to stage. What I mean is that you can't checkout a branch and hack it until it works before you merge it back in. You need to hack /etc in place and doing so means that only one branch can be active at a time. I can't think of a clean way to enforce that so we have to rely on legacy communication systems: tell the other admins on which server you are working or suffer.

When someone asks me if he should use Git or Mercurial, I always reply "yes." Both tools can do the same tricks and the interface differences are marginal. But the need to hack in place makes Git with it's internally checked out branches a favorite. Mercurial 0.9.5 has branches but those are not exactly as fancy as Git ones. On the other hand, git checkout will revert the permissions to world readable so it's probably not a good idea to branch unless you add another layer to take care of shadow, libnss-ldap.secret, and all the other files you don't want the world to see.

Michael Prokop's solution is extremely clean: it hooks into apt so you can easily tell the automatic edits from the manual ones. I only wrapped it in a convenient kick start script and I added an ecom command (for etc commit) that takes a user name as the first parameter and passes the rest to hg commit. I suspect that Michael runs in an environment where most of the stuff is done with sudo while we work root most of the time so we can't rely on $LOGNAME to get a per-user commit log. With my setup I would do:

     emacs logrotate.d/apache2
     emacs logrotate.d/foo
     emacs foo.conf
     hg add
     ecom YGingras -m "more aggressive log rotations and renice of analysers"

I also force Mercurial 0.9.5 because we want to track symlinks. I keep it out of the global path to prevent any possible breakage of running systems.

Here is hgetc 0.1. Have fun!

update: I have to mention etckeeper, a framework to track /etc with git that bypasses the world readable limitation by using an external store for meta-data. It's a really neat solution but it will be harder to setup on a non-apt system. We maintain a number of RPM based systems and it's easier for us to use the same solution all over the place.