config-package-dev: building Debian configuration packages

This page describes the config-package-dev package, a system that allows easy and efficient creation of Debian configuration packages. Configuration packages are packages whose purpose is to configure a Debian system, rather than add new software. config-package-dev consists of an extension to Debhelper, as well as set of modules for the Common Debian Build System. The config-package-dev system was originally created by Anders Kaseorg and Tim Abbott as part of the MIT Debathena project; Debhelper support was added by Geoffrey Thomas.

Contents

Install config-package-dev

config-package-dev is included in Debian and Ubuntu (and should therefore be in downstream distributions). You can just install the config-package-dev package with your package manager.

Browse source and examples

The config-package-dev source code is maintained in git:

git clone git://debathena.mit.edu/config-package-dev.git

Provided your apt repositories are appropriately set up, apt-get source config-package-dev will also work to download the source for your distribution's version of config-package-dev.

You can also browse the code online via gitweb. Bug reports, comments, and patches are welcome to config-package-dev@mit.edu.

Several simple examples are in the examples/ directory in the source. See below for some more annotated examples, as well as trunk/debathena/config in the Debathena Subversion repository for several production packages using config-package-dev. (Most of these are still using CDBS, although we will be converting our own repository to Debhelper 7-style rules files shortly.)

Overview of the system

config-packge-dev supports three operations: displace, transform, and hide. All of these use the dpkg-divert operation for moving packaged files to alternative locations.

The displace operation consists of moving a packaged file aside and providing a different version of that file. For instance, debathena-nsswitch-config displaces /etc/nsswitch.conf to /etc/nsswitch.conf.debathena-orig, installs its customized configuration to /etc/nsswitch.conf.debathena, and place a symlink from nsswitch.conf to the customized version.

In addition to configuration files, a common use is to install a wrapper script for an executable. debathena-pidgin-config, for example, displaces /usr/bin/pidgin and provides a wrapper script that configures MIT Jabber in the user's home directory and then launches /usr/bin/pidgin.debathena-orig. Because dpkg-divert is used, updated versions of Pidgin continue being safely installed at /usr/bin/pidgin.debathena-orig.

The transform operation is a special case of the displace operation. At build time, a “transform script” is applied to the original source, and the result is used as the replacement in the displace operation. A common use of this is to change one value in a config file without needing to re-type the entire config file (and risk bit-rot). In fact, debathena-nsswitch-config uses a three-line Perl transform script to add references to the specific NSS backends that Debathena needs, while leaving the rest of the file matching the default upstream configuration of the distro being used.

The hide operation is yet another special case of the displace operation, namely that there is no replacement or symlink. Instead, the file is diverted to a unique path on the target system specific to the package running the hide operation (namely into /usr/share/package), instead of being left in the same directory. A common use of this is to suppress a snippet file in a configuration directory like /etc/cron.d. The standard displace mechanism would leave three files executed by cron (the symlink, the .debathena file, and the .debathena-orig), defeating the purpose of customizing it. Using the hide operation and installing a differently-named crontab leaves just that one file run by cron, as intended.

These operations all use a displace extension, which defaults to a dot followed by the first hyphen-separated word in the package name. In these examples, since all the packages start with debathena-, the displace extension has been .debathena. We recommend you follow this naming convention, but it can be customized. Note that the system does not handle changing this extension on upgrade.

What you need to begin

You will need to download and install the config-package-dev package from the link above.

If you’re not familiar with the process of building Debian packages, you may want to look at Debian's packaging tutorial [PDF].

We recommend using config-package-dev with sbuild and schroot, pbuilder, or another framework for building Debian packages in clean environments.

This article documents the Debhelper interface of config-package-dev, using Debhelper 7-style rules files. The CDBS interface from versions 4 and earlier is still present, but will not receive further development attention. Some documentation for it is present in the source code. We believe the Debhelper interface is easier to understand and use, and encourage you to migrate from CDBS if you have existing packages using this system.

Interface

To start using the config-package-dev extension module with the dh sequencer, simply pass the --with=config-package. For instance, a simple debian/rules file — sufficient in most cases — would be

#!/usr/bin/make -f

%:
	dh $@ --with=config-package

Like other Debhelper commands, the config-package-dev system is controlled through files in the debian subdirectory of the package. These files are generally named debian/package.type (e.g., debian/debathena-kerberos-config.displace), although for single-binary packages, they can just be named debian/type.

debian/package.displace
List of absolute paths to files to be diverted, one per line. Each path should contain the displace extension as a substring at least once (only the last instance of the displace extension in the filename will be stripped). Note that this will not install a replacement file automatically; if you want one, you must do that yourself (probably using debian/package.install, via the dh_install program) and you must install it to the path you specified. config-package-dev will create a symlink from the path without the displace extension.
debian/package.hide
List of absolute paths to files to be diverted into a unique path in /usr/share/package/. Note that this will not install a replacement file automatically; if you want one, you must do that yourself (probably using dh_install), and you must give it a different name than the original file.
debian/package.transform

Each line in the file specifies a transformation. A transformation consists of two space-separated fields: the full path of the target file including the displace extension and the transformation command itself. The transformation can either be a single shell command, or an executable file in the debian directory. The transformation takes the original source of the file on stdin and prints its transformation on stdout. Transformations are typically performed by perl, sed, or awk, but there is no limitation on what can be used as a transformation.

For example, to transform /etc/school.conf by replacing all occurrences of the word 'Harvard' with the word 'MIT', you might specify the following line:

/etc/school.conf.debathena sed -e 's/Harvard/MIT/g'

Or, storing the command in a separate script (which was the only option available with CDBS):

/etc/school.conf.debathena debian/transform_school.conf.debathena

If the transformation script fails, the package build fails. You can use this with e.g. Perl's or die syntax to make sure that the source file of the transformation has not changed from what you expected.

Under normal operation, the source (passed on stdin) for the transformation is the name of the diversion without the divert extension. In some cases, you may wish to use a different source (e.g. a sample configuration file in /usr/share/doc). You can specify this source as an optional field between the diversion filename and the transformation. This field must begin with a '<' immediately followed by the full path to the source. Taking the example above, we might alter it as follows:

/etc/school.conf.debathena </usr/share/doc/school/conf.example sed -e
's/Harvard/MIT/g'

Note that there is no “untransform“ operation. Because a transform operation is a special case of a displace operation, the “undisplace” operation is the correct way of removing a no-longer-needed transformation in future versions of the package.

debian/package.undisplace
List of absolute paths to files that were displaced in old versions of this package, but whose displace operations should be undone when this version of the package is installed. Each file should be listed as it was in debian/package.displace, namely full path including divert extension. This is primarily useful for removing a now-unecessary diversion from the package on an upgrade.
debian/package.unhide
As with the previous, but for undoing hide operations.

Examples

The config-package-dev package contains a number of simple examples ready for modification. We explain one in detail here.

Lynx is a text-mode web browser whose home page can be configured in the lynx.cfg global configuration file. We'd like to default the home page to MIT's, instead of to the Lynx web site.

One can implement a config-package-dev package to implement this change as follows (the parts of the files relevant to config-package-dev are highlighted in green):

debian/control:

Source: debathena-transform-example
Section: config
Priority: extra
Maintainer: Geoffrey Thomas <geofft@mit.edu>
Build-Depends: debhelper (>= 7.0.0~), config-package-dev (>= 5.0), lynx
Standards-Version: 3.9.2

Package: debathena-transform-example
Architecture: all
Depends: ${misc:Depends}, lynx
Provides: ${diverted-files}
Conflicts: ${diverted-files}
Description: Example config-package-dev package
 This is an example config-package-dev package.

debian/rules:

#!/usr/bin/make -f

%:
        dh $@ --with=config-package

debian/debathena-transform-example.transform:

/etc/lynx-cur/lynx.cfg.debathena perl -0pe 's|^#STARTFILE:.*$|$&\nSTARTFILE:http://web.mit.edu/|m or die'

The resulting package will, when installed, divert /etc/lynx-cur/lynx.cfg and replace it with (a symlink to) a file identical to the original lynx.cfg distributed by Debian, except with the home page set to http://web.mit.edu/.

You can browse the examples in the config-package-dev package here.

Motivation and philosophy

Configuration packages have several advantages over other approaches (such as the preseed file approach and Cfengine) for distributing site-wide configuration such as the MIT-specific configurations distributed as part of Debathena.

These properties make configuration packages effective for distributing site configurations at organizations on the scale of a university. They make it possible to configure site defaults so that interacting with any site services "just works", without restricting the ability of individual users at the site to control their machine.

Discussions of configuration packages on the Debian Wiki may also be of interest, as might be the thread on debian-devel discussing the weaknesses of the system and the technical motivation for the system..

A related project relevant for supporting multiple distributions is the Debathena build system which allows a single package-version pair to appear in an apt repository multiple times (we append tags like ~ubuntu7.04 to the version number at build time). It also contains code to automate the process of building packages for many different architectures; however, it is not documented for release. If you would like us to release it, send us an email at debathena@mit.edu.

Architecture

The config-package-dev system uses dpkg-divert to move aside configuration files. dpkg-divert is a standard Debian feature that configures dpkg, the Debian package manager, to move any file packaged to be installed at a given path to a different location. Using dpkg-divert for configuration files means that when Debian the upstream upgrades the relevant configuration file, they will not override the site changes. It also means that any Debian-wide updates will be preserved if the configuration package is ever uninstalled. Using a symlink here rather than a direct replacement is necessary for technical reasons related to how Debian unpacks files. The diversion system in config-package-dev has been carefully tested to support uninstallation, upgrades, and cleanly recovering from when you hit ^C in the middle of installing something (or installation fails for some other reason).

Our configuration system is implemented in a single Debhelper command, dh_configpackage. While the config-package sequencer extension module automatically, runs it, it can be used directly in old-style Debhelper rules files. You can also use a conditional override_dh_configpackage: Makefile target with debhelper (>= 7.0.50~) to call dh_configpackage with options.

All operations (displace, hide, transform, undisplace, unhide) run by default on each line of the file debian/package.operation. For the displace and hide operations, and for undoing them, these files are a list of paths, one per line; the transform operation has additional syntax to specify the transformation. The lists can also be provided on the command line by providing arguments to --operation option. Each time --operation is specified with an argument, it is interpreted as if it were a line in the debian/package.operation file. If any of these command-line options are present, then the files in debian/ are ignored for this run of dh_configpackage.

The displace operation

If file.debathena appears in this list, then on postinst, file will be diverted to file.debathena-orig, and a symlink file -> file.debathena will be installed. After using our system to displace /etc/ldap/ldap.conf, the state of the system will be:

$ ls -l /etc/ldap/ldap.conf*
lrwxrwxrwx 1 root root  19 2007-08-13 17:07 /etc/ldap/ldap.conf -> ldap.conf.debathena
-rw-r--r-- 1 root root 347 2007-08-15 17:58 /etc/ldap/ldap.conf.debathena
-rw-r--r-- 1 root root 333 2007-08-13 17:27 /etc/ldap/ldap.conf.debathena-orig

$ /usr/sbin/dpkg-divert --list | grep ldap
diversion of /etc/ldap/ldap.conf to /etc/ldap/ldap.conf.debathena-orig by debathena-ldap-config

Note that the displace extension can appear anywhere in the filename. Thus, /usr/share/man/man1/manpage.1.gz would be diverted to /usr/share/man/man1/manpage.debathena-orig.1.gz and a symlink created pointing /usr/share/man/man1/manpage.1.gz to /usr/share/man/man1/manpage.debathena.1.gz, so that man manpage.debathena-orig, man manpage.debathena, and man manpage all behave as expected.

Also note that when displacing files, you will need to create and install the replacement file file.debathena (probably using dh_install) yourself. See the examples package.install files below for how to do this.

The hide operation

The system just described does not work well for replacing files in directories such as /etc/cron.d where all files in the directory are used -- in the case of diverting /etc/cron.daily/lprng, the original cron job would get run and the new cron job would be run twice. You can banish such files into /usr/share/package with the hide operation. Fo example, in Debian etch, the lprng package contained a buggy cron job. After hiding the file in a package named lprng-no-cron, one has:

$ /usr/sbin/dpkg-divert --list | grep cron
diversion of /etc/cron.daily/lprng to /usr/share/lprng-no-cron/etc++cron.daily++lprng by lprng-no-cron

If you want to instead replace the cron job with a different one, you should just install the new cron just using dh_install (or in the case of cron, it’s probably better to use dh_installcron by naming your new cron file debian/foo.cron.daily). Do not give the replacement cron job the same name as the original one, instead give it a different name, like /etc/cron.daily/lprng.divert. If you do use the same name, it will not be possible to install your package in the same apt invocation that installs the package containing the original cron job.

The encoding of the filename using + characters shown in the last example is also used to ensure two packages that divert the same file conflict; if both packages divert /etc/cron.daily/lprng, they will both Provides: and Conflicts: configures-etc++cron.daily++lprng. The encoding used in both cases is a bijective map from legal ASCII filenames to package names. In order to enabled this you need to add ${diverted-files} to your Provides and Conflicts lines of your debian/control file; see the examples below for details.

The transform operation

The transform operation builds on the displace operation. There are three parts to each line: the file name of the form path/file.debathena, an optional source file for the transformation with '<' prepended, and the command to run for the transformation. dh_configpackage will check to make sure that path/file is unmodified (if necessary, it will check the origins of diversions, so it can be used even if path/file is currently displaced). It will then transform the unmodified configuration file by piping it through the transformation command, and install the result as path/file.debathena. Finally, it will displace path/file as usual. If the optional source file is given, then the transform operation will use that file as input to the transformation command.

Common Issues

Running code in a postinst script
If you want to run some other code in the postinst script of your configuration package, just write a normal postinst script, but include somewhere in the script the line
#DEBHELPER#
That line will be replaced with the auto-generated postinst code for the package, including the diversion code generated by config-package-dev. The example script /usr/share/debhelper/dh_make/debian/postinst.ex (provided in the dh-make package) uses this directive, but note that in several cases, you will want to move the line before your configuration code, for instance, if you want to restart a daemon using the new configuration. Code in the postinst before the #DEBHELPER# line will run before the diversions are in place.
Debian packages using debconf (or otherwise auto-generating configuration files)
Some packages auto-generate their configuration files on installation, so that those configuration files are not managed by dpkg. One should carefully read and test that package’s maintainer scripts (especially postinst) to verify that they will not override our dpkg-divert and symlink mechanism on package upgrades (otherwise, you should probably file a bug report). Also, such packages may not provide a clean upstream configuration file; however, one can often extract the clean upstream configuration from /var/lib/dpkg/info/package.postinst.
Debian packages using ucf
Some packages use ucf to check whether a configuration file shipped by the package has been modified to decide whether to replace it. Ideally, ucf should be changed to follow dpkg-divert when looking for configuration files (this is Debian #477773).

Contact

As of spring 2013, our mailing list is config-package-dev@mit.edu. You can also join the list or view the archive via Mailman.

This page was written by Tim Abbott with help from Anders Kaseorg, and significantly updated for Debhelper by Geoffrey Thomas, with some text taken from Jonathan Reed's man page for dh_configpackage.