source: trunk/debathena/NOTES @ 25465

Revision 25465, 25.7 KB checked in by jdreed, 12 years ago (diff)
The contents of NOTES are now in the wiki
2** NOTE **
4As of 24 April 2012, this file should be considered deprecated.  The
5canonical version of the information contained in this file is the
6Debathena Trac wiki (, in particular,
7the section titled "Release Engineering".  This file should be
8maintained for posterity, but should not be updated.
[22689]15This hierarchy contains Debian/Ubuntu-specific materials, also known
16as "Debathena".  The contents are:
18* debathena - Debathena-specific software packages such as PAM and NSS
19  modules.
21* config - Packages for configuring native system software in a manner
22  appropriate for Athena.
24* meta - Packages which contain nothing but dependencies on other
25  packages and serve as an installation convenience.
27* scripts - Build scripts and supporting materials.
29Debian software used by Debathena:
31  * schroot - Used to manage build chroot environments for each
[24193]32    Debian/Ubuntu version.  We use the device schroot type with aufs
33    union mounts, which allows rapid construction of ephemeral copies
34    of template "source" chroots, so that every binary package build
35    is done in a clean environment.
37  * debuild - Used to create Debian source packages from package
38    source directories.
40  * sbuild - Used to build binary packages from source packages inside
41    schroot environments.
43  * equivs - Used to create packages which only contain dependency
44    information.  Somewhat of a dirty hack, since it doesn't keep
45    proper changelogs, but it reduces overhead.
47  * CDBS (Common Debian Build System) - Referenced by debian/rules
48    files in packages.  Contains standard build rules to cut down on
49    per-package boilerplate.
51  * reprepro - Used to upload packages into the apt repositories.
53  * approx - Used to create a local cache of Debian packages on the
54    build server.  This cache is referenced by the build chroots for
55    improved performance.
[24928]57The remainder of this file documents procedures useful to Debathena
58developers and maintainers.
[22787]60Developers: Preferences setup
[22787]63You will probably want a $HOME/.devscripts file containing the
[22787]66DEBUILD_DPKG_BUILDPACKAGE_OPTS="-sa -us -uc -i -I.svn"
[22787]68This will save you from having to specify those options every time you
[24928]69run debuild.  The options mean:
[22787]71  * Look for original source as a tarfile or create one.
72  * Do not sign the source package.
73  * Do not sign the changes file.
74  * Ignore common version control metadata files when creating diffs.
75  * Ignore .svn paths when creating tarballs.
[23046]77You will also want a $HOME/.sbuildrc file containing the following:
79  $nolog = 1;
80  $mailto = 'yourusername';
81  $log_dir = '/tmp/sbuild-logs';
[23403]82  $maintainer_name = 'Debathena Project <>';
[23046]83  $force_orig_source = 1;
84  $sbuild_mode = "user";
85  1;
[22787]87You should also set the environment variable DEBATHENA_APT to
[22787]90Developers: Preparing a change
[22787]93To prepare a change to a regular package (a source tree containing a
94debian/ subdir), make the edits in a checkout and record a changelog
95entry.  You can either edit debian/changelog using emacs changelog
96mode (C-c C-v to add a new version entry, C-c C-a to add a change
[23752]97entry, C-c C-f to finalize the entry) or you can run "dadch" from
98the debathena locker / SVN scripts directory.
[22787]100When creating a new version entry, bump the upstream version number
101(to 10.0.0 if it was not already that high) if you are changing the
102main package source.  Otherwise, just bump the Debian version
103component (change 0debathena1 to 0debathena2, for instance).
[22787]105Developers: Building a package for test purposes on one platform
[22787]108After you have prepared a change, you will want to test that it builds
109and perhaps that it works before committing it.  First, if it is an
110Athena source directory using autoconf, run "daconfiscate" to set up
111the autoconf boilerplate which we don't check in.  Second, run
112"daorig" to copy or create an orig tarball in the parent directory if
113necessary.  Third, run "debuild".  The resulting package will be
114placed in the parent directory.
[22787]116In order to test if the package works, you can install it with "dpkg
117-i filename.deb".
[22787]119Developers: Building a package for test purposes on all platforms
[22787]122If the package you are working on interacts with the native OS in ways
123that might vary from platform to platform, you may want to do a test
124build for all platforms.  You will need to do this on or another machine which has been set up with
126build schroots.
128As above, run daconfiscate (if necessary) and then daorig.  Then run
129"debuild -S" to create a source package.  Now cd into the parent
130directory and identify the .dsc file created by debuild -S; it will
131have a name like debathena-just_9.4.0-0debathena2.dsc.  Run "da
132sbuildhack filename.dsc" to perform the package builds.  Each build
133will take place inside an ephemeral chroot based on a snapshot of a
134template for a particular Debian or Ubuntu version.  If a build fails
135and it's not obvious from the build log why, you may need to create
136your own ephemeral chroot session with a command like "schroot -c
137gutsy-amd64-sbuild /bin/sh" and then run debuild from within the
138package sources.
140If the build is successful, it will create a set of packages with
141names like debathena-just_9.4.0-0debathena2~ubuntu6.06_amd64.deb.
143Developers: Building an equivs package
146Most of the packages under debathena/meta are faked up using equivs.
147To build one, just run:
149  equivs-build --full filename.equivs
151These equivs files make reference to ../common, so you must have a
152checkout of debathena/meta/common alongside the particular
153meta-package you are building.
[23018]155Developers: The meaning of metapackages
158If you are adding a new package to the repository, you will probably
159at some point want to add it to one of the metapackages so that it
160doesn't have to be installed by hand.  Here are some descriptions
161which may help identify which metapackage is best:
163* locker: Provides access to Athena locker software--AFS and
164  automounter configuration, locker-related utilities, etc.
166* clients: Provides clients (either locally-written, like athinfo and
167  Discuss, or configurations) for Athena services, as well as
168  Athena-specific utility programs like "jot".  Configurations for
169  graphical client software are generally in the workstation package
170  instead, in order to make this package less intrusive.
172* standard: Implies locker and clients.  Also provides Athena shell
173  customizations and dotfiles.
175* login: Implies standard.  Configurations to merge the MIT user
176  namespace into the local machine namespace for the purpose of user
177  lookups and authentication.
[23752]179* login-graphical: Implies login.  Configurations for the graphical login
[23018]180  system and graphical client software intended to provide a standard
[23752]181  X login experience using Athena home directories.
[23247]183* extra-software: Implies a set of Debian packages common to cluster
184  machines and typical workstations.  The resulting software set is
185  rather large, and thus may not be desirable to all workstation
186  configurations.  Only stock Debian packages belong in this
187  metapackage; do not add other Debathena packages to it.
[23752]189* workstation: Implies login-graphical and extra-software.  The use case
190  is the same as that of the old "private workstations", so it includes
191  more centrally-managed configuration than standard or login does
192  (unattended updates, etc.). New as of May 2009.
[23752]194* thirdparty: Software supported by 3partysw that upstream Ubuntu
195  already happens to be keeping up to date, and is therefore easier to
196  install via packages than through lockers. This package is HUGE (1 GB of
197  network traffic, 3 GB installed), and is also not strongly tested on
198  Debian or on older releases of Ubuntu.
200* cluster: Implies workstation and thirdparty. Configures public cluster
201  machines (login-time LVM snapshots, cleanups between logins, public
202  root password, etc.). The snapshot code requires a particular LVM
203  layout that the PXE (netboot) cluster installer sets up.
[23018]205* debian-dev: Intended for developers of the system itself; provides a
206  set of Debian packages used by Debathena for development.
208For the most part a package should be listed in the "Depends:" line of
209a metapackage, but in some cases it is appropriate to hedge by using
210"Recommends:", which will cause aptitude to succeed even if the
211package is unavailable.  For example, a package which doesn't exist in
212all Debian/Ubuntu suites or isn't free can be listed under
213"Recommends:" so that our metapackages still work in all environments.
[23158]215Developers: Index of cluster packages
218Some private workstation admins may want specific pieces of cluster
219infrastructure.  With Athena 9.4, they would typically install the
220entire Athena software suite and then turn off features they did not
221want via /etc/athena/rc.conf.  That is not generally possible in
222Athena 10; a debathena-cluster machine acts like a cluster machine in
223all respects.  Instead, the machine owner should install
224debathena-workstation and then install specific packages they might be
225interested in.  At this time, candidate packages include:
227  * debathena-dns-config: Installs a caching DNS resolver and
228    configures the machine to use it.
230  * debathena-tmp-cleaner: Cleans files in /tmp and /var/tmp which
231    have not been recently accessed.
233  * debathena-reactivate: Causes each graphical login to be performed
234    in a separate ephemeral snapshot of the chroot.  This package
235    requires the root filesystem to be an LVM logical volume inside a
236    volume group with 21GB of free space for the snapshots; if that is
237    not the case, it will break graphical logins.  Install with care.
239  * debathena-auto-update: Periodically updates the machine's software
240    and reboots (when no one is logged in) if necessary.
242Cluster packages which are generally not of interest to private
243machine owners include:
[23177]245  * debathena-cluster-login-config: Configures a variety of system
246    services to implement the cluster login policy (no tty logins, no
247    user switching, screensaver logout button after 20 minutes, etc.).
[23158]249  * debathena-clusterinfo: Looks up the machine's cluster information
250    in Hesiod and caches it on local disk.
252  * debathena-larvnet: Reports to the central larvnet server whether
253    anyone has a graphical login to the machine.
[23162]255  * debathena-syslog-config: Configures sysklogd so that some syslog
256    messages are forwarded to a central logging host.
[22787]258Release engineer: Bootstrapping the project infrastructure
[22764]261  1. Create the package repository (detailed instructions on this
262     pending).  Set the DEBATHENA_APT environment variable to point to
263     the package repository.  Put a copy of the debathena "scripts"
264     subdir in your path.
266  2. Create the build area.
[24565]268  3. Run 'dareprepro export' to set up the repository.  Build each
269     equivs package under meta/ using 'equivs-build --full *.equivs'
270     and upload each with 'daupload-equivs-proposed *.changes',
271     followed by 'damove -proposed "" PACKAGE' (replacing PACKAGE with
272     the name of each equivs package you are moving).
274  4. Set up the build server.  The basic structure of the apt
275     repository must work for make-chroot to succeed, so this must
276     happen after step 3.
278  5. For each normal Debian package in dependency order, cd into its
279     directory in the build area and run "da sbuildhack *.dsc" and
[22787]280     "daupload-release *_source.changes".  If the package contains
281     only an "Architecture: all" binary package, pass the -A option to
282     both commands.
284     The all-packages script can generate an approximation of the
285     package list in dependency order, but it doesn't work right yet,
286     and ideally it would be possible to do several builds in parallel
287     using a Makefile like the one in scripts/build-server/build-all.
288     Improvements to this machinery are pending.
[22787]290  6. For each package under third, run "da ./debathenify-PKG source
[23084]291     binary upload".  Any created directories under third/openafs/meta
292     should be chmodded 777 to work around a perl/AFS permissions
293     issue with File::Temp; if this is not done, OpenAFS metapackage
294     builds will fail for other users.
[23090]296Release engineer: Updating the apt repository
299  1. ssh to the build server as the builder account and change to the
300     canonical build directory.
302  2. Run "gen-packages" to update the package list.  (Or "gen-packages
303     -c" if you know the AFS checkout of the source tree is up to date;
304     it should update every half hour.)
306  3. Run "ood-packages" to produce a list of out-of-date packages.
308  4. For each out of date package, run "dasource PKG".  Then change to
309     the package directory and run "da sbuildhack *.dsc" and
310     "daupload-release *_source.changes".  If the package contains
311     only an "Architecture: all" binary package, pass the -A option to
312     both commands.
314  5. svn update the meta directory.  If there are new subdirectories,
315     chmod them 777 to work around a perl/AFS permissions issue with
316     File::Temp.  For each updated subdir, change to it, run
317     "equivs-build --full *.equivs", and then "daequivsupload
318     FILENAME.changes" on the produced changes file.
320  6. svn update the third directory.  You can let autodebathenify
321     handle the updated scripts, or you can touch
322     ~/autodebathenify.suppress, make sure it's not running, and run
323     "da ./debathenify-PKG source binary upload" in each updated
324     directory.
326Sometimes you may have to mix up the order of the above steps in order
327to handle build dependencies.
[22787]329Release engineer: Setting up a build server
[23752]332  1. Install either Debian testing (or stable, if it's new enough, but
333     see the note about sbuild below) or the latest Ubuntu release.
334     The build server must be installed with free space in an LVM
[24237]335     volume group.  The build chroots consume 4GB each.
[24237]337  2. Install debathena-login as per the the instructions in
[24237]340  3. apt-key add
341     /afs/
[23248]343  4. Install the packages listed in scripts/build-server/packages
344     (using "aptitude install").
[23752]346     Note that currently the build system uses the latest version of
[24193]347     sbuild, so if you have not installed Debian testing, you'll need
348     to set up apt pinning to grab sbuild from there. See the
349     apt_preferences man page. You'll also need to grab schroot from
350     Debian experimental.
[24237]352  5. Install the aufs-modules-2.6-amd64 package.
[24237]354  6. Append to /etc/approx/approx.conf the contents of
[23752]355      scripts/build-server/approx.conf.tail.
356     Run: /etc/init.d/approx restart
[24303]358  7. Apply scripts/build-server/mount-defaults.patch and
359     scripts/build-server/pam-schroot.patch.
[24237]361  8. For each supported DIST (see scripts/ run:
[24928]363       scripts/build-server/make-chroot DIST i386
364       scripts/build-server/make-chroot DIST amd64
[24928]366     Example: scripts/build-server/make-chroot intrepid i386
[24237]368  9. Create a local account for builder with:
[24237]370       adduser --uid 1047 --disabled-password builder
[24237]372     Make the home directory mode 700.  Install a
373     daemon/ keytab in the home directory as
374     "keytab".  Install a copy of the secret repository-signing key
375     ( in the home directory's keyring with
376     something like:
[24237]378       kinit builder
[24262]379       gpg --export-secret-keys | \
[24237]380         ssh -l builder machinename gpg --import
[24237]382     Create a file named .sbuildrc in builder's homedir containing:
[24262]384       $mailto = undef;
[24237]385       $log_dir = '/tmp/sbuild-logs';
386       $maintainer_name = 'Debathena Project <>';
387       $force_orig_source = 1;
388       $sbuild_mode = "user";
389       1;
[24237]391     Create a file named .ssh/config in builder's homedir containing:
[24237]393       Host
394         User debuildsvn
[24237]396     Add builder to the sbuild group in /etc/group.
[24237]398     Copy scripts/build-server/autodebathenify to builder's homedir.
399     Create a file named autodebathenify.config in builder's homedir
400     containing:
403       scripts_dir=/mit/debathena/bin
404       build_dir=/mit/debathena/packages/third
405       packages="lprng openafs"
406       export DEBATHENA_APT=/mit/debathena/apt
[24237]408     Copy scripts/build-server/autodebathenify.cron to builder's
409     homedir and install it with "crontab autodebathenify.cron".
[24262]411     Copy scripts/build-server/autolivebuilder to builder's
412     homedir. Create a file called autolivebuilder.config in builder's
413     homedir containing:
416       release_version='9.04'
417       release='jaunty'
418       arch='i386'
419       mirror=''
420       gpg_opts=("-u" "0D8A9E8F")
421       live_dir=/net/
[24237]423     Create a file named .devscripts in builder's homedir containing:
[24237]425       DEBUILD_DPKG_BUILDPACKAGE_OPTS="-sa -us -uc -i -I.svn"
[24237]427     In builder's homedir, append to .bashrc:
[24237]429       add debathena
430       export PATH=$PATH:~/bin
431       export DEBATHENA_APT=/mit/debathena/apt
[22787]434Release engineer: Removing a build chroot on the build server
[22787]437  1. Run VG=/dev/blah scripts/clean-schroots as root to make sure that
438     the build chroot is not mounted, substituting the name of the
439     volume group for blah.
[25325]441  2. Look in /etc/schroot/chroot.d/ and delete the file
[22729]442     corresponding to the chroot.
444  3. Run lvchange -an blah/chrootname
445     substituting the name of the volume group for blah and the chroot
446     name for chroot.  Example: lvchange -an dink/gutsy-i386-sbuild
[22764]448  4. Run lvremove blah/chrootname
[22985]450Release engineer: Removing a dist from the apt repository
453  1. Inside the apt repository, edit conf/distributions and remove the
454     distribution section.
[24565]456  2. Run dareprepro --delete clearvanished
[22787]458Release engineer: Setting up a canonical build area
461  1. Create an empty directory and cd into it.  The canonical build
[23752]462     area lives in /mit/debathena/packages/.
[22787]464  2. Run gen-packages to create the table of normal Debian packages.
[22787]466  3. Run dasource to create subdirs and source packages for each
467     normal Debian package.
469  4. Create checkouts of the meta and third directories:
471     svn co svn+ssh://
472     svn co svn+ssh://
[23084]473     chmod 777 meta/*
[23084]475     A couple of subdirectories of debathena/meta are normal Debian
[22764]476     packages, so this will create redundant copies of those.  Ignore
[23084]477     them; they won't be used.
[24928]479Release engineer: Adding a new suite or rebuilding a suite
482This process is rarely performed and the infrastructure for it is
[24928]483imperfect.  Substitute the name of the new suite for $suite in all
[22957]484steps below.
[22958]486  1. Make sure the apt repository is up to date with respect to the
[24875]487     source tree for the existing dists. Check
488 and
489 More importantly, make
490     sure there is nothing in svn that isn't built. (An easy way to do
491     this is running the check-unbuilt-packages script.)
[25325]493  2. Add the new dist to scripts/ (and svn up into
494     the locker).  (It is not necessary to add the new dist to codes
495     at this point, but it must be present in the gettag conditional.)
496     If you are building an unreleased distribution, add an additional
497     ~0.1 tag; if you're rebilding, bump that tag.
499  3. Create the new distribution in the apt repository's configuration
[24928]500     file by editing /mit/debathena/apt/conf/gen-distributions and
501     running it with output to the 'distributions' file in the same
[25325]502     directory.  Create the skeleton of the dist by running
503     "dareprepro export" on the build server. (You don't need to do
504     this if you're rebuilding.)
[24928]506  4. Add an appropriate sources.list.d/*/$suite.list entry in
507     scripts/build-server. Commit and update
508     /mit/debathena/bin/build-server.
[25325]510  5. On the build server (as root), create a chroot for the new distribution:
[25325]512     First, check if /usr/share/debootstrap/scripts/$suite exists.
513     If not, it may require downloading and installing a
[22957]514     more recent version of the debootstrap package from the
[24928]515     -backports dist corresponding to the build server's OS. (You
516     presumably don't need to do this if you're rebuilding.)
[25325]518     You could also just go into that directory and do
519     "ln -s gutsy $suite" since it apparently hasn't changed in years.
521        /mit/debathena/bin/build-server/make-chroot $suite i386
522        /mit/debathena/bin/build-server/make-chroot $suite amd64
[25402]525  6. Fire up screen, ideally making use of the "-S" option to identify
526     your screen session.  e.g. "screen -S joeuser.oneiric_build".
[24928]528  7. mkdir /mit/debathena/machines/awesome-build-server/stamps.$suite.
[25402]530  8. cd into a checkout (probably on your local machine) of
531     debathena/scripts/build-server/build-all.
[22958]533  9. Edit Makefile (and check in the edit) so that suite is the new
[22957]534     distribution and psuite is the previously most recent Debian or
[24928]535     Ubuntu distribution.  psuite is used to generate the order of
536     build-dependencies, so choose the closest distribution.  If you are
537     rebuilding, you can set psuite to suite.
[25402]539  10. cd to /mit/debathena/bin/build-server/build-all, which is a
540      totally logical place to run the build from.  Run "make
[24928]543  11. Run "make -k all".
[22958]544      You can watch the builds happen in the other windows of the
545      screen session.  It's possible to do several builds at once with
[25325]546      make -j N (except as of 7/2011 this is a bad idea)
[23094]548Release engineer: Maintaining autodebathenify
[23185]551autodebathenify is a cron job which runs on the builder account and
552keeps the OpenAFS modules and modified packages under debathena/third
553up to date when there are upstream changes.  It relies on the
556  * builder@ATHENA.MIT.EDU has access to a local account named
557    debuildsvn on with read access to the athena
558    repository.
560  * builder@ATHENA.MIT.EDU has write access to the canonical build
561    area and apt repository in the dev cell.
563  * The builder account on the build machine has a copy of the signing
564    private key for the apt repository.
566The cron job will silently exit if it detects that it is already
567running.  If it fails for any other reason, it will send mail to and touch a file named autodebathenify.suppress
569which will prevent it from running again until manual intervention.
570It is important to get autodebathenify running smoothly or the apt
571repository will become out of date with respect to third-party
572packages, which in turn will compromise the user experience.
574Even if you don't notice failure mail, it's good to check on the
575status of autodebathenify from time to time.  To do this, get tickets
576as builder and ssh to linux-build-10 as builder.  Run "ls -l logs" to
577see what's been going on.  A logfile with size around 48K indicates
578that autodebathenify ran normally but didn't find any new work to do.
579A logfile with size around 60 means that autodebathenify.suppress
580exists and the cron job has stopped running.  A longer logfile
581indicates that autodebathenify attempted to build and upload a
584Possible failure cases include:
586  * One of the upstream apt repositories timed out during an "apt-get
[23119]587    update" of an sbuild chroot.  If this happens, you can just remove
588    the autodebathenify.suppress file to get the cron job to start
589    running again.
591  * autodebathenify tried to build an upstream version which already
592    exists.  If this happens, the build will fail at upload time.  If
593    this happens, there is a bug in the part of the debathenify-*
594    script which checks whether our apt repository already has the
595    expected version of the built package.
597  * Debian and Ubuntu have .orig files with the same name but
598    different contexts for a particular third-party package.  This has
599    only happened once, with bash (which is currently not configured
600    into autodebathenify for that reason).  We do not have a general
601    mechanism for resolving such issues at this time.
603  * The new upstream version of the package is sufficiently different
604    to cause our modifications to fail to apply, and our debathenify-*
605    script needs to be adapted appropriately.
607The cron job is run out of builder's home directory, but its canonical
608source location is in debathena/scripts/build-server in this
609repository, should it need to be modified.
[23191]611After several months of operation, LVM snapshots on the build server
612will start to become slower and slower, as /etc/lvm/cache/.cache grows
613in size.  Turning off the cache entirely seems to cause schroot to
614occasionally fail; the current solution is to rm /etc/lvm/cache/.cache
615every so often.
[23077]617Release engineer: apt repository HTTP server setup
[23752]620The apt repository server ( is hosted on out of the debathena locker. To replicate it on a
622dedicated server, install Apache and AFS (e.g., debathena-standard
623and apache2), and configure the document root to be
624  /mit/debathena/web_scripts
[23752]626There's also a crontab in /mit/debathena/cron_scripts that updates
627the package lists on the website. It expects to be able to edit the
628web_scripts directory; on scripts, it does so with daemon.scripts
631Trac ( expects to use
632for its database.
Note: See TracBrowser for help on using the repository browser.