source: trunk/debathena/NOTES @ 23119

Revision 23119, 22.5 KB checked in by ghudson, 16 years ago (diff)
Alter the autodebathenify maintenance documentation now that most "apt-get update" failures won't result in suppression of future runs.
[22689]1This hierarchy contains Debian/Ubuntu-specific materials, also known
2as "Debathena".  The contents are:
4* debathena - Debathena-specific software packages such as PAM and NSS
5  modules.
7* config - Packages for configuring native system software in a manner
8  appropriate for Athena.
10* meta - Packages which contain nothing but dependencies on other
11  packages and serve as an installation convenience.
13* scripts - Build scripts and supporting materials.
15Debathena is a SIPB project, and its infrastructure and procedures
16will need to be adapted for Athena 10.  For the moment this file will
17document the Debathena procedures as they are, not as they will be.
18The current procedures do not even use this svn repository yet.
20Debian software used by Debathena:
22  * schroot - Used to manage build chroot environments for each
23    Debian/Ubuntu version.  We use the lvm-snapshot schroot type,
24    which allows rapid construction of ephemeral copies of template
25    "source" chroots, so that every binary package build is done in a
26    clean environment.
28  * debuild - Used to create Debian source packages from package
29    source directories.
31  * sbuild - Used to build binary packages from source packages inside
32    schroot environments.
34  * equivs - Used to create packages which only contain dependency
35    information.  Somewhat of a dirty hack, since it doesn't keep
36    proper changelogs, but it reduces overhead.
38  * CDBS (Common Debian Build System) - Referenced by debian/rules
39    files in packages.  Contains standard build rules to cut down on
40    per-package boilerplate.
42  * reprepro - Used to upload packages into the apt repositories.
44  * approx - Used to create a local cache of Debian packages on the
45    build server.  This cache is referenced by the build chroots for
46    improved performance.
[22787]48The remainder of this file documents procedures useful to Athena 10
49developers and the release engineer.
[22787]51Developers: Preferences setup
[22787]54You will probably want a $HOME/.devscripts file containing the
[22787]57DEBUILD_DPKG_BUILDPACKAGE_OPTS="-sa -us -uc -i -I.svn"
[22787]59This will save you from having to specify those options every time you
60run debuild.  Athena 10 scripts do not assume the above preferences,
61but the instructions in this file do.  The options mean:
[22787]63  * Look for original source as a tarfile or create one.
64  * Do not sign the source package.
65  * Do not sign the changes file.
66  * Ignore common version control metadata files when creating diffs.
67  * Ignore .svn paths when creating tarballs.
[23046]69You will also want a $HOME/.sbuildrc file containing the following:
71  $nolog = 1;
72  $mailto = 'yourusername';
73  $log_dir = '/tmp/sbuild-logs';
74  $maintainer_name = 'Debian-Athena Project <>';
75  $force_orig_source = 1;
76  $sbuild_mode = "user";
77  1;
[22787]79You should also set the environment variable DEBATHENA_APT to
[22787]82Developers: Preparing a change
[22787]85To prepare a change to a regular package (a source tree containing a
86debian/ subdir), make the edits in a checkout and record a changelog
87entry.  You can either edit debian/changelog using emacs changelog
88mode (C-c C-v to add a new version entry, C-c C-a to add a change
89entry, C-c C-f to finalize the entry) or you can run "dadch".
[22787]91When creating a new version entry, bump the upstream version number
92(to 10.0.0 if it was not already that high) if you are changing the
93main package source.  Otherwise, just bump the Debian version
94component (change 0debathena1 to 0debathena2, for instance).
[22787]96Developers: Building a package for test purposes on one platform
[22787]99After you have prepared a change, you will want to test that it builds
100and perhaps that it works before committing it.  First, if it is an
101Athena source directory using autoconf, run "daconfiscate" to set up
102the autoconf boilerplate which we don't check in.  Second, run
103"daorig" to copy or create an orig tarball in the parent directory if
104necessary.  Third, run "debuild".  The resulting package will be
105placed in the parent directory.
[22787]107In order to test if the package works, you can install it with "dpkg
108-i filename.deb".
[22787]110Developers: Building a package for test purposes on all platforms
[22787]113If the package you are working on interacts with the native OS in ways
114that might vary from platform to platform, you may want to do a test
115build for all platforms.  You will need to do this on or another machine which has been set up with
117build schroots.
119As above, run daconfiscate (if necessary) and then daorig.  Then run
120"debuild -S" to create a source package.  Now cd into the parent
121directory and identify the .dsc file created by debuild -S; it will
122have a name like debathena-just_9.4.0-0debathena2.dsc.  Run "da
123sbuildhack filename.dsc" to perform the package builds.  Each build
124will take place inside an ephemeral chroot based on a snapshot of a
125template for a particular Debian or Ubuntu version.  If a build fails
126and it's not obvious from the build log why, you may need to create
127your own ephemeral chroot session with a command like "schroot -c
128gutsy-amd64-sbuild /bin/sh" and then run debuild from within the
129package sources.
131If the build is successful, it will create a set of packages with
132names like debathena-just_9.4.0-0debathena2~ubuntu6.06_amd64.deb.
134Developers: Building an equivs package
137Most of the packages under debathena/meta are faked up using equivs.
138To build one, just run:
140  equivs-build --full filename.equivs
142These equivs files make reference to ../common, so you must have a
143checkout of debathena/meta/common alongside the particular
144meta-package you are building.
[23018]146Developers: The meaning of metapackages
149If you are adding a new package to the repository, you will probably
150at some point want to add it to one of the metapackages so that it
151doesn't have to be installed by hand.  Here are some descriptions
152which may help identify which metapackage is best:
154* locker: Provides access to Athena locker software--AFS and
155  automounter configuration, locker-related utilities, etc.
157* clients: Provides clients (either locally-written, like athinfo and
158  Discuss, or configurations) for Athena services, as well as
159  Athena-specific utility programs like "jot".  Configurations for
160  graphical client software are generally in the workstation package
161  instead, in order to make this package less intrusive.
163* standard: Implies locker and clients.  Also provides Athena shell
164  customizations and dotfiles.
166* login: Implies standard.  Configurations to merge the MIT user
167  namespace into the local machine namespace for the purpose of user
168  lookups and authentication.
170* workstation: Implies login.  Configurations for the graphical login
171  system and graphical client software intended to provide a standard
172  X login experience using Athena home directories.  Still in
173  development.
175* cluster-software: Provides a set of Debian packages common to
176  cluster machines.  The resulting software set is rather large, and
177  thus may not be desirable to all workstation configurations.  Only
178  stock Debian packages belong in this metapackage; do not add
179  other Debathena packages to it.
181* cluster: Implies workstation and cluster-software.  Also contains
182  configurations for self-maintenance of machines (unattended updates,
[23102]183  cleanups between logins, etc.).
185* debian-dev: Intended for developers of the system itself; provides a
186  set of Debian packages used by Debathena for development.
188For the most part a package should be listed in the "Depends:" line of
189a metapackage, but in some cases it is appropriate to hedge by using
190"Recommends:", which will cause aptitude to succeed even if the
191package is unavailable.  For example, a package which doesn't exist in
192all Debian/Ubuntu suites or isn't free can be listed under
193"Recommends:" so that our metapackages still work in all environments.
[22787]195Release engineer: Bootstrapping the project infrastructure
[22764]198  1. Create the package repository (detailed instructions on this
199     pending).  Set the DEBATHENA_APT environment variable to point to
200     the package repository.  Put a copy of the debathena "scripts"
201     subdir in your path.
203  2. Create the build area.
205  3. Build each equivs package under meta/ using "equivs-build --full
206     *.equivs" and upload each with "daequivsupload *.changes".  This
207     has the side-effect of creating the basic structure of the
208     package repository.
210  4. Set up the build server.  The basic structure of the apt
211     repository must work for make-chroot to succeed, so this must
212     happen after step 3.
214  5. For each normal Debian package in dependency order, cd into its
215     directory in the build area and run "da sbuildhack *.dsc" and
[22787]216     "daupload-release *_source.changes".  If the package contains
217     only an "Architecture: all" binary package, pass the -A option to
218     both commands.
220     The all-packages script can generate an approximation of the
221     package list in dependency order, but it doesn't work right yet,
222     and ideally it would be possible to do several builds in parallel
223     using a Makefile like the one in scripts/build-server/build-all.
224     Improvements to this machinery are pending.
[22787]226  6. For each package under third, run "da ./debathenify-PKG source
[23084]227     binary upload".  Any created directories under third/openafs/meta
228     should be chmodded 777 to work around a perl/AFS permissions
229     issue with File::Temp; if this is not done, OpenAFS metapackage
230     builds will fail for other users.
[23090]232Release engineer: Updating the apt repository
235  1. ssh to the build server as the builder account and change to the
236     canonical build directory.
238  2. Run "gen-packages" to update the package list.  (Or "gen-packages
239     -c" if you know the AFS checkout of the source tree is up to date;
240     it should update every half hour.)
242  3. Run "ood-packages" to produce a list of out-of-date packages.
244  4. For each out of date package, run "dasource PKG".  Then change to
245     the package directory and run "da sbuildhack *.dsc" and
246     "daupload-release *_source.changes".  If the package contains
247     only an "Architecture: all" binary package, pass the -A option to
248     both commands.
250  5. svn update the meta directory.  If there are new subdirectories,
251     chmod them 777 to work around a perl/AFS permissions issue with
252     File::Temp.  For each updated subdir, change to it, run
253     "equivs-build --full *.equivs", and then "daequivsupload
254     FILENAME.changes" on the produced changes file.
256  6. svn update the third directory.  You can let autodebathenify
257     handle the updated scripts, or you can touch
258     ~/autodebathenify.suppress, make sure it's not running, and run
259     "da ./debathenify-PKG source binary upload" in each updated
260     directory.
262Sometimes you may have to mix up the order of the above steps in order
263to handle build dependencies.
[22787]265Release engineer: Setting up a build server
[22729]268  1. The build server must be installed with free space in an LVM
[22787]269     volume group.  The build chroots consume 2GB each.  There is a
270     known memory corruption issue with LVM snapshots in the kernel
271     used in Ubuntu Gutsy (which is based on 2.6.22), so use a newer
272     kernel such as the one in Ubuntu Hardy (based on 2.6.24) instead.
[22743]274  2. Install debathena-standard as per the the instructions in
[22743]277  3. apt-key add /afs/
279  4. Install the packages listed in
[22729]280     scripts/build-server/packages (using "aptitude install")
[22743]282  5. Install debathena-login, debathena-ssh-server, and
[22729]283     debathena-build-depends (using "aptitude install").
285     (Depending on how recently debathena-build-depends was rebuilt,
286     additional packages might need to be installed to satisfy the
287     build-depends of newer packages.  This can be taken care of later
288     when an error occurs building a source package.)
[22743]290  6. Edit /etc/security/access.conf and add a first line:
[22729]291     -:ALL EXCEPT root <developer usernames>:ALL
[22788]293  7. Edit /etc/pam.d/schroot, comment out "@include common-session",
[22787]294     and add:
[22787]296       # Basic pam_unix session module in place of common-session.
297       session required
299  8. Edit /etc/group and add the developers to the sbuild group.
301  9. Create /etc/passwd entries for each developer with "hesinfo
[22729]302     username passwd >> /etc/passwd" and then run pwconv.
304     (This is not necessary for the login system on the main root
305     environment, but is for the chroot environments.)
[22787]307  10. Append to /etc/approx/approx.conf the contents of
308       scripts/build-server/approx.conf.tail.
309      Change the last line from to
310       file:///afs/
311      Add "$interval 0" above the repository lines (only necessary if
312       the version of approx as reported by "dpkg -l approx" is less
313       than 3.0)
314      Run: /etc/init.d/approx restart
[22984]316  11. Apply scripts/build-server/mount-defaults.patch.
[22787]318  12. For each supported DIST (see scripts/ run:
[22743]320        VG=/dev/blah scripts/build-server/make-chroot DIST i386
321        VG=/dev/blah scripts/build-server/make-chroot DIST amd64
[22743]323      substituting the name of the volume group for blah.  Omit the
324      amd64 line if DIST is sarge.
[22743]326      Example: VG=/dev/dink scripts/build-server/make-chroot gutsy i386
[23046]328  13. Create a local account for builder with:
330        adduser --uid 1047 --disabled-password builder
332      Make the home directory mode 700.  Install a
333      daemon/ keytab in the home directory as
334      "keytab".  Install a copy of the secret repository-signing key
335      ( in the home directory's keyring with
336      something like:
338        kinit builder
339        gpg --export-secret-keys | \
340          ssh -l builder machinename gpg --import
342      Create a file named .sbuildrc in builder's homedir containing:
344        $nolog = 1;
345        $mailto = '';
346        $log_dir = '/tmp/sbuild-logs';
347        $maintainer_name = 'Debian-Athena Project <>';
348        $force_orig_source = 1;
349        $sbuild_mode = "user";
350        1;
[23050]352      Create a file named .ssh/config in builder's homedir containing:
354        Host
355          User debuildsvn
[23046]357      Add builder to the sbuild group in /etc/group.
359      Copy scripts/build-server/autodebathenify to builder's homedir.
360      Create a file named autodebathenify.config in builder's homedir
361      containing:
364        scripts_dir=/afs/
365        build_dir=/afs/
366        packages="cyrus-sasl2-mit evolution-data-server lprng openafs tcsh"
367        export DEBATHENA_APT=/afs/
369      Copy scripts/build-server/autodebathenify.cron to builder's
370      homedir and install it with "crontab autodebathenify.cron".
[23085]372      Create a file named .devscripts in builder's homedir containing:
374        DEBUILD_DPKG_BUILDPACKAGE_OPTS="-sa -us -uc -i -I.svn"
[23065]376      In builder's homedir, append to .profile:
378        PATH=${PATH}:/afs/
379        export DEBATHENA_APT=/afs/
381      and to .bashrc:
383        bld=/afs/
[22787]385Release engineer: Removing a build chroot on the build server
[22787]388  1. Run VG=/dev/blah scripts/clean-schroots as root to make sure that
389     the build chroot is not mounted, substituting the name of the
390     volume group for blah.
392  2. Edit /etc/schroot/schroot.conf and delete the section
393     corresponding to the chroot.
395  3. Run lvchange -an blah/chrootname
396     substituting the name of the volume group for blah and the chroot
397     name for chroot.  Example: lvchange -an dink/gutsy-i386-sbuild
[22764]399  4. Run lvremove blah/chrootname
[22985]401Release engineer: Removing a dist from the apt repository
404  1. Inside the apt repository, edit conf/distributions and remove the
405     distribution section.
407  2. Run reprepro -Vb $DEBATHENA_APT --delete clearvanished
[22787]409Release engineer: Setting up a canonical build area
412  1. Create an empty directory and cd into it.  The canonical build
413     area lives in /afs/
[22787]415  2. Run gen-packages to create the table of normal Debian packages.
[22787]417  3. Run dasource to create subdirs and source packages for each
418     normal Debian package.
420  4. Create checkouts of the meta and third directories:
422     svn co svn+ssh://
423     svn co svn+ssh://
[23084]424     chmod 777 meta/*
[23084]426     A couple of subdirectories of debathena/meta are normal Debian
[22764]427     packages, so this will create redundant copies of those.  Ignore
[23084]428     them; they won't be used.
430Release engineer: Adding a new suite
433This process is rarely performed and the infrastructure for it is
434imperfect.  Substitute the name of the new suite for "newdist" in all
435steps below.
[22958]437  1. Make sure the apt repository is up to date with respect to the
438     source tree for the existing dists.
440  2. Add the new dist to scripts/  (It is not
441     necessary to add the new dist to codes at this point, but it must
442     be present in the gettag conditional.)
444  3. Create the new distribution in the apt repository's configuration
[22957]445     file.  Create the skeleton of the dist by installing at least one
446     equivs package from meta/ with "reprepro -Vb $DEBATHENA_APT
447     include newdistname file.changes".
[22958]449  4. On the build server, create a chroot for the new distribution as
[22957]450     documented above.  This may require downloading and installing a
451     more recent version of the debootstrap package from the
452     -backports dist corresponding to the build server's OS.
[22958]454  5. Set the DEBATHENA_BUILD_AREA environment variable to point to the
[22957]455     build area.
[22958]457  6. Fire up screen.
[22958]459  7. mkdir $DEBATHENA_BUILD_AREA/stamps.newdist.
[22958]461  8. cd into a checkout of debathena/scripts/build-server/build-all.
[22958]463  9. Edit Makefile (and check in the edit) so that suite is the new
[22957]464     distribution and psuite is the previously most recent Debian or
465     Ubuntu distribution.
[22958]467  10. Run "make".
[22958]469  11. Run "make -k all STAMPS=$DEBATHENA_BUILD_AREA/stamps.newdist".
470      You can watch the builds happen in the other windows of the
471      screen session.  It's possible to do several builds at once with
472      make -j N.
[22958]474  12. debathenify packages will fail out; they must be built by hand.
[22957]475      When the build fails on one, cd into third/packagename in the
476      build area and run "./debathenify newdist-amd64 -A source binary
477      upload" and "./debathenify newdist-i386 binary upload".  Then
478      touch $DEBATHENA_BUILD_AREA/stamps.newdist/packagename.done" and
479      restart the build.
[23084]481      The newly created third/openafs/meta directories should be
482      chmodded 777 to work around a perl/AFS permissions issue.
[23094]484Release engineer: Maintaining autodebathenify
487autodebathenify is a cron job which runs hourly on the builder account
488and keeps the OpenAFS modules and modified packages under
489debathena/third up to date when there are upstream changes.  It relies
490on the following:
492  * builder@ATHENA.MIT.EDU has access to a local account named
493    debuildsvn on with read access to the athena
494    repository.
496  * builder@ATHENA.MIT.EDU has write access to the canonical build
497    area and apt repository in the dev cell.
499  * The builder account on the build machine has a copy of the signing
500    private key for the apt repository.
502The cron job will silently exit if it detects that it is already
503running.  If it fails for any other reason, it will send mail to and touch a file named autodebathenify.suppress
505which will prevent it from running again until manual intervention.
506It is important to get autodebathenify running smoothly or the apt
507repository will become out of date with respect to third-party
508packages, which in turn will compromise the user experience.
510Even if you don't notice failure mail, it's good to check on the
511status of autodebathenify from time to time.  To do this, get tickets
512as builder and ssh to linux-build-10 as builder.  Run "ls -l logs" to
513see what's been going on.  A logfile with size around 48K indicates
514that autodebathenify ran normally but didn't find any new work to do.
515A logfile with size around 60 means that autodebathenify.suppress
516exists and the cron job has stopped running.  A longer logfile
517indicates that autodebathenify attempted to build and upload a
520Possible failure cases include:
522  * One of the upstream apt repositories timed out during an "apt-get
[23119]523    update" of an sbuild chroot.  If this happens, you can just remove
524    the autodebathenify.suppress file to get the cron job to start
525    running again.
527  * autodebathenify tried to build an upstream version which already
528    exists.  If this happens, the build will fail at upload time.  If
529    this happens, there is a bug in the part of the debathenify-*
530    script which checks whether our apt repository already has the
531    expected version of the built package.
533  * Debian and Ubuntu have .orig files with the same name but
534    different contexts for a particular third-party package.  This has
535    only happened once, with bash (which is currently not configured
536    into autodebathenify for that reason).  We do not have a general
537    mechanism for resolving such issues at this time.
539  * The new upstream version of the package is sufficiently different
540    to cause our modifications to fail to apply, and our debathenify-*
541    script needs to be adapted appropriately.
543The cron job is run out of builder's home directory, but its canonical
544source location is in debathena/scripts/build-server in this
545repository, should it need to be modified.
[23077]547Release engineer: apt repository HTTP server setup
550The apt repository server ( is an ops virtual image
551with httpd and AFS installed.  At the moment, the only customization
552is two changes to /etc/httpd/conf/httpd.conf:
554  * The DocumentRoot is set to "/afs/".
555  * The Directory entry for /var/html/www is also changed to
556    "/afs/".
Note: See TracBrowser for help on using the repository browser.