source: trunk/debathena/NOTES @ 25325

Revision 25325, 25.1 KB checked in by jdreed, 13 years ago (diff)
Clarify some stuff about deleting/staging a chroot
[22689]1This hierarchy contains Debian/Ubuntu-specific materials, also known
2as "Debathena".  The contents are:
4* debathena - Debathena-specific software packages such as PAM and NSS
5  modules.
7* config - Packages for configuring native system software in a manner
8  appropriate for Athena.
10* meta - Packages which contain nothing but dependencies on other
11  packages and serve as an installation convenience.
13* scripts - Build scripts and supporting materials.
15Debian software used by Debathena:
17  * schroot - Used to manage build chroot environments for each
[24193]18    Debian/Ubuntu version.  We use the device schroot type with aufs
19    union mounts, which allows rapid construction of ephemeral copies
20    of template "source" chroots, so that every binary package build
21    is done in a clean environment.
23  * debuild - Used to create Debian source packages from package
24    source directories.
26  * sbuild - Used to build binary packages from source packages inside
27    schroot environments.
29  * equivs - Used to create packages which only contain dependency
30    information.  Somewhat of a dirty hack, since it doesn't keep
31    proper changelogs, but it reduces overhead.
33  * CDBS (Common Debian Build System) - Referenced by debian/rules
34    files in packages.  Contains standard build rules to cut down on
35    per-package boilerplate.
37  * reprepro - Used to upload packages into the apt repositories.
39  * approx - Used to create a local cache of Debian packages on the
40    build server.  This cache is referenced by the build chroots for
41    improved performance.
[24928]43The remainder of this file documents procedures useful to Debathena
44developers and maintainers.
[22787]46Developers: Preferences setup
[22787]49You will probably want a $HOME/.devscripts file containing the
[22787]52DEBUILD_DPKG_BUILDPACKAGE_OPTS="-sa -us -uc -i -I.svn"
[22787]54This will save you from having to specify those options every time you
[24928]55run debuild.  The options mean:
[22787]57  * Look for original source as a tarfile or create one.
58  * Do not sign the source package.
59  * Do not sign the changes file.
60  * Ignore common version control metadata files when creating diffs.
61  * Ignore .svn paths when creating tarballs.
[23046]63You will also want a $HOME/.sbuildrc file containing the following:
65  $nolog = 1;
66  $mailto = 'yourusername';
67  $log_dir = '/tmp/sbuild-logs';
[23403]68  $maintainer_name = 'Debathena Project <>';
[23046]69  $force_orig_source = 1;
70  $sbuild_mode = "user";
71  1;
[22787]73You should also set the environment variable DEBATHENA_APT to
[22787]76Developers: Preparing a change
[22787]79To prepare a change to a regular package (a source tree containing a
80debian/ subdir), make the edits in a checkout and record a changelog
81entry.  You can either edit debian/changelog using emacs changelog
82mode (C-c C-v to add a new version entry, C-c C-a to add a change
[23752]83entry, C-c C-f to finalize the entry) or you can run "dadch" from
84the debathena locker / SVN scripts directory.
[22787]86When creating a new version entry, bump the upstream version number
87(to 10.0.0 if it was not already that high) if you are changing the
88main package source.  Otherwise, just bump the Debian version
89component (change 0debathena1 to 0debathena2, for instance).
[22787]91Developers: Building a package for test purposes on one platform
[22787]94After you have prepared a change, you will want to test that it builds
95and perhaps that it works before committing it.  First, if it is an
96Athena source directory using autoconf, run "daconfiscate" to set up
97the autoconf boilerplate which we don't check in.  Second, run
98"daorig" to copy or create an orig tarball in the parent directory if
99necessary.  Third, run "debuild".  The resulting package will be
100placed in the parent directory.
[22787]102In order to test if the package works, you can install it with "dpkg
103-i filename.deb".
[22787]105Developers: Building a package for test purposes on all platforms
[22787]108If the package you are working on interacts with the native OS in ways
109that might vary from platform to platform, you may want to do a test
110build for all platforms.  You will need to do this on or another machine which has been set up with
112build schroots.
114As above, run daconfiscate (if necessary) and then daorig.  Then run
115"debuild -S" to create a source package.  Now cd into the parent
116directory and identify the .dsc file created by debuild -S; it will
117have a name like debathena-just_9.4.0-0debathena2.dsc.  Run "da
118sbuildhack filename.dsc" to perform the package builds.  Each build
119will take place inside an ephemeral chroot based on a snapshot of a
120template for a particular Debian or Ubuntu version.  If a build fails
121and it's not obvious from the build log why, you may need to create
122your own ephemeral chroot session with a command like "schroot -c
123gutsy-amd64-sbuild /bin/sh" and then run debuild from within the
124package sources.
126If the build is successful, it will create a set of packages with
127names like debathena-just_9.4.0-0debathena2~ubuntu6.06_amd64.deb.
129Developers: Building an equivs package
132Most of the packages under debathena/meta are faked up using equivs.
133To build one, just run:
135  equivs-build --full filename.equivs
137These equivs files make reference to ../common, so you must have a
138checkout of debathena/meta/common alongside the particular
139meta-package you are building.
[23018]141Developers: The meaning of metapackages
144If you are adding a new package to the repository, you will probably
145at some point want to add it to one of the metapackages so that it
146doesn't have to be installed by hand.  Here are some descriptions
147which may help identify which metapackage is best:
149* locker: Provides access to Athena locker software--AFS and
150  automounter configuration, locker-related utilities, etc.
152* clients: Provides clients (either locally-written, like athinfo and
153  Discuss, or configurations) for Athena services, as well as
154  Athena-specific utility programs like "jot".  Configurations for
155  graphical client software are generally in the workstation package
156  instead, in order to make this package less intrusive.
158* standard: Implies locker and clients.  Also provides Athena shell
159  customizations and dotfiles.
161* login: Implies standard.  Configurations to merge the MIT user
162  namespace into the local machine namespace for the purpose of user
163  lookups and authentication.
[23752]165* login-graphical: Implies login.  Configurations for the graphical login
[23018]166  system and graphical client software intended to provide a standard
[23752]167  X login experience using Athena home directories.
[23247]169* extra-software: Implies a set of Debian packages common to cluster
170  machines and typical workstations.  The resulting software set is
171  rather large, and thus may not be desirable to all workstation
172  configurations.  Only stock Debian packages belong in this
173  metapackage; do not add other Debathena packages to it.
[23752]175* workstation: Implies login-graphical and extra-software.  The use case
176  is the same as that of the old "private workstations", so it includes
177  more centrally-managed configuration than standard or login does
178  (unattended updates, etc.). New as of May 2009.
[23752]180* thirdparty: Software supported by 3partysw that upstream Ubuntu
181  already happens to be keeping up to date, and is therefore easier to
182  install via packages than through lockers. This package is HUGE (1 GB of
183  network traffic, 3 GB installed), and is also not strongly tested on
184  Debian or on older releases of Ubuntu.
186* cluster: Implies workstation and thirdparty. Configures public cluster
187  machines (login-time LVM snapshots, cleanups between logins, public
188  root password, etc.). The snapshot code requires a particular LVM
189  layout that the PXE (netboot) cluster installer sets up.
[23018]191* debian-dev: Intended for developers of the system itself; provides a
192  set of Debian packages used by Debathena for development.
194For the most part a package should be listed in the "Depends:" line of
195a metapackage, but in some cases it is appropriate to hedge by using
196"Recommends:", which will cause aptitude to succeed even if the
197package is unavailable.  For example, a package which doesn't exist in
198all Debian/Ubuntu suites or isn't free can be listed under
199"Recommends:" so that our metapackages still work in all environments.
[23158]201Developers: Index of cluster packages
204Some private workstation admins may want specific pieces of cluster
205infrastructure.  With Athena 9.4, they would typically install the
206entire Athena software suite and then turn off features they did not
207want via /etc/athena/rc.conf.  That is not generally possible in
208Athena 10; a debathena-cluster machine acts like a cluster machine in
209all respects.  Instead, the machine owner should install
210debathena-workstation and then install specific packages they might be
211interested in.  At this time, candidate packages include:
213  * debathena-dns-config: Installs a caching DNS resolver and
214    configures the machine to use it.
216  * debathena-tmp-cleaner: Cleans files in /tmp and /var/tmp which
217    have not been recently accessed.
219  * debathena-reactivate: Causes each graphical login to be performed
220    in a separate ephemeral snapshot of the chroot.  This package
221    requires the root filesystem to be an LVM logical volume inside a
222    volume group with 21GB of free space for the snapshots; if that is
223    not the case, it will break graphical logins.  Install with care.
225  * debathena-auto-update: Periodically updates the machine's software
226    and reboots (when no one is logged in) if necessary.
228Cluster packages which are generally not of interest to private
229machine owners include:
[23177]231  * debathena-cluster-login-config: Configures a variety of system
232    services to implement the cluster login policy (no tty logins, no
233    user switching, screensaver logout button after 20 minutes, etc.).
[23158]235  * debathena-clusterinfo: Looks up the machine's cluster information
236    in Hesiod and caches it on local disk.
238  * debathena-larvnet: Reports to the central larvnet server whether
239    anyone has a graphical login to the machine.
[23162]241  * debathena-syslog-config: Configures sysklogd so that some syslog
242    messages are forwarded to a central logging host.
[22787]244Release engineer: Bootstrapping the project infrastructure
[22764]247  1. Create the package repository (detailed instructions on this
248     pending).  Set the DEBATHENA_APT environment variable to point to
249     the package repository.  Put a copy of the debathena "scripts"
250     subdir in your path.
252  2. Create the build area.
[24565]254  3. Run 'dareprepro export' to set up the repository.  Build each
255     equivs package under meta/ using 'equivs-build --full *.equivs'
256     and upload each with 'daupload-equivs-proposed *.changes',
257     followed by 'damove -proposed "" PACKAGE' (replacing PACKAGE with
258     the name of each equivs package you are moving).
260  4. Set up the build server.  The basic structure of the apt
261     repository must work for make-chroot to succeed, so this must
262     happen after step 3.
264  5. For each normal Debian package in dependency order, cd into its
265     directory in the build area and run "da sbuildhack *.dsc" and
[22787]266     "daupload-release *_source.changes".  If the package contains
267     only an "Architecture: all" binary package, pass the -A option to
268     both commands.
270     The all-packages script can generate an approximation of the
271     package list in dependency order, but it doesn't work right yet,
272     and ideally it would be possible to do several builds in parallel
273     using a Makefile like the one in scripts/build-server/build-all.
274     Improvements to this machinery are pending.
[22787]276  6. For each package under third, run "da ./debathenify-PKG source
[23084]277     binary upload".  Any created directories under third/openafs/meta
278     should be chmodded 777 to work around a perl/AFS permissions
279     issue with File::Temp; if this is not done, OpenAFS metapackage
280     builds will fail for other users.
[23090]282Release engineer: Updating the apt repository
285  1. ssh to the build server as the builder account and change to the
286     canonical build directory.
288  2. Run "gen-packages" to update the package list.  (Or "gen-packages
289     -c" if you know the AFS checkout of the source tree is up to date;
290     it should update every half hour.)
292  3. Run "ood-packages" to produce a list of out-of-date packages.
294  4. For each out of date package, run "dasource PKG".  Then change to
295     the package directory and run "da sbuildhack *.dsc" and
296     "daupload-release *_source.changes".  If the package contains
297     only an "Architecture: all" binary package, pass the -A option to
298     both commands.
300  5. svn update the meta directory.  If there are new subdirectories,
301     chmod them 777 to work around a perl/AFS permissions issue with
302     File::Temp.  For each updated subdir, change to it, run
303     "equivs-build --full *.equivs", and then "daequivsupload
304     FILENAME.changes" on the produced changes file.
306  6. svn update the third directory.  You can let autodebathenify
307     handle the updated scripts, or you can touch
308     ~/autodebathenify.suppress, make sure it's not running, and run
309     "da ./debathenify-PKG source binary upload" in each updated
310     directory.
312Sometimes you may have to mix up the order of the above steps in order
313to handle build dependencies.
[22787]315Release engineer: Setting up a build server
[23752]318  1. Install either Debian testing (or stable, if it's new enough, but
319     see the note about sbuild below) or the latest Ubuntu release.
320     The build server must be installed with free space in an LVM
[24237]321     volume group.  The build chroots consume 4GB each.
[24237]323  2. Install debathena-login as per the the instructions in
[24237]326  3. apt-key add
327     /afs/
[23248]329  4. Install the packages listed in scripts/build-server/packages
330     (using "aptitude install").
[23752]332     Note that currently the build system uses the latest version of
[24193]333     sbuild, so if you have not installed Debian testing, you'll need
334     to set up apt pinning to grab sbuild from there. See the
335     apt_preferences man page. You'll also need to grab schroot from
336     Debian experimental.
[24237]338  5. Install the aufs-modules-2.6-amd64 package.
[24237]340  6. Append to /etc/approx/approx.conf the contents of
[23752]341      scripts/build-server/approx.conf.tail.
342     Run: /etc/init.d/approx restart
[24303]344  7. Apply scripts/build-server/mount-defaults.patch and
345     scripts/build-server/pam-schroot.patch.
[24237]347  8. For each supported DIST (see scripts/ run:
[24928]349       scripts/build-server/make-chroot DIST i386
350       scripts/build-server/make-chroot DIST amd64
[24928]352     Example: scripts/build-server/make-chroot intrepid i386
[24237]354  9. Create a local account for builder with:
[24237]356       adduser --uid 1047 --disabled-password builder
[24237]358     Make the home directory mode 700.  Install a
359     daemon/ keytab in the home directory as
360     "keytab".  Install a copy of the secret repository-signing key
361     ( in the home directory's keyring with
362     something like:
[24237]364       kinit builder
[24262]365       gpg --export-secret-keys | \
[24237]366         ssh -l builder machinename gpg --import
[24237]368     Create a file named .sbuildrc in builder's homedir containing:
[24262]370       $mailto = undef;
[24237]371       $log_dir = '/tmp/sbuild-logs';
372       $maintainer_name = 'Debathena Project <>';
373       $force_orig_source = 1;
374       $sbuild_mode = "user";
375       1;
[24237]377     Create a file named .ssh/config in builder's homedir containing:
[24237]379       Host
380         User debuildsvn
[24237]382     Add builder to the sbuild group in /etc/group.
[24237]384     Copy scripts/build-server/autodebathenify to builder's homedir.
385     Create a file named autodebathenify.config in builder's homedir
386     containing:
389       scripts_dir=/mit/debathena/bin
390       build_dir=/mit/debathena/packages/third
391       packages="lprng openafs"
392       export DEBATHENA_APT=/mit/debathena/apt
[24237]394     Copy scripts/build-server/autodebathenify.cron to builder's
395     homedir and install it with "crontab autodebathenify.cron".
[24262]397     Copy scripts/build-server/autolivebuilder to builder's
398     homedir. Create a file called autolivebuilder.config in builder's
399     homedir containing:
402       release_version='9.04'
403       release='jaunty'
404       arch='i386'
405       mirror=''
406       gpg_opts=("-u" "0D8A9E8F")
407       live_dir=/net/
[24237]409     Create a file named .devscripts in builder's homedir containing:
[24237]411       DEBUILD_DPKG_BUILDPACKAGE_OPTS="-sa -us -uc -i -I.svn"
[24237]413     In builder's homedir, append to .bashrc:
[24237]415       add debathena
416       export PATH=$PATH:~/bin
417       export DEBATHENA_APT=/mit/debathena/apt
[22787]420Release engineer: Removing a build chroot on the build server
[22787]423  1. Run VG=/dev/blah scripts/clean-schroots as root to make sure that
424     the build chroot is not mounted, substituting the name of the
425     volume group for blah.
[25325]427  2. Look in /etc/schroot/chroot.d/ and delete the file
[22729]428     corresponding to the chroot.
430  3. Run lvchange -an blah/chrootname
431     substituting the name of the volume group for blah and the chroot
432     name for chroot.  Example: lvchange -an dink/gutsy-i386-sbuild
[22764]434  4. Run lvremove blah/chrootname
[22985]436Release engineer: Removing a dist from the apt repository
439  1. Inside the apt repository, edit conf/distributions and remove the
440     distribution section.
[24565]442  2. Run dareprepro --delete clearvanished
[22787]444Release engineer: Setting up a canonical build area
447  1. Create an empty directory and cd into it.  The canonical build
[23752]448     area lives in /mit/debathena/packages/.
[22787]450  2. Run gen-packages to create the table of normal Debian packages.
[22787]452  3. Run dasource to create subdirs and source packages for each
453     normal Debian package.
455  4. Create checkouts of the meta and third directories:
457     svn co svn+ssh://
458     svn co svn+ssh://
[23084]459     chmod 777 meta/*
[23084]461     A couple of subdirectories of debathena/meta are normal Debian
[22764]462     packages, so this will create redundant copies of those.  Ignore
[23084]463     them; they won't be used.
[24928]465Release engineer: Adding a new suite or rebuilding a suite
468This process is rarely performed and the infrastructure for it is
[24928]469imperfect.  Substitute the name of the new suite for $suite in all
[22957]470steps below.
[22958]472  1. Make sure the apt repository is up to date with respect to the
[24875]473     source tree for the existing dists. Check
474 and
475 More importantly, make
476     sure there is nothing in svn that isn't built. (An easy way to do
477     this is running the check-unbuilt-packages script.)
[25325]479  2. Add the new dist to scripts/ (and svn up into
480     the locker).  (It is not necessary to add the new dist to codes
481     at this point, but it must be present in the gettag conditional.)
482     If you are building an unreleased distribution, add an additional
483     ~0.1 tag; if you're rebilding, bump that tag.
485  3. Create the new distribution in the apt repository's configuration
[24928]486     file by editing /mit/debathena/apt/conf/gen-distributions and
487     running it with output to the 'distributions' file in the same
[25325]488     directory.  Create the skeleton of the dist by running
489     "dareprepro export" on the build server. (You don't need to do
490     this if you're rebuilding.)
[24928]492  4. Add an appropriate sources.list.d/*/$suite.list entry in
493     scripts/build-server. Commit and update
494     /mit/debathena/bin/build-server.
[25325]496  5. On the build server (as root), create a chroot for the new distribution:
[25325]498     First, check if /usr/share/debootstrap/scripts/$suite exists.
499     If not, it may require downloading and installing a
[22957]500     more recent version of the debootstrap package from the
[24928]501     -backports dist corresponding to the build server's OS. (You
502     presumably don't need to do this if you're rebuilding.)
[25325]504     You could also just go into that directory and do
505     "ln -s gutsy $suite" since it apparently hasn't changed in years.
507        /mit/debathena/bin/build-server/make-chroot $suite i386
508        /mit/debathena/bin/build-server/make-chroot $suite amd64
[22958]511  6. Fire up screen.
[24928]513  7. mkdir /mit/debathena/machines/awesome-build-server/stamps.$suite.
[22958]515  8. cd into a checkout of debathena/scripts/build-server/build-all.
[22958]517  9. Edit Makefile (and check in the edit) so that suite is the new
[22957]518     distribution and psuite is the previously most recent Debian or
[24928]519     Ubuntu distribution.  psuite is used to generate the order of
520     build-dependencies, so choose the closest distribution.  If you are
521     rebuilding, you can set psuite to suite.
[22958]523  10. Run "make".
[24928]525  11. Run "make -k all".
[22958]526      You can watch the builds happen in the other windows of the
527      screen session.  It's possible to do several builds at once with
[25325]528      make -j N (except as of 7/2011 this is a bad idea)
[23094]530Release engineer: Maintaining autodebathenify
[23185]533autodebathenify is a cron job which runs on the builder account and
534keeps the OpenAFS modules and modified packages under debathena/third
535up to date when there are upstream changes.  It relies on the
538  * builder@ATHENA.MIT.EDU has access to a local account named
539    debuildsvn on with read access to the athena
540    repository.
542  * builder@ATHENA.MIT.EDU has write access to the canonical build
543    area and apt repository in the dev cell.
545  * The builder account on the build machine has a copy of the signing
546    private key for the apt repository.
548The cron job will silently exit if it detects that it is already
549running.  If it fails for any other reason, it will send mail to and touch a file named autodebathenify.suppress
551which will prevent it from running again until manual intervention.
552It is important to get autodebathenify running smoothly or the apt
553repository will become out of date with respect to third-party
554packages, which in turn will compromise the user experience.
556Even if you don't notice failure mail, it's good to check on the
557status of autodebathenify from time to time.  To do this, get tickets
558as builder and ssh to linux-build-10 as builder.  Run "ls -l logs" to
559see what's been going on.  A logfile with size around 48K indicates
560that autodebathenify ran normally but didn't find any new work to do.
561A logfile with size around 60 means that autodebathenify.suppress
562exists and the cron job has stopped running.  A longer logfile
563indicates that autodebathenify attempted to build and upload a
566Possible failure cases include:
568  * One of the upstream apt repositories timed out during an "apt-get
[23119]569    update" of an sbuild chroot.  If this happens, you can just remove
570    the autodebathenify.suppress file to get the cron job to start
571    running again.
573  * autodebathenify tried to build an upstream version which already
574    exists.  If this happens, the build will fail at upload time.  If
575    this happens, there is a bug in the part of the debathenify-*
576    script which checks whether our apt repository already has the
577    expected version of the built package.
579  * Debian and Ubuntu have .orig files with the same name but
580    different contexts for a particular third-party package.  This has
581    only happened once, with bash (which is currently not configured
582    into autodebathenify for that reason).  We do not have a general
583    mechanism for resolving such issues at this time.
585  * The new upstream version of the package is sufficiently different
586    to cause our modifications to fail to apply, and our debathenify-*
587    script needs to be adapted appropriately.
589The cron job is run out of builder's home directory, but its canonical
590source location is in debathena/scripts/build-server in this
591repository, should it need to be modified.
[23191]593After several months of operation, LVM snapshots on the build server
594will start to become slower and slower, as /etc/lvm/cache/.cache grows
595in size.  Turning off the cache entirely seems to cause schroot to
596occasionally fail; the current solution is to rm /etc/lvm/cache/.cache
597every so often.
[23077]599Release engineer: apt repository HTTP server setup
[23752]602The apt repository server ( is hosted on out of the debathena locker. To replicate it on a
604dedicated server, install Apache and AFS (e.g., debathena-standard
605and apache2), and configure the document root to be
606  /mit/debathena/web_scripts
[23752]608There's also a crontab in /mit/debathena/cron_scripts that updates
609the package lists on the website. It expects to be able to edit the
610web_scripts directory; on scripts, it does so with daemon.scripts
613Trac ( expects to use
614for its database.
Note: See TracBrowser for help on using the repository browser.