source: trunk/debathena/NOTES @ 23191

This hierarchy contains Debian/Ubuntu-specific materials, also known
as "Debathena".  The contents are:

* debathena - Debathena-specific software packages such as PAM and NSS
  modules.

* config - Packages for configuring native system software in a manner
  appropriate for Athena.

* meta - Packages which contain nothing but dependencies on other
  packages and serve as an installation convenience.

* scripts - Build scripts and supporting materials.

Debathena is a SIPB project, and its infrastructure and procedures
will need to be adapted for Athena 10.  For the moment this file will
document the Debathena procedures as they are, not as they will be.
The current procedures do not even use this svn repository yet.

Debian software used by Debathena:

  * schroot - Used to manage build chroot environments for each
    Debian/Ubuntu version.  We use the lvm-snapshot schroot type,
    which allows rapid construction of ephemeral copies of template
    "source" chroots, so that every binary package build is done in a
    clean environment.

  * debuild - Used to create Debian source packages from package
    source directories.

  * sbuild - Used to build binary packages from source packages inside
    schroot environments.

  * equivs - Used to create packages which only contain dependency
    information.  Somewhat of a dirty hack, since it doesn't keep
    proper changelogs, but it reduces overhead.

  * CDBS (Common Debian Build System) - Referenced by debian/rules
    files in packages.  Contains standard build rules to cut down on
    per-package boilerplate.

  * reprepro - Used to upload packages into the apt repositories.

  * approx - Used to create a local cache of Debian packages on the
    build server.  This cache is referenced by the build chroots for
    improved performance.

The remainder of this file documents procedures useful to Athena 10
developers and the release engineer.

Developers: Preferences setup
-----------------------------

You will probably want a $HOME/.devscripts file containing the
following:

DEBUILD_DPKG_BUILDPACKAGE_OPTS="-sa -us -uc -i -I.svn"

This will save you from having to specify those options every time you
run debuild.  Athena 10 scripts do not assume the above preferences,
but the instructions in this file do.  The options mean:

  * Look for original source as a tarfile or create one.
  * Do not sign the source package.
  * Do not sign the changes file.
  * Ignore common version control metadata files when creating diffs.
  * Ignore .svn paths when creating tarballs.

You will also want a $HOME/.sbuildrc file containing the following:

  $nolog = 1;
  $mailto = 'yourusername';
  $log_dir = '/tmp/sbuild-logs';
  $maintainer_name = 'Debian-Athena Project <debathena@mit.edu>';
  $force_orig_source = 1;
  $sbuild_mode = "user";
  1;

You should also set the environment variable DEBATHENA_APT to
"/afs/dev.mit.edu/system/athena10/apt".

Developers: Preparing a change
------------------------------

To prepare a change to a regular package (a source tree containing a
debian/ subdir), make the edits in a checkout and record a changelog
entry.  You can either edit debian/changelog using emacs changelog
mode (C-c C-v to add a new version entry, C-c C-a to add a change
entry, C-c C-f to finalize the entry) or you can run "dadch".

When creating a new version entry, bump the upstream version number
(to 10.0.0 if it was not already that high) if you are changing the
main package source.  Otherwise, just bump the Debian version
component (change 0debathena1 to 0debathena2, for instance).

Developers: Building a package for test purposes on one platform
----------------------------------------------------------------

After you have prepared a change, you will want to test that it builds
and perhaps that it works before committing it.  First, if it is an
Athena source directory using autoconf, run "daconfiscate" to set up
the autoconf boilerplate which we don't check in.  Second, run
"daorig" to copy or create an orig tarball in the parent directory if
necessary.  Third, run "debuild".  The resulting package will be
placed in the parent directory.

In order to test if the package works, you can install it with "dpkg
-i filename.deb".

Developers: Building a package for test purposes on all platforms
-----------------------------------------------------------------

If the package you are working on interacts with the native OS in ways
that might vary from platform to platform, you may want to do a test
build for all platforms.  You will need to do this on
linux-build-10.mit.edu or another machine which has been set up with
build schroots.

As above, run daconfiscate (if necessary) and then daorig.  Then run
"debuild -S" to create a source package.  Now cd into the parent
directory and identify the .dsc file created by debuild -S; it will
have a name like debathena-just_9.4.0-0debathena2.dsc.  Run "da
sbuildhack filename.dsc" to perform the package builds.  Each build
will take place inside an ephemeral chroot based on a snapshot of a
template for a particular Debian or Ubuntu version.  If a build fails
and it's not obvious from the build log why, you may need to create
your own ephemeral chroot session with a command like "schroot -c
gutsy-amd64-sbuild /bin/sh" and then run debuild from within the
package sources.

If the build is successful, it will create a set of packages with
names like debathena-just_9.4.0-0debathena2~ubuntu6.06_amd64.deb.

Developers: Building an equivs package
--------------------------------------

Most of the packages under debathena/meta are faked up using equivs.
To build one, just run:

  equivs-build --full filename.equivs

These equivs files make reference to ../common, so you must have a
checkout of debathena/meta/common alongside the particular
meta-package you are building.

Developers: The meaning of metapackages
---------------------------------------

If you are adding a new package to the repository, you will probably
at some point want to add it to one of the metapackages so that it
doesn't have to be installed by hand.  Here are some descriptions
which may help identify which metapackage is best:

* locker: Provides access to Athena locker software--AFS and
  automounter configuration, locker-related utilities, etc.

* clients: Provides clients (either locally-written, like athinfo and
  Discuss, or configurations) for Athena services, as well as
  Athena-specific utility programs like "jot".  Configurations for
  graphical client software are generally in the workstation package
  instead, in order to make this package less intrusive.

* standard: Implies locker and clients.  Also provides Athena shell
  customizations and dotfiles.

* login: Implies standard.  Configurations to merge the MIT user
  namespace into the local machine namespace for the purpose of user
  lookups and authentication.

* workstation: Implies login.  Configurations for the graphical login
  system and graphical client software intended to provide a standard
  X login experience using Athena home directories.  Still in
  development.

* cluster-software: Provides a set of Debian packages common to
  cluster machines.  The resulting software set is rather large, and
  thus may not be desirable to all workstation configurations.  Only
  stock Debian packages belong in this metapackage; do not add
  other Debathena packages to it.

* cluster: Implies workstation and cluster-software.  Also contains
  configurations for self-maintenance of machines (unattended updates,
  cleanups between logins, etc.).

* debian-dev: Intended for developers of the system itself; provides a
  set of Debian packages used by Debathena for development.

For the most part a package should be listed in the "Depends:" line of
a metapackage, but in some cases it is appropriate to hedge by using
"Recommends:", which will cause aptitude to succeed even if the
package is unavailable.  For example, a package which doesn't exist in
all Debian/Ubuntu suites or isn't free can be listed under
"Recommends:" so that our metapackages still work in all environments.

Developers: Index of cluster packages
-------------------------------------

Some private workstation admins may want specific pieces of cluster
infrastructure.  With Athena 9.4, they would typically install the
entire Athena software suite and then turn off features they did not
want via /etc/athena/rc.conf.  That is not generally possible in
Athena 10; a debathena-cluster machine acts like a cluster machine in
all respects.  Instead, the machine owner should install
debathena-workstation and then install specific packages they might be
interested in.  At this time, candidate packages include:

  * debathena-dns-config: Installs a caching DNS resolver and
    configures the machine to use it.

  * debathena-tmp-cleaner: Cleans files in /tmp and /var/tmp which
    have not been recently accessed.

  * debathena-reactivate: Causes each graphical login to be performed
    in a separate ephemeral snapshot of the chroot.  This package
    requires the root filesystem to be an LVM logical volume inside a
    volume group with 21GB of free space for the snapshots; if that is
    not the case, it will break graphical logins.  Install with care.

  * debathena-auto-update: Periodically updates the machine's software
    and reboots (when no one is logged in) if necessary.

Cluster packages which are generally not of interest to private
machine owners include:

  * debathena-cluster-login-config: Configures a variety of system
    services to implement the cluster login policy (no tty logins, no
    user switching, screensaver logout button after 20 minutes, etc.).

  * debathena-clusterinfo: Looks up the machine's cluster information
    in Hesiod and caches it on local disk.

  * debathena-larvnet: Reports to the central larvnet server whether
    anyone has a graphical login to the machine.

  * debathena-syslog-config: Configures sysklogd so that some syslog
    messages are forwarded to a central logging host.

Release engineer: Bootstrapping the project infrastructure
----------------------------------------------------------

  1. Create the package repository (detailed instructions on this
     pending).  Set the DEBATHENA_APT environment variable to point to
     the package repository.  Put a copy of the debathena "scripts"
     subdir in your path.

  2. Create the build area.

  3. Build each equivs package under meta/ using "equivs-build --full
     *.equivs" and upload each with "daequivsupload *.changes".  This
     has the side-effect of creating the basic structure of the
     package repository.

  4. Set up the build server.  The basic structure of the apt
     repository must work for make-chroot to succeed, so this must
     happen after step 3.

  5. For each normal Debian package in dependency order, cd into its
     directory in the build area and run "da sbuildhack *.dsc" and
     "daupload-release *_source.changes".  If the package contains
     only an "Architecture: all" binary package, pass the -A option to
     both commands.

     The all-packages script can generate an approximation of the
     package list in dependency order, but it doesn't work right yet,
     and ideally it would be possible to do several builds in parallel
     using a Makefile like the one in scripts/build-server/build-all.
     Improvements to this machinery are pending.

  6. For each package under third, run "da ./debathenify-PKG source
     binary upload".  Any created directories under third/openafs/meta
     should be chmodded 777 to work around a perl/AFS permissions
     issue with File::Temp; if this is not done, OpenAFS metapackage
     builds will fail for other users.

Release engineer: Updating the apt repository
---------------------------------------------

  1. ssh to the build server as the builder account and change to the
     canonical build directory.

  2. Run "gen-packages" to update the package list.  (Or "gen-packages
     -c" if you know the AFS checkout of the source tree is up to date;
     it should update every half hour.)

  3. Run "ood-packages" to produce a list of out-of-date packages.

  4. For each out of date package, run "dasource PKG".  Then change to
     the package directory and run "da sbuildhack *.dsc" and
     "daupload-release *_source.changes".  If the package contains
     only an "Architecture: all" binary package, pass the -A option to
     both commands.

  5. svn update the meta directory.  If there are new subdirectories,
     chmod them 777 to work around a perl/AFS permissions issue with
     File::Temp.  For each updated subdir, change to it, run
     "equivs-build --full *.equivs", and then "daequivsupload
     FILENAME.changes" on the produced changes file.

  6. svn update the third directory.  You can let autodebathenify
     handle the updated scripts, or you can touch
     ~/autodebathenify.suppress, make sure it's not running, and run
     "da ./debathenify-PKG source binary upload" in each updated
     directory.

Sometimes you may have to mix up the order of the above steps in order
to handle build dependencies.

Release engineer: Setting up a build server
-------------------------------------------

  1. The build server must be installed with free space in an LVM
     volume group.  The build chroots consume 2GB each.  There is a
     known memory corruption issue with LVM snapshots in the kernel
     used in Ubuntu Gutsy (which is based on 2.6.22), so use a newer
     kernel such as the one in Ubuntu Hardy (based on 2.6.24) instead.

  2. Install debathena-standard as per the the instructions in
     http://debathena.mit.edu/install.

  3. apt-key add /afs/dev.mit.edu/system/athena10/apt/athena10-archive.asc

  4. Install the packages listed in
     scripts/build-server/packages (using "aptitude install")

  5. Install debathena-login, debathena-ssh-server, and
     debathena-build-depends (using "aptitude install").

     (Depending on how recently debathena-build-depends was rebuilt,
     additional packages might need to be installed to satisfy the
     build-depends of newer packages.  This can be taken care of later
     when an error occurs building a source package.)

  6. Edit /etc/security/access.conf and add a first line:
     -:ALL EXCEPT root <developer usernames>:ALL

  7. Edit /etc/pam.d/schroot, comment out "@include common-session",
     and add:

       # Basic pam_unix session module in place of common-session.
       session required         pam_unix.so

  8. Edit /etc/group and add the developers to the sbuild group.

  9. Create /etc/passwd entries for each developer with "hesinfo
     username passwd >> /etc/passwd" and then run pwconv.

     (This is not necessary for the login system on the main root
     environment, but is for the chroot environments.)

  10. Append to /etc/approx/approx.conf the contents of
       scripts/build-server/approx.conf.tail.
      Change the last line from http://debathena.mit.edu/apt to
       file:///afs/dev.mit.edu/system/athena10/apt
      Add "$interval 0" above the repository lines (only necessary if
       the version of approx as reported by "dpkg -l approx" is less
       than 3.0)
      Run: /etc/init.d/approx restart

  11. Apply scripts/build-server/mount-defaults.patch.

  12. Edit /etc/lvm/lvm.conf and change "archive = 1" to "archive = 0"
      in order to avoid generating a bazillion backup files under
      /etc/lvm/archive.

  13. For each supported DIST (see scripts/debian-versions.sh) run:

        VG=/dev/blah scripts/build-server/make-chroot DIST i386
        VG=/dev/blah scripts/build-server/make-chroot DIST amd64

      substituting the name of the volume group for blah.  Omit the
      amd64 line if DIST is sarge.

      Example: VG=/dev/dink scripts/build-server/make-chroot gutsy i386

  14. Create a local account for builder with:

        adduser --uid 1047 --disabled-password builder

      Make the home directory mode 700.  Install a
      daemon/linux-build-10.mit.edu keytab in the home directory as
      "keytab".  Install a copy of the secret repository-signing key
      (athena10@mit.edu) in the home directory's keyring with
      something like:

        kinit builder
        gpg --export-secret-keys athena10@mit.edu | \
          ssh -l builder machinename gpg --import

      Create a file named .sbuildrc in builder's homedir containing:

        $nolog = 1;
        $mailto = 'source-wash@mit.edu';
        $log_dir = '/tmp/sbuild-logs';
        $maintainer_name = 'Debian-Athena Project <debathena@mit.edu>';
        $force_orig_source = 1;
        $sbuild_mode = "user";
        1;

      Create a file named .ssh/config in builder's homedir containing:

        Host svn.mit.edu
          User debuildsvn

      Add builder to the sbuild group in /etc/group.

      Copy scripts/build-server/autodebathenify to builder's homedir.
      Create a file named autodebathenify.config in builder's homedir
      containing:

        error_addr=source-wash@mit.edu
        scripts_dir=/afs/dev.mit.edu/source/src-svn/debathena/scripts
        build_dir=/afs/dev.mit.edu/project/release/10/build/third
        packages="cyrus-sasl2-mit evolution-data-server lprng openafs tcsh"
        export DEBATHENA_APT=/afs/dev.mit.edu/system/athena10/apt

      Copy scripts/build-server/autodebathenify.cron to builder's
      homedir and install it with "crontab autodebathenify.cron".

      Create a file named .devscripts in builder's homedir containing:

        DEBUILD_DPKG_BUILDPACKAGE_OPTS="-sa -us -uc -i -I.svn"

      In builder's homedir, append to .profile:

        PATH=${PATH}:/afs/dev.mit.edu/source/src-svn/debathena/scripts
        export DEBATHENA_APT=/afs/dev.mit.edu/system/athena10/apt

      and to .bashrc:

        bld=/afs/dev.mit.edu/project/release/10/build

Release engineer: Removing a build chroot on the build server
-------------------------------------------------------------

  1. Run VG=/dev/blah scripts/clean-schroots as root to make sure that
     the build chroot is not mounted, substituting the name of the
     volume group for blah.

  2. Edit /etc/schroot/schroot.conf and delete the section
     corresponding to the chroot.

  3. Run lvchange -an blah/chrootname
     substituting the name of the volume group for blah and the chroot
     name for chroot.  Example: lvchange -an dink/gutsy-i386-sbuild

  4. Run lvremove blah/chrootname

Release engineer: Removing a dist from the apt repository
---------------------------------------------------------

  1. Inside the apt repository, edit conf/distributions and remove the
     distribution section.

  2. Run reprepro -Vb $DEBATHENA_APT --delete clearvanished

Release engineer: Setting up a canonical build area
---------------------------------------------------

  1. Create an empty directory and cd into it.  The canonical build
     area lives in /afs/dev.mit.edu/project/release/10/build.

  2. Run gen-packages to create the table of normal Debian packages.

  3. Run dasource to create subdirs and source packages for each
     normal Debian package.

  4. Create checkouts of the meta and third directories:

     svn co svn+ssh://svn.mit.edu/athena/trunk/debathena/meta
     svn co svn+ssh://svn.mit.edu/athena/trunk/debathena/third
     chmod 777 meta/*

     A couple of subdirectories of debathena/meta are normal Debian
     packages, so this will create redundant copies of those.  Ignore
     them; they won't be used.

Release engineer: Adding a new suite
------------------------------------

This process is rarely performed and the infrastructure for it is
imperfect.  Substitute the name of the new suite for "newdist" in all
steps below.

  1. Make sure the apt repository is up to date with respect to the
     source tree for the existing dists.

  2. Add the new dist to scripts/debian-versions.sh.  (It is not
     necessary to add the new dist to codes at this point, but it must
     be present in the gettag conditional.)

  3. Create the new distribution in the apt repository's configuration
     file.  Create the skeleton of the dist by installing at least one
     equivs package from meta/ with "reprepro -Vb $DEBATHENA_APT
     include newdistname file.changes".

  4. On the build server, create a chroot for the new distribution as
     documented above.  This may require downloading and installing a
     more recent version of the debootstrap package from the
     -backports dist corresponding to the build server's OS.

  5. Set the DEBATHENA_BUILD_AREA environment variable to point to the
     build area.

  6. Fire up screen.

  7. mkdir $DEBATHENA_BUILD_AREA/stamps.newdist.

  8. cd into a checkout of debathena/scripts/build-server/build-all.

  9. Edit Makefile (and check in the edit) so that suite is the new
     distribution and psuite is the previously most recent Debian or
     Ubuntu distribution.

  10. Run "make deps.mk".

  11. Run "make -k all STAMPS=$DEBATHENA_BUILD_AREA/stamps.newdist".
      You can watch the builds happen in the other windows of the
      screen session.  It's possible to do several builds at once with
      make -j N.

  12. debathenify packages will fail out; they must be built by hand.
      When the build fails on one, cd into third/packagename in the
      build area and run "./debathenify newdist-amd64 -A source binary
      upload" and "./debathenify newdist-i386 binary upload".  Then
      touch $DEBATHENA_BUILD_AREA/stamps.newdist/packagename.done" and
      restart the build.

      The newly created third/openafs/meta directories should be
      chmodded 777 to work around a perl/AFS permissions issue.

Release engineer: Maintaining autodebathenify
---------------------------------------------

autodebathenify is a cron job which runs on the builder account and
keeps the OpenAFS modules and modified packages under debathena/third
up to date when there are upstream changes.  It relies on the
following:

  * builder@ATHENA.MIT.EDU has access to a local account named
    debuildsvn on svn.mit.edu with read access to the athena
    repository.

  * builder@ATHENA.MIT.EDU has write access to the canonical build
    area and apt repository in the dev cell.

  * The builder account on the build machine has a copy of the signing
    private key for the apt repository.

The cron job will silently exit if it detects that it is already
running.  If it fails for any other reason, it will send mail to
source-wash@mit.edu and touch a file named autodebathenify.suppress
which will prevent it from running again until manual intervention.
It is important to get autodebathenify running smoothly or the apt
repository will become out of date with respect to third-party
packages, which in turn will compromise the user experience.

Even if you don't notice failure mail, it's good to check on the
status of autodebathenify from time to time.  To do this, get tickets
as builder and ssh to linux-build-10 as builder.  Run "ls -l logs" to
see what's been going on.  A logfile with size around 48K indicates
that autodebathenify ran normally but didn't find any new work to do.
A logfile with size around 60 means that autodebathenify.suppress
exists and the cron job has stopped running.  A longer logfile
indicates that autodebathenify attempted to build and upload a
package.

Possible failure cases include:

  * One of the upstream apt repositories timed out during an "apt-get
    update" of an sbuild chroot.  If this happens, you can just remove
    the autodebathenify.suppress file to get the cron job to start
    running again.

  * autodebathenify tried to build an upstream version which already
    exists.  If this happens, the build will fail at upload time.  If
    this happens, there is a bug in the part of the debathenify-*
    script which checks whether our apt repository already has the
    expected version of the built package.

  * Debian and Ubuntu have .orig files with the same name but
    different contexts for a particular third-party package.  This has
    only happened once, with bash (which is currently not configured
    into autodebathenify for that reason).  We do not have a general
    mechanism for resolving such issues at this time.

  * The new upstream version of the package is sufficiently different
    to cause our modifications to fail to apply, and our debathenify-*
    script needs to be adapted appropriately.

The cron job is run out of builder's home directory, but its canonical
source location is in debathena/scripts/build-server in this
repository, should it need to be modified.

After several months of operation, LVM snapshots on the build server
will start to become slower and slower, as /etc/lvm/cache/.cache grows
in size.  Turning off the cache entirely seems to cause schroot to
occasionally fail; the current solution is to rm /etc/lvm/cache/.cache
every so often.

Release engineer: apt repository HTTP server setup
--------------------------------------------------

The apt repository server (athena10.mit.edu) is an ops virtual image
with httpd and AFS installed.  At the moment, the only customization
is two changes to /etc/httpd/conf/httpd.conf:

  * The DocumentRoot is set to "/afs/dev.mit.edu/system/athena10".
  * The Directory entry for /var/html/www is also changed to
    "/afs/dev.mit.edu/system/athena10".