source: trunk/debathena/NOTES @ 22764

This hierarchy contains Debian/Ubuntu-specific materials, also known
as "Debathena".  The contents are:

* debathena - Debathena-specific software packages such as PAM and NSS
  modules.

* config - Packages for configuring native system software in a manner
  appropriate for Athena.

* meta - Packages which contain nothing but dependencies on other
  packages and serve as an installation convenience.

* scripts - Build scripts and supporting materials.

Debathena is a SIPB project, and its infrastructure and procedures
will need to be adapted for Athena 10.  For the moment this file will
document the Debathena procedures as they are, not as they will be.
The current procedures do not even use this svn repository yet.

Debian software used by Debathena:

  * schroot - Used to manage build chroot environments for each
    Debian/Ubuntu version.  We use the lvm-snapshot schroot type,
    which allows rapid construction of ephemeral copies of template
    "source" chroots, so that every binary package build is done in a
    clean environment.

  * debuild - Used to create Debian source packages from package
    source directories.

  * sbuild - Used to build binary packages from source packages inside
    schroot environments.

  * equivs - Used to create packages which only contain dependency
    information.  Somewhat of a dirty hack, since it doesn't keep
    proper changelogs, but it reduces overhead.

  * CDBS (Common Debian Build System) - Referenced by debian/rules
    files in packages.  Contains standard build rules to cut down on
    per-package boilerplate.

  * reprepro - Used to upload packages into the apt repositories.

  * approx - Used to create a local cache of Debian packages on the
    build server.  This cache is referenced by the build chroots for
    improved performance.

Procedures used by Debathena for handling day-to-day package changes
for regular packages:

  * To prepare a change to a regular package, the contents are edited
    directly in the /mit/debathena/packages subdir.

  * To record the prepared change in the changelog:
    (From inside the package source directory)
    dadch

  * To build and deploy the changed package for all environments:
    (From inside the package source directory, on debuild.mit.edu)
    debuild -S
    cd ..
    da sbuildhack <filename>.dsc

  * Generally the package is tested by manually installing it on test
    machines with dpkg -i.

  * To deploy the package to the repository:
    (From the package source parent directory, on debuild.mit.edu as
    the debuild user)
    daupload-release <filename>_source.changes

  * If the package is architecture-independent, the -A option is
    passed to da and daupload-release.

For an equivs-built package, the process is simpler since there is no
source package and only one binary package:

  * To prepare a change to the control file, the contents are edited
    directly in the /mit/debathena/packages subdir.

  * To build the binary package:
    (From the package directory, on debuild.mit.edu)
    equivs-build --full <filename>.equivs

  * To deploy the package to the repository:
    daequivsupload <filename>.changes

Adding a new distribution:

  <not yet written>

Bootstrapping a build area, build server, and apt repository:
(Adapted for Athena 10)

  1. Create the package repository (detailed instructions on this
     pending).  Set the DEBATHENA_APT environment variable to point to
     the package repository.  Put a copy of the debathena "scripts"
     subdir in your path.

  2. Create the build area.

  3. Build each equivs package under meta/ using "equivs-build --full
     *.equivs" and upload each with "daequivsupload *.changes".  This
     has the side-effect of creating the basic structure of the
     package repository.

  4. Set up the build server.  The basic structure of the apt
     repository must work for make-chroot to succeed, so this must
     happen after step 3.

  5. For each normal Debian package in dependency order, cd into its
     directory in the build area and run "da sbuildhack *.dsc" and
     "daupload-release *_source.changes".

     The all-packages script can generate an approximation of the
     package list in dependency order, but it doesn't work right yet,
     and ideally it would be possible to do several builds in parallel
     using a Makefile like the one in scripts/build-server/build-all.
     Improvements to this machinery are pending.

  6. Build the packages under third (instructions pending).

Setting up a build server:
(Adapted for Athena 10)

  1. The build server must be installed with free space in an LVM
     volume group.  The build chroots consume 2GB each.

  2. Install debathena-standard as per the the instructions in
     http://debathena.mit.edu/install.

  3. apt-key add /afs/dev.mit.edu/system/athena10/apt/athena10-archive.asc

  4. Install the packages listed in
     scripts/build-server/packages (using "aptitude install")

  5. Install debathena-login, debathena-ssh-server, and
     debathena-build-depends (using "aptitude install").

     (Depending on how recently debathena-build-depends was rebuilt,
     additional packages might need to be installed to satisfy the
     build-depends of newer packages.  This can be taken care of later
     when an error occurs building a source package.)

  6. Edit /etc/security/access.conf and add a first line:
     -:ALL EXCEPT root <developer usernames>:ALL

  7. Edit /etc/group and add the developers to the sbuild group.

  8. Create /etc/passwd entries for each developer with "hesinfo
     username passwd >> /etc/passwd" and then run pwconv.

     (This is not necessary for the login system on the main root
     environment, but is for the chroot environments.)

  9. Append to /etc/approx/approx.conf the contents of
      scripts/build-server/approx.conf.tail.
     Change the last line from http://debathena.mit.edu/apt to
      file:///afs/dev.mit.edu/system/athena10/apt
     Add "$interval 0" above the repository lines (only necessary if
      the version of approx as reported by "dpkg -l approx" is less
      than 3.0)
     Run: /etc/init.d/approx restart

  10. Apply scripts/build-server/10mount.patch.

  11. For each supported DIST (see scripts/debian-versions.sh) run:

        VG=/dev/blah scripts/build-server/make-chroot DIST i386
        VG=/dev/blah scripts/build-server/make-chroot DIST amd64

      substituting the name of the volume group for blah.  Omit the
      amd64 line if DIST is sarge.

      Example: VG=/dev/dink scripts/build-server/make-chroot gutsy i386

Removing a build chroot on the build server:

  1. Run VG=/dev/blah scripts/clean-schroots to make sure that the
     build chroot is not mounted, substituting the name of the volume
     group for blah.

  2. Edit /etc/schroot/schroot.conf and delete the section
     corresponding to the chroot.

  3. Run lvchange -an blah/chrootname
     substituting the name of the volume group for blah and the chroot
     name for chroot.  Example: lvchange -an dink/gutsy-i386-sbuild

  4. Run lvremove blah/chrootname

Setting up a build area:
(Adapted for Athena 10)

  1. Create an empty directory and cd into it.  The canonical build
     area lives in /afs/dev.mit.edu/project/release/10/build.

  2. Run scripts/gen-packages to create the table of normal Debian
     packages.

  3. Run scripts/dasource to create subdirs and source packages for
     each normal Debian package.

  4. Create checkouts of the meta and third directories:

     svn co svn+ssh://svn.mit.edu/athena/trunk/debathena/meta
     svn co svn+ssh://svn.mit.edu/athena/trunk/debathena/third

     (A couple of subdirectories of debathena/meta are normal Debian
     packages, so this will create redundant copies of those.  Ignore
     them; they won't be used.)