wiki:AddingOrRebuildingASuite

Bootstrapping A New Release

  1. Create build chroot using scripts/build-server/make-chroot
    • Make sure resulting chroot actually has universe enabled.
  2. Update the buildsystem configuration with tags. Do not forget to add ~0.1.
    • Build system config live in dabuildsys/config.py in build-system.git (checked out on build server home directory).
    • Tags live in debian-versions.sh in scripts.git (checked out at /mit/debathena/bin).
  3. Update /mit/debathena/apt/conf
    • Do not forget to run git commit there
  4. Run dareprepro export.
  5. Run dapopulate with appropriate flags. An example would be: dapopulate --handle-broken=include --allow-out-of-date --use-development utopic trusty (populate utopic with sources from trusty).
  6. Fix debathena-machtype
  7. Open a tmux, and in it, run dabuild on the development repo. You will probably need to specify -B with the name of preceeding repo, for example: dabuild utopic-development -B trusty.

STUFF BELOW THIS LINE MAY BE OUT OF DATE

This covers the process of building for a new suite (version, e.g. "precise"). This process is typically begun 1-2 months before the suite's release date.

Substitute the codename of the suite for $suite in the directions below where applicable.

Bootstrapping a new suite

  1. Add the new distribution to scripts/debian-versions.sh. This is done by adding a new case in the gettag function. Assuming the suite has not yet been released, the value should have an additional ~0.1 tag appended to it. (The new suite should not be added to the DEBIAN_CODES variable until it has been released, as the value of that line determines whether to include the new suite when doing a "normal" build of a single package. The ~0.1 ensures that pre-release packages sort lower than released packages -- e.g. debathena-foo_1.0~ubuntu12.04 will be "newer" than debathena-foo_1.0~ubuntu12.04~0.1)
  1. Add an appropriate entry in scripts/build-server/sources.list.d/{debian|ubuntu}/$suite.list.
  1. Commit your changes from steps 1 and 2 above.
  1. Update (svn up) /mit/debathena/bin/build-server with the changes you just committed.
  1. Create the new distribution in the apt repository's configuration file by editing /mit/debathena/apt/conf/gen-distributions (not under version control) and running it with output to the distributions file in the same directory (you may wish to first run gen-distributions and sanity-check the output on stdout). Create the skeleton of the dist by running "dareprepro export" on the build server.
  1. Log in to the build server as root. Check if /usr/share/debootstrap/scripts/$suite exists. This may require a more recent version of the debootstrap package, or you can just ln -s gutsy $suite since apparently it hasn't changed since Gutsy. Unless you're building for Debian, in which cause symlink it to sid, not gutsy. Then create the chroots:
    /mit/debathena/bin/build-server/make-chroot $suite i386
    /mit/debathena/bin/build-server/make-chroot $suite amd64
    
  1. Stop. Go make sure the apt repository is up to date with respect to the source tree for the existing dists. Check http://debathena.mit.edu/package-list/development and http://debathena.mit.edu/package-list/proposed. More importantly, make sure there is nothing in svn that isn't built. (An easy way to do this is running the check-unbuilt-packages script.)

NOTE: This is very important, because build-all uses the source packages that are already there. So, let's say a build fails, and you commit a fix. It's still going to use the old .dsc file, until you build that package normally.

  1. Log out of the build server, and log back in as builder.
  1. Start screen, ideally making use of the "-S" option to identify your screen session. e.g. screen -S joeuser.oneiric_build.

NOTE: Because you're inside of screen now, if you want to set variables like DEBATHENA_MIRROR that get recognized by chroot-sources or do-build (or anything called as part of the build process), you need to use screen's "setenv" command. New screen windows inherit from the environment that _launched_ screen, not from window 0 (where you are now). e.g. export DEBATHENA_MIRROR=foo won't work. Instead use screen -X setenv DEBATHENA_MIRROR foo

  1. mkdir /mit/debathena/machines/awesome-build-server/stamps.$suite
  1. On your local machine, cd into a checkout of debathena/scripts/build-server/build-all. Edit the Makefile so that the variable suite is the codename of the new suite, and psuite is the codename of the most recent Debian or Ubuntu distribution.
  1. Check in your changes from the previous step, and update (svn up) /mit/debathena/bin/build-server/build-all
  1. Back in the screen session on the build server, cd to /mit/debathena/bin/build-server/build-all and run make deps.mk
  1. Run make -k all. This begins the build. The actual builds happen in window 1 of screen. It may or may not be possible or a good idea to run several builds at once with make -j <N> all.

Now, the build will almost certainly fail at some point, because of upstream changes, or incompatibilities, or whatever. When that happens, you will be prompted (in window 1) to acknowledge the error by pressing <Enter>. Make a note of the package that failed, and the failure. Let the build continue as much as it can.

When you go back to fix changes, you need to not only check the fixes in to svn, but you also need to build that package normally and upload it (ideally to -development), because: a) you want to avoid version skew between suites b) the build process uses the latest source package in the repo.

Once you have fixed, built, and uploaded the new package, you can restart the build with make -k all. Lather, rinse, repeat until everything is built.

Re-building a suite

This procedure is for a full rebuild of a suite, not picking up where a failed build left off.

  1. Go-back and re-read the part about the apt repository being up to date in step 7 above.
  1. Edit scripts/debian-versions.sh and increment the tag for the suite by 0.1. If you are doing the final build for a released suite, remove the tag entirely.
  1. Edit debathena/scripts/build-server/build-all/Makefile and set psuite to be the same as suite, since you're re-building.
  1. Check in the changes from above and svn up in the locker.
  1. Either delete /mit/debathena/machines/awesome-build-server/stamps.$suite (from the last build) or move it out of the way.
  1. Log in to the build server, start up screen, using the "-S" option to identify your screen session with your username.
  1. cd /mit/debathena/bin/build-server/build-all and run make deps.mk.
  1. Run make -k all. See the notes above about what you need to do when fixing a package.

Moving a suite to production

The build uploads built packages to the -development repository for the suite in question. You will need to move them to production.

The best way to do this is with dareprepro on the build server.

First, run this command, which should list all the packages in -development, and sanity check the output. In particular, look for packages that still have an 0.1 tag, for example

dareprepro listmatched $suite-development '*'

If everything looks good, you can do the actual move with:

dareprepro copymatched $suite $suite-development '*'`
dareprepro removematched $suite-development '*'