source: trunk/debathena/NOTES @ 25465

Revision 25465, 25.7 KB checked in by jdreed, 12 years ago (diff)
The contents of NOTES are now in the wiki
Line 
1
2** NOTE **
3
4As of 24 April 2012, this file should be considered deprecated.  The
5canonical version of the information contained in this file is the
6Debathena Trac wiki (http://debathena.mit.edu/trac/wiki), in particular,
7the section titled "Release Engineering".  This file should be
8maintained for posterity, but should not be updated.
9
10**
11
12
13
14
15This hierarchy contains Debian/Ubuntu-specific materials, also known
16as "Debathena".  The contents are:
17
18* debathena - Debathena-specific software packages such as PAM and NSS
19  modules.
20
21* config - Packages for configuring native system software in a manner
22  appropriate for Athena.
23
24* meta - Packages which contain nothing but dependencies on other
25  packages and serve as an installation convenience.
26
27* scripts - Build scripts and supporting materials.
28
29Debian software used by Debathena:
30
31  * schroot - Used to manage build chroot environments for each
32    Debian/Ubuntu version.  We use the device schroot type with aufs
33    union mounts, which allows rapid construction of ephemeral copies
34    of template "source" chroots, so that every binary package build
35    is done in a clean environment.
36
37  * debuild - Used to create Debian source packages from package
38    source directories.
39
40  * sbuild - Used to build binary packages from source packages inside
41    schroot environments.
42
43  * equivs - Used to create packages which only contain dependency
44    information.  Somewhat of a dirty hack, since it doesn't keep
45    proper changelogs, but it reduces overhead.
46
47  * CDBS (Common Debian Build System) - Referenced by debian/rules
48    files in packages.  Contains standard build rules to cut down on
49    per-package boilerplate.
50
51  * reprepro - Used to upload packages into the apt repositories.
52
53  * approx - Used to create a local cache of Debian packages on the
54    build server.  This cache is referenced by the build chroots for
55    improved performance.
56
57The remainder of this file documents procedures useful to Debathena
58developers and maintainers.
59
60Developers: Preferences setup
61-----------------------------
62
63You will probably want a $HOME/.devscripts file containing the
64following:
65
66DEBUILD_DPKG_BUILDPACKAGE_OPTS="-sa -us -uc -i -I.svn"
67
68This will save you from having to specify those options every time you
69run debuild.  The options mean:
70
71  * Look for original source as a tarfile or create one.
72  * Do not sign the source package.
73  * Do not sign the changes file.
74  * Ignore common version control metadata files when creating diffs.
75  * Ignore .svn paths when creating tarballs.
76
77You will also want a $HOME/.sbuildrc file containing the following:
78
79  $nolog = 1;
80  $mailto = 'yourusername';
81  $log_dir = '/tmp/sbuild-logs';
82  $maintainer_name = 'Debathena Project <debathena@mit.edu>';
83  $force_orig_source = 1;
84  $sbuild_mode = "user";
85  1;
86
87You should also set the environment variable DEBATHENA_APT to
88"/afs/sipb.mit.edu/project/debathena/apt".
89
90Developers: Preparing a change
91------------------------------
92
93To prepare a change to a regular package (a source tree containing a
94debian/ subdir), make the edits in a checkout and record a changelog
95entry.  You can either edit debian/changelog using emacs changelog
96mode (C-c C-v to add a new version entry, C-c C-a to add a change
97entry, C-c C-f to finalize the entry) or you can run "dadch" from
98the debathena locker / SVN scripts directory.
99
100When creating a new version entry, bump the upstream version number
101(to 10.0.0 if it was not already that high) if you are changing the
102main package source.  Otherwise, just bump the Debian version
103component (change 0debathena1 to 0debathena2, for instance).
104
105Developers: Building a package for test purposes on one platform
106----------------------------------------------------------------
107
108After you have prepared a change, you will want to test that it builds
109and perhaps that it works before committing it.  First, if it is an
110Athena source directory using autoconf, run "daconfiscate" to set up
111the autoconf boilerplate which we don't check in.  Second, run
112"daorig" to copy or create an orig tarball in the parent directory if
113necessary.  Third, run "debuild".  The resulting package will be
114placed in the parent directory.
115
116In order to test if the package works, you can install it with "dpkg
117-i filename.deb".
118
119Developers: Building a package for test purposes on all platforms
120-----------------------------------------------------------------
121
122If the package you are working on interacts with the native OS in ways
123that might vary from platform to platform, you may want to do a test
124build for all platforms.  You will need to do this on
125linux-build-10.mit.edu or another machine which has been set up with
126build schroots.
127
128As above, run daconfiscate (if necessary) and then daorig.  Then run
129"debuild -S" to create a source package.  Now cd into the parent
130directory and identify the .dsc file created by debuild -S; it will
131have a name like debathena-just_9.4.0-0debathena2.dsc.  Run "da
132sbuildhack filename.dsc" to perform the package builds.  Each build
133will take place inside an ephemeral chroot based on a snapshot of a
134template for a particular Debian or Ubuntu version.  If a build fails
135and it's not obvious from the build log why, you may need to create
136your own ephemeral chroot session with a command like "schroot -c
137gutsy-amd64-sbuild /bin/sh" and then run debuild from within the
138package sources.
139
140If the build is successful, it will create a set of packages with
141names like debathena-just_9.4.0-0debathena2~ubuntu6.06_amd64.deb.
142
143Developers: Building an equivs package
144--------------------------------------
145
146Most of the packages under debathena/meta are faked up using equivs.
147To build one, just run:
148
149  equivs-build --full filename.equivs
150
151These equivs files make reference to ../common, so you must have a
152checkout of debathena/meta/common alongside the particular
153meta-package you are building.
154
155Developers: The meaning of metapackages
156---------------------------------------
157
158If you are adding a new package to the repository, you will probably
159at some point want to add it to one of the metapackages so that it
160doesn't have to be installed by hand.  Here are some descriptions
161which may help identify which metapackage is best:
162
163* locker: Provides access to Athena locker software--AFS and
164  automounter configuration, locker-related utilities, etc.
165
166* clients: Provides clients (either locally-written, like athinfo and
167  Discuss, or configurations) for Athena services, as well as
168  Athena-specific utility programs like "jot".  Configurations for
169  graphical client software are generally in the workstation package
170  instead, in order to make this package less intrusive.
171
172* standard: Implies locker and clients.  Also provides Athena shell
173  customizations and dotfiles.
174
175* login: Implies standard.  Configurations to merge the MIT user
176  namespace into the local machine namespace for the purpose of user
177  lookups and authentication.
178
179* login-graphical: Implies login.  Configurations for the graphical login
180  system and graphical client software intended to provide a standard
181  X login experience using Athena home directories.
182
183* extra-software: Implies a set of Debian packages common to cluster
184  machines and typical workstations.  The resulting software set is
185  rather large, and thus may not be desirable to all workstation
186  configurations.  Only stock Debian packages belong in this
187  metapackage; do not add other Debathena packages to it.
188
189* workstation: Implies login-graphical and extra-software.  The use case
190  is the same as that of the old "private workstations", so it includes
191  more centrally-managed configuration than standard or login does
192  (unattended updates, etc.). New as of May 2009.
193
194* thirdparty: Software supported by 3partysw that upstream Ubuntu
195  already happens to be keeping up to date, and is therefore easier to
196  install via packages than through lockers. This package is HUGE (1 GB of
197  network traffic, 3 GB installed), and is also not strongly tested on
198  Debian or on older releases of Ubuntu.
199
200* cluster: Implies workstation and thirdparty. Configures public cluster
201  machines (login-time LVM snapshots, cleanups between logins, public
202  root password, etc.). The snapshot code requires a particular LVM
203  layout that the PXE (netboot) cluster installer sets up.
204
205* debian-dev: Intended for developers of the system itself; provides a
206  set of Debian packages used by Debathena for development.
207
208For the most part a package should be listed in the "Depends:" line of
209a metapackage, but in some cases it is appropriate to hedge by using
210"Recommends:", which will cause aptitude to succeed even if the
211package is unavailable.  For example, a package which doesn't exist in
212all Debian/Ubuntu suites or isn't free can be listed under
213"Recommends:" so that our metapackages still work in all environments.
214
215Developers: Index of cluster packages
216-------------------------------------
217
218Some private workstation admins may want specific pieces of cluster
219infrastructure.  With Athena 9.4, they would typically install the
220entire Athena software suite and then turn off features they did not
221want via /etc/athena/rc.conf.  That is not generally possible in
222Athena 10; a debathena-cluster machine acts like a cluster machine in
223all respects.  Instead, the machine owner should install
224debathena-workstation and then install specific packages they might be
225interested in.  At this time, candidate packages include:
226
227  * debathena-dns-config: Installs a caching DNS resolver and
228    configures the machine to use it.
229
230  * debathena-tmp-cleaner: Cleans files in /tmp and /var/tmp which
231    have not been recently accessed.
232
233  * debathena-reactivate: Causes each graphical login to be performed
234    in a separate ephemeral snapshot of the chroot.  This package
235    requires the root filesystem to be an LVM logical volume inside a
236    volume group with 21GB of free space for the snapshots; if that is
237    not the case, it will break graphical logins.  Install with care.
238
239  * debathena-auto-update: Periodically updates the machine's software
240    and reboots (when no one is logged in) if necessary.
241
242Cluster packages which are generally not of interest to private
243machine owners include:
244
245  * debathena-cluster-login-config: Configures a variety of system
246    services to implement the cluster login policy (no tty logins, no
247    user switching, screensaver logout button after 20 minutes, etc.).
248
249  * debathena-clusterinfo: Looks up the machine's cluster information
250    in Hesiod and caches it on local disk.
251
252  * debathena-larvnet: Reports to the central larvnet server whether
253    anyone has a graphical login to the machine.
254
255  * debathena-syslog-config: Configures sysklogd so that some syslog
256    messages are forwarded to a central logging host.
257
258Release engineer: Bootstrapping the project infrastructure
259----------------------------------------------------------
260
261  1. Create the package repository (detailed instructions on this
262     pending).  Set the DEBATHENA_APT environment variable to point to
263     the package repository.  Put a copy of the debathena "scripts"
264     subdir in your path.
265
266  2. Create the build area.
267
268  3. Run 'dareprepro export' to set up the repository.  Build each
269     equivs package under meta/ using 'equivs-build --full *.equivs'
270     and upload each with 'daupload-equivs-proposed *.changes',
271     followed by 'damove -proposed "" PACKAGE' (replacing PACKAGE with
272     the name of each equivs package you are moving).
273
274  4. Set up the build server.  The basic structure of the apt
275     repository must work for make-chroot to succeed, so this must
276     happen after step 3.
277
278  5. For each normal Debian package in dependency order, cd into its
279     directory in the build area and run "da sbuildhack *.dsc" and
280     "daupload-release *_source.changes".  If the package contains
281     only an "Architecture: all" binary package, pass the -A option to
282     both commands.
283
284     The all-packages script can generate an approximation of the
285     package list in dependency order, but it doesn't work right yet,
286     and ideally it would be possible to do several builds in parallel
287     using a Makefile like the one in scripts/build-server/build-all.
288     Improvements to this machinery are pending.
289
290  6. For each package under third, run "da ./debathenify-PKG source
291     binary upload".  Any created directories under third/openafs/meta
292     should be chmodded 777 to work around a perl/AFS permissions
293     issue with File::Temp; if this is not done, OpenAFS metapackage
294     builds will fail for other users.
295
296Release engineer: Updating the apt repository
297---------------------------------------------
298
299  1. ssh to the build server as the builder account and change to the
300     canonical build directory.
301
302  2. Run "gen-packages" to update the package list.  (Or "gen-packages
303     -c" if you know the AFS checkout of the source tree is up to date;
304     it should update every half hour.)
305
306  3. Run "ood-packages" to produce a list of out-of-date packages.
307
308  4. For each out of date package, run "dasource PKG".  Then change to
309     the package directory and run "da sbuildhack *.dsc" and
310     "daupload-release *_source.changes".  If the package contains
311     only an "Architecture: all" binary package, pass the -A option to
312     both commands.
313
314  5. svn update the meta directory.  If there are new subdirectories,
315     chmod them 777 to work around a perl/AFS permissions issue with
316     File::Temp.  For each updated subdir, change to it, run
317     "equivs-build --full *.equivs", and then "daequivsupload
318     FILENAME.changes" on the produced changes file.
319
320  6. svn update the third directory.  You can let autodebathenify
321     handle the updated scripts, or you can touch
322     ~/autodebathenify.suppress, make sure it's not running, and run
323     "da ./debathenify-PKG source binary upload" in each updated
324     directory.
325
326Sometimes you may have to mix up the order of the above steps in order
327to handle build dependencies.
328
329Release engineer: Setting up a build server
330-------------------------------------------
331
332  1. Install either Debian testing (or stable, if it's new enough, but
333     see the note about sbuild below) or the latest Ubuntu release.
334     The build server must be installed with free space in an LVM
335     volume group.  The build chroots consume 4GB each.
336
337  2. Install debathena-login as per the the instructions in
338     http://debathena.mit.edu/install.
339
340  3. apt-key add
341     /afs/sipb.mit.edu/project/debathena/apt/debathena-archive-keyring.asc
342
343  4. Install the packages listed in scripts/build-server/packages
344     (using "aptitude install").
345
346     Note that currently the build system uses the latest version of
347     sbuild, so if you have not installed Debian testing, you'll need
348     to set up apt pinning to grab sbuild from there. See the
349     apt_preferences man page. You'll also need to grab schroot from
350     Debian experimental.
351
352  5. Install the aufs-modules-2.6-amd64 package.
353
354  6. Append to /etc/approx/approx.conf the contents of
355      scripts/build-server/approx.conf.tail.
356     Run: /etc/init.d/approx restart
357
358  7. Apply scripts/build-server/mount-defaults.patch and
359     scripts/build-server/pam-schroot.patch.
360
361  8. For each supported DIST (see scripts/debian-versions.sh) run:
362
363       scripts/build-server/make-chroot DIST i386
364       scripts/build-server/make-chroot DIST amd64
365
366     Example: scripts/build-server/make-chroot intrepid i386
367
368  9. Create a local account for builder with:
369
370       adduser --uid 1047 --disabled-password builder
371
372     Make the home directory mode 700.  Install a
373     daemon/linux-build-10.mit.edu keytab in the home directory as
374     "keytab".  Install a copy of the secret repository-signing key
375     (debathena@mit.edu) in the home directory's keyring with
376     something like:
377
378       kinit builder
379       gpg --export-secret-keys debathena@mit.edu | \
380         ssh -l builder machinename gpg --import
381
382     Create a file named .sbuildrc in builder's homedir containing:
383
384       $mailto = undef;
385       $log_dir = '/tmp/sbuild-logs';
386       $maintainer_name = 'Debathena Project <debathena@mit.edu>';
387       $force_orig_source = 1;
388       $sbuild_mode = "user";
389       1;
390
391     Create a file named .ssh/config in builder's homedir containing:
392
393       Host svn.mit.edu
394         User debuildsvn
395
396     Add builder to the sbuild group in /etc/group.
397
398     Copy scripts/build-server/autodebathenify to builder's homedir.
399     Create a file named autodebathenify.config in builder's homedir
400     containing:
401
402       error_addr=debathena-root@mit.edu
403       scripts_dir=/mit/debathena/bin
404       build_dir=/mit/debathena/packages/third
405       packages="lprng openafs"
406       export DEBATHENA_APT=/mit/debathena/apt
407
408     Copy scripts/build-server/autodebathenify.cron to builder's
409     homedir and install it with "crontab autodebathenify.cron".
410
411     Copy scripts/build-server/autolivebuilder to builder's
412     homedir. Create a file called autolivebuilder.config in builder's
413     homedir containing:
414
415       error_addr=debathena-root@mit.edu
416       release_version='9.04'
417       release='jaunty'
418       arch='i386'
419       mirror='mirrors.mit.edu'
420       gpg_opts=("-u" "0D8A9E8F")
421       live_dir=/net/sipb-isilon-sc.mit.edu/ifs/mirror/sipb/debathena/livecd
422
423     Create a file named .devscripts in builder's homedir containing:
424
425       DEBUILD_DPKG_BUILDPACKAGE_OPTS="-sa -us -uc -i -I.svn"
426
427     In builder's homedir, append to .bashrc:
428
429       add debathena
430       export PATH=$PATH:~/bin
431       export DEBATHENA_APT=/mit/debathena/apt
432
433
434Release engineer: Removing a build chroot on the build server
435-------------------------------------------------------------
436
437  1. Run VG=/dev/blah scripts/clean-schroots as root to make sure that
438     the build chroot is not mounted, substituting the name of the
439     volume group for blah.
440
441  2. Look in /etc/schroot/chroot.d/ and delete the file
442     corresponding to the chroot.
443
444  3. Run lvchange -an blah/chrootname
445     substituting the name of the volume group for blah and the chroot
446     name for chroot.  Example: lvchange -an dink/gutsy-i386-sbuild
447
448  4. Run lvremove blah/chrootname
449
450Release engineer: Removing a dist from the apt repository
451---------------------------------------------------------
452
453  1. Inside the apt repository, edit conf/distributions and remove the
454     distribution section.
455
456  2. Run dareprepro --delete clearvanished
457
458Release engineer: Setting up a canonical build area
459---------------------------------------------------
460
461  1. Create an empty directory and cd into it.  The canonical build
462     area lives in /mit/debathena/packages/.
463
464  2. Run gen-packages to create the table of normal Debian packages.
465
466  3. Run dasource to create subdirs and source packages for each
467     normal Debian package.
468
469  4. Create checkouts of the meta and third directories:
470
471     svn co svn+ssh://svn.mit.edu/athena/trunk/debathena/meta
472     svn co svn+ssh://svn.mit.edu/athena/trunk/debathena/third
473     chmod 777 meta/*
474
475     A couple of subdirectories of debathena/meta are normal Debian
476     packages, so this will create redundant copies of those.  Ignore
477     them; they won't be used.
478
479Release engineer: Adding a new suite or rebuilding a suite
480----------------------------------------------------------
481
482This process is rarely performed and the infrastructure for it is
483imperfect.  Substitute the name of the new suite for $suite in all
484steps below.
485
486  1. Make sure the apt repository is up to date with respect to the
487     source tree for the existing dists. Check
488     debathena.mit.edu/package-list/development and
489     debathena.mit.edu/package-list/proposed. More importantly, make
490     sure there is nothing in svn that isn't built. (An easy way to do
491     this is running the check-unbuilt-packages script.)
492
493  2. Add the new dist to scripts/debian-versions.sh (and svn up into
494     the locker).  (It is not necessary to add the new dist to codes
495     at this point, but it must be present in the gettag conditional.)
496     If you are building an unreleased distribution, add an additional
497     ~0.1 tag; if you're rebilding, bump that tag.
498
499  3. Create the new distribution in the apt repository's configuration
500     file by editing /mit/debathena/apt/conf/gen-distributions and
501     running it with output to the 'distributions' file in the same
502     directory.  Create the skeleton of the dist by running
503     "dareprepro export" on the build server. (You don't need to do
504     this if you're rebuilding.)
505
506  4. Add an appropriate sources.list.d/*/$suite.list entry in
507     scripts/build-server. Commit and update
508     /mit/debathena/bin/build-server.
509
510  5. On the build server (as root), create a chroot for the new distribution:
511
512     First, check if /usr/share/debootstrap/scripts/$suite exists.
513     If not, it may require downloading and installing a
514     more recent version of the debootstrap package from the
515     -backports dist corresponding to the build server's OS. (You
516     presumably don't need to do this if you're rebuilding.)
517
518     You could also just go into that directory and do
519     "ln -s gutsy $suite" since it apparently hasn't changed in years.
520
521        /mit/debathena/bin/build-server/make-chroot $suite i386
522        /mit/debathena/bin/build-server/make-chroot $suite amd64
523
524
525  6. Fire up screen, ideally making use of the "-S" option to identify
526     your screen session.  e.g. "screen -S joeuser.oneiric_build".
527
528  7. mkdir /mit/debathena/machines/awesome-build-server/stamps.$suite.
529
530  8. cd into a checkout (probably on your local machine) of
531     debathena/scripts/build-server/build-all.
532
533  9. Edit Makefile (and check in the edit) so that suite is the new
534     distribution and psuite is the previously most recent Debian or
535     Ubuntu distribution.  psuite is used to generate the order of
536     build-dependencies, so choose the closest distribution.  If you are
537     rebuilding, you can set psuite to suite.
538
539  10. cd to /mit/debathena/bin/build-server/build-all, which is a
540      totally logical place to run the build from.  Run "make
541      deps.mk".
542
543  11. Run "make -k all".
544      You can watch the builds happen in the other windows of the
545      screen session.  It's possible to do several builds at once with
546      make -j N (except as of 7/2011 this is a bad idea)
547
548Release engineer: Maintaining autodebathenify
549---------------------------------------------
550
551autodebathenify is a cron job which runs on the builder account and
552keeps the OpenAFS modules and modified packages under debathena/third
553up to date when there are upstream changes.  It relies on the
554following:
555
556  * builder@ATHENA.MIT.EDU has access to a local account named
557    debuildsvn on svn.mit.edu with read access to the athena
558    repository.
559
560  * builder@ATHENA.MIT.EDU has write access to the canonical build
561    area and apt repository in the dev cell.
562
563  * The builder account on the build machine has a copy of the signing
564    private key for the apt repository.
565
566The cron job will silently exit if it detects that it is already
567running.  If it fails for any other reason, it will send mail to
568source-wash@mit.edu and touch a file named autodebathenify.suppress
569which will prevent it from running again until manual intervention.
570It is important to get autodebathenify running smoothly or the apt
571repository will become out of date with respect to third-party
572packages, which in turn will compromise the user experience.
573
574Even if you don't notice failure mail, it's good to check on the
575status of autodebathenify from time to time.  To do this, get tickets
576as builder and ssh to linux-build-10 as builder.  Run "ls -l logs" to
577see what's been going on.  A logfile with size around 48K indicates
578that autodebathenify ran normally but didn't find any new work to do.
579A logfile with size around 60 means that autodebathenify.suppress
580exists and the cron job has stopped running.  A longer logfile
581indicates that autodebathenify attempted to build and upload a
582package.
583
584Possible failure cases include:
585
586  * One of the upstream apt repositories timed out during an "apt-get
587    update" of an sbuild chroot.  If this happens, you can just remove
588    the autodebathenify.suppress file to get the cron job to start
589    running again.
590
591  * autodebathenify tried to build an upstream version which already
592    exists.  If this happens, the build will fail at upload time.  If
593    this happens, there is a bug in the part of the debathenify-*
594    script which checks whether our apt repository already has the
595    expected version of the built package.
596
597  * Debian and Ubuntu have .orig files with the same name but
598    different contexts for a particular third-party package.  This has
599    only happened once, with bash (which is currently not configured
600    into autodebathenify for that reason).  We do not have a general
601    mechanism for resolving such issues at this time.
602
603  * The new upstream version of the package is sufficiently different
604    to cause our modifications to fail to apply, and our debathenify-*
605    script needs to be adapted appropriately.
606
607The cron job is run out of builder's home directory, but its canonical
608source location is in debathena/scripts/build-server in this
609repository, should it need to be modified.
610
611After several months of operation, LVM snapshots on the build server
612will start to become slower and slower, as /etc/lvm/cache/.cache grows
613in size.  Turning off the cache entirely seems to cause schroot to
614occasionally fail; the current solution is to rm /etc/lvm/cache/.cache
615every so often.
616
617Release engineer: apt repository HTTP server setup
618--------------------------------------------------
619
620The apt repository server (debathena.mit.edu) is hosted on
621scripts.mit.edu out of the debathena locker. To replicate it on a
622dedicated server, install Apache and AFS (e.g., debathena-standard
623and apache2), and configure the document root to be
624  /mit/debathena/web_scripts
625
626There's also a crontab in /mit/debathena/cron_scripts that updates
627the package lists on the website. It expects to be able to edit the
628web_scripts directory; on scripts, it does so with daemon.scripts
629permissions.
630
631Trac (http://debathena.mit.edu/trac/) expects to use sql.mit.edu
632for its database.
Note: See TracBrowser for help on using the repository browser.