source: trunk/debathena/NOTES @ 25325

Revision 25325, 25.1 KB checked in by jdreed, 13 years ago (diff)
Clarify some stuff about deleting/staging a chroot
Line 
1This hierarchy contains Debian/Ubuntu-specific materials, also known
2as "Debathena".  The contents are:
3
4* debathena - Debathena-specific software packages such as PAM and NSS
5  modules.
6
7* config - Packages for configuring native system software in a manner
8  appropriate for Athena.
9
10* meta - Packages which contain nothing but dependencies on other
11  packages and serve as an installation convenience.
12
13* scripts - Build scripts and supporting materials.
14
15Debian software used by Debathena:
16
17  * schroot - Used to manage build chroot environments for each
18    Debian/Ubuntu version.  We use the device schroot type with aufs
19    union mounts, which allows rapid construction of ephemeral copies
20    of template "source" chroots, so that every binary package build
21    is done in a clean environment.
22
23  * debuild - Used to create Debian source packages from package
24    source directories.
25
26  * sbuild - Used to build binary packages from source packages inside
27    schroot environments.
28
29  * equivs - Used to create packages which only contain dependency
30    information.  Somewhat of a dirty hack, since it doesn't keep
31    proper changelogs, but it reduces overhead.
32
33  * CDBS (Common Debian Build System) - Referenced by debian/rules
34    files in packages.  Contains standard build rules to cut down on
35    per-package boilerplate.
36
37  * reprepro - Used to upload packages into the apt repositories.
38
39  * approx - Used to create a local cache of Debian packages on the
40    build server.  This cache is referenced by the build chroots for
41    improved performance.
42
43The remainder of this file documents procedures useful to Debathena
44developers and maintainers.
45
46Developers: Preferences setup
47-----------------------------
48
49You will probably want a $HOME/.devscripts file containing the
50following:
51
52DEBUILD_DPKG_BUILDPACKAGE_OPTS="-sa -us -uc -i -I.svn"
53
54This will save you from having to specify those options every time you
55run debuild.  The options mean:
56
57  * Look for original source as a tarfile or create one.
58  * Do not sign the source package.
59  * Do not sign the changes file.
60  * Ignore common version control metadata files when creating diffs.
61  * Ignore .svn paths when creating tarballs.
62
63You will also want a $HOME/.sbuildrc file containing the following:
64
65  $nolog = 1;
66  $mailto = 'yourusername';
67  $log_dir = '/tmp/sbuild-logs';
68  $maintainer_name = 'Debathena Project <debathena@mit.edu>';
69  $force_orig_source = 1;
70  $sbuild_mode = "user";
71  1;
72
73You should also set the environment variable DEBATHENA_APT to
74"/afs/sipb.mit.edu/project/debathena/apt".
75
76Developers: Preparing a change
77------------------------------
78
79To prepare a change to a regular package (a source tree containing a
80debian/ subdir), make the edits in a checkout and record a changelog
81entry.  You can either edit debian/changelog using emacs changelog
82mode (C-c C-v to add a new version entry, C-c C-a to add a change
83entry, C-c C-f to finalize the entry) or you can run "dadch" from
84the debathena locker / SVN scripts directory.
85
86When creating a new version entry, bump the upstream version number
87(to 10.0.0 if it was not already that high) if you are changing the
88main package source.  Otherwise, just bump the Debian version
89component (change 0debathena1 to 0debathena2, for instance).
90
91Developers: Building a package for test purposes on one platform
92----------------------------------------------------------------
93
94After you have prepared a change, you will want to test that it builds
95and perhaps that it works before committing it.  First, if it is an
96Athena source directory using autoconf, run "daconfiscate" to set up
97the autoconf boilerplate which we don't check in.  Second, run
98"daorig" to copy or create an orig tarball in the parent directory if
99necessary.  Third, run "debuild".  The resulting package will be
100placed in the parent directory.
101
102In order to test if the package works, you can install it with "dpkg
103-i filename.deb".
104
105Developers: Building a package for test purposes on all platforms
106-----------------------------------------------------------------
107
108If the package you are working on interacts with the native OS in ways
109that might vary from platform to platform, you may want to do a test
110build for all platforms.  You will need to do this on
111linux-build-10.mit.edu or another machine which has been set up with
112build schroots.
113
114As above, run daconfiscate (if necessary) and then daorig.  Then run
115"debuild -S" to create a source package.  Now cd into the parent
116directory and identify the .dsc file created by debuild -S; it will
117have a name like debathena-just_9.4.0-0debathena2.dsc.  Run "da
118sbuildhack filename.dsc" to perform the package builds.  Each build
119will take place inside an ephemeral chroot based on a snapshot of a
120template for a particular Debian or Ubuntu version.  If a build fails
121and it's not obvious from the build log why, you may need to create
122your own ephemeral chroot session with a command like "schroot -c
123gutsy-amd64-sbuild /bin/sh" and then run debuild from within the
124package sources.
125
126If the build is successful, it will create a set of packages with
127names like debathena-just_9.4.0-0debathena2~ubuntu6.06_amd64.deb.
128
129Developers: Building an equivs package
130--------------------------------------
131
132Most of the packages under debathena/meta are faked up using equivs.
133To build one, just run:
134
135  equivs-build --full filename.equivs
136
137These equivs files make reference to ../common, so you must have a
138checkout of debathena/meta/common alongside the particular
139meta-package you are building.
140
141Developers: The meaning of metapackages
142---------------------------------------
143
144If you are adding a new package to the repository, you will probably
145at some point want to add it to one of the metapackages so that it
146doesn't have to be installed by hand.  Here are some descriptions
147which may help identify which metapackage is best:
148
149* locker: Provides access to Athena locker software--AFS and
150  automounter configuration, locker-related utilities, etc.
151
152* clients: Provides clients (either locally-written, like athinfo and
153  Discuss, or configurations) for Athena services, as well as
154  Athena-specific utility programs like "jot".  Configurations for
155  graphical client software are generally in the workstation package
156  instead, in order to make this package less intrusive.
157
158* standard: Implies locker and clients.  Also provides Athena shell
159  customizations and dotfiles.
160
161* login: Implies standard.  Configurations to merge the MIT user
162  namespace into the local machine namespace for the purpose of user
163  lookups and authentication.
164
165* login-graphical: Implies login.  Configurations for the graphical login
166  system and graphical client software intended to provide a standard
167  X login experience using Athena home directories.
168
169* extra-software: Implies a set of Debian packages common to cluster
170  machines and typical workstations.  The resulting software set is
171  rather large, and thus may not be desirable to all workstation
172  configurations.  Only stock Debian packages belong in this
173  metapackage; do not add other Debathena packages to it.
174
175* workstation: Implies login-graphical and extra-software.  The use case
176  is the same as that of the old "private workstations", so it includes
177  more centrally-managed configuration than standard or login does
178  (unattended updates, etc.). New as of May 2009.
179
180* thirdparty: Software supported by 3partysw that upstream Ubuntu
181  already happens to be keeping up to date, and is therefore easier to
182  install via packages than through lockers. This package is HUGE (1 GB of
183  network traffic, 3 GB installed), and is also not strongly tested on
184  Debian or on older releases of Ubuntu.
185
186* cluster: Implies workstation and thirdparty. Configures public cluster
187  machines (login-time LVM snapshots, cleanups between logins, public
188  root password, etc.). The snapshot code requires a particular LVM
189  layout that the PXE (netboot) cluster installer sets up.
190
191* debian-dev: Intended for developers of the system itself; provides a
192  set of Debian packages used by Debathena for development.
193
194For the most part a package should be listed in the "Depends:" line of
195a metapackage, but in some cases it is appropriate to hedge by using
196"Recommends:", which will cause aptitude to succeed even if the
197package is unavailable.  For example, a package which doesn't exist in
198all Debian/Ubuntu suites or isn't free can be listed under
199"Recommends:" so that our metapackages still work in all environments.
200
201Developers: Index of cluster packages
202-------------------------------------
203
204Some private workstation admins may want specific pieces of cluster
205infrastructure.  With Athena 9.4, they would typically install the
206entire Athena software suite and then turn off features they did not
207want via /etc/athena/rc.conf.  That is not generally possible in
208Athena 10; a debathena-cluster machine acts like a cluster machine in
209all respects.  Instead, the machine owner should install
210debathena-workstation and then install specific packages they might be
211interested in.  At this time, candidate packages include:
212
213  * debathena-dns-config: Installs a caching DNS resolver and
214    configures the machine to use it.
215
216  * debathena-tmp-cleaner: Cleans files in /tmp and /var/tmp which
217    have not been recently accessed.
218
219  * debathena-reactivate: Causes each graphical login to be performed
220    in a separate ephemeral snapshot of the chroot.  This package
221    requires the root filesystem to be an LVM logical volume inside a
222    volume group with 21GB of free space for the snapshots; if that is
223    not the case, it will break graphical logins.  Install with care.
224
225  * debathena-auto-update: Periodically updates the machine's software
226    and reboots (when no one is logged in) if necessary.
227
228Cluster packages which are generally not of interest to private
229machine owners include:
230
231  * debathena-cluster-login-config: Configures a variety of system
232    services to implement the cluster login policy (no tty logins, no
233    user switching, screensaver logout button after 20 minutes, etc.).
234
235  * debathena-clusterinfo: Looks up the machine's cluster information
236    in Hesiod and caches it on local disk.
237
238  * debathena-larvnet: Reports to the central larvnet server whether
239    anyone has a graphical login to the machine.
240
241  * debathena-syslog-config: Configures sysklogd so that some syslog
242    messages are forwarded to a central logging host.
243
244Release engineer: Bootstrapping the project infrastructure
245----------------------------------------------------------
246
247  1. Create the package repository (detailed instructions on this
248     pending).  Set the DEBATHENA_APT environment variable to point to
249     the package repository.  Put a copy of the debathena "scripts"
250     subdir in your path.
251
252  2. Create the build area.
253
254  3. Run 'dareprepro export' to set up the repository.  Build each
255     equivs package under meta/ using 'equivs-build --full *.equivs'
256     and upload each with 'daupload-equivs-proposed *.changes',
257     followed by 'damove -proposed "" PACKAGE' (replacing PACKAGE with
258     the name of each equivs package you are moving).
259
260  4. Set up the build server.  The basic structure of the apt
261     repository must work for make-chroot to succeed, so this must
262     happen after step 3.
263
264  5. For each normal Debian package in dependency order, cd into its
265     directory in the build area and run "da sbuildhack *.dsc" and
266     "daupload-release *_source.changes".  If the package contains
267     only an "Architecture: all" binary package, pass the -A option to
268     both commands.
269
270     The all-packages script can generate an approximation of the
271     package list in dependency order, but it doesn't work right yet,
272     and ideally it would be possible to do several builds in parallel
273     using a Makefile like the one in scripts/build-server/build-all.
274     Improvements to this machinery are pending.
275
276  6. For each package under third, run "da ./debathenify-PKG source
277     binary upload".  Any created directories under third/openafs/meta
278     should be chmodded 777 to work around a perl/AFS permissions
279     issue with File::Temp; if this is not done, OpenAFS metapackage
280     builds will fail for other users.
281
282Release engineer: Updating the apt repository
283---------------------------------------------
284
285  1. ssh to the build server as the builder account and change to the
286     canonical build directory.
287
288  2. Run "gen-packages" to update the package list.  (Or "gen-packages
289     -c" if you know the AFS checkout of the source tree is up to date;
290     it should update every half hour.)
291
292  3. Run "ood-packages" to produce a list of out-of-date packages.
293
294  4. For each out of date package, run "dasource PKG".  Then change to
295     the package directory and run "da sbuildhack *.dsc" and
296     "daupload-release *_source.changes".  If the package contains
297     only an "Architecture: all" binary package, pass the -A option to
298     both commands.
299
300  5. svn update the meta directory.  If there are new subdirectories,
301     chmod them 777 to work around a perl/AFS permissions issue with
302     File::Temp.  For each updated subdir, change to it, run
303     "equivs-build --full *.equivs", and then "daequivsupload
304     FILENAME.changes" on the produced changes file.
305
306  6. svn update the third directory.  You can let autodebathenify
307     handle the updated scripts, or you can touch
308     ~/autodebathenify.suppress, make sure it's not running, and run
309     "da ./debathenify-PKG source binary upload" in each updated
310     directory.
311
312Sometimes you may have to mix up the order of the above steps in order
313to handle build dependencies.
314
315Release engineer: Setting up a build server
316-------------------------------------------
317
318  1. Install either Debian testing (or stable, if it's new enough, but
319     see the note about sbuild below) or the latest Ubuntu release.
320     The build server must be installed with free space in an LVM
321     volume group.  The build chroots consume 4GB each.
322
323  2. Install debathena-login as per the the instructions in
324     http://debathena.mit.edu/install.
325
326  3. apt-key add
327     /afs/sipb.mit.edu/project/debathena/apt/debathena-archive-keyring.asc
328
329  4. Install the packages listed in scripts/build-server/packages
330     (using "aptitude install").
331
332     Note that currently the build system uses the latest version of
333     sbuild, so if you have not installed Debian testing, you'll need
334     to set up apt pinning to grab sbuild from there. See the
335     apt_preferences man page. You'll also need to grab schroot from
336     Debian experimental.
337
338  5. Install the aufs-modules-2.6-amd64 package.
339
340  6. Append to /etc/approx/approx.conf the contents of
341      scripts/build-server/approx.conf.tail.
342     Run: /etc/init.d/approx restart
343
344  7. Apply scripts/build-server/mount-defaults.patch and
345     scripts/build-server/pam-schroot.patch.
346
347  8. For each supported DIST (see scripts/debian-versions.sh) run:
348
349       scripts/build-server/make-chroot DIST i386
350       scripts/build-server/make-chroot DIST amd64
351
352     Example: scripts/build-server/make-chroot intrepid i386
353
354  9. Create a local account for builder with:
355
356       adduser --uid 1047 --disabled-password builder
357
358     Make the home directory mode 700.  Install a
359     daemon/linux-build-10.mit.edu keytab in the home directory as
360     "keytab".  Install a copy of the secret repository-signing key
361     (debathena@mit.edu) in the home directory's keyring with
362     something like:
363
364       kinit builder
365       gpg --export-secret-keys debathena@mit.edu | \
366         ssh -l builder machinename gpg --import
367
368     Create a file named .sbuildrc in builder's homedir containing:
369
370       $mailto = undef;
371       $log_dir = '/tmp/sbuild-logs';
372       $maintainer_name = 'Debathena Project <debathena@mit.edu>';
373       $force_orig_source = 1;
374       $sbuild_mode = "user";
375       1;
376
377     Create a file named .ssh/config in builder's homedir containing:
378
379       Host svn.mit.edu
380         User debuildsvn
381
382     Add builder to the sbuild group in /etc/group.
383
384     Copy scripts/build-server/autodebathenify to builder's homedir.
385     Create a file named autodebathenify.config in builder's homedir
386     containing:
387
388       error_addr=debathena-root@mit.edu
389       scripts_dir=/mit/debathena/bin
390       build_dir=/mit/debathena/packages/third
391       packages="lprng openafs"
392       export DEBATHENA_APT=/mit/debathena/apt
393
394     Copy scripts/build-server/autodebathenify.cron to builder's
395     homedir and install it with "crontab autodebathenify.cron".
396
397     Copy scripts/build-server/autolivebuilder to builder's
398     homedir. Create a file called autolivebuilder.config in builder's
399     homedir containing:
400
401       error_addr=debathena-root@mit.edu
402       release_version='9.04'
403       release='jaunty'
404       arch='i386'
405       mirror='mirrors.mit.edu'
406       gpg_opts=("-u" "0D8A9E8F")
407       live_dir=/net/sipb-isilon-sc.mit.edu/ifs/mirror/sipb/debathena/livecd
408
409     Create a file named .devscripts in builder's homedir containing:
410
411       DEBUILD_DPKG_BUILDPACKAGE_OPTS="-sa -us -uc -i -I.svn"
412
413     In builder's homedir, append to .bashrc:
414
415       add debathena
416       export PATH=$PATH:~/bin
417       export DEBATHENA_APT=/mit/debathena/apt
418
419
420Release engineer: Removing a build chroot on the build server
421-------------------------------------------------------------
422
423  1. Run VG=/dev/blah scripts/clean-schroots as root to make sure that
424     the build chroot is not mounted, substituting the name of the
425     volume group for blah.
426
427  2. Look in /etc/schroot/chroot.d/ and delete the file
428     corresponding to the chroot.
429
430  3. Run lvchange -an blah/chrootname
431     substituting the name of the volume group for blah and the chroot
432     name for chroot.  Example: lvchange -an dink/gutsy-i386-sbuild
433
434  4. Run lvremove blah/chrootname
435
436Release engineer: Removing a dist from the apt repository
437---------------------------------------------------------
438
439  1. Inside the apt repository, edit conf/distributions and remove the
440     distribution section.
441
442  2. Run dareprepro --delete clearvanished
443
444Release engineer: Setting up a canonical build area
445---------------------------------------------------
446
447  1. Create an empty directory and cd into it.  The canonical build
448     area lives in /mit/debathena/packages/.
449
450  2. Run gen-packages to create the table of normal Debian packages.
451
452  3. Run dasource to create subdirs and source packages for each
453     normal Debian package.
454
455  4. Create checkouts of the meta and third directories:
456
457     svn co svn+ssh://svn.mit.edu/athena/trunk/debathena/meta
458     svn co svn+ssh://svn.mit.edu/athena/trunk/debathena/third
459     chmod 777 meta/*
460
461     A couple of subdirectories of debathena/meta are normal Debian
462     packages, so this will create redundant copies of those.  Ignore
463     them; they won't be used.
464
465Release engineer: Adding a new suite or rebuilding a suite
466----------------------------------------------------------
467
468This process is rarely performed and the infrastructure for it is
469imperfect.  Substitute the name of the new suite for $suite in all
470steps below.
471
472  1. Make sure the apt repository is up to date with respect to the
473     source tree for the existing dists. Check
474     debathena.mit.edu/package-list/development and
475     debathena.mit.edu/package-list/proposed. More importantly, make
476     sure there is nothing in svn that isn't built. (An easy way to do
477     this is running the check-unbuilt-packages script.)
478
479  2. Add the new dist to scripts/debian-versions.sh (and svn up into
480     the locker).  (It is not necessary to add the new dist to codes
481     at this point, but it must be present in the gettag conditional.)
482     If you are building an unreleased distribution, add an additional
483     ~0.1 tag; if you're rebilding, bump that tag.
484
485  3. Create the new distribution in the apt repository's configuration
486     file by editing /mit/debathena/apt/conf/gen-distributions and
487     running it with output to the 'distributions' file in the same
488     directory.  Create the skeleton of the dist by running
489     "dareprepro export" on the build server. (You don't need to do
490     this if you're rebuilding.)
491
492  4. Add an appropriate sources.list.d/*/$suite.list entry in
493     scripts/build-server. Commit and update
494     /mit/debathena/bin/build-server.
495
496  5. On the build server (as root), create a chroot for the new distribution:
497
498     First, check if /usr/share/debootstrap/scripts/$suite exists.
499     If not, it may require downloading and installing a
500     more recent version of the debootstrap package from the
501     -backports dist corresponding to the build server's OS. (You
502     presumably don't need to do this if you're rebuilding.)
503
504     You could also just go into that directory and do
505     "ln -s gutsy $suite" since it apparently hasn't changed in years.
506
507        /mit/debathena/bin/build-server/make-chroot $suite i386
508        /mit/debathena/bin/build-server/make-chroot $suite amd64
509
510
511  6. Fire up screen.
512
513  7. mkdir /mit/debathena/machines/awesome-build-server/stamps.$suite.
514
515  8. cd into a checkout of debathena/scripts/build-server/build-all.
516
517  9. Edit Makefile (and check in the edit) so that suite is the new
518     distribution and psuite is the previously most recent Debian or
519     Ubuntu distribution.  psuite is used to generate the order of
520     build-dependencies, so choose the closest distribution.  If you are
521     rebuilding, you can set psuite to suite.
522
523  10. Run "make deps.mk".
524
525  11. Run "make -k all".
526      You can watch the builds happen in the other windows of the
527      screen session.  It's possible to do several builds at once with
528      make -j N (except as of 7/2011 this is a bad idea)
529
530Release engineer: Maintaining autodebathenify
531---------------------------------------------
532
533autodebathenify is a cron job which runs on the builder account and
534keeps the OpenAFS modules and modified packages under debathena/third
535up to date when there are upstream changes.  It relies on the
536following:
537
538  * builder@ATHENA.MIT.EDU has access to a local account named
539    debuildsvn on svn.mit.edu with read access to the athena
540    repository.
541
542  * builder@ATHENA.MIT.EDU has write access to the canonical build
543    area and apt repository in the dev cell.
544
545  * The builder account on the build machine has a copy of the signing
546    private key for the apt repository.
547
548The cron job will silently exit if it detects that it is already
549running.  If it fails for any other reason, it will send mail to
550source-wash@mit.edu and touch a file named autodebathenify.suppress
551which will prevent it from running again until manual intervention.
552It is important to get autodebathenify running smoothly or the apt
553repository will become out of date with respect to third-party
554packages, which in turn will compromise the user experience.
555
556Even if you don't notice failure mail, it's good to check on the
557status of autodebathenify from time to time.  To do this, get tickets
558as builder and ssh to linux-build-10 as builder.  Run "ls -l logs" to
559see what's been going on.  A logfile with size around 48K indicates
560that autodebathenify ran normally but didn't find any new work to do.
561A logfile with size around 60 means that autodebathenify.suppress
562exists and the cron job has stopped running.  A longer logfile
563indicates that autodebathenify attempted to build and upload a
564package.
565
566Possible failure cases include:
567
568  * One of the upstream apt repositories timed out during an "apt-get
569    update" of an sbuild chroot.  If this happens, you can just remove
570    the autodebathenify.suppress file to get the cron job to start
571    running again.
572
573  * autodebathenify tried to build an upstream version which already
574    exists.  If this happens, the build will fail at upload time.  If
575    this happens, there is a bug in the part of the debathenify-*
576    script which checks whether our apt repository already has the
577    expected version of the built package.
578
579  * Debian and Ubuntu have .orig files with the same name but
580    different contexts for a particular third-party package.  This has
581    only happened once, with bash (which is currently not configured
582    into autodebathenify for that reason).  We do not have a general
583    mechanism for resolving such issues at this time.
584
585  * The new upstream version of the package is sufficiently different
586    to cause our modifications to fail to apply, and our debathenify-*
587    script needs to be adapted appropriately.
588
589The cron job is run out of builder's home directory, but its canonical
590source location is in debathena/scripts/build-server in this
591repository, should it need to be modified.
592
593After several months of operation, LVM snapshots on the build server
594will start to become slower and slower, as /etc/lvm/cache/.cache grows
595in size.  Turning off the cache entirely seems to cause schroot to
596occasionally fail; the current solution is to rm /etc/lvm/cache/.cache
597every so often.
598
599Release engineer: apt repository HTTP server setup
600--------------------------------------------------
601
602The apt repository server (debathena.mit.edu) is hosted on
603scripts.mit.edu out of the debathena locker. To replicate it on a
604dedicated server, install Apache and AFS (e.g., debathena-standard
605and apache2), and configure the document root to be
606  /mit/debathena/web_scripts
607
608There's also a crontab in /mit/debathena/cron_scripts that updates
609the package lists on the website. It expects to be able to edit the
610web_scripts directory; on scripts, it does so with daemon.scripts
611permissions.
612
613Trac (http://debathena.mit.edu/trac/) expects to use sql.mit.edu
614for its database.
Note: See TracBrowser for help on using the repository browser.