source: trunk/debathena/NOTES @ 24875

Revision 24875, 25.0 KB checked in by lizdenys, 14 years ago (diff)
added how to check that the apt repo is up to date
Line 
1This hierarchy contains Debian/Ubuntu-specific materials, also known
2as "Debathena".  The contents are:
3
4* debathena - Debathena-specific software packages such as PAM and NSS
5  modules.
6
7* config - Packages for configuring native system software in a manner
8  appropriate for Athena.
9
10* meta - Packages which contain nothing but dependencies on other
11  packages and serve as an installation convenience.
12
13* scripts - Build scripts and supporting materials.
14
15Debathena is a SIPB project, and its infrastructure and procedures
16will need to be adapted for Athena 10.  For the moment this file will
17document the Debathena procedures as they are, not as they will be.
18The current procedures do not even use this svn repository yet.
19
20Debian software used by Debathena:
21
22  * schroot - Used to manage build chroot environments for each
23    Debian/Ubuntu version.  We use the device schroot type with aufs
24    union mounts, which allows rapid construction of ephemeral copies
25    of template "source" chroots, so that every binary package build
26    is done in a clean environment.
27
28  * debuild - Used to create Debian source packages from package
29    source directories.
30
31  * sbuild - Used to build binary packages from source packages inside
32    schroot environments.
33
34  * equivs - Used to create packages which only contain dependency
35    information.  Somewhat of a dirty hack, since it doesn't keep
36    proper changelogs, but it reduces overhead.
37
38  * CDBS (Common Debian Build System) - Referenced by debian/rules
39    files in packages.  Contains standard build rules to cut down on
40    per-package boilerplate.
41
42  * reprepro - Used to upload packages into the apt repositories.
43
44  * approx - Used to create a local cache of Debian packages on the
45    build server.  This cache is referenced by the build chroots for
46    improved performance.
47
48The remainder of this file documents procedures useful to Athena 10
49developers and the release engineer.
50
51Developers: Preferences setup
52-----------------------------
53
54You will probably want a $HOME/.devscripts file containing the
55following:
56
57DEBUILD_DPKG_BUILDPACKAGE_OPTS="-sa -us -uc -i -I.svn"
58
59This will save you from having to specify those options every time you
60run debuild.  Athena 10 scripts do not assume the above preferences,
61but the instructions in this file do.  The options mean:
62
63  * Look for original source as a tarfile or create one.
64  * Do not sign the source package.
65  * Do not sign the changes file.
66  * Ignore common version control metadata files when creating diffs.
67  * Ignore .svn paths when creating tarballs.
68
69You will also want a $HOME/.sbuildrc file containing the following:
70
71  $nolog = 1;
72  $mailto = 'yourusername';
73  $log_dir = '/tmp/sbuild-logs';
74  $maintainer_name = 'Debathena Project <debathena@mit.edu>';
75  $force_orig_source = 1;
76  $sbuild_mode = "user";
77  1;
78
79You should also set the environment variable DEBATHENA_APT to
80"/afs/dev.mit.edu/system/athena10/apt".
81
82Developers: Preparing a change
83------------------------------
84
85To prepare a change to a regular package (a source tree containing a
86debian/ subdir), make the edits in a checkout and record a changelog
87entry.  You can either edit debian/changelog using emacs changelog
88mode (C-c C-v to add a new version entry, C-c C-a to add a change
89entry, C-c C-f to finalize the entry) or you can run "dadch" from
90the debathena locker / SVN scripts directory.
91
92When creating a new version entry, bump the upstream version number
93(to 10.0.0 if it was not already that high) if you are changing the
94main package source.  Otherwise, just bump the Debian version
95component (change 0debathena1 to 0debathena2, for instance).
96
97Developers: Building a package for test purposes on one platform
98----------------------------------------------------------------
99
100After you have prepared a change, you will want to test that it builds
101and perhaps that it works before committing it.  First, if it is an
102Athena source directory using autoconf, run "daconfiscate" to set up
103the autoconf boilerplate which we don't check in.  Second, run
104"daorig" to copy or create an orig tarball in the parent directory if
105necessary.  Third, run "debuild".  The resulting package will be
106placed in the parent directory.
107
108In order to test if the package works, you can install it with "dpkg
109-i filename.deb".
110
111Developers: Building a package for test purposes on all platforms
112-----------------------------------------------------------------
113
114If the package you are working on interacts with the native OS in ways
115that might vary from platform to platform, you may want to do a test
116build for all platforms.  You will need to do this on
117linux-build-10.mit.edu or another machine which has been set up with
118build schroots.
119
120As above, run daconfiscate (if necessary) and then daorig.  Then run
121"debuild -S" to create a source package.  Now cd into the parent
122directory and identify the .dsc file created by debuild -S; it will
123have a name like debathena-just_9.4.0-0debathena2.dsc.  Run "da
124sbuildhack filename.dsc" to perform the package builds.  Each build
125will take place inside an ephemeral chroot based on a snapshot of a
126template for a particular Debian or Ubuntu version.  If a build fails
127and it's not obvious from the build log why, you may need to create
128your own ephemeral chroot session with a command like "schroot -c
129gutsy-amd64-sbuild /bin/sh" and then run debuild from within the
130package sources.
131
132If the build is successful, it will create a set of packages with
133names like debathena-just_9.4.0-0debathena2~ubuntu6.06_amd64.deb.
134
135Developers: Building an equivs package
136--------------------------------------
137
138Most of the packages under debathena/meta are faked up using equivs.
139To build one, just run:
140
141  equivs-build --full filename.equivs
142
143These equivs files make reference to ../common, so you must have a
144checkout of debathena/meta/common alongside the particular
145meta-package you are building.
146
147Developers: The meaning of metapackages
148---------------------------------------
149
150If you are adding a new package to the repository, you will probably
151at some point want to add it to one of the metapackages so that it
152doesn't have to be installed by hand.  Here are some descriptions
153which may help identify which metapackage is best:
154
155* locker: Provides access to Athena locker software--AFS and
156  automounter configuration, locker-related utilities, etc.
157
158* clients: Provides clients (either locally-written, like athinfo and
159  Discuss, or configurations) for Athena services, as well as
160  Athena-specific utility programs like "jot".  Configurations for
161  graphical client software are generally in the workstation package
162  instead, in order to make this package less intrusive.
163
164* standard: Implies locker and clients.  Also provides Athena shell
165  customizations and dotfiles.
166
167* login: Implies standard.  Configurations to merge the MIT user
168  namespace into the local machine namespace for the purpose of user
169  lookups and authentication.
170
171* login-graphical: Implies login.  Configurations for the graphical login
172  system and graphical client software intended to provide a standard
173  X login experience using Athena home directories.
174
175* extra-software: Implies a set of Debian packages common to cluster
176  machines and typical workstations.  The resulting software set is
177  rather large, and thus may not be desirable to all workstation
178  configurations.  Only stock Debian packages belong in this
179  metapackage; do not add other Debathena packages to it.
180
181* workstation: Implies login-graphical and extra-software.  The use case
182  is the same as that of the old "private workstations", so it includes
183  more centrally-managed configuration than standard or login does
184  (unattended updates, etc.). New as of May 2009.
185
186* thirdparty: Software supported by 3partysw that upstream Ubuntu
187  already happens to be keeping up to date, and is therefore easier to
188  install via packages than through lockers. This package is HUGE (1 GB of
189  network traffic, 3 GB installed), and is also not strongly tested on
190  Debian or on older releases of Ubuntu.
191
192* cluster: Implies workstation and thirdparty. Configures public cluster
193  machines (login-time LVM snapshots, cleanups between logins, public
194  root password, etc.). The snapshot code requires a particular LVM
195  layout that the PXE (netboot) cluster installer sets up.
196
197* debian-dev: Intended for developers of the system itself; provides a
198  set of Debian packages used by Debathena for development.
199
200For the most part a package should be listed in the "Depends:" line of
201a metapackage, but in some cases it is appropriate to hedge by using
202"Recommends:", which will cause aptitude to succeed even if the
203package is unavailable.  For example, a package which doesn't exist in
204all Debian/Ubuntu suites or isn't free can be listed under
205"Recommends:" so that our metapackages still work in all environments.
206
207Developers: Index of cluster packages
208-------------------------------------
209
210Some private workstation admins may want specific pieces of cluster
211infrastructure.  With Athena 9.4, they would typically install the
212entire Athena software suite and then turn off features they did not
213want via /etc/athena/rc.conf.  That is not generally possible in
214Athena 10; a debathena-cluster machine acts like a cluster machine in
215all respects.  Instead, the machine owner should install
216debathena-workstation and then install specific packages they might be
217interested in.  At this time, candidate packages include:
218
219  * debathena-dns-config: Installs a caching DNS resolver and
220    configures the machine to use it.
221
222  * debathena-tmp-cleaner: Cleans files in /tmp and /var/tmp which
223    have not been recently accessed.
224
225  * debathena-reactivate: Causes each graphical login to be performed
226    in a separate ephemeral snapshot of the chroot.  This package
227    requires the root filesystem to be an LVM logical volume inside a
228    volume group with 21GB of free space for the snapshots; if that is
229    not the case, it will break graphical logins.  Install with care.
230
231  * debathena-auto-update: Periodically updates the machine's software
232    and reboots (when no one is logged in) if necessary.
233
234Cluster packages which are generally not of interest to private
235machine owners include:
236
237  * debathena-cluster-login-config: Configures a variety of system
238    services to implement the cluster login policy (no tty logins, no
239    user switching, screensaver logout button after 20 minutes, etc.).
240
241  * debathena-clusterinfo: Looks up the machine's cluster information
242    in Hesiod and caches it on local disk.
243
244  * debathena-larvnet: Reports to the central larvnet server whether
245    anyone has a graphical login to the machine.
246
247  * debathena-syslog-config: Configures sysklogd so that some syslog
248    messages are forwarded to a central logging host.
249
250Release engineer: Bootstrapping the project infrastructure
251----------------------------------------------------------
252
253  1. Create the package repository (detailed instructions on this
254     pending).  Set the DEBATHENA_APT environment variable to point to
255     the package repository.  Put a copy of the debathena "scripts"
256     subdir in your path.
257
258  2. Create the build area.
259
260  3. Run 'dareprepro export' to set up the repository.  Build each
261     equivs package under meta/ using 'equivs-build --full *.equivs'
262     and upload each with 'daupload-equivs-proposed *.changes',
263     followed by 'damove -proposed "" PACKAGE' (replacing PACKAGE with
264     the name of each equivs package you are moving).
265
266  4. Set up the build server.  The basic structure of the apt
267     repository must work for make-chroot to succeed, so this must
268     happen after step 3.
269
270  5. For each normal Debian package in dependency order, cd into its
271     directory in the build area and run "da sbuildhack *.dsc" and
272     "daupload-release *_source.changes".  If the package contains
273     only an "Architecture: all" binary package, pass the -A option to
274     both commands.
275
276     The all-packages script can generate an approximation of the
277     package list in dependency order, but it doesn't work right yet,
278     and ideally it would be possible to do several builds in parallel
279     using a Makefile like the one in scripts/build-server/build-all.
280     Improvements to this machinery are pending.
281
282  6. For each package under third, run "da ./debathenify-PKG source
283     binary upload".  Any created directories under third/openafs/meta
284     should be chmodded 777 to work around a perl/AFS permissions
285     issue with File::Temp; if this is not done, OpenAFS metapackage
286     builds will fail for other users.
287
288Release engineer: Updating the apt repository
289---------------------------------------------
290
291  1. ssh to the build server as the builder account and change to the
292     canonical build directory.
293
294  2. Run "gen-packages" to update the package list.  (Or "gen-packages
295     -c" if you know the AFS checkout of the source tree is up to date;
296     it should update every half hour.)
297
298  3. Run "ood-packages" to produce a list of out-of-date packages.
299
300  4. For each out of date package, run "dasource PKG".  Then change to
301     the package directory and run "da sbuildhack *.dsc" and
302     "daupload-release *_source.changes".  If the package contains
303     only an "Architecture: all" binary package, pass the -A option to
304     both commands.
305
306  5. svn update the meta directory.  If there are new subdirectories,
307     chmod them 777 to work around a perl/AFS permissions issue with
308     File::Temp.  For each updated subdir, change to it, run
309     "equivs-build --full *.equivs", and then "daequivsupload
310     FILENAME.changes" on the produced changes file.
311
312  6. svn update the third directory.  You can let autodebathenify
313     handle the updated scripts, or you can touch
314     ~/autodebathenify.suppress, make sure it's not running, and run
315     "da ./debathenify-PKG source binary upload" in each updated
316     directory.
317
318Sometimes you may have to mix up the order of the above steps in order
319to handle build dependencies.
320
321Release engineer: Setting up a build server
322-------------------------------------------
323
324  1. Install either Debian testing (or stable, if it's new enough, but
325     see the note about sbuild below) or the latest Ubuntu release.
326     The build server must be installed with free space in an LVM
327     volume group.  The build chroots consume 4GB each.
328
329  2. Install debathena-login as per the the instructions in
330     http://debathena.mit.edu/install.
331
332  3. apt-key add
333     /afs/sipb.mit.edu/project/debathena/apt/debathena-archive-keyring.asc
334
335  4. Install the packages listed in scripts/build-server/packages
336     (using "aptitude install").
337
338     Note that currently the build system uses the latest version of
339     sbuild, so if you have not installed Debian testing, you'll need
340     to set up apt pinning to grab sbuild from there. See the
341     apt_preferences man page. You'll also need to grab schroot from
342     Debian experimental.
343
344  5. Install the aufs-modules-2.6-amd64 package.
345
346  6. Append to /etc/approx/approx.conf the contents of
347      scripts/build-server/approx.conf.tail.
348     Run: /etc/init.d/approx restart
349
350  7. Apply scripts/build-server/mount-defaults.patch and
351     scripts/build-server/pam-schroot.patch.
352
353  8. For each supported DIST (see scripts/debian-versions.sh) run:
354
355       VG=/dev/blah scripts/build-server/make-chroot DIST i386
356       VG=/dev/blah scripts/build-server/make-chroot DIST amd64
357
358     substituting the name of the volume group for blah.
359
360     Example: VG=/dev/dink scripts/build-server/make-chroot intrepid i386
361
362  9. Create a local account for builder with:
363
364       adduser --uid 1047 --disabled-password builder
365
366     Make the home directory mode 700.  Install a
367     daemon/linux-build-10.mit.edu keytab in the home directory as
368     "keytab".  Install a copy of the secret repository-signing key
369     (debathena@mit.edu) in the home directory's keyring with
370     something like:
371
372       kinit builder
373       gpg --export-secret-keys debathena@mit.edu | \
374         ssh -l builder machinename gpg --import
375
376     Create a file named .sbuildrc in builder's homedir containing:
377
378       $mailto = undef;
379       $log_dir = '/tmp/sbuild-logs';
380       $maintainer_name = 'Debathena Project <debathena@mit.edu>';
381       $force_orig_source = 1;
382       $sbuild_mode = "user";
383       1;
384
385     Create a file named .ssh/config in builder's homedir containing:
386
387       Host svn.mit.edu
388         User debuildsvn
389
390     Add builder to the sbuild group in /etc/group.
391
392     Copy scripts/build-server/autodebathenify to builder's homedir.
393     Create a file named autodebathenify.config in builder's homedir
394     containing:
395
396       error_addr=debathena-root@mit.edu
397       scripts_dir=/mit/debathena/bin
398       build_dir=/mit/debathena/packages/third
399       packages="lprng openafs"
400       export DEBATHENA_APT=/mit/debathena/apt
401
402     Copy scripts/build-server/autodebathenify.cron to builder's
403     homedir and install it with "crontab autodebathenify.cron".
404
405     Copy scripts/build-server/autolivebuilder to builder's
406     homedir. Create a file called autolivebuilder.config in builder's
407     homedir containing:
408
409       error_addr=debathena-root@mit.edu
410       release_version='9.04'
411       release='jaunty'
412       arch='i386'
413       mirror='mirrors.mit.edu'
414       gpg_opts=("-u" "0D8A9E8F")
415       live_dir=/net/sipb-isilon-sc.mit.edu/ifs/mirror/sipb/debathena/livecd
416
417     Create a file named .devscripts in builder's homedir containing:
418
419       DEBUILD_DPKG_BUILDPACKAGE_OPTS="-sa -us -uc -i -I.svn"
420
421     In builder's homedir, append to .bashrc:
422
423       add debathena
424       export PATH=$PATH:~/bin
425       export DEBATHENA_APT=/mit/debathena/apt
426
427
428Release engineer: Removing a build chroot on the build server
429-------------------------------------------------------------
430
431  1. Run VG=/dev/blah scripts/clean-schroots as root to make sure that
432     the build chroot is not mounted, substituting the name of the
433     volume group for blah.
434
435  2. Edit /etc/schroot/schroot.conf and delete the section
436     corresponding to the chroot.
437
438  3. Run lvchange -an blah/chrootname
439     substituting the name of the volume group for blah and the chroot
440     name for chroot.  Example: lvchange -an dink/gutsy-i386-sbuild
441
442  4. Run lvremove blah/chrootname
443
444Release engineer: Removing a dist from the apt repository
445---------------------------------------------------------
446
447  1. Inside the apt repository, edit conf/distributions and remove the
448     distribution section.
449
450  2. Run dareprepro --delete clearvanished
451
452Release engineer: Setting up a canonical build area
453---------------------------------------------------
454
455  1. Create an empty directory and cd into it.  The canonical build
456     area lives in /mit/debathena/packages/.
457
458  2. Run gen-packages to create the table of normal Debian packages.
459
460  3. Run dasource to create subdirs and source packages for each
461     normal Debian package.
462
463  4. Create checkouts of the meta and third directories:
464
465     svn co svn+ssh://svn.mit.edu/athena/trunk/debathena/meta
466     svn co svn+ssh://svn.mit.edu/athena/trunk/debathena/third
467     chmod 777 meta/*
468
469     A couple of subdirectories of debathena/meta are normal Debian
470     packages, so this will create redundant copies of those.  Ignore
471     them; they won't be used.
472
473Release engineer: Adding a new suite
474------------------------------------
475
476This process is rarely performed and the infrastructure for it is
477imperfect.  Substitute the name of the new suite for "newdist" in all
478steps below.
479
480  1. Make sure the apt repository is up to date with respect to the
481     source tree for the existing dists. Check
482     debathena.mit.edu/package-list/development and
483     debathena.mit.edu/package-list/proposed. More importantly, make
484     sure there is nothing in svn that isn't built. (An easy way to do
485     this is running the check-unbuilt-packages script.)
486
487  2. Add the new dist to scripts/debian-versions.sh.  (It is not
488     necessary to add the new dist to codes at this point, but it must
489     be present in the gettag conditional.)
490
491  3. Create the new distribution in the apt repository's configuration
492     file.  Create the skeleton of the dist by running "dareprepro
493     export".
494
495  4. On the build server, create a chroot for the new distribution as
496     documented above.  This may require downloading and installing a
497     more recent version of the debootstrap package from the
498     -backports dist corresponding to the build server's OS.
499
500  5. Set the DEBATHENA_BUILD_AREA environment variable to point to the
501     build area.
502
503  6. Fire up screen.
504
505  7. mkdir $DEBATHENA_BUILD_AREA/stamps.newdist.
506
507  8. cd into a checkout of debathena/scripts/build-server/build-all.
508
509  9. Edit Makefile (and check in the edit) so that suite is the new
510     distribution and psuite is the previously most recent Debian or
511     Ubuntu distribution.
512
513  10. Run "make deps.mk".
514
515  11. Run "make -k all STAMPS=$DEBATHENA_BUILD_AREA/stamps.newdist".
516      You can watch the builds happen in the other windows of the
517      screen session.  It's possible to do several builds at once with
518      make -j N.
519
520  12. debathenify packages will fail out; they must be built by hand.
521      When the build fails on one, cd into third/packagename in the
522      build area and run "./debathenify newdist-amd64 -A source binary
523      upload" and "./debathenify newdist-i386 binary upload".  Then
524      touch $DEBATHENA_BUILD_AREA/stamps.newdist/packagename.done" and
525      restart the build.
526
527      The newly created third/openafs/meta directories should be
528      chmodded 777 to work around a perl/AFS permissions issue.
529
530Release engineer: Maintaining autodebathenify
531---------------------------------------------
532
533autodebathenify is a cron job which runs on the builder account and
534keeps the OpenAFS modules and modified packages under debathena/third
535up to date when there are upstream changes.  It relies on the
536following:
537
538  * builder@ATHENA.MIT.EDU has access to a local account named
539    debuildsvn on svn.mit.edu with read access to the athena
540    repository.
541
542  * builder@ATHENA.MIT.EDU has write access to the canonical build
543    area and apt repository in the dev cell.
544
545  * The builder account on the build machine has a copy of the signing
546    private key for the apt repository.
547
548The cron job will silently exit if it detects that it is already
549running.  If it fails for any other reason, it will send mail to
550source-wash@mit.edu and touch a file named autodebathenify.suppress
551which will prevent it from running again until manual intervention.
552It is important to get autodebathenify running smoothly or the apt
553repository will become out of date with respect to third-party
554packages, which in turn will compromise the user experience.
555
556Even if you don't notice failure mail, it's good to check on the
557status of autodebathenify from time to time.  To do this, get tickets
558as builder and ssh to linux-build-10 as builder.  Run "ls -l logs" to
559see what's been going on.  A logfile with size around 48K indicates
560that autodebathenify ran normally but didn't find any new work to do.
561A logfile with size around 60 means that autodebathenify.suppress
562exists and the cron job has stopped running.  A longer logfile
563indicates that autodebathenify attempted to build and upload a
564package.
565
566Possible failure cases include:
567
568  * One of the upstream apt repositories timed out during an "apt-get
569    update" of an sbuild chroot.  If this happens, you can just remove
570    the autodebathenify.suppress file to get the cron job to start
571    running again.
572
573  * autodebathenify tried to build an upstream version which already
574    exists.  If this happens, the build will fail at upload time.  If
575    this happens, there is a bug in the part of the debathenify-*
576    script which checks whether our apt repository already has the
577    expected version of the built package.
578
579  * Debian and Ubuntu have .orig files with the same name but
580    different contexts for a particular third-party package.  This has
581    only happened once, with bash (which is currently not configured
582    into autodebathenify for that reason).  We do not have a general
583    mechanism for resolving such issues at this time.
584
585  * The new upstream version of the package is sufficiently different
586    to cause our modifications to fail to apply, and our debathenify-*
587    script needs to be adapted appropriately.
588
589The cron job is run out of builder's home directory, but its canonical
590source location is in debathena/scripts/build-server in this
591repository, should it need to be modified.
592
593After several months of operation, LVM snapshots on the build server
594will start to become slower and slower, as /etc/lvm/cache/.cache grows
595in size.  Turning off the cache entirely seems to cause schroot to
596occasionally fail; the current solution is to rm /etc/lvm/cache/.cache
597every so often.
598
599Release engineer: apt repository HTTP server setup
600--------------------------------------------------
601
602The apt repository server (debathena.mit.edu) is hosted on
603scripts.mit.edu out of the debathena locker. To replicate it on a
604dedicated server, install Apache and AFS (e.g., debathena-standard
605and apache2), and configure the document root to be
606  /mit/debathena/web_scripts
607
608There's also a crontab in /mit/debathena/cron_scripts that updates
609the package lists on the website. It expects to be able to edit the
610web_scripts directory; on scripts, it does so with daemon.scripts
611permissions.
612
613Trac (http://debathena.mit.edu/trac/) expects to use sql.mit.edu
614for its database.
Note: See TracBrowser for help on using the repository browser.