source: trunk/debathena/NOTES @ 23046

Revision 23046, 17.1 KB checked in by ghudson, 16 years ago (diff)
Document the need for a .sbuildrc for developers. debathenify scripts no longer require schroots to be upgraded. Document the local builder account on build server, including autodebathenify setup.
Line 
1This hierarchy contains Debian/Ubuntu-specific materials, also known
2as "Debathena".  The contents are:
3
4* debathena - Debathena-specific software packages such as PAM and NSS
5  modules.
6
7* config - Packages for configuring native system software in a manner
8  appropriate for Athena.
9
10* meta - Packages which contain nothing but dependencies on other
11  packages and serve as an installation convenience.
12
13* scripts - Build scripts and supporting materials.
14
15Debathena is a SIPB project, and its infrastructure and procedures
16will need to be adapted for Athena 10.  For the moment this file will
17document the Debathena procedures as they are, not as they will be.
18The current procedures do not even use this svn repository yet.
19
20Debian software used by Debathena:
21
22  * schroot - Used to manage build chroot environments for each
23    Debian/Ubuntu version.  We use the lvm-snapshot schroot type,
24    which allows rapid construction of ephemeral copies of template
25    "source" chroots, so that every binary package build is done in a
26    clean environment.
27
28  * debuild - Used to create Debian source packages from package
29    source directories.
30
31  * sbuild - Used to build binary packages from source packages inside
32    schroot environments.
33
34  * equivs - Used to create packages which only contain dependency
35    information.  Somewhat of a dirty hack, since it doesn't keep
36    proper changelogs, but it reduces overhead.
37
38  * CDBS (Common Debian Build System) - Referenced by debian/rules
39    files in packages.  Contains standard build rules to cut down on
40    per-package boilerplate.
41
42  * reprepro - Used to upload packages into the apt repositories.
43
44  * approx - Used to create a local cache of Debian packages on the
45    build server.  This cache is referenced by the build chroots for
46    improved performance.
47
48The remainder of this file documents procedures useful to Athena 10
49developers and the release engineer.
50
51Developers: Preferences setup
52-----------------------------
53
54You will probably want a $HOME/.devscripts file containing the
55following:
56
57DEBUILD_DPKG_BUILDPACKAGE_OPTS="-sa -us -uc -i -I.svn"
58
59This will save you from having to specify those options every time you
60run debuild.  Athena 10 scripts do not assume the above preferences,
61but the instructions in this file do.  The options mean:
62
63  * Look for original source as a tarfile or create one.
64  * Do not sign the source package.
65  * Do not sign the changes file.
66  * Ignore common version control metadata files when creating diffs.
67  * Ignore .svn paths when creating tarballs.
68
69You will also want a $HOME/.sbuildrc file containing the following:
70
71  $nolog = 1;
72  $mailto = 'yourusername';
73  $log_dir = '/tmp/sbuild-logs';
74  $maintainer_name = 'Debian-Athena Project <debathena@mit.edu>';
75  $force_orig_source = 1;
76  $sbuild_mode = "user";
77  1;
78
79You should also set the environment variable DEBATHENA_APT to
80"/afs/dev.mit.edu/system/athena10/apt".
81
82Developers: Preparing a change
83------------------------------
84
85To prepare a change to a regular package (a source tree containing a
86debian/ subdir), make the edits in a checkout and record a changelog
87entry.  You can either edit debian/changelog using emacs changelog
88mode (C-c C-v to add a new version entry, C-c C-a to add a change
89entry, C-c C-f to finalize the entry) or you can run "dadch".
90
91When creating a new version entry, bump the upstream version number
92(to 10.0.0 if it was not already that high) if you are changing the
93main package source.  Otherwise, just bump the Debian version
94component (change 0debathena1 to 0debathena2, for instance).
95
96Developers: Building a package for test purposes on one platform
97----------------------------------------------------------------
98
99After you have prepared a change, you will want to test that it builds
100and perhaps that it works before committing it.  First, if it is an
101Athena source directory using autoconf, run "daconfiscate" to set up
102the autoconf boilerplate which we don't check in.  Second, run
103"daorig" to copy or create an orig tarball in the parent directory if
104necessary.  Third, run "debuild".  The resulting package will be
105placed in the parent directory.
106
107In order to test if the package works, you can install it with "dpkg
108-i filename.deb".
109
110Developers: Building a package for test purposes on all platforms
111-----------------------------------------------------------------
112
113If the package you are working on interacts with the native OS in ways
114that might vary from platform to platform, you may want to do a test
115build for all platforms.  You will need to do this on
116linux-build-10.mit.edu or another machine which has been set up with
117build schroots.
118
119As above, run daconfiscate (if necessary) and then daorig.  Then run
120"debuild -S" to create a source package.  Now cd into the parent
121directory and identify the .dsc file created by debuild -S; it will
122have a name like debathena-just_9.4.0-0debathena2.dsc.  Run "da
123sbuildhack filename.dsc" to perform the package builds.  Each build
124will take place inside an ephemeral chroot based on a snapshot of a
125template for a particular Debian or Ubuntu version.  If a build fails
126and it's not obvious from the build log why, you may need to create
127your own ephemeral chroot session with a command like "schroot -c
128gutsy-amd64-sbuild /bin/sh" and then run debuild from within the
129package sources.
130
131If the build is successful, it will create a set of packages with
132names like debathena-just_9.4.0-0debathena2~ubuntu6.06_amd64.deb.
133
134Developers: Building an equivs package
135--------------------------------------
136
137Most of the packages under debathena/meta are faked up using equivs.
138To build one, just run:
139
140  equivs-build --full filename.equivs
141
142These equivs files make reference to ../common, so you must have a
143checkout of debathena/meta/common alongside the particular
144meta-package you are building.
145
146Developers: The meaning of metapackages
147---------------------------------------
148
149If you are adding a new package to the repository, you will probably
150at some point want to add it to one of the metapackages so that it
151doesn't have to be installed by hand.  Here are some descriptions
152which may help identify which metapackage is best:
153
154* locker: Provides access to Athena locker software--AFS and
155  automounter configuration, locker-related utilities, etc.
156
157* clients: Provides clients (either locally-written, like athinfo and
158  Discuss, or configurations) for Athena services, as well as
159  Athena-specific utility programs like "jot".  Configurations for
160  graphical client software are generally in the workstation package
161  instead, in order to make this package less intrusive.
162
163* standard: Implies locker and clients.  Also provides Athena shell
164  customizations and dotfiles.
165
166* login: Implies standard.  Configurations to merge the MIT user
167  namespace into the local machine namespace for the purpose of user
168  lookups and authentication.
169
170* workstation: Implies login.  Configurations for the graphical login
171  system and graphical client software intended to provide a standard
172  X login experience using Athena home directories.  Still in
173  development.
174
175* cluster-software: Provides a set of Debian packages common to
176  cluster machines.  The resulting software set is rather large, and
177  thus may not be desirable to all workstation configurations.  Only
178  stock Debian packages belong in this metapackage; do not add
179  other Debathena packages to it.
180
181* cluster: Implies workstation and cluster-software.  Also contains
182  configurations for self-maintenance of machines (unattended updates,
183  cleanups between logins, etc.).  Does not exist yet.
184
185* debian-dev: Intended for developers of the system itself; provides a
186  set of Debian packages used by Debathena for development.
187
188For the most part a package should be listed in the "Depends:" line of
189a metapackage, but in some cases it is appropriate to hedge by using
190"Recommends:", which will cause aptitude to succeed even if the
191package is unavailable.  For example, a package which doesn't exist in
192all Debian/Ubuntu suites or isn't free can be listed under
193"Recommends:" so that our metapackages still work in all environments.
194
195Release engineer: Bootstrapping the project infrastructure
196----------------------------------------------------------
197
198  1. Create the package repository (detailed instructions on this
199     pending).  Set the DEBATHENA_APT environment variable to point to
200     the package repository.  Put a copy of the debathena "scripts"
201     subdir in your path.
202
203  2. Create the build area.
204
205  3. Build each equivs package under meta/ using "equivs-build --full
206     *.equivs" and upload each with "daequivsupload *.changes".  This
207     has the side-effect of creating the basic structure of the
208     package repository.
209
210  4. Set up the build server.  The basic structure of the apt
211     repository must work for make-chroot to succeed, so this must
212     happen after step 3.
213
214  5. For each normal Debian package in dependency order, cd into its
215     directory in the build area and run "da sbuildhack *.dsc" and
216     "daupload-release *_source.changes".  If the package contains
217     only an "Architecture: all" binary package, pass the -A option to
218     both commands.
219
220     The all-packages script can generate an approximation of the
221     package list in dependency order, but it doesn't work right yet,
222     and ideally it would be possible to do several builds in parallel
223     using a Makefile like the one in scripts/build-server/build-all.
224     Improvements to this machinery are pending.
225
226  6. For each package under third, run "da ./debathenify-PKG source
227     binary upload".
228
229Release engineer: Setting up a build server
230-------------------------------------------
231
232  1. The build server must be installed with free space in an LVM
233     volume group.  The build chroots consume 2GB each.  There is a
234     known memory corruption issue with LVM snapshots in the kernel
235     used in Ubuntu Gutsy (which is based on 2.6.22), so use a newer
236     kernel such as the one in Ubuntu Hardy (based on 2.6.24) instead.
237
238  2. Install debathena-standard as per the the instructions in
239     http://debathena.mit.edu/install.
240
241  3. apt-key add /afs/dev.mit.edu/system/athena10/apt/athena10-archive.asc
242
243  4. Install the packages listed in
244     scripts/build-server/packages (using "aptitude install")
245
246  5. Install debathena-login, debathena-ssh-server, and
247     debathena-build-depends (using "aptitude install").
248
249     (Depending on how recently debathena-build-depends was rebuilt,
250     additional packages might need to be installed to satisfy the
251     build-depends of newer packages.  This can be taken care of later
252     when an error occurs building a source package.)
253
254  6. Edit /etc/security/access.conf and add a first line:
255     -:ALL EXCEPT root <developer usernames>:ALL
256
257  7. Edit /etc/pam.d/schroot, comment out "@include common-session",
258     and add:
259
260       # Basic pam_unix session module in place of common-session.
261       session required         pam_unix.so
262
263  8. Edit /etc/group and add the developers to the sbuild group.
264
265  9. Create /etc/passwd entries for each developer with "hesinfo
266     username passwd >> /etc/passwd" and then run pwconv.
267
268     (This is not necessary for the login system on the main root
269     environment, but is for the chroot environments.)
270
271  10. Append to /etc/approx/approx.conf the contents of
272       scripts/build-server/approx.conf.tail.
273      Change the last line from http://debathena.mit.edu/apt to
274       file:///afs/dev.mit.edu/system/athena10/apt
275      Add "$interval 0" above the repository lines (only necessary if
276       the version of approx as reported by "dpkg -l approx" is less
277       than 3.0)
278      Run: /etc/init.d/approx restart
279
280  11. Apply scripts/build-server/mount-defaults.patch.
281
282  12. For each supported DIST (see scripts/debian-versions.sh) run:
283
284        VG=/dev/blah scripts/build-server/make-chroot DIST i386
285        VG=/dev/blah scripts/build-server/make-chroot DIST amd64
286
287      substituting the name of the volume group for blah.  Omit the
288      amd64 line if DIST is sarge.
289
290      Example: VG=/dev/dink scripts/build-server/make-chroot gutsy i386
291
292  13. Create a local account for builder with:
293
294        adduser --uid 1047 --disabled-password builder
295
296      Make the home directory mode 700.  Install a
297      daemon/linux-build-10.mit.edu keytab in the home directory as
298      "keytab".  Install a copy of the secret repository-signing key
299      (athena10@mit.edu) in the home directory's keyring with
300      something like:
301
302        kinit builder
303        gpg --export-secret-keys athena10@mit.edu | \
304          ssh -l builder machinename gpg --import
305
306      Create a file named .sbuildrc in builder's homedir containing:
307
308        $nolog = 1;
309        $mailto = 'source-wash@mit.edu';
310        $log_dir = '/tmp/sbuild-logs';
311        $maintainer_name = 'Debian-Athena Project <debathena@mit.edu>';
312        $force_orig_source = 1;
313        $sbuild_mode = "user";
314        1;
315
316      Add builder to the sbuild group in /etc/group.
317
318      Copy scripts/build-server/autodebathenify to builder's homedir.
319      Create a file named autodebathenify.config in builder's homedir
320      containing:
321
322        error_addr=source-wash@mit.edu
323        scripts_dir=/afs/dev.mit.edu/source/src-svn/debathena/scripts
324        build_dir=/afs/dev.mit.edu/project/release/10/build/third
325        packages="cyrus-sasl2-mit evolution-data-server lprng openafs tcsh"
326        export DEBATHENA_APT=/afs/dev.mit.edu/system/athena10/apt
327
328      Copy scripts/build-server/autodebathenify.cron to builder's
329      homedir and install it with "crontab autodebathenify.cron".
330
331Release engineer: Removing a build chroot on the build server
332-------------------------------------------------------------
333
334  1. Run VG=/dev/blah scripts/clean-schroots as root to make sure that
335     the build chroot is not mounted, substituting the name of the
336     volume group for blah.
337
338  2. Edit /etc/schroot/schroot.conf and delete the section
339     corresponding to the chroot.
340
341  3. Run lvchange -an blah/chrootname
342     substituting the name of the volume group for blah and the chroot
343     name for chroot.  Example: lvchange -an dink/gutsy-i386-sbuild
344
345  4. Run lvremove blah/chrootname
346
347Release engineer: Removing a dist from the apt repository
348---------------------------------------------------------
349
350  1. Inside the apt repository, edit conf/distributions and remove the
351     distribution section.
352
353  2. Run reprepro -Vb $DEBATHENA_APT --delete clearvanished
354
355Release engineer: Setting up a canonical build area
356---------------------------------------------------
357
358  1. Create an empty directory and cd into it.  The canonical build
359     area lives in /afs/dev.mit.edu/project/release/10/build.
360
361  2. Run gen-packages to create the table of normal Debian packages.
362
363  3. Run dasource to create subdirs and source packages for each
364     normal Debian package.
365
366  4. Create checkouts of the meta and third directories:
367
368     svn co svn+ssh://svn.mit.edu/athena/trunk/debathena/meta
369     svn co svn+ssh://svn.mit.edu/athena/trunk/debathena/third
370
371     (A couple of subdirectories of debathena/meta are normal Debian
372     packages, so this will create redundant copies of those.  Ignore
373     them; they won't be used.)
374
375Release engineer: Adding a new suite
376------------------------------------
377
378This process is rarely performed and the infrastructure for it is
379imperfect.  Substitute the name of the new suite for "newdist" in all
380steps below.
381
382  1. Make sure the apt repository is up to date with respect to the
383     source tree for the existing dists.
384
385  2. Add the new dist to scripts/debian-versions.sh.  (It is not
386     necessary to add the new dist to codes at this point, but it must
387     be present in the gettag conditional.)
388
389  3. Create the new distribution in the apt repository's configuration
390     file.  Create the skeleton of the dist by installing at least one
391     equivs package from meta/ with "reprepro -Vb $DEBATHENA_APT
392     include newdistname file.changes".
393
394  4. On the build server, create a chroot for the new distribution as
395     documented above.  This may require downloading and installing a
396     more recent version of the debootstrap package from the
397     -backports dist corresponding to the build server's OS.
398
399  5. Set the DEBATHENA_BUILD_AREA environment variable to point to the
400     build area.
401
402  6. Fire up screen.
403
404  7. mkdir $DEBATHENA_BUILD_AREA/stamps.newdist.
405
406  8. cd into a checkout of debathena/scripts/build-server/build-all.
407
408  9. Edit Makefile (and check in the edit) so that suite is the new
409     distribution and psuite is the previously most recent Debian or
410     Ubuntu distribution.
411
412  10. Run "make deps.mk".
413
414  11. Run "make -k all STAMPS=$DEBATHENA_BUILD_AREA/stamps.newdist".
415      You can watch the builds happen in the other windows of the
416      screen session.  It's possible to do several builds at once with
417      make -j N.
418
419  12. debathenify packages will fail out; they must be built by hand.
420      When the build fails on one, cd into third/packagename in the
421      build area and run "./debathenify newdist-amd64 -A source binary
422      upload" and "./debathenify newdist-i386 binary upload".  Then
423      touch $DEBATHENA_BUILD_AREA/stamps.newdist/packagename.done" and
424      restart the build.
425
426  13. Go into third/openafs in the build area and build AFS modules
427      for the new suite's kernels.  (Instructions pending.)
Note: See TracBrowser for help on using the repository browser.