source: trunk/debathena/NOTES @ 23018

Revision 23018, 15.5 KB checked in by ghudson, 16 years ago (diff)
In NOTES, add a section documenting the metapackages.
Line 
1This hierarchy contains Debian/Ubuntu-specific materials, also known
2as "Debathena".  The contents are:
3
4* debathena - Debathena-specific software packages such as PAM and NSS
5  modules.
6
7* config - Packages for configuring native system software in a manner
8  appropriate for Athena.
9
10* meta - Packages which contain nothing but dependencies on other
11  packages and serve as an installation convenience.
12
13* scripts - Build scripts and supporting materials.
14
15Debathena is a SIPB project, and its infrastructure and procedures
16will need to be adapted for Athena 10.  For the moment this file will
17document the Debathena procedures as they are, not as they will be.
18The current procedures do not even use this svn repository yet.
19
20Debian software used by Debathena:
21
22  * schroot - Used to manage build chroot environments for each
23    Debian/Ubuntu version.  We use the lvm-snapshot schroot type,
24    which allows rapid construction of ephemeral copies of template
25    "source" chroots, so that every binary package build is done in a
26    clean environment.
27
28  * debuild - Used to create Debian source packages from package
29    source directories.
30
31  * sbuild - Used to build binary packages from source packages inside
32    schroot environments.
33
34  * equivs - Used to create packages which only contain dependency
35    information.  Somewhat of a dirty hack, since it doesn't keep
36    proper changelogs, but it reduces overhead.
37
38  * CDBS (Common Debian Build System) - Referenced by debian/rules
39    files in packages.  Contains standard build rules to cut down on
40    per-package boilerplate.
41
42  * reprepro - Used to upload packages into the apt repositories.
43
44  * approx - Used to create a local cache of Debian packages on the
45    build server.  This cache is referenced by the build chroots for
46    improved performance.
47
48The remainder of this file documents procedures useful to Athena 10
49developers and the release engineer.
50
51Developers: Preferences setup
52-----------------------------
53
54You will probably want a $HOME/.devscripts file containing the
55following:
56
57DEBUILD_DPKG_BUILDPACKAGE_OPTS="-sa -us -uc -i -I.svn"
58
59This will save you from having to specify those options every time you
60run debuild.  Athena 10 scripts do not assume the above preferences,
61but the instructions in this file do.  The options mean:
62
63  * Look for original source as a tarfile or create one.
64  * Do not sign the source package.
65  * Do not sign the changes file.
66  * Ignore common version control metadata files when creating diffs.
67  * Ignore .svn paths when creating tarballs.
68
69You should also set the environment variable DEBATHENA_APT to
70"/afs/dev.mit.edu/system/athena10/apt".
71
72Developers: Preparing a change
73------------------------------
74
75To prepare a change to a regular package (a source tree containing a
76debian/ subdir), make the edits in a checkout and record a changelog
77entry.  You can either edit debian/changelog using emacs changelog
78mode (C-c C-v to add a new version entry, C-c C-a to add a change
79entry, C-c C-f to finalize the entry) or you can run "dadch".
80
81When creating a new version entry, bump the upstream version number
82(to 10.0.0 if it was not already that high) if you are changing the
83main package source.  Otherwise, just bump the Debian version
84component (change 0debathena1 to 0debathena2, for instance).
85
86Developers: Building a package for test purposes on one platform
87----------------------------------------------------------------
88
89After you have prepared a change, you will want to test that it builds
90and perhaps that it works before committing it.  First, if it is an
91Athena source directory using autoconf, run "daconfiscate" to set up
92the autoconf boilerplate which we don't check in.  Second, run
93"daorig" to copy or create an orig tarball in the parent directory if
94necessary.  Third, run "debuild".  The resulting package will be
95placed in the parent directory.
96
97In order to test if the package works, you can install it with "dpkg
98-i filename.deb".
99
100Developers: Building a package for test purposes on all platforms
101-----------------------------------------------------------------
102
103If the package you are working on interacts with the native OS in ways
104that might vary from platform to platform, you may want to do a test
105build for all platforms.  You will need to do this on
106linux-build-10.mit.edu or another machine which has been set up with
107build schroots.
108
109As above, run daconfiscate (if necessary) and then daorig.  Then run
110"debuild -S" to create a source package.  Now cd into the parent
111directory and identify the .dsc file created by debuild -S; it will
112have a name like debathena-just_9.4.0-0debathena2.dsc.  Run "da
113sbuildhack filename.dsc" to perform the package builds.  Each build
114will take place inside an ephemeral chroot based on a snapshot of a
115template for a particular Debian or Ubuntu version.  If a build fails
116and it's not obvious from the build log why, you may need to create
117your own ephemeral chroot session with a command like "schroot -c
118gutsy-amd64-sbuild /bin/sh" and then run debuild from within the
119package sources.
120
121If the build is successful, it will create a set of packages with
122names like debathena-just_9.4.0-0debathena2~ubuntu6.06_amd64.deb.
123
124Developers: Building an equivs package
125--------------------------------------
126
127Most of the packages under debathena/meta are faked up using equivs.
128To build one, just run:
129
130  equivs-build --full filename.equivs
131
132These equivs files make reference to ../common, so you must have a
133checkout of debathena/meta/common alongside the particular
134meta-package you are building.
135
136Developers: The meaning of metapackages
137---------------------------------------
138
139If you are adding a new package to the repository, you will probably
140at some point want to add it to one of the metapackages so that it
141doesn't have to be installed by hand.  Here are some descriptions
142which may help identify which metapackage is best:
143
144* locker: Provides access to Athena locker software--AFS and
145  automounter configuration, locker-related utilities, etc.
146
147* clients: Provides clients (either locally-written, like athinfo and
148  Discuss, or configurations) for Athena services, as well as
149  Athena-specific utility programs like "jot".  Configurations for
150  graphical client software are generally in the workstation package
151  instead, in order to make this package less intrusive.
152
153* standard: Implies locker and clients.  Also provides Athena shell
154  customizations and dotfiles.
155
156* login: Implies standard.  Configurations to merge the MIT user
157  namespace into the local machine namespace for the purpose of user
158  lookups and authentication.
159
160* workstation: Implies login.  Configurations for the graphical login
161  system and graphical client software intended to provide a standard
162  X login experience using Athena home directories.  Still in
163  development.
164
165* cluster-software: Provides a set of Debian packages common to
166  cluster machines.  The resulting software set is rather large, and
167  thus may not be desirable to all workstation configurations.  Only
168  stock Debian packages belong in this metapackage; do not add
169  other Debathena packages to it.
170
171* cluster: Implies workstation and cluster-software.  Also contains
172  configurations for self-maintenance of machines (unattended updates,
173  cleanups between logins, etc.).  Does not exist yet.
174
175* debian-dev: Intended for developers of the system itself; provides a
176  set of Debian packages used by Debathena for development.
177
178For the most part a package should be listed in the "Depends:" line of
179a metapackage, but in some cases it is appropriate to hedge by using
180"Recommends:", which will cause aptitude to succeed even if the
181package is unavailable.  For example, a package which doesn't exist in
182all Debian/Ubuntu suites or isn't free can be listed under
183"Recommends:" so that our metapackages still work in all environments.
184
185Release engineer: Bootstrapping the project infrastructure
186----------------------------------------------------------
187
188  1. Create the package repository (detailed instructions on this
189     pending).  Set the DEBATHENA_APT environment variable to point to
190     the package repository.  Put a copy of the debathena "scripts"
191     subdir in your path.
192
193  2. Create the build area.
194
195  3. Build each equivs package under meta/ using "equivs-build --full
196     *.equivs" and upload each with "daequivsupload *.changes".  This
197     has the side-effect of creating the basic structure of the
198     package repository.
199
200  4. Set up the build server.  The basic structure of the apt
201     repository must work for make-chroot to succeed, so this must
202     happen after step 3.
203
204  5. For each normal Debian package in dependency order, cd into its
205     directory in the build area and run "da sbuildhack *.dsc" and
206     "daupload-release *_source.changes".  If the package contains
207     only an "Architecture: all" binary package, pass the -A option to
208     both commands.
209
210     The all-packages script can generate an approximation of the
211     package list in dependency order, but it doesn't work right yet,
212     and ideally it would be possible to do several builds in parallel
213     using a Makefile like the one in scripts/build-server/build-all.
214     Improvements to this machinery are pending.
215
216  6. For each package under third, run "da ./debathenify-PKG source
217     binary upload".  This infrastructure depends on the chroots being
218     up to date, so run "all-schroots upgrade-schroot" beforehand if
219     substantial time has passed since they were created.
220
221Release engineer: Setting up a build server
222-------------------------------------------
223
224  1. The build server must be installed with free space in an LVM
225     volume group.  The build chroots consume 2GB each.  There is a
226     known memory corruption issue with LVM snapshots in the kernel
227     used in Ubuntu Gutsy (which is based on 2.6.22), so use a newer
228     kernel such as the one in Ubuntu Hardy (based on 2.6.24) instead.
229
230  2. Install debathena-standard as per the the instructions in
231     http://debathena.mit.edu/install.
232
233  3. apt-key add /afs/dev.mit.edu/system/athena10/apt/athena10-archive.asc
234
235  4. Install the packages listed in
236     scripts/build-server/packages (using "aptitude install")
237
238  5. Install debathena-login, debathena-ssh-server, and
239     debathena-build-depends (using "aptitude install").
240
241     (Depending on how recently debathena-build-depends was rebuilt,
242     additional packages might need to be installed to satisfy the
243     build-depends of newer packages.  This can be taken care of later
244     when an error occurs building a source package.)
245
246  6. Edit /etc/security/access.conf and add a first line:
247     -:ALL EXCEPT root <developer usernames>:ALL
248
249  7. Edit /etc/pam.d/schroot, comment out "@include common-session",
250     and add:
251
252       # Basic pam_unix session module in place of common-session.
253       session required         pam_unix.so
254
255  8. Edit /etc/group and add the developers to the sbuild group.
256
257  9. Create /etc/passwd entries for each developer with "hesinfo
258     username passwd >> /etc/passwd" and then run pwconv.
259
260     (This is not necessary for the login system on the main root
261     environment, but is for the chroot environments.)
262
263  10. Append to /etc/approx/approx.conf the contents of
264       scripts/build-server/approx.conf.tail.
265      Change the last line from http://debathena.mit.edu/apt to
266       file:///afs/dev.mit.edu/system/athena10/apt
267      Add "$interval 0" above the repository lines (only necessary if
268       the version of approx as reported by "dpkg -l approx" is less
269       than 3.0)
270      Run: /etc/init.d/approx restart
271
272  11. Apply scripts/build-server/mount-defaults.patch.
273
274  12. For each supported DIST (see scripts/debian-versions.sh) run:
275
276        VG=/dev/blah scripts/build-server/make-chroot DIST i386
277        VG=/dev/blah scripts/build-server/make-chroot DIST amd64
278
279      substituting the name of the volume group for blah.  Omit the
280      amd64 line if DIST is sarge.
281
282      Example: VG=/dev/dink scripts/build-server/make-chroot gutsy i386
283
284Release engineer: Removing a build chroot on the build server
285-------------------------------------------------------------
286
287  1. Run VG=/dev/blah scripts/clean-schroots as root to make sure that
288     the build chroot is not mounted, substituting the name of the
289     volume group for blah.
290
291  2. Edit /etc/schroot/schroot.conf and delete the section
292     corresponding to the chroot.
293
294  3. Run lvchange -an blah/chrootname
295     substituting the name of the volume group for blah and the chroot
296     name for chroot.  Example: lvchange -an dink/gutsy-i386-sbuild
297
298  4. Run lvremove blah/chrootname
299
300Release engineer: Removing a dist from the apt repository
301---------------------------------------------------------
302
303  1. Inside the apt repository, edit conf/distributions and remove the
304     distribution section.
305
306  2. Run reprepro -Vb $DEBATHENA_APT --delete clearvanished
307
308Release engineer: Setting up a canonical build area
309---------------------------------------------------
310
311  1. Create an empty directory and cd into it.  The canonical build
312     area lives in /afs/dev.mit.edu/project/release/10/build.
313
314  2. Run gen-packages to create the table of normal Debian packages.
315
316  3. Run dasource to create subdirs and source packages for each
317     normal Debian package.
318
319  4. Create checkouts of the meta and third directories:
320
321     svn co svn+ssh://svn.mit.edu/athena/trunk/debathena/meta
322     svn co svn+ssh://svn.mit.edu/athena/trunk/debathena/third
323
324     (A couple of subdirectories of debathena/meta are normal Debian
325     packages, so this will create redundant copies of those.  Ignore
326     them; they won't be used.)
327
328Release engineer: Adding a new suite
329------------------------------------
330
331This process is rarely performed and the infrastructure for it is
332imperfect.  Substitute the name of the new suite for "newdist" in all
333steps below.
334
335  1. Make sure the apt repository is up to date with respect to the
336     source tree for the existing dists.
337
338  2. Add the new dist to scripts/debian-versions.sh.  (It is not
339     necessary to add the new dist to codes at this point, but it must
340     be present in the gettag conditional.)
341
342  3. Create the new distribution in the apt repository's configuration
343     file.  Create the skeleton of the dist by installing at least one
344     equivs package from meta/ with "reprepro -Vb $DEBATHENA_APT
345     include newdistname file.changes".
346
347  4. On the build server, create a chroot for the new distribution as
348     documented above.  This may require downloading and installing a
349     more recent version of the debootstrap package from the
350     -backports dist corresponding to the build server's OS.
351
352  5. Set the DEBATHENA_BUILD_AREA environment variable to point to the
353     build area.
354
355  6. Fire up screen.
356
357  7. mkdir $DEBATHENA_BUILD_AREA/stamps.newdist.
358
359  8. cd into a checkout of debathena/scripts/build-server/build-all.
360
361  9. Edit Makefile (and check in the edit) so that suite is the new
362     distribution and psuite is the previously most recent Debian or
363     Ubuntu distribution.
364
365  10. Run "make deps.mk".
366
367  11. Run "make -k all STAMPS=$DEBATHENA_BUILD_AREA/stamps.newdist".
368      You can watch the builds happen in the other windows of the
369      screen session.  It's possible to do several builds at once with
370      make -j N.
371
372  12. debathenify packages will fail out; they must be built by hand.
373      When the build fails on one, cd into third/packagename in the
374      build area and run "./debathenify newdist-amd64 -A source binary
375      upload" and "./debathenify newdist-i386 binary upload".  Then
376      touch $DEBATHENA_BUILD_AREA/stamps.newdist/packagename.done" and
377      restart the build.
378
379  13. Go into third/openafs in the build area and build AFS modules
380      for the new suite's kernels.  (Instructions pending.)
Note: See TracBrowser for help on using the repository browser.