source: trunk/debathena/NOTES @ 22957

Revision 22957, 12.7 KB checked in by ghudson, 16 years ago (diff)
Document current infrastructure for adding a suite.
Line 
1This hierarchy contains Debian/Ubuntu-specific materials, also known
2as "Debathena".  The contents are:
3
4* debathena - Debathena-specific software packages such as PAM and NSS
5  modules.
6
7* config - Packages for configuring native system software in a manner
8  appropriate for Athena.
9
10* meta - Packages which contain nothing but dependencies on other
11  packages and serve as an installation convenience.
12
13* scripts - Build scripts and supporting materials.
14
15Debathena is a SIPB project, and its infrastructure and procedures
16will need to be adapted for Athena 10.  For the moment this file will
17document the Debathena procedures as they are, not as they will be.
18The current procedures do not even use this svn repository yet.
19
20Debian software used by Debathena:
21
22  * schroot - Used to manage build chroot environments for each
23    Debian/Ubuntu version.  We use the lvm-snapshot schroot type,
24    which allows rapid construction of ephemeral copies of template
25    "source" chroots, so that every binary package build is done in a
26    clean environment.
27
28  * debuild - Used to create Debian source packages from package
29    source directories.
30
31  * sbuild - Used to build binary packages from source packages inside
32    schroot environments.
33
34  * equivs - Used to create packages which only contain dependency
35    information.  Somewhat of a dirty hack, since it doesn't keep
36    proper changelogs, but it reduces overhead.
37
38  * CDBS (Common Debian Build System) - Referenced by debian/rules
39    files in packages.  Contains standard build rules to cut down on
40    per-package boilerplate.
41
42  * reprepro - Used to upload packages into the apt repositories.
43
44  * approx - Used to create a local cache of Debian packages on the
45    build server.  This cache is referenced by the build chroots for
46    improved performance.
47
48The remainder of this file documents procedures useful to Athena 10
49developers and the release engineer.
50
51Developers: Preferences setup
52-----------------------------
53
54You will probably want a $HOME/.devscripts file containing the
55following:
56
57DEBUILD_DPKG_BUILDPACKAGE_OPTS="-sa -us -uc -i -I.svn"
58
59This will save you from having to specify those options every time you
60run debuild.  Athena 10 scripts do not assume the above preferences,
61but the instructions in this file do.  The options mean:
62
63  * Look for original source as a tarfile or create one.
64  * Do not sign the source package.
65  * Do not sign the changes file.
66  * Ignore common version control metadata files when creating diffs.
67  * Ignore .svn paths when creating tarballs.
68
69You should also set the environment variable DEBATHENA_APT to
70"/afs/dev.mit.edu/system/athena10/apt".
71
72Developers: Preparing a change
73------------------------------
74
75To prepare a change to a regular package (a source tree containing a
76debian/ subdir), make the edits in a checkout and record a changelog
77entry.  You can either edit debian/changelog using emacs changelog
78mode (C-c C-v to add a new version entry, C-c C-a to add a change
79entry, C-c C-f to finalize the entry) or you can run "dadch".
80
81When creating a new version entry, bump the upstream version number
82(to 10.0.0 if it was not already that high) if you are changing the
83main package source.  Otherwise, just bump the Debian version
84component (change 0debathena1 to 0debathena2, for instance).
85
86Developers: Building a package for test purposes on one platform
87----------------------------------------------------------------
88
89After you have prepared a change, you will want to test that it builds
90and perhaps that it works before committing it.  First, if it is an
91Athena source directory using autoconf, run "daconfiscate" to set up
92the autoconf boilerplate which we don't check in.  Second, run
93"daorig" to copy or create an orig tarball in the parent directory if
94necessary.  Third, run "debuild".  The resulting package will be
95placed in the parent directory.
96
97In order to test if the package works, you can install it with "dpkg
98-i filename.deb".
99
100Developers: Building a package for test purposes on all platforms
101-----------------------------------------------------------------
102
103If the package you are working on interacts with the native OS in ways
104that might vary from platform to platform, you may want to do a test
105build for all platforms.  You will need to do this on
106linux-build-10.mit.edu or another machine which has been set up with
107build schroots.
108
109As above, run daconfiscate (if necessary) and then daorig.  Then run
110"debuild -S" to create a source package.  Now cd into the parent
111directory and identify the .dsc file created by debuild -S; it will
112have a name like debathena-just_9.4.0-0debathena2.dsc.  Run "da
113sbuildhack filename.dsc" to perform the package builds.  Each build
114will take place inside an ephemeral chroot based on a snapshot of a
115template for a particular Debian or Ubuntu version.  If a build fails
116and it's not obvious from the build log why, you may need to create
117your own ephemeral chroot session with a command like "schroot -c
118gutsy-amd64-sbuild /bin/sh" and then run debuild from within the
119package sources.
120
121If the build is successful, it will create a set of packages with
122names like debathena-just_9.4.0-0debathena2~ubuntu6.06_amd64.deb.
123
124Developers: Building an equivs package
125--------------------------------------
126
127Most of the packages under debathena/meta are faked up using equivs.
128To build one, just run:
129
130  equivs-build --full filename.equivs
131
132These equivs files make reference to ../common, so you must have a
133checkout of debathena/meta/common alongside the particular
134meta-package you are building.
135
136Release engineer: Bootstrapping the project infrastructure
137----------------------------------------------------------
138
139  1. Create the package repository (detailed instructions on this
140     pending).  Set the DEBATHENA_APT environment variable to point to
141     the package repository.  Put a copy of the debathena "scripts"
142     subdir in your path.
143
144  2. Create the build area.
145
146  3. Build each equivs package under meta/ using "equivs-build --full
147     *.equivs" and upload each with "daequivsupload *.changes".  This
148     has the side-effect of creating the basic structure of the
149     package repository.
150
151  4. Set up the build server.  The basic structure of the apt
152     repository must work for make-chroot to succeed, so this must
153     happen after step 3.
154
155  5. For each normal Debian package in dependency order, cd into its
156     directory in the build area and run "da sbuildhack *.dsc" and
157     "daupload-release *_source.changes".  If the package contains
158     only an "Architecture: all" binary package, pass the -A option to
159     both commands.
160
161     The all-packages script can generate an approximation of the
162     package list in dependency order, but it doesn't work right yet,
163     and ideally it would be possible to do several builds in parallel
164     using a Makefile like the one in scripts/build-server/build-all.
165     Improvements to this machinery are pending.
166
167  6. For each package under third, run "da ./debathenify-PKG source
168     binary upload".  This infrastructure depends on the chroots being
169     up to date, so run "all-schroots upgrade-schroot" beforehand if
170     substantial time has passed since they were created.
171
172Release engineer: Setting up a build server
173-------------------------------------------
174
175  1. The build server must be installed with free space in an LVM
176     volume group.  The build chroots consume 2GB each.  There is a
177     known memory corruption issue with LVM snapshots in the kernel
178     used in Ubuntu Gutsy (which is based on 2.6.22), so use a newer
179     kernel such as the one in Ubuntu Hardy (based on 2.6.24) instead.
180
181  2. Install debathena-standard as per the the instructions in
182     http://debathena.mit.edu/install.
183
184  3. apt-key add /afs/dev.mit.edu/system/athena10/apt/athena10-archive.asc
185
186  4. Install the packages listed in
187     scripts/build-server/packages (using "aptitude install")
188
189  5. Install debathena-login, debathena-ssh-server, and
190     debathena-build-depends (using "aptitude install").
191
192     (Depending on how recently debathena-build-depends was rebuilt,
193     additional packages might need to be installed to satisfy the
194     build-depends of newer packages.  This can be taken care of later
195     when an error occurs building a source package.)
196
197  6. Edit /etc/security/access.conf and add a first line:
198     -:ALL EXCEPT root <developer usernames>:ALL
199
200  7. Edit /etc/pam.d/schroot, comment out "@include common-session",
201     and add:
202
203       # Basic pam_unix session module in place of common-session.
204       session required         pam_unix.so
205
206  8. Edit /etc/group and add the developers to the sbuild group.
207
208  9. Create /etc/passwd entries for each developer with "hesinfo
209     username passwd >> /etc/passwd" and then run pwconv.
210
211     (This is not necessary for the login system on the main root
212     environment, but is for the chroot environments.)
213
214  10. Append to /etc/approx/approx.conf the contents of
215       scripts/build-server/approx.conf.tail.
216      Change the last line from http://debathena.mit.edu/apt to
217       file:///afs/dev.mit.edu/system/athena10/apt
218      Add "$interval 0" above the repository lines (only necessary if
219       the version of approx as reported by "dpkg -l approx" is less
220       than 3.0)
221      Run: /etc/init.d/approx restart
222
223  11. Apply scripts/build-server/10mount.patch.
224
225  12. For each supported DIST (see scripts/debian-versions.sh) run:
226
227        VG=/dev/blah scripts/build-server/make-chroot DIST i386
228        VG=/dev/blah scripts/build-server/make-chroot DIST amd64
229
230      substituting the name of the volume group for blah.  Omit the
231      amd64 line if DIST is sarge.
232
233      Example: VG=/dev/dink scripts/build-server/make-chroot gutsy i386
234
235Release engineer: Removing a build chroot on the build server
236-------------------------------------------------------------
237
238  1. Run VG=/dev/blah scripts/clean-schroots as root to make sure that
239     the build chroot is not mounted, substituting the name of the
240     volume group for blah.
241
242  2. Edit /etc/schroot/schroot.conf and delete the section
243     corresponding to the chroot.
244
245  3. Run lvchange -an blah/chrootname
246     substituting the name of the volume group for blah and the chroot
247     name for chroot.  Example: lvchange -an dink/gutsy-i386-sbuild
248
249  4. Run lvremove blah/chrootname
250
251Release engineer: Setting up a canonical build area
252---------------------------------------------------
253
254  1. Create an empty directory and cd into it.  The canonical build
255     area lives in /afs/dev.mit.edu/project/release/10/build.
256
257  2. Run gen-packages to create the table of normal Debian packages.
258
259  3. Run dasource to create subdirs and source packages for each
260     normal Debian package.
261
262  4. Create checkouts of the meta and third directories:
263
264     svn co svn+ssh://svn.mit.edu/athena/trunk/debathena/meta
265     svn co svn+ssh://svn.mit.edu/athena/trunk/debathena/third
266
267     (A couple of subdirectories of debathena/meta are normal Debian
268     packages, so this will create redundant copies of those.  Ignore
269     them; they won't be used.)
270
271Release engineer: Adding a new suite
272------------------------------------
273
274This process is rarely performed and the infrastructure for it is
275imperfect.  Substitute the name of the new suite for "newdist" in all
276steps below.
277
278  1. Create the new distribution in the apt repository's configuration
279     file.  Create the skeleton of the dist by installing at least one
280     equivs package from meta/ with "reprepro -Vb $DEBATHENA_APT
281     include newdistname file.changes".
282
283  2. On the build server, create a chroot for the new distribution as
284     documented above.  This may require downloading and installing a
285     more recent version of the debootstrap package from the
286     -backports dist corresponding to the build server's OS.
287
288  3. Set the DEBATHENA_BUILD_AREA environment variable to point to the
289     build area.
290
291  4. Fire up screen.
292
293  5. mkdir $DEBATHENA_BUILD_AREA/stamps.newdist.
294
295  6. cd into a checkout of debathena/scripts/build-server/build-all.
296
297  7. Edit Makefile (and check in the edit) so that suite is the new
298     distribution and psuite is the previously most recent Debian or
299     Ubuntu distribution.
300
301  8. Run "make deps.mk".
302
303  9. Run "make all STAMPS=$DEBATHENA_BUILD_AREA/stamps.newdist".  You
304     can watch the builds happen in the other windows of the screen
305     session.  It's possible to do several builds at once with make -j
306     N.
307
308  10. debathenify packages will fail out; they must be built by hand.
309      When the build fails on one, cd into third/packagename in the
310      build area and run "./debathenify newdist-amd64 -A source binary
311      upload" and "./debathenify newdist-i386 binary upload".  Then
312      touch $DEBATHENA_BUILD_AREA/stamps.newdist/packagename.done" and
313      restart the build.
314
315  11. Go into third/openafs in the build area and build AFS modules
316      for the new suite's kernels.  (Instructions pending.)
Note: See TracBrowser for help on using the repository browser.