Introduction
I recently decided to review the source and package development cycles I was following, and to update my build scripts accordingly. This article documents that review.
Before reading any further, there are some aspects of the scripts I wrote that may make them unsuitable for other people to use:
- the scripts require use of Subversion to store program sources and Debian package control files
- the scripts expect that a copy of the unpacked source tarballs are not stored inside the module containing the Debian package control files; normally the program developer stores his sources in a module in one repository (e.g. as the Dizzy developer does here) but the package developer also stores a copy of the source files along with the Debian package control files in another module (e.g. as Debian do for Dizzy here); however, I store the program sources in one repository (e.g. my FAD sources are here), my package control files somewhere else (e.g. my FAD Debian package control files are here), but the package control files do not include a copy of the source files (e.g. the FAD Debian package control files do not include a copy of the FAD sources)
- the scripts depend on ADE
- adech(1) currently insists on release numbers being structured <epoch-index>.<feature-index>.<bugfix-index>; this may not suit everybody
If you’re impatient and want to skip ahead then the build scripts are here and an example of their usage for developing a program and a package starts here.
Why review?
First, I recently upgraded from Debian 5 (lenny) to Debian 6 (squeeze); this meant I upgraded from dpkg-buildpackage version 1.14.31 to dpkg-buildpackage version 1.15.8.10, whereupon my build scripts started reporting:
dh: Compatibility levels before 5 are deprecated.
Thankfully, this was not an error, it just a warning, but I needed to fix it.
Second, while I was constantly swapping my “Program Developer” and “Package Developer” hats, I was seeing some symmetry between the two development processes. I wondered if restructuring both cycles to increase the symmetry might result in both processes becoming clearer and simpler.
Third, I am not a Debian developer, but it makes sense that I periodically check what is currently best practice in the Debian developer community (preferrably before addressing the first issue). I discovered the following improvements had been made in the Debian packaging toolset:
- debhelper‘s range of supported Debian package format values has shifted up beyond the one I was using
- debhelper now includes dh
- Quilt is now the standard patch format
Fourth, my own tools had improved:
- ADE (a script development toolset) now includes adech(1) to manipulate sources’ ChangeLog files
- PAA (an package and mirror administration tool) can support the UNRELEASED Debian distribution (actually, it did a long time ago, but I only realised recently!)
Fifth, while the Debian New Maintainers’ Guide is an excellent reference and The Debian Wiki’s Introduction to Debian Packaging is an excellent introduction and even mentions workflow, neither describe the package development cycle.
Terminology
Some terms need to be defined:
- sources: source code and supporting files used to build a program (e.g. xx.c and Makefile might the sources for a xx program)
- tarball: compressed tar file containing the sources (e.g. xx-1.0.tar.gz might contain xx-1.0/xx.c and xx-1.0/Makefile)
- program development cycle: the cyclical executing of steps to develop sources into tarballs
- Debian package control files: source code and supporting files excluding the tarball used to build a Debian package (e.g. control, copyright, changelog, rules might be the package control files for the xx Debian package)
- Debian package: the Debian package file (e.g. xx_1.0-1_i386.deb might be the Debian package for the xx program)
- Debian package development cycle: the cyclical executing of steps to develop Debian package control files into Debian packages
This article will stick to these terms from here on.
The program development cycle
The normal program development cycle goes something like this:
Some steps might be expanded (e.g. “Schedule release” might involve collecting requirements or submitting a change request to a change review board and awaiting approval); other steps might be reduced or omitted (e.g. after “Regression test binaries” perhaps “Regression test tarball” is redundant).
The package development cycle
The package development cycle is similar to the program development cycle: both involve developing text files (package control files and sources, respectively), which are the inputs to a process (running dpkg-buildpackage and compiling, respectively), which goes on to produce output files (packages and tarballs, respectively); both involve testing, stamping the developed files with a release number and ultimately remaking the output files using on the stamped versions of the input files.
The following diagram illustrates the similarities:
Optimising the cycles
First, in order to reduce the amount of time the user has to be present and the amount of key-presses the user has to type, I considered tweaking the order of steps to put those that are non-interactive into clusters. Those clustered steps could then be implemented as single script or as individual scripts called by a single wrapper script.
For example: “Compile sources”, “Regression test binaries” and “Make tarball” are three non-interactive steps that are already clustered together; they could be implemented in a new “test-sources” script.
This approach could be taken too far, although what constitutes too far could be considered a matter of taste.
For example: “Release sources” and “Make tarball” are two non-interactive steps that are already clustered together; they could be implemented in a new “test-sources” script, but I thought most developers, myself included, want to do this step themselves.
Second, any non-interactive step is a candidate for having non-interactive enhancements added to it.
For example: the “Release sources” and “Release package control files” steps are non-interactive because the information they require is already cached (in doc/ChangeLog and debian/changelog, respectively), and given that these are the only steps that permanently change files, we might repeat the regression tests (“Regression test sources” and “Regression test package control files”, respectively) before really releasing the files.
The clustering and the addition of the example enhancement are illustrated in the diagram below:
Third, I want to enter the cycle as soon as possible, even before I have working sources or a working package; this way as much as possible of the development is subject to the designed change management protocols. To this end, I first create an almost empty Subversion module for sources or for package control files, commit them and then immediately submit a bug report saying “It doesn’t work.” I call this “Seeding”.
Fourth, the above diagrams show that the loops are repeated without any pause, but this is not really the case; in reality the developer is blocked, awaiting an event that unblocks him. For the package development cycle this event is one of:
- the program developer releases a new version of the sources (the package developer will need to package it)
- a bug is detected in the package (by the package developer himself or by somebody upstream)
- a bug in detected in the program but is reported to the package developer (by somebody upstream)
Of course, similar events block the program developer.
The seeding steps and the handling of blocking events are illustrated in the diagram below:
One can easily imagine the complex web of program and package development cycles, like the two above, that form a large part of the free software community.
The scripts
Looking at the colour coding in the above diagram, scripts for the following steps are needed:
| sources task | sources script name | package control task | package control script name |
|---|---|---|---|
| seed sources | prologue-sources.pasta-net | seed package control files | prologue-pkgctrl.pasta-net |
| schedule release of sources | schedule-sources | schedule release of package control files | schedule-pkgctrl |
| regression test sources | test-sources | regression test package control files | test-pkgctrl |
| release sources | release-sources | release test package control files | release-pkgctrl |
| make tarball | make-tarball | make package | make-package |
| publish tarball | epilogue-sources.pasta-net | publish package | epilogue-pkgctrl.pasta-net |
Note that:
- the script to perform the “Seed sources” and “Seed package control files” steps are called “prologue-sources” and “prologue-pkgctrl”; it is expected that these scripts will do the seeding in a very site-specific manner and will do other site-specific actions at the same time; the links above are for example scripts; you must write your own
- similarly, the script to perform the “Publish tarball” and “Publish package” steps are called “epilogue-sources” and “epilogue-pkgctrl”; it is expected that these scripts will do the publishing in a very site-specific manner and will do other site-specific actions at the same time; the links above are for example scripts; you must write your own.
You can download scripts or a Debian package of them here.
Using the scripts
This section is taken from the bs(1) man page.
BS(1) BS(1)
EXAMPLES
In this example session, we create a new program 'foo' and package it.
We base the configuration on the example configuration file included
with bs, but use a new local Subversion repository and keep our working
copies in ~/dev:
user$ cp /usr/share/doc/bs/examples/bs.conf ~/etc/
user$ export BS_CONFIG=~/etc/bs.conf
user$ svnadmin create ~/var/svnrepo
user$ echo SOURCES_SVNREPO_URL_PREFIX=file://~/var/svnrepo \
>> ~/etc/bs.conf
user$ echo PKGCTRL_SVNREPO_URL_PREFIX=file://~/var/svnrepo \
>> ~/etc/bs.conf
user$ echo SOURCES_SVNWC_DIR_PREFIX=~/dev >> ~/etc/bs.conf
user$ echo PKGCTRL_SVNWC_DIR_PREFIX=~/dev >> ~/etc/bs.conf
We create the 'foo' program sources using the example prologue-sources
script included with bs (this particular example script uses adegmt(1)
to generate the module, complete with man pages and a basic regression
test suite. Adegmt(1) needs to be told the name of a template module,
so the script prompts for this information):
user$ bs ps foo
template: lxshell
At this point we have a fully functional 'foo' program source tree in
the Subversion repository and in ~/dev.
Immediately, we submit a bug report stating that the program does not
do what it is supposed to do. It may seem odd to submit a bug report
even before the program sources have been written but it reduces the
amount of effort that is outside of the development cycle and therefore
not subject to the cycle's control, its checks and its logging. Let us
suppose that this bug report is assigned the ID 'FOO#001'.
We announce our intent to fix this bug by scheduling a new release:
user$ bs ss foo
bug ID []: FOO#001
bump type (b)ug, (f)eature or (r)ewrite [b]:
We test the program sources:
user$ bs ts foo
M foo/doc/ChangeLog
foo/doc/Makefile: no keywords property
foo/man/Makefile: no keywords property
foo/man/foo-config.1: no keywords property
...
? foo/bin/foo
? foo/bin/foo-config
? foo/bin/foodevsh
...
The error messages above relate to missing Subversion properties and
the automated modification of the ChangeLog made by the previous step;
these are fixed by using svn pset svn:keywords and svn pset svn:ignore
and then committing all changes back to the repository:
user$ svn commit -qm " * first proper version" foo
If we repeat the tests then no errors are shown:
user$ bs ts foo
Now we are free to release the sources and make the tarball:
user$ bs rs foo
user$ bs mt foo
user$ ls -l /pub/computing/software/local/sources/localpublic
-rw-r--r-- 1 alexis alexis 11964 Jan 16 11:25 foo-0.tar.gz
(Note that the initial release number is 0; see adech(1) for an expla-
nation.)
Now that the sources have been released, we take off our program devel-
oper hat and put on our package developer hat.
We create the 'foo' Debian package control files using the example pro-
logue-pkgctrl script included with bs:
user$ bs pp foo
Immediately, we submit a bug report stating that the package does not
do what it is supposed to do. It may seem odd to submit a bug report
even before the package control files have been written but it reduces
the amount of effort that is outside of the development cycle and
therefore not subject to its control, its checks and its logging. Let
us suppose that this bug report is assigned the ID 'FOOPKG#001'.
We announce our intent to fix this bug by scheduling a new release of
the package:
user$ bs sp foo
schedule-pkgctrl: WARNING: don't forget to update the changelog
(it's got dummy text in)
We test the program sources:
user$ bs tp foo
M /home/alexis/dev/def/foo.debian/changelog
/home/alexis/dev/def/foo.debian/watch: no keywords property
/home/alexis/dev/def/foo.debian/rules: no keywords property
E: foo: description-is-dh_make-template
E: foo: section-is-dh_make-template
W: foo: superfluous-clutter-in-homepage
...
The error messages above relate to missing Subversion properties, the
automated modification of the ChangeLog made by the previous step and
lintian(1) errors; these are fixed by using svn pset svn:keywords and
svn pset svn:ignore, fixing the lintian errors (the nature and descrip-
tion of which is outside the scope of this document) and then commit-
ting all changes back to the repository:
user$ svn commit -qm " * correct svn keywords" foo.debian
If we repeat the tests then no errors are shown:
user$ bs tp foo
Now we are free to release the Debian package control files and make
the package:
user$ bs rp foo
user$ bs mp foo
user$ ls -l /pub/computing/software/local/debian/localpublic.queue/
-rw-r--r-- 1 alexis alexis 5384 Jan 16 11:43 foo_0-2_all.deb
The make-package step puts the package in a directory specified by the
configuration file but it does not regenerate any repository control
files. The example epilogue-package file does that:
root# bs -f ~user/etc/bs.conf ep foo
(Note that, in this example, this step is run as root, necessitating
the use of the -f option or the temporary setting of the BS_CONFIG
environment variable.)