DEBIAN GFORGE MAINTAINER HOWTO ------------------------------ Here is a short HOWTO explaining a few of the tricks that are used by the Debian Gforge packages. ATTENTION TO NMUERS ------------------- If you intend to NMU this package, *please* read it thoroughly. While you can make it up as you do it as far as the helpers are organised, the database handling is, to say the least, very fragile. And errors are fatal. So go read the section entitled "THE DATABASE". Do it now. Don't skip it. Really. HELPERS (DEBHELPER & DEBCONF) ----------------------------- The package uses Debhelper and Debconf. While not all features of are used (no Emacsen modules, no shared libraries, no rollback features in Debconf, etc.), some of them are. Where they are, it is in a fairly straightforward way. No black magic or advanced features are used. PACKAGE ORGANISATION -------------------- Historically, the package was monolithic: there was only one (rather large) package, called "sourceforge". This package went and changed things all over the system, configuring a database, a web server, the mail transfer agent, etc. In order to keep a semblance of order in all the actions involved, we separated them by "subsystem". There was the database subsystem, the web server subsystem, the LDAP subsystem, etc. These subsystems were each represented by a script handling most of it. These scripts are called deb-specific/install-*.sh. The main maintainer scripts (postinst, prerm, cron jobs and suchlike) called these scripts in turn. The install-*.sh scripts handle different parameters: "configure", "purge" and "update" have rather explicit names (I hope); "configure-files" and "purge-files" are special targets. They are used as a way for the postinst to delegate the task of computing a proposed change in a configuration file to the subsystem. The postinst then uses Debconf to ask whether the proposed file should be used, takes appropriate action, then lets the subsystem finish its configuration. Similar things happen for prerm scripts. The package is now split into several pkgname-* packages. Some of them still contain some install-*.sh scripts (usually at most one). Each pkgname-* package installs its corresponding subsystem, or a semblance thereof. For instance, it is planned that the database can be hosted on a different server than the website. But the website still needs to know where the database server is. In this case, the database host will have to install pkgname-db-local (or whatever the package is named), and the web server will have to install pkgname-db-remote. The -db-local package will still install the database (thus invoking install-db.sh), but -db-remote will only install what is needed by the database *clients* (basically, the host where the database is installed and the appropriate password). There are therefore some areas where packages overlap: the database password is a variable "provided" by both the -db-local and -db-remote packages. Debconf is intelligent enough not to ask the corresponding question twice, but it is interesting nevertheless to keep the appropriate variables where they belong, and only there. Hence the use of a Debhelper-like trick, as described below. DSF-HELPER ---------- This is the most tricky part of the source package. The maintainer scripts and Debconf templates are not used "as is", but they are instead generated from templates. In much the same way as Debhelper replaces #DEBHELPER# lines in maintainer scripts by appropriate chunks of code to add the needed functionality to packages, this package builds the maintainer scripts (and Debconf templates) from templates by inserting bits of text in them. These bits can be either simple text (like for Debconf templates), or bits of code (like the appropriate code to handle one particular Debconf variable in a .config file, or a function to repeatedly ask for a password until two consecutive answers match). I call this trick DSF-Helper (for "Debian Sourceforge helper"). It's largely inspired from Debhelper (particularly dh_installdeb) in both its concepts and implementation, and it might result in a patch submitted against Debhelper proper when I'm confident it works and is useful. It is currently implemented in Perl. The "bits of stuff" are grouped by identifiers. For each identifier, you can have one chunk of text for each family of generated files (currently the families are .templates, .config, .preinst, .postinst, .prerm and .postrm). For instance, a Debconf variable shared between several packages will have one chunk for the .templates file (containing the Debconf template), one for the .config file (containing the appropriate Debconf call), and one for the .postinst file (containing code to turn this Debconf variable into a line in a configuration file). Each subpackage can then use some of these "bits of stuff" in its files. To do so, the files must be named *.dsfh-in and include lines like #DSFHELPER:identifier#. These files will be processed by dsf-helper.pl and turned into the appropriate files, with the keywords replaced by the appropriate text. Now for a few examples. - get-debconf-password: this is a simple shell function looping until the user types the same passwrd twice. This function is mostly useful in .config files, hence the "group" consists of the sole debian/dsf-helper/get-debconf-password.config file. To use it in a .config file, just rename that .config file as .config.dsfh-in and include #DSFHELPER:get-debconf-password# in it. - ldap-variables: this one involves both Debconf templates and .config code. Just include #DSFHELPER:ldap-variables# in both the .templates.dsfh-in and .config.dsfh-in, and DSF-Helper will insert the appropriate chunk of text in the appropriate file. The rationale behind DSF-Helper is that the code handling, say, one particular Debconf variable is likely to change from time to time, and to be added to one subpackage and removed from another. It can become a big hassle just to maintain the code in different files and keep it consistent, and creating a new subpackage is also a tedious task. DSF-Helper makes these tasks a bit more automated. Each bit of code is only maintained in one file, and it's propagated into every package at package building time. Creating a new package can be "just" a matter of picking the appropriate bits of code, and DSF-Helper will put them where needed. To draw a comparison with compiled C code, DSF-Helper is separate compilation (each function in its own file) made into static binaries. The "static" part is this: I could of course have put all the bits of code into one external file, and source it at run time, but that cannot work for .config, .preinst and .postrm scripts since they are executed when the package is not installed (not yet unpacked or already removed). THE DATABASE ------------ It is vitally important that extreme care be taken for changes in the database. A smooth upgrade path has been provided so far by carefully sequencing the changes in the database and making sure they will not conflict with each other or some other trick. This section is targetted at people who touch deb-specific/db-upgrade.pl. Please read it thoroughly, and don't skip paragraphs. A single mistake can be a nightmare to fix (and believe me, I know that). The database, as created by the package, has a lot of tables for the software proper, plus one especially used to store a version of the database scheme. That one table is named debian_meta_data, and contains two fields (named 'key' and 'value'). Two rows are currently used: the one for which the key is 'db-version', and the one for which the key is 'current-path'. Together they store the current status of the database. The value of 'current-path' is only used during the first installation of Sourceforge 2.6. If that installation is done on a fresh machine, the value will be 'scratch-to-2.6'. If the installation is done as an upgrade on Sourceforge 2.5, the value will be '2.5-to-2.6'. In any case, when that first installation is completed, the row is deleted. There's currently no other use for this row. The value of 'db-version' is a string encoding a version. The ordering method for these strings is the one provided by dpkg --compare-versions. The values currently used more or less match the package versions, but you shouldn't depend on it. Special procedures (upgrading from 2.5, or installing a fresh 2.6, or future cases maybe) involve names not corresponding to a package version. They still must be ordered according to dpkg. The database upgrader (db-upgrade.pl) can be seen as walking along a path. Either that path is an explicit one (during first install or upgrade from 2.5), or it is the default one (upgrading betwen versions of 2.6). In any case, the walking is made in steps. Each step has a target version (where it leads) and a series of actions to perform to reach that target. If the current version of the database is lesser than the target version, the actions are performed, and the current version is updated to the target version. If not, that step is skipped. It is very important that steps are attempted in the correct order (ascending order of target versions), otherwise steps will be skipped. It is also very important that the actions are dependable. Generally, when executing the actions for step n, you can depend on the database being compliant with the last step before n. Tip #1: don't assume things unless you are certain they are true. Don't assume some value is missing from a table simply because it's missing from yours. Don't assume some value is available. Tip #2: test your patch. Install Sourceforge 2.6 from scratch. Install it over a freshly installed Sourceforge 2.5. Install it over a previous version of Sourceforge 2.6. If at all possible, install it over a non-empty database coming from 2.5, then do it again over a non-empty database of 2.6. If *any* of these break, *don't upload*. Period. Tip #3: remember, there's no way back. Once your package has been installed by some user, the database version has irreversibly been bumped up. While you can sometimes revert changes by hand on your local database, you can't provide a rollback for potential thousands of users (even it the actual number is more in the tens than in the thousands). Tip #4: the db-upgrade.pl script has evolved over time to involve a few functions, such as get_db_version and update_db_version. Use them. Your best bet would be to cut and paste a block, change the $target, and change the actions. Tip #5: pay attention to the ordering of the blocks. You can't go back in time, so all your changes must be at the end of the series of actions. There. Thanks for reading. I know it sounds boring, but I can guarantee you will avoid problems if you understand this. Maybe not all problems, but definitely some of them. CONTRIBUTING ------------ The source is maintained in bzr (see debian/control's Vcs-Bzr: field). A copy can be checked-out using : debcheckout gforge. -- Roland Mas