Synopsis

 Syntax: em_autogrip -b PATH [OPTIONS] [COMMAND [PACKAGES ...]]
         em_autogrip -?|-h|--help|--version
 Commands:
 -b|--base-path PATH:           path to the top level grip directory [required]
 -p|--package PACKAGES ... :    add binary package(s) to the repository
 -s|--source  PACKAGES ... :    add source package(s) to the repository
 -t|--testing:                  only work on testing instead of unstable
   --noskipold:                 reprepro option for newly added packages
   --missing:                   print a list of missing source packages
   --build-depends:             print a list of missing build dependencies
   --britney:                   print the status of testing migrations
 -?|-h|--help|--version:        print this help message and exit

Options:
 -n|--dry-run:                  check which packages would be processed
 -m|--mirror MIRROR:            use a different Debian mirror for setup
                                [default: http://ftp.uk.debian.org/debian]
   --filter-name STRING:        alternative name for the filter repository
   --grip-name STRING:          alternative name for the grip repository
   --add-new:                   if a source package is found to be missing,
                                 or outdated in unstable, add it to the list.

Description

The default is to update all the packages so far existing in the filter repository, in all architectures.

After adding binary packages, ensure that em_autogrip is run without any options so that any missing source packages and any other Emdebian TDebs can be updated.

Although em_autogrip will setup the initial configuration files for the repository, it will not modify any existing files *except* the pkglist filter that prevents the mirror adding unwanted packages.

The mirror option only has an effect if there is no repository already found at the specified directory.

In particular, em_autogrip will only handle unstable by default. Migrations to testing and stable, even the creation of testing and stable, are not handled by em_autogrip. em_autogrip defaults to including packages into unstable and will only include packages directly into testing in "--testing" mode when updating to versions of packages uploaded into Debian via testing-proposed-updates or when catching up with a new (or stalled) repository. "--testing" mode requires a pre-configured testing configuration in "reprepro".

Public repositories should also use Secure-Apt by adding a value for SignWith: to each distribution in the Grip repository (there is no point signing the filter repository as it should not be public and is merely a filtered copy of existing, officially signed, repositories).

em_autogrip also updates the locale repository, shared by Emdebian Grip and Emdebian Crush.

Note that em_autogrip will only update the *binary* package(s) specified, even when it includes the full source package. This is down to how reprepro runs the filtered update - all binary packages expected to be listed in 'dpkg --get-selections' are included and even if a source package includes another binary, it will not be downloaded in the reprepro update. As em_autogrip does not actually build any packages from source, unless reprepro downloads the pre-built binary into the filter repository, that binary package will not be available to em_autogrip. This means that the same source package in Debian may be listed as generating a *smaller* number of binary packages in Emdebian Grip.

Secure Apt and reprepro

The secret key for the GnuPG key specified with SignWith: needs to be in the secret keyring of each user performing repository updates.

To verify the release in update rules, copy /etc/apt/trusted.gpg to ~/.gnupg/trustedkeys.gpg for all users who need to run updates. To add keys to the list available for "gpgv" use:

 C<gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --import keys.gpg>

Bugs

Problems with the automatic gripping of packages:

 1. Source packages need to complement binary packages.
 2. Binary packages with the same name as the source package cause both
    to be included
 3. Some such binary packages cause unwanted dependencies to be added.
 4. Some Architecture:all packages are dependencies of packages that only
    exist on some architectures, which breaks the edos-debcheck.

An example of 3. is lsb. An example of 4. is acpi-support-base.

The source package lsb is needed to complement lsb-desktop but the lsb binary package is a meta-package for the entire lsb suite which brings in all of Qt.

acpi-support-base is Architecture: all but depends on acpid which is Architecture: any [i386 amd64] - i.e. acpi-support-base should only exist on i386 and amd64 but as it is Architecture: all, it gets added to arm, armel, mips, mipsel and powerpc as well - at which point it has to be removed. There are ongoing discussions about such packages.

 http://lists.debian.org/debian-devel/2009/01/msg00246.html

Signal:Noise ratio in output

One important point here - reprepro outputs a lot of messages and may include lots of statements about errors and checksum mismatches, 'skipping foo' and 'downgrading bar' from and to the same version. The problem is that the useful information is hidden within all the noise, so not all reprepro STDERR (or STDOUT) output can be simply ignored. For now, just go by effects. If something is broken, look for errors that relate specifically to that package but ignore ``errors'' where everything is fine. Something like that. More work in the Emdebian::Grip module should isolate duplicate operations and unnecessary work, which in turn, should cut out most of the noise.

Using add-new

In --testing mode, em_autogrip checks for packages that have missing or outdated source packages in unstable and outputs a sample command that can be run to fill the gap. If --add-new is used, that sample command will be run - it does mean that --add-new requires --testing and that a second run of --testing without --add-new will be needed. This support is part of grip_cron.sh

Build dependencies

In the absence of a quicker way to identify which real package Provides: a virtual dependency, apt-cache showpkg is used against the main system cache. If this machine is not running Debian unstable, the list may be inaccurate or skip dependencies that are provided by packages that are only available in unstable (or if running stable, packages which are only in unstable or testing).

Repetition

If a package fails to build from source in Debian, "em_autogrip" will keep on trying to update it until the same version exists in the filter repository for all supported architectures.

Equally, manual tinkering with packages in the Grip repository, e.g. adding modified versions for testing, will cause the original Debian version to keep appearing in the "em_autogrip" updates and reprepro will ignore the built package as long as the modified version is higher.

Old packages

"em_autogrip" does not handle removals from the archive - these are manual within Debian too. Packages that only exist in stable or oldstable will confuse "em_autogrip", especially if the old package name is 'Provided' by another package which already exists in Grip. e.g. postgresql.

Adding lots of packages in one run

Sometimes, perhaps when setting up a new mirror, a full list of packages already exists on another site. Copying that pkglist into the new site will clear that list as the filter repository on the new site is empty. To avoid this problem, create the pkglist you need, then run the filter update run directly:

 reprepro -b /PATH/filter -v update

Now run "em_autogrip" without specifying any packages.

 em_autogrip -b /PATH/

Note that "reprepro" needs the path to the filter directory, "em_autogrip" needs the path to the directory above where it can find ./filter/, ./grip/ and ./locale/.

"em_autogrip" will then update the pkglist file with the final contents of the filter repository.

Ubuntu / non-Debian sources/suites

Emdebian Grip is still Debian, so although non-Debian repositories can be supported, the resulting Grip repository still requires a Debian-like layout. In particular, an 'unstable' suite must exist, even if the codename of that suite is not called 'sid'. Equally, if the repository is to support britney migrations, a suite called 'testing' must exist.

Remember, suites will change when a Debian stable release is made (i.e. testing points to something else after the release compared to what it contained before the release). Codenames do not change - squeeze always contains squeeze, even once squeeze is released as stable.

Architecture list

The list of architectures supported by a particular Grip setup cannot be easily changed - a lot of repository updates are needed before new architectures can be added to the array. Existing architectures can be dropped relatively easily. Sequence is unimportant.

 @archlist = qw/i386 amd64 arm armel powerpc mips mipsel/;

recursive edos considered risky

Recursion is still risky so edos is left as a manual step. The problem appears to be that once the repository gets out of step with Debian, an update must happen before edos can be resolved. Once the repository is up to date, edos can be run, apparently, without problems. Testing continues to see if simply moving the function lower in the flow resolves the problems.

Copyright and Licence

 Copyright (C) 2007-2010 Neil Williams <[email protected]>
 This package is free software; you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
 the Free Software Foundation; either version 3 of the License, or
 (at your option) any later version.
 This program is distributed in the hope that it will be useful,
 but WITHOUT ANY WARRANTY; without even the implied warranty of
 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 GNU General Public License for more details.
 You should have received a copy of the GNU General Public License
 along with this program.  If not, see <http://www.gnu.org/licenses/>.