xdeb fails if dpkg-cross has nothing to do

From what I'm seeing, this also causes more problems than just requiring xdeb to be restarted. Especially when using --only-explicit, the import failure means that, since the package has already been downloaded and the .changes file generated, the package is treated as "built" which stops any further reprocessing of the package upon next invocation... including not attempting to cross-install the packages, so xdeb happily carries on and tries to build the target package without all the cross build-deps installed.

Example:
$ ./xdeb/xdeb.py -a armel -d xdeb-scratch -b xdeb-scratch --only-explicit --prefer-apt qemu-linaro
[...]
dpkg-cross -a armel -b -X heimdal-dev -X krb5-multidev -X ssh-krb5 libkrb5-dev_1.8.3+dfsg-4ubuntu1_armel.deb libkrb5-3_1.8.3+dfsg-4ubuntu1_armel.deb libgssapi-krb5-2_1.8.3+dfsg-4ubuntu1_armel.deb libgssrpc4_1.8.3+dfsg-4ubuntu1_armel.deb libkadm5srv-mit7_1.8.3+dfsg-4ubuntu1_armel.deb libkadm5clnt-mit7_1.8.3+dfsg-4ubuntu1_armel.deb libk5crypto3_1.8.3+dfsg-4ubuntu1_armel.deb libkdb5-4_1.8.3+dfsg-4ubuntu1_armel.deb libkrb5support0_1.8.3+dfsg-4ubuntu1_armel.deb
Building libk5crypto3-armel-cross_1.8.3+dfsg-4ubuntu1_all.deb
dpkg-deb: building package `libk5crypto3-armel-cross' in `./libk5crypto3-armel-cross_1.8.3+dfsg-4ubuntu1_all.deb'.
dpkg-cross: package libkrb5-dev doesn't provide any useful files. Skipping.
Building libkadm5srv-mit7-armel-cross_1.8.3+dfsg-4ubuntu1_all.deb
dpkg-deb: building package `libkadm5srv-mit7-armel-cross' in `./libkadm5srv-mit7-armel-cross_1.8.3+dfsg-4ubuntu1_all.deb'.
Building libkrb5-3-armel-cross_1.8.3+dfsg-4ubuntu1_all.deb
dpkg-deb: building package `libkrb5-3-armel-cross' in `./libkrb5-3-armel-cross_1.8.3+dfsg-4ubuntu1_all.deb'.
Building libgssrpc4-armel-cross_1.8.3+dfsg-4ubuntu1_all.deb
dpkg-deb: building package `libgssrpc4-armel-cross' in `./libgssrpc4-armel-cross_1.8.3+dfsg-4ubuntu1_all.deb'.
Building libkadm5clnt-mit7-armel-cross_1.8.3+dfsg-4ubuntu1_all.deb
dpkg-deb: building package `libkadm5clnt-mit7-armel-cross' in `./libkadm5clnt-mit7-armel-cross_1.8.3+dfsg-4ubuntu1_all.deb'.
Building libkdb5-4-armel-cross_1.8.3+dfsg-4ubuntu1_all.deb
dpkg-deb: building package `libkdb5-4-armel-cross' in `./libkdb5-4-armel-cross_1.8.3+dfsg-4ubuntu1_all.deb'.
Building libkrb5support0-armel-cross_1.8.3+dfsg-4ubuntu1_all.deb
dpkg-deb: building package `libkrb5support0-armel-cross' in `./libkrb5support0-armel-cross_1.8.3+dfsg-4ubuntu1_all.deb'.
Building libgssapi-krb5-2-armel-cross_1.8.3+dfsg-4ubuntu1_all.deb
dpkg-deb: building package `libgssapi-krb5-2-armel-cross' in `./libgssapi-krb5-2-armel-cross_1.8.3+dfsg-4ubuntu1_all.deb'.
Traceback (most recent call last):
  File "./xdeb/xdeb.py", line 946, in <module>
    main()
  File "./xdeb/xdeb.py", line 930, in main
    native_crossed.extend(native_import(options, src))
  File "./xdeb/xdeb.py", line 560, in native_import
    crossed_debs = cross_convert(options, debs, options.builddirs[0])
  File "./xdeb/xdeb.py", line 487, in cross_convert
    spawn(convert, cwd=outdir)
  File "/home/devel/canonical/linaro/xdeb/utils.py", line 29, in spawn
    raise SubprocessException, ret
utils.SubprocessException: 1

And running the same xdeb command with --sequence now shows:

Build sequence: krb5 keyutils* zlib* tcp-wrappers* bluez* x11proto-kb* pulseaudio* xcb-util* libxtst* libxcb* curl* libpthread-stubs* x11proto-xext* libxau* gpm* libsndfile* libxdmcp* alsa-lib* libgpg-error* x11proto-input* libice* nas* gnutls26* libxt* libpng* libxdamage* libxext* libidn* esound* db4.8* ncurses* pcre3* glib2.0* python2.7* x11proto-core* libdrm* openldap* libogg* flac* brltty* libsdl1.2* mesa* util-linux* libxfixes* vde2* libcaca* libgcrypt11* slang2* dbus* libx11* openssl* e2fsprogs* cyrus-...

I tried to reproduce this with the just-merged xdeb-graph branch and get the log below. Which fails differently to the above. It seems to be entirely failing to install any host-arch packages, so falls over as soon as configure looks for one (zlib).

I'll look in to that.

#./xdeb.py -a armel -d xdeb-scratch -b xdeb-scratch --only-explicit --prefer-apt qemu-linaro
Building source package cache ...
Using file archive.ubuntu.com_ubuntu_dists_natty_main_source_Sources for apt cache
Using file archive.ubuntu.com_ubuntu_dists_natty_universe_source_Sources for apt cache
Building working tree cache ...
Build sequence: keyutils pango1.0 libxtst gdbm libsndfile libxdmcp atk1.0 fontconfig gdk-pixbuf pixman freetype libidn qt-x11-free libdrm brltty pcre3 util-linux nspr dbus libxcomposite libvorbis libsm avahi x11proto-dri2 cairo udev bluez tcp-wrappers libjpeg6b libxau nss x11proto-fixes libxi libgpg-error libice gnutls26 libxt libpng freetds libxext qt4-x11 python2.7 x11proto-core libsdl1.2 xft postgresql-8.4 openssl libogg libxxf86vm tiff libxcursor vde2 libxcb libmng cups libgcrypt11 db4.8 libxinerama libxrandr libxmu libdatrie ncurses flac xcb-util x11proto-gl zlib libjpeg8 libxrender libtasn1-3 libselinux lcms krb5 sqlite cyrus-sasl2 expat pulseaudio gtk+2.0 libpthread-stubs curl alsa-lib x11proto-xext nas sqlite3 libxdamage qt-assistant-compat unixodbc glib2.0 openldap libtool mesa libxfixes x11proto-record qtwebkit-source x11proto-render libx11 libiodbc2 e2fsprogs libthai mysql-5.1 libsepol qemu-linaro*
Installing native packages: []

===== Building qemu-linaro_0.14.50-2011.03-1-0ubuntu1 =====

dpkg-checkbuilddeps
dpkg-architecture: warning: Specified GNU system type arm-linux-gnueabi does not match gcc system type x86_64-linux-gnu.
debuild --no-lintian -eUSER -eCONFIG_SITE=/etc/dpkg-cross/cross-config.armel -eDEB_BUILD_OPTIONS=nocheck -eGTEST_INCLUDEDIR=/usr/arm-linux-gnueabi/include -eGTEST_LIBDIR=/usr/arm-linux-gnueabi/lib -ePKG_CONFIG_LIBDIR=/usr/arm-linux-gnueabi/lib/pkgconfig:/usr/arm-linux-gnueabi/share/pkgconfig:/usr/share/pkgconfig -aarmel -b -uc -us -tc
dpkg-architecture: warning: Specified GNU system type arm-linux-gnueabi does not match gcc system type x86_64-linux-gnu.
dpkg-architecture: warning: Specified GNU system type arm-linux-gnueabi does not match gcc system type x86_64-linux-gnu.
 dpkg-buildpackage -rfakeroot -d -us -uc -aarmel -b -tc
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor):
dpkg-buildpackage: export CXXFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export FFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export LDFLAGS from dpkg-buildflags (origin: vendor): -Wl,-Bsymbolic-functions
dpkg-buildpackage: source package qemu-linaro
dpkg-buildpackage: source version 0.14.50-2011.03-1-0ubuntu1
dpkg-buildpackage: source changed by Steve Langasek <email address hidden>
dpkg-architecture: warning: Specified GNU system type arm-linux-gnueabi does not match gcc system type x86_64-linux-gnu.
 dpkg-source --before-build qemu-linaro
dpkg-buildpackage: host architecture armel
...

On Fri, Mar 11, 2011 at 12:18:29AM -0000, Wookey wrote:
> #./xdeb.py -a armel -d xdeb-scratch -b xdeb-scratch --only-explicit --prefer-apt qemu-linaro
> Building source package cache ...
> Using file archive.ubuntu.com_ubuntu_dists_natty_main_source_Sources for apt cache
> Using file archive.ubuntu.com_ubuntu_dists_natty_universe_source_Sources for apt cache
> Building working tree cache ...
> Build sequence: keyutils pango1.0 libxtst gdbm libsndfile libxdmcp atk1.0 fontconfig gdk-pixbuf pixman freetype libidn qt-x11-free libdrm brltty pcre3 util-linux nspr dbus libxcomposite libvorbis libsm avahi x11proto-dri2 cairo udev bluez tcp-wrappers libjpeg6b libxau nss x11proto-fixes libxi libgpg-error libice gnutls26 libxt libpng freetds libxext qt4-x11 python2.7 x11proto-core libsdl1.2 xft postgresql-8.4 openssl libogg libxxf86vm tiff libxcursor vde2 libxcb libmng cups libgcrypt11 db4.8 libxinerama libxrandr libxmu libdatrie ncurses flac xcb-util x11proto-gl zlib libjpeg8 libxrender libtasn1-3 libselinux lcms krb5 sqlite cyrus-sasl2 expat pulseaudio gtk+2.0 libpthread-stubs curl alsa-lib x11proto-xext nas sqlite3 libxdamage qt-assistant-compat unixodbc glib2.0 openldap libtool mesa libxfixes x11proto-record qtwebkit-source x11proto-render libx11 libiodbc2 e2fsprogs libthai mysql-5.1 libsepol qemu-linaro*
> Installing native packages: []

This is very different from what I get; this shows that it thinks
qemu-linaro is the only package that needs processing, whereas mine shows
that all the packages need processing (for cross-conversion, if nothing
else). Does your xdeb-scratch directory have existing builds?

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

No that scratch directory was deliberately blanked. Not sure what was going on there. I've reproduced the problem now. Here is a rather simpler test-case:
After downloading libidn11 packages
wget -N http://ports.ubuntu.com/ubuntu-ports/pool/universe/libi/libidn/idn_1.18-1build1_armel.deb
wget -N http://ports.ubuntu.com/ubuntu-ports/pool/main/libi/libidn/libidn11-dev_1.18-1build1_armel.deb
wget -N http://ports.ubuntu.com/ubuntu-ports/pool/main/libi/libidn/libidn11_1.18-1build1_armel.deb
wget -N http://ports.ubuntu.com/ubuntu-ports/pool/universe/libi/libidn/libidn11-java_1.18-1build1_all.deb

dpkg-cross -a armel -b -X install-info -X dpkg -X pkg-config libidn11-dev_1.18-1build1_armel.deb libidn11_1.18-1build1_armel.deb libidn11-java_1.18-1build1_all.deb
is run and fails with dpkg-cross: package libidn11-java doesn't provide any useful files. Skipping.

So the real problem here is that xdeb thought libidn11-java was a crossable package, downloaded it and tried to cross it. The question is should we fix this by telling xdeb that it's not meaningfully crossable, or by having it automatically skip over such failures and press on in the hope that it won't matter?

On the one hand the way xdeb works it expects to be able to identify crossable packages from it's heuristics+black/whitelists - doing this lets it remove them from the cross-dependency tree in the -cross packages. On the other hand otherwise-identified, but actually cross-null packages are harmless. If we decided that such packages were not an error, then we'd need to turn on the '--convert-anyway' flag so that the empty packages were there to satisfy the dependencies on them which have not been pruned. (this is how xapt works - just do the whole damn lot). Apart from polluting the package name space with a lot of -cross packages it doesn't do much harm, but is best discouraged outside a chroot becuase it'll make a mess eventually.

On Mon, Apr 18, 2011 at 03:48:06PM -0000, Wookey wrote:
> So the real problem here is that xdeb thought libidn11-java was a
> crossable package, downloaded it and tried to cross it. The question is
> should we fix this by telling xdeb that it's not meaningfully crossable,
> or by having it automatically skip over such failures and press on in
> the hope that it won't matter?

I don't think we want to do the latter. I was thinking more in terms of
trapping the error, taking dpkg-cross's advice that this is not a crossable
package, and rerunning the dpkg-cross with that package both excluded from
the crossing list and added to the list of dependency exclusions.

> On the one hand the way xdeb works it expects to be able to identify
> crossable packages from it's heuristics+black/whitelists - doing this
> lets it remove them from the cross-dependency tree in the -cross
> packages. On the other hand otherwise-identified, but actually cross-
> null packages are harmless. If we decided that such packages were not an
> error, then we'd need to turn on the '--convert-anyway' flag so that the
> empty packages were there to satisfy the dependencies on them which have
> not been pruned. (this is how xapt works - just do the whole damn lot).

Right, force-crossing the packages might also be ok. I'm pretty sure the
one thing we *don't* want to do is to just ignore the xdeb non-zero exit,
since that could mean some of the -cross packages that *have* been generated
will be uninstallable due to non-existent deps.

Given the choice between force-crossing empty packages, and trapping and
re-crossing a reduced list of binaries, I have no opinion. The former would
be easier to implement; do you see any significant reason not to do this?

> Apart from polluting the package name space with a lot of -cross
> packages it doesn't do much harm, but is best discouraged outside a
> chroot becuase it'll make a mess eventually.

Right; I wouldn't expect xdeb to be run outside a dedicated development
environment anyway.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

This bug was fixed in the package xdeb - 0.6.5

---------------
xdeb (0.6.5) unstable; urgency=low

  [ Steve Langasek ]
  * If a package is Multi-Arch: foreign, we don't need to build it unless we
    also want to install it in its own right.
  * When calling apt-cache show, qualify the package name with an
    explicit architecture; this guards against attempts to import
    not-for-us binaries that are in the apt cache in a multiarch
    environment. LP: #752287.

  [ Wookey ]
  * Add some packages to black/whitelists
  * Improve graphing to only include binary deps that are
    actually depended on in the dependency tree
  * Update Standards-version
  * Depend on multiarch-capable dpkg-cross and invoke it with
    --convert-multiarch
  * Fix lintian test whinge
  * Don't fail if null dpkg-cross packages are found LP: #731079
  * Include lib64c in toolchain packages list - avoids
     multipleprovides exception on 64-bit arches. LP: #75574
  * Let xdeb grok binary package names as well as source ones. LP: #778506
  * Use 'apt-get download' rather than wget for native imports, provided
    that multiarch is configured. LP: #851427

  [ Steve McIntyre ]
  * Add initial support for armhf. LP: #772526

  [ Colin Watson ]
  * Convert to dh_python2.
  * Consistently use 'import utils' rather than 'from utils import ...'.
  * Suppress unusednames=_NATIVE_IMPORT_SOURCE_NAME in config.py, apparently
    due to a pychecker bug.
  * Split option parsing into a separate function to placate a pychecker
    warning about the length of main.
  * Extend need_loop_break hack to cover python2.7's build-dependency on
    libbluetooth-dev.
  * Add python2.7-dev to whitelist.
  * Fix installation of binutils-multiarch when cross-compiling.
  * Tolerate the removal of the fields collection in Lintian 2.5.2.
  * Don't install build-dependencies in --sequence mode.
  * Fix caching in Provides resolver.
  * Only skip Multi-Arch: foreign packages if the system is configured to be
    able to install packages of the appropriate foreign architecture.
  * Revamp build-dependency installation. We now point 'apt-get install' at
    a combination of the system sources.list and the destination directory
    rather than using 'dpkg -i', and we install build-dependencies
    immediately before building each package as well as in a block at the
    start of the run; this also allows us to be more selective about which
    crossed packages we install. The 'builddep_whitelist' configuration
    option is now unnecessary and has been removed.
  * Fix dependency resolver. We now only expand dependencies for binaries
    that are explicitly required to satisfy (build-)dependencies, rather
    than trying to expand all binaries in every source package we encounter.
  * If rebuilding a source package that had already been built, force a full
    regeneration of the apt repository and purge any previous incarnations
    of those binary packages.
  * Fix native_import to work again even if multiarch isn't configured.
  * Remove explicit code to install native imports; the new build-dependency
    installer can deal with that by itself.
  * Remove libreadline-dev from cross_...