multistrap needs to be updated for new apt and cross-tools in main

It would be nice to remove the --force-yes stuff and more correctly deal with archive-keyrings in chroot configs, but right now I don;t know how to make that work (see https://bugs.launchpad.net/ubuntu/+source/apt/+bug/646499 )

as commented in bug #645147, I'm not going to upload anything to the Ubuntu archive that forcibly bypasses apt's gpg checking. I'll have a look at the apt bug.

 [UbuntuToolchains]
-packages=g++-4.4-arm-linux-gnueabi linux-libc-dev-armel-cross binutils-arm-linux-gnueabi
-source=http://archive.ubuntu.com/ubuntu
+packages=gcc-arm-linux-gnueabi libc-dev-armel-cross linux-libc-dev-armel-cross
+source=[arch=i386,amd64] http://archive.ubuntu.com/ubuntu
 suite=maverick
 components=main universe
 keyring=ubuntu-keyring

Why are you enabling multiarch sources here? Unless otherwise annotated, any sources.list entry is native-arch-only, which is what we want here; this syntax will cause pointless downloads of Packages files on apt-get update that correspond to neither the host nor the build arch.

oh, why are we switching from g++-4.4-arm-linux-gnueabi to *gcc*-arm-linux-gnueabi? The latter doesn't pull in g++, does it?

Attached is my proposed debdiff for this, fixing multistrap to use the host keyring instead of turning off gpg checking. This is sufficient for running multistrap with the ubuntu/armel profile from an existing Ubuntu system (i.e., a system that has the Ubuntu keyring already in trusted.gpg). It does not solve the problem of bootstrapping a chroot from repositories with whose signing key you don't already have a trust relationship (e.g., bootstrapping an Ubuntu chroot from a Debian system; or bootstrapping from a PPA), so a more general solution is probably wanted here, but I think that shouldn't block us from getting a fix in for the common case.

This also fixes the s/gcc/g++/ nit mentioned in the previous comment.

I've kept in the added 'reinstall=binutils-multiarch' line, but you don't mention in the changelog why this change is needed - why does this package need reinstallation?

The binutils-multiarch re-install line is there because it is the fairly crufty way we deal with diversions.
Nothing guarantees correct ordering (pre-depends ignored) so diverts may happen before the thing they direct are present. Simplest fix is to re-install any package doing diversions at the end. In a base cross-chroot that's just binutils-multiarch.
Declarative diverts would let us deal with this without explicit config.

The attached patch is pretty-much the same as yours except that it also ensures --force-yes is correctly used for the re-install option if noauth is set.

On Fri, Oct 01, 2010 at 02:05:17AM -0000, Wookey wrote:
> The binutils-multiarch re-install line is there because it is the fairly
> crufty way we deal with diversions. Nothing guarantees correct ordering
> (pre-depends ignored) so diverts may happen before the thing they direct
> are present. Simplest fix is to re-install any package doing diversions
> at the end. In a base cross-chroot that's just binutils-multiarch.
> Declarative diverts would let us deal with this without explicit config.

Well, then you're working around a bug that does not exist. It's perfectly
allowed to divert a file before that file exists, and dpkg will do the right
thing on unpack. So I'm going to drop this part of the patch for upload.

> The attached patch is pretty-much the same as yours except that it also
> ensures --force-yes is correctly used for the re-install option if
> noauth is set.

Right, good point. noauth should never be used, but if someone's going to
use it, it should work...

Uploaded to the maverick queue, thanks!

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

This bug was fixed in the package multistrap - 2.1.6ubuntu3

---------------
multistrap (2.1.6ubuntu3) maverick; urgency=low

  * cross/ubuntu/armel.conf: Set config to use maverick now that all the
    packages are in the archive. LP: #646901
  * Always use the host apt keyring when bootstrapping, since we don't
    have one yet in the target root.

  [ Wookey <email address hidden> ]
  * Point UbuntuToolchains config at the gcc-defaults metapackage instead of
    at g++-4.4-arm-linux-gnueabi.
 -- Steve Langasek <email address hidden> Thu, 30 Sep 2010 19:26:29 +0000

The re-install issue is that we can't run pre-insts when doing the initial unpacking to make a system with multistrap (because multistrap is designed to work for cross rootfs creation). We have to use dpkg -X because dpkg --unpack doesn't have a 'don't run preinsts' option.

The only useful thing most preinsts do is run dpkg-diverts, so the re-install option exists to allow pre-insts to be run later once there is enough system to run them on and we are running on the right architecture. In practice this seems to work well. For cross-chroots - until we are happy that we can leave binutils-multiarch out, I think the reinstall option should stay.

Still, as we are trying to get rid of it maybe it doesn't matter that much - but it'll be confusing if the package is present but in fact its diverts aren't. I guess I can run a set of builds now and find out how much it really breaks

On Fri, Oct 01, 2010 at 02:10:58PM -0000, Wookey wrote:
> The re-install issue is that we can't run pre-insts when doing the
> initial unpacking to make a system with multistrap (because multistrap
> is designed to work for cross rootfs creation). We have to use dpkg -X
> because dpkg --unpack doesn't have a 'don't run preinsts' option.

Ah. But in that case, because the unpack ordering is not guaranteed, you
would need to reinstall *both* binutils and binutils-multiarch, because the
unpacked files could be from either package. You don't want to end up with
/usr/bin/objdump.single identical to /usr/bin/objdump!

> The only useful thing most preinsts do is run dpkg-diverts, so the re-
> install option exists to allow pre-insts to be run later once there is
> enough system to run them on and we are running on the right
> architecture. In practice this seems to work well. For cross-chroots -
> until we are happy that we can leave binutils-multiarch out, I think the
> reinstall option should stay.

Understanding now what this is meant to do, I agree that this should be kept
for the time being, though it should also have 'binutils' added to the list
of reinstalled packages if you want it to be reliable. Do you think another
upload is needed to get this into 10.10?

Long-term, I think multistrap shouldn't be ignoring the distinctions between
essential/required packages and other packages. The only packages you
should need to dpkg -X are these; a three-pass dpkg -X, dpkg --unpack, dpkg
--configure of the base system, followed by plain installation of any
additional packages, would be reliable without any need to hard-code lists
of packages that contain preinsts.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

On Fri, 01 Oct 2010 18:41:03 -0000
Steve Langasek <email address hidden> wrote:

> Ah. But in that case, because the unpack ordering is not guaranteed,
> you would need to reinstall *both* binutils and binutils-multiarch,

The unpack ordering *could* be guaranteed as multistrap handles each
package individually. It is conceivable that a method can be
implemented to specify such a sequence - probably as a config option.
e.g. the *order* of packages listed in "reinstall=..." could be
rendered significant.

> because the unpacked files could be from either package. You don't
> want to end up with /usr/bin/objdump.single identical
> to /usr/bin/objdump!

That isn't that much of an issue specifically for binutils - nothing
actually calls the .single variants - for binutils* itself, diverts
could mostly be replaced by Replaces: without problems. The issue of
preinst scripts and foreign chroots is wider than just binutils though.

> Long-term, I think multistrap shouldn't be ignoring the distinctions
> between essential/required packages and other packages.

Sorry, that forces multistrap backwards into the debootstrap model and
that just isn't useful. Multistrap needs to work principally for foreign
chroots - it's use with pdebuild-cross for cross-building chroots and
other "native" chroots is secondary to it's main purpose. In a foreign
environment, there can be no useful division without leaving hundreds
of megabytes of .deb files in /var/cache/apt/archives/ completely
untouched or partially unpacked and retained anyway (so that the .deb
can be reinstalled or replaced files recovered), wasting space in the
unconfigured rootfs. That makes it even slower to unpack the
unnecessarily large rootfs and then install most of the packages when
all that is actually needed is to configure the unpacked packages with
nothing at all in /var/cache/apt/archives.

> The only
> packages you should need to dpkg -X are these; a three-pass dpkg -X,
> dpkg --unpack, dpkg
> --configure of the base system, followed by plain installation of any
> additional packages, would be reliable without any need to hard-code
> lists of packages that contain preinsts.

That is a backwards step when all that is actually needed is a sensible
way of unpacking foreign .deb's without running preinsts. dpkg-divert
is the biggest issue with doing that because files get replaced if
the .deb is extracted before the preinst can run.

Therefore, some way of manipulating diversions in a changed root
directory / path is sorely needed. Multistrap can easily identify which
packages contain the relevant snippets in their preinst scripts and
then store that information. The step I haven't had time to investigate
(because I'm moving house soon) is how to modify the files
in /path/to/var/lib/dpkg/ and elsewhere such that it looks to dpkg etc.
as if the preinst has successfully completed before that particular .deb
is extracted. All such manipulations need to be done using the tools
running natively outside the foreign chroot.

The other roles performed by preinst scripts may also need to have
implementations that can be executed from *above* the root directory of
the final system and by nat...