d-i can't install against an https mirror

Status changed to 'Confirmed' because the bug affects multiple users.

I'm separating this out again because installing from an HTTPS mirror requires more work than handling preseeding from HTTPS. This does depend on bug 833994 and bug 1172101, but it isn't really a duplicate of either.

choose-mirror is just one of the components that will need work, but it'll do for a start.

I looked into the debootstrap side of this, and thought about having debootstrap install apt-transport-https if installing from an HTTPS mirror. (At the moment it observes that apt can't handle HTTPS by itself and configures the end result to use an HTTP mirror, although it does manage to do the actual bootstrap from HTTPS.)

The tricky bit here is that we'd also need to copy in any relevant certificates. With the work I've been doing recently (https://lists.debian.org/debian-boot/2014/02/msg00084.html etc.), it's possible to build d-i with the necessary certificates for your local mirror, so they'll be in /usr/lib/ssl/certs/; we could just copy the contents of that directory across. On the other hand, if you're debootstrapping from a full Unix system which just happens not to have the Debian keyring installed, there might be quite a lot of certificates in that directory, and we can't tell which is used.

To avoid this problem, I think it's best to handle the installation of apt-transport-https and the certificate copying in base-installer instead. That way we know it's d-i-specific and that any certificates we find are ones that the person who prepared the installation medium explicitly wanted.

Ah, it's not quite that simple. We need to install apt-transport-https in debootstrap, since otherwise we don't have the apt configuration we need to do so until we've installed apt-transport-https!

This bug was fixed in the package debootstrap - 1.0.58

---------------
debootstrap (1.0.58) unstable; urgency=medium

  * Policy version 3.9.5: no changes required.
  * Install apt-transport-https if installing from an HTTPS mirror
    (LP: #1135163). It may still be necessary to copy certificates into
    place, but there's at least a reasonable chance that somebody installing
    from HTTPS may want to keep using it, and we have to install
    apt-transport-https at this point otherwise they won't be able to do
    that end-to-end.

 -- Colin Watson <email address hidden> Tue, 11 Feb 2014 17:46:41 +0000

apt-setup needs some work to handle the proxy correctly and fix up details of timeout handling.

This bug was fixed in the package choose-mirror - 2.55ubuntu1

---------------
choose-mirror (2.55ubuntu1) trusty; urgency=medium

  * Resynchronise with Debian. Remaining changes:
    - Use Ubuntu mirrors.
    - Ubuntu branding.
    - Set default country to GB.
    - Never ask mirror/suite, mark it untranslatable, and give it
      appropriate choices for Ubuntu.
    - Pick CC.archive.ubuntu.com (for the country selected in the installer)
      as the default mirror. For ports architectures (armel, powerpc) use
      ports.ubuntu.com instead.
    - Drop the priorities of the country/mirror questions to medium if we're
      installing from a CD that includes the base system.
    - Support setting OVERRIDE_BASE_INSTALLABLE in the environment to force
      choose-mirror to assume that the base system is installable.
    - Skip mirror validation if the base system is installable and we're
      installing from a mirror in the masterlist.
    - Don't check if archive.ubuntu.com is reachable before asking for a
      proxy.
    - Allow preseeding the default CC.archive.ubuntu.com setup by preseeding
      mirror/http/mirror to CC.archive.ubuntu.com, and likewise for
      mirror/ftp/mirror.
    - Force xgettext to use UTF-8 encoding when generating templates files,
      to cope with Côte d'Ivoire.
    - Only default mirror/country to the value of debian-installer/country
      if the latter has any mirrors; otherwise, fall through to defaulting
      to GB.
    - Exclude all-countries mirrors such as ports.ubuntu.com when
      determining whether there is a mirror in the country specified in
      debian-installer/country.
    - If the selected country has no mirror (excluding all-countries
      mirrors), then fall back to the default.

choose-mirror (2.55) unstable; urgency=medium

  * Add HTTPS support (LP: #1135163). This is enabled by default here as it
    isn't unreasonably large, but won't be usable unless GNU wget has been
    added to the initrd.
  * Update Mirrors.masterlist.

choose-mirror (2.54) unstable; urgency=low

  * Update Mirrors.masterlist
  * Add myself to Uploaders.

choose-mirror (2.53) unstable; urgency=low

  [ Cyril Brulebois ]
  * Update Mirrors.masterlist

  [ Updated translations ]
  * Bosnian (bs.po) by Amila Valjevčić
  * Hungarian (hu.po) by Judit Gyimesi

choose-mirror (2.52) unstable; urgency=low

  [ Colin Watson ]
  * Make check-masterlist a no-op target if MIRRORLISTURL is undefined.

  [ Updated translations ]
  * Tajik (tg.po) by Victor Ibragimov
 -- Colin Watson <email address hidden> Wed, 12 Feb 2014 15:14:37 +0000

I've found one further issue, and a somewhat thorny one. Where we resolve it probably depends on upstream response. It doesn't affect the "ignore SSL certificates" case, so you may not be so worried about this; but I do think we need to sort it out.

apt-transport-https uses libcurl to download from HTTPS. This works fine if you're using a mirror whose root of trust is in ca-certificates, or if you've disabled CA certificate checking. However, we're specifically using libcurl3-gnutls, and cURL's GnuTLS backend only supports using a CA bundle, not a directory. As a result it is awkward to copy certificates from d-i to the target system.

The options I can think of at the moment are:

 * add CA directory support to cURL's GnuTLS backend (which would make some sense given that wget supports this with GnuTLS)
 * have apt-transport-https scan the CA directory itself manually
 * integrate with ca-certificates to merge the certificates provided to d-i into the bundle (this might be a nice thing to do anyway, since it would place the installer-provided certificates under /usr/local/)

I've added a curl task to this bug to represent the first of these options, but I'll need to think about this a bit and perhaps discuss it with some other people.

OK, it turns out I can handle this quite easily by installing ca-certificates, copying into /usr/local/share/ca-certificates/, and running update-ca-certificates from base-installer; so I'll drop the curl task as unnecessary.

This bug was fixed in the package base-installer - 1.143ubuntu1

---------------
base-installer (1.143ubuntu1) trusty; urgency=medium

  [ Colin Watson ]
  * Reverse lots of our delta against Debian relating to architectures no
    longer supported in Ubuntu (armel, hppa, ia64, sparc), to make merging
    from Debian less painful.
  * Resynchronise with Debian. Remaining changes:
    - Use and depend on the Ubuntu keyring.
    - Enable the restricted component by default, unless
      apt-setup/restricted is preseeded to false.
    - Set up the default sources.list to look in -updates and -security (the
      latter from apt-setup/security_host and apt-setup/security_path) as
      well as the unadorned suite; also -proposed if apt-setup/proposed is
      true.
    - Use Ubuntu kernel image names.
    - Allow preseeding base-installer/kernel/override-image to force a given
      kernel to be used.
    - Install busybox-initramfs rather than busybox.
    - Revert Joey's patch to call base-installer.d hooks after running
      debootstrap, which broke console-setup's expectation of being able to
      insert its configuration file into /target before console-setup is
      installed by debootstrap.
    - Add armhf/keystone, armhf/omap, and armhf/omap4 subarchitectures.
    - Install kernel headers to match the kernel. This may be overridden by
      setting base-installer/kernel/headers to false.
    - Add base-installer/kernel/backports-modules template, which may be
      preseeded to install selected linux-backports-modules-* packages.
    - Move some kernel installation support code from bootstrap-base to
      base-installer, and allow live-installer to override the title
      template used by that.
    - Add powerpc/e500 and powerpc/e500mc subarchitectures.
    - Add overlay archive support.
    - Add ppc64el architecture.
  * Remove code for some old Ubuntu i386 kernel names that no longer exist.

  [ dann frazier ]
  * Add arm64 kernel detection (LP: #1263756).

base-installer (1.143) unstable; urgency=medium

  * Add HTTPS support: pass --no-check-certificate to debootstrap if
    debian-installer/allow_unauthenticated_ssl=true, set https_proxy if
    necessary, and copy any certificates that exist in d-i into the target
    system (LP: #1135163).

base-installer (1.142) unstable; urgency=medium

  * pkgdetails.c: Only interpret percentages following whitespace, to cope
    with GNU wget outputting the local file name (which may contain "%" due
    to URL-encoding) after it finishes the download (LP: #1172101).

base-installer (1.141) unstable; urgency=low

  [ Ian Campbell ]
  * armhf: Support armmp kernel flavour on mx5 and express and as default on
    generic platforms.
  * armhf: Select armmp-lpae kernel flavour on generic platforms when the
    hardware feature is available.

base-installer (1.140) unstable; urgency=medium

  [ Aurelien Jarno ]
  * mips: fix octeon cpuinfo.

base-installer (1.139) unstable; urgency=medium

  * mips: Drop support for non-supported architectures.
  * mips: Add support for octeon kernels.
  * mipsel: Drop support for non-supported architectures
  * mipsel: Add support for loongson-3a kernels.

bas...

This bug was fixed in the package apt-setup - 1:0.80ubuntu6

---------------
apt-setup (1:0.80ubuntu6) trusty; urgency=medium

  * Cherry-pick from trunk:
    - Handle proxy and timeout configuration appropriately when installing
      over HTTPS (LP: #1135163).
 -- Colin Watson <email address hidden> Fri, 14 Feb 2014 15:12:46 +0000

Colin, thank you very much for your work on this issue and I'm currently testing https installs in our environment now. I had a question on how to properly set the installer to use a https mirror. The installer PXE configuration line I'm using is:

KERNEL pxelinux.cfg/kernel
APPEND append initrd=pxelinux.cfg/initrd.img vga=normal fb=false auto=true priority=critical auto-install/enable=true debian-installer/allow_unauthenticated_ssl=true url=https://buildhost/trusty/preseed.cfg.pl media=https://buildhost/Ubuntu-14.04-amd64/

These are the stock kernel and initrd available for Ubuntu 14.04, and it's hitting a nightly sync'd mirror that's available on https://buildhost/Ubuntu-14.04-amd64/. This site has a self-signed certificate, which debian-installer/allow_unauthenticated_ssl=true allows us to download and use the preseed.cfg.pl over https.

The base-install of packages still uses http however, which I can see by looking at the web server logs. I've tried the following based off of your Debian list emails (https://lists.debian.org/debian-boot/2014/02/msg00084.html):

1. Adding mirror/protocol=https in the append line, this results in the install prompting for a mirror
2. Adding debian-installer/mirror/protocol=https to the append line, this installs, but still goes over http
3. Adding d-i mirror/protocol string https in the preseed file, this has the same result as 1

I tried digging through the documentation for base-installer, debootstrap, and debian-installer, but other than the emails on the Debian list and change log entries, I couldn't find the exact syntax or place to tell the installer to use https for our custom mirror.

This is possible, and if so could you provide the needed syntax in order for packages to go over https as well? Thank you.

Try the following:

d-i mirror/country string manual
d-i mirror/protocol string https
d-i mirror/https/hostname string buildhost
d-i mirror/https/directory string /Ubuntu-14.04-amd64

Thank you Philipp, with mirror/https settings everything is now fully going over https.

Here's our working configuration with unauthenticated https installs for 14.04 in case anyone else comes across this:

PXE Config:
KERNEL pxelinux.cfg/kernel
APPEND append initrd=pxelinux.cfg/initrd.img vga=normal fb=false auto=true priority=critical auto-install/enable=true debian-installer/allow_unauthenticated_ssl=true url=https://buildhost/trusty/preseed.cfg.pl media=https://buildhost/ubuntu/

Preseed Mirror settings:
d-i mirror/protocol string https
d-i mirror/country string manual
d-i mirror/https/hostname string buildhost
d-i mirror/https/directory string /ubuntu/
d-i mirror/https/proxy string
d-i mirror/https/mirror select mirror

Hello Thomas, or anyone else affected,

Accepted base-installer into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/base-installer/1.122ubuntu7.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Thomas, or anyone else affected,

Accepted debootstrap into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/debootstrap/1.0.40~ubuntu0.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Thomas, or anyone else affected,

Accepted apt-setup into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/apt-setup/1:0.55ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Thomas, or anyone else affected,

Accepted choose-mirror into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/choose-mirror/2.39ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

It seems that an installer is in -proposed as well: 20101020ubuntu136.19 -- I'll test it next week with our preseeds. Thanks!

The kernel situation with proposed gave me some grief. Hence my testing last week did not work out. I did a clean installation via an nginx reverse proxy and a custom CA (which is *not* the setup at Google, but I had to fudge something for verification instead of testing against our own archive) and it worked right until after debootstrap. The cert was not copied in the chroot. On the other hand, in the real setup ca-certificates as found in Ubuntu is enough to verify certs. I'm not sure if trusty fails the same way, but it might. So this can be marked verified from our side but there's a caveat for other users. (But this is clearly not a regression over precise as it is now.)

This bug was fixed in the package apt-setup - 1:0.55ubuntu4.1

---------------
apt-setup (1:0.55ubuntu4.1) precise; urgency=medium

  * Handle proxy and timeout configuration appropriately when installing
    over HTTPS (LP: #1135163).
 -- Colin Watson <email address hidden> Mon, 23 Jun 2014 17:13:49 +0100

The verification of the Stable Release Update for apt-setup has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package base-installer - 1.122ubuntu7.4

---------------
base-installer (1.122ubuntu7.4) precise; urgency=medium

  * pkgdetails.c: Only interpret percentages following whitespace, to cope
    with GNU wget outputting the local file name (which may contain "%" due
    to URL-encoding) after it finishes the download (LP: #1172101).
  * Add HTTPS support: pass --no-check-certificate to debootstrap if
    debian-installer/allow_unauthenticated_ssl=true, set https_proxy if
    necessary, and copy any certificates that exist in d-i into the target
    system (LP: #1135163).
 -- Colin Watson <email address hidden> Mon, 23 Jun 2014 16:13:11 +0100

This bug was fixed in the package debootstrap - 1.0.40~ubuntu0.6

---------------
debootstrap (1.0.40~ubuntu0.6) precise; urgency=medium

  * pkgdetails_perl: Only interpret percentages following whitespace, to
    cope with GNU wget outputting the local file name (which may contain "%"
    due to URL-encoding) after it finishes the download (LP: #1172101).
  * Install apt-transport-https and ca-certificates if installing from an
    HTTPS mirror (LP: #1135163).
 -- Colin Watson <email address hidden> Mon, 23 Jun 2014 16:40:02 +0100

This bug was fixed in the package choose-mirror - 2.39ubuntu4.1

---------------
choose-mirror (2.39ubuntu4.1) precise; urgency=medium

  * Add HTTPS support (LP: #1135163). This is enabled by default here as it
    isn't unreasonably large, but won't be usable unless GNU wget has been
    added to the initrd.
 -- Colin Watson <email address hidden> Mon, 23 Jun 2014 16:54:16 +0100