Merge ~litios/ubuntu-security-tools:build-sources-list/use-auth-conf into ubuntu-security-tools:master

[David Fernandez Gonzalez]

Build-sources-list currently embeds the credentials for external PPA in the sources list file. This is deprecated in APT and we should use auth conf files.

This PR addresses this issue by creating the proper auth conf file for the ESM PPAs.

Also, note that sec-buildenv will start relying on this soon: https://github.com/canonical/sec-buildenv/issues/91

Another change is added to the PR. For customer PPA / ESM purposes, we also need -updates to be added to the sources lists. See https://code.launchpad.net/~litios/ubuntu-security-tools/+git/ubuntu-security-tools/+ref/ppas-tag-migration for handling the different pockets that will now coexist (MR tbd)

Testing: https://pastebin.canonical.com/p/B6FxRCF9pd/

[Alex Murray (alexmurray)]

This is a great improvement but I have concerns about having it silently write the the auth conf file behind the scenes via sudo - also to support umt download the auth conf file needs to have world readable permissions as per https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1904068

[Alex Murray (alexmurray)]

review: Needs Fixing

[David Fernandez Gonzalez (litios)]

Hey Alex, thanks for the review!

I refactored it to be a new file and added an echo statement in the original file for people to know they should run it too to get the credentials.

Regarding the permission issue, having the group as sudo for the file should allow the main user and any other user in the sudo group to be able to read the file. It's the same as we are doing for the files in sources.list.d too. https://pastebin.canonical.com/p/3CXGFDPzqV/.

It makes it work for umt while not setting world-read permission. But let me know if you find an issue with this and I can revert it to world-read.

[Spyros Seimenis (sespiros)]

Great improvement idd although the more I edit build-sources-list the more I feel it needs a bigger revamp.

In sec-buildenv I create esm-ppa.conf with permissions "640 root sudo" and I haven't encountered issues.

I also think it's a bit awkward to run sudo in the (now) new script but that's exactly what build-sources-list.install does at the moment.

I suggest having 3 scripts in total so that we also avoid duplicating things:

1) Have the new esm-auth-conf script simply output the right auth configuration and rename it to setup-esm-auth-conf

2) Have build-sources-list.install run both build-sources-list and esm-auth-conf and then install both source list and auth file. Maybe also rename it to something like setup-sources-list.

3) Remove sudo from build-sources-list.install, let if produce an error if it is not run with sudo

We would also need to update sec-buildenv and the wiki.

+ minor comments

review: Needs Fixing

[Steve Beattie (sbeattie)]

On Wed, Oct 18, 2023 at 12:00:43PM -0000, Spyros Seimenis wrote:
> 3) Remove sudo from build-sources-list.install, let if produce an error if it is not run with sudo

One challenge here is that the auth credentials are queried
from launchpad through an authenticated connection via
launchpadlib. Depending on what the user has done before this and/or
how sudo is invoked, it could end up with either a cached credential
in the /root/ homedir *or* root owned files in the user's directory.

I did not like embedding sudo in the original script, but trying to
navigate things that need to happen as the user and things that need
to happen as root was tricky, which is the current compromise is the
way it is.

One comment I'd also make is that when I thought about doing this,
my intent was to create an auth.conf.d file per ppa.

As to blatting over the existing one, I'm not sure the ppa
subscription credentials ever change -- maybe if you unsubscribe and
then resubscribe to the ppa -- but if they did, I'd want to think
carefully about what situations the existing file should or should
not get overwritten.

+1 on adding the esm-*-updates ppas by default.

--
Steve Beattie
<email address hidden>

[David Fernandez Gonzalez (litios)]

Hi everyone, thanks for the reviews.
I've made several changes based on the feedback:

----

> 1) Have the new esm-auth-conf script simply output the right auth configuration.

  - Updated!

> 2) Have build-sources-list.install run both build-sources-list and esm-auth-conf and then install both source list and auth file. Maybe also rename it to something like setup-sources-list.

  - Related to 1), now the installation is handled by setup-sources-list.install.

> 3) One comment I'd also make is that when I thought about doing this, my intent was to create an auth.conf.d file per ppa.

  - Applied too! This was an interesting combination with 2) so the bash to do so is not the easiest. It works though. One thing is that I'm using the auth.conf.d itself with a sudo tmp directory so everything stays "secure" and no race condition is present.

----

I did not rename the script to setup-esm-auth-conf as I think the word `setup` means we are actually installing something (as the other script called setup-sources-list)

In regards to:

> 3) Remove sudo from build-sources-list.install, let it produce an error if it is not run with sudo

I agree we can have a discussion for this but it seems like an issue for another PR. We need the -updates set up ASAP for OVAL and I feel like also dealing with the sudo issue in this PR may things take longer for this particular change.

In regards to:

> As to blatting over the existing one, I'm not sure the ppa
subscription credentials ever change -- maybe if you unsubscribe and
then resubscribe to the ppa -- but if they did, I'd want to think
carefully about what situations the existing file should or should
not get overwritten.

If it doesn't change, overriding it wouldn't make a difference. If it does change, that means the old credentials are no longer valid (I assume) and therefore there is no use in keeping it, overriding makes sense to me for that particular case.

Let me know of any improvements!

[Spyros Seimenis (sespiros)]

review: Approve

[Alex Murray (alexmurray)]

LGTM - the one thing perhaps would be to have build-sources-list have a comment or even output a message saying that in general it shouldn't be run directly now and instead to run setup-sources-list.install - but that is fine to leave for later.

review: Approve

[David Fernandez Gonzalez (litios)]

Added to the help of both scripts, as the output is used to create the files. Thanks for all the reviews :)

diff --git a/build-tools/build-sources-list b/build-tools/build-sources-list
index 97c8e0a..73d4c28 100755
--- a/build-tools/build-sources-list
+++ b/build-tools/build-sources-list
@@ -24,7 +24,8 @@ debmirror, you might want to specify '--add-fallbacks' to this script, which
 will add the official Ubuntu archive after your mirrors. This will ensure you
 have the most up to date sources.
 
-Typical usage:
+Typical usage should be through the use of setup-sources-list.install.
+In case you want to run it yourself, the expected usage is:
 $ build-sources-list | sudo sh -c 'cat > /etc/apt/sources.list.d/ubuntu-security.list && \
                                    chgrp root.sudo /etc/apt/sources.list.d/ubuntu-security.list && \
                                    chmod 640 /etc/apt/sources.list.d/ubuntu-security.list'
@@ -63,34 +64,6 @@ deb-src http://ppa.launchpad.net/${team}/${ppa}/ubuntu ${release} main
 EOM
 }
 
-emit_private_ppa_overlay() {
-    local team=$1
-    local ppa=$2
-    local release=$3
-    local keyring="$4"
-    local url
-    local signed_by
-
-    url="$("$UST"/lp_tools/get_ppa_subscription_url.py "$team" "$ppa")"
-    if [ "$?" -ne 0 ] ; then
-        echo "failed to get subscription info for $team/$ppa" 1>&2
-        return
-    fi
-    if [ -n "$keyring" ] ; then
-        signed_by="[signed-by=$keyring]"
-    fi
-
-    cat <<EOM
-# Reference: https://launchpad.net/~${team}/+archive/ubuntu/$ppa
-#
-# this should be protected via mode 640 root:sudo
-# ideally, we'd separate out the auth bits into files in /etc/apt/auth.conf.d/
-deb-src $signed_by $url $release main
-
-EOM
-    echo "# IMPORTANT: if this is the first time you have configured ppa:$team/$ppa, you will need to import the ppa archive key into ${ESM_PPA_KEYRING}" 1>&2
-}
-
 cat <<EOM
 #
 # Output of 'build-sources-list' for ubuntu-security-tools
@@ -125,21 +98,28 @@ EOM
     # ESM overlay ppas
     elif echo "$r" | grep -q "/esm" ; then
         ppa_overlay_release=$(echo "$r" | cut -f 1 -d '/')
-        ppa_overlay="esm"
         if [ $ppa_overlay_release = "trusty" ]; then
-            ppa_overlay="esm-infra-security"
+            cat <<EOM
+deb-src [signed-by=${ESM_PPA_KEYRING}] https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-infra-security/ubuntu $ppa_overlay_release main
+deb-src [signed-by=${ESM_PPA_KEYRING}] https://private-ppa.launchpadcontent.net/ubuntu-esm/esm-infra-updates/ubuntu $ppa_overlay_release main
+EOM
+        else
+            cat <<EOM
+deb-src [signed-by=${ESM_PPA_KEYRING}] https://private-ppa.launchpadcontent.net/ubuntu-esm/esm/ubuntu $ppa_overlay_release main
+EOM
         fi
-
-        emit_private_ppa_overlay "ubuntu-esm" "$ppa_overlay" "${ppa_overlay_release}" "${ESM_PPA_KEYRING}"
         echo "# Requires gpg keyid DBB1FC89762BF6B96707C4059BC0A1A1622CF918 stored in ${ESM_PPA_KEYRING}"
     # newer ESM releases are in the esm-infra-security and
     # esm-apps-security ppas
     elif echo "$r" | grep -q "esm-" ; then
         ppa_overlay_release=$(echo "$r" | cut -f 2 -d '/')
         product=$(echo "$r" | cut -f 1 -d '/')
-        ppa_overlay="$product-security"
-
-        emit_private_ppa_overlay "ubuntu-esm" "$ppa_overlay" "${ppa_overlay_release}" "${ESM_PPA_KEYRING}"
+        for pocket in "security" "updates" ; do
+            ppa_overlay="$product-$pocket"
+            cat <<EOM
+deb-src [signed-by=${ESM_PPA_KEYRING}] https://private-ppa.launchpadcontent.net/ubuntu-esm/$ppa_overlay/ubuntu $ppa_overlay_release main
+EOM
+        done
         echo "# Requires gpg keyid DBB1FC89762BF6B96707C4059BC0A1A1622CF918 stored in ${ESM_PPA_KEYRING}"
     # normal ubuntu releases.
     else
diff --git a/build-tools/esm-auth-conf b/build-tools/esm-auth-conf
 new file mode 100755
index 0000000..8e9c154
--- /dev/null
+++ b/build-tools/esm-auth-conf
@@ -0,0 +1,41 @@
+#!/bin/bash
+#
+# Copyright (C) 2023 Canonical, Ltd.
+# Author: David Fernandez Gonzalez <[e-mail]>
+# License: GPLv3
+#
+
+set -e
+
+help() {
+    cat <<EOM
+Usage: esm-auth-conf
+
+This script will query Launchpad to obtaing the credentials needed
+by apt to query ESM PPAs and output this information to stdout.
+
+Typical usage should be through the use of setup-sources-list.install.
+EOM
+}
+
+add_esm_auth_conf() {
+   ppas=(
+        "esm-infra-updates"
+        "esm-infra-security"
+        "esm-apps-updates"
+        "esm-apps-security"
+        "esm"
+    )
+
+    for ppa in "${ppas[@]}"; do
+        echo "# Auth info for ${ppa}"
+        $UST/lp_tools/get_ppa_subscription_url.py -a ubuntu-esm ${ppa}
+    done
+}
+
+if [ "$1" = "help" ] || [ "$1" = "--help" ] || [ "$1" = "-h" ]; then
+    help
+    exit 0
+fi
+
+add_esm_auth_conf
diff --git a/build-tools/build-sources-list.install b/build-tools/setup-sources-list.install
 similarity index 89%
 rename from build-tools/build-sources-list.install
 rename to build-tools/setup-sources-list.install
index 8fd7e34..11280e7 100755
--- a/build-tools/build-sources-list.install
+++ b/build-tools/setup-sources-list.install
@@ -10,9 +10,11 @@ set -e
 # command
 
 SOURCES_LIST="/etc/apt/sources.list.d/ubuntu-security.list"
-PER_REPO_GPG_DIR="/etc/apt/keyrings/"
+PER_REPO_GPG_DIR="/etc/apt/keyrings"
 ESM_GPG_PUBLIC_KEY="${PER_REPO_GPG_DIR}/ubuntu-esm-ppas.gpg"
 ESM_GPG_KEY_FINGERPRINT="DBB1FC89762BF6B96707C4059BC0A1A1622CF918"
+AUTH_CONF_DIRECTORY="/etc/apt/auth.conf.d"
+AUTH_CONF_TMP_DIRECTORY="esm-tmp-auth-conf"
 
 help() {
     cat <<EOM
@@ -62,6 +64,15 @@ echo "Setting up the sources.list in ${SOURCES_LIST}"
                 chgrp sudo ${SOURCES_LIST} && \
 		chmod 640 ${SOURCES_LIST}"
 
+echo "Setting up the auth.conf files for ESM"
+sudo sh -c "mkdir -p ${AUTH_CONF_DIRECTORY}/${AUTH_CONF_TMP_DIRECTORY} && \
+            chmod 640 ${AUTH_CONF_DIRECTORY}/${AUTH_CONF_TMP_DIRECTORY}"
+sudo sh -c " echo \"$("${UST}"/build-tools/esm-auth-conf)\" | awk -v RS= '{print > (\"${AUTH_CONF_DIRECTORY}/${AUTH_CONF_TMP_DIRECTORY}/\"\$5\".conf\")}'"
+sudo sh -c "chgrp sudo -R ${AUTH_CONF_DIRECTORY}/${AUTH_CONF_TMP_DIRECTORY} && \
+		chmod 640 -R ${AUTH_CONF_DIRECTORY}/${AUTH_CONF_TMP_DIRECTORY} && \
+        mv ${AUTH_CONF_DIRECTORY}/${AUTH_CONF_TMP_DIRECTORY}/* ${AUTH_CONF_DIRECTORY}/ && \
+        rm -rf ${AUTH_CONF_DIRECTORY}/${AUTH_CONF_TMP_DIRECTORY}"
+
 if [ -f "${ESM_GPG_PUBLIC_KEY}" ] ; then
     echo "ppa key file ${ESM_GPG_PUBLIC_KEY} already exists"
 else