Merge ~lamoura/ubuntu/+source/ubuntu-advantage-tools:upload-29.2-mantic into ubuntu/+source/ubuntu-advantage-tools:ubuntu/devel

[Lucas Albuquerque Medeiros de Moura]

New release addressing the bugs identified on the SRU review of 29.1

[Lucas Albuquerque Medeiros de Moura (lamoura)]

Xenial:
 - Source ubuntu-advantage-tools/29.2~16.04~rc1: Published
    + amd64: https://autopkgtest.ubuntu.com/request.cgi?release=xenial&package=ubuntu-advantage-tools&arch=amd64&trigger=ubuntu-advantage-tools%2F29.2~16.04~rc1&ppa=ua-client%2Fstaging♻️
    + arm64: https://autopkgtest.ubuntu.com/request.cgi?release=xenial&package=ubuntu-advantage-tools&arch=arm64&trigger=ubuntu-advantage-tools%2F29.2~16.04~rc1&ppa=ua-client%2Fstaging♻️
    + armhf: https://autopkgtest.ubuntu.com/request.cgi?release=xenial&package=ubuntu-advantage-tools&arch=armhf&trigger=ubuntu-advantage-tools%2F29.2~16.04~rc1&ppa=ua-client%2Fstaging♻️
    + ppcel64: https://autopkgtest.ubuntu.com/request.cgi?release=xenial&package=ubuntu-advantage-tools&arch=ppcel64&trigger=ubuntu-advantage-tools%2F29.2~16.04~rc1&ppa=ua-client%2Fstaging♻️
    + riscv64: https://autopkgtest.ubuntu.com/request.cgi?release=xenial&package=ubuntu-advantage-tools&arch=riscv64&trigger=ubuntu-advantage-tools%2F29.2~16.04~rc1&ppa=ua-client%2Fstaging♻️
    + s390x: https://autopkgtest.ubuntu.com/request.cgi?release=xenial&package=ubuntu-advantage-tools&arch=s390x&trigger=ubuntu-advantage-tools%2F29.2~16.04~rc1&ppa=ua-client%2Fstaging♻️
    + amd64: https://autopkgtest.ubuntu.com/request.cgi?release=xenial&package=ubuntu-advantage-tools&arch=amd64&trigger=ubuntu-advantage-tools%2F29.2~16.04~rc1&ppa=ua-client%2Fstaging💍
    + arm64: https://autopkgtest.ubuntu.com/request.cgi?release=xenial&package=ubuntu-advantage-tools&arch=arm64&trigger=ubuntu-advantage-tools%2F29.2~16.04~rc1&ppa=ua-client%2Fstaging💍
    + armhf: https://autopkgtest.ubuntu.com/request.cgi?release=xenial&package=ubuntu-advantage-tools&arch=armhf&trigger=ubuntu-advantage-tools%2F29.2~16.04~rc1&ppa=ua-client%2Fstaging💍
    + ppcel64: https://autopkgtest.ubuntu.com/request.cgi?release=xenial&package=ubuntu-advantage-tools&arch=ppcel64&trigger=ubuntu-advantage-tools%2F29.2~16.04~rc1&ppa=ua-client%2Fstaging💍
    + riscv64: https://autopkgtest.ubuntu.com/request.cgi?release=xenial&package=ubuntu-advantage-tools&arch=riscv64&trigger=ubuntu-advantage-tools%2F29.2~16.04~rc1&ppa=ua-client%2Fstaging💍
    + s390x: https://autopkgtest.ubuntu.com/request.cgi?release=xenial&package=ubuntu-advantage-tools&arch=s390x&trigger=ubuntu-advantage-tools%2F29.2~16.04~rc1&ppa=ua-client%2Fstaging💍

Bionic:
  - Source ubuntu-advantage-tools/29.2~18.04~rc1: Published
    + amd64: https://autopkgtest.ubuntu.com/request.cgi?release=bionic&package=ubuntu-advantage-tools&arch=amd64&trigger=ubuntu-advantage-tools%2F29.2~18.04~rc1&ppa=ua-client%2Fstaging♻️
    + arm64: https://autopkgtest.ubuntu.com/request.cgi?release=bionic&package=ubuntu-advantage-tools&arch=arm64&trigger=ubuntu-advantage-tools%2F29.2~18.04~rc1&ppa=ua-client%2Fstaging♻️
    + armhf: https://autopkgtest.ubuntu.com/request.cgi?release=bionic&package=ubuntu-advantage-tools&arch=armhf&trigger=ubuntu-advantage-tools%2F29.2~18.04~rc1&ppa=ua-client%2Fstaging♻️
    + ppcel64: https://autopkgtest.ubuntu.com/request.cgi?release=bionic&package=ubuntu-advantage-tools&arch=ppcel64&trigger=ubuntu-advantage-tools%2F29.2~18.04~rc1&ppa=ua-client%2Fstaging♻️
    + riscv64: https://autopkgtest.ubuntu.com/r...

[Sergio Durigan Junior (sergiodj)]

Thanks, Lucas.

There's a small nit on sru/release-29/test-key-rename-failure.sh that needs fixing, but other than that the changes LGTM. I can sponsor the package as soon as you fix the issue.

review: Needs Fixing

[Sergio Durigan Junior (sergiodj)]

So I talked to Lucas in private, and while pedantically speaking the comparison operator '==' shouldn't be used when comparing strings with '[', bash still supports it (although it is not the standard) and this won't affect the release per se.

Therefore, in order to avoid spending even more time changing the code, I will go ahead and upload the package as is. Lucas also told me that they plan to tackle this specific issue in the next releases.

Uploaded:

$ dput ubuntu-advantage-tools_29.2_source.changes
Trying to upload package to ubuntu
Checking signature on .changes
gpg: /home/sergio/work/ubuntu-advantage-tools/ubuntu-advantage-tools_29.2_source.changes: Valid signature from 106DA1C8C3CBBF14
Checking signature on .dsc
gpg: /home/sergio/work/ubuntu-advantage-tools/ubuntu-advantage-tools_29.2.dsc: Valid signature from 106DA1C8C3CBBF14
Uploading to ubuntu (via ftp to upload.ubuntu.com):
  Uploading ubuntu-advantage-tools_29.2.dsc: done.
  Uploading ubuntu-advantage-tools_29.2.tar.xz: done.
  Uploading ubuntu-advantage-tools_29.2_source.buildinfo: done.
  Uploading ubuntu-advantage-tools_29.2_source.changes: done.
Successfully uploaded packages.

review: Approve

diff --git a/debian/changelog b/debian/changelog
index 6d7a651..1378858 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,12 @@
+ubuntu-advantage-tools (29.2) mantic; urgency=medium
+
+  * d/ubuntu-advantage-tools.postinst:
+    - replace deb-systemd-invoke back to systemctl
+  * proxy: alert user if ca-certificates is not installed when using
+    a TLS-in-TLS proxy
+
+ -- Lucas Moura <[e-mail]>  Thu, 17 Aug 2023 19:45:48 -0300
+
 ubuntu-advantage-tools (29.1) mantic; urgency=medium
 
   * anbox: allow enabling service on container using the --access-only flag
diff --git a/debian/ubuntu-advantage-tools.postinst b/debian/ubuntu-advantage-tools.postinst
index b332333..b11748f 100644
--- a/debian/ubuntu-advantage-tools.postinst
+++ b/debian/ubuntu-advantage-tools.postinst
@@ -256,11 +256,11 @@ remove_old_systemd_units() {
             # The failed state is ephemeral and only needs to be cleared if
             # it is there so that the system doesn't say it is degraded.
             # If the old timer was not running, then this is a noop.
-            deb-systemd-invoke --system daemon-reload > /dev/null || true
-            deb-systemd-invoke reset-failed ua-license-check.timer > /dev/null 2>&1 || true
+            systemctl --system daemon-reload > /dev/null || true
+            systemctl reset-failed ua-license-check.timer > /dev/null 2>&1 || true
             # In rare race-condition scenarios, the service can also get into
             # the same failed state.
-            deb-systemd-invoke reset-failed ua-license-check.service > /dev/null 2>&1 || true
+            systemctl reset-failed ua-license-check.service > /dev/null 2>&1 || true
         fi
         deb-systemd-helper purge ua-license-check.path > /dev/null || true
         deb-systemd-helper unmask ua-license-check.path > /dev/null || true
diff --git a/features/proxy_config.feature b/features/proxy_config.feature
index 0b6359d..755f5dc 100644
--- a/features/proxy_config.feature
+++ b/features/proxy_config.feature
@@ -1257,7 +1257,19 @@ Feature: Proxy configuration
         Proxy authentication failed
         """
 
-        When I run `pro config set https_proxy=https://someuser:somepassword@$behave_var{machine-name proxy}.lxd:3129` with sudo
+        When I run `apt remove ca-certificates -y` with sudo
+        And I run `rm -f /etc/ssl/certs/ca-certificates.crt` with sudo
+        And I verify that running `pro config set https_proxy=https://someuser:somepassword@$behave_var{machine-name proxy}.lxd:3129` `with sudo` exits `1`
+        Then stderr matches regexp:
+        """
+        Failed to access URL: https://.*
+        Cannot verify certificate of server
+        Please install "ca-certificates" and try again.
+        """
+
+        When I run `apt install ca-certificates -y` with sudo
+        And I run `update-ca-certificates` with sudo
+        And I run `pro config set https_proxy=https://someuser:somepassword@$behave_var{machine-name proxy}.lxd:3129` with sudo
         And I run `pro config set ua_apt_https_proxy=https://someuser:somepassword@$behave_var{machine-name proxy}.lxd:3129` with sudo
 
         When I run `truncate -s 0 /var/log/squid/access.log` `with sudo` on the `proxy` machine
@@ -1285,7 +1297,8 @@ Feature: Proxy configuration
         """
 
         # Pre-install canonical-livepatch to tell it to trust the cert
-        When I run `snap install canonical-livepatch` with sudo
+        When I run `apt install snapd -y` with sudo
+        And I run `snap install canonical-livepatch` with sudo
         And I run shell command `canonical-livepatch config ca-certs=@stdin < /usr/local/share/ca-certificates/ca.crt` with sudo
 
         When I run `truncate -s 0 /var/log/squid/access.log` `with sudo` on the `proxy` machine
diff --git a/preferences.d/ubuntu-pro-esm-apps b/preferences.d/ubuntu-pro-esm-apps
index aa60598..5ded6f8 100644
--- a/preferences.d/ubuntu-pro-esm-apps
+++ b/preferences.d/ubuntu-pro-esm-apps
@@ -1,7 +1,10 @@
-# This file was created by ubuntu-advantage-tools
-
-# Pin esm-apps packages to a slightly higher value than the archive,
-# so those are preferred when the service is enabled
+# This file is used by Ubuntu Pro and supplied by the ubuntu-advantage-tools
+# package. It has no effect if Ubuntu Pro services are not in use since no
+# other apt repositories are expected to match o=UbuntuESMApps.
+#
+# Pin esm-apps packages to a slightly higher value than the default,
+# so those are preferred over a non-ESM package from the archive when the
+# service is enabled.
 
 Package: *
 Pin: release o=UbuntuESMApps
diff --git a/preferences.d/ubuntu-pro-esm-infra b/preferences.d/ubuntu-pro-esm-infra
index af9bf5a..e956e0a 100644
--- a/preferences.d/ubuntu-pro-esm-infra
+++ b/preferences.d/ubuntu-pro-esm-infra
@@ -1,8 +1,10 @@
-# This file was created by ubuntu-advantage-tools
-
-# Pin esm-infra packages to a slightly higher value than the archive,
-# so those are preferred when the service is enabled
-
+# This file is used by Ubuntu Pro and supplied by the ubuntu-advantage-tools
+# package. It has no effect if Ubuntu Pro services are not in use since no
+# other apt repositories are expected to match o=UbuntuESM.
+#
+# Pin esm-infra packages to a slightly higher value than the default,
+# so those are preferred over a non-ESM package from the archive when the
+# service is enabled.
 Package: *
 Pin: release o=UbuntuESM
 Pin-Priority: 510
diff --git a/sru/release-29/test-esm-pinning.sh b/sru/release-29/test-esm-pinning.sh
index b9db8c6..b64d8e3 100755
--- a/sru/release-29/test-esm-pinning.sh
+++ b/sru/release-29/test-esm-pinning.sh
@@ -104,6 +104,12 @@ check_esm_pin "infra" 510
 check_esm_pin "apps" 510
 echo -e "###########################################\n"
 
+# Check pin for archive packages
+echo -e "\n* Check pins for package in archive"
+echo "###########################################"
+lxc exec $name -- apt-cache policy toilet python3-ipdb
+echo -e "###########################################\n"
+
 # Disable esm-apps and esm-infra
 # ----------------------------------------------------------------
 disable_esm_services
diff --git a/sru/release-29/test-key-rename-failure.sh b/sru/release-29/test-key-rename-failure.sh
new file mode 100644
index 0000000..6ad58e8
--- /dev/null
+++ b/sru/release-29/test-key-rename-failure.sh
@@ -0,0 +1,118 @@
+#!/bin/bash
+set -e
+
+series=$1
+token=$2
+install_from=$3 # either path to a .deb, or 'staging', or 'proposed'
+
+name=$series-dev
+
+function cleanup {
+  lxc delete $name --force
+}
+
+function on_err {
+  echo -e "Test Failed"
+  cleanup
+  exit 1
+}
+trap on_err ERR
+
+lxc launch ubuntu-daily:$series $name
+sleep 5
+
+# Install latest ubuntu-advantage-tools
+lxc exec $name -- apt-get update > /dev/null
+lxc exec $name -- apt-get install  -y ubuntu-advantage-tools > /dev/null
+echo -e "\n* Latest u-a-t is installed"
+echo "###########################################"
+lxc exec $name -- apt-cache policy ubuntu-advantage-tools
+echo -e "###########################################\n"
+
+# Attach so we can have some gpg keys in place
+lxc exec $name -- pro attach $token
+echo -e "\n* Pro is attached"
+echo "###########################################"
+lxc exec $name -- pro status --wait
+echo -e "###########################################\n"
+
+# Upgrade u-a-t to new version
+# ----------------------------------------------------------------
+if [ $install_from == 'staging' ]; then
+  lxc exec $name -- sudo add-apt-repository ppa:ua-client/staging -y > /dev/null
+  lxc exec $name -- apt-get update > /dev/null
+  lxc exec $name -- apt-get install ubuntu-advantage-tools -y > /dev/null
+elif [ $install_from == 'proposed' ]; then
+  lxc exec $name -- sh -c "echo \"deb http://archive.ubuntu.com/ubuntu $series-proposed main\" | tee /etc/apt/sources.list.d/proposed.list"
+  lxc exec $name -- apt-get update > /dev/null
+  lxc exec $name -- apt-get install ubuntu-advantage-tools -y > /dev/null
+else
+  lxc file push $install_from $name/new-ua.deb
+  lxc exec $name -- dpkg -i /new-ua.deb > /dev/null
+fi
+# ----------------------------------------------------------------
+
+echo -e "\n* Renaiming keys back to their original name"
+echo "###########################################"
+lxc exec $name -- mv /etc/apt/trusted.gpg.d/ubuntu-pro-esm-infra.gpg /etc/apt/trusted.gpg.d/ubuntu-advantage-esm-infra-trusty.gpg
+lxc exec $name -- mv /etc/apt/trusted.gpg.d/ubuntu-pro-esm-apps.gpg /etc/apt/trusted.gpg.d/ubuntu-advantage-esm-apps.gpg
+echo -e "###########################################\n"
+
+echo -e "\n* List gpg keys"
+echo "###########################################"
+lxc exec $name -- ls /etc/apt/trusted.gpg.d/
+echo -e "###########################################\n"
+
+# Modify the postinst script to fail after renaming just one key
+lxc exec $name -- sed -i "s/^\s\+SERVICES=.*/error/g" /var/lib/dpkg/info/ubuntu-advantage-tools.postinst
+echo -e "\n* Running postinst script again"
+echo "###########################################"
+lxc exec $name -- dpkg-reconfigure ubuntu-advantage-tools || true
+echo -e "###########################################\n"
+
+# Check that only one key got renamed
+echo -e "\n* List gpg keys"
+echo "###########################################"
+lxc exec $name -- ls /etc/apt/trusted.gpg.d/
+echo -e "###########################################\n"
+
+# Make sure the services are still enabled
+echo -e "\n* Check that services are still enabled"
+echo "###########################################"
+lxc exec $name -- pro status --wait
+echo -e "###########################################\n"
+
+# Check that apt update is working as expected
+echo -e "\n* Check that APT update is working as expected"
+echo "###########################################"
+lxc exec $name -- apt update
+echo -e "###########################################\n"
+
+# Disable service which key was not renamed
+echo -e "\n* Disable esm-apps service which gpg key was not renamed"
+echo "###########################################"
+lxc exec $name -- pro disable esm-apps
+lxc exec $name -- pro status
+echo -e "###########################################\n"
+
+# Check that the GPG key is still there
+echo -e "\n* List gpg keys"
+echo "###########################################"
+lxc exec $name -- ls /etc/apt/trusted.gpg.d/
+echo -e "###########################################\n"
+
+# Enable esm-apps
+echo -e "\n* Enable esm-apps and check new gpg key is created"
+echo "###########################################"
+lxc exec $name -- pro enable esm-apps
+lxc exec $name -- pro status
+lxc exec $name -- ls /etc/apt/trusted.gpg.d/
+echo -e "###########################################\n"
+
+# Check that the new GPG key is there
+echo -e "\n* check ubuntu-pro-esm-apps.gpg key now exists"
+echo "###########################################"
+lxc exec $name -- ls /etc/apt/trusted.gpg.d/
+echo -e "###########################################\n"
+
+cleanup
diff --git a/uaclient/cli.py b/uaclient/cli.py
index 120158b..9807a59 100644
--- a/uaclient/cli.py
+++ b/uaclient/cli.py
@@ -2000,6 +2000,20 @@ def main_error_handler(func):
             _warn_about_new_version()
 
             sys.exit(1)
+        except exceptions.PycurlCACertificatesError as exc:
+            tmpl = messages.SSL_VERIFICATION_ERROR_CA_CERTIFICATES
+            if apt.is_installed("ca-certificates"):
+                tmpl = messages.SSL_VERIFICATION_ERROR_OPENSSL_CONFIG
+            msg = tmpl.format(url=exc.url)
+            event.error(error_msg=msg.msg, error_code=msg.name)
+            event.info(info_msg=msg.msg, file_type=sys.stderr)
+
+            lock.clear_lock_file_if_present()
+            event.process_events()
+
+            _warn_about_new_version()
+
+            sys.exit(1)
         except exceptions.UserFacingError as exc:
             with util.disable_log_to_console():
                 LOG.error(exc.msg)
diff --git a/uaclient/exceptions.py b/uaclient/exceptions.py
index 447b4eb..156abb2 100644
--- a/uaclient/exceptions.py
+++ b/uaclient/exceptions.py
@@ -522,6 +522,13 @@ class PycurlError(UserFacingError):
         super().__init__(msg=msg.msg, msg_code=msg.name)
 
 
+class PycurlCACertificatesError(UserFacingError):
+    def __init__(self, url: str) -> None:
+        msg = messages.PYCURL_CA_CERTIFICATES_ERROR
+        self.url = url
+        super().__init__(msg=msg.msg, msg_code=msg.name)
+
+
 class ProxyAuthenticationFailed(UserFacingError):
     def __init__(self) -> None:
         msg = messages.PROXY_AUTH_FAIL
diff --git a/uaclient/http/__init__.py b/uaclient/http/__init__.py
index 36b616e..6540f85 100644
--- a/uaclient/http/__init__.py
+++ b/uaclient/http/__init__.py
@@ -72,6 +72,8 @@ def validate_proxy(
             raise
         except exceptions.ProxyAuthenticationFailed:
             raise
+        except exceptions.PycurlCACertificatesError:
+            raise
         except Exception as e:
             with util.disable_log_to_console():
                 msg = getattr(e, "reason", str(e))
@@ -208,7 +210,9 @@ def should_use_pycurl(https_proxy, target_url):
     return ret
 
 
-def _handle_pycurl_error(error, authentication_error_code):
+def _handle_pycurl_error(
+    error, url, authentication_error_code, ca_certificates_error_code
+):
     code = None
     msg = None
     if len(error.args) > 0:
@@ -221,6 +225,8 @@ def _handle_pycurl_error(error, authentication_error_code):
         and "HTTP code 407 from proxy" in msg
     ):
         raise exceptions.ProxyAuthenticationFailed()
+    elif code == ca_certificates_error_code:
+        raise exceptions.PycurlCACertificatesError(url)
     else:
         raise exceptions.PycurlError(error)
 
@@ -302,7 +308,12 @@ def _readurl_pycurl_https_in_https(
     try:
         c.perform()
     except pycurl.error as e:
-        _handle_pycurl_error(e, authentication_error_code=pycurl.E_RECV_ERROR)
+        _handle_pycurl_error(
+            e,
+            url=req.get_full_url(),
+            authentication_error_code=pycurl.E_RECV_ERROR,
+            ca_certificates_error_code=pycurl.E_SSL_CACERT_BADFILE,
+        )
 
     code = int(c.getinfo(pycurl.RESPONSE_CODE))
     body = body_output.getvalue().decode("utf-8")
diff --git a/uaclient/http/tests/test_http.py b/uaclient/http/tests/test_http.py
index 9b00c30..1161e76 100644
--- a/uaclient/http/tests/test_http.py
+++ b/uaclient/http/tests/test_http.py
@@ -453,9 +453,16 @@ class TestHandlePycurlError:
                 ("PYCURL_ERROR", "HTTP code 407 from proxy: proxy"),
                 exceptions.ProxyAuthenticationFailed,
             ),
+            (("PYCURL_ERROR", "test"), exceptions.PycurlError),
+            (
+                ("CA_CERTIFICATES_ERROR", "test"),
+                exceptions.PycurlCACertificatesError,
+            ),
         ),
     )
     def test_handle_pycurl_error(self, error_args, expected_exception):
         with pytest.raises(expected_exception):
             m_error = mock.MagicMock(args=error_args)
-            http._handle_pycurl_error(m_error, "PYCURL_ERROR")
+            http._handle_pycurl_error(
+                m_error, "url", "PYCURL_ERROR", "CA_CERTIFICATES_ERROR"
+            )
diff --git a/uaclient/messages.py b/uaclient/messages.py
index 4fd06d8..a5cee3c 100644
--- a/uaclient/messages.py
+++ b/uaclient/messages.py
@@ -1411,6 +1411,9 @@ PYCURL_REQUIRED = NamedMessage(
     ),
 )
 PYCURL_ERROR = FormattedNamedMessage("pycurl-error", "PycURL Error: {e}")
+PYCURL_CA_CERTIFICATES_ERROR = NamedMessage(
+    "pycurl-ca-certificates-error", "Problem reading SSL CA certificates"
+)
 
 PROXY_AUTH_FAIL = NamedMessage(
     "proxy-auth-fail", "Proxy authentication failed"
diff --git a/uaclient/version.py b/uaclient/version.py
index b8e1f94..5191f37 100644
--- a/uaclient/version.py
+++ b/uaclient/version.py
@@ -15,7 +15,7 @@ from uaclient.defaults import CANDIDATE_CACHE_PATH, UAC_RUN_PATH
 from uaclient.exceptions import ProcessExecutionError
 from uaclient.system import subp
 
-__VERSION__ = "29.1"
+__VERSION__ = "29.2"
 PACKAGED_VERSION = "@@PACKAGED_VERSION@@"
 
 CANDIDATE_REGEX = r"Candidate: (?P<candidate>.*?)\n"