esm-cache.service throw apt.cache.FetchFailedException when connecting to https://esm.ubuntu.com fails

This might be a bit user visible if the crash reporter causes a popup to show on desktops:

# ll /var/crash/_usr_lib_ubuntu-advantage_esm_cache.py.0.crash
-rw-r----- 1 root root 23576 Jan 27 12:48 /var/crash/_usr_lib_ubuntu-advantage_esm_cache.py.0.crash

I am in the same situation. None of our servers have internet access. Our APT sources point to a local Artifactory instance. No HTTPS proxy is in use.

$ systemctl --failed
  UNIT LOAD ACTIVE SUB DESCRIPTION
● esm-cache.service loaded failed failed Update the local ESM caches

LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.

1 loaded units listed.

$ journalctl -o cat -u esm-cache.service
Starting Update the local ESM caches...
Traceback (most recent call last):
  File "/usr/lib/ubuntu-advantage/esm_cache.py", line 22, in <module>
    main(cfg)
  File "/usr/lib/ubuntu-advantage/esm_cache.py", line 11, in main
    update_esm_caches(cfg)
  File "/usr/lib/python3/dist-packages/uaclient/apt.py", line 636, in update_esm_caches
    cache.update()
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 593, in update
    raise FetchFailedException()
apt.cache.FetchFailedException
esm-cache.service: Main process exited, code=exited, status=1/FAILURE
esm-cache.service: Failed with result 'exit-code'.
Failed to start Update the local ESM caches.

$ apt-cache policy ubuntu-advantage-tools
ubuntu-advantage-tools:
  Installed: 27.13.2~20.04.1
  Candidate: 27.13.2~20.04.1
  Version table:
 *** 27.13.2~20.04.1 500
        500 https://LOCALDOMAIN/artifactory/ubuntu-debian-remote focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     20.3 500
        500 https://LOCALDOMAIN/artifactory/ubuntu-debian-remote focal/main amd64 Packages

$ lsb_release -rd
Description: Ubuntu 20.04.5 LTS
Release: 20.04

$uname -a
Linux mail 5.4.0-137-generic #154-Ubuntu SMP Thu Jan 5 17:03:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Status changed to 'Confirmed' because the bug affects multiple users.

> I am in the same situation. None of our servers have internet access.

@Jonas, out of curiosity, how did you handle this in the previous versions of ubuntu-advantage-tools, where esm.ubuntu.com was explicitly a sources.list entry (or separate file)? Did you just comment the esm lines?

@Andreas, the system sources.list entry was only ever in place for esm-infra which was only for ESM distros (Trusty, Xenial). esm-apps (which applies to Bionic onwards) never got a system sources.list entry. So it makes sense that this hasn't show up until now.

Status changed to 'Confirmed' because the bug affects multiple users.

I am in the same situation.
# ll /var/crash/_usr_lib_ubuntu-advantage_esm_cache.py.0.crash
-rw-r----- 1 root root 23576 Jan 27 12:48 /var/crash/_usr_lib_ubuntu-advantage_esm_cache.py.0.crash

> We are creating an automated test for this scenario, which will run on all future SRUs of ubuntu-advantage-tools. This test will be used for verification of this SRU.

This is fine (and preferable), but for SRU verification of this SRU, can we please also ensure that 1) the test does correctly fail if the scenario fails (eg. cause the test to fail on the version currently in -updates); and 2) the test did actually run when you ran the test suite? Just satisfy yourself please, to ensure that we don't assume all is OK in case one of those things doesn't happen. You may already have taken steps to ensure this - in that case nothing further needed.

Hello Simon, or anyone else affected,

Accepted ubuntu-advantage-tools into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.13.3~22.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Simon, or anyone else affected,

Accepted ubuntu-advantage-tools into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.13.3~22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Simon, or anyone else affected,

Accepted ubuntu-advantage-tools into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.13.3~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Simon, or anyone else affected,

Accepted ubuntu-advantage-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.13.3~18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Simon, or anyone else affected,

Accepted ubuntu-advantage-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/27.13.3~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Verification for Jammy:

# enable -proposed
# run `apt-get update`
root@sdeziel-lemur:~# apt-get update && apt-get dist-upgrade
...
Fetched 935 kB in 1s (867 kB/s)
Reading package lists... Done

# pull the updated package
root@sdeziel-lemur:~# apt-get install ubuntu-advantage-tools
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be upgraded:
   ubuntu-advantage-tools (27.13.2~22.04.1 => 27.13.3~22.04.1)
1 upgraded, 0 newly installed, 0 to remove and 44 not upgraded.
Need to get 172 kB of archives.
After this operation, 1,024 B of additional disk space will be used.
Get:1 https://apt.sdeziel.info/archive/ubuntu jammy-proposed/main amd64 ubuntu-advantage-tools amd64 27.13.3~22.04.1 [172 kB]
Fetched 172 kB in 1s (256 kB/s)
[master fc03eab] saving uncommitted changes in /etc prior to apt run
 Author: Simon Deziel <email address hidden>
 1 file changed, 1 insertion(+), 1 deletion(-)
Reading changelogs... Done
apt-listchanges: Do you want to continue? [Y/n]
Preconfiguring packages ...
(Reading database ... 135862 files and directories currently installed.)
Preparing to unpack .../ubuntu-advantage-tools_27.13.3~22.04.1_amd64.deb ...
Unpacking ubuntu-advantage-tools (27.13.3~22.04.1) over (27.13.2~22.04.1) ...
Setting up ubuntu-advantage-tools (27.13.3~22.04.1) ...
ua-timer.timer is a disabled or a static unit, not starting it.
Processing triggers for man-db (2.10.2-1) ...

# run `apt-get update` to trigger the esm apt hook
root@sdeziel-lemur:~# apt-get update
...
Fetched 483 kB in 1s (342 kB/s)
Reading package lists... Done

# check esm-cache journal output:

$ journalctl -fu esm-cache
Jan 31 20:16:06 sdeziel-lemur python3[2395775]: File "/usr/lib/ubuntu-advantage/esm_cache.py", line 11, in main
Jan 31 20:16:06 sdeziel-lemur python3[2395775]: update_esm_caches(cfg)
Jan 31 20:16:06 sdeziel-lemur python3[2395775]: File "/usr/lib/python3/dist-packages/uaclient/apt.py", line 636, in update_esm_caches
Jan 31 20:16:06 sdeziel-lemur python3[2395775]: cache.update()
Jan 31 20:16:06 sdeziel-lemur python3[2395775]: File "/usr/lib/python3/dist-packages/apt/cache.py", line 575, in update
Jan 31 20:16:06 sdeziel-lemur python3[2395775]: raise FetchFailedException()
Jan 31 20:16:06 sdeziel-lemur python3[2395775]: apt.cache.FetchFailedException
Jan 31 20:16:06 sdeziel-lemur systemd[1]: esm-cache.service: Main process exited, code=exited, status=1/FAILURE
Jan 31 20:16:06 sdeziel-lemur systemd[1]: esm-cache.service: Failed with result 'exit-code'.
Jan 31 20:16:06 sdeziel-lemur systemd[1]: Failed to start Update the local ESM caches.
Jan 31 20:17:30 sdeziel-lemur systemd[1]: Starting Update the local ESM caches...
Jan 31 20:17:38 sdeziel-lemur python3[2398857]: Failed to fetch the ESM Apt Cache
Jan 31 20:17:38 sdeziel-lemur systemd[1]: esm-cache.service: Deactivated successfully.
Jan 31 20:17:38 sdeziel-lemur systemd[1]: Finished Update the local ESM caches.

# confirm no service failed:

root@sdeziel-lemur:~# systemctl --failed
  UNIT LOAD ACTIVE SUB DESCRIPTION
0 loaded units listed.

# confirm that no *new* crash reports were generate...

All autopkgtests for the newly accepted ubuntu-advantage-tools (27.13.3~18.04.1) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-advantage-tools/unknown (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#ubuntu-advantage-tools

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

I'm attaching both automated and manual tests confirming the fix in -proposed is working for all releases.

Also included is an example of the automated test failing when using 27.13.2 from -updates.

Marking verification done for all remaining releases.

The failure on bionic/arm64 is unrelated:

autopkgtest for ubuntu-advantage-tools/unknown: arm64: Regression ♻

(...)
Installing for arm64-efi platform.
grub-install: error: efibootmgr: not found.
Failed: grub-install --target=arm64-efi
WARNING: Bootloader is not properly installed, system may not be bootable
Sourcing file `/etc/default/grub'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.15.0-202-generic
Found initrd image: /boot/initrd.img-4.15.0-202-generic
Adding boot menu entry for EFI firmware configuration
done
Setting up grub-efi-arm64-signed (1.187.2~18.04.1+2.06-2ubuntu14) ...
Installing for arm64-efi platform.
grub-install: error: efibootmgr: not found.
dpkg: error processing package grub-efi-arm64-signed (--configure):
 installed grub-efi-arm64-signed package post-installation script subprocess returned error exit status 1
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Errors were encountered while processing:
 grub-efi-arm64-signed
E: Sub-process /usr/bin/dpkg returned an error code (1)
autopkgtest [12:49:06]: ERROR: erroneous package: testbed setup commands failed with status 100
blame:
badpkg: testbed setup commands failed with status 100

It was highlighted on #ubuntu-devel irc[1] and is being discussed.

Looks like the bug is https://bugs.launchpad.net/ubuntu/+source/grub2-signed/+bug/1936857

1. https://irclogs.ubuntu.com/2023/02/01/%23ubuntu-devel.html#t17:29

I verified the attached logs and am satisfied that they show the executed planned test case, and that the results are correct.

In all cases, manual and automated, the crash no longer happens, and we get a warning in the logs.

I also checked the forced-failure test case on jammy, and the automated test caught the failure correctly.

The package built correctly in all architectures and ubuntu releases it was meant for.

There is one DEP8 regression in bionic/arm64, but that is caused by an unrelated update to grub/shim/signed and is being handled in https://bugs.launchpad.net/ubuntu/+source/grub2-signed/+bug/1936857

There is no SRU freeze ongoing at the moment.

The previous ubuntu-advantage-tools SRU is in a halted phasing state, which is what this bug here is about.

(deleted comment: was meant for the other bug)

This bug was fixed in the package ubuntu-advantage-tools - 27.13.3~23.04.1

---------------
ubuntu-advantage-tools (27.13.3~23.04.1) lunar; urgency=medium

  * d/ubuntu-advantage-tools.preinst: (LP: #2004279)
    - correct second set of md5sums to continue avoiding a dpkg conf prompt
      if the only change to the original config file was to the apt_news flag
    - restore correct default uaclient.conf when upgrading from 27.13.X and
      the only conf change is apt_news
  * esm-cache.service:
    - Catch errors when esm.ubuntu.com is unreachable to avoid causing crash
      reports and degraded systemd status from this non-critical service
      (LP: #2004130)

 -- Grant Orndorff <email address hidden> Mon, 30 Jan 2023 09:34:43 -0500

This bug was fixed in the package ubuntu-advantage-tools - 27.13.3~22.10.1

---------------
ubuntu-advantage-tools (27.13.3~22.10.1) kinetic; urgency=medium

  * Backport new upstream release: (LP: #2004130 and LP: #2004279) to
    kinetic

ubuntu-advantage-tools (27.13.3~23.04.1) lunar; urgency=medium

  * d/ubuntu-advantage-tools.preinst: (LP: #2004279)
    - correct second set of md5sums to continue avoiding a dpkg conf prompt
      if the only change to the original config file was to the apt_news flag
    - restore correct default uaclient.conf when upgrading from 27.13.X and
      the only conf change is apt_news
  * esm-cache.service:
    - Catch errors when esm.ubuntu.com is unreachable to avoid causing crash
      reports and degraded systemd status from this non-critical service
      (LP: #2004130)

 -- Grant Orndorff <email address hidden> Tue, 31 Jan 2023 10:08:19 -0500

The verification of the Stable Release Update for ubuntu-advantage-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package ubuntu-advantage-tools - 27.13.3~22.04.1

---------------
ubuntu-advantage-tools (27.13.3~22.04.1) jammy; urgency=medium

  * Backport new upstream release: (LP: #2004130 and LP: #2004279) to
    jammy

ubuntu-advantage-tools (27.13.3~23.04.1) lunar; urgency=medium

  * d/ubuntu-advantage-tools.preinst: (LP: #2004279)
    - correct second set of md5sums to continue avoiding a dpkg conf prompt
      if the only change to the original config file was to the apt_news flag
    - restore correct default uaclient.conf when upgrading from 27.13.X and
      the only conf change is apt_news
  * esm-cache.service:
    - Catch errors when esm.ubuntu.com is unreachable to avoid causing crash
      reports and degraded systemd status from this non-critical service
      (LP: #2004130)

 -- Grant Orndorff <email address hidden> Tue, 31 Jan 2023 10:08:06 -0500

This bug was fixed in the package ubuntu-advantage-tools - 27.13.3~20.04.1

---------------
ubuntu-advantage-tools (27.13.3~20.04.1) focal; urgency=medium

  * Backport new upstream release: (LP: #2004130 and LP: #2004279) to
    focal

ubuntu-advantage-tools (27.13.3~23.04.1) lunar; urgency=medium

  * d/ubuntu-advantage-tools.preinst: (LP: #2004279)
    - correct second set of md5sums to continue avoiding a dpkg conf prompt
      if the only change to the original config file was to the apt_news flag
    - restore correct default uaclient.conf when upgrading from 27.13.X and
      the only conf change is apt_news
  * esm-cache.service:
    - Catch errors when esm.ubuntu.com is unreachable to avoid causing crash
      reports and degraded systemd status from this non-critical service
      (LP: #2004130)

 -- Grant Orndorff <email address hidden> Tue, 31 Jan 2023 10:08:02 -0500

This bug was fixed in the package ubuntu-advantage-tools - 27.13.3~18.04.1

---------------
ubuntu-advantage-tools (27.13.3~18.04.1) bionic; urgency=medium

  * Backport new upstream release: (LP: #2004130 and LP: #2004279) to
    bionic

ubuntu-advantage-tools (27.13.3~23.04.1) lunar; urgency=medium

  * d/ubuntu-advantage-tools.preinst: (LP: #2004279)
    - correct second set of md5sums to continue avoiding a dpkg conf prompt
      if the only change to the original config file was to the apt_news flag
    - restore correct default uaclient.conf when upgrading from 27.13.X and
      the only conf change is apt_news
  * esm-cache.service:
    - Catch errors when esm.ubuntu.com is unreachable to avoid causing crash
      reports and degraded systemd status from this non-critical service
      (LP: #2004130)

 -- Grant Orndorff <email address hidden> Tue, 31 Jan 2023 10:07:56 -0500

This bug was fixed in the package ubuntu-advantage-tools - 27.13.3~16.04.1

---------------
ubuntu-advantage-tools (27.13.3~16.04.1) xenial; urgency=medium

  * Backport new upstream release: (LP: #2004130 and LP: #2004279) to
    xenial

ubuntu-advantage-tools (27.13.3~23.04.1) lunar; urgency=medium

  * d/ubuntu-advantage-tools.preinst: (LP: #2004279)
    - correct second set of md5sums to continue avoiding a dpkg conf prompt
      if the only change to the original config file was to the apt_news flag
    - restore correct default uaclient.conf when upgrading from 27.13.X and
      the only conf change is apt_news
  * esm-cache.service:
    - Catch errors when esm.ubuntu.com is unreachable to avoid causing crash
      reports and degraded systemd status from this non-critical service
      (LP: #2004130)

 -- Grant Orndorff <email address hidden> Tue, 31 Jan 2023 10:07:50 -0500