Comment 4 for bug 1657440
Revision history for this message
| Samuel Matzek (smatzek) wrote Re: apt won't redownload Release.gpg | #4 |
© 2004
Canonical Ltd.
•
Terms of use
•
Data privacy
•
Contact Launchpad Support
•
Blog
•
Careers
•
System status
•
35a32c5
(Get the code!)
This upstream patch needs to be backported to the 1.2 series for Xenial. If left unfixed in Xenial it opens a timing window every hour with Ubuntu Cloud Archive where users can fall into the "Some packages could not be authenticated" state without Release.gpg that is described in the description. This state will not self correct until an hour has passed an another apt-get update is run. This really impacts automated deployment technologies such as Juju and Ansible because if they hit this hourly window with UCA their fallback retries on apt-get update will not work and the automated deployments fail. The noted upstream Debian bug is specifically about trying to do an apt-get update without adding the keys first. The timing window that users can hit will occur even if you add the keys first.
Now for background information to explain the assertions above. Ubuntu Cloud Archive updates its files, and more importantly the timestamps on its files including the Release and Release.gpg file every hour. The timestamps are updated to be 45 minutes past the hour. The UCA servers start to reflect these changes around 50 minutes after the hour with a rolling update of the Packages and then the Release.* files. They are not updated as an atomic unit as seen from an HTTP client.
So the order of events is:
1. User or automation adds keys by installing the 'ubuntu-
2. User adds the UCA repo using the Ansible apt_repository or other technique, possilbly just adding the repo to a sources list file under /etc/apt/
3. Either the tooling (apt_repository module) or the user triggers an apt-get update or other apt cache update trigger. If this cache update hits the timing window when UCA is being updated you can get into the state where you have the Release file but not the Release.gpg file without triggering a cache or apt-get update failure. A recreation Python main which uses straight python-apt can show this. I will attach my recreation program and output showing the error case.
4. At this point, as shown in the original description no further apt-get updates will fix the situation and any package installs from UCA will fail with "Some packages could not be authenticated".
While the timing window may seem small, probably a minute each hour, with complex multi-node OpenStack deployments using Ansible we are seeing this occur fairly frequently. Given the 'juju' in the host name in the original description I suspect that multi-node orchestrated Juju charm deployments using UCA are also hitting this often.
The bug is particularly harmful to automated deploy tooling because while the deploy tooling normally has apt-get update retries or periodic updates throughout the process, once this error state is entered the apt-get updates do not work to correct it until after an hour has passed and UCA has updated itself. The deployment tooling normally times and fails much sooner than an hour of retries.
Here is the annotated log output of the recreation script:
#####
# In this snippet we see apt update not pulling down the Release.gpg that was deleted right before the update to test
# its ability to pull down a new copy.
#####
Before update 2017-02-06 22:50:01.915
After update 2017-02-06 22:50:03.044
ctime changed for file /var/lib/
Path does not exist /var/lib/
Before update 2017-02-06 22:50:03.045
After update 2017-02-06 22:50:06.387
ctime changed for file /var/lib/
mtime changed for file /var/lib/
Path does not exist /var/lib/
#####
# In this iteration of the loop we see that the apt update has removed the Release file as well as not pulling down the Release.gpg.
# UCA's hourly update is in progress.
#####
Before update 2017-02-06 22:50:06.387
After update 2017-02-06 22:50:07.536
Path does not exist /var/lib/
Path does not exist /var/lib/
#####
# In this iteration of the loop we see that the apt update has updates the packages file and recreated the Release file.
# However, it did NOT pull down the Release.gpg file. If an automated tool were to hit this window they enter the bug state
# where no amount of updates will fix this until after the next hourly UCA update.
#####
Before update 2017-02-06 22:50:07.536
After update 2017-02-06 22:50:09.966
ctime changed for file /var/lib/
ctime changed for file /var/lib/
mtime changed for file /var/lib/
Path does not exist /var/lib/
#####
# In this iteration of the loop the Release file has been removed by the update call again.
#####
Before update 2017-02-06 22:50:09.967
After update 2017-02-06 22:50:12.491
Path does not exist /var/lib/
Path does not exist /var/lib/
#####
# Now the update call put the release file back but not the Release.gpg, again, entering the bug state.
#####
Before update 2017-02-06 22:50:12.492
After update 2017-02-06 22:50:14.869
ctime changed for file /var/lib/
Path does not exist /var/lib/
#####
# Update removed the Release file again.
#####
Before update 2017-02-06 22:50:14.870
After update 2017-02-06 22:50:17.336
Path does not exist /var/lib/
Path does not exist /var/lib/
#####
# Now finally, both Release and Release.gpg are there.
#####
Before update 2017-02-06 22:50:17.336
After update 2017-02-06 22:50:19.885
ctime changed for file /var/lib/
ctime changed for file /var/lib/
mtime changed for file /var/lib/
#####
# The recreate program goes along its way, having deleted the Release.gpg and it is not pulled down by update.
#####
Before update 2017-02-06 22:50:19.886
After update 2017-02-06 22:50:22.208
ctime changed for file /var/lib/
Path does not exist /var/lib/