Ubuntu
apt-clone package

autopkgtest fails - test_clone_upgrade.py

Bug #1902830 reported by Christian Ehrhardt 
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
apt-clone (Ubuntu)
Fix Released
Undecided
Unassigned

Bug Description

It seems this autopkgtest is broken in 21.04 and the issue is affecting all architecture at the moment.

There is nothing yet in -proposed that would indicate a fix is on the way (if it would be an upload to the package).

# The Issue

======================================================================
FAIL: test_clone_upgrade_synthetic (__main__.TestCloneUpgrade)
test clone upgrade with on-the-fly generated chroots
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_clone_upgrade.py", line 59, in test_clone_upgrade_synthetic
    self.assertTrue(len(cache.get_changes()) > 0)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 2 tests in 12.801s

Fail log:
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/a/apt-clone/20201027_023127_1a678@/log.gz
And a bunch of retries on the infra by

# Retries

vorlon: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/a/apt-clone/20201028_172522_e07e0@/log.gz

juliank: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/a/apt-clone/20201028_095059_2ef15@/log.gz

bdmurray: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/a/apt-clone/20201104_010337_b3a3a@/log.gz

But I found no bug that would document the insights so far.
Hence I'm filing this one and set the mentioned people on CC here.
Especially thins this test blocks python-apt which breaks automated testing using add-apt-repository I'd expect more people will come by and take a look - so documenting a bit can't hurt in this case.

# Repro

This fails in local VM autokgtest with/without the packages in proposed.

# History

Interestingly there seems to be a pattern of this hitting early -dev cycles (in some of them even being an issue up to late in the cycle).
Groovy:
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-groovy/groovy/amd64/a/apt-clone/20200502_094132_932ac@/log.gz
Focal:
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal/focal/amd64/a/apt-clone/20191020_095345_3b0ca@/log.gz
Bionic:
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic/bionic/amd64/a/apt-clone/20171026_195549_b2b60@/log.gz

Therefore this might - for some of us - be a known re-occuring case
and just wait for something (an image pushed, a release being mentioned
in a data file, ...).
The fact that juliank+bdmurray+vorlon have retried the test makes me think
it is expected to work already but doesn't?

Looking at test hints we have had:
badtest (2015)
https://git.launchpad.net/~ubuntu-release/britney/+git/hints-ubuntu/commit/?id=32c975e7cb91000fde6a1590b2a34b6677f1fb2c
badtest (2018)
https://git.launchpad.net/~ubuntu-release/britney/+git/hints-ubuntu/commit/?id=8bdef30c2deb13ff337d27fd2c1fb3efbc8c8d6a
(mass) convert badtest to a reset (2020):
https://git.launchpad.net/~ubuntu-release/britney/+git/hints-ubuntu/commit/?id=22d2cf67f10a1a5322670dcc9b40bbfeed250a7a

That finally got me to bug 1510060.
Seems this keeps to continue coming and going.

I'll maybe propose to change it back to a badtest.

See original description
Christian Ehrhardt  (paelzer)
description: updated
Christian Ehrhardt  (paelzer)
description: updated
description: updated
Christian Ehrhardt  (paelzer)
description: updated
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

The test results are now empty for more meta packages (not just ubuntu-standard) which at first it seems slightly different than bug 1510060. But it still is the same issue overall.

--------- ubuntu-standard -----
[]
-------------
--------- ubuntu-desktop -----
[]
-------------
--------- kubuntu-desktop -----
[<Package: name:'python3-sip' architecture='amd64' id:27681>, <Package: name:'libsrt1-gnutls' architecture='amd64' id:30322>]
-------------
--------- xubuntu-desktop -----
[]
-------------

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I'm not going deeper into this case as it seems everyone who did in the past ended up badtesting it anyway. I'm not assuming I can beat the combined past debugging/analysis of pitti, mvo and vorlon and therefore proposed to mark it a badtest again (as we had it before).

https://code.launchpad.net/~paelzer/britney/+git/hints-ubuntu/+merge/393306

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Actually one can debug a bit via:
$ export APT_CLONE_DEBUG_RESOLVER=1
$ PYTONPATH=.. python3.8 test_clone_upgrade.py

The libsrt1-gnutls that shows up as proper value in the kubuntu-desktop is mentioned then as:
  Installing libsrt1-gnutls:amd64 as Depends of libavformat58:amd64
But it is listed the same way when handling xubuntu-desktop and missing there.

 libsrt1-gnutls | 1.4.1-5build1 | groovy/universe | amd64, arm64, armhf, i386, ppc64el, riscv64, s390x
 libsrt1-gnutls | 1.4.2-1 | hirsute/universe | amd64, arm64, armhf, i386, ppc64el, riscv64, s390x

So it listed a package that is upgraded.
When doing the initial grab of installed packages it logged
libsrt1-gnutls 1.4.1-5build1 1

The test does
1. create an env of the -2 release (groovy in this case)
   old = self._create_fake_upgradable_root(supported[-2], meta=meta)
2. stores a package selection list of OLD in tarball
3. create an env of the -1 release (hirsute in this case)
   new = self._create_fake_upgradable_root(supported[-1], meta=meta)
4. creates an apt-cache of the NEW env
   cache = apt.Cache(rootdir=new)
5. restores package selection of the "clone of NEW" to the state of "OLD" from tarball

It expects that there are upgrades available under the meta packages of ubuntu-standard""ubuntu-desktop" "kubuntu-desktop" and "xubuntu-desktop". But in the beginning of a release (or at any later time if there are apt network issues) there are none.

I've found that we can make this much more reliable (especially early on when -dev ~= -dev-1 by comparing it with the first release that distro info reports.

With that I have found a fix that eliminates the false positive.
=> https://code.launchpad.net/~paelzer/apt-clone/apt-clone-fix-fails-early-cycle/+merge/393310

This would still be a new upload, so we could accept the badtest now as-is and upload a fixed version later ... up to you (release team people).

@mvo are you still reviewing/merging/uploading those MPs to apt-clone?

Christian Ehrhardt  (paelzer)
Changed in apt-clone (Ubuntu):
status: New → Triaged
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

This is fixed in 0.4.2 synced into Ubuntu by MVO (thanks!).
It needs a follow on 0.4.3 which already is on the way.

Changed in apt-clone (Ubuntu):
status: Triaged → In Progress
Revision history for this message
Brian Murray (brian-murray) wrote :

0.4.3ubuntu1 is available in hirsute and the autopkgtests have been passing in impish, so I'm setting this to Fix Released.

summary: - autopkgtest fails in hirsute - test_clone_upgrade.py
+ autopkgtest fails - test_clone_upgrade.py
Changed in apt-clone (Ubuntu):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.