vmtest missing images even after a full vmtest-sync-images run

Bug #1908543 reported by Paride Legovini
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
curtin
Fix Released
Undecided
Unassigned

Bug Description

Even after a full vmtest-sync-images run, done from scratch, the vmtest harness can't find some images. I suspect because of MAAS changing the image "channel" names ("daily" to "candidate" and "release" to "stable").

Reproducer (starting from an empty /srv/images):

$ tools/vmtest-sync-images
2020-12-17 09:36:52,621 - tests.vmtests - INFO - Logfile: /tmp/vmtest-2020-12-17T093652.621147.log . Working dir: /tmp/vmtest-2020-12-17T093652.621147
Acquiring an exclusive lock on /srv/images...
Acquiring an exclusive lock on /srv/images/vmtest-images.lock...
2020-12-17 09:36:53,725 - tests.vmtests - INFO - Syncing images from http://images.maas.io/ephemeral-v3/daily/streams/v1/index.sjson with filters=['os=ubuntu', 'release~bionic|focal|groovy|xenial', 'ftype~(boot-initrd|boot-kernel|root-tgz|squashfs)', 'arch~arm64']
=> xenial/arm64/20201210/ga-16.04/generic/boot-initrd [38927557]
...............................................................................
=> xenial/arm64/20201210/ga-16.04/generic/boot-kernel [14957568]
...............................................................................
=> xenial/arm64/20201210/squashfs [157917184]
...............................................................................
=> xenial/arm64/20201210/hwe-16.04/generic/boot-initrd [55585844]
...............................................................................
=> xenial/arm64/20201210/hwe-16.04/generic/boot-kernel [8313411]
...............................................................................
=> xenial/arm64/20201210/hwe-16.04-edge/generic/boot-initrd [55583928]
...............................................................................
=> xenial/arm64/20201210/hwe-16.04-edge/generic/boot-kernel [8313411]
...............................................................................
=> xenial/arm64/20201210/xgene-uboot/generic/boot-initrd.xgene [38928055]
...............................................................................
=> xenial/arm64/20201210/xgene-uboot/generic/boot-kernel.xgene [14957632]
...............................................................................
=> xenial/arm64/20201210/xgene-uboot-mustang/generic/boot-initrd.xgene-mustang [38931008]
...............................................................................
=> xenial/arm64/20201210/xgene-uboot-mustang/generic/boot-kernel.xgene-mustang [14957632]
...............................................................................
=> bionic/arm64/20201211.1/ga-18.04/generic/boot-initrd [57951687]
...............................................................................
=> bionic/arm64/20201211.1/ga-18.04/generic/boot-kernel [8479229]
...............................................................................
=> bionic/arm64/20201211.1/squashfs [183754752]
...............................................................................
=> bionic/arm64/20201211.1/hwe-18.04/generic/boot-initrd [68801250]
...............................................................................
=> bionic/arm64/20201211.1/hwe-18.04/generic/boot-kernel [10741020]
...............................................................................
=> bionic/arm64/20201211.1/hwe-18.04-edge/generic/boot-initrd [68802206]
...............................................................................
=> bionic/arm64/20201211.1/hwe-18.04-edge/generic/boot-kernel [10741020]
...............................................................................
=> bionic/arm64/20201211.1/xgene-uboot/generic/boot-initrd.xgene [57956804]
...............................................................................
=> bionic/arm64/20201211.1/xgene-uboot/generic/boot-kernel.xgene [8479293]
...............................................................................
=> bionic/arm64/20201211.1/xgene-uboot-mustang/generic/boot-initrd.xgene-mustang [57953827]
...............................................................................
=> bionic/arm64/20201211.1/xgene-uboot-mustang/generic/boot-kernel.xgene-mustang [8479293]
...............................................................................
=> focal/arm64/20201211/ga-20.04/generic/boot-initrd [88954428]
...............................................................................
=> focal/arm64/20201211/ga-20.04/generic/boot-kernel [10633925]
...............................................................................
=> focal/arm64/20201211/squashfs [343797760]
...............................................................................

=> focal/arm64/20201211/hwe-20.04/generic/boot-initrd [88954454]
...............................................................................
=> focal/arm64/20201211/hwe-20.04/generic/boot-kernel [10633925]
...............................................................................
=> focal/arm64/20201211/hwe-20.04-edge/generic/boot-initrd [93401428]
...............................................................................
=> focal/arm64/20201211/hwe-20.04-edge/generic/boot-kernel [11349848]
...............................................................................
=> focal/arm64/20201211/xgene-uboot/generic/boot-initrd.xgene [88954464]
...............................................................................
=> focal/arm64/20201211/xgene-uboot/generic/boot-kernel.xgene [10633989]
...............................................................................
=> focal/arm64/20201211/xgene-uboot-mustang/generic/boot-initrd.xgene-mustang [88954442]
...............................................................................
=> focal/arm64/20201211/xgene-uboot-mustang/generic/boot-kernel.xgene-mustang [10633989]
...............................................................................
=> groovy/arm64/20201210/ga-20.10/generic/boot-initrd [93633441]
...............................................................................
=> groovy/arm64/20201210/ga-20.10/generic/boot-kernel [12672563]
...............................................................................
=> groovy/arm64/20201210/squashfs [346574848]
...............................................................................
=> groovy/arm64/20201210/xgene-uboot/generic/boot-initrd.xgene [93633365]
...............................................................................
=> groovy/arm64/20201210/xgene-uboot/generic/boot-kernel.xgene [12672627]
...............................................................................
=> groovy/arm64/20201210/xgene-uboot-mustang/generic/boot-initrd.xgene-mustang [93633576]
...............................................................................
=> groovy/arm64/20201210/xgene-uboot-mustang/generic/boot-kernel.xgene-mustang [12672627]
...............................................................................
2020-12-17 09:47:26,419 - tests.vmtests - INFO - Syncing images from http://images.maas.io/ephemeral-v3/daily/streams/v1/index.sjson with filters=['os=centos', 'release~centos66|centos70', 'ftype~(boot-initrd|boot-kernel|root-tgz|squashfs)', 'arch~arm64']

$ tools/jenkins-runner -p 2 tests/vmtests/test_uefi_basic.py:XenialGAUefiTestBasic
CURTIN_VMTEST_CURTIN_EXE_VERSION=20.2-24-ge4e1ee0b
CURTIN_VMTEST_CURTIN_VERSION=20.2-24-ge4e1ee0b
CURTIN_VMTEST_IMAGE_SYNC=0
CURTIN_VMTEST_ISCSI_PORTAL=10.247.8.17:29502
CURTIN_VMTEST_KEEP_DATA_FAIL=logs,collect
CURTIN_VMTEST_KEEP_DATA_PASS=logs,collect
CURTIN_VMTEST_LOG=/jenkins/servers/server/workspace/curtin-autoland-test/nodes/metal-arm64/curtin/output/debug.log
CURTIN_VMTEST_PARALLEL=2
CURTIN_VMTEST_REUSE_TOPDIR=0
CURTIN_VMTEST_SHUFFLE_TESTS=1
CURTIN_VMTEST_TAR_DISKS=0
CURTIN_VMTEST_TOPDIR=/jenkins/servers/server/workspace/curtin-autoland-test/nodes/metal-arm64/curtin/output
TGT_IPC_SOCKET=/jenkins/servers/server/workspace/curtin-autoland-test/nodes/metal-arm64/curtin/output/tgt.d/socket
TGT_LOG_D=/jenkins/servers/server/workspace/curtin-autoland-test/nodes/metal-arm64/curtin/output/tgt.d
TGT_PID=1584
TGT_PORTAL=10.247.8.17:29502
http_proxy=
https_proxy=
no_proxy=
Quering synced ephemeral images/kernels in /srv/images
======================================================================================
 Release Codename ImageDate Arch /SubArch Path
--------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------
======================================================================================

Thu, 17 Dec 2020 09:48:57 -0500: vmtest start: nosetests3 --process-timeout=86400 --processes=2 -vv --nologcapture tests/vmtests/test_uefi_basic.py:XenialGAUefiTestBasic
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
2020-12-17 09:48:59,591 - vmtests - INFO - Logfile: /jenkins/servers/server/workspace/curtin-autoland-test/nodes/metal-arm64/curtin/output/debug.log . Working dir: /jenkins/servers/server/workspace/curtin-autoland-test/nodes/metal-arm64/curtin/output
2020-12-17 09:48:59,669 - XenialGAUefiTestBasic - INFO - Logfile: /jenkins/servers/server/workspace/curtin-autoland-test/nodes/metal-arm64/curtin/output/debug.log . Working dir: /jenkins/servers/server/workspace/curtin-autoland-test/nodes/metal-arm64/curtin/output
2020-12-17 09:48:59,677 - XenialGAUefiTestBasic - INFO - Starting setup for testclass: XenialGAUefiTestBasic (ubuntu/xenial -> ubuntu/xenial)
2020-12-17 09:48:59,957 - XenialGAUefiTestBasic - INFO - Using tempdir: /jenkins/servers/server/workspace/curtin-autoland-test/nodes/metal-arm64/curtin/output/XenialGAUefiTestBasic
2020-12-17 09:48:59,959 - XenialGAUefiTestBasic - INFO - Loading testcase config file: examples/tests/uefi_basic.yaml
ERROR

======================================================================
ERROR: test suite for <class 'vmtests.test_uefi_basic.XenialGAUefiTestBasic'>
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nose/plugins/multiprocess.py", line 788, in run
    self.setUp()
  File "/usr/lib/python3/dist-packages/nose/suite.py", line 292, in setUp
    self.setupContext(ancestor)
  File "/usr/lib/python3/dist-packages/nose/plugins/multiprocess.py", line 770, in setupContext
    super(NoSharedFixtureContextSuite, self).setupContext(context)
  File "/usr/lib/python3/dist-packages/nose/suite.py", line 315, in setupContext
    try_run(context, names)
  File "/usr/lib/python3/dist-packages/nose/util.py", line 471, in try_run
    return func()
  File "/jenkins/servers/server/workspace/curtin-autoland-test/nodes/metal-arm64/curtin/tests/vmtests/__init__.py", line 959, in setUpClass
    ftypes = cls.get_test_files()
  File "/jenkins/servers/server/workspace/curtin-autoland-test/nodes/metal-arm64/curtin/tests/vmtests/__init__.py", line 671, in get_test_files
    ftypes=('boot-initrd', 'boot-kernel', cls.ephemeral_ftype))
  File "/jenkins/servers/server/workspace/curtin-autoland-test/nodes/metal-arm64/curtin/tests/vmtests/__init__.py", line 262, in get_images
    "syncing disabled:\n%s" % query_str)
ValueError: Required images not found and syncing disabled:
python3 tests/vmtests/image_sync.py query /srv/images ftype~(boot-initrd|boot-kernel|squashfs) release=xenial arch=arm64 os=ubuntu kflavor=generic krel=xenial subarch=ga-16.04

----------------------------------------------------------------------
Ran 0 tests in 0.388s

FAILED (errors=1)
Thu, 17 Dec 2020 09:49:00 -0500: vmtest end [1] in 7s

Related branches

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :
Revision history for this message
Ryan Harper (raharper) wrote :

I believe I understand what's going wrong; thanks to @OddBloke for the tip. tests/vmtests/image_sync.py syncs from the daily link, which is now a URL redirect to stable and as a result, the *content* of the streams file is also renamed:

com.ubuntu.maas:stable:v3:download.json

which used to be

com.ubuntu.maas:daily:v3:download.json

When curtin's image_sync goes to look for the synced content, it would only ever look for the content in these previously downloaded streams data files

If we change the URL to stable, and the STREAM_BASE (which points to the file where vmtest caches which content it pulled down) then the sync works correctly.

So this:

% git diff tests/vmtests/image_sync.py
diff --git a/tests/vmtests/image_sync.py b/tests/vmtests/image_sync.py
index e460e028..9183f18a 100644
--- a/tests/vmtests/image_sync.py
+++ b/tests/vmtests/image_sync.py
@@ -31,10 +31,10 @@ def environ_get(key, default):
     long_key = 'CURTIN_VMTEST_' + key
     return os.environ.get(long_key, os.environ.get(key, default))

-
+MAAS_STREAM = environ_get("MAAS_IMAGE_STREAM", "stable")
 IMAGE_SRC_URL = environ_get(
     'IMAGE_SRC_URL',
- "http://images.maas.io/ephemeral-v3/daily/streams/v1/index.sjson")
+ "http://images.maas.io/ephemeral-v3/%s/streams/v1/index.sjson" % MAAS_STREAM)
 IMAGE_DIR = environ_get("IMAGE_DIR", "/srv/images")

 KEYRING = environ_get(
@@ -43,7 +43,7 @@ KEYRING = environ_get(
 ITEM_NAME_FILTERS = \
     ['ftype~(boot-initrd|boot-kernel|root-tgz|squashfs)']
 FORMAT_JSON = 'JSON'
-STREAM_BASE = 'com.ubuntu.maas:daily'
+STREAM_BASE = 'com.ubuntu.maas:%s' % MAAS_STREAM
 VMTEST_CONTENT_ID_PATH_MAP = {
     STREAM_BASE + ":v3:download": "streams/v1/vmtest.json",
     STREAM_BASE + ":centos-bases-download": "streams/v1/vmtest-centos.json",

Fixes things for syncing from stable.

To your second question, should curtin use stable or candidate, I believe curtin should test stable primarily and I'd like to add a jenkins job to run candidate.

The challenge is that we can't easily mix the same images repo without a bigger refactor in image_sync.py as the /srv/images/.vmtest_data/* content is coupled to the stream_base.

Depending on how frequently we think we should run against candidate (once a week?) we could sync candidate to a different IMAGE_DIR; and with a fix to accept which stream.

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

This bug is fixed with commit e387c747 to curtin on branch master.
To view that commit see the following URL:
https://git.launchpad.net/curtin/commit/?id=e387c747

Changed in curtin:
status: New → Fix Committed
Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote : Fixed in curtin version 21.1.

This bug is believed to be fixed in curtin in version 21.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in curtin:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.