Merge ~smoser/ubuntu/+source/lvm2:feature/autopkg-test into ubuntu/+source/lvm2:ubuntu/devel

[Scott Moser]

Add autopkg tests for lvm2 based on upstream 'lvm2' test suite.

The lvm2 package currently does not have any autopkg tests. This makes it
hard to have any confidence in an SRU. The changes package the upstream
provided testsuite, and provide the ability to run those tests from
autopkgtest.

The same merge request has been submitted to debian [1].

Running the upstream testsuite in an autopackage test has has some
failures. I posted about those failures upstream at [2].

Rather than root-cause and fix each of the tests, I have added a file
debian/tests/testsuite-skips which contains a list of tests that will be
skipped and comments describing what failed.

The testsuite runner in (debian/test/testsuite) will read that file and
not execute the tests that have known issues. Running all the tests can
be accomplished by removing or truncating the file. To run all tests
without changes to the package you can do:

    autopkgtest --setup-commands="touch /run/lvm2-testsuite-skips"

--
 [1] https://salsa.debian.org/lvm-team/lvm2/-/merge_requests/6
 [2] https://listman.redhat.com/archives/linux-lvm/2023-June/026372.html

ea5b449... by Scott Moser on 2023-11-03

releasing package lvm2 version 2.03.16-2ubuntu2

[Scott Moser (smoser)]

An upload of d6f1d737aaa13395ce3f1be0337b2b8f4a624a86
with series noble and version 2.03.16-2ubuntu2~ppa2
is being built in smoser/ppa here

https://launchpad.net/~smoser/+archive/ubuntu/ppa/+build/26957152

4ab5cf6... by Scott Moser on 2023-11-03

update changelog - Remove obsolete dependency on lsb-base

dfbdb55... by Scott Moser on 2023-11-03

Add failures/skips found running on noble.

These added skips failed today running the ppa build on official
infrastruture as described in
https://wiki.ubuntu.com/ProposedMigration#Testing_against_a_PPA

[Scott Moser (smoser)]

ea5b449884b26ed9df21d575c4189859b66ec328 is pushed to smoser/ppa as ~ppa4
https://launchpad.net/~smoser/+archive/ubuntu/ppa/+packages

I requested test of it just now (Fri, 03 Nov 2023 15:30:22 -0400)
it just started building https://autopkgtest.ubuntu.com/running#pkg-lvm2

So hopefully in about 1hr results will be in
https://autopkgtest.ubuntu.com/results/autopkgtest-noble-smoser-ppa/?format=plain

[Lucas Kanashiro (lucaskanashiro)]

Thanks for working on this Scott! At this point we need to target this upload to the next development release (OO series), since Noble was released today. Or make this a SRU.

This type of change would be really good to land first in Debian, I saw that the Debian maintainer did not reply in your salsa MR yet, I took the liberty to ping him on IRC to provide some feedback. If he provides a good feedback and demonstrated the willingness to merge it, I think we can avoid adding the delta in Ubuntu and get it in Ubuntu in the next merge. On the other hand, this package already has a delta, so adding this on top is not a big deal.

Anyway, I see you are a core-dev and actually do not need a sponsor, am I right? I am unsubscring ~ubuntu-sponsors for now. If for any reason you do not have the rights to upload the package, please subscribe ~ubuntu-sponsors again.

[Lucas Kanashiro (lucaskanashiro)]

Sorry, I though I could remove the review slot for ~ubuntu-sponsors, but I can't. It is still there :)

[Scott Moser (smoser)]

I am a core dev, but i'm not active enough to feel it is responsible to dump a change like this without other review (i really don't think anyone should do something without review.

wrt debian, yeah, it'd be really nice to have it there.

I put this together because I wanted to make an SRU of a fix, but didn't have any way to sanely check for regression in a core package. After quite some time working, I kind of abandoned this stuff because I don't think the upstream tests are reliable. :-(

Unmerged commits

ea5b449... by Scott Moser on 2023-11-03

releasing package lvm2 version 2.03.16-2ubuntu2

94e1f83... by Scott Moser on 2023-06-09

update changelog

dfbdb55... by Scott Moser on 2023-11-03

Add failures/skips found running on noble.

These added skips failed today running the ppa build on official
infrastruture as described in
https://wiki.ubuntu.com/ProposedMigration#Testing_against_a_PPA

fa7a7df... by Scott Moser on 2023-06-09

Add debian/tests to run the lvm2 testsuite.

e05433f... by Scott Moser on 2023-06-08

update changelog - Add lvm2-tests package.

b16a7df... by Scott Moser on 2023-06-16

Add lintian override for lvm2-testsuite's shell files.

Lintian would complain about dozens of .sh files in
/usr/share/lvm2-testsuite/shell/

It does not seem useful to make these executable or worth
the effort to move them.
lvm2-testsuite binary: script-not-executable

2c89bc0... by Scott Moser on 2023-06-07

Add lvm2-testsuite package.

Add an lvm2-tests package for invoking the lvm test suite.

4ab5cf6... by Scott Moser on 2023-11-03

update changelog - Remove obsolete dependency on lsb-base

a8859e6... by Bastian Blank on 2023-09-30

Remove obsolete dependency on lsb-base

(cherry picked from commit a65c2e4d2a27c3f206d62526a3d5908556e71504)

b2a71e1... by Dave Jones on 2023-06-07

changelog

diff --git a/debian/changelog b/debian/changelog
index b89e409..4986fb2 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,11 @@
+lvm2 (2.03.16-2ubuntu2) noble; urgency=medium
+
+  * Remove obsolete dependency on lsb-base 
+  * Add lvm2-testsuite package.
+  * Add an autopkgtest that uses the lvm2-testsuite.
+
+ -- Scott Moser <[e-mail]>  Fri, 03 Nov 2023 10:34:51 -0400
+
 lvm2 (2.03.16-2ubuntu1) mantic; urgency=medium
 
   * Merge from Debian unstable (LP: #2020464). Remaining changes:
diff --git a/debian/control b/debian/control
index fff01ca..941ab02 100644
--- a/debian/control
+++ b/debian/control
@@ -43,7 +43,6 @@ Pre-Depends:
 Depends:
  ${shlibs:Depends},
  ${misc:Depends},
- lsb-base,
  dmsetup (>= ${devmapper:Version}~),
  dmeventd (>= ${devmapper:Version}~),
 Recommends:
@@ -279,3 +278,24 @@ Multi-Arch: same
 Description: LVM2 libraries - development files
  This package contains files needed to develop applications that use the
  lvm2app library.
+
+Package: lvm2-testsuite
+Priority: optional
+Architecture: any
+Depends:
+ ${misc:Depends},
+ ${shlibs:Depends},
+ binutils,
+ bsdextrautils,
+ lvm2,
+ lvm2-dbusd,
+ mdadm,
+ python3,
+ python3-dbus,
+ python3-pyudev,
+ thin-provisioning-tools,
+ xfsprogs,
+Multi-Arch: foreign
+Description: testsuite for excercising LVM2
+ This package provides LVM2's test suite.  It is mainly
+ intended for use with the autopkgtest system.
diff --git a/debian/lvm2-testsuite.install b/debian/lvm2-testsuite.install
new file mode 100644
index 0000000..a7e0186
--- /dev/null
+++ b/debian/lvm2-testsuite.install
@@ -0,0 +1,3 @@
+usr/bin/lvm2-testsuite
+usr/libexec/lvm2-testsuite
+usr/share/lvm2-testsuite
diff --git a/debian/lvm2-testsuite.lintian-overrides b/debian/lvm2-testsuite.lintian-overrides
new file mode 100644
index 0000000..62c4742
--- /dev/null
+++ b/debian/lvm2-testsuite.lintian-overrides
@@ -0,0 +1,3 @@
+# Upstream provides shell files in /usr/share/lvm2-testsuite/shell/ that
+# are not meant to executable on their own but are used by the testsuite.
+lvm2-testsuite binary: script-not-executable
diff --git a/debian/rules b/debian/rules
index b6153d0..2333b7a 100755
--- a/debian/rules
+++ b/debian/rules
@@ -111,6 +111,11 @@ dh_auto_install.%:
 		install_systemd_units \
 		install_tmpfiles_configuration \
 		LIB_VERSION_DM=$(DEVMAPPER_ABINAME)
+	# install test suite and remove unit tests.
+	dh_auto_install --sourcedirectory=debian/build/build_$* \
+		--destdir=debian/build/install_$* -- -C test install \
+		LIB_VERSION_DM=$(DEVMAPPER_ABINAME)
+	rm -f debian/build/install_$*/usr/share/lvm2-testsuite/unit/unit-test
 
 override_dh_auto_clean:
 	rm -rf debian/build
diff --git a/debian/tests/control b/debian/tests/control
new file mode 100644
index 0000000..f1b53d0
--- /dev/null
+++ b/debian/tests/control
@@ -0,0 +1,3 @@
+Tests: testsuite
+Restrictions: needs-root, isolation-machine, breaks-testbed, allow-stderr
+Depends: bash, lvm2-testsuite
diff --git a/debian/tests/testsuite b/debian/tests/testsuite
new file mode 100755
index 0000000..0bb1210
--- /dev/null
+++ b/debian/tests/testsuite
@@ -0,0 +1,223 @@
+#!/bin/bash
+DEFAULT_OUTPUT_D="${AUTOPKGTEST_ARTIFACTS:-.}/testsuite-results"
+TESTSUITE_DIR=${TESTSUITE_DIR:-/usr/share/lvm2-testsuite}
+info() { stderr "INFO:" "$@"; }
+stderr() { echo "$@" 1>&2; }
+warn() { stderr "WARN:" "$@"; }
+fail() { stderr "FATAL:" "$@"; exit 1; }
+vrun() {
+    local rc=""
+    stderr "execute:" "$@"
+    "$@"
+    rc=$?
+    [ "$rc" = 0 ] || stderr "command failed [$rc]:" "$@"
+    return $rc
+}
+
+
+Usage() {
+    cat <<EOF
+${0##*/} [options] [output-dir]
+
+    --skip-installs      do not install any packages
+    --skip-masks         do not mask or stop any systemd services
+
+    output-dir defaults to "$DEFAULT_OUTPUT_D"
+EOF
+}
+
+is_installed() {
+    local pkg="$1" out="" rc=""
+    out=$(LANG=C.UTF-8 dpkg-query --show \
+        --showformat='${db:Status-Abbrev} ${Package}\n' "$pkg" 2>&1)
+    rc=$?
+    [ $rc -eq 1 ] && return 1
+    [ $rc -eq 0 ] || fail "'dpkg-query $pkg' failed [$rc]: $out"
+
+    set -- $out
+    [ $# -eq 2 ] || {
+        warn "dpkg-query $pkg got $# bits of output. expected 2. output: $out"
+        return 1
+    }
+    [ "$1" = "ii" ]
+}
+
+apt_update() {
+    vrun apt-get -q \
+        "--option=APT::Update::Error-Mode=any" update
+}
+
+apt_install() {
+    local pkg="$1"
+    apt_update || { stderr "apt-get updated failed $?"; return 1; }
+    set -- env LANG=C.UTF-8 LC_ALL=C.UTF-8 \
+        DEBIAN_FRONTEND=noninteractive \
+        apt-get \
+            --option=Dpkg::Options::=--force-confnew \
+            --assume-yes \
+            install "$pkg"
+    vrun "$@"
+}
+
+check_kernel_modules() {
+    local uname_r="" out="" mpkg=""
+    uname_r=$(uname -r) || {
+        stderr "uname -r failed $?"
+        return 1
+    }
+    mpkg="linux-modules-${uname_r}"
+    if is_installed "$mpkg"; then
+        info "Running kernel $uname_r. modules installed from $mpkg"
+        return 0
+    fi
+
+    info "kernel package $mpkg not installed. Attempting installation."
+    apt_install "$mpkg" ||
+        { stderr "installation of package $mpkg failed"; return 1; }
+
+    info "Running kernel $uname_r. modules were installed from $mpkg"
+}
+
+stop_and_mask_units() {
+    local m="" out="" unit="" state=""
+    for m in "$@"; do
+        out=$(systemctl --legend=false list-unit-files "$m.*" |
+            awk '{printf("%s:%s ", $1, $2)}')
+        if [ -z "$out" ]; then
+            stderr "$m: no matched units"
+            continue
+        fi
+        for tok in $out; do
+            unit="${tok%:*}"
+            state="${tok#*:}"
+            if [ "$state" != "masked" ]; then
+                vrun systemctl mask "$unit" || return 1
+            fi
+            vrun systemctl stop "$unit" || return 1
+        done
+    done
+    return 0
+}
+
+# read_skips(skipfile, testdir)
+#   Output a comma delimited list of tests to skip as read from skipfile.
+#
+#   skipfile: a file containing a list of tests to skip.
+#     Each line in skipfile is whitespace delimited testname and reason
+#     comments are supported with '#' character.
+#     Example:
+#         # blocksize-3 fails occasionally.
+#         shell/integrity-blocksize-3.sh           transient
+#         ...
+#   testdir: the lvm testsuite dir.  Each testname listed must be an existing
+#     test in testdir.
+read_skips() {
+    local skipfile="$1" tdir="$2"
+    local missing="" skipnames="" skipreasons="" skips="" line=""
+    missing=()
+    skipnames=()
+    skipreasons=()
+    skips=""
+
+    while read line; do
+        line=${line%%#*}
+        [ -z "$line" ] && continue
+        set -- $line
+        name=$1
+        reason=$2
+        if [ -n "$tdir" ] && [ ! -f "$tdir/$name" ]; then
+            missing=( "${missing[@]}" "$name" )
+            continue
+        fi
+        skipnames=( "${skipnames[@]}" "$name" )
+        skipreasons=( "${skipreasons[@]}" "$reason" )
+        skips="$skips,$name"
+    done <"$skipfile"
+    skips=${skips#,}
+
+    stderr "skipping ${#skipnames[@]} tests from ${skipfile}"
+    i=0
+    while [ "$i" -lt "${#skipnames[@]}" ]; do
+        printf "  %-13s %s\n" "${skipreasons[$i]}" "${skipnames[$i]}"
+        i=$((i+1))
+    done 1>&2
+
+    if [ "${#missing[@]}" != 0 ]; then
+        warn "tests listed in skiplist are not present in '$tdir'"
+        warn "update skipfile '$skipfile'"
+        for t in "${missing[@]}"; do
+            stderr "  $t"
+        done
+    fi
+
+    echo "$skips"
+}
+
+
+main() {
+    local sopts="hv"
+    local lopts="help,skip-installs,skip-masks,skip-file:,verbose"
+    local name="${0##*/}" out=""
+    out=$(getopt --name "$name" \
+        --options "$sopts" --long "$lopts" -- "$@") &&
+        eval set -- "$out" ||
+        { bad_Usage; return; }
+
+    local cur="" next="" t="" mydir=""
+    local skipfile="" do_installs=true do_masks=true
+    while [ $# -ne 0 ]; do
+        cur="$1"; next="$2";
+        case "$cur" in
+            -h|--help) Usage ; exit 0;;
+               --skip-installs) do_installs=false;;
+               --skip-masks) do_masks=false;;
+               --skip-file) skipfile=$next; shift;;
+            --) shift; break;;
+        esac
+        shift;
+    done
+
+    [ $# -eq 1 -o $# -eq 0 ] || fail "$0 got $# args ($*), expected 0 or 1"
+
+    outdir="${DEFAULT_OUTPUT_D}"
+    [ $# -eq 0 ] || {
+        outdir="$1"
+        shift;
+    }
+
+    t=$(dirname "$0") && mydir=$(cd "$t" && pwd) ||
+        fail "could not change dir to to $t"
+
+    if [ -z "$skipfile" ]; then
+        # You can use an empty skipfile (and run all the tests) by:
+        #   autopkgtest --setup-commands="touch /run/lvm2-testsuite-skips"
+        for f in "$mydir/testsuite-skips" /run/lvm2-testsuite-skips; do
+            [ -f "$f" ] && skipfile="$f"
+        done
+    fi
+    [ -e "$skipfile" ] || fail "skipfile '$skipfile' does not exist."
+
+    if [ "$do_installs" = "true" ]; then
+        check_kernel_modules || fail "check_kernel_modules failed"
+    fi
+
+    skips=$(read_skips "$skipfile" "${TESTSUITE_DIR}") ||
+        fail "failed reading skips from $skipfile"
+
+    if [ "$do_masks" = "true" ]; then
+        stop_and_mask_units dm-event lvm2-lvmpolld lvm2-monitor lvm2-lvmdbusd ||
+            fail "masking systemd units failed"
+    fi
+
+    mkdir -p "$outdir" || fail "failed to create output dir '$outdir'"
+
+    vrun lvm2-testsuite \
+        --flavours udev-vanilla \
+        ${skips:+--skip "${skips}"} \
+        --outdir "$outdir"
+    rc=$?
+    stderr "result $rc in ${SECONDS}s"
+    exit $rc
+}
+
+main "$@"
diff --git a/debian/tests/testsuite-skips b/debian/tests/testsuite-skips
new file mode 100644
index 0000000..898da46
--- /dev/null
+++ b/debian/tests/testsuite-skips
@@ -0,0 +1,104 @@
+# failed 2.03.16-2ubuntu2 on noble 6.5.0-9-generic
+shell/cache-single-options.sh            unknown
+
+# failed 2.03.16-2ubuntu2 on noble 6.5.0-9-generic
+shell/cache-single-split.sh              unknown
+
+# failed 2.03.16-2ubuntu2 on noble 6.5.0-9-generic
+shell/cache-single-types.sh              unknown
+
+# failed 2.03.16-2ubuntu2 on noble 6.5.0-9-generic
+shell/cache-single-usage.sh              unknown
+
+# failed 2.03.16-2ubuntu2 on noble 6.5.0-9-generic
+shell/fsadm-renamed.sh                   unknown
+
+# failed 2.03.16-2ubuntu2 on noble 6.5.0-9-generic
+shell/integrity-blocksize-2.sh           unknown
+
+# 2 pass/2 fail: modprobe: FATAL: Module scsi_debug is in use.
+shell/integrity-blocksize-3.sh           transient
+
+# failed 2.03.16-2ubuntu2 on noble 6.5.0-9-generic
+shell/integrity-blocksize.sh             unknown
+
+# failed 2.03.16-2ubuntu2 on noble 6.5.0-9-generic
+shell/integrity-dmeventd.sh              unknown
+
+# failed 2.03.16-2ubuntu2 on noble 6.5.0-9-generic
+shell/integrity-large.sh                 unknown
+
+# failed 2.03.16-2ubuntu2 on noble 6.5.0-9-generic
+shell/integrity-misc.sh                  unknown
+
+# failed 2.03.16-2ubuntu2 on noble 6.5.0-9-generic
+shell/integrity-syncaction.sh            unknown
+
+# failed 2.03.16-2ubuntu2 on noble 6.5.0-9-generic
+shell/integrity.sh                       unknown
+
+# 1 pass/2 fail: Flushing of LVMTEST131033vg/LV1 not aborted
+shell/lvconvert-cache-abort.sh           transient
+
+# 3 pass/1 fail: Waiting for pvmove LV to get activated has timed out
+shell/pvmove-abort-all.sh                timeout
+
+# 3 pass/1 fail: unclear
+shell/pvmove-abort.sh                    transient
+
+# ERROR: Failed to change execution mode to "lvm shell (non-fork)"
+api/dbustest.sh                          expected-fail
+
+# failed 2.03.16-2ubuntu2 on noble 6.5.0-9-generic
+shell/large-physical-sector-size.sh      unknown
+
+# failed 2.03.16-2ubuntu2 on noble 6.5.0-9-generic
+shell/lvchange-thin.sh                   unknown
+
+# no obvious failure - shell/lvconvert-repair-thin.sh:64
+shell/lvconvert-repair-thin.sh           expected-fail
+
+# # Prepare 5x ~1P sized devices
+shell/lvcreate-large-raid.sh             space-req
+
+# # 16T device
+shell/lvcreate-thin-limits.sh            space-req
+
+# #lvm-conf-error.sh:56+ grep 'Failed to load config file mnt/lvm.conf' out
+shell/lvm-conf-error.sh                  expected-fail
+
+# fsadm: Fsck ext3 failed.
+shell/lvresize-full.sh                   expected-fail
+
+# Not enough space to build proposed filesystem while setting up superblock
+shell/pvmove-basic.sh                    space-req
+
+# Unable to merge invalidated snapshot LV LVMTEST486646vg1/LV1_snap
+shell/pvscan-autoactivation-polling.sh   expected-fail
+
+# failed 2.03.16-2ubuntu2 on noble 6.5.0-9-generic
+shell/profiles-thin.sh                   unknown
+
+# failed 2.03.16-2ubuntu2 on noble 6.5.0-9-generic
+shell/writecache-misc.sh                 unknown
+
+# failed 2.03.16-2ubuntu2 on noble 6.5.0-9-generic
+shell/writecache-split.sh                unknown
+
+# failed 2.03.16-2ubuntu2 on noble 6.5.0-9-generic
+shell/duplicate-vgnames.sh               unknown
+
+# no obvious failure - shell/snapshot-merge.sh:133
+shell/snapshot-merge.sh                  expected-fail
+
+# Prepare ~1P sized devices
+shell/thin-large.sh                      space-req
+
+# 5 pass/1 fail: modprobe: FATAL: Module scsi_debug is in use.
+shell/writecache-cache-blocksize.sh      transient
+
+# failed 2.03.16-2ubuntu2 on noble 6.5.0-9-generic
+shell/lvextend-caches-on-thindata.sh     unknown
+
+# failed 2.03.16-2ubuntu2 on noble 6.5.0-9-generic
+shell/lvextend-caches.sh                 unknown