Merge ~andersson123/autopkgtest-cloud:preserve-all-proposed into autopkgtest-cloud:master

[Tim Andersson]

[Tim Andersson (andersson123)]

I need to add another column to the "results" table I believe, then access that column instead of preserving all-proposed in the "triggers" column of the "results" table.

[Tim Andersson (andersson123)]

Done - ready for review. It's tested but I'm going to rebuild the charms now that I've cleaned up the git history and test again just to be sure.

[Tim Andersson (andersson123)]

Tested again off the most recent charm - works.

[Tim Andersson (andersson123)]

I need to amend this to make the column for all-proposed not be JUST for all-proposed, include any environmental variables

[Tim Andersson (andersson123)]

well, not environmental, but you get it

[Tim Andersson (andersson123)]

https://code.launchpad.net/~andersson123/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/453426/comments/1223250

this is done now

[Tim Andersson (andersson123)]

This is ready for re-review

[Skia (hyask)]

I've added a few comments, then this will need testing on staging.

This is very likely to superseed this MP, which is a very good thing!
https://code.launchpad.net/~xnox/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/403628

review: Needs Fixing

[Tim Andersson (andersson123)]

Amended now, please re-review! Thanks for the review!

[Skia (hyask)]

review: Needs Fixing

[Tim Andersson (andersson123)]

Amended, please re-review

[Skia (hyask)]

Looking good, thanks!

review: Approve

[Tim Andersson (andersson123)]

CI failure is unrelated transient issue. I'm going to go ahead and merge this.

[Skia (hyask)]

Some additional thoughts that should be addressed afterwards :-)

diff --git a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/.coverage b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/.coverage
new file mode 100644
index 0000000..e617275
Binary files /dev/null and b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/.coverage differ
diff --git a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker
index 419e5f6..ea31096 100755
--- a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker
@@ -319,7 +319,7 @@ def request_matches_per_package(package, arch, release, s):
     )
 
 
-def process_output_dir(dir, pkgname, code, triggers):
+def process_output_dir(dir, pkgname, code, triggers, all_proposed):
     """Post-process output directory"""
 
     files = set(os.listdir(dir))
@@ -349,6 +349,15 @@ def process_output_dir(dir, pkgname, code, triggers):
                 json.dump(d, testinfo, indent=True)
             files.add("testinfo.json")
 
+    # add all-proposed to testinfo.json if test was triggered with all-proposed=1
+    if all_proposed:
+        d = {}
+        with open(os.path.join(dir, "testinfo.json"), "r") as testinfo:
+            d = json.load(testinfo)
+        d["all-proposed"] = 1
+        with open(os.path.join(dir, "testinfo.json"), "w") as testinfo:
+            json.dump(d, testinfo, indent=True)
+
     with open(os.path.join(dir, "testpkg-version"), "r") as tpv:
         testpkg_version = tpv.read().split()[1]
 
@@ -1235,7 +1244,11 @@ def request(msg):
                     f.write("%s\n" % params["readable-by"])
 
         (testpkg_version, duration, requester) = process_output_dir(
-            out_dir, pkgname, code, params.get("triggers", [])
+            out_dir,
+            pkgname,
+            code,
+            params.get("triggers", []),
+            True if params.get("all-proposed", 0) == 1 else False,
         )
 
         # If two tests for the same package with different triggers finish at the
@@ -1358,6 +1371,7 @@ def request(msg):
             "requester": requester,
             "swift_dir": swift_dir,
             "triggers": triggers,
+            "all-proposed": 1 if "all-proposed" in params else 0,
         }
     )
     complete_amqp.basic_publish(
diff --git a/charms/focal/autopkgtest-web/webcontrol/browse.cgi b/charms/focal/autopkgtest-web/webcontrol/browse.cgi
index 7355d6b..9a86a66 100755
--- a/charms/focal/autopkgtest-web/webcontrol/browse.cgi
+++ b/charms/focal/autopkgtest-web/webcontrol/browse.cgi
@@ -296,16 +296,22 @@ def package_release_arch(package, release, arch, _=None):
     seen = set()
     results = []
     for row in db_con.execute(
-        "SELECT run_id, version, triggers, duration, exitcode, requester FROM result "
+        "SELECT run_id, version, triggers, duration, exitcode, requester, env FROM result "
         "WHERE test_id=? "
         "ORDER BY run_id DESC",
         (test_id,),
     ):
         requester = row[5] if row[5] else "-"
         code = human_exitcode(row[4])
+        version = row[1]
+        triggers = row[2]
+        env = row[
+            6
+        ]  # string of comma separated env variables e.g. all-proposed=1,test-name=mytest
+
         identifier = (
-            row[1],
-            row[2],
+            version,
+            triggers,
         )  # Version + triggers uniquely identifies this result
         show_retry = code != "pass" and identifier not in seen
         seen.add(identifier)
@@ -319,14 +325,16 @@ def package_release_arch(package, release, arch, _=None):
         )
         results.append(
             (
-                row[1],
-                row[2],
+                version,
+                triggers,
+                env,
                 human_date(row[0]),
                 human_sec(row[3]),
                 requester,
                 code,
                 url,
                 show_retry,
+                "&all-proposed=1" if "all-proposed=1" in env else "",
             )
         )
 
diff --git a/charms/focal/autopkgtest-web/webcontrol/download-all-results b/charms/focal/autopkgtest-web/webcontrol/download-all-results
index 412771a..436c806 100755
--- a/charms/focal/autopkgtest-web/webcontrol/download-all-results
+++ b/charms/focal/autopkgtest-web/webcontrol/download-all-results
@@ -172,11 +172,18 @@ def fetch_one_result(url):
         test_triggers,
         exitcode,
     )
+    env_vars = []
+    env_spec = {
+        "all-proposed": "all-proposed=1",
+    }
+    for env, spec in env_spec.items():
+        if env in testinfo.keys():
+            env_vars.append(spec)
 
     try:
         c = db_con.cursor()
         c.execute(
-            "INSERT INTO result VALUES (?, ?, ?, ?, ?, ?, ?)",
+            "INSERT INTO result VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
             (
                 test_id,
                 run_id,
@@ -185,6 +192,7 @@ def fetch_one_result(url):
                 duration,
                 exitcode,
                 requester,
+                ",".join(env_vars),
             ),
         )
         db_con.commit()
@@ -236,6 +244,15 @@ if __name__ == "__main__":
     try:
         for release in releases:
             db_con = init_db(config["web"]["database"])
+            # check if env column exists
+            # if not, create it
+            try:
+                c = db_con.cursor()
+                c.execute("SELECT env FROM result")
+            except sqlite3.OperationalError as e:
+                if "no such column" in str(e):
+                    c = db_con.cursor()
+                    c.execute("ALTER TABLE result ADD COLUMN env TEXT")
             fetch_container(
                 release,
                 os.path.join(
diff --git a/charms/focal/autopkgtest-web/webcontrol/download-results b/charms/focal/autopkgtest-web/webcontrol/download-results
index d97b98a..ca0de53 100755
--- a/charms/focal/autopkgtest-web/webcontrol/download-results
+++ b/charms/focal/autopkgtest-web/webcontrol/download-results
@@ -79,12 +79,20 @@ def process_message(msg, db_con):
         msg.channel.basic_ack(msg.delivery_tag)
         return
 
+    env_vars = []
+    env_spec = {
+        "all-proposed": "all-proposed=1",
+    }
+    for env, spec in env_spec.items():
+        if env in info:
+            env_vars.append(spec)
+
     test_id = get_test_id(db_con, release, arch, package)
 
     try:
         c = db_con.cursor()
         c.execute(
-            "INSERT INTO result VALUES (?, ?, ?, ?, ?, ?, ?)",
+            "INSERT INTO result VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
             (
                 test_id,
                 run_id,
@@ -93,6 +101,7 @@ def process_message(msg, db_con):
                 duration,
                 exitcode,
                 requester,
+                ",".join(env_vars),
             ),
         )
         db_con.commit()
@@ -108,6 +117,15 @@ if __name__ == "__main__":
     )
 
     db_con = db_connect()
+
+    try:
+        c = db_con.cursor()
+        c.execute("SELECT env FROM result")
+    except sqlite3.OperationalError as e:
+        if "no such column" in str(e):
+            c = db_con.cursor()
+            c.execute("ALTER TABLE result ADD COLUMN env TEXT")
+
     amqp_con = amqp_connect()
     status_ch = amqp_con.channel()
     status_ch.access_request("/complete", active=True, read=True, write=False)
diff --git a/charms/focal/autopkgtest-web/webcontrol/helpers/utils.py b/charms/focal/autopkgtest-web/webcontrol/helpers/utils.py
index 6fbb09c..8ad9eeb 100644
--- a/charms/focal/autopkgtest-web/webcontrol/helpers/utils.py
+++ b/charms/focal/autopkgtest-web/webcontrol/helpers/utils.py
@@ -43,6 +43,7 @@ def init_db(path):
             "  duration INTEGER, "
             "  exitcode INTEGER, "
             "  requester TEXT, "
+            "  env TEXT, "
             "  PRIMARY KEY(test_id, run_id), "
             "  FOREIGN KEY(test_id) REFERENCES test(id))"
         )
diff --git a/charms/focal/autopkgtest-web/webcontrol/templates/browse-results.html b/charms/focal/autopkgtest-web/webcontrol/templates/browse-results.html
index ecf47e2..6ae5ab8 100644
--- a/charms/focal/autopkgtest-web/webcontrol/templates/browse-results.html
+++ b/charms/focal/autopkgtest-web/webcontrol/templates/browse-results.html
@@ -6,6 +6,7 @@
     <tr>
       <td><b>Version</b></td>
       <td><b>Triggers</b></td>
+      <td><b>Env</b></td>
       <td><b>Date</b></td>
       <td><b>Duration</b></td>
       <td><b>Requester</b></td>
@@ -28,11 +29,12 @@
       </td>
       <td class="{{row[5]}}">{{row[5]}}</td>
       <td class="nowrap">
-        <a href="{{row[6]}}/log.gz">log</a> &emsp;
-        <a href="{{row[6]}}/artifacts.tar.gz">artifacts</a> &emsp;
-        {% if row[7] %}
+        <a href="{{row[7]}}/log.gz">log</a> &emsp;
+        <a href="{{row[7]}}/artifacts.tar.gz">artifacts</a> &emsp;
+        {% if row[8] %}
           {% set ts = row[1].split()|map('urlencode')|join("&trigger=")|safe %}
-          <a href="{{base_url}}request.cgi?release={{release}}&arch={{arch}}&package={{package|urlencode}}&trigger={{ts}}">&#9851;</a>
+          {% set all_proposed = row[9] %}
+          <a href="{{base_url}}request.cgi?release={{release}}&arch={{arch}}&package={{package|urlencode}}&trigger={{ts}}{{all_proposed}}">&#9851;</a>
         {% endif %}
       </td>
     </tr>