Merge lp:~seb128/ubuntu-archive-scripts/verdict-not-package into lp:ubuntu-archive-scripts

[Sebastien Bacher]

the 'verdict" entries in the autopkgtest sections are not packages

fix the code with the new britney output

Using the new britney version the yaml has such entries

    autopkgtest:
      verdict: REJECTED_TEMPORARILY

or the code assumed the autopkgtest section only includes packages entries and hits an error

Traceback (most recent call last):
  File "./generate-team-p-m", line 352, in <module>
    main()
  File "./generate-team-p-m", line 265, in main
    for arch, result in sorted(results.items()):
AttributeError: 'str' object has no attribute 'items'

ignoring those lines makes the report work again

[Iain Lane (laney)]

Thanks.

Just wanted to say that I'm talking to upstream about restructuring the YAML due to this (it would be unfortunate if we ever get a package called 'verdict'), so the fix required might look a bit different in the end.

review: Abstain

[Iain Lane (laney)]

Looks good but isn't this also a problem for update-excuse and block-bugs bugs too?

review: Approve

[Iain Lane (laney)]

(the restructuring I mentioned will probably happen but not right now)

[lp:~seb128/ubuntu-archive-scripts/verdict-not-package]

283. By Sebastien Bacher on 2020-06-30

ignore verdict entries for block-bug and update-excuse

[Sebastien Bacher (seb128)]

thanks for reviewing!

> Looks good but isn't this also a problem for update-excuse and block-bugs bugs too?

it is but the fix was a bit less trivial since those sections are handled in the template and not in the code, I've pushed extra changes that should do the job

[Iain Lane (laney)]

Ah yeah, this works I guess but you could also delete those entries before handing the dictionary to the template, that might be cleaner? up to you, something like this (not tested!):

=== modified file 'generate-team-p-m'
--- old/generate-team-p-m 2020-04-07 10:02:51 +0000
+++ new/generate-team-p-m 2020-06-30 15:06:56 +0000
@@ -256,6 +256,11 @@
         in_proposed_packages[source_package_name] = prob
         prob.regressions = []
         prob.waiting = []
+ for policy in ['autopkgtest', 'update-excuse', 'block-bugs']:
+ try:
+ del item['policy_info'][policy]['verdict']
+ except KeyError:
+ pass
         if 'autopkgtest' in item['reason']:
             for package, results in sorted(item['policy_info']['autopkgtest'].items()):
                 regr_arches = []

[Iain Lane (laney)]

bah, Launchpad messed up my indentation there, hopefully you get the idea

[lp:~seb128/ubuntu-archive-scripts/verdict-not-package]

284. By Sebastien Bacher on 2020-06-30

exclude the verdit entries from the code it's simpler, thanks Laney!

285. By Sebastien Bacher on 2020-06-30

simply the template code again

286. By Sebastien Bacher on 2020-06-30

try reading the xz report first, if not fallback to the uncompressed one

287. By Sebastien Bacher on 2020-07-01

there is no need to read the report before iterating

288. By Sebastien Bacher on 2020-07-01

specify the urllib error to catch

289. By Sebastien Bacher on 2020-07-01

indicate when the fallback url is read and display the error

290. By Sebastien Bacher on 2020-07-01

don't special case verdicts entries, they are filter out now

[Iain Lane (laney)]

lgtm

review: Approve

=== modified file 'generate-team-p-m'
--- generate-team-p-m	2020-04-07 10:02:51 +0000
+++ generate-team-p-m	2020-07-01 08:20:30 +0000
@@ -23,10 +23,12 @@
 import os
 import threading
 from urllib.request import urlopen
+import urllib.error
 
 import attr
 from jinja2 import Environment, FileSystemLoader
 import yaml
+import lzma
 
 env = Environment(
     loader=FileSystemLoader(os.path.dirname(os.path.abspath(__file__)) + '/templates'),
@@ -238,9 +240,13 @@
 
     print("fetching yaml")
     if args.excuses_yaml:
-        yaml_text = open(args.excuses_yaml).read()
+        yaml_text = open(args.excuses_yaml)
     else:
-        yaml_text = urlopen("https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.yaml").read()
+        try:
+            yaml_text = lzma.open(urlopen("https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.yaml.xz"))
+        except urllib.error.HTTPError as e:
+            print("Reading fallback yaml (%s)" % e)
+            yaml_text = urlopen("https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.yaml")
     print("parsing yaml")
     # The CSafeLoader is ten times faster than the regular one
     excuses = yaml.load(yaml_text, Loader=yaml.CSafeLoader)
@@ -256,6 +262,12 @@
         in_proposed_packages[source_package_name] = prob
         prob.regressions = []
         prob.waiting = []
+        # The verdict entries are not items to list on the report
+        for policy in ['autopkgtest', 'update-excuse', 'block-bugs']:
+            try:
+                del item['policy_info'][policy]['verdict']
+            except KeyError:
+                pass
         if 'autopkgtest' in item['reason']:
             for package, results in sorted(item['policy_info']['autopkgtest'].items()):
                 regr_arches = []