Merge ~pfsmorigo/ubuntu-cve-tracker:usn_pocket into ubuntu-cve-tracker:master

[Paulo Flabiano Smorigo]

af16ec9... by Paulo Flabiano Smorigo on 2021-03-11

scripts/sis-generate-usn: check srcver instead of opt.esm

Signed-off-by: Paulo Flabiano Smorigo <email address hidden>

[Steve Beattie (sbeattie)]

I suspect we also want to tag the pocket for the allbinaries records, which contains the binaries that we pruned out from the 'binaries' set of packages. It is likely the web-team will need to accept those eventually as well.

5d45b05... by Paulo Flabiano Smorigo on 2021-03-15

scripts/sis-changes: Use esm-{apps,infra} for pocket

Signed-off-by: Paulo Flabiano Smorigo <email address hidden>

abd82b9... by Paulo Flabiano Smorigo on 2021-03-15

scripts/sis-generate-usn: Check the URL fields before use

Signed-off-by: Paulo Flabiano Smorigo <email address hidden>

29393eb... by Paulo Flabiano Smorigo on 2021-03-15

scripts/sis-generate-usn: Add --pocket-all option to the all binaries

Signed-off-by: Paulo Flabiano Smorigo <email address hidden>

7af9053... by Paulo Flabiano Smorigo on 2021-03-15

scripts/sis-generate-usn: change read command to press enter

Signed-off-by: Paulo Flabiano Smorigo <email address hidden>

[Steve Beattie (sbeattie)]

I have gone ahead and merge this proposal after:

  - rebasing against origin/master,
  - dropping the commit that made the no longer needed change for the HIDDEN prompt
  - added a couple of commits that expanded the json slightly, so that at the lowest level it's a dict rather than a list (so we don't have to rmember that version is position 0, etc) as well as including an 'origin' field which records which ppa(s) sis-changes collectied information against. This can be used as part of the heuristic for determining what the pocket should be.
  - fixed a couple of bugs for things like source packages that only produce a binary for one architecture (and not the 'all' architecture), as well as changes files that contain descriptions for phantom binary packages.

The necessary changes to lp:usn-tool/usn.py have already landed.

This should not break USN publication, but if it does, a revert can be done via:

  git revert -s -m 1 9879c82bc7902e287d17cb3840b370497a887cba

If done correctly, it should auto-generate a commit message like:

  This reverts commit 9879c82bc7902e287d17cb3840b370497a887cba, reversing
  changes made to 36ea6d05426692b92ed4dacd346a45cc0de01e6a.

Thanks!

review: Approve

diff --git a/scripts/publish-usn-to-website-api.py b/scripts/publish-usn-to-website-api.py
index d867302..617bcf8 100755
--- a/scripts/publish-usn-to-website-api.py
+++ b/scripts/publish-usn-to-website-api.py
@@ -183,10 +183,12 @@ for notice_id, notice in payload:
             else:
                 source_link = ""
                 version_link = ""
+            pocket = info["pocket"] if "pocket" in info else ""
             release_packages[codename].append(
                 {
                     "name": name,
                     "version": info["version"],
+                    "pocket": pocket,
                     "is_source": "false",
                     "source_link": source_link,
                     "version_link": version_link,
diff --git a/scripts/sis-changes b/scripts/sis-changes
index aaebaea..6137d44 100755
--- a/scripts/sis-changes
+++ b/scripts/sis-changes
@@ -346,15 +346,26 @@ def load_pkg_details_from_lp(pkgs, binaries, pkg, item):
 
     binaries.setdefault(pkg, dict())
     binaries[pkg].setdefault(rel, dict())
-    # map binaries as pkg -> rel -> binary -> version - this is used by
+    # map binaries as pkg -> rel -> binary -> [version, pocket] - this is used by
     # sis-generate-usn to know what the actual binary package version is
     # since this may be different than the version of the source
     # package. Also check that all built binaries have actually published
     # into the PPA
     for binary in item.getPublishedBinaries():
         # collect binary versions
-        binaries[pkg][rel].setdefault(binary.binary_package_name,
-                                      binary.binary_package_version)
+
+        url = ""
+        all_arch = cve_lib.get_all_arch(rel)
+        # First check the current arch, if there is no result, check the "all" arch
+        if binary.binary_package_name in pkgs[pkg][rel][arch]['binaries']:
+           url = pkgs[pkg][rel][arch]['binaries'][binary.binary_package_name]
+        elif binary.binary_package_name in pkgs[pkg][rel][all_arch]['binaries']:
+           url = pkgs[pkg][rel][all_arch]['binaries'][binary.binary_package_name]
+        pocket = ""
+        for esm_pocket in ["esm-infra", "esm-apps"]:
+            if esm_pocket in url:
+                pocket = esm_pocket
+        binaries[pkg][rel].setdefault(binary.binary_package_name, [binary.binary_package_version, pocket])
         if opt.skip_build_check is True:
             print("WARNING: skipping binary publication check. Please check manually.", file=sys.stderr)
         else:
diff --git a/scripts/sis-generate-usn b/scripts/sis-generate-usn
index 5ddc86c..b002fb6 100755
--- a/scripts/sis-generate-usn
+++ b/scripts/sis-generate-usn
@@ -26,7 +26,7 @@ import cve_lib
 
 opter = optparse.OptionParser()
 opter.add_option("--debug", help="Verbose processing output", action='store_true')
-opter.add_option("--esm", help="Generate template with comments for esm updates", default=None)
+opter.add_option("--esm", help="Set to be an ESM release", default=None)
 opter.add_option("--kernel-mode", help="Use special kernel mode", action='store_true')
 opter.add_option("--no-new-warn", help="Do not warn about or error out on NEW binaries", action='store_true')
 opter.add_option("--filter-bins", metavar="REGEX", help="Only include binary packages matching the REGEX in report", default=None)
@@ -341,6 +341,11 @@ def lp_find_url(source, srcver, filename, release, arch):
 
 def find_url(source, srcver, filename, release, arch):
     '''Try to find the archive URL for a given .deb filename'''
+
+    # For ESM there is no files in archive
+    if re.match("^.*[+~]esm[0-9]*$", srcver):
+        return ""
+
     filepath, base = find_archive_path(source, release, arch, filename)
 
     return base_to_url(base,release,arch)
@@ -817,11 +822,12 @@ for release in releases:
             # package, so override the source package version with the
             # binary package version if we have this available
             try:
-                version = binaries[source][release][deb]
+                version = binaries[source][release][deb][0]
             except KeyError as e:
                 pass
+            pocket = "--pocket %s" % binaries[source][release][deb][1] if binaries[source][release][deb][1] else ""
             if not filter_dbg.search(deb) and (not opt.filter_bins or opt.filter_bins.search(deb)):
-                print('  usn.py --db "$DB" $USN --release %s --package %s --binary-version %s' % (release,deb,version))
+                print('  usn.py --db "$DB" $USN --release %s --package %s --binary-version %s %s' % (release, deb, version, pocket))
         print()
 
 # URLs
@@ -838,7 +844,10 @@ for release in releases:
                 continue
             for name in info[release][source]['arch'][arch]:
                 size, md5, url = info[release][source]['arch'][arch][name]
-                print('usn.py --db "$DB" $USN --release %s --arch %s --url %s --url-size %s --url-md5 %s' % (release, arch, url, size, md5))
+                if size and md5 and url:
+                    print('usn.py --db "$DB" $USN --release %s --arch %s --url %s --url-size %s --url-md5 %s' % (release, arch, url, size, md5))
+                else:
+                    print('# No valid URL found for %s %s' % (release, arch))
             print()
         print()
     print()
@@ -854,11 +863,12 @@ for release in releases:
             # package, so override the source package version with the
             # binary package version if we have this available
             try:
-                version = binaries[source][release][deb]
+                version = binaries[source][release][deb][0]
             except KeyError as e:
                 pass
             if not filter_dbg.search(deb) and (not opt.filter_bins or opt.filter_bins.search(deb)):
-                print('  usn.py --db "$DB" $USN --release %s --package %s --all-binary-version %s' % (release,deb,version))
+                pocket = "--pocket-all %s" % binaries[source][release][deb][1] if binaries[source][release][deb][1] else ""
+                print('  usn.py --db "$DB" $USN --release %s --package %s --all-binary-version %s %s' % (release,deb,version, pocket))
         print()
 
 print('if [ "${HIDDEN,,}" = "true" ]; then')
@@ -875,7 +885,7 @@ print('    echo IMPORTANT: Keep $USN.json until it becomes public.')
 print('    echo ""')
 print('    exit')
 print('else')
-print('    read -p "Refresh local USN database and upload it to remote master? [yN] " -n 1 -r')
+print('    read -p "Refresh local USN database and upload it to remote master? [yN] " -r')
 print('    [[ ! $REPLY =~ ^[Yy]$ ]] && exit')
 print('fi')
 print('')