Merge ~sespiros/ubuntu-security-tools/+git/ubuntu-security-tools:fix-release-info into ubuntu-security-tools:master

[Spyros Seimenis]

Changed most of the umt references to details['release'] to use details['base_release']. The rest of the cases which use details['release'] do so because they need the full release name and not just the base/parent release.

Now that umt depends on source_map and cve_lib anyway, maybe those parts could be refactored further in the future to entirely remove the release_list field from .ubuntu-security-tools.conf and use cve_lib instead.

[Spyros Seimenis (sespiros)]

For the reviewer: link to the MP that started it all https://code.launchpad.net/~sespiros/ubuntu-security-tools/+git/ubuntu-security-tools/+merge/424782

[Alex Murray (alexmurray)]

I am not in love with changing so much code to use a new name... although I can understand the motivation. What about a different word than base_release - since this is what we would use in the changelog entry, looking at it for inspiration https://www.debian.org/doc/debian-policy/ch-source.html#s-dpkgchangelog I see they use the term distribution(s) - what if we used that instead?

[Spyros Seimenis (sespiros)]

I agree that it seems like a big change but there were 2 ways to fix the changelog/repo commands bug:
1) revert my previous fix; that means that the release field will keep not detecting ESM releases properly, the ESM chroot naming standardization was just a case that worked as a motivation to make all these changes.
2) keep the new field (base_release) and differentiate code that needs a full release name from code that needs just the base release (this patch)

Regarding the name, "distribution" could be an alternative one but I haven't seen that used in any of our tools. cve_lib for example uses the term "parent" https://git.launchpad.net/ubuntu-cve-tracker/tree/scripts/cve_lib.py#n113 to indicate the same thing, so that could be a candidate.

If we wanted to be pedantic, according to
- https://docs.ubuntu.com/landscape/en/repositories and
- https://wiki.ubuntu.com/DevelopmentCodeNames

technically a release is "Ubuntu" plus the version "X.YY" (and I guess the minor versions as well). "series" could also be the term we could use instead of "base_release" ("parent" in cve_lib) and maybe "subproject" instead of "release" (subproject is the term cve_lib uses).

[Alex Murray (alexmurray)]

series sounds like a good option - but it would be good to canvas the team to see if anyone else has better ideas.

[Emilia Torino (emitorino)]

+1 to refactor and use cve_lib as single source of this type of information

[Alex Murray (alexmurray)]

review: Approve

diff --git a/build-tools/umt b/build-tools/umt
index 27ca44c..603c412 100755
--- a/build-tools/umt
+++ b/build-tools/umt
@@ -379,10 +379,10 @@ def cmd_changelog():
     validate_toplevel()
 
     is_devel_release = False
-    if details['release'] != ust['release_devel']:
-        release = details['release'] + '-security'
+    if details['series'] != ust['release_devel']:
+        release = details['series'] + '-security'
     else:
-        release = details['release']
+        release = details['series']
         is_devel_release = True
 
     if opt.increment:
@@ -564,7 +564,7 @@ def perform_source_build(details, opt):
         build_params.append('-sa')
 
     if not opt.skip_maintainer:
-        update_maintainer(details['release'], details['version'], opt.simple_maintainer_update)
+        update_maintainer(details['series'], details['version'], opt.simple_maintainer_update)
     build_source(build_params, build_script, details,
                  force_dpkg_source=force_dpkg_source)
     process_source_files(details, source_dest, merge=opt.merge)
@@ -652,8 +652,8 @@ def perform_binary_build(details, opt):
     # If opt.sbuild_dep_resolver is not specified, look for per package overrides
     if details["build_dep_resolver"] == None:
         if details['package'] in sbuild_dep_resolver_overrides and \
-           details['release'] in sbuild_dep_resolver_overrides[details['package']]:
-            details["build_dep_resolver"] = sbuild_dep_resolver_overrides[details['package']][details['release']]
+           details['series'] in sbuild_dep_resolver_overrides[details['package']]:
+            details["build_dep_resolver"] = sbuild_dep_resolver_overrides[details['package']][details['series']]
 
     if details["build_dep_resolver"] in ['apt', 'aptitude', 'internal']:
         buildopts.append('--build-dep-resolver=%s' % details["build_dep_resolver"])
@@ -761,7 +761,7 @@ def cmd_build_orig():
     opt.skip = True
     # Force the release as we can't auto-discover based on directory name
     if opt.release == False:
-        opt.release = details['release']
+        opt.release = details['series']
     details = prepare_build(opt)
     perform_source_build(details, opt)
     perform_binary_build(details, opt)
@@ -919,7 +919,7 @@ def cmd_check():
     (rc, report) = runcmd(command, stdout=None)
 
     # Check if we should warn about a Vcs-Bzr tag
-    if details['release'] == ust['release_devel']:
+    if details['series'] == ust['release_devel']:
         try:
             for line in open('debian/control').readlines():
                 if line.lower().startswith("vcs-bzr:"):
@@ -928,7 +928,7 @@ def cmd_check():
             pass
 
     # Print out some special-case warnings before exiting
-    emit_special_warnings(details['package'], details['release'])
+    emit_special_warnings(details['package'], details['series'])
 
     # finally check changes file as well for missing LP: # references
     changes_file = dscfile.replace(".dsc", "_source.changes")
@@ -1156,7 +1156,7 @@ def download_binaries_from_ppa(details, binary_dest, version, ppa, arch, display
         err("could not find '%s'" % sis_changes)
         sys.exit(1)
 
-    repo_dest = os.path.join(ust['package_tools_repo_base'], details['release'])
+    repo_dest = os.path.join(ust['package_tools_repo_base'], details['series'])
     binaries = glob.glob(repo_dest + '/%s_%s_*' % (details['package'], version))
     copy_binaries = False
     if len(binaries) > 0:
@@ -1170,7 +1170,7 @@ def download_binaries_from_ppa(details, binary_dest, version, ppa, arch, display
 
     if not copy_binaries:
         copy_sppa_to_repos = os.path.join(os.environ['UST'], "repo-tools", 'copy_sppa_to_repos')
-        args = [copy_sppa_to_repos, '--ppa', ppa, '--force-binaries', '--release', details['release']]
+        args = [copy_sppa_to_repos, '--ppa', ppa, '--force-binaries', '--release', details['series']]
         if arch:
             args += ['--arch', arch]
         args.append(details['package'])
@@ -1207,24 +1207,24 @@ def cmd_compare_bin():
 
         # always use debian-installer in release pocket for non-devel releases
         installer_components = 'main/debian-installer'
-        if details['release'] == ust['release_devel'] or not use_installer:
+        if details['series'] == ust['release_devel'] or not use_installer:
             installer_components = ''
         s='''
 deb [arch=%s] %s/ %s %s %s
 deb-src %s/ %s %s
-''' % (details['arch'], ust.get_sbuild_mirror(details['arch']), details['release'], components, installer_components,
-           ust.get_sbuild_srcmirror(details['arch']), details['release'], components)
+''' % (details['arch'], ust.get_sbuild_mirror(details['arch']), details['series'], components, installer_components,
+           ust.get_sbuild_srcmirror(details['arch']), details['series'], components)
 
-        if details['release'] != ust['release_devel']:
+        if details['series'] != ust['release_devel']:
             s += '''
 deb [arch=%s] %s/ %s-updates %s %s
 deb-src %s/ %s-updates %s
 deb [arch=%s] %s/ %s-security %s %s
 deb-src %s/ %s-security %s
-''' % (details['arch'], ust.get_sbuild_mirror(details['arch']), details['release'], components, installer_components,
-               ust.get_sbuild_srcmirror(details['arch']), details['release'], components,
-               details['arch'], ust.get_sbuild_mirror(details['arch']), details['release'], components, installer_components,
-               ust.get_sbuild_srcmirror(details['arch']), details['release'], components)
+''' % (details['arch'], ust.get_sbuild_mirror(details['arch']), details['series'], components, installer_components,
+               ust.get_sbuild_srcmirror(details['arch']), details['series'], components,
+               details['arch'], ust.get_sbuild_mirror(details['arch']), details['series'], components, installer_components,
+               ust.get_sbuild_srcmirror(details['arch']), details['series'], components)
 
         # as of quantal, devel packages can be waiting in -proposed, so
         # don't do this just for non-devel releases
@@ -1232,15 +1232,15 @@ deb-src %s/ %s-security %s
             s += '''
 deb [arch=%s] %s/ %s-proposed %s %s
 deb-src %s/ %s-proposed %s
-''' % (details['arch'], ust.get_sbuild_mirror(details['arch']), details['release'], components, installer_components,
-               ust.get_sbuild_srcmirror(details['arch']), details['release'], components)
+''' % (details['arch'], ust.get_sbuild_mirror(details['arch']), details['series'], components, installer_components,
+               ust.get_sbuild_srcmirror(details['arch']), details['series'], components)
 
         if 'test_repo' in details and details['test_repo']:
             s += '''
 deb [arch=%s] %s/ %s/
 deb-src %s/ %s/
-''' % (details['arch'], ust['package_tools_repo_url'], details['release'],
-              ust['package_tools_repo_url'], details['release'])
+''' % (details['arch'], ust['package_tools_repo_url'], details['series'],
+              ust['package_tools_repo_url'], details['series'])
 
         return s
 
@@ -1489,7 +1489,7 @@ Acquire::Languages "none";
         details['arch'] = arch
 
     if opt.release:
-        details['release'] = opt.release
+        details['series'] = opt.release
 
     if opt.proposed:
         details['proposed'] = True
@@ -1572,7 +1572,7 @@ def copy_to_repo(opt, details, quiet=False, source_dest=source_dest, binary_dest
     if not quiet:
         print_details(details)
 
-    repo_dest = os.path.join(repo_base, details['release'])
+    repo_dest = os.path.join(repo_base, details['series'])
 
     if opt.purge and os.path.isdir(repo_dest):
         print("Cleaning out '%s'" % (repo_dest))
@@ -1620,7 +1620,7 @@ def copy_to_repo(opt, details, quiet=False, source_dest=source_dest, binary_dest
                 print("Copying '%s'..." % (os.path.join(binary_dest, f)))
             shutil.copy(os.path.join(binary_dest, f), repo_dest)
 
-    (rc, report) = runcmd([update_repo, details['release']], stdin=sys.stdin, stdout=None)
+    (rc, report) = runcmd([update_repo, details['series']], stdin=sys.stdin, stdout=None)
     if rc != 0:
         err("failure running '%s':\n%s" % (update_repo, report))
         sys.exit(1)
@@ -1672,7 +1672,7 @@ def cmd_upload():
 
     # adjust destination if debian
     init_dest = opt.destination
-    if details['release'] == "unstable":  # XXX: add other releases
+    if details['series'] == "unstable":  # XXX: add other releases
         init_dest = "debian"
 
     details = parse_changes_file(changes_file)
@@ -1971,7 +1971,7 @@ def cmd_adt():
     }
 
     details = parse_package_details(skip_sanity = True)
-    release = details['release']
+    release = details['series']
 
     valid_releases = []
     for rel in source_map.cve_lib.releases:
@@ -2647,7 +2647,7 @@ def run_qrt_tests(opt, args, details):
         # installed (--only-upgrade) and mark them as auto installed
         # (--mark-auto)
         apt_args += "--only-upgrade "
-        release_num = BetterUbuntuDistroInfo().get_release(details['release'])
+        release_num = BetterUbuntuDistroInfo().get_release(details['series'])
         if release_num:
             if release_num > "20.04":
                 apt_args += "--mark-auto "
@@ -3418,7 +3418,7 @@ def prepare_tool_env(opt, details):
             os.environ['DEB_BUILD_HARDENING'] = '1'
 
         # Parallel builds only started with karmic and later
-        if opt.parallel and details['release'] not in ['dapper','jaunty']:
+        if opt.parallel and details['series'] not in ['dapper','jaunty']:
             _append_deb_build_options('parallel=%s' % (opt.parallel), "parallel=")
     except:
         pass
@@ -3698,7 +3698,7 @@ do
         rm -f "$SRCS"
     fi
 done
-''' % (dropped_pockets, details['release'])
+''' % (dropped_pockets, details['series'])
 
     # add the updates pocket if building a proposed package if it isn't already
     # in the sources.list
@@ -3712,7 +3712,7 @@ if ! grep -q '^[^#].*\-updates' /etc/apt/sources.list ; then
     mv -f "$SRCS" /etc/apt/sources.list
     need_update=1
 fi
-''' % (details['release'], details['release'], details['release'])
+''' % (details['series'], details['series'], details['series'])
 
     # add the proposed pocket if told to do so
     if force_proposed == True:
@@ -3725,7 +3725,7 @@ if ! grep -q '^[^#].*\-proposed' /etc/apt/sources.list ; then
     mv -f "$SRCS" /etc/apt/sources.list
     need_update=1
 fi
-''' % (details['release'], details['release'], details['release'])
+''' % (details['series'], details['series'], details['series'])
 
     # Add local repo if needed
     if details['additional-repo']:
@@ -3737,12 +3737,12 @@ echo "deb %s %s/" >> "$SRCS"
 mv -f "$SRCS" /etc/apt/sources.list
 need_update=1
 ''' % (details['additional-repo'],
-       details['additional-repo'], details['release'])
+       details['additional-repo'], details['series'])
 
     # Update components. Xenial and higher now requires universe to build
     # main packages. release_num can be None if release is unknown, like
     # when distro_info hasn't been updated yet for a new dev release
-    release_num = BetterUbuntuDistroInfo().get_release(details['release'])
+    release_num = BetterUbuntuDistroInfo().get_release(details['series'])
     if release_num and release_num < "16.04":
         allowed_comps = { 'main'       : ':main:',
                           'restricted' : ':main:restricted:',
@@ -3790,7 +3790,7 @@ fi
 ''' % (allowed_comps[details['component']], details['component'])
 
     # Hack to work around lucid dbus trying to start in the chroot
-    if details['release'] == "lucid":
+    if details['series'] == "lucid":
         build_script = build_script + '''
 dpkg-divert --divert /etc/init/dbus.conf.disabled --rename /etc/init/dbus.conf
 '''
@@ -4054,14 +4054,14 @@ def parse_package_details(release = None, skip_sanity = False, force_component =
         details.setdefault('release', details['changelog_release'])
 
     # Find base release
-    base_release = details["release"]
-    if 'esm' in base_release:
-        if 'trusty' in base_release:
-            base_release = 'trusty'
+    series = details["release"]
+    if 'esm' in series:
+        if 'trusty' in series:
+            series = 'trusty'
         else:
-            base_release = base_release.split('/')[1]
+            series = series.split('/')[1]
 
-    details['base_release'] = base_release
+    details['series'] = series
 
     # Find previous version
     previous_version = ''
@@ -4091,11 +4091,11 @@ def parse_package_details(release = None, skip_sanity = False, force_component =
     # Perform sanity checking
     if not skip_sanity:
         # Sanity check release
-        if details['changelog_release'] != details['base_release']:
-            err("oops! changelog release (%s) != '%s'" % (details['changelog_release'], details['base_release']))
+        if details['changelog_release'] != details['series']:
+            err("oops! changelog release (%s) != '%s'" % (details['changelog_release'], details['series']))
             sys.exit(1)
         # Sanity check pocket
-        if (details['release'] != ust['release_devel']) and (details['release'] != 'unstable') and \
+        if (details['series'] != ust['release_devel']) and (details['series'] != 'unstable') and \
                 (details['pocket'] != 'security') and \
                 (details['pocket'] != 'backports') and \
                 (details['pocket'] != 'proposed'):