Merge ~the-meulengracht/snappy-hwe-snaps/+git/jenkins-jobs:master into ~snappy-hwe-team/snappy-hwe-snaps/+git/jenkins-jobs:master

[Philip Meulengracht]

This MP contains the following:
 - Support for focal
 - Migration to python3

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:663e605c99a54e7065b741d5db521abcd7ae7718
https://jenkins.canonical.com/system-enablement/job/snappy-hwe-snaps-snap-build-prepare/719/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/snappy-hwe-snaps-snap-build/ARCHITECTURE=amd64/742
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/snappy-hwe-snaps-snap-build/ARCHITECTURE=arm64/742
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/snappy-hwe-snaps-snap-build/ARCHITECTURE=armhf/742
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/snappy-hwe-snaps-snap-build/ARCHITECTURE=i386/742
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/snappy-hwe-snaps-snap-docs/1523
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/snappy-hwe-snaps-snap-cleanup/1265
    None: https://jenkins.canonical.com/system-enablement/job/snappy-hwe-snaps-snap-update-mp/1332/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/snappy-hwe-snaps-snap-build-prepare/719/rebuild

review: Approve (continuous-integration)

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

Thanks for this! I've a couple of comments.

Also, I have run flake8 on the python scripts, that should help to find issues in the migration to python3. See https://paste.ubuntu.com/p/cBMsq89xvZ/ . I think most things are harmless, except maybe:

./tools/vote-on-merge-proposal.py:76:10: E999 SyntaxError: Missing parentheses in call to 'print'. Did you mean print(...)?

Anyway, it is a good opportunity to clean-up these warnings, although maybe you can ignore the "line too long" ones.

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:d20dc2f3493ffbf7c1e1bbaf516dfe1689cfbf64
https://jenkins.canonical.com/system-enablement/job/snappy-hwe-snaps-snap-build-prepare/720/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/snappy-hwe-snaps-snap-build/ARCHITECTURE=amd64/743
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/snappy-hwe-snaps-snap-build/ARCHITECTURE=arm64/743
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/snappy-hwe-snaps-snap-build/ARCHITECTURE=armhf/743
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/snappy-hwe-snaps-snap-build/ARCHITECTURE=i386/743
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/snappy-hwe-snaps-snap-docs/1524
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/snappy-hwe-snaps-snap-cleanup/1266
    None: https://jenkins.canonical.com/system-enablement/job/snappy-hwe-snaps-snap-update-mp/1333/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/snappy-hwe-snaps-snap-build-prepare/720/rebuild

review: Approve (continuous-integration)

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

LGTM, thanks for the changes!

review: Approve

diff --git a/3rdparty/repo b/3rdparty/repo
index f9eb9e8..c069798 100644
--- a/3rdparty/repo
+++ b/3rdparty/repo
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # repo default configuration
 #
diff --git a/jobs/infrastructure/credentials-1-launchpad.py b/jobs/infrastructure/credentials-1-launchpad.py
index 59dd706..61e36de 100644
--- a/jobs/infrastructure/credentials-1-launchpad.py
+++ b/jobs/infrastructure/credentials-1-launchpad.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # Copyright (C) 2016 Canonical Ltd
 #
@@ -82,5 +82,6 @@ def get_launchpad(cred_path, launchpadlib_dir=None):
                                 authorization_engine=authorization_engine,
                                 launchpadlib_dir=launchpadlib_dir)
 
+
 if __name__ == '__main__':
     get_launchpad("{credentials_path}")
diff --git a/jobs/infrastructure/prepare-0-install.sh b/jobs/infrastructure/prepare-0-install.sh
index d50e5a9..afffe90 100644
--- a/jobs/infrastructure/prepare-0-install.sh
+++ b/jobs/infrastructure/prepare-0-install.sh
@@ -16,6 +16,9 @@
 
 set -ex
 
+JENKINS_JOBS_GIT_REPO="{jobs-git-repo}"
+JENKINS_JOBS_GIT_REPO_BRANCH="{jobs-git-repo-branch}"
+
 # Use correct home and preserve environment variables when calling `sudo ...`
 SUDO="sudo -E -H"
 
@@ -26,24 +29,27 @@ $SUDO apt-get install --yes software-properties-common
 # build tools as used in Launchpad
 $SUDO add-apt-repository --yes ppa:launchpad/buildd-staging
 $SUDO add-apt-repository --yes ppa:jenkaas-hackers/tools
-$SUDO add-apt-repository --yes ppa:snappy-hwe-team/ci-tools
+
+version=$(lsb_release -sc)
+if [ "$version" = "xenial" ]; then
+    $SUDO add-apt-repository --yes ppa:snappy-hwe-team/ci-tools
+fi
 
 $SUDO apt-get update
 
 echo "postfix postfix/mailname string $(hostname)" | $SUDO debconf-set-selections
 echo  "postfix postfix/main_mailer_type string 'Internet Site'" | $SUDO debconf-set-selections
 
+# install shared packages
 $SUDO apt-get install --yes \
     git \
-    python \
-    python-launchpadlib \
-    python-bzrlib \
-    python-lockfile \
-    python-yaml \
-    python-jenkins \
-    python-requests \
-    python-git \
-    python-pip \
+    python3 \
+    python3-launchpadlib \
+    python3-lockfile \
+    python3-yaml \
+    python3-jenkins \
+    python3-requests \
+    python3-git \
     python3-debian \
     python3-pip \
     tarmac \
@@ -58,7 +64,6 @@ $SUDO apt-get install --yes \
     jq \
     postfix \
     abootimg \
-    android-tools-fsutils \
     golang-go \
     dpkg-dev \
     devscripts \
@@ -66,31 +71,28 @@ $SUDO apt-get install --yes \
     bzr \
     dh-systemd \
     expect \
-    tcl \
-    {install_packages}
+    tcl {install_packages}
+
+# install additional packages based on distro
+if [ "$version" != "xenial" ]; then
+    $SUDO apt-get install --yes \
+        android-sdk-libsparse-utils \
+        android-sdk-ext4-utils
+else
+    $SUDO apt-get install --yes \
+        android-tools-fsutils
+fi
 
 # Install snaps
 $SUDO snap install snapcraft
 $SUDO snap install yq
 
-# Install additional sofware from the python package database
-$SUDO pip2 install --upgrade pip
-$SUDO pip2 install jenkins-job-builder pid
+export http_proxy="{http_proxy}"
+export https_proxy="{https_proxy}"
 
 # documentation-builder is normally available as a snap but we can't
 # use snaps yet on our agents.
 $SUDO pip3 install --upgrade pip
+$SUDO pip3 install --ignore-installed PyYAML
+$SUDO pip3 install jenkins-job-builder pid
 $SUDO pip3 install ubuntudesign.documentation-builder
-
-# Now http & https proxy is not needed, it will in fact interfere with curl
-# below
-
-unset http_proxy
-unset https_proxy
-
-# Fetch and install latest repo; we still can't use snaps reliably on our
-# agents so we go with the upstream release.
-$SUDO mkdir -p /usr/local/bin
-curl $JENKINS_JOBS_GIT_REPO/plain/3rdparty/repo?h=$JENKINS_JOBS_GIT_REPO_BRANCH | $SUDO tee /usr/local/bin/repo
-$SUDO mv $HOME/repo /usr/local/bin/repo
-$SUDO chmod +x /usr/local/bin/repo
diff --git a/tools/automerge-mps.py b/tools/automerge-mps.py
index 2e1e0b6..150da07 100755
--- a/tools/automerge-mps.py
+++ b/tools/automerge-mps.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
 #
 # Copyright (C) 2016 Canonical Ltd
diff --git a/tools/delete-ci-repo.py b/tools/delete-ci-repo.py
index 35c762d..3a04613 100755
--- a/tools/delete-ci-repo.py
+++ b/tools/delete-ci-repo.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 #
 # Copyright (C) 2017 Canonical Ltd
 #
@@ -14,8 +14,6 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-from launchpadlib.launchpad import Launchpad
-
 from argparse import ArgumentParser
 
 import se_utils
diff --git a/tools/se_utils/__init__.py b/tools/se_utils/__init__.py
index 64cc0cc..e78eb6b 100644
--- a/tools/se_utils/__init__.py
+++ b/tools/se_utils/__init__.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
 #
 # Copyright (C) 2016 Canonical Ltd
@@ -18,7 +18,6 @@
 import atexit
 import sys
 import time
-import logging
 import os
 import re
 import yaml
@@ -28,11 +27,13 @@ from lazr.restfulclient.errors import HTTPError
 from launchpadlib.launchpad import Launchpad
 from launchpadlib.credentials import UnencryptedFileCredentialStore
 
+
 class LaunchpadVote():
     APPROVE = 'Approve'
     DISAPPROVE = 'Disapprove'
     NEEDS_FIXING = 'Needs Fixing'
 
+
 ACCESS_TOKEN_POLL_TIME = 10
 WAITING_FOR_USER = """Open this link:
 {}
@@ -58,7 +59,7 @@ class AuthorizeRequestTokenWithConsole(RequestTokenAuthorizationEngine):
 
         """
         authorization_url = self.authorization_url(request_token)
-        print WAITING_FOR_USER.format(authorization_url)
+        print(WAITING_FOR_USER.format(authorization_url))
         # if we don't flush we may not see the message
         sys.stdout.flush()
         while credentials.access_token is None:
@@ -67,7 +68,7 @@ class AuthorizeRequestTokenWithConsole(RequestTokenAuthorizationEngine):
                 credentials.exchange_request_token_for_access_token(
                     self.web_root)
                 break
-            except HTTPError, e:
+            except HTTPError as e:
                 if e.response.status == 403:
                     # The user decided not to authorize this
                     # application.
@@ -79,6 +80,7 @@ class AuthorizeRequestTokenWithConsole(RequestTokenAuthorizationEngine):
                     # There was an error accessing the server.
                     raise e
 
+
 # launchpadlib is not thread/process safe so we are creating launchpadlib
 # cache in /tmp per process which gets cleaned up at the end
 # see also lp:459418 and lp:1025153
@@ -97,8 +99,8 @@ def get_launchpad(launchpadlib_dir=None, credential_store_path=None, lp_app=None
     the default """
     store = UnencryptedFileCredentialStore(credential_store_path)
     authorization_engine = AuthorizeRequestTokenWithConsole(lp_env, lp_app)
-    lib_dir=launchpad_cachedir
-    if launchpadlib_dir != None:
+    lib_dir = launchpad_cachedir
+    if launchpadlib_dir is not None:
         lib_dir = launchpadlib_dir
     return Launchpad.login_with(lp_app, lp_env,
                                 credential_store=store,
@@ -106,6 +108,7 @@ def get_launchpad(launchpadlib_dir=None, credential_store_path=None, lp_app=None
                                 launchpadlib_dir=lib_dir,
                                 version='devel')
 
+
 # Load configuration for the current agent we're running on. All agents were
 # provisioned when they were setup with a proper configuration. See
 # https://wiki.canonical.com/InformationInfrastructure/Jenkaas/UserDocs for
@@ -121,12 +124,14 @@ def load_config():
     print("ERROR: No config file found")
     sys.exit(1)
 
+
 # Return a configuration option from the agent configuration specified by the
 # name argument.
 def get_config_option(name):
     config = load_config()
     return config[name]
 
+
 def get_branch_handle_from_url(lp_handle, url):
     """ Return a branch/repo handle for the given url.
     Returns a launchpad branch or git repository handle for the given url.
@@ -147,6 +152,7 @@ def get_branch_handle_from_url(lp_handle, url):
         print('fetching branch: ' + name)
         return lp_handle.branches.getByUrl(url=name)
 
+
 def get_branch_from_mp(merge_proposal):
     """Return a link to branch given a link to a merge proposal.
 
@@ -157,11 +163,12 @@ def get_branch_from_mp(merge_proposal):
 
     :param merge_proposal: url of a launchpad merge proposal
     """
-    m = re.search('(.*)\+merge/[0-9]+$', merge_proposal)
+    m = re.search(r'(.*)\+merge/[0-9]+$', merge_proposal)
     if m:
         return m.group(1)
     return None
 
+
 def get_mp_handle_from_url(lp_handle, merge_proposal_link):
     """ Get launchpad handle for merge proposal given a merge proposal URL.
 
@@ -173,7 +180,7 @@ def get_mp_handle_from_url(lp_handle, merge_proposal_link):
         print('Unable to get branch link from merge proposal link.')
         return None
 
-    branch =  get_branch_handle_from_url(lp_handle, branch_link)
+    branch = get_branch_handle_from_url(lp_handle, branch_link)
     if not branch:
         print('Branch {} does not exist'.format(branch_link))
         return None
@@ -187,6 +194,7 @@ def get_mp_handle_from_url(lp_handle, merge_proposal_link):
 
     return None
 
+
 def get_vote_subject(mp):
     """Given a mp handle return a subject for the vote message
 
@@ -209,4 +217,4 @@ def get_vote_subject(mp):
     else:
         return 'Re: [Merge] {} into {}'.format(
             mp.source_branch.display_name,
-            mp.target_branch.display_name)
 \ No newline at end of file
+            mp.target_branch.display_name)
diff --git a/tools/shyaml b/tools/shyaml
index e4618ec..ea3db4f 100755
--- a/tools/shyaml
+++ b/tools/shyaml
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # Taken from upstream git repository https://github.com/0k/shyaml
 # at revision d77e30599a0971c51896ef97d21883550e7e9979
diff --git a/tools/trigger-ci.py b/tools/trigger-ci.py
index 65f3b3e..9267a16 100755
--- a/tools/trigger-ci.py
+++ b/tools/trigger-ci.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
 #
 # Copyright (C) 2016 Canonical Ltd
diff --git a/tools/unstage-from-manifest.py b/tools/unstage-from-manifest.py
index e189492..e3f79ec 100755
--- a/tools/unstage-from-manifest.py
+++ b/tools/unstage-from-manifest.py
@@ -22,7 +22,6 @@
 # store.
 
 import sys
-import yaml
 from tools import yaml_utils
 
 
@@ -34,7 +33,7 @@ def remove_from_staged(pkg_list_path, manifest_path, out_manifest_path):
     try:
         with open(pkg_list_path) as pkg_list_f:
             pkg_list = pkg_list_f.read().splitlines()
-    except FileNotFoundError as e:
+    except FileNotFoundError:
         print(pkg_list_path, 'not found, just copying manifest')
         pkg_list = []
 
diff --git a/tools/update-merge-proposal.py b/tools/update-merge-proposal.py
index 0054659..261fa6d 100755
--- a/tools/update-merge-proposal.py
+++ b/tools/update-merge-proposal.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
 #
 # Copyright (C) 2017 Canonical Ltd
@@ -15,14 +15,6 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import atexit
-import sys
-import time
-import logging
-import os
-import yaml
-import re
-from shutil import rmtree
 from argparse import ArgumentParser
 
 import se_utils
@@ -54,7 +46,7 @@ content = content.decode('string_escape')
 if not args['approve'] and not args['disapprove']:
     parser.error('neither approved or disapproved the MP')
 
-vote=""
+vote = ""
 if args["approve"]:
     vote = se_utils.LaunchpadVote.APPROVE
 elif args["disapprove"]:
diff --git a/tools/vote-on-merge-proposal.py b/tools/vote-on-merge-proposal.py
index bec6da5..96838a7 100755
--- a/tools/vote-on-merge-proposal.py
+++ b/tools/vote-on-merge-proposal.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
 #
 # Copyright (C) 2016 Canonical Ltd
@@ -20,7 +20,6 @@ import sys
 import time
 import logging
 import os
-import yaml
 import re
 from shutil import rmtree
 from argparse import ArgumentParser
@@ -29,7 +28,7 @@ from lazr.restfulclient.errors import HTTPError
 from launchpadlib.launchpad import Launchpad
 from launchpadlib.credentials import UnencryptedFileCredentialStore
 from jlp import get_config_option
-from jlp import launchpadutils, jenkinsutils, logger
+from jlp import jenkinsutils
 
 logger = logging.getLogger('jenkins-launchpad-plugin')
 stdout_handler = logging.StreamHandler(stream=sys.stdout)
@@ -73,7 +72,7 @@ class AuthorizeRequestTokenWithConsole(RequestTokenAuthorizationEngine):
 
         """
         authorization_url = self.authorization_url(request_token)
-        print WAITING_FOR_USER.format(authorization_url)
+        print(WAITING_FOR_USER.format(authorization_url))
         # if we don't flush we may not see the message
         sys.stdout.flush()
         while credentials.access_token is None:
@@ -82,7 +81,7 @@ class AuthorizeRequestTokenWithConsole(RequestTokenAuthorizationEngine):
                 credentials.exchange_request_token_for_access_token(
                     self.web_root)
                 break
-            except HTTPError, e:
+            except HTTPError as e:
                 if e.response.status == 403:
                     # The user decided not to authorize this
                     # application.
@@ -110,6 +109,7 @@ def get_launchpad(launchpadlib_dir=None):
                                 launchpadlib_dir=launchpadlib_dir,
                                 version='devel')
 
+
 def get_branch_handle_from_url(lp_handle, url):
     """ Return a branch/repo handle for the given url.
     Returns a launchpad branch or git repository handle for the given url.
@@ -130,6 +130,7 @@ def get_branch_handle_from_url(lp_handle, url):
         logger.debug('fetching branch: ' + name)
         return lp_handle.branches.getByUrl(url=name)
 
+
 def get_branch_from_mp(merge_proposal):
     """Return a link to branch given a link to a merge proposal.
 
@@ -140,11 +141,12 @@ def get_branch_from_mp(merge_proposal):
 
     :param merge_proposal: url of a launchpad merge proposal
     """
-    m = re.search('(.*)\+merge/[0-9]+$', merge_proposal)
+    m = re.search(r'(.*)\+merge/[0-9]+$', merge_proposal)
     if m:
         return m.group(1)
     return None
 
+
 def get_mp_handle_from_url(lp_handle, merge_proposal_link):
     """ Get launchpad handle for merge proposal given a merge proposal URL.
 
@@ -156,7 +158,7 @@ def get_mp_handle_from_url(lp_handle, merge_proposal_link):
         logger.error('Unable to get branch link from merge proposal link.')
         return None
 
-    branch =  get_branch_handle_from_url(lp_handle, branch_link)
+    branch = get_branch_handle_from_url(lp_handle, branch_link)
     if not branch:
         logger.debug('Branch {} does not exist'.format(branch_link))
         return None
@@ -170,11 +172,13 @@ def get_mp_handle_from_url(lp_handle, merge_proposal_link):
 
     return None
 
+
 class LaunchpadVote():
     APPROVE = 'Approve'
     DISAPPROVE = 'Disapprove'
     NEEDS_FIXING = 'Needs Fixing'
 
+
 def get_vote_subject(mp):
     """Given a mp handle return a subject for the vote message
 
@@ -241,6 +245,7 @@ def disapprove_mp(mp, revision, build_url, reason=None):
                      subject=get_vote_subject(mp),
                      content=content)
 
+
 # launchpadlib is not thread/process safe so we are creating launchpadlib
 # cache in /tmp per process which gets cleaned up at the end
 # see also lp:459418 and lp:1025153
@@ -265,7 +270,4 @@ reason = ''
 if overal_status == 'PASSED':
     approve_mp(mp, args['revision'], args['build_url'])
 else:  # status == False corresponds to NOT 'PASSED'
-    disapprove_mp(mp,
-                                    args['revision'],
-                                    args['build_url'],
-                                    reason)
+    disapprove_mp(mp, args['revision'], args['build_url'], reason)