Merge ~fourdollars/pc-enablement/+git/oem-scripts:master into ~oem-solutions-engineers/pc-enablement/+git/oem-scripts:master

[Shih-Yuan Lee]

linux-image-4.15.0-1104-oem was pending publication so this commit aims to catch it.

$ pkg-list --apt-dir /tmp/apt.W1fko1dluV linux-oem --exclude /tmp/apt.W1fko1dluV/all.list
amd64-microcode 3.20191021.1+really3.20181128.1~ubuntu0.18.04.1
intel-microcode 3.20201110.0ubuntu0.18.04.2
iucode-tool 2.3.1-1
linux-firmware 1.173.19
linux-headers-4.15.0-1104-oem 4.15.0-1104.115
linux-headers-oem 4.15.0.1104.108
linux-image-oem 4.15.0.1104.108
linux-oem 4.15.0.1104.108
linux-oem-headers-4.15.0-1104 4.15.0-1104.115

$ pkg-list --apt-dir /tmp/apt.W1fko1dluV linux-oem --exclude /tmp/apt.W1fko1dluV/all.list --fail-unavailable
<ERROR> linux-image-4.15.0-1104-oem is unavailable.

$ run-autopkgtest lxc focal -t pkg-list -C
...
pkg-list PASS

autopkgtest-build-lxc-23ad9ac-focal-complete.log https://paste.ubuntu.com/p/gBX6gfMKX5/

$ grep ERROR autopkgtest-build-lxc-23ad9ac-focal-complete.log -A 1
<ERROR> python3:any is unavailable.
It is usual to see some Debian package still unavailable in proposed channel.

[Alex Tu (alextu)]

I'm thinking of the possibility of a unit test for pkg-list to capture all scenario of this script like what I did in pc-sanity package [1].

So, that we have confidence that any change would not impact expected behavior and the new scenarios could also be reviewed and verified easily through unit tests.

[1] https://git.launchpad.net/plainbox-provider-pc-sanity/tree/test

[Alex Tu (alextu)]

LGTM.

[Shih-Yuan Lee (fourdollars)]

$ cat autopkgtest-build-lxc-7eeaacd-focal-summary.log
mir-bug-verification SKIP Test requires machine-level isolation but testbed does not provide that
autopkgtest-collect-credentials PASS
autopkgtest-oem-scripts-auto PASS
pkg-somerville-meta PASS
pkg-sutton-meta PASS
pkg-stella-meta PASS
launchpad-api PASS
get-private-ppa PASS
jq-lp PASS
mir-bug PASS
run-autopkgtest PASS
bug-bind PASS
pkg-list PASS
setup-apt-dir PASS
git-url-insteadof-setting PASS

autopkgtest-build-lxc-7eeaacd-focal-complete.log https://paste.ubuntu.com/p/WVJ53xfwyC/

[Shih-Yuan Lee (fourdollars)]

$ ./oem-meta-packages subscribe
<INFO> Using oem-scripts login
<INFO> Loading KWallet
<INFO> Loading SecretService
<INFO> Loading Windows
<INFO> Loading chainer
<INFO> Loading macOS
<INFO> Checking the subscriptions for oem-somerville-meera-tgl-meta...
<INFO> ubuntu/oem-somerville-meera-tgl-meta has subscribed oem-solutions-engineers.
<INFO> Checking the subscriptions for oem-sutton.bachman-banaing-meta...
<INFO> ubuntu/oem-sutton.bachman-banaing-meta has subscribed oem-solutions-engineers.
<INFO> Checking the subscriptions for oem-sutton.newell-cadyna-meta...
<INFO> ubuntu/oem-sutton.newell-cadyna-meta has subscribed oem-solutions-engineers.
<INFO> Checking the subscriptions for oem-sutton.newell-adalbrechta-meta...
<INFO> ubuntu/oem-sutton.newell-adalbrechta-meta has subscribed oem-solutions-engineers.
<INFO> Checking the subscriptions for oem-sutton.bachman-banner-meta...
<INFO> ubuntu/oem-sutton.bachman-banner-meta has subscribed oem-solutions-engineers.
<INFO> Checking the subscriptions for oem-sutton.newell-cade-meta...
<INFO> ubuntu/oem-sutton.newell-cade-meta has subscribed oem-solutions-engineers.
<INFO> Checking the subscriptions for oem-sutton.newell-cadence-meta...
<INFO> ubuntu/oem-sutton.newell-cadence-meta has subscribed oem-solutions-engineers.
<INFO> Checking the subscriptions for oem-sutton.simon-adken-meta...
<INFO> ubuntu/oem-sutton.simon-adken-meta has subscribed oem-solutions-engineers.
<INFO> Checking the subscriptions for oem-somerville-beric-icl-meta...
<INFO> ubuntu/oem-somerville-beric-icl-meta has subscribed oem-solutions-engineers.
<INFO> Checking the subscriptions for oem-sutton.newell-cameo-meta...
<INFO> ubuntu/oem-sutton.newell-cameo-meta has subscribed oem-solutions-engineers.
<INFO> Checking the subscriptions for oem-somerville-samwell-tgl-meta...
<INFO> ubuntu/oem-somerville-samwell-tgl-meta has subscribed oem-solutions-engineers.
<INFO> Checking the subscriptions for oem-sutton.newell-ace-meta...
<INFO> ubuntu/oem-sutton.newell-ace-meta has subscribed oem-solutions-engineers.
<INFO> Checking the subscriptions for oem-somerville-bulbasaur-meta...
<INFO> ubuntu/oem-somerville-bulbasaur-meta has subscribed oem-solutions-engineers.
<INFO> Checking the subscriptions for oem-sutton.bachman-bale-meta...
<INFO> ubuntu/oem-sutton.bachman-bale-meta has subscribed oem-solutions-engineers.
<INFO> Checking the subscriptions for oem-somerville-three-eyed-raven-meta...
<INFO> ubuntu/oem-somerville-three-eyed-raven-meta has subscribed oem-solutions-engineers.
<INFO> Checking the subscriptions for oem-somerville-caterpie-meta...
<INFO> ubuntu/oem-somerville-caterpie-meta has subscribed oem-solutions-engineers.
<INFO> Checking the subscriptions for oem-somerville-melisa-meta...
<INFO> ubuntu/oem-somerville-melisa-meta has subscribed oem-solutions-engineers.
<INFO> Checking the subscriptions for oem-qemu-meta...
<INFO> ubuntu/oem-qemu-meta has subscribed oem-solutions-engineers.
<INFO> Checking the subscriptions for oem-sutton.bachman-baker-meta...
<INFO> ubuntu/oem-sutton.bachman-baker-meta has subscribed oem-solutions-engineers.
<INFO> Checking the subscriptions for oem-sutton.newell-candice-meta...
<INFO> ubuntu/oem-sutto...

[Yuan-Chen Cheng (ycheng-twn)]

for change related to pkg-list, they are LGTM.

[Yuan-Chen Cheng (ycheng-twn)]

LGTM

review: Approve

[Yuan-Chen Cheng (ycheng-twn)]

note: talk in the f2f with $4, I think there could be some other better place to do what oem-meta-packages is doing.

However, he can't think any other script fits. Give so, approve it.

diff --git a/debian/changelog b/debian/changelog
index 4545eb5..20ad464 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,16 @@
+oem-scripts (0.71) UNRELEASED; urgency=medium
+
+  * pkg-list: Add an option '--fail-unavailable' to return unavailable error.
+  * pkg-list: Refine the checking for depends.
+  * oem-meta-packages,
+    oem_scripts/LaunchpadLogin.py,
+    setup.py,
+    tests/test_flake8,
+    tests/test_pep8: Add oem-meta-packages to subscribe the bugs of oem meta
+    packages.
+
+ -- Shih-Yuan Lee (FourDollars) <[e-mail]>  Fri, 04 Dec 2020 14:50:37 +0800
+
 oem-scripts (0.70) focal; urgency=medium
 
   * mir-bug,
diff --git a/debian/tests/pkg-list b/debian/tests/pkg-list
index ae02e8d..92dfe33 100644
--- a/debian/tests/pkg-list
+++ b/debian/tests/pkg-list
@@ -1,9 +1,13 @@
 #!/bin/bash
 
+echo "=== Enable focal proposed channel ==="
 APTDIR=$(setup-apt-dir.sh -c focal -m http://uk.archive.ubuntu.com/ubuntu -p | tail -n1)
 
 echo "=== ubuntu-desktop --recommends ==="
 pkg-list --apt-dir "$APTDIR" ubuntu-desktop --recommends | tee ubuntu-desktop.list
+if ! pkg-list --apt-dir "$APTDIR" ubuntu-desktop --recommends --fail-unavailable; then
+    echo "It is usual to see some Debian package still unavailable in proposed channel."
+fi
 
 echo "=== dkms ==="
 pkg-list --apt-dir "$APTDIR" dkms --exclude ubuntu-desktop.list | tee dkms.list
diff --git a/oem-meta-packages b/oem-meta-packages
new file mode 100755
index 0000000..2deb366
--- /dev/null
+++ b/oem-meta-packages
@@ -0,0 +1,151 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+# Copyright (C) 2020  Canonical Ltd.
+# Author: Shih-Yuan Lee (FourDollars) <[e-mail]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+import argparse
+import difflib
+import lazr
+import os
+import types
+
+from apt import apt_pkg
+from logging import info, warning, error, debug  # , critical
+from oem_scripts.LaunchpadLogin import LaunchpadLogin
+from oem_scripts.logging import setup_logging
+
+parser = argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter,
+                                 epilog="""
+Make all bugs of all oem meta packages be subscribed by oem-solutions-engineers.
+ (search current apt source for package lists)
+
+Check "To all bugs in oem-qemu-meta in Ubuntu:" on https://launchpad.net/ubuntu/+source/oem-qemu-meta/+subscribe for example.
+
+examples:
+    oem-meta-packages subscribe""")
+
+parser.add_argument("-d", "--debug",
+                    help="print debug messages", action="store_true")
+parser.add_argument("-q", "--quiet",
+                    help="Don't print info messages", action="store_true")
+parser.add_argument("-v", "--verbose", action="store_true",
+                    help="print verbose messages")
+parser.add_argument("--apt-dir",
+                    type=str,
+                    help="specify the dir for apt")
+
+subparsers = parser.add_subparsers(dest="subcommand")
+
+subscribe = subparsers.add_parser('subscribe', help='[-h] [--yes]')
+subscribe.add_argument("--yes",
+                       help="Say yes for all prompts.", action="store_true")
+
+unsubscribe = subparsers.add_parser('unsubscribe', help='[-h] [--yes] pkgName')
+unsubscribe.add_argument("pkgName", type=str,
+                         help="Specify the package name to unsubscribe.")
+unsubscribe.add_argument("--yes",
+                         help="Say yes for all prompts.", action="store_true")
+
+args = parser.parse_args()
+
+setup_logging(debug=args.debug, quiet=args.quiet)
+
+if args.subcommand:
+    login = LaunchpadLogin()
+    lp = login.lp
+
+if args.apt_dir:
+    apt_pkg.init_config()
+    if args.debug:
+        old = apt_pkg.config.dump()
+    apt_pkg.config.set("Dir", args.apt_dir)
+    apt_pkg.config.set("Dir::State::status", os.path.join(args.apt_dir, "var/lib/dpkg/status"))
+    if args.debug:
+        new = apt_pkg.config.dump()
+        d = difflib.Differ()
+        diff = d.compare(old.split('\n'), new.split('\n'))
+        for line in diff:
+            debug(line.strip())
+    apt_pkg.init_system()
+
+
+def yes_or_ask(yes: bool, message: str) -> bool:
+    if yes:
+        print(f"> \033[1;34m{message}\033[1;0m (y/n) y")
+        return True
+    while True:
+        res = input(f"> \033[1;34m{message}\033[1;0m (y/n) ").lower()
+        if res not in {"y", "n"}:
+            continue
+        if res == 'y':
+            return True
+        else:
+            return False
+
+
+def _debug_obj(pkg) -> None:
+
+    if not args.debug:
+        return
+
+    debug(dir(pkg))
+
+    for attr in dir(pkg):
+        if not attr.startswith('__'):
+            if not isinstance(pkg.__getattribute__(attr), types.BuiltinFunctionType):
+                debug(f"{attr}: {pkg.__getattribute__(attr)}")
+
+
+cache = apt_pkg.Cache(progress=None)
+
+if args.subcommand == 'subscribe':
+    for pkg in cache.packages:
+        name = pkg.name
+        if not name.startswith('oem-') or not name.endswith('-meta'):
+            continue
+        info(f"Checking the subscriptions for {name}...")
+        source = lp.distributions['ubuntu'].getSourcePackage(name=name)
+        if 'oem-solutions-engineers' in map(lambda x: x.subscriber.name, source.getSubscriptions()):
+            info(f"ubuntu/{name} has subscribed oem-solutions-engineers.")
+            continue
+        warning(f"ubuntu/{name} didn't subscribe oem-solutions-engineers yet.")
+        if yes_or_ask(args.yes, f"Would you like to subscribe 'oem-solutions-engineers' for ubuntu/{name}?"):
+            try:
+                # When a person is subscribed to a source package, one actually subscribe all bugs for it.
+                source.addBugSubscription(subscriber=lp.people['oem-solutions-engineers'])
+            except lazr.restfulclient.errors.Unauthorized as e:
+                error(f"{lp.me.name} does not have permission to subscribe oem-solutions-engineers.")
+                if args.verbose:
+                    print(e)
+                exit(1)
+elif args.subcommand == 'unsubscribe':
+    source = lp.distributions['ubuntu'].getSourcePackage(name=args.pkgName)
+    subscriptions = source.getSubscriptions()
+    for subscription in subscriptions:
+        if subscription.subscriber.name == 'oem-solutions-engineers':
+            info(f"ubuntu/{args.pkgName} has subscribed oem-solutions-engineers.")
+            if yes_or_ask(args.yes, f"Would you like to unsubscribe 'oem-solutions-engineers' for ubuntu/{args.pkgName}?"):
+                try:
+                    source.removeBugSubscription(subscriber=lp.people['oem-solutions-engineers'])
+                except lazr.restfulclient.errors.Unauthorized as e:
+                    error(f"{lp.me.name} does not have permission to unsubscribe oem-solutions-engineers.")
+                    if args.verbose:
+                        print(e)
+                    exit(1)
+            exit(0)
+else:
+    parser.print_help()
diff --git a/oem_scripts/LaunchpadLogin.py b/oem_scripts/LaunchpadLogin.py
index 25c314f..0583ad2 100644
--- a/oem_scripts/LaunchpadLogin.py
+++ b/oem_scripts/LaunchpadLogin.py
@@ -67,7 +67,7 @@ class LaunchpadLogin():
                 logging.info("Using anonymously login")
                 self.lp = Launchpad.login_anonymously(application_name, service_root)
             elif ":" in launchpad_token:
-                oauth_token, oauth_token_secret, oauth_consumer_key = launchpad_token.split(":")
+                oauth_token, oauth_token_secret, oauth_consumer_key = launchpad_token.split(":", maxsplit=2)
                 self.lp = Launchpad.login(oauth_consumer_key,
                                           oauth_token,
                                           oauth_token_secret,
diff --git a/pkg-list b/pkg-list
index ac844db..42351d5 100755
--- a/pkg-list
+++ b/pkg-list
@@ -25,7 +25,7 @@ import sys
 import types
 
 from apt import apt_pkg
-from logging import debug, error, critical  # info, warning
+from logging import debug, error, critical, info  # , warning
 from urllib.parse import urljoin
 
 
@@ -59,6 +59,9 @@ parser.add_argument("--suggests",
 parser.add_argument("--non-installed",
                     action="store_true",
                     help="only get non-installed packages per check current running environments")
+parser.add_argument("--fail-unavailable",
+                    action="store_true",
+                    help="Return error when any package is unavailable.")
 parser.add_argument("--exclude",
                     metavar='pkg.list',
                     type=argparse.FileType('r', encoding='UTF-8'),
@@ -133,9 +136,18 @@ def get_depends(pkg_name: str, depends_list: list, recommends: bool, suggests: b
 
     pkg = cache[pkg_name]
 
+    info(f"check {pkg_name}")
     _debug_pkg(pkg)
 
+    if args.fail_unavailable and not pkg.has_versions:
+        error(f"{pkg_name} is unavailable.")
+        exit(1)
+
     for version in pkg.version_list:
+        if pkg_name in map(lambda x: x[0], depends_list):
+            continue
+        info(f"version {version.ver_str}")
+        _debug_pkg(version)
         url = None
 
         if not version.downloadable:
@@ -160,22 +172,21 @@ def get_depends(pkg_name: str, depends_list: list, recommends: bool, suggests: b
         item = (pkg_name, version.ver_str, url, record.hashes.find("MD5Sum"), record.hashes.find("SHA1"), record.hashes.find("SHA256"))
         depends_list.append(item)
 
-        if 'Depends' not in version.depends_list_str:
-            continue
-
-        for depends in version.depends_list_str['Depends']:
-            (name, ver, _) = depends[0]
-            get_depends(name, depends_list, recommends=recommends, suggests=suggests, non_installed=non_installed)
-
-        if recommends and 'Recommends' in version.depends_list_str:
-            for depends in version.depends_list_str['Recommends']:
-                (name, ver, _) = depends[0]
-                get_depends(name, depends_list, recommends=recommends, suggests=suggests, non_installed=non_installed)
-
-        if suggests and 'Suggests' in version.depends_list_str:
-            for depends in version.depends_list_str['Suggests']:
-                (name, ver, _) = depends[0]
-                get_depends(name, depends_list, recommends=recommends, suggests=suggests, non_installed=non_installed)
+        for target in ('PreDepends', 'Depends', 'Recommends', 'Suggests'):
+            if target == 'Recommends' and not recommends:
+                continue
+            if target == 'Suggests' and not suggests:
+                continue
+            if target not in version.depends_list_str:
+                continue
+            for depends in version.depends_list_str[target]:
+                for depend in depends:
+                    (name, ver, _) = depend
+                    pkg = cache[name]
+                    if name in map(lambda x: x[0], depends_list):
+                        continue
+                    info(f"{pkg_name} {target} {name}")
+                    get_depends(name, depends_list, recommends=recommends, suggests=suggests, non_installed=non_installed)
 
 
 if args.apt_dir:
diff --git a/setup.py b/setup.py
index c598ec5..7984690 100644
--- a/setup.py
+++ b/setup.py
@@ -45,6 +45,7 @@ Also there is a meta package oem-dev-tools that installs all scripts''',
           'lp-bug',
           'live-build-image-chroot.sh',
           'oem-getiso',
+          'oem-meta-packages',
           'pkg-list',
           'pkg-oem-meta',
           'rename-everything.py',
diff --git a/tests/test_flake8 b/tests/test_flake8
index 069a89d..c4b8daa 100755
--- a/tests/test_flake8
+++ b/tests/test_flake8
@@ -3,4 +3,5 @@
 flake8 --ignore E501 \
     lp-bug \
     mir-bug \
+    oem-meta-packages \
     pkg-list
diff --git a/tests/test_pep8 b/tests/test_pep8
index 22fdd7e..e09071e 100755
--- a/tests/test_pep8
+++ b/tests/test_pep8
@@ -11,4 +11,5 @@ pep8 --ignore=E501 \
     lp-bug \
     mir-bug \
     oem-getiso \
+    oem-meta-packages \
     pkg-list