Merge ~pieq/bugit/+git/qabro:fix-sosreport-dpkg-logs into bugit:master

[Pierre Equoy]

See commit message for more info

–

To test (on Ubuntu classic):

1. Run `snapcraft` from this branch to build a `qabro_0.13dev_amd64.snap` file
2. Install the snap:
    sudo snap install qabro_0.13dev_amd64.snap --devmode
3. Run sosreport from the snap:
    sudo qabro.sosreport
It will generate a /root/snap/qabro/common/sosreport-*.tar.gz archive

4. Copy the archive and modify the permissions, then open it to check the content of `sos_commands/dpkg/`:

    sudo cp /root/snap/qabro/common/sosreport-*.tar.gz /tmp
    sudo chown $USER:$USER /tmp/sosreport-*.tar.gz
    xdg-open /tmp/sosreport-*.tar.gz

You should find the list of packages installed on your system.

Unmerged commits

8ffa378... by Pierre Equoy on 2020-06-12

Add dpkg and patch sosreport to provide proper `dpkg -l` logs

Until now, on classic systems, the list of packages installed (the
output of `dpkg -l`) was empty in the archive report generated by
sosreport.

This commit addresses the issue. The list of installed packages can now
be found in `sos_commands/dpkg/`.

LP: #1880071

d971493... by Pierre Equoy on 2020-06-04

Add features and vendors tag lists

We used to have our own list of "bug type" tags. We are now using two
up-to-date lists of features and vendors tags that are shared among
teams.

These two lists are shown in two columns under the text field to enter
tags. The user can select as many impacted features/vendors as required,
and this updates the list of tags submitted to Launchpad dynamically.

LP: #1880538, #1880071

a059e29... by Pierre Equoy on 2020-06-04

Cleanup code look

36143db... by Pierre Equoy on 2020-06-04

Update for sosreport 3.9

sosreport 3.9 is now available in the bionic-updates pocket, so we need
to make the following changes to be able to build QABro from now on:

- remove the apt-no-auth-conf patch (it's been merged in sosreport 3.9)
and remove it from the snapcraft recipe
- adapt the env-path-snap patch
- modify the way we retrieve the path to the .tar.gz archive generated
by sosreport

985f5a4... by Pierre Equoy on 2020-06-04

Bump to v0.13dev

diff --git a/patches/sosreport-apt-no-auth-conf.patch b/patches/sosreport-apt-no-auth-conf.patch
 deleted file mode 100644
index b2c6514..0000000
--- a/patches/sosreport-apt-no-auth-conf.patch
+++ /dev/null
@@ -1,11 +0,0 @@
---- a/sos/plugins/apt.py
-+++ /usr/share/sosreport/sos/plugins/apt.py
-@@ -23,6 +23,8 @@
-             "/etc/apt", "/var/log/apt"
-         ])
- 
-+        self.add_forbidden_path("/etc/apt/auth.conf")
-+
-         self.add_cmd_output([
-             "apt-get check",
-             "apt-config dump",
diff --git a/patches/sosreport-dpkg-l.patch b/patches/sosreport-dpkg-l.patch
 new file mode 100644
index 0000000..c73c141
--- /dev/null
+++ b/patches/sosreport-dpkg-l.patch
@@ -0,0 +1,12 @@
+diff -u /tmp/sos-3.9/sos/plugins/dpkg.py usr/share/sosreport/sos/plugins/dpkg.py
+--- /tmp/sos-3.9/sos/plugins/dpkg.py	2020-06-12 15:00:20.955904359 +0800
++++ usr/share/sosreport/sos/plugins/dpkg.py	2020-06-12 15:01:18.930341885 +0800
+@@ -19,7 +19,7 @@
+     profiles = ('sysmgmt', 'packagemanager')
+ 
+     def setup(self):
+-        self.add_cmd_output("dpkg -l", root_symlink="installed-debs")
++        self.add_cmd_output("dpkg -l --admindir=/var/lib/snapd/hostfs/var/lib/dpkg", root_symlink="installed-debs")
+         if self.get_option("verify"):
+             self.add_cmd_output("dpkg -V")
+             self.add_cmd_output("dpkg -C")
diff --git a/patches/sosreport-env-path-snap.patch b/patches/sosreport-env-path-snap.patch
index b4f95c1..f20b1c8 100644
--- a/patches/sosreport-env-path-snap.patch
+++ b/patches/sosreport-env-path-snap.patch
@@ -1,22 +1,15 @@
-diff -ru /tmp/sos-3.5/sos/policies/ubuntu.py usr/share/sosreport/sos/policies/ubuntu.py
---- /tmp/sos-3.5/sos/policies/ubuntu.py	        2017-11-02 20:54:47.000000000 +0800
-+++ usr/share/sosreport/sos/policies/ubuntu.py	2018-10-24 15:28:46.439206314 +0800
-@@ -3,11 +3,18 @@
- from sos.plugins import UbuntuPlugin, DebianPlugin
- from sos.policies.debian import DebianPolicy
- 
-+import os
- 
- class UbuntuPolicy(DebianPolicy):
-     distro = "Ubuntu"
-     vendor = "Ubuntu"
-     vendor_url = "http://www.ubuntu.com/"
-+    PATH = ("/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
-+            ":/usr/local/sbin:/usr/local/bin"
-+            ":$SNAP/usr/sbin:$SNAP/usr/bin:$SNAP/sbin:$SNAP/bin:$SNAP/usr/games"
-+            ":$SNAP/usr/local/games:$SNAP/usr/local/sbin:$SNAP/usr/local/bin"
-+           )
+diff -u /tmp/sos-3.9/sos/policies/ubuntu.py usr/share/sosreport/sos/policies/ubuntu.py
+--- /tmp/sos-3.9/sos/policies/ubuntu.py	2020-06-03 15:14:14.151559473 +0800
++++ usr/share/sosreport/sos/policies/ubuntu.py	2020-06-03 15:17:11.178575436 +0800
+@@ -11,7 +11,10 @@
+     vendor = "Canonical"
+     vendor_url = "https://www.ubuntu.com/"
+     PATH = "/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games" \
+-           + ":/usr/local/sbin:/usr/local/bin:/snap/bin"
++           + ":/usr/local/sbin:/usr/local/bin:/snap/bin" \
++           + ":$SNAP/usr/sbin:$SNAP/usr/bin:$SNAP/sbin:$SNAP/bin:$SNAP/usr/games" \
++           + ":$SNAP/usr/local/games:$SNAP/usr/local/sbin:$SNAP/usr/local/bin"
 +    PATH = os.path.expandvars(PATH)
- 
-     def __init__(self, sysroot=None):
-         super(UbuntuPolicy, self).__init__(sysroot=sysroot)
+     _upload_url = "https://files.support.canonical.com/uploads/"
+     _upload_user = "ubuntu"
+     _upload_password = "ubuntu"
diff --git a/qabro/__version__.py b/qabro/__version__.py
index 0fe7cd3..a5c2cc5 100644
--- a/qabro/__version__.py
+++ b/qabro/__version__.py
@@ -1,5 +1,5 @@
 __title__ = 'qabro'
-__version__ = '0.12'
+__version__ = '0.13dev'
 __description__ = 'Tool to generate a Launchpad bug report and attach useful logs (using sosreport and the like).'
 __author__ = 'Pierre Equoy'
 __author_email__ = '[e-mail]'
diff --git a/qabro/bug_assistant.py b/qabro/bug_assistant.py
index 1ab9532..a32fd96 100644
--- a/qabro/bug_assistant.py
+++ b/qabro/bug_assistant.py
@@ -230,7 +230,11 @@ class AttachmentAssistant:
                            "sosreport", "--no-report", "--batch", "--tmp-dir",
                            home_dir, "-z", "gzip"]
                 output = subprocess.check_output(command).decode('utf-8')
-                archive_path = output.splitlines()[-6].strip()
+                # sosreport outputs a summary that includes a line with a
+                # path to the archive. We retrieve this:
+                for line in output.splitlines():
+                    if ".tar.gz" in line:
+                        archive_path = line.strip()
                 print("Report generated: {}".format(archive_path))
                 command = ["sudo", "chown", "{}:{}".format(user, user), archive_path]
                 subprocess.check_call(command)
diff --git a/qabro/data/tags.json b/qabro/data/tags.json
 new file mode 100644
index 0000000..8a779a5
--- /dev/null
+++ b/qabro/data/tags.json
@@ -0,0 +1,68 @@
+[{
+    "category": "features",
+    "type": "public",
+    "description": "Features",
+    "tags": {
+      "Audio": ["hwe-audio"],
+      "Bluetooth": ["hwe-bluetooth"],
+      "Brightness": ["hwe-brightness"],
+      "CANBus": ["hwe-canbus"],
+      "Camera": ["oem-camera"],
+      "Checkbox": ["checkbox test-case"],
+      "External Storage": ["oem-storage"],
+      "Fingerprint Reader": ["hwe-fingerprint"],
+      "Firmware": ["hwe-firmware"],
+      "Full Disk Encryption": ["oem-fde"],
+      "GPIO": ["hwe-gpio"],
+      "Hotkeys": ["hwe-hotkeys"],
+      "I2C": ["hwe-i2c"],
+      "Install": ["hwe-installer"],
+      "LED": ["hwe-led"],
+      "Media Card": ["hwe-media"],
+      "Missing driver": ["hwe-needs-driver"],
+      "Model Assertion": ["oem-assertions"],
+      "Model Pivot / Remodelling": ["oem-assertions"],
+      "Networking (ethernet)": ["hwe-networking-ethernet", "oem-networking"],
+      "Networking (modem)": ["hwe-networking-modem", "oem-networking"],
+      "Networking (wifi)": ["hwe-networking-wifi", "oem-networking"],
+      "Other Problem": ["oem-other"],
+      "Performance": ["oem-performance"],
+      "Power Management": ["hwe-suspend-resume"],
+      "Power On/Off": ["hwe-powercycle"],
+      "Recovery": ["oem-recovery"],
+      "Secure Boot": ["oem-secureboot"],
+      "Sensor": ["hwe-sensor"],
+      "Serial Assertion": ["oem-assertions"],
+      "Serial": ["hwe-serial"],
+      "Snapd": ["oem-snapd"],
+      "Store": ["oem-store"],
+      "Stress": ["oem-stress"],
+      "TPM": ["hwe-tpm"],
+      "Touchpad": ["hwe-touchpad"],
+      "Touchscreen": ["oem-touchscreen"],
+      "Video": ["hwe-graphics"],
+      "Watchdog": ["hwe-watchdog"],
+      "Zigbee": ["hwe-zigbee"]
+    }
+  },
+  {
+    "category": "vendors",
+    "type": "public",
+    "description": "Vendors",
+    "tags": {
+      "AMD": ["ihv-amd"],
+      "Atheros/Qualcomm": ["ihv-qualcomm-atheros"],
+      "Gemalto": ["ihv-gemalto"],
+      "Intel": ["ihv-intel"],
+      "MTK": ["ihv-mtk"],
+      "Marvell": ["ihv-marvell"],
+      "Mighty Gecko": ["ihv-mightygecko"],
+      "Nvidia": ["ihv-nvidia"],
+      "Quectel": ["ihv-quectel"],
+      "Realtek": ["ihv-realtek"],
+      "Redpine": ["ihv-redpine"],
+      "Sierra": ["ihv-sierra"],
+      "Telegesis": ["ihv-telegesis"],
+      "Telit": ["ihv-telit"]
+  }
+}]
 \ No newline at end of file
diff --git a/qabro/ui.py b/qabro/ui.py
index ab8a343..695b3ca 100644
--- a/qabro/ui.py
+++ b/qabro/ui.py
@@ -1,8 +1,10 @@
+import json
+from pathlib import Path
 import sys
 import urwid
 
-from qabro.bug_assistant import AttachmentAssistant, BugReport
 from qabro.__version__ import __version__ as qversion, __title__ as qtitle
+from qabro.bug_assistant import AttachmentAssistant, BugReport
 
 
 class ReportScreen:
@@ -11,7 +13,7 @@ class ReportScreen:
         ('title', 'white', 'black', 'bold'),
         ('key', 'yellow,bold', ''),
         ('error', 'dark red,bold', ''),
-        ]
+    ]
 
     footer_text = [
         qtitle + ' v' + qversion + '    ',
@@ -26,31 +28,20 @@ class ReportScreen:
                    'Medium',
                    'High',
                    'Critical')
-    bug_types = {"Audio": "hwe-audio",
-                 "Bluetooth": "hwe-bluetooth",
-                 "Camera": "oem-camera",
-                 "External storage": "oem-storage",
-                 "Touchpad": "hwe-touchpad",
-                 "Touchscreen": "oem-touchscreen",
-                 "Install": "hwe-installer",
-                 "Media Card": "hwe-media",
-                 "Networking (ethernet)": "hwe-networking-ethernet",
-                 "Networking (wifi)": "hwe-networking-wifi",
-                 "Networking (modem)": "hwe-networking-modem",
-                 "Power management": "hwe-suspend-resume",
-                 "Checkbox": "checkbox test-case",
-                 "Video": "hwe-graphics",
-                 "Hotkeys": "hwe-hotkeys",
-                 "Firmware": "hwe-firmware",
-                 "Missing driver": "hwe-needs-driver",
-                 "Performance": "oem-performance",
-                 "Other problem": "oem-other"}
 
     default_description = ("[Summary]\n\n"
                            "[Steps to reproduce]\n\n"
                            "[Expected result]\n\n"
                            "[Actual result]\n\n"
                            "[Failure rate]\n\n")
+    with open(Path(__file__).parent / "data/tags.json") as f:
+        _data = json.load(f)
+        tags = dict()
+        for d in _data:
+            if d["category"] == "features":
+                tags["features"] = d["tags"]
+            elif d["category"] == "vendors":
+                tags["vendors"] = d["tags"]
 
     def __init__(self, project='', tags='', assignee=''):
         # Adding standard info (CPU, BIOS, GPU, etc.)  to the description
@@ -60,7 +51,7 @@ class ReportScreen:
 
         self._title = urwid.LineBox(urwid.Edit(), 'Bug Title')
         self._description = urwid.LineBox(urwid.Edit(edit_text=self.default_description,
-                                          multiline=True),
+                                                     multiline=True),
                                           "Description (including steps to reproduce)")
         self._project = urwid.LineBox(urwid.Edit(edit_text=project), 'Project')
         self._series = urwid.LineBox(urwid.Edit(), 'Series')
@@ -70,17 +61,28 @@ class ReportScreen:
         self._importance = []
         for i in self.importances:
             rb = urwid.RadioButton(self._importance, i, 'first True', None)
-        self._bug_type = []
-        for i in sorted(self.bug_types):
-            rb = urwid.RadioButton(self._bug_type, i, state=False,
-                                   on_state_change=self.add_tag)
+        self._bug_features = []
+        for feature in sorted(self.tags["features"]):
+            cb = urwid.CheckBox(label=feature, state=False,
+                                on_state_change=self.toggle_tag,
+                                user_data=self.tags["features"])
+            self._bug_features.append(cb)
+        self._bug_vendors = []
+        for vendor in sorted(self.tags["vendors"]):
+            cb = urwid.CheckBox(label=vendor, state=False,
+                                on_state_change=self.toggle_tag,
+                                user_data=self.tags["vendors"])
+            self._bug_vendors.append(cb)
         self._assignee = urwid.LineBox(urwid.Edit(edit_text=assignee), 'Assigned To (Launchpad ID)')
         self._tags = urwid.LineBox(urwid.Edit(edit_text=tags), 'Tags')
         status_list = urwid.LineBox(urwid.Pile(self._status), 'Status')
         importance_list = urwid.LineBox(urwid.Pile(self._importance), 'Importance')
-        bug_types_list = urwid.LineBox(urwid.Pile(self._bug_type), 'Bug Type')
+        self.bug_features_list = urwid.LineBox(urwid.Pile(self._bug_features), 'Impacted Feature(s)')
+        self.bug_vendors_list = urwid.LineBox(urwid.Pile(self._bug_vendors), 'Impacted Vendor(s)')
         self.message = urwid.Text('')
         self.footer = urwid.AttrMap(urwid.Text(self.footer_text), 'foot')
+        self.bug_tags_lists = urwid.Columns([self.bug_features_list,
+                                             self.bug_vendors_list])
         self.pile = urwid.Pile([self.message,
                                 self._title,
                                 self._description,
@@ -90,20 +92,27 @@ class ReportScreen:
                                 importance_list,
                                 self._assignee,
                                 self._tags,
-                                bug_types_list])
+                                self.bug_tags_lists])
         self.view = urwid.Frame(urwid.Filler(self.pile, 'top'), footer=self.footer)
 
-    def add_tag(self, rb, state):
+    def toggle_tag(self, cb, state, user_data):
+        tags = user_data[cb.label]
         if state:
-            # for the selected radio button
-            tag = self.bug_types[rb.label]
-            self._tags.base_widget.edit_text += ' ' + tag
+            # if checkbox is ticked
+            self._tags.base_widget.edit_text += ' ' + ' '.join(tags)
         else:
-            # for the previously selected radio button
-            tag = self.bug_types[rb.label]
-            self._tags.base_widget.edit_text = self._tags.base_widget.edit_text.replace(tag, '')
-        # clean up the list of tags to remove unwanted white spaces
-        self._tags.base_widget.edit_text = " ".join(self._tags.base_widget.edit_text.split())
+            # if checkbox is unticked
+            for tag in tags:
+                self._tags.base_widget.edit_text = self._tags.base_widget.edit_text.replace(tag, '')
+        # clean up the list of tags to
+        # - remove unwanted white spaces
+        # - remove duplicated tags
+        current_tags = self._tags.base_widget.edit_text.split()
+        dedup_tags = []
+        for tag in current_tags:
+            if tag not in dedup_tags:
+                dedup_tags.append(tag)
+        self._tags.base_widget.edit_text = " ".join(dedup_tags)
 
     def run(self):
         """Run the urwid MainLoop."""
diff --git a/setup.py b/setup.py
index 9ca0a66..d01f071 100644
--- a/setup.py
+++ b/setup.py
@@ -20,6 +20,7 @@ qabro is a tool to report bugs against Launchpad projects and
 automatically attach relevent information to them (log files).
 """,
     packages=packages,
+    package_data={'qabro': ['data/tags.json']},
     entry_points={
         'console_scripts': ['qabro = qabro.__init__:main']
     },
diff --git a/snap/snapcraft.yaml b/snap/snapcraft.yaml
index e307ff0..de2580d 100644
--- a/snap/snapcraft.yaml
+++ b/snap/snapcraft.yaml
@@ -43,6 +43,11 @@ parts:
       - python3-urwid
       - ubuntu-report
 
+  dpkg:
+    plugin: nil
+    stage-packages:
+      - dpkg
+
   sosreport:
     after: [patches]
     plugin: nil
@@ -56,8 +61,9 @@ parts:
       # variable used by sosreport to call other commands.
       patch $SNAPCRAFT_PART_INSTALL/usr/share/sosreport/sos/policies/ubuntu.py $SNAPCRAFT_STAGE/patches/sosreport-env-path-snap.patch
       # patches from YC Chen
-      patch $SNAPCRAFT_PART_INSTALL/usr/share/sosreport/sos/plugins/apt.py $SNAPCRAFT_STAGE/patches/sosreport-apt-no-auth-conf.patch
       patch $SNAPCRAFT_PART_INSTALL/usr/share/sosreport/sos/plugins/apt.py $SNAPCRAFT_STAGE/patches/sosreport-apt-remove-user-pwd.patch
+      # patch to be able to have data from `dpkg -l` command
+      patch $SNAPCRAFT_PART_INSTALL/usr/share/sosreport/sos/plugins/dpkg.py $SNAPCRAFT_STAGE/patches/sosreport-dpkg-l.patch
 
   sosreport-plugins:
     after: [sosreport]