Merge lp:~mvo/click/click-check-libs into lp:click/devel

[Michael Vogt]

Add new click verify --check-libs command

This branch will add a new command:
  click verify --check-libs com.ubuntu.gallery_2.9.1.1056_armhf.click

to check if there are libraries by any ELF binary in the click that are not part of the declared framework. It will use the existing click chroot to find information about the libraries (we might use apt-file too, but it seems like the chroot is the better option).

When its run it will inspect the objdump -p output and check for the NEEDED section. If the library that is NEEDED is not a direct dependency of ubuntu-sdk-libs or bundled in the click or part of the IGNORE_LIST it will complain. Example output:

$ PYTHONPATH=. ./bin/click verify --check-libs com.ubuntu.gallery_2.9.1.1056_armhf.click
WARNING:root:Failed to get files for 'ubuntu-ui-toolkit-theme:armhf'
...
Missing libraries:
libQt5Core.so.5
libQt5Gui.so.5
libQt5Qml.so.5
libQt5Quick.so.5
libQt5Sql.so.5
libQt5Widgets.so.5
libcontent-hub.so.0
libexpat.so.1
libmediainfo.so.0

Which indicates that we probably need to tweak ubuntu-sdk-libs to include more of qt5 directly.

I would love to get feedback on the approach and the IGNORE_LIST. I haven't written integration tests yet because I want to get approval for the general approach first, but if it looks good I'm happy to write the missing tests.

Thanks!
 Michael

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:521
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~mvo/click/click-check-libs/+merge/235960/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/click-devel-ci/76/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-amd64-ci/78
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/76
        deb: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/76/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-i386-ci/76

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/click-devel-ci/76/rebuild

review: Needs Fixing (continuous-integration)

Unmerged revisions

521. By Michael Vogt on 2014-09-25

click/elfinspector.py: add IGNORE_LIST for base libs like libc

520. By Michael Vogt on 2014-09-25

really add some missing tests

519. By Michael Vogt on 2014-09-25

fix pep8 error, add some missing tests

518. By Michael Vogt on 2014-09-25

cleanup, much nicer now

517. By Michael Vogt on 2014-09-09

initial (ugly) draft to get missing libs from a click, you can do "click verify --check-libs some.click" now

=== modified file 'click/commands/verify.py'
--- click/commands/verify.py	2014-08-07 21:51:27 +0000
+++ click/commands/verify.py	2014-09-25 13:09:48 +0000
@@ -32,10 +32,23 @@
     parser.add_option(
         "--allow-unauthenticated", action="store_true", default=False,
         help="allow installing packages with no sigantures")
+    parser.add_option(
+        "--check-libs", action="store_true", default=False,
+        help="also check shared libraries")
     options, args = parser.parse_args(argv)
     if len(args) < 1:
         parser.error("need package file name")
     package_path = args[0]
+
+    if options.check_libs:
+        from click.elfinspector import ElfInspector
+        elf = ElfInspector()
+        missing_libs = elf.check_for_libs_outside_of_framework(package_path)
+        if missing_libs:
+            print("Missing libraries:")
+            print("\n".join(sorted(missing_libs)))
+            return 1
+
     installer = ClickInstaller(
         db=None, force_missing_framework=options.force_missing_framework,
         allow_unauthenticated=options.allow_unauthenticated)

=== added file 'click/elfinspector.py'
--- click/elfinspector.py	1970-01-01 00:00:00 +0000
+++ click/elfinspector.py	2014-09-25 13:09:48 +0000
@@ -0,0 +1,196 @@
+#
+# Copyright (C) 2013-2014 Canonical Ltd.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 3 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# Parts based on lp: ~pkgme-binary-dev/pkgme-devportal/trunk
+
+import json
+import fnmatch
+import logging
+import os
+import shutil
+import subprocess
+import tempfile
+
+#from .sohelper import get_supported_so_libs_for_click_chroot
+import click.chroot
+
+
+class Dependency:
+    def __init__(self, name):
+        self.name = name
+    def __unicode__(self):
+        return self.name
+    def __repr__(self):
+        return "<Dependency: %s>" % self.name
+
+
+def get_dependencies(chroot_dir, pkgname):
+    cmd = ["apt-cache",
+           "-o", "Dir=%s" % chroot_dir,
+           "depends", pkgname]
+    output = subprocess.check_output(cmd,
+                                     universal_newlines=True)
+    or_depends = []
+    for line in output.splitlines():
+        prefix, sep, dep = line.strip().partition(":")
+        if prefix != "Depends":
+            continue
+        dep = dep.strip().strip(">").strip("<")
+        # FIXME: deal with or-deps
+        or_depends.append([Dependency(dep)])
+    return or_depends
+
+
+def get_installed_files_for_pkg(chroot_dir, full_pkgname):
+    # we might also use "dpkg -L --rootdir= chroot_dir"
+    # but that is quite a bit of fork/exec overhead
+    dpkg_list_file = os.path.join(
+            chroot_dir, "var", "lib", "dpkg", "info", full_pkgname+".list")
+    if not os.path.exists(dpkg_list_file):
+        logging.warning("Failed to get files for '%s'" % full_pkgname)
+        return []
+    with open(dpkg_list_file) as f:
+        return f.read().splitlines()
+
+
+def get_libs_for_framework(chroot_dir, framework, so_files,
+                           depth=1, max_depth=1):
+    if depth > max_depth:
+        return
+    # if only we had python-apt in the click chroots
+    dependencies = get_dependencies(chroot_dir, framework)
+    for or_dep in dependencies:
+        for dep in or_dep:
+            pkgname = dep.name
+            so_files |= set(
+                [os.path.basename(p)
+                 for p in get_installed_files_for_pkg(chroot_dir, pkgname)
+                 if fnmatch.fnmatch(p, "*.so.*")])
+            get_libs_for_framework(
+                chroot_dir, pkgname, so_files, depth + 1, max_depth)
+
+
+def get_supported_so_libs_for_click_chroot(arch, framework, series,
+                                           libs_pkgname):
+    chroot = click.chroot.ClickChroot(arch, framework, series=series)
+    chroot_dir = os.path.join(chroot.chroots_dir, chroot.full_name)
+    # the output so files is stored here, slightly ugly
+    output_so_files = set()
+    get_libs_for_framework(chroot_dir, libs_pkgname, output_so_files)
+    return output_so_files
+
+
+def get_elf_files(path):
+    """Find all of the ELF exectuable binaries underneath 'path'."""
+    binaries = set()
+    for root, dirs, files in os.walk(path):
+        for f in files:
+            path = os.path.join(root, f)
+            with open(path, 'rb') as f:
+                header = f.read(4)
+                #import binascii
+                #print(binascii.hexlify(header))
+                if header == b'\x7fELF':
+                    # we could yield here too but the amount of data is
+                    # likely small
+                    binaries.add(path)
+    return binaries
+
+
+def get_required_libraries(path_to_elf):
+    required_libraries = set()
+    output = subprocess.check_output(
+        ["objdump", '-f', '-p', path_to_elf], universal_newlines=True)
+    for line in output.splitlines():
+        line = line.strip()
+        if line.startswith("NEEDED"):
+            soname = line.split()[1].strip()
+            # special case as its arch-dependant
+            if not soname.startswith("ld-linux-"):
+                required_libraries.add(soname)
+    return required_libraries
+
+
+class ElfInspector:
+
+    LIBS_FOR_FRAMEWORK = {}
+    IGNORE_LIST = set([
+        "libc.so.6",
+        "libdl.so.2",
+        "libgcc_s.so.1",
+        "libm.so.6",
+        "libpthread.so.0",
+        "libstdc++.so.6",
+        "libz.so.1",
+    ])
+
+    def _open(self, clickfile):
+        self.clickfile = clickfile
+        self.unpack_dir = self._unpack(self.clickfile)
+        self.manifest = json.load(open(os.path.join(
+            self.unpack_dir,"CONTROL", "manifest")))
+
+    def _close(self):
+        shutil.rmtree(self.unpack_dir)
+
+    def _unpack(self, clickfile):
+        tempdir = tempfile.mkdtemp()
+        subprocess.check_call(
+            ["dpkg-deb", "-e", clickfile, os.path.join(tempdir, "CONTROL")])
+        subprocess.check_call(
+            ["dpkg-deb", "-x", clickfile, tempdir])
+        return tempdir
+
+    def _load_supported_libs_for_framework(self, arch, framework):
+        if framework not in self.LIBS_FOR_FRAMEWORK:
+            supported_libs = set()
+            supported_libs = get_supported_so_libs_for_click_chroot(
+                arch, framework, None, "ubuntu-sdk-libs:%s" % arch)
+            self.LIBS_FOR_FRAMEWORK[framework] = set(supported_libs)
+        return self.LIBS_FOR_FRAMEWORK[framework]
+
+    def _get_bundled_libs_from_click(self):
+        bundled_libs = set()
+        # FIXME: too strict? clicks might ship there own wrappers
+        # FIXME2: too loose? ubuntu-app-launch uses lib/$(ARCH)/bin
+        bundled_libs_dir = os.path.join(self.unpack_dir, "lib")
+        for root, dirnames, filenames in os.walk(bundled_libs_dir):
+            bundled_libs |= set(fnmatch.filter(filenames, "*.so.*"))
+        return bundled_libs
+
+    def _verify_libs_for_framework(self, arch, framework, required_libs):
+        missing_libs = set()
+        supported_libs = self._load_supported_libs_for_framework(
+            arch, framework)
+        supported_libs |= self._get_bundled_libs_from_click()
+        libs_not_available_in_framework = required_libs - supported_libs
+        if len(libs_not_available_in_framework) > 0:
+            missing_libs |= libs_not_available_in_framework
+        return missing_libs
+
+    def check_for_libs_outside_of_framework(self, package_path):
+        self._open(package_path)
+        binaries = get_elf_files(self.unpack_dir)
+        required_libs_for_binary = {}
+        for binary in binaries:
+            required_libs_for_binary[binary] = get_required_libraries(binary)
+            required_libs_for_binary[binary] -= self.IGNORE_LIST
+        missing_libs = set()
+        for binary, required_libs in required_libs_for_binary.items():
+            missing_libs |= self._verify_libs_for_framework(
+                self.manifest['architecture'], self.manifest['framework'],
+                required_libs)
+        self._close()
+        return missing_libs

=== added file 'click/tests/test_elfinspector.py'
--- click/tests/test_elfinspector.py	1970-01-01 00:00:00 +0000
+++ click/tests/test_elfinspector.py	2014-09-25 13:09:48 +0000
@@ -0,0 +1,57 @@
+# Copyright (C) 2014 Canonical Ltd.
+# Author: Michael Vogt <[e-mail]>
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 3 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""Unit tests for click.elfinspector
+
+"""
+
+from __future__ import print_function
+
+__metaclass__ = type
+__all__ = [
+    'TestClickElfInspectorHelpers',
+    ]
+
+
+from click.tests.helpers import TestCase
+from click.elfinspector import (
+    get_dependencies,
+    get_elf_files,
+    get_installed_files_for_pkg,
+    get_required_libraries,
+)
+
+class TestClickElfInspectorHelpers(TestCase):
+
+    def test_get_dependencies(self):
+        dependencies = get_dependencies("/", "apt")
+        self.assertTrue(len(dependencies) > 2)
+        self.assertIn("libc6", [or_dep[0].name for or_dep in dependencies])
+
+    def test_get_installed_files_for_pkg(self):
+        apt_owned_files = get_installed_files_for_pkg("/", "apt")
+        self.assertIn("/usr/bin/apt-get", apt_owned_files)
+
+    def test_get_elf_files(self):
+        files = get_elf_files("/bin")
+        self.assertIn("/bin/mv", files)
+
+    def test_get_elf_files_no_elf_files(self):
+        files = get_elf_files("/var/lib/apt/periodic/")
+        self.assertEqual(files, set([]))
+
+    def test_get_required_libraries(self):
+        required_libs = get_required_libraries("/bin/mv")
+        self.assertIn("libc.so.6", required_libs)