Merge lp:~mvo/click/click-check-libs into lp:click/devel

Proposed by Michael Vogt on 2014-09-25
Status: Needs review
Proposed branch: lp:~mvo/click/click-check-libs
Merge into: lp:click/devel
Diff against target: 290 lines (+266/-0)
3 files modified
click/commands/verify.py (+13/-0)
click/elfinspector.py (+196/-0)
click/tests/test_elfinspector.py (+57/-0)
To merge this branch: bzr merge lp:~mvo/click/click-check-libs
Reviewer Review Type Date Requested Status
PS Jenkins bot (community) continuous-integration Needs Fixing on 2014-09-25
click hackers 2014-09-25 Pending
Review via email: mp+235960@code.launchpad.net

Commit message

Add new click verify --check-libs command

Description of the change

This branch will add a new command:
  click verify --check-libs com.ubuntu.gallery_2.9.1.1056_armhf.click

to check if there are libraries by any ELF binary in the click that are not part of the declared framework. It will use the existing click chroot to find information about the libraries (we might use apt-file too, but it seems like the chroot is the better option).

When its run it will inspect the objdump -p output and check for the NEEDED section. If the library that is NEEDED is not a direct dependency of ubuntu-sdk-libs or bundled in the click or part of the IGNORE_LIST it will complain. Example output:

$ PYTHONPATH=. ./bin/click verify --check-libs com.ubuntu.gallery_2.9.1.1056_armhf.click
WARNING:root:Failed to get files for 'ubuntu-ui-toolkit-theme:armhf'
...
Missing libraries:
libQt5Core.so.5
libQt5Gui.so.5
libQt5Qml.so.5
libQt5Quick.so.5
libQt5Sql.so.5
libQt5Widgets.so.5
libcontent-hub.so.0
libexpat.so.1
libmediainfo.so.0

Which indicates that we probably need to tweak ubuntu-sdk-libs to include more of qt5 directly.

I would love to get feedback on the approach and the IGNORE_LIST. I haven't written integration tests yet because I want to get approval for the general approach first, but if it looks good I'm happy to write the missing tests.

Thanks!
 Michael

To post a comment you must log in.
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:521
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~mvo/click/click-check-libs/+merge/235960/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/click-devel-ci/76/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-amd64-ci/78
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/76
        deb: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/76/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-i386-ci/76

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/click-devel-ci/76/rebuild

review: Needs Fixing (continuous-integration)

Unmerged revisions

521. By Michael Vogt on 2014-09-25

click/elfinspector.py: add IGNORE_LIST for base libs like libc

520. By Michael Vogt on 2014-09-25

really add some missing tests

519. By Michael Vogt on 2014-09-25

fix pep8 error, add some missing tests

518. By Michael Vogt on 2014-09-25

cleanup, much nicer now

517. By Michael Vogt on 2014-09-09

initial (ugly) draft to get missing libs from a click, you can do "click verify --check-libs some.click" now

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'click/commands/verify.py'
--- click/commands/verify.py 2014-08-07 21:51:27 +0000
+++ click/commands/verify.py 2014-09-25 13:09:48 +0000
@@ -32,10 +32,23 @@
32 parser.add_option(32 parser.add_option(
33 "--allow-unauthenticated", action="store_true", default=False,33 "--allow-unauthenticated", action="store_true", default=False,
34 help="allow installing packages with no sigantures")34 help="allow installing packages with no sigantures")
35 parser.add_option(
36 "--check-libs", action="store_true", default=False,
37 help="also check shared libraries")
35 options, args = parser.parse_args(argv)38 options, args = parser.parse_args(argv)
36 if len(args) < 1:39 if len(args) < 1:
37 parser.error("need package file name")40 parser.error("need package file name")
38 package_path = args[0]41 package_path = args[0]
42
43 if options.check_libs:
44 from click.elfinspector import ElfInspector
45 elf = ElfInspector()
46 missing_libs = elf.check_for_libs_outside_of_framework(package_path)
47 if missing_libs:
48 print("Missing libraries:")
49 print("\n".join(sorted(missing_libs)))
50 return 1
51
39 installer = ClickInstaller(52 installer = ClickInstaller(
40 db=None, force_missing_framework=options.force_missing_framework,53 db=None, force_missing_framework=options.force_missing_framework,
41 allow_unauthenticated=options.allow_unauthenticated)54 allow_unauthenticated=options.allow_unauthenticated)
4255
=== added file 'click/elfinspector.py'
--- click/elfinspector.py 1970-01-01 00:00:00 +0000
+++ click/elfinspector.py 2014-09-25 13:09:48 +0000
@@ -0,0 +1,196 @@
1#
2# Copyright (C) 2013-2014 Canonical Ltd.
3#
4# This program is free software: you can redistribute it and/or modify
5# it under the terms of the GNU General Public License as published by
6# the Free Software Foundation; version 3 of the License.
7#
8# This program is distributed in the hope that it will be useful,
9# but WITHOUT ANY WARRANTY; without even the implied warranty of
10# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
11# GNU General Public License for more details.
12#
13# You should have received a copy of the GNU General Public License
14# along with this program. If not, see <http://www.gnu.org/licenses/>.
15#
16# Parts based on lp: ~pkgme-binary-dev/pkgme-devportal/trunk
17
18import json
19import fnmatch
20import logging
21import os
22import shutil
23import subprocess
24import tempfile
25
26#from .sohelper import get_supported_so_libs_for_click_chroot
27import click.chroot
28
29
30class Dependency:
31 def __init__(self, name):
32 self.name = name
33 def __unicode__(self):
34 return self.name
35 def __repr__(self):
36 return "<Dependency: %s>" % self.name
37
38
39def get_dependencies(chroot_dir, pkgname):
40 cmd = ["apt-cache",
41 "-o", "Dir=%s" % chroot_dir,
42 "depends", pkgname]
43 output = subprocess.check_output(cmd,
44 universal_newlines=True)
45 or_depends = []
46 for line in output.splitlines():
47 prefix, sep, dep = line.strip().partition(":")
48 if prefix != "Depends":
49 continue
50 dep = dep.strip().strip(">").strip("<")
51 # FIXME: deal with or-deps
52 or_depends.append([Dependency(dep)])
53 return or_depends
54
55
56def get_installed_files_for_pkg(chroot_dir, full_pkgname):
57 # we might also use "dpkg -L --rootdir= chroot_dir"
58 # but that is quite a bit of fork/exec overhead
59 dpkg_list_file = os.path.join(
60 chroot_dir, "var", "lib", "dpkg", "info", full_pkgname+".list")
61 if not os.path.exists(dpkg_list_file):
62 logging.warning("Failed to get files for '%s'" % full_pkgname)
63 return []
64 with open(dpkg_list_file) as f:
65 return f.read().splitlines()
66
67
68def get_libs_for_framework(chroot_dir, framework, so_files,
69 depth=1, max_depth=1):
70 if depth > max_depth:
71 return
72 # if only we had python-apt in the click chroots
73 dependencies = get_dependencies(chroot_dir, framework)
74 for or_dep in dependencies:
75 for dep in or_dep:
76 pkgname = dep.name
77 so_files |= set(
78 [os.path.basename(p)
79 for p in get_installed_files_for_pkg(chroot_dir, pkgname)
80 if fnmatch.fnmatch(p, "*.so.*")])
81 get_libs_for_framework(
82 chroot_dir, pkgname, so_files, depth + 1, max_depth)
83
84
85def get_supported_so_libs_for_click_chroot(arch, framework, series,
86 libs_pkgname):
87 chroot = click.chroot.ClickChroot(arch, framework, series=series)
88 chroot_dir = os.path.join(chroot.chroots_dir, chroot.full_name)
89 # the output so files is stored here, slightly ugly
90 output_so_files = set()
91 get_libs_for_framework(chroot_dir, libs_pkgname, output_so_files)
92 return output_so_files
93
94
95def get_elf_files(path):
96 """Find all of the ELF exectuable binaries underneath 'path'."""
97 binaries = set()
98 for root, dirs, files in os.walk(path):
99 for f in files:
100 path = os.path.join(root, f)
101 with open(path, 'rb') as f:
102 header = f.read(4)
103 #import binascii
104 #print(binascii.hexlify(header))
105 if header == b'\x7fELF':
106 # we could yield here too but the amount of data is
107 # likely small
108 binaries.add(path)
109 return binaries
110
111
112def get_required_libraries(path_to_elf):
113 required_libraries = set()
114 output = subprocess.check_output(
115 ["objdump", '-f', '-p', path_to_elf], universal_newlines=True)
116 for line in output.splitlines():
117 line = line.strip()
118 if line.startswith("NEEDED"):
119 soname = line.split()[1].strip()
120 # special case as its arch-dependant
121 if not soname.startswith("ld-linux-"):
122 required_libraries.add(soname)
123 return required_libraries
124
125
126class ElfInspector:
127
128 LIBS_FOR_FRAMEWORK = {}
129 IGNORE_LIST = set([
130 "libc.so.6",
131 "libdl.so.2",
132 "libgcc_s.so.1",
133 "libm.so.6",
134 "libpthread.so.0",
135 "libstdc++.so.6",
136 "libz.so.1",
137 ])
138
139 def _open(self, clickfile):
140 self.clickfile = clickfile
141 self.unpack_dir = self._unpack(self.clickfile)
142 self.manifest = json.load(open(os.path.join(
143 self.unpack_dir,"CONTROL", "manifest")))
144
145 def _close(self):
146 shutil.rmtree(self.unpack_dir)
147
148 def _unpack(self, clickfile):
149 tempdir = tempfile.mkdtemp()
150 subprocess.check_call(
151 ["dpkg-deb", "-e", clickfile, os.path.join(tempdir, "CONTROL")])
152 subprocess.check_call(
153 ["dpkg-deb", "-x", clickfile, tempdir])
154 return tempdir
155
156 def _load_supported_libs_for_framework(self, arch, framework):
157 if framework not in self.LIBS_FOR_FRAMEWORK:
158 supported_libs = set()
159 supported_libs = get_supported_so_libs_for_click_chroot(
160 arch, framework, None, "ubuntu-sdk-libs:%s" % arch)
161 self.LIBS_FOR_FRAMEWORK[framework] = set(supported_libs)
162 return self.LIBS_FOR_FRAMEWORK[framework]
163
164 def _get_bundled_libs_from_click(self):
165 bundled_libs = set()
166 # FIXME: too strict? clicks might ship there own wrappers
167 # FIXME2: too loose? ubuntu-app-launch uses lib/$(ARCH)/bin
168 bundled_libs_dir = os.path.join(self.unpack_dir, "lib")
169 for root, dirnames, filenames in os.walk(bundled_libs_dir):
170 bundled_libs |= set(fnmatch.filter(filenames, "*.so.*"))
171 return bundled_libs
172
173 def _verify_libs_for_framework(self, arch, framework, required_libs):
174 missing_libs = set()
175 supported_libs = self._load_supported_libs_for_framework(
176 arch, framework)
177 supported_libs |= self._get_bundled_libs_from_click()
178 libs_not_available_in_framework = required_libs - supported_libs
179 if len(libs_not_available_in_framework) > 0:
180 missing_libs |= libs_not_available_in_framework
181 return missing_libs
182
183 def check_for_libs_outside_of_framework(self, package_path):
184 self._open(package_path)
185 binaries = get_elf_files(self.unpack_dir)
186 required_libs_for_binary = {}
187 for binary in binaries:
188 required_libs_for_binary[binary] = get_required_libraries(binary)
189 required_libs_for_binary[binary] -= self.IGNORE_LIST
190 missing_libs = set()
191 for binary, required_libs in required_libs_for_binary.items():
192 missing_libs |= self._verify_libs_for_framework(
193 self.manifest['architecture'], self.manifest['framework'],
194 required_libs)
195 self._close()
196 return missing_libs
0197
=== added file 'click/tests/test_elfinspector.py'
--- click/tests/test_elfinspector.py 1970-01-01 00:00:00 +0000
+++ click/tests/test_elfinspector.py 2014-09-25 13:09:48 +0000
@@ -0,0 +1,57 @@
1# Copyright (C) 2014 Canonical Ltd.
2# Author: Michael Vogt <mvo@ubuntu.com>
3
4# This program is free software: you can redistribute it and/or modify
5# it under the terms of the GNU General Public License as published by
6# the Free Software Foundation; version 3 of the License.
7#
8# This program is distributed in the hope that it will be useful,
9# but WITHOUT ANY WARRANTY; without even the implied warranty of
10# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
11# GNU General Public License for more details.
12#
13# You should have received a copy of the GNU General Public License
14# along with this program. If not, see <http://www.gnu.org/licenses/>.
15
16"""Unit tests for click.elfinspector
17
18"""
19
20from __future__ import print_function
21
22__metaclass__ = type
23__all__ = [
24 'TestClickElfInspectorHelpers',
25 ]
26
27
28from click.tests.helpers import TestCase
29from click.elfinspector import (
30 get_dependencies,
31 get_elf_files,
32 get_installed_files_for_pkg,
33 get_required_libraries,
34)
35
36class TestClickElfInspectorHelpers(TestCase):
37
38 def test_get_dependencies(self):
39 dependencies = get_dependencies("/", "apt")
40 self.assertTrue(len(dependencies) > 2)
41 self.assertIn("libc6", [or_dep[0].name for or_dep in dependencies])
42
43 def test_get_installed_files_for_pkg(self):
44 apt_owned_files = get_installed_files_for_pkg("/", "apt")
45 self.assertIn("/usr/bin/apt-get", apt_owned_files)
46
47 def test_get_elf_files(self):
48 files = get_elf_files("/bin")
49 self.assertIn("/bin/mv", files)
50
51 def test_get_elf_files_no_elf_files(self):
52 files = get_elf_files("/var/lib/apt/periodic/")
53 self.assertEqual(files, set([]))
54
55 def test_get_required_libraries(self):
56 required_libs = get_required_libraries("/bin/mv")
57 self.assertIn("libc.so.6", required_libs)

Subscribers

People subscribed via source and target branches

to all changes: