Merge lp:~mvo/click/click-check-libs into lp:click/devel

Proposed by Michael Vogt on 2014-09-25
Status: Needs review
Proposed branch: lp:~mvo/click/click-check-libs
Merge into: lp:click/devel
Diff against target: 290 lines (+266/-0)
3 files modified
click/commands/verify.py (+13/-0)
click/elfinspector.py (+196/-0)
click/tests/test_elfinspector.py (+57/-0)
To merge this branch: bzr merge lp:~mvo/click/click-check-libs
Reviewer Review Type Date Requested Status
PS Jenkins bot (community) continuous-integration Needs Fixing on 2014-09-25
click hackers 2014-09-25 Pending
Review via email: mp+235960@code.launchpad.net

Commit message

Add new click verify --check-libs command

Description of the change

This branch will add a new command:
  click verify --check-libs com.ubuntu.gallery_2.9.1.1056_armhf.click

to check if there are libraries by any ELF binary in the click that are not part of the declared framework. It will use the existing click chroot to find information about the libraries (we might use apt-file too, but it seems like the chroot is the better option).

When its run it will inspect the objdump -p output and check for the NEEDED section. If the library that is NEEDED is not a direct dependency of ubuntu-sdk-libs or bundled in the click or part of the IGNORE_LIST it will complain. Example output:

$ PYTHONPATH=. ./bin/click verify --check-libs com.ubuntu.gallery_2.9.1.1056_armhf.click
WARNING:root:Failed to get files for 'ubuntu-ui-toolkit-theme:armhf'
...
Missing libraries:
libQt5Core.so.5
libQt5Gui.so.5
libQt5Qml.so.5
libQt5Quick.so.5
libQt5Sql.so.5
libQt5Widgets.so.5
libcontent-hub.so.0
libexpat.so.1
libmediainfo.so.0

Which indicates that we probably need to tweak ubuntu-sdk-libs to include more of qt5 directly.

I would love to get feedback on the approach and the IGNORE_LIST. I haven't written integration tests yet because I want to get approval for the general approach first, but if it looks good I'm happy to write the missing tests.

Thanks!
 Michael

To post a comment you must log in.
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:521
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~mvo/click/click-check-libs/+merge/235960/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/click-devel-ci/76/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-amd64-ci/78
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/76
        deb: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/76/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-i386-ci/76

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/click-devel-ci/76/rebuild

review: Needs Fixing (continuous-integration)

Unmerged revisions

521. By Michael Vogt on 2014-09-25

click/elfinspector.py: add IGNORE_LIST for base libs like libc

520. By Michael Vogt on 2014-09-25

really add some missing tests

519. By Michael Vogt on 2014-09-25

fix pep8 error, add some missing tests

518. By Michael Vogt on 2014-09-25

cleanup, much nicer now

517. By Michael Vogt on 2014-09-09

initial (ugly) draft to get missing libs from a click, you can do "click verify --check-libs some.click" now

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'click/commands/verify.py'
2--- click/commands/verify.py 2014-08-07 21:51:27 +0000
3+++ click/commands/verify.py 2014-09-25 13:09:48 +0000
4@@ -32,10 +32,23 @@
5 parser.add_option(
6 "--allow-unauthenticated", action="store_true", default=False,
7 help="allow installing packages with no sigantures")
8+ parser.add_option(
9+ "--check-libs", action="store_true", default=False,
10+ help="also check shared libraries")
11 options, args = parser.parse_args(argv)
12 if len(args) < 1:
13 parser.error("need package file name")
14 package_path = args[0]
15+
16+ if options.check_libs:
17+ from click.elfinspector import ElfInspector
18+ elf = ElfInspector()
19+ missing_libs = elf.check_for_libs_outside_of_framework(package_path)
20+ if missing_libs:
21+ print("Missing libraries:")
22+ print("\n".join(sorted(missing_libs)))
23+ return 1
24+
25 installer = ClickInstaller(
26 db=None, force_missing_framework=options.force_missing_framework,
27 allow_unauthenticated=options.allow_unauthenticated)
28
29=== added file 'click/elfinspector.py'
30--- click/elfinspector.py 1970-01-01 00:00:00 +0000
31+++ click/elfinspector.py 2014-09-25 13:09:48 +0000
32@@ -0,0 +1,196 @@
33+#
34+# Copyright (C) 2013-2014 Canonical Ltd.
35+#
36+# This program is free software: you can redistribute it and/or modify
37+# it under the terms of the GNU General Public License as published by
38+# the Free Software Foundation; version 3 of the License.
39+#
40+# This program is distributed in the hope that it will be useful,
41+# but WITHOUT ANY WARRANTY; without even the implied warranty of
42+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
43+# GNU General Public License for more details.
44+#
45+# You should have received a copy of the GNU General Public License
46+# along with this program. If not, see <http://www.gnu.org/licenses/>.
47+#
48+# Parts based on lp: ~pkgme-binary-dev/pkgme-devportal/trunk
49+
50+import json
51+import fnmatch
52+import logging
53+import os
54+import shutil
55+import subprocess
56+import tempfile
57+
58+#from .sohelper import get_supported_so_libs_for_click_chroot
59+import click.chroot
60+
61+
62+class Dependency:
63+ def __init__(self, name):
64+ self.name = name
65+ def __unicode__(self):
66+ return self.name
67+ def __repr__(self):
68+ return "<Dependency: %s>" % self.name
69+
70+
71+def get_dependencies(chroot_dir, pkgname):
72+ cmd = ["apt-cache",
73+ "-o", "Dir=%s" % chroot_dir,
74+ "depends", pkgname]
75+ output = subprocess.check_output(cmd,
76+ universal_newlines=True)
77+ or_depends = []
78+ for line in output.splitlines():
79+ prefix, sep, dep = line.strip().partition(":")
80+ if prefix != "Depends":
81+ continue
82+ dep = dep.strip().strip(">").strip("<")
83+ # FIXME: deal with or-deps
84+ or_depends.append([Dependency(dep)])
85+ return or_depends
86+
87+
88+def get_installed_files_for_pkg(chroot_dir, full_pkgname):
89+ # we might also use "dpkg -L --rootdir= chroot_dir"
90+ # but that is quite a bit of fork/exec overhead
91+ dpkg_list_file = os.path.join(
92+ chroot_dir, "var", "lib", "dpkg", "info", full_pkgname+".list")
93+ if not os.path.exists(dpkg_list_file):
94+ logging.warning("Failed to get files for '%s'" % full_pkgname)
95+ return []
96+ with open(dpkg_list_file) as f:
97+ return f.read().splitlines()
98+
99+
100+def get_libs_for_framework(chroot_dir, framework, so_files,
101+ depth=1, max_depth=1):
102+ if depth > max_depth:
103+ return
104+ # if only we had python-apt in the click chroots
105+ dependencies = get_dependencies(chroot_dir, framework)
106+ for or_dep in dependencies:
107+ for dep in or_dep:
108+ pkgname = dep.name
109+ so_files |= set(
110+ [os.path.basename(p)
111+ for p in get_installed_files_for_pkg(chroot_dir, pkgname)
112+ if fnmatch.fnmatch(p, "*.so.*")])
113+ get_libs_for_framework(
114+ chroot_dir, pkgname, so_files, depth + 1, max_depth)
115+
116+
117+def get_supported_so_libs_for_click_chroot(arch, framework, series,
118+ libs_pkgname):
119+ chroot = click.chroot.ClickChroot(arch, framework, series=series)
120+ chroot_dir = os.path.join(chroot.chroots_dir, chroot.full_name)
121+ # the output so files is stored here, slightly ugly
122+ output_so_files = set()
123+ get_libs_for_framework(chroot_dir, libs_pkgname, output_so_files)
124+ return output_so_files
125+
126+
127+def get_elf_files(path):
128+ """Find all of the ELF exectuable binaries underneath 'path'."""
129+ binaries = set()
130+ for root, dirs, files in os.walk(path):
131+ for f in files:
132+ path = os.path.join(root, f)
133+ with open(path, 'rb') as f:
134+ header = f.read(4)
135+ #import binascii
136+ #print(binascii.hexlify(header))
137+ if header == b'\x7fELF':
138+ # we could yield here too but the amount of data is
139+ # likely small
140+ binaries.add(path)
141+ return binaries
142+
143+
144+def get_required_libraries(path_to_elf):
145+ required_libraries = set()
146+ output = subprocess.check_output(
147+ ["objdump", '-f', '-p', path_to_elf], universal_newlines=True)
148+ for line in output.splitlines():
149+ line = line.strip()
150+ if line.startswith("NEEDED"):
151+ soname = line.split()[1].strip()
152+ # special case as its arch-dependant
153+ if not soname.startswith("ld-linux-"):
154+ required_libraries.add(soname)
155+ return required_libraries
156+
157+
158+class ElfInspector:
159+
160+ LIBS_FOR_FRAMEWORK = {}
161+ IGNORE_LIST = set([
162+ "libc.so.6",
163+ "libdl.so.2",
164+ "libgcc_s.so.1",
165+ "libm.so.6",
166+ "libpthread.so.0",
167+ "libstdc++.so.6",
168+ "libz.so.1",
169+ ])
170+
171+ def _open(self, clickfile):
172+ self.clickfile = clickfile
173+ self.unpack_dir = self._unpack(self.clickfile)
174+ self.manifest = json.load(open(os.path.join(
175+ self.unpack_dir,"CONTROL", "manifest")))
176+
177+ def _close(self):
178+ shutil.rmtree(self.unpack_dir)
179+
180+ def _unpack(self, clickfile):
181+ tempdir = tempfile.mkdtemp()
182+ subprocess.check_call(
183+ ["dpkg-deb", "-e", clickfile, os.path.join(tempdir, "CONTROL")])
184+ subprocess.check_call(
185+ ["dpkg-deb", "-x", clickfile, tempdir])
186+ return tempdir
187+
188+ def _load_supported_libs_for_framework(self, arch, framework):
189+ if framework not in self.LIBS_FOR_FRAMEWORK:
190+ supported_libs = set()
191+ supported_libs = get_supported_so_libs_for_click_chroot(
192+ arch, framework, None, "ubuntu-sdk-libs:%s" % arch)
193+ self.LIBS_FOR_FRAMEWORK[framework] = set(supported_libs)
194+ return self.LIBS_FOR_FRAMEWORK[framework]
195+
196+ def _get_bundled_libs_from_click(self):
197+ bundled_libs = set()
198+ # FIXME: too strict? clicks might ship there own wrappers
199+ # FIXME2: too loose? ubuntu-app-launch uses lib/$(ARCH)/bin
200+ bundled_libs_dir = os.path.join(self.unpack_dir, "lib")
201+ for root, dirnames, filenames in os.walk(bundled_libs_dir):
202+ bundled_libs |= set(fnmatch.filter(filenames, "*.so.*"))
203+ return bundled_libs
204+
205+ def _verify_libs_for_framework(self, arch, framework, required_libs):
206+ missing_libs = set()
207+ supported_libs = self._load_supported_libs_for_framework(
208+ arch, framework)
209+ supported_libs |= self._get_bundled_libs_from_click()
210+ libs_not_available_in_framework = required_libs - supported_libs
211+ if len(libs_not_available_in_framework) > 0:
212+ missing_libs |= libs_not_available_in_framework
213+ return missing_libs
214+
215+ def check_for_libs_outside_of_framework(self, package_path):
216+ self._open(package_path)
217+ binaries = get_elf_files(self.unpack_dir)
218+ required_libs_for_binary = {}
219+ for binary in binaries:
220+ required_libs_for_binary[binary] = get_required_libraries(binary)
221+ required_libs_for_binary[binary] -= self.IGNORE_LIST
222+ missing_libs = set()
223+ for binary, required_libs in required_libs_for_binary.items():
224+ missing_libs |= self._verify_libs_for_framework(
225+ self.manifest['architecture'], self.manifest['framework'],
226+ required_libs)
227+ self._close()
228+ return missing_libs
229
230=== added file 'click/tests/test_elfinspector.py'
231--- click/tests/test_elfinspector.py 1970-01-01 00:00:00 +0000
232+++ click/tests/test_elfinspector.py 2014-09-25 13:09:48 +0000
233@@ -0,0 +1,57 @@
234+# Copyright (C) 2014 Canonical Ltd.
235+# Author: Michael Vogt <mvo@ubuntu.com>
236+
237+# This program is free software: you can redistribute it and/or modify
238+# it under the terms of the GNU General Public License as published by
239+# the Free Software Foundation; version 3 of the License.
240+#
241+# This program is distributed in the hope that it will be useful,
242+# but WITHOUT ANY WARRANTY; without even the implied warranty of
243+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
244+# GNU General Public License for more details.
245+#
246+# You should have received a copy of the GNU General Public License
247+# along with this program. If not, see <http://www.gnu.org/licenses/>.
248+
249+"""Unit tests for click.elfinspector
250+
251+"""
252+
253+from __future__ import print_function
254+
255+__metaclass__ = type
256+__all__ = [
257+ 'TestClickElfInspectorHelpers',
258+ ]
259+
260+
261+from click.tests.helpers import TestCase
262+from click.elfinspector import (
263+ get_dependencies,
264+ get_elf_files,
265+ get_installed_files_for_pkg,
266+ get_required_libraries,
267+)
268+
269+class TestClickElfInspectorHelpers(TestCase):
270+
271+ def test_get_dependencies(self):
272+ dependencies = get_dependencies("/", "apt")
273+ self.assertTrue(len(dependencies) > 2)
274+ self.assertIn("libc6", [or_dep[0].name for or_dep in dependencies])
275+
276+ def test_get_installed_files_for_pkg(self):
277+ apt_owned_files = get_installed_files_for_pkg("/", "apt")
278+ self.assertIn("/usr/bin/apt-get", apt_owned_files)
279+
280+ def test_get_elf_files(self):
281+ files = get_elf_files("/bin")
282+ self.assertIn("/bin/mv", files)
283+
284+ def test_get_elf_files_no_elf_files(self):
285+ files = get_elf_files("/var/lib/apt/periodic/")
286+ self.assertEqual(files, set([]))
287+
288+ def test_get_required_libraries(self):
289+ required_libs = get_required_libraries("/bin/mv")
290+ self.assertIn("libc.so.6", required_libs)

Subscribers

People subscribed via source and target branches

to all changes: