Merge lp:~mvo/click/generate-framework-initial.libs into lp:click/devel

[Michael Vogt]

Generate list of shared libraries available in a fresh click chroot

This branch generates a list of initial libraries in a click chroot.

This is done in order to support the following two use-cases:
(1) detect if a click package links against libraries that are not part of the framework (and needs bundling)
(2) automatically bundle libraries when "click build --with-bundle-libraries" (or somesuch) is used

For (1) I have a branch lp:~mvo/click-reviewers-tools/check-libs that "just" needs to get the data what libs are available for what framework/arch combination.

If this approach to generate the data looks sensible I will update the click-reviewers-tools branch to use the data from this branch. Otherwise I would love to discuss other options.

Thanks for your consideration,
 Michael

p.s. Note that this is currently not looking at the individual symbols, just at the binaries DT_NEEDED values. If we want to look at the individual symbols then a more sophisticated approach will be needed.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:487
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~mvo/click/generate-framework-initial.libs/+merge/225810/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/click-devel-ci/26/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-amd64-ci/28/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/26/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-i386-ci/26/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/click-devel-ci/26/rebuild

review: Needs Fixing (continuous-integration)

[Colin Watson (cjwatson)]

I think if we're going to do this we need some way to ensure that the list is kept up to date when people upgrade the chroot.

review: Needs Fixing

[Colin Watson (cjwatson)]

We should probably only take the top-level dependencies, too. That implies that perhaps we can just figure this out from the framework package's dependencies at run-time?

[Michael Vogt (mvo)]

Thanks for the feedback! If we only take the so files of the toplevel dependencies of ubuntu-sdk-libs into account, then the list of supported "*.so.*" files is rather small:
"""
libQt5Script.so.5
libQt5Script.so.5.3
libQt5Script.so.5.3.1
libcontent-hub.so.0
libcontent-hub.so.0.0.1
libqt5keychain.so.0
libqt5keychain.so.0.3.0
"""

With one additional dependency level I get a list of 195 libs but I'm not sure that is really what we want (i.e. why two levels and not all levels). If we only want to follow the first dependency level, then it seems like the sdk-libs ubuntu-touch seed needs more entries. It will also mean that for something that uses e.g. Qt5GUI.so we need to ship ~30 additional libs including libX11.so.6, libstdc++.so.6, libpcre.so.3 etc if we take this narrow approach.

[Colin Watson (cjwatson)]

I think it's definitely worth looking at fleshing out sdk-libs, although I think germinate-update-metapackage might skip things in inner seeds; worth experimenting.

Why would we need to ship libX11.so.6 et al? We should only need to look at direct DT_NEEDED entries in object linkage, not follow the whole ldd chain. Hopefully binaries that link against Qt5GUI don't in general also overlink against all its transitive dependencies.

[Michael Vogt (mvo)]

Thanks Colin!

I created a (somewhat ugly) branch in lp:~mvo/click/click-check-libs that implements your suggestion to use only the first dependency level. Here is a example output:
"""
$ PYTHONPATH=. ./bin/click verify --allow-unauthenticated --check-libs com.ubuntu.camera_3.0.0.387_armhf.click
[..warnings about missing pkgs in the click chroot.. ]
libQt5Core.so.5
libQt5Gui.so.5
libQt5Multimedia.so.5
libQt5Qml.so.5
libQt5Quick.so.5
libc.so.6
libgcc_s.so.1
libstdc++.so.6
"""
which looks not bad, once libQt5* is explicitely seeded. What should happen with libc/libgcc/libstdc++, should this be added to a default whitelist?

[Michael Vogt (mvo)]

Some more libs I noticed when checking our clicks:
libpam.so.0
libgcc_s.so.1
libubuntuoneauth-2.0.so.0
libexpat.so.1
libm.so.6
libz.so.1
libcontent-hub.so.0

The libcontent-hub.so.0 indicates that we need to ensure to install the full ubuntu-sdk-libs:$arch into the click chroot - alternatively we could use apt-file to get this data.

[Michael Vogt (mvo)]

This MP is no longer needed but the comments maybe insightful so I don't delete it right away.

Unmerged revisions

487. By Michael Vogt on 2014-07-07

Generate list of initially installed libs on chroot creation

This will allow us to detect if a click package links against
libraries that are not part of the click package and not part
of a framework.

=== modified file 'click/chroot.py'
--- click/chroot.py	2014-07-03 11:03:24 +0000
+++ click/chroot.py	2014-07-07 11:29:20 +0000
@@ -318,7 +318,7 @@
                 """) % proxy)
         return apt_conf_f
 
-    def _generate_finish_script(self, mount, build_pkgs):
+    def _generate_finish_script(self, mount, build_pkgs, cross_libdir):
         finish_script = "%s/finish.sh" % mount
         with open(finish_script, 'w') as finish:
             finish.write(dedent("""\
@@ -342,11 +342,19 @@
             if [ ! -r /dev/stdin ];  then ln -s /proc/self/fd/0 /dev/stdin;  fi
             if [ ! -r /dev/stdout ]; then ln -s /proc/self/fd/1 /dev/stdout; fi
             if [ ! -r /dev/stderr ]; then ln -s /proc/self/fd/2 /dev/stderr; fi
+            # Generate what libs we have
+            echo "Generating default libs list"
+            mkdir -p /var/lib/click
+            find {cross_libdir} -name "*.so.*" | while read -r file
+            do
+              echo $file
+            done | sort | uniq > /var/lib/click/initial.libs
             # Clean up
             rm /finish.sh
             apt-get clean
             """).format(target_arch=self.target_arch,
-                        build_pkgs=' '.join(build_pkgs)))
+                        build_pkgs=' '.join(build_pkgs),
+                        cross_libdir=cross_libdir))
         return finish_script
 
     def _debootstrap(self, components, mount):
@@ -434,7 +442,13 @@
         os.remove("%s/sbin/initctl" % mount)
         os.symlink("%s/bin/true" % mount, "%s/sbin/initctl" % mount)
         self._generate_apt_proxy_file(mount, proxy)
-        finish_script = self._generate_finish_script(mount, build_pkgs)
+        if self.native_arch == self.target_arch:
+            cross_libdir = "/lib /lib64 /usr/lib"
+        else:
+            triplet = self.dpkg_architecture["DEB_HOST_GNU_TYPE"]
+            cross_libdir = "/usr/lib/%s" % triplet
+        finish_script = self._generate_finish_script(
+            mount, build_pkgs, cross_libdir)
         self._make_executable(finish_script)
         command = ["/finish.sh"]
         ret_code = self.maint(*command)

=== modified file 'click/tests/test_chroot.py'
--- click/tests/test_chroot.py	2014-07-02 09:07:13 +0000
+++ click/tests/test_chroot.py	2014-07-07 11:29:20 +0000
@@ -216,7 +216,8 @@
         chroot = ClickChroot("i386", "ubuntu-sdk-14.04")
         finish_script = chroot._generate_finish_script(
             self.temp_dir,
-            ["build-pkg-1", "build-pkg-2"])
+            ["build-pkg-1", "build-pkg-2"],
+            "/usr/lib/arm-linux-meep")
         with open(finish_script) as f:
             self.assertEqual(f.read(), dedent("""\
             #!/bin/bash
@@ -239,6 +240,13 @@
             if [ ! -r /dev/stdin ];  then ln -s /proc/self/fd/0 /dev/stdin;  fi
             if [ ! -r /dev/stdout ]; then ln -s /proc/self/fd/1 /dev/stdout; fi
             if [ ! -r /dev/stderr ]; then ln -s /proc/self/fd/2 /dev/stderr; fi
+            # Generate what libs we have
+            echo "Generating default libs list"
+            mkdir -p /var/lib/click
+            find /usr/lib/arm-linux-meep -name "*.so.*" | while read -r file
+            do
+              echo $file
+            done | sort | uniq > /var/lib/click/initial.libs
             # Clean up
             rm /finish.sh
             apt-get clean
@@ -338,3 +346,31 @@
                 ("ubuntu-sdk-14.04-dev99", "ubuntu-sdk-14.04"),
             ):
             self.assertEqual(strip_dev_series_from_framework(have), want)
+
+    def test_chroot_create_generates_finished_script_native(self):
+        self.use_temp_dir()
+        chroot = FakeClickChroot(
+            "i386", "ubuntu-sdk-14.04",
+            chroots_dir=self.temp_dir, temp_dir=self.temp_dir)
+        chroot.native_arch = "i386"
+        chroot.create()
+        finish_script = os.path.join(
+            self.temp_dir, "click-ubuntu-sdk-14.04-i386", "finish.sh")
+        with open(finish_script) as f:
+            self.assertIn(
+                'find /lib /lib64 /usr/lib -name "*.so.*" |', f.read())
+
+    def test_chroot_create_generates_finished_script_foreign(self):
+        self.use_temp_dir()
+        chroot = FakeClickChroot(
+            "armhf", "ubuntu-sdk-14.04", 
+            chroots_dir=self.temp_dir, temp_dir=self.temp_dir)
+        chroot.native_arch = "i386"
+        chroot.create()
+        finish_script = os.path.join(
+            self.temp_dir, "click-ubuntu-sdk-14.04-armhf", "finish.sh")
+        with open(finish_script) as f:
+            haystack = f.read()
+            self.assertIn(
+                'find /usr/lib/arm-linux-gnueabihf -name "*.so.*" |',
+                haystack)