Merge lp:~mikemc/ubuntuone-windows-installer/package-everything into lp:ubuntuone-windows-installer

[Mike McCracken]

- Package all helper apps, use symlinks to share resources (LP: #1025342)

- Package all helper apps, use symlinks to share resources (LP: #1025342)

This also includes a couple other changes, including some code cleanup.

NOTE to testers:
The script currently expects to be run from a branch of ubuntuone-windows-installer inside the parts/ directory of a buildout. However, because the buildout environment itself is a checkout of the windows installer code, this is a little silly. It dates from before I understood the buildout.
That is fixed, but saved for a future merge.

To run with successful feelings, you need PYTHONPATH to include:
- a built copy of lp:~diego.sarmentero/+junk/python-macfsevents
- current trunk dirspec
- the same directory as setup-mac.py (to get conf.py)
- current trunk of ubuntu-sso-client, ubuntuone-client, and ubuntuone-storage-protocol.

You will also need to run setup-mac using the buildout python.

To run the resulting executable, you can do the following:

% XDG_DATA_HOME=~/Documents/Canonical/Source/buildout-env/scripts/devsetup/parts/ubuntuone-storage-protocol/data XDG_CONFIG_HOME=/Users/mmccrack/Documents/Canonical/Source/buildout-env/scripts/devsetup/parts/ubuntuone-client/data/ U1_DEBUG=1 dist/ubuntuone-control-panel-qt/UbuntuOne.app/Contents/MacOS/ubuntuone-control-panel-qt

However, with current trunk of storage-protocol as of this writing, you will see errors trying to find the SSL certs in /etc/ssl. See bug #1025950 for progress on that issue.

[Brian Curtin (brian.curtin)]

I'll leave the actual approval to someone who has knowledge of the deeper Mac things in here, so just entering as a comment for now, but on the surface nothing jumps out as being incorrect as just general Python code.

99 + print "WARNING: got OSError %r in rmtree(%r)." \
100 + % (e, os.path.join(INSTALL_DIR, "lib"))

Could you wrap this in parentheses rather than using the '\' line continuation?

[lp:~mikemc/ubuntuone-windows-installer/package-everything]

130. By Mike McCracken on 2012-07-18

style fix and comment-out the right unused executable

[Alejandro J. Cura (alecu)]

Great branch!

review: Approve

[Manuel de la Peña (mandel)]

Some stupid comments:

Is there a errno in a constant to be user here:

98 + if errno != 2:

is better than simply using 2, right?

What happens in '.py' is already present, we might as well test for it just in case:

157 + mainscript = os.path.join("bin", exename + ".py")

Shall we just remove the following:

189 + #NOTE: Disabled because this is not used as of 07/18/2012.
190 + # sso_ssl_app_path = do_setup(True,
191 + # "com.ubuntu.sso.ssl-cert",
192 + # "ubuntu-sso-ssl-certificate-qt")

Does it really give any required info for the next maintainers?

In the following:

406 + # copy everything in resources that isn't include, lib, or *.app:
407 + resources_to_copy = [p for p in helper_src_resources if
408 + p not in ['include', 'lib']]
409 +
410 + for rsrc_name in resources_to_copy:
411 + shutil.copy(os.path.join(helper_src_resources_path, rsrc_name),
412 + os.path.join(helper_dest_resources_path, rsrc_name))
413 +
414 + # link to resources/include and resources/lib from the main bundle:
415 + resources_to_link = [os.path.join("Resources", p)
416 + for p in ['include', 'lib']]

['include', 'lib'] are the include and resources, would it be a good idea to make it a constants so that we don't have to chance it in the code. That way, if we have too add other strings we just have to modify it in a single place. Maybe the 'Resources' stirng is also a good candidate for a constant.

In the following:

337 + if len(dynload_paths) != 1:
338 + raise Exception("can't expand ", dynload_glob)

maybe len(dynload_paths) < 1 or len(dynload_paths) == 0, I don't exactly know if we should fail also if the length is bigger.

Besides the above I ran the script and everything seems to be ok.

review: Needs Fixing

[lp:~mikemc/ubuntuone-windows-installer/package-everything]

131. By Mike McCracken on 2012-07-25

Use errno constant for readability

132. By Mike McCracken on 2012-07-25

Add constant for Resources/lib and Resources/include

133. By Mike McCracken on 2012-07-25

Remove unused packaging code for sso-ssl-certificate-qt

134. By Mike McCracken on 2012-07-25

Improve error messages for globbing

[Mike McCracken (mikemc)]

> Some stupid comments:
>
> Is there a errno in a constant to be user here:
>
> 98 + if errno != 2:
>
> is better than simply using 2, right?

Good point - I forgot about the errno module.
I've changed it to check against errno.ENOENT.

> What happens in '.py' is already present, we might as well test for it just in
> case:
>
> 157 + mainscript = os.path.join("bin", exename + ".py")

The bin/exename.py file is written as part of the prepare step, see line 210 (of the file).

In the line you quoted, it could be there (and about to get overwritten) or not there (and that's OK).

If the file isn't there and you run 'python setup-mac.py py2app' instead of 'python setup-mac.py prepare py2app', you will get a pretty readable error from py2app about it.
It's not simple for us to check it here, though, because it might be OK for it to be missing - if we are going to do the prepare step and it's the first time we're calling setup() on this run of the script.

I've left it as-is, let me know if you think it should do more.

> Shall we just remove the following:
>
> 189 + #NOTE: Disabled because this is not used as of 07/18/2012.
> 190 + # sso_ssl_app_path = do_setup(True,
> 191 + # "com.ubuntu.sso.ssl-cert",
> 192 + # "ubuntu-sso-ssl-certificate-qt")
>
> Does it really give any required info for the next maintainers?

Not really, removed.

> In the following:
>
> 406 + # copy everything in resources that isn't include, lib, or *.app:
> 407 + resources_to_copy = [p for p in helper_src_resources if
> 408 + p not in ['include', 'lib']]
> 409 +
> 410 + for rsrc_name in resources_to_copy:
> 411 + shutil.copy(os.path.join(helper_src_resources_path, rsrc_name),
> 412 + os.path.join(helper_dest_resources_path, rsrc_name))
> 413 +
> 414 + # link to resources/include and resources/lib from the main bundle:
> 415 + resources_to_link = [os.path.join("Resources", p)
> 416 + for p in ['include', 'lib']]
>
> ['include', 'lib'] are the include and resources, would it be a good idea to
> make it a constants so that we don't have to chance it in the code. That way,
> if we have too add other strings we just have to modify it in a single place.

Agreed. I've made that change.

> Maybe the 'Resources' stirng is also a good candidate for a constant.

I didn't do this, because "Resources" will never change.

> In the following:
>
> 337 + if len(dynload_paths) != 1:
> 338 + raise Exception("can't expand ", dynload_glob)
>
> maybe len(dynload_paths) < 1 or len(dynload_paths) == 0, I don't exactly know
> if we should fail also if the length is bigger.

We should fail if the length is bigger, because then we don't know which one to use.
Maybe we'll end up packaging modules from multiple python versions at some point and then we'll need to make more changes to handle copying stuff from each of them. For now this code should break if it's != 1, though.

However, it could use a more descriptive error message. I've changed that.

> Besides the above I ran the script and everything seems to be ok.

Great!

[Manuel de la Peña (mandel)]

Cool, approving.

review: Approve

=== modified file 'scripts/setup-mac.py'
--- scripts/setup-mac.py	2012-07-13 19:02:48 +0000
+++ scripts/setup-mac.py	2012-07-25 20:04:22 +0000
@@ -31,11 +31,13 @@
 
 
 import copy
+import errno
 import glob
 import os
 import shutil
 import subprocess
 import sys
+from zipfile import ZipFile
 
 import conf
 
@@ -51,6 +53,15 @@
     sys.exit()
 
 
+APP_NAMES = {'ubuntu-sso-login': 'Ubuntu SSO Helper',
+             'ubuntu-sso-login-qt': 'Ubuntu Single Sign-On',
+             'ubuntu-sso-ssl-certificate-qt': 'Ubuntu SSO SSL Certificate',
+             'ubuntu-sso-proxy-creds-qt': 'Ubuntu SSO Proxy Credentials',
+             'ubuntuone-syncdaemon': 'UbuntuOne Syncdaemon',
+             'ubuntuone-control-panel-qt': 'UbuntuOne',
+             'ubuntuone-proxy-tunnel': 'UbuntuOne Proxy Tunnel',
+             }
+
 LOG_LEVEL = "DEBUG"
 LOG_FILE_SIZE = "1000000"
 
@@ -86,12 +97,38 @@
 
 # pylint is complaining about parent classes having too many methods
 # pylint: disable=R0904
+class Clean(Command):
+    """Remove built products from previous runs."""
+
+    user_options = []
+
+    def initialize_options(self):
+        """No-op."""
+
+    def finalize_options(self):
+        """No-op."""
+
+    def run(self):
+        """Clean up."""
+        for d in [INSTALL_DIR, 'dist', 'build']:
+            if os.path.exists(d):
+                print "Clean: removing", d
+                shutil.rmtree(d)
+        print "NOTE: exiting. 'clean' cannot be chained, "
+        print "      because we call setup() multiple times."
+        sys.exit()
+
+# pylint: enable=R0904
+
+
+# pylint is complaining about parent classes having too many methods
+# pylint: disable=R0904
 class DoNotPrepareAnything(Command):
     """No-op, lets us call setup multiple times without re-preparing."""
 
-    user_options = user_options = [("from-trunk", None, "noop."),
-                                   ("only-prepare=", None, "noop."),
-                                   ("source-dir=", None, "noop.")]
+    user_options = [("from-trunk", None, "noop."),
+                    ("only-prepare=", None, "noop."),
+                    ("source-dir=", None, "noop.")]
 
     def initialize_options(self):
         """No-op."""
@@ -101,11 +138,9 @@
 
     def finalize_options(self):
         """No-op."""
-        pass
 
     def run(self):
         """No-op."""
-        pass
 
 # pylint: enable=R0904
 
@@ -156,7 +191,7 @@
 
         print "Starting Prepare step:"
 
-        for folder in ["bin", INSTALL_DIR, "data"]:
+        for folder in ["bin", INSTALL_DIR, "data", "build"]:
             if not os.path.isdir(folder):
                 os.mkdir(folder)
 
@@ -216,8 +251,10 @@
         try:
             shutil.rmtree(os.path.join(INSTALL_DIR, "lib"))
         except OSError, e:
-            print "WARNING: OSError %r removing %r" \
-                  % (e, os.path.join(INSTALL_DIR, "lib"))
+            error, msg = e
+            if error != errno.ENOENT:
+                print ("WARNING: got OSError %r in rmtree(%r)."
+                       % (e, os.path.join(INSTALL_DIR, "lib")))
 
         start_dir = os.getcwd()
 
@@ -407,32 +444,48 @@
                                        "/imageformats/*.dylib")
     dylib_paths = qt_imageformat_plugins + qt_network_plugins
 
+    data_files = glob.glob("data/*")
+
     master_plist = Plist.fromFile('data/macapp_template.plist')
     master_options = {"includes": ['google.protobuf.descriptor',
                                    'sip',
                                    'twisted.web.resource',
                                    'twisted.web.client',
-                                   'ubuntu_sso.qt',
-                                   'ubuntu_sso.qt.ui',
-                                   'ubuntuone.storageprotocol',
-                                   'oauth'],
+                                   'oauth'
+                                   ],
                       "excludes": ["fsm", "PyQt4.uic"],
                       "frameworks": dylib_paths,
-                      "resources": ['data/qt.conf'],
+                      "resources": data_files,
                       "strip": True,
                       }
 
     # make sure we can see the prepared libraries:
     sys.path.insert(0, os.path.join(INSTALL_DIR, "lib", "site-packages"))
 
-    def do_setup(name, is_background, id, exename,
-                 mainscript,
+    helper_app_paths = []
+
+    def do_setup(is_background, id, exename,
                  verbose=False,
-                 doprepare=False):
+                 doprepare=False,
+                 is_main_app=False):
         """Customize plist & options and run setup.
 
+        - is_background controls showing a dock icon and menubar
+        - id is the bundle id, in reverse dns style
+        - exename is the name of the exe to be generated, usually
+          should just be the basename of the main script without .py
+
+        - verbose - sent to py2app, controls spewage
+        - doprepare - set to True only the first call of setup()
+
+        - is_main_app - if False, adds app path to list of helpers,
+          which have their dependencies merged into the main app
+          bundle in a post-processing step below.
+
         Returns the path of the generated .app bundle.
         """
+        name = APP_NAMES[exename]
+        mainscript = os.path.join("bin", exename + ".py")
 
         print "calling setup for:", name
         plist = copy.deepcopy(master_plist)
@@ -451,50 +504,47 @@
         else:
             prepare_class = DoNotPrepareAnything
 
-        setup(cmdclass=dict(prepare=prepare_class),
+        setup(cmdclass=dict(prepare=prepare_class,
+                            clean=Clean),
               app=[mainscript],
               options=dict(py2app=options),
               verbose=verbose)
 
-        return os.path.join("dist", exename, name + ".app")
+        app_path = os.path.join("dist", exename, name + ".app")
+        if not is_main_app:
+            helper_app_paths.append(app_path)
+        return app_path
 
-    sso_login_qt_app_path = do_setup("Ubuntu Single Sign-On",
-                                     True,
+    sso_login_qt_app_path = do_setup(True,
                                      "com.ubuntu.sso.login-qt",
                                      "ubuntu-sso-login-qt",
-                                     os.path.join("bin",
-                                                  "ubuntu-sso-login-qt.py"),
                                      doprepare=True)
 
-    sso_login_app_path = do_setup("Ubuntu SSO Helper",
-                                  True,
+    proxy_creds_app_path = do_setup(True,
+                                    "com.ubuntu.sso.ssl-cert",
+                                    "ubuntu-sso-proxy-creds-qt")
+
+    sso_login_app_path = do_setup(True,
                                   "com.ubuntu.sso.login",
-                                  "ubuntu-sso-login",
-                                  os.path.join("bin", "ubuntu-sso-login.py"))
+                                  "ubuntu-sso-login")
 
-    proxy_tunnel_app_path = do_setup("UbuntuOne Proxy Tunnel",
-                                     True,
+    proxy_tunnel_app_path = do_setup(True,
                                      "com.ubuntu.one.proxy-tunnel",
-                                     "ubuntuone-proxy-tunnel",
-                                     os.path.join("bin",
-                                                  "ubuntuone-proxy-tunnel.py"))
+                                     "ubuntuone-proxy-tunnel")
 
-    syncdaemon_app_path = do_setup("UbuntuOne Syncdaemon",
-                                   True,
+    syncdaemon_app_path = do_setup(True,
                                    "com.ubuntu.one.syncdaemon",
-                                   "ubuntuone-syncdaemon",
-                                   os.path.join("bin",
-                                                "ubuntuone-syncdaemon.py"))
+                                   "ubuntuone-syncdaemon")
 
-    control_panel_script = os.path.join("bin", "ubuntuone-control-panel-qt.py")
-    control_panel_app_path = do_setup("UbuntuOne",
-                                      False,
+    control_panel_app_path = do_setup(False,
                                       "com.ubuntu.one.controlpanel",
-                                      "ubuntuone-control-panel",
-                                      control_panel_script)
+                                      "ubuntuone-control-panel-qt",
+                                      is_main_app=True)
 
-    control_panel_resources_path = os.path.join(control_panel_app_path,
-                                                "Contents", "Resources")
+    control_panel_contents_path = os.path.join(control_panel_app_path,
+                                               "Contents")
+    control_panel_resources_path = os.path.join(control_panel_contents_path,
+                                                "Resources")
 
     # only do tweaks if you've done setup.py py2app
     if 'py2app' not in sys.argv:
@@ -505,60 +555,171 @@
 
     print "Starting post-setup tweaks"
 
-    # move plugins around
-    frameworks_paths = [os.path.join(p, "Contents", "Frameworks") for p in
-                        sso_login_qt_app_path,
-                        sso_login_app_path,
-                        control_panel_app_path]
+    # move Qt plugins into Frameworks/plugins/subdir/
+    control_panel_frameworks_path = os.path.join(control_panel_contents_path,
+                                                 "Frameworks")
 
     plugin_names = [os.path.basename(f) for f in dylib_paths]
 
-    for f_path in frameworks_paths:
-        bearers_dir = os.path.join(f_path, "plugins", "bearer")
-        imagefmts_dir = os.path.join(f_path, "plugins", "imageformats")
-        try:
-            os.makedirs(imagefmts_dir)
-        except Exception as e:
-            print "ignoring exception in makedirs(%r): %r" % (imagefmts_dir,
-                                                              e)
-
-        try:
-            os.makedirs(bearers_dir)
-        except Exception as e:
-            print "ignoring exception in makedirs(%r): %r" % (bearers_dir, e)
-
-        try:
-            for plugin_name in plugin_names:
-                if "bearer" in plugin_name:
-                    print "moving ", os.path.join(f_path, plugin_name), "to"
-                    print "       ", os.path.join(bearers_dir, plugin_name)
-                    os.rename(os.path.join(f_path, plugin_name),
-                              os.path.join(bearers_dir, plugin_name))
-                else:
-                    os.rename(os.path.join(f_path, plugin_name),
-                              os.path.join(imagefmts_dir, plugin_name))
-                    print "moving ", os.path.join(f_path, plugin_name), "to"
-                    print "       ", os.path.join(imagefmts_dir, plugin_name)
-        except Exception as e:
-            print "ERROR moving:", e
-
-    print "copying helper apps into main app"
-
-    def copy_helper(helper_path):
-        """Copy sub-apps into main app resources folder."""
+    bearers_dir = os.path.join(control_panel_frameworks_path,
+                               "plugins", "bearer")
+    imagefmts_dir = os.path.join(control_panel_frameworks_path,
+                                 "plugins", "imageformats")
+    try:
+        os.makedirs(imagefmts_dir)
+    except Exception as e:
+        print "ignoring exception in makedirs(%r): %r" % (imagefmts_dir,
+                                                          e)
+
+    try:
+        os.makedirs(bearers_dir)
+    except Exception as e:
+        print "ignoring exception in makedirs(%r): %r" % (bearers_dir, e)
+
+    try:
+        for plugin_name in plugin_names:
+            plugin_src = os.path.join(control_panel_frameworks_path,
+                                      plugin_name)
+
+            if "bearer" in plugin_name:
+                plugin_dest = os.path.join(bearers_dir, plugin_name)
+            else:
+                plugin_dest = os.path.join(imagefmts_dir, plugin_name)
+
+            os.rename(plugin_src, plugin_dest)
+
+    except Exception as e:
+        print "WARNING, got exception moving plugins around:", e
+
+    def get_packages_zip_file(resources_path):
+        """Get a ZipFile instance of the site_packages for an app."""
+        zip_glob = os.path.join(resources_path,
+                                'lib', 'python*',
+                                'site-packages.zip')
+        zip_paths = glob.glob(zip_glob)
+        if len(zip_paths) != 1:
+            raise Exception("zip_glob %r expanded to %r, wanted 1 path." %
+                            (zip_glob, zip_paths))
+
+        return ZipFile(zip_paths[0], 'a')
+
+    def get_dynload_files(resources_path):
+        """Get a set of files in the lib-dynload directory."""
+        file_set = set()
+        dynload_glob = os.path.join(resources_path,
+                                    'lib', 'python*',
+                                    'lib-dynload')
+        dynload_paths = glob.glob(dynload_glob)
+        if len(dynload_paths) != 1:
+            raise Exception("dynload_glob %r expanded to %r, wanted 1 path." %
+                            (dynload_glob, dynload_paths))
+
+        dynload_path = dynload_paths[0]
+        for root, dirs, files in os.walk(dynload_path):
+            for f in files:
+                path = os.path.join(root, f)
+                relpath = os.path.relpath(path, dynload_path)
+                file_set.add(relpath)
+        return dynload_path, file_set
+
+    main_zip = get_packages_zip_file(control_panel_resources_path)
+    main_zip_files = set(main_zip.namelist())
+
+    (main_dynload_path,
+     main_dynload_files) = get_dynload_files(control_panel_resources_path)
+
+    print "Copying helper apps into main app"
+
+    for helper_path in helper_app_paths:
+
+        print "Processing", helper_path
 
         helper_app_name = os.path.basename(helper_path)
-        dest = os.path.join(control_panel_resources_path,
+        helper_src_contents_path = os.path.join(helper_path,
+                                         "Contents")
+
+        helper_dest = os.path.join(control_panel_resources_path,
                             helper_app_name)
-        if os.path.exists(dest):
-            print "removing", dest
-            shutil.rmtree(dest)
-
-        shutil.copytree(helper_path, dest)
-
-    copy_helper(sso_login_qt_app_path)
-    copy_helper(sso_login_app_path)
-    copy_helper(proxy_tunnel_app_path)
-    copy_helper(syncdaemon_app_path)
+        if os.path.exists(helper_dest):
+            print "%r exists. removing it." % helper_dest
+            shutil.rmtree(helper_dest)
+
+        helper_dest_contents_path = os.path.join(helper_dest, "Contents")
+
+        os.makedirs(helper_dest_contents_path)
+
+        # copy actual Contents/MacOS and plist, symlink everything else
+
+        shutil.copytree(os.path.join(helper_src_contents_path, "MacOS"),
+                        os.path.join(helper_dest_contents_path, "MacOS"))
+
+        shutil.copy(os.path.join(helper_src_contents_path, "Info.plist"),
+                    os.path.join(helper_dest_contents_path, "Info.plist"))
+
+        # link to the rest of the items in the main app bundle, except:
+        # - the .app bundles, to avoid circular links
+        # __boot__.py needs to be the one from the subapp
+
+        resource_subdirs_to_link = ['include', 'lib']
+
+        helper_src_resources_path = os.path.join(helper_src_contents_path,
+                                                 "Resources")
+
+        helper_dest_resources_path = os.path.join(helper_dest_contents_path,
+                                                  "Resources")
+        os.mkdir(helper_dest_resources_path)
+
+        helper_src_resources = os.listdir(helper_src_resources_path)
+
+        # copy everything in resources that isn't include, lib, or *.app:
+        resources_to_copy = [p for p in helper_src_resources if
+                             p not in resource_subdirs_to_link]
+
+        for rsrc_name in resources_to_copy:
+            shutil.copy(os.path.join(helper_src_resources_path, rsrc_name),
+                        os.path.join(helper_dest_resources_path, rsrc_name))
+
+        # link to resources/include and resources/lib from the main bundle:
+        resources_to_link = [os.path.join("Resources", p)
+                             for p in resource_subdirs_to_link]
+
+        # also link Frameworks directory
+        for src_item in ["Frameworks", "PkgInfo"] + resources_to_link:
+            src_path = os.path.join(os.path.pardir,
+                                    os.path.pardir,
+                                    os.path.pardir,
+                                    src_item)
+            # add another pardir to make Resources/* items work:
+            if src_item.startswith("Resources"):
+                src_path = os.path.join(os.path.pardir, src_path)
+
+            link_name = os.path.join(helper_dest_contents_path, src_item)
+            os.symlink(src_path, link_name)
+
+        print "Merging site-packages.zip files"
+        helper_zip = get_packages_zip_file(helper_src_resources_path)
+        helper_zip_files = set(helper_zip.namelist())
+        add_from_helper = helper_zip_files - main_zip_files
+
+        for name in add_from_helper:
+            main_zip.writestr(name, helper_zip.read(name))
+        helper_zip.close()
+
+        print "Merging lib-dynload contents"
+        (helper_dynload_path,
+         helper_dynload_files) = get_dynload_files(helper_src_resources_path)
+        add_from_helper = helper_dynload_files - main_dynload_files
+
+        for filename in add_from_helper:
+            src = os.path.join(helper_dynload_path,
+                               filename)
+            dest = os.path.join(main_dynload_path,
+                                filename)
+            dest_dir = os.path.dirname(dest)
+            if not os.path.exists(dest_dir):
+                os.makedirs(dest_dir)
+            shutil.copy(src, dest)
+
+    main_zip.close()
 
     print "DONE. see dist/ for the .app."