Merge lp:~mterry/snapcraft/ros-project into lp:~snappy-dev/snapcraft/core

[Michael Terry]

Add simple ROS project support.

This is very preliminary. You have to manually add all the depends for your leaf ROS project in the parts list (from catkin up).

This isn't very different from other project types (python3-project for example). But it does make it a bit annoying to deal with. So this is a sort of minimally-viable-product branch.

There is an example with an extensive dependency hierarchy. But it is not a real working example! It needs Ubuntu packages included that aren't captured in snapcraft.yaml (because they would conflict with each other). To fix that, we need to allow the ubuntu plugin to install multiple packages in one part or dependency-flattening support.

This plugin would really benefit from dependency-injecting-and-flattening support. That way it could automatically create the very verbose dependency hierarchy from a few top-level parts. And it could inject all the Ubuntu package dependencies that are listed in each component's package.xml. But that's stage 2. This first branch is just basic ROS support.

I do a new thing in this branch, which is to hold data in a plugin-wide cache dir (~/.cache/snapcraft/plugins/ros-project). I use this for the rosdistro information that tells us where each branch for a given project is, rather than download the whole git branch each time. Might be useful idea in other plugins too (like go1.4 for the Go tarball?).

Add simple ROS project support.

This is very preliminary. You have to manually add all the depends for your leaf ROS project in the parts list (from catkin up).

This isn't very different from other project types (python3-project for example). But it does make it a bit annoying to deal with. So this is a sort of minimally-viable-product branch.

There is an example with an extensive dependency hierarchy. But it is not a real working example! It needs Ubuntu packages included that aren't captured in snapcraft.yaml (because they would conflict with each other). To fix that, we need to allow the ubuntu plugin to install multiple packages in one part or dependency-flattening support.

This plugin would really benefit from dependency-injecting-and-flattening support. That way it could automatically create the very verbose dependency hierarchy from a few top-level parts. And it could inject all the Ubuntu package dependencies that are listed in each component's package.xml. But that's stage 2. This first branch is just basic ROS support.

I do a new thing in this branch, which is to hold data in a plugin-wide cache dir (~/.cache/snapcraft/plugins/ros-project). I use this for the rosdistro information that tells us where each branch for a given project is, rather than download the whole git branch each time. Might be useful idea in other plugins too (like go1.4 for the Go tarball?).

[lp:~mterry/snapcraft/ros-project]

110. By Michael Terry on 2015-07-28

Use package name rather than part name in file tree

111. By Michael Terry on 2015-07-29

Merge from trunk

112. By Michael Terry on 2015-07-29

Don't use installdir in env() call, rather use given root

[Sergio Schvezov (sergiusens)]

I think ted is working on something based or not on this, but I want to get rid of the clutter here :-)

Thanks for contribution in any case, it opened up the path for the work ahead.

review: Disapprove

Unmerged revisions

112. By Michael Terry on 2015-07-29

Don't use installdir in env() call, rather use given root

111. By Michael Terry on 2015-07-29

Merge from trunk

110. By Michael Terry on 2015-07-28

Use package name rather than part name in file tree

109. By Michael Terry on 2015-07-28

Add README detailing some of the limitations of the example

108. By Michael Terry on 2015-07-28

Expand whole dependency hierarchy for test

107. By Michael Terry on 2015-07-27

Get python build depends in a generic way

106. By Michael Terry on 2015-07-27

Merge from trunk

105. By Michael Terry on 2015-07-27

Another snapshot

104. By Michael Terry on 2015-07-22

Snapshot of ros work

=== added directory 'examples/ros'
=== added file 'examples/ros/README'
--- examples/ros/README	1970-01-01 00:00:00 +0000
+++ examples/ros/README	2015-07-29 17:26:38 +0000
@@ -0,0 +1,7 @@
+This example is not a real working example.  It's just a technical demo of the dependency hierarchy and proof that we can build a bunch of ROS projects.  But it currently relies on packages to be installed on the host in order to build.  And those packages should really be included in the snap in order for the code to work once installed.
+
+To fix this, we need to allow multiple Ubuntu packages to not conflict with each other (either multiple packages listed in one part or dependency flattening).
+
+Also, ideally the user wouldn't have to list the whole hierarchy like in this snapcraft.yaml example.  Ideally, snapcraft would inject all those dependencies for you and you just list the top-level part.
+
+So this example is incomplete.  It is work-in-progress.

=== added directory 'examples/ros/meta'
=== added file 'examples/ros/meta/package.yaml'
--- examples/ros/meta/package.yaml	1970-01-01 00:00:00 +0000
+++ examples/ros/meta/package.yaml	2015-07-29 17:26:38 +0000
@@ -0,0 +1,3 @@
+name: rosexample
+version: 1.0
+vendor: "Mike Terry <[e-mail]>"

=== added file 'examples/ros/meta/readme.md'
--- examples/ros/meta/readme.md	1970-01-01 00:00:00 +0000
+++ examples/ros/meta/readme.md	2015-07-29 17:26:38 +0000
@@ -0,0 +1,1 @@
+ros example

=== added file 'examples/ros/snapcraft.yaml'
--- examples/ros/snapcraft.yaml	1970-01-01 00:00:00 +0000
+++ examples/ros/snapcraft.yaml	2015-07-29 17:26:38 +0000
@@ -0,0 +1,141 @@
+parts:
+  catkin:
+    plugin: ros-project
+    release: jade
+  cpp_common:
+    plugin: ros-project
+    release: jade
+    after: [catkin]
+  navigation:
+    plugin: ros-project
+    release: jade
+    after: [catkin]
+  genmsg:
+    plugin: ros-project
+    release: jade
+    after: [catkin]
+  gencpp:
+    plugin: ros-project
+    release: jade
+    after: [catkin, genmsg]
+  geneus:
+    plugin: ros-project
+    release: jade
+    after: [catkin, genmsg]
+  genlisp:
+    plugin: ros-project
+    release: jade
+    after: [catkin, genmsg]
+  genpy:
+    plugin: ros-project
+    release: jade
+    after: [catkin, genmsg]
+  rostime:
+    plugin: ros-project
+    release: jade
+    after: [catkin, cpp_common]
+  rosunit:
+    plugin: ros-project
+    release: jade
+    after: [catkin]
+  rosconsole:
+    plugin: ros-project
+    release: jade
+    after: [catkin, rostime, rosunit]
+  message_generation:
+    plugin: ros-project
+    release: jade
+    after: [catkin, gencpp]
+  roscpp_traits:
+    plugin: ros-project
+    release: jade
+    after: [catkin]
+  roscpp_serialization:
+    plugin: ros-project
+    release: jade
+    after: [catkin, cpp_common, roscpp_traits, rostime]
+  std_msgs:
+    plugin: ros-project
+    release: jade
+    after: [catkin, message_generation, geneus, genlisp, genpy]
+  message_runtime:
+    plugin: ros-project
+    release: jade
+    after: [catkin]
+  rosgraph_msgs:
+    plugin: ros-project
+    release: jade
+    after: [catkin, message_generation, geneus, genlisp, genpy, std_msgs, message_runtime, cpp_common, roscpp_serialization]
+  xmlrpcpp:
+    plugin: ros-project
+    release: jade
+    after: [catkin, cpp_common]
+  roscpp:
+    plugin: ros-project
+    release: jade
+    after: [catkin, cpp_common, message_generation, geneus, genlisp, genpy, rosconsole, roscpp_serialization, rosgraph_msgs, xmlrpcpp]
+  angles:
+    plugin: ros-project
+    release: jade
+    after: [catkin]
+  geometry_msgs:
+    plugin: ros-project
+    release: jade
+    after: [catkin, message_generation, geneus, genlisp, genpy, std_msgs, message_runtime, cpp_common, roscpp_serialization]
+  rostest:
+    plugin: ros-project
+    release: jade
+    after: [catkin]
+  message_filters:
+    plugin: ros-project
+    release: jade
+    after: [catkin, roscpp, rostest]
+  sensor_msgs:
+    plugin: ros-project
+    release: jade
+    after: [catkin, geometry_msgs]
+  actionlib_msgs:
+    plugin: ros-project
+    release: jade
+    after: [catkin, message_generation, geneus, genlisp, genpy, std_msgs, message_runtime, cpp_common, roscpp_serialization]
+  actionlib:
+    plugin: ros-project
+    release: jade
+    after: [catkin, actionlib_msgs, roscpp, rostest]
+  rosgraph:
+    plugin: ros-project
+    release: jade
+    after: [catkin]
+  rospy:
+    plugin: ros-project
+    release: jade
+    after: [catkin]
+  tf2_msgs:
+    plugin: ros-project
+    release: jade
+    after: [catkin, geometry_msgs, actionlib_msgs]
+  tf2:
+    plugin: ros-project
+    release: jade
+    after: [catkin, geometry_msgs, tf2_msgs]
+  tf2_py:
+    plugin: ros-project
+    release: jade
+    after: [catkin, rospy, tf2]
+  tf2_ros:
+    plugin: ros-project
+    release: jade
+    after: [catkin, actionlib, geometry_msgs, message_filters, rosgraph, rospy, tf2, tf2_py]
+  tf:
+    plugin: ros-project
+    release: jade
+    after: [catkin, angles, sensor_msgs, tf2_ros]
+  nav_msgs:
+    plugin: ros-project
+    release: jade
+    after: [catkin, geometry_msgs, actionlib_msgs]
+  map_server:
+    plugin: ros-project
+    release: jade
+    after: [catkin, roscpp, tf, nav_msgs]
+snappy-metadata: meta

=== added file 'plugins/ros-project.yaml'
--- plugins/ros-project.yaml	1970-01-01 00:00:00 +0000
+++ plugins/ros-project.yaml	2015-07-29 17:26:38 +0000
@@ -0,0 +1,7 @@
+module: ros_project
+build-tools: [cmake]
+options:
+    package:
+        required: false
+    release:
+        required: true

=== modified file 'snapcraft/__init__.py'
--- snapcraft/__init__.py	2015-07-29 04:33:25 +0000
+++ snapcraft/__init__.py	2015-07-29 17:26:38 +0000
@@ -65,33 +65,36 @@
     def isurl(self, url):
         return urllib.parse.urlparse(url).scheme != ""
 
-    def pull_bzr(self, source, source_tag=None):
+    def pull_bzr(self, source, source_tag=None, destdir=None):
+        destdir = destdir or self.sourcedir
         tag_opts = []
         if source_tag:
             tag_opts = ['-r', 'tag:' + source_tag]
-        if os.path.exists(os.path.join(self.sourcedir, ".bzr")):
-            return self.run(['bzr', 'pull'] + tag_opts + [source, '-d', self.sourcedir], cwd=os.getcwd())
+        if os.path.exists(os.path.join(destdir, ".bzr")):
+            return self.run(['bzr', 'pull'] + tag_opts + [source, '-d', destdir], cwd=os.getcwd())
         else:
-            os.rmdir(self.sourcedir)
-            return self.run(['bzr', 'branch'] + tag_opts + [source, self.sourcedir], cwd=os.getcwd())
-
-    def pull_git(self, source, source_tag=None, source_branch=None):
+            os.rmdir(destdir)
+            return self.run(['bzr', 'branch'] + tag_opts + [source, destdir], cwd=os.getcwd())
+
+    def pull_git(self, source, source_tag=None, source_branch=None, destdir=None):
+        destdir = destdir or self.sourcedir
+
         if source_tag and source_branch:
             logger.error("You can't specify both source-tag and source-branch for a git source (part '%s')." % self.name)
             snapcraft.common.fatal()
 
-        if os.path.exists(os.path.join(self.sourcedir, ".git")):
+        if os.path.exists(os.path.join(destdir, ".git")):
             refspec = 'HEAD'
             if source_branch:
                 refspec = 'refs/heads/' + source_branch
             elif source_tag:
                 refspec = 'refs/tags/' + source_tag
-            return self.run(['git', '-C', self.sourcedir, 'pull', source, refspec], cwd=os.getcwd())
+            return self.run(['git', '-C', destdir, 'pull', source, refspec], cwd=os.getcwd())
         else:
             branch_opts = []
             if source_tag or source_branch:
                 branch_opts = ['--branch', source_tag or source_branch]
-            return self.run(['git', 'clone'] + branch_opts + [source, self.sourcedir], cwd=os.getcwd())
+            return self.run(['git', 'clone'] + branch_opts + [source, destdir], cwd=os.getcwd())
 
     def pull_tarball(self, source, destdir=None):
         destdir = destdir or self.sourcedir

=== modified file 'snapcraft/common.py'
--- snapcraft/common.py	2015-07-23 17:19:25 +0000
+++ snapcraft/common.py	2015-07-29 17:26:38 +0000
@@ -29,15 +29,17 @@
 env = []
 
 
-def assemble_env():
-    return '\n'.join(['export ' + e for e in env])
-
-
-def run(cmd, **kwargs):
+def assemble_env(extra_env=[]):
+    # Use a set to reduce duplicates (which can sometimes cause very long
+    # lines that exceed linux limits) and order shouldn't matter for these.
+    return '\n'.join(['export ' + e for e in set(env + extra_env)])
+
+
+def run(cmd, extra_env=[], **kwargs):
     assert isinstance(cmd, list), "run command must be a list"
     # FIXME: This is gross to keep writing this, even when env is the same
     with tempfile.NamedTemporaryFile(mode='w+') as f:
-        f.write(assemble_env())
+        f.write(assemble_env(extra_env=extra_env))
         f.write('\n')
         f.write('exec $*')
         f.flush()
@@ -63,3 +65,9 @@
 
 def get_plugindir():
     return _plugindir
+
+
+def get_cachedir():
+    home_cache = os.path.join(os.environ['HOME'], '.cache')
+    env_cache = os.environ.get('XDG_CACHE_HOME', home_cache)
+    return os.path.join(env_cache, 'snapcraft')

=== modified file 'snapcraft/plugin.py'
--- snapcraft/plugin.py	2015-07-24 16:55:44 +0000
+++ snapcraft/plugin.py	2015-07-29 17:26:38 +0000
@@ -113,9 +113,12 @@
 
         for propName in dir(module):
             prop = getattr(module, propName)
-            if issubclass(prop, snapcraft.BasePlugin):
-                self.code = prop(part_name, options)
-                break
+            try:
+                if issubclass(prop, snapcraft.BasePlugin):
+                    self.code = prop(part_name, options)
+                    break
+            except TypeError:
+                continue
 
     def __str__(self):
         return self.part_names[0]

=== modified file 'snapcraft/plugins/autotools_project.py'
--- snapcraft/plugins/autotools_project.py	2015-07-22 18:23:16 +0000
+++ snapcraft/plugins/autotools_project.py	2015-07-29 17:26:38 +0000
@@ -15,10 +15,10 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 import os
-from snapcraft.plugins.make_project import MakePlugin
-
-
-class AutotoolsPlugin(MakePlugin):
+from snapcraft.plugins import make_project
+
+
+class AutotoolsPlugin(make_project.MakePlugin):
     def __init__(self, name, options):
         super().__init__(name, options)
         if self.options.configflags is None:

=== modified file 'snapcraft/plugins/cmake_project.py'
--- snapcraft/plugins/cmake_project.py	2015-07-22 18:23:16 +0000
+++ snapcraft/plugins/cmake_project.py	2015-07-29 17:26:38 +0000
@@ -14,10 +14,10 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-from snapcraft.plugins.make_project import MakePlugin
-
-
-class CMakePlugin(MakePlugin):
+from snapcraft.plugins import make_project
+
+
+class CMakePlugin(make_project.MakePlugin):
     def __init__(self, name, options):
         super().__init__(name, options)
         if self.options.configflags is None:

=== modified file 'snapcraft/plugins/go14_project.py'
--- snapcraft/plugins/go14_project.py	2015-07-22 18:23:16 +0000
+++ snapcraft/plugins/go14_project.py	2015-07-29 17:26:38 +0000
@@ -37,5 +37,5 @@
         return self.run(['cp', '-a', os.path.join(self.builddir, 'bin'), self.installdir])
 
     def run(self, cmd, **kwargs):
-        cmd = ['env', 'GOPATH=' + self.builddir] + cmd
-        return super().run(cmd, **kwargs)
+        extra_env = ['GOPATH=' + self.builddir]
+        return super().run(cmd, extra_env=extra_env, **kwargs)

=== added file 'snapcraft/plugins/ros_project.py'
--- snapcraft/plugins/ros_project.py	1970-01-01 00:00:00 +0000
+++ snapcraft/plugins/ros_project.py	2015-07-29 17:26:38 +0000
@@ -0,0 +1,136 @@
+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
+#
+# Copyright (C) 2015 Canonical Ltd
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import logging
+import os
+import xml.etree.ElementTree as ET  # vulnerabilities aren't a concern here
+
+import yaml
+
+from snapcraft import common
+from snapcraft.plugins import make_project
+
+
+logger = logging.getLogger(__name__)
+
+
+class RosProjectPlugin(make_project.MakePlugin):
+
+    def __init__(self, name, options):
+        super().__init__(name, options)
+        if not self.options.package:
+            # User didn't specify a package, use the part name
+            if name == 'ros-project':
+                logger.error('Part {} needs either a package option or a name'.format(name))
+                common.fatal()
+            self.options.package = name
+        self.packageinfo = ET.ElementTree()
+
+    def pull(self):
+        # Get main rosdistro info
+        url, tag = self.get_package_info()
+        # per http://www.ros.org/reps/rep-0141.html all release urls are git
+        if not self.get_source(url, source_tag=tag, source_type='git'):
+            return False
+
+        self.parse_package_xml()
+        return self.install_build_depends()
+
+    def build(self):
+        # We use a /ros/NAME prefix because the ROS build scripts don't handle
+        # an empty prefix well (which is our normal cmake prefix).  And each
+        # ROS project installs some common files that would conflict if we
+        # don't namespace them.  So we put them each in their own bucket.
+        return self.run(['cmake', '.', '-DCMAKE_INSTALL_PREFIX=/ros/{}'.format(self.options.package)]) and \
+            super().build()
+
+    def env(self, root):
+        env = []
+
+        package_root = os.path.join(root, 'ros', self.options.package)
+
+        # Our own python modules
+        pydirs = self.get_python_dirs(os.path.join(package_root, 'lib'))
+        # And our own build-depends sometimes need to be exported (listed in
+        # package.xml via <build_export_depend>, but let's just do them all)
+        pydirs += self.get_python_dirs(os.path.join(package_root, 'usr', 'local', 'lib'))
+        if pydirs:
+            env += ['PYTHONPATH={}:$PYTHONPATH'.format(':'.join(pydirs))]
+
+        # Tell cmake how to find our macro files
+        env += ['CMAKE_PREFIX_PATH={}:$CMAKE_PREFIX_PATH'.format(os.path.join(root, 'ros', self.options.package))]
+
+        return env
+
+    def snap_files(self):
+        # Skip our cmake files
+        return (['*'], ['ros/{0}/share/{0}/cmake'.format(self.options.package)])
+
+    def parse_package_xml(self):
+        package_file = os.path.join(self.sourcedir, 'package.xml')
+        if os.path.exists(package_file):
+            self.packageinfo.parse(package_file)
+
+    def install_build_depends(self):
+        depends = self.packageinfo.findall('./depend')
+        build_depends = self.packageinfo.findall('./build_depend')
+        pyprefix = 'python-'
+        pydir = os.path.join(self.installdir, 'ros', self.options.package)
+        for depend in depends + build_depends:
+            if depend.text.startswith(pyprefix):
+                self.makedirs(pydir)
+                pkgname = depend.text[len(pyprefix):].replace('-', '_')
+                if not self.run(['pip3', 'install', '--system', '--root', pydir, pkgname]):
+                    return False
+        return True
+
+    def get_package_info(self):
+        rosdistrodir = os.path.join(common.get_cachedir(), 'plugins', 'ros-project', 'rosdistro')
+        self.makedirs(rosdistrodir)
+        self.pull_git('git://github.com/ros/rosdistro.git', destdir=rosdistrodir)
+        with open(os.path.join(rosdistrodir, self.options.release, 'distribution.yaml')) as fp:
+            return self.find_package_info(yaml.load(fp))
+
+    def find_package_info(self, distinfo):
+        for repo in distinfo['repositories']:
+            if 'release' not in distinfo['repositories'][repo]:
+                continue
+            releaseinfo = distinfo['repositories'][repo]['release']
+            packages = releaseinfo.get('packages', [repo])
+            if self.options.package in packages:
+                tagscheme = releaseinfo['tags']['release']
+                tagscheme = tagscheme.replace('{package}', self.options.package)
+                tagscheme = tagscheme.replace('{version}', releaseinfo['version'])
+                return releaseinfo['url'], tagscheme
+        logger.error('Could not find release information for ROS package {}'.format(self.options.package))
+        common.fatal()
+        return None, None
+
+    def get_python_dirs(self, root):
+        try:
+            alldirs = os.listdir(root)
+        except Exception:
+            return []
+        return [os.path.join(root, x, 'dist-packages') for x in alldirs if x.startswith('python')]
+
+    def run(self, cmd, **kwargs):
+        extra_env = []
+        pylibdir = os.path.join(self.installdir, 'ros', self.options.package, 'usr', 'local', 'lib')
+        pydirs = self.get_python_dirs(pylibdir)
+        if pydirs:
+            pypaths = ':'.join(pydirs)
+            extra_env = ['PYTHONPATH={}:$PYTHONPATH'.format(pypaths)]
+        return super().run(cmd, extra_env=extra_env, **kwargs)