Merge lp:~mvo/snapcraft/python3-project into lp:~snappy-dev/snapcraft/core

[Michael Vogt]

Add python3 plugin

This is a bit of a RFC branch.

It adds a python3-project plugin for python3 parts. It is not using pip/pypi (for projects not on pypi). My idea was to have a python3-project and a pip3-project. But I do wonder if they should be separate or if python3-project should simply support both and we infer that its pip from the source: option (i.e. "source: pip:pkgname").

The other question I have is what the best layout for my py3-sponeshaker example is. The idea is that its a example that uses a github py3 module to do sha3 and builds a simple cmdline tool on top of this that uses the sha3 python-module. The sha3sum.py is just a single file that needs to be in the top-level of the snap. I wonder if we should have "copy" plugin or something like this that just puts the one file into the snap. What do you think is the best approach here?

And finally - python3 seems to be not have a official linux binary download. Which is a bit of a bummer. But it seems its not too terrible as building on my box takes only ~2min (and ~170mb diskspace). I will look into this next.

[Michael Terry (mterry)]

Spongeshaker is such a good project name. :)

I think it makes sense to have a "python3-project" plugin for setup.py-style projects.

And it probably makes sense to also have a "pip3" (or pypi3?) plugin for the use case of using the parts list as dependencies for your own project. I'm also not 100% on whether that's better or worse than "source: pip:spongeshaker"

As for the layout... I think it does make sense to offer a simple "copy" or "files" plugin. Could take a map of input files to output path?

I bet the snapcraft.yaml for your example project COULD look like:

"""
parts:
  spongeshaker:
    plugin: pip3
project:
  plugin: copy
  files:
    sha3sum.py: sha3sum
snappy-metadata: meta
"""

(Uses some concepts from my recent "discussion items" email.) How does that feel?

And as for how to get python3... Maybe we could use the ubuntu package? But getting it and building it from upstream is also pretty nice if it's not too bad. Does it have many dependencies?

A couple small comments inline too.

[Michael Vogt (mvo)]

Thanks! I renamed the example and addressed the inline comments. Having a python3-project and a pypi3/pip3 plugin sounds reasonable. I create a branch on top of this one for the copy plugin so that the example is complete (lp:~mvo/snapcraft/copy-plugin). The example snapcraft.yaml is slightly different from the above proposal as there is no "project" yet AIUI. Should be trivial to migrate though.

[Barry Warsaw (barry)]

On Jul 14, 2015, at 12:33 PM, Michael Vogt wrote:

>And finally - python3 seems to be not have a official linux binary
>download. Which is a bit of a bummer.

It's pretty much impossible to provide that upstream, given the wide range of
ABI differences among the Linux landscape.

[Michael Terry (mterry)]

More comments.

[Michael Vogt (mvo)]

@Barry: aha, thanks! Thats useful to know :)

@Michael: Thanks a bunch for the review. I addressed the points (hopefully) and this should be ready.

[Michael Terry (mterry)]

A couple comments, mostly about the environment stuff.

Looks nice and clean. Encouraging, in terms of how hard it is to write plugins.

[Michael Terry (mterry)]

Oh hrm. I asked about testing the snap. Snapcraft's default "wrap the user's binaries in a script that sets up the environment for them" logic wouldn't find your sha3sum script, because it's not in bin/ or usr/bin/...

Now that Snapcraft will know the metadata for the snap ahead of time... Maybe it should only wrap the binaries listed in package.yaml. That way it could find them anywhere.

[Michael Vogt (mvo)]

Thanks for your comments, I replied inline and fixed the issues.

As for:
> Snapcraft's default "wrap the user's binaries in a script that sets up the environment for them
> logic wouldn't find your sha3sum script, because it's not in bin/ or usr/bin/...

I copy the sha3sum.py into bin/ in the copy-plugin branch now. I think the idea to only wrap stuff that is mentioned in binaries in package.yaml is great.

[Michael Terry (mterry)]

OK, looks nice. :)

Here's what I found out about import paths: Ubuntu's version of python (and maybe upstream's too?) looks at where it's running from to figure out it's prefix. And installs a bunch of standard import locations into sys.path.

I had a hard time believing your imports were working ("PYTHONPATH=%s:$PYTHONPATH" % root" in particular should be %s/usr/lib/python2.7/dist-packages/ or the like). So I wanted to make a python2 version, something I knew wouldn't be on my install of snappy.

Here's what sys.path looks like with NO extra variables set (besides PATH to find python2 in the first place):

['/apps/spongeshaker.sideload/0/bin', '/apps/spongeshaker.sideload/0', '/apps/spongeshaker.sideload/0/usr/lib/python2.7', '/apps/spongeshaker.sideload/0/usr/lib/python2.7/plat-x86_64-linux-gnu', '/apps/spongeshaker.sideload/0/usr/lib/python2.7/lib-tk', '/apps/spongeshaker.sideload/0/usr/lib/python2.7/lib-old', '/apps/spongeshaker.sideload/0/usr/lib/python2.7/lib-dynload', '/apps/spongeshaker.sideload/0/usr/lib/python2.7/dist-packages']

So... I think we don't need ANY of the env() lines in this branch. PATH is automatically done for bin and usr/bin by snapcraft. PYTHONHOME and PYTHONPATH are automatically discovered by python.

So that's nice...!

So I guess drop the env() bits and I'm good with this branch. I can later propose my python2 version (which is largely a copy/paste).

Ideally the python3-project plugin would also get all dependencies listed in requirements.txt or in setup.py and install them into the same part. But that can come along with the pip3 plugin (similar logic I'm guessing).

[Michael Vogt (mvo)]

Thanks a lot Mike for digging into the details of this. Thats indeed great news. I removed them all now (except for PATH).

[Michael Terry (mterry)]

Nice! :) Setting PATH can also be left out, since snapcraft core does the standard PATH locations (bin and usr/bin) for all the parts. But it's harmless.

review: Approve

[Michael Terry (mterry)]

This needs a commit message and I also noticed one thing! You still have snapFiles, when you want snap_files now.

review: Needs Fixing

[Michael Terry (mterry)]

And pullBranch is now pull_branch

[Michael Vogt (mvo)]

Thanks again. I fixed snap_files() in r87 and removed setting PATH along the way.

[lp:~mvo/snapcraft/python3-project]

87. By Michael Vogt on 2015-07-17

fix snapFiles->snap_files and remove env() complettely (snapcraft sets this already). Thanks mterry!

88. By Michael Vogt on 2015-07-17

merged lp:snapcraft and renamed pullBranch -> pull_branch

[Michael Terry (mterry)]

Looks great!

review: Approve

=== added directory 'examples/py3-project'
=== added directory 'examples/py3-project/meta'
=== added file 'examples/py3-project/meta/package.yaml'
--- examples/py3-project/meta/package.yaml	1970-01-01 00:00:00 +0000
+++ examples/py3-project/meta/package.yaml	2015-07-17 07:14:10 +0000
@@ -0,0 +1,5 @@
+name: spongeshaker
+version: 0
+vendor: [e-mail]
+binaries:
+ - name: ./sha3sum

=== added file 'examples/py3-project/meta/readme.md'
--- examples/py3-project/meta/readme.md	1970-01-01 00:00:00 +0000
+++ examples/py3-project/meta/readme.md	2015-07-17 07:14:10 +0000
@@ -0,0 +1,1 @@
+A python sha3 implementation

=== added file 'examples/py3-project/sha3sum.py'
--- examples/py3-project/sha3sum.py	1970-01-01 00:00:00 +0000
+++ examples/py3-project/sha3sum.py	2015-07-17 07:14:10 +0000
@@ -0,0 +1,15 @@
+#!/usr/bin/env python3
+
+import sys
+
+import spongeshaker.sha3
+
+
+if __name__ == "__main__":
+    # 224 is the default from sha3sum
+    h = spongeshaker.sha3.sha3_224()
+    with open(sys.argv[1], "rb") as fp:
+        data = fp.read()
+    h.update(data)
+    print(h.hexdigest())
+

=== added file 'examples/py3-project/snapcraft.yaml'
--- examples/py3-project/snapcraft.yaml	1970-01-01 00:00:00 +0000
+++ examples/py3-project/snapcraft.yaml	2015-07-17 07:14:10 +0000
@@ -0,0 +1,6 @@
+parts:
+  spongeshaker:
+    plugin: python3-project
+    source: git://github.com/markokr/spongeshaker.git
+snap:
+  meta: meta

=== added file 'plugins/python3-project.yaml'
--- plugins/python3-project.yaml	1970-01-01 00:00:00 +0000
+++ plugins/python3-project.yaml	2015-07-17 07:14:10 +0000
@@ -0,0 +1,8 @@
+module: python3_project
+requires:
+  - python3
+options:
+  source:
+    required: true
+  configflags:
+    required: false

=== added file 'plugins/python3.yaml'
=== added file 'snapcraft/plugins/python3.py'
--- snapcraft/plugins/python3.py	1970-01-01 00:00:00 +0000
+++ snapcraft/plugins/python3.py	2015-07-17 07:14:10 +0000
@@ -0,0 +1,43 @@
+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
+#
+# Copyright (C) 2015 Canonical Ltd
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import snapcraft
+from snapcraft.plugins.ubuntu import UbuntuPlugin
+
+
+class Python3Plugin(snapcraft.BasePlugin):
+
+    def __init__(self, name, options):
+        super().__init__(name, options)
+
+        class Py3Options:
+            package = "python3-dev"
+        self.ubuntu = UbuntuPlugin(name, Py3Options())
+
+    # note that we don't need to set PYTHONHOME here,
+    # python discovers this automatically from it installed
+    # location, see https://code.launchpad.net/~mvo/snapcraft/python3-project/+merge/264521/comments/664308
+    #
+    # PATH is automatically set by snapcraft
+
+    def pull(self):
+        return self.ubuntu.pull()
+
+    def build(self):
+        return self.ubuntu.build()
+
+    def snap_files(self):
+        return self.ubuntu.snap_files()

=== added file 'snapcraft/plugins/python3_project.py'
--- snapcraft/plugins/python3_project.py	1970-01-01 00:00:00 +0000
+++ snapcraft/plugins/python3_project.py	2015-07-17 07:14:10 +0000
@@ -0,0 +1,31 @@
+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
+#
+# Copyright (C) 2015 Canonical Ltd
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import snapcraft
+
+
+class Python3ProjectPlugin(snapcraft.BasePlugin):
+
+    # note that we don't need to setup env(), python figures it out
+    # see python3.py for more details
+
+    def pull(self):
+        return self.pull_branch(self.options.source)
+
+    def build(self):
+        return self.run(
+            ["python3", "setup.py", "install", "--install-layout=deb",
+             "--prefix=%s/usr" % self.installdir])