Merge lp:~ted/snapcraft/python-pip into lp:~snappy-dev/snapcraft/core

[Ted Gould]

Add in support for PIP requirements.txt files in the python plugins.

Add in support for PIP requirements.txt files in the python plugins.

[lp:~ted/snapcraft/python-pip]

149. By Ted Gould on 2015-09-14

Update to trunk

[Sergio Schvezov (sergiusens)]

Are you rendering the py3 and py2 examples useless here by removing handle_source options?

I also bring this question up often enough (internally at least); with this implemented in python instead of python-project, what is the reason for python-project (or any .*-project at all?)

[Sergio Schvezov (sergiusens)]

hey this looks good, it just seems that py2 took a different path than py3.

This also removes the use case (which is rather neat) that the py2-project and py3-project examples bring in.

review: Needs Fixing

[Sergio Schvezov (sergiusens)]

python2 and python3 for pip is fine fwiw

[lp:~ted/snapcraft/python-pip]

150. By Ted Gould on 2015-09-14

Grabbin' trunk

151. By Ted Gould on 2015-09-16

Put PYTHONPATH in the overall environment instead of a custom one

152. By Ted Gould on 2015-09-16

Move PYTHONPATH to the env function in Python 2 as well

153. By Ted Gould on 2015-09-16

Switch python2 to use target as well

[Ted Gould (ted)]

So, some slight modifications. Switched from --root to --target as it seems in recursive cases that causes "bad things to happen" in that it appends twice. But target doesn't have that problem. Also moved the PYTHONPATH to the standard snapcraft environment handling. Now I can build a Nova snap:

https://github.com/ted-gould/nova/blob/master/snapcraft.yaml

Still would like to figure out if we can fix the command line support in setuptools to not put in the full path in a shebang, but that's not really a pip issue.

[lp:~ted/snapcraft/python-pip]

154. By Ted Gould on 2015-09-17

Put back a require on python2

155. By Ted Gould on 2015-09-17

Return the source options to python2 project

[Ted Gould (ted)]

On Tue, 2015-09-15 at 05:53 +0000, Sergio Schvezov wrote:

> hey this looks good, it just seems that py2 took a different path than py3.
>
> This also removes the use case (which is rather neat) that the py2-project and py3-project examples bring in.

<snip>
I was confused, I didn't see your comments in the diff... handled now.
Thanks!

[lp:~ted/snapcraft/python-pip]

156. By Ted Gould on 2015-09-17

Updating to trunk

[Sergio Schvezov (sergiusens)]

This is what I see when doing

cd examples/webcam-webui
../../bin/snapcraft

...
Building config
python3 setup.py install --install-layout=deb --prefix=/home/sergiusens/source/launchpad.net/snapcraft/examples/webcam-webui/parts/config/install/usr
running install
Checking .pth file support in /home/sergiusens/source/launchpad.net/snapcraft/examples/webcam-webui/parts/config/install/usr/lib/python3/dist-packages/
/home/sergiusens/source/launchpad.net/snapcraft/examples/webcam-webui/parts/python3/install/usr/bin/python3 -E -c pass
TEST FAILED: /home/sergiusens/source/launchpad.net/snapcraft/examples/webcam-webui/parts/config/install/usr/lib/python3/dist-packages/ does NOT support .pth files
error: bad install directory or PYTHONPATH

You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from. The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /home/sergiusens/source/launchpad.net/snapcraft/examples/webcam-webui/parts/config/install/usr/lib/python3/dist-packages/

and your PYTHONPATH environment variable currently contains:

    '/home/sergiusens/source/launchpad.net/snapcraft/examples/webcam-webui/parts/python3/install/usr/lib/python3/dist-packages'

Here are some of your options for correcting the problem:

* You can choose a different installation directory, i.e., one that is
  on PYTHONPATH or supports .pth files

* You can add the installation directory to the PYTHONPATH environment
  variable. (It must then also be on PYTHONPATH whenever you run
  Python and want to use the package(s) you are installing.)

* You can set up the installation directory to support ".pth" files by
  using one of the approaches described here:

  https://pythonhosted.org/setuptools/easy_install.html#custom-installation-locations

Please make the appropriate changes for your system and try again.
Failed doing build for config!

[lp:~ted/snapcraft/python-pip]

157. By Ted Gould on 2015-09-17

Updating trunk

158. By Ted Gould on 2015-09-17

Grab sergio's test

159. By Sergio Schvezov on 2015-09-18

Environment fixes

[Sergio Schvezov (sergiusens)]

Nice

review: Approve

[Sergio Schvezov (sergiusens)]

Thanks! This was a tough one!

=== added directory 'integration-tests/data/pip-requirements'
=== added file 'integration-tests/data/pip-requirements/icon.svg'
--- integration-tests/data/pip-requirements/icon.svg	1970-01-01 00:00:00 +0000
+++ integration-tests/data/pip-requirements/icon.svg	2015-09-18 03:31:26 +0000
@@ -0,0 +1,2 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<svg />

=== added file 'integration-tests/data/pip-requirements/requirements.txt'
--- integration-tests/data/pip-requirements/requirements.txt	1970-01-01 00:00:00 +0000
+++ integration-tests/data/pip-requirements/requirements.txt	2015-09-18 03:31:26 +0000
@@ -0,0 +1,1 @@
+argparse==1.2.2

=== added file 'integration-tests/data/pip-requirements/snapcraft.yaml'
--- integration-tests/data/pip-requirements/snapcraft.yaml	1970-01-01 00:00:00 +0000
+++ integration-tests/data/pip-requirements/snapcraft.yaml	2015-09-18 03:31:26 +0000
@@ -0,0 +1,14 @@
+name: pip-test
+version: 1
+vendor: PIP Tests <[e-mail]>
+summary: Why do snapcraft files require a summary?
+description: This seems a bit excessive too
+icon: icon.svg
+
+parts:
+  python2:
+    type: python2
+    requirements: requirements.txt
+  python3:
+    type: python3
+    requirements: requirements.txt

=== modified file 'integration-tests/units/jobs.pxu'
--- integration-tests/units/jobs.pxu	2015-09-17 20:52:11 +0000
+++ integration-tests/units/jobs.pxu	2015-09-18 03:31:26 +0000
@@ -315,3 +315,14 @@
     test -e parts/mercurial/src/second
     ${SNAPCRAFT} pull
     test -e parts/mercurial/src/second
+
+id: snapcraft/normal/pip-requirements
+plugin: shell
+estimated_duration: 60
+command:
+    set -ex
+    cp -rT $PLAINBOX_PROVIDER_DATA/pip-requirements .
+    ${SNAPCRAFT} pull
+    test -f parts/python2/install/usr/lib/python2.7/argparse.py
+    test -f parts/python3/install/usr/lib/python3.4/argparse.py
+

=== modified file 'plugins/python2.yaml'
--- plugins/python2.yaml	2015-07-17 14:44:45 +0000
+++ plugins/python2.yaml	2015-09-18 03:31:26 +0000
@@ -0,0 +1,3 @@
+options:
+  requirements:
+    required: false

=== modified file 'plugins/python3.yaml'
--- plugins/python3.yaml	2015-07-15 15:28:27 +0000
+++ plugins/python3.yaml	2015-09-18 03:31:26 +0000
@@ -0,0 +1,3 @@
+options:
+  requirements:
+    required: false

=== modified file 'snapcraft/plugins/python2.py'
--- snapcraft/plugins/python2.py	2015-09-17 06:24:37 +0000
+++ snapcraft/plugins/python2.py	2015-09-18 03:31:26 +0000
@@ -14,6 +14,7 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+import os
 import snapcraft
 
 
@@ -24,6 +25,14 @@
         'python-setuptools',
     ]
 
+    def __init__(self, name, options):
+        if options.requirements:
+            self.requirements = options.requirements
+            self._PLUGIN_STAGE_PACKAGES.extend(['python-pkg-resources', 'python-setuptools'])
+        else:
+            self.requirements = None
+        super().__init__(name, options)
+
     def snap_fileset(self):
         return [
             '-usr/share',
@@ -32,3 +41,19 @@
     # note that we don't need to set PYTHONHOME here,
     # python discovers this automatically from it installed
     # location.  And PATH is automatically set by snapcraft.
+
+    def env(self, root):
+        return ["PYTHONPATH=%s" % os.path.join(
+            root, 'usr', 'lib', 'python2.7', 'dist-packages')]
+
+    def pull(self):
+        # A nice idea here would be to be asking setup tools
+        # to use the deb layout, but that doesn't work with
+        # prefix sadly
+
+        if self.requirements and not (self.run(
+                ['ln', '-s', os.path.join(self.installdir, 'usr', 'lib', 'python2.7', 'dist-packages'), os.path.join(self.installdir, 'usr', 'lib', 'python2.7', 'site-packages')]) and self.run(
+                ['python2', os.path.join(self.installdir, 'usr', 'bin', 'easy_install'), '--prefix', os.path.join(self.installdir, 'usr'), 'pip']) and self.run(
+                ['python2', os.path.join(self.installdir, 'usr', 'bin', 'pip2'), 'install', '--target', os.path.join(self.installdir, 'usr', 'lib', 'python2.7', 'site-packages'), '--requirement', os.path.join(os.getcwd(), self.requirements)])):
+            return False
+        return True

=== modified file 'snapcraft/plugins/python2_project.py'
--- snapcraft/plugins/python2_project.py	2015-09-11 19:53:38 +0000
+++ snapcraft/plugins/python2_project.py	2015-09-18 03:31:26 +0000
@@ -1,6 +1,6 @@
 # -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
 #
-# Copyright (C) 2015 Canonical Ltd
+# Copyright © 2015 Canonical Ltd
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License version 3 as
@@ -33,13 +33,10 @@
         # and be in the PYTHONPATH. It's harmless if setuptools isn't
         # used.
         os.makedirs(self.dist_packages_dir, exist_ok=True)
-        env = os.environ.copy()
-        env['PYTHONPATH'] = self.dist_packages_dir
 
         return self.run(
             ['python2', 'setup.py', 'install', '--install-layout=deb',
-             '--prefix={}/usr'.format(self.installdir)],
-            env=env)
+             '--prefix={}/usr'.format(self.installdir)])
 
     @property
     def dist_packages_dir(self):

=== modified file 'snapcraft/plugins/python3.py'
--- snapcraft/plugins/python3.py	2015-09-17 06:24:37 +0000
+++ snapcraft/plugins/python3.py	2015-09-18 03:31:26 +0000
@@ -14,6 +14,7 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+import os
 import snapcraft
 
 
@@ -24,6 +25,14 @@
         'python3-setuptools',
     ]
 
+    def __init__(self, name, options):
+        if options.requirements:
+            self.requirements = options.requirements
+            self._PLUGIN_STAGE_PACKAGES.extend(['python3-pkg-resources', 'python3-setuptools'])
+        else:
+            self.requirements = None
+        super().__init__(name, options)
+
     def snap_fileset(self):
         return [
             '-usr/share',
@@ -34,3 +43,19 @@
     # location, see https://code.launchpad.net/~mvo/snapcraft/python3-project/+merge/264521/comments/664308
     #
     # PATH is automatically set by snapcraft
+
+    def env(self, root):
+        return ["PYTHONPATH=%s" % os.path.join(
+            root, 'usr', 'lib', 'python3', 'dist-packages')]
+
+    def pull(self):
+        # A nice idea here would be to be asking setup tools
+        # to use the deb layout, but that doesn't work with
+        # prefix sadly
+
+        if self.requirements and not (self.run(
+                ['ln', '-s', os.path.join(self.installdir, 'usr', 'lib', 'python3', 'dist-packages'), os.path.join(self.installdir, 'usr', 'lib', 'python3.4', 'site-packages')]) and self.run(
+                ['python3', os.path.join(self.installdir, 'usr', 'bin', 'easy_install3'), '--prefix', os.path.join(self.installdir, 'usr'), 'pip']) and self.run(
+                ['python3', os.path.join(self.installdir, 'usr', 'bin', 'pip3'), 'install', '--target', os.path.join(self.installdir, 'usr', 'lib', 'python3.4', 'site-packages'), '--requirement', os.path.join(os.getcwd(), self.requirements)])):
+            return False
+        return True

=== modified file 'snapcraft/plugins/python3_project.py'
--- snapcraft/plugins/python3_project.py	2015-09-11 19:53:38 +0000
+++ snapcraft/plugins/python3_project.py	2015-09-18 03:31:26 +0000
@@ -24,6 +24,10 @@
     # note that we don't need to setup env(), python figures it out
     # see python3.py for more details
 
+    def env(self, root):
+        return ["PYTHONPATH=%s" % os.path.join(
+            root, 'usr', 'lib', 'python3', 'dist-packages')]
+
     def pull(self):
         return self.handle_source_options()
 
@@ -33,12 +37,9 @@
         # and be in the PYTHONPATH. It's harmless if setuptools isn't
         # used.
         os.makedirs(self.dist_packages_dir, exist_ok=True)
-        env = os.environ.copy()
-        env['PYTHONPATH'] = self.dist_packages_dir
         return self.run(
             ['python3', 'setup.py', 'install', '--install-layout=deb',
-             '--prefix=%s/usr' % self.installdir],
-            env=env)
+             '--prefix=%s/usr' % self.installdir])
 
     @property
     def dist_packages_dir(self):

=== modified file 'snapcraft/yaml.py'
--- snapcraft/yaml.py	2015-09-16 19:31:05 +0000
+++ snapcraft/yaml.py	2015-09-18 03:31:26 +0000
@@ -183,11 +183,13 @@
 
         for dep in part.deps:
             root = dep.installdir
-            env += self.runtime_env(root)
-            env += self.build_env(root)
             env += dep.env(root)
             env += self.build_env_for_part(dep)
 
+        env += part.env(part.installdir)
+        env += self.runtime_env(part.installdir)
+        env += self.build_env(part.installdir)
+
         return env
 
     def stage_env(self):