Merge lp:~sergiusens/snapcraft/wiki into lp:~snappy-dev/snapcraft/core

[Sergio Schvezov]

Try to use wiki when after in part is not local

[Daniel Holbach (dholbach)]

A few small things:
 - examples/downloader-with-wiki-parts/icon.png is missing
 - "curl based downloaded" → "curl based downloader"?

I wasn't part of the design decisions regarding the wiki plugin, but I'm not quite sure how we could safe-guard against tampering of wiki pages. Do we want to look up who made the last edit of the page or something like that or introduce some kind of other security measure?

review: Needs Fixing

[Sergio Schvezov (sergiusens)]

> A few small things:
> - examples/downloader-with-wiki-parts/icon.png is missing

ah, always forget to bzr add :-) (I also added test_wiki.py)

> - "curl based downloaded" → "curl based downloader"?
>
> I wasn't part of the design decisions regarding the wiki plugin, but I'm not
> quite sure how we could safe-guard against tampering of wiki pages. Do we want
> to look up who made the last edit of the page or something like that or
> introduce some kind of other security measure?

The idea was to lock down the wiki endpoint to specific people only.

[Daniel Holbach (dholbach)]

Good work!

review: Approve

[Sergio Schvezov (sergiusens)]

Taking advantage of community reviewers due to the high load.

review: Approve

[Snappy Tarmac (snappydevtarmac)]

The attempt to merge lp:~sergiusens/snapcraft/wiki into lp:snapcraft failed. Below is the output from the failed tests.

The project has gotten complex.
Here's the list of units exceeding 10:
- snapcraft/cmds.py:
  246:1: 'cmd' 12

cp --preserve=all -R zzz /tmp/tmpsetsbpul/parts/copy/install/zzz
cp --preserve=all -R src /tmp/tmpq0hhcvas/parts/copy/install/dst
cp --preserve=all -R src /tmp/tmpuu5cjsbn/parts/copy/install/dir/dst

.........E...................Warning: unable to find "test_relexepath" in the path
..........................................EE
======================================================================
ERROR: snapcraft.tests.test_cmds (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.4/unittest/case.py", line 57, in testPartExecutor
    yield
  File "/usr/lib/python3.4/unittest/case.py", line 574, in run
    testMethod()
  File "/usr/lib/python3.4/unittest/loader.py", line 32, in testFailure
    raise exception
ImportError: Failed to import test module: snapcraft.tests.test_cmds
Traceback (most recent call last):
  File "/usr/lib/python3.4/unittest/loader.py", line 312, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.4/unittest/loader.py", line 290, in _get_module_from_name
    __import__(name)
  File "/tmp/tarmac/branch.9LlwxZ/snapcraft/tests/test_cmds.py", line 24, in <module>
    from snapcraft import (
  File "/tmp/tarmac/branch.9LlwxZ/snapcraft/cmds.py", line 27, in <module>
    import snapcraft.yaml
  File "/tmp/tarmac/branch.9LlwxZ/snapcraft/yaml.py", line 25, in <module>
    import snapcraft.wiki
  File "/tmp/tarmac/branch.9LlwxZ/snapcraft/wiki.py", line 18, in <module>
    import requests
ImportError: No module named 'requests'

======================================================================
ERROR: snapcraft.tests.test_wiki (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.4/unittest/case.py", line 57, in testPartExecutor
    yield
  File "/usr/lib/python3.4/unittest/case.py", line 574, in run
    testMethod()
  File "/usr/lib/python3.4/unittest/loader.py", line 32, in testFailure
    raise exception
ImportError: Failed to import test module: snapcraft.tests.test_wiki
Traceback (most recent call last):
  File "/usr/lib/python3.4/unittest/loader.py", line 312, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.4/unittest/loader.py", line 290, in _get_module_from_name
    __import__(name)
  File "/tmp/tarmac/branch.9LlwxZ/snapcraft/tests/test_wiki.py", line 19, in <module>
    import snapcraft.wiki
  File "/tmp/tarmac/branch.9LlwxZ/snapcraft/wiki.py", line 18, in <module>
    import requests
ImportError: No module named 'requests'

======================================================================
ERROR: snapcraft.tests.test_yaml (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/li...

[John Lenton (chipaca)]

review: Approve

[Sergio Schvezov (sergiusens)]

after review (and tarmac now has python3-requests)

review: Approve

[lp:~sergiusens/snapcraft/wiki]

167. By Sergio Schvezov on 2015-09-16

Try to use wiki when after in part is not local by sergiusens approved by sergiusens,chipaca,dholbach

=== added directory 'examples/downloader-with-wiki-parts'
=== added file 'examples/downloader-with-wiki-parts/Makefile'
--- examples/downloader-with-wiki-parts/Makefile	1970-01-01 00:00:00 +0000
+++ examples/downloader-with-wiki-parts/Makefile	2015-09-15 21:52:37 +0000
@@ -0,0 +1,11 @@
+# -*- Mode:Makefile; indent-tabs-mode:t; tab-width:4 -*-
+
+all:
+	gcc -Wall -Wextra -Werror -pedantic -o test ./test.c $(CFLAGS) $(LDFLAGS) -lcurl -lssl -lcrypto -lssl -lcrypto -lz
+
+install:
+	install -d -m755  $(DESTDIR)/bin/
+	install -m755 ./test $(DESTDIR)/bin/test
+
+clean:
+	rm -f test

=== added file 'examples/downloader-with-wiki-parts/icon.png'
Binary files examples/downloader-with-wiki-parts/icon.png	1970-01-01 00:00:00 +0000 and examples/downloader-with-wiki-parts/icon.png	2015-09-15 21:52:37 +0000 differ
=== added file 'examples/downloader-with-wiki-parts/snapcraft.yaml'
--- examples/downloader-with-wiki-parts/snapcraft.yaml	1970-01-01 00:00:00 +0000
+++ examples/downloader-with-wiki-parts/snapcraft.yaml	2015-09-15 21:52:37 +0000
@@ -0,0 +1,15 @@
+name: downloader
+version: 1.0
+vendor: "Sergio Schvezov <[e-mail]>"
+summary: curl based downloader
+description: this is an example package
+icon: icon.png
+binaries:
+    - name: bin/test
+
+parts:
+    main:
+        type: make-project
+        source: .
+        after:
+            - curl

=== added file 'examples/downloader-with-wiki-parts/test.c'
--- examples/downloader-with-wiki-parts/test.c	1970-01-01 00:00:00 +0000
+++ examples/downloader-with-wiki-parts/test.c	2015-09-15 21:52:37 +0000
@@ -0,0 +1,17 @@
+#include <stdio.h>
+#include <curl/curl.h>
+
+int main(void)
+{
+  CURL *curl;
+
+  curl = curl_easy_init();
+  if(curl) {
+    curl_easy_setopt(curl, CURLOPT_URL, "ubuntu.com");
+    curl_easy_perform(curl);
+
+    curl_easy_cleanup(curl);
+  }
+
+  return 0;
+}

=== modified file 'runtests.sh'
--- runtests.sh	2015-08-26 04:20:25 +0000
+++ runtests.sh	2015-09-15 21:52:37 +0000
@@ -31,7 +31,7 @@
 
 # mccabe in 'warning' mode as we have high complexity
 mccabe_list=
-for unit in $(find . -type f -name '*.py')
+for unit in $(find snapcraft -type f -name '*.py')
 do
   output=$(python3 -m mccabe --min 10 "$unit")
   [ -n "$output" ] && mccabe_list="- $unit:\n  $output\n$mccabe_list"

=== added file 'snapcraft/tests/test_wiki.py'
--- snapcraft/tests/test_wiki.py	1970-01-01 00:00:00 +0000
+++ snapcraft/tests/test_wiki.py	2015-09-15 21:52:37 +0000
@@ -0,0 +1,50 @@
+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
+#
+# Copyright (C) 2015 Canonical Ltd
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import unittest.mock
+
+import snapcraft.wiki
+
+from snapcraft.tests import TestCase
+
+
+class TestYaml(TestCase):
+
+    def setUp(self):
+        super().setUp()
+
+        class Content:
+
+            @property
+            def text(self):
+                return '''{{{part1:
+  type: go
+}}}'''
+
+        patcher = unittest.mock.patch('requests.get')
+        self.mock_requests = patcher.start()
+        self.mock_requests.return_value = Content()
+        self.addCleanup(patcher.stop)
+
+    def test_get_part(self):
+        w = snapcraft.wiki.Wiki()
+
+        self.mock_requests.assert_called_once_with(
+            'https://wiki.ubuntu.com/Snappy/Parts',
+            params={'action': 'raw'})
+
+        self.assertEqual(w.get_part('part1'), {'type': 'go'})
+        self.assertEqual(w.get_part('part2'), None)

=== modified file 'snapcraft/tests/test_yaml.py'
--- snapcraft/tests/test_yaml.py	2015-09-14 19:22:59 +0000
+++ snapcraft/tests/test_yaml.py	2015-09-15 21:52:37 +0000
@@ -39,6 +39,10 @@
         mock_wrap_exe.return_value = True
         self.addCleanup(patcher.stop)
 
+        patcher = unittest.mock.patch('snapcraft.wiki.Wiki')
+        self.mock_wiki = patcher.start()
+        self.addCleanup(patcher.stop)
+
     def make_snapcraft_yaml(self, content):
         tempdirObj = tempfile.TemporaryDirectory()
         self.addCleanup(tempdirObj.cleanup)
@@ -66,6 +70,32 @@
             'stage': [], 'snap': [],
         })
 
+        self.assertFalse(self.mock_wiki.get_part.called)
+
+    @unittest.mock.patch('snapcraft.yaml.Config.load_plugin')
+    def test_config_loads_plugins_with_wiki_part(self, mock_loadPlugin):
+        self.make_snapcraft_yaml("""name: test
+version: "1"
+vendor: me <[e-mail]>
+summary: test
+description: test
+icon: my-icon.png
+
+parts:
+  part1:
+    after:
+      - part2wiki
+    type: go
+    stage-packages: [fswebcam]
+""")
+        snapcraft.yaml.Config()
+        mock_loadPlugin.assert_called_with('part1', 'go', {
+            'stage-packages': ['fswebcam'],
+            'stage': [], 'snap': [],
+        })
+
+        self.assertFalse(self.mock_wiki.get_part.called)
+
     def test_config_raises_on_missing_snapcraft_yaml(self):
         fake_logger = fixtures.FakeLogger(level=logging.ERROR)
         self.useFixture(fake_logger)
@@ -99,6 +129,7 @@
             snapcraft.yaml.Config()
 
         self.assertEqual(raised.exception.message, 'circular dependency chain found in parts definition')
+        self.assertFalse(self.mock_wiki.get_part.called)
 
     @unittest.mock.patch('snapcraft.yaml.Config.load_plugin')
     def test_invalid_yaml_missing_name(self, mock_loadPlugin):

=== added file 'snapcraft/wiki.py'
--- snapcraft/wiki.py	1970-01-01 00:00:00 +0000
+++ snapcraft/wiki.py	2015-09-15 21:52:37 +0000
@@ -0,0 +1,50 @@
+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
+#
+# Copyright (C) 2015 Canonical Ltd
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import logging
+import requests
+import yaml
+
+PARTS_URI = 'https://wiki.ubuntu.com/Snappy/Parts'
+PARTS_URI_PARAMS = {'action': 'raw'}
+
+_WIKI_OPEN = '{{{'
+_WIKI_CLOSE = '}}}'
+
+logging.getLogger("urllib3").setLevel(logging.CRITICAL)
+
+
+class Wiki:
+
+    def __init__(self):
+        self.wiki_parts = self._fetch()
+
+    def _fetch(self):
+        raw_content = requests.get(PARTS_URI, params=PARTS_URI_PARAMS)
+        content = raw_content.text.strip()
+
+        if content.startswith(_WIKI_OPEN):
+            content = content[len(_WIKI_OPEN):].strip()
+        if content.endswith(_WIKI_CLOSE):
+            content = content[:-len(_WIKI_CLOSE)]
+
+        return yaml.load(content)
+
+    def get_part(self, name):
+        try:
+            return self.wiki_parts[name]
+        except KeyError:
+            return None

=== modified file 'snapcraft/yaml.py'
--- snapcraft/yaml.py	2015-09-08 18:06:03 +0000
+++ snapcraft/yaml.py	2015-09-15 21:52:37 +0000
@@ -22,6 +22,7 @@
 import os.path
 
 import snapcraft.plugin
+import snapcraft.wiki
 from snapcraft import common
 
 
@@ -114,6 +115,7 @@
 
     def _compute_part_dependencies(self, after_requests):
         '''Gather the lists of dependencies and adds to all_parts.'''
+        w = snapcraft.wiki.Wiki()
 
         for part in self.all_parts:
             dep_names = part.config.get('requires', []) + after_requests.get(part.names()[0], [])
@@ -125,6 +127,11 @@
                         found = True
                         break
                 if not found:
+                    wiki_part = w.get_part(dep)
+                    found = True if wiki_part else False
+                    if found:
+                        part.deps.append(self.load_plugin(dep, wiki_part['type'], wiki_part))
+                if not found:
                     raise SnapcraftLogicError('part name missing {}'.format(dep))
 
     def _sort_parts(self):