Merge lp:~beuno/click-reviewers-tools/deprecate-14-10-dev1 into lp:click-reviewers-tools

[Martin Albisetti]

This branch mostly adds 14-10-dev2 and deprecates 14-10-dev1.
Because it bothered me so much, I refactored the way we handle frameworks into a central static list which should be easy to update.

[lp:~beuno/click-reviewers-tools/deprecate-14-10-dev1]

203. By Daniel Holbach on 2014-06-30

add script to update frameworks definition from branch, FRAMEWORKS_BRANCH_URL will have to be updated to be something with ~ubuntu-core-dev

204. By Daniel Holbach on 2014-06-30

ran ./debian/scripts/update-frameworks

205. By Daniel Holbach on 2014-06-30

make use of new data file

206. By Daniel Holbach on 2014-06-30

move update-frameworks to bin, make it not depend on bzr

207. By Daniel Holbach on 2014-06-30

move functionality into frameworks.py

208. By Daniel Holbach on 2014-07-01

 - deal with urllib issues (no access to the net, etc.)
 - save frameworks definition in ~/.cache
 - update cache every week
 - add error messages

209. By Daniel Holbach on 2014-07-01

refine update logic, add local_copy parameter, which can be used for builds and tests

210. By Daniel Holbach on 2014-07-01

add missing import

211. By Daniel Holbach on 2014-07-01

rename variables to make more sense, simplify read_frameworks_file logic somewhat, introduce a Frameworks class

212. By Daniel Holbach on 2014-07-01

pass filename to get_frameworks_file

213. By Daniel Holbach on 2014-07-01

call update-frameworks in clean target

214. By Daniel Holbach on 2014-07-01

use Frameworks class

215. By Daniel Holbach on 2014-07-01

remove debian/scripts

216. By Daniel Holbach on 2014-07-01

change URL and explain that this bazaar.lp.net hack will be gone soon

217. By Daniel Holbach on 2014-07-02

move makedirs call to the right place

[Ricardo Kirkner (ricardokirkner)]

Looks good overall, but needs a few small tweaks

[lp:~beuno/click-reviewers-tools/deprecate-14-10-dev1]

218. By Martin Albisetti on 2014-07-02

Fixes from Daniel

[Ricardo Kirkner (ricardokirkner)]

<dholbach> pindonga, about the two other things I guess it'd help to file general bugs, so we can fix it everywhere else as well - the unreadable variable names were probably used, since the call is basically used in every check
<pindonga> dholbach, so you're saying you're not fixing these 2 issues for consistency with the rest of the code?
<pindonga> dholbach, just so I understand the reason
<dholbach> beuno, ^ do you have an opinion on this?
<beuno> dholbach, pindonga, I'm happy to favor consistency, yes

review: Approve

[lp:~beuno/click-reviewers-tools/deprecate-14-10-dev1]

219. By Martin Albisetti on 2014-07-02

More fixes from Daniel

[Jamie Strandboge (jdstrand)]

I like the approach, though I have a couple of comments:
 * ./run-pep8 shows some problems-- these should be fixed
 * it was a little odd at first seeing all the stuff in frameworks.py about USER_DATA_FILE and cr_lint.py not using it, but then I realized that the scripts are always going to use data/frameworks.json and the rest of frameworks.py is just about implementing what is needed for bin/update-frameworks to work
 * there are no tests for frameworks.py-- not sure how feasible these are
 * it would be nice if the change to test_check_framework() didn't have to keep happening-- ie, if we could just take anything from data/frameworks.json and put it in there. That is an improvement and doesn't have to be fixed for the merge

Please fix the pep8 and if you are feeling industrious, modify test_check_framework() as mentioned. After that, feel free to commit.

review: Approve

[Jamie Strandboge (jdstrand)]

review: Needs Fixing

[Jamie Strandboge (jdstrand)]

I fixed the pep8 issues and committed. Thanks!

=== added file 'bin/update-frameworks'
--- bin/update-frameworks	1970-01-01 00:00:00 +0000
+++ bin/update-frameworks	2014-07-02 16:33:58 +0000
@@ -0,0 +1,20 @@
+#!/usr/bin/python3
+
+from clickreviews import frameworks
+import sys
+
+def main(filename=None):
+    frameworks.get_frameworks_file(filename)
+
+if __name__ == '__main__':
+    try:
+        filename = None
+        if len(sys.argv) > 2:
+            print("Usage: %s [file]" % sys.argv[0])
+            sys.exit(1)
+        elif len(sys.argv) == 2:
+            filename = sys.argv[1]
+        main(filename)
+    except KeyboardInterrupt:
+        print('Aborted.', file=sys.stderr)
+        sys.exit(1)

=== modified file 'clickreviews/cr_lint.py'
--- clickreviews/cr_lint.py	2014-05-30 08:33:09 +0000
+++ clickreviews/cr_lint.py	2014-07-02 16:33:58 +0000
@@ -21,6 +21,7 @@
 import os
 import re
 
+from clickreviews.frameworks import Frameworks
 from clickreviews.cr_common import ClickReview, open_file_read, cmd
 
 CONTROL_FILE_NAMES = ["control", "manifest", "md5sums", "preinst"]
@@ -529,26 +530,37 @@
 
     def check_framework(self):
         '''Check framework()'''
-        t = 'info'
         n = 'framework'
-        s = 'OK'
-        if self.manifest['framework'] not in self.valid_frameworks:
+        l = "http://askubuntu.com/questions/460512/what-framework-should-i-use-in-my-manifest-file"
+        local_copy = os.path.join(os.path.dirname(__file__), 
+                                  '../data/frameworks.json')
+        frameworks = Frameworks(local_copy)
+        if self.manifest['framework'] in frameworks.AVAILABLE_FRAMEWORKS:
+            t = 'info'
+            s = 'OK'
+            self._add_result(t, n, s)
+            # If it's an available framework, we're done checking
+            return
+        elif self.manifest['framework'] in frameworks.DEPRECATED_FRAMEWORKS:
+            t = 'warn'
+            s = "'%s' is deprecated. Please use a newer framework" % \
+                self.manifest['framework']
+            self._add_result(t, n, s, l)
+            return
+        elif self.manifest['framework'] in frameworks.OBSOLETE_FRAMEWORKS:
+            t = 'error'
+            s = "'%s' is obsolete. Please use a newer framework" % \
+                self.manifest['framework']
+            self._add_result(t, n, s, l)
+            return
+        else:
+            # None of the above checks triggered, this is an unknown framework
             t = 'error'
             s = "'%s' is not a supported framework" % \
                 self.manifest['framework']
-        self._add_result(t, n, s)
-
-        deprecated_frameworks = ['ubuntu-sdk-13.10']
-        t = 'info'
-        n = 'current framework'
-        s = 'OK'
-        l = None
-        if self.manifest['framework'] in deprecated_frameworks:
-            t = 'warn'
-            s = "'%s' is deprecated. Please use a newer framework" % \
-                self.manifest['framework']
-            l = "http://askubuntu.com/questions/460512/what-framework-should-i-use-in-my-manifest-file"
-        self._add_result(t, n, s, l)
+            self._add_result(t, n, s, l)
+
+
 
     def check_click_local_extensions(self):
         '''Report any click local extensions'''

=== added file 'clickreviews/frameworks.py'
--- clickreviews/frameworks.py	1970-01-01 00:00:00 +0000
+++ clickreviews/frameworks.py	2014-07-02 16:33:58 +0000
@@ -0,0 +1,93 @@
+"""
+This file defines all known frameworks and their current status.
+Frameworks are currenly tracked in: http://goo.gl/z9ohJ3
+"""
+
+import json
+import os
+import re
+from socket import timeout
+import sys
+import time
+from urllib import request, parse
+from urllib.error import HTTPError, URLError
+
+DATA_DIR = os.path.join(os.path.expanduser('~/.cache/ubuntu-frameworks/'))
+USER_DATA_FILE = os.path.join(DATA_DIR, 'frameworks.json')
+
+# XXX: This is a hack and will be gone, as soon as myapps has an API for this.
+FRAMEWORKS_DATA_URL = \
+        "http://bazaar.launchpad.net/~ubuntu-core-dev/+junk/frameworks/view/head:/frameworks.json"
+
+UPDATE_INTERVAL = 60*60*24*7
+
+def update_is_necessary():
+    return (not os.path.exists(USER_DATA_FILE)) or \
+            (time.time()-os.path.getctime(USER_DATA_FILE) >= UPDATE_INTERVAL)
+
+def update_is_possible():
+    update = True
+    try:
+        request.urlopen(FRAMEWORKS_DATA_URL)
+    except (HTTPError, URLError):
+        update = False
+    except timeout:
+        update = False
+    return update
+
+def abort(msg=None):
+    if msg:
+        print(msg, file=sys.stderr)
+    print('Aborted.', file=sys.stderr)
+    sys.exit(1)
+
+def get_frameworks_file(data_dir=DATA_DIR):
+    try:
+        f = request.urlopen(FRAMEWORKS_DATA_URL)
+    except (HTTPError, URLError) as error:
+        abort('Data not retrieved because %s.' % error)
+    except timeout:
+        abort('Socket timed out.')
+    html = f.read()
+    # XXX: This is a hack and will be gone, as soon as myapps has an API for this.
+    link = re.findall(b'<a href="(\S+?)">download file</a>', html)
+    if not link:
+        abort()
+    download_link = '{}://{}/{}'.format(\
+            parse.urlparse(FRAMEWORKS_DATA_URL).scheme,
+            parse.urlparse(FRAMEWORKS_DATA_URL).netloc,
+            link[0].decode("utf-8"))
+    f = request.urlopen(download_link)
+    if not f:
+        abort()
+    if os.path.exists(USER_DATA_FILE):
+        os.remove(USER_DATA_FILE)
+    if not os.path.exists(DATA_DIR):
+        os.makedirs(DATA_DIR)
+    with open(USER_DATA_FILE, 'bw') as local_file:
+        local_file.write(f.read())
+
+def read_frameworks_file(local_copy=None):
+    if update_is_necessary() and update_is_possible():
+        get_frameworks_file()
+    if not os.path.exists(USER_DATA_FILE):
+        if local_copy:
+            return json.loads(open(local_copy, 'r').read())
+        return {}
+    return json.loads(open(USER_DATA_FILE, 'r').read())
+
+class Frameworks(object):
+    DEPRECATED_FRAMEWORKS = []
+    OBSOLETE_FRAMEWORKS = []
+    AVAILABLE_FRAMEWORKS = []
+
+    def __init__(self, local_copy=None):
+        self.FRAMEWORKS = read_frameworks_file(local_copy)
+
+        for k, v in self.FRAMEWORKS.items():
+            if v == 'deprecated':
+                self.DEPRECATED_FRAMEWORKS.append(k)
+            elif v == 'obsolete':
+                self.OBSOLETE_FRAMEWORKS.append(k)
+            elif v == 'available':
+                self.AVAILABLE_FRAMEWORKS.append(k)

=== modified file 'clickreviews/tests/test_cr_lint.py'
--- clickreviews/tests/test_cr_lint.py	2014-05-30 08:33:09 +0000
+++ clickreviews/tests/test_cr_lint.py	2014-07-02 16:33:58 +0000
@@ -601,11 +601,11 @@
 
     def test_check_framework(self):
         '''Test check_framework()'''
-        self.set_test_manifest("framework", "ubuntu-sdk-14.04-qml-dev1")
+        self.set_test_manifest("framework", "ubuntu-sdk-14.10-qml-dev2")
         c = ClickReviewLint(self.test_name)
         c.check_framework()
         r = c.click_report
-        expected_counts = {'info': 2, 'warn': 0, 'error': 0}
+        expected_counts = {'info': 1, 'warn': 0, 'error': 0}
         self.check_results(r, expected_counts)
 
     def test_check_framework_bad(self):
@@ -626,6 +626,15 @@
         expected_counts = {'info': None, 'warn': 1, 'error': 0}
         self.check_results(r, expected_counts)
 
+    def test_check_framework_obsolete(self):
+        '''Test check_framework() - obsolete'''
+        self.set_test_manifest("framework", "ubuntu-sdk-14.10-qml-dev1")
+        c = ClickReviewLint(self.test_name)
+        c.check_framework()
+        r = c.click_report
+        expected_counts = {'info': None, 'warn': 0, 'error': 1}
+        self.check_results(r, expected_counts)
+
     def test_check_hooks(self):
         '''Test check_hooks()'''
         self.set_test_manifest("framework", "ubuntu-sdk-13.10")

=== added directory 'data'
=== added file 'data/frameworks.json'
--- data/frameworks.json	1970-01-01 00:00:00 +0000
+++ data/frameworks.json	2014-07-02 16:33:58 +0000
@@ -0,0 +1,19 @@
+{
+ "ubuntu-sdk-13.10": "deprecated",
+ "ubuntu-sdk-14.04-dev1": "deprecated",
+ "ubuntu-sdk-14.04-html-dev1": "deprecated",
+ "ubuntu-sdk-14.04-papi-dev1": "deprecated",
+ "ubuntu-sdk-14.04-qml-dev1": "deprecated",
+ "ubuntu-sdk-14.04": "available",
+ "ubuntu-sdk-14.04-html": "available",
+ "ubuntu-sdk-14.04-papi": "available",
+ "ubuntu-sdk-14.04-qml": "available",
+ "ubuntu-sdk-14.10-dev1": "obsolete",
+ "ubuntu-sdk-14.10-html-dev1": "obsolete",
+ "ubuntu-sdk-14.10-papi-dev1": "obsolete",
+ "ubuntu-sdk-14.10-qml-dev1": "obsolete",
+ "ubuntu-sdk-14.10-dev2": "available",
+ "ubuntu-sdk-14.10-html-dev2": "available",
+ "ubuntu-sdk-14.10-papi-dev2": "available",
+ "ubuntu-sdk-14.10-qml-dev2": "available"
+}

=== modified file 'debian/rules'
--- debian/rules	2013-09-26 10:42:09 +0000
+++ debian/rules	2014-07-02 16:33:58 +0000
@@ -18,7 +18,7 @@
 	rm -rf build *.egg-info .pybuild
 	find -name \*.pyc -print0 | xargs -0r rm -f
 	find -name __pycache__ -print0 | xargs -0r rm -rf
-
+	-$(shell python3 ./bin/update-frameworks ./data)
 
 override_dh_auto_build:
 	dh_auto_build