Merge lp:~dholbach/help-app/1433525 into lp:help-app

[Daniel Holbach]

Add markdown extension to make external links use target="_blank".

Add markdown extension to make external links use target="_blank".

[lp:~dholbach/help-app/1433525]

137. By Daniel Holbach on 2015-05-12

simplify python2/3 urlparse code

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

PASSED: Continuous integration, rev:137
http://91.189.93.70:8080/job/help-app-ci/26/
Executed test runs:
    SUCCESS: http://91.189.93.70:8080/job/help-app-utopic-amd64-ci/26
    SUCCESS: http://91.189.93.70:8080/job/help-app-vivid-amd64-ci/28

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/help-app-ci/26/rebuild

review: Approve (continuous-integration)

[lp:~dholbach/help-app/1433525]

138. By Daniel Holbach on 2015-05-13

simplify code in link checking somewhat, this will allow us to add more tests to check links, add more atomic tests test_simple_local_broken_link, test_simple_local_working_link, test_non_existing_build

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

FAILED: Continuous integration, rev:138
http://91.189.93.70:8080/job/help-app-ci/27/
Executed test runs:
    FAILURE: http://91.189.93.70:8080/job/help-app-utopic-amd64-ci/27/console
    FAILURE: http://91.189.93.70:8080/job/help-app-vivid-amd64-ci/29/console

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/help-app-ci/27/rebuild

review: Needs Fixing (continuous-integration)

[lp:~dholbach/help-app/1433525]

139. By Daniel Holbach on 2015-05-13

fix tests run during package build, by fixing override of build location (temporary dir to avoid changing live data)

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

PASSED: Continuous integration, rev:139
http://91.189.93.70:8080/job/help-app-ci/28/
Executed test runs:
    SUCCESS: http://91.189.93.70:8080/job/help-app-utopic-amd64-ci/28
    SUCCESS: http://91.189.93.70:8080/job/help-app-vivid-amd64-ci/30

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/help-app-ci/28/rebuild

review: Approve (continuous-integration)

[lp:~dholbach/help-app/1433525]

140. By Daniel Holbach on 2015-05-13

extend and generalise MyHTMLParser and HTMLLinkChecker so we can run other checks on links as well - added a test to see if the markdown extension for external links was run

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

PASSED: Continuous integration, rev:140
http://91.189.93.70:8080/job/help-app-ci/29/
Executed test runs:
    SUCCESS: http://91.189.93.70:8080/job/help-app-utopic-amd64-ci/29
    SUCCESS: http://91.189.93.70:8080/job/help-app-vivid-amd64-ci/31

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/help-app-ci/29/rebuild

review: Approve (continuous-integration)

[lp:~dholbach/help-app/1433525]

141. By Daniel Holbach on 2015-05-13

fix links in the CSS

[Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot)]

PASSED: Continuous integration, rev:141
http://91.189.93.70:8080/job/help-app-ci/30/
Executed test runs:
    SUCCESS: http://91.189.93.70:8080/job/help-app-utopic-amd64-ci/30
    SUCCESS: http://91.189.93.70:8080/job/help-app-vivid-amd64-ci/32

Click here to trigger a rebuild:
http://91.189.93.70:8080/job/help-app-ci/30/rebuild

review: Approve (continuous-integration)

[Nicholas Skaggs (nskaggs)]

LGTM. Couldn't check on device; will do before release.

review: Approve

=== modified file 'HACKING'
--- HACKING	2015-03-31 12:27:56 +0000
+++ HACKING	2015-05-13 13:46:52 +0000
@@ -86,7 +86,7 @@
 
   sudo apt install python-pelican po4a make bzrtools \
 	ubuntu-html5-ui-toolkit python3-polib python3-magic \
-	python3-markdown pep8 pyflakes
+	python3-markdown pep8 pyflakes python-magic python-polib
 
 This will install the necessary files, so you can build the app or web build
 locally and check if your changes actually make sense and look and work well.
 
=== modified file 'debian/control'
--- debian/control	2015-03-25 10:14:27 +0000
+++ debian/control	2015-05-13 13:46:52 +0000
@@ -8,6 +8,8 @@
                po4a,
                pyflakes,
                python-pelican (>= 3.5.0~),
+               python-magic,
+               python-polib,
                python3-magic,
                python3-markdown,
                python3-polib,
 
=== added file 'internals/local/external-links.py'
--- internals/local/external-links.py	1970-01-01 00:00:00 +0000
+++ internals/local/external-links.py	2015-05-13 13:46:52 +0000
@@ -0,0 +1,26 @@
+from __future__ import absolute_import
+from __future__ import unicode_literals
+from markdown import Extension
+from markdown.treeprocessors import Treeprocessor
+
+from translations.utils import link_is_local
+
+
+class ExternalLinksProcessor(Treeprocessor):
+    def run(self, doc):
+        for elem in doc.iter('a'):
+            if 'href' in elem.attrib and \
+               not link_is_local(elem.get('href')):
+                elem.set('target', '_blank')
+
+
+class ExternalLinksExtension(Extension):
+    def extendMarkdown(self, md, md_globals):
+        md.treeprocessors.add('external-links',
+                              ExternalLinksProcessor(md.parser),
+                              '_end')
+        md.registerExtension(self)
+
+
+def makeExtension(*args, **kwargs):
+    return ExternalLinksExtension(*args, **kwargs)
 
=== modified file 'internals/pelicanconf.py'
--- internals/pelicanconf.py	2015-03-31 08:18:07 +0000
+++ internals/pelicanconf.py	2015-05-13 13:46:52 +0000
@@ -54,7 +54,7 @@
 TAG_SAVE_AS = ''
 THEME = TOP_LEVEL_DIR+'static/themes/app/'
 
-MD_EXTENSIONS = ['local.q-and-a', 'attr_list', 'toc']
+MD_EXTENSIONS = ['local.q-and-a', 'local.external-links', 'attr_list', 'toc']
 
 META_TAGS = [
     '[TOC]',
 
=== modified file 'internals/tests/test_links.py'
--- internals/tests/test_links.py	2015-03-31 06:25:44 +0000
+++ internals/tests/test_links.py	2015-05-13 13:46:52 +0000
@@ -5,22 +5,12 @@
 import subprocess
 import tempfile
 from unittest import TestCase
-import urllib.parse
-
-from translations.utils import use_top_level_dir
-
-
-def require_build(build):
-    tempdir = tempfile.mkdtemp()
-    env = {}
-    if build == 'app':
-        env = {'OUTPUTDIR_APP': tempdir}
-    if build == 'web':
-        env = {'OUTPUTDIR_WEB': tempdir}
-    pwd = use_top_level_dir()
-    ret = subprocess.call(['make', '-es', build], env=env)
-    os.chdir(pwd)
-    return (ret, tempdir)
+
+from translations.utils import (
+    link_is_anchor,
+    link_is_local,
+    use_top_level_dir,
+)
 
 
 def clean_tempdir(tempdir):
@@ -28,53 +18,165 @@
         shutil.rmtree(tempdir)
 
 
+class BuildRunner():
+    def __init__(self, build):
+        self.tempdir = tempfile.mkdtemp()
+        self.env = {}
+        self.build = build
+        self.html_files = []
+        self.rc = None
+        if self.build == 'app':
+            self.env = {'APP_DIR': self.tempdir}
+        if self.build == 'web':
+            self.env = {'OUTPUTDIR_WEB': self.tempdir}
+        self.pwd = use_top_level_dir()
+        self.run_build()
+        self.find_html_files()
+
+    def run_build(self):
+        self.rc = subprocess.call(['make', '-es', self.build], env=self.env)
+        os.chdir(self.pwd)
+
+    def find_html_files(self):
+        for dirpath, dirnames, filenames in os.walk(self.tempdir):
+            self.html_files.extend([os.path.join(dirpath, fn)
+                                    for fn in filenames
+                                    if fn.endswith('.html')])
+
+    def __del__(self):
+        os.chdir(self.pwd)
+        clean_tempdir(self.tempdir)
+
+
 class MyHTMLParser(HTMLParser):
     links = []
 
     def handle_starttag(self, tag, attrs):
-        if tag == "a":
+        if tag == 'a':
+            found = {}
             for name, value in attrs:
-                if name == "href":
-                    url = urllib.parse.urlparse(value)
-                    if not value.startswith('#') and \
-                       url.netloc in [None, '', 'localhost']:
-                        self.links += [value]
+                if name == 'href' and \
+                   not link_is_anchor(value) and \
+                   value != '/':
+                    found['href'] = value
+                if name == 'target':
+                    found['target'] = value
+            if found != {}:
+                self.links += [found]
 
-    def reload(self):
-        links = self.links
+    def feed(self, data):
         self.links = []
-        return links
+        super(MyHTMLParser, self).feed(data)
+
+
+class HTMLLinkChecker():
+    def __init__(self, html_files):
+        self.local_link_results = []
+        self.htmlparser = MyHTMLParser()
+        self.links = {}
+        for fn in html_files:
+            html = codecs.open(fn, encoding='utf-8').read()
+            self.links[fn] = self._read_links(html)
+
+    def _read_links(self, html):
+        self.htmlparser.feed(html)
+        return self.htmlparser.links
+
+    def check_local_links(self):
+        self.local_link_results = []
+        pwd = os.getcwd()
+        for filename in self.links:
+            for link in [a['href'] for a
+                         in self.links[filename]
+                         if link_is_local(a['href'])]:
+                os.chdir(os.path.dirname(filename))
+                full_link_fn = os.path.join(os.path.dirname(filename), link)
+                rel_path = os.path.relpath(full_link_fn, filename)
+                path = os.path.normpath(os.path.join(filename, rel_path))
+                self.local_link_results += [os.path.exists(path)]
+        os.chdir(pwd)
+
+    def check_external_links(self):
+        self.external_link_results = []
+        for filename in self.links:
+            for link in [a for a
+                         in self.links[filename]
+                         if not link_is_local(a['href'])]:
+                self.external_link_results += [('target' in link and
+                                                link['target'] == '_blank')]
 
 
 class HelpTestCase(TestCase):
     def __init__(self, *args):
         self.pwd = os.getcwd()
-        self.htmlparser = MyHTMLParser()
+        self.build_runner = None
         TestCase.__init__(self, *args)
+        self.html_link_checker = None
 
     def __del__(self):
         os.chdir(self.pwd)
 
+    def _require_build(self, build):
+        if self.build_runner is None or \
+           self.build_runner.build != build:
+            self.build_runner = BuildRunner(build)
+
     def _test_local_links(self, build):
-        (ret, tempdir) = require_build(build)
-        self.assertEqual(ret, 0)
-        for dirpath, dirnames, filenames in os.walk(tempdir):
-            for fn in filenames:
-                full_fn = os.path.join(dirpath, fn)
-                os.chdir(dirpath)
-                if full_fn.endswith('.html'):
-                    html = codecs.open(full_fn, encoding='utf-8').read()
-                    self.htmlparser.feed(html)
-                links = self.htmlparser.reload()
-                for link in links:
-                    rel_path = os.path.relpath(link, full_fn)
-                    path = os.path.normpath(os.path.join(full_fn, rel_path))
-                    self.assertTrue(os.path.exists(path))
-        clean_tempdir(tempdir)
-        return True
-
-    def test_local_phone_links(self):
-        return self._test_local_links('html')
-
-    def test_local_web_links(self):
-        return self._test_local_links('web')
+        self._require_build(build)
+        self.html_link_checker = HTMLLinkChecker(self.build_runner.html_files)
+        self.html_link_checker.check_local_links()
+
+    def _test_external_links(self, build):
+        self._require_build(build)
+        self.html_link_checker = HTMLLinkChecker(self.build_runner.html_files)
+        self.html_link_checker.check_external_links()
+
+    def test_simple_local_broken_link(self):
+        dirname = tempfile.mkdtemp()
+        filename = os.path.join(dirname, 'first-test.html')
+        with open(filename, 'w') as f:
+            f.write('<html><body>' +
+                    '<a href="nonexisting-test.html">broken test link</a>' +
+                    '</body><html>')
+        self.html_link_checker = HTMLLinkChecker([filename])
+        self.html_link_checker.check_local_links()
+        self.assertNotIn(True, self.html_link_checker.local_link_results)
+        clean_tempdir(dirname)
+
+    def test_simple_local_working_link(self):
+        html = '<html><body>' + \
+               '<a href="second-test.html">broken test link</a>' + \
+               '</body><html>'
+        dirname = tempfile.mkdtemp()
+        filename1 = os.path.join(dirname, 'first-test.html')
+        with open(filename1, 'w') as f:
+            f.write(html)
+        filename2 = os.path.join(dirname, 'second-test.html')
+        with open(filename2, 'w') as f:
+            f.write(html)
+        self.html_link_checker = HTMLLinkChecker([filename1])
+        self.html_link_checker.check_local_links()
+        self.assertEqual([True], self.html_link_checker.local_link_results)
+        clean_tempdir(dirname)
+
+    def test_non_existing_build(self):
+        self._require_build('not-existing')
+        self.assertNotEqual(self.build_runner.rc, 0)
+
+    def test_local_links_in_phone_build(self):
+        self._test_local_links('app')
+        self.assertEqual(self.build_runner.rc, 0)
+        self.assertNotEqual(self.build_runner.html_files, [])
+        self.assertNotIn(False, self.html_link_checker.local_link_results)
+
+    def test_external_links_in_phone_build(self):
+        self._test_external_links('app')
+        self.assertEqual(self.build_runner.rc, 0)
+        self.assertNotEqual(self.build_runner.html_files, [])
+        self.assertNotIn(False, self.html_link_checker.external_link_results)
+
+    def test_local_links_in_web_build(self):
+        self._test_local_links('web')
+        self.assertEqual(self.build_runner.rc, 0)
+        self.assertNotEqual(self.build_runner.html_files, [])
+        self.assertNotIn(False, self.html_link_checker.local_link_results)
 
=== modified file 'internals/translations/build.py'
--- internals/translations/build.py	2015-03-31 11:38:30 +0000
+++ internals/translations/build.py	2015-05-13 13:46:52 +0000
@@ -20,7 +20,7 @@
 try:
     import polib
 except ImportError:
-    require('python3-polib')
+    require('python3-polib pyton-polib')
 
 from pelicanconf import (
     DOCS_PATH,
 
=== modified file 'internals/translations/utils.py'
--- internals/translations/utils.py	2015-03-31 07:57:56 +0000
+++ internals/translations/utils.py	2015-05-13 13:46:52 +0000
@@ -1,6 +1,10 @@
 import os
 import sys
 import tempfile
+if sys.version_info.major == 2:
+    import urlparse
+else:
+    import urllib.parse as urlparse
 
 
 def require(package):
@@ -11,7 +15,7 @@
 try:
     import magic
 except:
-    require('python3-magic')
+    require('python3-magic python-magic')
 
 try:
     import markdown
@@ -37,6 +41,15 @@
 ]
 
 
+def link_is_anchor(link):
+    return link.startswith('#')
+
+
+def link_is_local(link):
+    url = urlparse.urlparse(link)
+    return url.netloc in [None, '', 'localhost']
+
+
 def normalise_path(path):
     return os.path.relpath(path, TOP_LEVEL_DIR)
 
 
=== modified file 'static/themes/app/static/css/app.css'
--- static/themes/app/static/css/app.css	2015-03-18 12:06:41 +0000
+++ static/themes/app/static/css/app.css	2015-05-13 13:46:52 +0000
@@ -47,13 +47,6 @@
    display: none;
 }
 
-/* Disable links until they work
-   properly */
-a {
-  pointer-events: none;
-  cursor: default;
-}
-
 /* Disable this when the app container
    properly supports tab scrolling.
    We hardcode support for showing