Ubuntu Developer Portal

Merge lp:~dholbach/developer-ubuntu-com/importer-post-deployment-fixes into lp:developer-ubuntu-com

  1. importer-post-deployment-fixes
  2. Merge into stable
Proposed by Daniel Holbach
Status: Merged
Approved by: David Callé
Approved revision: 212
Merged at revision: 194
Proposed branch: lp:~dholbach/developer-ubuntu-com/importer-post-deployment-fixes
Merge into: lp:developer-ubuntu-com
Diff against target: 480 lines (+189/-72)
7 files modified
md_importer/importer/__init__.py (+12/-0)
md_importer/importer/article.py (+37/-22)
md_importer/importer/publish.py (+54/-22)
md_importer/importer/repo.py (+0/-1)
md_importer/tests/test_branch_import.py (+40/-6)
md_importer/tests/test_link_rewrite.py (+43/-20)
md_importer/tests/utils.py (+3/-1)
To merge this branch: bzr merge lp:~dholbach/developer-ubuntu-com/importer-post-deployment-fixes
Reviewer Review Type Date Requested Status
Ubuntu App Developer site developers Pending
Review via email: mp+284309@code.launchpad.net

Description of the change

This is ready to land now.

List of fixes:
 - only use a shortlist of markdown extensions
 - fix the rewriting of links in articles (links between articles in the
   same branch), fix and extend tests
 - simplify code somewhat, remove useless bits
 - fix stripping of tags like <body>, add tests

To post a comment you must log in.
lp:~dholbach/developer-ubuntu-com/importer-post-deployment-fixes updated
192. By Daniel Holbach

only set page to public_object if it exists

193. By Daniel Holbach

remove unnecessary reset of repo.pages

194. By Daniel Holbach

merge from trunk

195. By Daniel Holbach

- make TestLinkRewrite link check explicit
- fix condition in TestLinkBrokenRewrite to not leave for loop early

196. By Daniel Holbach

bring TestSnapcraftLinkRewrite test closer to reality and just import what's important to us, fix condition to not leave for loop early

197. By Daniel Holbach

remove unnecessary line

198. By Daniel Holbach

avoid 'break' in the for loop

199. By Daniel Holbach

break out update_page functionality into separate function

200. By Daniel Holbach

store list of local images and if links were rewritten in the article object, use the new update_page function

201. By Daniel Holbach

add TODO item, make pyflakes and pep8 happy

202. By Daniel Holbach

remove body/html tags after soup.prettify

203. By Daniel Holbach

add test to ensure we strip all <body> tags from the imported articles

204. By Daniel Holbach

make sure internal links start with '/'

205. By Daniel Holbach

fix tests wrt fixed links

206. By Daniel Holbach

remove stray print

207. By Daniel Holbach

make regexes for stripping body/html/head tags clearer

208. By Daniel Holbach

drop pymdownx.headeranchor - it creates problems (order of link attributes gets mixed up depending on which html output we use), and we don't need it on the page

209. By Daniel Holbach

- when comparing HTML, always use clean_html from djangocms_text_ckeditor
  and soup.prettify, so we're looking at the same output style
- add convenience function find_text_plugin
- check not only if the draft's html has changed, but also if the published
  version changed
- update test as well, check only published pages

210. By Daniel Holbach

make sure we don't have 'None' as slug for the root node, add tests (one for the URL, one for links in the HTML)

211. By Daniel Holbach

cater to the use-case where we just import snappy docs, but have no release_alias (ie current) set

212. By Daniel Holbach

fix typo

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'md_importer/importer/__init__.py'
--- md_importer/importer/__init__.py 2016-01-12 11:44:04 +0000
+++ md_importer/importer/__init__.py 2016-03-02 11:13:56 +0000
@@ -3,3 +3,15 @@
3DEFAULT_LANG = LANGUAGE_CODE3DEFAULT_LANG = LANGUAGE_CODE
4HOME_PAGE_URL = '/{}/'.format(DEFAULT_LANG)4HOME_PAGE_URL = '/{}/'.format(DEFAULT_LANG)
5SUPPORTED_ARTICLE_TYPES = ['.md', '.html']5SUPPORTED_ARTICLE_TYPES = ['.md', '.html']
6
7# Instead of just using pymdownx.github, we go with these because of
8# https://github.com/facelessuser/pymdown-extensions/issues/11
9MARKDOWN_EXTENSIONS = [
10 'markdown.extensions.tables',
11 'pymdownx.magiclink',
12 'pymdownx.betterem',
13 'pymdownx.tilde',
14 'pymdownx.githubemoji',
15 'pymdownx.tasklist',
16 'pymdownx.superfences',
17]
618
=== modified file 'md_importer/importer/article.py'
--- md_importer/importer/article.py 2016-01-15 13:56:34 +0000
+++ md_importer/importer/article.py 2016-03-02 11:13:56 +0000
@@ -8,9 +8,10 @@
88
9from . import (9from . import (
10 DEFAULT_LANG,10 DEFAULT_LANG,
11 MARKDOWN_EXTENSIONS,
11 SUPPORTED_ARTICLE_TYPES,12 SUPPORTED_ARTICLE_TYPES,
12)13)
13from .publish import get_or_create_page, slugify14from .publish import get_or_create_page, slugify, update_page
1415
15if sys.version_info.major == 2:16if sys.version_info.major == 2:
16 from urlparse import urlparse17 from urlparse import urlparse
@@ -27,18 +28,18 @@
27 self.write_to = slugify(self.fn)28 self.write_to = slugify(self.fn)
28 self.full_url = write_to29 self.full_url = write_to
29 self.slug = os.path.basename(self.full_url)30 self.slug = os.path.basename(self.full_url)
31 self.links_rewritten = False
32 self.local_images = []
3033
31 def _find_local_images(self):34 def _find_local_images(self):
32 '''Local images are currently not supported.'''35 '''Local images are currently not supported.'''
33 soup = BeautifulSoup(self.html, 'html5lib')36 soup = BeautifulSoup(self.html, 'html5lib')
34 local_images = []
35 for img in soup.find_all('img'):37 for img in soup.find_all('img'):
36 if img.has_attr('src'):38 if img.has_attr('src'):
37 (scheme, netloc, path, params, query, fragment) = \39 (scheme, netloc, path, params, query, fragment) = \
38 urlparse(img.attrs['src'])40 urlparse(img.attrs['src'])
39 if scheme not in ['http', 'https']:41 if scheme not in ['http', 'https']:
40 local_images.extend([img.attrs['src']])42 self.local_images.extend([img.attrs['src']])
41 return local_images
4243
43 def read(self):44 def read(self):
44 if os.path.splitext(self.fn)[1] not in SUPPORTED_ARTICLE_TYPES:45 if os.path.splitext(self.fn)[1] not in SUPPORTED_ARTICLE_TYPES:
@@ -50,13 +51,13 @@
50 self.html = markdown.markdown(51 self.html = markdown.markdown(
51 f.read(),52 f.read(),
52 output_format='html5',53 output_format='html5',
53 extensions=['pymdownx.github'])54 extensions=MARKDOWN_EXTENSIONS)
54 elif self.fn.endswith('.html'):55 elif self.fn.endswith('.html'):
55 self.html = f.read()56 self.html = f.read()
56 local_images = self._find_local_images()57 self._find_local_images()
57 if local_images:58 if self.local_images:
58 logging.error('Found the following local image(s): {}'.format(59 logging.error('Found the following local image(s): {}'.format(
59 ', '.join(local_images)60 ', '.join(self.local_images)
60 ))61 ))
61 return False62 return False
62 self.title = self._read_title()63 self.title = self._read_title()
@@ -73,10 +74,15 @@
73 return slugify(self.fn).replace('-', ' ').title()74 return slugify(self.fn).replace('-', ' ').title()
7475
75 def _remove_body_and_html_tags(self):76 def _remove_body_and_html_tags(self):
76 self.html = re.sub(r"<html>\n\s<body>\n", "", self.html,77 for regex in [
77 flags=re.MULTILINE)78 # These are added by markdown.markdown
78 self.html = re.sub(r"\s<\/body>\n<\/html>", "", self.html,79 r'\s*<html>\s*<body>\s*',
79 flags=re.MULTILINE)80 r'\s*<\/body>\s*<\/html>\s*',
81 # This is added by BeautifulSoup.prettify
82 r'\s*<html>\s*<head>\s*<\/head>\s*<body>\s*',
83 ]:
84 self.html = re.sub(regex, '', self.html,
85 flags=re.MULTILINE)
8086
81 def _use_developer_site_style(self):87 def _use_developer_site_style(self):
82 begin = (u"<div class=\"row no-border\">"88 begin = (u"<div class=\"row no-border\">"
@@ -92,7 +98,6 @@
9298
93 def replace_links(self, titles, url_map):99 def replace_links(self, titles, url_map):
94 soup = BeautifulSoup(self.html, 'html5lib')100 soup = BeautifulSoup(self.html, 'html5lib')
95 change = False
96 for link in soup.find_all('a'):101 for link in soup.find_all('a'):
97 if not link.has_attr('class') or \102 if not link.has_attr('class') or \
98 'headeranchor-link' not in link.attrs['class']:103 'headeranchor-link' not in link.attrs['class']:
@@ -100,10 +105,12 @@
100 if title.endswith(link.attrs['href']) and \105 if title.endswith(link.attrs['href']) and \
101 link.attrs['href'] != url_map[title].full_url:106 link.attrs['href'] != url_map[title].full_url:
102 link.attrs['href'] = url_map[title].full_url107 link.attrs['href'] = url_map[title].full_url
103 change = True108 if not link.attrs['href'].startswith('/'):
104 if change:109 link.attrs['href'] = '/' + link.attrs['href']
110 self.links_rewritten = True
111 if self.links_rewritten:
105 self.html = soup.prettify()112 self.html = soup.prettify()
106 return change113 self._remove_body_and_html_tags()
107114
108 def add_to_db(self):115 def add_to_db(self):
109 '''Publishes pages in their branch alias namespace.'''116 '''Publishes pages in their branch alias namespace.'''
@@ -112,13 +119,19 @@
112 html=self.html)119 html=self.html)
113 if not self.page:120 if not self.page:
114 return False121 return False
115 self.full_url = self.page.get_absolute_url()122 self.full_url = re.sub(
123 r'^\/None\/', '/{}/'.format(DEFAULT_LANG),
124 self.page.get_absolute_url())
116 return True125 return True
117126
118 def publish(self):127 def publish(self):
128 if self.links_rewritten:
129 update_page(self.page, title=self.title, full_url=self.full_url,
130 menu_title=self.title, html=self.html)
119 if self.page.is_dirty(DEFAULT_LANG):131 if self.page.is_dirty(DEFAULT_LANG):
120 self.page.publish(DEFAULT_LANG)132 self.page.publish(DEFAULT_LANG)
121 self.page = self.page.get_public_object()133 if self.page.get_public_object():
134 self.page = self.page.get_public_object()
122 return self.page135 return self.page
123136
124137
@@ -128,14 +141,16 @@
128 def read(self):141 def read(self):
129 if not Article.read(self):142 if not Article.read(self):
130 return False143 return False
131 self.release_alias = re.findall(r'snappy/guides/(\S+?)/\S+?',144 matches = re.findall(r'snappy/guides/(\S+?)/\S+?',
132 self.full_url)[0]145 self.full_url)
146 if matches:
147 self.release_alias = matches[0]
133 self._make_snappy_mods()148 self._make_snappy_mods()
134 return True149 return True
135150
136 def _make_snappy_mods(self):151 def _make_snappy_mods(self):
137 # Make sure the reader knows which documentation she is browsing152 # Make sure the reader knows which documentation she is browsing
138 if self.release_alias != 'current':153 if self.release_alias and self.release_alias != 'current':
139 before = (u"<div class=\"row no-border\">\n"154 before = (u"<div class=\"row no-border\">\n"
140 "<div class=\"eight-col\">\n")155 "<div class=\"eight-col\">\n")
141 after = (u"<div class=\"row no-border\">\n"156 after = (u"<div class=\"row no-border\">\n"
@@ -158,6 +173,6 @@
158 redirect="/snappy/guides/current/{}".format(self.slug))173 redirect="/snappy/guides/current/{}".format(self.slug))
159 if not page:174 if not page:
160 return False175 return False
161 else:176 elif self.release_alias:
162 self.title += " (%s)" % (self.release_alias,)177 self.title += " (%s)" % (self.release_alias,)
163 return Article.add_to_db(self)178 return Article.add_to_db(self)
164179
=== modified file 'md_importer/importer/publish.py'
--- md_importer/importer/publish.py 2016-01-15 13:58:39 +0000
+++ md_importer/importer/publish.py 2016-03-02 11:13:56 +0000
@@ -4,11 +4,18 @@
4from cms.models import Title4from cms.models import Title
5from djangocms_text_ckeditor.html import clean_html5from djangocms_text_ckeditor.html import clean_html
66
7from bs4 import BeautifulSoup
7import logging8import logging
8import re9import re
9import os10import os
1011
1112
13def _compare_html(html_a, html_b):
14 soup_a = BeautifulSoup(html_a, 'html5lib')
15 soup_b = BeautifulSoup(html_b, 'html5lib')
16 return (clean_html(soup_a.prettify()) == clean_html(soup_b.prettify()))
17
18
12def slugify(filename):19def slugify(filename):
13 return os.path.basename(filename).replace('.md', '').replace('.html', '')20 return os.path.basename(filename).replace('.md', '').replace('.html', '')
1421
@@ -32,6 +39,51 @@
32 return parent_pages[0].page39 return parent_pages[0].page
3340
3441
42def find_text_plugin(page):
43 # We create the page, so we know there's just one placeholder
44 placeholder = page.placeholders.all()[0]
45 if placeholder.get_plugins():
46 return (
47 placeholder,
48 placeholder.get_plugins()[0].get_plugin_instance()[0]
49 )
50 return (placeholder, None)
51
52
53def update_page(page, title, full_url, menu_title=None,
54 in_navigation=True, redirect=None, html=None):
55 if page.get_title() != title:
56 page.title = title
57 if page.get_menu_title() != menu_title:
58 page.menu_title = menu_title
59 if page.in_navigation != in_navigation:
60 page.in_navigation = in_navigation
61 if page.get_redirect() != redirect:
62 page.redirect = redirect
63 if html:
64 update = True
65 (placeholder, plugin) = find_text_plugin(page)
66 if plugin:
67 if _compare_html(html, plugin.body):
68 update = False
69 elif page.get_public_object():
70 (dummy, published_plugin) = \
71 find_text_plugin(page.get_public_object())
72 if published_plugin:
73 if _compare_html(html, published_plugin.body):
74 update = False
75 if update:
76 plugin.body = html
77 plugin.save()
78 else:
79 # Reset draft
80 page.get_draft_object().revert(DEFAULT_LANG)
81 else:
82 add_plugin(
83 placeholder, 'RawHtmlPlugin',
84 DEFAULT_LANG, body=html)
85
86
35def get_or_create_page(title, full_url, menu_title=None,87def get_or_create_page(title, full_url, menu_title=None,
36 in_navigation=True, redirect=None, html=None):88 in_navigation=True, redirect=None, html=None):
37 # First check if pages already exist.89 # First check if pages already exist.
@@ -39,26 +91,8 @@
39 path__regex=full_url).filter(publisher_is_draft=True)91 path__regex=full_url).filter(publisher_is_draft=True)
40 if pages:92 if pages:
41 page = pages[0].page93 page = pages[0].page
42 if page.get_title() != title:94 update_page(page, title, full_url, menu_title, in_navigation,
43 page.title = title95 redirect, html)
44 if page.get_menu_title() != menu_title:
45 page.menu_title = menu_title
46 if page.in_navigation != in_navigation:
47 page.in_navigation = in_navigation
48 if page.get_redirect() != redirect:
49 page.redirect = redirect
50 if html:
51 # We create the page, so we know there's just one placeholder
52 placeholder = page.placeholders.all()[0]
53 if placeholder.get_plugins():
54 plugin = placeholder.get_plugins()[0].get_plugin_instance()[0]
55 if plugin.body != clean_html(html, full=False):
56 plugin.body = html
57 plugin.save()
58 else:
59 add_plugin(
60 placeholder, 'RawHtmlPlugin',
61 DEFAULT_LANG, body=html)
62 else:96 else:
63 parent = _find_parent(full_url)97 parent = _find_parent(full_url)
64 if not parent:98 if not parent:
@@ -70,6 +104,4 @@
70 position='last-child', redirect=redirect)104 position='last-child', redirect=redirect)
71 placeholder = page.placeholders.get()105 placeholder = page.placeholders.get()
72 add_plugin(placeholder, 'RawHtmlPlugin', DEFAULT_LANG, body=html)106 add_plugin(placeholder, 'RawHtmlPlugin', DEFAULT_LANG, body=html)
73 placeholder = page.placeholders.all()[0]
74 plugin = placeholder.get_plugins()[0].get_plugin_instance()[0]
75 return page107 return page
76108
=== modified file 'md_importer/importer/repo.py'
--- md_importer/importer/repo.py 2016-01-15 18:54:50 +0000
+++ md_importer/importer/repo.py 2016-03-02 11:13:56 +0000
@@ -118,7 +118,6 @@
118 logging.error('Publishing of {} aborted.'.format(self.origin))118 logging.error('Publishing of {} aborted.'.format(self.origin))
119 return False119 return False
120 article.replace_links(self.titles, self.url_map)120 article.replace_links(self.titles, self.url_map)
121 self.pages = []
122 for article in self.imported_articles:121 for article in self.imported_articles:
123 self.pages.extend([article.publish()])122 self.pages.extend([article.publish()])
124 if self.index_page:123 if self.index_page:
125124
=== modified file 'md_importer/tests/test_branch_import.py'
--- md_importer/tests/test_branch_import.py 2016-01-15 13:59:32 +0000
+++ md_importer/tests/test_branch_import.py 2016-03-02 11:13:56 +0000
@@ -2,9 +2,10 @@
2import pytz2import pytz
3import shutil3import shutil
44
5from cms.models import CMSPlugin, Page5from cms.models import Page
66
7from md_importer.importer.article import Article7from md_importer.importer.article import Article
8from md_importer.importer.publish import find_text_plugin
8from .utils import TestLocalBranchImport9from .utils import TestLocalBranchImport
910
1011
@@ -66,6 +67,39 @@
66 self.assertEqual(page.parent_id, self.root.id)67 self.assertEqual(page.parent_id, self.root.id)
6768
6869
70class TestArticleHTMLTagsAfterImport(TestLocalBranchImport):
71 def runTest(self):
72 self.create_repo('data/snapcraft-test')
73 self.repo.add_directive('docs', '')
74 self.assertEqual(len(self.repo.directives), 1)
75 self.assertTrue(self.repo.execute_import_directives())
76 self.assertGreater(len(self.repo.imported_articles), 3)
77 self.assertTrue(self.repo.publish())
78 pages = Page.objects.all()
79 self.assertGreater(pages.count(), len(self.repo.imported_articles))
80 for article in self.repo.imported_articles:
81 self.assertIsInstance(article, Article)
82 self.assertNotIn('<body>', article.html)
83 self.assertNotIn('&lt;body&gt;', article.html)
84
85
86class TestNoneInURLAfterImport(TestLocalBranchImport):
87 def runTest(self):
88 self.create_repo('data/snapcraft-test')
89 self.repo.add_directive('docs', '')
90 self.assertEqual(len(self.repo.directives), 1)
91 self.assertTrue(self.repo.execute_import_directives())
92 self.assertGreater(len(self.repo.imported_articles), 3)
93 self.assertTrue(self.repo.publish())
94 pages = Page.objects.all()
95 self.assertGreater(pages.count(), len(self.repo.imported_articles))
96 for article in self.repo.imported_articles:
97 self.assertIsInstance(article, Article)
98 self.assertNotIn('/None/', article.full_url)
99 for page in pages:
100 self.assertIsNotNone(page.get_slug())
101
102
69class TestTwiceImport(TestLocalBranchImport):103class TestTwiceImport(TestLocalBranchImport):
70 '''Run import on the same contents twice, make sure we don't104 '''Run import on the same contents twice, make sure we don't
71 add new pages over and over again.'''105 add new pages over and over again.'''
@@ -101,9 +135,9 @@
101 self.assertEqual(135 self.assertEqual(
102 Page.objects.filter(publisher_is_draft=False).count(),136 Page.objects.filter(publisher_is_draft=False).count(),
103 len(self.repo.imported_articles)+1) # articles + root137 len(self.repo.imported_articles)+1) # articles + root
138 shutil.rmtree(self.tempdir)
104 # Take the time before publishing the second import139 # Take the time before publishing the second import
105 now = datetime.now(pytz.utc)140 now = datetime.now(pytz.utc)
106 shutil.rmtree(self.tempdir)
107 # Run second import141 # Run second import
108 self.create_repo('data/snapcraft-test')142 self.create_repo('data/snapcraft-test')
109 self.repo.add_directive('docs', '')143 self.repo.add_directive('docs', '')
@@ -112,7 +146,7 @@
112 self.assertTrue(self.repo.execute_import_directives())146 self.assertTrue(self.repo.execute_import_directives())
113 self.assertTrue(self.repo.publish())147 self.assertTrue(self.repo.publish())
114 # Check the page's plugins148 # Check the page's plugins
115 for plugin_change in CMSPlugin.objects.filter(149 for page in Page.objects.filter(publisher_is_draft=False):
116 plugin_type='RawHtmlPlugin').order_by(150 if page != self.root:
117 '-changed_date'):151 (dummy, plugin) = find_text_plugin(page)
118 self.assertGreater(now, plugin_change.changed_date)152 self.assertGreater(now, plugin.changed_date)
119153
=== modified file 'md_importer/tests/test_link_rewrite.py'
--- md_importer/tests/test_link_rewrite.py 2016-01-11 14:38:51 +0000
+++ md_importer/tests/test_link_rewrite.py 2016-03-02 11:13:56 +0000
@@ -30,6 +30,11 @@
30 link.attrs['href'],30 link.attrs['href'],
31 ', '.join([p.get_absolute_url() for p in pages])))31 ', '.join([p.get_absolute_url() for p in pages])))
32 self.assertIn(page, pages)32 self.assertIn(page, pages)
33 if article.slug == 'file1':
34 for link in soup.find_all('a'):
35 if not link.has_attr('class') or \
36 'headeranchor-link' not in link.attrs['class']:
37 self.assertEqual(link.attrs['href'], '/file2')
3338
3439
35class TestLinkBrokenRewrite(TestLocalBranchImport):40class TestLinkBrokenRewrite(TestLocalBranchImport):
@@ -45,12 +50,34 @@
45 self.assertEqual(article.page.parent, self.root)50 self.assertEqual(article.page.parent, self.root)
46 soup = BeautifulSoup(article.html, 'html5lib')51 soup = BeautifulSoup(article.html, 'html5lib')
47 for link in soup.find_all('a'):52 for link in soup.find_all('a'):
48 if link.has_attr('class') and \53 if not link.has_attr('class') or \
49 'headeranchor-link' in link.attrs['class']:54 'headeranchor-link' not in link.attrs['class']:
50 break55 page = self.check_local_link(link.attrs['href'])
51 page = self.check_local_link(link.attrs['href'])56 self.assertIsNone(page)
52 self.assertIsNone(page)57 self.assertNotIn(page, pages)
53 self.assertNotIn(page, pages)58
59
60class TestNoneNotInLinks(TestLocalBranchImport):
61 def runTest(self):
62 self.create_repo('data/snapcraft-test')
63 snappy_page = db_add_empty_page('Snappy', self.root)
64 self.assertFalse(snappy_page.publisher_is_draft)
65 build_apps = db_add_empty_page('Build Apps', snappy_page)
66 self.assertFalse(build_apps.publisher_is_draft)
67 self.assertEqual(
68 3, Page.objects.filter(publisher_is_draft=False).count())
69 self.repo.add_directive('docs/intro.md', 'snappy/build-apps/current')
70 self.repo.add_directive('docs', 'snappy/build-apps/current')
71 self.assertTrue(self.repo.execute_import_directives())
72 self.assertTrue(self.repo.publish())
73 pages = Page.objects.all()
74 for article in self.repo.imported_articles:
75 self.assertTrue(isinstance(article, Article))
76 self.assertGreater(len(article.html), 0)
77 soup = BeautifulSoup(article.html, 'html5lib')
78 for link in soup.find_all('a'):
79 if is_local_link(link):
80 self.assertFalse(link.attrs['href'].startswith('/None/'))
5481
5582
56class TestSnapcraftLinkRewrite(TestLocalBranchImport):83class TestSnapcraftLinkRewrite(TestLocalBranchImport):
@@ -62,25 +89,21 @@
62 self.assertFalse(build_apps.publisher_is_draft)89 self.assertFalse(build_apps.publisher_is_draft)
63 self.assertEqual(90 self.assertEqual(
64 3, Page.objects.filter(publisher_is_draft=False).count())91 3, Page.objects.filter(publisher_is_draft=False).count())
65 self.repo.add_directive('docs', 'snappy/build-apps/devel')92 self.repo.add_directive('docs/intro.md', 'snappy/build-apps/current')
66 self.repo.add_directive('README.md', 'snappy/build-apps/devel')93 self.repo.add_directive('docs', 'snappy/build-apps/current')
67 self.repo.add_directive(
68 'HACKING.md', 'snappy/build-apps/devel/hacking')
69 self.assertTrue(self.repo.execute_import_directives())94 self.assertTrue(self.repo.execute_import_directives())
70 self.assertTrue(self.repo.publish())95 self.assertTrue(self.repo.publish())
71 pages = Page.objects.all()96 pages = Page.objects.all()
72 for article in self.repo.imported_articles:97 for article in self.repo.imported_articles:
73 self.assertTrue(isinstance(article, Article))98 self.assertTrue(isinstance(article, Article))
74 self.assertGreater(len(article.html), 0)99 self.assertGreater(len(article.html), 0)
75 for article in self.repo.imported_articles:
76 soup = BeautifulSoup(article.html, 'html5lib')100 soup = BeautifulSoup(article.html, 'html5lib')
77 for link in soup.find_all('a'):101 for link in soup.find_all('a'):
78 if not is_local_link(link):102 if is_local_link(link):
79 break103 page = self.check_local_link(link.attrs['href'])
80 page = self.check_local_link(link.attrs['href'])104 self.assertIsNotNone(
81 self.assertIsNotNone(105 page,
82 page,106 msg='Link {} not found. Available pages: {}'.format(
83 msg='Link {} not found. Available pages: {}'.format(107 link.attrs['href'],
84 link.attrs['href'],108 ', '.join([p.get_absolute_url() for p in pages])))
85 ', '.join([p.get_absolute_url() for p in pages])))109 self.assertIn(page, pages)
86 self.assertIn(page, pages)
87110
=== modified file 'md_importer/tests/utils.py'
--- md_importer/tests/utils.py 2016-01-11 14:38:51 +0000
+++ md_importer/tests/utils.py 2016-03-02 11:13:56 +0000
@@ -55,8 +55,10 @@
55 self.assertEqual(self.fetch_retcode, 0)55 self.assertEqual(self.fetch_retcode, 0)
5656
57 def check_local_link(self, url):57 def check_local_link(self, url):
58 if not url.startswith('/'):
59 url = '/' + url
58 if not url.startswith('/{}/'.format(DEFAULT_LANG)):60 if not url.startswith('/{}/'.format(DEFAULT_LANG)):
59 url = '/{}/{}/'.format(DEFAULT_LANG, url)61 url = '/{}'.format(DEFAULT_LANG) + url
60 request = self.get_request(url)62 request = self.get_request(url)
61 page = get_page_from_request(request)63 page = get_page_from_request(request)
62 return page64 return page

Subscribers

People subscribed via source and target branches