Merge lp:~dholbach/developer-ubuntu-com/rework-importer into lp:~developer-ubuntu-com-dev/developer-ubuntu-com/django-1.8-cms-2.3
- rework-importer
- Merge into django-1.8-cms-2.3
Status: | Superseded | ||||||||
---|---|---|---|---|---|---|---|---|---|
Proposed branch: | lp:~dholbach/developer-ubuntu-com/rework-importer | ||||||||
Merge into: | lp:~developer-ubuntu-com-dev/developer-ubuntu-com/django-1.8-cms-2.3 | ||||||||
Diff against target: |
1625 lines (+953/-399) 28 files modified
TODO (+13/-0) developer_portal/admin.py (+1/-18) developer_portal/migrations/0001_initial.py (+0/-9) developer_portal/models.py (+0/-21) developer_portal/settings.py (+2/-0) locale/de.po (+2/-2) locale/en_GB.po (+2/-2) locale/es.po (+2/-2) locale/ug.po (+2/-2) locale/zh_CN.po (+2/-2) md_importer/admin.py (+36/-0) md_importer/management/commands/import-external-docs-branches.py (+35/-335) md_importer/management/importer/article.py (+118/-0) md_importer/management/importer/publish.py (+64/-0) md_importer/management/importer/repo.py (+168/-0) md_importer/management/importer/source.py (+60/-0) md_importer/migrations/0001_initial.py (+46/-0) md_importer/models.py (+54/-0) md_importer/tests/__init__.py (+8/-0) md_importer/tests/data/link-test/file1.md (+5/-0) md_importer/tests/data/link-test/file2.md (+3/-0) md_importer/tests/test_branch_fetch.py (+42/-0) md_importer/tests/test_branch_import.py (+67/-0) md_importer/tests/test_link_rewrite.py (+44/-0) md_importer/tests/test_snappy_import.py (+70/-0) md_importer/tests/test_utils.py (+33/-0) md_importer/tests/utils.py (+67/-0) requirements.txt (+7/-6) |
||||||||
To merge this branch: | bzr merge lp:~dholbach/developer-ubuntu-com/rework-importer | ||||||||
Related bugs: |
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Ubuntu App Developer site developers | Pending | ||
Review via email:
|
Commit message
Description of the change
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Daniel Holbach (dholbach) wrote : | # |
- 252. By Daniel Holbach
-
break up tests into individual files
- 253. By Daniel Holbach
-
update TODO
- 254. By Daniel Holbach
-
improve create_repo helper function
- 255. By Daniel Holbach
-
allow SourceCode.
_get_branch to use local docs (ie for testing) - 256. By Daniel Holbach
-
first cut at testing local links
- 257. By Daniel Holbach
-
fix simple link rewrite functionality and test
- 258. By Daniel Holbach
-
move md_importer into its own app
- 259. By Daniel Holbach
-
add migration for md_importer
- 260. By Daniel Holbach
- 261. By Daniel Holbach
-
make directory structure clearer
- 262. By Daniel Holbach
-
adapt to new name of management command
- 263. By Daniel Holbach
-
add more readable error message
- 264. By Daniel Holbach
-
add more local test data (imported just the docs from snappy's and snapcraft's git master branches)
- 265. By Daniel Holbach
-
WIP commit:
- move and update TODO file
- define some global values for the importer centrally
- don't sys.exit() when adding pages to the db
- add debug messages
- break _find_parent() into its own function
- mostly make use of local test data
- drop singleton, use separate test classes instead
- add more assertions in tests - 266. By Daniel Holbach
-
remove debug prints, make Repo class variables actual members
- 267. By Daniel Holbach
-
- make pep8 happy, remove unnecessary imports
- use TestLocalBranchImport as base class for almost everything,
reduces a lot of duplication - 268. By Daniel Holbach
-
remove unnecessary code, fix publishing of pages in tests
- 269. By Daniel Holbach
-
fix test_link_rewrite by fixing the URL
- 270. By Daniel Holbach
-
make Article class variables actual members
- 271. By Daniel Holbach
-
add new test to check links in snapcraft import, move link checking function into TestLocalBranch
Import - 272. By Daniel Holbach
-
merge lp:~developer-ubuntu-com-dev/developer-ubuntu-com/django-1.8-cms-2.3
- 273. By Daniel Holbach
-
allow empty strings in import directives
- 274. By Daniel Holbach
-
update migration
- 275. By Daniel Holbach
-
check if articles were added below home
- 276. By Daniel Holbach
-
remove unnecessary call to set a page's publisher_is_draft bit
- 277. By Daniel Holbach
-
give article a .publish() method which gives back the page's public_object
- 278. By Daniel Holbach
-
rename home_page to root_page, use cms.test_
utils.testcases .CMSTestCase, make sure we use public_object wherever possible - 279. By Daniel Holbach
-
update tests accordingly
- 280. By Daniel Holbach
-
disregard anchors, make link checking more flexible (ie not only check for intro.md, but also things like docs/intro.md
- 281. By Daniel Holbach
-
make URL work, even if LANG is already part of it
- 282. By Daniel Holbach
-
links look like they're working now
- 283. By Daniel Holbach
-
add test with a broken link, add convenience function is_local_link, modify tests
- 284. By Daniel Holbach
-
import doc fix from https:/
/bugs.launchpad .net/developer- ubuntu- com/+bug/ 1531200 - 285. By Daniel Holbach
-
bug fixed
- 286. By Daniel Holbach
-
stop import if local images are found, add tests
- 287. By Daniel Holbach
-
update TODO
- 288. By Daniel Holbach
-
add test to see if importing the same content twice results in the same number of pages
- 289. By Daniel Holbach
-
when replacing links, only update HTML if things actually change, only publish if page is dirty
- 290. By Daniel Holbach
-
only update page attributes if they actually change, only update the text plugin if the html actually changes, veryify with djangocms_
text_ckeditor. html.clean_ html, add a text plugin to a page, even if html is empty - 291. By Daniel Holbach
-
add test to check if running an import twice will update the articles in question
- 292. By Daniel Holbach
-
update TODO
- 293. By Daniel Holbach
-
add misc tests - forgot to 'bzr add'
- 294. By Daniel Holbach
-
make update-mtemplate work again, simplify it
- 295. By Daniel Holbach
-
update .pot file
- 296. By Daniel Holbach
-
add actual page object to repo.pages
- 297. By Daniel Holbach
-
use UTC for ImportedArticle
.last_import, simplify ImportedArticle cleanup - 298. By Daniel Holbach
-
use repo instead of branch consistently
- 299. By Daniel Holbach
-
break out the process of importing a branch into its own module, add a first simple test for it
- 300. By Daniel Holbach
-
fix clean up of imported articles, add a test
- 301. By Daniel Holbach
-
update TODO
Unmerged revisions
- 301. By Daniel Holbach
-
update TODO
- 300. By Daniel Holbach
-
fix clean up of imported articles, add a test
- 299. By Daniel Holbach
-
break out the process of importing a branch into its own module, add a first simple test for it
- 298. By Daniel Holbach
-
use repo instead of branch consistently
- 297. By Daniel Holbach
-
use UTC for ImportedArticle
.last_import, simplify ImportedArticle cleanup - 296. By Daniel Holbach
-
add actual page object to repo.pages
- 295. By Daniel Holbach
-
update .pot file
- 294. By Daniel Holbach
-
make update-mtemplate work again, simplify it
- 293. By Daniel Holbach
-
add misc tests - forgot to 'bzr add'
- 292. By Daniel Holbach
-
update TODO
Preview Diff
1 | === added file 'TODO' |
2 | --- TODO 1970-01-01 00:00:00 +0000 |
3 | +++ TODO 2016-01-06 12:40:31 +0000 |
4 | @@ -0,0 +1,13 @@ |
5 | +- fix branch indicator (devel vs. stable, 1.x vs devel, etc.) |
6 | + |
7 | +- fix links |
8 | +- add tests for links |
9 | + |
10 | +- check if https://bugs.launchpad.net/developer-ubuntu-com/+bug/1531200 works |
11 | + |
12 | +- import pictures (LP: #1511677) |
13 | +- unpublished docs: some articles show up with a blue circle, so |
14 | + unpublished although they're accessible(?) |
15 | +- add comment at top of RawHTML plugins, so users know not to edit them, |
16 | + currently blocked on LP: #1523925 |
17 | +- if article already exists, check if HTML actually changed, if not, don't update. |
18 | |
19 | === modified file 'developer_portal/admin.py' |
20 | --- developer_portal/admin.py 2015-11-02 12:17:27 +0000 |
21 | +++ developer_portal/admin.py 2016-01-06 12:40:31 +0000 |
22 | @@ -4,22 +4,12 @@ |
23 | from reversion.admin import VersionAdmin |
24 | |
25 | from cms.extensions import TitleExtensionAdmin |
26 | -from .models import ExternalDocsBranch, SEOExtension |
27 | -from django.core.management import call_command |
28 | +from .models import SEOExtension |
29 | |
30 | __all__ = ( |
31 | ) |
32 | |
33 | |
34 | -def import_selected_external_docs_branches(modeladmin, request, queryset): |
35 | - branches = [] |
36 | - for branch in queryset: |
37 | - branches.append(branch.docs_namespace) |
38 | - call_command('import-external-docs-branches', *branches) |
39 | - import_selected_external_docs_branches.short_description = \ |
40 | - "Import selected branches" |
41 | - |
42 | - |
43 | class RevisionAdmin(admin.ModelAdmin): |
44 | list_display = ('date_created', 'user', 'comment') |
45 | list_display_links = ('date_created', ) |
46 | @@ -37,13 +27,6 @@ |
47 | admin.site.register(Version, VersionAdmin) |
48 | |
49 | |
50 | -class ExternalDocsBranchAdmin(admin.ModelAdmin): |
51 | - list_display = ('lp_origin', 'docs_namespace') |
52 | - list_filter = ('lp_origin', 'docs_namespace') |
53 | - actions = [import_selected_external_docs_branches] |
54 | - |
55 | -admin.site.register(ExternalDocsBranch, ExternalDocsBranchAdmin) |
56 | - |
57 | class SEOExtensionAdmin(TitleExtensionAdmin): |
58 | pass |
59 | |
60 | |
61 | === modified file 'developer_portal/migrations/0001_initial.py' |
62 | --- developer_portal/migrations/0001_initial.py 2015-11-17 13:38:11 +0000 |
63 | +++ developer_portal/migrations/0001_initial.py 2016-01-06 12:40:31 +0000 |
64 | @@ -11,15 +11,6 @@ |
65 | |
66 | operations = [ |
67 | migrations.CreateModel( |
68 | - name='ExternalDocsBranch', |
69 | - fields=[ |
70 | - ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)), |
71 | - ('lp_origin', models.CharField(help_text='External branch location, ie: lp:snappy/15.04 or https://github.com/ubuntu-core/snappy.git', max_length=200)), |
72 | - ('docs_namespace', models.CharField(help_text='Path alias we want to use for the docs, ie "snappy/guides/15.04" or "snappy/guides/latest", etc.', max_length=120)), |
73 | - ('index_doc', models.CharField(help_text='File name of doc to be used as index document, ie "intro.md"', max_length=120, blank=True)), |
74 | - ], |
75 | - ), |
76 | - migrations.CreateModel( |
77 | name='RawHtml', |
78 | fields=[ |
79 | ('cmsplugin_ptr', models.OneToOneField(parent_link=True, auto_created=True, primary_key=True, serialize=False, to='cms.CMSPlugin')), |
80 | |
81 | === modified file 'developer_portal/models.py' |
82 | --- developer_portal/models.py 2015-11-02 16:47:26 +0000 |
83 | +++ developer_portal/models.py 2016-01-06 12:40:31 +0000 |
84 | @@ -1,7 +1,5 @@ |
85 | from django.db import models |
86 | -from django.utils.translation import ugettext_lazy as _ |
87 | |
88 | -from cms.models import CMSPlugin |
89 | from cms.extensions import TitleExtension |
90 | from cms.extensions.extension_pool import extension_pool |
91 | from djangocms_text_ckeditor.html import extract_images |
92 | @@ -20,25 +18,6 @@ |
93 | AbstractText.save(self, *args, **kwargs) |
94 | |
95 | |
96 | -class ExternalDocsBranch(models.Model): |
97 | - # We originally assumed that branches would also live in LP, |
98 | - # well, we were wrong, but let's keep the name around. It's |
99 | - # no use having a schema/data migration just for this. |
100 | - lp_origin = models.CharField( |
101 | - max_length=200, |
102 | - help_text=_('External branch location, ie: lp:snappy/15.04 or ' |
103 | - 'https://github.com/ubuntu-core/snappy.git')) |
104 | - docs_namespace = models.CharField( |
105 | - max_length=120, |
106 | - help_text=_('Path alias we want to use for the docs, ' |
107 | - 'ie "snappy/guides/15.04" or ' |
108 | - '"snappy/guides/latest", etc.')) |
109 | - index_doc = models.CharField( |
110 | - max_length=120, |
111 | - help_text=_('File name of doc to be used as index document, ' |
112 | - 'ie "intro.md"'), |
113 | - blank=True) |
114 | - |
115 | class SEOExtension(TitleExtension): |
116 | keywords = models.CharField(max_length=256) |
117 | |
118 | |
119 | === modified file 'developer_portal/settings.py' |
120 | --- developer_portal/settings.py 2015-12-11 16:10:37 +0000 |
121 | +++ developer_portal/settings.py 2016-01-06 12:40:31 +0000 |
122 | @@ -76,6 +76,8 @@ |
123 | 'store_data', |
124 | |
125 | 'api_docs', |
126 | + |
127 | + 'md_importer', |
128 | ] |
129 | |
130 | MIDDLEWARE_CLASSES = ( |
131 | |
132 | === modified file 'locale/de.po' |
133 | --- locale/de.po 2015-11-24 05:38:32 +0000 |
134 | +++ locale/de.po 2016-01-06 12:40:31 +0000 |
135 | @@ -15,8 +15,8 @@ |
136 | "Content-Type: text/plain; charset=UTF-8\n" |
137 | "Content-Transfer-Encoding: 8bit\n" |
138 | "Plural-Forms: nplurals=2; plural=n != 1;\n" |
139 | -"X-Launchpad-Export-Date: 2015-11-24 05:38+0000\n" |
140 | -"X-Generator: Launchpad (build 17850)\n" |
141 | +"X-Launchpad-Export-Date: 2015-12-03 06:16+0000\n" |
142 | +"X-Generator: Launchpad (build 17856)\n" |
143 | |
144 | #: developer_portal/cms_plugins.py:10 |
145 | msgid "Raw HTML" |
146 | |
147 | === modified file 'locale/en_GB.po' |
148 | --- locale/en_GB.po 2015-11-24 05:38:32 +0000 |
149 | +++ locale/en_GB.po 2016-01-06 12:40:31 +0000 |
150 | @@ -15,8 +15,8 @@ |
151 | "Content-Type: text/plain; charset=UTF-8\n" |
152 | "Content-Transfer-Encoding: 8bit\n" |
153 | "Plural-Forms: nplurals=2; plural=n != 1;\n" |
154 | -"X-Launchpad-Export-Date: 2015-11-24 05:38+0000\n" |
155 | -"X-Generator: Launchpad (build 17850)\n" |
156 | +"X-Launchpad-Export-Date: 2015-12-03 06:16+0000\n" |
157 | +"X-Generator: Launchpad (build 17856)\n" |
158 | |
159 | #: developer_portal/cms_plugins.py:10 |
160 | msgid "Raw HTML" |
161 | |
162 | === modified file 'locale/es.po' |
163 | --- locale/es.po 2015-11-24 05:38:32 +0000 |
164 | +++ locale/es.po 2016-01-06 12:40:31 +0000 |
165 | @@ -15,8 +15,8 @@ |
166 | "Content-Type: text/plain; charset=UTF-8\n" |
167 | "Content-Transfer-Encoding: 8bit\n" |
168 | "Plural-Forms: nplurals=2; plural=n != 1;\n" |
169 | -"X-Launchpad-Export-Date: 2015-11-24 05:38+0000\n" |
170 | -"X-Generator: Launchpad (build 17850)\n" |
171 | +"X-Launchpad-Export-Date: 2015-12-03 06:16+0000\n" |
172 | +"X-Generator: Launchpad (build 17856)\n" |
173 | |
174 | #: developer_portal/cms_plugins.py:10 |
175 | msgid "Raw HTML" |
176 | |
177 | === modified file 'locale/ug.po' |
178 | --- locale/ug.po 2015-11-24 05:38:32 +0000 |
179 | +++ locale/ug.po 2016-01-06 12:40:31 +0000 |
180 | @@ -15,8 +15,8 @@ |
181 | "Content-Type: text/plain; charset=UTF-8\n" |
182 | "Content-Transfer-Encoding: 8bit\n" |
183 | "Plural-Forms: nplurals=1; plural=0;\n" |
184 | -"X-Launchpad-Export-Date: 2015-11-24 05:38+0000\n" |
185 | -"X-Generator: Launchpad (build 17850)\n" |
186 | +"X-Launchpad-Export-Date: 2015-12-03 06:16+0000\n" |
187 | +"X-Generator: Launchpad (build 17856)\n" |
188 | |
189 | #: developer_portal/cms_plugins.py:10 |
190 | msgid "Raw HTML" |
191 | |
192 | === modified file 'locale/zh_CN.po' |
193 | --- locale/zh_CN.po 2015-11-24 05:38:32 +0000 |
194 | +++ locale/zh_CN.po 2016-01-06 12:40:31 +0000 |
195 | @@ -15,8 +15,8 @@ |
196 | "Content-Type: text/plain; charset=UTF-8\n" |
197 | "Content-Transfer-Encoding: 8bit\n" |
198 | "Plural-Forms: nplurals=1; plural=0;\n" |
199 | -"X-Launchpad-Export-Date: 2015-11-24 05:38+0000\n" |
200 | -"X-Generator: Launchpad (build 17850)\n" |
201 | +"X-Launchpad-Export-Date: 2015-12-03 06:16+0000\n" |
202 | +"X-Generator: Launchpad (build 17856)\n" |
203 | |
204 | #: developer_portal/cms_plugins.py:10 |
205 | msgid "Raw HTML" |
206 | |
207 | === added directory 'md_importer' |
208 | === added file 'md_importer/__init__.py' |
209 | === added file 'md_importer/admin.py' |
210 | --- md_importer/admin.py 1970-01-01 00:00:00 +0000 |
211 | +++ md_importer/admin.py 2016-01-06 12:40:31 +0000 |
212 | @@ -0,0 +1,36 @@ |
213 | +from django.contrib import admin |
214 | + |
215 | +from .models import ( |
216 | + ExternalDocsBranch, ExternalDocsBranchImportDirective, |
217 | + ImportedArticle, |
218 | +) |
219 | +from django.core.management import call_command |
220 | + |
221 | +__all__ = ( |
222 | +) |
223 | + |
224 | + |
225 | +def import_selected_external_docs_branches(modeladmin, request, queryset): |
226 | + branches = [] |
227 | + for branch in queryset: |
228 | + branches.append(branch.origin) |
229 | + call_command('import-external-docs-branches', *branches) |
230 | + import_selected_external_docs_branches.short_description = \ |
231 | + "Import selected branches" |
232 | + |
233 | + |
234 | +@admin.register(ExternalDocsBranch) |
235 | +class ExternalDocsBranchAdmin(admin.ModelAdmin): |
236 | + list_display = ('origin', 'post_checkout_command', 'branch_name',) |
237 | + list_filter = ('origin', 'post_checkout_command', 'branch_name',) |
238 | + actions = [import_selected_external_docs_branches] |
239 | + |
240 | + |
241 | +@admin.register(ExternalDocsBranchImportDirective) |
242 | +class ExternalDocsBranchImportDirectiveAdmin(admin.ModelAdmin): |
243 | + pass |
244 | + |
245 | + |
246 | +@admin.register(ImportedArticle) |
247 | +class ImportedArticleAdmin(admin.ModelAdmin): |
248 | + pass |
249 | |
250 | === added directory 'md_importer/management' |
251 | === added file 'md_importer/management/__init__.py' |
252 | === added directory 'md_importer/management/commands' |
253 | === added file 'md_importer/management/commands/__init__.py' |
254 | === renamed file 'developer_portal/management/commands/import-external-docs-branches.py' => 'md_importer/management/commands/import-external-docs-branches.py' |
255 | --- developer_portal/management/commands/import-external-docs-branches.py 2015-11-06 09:48:07 +0000 |
256 | +++ md_importer/management/commands/import-external-docs-branches.py 2016-01-06 12:40:31 +0000 |
257 | @@ -1,300 +1,18 @@ |
258 | from django.core.management.base import BaseCommand |
259 | from django.core.management import call_command |
260 | -from django.db import transaction |
261 | - |
262 | -from cms.api import create_page, add_plugin |
263 | -from cms.models import Page, Title |
264 | -from cms.utils import page_resolver |
265 | - |
266 | -from bs4 import BeautifulSoup |
267 | -import codecs |
268 | -import glob |
269 | + |
270 | +from ..importer.repo import create_repo |
271 | + |
272 | +import datetime |
273 | import logging |
274 | -import markdown |
275 | -import os |
276 | -import re |
277 | import shutil |
278 | -import subprocess |
279 | -import sys |
280 | import tempfile |
281 | |
282 | -from developer_portal.models import ExternalDocsBranch |
283 | - |
284 | -DOCS_DIRNAME = 'docs' |
285 | - |
286 | - |
287 | -class DBActions: |
288 | - added_pages = [] |
289 | - removed_pages = [] |
290 | - |
291 | - def add_page(self, **kwargs): |
292 | - self.added_pages += [kwargs] |
293 | - |
294 | - def remove_page(self, page_id): |
295 | - self.removed_pages += [page_id] |
296 | - |
297 | - @transaction.atomic() |
298 | - def run(self): |
299 | - for added_page in self.added_pages: |
300 | - page = get_or_create_page(**added_page) |
301 | - page.publish('en') |
302 | - |
303 | - # Only remove pages created by a script! |
304 | - Page.objects.filter(id__in=self.removed_pages, |
305 | - created_by="script").delete() |
306 | - |
307 | - # https://stackoverflow.com/questions/33284171/ |
308 | - call_command('cms', 'fix-tree') |
309 | - |
310 | - |
311 | -class MarkdownFile: |
312 | - html = None |
313 | - |
314 | - def __init__(self, fn, docs_namespace, db_actions, slug_override=None): |
315 | - self.fn = fn |
316 | - self.docs_namespace = docs_namespace |
317 | - self.db_actions = db_actions |
318 | - if slug_override: |
319 | - self.slug = slug_override |
320 | - else: |
321 | - self.slug = slugify(self.fn) |
322 | - self.full_url = os.path.join(self.docs_namespace, self.slug) |
323 | - with codecs.open(self.fn, 'r', encoding='utf-8') as f: |
324 | - self.html = markdown.markdown( |
325 | - f.read(), |
326 | - output_format="html5", |
327 | - extensions=['markdown.extensions.tables']) |
328 | - self.release_alias = self._get_release_alias() |
329 | - self.title = self._read_title() |
330 | - self._remove_body_and_html_tags() |
331 | - self._use_developer_site_style() |
332 | - |
333 | - def _get_release_alias(self): |
334 | - alias = re.findall(r'/tmp/tmp\S+?/(\S+?)/%s/\S+?' % DOCS_DIRNAME, |
335 | - self.fn) |
336 | - return alias[0] |
337 | - |
338 | - def _read_title(self): |
339 | - soup = BeautifulSoup(self.html, 'html5lib') |
340 | - if soup.title: |
341 | - return soup.title.text |
342 | - if soup.h1: |
343 | - return soup.h1.text |
344 | - return slugify(self.fn).replace('-', ' ').title() |
345 | - |
346 | - def _remove_body_and_html_tags(self): |
347 | - self.html = re.sub(r"<html>\n\s<body>\n", "", self.html, |
348 | - flags=re.MULTILINE) |
349 | - self.html = re.sub(r"\s<\/body>\n<\/html>", "", self.html, |
350 | - flags=re.MULTILINE) |
351 | - |
352 | - def _use_developer_site_style(self): |
353 | - begin = (u"<div class=\"row no-border\">" |
354 | - "\n<div class=\"eight-col\">\n") |
355 | - end = u"</div>\n</div>" |
356 | - self.html = begin + self.html + end |
357 | - self.html = self.html.replace( |
358 | - "<pre><code>", |
359 | - "</div><div class=\"twelve-col\"><pre><code>") |
360 | - self.html = self.html.replace( |
361 | - "</code></pre>", |
362 | - "</code></pre></div><div class=\"eight-col\">") |
363 | - |
364 | - def replace_links(self, titles, url_map): |
365 | - for title in titles: |
366 | - local_md_fn = os.path.basename(title) |
367 | - url = u'/'+url_map[title] |
368 | - # Replace links of the form <a href="/path/somefile.md"> first |
369 | - href = u"<a href=\"{}\">".format(url) |
370 | - md_href = u"<a href=\"{}\">".format(local_md_fn) |
371 | - self.html = self.html.replace(md_href, href) |
372 | - |
373 | - # Now we can replace free-standing "somefile.md" references in |
374 | - # the HTML |
375 | - link = href + u"{}</a>".format(titles[title]) |
376 | - self.html = self.html.replace(local_md_fn, link) |
377 | - |
378 | - def publish(self): |
379 | - '''Publishes pages in their branch alias namespace.''' |
380 | - self.db_actions.add_page( |
381 | - title=self.title, full_url=self.full_url, menu_title=self.title, |
382 | - html=self.html) |
383 | - |
384 | - |
385 | -class SnappyMarkdownFile(MarkdownFile): |
386 | - def __init__(self, fn, docs_namespace, db_actions): |
387 | - MarkdownFile.__init__(self, fn, docs_namespace, db_actions) |
388 | - self._make_snappy_mods() |
389 | - |
390 | - def _make_snappy_mods(self): |
391 | - # Make sure the reader knows which documentation she is browsing |
392 | - if self.release_alias != 'current': |
393 | - before = (u"<div class=\"row no-border\">\n" |
394 | - "<div class=\"eight-col\">\n") |
395 | - after = (u"<div class=\"row no-border\">\n" |
396 | - "<div class=\"box pull-three three-col\">" |
397 | - "<p>You are browsing the Snappy <code>%s</code> " |
398 | - "documentation.</p>" |
399 | - "<p><a href=\"/snappy/guides/current/%s\">" |
400 | - "Back to the latest stable release ›" |
401 | - "</a></p></div>\n" |
402 | - "<div class=\"eight-col\">\n") % (self.release_alias, |
403 | - self.slug, ) |
404 | - self.html = self.html.replace(before, after) |
405 | - |
406 | - def publish(self): |
407 | - if self.release_alias == "current": |
408 | - # Add a guides/<page> redirect to guides/current/<page> |
409 | - self.db_actions.add_page( |
410 | - title=self.title, |
411 | - full_url=self.full_url.replace('/current', ''), |
412 | - redirect="/snappy/guides/current/%s" % (self.slug)) |
413 | - else: |
414 | - self.title += " (%s)" % (self.release_alias,) |
415 | - MarkdownFile.publish(self) |
416 | - |
417 | - |
418 | -def slugify(filename): |
419 | - return os.path.basename(filename).replace('.md', '') |
420 | - |
421 | - |
422 | -class LocalBranch: |
423 | - titles = {} |
424 | - url_map = {} |
425 | - |
426 | - def __init__(self, dirname, external_branch, db_actions): |
427 | - self.dirname = dirname |
428 | - self.docs_path = os.path.join(self.dirname, DOCS_DIRNAME) |
429 | - self.doc_fns = glob.glob(self.docs_path+'/*.md') |
430 | - self.md_files = [] |
431 | - self.external_branch = external_branch |
432 | - self.docs_namespace = self.external_branch.docs_namespace |
433 | - self.release_alias = os.path.basename(self.docs_namespace) |
434 | - self.index_doc_title = self.release_alias.title() |
435 | - self.index_doc = self.external_branch.index_doc |
436 | - self.db_actions = db_actions |
437 | - self.markdown_class = MarkdownFile |
438 | - |
439 | - def import_markdown(self): |
440 | - for doc_fn in self.doc_fns: |
441 | - if self.index_doc and os.path.basename(doc_fn) == self.index_doc: |
442 | - md_file = self.markdown_class( |
443 | - doc_fn, |
444 | - os.path.dirname(self.docs_namespace), |
445 | - self.db_actions, |
446 | - slug_override=os.path.basename(self.docs_namespace)) |
447 | - self.md_files.insert(0, md_file) |
448 | - else: |
449 | - md_file = self.markdown_class(doc_fn, self.docs_namespace, |
450 | - self.db_actions) |
451 | - self.md_files += [md_file] |
452 | - self.titles[md_file.fn] = md_file.title |
453 | - self.url_map[md_file.fn] = md_file.full_url |
454 | - if not self.index_doc: |
455 | - self._create_fake_index_doc() |
456 | - for md_file in self.md_files: |
457 | - md_file.replace_links(self.titles, self.url_map) |
458 | - |
459 | - def remove_old_pages(self): |
460 | - imported_page_urls = set([md_file.full_url |
461 | - for md_file in self.md_files]) |
462 | - index_doc = page_resolver.get_page_queryset_from_path( |
463 | - self.docs_namespace) |
464 | - db_pages = [] |
465 | - if len(index_doc): |
466 | - # All pages in this namespace currently in the database |
467 | - db_pages = index_doc[0].get_descendants().all() |
468 | - for db_page in db_pages: |
469 | - still_relevant = False |
470 | - for url in imported_page_urls: |
471 | - if url in db_page.get_absolute_url(): |
472 | - still_relevant = True |
473 | - break |
474 | - # At this point we know that there's no match and the page |
475 | - # can be deleted. |
476 | - if not still_relevant: |
477 | - self.db_actions.remove_page(db_page.id) |
478 | - |
479 | - def publish(self): |
480 | - for md_file in self.md_files: |
481 | - md_file.publish() |
482 | - |
483 | - def _create_fake_index_doc(self): |
484 | - '''Creates a fake index page at the top of the branches |
485 | - docs namespace.''' |
486 | - |
487 | - if self.docs_namespace == "current": |
488 | - redirect = "/snappy/guides" |
489 | - else: |
490 | - redirect = None |
491 | - |
492 | - in_navigation = False |
493 | - menu_title = None |
494 | - list_pages = "" |
495 | - for page in self.md_files: |
496 | - list_pages += "<li><a href=\"%s\">%s</a></li>" \ |
497 | - % (os.path.basename(page.full_url), page.title) |
498 | - landing = ( |
499 | - u"<div class=\"row\"><div class=\"eight-col\">\n" |
500 | - "<p>This section contains documentation for the " |
501 | - "<code>%s</code> Snappy branch.</p>" |
502 | - "<p><ul class=\"list-ubuntu\">%s</ul></p>\n" |
503 | - "<p>Auto-imported from <a " |
504 | - "href=\"https://github.com/ubuntu-core/snappy\">%s</a>.</p>\n" |
505 | - "</div></div>") % (self.release_alias, list_pages, |
506 | - self.external_branch.lp_origin) |
507 | - self.db_actions.add_page( |
508 | - title=self.index_doc_title, full_url=self.docs_namespace, |
509 | - in_navigation=in_navigation, redirect=redirect, html=landing, |
510 | - menu_title=menu_title) |
511 | - |
512 | - |
513 | -class SnappyLocalBranch(LocalBranch): |
514 | - def __init__(self, dirname, external_branch, db_actions): |
515 | - LocalBranch.__init__(self, dirname, external_branch, db_actions) |
516 | - self.markdown_class = SnappyMarkdownFile |
517 | - self.index_doc_title = 'Snappy documentation' |
518 | - if self.release_alias != 'current': |
519 | - self.index_doc_title += ' (%s)' % self.release_alias |
520 | - |
521 | - |
522 | -def get_or_create_page(title, full_url, menu_title=None, |
523 | - in_navigation=True, redirect=None, html=None): |
524 | - # First check if pages already exist. |
525 | - pages = Title.objects.select_related('page').filter(path__regex=full_url) |
526 | - if pages: |
527 | - page = pages[0].page |
528 | - page.title = title |
529 | - page.publisher_is_draft = True |
530 | - page.menu_title = menu_title |
531 | - page.in_navigation = in_navigation |
532 | - page.redirect = redirect |
533 | - if html: |
534 | - # We create the page, so we know there's just one placeholder |
535 | - placeholder = page.placeholders.all()[0] |
536 | - if placeholder.get_plugins(): |
537 | - plugin = placeholder.get_plugins()[0].get_plugin_instance()[0] |
538 | - plugin.body = html |
539 | - plugin.save() |
540 | - else: |
541 | - add_plugin(placeholder, 'RawHtmlPlugin', 'en', body=html) |
542 | - else: |
543 | - parent_pages = Title.objects.select_related('page').filter( |
544 | - path__regex=os.path.dirname(full_url)) |
545 | - if not parent_pages: |
546 | - print('Parent %s not found.' % os.path.dirname(full_url)) |
547 | - sys.exit(1) |
548 | - parent = parent_pages[0].page |
549 | - |
550 | - slug = os.path.basename(full_url) |
551 | - page = create_page( |
552 | - title, "default.html", "en", slug=slug, parent=parent, |
553 | - menu_title=menu_title, in_navigation=in_navigation, |
554 | - position="last-child", redirect=redirect) |
555 | - if html: |
556 | - placeholder = page.placeholders.get() |
557 | - add_plugin(placeholder, 'RawHtmlPlugin', 'en', body=html) |
558 | - return page |
559 | +from developer_portal.models import ( |
560 | + ExternalDocsBranch, |
561 | + ExternalDocsBranchImportDirective, |
562 | + ImportedArticle, |
563 | +) |
564 | |
565 | |
566 | def import_branches(selection): |
567 | @@ -302,52 +20,34 @@ |
568 | logging.error('No branches registered in the ' |
569 | 'ExternalDocsBranch table yet.') |
570 | return |
571 | - tempdir = tempfile.mkdtemp() |
572 | - db_actions = DBActions() |
573 | for branch in ExternalDocsBranch.objects.filter( |
574 | - docs_namespace__regex=selection): |
575 | - checkout_location = os.path.join( |
576 | - tempdir, os.path.basename(branch.docs_namespace)) |
577 | - sourcecode = SourceCode(branch.lp_origin, checkout_location) |
578 | - if sourcecode.get() != 0: |
579 | - logging.error( |
580 | - 'Could not check out branch "%s".' % branch.lp_origin) |
581 | - if os.path.exists(checkout_location): |
582 | - shutil.rmtree(checkout_location) |
583 | + origin__regex=selection, active=True): |
584 | + tempdir = tempfile.mkdtemp() |
585 | + repo = create_repo(tempdir, branch.origin, branch.branch_name, |
586 | + branch.post_checkout_command) |
587 | + if repo.get() != 0: |
588 | break |
589 | - if branch.lp_origin.startswith('lp:snappy') or \ |
590 | - 'snappy' in branch.lp_origin.split(':')[1].split('.git')[0].split('/'): |
591 | - local_branch = SnappyLocalBranch(checkout_location, branch, |
592 | - db_actions) |
593 | - else: |
594 | - local_branch = LocalBranch(checkout_location, branch, db_actions) |
595 | - local_branch.import_markdown() |
596 | - local_branch.publish() |
597 | - local_branch.remove_old_pages() |
598 | - shutil.rmtree(tempdir) |
599 | - db_actions.run() |
600 | - |
601 | - |
602 | -class SourceCode(): |
603 | - def __init__(self, branch_origin, checkout_location): |
604 | - self.branch_origin = branch_origin |
605 | - self.checkout_location = checkout_location |
606 | - |
607 | - def get(self): |
608 | - if self.branch_origin.startswith('lp:') and \ |
609 | - os.path.exists('/usr/bin/bzr'): |
610 | - return subprocess.call([ |
611 | - 'bzr', 'checkout', '--lightweight', self.branch_origin, |
612 | - self.checkout_location]) |
613 | - if self.branch_origin.startswith('https://github.com') and \ |
614 | - self.branch_origin.endswith('.git') and \ |
615 | - os.path.exists('/usr/bin/git'): |
616 | - return subprocess.call([ |
617 | - 'git', 'clone', '-q', self.branch_origin, |
618 | - self.checkout_location]) |
619 | - logging.error( |
620 | - 'Branch format "{}" not understood.'.format(self.branch_origin)) |
621 | - return 1 |
622 | + for directive in ExternalDocsBranchImportDirective.objects.filter( |
623 | + external_docs_branch=branch): |
624 | + repo.add_directive(directive.import_from, |
625 | + directive.write_to) |
626 | + repo.execute_import_directives() |
627 | + repo.publish() |
628 | + for page in repo.pages: |
629 | + ImportedArticle.objects.get_or_create( |
630 | + branch=branch, |
631 | + page=page, |
632 | + last_import=datetime.datetime.now()) |
633 | + |
634 | + # The import is done, now let's clean up. |
635 | + for old_article in ImportedArticle.objects.filter(branch=branch): |
636 | + if old_article.page not in repo.pages and \ |
637 | + old_article.page.changed_by in ['python-api', 'script']: |
638 | + old_article.page.delete() |
639 | + shutil.rmtree(tempdir) |
640 | + |
641 | + # https://stackoverflow.com/questions/33284171/ |
642 | + call_command('cms', 'fix-tree') |
643 | |
644 | |
645 | class Command(BaseCommand): |
646 | |
647 | === added directory 'md_importer/management/importer' |
648 | === added file 'md_importer/management/importer/__init__.py' |
649 | === added file 'md_importer/management/importer/article.py' |
650 | --- md_importer/management/importer/article.py 1970-01-01 00:00:00 +0000 |
651 | +++ md_importer/management/importer/article.py 2016-01-06 12:40:31 +0000 |
652 | @@ -0,0 +1,118 @@ |
653 | +from bs4 import BeautifulSoup |
654 | +import codecs |
655 | +import logging |
656 | +import markdown |
657 | +import os |
658 | +import re |
659 | + |
660 | +from .publish import get_or_create_page, slugify |
661 | + |
662 | + |
663 | +class Article: |
664 | + html = None |
665 | + page = None |
666 | + title = "" |
667 | + |
668 | + def __init__(self, fn, write_to): |
669 | + self.fn = fn |
670 | + self.write_to = slugify(self.fn) |
671 | + self.full_url = write_to |
672 | + self.slug = os.path.basename(self.full_url) |
673 | + |
674 | + def read(self): |
675 | + with codecs.open(self.fn, 'r', encoding='utf-8') as f: |
676 | + if self.fn.endswith('.md'): |
677 | + self.html = markdown.markdown( |
678 | + f.read(), |
679 | + output_format='html5', |
680 | + extensions=['pymdownx.github']) |
681 | + elif self.fn.endswith('.html'): |
682 | + self.html = f.read() |
683 | + else: |
684 | + logging.error("Don't know how to interpret '{}'.".format( |
685 | + self.fn)) |
686 | + return False |
687 | + self.title = self._read_title() |
688 | + self._remove_body_and_html_tags() |
689 | + self._use_developer_site_style() |
690 | + return True |
691 | + |
692 | + def _read_title(self): |
693 | + soup = BeautifulSoup(self.html, 'html5lib') |
694 | + if soup.title: |
695 | + return soup.title.text |
696 | + if soup.h1: |
697 | + return soup.h1.text |
698 | + return slugify(self.fn).replace('-', ' ').title() |
699 | + |
700 | + def _remove_body_and_html_tags(self): |
701 | + self.html = re.sub(r"<html>\n\s<body>\n", "", self.html, |
702 | + flags=re.MULTILINE) |
703 | + self.html = re.sub(r"\s<\/body>\n<\/html>", "", self.html, |
704 | + flags=re.MULTILINE) |
705 | + |
706 | + def _use_developer_site_style(self): |
707 | + begin = (u"<div class=\"row no-border\">" |
708 | + "\n<div class=\"eight-col\">\n") |
709 | + end = u"</div>\n</div>" |
710 | + self.html = begin + self.html + end |
711 | + self.html = self.html.replace( |
712 | + "<pre><code>", |
713 | + "</div><div class=\"twelve-col\"><pre><code>") |
714 | + self.html = self.html.replace( |
715 | + "</code></pre>", |
716 | + "</code></pre></div><div class=\"eight-col\">") |
717 | + |
718 | + def replace_links(self, titles, url_map): |
719 | + soup = BeautifulSoup(self.html, 'html5lib') |
720 | + for link in soup.find_all('a'): |
721 | + for title in titles: |
722 | + if link.attrs['href'] == os.path.basename(title): |
723 | + link.attrs['href'] = url_map[title].full_url |
724 | + self.html = soup.prettify() |
725 | + |
726 | + def add_to_db(self): |
727 | + '''Publishes pages in their branch alias namespace.''' |
728 | + self.page = get_or_create_page( |
729 | + title=self.title, full_url=self.full_url, menu_title=self.title, |
730 | + html=self.html) |
731 | + self.full_url = self.page.get_absolute_url() |
732 | + |
733 | + |
734 | +class SnappyArticle(Article): |
735 | + release_alias = None |
736 | + |
737 | + def read(self): |
738 | + if not Article.read(self): |
739 | + return False |
740 | + self.release_alias = re.findall(r'snappy/guides/(\S+?)/\S+?', |
741 | + self.full_url)[0] |
742 | + self._make_snappy_mods() |
743 | + return True |
744 | + |
745 | + def _make_snappy_mods(self): |
746 | + # Make sure the reader knows which documentation she is browsing |
747 | + if self.release_alias != 'current': |
748 | + before = (u"<div class=\"row no-border\">\n" |
749 | + "<div class=\"eight-col\">\n") |
750 | + after = (u"<div class=\"row no-border\">\n" |
751 | + "<div class=\"box pull-three three-col\">" |
752 | + "<p>You are browsing the Snappy <code>%s</code> " |
753 | + "documentation.</p>" |
754 | + "<p><a href=\"/snappy/guides/current/%s\">" |
755 | + "Back to the latest stable release ›" |
756 | + "</a></p></div>\n" |
757 | + "<div class=\"eight-col\">\n") % (self.release_alias, |
758 | + self.slug, ) |
759 | + self.html = self.html.replace(before, after) |
760 | + |
761 | + def add_to_db(self): |
762 | + if self.release_alias == "current": |
763 | + # Add a guides/<page> redirect to guides/current/<page> |
764 | + get_or_create_page( |
765 | + title=self.title, |
766 | + full_url=self.full_url.replace('/current', ''), |
767 | + redirect="/snappy/guides/current/{}".format(self.slug)) |
768 | + else: |
769 | + self.title += " (%s)" % (self.release_alias,) |
770 | + Article.add_to_db(self) |
771 | |
772 | === added file 'md_importer/management/importer/publish.py' |
773 | --- md_importer/management/importer/publish.py 1970-01-01 00:00:00 +0000 |
774 | +++ md_importer/management/importer/publish.py 2016-01-06 12:40:31 +0000 |
775 | @@ -0,0 +1,64 @@ |
776 | +from cms.api import create_page, add_plugin |
777 | +from cms.models import Title |
778 | + |
779 | +import logging |
780 | +import os |
781 | +import sys |
782 | + |
783 | +# XXX: use this once we RawHTML plugins don't strip comments (LP: #1523925) |
784 | +START_TEXT = """ |
785 | +<!-- |
786 | +branch id: {} |
787 | + |
788 | +THIS PAGE IS AUTOMATICALLY UPDATED. |
789 | +DON'T EDIT IT - CHANGES WILL BE OVERWRITTEN. |
790 | +--> |
791 | +""" |
792 | + |
793 | + |
794 | +def slugify(filename): |
795 | + return os.path.basename(filename).replace('.md', '').replace('.html', '') |
796 | + |
797 | + |
798 | +def get_or_create_page(title, full_url, menu_title=None, |
799 | + in_navigation=True, redirect=None, html=None): |
800 | + # First check if pages already exist. |
801 | + if full_url.startswith('/'): |
802 | + full_url = full_url[1:] |
803 | + pages = Title.objects.select_related('page').filter( |
804 | + path__regex=full_url).filter(publisher_is_draft=True) |
805 | + if pages: |
806 | + page = pages[0].page |
807 | + page.title = title |
808 | + page.publisher_is_draft = True |
809 | + page.menu_title = menu_title |
810 | + page.in_navigation = in_navigation |
811 | + page.redirect = redirect |
812 | + if html: |
813 | + # We create the page, so we know there's just one placeholder |
814 | + placeholder = page.placeholders.all()[0] |
815 | + if placeholder.get_plugins(): |
816 | + plugin = placeholder.get_plugins()[0].get_plugin_instance()[0] |
817 | + plugin.body = html |
818 | + plugin.save() |
819 | + else: |
820 | + add_plugin(placeholder, 'RawHtmlPlugin', 'en', body=html) |
821 | + else: |
822 | + parent_pages = Title.objects.select_related('page').filter( |
823 | + path__regex=os.path.dirname(full_url)).filter( |
824 | + publisher_is_draft=True) |
825 | + if not parent_pages: |
826 | + logging.error('Parent {} not found.'.format( |
827 | + os.path.dirname(full_url))) |
828 | + sys.exit(1) |
829 | + parent = parent_pages[0].page |
830 | + |
831 | + slug = os.path.basename(full_url) |
832 | + page = create_page( |
833 | + title, "default.html", "en", slug=slug, parent=parent, |
834 | + menu_title=menu_title, in_navigation=in_navigation, |
835 | + position="last-child", redirect=redirect) |
836 | + if html: |
837 | + placeholder = page.placeholders.get() |
838 | + add_plugin(placeholder, 'RawHtmlPlugin', 'en', body=html) |
839 | + return page |
840 | |
841 | === added file 'md_importer/management/importer/repo.py' |
842 | --- md_importer/management/importer/repo.py 1970-01-01 00:00:00 +0000 |
843 | +++ md_importer/management/importer/repo.py 2016-01-06 12:40:31 +0000 |
844 | @@ -0,0 +1,168 @@ |
845 | +from .article import Article, SnappyArticle |
846 | +from .publish import get_or_create_page, slugify |
847 | +from .source import SourceCode |
848 | + |
849 | +from cms.api import publish_pages |
850 | + |
851 | +import glob |
852 | +import logging |
853 | +import os |
854 | +import shutil |
855 | + |
856 | + |
857 | +def create_repo(tempdir, origin, branch_name, post_checkout_command): |
858 | + if os.path.exists(origin): |
859 | + if 'snappy' in origin: |
860 | + repo_class = SnappyRepo |
861 | + else: |
862 | + repo_class = Repo |
863 | + else: |
864 | + if origin.startswith('lp:snappy') or \ |
865 | + 'snappy' in origin.split(':')[1].split('.git')[0].split('/'): |
866 | + repo_class = SnappyRepo |
867 | + else: |
868 | + repo_class = Repo |
869 | + return repo_class(tempdir, origin, branch_name, post_checkout_command) |
870 | + |
871 | + |
872 | +class Repo: |
873 | + titles = {} |
874 | + url_map = {} |
875 | + index_doc_url = None |
876 | + index_page = None |
877 | + release_alias = None |
878 | + directives = [] |
879 | + imported_articles = [] |
880 | + # On top of the pages in imported_articles this also includes index_page |
881 | + pages = [] |
882 | + |
883 | + def __init__(self, tempdir, origin, branch_name, post_checkout_command): |
884 | + self.origin = origin |
885 | + self.branch_name = branch_name |
886 | + self.post_checkout_command = post_checkout_command |
887 | + branch_nick = os.path.basename(self.origin.replace('.git', '')) |
888 | + self.checkout_location = os.path.join( |
889 | + tempdir, branch_nick) |
890 | + self.index_doc_title = branch_nick |
891 | + self.article_class = Article |
892 | + |
893 | + # Only used to speed up tests - allows reusing same object without |
894 | + # having to redownload the source again |
895 | + def reset(self): |
896 | + self.article_class = Article |
897 | + self.directives = [] |
898 | + self.imported_articles = [] |
899 | + |
900 | + def get(self): |
901 | + sourcecode = SourceCode(self.origin, self.checkout_location, |
902 | + self.branch_name, self.post_checkout_command) |
903 | + if sourcecode.get() != 0: |
904 | + logging.error( |
905 | + 'Could not check out branch "{}".'.format(self.origin)) |
906 | + shutil.rmtree(self.checkout_location) |
907 | + return 1 |
908 | + return 0 |
909 | + |
910 | + def add_directive(self, import_from, write_to): |
911 | + self.directives += [ |
912 | + { |
913 | + 'import_from': os.path.join(self.checkout_location, |
914 | + import_from), |
915 | + 'write_to': write_to |
916 | + } |
917 | + ] |
918 | + |
919 | + def execute_import_directives(self): |
920 | + import_list = [] |
921 | + # Import single files first |
922 | + for directive in [d for d in self.directives |
923 | + if os.path.isfile(d['import_from'])]: |
924 | + import_list += [ |
925 | + (directive['import_from'], directive['write_to']) |
926 | + ] |
927 | + # Import directories next |
928 | + for directive in [d for d in self.directives |
929 | + if os.path.isdir(d['import_from'])]: |
930 | + for fn in glob.glob('{}/*'.format(directive['import_from'])): |
931 | + if fn not in [a[0] for a in import_list]: |
932 | + import_list += [ |
933 | + (fn, os.path.join(directive['write_to'], slugify(fn))) |
934 | + ] |
935 | + # If we import into a namespace and don't have an index doc, |
936 | + # we need to write one. |
937 | + if directive['write_to'] not in [x[1] for x in import_list]: |
938 | + self.index_doc_url = directive['write_to'] |
939 | + if self.index_doc_url: |
940 | + self._create_fake_index_page() |
941 | + # The actual import |
942 | + for entry in import_list: |
943 | + article = self._read_article(entry[0], entry[1]) |
944 | + if article: |
945 | + self.imported_articles += [article] |
946 | + self.titles[article.fn] = article.title |
947 | + self.url_map[article.fn] = article |
948 | + if self.index_doc_url: |
949 | + self._write_fake_index_doc() |
950 | + |
951 | + def _read_article(self, fn, write_to): |
952 | + article = self.article_class(fn, write_to) |
953 | + if article.read(): |
954 | + return article |
955 | + return None |
956 | + |
957 | + def publish(self): |
958 | + for article in self.imported_articles: |
959 | + article.add_to_db() |
960 | + article.replace_links(self.titles, self.url_map) |
961 | + self.pages = [article.page for article in self.imported_articles] |
962 | + if self.index_page: |
963 | + self.pages.extend([self.index_page]) |
964 | + publish_pages(self.pages) |
965 | + |
966 | + def _create_fake_index_page(self): |
967 | + '''Creates a fake index page at the top of the branches |
968 | + docs namespace.''' |
969 | + |
970 | + if self.index_doc_url.endswith('current'): |
971 | + redirect = '/snappy/guides' |
972 | + else: |
973 | + redirect = None |
974 | + self.index_page = get_or_create_page( |
975 | + title=self.index_doc_title, full_url=self.index_doc_url, |
976 | + in_navigation=False, redirect=redirect, html='', |
977 | + menu_title=None) |
978 | + |
979 | + def _write_fake_index_doc(self): |
980 | + list_pages = '' |
981 | + for article in [a for a |
982 | + in self.imported_articles |
983 | + if a.full_url.startswith(self.index_doc_url)]: |
984 | + list_pages += '<li><a href=\"{}\">{}</a></li>'.format( |
985 | + os.path.basename(article.full_url), article.title) |
986 | + self.index_page.html = ( |
987 | + u'<div class=\"row\"><div class=\"eight-col\">\n' |
988 | + '<p>This section contains documentation for the ' |
989 | + '<code>{}</code> Snappy branch.</p>' |
990 | + '<p><ul class=\"list-ubuntu\">{}</ul></p>\n' |
991 | + '<p>Auto-imported from <a ' |
992 | + 'href=\"{}\">{}</a>.</p>\n' |
993 | + '</div></div>'.format(self.release_alias, list_pages, |
994 | + self.origin, self.origin)) |
995 | + |
996 | + |
997 | +class SnappyRepo(Repo): |
998 | + def __init__(self, tempdir, origin, branch_name, post_checkout_command): |
999 | + Repo.__init__(self, tempdir, origin, branch_name, |
1000 | + post_checkout_command) |
1001 | + self.article_class = SnappyArticle |
1002 | + self.index_doc_title = 'Snappy documentation' |
1003 | + |
1004 | + def _create_fake_index_page(self): |
1005 | + self.release_alias = os.path.basename(self.index_doc_url) |
1006 | + if not self.index_doc_url.endswith('current'): |
1007 | + self.index_doc_title += ' ({})'.format(self.release_alias) |
1008 | + Repo._create_fake_index_page(self) |
1009 | + |
1010 | + def reset(self): |
1011 | + Repo.reset(self) |
1012 | + self.article_class = SnappyArticle |
1013 | |
1014 | === added file 'md_importer/management/importer/source.py' |
1015 | --- md_importer/management/importer/source.py 1970-01-01 00:00:00 +0000 |
1016 | +++ md_importer/management/importer/source.py 2016-01-06 12:40:31 +0000 |
1017 | @@ -0,0 +1,60 @@ |
1018 | +import logging |
1019 | +import os |
1020 | +import shutil |
1021 | +import subprocess |
1022 | + |
1023 | + |
1024 | +class SourceCode(): |
1025 | + def __init__(self, origin, checkout_location, branch_name, |
1026 | + post_checkout_command): |
1027 | + self.origin = origin |
1028 | + self.checkout_location = checkout_location |
1029 | + self.branch_name = branch_name |
1030 | + self.post_checkout_command = post_checkout_command |
1031 | + |
1032 | + def get(self): |
1033 | + res = self._get_branch() |
1034 | + if res == 0 and self.post_checkout_command: |
1035 | + res = self._post_checkout() |
1036 | + return res |
1037 | + return res |
1038 | + |
1039 | + def _get_branch(self): |
1040 | + if os.path.exists(self.origin): |
1041 | + shutil.copytree( |
1042 | + self.origin, |
1043 | + self.checkout_location) |
1044 | + return 0 |
1045 | + if self.origin.startswith('lp:') and \ |
1046 | + os.path.exists('/usr/bin/bzr'): |
1047 | + return subprocess.call([ |
1048 | + 'bzr', 'checkout', '--lightweight', self.origin, |
1049 | + self.checkout_location]) |
1050 | + if self.origin.startswith('https://github.com') and \ |
1051 | + self.origin.endswith('.git') and \ |
1052 | + os.path.exists('/usr/bin/git'): |
1053 | + retcode = subprocess.call([ |
1054 | + 'git', 'clone', '--quiet', self.origin, |
1055 | + self.checkout_location]) |
1056 | + if retcode == 0 and self.branch_name: |
1057 | + pwd = os.getcwd() |
1058 | + os.chdir(self.checkout_location) |
1059 | + retcode = subprocess.call(['git', 'checkout', '--quiet', |
1060 | + self.branch_name]) |
1061 | + os.chdir(pwd) |
1062 | + return retcode |
1063 | + logging.error( |
1064 | + 'Branch format "{}" not understood.'.format(self.origin)) |
1065 | + return 1 |
1066 | + |
1067 | + def _post_checkout(self): |
1068 | + pwd = os.getcwd() |
1069 | + os.chdir(self.checkout_location) |
1070 | + process = subprocess.Popen(self.post_checkout_command.split(), |
1071 | + stdout=subprocess.PIPE) |
1072 | + (out, err) = process.communicate() |
1073 | + retcode = process.wait() |
1074 | + os.chdir(pwd) |
1075 | + if retcode != 0: |
1076 | + logging.error(out) |
1077 | + return retcode |
1078 | |
1079 | === added directory 'md_importer/migrations' |
1080 | === added file 'md_importer/migrations/0001_initial.py' |
1081 | --- md_importer/migrations/0001_initial.py 1970-01-01 00:00:00 +0000 |
1082 | +++ md_importer/migrations/0001_initial.py 2016-01-06 12:40:31 +0000 |
1083 | @@ -0,0 +1,46 @@ |
1084 | +# -*- coding: utf-8 -*- |
1085 | +from __future__ import unicode_literals |
1086 | + |
1087 | +from django.db import migrations, models |
1088 | + |
1089 | + |
1090 | +class Migration(migrations.Migration): |
1091 | + |
1092 | + dependencies = [ |
1093 | + ('cms', '0013_urlconfrevision'), |
1094 | + ] |
1095 | + |
1096 | + operations = [ |
1097 | + migrations.CreateModel( |
1098 | + name='ExternalDocsBranch', |
1099 | + fields=[ |
1100 | + ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)), |
1101 | + ('origin', models.CharField(help_text='External branch location, ie: lp:snappy/15.04 or https://github.com/ubuntu-core/snappy.git', max_length=200)), |
1102 | + ('branch_name', models.CharField(help_text='For use with git branches, ie: "master" or "15.04" or "1.x".', max_length=200, blank=True)), |
1103 | + ('post_checkout_command', models.CharField(help_text='Command to run after checkout of the branch.', max_length=100, blank=True)), |
1104 | + ('active', models.BooleanField(default=True)), |
1105 | + ], |
1106 | + options={ |
1107 | + 'verbose_name': 'external docs branch', |
1108 | + 'verbose_name_plural': 'external docs branches', |
1109 | + }, |
1110 | + ), |
1111 | + migrations.CreateModel( |
1112 | + name='ExternalDocsBranchImportDirective', |
1113 | + fields=[ |
1114 | + ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)), |
1115 | + ('import_from', models.CharField(help_text='File or directory to import from the branch. Ie: "docs/intro.md" (file) or "docs" (complete directory), etc.', max_length=150)), |
1116 | + ('write_to', models.CharField(help_text='Article URL (for a specific file) or article namespace for a directory or a set of files.', max_length=150)), |
1117 | + ('external_docs_branch', models.ForeignKey(to='md_importer.ExternalDocsBranch')), |
1118 | + ], |
1119 | + ), |
1120 | + migrations.CreateModel( |
1121 | + name='ImportedArticle', |
1122 | + fields=[ |
1123 | + ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)), |
1124 | + ('last_import', models.DateTimeField(help_text='Datetime of last import.', verbose_name='Datetime')), |
1125 | + ('branch', models.ForeignKey(to='md_importer.ExternalDocsBranch')), |
1126 | + ('page', models.ForeignKey(to='cms.Page')), |
1127 | + ], |
1128 | + ), |
1129 | + ] |
1130 | |
1131 | === added file 'md_importer/migrations/__init__.py' |
1132 | === added file 'md_importer/models.py' |
1133 | --- md_importer/models.py 1970-01-01 00:00:00 +0000 |
1134 | +++ md_importer/models.py 2016-01-06 12:40:31 +0000 |
1135 | @@ -0,0 +1,54 @@ |
1136 | +from django.db import models |
1137 | +from django.utils.translation import ugettext_lazy as _ |
1138 | + |
1139 | +from cms.models import Page |
1140 | + |
1141 | + |
1142 | +class ExternalDocsBranch(models.Model): |
1143 | + origin = models.CharField( |
1144 | + max_length=200, |
1145 | + help_text=_('External branch location, ie: lp:snappy/15.04 or ' |
1146 | + 'https://github.com/ubuntu-core/snappy.git')) |
1147 | + branch_name = models.CharField( |
1148 | + max_length=200, |
1149 | + help_text=_('For use with git branches, ie: "master" or "15.04" ' |
1150 | + 'or "1.x".'), |
1151 | + blank=True) |
1152 | + post_checkout_command = models.CharField( |
1153 | + max_length=100, |
1154 | + help_text=_('Command to run after checkout of the branch.'), |
1155 | + blank=True) |
1156 | + active = models.BooleanField(default=True) |
1157 | + |
1158 | + def __str__(self): |
1159 | + if self.branch_name: |
1160 | + return "{} - {}".format(self.origin, self.branch_name) |
1161 | + return "{}".format(self.origin) |
1162 | + |
1163 | + class Meta: |
1164 | + verbose_name = "external docs branch" |
1165 | + verbose_name_plural = "external docs branches" |
1166 | + |
1167 | + |
1168 | +class ExternalDocsBranchImportDirective(models.Model): |
1169 | + external_docs_branch = models.ForeignKey(ExternalDocsBranch) |
1170 | + import_from = models.CharField( |
1171 | + max_length=150, |
1172 | + help_text=_('File or directory to import from the branch. ' |
1173 | + 'Ie: "docs/intro.md" (file) or ' |
1174 | + '"docs" (complete directory), etc.')) |
1175 | + write_to = models.CharField( |
1176 | + max_length=150, |
1177 | + help_text=_('Article URL (for a specific file) or article namespace ' |
1178 | + 'for a directory or a set of files.')) |
1179 | + |
1180 | + def __str__(self): |
1181 | + return "{} -- {}".format(self.external_docs_branch, |
1182 | + self.import_from) |
1183 | + |
1184 | + |
1185 | +class ImportedArticle(models.Model): |
1186 | + page = models.ForeignKey(Page) |
1187 | + branch = models.ForeignKey(ExternalDocsBranch) |
1188 | + last_import = models.DateTimeField( |
1189 | + _('Datetime'), help_text=_('Datetime of last import.')) |
1190 | |
1191 | === added directory 'md_importer/tests' |
1192 | === added file 'md_importer/tests/__init__.py' |
1193 | --- md_importer/tests/__init__.py 1970-01-01 00:00:00 +0000 |
1194 | +++ md_importer/tests/__init__.py 2016-01-06 12:40:31 +0000 |
1195 | @@ -0,0 +1,8 @@ |
1196 | +import shutil |
1197 | + |
1198 | +from utils import SnapcraftTestRepo, SnappyTestRepo |
1199 | + |
1200 | + |
1201 | +def tearDownModule(): |
1202 | + shutil.rmtree(SnapcraftTestRepo().tempdir) |
1203 | + shutil.rmtree(SnappyTestRepo().tempdir) |
1204 | |
1205 | === added directory 'md_importer/tests/data' |
1206 | === added directory 'md_importer/tests/data/link-test' |
1207 | === added file 'md_importer/tests/data/link-test/file1.md' |
1208 | --- md_importer/tests/data/link-test/file1.md 1970-01-01 00:00:00 +0000 |
1209 | +++ md_importer/tests/data/link-test/file1.md 2016-01-06 12:40:31 +0000 |
1210 | @@ -0,0 +1,5 @@ |
1211 | +# Test |
1212 | + |
1213 | +This is a link to [the file2 file](file2.md). |
1214 | + |
1215 | +That's it. |
1216 | |
1217 | === added file 'md_importer/tests/data/link-test/file2.md' |
1218 | --- md_importer/tests/data/link-test/file2.md 1970-01-01 00:00:00 +0000 |
1219 | +++ md_importer/tests/data/link-test/file2.md 2016-01-06 12:40:31 +0000 |
1220 | @@ -0,0 +1,3 @@ |
1221 | +# File 2 |
1222 | + |
1223 | +Here's just some text. |
1224 | |
1225 | === added file 'md_importer/tests/test_branch_fetch.py' |
1226 | --- md_importer/tests/test_branch_fetch.py 1970-01-01 00:00:00 +0000 |
1227 | +++ md_importer/tests/test_branch_fetch.py 2016-01-06 12:40:31 +0000 |
1228 | @@ -0,0 +1,42 @@ |
1229 | +import os |
1230 | +import shutil |
1231 | +import tempfile |
1232 | + |
1233 | +from django.test import TestCase |
1234 | + |
1235 | +from ..management.importer.repo import create_repo, Repo |
1236 | +from .utils import SnapcraftTestRepo |
1237 | + |
1238 | + |
1239 | +class TestBranchFetch(TestCase): |
1240 | + def test_git_fetch(self): |
1241 | + snapcraft = SnapcraftTestRepo() |
1242 | + snapcraft.repo.reset() |
1243 | + self.assertEqual(snapcraft.fetch_retcode, 0) |
1244 | + self.assertTrue(isinstance(snapcraft.repo, Repo)) |
1245 | + |
1246 | + def test_bzr_fetch(self): |
1247 | + tempdir = tempfile.mkdtemp() |
1248 | + repo = create_repo( |
1249 | + tempdir, |
1250 | + 'lp:snapcraft', # outdated, but should work for testing |
1251 | + '', |
1252 | + '') |
1253 | + ret = repo.get() |
1254 | + shutil.rmtree(tempdir) |
1255 | + self.assertEqual(ret, 0) |
1256 | + self.assertTrue(isinstance(repo, Repo)) |
1257 | + |
1258 | + def test_post_checkout_command(self): |
1259 | + tempdir = tempfile.mkdtemp() |
1260 | + repo = create_repo( |
1261 | + tempdir, |
1262 | + 'lp:snapcraft', |
1263 | + '', |
1264 | + 'touch something.html' |
1265 | + ) |
1266 | + ret = repo.get() |
1267 | + self.assertEqual(ret, 0) |
1268 | + self.assertTrue(os.path.exists( |
1269 | + os.path.join(repo.checkout_location, 'something.html'))) |
1270 | + shutil.rmtree(tempdir) |
1271 | |
1272 | === added file 'md_importer/tests/test_branch_import.py' |
1273 | --- md_importer/tests/test_branch_import.py 1970-01-01 00:00:00 +0000 |
1274 | +++ md_importer/tests/test_branch_import.py 2016-01-06 12:40:31 +0000 |
1275 | @@ -0,0 +1,67 @@ |
1276 | +from django.test import TestCase |
1277 | + |
1278 | +from cms.api import publish_pages |
1279 | +from cms.models import Page |
1280 | + |
1281 | +from ..management.importer.article import Article |
1282 | +from .utils import ( |
1283 | + db_create_home_page, |
1284 | + db_empty_page_list, |
1285 | + SnapcraftTestRepo, |
1286 | +) |
1287 | + |
1288 | + |
1289 | +class TestBranchImport(TestCase): |
1290 | + def test_1dir_import(self): |
1291 | + db_empty_page_list() |
1292 | + db_create_home_page() |
1293 | + snapcraft = SnapcraftTestRepo() |
1294 | + snapcraft.repo.reset() |
1295 | + snapcraft.repo.add_directive('docs', '/') |
1296 | + snapcraft.repo.execute_import_directives() |
1297 | + snapcraft.repo.publish() |
1298 | + pages = Page.objects.all() |
1299 | + self.assertGreater(len(pages), 3) |
1300 | + for article in snapcraft.repo.imported_articles: |
1301 | + self.assertTrue(isinstance(article, Article)) |
1302 | + |
1303 | + def test_1dir_and_2files_import(self): |
1304 | + db_empty_page_list() |
1305 | + db_create_home_page() |
1306 | + snapcraft = SnapcraftTestRepo() |
1307 | + snapcraft.repo.reset() |
1308 | + snapcraft.repo.add_directive('docs', '/') |
1309 | + snapcraft.repo.add_directive('README.md', '/') |
1310 | + snapcraft.repo.add_directive('HACKING.md', '/hacking') |
1311 | + snapcraft.repo.execute_import_directives() |
1312 | + snapcraft.repo.publish() |
1313 | + pages = Page.objects.all() |
1314 | + self.assertGreater(len(pages), 5) |
1315 | + self.assertIn(u'/en/', [p.get_absolute_url() for p in pages]) |
1316 | + self.assertIn(u'/en/hacking/', [p.get_absolute_url() for p in pages]) |
1317 | + |
1318 | + # Check if all importe article has 'home' as parent |
1319 | + def test_articletree_1file_import(self): |
1320 | + db_empty_page_list() |
1321 | + home = db_create_home_page() |
1322 | + publish_pages([[home]]) |
1323 | + snapcraft = SnapcraftTestRepo() |
1324 | + snapcraft.repo.reset() |
1325 | + snapcraft.repo.add_directive('README.md', '/readme') |
1326 | + snapcraft.repo.execute_import_directives() |
1327 | + snapcraft.repo.publish() |
1328 | + self.assertEqual(Page.objects.count(), 1+1) # readme + home |
1329 | + self.assertTrue(snapcraft.repo.pages[0].parent == home) |
1330 | + |
1331 | + # Check if all imported articles have 'home' as parent |
1332 | + def test_articletree_1dir_import(self): |
1333 | + db_empty_page_list() |
1334 | + home = db_create_home_page() |
1335 | + snapcraft = SnapcraftTestRepo() |
1336 | + snapcraft.repo.reset() |
1337 | + snapcraft.repo.add_directive('docs', '/') |
1338 | + snapcraft.repo.execute_import_directives() |
1339 | + snapcraft.repo.publish() |
1340 | + for page in Page.objects.filter(publisher_is_draft=False): |
1341 | + if page.parent is not None: |
1342 | + self.assertEqual(page.parent_id, home.id) |
1343 | |
1344 | === added file 'md_importer/tests/test_link_rewrite.py' |
1345 | --- md_importer/tests/test_link_rewrite.py 1970-01-01 00:00:00 +0000 |
1346 | +++ md_importer/tests/test_link_rewrite.py 2016-01-06 12:40:31 +0000 |
1347 | @@ -0,0 +1,44 @@ |
1348 | +from bs4 import BeautifulSoup |
1349 | +import os |
1350 | +import shutil |
1351 | +import tempfile |
1352 | + |
1353 | +from django.http.response import HttpResponseNotFound |
1354 | +from django.test import TestCase, Client |
1355 | + |
1356 | +from cms.models import Page |
1357 | + |
1358 | +from ..management.importer.article import Article |
1359 | +from ..management.importer.repo import create_repo |
1360 | +from .utils import ( |
1361 | + db_create_home_page, |
1362 | + db_empty_page_list, |
1363 | +) |
1364 | + |
1365 | + |
1366 | +class TestLinkRewrite(TestCase): |
1367 | + def test_simple_case(self): |
1368 | + db_empty_page_list() |
1369 | + db_create_home_page() |
1370 | + tempdir = tempfile.mkdtemp() |
1371 | + repo = create_repo( |
1372 | + tempdir, |
1373 | + os.path.join(os.path.dirname(__file__), 'data/link-test'), |
1374 | + '', |
1375 | + '') |
1376 | + self.assertEqual(repo.get(), 0) |
1377 | + repo.add_directive('', '/') |
1378 | + repo.execute_import_directives() |
1379 | + repo.publish() |
1380 | + pages = Page.objects.all() |
1381 | + self.assertEqual(pages.count(), 1+2) # Home + 2 articles |
1382 | + c = Client() |
1383 | + for article in repo.imported_articles: |
1384 | + self.assertTrue(isinstance(article, Article)) |
1385 | + soup = BeautifulSoup(article.html, 'html5lib') |
1386 | + for link in soup.find_all('a'): |
1387 | + if not link.has_attr('class') or \ |
1388 | + 'headeranchor-link' not in link.attrs['class']: |
1389 | + res = c.get(link.attrs['href']) |
1390 | + self.assertNotIsInstance(res, HttpResponseNotFound) |
1391 | + shutil.rmtree(tempdir) |
1392 | |
1393 | === added file 'md_importer/tests/test_snappy_import.py' |
1394 | --- md_importer/tests/test_snappy_import.py 1970-01-01 00:00:00 +0000 |
1395 | +++ md_importer/tests/test_snappy_import.py 2016-01-06 12:40:31 +0000 |
1396 | @@ -0,0 +1,70 @@ |
1397 | +from django.test import TestCase |
1398 | + |
1399 | +from cms.api import publish_pages |
1400 | +from cms.models import Page |
1401 | + |
1402 | +from ..management.importer.repo import SnappyRepo |
1403 | +from ..management.importer.article import SnappyArticle |
1404 | +from .utils import ( |
1405 | + db_add_empty_page, |
1406 | + db_create_home_page, |
1407 | + db_empty_page_list, |
1408 | + SnappyTestRepo, |
1409 | +) |
1410 | + |
1411 | + |
1412 | +class TestSnappyImport(TestCase): |
1413 | + def test_snappy_devel_import(self): |
1414 | + db_empty_page_list() |
1415 | + home = db_create_home_page() |
1416 | + snappy_page = db_add_empty_page('Snappy', home) |
1417 | + guides = db_add_empty_page('Guides', snappy_page) |
1418 | + publish_pages([home, snappy_page, guides]) |
1419 | + snappy = SnappyTestRepo() |
1420 | + snappy.repo.reset() |
1421 | + self.assertEqual(snappy.fetch_retcode, 0) |
1422 | + self.assertTrue(isinstance(snappy.repo, SnappyRepo)) |
1423 | + snappy.repo.add_directive('docs', '/snappy/guides/devel') |
1424 | + snappy.repo.execute_import_directives() |
1425 | + snappy.repo.publish() |
1426 | + for article in snappy.repo.imported_articles: |
1427 | + self.assertTrue(isinstance(article, SnappyArticle)) |
1428 | + self.assertGreater(len(snappy.repo.pages), 0) |
1429 | + devel = Page.objects.filter(parent=guides.get_public_object()) |
1430 | + self.assertEqual(devel.count(), 1) |
1431 | + for page in Page.objects.filter(publisher_is_draft=False): |
1432 | + if page not in [home, snappy_page, guides, devel[0]]: |
1433 | + self.assertEqual(page.parent, devel[0]) |
1434 | + |
1435 | + def test_snappy_current_import(self): |
1436 | + db_empty_page_list() |
1437 | + home = db_create_home_page() |
1438 | + snappy_page = db_add_empty_page('Snappy', home) |
1439 | + guides = db_add_empty_page('Guides', snappy_page) |
1440 | + publish_pages([home, snappy_page, guides]) |
1441 | + snappy = SnappyTestRepo() |
1442 | + snappy.repo.reset() |
1443 | + self.assertTrue(isinstance(snappy.repo, SnappyRepo)) |
1444 | + snappy.repo.add_directive('docs', '/snappy/guides/current') |
1445 | + snappy.repo.execute_import_directives() |
1446 | + snappy.repo.publish() |
1447 | + number_of_articles = len(snappy.repo.imported_articles) |
1448 | + for article in snappy.repo.imported_articles: |
1449 | + self.assertTrue(isinstance(article, SnappyArticle)) |
1450 | + self.assertGreater(number_of_articles, 0) |
1451 | + pages = Page.objects.all() |
1452 | + current_search = [ |
1453 | + a for a in pages |
1454 | + if a.get_slug('current') and |
1455 | + a.get_absolute_url().endswith('snappy/guides/current/')] |
1456 | + self.assertEqual(len(current_search), 1) |
1457 | + current = current_search[0] |
1458 | + nav_pages = [home, snappy_page, guides, current] |
1459 | + # 1 imported article, 1 redirect |
1460 | + self.assertEqual( |
1461 | + number_of_articles*2, pages.count()-len(nav_pages)) |
1462 | + for page in [a for a in pages if a not in nav_pages]: |
1463 | + if page.get_redirect('en'): |
1464 | + self.assertEqual(page.parent, guides) |
1465 | + else: |
1466 | + self.assertEqual(page.parent, current) |
1467 | |
1468 | === added file 'md_importer/tests/test_utils.py' |
1469 | --- md_importer/tests/test_utils.py 1970-01-01 00:00:00 +0000 |
1470 | +++ md_importer/tests/test_utils.py 2016-01-06 12:40:31 +0000 |
1471 | @@ -0,0 +1,33 @@ |
1472 | +from django.test import TestCase |
1473 | + |
1474 | +from cms.api import publish_pages |
1475 | +from cms.models import Page |
1476 | + |
1477 | +from .utils import ( |
1478 | + db_add_empty_page, |
1479 | + db_create_home_page, |
1480 | + db_empty_page_list, |
1481 | +) |
1482 | + |
1483 | + |
1484 | +class PageDBActivities(TestCase): |
1485 | + def test_empty_page_list(self): |
1486 | + db_empty_page_list() |
1487 | + self.assertEqual(Page.objects.count(), 0) |
1488 | + |
1489 | + def test_create_home_page(self): |
1490 | + db_empty_page_list() |
1491 | + home = db_create_home_page() |
1492 | + publish_pages([home]) |
1493 | + self.assertNotEqual(home, None) |
1494 | + self.assertEqual(Page.objects.count(), 1) |
1495 | + |
1496 | + def test_simple_articletree(self): |
1497 | + db_empty_page_list() |
1498 | + home = db_create_home_page() |
1499 | + snappy = db_add_empty_page('Snappy', home) |
1500 | + guides = db_add_empty_page('Guides', snappy) |
1501 | + publish_pages([home, snappy, guides]) |
1502 | + self.assertEqual(Page.objects.count(), 3) |
1503 | + self.assertEqual(guides.parent, snappy) |
1504 | + self.assertEqual(snappy.parent, home) |
1505 | |
1506 | === added file 'md_importer/tests/utils.py' |
1507 | --- md_importer/tests/utils.py 1970-01-01 00:00:00 +0000 |
1508 | +++ md_importer/tests/utils.py 2016-01-06 12:40:31 +0000 |
1509 | @@ -0,0 +1,67 @@ |
1510 | +import shutil |
1511 | +import tempfile |
1512 | + |
1513 | +from django.utils.text import slugify |
1514 | + |
1515 | +from cms.api import create_page |
1516 | +from cms.models import Page |
1517 | + |
1518 | +from ..management.importer.repo import create_repo |
1519 | + |
1520 | + |
1521 | +class Singleton(type): |
1522 | + _instances = {} |
1523 | + |
1524 | + def __call__(cls, *args, **kwargs): |
1525 | + if cls not in cls._instances: |
1526 | + cls._instances[cls] = super(Singleton, cls).__call__(*args, |
1527 | + **kwargs) |
1528 | + return cls._instances[cls] |
1529 | + |
1530 | + |
1531 | +# We are going to re-use this one, so we don't have to checkout the git |
1532 | +# repo all the time. |
1533 | +class SnapcraftTestRepo(): |
1534 | + __metaclass__ = Singleton |
1535 | + |
1536 | + def __init__(self): |
1537 | + self.tempdir = tempfile.mkdtemp() |
1538 | + self.repo = create_repo( |
1539 | + self.tempdir, |
1540 | + 'https://github.com/ubuntu-core/snapcraft.git', |
1541 | + 'master', |
1542 | + '') |
1543 | + self.fetch_retcode = self.repo.get() |
1544 | + |
1545 | + |
1546 | +class SnappyTestRepo(): |
1547 | + __metaclass__ = Singleton |
1548 | + |
1549 | + def __init__(self): |
1550 | + self.tempdir = tempfile.mkdtemp() |
1551 | + self.repo = create_repo( |
1552 | + self.tempdir, |
1553 | + 'https://github.com/ubuntu-core/snappy.git', |
1554 | + 'master', |
1555 | + '') |
1556 | + self.fetch_retcode = self.repo.get() |
1557 | + |
1558 | + |
1559 | +def tearDownModule(): |
1560 | + shutil.rmtree(SnapcraftTestRepo().tempdir) |
1561 | + shutil.rmtree(SnappyTestRepo().tempdir) |
1562 | + |
1563 | + |
1564 | +def db_empty_page_list(): |
1565 | + Page.objects.all().delete() |
1566 | + |
1567 | + |
1568 | +def db_create_home_page(): |
1569 | + home = create_page('Test import', 'default.html', 'en', slug='home') |
1570 | + return home |
1571 | + |
1572 | + |
1573 | +def db_add_empty_page(title, parent): |
1574 | + page = create_page(title, 'default.html', 'en', slug=slugify(title), |
1575 | + parent=parent) |
1576 | + return page |
1577 | |
1578 | === modified file 'requirements.txt' |
1579 | --- requirements.txt 2015-12-11 16:10:30 +0000 |
1580 | +++ requirements.txt 2016-01-06 12:40:31 +0000 |
1581 | @@ -1,4 +1,4 @@ |
1582 | -Django==1.8.7 |
1583 | +Django==1.8.8 |
1584 | django-template-debug==0.3.5 |
1585 | oslo.config==3.1.0 |
1586 | oslo.i18n==3.1.0 |
1587 | @@ -7,6 +7,7 @@ |
1588 | Pillow==2.9.0 |
1589 | cmsplugin-zinnia==0.8 |
1590 | Markdown==2.6.5 |
1591 | +pymdown-extensions==1.0.1 |
1592 | beautifulsoup4==4.4.1 |
1593 | dj-database-url==0.3.0 |
1594 | django-admin-enhancer==1.0.0 |
1595 | @@ -20,8 +21,8 @@ |
1596 | django-meta==0.3.1 |
1597 | django-meta-mixin==0.2.1 |
1598 | django-missing==0.1.15 |
1599 | -django-parler==1.5.1 |
1600 | -django-polymorphic==0.7.2 |
1601 | +django-parler==1.6 |
1602 | +django-polymorphic==0.8.1 |
1603 | django-reversion==1.9.3 |
1604 | django-sekizai==0.9.0 |
1605 | django-swiftstorage==1.1.0 |
1606 | @@ -31,11 +32,11 @@ |
1607 | django-taggit-templatetags==0.2.5 |
1608 | django-templatetag-sugar==1.0 |
1609 | django-xmlrpc==0.1.5 |
1610 | -djangocms-admin-style==1.0.7 |
1611 | +djangocms-admin-style==1.0.8 |
1612 | djangocms-link==1.7.1 |
1613 | djangocms-picture==0.2.0 |
1614 | djangocms-snippet==1.7.1 |
1615 | -djangocms-text-ckeditor==2.8.0 |
1616 | +djangocms-text-ckeditor==2.8.1 |
1617 | djangocms-utils==0.9.5 |
1618 | djangocms-video==0.2.0 |
1619 | python-keystoneclient==1.3.3 |
1620 | @@ -49,4 +50,4 @@ |
1621 | django-pygments==0.1 |
1622 | django-openid-auth==0.7 |
1623 | python-openid==2.2.5 |
1624 | -djangorestframework==3.3.1 |
1625 | +djangorestframework==3.3.2 |
sqlite> select * from developer_ portal_ externaldocsbra nch; /github. com/ubuntu- core/snapcraft. git|1|master| /github. com/ubuntu- core/snapcraft. git|1|1.x| /github. com/ubuntu- core/snappy. git|1|master| /github. com/ubuntu- core/snappy. git|1|15.04| portal_ externaldocsbra nchimportdirect ive; md|snappy/ build-apps/ devel|1 snappy/ build-apps/ devel|1 md|snappy/ build-apps/ |2 snappy/ build-apps/ |2 snappy/ guides/ devel|3 snappy/ guides/ current| 4 md|snappy/ build-apps/ hacking| 1
1|https:/
2|https:/
3|https:/
4|https:/
sqlite> select * from developer_
1|README.
2|docs|
3|README.
4|docs|
5|docs|
6|docs|
7|HACKING.
sqlite>