Ubuntu Developer Portal

Merge lp:~dholbach/developer-ubuntu-com/rework-importer into lp:developer-ubuntu-com

  1. rework-importer
  2. Merge into stable
Proposed by Daniel Holbach on 2015-11-17
Status: Superseded
Proposed branch: lp:~dholbach/developer-ubuntu-com/rework-importer
Merge into: lp:developer-ubuntu-com
Prerequisite: lp:~developer-ubuntu-com-dev/developer-ubuntu-com/1470715
Diff against target: 886 lines (+636/-75) (has conflicts)
12 files modified
TODO (+7/-0)
developer_portal/admin.py (+22/-4)
developer_portal/management/commands/import-external-docs-branches.py (+42/-56)
developer_portal/management/importer/article.py (+126/-0)
developer_portal/management/importer/local_branch.py (+132/-0)
developer_portal/management/importer/publish.py (+60/-0)
developer_portal/management/importer/source.py (+51/-0)
developer_portal/migrations/0002_rework_importer.py (+59/-0)
developer_portal/migrations/0003_track_imported_articles.py (+24/-0)
developer_portal/models.py (+46/-15)
developer_portal/settings.py (+20/-0)
requirements.txt (+47/-0)
Text conflict in developer_portal/admin.py
Text conflict in developer_portal/management/commands/import-external-docs-branches.py
Text conflict in developer_portal/settings.py
Text conflict in requirements.txt
To merge this branch: bzr merge lp:~dholbach/developer-ubuntu-com/rework-importer
Reviewer Review Type Date Requested Status
Ubuntu App Developer site developers 2015-11-17 Pending
Review via email: mp+277722@code.launchpad.net
To post a comment you must log in.
lp:~dholbach/developer-ubuntu-com/rework-importer updated on 2016-01-18
183. By Daniel Holbach on 2015-11-18

make admin interface for ExternalDocsBranch* usable

184. By Daniel Holbach on 2015-11-19

first cut of importer reorg
 - break up into individual files
 - simplify API a bit, make it more easily extendable and testable
 - next steps: implement removal of old articles, fix links

185. By Daniel Holbach on 2015-11-19

we haven't implemented globs yet

186. By Daniel Holbach on 2015-12-01

merged from trunk

187. By Daniel Holbach on 2015-12-02

add TODO

188. By Daniel Holbach on 2015-12-04

merge trunk

189. By Daniel Holbach on 2015-12-04

merged lp:~dholbach/developer-ubuntu-com/django-cms-update

190. By Daniel Holbach on 2015-12-04

some more package updates

191. By Daniel Holbach on 2015-12-04

revert reversion upgrade

192. By Daniel Holbach on 2015-12-04

update migations locations

193. By Daniel Holbach on 2015-12-04

rename branch_origin to origin, add branch_name and add functionatlity so users can specify which branch of a git branch to import form

194. By Daniel Holbach on 2015-12-04

make ExternalDocsBranch.__str__ more revealing

195. By Daniel Holbach on 2015-12-04

update SnappyLocalBranch c'tor to be inline with the added branch_name field

196. By Daniel Holbach on 2015-12-04

make sure we run 'checkout <branch>' in the right directory

197. By Daniel Holbach on 2015-12-07

update member name

198. By Daniel Holbach on 2015-12-07

fix imports, return published articles

199. By Daniel Holbach on 2015-12-08

update TODO

200. By Daniel Holbach on 2015-12-08

make pep8 happy, use .format()

201. By Daniel Holbach on 2015-12-08

allow us to track which pages were imported from which branch

202. By Daniel Holbach on 2015-12-08

make pep8 happy

203. By Daniel Holbach on 2015-12-08

first cut at tracking docs-branch/page in a separate table

204. By Daniel Holbach on 2015-12-09

fix check if directory-import would import already file-imported articles

205. By Daniel Holbach on 2015-12-09

safe datetime of last_import, fix removal condition of old articles

206. By Daniel Holbach on 2015-12-09

remove unused import

207. By Daniel Holbach on 2015-12-09

merge from lp:~developer-ubuntu-com-dev/developer-ubuntu-com/django-1.8-cms-2.3

208. By Daniel Holbach on 2015-12-09

add check if the last change was made by our script

209. By Daniel Holbach on 2015-12-14

merge lp:~developer-ubuntu-com-dev/developer-ubuntu-com/django-1.8-cms-2.3 again

210. By Daniel Holbach on 2015-12-14

make __str__ representations of ExternalDocsBranchImportDirective and ExternalDocsBranch more readable

211. By Daniel Holbach on 2015-12-14

fix syntax error

212. By Daniel Holbach on 2015-12-14

every branch gets its own tempdir

213. By Daniel Holbach on 2015-12-14

help with paths starting with '/'

214. By Daniel Holbach on 2015-12-14

every branch gets its own tempdir

215. By Daniel Holbach on 2015-12-14

add fenced_code markdown extension - only change links if they are in our url_map

216. By Daniel Holbach on 2015-12-14

allow github style markdown

217. By Daniel Holbach on 2015-12-14

remove url_map checks, they are useless

218. By Daniel Holbach on 2015-12-14

update TODO

219. By Daniel Holbach on 2015-12-14

support local repositories - useful for testing

220. By Daniel Holbach on 2015-12-14

add first tests

221. By Daniel Holbach on 2015-12-14

add another test

222. By Daniel Holbach on 2015-12-14

pep8 fix

223. By Daniel Holbach on 2015-12-15

rename tests to be clearer, rewrite tests, so we can reuse a repository we already checked out

224. By Daniel Holbach on 2015-12-15

rename local_branch to repo - makes more sense these days

225. By Daniel Holbach on 2015-12-15

add test for post_checkout_command, fix post_checkout_command logic by chdir()ing correctly, make pep8+pyflakes happy

226. By Daniel Holbach on 2015-12-15

make create_home_page return home, test page db activities

227. By Daniel Holbach on 2015-12-15

check if all imported articles have 'home' as parent

228. By Daniel Holbach on 2015-12-15

add comment, pep8

229. By Daniel Holbach on 2015-12-15

add function to reset a repo - only useful for tests, where we reuse the same repo to avoid redownloading, add test where only one file is imported and we check the article tree

230. By Daniel Holbach on 2015-12-15

filter on publisher_is_draft=True

231. By Daniel Holbach on 2015-12-15

clean up after the test

232. By Daniel Holbach on 2015-12-16

make index_doc_title available everywhere

233. By Daniel Holbach on 2015-12-16

add db_add_empty_page convenience function, add simple articletree test, add test for the import of snappy devel

234. By Daniel Holbach on 2015-12-16

make git quiet

235. By Daniel Holbach on 2015-12-16

add create_repo convenience function

236. By Daniel Holbach on 2015-12-16

use create_repo function, add assertions for SnappyRepo vs Repo instantiation

237. By Daniel Holbach on 2015-12-17

add SnappyTestRepo singleton for re-use, rename variables

238. By Daniel Holbach on 2015-12-17

add assertions if imported articles have the correct type - drop useless tempdir creation

239. By Daniel Holbach on 2015-12-17

function article.get was renamed to article.read ages ago...

240. By Daniel Holbach on 2015-12-17

add test for 'current' snappy import (with check for redirects)

241. By Daniel Holbach on 2015-12-17

remove superfluous .get() call, add assertion that fetching worked fine

242. By Daniel Holbach on 2015-12-17

move snappy import tests into their own class

243. By Daniel Holbach on 2015-12-18

reset snappy test repos at the beginning of the test too

244. By Daniel Holbach on 2015-12-18

add assertion if snappy repo produces snappy articles

245. By Daniel Holbach on 2015-12-18

add reset() SnappyRepo to as well, rename variables and function names to be clearer, produce index_doc first, then add other pages, then refer to them in index_doc - this makes sure the parent (index_doc) is in the db first

246. By Daniel Holbach on 2015-12-18

merge lp:~developer-ubuntu-com-dev/developer-ubuntu-com/django-1.8-cms-2.3 - resolve conflicts

247. By Daniel Holbach on 2015-12-21

<ojii|work> because publishing etc will change a pages PK
<ojii|work> Page().publish() creates a new copy of the page (and deletes the old one)

248. By Daniel Holbach on 2015-12-21

improve variable usage

249. By Daniel Holbach on 2015-12-21

introduce repo.pages to distinguish from .imported_articles

250. By Daniel Holbach on 2015-12-21

fix tests

251. By Daniel Holbach on 2015-12-22

check for both 'python-api' and 'script'

252. By Daniel Holbach on 2015-12-22

break up tests into individual files

253. By Daniel Holbach on 2015-12-22

update TODO

254. By Daniel Holbach on 2016-01-05

improve create_repo helper function

255. By Daniel Holbach on 2016-01-05

allow SourceCode._get_branch to use local docs (ie for testing)

256. By Daniel Holbach on 2016-01-05

first cut at testing local links

257. By Daniel Holbach on 2016-01-05

fix simple link rewrite functionality and test

258. By Daniel Holbach on 2016-01-06

move md_importer into its own app

259. By Daniel Holbach on 2016-01-06

add migration for md_importer

260. By Daniel Holbach on 2016-01-06

merge lp:~dholbach/developer-ubuntu-com/django-cms-update

261. By Daniel Holbach on 2016-01-06

make directory structure clearer

262. By Daniel Holbach on 2016-01-06

adapt to new name of management command

263. By Daniel Holbach on 2016-01-06

add more readable error message

264. By Daniel Holbach on 2016-01-08

add more local test data (imported just the docs from snappy's and snapcraft's git master branches)

265. By Daniel Holbach on 2016-01-08

WIP commit:
- move and update TODO file
- define some global values for the importer centrally
- don't sys.exit() when adding pages to the db
- add debug messages
- break _find_parent() into its own function
- mostly make use of local test data
- drop singleton, use separate test classes instead
- add more assertions in tests

266. By Daniel Holbach on 2016-01-08

remove debug prints, make Repo class variables actual members

267. By Daniel Holbach on 2016-01-08

- make pep8 happy, remove unnecessary imports
- use TestLocalBranchImport as base class for almost everything,
  reduces a lot of duplication

268. By Daniel Holbach on 2016-01-08

remove unnecessary code, fix publishing of pages in tests

269. By Daniel Holbach on 2016-01-08

fix test_link_rewrite by fixing the URL

270. By Daniel Holbach on 2016-01-08

make Article class variables actual members

271. By Daniel Holbach on 2016-01-08

add new test to check links in snapcraft import, move link checking function into TestLocalBranchImport

272. By Daniel Holbach on 2016-01-11

merge lp:~developer-ubuntu-com-dev/developer-ubuntu-com/django-1.8-cms-2.3

273. By Daniel Holbach on 2016-01-11

allow empty strings in import directives

274. By Daniel Holbach on 2016-01-11

update migration

275. By Daniel Holbach on 2016-01-11

check if articles were added below home

276. By Daniel Holbach on 2016-01-11

remove unnecessary call to set a page's publisher_is_draft bit

277. By Daniel Holbach on 2016-01-11

give article a .publish() method which gives back the page's public_object

278. By Daniel Holbach on 2016-01-11

rename home_page to root_page, use cms.test_utils.testcases.CMSTestCase, make sure we use public_object wherever possible

279. By Daniel Holbach on 2016-01-11

update tests accordingly

280. By Daniel Holbach on 2016-01-11

disregard anchors, make link checking more flexible (ie not only check for intro.md, but also things like docs/intro.md

281. By Daniel Holbach on 2016-01-11

make URL work, even if LANG is already part of it

282. By Daniel Holbach on 2016-01-11

links look like they're working now

283. By Daniel Holbach on 2016-01-11

add test with a broken link, add convenience function is_local_link, modify tests

284. By Daniel Holbach on 2016-01-11

import doc fix from https://bugs.launchpad.net/developer-ubuntu-com/+bug/1531200

285. By Daniel Holbach on 2016-01-11

bug fixed

286. By Daniel Holbach on 2016-01-12

stop import if local images are found, add tests

287. By Daniel Holbach on 2016-01-12

update TODO

288. By Daniel Holbach on 2016-01-12

add test to see if importing the same content twice results in the same number of pages

289. By Daniel Holbach on 2016-01-15

when replacing links, only update HTML if things actually change, only publish if page is dirty

290. By Daniel Holbach on 2016-01-15

only update page attributes if they actually change, only update the text plugin if the html actually changes, veryify with djangocms_text_ckeditor.html.clean_html, add a text plugin to a page, even if html is empty

291. By Daniel Holbach on 2016-01-15

add test to check if running an import twice will update the articles in question

292. By Daniel Holbach on 2016-01-15

update TODO

293. By Daniel Holbach on 2016-01-15

add misc tests - forgot to 'bzr add'

294. By Daniel Holbach on 2016-01-15

make update-mtemplate work again, simplify it

295. By Daniel Holbach on 2016-01-15

update .pot file

296. By Daniel Holbach on 2016-01-15

add actual page object to repo.pages

297. By Daniel Holbach on 2016-01-15

use UTC for ImportedArticle.last_import, simplify ImportedArticle cleanup

298. By Daniel Holbach on 2016-01-15

use repo instead of branch consistently

299. By Daniel Holbach on 2016-01-15

break out the process of importing a branch into its own module, add a first simple test for it

300. By Daniel Holbach on 2016-01-16

fix clean up of imported articles, add a test

301. By Daniel Holbach on 2016-01-18

update TODO

Unmerged revisions

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added file 'TODO'
2--- TODO 1970-01-01 00:00:00 +0000
3+++ TODO 2015-12-09 15:41:23 +0000
4@@ -0,0 +1,7 @@
5+- import pictures (LP: #1511677)
6+- unpublished docs
7+- add comment at top of RawHTML plugins, so users know not to edit them,
8+ currently blocked on LP: #1523925
9+- when deleting pages, check if last rev was made by python-api
10+
11+-
12
13=== modified file 'developer_portal/admin.py'
14--- developer_portal/admin.py 2015-12-04 11:49:10 +0000
15+++ developer_portal/admin.py 2015-12-09 15:41:23 +0000
16@@ -4,7 +4,10 @@
17 from reversion.admin import VersionAdmin
18
19 from cms.extensions import TitleExtensionAdmin
20-from .models import ExternalDocsBranch, SEOExtension
21+from .models import (
22+ ExternalDocsBranch, ExternalDocsBranchImportDirective,
23+ ImportedArticle, SEOExtension
24+)
25 from django.core.management import call_command
26
27 __all__ = (
28@@ -13,7 +16,12 @@
29
30 def import_selected_external_docs_branches(modeladmin, request, queryset):
31 for branch in queryset:
32+<<<<<<< TREE
33 call_command('import-external-docs-branches', branch.docs_namespace)
34+=======
35+ branches.append(branch.origin)
36+ call_command('import-external-docs-branches', *branches)
37+>>>>>>> MERGE-SOURCE
38 import_selected_external_docs_branches.short_description = \
39 "Import selected branches"
40
41@@ -35,12 +43,22 @@
42 admin.site.register(Version, VersionAdmin)
43
44
45+@admin.register(ExternalDocsBranch)
46 class ExternalDocsBranchAdmin(admin.ModelAdmin):
47- list_display = ('lp_origin', 'docs_namespace')
48- list_filter = ('lp_origin', 'docs_namespace')
49+ list_display = ('origin', 'post_checkout_command', 'branch_name',)
50+ list_filter = ('origin', 'post_checkout_command', 'branch_name',)
51 actions = [import_selected_external_docs_branches]
52
53-admin.site.register(ExternalDocsBranch, ExternalDocsBranchAdmin)
54+
55+@admin.register(ExternalDocsBranchImportDirective)
56+class ExternalDocsBranchImportDirectiveAdmin(admin.ModelAdmin):
57+ pass
58+
59+
60+@admin.register(ImportedArticle)
61+class ImportedArticleAdmin(admin.ModelAdmin):
62+ pass
63+
64
65 class SEOExtensionAdmin(TitleExtensionAdmin):
66 pass
67
68=== modified file 'developer_portal/management/commands/import-external-docs-branches.py'
69--- developer_portal/management/commands/import-external-docs-branches.py 2015-12-04 11:49:10 +0000
70+++ developer_portal/management/commands/import-external-docs-branches.py 2015-12-09 15:41:23 +0000
71@@ -1,23 +1,14 @@
72 from django.core.management.base import BaseCommand
73 from django.core.management import call_command
74-from django.db import transaction
75-
76-from cms.api import create_page, add_plugin
77-from cms.models import Page, Title
78-from cms.utils import page_resolver
79-
80-from bs4 import BeautifulSoup
81-import codecs
82-import glob
83+
84+from ..importer.local_branch import LocalBranch, SnappyLocalBranch
85+
86+import datetime
87 import logging
88-import markdown
89-import os
90-import re
91 import shutil
92-import subprocess
93-import sys
94 import tempfile
95
96+<<<<<<< TREE
97 from developer_portal.models import ExternalDocsBranch
98
99 DOCS_DIRNAME = 'docs'
100@@ -295,6 +286,13 @@
101 placeholder = page.placeholders.get()
102 add_plugin(placeholder, 'RawHtmlPlugin', 'en', body=html)
103 return page
104+=======
105+from developer_portal.models import (
106+ ExternalDocsBranch,
107+ ExternalDocsBranchImportDirective,
108+ ImportedArticle,
109+)
110+>>>>>>> MERGE-SOURCE
111
112
113 def import_branches(selection):
114@@ -303,51 +301,39 @@
115 'ExternalDocsBranch table yet.')
116 return
117 tempdir = tempfile.mkdtemp()
118- db_actions = DBActions()
119 for branch in ExternalDocsBranch.objects.filter(
120- docs_namespace__regex=selection):
121- checkout_location = os.path.join(
122- tempdir, os.path.basename(branch.docs_namespace))
123- sourcecode = SourceCode(branch.lp_origin, checkout_location)
124- if sourcecode.get() != 0:
125- logging.error(
126- 'Could not check out branch "%s".' % branch.lp_origin)
127- if os.path.exists(checkout_location):
128- shutil.rmtree(checkout_location)
129+ origin__regex=selection, active=True):
130+ url = branch.origin
131+ if url.startswith('lp:snappy') or \
132+ 'snappy' in url.split(':')[1].split('.git')[0].split('/'):
133+ branch_class = SnappyLocalBranch
134+ else:
135+ branch_class = LocalBranch
136+ local_branch = branch_class(tempdir, branch.origin, branch.branch_name,
137+ branch.post_checkout_command)
138+ if local_branch.get() != 0:
139 break
140- if branch.lp_origin.startswith('lp:snappy') or \
141- 'snappy' in branch.lp_origin.split(':')[1].split('.git')[0].split('/'):
142- local_branch = SnappyLocalBranch(checkout_location, branch,
143- db_actions)
144- else:
145- local_branch = LocalBranch(checkout_location, branch, db_actions)
146- local_branch.import_markdown()
147- local_branch.publish()
148- local_branch.remove_old_pages()
149+ for directive in ExternalDocsBranchImportDirective.objects.filter(
150+ external_docs_branch=branch):
151+ local_branch.add_directive(directive.import_from,
152+ directive.write_to)
153+ local_branch.execute_import_directives()
154+ imported_articles = local_branch.publish()
155+ for imported_article in imported_articles:
156+ ImportedArticle.objects.get_or_create(
157+ branch=branch,
158+ page=imported_article.page,
159+ last_import=datetime.datetime.now())
160+
161+ # The import is done, now let's clean up.
162+ for old_article in ImportedArticle.objects.filter(branch=branch):
163+ if old_article.page not in [a.page for a in imported_articles]:
164+ # XXX: check if last revision was made by 'python-api'?
165+ old_article.page.delete()
166 shutil.rmtree(tempdir)
167- db_actions.run()
168-
169-
170-class SourceCode():
171- def __init__(self, branch_origin, checkout_location):
172- self.branch_origin = branch_origin
173- self.checkout_location = checkout_location
174-
175- def get(self):
176- if self.branch_origin.startswith('lp:') and \
177- os.path.exists('/usr/bin/bzr'):
178- return subprocess.call([
179- 'bzr', 'checkout', '--lightweight', self.branch_origin,
180- self.checkout_location])
181- if self.branch_origin.startswith('https://github.com') and \
182- self.branch_origin.endswith('.git') and \
183- os.path.exists('/usr/bin/git'):
184- return subprocess.call([
185- 'git', 'clone', '-q', self.branch_origin,
186- self.checkout_location])
187- logging.error(
188- 'Branch format "{}" not understood.'.format(self.branch_origin))
189- return 1
190+
191+ # https://stackoverflow.com/questions/33284171/
192+ call_command('cms', 'fix-tree')
193
194
195 class Command(BaseCommand):
196
197=== added directory 'developer_portal/management/importer'
198=== added file 'developer_portal/management/importer/__init__.py'
199=== added file 'developer_portal/management/importer/article.py'
200--- developer_portal/management/importer/article.py 1970-01-01 00:00:00 +0000
201+++ developer_portal/management/importer/article.py 2015-12-09 15:41:23 +0000
202@@ -0,0 +1,126 @@
203+from bs4 import BeautifulSoup
204+import codecs
205+import logging
206+import markdown
207+import os
208+import re
209+
210+from .publish import get_or_create_page, slugify
211+
212+
213+class Article:
214+ html = None
215+ page = None
216+ title = ""
217+
218+ def __init__(self, fn, write_to):
219+ self.fn = fn
220+ self.write_to = slugify(self.fn)
221+ self.full_url = write_to
222+ self.slug = os.path.basename(self.full_url)
223+
224+ def read(self):
225+ with codecs.open(self.fn, 'r', encoding='utf-8') as f:
226+ if self.fn.endswith('.md'):
227+ self.html = markdown.markdown(
228+ f.read(),
229+ output_format='html5',
230+ extensions=['markdown.extensions.tables'])
231+ elif self.fn.endswith('.html'):
232+ self.html = f.read()
233+ else:
234+ logging.error("Don't know how to interpret '{}'.".format(
235+ self.fn))
236+ return False
237+ self.title = self._read_title()
238+ self._remove_body_and_html_tags()
239+ self._use_developer_site_style()
240+ return True
241+
242+ def _read_title(self):
243+ soup = BeautifulSoup(self.html, 'html5lib')
244+ if soup.title:
245+ return soup.title.text
246+ if soup.h1:
247+ return soup.h1.text
248+ return slugify(self.fn).replace('-', ' ').title()
249+
250+ def _remove_body_and_html_tags(self):
251+ self.html = re.sub(r"<html>\n\s<body>\n", "", self.html,
252+ flags=re.MULTILINE)
253+ self.html = re.sub(r"\s<\/body>\n<\/html>", "", self.html,
254+ flags=re.MULTILINE)
255+
256+ def _use_developer_site_style(self):
257+ begin = (u"<div class=\"row no-border\">"
258+ "\n<div class=\"eight-col\">\n")
259+ end = u"</div>\n</div>"
260+ self.html = begin + self.html + end
261+ self.html = self.html.replace(
262+ "<pre><code>",
263+ "</div><div class=\"twelve-col\"><pre><code>")
264+ self.html = self.html.replace(
265+ "</code></pre>",
266+ "</code></pre></div><div class=\"eight-col\">")
267+
268+ def replace_links(self, titles, url_map):
269+ for title in titles:
270+ local_md_fn = os.path.basename(title)
271+ url = u'/'+url_map[title]
272+ # Replace links of the form <a href="/path/somefile.md"> first
273+ href = u"<a href=\"{}\">".format(url)
274+ md_href = u"<a href=\"{}\">".format(local_md_fn)
275+ self.html = self.html.replace(md_href, href)
276+
277+ # Now we can replace free-standing "somefile.md" references in
278+ # the HTML
279+ link = href + u"{}</a>".format(titles[title])
280+ self.html = self.html.replace(local_md_fn, link)
281+
282+ def publish(self):
283+ '''Publishes pages in their branch alias namespace.'''
284+ page = get_or_create_page(
285+ title=self.title, full_url=self.full_url, menu_title=self.title,
286+ html=self.html)
287+ self.page = page
288+ self.page.publish('en')
289+
290+
291+class SnappyArticle(Article):
292+ release_alias = None
293+
294+ def get(self):
295+ if not Article.get(self):
296+ return False
297+ self.release_alias = re.findall(r'snappy/guides/(\S+?)/\S+?',
298+ self.full_url)[0]
299+ self._make_snappy_mods()
300+ return True
301+
302+ def _make_snappy_mods(self):
303+ # Make sure the reader knows which documentation she is browsing
304+ if self.release_alias != 'current':
305+ before = (u"<div class=\"row no-border\">\n"
306+ "<div class=\"eight-col\">\n")
307+ after = (u"<div class=\"row no-border\">\n"
308+ "<div class=\"box pull-three three-col\">"
309+ "<p>You are browsing the Snappy <code>%s</code> "
310+ "documentation.</p>"
311+ "<p><a href=\"/snappy/guides/current/%s\">"
312+ "Back to the latest stable release &rsaquo;"
313+ "</a></p></div>\n"
314+ "<div class=\"eight-col\">\n") % (self.release_alias,
315+ self.slug, )
316+ self.html = self.html.replace(before, after)
317+
318+ def publish(self):
319+ if self.release_alias == "current":
320+ # Add a guides/<page> redirect to guides/current/<page>
321+ page = get_or_create_page(
322+ title=self.title,
323+ full_url=self.full_url.replace('/current', ''),
324+ redirect="/snappy/guides/current/{}".format(self.slug))
325+ page.publish('en')
326+ else:
327+ self.title += " (%s)" % (self.release_alias,)
328+ Article.publish(self)
329
330=== added file 'developer_portal/management/importer/local_branch.py'
331--- developer_portal/management/importer/local_branch.py 1970-01-01 00:00:00 +0000
332+++ developer_portal/management/importer/local_branch.py 2015-12-09 15:41:23 +0000
333@@ -0,0 +1,132 @@
334+from .article import Article, SnappyArticle
335+from .publish import get_or_create_page, slugify
336+from .source import SourceCode
337+
338+import glob
339+import logging
340+import os
341+import shutil
342+
343+
344+class LocalBranch:
345+ titles = {}
346+ url_map = {}
347+ index_doc_title = None
348+ index_doc = None
349+ release_alias = None
350+
351+ def __init__(self, tempdir, origin, branch_name, post_checkout_command):
352+ self.origin = origin
353+ self.branch_name = branch_name
354+ self.post_checkout_command = post_checkout_command
355+ self.checkout_location = os.path.join(
356+ tempdir,
357+ os.path.basename(self.origin.replace('.git', '')))
358+ self.article_class = Article
359+ self.directives = []
360+ self.imported_articles = []
361+
362+ def get(self):
363+ sourcecode = SourceCode(self.origin, self.checkout_location,
364+ self.branch_name, self.post_checkout_command)
365+ if sourcecode.get() != 0:
366+ logging.error(
367+ 'Could not check out branch "{}".'.format(self.origin))
368+ shutil.rmtree(self.checkout_location)
369+ return 1
370+ return 0
371+
372+ def add_directive(self, import_from, write_to):
373+ self.directives += [
374+ {
375+ 'import_from': os.path.join(self.checkout_location,
376+ import_from),
377+ 'write_to': write_to
378+ }
379+ ]
380+
381+ def execute_import_directives(self):
382+ import_list = []
383+ # Import single files first
384+ for directive in [d for d in self.directives
385+ if os.path.isfile(d['import_from'])]:
386+ import_list += [
387+ (directive['import_from'], directive['write_to'])
388+ ]
389+ # Import directories next
390+ for directive in [d for d in self.directives
391+ if os.path.isdir(d['import_from'])]:
392+ for fn in glob.glob('{}/*'.format(directive['import_from'])):
393+ if fn not in [a[0] for a in import_list]:
394+ import_list += [
395+ (fn, os.path.join(directive['write_to'], slugify(fn)))
396+ ]
397+ # If we import into a namespace and don't have an index doc,
398+ # we need to write one.
399+ if directive['write_to'] not in [x[1] for x in import_list]:
400+ self.index_doc = directive['write_to']
401+ # The actual import
402+ for entry in import_list:
403+ article = self._read_article(entry[0], entry[1])
404+ if article:
405+ self.imported_articles += [article]
406+ self.titles[article.fn] = article.title
407+ self.url_map[article.fn] = article.full_url
408+ for article in self.imported_articles:
409+ article.replace_links(self.titles, self.url_map)
410+ if self.index_doc:
411+ self._create_fake_index_docs()
412+
413+ def _read_article(self, fn, write_to):
414+ article = self.article_class(fn, write_to)
415+ if article.read():
416+ return article
417+ return None
418+
419+ def publish(self):
420+ for article in self.imported_articles:
421+ article.publish()
422+ return self.imported_articles
423+
424+ def _create_fake_index_docs(self):
425+ '''Creates a fake index page at the top of the branches
426+ docs namespace.'''
427+
428+ if self.index_doc.endswith('current'):
429+ redirect = '/snappy/guides'
430+ else:
431+ redirect = None
432+ list_pages = ''
433+ for article in [a for a
434+ in self.imported_articles
435+ if a.full_url.startswith(self.index_doc)]:
436+ list_pages += '<li><a href=\"{}\">{}</a></li>'.format(
437+ os.path.basename(article.full_url), article.title)
438+ landing = (
439+ u'<div class=\"row\"><div class=\"eight-col\">\n'
440+ '<p>This section contains documentation for the '
441+ '<code>{}</code> Snappy branch.</p>'
442+ '<p><ul class=\"list-ubuntu\">{}</ul></p>\n'
443+ '<p>Auto-imported from <a '
444+ 'href=\"https://github.com/ubuntu-core/snappy\">%s</a>.</p>\n'
445+ '</div></div>'.format(self.release_alias, list_pages,
446+ self.origin))
447+ page = get_or_create_page(
448+ title=self.index_doc_title, full_url=self.index_doc,
449+ in_navigation=False, redirect=redirect, html=landing,
450+ menu_title=None)
451+ page.publish('en')
452+
453+
454+class SnappyLocalBranch(LocalBranch):
455+ def __init__(self, tempdir, origin, branch_name, post_checkout_command):
456+ LocalBranch.__init__(self, tempdir, origin, branch_name,
457+ post_checkout_command)
458+ self.article_class = SnappyArticle
459+ self.index_doc_title = 'Snappy documentation'
460+
461+ def _create_fake_index_docs(self):
462+ self.release_alias = os.path.basename(self.index_doc)
463+ if not self.index_doc.endswith('current'):
464+ self.index_doc_title += ' ({})'.format(self.release_alias)
465+ LocalBranch._create_fake_index_docs(self)
466
467=== added file 'developer_portal/management/importer/publish.py'
468--- developer_portal/management/importer/publish.py 1970-01-01 00:00:00 +0000
469+++ developer_portal/management/importer/publish.py 2015-12-09 15:41:23 +0000
470@@ -0,0 +1,60 @@
471+from cms.api import create_page, add_plugin
472+from cms.models import Title
473+
474+import logging
475+import os
476+import sys
477+
478+# XXX: use this once we RawHTML plugins don't strip comments (LP: #1523925)
479+START_TEXT = """
480+<!--
481+branch id: {}
482+
483+THIS PAGE IS AUTOMATICALLY UPDATED.
484+DON'T EDIT IT - CHANGES WILL BE OVERWRITTEN.
485+-->
486+"""
487+
488+
489+def slugify(filename):
490+ return os.path.basename(filename).replace('.md', '').replace('.html', '')
491+
492+
493+def get_or_create_page(title, full_url, menu_title=None,
494+ in_navigation=True, redirect=None, html=None):
495+ # First check if pages already exist.
496+ pages = Title.objects.select_related('page').filter(path__regex=full_url)
497+ if pages:
498+ page = pages[0].page
499+ page.title = title
500+ page.publisher_is_draft = True
501+ page.menu_title = menu_title
502+ page.in_navigation = in_navigation
503+ page.redirect = redirect
504+ if html:
505+ # We create the page, so we know there's just one placeholder
506+ placeholder = page.placeholders.all()[0]
507+ if placeholder.get_plugins():
508+ plugin = placeholder.get_plugins()[0].get_plugin_instance()[0]
509+ plugin.body = html
510+ plugin.save()
511+ else:
512+ add_plugin(placeholder, 'RawHtmlPlugin', 'en', body=html)
513+ else:
514+ parent_pages = Title.objects.select_related('page').filter(
515+ path__regex=os.path.dirname(full_url))
516+ if not parent_pages:
517+ logging.error('Parent {} not found.'.format(
518+ os.path.dirname(full_url)))
519+ sys.exit(1)
520+ parent = parent_pages[0].page
521+
522+ slug = os.path.basename(full_url)
523+ page = create_page(
524+ title, "default.html", "en", slug=slug, parent=parent,
525+ menu_title=menu_title, in_navigation=in_navigation,
526+ position="last-child", redirect=redirect)
527+ if html:
528+ placeholder = page.placeholders.get()
529+ add_plugin(placeholder, 'RawHtmlPlugin', 'en', body=html)
530+ return page
531
532=== added file 'developer_portal/management/importer/source.py'
533--- developer_portal/management/importer/source.py 1970-01-01 00:00:00 +0000
534+++ developer_portal/management/importer/source.py 2015-12-09 15:41:23 +0000
535@@ -0,0 +1,51 @@
536+import logging
537+import os
538+import subprocess
539+
540+
541+class SourceCode():
542+ def __init__(self, origin, checkout_location, branch_name,
543+ post_checkout_command):
544+ self.origin = origin
545+ self.checkout_location = checkout_location
546+ self.branch_name = branch_name
547+ self.post_checkout_command = post_checkout_command
548+
549+ def get(self):
550+ res = self._get_branch()
551+ if res == 0 and self.post_checkout_command:
552+ res = self._post_checkout()
553+ return res
554+ return res
555+
556+ def _get_branch(self):
557+ if self.origin.startswith('lp:') and \
558+ os.path.exists('/usr/bin/bzr'):
559+ return subprocess.call([
560+ 'bzr', 'checkout', '--lightweight', self.origin,
561+ self.checkout_location])
562+ if self.origin.startswith('https://github.com') and \
563+ self.origin.endswith('.git') and \
564+ os.path.exists('/usr/bin/git'):
565+ retcode = subprocess.call([
566+ 'git', 'clone', '-q', self.origin,
567+ self.checkout_location])
568+ if retcode == 0 and self.branch_name:
569+ pwd = os.getcwd()
570+ os.chdir(self.checkout_location)
571+ retcode = subprocess.call(['git', 'checkout',
572+ self.branch_name])
573+ os.chdir(pwd)
574+ return retcode
575+ logging.error(
576+ 'Branch format "{}" not understood.'.format(self.origin))
577+ return 1
578+
579+ def _post_checkout(self):
580+ process = subprocess.Popen(self.post_checkout_command.split(),
581+ stdout=subprocess.PIPE)
582+ (out, err) = process.communicate()
583+ retcode = process.wait()
584+ if retcode != 0:
585+ logging.error(out)
586+ return retcode
587
588=== added file 'developer_portal/migrations/0002_rework_importer.py'
589--- developer_portal/migrations/0002_rework_importer.py 1970-01-01 00:00:00 +0000
590+++ developer_portal/migrations/0002_rework_importer.py 2015-12-09 15:41:23 +0000
591@@ -0,0 +1,59 @@
592+# -*- coding: utf-8 -*-
593+from __future__ import unicode_literals
594+
595+from django.db import migrations, models
596+
597+
598+class Migration(migrations.Migration):
599+
600+ dependencies = [
601+ ('developer_portal', '0001_initial'),
602+ ]
603+
604+ operations = [
605+ migrations.CreateModel(
606+ name='ExternalDocsBranchImportDirective',
607+ fields=[
608+ ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
609+ ('import_from', models.CharField(help_text='File or directory to import from the branch. Ie: "docs/intro.md" (file) or "docs" (complete directory), etc.', max_length=150)),
610+ ('write_to', models.CharField(help_text='Article URL (for a specific file) or article namespace for a directory or a set of files.', max_length=150)),
611+ ],
612+ ),
613+ migrations.AlterModelOptions(
614+ name='externaldocsbranch',
615+ options={'verbose_name': 'external docs branch', 'verbose_name_plural': 'external docs branches'},
616+ ),
617+ migrations.RenameField(
618+ model_name='externaldocsbranch',
619+ old_name='lp_origin',
620+ new_name='origin',
621+ ),
622+ migrations.RemoveField(
623+ model_name='externaldocsbranch',
624+ name='docs_namespace',
625+ ),
626+ migrations.RemoveField(
627+ model_name='externaldocsbranch',
628+ name='index_doc',
629+ ),
630+ migrations.AddField(
631+ model_name='externaldocsbranch',
632+ name='active',
633+ field=models.BooleanField(default=True),
634+ ),
635+ migrations.AddField(
636+ model_name='externaldocsbranch',
637+ name='branch_name',
638+ field=models.CharField(help_text='For use with git branches, ie: "master" or "15.04" or "1.x".', max_length=200, blank=True),
639+ ),
640+ migrations.AddField(
641+ model_name='externaldocsbranch',
642+ name='post_checkout_command',
643+ field=models.CharField(help_text='Command to run after checkout of the branch.', max_length=100, blank=True),
644+ ),
645+ migrations.AddField(
646+ model_name='externaldocsbranchimportdirective',
647+ name='external_docs_branch',
648+ field=models.ForeignKey(to='developer_portal.ExternalDocsBranch'),
649+ ),
650+ ]
651
652=== added file 'developer_portal/migrations/0003_track_imported_articles.py'
653--- developer_portal/migrations/0003_track_imported_articles.py 1970-01-01 00:00:00 +0000
654+++ developer_portal/migrations/0003_track_imported_articles.py 2015-12-09 15:41:23 +0000
655@@ -0,0 +1,24 @@
656+# -*- coding: utf-8 -*-
657+from __future__ import unicode_literals
658+
659+from django.db import migrations, models
660+
661+
662+class Migration(migrations.Migration):
663+
664+ dependencies = [
665+ ('cms', '0013_urlconfrevision'),
666+ ('developer_portal', '0002_rework_importer'),
667+ ]
668+
669+ operations = [
670+ migrations.CreateModel(
671+ name='ImportedArticle',
672+ fields=[
673+ ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
674+ ('last_import', models.DateTimeField(help_text='Datetime of last import.', verbose_name='Datetime')),
675+ ('branch', models.ForeignKey(to='developer_portal.ExternalDocsBranch')),
676+ ('page', models.ForeignKey(to='cms.Page')),
677+ ],
678+ ),
679+ ]
680
681=== modified file 'developer_portal/models.py'
682--- developer_portal/models.py 2015-11-02 16:47:26 +0000
683+++ developer_portal/models.py 2015-12-09 15:41:23 +0000
684@@ -1,9 +1,9 @@
685 from django.db import models
686 from django.utils.translation import ugettext_lazy as _
687
688-from cms.models import CMSPlugin
689 from cms.extensions import TitleExtension
690 from cms.extensions.extension_pool import extension_pool
691+from cms.models import Page
692 from djangocms_text_ckeditor.html import extract_images
693 from djangocms_text_ckeditor.models import AbstractText
694
695@@ -21,23 +21,54 @@
696
697
698 class ExternalDocsBranch(models.Model):
699- # We originally assumed that branches would also live in LP,
700- # well, we were wrong, but let's keep the name around. It's
701- # no use having a schema/data migration just for this.
702- lp_origin = models.CharField(
703+ origin = models.CharField(
704 max_length=200,
705 help_text=_('External branch location, ie: lp:snappy/15.04 or '
706 'https://github.com/ubuntu-core/snappy.git'))
707- docs_namespace = models.CharField(
708- max_length=120,
709- help_text=_('Path alias we want to use for the docs, '
710- 'ie "snappy/guides/15.04" or '
711- '"snappy/guides/latest", etc.'))
712- index_doc = models.CharField(
713- max_length=120,
714- help_text=_('File name of doc to be used as index document, '
715- 'ie "intro.md"'),
716- blank=True)
717+ branch_name = models.CharField(
718+ max_length=200,
719+ help_text=_('For use with git branches, ie: "master" or "15.04" '
720+ 'or "1.x".'),
721+ blank=True)
722+ post_checkout_command = models.CharField(
723+ max_length=100,
724+ help_text=_('Command to run after checkout of the branch.'),
725+ blank=True)
726+ active = models.BooleanField(default=True)
727+
728+ def __str__(self):
729+ if self.branch_name:
730+ return "{} {}".format(self.origin, self.branch_name)
731+ return "{}".format(self.origin)
732+
733+ class Meta:
734+ verbose_name = "external docs branch"
735+ verbose_name_plural = "external docs branches"
736+
737+
738+class ExternalDocsBranchImportDirective(models.Model):
739+ external_docs_branch = models.ForeignKey(ExternalDocsBranch)
740+ import_from = models.CharField(
741+ max_length=150,
742+ help_text=_('File or directory to import from the branch. '
743+ 'Ie: "docs/intro.md" (file) or '
744+ '"docs" (complete directory), etc.'))
745+ write_to = models.CharField(
746+ max_length=150,
747+ help_text=_('Article URL (for a specific file) or article namespace '
748+ 'for a directory or a set of files.'))
749+
750+ def __str__(self):
751+ return "{} -- {}".format(self.external_docs_branch.origin,
752+ self.import_from)
753+
754+
755+class ImportedArticle(models.Model):
756+ page = models.ForeignKey(Page)
757+ branch = models.ForeignKey(ExternalDocsBranch)
758+ last_import = models.DateTimeField(
759+ _('Datetime'), help_text=_('Datetime of last import.'))
760+
761
762 class SEOExtension(TitleExtension):
763 keywords = models.CharField(max_length=256)
764
765=== modified file 'developer_portal/settings.py'
766--- developer_portal/settings.py 2015-12-04 16:09:27 +0000
767+++ developer_portal/settings.py 2015-12-09 15:41:23 +0000
768@@ -318,6 +318,26 @@
769 #'PAGINATE_BY': 10,
770 }
771
772+<<<<<<< TREE
773+=======
774+MIGRATION_MODULES = {
775+ 'cms': 'cms.migrations',
776+ 'cmsplugin_zinnia': 'cmsplugin_zinnia.migrations',
777+ 'djangocms_link': 'djangocms_link.migrations',
778+ 'djangocms_picture': 'djangocms_picture.migrations',
779+ 'djangocms_snippet': 'djangocms_snippet.migrations',
780+ 'djangocms_text_ckeditor': 'djangocms_text_ckeditor.migrations',
781+ 'djangocms_video': 'djangocms_video.migrations',
782+ 'django_comments': 'django_comments.migrations',
783+ 'menus': 'menus.migrations',
784+ 'rest_framework.authtoken': 'rest_framework.authtoken.migrations',
785+ 'reversion': 'reversion.migrations',
786+ 'tagging': 'tagging.migrations',
787+ 'taggit': 'taggit.migrations',
788+ 'zinnia': 'zinnia.migrations',
789+}
790+
791+>>>>>>> MERGE-SOURCE
792 LOGGING = {
793 'version': 1,
794 'disable_existing_loggers': False,
795
796=== modified file 'requirements.txt'
797--- requirements.txt 2015-12-04 11:49:10 +0000
798+++ requirements.txt 2015-12-09 15:41:23 +0000
799@@ -1,3 +1,4 @@
800+<<<<<<< TREE
801 oslo.config==1.6.0
802 oslo.i18n==1.2.0
803 oslo.serialization==1.2.0
804@@ -8,12 +9,25 @@
805 Pillow==2.6.1
806 beautifulsoup4==4.4.0
807 cmsplugin-zinnia==0.6
808+=======
809+Django==1.8.7
810+django-template-debug==0.3.5
811+oslo.config==3.0.0
812+oslo.i18n==2.7.0
813+oslo.serialization==2.0.0
814+oslo.utils==2.8.0
815+Pillow==2.9.0
816+cmsplugin-zinnia==0.8
817+Markdown==2.6.5
818+beautifulsoup4==4.4.1
819+>>>>>>> MERGE-SOURCE
820 dj-database-url==0.3.0
821 django-admin-enhancer==0.1.3.1
822 django-appconf==0.6
823 django-blog-zinnia==0.14.2
824 django-ckeditor==4.4.7
825 django-ckeditor-updated==4.4.4
826+<<<<<<< TREE
827 django-classy-tags==0.5.1
828 django-cms==3.0.6
829 django-contrib-comments==1.5
830@@ -25,23 +39,56 @@
831 django-polymorphic==0.6
832 django-reversion==1.8.4
833 django-sekizai==0.7
834+=======
835+django-classy-tags==0.6.2
836+django-cms==3.2.0
837+django-contrib-comments==1.6.1
838+django-meta==0.3.1
839+django-meta-mixin==0.2.1
840+django-missing==0.1.15
841+django-parler==1.5.1
842+django-polymorphic==0.7.2
843+django-reversion==1.9.3
844+django-sekizai==0.8.2
845+>>>>>>> MERGE-SOURCE
846 django-swiftstorage==1.1.0
847+<<<<<<< TREE
848 django-tagging==0.3.3
849 django-taggit==0.12.2
850 django-taggit-autosuggest==0.2.5
851+=======
852+django-tagging==0.4
853+django-taggit==0.17.5
854+django-taggit-autosuggest==0.2.8
855+>>>>>>> MERGE-SOURCE
856 django-taggit-templatetags==0.2.5
857 django-templatetag-sugar==1.0
858 django-xmlrpc==0.1.5
859+<<<<<<< TREE
860 djangocms-admin-style==0.2.2
861 djangocms-link==1.5
862 djangocms-picture==0.1
863 djangocms-snippet==1.3
864 djangocms-text-ckeditor==2.4.2
865+=======
866+djangocms-admin-style==1.0.7
867+djangocms-link==1.7.1
868+djangocms-picture==0.2.0
869+djangocms-snippet==1.7.1
870+djangocms-text-ckeditor==2.8.0
871+>>>>>>> MERGE-SOURCE
872 djangocms-utils==0.9.5
873+<<<<<<< TREE
874 djangocms-video==0.1
875 python-keystoneclient==0.11.2
876 python-swiftclient==2.3.1
877 pytz==2014.7
878+=======
879+djangocms-video==0.2.0
880+python-keystoneclient==1.3.3
881+python-swiftclient==2.6.0
882+pytz==2015.7
883+>>>>>>> MERGE-SOURCE
884 simple-translation==0.8.6
885 simplejson==3.6.5
886 wsgiref==0.1.2

Subscribers

People subscribed via source and target branches