Merge lp:~dholbach/developer-ubuntu-com/rework-importer into lp:developer-ubuntu-com

[Daniel Holbach]

[lp:~dholbach/developer-ubuntu-com/rework-importer]

183. By Daniel Holbach on 2015-11-18

make admin interface for ExternalDocsBranch* usable

184. By Daniel Holbach on 2015-11-19

first cut of importer reorg
 - break up into individual files
 - simplify API a bit, make it more easily extendable and testable
 - next steps: implement removal of old articles, fix links

185. By Daniel Holbach on 2015-11-19

we haven't implemented globs yet

186. By Daniel Holbach on 2015-12-01

merged from trunk

187. By Daniel Holbach on 2015-12-02

add TODO

188. By Daniel Holbach on 2015-12-04

merge trunk

189. By Daniel Holbach on 2015-12-04

merged lp:~dholbach/developer-ubuntu-com/django-cms-update

190. By Daniel Holbach on 2015-12-04

some more package updates

191. By Daniel Holbach on 2015-12-04

revert reversion upgrade

192. By Daniel Holbach on 2015-12-04

update migations locations

193. By Daniel Holbach on 2015-12-04

rename branch_origin to origin, add branch_name and add functionatlity so users can specify which branch of a git branch to import form

194. By Daniel Holbach on 2015-12-04

make ExternalDocsBranch.__str__ more revealing

195. By Daniel Holbach on 2015-12-04

update SnappyLocalBranch c'tor to be inline with the added branch_name field

196. By Daniel Holbach on 2015-12-04

make sure we run 'checkout <branch>' in the right directory

197. By Daniel Holbach on 2015-12-07

update member name

198. By Daniel Holbach on 2015-12-07

fix imports, return published articles

199. By Daniel Holbach on 2015-12-08

update TODO

200. By Daniel Holbach on 2015-12-08

make pep8 happy, use .format()

201. By Daniel Holbach on 2015-12-08

allow us to track which pages were imported from which branch

202. By Daniel Holbach on 2015-12-08

make pep8 happy

203. By Daniel Holbach on 2015-12-08

first cut at tracking docs-branch/page in a separate table

204. By Daniel Holbach on 2015-12-09

fix check if directory-import would import already file-imported articles

205. By Daniel Holbach on 2015-12-09

safe datetime of last_import, fix removal condition of old articles

206. By Daniel Holbach on 2015-12-09

remove unused import

207. By Daniel Holbach on 2015-12-09

merge from lp:~developer-ubuntu-com-dev/developer-ubuntu-com/django-1.8-cms-2.3

208. By Daniel Holbach on 2015-12-09

add check if the last change was made by our script

209. By Daniel Holbach on 2015-12-14

merge lp:~developer-ubuntu-com-dev/developer-ubuntu-com/django-1.8-cms-2.3 again

210. By Daniel Holbach on 2015-12-14

make __str__ representations of ExternalDocsBranchImportDirective and ExternalDocsBranch more readable

211. By Daniel Holbach on 2015-12-14

fix syntax error

212. By Daniel Holbach on 2015-12-14

every branch gets its own tempdir

213. By Daniel Holbach on 2015-12-14

help with paths starting with '/'

214. By Daniel Holbach on 2015-12-14

every branch gets its own tempdir

215. By Daniel Holbach on 2015-12-14

add fenced_code markdown extension - only change links if they are in our url_map

216. By Daniel Holbach on 2015-12-14

allow github style markdown

217. By Daniel Holbach on 2015-12-14

remove url_map checks, they are useless

218. By Daniel Holbach on 2015-12-14

update TODO

219. By Daniel Holbach on 2015-12-14

support local repositories - useful for testing

220. By Daniel Holbach on 2015-12-14

add first tests

221. By Daniel Holbach on 2015-12-14

add another test

222. By Daniel Holbach on 2015-12-14

pep8 fix

223. By Daniel Holbach on 2015-12-15

rename tests to be clearer, rewrite tests, so we can reuse a repository we already checked out

224. By Daniel Holbach on 2015-12-15

rename local_branch to repo - makes more sense these days

225. By Daniel Holbach on 2015-12-15

add test for post_checkout_command, fix post_checkout_command logic by chdir()ing correctly, make pep8+pyflakes happy

226. By Daniel Holbach on 2015-12-15

make create_home_page return home, test page db activities

227. By Daniel Holbach on 2015-12-15

check if all imported articles have 'home' as parent

228. By Daniel Holbach on 2015-12-15

add comment, pep8

229. By Daniel Holbach on 2015-12-15

add function to reset a repo - only useful for tests, where we reuse the same repo to avoid redownloading, add test where only one file is imported and we check the article tree

230. By Daniel Holbach on 2015-12-15

filter on publisher_is_draft=True

231. By Daniel Holbach on 2015-12-15

clean up after the test

232. By Daniel Holbach on 2015-12-16

make index_doc_title available everywhere

233. By Daniel Holbach on 2015-12-16

add db_add_empty_page convenience function, add simple articletree test, add test for the import of snappy devel

234. By Daniel Holbach on 2015-12-16

make git quiet

235. By Daniel Holbach on 2015-12-16

add create_repo convenience function

236. By Daniel Holbach on 2015-12-16

use create_repo function, add assertions for SnappyRepo vs Repo instantiation

237. By Daniel Holbach on 2015-12-17

add SnappyTestRepo singleton for re-use, rename variables

238. By Daniel Holbach on 2015-12-17

add assertions if imported articles have the correct type - drop useless tempdir creation

239. By Daniel Holbach on 2015-12-17

function article.get was renamed to article.read ages ago...

240. By Daniel Holbach on 2015-12-17

add test for 'current' snappy import (with check for redirects)

241. By Daniel Holbach on 2015-12-17

remove superfluous .get() call, add assertion that fetching worked fine

242. By Daniel Holbach on 2015-12-17

move snappy import tests into their own class

243. By Daniel Holbach on 2015-12-18

reset snappy test repos at the beginning of the test too

244. By Daniel Holbach on 2015-12-18

add assertion if snappy repo produces snappy articles

245. By Daniel Holbach on 2015-12-18

add reset() SnappyRepo to as well, rename variables and function names to be clearer, produce index_doc first, then add other pages, then refer to them in index_doc - this makes sure the parent (index_doc) is in the db first

246. By Daniel Holbach on 2015-12-18

merge lp:~developer-ubuntu-com-dev/developer-ubuntu-com/django-1.8-cms-2.3 - resolve conflicts

247. By Daniel Holbach on 2015-12-21

<ojii|work> because publishing etc will change a pages PK
<ojii|work> Page().publish() creates a new copy of the page (and deletes the old one)

248. By Daniel Holbach on 2015-12-21

improve variable usage

249. By Daniel Holbach on 2015-12-21

introduce repo.pages to distinguish from .imported_articles

250. By Daniel Holbach on 2015-12-21

fix tests

251. By Daniel Holbach on 2015-12-22

check for both 'python-api' and 'script'

252. By Daniel Holbach on 2015-12-22

break up tests into individual files

253. By Daniel Holbach on 2015-12-22

update TODO

254. By Daniel Holbach on 2016-01-05

improve create_repo helper function

255. By Daniel Holbach on 2016-01-05

allow SourceCode._get_branch to use local docs (ie for testing)

256. By Daniel Holbach on 2016-01-05

first cut at testing local links

257. By Daniel Holbach on 2016-01-05

fix simple link rewrite functionality and test

258. By Daniel Holbach on 2016-01-06

move md_importer into its own app

259. By Daniel Holbach on 2016-01-06

add migration for md_importer

260. By Daniel Holbach on 2016-01-06

merge lp:~dholbach/developer-ubuntu-com/django-cms-update

261. By Daniel Holbach on 2016-01-06

make directory structure clearer

262. By Daniel Holbach on 2016-01-06

adapt to new name of management command

263. By Daniel Holbach on 2016-01-06

add more readable error message

264. By Daniel Holbach on 2016-01-08

add more local test data (imported just the docs from snappy's and snapcraft's git master branches)

265. By Daniel Holbach on 2016-01-08

WIP commit:
- move and update TODO file
- define some global values for the importer centrally
- don't sys.exit() when adding pages to the db
- add debug messages
- break _find_parent() into its own function
- mostly make use of local test data
- drop singleton, use separate test classes instead
- add more assertions in tests

266. By Daniel Holbach on 2016-01-08

remove debug prints, make Repo class variables actual members

267. By Daniel Holbach on 2016-01-08

- make pep8 happy, remove unnecessary imports
- use TestLocalBranchImport as base class for almost everything,
  reduces a lot of duplication

268. By Daniel Holbach on 2016-01-08

remove unnecessary code, fix publishing of pages in tests

269. By Daniel Holbach on 2016-01-08

fix test_link_rewrite by fixing the URL

270. By Daniel Holbach on 2016-01-08

make Article class variables actual members

271. By Daniel Holbach on 2016-01-08

add new test to check links in snapcraft import, move link checking function into TestLocalBranchImport

272. By Daniel Holbach on 2016-01-11

merge lp:~developer-ubuntu-com-dev/developer-ubuntu-com/django-1.8-cms-2.3

273. By Daniel Holbach on 2016-01-11

allow empty strings in import directives

274. By Daniel Holbach on 2016-01-11

update migration

275. By Daniel Holbach on 2016-01-11

check if articles were added below home

276. By Daniel Holbach on 2016-01-11

remove unnecessary call to set a page's publisher_is_draft bit

277. By Daniel Holbach on 2016-01-11

give article a .publish() method which gives back the page's public_object

278. By Daniel Holbach on 2016-01-11

rename home_page to root_page, use cms.test_utils.testcases.CMSTestCase, make sure we use public_object wherever possible

279. By Daniel Holbach on 2016-01-11

update tests accordingly

280. By Daniel Holbach on 2016-01-11

disregard anchors, make link checking more flexible (ie not only check for intro.md, but also things like docs/intro.md

281. By Daniel Holbach on 2016-01-11

make URL work, even if LANG is already part of it

282. By Daniel Holbach on 2016-01-11

links look like they're working now

283. By Daniel Holbach on 2016-01-11

add test with a broken link, add convenience function is_local_link, modify tests

284. By Daniel Holbach on 2016-01-11

import doc fix from https://bugs.launchpad.net/developer-ubuntu-com/+bug/1531200

285. By Daniel Holbach on 2016-01-11

bug fixed

286. By Daniel Holbach on 2016-01-12

stop import if local images are found, add tests

287. By Daniel Holbach on 2016-01-12

update TODO

288. By Daniel Holbach on 2016-01-12

add test to see if importing the same content twice results in the same number of pages

289. By Daniel Holbach on 2016-01-15

when replacing links, only update HTML if things actually change, only publish if page is dirty

290. By Daniel Holbach on 2016-01-15

only update page attributes if they actually change, only update the text plugin if the html actually changes, veryify with djangocms_text_ckeditor.html.clean_html, add a text plugin to a page, even if html is empty

291. By Daniel Holbach on 2016-01-15

add test to check if running an import twice will update the articles in question

292. By Daniel Holbach on 2016-01-15

update TODO

293. By Daniel Holbach on 2016-01-15

add misc tests - forgot to 'bzr add'

294. By Daniel Holbach on 2016-01-15

make update-mtemplate work again, simplify it

295. By Daniel Holbach on 2016-01-15

update .pot file

296. By Daniel Holbach on 2016-01-15

add actual page object to repo.pages

297. By Daniel Holbach on 2016-01-15

use UTC for ImportedArticle.last_import, simplify ImportedArticle cleanup

298. By Daniel Holbach on 2016-01-15

use repo instead of branch consistently

299. By Daniel Holbach on 2016-01-15

break out the process of importing a branch into its own module, add a first simple test for it

300. By Daniel Holbach on 2016-01-16

fix clean up of imported articles, add a test

301. By Daniel Holbach on 2016-01-18

update TODO

Unmerged revisions

=== added file 'TODO'
--- TODO	1970-01-01 00:00:00 +0000
+++ TODO	2015-12-09 15:41:23 +0000
@@ -0,0 +1,7 @@
+- import pictures (LP: #1511677)
+- unpublished docs
+- add comment at top of RawHTML plugins, so users know not to edit them,
+  currently blocked on LP: #1523925
+- when deleting pages, check if last rev was made by python-api
+
+- 

=== modified file 'developer_portal/admin.py'
--- developer_portal/admin.py	2015-12-04 11:49:10 +0000
+++ developer_portal/admin.py	2015-12-09 15:41:23 +0000
@@ -4,7 +4,10 @@
 from reversion.admin import VersionAdmin
 
 from cms.extensions import TitleExtensionAdmin
-from .models import ExternalDocsBranch, SEOExtension
+from .models import (
+    ExternalDocsBranch, ExternalDocsBranchImportDirective,
+    ImportedArticle, SEOExtension
+)
 from django.core.management import call_command
 
 __all__ = (
@@ -13,7 +16,12 @@
 
 def import_selected_external_docs_branches(modeladmin, request, queryset):
     for branch in queryset:
+<<<<<<< TREE
         call_command('import-external-docs-branches', branch.docs_namespace)
+=======
+        branches.append(branch.origin)
+    call_command('import-external-docs-branches', *branches)
+>>>>>>> MERGE-SOURCE
     import_selected_external_docs_branches.short_description = \
         "Import selected branches"
 
@@ -35,12 +43,22 @@
 admin.site.register(Version, VersionAdmin)
 
 
[e-mail](ExternalDocsBranch)
 class ExternalDocsBranchAdmin(admin.ModelAdmin):
-    list_display = ('lp_origin', 'docs_namespace')
-    list_filter = ('lp_origin', 'docs_namespace')
+    list_display = ('origin', 'post_checkout_command', 'branch_name',)
+    list_filter = ('origin', 'post_checkout_command', 'branch_name',)
     actions = [import_selected_external_docs_branches]
 
-admin.site.register(ExternalDocsBranch, ExternalDocsBranchAdmin)
+
[e-mail](ExternalDocsBranchImportDirective)
+class ExternalDocsBranchImportDirectiveAdmin(admin.ModelAdmin):
+    pass
+
+
[e-mail](ImportedArticle)
+class ImportedArticleAdmin(admin.ModelAdmin):
+    pass
+
 
 class SEOExtensionAdmin(TitleExtensionAdmin):
     pass

=== modified file 'developer_portal/management/commands/import-external-docs-branches.py'
--- developer_portal/management/commands/import-external-docs-branches.py	2015-12-04 11:49:10 +0000
+++ developer_portal/management/commands/import-external-docs-branches.py	2015-12-09 15:41:23 +0000
@@ -1,23 +1,14 @@
 from django.core.management.base import BaseCommand
 from django.core.management import call_command
-from django.db import transaction
-
-from cms.api import create_page, add_plugin
-from cms.models import Page, Title
-from cms.utils import page_resolver
-
-from bs4 import BeautifulSoup
-import codecs
-import glob
+
+from ..importer.local_branch import LocalBranch, SnappyLocalBranch
+
+import datetime
 import logging
-import markdown
-import os
-import re
 import shutil
-import subprocess
-import sys
 import tempfile
 
+<<<<<<< TREE
 from developer_portal.models import ExternalDocsBranch
 
 DOCS_DIRNAME = 'docs'
@@ -295,6 +286,13 @@
             placeholder = page.placeholders.get()
             add_plugin(placeholder, 'RawHtmlPlugin', 'en', body=html)
     return page
+=======
+from developer_portal.models import (
+    ExternalDocsBranch,
+    ExternalDocsBranchImportDirective,
+    ImportedArticle,
+)
+>>>>>>> MERGE-SOURCE
 
 
 def import_branches(selection):
@@ -303,51 +301,39 @@
                       'ExternalDocsBranch table yet.')
         return
     tempdir = tempfile.mkdtemp()
-    db_actions = DBActions()
     for branch in ExternalDocsBranch.objects.filter(
-            docs_namespace__regex=selection):
-        checkout_location = os.path.join(
-            tempdir, os.path.basename(branch.docs_namespace))
-        sourcecode = SourceCode(branch.lp_origin, checkout_location)
-        if sourcecode.get() != 0:
-            logging.error(
-                'Could not check out branch "%s".' % branch.lp_origin)
-            if os.path.exists(checkout_location):
-                shutil.rmtree(checkout_location)
+            origin__regex=selection, active=True):
+        url = branch.origin
+        if url.startswith('lp:snappy') or \
+           'snappy' in url.split(':')[1].split('.git')[0].split('/'):
+            branch_class = SnappyLocalBranch
+        else:
+            branch_class = LocalBranch
+        local_branch = branch_class(tempdir, branch.origin, branch.branch_name,
+                                    branch.post_checkout_command)
+        if local_branch.get() != 0:
             break
-        if branch.lp_origin.startswith('lp:snappy') or \
-           'snappy' in branch.lp_origin.split(':')[1].split('.git')[0].split('/'):
-            local_branch = SnappyLocalBranch(checkout_location, branch,
-                                             db_actions)
-        else:
-            local_branch = LocalBranch(checkout_location, branch, db_actions)
-        local_branch.import_markdown()
-        local_branch.publish()
-        local_branch.remove_old_pages()
+        for directive in ExternalDocsBranchImportDirective.objects.filter(
+                external_docs_branch=branch):
+            local_branch.add_directive(directive.import_from,
+                                       directive.write_to)
+        local_branch.execute_import_directives()
+        imported_articles = local_branch.publish()
+        for imported_article in imported_articles:
+            ImportedArticle.objects.get_or_create(
+                branch=branch,
+                page=imported_article.page,
+                last_import=datetime.datetime.now())
+
+        # The import is done, now let's clean up.
+        for old_article in ImportedArticle.objects.filter(branch=branch):
+            if old_article.page not in [a.page for a in imported_articles]:
+                # XXX: check if last revision was made by 'python-api'?
+                old_article.page.delete()
     shutil.rmtree(tempdir)
-    db_actions.run()
-
-
-class SourceCode():
-    def __init__(self, branch_origin, checkout_location):
-        self.branch_origin = branch_origin
-        self.checkout_location = checkout_location
-
-    def get(self):
-        if self.branch_origin.startswith('lp:') and \
-           os.path.exists('/usr/bin/bzr'):
-            return subprocess.call([
-                'bzr', 'checkout', '--lightweight', self.branch_origin,
-                self.checkout_location])
-        if self.branch_origin.startswith('https://github.com') and \
-           self.branch_origin.endswith('.git') and \
-           os.path.exists('/usr/bin/git'):
-            return subprocess.call([
-                'git', 'clone', '-q', self.branch_origin,
-                self.checkout_location])
-        logging.error(
-            'Branch format "{}" not understood.'.format(self.branch_origin))
-        return 1
+
+    # https://stackoverflow.com/questions/33284171/
+    call_command('cms', 'fix-tree')
 
 
 class Command(BaseCommand):

=== added directory 'developer_portal/management/importer'
=== added file 'developer_portal/management/importer/__init__.py'
=== added file 'developer_portal/management/importer/article.py'
--- developer_portal/management/importer/article.py	1970-01-01 00:00:00 +0000
+++ developer_portal/management/importer/article.py	2015-12-09 15:41:23 +0000
@@ -0,0 +1,126 @@
+from bs4 import BeautifulSoup
+import codecs
+import logging
+import markdown
+import os
+import re
+
+from .publish import get_or_create_page, slugify
+
+
+class Article:
+    html = None
+    page = None
+    title = ""
+
+    def __init__(self, fn, write_to):
+        self.fn = fn
+        self.write_to = slugify(self.fn)
+        self.full_url = write_to
+        self.slug = os.path.basename(self.full_url)
+
+    def read(self):
+        with codecs.open(self.fn, 'r', encoding='utf-8') as f:
+            if self.fn.endswith('.md'):
+                self.html = markdown.markdown(
+                    f.read(),
+                    output_format='html5',
+                    extensions=['markdown.extensions.tables'])
+            elif self.fn.endswith('.html'):
+                self.html = f.read()
+            else:
+                logging.error("Don't know how to interpret '{}'.".format(
+                    self.fn))
+                return False
+        self.title = self._read_title()
+        self._remove_body_and_html_tags()
+        self._use_developer_site_style()
+        return True
+
+    def _read_title(self):
+        soup = BeautifulSoup(self.html, 'html5lib')
+        if soup.title:
+            return soup.title.text
+        if soup.h1:
+            return soup.h1.text
+        return slugify(self.fn).replace('-', ' ').title()
+
+    def _remove_body_and_html_tags(self):
+        self.html = re.sub(r"<html>\n\s<body>\n", "", self.html,
+                           flags=re.MULTILINE)
+        self.html = re.sub(r"\s<\/body>\n<\/html>", "", self.html,
+                           flags=re.MULTILINE)
+
+    def _use_developer_site_style(self):
+        begin = (u"<div class=\"row no-border\">"
+                 "\n<div class=\"eight-col\">\n")
+        end = u"</div>\n</div>"
+        self.html = begin + self.html + end
+        self.html = self.html.replace(
+            "<pre><code>",
+            "</div><div class=\"twelve-col\"><pre><code>")
+        self.html = self.html.replace(
+            "</code></pre>",
+            "</code></pre></div><div class=\"eight-col\">")
+
+    def replace_links(self, titles, url_map):
+        for title in titles:
+            local_md_fn = os.path.basename(title)
+            url = u'/'+url_map[title]
+            # Replace links of the form <a href="/path/somefile.md"> first
+            href = u"<a href=\"{}\">".format(url)
+            md_href = u"<a href=\"{}\">".format(local_md_fn)
+            self.html = self.html.replace(md_href, href)
+
+            # Now we can replace free-standing "somefile.md" references in
+            # the HTML
+            link = href + u"{}</a>".format(titles[title])
+            self.html = self.html.replace(local_md_fn, link)
+
+    def publish(self):
+        '''Publishes pages in their branch alias namespace.'''
+        page = get_or_create_page(
+            title=self.title, full_url=self.full_url, menu_title=self.title,
+            html=self.html)
+        self.page = page
+        self.page.publish('en')
+
+
+class SnappyArticle(Article):
+    release_alias = None
+
+    def get(self):
+        if not Article.get(self):
+            return False
+        self.release_alias = re.findall(r'snappy/guides/(\S+?)/\S+?',
+                                        self.full_url)[0]
+        self._make_snappy_mods()
+        return True
+
+    def _make_snappy_mods(self):
+        # Make sure the reader knows which documentation she is browsing
+        if self.release_alias != 'current':
+            before = (u"<div class=\"row no-border\">\n"
+                      "<div class=\"eight-col\">\n")
+            after = (u"<div class=\"row no-border\">\n"
+                     "<div class=\"box pull-three three-col\">"
+                     "<p>You are browsing the Snappy <code>%s</code> "
+                     "documentation.</p>"
+                     "<p><a href=\"/snappy/guides/current/%s\">"
+                     "Back to the latest stable release &rsaquo;"
+                     "</a></p></div>\n"
+                     "<div class=\"eight-col\">\n") % (self.release_alias,
+                                                       self.slug, )
+            self.html = self.html.replace(before, after)
+
+    def publish(self):
+        if self.release_alias == "current":
+            # Add a guides/<page> redirect to guides/current/<page>
+            page = get_or_create_page(
+                title=self.title,
+                full_url=self.full_url.replace('/current', ''),
+                redirect="/snappy/guides/current/{}".format(self.slug))
+            page.publish('en')
+        else:
+            self.title += " (%s)" % (self.release_alias,)
+        Article.publish(self)

=== added file 'developer_portal/management/importer/local_branch.py'
--- developer_portal/management/importer/local_branch.py	1970-01-01 00:00:00 +0000
+++ developer_portal/management/importer/local_branch.py	2015-12-09 15:41:23 +0000
@@ -0,0 +1,132 @@
+from .article import Article, SnappyArticle
+from .publish import get_or_create_page, slugify
+from .source import SourceCode
+
+import glob
+import logging
+import os
+import shutil
+
+
+class LocalBranch:
+    titles = {}
+    url_map = {}
+    index_doc_title = None
+    index_doc = None
+    release_alias = None
+
+    def __init__(self, tempdir, origin, branch_name, post_checkout_command):
+        self.origin = origin
+        self.branch_name = branch_name
+        self.post_checkout_command = post_checkout_command
+        self.checkout_location = os.path.join(
+            tempdir,
+            os.path.basename(self.origin.replace('.git', '')))
+        self.article_class = Article
+        self.directives = []
+        self.imported_articles = []
+
+    def get(self):
+        sourcecode = SourceCode(self.origin, self.checkout_location,
+                                self.branch_name, self.post_checkout_command)
+        if sourcecode.get() != 0:
+            logging.error(
+                'Could not check out branch "{}".'.format(self.origin))
+            shutil.rmtree(self.checkout_location)
+            return 1
+        return 0
+
+    def add_directive(self, import_from, write_to):
+        self.directives += [
+            {
+                'import_from': os.path.join(self.checkout_location,
+                                            import_from),
+                'write_to': write_to
+            }
+        ]
+
+    def execute_import_directives(self):
+        import_list = []
+        # Import single files first
+        for directive in [d for d in self.directives
+                          if os.path.isfile(d['import_from'])]:
+            import_list += [
+                (directive['import_from'], directive['write_to'])
+            ]
+        # Import directories next
+        for directive in [d for d in self.directives
+                          if os.path.isdir(d['import_from'])]:
+            for fn in glob.glob('{}/*'.format(directive['import_from'])):
+                if fn not in [a[0] for a in import_list]:
+                    import_list += [
+                        (fn, os.path.join(directive['write_to'], slugify(fn)))
+                    ]
+            # If we import into a namespace and don't have an index doc,
+            # we need to write one.
+            if directive['write_to'] not in [x[1] for x in import_list]:
+                self.index_doc = directive['write_to']
+        # The actual import
+        for entry in import_list:
+            article = self._read_article(entry[0], entry[1])
+            if article:
+                self.imported_articles += [article]
+                self.titles[article.fn] = article.title
+                self.url_map[article.fn] = article.full_url
+        for article in self.imported_articles:
+            article.replace_links(self.titles, self.url_map)
+        if self.index_doc:
+            self._create_fake_index_docs()
+
+    def _read_article(self, fn, write_to):
+        article = self.article_class(fn, write_to)
+        if article.read():
+            return article
+        return None
+
+    def publish(self):
+        for article in self.imported_articles:
+            article.publish()
+        return self.imported_articles
+
+    def _create_fake_index_docs(self):
+        '''Creates a fake index page at the top of the branches
+           docs namespace.'''
+
+        if self.index_doc.endswith('current'):
+            redirect = '/snappy/guides'
+        else:
+            redirect = None
+        list_pages = ''
+        for article in [a for a
+                        in self.imported_articles
+                        if a.full_url.startswith(self.index_doc)]:
+            list_pages += '<li><a href=\"{}\">{}</a></li>'.format(
+                os.path.basename(article.full_url), article.title)
+        landing = (
+            u'<div class=\"row\"><div class=\"eight-col\">\n'
+            '<p>This section contains documentation for the '
+            '<code>{}</code> Snappy branch.</p>'
+            '<p><ul class=\"list-ubuntu\">{}</ul></p>\n'
+            '<p>Auto-imported from <a '
+            'href=\"https://github.com/ubuntu-core/snappy\">%s</a>.</p>\n'
+            '</div></div>'.format(self.release_alias, list_pages,
+                                  self.origin))
+        page = get_or_create_page(
+            title=self.index_doc_title, full_url=self.index_doc,
+            in_navigation=False, redirect=redirect, html=landing,
+            menu_title=None)
+        page.publish('en')
+
+
+class SnappyLocalBranch(LocalBranch):
+    def __init__(self, tempdir, origin, branch_name, post_checkout_command):
+        LocalBranch.__init__(self, tempdir, origin, branch_name,
+                             post_checkout_command)
+        self.article_class = SnappyArticle
+        self.index_doc_title = 'Snappy documentation'
+
+    def _create_fake_index_docs(self):
+        self.release_alias = os.path.basename(self.index_doc)
+        if not self.index_doc.endswith('current'):
+            self.index_doc_title += ' ({})'.format(self.release_alias)
+        LocalBranch._create_fake_index_docs(self)

=== added file 'developer_portal/management/importer/publish.py'
--- developer_portal/management/importer/publish.py	1970-01-01 00:00:00 +0000
+++ developer_portal/management/importer/publish.py	2015-12-09 15:41:23 +0000
@@ -0,0 +1,60 @@
+from cms.api import create_page, add_plugin
+from cms.models import Title
+
+import logging
+import os
+import sys
+
+# XXX: use this once we RawHTML plugins don't strip comments (LP: #1523925)
+START_TEXT = """
+<!--
+branch id: {}
+
+THIS PAGE IS AUTOMATICALLY UPDATED.
+DON'T EDIT IT - CHANGES WILL BE OVERWRITTEN.
+-->
+"""
+
+
+def slugify(filename):
+    return os.path.basename(filename).replace('.md', '').replace('.html', '')
+
+
+def get_or_create_page(title, full_url, menu_title=None,
+                       in_navigation=True, redirect=None, html=None):
+    # First check if pages already exist.
+    pages = Title.objects.select_related('page').filter(path__regex=full_url)
+    if pages:
+        page = pages[0].page
+        page.title = title
+        page.publisher_is_draft = True
+        page.menu_title = menu_title
+        page.in_navigation = in_navigation
+        page.redirect = redirect
+        if html:
+            # We create the page, so we know there's just one placeholder
+            placeholder = page.placeholders.all()[0]
+            if placeholder.get_plugins():
+                plugin = placeholder.get_plugins()[0].get_plugin_instance()[0]
+                plugin.body = html
+                plugin.save()
+            else:
+                add_plugin(placeholder, 'RawHtmlPlugin', 'en', body=html)
+    else:
+        parent_pages = Title.objects.select_related('page').filter(
+            path__regex=os.path.dirname(full_url))
+        if not parent_pages:
+            logging.error('Parent {} not found.'.format(
+                os.path.dirname(full_url)))
+            sys.exit(1)
+        parent = parent_pages[0].page
+
+        slug = os.path.basename(full_url)
+        page = create_page(
+            title, "default.html", "en", slug=slug, parent=parent,
+            menu_title=menu_title, in_navigation=in_navigation,
+            position="last-child", redirect=redirect)
+        if html:
+            placeholder = page.placeholders.get()
+            add_plugin(placeholder, 'RawHtmlPlugin', 'en', body=html)
+    return page

=== added file 'developer_portal/management/importer/source.py'
--- developer_portal/management/importer/source.py	1970-01-01 00:00:00 +0000
+++ developer_portal/management/importer/source.py	2015-12-09 15:41:23 +0000
@@ -0,0 +1,51 @@
+import logging
+import os
+import subprocess
+
+
+class SourceCode():
+    def __init__(self, origin, checkout_location, branch_name,
+                 post_checkout_command):
+        self.origin = origin
+        self.checkout_location = checkout_location
+        self.branch_name = branch_name
+        self.post_checkout_command = post_checkout_command
+
+    def get(self):
+        res = self._get_branch()
+        if res == 0 and self.post_checkout_command:
+            res = self._post_checkout()
+            return res
+        return res
+
+    def _get_branch(self):
+        if self.origin.startswith('lp:') and \
+           os.path.exists('/usr/bin/bzr'):
+            return subprocess.call([
+                'bzr', 'checkout', '--lightweight', self.origin,
+                self.checkout_location])
+        if self.origin.startswith('https://github.com') and \
+           self.origin.endswith('.git') and \
+           os.path.exists('/usr/bin/git'):
+            retcode = subprocess.call([
+                'git', 'clone', '-q', self.origin,
+                self.checkout_location])
+            if retcode == 0 and self.branch_name:
+                pwd = os.getcwd()
+                os.chdir(self.checkout_location)
+                retcode = subprocess.call(['git', 'checkout',
+                                           self.branch_name])
+                os.chdir(pwd)
+            return retcode
+        logging.error(
+            'Branch format "{}" not understood.'.format(self.origin))
+        return 1
+
+    def _post_checkout(self):
+        process = subprocess.Popen(self.post_checkout_command.split(),
+                                   stdout=subprocess.PIPE)
+        (out, err) = process.communicate()
+        retcode = process.wait()
+        if retcode != 0:
+            logging.error(out)
+        return retcode

=== added file 'developer_portal/migrations/0002_rework_importer.py'
--- developer_portal/migrations/0002_rework_importer.py	1970-01-01 00:00:00 +0000
+++ developer_portal/migrations/0002_rework_importer.py	2015-12-09 15:41:23 +0000
@@ -0,0 +1,59 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('developer_portal', '0001_initial'),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='ExternalDocsBranchImportDirective',
+            fields=[
+                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
+                ('import_from', models.CharField(help_text='File or directory to import from the branch. Ie: "docs/intro.md" (file) or "docs" (complete directory), etc.', max_length=150)),
+                ('write_to', models.CharField(help_text='Article URL (for a specific file) or article namespace for a directory or a set of files.', max_length=150)),
+            ],
+        ),
+        migrations.AlterModelOptions(
+            name='externaldocsbranch',
+            options={'verbose_name': 'external docs branch', 'verbose_name_plural': 'external docs branches'},
+        ),
+        migrations.RenameField(
+            model_name='externaldocsbranch',
+            old_name='lp_origin',
+            new_name='origin',
+        ),
+        migrations.RemoveField(
+            model_name='externaldocsbranch',
+            name='docs_namespace',
+        ),
+        migrations.RemoveField(
+            model_name='externaldocsbranch',
+            name='index_doc',
+        ),
+        migrations.AddField(
+            model_name='externaldocsbranch',
+            name='active',
+            field=models.BooleanField(default=True),
+        ),
+        migrations.AddField(
+            model_name='externaldocsbranch',
+            name='branch_name',
+            field=models.CharField(help_text='For use with git branches, ie: "master" or "15.04" or "1.x".', max_length=200, blank=True),
+        ),
+        migrations.AddField(
+            model_name='externaldocsbranch',
+            name='post_checkout_command',
+            field=models.CharField(help_text='Command to run after checkout of the branch.', max_length=100, blank=True),
+        ),
+        migrations.AddField(
+            model_name='externaldocsbranchimportdirective',
+            name='external_docs_branch',
+            field=models.ForeignKey(to='developer_portal.ExternalDocsBranch'),
+        ),
+    ]

=== added file 'developer_portal/migrations/0003_track_imported_articles.py'
--- developer_portal/migrations/0003_track_imported_articles.py	1970-01-01 00:00:00 +0000
+++ developer_portal/migrations/0003_track_imported_articles.py	2015-12-09 15:41:23 +0000
@@ -0,0 +1,24 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('cms', '0013_urlconfrevision'),
+        ('developer_portal', '0002_rework_importer'),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='ImportedArticle',
+            fields=[
+                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
+                ('last_import', models.DateTimeField(help_text='Datetime of last import.', verbose_name='Datetime')),
+                ('branch', models.ForeignKey(to='developer_portal.ExternalDocsBranch')),
+                ('page', models.ForeignKey(to='cms.Page')),
+            ],
+        ),
+    ]

=== modified file 'developer_portal/models.py'
--- developer_portal/models.py	2015-11-02 16:47:26 +0000
+++ developer_portal/models.py	2015-12-09 15:41:23 +0000
@@ -1,9 +1,9 @@
 from django.db import models
 from django.utils.translation import ugettext_lazy as _
 
-from cms.models import CMSPlugin
 from cms.extensions import TitleExtension
 from cms.extensions.extension_pool import extension_pool
+from cms.models import Page
 from djangocms_text_ckeditor.html import extract_images
 from djangocms_text_ckeditor.models import AbstractText
 
@@ -21,23 +21,54 @@
 
 
 class ExternalDocsBranch(models.Model):
-    # We originally assumed that branches would also live in LP,
-    # well, we were wrong, but let's keep the name around. It's
-    # no use having a schema/data migration just for this.
-    lp_origin = models.CharField(
+    origin = models.CharField(
         max_length=200,
         help_text=_('External branch location, ie: lp:snappy/15.04 or '
                     'https://github.com/ubuntu-core/snappy.git'))
-    docs_namespace = models.CharField(
-        max_length=120,
-        help_text=_('Path alias we want to use for the docs, '
-                    'ie "snappy/guides/15.04" or '
-                    '"snappy/guides/latest", etc.'))
-    index_doc = models.CharField(
-        max_length=120,
-        help_text=_('File name of doc to be used as index document, '
-                    'ie "intro.md"'),
-        blank=True)
+    branch_name = models.CharField(
+        max_length=200,
+        help_text=_('For use with git branches, ie: "master" or "15.04" '
+                    'or "1.x".'),
+        blank=True)
+    post_checkout_command = models.CharField(
+        max_length=100,
+        help_text=_('Command to run after checkout of the branch.'),
+        blank=True)
+    active = models.BooleanField(default=True)
+
+    def __str__(self):
+        if self.branch_name:
+            return "{} {}".format(self.origin, self.branch_name)
+        return "{}".format(self.origin)
+
+    class Meta:
+        verbose_name = "external docs branch"
+        verbose_name_plural = "external docs branches"
+
+
+class ExternalDocsBranchImportDirective(models.Model):
+    external_docs_branch = models.ForeignKey(ExternalDocsBranch)
+    import_from = models.CharField(
+        max_length=150,
+        help_text=_('File or directory to import from the branch. '
+                    'Ie: "docs/intro.md" (file) or '
+                    '"docs" (complete directory), etc.'))
+    write_to = models.CharField(
+        max_length=150,
+        help_text=_('Article URL (for a specific file) or article namespace '
+                    'for a directory or a set of files.'))
+
+    def __str__(self):
+        return "{} -- {}".format(self.external_docs_branch.origin,
+                                 self.import_from)
+
+
+class ImportedArticle(models.Model):
+    page = models.ForeignKey(Page)
+    branch = models.ForeignKey(ExternalDocsBranch)
+    last_import = models.DateTimeField(
+        _('Datetime'), help_text=_('Datetime of last import.'))
+
 
 class SEOExtension(TitleExtension):
     keywords = models.CharField(max_length=256)

=== modified file 'developer_portal/settings.py'
--- developer_portal/settings.py	2015-12-04 16:09:27 +0000
+++ developer_portal/settings.py	2015-12-09 15:41:23 +0000
@@ -318,6 +318,26 @@
     #'PAGINATE_BY': 10,
 }
 
+<<<<<<< TREE
+=======
+MIGRATION_MODULES = {
+    'cms': 'cms.migrations',
+    'cmsplugin_zinnia': 'cmsplugin_zinnia.migrations',
+    'djangocms_link': 'djangocms_link.migrations',
+    'djangocms_picture': 'djangocms_picture.migrations',
+    'djangocms_snippet': 'djangocms_snippet.migrations',
+    'djangocms_text_ckeditor': 'djangocms_text_ckeditor.migrations',
+    'djangocms_video': 'djangocms_video.migrations',
+    'django_comments': 'django_comments.migrations',
+    'menus': 'menus.migrations',
+    'rest_framework.authtoken': 'rest_framework.authtoken.migrations',
+    'reversion': 'reversion.migrations',
+    'tagging': 'tagging.migrations',
+    'taggit': 'taggit.migrations',
+    'zinnia': 'zinnia.migrations',
+}
+
+>>>>>>> MERGE-SOURCE
 LOGGING = {
     'version': 1,
     'disable_existing_loggers': False,

=== modified file 'requirements.txt'
--- requirements.txt	2015-12-04 11:49:10 +0000
+++ requirements.txt	2015-12-09 15:41:23 +0000
@@ -1,3 +1,4 @@
+<<<<<<< TREE
 oslo.config==1.6.0
 oslo.i18n==1.2.0
 oslo.serialization==1.2.0
@@ -8,12 +9,25 @@
 Pillow==2.6.1
 beautifulsoup4==4.4.0
 cmsplugin-zinnia==0.6
+=======
+Django==1.8.7
+django-template-debug==0.3.5
+oslo.config==3.0.0
+oslo.i18n==2.7.0
+oslo.serialization==2.0.0
+oslo.utils==2.8.0
+Pillow==2.9.0
+cmsplugin-zinnia==0.8
+Markdown==2.6.5
+beautifulsoup4==4.4.1
+>>>>>>> MERGE-SOURCE
 dj-database-url==0.3.0
 django-admin-enhancer==0.1.3.1
 django-appconf==0.6
 django-blog-zinnia==0.14.2
 django-ckeditor==4.4.7
 django-ckeditor-updated==4.4.4
+<<<<<<< TREE
 django-classy-tags==0.5.1
 django-cms==3.0.6
 django-contrib-comments==1.5
@@ -25,23 +39,56 @@
 django-polymorphic==0.6
 django-reversion==1.8.4
 django-sekizai==0.7
+=======
+django-classy-tags==0.6.2
+django-cms==3.2.0
+django-contrib-comments==1.6.1
+django-meta==0.3.1
+django-meta-mixin==0.2.1
+django-missing==0.1.15
+django-parler==1.5.1
+django-polymorphic==0.7.2
+django-reversion==1.9.3
+django-sekizai==0.8.2
+>>>>>>> MERGE-SOURCE
 django-swiftstorage==1.1.0
+<<<<<<< TREE
 django-tagging==0.3.3
 django-taggit==0.12.2
 django-taggit-autosuggest==0.2.5
+=======
+django-tagging==0.4
+django-taggit==0.17.5
+django-taggit-autosuggest==0.2.8
+>>>>>>> MERGE-SOURCE
 django-taggit-templatetags==0.2.5
 django-templatetag-sugar==1.0
 django-xmlrpc==0.1.5
+<<<<<<< TREE
 djangocms-admin-style==0.2.2
 djangocms-link==1.5
 djangocms-picture==0.1
 djangocms-snippet==1.3
 djangocms-text-ckeditor==2.4.2
+=======
+djangocms-admin-style==1.0.7
+djangocms-link==1.7.1
+djangocms-picture==0.2.0
+djangocms-snippet==1.7.1
+djangocms-text-ckeditor==2.8.0
+>>>>>>> MERGE-SOURCE
 djangocms-utils==0.9.5
+<<<<<<< TREE
 djangocms-video==0.1
 python-keystoneclient==0.11.2
 python-swiftclient==2.3.1
 pytz==2014.7
+=======
+djangocms-video==0.2.0
+python-keystoneclient==1.3.3
+python-swiftclient==2.6.0
+pytz==2015.7
+>>>>>>> MERGE-SOURCE
 simple-translation==0.8.6
 simplejson==3.6.5
 wsgiref==0.1.2