Merge lp:~dholbach/developer-ubuntu-com/rework-importer into lp:~developer-ubuntu-com-dev/developer-ubuntu-com/django-1.8-cms-2.3

[Daniel Holbach]

[Daniel Holbach (dholbach)]

sqlite> select * from developer_portal_externaldocsbranch;
1|https://github.com/ubuntu-core/snapcraft.git|1|master|
2|https://github.com/ubuntu-core/snapcraft.git|1|1.x|
3|https://github.com/ubuntu-core/snappy.git|1|master|
4|https://github.com/ubuntu-core/snappy.git|1|15.04|
sqlite> select * from developer_portal_externaldocsbranchimportdirective;
1|README.md|snappy/build-apps/devel|1
2|docs|snappy/build-apps/devel|1
3|README.md|snappy/build-apps/|2
4|docs|snappy/build-apps/|2
5|docs|snappy/guides/devel|3
6|docs|snappy/guides/current|4
7|HACKING.md|snappy/build-apps/hacking|1
sqlite>

[lp:~dholbach/developer-ubuntu-com/rework-importer]

252. By Daniel Holbach on 2015-12-22

break up tests into individual files

253. By Daniel Holbach on 2015-12-22

update TODO

254. By Daniel Holbach on 2016-01-05

improve create_repo helper function

255. By Daniel Holbach on 2016-01-05

allow SourceCode._get_branch to use local docs (ie for testing)

256. By Daniel Holbach on 2016-01-05

first cut at testing local links

257. By Daniel Holbach on 2016-01-05

fix simple link rewrite functionality and test

258. By Daniel Holbach on 2016-01-06

move md_importer into its own app

259. By Daniel Holbach on 2016-01-06

add migration for md_importer

260. By Daniel Holbach on 2016-01-06

merge lp:~dholbach/developer-ubuntu-com/django-cms-update

261. By Daniel Holbach on 2016-01-06

make directory structure clearer

262. By Daniel Holbach on 2016-01-06

adapt to new name of management command

263. By Daniel Holbach on 2016-01-06

add more readable error message

264. By Daniel Holbach on 2016-01-08

add more local test data (imported just the docs from snappy's and snapcraft's git master branches)

265. By Daniel Holbach on 2016-01-08

WIP commit:
- move and update TODO file
- define some global values for the importer centrally
- don't sys.exit() when adding pages to the db
- add debug messages
- break _find_parent() into its own function
- mostly make use of local test data
- drop singleton, use separate test classes instead
- add more assertions in tests

266. By Daniel Holbach on 2016-01-08

remove debug prints, make Repo class variables actual members

267. By Daniel Holbach on 2016-01-08

- make pep8 happy, remove unnecessary imports
- use TestLocalBranchImport as base class for almost everything,
  reduces a lot of duplication

268. By Daniel Holbach on 2016-01-08

remove unnecessary code, fix publishing of pages in tests

269. By Daniel Holbach on 2016-01-08

fix test_link_rewrite by fixing the URL

270. By Daniel Holbach on 2016-01-08

make Article class variables actual members

271. By Daniel Holbach on 2016-01-08

add new test to check links in snapcraft import, move link checking function into TestLocalBranchImport

272. By Daniel Holbach on 2016-01-11

merge lp:~developer-ubuntu-com-dev/developer-ubuntu-com/django-1.8-cms-2.3

273. By Daniel Holbach on 2016-01-11

allow empty strings in import directives

274. By Daniel Holbach on 2016-01-11

update migration

275. By Daniel Holbach on 2016-01-11

check if articles were added below home

276. By Daniel Holbach on 2016-01-11

remove unnecessary call to set a page's publisher_is_draft bit

277. By Daniel Holbach on 2016-01-11

give article a .publish() method which gives back the page's public_object

278. By Daniel Holbach on 2016-01-11

rename home_page to root_page, use cms.test_utils.testcases.CMSTestCase, make sure we use public_object wherever possible

279. By Daniel Holbach on 2016-01-11

update tests accordingly

280. By Daniel Holbach on 2016-01-11

disregard anchors, make link checking more flexible (ie not only check for intro.md, but also things like docs/intro.md

281. By Daniel Holbach on 2016-01-11

make URL work, even if LANG is already part of it

282. By Daniel Holbach on 2016-01-11

links look like they're working now

283. By Daniel Holbach on 2016-01-11

add test with a broken link, add convenience function is_local_link, modify tests

284. By Daniel Holbach on 2016-01-11

import doc fix from https://bugs.launchpad.net/developer-ubuntu-com/+bug/1531200

285. By Daniel Holbach on 2016-01-11

bug fixed

286. By Daniel Holbach on 2016-01-12

stop import if local images are found, add tests

287. By Daniel Holbach on 2016-01-12

update TODO

288. By Daniel Holbach on 2016-01-12

add test to see if importing the same content twice results in the same number of pages

289. By Daniel Holbach on 2016-01-15

when replacing links, only update HTML if things actually change, only publish if page is dirty

290. By Daniel Holbach on 2016-01-15

only update page attributes if they actually change, only update the text plugin if the html actually changes, veryify with djangocms_text_ckeditor.html.clean_html, add a text plugin to a page, even if html is empty

291. By Daniel Holbach on 2016-01-15

add test to check if running an import twice will update the articles in question

292. By Daniel Holbach on 2016-01-15

update TODO

293. By Daniel Holbach on 2016-01-15

add misc tests - forgot to 'bzr add'

294. By Daniel Holbach on 2016-01-15

make update-mtemplate work again, simplify it

295. By Daniel Holbach on 2016-01-15

update .pot file

296. By Daniel Holbach on 2016-01-15

add actual page object to repo.pages

297. By Daniel Holbach on 2016-01-15

use UTC for ImportedArticle.last_import, simplify ImportedArticle cleanup

298. By Daniel Holbach on 2016-01-15

use repo instead of branch consistently

299. By Daniel Holbach on 2016-01-15

break out the process of importing a branch into its own module, add a first simple test for it

300. By Daniel Holbach on 2016-01-16

fix clean up of imported articles, add a test

301. By Daniel Holbach on 2016-01-18

update TODO

Unmerged revisions

301. By Daniel Holbach on 2016-01-18

update TODO

300. By Daniel Holbach on 2016-01-16

fix clean up of imported articles, add a test

299. By Daniel Holbach on 2016-01-15

break out the process of importing a branch into its own module, add a first simple test for it

298. By Daniel Holbach on 2016-01-15

use repo instead of branch consistently

297. By Daniel Holbach on 2016-01-15

use UTC for ImportedArticle.last_import, simplify ImportedArticle cleanup

296. By Daniel Holbach on 2016-01-15

add actual page object to repo.pages

295. By Daniel Holbach on 2016-01-15

update .pot file

294. By Daniel Holbach on 2016-01-15

make update-mtemplate work again, simplify it

293. By Daniel Holbach on 2016-01-15

add misc tests - forgot to 'bzr add'

292. By Daniel Holbach on 2016-01-15

update TODO

=== added file 'TODO'
--- TODO	1970-01-01 00:00:00 +0000
+++ TODO	2016-01-06 12:40:31 +0000
@@ -0,0 +1,13 @@
+- fix branch indicator (devel vs. stable, 1.x vs devel, etc.)
+
+- fix links
+- add tests for links
+
+- check if https://bugs.launchpad.net/developer-ubuntu-com/+bug/1531200 works
+
+- import pictures (LP: #1511677)
+- unpublished docs: some articles show up with a blue circle, so 
+  unpublished although they're accessible(?)
+- add comment at top of RawHTML plugins, so users know not to edit them,
+  currently blocked on LP: #1523925
+- if article already exists, check if HTML actually changed, if not, don't update.
 
=== modified file 'developer_portal/admin.py'
--- developer_portal/admin.py	2015-11-02 12:17:27 +0000
+++ developer_portal/admin.py	2016-01-06 12:40:31 +0000
@@ -4,22 +4,12 @@
 from reversion.admin import VersionAdmin
 
 from cms.extensions import TitleExtensionAdmin
-from .models import ExternalDocsBranch, SEOExtension
-from django.core.management import call_command
+from .models import SEOExtension
 
 __all__ = (
 )
 
 
-def import_selected_external_docs_branches(modeladmin, request, queryset):
-    branches = []
-    for branch in queryset:
-        branches.append(branch.docs_namespace)
-    call_command('import-external-docs-branches', *branches)
-    import_selected_external_docs_branches.short_description = \
-        "Import selected branches"
-
-
 class RevisionAdmin(admin.ModelAdmin):
     list_display = ('date_created', 'user', 'comment')
     list_display_links = ('date_created', )
@@ -37,13 +27,6 @@
 admin.site.register(Version, VersionAdmin)
 
 
-class ExternalDocsBranchAdmin(admin.ModelAdmin):
-    list_display = ('lp_origin', 'docs_namespace')
-    list_filter = ('lp_origin', 'docs_namespace')
-    actions = [import_selected_external_docs_branches]
-
-admin.site.register(ExternalDocsBranch, ExternalDocsBranchAdmin)
-
 class SEOExtensionAdmin(TitleExtensionAdmin):
     pass
 
 
=== modified file 'developer_portal/migrations/0001_initial.py'
--- developer_portal/migrations/0001_initial.py	2015-11-17 13:38:11 +0000
+++ developer_portal/migrations/0001_initial.py	2016-01-06 12:40:31 +0000
@@ -11,15 +11,6 @@
 
     operations = [
         migrations.CreateModel(
-            name='ExternalDocsBranch',
-            fields=[
-                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
-                ('lp_origin', models.CharField(help_text='External branch location, ie: lp:snappy/15.04 or https://github.com/ubuntu-core/snappy.git', max_length=200)),
-                ('docs_namespace', models.CharField(help_text='Path alias we want to use for the docs, ie "snappy/guides/15.04" or "snappy/guides/latest", etc.', max_length=120)),
-                ('index_doc', models.CharField(help_text='File name of doc to be used as index document, ie "intro.md"', max_length=120, blank=True)),
-            ],
-        ),
-        migrations.CreateModel(
             name='RawHtml',
             fields=[
                 ('cmsplugin_ptr', models.OneToOneField(parent_link=True, auto_created=True, primary_key=True, serialize=False, to='cms.CMSPlugin')),
 
=== modified file 'developer_portal/models.py'
--- developer_portal/models.py	2015-11-02 16:47:26 +0000
+++ developer_portal/models.py	2016-01-06 12:40:31 +0000
@@ -1,7 +1,5 @@
 from django.db import models
-from django.utils.translation import ugettext_lazy as _
 
-from cms.models import CMSPlugin
 from cms.extensions import TitleExtension
 from cms.extensions.extension_pool import extension_pool
 from djangocms_text_ckeditor.html import extract_images
@@ -20,25 +18,6 @@
         AbstractText.save(self, *args, **kwargs)
 
 
-class ExternalDocsBranch(models.Model):
-    # We originally assumed that branches would also live in LP,
-    # well, we were wrong, but let's keep the name around. It's
-    # no use having a schema/data migration just for this.
-    lp_origin = models.CharField(
-        max_length=200,
-        help_text=_('External branch location, ie: lp:snappy/15.04 or '
-                    'https://github.com/ubuntu-core/snappy.git'))
-    docs_namespace = models.CharField(
-        max_length=120,
-        help_text=_('Path alias we want to use for the docs, '
-                    'ie "snappy/guides/15.04" or '
-                    '"snappy/guides/latest", etc.'))
-    index_doc = models.CharField(
-        max_length=120,
-        help_text=_('File name of doc to be used as index document, '
-                    'ie "intro.md"'),
-        blank=True)
-
 class SEOExtension(TitleExtension):
     keywords = models.CharField(max_length=256)
 
 
=== modified file 'developer_portal/settings.py'
--- developer_portal/settings.py	2015-12-11 16:10:37 +0000
+++ developer_portal/settings.py	2016-01-06 12:40:31 +0000
@@ -76,6 +76,8 @@
     'store_data',
 
     'api_docs',
+
+    'md_importer',
 ]
 
 MIDDLEWARE_CLASSES = (
 
=== modified file 'locale/de.po'
--- locale/de.po	2015-11-24 05:38:32 +0000
+++ locale/de.po	2016-01-06 12:40:31 +0000
@@ -15,8 +15,8 @@
 "Content-Type: text/plain; charset=UTF-8\n"
 "Content-Transfer-Encoding: 8bit\n"
 "Plural-Forms: nplurals=2; plural=n != 1;\n"
-"X-Launchpad-Export-Date: 2015-11-24 05:38+0000\n"
-"X-Generator: Launchpad (build 17850)\n"
+"X-Launchpad-Export-Date: 2015-12-03 06:16+0000\n"
+"X-Generator: Launchpad (build 17856)\n"
 
 #: developer_portal/cms_plugins.py:10
 msgid "Raw HTML"
 
=== modified file 'locale/en_GB.po'
--- locale/en_GB.po	2015-11-24 05:38:32 +0000
+++ locale/en_GB.po	2016-01-06 12:40:31 +0000
@@ -15,8 +15,8 @@
 "Content-Type: text/plain; charset=UTF-8\n"
 "Content-Transfer-Encoding: 8bit\n"
 "Plural-Forms: nplurals=2; plural=n != 1;\n"
-"X-Launchpad-Export-Date: 2015-11-24 05:38+0000\n"
-"X-Generator: Launchpad (build 17850)\n"
+"X-Launchpad-Export-Date: 2015-12-03 06:16+0000\n"
+"X-Generator: Launchpad (build 17856)\n"
 
 #: developer_portal/cms_plugins.py:10
 msgid "Raw HTML"
 
=== modified file 'locale/es.po'
--- locale/es.po	2015-11-24 05:38:32 +0000
+++ locale/es.po	2016-01-06 12:40:31 +0000
@@ -15,8 +15,8 @@
 "Content-Type: text/plain; charset=UTF-8\n"
 "Content-Transfer-Encoding: 8bit\n"
 "Plural-Forms: nplurals=2; plural=n != 1;\n"
-"X-Launchpad-Export-Date: 2015-11-24 05:38+0000\n"
-"X-Generator: Launchpad (build 17850)\n"
+"X-Launchpad-Export-Date: 2015-12-03 06:16+0000\n"
+"X-Generator: Launchpad (build 17856)\n"
 
 #: developer_portal/cms_plugins.py:10
 msgid "Raw HTML"
 
=== modified file 'locale/ug.po'
--- locale/ug.po	2015-11-24 05:38:32 +0000
+++ locale/ug.po	2016-01-06 12:40:31 +0000
@@ -15,8 +15,8 @@
 "Content-Type: text/plain; charset=UTF-8\n"
 "Content-Transfer-Encoding: 8bit\n"
 "Plural-Forms: nplurals=1; plural=0;\n"
-"X-Launchpad-Export-Date: 2015-11-24 05:38+0000\n"
-"X-Generator: Launchpad (build 17850)\n"
+"X-Launchpad-Export-Date: 2015-12-03 06:16+0000\n"
+"X-Generator: Launchpad (build 17856)\n"
 
 #: developer_portal/cms_plugins.py:10
 msgid "Raw HTML"
 
=== modified file 'locale/zh_CN.po'
--- locale/zh_CN.po	2015-11-24 05:38:32 +0000
+++ locale/zh_CN.po	2016-01-06 12:40:31 +0000
@@ -15,8 +15,8 @@
 "Content-Type: text/plain; charset=UTF-8\n"
 "Content-Transfer-Encoding: 8bit\n"
 "Plural-Forms: nplurals=1; plural=0;\n"
-"X-Launchpad-Export-Date: 2015-11-24 05:38+0000\n"
-"X-Generator: Launchpad (build 17850)\n"
+"X-Launchpad-Export-Date: 2015-12-03 06:16+0000\n"
+"X-Generator: Launchpad (build 17856)\n"
 
 #: developer_portal/cms_plugins.py:10
 msgid "Raw HTML"
 
=== added directory 'md_importer'
=== added file 'md_importer/__init__.py'
=== added file 'md_importer/admin.py'
--- md_importer/admin.py	1970-01-01 00:00:00 +0000
+++ md_importer/admin.py	2016-01-06 12:40:31 +0000
@@ -0,0 +1,36 @@
+from django.contrib import admin
+
+from .models import (
+    ExternalDocsBranch, ExternalDocsBranchImportDirective,
+    ImportedArticle,
+)
+from django.core.management import call_command
+
+__all__ = (
+)
+
+
+def import_selected_external_docs_branches(modeladmin, request, queryset):
+    branches = []
+    for branch in queryset:
+        branches.append(branch.origin)
+    call_command('import-external-docs-branches', *branches)
+    import_selected_external_docs_branches.short_description = \
+        "Import selected branches"
+
+
[e-mail](ExternalDocsBranch)
+class ExternalDocsBranchAdmin(admin.ModelAdmin):
+    list_display = ('origin', 'post_checkout_command', 'branch_name',)
+    list_filter = ('origin', 'post_checkout_command', 'branch_name',)
+    actions = [import_selected_external_docs_branches]
+
+
[e-mail](ExternalDocsBranchImportDirective)
+class ExternalDocsBranchImportDirectiveAdmin(admin.ModelAdmin):
+    pass
+
+
[e-mail](ImportedArticle)
+class ImportedArticleAdmin(admin.ModelAdmin):
+    pass
 
=== added directory 'md_importer/management'
=== added file 'md_importer/management/__init__.py'
=== added directory 'md_importer/management/commands'
=== added file 'md_importer/management/commands/__init__.py'
=== renamed file 'developer_portal/management/commands/import-external-docs-branches.py' => 'md_importer/management/commands/import-external-docs-branches.py'
--- developer_portal/management/commands/import-external-docs-branches.py	2015-11-06 09:48:07 +0000
+++ md_importer/management/commands/import-external-docs-branches.py	2016-01-06 12:40:31 +0000
@@ -1,300 +1,18 @@
 from django.core.management.base import BaseCommand
 from django.core.management import call_command
-from django.db import transaction
-
-from cms.api import create_page, add_plugin
-from cms.models import Page, Title
-from cms.utils import page_resolver
-
-from bs4 import BeautifulSoup
-import codecs
-import glob
+
+from ..importer.repo import create_repo
+
+import datetime
 import logging
-import markdown
-import os
-import re
 import shutil
-import subprocess
-import sys
 import tempfile
 
-from developer_portal.models import ExternalDocsBranch
-
-DOCS_DIRNAME = 'docs'
-
-
-class DBActions:
-    added_pages = []
-    removed_pages = []
-
-    def add_page(self, **kwargs):
-        self.added_pages += [kwargs]
-
-    def remove_page(self, page_id):
-        self.removed_pages += [page_id]
-
-    @transaction.atomic()
-    def run(self):
-        for added_page in self.added_pages:
-            page = get_or_create_page(**added_page)
-            page.publish('en')
-
-        # Only remove pages created by a script!
-        Page.objects.filter(id__in=self.removed_pages,
-                            created_by="script").delete()
-
-        # https://stackoverflow.com/questions/33284171/
-        call_command('cms', 'fix-tree')
-
-
-class MarkdownFile:
-    html = None
-
-    def __init__(self, fn, docs_namespace, db_actions, slug_override=None):
-        self.fn = fn
-        self.docs_namespace = docs_namespace
-        self.db_actions = db_actions
-        if slug_override:
-            self.slug = slug_override
-        else:
-            self.slug = slugify(self.fn)
-        self.full_url = os.path.join(self.docs_namespace, self.slug)
-        with codecs.open(self.fn, 'r', encoding='utf-8') as f:
-            self.html = markdown.markdown(
-                f.read(),
-                output_format="html5",
-                extensions=['markdown.extensions.tables'])
-        self.release_alias = self._get_release_alias()
-        self.title = self._read_title()
-        self._remove_body_and_html_tags()
-        self._use_developer_site_style()
-
-    def _get_release_alias(self):
-        alias = re.findall(r'/tmp/tmp\S+?/(\S+?)/%s/\S+?' % DOCS_DIRNAME,
-                           self.fn)
-        return alias[0]
-
-    def _read_title(self):
-        soup = BeautifulSoup(self.html, 'html5lib')
-        if soup.title:
-            return soup.title.text
-        if soup.h1:
-            return soup.h1.text
-        return slugify(self.fn).replace('-', ' ').title()
-
-    def _remove_body_and_html_tags(self):
-        self.html = re.sub(r"<html>\n\s<body>\n", "", self.html,
-                           flags=re.MULTILINE)
-        self.html = re.sub(r"\s<\/body>\n<\/html>", "", self.html,
-                           flags=re.MULTILINE)
-
-    def _use_developer_site_style(self):
-        begin = (u"<div class=\"row no-border\">"
-                 "\n<div class=\"eight-col\">\n")
-        end = u"</div>\n</div>"
-        self.html = begin + self.html + end
-        self.html = self.html.replace(
-            "<pre><code>",
-            "</div><div class=\"twelve-col\"><pre><code>")
-        self.html = self.html.replace(
-            "</code></pre>",
-            "</code></pre></div><div class=\"eight-col\">")
-
-    def replace_links(self, titles, url_map):
-        for title in titles:
-            local_md_fn = os.path.basename(title)
-            url = u'/'+url_map[title]
-            # Replace links of the form <a href="/path/somefile.md"> first
-            href = u"<a href=\"{}\">".format(url)
-            md_href = u"<a href=\"{}\">".format(local_md_fn)
-            self.html = self.html.replace(md_href, href)
-
-            # Now we can replace free-standing "somefile.md" references in
-            # the HTML
-            link = href + u"{}</a>".format(titles[title])
-            self.html = self.html.replace(local_md_fn, link)
-
-    def publish(self):
-        '''Publishes pages in their branch alias namespace.'''
-        self.db_actions.add_page(
-            title=self.title, full_url=self.full_url, menu_title=self.title,
-            html=self.html)
-
-
-class SnappyMarkdownFile(MarkdownFile):
-    def __init__(self, fn, docs_namespace, db_actions):
-        MarkdownFile.__init__(self, fn, docs_namespace, db_actions)
-        self._make_snappy_mods()
-
-    def _make_snappy_mods(self):
-        # Make sure the reader knows which documentation she is browsing
-        if self.release_alias != 'current':
-            before = (u"<div class=\"row no-border\">\n"
-                      "<div class=\"eight-col\">\n")
-            after = (u"<div class=\"row no-border\">\n"
-                     "<div class=\"box pull-three three-col\">"
-                     "<p>You are browsing the Snappy <code>%s</code> "
-                     "documentation.</p>"
-                     "<p><a href=\"/snappy/guides/current/%s\">"
-                     "Back to the latest stable release &rsaquo;"
-                     "</a></p></div>\n"
-                     "<div class=\"eight-col\">\n") % (self.release_alias,
-                                                       self.slug, )
-            self.html = self.html.replace(before, after)
-
-    def publish(self):
-        if self.release_alias == "current":
-            # Add a guides/<page> redirect to guides/current/<page>
-            self.db_actions.add_page(
-                title=self.title,
-                full_url=self.full_url.replace('/current', ''),
-                redirect="/snappy/guides/current/%s" % (self.slug))
-        else:
-            self.title += " (%s)" % (self.release_alias,)
-        MarkdownFile.publish(self)
-
-
-def slugify(filename):
-    return os.path.basename(filename).replace('.md', '')
-
-
-class LocalBranch:
-    titles = {}
-    url_map = {}
-
-    def __init__(self, dirname, external_branch, db_actions):
-        self.dirname = dirname
-        self.docs_path = os.path.join(self.dirname, DOCS_DIRNAME)
-        self.doc_fns = glob.glob(self.docs_path+'/*.md')
-        self.md_files = []
-        self.external_branch = external_branch
-        self.docs_namespace = self.external_branch.docs_namespace
-        self.release_alias = os.path.basename(self.docs_namespace)
-        self.index_doc_title = self.release_alias.title()
-        self.index_doc = self.external_branch.index_doc
-        self.db_actions = db_actions
-        self.markdown_class = MarkdownFile
-
-    def import_markdown(self):
-        for doc_fn in self.doc_fns:
-            if self.index_doc and os.path.basename(doc_fn) == self.index_doc:
-                md_file = self.markdown_class(
-                    doc_fn,
-                    os.path.dirname(self.docs_namespace),
-                    self.db_actions,
-                    slug_override=os.path.basename(self.docs_namespace))
-                self.md_files.insert(0, md_file)
-            else:
-                md_file = self.markdown_class(doc_fn, self.docs_namespace,
-                                              self.db_actions)
-                self.md_files += [md_file]
-            self.titles[md_file.fn] = md_file.title
-            self.url_map[md_file.fn] = md_file.full_url
-        if not self.index_doc:
-            self._create_fake_index_doc()
-        for md_file in self.md_files:
-            md_file.replace_links(self.titles, self.url_map)
-
-    def remove_old_pages(self):
-        imported_page_urls = set([md_file.full_url
-                                  for md_file in self.md_files])
-        index_doc = page_resolver.get_page_queryset_from_path(
-            self.docs_namespace)
-        db_pages = []
-        if len(index_doc):
-            # All pages in this namespace currently in the database
-            db_pages = index_doc[0].get_descendants().all()
-        for db_page in db_pages:
-            still_relevant = False
-            for url in imported_page_urls:
-                if url in db_page.get_absolute_url():
-                    still_relevant = True
-                    break
-            # At this point we know that there's no match and the page
-            # can be deleted.
-            if not still_relevant:
-                self.db_actions.remove_page(db_page.id)
-
-    def publish(self):
-        for md_file in self.md_files:
-            md_file.publish()
-
-    def _create_fake_index_doc(self):
-        '''Creates a fake index page at the top of the branches
-           docs namespace.'''
-
-        if self.docs_namespace == "current":
-            redirect = "/snappy/guides"
-        else:
-            redirect = None
-
-        in_navigation = False
-        menu_title = None
-        list_pages = ""
-        for page in self.md_files:
-            list_pages += "<li><a href=\"%s\">%s</a></li>" \
-                % (os.path.basename(page.full_url), page.title)
-        landing = (
-            u"<div class=\"row\"><div class=\"eight-col\">\n"
-            "<p>This section contains documentation for the "
-            "<code>%s</code> Snappy branch.</p>"
-            "<p><ul class=\"list-ubuntu\">%s</ul></p>\n"
-            "<p>Auto-imported from <a "
-            "href=\"https://github.com/ubuntu-core/snappy\">%s</a>.</p>\n"
-            "</div></div>") % (self.release_alias, list_pages,
-                               self.external_branch.lp_origin)
-        self.db_actions.add_page(
-            title=self.index_doc_title, full_url=self.docs_namespace,
-            in_navigation=in_navigation, redirect=redirect, html=landing,
-            menu_title=menu_title)
-
-
-class SnappyLocalBranch(LocalBranch):
-    def __init__(self, dirname, external_branch, db_actions):
-        LocalBranch.__init__(self, dirname, external_branch, db_actions)
-        self.markdown_class = SnappyMarkdownFile
-        self.index_doc_title = 'Snappy documentation'
-        if self.release_alias != 'current':
-            self.index_doc_title += ' (%s)' % self.release_alias
-
-
-def get_or_create_page(title, full_url, menu_title=None,
-                       in_navigation=True, redirect=None, html=None):
-    # First check if pages already exist.
-    pages = Title.objects.select_related('page').filter(path__regex=full_url)
-    if pages:
-        page = pages[0].page
-        page.title = title
-        page.publisher_is_draft = True
-        page.menu_title = menu_title
-        page.in_navigation = in_navigation
-        page.redirect = redirect
-        if html:
-            # We create the page, so we know there's just one placeholder
-            placeholder = page.placeholders.all()[0]
-            if placeholder.get_plugins():
-                plugin = placeholder.get_plugins()[0].get_plugin_instance()[0]
-                plugin.body = html
-                plugin.save()
-            else:
-                add_plugin(placeholder, 'RawHtmlPlugin', 'en', body=html)
-    else:
-        parent_pages = Title.objects.select_related('page').filter(
-            path__regex=os.path.dirname(full_url))
-        if not parent_pages:
-            print('Parent %s not found.' % os.path.dirname(full_url))
-            sys.exit(1)
-        parent = parent_pages[0].page
-
-        slug = os.path.basename(full_url)
-        page = create_page(
-            title, "default.html", "en", slug=slug, parent=parent,
-            menu_title=menu_title, in_navigation=in_navigation,
-            position="last-child", redirect=redirect)
-        if html:
-            placeholder = page.placeholders.get()
-            add_plugin(placeholder, 'RawHtmlPlugin', 'en', body=html)
-    return page
+from developer_portal.models import (
+    ExternalDocsBranch,
+    ExternalDocsBranchImportDirective,
+    ImportedArticle,
+)
 
 
 def import_branches(selection):
@@ -302,52 +20,34 @@
         logging.error('No branches registered in the '
                       'ExternalDocsBranch table yet.')
         return
-    tempdir = tempfile.mkdtemp()
-    db_actions = DBActions()
     for branch in ExternalDocsBranch.objects.filter(
-            docs_namespace__regex=selection):
-        checkout_location = os.path.join(
-            tempdir, os.path.basename(branch.docs_namespace))
-        sourcecode = SourceCode(branch.lp_origin, checkout_location)
-        if sourcecode.get() != 0:
-            logging.error(
-                'Could not check out branch "%s".' % branch.lp_origin)
-            if os.path.exists(checkout_location):
-                shutil.rmtree(checkout_location)
+            origin__regex=selection, active=True):
+        tempdir = tempfile.mkdtemp()
+        repo = create_repo(tempdir, branch.origin, branch.branch_name,
+                           branch.post_checkout_command)
+        if repo.get() != 0:
             break
-        if branch.lp_origin.startswith('lp:snappy') or \
-           'snappy' in branch.lp_origin.split(':')[1].split('.git')[0].split('/'):
-            local_branch = SnappyLocalBranch(checkout_location, branch,
-                                             db_actions)
-        else:
-            local_branch = LocalBranch(checkout_location, branch, db_actions)
-        local_branch.import_markdown()
-        local_branch.publish()
-        local_branch.remove_old_pages()
-    shutil.rmtree(tempdir)
-    db_actions.run()
-
-
-class SourceCode():
-    def __init__(self, branch_origin, checkout_location):
-        self.branch_origin = branch_origin
-        self.checkout_location = checkout_location
-
-    def get(self):
-        if self.branch_origin.startswith('lp:') and \
-           os.path.exists('/usr/bin/bzr'):
-            return subprocess.call([
-                'bzr', 'checkout', '--lightweight', self.branch_origin,
-                self.checkout_location])
-        if self.branch_origin.startswith('https://github.com') and \
-           self.branch_origin.endswith('.git') and \
-           os.path.exists('/usr/bin/git'):
-            return subprocess.call([
-                'git', 'clone', '-q', self.branch_origin,
-                self.checkout_location])
-        logging.error(
-            'Branch format "{}" not understood.'.format(self.branch_origin))
-        return 1
+        for directive in ExternalDocsBranchImportDirective.objects.filter(
+                external_docs_branch=branch):
+            repo.add_directive(directive.import_from,
+                               directive.write_to)
+        repo.execute_import_directives()
+        repo.publish()
+        for page in repo.pages:
+            ImportedArticle.objects.get_or_create(
+                branch=branch,
+                page=page,
+                last_import=datetime.datetime.now())
+
+        # The import is done, now let's clean up.
+        for old_article in ImportedArticle.objects.filter(branch=branch):
+            if old_article.page not in repo.pages and \
+               old_article.page.changed_by in ['python-api', 'script']:
+                old_article.page.delete()
+        shutil.rmtree(tempdir)
+
+    # https://stackoverflow.com/questions/33284171/
+    call_command('cms', 'fix-tree')
 
 
 class Command(BaseCommand):
 
=== added directory 'md_importer/management/importer'
=== added file 'md_importer/management/importer/__init__.py'
=== added file 'md_importer/management/importer/article.py'
--- md_importer/management/importer/article.py	1970-01-01 00:00:00 +0000
+++ md_importer/management/importer/article.py	2016-01-06 12:40:31 +0000
@@ -0,0 +1,118 @@
+from bs4 import BeautifulSoup
+import codecs
+import logging
+import markdown
+import os
+import re
+
+from .publish import get_or_create_page, slugify
+
+
+class Article:
+    html = None
+    page = None
+    title = ""
+
+    def __init__(self, fn, write_to):
+        self.fn = fn
+        self.write_to = slugify(self.fn)
+        self.full_url = write_to
+        self.slug = os.path.basename(self.full_url)
+
+    def read(self):
+        with codecs.open(self.fn, 'r', encoding='utf-8') as f:
+            if self.fn.endswith('.md'):
+                self.html = markdown.markdown(
+                    f.read(),
+                    output_format='html5',
+                    extensions=['pymdownx.github'])
+            elif self.fn.endswith('.html'):
+                self.html = f.read()
+            else:
+                logging.error("Don't know how to interpret '{}'.".format(
+                    self.fn))
+                return False
+        self.title = self._read_title()
+        self._remove_body_and_html_tags()
+        self._use_developer_site_style()
+        return True
+
+    def _read_title(self):
+        soup = BeautifulSoup(self.html, 'html5lib')
+        if soup.title:
+            return soup.title.text
+        if soup.h1:
+            return soup.h1.text
+        return slugify(self.fn).replace('-', ' ').title()
+
+    def _remove_body_and_html_tags(self):
+        self.html = re.sub(r"<html>\n\s<body>\n", "", self.html,
+                           flags=re.MULTILINE)
+        self.html = re.sub(r"\s<\/body>\n<\/html>", "", self.html,
+                           flags=re.MULTILINE)
+
+    def _use_developer_site_style(self):
+        begin = (u"<div class=\"row no-border\">"
+                 "\n<div class=\"eight-col\">\n")
+        end = u"</div>\n</div>"
+        self.html = begin + self.html + end
+        self.html = self.html.replace(
+            "<pre><code>",
+            "</div><div class=\"twelve-col\"><pre><code>")
+        self.html = self.html.replace(
+            "</code></pre>",
+            "</code></pre></div><div class=\"eight-col\">")
+
+    def replace_links(self, titles, url_map):
+        soup = BeautifulSoup(self.html, 'html5lib')
+        for link in soup.find_all('a'):
+            for title in titles:
+                if link.attrs['href'] == os.path.basename(title):
+                    link.attrs['href'] = url_map[title].full_url
+        self.html = soup.prettify()
+
+    def add_to_db(self):
+        '''Publishes pages in their branch alias namespace.'''
+        self.page = get_or_create_page(
+            title=self.title, full_url=self.full_url, menu_title=self.title,
+            html=self.html)
+        self.full_url = self.page.get_absolute_url()
+
+
+class SnappyArticle(Article):
+    release_alias = None
+
+    def read(self):
+        if not Article.read(self):
+            return False
+        self.release_alias = re.findall(r'snappy/guides/(\S+?)/\S+?',
+                                        self.full_url)[0]
+        self._make_snappy_mods()
+        return True
+
+    def _make_snappy_mods(self):
+        # Make sure the reader knows which documentation she is browsing
+        if self.release_alias != 'current':
+            before = (u"<div class=\"row no-border\">\n"
+                      "<div class=\"eight-col\">\n")
+            after = (u"<div class=\"row no-border\">\n"
+                     "<div class=\"box pull-three three-col\">"
+                     "<p>You are browsing the Snappy <code>%s</code> "
+                     "documentation.</p>"
+                     "<p><a href=\"/snappy/guides/current/%s\">"
+                     "Back to the latest stable release &rsaquo;"
+                     "</a></p></div>\n"
+                     "<div class=\"eight-col\">\n") % (self.release_alias,
+                                                       self.slug, )
+            self.html = self.html.replace(before, after)
+
+    def add_to_db(self):
+        if self.release_alias == "current":
+            # Add a guides/<page> redirect to guides/current/<page>
+            get_or_create_page(
+                title=self.title,
+                full_url=self.full_url.replace('/current', ''),
+                redirect="/snappy/guides/current/{}".format(self.slug))
+        else:
+            self.title += " (%s)" % (self.release_alias,)
+        Article.add_to_db(self)
 
=== added file 'md_importer/management/importer/publish.py'
--- md_importer/management/importer/publish.py	1970-01-01 00:00:00 +0000
+++ md_importer/management/importer/publish.py	2016-01-06 12:40:31 +0000
@@ -0,0 +1,64 @@
+from cms.api import create_page, add_plugin
+from cms.models import Title
+
+import logging
+import os
+import sys
+
+# XXX: use this once we RawHTML plugins don't strip comments (LP: #1523925)
+START_TEXT = """
+<!--
+branch id: {}
+
+THIS PAGE IS AUTOMATICALLY UPDATED.
+DON'T EDIT IT - CHANGES WILL BE OVERWRITTEN.
+-->
+"""
+
+
+def slugify(filename):
+    return os.path.basename(filename).replace('.md', '').replace('.html', '')
+
+
+def get_or_create_page(title, full_url, menu_title=None,
+                       in_navigation=True, redirect=None, html=None):
+    # First check if pages already exist.
+    if full_url.startswith('/'):
+        full_url = full_url[1:]
+    pages = Title.objects.select_related('page').filter(
+        path__regex=full_url).filter(publisher_is_draft=True)
+    if pages:
+        page = pages[0].page
+        page.title = title
+        page.publisher_is_draft = True
+        page.menu_title = menu_title
+        page.in_navigation = in_navigation
+        page.redirect = redirect
+        if html:
+            # We create the page, so we know there's just one placeholder
+            placeholder = page.placeholders.all()[0]
+            if placeholder.get_plugins():
+                plugin = placeholder.get_plugins()[0].get_plugin_instance()[0]
+                plugin.body = html
+                plugin.save()
+            else:
+                add_plugin(placeholder, 'RawHtmlPlugin', 'en', body=html)
+    else:
+        parent_pages = Title.objects.select_related('page').filter(
+            path__regex=os.path.dirname(full_url)).filter(
+            publisher_is_draft=True)
+        if not parent_pages:
+            logging.error('Parent {} not found.'.format(
+                os.path.dirname(full_url)))
+            sys.exit(1)
+        parent = parent_pages[0].page
+
+        slug = os.path.basename(full_url)
+        page = create_page(
+            title, "default.html", "en", slug=slug, parent=parent,
+            menu_title=menu_title, in_navigation=in_navigation,
+            position="last-child", redirect=redirect)
+        if html:
+            placeholder = page.placeholders.get()
+            add_plugin(placeholder, 'RawHtmlPlugin', 'en', body=html)
+    return page
 
=== added file 'md_importer/management/importer/repo.py'
--- md_importer/management/importer/repo.py	1970-01-01 00:00:00 +0000
+++ md_importer/management/importer/repo.py	2016-01-06 12:40:31 +0000
@@ -0,0 +1,168 @@
+from .article import Article, SnappyArticle
+from .publish import get_or_create_page, slugify
+from .source import SourceCode
+
+from cms.api import publish_pages
+
+import glob
+import logging
+import os
+import shutil
+
+
+def create_repo(tempdir, origin, branch_name, post_checkout_command):
+    if os.path.exists(origin):
+        if 'snappy' in origin:
+            repo_class = SnappyRepo
+        else:
+            repo_class = Repo
+    else:
+        if origin.startswith('lp:snappy') or \
+           'snappy' in origin.split(':')[1].split('.git')[0].split('/'):
+            repo_class = SnappyRepo
+        else:
+            repo_class = Repo
+    return repo_class(tempdir, origin, branch_name, post_checkout_command)
+
+
+class Repo:
+    titles = {}
+    url_map = {}
+    index_doc_url = None
+    index_page = None
+    release_alias = None
+    directives = []
+    imported_articles = []
+    # On top of the pages in imported_articles this also includes index_page
+    pages = []
+
+    def __init__(self, tempdir, origin, branch_name, post_checkout_command):
+        self.origin = origin
+        self.branch_name = branch_name
+        self.post_checkout_command = post_checkout_command
+        branch_nick = os.path.basename(self.origin.replace('.git', ''))
+        self.checkout_location = os.path.join(
+            tempdir, branch_nick)
+        self.index_doc_title = branch_nick
+        self.article_class = Article
+
+    # Only used to speed up tests - allows reusing same object without
+    # having to redownload the source again
+    def reset(self):
+        self.article_class = Article
+        self.directives = []
+        self.imported_articles = []
+
+    def get(self):
+        sourcecode = SourceCode(self.origin, self.checkout_location,
+                                self.branch_name, self.post_checkout_command)
+        if sourcecode.get() != 0:
+            logging.error(
+                    'Could not check out branch "{}".'.format(self.origin))
+            shutil.rmtree(self.checkout_location)
+            return 1
+        return 0
+
+    def add_directive(self, import_from, write_to):
+        self.directives += [
+            {
+                'import_from': os.path.join(self.checkout_location,
+                                            import_from),
+                'write_to': write_to
+            }
+        ]
+
+    def execute_import_directives(self):
+        import_list = []
+        # Import single files first
+        for directive in [d for d in self.directives
+                          if os.path.isfile(d['import_from'])]:
+            import_list += [
+                (directive['import_from'], directive['write_to'])
+            ]
+        # Import directories next
+        for directive in [d for d in self.directives
+                          if os.path.isdir(d['import_from'])]:
+            for fn in glob.glob('{}/*'.format(directive['import_from'])):
+                if fn not in [a[0] for a in import_list]:
+                    import_list += [
+                        (fn, os.path.join(directive['write_to'], slugify(fn)))
+                    ]
+            # If we import into a namespace and don't have an index doc,
+            # we need to write one.
+            if directive['write_to'] not in [x[1] for x in import_list]:
+                self.index_doc_url = directive['write_to']
+        if self.index_doc_url:
+            self._create_fake_index_page()
+        # The actual import
+        for entry in import_list:
+            article = self._read_article(entry[0], entry[1])
+            if article:
+                self.imported_articles += [article]
+                self.titles[article.fn] = article.title
+                self.url_map[article.fn] = article
+        if self.index_doc_url:
+            self._write_fake_index_doc()
+
+    def _read_article(self, fn, write_to):
+        article = self.article_class(fn, write_to)
+        if article.read():
+            return article
+        return None
+
+    def publish(self):
+        for article in self.imported_articles:
+            article.add_to_db()
+            article.replace_links(self.titles, self.url_map)
+        self.pages = [article.page for article in self.imported_articles]
+        if self.index_page:
+            self.pages.extend([self.index_page])
+        publish_pages(self.pages)
+
+    def _create_fake_index_page(self):
+        '''Creates a fake index page at the top of the branches
+           docs namespace.'''
+
+        if self.index_doc_url.endswith('current'):
+            redirect = '/snappy/guides'
+        else:
+            redirect = None
+        self.index_page = get_or_create_page(
+            title=self.index_doc_title, full_url=self.index_doc_url,
+            in_navigation=False, redirect=redirect, html='',
+            menu_title=None)
+
+    def _write_fake_index_doc(self):
+        list_pages = ''
+        for article in [a for a
+                        in self.imported_articles
+                        if a.full_url.startswith(self.index_doc_url)]:
+            list_pages += '<li><a href=\"{}\">{}</a></li>'.format(
+                os.path.basename(article.full_url), article.title)
+        self.index_page.html = (
+            u'<div class=\"row\"><div class=\"eight-col\">\n'
+            '<p>This section contains documentation for the '
+            '<code>{}</code> Snappy branch.</p>'
+            '<p><ul class=\"list-ubuntu\">{}</ul></p>\n'
+            '<p>Auto-imported from <a '
+            'href=\"{}\">{}</a>.</p>\n'
+            '</div></div>'.format(self.release_alias, list_pages,
+                                  self.origin, self.origin))
+
+
+class SnappyRepo(Repo):
+    def __init__(self, tempdir, origin, branch_name, post_checkout_command):
+        Repo.__init__(self, tempdir, origin, branch_name,
+                      post_checkout_command)
+        self.article_class = SnappyArticle
+        self.index_doc_title = 'Snappy documentation'
+
+    def _create_fake_index_page(self):
+        self.release_alias = os.path.basename(self.index_doc_url)
+        if not self.index_doc_url.endswith('current'):
+            self.index_doc_title += ' ({})'.format(self.release_alias)
+        Repo._create_fake_index_page(self)
+
+    def reset(self):
+        Repo.reset(self)
+        self.article_class = SnappyArticle
 
=== added file 'md_importer/management/importer/source.py'
--- md_importer/management/importer/source.py	1970-01-01 00:00:00 +0000
+++ md_importer/management/importer/source.py	2016-01-06 12:40:31 +0000
@@ -0,0 +1,60 @@
+import logging
+import os
+import shutil
+import subprocess
+
+
+class SourceCode():
+    def __init__(self, origin, checkout_location, branch_name,
+                 post_checkout_command):
+        self.origin = origin
+        self.checkout_location = checkout_location
+        self.branch_name = branch_name
+        self.post_checkout_command = post_checkout_command
+
+    def get(self):
+        res = self._get_branch()
+        if res == 0 and self.post_checkout_command:
+            res = self._post_checkout()
+            return res
+        return res
+
+    def _get_branch(self):
+        if os.path.exists(self.origin):
+            shutil.copytree(
+                self.origin,
+                self.checkout_location)
+            return 0
+        if self.origin.startswith('lp:') and \
+           os.path.exists('/usr/bin/bzr'):
+            return subprocess.call([
+                'bzr', 'checkout', '--lightweight', self.origin,
+                self.checkout_location])
+        if self.origin.startswith('https://github.com') and \
+           self.origin.endswith('.git') and \
+           os.path.exists('/usr/bin/git'):
+            retcode = subprocess.call([
+                'git', 'clone', '--quiet', self.origin,
+                self.checkout_location])
+            if retcode == 0 and self.branch_name:
+                pwd = os.getcwd()
+                os.chdir(self.checkout_location)
+                retcode = subprocess.call(['git', 'checkout', '--quiet',
+                                           self.branch_name])
+                os.chdir(pwd)
+            return retcode
+        logging.error(
+            'Branch format "{}" not understood.'.format(self.origin))
+        return 1
+
+    def _post_checkout(self):
+        pwd = os.getcwd()
+        os.chdir(self.checkout_location)
+        process = subprocess.Popen(self.post_checkout_command.split(),
+                                   stdout=subprocess.PIPE)
+        (out, err) = process.communicate()
+        retcode = process.wait()
+        os.chdir(pwd)
+        if retcode != 0:
+            logging.error(out)
+        return retcode
 
=== added directory 'md_importer/migrations'
=== added file 'md_importer/migrations/0001_initial.py'
--- md_importer/migrations/0001_initial.py	1970-01-01 00:00:00 +0000
+++ md_importer/migrations/0001_initial.py	2016-01-06 12:40:31 +0000
@@ -0,0 +1,46 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('cms', '0013_urlconfrevision'),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='ExternalDocsBranch',
+            fields=[
+                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
+                ('origin', models.CharField(help_text='External branch location, ie: lp:snappy/15.04 or https://github.com/ubuntu-core/snappy.git', max_length=200)),
+                ('branch_name', models.CharField(help_text='For use with git branches, ie: "master" or "15.04" or "1.x".', max_length=200, blank=True)),
+                ('post_checkout_command', models.CharField(help_text='Command to run after checkout of the branch.', max_length=100, blank=True)),
+                ('active', models.BooleanField(default=True)),
+            ],
+            options={
+                'verbose_name': 'external docs branch',
+                'verbose_name_plural': 'external docs branches',
+            },
+        ),
+        migrations.CreateModel(
+            name='ExternalDocsBranchImportDirective',
+            fields=[
+                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
+                ('import_from', models.CharField(help_text='File or directory to import from the branch. Ie: "docs/intro.md" (file) or "docs" (complete directory), etc.', max_length=150)),
+                ('write_to', models.CharField(help_text='Article URL (for a specific file) or article namespace for a directory or a set of files.', max_length=150)),
+                ('external_docs_branch', models.ForeignKey(to='md_importer.ExternalDocsBranch')),
+            ],
+        ),
+        migrations.CreateModel(
+            name='ImportedArticle',
+            fields=[
+                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
+                ('last_import', models.DateTimeField(help_text='Datetime of last import.', verbose_name='Datetime')),
+                ('branch', models.ForeignKey(to='md_importer.ExternalDocsBranch')),
+                ('page', models.ForeignKey(to='cms.Page')),
+            ],
+        ),
+    ]
 
=== added file 'md_importer/migrations/__init__.py'
=== added file 'md_importer/models.py'
--- md_importer/models.py	1970-01-01 00:00:00 +0000
+++ md_importer/models.py	2016-01-06 12:40:31 +0000
@@ -0,0 +1,54 @@
+from django.db import models
+from django.utils.translation import ugettext_lazy as _
+
+from cms.models import Page
+
+
+class ExternalDocsBranch(models.Model):
+    origin = models.CharField(
+        max_length=200,
+        help_text=_('External branch location, ie: lp:snappy/15.04 or '
+                    'https://github.com/ubuntu-core/snappy.git'))
+    branch_name = models.CharField(
+        max_length=200,
+        help_text=_('For use with git branches, ie: "master" or "15.04" '
+                    'or "1.x".'),
+        blank=True)
+    post_checkout_command = models.CharField(
+        max_length=100,
+        help_text=_('Command to run after checkout of the branch.'),
+        blank=True)
+    active = models.BooleanField(default=True)
+
+    def __str__(self):
+        if self.branch_name:
+            return "{} - {}".format(self.origin, self.branch_name)
+        return "{}".format(self.origin)
+
+    class Meta:
+        verbose_name = "external docs branch"
+        verbose_name_plural = "external docs branches"
+
+
+class ExternalDocsBranchImportDirective(models.Model):
+    external_docs_branch = models.ForeignKey(ExternalDocsBranch)
+    import_from = models.CharField(
+        max_length=150,
+        help_text=_('File or directory to import from the branch. '
+                    'Ie: "docs/intro.md" (file) or '
+                    '"docs" (complete directory), etc.'))
+    write_to = models.CharField(
+        max_length=150,
+        help_text=_('Article URL (for a specific file) or article namespace '
+                    'for a directory or a set of files.'))
+
+    def __str__(self):
+        return "{} -- {}".format(self.external_docs_branch,
+                                 self.import_from)
+
+
+class ImportedArticle(models.Model):
+    page = models.ForeignKey(Page)
+    branch = models.ForeignKey(ExternalDocsBranch)
+    last_import = models.DateTimeField(
+        _('Datetime'), help_text=_('Datetime of last import.'))
 
=== added directory 'md_importer/tests'
=== added file 'md_importer/tests/__init__.py'
--- md_importer/tests/__init__.py	1970-01-01 00:00:00 +0000
+++ md_importer/tests/__init__.py	2016-01-06 12:40:31 +0000
@@ -0,0 +1,8 @@
+import shutil
+
+from utils import SnapcraftTestRepo, SnappyTestRepo
+
+
+def tearDownModule():
+    shutil.rmtree(SnapcraftTestRepo().tempdir)
+    shutil.rmtree(SnappyTestRepo().tempdir)
 
=== added directory 'md_importer/tests/data'
=== added directory 'md_importer/tests/data/link-test'
=== added file 'md_importer/tests/data/link-test/file1.md'
--- md_importer/tests/data/link-test/file1.md	1970-01-01 00:00:00 +0000
+++ md_importer/tests/data/link-test/file1.md	2016-01-06 12:40:31 +0000
@@ -0,0 +1,5 @@
+# Test
+
+This is a link to [the file2 file](file2.md).
+
+That's it.
 
=== added file 'md_importer/tests/data/link-test/file2.md'
--- md_importer/tests/data/link-test/file2.md	1970-01-01 00:00:00 +0000
+++ md_importer/tests/data/link-test/file2.md	2016-01-06 12:40:31 +0000
@@ -0,0 +1,3 @@
+# File 2
+
+Here's just some text.
 
=== added file 'md_importer/tests/test_branch_fetch.py'
--- md_importer/tests/test_branch_fetch.py	1970-01-01 00:00:00 +0000
+++ md_importer/tests/test_branch_fetch.py	2016-01-06 12:40:31 +0000
@@ -0,0 +1,42 @@
+import os
+import shutil
+import tempfile
+
+from django.test import TestCase
+
+from ..management.importer.repo import create_repo, Repo
+from .utils import SnapcraftTestRepo
+
+
+class TestBranchFetch(TestCase):
+    def test_git_fetch(self):
+        snapcraft = SnapcraftTestRepo()
+        snapcraft.repo.reset()
+        self.assertEqual(snapcraft.fetch_retcode, 0)
+        self.assertTrue(isinstance(snapcraft.repo, Repo))
+
+    def test_bzr_fetch(self):
+        tempdir = tempfile.mkdtemp()
+        repo = create_repo(
+            tempdir,
+            'lp:snapcraft',  # outdated, but should work for testing
+            '',
+            '')
+        ret = repo.get()
+        shutil.rmtree(tempdir)
+        self.assertEqual(ret, 0)
+        self.assertTrue(isinstance(repo, Repo))
+
+    def test_post_checkout_command(self):
+        tempdir = tempfile.mkdtemp()
+        repo = create_repo(
+            tempdir,
+            'lp:snapcraft',
+            '',
+            'touch something.html'
+        )
+        ret = repo.get()
+        self.assertEqual(ret, 0)
+        self.assertTrue(os.path.exists(
+            os.path.join(repo.checkout_location, 'something.html')))
+        shutil.rmtree(tempdir)
 
=== added file 'md_importer/tests/test_branch_import.py'
--- md_importer/tests/test_branch_import.py	1970-01-01 00:00:00 +0000
+++ md_importer/tests/test_branch_import.py	2016-01-06 12:40:31 +0000
@@ -0,0 +1,67 @@
+from django.test import TestCase
+
+from cms.api import publish_pages
+from cms.models import Page
+
+from ..management.importer.article import Article
+from .utils import (
+    db_create_home_page,
+    db_empty_page_list,
+    SnapcraftTestRepo,
+)
+
+
+class TestBranchImport(TestCase):
+    def test_1dir_import(self):
+        db_empty_page_list()
+        db_create_home_page()
+        snapcraft = SnapcraftTestRepo()
+        snapcraft.repo.reset()
+        snapcraft.repo.add_directive('docs', '/')
+        snapcraft.repo.execute_import_directives()
+        snapcraft.repo.publish()
+        pages = Page.objects.all()
+        self.assertGreater(len(pages), 3)
+        for article in snapcraft.repo.imported_articles:
+            self.assertTrue(isinstance(article, Article))
+
+    def test_1dir_and_2files_import(self):
+        db_empty_page_list()
+        db_create_home_page()
+        snapcraft = SnapcraftTestRepo()
+        snapcraft.repo.reset()
+        snapcraft.repo.add_directive('docs', '/')
+        snapcraft.repo.add_directive('README.md', '/')
+        snapcraft.repo.add_directive('HACKING.md', '/hacking')
+        snapcraft.repo.execute_import_directives()
+        snapcraft.repo.publish()
+        pages = Page.objects.all()
+        self.assertGreater(len(pages), 5)
+        self.assertIn(u'/en/', [p.get_absolute_url() for p in pages])
+        self.assertIn(u'/en/hacking/', [p.get_absolute_url() for p in pages])
+
+    # Check if all importe article has 'home' as parent
+    def test_articletree_1file_import(self):
+        db_empty_page_list()
+        home = db_create_home_page()
+        publish_pages([[home]])
+        snapcraft = SnapcraftTestRepo()
+        snapcraft.repo.reset()
+        snapcraft.repo.add_directive('README.md', '/readme')
+        snapcraft.repo.execute_import_directives()
+        snapcraft.repo.publish()
+        self.assertEqual(Page.objects.count(), 1+1)  # readme + home
+        self.assertTrue(snapcraft.repo.pages[0].parent == home)
+
+    # Check if all imported articles have 'home' as parent
+    def test_articletree_1dir_import(self):
+        db_empty_page_list()
+        home = db_create_home_page()
+        snapcraft = SnapcraftTestRepo()
+        snapcraft.repo.reset()
+        snapcraft.repo.add_directive('docs', '/')
+        snapcraft.repo.execute_import_directives()
+        snapcraft.repo.publish()
+        for page in Page.objects.filter(publisher_is_draft=False):
+            if page.parent is not None:
+                self.assertEqual(page.parent_id, home.id)
 
=== added file 'md_importer/tests/test_link_rewrite.py'
--- md_importer/tests/test_link_rewrite.py	1970-01-01 00:00:00 +0000
+++ md_importer/tests/test_link_rewrite.py	2016-01-06 12:40:31 +0000
@@ -0,0 +1,44 @@
+from bs4 import BeautifulSoup
+import os
+import shutil
+import tempfile
+
+from django.http.response import HttpResponseNotFound
+from django.test import TestCase, Client
+
+from cms.models import Page
+
+from ..management.importer.article import Article
+from ..management.importer.repo import create_repo
+from .utils import (
+    db_create_home_page,
+    db_empty_page_list,
+)
+
+
+class TestLinkRewrite(TestCase):
+    def test_simple_case(self):
+        db_empty_page_list()
+        db_create_home_page()
+        tempdir = tempfile.mkdtemp()
+        repo = create_repo(
+            tempdir,
+            os.path.join(os.path.dirname(__file__), 'data/link-test'),
+            '',
+            '')
+        self.assertEqual(repo.get(), 0)
+        repo.add_directive('', '/')
+        repo.execute_import_directives()
+        repo.publish()
+        pages = Page.objects.all()
+        self.assertEqual(pages.count(), 1+2)  # Home + 2 articles
+        c = Client()
+        for article in repo.imported_articles:
+            self.assertTrue(isinstance(article, Article))
+            soup = BeautifulSoup(article.html, 'html5lib')
+            for link in soup.find_all('a'):
+                if not link.has_attr('class') or \
+                   'headeranchor-link' not in link.attrs['class']:
+                    res = c.get(link.attrs['href'])
+                    self.assertNotIsInstance(res, HttpResponseNotFound)
+        shutil.rmtree(tempdir)
 
=== added file 'md_importer/tests/test_snappy_import.py'
--- md_importer/tests/test_snappy_import.py	1970-01-01 00:00:00 +0000
+++ md_importer/tests/test_snappy_import.py	2016-01-06 12:40:31 +0000
@@ -0,0 +1,70 @@
+from django.test import TestCase
+
+from cms.api import publish_pages
+from cms.models import Page
+
+from ..management.importer.repo import SnappyRepo
+from ..management.importer.article import SnappyArticle
+from .utils import (
+    db_add_empty_page,
+    db_create_home_page,
+    db_empty_page_list,
+    SnappyTestRepo,
+)
+
+
+class TestSnappyImport(TestCase):
+    def test_snappy_devel_import(self):
+        db_empty_page_list()
+        home = db_create_home_page()
+        snappy_page = db_add_empty_page('Snappy', home)
+        guides = db_add_empty_page('Guides', snappy_page)
+        publish_pages([home, snappy_page, guides])
+        snappy = SnappyTestRepo()
+        snappy.repo.reset()
+        self.assertEqual(snappy.fetch_retcode, 0)
+        self.assertTrue(isinstance(snappy.repo, SnappyRepo))
+        snappy.repo.add_directive('docs', '/snappy/guides/devel')
+        snappy.repo.execute_import_directives()
+        snappy.repo.publish()
+        for article in snappy.repo.imported_articles:
+            self.assertTrue(isinstance(article, SnappyArticle))
+        self.assertGreater(len(snappy.repo.pages), 0)
+        devel = Page.objects.filter(parent=guides.get_public_object())
+        self.assertEqual(devel.count(), 1)
+        for page in Page.objects.filter(publisher_is_draft=False):
+            if page not in [home, snappy_page, guides, devel[0]]:
+                self.assertEqual(page.parent, devel[0])
+
+    def test_snappy_current_import(self):
+        db_empty_page_list()
+        home = db_create_home_page()
+        snappy_page = db_add_empty_page('Snappy', home)
+        guides = db_add_empty_page('Guides', snappy_page)
+        publish_pages([home, snappy_page, guides])
+        snappy = SnappyTestRepo()
+        snappy.repo.reset()
+        self.assertTrue(isinstance(snappy.repo, SnappyRepo))
+        snappy.repo.add_directive('docs', '/snappy/guides/current')
+        snappy.repo.execute_import_directives()
+        snappy.repo.publish()
+        number_of_articles = len(snappy.repo.imported_articles)
+        for article in snappy.repo.imported_articles:
+            self.assertTrue(isinstance(article, SnappyArticle))
+        self.assertGreater(number_of_articles, 0)
+        pages = Page.objects.all()
+        current_search = [
+            a for a in pages
+            if a.get_slug('current') and
+            a.get_absolute_url().endswith('snappy/guides/current/')]
+        self.assertEqual(len(current_search), 1)
+        current = current_search[0]
+        nav_pages = [home, snappy_page, guides, current]
+        # 1 imported article, 1 redirect
+        self.assertEqual(
+            number_of_articles*2, pages.count()-len(nav_pages))
+        for page in [a for a in pages if a not in nav_pages]:
+            if page.get_redirect('en'):
+                self.assertEqual(page.parent, guides)
+            else:
+                self.assertEqual(page.parent, current)
 
=== added file 'md_importer/tests/test_utils.py'
--- md_importer/tests/test_utils.py	1970-01-01 00:00:00 +0000
+++ md_importer/tests/test_utils.py	2016-01-06 12:40:31 +0000
@@ -0,0 +1,33 @@
+from django.test import TestCase
+
+from cms.api import publish_pages
+from cms.models import Page
+
+from .utils import (
+    db_add_empty_page,
+    db_create_home_page,
+    db_empty_page_list,
+)
+
+
+class PageDBActivities(TestCase):
+    def test_empty_page_list(self):
+        db_empty_page_list()
+        self.assertEqual(Page.objects.count(), 0)
+
+    def test_create_home_page(self):
+        db_empty_page_list()
+        home = db_create_home_page()
+        publish_pages([home])
+        self.assertNotEqual(home, None)
+        self.assertEqual(Page.objects.count(), 1)
+
+    def test_simple_articletree(self):
+        db_empty_page_list()
+        home = db_create_home_page()
+        snappy = db_add_empty_page('Snappy', home)
+        guides = db_add_empty_page('Guides', snappy)
+        publish_pages([home, snappy, guides])
+        self.assertEqual(Page.objects.count(), 3)
+        self.assertEqual(guides.parent, snappy)
+        self.assertEqual(snappy.parent, home)
 
=== added file 'md_importer/tests/utils.py'
--- md_importer/tests/utils.py	1970-01-01 00:00:00 +0000
+++ md_importer/tests/utils.py	2016-01-06 12:40:31 +0000
@@ -0,0 +1,67 @@
+import shutil
+import tempfile
+
+from django.utils.text import slugify
+
+from cms.api import create_page
+from cms.models import Page
+
+from ..management.importer.repo import create_repo
+
+
+class Singleton(type):
+    _instances = {}
+
+    def __call__(cls, *args, **kwargs):
+        if cls not in cls._instances:
+            cls._instances[cls] = super(Singleton, cls).__call__(*args,
+                                                                 **kwargs)
+        return cls._instances[cls]
+
+
+# We are going to re-use this one, so we don't have to checkout the git
+# repo all the time.
+class SnapcraftTestRepo():
+    __metaclass__ = Singleton
+
+    def __init__(self):
+        self.tempdir = tempfile.mkdtemp()
+        self.repo = create_repo(
+            self.tempdir,
+            'https://github.com/ubuntu-core/snapcraft.git',
+            'master',
+            '')
+        self.fetch_retcode = self.repo.get()
+
+
+class SnappyTestRepo():
+    __metaclass__ = Singleton
+
+    def __init__(self):
+        self.tempdir = tempfile.mkdtemp()
+        self.repo = create_repo(
+            self.tempdir,
+            'https://github.com/ubuntu-core/snappy.git',
+            'master',
+            '')
+        self.fetch_retcode = self.repo.get()
+
+
+def tearDownModule():
+    shutil.rmtree(SnapcraftTestRepo().tempdir)
+    shutil.rmtree(SnappyTestRepo().tempdir)
+
+
+def db_empty_page_list():
+    Page.objects.all().delete()
+
+
+def db_create_home_page():
+    home = create_page('Test import', 'default.html', 'en', slug='home')
+    return home
+
+
+def db_add_empty_page(title, parent):
+    page = create_page(title, 'default.html', 'en', slug=slugify(title),
+                       parent=parent)
+    return page
 
=== modified file 'requirements.txt'
--- requirements.txt	2015-12-11 16:10:30 +0000
+++ requirements.txt	2016-01-06 12:40:31 +0000
@@ -1,4 +1,4 @@
-Django==1.8.7
+Django==1.8.8
 django-template-debug==0.3.5
 oslo.config==3.1.0
 oslo.i18n==3.1.0
@@ -7,6 +7,7 @@
 Pillow==2.9.0
 cmsplugin-zinnia==0.8
 Markdown==2.6.5
+pymdown-extensions==1.0.1
 beautifulsoup4==4.4.1
 dj-database-url==0.3.0
 django-admin-enhancer==1.0.0
@@ -20,8 +21,8 @@
 django-meta==0.3.1
 django-meta-mixin==0.2.1
 django-missing==0.1.15
-django-parler==1.5.1
-django-polymorphic==0.7.2
+django-parler==1.6
+django-polymorphic==0.8.1
 django-reversion==1.9.3
 django-sekizai==0.9.0
 django-swiftstorage==1.1.0
@@ -31,11 +32,11 @@
 django-taggit-templatetags==0.2.5
 django-templatetag-sugar==1.0
 django-xmlrpc==0.1.5
-djangocms-admin-style==1.0.7
+djangocms-admin-style==1.0.8
 djangocms-link==1.7.1
 djangocms-picture==0.2.0
 djangocms-snippet==1.7.1
-djangocms-text-ckeditor==2.8.0
+djangocms-text-ckeditor==2.8.1
 djangocms-utils==0.9.5
 djangocms-video==0.2.0
 python-keystoneclient==1.3.3
@@ -49,4 +50,4 @@
 django-pygments==0.1
 django-openid-auth==0.7
 python-openid==2.2.5
-djangorestframework==3.3.1
+djangorestframework==3.3.2