Merge lp:~savoirfairelinux-openerp/openupgrade-addons/7.0 into lp:openupgrade-addons

[Sandy Carter (http://www.savoirfairelinux.com)]

Handling migration of wiki to document_page.

Groups turned into parent pages with type 'category'
Wiky syntax converted to html using specifically made python port of Wiky.js

Leftover from 6.1 after Upgrade:
 - Dashboards
 - OpenERP Web mobile
 - Process
 - web Dashboard

Missing from 7.0 after Upgrade:
 - Enterprise Process
 - View Editor

Lost data:
 - wiki_wiki.tags
 - wiki_wiki.minor_edit
 - wiki_wiki.review
 - wiki_wiki.summary
 - wiki_wiki.toc
 - wiki_wiki.section
 - wiki_wiki_history.minor_edit
 - wiki_groups.summary
 - wiki_make_index
 - wiki_wiki_page_open

Known issues:
  Views are messed up after upgrade, this is resolved by installing 'Enterprise Process' after the upgrade.

[Sylvain LE GAL (GRAP) (sylvain-legal)]

I don't use neither know the 'wiki' module.
Sorry.

review: Abstain

[Holger Brunn (Therp) (hbrunn)]

Looks great! But this should be a migration script of the wiki module, not of the document_page module. So I'd advise to create an empty wiki addon that simply contains the script you developed. (and appropriate __openerp__.py, __init__.py of course)

Then on #343, #349, #408f, I think you need to copy data between the tables and not rename them

Please put the information about lost data into a file called user_notes.txt, the loang term goal is to present the information there to the user, giving a summary what points should be given attention.

When setting up your fake wiki module for 7.0, simply let it depend on Enterprise Process, then this addon will be installed automatically on upgrade.

review: Needs Fixing

[Stefan Rijnhart (Opener) (stefan-opener)]

Thanks for your contribution! I think the idea of rendering the wiki in Python is very elegant.

Holger, I think putting this script in the document_page module is very defendable. Looking at the code, it is clear that the document_page module is not only conceptually the replacement of the wiki module, but also a direct rewrite of it. This is actually already encoded in the base module's pre script that triggers the module renames.

The renaming of the tables and models (ll.343..352) goes together with this idea. Small fixes here: I think you do still need to rename the history model. You do not need to rename the create_menu table as this is a transient model.

I do not see why you would make such an effort to precreate and fill columns in the pre-script. Typically, this is done in the post-script. If it is to allow the orm to set a database constraint 'not null', remember that if you fill them properly in the post-script, the orm will set the constraint at the next, regular upgrade. Can you say if that was your main concern?

l.364: I am guessing the comment 'Put wiki_wiki content into wiki_groups' should be the other way around.

This module aggressively drops deprecated columns. I am generally in favour of prefixing them with the generated openupgrade prefix instead, to prevent unintentional loss of information. But pending the creation of the ephemeral service module to drop deprecated columns in a controlled fashion, I will not block this proposal for it.

review: Needs Fixing

[Sandy Carter (http://www.savoirfairelinux.com) (sandy-carter)]

Looking at the base migration script, I came to the conclusion that document_page was the simplified continuation of wiki_wiki. There is also the fact that modules that depened on wiki_wiki in 6.* now depend on document_page.

The issue with the views that was resolved by Enterprise process seems to be fixed.

I wasn't sure whether to put my precreate functions in pre, or post. That's fixed now.

What is the generated prefix? Is it version? How would you name it?

[Stefan Rijnhart (Opener) (stefan-opener)]

Hi Sandy, to retrieve the legacy name for columns or tables, you can use

    openupgrade.get_legacy_name(existing_table_or_column_name)

In rename_columns, you can specify (existing_column_name, None). That will call get_legacy_name internally.

[Sandy Carter (http://www.savoirfairelinux.com) (sandy-carter)]

This should address all your issues, Stefan.
Holger, I hope this works too.

review: Needs Resubmitting

[Stefan Rijnhart (Opener) (stefan-opener)]

Thanks for updating this branch! Code approved, but I think your comments in user_notes.txt about leftover modules, missing modules and known issues are a little off. For instance, the Process module in 6.1 is the same module as the Enterprice process in 7.0, and it will always be installed. Could you perhaps just remove these three comments and keep the first two lines of the file and the lost data stanza?

I think the issue you are experiencing with the views after the upgrade may be similar to https://bugs.launchpad.net/openupgrade-server/+bug/1226086, which fixed itself after the next (regular) module upgrade.

review: Needs Fixing

[Sandy Carter (http://www.savoirfairelinux.com) (sandy-carter)]

Done

[Stefan Rijnhart (Opener) (stefan-opener)]

Approved. Thanks again!

review: Approve

[Sandy Carter (http://www.savoirfairelinux.com) (sandy-carter)]

review: Approve

[Maxime Chambreuil (http://www.savoirfairelinux.com) (max3903)]

lgtm

review: Approve (code review)

[Holger Brunn (Therp) (hbrunn)]

review: Approve (code review)

=== added file 'document_page/migrations/7.0.1.0.1/post-migration.py'
--- document_page/migrations/7.0.1.0.1/post-migration.py	1970-01-01 00:00:00 +0000
+++ document_page/migrations/7.0.1.0.1/post-migration.py	2013-10-15 19:37:17 +0000
@@ -0,0 +1,306 @@
+# -*- coding: utf-8 -*-
+##############################################################################
+#
+#    OpenERP, Open Source Management Solution
+#    This module copyright (C) 2013 Savoir-faire Linux
+#    (<http://www.savoirfairelinux.com>).
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU Affero General Public License as
+#    published by the Free Software Foundation, either version 3 of the
+#    License, or (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU Affero General Public License for more details.
+#
+#    You should have received a copy of the GNU Affero General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+##############################################################################
+
+from openerp.openupgrade import openupgrade
+from openerp import pooler, SUPERUSER_ID
+from openerp.openupgrade.openupgrade import logged_query
+
+
+def combine_wiki_groups_document_page(cr):
+    """Put wiki_groups content into wiki_wiki, then delete wiki_groups, conserve parent_id"""
+    logged_query(cr, """ALTER TABLE document_page ADD COLUMN old_id integer;""")
+    logged_query(cr, """\
+INSERT INTO document_page(create_uid, create_date, write_date, name, content, type, old_id)
+SELECT create_uid, create_date, write_date, name, content, 'category' AS type, id
+FROM wiki_groups
+ORDER BY id ASC;""")
+    logged_query(cr, """\
+UPDATE document_page w
+SET parent_id = (SELECT id FROM document_page WHERE old_id = w.group_id LIMIT 1)
+WHERE group_id IS NOT null;\
+""")
+    openupgrade.drop_columns(cr, [('document_page', 'group_id'), ('document_page', 'old_id')])
+
+
+def migrate_wiki_to_html(cr, pool):
+    document_page_obj = pool.get('document.page')
+    wiky = Wiky()
+    logged_query(cr, """\
+SELECT id, content
+FROM document_page
+WHERE content is not NULL;
+""")
+    for page_line_id, content in cr.fetchall():
+        document_page_obj.write(
+            cr, SUPERUSER_ID, [page_line_id],
+            {'content': wiky.process(content)}
+        )
+
+
[e-mail]()
+def migrate(cr, version):
+    pool = pooler.get_pool(cr.dbname)
+    combine_wiki_groups_document_page(cr)
+    migrate_wiki_to_html(cr, pool)
+    logged_query(cr, """DROP TABLE wiki_wiki_page_open;""")
+    logged_query(cr, """DROP TABLE wiki_make_index;""")
+    logged_query(cr, """DROP TABLE wiki_create_menu""")
+    logged_query(cr, """DROP TABLE wiki_groups""")
+
+
+##############################################################################
+#
+#    Wiky.py - Python library to converts Wiki MarkUp language to HTML.
+#    Based on Wiki.js by Tanin Na Nakorn
+#
+#    Copyright © 2013 Sandy Carter <[e-mail]>
+#    This work is free. You can redistribute it and/or modify it under the
+#    terms of the Creative Commons Attribution 3.0 Unported License.
+#    (http://creativecommons.org/licenses/by/3.0/legalcode)
+#
+##############################################################################
+
+import re
+
+re_h3 = re.compile("^===[^=]+===$")
+re_h2 = re.compile("^==[^=]+==$")
+re_h1 = re.compile("^=[^=]+=$")
+re_indent = re.compile("^:+")
+re_hr = re.compile("^-{4}")
+re_ul = re.compile("^\*+ ")
+re_ol = re.compile("^#+ ")
+re_ul_ol = re.compile("^(\*+|#+) ")
+re_ul_li = re.compile("^(\*+|##+):? ")
+re_ol_li = re.compile("^(\*\*+|#+):? ")
+re_ul_ol_li = re.compile("^(\*+|#+):? ")
+re_youtube = re.compile("^(https?://)?(www\.)?youtube.com/(watch\?(.*)v=|embed/)([^&]+)")
+re_b_i = re.compile("'''''(([^']|([^']('{1,4})?[^']))+)'''''")
+re_b = re.compile("'''(([^']|([^'](''?)?[^']))+)'''")
+re_i = re.compile("''(([^']|([^']'?[^']))+)''")
+
+
+class Wiky:
+    def __init__(self, link_image=None):
+        self.link_image = link_image
+
+    def process(self, wikitext):
+        lines = wikitext.split("\n")
+        html = ""
+        i = 0
+        while i < len(lines):
+            line = lines[i]
+            if re_h3.match(line):
+                html += "<h3>%s</h3>" % line[3:-3]
+            elif re_h2.match(line):
+                html += "<h2>%s</h2>" % line[2:-2]
+            elif re_h1.match(line):
+                html += "<h1>%s</h1>" % line[1:-1]
+            elif re_hr.match(line):
+                html += "<hr/>"
+            elif re_indent.match(line):
+                start = i
+                while i < len(lines) and re_indent.match(lines[i]):
+                    i += 1
+                i -= 1
+                html += self.process_indent(lines[start: i + 1])
+            elif re_ul.match(line):
+                start = i
+                while i < len(lines) and re_ul_li.match(lines[i]):
+                    i += 1
+                i -= 1
+                html += self.process_bullet_point(lines[start: i + 1])
+            elif re_ol.match(line):
+                start = i
+                while i < len(lines) and re_ol_li.match(lines[i]):
+                    i += 1
+                i -= 1
+                html += self.process_bullet_point(lines[start: i + 1])
+            else:
+                html += self.process_normal(line)
+            html += "<br/>\n"
+            i += 1
+        return html
+
+    def process_indent(self, lines):
+        html = "\n<dl>\n"
+        i = 0
+        while i < len(lines):
+            line = lines[i]
+            html += "<dd>"
+            this_count = len(re_indent.match(line).group(0))
+            html += self.process_normal(line[this_count:])
+
+            nested_end = i
+            j = i + 1
+            while j < len(lines):
+                nested_count = len(re_indent.match(lines[j]).group(0))
+                if nested_count <= this_count:
+                    break
+                else:
+                    nested_end = j
+                j += 1
+
+            if nested_end > i:
+                html += self.process_indent(lines[i + 1: nested_end + 1])
+                i = nested_end
+
+            html += "</dd>\n"
+            i += 1
+        html += "</dl>\n"
+        return html
+
+    def process_bullet_point(self, lines):
+        if not len(lines):
+            return ""
+        html = "<ul>" if lines[0][0] == "*" else "<ol>"
+        html += '\n'
+        i = 0
+        while i < len(lines):
+            line = lines[i]
+            html += "<li>"
+            this_count = len(re_ul_ol.match(line).group(1))
+            html += self.process_normal(line[this_count+1:])
+
+            # continue previous with #:
+            nested_end = i
+            j = i + 1
+            while j < len(lines):
+                nested_count = len(re_ul_ol_li.match(lines[j]).group(1))
+                if nested_count < this_count:
+                    break
+                elif lines[j][nested_count] == ':':
+                    html += "<br/>" + self.process_normal(lines[j][nested_count + 2:])
+                    nested_end = j
+                else:
+                    break
+                j += 1
+            i = nested_end
+
+            # nested bullet point
+            nested_end = i
+            j = i + 1
+            while j < len(lines):
+                nested_count = len(re_ul_ol_li.match(lines[j]).group(1))
+                if nested_count <= this_count:
+                    break
+                else:
+                    nested_end = j
+                j += 1
+
+            if nested_end > i:
+                html += self.process_bullet_point(lines[i + 1: nested_end + 1])
+                i = nested_end
+
+            # continue previous with #:
+            nested_end = i
+            j = i + 1
+            while j < len(lines):
+                nested_count = len(re_ul_ol_li.match(lines[j]).group(1))
+                if nested_count < this_count:
+                    break
+                elif lines[j][nested_count] == ':':
+                    html += self.process_normal(lines[j][nested_count + 2:])
+                    nested_end = j
+                else:
+                    break
+                j += 1
+            i = nested_end
+            html += "</li>\n"
+            i += 1
+        html += "</ul>" if lines[0][0] == "*" else "</ol>"
+        html += '\n'
+        return html
+
+    def process_url(self, txt):
+        css = ('style="background: url(\"%s\") no-repeat scroll '
+               'right center transparent;padding-right: 13px;"'
+               % self.link_image) if self.link_image else ''
+        try:
+            index = txt.index(" ")
+            url = txt[:index]
+            label = txt[index + 1:]
+        except ValueError:
+            label = url = txt
+        return """<a href="%s" %s>%s</a>""" % (url, css, label)
+
+    @staticmethod
+    def process_image(txt):
+        try:
+            index = txt.index(" ")
+            url = txt[:index]
+            label = txt[index + 1:]
+        except ValueError:
+            url = txt
+            label = ""
+        return '<img src="%s" alt="%s" />' % (url, label)
+
+    @staticmethod
+    def process_video(url):
+        m = re_youtube.match(url)
+        if not m:
+            return "<b>%s is an invalid YouTube URL</b>" % url
+        url = "http://www.youtube.com/embed/" + m.group(5)
+        return '<iframe width="480" height="390" src="%s" frameborder="0" allowfullscreen=""></iframe>' % url
+
+    def process_normal(self, wikitext):
+        # Image
+        while True:
+            try:
+                index = wikitext.index("[[File:")
+                end_index = wikitext.index("]]", index + 7)
+                wikitext = (wikitext[:index] +
+                            self.process_image(wikitext[index + 7:end_index]) +
+                            wikitext[end_index + 2:])
+            except ValueError:
+                break
+
+        # Video
+        while True:
+            try:
+                index = wikitext.index("[[Video:")
+                end_index = wikitext.index("]]", index + 8)
+                wikitext = (wikitext[:index] +
+                            self.process_video(wikitext[index+8:end_index]) +
+                            wikitext[end_index + 2:])
+            except ValueError:
+                break
+
+        # URL
+        for protocol in ["http", "ftp", "news"]:
+            end_index = -1
+            while True:
+                try:
+                    index = wikitext.index("[%s://" % protocol, end_index + 1)
+                    end_index = wikitext.index("]", index + len(protocol) + 4)
+                    wikitext = (wikitext[:index] +
+                                self.process_url(wikitext[index+1:end_index]) +
+                                wikitext[end_index+1:])
+                except ValueError:
+                    break
+
+        # Bold, Italics, Emphasis
+        wikitext = re_b_i.sub("<b><i>\1</i></b>", wikitext)
+        wikitext = re_b.sub("<b>\1</b>", wikitext)
+        wikitext = re_i.sub("<i>\1</i>", wikitext)
+
+        return wikitext
+# vim:expandtab:smartindent:tabstop=4:softtabstop=4:shiftwidth=4:

=== added file 'document_page/migrations/7.0.1.0.1/pre-migration.py'
--- document_page/migrations/7.0.1.0.1/pre-migration.py	1970-01-01 00:00:00 +0000
+++ document_page/migrations/7.0.1.0.1/pre-migration.py	2013-10-15 19:37:17 +0000
@@ -0,0 +1,60 @@
+# -*- coding: utf-8 -*-
+##############################################################################
+#
+#    OpenERP, Open Source Management Solution
+#    This module copyright (C) 2013 Savoir-faire Linux
+#    (<http://www.savoirfairelinux.com>).
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU Affero General Public License as
+#    published by the Free Software Foundation, either version 3 of the
+#    License, or (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU Affero General Public License for more details.
+#
+#    You should have received a copy of the GNU Affero General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+##############################################################################
+
+from openerp.openupgrade import openupgrade
+
+column_renames = {
+    'wiki_wiki': [
+        ('text_area', 'content'),
+        ('tags', None),
+        ('minor_edit', None),
+        ('review', None),
+        ('summary', None),
+        ('toc', None),
+        ('section', None),
+    ],
+    'wiki_groups': [
+        ('template', 'content'),
+    ],
+    'wiki_wiki_history': [
+        ('text_area', 'content'),
+        ('wiki_id', 'page_id'),
+        ('minor_edit', None),
+    ],
+}
+
+table_renames = [
+    ('wiki_wiki', 'document_page'),
+    ('wiki_wiki_history', 'document_page_history'),
+]
+
+model_renames = [
+    ('wiki.wiki', 'document.page'),
+    ('wiki.wiki.history', 'document.page.history'),
+]
+
+
[e-mail]()
+def migrate(cr, version):
+    openupgrade.rename_columns(cr, column_renames)
+    openupgrade.rename_tables(cr, table_renames)
+    openupgrade.rename_models(cr, model_renames)

=== added file 'document_page/migrations/7.0.1.0.1/user_notes.txt'
--- document_page/migrations/7.0.1.0.1/user_notes.txt	1970-01-01 00:00:00 +0000
+++ document_page/migrations/7.0.1.0.1/user_notes.txt	2013-10-15 19:37:17 +0000
@@ -0,0 +1,14 @@
+Groups turned into parent pages with type 'category'
+Wiky syntax converted to html using specifically made python port of Wiky.js
+
+Lost data:
+ - wiki_wiki.tags
+ - wiki_wiki.minor_edit
+ - wiki_wiki.review
+ - wiki_wiki.summary
+ - wiki_wiki.toc
+ - wiki_wiki.section
+ - wiki_wiki_history.minor_edit
+ - wiki_groups.summary
+ - wiki_make_index
+ - wiki_wiki_page_open