Merge lp:~dholbach/help-app/1425010 into lp:~ubuntu-touch-coreapps-drivers/help-app/trunk

[Daniel Holbach]

This is not a complete fix for bug 1425010, but is where I got stuck for now. Links are broken again.

[lp:~dholbach/help-app/1425010]

83. By Daniel Holbach on 2015-02-25

add code for adding a fake en-US profile - write translated files to pages/<lang>/<file.md>

84. By Daniel Holbach on 2015-02-25

define location of .pot file just once

85. By Daniel Holbach on 2015-02-25

rewrite links again, we are going to go with the solution of <filename>.<lang>.html, change paths accordingly

86. By Daniel Holbach on 2015-02-25

remove stray print statements

87. By Daniel Holbach on 2015-02-25

don't generate pointless duplicates of our pages, the 'original' is the 'en-us' variant

88. By Daniel Holbach on 2015-02-25

make pep8 happy

89. By Daniel Holbach on 2015-02-25

update .bzrignore list

90. By Daniel Holbach on 2015-02-25

remove 'lang' from markdown files and its special-casing, we get it for free from the generated filenames - we can likely remove more code now

91. By Daniel Holbach on 2015-02-25

remove unused code

92. By Daniel Holbach on 2015-02-25

simplify code some more

93. By Daniel Holbach on 2015-02-25

simplify Makefile logic ('make html' will work in top and 'edit-here' directory)

94. By Daniel Holbach on 2015-02-25

restructure functions in the translations classes to make more sense

95. By Daniel Holbach on 2015-02-25

make pep8 happy

[David Planella (dpm)]

Looks good to me, although as discussed yesterday on IRC, I'm still not sure about creating the fake en_US translation.

For the broken links, it seems what is happening is that during the build pelican is trying to find e.g. faq.html, which is not built (we build faq.en-us.html instead) so it gets confused and outputs the link as './'. If the index.md file is modified as below, then the generated links work (but for English only).

[Take me to the FAQ!]({filename}faq.en-us.md)

review: Needs Fixing

[lp:~dholbach/help-app/1425010]

96. By Daniel Holbach on 2015-02-26

rewrite links also for the case of en_US

[David Planella (dpm)]

This works well now. I've noticed something else, not sure if it needs to be fixed in this branch, though:

Whenever I click on the top-level 'de' links to switch to the German translation, the German pages contain an additional 'en' link that does not work. I think it should just be 'de' also for the German pages.

Translations: en-us en

Going forward, once we can load translations based on the user language, I'd recommend removing the translation links - languages should be set automatically for a smooth experience.

[Daniel Holbach (dholbach)]

I filed bug 1425924.

[Daniel Holbach (dholbach)]

<dholbach> dpm, the links were added automagically by pelican - I'll see what we can do and file a separate bug if that's ok
 dpm, I think we're now at the point where we can start adding content, fixing the styling and everything else
 dpm, shall I fix it in the same branch or in a separate branch?
<dpm> dholbach, yeah, let's do a separate branch
<dholbach> ok cool
 I'll land the other one, ok?
<dpm> dholbach, yeah. ...

review: Approve

=== modified file '.bzrignore'
--- .bzrignore	2015-02-20 16:06:22 +0000
+++ .bzrignore	2015-02-26 11:59:44 +0000
@@ -1,5 +1,6 @@
 app/www
 *.click
 edit-here/cache
-edit-here/content/pages/lang-??-*
 edit-here/backup
+edit-here/po/en_US.po
+edit-here/content/pages/*.*.md

=== modified file 'Makefile'
--- Makefile	2015-02-23 10:21:42 +0000
+++ Makefile	2015-02-26 11:59:44 +0000
@@ -11,7 +11,6 @@
 	cd app && click build . && mv *.click ..
 
 html: clean
-	cd edit-here && ./generate-translations
 	make -C edit-here html
 
 serve:

=== modified file 'edit-here/Makefile'
--- edit-here/Makefile	2015-02-13 16:08:50 +0000
+++ edit-here/Makefile	2015-02-26 11:59:44 +0000
@@ -58,6 +58,7 @@
 	@echo '                                                                       '
 
 html:
+	./generate-translations
 	$(PELICAN) $(INPUTDIR) -o $(OUTPUTDIR) -s $(CONFFILE) $(PELICANOPTS)
 
 clean:

=== modified file 'edit-here/content/pages/apps.md'
--- edit-here/content/pages/apps.md	2015-02-20 16:05:55 +0000
+++ edit-here/content/pages/apps.md	2015-02-26 11:59:44 +0000
@@ -1,5 +1,4 @@
 Title: FAQ - Apps
-Lang: en
 
 Are you looking for a good way to do X and can't find it in the app
 store? Want some suggestions to solve a particular problem?

=== modified file 'edit-here/content/pages/faq.md'
--- edit-here/content/pages/faq.md	2015-02-20 16:05:55 +0000
+++ edit-here/content/pages/faq.md	2015-02-26 11:59:44 +0000
@@ -1,5 +1,4 @@
 Title: Get your questions answered.
-Lang: en
 
 We divided the questions up into categories, so you can spot yours more 
 easily. Here we go:

=== modified file 'edit-here/content/pages/get-in-touch.md'
--- edit-here/content/pages/get-in-touch.md	2015-02-20 16:05:55 +0000
+++ edit-here/content/pages/get-in-touch.md	2015-02-26 11:59:44 +0000
@@ -1,5 +1,4 @@
 Title: Get in touch
-Lang: en
 
 Ubuntu has a huge community. You can easily get in touch with experts and
 other enthusiasts. One good way to do so, particularly if you **have 

=== modified file 'edit-here/content/pages/index.md'
--- edit-here/content/pages/index.md	2015-02-20 16:05:55 +0000
+++ edit-here/content/pages/index.md	2015-02-26 11:59:44 +0000
@@ -1,5 +1,4 @@
 Title: Welcome to Help for Ubuntu for devices!
-Lang: en
 
 The world-wide Ubuntu community wants to give you the best possible 
 experience on your Ubuntu device. This is why we collected:

=== modified file 'edit-here/generate-pot'
--- edit-here/generate-pot	2015-02-24 09:17:31 +0000
+++ edit-here/generate-pot	2015-02-26 11:59:44 +0000
@@ -10,8 +10,6 @@
 
 def main():
     translations = Translations()
-    if not translations.clean_documents():
-        sys.exit(1)
     translations.generate_pot_file()
     if not translations.update_po_files():
         sys.exit(1)

=== modified file 'edit-here/pelicanconf.py'
--- edit-here/pelicanconf.py	2015-02-24 09:35:13 +0000
+++ edit-here/pelicanconf.py	2015-02-26 11:59:44 +0000
@@ -14,11 +14,13 @@
 
 RELATIVE_URLS = True
 DIRECT_TEMPLATES = []
+
 SLUGIFY_SOURCE = 'basename'
-PAGE_URL =          '{slug}.html'
-PAGE_SAVE_AS =      '{slug}.html'
-PAGE_LANG_URL =     '{slug}.html'
-PAGE_LANG_SAVE_AS = '{slug}.html'
+FILENAME_METADATA = '(?P<slug>\S+)\.(?P<lang>\S+)\.*'
+PAGE_URL =          ''
+PAGE_SAVE_AS =      ''
+PAGE_LANG_URL =     '{slug}.{lang}.html'
+PAGE_LANG_SAVE_AS = '{slug}.{lang}.html'
 
 # Feed generation is usually not desired when developing
 FEED_ALL_ATOM = None

=== modified file 'edit-here/po/de.po'
--- edit-here/po/de.po	2015-02-24 13:17:06 +0000
+++ edit-here/po/de.po	2015-02-26 11:59:44 +0000
@@ -33,7 +33,7 @@
 #. type: Bullet: '  * '
 #: content/pages/faq.md:7
 msgid "[Apps]({filename}apps.md)"
-msgstr "[Apps]({filename}lang-de-apps.md)"
+msgstr "[Apps]({filename}apps.de.md)"
 
 #. type: Bullet: '  * '
 #: content/pages/faq.md:7
@@ -81,14 +81,14 @@
 #, fuzzy, no-wrap
 #| msgid "[Take me to the FAQ!](faq.html)  \n"
 msgid "[Take me to the FAQ!]({filename}faq.md)  \n"
-msgstr "[Zu den Antworten!]({filename}lang-de-faq.md)  \n"
+msgstr "[Zu den Antworten!]({filename}faq.de.md)  \n"
 
 #. type: Plain text
 #: content/pages/index.md:14
 #, fuzzy
 #| msgid "[Get in touch](get-in-touch.html)"
 msgid "[Get in touch]({filename}get-in-touch.md)"
-msgstr "[Kontakt]({filename}lang-de-get-in-touch.md)"
+msgstr "[Kontakt]({filename}get-in-touch.de.md)"
 
 #. type: Plain text
 #: content/pages/apps.md:2

=== modified file 'edit-here/translations.py'
--- edit-here/translations.py	2015-02-24 13:16:43 +0000
+++ edit-here/translations.py	2015-02-26 11:59:44 +0000
@@ -6,44 +6,35 @@
 import re
 import shutil
 import subprocess
-import tempfile
 
 from pelicanconf import PATH
 
-METADATA_TAGS = [
-    'title',
-    'lang',
-    'date',
-    ]
-
 # This defines how complete we expect translations to be before we
 # generate HTML from them. Needs to be string.
 TRANSLATION_COMPLETION_PERCENTAGE = '0'
 
+BCP47_OVERRIDES = (
+    ('zh_CN', 'zh-hans'),
+    ('zh_TW', 'zh-hant'),
+)
+
 
 class PO4A(object):
-    def __init__(self, translations_dir, temp_dir):
+    def __init__(self, pot_file):
         self.default_args = [
             '-f', 'text',
             '-o', 'markdown',
             '-M', 'utf-8',
             ]
-        self.translations_dir = translations_dir
-        self.temp_dir = temp_dir
+        self.pot_file = pot_file
 
-    def run(self, po4a_command, additional_args, with_output=False,
-            working_dir=None):
-        if not working_dir:
-            working_dir = self.temp_dir
-        pwd = os.getcwd()
+    def run(self, po4a_command, additional_args, with_output=False):
         args = copy.copy(self.default_args)
         args += additional_args
-        os.chdir(working_dir)
         if with_output:
             ret = subprocess.Popen([po4a_command]+args, stdout=subprocess.PIPE)
         else:
             ret = subprocess.call([po4a_command]+args)
-        os.chdir(pwd)
         return ret
 
     def gettextize(self, documents):
@@ -51,7 +42,7 @@
         for document in documents:
             args += ['-m', document]
         args += [
-            '-p', os.path.join(self.translations_dir, 'help.pot'),
+            '-p', self.pot_file,
             '-L', 'utf-8',
             ]
         return self.run('po4a-gettextize', args)
@@ -74,29 +65,80 @@
             '-p', lang.file_name,
             '-L', 'utf-8',
             ]
-        return self.run('po4a-translate', args, with_output=True,
-                        working_dir=os.path.join(PATH, '..'))
+        return self.run('po4a-translate', args, with_output=True)
+
+
+class Documents(object):
+    def __init__(self):
+        self.docs = []
+        for dirpath, dirnames, filenames in os.walk(PATH):
+            for filename in filenames:
+                self.docs += [os.path.join(dirpath, filename)]
+
+    def translated_doc(self, file_name, lang):
+        match = [doc for doc in self.docs
+                 if os.path.basename(doc) == os.path.basename(file_name)]
+        if not match:
+            return None
+        return '%s.%s.md' % (match[0].split('.md')[0],
+                             lang.bcp47_code)
+
+    def _call_po4a_translate(self, doc, lang, po4a):
+        res = po4a.translate(doc, lang)
+        output = codecs.decode(res.communicate()[0])
+        broken_title_line = [line for line in output.split('\n')
+                             if line.lower().startswith('title:')][0]
+        rest = [line for line in output.split('\n')
+                if not line.lower().startswith('title')]
+        output = '\n'.join(rest)
+        return (broken_title_line, output)
+
+    def write_translated_markdown(self, lang, po4a):
+        for doc in self.docs:
+            (broken_title_line, output) = \
+                self._call_po4a_translate(doc, lang, po4a)
+            new_path = self.translated_doc(doc, lang)
+            text = "%s\nDate:\n\n" % (broken_title_line)
+            text += output
+            if os.path.exists(new_path):
+                os.remove(new_path)
+            if not os.path.exists(os.path.dirname(new_path)):
+                os.makedirs(os.path.dirname(new_path))
+            with open(new_path, 'w', encoding='utf-8') as f:
+                f.write(text)
 
 
 class Language(object):
-    def __init__(self, po_file):
+    def __init__(self, po_file, documents):
         self.file_name = po_file
-        self.gettext_lang_code = os.path.basename(po_file).split('.po')[0]
+        self.gettext_code = os.path.basename(po_file).split('.po')[0]
+        self.bcp47_code = self._find_bcp47_code()
+        self.documents = documents
+
+    def _find_bcp47_code(self):
+        if self.gettext_code not in [c[0] for c in BCP47_OVERRIDES]:
+            return self.gettext_code.lower().replace('_', '-')
+        return [c[1] for c in BCP47_OVERRIDES
+                if c[0] == self.gettext_code][0]
 
     def rewrite_links(self):
         po_file = polib.pofile(self.file_name)
-        for entry_group in [po_file.untranslated_entries(),
-                            po_file.translated_entries(),
-                            po_file.fuzzy_entries()]:
+        link_regex = r'\[.+?\]\(\{filename\}(.+?)\)'
+        for entry_group in [po_file.translated_entries(),
+                            po_file.fuzzy_entries(),
+                            po_file.untranslated_entries()]:
             for entry in entry_group:
                 if '{filename}' in entry.msgid:
-                    if not entry.msgstr:
+                    link_msgid = re.findall(link_regex, entry.msgid)[0]
+                    link_msgstr = list(re.findall(link_regex, entry.msgstr))
+                    translated_doc = os.path.basename(
+                        self.documents.translated_doc(
+                            link_msgid, self))
+                    if not link_msgstr:
                         entry.msgstr = entry.msgid
-                    link = re.findall(r'\[.+?\]\(\{filename\}(.+?)\)',
-                                      entry.msgid)[0]
-                    entry.msgstr = entry.msgstr.replace(
-                        link,
-                        'lang-%s-%s' % (self.gettext_lang_code, link))
+                        link_msgstr = [link_msgid]
+                    entry.msgstr = entry.msgstr.replace(link_msgstr[0],
+                                                        translated_doc)
         po_file.save(self.file_name)
 
 
@@ -105,21 +147,19 @@
         self._cleanup()
         self.translations_dir = os.path.abspath(os.path.join(PATH, '../po'))
         self.available_languages = []
+        self.documents = Documents()
         for po_filename in glob.glob(self.translations_dir+'/*.po'):
-            self.available_languages += [Language(po_filename)]
-        self.documents = self._find_documents()
-        self.temp_dir = tempfile.mkdtemp()
-        self.po4a = PO4A(self.translations_dir, self.temp_dir)
+            self.available_languages += [Language(po_filename, self.documents)]
+        self.fake_lang_code = 'en_US'
+        self.fake_po_file = os.path.join(self.translations_dir,
+                                         '%s.po' % self.fake_lang_code)
+        self.pot_file = os.path.join(self.translations_dir,
+                                     "help.pot")
+        self.po4a = PO4A(self.pot_file)
 
     def __del__(self):
-        shutil.rmtree(self.temp_dir)
-
-    def _find_documents(self):
-        documents = []
-        for dirpath, dirnames, filenames in os.walk(PATH):
-            for filename in filenames:
-                documents += [os.path.join(dirpath, filename)]
-        return documents
+        if os.path.exists(self.fake_po_file):
+            os.remove(self.fake_po_file)
 
     def _cleanup(self):
         r = subprocess.Popen(['bzr', 'ignored'], stdout=subprocess.PIPE)
@@ -133,80 +173,27 @@
             except NotADirectoryError:
                 os.remove(f)
 
-    def _remove_metadata(self, filename):
-        new_dir = os.path.join(self.temp_dir, os.path.dirname(filename))
-        os.makedirs(new_dir, exist_ok=True)
-        new_filename = os.path.join(new_dir, os.path.basename(filename))
-        shutil.copy(filename, new_filename)
-        lines = open(new_filename).readlines()
-        title_line = [x for x in lines if
-                      (x.startswith('Title:') or x.startswith('title:'))]
-        index = 0
-        for line in lines:
-            if not [x for x in METADATA_TAGS
-                    if line.lower().startswith(x+':')] and \
-               line.strip() != '':
-                index = lines.index(line)
-                break
-        if not title_line:
-            print('No line starting with "Title: " found in "%s".' %
-                  new_filename)
-            return False
-        os.remove(new_filename)
-        with open(new_filename, 'w', encoding='utf-8') as new_file:
-            new_file.write(title_line[0]+'\n')
-            new_file.write(''.join(lines[index:]))
-        return True
-
-    def clean_documents(self):
-        for document in self.documents:
-            if not self._remove_metadata(document):
-                return False
-        return True
-
     def generate_pot_file(self):
-        return self.po4a.gettextize(self.documents)
+        return self.po4a.gettextize(self.documents.docs)
 
     def update_po_files(self):
-        return self.po4a.updatepo(self.available_languages, self.documents)
-
-    def _call_po4a_translate(self, doc, lang):
-        res = self.po4a.translate(doc, lang)
-        output = codecs.decode(res.communicate()[0])
-        broken_title_line = [line for line in output.split('\n')
-                             if line.lower().startswith('title:')][0]
-        rest = [line for line in output.split('\n')
-                if not line.lower().startswith('title')]
-        output = '\n'.join(rest)
-        return (broken_title_line, output)
-
-    def _new_header(self, lang, broken_title_line):
-        title_line = broken_title_line
-        for metadata_tag in [x for x in METADATA_TAGS if not x == 'title']:
-            title_line = title_line.split(metadata_tag)[0]
-            title_line = title_line.split(metadata_tag.capitalize())[0]
-            title_line = title_line.split(metadata_tag.upper())[0]
-        return "%s\nLang: %s\nDate:\n\n" % \
-            (title_line, lang.gettext_lang_code)
+        return self.po4a.updatepo(self.available_languages,
+                                  self.documents.docs)
 
     def rewrite_links(self):
+        self._generate_fake_pofile()
         for lang in self.available_languages:
             lang.rewrite_links()
 
+    # we generate a fake translation for en-US which is going to be
+    # the default
+    def _generate_fake_pofile(self):
+        if os.path.exists(self.fake_po_file):
+            os.remove(self.fake_po_file)
+        shutil.copy(self.pot_file, self.fake_po_file)
+        self.available_languages += [Language(self.fake_po_file,
+                                              self.documents)]
+
     def generate_translations(self):
         for lang in self.available_languages:
-            for doc in self.documents:
-                (broken_title_line, output) = \
-                    self._call_po4a_translate(doc, lang)
-                new_path = os.path.join(PATH, 'pages',
-                                        'lang-%s-%s' %
-                                        (lang.gettext_lang_code,
-                                         os.path.basename(doc)))
-                text = self._new_header(lang, broken_title_line)
-                text += output
-                if os.path.exists(new_path):
-                    os.remove(new_path)
-                if not os.path.exists(os.path.dirname(new_path)):
-                    os.makedirs(os.path.dirname(new_path))
-                with open(new_path, 'w', encoding='utf-8') as f:
-                    f.write(text)
+            self.documents.write_translated_markdown(lang, self.po4a)