Merge lp:~songofacandy/bzr/i18n-msgfmt into lp:bzr

[John A Meinel]

add build_mo command to setup.py

This merge proposal adds build_mo command to setup.py
This command builds bzrlib/locale/<LANG>/LC_MESSAGES/*.mo files from po/*.po.

The directory format draft is described at http://wiki.bazaar.canonical.com/DraftSpecs/I18nSupport.

(resubmitted to set the Prerequisite branch)

[methane (songofacandy)]

Uh, this mp is based on lp:~songofacandy/bzr/i18n-msgextract.
Extra diffs may be removed after that branch is merged.

[John A Meinel (jameinel)]

Do you know that you can set a Prerequisite branch as part of submitting the merge proposal? Changes in the prerequisite branch won't be shown in the diff. I think you can resubmit this and add the appropriate prereq. I'll try.

[methane (songofacandy)]

Oh, I didn't know that. Thank you.

On Mon, May 16, 2011 at 6:48 PM, John A Meinel <email address hidden> wrote:
> Do you know that you can set a Prerequisite branch as part of submitting the merge proposal? Changes in the prerequisite branch won't be shown in the diff. I think you can resubmit this and add the appropriate prereq. I'll try.
> --
> https://code.launchpad.net/~songofacandy/bzr/i18n-msgfmt/+merge/61030
> You are the owner of lp:~songofacandy/bzr/i18n-msgfmt.
>

--
INADA Naoki  <email address hidden>

[Martin Pool (mbp)]

Sorry if I missed this in the list thread, but: can you tell me why we're adding a build_mo command, rather than using the external ones?

+ if 'en' in self.lang:
+ if find_executable('msginit') is None:
+ log.warn("GNU gettext msginit utility not found!")
+ log.warn("Skip creating English PO file.")
+ else:
+ log.info('Creating English PO file...')
+ pot = (self.prj_name or 'messages') + '.pot'
+ if self.prj_name:
+ en_po = '%s-en.po' % self.prj_name
+ else:
+ en_po = 'en.po'
+ self.spawn(['msginit',
+ '--no-translator',
+ '-l', 'en',
+ '-i', os.path.join(self.source_dir, pot),
+ '-o', os.path.join(self.source_dir, en_po),
+ ])
+

Could you explain this a bit more to me?

Thanks

review: Needs Information

[methane (songofacandy)]

On Wed, May 18, 2011 at 12:01 AM, Martin Pool <email address hidden> wrote:
> Review: Needs Information
> Sorry if I missed this in the list thread, but: can you tell me why we're adding a build_mo command, rather than using the external ones?

setup.py command or Makefile command is needed with other tool, too.
So the problem is "msgfmt.py or external msgfmt command."

The advantage of msgfmt.py is that users don't need msgfmt command.
Users building bzr from source must have Python, of course.
But msgfmt tool is extra dependency for not only windows users, but also
Ubuntu users. Ubuntu doesn't installs msgfmt default and not all users
have root. Installing msgfmt is pain for non root users. Please imagine
"pip install --user bzr".

On the other hand, I don't imagine advantage of external msgfmt tool.

>
> +        if 'en' in self.lang:
> +            if find_executable('msginit') is None:
> +                log.warn("GNU gettext msginit utility not found!")
> +                log.warn("Skip creating English PO file.")
> +            else:
> +                log.info('Creating English PO file...')
> +                pot = (self.prj_name or 'messages') + '.pot'
> +                if self.prj_name:
> +                    en_po = '%s-en.po' % self.prj_name
> +                else:
> +                    en_po = 'en.po'
> +                self.spawn(['msginit',
> +                    '--no-translator',
> +                    '-l', 'en',
> +                    '-i', os.path.join(self.source_dir, pot),
> +                    '-o', os.path.join(self.source_dir, en_po),
> +                    ])
> +
>
> Could you explain this a bit more to me?

I've copied it from bzr-explorer and have didn't mind...
SImply, adding empty english po file is enough solution to remove
msginit dependency.

>
> Thanks
> --
> https://code.launchpad.net/~songofacandy/bzr/i18n-msgfmt/+merge/61082
> You are the owner of lp:~songofacandy/bzr/i18n-msgfmt.
>

--
INADA Naoki  <email address hidden>

[Martin Pool (mbp)]

On 17 May 2011 18:49, INADA Naoki <email address hidden> wrote:
> On Wed, May 18, 2011 at 12:01 AM, Martin Pool <email address hidden> wrote:
>> Review: Needs Information
>> Sorry if I missed this in the list thread, but: can you tell me why we're adding a build_mo command, rather than using the external ones?
>
> setup.py command or Makefile command is needed with other tool, too.
> So the problem is "msgfmt.py or external msgfmt command."
>
> The advantage of msgfmt.py is that users don't need msgfmt command.
> Users building bzr from source must have Python, of course.
> But msgfmt tool is extra dependency for not only windows users, but also
> Ubuntu users. Ubuntu doesn't installs msgfmt default and not all users
> have root. Installing msgfmt is pain for non root users. Please imagine
> "pip install --user bzr".
>
> On the other hand, I don't imagine advantage of external msgfmt tool.

ok, that makes sense. thanks!

[Martin Pool (mbp)]

sent to pqm by email

=== modified file 'setup.py'
--- setup.py	2011-05-11 16:35:34 +0000
+++ setup.py	2011-05-17 17:51:36 +0000
@@ -69,7 +69,8 @@
                                         'tests/ssl_certs/ca.crt',
                                         'tests/ssl_certs/server_without_pass.key',
                                         'tests/ssl_certs/server_with_pass.key',
-                                        'tests/ssl_certs/server.crt'
+                                        'tests/ssl_certs/server.crt',
+                                        'locale/*/LC_MESSAGES/*.mo',
                                        ]},
            }
 
@@ -152,6 +153,10 @@
     Generate bzr.1.
     """
 
+    sub_commands = build.sub_commands + [
+            ('build_mo', lambda _: True),
+            ]
+
     def run(self):
         build.run(self)
 
@@ -163,8 +168,12 @@
 ## Setup
 ########################
 
+from tools.build_mo import build_mo
+
 command_classes = {'install_scripts': my_install_scripts,
-                   'build': bzr_build}
+                   'build': bzr_build,
+                   'build_mo': build_mo,
+                   }
 from distutils import log
 from distutils.errors import CCompilerError, DistutilsPlatformError
 from distutils.extension import Extension

=== added file 'tools/build_mo.py'
--- tools/build_mo.py	1970-01-01 00:00:00 +0000
+++ tools/build_mo.py	2011-05-17 17:51:36 +0000
@@ -0,0 +1,107 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2007 Lukáš Lalinský <[e-mail]>
+# Copyright (C) 2007,2009 Alexander Belchenko <[e-mail]>
+# Copyright 2011 Canonical Ltd.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+
+# This code is bring from bzr-explorer and modified for bzr.
+
+"""build_mo command for setup.py"""
+
+from distutils import log
+from distutils.command.build import build
+from distutils.core import Command
+from distutils.dep_util import newer
+from distutils.spawn import find_executable
+import os
+import re
+
+import msgfmt
+
+class build_mo(Command):
+    """Subcommand of build command: build_mo"""
+
+    description = 'compile po files to mo files'
+
+    # List of options:
+    #   - long name,
+    #   - short name (None if no short name),
+    #   - help string.
+    user_options = [('build-dir=', 'd', 'Directory to build locale files'),
+                    ('output-base=', 'o', 'mo-files base name'),
+                    ('source-dir=', None, 'Directory with sources po files'),
+                    ('force', 'f', 'Force creation of mo files'),
+                    ('lang=', None, 'Comma-separated list of languages '
+                                    'to process'),
+                   ]
+
+    boolean_options = ['force']
+
+    def initialize_options(self):
+        self.build_dir = None
+        self.output_base = None
+        self.source_dir = None
+        self.force = None
+        self.lang = None
+
+    def finalize_options(self):
+        self.set_undefined_options('build', ('force', 'force'))
+        self.prj_name = self.distribution.get_name()
+        if self.build_dir is None:
+            self.build_dir = 'bzrlib/locale'
+        if not self.output_base:
+            self.output_base = self.prj_name or 'messages'
+        if self.source_dir is None:
+            self.source_dir = 'po'
+        if self.lang is None:
+            if self.prj_name:
+                re_po = re.compile(r'^(?:%s-)?([a-zA-Z_]+)\.po$' % self.prj_name)
+            else:
+                re_po = re.compile(r'^([a-zA-Z_]+)\.po$')
+            self.lang = []
+            for i in os.listdir(self.source_dir):
+                mo = re_po.match(i)
+                if mo:
+                    self.lang.append(mo.group(1))
+        else:
+            self.lang = [i.strip() for i in self.lang.split(',') if i.strip()]
+
+    def run(self):
+        """Run msgfmt for each language"""
+        if not self.lang:
+            return
+
+        basename = self.output_base
+        if not basename.endswith('.mo'):
+            basename += '.mo'
+
+        po_prefix = ''
+        if self.prj_name:
+            po_prefix = self.prj_name + '-'
+        for lang in self.lang:
+            po = os.path.join('po', lang + '.po')
+            if not os.path.isfile(po):
+                po = os.path.join('po', po_prefix + lang + '.po')
+            dir_ = os.path.join(self.build_dir, lang, 'LC_MESSAGES')
+            self.mkpath(dir_)
+            mo = os.path.join(dir_, basename)
+            if self.force or newer(po, mo):
+                log.info('Compile: %s -> %s' % (po, mo))
+                msgfmt.make(po,  mo)
+
+
+build.sub_commands.insert(0, ('build_mo', None))

=== added file 'tools/msgfmt.py'
--- tools/msgfmt.py	1970-01-01 00:00:00 +0000
+++ tools/msgfmt.py	2011-05-17 17:51:36 +0000
@@ -0,0 +1,208 @@
+#! /usr/bin/env python
+# -*- coding: utf-8 -*-
+# Written by Martin v. Lowis <[e-mail]>
+
+# This script is copied from Python/Tools/i18n/msgfmt.py
+# It licensed under PSF license which compatible with GPL.
+#
+# ChangeLog
+# * Convert encoding from iso-8859-1 to utf-8
+# * Fix http://bugs.python.org/issue9741
+
+"""Generate binary message catalog from textual translation description.
+
+This program converts a textual Uniforum-style message catalog (.po file) into
+a binary GNU catalog (.mo file).  This is essentially the same function as the
+GNU msgfmt program, however, it is a simpler implementation.
+
+Usage: msgfmt.py [OPTIONS] filename.po
+
+Options:
+    -o file
+    --output-file=file
+        Specify the output file to write to.  If omitted, output will go to a
+        file named filename.mo (based off the input file name).
+
+    -h
+    --help
+        Print this message and exit.
+
+    -V
+    --version
+        Display version information and exit.
+"""
+
+import sys
+import os
+import getopt
+import struct
+import array
+
+__version__ = "1.1"
+
+MESSAGES = {}
+
+
+def usage(code, msg=''):
+    print >> sys.stderr, __doc__
+    if msg:
+        print >> sys.stderr, msg
+    sys.exit(code)
+
+
+def add(id, str, fuzzy):
+    "Add a non-fuzzy translation to the dictionary."
+    global MESSAGES
+    if not fuzzy and str:
+        MESSAGES[id] = str
+
+
+def generate():
+    "Return the generated output."
+    global MESSAGES
+    keys = MESSAGES.keys()
+    # the keys are sorted in the .mo file
+    keys.sort()
+    offsets = []
+    ids = strs = ''
+    for id in keys:
+        # For each string, we need size and file offset.  Each string is NUL
+        # terminated; the NUL does not count into the size.
+        offsets.append((len(ids), len(id), len(strs), len(MESSAGES[id])))
+        ids += id + '\0'
+        strs += MESSAGES[id] + '\0'
+    output = ''
+    # The header is 7 32-bit unsigned integers.  We don't use hash tables, so
+    # the keys start right after the index tables.
+    # translated string.
+    keystart = 7*4+16*len(keys)
+    # and the values start after the keys
+    valuestart = keystart + len(ids)
+    koffsets = []
+    voffsets = []
+    # The string table first has the list of keys, then the list of values.
+    # Each entry has first the size of the string, then the file offset.
+    for o1, l1, o2, l2 in offsets:
+        koffsets += [l1, o1+keystart]
+        voffsets += [l2, o2+valuestart]
+    offsets = koffsets + voffsets
+    output = struct.pack("Iiiiiii",
+                         0x950412deL,       # Magic
+                         0,                 # Version
+                         len(keys),         # # of entries
+                         7*4,               # start of key index
+                         7*4+len(keys)*8,   # start of value index
+                         0, 0)              # size and offset of hash table
+    output += array.array("i", offsets).tostring()
+    output += ids
+    output += strs
+    return output
+
+
+def make(filename, outfile):
+    # Fix http://bugs.python.org/issue9741
+    global MESSAGES
+    MESSAGES.clear()
+    ID = 1
+    STR = 2
+
+    # Compute .mo name from .po name and arguments
+    if filename.endswith('.po'):
+        infile = filename
+    else:
+        infile = filename + '.po'
+    if outfile is None:
+        outfile = os.path.splitext(infile)[0] + '.mo'
+
+    try:
+        lines = open(infile).readlines()
+    except IOError, msg:
+        print >> sys.stderr, msg
+        sys.exit(1)
+
+    section = None
+    fuzzy = 0
+
+    # Parse the catalog
+    lno = 0
+    for l in lines:
+        lno += 1
+        # If we get a comment line after a msgstr, this is a new entry
+        if l[0] == '#' and section == STR:
+            add(msgid, msgstr, fuzzy)
+            section = None
+            fuzzy = 0
+        # Record a fuzzy mark
+        if l[:2] == '#,' and 'fuzzy' in l:
+            fuzzy = 1
+        # Skip comments
+        if l[0] == '#':
+            continue
+        # Now we are in a msgid section, output previous section
+        if l.startswith('msgid'):
+            if section == STR:
+                add(msgid, msgstr, fuzzy)
+            section = ID
+            l = l[5:]
+            msgid = msgstr = ''
+        # Now we are in a msgstr section
+        elif l.startswith('msgstr'):
+            section = STR
+            l = l[6:]
+        # Skip empty lines
+        l = l.strip()
+        if not l:
+            continue
+        # XXX: Does this always follow Python escape semantics?
+        l = eval(l)
+        if section == ID:
+            msgid += l
+        elif section == STR:
+            msgstr += l
+        else:
+            print >> sys.stderr, 'Syntax error on %s:%d' % (infile, lno), \
+                  'before:'
+            print >> sys.stderr, l
+            sys.exit(1)
+    # Add last entry
+    if section == STR:
+        add(msgid, msgstr, fuzzy)
+
+    # Compute output
+    output = generate()
+
+    try:
+        open(outfile,"wb").write(output)
+    except IOError,msg:
+        print >> sys.stderr, msg
+
+
+def main():
+    try:
+        opts, args = getopt.getopt(sys.argv[1:], 'hVo:',
+                                   ['help', 'version', 'output-file='])
+    except getopt.error, msg:
+        usage(1, msg)
+
+    outfile = None
+    # parse options
+    for opt, arg in opts:
+        if opt in ('-h', '--help'):
+            usage(0)
+        elif opt in ('-V', '--version'):
+            print >> sys.stderr, "msgfmt.py", __version__
+            sys.exit(0)
+        elif opt in ('-o', '--output-file'):
+            outfile = arg
+    # do it
+    if not args:
+        print >> sys.stderr, 'No input file given'
+        print >> sys.stderr, "Try `msgfmt --help' for more information."
+        return
+
+    for filename in args:
+        make(filename, outfile)
+
+
+if __name__ == '__main__':
+    main()