Merge lp:~songofacandy/bzr/i18n-msgfmt into lp:bzr

Proposed by John A Meinel on 2011-05-16
Status: Merged
Approved by: Martin Pool on 2011-05-18
Approved revision: 5839
Merged at revision: 5891
Proposed branch: lp:~songofacandy/bzr/i18n-msgfmt
Merge into: lp:bzr
Prerequisite: lp:~songofacandy/bzr/i18n-msgextract
Diff against target: 363 lines (+326/-2)
3 files modified
setup.py (+11/-2)
tools/build_mo.py (+107/-0)
tools/msgfmt.py (+208/-0)
To merge this branch: bzr merge lp:~songofacandy/bzr/i18n-msgfmt
Reviewer Review Type Date Requested Status
Martin Pool 2011-05-16 Needs Information on 2011-05-17
Review via email: mp+61082@code.launchpad.net

This proposal supersedes a proposal from 2011-05-15.

Commit message

add build_mo command to setup.py

Description of the change

This merge proposal adds build_mo command to setup.py
This command builds bzrlib/locale/<LANG>/LC_MESSAGES/*.mo files from po/*.po.

The directory format draft is described at http://wiki.bazaar.canonical.com/DraftSpecs/I18nSupport.

(resubmitted to set the Prerequisite branch)

To post a comment you must log in.
INADA Naoki (songofacandy) wrote : Posted in a previous version of this proposal

Uh, this mp is based on lp:~songofacandy/bzr/i18n-msgextract.
Extra diffs may be removed after that branch is merged.

John A Meinel (jameinel) wrote : Posted in a previous version of this proposal

Do you know that you can set a Prerequisite branch as part of submitting the merge proposal? Changes in the prerequisite branch won't be shown in the diff. I think you can resubmit this and add the appropriate prereq. I'll try.

INADA Naoki (songofacandy) wrote : Posted in a previous version of this proposal

Oh, I didn't know that. Thank you.

On Mon, May 16, 2011 at 6:48 PM, John A Meinel <email address hidden> wrote:
> Do you know that you can set a Prerequisite branch as part of submitting the merge proposal? Changes in the prerequisite branch won't be shown in the diff. I think you can resubmit this and add the appropriate prereq. I'll try.
> --
> https://code.launchpad.net/~songofacandy/bzr/i18n-msgfmt/+merge/61030
> You are the owner of lp:~songofacandy/bzr/i18n-msgfmt.
>

--
INADA Naoki  <email address hidden>

lp:~songofacandy/bzr/i18n-msgfmt updated on 2011-05-17
5838. By INADA Naoki on 2011-05-17

merge from lp:bzr

Martin Pool (mbp) wrote :

Sorry if I missed this in the list thread, but: can you tell me why we're adding a build_mo command, rather than using the external ones?

+ if 'en' in self.lang:
+ if find_executable('msginit') is None:
+ log.warn("GNU gettext msginit utility not found!")
+ log.warn("Skip creating English PO file.")
+ else:
+ log.info('Creating English PO file...')
+ pot = (self.prj_name or 'messages') + '.pot'
+ if self.prj_name:
+ en_po = '%s-en.po' % self.prj_name
+ else:
+ en_po = 'en.po'
+ self.spawn(['msginit',
+ '--no-translator',
+ '-l', 'en',
+ '-i', os.path.join(self.source_dir, pot),
+ '-o', os.path.join(self.source_dir, en_po),
+ ])
+

Could you explain this a bit more to me?

Thanks

review: Needs Information
INADA Naoki (songofacandy) wrote :

On Wed, May 18, 2011 at 12:01 AM, Martin Pool <email address hidden> wrote:
> Review: Needs Information
> Sorry if I missed this in the list thread, but: can you tell me why we're adding a build_mo command, rather than using the external ones?

setup.py command or Makefile command is needed with other tool, too.
So the problem is "msgfmt.py or external msgfmt command."

The advantage of msgfmt.py is that users don't need msgfmt command.
Users building bzr from source must have Python, of course.
But msgfmt tool is extra dependency for not only windows users, but also
Ubuntu users. Ubuntu doesn't installs msgfmt default and not all users
have root. Installing msgfmt is pain for non root users. Please imagine
"pip install --user bzr".

On the other hand, I don't imagine advantage of external msgfmt tool.

>
> +        if 'en' in self.lang:
> +            if find_executable('msginit') is None:
> +                log.warn("GNU gettext msginit utility not found!")
> +                log.warn("Skip creating English PO file.")
> +            else:
> +                log.info('Creating English PO file...')
> +                pot = (self.prj_name or 'messages') + '.pot'
> +                if self.prj_name:
> +                    en_po = '%s-en.po' % self.prj_name
> +                else:
> +                    en_po = 'en.po'
> +                self.spawn(['msginit',
> +                    '--no-translator',
> +                    '-l', 'en',
> +                    '-i', os.path.join(self.source_dir, pot),
> +                    '-o', os.path.join(self.source_dir, en_po),
> +                    ])
> +
>
> Could you explain this a bit more to me?

I've copied it from bzr-explorer and have didn't mind...
SImply, adding empty english po file is enough solution to remove
msginit dependency.

>
> Thanks
> --
> https://code.launchpad.net/~songofacandy/bzr/i18n-msgfmt/+merge/61082
> You are the owner of lp:~songofacandy/bzr/i18n-msgfmt.
>

--
INADA Naoki  <email address hidden>

lp:~songofacandy/bzr/i18n-msgfmt updated on 2011-05-17
5839. By INADA Naoki on 2011-05-17

Remove msginit dependency.

Martin Pool (mbp) wrote :

On 17 May 2011 18:49, INADA Naoki <email address hidden> wrote:
> On Wed, May 18, 2011 at 12:01 AM, Martin Pool <email address hidden> wrote:
>> Review: Needs Information
>> Sorry if I missed this in the list thread, but: can you tell me why we're adding a build_mo command, rather than using the external ones?
>
> setup.py command or Makefile command is needed with other tool, too.
> So the problem is "msgfmt.py or external msgfmt command."
>
> The advantage of msgfmt.py is that users don't need msgfmt command.
> Users building bzr from source must have Python, of course.
> But msgfmt tool is extra dependency for not only windows users, but also
> Ubuntu users. Ubuntu doesn't installs msgfmt default and not all users
> have root. Installing msgfmt is pain for non root users. Please imagine
> "pip install --user bzr".
>
> On the other hand, I don't imagine advantage of external msgfmt tool.

ok, that makes sense. thanks!

Martin Pool (mbp) wrote :

sent to pqm by email

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'setup.py'
--- setup.py 2011-05-11 16:35:34 +0000
+++ setup.py 2011-05-17 17:51:36 +0000
@@ -69,7 +69,8 @@
69 'tests/ssl_certs/ca.crt',69 'tests/ssl_certs/ca.crt',
70 'tests/ssl_certs/server_without_pass.key',70 'tests/ssl_certs/server_without_pass.key',
71 'tests/ssl_certs/server_with_pass.key',71 'tests/ssl_certs/server_with_pass.key',
72 'tests/ssl_certs/server.crt'72 'tests/ssl_certs/server.crt',
73 'locale/*/LC_MESSAGES/*.mo',
73 ]},74 ]},
74 }75 }
7576
@@ -152,6 +153,10 @@
152 Generate bzr.1.153 Generate bzr.1.
153 """154 """
154155
156 sub_commands = build.sub_commands + [
157 ('build_mo', lambda _: True),
158 ]
159
155 def run(self):160 def run(self):
156 build.run(self)161 build.run(self)
157162
@@ -163,8 +168,12 @@
163## Setup168## Setup
164########################169########################
165170
171from tools.build_mo import build_mo
172
166command_classes = {'install_scripts': my_install_scripts,173command_classes = {'install_scripts': my_install_scripts,
167 'build': bzr_build}174 'build': bzr_build,
175 'build_mo': build_mo,
176 }
168from distutils import log177from distutils import log
169from distutils.errors import CCompilerError, DistutilsPlatformError178from distutils.errors import CCompilerError, DistutilsPlatformError
170from distutils.extension import Extension179from distutils.extension import Extension
171180
=== added file 'tools/build_mo.py'
--- tools/build_mo.py 1970-01-01 00:00:00 +0000
+++ tools/build_mo.py 2011-05-17 17:51:36 +0000
@@ -0,0 +1,107 @@
1# -*- coding: utf-8 -*-
2#
3# Copyright (C) 2007 Lukáš Lalinský <lalinsky@gmail.com>
4# Copyright (C) 2007,2009 Alexander Belchenko <bialix@ukr.net>
5# Copyright 2011 Canonical Ltd.
6#
7# This program is free software; you can redistribute it and/or
8# modify it under the terms of the GNU General Public License
9# as published by the Free Software Foundation; either version 2
10# of the License, or (at your option) any later version.
11#
12# This program is distributed in the hope that it will be useful,
13# but WITHOUT ANY WARRANTY; without even the implied warranty of
14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
15# GNU General Public License for more details.
16#
17# You should have received a copy of the GNU General Public License
18# along with this program; if not, write to the Free Software
19# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
20
21# This code is bring from bzr-explorer and modified for bzr.
22
23"""build_mo command for setup.py"""
24
25from distutils import log
26from distutils.command.build import build
27from distutils.core import Command
28from distutils.dep_util import newer
29from distutils.spawn import find_executable
30import os
31import re
32
33import msgfmt
34
35class build_mo(Command):
36 """Subcommand of build command: build_mo"""
37
38 description = 'compile po files to mo files'
39
40 # List of options:
41 # - long name,
42 # - short name (None if no short name),
43 # - help string.
44 user_options = [('build-dir=', 'd', 'Directory to build locale files'),
45 ('output-base=', 'o', 'mo-files base name'),
46 ('source-dir=', None, 'Directory with sources po files'),
47 ('force', 'f', 'Force creation of mo files'),
48 ('lang=', None, 'Comma-separated list of languages '
49 'to process'),
50 ]
51
52 boolean_options = ['force']
53
54 def initialize_options(self):
55 self.build_dir = None
56 self.output_base = None
57 self.source_dir = None
58 self.force = None
59 self.lang = None
60
61 def finalize_options(self):
62 self.set_undefined_options('build', ('force', 'force'))
63 self.prj_name = self.distribution.get_name()
64 if self.build_dir is None:
65 self.build_dir = 'bzrlib/locale'
66 if not self.output_base:
67 self.output_base = self.prj_name or 'messages'
68 if self.source_dir is None:
69 self.source_dir = 'po'
70 if self.lang is None:
71 if self.prj_name:
72 re_po = re.compile(r'^(?:%s-)?([a-zA-Z_]+)\.po$' % self.prj_name)
73 else:
74 re_po = re.compile(r'^([a-zA-Z_]+)\.po$')
75 self.lang = []
76 for i in os.listdir(self.source_dir):
77 mo = re_po.match(i)
78 if mo:
79 self.lang.append(mo.group(1))
80 else:
81 self.lang = [i.strip() for i in self.lang.split(',') if i.strip()]
82
83 def run(self):
84 """Run msgfmt for each language"""
85 if not self.lang:
86 return
87
88 basename = self.output_base
89 if not basename.endswith('.mo'):
90 basename += '.mo'
91
92 po_prefix = ''
93 if self.prj_name:
94 po_prefix = self.prj_name + '-'
95 for lang in self.lang:
96 po = os.path.join('po', lang + '.po')
97 if not os.path.isfile(po):
98 po = os.path.join('po', po_prefix + lang + '.po')
99 dir_ = os.path.join(self.build_dir, lang, 'LC_MESSAGES')
100 self.mkpath(dir_)
101 mo = os.path.join(dir_, basename)
102 if self.force or newer(po, mo):
103 log.info('Compile: %s -> %s' % (po, mo))
104 msgfmt.make(po, mo)
105
106
107build.sub_commands.insert(0, ('build_mo', None))
0108
=== added file 'tools/msgfmt.py'
--- tools/msgfmt.py 1970-01-01 00:00:00 +0000
+++ tools/msgfmt.py 2011-05-17 17:51:36 +0000
@@ -0,0 +1,208 @@
1#! /usr/bin/env python
2# -*- coding: utf-8 -*-
3# Written by Martin v. Lowis <loewis@informatik.hu-berlin.de>
4
5# This script is copied from Python/Tools/i18n/msgfmt.py
6# It licensed under PSF license which compatible with GPL.
7#
8# ChangeLog
9# * Convert encoding from iso-8859-1 to utf-8
10# * Fix http://bugs.python.org/issue9741
11
12"""Generate binary message catalog from textual translation description.
13
14This program converts a textual Uniforum-style message catalog (.po file) into
15a binary GNU catalog (.mo file). This is essentially the same function as the
16GNU msgfmt program, however, it is a simpler implementation.
17
18Usage: msgfmt.py [OPTIONS] filename.po
19
20Options:
21 -o file
22 --output-file=file
23 Specify the output file to write to. If omitted, output will go to a
24 file named filename.mo (based off the input file name).
25
26 -h
27 --help
28 Print this message and exit.
29
30 -V
31 --version
32 Display version information and exit.
33"""
34
35import sys
36import os
37import getopt
38import struct
39import array
40
41__version__ = "1.1"
42
43MESSAGES = {}
44
45
46def usage(code, msg=''):
47 print >> sys.stderr, __doc__
48 if msg:
49 print >> sys.stderr, msg
50 sys.exit(code)
51
52
53def add(id, str, fuzzy):
54 "Add a non-fuzzy translation to the dictionary."
55 global MESSAGES
56 if not fuzzy and str:
57 MESSAGES[id] = str
58
59
60def generate():
61 "Return the generated output."
62 global MESSAGES
63 keys = MESSAGES.keys()
64 # the keys are sorted in the .mo file
65 keys.sort()
66 offsets = []
67 ids = strs = ''
68 for id in keys:
69 # For each string, we need size and file offset. Each string is NUL
70 # terminated; the NUL does not count into the size.
71 offsets.append((len(ids), len(id), len(strs), len(MESSAGES[id])))
72 ids += id + '\0'
73 strs += MESSAGES[id] + '\0'
74 output = ''
75 # The header is 7 32-bit unsigned integers. We don't use hash tables, so
76 # the keys start right after the index tables.
77 # translated string.
78 keystart = 7*4+16*len(keys)
79 # and the values start after the keys
80 valuestart = keystart + len(ids)
81 koffsets = []
82 voffsets = []
83 # The string table first has the list of keys, then the list of values.
84 # Each entry has first the size of the string, then the file offset.
85 for o1, l1, o2, l2 in offsets:
86 koffsets += [l1, o1+keystart]
87 voffsets += [l2, o2+valuestart]
88 offsets = koffsets + voffsets
89 output = struct.pack("Iiiiiii",
90 0x950412deL, # Magic
91 0, # Version
92 len(keys), # # of entries
93 7*4, # start of key index
94 7*4+len(keys)*8, # start of value index
95 0, 0) # size and offset of hash table
96 output += array.array("i", offsets).tostring()
97 output += ids
98 output += strs
99 return output
100
101
102def make(filename, outfile):
103 # Fix http://bugs.python.org/issue9741
104 global MESSAGES
105 MESSAGES.clear()
106 ID = 1
107 STR = 2
108
109 # Compute .mo name from .po name and arguments
110 if filename.endswith('.po'):
111 infile = filename
112 else:
113 infile = filename + '.po'
114 if outfile is None:
115 outfile = os.path.splitext(infile)[0] + '.mo'
116
117 try:
118 lines = open(infile).readlines()
119 except IOError, msg:
120 print >> sys.stderr, msg
121 sys.exit(1)
122
123 section = None
124 fuzzy = 0
125
126 # Parse the catalog
127 lno = 0
128 for l in lines:
129 lno += 1
130 # If we get a comment line after a msgstr, this is a new entry
131 if l[0] == '#' and section == STR:
132 add(msgid, msgstr, fuzzy)
133 section = None
134 fuzzy = 0
135 # Record a fuzzy mark
136 if l[:2] == '#,' and 'fuzzy' in l:
137 fuzzy = 1
138 # Skip comments
139 if l[0] == '#':
140 continue
141 # Now we are in a msgid section, output previous section
142 if l.startswith('msgid'):
143 if section == STR:
144 add(msgid, msgstr, fuzzy)
145 section = ID
146 l = l[5:]
147 msgid = msgstr = ''
148 # Now we are in a msgstr section
149 elif l.startswith('msgstr'):
150 section = STR
151 l = l[6:]
152 # Skip empty lines
153 l = l.strip()
154 if not l:
155 continue
156 # XXX: Does this always follow Python escape semantics?
157 l = eval(l)
158 if section == ID:
159 msgid += l
160 elif section == STR:
161 msgstr += l
162 else:
163 print >> sys.stderr, 'Syntax error on %s:%d' % (infile, lno), \
164 'before:'
165 print >> sys.stderr, l
166 sys.exit(1)
167 # Add last entry
168 if section == STR:
169 add(msgid, msgstr, fuzzy)
170
171 # Compute output
172 output = generate()
173
174 try:
175 open(outfile,"wb").write(output)
176 except IOError,msg:
177 print >> sys.stderr, msg
178
179
180def main():
181 try:
182 opts, args = getopt.getopt(sys.argv[1:], 'hVo:',
183 ['help', 'version', 'output-file='])
184 except getopt.error, msg:
185 usage(1, msg)
186
187 outfile = None
188 # parse options
189 for opt, arg in opts:
190 if opt in ('-h', '--help'):
191 usage(0)
192 elif opt in ('-V', '--version'):
193 print >> sys.stderr, "msgfmt.py", __version__
194 sys.exit(0)
195 elif opt in ('-o', '--output-file'):
196 outfile = arg
197 # do it
198 if not args:
199 print >> sys.stderr, 'No input file given'
200 print >> sys.stderr, "Try `msgfmt --help' for more information."
201 return
202
203 for filename in args:
204 make(filename, outfile)
205
206
207if __name__ == '__main__':
208 main()