Merge lp:~songofacandy/bzr/i18n-msgfmt into lp:bzr

Proposed by John A Meinel on 2011-05-16
Status: Merged
Approved by: Martin Pool on 2011-05-18
Approved revision: 5839
Merged at revision: 5891
Proposed branch: lp:~songofacandy/bzr/i18n-msgfmt
Merge into: lp:bzr
Prerequisite: lp:~songofacandy/bzr/i18n-msgextract
Diff against target: 363 lines (+326/-2)
3 files modified
setup.py (+11/-2)
tools/build_mo.py (+107/-0)
tools/msgfmt.py (+208/-0)
To merge this branch: bzr merge lp:~songofacandy/bzr/i18n-msgfmt
Reviewer Review Type Date Requested Status
Martin Pool 2011-05-16 Needs Information on 2011-05-17
Review via email: mp+61082@code.launchpad.net

This proposal supersedes a proposal from 2011-05-15.

Commit Message

add build_mo command to setup.py

Description of the Change

This merge proposal adds build_mo command to setup.py
This command builds bzrlib/locale/<LANG>/LC_MESSAGES/*.mo files from po/*.po.

The directory format draft is described at http://wiki.bazaar.canonical.com/DraftSpecs/I18nSupport.

(resubmitted to set the Prerequisite branch)

To post a comment you must log in.
INADA Naoki (songofacandy) wrote : Posted in a previous version of this proposal

Uh, this mp is based on lp:~songofacandy/bzr/i18n-msgextract.
Extra diffs may be removed after that branch is merged.

John A Meinel (jameinel) wrote : Posted in a previous version of this proposal

Do you know that you can set a Prerequisite branch as part of submitting the merge proposal? Changes in the prerequisite branch won't be shown in the diff. I think you can resubmit this and add the appropriate prereq. I'll try.

INADA Naoki (songofacandy) wrote : Posted in a previous version of this proposal

Oh, I didn't know that. Thank you.

On Mon, May 16, 2011 at 6:48 PM, John A Meinel <email address hidden> wrote:
> Do you know that you can set a Prerequisite branch as part of submitting the merge proposal? Changes in the prerequisite branch won't be shown in the diff. I think you can resubmit this and add the appropriate prereq. I'll try.
> --
> https://code.launchpad.net/~songofacandy/bzr/i18n-msgfmt/+merge/61030
> You are the owner of lp:~songofacandy/bzr/i18n-msgfmt.
>

--
INADA Naoki  <email address hidden>

lp:~songofacandy/bzr/i18n-msgfmt updated on 2011-05-17
5838. By INADA Naoki on 2011-05-17

merge from lp:bzr

Martin Pool (mbp) wrote :

Sorry if I missed this in the list thread, but: can you tell me why we're adding a build_mo command, rather than using the external ones?

+ if 'en' in self.lang:
+ if find_executable('msginit') is None:
+ log.warn("GNU gettext msginit utility not found!")
+ log.warn("Skip creating English PO file.")
+ else:
+ log.info('Creating English PO file...')
+ pot = (self.prj_name or 'messages') + '.pot'
+ if self.prj_name:
+ en_po = '%s-en.po' % self.prj_name
+ else:
+ en_po = 'en.po'
+ self.spawn(['msginit',
+ '--no-translator',
+ '-l', 'en',
+ '-i', os.path.join(self.source_dir, pot),
+ '-o', os.path.join(self.source_dir, en_po),
+ ])
+

Could you explain this a bit more to me?

Thanks

review: Needs Information
INADA Naoki (songofacandy) wrote :

On Wed, May 18, 2011 at 12:01 AM, Martin Pool <email address hidden> wrote:
> Review: Needs Information
> Sorry if I missed this in the list thread, but: can you tell me why we're adding a build_mo command, rather than using the external ones?

setup.py command or Makefile command is needed with other tool, too.
So the problem is "msgfmt.py or external msgfmt command."

The advantage of msgfmt.py is that users don't need msgfmt command.
Users building bzr from source must have Python, of course.
But msgfmt tool is extra dependency for not only windows users, but also
Ubuntu users. Ubuntu doesn't installs msgfmt default and not all users
have root. Installing msgfmt is pain for non root users. Please imagine
"pip install --user bzr".

On the other hand, I don't imagine advantage of external msgfmt tool.

>
> +        if 'en' in self.lang:
> +            if find_executable('msginit') is None:
> +                log.warn("GNU gettext msginit utility not found!")
> +                log.warn("Skip creating English PO file.")
> +            else:
> +                log.info('Creating English PO file...')
> +                pot = (self.prj_name or 'messages') + '.pot'
> +                if self.prj_name:
> +                    en_po = '%s-en.po' % self.prj_name
> +                else:
> +                    en_po = 'en.po'
> +                self.spawn(['msginit',
> +                    '--no-translator',
> +                    '-l', 'en',
> +                    '-i', os.path.join(self.source_dir, pot),
> +                    '-o', os.path.join(self.source_dir, en_po),
> +                    ])
> +
>
> Could you explain this a bit more to me?

I've copied it from bzr-explorer and have didn't mind...
SImply, adding empty english po file is enough solution to remove
msginit dependency.

>
> Thanks
> --
> https://code.launchpad.net/~songofacandy/bzr/i18n-msgfmt/+merge/61082
> You are the owner of lp:~songofacandy/bzr/i18n-msgfmt.
>

--
INADA Naoki  <email address hidden>

lp:~songofacandy/bzr/i18n-msgfmt updated on 2011-05-17
5839. By INADA Naoki on 2011-05-17

Remove msginit dependency.

Martin Pool (mbp) wrote :

On 17 May 2011 18:49, INADA Naoki <email address hidden> wrote:
> On Wed, May 18, 2011 at 12:01 AM, Martin Pool <email address hidden> wrote:
>> Review: Needs Information
>> Sorry if I missed this in the list thread, but: can you tell me why we're adding a build_mo command, rather than using the external ones?
>
> setup.py command or Makefile command is needed with other tool, too.
> So the problem is "msgfmt.py or external msgfmt command."
>
> The advantage of msgfmt.py is that users don't need msgfmt command.
> Users building bzr from source must have Python, of course.
> But msgfmt tool is extra dependency for not only windows users, but also
> Ubuntu users. Ubuntu doesn't installs msgfmt default and not all users
> have root. Installing msgfmt is pain for non root users. Please imagine
> "pip install --user bzr".
>
> On the other hand, I don't imagine advantage of external msgfmt tool.

ok, that makes sense. thanks!

Martin Pool (mbp) wrote :

sent to pqm by email

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'setup.py'
2--- setup.py 2011-05-11 16:35:34 +0000
3+++ setup.py 2011-05-17 17:51:36 +0000
4@@ -69,7 +69,8 @@
5 'tests/ssl_certs/ca.crt',
6 'tests/ssl_certs/server_without_pass.key',
7 'tests/ssl_certs/server_with_pass.key',
8- 'tests/ssl_certs/server.crt'
9+ 'tests/ssl_certs/server.crt',
10+ 'locale/*/LC_MESSAGES/*.mo',
11 ]},
12 }
13
14@@ -152,6 +153,10 @@
15 Generate bzr.1.
16 """
17
18+ sub_commands = build.sub_commands + [
19+ ('build_mo', lambda _: True),
20+ ]
21+
22 def run(self):
23 build.run(self)
24
25@@ -163,8 +168,12 @@
26 ## Setup
27 ########################
28
29+from tools.build_mo import build_mo
30+
31 command_classes = {'install_scripts': my_install_scripts,
32- 'build': bzr_build}
33+ 'build': bzr_build,
34+ 'build_mo': build_mo,
35+ }
36 from distutils import log
37 from distutils.errors import CCompilerError, DistutilsPlatformError
38 from distutils.extension import Extension
39
40=== added file 'tools/build_mo.py'
41--- tools/build_mo.py 1970-01-01 00:00:00 +0000
42+++ tools/build_mo.py 2011-05-17 17:51:36 +0000
43@@ -0,0 +1,107 @@
44+# -*- coding: utf-8 -*-
45+#
46+# Copyright (C) 2007 Lukáš Lalinský <lalinsky@gmail.com>
47+# Copyright (C) 2007,2009 Alexander Belchenko <bialix@ukr.net>
48+# Copyright 2011 Canonical Ltd.
49+#
50+# This program is free software; you can redistribute it and/or
51+# modify it under the terms of the GNU General Public License
52+# as published by the Free Software Foundation; either version 2
53+# of the License, or (at your option) any later version.
54+#
55+# This program is distributed in the hope that it will be useful,
56+# but WITHOUT ANY WARRANTY; without even the implied warranty of
57+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
58+# GNU General Public License for more details.
59+#
60+# You should have received a copy of the GNU General Public License
61+# along with this program; if not, write to the Free Software
62+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
63+
64+# This code is bring from bzr-explorer and modified for bzr.
65+
66+"""build_mo command for setup.py"""
67+
68+from distutils import log
69+from distutils.command.build import build
70+from distutils.core import Command
71+from distutils.dep_util import newer
72+from distutils.spawn import find_executable
73+import os
74+import re
75+
76+import msgfmt
77+
78+class build_mo(Command):
79+ """Subcommand of build command: build_mo"""
80+
81+ description = 'compile po files to mo files'
82+
83+ # List of options:
84+ # - long name,
85+ # - short name (None if no short name),
86+ # - help string.
87+ user_options = [('build-dir=', 'd', 'Directory to build locale files'),
88+ ('output-base=', 'o', 'mo-files base name'),
89+ ('source-dir=', None, 'Directory with sources po files'),
90+ ('force', 'f', 'Force creation of mo files'),
91+ ('lang=', None, 'Comma-separated list of languages '
92+ 'to process'),
93+ ]
94+
95+ boolean_options = ['force']
96+
97+ def initialize_options(self):
98+ self.build_dir = None
99+ self.output_base = None
100+ self.source_dir = None
101+ self.force = None
102+ self.lang = None
103+
104+ def finalize_options(self):
105+ self.set_undefined_options('build', ('force', 'force'))
106+ self.prj_name = self.distribution.get_name()
107+ if self.build_dir is None:
108+ self.build_dir = 'bzrlib/locale'
109+ if not self.output_base:
110+ self.output_base = self.prj_name or 'messages'
111+ if self.source_dir is None:
112+ self.source_dir = 'po'
113+ if self.lang is None:
114+ if self.prj_name:
115+ re_po = re.compile(r'^(?:%s-)?([a-zA-Z_]+)\.po$' % self.prj_name)
116+ else:
117+ re_po = re.compile(r'^([a-zA-Z_]+)\.po$')
118+ self.lang = []
119+ for i in os.listdir(self.source_dir):
120+ mo = re_po.match(i)
121+ if mo:
122+ self.lang.append(mo.group(1))
123+ else:
124+ self.lang = [i.strip() for i in self.lang.split(',') if i.strip()]
125+
126+ def run(self):
127+ """Run msgfmt for each language"""
128+ if not self.lang:
129+ return
130+
131+ basename = self.output_base
132+ if not basename.endswith('.mo'):
133+ basename += '.mo'
134+
135+ po_prefix = ''
136+ if self.prj_name:
137+ po_prefix = self.prj_name + '-'
138+ for lang in self.lang:
139+ po = os.path.join('po', lang + '.po')
140+ if not os.path.isfile(po):
141+ po = os.path.join('po', po_prefix + lang + '.po')
142+ dir_ = os.path.join(self.build_dir, lang, 'LC_MESSAGES')
143+ self.mkpath(dir_)
144+ mo = os.path.join(dir_, basename)
145+ if self.force or newer(po, mo):
146+ log.info('Compile: %s -> %s' % (po, mo))
147+ msgfmt.make(po, mo)
148+
149+
150+build.sub_commands.insert(0, ('build_mo', None))
151
152=== added file 'tools/msgfmt.py'
153--- tools/msgfmt.py 1970-01-01 00:00:00 +0000
154+++ tools/msgfmt.py 2011-05-17 17:51:36 +0000
155@@ -0,0 +1,208 @@
156+#! /usr/bin/env python
157+# -*- coding: utf-8 -*-
158+# Written by Martin v. Lowis <loewis@informatik.hu-berlin.de>
159+
160+# This script is copied from Python/Tools/i18n/msgfmt.py
161+# It licensed under PSF license which compatible with GPL.
162+#
163+# ChangeLog
164+# * Convert encoding from iso-8859-1 to utf-8
165+# * Fix http://bugs.python.org/issue9741
166+
167+"""Generate binary message catalog from textual translation description.
168+
169+This program converts a textual Uniforum-style message catalog (.po file) into
170+a binary GNU catalog (.mo file). This is essentially the same function as the
171+GNU msgfmt program, however, it is a simpler implementation.
172+
173+Usage: msgfmt.py [OPTIONS] filename.po
174+
175+Options:
176+ -o file
177+ --output-file=file
178+ Specify the output file to write to. If omitted, output will go to a
179+ file named filename.mo (based off the input file name).
180+
181+ -h
182+ --help
183+ Print this message and exit.
184+
185+ -V
186+ --version
187+ Display version information and exit.
188+"""
189+
190+import sys
191+import os
192+import getopt
193+import struct
194+import array
195+
196+__version__ = "1.1"
197+
198+MESSAGES = {}
199+
200+
201+def usage(code, msg=''):
202+ print >> sys.stderr, __doc__
203+ if msg:
204+ print >> sys.stderr, msg
205+ sys.exit(code)
206+
207+
208+def add(id, str, fuzzy):
209+ "Add a non-fuzzy translation to the dictionary."
210+ global MESSAGES
211+ if not fuzzy and str:
212+ MESSAGES[id] = str
213+
214+
215+def generate():
216+ "Return the generated output."
217+ global MESSAGES
218+ keys = MESSAGES.keys()
219+ # the keys are sorted in the .mo file
220+ keys.sort()
221+ offsets = []
222+ ids = strs = ''
223+ for id in keys:
224+ # For each string, we need size and file offset. Each string is NUL
225+ # terminated; the NUL does not count into the size.
226+ offsets.append((len(ids), len(id), len(strs), len(MESSAGES[id])))
227+ ids += id + '\0'
228+ strs += MESSAGES[id] + '\0'
229+ output = ''
230+ # The header is 7 32-bit unsigned integers. We don't use hash tables, so
231+ # the keys start right after the index tables.
232+ # translated string.
233+ keystart = 7*4+16*len(keys)
234+ # and the values start after the keys
235+ valuestart = keystart + len(ids)
236+ koffsets = []
237+ voffsets = []
238+ # The string table first has the list of keys, then the list of values.
239+ # Each entry has first the size of the string, then the file offset.
240+ for o1, l1, o2, l2 in offsets:
241+ koffsets += [l1, o1+keystart]
242+ voffsets += [l2, o2+valuestart]
243+ offsets = koffsets + voffsets
244+ output = struct.pack("Iiiiiii",
245+ 0x950412deL, # Magic
246+ 0, # Version
247+ len(keys), # # of entries
248+ 7*4, # start of key index
249+ 7*4+len(keys)*8, # start of value index
250+ 0, 0) # size and offset of hash table
251+ output += array.array("i", offsets).tostring()
252+ output += ids
253+ output += strs
254+ return output
255+
256+
257+def make(filename, outfile):
258+ # Fix http://bugs.python.org/issue9741
259+ global MESSAGES
260+ MESSAGES.clear()
261+ ID = 1
262+ STR = 2
263+
264+ # Compute .mo name from .po name and arguments
265+ if filename.endswith('.po'):
266+ infile = filename
267+ else:
268+ infile = filename + '.po'
269+ if outfile is None:
270+ outfile = os.path.splitext(infile)[0] + '.mo'
271+
272+ try:
273+ lines = open(infile).readlines()
274+ except IOError, msg:
275+ print >> sys.stderr, msg
276+ sys.exit(1)
277+
278+ section = None
279+ fuzzy = 0
280+
281+ # Parse the catalog
282+ lno = 0
283+ for l in lines:
284+ lno += 1
285+ # If we get a comment line after a msgstr, this is a new entry
286+ if l[0] == '#' and section == STR:
287+ add(msgid, msgstr, fuzzy)
288+ section = None
289+ fuzzy = 0
290+ # Record a fuzzy mark
291+ if l[:2] == '#,' and 'fuzzy' in l:
292+ fuzzy = 1
293+ # Skip comments
294+ if l[0] == '#':
295+ continue
296+ # Now we are in a msgid section, output previous section
297+ if l.startswith('msgid'):
298+ if section == STR:
299+ add(msgid, msgstr, fuzzy)
300+ section = ID
301+ l = l[5:]
302+ msgid = msgstr = ''
303+ # Now we are in a msgstr section
304+ elif l.startswith('msgstr'):
305+ section = STR
306+ l = l[6:]
307+ # Skip empty lines
308+ l = l.strip()
309+ if not l:
310+ continue
311+ # XXX: Does this always follow Python escape semantics?
312+ l = eval(l)
313+ if section == ID:
314+ msgid += l
315+ elif section == STR:
316+ msgstr += l
317+ else:
318+ print >> sys.stderr, 'Syntax error on %s:%d' % (infile, lno), \
319+ 'before:'
320+ print >> sys.stderr, l
321+ sys.exit(1)
322+ # Add last entry
323+ if section == STR:
324+ add(msgid, msgstr, fuzzy)
325+
326+ # Compute output
327+ output = generate()
328+
329+ try:
330+ open(outfile,"wb").write(output)
331+ except IOError,msg:
332+ print >> sys.stderr, msg
333+
334+
335+def main():
336+ try:
337+ opts, args = getopt.getopt(sys.argv[1:], 'hVo:',
338+ ['help', 'version', 'output-file='])
339+ except getopt.error, msg:
340+ usage(1, msg)
341+
342+ outfile = None
343+ # parse options
344+ for opt, arg in opts:
345+ if opt in ('-h', '--help'):
346+ usage(0)
347+ elif opt in ('-V', '--version'):
348+ print >> sys.stderr, "msgfmt.py", __version__
349+ sys.exit(0)
350+ elif opt in ('-o', '--output-file'):
351+ outfile = arg
352+ # do it
353+ if not args:
354+ print >> sys.stderr, 'No input file given'
355+ print >> sys.stderr, "Try `msgfmt --help' for more information."
356+ return
357+
358+ for filename in args:
359+ make(filename, outfile)
360+
361+
362+if __name__ == '__main__':
363+ main()