Merge lp:~songofacandy/bzr/i18n-msgextract into lp:bzr

Proposed by INADA Naoki on 2011-05-05
Status: Merged
Approved by: Vincent Ladeuil on 2011-05-16
Approved revision: 5852
Merged at revision: 5873
Proposed branch: lp:~songofacandy/bzr/i18n-msgextract
Merge into: lp:bzr
Diff against target: 463 lines (+419/-0)
5 files modified
Makefile (+20/-0)
bzrlib/builtins.py (+10/-0)
bzrlib/export_pot.py (+241/-0)
bzrlib/tests/__init__.py (+1/-0)
bzrlib/tests/test_export_pot.py (+147/-0)
To merge this branch: bzr merge lp:~songofacandy/bzr/i18n-msgextract
Reviewer Review Type Date Requested Status
Vincent Ladeuil 2011-05-05 Approve on 2011-05-13
Review via email: mp+60033@code.launchpad.net

Commit message

Extract messages for translation.

Description of the change

I18n part2 - extract messages for translate.

To post a comment you must log in.
Alexander Belchenko (bialix) wrote :

INADA Naoki пишет:
> INADA Naoki has proposed merging lp:~songofacandy/bzr/i18n-msgextract into lp:bzr.
>
> Requested reviews:
> bzr-core (bzr-core)
>
> For more details, see:
> https://code.launchpad.net/~songofacandy/bzr/i18n-msgextract/+merge/60033
>
> I18n part2 - extract messages for translate.

I think it will be very cool if we can automatically extract help
strings for command-line options, so we can avoid wrapping them into
N_() function calls. I fear that N_() will slowdown the CLI. Of course
somebody have to check the time, maybe I'm totally wrong here.

--
All the dude wanted was his rug back

INADA Naoki (songofacandy) wrote :

> I think it will be very cool if we can automatically extract help
> strings for command-line options, so we can avoid wrapping them into
> N_() function calls. I fear that N_() will slowdown the CLI. Of course
> somebody have to check the time, maybe I'm totally wrong here.

Done.

Alexander Belchenko (bialix) wrote :

>
> > I think it will be very cool if we can automatically extract help
> > strings for command-line options, so we can avoid wrapping them into
> > N_() function calls. I fear that N_() will slowdown the CLI. Of course
> > somebody have to check the time, maybe I'm totally wrong here.
>
> Done.

Cool!

Alexander Belchenko (bialix) wrote :

INADA Naoki пишет:
> INADA Naoki has proposed merging lp:~songofacandy/bzr/i18n-msgextract into lp:bzr.
>
> Requested reviews:
> bzr-core (bzr-core)
> Related bugs:
> Bug #83941 in Bazaar: "Bzr doesn't speak my tongue"
> https://bugs.launchpad.net/bzr/+bug/83941
>
> For more details, see:
> https://code.launchpad.net/~songofacandy/bzr/i18n-msgextract/+merge/60033
>
> I18n part2 - extract messages for translate.

Why did you put your new python script into filename without .py
extension? That makes windows life a bit harder. Is it really important
to omit .py extension here?

--
All the dude wanted was his rug back

INADA Naoki (songofacandy) wrote :

> Why did you put your new python script into filename without .py
> extension? That makes windows life a bit harder. Is it really important
> to omit .py extension here?

No reason. It is because it is originally from mercurial and named hggettext.
I've added .py extension.

John A Meinel (jameinel) wrote :

That isn't all of the .py files. Is there a reason not to do:

$(PYTHON) tools/bzrgettext bzrlib/*.py \
   bzrlib/*/*.py \

I guess you would get tests/ which isn't really necessary but you might want to be getting stuff like transport/ and ui/

Also, for plugins, you really need:
 bzrlib/plugins/*/*.py

Because cmd_* functions will be in the plugins themselves, not in the containing dir.

If we are doing xargs anyway, why not use a "find" command.

INADA Naoki (songofacandy) wrote :

Fixed.

I've found some modules raises error on importing because of lacking some modules.
Modules providing commands should be able to import while exporting command helps.
Otherwise, help messages of the commands are not translated.

I don't think this is a big problem because updating pot is as special as making
package. Only few developers and buildbots do this task.
Is this an acceptable limitation?

Vincent Ladeuil (vila) wrote :

> Fixed.
>
> I've found some modules raises error on importing because of lacking some
> modules.

Can you elaborate on that ?

> Modules providing commands should be able to import while exporting command
> helps.
> Otherwise, help messages of the commands are not translated.

Right, this could be addressed by using the command registry probably.

>
> I don't think this is a big problem because updating pot is as special as
> making
> package. Only few developers and buildbots do this task.
> Is this an acceptable limitation?

Yes and no, it depends ;)

More importantly, I think we need... tests ;)

Especially for the case you're encountering right now but more
generally so we can clearly define which texts should be
translated and ensure that a test will fail if whatever code
modification happens to escape the collection stage.

22 + xgettext --package-name "Bazaar" \

Really ? Is it just some internal id or can it be referenced by
say, launchpad ? If the later is true, we probably want bzr no ?

134 +def importpath(path):

I think we have some better implementation for that in pyutils
(get_name_object ?), this may also be related to the import
errors you're encountering.

I don't want to sound NIH-ish here, but... it seems to me we'll
do a better job (in terms of coverage and precision) by using
bzrlib and all its facilities no ?

Getting all commands, their help and all exceptions while
excluding tests for example sounds like an uphill battle to fight
with find and grep...

Do you get a feeling about how close you're coming to get *all*
the needed strings and can you categorize them (my intuition
being that there are ways to get them all reliably and precisely
by leveraging some existing APIs... and if we can do that,
designing for tests for them should be straightforward).

review: Needs Information
INADA Naoki (songofacandy) wrote :
Download full text (3.8 KiB)

> > Fixed.
> >
> > I've found some modules raises error on importing because of lacking some
> > modules.
>
> Can you elaborate on that ?

Before ignoreing "bzrlib/doc", there are four import errors.
Can't import 'bzrlib/doc_generate/builders/texinfo.py': No module named sphinx
Can't import 'bzrlib/doc_generate/writers/texinfo.py': No module named docutils
Can't import 'bzrlib/util/simplemapi.py': name 'windll' is not defined
Can't import 'bzrlib/transport/ftp/_gssapi.py': Unable to import library "kerberos": No module named kerberos

I think bzrlib/doc and bzrlib/doc_generate should be excluded so I added filter to Makefile.
Both of bzrlib.util.simplemapi and bzrlib.transport.ftp._gssapi doesn't provide
any commands.

> > Modules providing commands should be able to import while exporting command
> > helps.
> > Otherwise, help messages of the commands are not translated.
>
> Right, this could be addressed by using the command registry probably.

Okey, I'll try command registry based approach.

> > I don't think this is a big problem because updating pot is as special as
> > making
> > package. Only few developers and buildbots do this task.
> > Is this an acceptable limitation?
>
> Yes and no, it depends ;)
>
> More importantly, I think we need... tests ;)
>
> Especially for the case you're encountering right now but more
> generally so we can clearly define which texts should be
> translated and ensure that a test will fail if whatever code
> modification happens to escape the collection stage.
>

How can I write tests for tools like bzrgettext?

One idea I have is making "xx" language. This language is translated
automatically from pot. For example, "Display status" translated to
"xx{{Display status}}".
With this language, test of command can be ensure that messages that
should be translated is really exported and translated.

> 22 + xgettext --package-name "Bazaar" \
>
> Really ? Is it just some internal id or can it be referenced by
> say, launchpad ? If the later is true, we probably want bzr no ?

I don't know that this name affects anyware.
Should I try to make test project for it and play with Rosetta?
Or can someone in bzr-core team help me by making test branch including
po/bzr.pot generated by my branch?

BTW, there are no reason to rename it to 'bzr'. I'll do it.

> 134 +def importpath(path):
>
> I think we have some better implementation for that in pyutils
> (get_name_object ?), this may also be related to the import
> errors you're encountering.
>
> I don't want to sound NIH-ish here, but... it seems to me we'll
> do a better job (in terms of coverage and precision) by using
> bzrlib and all its facilities no ?
>
> Getting all commands, their help and all exceptions while
> excluding tests for example sounds like an uphill battle to fight
> with find and grep...

As I said above, I'll try command registry based approach.
With this approach, there are no need to manualy importing like
"importpath()".

> Do you get a feeling about how close you're coming to get *all*
> the needed strings and can you categorize them (my intuition
> being that there are ways to get them all reliably and precisely
> by leverag...

Read more...

Vincent Ladeuil (vila) wrote :

> It is far from *all* because there are no N_() and gettext() now.
> I can't imagine how far.
>

Fair enough.

> But I think message categories that is most important to users is:
> * help topics
> * command help
> * Error messages giving a important hint to user. (ex. NotWorkingTree)

Right, so I think a better approach would be to indeed focus on
these ones and neglect (to begin with) the other strings. This
will also reduce the amount of needed translations while we
bootstrap the whole process.

Defining a *bzr* command to do that (may be hidden as this is not
targeted at regular bzr users) will make things simpler.

>
> About help topics, I'll implement scanning of it to bzrgettext.py.
> But we need to decide how handle text files under bzrlib/help_topics/en/.

bzrlib.help is probably the way to go then since it already
provides the registry for that and several other utilities who
already extract the relevant texts (and may in fact be the site
where the localization should occur).

>
> About command help, I don't know how to prepare command registry that
> includes all bundled plugins but does not include 3rd party plugins.

Good point. But again, defining a proper bzr command will mean we
can use BZR_PLUGIN_PATH and friends to control which plugins are
loaded/seen by bzrlib. If you want to focus on bzr and its
bundled plugins, using 'BZR_PLUGINS_PATH=-site' will do just
that.

>
> About error messages, looking on bzrlib.errors is enough.

I think so too, filtering the bzrlib.errors module for classes
inheriting from BzrError should do.

This would miss some errors defined locally in some modules but
we could ignore them to start with and file bugs for them later.

Finally, an important point is the order of the strings in the
generated file.

Relying on 'find' means that different users running the script
are likely to get different file orders (hence vastly different
file content IIUC), whereas relying on internal registries means
that we can force the lexicographical order on command names or
help topics to ensure a consistent order.

This will also means that we can rely on the existing tests for
coverage and focus on tests specific to the problem we're
addressing here (string order or avoiding duplicate strings (if
that matters, I don't know) for example).

231 + with bzrlib.initialize():

2.6 specific you evil :)

Alexander Belchenko (bialix) wrote :

INADA Naoki пишет:
>
> But I think message categories that is most important to users is:
> * help topics
> * command help
> * Error messages giving a important hint to user. (ex. NotWorkingTree)
>
> About help topics, I'll implement scanning of it to bzrgettext.py.
> But we need to decide how handle text files under bzrlib/help_topics/en/.

I think the initial idea was to put translations into corresponding text
files under bzrlib/help_topics/$LANG_CODE/

Is it not what you expect? What's you intent here? Extract and convert
these files into PO files? I suppose big and long help topics should be
translated not paragraph by paragraph, but as the whole text. Therefore
I'd prefer to extract all other help topics from python modules and put
them into plain text files. That won't work very good with Launchpad,
but this is another problem.

I think we should discuss this more broadly in bzr ML, because in the
past some people have concerns about txt files vs py files. IIRC Aaron
didn't like the fact we're using txt files.

But! In my opinion such text files is much better for translators.

PO files are good only for relatively short strings. They're very bad re
context.

--
All the dude wanted was his rug back

INADA Naoki (songofacandy) wrote :

> > But I think message categories that is most important to users is:
> > * help topics
> > * command help
> > * Error messages giving a important hint to user. (ex. NotWorkingTree)
>
> Right, so I think a better approach would be to indeed focus on
> these ones and neglect (to begin with) the other strings. This
> will also reduce the amount of needed translations while we
> bootstrap the whole process.
>
> Defining a *bzr* command to do that (may be hidden as this is not
> targeted at regular bzr users) will make things simpler.
>
...
> > About command help, I don't know how to prepare command registry that
> > includes all bundled plugins but does not include 3rd party plugins.
>
> Good point. But again, defining a proper bzr command will mean we
> can use BZR_PLUGIN_PATH and friends to control which plugins are
> loaded/seen by bzrlib. If you want to focus on bzr and its
> bundled plugins, using 'BZR_PLUGINS_PATH=-site' will do just
> that.
>

OK. I'll do it.

I want to bzr command is usable to 3rd party plugins too, if possible.

But we should focus on starting translation on Launchpad to give
translaters enough time before bzr-2.4. So BZR_PLUGINS_PATH=-site is
good starting point.

> > About help topics, I'll implement scanning of it to bzrgettext.py.
> > But we need to decide how handle text files under bzrlib/help_topics/en/.
>
> bzrlib.help is probably the way to go then since it already
> provides the registry for that and several other utilities who
> already extract the relevant texts (and may in fact be the site
> where the localization should occur).

As Alexander mentioned, I said about help_topics/en/*.txt
should be translated with Rosetta or not.
I'll post to ML about this.

> >
> > About error messages, looking on bzrlib.errors is enough.
>
> I think so too, filtering the bzrlib.errors module for classes
> inheriting from BzrError should do.
>
> This would miss some errors defined locally in some modules but
> we could ignore them to start with and file bugs for them later.
>

bzrerrors() function in bzrgettext.py filters Error classes with
internal_error=True.

> Finally, an important point is the order of the strings in the
> generated file.
>
> Relying on 'find' means that different users running the script
> are likely to get different file orders (hence vastly different
> file content IIUC), whereas relying on internal registries means
> that we can force the lexicographical order on command names or
> help topics to ensure a consistent order.
>
> This will also means that we can rely on the existing tests for
> coverage and focus on tests specific to the problem we're
> addressing here (string order or avoiding duplicate strings (if
> that matters, I don't know) for example).

OK, I'll mind to order.

>
> 231 + with bzrlib.initialize():
>
> 2.6 specific you evil :)

It can run under 2.5, because there is "from __future__ import with_statement".
But when bzrgettext.py is bzr command, it should be able to run under
2.4 until dropping 2.4 support is decided.

John A Meinel (jameinel) wrote :

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/06/2011 08:45 AM, INADA Naoki wrote:
> Fixed.
>
> I've found some modules raises error on importing because of lacking some modules.
> Modules providing commands should be able to import while exporting command helps.
> Otherwise, help messages of the commands are not translated.
>
> I don't think this is a big problem because updating pot is as special as making
> package. Only few developers and buildbots do this task.
> Is this an acceptable limitation?
>

Modules ideally should not have dependency issues. ie You should be able
to run "python -c 'import bzrlib.XXX'" for any XXX and not have it fail.
 If there are circular dependencies or missing dependencies, we should
fix this.

I had thought you were just grepping the code, but if you are importing
it, then we should probably go via a different path.

 from bzrlib import initialize, plugin, commands
 initialize().__enter__()
 plugin.load_plugins()

 # do stuff with the registered commands. Which will now include plugin-
 # provided commands.

And then whatever else you need. Once you've done initialize +
load_plugins, I think everything should be in a happy enough state for
importing.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk3EBOsACgkQJdeBCYSNAAP7BwCeM5NWtQP/Cs03o8ssj/gtafcg
D/8AnAydV84uiweq+VyhiZrImKorY0m7
=0Io/
-----END PGP SIGNATURE-----

Martin Packman (gz) wrote :

Would using _ast be an alternative to needing to import the modules and worry about dependencies?

+ if fmt:
+ poentry('bzrlib/erros.py', inspect.findsource(klass)[1], fmt)

Typo, "errors.py"?

INADA Naoki (songofacandy) wrote :

Yey! I did it!
This is my first bzr command!

INADA Naoki (songofacandy) wrote :

Then, what kind of tests should I write?

Vincent Ladeuil (vila) wrote :

>>>>> INADA Naoki <email address hidden> writes:

    > Yey! I did it!

Hurray !

    > This is my first bzr command!

Congrats :)

INADA Naoki (songofacandy) wrote :

Then, last things I should do is writing tests more?
The way of extracting messages is OK?

Vincent Ladeuil (vila) wrote :

I think that's good enough to start experimenting, thanks !

I'm still unclear about the overall approach to extract the messages but I don't want to block on that either. We can still revisit it later once we all know a bit more about i18n and l10n ;)

Unless the second reviewer objects, I think we should land this and see how it goes for the whole translation process.

review: Approve
INADA Naoki (songofacandy) wrote :

I want to enable extracting help topics.
It is implemented in lp:~songofacandy/bzr/i18n
Should it be separeted merge request?

Vincent Ladeuil (vila) wrote :

> Should it be separeted merge request?

Yes.

*This* mp has been approved and only needs a second review.

Start a new one for the new work so we can land *this* mp (when approved) while discussing the other one.

Vincent Ladeuil (vila) wrote :

It seems there are too many possible ways to address the message extraction and that we won't progress without making at least some experiments.

This proposal implements one way to extract the messages and it's worth trying.

So I'll land it and if we change our mind later, well fix the fallouts if needed.

Vincent Ladeuil (vila) wrote :

@Naoki: Also, it would be nice if you collect the knowledge you acquire in some document we can refer to *in* the code base (doc/developers/i18n.txt comes to mind)

Don't worry about it being perfect or even finalized, just put what you know today there and we'll refine it as we get a better collective understanding.

Vincent Ladeuil (vila) wrote :

sent to pqm by email

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'Makefile'
2--- Makefile 2011-04-18 00:44:08 +0000
3+++ Makefile 2011-05-12 12:21:48 +0000
4@@ -420,6 +420,26 @@
5 $(PYTHON) tools/win32/ostools.py remove dist
6
7
8+# i18n targets
9+
10+.PHONY: update-pot po/bzr.pot
11+update-pot: po/bzr.pot
12+
13+TRANSLATABLE_PYFILES:=$(shell find bzrlib -name '*.py' \
14+ | grep -v 'bzrlib/tests/' \
15+ | grep -v 'bzrlib/doc' \
16+ )
17+
18+po/bzr.pot: $(PYFILES) $(DOCFILES)
19+ $(PYTHON) ./bzr export-pot > po/bzr.pot
20+ echo $(TRANSLATABLE_PYFILES) | xargs \
21+ xgettext --package-name "bzr" \
22+ --msgid-bugs-address "<bazaar@canonical.com>" \
23+ --copyright-holder "Canonical" \
24+ --from-code ISO-8859-1 --join --sort-by-file --add-comments=i18n: \
25+ -d bzr -p po -o bzr.pot
26+
27+
28 ### Packaging Targets ###
29
30 .PHONY: dist check-dist-tarball
31
32=== modified file 'bzrlib/builtins.py'
33--- bzrlib/builtins.py 2011-05-03 13:53:46 +0000
34+++ bzrlib/builtins.py 2011-05-12 12:21:48 +0000
35@@ -6154,6 +6154,16 @@
36 self.outf.write('%s %s\n' % (path, location))
37
38
39+class cmd_export_pot(Command):
40+ __doc__ = """Export command helps and error messages in po format."""
41+
42+ hidden = True
43+
44+ def run(self):
45+ from bzrlib.export_pot import export_pot
46+ export_pot(self.outf)
47+
48+
49 def _register_lazy_builtins():
50 # register lazy builtins from other modules; called at startup and should
51 # be only called once.
52
53=== added file 'bzrlib/export_pot.py'
54--- bzrlib/export_pot.py 1970-01-01 00:00:00 +0000
55+++ bzrlib/export_pot.py 2011-05-12 12:21:48 +0000
56@@ -0,0 +1,241 @@
57+# Copyright (C) 2011 Canonical Ltd
58+#
59+# This program is free software; you can redistribute it and/or modify
60+# it under the terms of the GNU General Public License as published by
61+# the Free Software Foundation; either version 2 of the License, or
62+# (at your option) any later version.
63+#
64+# This program is distributed in the hope that it will be useful,
65+# but WITHOUT ANY WARRANTY; without even the implied warranty of
66+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
67+# GNU General Public License for more details.
68+#
69+# You should have received a copy of the GNU General Public License
70+# along with this program; if not, write to the Free Software
71+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
72+
73+# The normalize function is taken from pygettext which is distributed
74+# with Python under the Python License, which is GPL compatible.
75+
76+"""Extract docstrings from Bazaar commands.
77+"""
78+
79+import inspect
80+import os
81+
82+from bzrlib import (
83+ commands as _mod_commands,
84+ errors,
85+ help_topics,
86+ plugin,
87+ )
88+from bzrlib.trace import (
89+ mutter,
90+ note,
91+ )
92+
93+
94+def _escape(s):
95+ s = (s.replace('\\', '\\\\')
96+ .replace('\n', '\\n')
97+ .replace('\r', '\\r')
98+ .replace('\t', '\\t')
99+ .replace('"', '\\"')
100+ )
101+ return s
102+
103+def _normalize(s):
104+ # This converts the various Python string types into a format that
105+ # is appropriate for .po files, namely much closer to C style.
106+ lines = s.split('\n')
107+ if len(lines) == 1:
108+ s = '"' + _escape(s) + '"'
109+ else:
110+ if not lines[-1]:
111+ del lines[-1]
112+ lines[-1] = lines[-1] + '\n'
113+ lines = map(_escape, lines)
114+ lineterm = '\\n"\n"'
115+ s = '""\n"' + lineterm.join(lines) + '"'
116+ return s
117+
118+
119+_FOUND_MSGID = None # set by entry function.
120+
121+def _poentry(outf, path, lineno, s, comment=None):
122+ if s in _FOUND_MSGID:
123+ return
124+ _FOUND_MSGID.add(s)
125+ if comment is None:
126+ comment = ''
127+ else:
128+ comment = "# %s\n" % comment
129+ mutter("Exporting msg %r at line %d in %r", s[:20], lineno, path)
130+ print >>outf, ('#: %s:%d\n' % (path, lineno) +
131+ comment+
132+ 'msgid %s\n' % _normalize(s) +
133+ 'msgstr ""\n')
134+
135+def _poentry_per_paragraph(outf, path, lineno, msgid):
136+ # TODO: How to split long help?
137+ paragraphs = msgid.split('\n\n')
138+ for p in paragraphs:
139+ _poentry(outf, path, lineno, p)
140+ lineno += p.count('\n') + 2
141+
142+_LAST_CACHE = _LAST_CACHED_SRC = None
143+
144+def _offsets_of_literal(src):
145+ global _LAST_CACHE, _LAST_CACHED_SRC
146+ if src == _LAST_CACHED_SRC:
147+ return _LAST_CACHE.copy()
148+
149+ import ast
150+ root = ast.parse(src)
151+ offsets = {}
152+ for node in ast.walk(root):
153+ if not isinstance(node, ast.Str):
154+ continue
155+ offsets[node.s] = node.lineno - node.s.count('\n')
156+
157+ _LAST_CACHED_SRC = src
158+ _LAST_CACHE = offsets.copy()
159+ return offsets
160+
161+def _standard_options(outf):
162+ from bzrlib.option import Option
163+ src = inspect.findsource(Option)[0]
164+ src = ''.join(src)
165+ path = 'bzrlib/option.py'
166+ offsets = _offsets_of_literal(src)
167+
168+ for name in sorted(Option.OPTIONS.keys()):
169+ opt = Option.OPTIONS[name]
170+ if getattr(opt, 'hidden', False):
171+ continue
172+ if getattr(opt, 'title', None):
173+ lineno = offsets.get(opt.title, 9999)
174+ if lineno == 9999:
175+ note("%r is not found in bzrlib/option.py" % opt.title)
176+ _poentry(outf, path, lineno, opt.title,
177+ 'title of %r option' % name)
178+ if getattr(opt, 'help', None):
179+ lineno = offsets.get(opt.help, 9999)
180+ if lineno == 9999:
181+ note("%r is not found in bzrlib/option.py" % opt.help)
182+ _poentry(outf, path, lineno, opt.help,
183+ 'help of %r option' % name)
184+
185+def _command_options(outf, path, cmd):
186+ src, default_lineno = inspect.findsource(cmd.__class__)
187+ offsets = _offsets_of_literal(''.join(src))
188+ for opt in cmd.takes_options:
189+ if isinstance(opt, str):
190+ continue
191+ if getattr(opt, 'hidden', False):
192+ continue
193+ name = opt.name
194+ if getattr(opt, 'title', None):
195+ lineno = offsets.get(opt.title, default_lineno)
196+ _poentry(outf, path, lineno, opt.title,
197+ 'title of %r option of %r command' % (name, cmd.name()))
198+ if getattr(opt, 'help', None):
199+ lineno = offsets.get(opt.help, default_lineno)
200+ _poentry(outf, path, lineno, opt.help,
201+ 'help of %r option of %r command' % (name, cmd.name()))
202+
203+
204+def _write_command_help(outf, cmd_name, cmd):
205+ path = inspect.getfile(cmd.__class__)
206+ if path.endswith('.pyc'):
207+ path = path[:-1]
208+ path = os.path.relpath(path)
209+ src, lineno = inspect.findsource(cmd.__class__)
210+ offsets = _offsets_of_literal(''.join(src))
211+ lineno = offsets[cmd.__doc__]
212+ doc = inspect.getdoc(cmd)
213+
214+ _poentry_per_paragraph(outf, path, lineno, doc)
215+ _command_options(outf, path, cmd)
216+
217+def _command_helps(outf):
218+ """Extract docstrings from path.
219+
220+ This respects the Bazaar cmdtable/table convention and will
221+ only extract docstrings from functions mentioned in these tables.
222+ """
223+ from glob import glob
224+
225+ # builtin commands
226+ for cmd_name in _mod_commands.builtin_command_names():
227+ command = _mod_commands.get_cmd_object(cmd_name, False)
228+ if command.hidden:
229+ continue
230+ note("Exporting messages from builtin command: %s", cmd_name)
231+ _write_command_help(outf, cmd_name, command)
232+
233+ plugin_path = plugin.get_core_plugin_path()
234+ core_plugins = glob(plugin_path + '/*/__init__.py')
235+ core_plugins = [os.path.basename(os.path.dirname(p))
236+ for p in core_plugins]
237+ # core plugins
238+ for cmd_name in _mod_commands.plugin_command_names():
239+ command = _mod_commands.get_cmd_object(cmd_name, False)
240+ if command.hidden:
241+ continue
242+ if command.plugin_name() not in core_plugins:
243+ # skip non-core plugins
244+ # TODO: Support extracting from third party plugins.
245+ continue
246+ note("Exporting messages from plugin command: %s in %s",
247+ cmd_name, command.plugin_name())
248+ _write_command_help(outf, cmd_name, command)
249+
250+
251+def _error_messages(outf):
252+ """Extract fmt string from bzrlib.errors."""
253+ path = errors.__file__
254+ if path.endswith('.pyc'):
255+ path = path[:-1]
256+ offsets = _offsets_of_literal(open(path).read())
257+
258+ base_klass = errors.BzrError
259+ for name in dir(errors):
260+ klass = getattr(errors, name)
261+ if not inspect.isclass(klass):
262+ continue
263+ if not issubclass(klass, base_klass):
264+ continue
265+ if klass is base_klass:
266+ continue
267+ if klass.internal_error:
268+ continue
269+ fmt = getattr(klass, "_fmt", None)
270+ if fmt:
271+ note("Exporting message from error: %s", name)
272+ _poentry(outf, 'bzrlib/errors.py',
273+ offsets.get(fmt, 9999), fmt)
274+
275+def _help_topics(outf):
276+ topic_registry = help_topics.topic_registry
277+ for key in topic_registry.keys():
278+ doc = topic_registry.get(key)
279+ if isinstance(doc, str):
280+ _poentry_per_paragraph(
281+ outf,
282+ 'dummy/help_topics/'+key+'/detail.txt',
283+ 1, doc)
284+
285+ summary = topic_registry.get_summary(key)
286+ if summary is not None:
287+ _poentry(outf, 'dummy/help_topics/'+key+'/summary.txt',
288+ 1, summary)
289+
290+def export_pot(outf):
291+ global _FOUND_MSGID
292+ _FOUND_MSGID = set()
293+ _standard_options(outf)
294+ _command_helps(outf)
295+ _error_messages(outf)
296+ # disable exporting help topics until we decide how to translate it.
297+ #_help_topics(outf)
298
299=== modified file 'bzrlib/tests/__init__.py'
300--- bzrlib/tests/__init__.py 2011-05-10 07:46:15 +0000
301+++ bzrlib/tests/__init__.py 2011-05-12 12:21:48 +0000
302@@ -3783,6 +3783,7 @@
303 'bzrlib.tests.test_eol_filters',
304 'bzrlib.tests.test_errors',
305 'bzrlib.tests.test_export',
306+ 'bzrlib.tests.test_export_pot',
307 'bzrlib.tests.test_extract',
308 'bzrlib.tests.test_fetch',
309 'bzrlib.tests.test_fixtures',
310
311=== added file 'bzrlib/tests/test_export_pot.py'
312--- bzrlib/tests/test_export_pot.py 1970-01-01 00:00:00 +0000
313+++ bzrlib/tests/test_export_pot.py 2011-05-12 12:21:48 +0000
314@@ -0,0 +1,147 @@
315+# Copyright (C) 2011 Canonical Ltd
316+#
317+# This program is free software; you can redistribute it and/or modify
318+# it under the terms of the GNU General Public License as published by
319+# the Free Software Foundation; either version 2 of the License, or
320+# (at your option) any later version.
321+#
322+# This program is distributed in the hope that it will be useful,
323+# but WITHOUT ANY WARRANTY; without even the implied warranty of
324+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
325+# GNU General Public License for more details.
326+#
327+# You should have received a copy of the GNU General Public License
328+# along with this program; if not, write to the Free Software
329+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
330+
331+from cStringIO import StringIO
332+import textwrap
333+
334+from bzrlib import (
335+ export_pot,
336+ tests,
337+ )
338+
339+class TestEscape(tests.TestCase):
340+
341+ def test_simple_escape(self):
342+ self.assertEqual(
343+ export_pot._escape('foobar'),
344+ 'foobar')
345+
346+ s = '''foo\nbar\r\tbaz\\"spam"'''
347+ e = '''foo\\nbar\\r\\tbaz\\\\\\"spam\\"'''
348+ self.assertEqual(export_pot._escape(s), e)
349+
350+ def test_complex_escape(self):
351+ s = '''\\r \\\n'''
352+ e = '''\\\\r \\\\\\n'''
353+ self.assertEqual(export_pot._escape(s), e)
354+
355+
356+class TestNormalize(tests.TestCase):
357+
358+ def test_single_line(self):
359+ s = 'foobar'
360+ e = '"foobar"'
361+ self.assertEqual(export_pot._normalize(s), e)
362+
363+ s = 'foo"bar'
364+ e = '"foo\\"bar"'
365+ self.assertEqual(export_pot._normalize(s), e)
366+
367+ def test_multi_lines(self):
368+ s = 'foo\nbar\n'
369+ e = '""\n"foo\\n"\n"bar\\n"'
370+ self.assertEqual(export_pot._normalize(s), e)
371+
372+ s = '\nfoo\nbar\n'
373+ e = ('""\n'
374+ '"\\n"\n'
375+ '"foo\\n"\n'
376+ '"bar\\n"')
377+ self.assertEqual(export_pot._normalize(s), e)
378+
379+
380+class PoEntryTestCase(tests.TestCase):
381+
382+ def setUp(self):
383+ self.overrideAttr(export_pot, '_FOUND_MSGID', set())
384+ self._outf = StringIO()
385+ super(PoEntryTestCase, self).setUp()
386+
387+ def check_output(self, expected):
388+ self.assertEqual(
389+ self._outf.getvalue(),
390+ textwrap.dedent(expected)
391+ )
392+
393+class TestPoEntry(PoEntryTestCase):
394+
395+ def test_simple(self):
396+ export_pot._poentry(self._outf, 'dummy', 1, "spam")
397+ export_pot._poentry(self._outf, 'dummy', 2, "ham", 'EGG')
398+ self.check_output('''\
399+ #: dummy:1
400+ msgid "spam"
401+ msgstr ""
402+
403+ #: dummy:2
404+ # EGG
405+ msgid "ham"
406+ msgstr ""
407+
408+ ''')
409+
410+ def test_duplicate(self):
411+ export_pot._poentry(self._outf, 'dummy', 1, "spam")
412+ # This should be ignored.
413+ export_pot._poentry(self._outf, 'dummy', 2, "spam", 'EGG')
414+
415+ self.check_output('''\
416+ #: dummy:1
417+ msgid "spam"
418+ msgstr ""\n
419+ ''')
420+
421+
422+class TestPoentryPerPergraph(PoEntryTestCase):
423+
424+ def test_single(self):
425+ export_pot._poentry_per_paragraph(
426+ self._outf,
427+ 'dummy',
428+ 10,
429+ '''foo\nbar\nbaz\n'''
430+ )
431+ self.check_output('''\
432+ #: dummy:10
433+ msgid ""
434+ "foo\\n"
435+ "bar\\n"
436+ "baz\\n"
437+ msgstr ""\n
438+ ''')
439+
440+ def test_multi(self):
441+ export_pot._poentry_per_paragraph(
442+ self._outf,
443+ 'dummy',
444+ 10,
445+ '''spam\nham\negg\n\nSPAM\nHAM\nEGG\n'''
446+ )
447+ self.check_output('''\
448+ #: dummy:10
449+ msgid ""
450+ "spam\\n"
451+ "ham\\n"
452+ "egg"
453+ msgstr ""
454+
455+ #: dummy:14
456+ msgid ""
457+ "SPAM\\n"
458+ "HAM\\n"
459+ "EGG\\n"
460+ msgstr ""\n
461+ ''')
462
463=== added directory 'po'