Merge lp:~songofacandy/bzr/i18n-msgextract into lp:bzr

Proposed by INADA Naoki on 2011-05-05
Status: Merged
Approved by: Vincent Ladeuil on 2011-05-16
Approved revision: 5852
Merged at revision: 5873
Proposed branch: lp:~songofacandy/bzr/i18n-msgextract
Merge into: lp:bzr
Diff against target: 463 lines (+419/-0)
5 files modified
Makefile (+20/-0)
bzrlib/builtins.py (+10/-0)
bzrlib/export_pot.py (+241/-0)
bzrlib/tests/__init__.py (+1/-0)
bzrlib/tests/test_export_pot.py (+147/-0)
To merge this branch: bzr merge lp:~songofacandy/bzr/i18n-msgextract
Reviewer Review Type Date Requested Status
Vincent Ladeuil 2011-05-05 Approve on 2011-05-13
Review via email: mp+60033@code.launchpad.net

Commit message

Extract messages for translation.

Description of the change

I18n part2 - extract messages for translate.

To post a comment you must log in.
Alexander Belchenko (bialix) wrote :

INADA Naoki пишет:
> INADA Naoki has proposed merging lp:~songofacandy/bzr/i18n-msgextract into lp:bzr.
>
> Requested reviews:
> bzr-core (bzr-core)
>
> For more details, see:
> https://code.launchpad.net/~songofacandy/bzr/i18n-msgextract/+merge/60033
>
> I18n part2 - extract messages for translate.

I think it will be very cool if we can automatically extract help
strings for command-line options, so we can avoid wrapping them into
N_() function calls. I fear that N_() will slowdown the CLI. Of course
somebody have to check the time, maybe I'm totally wrong here.

--
All the dude wanted was his rug back

INADA Naoki (songofacandy) wrote :

> I think it will be very cool if we can automatically extract help
> strings for command-line options, so we can avoid wrapping them into
> N_() function calls. I fear that N_() will slowdown the CLI. Of course
> somebody have to check the time, maybe I'm totally wrong here.

Done.

Alexander Belchenko (bialix) wrote :

>
> > I think it will be very cool if we can automatically extract help
> > strings for command-line options, so we can avoid wrapping them into
> > N_() function calls. I fear that N_() will slowdown the CLI. Of course
> > somebody have to check the time, maybe I'm totally wrong here.
>
> Done.

Cool!

Alexander Belchenko (bialix) wrote :

INADA Naoki пишет:
> INADA Naoki has proposed merging lp:~songofacandy/bzr/i18n-msgextract into lp:bzr.
>
> Requested reviews:
> bzr-core (bzr-core)
> Related bugs:
> Bug #83941 in Bazaar: "Bzr doesn't speak my tongue"
> https://bugs.launchpad.net/bzr/+bug/83941
>
> For more details, see:
> https://code.launchpad.net/~songofacandy/bzr/i18n-msgextract/+merge/60033
>
> I18n part2 - extract messages for translate.

Why did you put your new python script into filename without .py
extension? That makes windows life a bit harder. Is it really important
to omit .py extension here?

--
All the dude wanted was his rug back

INADA Naoki (songofacandy) wrote :

> Why did you put your new python script into filename without .py
> extension? That makes windows life a bit harder. Is it really important
> to omit .py extension here?

No reason. It is because it is originally from mercurial and named hggettext.
I've added .py extension.

John A Meinel (jameinel) wrote :

That isn't all of the .py files. Is there a reason not to do:

$(PYTHON) tools/bzrgettext bzrlib/*.py \
   bzrlib/*/*.py \

I guess you would get tests/ which isn't really necessary but you might want to be getting stuff like transport/ and ui/

Also, for plugins, you really need:
 bzrlib/plugins/*/*.py

Because cmd_* functions will be in the plugins themselves, not in the containing dir.

If we are doing xargs anyway, why not use a "find" command.

INADA Naoki (songofacandy) wrote :

Fixed.

I've found some modules raises error on importing because of lacking some modules.
Modules providing commands should be able to import while exporting command helps.
Otherwise, help messages of the commands are not translated.

I don't think this is a big problem because updating pot is as special as making
package. Only few developers and buildbots do this task.
Is this an acceptable limitation?

Vincent Ladeuil (vila) wrote :

> Fixed.
>
> I've found some modules raises error on importing because of lacking some
> modules.

Can you elaborate on that ?

> Modules providing commands should be able to import while exporting command
> helps.
> Otherwise, help messages of the commands are not translated.

Right, this could be addressed by using the command registry probably.

>
> I don't think this is a big problem because updating pot is as special as
> making
> package. Only few developers and buildbots do this task.
> Is this an acceptable limitation?

Yes and no, it depends ;)

More importantly, I think we need... tests ;)

Especially for the case you're encountering right now but more
generally so we can clearly define which texts should be
translated and ensure that a test will fail if whatever code
modification happens to escape the collection stage.

22 + xgettext --package-name "Bazaar" \

Really ? Is it just some internal id or can it be referenced by
say, launchpad ? If the later is true, we probably want bzr no ?

134 +def importpath(path):

I think we have some better implementation for that in pyutils
(get_name_object ?), this may also be related to the import
errors you're encountering.

I don't want to sound NIH-ish here, but... it seems to me we'll
do a better job (in terms of coverage and precision) by using
bzrlib and all its facilities no ?

Getting all commands, their help and all exceptions while
excluding tests for example sounds like an uphill battle to fight
with find and grep...

Do you get a feeling about how close you're coming to get *all*
the needed strings and can you categorize them (my intuition
being that there are ways to get them all reliably and precisely
by leveraging some existing APIs... and if we can do that,
designing for tests for them should be straightforward).

review: Needs Information
INADA Naoki (songofacandy) wrote :
Download full text (3.8 KiB)

> > Fixed.
> >
> > I've found some modules raises error on importing because of lacking some
> > modules.
>
> Can you elaborate on that ?

Before ignoreing "bzrlib/doc", there are four import errors.
Can't import 'bzrlib/doc_generate/builders/texinfo.py': No module named sphinx
Can't import 'bzrlib/doc_generate/writers/texinfo.py': No module named docutils
Can't import 'bzrlib/util/simplemapi.py': name 'windll' is not defined
Can't import 'bzrlib/transport/ftp/_gssapi.py': Unable to import library "kerberos": No module named kerberos

I think bzrlib/doc and bzrlib/doc_generate should be excluded so I added filter to Makefile.
Both of bzrlib.util.simplemapi and bzrlib.transport.ftp._gssapi doesn't provide
any commands.

> > Modules providing commands should be able to import while exporting command
> > helps.
> > Otherwise, help messages of the commands are not translated.
>
> Right, this could be addressed by using the command registry probably.

Okey, I'll try command registry based approach.

> > I don't think this is a big problem because updating pot is as special as
> > making
> > package. Only few developers and buildbots do this task.
> > Is this an acceptable limitation?
>
> Yes and no, it depends ;)
>
> More importantly, I think we need... tests ;)
>
> Especially for the case you're encountering right now but more
> generally so we can clearly define which texts should be
> translated and ensure that a test will fail if whatever code
> modification happens to escape the collection stage.
>

How can I write tests for tools like bzrgettext?

One idea I have is making "xx" language. This language is translated
automatically from pot. For example, "Display status" translated to
"xx{{Display status}}".
With this language, test of command can be ensure that messages that
should be translated is really exported and translated.

> 22 + xgettext --package-name "Bazaar" \
>
> Really ? Is it just some internal id or can it be referenced by
> say, launchpad ? If the later is true, we probably want bzr no ?

I don't know that this name affects anyware.
Should I try to make test project for it and play with Rosetta?
Or can someone in bzr-core team help me by making test branch including
po/bzr.pot generated by my branch?

BTW, there are no reason to rename it to 'bzr'. I'll do it.

> 134 +def importpath(path):
>
> I think we have some better implementation for that in pyutils
> (get_name_object ?), this may also be related to the import
> errors you're encountering.
>
> I don't want to sound NIH-ish here, but... it seems to me we'll
> do a better job (in terms of coverage and precision) by using
> bzrlib and all its facilities no ?
>
> Getting all commands, their help and all exceptions while
> excluding tests for example sounds like an uphill battle to fight
> with find and grep...

As I said above, I'll try command registry based approach.
With this approach, there are no need to manualy importing like
"importpath()".

> Do you get a feeling about how close you're coming to get *all*
> the needed strings and can you categorize them (my intuition
> being that there are ways to get them all reliably and precisely
> by leverag...

Read more...

Vincent Ladeuil (vila) wrote :

> It is far from *all* because there are no N_() and gettext() now.
> I can't imagine how far.
>

Fair enough.

> But I think message categories that is most important to users is:
> * help topics
> * command help
> * Error messages giving a important hint to user. (ex. NotWorkingTree)

Right, so I think a better approach would be to indeed focus on
these ones and neglect (to begin with) the other strings. This
will also reduce the amount of needed translations while we
bootstrap the whole process.

Defining a *bzr* command to do that (may be hidden as this is not
targeted at regular bzr users) will make things simpler.

>
> About help topics, I'll implement scanning of it to bzrgettext.py.
> But we need to decide how handle text files under bzrlib/help_topics/en/.

bzrlib.help is probably the way to go then since it already
provides the registry for that and several other utilities who
already extract the relevant texts (and may in fact be the site
where the localization should occur).

>
> About command help, I don't know how to prepare command registry that
> includes all bundled plugins but does not include 3rd party plugins.

Good point. But again, defining a proper bzr command will mean we
can use BZR_PLUGIN_PATH and friends to control which plugins are
loaded/seen by bzrlib. If you want to focus on bzr and its
bundled plugins, using 'BZR_PLUGINS_PATH=-site' will do just
that.

>
> About error messages, looking on bzrlib.errors is enough.

I think so too, filtering the bzrlib.errors module for classes
inheriting from BzrError should do.

This would miss some errors defined locally in some modules but
we could ignore them to start with and file bugs for them later.

Finally, an important point is the order of the strings in the
generated file.

Relying on 'find' means that different users running the script
are likely to get different file orders (hence vastly different
file content IIUC), whereas relying on internal registries means
that we can force the lexicographical order on command names or
help topics to ensure a consistent order.

This will also means that we can rely on the existing tests for
coverage and focus on tests specific to the problem we're
addressing here (string order or avoiding duplicate strings (if
that matters, I don't know) for example).

231 + with bzrlib.initialize():

2.6 specific you evil :)

Alexander Belchenko (bialix) wrote :

INADA Naoki пишет:
>
> But I think message categories that is most important to users is:
> * help topics
> * command help
> * Error messages giving a important hint to user. (ex. NotWorkingTree)
>
> About help topics, I'll implement scanning of it to bzrgettext.py.
> But we need to decide how handle text files under bzrlib/help_topics/en/.

I think the initial idea was to put translations into corresponding text
files under bzrlib/help_topics/$LANG_CODE/

Is it not what you expect? What's you intent here? Extract and convert
these files into PO files? I suppose big and long help topics should be
translated not paragraph by paragraph, but as the whole text. Therefore
I'd prefer to extract all other help topics from python modules and put
them into plain text files. That won't work very good with Launchpad,
but this is another problem.

I think we should discuss this more broadly in bzr ML, because in the
past some people have concerns about txt files vs py files. IIRC Aaron
didn't like the fact we're using txt files.

But! In my opinion such text files is much better for translators.

PO files are good only for relatively short strings. They're very bad re
context.

--
All the dude wanted was his rug back

INADA Naoki (songofacandy) wrote :

> > But I think message categories that is most important to users is:
> > * help topics
> > * command help
> > * Error messages giving a important hint to user. (ex. NotWorkingTree)
>
> Right, so I think a better approach would be to indeed focus on
> these ones and neglect (to begin with) the other strings. This
> will also reduce the amount of needed translations while we
> bootstrap the whole process.
>
> Defining a *bzr* command to do that (may be hidden as this is not
> targeted at regular bzr users) will make things simpler.
>
...
> > About command help, I don't know how to prepare command registry that
> > includes all bundled plugins but does not include 3rd party plugins.
>
> Good point. But again, defining a proper bzr command will mean we
> can use BZR_PLUGIN_PATH and friends to control which plugins are
> loaded/seen by bzrlib. If you want to focus on bzr and its
> bundled plugins, using 'BZR_PLUGINS_PATH=-site' will do just
> that.
>

OK. I'll do it.

I want to bzr command is usable to 3rd party plugins too, if possible.

But we should focus on starting translation on Launchpad to give
translaters enough time before bzr-2.4. So BZR_PLUGINS_PATH=-site is
good starting point.

> > About help topics, I'll implement scanning of it to bzrgettext.py.
> > But we need to decide how handle text files under bzrlib/help_topics/en/.
>
> bzrlib.help is probably the way to go then since it already
> provides the registry for that and several other utilities who
> already extract the relevant texts (and may in fact be the site
> where the localization should occur).

As Alexander mentioned, I said about help_topics/en/*.txt
should be translated with Rosetta or not.
I'll post to ML about this.

> >
> > About error messages, looking on bzrlib.errors is enough.
>
> I think so too, filtering the bzrlib.errors module for classes
> inheriting from BzrError should do.
>
> This would miss some errors defined locally in some modules but
> we could ignore them to start with and file bugs for them later.
>

bzrerrors() function in bzrgettext.py filters Error classes with
internal_error=True.

> Finally, an important point is the order of the strings in the
> generated file.
>
> Relying on 'find' means that different users running the script
> are likely to get different file orders (hence vastly different
> file content IIUC), whereas relying on internal registries means
> that we can force the lexicographical order on command names or
> help topics to ensure a consistent order.
>
> This will also means that we can rely on the existing tests for
> coverage and focus on tests specific to the problem we're
> addressing here (string order or avoiding duplicate strings (if
> that matters, I don't know) for example).

OK, I'll mind to order.

>
> 231 + with bzrlib.initialize():
>
> 2.6 specific you evil :)

It can run under 2.5, because there is "from __future__ import with_statement".
But when bzrgettext.py is bzr command, it should be able to run under
2.4 until dropping 2.4 support is decided.

John A Meinel (jameinel) wrote :

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/06/2011 08:45 AM, INADA Naoki wrote:
> Fixed.
>
> I've found some modules raises error on importing because of lacking some modules.
> Modules providing commands should be able to import while exporting command helps.
> Otherwise, help messages of the commands are not translated.
>
> I don't think this is a big problem because updating pot is as special as making
> package. Only few developers and buildbots do this task.
> Is this an acceptable limitation?
>

Modules ideally should not have dependency issues. ie You should be able
to run "python -c 'import bzrlib.XXX'" for any XXX and not have it fail.
 If there are circular dependencies or missing dependencies, we should
fix this.

I had thought you were just grepping the code, but if you are importing
it, then we should probably go via a different path.

 from bzrlib import initialize, plugin, commands
 initialize().__enter__()
 plugin.load_plugins()

 # do stuff with the registered commands. Which will now include plugin-
 # provided commands.

And then whatever else you need. Once you've done initialize +
load_plugins, I think everything should be in a happy enough state for
importing.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk3EBOsACgkQJdeBCYSNAAP7BwCeM5NWtQP/Cs03o8ssj/gtafcg
D/8AnAydV84uiweq+VyhiZrImKorY0m7
=0Io/
-----END PGP SIGNATURE-----

Martin Packman (gz) wrote :

Would using _ast be an alternative to needing to import the modules and worry about dependencies?

+ if fmt:
+ poentry('bzrlib/erros.py', inspect.findsource(klass)[1], fmt)

Typo, "errors.py"?

INADA Naoki (songofacandy) wrote :

Yey! I did it!
This is my first bzr command!

INADA Naoki (songofacandy) wrote :

Then, what kind of tests should I write?

Vincent Ladeuil (vila) wrote :

>>>>> INADA Naoki <email address hidden> writes:

    > Yey! I did it!

Hurray !

    > This is my first bzr command!

Congrats :)

INADA Naoki (songofacandy) wrote :

Then, last things I should do is writing tests more?
The way of extracting messages is OK?

Vincent Ladeuil (vila) wrote :

I think that's good enough to start experimenting, thanks !

I'm still unclear about the overall approach to extract the messages but I don't want to block on that either. We can still revisit it later once we all know a bit more about i18n and l10n ;)

Unless the second reviewer objects, I think we should land this and see how it goes for the whole translation process.

review: Approve
INADA Naoki (songofacandy) wrote :

I want to enable extracting help topics.
It is implemented in lp:~songofacandy/bzr/i18n
Should it be separeted merge request?

Vincent Ladeuil (vila) wrote :

> Should it be separeted merge request?

Yes.

*This* mp has been approved and only needs a second review.

Start a new one for the new work so we can land *this* mp (when approved) while discussing the other one.

Vincent Ladeuil (vila) wrote :

It seems there are too many possible ways to address the message extraction and that we won't progress without making at least some experiments.

This proposal implements one way to extract the messages and it's worth trying.

So I'll land it and if we change our mind later, well fix the fallouts if needed.

Vincent Ladeuil (vila) wrote :

@Naoki: Also, it would be nice if you collect the knowledge you acquire in some document we can refer to *in* the code base (doc/developers/i18n.txt comes to mind)

Don't worry about it being perfect or even finalized, just put what you know today there and we'll refine it as we get a better collective understanding.

Vincent Ladeuil (vila) wrote :

sent to pqm by email

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'Makefile'
--- Makefile 2011-04-18 00:44:08 +0000
+++ Makefile 2011-05-12 12:21:48 +0000
@@ -420,6 +420,26 @@
420 $(PYTHON) tools/win32/ostools.py remove dist420 $(PYTHON) tools/win32/ostools.py remove dist
421421
422422
423# i18n targets
424
425.PHONY: update-pot po/bzr.pot
426update-pot: po/bzr.pot
427
428TRANSLATABLE_PYFILES:=$(shell find bzrlib -name '*.py' \
429 | grep -v 'bzrlib/tests/' \
430 | grep -v 'bzrlib/doc' \
431 )
432
433po/bzr.pot: $(PYFILES) $(DOCFILES)
434 $(PYTHON) ./bzr export-pot > po/bzr.pot
435 echo $(TRANSLATABLE_PYFILES) | xargs \
436 xgettext --package-name "bzr" \
437 --msgid-bugs-address "<bazaar@canonical.com>" \
438 --copyright-holder "Canonical" \
439 --from-code ISO-8859-1 --join --sort-by-file --add-comments=i18n: \
440 -d bzr -p po -o bzr.pot
441
442
423### Packaging Targets ###443### Packaging Targets ###
424444
425.PHONY: dist check-dist-tarball445.PHONY: dist check-dist-tarball
426446
=== modified file 'bzrlib/builtins.py'
--- bzrlib/builtins.py 2011-05-03 13:53:46 +0000
+++ bzrlib/builtins.py 2011-05-12 12:21:48 +0000
@@ -6154,6 +6154,16 @@
6154 self.outf.write('%s %s\n' % (path, location))6154 self.outf.write('%s %s\n' % (path, location))
61556155
61566156
6157class cmd_export_pot(Command):
6158 __doc__ = """Export command helps and error messages in po format."""
6159
6160 hidden = True
6161
6162 def run(self):
6163 from bzrlib.export_pot import export_pot
6164 export_pot(self.outf)
6165
6166
6157def _register_lazy_builtins():6167def _register_lazy_builtins():
6158 # register lazy builtins from other modules; called at startup and should6168 # register lazy builtins from other modules; called at startup and should
6159 # be only called once.6169 # be only called once.
61606170
=== added file 'bzrlib/export_pot.py'
--- bzrlib/export_pot.py 1970-01-01 00:00:00 +0000
+++ bzrlib/export_pot.py 2011-05-12 12:21:48 +0000
@@ -0,0 +1,241 @@
1# Copyright (C) 2011 Canonical Ltd
2#
3# This program is free software; you can redistribute it and/or modify
4# it under the terms of the GNU General Public License as published by
5# the Free Software Foundation; either version 2 of the License, or
6# (at your option) any later version.
7#
8# This program is distributed in the hope that it will be useful,
9# but WITHOUT ANY WARRANTY; without even the implied warranty of
10# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
11# GNU General Public License for more details.
12#
13# You should have received a copy of the GNU General Public License
14# along with this program; if not, write to the Free Software
15# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
16
17# The normalize function is taken from pygettext which is distributed
18# with Python under the Python License, which is GPL compatible.
19
20"""Extract docstrings from Bazaar commands.
21"""
22
23import inspect
24import os
25
26from bzrlib import (
27 commands as _mod_commands,
28 errors,
29 help_topics,
30 plugin,
31 )
32from bzrlib.trace import (
33 mutter,
34 note,
35 )
36
37
38def _escape(s):
39 s = (s.replace('\\', '\\\\')
40 .replace('\n', '\\n')
41 .replace('\r', '\\r')
42 .replace('\t', '\\t')
43 .replace('"', '\\"')
44 )
45 return s
46
47def _normalize(s):
48 # This converts the various Python string types into a format that
49 # is appropriate for .po files, namely much closer to C style.
50 lines = s.split('\n')
51 if len(lines) == 1:
52 s = '"' + _escape(s) + '"'
53 else:
54 if not lines[-1]:
55 del lines[-1]
56 lines[-1] = lines[-1] + '\n'
57 lines = map(_escape, lines)
58 lineterm = '\\n"\n"'
59 s = '""\n"' + lineterm.join(lines) + '"'
60 return s
61
62
63_FOUND_MSGID = None # set by entry function.
64
65def _poentry(outf, path, lineno, s, comment=None):
66 if s in _FOUND_MSGID:
67 return
68 _FOUND_MSGID.add(s)
69 if comment is None:
70 comment = ''
71 else:
72 comment = "# %s\n" % comment
73 mutter("Exporting msg %r at line %d in %r", s[:20], lineno, path)
74 print >>outf, ('#: %s:%d\n' % (path, lineno) +
75 comment+
76 'msgid %s\n' % _normalize(s) +
77 'msgstr ""\n')
78
79def _poentry_per_paragraph(outf, path, lineno, msgid):
80 # TODO: How to split long help?
81 paragraphs = msgid.split('\n\n')
82 for p in paragraphs:
83 _poentry(outf, path, lineno, p)
84 lineno += p.count('\n') + 2
85
86_LAST_CACHE = _LAST_CACHED_SRC = None
87
88def _offsets_of_literal(src):
89 global _LAST_CACHE, _LAST_CACHED_SRC
90 if src == _LAST_CACHED_SRC:
91 return _LAST_CACHE.copy()
92
93 import ast
94 root = ast.parse(src)
95 offsets = {}
96 for node in ast.walk(root):
97 if not isinstance(node, ast.Str):
98 continue
99 offsets[node.s] = node.lineno - node.s.count('\n')
100
101 _LAST_CACHED_SRC = src
102 _LAST_CACHE = offsets.copy()
103 return offsets
104
105def _standard_options(outf):
106 from bzrlib.option import Option
107 src = inspect.findsource(Option)[0]
108 src = ''.join(src)
109 path = 'bzrlib/option.py'
110 offsets = _offsets_of_literal(src)
111
112 for name in sorted(Option.OPTIONS.keys()):
113 opt = Option.OPTIONS[name]
114 if getattr(opt, 'hidden', False):
115 continue
116 if getattr(opt, 'title', None):
117 lineno = offsets.get(opt.title, 9999)
118 if lineno == 9999:
119 note("%r is not found in bzrlib/option.py" % opt.title)
120 _poentry(outf, path, lineno, opt.title,
121 'title of %r option' % name)
122 if getattr(opt, 'help', None):
123 lineno = offsets.get(opt.help, 9999)
124 if lineno == 9999:
125 note("%r is not found in bzrlib/option.py" % opt.help)
126 _poentry(outf, path, lineno, opt.help,
127 'help of %r option' % name)
128
129def _command_options(outf, path, cmd):
130 src, default_lineno = inspect.findsource(cmd.__class__)
131 offsets = _offsets_of_literal(''.join(src))
132 for opt in cmd.takes_options:
133 if isinstance(opt, str):
134 continue
135 if getattr(opt, 'hidden', False):
136 continue
137 name = opt.name
138 if getattr(opt, 'title', None):
139 lineno = offsets.get(opt.title, default_lineno)
140 _poentry(outf, path, lineno, opt.title,
141 'title of %r option of %r command' % (name, cmd.name()))
142 if getattr(opt, 'help', None):
143 lineno = offsets.get(opt.help, default_lineno)
144 _poentry(outf, path, lineno, opt.help,
145 'help of %r option of %r command' % (name, cmd.name()))
146
147
148def _write_command_help(outf, cmd_name, cmd):
149 path = inspect.getfile(cmd.__class__)
150 if path.endswith('.pyc'):
151 path = path[:-1]
152 path = os.path.relpath(path)
153 src, lineno = inspect.findsource(cmd.__class__)
154 offsets = _offsets_of_literal(''.join(src))
155 lineno = offsets[cmd.__doc__]
156 doc = inspect.getdoc(cmd)
157
158 _poentry_per_paragraph(outf, path, lineno, doc)
159 _command_options(outf, path, cmd)
160
161def _command_helps(outf):
162 """Extract docstrings from path.
163
164 This respects the Bazaar cmdtable/table convention and will
165 only extract docstrings from functions mentioned in these tables.
166 """
167 from glob import glob
168
169 # builtin commands
170 for cmd_name in _mod_commands.builtin_command_names():
171 command = _mod_commands.get_cmd_object(cmd_name, False)
172 if command.hidden:
173 continue
174 note("Exporting messages from builtin command: %s", cmd_name)
175 _write_command_help(outf, cmd_name, command)
176
177 plugin_path = plugin.get_core_plugin_path()
178 core_plugins = glob(plugin_path + '/*/__init__.py')
179 core_plugins = [os.path.basename(os.path.dirname(p))
180 for p in core_plugins]
181 # core plugins
182 for cmd_name in _mod_commands.plugin_command_names():
183 command = _mod_commands.get_cmd_object(cmd_name, False)
184 if command.hidden:
185 continue
186 if command.plugin_name() not in core_plugins:
187 # skip non-core plugins
188 # TODO: Support extracting from third party plugins.
189 continue
190 note("Exporting messages from plugin command: %s in %s",
191 cmd_name, command.plugin_name())
192 _write_command_help(outf, cmd_name, command)
193
194
195def _error_messages(outf):
196 """Extract fmt string from bzrlib.errors."""
197 path = errors.__file__
198 if path.endswith('.pyc'):
199 path = path[:-1]
200 offsets = _offsets_of_literal(open(path).read())
201
202 base_klass = errors.BzrError
203 for name in dir(errors):
204 klass = getattr(errors, name)
205 if not inspect.isclass(klass):
206 continue
207 if not issubclass(klass, base_klass):
208 continue
209 if klass is base_klass:
210 continue
211 if klass.internal_error:
212 continue
213 fmt = getattr(klass, "_fmt", None)
214 if fmt:
215 note("Exporting message from error: %s", name)
216 _poentry(outf, 'bzrlib/errors.py',
217 offsets.get(fmt, 9999), fmt)
218
219def _help_topics(outf):
220 topic_registry = help_topics.topic_registry
221 for key in topic_registry.keys():
222 doc = topic_registry.get(key)
223 if isinstance(doc, str):
224 _poentry_per_paragraph(
225 outf,
226 'dummy/help_topics/'+key+'/detail.txt',
227 1, doc)
228
229 summary = topic_registry.get_summary(key)
230 if summary is not None:
231 _poentry(outf, 'dummy/help_topics/'+key+'/summary.txt',
232 1, summary)
233
234def export_pot(outf):
235 global _FOUND_MSGID
236 _FOUND_MSGID = set()
237 _standard_options(outf)
238 _command_helps(outf)
239 _error_messages(outf)
240 # disable exporting help topics until we decide how to translate it.
241 #_help_topics(outf)
0242
=== modified file 'bzrlib/tests/__init__.py'
--- bzrlib/tests/__init__.py 2011-05-10 07:46:15 +0000
+++ bzrlib/tests/__init__.py 2011-05-12 12:21:48 +0000
@@ -3783,6 +3783,7 @@
3783 'bzrlib.tests.test_eol_filters',3783 'bzrlib.tests.test_eol_filters',
3784 'bzrlib.tests.test_errors',3784 'bzrlib.tests.test_errors',
3785 'bzrlib.tests.test_export',3785 'bzrlib.tests.test_export',
3786 'bzrlib.tests.test_export_pot',
3786 'bzrlib.tests.test_extract',3787 'bzrlib.tests.test_extract',
3787 'bzrlib.tests.test_fetch',3788 'bzrlib.tests.test_fetch',
3788 'bzrlib.tests.test_fixtures',3789 'bzrlib.tests.test_fixtures',
37893790
=== added file 'bzrlib/tests/test_export_pot.py'
--- bzrlib/tests/test_export_pot.py 1970-01-01 00:00:00 +0000
+++ bzrlib/tests/test_export_pot.py 2011-05-12 12:21:48 +0000
@@ -0,0 +1,147 @@
1# Copyright (C) 2011 Canonical Ltd
2#
3# This program is free software; you can redistribute it and/or modify
4# it under the terms of the GNU General Public License as published by
5# the Free Software Foundation; either version 2 of the License, or
6# (at your option) any later version.
7#
8# This program is distributed in the hope that it will be useful,
9# but WITHOUT ANY WARRANTY; without even the implied warranty of
10# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
11# GNU General Public License for more details.
12#
13# You should have received a copy of the GNU General Public License
14# along with this program; if not, write to the Free Software
15# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
16
17from cStringIO import StringIO
18import textwrap
19
20from bzrlib import (
21 export_pot,
22 tests,
23 )
24
25class TestEscape(tests.TestCase):
26
27 def test_simple_escape(self):
28 self.assertEqual(
29 export_pot._escape('foobar'),
30 'foobar')
31
32 s = '''foo\nbar\r\tbaz\\"spam"'''
33 e = '''foo\\nbar\\r\\tbaz\\\\\\"spam\\"'''
34 self.assertEqual(export_pot._escape(s), e)
35
36 def test_complex_escape(self):
37 s = '''\\r \\\n'''
38 e = '''\\\\r \\\\\\n'''
39 self.assertEqual(export_pot._escape(s), e)
40
41
42class TestNormalize(tests.TestCase):
43
44 def test_single_line(self):
45 s = 'foobar'
46 e = '"foobar"'
47 self.assertEqual(export_pot._normalize(s), e)
48
49 s = 'foo"bar'
50 e = '"foo\\"bar"'
51 self.assertEqual(export_pot._normalize(s), e)
52
53 def test_multi_lines(self):
54 s = 'foo\nbar\n'
55 e = '""\n"foo\\n"\n"bar\\n"'
56 self.assertEqual(export_pot._normalize(s), e)
57
58 s = '\nfoo\nbar\n'
59 e = ('""\n'
60 '"\\n"\n'
61 '"foo\\n"\n'
62 '"bar\\n"')
63 self.assertEqual(export_pot._normalize(s), e)
64
65
66class PoEntryTestCase(tests.TestCase):
67
68 def setUp(self):
69 self.overrideAttr(export_pot, '_FOUND_MSGID', set())
70 self._outf = StringIO()
71 super(PoEntryTestCase, self).setUp()
72
73 def check_output(self, expected):
74 self.assertEqual(
75 self._outf.getvalue(),
76 textwrap.dedent(expected)
77 )
78
79class TestPoEntry(PoEntryTestCase):
80
81 def test_simple(self):
82 export_pot._poentry(self._outf, 'dummy', 1, "spam")
83 export_pot._poentry(self._outf, 'dummy', 2, "ham", 'EGG')
84 self.check_output('''\
85 #: dummy:1
86 msgid "spam"
87 msgstr ""
88
89 #: dummy:2
90 # EGG
91 msgid "ham"
92 msgstr ""
93
94 ''')
95
96 def test_duplicate(self):
97 export_pot._poentry(self._outf, 'dummy', 1, "spam")
98 # This should be ignored.
99 export_pot._poentry(self._outf, 'dummy', 2, "spam", 'EGG')
100
101 self.check_output('''\
102 #: dummy:1
103 msgid "spam"
104 msgstr ""\n
105 ''')
106
107
108class TestPoentryPerPergraph(PoEntryTestCase):
109
110 def test_single(self):
111 export_pot._poentry_per_paragraph(
112 self._outf,
113 'dummy',
114 10,
115 '''foo\nbar\nbaz\n'''
116 )
117 self.check_output('''\
118 #: dummy:10
119 msgid ""
120 "foo\\n"
121 "bar\\n"
122 "baz\\n"
123 msgstr ""\n
124 ''')
125
126 def test_multi(self):
127 export_pot._poentry_per_paragraph(
128 self._outf,
129 'dummy',
130 10,
131 '''spam\nham\negg\n\nSPAM\nHAM\nEGG\n'''
132 )
133 self.check_output('''\
134 #: dummy:10
135 msgid ""
136 "spam\\n"
137 "ham\\n"
138 "egg"
139 msgstr ""
140
141 #: dummy:14
142 msgid ""
143 "SPAM\\n"
144 "HAM\\n"
145 "EGG\\n"
146 msgstr ""\n
147 ''')
0148
=== added directory 'po'