Merge lp:~lovesyao/bzr/windows-utf8 into lp:~bzr/bzr/trunk-old

[Nazo]

[Robert Collins (lifeless)]

I'm not in a position to comment on the correctness of this. I would like to note that it really belongs deeper in the library where our existing encoding support is.

review: Abstain

[John A Meinel (jameinel)]

Something like this seems potentially very nice. However as Robert mentions, there are some changes that need to happen.

1) All of the logic should be moved out of 'bzr' and into something like 'bzrlib/win32tools.py'. It would be nice to have it as an independent class that perhaps wasn't strictly tied to Windows, so we could write unit tests for it that run everywhere.

For example, if you can just subclass with:

class LoggingConsoleWriter(WindowsConsoleWriter):

  def _write(self, text):
    self._log.append(text)

Then you can test that things are getting mapped into UTF-16, etc properly.

+ def writelines(self, text):
+ self.write(text + "\n")
+

^- I'm 95% sure that this is completely wrong. "writelines(self, lines)" is the api, where 'lines' is an iterable of strings, not a simple text string. (I'm guessing you are thinking of the Java api, writeline(), though my Java is weak, I do remember something that would add '\n' for you.)

Also, having unit tests lets us test stuff like _color() is getting called on bad characters, etc.

I think the reason you are seeing UTF-8 weirdness, is that we have defined the 'note()' functionality as writing out to a UTF-8 stream, which goes to stdout. (Which is probably also why you ended up realizing that you needed to overwrite stderr as well as stdout.)

My guess is that replacing sys.stdout and sys.stderr unilaterally is not the correct fix, and instead we should be just hooking into things like 'note()' and making sure it gets formatted correctly.

This can also get hooked into our infrastructure for "Command.encoding_type" which handles stuff like "the bytes of files should be output exactly as read from disk" but "log should show User names in appropriate encoding."

review: Needs Fixing

[Alexander Belchenko (bialix)]

> Something like this seems potentially very nice. However as Robert mentions,
> there are some changes that need to happen.
>
> 1) All of the logic should be moved out of 'bzr' and into something like
> 'bzrlib/win32tools.py'.

You mean 'bzrlib/win32utils.py' actually.

[Martin Pool (mbp)]

beyond jam's comment, it would be nice to be clear on just what bug is causing this.

I guess that it is that sys.stdout.write(u'文字化け') gets mangled, so it's nothing specific to our note() or logging infrastructure. It's conceivable (I'm not sure) that on Windows these streams are irretrievably in an encoding that cannot represent all unicode characters and the best thing really is to use a win32 api that is unicode-clean. In that case perhaps replacing something at a very global level really is best, though I might try to do it more through UIFactory than sys.stdout.

Supporting colors would be interesting but keying off reserved unicode characters to do it is a bit weird.

[Parth Malwankar (parthm)]

>
> Supporting colors would be interesting but keying off reserved unicode
> characters to do it is a bit weird.

It would be good to have bzrlib support color on windows and unix.
As this feature is not present, plugins end up with their own support. E.g. lp:bzrtools[1] and lp:bzr-grep[2] have their own implementation of color (unix only). I know there is a PyPI package (colorama[3] if I remember the code correctly) that supports color on both windows and unix. On Windows, from the developer/user perspective it translates the standard unix escape sequence into the appropriate windows equivalent. Maybe we can just use colorama or use a similar approach. Having bzrlib color support will also allow other commands to log etc. to have color.

Hmm. Maybe this should be a separate wishlist bug.

[1] http://bazaar.launchpad.net/~abentley/bzrtools/bzrtools.dev/annotate/head:/terminal.py
[2] http://bazaar.launchpad.net/~bzr/bzr-grep/trunk/annotate/head:/termcolor.py
[3] http://pypi.python.org/pypi/colorama/0.1.14

Unmerged revisions

3844. By Nazo on 2008-11-20

Fix mojibake of utf_8 character for stderr on Windows Command Prompt

3843. By Nazo on 2008-11-20

cleanup code

3842. By Nazo on 2008-11-19

Fix mojibake of utf_8 character on Windows Command Prompt

=== modified file 'bzr'
--- bzr	2009-08-28 05:11:10 +0000
+++ bzr	2009-08-31 04:38:40 +0000
@@ -53,6 +53,91 @@
     os.unsetenv(REINVOKE)
 
 
+if sys.platform == 'win32':
+    from ctypes import windll, Structure, c_short, c_ushort, byref
+    import codecs
+
+    class COORD(Structure):
+        _fields_ = [
+            ("X", c_short),
+            ("Y", c_short)
+        ]
+
+    class SMALL_RECT(Structure):
+        _fields_ = [
+            ("Left", c_short),
+            ("Top", c_short),
+            ("Right", c_short),
+            ("Bottom", c_short)
+        ]
+
+    class CONSOLE_SCREEN_BUFFER_INFO(Structure):
+        _fields_ = [
+            ("dwSize", COORD),
+            ("dwCursorPosition", COORD),
+            ("wAttributes", c_ushort),
+            ("srWindow", SMALL_RECT),
+            ("dwMaximumWindowSize", COORD)
+        ]
+
+    STD_OUTPUT_HANDLE = -11
+    STD_ERROR_HANDLE = -12
+
+    def consolereplace_errors(exception):
+        if not isinstance(exception, UnicodeDecodeError):
+            raise TypeError("consolereplace() cannot handle %r" % exception)
+        l = [u"\\%x" % ord(exception.object[pos]) for pos in xrange(exception.start, exception.end)]
+        return (u"\ufffe%s\ufeff" % u"".join(l), exception.end)
+
+    codecs.register_error("consolereplace", consolereplace_errors)
+
+    FOREGROUND_RED = 4
+
+    class WindowsConsoleWriter:
+        def __init__(self, stdhandle, errors='strict'):
+            self.errors = errors
+            self.encoding = 'utf_16'
+            self.handle = windll.kernel32.GetStdHandle(stdhandle)
+
+            csbi = CONSOLE_SCREEN_BUFFER_INFO()
+            windll.kernel32.GetConsoleScreenBufferInfo(self.handle, byref(csbi))
+
+            self.defattr = csbi.wAttributes
+
+        def writelines(self, text):
+            self.write(text + "\n")
+
+        def _write(self, text):
+            windll.kernel32.WriteConsoleW(self.handle, text, len(text), 0, 0)
+
+        def _color(self, color):
+            windll.kernel32.SetConsoleTextAttribute(self.handle, color)
+
+        def write(self, text):
+            text = unicode(text, 'utf_8', errors='consolereplace')
+            i = 0
+            j = 0
+            for c in text:
+                if c == u'\ufffe':
+                    self._write(text[i:j])
+                    i = j+1
+                    self._color(FOREGROUND_RED)
+                elif c == u'\ufeff':
+                    self._write(text[i:j])
+                    i = j+1
+                    self._color(self.defattr)
+                j+=1
+            self._write(text[i:j])
+
+        def isatty(self): return True 
+
+        def flush(self): pass
+
+    if sys.stdout.isatty():
+        sys.stdout = WindowsConsoleWriter(STD_OUTPUT_HANDLE)
+    if sys.stderr.isatty():
+        sys.stderr = WindowsConsoleWriter(STD_ERROR_HANDLE)
+
 profiling = False
 if '--profile-imports' in sys.argv:
     sys.argv.remove('--profile-imports')