Merge lp:~qzhang/lava-dispatcher/fix-unicode-encode-err into lp:lava-dispatcher

[Spring Zhang]

fix Bug #788846

[Zygmunt Krynicki (zyga)]

Is self.serialio expecting binary strings or unicode strings? Do you do any extra manipulation on that string? This code seems to do nothing useful because:

1) Either the serial line expects unicode strings and the world is upside down (this is not likely)
2) The serial line is happy with any bytes you send it and you just corrupt the 8th bit for no good reason.

review: Needs Information

[Paul Larson (pwlars)]

11 + unicode_text = text.decode('ascii', 'replace')
12 + ascii_text = unicode_text.encode('ascii', 'replace')
I *think* it should not be necessary to do both of these, only one. The issue, as I understand it, is the mixture of 8-bit strings and unicode. Were you able to reliably reproduce what Michael was seeing so that you can know the fix works?

review: Needs Information

[Spring Zhang (qzhang)]

> 11 + unicode_text = text.decode('ascii', 'replace')
> 12 + ascii_text = unicode_text.encode('ascii', 'replace')
> I *think* it should not be necessary to do both of these, only one. The
> issue, as I understand it, is the mixture of 8-bit strings and unicode. Were
> you able to reliably reproduce what Michael was seeing so that you can know
> the fix works?
Only line 12 will raise the same exception, so I think it needs to decode then encode back to the original forms, or only decode is necessary?
I have met the exact issue which Michael met, and do some test after the commit and no such exception found.

[Spring Zhang (qzhang)]

> Is self.serialio expecting binary strings or unicode strings? Do you do any
> extra manipulation on that string? This code seems to do nothing useful
> because:
No extra manipulation on the string, except the code in this commit. self.SerialIO just catch the output of serial output text and put it to both logfile and StringIO, it doesn't expect unicode or other strings, the exception raises because StringIO can't mix 8bit and Unicode string, but it goes to StringIO if no handling.

>
> 1) Either the serial line expects unicode strings and the world is upside down
> (this is not likely)
We can also encode it by Unicode not Ascii, we can determine it if unicode is more compatible.

> 2) The serial line is happy with any bytes you send it and you just corrupt
> the 8th bit for no good reason.
Exactly for serial line but StringIO won't accept mixed strings. And most characters from serial line are 8bit and StringIO corrupt with a Unicode byte.

[Loïc Minier (lool)]

Is this still active?

[Spring Zhang (qzhang)]

Ah, yes, but I can't meet the bug(#788846) when I use LAVA now, I'd like the submitter to verify if it's also ok now?

[Paul Larson (pwlars)]

I believe this was fixed already, please let me know if you see it again though.

[Spring Zhang (qzhang)]

Could you point out the branch which fixes it? I'm curious.

[Loïc Minier (lool)]

Should we reject this mp?

[Spring Zhang (qzhang)]

The issue is already fixed by some uncertain code change.

Unmerged revisions

60. By Spring Zhang on 2011-06-01

Fix error "UnicodeDecodeError: 'ascii' codec can't decode" when get_value in StringIO

=== modified file 'lava/dispatcher/client.py'
--- lava/dispatcher/client.py	2011-05-26 22:31:59 +0000
+++ lava/dispatcher/client.py	2011-06-01 09:22:03 +0000
@@ -120,7 +120,12 @@
         self.logfile = logfile
 
     def write(self, text):
-        self.serialio.write(text)
+        #in case characters which are not in rage(0..128) 
+        #throwing UnicodeEncodeError when get_value() from SerialIO
+        unicode_text = text.decode('ascii', 'replace')
+        ascii_text = unicode_text.encode('ascii', 'replace')
+        self.serialio.write(ascii_text)
+        #let logfile to record all logs
         self.logfile.write(text)
 
     def close(self):