Merge lp:~nobuto/ubuntu/wily/unzip/fallback-encoding into lp:ubuntu/wily/unzip

[Nobuto Murata]

* add fallback charsets for Windows archives in CJKV+th locale,
  inspired by Ubuntu Kylin way, LP: #1422290
* add autopkgtest to make sure all of 3 types of zip files can be
  extracted properly

[Steve Langasek (vorlon)]

This leaks a variable with a common name into the user's shell environment and is not appropriate. It is also against Debian policy, which specifies that packages should not rely on environment variables for correct operation. I can see three possible ways that this could be done instead:

 - use a 'local' variable in the profile snippet to avoid leaking the $charset variable into the shell's variables;
 - provide these as wrapper scripts instead around the real zip,unzip commands which set the -O option if not set; or
 - (preferred) patch this default behavior into the upstream code.

Note that the -O option to unzip is already a Debian-local patch (debian/patches/20-unzip60-alt-iconv-utf8). Support for these defaults should be merged into that patch.

In particular, it appears that dos_charset_map in unix/unix.c can be extended with the following mappings for equivalent results:

{ "EUC-JP", "CP932" },
{ "EUC-KR", "CP949" },
{ "TIS-620", "CP874" },
{ "GB2312", "CP936" },
{ "BIG5", "CP950" },

This covers all of the languages included in your patch *except* for Vietnamese; Vietnamese uses a UTF-8 locale by default, so it's impossible to infer the correct codepage of the DOS-encoded zip file from the Unix charset for that case. You could handle the (non-default) vi_VN.TCVN locale by adding:

{ "TCVN5712-1", "CP1258" },

All Unix charset values taken from /usr/share/i18n/SUPPORTED.

Personally I would find it somewhat unsatisfactory that the codepage is only correctly deduced when using a non-UTF8 locale for the *other* languages, but this is a limitation of your existing patch and the existing Debian code and UTF-8 is not commonly used as the locale for these other languages, so changing as above is probably sufficient.

review: Needs Fixing

Unmerged revisions

31. By Nobuto Murata on 2015-08-23

* add fallback charsets for Windows archives in CJKV+th locale,
  inspired by Ubuntu Kylin way, LP: #1422290
* add autopkgtest to make sure all of 3 types of zip files can be
  extracted properly

=== modified file 'debian/changelog'
--- debian/changelog	2015-05-22 12:31:51 +0000
+++ debian/changelog	2015-08-23 14:28:47 +0000
@@ -1,3 +1,12 @@
+unzip (6.0-17ubuntu2) UNRELEASED; urgency=medium
+
+  * add fallback charsets for Windows archives in CJKV+th locale, 
+    inspired by Ubuntu Kylin way, LP: #1422290
+  * add autopkgtest to make sure all of 3 types of zip files can be 
+    extracted properly
+
+ -- Nobuto Murata <[e-mail]>  Sun, 23 Aug 2015 22:38:52 +0900
+
 unzip (6.0-17ubuntu1) wily; urgency=medium
 
   * Resynchronise with Debian. Remaining changes:

=== modified file 'debian/control'
--- debian/control	2015-05-22 12:31:51 +0000
+++ debian/control	2015-08-23 14:28:47 +0000
@@ -6,6 +6,7 @@
 Standards-Version: 3.9.6
 Build-Depends: debhelper (>= 9), libbz2-dev
 Homepage: http://www.info-zip.org/UnZip.html
+XS-Testsuite: autopkgtest
 
 Package: unzip
 Architecture: any

=== added directory 'debian/tests'
=== added file 'debian/tests/control'
--- debian/tests/control	1970-01-01 00:00:00 +0000
+++ debian/tests/control	2015-08-23 14:28:47 +0000
@@ -0,0 +1,2 @@
+Tests: fallback-encoding
+Depends: unzip, language-pack-ja

=== added file 'debian/tests/fallback-encoding'
--- debian/tests/fallback-encoding	1970-01-01 00:00:00 +0000
+++ debian/tests/fallback-encoding	2015-08-23 14:28:47 +0000
@@ -0,0 +1,57 @@
+#!/bin/sh
+# autopkgtest check: Extract 3 types of zip file and verify those can be
+# extracted with expected path names.
+# Author: Nobuto Murata <[e-mail]>
+
+set -e
+
+WORKDIR=$(mktemp -d)
+trap "rm -rf $WORKDIR" 0 INT QUIT ABRT PIPE TERM
+cd $WORKDIR
+
+#### zip file type 1: unx and UTF-8 ####
+# Created by zip command on Ubuntu
+mkdir unx-utf8
+cat <<EOF | base64 -d > unx-utf8/test.zip
+UEsDBAoAAAgAAEcGvUYAAAAAAAAAAAAAAAAWABwA44OG44K544OI44OV44Kp44Or44OAL1VUCQAD
+NjlnVTc5Z1V1eAsAAQToAwAABOgDAABQSwMECgAACAAAFXy8RgAAAAAAAAAAAAAAAC8AHADjg4bj
+grnjg4jjg5Xjgqnjg6vjg4Av44OG44K544OI44OV44Kh44Kk44OrLnR4dFVUCQADirZmVbs7Z1V1
+eAsAAQToAwAABOgDAABQSwECHgMKAAAIAABHBr1GAAAAAAAAAAAAAAAAFgAYAAAAAAAAABAA/UEA
+AAAA44OG44K544OI44OV44Kp44Or44OAL1VUBQADNjlnVXV4CwABBOgDAAAE6AMAAFBLAQIeAwoA
+AAgAABV8vEYAAAAAAAAAAAAAAAAvABgAAAAAAAAAAAC0gVAAAADjg4bjgrnjg4jjg5Xjgqnjg6vj
+g4Av44OG44K544OI44OV44Kh44Kk44OrLnR4dFVUBQADirZmVXV4CwABBOgDAAAE6AMAAFBLBQYA
+AAAAAgACANEAAAC5AAAAAAA=
+EOF
+
+#### zip file type 2: fat and UTF-8 ####
+# Created by Google Drive folder export
+mkdir fat-utf8
+cat <<EOF | base64 -d > fat-utf8/test.zip
+UEsDBBQACAgIAC1IvEYAAAAAAAAAAAAAAAAvADgA44OG44K544OI44OV44Kp44Or44OAL+ODhuOC
+ueODiOODleOCoeOCpOODqy50eHR1cDQAAQvM2DTjg4bjgrnjg4jjg5Xjgqnjg6vjg4Av44OG44K5
+44OI44OV44Kh44Kk44OrLnR4dAMAUEsHCAAAAAACAAAAAAAAAFBLAQIUABQACAgIAC1IvEYAAAAA
+AgAAAAAAAAAvADgAAAAAAAAAAAAAAAAAAADjg4bjgrnjg4jjg5Xjgqnjg6vjg4Av44OG44K544OI
+44OV44Kh44Kk44OrLnR4dHVwNAABC8zYNOODhuOCueODiOODleOCqeODq+ODgC/jg4bjgrnjg4jj
+g5XjgqHjgqTjg6sudHh0UEsFBgAAAAABAAEAlQAAAJcAAAAAAA==
+EOF
+
+#### zip file type 3: fat and CP932 ####
+# Created by embeded tool on Windows Server 2012 R2 with Japanese locale
+mkdir fat-cp932
+cat <<EOF | base64 -d > fat-cp932/test.zip
+UEsDBBQAAAAAABV8vEYAAAAAAAAAAAAAAAAhAAAAg2WDWINng3SDSIOLg18vg2WDWINng3SDQIND
+g4sudHh0UEsBAhQAFAAAAAAAFXy8RgAAAAAAAAAAAAAAACEAAAAAAAAAAAAgAAAAAAAAAINlg1iD
+Z4N0g0iDi4NfL4Nlg1iDZ4N0g0CDQ4OLLnR4dFBLBQYAAAAAAQABAE8AAAA/AAAAAAA=
+EOF
+
+
+# spawn login shell to read fallback charset per locale from /etc/profile.d/
+env LC_CTYPE='ja_JP.UTF-8' sh -e -l -c '
+    for type in unx-utf8 fat-utf8 fat-cp932; do
+        # all zip files contain the path "テストフォルダ/テストファイル.txt"
+        zipinfo $type/test.zip
+        unzip $type/test.zip -d $type
+        stat $type/テストフォルダ/テストファイル.txt >/dev/null
+        echo "extract $type: OK\n"
+    done
+'

=== added file 'debian/unzip-fallback-charset.sh'
--- debian/unzip-fallback-charset.sh	1970-01-01 00:00:00 +0000
+++ debian/unzip-fallback-charset.sh	2015-08-23 14:28:47 +0000
@@ -0,0 +1,36 @@
+# LP: #1422290
+# export fallback charset per locale in addition to UTF-8. Those fallbacks
+# are needed to handle zip files created on Windows systems.
+
+locale=${LC_ALL:-$LC_CTYPE}
+locale=${locale:-$LANG}
+locale=${locale%%.*}
+
+case "$locale" in
+    ja_JP)
+        charset=CP932
+    ;;
+    ko_KR)
+        charset=CP949
+    ;;
+    th_TH)
+        charset=CP874
+    ;;
+    vi_VN)
+        charset=CP1258
+    ;;
+    zh_CN)
+        charset=CP936
+    ;;
+    zh_TW)
+        charset=CP950
+    ;;
+    *)
+        charset=
+    ;;
+esac
+
+if [ -n "$charset" ]; then
+    export UNZIP="-O $charset"
+    export ZIPINFO="-O $charset"
+fi

=== modified file 'debian/unzip.install'
--- debian/unzip.install	2015-05-17 12:41:52 +0000
+++ debian/unzip.install	2015-08-23 14:28:47 +0000
@@ -1,2 +1,3 @@
+debian/unzip-fallback-charset.sh etc/profile.d
 usr/bin/*
 usr/man/* usr/share/man