Merge lp:~pascal-devuyst/apt-ddtp-tools/DebianSync into lp:apt-ddtp-tools

[Pascal De Vuyst]

Hi Michael,

I have modified your update-from-debian.sh, gen-sync.sh,
po2translation.py and cmp-translations.py scripts in apt-ddtp-tools to
give a list of descriptions missing in debian:
* Translation-en instead of Packages
* Fixed typo's

* update-from-debian.sh: variables for debian and ubuntu mirrors and
  only get Translation for specific language in series.sh
* gen-sync.sh: remove-duplicates.py is missing so commented it out
* po2translation.py: mark missing translation chunks as <trans> like
  ddtp.debian.net <http://ddtp.debian.net> does, fixed error of extra
  trailing \n with enumerations, Description-en instead of _Description
* cmp-translations.py: make it work with Translation-main-$lang files in
  missing-in-debian and output alphabetically

I created ddtp-get.py to fetch a package description from
ddtp.debian.net <http://ddtp.debian.net>, mail-new-ddtp.py to send a
package description to debian using email interface and sync-missing to
send descriptions missing in debian back to debian (as suggested on
<email address hidden> <mailto:debian-i18n%40lists.debian.org>).
Please read DebianSync.

Please consider including these changes into apt-ddtp-tools and feel
free to improve where necessary.

Thanks in advance,
Pascal

[Michael Vogt (mvo)]

Hello Pascal,

thanks a lot for this branch and sorry for my slow reply.

This looks good, but I do have some questions:

=== modified file 'po2translation.py'
--- po2translation.py 2010-12-20 13:22:27 +0000
+++ po2translation.py 2013-05-30 07:21:40 +0000
@@ -41,7 +41,7 @@
                # we have no translation for this chunk
                if only_do_full_translations:
                        return ""
- short_descr_trans = short_descr
+ short_descr_trans = "<trans>"
        else:
                have_translation = True

Why is this needed? Will this <trans> get filtered out later?
@@ -91,7 +91,7 @@
                        for line in chunk.split("\n"):
                                if line:
                                        new_chunk += " %s\n" % line
- chunk = new_chunk
+ chunk = new_chunk[:-1] # get rid of trailing \n
                # make sure here that chunks are seperated with "\n .\n"
                if long_descr_str == "":
                        long_descr_str = "%s" % chunk

Its probably best to use chunk = new_chunk.rstrip("\n") here to be robust against chunks that do not end with \n.

For the sending of mails, I wonder if it wouldn't be easier to just call the sendmail binary (maybe as a optional way) for people who already have a installed and configured sendmail.

It would be nice if the return codes (253, 252 etc) also have symbolic names somewhere.

I hope the above makes sense, please let me know if you have any questions. Its a nice touch that you also fix the hardcoded ubuntu mirror and make that configurable instead, its only a small part of the large patch but still a nice touch :)

Out of curiosity, when we send new translations to the email interface, do we have to be careful about not sending too much? Will it enter some sort of queue/review process in debian? Should we notify them before we send the stuff back or did you already started the sending back ?

Thanks!
 Michael

[Pierre Slamich (pierre-slamich)]

Was also wondering about throttling to ensure we don't crash upstream's servers

Unmerged revisions

90. By Pascal De Vuyst on 2013-05-06

update-from-debian.sh, gen-sync.sh, po2translation.py and cmp-translations.py:
* Translation-en instead of Packages
* Fixed typo's

* update-from-debian.sh: variables for debian and ubuntu mirrors and only get
  Translation for specific language in series.sh
* gen-sync.sh: remove-duplicates.py is missing so commented it out
* po2translation.py: mark missing translation chunks as <trans> like
  ddtp.debian.net does, fixed error of extra trailing \n with enumerations,
  Description-en instead of _Description
* cmp-translations.py: make it work with Translation-main-$lang files in
  missing-in-debian and output alphabetically

I created ddtp-get.py to fetch a package description from ddtp.debian.net,
mail-new-ddtp.py to send a package description to debian using email interface
and sync-missing to send descriptions missing in debian back to debian (as
suggested on <email address hidden>). Please read DebianSync.

=== added file 'DebianSync'
--- DebianSync	1970-01-01 00:00:00 +0000
+++ DebianSync	2013-05-06 18:35:30 +0000
@@ -0,0 +1,30 @@
+#1) Generate list of translations missing in Debian:
+
+#Preparations:
+sudo apt-get install distro-info lftp
+#Edit apt-ddtp-tools/series.sh and set SERIES, LNG and archive mirrors!
+#Get recent $lang.po from ddtp branch or export them for each $comp by email
+#and link (ln -s) or put them into apt-ddtp-tools/ubuntu/$comp/$lang.po
+
+cd apt-ddtp-tools/ubuntu/debian-ddtp-import
+./update-from-debian.sh
+./gen-sync.sh
+#Translations missing in Debian are listed in
+#./missing-in-debian/Translation-$comp-$lang
+
+#2) Send missing translations to Debian:
+#Preparations: edit ./mail-new-ddtp.py and set SMTP_SERVER, SMTP_PORT, SENDER 
+#and PASSWORD
+./sync-missing.py ./missing-in-debian/Translation-$comp-$lang
+
+#3) Get and send a translation directly from/to Debian:
+./ddtp-get.py $package $lang
+#Edit new.ddtp and change the translation:
+editor new.ddtp
+#Send the translation to Debian:
+./mail-new-ddtp.py
+#A file $lang.avoid can be use to avoid words in the translation (frequent
+#errors). The format is word <tab> suggestion. If an avoid word occurs in the
+#translation it will not be send until corrected.
+#./mail-new-ddtp.py -f or ./mail-new-ddtp.py --force can be used to
+#send translations anyway even if avoid words occur.

=== modified file 'cmp-translations.py'
--- cmp-translations.py	2010-12-20 13:25:44 +0000
+++ cmp-translations.py	2013-05-06 18:35:30 +0000
@@ -19,13 +19,13 @@
 
     if len(sys.argv) < 3:
         print "Need two translation files left-file right-file (missing-from-right)"
-        print "the paramter 'missing-from-right' is option and will create "
+        print "the parameter 'missing-from-right' is option and will create "
         print "a file with the missing Translation stuff from the right hand side"
         sys.exit(1)
     
     # sanity check
-    lang = os.path.basename(sys.argv[1]).split("-")[1]
-    if lang != os.path.basename(sys.argv[2]).split("-")[1]:
+    lang = os.path.basename(sys.argv[1]).rsplit("-",1)[1]
+    if lang != os.path.basename(sys.argv[2]).rsplit("-",1)[1]:
         print "Different languages, does not make sense to compare them"
         sys.exit(1)
 
@@ -68,7 +68,7 @@
     # output what is missing in the left
     if len(set(left_hand.keys()).difference(set(right_hand.keys()))) > 0:
         f=open(missing_on_left,"w")
-        for pkg in set(left_hand.keys()).difference(set(right_hand.keys())):
+        for pkg in sorted(set(left_hand.keys()).difference(set(right_hand.keys()))):
             f.write("Package: %s\n" % pkg)
             f.write("Description-md5: %s\n" % left_hand[pkg][0])
             f.write("Description-%s: %s\n\n" % (lang, left_hand[pkg][1]))

=== modified file 'po2translation.py'
--- po2translation.py	2010-12-20 13:22:27 +0000
+++ po2translation.py	2013-05-06 18:35:30 +0000
@@ -41,7 +41,7 @@
 		# we have no translation for this chunk
 		if only_do_full_translations:
 			return ""
-		short_descr_trans = short_descr
+		short_descr_trans = "<trans>"
 	else:
 		have_translation = True
 		
@@ -61,7 +61,7 @@
 		if not chunk_translated:
 			if only_do_full_translations:
 				return ""
-			chunk_translated = chunk
+			chunk_translated = "<trans>"
 		else:
 			have_translation=True
 		# strip away any \r and \n\n to avoid anything like bug #195383
@@ -91,7 +91,7 @@
 			for line in chunk.split("\n"):
 				if line:
 					new_chunk += " %s\n" % line
-			chunk = new_chunk
+			chunk = new_chunk[:-1] # get rid of trailing \n
 		# make sure here that chunks are seperated with "\n .\n"
 		if long_descr_str == "":
 			long_descr_str = "%s" % chunk
@@ -120,7 +120,7 @@
 	translated_descriptions = 0
 	while parser.step():
 		pkg = parser.section.get("Package")
-		descr = parser.section.get("_Description")
+		descr = parser.section.get("Description-en")
 		# work with a regular packages file too
 		if not descr:
 			descr = parser.section.get("Description")
@@ -136,7 +136,7 @@
                             format="%(asctime)s %(message)s")
 
 	if len(sys.argv) < 4:
-		print "need a Packages file a po file and a language as argument"
+		print "need a Translation-en/Packages file a po file and a language as argument"
 		print "it will search for the mo file in "
 		print "mo/$lang/LC_MESSAGES/packages.mo"
 		sys.exit(1)

=== modified file 'series.sh'
--- series.sh	2013-04-09 18:45:59 +0000
+++ series.sh	2013-05-06 18:35:30 +0000
@@ -1,7 +1,15 @@
-
-# the series to use (default auto-detect)
-#SERIES=$(lsb_release -s -c)
-SERIES="raring"
+#The series to use:
+#SERIES="raring"
+#Current development version short codename (requires distro-info package):
+SERIES=$(distro-info --devel)
 
 DIST="$SERIES"
 
+#update-from-debian.sh language:
+LNG=nl
+#All languages:
+#LNG=""
+
+#Archive mirrors to use:
+UBUNTUMIRROR=be.archive.ubuntu.com
+DEBIANMIRROR=ftp.be.debian.org

=== added file 'ubuntu/debian-ddtp-import/ddtp-get.py'
--- ubuntu/debian-ddtp-import/ddtp-get.py	1970-01-01 00:00:00 +0000
+++ ubuntu/debian-ddtp-import/ddtp-get.py	2013-05-06 18:35:30 +0000
@@ -0,0 +1,48 @@
+#!/usr/bin/python
+
+import sys, urllib2, re, hashlib
+
+if len(sys.argv) < 3:
+  print 'please specify package and language'
+  exit(1)
+
+f = urllib2.urlopen('https://ddtp.debian.net/ddt.cgi?package=' + sys.argv[1])
+pkg = f.readlines()
+
+i = None
+for line in pkg:
+    if re.search('active', line):
+        try:
+          i = re.search('id=([0-9]*)',prevline).group(1)
+        except AttributeError:
+          i = None
+	break
+    prevline = line
+
+if i == None:
+  print "package " + sys.argv[1] + " not active or not present in DDTP"
+  exit(254)
+else:
+  print 'Id: ' + i
+
+f = urllib2.urlopen('https://ddtp.debian.net/ddt.cgi?desc_id='+i+'&getuntrans=' + sys.argv[2])
+trans = f.read()
+
+descr = trans[trans.find('Description:')+13:trans.find('Description-')]
+m = hashlib.md5()
+m.update(descr)
+print 'Description-md5: ' + m.hexdigest()
+print trans
+descr2 = ''
+if trans.find('<trans>') == -1:
+   print 'translated to ' + sys.argv[2]
+   descr2 = trans[trans.find('Description-')+14+len(sys.argv[2]):trans.find('\n\n')] + '\n'
+   exitval = 253
+else:
+   print 'not yet translated to ' + sys.argv[2]
+   exitval = 252
+f = open('new.ddtp', 'w')
+message='# Package(s): ' + sys.argv[1] + '\nDescription: ' + descr[:-1] + '\nDescription-' + sys.argv[2] + '.UTF-8: ' + descr2
+f.write(message)
+f.close()
+exit(exitval)

=== modified file 'ubuntu/debian-ddtp-import/gen-sync.sh'
--- ubuntu/debian-ddtp-import/gen-sync.sh	2010-12-20 13:23:42 +0000
+++ ubuntu/debian-ddtp-import/gen-sync.sh	2013-05-06 18:35:30 +0000
@@ -25,17 +25,17 @@
 	LANG=${LANG#Translation-}
 	LANG=${LANG%.bz2}
 	echo "working on $LANG"
-	if ! ../../po2translation sid/main/Packages ../$comp/$LANG.po $LANG --full-translations > tmp/Translation-$LANG; then
+	if ! ../../po2translation sid/main/Translation-en ../$comp/$LANG.po $LANG --full-translations > tmp/Translation-$LANG; then
 	    continue
 	fi
 	# now check what translations are there but not in debian and
-	# geneate a file for this
+	# generate a file for this
 	../../cmp-translations.py ${f%.bz2} tmp/Translation-$LANG missing-in-debian/Translation-$comp-$LANG | tee missing-in-debian/Translation-$comp-$LANG.stats
 
-	# remove duplicates for now (ddtp has a bug)
-	if [ -e missing-in-debian/Translation-$comp-$LANG ]; then
-	    ../../remove-duplicates.py sid/main/i18n/Translation-$LANG missing-in-debian/Translation-$comp-$LANG > missing-in-debian/Translation-.no.same.md5.as.in.debian-$comp-${LANG}
-	fi
+	## remove duplicates for now (ddtp has a bug)
+	#if [ -e missing-in-debian/Translation-$comp-$LANG ]; then
+	#    ../../remove-duplicates.py sid/main/i18n/Translation-$LANG missing-in-debian/Translation-$comp-$LANG > missing-in-debian/Translation-.no.same.md5.as.in.debian-$comp-${LANG}
+	#fi
 
     done
 

=== added file 'ubuntu/debian-ddtp-import/mail-new-ddtp.py'
--- ubuntu/debian-ddtp-import/mail-new-ddtp.py	1970-01-01 00:00:00 +0000
+++ ubuntu/debian-ddtp-import/mail-new-ddtp.py	2013-05-06 18:35:30 +0000
@@ -0,0 +1,73 @@
+#!/usr/bin/python
+
+import sys, smtplib
+
+SMTP_SERVER = 'smtp.gmail.com'
+SMTP_PORT = 587
+SENDER = '[e-mail]'
+PASSWORD = 'xxxx'
+
+f = open('new.ddtp', 'r')
+s = f.read()
+f.close()
+pkg = s[s.find('Package(s):')+12:s.find('\n')]
+lang = s[s.find('Description-')+12:s.find('.UTF-8')]
+tstart = s.find('Description-')+len(lang)+20
+short = s[tstart:s.find('\n',tstart)]
+trans = s[tstart:s.find('\n\n',tstart)]
+
+if len(short) > 80:
+  print "WARNING: translated short description too long (>80 characters), please edit and resend."
+  exit(1)
+
+try:
+ f = open(lang+'.avoid', 'r')
+ avoid = f.readlines()
+ f.close()
+except IOError:
+ avoid = []
+
+try:
+  if sys.argv[1] == "--force" or sys.argv[1] == "-f":
+    force = True
+except IndexError:
+  force = False
+
+av = False
+for item in avoid:
+  av0 = item.strip('\n').split('\t')[0]
+  av1 = item.strip('\n').split('\t')[1]
+  if av0 in trans:
+     print av0 + " should be translated as " + av1
+     av = True
+     continue
+if av == True and force == False: exit(1)
+
+recipient = '[e-mail]'
+subject = pkg + ' ' + lang
+ 
+from email.mime.multipart import MIMEMultipart
+msg = MIMEMultipart()
+msg['Subject'] = subject
+msg['To'] = recipient
+msg['From'] = SENDER
+
+from email.MIMEText import MIMEText
+ 
+filename = "new.ddtp"
+f = open(filename,'r')
+attachment = MIMEText(f.read())
+f.close()
+attachment.add_header('Content-Disposition', 'attachment', filename=filename)
+msg.attach(attachment)
+
+session = smtplib.SMTP(SMTP_SERVER, SMTP_PORT)
+
+session.ehlo()
+session.starttls()
+session.ehlo
+session.login(SENDER, PASSWORD)
+ 
+session.sendmail(SENDER, recipient, msg.as_string())
+print "sent " + subject
+session.quit()

=== added file 'ubuntu/debian-ddtp-import/nl.avoid'
--- ubuntu/debian-ddtp-import/nl.avoid	1970-01-01 00:00:00 +0000
+++ ubuntu/debian-ddtp-import/nl.avoid	2013-05-06 18:35:30 +0000
@@ -0,0 +1,3 @@
+dummy	leeg
+frontend	interface
+development	ontwikkelings

=== added file 'ubuntu/debian-ddtp-import/sync-missing.py'
--- ubuntu/debian-ddtp-import/sync-missing.py	1970-01-01 00:00:00 +0000
+++ ubuntu/debian-ddtp-import/sync-missing.py	2013-05-06 18:35:30 +0000
@@ -0,0 +1,36 @@
+#!/usr/bin/python
+
+import sys, apt_pkg, subprocess, hashlib
+
+if len(sys.argv) < 2:
+  print 'please specify Translation file'
+  exit()
+
+lang = sys.argv[1].rsplit("-",1)[1]
+
+translation = []
+rt = apt_pkg.ParseTagFile(open(sys.argv[1]))
+while rt.Step():
+  translation.append( (rt.Section.get("Package"),rt.Section.get("Description-md5"),rt.Section.get("Description-"+lang)) )
+
+for item in translation:
+  print "-" * 75
+  r = subprocess.call(['./ddtp-get.py',item[0],lang])
+  if r == 253 or r == 254: #translated | package not active or not present in DDTP
+    continue
+  f = open('new.ddtp', 'r')
+  trans = f.read()
+  f.close()
+  descr = trans[trans.find('Description:')+13:trans.find('Description-')]
+  if r == 252: #not yet translated
+    m = hashlib.md5()
+    m.update(descr)
+    if m.hexdigest() == item[1]:
+      f = open('new.ddtp','a')
+      f.write(item[2]+'\n')
+      f = open('new.ddtp','r')
+      print f.read()
+      f.close()
+      subprocess.call(['./mail-new-ddtp.py','--force'])
+    else:
+      print "md5 doesn't match"

=== modified file 'ubuntu/debian-ddtp-import/update-from-debian.sh'
--- ubuntu/debian-ddtp-import/update-from-debian.sh	2012-04-17 13:22:53 +0000
+++ ubuntu/debian-ddtp-import/update-from-debian.sh	2013-05-06 18:35:30 +0000
@@ -2,25 +2,25 @@
 
 set -e
 
-# include series and DIST overide
+# include series and DIST override
 . ../../series.sh
 
 TRANSLATIONS2PO=`pwd`/../../translation2po
 
-# get the debian translatins
+# get the debian translations
 #rm -f Translations.tar.gz
 #wget http://ddtp.debian.net/debian/Translations.tar.gz
 #tar xzvf Translations.tar.gz
 
-# get current packages file
+# get current Translation-en file
 mkdir -p sid/main/ || true
 cd sid/main
-rm -f Packages
-wget -c ftp://ftp.de.debian.org/debian/dists/sid/main/binary-i386/Packages.gz
-gunzip Packages.gz
+rm -f Translation-en
+wget -c "ftp://$DEBIANMIRROR/debian/dists/sid/main/i18n/Translation-en.bz2"
+bunzip2 Translation-en.bz2
 cd ../..
 
-# *sigh* now that debian does no longer publish the transltions tarball
+# *sigh* now that debian does no longer publish the translations tarball
 # on debian.net we need to get it elsewhere
 mkdir -p sid/main/i18n/ || true
 cd sid/main/i18n ;
@@ -28,7 +28,8 @@
 # mget -c is playing tricks on me
 rm -f *.bz2
 
-echo "mget -c *.bz2"|lftp 'ftp://ftp.de.debian.org/debian/dists/sid/main/i18n'
+# if $LNG only get Translation for that language
+echo "mget -c *$LNG.bz2"|lftp "ftp://$DEBIANMIRROR/debian/dists/sid/main/i18n"
 for i in *.bz2; do
      bunzip2 -c $i > ${i%.bz2} 
 done
@@ -49,12 +50,12 @@
     rm -f $comp/*
     cd $comp
     # the translations file
-    wget http://archive.ubuntu.com/ubuntu/dists/$DIST/$comp/i18n/Translation-en
+    wget "http://$UBUNTUMIRROR/ubuntu/dists/$DIST/$comp/i18n/Translation-en"
     for f in ../sid/main/i18n/Translation-*.bz2; do
 	file=${f%.bz2} 
 	filename=$(basename $file)
 	lang=${filename#Translation-}
-	echo "$TRANSLATIONS2PO Packages $file $lang >$lang.po"
+	echo "$TRANSLATIONS2PO Translation-en $file $lang >$lang.po"
 	$TRANSLATIONS2PO Translation-en $file $lang >$lang.po
     done
     cd ..