Merge ~darinmiller/ka:master into ka:master

[Darin Miller]

santa_requested a code cleanup-replacing manually built path strings in the ka python scripts with the os.path.join function.

The following grep commands were performed to find scripts with manual path builds:
- grep "\'/\'" *
- grep "\"/\"" *

The following scripts were modified:

        modified: do-all
        modified: get-kci-tarball
        modified: ka-update-metadata
        modified: kubuntu-create-sru-branches
        modified: staging-upload
        modified: ubuntu-archive-upload
        modified: uploadsource

PLEASE double check my edits that I did not extend a path build beyond the parameter in which the path was built. i.e. some functions were passing manually built paths in the function invocation. Since I do not know how or may not have the privilege to run these scripts...

  THESE EDITS ARE UNTESTED!

[Jose Manuel Santamaria Lema (panfaust)]

Hi Darin,

I have just read the diff and it seems mostly ok, I plan to use KA with these changes for a while to check everything works as expected. Since you mention that you may or may not have permissions to test the changes yourself, let's see one by one.

do-all:
This command iterates over several directories executing the command given, so if you have a set of clones created with git-clone-all you should be able to test a "do-all ls" against that set of clones.

get-kci-tarball:
This program creates a 'fake' kci tarball when executed from a particular clone, so inside any clone of frameworks/plasma/apps packaging you should be able to execute this and get an orig tarball in the build-area directory. The tarball is created using KDE's git anonymously, so you shouldn't need any kind of privilege to test it.

ka-update-metadata:
This is something we use to execute when providing a new release (see the README.ng) to test this maybe we could mentor you to do something real, like any frameworks/plasma/apps backport we don't have right now or something like that. We could arrange this by IRC. This way you will get more familiar with our KA scriptery.

uploadsource:
This one is just an optional program from KA meant to be a small wrapper around dput/dupload. If I'm not mistaken, you could test it uploading any package from frameworks/plasma/apps to any of your ppas like this:
uploadsource -t <your_ppa>

kubuntu-create-sru-branches, staging-upload, ubuntu-archive-upload:
These ones are the old ugly monolithic scripts which we replaced by UNIX-like git-clone-all/do-all/gbp-* and friends after the "Slice" operation[1]. I would drop the changes on these, since what I'm planning to do is writing a mail to kubuntu-devel mailing list notifying that I'm going to move them to an "attic/" directory as part of the "Fir Tree" operation[2]. This way it would be easier to
know if a script is obsolete or not without removing it completely just in case we want to recycle any code in them.

[1]https://phabricator.kde.org/w/kubuntu/black-operations/slice/
[2]https://phabricator.kde.org/w/kubuntu/black-operations/fir-tree/

[Jose Manuel Santamaria Lema (panfaust)]

Hi,

since we needed to add a new feature to do-all I have just merged into master the proposed changes, they seem to work fine so far. If anything, we could fix it later.

Thanks for the patch Darin.

review: Approve

diff --git a/do-all b/do-all
index 39ce3f7..f2d908f 100755
--- a/do-all
+++ b/do-all
@@ -84,19 +84,19 @@ n = len(dirs)
 for directory in dirs:
     i += 1
     print("----------------------------------------------------------------------")
-    print("do-all current directory: %s (%s of %s)" % (directory+'/'+subdirectory,i,n))
+    print("do-all current directory: %s (%s of %s)" % (os.path.join(directory, subdirectory),i,n))
     print("----------------------------------------------------------------------")
     sys.stdout.flush()
     try:
-        os.chdir(old_cwd + '/' + directory + '/' + subdirectory )
+        os.chdir(os.path.join(old_cwd, directory, subdirectory))
     except:
-        dirs_not_accessible.append(directory + '/' + subdirectory)
+        dirs_not_accessible.append(os.path.join(directory, subdirectory))
         print("directory not accessible, failed to 'cd' into it")
         continue
     if directory in skip_list:
         print("As requested by the configuration file,\n"
             "the command wasn't executed in this directory")
-        dirs_ignored.append(directory + '/' + subdirectory)
+        dirs_ignored.append(os.path.join(directory, subdirectory))
         continue
     try:
         returncode = subprocess.call(command)
@@ -109,7 +109,7 @@ for directory in dirs:
         #http://www.tldp.org/LDP/abs/html/exitcodes.html
         returncode = 127
     if returncode != 0:
-        dirs_failed.append((directory+'/'+subdirectory,returncode))
+        dirs_failed.append((os.path.join(directory, subdirectory), returncode))
     sys.stdout.flush()
 
 #Print summary
diff --git a/get-kci-tarball b/get-kci-tarball
index 7675f20..fbf3271 100755
--- a/get-kci-tarball
+++ b/get-kci-tarball
@@ -58,7 +58,7 @@ kde_git_repository_url = "https://anongit.kde.org/%s" % upstream_name
 
 #Create temporary directory to clone the repository.
 temp_dir = tempfile.TemporaryDirectory()
-kde_clone_dir_name = temp_dir.name+"/"+upstream_name
+kde_clone_dir_name = os.path.join(temp_dir.name, upstream_name)
 
 #Clone repository
 returncode = subprocess.call(["git","clone",kde_git_repository_url,kde_clone_dir_name])
diff --git a/ka-update-metadata b/ka-update-metadata
index a99c489..dcb6cd1 100755
--- a/ka-update-metadata
+++ b/ka-update-metadata
@@ -77,7 +77,7 @@ for d in dirs:
         d = d.split("-opensource-src")[0]
     #Find out source package version
     changelog = Changelog()
-    changelog_file_name = d + '/' + subdir + '/debian/changelog'
+    changelog_file_name = os.path.join(d, subdir, 'debian/changelog')
     try:
         changelog.parse_changelog(codecs.open(changelog_file_name, 'r', 'utf-8'))
     except FileNotFoundError:
@@ -114,7 +114,7 @@ for d in dirs:
     # With the info parsed from $PWD/*/debian/control populate:
     # - build_depends_set <- with all the binary packages appearing in Build-Depends[-Indep]
     # - dev_package_version_map <- with all the binary packages with their versions
-    control_file_name = d + '/' + subdir + '/debian/control'
+    control_file_name = os.path.join(d, subdir, 'debian/control')
     try:
         control_file = deb822.Packages.iter_paragraphs(open(control_file_name, 'r'));
     except FileNotFoundError:
diff --git a/kubuntu-create-sru-branches b/kubuntu-create-sru-branches
index c4bfd13..a71288f 100755
--- a/kubuntu-create-sru-branches
+++ b/kubuntu-create-sru-branches
@@ -58,7 +58,7 @@ for package in packages:
         print("Branching %s ..." % branch_url)
         subprocess.check_call(["bzr", "branch", branch_url])
 
-        os.chdir(workdir + "/" + package)
+        os.chdir(os.path.join(workdir, package))
         with open("debian/control", "r") as f:
             control = f.read()
 
diff --git a/staging-upload b/staging-upload
index 3e9eb43..57e2842 100755
--- a/staging-upload
+++ b/staging-upload
@@ -186,12 +186,12 @@ for package in packages:
             continue
 
     try:
-        os.mkdir(basedir + "/" + package)
+        os.mkdir(os.path.join(basedir, package))
     except OSError:
-        print("removing existing " + basedir + "/" + package)
-        shutil.rmtree(basedir + "/" + package)
-        os.mkdir(basedir + "/" + package)
-    os.chdir(basedir + "/" + package)
+        print("removing existing " os.path.join(basedir, package))
+        shutil.rmtree(os.path.join(basedir, package))
+        os.mkdir(os.path.join(basedir, package))
+    os.chdir(os.path.join(basedir, package))
     if package in packagingExceptions:
         upstreamVersion = packagingExceptions[package]
     else:
@@ -215,7 +215,7 @@ for package in packages:
         subprocess.check_call(["rsync", "--progress", "-z", "--copy-links", "--compress-level=9", "-e ssh", remote, "."])
     except:
         print("==== Skipping %s, failed to fetch tarball from depot.kde.org, moving to manual/" % (package))
-        shutil.move(basedir + "/" + package, basedir + "/manual/" + package)
+        shutil.move(os.path.join(basedir, package), os.path.join(basedir, "manual", package))
         moved.append((package, "failed to fetch tarball from depot"))
         continue
     orig = "%s_%s.orig.tar.xz" % (package, upstreamVersion)
@@ -237,7 +237,7 @@ for package in packages:
         match = re.match(re.escape(package) + r'_([\d\.a-z]+)-', dscname)
         if not match:
             print("==== Skipping %s, version parsing failed, moving to manual/" % (package,))
-            shutil.move(basedir + "/" + package, basedir + "/manual/" + package)
+            shutil.move(os.path.join(basedir, package), os.path.join(basedir, "manual", package))
             moved.append((package, "version parsing failed"))
             continue
 
@@ -297,7 +297,7 @@ for package in packages:
 
         if not sanitizeBranch(package):
             print "==== Skipping %s, git branch has unexpected content, moving to manual/" % (package,)
-            shutil.move(basedir + "/" + package, basedir + "/manual/" + package)
+            shutil.move(os.path.join(basedir, package), os.path.join(basedir, "manual" , package))
             moved.append((package, "git unclean or out of sync"))
             continue
 
@@ -346,7 +346,7 @@ for package in packages:
         # package build failed, manual intervention necessary
         print "==== " + package + " build was interrupted, moving to manual/"
         shutil.rmtree(basedir + "/manual/" + package, True)
-        shutil.move(basedir + "/" + package, basedir + "/manual/" + package)
+        shutil.move(os.path.join(basedir, package), os.path.join(basedir, "manual", package))
         moved.append((package, "build was interrupted"))
 
 print("\n\n=== Packages are ready for upload in " + uploaddir)
diff --git a/ubuntu-archive-upload b/ubuntu-archive-upload
index c32f3f1..591acaa 100755
--- a/ubuntu-archive-upload
+++ b/ubuntu-archive-upload
@@ -126,7 +126,7 @@ for package in packages:
     except IndexError:
         print "=== %s not in the archive, new package?" % (package)
 
-    packageDir = basedir + "/" + package
+    packageDir = os.path.join(basedir, package)
     try:
       os.mkdir(packageDir)
     except OSError:
@@ -145,7 +145,7 @@ for package in packages:
         if package == "kde4libs":
             print "MMM gitName = kde4libs"
             gitName = "kde4libs" # as an exception this one has git repo named after package
-        subprocess.check_call(["git", "clone", debian_git + "/" + gitName, "git"])
+        subprocess.check_call(["git", "clone", os.path.join(debian_git, gitName), "git"])
         os.chdir("git")
         subprocess.check_call(["git", "checkout", "kubuntu_"+release+"_archive"])
     else:
diff --git a/uploadsource b/uploadsource
index 69a8f83..6ca1df7 100755
--- a/uploadsource
+++ b/uploadsource
@@ -15,6 +15,7 @@ import subprocess
 import sys
 import argparse
 import glob
+import os
 
 from debian.changelog import Changelog, Version
 
@@ -62,7 +63,7 @@ if changelog.debian_revision != None:
     version += "-" + changelog.debian_revision
 
 #Find out the filename to upload
-file_pattern = ka_config['areas']['upload-area'] + "/" + src_pkg + "_" + version + "*_source.changes"
+file_pattern = os.path.join(ka_config['areas']['upload-area'], src_pkg + "_" + version + "*_source.changes")
 files = glob.glob(file_pattern)
 
 if not files: