Merge lp:~nskaggs/phablet-tools/fix-1371241 into lp:phablet-tools

[Nicholas Skaggs]

allow source from outside release pocket, bug 1371241

allow source from outside release pocket, bug 1371241

[lp:~nskaggs/phablet-tools/fix-1371241]

313. By Nicholas Skaggs on 2014-09-18

don't require release pocket or published for --depends either

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:312
http://jenkins.qa.ubuntu.com/job/phablet-tools-ci/402/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-utopic-amd64-ci/83
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-utopic-armhf-ci/83
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-utopic-i386-ci/83

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/phablet-tools-ci/402/rebuild

review: Approve (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:313
http://jenkins.qa.ubuntu.com/job/phablet-tools-ci/403/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-utopic-amd64-ci/84
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-utopic-armhf-ci/84
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-utopic-i386-ci/84

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/phablet-tools-ci/403/rebuild

review: Approve (continuous-integration)

[Colin Watson (cjwatson)]

Removing those filters entirely usually isn't a good idea. This looks as though it's potentially going to have some bad consequences, including:

 * fetching packages from -proposed that have not yet been promoted to release
 * fetching deleted packages

Judging from the bug report, you just need to make sure that you get status="Pending" publications as well. Have a look at the find_newest_publication function in http://bazaar.launchpad.net/~ubuntu-archive/ubuntu-archive-tools/trunk/view/head:/lputils.py - that might be a better approach.

review: Needs Fixing

[Nicholas Skaggs (nskaggs)]

Colin, the rationale was if it's in the image you want the source no matter it's status. Thanks for the link to find_newest_publication, that looks handy.

I'm concerned however that we would also need to pull superseded packages. The archive might have moved on, but the image I have on my device still runs an older version of the package. Are we still covered in this scenario?

On fetching -proposed or deleted packages, I agree this change will nab those. And we're saying we want them if they are in the image. A deleted package though is really iffy. Is there any danger in version collision with a deleted package (aka, there is a published version 20140119ubuntu1, and a deleted version 20140119ubuntu1. We don't want to pull the deleted version in this scenario) ?

I'm happy to tweak the MP so long as we can cover the superseded scenario. The other statuses aren't likely to come into play.

[Colin Watson (cjwatson)]

It's not totally clear to me why you care about a package already on your image. Perhaps I'm misunderstanding something here. That said, when you're selecting a specific version (as is the case in get_source_package_tests here, although *not* in get_python_binary_package) it seems reasonable to remove the status and pocket restrictions; they don't seem likely to help matters when you already know the version you want.

Version collisions with deleted packages within a single archive are disallowed by Launchpad, so you don't need to worry about that.

[Colin Watson (cjwatson)]

17:14 <brendand> cjwatson, i *think* you approve?
17:14 <cjwatson> brendand: no, the second half makes sense now but I remain unconvinced by the first half
17:14 <cjwatson> see the bit where I say "*not*"
17:15 <brendand> cjwatson, so why do we care about a package already in the image?
17:15 <cjwatson> that's the question I asked
17:15 <brendand> cjwatson, we want to get the tarball of autopilot tests which match that version
17:16 <cjwatson> brendand: That makes sense for get_source_package_tests. What about get_python_binary_package?
17:16 <cjwatson> There are two quite different functions being changed here.
17:16 <cjwatson> get_python_binary_package does not nominate a specific version, so there is clearly no version-matching going on.
17:17 <cjwatson> It looks to me as though get_python_binary_package just wants basically the latest published thing that apt would give you. If that's correct then removing these restrictions is wrong.
17:19 <brendand> cjwatson, i wasn't expecting get_python_binary_package to be changed so i overlooked that
17:19 <cjwatson> (Because it's possible for a newer version to be removed and then deleted; or for a version to be in -proposed that hasn't yet made it to the release pocket because it fails autopkgtests or something; or ...)
17:19 <brendand> cjwatson, perhaps balloons_ can comment on that
17:19 <cjwatson> I'll copy and paste this into the MP to clarify

[Nicholas Skaggs (nskaggs)]

@cjwatson, thanks for the comments. You are correct. We want the latest dependency that apt would give for the get_python_binary_package call.

[lp:~nskaggs/phablet-tools/fix-1371241]

314. By Nicholas Skaggs on 2014-09-29

do check for --depends

Unmerged revisions

314. By Nicholas Skaggs on 2014-09-29

do check for --depends

313. By Nicholas Skaggs on 2014-09-18

don't require release pocket or published for --depends either

312. By Nicholas Skaggs on 2014-09-18

allow source from outside release pocket, bug 1371241

=== modified file 'phablet-click-test-setup'
--- phablet-click-test-setup	2014-09-17 17:06:39 +0000
+++ phablet-click-test-setup	2014-09-29 16:25:51 +0000
@@ -156,6 +156,7 @@
     tmp_dir = tempfile.mkdtemp()
     atexit.register(cleanup, tmp_dir)
     print('Fetching %s - into %s' % (package, tmp_dir))
+<<<<<<< TREE
     try:
         sp = lp.archive.getPublishedSources(
             source_name=package,
@@ -202,6 +203,40 @@
 
 
 def get_bzr_tests(branch, revision, autopilot_dir, target_dir):
+=======
+    sp = lp.archive.getPublishedSources(
+        source_name=package,
+        version=version,
+        distro_series=lp.series,
+        exact_match=True)
+    # TODO needs filtering
+    source_url = sp[0].sourceFileUrls()[0]
+    print('Downloading %s version %s from %s' % (package, version, source_url))
+    url = urllib2.urlopen(source_url)
+    data = url.read()
+
+    target = path.join(tmp_dir, 'source_file')
+
+    with open(target, "wb") as p:
+        p.write(data)
+    check_call(['tar', '-xf', target], cwd=tmp_dir)
+
+    print('Keeping tests from obtained package')
+    package_source = filter((lambda x: x.startswith(package)),
+                            os.listdir(tmp_dir))
+    # Just let an exception be thrown if more than one match which means
+    # there's a problem somewhere
+    test_base_dir = path.join(tmp_dir, package_source[0], 'tests', 'autopilot')
+    test_dirs = filter((lambda x: path.isdir(path.join(test_base_dir, x))),
+                       os.listdir(test_base_dir))
+    for test_dir in test_dirs:
+        test_dir = path.join(test_base_dir, test_dir)
+        print('Moving %s to %s' % (test_dir, target_dir))
+        shutil.move(test_dir, target_dir)
+
+
+def get_bzr_tests(branch, revision, autopilot_dir, target_dir, sub_dir):
+>>>>>>> MERGE-SOURCE
     tmp_dir = tempfile.mkdtemp()
     atexit.register(cleanup, tmp_dir)
     print('Checking out %s to %s' % (branch, path.join(tmp_dir, 'work')))