Merge lp:~roadmr/capomastro/single-build into lp:capomastro

[Daniel Manrique]

Create archiveartifacts for non-rebuilt dependencies when building a project (LP: #1450070).

Fix some documentation and view issues.

Essentially this fixes the "partial dependency rebuilds" bug by ensuring that non-rebuilt dependencies have their artifacts properly linked (by creating a new archiveartifact pointing to the old artifact and build). It includes the code to fix this in projects.helpers as well as tests. Caveat: the tests only verify the scenario where a non-rebuilt dependency already had a known good build. They don't test the scenario where a dependency has never been built (which will result in an incomplete set of artifacts, but that's expected).

I also snuck in a few more changes:

Related to the partial dep debuild changes:
70a556e projects:helpers: Test for non-rebuilt dependency behavior
f92c230 projects:helpers: Document arguments for a confusing method
12c6b7b projects:helpers: Variable renaming in build_project for clarity.
bc5485a projects:helpers: non-rebuilt dependencies should have ArchiveArtifacts created from the dependency's last successful build

Two view fixes sent by Caio:
dabbdcc Added a textual description of what happens during the build process
9f6f044 Fix labels in project build form and template

Small documentation tweaks
b878458 docs: Clarify deb packages that need to be installed before running the next step

[Caio Begotti (caio1982)]

Just finished testing it all on Wendigo and it works as discussed and expected, even with the Swift storage.

review: Approve

=== modified file 'docs/source/overview.rst'
--- docs/source/overview.rst	2015-03-17 15:05:05 +0000
+++ docs/source/overview.rst	2015-06-17 17:54:20 +0000
@@ -57,3 +57,52 @@
 This shows the data model and application interdependencies.
 
 .. graphviz:: apps.dot
+
+Overview of what happens
+------------------------
+
+The main entities are the Project and the Dependency.
+
+A dependency maps directly to a Jenkins job which produces a set of artifacts.
+
+Each job is associated with a job type. The job type can be thought of as a "class" while the job itself is an "instance". Each "instance" of the job
+is linked to a dependency (one-to-one correspondence).
+
+Dependencies can be built on their own or as part of a project. We'll look at building a project which triggers its dependency builds first.
+
+Requesting a project build will::
+
+    * Create a ProjectBuild with data about this particular request.
+    * For each dependency associated with this project:
+        * If the dependency was marked to be built:
+          * Launch a Jenkins Build of the dependency.
+          * Create a ProjectBuildDependency linking:
+            * The Project
+            * The Dependency
+            * The particular Build of this dependency (from Jenkins)
+        * If the dependency was NOT marked to be built (use old one):
+          * Create a ProjectBuildDependency linking:
+            * This ProjectBuild (it implicits a link to the Project)
+            * This Dependency
+            * The *PREVIOUS* existing Build of this dependency (from Jenkins)
+          * Look at the old Build
+            * For each Artifact linked to the old Build (X):
+              * create an ArchiveArtifact (Y) linking:
+                * This Dependency (can get from the PBD?)
+                * This ProjectBuildDependency
+                * This Build (can get it from the Artifact?)
+                * This Artifact
+                * The current Archive
+              * schedule a link_artifact_in_archive task with the ArchiveArtifact from the OLD build (X) as source and the ArchiveArtifact from the NEW build (Y) as destination.
+    * Wait for notification from Jenkins about completed builds. When one is received:
+      * For each artifact in the build as notified by Jenkins, create an Artifact item, link it to the Build. Then get the current Archive and ask it to archive the Artifact. This creates an ArchiveArtifact record linked to:
+        * The ProjectBuildDependency
+        * the Artifact (from the Jenkins side of things).
+        * the Dependency (is this redundant? we have this from PBD)
+        * The Archive itself
+        * Note: There's a link from Build to Artifact and from Build to ArchiveArtifact, but Artifact could be used as intermediate table and remove the ArchiveArtifact -> Build link.
+
+
+Requesting just a dependency build will::
+
+    * We don't know yet - this is to be researched.

=== modified file 'docs/source/setup.rst'
--- docs/source/setup.rst	2015-06-02 18:24:53 +0000
+++ docs/source/setup.rst	2015-06-17 17:54:20 +0000
@@ -11,6 +11,20 @@
 * RabbitMQ, for message handling
 * Celery, tasks management
 
+Prerequisites
+-------------
+
+Add this repository (for jenkins)::
+
+    sudo apt-get install wget
+    wget -q -O - https://jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
+    sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'
+    sudo apt-get update
+
+Packages to install::
+
+    sudo apt-get install rabbitmq-server postgresql-9.3 libpq-dev tox python-dev libffi-dev python-cffi jenkins
+
 Environment
 -----------
 

=== modified file 'projects/forms.py'
--- projects/forms.py	2015-05-06 16:49:56 +0000
+++ projects/forms.py	2015-06-17 17:54:20 +0000
@@ -81,5 +81,5 @@
     interval = forms.ModelChoiceField(label='When to build',
                                       widget=forms.RadioSelect(),
                                       queryset=IntervalSchedule.objects.all(),
-                                      empty_label="Now",
+                                      empty_label="now",
                                       required=False)

=== modified file 'projects/helpers.py'
--- projects/helpers.py	2015-04-15 21:55:45 +0000
+++ projects/helpers.py	2015-06-17 17:54:20 +0000
@@ -1,9 +1,11 @@
 import json
 
+from archives.models import ArchiveArtifact
+from archives.tasks import link_artifact_in_archive
+from djcelery.models import PeriodicTask
 from jenkins.tasks import build_job
 from jenkins.models import Build
 from projects.models import Project, ProjectDependency
-from djcelery.models import PeriodicTask
 
 def build_dependency(dependency, build_id=None, user=None):
     """
@@ -46,8 +48,8 @@
     if automated:
         options["phase"] = Build.FINALIZED
 
-    previous_build = project.get_current_projectbuild()
-    build = ProjectBuild.objects.create(**options)
+    previous_project_build = project.get_current_projectbuild()
+    project_build = ProjectBuild.objects.create(**options)
 
     if dependencies:
         filter_args = {"dependency__in": dependencies}
@@ -62,12 +64,14 @@
     if not automated:
         for dependency in dependencies_to_build.order_by(
                 "dependency__job__pk"):
-            kwargs = {"projectbuild": build,
+            kwargs = {"projectbuild": project_build,
                       "dependency": dependency.dependency}
             ProjectBuildDependency.objects.create(**kwargs)
             if queue_build:
                 build_dependency(
-                    dependency.dependency, build_id=build.build_key, user=user)
+                    dependency.dependency,
+                    build_id=project_build.build_key,
+                    user=user)
 
     # If it's automated, then we create a ProjectBuildDependency for each
     # dependency of the project and prepopulate it with the last known build.
@@ -77,14 +81,68 @@
     else:
         remaining_builds = dependencies_not_to_build
 
+    # Walk each non-built dependency
     for dependency in remaining_builds:
+        # last_known_build is a jenkins.Build instance
         last_known_build = get_last_build_for_dependency(
-            dependency, previous_build)
-        kwargs = {"projectbuild": build,
+            dependency, previous_project_build)
+        kwargs = {"projectbuild": project_build,
                   "dependency": dependency.dependency,
                   "build": last_known_build}
-        ProjectBuildDependency.objects.create(**kwargs)
-    return build
+        # Link the new project build with this dependency but using the
+        # last known build. This is done using the intermediate
+        # ProjectBuildDependency table.
+        new_project_build_dependency = ProjectBuildDependency.objects.create(
+            **kwargs)
+        # For each archived artifact (i.e. artifact already on Capomastro's
+        # archiver, previously pulled from Jenkins), we need to create a
+        # "copy" of the ArchiveArtifact. The two differences are: it should
+        # be linked to the *new* project build dependency (so it's associated
+        # with the new project build rather than the old one), and its
+        # archived_path should match the one for the new project build.
+        #
+        # For this we look at the last known build's full set of artifacts,
+        # filtering by those whose last known build matches the one we
+        # identified earlier.
+        if not last_known_build:
+            continue
+        for archiveartifact in last_known_build.archiveartifact_set.filter(
+                projectbuild_dependency__build=last_known_build):
+            # The filename is one of the items that changes. Use the
+            # get_path_for_artifact method to get a filename matching the
+            # new ProjectBuildDependency's archival path.
+            policy = archiveartifact.archive.get_policy()
+            new_filename = policy.get_path_for_artifact(
+                archiveartifact.artifact,
+                build=archiveartifact.build, dependency=dependency.dependency,
+                projectbuild=new_project_build_dependency.projectbuild)
+            # Build the arguments for the new ArchiveArtifact. It's clear here
+            # that the two items that change are:
+            #     * archived_path as calculated above.
+            #     * projectbuild_dependency so it eventually links to the
+            #       new project build.
+            # FIXME: I'm lazy and used dependency.dependency, it would be
+            # better to use archiveartifact.dependency so we keep the "copy"
+            # scoped to the archiveartifact.
+            kwargs = {'archive': archiveartifact.archive,
+                      'artifact': archiveartifact.artifact,
+                      'archived_path': new_filename,
+                      'archived_size': archiveartifact.archived_size,
+                      'build': archiveartifact.build,
+                      'projectbuild_dependency': new_project_build_dependency,
+                      'dependency' : dependency.dependency}
+            # Maybe we already have an ArchiveArtifact with these parameters?
+            # if so, retrieve it.
+            # FIXME: I don't know why this could happen (multiple artifacts),
+            # need to research better.
+            newaa, created = ArchiveArtifact.objects.get_or_create(**kwargs)
+            # Finally dispatch a celery task to create the actual archive
+            # link from the old artifact to the new. This "syncs" the
+            # filesystem or archiver with the state that's stored in the
+            # database.
+            link_artifact_in_archive.delay(archiveartifact.pk, newaa.pk)
+
+    return project_build
 
 
 def get_last_build_for_dependency(dependency, previous_build=None):
@@ -92,6 +150,10 @@
     Return the last known build for the provided ProjectDependency dependency,
     which is defined as the current build associated with itself if it's not
     auto-tracked, or the most recent build for auto-tracked cases.
+
+    arguments:
+    dependency should be a ProjectDependency.
+    previous_build should be a ProjectBuild, or None.
     """
     if dependency.auto_track:
         if previous_build:

=== modified file 'projects/templates/projects/projectbuild_form.html'
--- projects/templates/projects/projectbuild_form.html	2015-04-16 16:36:49 +0000
+++ projects/templates/projects/projectbuild_form.html	2015-06-17 17:54:20 +0000
@@ -9,11 +9,11 @@
   <div class="row">
     <div>
       <h2>Build or Schedule Project {{ project.name }}</h2>
-      <h3>{{ project.name }}</h3>
       <p>{{ project.description|default:"" }}</p>
       {% if dependency_count > 0 %}
-        <h3>Selection</h3>
-        <p>Please select all the dependencies you want to build.</p>
+        <h3>Select dependencies to build</h3>
+	<p>Dependencies not selected will be skipped and will default to the current versions for this project.</p>
+	<p>Artifacts of skipped dependencies will be reprocessed for archival.</p>
         <form action="" method="post" class="form" id="buildproject-form">
           {% csrf_token %}
           {% bootstrap_form form %}
@@ -21,7 +21,6 @@
             <button type="submit" class="btn btn-primary">Build</button>
           {% endbuttons %}
         </form>
-        <p>Any dependencies not selected will not be built and will default to the current built versions for this project.</p>
       {% else %}
         <h3>Selection</h3>
         <p>Unable to select dependencies.</p>

=== modified file 'projects/tests/test_helpers.py'
--- projects/tests/test_helpers.py	2015-04-15 21:55:45 +0000
+++ projects/tests/test_helpers.py	2015-06-17 17:54:20 +0000
@@ -5,12 +5,14 @@
 import mock
 
 from djcelery.models import PERIOD_CHOICES, IntervalSchedule, PeriodicTask
+from jenkins.models import Build
 from projects.models import (
     ProjectBuild, ProjectDependency, ProjectBuildDependency)
 from projects.helpers import (
     build_project, build_dependency, project_for_periodic_task)
-from .factories import ProjectFactory, DependencyFactory
-from jenkins.tests.factories import BuildFactory
+from .factories import ProjectFactory, ProjectBuildFactory, DependencyFactory
+from archives.tests.factories import ArchiveArtifactFactory
+from jenkins.tests.factories import BuildFactory, ArtifactFactory
 
 
 class PeriodicTaskTest(TestCase):
@@ -220,6 +222,83 @@
             projectbuild=build2, dependency=dependency2)
         self.assertEqual(built_dependency2.build, built_dependency1.build)
 
+    def test_build_project_non_automated_partial_build(self):
+        """
+        When some of the dependencies are selected to be built, the final
+        ProjectBuild should have PBDs and ArchiveArtifacts for all
+        dependencies. The non-rebuilt ones should link to previous artifacts.
+        """
+        project = ProjectFactory.create()
+        dependency1 = DependencyFactory.create()
+        pd1 = ProjectDependency.objects.create(
+            project=project, dependency=dependency1)
+
+        dependency2 = DependencyFactory.create()
+        ProjectDependency.objects.create(
+            project=project, dependency=dependency2)
+
+        # When a project is built a ProjectBuildDependency appears.
+        # We only care about dependency1 as dependency2 will be built in
+        # the second build.
+        first_build = ProjectBuildFactory.create(project=project)
+        # An artifact appears and is archived
+        artifact = ArtifactFactory.create()
+        archive_artifact = ArchiveArtifactFactory.create(artifact=artifact)
+        archive_artifact.build = artifact.build  # Bad normalization!
+        archive_artifact.dependency = dependency1  # Bad normalization!
+        archive_artifact.save()
+        archive_artifact.build.projectdependency_set.add(pd1)
+        # Link the artifact to the projectbuilddependency
+        first_pbd = ProjectBuildDependency.objects.create(
+            dependency=dependency1,
+            build=archive_artifact.build,
+            projectbuild=first_build)
+        archive_artifact.projectbuild_dependency = first_pbd
+        archive_artifact.save()
+        # Mark the project build as successfully finalized
+        first_pbd.projectbuild.phase = Build.FINALIZED
+        first_pbd.projectbuild.status = "SUCCESS"
+        first_pbd.projectbuild.save()
+
+        # Now try the new build
+        with mock.patch("projects.helpers.build_job") as mock_build_job, mock.patch("projects.helpers.link_artifact_in_archive") as mock_link_artifact:
+            new_build = build_project(project, dependencies=[dependency2],
+                                      queue_build=True)
+            self.assertIsInstance(new_build, ProjectBuild)
+
+        # Verification time
+        build_dependencies = ProjectBuildDependency.objects.filter(
+            projectbuild=new_build)
+        # New build should still have 2 build_dependencies.
+        self.assertEqual(2, build_dependencies.count())
+        # We only called build_job for dependency2, as dependency1 was already
+        # built and was just re-linked
+        mock_build_job.delay.assert_has_calls(
+            [mock.call(dependency2.job.pk, build_id=new_build.build_key)])
+        # Ensure both dependencies are in the new build, even though only one
+        # of them was explicitly rebuilt.
+        self.assertEqual(
+            sorted([dependency1.pk, dependency2.pk]),
+            sorted(list(build_dependencies.values_list("dependency",
+                                                       flat=True))))
+        # Ensure the artifacts and all for the non-explicitly-built dep are
+        # what we want.
+        dep1_build = build_dependencies.filter(dependency=dependency1)[0]
+        self.assertEqual(dep1_build.projectbuild, new_build)
+        # The build should be the same as the existing archiveartifact's build
+        self.assertEqual(dep1_build.build, archive_artifact.build)
+        # The archiveartifact_set should contain one item...
+        self.assertEqual(1, len(dep1_build.archiveartifact_set.all()))
+        # And it should NOT be the old archiveartifact.
+        self.assertNotIn(archive_artifact,
+                         dep1_build.archiveartifact_set.all())
+        # Now let's get the new archiveartifact and ensure we called
+        # link_artifacts_in_archive
+        new_archive_artifact = dep1_build.archiveartifact_set.all()[0]
+
+        mock_link_artifact.delay.assert_has_calls(
+            [mock.call(archive_artifact.pk, new_archive_artifact.pk)])
+
 
 class BuildDependencyTest(TestCase):