Merge lp:~canonical-ci-engineering/ubuntu-ci-services-itself/pm into lp:ubuntu-ci-services-itself

[Ursula Junque]

Adding the design, modeling and beginning API calls for Houston.

Adding the design, modelling and part of the REST API calls for Houston (former project manager) phase 0.

Design section explains the approach to process source packages only.

NOTE: Nicely formatted page: http://162.213.34.2/rtd/components/houston.html

[Vincent Ladeuil (vila)]

Overall: sounds great but already a bit complex, I wonder if we can simplify things a bit.

Some details:

24 +* Runs as a persistent service, when it goes down, the engine halts.
25 +* State needs to serialized whenever it is updated, so that status can be regenerated on a restart.

+1

57 +* PPA Assigner - Provides two PPAs to perform package builds, one for MPs themselves, another for MPs + trunk.

Isn't phase 0 using a single PPA only ?

60 +* Integration Test Runner - Runs the specified tests on up to 3 images.
61 +

Just 'Test Runner', the 'Integration' is silent ;)

80 + package: chris-foobar-pkg
81 + version: 1.1-0ubuntu2
82 + source: chris-foobar-src

Is there a strong reason to use ':' or can we use '=' instead ?

I don't want to bikeshed there so if you feel strongly about, go for
it. Otherwise, this is configuration data and I'd rather use '=' to make
sure we keep it simple and don't turn it into a full json that only experts
can decipher ;)

233 + WORKFLOW_STEPS = (
239 + WORKFLOW_STEP_STATUSES = (

I don't get why those are split, aren't them all just states for the ticket ?

1 === added file 'docs/components/houston.rst'
380 === removed file 'docs/components/project-manager.rst'

O.M.G. !

This is a sin, please use 'bzr mv docs/components/project-manger.rst docs/components/houston.rst' !

A kitten dies everytime you do remove+add ;-)

More seriously, bzr supports renames, use that feature, it preserves
history, in this specific case it would have more clearly show what you did
change in this proposal.

....

Ha right, that was a rewrite on the side and the remove was revno 27.

Ok, the kitten survived after all ;)

Summary, no need to block on the questions asked above, the answers can come in a followup, this is significant progress, let's land it.

review: Approve

[Francis Ginther (fginther)]

In general this is all good for phase 0, I don't see anything missing.

Just some small nit-picking:
 - For "submitting a file to project manager" would it be easier to state the format needed (yaml, json, etc.) and the required keys?
 - The Interactions/Ticket System reflects the older design. Might just want to focus on the Phase 0 plan of interacting with the Lander and the UI.
 - I can't think of an immediate need to have multiple users defined for phase 0. But I can see the potential need for having at least a default user defined.

As these don't appear to directly impact the design, I'll approve.

review: Approve

[Chris Johnston (cjohnston)]

Attempt to merge into lp:ubuntu-ci-services-itself failed due to conflicts:

duplicate in docs/components/ticket-system.rst.moved

[Vincent Ladeuil (vila)]

> duplicate in docs/components/ticket-system.rst.moved

Probably caused by someone modifying that file on trunk since you branched, you need to merge locally, resolve the conflict and push again.

[Chris Johnston (cjohnston)]

This was superseded by another MP

Unmerged revisions

35. By Ursula Junque on 2013-12-09

Changing references from houston to ticket system

34. By Ursula Junque on 2013-12-09

Renaming houston to ticket system

33. By Chris Johnston on 2013-12-06

Fix build error

32. By Ursula Junque on 2013-12-06

Added set_subticket_status

31. By Ursula Junque on 2013-12-06

Added 'Not Started' status to workflow steps

30. By Ursula Junque on 2013-12-06

More formatting

29. By Ursula Junque on 2013-12-06

REST API in same title level as other to show in planned page

28. By Ursula Junque on 2013-12-06

API formatting

27. By Ursula Junque on 2013-12-06

removing ticket system as it's inside houston now

26. By Ursula Junque on 2013-12-06

s/project-manager/houston/

=== modified file 'docs/components/planned.rst'
--- docs/components/planned.rst	2013-12-06 15:26:40 +0000
+++ docs/components/planned.rst	2013-12-09 17:50:28 +0000
@@ -4,7 +4,7 @@
 .. toctree::
    :maxdepth: 2
 
-   project-manager
+   houston
    ticket-manager
    web-server
    branch-listener

=== removed file 'docs/components/project-manager.rst'
--- docs/components/project-manager.rst	2013-11-25 23:32:31 +0000
+++ docs/components/project-manager.rst	1970-01-01 00:00:00 +0000
@@ -1,41 +0,0 @@
-Project Manager
-===============
-
-.. toctree::
-   :maxdepth: 2
-
-   ticket-system
-   test-database
-
-*Purpose:*
-
-* Brains of the CI system. Contains project centric knowledge such as location of trunk names of integration tests, point of contact, etc. Also knows status of current MPs being built and a queue of MPs to process.
-* The project manager maintains the following sets of data:
-  * The history and status of tickets.
-  * The definition and configuration of individual projects.
-  * The test database.
-
-*Deployment:*
-
-* Can run as a juju service.
-* Runs as a persistent service, when it goes down, the engine halts.
-* State needs to serialized whenever it is updated, so that status can be regenerated on a restart.
-* Public facing UI is provided by the Web Server.
-
-*Interactions:*
-
-* Ticket Manager - Requests project metadata and provides status updates on tickets active in the CI system.
-* Landing Manager - Requests project metadata and provides updates on status of a branch through the CI system.
-* Web Server - Provides the UI.
-
-*Development Plan:*
-
-* Define API.
-* Define backend ticket storage.
-* Define backend project specification storage.
-* Define backend test database storage.
-* Integrate with the Web Server.
-* Integrate with the Landing Manager, supply metadata to build and test a single project and generate an automatic low-cost ticket.
-* Handle status updates from the Landing Manager.
-* Integrate with the Ticket Manager to supply metadata to establish a ticket workflow with multiple projects and tests.
-* Handle status updates from the Ticket Manager.

=== removed file 'docs/components/test-database.rst'
--- docs/components/test-database.rst	2013-11-16 10:12:08 +0000
+++ docs/components/test-database.rst	1970-01-01 00:00:00 +0000
@@ -1,16 +0,0 @@
-Tests Database (sub-function of Project Manager)
-================================================
-
-*Purpose:*
-
-* Repository of tests per project. (What else?)
-
-*Deployment:*
-
-* Built into the Project Manager
-
-*Interactions:*
-
-* Ticket System - Supplies available and default test suites.
-* Integration Test Service - Supplies test provisioning and execution details.
-

=== added file 'docs/components/ticket-system.rst'
--- docs/components/ticket-system.rst	1970-01-01 00:00:00 +0000
+++ docs/components/ticket-system.rst	2013-12-09 17:50:28 +0000
@@ -0,0 +1,361 @@
+Ticket System (former Project Manager)
+==================================================
+
+
+Purpose
+*******
+
+The Ticket System contains project centric knowledge such as location of trunk names of integration tests, point of contact, etc. as well as the ticketing management for the CI Engine which includes the status of current MPs being built and a queue of MPs to process.
+
+It maintains the following sets of data:
+
+* The history and status of tickets (the ticket tracker).
+* The definition and configuration of projects, their branches, source packages and produced binary packages (the project manager).
+* The test database. (This will not be part of phase 0)
+
+Deployment
+**********
+
+* Can run as a juju service.
+* Runs as a persistent service, when it goes down, the engine halts.
+* State needs to serialized whenever it is updated, so that status can be regenerated on a restart.
+
+Development Plan
+****************
+
+Phase 0
+-------
+
+* Define API.
+* Define backend ticket storage.
+* Define backend project specification storage.
+* Handle status updates from the Landing Manager.
+* Provide read API for the Web Server.
+
+Future
+------
+
+* Integrate with the Landing Manager, supply metadata to build and test a single project and generate an automatic low-cost ticket (phase 0 will accept only source package uploads directly, the branch polling for new merges and creation of automatic tickets is in the future.)
+* Define backend test database storage, and if it should be a separated system component.
+
+Interactions
+************
+
+Project Manager
+---------------
+
+* Provide API for any system component that needs information about a project/branch/merge proposal/source package/binary package.
+
+Ticket System
+-------------
+
+* Provide API for Web UI
+* PPA Assigner - Provides two PPAs to perform package builds, one for MPs themselves, another for MPs + trunk.
+* Branch/Source Builder - MP branches are dispatched to the builder.
+* Image Builder - Generates up to 3 images to be used for testing: feature branches + base image, feature branches + latest image, feature branches + trunk + latest image.
+* Integration Test Runner - Runs the specified tests on up to 3 images.
+
+
+Design
+******
+
+Project Manager
+---------------
+
+In phase 0 the project manager will store and manage information about:
+
+ * Source package upload: component that will be tracked by a subticket. Is a direct source package upload.
+ * Source package: a source package as found in the archive.
+
+We'll be provided a list of binary packages that should be monitored and tested, and a script outside project manager will generate the respective sources list.
+
+This data will be updated by submitting a file to project manager, that should contain the binary name, source package name and version. Entries are separated by a newline. As below:
+
+::
+
+  package: chris-foobar-pkg
+  version: 1.1-0ubuntu2
+  source: chris-foobar-src
+
+  ...
+
+
+For next phases, we'll extend it to store info about:
+
+ * Binary package: a binary package as found in the archive, produced by a known source package. 
+ * Upstream project: a launchpad project, that might have branches.
+ * Bazaar branches: launchpad bazaar branches, might be associated with a source package. A branch can produce only one source package at a time.
+ * Merge proposal: a launchpad merge proposal, that will be tracked by a subticket.
+ 
+
+
+Ticket System
+-------------
+
+Tickets and subtickets
+~~~~~~~~~~~~~~~~~~~~~~
+
+A ticket is a way of grouping parts of a bug fix or feature.
+In the ticket you'll describe the objective, add relevant information like
+blueprints or other references and the ticket status will reflect the status
+of its subtickets. Other system components will rely on the ticket status to take actions.
+
+A subticket refers to a source package upload or a merge proposal.
+
+When a developer submits the package upload request, the ticket system creates a subticket for each uploaded package and/or merge proposal, and a ticket that contains all of them.
+
+In phase 0, it won't be possible to add more subtickets to a ticket after ticket is created (no editing). To create a ticket with several parts, the developer needs to submit all of them at once.
+If a subticket fails, the developer needs to submit another request that will generate another ticket with a subticket.
+
+The ticket and subtickets will keep only the current step of the workflow they are and their status, as defined in ticket and subticket models.
+
+In phase 1, if the subticket processing fails, the subticket is closed as failed, and the developer can submit another source package upload to the same ticket. It'll also be possible to add merge proposal subtickets.
+
+
+Ticket System features
+~~~~~~~~~~~~~~~~~~~~~~
+
+Create, monitor and manage ticket related activities
+
+* Define and create a ticket including:
+
+ * describe the targeted feature
+ * Links to any bugs/BPs/other design docs that are related
+ * People/teams working/owning on the ticket
+ * Components involved (packages/PPAs/feature branches/infrastructure required to code/test/build/deploy)
+ * PPA(s) dedicated to the ticket
+ * Describe the test suites that need to be run (smarts can be added in the long run to assist users in choosing the Right test to run). In phase 0 it consists on the dep8 tests in all source packages that are relevant.
+ * Knowing what image is supposed to be ‘frozen’ for developing against
+
+* Provides test and status of a ticket:
+
+ * Built images in the image store
+ * Status of the ticket as it progresses through the CI engine
+ * Clear indication of merge/build/test failures and where the failure occurred, make it obvious to developers what a corrective course of action should be
+ * Provide a means to execute a landing when all criteria have passed
+
+* Provides ticket management:
+
+ * list and status of all tickets
+
+
+REST APIs
+*********
+
+Tickets
+-------
+
+list
+~~~~
+
+List all open tickets.
+
+create_ticket
+~~~~~~~~~~~~~
+
+Create a ticket from one or more source package uploads.
+
+get_ticket
+~~~~~~~~~~
+
+Return the ticket given its id.
+
+get_ticket_status
+~~~~~~~~~~~~~~~~~
+
+Get the ticket status.
+
+set_ticket_status
+~~~~~~~~~~~~~~~~~
+
+Set the ticket status.
+
+set_subticket_status
+~~~~~~~~~~~~~~~~~~~~
+
+Set a subticket status, given the ticket number and the source package name.
+
+Person
+------
+
+add_person
+~~~~~~~~~~
+
+Add a person to the database.
+
+get_person
+~~~~~~~~~~
+
+Return a person from the database, given it's nickname/email.
+
+Models
+******
+
+Ticket System will be django + REST + Postgres. It will have apps and data models similar to the following.
+
+Phase 0
+-------
+
+sourcepackage
+~~~~~~~~~~~~~
+
+::
+
+ class SourcePackage(models.Model):
+    name = models.CharField(max_length=4096)
+
+sourcepackageupload
+~~~~~~~~~~~~~~~~~~~
+
+::
+ 
+ class SourcePackageUpload(models.Model):
+    sourcepackage = models.ForeignKey("SourcePackage")
+    version = models.CharField(max_length=4096)
+    date = models.DateTimeField()
+    artifact = models.ForeignKey("Artifact")  # Provided by the artifact manager
+
+
+ticket
+~~~~~~
+
+A ticket can contain only one active subticket per source package. This
+way we can reference the ticket externally by source package name and ticket
+number.
+
+::
+
+ class Ticket(models.Model):
+    WORKFLOW_STEPS = (
+        (PACKAGE_BUILDING, "Package Building"),
+        (IMAGE_BUILDING, "Image Building"),
+        (IMAGE_TESTING, "Image Testing"),
+        (PACKAGE_PUBLISHING, "Package Publishing"),
+    )
+    WORKFLOW_STEP_STATUSES = (
+        (PKG_BUILD_WAITING, "Not Started"),
+        (PKG_BUILD_INPROGRESS, "In Progress"),
+        (PKG_BUILD_DONE, "Completed"),
+        (PKG_BUILD_FAILED, "Failed"),
+        (IMAGE_BUILD_WAITING, "Not Started"),
+        (IMAGE_BUILD_INPROGRESS, "In Progress"),
+        (IMAGE_BUILD_DONE, "Completed"),
+        (IMAGE_BUILD_FAILED, "Failed"),
+        (IMAGE_TESTS_WAITING, "Not Started"),
+        (IMAGE_TESTS_INPROGRESS, "In Progress"),
+        (IMAGE_TESTS_PASSED, "Passed"),
+        (IMAGE_TESTS_FAILED, "Failed"),
+        (PKG_PUBLISHING_DONE, "Completed"),
+        (PKG_PUBLISHING_FAILED, "Failed"),
+    )
+    owner = models.ForeignKey("Person")
+    title = models.CharField(max_length=4096)
+    description = models.TextField()
+    bug_id = models.IntegerField(null=True, blank=True)
+    current_workflow_step = models.CharField(choices=WORKFLOW_STEPS)
+    status = models.CharField(choices=WORKFLOW_STEP_STATUSES)
+    base_image = models.CharField(max_length=20)
+
+subticket
+~~~~~~~~~
+
+::
+
+ class SubTicket(models.Model):
+    WORKFLOW_STEPS = (
+        (PACKAGE_BUILDING, "Package Building"),
+    )
+    WORKFLOW_STEP_STATUSES = (
+        (PKG_BUILD_WAITING, "Not Started"),
+        (PKG_BUILD_INPROGRESS, "In Progress"),
+        (PKG_BUILD_DONE, "Completed"),
+        (PKG_BUILD_FAILED, "Failed"),
+    )
+    current_workflow_step = models.CharField(choices=WORKFLOW_STEPS)
+    status = models.CharField(choices=WORKFLOW_STEP_STATUSES)
+    ticket = models.ForeignKey("Ticket")
+    assignee = models.ForeignKey("Person") 
+    sourcepackageupload = models.ForeignKey("SourcePackageUpload")
+
+
+artifact
+~~~~~~~~
+
+::
+ 
+ class Artifact(models.Model):
+    type = models.CharField(choices=["test_result", "source_upload", "test_failure_log"]
+    ticket_component = models.ForeignKey("TicketComponent")
+    reference = models.CharField(max_length=4096)  # Provided by the artifact manager
+
+person
+~~~~~~
+
+::
+
+ class Person(models.Model):
+    name = models.CharField(max_length=4096)
+    email = models.EmailField(max_length=200)
+    is_team = models.BooleanField(default=False)
+
+
+Future Apps/Models
+------------------
+
+testsuites
+~~~~~~~~~~
+
+::
+
+ class TestSuites(models.Model):
+    pass
+
+
+project
+~~~~~~~
+
+::
+
+ class Project(models.Model):
+    # Class that defines an upstream project.
+    name = models.CharField(max_length=4096)
+    display_name = models.CharField(max_length=4096)
+    maintainer = models.ForeignKey("Person")
+    contact = models.ForeignKey("Person")
+    description = models.TextField()
+
+branch
+~~~~~~
+
+::
+
+ class Branch(models.Model):
+    unique_name = models.CharField(max_length=4096)
+    owner = models.ForeignKey("Person")
+    type = models.CharField(choice=["trunk", "development", "maintenance", "regular"])
+    project = models.ForeignKey("Project", null=True, blank=True)
+
+
+binarypackage
+~~~~~~~~~~~~~
+
+::
+
+ class BinaryPackage(models.Model):
+    name = models.CharField(max_length=4096)
+    sourcepackage = models.ForeignKey("SourcePackage")
+    seeded = models.BooleanField(default=False)
+    
+
+mergeproposal
+~~~~~~~~~~~~~
+
+::
+
+ class MergeProposal(models.Model):
+     branch = models.ForeignKey("Branch")
+     submitter = models.ForeignKey("Person")
+     status = models.CharField(choices=["work_in_progress", "needs_review", "approved", "rejected", "merged"])
+     lp_weblink = models.CharField(max_length=4096)
+
+

=== renamed file 'docs/components/ticket-system.rst' => 'docs/components/ticket-system.rst.moved'