Merge lp:~fginther/ubuntu-ci-services-itself/bsb-dput-3 into lp:ubuntu-ci-services-itself

[Francis Ginther]

Adds the dput and ppa_watch functionality to the branch source builder. This is very raw, most of the functionality was pulled directly from lp:cupstream2distro with small changes related to the input parameters and the deployment environment.

Adds the dput and ppa_watch functionality to the branch source builder. This is very raw, most of the functionality was pulled directly from lp:cupstream2distro with small changes related to the input parameters and the deployment environment.

The series and archive PPA are currently hard coded. This will be removed by a subsequent MP.

You'll also notice no tests here. The cupstream2distro code also has no tests for these modules. I need to address this. For now, I'm banking on the fact that this code is already consumed by the daily-release process.

[Francis Ginther (fginther)]

This is a follow up to bsb-dput-2. I screwed up the branch and had to resubmit the whole thing.

> I'm skipping the cupstream2distro code since its mostly copy/paste.
>
> = run-worker:
>
> 2050 + if amqp_utils.progress_completed(trigger):
> 2051 + log.error('Unable to notify progress-trigger completion of action')
>
> the progress_completed function already logs an error, so you really
> only need to check the return code for it if you are going to handle
> failure in some special way. Since we don't here, I'd delete the
> log.error line.

Fixed.

> You also need to supply a payload to the "progress_completed" call. The
> data returned from this function winds up being the data that will be
> passed to the image builder. I think it needs to contain:
>
> {
> "ppa": "<foo>"
> }
>
> so we can remove the hack from:
>
> <http://bazaar.launchpad.net/~canonical-ci-engineering/ubuntu-ci-services-
> itself/trunk/view/head:/charms/precise/lander-
> jenkins/templates/jobs/lander_master.xml#L170>

Fixed, good catch on removing the hack.

> 2052 + except EnvironmentError as e:
>
> I think I'd just catch "Exception" here. That way nothing slips through
> in a way where we won't call amqp_utils.progress_failed

Fixed

> = upload_package:
>
> 2141 + file_name = source.split('/')[-1]
>
> I think you can just os.path.basename to make it more obvious what your
> intent is:
>
> os.path.basename(source)

Fixed, I also removed the hack to remove the container name from the filename (the data-store module has been updated to just use the filename).

> 2186 + logging.basicConfig(level=logging.INFO, format="%(asctime)s
> %(levelname)s "
> 2187 + "%(message)s")
>
> I think that should be moved to "main" instead. Otherwise you might just
> override the logging config of the process that's using this API.

Fixed, also fixed in watch_ppa.py

review: Needs Resubmitting

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:195
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/142/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/142/rebuild

review: Approve (continuous-integration)

[Andy Doan (doanac)]

the most exciting thing i've seen today!

I saw we merge this and then start testing it hard. I noticed one small thing:

2035 + archive_ppa = 'ppa:ci-engineering-airline/ci-archive'
2036 + series = 'saucy'

once this is merged into trunk, I think you can pick up some work Chris added today:

  http://bazaar.launchpad.net/~canonical-ci-engineering/ubuntu-ci-services-itself/trunk/revision/231

[Francis Ginther (fginther)]

> the most exciting thing i've seen today!
>
> I saw we merge this and then start testing it hard. I noticed one small thing:
>
> 2035 + archive_ppa = 'ppa:ci-engineering-airline/ci-archive'
> 2036 + series = 'saucy'
>
> once this is merged into trunk, I think you can pick up some work Chris added
> today:
>
> http://bazaar.launchpad.net/~canonical-ci-engineering/ubuntu-ci-services-
> itself/trunk/revision/231

Right, I also noticed the ticket system change today, but didn't want to further delay this MP.

[Andy Doan (doanac)]

review: Approve

=== added directory 'branch-source-builder/cupstream2distro'
=== added file 'branch-source-builder/cupstream2distro/__init__.py'
=== added file 'branch-source-builder/cupstream2distro/branchhandling.py'
--- branch-source-builder/cupstream2distro/branchhandling.py	1970-01-01 00:00:00 +0000
+++ branch-source-builder/cupstream2distro/branchhandling.py	2014-02-11 20:59:14 +0000
@@ -0,0 +1,283 @@
+# -*- coding: utf-8 -*-
+# Copyright (C) 2012 Canonical
+#
+# Authors:
+#  Didier Roche
+#
+# This program is free software; you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free Software
+# Foundation; version 3.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+import ConfigParser
+from collections import defaultdict
+import logging
+import os
+import re
+import subprocess
+
+from .settings import BRANCH_URL, IGNORECHANGELOG_COMMIT, PACKAGING_MERGE_COMMIT_MESSAGE, PROJECT_CONFIG_SUFFIX, SILO_PACKAGING_RELEASE_COMMIT_MESSAGE
+
+
+def get_branch(branch_url, dest_dir):
+    '''Grab a branch'''
+    instance = subprocess.Popen(["bzr", "branch", branch_url, dest_dir], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (stdout, stderr) = instance.communicate()
+    if instance.returncode != 0:
+        raise Exception(stderr.decode("utf-8").strip())
+
+
+def get_tip_bzr_revision():
+    '''Get latest revision in bzr'''
+    instance = subprocess.Popen(["bzr", "log", "-c", "-1", "--line"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (stdout, stderr) = instance.communicate()
+    if instance.returncode != 0:
+        raise Exception(stderr.decode("utf-8").strip())
+    return (int(stdout.split(':')[0]))
+
+
+def collect_author_commits(content_to_parse, bugs_to_skip, additional_stamp=""):
+    '''return a tuple of a dict with authors and commits message from the content to parse
+
+    bugs_to_skip is a set of bugs we need to skip
+
+    Form: ({Author: [commit_message]}, set(bugs))'''
+
+    author_commit = defaultdict(list)
+    all_bugs = set()
+
+    current_authors = set()
+    current_commit = ""
+    current_bugs = set()
+    commit_message_stenza = False
+    for line in content_to_parse.splitlines():
+        # new revision, collect what we have found
+        if line.startswith("------------------------------------------------------------"):
+            # try to decipher a special case: we have some commits which were already in bugs_to_skip,
+            # so we eliminate them.
+            # Also ignore when having IGNORECHANGELOG_COMMIT
+            if (current_bugs and not (current_bugs - bugs_to_skip)) or IGNORECHANGELOG_COMMIT in current_commit:
+                current_authors = set()
+                current_commit = ""
+                current_bugs = set()
+                continue
+            current_bugs -= bugs_to_skip
+            commit_message = current_commit + _format_bugs(current_bugs)
+            for author in current_authors:
+                if not author.startswith("Launchpad "):
+                    author_commit[author].append(commit_message)
+            all_bugs = all_bugs.union(current_bugs)
+            current_authors = set()
+            current_commit = ""
+            current_bugs = set()
+
+        # we ignore this commit if we have a changelog provided as part of the diff
+        if line.startswith("=== modified file 'debian/changelog'"):
+            current_authors = set()
+            current_commit = ""
+            current_bugs = set()
+
+        if line.startswith("author: "):
+            current_authors = _extract_authors(line[8:])
+        # if direct commit to trunk
+        elif not current_authors and line.startswith("committer: "):
+            current_authors = _extract_authors(line[11:])
+        # file the commit message log
+        elif commit_message_stenza:
+            if line.startswith("diff:"):
+                commit_message_stenza = False
+                current_commit, current_bugs = _extract_commit_bugs(current_commit, additional_stamp)
+            else:
+                line = line[2:] # Dedent the message provided by bzr
+                if line[0:2] in ('* ', '- '): # paragraph line.
+                    line = line[2:] # Remove bullet
+                    if line[-1] != '.': # Grammar nazi...
+                        line += '.' # ... or the lines will be merged.
+                line = line + ' ' # Add a space to preserve lines
+                current_commit += line
+                #  Maybe add something like that
+                #for content in mp.commit_message.split('\n'):
+                #    content = content.strip()
+                #    if content.startswith('-') or content.startswith('*'):
+                #        content = content[1:]
+                #    description_content.append(content.strip())
+        elif line.startswith("message:"):
+            commit_message_stenza = True
+        elif line.startswith("fixes bug: "):
+            current_bugs = current_bugs.union(_return_bugs(line[11:]))
+
+    return (dict(author_commit), all_bugs)
+
+
+def _format_bugs(bugs):
+    '''Format a list of launchpad bugs.'''
+    if bugs:
+        msg = ' (LP: {})'.format(', '.join(['#{}'.format(b) for b in bugs]))
+    else:
+        msg = ''
+    return msg
+
+
+def _extract_commit_bugs(commit_message, additional_stamp=""):
+    '''extract relevant commit message part and bugs number from a commit message'''
+
+    current_bugs = _return_bugs(commit_message)
+    changelog_content = " ".join(commit_message.rsplit('Fixes: ')[0].rsplit('Approved by ')[0].split())
+    if additional_stamp:
+        changelog_content = changelog_content + " " + additional_stamp
+    return (changelog_content, current_bugs)
+
+
+def _return_bugs(string):
+    '''return a set of bugs from string'''
+
+    # we are trying to match in the commit message:
+    # bug #12345, bug#12345, bug12345, bug 12345
+    # lp: #12345, lp:#12345, lp:12345, lp: 12345
+    # lp #12345, lp#12345, lp12345, lp 12345,
+    # Fix #12345, Fix 12345, Fix: 12345, Fix12345, Fix: #12345,
+    # Fixes #12345, Fixes 12345, Fixes: 12345, Fixes:12345, Fixes: #12345
+    # *launchpad.net/bugs/1234567890 and
+    # #12345 (but not 12345 for false positive)
+    # Support multiple bugs per commit
+    bug_numbers = set()
+    bug_regexp = re.compile("((lp|bug|fix(es)?)[: #]*|#|launchpad.net/bugs/)(\d{5,})", re.IGNORECASE)
+    for match in bug_regexp.findall(string):
+        logging.debug("Bug regexp match: {}".format(match[-1]))
+        bug_numbers.add(int(match[-1]))
+    return bug_numbers
+
+
+def _extract_authors(string):
+    '''return a authors set from string, ignoring emails'''
+
+    authors = set()
+    for author_with_mail in string.split(", "):
+        author = author_with_mail.rsplit(' <')[0]
+        logging.debug("Found {} as author".format(author))
+        authors.add(author)
+    return authors
+
+
+def return_log_diff(starting_rev):
+    '''Return the relevant part of the cvs log since starting_rev'''
+
+    instance = subprocess.Popen(["bzr", "log", "-r", "{}..".format(starting_rev), "--show-diff", "--forward"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (stdout, stderr) = instance.communicate()
+    if instance.returncode != 0:
+        raise Exception(stderr.decode("utf-8").strip())
+    return stdout
+
+
+def return_log_diff_since_last_release(content_to_parse):
+    '''From a bzr log content, return only the log diff since the latest release'''
+    after_release = content_to_parse.split(SILO_PACKAGING_RELEASE_COMMIT_MESSAGE.format(''))[-1]
+    sep = '------------------------------------------------------------'
+    sep_index = after_release.find(sep)
+    if sep_index != 1:
+        after_release = after_release[sep_index:]
+    return after_release
+
+
+def commit_release(new_package_version, tip_bzr_rev=None):
+    '''Commit latest release'''
+    if not tip_bzr_rev:
+        message = SILO_PACKAGING_RELEASE_COMMIT_MESSAGE.format(new_package_version)
+    else:
+        message = "Releasing {}, based on r{}".format(new_package_version, tip_bzr_rev)
+    if subprocess.call(["bzr", "commit", "-m", message]) != 0:
+        raise Exception("The above command returned an error.")
+
+
+def _get_parent_branch(source_package_name):
+    '''Get parent branch from config'''
+    config = ConfigParser.RawConfigParser()
+    config.read("{}.{}".format(source_package_name, PROJECT_CONFIG_SUFFIX))
+    return config.get('Branch', 'branch')
+
+
+def propose_branch_for_merging(source_package_name, version, tip_rev, branch):
+    '''Propose and commit a branch upstream'''
+
+    parent_branch = _get_parent_branch(source_package_name)
+    # suppress browser opening
+    env = os.environ.copy()
+    env["BROWSER"] = "echo"
+    env["BZR_EDITOR"] = "echo"
+
+    os.chdir(source_package_name)
+    if subprocess.call(["bzr", "push", BRANCH_URL.format(source_package_name, version.replace("~", "").replace(":", "")), "--overwrite"]) != 0:
+        raise Exception("The push command returned an error.")
+    mergeinstance = subprocess.Popen(["bzr", "lp-propose-merge", parent_branch, "-m", PACKAGING_MERGE_COMMIT_MESSAGE.format(version, tip_rev, branch), "--approve"], stdin=subprocess.PIPE, env=env)
+    mergeinstance.communicate(input="y")
+    if mergeinstance.returncode != 0:
+        raise Exception("The lp-propose command returned an error.")
+    os.chdir('..')
+
+
+def merge_branch_with_parent_into(local_branch_uri, lp_parent_branch, dest_uri, commit_message, revision):
+    """Merge local branch into lp_parent_branch at revision"""
+    success = False
+    cur_dir = os.path.abspath('.')
+    subprocess.call(["bzr", "branch", "-r", str(revision), lp_parent_branch, dest_uri])
+    os.chdir(dest_uri)
+    if subprocess.call(["bzr", "merge", local_branch_uri]) == 0:
+        subprocess.call(["bzr", "commit", "-m", commit_message])
+        success = True
+    os.chdir(cur_dir)
+    return success
+
+
+def merge_branch(uri_to_merge, lp_parent_branch, commit_message, authors=set()):
+    """Resync with targeted branch if possible"""
+    success = False
+    cur_dir = os.path.abspath('.')
+    os.chdir(uri_to_merge)
+    lp_parent_branch = lp_parent_branch.replace("https://code.launchpad.net/", "lp:")
+    if subprocess.call(["bzr", "merge", lp_parent_branch]) == 0:
+        cmd = ["bzr", "commit", "-m", commit_message, "--unchanged"]
+        for author in authors:
+            cmd.extend(['--author', author])
+        subprocess.call(cmd)
+        success = True
+    os.chdir(cur_dir)
+    return success
+
+def push_to_branch(source_uri, lp_parent_branch, overwrite=False):
+    """Push source to parent branch"""
+    success = False
+    cur_dir = os.path.abspath('.')
+    os.chdir(source_uri)
+    lp_parent_branch = lp_parent_branch.replace("https://code.launchpad.net/", "lp:")
+    command = ["bzr", "push", lp_parent_branch]
+    if overwrite:
+        command.append("--overwrite")
+    if subprocess.call(command) == 0:
+        success = True
+    os.chdir(cur_dir)
+    return success
+
+def grab_committers_compared_to(source_uri, lp_branch_to_scan):
+    """Return unique list of committers for a given branch"""
+    committers = set()
+    cur_dir = os.path.abspath('.')
+    os.chdir(source_uri)
+    lp_branch_to_scan = lp_branch_to_scan.replace("https://code.launchpad.net/", "lp:")
+    instance = subprocess.Popen(["bzr", "missing", lp_branch_to_scan, "--other"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (stdout, stderr) = instance.communicate()
+    if stderr != "":
+        raise Exception("bzr missing on {} returned a failure: {}".format(lp_branch_to_scan, stderr.decode("utf-8").strip()))
+    committer_regexp = re.compile("\ncommitter: (.*)\n")
+    for match in committer_regexp.findall(stdout):
+        for committer in match.split(', '):
+            committers.add(committer)
+    os.chdir(cur_dir)
+    return committers

=== added file 'branch-source-builder/cupstream2distro/launchpadmanager.py'
--- branch-source-builder/cupstream2distro/launchpadmanager.py	1970-01-01 00:00:00 +0000
+++ branch-source-builder/cupstream2distro/launchpadmanager.py	2014-02-11 20:59:14 +0000
@@ -0,0 +1,157 @@
+# -*- coding: utf-8 -*-
+# Copyright (C) 2012 Canonical
+#
+# Authors:
+#  Didier Roche
+#
+# This program is free software; you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free Software
+# Foundation; version 3.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
+# details.cat
+#
+# You should have received a copy of the GNU General Public License along with
+# this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+from __future__ import unicode_literals
+
+from launchpadlib.launchpad import Launchpad
+import lazr
+import logging
+import os
+launchpad = None
+
+from .settings import ARCHS_TO_EVENTUALLY_IGNORE, ARCHS_TO_UNCONDITIONALLY_IGNORE, VIRTUALIZED_PPA_ARCH, CRED_FILE_PATH, COMMON_LAUNCHPAD_CACHE_DIR
+
+
+def get_launchpad(use_staging=False, use_cred_file=os.path.expanduser(CRED_FILE_PATH)):
+    '''Get THE Launchpad'''
+    global launchpad
+    if not launchpad:
+        if use_staging:
+            server = 'staging'
+        else:
+            server = 'production'
+
+        launchpadlib_dir = COMMON_LAUNCHPAD_CACHE_DIR
+        if not os.path.exists(launchpadlib_dir):
+            os.makdedirs(launchpadlib_dir)
+
+        if use_cred_file:
+            launchpad = Launchpad.login_with('cupstream2distro', server, allow_access_levels=["WRITE_PRIVATE"],
+                                             version='devel',  # devel because copyPackage is only available there
+                                             credentials_file=use_cred_file,
+                                             launchpadlib_dir=launchpadlib_dir)
+        else:
+            launchpad = Launchpad.login_with('cupstream2distro', server, allow_access_levels=["WRITE_PRIVATE"],
+                                             version='devel',  # devel because copyPackage is only available there
+                                             launchpadlib_dir=launchpadlib_dir)
+
+    return launchpad
+
+
+def get_ubuntu():
+    '''Get the ubuntu distro'''
+    lp = get_launchpad()
+    return lp.distributions['ubuntu']
+
+
+def get_ubuntu_archive():
+    '''Get the ubuntu main archive'''
+    return get_ubuntu().main_archive
+
+
+def get_series(series_name):
+    '''Return the launchpad object for the requested series'''
+    return get_ubuntu().getSeries(name_or_version=series_name)
+
+
+def get_bugs_titles(author_bugs):
+    lp = get_launchpad()
+    author_bugs_with_title = author_bugs.copy()
+    for author in author_bugs:
+        bug_title_sets = set()
+        for bug in author_bugs[author]:
+            try:
+                bug_title_sets.add("{} (LP: #{})".format(lp.bugs[bug].title, bug))
+            except KeyError:
+                # still list non existing or if launchpad timeouts bugs
+                bug_title_sets.add(u"Fix LP: #{}".format(bug))
+        author_bugs_with_title[author] = bug_title_sets
+
+    return author_bugs_with_title
+
+
+def open_bugs_for_source(bugs_list, source_name, series_name):
+    lp = get_launchpad()
+    ubuntu = get_ubuntu()
+
+    # don't nominate for current series
+    if ubuntu.current_series.name == series_name:
+        package = ubuntu.getSourcePackage(name=source_name)
+    else:
+        series = get_series(series_name)
+        package = series.getSourcePackage(name=source_name)
+
+    for bug_num in bugs_list:
+        try:
+            bug = lp.bugs[bug_num]
+            bug.addTask(target=package)
+            bug.lp_save()
+        except (KeyError, lazr.restfulclient.errors.BadRequest, lazr.restfulclient.errors.ServerError):
+            # ignore non existing or available bugs
+            logging.info("Can't synchronize upstream/downstream bugs for bug #{}. Not blocking on that.".format(bug_num))
+
+
+def get_available_and_all_archs(series, ppa=None):
+    '''Return a set of available archs, and the all arch'''
+    available_arch = set()
+    if ppa and ppa.require_virtualized:
+        available_arch = set(VIRTUALIZED_PPA_ARCH)
+        arch_all_arch = VIRTUALIZED_PPA_ARCH[0]
+    else:
+        for arch in series.architectures:
+            # HACK: filters armel as it's still seen as available on raring: https://launchpad.net/bugs/1077257
+            if arch.architecture_tag == "armel":
+                continue
+            available_arch.add(arch.architecture_tag)
+            if arch.is_nominated_arch_indep:
+                arch_all_arch = arch.architecture_tag
+
+    return (available_arch, arch_all_arch)
+
+
+def get_ignored_archs():
+    '''The archs we can eventually ignore if nothing is published in dest'''
+    return (ARCHS_TO_EVENTUALLY_IGNORE, ARCHS_TO_UNCONDITIONALLY_IGNORE)
+
+
+def get_ppa(ppa_name):
+    '''Return a launchpad ppa'''
+    if ppa_name.startswith("ppa:"):
+        ppa_name = ppa_name[4:]
+    ppa_dispatch = ppa_name.split("/")
+    return get_launchpad().people[ppa_dispatch[0]].getPPAByName(name=ppa_dispatch[1])
+
+def is_series_current(series_name):
+    '''Return if series_name is the edge development version'''
+    return get_ubuntu().current_series.name == series_name
+
+def get_resource_from_url(url):
+    '''Return a lp resource from a launchpad url'''
+    lp = get_launchpad()
+    url = lp.resource_type_link.replace("/#service-root", "") + url.split("launchpad.net")[1]
+    return lp.load(url)
+
+def get_resource_from_token(url):
+    '''Return a lp resource from a launchpad token'''
+    lp = get_launchpad()
+    return lp.load(url)
+
+def is_dest_ubuntu_archive(series_link):
+    '''return if series_link is the ubuntu archive'''
+    return series_link == get_ubuntu_archive().self_link

=== added file 'branch-source-builder/cupstream2distro/packageinppa.py'
--- branch-source-builder/cupstream2distro/packageinppa.py	1970-01-01 00:00:00 +0000
+++ branch-source-builder/cupstream2distro/packageinppa.py	2014-02-11 20:59:14 +0000
@@ -0,0 +1,225 @@
+# -*- coding: utf-8 -*-
+# Copyright (C) 2012 Canonical
+#
+# Authors:
+#  Didier Roche
+#
+# This program is free software; you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free Software
+# Foundation; version 3.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+import logging
+import os
+import re
+
+
+class PackageInPPA():
+
+    (BUILDING, FAILED, PUBLISHED) = range(3)
+
+    def __init__(self, source_name, version, ppa, destarchive, series,
+                 available_archs_in_ppa, arch_all_arch, archs_to_eventually_ignore,
+                 archs_to_unconditionually_ignore, package_archs=None):
+        self.source_name = source_name
+        self.version = version
+        self.series = series
+        self.arch_all_arch = arch_all_arch
+        self.ppa = ppa
+        self.current_status = {}
+
+        # Get archs we should look at
+        version_for_source_file = version.split(':')[-1]
+        if not package_archs:
+            dsc_filename = "{}_{}.dsc".format(source_name, version_for_source_file)
+            regexp = re.compile("^Architecture: (.*)\n")
+            for line in open(dsc_filename):
+                arch_lists = regexp.findall(line)
+                if arch_lists:
+                    package_archs = arch_lists[0]
+                    break
+
+        if not package_archs:
+            raise EnvironmentError("Can't determine package Architecture.")
+        if "any" in package_archs:
+            self.archs = available_archs_in_ppa.copy()
+        elif package_archs == "all":
+            self.archs = set([self.arch_all_arch])
+        else:
+            archs_supported_by_package = set()
+            for arch in package_archs.split():
+                archs_supported_by_package.add(arch)
+            self.archs = archs_supported_by_package.intersection(available_archs_in_ppa)
+        # ignore some eventual archs if doesn't exist in latest published version in dest
+        archs_to_eventually_ignore = archs_to_eventually_ignore.copy()
+        if archs_to_eventually_ignore:
+            try:
+                previous_source = destarchive.getPublishedSources(exact_match=True, source_name=self.source_name,
+                                                                  distro_series=self.series, status="Published")[0]
+                for binary in previous_source.getPublishedBinaries():
+                    if binary.architecture_specific and binary.distro_arch_series.architecture_tag in archs_to_eventually_ignore:
+                        archs_to_eventually_ignore -= set([binary.distro_arch_series.architecture_tag])
+                    if not archs_to_eventually_ignore:
+                        break
+
+            except IndexError:
+                # no package in dest, don't wait on any archs_to_eventually_ignore
+                pass
+            # remove from the inspection remaining archs to ignore
+            if archs_to_eventually_ignore:
+                self.archs -= archs_to_eventually_ignore
+            if archs_to_unconditionually_ignore:
+                self.archs -= archs_to_unconditionually_ignore
+
+
+    def __repr__(self):
+        return '{} - {}'.format(self.source_name, self.version)
+
+
+    def get_status(self, on_particular_arch=None):
+        '''Look at the package status in the ppa
+
+        Can scope to a particular arch to watch for'''
+
+        self._refresh_status()
+        if not self.current_status:
+            return None
+
+        current_package_building = False
+        current_package_failed = False
+        for arch in self.current_status:
+            if on_particular_arch and arch != on_particular_arch:
+                continue
+            str_status = "published"
+            if self.current_status[arch] == PackageInPPA.BUILDING:
+                current_package_building = True
+                str_status = "building"
+            if self.current_status[arch] == PackageInPPA.FAILED:
+                current_package_failed = True
+                str_status = "failed"
+            logging.info("arch: {}, status: {}".format(arch, str_status))
+
+        if current_package_building:
+            return self.BUILDING
+        # no more package is building, if one failed, time to signal it
+        if current_package_failed:
+            return self.FAILED
+        # if it's not None, not BUILDING, nor FAILED, it's PUBLISHED
+        return self.PUBLISHED
+
+    def _refresh_archs_skipped(self):
+        '''Refresh archs that we should skip for this build'''
+
+        for arch in self.archs.copy():
+            if os.path.isfile("{}.{}.ignore".format(self.source_name, arch)):
+                logging.warning("Request to ignore {} on {}.".format(self.source_name, arch))
+                try:
+                    self.archs.remove(arch)
+                except ValueError:
+                    logging.warning("Request to ignore {} on {} has been proceeded, but this one wasn't in the list we were monitor for.".format(self.source_name, arch))
+                try:
+                    self.current_status.pop(arch)
+                except KeyError:
+                    pass
+
+    def _refresh_status(self):
+        '''Refresh status from the ppa'''
+
+        self._refresh_archs_skipped()
+
+        # first step, get the source published
+        if not self.current_status:
+            (self.current_status, self.source) = self._get_status_for_source_package_in_ppa()
+        # check the binary status
+        if self.current_status:
+            self.current_status = self._get_status_for_binary_packages_in_ppa()
+
+    def _get_status_for_source_package_in_ppa(self):
+        '''Return current_status for source package in ppa.
+
+        The status is dict (if not None) with {arch: status} and can be:
+            - None -> not visible yet
+            - BUILDING -> currently Building (or waiting to build)
+            - FAILED -> Build failed for this arch or has been canceled
+            - PUBLISHED -> All packages (including arch:all from other archs) published.
+
+            Only the 2 first status are returned by this call. See _get_status_for_binary_packages_in_ppa
+            for the others.'''
+
+        try:
+            source = self.ppa.getPublishedSources(exact_match=True, source_name=self.source_name, version=self.version, distro_series=self.series)[0]
+            logging.info("Source available in ppa")
+            current_status = {}
+            for arch in self.archs:
+                current_status[arch] = self.BUILDING
+            return (current_status, source)
+        except (KeyError, IndexError):
+            logging.info("Source not yet available")
+            return ({}, None)
+
+    def _get_status_for_binary_packages_in_ppa(self):
+        '''Return current status for package in ppa
+
+        The status is dict (if not None) with {arch: status} and can be:
+            - None -> not visible yet
+            - BUILDING -> currently Building (or waiting to build)
+            - FAILED -> Build failed for this arch or has been canceled
+            - PUBLISHED -> All packages (including arch:all from other archs) published.
+
+            Only the 3 last statuses are returned by this call. See _get_status_for_source_package_in_ppa
+            for the other.'''
+
+        # Try to see if all binaries availables for this arch are built, including arch:all on other archs
+        status = self.current_status
+        at_least_one_published_binary = False
+        for binary in self.source.getPublishedBinaries():
+            at_least_one_published_binary = True
+            # all binaries for an arch are published at the same time
+            # launchpad is lying, it's telling that archs not in the ppa are built (for arch:all). Even for non supported arch!
+            # for instance, we can have the case of self.arch_all_arch (arch:all), built before the others and amd64 will be built for it
+            if binary.status == "Published" and (binary.distro_arch_series.architecture_tag == self.arch_all_arch or
+               (binary.distro_arch_series.architecture_tag != self.arch_all_arch and binary.architecture_specific)):
+                status[binary.distro_arch_series.architecture_tag] = self.PUBLISHED
+
+        # Looking for builds on archs still BUILDING (just loop on builds once to avoid too many lp requests)
+        needs_checking_build = False
+        build_state_failed = ('Failed to build', 'Chroot problem', 'Failed to upload', 'Cancelled build', 'Build for superseded Source')
+        for arch in self.archs:
+            if self.current_status[arch] == self.BUILDING:
+                needs_checking_build = True
+        if needs_checking_build:
+            for build in self.source.getBuilds():
+                # ignored archs
+                if not build.arch_tag in self.current_status:
+                    continue
+                if self.current_status[build.arch_tag] == self.BUILDING:
+                    if build.buildstate in build_state_failed:
+                        logging.error("{}: Build {} ({}) failed because of {}".format(build.arch_tag, build.title,
+                                                                                       build.web_link, build.buildstate))
+                        status[build.arch_tag] = self.FAILED
+                    # Another launchpad trick: if a binary arch was published, but then is superseeded, getPublishedBinaries() won't list
+                    # those binaries anymore. So it's seen as BUILDING again.
+                    # If there is a successful build record of it and the source is superseded, it means that it built fine at some point,
+                    # Another arch will fail as superseeded.
+                    # We don't just retain the old state of "PUBLISHED" because maybe we started the script with that situation already
+                    elif build.buildstate not in build_state_failed and self.source.status == "Superseded":
+                        status[build.arch_tag] = self.PUBLISHED
+
+        # There is no way to know if there are some arch:all packages (and there are not in publishedBinaries for this arch until
+        # it's built on arch_all_arch). So mark all arch to BUILDING if self.arch_all_arch is building or FAILED if it failed.
+        if self.arch_all_arch in status and status[self.arch_all_arch] != self.PUBLISHED:
+            for arch in self.archs:
+                if status[arch] == self.PUBLISHED:
+                    status[arch] = status[self.arch_all_arch]
+                    if arch != self.arch_all_arch and status[arch] == self.FAILED:
+                        logging.error("{} marked as FAILED because {} build FAILED and we may miss arch:all packages".format(arch, self.arch_all_arch))
+
+        return status

=== added file 'branch-source-builder/cupstream2distro/packageinppamanager.py'
--- branch-source-builder/cupstream2distro/packageinppamanager.py	1970-01-01 00:00:00 +0000
+++ branch-source-builder/cupstream2distro/packageinppamanager.py	2014-02-11 20:59:14 +0000
@@ -0,0 +1,109 @@
+# -*- coding: utf-8 -*-
+# Copyright (C) 2012 Canonical
+#
+# Authors:
+#  Didier Roche
+#
+# This program is free software; you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free Software
+# Foundation; version 3.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+import ConfigParser
+import logging
+import os
+import re
+
+from .branchhandling import _get_parent_branch
+from .packageinppa import PackageInPPA
+from .settings import PROJECT_CONFIG_SUFFIX
+
+
+def _ensure_removed_from_set(target_set, content_to_remove):
+    '''Silent removal from an existing set'''
+    try:
+        target_set.remove(content_to_remove)
+    except KeyError:
+        pass  # in case we missed the "build" step
+
+
+def get_all_packages_uploaded():
+    '''Get (package, version, rev, branch) of all packages uploaded'''
+
+    # we do not rely on the .changes files but in the config file
+    # because we need the exact version (which can have an epoch)
+    result = set()
+    source_package_regexp = re.compile("(.*).{}$".format(PROJECT_CONFIG_SUFFIX))
+    for file in os.listdir('.'):
+        substract = source_package_regexp.findall(file)
+        if substract:
+            version = _get_current_packaging_version_from_config(substract[0])
+            rev = _get_current_rev_from_config(substract[0])
+            branch = _get_parent_branch(substract[0])
+            result.add((substract[0], version, rev, branch))
+    foo = set()
+    foo.add(('autopilot', '1.3.1+13.10.20131003.1-0ubuntu2~fginther.1',
+            '100', 'lp:autopilot'))
+    foo.add(('autopilot', '1.3.1+13.10.20131003.1-0ubuntu2~fginther.2',
+            '105', 'lp:autopilot'))
+    return foo
+    return result
+
+
+def get_packages_and_versions_uploaded():
+    '''Get (package, version) of all packages uploaded. We can have duplicates'''
+
+    # we do not rely on the .changes files but in the config file
+    # because we need the exact version (which can have an epoch)
+    result = set()
+    source_package_regexp = re.compile("(.*).{}.*$".format(PROJECT_CONFIG_SUFFIX))
+    for file in os.listdir('.'):
+        substract = source_package_regexp.findall(file)
+        if substract:
+            config = ConfigParser.RawConfigParser()
+            config.read(file)
+            result.add((substract[0], config.get('Package', 'packaging_version')))
+    return result
+
+
+def update_all_packages_status(packages_not_in_ppa, packages_building, packages_failed, particular_arch=None):
+    '''Update all packages status, checking in the ppa'''
+
+    for current_package in (packages_not_in_ppa.union(packages_building)):
+        logging.info("current_package: " + current_package.source_name + " " + current_package.version)
+        package_status = current_package.get_status(particular_arch)
+        if package_status != None:  # global package_status can be 0 (building), 1 (failed), 2 (published)
+            # if one arch building, still considered as building
+            if package_status == PackageInPPA.BUILDING:
+                _ensure_removed_from_set(packages_not_in_ppa, current_package)  # maybe already removed
+                packages_building.add(current_package)
+            # if one arch failed, considered as failed
+            elif package_status == PackageInPPA.FAILED:
+                _ensure_removed_from_set(packages_building, current_package)  # in case we missed the "build" step
+                _ensure_removed_from_set(packages_not_in_ppa, current_package)  # in case we missed the "wait" step
+                packages_failed.add(current_package)
+            elif package_status == PackageInPPA.PUBLISHED:
+                _ensure_removed_from_set(packages_building, current_package)  # in case we missed the "build" step
+                _ensure_removed_from_set(packages_not_in_ppa, current_package)  # in case we missed the "wait" step
+
+
+def _get_current_packaging_version_from_config(source_package_name):
+    '''Get current packaging version from the saved config'''
+    config = ConfigParser.RawConfigParser()
+    config.read("{}.{}".format(source_package_name, PROJECT_CONFIG_SUFFIX))
+    return config.get('Package', 'packaging_version')
+
+
+def _get_current_rev_from_config(source_package_name):
+    '''Get current tip revision from the saved config'''
+    config = ConfigParser.RawConfigParser()
+    config.read("{}.{}".format(source_package_name, PROJECT_CONFIG_SUFFIX))
+    return config.get('Branch', 'rev')

=== added file 'branch-source-builder/cupstream2distro/packagemanager.py'
--- branch-source-builder/cupstream2distro/packagemanager.py	1970-01-01 00:00:00 +0000
+++ branch-source-builder/cupstream2distro/packagemanager.py	2014-02-11 20:59:14 +0000
@@ -0,0 +1,504 @@
+# -*- coding: utf-8 -*-
+# Copyright (C) 2012-2014 Canonical
+#
+# Authors:
+#  Didier Roche
+#  Rodney Dawes
+#
+# This program is free software; you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free Software
+# Foundation; version 3.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+import datetime
+import fileinput
+import logging
+import os
+import re
+import shutil
+import sys
+import subprocess
+import urllib
+
+import launchpadmanager
+import settings
+from .utils import ignored
+
+
+def get_current_version_for_series(source_package_name, series_name, ppa_name=None, dest=None):
+    '''Get current version for a package name in that series'''
+    series = launchpadmanager.get_series(series_name)
+    if not dest:
+        if ppa_name:
+            dest = launchpadmanager.get_ppa(ppa_name)
+        else:
+            dest = launchpadmanager.get_ubuntu_archive()
+    source_collection = dest.getPublishedSources(exact_match=True, source_name=source_package_name, distro_series=series)
+    try:
+        # cjwatson told that list always have the more recently published first (even if removed)
+        return source_collection[0].source_package_version
+    # was never in the dest, set the lowest possible version
+    except IndexError:
+        return "0"
+
+
+def is_version_for_series_in_dest(source_package_name, version, series, dest, pocket="Release"):
+    '''Return if version for a package name in that series is in dest'''
+    return dest.getPublishedSources(exact_match=True, source_name=source_package_name, version=version,
+                                    distro_series=series, pocket=pocket).total_size > 0
+
+def is_version_in_queue(source_package_name, version, dest_serie, queue):
+    '''Return if version for a package name in that series is in dest'''
+    return dest_serie.getPackageUploads(exact_match=True, name=source_package_name, version=version,
+                                        status=queue).total_size > 0
+
+
+def is_version1_higher_than_version2(version1, version2):
+    '''return if version1 is higher than version2'''
+    return (subprocess.call(["dpkg", "--compare-versions", version1, 'gt', version2], stdout=subprocess.PIPE, stderr=subprocess.PIPE) == 0)
+
+
+def is_version_in_changelog(version, f):
+    '''Return if the version is in the upstream changelog (released)'''
+
+    if version == "0":
+        return True
+
+    desired_changelog_line = re.compile("\({}\) (?!UNRELEASED).*\; urgency=".format(version.replace('+', '\+')))
+    for line in f.readlines():
+        if desired_changelog_line.search(line):
+            return True
+
+    return False
+
+
+def get_latest_upstream_bzr_rev(f, dest_ppa=None):
+    '''Report latest bzr rev in the file
+
+    If dest_ppa, first try to fetch the dest ppa tag. Otherwise, fallback to first distro version'''
+    distro_regex = re.compile("{} (\d+)".format(settings.REV_STRING_FORMAT))
+    destppa_regexp = re.compile("{} (\d+) \(ppa:{}\)".format(settings.REV_STRING_FORMAT, dest_ppa))
+    distro_rev = None
+    candidate_destppa_rev = None
+    candidate_distro_rev = None
+    destppa_element_found = False
+
+    # handle marker spread on two lines
+    end_of_line_regexp = re.compile(" *(.*\))")
+    previous_line = None
+
+    for line in f:
+        line = line[:-1]
+
+        if previous_line:
+            try:
+                line = previous_line + end_of_line_regexp.findall(line)[0]
+            except IndexError:
+                None
+            previous_line = None
+
+        if dest_ppa:
+            try:
+                candidate_destppa_rev = int(destppa_regexp.findall(line)[0])
+                destppa_element_found = True
+            except IndexError:
+                destppa_element_found = False
+        if not distro_rev and not destppa_element_found and not "(ppa:" in line:
+            try:
+                candidate_distro_rev = int(distro_regex.findall(line)[0])
+                distro_element_found = True
+            except IndexError:
+                distro_element_found = False
+
+        # try to catchup next line if we have a marker start without anything found
+        if settings.REV_STRING_FORMAT in line and (dest_ppa and not destppa_element_found) and not distro_element_found:
+            previous_line = line
+
+        if line.startswith(" -- "):
+            # first grab the dest ppa
+            if candidate_destppa_rev:
+                return candidate_destppa_rev
+            if not distro_rev and candidate_distro_rev:
+                distro_rev = candidate_distro_rev
+            if not dest_ppa and distro_rev:
+                return distro_rev
+
+    # we didn't find any dest ppa result but there is a distro_rev one
+    if dest_ppa and distro_rev:
+        return distro_rev
+
+    # we force a bootstrap commit for new components
+    return 0
+
+
+def list_packages_info_in_str(packages_set):
+    '''Return the packages info in a string'''
+
+    results = []
+    for package in packages_set:
+        results.append("{} ({})".format(package.source_name, package.version))
+    return " ".join(results)
+
+
+def get_packaging_version():
+    '''Get current packaging rev'''
+    instance = subprocess.Popen(["dpkg-parsechangelog"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (stdout, stderr) = instance.communicate()
+    if instance.returncode != 0:
+        raise Exception(stderr.decode("utf-8").strip())
+    expr = re.compile("Version: (.*)")
+    for line in stdout.splitlines():
+        packaging_version = expr.findall(line)
+        if packaging_version:
+            return packaging_version[0]
+
+    raise Exception("Didn't find any Version in the package: {}".format(stdout))
+
+
+def get_source_package_from_dest(source_package_name, dest_archive, dest_current_version, series_name):
+    '''Download and return a path containing a checkout of the current dest version.
+
+    None if this package was never published to dest archive'''
+
+    if dest_current_version == "0":
+        logging.info("This package was never released to the destination archive, don't return downloaded source")
+        return None
+
+    logging.info("Grab code for {} ({}) from {}".format(source_package_name, dest_current_version, series_name))
+    source_package_download_dir = os.path.join('ubuntu', source_package_name)
+    series = launchpadmanager.get_series(series_name)
+    with ignored(OSError):
+        os.makedirs(source_package_download_dir)
+    os.chdir(source_package_download_dir)
+
+    try:
+        sourcepkg = dest_archive.getPublishedSources(status="Published", exact_match=True, source_name=source_package_name, distro_series=series, version=dest_current_version)[0]
+    except IndexError:
+        raise Exception("Couldn't get in the destination the expected version")
+    logging.info('Downloading %s version %s', source_package_name, dest_current_version)
+    for url in sourcepkg.sourceFileUrls():
+        urllib.urlretrieve(url, urllib.unquote(url.split('/')[-1]))
+    instance = subprocess.Popen("dpkg-source -x *dsc", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (stdout, stderr) = instance.communicate()
+    if instance.returncode != 0:
+        raise Exception(stderr.decode("utf-8").strip())
+
+    # check the dir exist
+    splitted_version = dest_current_version.split(':')[-1].split('-')  # remove epoch is there is one
+    # TODO: debian version (like -3) is not handled here.
+    # We do handle 42ubuntu1 though (as splitted_version[0] can contain "ubuntu")
+    if "ubuntu" in splitted_version[-1] and len(splitted_version) > 1:  # don't remove last item for the case where we had a native version (-0.35.2) without ubuntu in it
+        splitted_version = splitted_version[:-1]
+    version_for_source_file = '-'.join(splitted_version)
+    source_directory_name = "{}-{}".format(source_package_name, version_for_source_file)
+    if not os.path.isdir(source_directory_name):
+        raise Exception("We tried to download and check that the directory {} is present, but it's not the case".format(source_directory_name))
+    os.chdir('../..')
+    return (os.path.join(source_package_download_dir, source_directory_name))
+
+
+def is_new_content_relevant_since_old_published_source(dest_version_source):
+    '''Return True if a new snapshot is needed
+
+    dest_version_source can be None if no released version was done before.'''
+
+    # we always released something not yet in ubuntu, no matter criterias are not met.
+    if not dest_version_source:
+        return True
+
+    # now check the relevance of the committed changes compared to the version in the repository (if any)
+    diffinstance = subprocess.Popen(['diff', '-Nrup', '.', dest_version_source], stdout=subprocess.PIPE)
+    filterinstance = subprocess.Popen(['filterdiff', '--clean', '-x', '*po', '-x', '*pot', '-x', '*local-options'], stdin=diffinstance.stdout, stdout=subprocess.PIPE)
+    lsdiffinstance = subprocess.Popen(['lsdiff'], stdin=filterinstance.stdout, stdout=subprocess.PIPE)
+    (relevant_changes, err) = subprocess.Popen(['grep', '-Ev', '.bzr|.pc'], stdin=lsdiffinstance.stdout, stdout=subprocess.PIPE).communicate()
+
+    # detect if the only change is a Vcs* target changes (with or without changelog edit). We won't release in that case
+    number_of_changed_files = relevant_changes.count("\n")
+    if ((number_of_changed_files == 1 and "debian/control" in relevant_changes) or
+       (number_of_changed_files == 2 and "debian/control" in relevant_changes and "debian/changelog" in relevant_changes)):
+        (results, err) = subprocess.Popen(['diff', os.path.join('debian', 'control'), os.path.join(dest_version_source, "debian", "control")], stdout=subprocess.PIPE).communicate()
+        for diff_line in results.split('\n'):
+            if diff_line.startswith("< ") or diff_line.startswith("> "):
+                if not diff_line[2:].startswith("Vcs-") and not diff_line[2:].startswith("#"):
+                    return True
+        return False
+
+    logging.debug("Relevant changes are:")
+    logging.debug(relevant_changes)
+
+    return (relevant_changes != '')
+
+
+def is_relevant_source_diff_from_previous_dest_version(newdsc_path, dest_version_source):
+    '''Extract and check if the generated source diff different from previous one'''
+
+    with ignored(OSError):
+        os.makedirs("generated")
+    extracted_generated_source = os.path.join("generated", newdsc_path.split('_')[0])
+    with ignored(OSError):
+        shutil.rmtree(extracted_generated_source)
+
+    # remove epoch is there is one
+    if subprocess.call(["dpkg-source", "-x", newdsc_path, extracted_generated_source]) != 0:
+        raise Exception("dpkg-source command returned an error.")
+
+    # now check the relevance of the committed changes compared to the version in the repository (if any)
+    diffinstance = subprocess.Popen(['diff', '-Nrup', extracted_generated_source, dest_version_source], stdout=subprocess.PIPE)
+    (diff, err) = subprocess.Popen(['filterdiff', '--clean', '-x', '*po', '-x', '*pot', '-x', '*local-options'], stdin=diffinstance.stdout, stdout=subprocess.PIPE).communicate()
+
+    # there is no important diff if the diff only contains 12 lines, corresponding to "Automatic daily release" marker in debian/changelog
+    if (diff.count('\n') <= 12):
+        return False
+    return True
+
+
+def _packaging_changes_between_dsc(oldsource_dsc, newsource_dsc):
+    '''Return if there has been a packaging change between two dsc files
+
+    We ignore the changelog only changes'''
+    if not oldsource_dsc:
+        return True
+    if not os.path.isfile(oldsource_dsc) or not os.path.isfile(newsource_dsc):
+        raise Exception("{} or {} doesn't not exist, can't create a diff".format(oldsource_dsc, newsource_dsc))
+    diffinstance = subprocess.Popen(['debdiff', oldsource_dsc, newsource_dsc], stdout=subprocess.PIPE)
+    filterinstance = subprocess.Popen(['filterdiff', '--clean', '-i', '*debian/*', '-x', '*changelog'], stdin=diffinstance.stdout, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (change_in_debian, filter_err) = filterinstance.communicate()
+    # we can't rely on diffinstance returncode as the signature key is maybe not present and it will exit with 1
+    if filterinstance.returncode != 0:
+        raise Exception("Error in diff: {}".format(filter_err.decode("utf-8").strip()))
+    return(change_in_debian != "")
+
+
+def generate_diff_between_dsc(diff_filepath, oldsource_dsc, newsource_dsc):
+    '''Generate a diff file in diff_filepath if there is a relevant packaging diff between 2 sources
+
+    The diff contains autotools files and cmakeries'''
+    if _packaging_changes_between_dsc(oldsource_dsc, newsource_dsc):
+        with open(diff_filepath, "w") as f:
+            if not oldsource_dsc:
+                f.writelines("This source is a new package, if the destination is ubuntu, please ensure it has been preNEWed by an archive admin before publishing that stack.")
+                return
+            f.write("/!\ Remember that this diff only represents packaging changes and build tools diff, not the whole content diff!\n\n")
+            diffinstance = subprocess.Popen(['debdiff', oldsource_dsc, newsource_dsc], stdout=subprocess.PIPE)
+            (changes_to_publish, err) = subprocess.Popen(['filterdiff', '--remove-timestamps', '--clean', '-i', '*setup.py',
+                                                          '-i', '*Makefile.am', '-i', '*configure.*', '-i', '*debian/*',
+                                                          '-i', '*CMakeLists.txt'], stdin=diffinstance.stdout, stdout=subprocess.PIPE).communicate()
+            f.write(changes_to_publish)
+
+
+def create_new_packaging_version(base_package_version, series_version, destppa=''):
+    '''Deliver a new packaging version, based on simple rules:
+
+    Version would be <upstream_version>.<series>.<yyyymmdd(.minor)>-0ubuntu1
+    if we already have something delivered today, it will be .minor, then, .minor+1…
+
+    We append the destination ppa name if we target a dest ppa and not distro'''
+    # to keep track of whether the package is native or not
+    native_pkg = False
+
+    today_version = datetime.date.today().strftime('%Y%m%d')
+    destppa = destppa.replace("-", '.').replace("_", ".").replace("/", ".")
+    # bootstrapping mode or direct upload or UNRELEASED for bumping to a new series
+    # TRANSITION
+    if not ("daily" in base_package_version or "+" in base_package_version):
+        # support both 42, 42-0ubuntu1
+        upstream_version = base_package_version.split('-')[0]
+        # if we have 42ubuntu1 like a wrong native version
+        if "ubuntu" in upstream_version:
+            upstream_version = upstream_version.split('ubuntu')[0]
+    elif not "-" in base_package_version and "+" in base_package_version:
+        # extract the day of previous daily upload and bump if already uploaded
+        regexp = re.compile("(.*)\+([\d\.]{5})\.(\d{8})\.?([\d]*).*")
+        try:
+            previous_day = regexp.findall(base_package_version)[0]
+            upstream_version = previous_day[0]
+            native_pkg = True
+            if (previous_day[1] == series_version and
+                previous_day[2] == today_version):
+                minor = 1
+                if previous_day[3]:  # second upload of the day
+                    minor = int(previous_day[3]) + 1
+                today_version = "{}.{}".format(today_version, minor)
+        except IndexError:
+            raise Exception(
+                "Unable to get previous day from native version: %s"
+                % base_package_version)
+    else:
+        # extract the day of previous daily upload and bump if already uploaded today
+        regexp = re.compile("(.*)\+([\d\.]{5})\.(\d{8})\.?([\d]*).*-.*")
+        try:
+            previous_day = regexp.findall(base_package_version)[0]
+        except IndexError:
+            # TRANSITION FALLBACK
+            try:
+                regexp = re.compile("(.*)(daily)([\d\.]{8})\.?([\d]*).*-.*")
+                previous_day = regexp.findall(base_package_version)[0]
+                # make the version compatible with the new version
+                previous_day = (previous_day[0], previous_day[1], "20" + previous_day[2].replace(".", ""), previous_day[3])
+            except IndexError:
+                raise Exception("Didn't find a correct versioning in the current package: {}".format(base_package_version))
+        upstream_version = previous_day[0]
+        if previous_day[1] == series_version and previous_day[2] == today_version:
+            minor = 1
+            if previous_day[3]:  # second upload of the day
+                minor = int(previous_day[3]) + 1
+            today_version = "{}.{}".format(today_version, minor)
+
+    new_upstream_version = "{upstream}+{series}.{date}{destppa}".format(
+        upstream=upstream_version, series=series_version,
+        date=today_version, destppa=destppa)
+    if native_pkg is not True:
+        new_upstream_version = "{}-0ubuntu1".format(new_upstream_version)
+
+    return new_upstream_version
+
+
+def get_packaging_sourcename():
+    '''Get current packaging source name'''
+    instance = subprocess.Popen(["dpkg-parsechangelog"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (stdout, stderr) = instance.communicate()
+    if instance.returncode != 0:
+        raise Exception(stderr.decode("utf-8").strip())
+    expr = re.compile("Source: (.*)")
+    for line in stdout.splitlines():
+        source_name = expr.findall(line)
+        if source_name:
+            return source_name[0]
+
+    raise Exception("Didn't find any source name in the package: {}".format(stdout))
+
+
+def collect_bugs_in_changelog_until_latest_snapshot(f, source_package_name):
+    '''Collect all bugs in the changelog until latest snapshot'''
+    bugs = set()
+    # matching only bug format that launchpad accepts
+    group_bugs_regexp = re.compile("lp: ?(.*\d{5,})", re.IGNORECASE)
+    bug_decipher_regexp = re.compile("(#\d{5,})+")
+    new_upload_changelog_regexp = re.compile(settings.NEW_CHANGELOG_PATTERN.format(source_package_name))
+    for line in f:
+        grouped_bugs_list = group_bugs_regexp.findall(line)
+        for grouped_bugs in grouped_bugs_list:
+            for bug in map(lambda bug_with_hash: bug_with_hash.replace('#', ''), bug_decipher_regexp.findall(grouped_bugs)):
+                bugs.add(bug)
+        #  a released upload to distro (automated or manual)n exit as bugs before were already covered
+        if new_upload_changelog_regexp.match(line):
+            return bugs
+
+    return bugs
+
+
+def update_changelog(new_package_version, series, tip_bzr_rev, authors_commits, dest_ppa=None):
+    '''Update the changelog for the incoming upload'''
+
+    dch_env = os.environ.copy()
+    for author in authors_commits:
+        dch_env["DEBFULLNAME"] = author
+        for bug_desc in authors_commits[author]:
+            if bug_desc.startswith('-'):
+                # Remove leading '-' or dch thinks (rightly) that it's an option
+                bug_desc = bug_desc[1:]
+            if bug_desc.startswith(' '):
+                # Remove leading spaces, there are useless and the result is
+                # prettier without them anyway ;)
+                bug_desc = bug_desc.strip()
+            cmd = ["dch", "--multimaint-merge", "--release-heuristic", "changelog",
+                   "-v{}".format(new_package_version), bug_desc]
+            subprocess.Popen(cmd, env=dch_env).communicate()
+
+    if tip_bzr_rev != None:
+        commit_message = "{} {}".format(settings.REV_STRING_FORMAT, tip_bzr_rev)
+        if dest_ppa:
+            commit_message += " ({})".format(dest_ppa)
+    else:
+        commit_message = ""
+
+    dch_env["DEBFULLNAME"] = settings.BOT_DEBFULLNAME
+    dch_env["DEBEMAIL"] = settings.BOT_DEBEMAIL
+    instance = subprocess.Popen(["dch", "--release-heuristic", "changelog",
+                                 "-v{}".format(new_package_version), commit_message],
+                                stderr=subprocess.PIPE, env=dch_env)
+    (stdout, stderr) = instance.communicate()
+    if instance.returncode != 0:
+        raise Exception(stderr.decode("utf-8").strip())
+    subprocess.call(["dch", "-r", "--distribution", series, "--force-distribution", ""], env=dch_env)
+
+    # in the case of no commit_message and no symbols file change, we have an addition [ DEBFULLNAME ] follow by an empty line
+    # better to remove both lines
+    subprocess.call(["sed", "-i", "/ \[ " + settings.BOT_DEBFULLNAME + " \]/{$q; N; /\\n$/d;}", "debian/changelog"])
+
+
+def build_source_package(series, distro_version, ppa=None):
+    '''Build the source package using the internal helper
+
+    Add the additional ppa inside the chroot if requested.'''
+
+    chroot_tool_dir = os.path.join(settings.ROOT_CU2D, "chroot-tools")
+    buildsource = os.path.join(chroot_tool_dir, "buildsource-chroot")
+    branch_dir = os.path.abspath('.')
+    parent_dir = os.path.abspath(os.path.dirname(branch_dir))
+    cowbuilder_env = os.environ.copy()
+    cowbuilder_env["HOME"] = chroot_tool_dir  # take the internal .pbuilderrc
+    cowbuilder_env["DIST"] = series
+    cmd = ["sudo", "-E", "cowbuilder", "--execute",
+           "--bindmounts", parent_dir,
+           "--bindmounts", settings.GNUPG_DIR,
+           "--", buildsource, branch_dir,
+           "--gnupg-parentdir", settings.GNUPG_DIR,
+           "--uid", str(os.getuid()), "--gid", str(os.getgid()),
+           "--gnupg-keyid", settings.BOT_KEY,
+           "--distro-version", distro_version]
+    if ppa:
+        cmd.extend(["--ppa", ppa])
+    instance = subprocess.Popen(cmd, env=cowbuilder_env)
+    instance.communicate()
+    if instance.returncode != 0:
+        raise Exception("%r returned: %s." % (cmd, instance.returncode))
+
+
+def upload_package(source, version, ppa, source_directory=None):
+    '''Upload the new package to a ppa'''
+    # remove epoch is there is one
+    here = None
+    if source_directory:
+        here = os.getcwd()
+        os.chdir(source_directory)
+    version_for_source_file = version.split(':')[-1]
+    cmd = ["dput", ppa,
+           "{}_{}_source.changes".format(source, version_for_source_file)]
+    if subprocess.call(cmd) != 0:
+        if here:
+            os.chdir(here)
+        raise Exception("%r returned an error." % (cmd,))
+    if here:
+        os.chdir(here)
+
+
+def refresh_symbol_files(packaging_version):
+    '''Refresh the symbols file having REPLACEME_TAG with version of the day.
+
+    Add a changelog entry if needed'''
+
+    new_upstream_version = packaging_version.split("-")[0]
+    replacement_done = False
+    for filename in os.listdir("debian"):
+        if filename.endswith("symbols"):
+            for line in fileinput.input(os.path.join('debian', filename), inplace=1):
+                if settings.REPLACEME_TAG in line:
+                    replacement_done = True
+                    line = line.replace(settings.REPLACEME_TAG, new_upstream_version)
+                sys.stdout.write(line)
+
+    if replacement_done:
+        dch_env = os.environ.copy()
+        dch_env["DEBFULLNAME"] = settings.BOT_DEBFULLNAME
+        dch_env["DEBEMAIL"] = settings.BOT_DEBEMAIL
+        subprocess.Popen(["dch", "debian/*symbols: auto-update new symbols to released version"], env=dch_env).communicate()
+        subprocess.call(["bzr", "commit", "-m", "Update symbols"])

=== added file 'branch-source-builder/cupstream2distro/settings.py'
--- branch-source-builder/cupstream2distro/settings.py	1970-01-01 00:00:00 +0000
+++ branch-source-builder/cupstream2distro/settings.py	2014-02-11 20:59:14 +0000
@@ -0,0 +1,121 @@
+# -*- coding: utf-8 -*-
+# Copyright (C) 2012 Canonical
+#
+# Authors:
+#  Didier Roche
+#
+# This program is free software; you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free Software
+# Foundation; version 3.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+import os
+import yaml
+
+REV_STRING_FORMAT = "Automatic snapshot from revision"
+NEW_CHANGELOG_PATTERN = "^{} \(.*\) (?!UNRELEASED)"
+PACKAGING_MERGE_COMMIT_MESSAGE = "Releasing {} (revision {} from {})"
+REPLACEME_TAG = "0replaceme"
+BRANCH_URL = "lp:~ps-jenkins/{}/latestsnapshot-{}"
+
+IGNORECHANGELOG_COMMIT = "#nochangelog"
+
+PROJECT_CONFIG_SUFFIX = "project"
+
+BOT_DEBFULLNAME = "Ubuntu daily release"
+BOT_DEBEMAIL = "[e-mail]"
+home_dir = "/srv/bsb_worker"
+CU2D_DIR = home_dir
+GNUPG_DIR = CU2D_DIR
+if not os.path.isdir(os.path.join(GNUPG_DIR, '.gnupg')):
+    GNUPG_DIR = home_dir
+CRED_FILE_PATH = os.path.join("/tmp", "launchpad.credentials")
+
+# TODO refactor into a ci-utils module
+def _unit_config():
+    path = os.path.join(home_dir, 'unit_config')
+    config = {}
+    try:
+        with open(path) as f:
+            config = yaml.safe_load(f.read())
+    except:
+        print('Unable to use unit_config(%s), defaulting values' % path)
+    return config
+_cfg = _unit_config()
+LAUNCHPAD_PPA_USER = _cfg.get('launchpad_user', None)
+LAUNCHPAD_API_BASE = _cfg.get(
+    'launchpad_api_base', 'https://api.launchpad.net/1.0')
+OAUTH_CONSUMER_KEY = _cfg.get('oauth_consumer_key', None)
+OAUTH_TOKEN = _cfg.get('oauth_token', None)
+OAUTH_TOKEN_SECRET = _cfg.get('oauth_token_secret', None)
+OAUTH_REALM = _cfg.get('oauth_realm', 'https://api.launchpad.net/')
+
+if not os.path.exists(CRED_FILE_PATH):
+    with open(CRED_FILE_PATH, 'w') as f:
+        f.write('[1]\n')
+        f.write('consumer_key = %s\n' % OAUTH_CONSUMER_KEY)
+        f.write('consumer_secret = \n')
+        f.write('access_token = %s\n' % OAUTH_TOKEN)
+        f.write('access_secret = %s\n' % OAUTH_TOKEN_SECRET)
+
+COMMON_LAUNCHPAD_CACHE_DIR = os.path.join("/tmp", "launchpad.cache")
+if not os.path.isdir(COMMON_LAUNCHPAD_CACHE_DIR):
+    os.makedirs(COMMON_LAUNCHPAD_CACHE_DIR)
+BOT_KEY = "B879A3E9"
+
+# selected arch for building arch:all packages
+VIRTUALIZED_PPA_ARCH = ["i386", "amd64"]
+# an arch we will ignore for publication if latest published version in dest doesn't build it
+ARCHS_TO_EVENTUALLY_IGNORE = set(['powerpc', 'arm64', 'ppc64el'])
+ARCHS_TO_UNCONDITIONALLY_IGNORE = set(['arm64', 'ppc64el'])
+SRU_PPA = "ubuntu-unity/sru-staging"
+
+TIME_BETWEEN_PPA_CHECKS = 60
+TIME_BETWEEN_STACK_CHECKS = 60
+TIME_BEFORE_STOP_LOOKING_FOR_SOURCE_PUBLISH = 20 * 60
+
+PUBLISHER_ARTEFACTS_FILENAME = 'publisher.xml'
+PREPARE_ARTEFACTS_FILENAME_FORMAT = 'prepare_{}.xml'
+
+OLD_STACK_DIR = 'old'
+PACKAGE_LIST_RSYNC_FILENAME_PREFIX = 'packagelist_rsync'
+PACKAGE_LIST_RSYNC_FILENAME_FORMAT = PACKAGE_LIST_RSYNC_FILENAME_PREFIX + '_{}-{}'
+RSYNC_PATTERN = "rsync://RSYNCSVR/cu2d_out/{}*".format(PACKAGE_LIST_RSYNC_FILENAME_PREFIX)
+
+ROOT_CU2D = os.path.join(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))
+DEFAULT_CONFIG_STACKS_DIR = os.path.join(os.path.dirname(ROOT_CU2D), 'cupstream2distro-config', 'stacks')
+STACK_STATUS_FILENAME = "stack.status"
+STACK_STARTED_FILENAME = "stack.started"
+STACK_BUILDING_FILENAME = "stack.building"
+
+STACK_RUNNING_DIR = "/iSCSI/jenkins/cu2d/work"
+STACK_STATUS_PUBLISHING_DIR = "/iSCSI/jenkins/cu2d/result_publishing"
+
+# for citrain
+SILO_NAME_LIST = []
+for i in xrange(1, 11):
+    SILO_NAME_LIST.append("landing-{:03d}".format(i))
+SILO_CONFIG_FILENAME = "config"
+SILO_BUILDPPA_SCHEME = "ci-train-ppa-service/{}"
+SILO_PACKAGING_RELEASE_COMMIT_MESSAGE = "Releasing {}"
+SILOS_RAW_DIR = "~/silos"
+SILOS_DIR = os.path.expanduser(SILOS_RAW_DIR)
+SILO_RSYNCDIR = "~/out"
+SILO_STATUS_RSYNCDIR = os.path.expanduser("~/status")
+CITRAIN_BINDIR = "~/citrain/citrain"
+(SILO_EMPTY, SILO_BUILTCHECKED, SILO_PUBLISHED, SILO_DONE) = range(4)
+
+SERIES_VERSION = {
+    'precise': '12.04',
+    'raring': '13.04',
+    'saucy': '13.10',
+    'trusty': '14.04'
+}

=== added file 'branch-source-builder/cupstream2distro/silomanager.py'
--- branch-source-builder/cupstream2distro/silomanager.py	1970-01-01 00:00:00 +0000
+++ branch-source-builder/cupstream2distro/silomanager.py	2014-02-11 20:59:14 +0000
@@ -0,0 +1,115 @@
+# -*- coding: utf-8 -*-
+# Copyright (C) 2014 Canonical
+#
+# Authors:
+#  Didier Roche
+#
+# This program is free software; you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free Software
+# Foundation; version 3.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+import json
+import logging
+import os
+import shutil
+
+from cupstream2distro.settings import SILO_CONFIG_FILENAME, SILO_NAME_LIST, SILO_STATUS_RSYNCDIR
+from cupstream2distro.utils import ignored
+
+
+def save_config(config, uri=''):
+    """Save config in uri and copy to outdir"""
+    silo_config_path = os.path.abspath(os.path.join(uri, SILO_CONFIG_FILENAME))
+    with ignored(OSError):
+        os.makedirs(uri)
+    try:
+        json.dump(config, open(silo_config_path, 'w'))
+    except TypeError as e:
+        logging.error("Can't save configuration: " + e.message)
+        os.remove(silo_config_path)
+        return False
+    # copy to outdir
+    with ignored(OSError):
+        os.makedirs(SILO_STATUS_RSYNCDIR)
+    silo_name = os.path.dirname(silo_config_path).split(os.path.sep)[-1]
+    dest = os.path.join(SILO_STATUS_RSYNCDIR, silo_name)
+    logging.debug("Copying configuration from {} to {}".format(silo_config_path, dest))
+    shutil.copy2(silo_config_path, os.path.join(SILO_STATUS_RSYNCDIR, silo_name))
+    return True
+
+def load_config(uri=None):
+    """return a loaded config
+
+    If no uri, load in the current directory"""
+    if not uri:
+        uri = os.path.abspath('.')
+    logging.debug("Reading configuration in {}".format(uri))
+    try:
+        return json.load(open(os.path.join(uri, SILO_CONFIG_FILENAME)))
+    # if silo isn't configured
+    except IOError:
+        pass
+    except ValueError as e:
+        logging.warning("Can't load configuration: " + e.message)
+    return None
+
+def remove_status_file(silo_name):
+    """Remove status file"""
+    os.remove(os.path.join(SILO_STATUS_RSYNCDIR, silo_name))
+
+
+def is_project_not_in_any_configs(project_name, series, dest, base_silo_uri, ignore_silo):
+    """Return true if the project for that serie in that dest is not in any configuration"""
+    logging.info("Checking if {} is already configured for {} ({}) in another silo".format(project_name, dest.name, series.name))
+    for silo_name in SILO_NAME_LIST:
+        # we are reconfiguring current silo, ignoring it
+        if ignore_silo == silo_name:
+            continue
+        config = load_config(os.path.join(base_silo_uri, silo_name))
+        if config:
+            if (config["global"]["dest"] == dest.self_link and config["global"]["series"] == series.self_link and
+                (project_name in config["mps"] or project_name in config["sources"])):
+                logging.error("{} is already prepared for the same serie and destination in {}".format(project_name, silo_name))
+                return False
+    return True
+
+
+def return_first_available_silo(base_silo_uri):
+    """Check which silos are free and return the first one"""
+    for silo_name in SILO_NAME_LIST:
+        if not os.path.isfile(os.path.join(base_silo_uri, silo_name, SILO_CONFIG_FILENAME)):
+            return silo_name
+    return None
+
+def get_config_step(config):
+    """Get configuration step"""
+    return config["global"]["step"]
+
+def set_config_step(config, new_step, uri=''):
+    """Set configuration step to new_step"""
+    config["global"]["step"] = new_step
+    return save_config(config, uri)
+
+def set_config_status(config, status, uri='', add_url=True):
+    """Change status to reflect latest status"""
+    build_url = os.getenv('BUILD_URL')
+    if add_url and build_url:
+        status = "{} ({}console)".format(status , build_url)
+    config["global"]["status"] = status
+    return save_config(config, uri)
+
+def get_all_projects(config):
+    """Get a list of all projets"""
+    projects = []
+    projects.extend(config["mps"])
+    projects.extend(config["sources"])
+    return projects

=== added file 'branch-source-builder/cupstream2distro/stack.py'
--- branch-source-builder/cupstream2distro/stack.py	1970-01-01 00:00:00 +0000
+++ branch-source-builder/cupstream2distro/stack.py	2014-02-11 20:59:14 +0000
@@ -0,0 +1,204 @@
+# -*- coding: utf-8 -*-
+# Copyright (C) 2012 Canonical
+#
+# Authors:
+#  Didier Roche
+#
+# This program is free software; you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free Software
+# Foundation; version 3.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+import logging
+import os
+import yaml
+
+from .settings import DEFAULT_CONFIG_STACKS_DIR, STACK_STATUS_FILENAME, STACK_STARTED_FILENAME
+from .utils import ignored
+
+_stacks_ref = {}
+
+# TODO: should be used by a metaclass
+def get_stack(release, stack_name):
+    try:
+        return _stacks_ref[release][stack_name]
+    except KeyError:
+        return Stack(release, stack_name)
+
+class Stack():
+
+    def __init__(self, release, stack_name):
+    	self.stack_name = stack_name
+    	self.release = release
+        self.statusfile = os.path.join('..', '..', release, stack_name, STACK_STATUS_FILENAME)
+        self.startedfile = os.path.join('..', '..', release, stack_name, STACK_STARTED_FILENAME)
+        self.stack_file_path = None
+        self._dependencies = None
+        self._rdependencies = None
+    	for stack_file_path in Stack.get_stacks_file_path(release):
+            if stack_file_path.split(os.path.sep)[-1] == "{}.cfg".format(stack_name):
+                self.stack_file_path = stack_file_path
+                break
+        if not self.stack_file_path:
+            raise Exception("{}.cfg for {} doesn't exist anywhere in {}".format(stack_name, release, self.get_root_stacks_dir()))
+
+        with open(self.stack_file_path, 'r') as f:
+            cfg = yaml.load(f)
+            try:
+                self.forced_manualpublish = cfg['stack']['manualpublish']
+            except (TypeError, KeyError):
+                self.forced_manualpublish = False
+        # register to the global dict
+        _stacks_ref.setdefault(release, {})[stack_name] = self
+
+    def get_status(self):
+        '''Return a stack status
+
+        0 is everything is fine and published
+        1 is the stack failed in a step
+        2 is the stack succeeded, but need manual publishing
+
+        Return None if the status is not available yet'''
+
+        cfg = yaml.load(open(self.stack_file_path))
+        with ignored(KeyError):
+            if cfg['stack']['status_ignored']:
+                return 0
+        if not os.path.isfile(self.statusfile):
+            return None
+        with open(self.statusfile, 'r') as f:
+            return(int(f.read()))
+
+    def is_started(self):
+        '''Return True if the stack is started (dep-wait or building)'''
+        if os.path.isfile(self.startedfile):
+            return True
+        return False
+
+    def is_building(self):
+        '''Return True if the stack is building'''
+        if os.path.isfile(self.startedfile):
+            return True
+        return False
+
+    def is_enabled(self):
+        '''Return True if the stack is enabled for daily release'''
+        with open(self.stack_file_path, 'r') as f:
+            cfg = yaml.load(f)
+            try:
+                if not cfg['stack']['enabled']:
+                    return False
+            except KeyError:
+                pass
+        return True
+
+    def get_direct_depending_stacks(self):
+        '''Get a list of direct depending stacks'''
+        if self._dependencies is not None:
+            return self._dependencies
+
+        with open(self.stack_file_path, 'r') as f:
+            cfg = yaml.load(f)
+            try:
+                deps_list = cfg['stack']['dependencies']
+                self._dependencies = []
+                if not deps_list:
+                    return self._dependencies
+                for item in deps_list:
+                    if isinstance(item, dict):
+                        (stackname, release) = (item["name"], item["release"])
+                    else:
+                        (stackname, release) = (item, self.release)
+                    self._dependencies.append(get_stack(release, stackname))
+                logging.info("{} ({}) dependency list is: {}".format(self.stack_name, self.release, ["{} ({})".format(stack.stack_name, stack.release) for stack in self._dependencies]))
+                return self._dependencies
+            except (TypeError, KeyError):
+                return []
+
+    def get_direct_rdepends_stack(self):
+        '''Get a list of direct rdepends'''
+        if self._rdependencies is not None:
+            return self._rdependencies
+
+        self._rdependencies = []
+        for stackfile in Stack.get_stacks_file_path(self.release):
+            path = stackfile.split(os.path.sep)
+            stack = get_stack(path[-2], path[-1].replace(".cfg", ""))
+            if self in stack.get_direct_depending_stacks():
+                self._rdependencies.append(stack)
+        return self._rdependencies
+
+    def generate_dep_status_message(self):
+        '''Return a list of potential problems from others stack which should block current publication'''
+
+        # TODO: get the first Stack object
+        # iterate over all stacks objects from dep chain
+        # call get_status on all of them
+
+        global_dep_status_info = []
+        for stack in self.get_direct_depending_stacks():
+            logging.info("Check status for {} ({})".format(stack.stack_name, stack.release))
+            status = stack.get_status()
+            message = None
+            # We should have a status for every stack
+            if status is None:
+                message = "Can't find status for {depstack} ({deprel}). This shouldn't happen unless the stack is currently running. If this is the case, it means that the current stack shouldn't be uploaded as the state is unknown.".format(depstack=stack, deprel=stack.release)
+            elif status == 1:
+                message = '''{depstack} ({deprel}) failed to publish. Possible causes are:
+        * the stack really didn't build/can't be prepared at all.
+        * the stack has integration tests not working with this previous stack.
+
+    What needs to be done:
+    Either:
+        * If we want to publish both stacks: retry the integration tests for {depstack} ({deprel}), including components from this stack (check with the whole PPA). If that works, both stacks should be published at the same time.
+    Or:
+        * If we only want to publish this stack: check that we can safely publish it by itself (e.g. without the stacks it depends on). The trick there is to make sure that the stack is not relying on, or affected by, a change that happened in one of its dependencies. Example: if the {depstack} ({deprel}) API changed in a way that affects any component of the current stack, and both stacks got updated in trunk, we need to make sure we don't land only one of the two stacks which would result in a broken state. Also think about potential ABI changes.'''.format(depstack=stack.stack_name, deprel=stack.release)
+            elif status == 2:
+                message = '''{depstack} ({deprel}) is in manually publish mode. Possible causes are:
+        * Some part of the stack has packaging changes
+        * This stack is depending on another stack not being published
+
+    What needs to be done:
+    Either:
+        * If {depstack} ({deprel}) can be published, we should publish both stacks at the same time.
+    Or:
+        * If we only want to publish this stack: check that we can safely publish it by itself (e.g. without the stacks it depends on). The trick there is to make sure that the stack is not relying on, or affected by, a change that happened in one of its dependencies. Example: if the {depstack} ({deprel}) API changed in a way that affects any component of the current stack, and both stacks got updated in trunk, we need to make sure we don't land only one of the two stacks which would result in a broken state. Also think about potential ABI changes.'''.format(depstack=stack.stack_name, deprel=stack.release)
+            elif status == 3 or status == -1:
+                message = '''{depstack} ({deprel}) has been manually aborted or failed for an unknown reason. Possible causes are:
+        * A job of this stack was stopped manually
+        * Jenkins had an internal error/shutdown
+
+    What needs to be done:
+        * If we only want to publish this stack: check that we can safely publish it by itself (e.g. without the stacks it depends on). The trick there is to make sure that the stack is not relying on, or affected by, a change that happened in one of its dependencies. Example: if the {depstack} ({deprel}) API changed in a way that affects any component of the current stack, and both stacks got updated in trunk, we need to make sure we don't land only one of the two stacks which would result in a broken state. Also think about potential ABI changes.'''.format(depstack=stack.stack_name, deprel=stack.release)
+
+            if message:
+                logging.warning(message)
+                global_dep_status_info.append(message)
+        return global_dep_status_info
+
+    @staticmethod
+    def get_root_stacks_dir():
+	    '''Get root stack dir'''
+	    return os.environ.get('CONFIG_STACKS_DIR', DEFAULT_CONFIG_STACKS_DIR)
+
+    @staticmethod
+    def get_stacks_file_path(release):
+        '''Return an iterator with all path for every discovered stack files'''
+        for root, dirs, files in os.walk(os.path.join(Stack.get_root_stacks_dir(), release)):
+            for candidate in files:
+                if candidate.endswith('.cfg'):
+                    yield os.path.join(root, candidate)
+
+    @staticmethod
+    def get_current_stack():
+        '''Return current stack object based on current path (release/stackname)'''
+        path = os.getcwd().split(os.path.sep)
+        return get_stack(path[-2], path[-1])

=== added file 'branch-source-builder/cupstream2distro/stacks.py'
--- branch-source-builder/cupstream2distro/stacks.py	1970-01-01 00:00:00 +0000
+++ branch-source-builder/cupstream2distro/stacks.py	2014-02-11 20:59:14 +0000
@@ -0,0 +1,89 @@
+# -*- coding: utf-8 -*-
+# Copyright (C) 2012 Canonical
+#
+# Authors:
+#  Didier Roche
+#
+# This program is free software; you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free Software
+# Foundation; version 3.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+import logging
+import os
+import yaml
+import subprocess
+
+from .settings import PACKAGE_LIST_RSYNC_FILENAME_PREFIX, RSYNC_PATTERN
+from .tools import get_packaging_diff_filename
+from .stack import Stack
+
+
+def _rsync_stack_files():
+    '''rsync all stack files'''
+    server = os.getenv('CU2D_RSYNCSVR')
+    if server == "none":
+        return
+    elif server:
+        remoteaddr = RSYNC_PATTERN.replace('RSYNCSVR', server)
+    else:
+        raise Exception('Please set environment variable CU2D_RSYNCSVR')
+
+    cmd = ["rsync", '--remove-source-files', '--timeout=60', remoteaddr, '.']
+    instance = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (stdout, stderr) = instance.communicate()
+    if instance.returncode not in (0, 23):
+        raise Exception(stderr.decode("utf-8").strip())
+
+
+def get_stack_files_to_sync():
+    '''Return a list of tuple: (file, release)'''
+    _rsync_stack_files()
+    for file in os.listdir('.'):
+        if file.startswith(PACKAGE_LIST_RSYNC_FILENAME_PREFIX):
+            yield (file, file.split('-')[-1])
+
+
+def get_allowed_projects(release):
+    '''Get all projects allowed to be uploaded for this release'''
+
+    projects = []
+    for file_path in Stack.get_stacks_file_path(release):
+        with open(file_path, 'r') as f:
+            cfg = yaml.load(f)
+            try:
+                projects_list = cfg['stack']['projects']
+            except (TypeError, KeyError):
+                logging.warning("{} seems broken in not having stack or projects keys".format(file_path))
+                continue
+            if not projects_list:
+                logging.warning("{} don't have any project list".format(file_path))
+                continue
+            for project in projects_list:
+                if isinstance(project, dict):
+                    projects.append(project.keys()[0])
+                else:
+                    projects.append(project)
+    return set(projects)
+
+def get_stack_packaging_change_status(source_version_list):
+    '''Return global package change status list
+
+    # FIXME: added too many infos now, should only be: (source, version)
+    source_version_list is a list of couples (source, version, tip_rev, target_branch)'''
+
+    packaging_change_status = []
+    for (source, version, tip_rev, target_branch) in source_version_list:
+        if os.path.exists(get_packaging_diff_filename(source, version)):
+            message = "Packaging change for {} ({}).".format(source, version)
+            logging.warning(message)
+            packaging_change_status.append(message)
+    return packaging_change_status

=== added file 'branch-source-builder/cupstream2distro/tools.py'
--- branch-source-builder/cupstream2distro/tools.py	1970-01-01 00:00:00 +0000
+++ branch-source-builder/cupstream2distro/tools.py	2014-02-11 20:59:14 +0000
@@ -0,0 +1,94 @@
+# -*- coding: utf-8 -*-
+# Copyright (C) 2012 Canonical
+#
+# Authors:
+#  Didier Roche
+#
+# This program is free software; you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free Software
+# Foundation; version 3.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+import ConfigParser
+import glob
+import os
+import shutil
+from xml.sax.saxutils import quoteattr, escape
+
+from .settings import PROJECT_CONFIG_SUFFIX
+from .utils import ignored
+
+WRAPPER_STRING = '''<testsuite errors="0" failures="{}" name="" tests="1" time="0.1">
+  <testcase classname="MarkUnstable" name={} time="0.0">{}</testcase>
+</testsuite>'''
+
+
+def generate_xml_artefacts(test_name, details, filename):
+    '''Generate a fake test name xml result for marking the build as unstable'''
+    failure = ""
+    errnum = 0
+    for detail in details:
+        errnum = 1
+        failure += '    <failure type="exception">{}</failure>\n'.format(escape(detail))
+    if failure:
+        failure = '\n{}'.format(failure)
+
+    with open(filename, 'w') as f:
+        f.write(WRAPPER_STRING.format(errnum, quoteattr(test_name), failure))
+
+
+def get_previous_distro_version_from_config(source_package_name):
+    '''Get previous packaging version which was in bzr from the saved config'''
+    config = ConfigParser.RawConfigParser()
+    config.read("{}.{}".format(source_package_name, PROJECT_CONFIG_SUFFIX))
+    return config.get('Package', 'dest_current_version')
+
+
+def save_project_config(source_package_name, branch, revision, dest_current_version, current_packaging_version):
+    '''Save branch and package configuration'''
+    config = ConfigParser.RawConfigParser()
+    config.add_section('Branch')
+    config.set('Branch', 'branch', branch)
+    config.set('Branch', 'rev', revision)
+    config.add_section('Package')
+    config.set('Package', 'dest_current_version', dest_current_version)
+    config.set('Package', 'packaging_version', current_packaging_version)
+    with open("{}.{}".format(source_package_name, PROJECT_CONFIG_SUFFIX), 'wb') as configfile:
+        config.write(configfile)
+
+
+def get_packaging_diff_filename(source_package_name, packaging_version):
+    '''Return the packaging diff filename'''
+
+    return "packaging_changes_{}_{}.diff".format(source_package_name, packaging_version)
+
+
+def mark_project_as_published(source_package_name, packaging_version):
+    '''Rename .project and eventual diff files so that if we do a partial rebuild, we don't try to republish them'''
+    project_filename = "{}.{}".format(source_package_name, PROJECT_CONFIG_SUFFIX)
+    os.rename(project_filename, "{}_{}".format(project_filename, packaging_version))
+    diff_filename = get_packaging_diff_filename(source_package_name, packaging_version)
+    if os.path.isfile(diff_filename):
+        os.rename(diff_filename, "{}.published".format(diff_filename))
+
+
+def clean_source(source):
+    """clean all related source content from current silos"""
+    with ignored(OSError):
+        shutil.rmtree(source)
+    with ignored(OSError):
+        os.remove("{}.{}".format(source, PROJECT_CONFIG_SUFFIX))
+    with ignored(OSError):
+        shutil.rmtree("ubuntu/{}".format(source))
+    for filename in glob.glob("{}_*".format(source)):
+        os.remove(filename)
+    for filename in glob.glob("packaging_changes_{}_*diff".format(source)):
+        os.remove(filename)

=== added file 'branch-source-builder/cupstream2distro/utils.py'
--- branch-source-builder/cupstream2distro/utils.py	1970-01-01 00:00:00 +0000
+++ branch-source-builder/cupstream2distro/utils.py	2014-02-11 20:59:14 +0000
@@ -0,0 +1,29 @@
+# -*- coding: utf-8 -*-
+# Copyright (C) 2013 Canonical
+#
+# Authors:
+#  Didier Roche
+#
+# This program is free software; you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free Software
+# Foundation; version 3.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+from contextlib import contextmanager
+
+
+# this is stolen from python 3.4 :)
+@contextmanager
+def ignored(*exceptions):
+    try:
+        yield
+    except exceptions:
+        pass

=== modified file 'branch-source-builder/run_worker'
--- branch-source-builder/run_worker	2014-01-06 19:47:57 +0000
+++ branch-source-builder/run_worker	2014-02-11 20:59:14 +0000
@@ -18,7 +18,10 @@
 import logging
 import os
 import sys
+import time
 
+import upload_package
+import watch_ppa
 
 logging.basicConfig(level=logging.INFO)
 log = logging.getLogger(__name__)
@@ -27,23 +30,59 @@
 # and add it, so we can always safely import stuff
 sys.path.append(os.path.join(os.path.dirname(__file__), '../ci-utils'))
 from ci_utils import amqp_utils
-
+TIME_BETWEEN_CHECKS = 60
 
 def on_message(msg):
-    log.info('on_message: %s', msg.body)
+    log.info('on_message: {}'.format(msg.body))
     params = json.loads(msg.body)
     sources = params['source_packages']
     ppa = params['ppa']
+    log.info('The PPA is: {}'.format(ppa))
     trigger = params['progress_trigger']
+
+    # Setup the output data to send back to the caller
+    # TODO: sources_packages are just the passed in source files, this
+    #       can be made to be more useful.
+    # TODO: artifacts will be populated with artifacts from this build.
+    out_data = {'source_packages': sources,
+                'ppa': ppa,
+                'artifacts': []}
     amqp_utils.progress_update(trigger, params)
 
-    # TODO build stuff
-
-    if amqp_utils.progress_completed(trigger):
-        log.error('Unable to notify progress-trigger completition of action')
-
-    # remove from queue so request becomes completed
-    msg.channel.basic_ack(msg.delivery_tag)
+    # TODO Replace these hardcoded parameters with params from the message
+    archive_ppa = 'ppa:ci-engineering-airline/ci-archive'
+    series = 'saucy'
+    try:
+        upload_list = upload_package.upload_source_packages(ppa, sources)
+        log.info('upload_list: {}'.format(upload_list))
+        start_time = time.time()
+        while True:
+            (ret, status) = watch_ppa.watch_ppa(start_time, series, ppa,
+                                                archive_ppa, None,
+                                                upload_list)
+            progress = {}
+            for key in status:
+                progress[key] = str(status[key])
+            log.info('progress: {}'.format(progress))
+            out_data['status'] = progress
+            amqp_utils.progress_update(trigger, out_data)
+            if ret == -1:
+                log.info('Going to sleep for {}'.format(TIME_BETWEEN_CHECKS))
+                time.sleep(TIME_BETWEEN_CHECKS)
+            else:
+                log.info('All done')
+                break
+
+        amqp_utils.progress_completed(trigger, out_data)
+    except Exception as e:
+        error_msg = 'Exception: {}'.format(e)
+        log.error(error_msg)
+        out_data['error_message'] = error_msg
+        amqp_utils.progress_failed(trigger, out_data)
+    finally:
+        # remove from queue so request becomes completed
+        log.info('Acking the request: {}'.format(msg.body))
+        msg.channel.basic_ack(msg.delivery_tag)
 
 
 if __name__ == '__main__':

=== added file 'branch-source-builder/upload_package.py'
--- branch-source-builder/upload_package.py	1970-01-01 00:00:00 +0000
+++ branch-source-builder/upload_package.py	2014-02-11 20:59:14 +0000
@@ -0,0 +1,137 @@
+#!/usr/bin/env python
+# Ubuntu CI Engine
+# Copyright 2013 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import argparse
+import atexit
+from dput.changes import parse_changes_file
+import json
+import logging
+import os
+import re
+import shutil
+import sys
+import tempfile
+import urllib2
+
+from cupstream2distro import packagemanager
+
+
+def parse_arguments():
+    parser = argparse.ArgumentParser(
+        description='Source package upload handler.')
+    parser.add_argument('-p', '--ppa',
+                        required=True,
+                        help='Target PPA for source package upload.')
+    parser.add_argument('-s', '--source',
+                        required=True,
+                        action='append',
+                        help='A source package file to include in the upload.')
+    return parser.parse_args()
+
+
+def cleanup(directory):
+     if os.path.exists(directory) and os.path.isdir(directory):
+         logging.info('Removing temp directory: {}'.format(directory))
+         shutil.rmtree(directory)
+
+
+def create_temp_dir():
+    directory = tempfile.mkdtemp()
+    atexit.register(cleanup, directory)
+    return directory
+
+
+def get_url_contents(url):
+    try:
+        req = urllib2.Request(url)
+        f = urllib2.urlopen(req)
+        response = f.read()
+        f.close()
+        return response
+    except IOError as e:
+        raise EnvironmentError('Failed to open url [{}]: {}'.format(url, e))
+
+
+def get_source_files(source_files):
+    location = create_temp_dir()
+    local_files = []
+    for source in source_files:
+        # Co-locate all the files into a temp directory
+        file_name = os.path.basename(source)
+        file_path = os.path.join(location, file_name)
+        logging.info('Retrieving source file: {}'.format(file_name))
+        with open(file_path, 'w') as out_file:
+            out_file.write(get_url_contents(source))
+        local_files.append(file_name)
+    return (location, local_files)
+
+
+def parse_dsc_file(dsc_file):
+    regexp = re.compile("^Architecture: (.*)\n")
+    with open(dsc_file) as in_file:
+        for line in in_file:
+            arch_lists = regexp.findall(line)
+            if arch_lists:
+                return arch_lists[0]
+
+
+def parse_source_files(source_directory, source_files):
+    package_list = []
+    for source in source_files:
+        source_name = os.path.join(source_directory, source)
+        if source.endswith('changes'):
+                changes = parse_changes_file(filename=source_name,
+                                             directory=source_directory)
+                package = {}
+                package['name'] = changes.get('Source')
+                package['version'] = changes.get('Version')
+                package['files'] = []
+                package['files'].append(changes.get_changes_file())
+                for package_file in changes.get_files():
+                    if os.path.exists(package_file):
+                        if package_file.endswith('.dsc'):
+                            package['architecture'] = parse_dsc_file(
+                                package_file)
+                        package['files'].append(package_file)
+                    else:
+                        raise EnvironmentError(
+                            'Not found: {}'.format(package_file))
+                package_list.append(package)
+    return package_list
+
+
+def upload_source_packages(ppa, upload_files):
+    '''Attempts source file upload into the PPA.'''
+    logging.info('Upload to the ppa: {}'.format(ppa))
+    (source_directory, source_files) = get_source_files(upload_files)
+    source_packages = parse_source_files(source_directory, source_files)
+    for package in source_packages:
+        packagemanager.upload_package(package['name'], package['version'],
+                                      ppa, source_directory)
+    # TODO Return the data set of packages uploaded
+    return source_packages
+
+
+def main():
+    logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s "
+                        "%(message)s")
+
+    args = parse_arguments()
+    upload_source_packages(args.ppa, args.source)
+
+
+if __name__ == '__main__':
+    sys.exit(main())

=== added file 'branch-source-builder/watch_ppa.py'
--- branch-source-builder/watch_ppa.py	1970-01-01 00:00:00 +0000
+++ branch-source-builder/watch_ppa.py	2014-02-11 20:59:14 +0000
@@ -0,0 +1,182 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+# Copyright (C) 2012 Canonical
+#
+# Authors:
+#  Didier Roche
+#
+# This program is free software; you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free Software
+# Foundation; version 3.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+
+import argparse
+import logging
+import os
+import sys
+import time
+
+from cupstream2distro import (launchpadmanager, packageinppamanager)
+from cupstream2distro.packageinppa import PackageInPPA
+from cupstream2distro.packagemanager import list_packages_info_in_str
+from cupstream2distro.settings import (
+    TIME_BETWEEN_PPA_CHECKS, TIME_BEFORE_STOP_LOOKING_FOR_SOURCE_PUBLISH)
+
+
+def parse_arguments():
+    parser = argparse.ArgumentParser(
+        description="Watch for published package in a ppa",
+        epilog="""series and ppa options can be set by the corresponding long
+        option name env variables as well""")
+    parser.add_argument("-s", "--series",
+                        help="Serie used to build the package")
+    parser.add_argument("-p", "--ppa",
+                        help="""PPA to publish this package to (for instance:
+                        'ubuntu-unity/daily-build')""")
+    parser.add_argument("-a", "--arch",
+                        default=None,
+                        help="Only consider the provided target")
+    parser.add_argument("-d", "--destppa",
+                        help="""Consider this destppa instead of
+                        {series}-proposed""")
+    return parser.parse_args()
+
+
+def get_environment_variables(args):
+    series = args.series
+    ppa = args.ppa
+    if not series:
+        series = os.getenv("series")
+    if not ppa:
+        ppa = os.getenv("ppa")
+    return (series, ppa)
+
+
+def get_ppa(ppa):
+    return launchpadmanager.get_ppa(ppa)
+
+
+def watch_ppa(time_start, series, ppa, dest_ppa, arch, upload_list):
+    # Prepare launchpad connection:
+    lp_series = launchpadmanager.get_series(series)
+    monitored_ppa = launchpadmanager.get_ppa(ppa)
+    if dest_ppa:
+        dest_archive = get_ppa(dest_ppa)
+    else:
+        dest_archive = launchpadmanager.get_ubuntu_archive()
+    logging.info('Series: {}'.format(lp_series))
+    logging.info('Monitoring PPA: {}'.format(monitored_ppa))
+
+    logging.info('Destination Archive: {}'.format(dest_archive))
+
+    # Get archs available and archs to ignore
+    (available_archs_in_ppa,
+     arch_all_arch) = launchpadmanager.get_available_and_all_archs(
+         lp_series, monitored_ppa)
+    (archs_to_eventually_ignore,
+     archs_to_unconditionally_ignore) = launchpadmanager.get_ignored_archs()
+    logging.info('Arches available in ppa: {}'.format(available_archs_in_ppa))
+    logging.info('All arch in ppa: {}'.format(arch_all_arch))
+    logging.info('Arches to eventually ignore: {}'.format(
+        archs_to_eventually_ignore))
+    logging.info('Arches to unconditionally ignore: {}'.format(
+        archs_to_unconditionally_ignore))
+
+    # Collecting all packages that have been uploaded to the ppa
+    packages_not_in_ppa = set()
+    packages_building = set()
+    packages_failed = set()
+    for source_package in upload_list:
+        source = source_package['name']
+        version = source_package['version']
+        archs = source_package['architecture']
+        logging.info('Inspecting upload: {} - {}'.format(source, version))
+        packages_not_in_ppa.add(PackageInPPA(source, version, monitored_ppa,
+                                             dest_archive, lp_series,
+                                             available_archs_in_ppa,
+                                             arch_all_arch,
+                                             archs_to_eventually_ignore,
+                                             archs_to_unconditionally_ignore,
+                                             package_archs=archs))
+
+    # packages_not_in_ppa are packages that were uploaded and are expeceted
+    # to eventually appear in the ppa.
+    logging.info('Packages not in PPA: {}'.format(
+        list_packages_info_in_str(packages_not_in_ppa)))
+    logging.info('Packages building: {}'.format(packages_building))
+    logging.info('Packages failed: {}'.format(packages_failed))
+
+    # Check the status regularly on all packages
+    # TODO The following is the original check loop. This can be extracted
+    #    and optimized.
+    logging.info("Checking the status for {}".format(
+        list_packages_info_in_str(
+            packages_not_in_ppa.union(packages_building))))
+    packageinppamanager.update_all_packages_status(
+        packages_not_in_ppa, packages_building, packages_failed, arch)
+
+    status = {'pending': packages_not_in_ppa,
+              'building': packages_building,
+              'failed': packages_failed}
+    # if we have no package building or failing and have wait for
+    # long enough to have some package appearing in the ppa, exit
+    if (packages_not_in_ppa and not packages_building and
+        ((time.time() - time_start) >
+         TIME_BEFORE_STOP_LOOKING_FOR_SOURCE_PUBLISH)):
+        # TODO return error on the missing packages
+        logging.info(
+            "Some source packages were never published in the ppa: "
+            "{}".format(list_packages_info_in_str(packages_not_in_ppa)))
+        return (1, status)
+
+    # break out of status check loop if all packages have arrived in
+    # the ppa and have completed building
+    if not packages_not_in_ppa and not packages_building:
+        if packages_failed:
+            # TODO Return package failure info
+            logging.info(
+                "Some of the packages failed to build: {}".format(
+                    list_packages_info_in_str(packages_failed)))
+            return (1, status)
+        return (0, status)
+
+    # -1 indicates to retry
+    # TODO return useful status about what is still in progress
+    return (-1, status)
+
+
+def main():
+    '''Provides usage through the command line.'''
+    logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s "
+                        "%(message)s")
+
+    args = parse_arguments()
+    (series, ppa) = get_environment_variables(args)
+
+    if not series or not ppa:
+        logging.error("Missing compulsory environment variables (ppa, series) "
+                      "watching: {}, series: {}".format(ppa, series))
+        return 1
+
+    # TODO UPLOAD LIST is not ready for use via the CLI yet.
+    # The original cu2d tools extracted this from files stored locally
+    time_start = time.time()
+    while True:
+        (ret, status) = watch_ppa(time_start, series, ppa,
+                                  args.destppa, args.arch, UPLOAD_LIST)
+        if ret == -1:
+            time.sleep(TIME_BETWEEN_PPA_CHECKS)
+        else:
+            return ret
+
+
+if __name__ == '__main__':
+    sys.exit(main())

=== modified file 'charms/precise/lander-jenkins/templates/jobs/lander_master.xml'
--- charms/precise/lander-jenkins/templates/jobs/lander_master.xml	2014-02-06 21:13:48 +0000
+++ charms/precise/lander-jenkins/templates/jobs/lander_master.xml	2014-02-11 20:59:14 +0000
@@ -168,13 +168,6 @@
 child_number=$TRIGGERED_BUILD_NUMBER_lander_branch_source_builder
 wget -O params.json "${JENKINS_URL}job/${LAST_TRIGGERED_JOB_NAME}/${child_number}/artifact/results/params.json"
 
-# TODO - temporary until the bsbuilder works properly
-cat > params.json <<EOF
-{
-  "ppa": "ppa:phablet-team/tools"
-}
-EOF
-
 /srv/lander_jenkins_sub/lander/bin/lander_merge_parameters.py --result-file params.json --service bsbuilder --output-file all.json --prior-file all.json
 
 # Convert to a format that can be passed to the child jobs

=== modified file 'juju-deployer/branch-source-builder.yaml'
--- juju-deployer/branch-source-builder.yaml	2014-01-31 09:15:34 +0000
+++ juju-deployer/branch-source-builder.yaml	2014-02-11 20:59:14 +0000
@@ -18,6 +18,9 @@
             options:
                 branch: lp:ubuntu-ci-services-itself
                 main: ./branch-source-builder/run_worker
+                unit-config: include-base64://configs/unit_config.yaml
+                packages: dput
+                pip-packages: dput
         rabbit:
             branch: lp:~canonical-ci-engineering/charms/precise/ubuntu-ci-services-itself/rabbitmq-server
             charm: rabbitmq