Merge lp:~robru/bileto/enable-britney into lp:bileto

[Robert Bruce Park]

Run britney inside bileto.

Experimental addition of britney to bileto instance.

[Robert Bruce Park (robru)]

This is so close to done I can smell it.

TODO:

* Expose britney output in flask

* Figure out why vivid doesn't work since xenial is working, and also figure out why the logging shows some things but not python tracebacks. I guess STDERR is getting lost somewhere along the way.

[Robert Bruce Park (robru)]

Ok, this is looking great for a first iteration:

* All requests in qa state 'Ready for QA' are automatically submitted for autopkgtests.
* Results automatically appear in the request page, with links to the excuses page.
* Publication is not blocked on this being successful
* The huge archive indexes that britney consumes are downloaded in parallel
* each britney is run in series due to incredible memory consumption (I'll be increasing the production unit from 2GB to 8GB when this goes live)

review: Approve

[Robert Bruce Park (robru)]

Here is the successful test run:

https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/landing-015/+packages

(note to self, make train post this on MRs)

[Robert Bruce Park (robru)]

Also here's the staging ticket with the links to the excuses pages:

https://requests.ci-train.staging.ubuntu.com/#/ticket/1

And the log:

https://requests.ci-train.staging.ubuntu.com/static/britney/last-run.txt

[Martin Pitt (pitti)]

> * each britney is run in series due to incredible memory consumption (I'll be increasing the production unit from 2GB to 8GB when this goes live)

Yes, it does that I'm afraid -- it slurps in the entire Packages_*/Sources indexes as Python dictionaries so that it can make efficient installability checks, reverse and forward dependency calculations, etc. So its RAM usage can't be significantly reduced.

Why does the ticket say "Ready for QA" while tests are still running, and thus packages are still "not considered"? That smells like a bug in evaluating excuses.yaml?

[Martin Pitt (pitti)]

A few nits, but by and large LGTM. I suggest adding s390x in a separate MP to not make this too messy.

review: Approve

[lp:~robru/bileto/enable-britney]

432. By Robert Bruce Park on 2015-12-14

Permanently enable verbose output for now.

433. By Robert Bruce Park on 2015-12-14

Fix amqp uri leak.

[Robert Bruce Park (robru)]

some replies inline.

[lp:~robru/bileto/enable-britney]

434. By Robert Bruce Park on 2015-12-14

Record a log of bzr update.

435. By Robert Bruce Park on 2015-12-14

Run autopkgtests on s390x also.

436. By Robert Bruce Park on 2015-12-14

Stupid bzr logging on stderr.

437. By Robert Bruce Park on 2015-12-14

Use series-specific configs with different arches.

438. By Robert Bruce Park on 2015-12-14

Hard-code s390x in ADT_ARCHES as it is ignored pre-xenial.

439. By Robert Bruce Park on 2015-12-14

Fix s390x in xenial.

440. By Robert Bruce Park on 2015-12-14

Drop back to every 15 minutes in cron.

=== added directory '.bzr-builddeb'
=== added file '.bzr-builddeb/default.conf'
--- .bzr-builddeb/default.conf	1970-01-01 00:00:00 +0000
+++ .bzr-builddeb/default.conf	2015-12-14 21:05:53 +0000
@@ -0,0 +1,2 @@
+[BUILDDEB]
+split = True

=== modified file 'Makefile'
--- Makefile	2015-11-09 23:57:30 +0000
+++ Makefile	2015-12-14 21:05:53 +0000
@@ -1,6 +1,10 @@
 envsubst = perl -pe 's/@([A-Z_]+)@/$$ENV{$$1} or die "$$1 undefined"/eg'
 apt = apt-get install -yq --force-yes
 
+britney_branch = lp:~ubuntu-release/britney/britney2-ubuntu
+britney_path = /var/lib/britney
+britney_prefix = /tmp/britney_
+
 check:
 	python3 -m coverage run -m nose2
 	python3 -m coverage report --include="t*/*" --show-missing --fail-under=100
@@ -13,19 +17,24 @@
 	./run.py
 
 install-common-deps:
-	$(apt) python3-launchpadlib python3-flask python3-flask-openid python3-flask-sqlalchemy python3-sqlalchemy python3-gunicorn
+	$(apt) python3-requests python3-launchpadlib python3-flask python3-flask-openid python3-flask-sqlalchemy python3-sqlalchemy python3-gunicorn
 
 install-test-deps: install-common-deps
 	$(apt) python3-pyflakes || $(apt) pyflakes
 	$(apt) pylint3 python3-pep8 python3-nose2 python3-coverage python3-mock
 
 install: install-common-deps
-	$(apt) uuid python3-psycopg2 gunicorn3 supervisor postgresql-client
+	$(apt) uuid python3-yaml python3-amqplib python3-psycopg2 gunicorn3 supervisor postgresql-client
 	$(envsubst) <files/supervisor.conf >/etc/supervisor/conf.d/bileto.conf
 	cp files/logrotate.conf /etc/logrotate.d/bileto
 	cp files/expire-sessions.sh /etc/cron.hourly/expire-sessions
+	[ -d "$(britney_path)" ] || bzr checkout --lightweight $(britney_branch) $(britney_path)
+	bzr update "$(britney_path)" 2>&1 | tee "$(britney_path).info"
+	cp britney/cron /etc/cron.d/bileto-britney
+	cp britney/expire.sh /etc/cron.daily/expire-britney-output
 
 install-user:
 	[ -f "$$HOME/bileto.api.token" ] || uuid > "$$HOME/bileto.api.token"
 	uuid > "$$HOME/bileto.instance.id"
+	mkdir --parents $(britney_prefix)data $(britney_prefix)output
 	./db_migrations.sh

=== added directory 'britney'
=== added file 'britney/Makefile'
--- britney/Makefile	1970-01-01 00:00:00 +0000
+++ britney/Makefile	2015-12-14 21:05:53 +0000
@@ -0,0 +1,6 @@
+envsubst = perl -pe 's/@([A-Z_]+)@/$$ENV{$$1} or die "$$1 undefined"/eg'
+input = /var/lib/bileto/britney/britney.conf.in
+
+%.conf: $(input) ~/amqp.uri
+	$(envsubst) <$(input) >$@.new
+	mv $@.new $@

=== added file 'britney/britney.conf.in'
--- britney/britney.conf.in	1970-01-01 00:00:00 +0000
+++ britney/britney.conf.in	2015-12-14 21:05:53 +0000
@@ -0,0 +1,58 @@
+# Configuration file for britney
+
+# Paths for control files
+TESTING           = /tmp/britney_data/%(SERIES)
+UNSTABLE          = /tmp/britney_data/@PPA@-%(SERIES)
+PARTIAL_UNSTABLE  = yes
+
+# Output
+NONINST_STATUS    = /tmp/britney_output/%(SERIES)/@PPA@/non-installable-status
+EXCUSES_OUTPUT    = /tmp/britney_output/%(SERIES)/@PPA@/excuses.html
+EXCUSES_YAML_OUTPUT = /tmp/britney_output/%(SERIES)/@PPA@/excuses.yaml
+UPGRADE_OUTPUT    = /tmp/britney_output/%(SERIES)/@PPA@/output.txt
+HEIDI_OUTPUT      = /tmp/britney_output/%(SERIES)/@PPA@/HeidiResult
+
+# List of release architectures
+ARCHITECTURES     = @ARCHES@ @PORTS_ARCHES@
+
+# if you're not in this list, arch: all packages are allowed to break on you
+NOBREAKALL_ARCHES = amd64
+
+# if you're in this list, your packages may not stay in sync with the source
+OUTOFSYNC_ARCHES  =
+
+# if you're in this list, your uninstallability count may increase
+BREAK_ARCHES      =
+
+# if you're in this list, you are a new architecture
+NEW_ARCHES        = @NEW_ARCHES@
+
+# priorities and delays
+MINDAYS_LOW       = 0
+MINDAYS_MEDIUM    = 0
+MINDAYS_HIGH      = 0
+MINDAYS_CRITICAL  = 0
+MINDAYS_EMERGENCY = 0
+DEFAULT_URGENCY   = medium
+
+# support for old libraries in testing (smooth update)
+# use ALL to enable smooth updates for all the sections
+#
+# naming a non-existent section will effectively disable new smooth
+# updates but still allow removals to occur
+SMOOTH_UPDATES    = badgers
+
+REMOVE_OBSOLETE   = no
+
+ADT_ENABLE        = yes
+ADT_DEBUG         = no
+ADT_ARCHES        = amd64 i386 armhf ppc64el s390x
+ADT_PPAS          = @TEAM@/@OVERLAY@ @TEAM@/@PPA@
+ADT_AMQP          = @AMQP_URI@
+# Swift base URL with the results (must be publicly readable and browsable)
+ADT_SWIFT_URL     = https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac
+
+BOOTTEST_ENABLE   = no
+BOOTTEST_DEBUG    = yes
+BOOTTEST_ARCHES   = armhf amd64
+BOOTTEST_FETCH    = yes

=== added file 'britney/cron'
--- britney/cron	1970-01-01 00:00:00 +0000
+++ britney/cron	2015-12-14 21:05:53 +0000
@@ -0,0 +1,2 @@
+# Trigger britney every 15 minutes
+*/15 * * * * www-data python3 /var/lib/bileto/britney/iterate.py

=== added file 'britney/expire.sh'
--- britney/expire.sh	1970-01-01 00:00:00 +0000
+++ britney/expire.sh	2015-12-14 21:05:53 +0000
@@ -0,0 +1,8 @@
+#!/bin/sh
+DIR="/tmp/britney_output"
+
+# Delete britney output files that are more than 3 months old.
+find $DIR -type f -mmin +7776000 -delete
+
+# Delete empty directories too.
+find $DIR -empty -type d -delete

=== added file 'britney/fetch-indexes'
--- britney/fetch-indexes	1970-01-01 00:00:00 +0000
+++ britney/fetch-indexes	2015-12-14 21:05:53 +0000
@@ -0,0 +1,139 @@
+#!/bin/sh
+# download current package indexes to data/<series>{,-proposed}/ for running
+# britney against a PPA. The PPA will play the role of "-proposed" (i. e.
+# "unstable" in britney terms, containing the updated packages to test), the
+# Ubuntu archive has the "-release" part (i. e. "testing" in britney terms, in
+# which the -proposed packages are being landed).
+#
+# Author: Martin Pitt <[e-mail]>
+# Author: Robert Bruce Park <[e-mail]>
+set -u
+
+export OVERLAY="stable-phone-overlay"
+export AMQP_URI=$(cat ~/amqp.uri)
+export TEAM=$(cat ~/ppa.team)
+export ARCHES="i386 amd64"
+export PORTS_ARCHES="armhf arm64 powerpc ppc64el"
+
+BRITNEY=/var/lib/britney/britney.py
+CODEDIR=$(dirname "$(readlink -f "$0")")
+DATADIR=/tmp/britney_data
+OUTDIR=/tmp/britney_output
+CACHE=$DATADIR/CACHE
+ARCHIVE=http://archive.ubuntu.com/ubuntu/dists
+PORTS=http://ports.ubuntu.com/dists
+
+usage() {
+cat <<EOF
+Usage: $0 SERIES [SILONAME]
+$1
+
+SERIES:
+    Ubuntu release codename, eg trusty, xenial, etc.
+
+SILONAME (optional):
+    Name of PPA to fetch; defaults to stable-phone-overlay.
+EOF
+exit 1
+}
+
+log() {
+    echo "$(date -Iseconds)" "$@"
+}
+
+# Echo something before running it.
+# Needed because 'set -x' goes to stderr which I don't want.
+loudly() {
+    log "$@"
+    "$@"
+}
+
+# Download files in parallel in background, only if there is an update
+refresh() {
+    DIR=$CACHE/$pocket/$(echo $1 | rev | cut --delimiter=/ --fields=2,3 | rev)
+    mkdir --parents $DIR
+    wget --directory-prefix $DIR --timestamping $1 --append-output $DIR/$$-wget-log --no-verbose &
+}
+
+[ $# -gt 0 ] && export SERIES="$1" && shift || usage "Missing series."
+
+if [ $# -gt 0 ]; then
+    export PPA="$1"
+else
+    export PPA="$OVERLAY"
+fi
+
+PPA_URL=http://ppa.launchpad.net/$TEAM/$PPA/ubuntu/dists/$SERIES/main
+
+# Enable s390x only for xenial or newer.
+case "$SERIES" in
+    precise|trusty|vivid|wily)
+        export NEW_ARCHES=" "
+        ;;
+    *)
+        export NEW_ARCHES="s390x"
+        export PORTS_ARCHES="$PORTS_ARCHES s390x"
+        ;;
+esac
+
+log 'Refreshing package indexes...'
+
+# Get archive bits
+if [ "$PPA" = "$OVERLAY" ]; then
+    for pocket in $SERIES $SERIES-updates $SERIES-proposed; do
+        for component in main restricted universe multiverse; do
+            for arch in $ARCHES; do
+                refresh $ARCHIVE/$pocket/$component/binary-$arch/Packages.gz
+            done
+            for arch in $PORTS_ARCHES; do
+                refresh $PORTS/$pocket/$component/binary-$arch/Packages.gz
+            done
+            refresh $ARCHIVE/$pocket/$component/source/Sources.bz2
+        done
+    done
+fi
+
+# Get ppa bits
+pocket=$PPA-$SERIES
+for arch in $ARCHES $PORTS_ARCHES; do
+    refresh $PPA_URL/binary-$arch/Packages.gz
+done
+refresh $PPA_URL/source/Sources.bz2
+
+wait  # for wgets to finish
+
+find $DATADIR -name "$$-wget-log*" -exec cat '{}' \; -delete
+
+log 'Building britney indexes...'
+
+if [ "$PPA" = "$OVERLAY" ]; then
+    # "Testing" is archive + stable overlay ppa
+    DEST=$DATADIR/$SERIES
+    mkdir --parents $DEST
+    bzcat $CACHE/$SERIES*/*/source/Sources.bz2 \
+          $CACHE/$OVERLAY-$SERIES/*/source/Sources.bz2 \
+          > $DEST/Sources
+    for arch in $ARCHES $PORTS_ARCHES; do
+        zcat $CACHE/$SERIES*/*/binary-$arch/Packages.gz \
+             $CACHE/$OVERLAY-$SERIES/*/binary-$arch/Packages.gz \
+             > $DEST/Packages_${arch}
+    done
+else
+    # "Unstable" is just silo PPA
+    DEST=$DATADIR/$PPA-$SERIES
+    mkdir --parents $DEST
+    bzcat $CACHE/$PPA-$SERIES/*/source/Sources.bz2 > $DEST/Sources
+    for arch in $ARCHES $PORTS_ARCHES; do
+        zcat $CACHE/$PPA-$SERIES/*/binary-$arch/Packages.gz > $DEST/Packages_${arch}
+    done
+    touch $DEST/Blocks
+
+    # Create config file atomically.
+    CONFIG="$DEST.conf"
+    make --file "$CODEDIR/Makefile" "$CONFIG"
+
+    mkdir --parents "$OUTDIR/$SERIES/$PPA/"
+    loudly $BRITNEY -v --config "$CONFIG" --series $SERIES | sed "s#$AMQP_URI#AMQP_URI#"
+fi
+
+log "$0 done."

=== added file 'britney/iterate.py'
--- britney/iterate.py	1970-01-01 00:00:00 +0000
+++ britney/iterate.py	2015-12-14 21:05:53 +0000
@@ -0,0 +1,150 @@
+#!/usr/bin/env python3
+
+"""Britney runner.
+
+This script iterates over all appropriate bileto requests, downloads archive
+indexes, and runs britney with those indexes as inputs.
+"""
+
+import sys
+import requests
+
+from glob import glob
+from time import time
+from json import dumps
+from shutil import copy2
+from subprocess import Popen, PIPE
+from collections import defaultdict
+from fcntl import LOCK_EX, LOCK_NB, flock
+from os.path import abspath, dirname, expanduser
+
+
+PUBLIC = '/static/britney/'
+OUTPUT = '/tmp/britney_output/'
+ATOMIC = OUTPUT + 'last-run.txt'
+LOG_NAME = OUTPUT + 'log.txt'
+LOCK_FD = open(LOG_NAME, 'a')
+sys.stdout = sys.stderr = LOCK_FD
+
+ORIG_EXCEPTHOOK = sys.excepthook
+API_PREFIX = 'http://127.0.0.1:8080/v1/'
+READ_API = API_PREFIX + 'silo/landing'
+WRITE_API = API_PREFIX + 'tickets?token='
+HOME = dirname(abspath(__file__))
+FETCH = HOME + '/fetch-indexes'
+EXCUSES = OUTPUT + '*/{}/excuses.html'
+SLASH = '/'
+PLUS = '+'
+SPACE = ' '
+EMPTY = ''
+BNEWLINE = b'\n'
+NEWLINE = '\n'
+
+
+def cleanup(*args, **kwargs):
+    """Actions that must be run no matter how badly everything exploded."""
+    LOCK_FD.close()
+    copy2(LOG_NAME, ATOMIC)
+    ORIG_EXCEPTHOOK(*args, **kwargs)
+
+
+def get_ppa(siloname):
+    """Extract just the ppa name from the siloname."""
+    return siloname[siloname.index(SLASH) + 1:]
+
+
+def read_token():
+    """Read API token."""
+    with open(expanduser('~/bileto.api.token')) as token:
+        return token.read().strip()
+
+
+def one_at_a_time():
+    """Prevent this script from running multiple times."""
+    flock(LOCK_FD, LOCK_EX | LOCK_NB)
+
+
+def fetch_indexes(*args):
+    """Invoke the index-fetcher script with the given args."""
+    return Popen((FETCH,) + args, stdout=PIPE, stderr=PIPE)
+
+
+def get_requests():
+    """Return list of all active requests from Bileto."""
+    resp = requests.get(READ_API).json()
+    return resp.get('requests', [])
+
+
+def update_requests(reqs, token):
+    """Send excuses links to bileto."""
+    for req in reqs:
+        request_id = req['request_id']
+        landing = get_ppa(req['siloname'])
+        files = glob(EXCUSES.format(landing))
+        if files:
+            new_req = dict(
+                request_id=request_id,
+                autopkgtest=NEWLINE.join(
+                    name.replace(OUTPUT, PUBLIC) for name in files))
+            resp = requests.post(
+                WRITE_API + token, dumps(new_req),
+                headers={'content-type': 'application/json'})
+            print('Updating request {}: {} {}'.format(
+                request_id, resp.status_code, resp.reason))
+
+
+def build_siloname_dict(reqs):
+    """Build a dict mapping series to silonames."""
+    runs = defaultdict(list)
+    for req in reqs:
+        landing = get_ppa(req['siloname'])
+        if 'Ready' in req['qa_signoff']:
+            for series in req['series'].split(PLUS):
+                runs[series].append(landing)
+    return runs
+
+
+def parallelize_fetching(silos):
+    """Invoke fetchers in parallel but serialize output on stdout."""
+    procs = []
+    # Parallelize most of the fetching...
+    for series in sorted(silos):
+        procs.append(fetch_indexes(series))
+    # ... serialize the output ...
+    for proc in procs:
+        print_proc(proc)
+    # ... and serialize the britney invocations
+    for series, silonames in sorted(silos.items()):
+        for siloname in sorted(silonames):
+            print_proc(fetch_indexes(series, siloname))
+
+
+def print_proc(proc):
+    """Print the output from a subprocess."""
+    sys.stdout.buffer.write(BNEWLINE.join(
+        (b'', SPACE.join(proc.args).encode('utf-8')) + proc.communicate()))
+
+
+def main():
+    """Handle errors."""
+    start = time()
+    token = read_token()
+    try:
+        one_at_a_time()
+        LOCK_FD.seek(0)
+        LOCK_FD.truncate()
+        reqs = get_requests()
+        parallelize_fetching(build_siloname_dict(reqs))
+        update_requests(reqs, token)
+    except BlockingIOError:
+        return 0
+    except (IOError, ConnectionRefusedError):
+        print('Failed to connect to Bileto. Is it running?')
+    print('\n\nCompleted in {:0.1f}s.'.format(time() - start))
+    cleanup()
+    return 0
+
+
+if __name__ == '__main__':
+    sys.excepthook = cleanup
+    sys.exit(main())

=== modified file 'db_migrations.sh'
--- db_migrations.sh	2015-11-19 23:38:57 +0000
+++ db_migrations.sh	2015-12-14 21:05:53 +0000
@@ -3,7 +3,7 @@
 # Super ultra-lightweight db migrations crafted by hand because the necessary
 # flask/sqlalchemy migrations bits aren't available for python3 in trusty.
 
-exit 0  # No migrations are currently required.
+# exit 0  # No migrations are currently required.
 
 CONNECTION=$(tr ':/@' ' ' < "$HOME/bileto.postgresql.info")
 USER=$(echo "$CONNECTION" | awk '{ print $2 }')
@@ -26,26 +26,30 @@
     $PG "SELECT pg_terminate_backend($pid);"
 done
 
+
+$PG "ALTER TABLE request ADD COLUMN autopkgtest text DEFAULT '';" || true
+
+rm -rf "$PGPASSFILE"
+exit 0
+
 # These ones have succeeded in production and only kept for reference
-# for column in job_log creator landers siloname sync_request published_versions distribution series dest qa_signoff; do
-#     $PG "ALTER TABLE request ALTER COLUMN $column SET DATA TYPE text;"
-# done
-# $PG "ALTER TABLE comment ALTER COLUMN siloname SET DATA TYPE text;"
-
-# $PG "ALTER TABLE comment ALTER COLUMN request_id SET DATA TYPE bigint;"
-# $PG "ALTER TABLE comment ALTER COLUMN id SET DATA TYPE bigint;"
-# $PG "ALTER TABLE request ALTER COLUMN request_id SET DATA TYPE bigint;"
-
-# $PG "ALTER TABLE comment ADD COLUMN log_url text DEFAULT '';" || true
-# $PG "ALTER TABLE comment ADD COLUMN published_versions text DEFAULT '';" || true
-# $PG "ALTER TABLE comment ADD COLUMN siloname varchar(1000) DEFAULT '';" || true
-
-# $PG "ALTER TABLE request ADD COLUMN job_log varchar(1000) DEFAULT '';" || true
-# $PG "ALTER TABLE request ADD COLUMN creator varchar(1000) DEFAULT '';" || true
-
-# for column in status sources search; do
-#     $PG "ALTER TABLE request ALTER COLUMN $column SET DATA TYPE text;"
-# done
-# $PG "ALTER TABLE comment ALTER COLUMN author SET DATA TYPE text;"
-
-rm -rf "$PGPASSFILE"
+for column in job_log creator landers siloname sync_request published_versions distribution series dest qa_signoff; do
+    $PG "ALTER TABLE request ALTER COLUMN $column SET DATA TYPE text;"
+done
+$PG "ALTER TABLE comment ALTER COLUMN siloname SET DATA TYPE text;"
+
+$PG "ALTER TABLE comment ALTER COLUMN request_id SET DATA TYPE bigint;"
+$PG "ALTER TABLE comment ALTER COLUMN id SET DATA TYPE bigint;"
+$PG "ALTER TABLE request ALTER COLUMN request_id SET DATA TYPE bigint;"
+
+$PG "ALTER TABLE comment ADD COLUMN log_url text DEFAULT '';" || true
+$PG "ALTER TABLE comment ADD COLUMN published_versions text DEFAULT '';" || true
+$PG "ALTER TABLE comment ADD COLUMN siloname varchar(1000) DEFAULT '';" || true
+
+$PG "ALTER TABLE request ADD COLUMN job_log varchar(1000) DEFAULT '';" || true
+$PG "ALTER TABLE request ADD COLUMN creator varchar(1000) DEFAULT '';" || true
+
+for column in status sources search; do
+    $PG "ALTER TABLE request ALTER COLUMN $column SET DATA TYPE text;"
+done
+$PG "ALTER TABLE comment ALTER COLUMN author SET DATA TYPE text;"

=== modified file 'debian/control'
--- debian/control	2015-12-09 06:31:56 +0000
+++ debian/control	2015-12-14 21:05:53 +0000
@@ -3,6 +3,7 @@
 Priority: extra
 Maintainer: Robert Bruce Park <[e-mail]>
 Build-Depends: debhelper (>= 9),
+               language-pack-en,
                pylint3,
                python3-coverage,
                python3-flask,

=== modified file 'tests/test_v1.py'
--- tests/test_v1.py	2015-11-20 20:25:42 +0000
+++ tests/test_v1.py	2015-12-14 21:05:53 +0000
@@ -117,11 +117,11 @@
             '/v1/tickets',
             data=json.dumps(dict(request_id=1, description='bar')),
             content_type='application/json')
-        self.assertEqual(ret.status_code, 201)
+        self.assertEqual(ret.status_code, 200)
         req = Request.query.get(1)
         self.assertEqual(req.description, 'bar')
         self.assertEqual(
-            req.search, 'fritz\nbar\n\n\n\n\n\n\n\n\n\nsiloname\n\nNew\n\n')
+            req.search, '\nfritz\nbar\n\n\n\n\n\n\n\n\n\nsiloname\n\nNew\n\n')
         self.assertEqual(Comment.query.get(1).text, 'Updated description.')
 
     def test_update_ticket_doesnt_exist(self):
@@ -451,6 +451,9 @@
         ret = self.app.get('/v1/metadata')
         self.assertDictEqual(parse(ret.data), dict(
             docs=dict(
+                autopkgtest='Autopkgtest Results\n\nThis field will '
+                            'automatically be filled with links to autopkgtest'
+                            ' results.',
                 comments='Audit Log',
                 creator='Request Creator\n\nLaunchpad nickname of the person '
                         'who created this request.',
@@ -493,7 +496,7 @@
             datalists=dict(
                 dest=[
                     '',
-                    'ci-train-ppa-service/ubuntu/stable-phone-overlay',
+                    'team-not-set/ubuntu/stable-phone-overlay',
                 ],
                 distribution=['ubuntu'],
                 qa_signoff=QA_SIGNOFF,
@@ -508,7 +511,9 @@
                     'ppa:~team/ubuntu/ppa,vivid',
                 ]),
             jenkins='https://ci-train.staging.ubuntu.com/',
+            ppa_team='team-not-set',
             titles=dict(
+                autopkgtest='Autopkgtest Results',
                 comments='Audit Log',
                 creator='Request Creator',
                 date='Creation Date',

=== modified file 'tickets/models.py'
--- tickets/models.py	2015-11-20 20:25:42 +0000
+++ tickets/models.py	2015-12-14 21:05:53 +0000
@@ -26,6 +26,10 @@
 
 PUBLISHABLE = ('publish without qa', 'qa granted')
 
+AUTOPKGTEST = """Autopkgtest Results
+
+This field will automatically be filled with links to autopkgtest results."""
+
 CREATOR = """Request Creator
 
 Launchpad nickname of the person who created this request."""
@@ -139,8 +143,7 @@
 
 class Request(db.Model, BiletoModel):
     """Contains all data associated with one landing request."""
-    request_id = db.Column(
-        BigInt, primary_key=True, doc='Request ID')
+    request_id = db.Column(BigInt, primary_key=True, doc='Request ID')
     date = db.Column(db.DateTime, default=right_now, doc='Creation Date')
 
     test_plan = db.Column(db.Text, default='', doc=TEST_PLAN)
@@ -148,6 +151,7 @@
     download_links = db.Column(db.Text, default='', doc=MANUAL_URLS)
     merge_proposals = db.Column(db.Text, default='', doc=MERGES)
 
+    autopkgtest = db.Column(db.Text, default='', doc=AUTOPKGTEST)
     job_log = db.Column(db.Text, default='', doc=JOB_LOG)
     creator = db.Column(db.Text, default='', doc=CREATOR)
     landers = db.Column(db.Text, default='', doc=LANDERS)

=== modified file 'tickets/settings.py'
--- tickets/settings.py	2015-10-19 19:05:19 +0000
+++ tickets/settings.py	2015-12-14 21:05:53 +0000
@@ -22,6 +22,7 @@
 SQLITE_PATH = expanduser('~/bileto.db')
 PG_PATH = expanduser('~/bileto.postgresql.info')
 JENKINS_PATH = expanduser('~/bileto.jenkins.url')
+PPA_TEAM_PATH = expanduser('~/ppa.team')
 
 
 def read_from_file(path):
@@ -31,6 +32,7 @@
             return data.read().strip()
 
 
+PPA_TEAM = read_from_file(PPA_TEAM_PATH) or 'team-not-set'
 API_TOKEN = read_from_file(API_PATH)
 PG_INFO = read_from_file(PG_PATH)
 JENKINS_INFO = read_from_file(

=== modified file 'tickets/static/app.js'
--- tickets/static/app.js	2015-11-18 20:54:13 +0000
+++ tickets/static/app.js	2015-12-14 21:05:53 +0000
@@ -1,5 +1,5 @@
 var EXCUSES = 'http://people.canonical.com/~ubuntu-archive/proposed-migration/';
-var PPA_PREFIX = 'https://launchpad.net/~ci-train-ppa-service/+archive/';
+var PPA_PREFIX = 'https://launchpad.net/~';
 var SOURCE_PREFIX = 'https://launchpad.net/ubuntu/+source/';
 var TEAM_PREFIX = 'https://launchpad.net/~';
 
@@ -33,6 +33,7 @@
     text = text || '';
     return text.replace(/\n/g, '<br />')
                .replace(/(#|bug)\s?(\d+)/g, ' http://pad.lv/$2')
+               .replace(/\/static\/[-A-Z0-9+&@#/%?=~_|!:,.;]*[A-Z0-9+&@#/%=~_|]/gi, shorter)
                .replace(/https?:\/\/[-A-Z0-9+&@#/%?=~_|!:,.;]*[A-Z0-9+&@#/%=~_|]/gi, shorter);
 }
 
@@ -45,10 +46,9 @@
     return text;
 }
 
-function display_siloname(text) {
+function display_siloname(ppa_team, text) {
     text = (text || '');
-    // TODO: Make PPA link link to correct staging/production PPA team
-    return a(PPA_PREFIX + text, text)
+    return a(PPA_PREFIX + ppa_team + '/+archive/' + text, text)
 }
 
 function link_excuses(match, packagelist) {

=== added symlink 'tickets/static/britney'
=== target is u'/tmp/britney_output'
=== modified file 'tickets/static/index.html'
--- tickets/static/index.html	2015-11-20 19:48:04 +0000
+++ tickets/static/index.html	2015-12-14 21:05:53 +0000
@@ -93,7 +93,7 @@
 
 <tr>
 <th>{{meta.titles.siloname}}</th>
-<td ng-bind-html="display_siloname(req.siloname)"></td>
+<td ng-bind-html="display_siloname(meta.ppa_team, req.siloname)"></td>
 </tr>
 
 <tr>
@@ -116,6 +116,11 @@
 <td>{{req[field]}}</td>
 </tr>
 
+<tr ng-show="req.autopkgtest">
+<th>{{meta.titles.autopkgtest}}</th>
+<td ng-bind-html="linkify(req.autopkgtest)"></td>
+</tr>
+
 <tr>
 <th>{{meta.titles.landers}}</th>
 <td ng-bind-html="display_landers(req.landers)"></td>

=== removed directory 'tickets/static/json'
=== modified file 'tickets/v1.py'
--- tickets/v1.py	2015-11-19 18:59:59 +0000
+++ tickets/v1.py	2015-12-14 21:05:53 +0000
@@ -25,7 +25,7 @@
 from tickets.teams import Teams
 from tickets.lplib import get_series, memoize
 from tickets.models import PUBLISHABLE, Request, Comment
-from tickets.settings import API_TOKEN, JENKINS_INFO, read_from_file
+from tickets.settings import API_TOKEN, JENKINS_INFO, PPA_TEAM, read_from_file
 
 
 LOGFILE = '/var/log/bileto.log'
@@ -120,6 +120,7 @@
     authenticate()
     assert_json()
     requestid = request.json.get('request_id')
+    success = 200 if requestid else 201
     if not requestid:
         req = Request()
         db.session.add(req)
@@ -145,7 +146,7 @@
             db.session.commit()
     req.update(**request.json)
     db.session.commit()
-    return jsonify(dict(request_id=req.request_id)), 201
+    return jsonify(dict(request_id=req.request_id)), success
 
 
 @app.route('/v1/tickets', methods=['GET'])
@@ -247,8 +248,7 @@
         datalists=dict(
             distribution=['ubuntu'],
             series=['xenial+vivid'] + series,
-            dest=[
-                '', 'ci-train-ppa-service/ubuntu/stable-phone-overlay'],
+            dest=['', '{}/ubuntu/stable-phone-overlay'.format(PPA_TEAM)],
             sync_request=['ubuntu,' + s for s in series] + [
                 '0',
                 'stable-overlay,vivid',
@@ -260,6 +260,7 @@
             ) + HIDDEN,
             qa_signoff=QA_SIGNOFF),
         jenkins=JENKINS_INFO,
+        ppa_team=PPA_TEAM,
     ))