Merge lp:~bloodearnest/canonical-identity-provider/sso-dev into lp:canonical-identity-provider/release

[Simon Davy]

Merge juju dev work
Notes:

1) This MP supports side-by-side dev envs. The juju one is enabled with
   I_CAN_HAZ_JUJU=yes env var.

The following files are forks specific to juju dev env. When the juju dev
env becomes default, these will replace/merge with their original files.

./django_project/settings_acceptance_juju.py
./django_project/settings_devel_juju.py
./scripts/acceptance-dev-juju.sh
./run-tests-juju
./Makefile.juju
./README.juju

2) The charm is checked out into ../sso-charm, and then used to deploy.
   You can deploy a custom charm branch by specifying CHARM_BRANCH. The
   following directories are bind mounted into the container

   $PWD -> /srv/login.ubuntu.com/devel/code/devel
   $PWD/logs -> /srv/login.ubuntu.com/devel/logs
   $PWD/django_project -> /srv/login.ubuntu.com/devel/etc
   $HOME -> $HOME

   This means all logs will be available in ./logs, and the charm will
   generate it's config into ./django_project, for easy inspection. $HOME
   mounting is mainly for convienince.

3) bootstrap: this now effectively does two things, both separate make
   targets in their own right.

   make dependencies - this is what the old bootstrap did (venv, cfgmgr,
                       wheels, install), plus it now also installs system
                       packages.

   make deploy - this will setup juju repo and envoronemtn and deploy
                       sso, and is idempotent. If you are already deployed,
                       it's pretty fast.

   So, for getting set up first time, we still use make bootstrap. But once
   we're set up, we probably want to use make dependencies/deploy as needed,
   but bootstrap will still work.

   We shouldn't need to teardown the juju env often. If we do, make
   juju-destroy will do that, and deploy will create it again and deploy.

4) config has changed quite a bit to support developing with the charm, but
   has also simplified a lot in the process, for both the charm and sso
   trunk.

   - We use explicit DJANGO_SETTINGS_MODULE values, which is more in-line
     with normal django usage.

     e.g. DJANGO_SETTINGS_MODULE=django_project.settings_devel_juju

     This is exposed on the charm, and allows for explicit staging/prod
     settings (in other projects, anyway - sso doesn't use them because it's
     open source)

   - all these modules import from 'settings', which are the charm generated
     settings.

   - the symlink from django_project/settings.py to ../../local_settings is
     gone (along with related issues)

   - A developer can create and modify django_project/settings_local.py to
     modify settings in development - this file is bzrignored.

5) DB: postgres now runs as system postgres in the container, configured via
   the charm and very close to the old db dev config. The big difference is
   that it's not running on /dev/shm, as this added significant complexity
   for little benefit. Happy to revisit this if people find it an issue.

   So:

   - start-db/stop-db are gone as now obsolete
   - reset-db/setup-db are preserved, but are the same command - drop the db
     and recreate it, run migrations
   - have proposed db-setup/db-reset as better tab-completion friendly
     names, but kept old names too

6) gunicorn: now runs constantly as charm configured service in the
   container, on port 8080, like prod, but with code reloading enabled.

   - gunicorn logs are in ./logs dir
   - For debugging, use make run as before, It stops the system gunicorn,
     and then runs it manually in the foreground on the same port. It uses
     the same config, but timeout is increased and logging is directed to
     stdout. When you've finished debugging, it will start gunicorn again
     in the container.

7) localmail: this is now installed by default in the container, and the dev
   configuration uses it by default. No more need to manually start up an
   smtp server.

   - logs will be in ./logs/localmail.log
   - mbox file at ./logs/localmail.mbox
   - simple web view at port 8880 on container

8) acceptance tests have been simplified a lot in the process

 - in dev, we no longer need to run sso/localmail in screen - they are
   always running

 - all server and client side config for dev acceptance tests was moved to
   the default development config. This means no config changes are need
   to run acceptance tests against a local dev server. This involved changing the default test emails, so as not to leak it, and couple of test changes to match

   No need to juggle files or change the server config at all :)

 - For staging prod, we use an explicit config to run the tests

   DJANGO_SETTINGS_MOD=django_project.settings_acceptace_juju

 - the acceptance db setup was moved into the Makefile

9) The Makefiles has seen a lot of work. It is more complex that it needs to
   be, in order to support side-by-side dev envs, but that should only be
   temporary.

  - Makefile.common - the tasks that do sso stuff, shared between old and
    new dev environments.

  - Makefile.db - old env db tasks

  - Makefile - new entrypoint that sshs into container if needed, and houses
    the commands to manage the lxc. If using old env, it just includes
    Makefile.common and Makefile.db, and the old env's definition of run and
    bootstrap.

  - Makefile.juju - the new juju specific commands, mainly about charms and
    deploying.

  Once this work has fully landed, it is anticipated that we will return to
  just a single Makefile, which will include a base common ols Makefile as
  a dependency.

  I also added a make help command, since we'd discussed the idea before, and I was adding a whole bunch of new commands, so it seemed like a good plan

[Daniel Manrique (roadmr)]

Awesome, thanks for this.

Maybe you want review by someone more familiar with all the makefile wizardry. In general it looks OK to me, the code looks sensible and the thing works very nicely (and I tried some uncommon scenarios). I made some minor comments below, mainly typos. There's one thing that may need clarification so I didn't "Approve" outright.

review: Needs Information

[Simon Davy (bloodearnest)]

Yeah, some of the makefile stuff is more complex that I'd like. Make is so good at many things, but it has some rough edges. I'm pretty comfortable with make, but I know not every one is. I'm hoping vila might spot some improvements I can make.

Are there any specific bits which look too much like magic?

[Simon Davy (bloodearnest)]

I've fixed the various typos too.

[Daniel Manrique (roadmr)]

+1 from me but don't forget to add a commit message :)

As for the makefile, nothing looks *too* magical, it's just unfamiliar due to rustiness with Makefiles. Maybe the way targets.mk is generated, but tbh I haven't looked at it too closely and it works anyway \o/.

review: Approve

[Simon Davy (bloodearnest)]

FYI, the juju deployer config we inherit from the charm is here:

https://bazaar.launchpad.net/~ubuntuone-pqm-team/canonical-is-charms/canonical-identity-provider/view/head:/deploy.yaml

[lp:~bloodearnest/canonical-identity-provider/sso-dev]

1366. By Simon Davy on 2015-11-16

add comment about order-only make pre-requisites

[lp:~bloodearnest/canonical-identity-provider/sso-dev]

1367. By Simon Davy on 2015-11-16

- whitespace
- remove unneeded sessionid config

1368. By Simon Davy on 2015-11-16

re-add missing SSH_ARGS

1369. By Simon Davy on 2015-11-17

 - add ssh_config file to avoid unwieldy cli args
 - fix location of .juju-repo

1370. By Simon Davy on 2015-11-17

clean up juju repo

[Natalia Bidart (nataliabidart)]

One more question, I ran:

$ make lint
make: /home/nessita/canonical/bloodearnest/sso-dev/env/bin/flake8: Command not found
Makefile.common:152: recipe for target 'lint' failed

any idea what is missing?

[Simon Davy (bloodearnest)]

On Tue, Nov 17, 2015 at 3:18 PM, Natalia Bidart
<email address hidden> wrote:
>
>
> Diff comments:
>
>>
>> === renamed file 'Makefile' => 'Makefile.common'
>> --- Makefile 2015-11-12 17:08:24 +0000
>> +++ Makefile.common 2015-11-17 13:23:47 +0000
>> @@ -1,9 +1,9 @@
>> -# Copyright (C) 2014 Canonical Ltd.
>> -
>> +# Copyright (C) 2014 Canonical Ltd.
>
> Since you are editing this file, would you mind updating the Copyright notice to 2014-2015?

Done

>> +#
>> CM ?= /usr/lib/config-manager/cm.py
>> CONFIGMANAGER = config-manager.txt
>> CONN_CHECK_CONFIG_PATH ?= /tmp/sso_conn_check_config.yaml
>> -DJANGO_SETTINGS_MODULE = settings
>> +DJANGO_SETTINGS_MODULE ?= settings
>> ENV = $(CURDIR)/env
>> JUJU_ENV ?= local
>> JUJU_REPO ?= ../.juju-repo
>> @@ -42,27 +42,32 @@
>> $(TARBALL_BUILD_DIR):
>> @mkdir -p $(TARBALL_BUILD_DIR)
>>
>> +## run django's collectstatic command, STATIC_ROOT defaults to ./staticfiles
>> collectstatic:
>> # this uses a modified collectstatic command, called linkstatic, where the
>> # --link option uses relative urls, so save space when tarballed
>> @DJANGO_SETTINGS_MODULE=django_project.settings_build $(DJANGO_MANAGE) linkstatic --noinput --link
>>
>> +## create a pre-gzipped version of static text assets
>> zipstatic:
>> @echo Gzipping static asset files...
>> @find $(STATIC_ROOT) -name \*.js -or -name \*.css -or -name \*.svg | xargs gzip -kf
>>
>> -# variation of compilemessages target that uses settings_build
>> +## variation of compilemessages target that uses settings_build
>> compilemessages-build:
>> @DJANGO_SETTINGS_MODULE=django_project.settings_build $(MAKE) compilemessages
>>
>> # note: fetch-sourcedeps needs to be manually called before this will work.
>> +## build a deployment tarbal
>
> Typo here? tarball

Fixed

>> build-tarball: install-wheels $(TARBALL_BUILD_PATH)
>>
>> +## write the bzr revno to lib/versioninfo.py
>> version:
>> bzr version-info --format=python > lib/versioninfo.py
>>
>> ### Wheels ###
>>
>> +## create/reinitialise the virtualenv
>> virtualenv:
>> @virtualenv --clear --system-site-packages $(ENV)
>>
>>
>> === added file 'Makefile.juju'
>> --- Makefile.juju 1970-01-01 00:00:00 +0000
>> +++ Makefile.juju 2015-11-17 13:23:47 +0000
>> @@ -0,0 +1,124 @@
>> +# sso specific config
>> +CHARM_SERIES = trusty
>> +CHARM = canonical-identity-provider
>> +CHARM_BRANCH = lp:~ubuntuone-pqm-team/canonical-is-charms/canonical-identity-provider
>> +CHARM_SRC = ../sso-charm
>> +DEPLOYER_CONFIG = $(CURDIR)/django_project/deploy.yaml
>> +DEPLOYER_TARGET = sso-app-dev
>> +
>> +# generic config
>> +LOG_DIRS ?= ./logs/www-oops ./logs/schema-updates
>> +ABS_CHARM_SRC = $(shell readlink -f $(CHARM_SRC))
>> +LOCAL_JUJU_CONFIG = .local.yaml
>> +JUJU_REPOSITORY ?= $(CURDIR)/.juju-repo
>> +JUJU_HOME ?= $(HOME)/.juju
>> +JENV = $(JUJU_HOME)/environments/$(JUJU_ENV).jenv
>> +export JUJU_REPOSITORY JUJU_ENV
>> +export I_CAN_HAZ_JUJU=yes
>> +
>> +# python config, overriding the non-juju config
>> +PYTHONPATH := $(LXC_BASEDIR)/etc:$(SRC_DIR):$(LIB_DIR):$(CURDIR)
>> +export PYTHONPATH
>> +
>> +
>> +
>> +$(CHARM_SRC):
>> + @bzr branch $...

[Simon Davy (bloodearnest)]

On Tue, Nov 17, 2015 at 3:18 PM, Natalia Bidart
<email address hidden> wrote:
> One more question, I ran:
>
> $ make lint
> make: /home/nessita/canonical/bloodearnest/sso-dev/env/bin/flake8: Command not found
> Makefile.common:152: recipe for target 'lint' failed
>
> any idea what is missing?

Assuming you've bootstrapped (or just make dependencies), not sure. I
haven;t altered any of the dev deps or make targets beyond bootrstrap
(which is now dependecnies) and make run.

It works for me here.

[Natalia Bidart (nataliabidart)]

More errors. On the host, on a sso-dev clean branch, I tried:

nessita@dali:~/canonical/bloodearnest/sso-dev$ export I_CAN_HAZ_JUJU=yes
nessita@dali:~/canonical/bloodearnest/sso-dev$ make clean
(Makefiles changed - rebuilding lxc target list)
unknown option -- -
usage: ssh [-1246AaCfGgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]
           [-D [bind_address:]port] [-E log_file] [-e escape_char]
           [-F configfile] [-I pkcs11] [-i identity_file]
           [-L [bind_address:]port:host:hostport] [-l login_name] [-m mac_spec]
           [-O ctl_cmd] [-o option] [-p port]
           [-Q cipher | cipher-auth | mac | kex | key]
           [-R [bind_address:]port:host:hostport] [-S ctl_path] [-W host:port]
           [-w local_tun[:remote_tun]] [user@]hostname [command]
Makefile:106: recipe for target 'clean' failed
make: *** [clean] Error 255

The main issue I see with this errors is that I have no idea what is going on, nor I can debug in a easy way.

[Simon Davy (bloodearnest)]

On Tue, Nov 17, 2015 at 9:05 PM, Natalia Bidart
<email address hidden> wrote:
> More errors. On the host, on a sso-dev clean branch, I tried:
>
> nessita@dali:~/canonical/bloodearnest/sso-dev$ export I_CAN_HAZ_JUJU=yes
> nessita@dali:~/canonical/bloodearnest/sso-dev$ make clean
> (Makefiles changed - rebuilding lxc target list)
> unknown option -- -
> usage: ssh [-1246AaCfGgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]
> [-D [bind_address:]port] [-E log_file] [-e escape_char]
> [-F configfile] [-I pkcs11] [-i identity_file]
> [-L [bind_address:]port:host:hostport] [-l login_name] [-m mac_spec]
> [-O ctl_cmd] [-o option] [-p port]
> [-Q cipher | cipher-auth | mac | kex | key]
> [-R [bind_address:]port:host:hostport] [-S ctl_path] [-W host:port]
> [-w local_tun[:remote_tun]] [user@]hostname [command]
> Makefile:106: recipe for target 'clean' failed
> make: *** [clean] Error 255

Hmm, this suggests your lxd is not set up correctly, it doesn't have
an IP currently.

The ssh command is on line 106:

@ssh -qtAF $(SSH_CONFIG_FILE) $(IP) -- $(MAKE) $(LXC_MAKE_FLAGS) -C
$(PWD) $@ $(LXC_MAKE_VARS)

I suspect the issue is that the $(IP) is coming out as empty, which is
confusing ssh

I will try and add an explicit check for an IP address, with suitable
error message. Previously, I have just been checking the container was
created, not also a) it was running and b) it has an IP address, which
are all things that are required so ssh in, but can fail.

Additionally, clean should be able to be run from host without ssh,
which I will fix

Would echoing the ssh command by default be better? It would make
things clearer in case of error, but at the expense of largely
irrelevant line noise on most command output.

--

Thanks

[lp:~bloodearnest/canonical-identity-provider/sso-dev]

1371. By Simon Davy on 2015-11-18

 - make clean now works in host w/o ssh
 - add explicit check for IP address for better error messages

[Simon Davy (bloodearnest)]

On Wed, Nov 18, 2015 at 10:34 AM, Simon Davy <email address hidden> wrote:
> On Tue, Nov 17, 2015 at 9:05 PM, Natalia Bidart
> <email address hidden> wrote:
>> More errors. On the host, on a sso-dev clean branch, I tried:
>>
>> nessita@dali:~/canonical/bloodearnest/sso-dev$ export I_CAN_HAZ_JUJU=yes
>> nessita@dali:~/canonical/bloodearnest/sso-dev$ make clean
>> (Makefiles changed - rebuilding lxc target list)
>> unknown option -- -
>> usage: ssh [-1246AaCfGgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]
>> [-D [bind_address:]port] [-E log_file] [-e escape_char]
>> [-F configfile] [-I pkcs11] [-i identity_file]
>> [-L [bind_address:]port:host:hostport] [-l login_name] [-m mac_spec]
>> [-O ctl_cmd] [-o option] [-p port]
>> [-Q cipher | cipher-auth | mac | kex | key]
>> [-R [bind_address:]port:host:hostport] [-S ctl_path] [-W host:port]
>> [-w local_tun[:remote_tun]] [user@]hostname [command]
>> Makefile:106: recipe for target 'clean' failed
>> make: *** [clean] Error 255
>
> Hmm, this suggests your lxd is not set up correctly, it doesn't have
> an IP currently.
>
> The ssh command is on line 106:
>
> @ssh -qtAF $(SSH_CONFIG_FILE) $(IP) -- $(MAKE) $(LXC_MAKE_FLAGS) -C
> $(PWD) $@ $(LXC_MAKE_VARS)
>
> I suspect the issue is that the $(IP) is coming out as empty, which is
> confusing ssh
>
> I will try and add an explicit check for an IP address, with suitable
> error message.

Done. If it can't find an IP address for the container, any command
that needs ssh will fail with a more helpful error.

Another option would be to auto-start the container it its not started, perhaps?

--
Simon

[Natalia Bidart (nataliabidart)]

I pulled the changes. A run of make clean without the env var and I run with it both succeded with the same output:

nessita@dali:~/canonical/bloodearnest/sso-dev$ make clean
rm -rf /home/nessita/canonical/bloodearnest/sso-dev/env
rm -rf branches/wheels
rm -rf ../.juju-repo
rm -rf branches/*
rm -rf logs/*.*
rm -rf staticfiles
rm -f lib/versioninfo.py
rm -f targets.mk
find -name '*.pyc' -delete
find -name '*.~*' -delete
nessita@dali:~/canonical/bloodearnest/sso-dev$ export I_CAN_HAZ_JUJU=yes
nessita@dali:~/canonical/bloodearnest/sso-dev$ make clean
(Makefiles changed - rebuilding lxc target list)
rm -rf
rm -rf
rm -rf
rm -rf branches/*
rm -rf logs/*.*
rm -rf staticfiles
rm -f lib/versioninfo.py
rm -f targets.mk
find -name '*.pyc' -delete
find -name '*.~*' -delete
nessita@dali:~/canonical/bloodearnest/sso-dev$

Were you expecting this or there is some randomness around? What confuses me is that I don't see the container in my list of containers:

nessita@dali:~/canonical/bloodearnest/sso-dev$ sudo lxc-ls --fancy
NAME STATE IPV4 IPV6 GROUPS AUTOSTART
----------------------------------------------------------------
cpi-trusty STOPPED - - - NO
juju-trusty-lxc-template STOPPED - - - NO
sca-trusty STOPPED - - - NO
sso-trusty STOPPED - - - NO

[Natalia Bidart (nataliabidart)]

Replying to:

"FWIW: that script is just:

#!/bin/bash
sudo service gunicorn stop
trap 'sudo service gunicorn start' INT
bash /srv/gunicorn/run_sso_wsgi.sh $@

We could actually get rid of this, perhaps. Some thing like:

run: gunicorn-stop
    @trap 'sudo service gunicorn start' INT;
/srv/gunicorn/run_sso_wsgi.sh --bind=$(ARGS) --reload
--error-logfile=- --access-logfile=- --timeout=99999

would probably work. I can have a look at that if you've prefer to get
rid of the script?"

I would prefer to get rid of the script mainly because having the script mean that when reading the Makefile it requires to go an open other script in other folder; and also sometimes I try to cleanup stuff and is not trivial to know where a script is used or not, to decide if it can be removed.

[Simon Davy (bloodearnest)]

On Wed, Nov 18, 2015 at 12:15 PM, Natalia Bidart
<email address hidden> wrote:
> I pulled the changes. A run of make clean without the env var and I run with it both succeded with the same output:
>
> nessita@dali:~/canonical/bloodearnest/sso-dev$ make clean
> rm -rf /home/nessita/canonical/bloodearnest/sso-dev/env
> rm -rf branches/wheels
> rm -rf ../.juju-repo
> rm -rf branches/*
> rm -rf logs/*.*
> rm -rf staticfiles
> rm -f lib/versioninfo.py
> rm -f targets.mk
> find -name '*.pyc' -delete
> find -name '*.~*' -delete
> nessita@dali:~/canonical/bloodearnest/sso-dev$ export I_CAN_HAZ_JUJU=yes
> nessita@dali:~/canonical/bloodearnest/sso-dev$ make clean
> (Makefiles changed - rebuilding lxc target list)
> rm -rf
> rm -rf
> rm -rf
> rm -rf branches/*
> rm -rf logs/*.*
> rm -rf staticfiles
> rm -f lib/versioninfo.py
> rm -f targets.mk
> find -name '*.pyc' -delete
> find -name '*.~*' -delete
> nessita@dali:~/canonical/bloodearnest/sso-dev$
>
> Were you expecting this or there is some randomness around?

This is correct. Most make targets are unaltered and shared between
old and new dev envs

> What confuses me is that I don't see the container in my list of containers:
>
> nessita@dali:~/canonical/bloodearnest/sso-dev$ sudo lxc-ls --fancy
> NAME STATE IPV4 IPV6 GROUPS AUTOSTART
> ----------------------------------------------------------------
> cpi-trusty STOPPED - - - NO
> juju-trusty-lxc-template STOPPED - - - NO
> sca-trusty STOPPED - - - NO
> sso-trusty STOPPED - - - NO

lxd and lxc containers are differently managed. Try

lxc list

FYI lxd containers are in /var/lib/lxd/containers, not /var/lib/lxc

--
Simon

[Simon Davy (bloodearnest)]

On Wed, Nov 18, 2015 at 12:24 PM, Natalia Bidart
<email address hidden> wrote:
> Replying to:
>
> "FWIW: that script is just:
>
> #!/bin/bash
> sudo service gunicorn stop
> trap 'sudo service gunicorn start' INT
> bash /srv/gunicorn/run_sso_wsgi.sh $@
>
> We could actually get rid of this, perhaps. Some thing like:
>
> run: gunicorn-stop
> @trap 'sudo service gunicorn start' INT;
> /srv/gunicorn/run_sso_wsgi.sh --bind=$(ARGS) --reload
> --error-logfile=- --access-logfile=- --timeout=99999
>
> would probably work. I can have a look at that if you've prefer to get
> rid of the script?"
>
> I would prefer to get rid of the script mainly because having the script mean that when reading the Makefile it requires to go an open other script in other folder; and also sometimes I try to cleanup stuff and is not trivial to know where a script is used or not, to decide if it can be removed.

Ok, I will try this approach.

--
Simon

[lp:~bloodearnest/canonical-identity-provider/sso-dev]

1372. By Simon Davy on 2015-11-18

- remove scripts/run_gunicorn.sh, handle trap in makefile instead
- deploy target now has dependencies as a prerequisite
- removed docs about deploy/dependencies targets to avoid confusion - bootstrap should be the main one
- auto matically update the sso-charm to latest on deploy

[Natalia Bidart (nataliabidart)]

Thank you for all the answers and improvements. I tried a new run from scratch, issuing:

$ make lxc-delete
$ make clean

$ make lxc-setup
Using saved parent location: bzr+ssh://bazaar.launchpad.net/~bloodearnest/+junk/ols-tools/
No revisions or tags to pull.
./branches/ols-tools/setup-devel-lxd sso /srv/login.ubuntu.com/devel
Using lxc sso
Checking ssh key set up...done
Setting up posgtresql init on /dev/shm on boot...done
Device .home removed from sso
Device .src removed from sso
Device .logs removed from sso
Device .etc removed from sso
Device .home added to sso
Device .src added to sso
Device .logs added to sso
Device .etc added to sso
Creating fake tarball...done.
Setting up nessita for passwordless sudo...done.
Removing landscape-common...done.
Checking for network access...done.
Updating package index...done.
Installing ppa deps...done.
Installing juju stable ppa...done.
Updating package index (again)...done.
Updating system packages...done.
Installing required packages...done.
Cleaning up packages...done.
Configuring lxc sshd...done.
Restarting ssh...done.
Updating sso juju environment to use lxc ip 10.0.3.41
Checking for basenode access...done
Checking for CAT access...done.
Updating basenode...done.
Installing basenode...done.

And then make bootstrap, which fails as follow (I'm connected to the VPN):

nessita@dali:~/canonical/bloodearnest/sso-dev$ make bootstrap
cat dependencies.txt | xargs sudo apt-get install -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
python-amqplib is already the newest version.
python-bson is already the newest version.
python-iso8601 is already the newest version.
python-launchpadlib is already the newest version.
python-m2crypto is already the newest version.
python-memcache is already the newest version.
python-psutil is already the newest version.
python-psycopg2 is already the newest version.
python-simplejson is already the newest version.
python-testresources is already the newest version.
python-bcrypt is already the newest version.
python-beautifulsoup is already the newest version.
python-virtualenv is already the newest version.
bzr is already the newest version.
python-lxml is already the newest version.
python-tz is already the newest version.
python-yaml is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
cat dependencies-devel.txt | xargs sudo apt-get install -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
build-essential is already the newest version.
libxslt1-dev is already the newest version.
make is already the newest version.
memcached is already the newest version.
python-dev is already the newest version.
screen is already the newest version.
swig is already the newest version.
config-manager is already the newest version.
firefox is already the newest version.
gettext is already the newest version.
libpq-dev is already the newest version.
libxml2-dev is already the newest version.
postgresql-client-9.3 is already the newest version.
postgresql-contrib-9....

[lp:~bloodearnest/canonical-identity-provider/sso-dev]

1373. By Simon Davy on 2015-11-18

 - fix jenv file not being delete
 - fix LXC_MAKE_ARGS partial rename
 - run db-reset on lxc-start, to cope with shared memory

[Simon Davy (bloodearnest)]

>> === modified file 'src/identityprovider/tests/test_command_cleanup.py'
>> --- src/identityprovider/tests/test_command_cleanup.py 2015-04-30 17:20:09 +0000
>> +++ src/identityprovider/tests/test_command_cleanup.py 2015-11-17 13:23:47 +0000
>> @@ -48,8 +48,8 @@
>>
>> def make_test_accounts(self, count=0, date_created=None):
>> for i in xrange(count):
>> - email = self.factory.make_email_address(prefix='isdtest+',
>> - domain='canonical.com')
>> + email = self.factory.make_email_address(prefix='testemail+',
>> + domain='example.com')
>
> What's the rationale for this change? The job needs to clear from the DB all email addressed created by the acceptance tests, which use the isdtest+ prefix, so changing it will not trigger the right cleanup.

So, one of the things I tried to do was "normalise" the dev acceptance
test configuration, so it could just be part default devel
configuration. This would mean that a) you wouldn't need access to the
config branch to run acceptance tests and b) could run the acceptance
tests against the dev server OOTB with no config changes, which
simplified stuff a lot.

The only thing in the *dev* acceptance settings that was at all
sensitive was the isdtest email stuff, as it might leak info about our
production config.

Now, the isdtest email stuff was already in the code base anyway (like
in the above test), but I went ahead an made the default dev config
use <email address hidden>, and updated some tests that were expecting
other config.

I don't *think* this will have broken the clean up job, as that uses
the configured emails, which are unchanged in prod or staging.

But maybe I missed something?

--
Thanks

[lp:~bloodearnest/canonical-identity-provider/sso-dev]

1374. By Simon Davy on 2015-11-19

make clean now works properly again

1375. By Simon Davy on 2015-11-23

merging

[Daniel Manrique (roadmr)]

Poke... what's the status on this? Simon, could I at least request, if feasible, for you to merge trunk into this? I did so but there was a (minor) conflict, I think I solved it well enough that my local copy is working but I'd be more at ease if I knew you did the right thing while merging :)

The actual reason I'm asking is this: I tried running acceptance tests in my old-style lxc setup and failed miserably. The set of config directives I used to just inject in my ../local_config/settings.py no longer "just works", and after spending a couple of hours trying, I gave up and decided to give sso-dev another whirl. Acceptance tests worked magically out of the box. So I'm wondering if perhaps the work done to remove the settings-by-symlink workflow broke the old way of doing acceptance testing.

Does this make sense to you? What do I need to do now in order to run acceptance tests locally? OTOH if it's easier to just use sso-dev, that's fair enough, but we'd need to push for this to land so that people wanting to run acceptance aren't left in the same limbo as me (or I was just being extra dumb today - always a possibility, particularly on Mondays).

Thanks and apologies for the rant!

[lp:~bloodearnest/canonical-identity-provider/sso-dev]

1376. By Simon Davy on 2015-12-08

merging

[Simon Davy (bloodearnest)]

On Mon, Dec 7, 2015 at 10:37 PM, Daniel Manrique
<email address hidden> wrote:
> Poke... what's the status on this? Simon, could I at least request, if feasible, for you to merge trunk into this?

Done. All tests passing.

> I did so but there was a (minor) conflict, I think I solved it well enough that my local copy is working but I'd be more at ease if I knew you did the right thing while merging :)

Yep, it was just bzr merge failing hard on trivial changes, for
unknown reasons :)

> The actual reason I'm asking is this: I tried running acceptance tests in my old-style lxc setup and failed miserably. The set of config directives I used to just inject in my ../local_config/settings.py no longer "just works", and after spending a couple of hours trying, I gave up and decided to give sso-dev another whirl. Acceptance tests worked magically out of the box. So I'm wondering if perhaps the work done to remove the settings-by-symlink workflow broke the old way of doing acceptance testing.

Hmm, it shouldn't have. ../local_config/settings.py is still used, but
just via PYTHONPATH, not a symlink. I did test the acceptances tests
with that change, with out issue. I will try again.

However, the old way of running acceptance tests (via run-tests and
scripts/acceptance-tests) mv's ../local_config/settings.py out of the
way, writes its own version there, runs the server and tests, and then
copies back your old settings.py. So, to my knowledge, you've never
been able to manually tweak ../local_config/settings.py when running
tests this way.

However, if you run the server yourself manually, and run the tests
separately with make run-acceptance, then your local settings.py is
used (and you'll need to have the acceptance relations

The sso-dev work removes all this complexity. The is now no difference
between devel settings and the acceptance tests settings, so no file
mv'ing needed. It also already has an sso server and mail server
running, so everything is much simpler when it comes to running

> Does this make sense to you? What do I need to do now in order to run acceptance tests locally? OTOH if it's easier to just use sso-dev, that's fair enough, but we'd need to push for this to land so that people wanting to run acceptance aren't left in the same limbo as me (or I was just being extra dumb today - always a possibility, particularly on Mondays).

I would like to land, but still waiting on review approval, there's
some outstanding comments. I will ask Natalia to have a look again.

> Thanks and apologies for the rant!

No worries - sorry it's not landed yet :(

--
Simon

[Simon Davy (bloodearnest)]

On Tue, Dec 8, 2015 at 11:18 AM, Simon Davy <email address hidden> wrote:
> On Mon, Dec 7, 2015 at 10:37 PM, Daniel Manrique
> <email address hidden> wrote:
>> Poke... what's the status on this? Simon, could I at least request, if feasible, for you to merge trunk into this?
>
> Done. All tests passing.
>
>> I did so but there was a (minor) conflict, I think I solved it well enough that my local copy is working but I'd be more at ease if I knew you did the right thing while merging :)
>
> Yep, it was just bzr merge failing hard on trivial changes, for
> unknown reasons :)
>
>> The actual reason I'm asking is this: I tried running acceptance tests in my old-style lxc setup and failed miserably. The set of config directives I used to just inject in my ../local_config/settings.py no longer "just works", and after spending a couple of hours trying, I gave up and decided to give sso-dev another whirl. Acceptance tests worked magically out of the box. So I'm wondering if perhaps the work done to remove the settings-by-symlink workflow broke the old way of doing acceptance testing.
>
> Hmm, it shouldn't have. ../local_config/settings.py is still used, but
> just via PYTHONPATH, not a symlink. I did test the acceptances tests
> with that change, with out issue. I will try again.

Using ./run-tests dev, the acceptance tests ran fine for me on r1374

Thanks

--
Simon

[lp:~bloodearnest/canonical-identity-provider/sso-dev]

1377. By Simon Davy on 2015-12-08

merging properly this time

[Daniel Manrique (roadmr)]

Argh, so I did ./run-tests dev (note, using trunk, not sso-dev) and it still won't run acceptance :( (r1376 here).

What I'm seeing is that ./run-tests dev starts by unconditionally bootstrapping, which I think destroys my ../local_config/settings.py, so by the time the acceptance tests start, the config is all borked and obviously they fail.

No worries though, I have a reasonable workaround by using sso-dev whenever I need acceptance tests run ;) (which is not often, really).

[lp:~bloodearnest/canonical-identity-provider/sso-dev]

1378. By Simon Davy on 2015-12-09

merging

1379. By Simon Davy on 2015-12-14

merging

[Daniel Manrique (roadmr)]

Ohnoes... here comes Daniel with more crazy problems...

Apparently once the container is shut down and restarted, part of the db configuration changes, so on next boot it fails with (ultimately) this:

django.db.utils.OperationalError: FATAL: password authentication failed for user "ssoadmin"

How to repro (I_CAN_HAZ_JUJU assumed to be set at all times):

(maybe delete any old sso containers with lxc delete)
(set up my crazy /src mount in the ols-dev profile)
get a checkout of sso-dev
merge latest trunk (I did this myself, maybe I screwed up?)
make lxc-setup
make bootstrap
ssh into container, change to /src/blahblah/sso-dev dir
make test ARGS=some.quick.test
*it works*
sudo poweroff (the container!)
lxc start sso
wait until juju stabilizes (on startup a series of config-changeds fire up)
ssh into container, change to /src/blahblah/sso-dev dir
make test ARGS=some.quick.test
* now it fails with the above-shown message *

Logging out of the container and redoing "make bootstrap" from the host brings things to a working state again, but I seem to recall this shouldn't be necessary.

[Simon Davy (bloodearnest)]

On Wed, Jan 6, 2016 at 10:16 PM, Daniel Manrique
<email address hidden> wrote:
> Ohnoes... here comes Daniel with more crazy problems...
>
> Apparently once the container is shut down and restarted, part of the db configuration changes, so on next boot it fails with (ultimately) this:
>
> django.db.utils.OperationalError: FATAL: password authentication failed for user "ssoadmin"
>
> How to repro (I_CAN_HAZ_JUJU assumed to be set at all times):
>
>
> (maybe delete any old sso containers with lxc delete)
> (set up my crazy /src mount in the ols-dev profile)
> get a checkout of sso-dev
> merge latest trunk (I did this myself, maybe I screwed up?)
> make lxc-setup
> make bootstrap
> ssh into container, change to /src/blahblah/sso-dev dir
> make test ARGS=some.quick.test
> *it works*
> sudo poweroff (the container!)
> lxc start sso
> wait until juju stabilizes (on startup a series of config-changeds fire up)
> ssh into container, change to /src/blahblah/sso-dev dir
> make test ARGS=some.quick.test
> * now it fails with the above-shown message *
>
> Logging out of the container and redoing "make bootstrap" from the host brings things to a working state again, but I seem to recall this shouldn't be necessary.

It shouldn't. I will look into it, and merge trunk in the process

FYI, there are lxc-start and lxc-stop make commands which I would
encourage rather than sudo poweroff

--
Simon

[lp:~bloodearnest/canonical-identity-provider/sso-dev]

1380. By Simon Davy on 2016-01-12

merging

1381. By Simon Davy on 2016-01-12

merging

1382. By Simon Davy on 2016-03-03

merging

1383. By Simon Davy on 2016-03-04

fix some small issues from user testing

[Daniel Manrique (roadmr)]

Hello :)

Unmerged revisions

1383. By Simon Davy on 2016-03-04

fix some small issues from user testing

1382. By Simon Davy on 2016-03-03

merging

1381. By Simon Davy on 2016-01-12

merging

1380. By Simon Davy on 2016-01-12

merging

1379. By Simon Davy on 2015-12-14

merging

1378. By Simon Davy on 2015-12-09

merging

1377. By Simon Davy on 2015-12-08

merging properly this time

1376. By Simon Davy on 2015-12-08

merging

1375. By Simon Davy on 2015-11-23

merging

1374. By Simon Davy on 2015-11-19

make clean now works properly again

=== modified file '.bzrignore'
--- .bzrignore	2015-10-09 16:03:29 +0000
+++ .bzrignore	2016-03-04 19:06:06 +0000
@@ -7,4 +7,8 @@
 node_modules
 staticfiles
 lib/versioninfo.py
-settings.py
+.juju-repo
+EXTRACTED
+django_project/settings.py
+django_project/settings_local.py
+targets.mk

=== added file 'Makefile'
--- Makefile	1970-01-01 00:00:00 +0000
+++ Makefile	2016-03-04 19:06:06 +0000
@@ -0,0 +1,162 @@
+.DEFAULT_GOAL = help
+# disable builtin rules, which are not used
+.SUFFIXES:
+
+
+# sso specific config
+SSO_LXC ?= sso
+# base dir inside lxc
+LXC_BASEDIR = /srv/login.ubuntu.com/devel
+CODE_DIR=$(LXC_BASEDIR)/code/devel
+
+# shared config between all dev envs
+WHEELS_BRANCH_URL ?= lp:~ubuntuone-pqm-team/$(PROJECT_NAME)/dependencies
+WHEELS_DIR = branches/wheels
+ENV = $(CURDIR)/env
+
+# commands that can be run on host or container
+clean:
+	rm -rf $(ENV)
+	rm -rf $(WHEELS_DIR)
+	rm -rf $(JUJU_REPOSITORY)
+	rm -rf branches/*
+	rm -rf logs/*.*
+	rm -rf staticfiles
+	rm -f lib/versioninfo.py
+	rm -f targets.mk
+	find -name '*.pyc' -delete
+	find -name '*.~*' -delete
+
+
+ifeq ("$(I_CAN_HAZ_JUJU)","yes")
+
+LXC_NAME = $(SSO_LXC)
+JUJU_ENV = $(LXC_NAME)
+JUJU_HOME ?= $(HOME)/.juju
+JENV = $(JUJU_HOME)/environments/$(JUJU_ENV).jenv
+JUJU_REPOSITORY ?= $(CURDIR)/.juju-repo
+JUJU_HOME ?= $(HOME)/.juju
+export JUJU_REPOSITORY JUJU_ENV 
+
+OLS_TOOLS_BRANCH ?= lp:~bloodearnest/+junk/ols-tools
+TOOLS = branches/ols-tools
+LXC_PATH = /var/lib/lxd/containers/$(LXC_NAME)
+LXC_STATE = $(shell lxc info $(LXC_NAME) | grep Status | awk '{ print $$2 }')
+IP = $(shell lxc info $(LXC_NAME) | grep eth0 | grep -v inet6 | awk '{ print $$3 }')
+HOST = $(shell hostname)
+SSH_CONFIG_FILE ?= django_project/ssh_config
+LXC_TARGETS ?= targets.mk
+# flags/vars for passing to make invocation inside lxc
+LXC_MAKE_FLAGS=--no-print-directory --no-builtin-rules
+LXC_MAKE_VARS = "I_CAN_HAZ_JUJU=yes"
+ERR_LOG=$(shell mktemp)
+
+APT_PROXY ?=
+export APT_PROXY
+
+# multiline error message
+define RUN_LXC_SETUP
+
+lxc container $(LXC_NAME) not found
+Please run the following to create it:
+make lxc-setup
+endef
+
+
+ifneq ($(HOST),$(LXC_NAME))
+# We are on the host machine, so ssh to the lxc and run the command
+
+# use the container dir as a fs-level proxy to whether the lxc exists
+$(LXC_PATH):
+	$(error $(RUN_LXC_SETUP))
+
+lxc-available: $(LXC_PATH)
+	@test -n "$(IP)" || { echo "Cannot find ip address for container $(LXC_NAME) - is it running? Try: make lxc-start"; exit 1; }
+
+# sadly (or maybe not), these can't be in config-manager.txt, as we need to
+# bootstrap to get those, and we can't bootstrap until we have an lxc.
+## fetch the shared ols-tools branch
+$(TOOLS):
+	@[ -d $@ ] && bzr pull -d $@ || bzr branch $(OLS_TOOLS_BRANCH) $@
+.PHONY: $(TOOLS)
+
+# this is idempotent (kinda), can be rerun to update the lxc
+## create/update the development lxc
+lxc-setup: $(TOOLS)
+	./branches/ols-tools/setup-devel-lxd $(LXC_NAME) $(LXC_BASEDIR)
+
+## start the lxc and reset the db
+lxc-start: $(LXC_PATH)
+	@[ "$(LXC_STATE)" = "Running" ] && echo "lxc $(LXC_NAME) already running" || (echo "Starting lxc $(LXC_NAME)" && lxc start $(LXC_NAME) && sleep 3)
+	@# reset db, as it's on shared memory, and thus been wiped
+	$(MAKE) db-reset
+
+## stop the lxc, including the juju env
+lxc-stop: $(LXC_PATH)
+	@[ "$(LXC_STATE)" != "Running" ] && echo "lxc $(LXC_NAME) already stopped" || (echo "Stopping lxc $(LXC_NAME)" && lxc stop $(LXC_NAME))
+
+## stop and delete the lxc
+lxc-delete: lxc-stop
+	@echo Destroying lxc $(LXC_NAME)
+	@lxc delete $(LXC_NAME)
+	@rm -f $(JENV)  # juju env is gone now, so clean up
+
+## convenient shortcut to ssh in
+ssh: lxc-available
+	JUJU_ENV=$(JUJU_ENV) ssh -tA -F $(SSH_CONFIG_FILE) $(IP) 'cd $(PWD) ; exec "$$SHELL" -l'
+
+# TODO: an easy way to set env vars inside the lxc
+# an explicit list of envvars to transfer as make args
+# if args is not present, don't set it, or else we override the default to be empty
+ifdef ARGS
+LXC_MAKE_VARS += ARGS=\"$(ARGS)\"
+endif
+ifdef DJANGO_SETTINGS_MODULE
+LXC_MAKE_VARS += DJANGO_SETTINGS_MODULE=$(DJANGO_SETTINGS_MODULE)
+endif
+ifdef SST_BASE_URL
+LXC_MAKE_VARS += SST_BASE_URL=$(SST_BASE_URL)
+endif
+
+
+# These rules build a list of the makefile targets that are available in the lxc
+# to use for providing tab completion on the host.
+# They utilise the special re-making behaviour where make will first try and
+# build any targets that are also included Makefiles, then restart parsing.
+# So in this case, make will try and build $(LXC_TARGETS) first, then
+# include it
+# https://www.gnu.org/software/make/manual/html_node/Remaking-Makefiles.html
+$(LXC_TARGETS): Makefile Makefile.common Makefile.juju
+	@echo "(Makefiles changed - rebuilding lxc target list)"
+	@echo -n "TARGETS=" > $(LXC_TARGETS)
+	@echo '$(shell make -nrRpq -f Makefile.juju -f Makefile.common | egrep -o "^[a-zA-Z0-9._/-]+:" | grep -v "^\." | grep -v Makefile | cut -d: -f1 | sort -u)' >> $(LXC_TARGETS)
+
+# the - means its absence is ignored on the first pass
+-include $(LXC_TARGETS)
+
+# We can now guarantee the file target list is generated and up to date
+$(TARGETS): lxc-available
+	@ssh -qtAF $(SSH_CONFIG_FILE) $(IP) -- $(MAKE) $(LXC_MAKE_FLAGS) -C $(PWD) $@ $(LXC_MAKE_VARS)
+
+else # we are in the lxc
+
+# here to avoid overriding non-juju config. Can be moved once juju is default
+DJANGO_SETTINGS_MODULE ?= django_project.settings_devel_juju
+
+include Makefile.common
+include Makefile.juju
+
+endif
+
+else # we are not using juju
+
+include Makefile.common
+include Makefile.db
+
+bootstrap: $(LOCAL_SETTINGS_PATH) virtualenv sourcedeps install-wheels-dev
+
+run: ARGS=0.0.0.0:8000
+run:
+	$(ENV)/bin/gunicorn django_project.wsgi:application --workers=2 --reload --pid=logs/gunicorn.pid --bind=$(ARGS) --timeout=99999
+
+endif

=== renamed file 'Makefile' => 'Makefile.common'
--- Makefile	2016-02-25 17:59:36 +0000
+++ Makefile.common	2016-03-04 19:06:06 +0000
@@ -1,12 +1,11 @@
-# Copyright (C) 2014  Canonical Ltd.
-
+# Copyright (C) 2015 Canonical Ltd.
+#
 CM ?= /usr/lib/config-manager/cm.py
 CONFIGMANAGER = config-manager.txt
 CONN_CHECK_CONFIG_PATH ?= /tmp/sso_conn_check_config.yaml
 DJANGO_SETTINGS_MODULE ?= settings
-ENV = $(CURDIR)/env
 JUJU_ENV ?= local
-JUJU_REPO ?= ../.juju-repo
+JUJU_REPOSITORY ?= ../.juju-repo
 LOCAL_SETTINGS_DIR ?= $(CURDIR)/../local_config
 LOCAL_SETTINGS_PATH = $(LOCAL_SETTINGS_DIR)/settings.py
 FLAKE8 = $(ENV)/bin/flake8
@@ -21,13 +20,11 @@
 TRANSLATIONS_DIR = $(CURDIR)/branches/translations/po
 # this is a bit hackey, could be an external script in python (Simon)
 UNITTEST_TARGETS = $(shell find $(SRC_DIR) -name test_\*.py | sed 's+\(^$(SRC_DIR)/\|.py\)++g' | sed 's+/+.+g')
-WHEELS_BRANCH_URL ?= lp:~ubuntuone-pqm-team/$(PROJECT_NAME)/dependencies
-WHEELS_DIR = branches/wheels
 STATIC_ROOT ?= $(shell grep -R STATIC_ROOT django_project/settings_base.py | awk '{print $$3}')
 # Create archives in labelled directories (ie. r27/$(PROJECT_NAME).tbz2)
 TARBALL_BUILD_LABEL ?= $(shell bzr version-info --custom --template="r{revno}\n")
 TARBALL_FILE_NAME = $(PROJECT_NAME).tbz2
-TARBALL_BUILDS_DIR ?= $(JUJU_REPO)/builds
+TARBALL_BUILDS_DIR ?= $(JUJU_REPOSITORY)/builds
 TARBALL_BUILD_DIR = $(TARBALL_BUILDS_DIR)/$(TARBALL_BUILD_LABEL)
 TARBALL_BUILD_PATH = $(TARBALL_BUILD_DIR)/$(TARBALL_FILE_NAME)
 
@@ -59,13 +56,16 @@
 	@DJANGO_SETTINGS_MODULE=django_project.settings_build $(MAKE) compilemessages 
 
 # note: fetch-sourcedeps needs to be manually called before this will work.
+## build a deployment tarball
 build-tarball: install-wheels $(TARBALL_BUILD_PATH)
 
+## write the bzr revno to lib/versioninfo.py
 version:
 	bzr version-info --format=python > lib/versioninfo.py
 
 ### Wheels ###
 
+## create/reinitialise the virtualenv
 virtualenv:
 	@virtualenv --clear --system-site-packages $(ENV)
 
@@ -73,24 +73,33 @@
 update-wheels: virtualenv
 	$(PIP) wheel -w $(WHEELS_DIR) -f $(WHEELS_DIR) $(ARGS)
 
+## install wheel dependencies
 install-wheels: ARGS=-r requirements.txt
 install-wheels: virtualenv
 	$(PIP) install --find-links=$(WHEELS_DIR) --no-index $(ARGS)
 
+## install development wheel dependencies
 install-wheels-dev: virtualenv
 	$(MAKE) install-wheels ARGS="-r requirements_devel.txt --pre"
 
+## setup localmail server in container
+localmail:
+	./scripts/setup-localmail.sh
+
 ### dependencies ###
 
 branches/last_build: $(CONFIGMANAGER)
 	$(CM) update $(CONFIGMANAGER)
 	touch $@
 
+## update wheels branch
 wheels:
 	[ -d $(WHEELS_DIR) ] && (cd $(WHEELS_DIR) && bzr pull) || (bzr branch $(WHEELS_BRANCH_URL) $(WHEELS_DIR))
 
+## fetch sourcedeps and wheels branch
 fetch-sourcedeps: branches/last_build wheels version
 
+## fetch sourcedeps
 sourcedeps: fetch-sourcedeps install-wheels
 
 ### local config ###
@@ -101,23 +110,8 @@
 $(LOCAL_SETTINGS_PATH): $(LOCAL_SETTINGS_DIR)
 	@cp django_project/settings.py.example $(LOCAL_SETTINGS_PATH)
 
-### for devel: start and stop the local postgresql DB ###
-
-include Makefile.db
-
 ### what you really need ###
 
-bootstrap: $(LOCAL_SETTINGS_PATH) virtualenv sourcedeps install-wheels-dev
-
-clean:
-	rm -rf $(ENV)
-	rm -rf $(WHEELS_DIR)
-	rm -rf branches/*
-	rm -rf staticfiles
-	rm -f lib/versioninfo.py
-	find -name '*.pyc' -delete
-	find -name '*.~*' -delete
-
 conn-check:
 	@echo "Generating conn-check config from Django settings.."
 	@sudo -u $$(ls -ld $(CURDIR) | awk '{print $$3}') PYTHONPATH=$(PYTHONPATH) \
@@ -142,17 +136,13 @@
 # Check python files (except for the south migration files that are not pep8
 # compliant and probably never will).
 lint:
-	@$(FLAKE8) --exclude='migrations' \
+	@$(FLAKE8) --exclude='migrations,django_project/settings.py' \
 		--filename='*.py' \
 		src/ django_project/ scripts/
 
 manage:
 	$(DJANGO_MANAGE) $(ARGS)
 
-run: ARGS=0.0.0.0:8000
-run: collectstatic
-	$(ENV)/bin/gunicorn django_project.wsgi:application --workers=2 --reload --pid=logs/gunicorn.pid --bind=$(ARGS) --timeout=99999 --error-logfile=- --access-logfile=- 
-
 start: bootstrap start-db
 
 stop: stop-db
@@ -186,4 +176,21 @@
 		cd -; \
 	done
 
-.PHONY: clean conn-check version makemessages compilemessages collectstatic run-acceptance docs
+## display this help message
+help:
+	$(info Available targets)
+	@awk '/^[a-zA-Z\-\_0-9]+:/ {                                   \
+		nb = sub( /^## /, "", helpMsg );                             \
+		if(nb == 0) {                                                \
+		helpMsg = $$0;                                             \
+		nb = sub( /^[^:]*:.* ## /, "", helpMsg );                  \
+		}                                                            \
+		if (nb)                                                      \
+		printf "\033[1;31m%-" width "s\033[0m %s\n", $$1, helpMsg; \
+	}                                                              \
+	{ helpMsg = $$0 }'                                             \
+	width=$$(grep -o '^[a-zA-Z_0-9]\+:' $(MAKEFILE_LIST) | wc -L)  \
+	$(MAKEFILE_LIST)
+
+
+.PHONY: conn-check version makemessages compilemessages collectstatic run-acceptance docs

=== added file 'Makefile.juju'
--- Makefile.juju	1970-01-01 00:00:00 +0000
+++ Makefile.juju	2016-03-04 19:06:06 +0000
@@ -0,0 +1,120 @@
+# sso specific config
+CHARM_SERIES = trusty
+CHARM = canonical-identity-provider
+CHARM_BRANCH = lp:~ubuntuone-pqm-team/canonical-is-charms/canonical-identity-provider
+CHARM_SRC = ../sso-charm
+DEPLOYER_CONFIG = $(CURDIR)/django_project/deploy.yaml
+DEPLOYER_TARGET = sso-app-dev
+
+
+LOG_DIRS ?= ./logs/www-oops ./logs/schema-updates
+ABS_CHARM_SRC = $(shell readlink -f $(CHARM_SRC))
+LOCAL_JUJU_CONFIG = .local.yaml
+export I_CAN_HAZ_JUJU=yes
+
+# python config, overriding the non-juju config
+PYTHONPATH := $(LXC_BASEDIR)/etc:$(SRC_DIR):$(LIB_DIR):$(CURDIR)
+export PYTHONPATH
+
+
+
+$(CHARM_SRC):
+	@test -d $@ && bzr pull -d $@ || bzr branch $(CHARM_BRANCH) $@
+
+
+# | is order only dependency - means if the dir is present, we're good. no need to rebuild
+# https://www.gnu.org/software/make/manual/html_node/Prerequisite-Types.html
+$(JUJU_REPOSITORY): $(CHARM_SRC)
+	@mkdir -p $(JUJU_REPOSITORY)/$(CHARM_SERIES)
+	@rm -f $(JUJU_REPOSITORY)/$(CHARM_SERIES)/$(CHARM)
+	@ln -sf $(ABS_CHARM_SRC) $(JUJU_REPOSITORY)/$(CHARM_SERIES)/$(CHARM)
+
+# make sure log subdirs are created with user ownership, or else they are owned by root.
+$(LOG_DIRS):
+	mkdir -p $@
+	sudo chown $$USER:$$USER $@
+
+$(LOCAL_JUJU_CONFIG):
+	@echo "$(DEPLOYER_TARGET): {services: {sso-app: {options: { user: $$USER }}}}" > $(LOCAL_JUJU_CONFIG)
+.INTERMEDIATE: $(LOCAL_JUJU_CONFIG)
+.DELETE_ON_ERROR: $(LOCAL_JUJU_CONFIG)
+
+
+# we need to stop gunicorn so we can relink the python in the venv
+gunicorn-start:
+	@test -f /etc/init/gunicorn.conf && sudo service gunicorn start || true
+gunicorn-stop:
+	@test -f /etc/init/gunicorn.conf && { sudo service gunicorn stop; sleep 1; } || true
+
+dependencies: system-dependencies system-dependencies-devel gunicorn-stop virtualenv sourcedeps install-wheels-dev gunicorn-start
+
+system-dependencies: dependencies.txt
+	cat dependencies.txt | xargs sudo apt-get install -y
+
+system-dependencies-devel: dependencies-devel.txt
+	@cat dependencies-devel.txt | xargs sudo apt-get install -y
+
+## install dependencies and deploy charms
+bootstrap: dependencies deploy
+
+# use jenv file as indication the environment is up or not
+$(JENV):
+	juju bootstrap --upload-tools
+
+# clean up any stale jenv files
+clean-old-jenv:
+	@test -f $(JENV) && grep -q $(shell hostname -I) $(JENV) || rm -f $(JENV)
+
+juju-bootstrap: clean-old-jenv $(JENV)
+
+deploy: dependencies juju-bootstrap $(JUJU_REPOSITORY) $(LOCAL_JUJU_CONFIG) | $(LOG_DIRS)
+	juju-deployer --local-mods -w 15 -s 0 --debug -c $(CHARM_SRC)/deploy.yaml -c $(DEPLOYER_CONFIG) -c $(LOCAL_JUJU_CONFIG) $(DEPLOYER_TARGET)
+	$(MAKE) setup-db localmail I_CAN_HAZ_JUJU=yes
+
+## destroy juju environment
+juju-destroy:
+	juju destroy-environment $(JUJU_ENV) --force || true
+	@# work around manual provider rubbishness, and remove charms
+	@echo Cleaning up manual provider leftovers
+	@sudo rm -rf /var/lib/juju/*
+	@sudo rm -f /etc/init/juju*
+	@sudo killall -q jujud || true
+	@sudo killall -q mongod || true
+	@# clean up any charm left overs
+	@sudo rm -f /etc/init/gunicorn*
+	@sudo rm -f /etc/init.d/gunicorn*
+	@sudo rm -rf /srv/gunicorn /srv/conn-check
+
+## destroy juju environment and rebuild
+redeploy: juju-destroy deploy
+
+# preserving old alias
+setup-db reset-db: db-setup
+
+## setup a clean database
+db-setup db-reset:
+	@# protect for when the juju env is not up yet
+	juju ssh sso-postgresql/0 true &>/dev/null && { $(CHARM_SRC)/scripts/create-dev-db; $(MAKE) migrate; }
+
+## migrate the db
+migrate:
+	$(DJANGO_MANAGE) migrate --noinput
+
+# this stops the system gunicorn and runs it manually, restarting on exit
+## run server in foreground for debugging
+run: ARGS=0.0.0.0:8080
+run: GUNICORN_DEVEL_ARGS=--reload --error-logfile=- --access-logfile=- --timeout=99999
+run: gunicorn-stop
+	@trap 'sudo service gunicorn start' INT; bash /srv/gunicorn/run_sso_wsgi.sh --bind=$(ARGS) $(GUNICORN_DEVEL_ARGS)
+
+## run acceptance tests in development
+acceptance-dev:
+	./scripts/acceptance-dev-juju.sh $(ARGS)
+.PHONY: acceptance-dev
+
+acceptance-db: db-reset
+	$(DJANGO_MANAGE) loaddata test
+	$(DJANGO_MANAGE) create_test_team
+	# add openid RP config
+	# we need this to explicitly allow unverified logins
+	$(DJANGO_MANAGE) add_openid_rp_config $$SST_BASE_URL/consumer --allow-unverified --allowed-user-attribs=\"fullname,nickname,email,language,account_verified\"

=== added file 'README.juju'
--- README.juju	1970-01-01 00:00:00 +0000
+++ README.juju	2016-03-04 19:06:06 +0000
@@ -0,0 +1,220 @@
+=============================
+Development environment setup
+=============================
+
+Getting started
+===============
+
+Note: currently you need to set I_CAN_HAZ_JUJU=yes in the environment to enable the juju dev env.
+
+0. Get the code
+::
+
+.. note::
+    You need to install, configure and setup bzr and ssh keys. See BAZAAR below.
+
+    bzr branch lp:canonical-identity-provider
+
+
+1. Install lxd
+::
+
+    The supported way to develop Canonical Identity Provider is using Juju and LXD
+
+    Follow the instructions here to install/configure lxd
+
+    https://linuxcontainers.org/lxd/getting-started-cli/
+
+    Optional: you don't need juju on the host (as it will be installed in the
+    lxc), but it is useful to be able to query the juju system direct from the
+    host
+
+    sudo apt-get install juju-core
+
+2. Create the lxd container.
+::
+
+    The following command will create an lxd container, in which everything
+    will be deployed.  This will create the container if it does not exist,
+    update it, install dependencies, and set up juju properly. It mounts the
+    current directory into the container, so you can edit your code on the
+    host. It also mirrors your user and mounts your home dir into the
+    container. This allows conveniences like ssh/bzr/bash configs to be used
+    inside the container. Some important configuration options that can be
+    configured via env vars or make args:
+
+    SSO_LXC - name of lxd container and juju environment. Default: sso
+
+    APT_PROXY - http://url of local apt proxy, highly recommended to have one
+    set up. Default: none 
+
+    DEVELOPER_PACKAGES - list of additional packages to install. Default: none
+
+    $ cd canonical-identity-provider
+    $ make lxc-setup [APT_PROXY=http://...]
+
+.. note::
+
+    You can access the container via the normal lxd mechanism (e.g lxc
+    exec), or via ssh, or just do
+
+    make ssh
+
+    If you create an lxd profile called ols-dev, this will be applied to the
+    container.  This is useful if you have an unusual $HOME set up, amongst
+    other things.  For example, to have all your lxd containers for OLS
+    projects have a special mount into your $HOME dir:
+
+    lxc profile create ols-dev
+    lxc profile device add ols-dev <name> disk source=/mnt/encrypted path=$HOME/work
+
+    By default, the development environment uses django_project/ssh_config
+    as its ssh config file. This is to provide a good default experience.
+    However, If you wish to configure ssh into the container yourself, you
+    can set SSH_CONFIG_FILE env var or make argument. Note: agent
+    forwarding is required, and is hardcoded by cli arguments in the
+    makefiles.
+
+    You should start, stop or destroy the container using the following:
+
+    make lxc-stop
+    make lxc-start
+    make lxc-delete  # this will wipe everything!
+
+
+3. Bootstrap the project
+
+    This will fetch python dependencies, and install them into a virtualenv. It
+    will also check out the juju charm, and bootstrap/deploy a juju environment
+    in the container, inluding postgresql, memcached, and gunicorn. Both of
+    these steps take a few minutes each the first time you do them, but
+    subsequent bootstraps are much quicker. When complete, the service should
+    be accessible on port 8080 of the container.
+
+    $ make bootstrap
+
+.. note::
+
+    For more commands, see 
+
+    $ make help
+
+
+4. Run the tests
+::
+
+    $ make test
+    
+    To run tests in 4 parallel
+
+    $ make test ARGS=-c4
+
+
+5. Debug the instance
+::
+    By default, the service is running on port 8080 under upstart, exactly
+    like production. If you wish to use a debugger, you need to use the
+    following command. This will stop the upstart controlled gunicorn, and
+    run it in the foreground, so you can attach to stdin/out
+
+    $ make run
+
+
+6. (Optional) Setup reCaptcha
+::
+
+    captcha is disabled by default. If you wish to enable it, you can do
+
+    juju set sso-app captcha_public_key=<your public key>
+    juju set sso-app captcha_private_key=<your private key>
+
+    You may also need to enable the gargoyle flag for captcha
+
+
+7. (Optional) Accessing the mail server
+::
+    The development set up includes an email server (lp:localmail), which
+    supports delivery via smtp on port 2025, and access via IMAP on port 2143,
+    or a simple web interface on port 8880.
+
+    Additionally, the logs are available in logs/localmail, and a mbox file is
+    at logs/localmail.mbox, which you can read with mutt or other mail client.
+
+
+8. (Optional) Create your own certificate and private key for SAML 2.0
+::
+    Required if you want to test interaction with a real Service Point.
+
+    See instructions in the django-saml2-idp upstream project:
+    http://code.google.com/p/django-saml2-idp/source/browse/trunk/idptest/keys/mk_keys.sh
+
+    Once you have the files (certificate.pem and private-key.pem), update the
+    saml_remotes config on the charm to point to them.
+
+
+9. (Optional) Running acceptance tests
+::
+
+    There is suite of selenium tests in ./acceptance. To run those against
+    your container:
+
+    make acceptance-dev
+
+    For more info:
+
+    make acceptance-dev ARGS=--help
+
+10. (Optional) CSS Hacking
+::
+
+    (https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager)
+
+    npm install
+    npm install -g gulp
+    gulp
+
+    Will start a watcher to lint, minify and concat CSS (see ./gulpfile.js)
+    CSS source can be found in src/identityprovider/static/src/css and is built
+    to src/identityprovider/static/css/ (see ./gulpfile.js)
+
+
+11. (Optional) Build deployment tarball
+::
+    To build a tarball that contains all the code, dependencies and the
+    virtualenv:
+
+    make fetch-sourcedeps
+    make build-tarball
+
+    Note: this is two separate steps as in Mojo specs we need to separate the
+    dependency collect step from the build step, as the build step runs in an
+    lxc with no internet access at all.
+    
+
+
+BAZAAR
+------
+You can install bzr like this:
+
+$ sudo apt-get install bzr
+$ bzr whoami "You Name <[e-mail]"
+
+You will also need to setup SSH keys. If you already have them, you will need
+to copy them to the machine you wish to use. (Otherwise, Launchpad won't talk
+to you.) Although, this suggests you can have multiple keys:
+
+https://launchpad.net/~YOURUSERNAME/+editsshkeys
+
+Background Reading
+------------------
+For those new to this project, this may help bring you up to speed quickly.
+The Identity Provider uses some conventions that you may not have seen before.
+These include:
+
+- virtualenv
+- preflight
+
+Enjoy!
+======
+
+We hope you enjoy using this software.

=== added file 'django_project/deploy.yaml'
--- django_project/deploy.yaml	1970-01-01 00:00:00 +0000
+++ django_project/deploy.yaml	2016-03-04 19:06:06 +0000
@@ -0,0 +1,25 @@
+sso-app-dev:
+  inherits:
+      - sso-dev-minimal
+  services:
+    sso-app:
+      to: [0]
+      options:
+        build_label: devel
+        deployment: devel
+        secret_key: development
+        email_hostport: localhost:2025
+        django_settings_module: django_project.settings_devel_juju
+        # use unix socket
+        db_host: ""
+        db_port: ""
+        # use db superuser for dev
+        db_user: ssoadmin
+        db_password: ssoadmin
+    sso-postgresql:
+      to: [0]
+      options:
+        # enable access via local socket
+        extra_pg_auth: local all sso md5,local all ssoadmin md5
+    sso-memcached:
+      to: [0]

=== modified file 'django_project/paths.py'
--- django_project/paths.py	2015-10-09 16:03:29 +0000
+++ django_project/paths.py	2016-03-04 19:06:06 +0000
@@ -29,7 +29,9 @@
 
 
 def setup_paths():
-    sys.path = list(get_paths(PATHS)) + sys.path + APPEND_PATHS
+    sys.path = list(get_paths(PATHS)) + sys.path
+    if 'I_CAN_HAZ_JUJU' not in os.environ:
+        sys.path += APPEND_PATHS
     os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'settings')
 
 

=== added symlink 'django_project/settings_acceptance_juju.py'
=== target is u'../branches/config/settings_acceptance_juju.py'
=== modified file 'django_project/settings_base.py'
--- django_project/settings_base.py	2016-01-27 12:00:39 +0000
+++ django_project/settings_base.py	2016-03-04 19:06:06 +0000
@@ -503,7 +503,7 @@
 SESSION_SAVE_EVERY_REQUEST = False
 SESSION_SERIALIZER = 'django.contrib.sessions.serializers.PickleSerializer'
 SETTINGS_ENCODING = 'utf-8'
-SETTINGS_MODULE = 'django_project.settings'
+SETTINGS_MODULE = 'settings'
 SHORT_DATETIME_FORMAT = 'm/d/Y P'
 SHORT_DATE_FORMAT = 'm/d/Y'
 SHOW_USER_NUM_AUTHLOGS = 50

=== added file 'django_project/settings_devel_juju.py'
--- django_project/settings_devel_juju.py	1970-01-01 00:00:00 +0000
+++ django_project/settings_devel_juju.py	2016-03-04 19:06:06 +0000
@@ -0,0 +1,54 @@
+import os
+
+os.environ.setdefault('SSO_ROOT_URL', 'http://0.0.0.0:8080/')
+from settings import *  # noqa
+
+ALLOWED_HOSTS = ['*']
+CSRF_COOKIE_SECURE = False
+DEBUG = True
+GARGOYLE_SWITCH_DEFAULTS = {
+    'TWOFACTOR': {'is_active': True},
+    'CAN_VIEW_SUPPORT_PHONE': {'is_active': True},
+    'CAPTCHA': {'is_active': False},
+    'PREFLIGHT': {'is_active': True},
+    'LOGIN_BY_TOKEN': {'is_active': True},
+}
+PASSWORD_HASHERS = [
+    'django.contrib.auth.hashers.SHA1PasswordHasher',
+    'identityprovider.auth.BCrypt13PasswordHasher',
+    'django.contrib.auth.hashers.PBKDF2PasswordHasher',
+]
+SESSION_COOKIE_SECURE = False
+STATSD_CLIENT = 'django_statsd.clients.null'
+TEMPLATE_DEBUG = True
+TESTING = True
+TWOFACTOR_MANDATORY_TEAMS = ['ubuntuone-hackers']
+# Load a minimal blacklist in development so as to not need too much
+# time and memory and not break any tests
+PASSWORD_BLACKLIST_FILE = os.path.join(BASE_DIR, "blacklist-devel.txt")
+
+# this config is only needed for running the sst acceptance tests
+# the server doesn't need this config
+EMAIL_ADDRESS_PATTERN = '[e-mail]'
+SSO_TEST_ACCOUNT_EMAIL = '[e-mail]'
+SSO_TEST_ACCOUNT_FULL_NAME = 'Test Account'
+SSO_TEST_ACCOUNT_PASSWORD = 'password'
+TEST_ACCOUNT_EMAIL = '[e-mail]'
+TEST_ACCOUNT_PASSWORD = 'password'
+TEST_DISCOVER_ROOT = SRC_DIR
+TEST_DISCOVER_TOP_LEVEL = SRC_DIR
+TEST_PROFILE = 'test.prof'
+TEST_RUNNER = 'testing.runner.TestRunner'
+
+# localmail imap server
+IMAP_PORT = 2143
+IMAP_USERNAME = 'DOESNOTMATTER'
+IMAP_PASSWORD = 'DOESNOTMATTER'
+IMAP_SERVER = 'localhost'
+IMAP_USE_SSL = False
+
+
+try:
+    from django_project.settings_local import *  # noqa
+except ImportError:
+    pass

=== added file 'django_project/ssh_config'
--- django_project/ssh_config	1970-01-01 00:00:00 +0000
+++ django_project/ssh_config	2016-03-04 19:06:06 +0000
@@ -0,0 +1,8 @@
+# disable all key checking and warnings
+CheckHostIP no
+UserKnownHostsFile /dev/null
+StrictHostKeyChecking no
+LogLevel ERROR
+# pass through JUJU_ENV
+SendEnv JUJU_ENV
+SendEnv I_CAN_HAZ_JUJU

=== modified file 'requirements_devel.txt'
--- requirements_devel.txt	2016-01-20 18:34:00 +0000
+++ requirements_devel.txt	2016-03-04 19:06:06 +0000
@@ -7,7 +7,7 @@
 fixtures==0.3.12
 flake8==2.4.0
 junitxml==0.7
-localmail==0.3
+localmail==0.4.1
 logilab-astng==0.24.3
 logilab-common==0.59.1
 mechanize==0.2.5

=== added file 'run-tests-juju'
--- run-tests-juju	1970-01-01 00:00:00 +0000
+++ run-tests-juju	2016-03-04 19:06:06 +0000
@@ -0,0 +1,52 @@
+#! /bin/bash
+# How the tests are run in Jenkins by Tarmac
+#
+# ./run-tests -- run unit/acceptance tests for an environment.
+#
+# Usage: ./run-tests [dev|staging|production]
+set -e
+
+TARGET=$1
+BOOTSTRAP=${2:-yes}
+
+if [ "$TARGET" = "production" ]; then
+    SST_BASE_URL="https://login.ubuntu.com"
+elif [ "$TARGET" = "staging" ]; then
+    SST_BASE_URL="https://login.staging.ubuntu.com"
+fi
+
+# Some Jenkins jobs copy workspace.tar.gz from successful build.
+if [ -r "workspace.tar.gz" ]; then
+    tar zxf workspace.tar.gz -C .
+fi
+
+# clean old results
+rm -rf results/*
+
+# make sure that dependencies are up to date
+if [ $BOOTSTRAP = "yes" ]; then
+    echo "Setting up lxc..."
+    make lxc-setup
+    echo "Bootstrapping..."
+    make clean && make bootstrap
+fi
+
+# Before setting up the DB, ensure developer generated all needed migrations
+# Will abort if there are pending migrations
+echo "Check missing Django migrations left behind. If output is not 'No changes detected', run will abort."
+make manage ARGS=makemigrations | tee /tmp/sso.log && grep "No changes detected" /tmp/sso.log > /dev/null
+
+# run tests
+if [[ -z ${TARGET} ]]; then
+    echo "Updating translation messages"
+    make makemessages
+    echo "Running canonical-identity-provider tests in tarmac"
+    make test ARGS="--noinput --failfast"
+else
+    # run acceptance tests if TARGET given
+    if [ ${TARGET} = "dev" ]; then
+        make acceptance-dev
+    else
+        make run-acceptance SST_BASE_URL="$SST_BASE_URL" DJANGO_SETTINGS_MODULE=django_project.settings_acceptance_juju 
+    fi
+fi

=== added file 'scripts/acceptance-dev-juju.sh'
--- scripts/acceptance-dev-juju.sh	1970-01-01 00:00:00 +0000
+++ scripts/acceptance-dev-juju.sh	2016-03-04 19:06:06 +0000
@@ -0,0 +1,66 @@
+#!/bin/bash
+
+set -ue
+
+function usage {
+  cat  << EOF
+Usage: $0 [-t TESTCASE] [-h|-f] [-p PORT]
+Run SSO acceptance tests on a random port.
+  -t [TESTCASE]   run using TESTCASE regex. Will not reset db
+  -h              run headless (default)
+  -f              run headful
+  -p [PORT]       use a specific port (e.g. -p 8001)
+  -c              force a clean db (if using -t option)
+  -1              stop on the first failure (AKA "fail-fast")
+EOF
+}
+
+# random port by default
+PORT=8080
+TESTCASE=
+HEADLESS="-x"
+CLEAN="false"
+FAILFAST=
+
+while getopts ":t:hfc1p:" opt; do
+  case $opt in
+    t) TESTCASE=$OPTARG ;;
+    h) HEADLESS="-x" ;;
+    f) HEADLESS="" ;;
+    p) PORT=$OPTARG ;;
+    c) CLEAN="true" ;;
+    1) FAILFAST="--failfast" ;;
+    \?)
+      echo "Invalid option: -$OPTARG" >&2
+      usage
+      exit 1
+      ;;
+    :)
+      echo "Option -$OPTARG requires an argument." >&2
+      usage
+      exit 1
+      ;;
+  esac
+done
+
+export PORT=$PORT
+export SSO_HOSTNAME="0.0.0.0:$PORT"
+export SSO_ROOT_URL="http://$SSO_HOSTNAME"
+export SST_BASE_URL="http://$SSO_HOSTNAME"
+
+# only reset db if we're doing a full run or explicitly told to
+if [ -z $TESTCASE ] || $CLEAN
+then
+  echo "Setting up acceptance database"
+  make acceptance-db
+  make collectstatic
+fi
+
+SST_ARGS="-s -r xml -q -c 1 $FAILFAST $HEADLESS"
+
+if [ -n "$TESTCASE" ]; then
+  SST_ARGS="$SST_ARGS \"$TESTCASE\""
+fi
+echo $SST_ARGS
+
+make run-acceptance ARGS="$SST_ARGS" DJANGO_SETTINGS_MODULE=django_project.settings_devel_juju

=== added file 'scripts/setup-localmail.sh'
--- scripts/setup-localmail.sh	1970-01-01 00:00:00 +0000
+++ scripts/setup-localmail.sh	2016-03-04 19:06:06 +0000
@@ -0,0 +1,47 @@
+#!/bin/bash
+set -eu
+
+sudo service localmail stop || true
+
+LOGS=$PWD/logs
+LOCALMAIL=/srv/localmail
+if [ ! -d $LOCALMAIL ]
+then
+    sudo mkdir -p $LOCALMAIL
+    sudo chown $USER:$USER $LOCALMAIL
+    virtualenv $LOCALMAIL
+    $LOCALMAIL/bin/pip install -U pip
+    
+fi
+$LOCALMAIL/bin/pip install -U localmail
+upstart=/tmp/localmail.conf
+cron=/tmp/localmail.sh
+
+cat > $upstart <<EOF
+description "localmail development smtp/imap server"
+
+start on (local-filesystems and net-device-up IFACE=eth0)
+stop on runlevel [!12345]
+
+# If the process quits unexpectadly trigger a respawn
+respawn
+respawn limit 10 5
+
+setuid $USER
+setgid $USER
+chdir $PWD
+
+exec $LOCALMAIL/bin/twistd -n --pidfile= --logfile $LOGS/localmail.log localmail --smtp 2025 --imap 2143 --http 8880 --file $LOGS/localmail.mbox
+EOF
+
+cat > $cron <<EOF
+#!/bin/bash
+service localmail stop
+rm -f $LOGS/localmail.mbox $LOGS/localmail.log
+service localmail start
+EOF
+
+sudo mv $upstart /etc/init/localmail.conf
+sudo mv $cron /etc/cron.daily/localmail
+sudo chmod +x /etc/cron.daily/localmail
+sudo service localmail start

=== modified file 'src/identityprovider/tests/test_command_cleanup.py'
--- src/identityprovider/tests/test_command_cleanup.py	2015-12-09 20:01:10 +0000
+++ src/identityprovider/tests/test_command_cleanup.py	2016-03-04 19:06:06 +0000
@@ -47,8 +47,8 @@
 
     def make_test_accounts(self, count=0, date_created=None):
         for i in xrange(count):
-            email = self.factory.make_email_address(prefix='isdtest+',
-                                                    domain='canonical.com')
+            email = self.factory.make_email_address(prefix='testemail+',
+                                                    domain='example.com')
 
             account = self.factory.make_account(email=email)
             if date_created is not None:
@@ -134,7 +134,7 @@
     def assert_testdata(self, accounts=0, emails=0, passwords=0):
         tomorrow = now().date() + timedelta(days=1)
         test_emails = EmailAddress.objects.filter(
-            email__iregex=r'^isdtest\+[^@]+@canonical\.com$',
+            email__iregex=r'^testemail\+[^@]+@example\.com$',
             date_created__lt=tomorrow)
         test_accounts = Account.objects.filter(
             displayname__startswith='Test Account')
@@ -209,7 +209,7 @@
     def test_cleanup_orphaned_accounts(self):
         self.make_test_accounts(count=10)
         emails = EmailAddress.objects.filter(
-            email__iregex=r'^isdtest\+[^@]+@canonical\.com$')
+            email__iregex=r'^testemail\+[^@]+@example\.com$')
         email_ids = emails.values_list('pk')[:5]
         EmailAddress.objects.filter(pk__in=email_ids).delete()
 

=== modified file 'src/webui/tests/test_views_ui.py'
--- src/webui/tests/test_views_ui.py	2016-01-13 19:11:31 +0000
+++ src/webui/tests/test_views_ui.py	2016-03-04 19:06:06 +0000
@@ -606,7 +606,6 @@
 
     def assert_logged_out_in_sso(self, response):
         self.assertTemplateUsed(response, 'registration/logout.html')
-        self.assertContains(response, 'You have been logged out')
         self.assertIsInstance(response.context['user'], AnonymousUser)
 
     def assert_redirect_as_per_request(self, response, redirected_to):