Merge lp:~vila/ubuntu-ci-services-itself/ppa-assigner-integration-test into lp:ubuntu-ci-services-itself

[Vincent Ladeuil]

First integration tests for ppa assigner using amulet and juju deployer.

Here is a first succesful attempt at running an integration test using juju
deployer and amulet.

Andy's ppa-assigner juju deployer script was chosen for that, sorry Andy ;)
Or rather, thanks Andy !

With the help of Martin we diagnosed hundreds^W a bunch of various issues
that we tried to capture but... long story short, we had to take a few
shortcuts ;)

Still, any of us should be able to reproduce the experience following the
instructions in the README.

To summarize:
- cloud credentials are assumed to be available, so source your novarc first.
- sshuttle needs to be setup manually each time you run 'juju bootstrap'

- you need to install amulet from lp:~vila/amulet/ci and setup your env
  appropriately (my PYTHONPATH includes ~/lib/python so I create a symlink
  there to the amulet sources)

From that point, the test run is automatic and can be re-run at will.

It will deploy the ppa-assigner or reuse the one you deploy before running
the test.

A single test has been created for now and the next step will be to turn it
into a regular python unittest one but we wanted to share that first
significant step.

That test issue a request to get the list of the ppas. The request succeeds
(200, OK) but the objects in the content is empty. I don't know if that's
because we've mis-configured something or a genuine bug, please advise !

But more importantly, try this at home and tell us what works and what
doesn't so we can find the Right Way to automate this further.

[Evan (ev)]

On 18 December 2013 17:51, Vincent Ladeuil <email address hidden> wrote:
> === added file 'tests/ppa_assigner/test.py'
> --- tests/ppa_assigner/test.py 1970-01-01 00:00:00 +0000
> +++ tests/ppa_assigner/test.py 2013-12-18 17:42:04 +0000
> +opd = os.path.dirname
> +
> +root_dir = opd(opd(opd((os.path.abspath(__file__)))))

Surely:

os.path.abspath(os.path.join ('../../../', __file__))

[Andy Doan (doanac)]

The README is great info, but its almost too much and too specific to canonistack. I wonder if we should create something like "README-juju" or README-canonistack for that stuff. I'm talking roughly about:

18 +Configure juju for canonistack (or whatever you feel is appropriate), make
to
50 +You'll need to interrupt this command once you're run 'juju destroy-environment' to ensure you didn't leave pending connections to juju instances that should not exist anymore.

Can you explain why:

142 + expose: True

was needed? I'm trying to understand if my charming was done incorrectly or amulet has some need of that or what?

I'm not sure about:

185 + script = {depl.juju_env: jd_script.values()[0]}

Should we be explicit and load jd_script['ppa-assigner-staging']?

As for the remainder of the script - i can see plenty of room for us to expand there soon! thanks

[Vincent Ladeuil (vila)]

> On 18 December 2013 17:51, Vincent Ladeuil <email address hidden> wrote:
> > === added file 'tests/ppa_assigner/test.py'
> > --- tests/ppa_assigner/test.py 1970-01-01 00:00:00 +0000
> > +++ tests/ppa_assigner/test.py 2013-12-18 17:42:04 +0000
> > +opd = os.path.dirname
> > +
> > +root_dir = opd(opd(opd((os.path.abspath(__file__)))))
>
> Surely:
>
> os.path.abspath(os.path.join ('../../../', __file__))

Yeah, right, as explained using '/' instead of os.dirsep is not portable. We don't care here, but old good habits... never die ;)

Plus, if __file__ is a relative path you end up with os.path.abspath('') which won't work I think.

So, yeah, this is hackish but should disappear so, I'll add a comment to that effect.

[Vincent Ladeuil (vila)]

> The README is great info, but its almost too much and too specific to
> canonistack. I wonder if we should create something like "README-juju" or
> README-canonistack for that stuff. I'm talking roughly about:
>
> 18 +Configure juju for canonistack (or whatever you feel is appropriate),
> make
> to
> 50 +You'll need to interrupt this command once you're run 'juju destroy-
> environment' to ensure you didn't leave pending connections to juju instances
> that should not exist anymore.

Will do.

>
>
> Can you explain why:
>
> 142 + expose: True
>
> was needed? I'm trying to understand if my charming was done incorrectly or
> amulet has some need of that or what?

Damn, sorry, I wanted to explain that part and will add a comment.

According to Martin, you can try to set it to False and from your desktop you shouldn't be able to access it anymore. So it's required to allow access from outside the cloud.

>
> I'm not sure about:
>
> 185 + script = {depl.juju_env: jd_script.values()[0]}
>
> Should we be explicit and load jd_script['ppa-assigner-staging']?

Well, we chose not to so if that name change, the test won't have to be updated.

>
> As for the remainder of the script - i can see plenty of room for us to expand
> there soon! thanks

Great.

[Vincent Ladeuil (vila)]

> > The README is great info, but its almost too much and too specific to
> > canonistack. I wonder if we should create something like "README-juju" or
> > README-canonistack for that stuff. I'm talking roughly about:
> >
> > 18 +Configure juju for canonistack (or whatever you feel is
> appropriate),
> > make
> > to
> > 50 +You'll need to interrupt this command once you're run 'juju
> destroy-
> > environment' to ensure you didn't leave pending connections to juju
> instances
> > that should not exist anymore.
>
> Will do.

Hmm, that's only ~30 lines, so I'd rather make sure we properly curate a single README file when these steps are automated rather than introducing several READMEs.

I feel more comfortable with too much instructions rather than too little (despite the discussion about shuttle on the mailing list we still had to re-discover some of the tricks explained here).

[Chris Johnston (cjohnston)]

Voting does not meet specified criteria. Required: Approve >= 1. Got: 1 Pending.

[Para Siva (psivaa)]

+1

review: Approve

=== modified file 'README'
--- README	2013-12-12 22:34:51 +0000
+++ README	2013-12-18 17:42:04 +0000
@@ -64,3 +64,58 @@
 
 Each service should include a juju-deployer config under the juju-deployer
 directory.
+
+ppa assigner testing with juju-deployer and amulet
+--------------------------------------------------
+
+Install pre-requisites:
+
+$ sudo apt-get install juju juju-deployer python-httplib2
+
+Branch lp:amulet and make it reachable.
+
+Configure juju for canonistack (or whatever you feel is appropriate), make
+sure you select the region there (see OS_REGION_NAME/NOVA_REGION):
+
+$ . ~/.canonistack/novarc
+
+Create a proper ~/.juju/environments.yaml, example:
+
+environments:
+  ## https://juju.ubuntu.com/docs/config-openstack.html
+  lcy02:
+    type: openstack
+    admin-secret: <your admin secret, juju can auto-create it>
+    # Globally unique swift bucket name
+    control-bucket: <you control-bucket secret, juju can auto-create it>
+    # Usually set via the env variable OS_REGION_NAME, but can be specified here
+    region: lcy02
+
+$ juju bootstrap
+
+Get the IP address of the juju state server from 'nova list --name machine-0':
+
+$ nova list --name machine-0 --fields networks
++--------------------------------------+-------------------------+
+| ID                                   | Networks                |
++--------------------------------------+-------------------------+
+| 0463d358-750c-4fc4-b38b-a618e2e7d034 | canonistack=10.55.32.64 |
++--------------------------------------+-------------------------+
+
+Start sshuttle in the foreground from a different terminal where you've sourced your credentials (we need a way to get that in a more script friendly way):
+
+$ sshuttle -r ubuntu@10.55.32.64 10.55.0.0/16
+
+You'll need to interrupt this command once you're run 'juju destroy-environment' to ensure you didn't leave pending connections to juju instances that should not exist anymore.
+
+Deploy ppa-assigner (mostly a smoke check, amulet will deploy it anyway or
+reuse the already deployed one):
+
+$ juju-deployer -c juju-deployer/ppa-assigner.yaml ppa-assigner-staging
+
+Run the ppa assigner test:
+
+$ tests/ppa_assigner/test.py 
+
+It currently fails because an empty ppa list is returned, feedback/fixes
+welcome :-)

=== added file 'TRICKS'
--- TRICKS	1970-01-01 00:00:00 +0000
+++ TRICKS	2013-12-18 17:42:04 +0000
@@ -0,0 +1,66 @@
+====================================================
+Tricks used while debugging juju/juju=deployer stuff
+====================================================
+
+Setup a public IP for the juju state server
+===========================================
+
+Used when trying to allow juju status to be reached without settting up
+~/.ssh/config to proxy via chinstrap.canonistack.com.
+
+Declare a public ip address where we can connect to the juju state server on
+canonistack via ssh.
+
+$ OS_REGION_NAME=lcy02 nova list
++--------------------------------------+----------------------+--------+------------+-------------+--------------------------+
+| ID                                   | Name                 | Status | Task State | Power State | Networks                 |
++--------------------------------------+----------------------+--------+------------+-------------+--------------------------+
+| a5f350da-b576-4e4e-be7e-ee639cbb9683 | juju-lcy02-machine-0 | ACTIVE | None       | Running     | canonistack=10.55.32.186 |
++--------------------------------------+----------------------+--------+------------+-------------+--------------------------+
+
+The ID above (a5f350da-b576-4e4e-be7e-ee639cbb9683) is the juju state server.
+
+We need a public IP to reach it:
+
+$ OS_REGION_NAME=lcy02 nova floating-ip-create
++---------------+-------------+----------+-------+
+| Ip            | Instance Id | Fixed Ip | Pool  |
++---------------+-------------+----------+-------+
+| 162.213.35.63 | None        | None     | lcy02 |
++---------------+-------------+----------+-------+
+
+Assign that IP address to the juju state server:
+
+$ OS_REGION_NAME=lcy02 nova add-floating-ip a5f350da-b576-4e4e-be7e-ee639cbb9683 162.213.35.63
+
+This was used to try to get juju-deployer working but didn't work in the end
+as canonistack has an known issue about public IPs routing from inside the
+region.
+
+This was diagnosed from, connecting to some juju instance:
+
+$ ssh 10.55.32.111
+
+ubuntu@juju-lcy02-machine-1:~$ cd /var/log/juju
+ubuntu@juju-lcy02-machine-1:/var/log/juju$ tail machine-1.log
+
[phone]:52:07 INFO juju runner.go:245 worker: restarting "api" in 3s
[phone]:52:10 INFO juju runner.go:253 worker: start "api"
[phone]:52:10 INFO juju apiclient.go:111 state/api: dialing "wss://162.213.35.63:17070/"
[phone]:53:13 ERROR juju apiclient.go:116 state/api: websocket.Dial wss://162.213.35.63:17070/: dial tcp 162.213.35.63:17070: connection timed out
+
+Which shows that canonistack can't reach a public IP from inside the cloud itself :-/
+
+Make sure juju tears down stuff properly
+========================================
+
+After 'juju destroy-environment' make sure 'nova list' also comes back
+empty.
+
+
+amulet things it has charms in the current directory
+====================================================
+
+We need:
+vila:~/ci/ubuntu-ci-services-itself/trunk :) $ ln -s precise/python-django django
+

=== modified file 'juju-deployer/ppa-assigner.yaml'
--- juju-deployer/ppa-assigner.yaml	2013-12-13 20:00:59 +0000
+++ juju-deployer/ppa-assigner.yaml	2013-12-18 17:42:04 +0000
@@ -2,6 +2,7 @@
     series: precise
     services:
         django:
+            expose: True
             charm: python-django
             branch: lp:~canonical-ci-engineering/charms/precise/python-django/ci-services
             options:
@@ -23,5 +24,7 @@
             options:
                 python_path: /srv/django/ci-utils:/srv/django/ppa-assigner
     relations:
-        - [django, "postgres:db"]
-        - [django, gunicorn]
+        # Relations should be explicit as amulet can't infer them otherwise
+        # even if there is a single one
+        - ["django:pgsql", "postgres:db"]
+        - ["django:wsgi", "gunicorn:wsgi-file"]

=== added directory 'tests'
=== added directory 'tests/ppa_assigner'
=== added file 'tests/ppa_assigner/test.py'
--- tests/ppa_assigner/test.py	1970-01-01 00:00:00 +0000
+++ tests/ppa_assigner/test.py	2013-12-18 17:42:04 +0000
@@ -0,0 +1,48 @@
+#!/usr/bin/env python
+
+import json
+import httplib2
+import yaml
+import os
+import sys
+
+opd = os.path.dirname
+
+root_dir = opd(opd(opd((os.path.abspath(__file__)))))
+
+import amulet
+
+def main():
+    # FIXME: Running amulet with sentries is vurrently bogus in multiple ways
+    # and we can get juju status ourselves -- vila 2013-12-18
+    depl = amulet.Deployment(sentries=False)
+    jd_path = os.path.join(root_dir, 'juju-deployer', 'ppa-assigner.yaml')
+    with open(jd_path) as f:
+        jd_script = yaml.safe_load(f.read())
+    # Override the juju env provided by ppa-assigner or amulet gets confused
+    script = {depl.juju_env: jd_script.values()[0]}
+    depl.load(script)
+    # This is a no-op if (as expected) the ppa-assigner has already been
+    # deployed, but it *will* be deployed if needed anyway.
+    depl.setup()
+
+    status = amulet.waiter.status(depl.juju_env)
+    units = status['services']['django']['units']
+    ip = units['django/0']['public-address']
+    port, _ = units['django/0']['open-ports'][0].split('/')
+    # Issue a simple command against the ppa assigner API
+    url = 'http://{}:{}/api/v1/ppa'.format(ip, port)
+    client = httplib2.Http()
+    resp, content = client.request(url, 'GET')
+    jcontent = json.loads(content)
+    # Target
+    # self.assertEqual('200', resp['status'])
+    # self.assertNotEqual([], jcontent['objects'])
+    if '200' != resp['status']:
+        sys.exit(1)
+    if [] == jcontent['objects']:
+        sys.exit(2)
+
+
+if __name__ == "__main__":
+    main()