Merge lp:~james-w/pkgme-service/fab-deploy-tasks into lp:pkgme-service

[James Westby]

Hi,

This adds some fab tasks for helping with spinning up dev instances.

There's no tests unfortunately.

There's a command to deploy to an existing instance, and one to spin
up an ec2 instance and deploy to it (like Launchpad's "ec2 demo".)

There's also a couple of commands to help with managing the ec2 instances.

The pre-requisites are rather convoluted, but that seems inevitable with ec2
unfortuanately, and they are documented as best as I know how at this time.

Thanks,

James

[Jonathan Lange (jml)]

This all looks OK to me.

It's a bit disappointing that there's not a Python library that provides some of the stuff that you hand-roll in the first few functions.

Any ideas on how we can prevent this from sprawling into a project the size of Launchpad's ec2test?

Also, when can we target this at a local virtual machine / chroot / LXC thing?

[Jonathan Lange (jml)]

review: Approve

[Jonathan Lange (jml)]

I consistently get this error:

$ fab deploy_to_ec2
Waiting for instance i-3c7d7f5e to start...
Instance started as ec2-23-20-25-159.compute-1.amazonaws.com
Waiting for ssh to come up
[<email address hidden>:22] run: sudo add-apt-repository ppa:canonical-sysadmins/puppet

Fatal error: Timed out trying to connect to ec2-23-20-25-159.compute-1.amazonaws.com

Aborting.

[Jonathan Lange (jml)]

review: Needs Fixing

[James Westby (james-w)]

On Thu, 19 Jan 2012 10:13:31 -0000, Jonathan Lange <email address hidden> wrote:
> It's a bit disappointing that there's not a Python library that provides some of the stuff that you hand-roll in the first few functions.

Indeed, I don't think that boto has a location it looks, or knows about
the ubuntu-specific script I use. I don't think it's worth turning it in
to a library at this point though.

> Any ideas on how we can prevent this from sprawling into a project the size of Launchpad's ec2test?

I wouldn't mind something the size of Launchpad's ec2test if it didn't
live in this project, or at least the parts that aren't specific to
pkgme-service.

Probably we'll abandon it all in favour of juju in a few months though.

> Also, when can we target this at a local virtual machine / chroot / LXC thing?

If you have one up the deploy_to_existing should work ok (it could do
with dealing with not having a fresh install a little better if you want
to re-use.)

We could add lxc without too much work I expect, but I haven't played
with lxc directly yet to know what to do.

Thanks,

James

[lp:~james-w/pkgme-service/fab-deploy-tasks]

29. By James Westby on 2012-01-19

Merge puppet improvements.

30. By James Westby on 2012-01-19

Merge puppet fixes.

31. By James Westby on 2012-01-19

Set the security group of the created instance so it is accessible.

[James Westby (james-w)]

On Thu, 19 Jan 2012 10:59:59 -0000, Jonathan Lange <email address hidden> wrote:
> I consistently get this error:
>
> $ fab deploy_to_ec2
> Waiting for instance i-3c7d7f5e to start...
> Instance started as ec2-23-20-25-159.compute-1.amazonaws.com
> Waiting for ssh to come up
> [<email address hidden>:22] run: sudo add-apt-repository ppa:canonical-sysadmins/puppet
>
> Fatal error: Timed out trying to connect to ec2-23-20-25-159.compute-1.amazonaws.com
>
> Aborting.

Hi,

This should now work with the latest changes pushed, as it now creates a
security group for the instance that gives the world access to 22,80,443
(I didn't think that locking it down to specific IP addresses gained us
much at all.)

Thanks,

James

[lp:~james-w/pkgme-service/fab-deploy-tasks]

32. By James Westby on 2012-01-19

Delete the security group before the instance.

[Jonathan Lange (jml)]

On Thu, Jan 19, 2012 at 3:06 PM, James Westby <email address hidden> wrote:
> On Thu, 19 Jan 2012 10:59:59 -0000, Jonathan Lange <email address hidden> wrote:
>> I consistently get this error:
>>
>> $ fab deploy_to_ec2
>> Waiting for instance i-3c7d7f5e to start...
>> Instance started as ec2-23-20-25-159.compute-1.amazonaws.com
>> Waiting for ssh to come up
>> [<email address hidden>:22] run: sudo add-apt-repository ppa:canonical-sysadmins/puppet
>>
>> Fatal error: Timed out trying to connect to ec2-23-20-25-159.compute-1.amazonaws.com
>>
>> Aborting.
>
> Hi,
>
> This should now work with the latest changes pushed, as it now creates a
> security group for the instance that gives the world access to 22,80,443
> (I didn't think that locking it down to specific IP addresses gained us
> much at all.)
>

Works now. Thanks.

jml

[Jonathan Lange (jml)]

review: Approve

=== modified file 'fabtasks/__init__.py'
--- fabtasks/__init__.py	2011-08-29 23:50:07 +0000
+++ fabtasks/__init__.py	2012-01-19 18:09:25 +0000
@@ -1,2 +1,3 @@
 from .bootstrap import *
+from .deploy import *
 from .django import *
 
=== added file 'fabtasks/deploy.py'
--- fabtasks/deploy.py	1970-01-01 00:00:00 +0000
+++ fabtasks/deploy.py	2012-01-19 18:09:25 +0000
@@ -0,0 +1,284 @@
+from datetime import datetime
+import os
+import subprocess
+import time
+
+from fabric.api import env, run
+from fabric.utils import abort, puts
+from fabric.operations import open_shell as _open_shell
+
+from boto import ec2
+
+
+RELEASE = 'lucid'
+ARCH = 'amd64'
+FS_TYPE = 'ebs'
+INSTANCE_TYPE = 't1.micro'
+USERNAME = 'ubuntu'
+REGION_NAME = 'us-east-1'
+
+NAME_KEY = "name"
+NAME_PREFIX = "fab/pkgme-service"
+
+ALL_HOSTS = "0.0.0.0/0"
+
+
+def open_shell():
+    """Open a shell on the remote host."""
+    _open_shell()
+
+
+def _get_aws_credentials():
+    """Get the AWS credentials for the user from ~/.ec2/aws_id.
+
+    :returns: a tuple of access key id, secret access key
+    """
+    key_id = None
+    secret_access_key = None
+    with open(os.path.expanduser("~/.ec2/aws_id")) as f:
+        for i, line in enumerate(f.readlines()):
+            if i == 0:
+               key_id = line.strip()
+            if i == 1:
+               secret_access_key = line.strip()
+    if key_id is None:
+        raise AssertionError("Missing key id in ~/.ec2/aws_id")
+    if secret_access_key is None:
+        raise AssertionError("Missing secret access key in ~/.ec2/aws_id")
+    return key_id, secret_access_key
+
+
+def _get_ami_id(region_name):
+    """Get the AMI to use for a particular region.
+
+    This consults the ubuntu-cloudimg-query tool to find out the best
+    AMI to use for a particular region.
+
+    :returns: the ami id (as a string)
+    """
+    proc = subprocess.Popen(
+        ['ubuntu-cloudimg-query', RELEASE, ARCH, FS_TYPE, region_name],
+        stdout=subprocess.PIPE)
+    stdout, _ = proc.communicate()
+    if proc.returncode != 0:
+        raise AssertionError("calling ubuntu-cloudimg-query failed")
+    return stdout.strip()
+
+
+def _get_security_group(conn, name):
+    security_group = conn.create_security_group(name,
+            "Access to the fab pkgme-service ec2 deployment %s" % name)
+    security_group.authorize('tcp', 22, 22, ALL_HOSTS)
+    security_group.authorize('tcp', 80, 80, ALL_HOSTS)
+    security_group.authorize('tcp', 443, 443, ALL_HOSTS)
+    return security_group
+
+
+def _new_ec2_instance(keypair):
+    """Starts a new ec2 instance, giving the specified keypair access.
+
+    This will use the AWS credentials from ~/.ec2/aws_id, and the
+    AMI recommended by ubuntu-cloudimg-query.
+
+    :returns: the boto Instance object of the launched instance. It
+        will be in a state where it can be accessed over ssh.
+    """
+    key_id, secret_access_key = _get_aws_credentials()
+    ami_id = _get_ami_id(REGION_NAME)
+    conn = ec2.connect_to_region(REGION_NAME, aws_access_key_id=key_id,
+        aws_secret_access_key=secret_access_key)
+    image = conn.get_image(ami_id)
+    now = datetime.utcnow()
+    name = "%s/%s" % (NAME_PREFIX, now.isoformat())
+    security_group = _get_security_group(conn, name)
+    reservation = image.run(instance_type=INSTANCE_TYPE, key_name=keypair,
+            security_groups=[security_group.name])
+    instance = reservation.instances[0]
+    puts("Waiting for instance %s to start..." % instance.id)
+    while True:
+        time.sleep(10)
+        instance.update()
+        if instance.state != 'pending':
+           break
+    if instance.state != 'running':
+        raise AssertionError("Instance failed to start")
+    puts("Instance started as %s" % instance.dns_name)
+    puts("Waiting for ssh to come up")
+    # FIXME: use something better than a sleep to determine this.
+    time.sleep(30)
+    instance.add_tag(NAME_KEY, name)
+    return instance
+
+
+def deploy_to_ec2(branch="lp:pkgme-service", use_staging_deps=True, pkgme_branch="lp:pkgme", pkgme_binary_branch="lp:pkgme-binary", keypair='ec2-keypair'):
+    """Deploy to a new ec2 instance.
+
+    This command will spin up an ec2 instance, and deploy to it.
+
+    To run this command you must first set up an ec2 account.
+
+        http://aws.amazon.com/ec2/
+
+    Once you have that create access keys for this to use. Go to
+
+        https://aws-portal.amazon.com/gp/aws/developer/account?ie=UTF8&action=access-key
+
+    and create an access key. Then create a file at ~/.ec2/aws_id and
+    put the "Access Key ID" on the first line, and the "Secret Access Key" on
+    the second line (with nothing else on either line.)
+
+    Next you will need an ec2 keypair. Go to the EC2 console at
+
+        https://console.aws.amazon.com/ec2/home?region=us-east-1
+
+    click on "Key Pairs" and then "Create Key Pair", name the keypair "ec2-keypair".
+    Save the resulting file as
+
+        ~/.ec2/ec2-keypair.pem
+
+    Now you are ready to deploy, so run
+
+       fab deploy_to_ec2:keypair=ec2-keypair -i ~/.ec2/ec2-keypair.pem
+
+    and wait.
+
+    Note that you will be responsible for terminating the instance after
+    use (see the destroy_ec2_instances command.)
+
+    You can also specify the following arguments:
+
+      * branch: the branch to deploy, defaults to lp:pkgme-service. To
+          test some in-progress work push to lp:~you/pkgme-service/something
+          and then specify that as the branch.
+
+      * pkgme_branch: the branch of pkgme to deploy, defaults to
+        lp:pkgme.
+
+      * pkgme_binary_branch: the branch of pkgme-binary to deploy,
+        defaults to lp:pkgme-binary.
+
+      * use_staging_deps: whether to use the staging PPA for
+        dependencies, as well as the production one, defaults to True.
+
+    Arguments are all specified by attaching them to the command name, e.g.
+
+      fab deploy_to_ec2:keypair=ec2-keypair,branch=lp:~me/pkgme-service/something
+    """
+    instance = _new_ec2_instance(keypair)
+    _set_instance_as_host(instance)
+    deploy_to_existing(branch=branch, use_staging_deps=use_staging_deps, pkgme_branch=pkgme_branch, pkgme_binary_branch=pkgme_binary_branch)
+
+
+def _set_instance_as_host(instance):
+    """Set the host to be acted on by fab to this instance."""
+    env.host_string = "%s@%s:22" % (USERNAME, instance.dns_name)
+
+
+def deploy_to_existing(branch="lp:pkgme-service", use_staging_deps=True, pkgme_branch="lp:pkgme", pkgme_binary_branch="lp:pkgme-binary"):
+    """Deploy to an existing instance.
+
+    This command will deploy to an existing instance. Don't use it on an
+    instance you care about as it may overwrite anything.
+
+    To specify the host to deploy to use the -H option of fab:
+
+      fab -H ubuntu@<host> deploy_to_existing
+
+    You can also specify the following arguments:
+
+      * branch: the branch to deploy, defaults to lp:pkgme-service. To
+          test some in-progress work push to lp:~you/pkgme-service/something
+          and then specify that as the branch.
+
+      * pkgme_branch: the branch of pkgme to deploy, defaults to
+        lp:pkgme.
+
+      * pkgme_binary_branch: the branch of pkgme-binary to deploy,
+        defaults to lp:pkgme-binary.
+
+      * use_staging_deps: whether to use the staging PPA for
+        dependencies, as well as the production one, defaults to True.
+
+    Arguments are all specified by attaching them to the command name, e.g.
+
+      fab deploy_to_ec2:keypair=ec2-keypair,branch=lp:~me/pkgme-service/something
+    """
+    # Get the puppet used by IS
+    run('sudo add-apt-repository ppa:canonical-sysadmins/puppet')
+    # Add our dependency PPA
+    run('sudo add-apt-repository ppa:canonical-ca-hackers/production')
+    if use_staging_deps:
+        # Add our dependency staging PPA
+        run('sudo add-apt-repository ppa:canonical-ca-hackers/staging')
+    run('sudo apt-get update -q')
+    # Upgrade the base system in case we are shipping any updates in
+    # our PPAs
+    run('sudo apt-get dist-upgrade -q -y --force-yes')
+    # Avoid a debconf note when installing rabbitmq-server on lucid
+    run('echo "rabbitmq-server rabbitmq-server/upgrade_previous note" | sudo debconf-set-selections')
+    # Install the dependencies needed to get puppet going
+    # TODO: move the rest of the dependencies to puppet
+    run('sudo apt-get install -q -y --force-yes pkgme-service-dependencies bzr apache2 libapache2-mod-wsgi rabbitmq-server postgresql-8.4 puppet')
+    # Grab the branches
+    # TODO: investigate re-using IS' config-manager config
+    run('bzr branch -q %s pkgme-service' % branch)
+    run('bzr branch -q %s pkgme-service/sourcecode/pkgme' % pkgme_branch)
+    run('bzr branch -q %s pkgme-service/sourcecode/pkgme-binary' % pkgme_binary_branch)
+    run('cd pkgme-service/sourcecode/pkgme && python setup.py build')
+    run('cd pkgme-service/sourcecode/pkgme-binary && python setup.py build')
+    # Grab canonical-memento and use it?
+    # Run puppet to set everything else up.
+    run('./pkgme-service/dev_config/apply')
+
+
+def _connect_to_ec2():
+    """Get a connection to ec2, using the credentials on disk."""
+    key_id, secret_access_key = _get_aws_credentials()
+    conn = ec2.connect_to_region(REGION_NAME, aws_access_key_id=key_id,
+        aws_secret_access_key=secret_access_key)
+    return conn
+
+
+def _get_started_ec2_instances(conn):
+    """Get the ec2 instances started from here."""
+    for reservation in conn.get_all_instances():
+        for instance in reservation.instances:
+            if instance.state == 'running':
+                tags = getattr(instance, "tags", {})
+                name = tags.get(NAME_KEY, None)
+                if name and name.startswith(NAME_PREFIX):
+                    yield instance
+
+
+def destroy_ec2_instances():
+    """Destroy any ec2 instances created by this deployment."""
+    conn = _connect_to_ec2()
+    for instance in _get_started_ec2_instances(conn):
+        puts("Stopping %s" % instance.id)
+        conn.delete_security_group(name=instance.tags[NAME_KEY])
+        instance.terminate()
+
+
+def last_ec2_launched():
+    """Cause other commands to act on the last ec2 instance launched from here.
+
+    Use this in a list of commands to have the rest of the commands act on the
+    last ec2 instance launched with deploy_to_ec2, e.g.
+
+        fab -i ~/.ec2/ec2-keypair.pem last_ec2_launched -- ls
+
+    This will then run ls on that instance.
+    """
+    def get_date(instance):
+        return instance.tags[NAME_KEY][len(NAME_PREFIX):]
+    last_instance = None
+    conn = _connect_to_ec2()
+    for instance in _get_started_ec2_instances(conn):
+        if last_instance is None:
+            last_instance = instance
+        else:
+            if get_date(instance) > get_date(last_instance):
+                last_instance = instance
+    if last_instance is None:
+        abort("No instances found.")
+    _set_instance_as_host(last_instance)