Merge lp:~james-w/pkgme-service/fab-deploy-tasks into lp:pkgme-service

Proposed by James Westby on 2012-01-18
Status: Merged
Approved by: Jonathan Lange on 2012-01-19
Approved revision: 32
Merged at revision: 26
Proposed branch: lp:~james-w/pkgme-service/fab-deploy-tasks
Merge into: lp:pkgme-service
Prerequisite: lp:~james-w/pkgme-service/puppet
Diff against target: 296 lines (+285/-0)
2 files modified
fabtasks/__init__.py (+1/-0)
fabtasks/deploy.py (+284/-0)
To merge this branch: bzr merge lp:~james-w/pkgme-service/fab-deploy-tasks
Reviewer Review Type Date Requested Status
Jonathan Lange (community) 2012-01-18 Approve on 2012-01-19
Review via email: mp+89165@code.launchpad.net

This proposal supersedes a proposal from 2012-01-18.

Description of the change

Hi,

This adds some fab tasks for helping with spinning up dev instances.

There's no tests unfortunately.

There's a command to deploy to an existing instance, and one to spin
up an ec2 instance and deploy to it (like Launchpad's "ec2 demo".)

There's also a couple of commands to help with managing the ec2 instances.

The pre-requisites are rather convoluted, but that seems inevitable with ec2
unfortuanately, and they are documented as best as I know how at this time.

Thanks,

James

To post a comment you must log in.
Jonathan Lange (jml) wrote :

This all looks OK to me.

It's a bit disappointing that there's not a Python library that provides some of the stuff that you hand-roll in the first few functions.

Any ideas on how we can prevent this from sprawling into a project the size of Launchpad's ec2test?

Also, when can we target this at a local virtual machine / chroot / LXC thing?

Jonathan Lange (jml) :
review: Approve
Jonathan Lange (jml) wrote :

I consistently get this error:

$ fab deploy_to_ec2
Waiting for instance i-3c7d7f5e to start...
Instance started as ec2-23-20-25-159.compute-1.amazonaws.com
Waiting for ssh to come up
[<email address hidden>:22] run: sudo add-apt-repository ppa:canonical-sysadmins/puppet

Fatal error: Timed out trying to connect to ec2-23-20-25-159.compute-1.amazonaws.com

Aborting.

Jonathan Lange (jml) :
review: Needs Fixing
James Westby (james-w) wrote :

On Thu, 19 Jan 2012 10:13:31 -0000, Jonathan Lange <email address hidden> wrote:
> It's a bit disappointing that there's not a Python library that provides some of the stuff that you hand-roll in the first few functions.

Indeed, I don't think that boto has a location it looks, or knows about
the ubuntu-specific script I use. I don't think it's worth turning it in
to a library at this point though.

> Any ideas on how we can prevent this from sprawling into a project the size of Launchpad's ec2test?

I wouldn't mind something the size of Launchpad's ec2test if it didn't
live in this project, or at least the parts that aren't specific to
pkgme-service.

Probably we'll abandon it all in favour of juju in a few months though.

> Also, when can we target this at a local virtual machine / chroot / LXC thing?

If you have one up the deploy_to_existing should work ok (it could do
with dealing with not having a fresh install a little better if you want
to re-use.)

We could add lxc without too much work I expect, but I haven't played
with lxc directly yet to know what to do.

Thanks,

James

29. By James Westby on 2012-01-19

Merge puppet improvements.

30. By James Westby on 2012-01-19

Merge puppet fixes.

31. By James Westby on 2012-01-19

Set the security group of the created instance so it is accessible.

James Westby (james-w) wrote :

On Thu, 19 Jan 2012 10:59:59 -0000, Jonathan Lange <email address hidden> wrote:
> I consistently get this error:
>
> $ fab deploy_to_ec2
> Waiting for instance i-3c7d7f5e to start...
> Instance started as ec2-23-20-25-159.compute-1.amazonaws.com
> Waiting for ssh to come up
> [<email address hidden>:22] run: sudo add-apt-repository ppa:canonical-sysadmins/puppet
>
> Fatal error: Timed out trying to connect to ec2-23-20-25-159.compute-1.amazonaws.com
>
> Aborting.

Hi,

This should now work with the latest changes pushed, as it now creates a
security group for the instance that gives the world access to 22,80,443
(I didn't think that locking it down to specific IP addresses gained us
much at all.)

Thanks,

James

32. By James Westby on 2012-01-19

Delete the security group before the instance.

Jonathan Lange (jml) wrote :

On Thu, Jan 19, 2012 at 3:06 PM, James Westby <email address hidden> wrote:
> On Thu, 19 Jan 2012 10:59:59 -0000, Jonathan Lange <email address hidden> wrote:
>> I consistently get this error:
>>
>> $ fab deploy_to_ec2
>> Waiting for instance i-3c7d7f5e to start...
>> Instance started as ec2-23-20-25-159.compute-1.amazonaws.com
>> Waiting for ssh to come up
>> [<email address hidden>:22] run: sudo add-apt-repository ppa:canonical-sysadmins/puppet
>>
>> Fatal error: Timed out trying to connect to ec2-23-20-25-159.compute-1.amazonaws.com
>>
>> Aborting.
>
> Hi,
>
> This should now work with the latest changes pushed, as it now creates a
> security group for the instance that gives the world access to 22,80,443
> (I didn't think that locking it down to specific IP addresses gained us
> much at all.)
>

Works now. Thanks.

jml

Jonathan Lange (jml) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'fabtasks/__init__.py'
2--- fabtasks/__init__.py 2011-08-29 23:50:07 +0000
3+++ fabtasks/__init__.py 2012-01-19 18:09:25 +0000
4@@ -1,2 +1,3 @@
5 from .bootstrap import *
6+from .deploy import *
7 from .django import *
8
9=== added file 'fabtasks/deploy.py'
10--- fabtasks/deploy.py 1970-01-01 00:00:00 +0000
11+++ fabtasks/deploy.py 2012-01-19 18:09:25 +0000
12@@ -0,0 +1,284 @@
13+from datetime import datetime
14+import os
15+import subprocess
16+import time
17+
18+from fabric.api import env, run
19+from fabric.utils import abort, puts
20+from fabric.operations import open_shell as _open_shell
21+
22+from boto import ec2
23+
24+
25+RELEASE = 'lucid'
26+ARCH = 'amd64'
27+FS_TYPE = 'ebs'
28+INSTANCE_TYPE = 't1.micro'
29+USERNAME = 'ubuntu'
30+REGION_NAME = 'us-east-1'
31+
32+NAME_KEY = "name"
33+NAME_PREFIX = "fab/pkgme-service"
34+
35+ALL_HOSTS = "0.0.0.0/0"
36+
37+
38+def open_shell():
39+ """Open a shell on the remote host."""
40+ _open_shell()
41+
42+
43+def _get_aws_credentials():
44+ """Get the AWS credentials for the user from ~/.ec2/aws_id.
45+
46+ :returns: a tuple of access key id, secret access key
47+ """
48+ key_id = None
49+ secret_access_key = None
50+ with open(os.path.expanduser("~/.ec2/aws_id")) as f:
51+ for i, line in enumerate(f.readlines()):
52+ if i == 0:
53+ key_id = line.strip()
54+ if i == 1:
55+ secret_access_key = line.strip()
56+ if key_id is None:
57+ raise AssertionError("Missing key id in ~/.ec2/aws_id")
58+ if secret_access_key is None:
59+ raise AssertionError("Missing secret access key in ~/.ec2/aws_id")
60+ return key_id, secret_access_key
61+
62+
63+def _get_ami_id(region_name):
64+ """Get the AMI to use for a particular region.
65+
66+ This consults the ubuntu-cloudimg-query tool to find out the best
67+ AMI to use for a particular region.
68+
69+ :returns: the ami id (as a string)
70+ """
71+ proc = subprocess.Popen(
72+ ['ubuntu-cloudimg-query', RELEASE, ARCH, FS_TYPE, region_name],
73+ stdout=subprocess.PIPE)
74+ stdout, _ = proc.communicate()
75+ if proc.returncode != 0:
76+ raise AssertionError("calling ubuntu-cloudimg-query failed")
77+ return stdout.strip()
78+
79+
80+def _get_security_group(conn, name):
81+ security_group = conn.create_security_group(name,
82+ "Access to the fab pkgme-service ec2 deployment %s" % name)
83+ security_group.authorize('tcp', 22, 22, ALL_HOSTS)
84+ security_group.authorize('tcp', 80, 80, ALL_HOSTS)
85+ security_group.authorize('tcp', 443, 443, ALL_HOSTS)
86+ return security_group
87+
88+
89+def _new_ec2_instance(keypair):
90+ """Starts a new ec2 instance, giving the specified keypair access.
91+
92+ This will use the AWS credentials from ~/.ec2/aws_id, and the
93+ AMI recommended by ubuntu-cloudimg-query.
94+
95+ :returns: the boto Instance object of the launched instance. It
96+ will be in a state where it can be accessed over ssh.
97+ """
98+ key_id, secret_access_key = _get_aws_credentials()
99+ ami_id = _get_ami_id(REGION_NAME)
100+ conn = ec2.connect_to_region(REGION_NAME, aws_access_key_id=key_id,
101+ aws_secret_access_key=secret_access_key)
102+ image = conn.get_image(ami_id)
103+ now = datetime.utcnow()
104+ name = "%s/%s" % (NAME_PREFIX, now.isoformat())
105+ security_group = _get_security_group(conn, name)
106+ reservation = image.run(instance_type=INSTANCE_TYPE, key_name=keypair,
107+ security_groups=[security_group.name])
108+ instance = reservation.instances[0]
109+ puts("Waiting for instance %s to start..." % instance.id)
110+ while True:
111+ time.sleep(10)
112+ instance.update()
113+ if instance.state != 'pending':
114+ break
115+ if instance.state != 'running':
116+ raise AssertionError("Instance failed to start")
117+ puts("Instance started as %s" % instance.dns_name)
118+ puts("Waiting for ssh to come up")
119+ # FIXME: use something better than a sleep to determine this.
120+ time.sleep(30)
121+ instance.add_tag(NAME_KEY, name)
122+ return instance
123+
124+
125+def deploy_to_ec2(branch="lp:pkgme-service", use_staging_deps=True, pkgme_branch="lp:pkgme", pkgme_binary_branch="lp:pkgme-binary", keypair='ec2-keypair'):
126+ """Deploy to a new ec2 instance.
127+
128+ This command will spin up an ec2 instance, and deploy to it.
129+
130+ To run this command you must first set up an ec2 account.
131+
132+ http://aws.amazon.com/ec2/
133+
134+ Once you have that create access keys for this to use. Go to
135+
136+ https://aws-portal.amazon.com/gp/aws/developer/account?ie=UTF8&action=access-key
137+
138+ and create an access key. Then create a file at ~/.ec2/aws_id and
139+ put the "Access Key ID" on the first line, and the "Secret Access Key" on
140+ the second line (with nothing else on either line.)
141+
142+ Next you will need an ec2 keypair. Go to the EC2 console at
143+
144+ https://console.aws.amazon.com/ec2/home?region=us-east-1
145+
146+ click on "Key Pairs" and then "Create Key Pair", name the keypair "ec2-keypair".
147+ Save the resulting file as
148+
149+ ~/.ec2/ec2-keypair.pem
150+
151+ Now you are ready to deploy, so run
152+
153+ fab deploy_to_ec2:keypair=ec2-keypair -i ~/.ec2/ec2-keypair.pem
154+
155+ and wait.
156+
157+ Note that you will be responsible for terminating the instance after
158+ use (see the destroy_ec2_instances command.)
159+
160+ You can also specify the following arguments:
161+
162+ * branch: the branch to deploy, defaults to lp:pkgme-service. To
163+ test some in-progress work push to lp:~you/pkgme-service/something
164+ and then specify that as the branch.
165+
166+ * pkgme_branch: the branch of pkgme to deploy, defaults to
167+ lp:pkgme.
168+
169+ * pkgme_binary_branch: the branch of pkgme-binary to deploy,
170+ defaults to lp:pkgme-binary.
171+
172+ * use_staging_deps: whether to use the staging PPA for
173+ dependencies, as well as the production one, defaults to True.
174+
175+ Arguments are all specified by attaching them to the command name, e.g.
176+
177+ fab deploy_to_ec2:keypair=ec2-keypair,branch=lp:~me/pkgme-service/something
178+ """
179+ instance = _new_ec2_instance(keypair)
180+ _set_instance_as_host(instance)
181+ deploy_to_existing(branch=branch, use_staging_deps=use_staging_deps, pkgme_branch=pkgme_branch, pkgme_binary_branch=pkgme_binary_branch)
182+
183+
184+def _set_instance_as_host(instance):
185+ """Set the host to be acted on by fab to this instance."""
186+ env.host_string = "%s@%s:22" % (USERNAME, instance.dns_name)
187+
188+
189+def deploy_to_existing(branch="lp:pkgme-service", use_staging_deps=True, pkgme_branch="lp:pkgme", pkgme_binary_branch="lp:pkgme-binary"):
190+ """Deploy to an existing instance.
191+
192+ This command will deploy to an existing instance. Don't use it on an
193+ instance you care about as it may overwrite anything.
194+
195+ To specify the host to deploy to use the -H option of fab:
196+
197+ fab -H ubuntu@<host> deploy_to_existing
198+
199+ You can also specify the following arguments:
200+
201+ * branch: the branch to deploy, defaults to lp:pkgme-service. To
202+ test some in-progress work push to lp:~you/pkgme-service/something
203+ and then specify that as the branch.
204+
205+ * pkgme_branch: the branch of pkgme to deploy, defaults to
206+ lp:pkgme.
207+
208+ * pkgme_binary_branch: the branch of pkgme-binary to deploy,
209+ defaults to lp:pkgme-binary.
210+
211+ * use_staging_deps: whether to use the staging PPA for
212+ dependencies, as well as the production one, defaults to True.
213+
214+ Arguments are all specified by attaching them to the command name, e.g.
215+
216+ fab deploy_to_ec2:keypair=ec2-keypair,branch=lp:~me/pkgme-service/something
217+ """
218+ # Get the puppet used by IS
219+ run('sudo add-apt-repository ppa:canonical-sysadmins/puppet')
220+ # Add our dependency PPA
221+ run('sudo add-apt-repository ppa:canonical-ca-hackers/production')
222+ if use_staging_deps:
223+ # Add our dependency staging PPA
224+ run('sudo add-apt-repository ppa:canonical-ca-hackers/staging')
225+ run('sudo apt-get update -q')
226+ # Upgrade the base system in case we are shipping any updates in
227+ # our PPAs
228+ run('sudo apt-get dist-upgrade -q -y --force-yes')
229+ # Avoid a debconf note when installing rabbitmq-server on lucid
230+ run('echo "rabbitmq-server rabbitmq-server/upgrade_previous note" | sudo debconf-set-selections')
231+ # Install the dependencies needed to get puppet going
232+ # TODO: move the rest of the dependencies to puppet
233+ run('sudo apt-get install -q -y --force-yes pkgme-service-dependencies bzr apache2 libapache2-mod-wsgi rabbitmq-server postgresql-8.4 puppet')
234+ # Grab the branches
235+ # TODO: investigate re-using IS' config-manager config
236+ run('bzr branch -q %s pkgme-service' % branch)
237+ run('bzr branch -q %s pkgme-service/sourcecode/pkgme' % pkgme_branch)
238+ run('bzr branch -q %s pkgme-service/sourcecode/pkgme-binary' % pkgme_binary_branch)
239+ run('cd pkgme-service/sourcecode/pkgme && python setup.py build')
240+ run('cd pkgme-service/sourcecode/pkgme-binary && python setup.py build')
241+ # Grab canonical-memento and use it?
242+ # Run puppet to set everything else up.
243+ run('./pkgme-service/dev_config/apply')
244+
245+
246+def _connect_to_ec2():
247+ """Get a connection to ec2, using the credentials on disk."""
248+ key_id, secret_access_key = _get_aws_credentials()
249+ conn = ec2.connect_to_region(REGION_NAME, aws_access_key_id=key_id,
250+ aws_secret_access_key=secret_access_key)
251+ return conn
252+
253+
254+def _get_started_ec2_instances(conn):
255+ """Get the ec2 instances started from here."""
256+ for reservation in conn.get_all_instances():
257+ for instance in reservation.instances:
258+ if instance.state == 'running':
259+ tags = getattr(instance, "tags", {})
260+ name = tags.get(NAME_KEY, None)
261+ if name and name.startswith(NAME_PREFIX):
262+ yield instance
263+
264+
265+def destroy_ec2_instances():
266+ """Destroy any ec2 instances created by this deployment."""
267+ conn = _connect_to_ec2()
268+ for instance in _get_started_ec2_instances(conn):
269+ puts("Stopping %s" % instance.id)
270+ conn.delete_security_group(name=instance.tags[NAME_KEY])
271+ instance.terminate()
272+
273+
274+def last_ec2_launched():
275+ """Cause other commands to act on the last ec2 instance launched from here.
276+
277+ Use this in a list of commands to have the rest of the commands act on the
278+ last ec2 instance launched with deploy_to_ec2, e.g.
279+
280+ fab -i ~/.ec2/ec2-keypair.pem last_ec2_launched -- ls
281+
282+ This will then run ls on that instance.
283+ """
284+ def get_date(instance):
285+ return instance.tags[NAME_KEY][len(NAME_PREFIX):]
286+ last_instance = None
287+ conn = _connect_to_ec2()
288+ for instance in _get_started_ec2_instances(conn):
289+ if last_instance is None:
290+ last_instance = instance
291+ else:
292+ if get_date(instance) > get_date(last_instance):
293+ last_instance = instance
294+ if last_instance is None:
295+ abort("No instances found.")
296+ _set_instance_as_host(last_instance)

Subscribers

People subscribed via source and target branches