Merge lp:~james-w/pkgme-service/fab-deploy-tasks into lp:pkgme-service
- fab-deploy-tasks
- Merge into trunk
Status: | Merged |
---|---|
Approved by: | Jonathan Lange |
Approved revision: | 32 |
Merged at revision: | 26 |
Proposed branch: | lp:~james-w/pkgme-service/fab-deploy-tasks |
Merge into: | lp:pkgme-service |
Prerequisite: | lp:~james-w/pkgme-service/puppet |
Diff against target: |
296 lines (+285/-0) 2 files modified
fabtasks/__init__.py (+1/-0) fabtasks/deploy.py (+284/-0) |
To merge this branch: | bzr merge lp:~james-w/pkgme-service/fab-deploy-tasks |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Jonathan Lange (community) | Approve | ||
Review via email: mp+89165@code.launchpad.net |
This proposal supersedes a proposal from 2012-01-18.
Commit message
Description of the change
Hi,
This adds some fab tasks for helping with spinning up dev instances.
There's no tests unfortunately.
There's a command to deploy to an existing instance, and one to spin
up an ec2 instance and deploy to it (like Launchpad's "ec2 demo".)
There's also a couple of commands to help with managing the ec2 instances.
The pre-requisites are rather convoluted, but that seems inevitable with ec2
unfortuanately, and they are documented as best as I know how at this time.
Thanks,
James
Jonathan Lange (jml) wrote : | # |
Jonathan Lange (jml) : | # |
Jonathan Lange (jml) wrote : | # |
I consistently get this error:
$ fab deploy_to_ec2
Waiting for instance i-3c7d7f5e to start...
Instance started as ec2-23-
Waiting for ssh to come up
[<email address hidden>:22] run: sudo add-apt-repository ppa:canonical-
Fatal error: Timed out trying to connect to ec2-23-
Aborting.
Jonathan Lange (jml) : | # |
James Westby (james-w) wrote : | # |
On Thu, 19 Jan 2012 10:13:31 -0000, Jonathan Lange <email address hidden> wrote:
> It's a bit disappointing that there's not a Python library that provides some of the stuff that you hand-roll in the first few functions.
Indeed, I don't think that boto has a location it looks, or knows about
the ubuntu-specific script I use. I don't think it's worth turning it in
to a library at this point though.
> Any ideas on how we can prevent this from sprawling into a project the size of Launchpad's ec2test?
I wouldn't mind something the size of Launchpad's ec2test if it didn't
live in this project, or at least the parts that aren't specific to
pkgme-service.
Probably we'll abandon it all in favour of juju in a few months though.
> Also, when can we target this at a local virtual machine / chroot / LXC thing?
If you have one up the deploy_to_existing should work ok (it could do
with dealing with not having a fresh install a little better if you want
to re-use.)
We could add lxc without too much work I expect, but I haven't played
with lxc directly yet to know what to do.
Thanks,
James
- 29. By James Westby
-
Merge puppet improvements.
- 30. By James Westby
-
Merge puppet fixes.
- 31. By James Westby
-
Set the security group of the created instance so it is accessible.
James Westby (james-w) wrote : | # |
On Thu, 19 Jan 2012 10:59:59 -0000, Jonathan Lange <email address hidden> wrote:
> I consistently get this error:
>
> $ fab deploy_to_ec2
> Waiting for instance i-3c7d7f5e to start...
> Instance started as ec2-23-
> Waiting for ssh to come up
> [<email address hidden>:22] run: sudo add-apt-repository ppa:canonical-
>
> Fatal error: Timed out trying to connect to ec2-23-
>
> Aborting.
Hi,
This should now work with the latest changes pushed, as it now creates a
security group for the instance that gives the world access to 22,80,443
(I didn't think that locking it down to specific IP addresses gained us
much at all.)
Thanks,
James
- 32. By James Westby
-
Delete the security group before the instance.
Jonathan Lange (jml) wrote : | # |
On Thu, Jan 19, 2012 at 3:06 PM, James Westby <email address hidden> wrote:
> On Thu, 19 Jan 2012 10:59:59 -0000, Jonathan Lange <email address hidden> wrote:
>> I consistently get this error:
>>
>> $ fab deploy_to_ec2
>> Waiting for instance i-3c7d7f5e to start...
>> Instance started as ec2-23-
>> Waiting for ssh to come up
>> [<email address hidden>:22] run: sudo add-apt-repository ppa:canonical-
>>
>> Fatal error: Timed out trying to connect to ec2-23-
>>
>> Aborting.
>
> Hi,
>
> This should now work with the latest changes pushed, as it now creates a
> security group for the instance that gives the world access to 22,80,443
> (I didn't think that locking it down to specific IP addresses gained us
> much at all.)
>
Works now. Thanks.
jml
Jonathan Lange (jml) : | # |
Preview Diff
1 | === modified file 'fabtasks/__init__.py' |
2 | --- fabtasks/__init__.py 2011-08-29 23:50:07 +0000 |
3 | +++ fabtasks/__init__.py 2012-01-19 18:09:25 +0000 |
4 | @@ -1,2 +1,3 @@ |
5 | from .bootstrap import * |
6 | +from .deploy import * |
7 | from .django import * |
8 | |
9 | === added file 'fabtasks/deploy.py' |
10 | --- fabtasks/deploy.py 1970-01-01 00:00:00 +0000 |
11 | +++ fabtasks/deploy.py 2012-01-19 18:09:25 +0000 |
12 | @@ -0,0 +1,284 @@ |
13 | +from datetime import datetime |
14 | +import os |
15 | +import subprocess |
16 | +import time |
17 | + |
18 | +from fabric.api import env, run |
19 | +from fabric.utils import abort, puts |
20 | +from fabric.operations import open_shell as _open_shell |
21 | + |
22 | +from boto import ec2 |
23 | + |
24 | + |
25 | +RELEASE = 'lucid' |
26 | +ARCH = 'amd64' |
27 | +FS_TYPE = 'ebs' |
28 | +INSTANCE_TYPE = 't1.micro' |
29 | +USERNAME = 'ubuntu' |
30 | +REGION_NAME = 'us-east-1' |
31 | + |
32 | +NAME_KEY = "name" |
33 | +NAME_PREFIX = "fab/pkgme-service" |
34 | + |
35 | +ALL_HOSTS = "0.0.0.0/0" |
36 | + |
37 | + |
38 | +def open_shell(): |
39 | + """Open a shell on the remote host.""" |
40 | + _open_shell() |
41 | + |
42 | + |
43 | +def _get_aws_credentials(): |
44 | + """Get the AWS credentials for the user from ~/.ec2/aws_id. |
45 | + |
46 | + :returns: a tuple of access key id, secret access key |
47 | + """ |
48 | + key_id = None |
49 | + secret_access_key = None |
50 | + with open(os.path.expanduser("~/.ec2/aws_id")) as f: |
51 | + for i, line in enumerate(f.readlines()): |
52 | + if i == 0: |
53 | + key_id = line.strip() |
54 | + if i == 1: |
55 | + secret_access_key = line.strip() |
56 | + if key_id is None: |
57 | + raise AssertionError("Missing key id in ~/.ec2/aws_id") |
58 | + if secret_access_key is None: |
59 | + raise AssertionError("Missing secret access key in ~/.ec2/aws_id") |
60 | + return key_id, secret_access_key |
61 | + |
62 | + |
63 | +def _get_ami_id(region_name): |
64 | + """Get the AMI to use for a particular region. |
65 | + |
66 | + This consults the ubuntu-cloudimg-query tool to find out the best |
67 | + AMI to use for a particular region. |
68 | + |
69 | + :returns: the ami id (as a string) |
70 | + """ |
71 | + proc = subprocess.Popen( |
72 | + ['ubuntu-cloudimg-query', RELEASE, ARCH, FS_TYPE, region_name], |
73 | + stdout=subprocess.PIPE) |
74 | + stdout, _ = proc.communicate() |
75 | + if proc.returncode != 0: |
76 | + raise AssertionError("calling ubuntu-cloudimg-query failed") |
77 | + return stdout.strip() |
78 | + |
79 | + |
80 | +def _get_security_group(conn, name): |
81 | + security_group = conn.create_security_group(name, |
82 | + "Access to the fab pkgme-service ec2 deployment %s" % name) |
83 | + security_group.authorize('tcp', 22, 22, ALL_HOSTS) |
84 | + security_group.authorize('tcp', 80, 80, ALL_HOSTS) |
85 | + security_group.authorize('tcp', 443, 443, ALL_HOSTS) |
86 | + return security_group |
87 | + |
88 | + |
89 | +def _new_ec2_instance(keypair): |
90 | + """Starts a new ec2 instance, giving the specified keypair access. |
91 | + |
92 | + This will use the AWS credentials from ~/.ec2/aws_id, and the |
93 | + AMI recommended by ubuntu-cloudimg-query. |
94 | + |
95 | + :returns: the boto Instance object of the launched instance. It |
96 | + will be in a state where it can be accessed over ssh. |
97 | + """ |
98 | + key_id, secret_access_key = _get_aws_credentials() |
99 | + ami_id = _get_ami_id(REGION_NAME) |
100 | + conn = ec2.connect_to_region(REGION_NAME, aws_access_key_id=key_id, |
101 | + aws_secret_access_key=secret_access_key) |
102 | + image = conn.get_image(ami_id) |
103 | + now = datetime.utcnow() |
104 | + name = "%s/%s" % (NAME_PREFIX, now.isoformat()) |
105 | + security_group = _get_security_group(conn, name) |
106 | + reservation = image.run(instance_type=INSTANCE_TYPE, key_name=keypair, |
107 | + security_groups=[security_group.name]) |
108 | + instance = reservation.instances[0] |
109 | + puts("Waiting for instance %s to start..." % instance.id) |
110 | + while True: |
111 | + time.sleep(10) |
112 | + instance.update() |
113 | + if instance.state != 'pending': |
114 | + break |
115 | + if instance.state != 'running': |
116 | + raise AssertionError("Instance failed to start") |
117 | + puts("Instance started as %s" % instance.dns_name) |
118 | + puts("Waiting for ssh to come up") |
119 | + # FIXME: use something better than a sleep to determine this. |
120 | + time.sleep(30) |
121 | + instance.add_tag(NAME_KEY, name) |
122 | + return instance |
123 | + |
124 | + |
125 | +def deploy_to_ec2(branch="lp:pkgme-service", use_staging_deps=True, pkgme_branch="lp:pkgme", pkgme_binary_branch="lp:pkgme-binary", keypair='ec2-keypair'): |
126 | + """Deploy to a new ec2 instance. |
127 | + |
128 | + This command will spin up an ec2 instance, and deploy to it. |
129 | + |
130 | + To run this command you must first set up an ec2 account. |
131 | + |
132 | + http://aws.amazon.com/ec2/ |
133 | + |
134 | + Once you have that create access keys for this to use. Go to |
135 | + |
136 | + https://aws-portal.amazon.com/gp/aws/developer/account?ie=UTF8&action=access-key |
137 | + |
138 | + and create an access key. Then create a file at ~/.ec2/aws_id and |
139 | + put the "Access Key ID" on the first line, and the "Secret Access Key" on |
140 | + the second line (with nothing else on either line.) |
141 | + |
142 | + Next you will need an ec2 keypair. Go to the EC2 console at |
143 | + |
144 | + https://console.aws.amazon.com/ec2/home?region=us-east-1 |
145 | + |
146 | + click on "Key Pairs" and then "Create Key Pair", name the keypair "ec2-keypair". |
147 | + Save the resulting file as |
148 | + |
149 | + ~/.ec2/ec2-keypair.pem |
150 | + |
151 | + Now you are ready to deploy, so run |
152 | + |
153 | + fab deploy_to_ec2:keypair=ec2-keypair -i ~/.ec2/ec2-keypair.pem |
154 | + |
155 | + and wait. |
156 | + |
157 | + Note that you will be responsible for terminating the instance after |
158 | + use (see the destroy_ec2_instances command.) |
159 | + |
160 | + You can also specify the following arguments: |
161 | + |
162 | + * branch: the branch to deploy, defaults to lp:pkgme-service. To |
163 | + test some in-progress work push to lp:~you/pkgme-service/something |
164 | + and then specify that as the branch. |
165 | + |
166 | + * pkgme_branch: the branch of pkgme to deploy, defaults to |
167 | + lp:pkgme. |
168 | + |
169 | + * pkgme_binary_branch: the branch of pkgme-binary to deploy, |
170 | + defaults to lp:pkgme-binary. |
171 | + |
172 | + * use_staging_deps: whether to use the staging PPA for |
173 | + dependencies, as well as the production one, defaults to True. |
174 | + |
175 | + Arguments are all specified by attaching them to the command name, e.g. |
176 | + |
177 | + fab deploy_to_ec2:keypair=ec2-keypair,branch=lp:~me/pkgme-service/something |
178 | + """ |
179 | + instance = _new_ec2_instance(keypair) |
180 | + _set_instance_as_host(instance) |
181 | + deploy_to_existing(branch=branch, use_staging_deps=use_staging_deps, pkgme_branch=pkgme_branch, pkgme_binary_branch=pkgme_binary_branch) |
182 | + |
183 | + |
184 | +def _set_instance_as_host(instance): |
185 | + """Set the host to be acted on by fab to this instance.""" |
186 | + env.host_string = "%s@%s:22" % (USERNAME, instance.dns_name) |
187 | + |
188 | + |
189 | +def deploy_to_existing(branch="lp:pkgme-service", use_staging_deps=True, pkgme_branch="lp:pkgme", pkgme_binary_branch="lp:pkgme-binary"): |
190 | + """Deploy to an existing instance. |
191 | + |
192 | + This command will deploy to an existing instance. Don't use it on an |
193 | + instance you care about as it may overwrite anything. |
194 | + |
195 | + To specify the host to deploy to use the -H option of fab: |
196 | + |
197 | + fab -H ubuntu@<host> deploy_to_existing |
198 | + |
199 | + You can also specify the following arguments: |
200 | + |
201 | + * branch: the branch to deploy, defaults to lp:pkgme-service. To |
202 | + test some in-progress work push to lp:~you/pkgme-service/something |
203 | + and then specify that as the branch. |
204 | + |
205 | + * pkgme_branch: the branch of pkgme to deploy, defaults to |
206 | + lp:pkgme. |
207 | + |
208 | + * pkgme_binary_branch: the branch of pkgme-binary to deploy, |
209 | + defaults to lp:pkgme-binary. |
210 | + |
211 | + * use_staging_deps: whether to use the staging PPA for |
212 | + dependencies, as well as the production one, defaults to True. |
213 | + |
214 | + Arguments are all specified by attaching them to the command name, e.g. |
215 | + |
216 | + fab deploy_to_ec2:keypair=ec2-keypair,branch=lp:~me/pkgme-service/something |
217 | + """ |
218 | + # Get the puppet used by IS |
219 | + run('sudo add-apt-repository ppa:canonical-sysadmins/puppet') |
220 | + # Add our dependency PPA |
221 | + run('sudo add-apt-repository ppa:canonical-ca-hackers/production') |
222 | + if use_staging_deps: |
223 | + # Add our dependency staging PPA |
224 | + run('sudo add-apt-repository ppa:canonical-ca-hackers/staging') |
225 | + run('sudo apt-get update -q') |
226 | + # Upgrade the base system in case we are shipping any updates in |
227 | + # our PPAs |
228 | + run('sudo apt-get dist-upgrade -q -y --force-yes') |
229 | + # Avoid a debconf note when installing rabbitmq-server on lucid |
230 | + run('echo "rabbitmq-server rabbitmq-server/upgrade_previous note" | sudo debconf-set-selections') |
231 | + # Install the dependencies needed to get puppet going |
232 | + # TODO: move the rest of the dependencies to puppet |
233 | + run('sudo apt-get install -q -y --force-yes pkgme-service-dependencies bzr apache2 libapache2-mod-wsgi rabbitmq-server postgresql-8.4 puppet') |
234 | + # Grab the branches |
235 | + # TODO: investigate re-using IS' config-manager config |
236 | + run('bzr branch -q %s pkgme-service' % branch) |
237 | + run('bzr branch -q %s pkgme-service/sourcecode/pkgme' % pkgme_branch) |
238 | + run('bzr branch -q %s pkgme-service/sourcecode/pkgme-binary' % pkgme_binary_branch) |
239 | + run('cd pkgme-service/sourcecode/pkgme && python setup.py build') |
240 | + run('cd pkgme-service/sourcecode/pkgme-binary && python setup.py build') |
241 | + # Grab canonical-memento and use it? |
242 | + # Run puppet to set everything else up. |
243 | + run('./pkgme-service/dev_config/apply') |
244 | + |
245 | + |
246 | +def _connect_to_ec2(): |
247 | + """Get a connection to ec2, using the credentials on disk.""" |
248 | + key_id, secret_access_key = _get_aws_credentials() |
249 | + conn = ec2.connect_to_region(REGION_NAME, aws_access_key_id=key_id, |
250 | + aws_secret_access_key=secret_access_key) |
251 | + return conn |
252 | + |
253 | + |
254 | +def _get_started_ec2_instances(conn): |
255 | + """Get the ec2 instances started from here.""" |
256 | + for reservation in conn.get_all_instances(): |
257 | + for instance in reservation.instances: |
258 | + if instance.state == 'running': |
259 | + tags = getattr(instance, "tags", {}) |
260 | + name = tags.get(NAME_KEY, None) |
261 | + if name and name.startswith(NAME_PREFIX): |
262 | + yield instance |
263 | + |
264 | + |
265 | +def destroy_ec2_instances(): |
266 | + """Destroy any ec2 instances created by this deployment.""" |
267 | + conn = _connect_to_ec2() |
268 | + for instance in _get_started_ec2_instances(conn): |
269 | + puts("Stopping %s" % instance.id) |
270 | + conn.delete_security_group(name=instance.tags[NAME_KEY]) |
271 | + instance.terminate() |
272 | + |
273 | + |
274 | +def last_ec2_launched(): |
275 | + """Cause other commands to act on the last ec2 instance launched from here. |
276 | + |
277 | + Use this in a list of commands to have the rest of the commands act on the |
278 | + last ec2 instance launched with deploy_to_ec2, e.g. |
279 | + |
280 | + fab -i ~/.ec2/ec2-keypair.pem last_ec2_launched -- ls |
281 | + |
282 | + This will then run ls on that instance. |
283 | + """ |
284 | + def get_date(instance): |
285 | + return instance.tags[NAME_KEY][len(NAME_PREFIX):] |
286 | + last_instance = None |
287 | + conn = _connect_to_ec2() |
288 | + for instance in _get_started_ec2_instances(conn): |
289 | + if last_instance is None: |
290 | + last_instance = instance |
291 | + else: |
292 | + if get_date(instance) > get_date(last_instance): |
293 | + last_instance = instance |
294 | + if last_instance is None: |
295 | + abort("No instances found.") |
296 | + _set_instance_as_host(last_instance) |
This all looks OK to me.
It's a bit disappointing that there's not a Python library that provides some of the stuff that you hand-roll in the first few functions.
Any ideas on how we can prevent this from sprawling into a project the size of Launchpad's ec2test?
Also, when can we target this at a local virtual machine / chroot / LXC thing?