Merge lp:~fwereade/pyjuju/cloud-init-class-used into lp:pyjuju
- cloud-init-class-used
- Merge into trunk
Status: | Merged | ||||
---|---|---|---|---|---|
Approved by: | Gustavo Niemeyer | ||||
Approved revision: | 341 | ||||
Merged at revision: | 342 | ||||
Proposed branch: | lp:~fwereade/pyjuju/cloud-init-class-used | ||||
Merge into: | lp:pyjuju | ||||
Prerequisite: | lp:~fwereade/pyjuju/cloud-init-class-prepare | ||||
Diff against target: |
2107 lines (+514/-880) 28 files modified
ensemble/providers/common/base.py (+3/-4) ensemble/providers/common/cloudinit.py (+12/-13) ensemble/providers/common/launch.py (+46/-170) ensemble/providers/common/tests/data/cloud_init_bootstrap (+1/-2) ensemble/providers/common/tests/data/cloud_init_bootstrap_zookeepers (+1/-2) ensemble/providers/common/tests/data/cloud_init_branch (+1/-2) ensemble/providers/common/tests/data/cloud_init_distro (+1/-2) ensemble/providers/common/tests/data/cloud_init_normal (+1/-2) ensemble/providers/common/tests/test_bootstrap.py (+102/-0) ensemble/providers/common/tests/test_cloudinit.py (+2/-2) ensemble/providers/common/tests/test_launch.py (+61/-224) ensemble/providers/common/tests/test_utils.py (+1/-2) ensemble/providers/common/utils.py (+3/-3) ensemble/providers/ec2/__init__.py (+10/-5) ensemble/providers/ec2/launch.py (+32/-53) ensemble/providers/ec2/tests/common.py (+2/-4) ensemble/providers/ec2/tests/data/bootstrap_cloud_init (+18/-0) ensemble/providers/ec2/tests/data/launch_cloud_init (+15/-0) ensemble/providers/ec2/tests/test_bootstrap.py (+16/-51) ensemble/providers/ec2/tests/test_launch.py (+79/-114) ensemble/providers/ec2/tests/test_utils.py (+40/-106) ensemble/providers/ec2/utils.py (+17/-69) ensemble/providers/orchestra/__init__.py (+9/-6) ensemble/providers/orchestra/launch.py (+23/-34) ensemble/providers/orchestra/tests/common.py (+2/-1) ensemble/providers/orchestra/tests/data/bootstrap_late_command (+5/-5) ensemble/providers/orchestra/tests/data/launch_late_command (+5/-4) ensemble/providers/orchestra/tests/test_launch.py (+6/-0) |
||||
To merge this branch: | bzr merge lp:~fwereade/pyjuju/cloud-init-class-used | ||||
Related bugs: |
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Kapil Thangavelu (community) | Approve | ||
Gustavo Niemeyer | Approve | ||
Review via email: mp+73634@code.launchpad.net |
Commit message
Description of the change
Another attempt at a cleaner cloud-init/
- 339. By William Reade
-
merge parent
- 340. By William Reade
-
renaming in CloudInit
- 341. By William Reade
-
get_current_ami now uses same keys as machine_data constraints dict
Gustavo Niemeyer (niemeyer) wrote : | # |
Thanks for the changes William.
[7]
+# XXX ideally should come from latest available or client release name.
FWIW, it should come from the formula Ubuntu series instead.
Gustavo Niemeyer (niemeyer) wrote : | # |
Please go ahead and merge this William. It's been in review for 2 weeks already.
Kapil Thangavelu (hazmat) wrote : | # |
looks great, bye-bye anonymous blobs. some comments.
[0]
- @param machine_data: a dictionary of data to pass along to the newly
- launched machine.
+ `machine_data`: a dictionary describing the newly launched machine.
'describing the machine to be launched.'
[1]
+ def set_provider_
+ self._provider_type = type_
type isn't a keyword
[2]
+ def set_machine_
+ self._machine_id = id
+
+ def set_instance_
+ self._instance_id = expr
+ if self._zookeeper:
+ require(
+ require(
what's the distinction between the two variables. instance id comes from the provider
implementation as a consequence of actually running the machine, its an output value, not
an input value to cloud init afaics.
its apparent a little bit latter into the diff, this is actually a shell command for retrieval
of instance id.. the naming should be reflective of that. perhaps set_instance_
what's the case for the instance id not being set? is that just for testing ease, if so it should probably be commented as such.
William Reade (fwereade) wrote : | # |
> looks great, bye-bye anonymous blobs. some comments.
>
> [0]
>
> - @param machine_data: a dictionary of data to pass along to the newly
> - launched machine.
> + `machine_data`: a dictionary describing the newly launched machine.
>
> 'describing the machine to be launched.'
Thanks
> [1]
>
> + def set_provider_
> + self._provider_type = type_
>
> type isn't a keyword
But it is a builtin, and I prefer not to shadow them unless I absopositively have to.
> [2]
> + def set_machine_
> + self._machine_id = id
> +
> + def set_instance_
> + self._instance_id = expr
>
> + if self._zookeeper:
> + require(
> + require(
>
> what's the distinction between the two variables. instance id comes from the
> provider
> implementation as a consequence of actually running the machine, its an output
> value, not
> an input value to cloud init afaics.
>
> its apparent a little bit latter into the diff, this is actually a shell
> command for retrieval
> of instance id.. the naming should be reflective of that. perhaps
> set_instance_
Good idea
> what's the case for the instance id not being set? is that just for testing
> ease, if so it should probably be commented as such.
It's just that it's not actually used, except when we're running a bootstrap node. This could well change in the future.
- 342. By William Reade
-
merge parent
- 343. By William Reade
-
fixes per hazmat's review
- 344. By William Reade
-
cloud-inits now contain lists of ssh keys, not list-of-lists of ssh keys
Preview Diff
1 | === modified file 'ensemble/providers/common/base.py' |
2 | --- ensemble/providers/common/base.py 2011-08-29 02:23:11 +0000 |
3 | +++ ensemble/providers/common/base.py 2011-09-13 10:56:30 +0000 |
4 | @@ -65,11 +65,10 @@ |
5 | def start_machine(self, machine_data, master=False): |
6 | """Start a machine in the provider. |
7 | |
8 | - @param machine_data: a dictionary of data to pass along to the newly |
9 | - launched machine. |
10 | + `machine_data`: a dictionary describing the machine to be launched. |
11 | |
12 | - @param master: if True, machine will initialize the ensemble admin |
13 | - and run a provisioning agent. |
14 | + `master`: if True, machine will initialize the ensemble admin and run |
15 | + a provisioning agent, in addition to running a machine agent |
16 | """ |
17 | raise NotImplementedError() |
18 | |
19 | |
20 | === modified file 'ensemble/providers/common/cloudinit.py' |
21 | --- ensemble/providers/common/cloudinit.py 2011-09-13 10:56:30 +0000 |
22 | +++ ensemble/providers/common/cloudinit.py 2011-09-13 10:56:30 +0000 |
23 | @@ -60,7 +60,7 @@ |
24 | self._ssh_keys = [] |
25 | self._provision = False |
26 | self._zookeeper = False |
27 | - self._zookeeper_names = [] |
28 | + self._zookeeper_hosts = [] |
29 | self._zookeeper_secret = None |
30 | |
31 | def add_ssh_key(self, key): |
32 | @@ -83,14 +83,14 @@ |
33 | def set_machine_id(self, id): |
34 | self._machine_id = id |
35 | |
36 | - def set_instance_id(self, expr): |
37 | + def set_instance_id_accessor(self, expr): |
38 | self._instance_id = expr |
39 | |
40 | def set_provider_type(self, type_): |
41 | self._provider_type = type_ |
42 | |
43 | def set_zookeeper_machines(self, machines): |
44 | - self._zookeeper_names = [m.private_dns_name for m in machines] |
45 | + self._zookeeper_hosts = [m.private_dns_name for m in machines] |
46 | |
47 | def set_zookeeper_secret(self, secret): |
48 | self._zookeeper_secret = secret |
49 | @@ -115,20 +115,19 @@ |
50 | require("_machine_id", "set_machine_id") |
51 | require("_provider_type", "set_provider_type") |
52 | if self._zookeeper: |
53 | - require("_instance_id", "set_instance_id") |
54 | + require("_instance_id", "set_instance_id_accessor") |
55 | require("_zookeeper_secret", "set_zookeeper_secret") |
56 | else: |
57 | - require("_zookeeper_names", "set_zookeeper_machines") |
58 | + require("_zookeeper_hosts", "set_zookeeper_machines") |
59 | if missing: |
60 | raise CloudInitError("Incomplete cloud-init: you need to call %s" |
61 | % ", ".join(missing)) |
62 | |
63 | - @property |
64 | - def _zookeeper_hosts(self): |
65 | - all_names = self._zookeeper_names[:] |
66 | + def _join_zookeeper_hosts(self): |
67 | + all_hosts = self._zookeeper_hosts[:] |
68 | if self._zookeeper: |
69 | - all_names.append("localhost") |
70 | - return ",".join(["%s:2181" % name for name in all_names]) |
71 | + all_hosts.append("localhost") |
72 | + return ",".join(["%s:2181" % host for host in all_hosts]) |
73 | |
74 | def _collect_packages(self): |
75 | packages = [ |
76 | @@ -150,13 +149,13 @@ |
77 | scripts.extend(_zookeeper_scripts( |
78 | self._instance_id, self._zookeeper_secret)) |
79 | scripts.extend(_machine_scripts( |
80 | - self._machine_id, self._zookeeper_hosts)) |
81 | + self._machine_id, self._join_zookeeper_hosts())) |
82 | if self._provision: |
83 | - scripts.extend(_provision_scripts(self._zookeeper_hosts)) |
84 | + scripts.extend(_provision_scripts(self._join_zookeeper_hosts())) |
85 | return scripts |
86 | |
87 | def _collect_machine_data(self): |
88 | return { |
89 | "machine-id": self._machine_id, |
90 | "ensemble-provider-type": self._provider_type, |
91 | - "ensemble-zookeeper-hosts": self._zookeeper_hosts} |
92 | + "ensemble-zookeeper-hosts": self._join_zookeeper_hosts()} |
93 | |
94 | === modified file 'ensemble/providers/common/launch.py' |
95 | --- ensemble/providers/common/launch.py 2011-08-17 16:01:43 +0000 |
96 | +++ ensemble/providers/common/launch.py 2011-09-13 10:56:30 +0000 |
97 | @@ -1,195 +1,71 @@ |
98 | -import copy |
99 | - |
100 | from twisted.internet.defer import inlineCallbacks, returnValue |
101 | |
102 | -from ensemble.errors import ProviderError |
103 | -from ensemble.state.auth import make_identity |
104 | - |
105 | +from .cloudinit import CloudInit |
106 | from .utils import get_user_authorized_keys |
107 | |
108 | -DEFAULT_REPOSITORIES = [ |
109 | - "ppa:ensemble/ppa"] |
110 | - |
111 | -DEFAULT_PACKAGES = [ |
112 | - "bzr", |
113 | - "byobu", |
114 | - "tmux", |
115 | - "python-setuptools", |
116 | - "python-twisted", |
117 | - "python-argparse", |
118 | - "python-txaws", |
119 | - "python-zookeeper"] |
120 | - |
121 | -BOOTSTRAP_PACKAGES = [ |
122 | - "bzr", |
123 | - "default-jre-headless", |
124 | - "zookeeper", |
125 | - "zookeeperd"] |
126 | - |
127 | - |
128 | -def _get_initialize_script(instance_id_command, admin_secret): |
129 | - admin_identity = make_identity("admin:%s" % admin_secret) |
130 | - template = ('ensemble-admin initialize ' |
131 | - '--instance-id=%s --admin-identity="%s"') |
132 | - return template % (instance_id_command, admin_identity) |
133 | - |
134 | - |
135 | -def _get_machine_agent_script(vars): |
136 | - template = ( |
137 | - "ENSEMBLE_MACHINE_ID=%(machine-id)s " |
138 | - "ENSEMBLE_ZOOKEEPER=%(ensemble-zookeeper-hosts)s " |
139 | - "python -m ensemble.agents.machine -n " |
140 | - "--logfile=/var/log/ensemble/machine-agent.log " |
141 | - "--pidfile=/var/run/ensemble/machine-agent.pid") |
142 | - return template % vars |
143 | - |
144 | - |
145 | -def _get_provision_agent_script(vars): |
146 | - template = ( |
147 | - "ENSEMBLE_ZOOKEEPER=%(ensemble-zookeeper-hosts)s " |
148 | - "python -m ensemble.agents.provision -n " |
149 | - "--logfile=/var/log/ensemble/provision-agent.log " |
150 | - "--pidfile=/var/run/ensemble/provision-agent.pid") |
151 | - return template % vars |
152 | - |
153 | |
154 | class LaunchMachine(object): |
155 | """Abstract class with generic instance-launching logic. |
156 | |
157 | - Constructing with master=True will cause the run method to |
158 | - construct a machine which is also running a zookeeper for the |
159 | - cluster, and a provisioning agent, as well as the usual machine |
160 | - agent. |
161 | - |
162 | To create your own subclass, you will certainly need to override |
163 | - C{start_machine} and C{get_instance_id_command}, and may find it |
164 | - convenient to override C{get_machine_variables} as well. |
165 | + `start_machine`, and will very probably want to implement it with the aid |
166 | + of a provider-specific subclass of `ProviderCloudInit`. |
167 | + |
168 | + `provider`: the necessary `MachineProvider` |
169 | + |
170 | + `master`: if True, the machine will run a zookeeper and a provisioning |
171 | + agent, in addition to the machine agent that every machine runs |
172 | """ |
173 | |
174 | - def __init__(self, provider, master=False): |
175 | + def __init__(self, provider, master=False, constraints=None): |
176 | self._provider = provider |
177 | self._master = master |
178 | + self._constraints = constraints or {} |
179 | |
180 | @inlineCallbacks |
181 | - def run(self, machine_data): |
182 | + def run(self, machine_id): |
183 | """Launch an instance node within the machine provider environment. |
184 | |
185 | - `machine_data`: a dictionary of data that is passed along to |
186 | - provided to the machine as serialized yaml. 'machine-id' is a |
187 | - required key, denoting the machine's zookeeper node for its |
188 | - machine agent. |
189 | + `machine_id`: the ensemble machine ID |
190 | """ |
191 | - if "machine-id" not in machine_data: |
192 | - raise ProviderError( |
193 | - "Machine state `machine-id` not provided in machine_data.") |
194 | - |
195 | - machine_variables = yield self.get_machine_variables(machine_data) |
196 | - provider_machines = yield self.start_machine( |
197 | - machine_variables, machine_data["machine-id"]) |
198 | - if self._master: |
199 | - yield self._on_master_launched(provider_machines) |
200 | - returnValue(provider_machines) |
201 | - |
202 | - def start_machine(self, machine_variables, machine_id): |
203 | + # XXX at some point, we'll want to start multiple zookeepers |
204 | + # that know about each other; for now, this'll do |
205 | + if self._master: |
206 | + zookeepers = [] |
207 | + else: |
208 | + zookeepers = yield self._provider.get_zookeeper_machines() |
209 | + |
210 | + machines = yield self.start_machine(machine_id, zookeepers) |
211 | + if self._master: |
212 | + yield self._on_new_zookeepers(machines) |
213 | + returnValue(machines) |
214 | + |
215 | + def start_machine(self, machine_id, zookeepers): |
216 | """Actually launch a machine for the appropriate provider. |
217 | |
218 | - `machine_variables`: non-provider-specific data, sufficient to |
219 | - define the machine's behaviour once it exists. |
220 | - |
221 | - `machine_id`: the external machine ID (also exists in the |
222 | - above bag of data) |
223 | - |
224 | - Returns a singe-entry list containing a ProviderMachine for the |
225 | + `machine_id`: the ensemble machine ID |
226 | + |
227 | + `zookeepers`: a list of `ProviderMachine`s already running zookeeper, |
228 | + which the machine will need to connect to. May be empty. |
229 | + |
230 | + Returns a singe-entry list containing a `ProviderMachine` for the |
231 | new instance |
232 | """ |
233 | raise NotImplementedError() |
234 | |
235 | - def get_instance_id_command(self): |
236 | - """Snippet to discover local instance-id |
237 | - |
238 | - @return: a snippet of shell script that evaluates to the |
239 | - local machine's instance_id. |
240 | - """ |
241 | - raise NotImplementedError() |
242 | - |
243 | - @inlineCallbacks |
244 | - def get_machine_variables(self, vars_in): |
245 | - """Get the variables used for launching a machine and its cloud-init""" |
246 | - vars_out = copy.deepcopy(vars_in) |
247 | - vars_out[ |
248 | - "ensemble-zookeeper-hosts"] = yield self._get_zookeeper_hosts() |
249 | - |
250 | - authorized_keys = get_user_authorized_keys(self._provider.config) |
251 | - vars_out["authorized_keys"] = authorized_keys |
252 | - vars_out["packages"] = self._get_packages() |
253 | - vars_out["repositories"] = list(DEFAULT_REPOSITORIES) |
254 | - vars_out["scripts"] = self._get_launch_scripts(vars_out) |
255 | - returnValue(vars_out) |
256 | - |
257 | - @inlineCallbacks |
258 | - def _get_zookeeper_hosts(self): |
259 | - if self._master: |
260 | - returnValue("localhost:2181") |
261 | - machines = yield self._provider.get_zookeeper_machines() |
262 | - hosts = [m.private_dns_name for m in machines] |
263 | - # format multiple hosts in string suitable for client |
264 | - returnValue(",".join(["%s:2181" % h for h in hosts])) |
265 | - |
266 | - def _get_packages(self): |
267 | - result = list(DEFAULT_PACKAGES) |
268 | - if self._master: |
269 | - result.extend(BOOTSTRAP_PACKAGES) |
270 | - return result |
271 | - |
272 | - def _get_launch_scripts(self, machine_data): |
273 | - """Get the scripts that will execute after the machine has launched.""" |
274 | - launch_scripts = self._get_install_scripts() |
275 | - |
276 | - # These should be in the deb |
277 | - launch_scripts.extend([ |
278 | - "sudo mkdir -p /var/lib/ensemble", |
279 | - "sudo mkdir -p /var/log/ensemble"]) |
280 | - |
281 | - if self._master: |
282 | - launch_scripts.append(self._get_initialize_script()) |
283 | - |
284 | - # Every machine has its own agent. |
285 | - machine_agent_script_template = ( |
286 | - "ENSEMBLE_MACHINE_ID=%(machine-id)s " |
287 | - "ENSEMBLE_ZOOKEEPER=%(ensemble-zookeeper-hosts)s " |
288 | - "python -m ensemble.agents.machine -n " |
289 | - "--logfile=/var/log/ensemble/machine-agent.log " |
290 | - "--pidfile=/var/run/ensemble/machine-agent.pid") |
291 | - launch_scripts.append(machine_agent_script_template % machine_data) |
292 | - |
293 | - if self._master: |
294 | - launch_scripts.append(_get_provision_agent_script(machine_data)) |
295 | - |
296 | - return launch_scripts |
297 | - |
298 | - def _get_install_scripts(self): |
299 | - branch = self._provider.config.get("ensemble-branch") |
300 | - if branch is None: |
301 | - return ["sudo apt-get -y install ensemble"] |
302 | - # If we're using a custom branch, pass it via cloud-init to the agents, |
303 | - # so the entire cluster uses it. |
304 | - return [ |
305 | - "sudo apt-get install -y python-txzookeeper", |
306 | - "sudo mkdir -p /usr/lib/ensemble", |
307 | - "cd /usr/lib/ensemble && sudo /usr/bin/bzr co %s ensemble" % \ |
308 | - branch, |
309 | - "cd /usr/lib/ensemble/ensemble && sudo python setup.py develop"] |
310 | - |
311 | - def _get_initialize_script(self): |
312 | - admin_secret = self._provider.config["admin-secret"] |
313 | - return _get_initialize_script( |
314 | - self.get_instance_id_command(), admin_secret) |
315 | - |
316 | - def _on_master_launched(self, machines): |
317 | - # TODO this should be part of Bootstrap (and should really extend, |
318 | - # rather than effectively replace, the result of |
319 | - # self._provider.get_zookeeper_machines) |
320 | + def _create_cloud_init(self, machine_id, zookeepers): |
321 | + config = self._provider.config |
322 | + cloud_init = CloudInit() |
323 | + cloud_init.add_ssh_key(get_user_authorized_keys(config)) |
324 | + cloud_init.set_machine_id(machine_id) |
325 | + cloud_init.set_zookeeper_machines(zookeepers) |
326 | + if config.get("ensemble-branch"): |
327 | + cloud_init.set_ensemble_source(config["ensemble-branch"]) |
328 | + if self._master: |
329 | + cloud_init.enable_bootstrap() |
330 | + cloud_init.set_zookeeper_secret(config["admin-secret"]) |
331 | + return cloud_init |
332 | + |
333 | + def _on_new_zookeepers(self, machines): |
334 | instance_ids = [m.instance_id for m in machines] |
335 | - d = self._provider.save_state({"zookeeper-instances": instance_ids}) |
336 | - d.addCallback(lambda _: machines) |
337 | - return d |
338 | + return self._provider.save_state({"zookeeper-instances": instance_ids}) |
339 | |
340 | === modified file 'ensemble/providers/common/tests/data/cloud_init_bootstrap' |
341 | --- ensemble/providers/common/tests/data/cloud_init_bootstrap 2011-09-13 10:56:30 +0000 |
342 | +++ ensemble/providers/common/tests/data/cloud_init_bootstrap 2011-09-13 10:56:30 +0000 |
343 | @@ -14,5 +14,4 @@ |
344 | -n --logfile=/var/log/ensemble/machine-agent.log --pidfile=/var/run/ensemble/machine-agent.pid', |
345 | 'ENSEMBLE_ZOOKEEPER=localhost:2181 python -m ensemble.agents.provision -n --logfile=/var/log/ensemble/provision-agent.log |
346 | --pidfile=/var/run/ensemble/provision-agent.pid'] |
347 | -ssh_authorized_keys: |
348 | -- [chubb] |
349 | +ssh_authorized_keys: [chubb] |
350 | |
351 | === modified file 'ensemble/providers/common/tests/data/cloud_init_bootstrap_zookeepers' |
352 | --- ensemble/providers/common/tests/data/cloud_init_bootstrap_zookeepers 2011-09-13 10:56:30 +0000 |
353 | +++ ensemble/providers/common/tests/data/cloud_init_bootstrap_zookeepers 2011-09-13 10:56:30 +0000 |
354 | @@ -15,5 +15,4 @@ |
355 | --pidfile=/var/run/ensemble/machine-agent.pid', 'ENSEMBLE_ZOOKEEPER=cotswold:2181,longleat:2181,localhost:2181 |
356 | python -m ensemble.agents.provision -n --logfile=/var/log/ensemble/provision-agent.log |
357 | --pidfile=/var/run/ensemble/provision-agent.pid'] |
358 | -ssh_authorized_keys: |
359 | -- [chubb] |
360 | +ssh_authorized_keys: [chubb] |
361 | |
362 | === modified file 'ensemble/providers/common/tests/data/cloud_init_branch' |
363 | --- ensemble/providers/common/tests/data/cloud_init_branch 2011-09-13 10:56:30 +0000 |
364 | +++ ensemble/providers/common/tests/data/cloud_init_branch 2011-09-13 10:56:30 +0000 |
365 | @@ -14,5 +14,4 @@ |
366 | sudo mkdir -p /var/log/ensemble, 'ENSEMBLE_MACHINE_ID=passport ENSEMBLE_ZOOKEEPER=cotswold:2181,longleat:2181 |
367 | python -m ensemble.agents.machine -n --logfile=/var/log/ensemble/machine-agent.log |
368 | --pidfile=/var/run/ensemble/machine-agent.pid'] |
369 | -ssh_authorized_keys: |
370 | -- [chubb] |
371 | +ssh_authorized_keys: [chubb] |
372 | |
373 | === modified file 'ensemble/providers/common/tests/data/cloud_init_distro' |
374 | --- ensemble/providers/common/tests/data/cloud_init_distro 2011-09-13 10:56:30 +0000 |
375 | +++ ensemble/providers/common/tests/data/cloud_init_distro 2011-09-13 10:56:30 +0000 |
376 | @@ -10,5 +10,4 @@ |
377 | -p /var/log/ensemble, 'ENSEMBLE_MACHINE_ID=passport ENSEMBLE_ZOOKEEPER=cotswold:2181,longleat:2181 |
378 | python -m ensemble.agents.machine -n --logfile=/var/log/ensemble/machine-agent.log |
379 | --pidfile=/var/run/ensemble/machine-agent.pid'] |
380 | -ssh_authorized_keys: |
381 | -- [chubb] |
382 | +ssh_authorized_keys: [chubb] |
383 | |
384 | === modified file 'ensemble/providers/common/tests/data/cloud_init_normal' |
385 | --- ensemble/providers/common/tests/data/cloud_init_normal 2011-09-13 10:56:30 +0000 |
386 | +++ ensemble/providers/common/tests/data/cloud_init_normal 2011-09-13 10:56:30 +0000 |
387 | @@ -12,5 +12,4 @@ |
388 | -p /var/log/ensemble, 'ENSEMBLE_MACHINE_ID=passport ENSEMBLE_ZOOKEEPER=cotswold:2181,longleat:2181 |
389 | python -m ensemble.agents.machine -n --logfile=/var/log/ensemble/machine-agent.log |
390 | --pidfile=/var/run/ensemble/machine-agent.pid'] |
391 | -ssh_authorized_keys: |
392 | -- [chubb] |
393 | +ssh_authorized_keys: [chubb] |
394 | |
395 | === added file 'ensemble/providers/common/tests/test_bootstrap.py' |
396 | --- ensemble/providers/common/tests/test_bootstrap.py 1970-01-01 00:00:00 +0000 |
397 | +++ ensemble/providers/common/tests/test_bootstrap.py 2011-09-13 10:56:30 +0000 |
398 | @@ -0,0 +1,102 @@ |
399 | +import logging |
400 | +import tempfile |
401 | + |
402 | +from twisted.internet.defer import fail, succeed |
403 | + |
404 | +from ensemble.errors import EnvironmentNotFound, ProviderError |
405 | +from ensemble.lib.testing import TestCase |
406 | +from ensemble.providers.common.base import MachineProviderBase |
407 | +from ensemble.providers.dummy import DummyMachine, FileStorage |
408 | + |
409 | + |
410 | +class SomeError(Exception): |
411 | + pass |
412 | + |
413 | + |
414 | +class WorkingFileStorage(FileStorage): |
415 | + |
416 | + def __init__(self): |
417 | + super(WorkingFileStorage, self).__init__(tempfile.mkdtemp()) |
418 | + |
419 | + |
420 | +class UnwritableFileStorage(object): |
421 | + |
422 | + def put(self, name, f): |
423 | + return fail(Exception("oh noes")) |
424 | + |
425 | + |
426 | +class DummyProvider(MachineProviderBase): |
427 | + |
428 | + def __init__(self, file_storage, zookeeper): |
429 | + self._file_storage = file_storage |
430 | + self._zookeeper = zookeeper |
431 | + |
432 | + def get_file_storage(self): |
433 | + return self._file_storage |
434 | + |
435 | + def get_zookeeper_machines(self): |
436 | + if isinstance(self._zookeeper, Exception): |
437 | + return fail(self._zookeeper) |
438 | + if self._zookeeper: |
439 | + return succeed([self._zookeeper]) |
440 | + return fail(EnvironmentNotFound()) |
441 | + |
442 | + def start_machine(self, machine_id, master=False): |
443 | + assert master is True |
444 | + return [DummyMachine("i-keepzoos")] |
445 | + |
446 | + |
447 | +class BootstrapTest(TestCase): |
448 | + |
449 | + def test_unknown_error(self): |
450 | + provider = DummyProvider(None, SomeError()) |
451 | + d = provider.bootstrap() |
452 | + self.assertFailure(d, SomeError) |
453 | + return d |
454 | + |
455 | + def test_zookeeper_exists(self): |
456 | + log = self.capture_logging("ensemble.common", level=logging.DEBUG) |
457 | + provider = DummyProvider( |
458 | + WorkingFileStorage(), DummyMachine("i-alreadykeepzoos")) |
459 | + d = provider.bootstrap() |
460 | + |
461 | + def verify(machines): |
462 | + (machine,) = machines |
463 | + self.assertTrue(isinstance(machine, DummyMachine)) |
464 | + self.assertEquals(machine.instance_id, "i-alreadykeepzoos") |
465 | + |
466 | + log_text = log.getvalue() |
467 | + self.assertIn( |
468 | + "Ensemble environment previously bootstrapped", log_text) |
469 | + self.assertNotIn("Launching", log_text) |
470 | + d.addCallback(verify) |
471 | + return d |
472 | + |
473 | + def test_bad_storage(self): |
474 | + provider = DummyProvider(UnwritableFileStorage(), None) |
475 | + d = provider.bootstrap() |
476 | + self.assertFailure(d, ProviderError) |
477 | + |
478 | + def verify(error): |
479 | + self.assertEquals( |
480 | + str(error), |
481 | + "Bootstrap aborted because file storage is not writable: " |
482 | + "oh noes") |
483 | + d.addCallback(verify) |
484 | + return d |
485 | + |
486 | + def test_create_zookeeper(self): |
487 | + log = self.capture_logging("ensemble.common", level=logging.DEBUG) |
488 | + provider = DummyProvider(WorkingFileStorage(), None) |
489 | + d = provider.bootstrap() |
490 | + |
491 | + def verify(machines): |
492 | + (machine,) = machines |
493 | + self.assertTrue(isinstance(machine, DummyMachine)) |
494 | + self.assertEquals(machine.instance_id, "i-keepzoos") |
495 | + |
496 | + log_text = log.getvalue() |
497 | + self.assertIn("Launching Ensemble bootstrap instance", log_text) |
498 | + self.assertNotIn("previously bootstrapped", log_text) |
499 | + d.addCallback(verify) |
500 | + return d |
501 | |
502 | === modified file 'ensemble/providers/common/tests/test_cloudinit.py' |
503 | --- ensemble/providers/common/tests/test_cloudinit.py 2011-09-13 10:56:30 +0000 |
504 | +++ ensemble/providers/common/tests/test_cloudinit.py 2011-09-13 10:56:30 +0000 |
505 | @@ -26,7 +26,7 @@ |
506 | cloud_init.add_ssh_key("chubb") |
507 | cloud_init.set_machine_id("passport") |
508 | cloud_init.set_provider_type("dummy") |
509 | - cloud_init.set_instance_id("token") |
510 | + cloud_init.set_instance_id_accessor("token") |
511 | cloud_init.set_zookeeper_secret("seekrit") |
512 | if with_zookeepers: |
513 | cloud_init.set_zookeeper_machines([ |
514 | @@ -54,7 +54,7 @@ |
515 | self.assertEquals( |
516 | str(error), |
517 | "Incomplete cloud-init: you need to call add_ssh_key, " |
518 | - "set_machine_id, set_provider_type, set_instance_id, " |
519 | + "set_machine_id, set_provider_type, set_instance_id_accessor, " |
520 | "set_zookeeper_secret") |
521 | |
522 | def test_source_validate(self): |
523 | |
524 | === modified file 'ensemble/providers/common/tests/test_launch.py' |
525 | --- ensemble/providers/common/tests/test_launch.py 2011-08-17 12:04:56 +0000 |
526 | +++ ensemble/providers/common/tests/test_launch.py 2011-09-13 10:56:30 +0000 |
527 | @@ -1,243 +1,80 @@ |
528 | -import logging |
529 | import tempfile |
530 | |
531 | -from twisted.internet.defer import fail, inlineCallbacks, succeed |
532 | +from twisted.internet.defer import fail, succeed |
533 | |
534 | -from ensemble.errors import EnvironmentNotFound, ProviderError |
535 | +from ensemble.errors import EnvironmentNotFound |
536 | from ensemble.lib.testing import TestCase |
537 | from ensemble.providers.common.base import MachineProviderBase |
538 | -from ensemble.providers.common.launch import ( |
539 | - BOOTSTRAP_PACKAGES, DEFAULT_PACKAGES, DEFAULT_REPOSITORIES, LaunchMachine) |
540 | +from ensemble.providers.common.launch import LaunchMachine |
541 | from ensemble.providers.dummy import DummyMachine, FileStorage |
542 | |
543 | -DEFAULT_INSTALL = ["sudo apt-get -y install ensemble"] |
544 | -CREATE_DIRS = ["sudo mkdir -p /var/lib/ensemble", |
545 | - "sudo mkdir -p /var/log/ensemble"] |
546 | - |
547 | - |
548 | -class DummyLaunchMachine(LaunchMachine): |
549 | - |
550 | - def start_machine(self, variables, data): |
551 | - if self._master: |
552 | - name = "bootstrapped-instance-id" |
553 | - else: |
554 | - name = "some-instance-id" |
555 | - return [DummyMachine(name)] |
556 | - |
557 | - def get_instance_id_command(self): |
558 | - return "$(magic)" |
559 | - |
560 | - |
561 | -class WorkingFileStorage(FileStorage): |
562 | - |
563 | - def __init__(self): |
564 | - super(WorkingFileStorage, self).__init__(tempfile.mkdtemp()) |
565 | - |
566 | - |
567 | -class UnwritableFileStorage(object): |
568 | - |
569 | - def put(self, name, f): |
570 | - return fail(Exception("oh noes")) |
571 | - |
572 | - |
573 | -def get_provider(launch_class=DummyLaunchMachine, config=None, zookeeper=True, |
574 | - file_storage_class=WorkingFileStorage): |
575 | + |
576 | +def launch_machine(test, master, zookeeper): |
577 | |
578 | class DummyProvider(MachineProviderBase): |
579 | |
580 | - def __init__(self, config): |
581 | - super(DummyProvider, self).__init__("venus", config) |
582 | - self._file_storage = file_storage_class() |
583 | - |
584 | - def get_zookeeper_machines(self): |
585 | - # this is mocked out to avoid insane complexity |
586 | - if isinstance(zookeeper, Exception): |
587 | - return fail(zookeeper) |
588 | - return succeed( |
589 | - [DummyMachine("zookeeper-instance-id", |
590 | - private_dns_name="zookeeper.internal")]) |
591 | + def __init__(self): |
592 | + self.config = {"admin-secret": "BLAH"} |
593 | + self._file_storage = FileStorage(tempfile.mkdtemp()) |
594 | |
595 | def get_file_storage(self): |
596 | return self._file_storage |
597 | |
598 | - def start_machine(self, machine_data, master=False): |
599 | - return launch_class(self, master).run(machine_data) |
600 | - |
601 | - create_config = {"authorized-keys": "abc"} |
602 | - if config is not None: |
603 | - create_config.update(config) |
604 | - return DummyProvider(create_config) |
605 | - |
606 | - |
607 | -def get_launch(launch_class=DummyLaunchMachine, config=None, |
608 | - zookeeper=True, master=False): |
609 | - provider = get_provider(launch_class, config, zookeeper) |
610 | - return launch_class(provider, master=master) |
611 | + def get_zookeeper_machines(self): |
612 | + if zookeeper: |
613 | + return succeed([zookeeper]) |
614 | + return fail(EnvironmentNotFound()) |
615 | + |
616 | + class DummyLaunchMachine(LaunchMachine): |
617 | + |
618 | + def start_machine(self, machine_id, zookeepers): |
619 | + test.assertEquals(machine_id, "1234") |
620 | + test.assertEquals(zookeepers, filter(None, [zookeeper])) |
621 | + test.assertEquals(self._master, master) |
622 | + test.assertEquals(self._constraints, {}) |
623 | + return succeed([DummyMachine("i-malive")]) |
624 | + |
625 | + provider = DummyProvider() |
626 | + d = DummyLaunchMachine(provider, master).run("1234") |
627 | + return provider, d |
628 | |
629 | |
630 | class LaunchMachineTest(TestCase): |
631 | |
632 | - def _verify_machines(self, machines): |
633 | - (machine,) = machines |
634 | - self.assertTrue(isinstance(machine, DummyMachine)) |
635 | - self.assertEquals(machine.instance_id, "some-instance-id") |
636 | - |
637 | - def test_no_machine_id(self): |
638 | - launch = get_launch() |
639 | - d = launch.run({}) |
640 | - self.assertFailure(d, ProviderError) |
641 | - return d |
642 | - |
643 | - def test_basic_run(self): |
644 | - launch = get_launch() |
645 | - d = launch.run({"machine-id": "machine-77"}) |
646 | - d.addCallback(self._verify_machines) |
647 | - return d |
648 | - |
649 | - @inlineCallbacks |
650 | - def test_bootstrap_run(self): |
651 | - provider = get_provider(config={"admin-secret": "whatever"}) |
652 | - launch = DummyLaunchMachine(provider, master=True) |
653 | - yield launch.run({"machine-id": "machine-32767"}) |
654 | - saved_state = yield provider.load_state() |
655 | - self.assertEquals( |
656 | - saved_state, {"zookeeper-instances": ["bootstrapped-instance-id"]}) |
657 | - |
658 | - def test_get_machine_variables_normal(self): |
659 | - launch = get_launch() |
660 | - d = launch.get_machine_variables({"machine-id": "machine-999"}) |
661 | - |
662 | - def verify_vars(vars): |
663 | - expect_vars = { |
664 | - "authorized_keys": "abc", |
665 | - "ensemble-zookeeper-hosts": "zookeeper.internal:2181", |
666 | - "machine-id": "machine-999", |
667 | - "packages": DEFAULT_PACKAGES, |
668 | - "repositories": DEFAULT_REPOSITORIES, |
669 | - "scripts": DEFAULT_INSTALL + CREATE_DIRS + |
670 | - ["ENSEMBLE_MACHINE_ID=machine-999 " |
671 | - "ENSEMBLE_ZOOKEEPER=zookeeper.internal:2181 " |
672 | - "python -m ensemble.agents.machine -n " |
673 | - "--logfile=/var/log/ensemble/machine-agent.log " |
674 | - "--pidfile=/var/run/ensemble/machine-agent.pid"]} |
675 | - self.assertEquals(vars, expect_vars) |
676 | - d.addCallback(verify_vars) |
677 | - return d |
678 | - |
679 | - def test_get_machine_variables_with_ensemble_branch(self): |
680 | - branch_name = "lp:~my/ensemble/branch" |
681 | - launch = get_launch(config={"ensemble-branch": branch_name}) |
682 | - d = launch.get_machine_variables({"machine-id": "machine-303"}) |
683 | - |
684 | - def verify_vars(vars): |
685 | - scripts = vars["scripts"] |
686 | - install_scripts = ["sudo apt-get install -y python-txzookeeper", |
687 | - "sudo mkdir -p /usr/lib/ensemble", |
688 | - "cd /usr/lib/ensemble && " |
689 | - "sudo /usr/bin/bzr co " |
690 | - + branch_name + " ensemble", |
691 | - "cd /usr/lib/ensemble/ensemble && " |
692 | - "sudo python setup.py develop"] |
693 | - install_count = len(install_scripts) |
694 | - self.assertEquals(scripts[:install_count], install_scripts) |
695 | - |
696 | - other_scripts = scripts[install_count:] |
697 | - self.assertEquals(other_scripts, |
698 | - CREATE_DIRS + [ |
699 | - "ENSEMBLE_MACHINE_ID=machine-303 " |
700 | - "ENSEMBLE_ZOOKEEPER=zookeeper.internal:2181 " |
701 | - "python -m ensemble.agents.machine -n " |
702 | - "--logfile=/var/log/ensemble/machine-agent.log " |
703 | - "--pidfile=/var/run/ensemble/machine-agent.pid"]) |
704 | - d.addCallback(verify_vars) |
705 | - return d |
706 | - |
707 | - def test_get_machine_variables_bootstrap(self): |
708 | - launch = get_launch(config={"admin-secret": "SEEKRIT"}, master=True) |
709 | - d = launch.get_machine_variables({"machine-id": "machine-757"}) |
710 | - |
711 | - def verify_vars(vars): |
712 | - expect_vars = { |
713 | - "authorized_keys": "abc", |
714 | - "ensemble-zookeeper-hosts": "localhost:2181", |
715 | - "machine-id": "machine-757", |
716 | - "packages": DEFAULT_PACKAGES + BOOTSTRAP_PACKAGES, |
717 | - "repositories": DEFAULT_REPOSITORIES} |
718 | - for key, value in expect_vars.items(): |
719 | - self.assertEquals(vars[key], value) |
720 | - |
721 | - scripts = vars["scripts"] |
722 | - self.assertEquals(scripts[:3], DEFAULT_INSTALL + CREATE_DIRS) |
723 | - admin_init, machine_agent, provision_agent = scripts[3:] |
724 | - |
725 | - self.assertTrue(admin_init.startswith( |
726 | - "ensemble-admin initialize " |
727 | - '--instance-id=$(magic) --admin-identity="admin:')) |
728 | - self.assertFalse("SEEKRIT" in admin_init) |
729 | - |
730 | - expect_machine = ( |
731 | - "ENSEMBLE_MACHINE_ID=machine-757 " |
732 | - "ENSEMBLE_ZOOKEEPER=localhost:2181 " |
733 | - "python -m ensemble.agents.machine -n " |
734 | - "--logfile=/var/log/ensemble/machine-agent.log " |
735 | - "--pidfile=/var/run/ensemble/machine-agent.pid") |
736 | - self.assertEquals(machine_agent, expect_machine) |
737 | - |
738 | - expect_provision = ( |
739 | - "ENSEMBLE_ZOOKEEPER=localhost:2181 " |
740 | - "python -m ensemble.agents.provision -n " |
741 | - "--logfile=/var/log/ensemble/provision-agent.log " |
742 | - "--pidfile=/var/run/ensemble/provision-agent.pid") |
743 | - self.assertEquals(provision_agent, expect_provision) |
744 | - d.addCallback(verify_vars) |
745 | - return d |
746 | - |
747 | - |
748 | -class BootstrapTest(TestCase): |
749 | - |
750 | - def test_bootstrap_unknown_error(self): |
751 | - class BBQError(Exception): |
752 | - pass |
753 | - provider = get_provider(zookeeper=BBQError()) |
754 | - d = provider.bootstrap() |
755 | - self.assertFailure(d, BBQError) |
756 | - return d |
757 | - |
758 | - def test_bootstrap_unwritable_storage(self): |
759 | - provider = get_provider(zookeeper=EnvironmentNotFound(), |
760 | - file_storage_class=UnwritableFileStorage) |
761 | - d = provider.bootstrap() |
762 | - self.assertFailure(d, ProviderError) |
763 | - return d |
764 | - |
765 | - def test_bootstrap_no_launch(self): |
766 | - log = self.capture_logging("ensemble.common", level=logging.DEBUG) |
767 | - provider = get_provider() |
768 | - d = provider.bootstrap() |
769 | - |
770 | - def verify_machines(machines): |
771 | - (machine,) = machines |
772 | - self.assertTrue(isinstance(machine, DummyMachine)) |
773 | - self.assertEquals(machine.instance_id, "zookeeper-instance-id") |
774 | - log_text = log.getvalue() |
775 | - self.assertIn( |
776 | - "Ensemble environment previously bootstrapped", log_text) |
777 | - self.assertNotIn("Launching", log_text) |
778 | - d.addCallback(verify_machines) |
779 | - return d |
780 | - |
781 | - def test_bootstrap_launch(self): |
782 | - log = self.capture_logging("ensemble.common", level=logging.DEBUG) |
783 | - provider = get_provider(config={"admin-secret": "SEEKRIT"}, |
784 | - zookeeper=EnvironmentNotFound()) |
785 | - d = provider.bootstrap() |
786 | - |
787 | - def verify_machines(machines): |
788 | - (machine,) = machines |
789 | - self.assertTrue(isinstance(machine, DummyMachine)) |
790 | - self.assertEquals(machine.instance_id, "bootstrapped-instance-id") |
791 | - log_text = log.getvalue() |
792 | - self.assertIn("Launching Ensemble bootstrap instance", log_text) |
793 | - self.assertNotIn("previously bootstrapped", log_text) |
794 | - d.addCallback(verify_machines) |
795 | + def assert_success(self, master, zookeeper): |
796 | + provider, d = launch_machine(self, master, zookeeper) |
797 | + |
798 | + def verify(machines): |
799 | + (machine,) = machines |
800 | + self.assertTrue(isinstance(machine, DummyMachine)) |
801 | + self.assertEquals(machine.instance_id, "i-malive") |
802 | + return provider |
803 | + d.addCallback(verify) |
804 | + return d |
805 | + |
806 | + def test_start_nonzookeeper_no_zookeepers(self): |
807 | + """Starting a non-zookeeper without a zookeeper is an error""" |
808 | + unused, d = launch_machine(self, False, None) |
809 | + self.assertFailure(d, EnvironmentNotFound) |
810 | + return d |
811 | + |
812 | + def test_start_zookeeper_no_zookeepers(self): |
813 | + """A new zookeeper should be recorded in provider state""" |
814 | + d = self.assert_success(True, None) |
815 | + |
816 | + def verify(provider): |
817 | + provider_state = yield provider.load_state() |
818 | + self.assertEquals( |
819 | + provider_state, {"zookeeper-instances": ["i-malive"]}) |
820 | + d.addCallback(verify) |
821 | + return d |
822 | + |
823 | + def test_works_with_zookeeper(self): |
824 | + """Creating a non-zookeeper machine should not alter provider state""" |
825 | + d = self.assert_success(False, DummyMachine("i-keepzoos")) |
826 | + |
827 | + def verify(provider): |
828 | + provider_state = yield provider.load_state() |
829 | + self.assertEquals(provider_state, False) |
830 | + d.addCallback(verify) |
831 | return d |
832 | |
833 | === modified file 'ensemble/providers/common/tests/test_utils.py' |
834 | --- ensemble/providers/common/tests/test_utils.py 2011-08-18 12:19:22 +0000 |
835 | +++ ensemble/providers/common/tests/test_utils.py 2011-09-13 10:56:30 +0000 |
836 | @@ -1,5 +1,4 @@ |
837 | import os |
838 | - |
839 | from yaml import load |
840 | |
841 | from twisted.python.failure import Failure |
842 | @@ -100,7 +99,7 @@ |
843 | scripts = ["wget http://lwn.net > /tmp/out"] |
844 | repositories = ["ppa:ensemble/ppa"] |
845 | output = format_cloud_init( |
846 | - "zebra", |
847 | + ["zebra"], |
848 | packages=packages, |
849 | scripts=scripts, |
850 | repositories=repositories, |
851 | |
852 | === modified file 'ensemble/providers/common/utils.py' |
853 | --- ensemble/providers/common/utils.py 2011-08-31 21:12:40 +0000 |
854 | +++ ensemble/providers/common/utils.py 2011-09-13 10:56:30 +0000 |
855 | @@ -1,6 +1,5 @@ |
856 | import logging |
857 | import os |
858 | - |
859 | from yaml import safe_dump |
860 | |
861 | from twisted.python.failure import Failure |
862 | @@ -40,6 +39,7 @@ |
863 | raise error |
864 | |
865 | |
866 | +# XXX this should either be singular, or return a list |
867 | def get_user_authorized_keys(config): |
868 | """Locate a public key for the user. |
869 | |
870 | @@ -85,7 +85,7 @@ |
871 | |
872 | @param authorized_keys The authorized SSH keys to be used when populating |
873 | the newly launched machine's. |
874 | - @type string |
875 | + @type list of strings |
876 | |
877 | @param packages A list of packages to be installed on a machine. |
878 | @type list of strings |
879 | @@ -104,7 +104,7 @@ |
880 | cloud_config = { |
881 | "apt-update": True, |
882 | "apt-upgrade": True, |
883 | - "ssh_authorized_keys": [authorized_keys], |
884 | + "ssh_authorized_keys": authorized_keys, |
885 | "packages": [], |
886 | "output": { "all": "| tee -a /var/log/cloud-init-output.log" }} |
887 | |
888 | |
889 | === modified file 'ensemble/providers/ec2/__init__.py' |
890 | --- ensemble/providers/ec2/__init__.py 2011-08-29 18:34:02 +0000 |
891 | +++ ensemble/providers/ec2/__init__.py 2011-09-13 10:56:30 +0000 |
892 | @@ -1,7 +1,8 @@ |
893 | import os |
894 | import re |
895 | |
896 | -from twisted.internet.defer import inlineCallbacks, returnValue |
897 | +from twisted.internet.defer import fail, inlineCallbacks, returnValue |
898 | + |
899 | from txaws.ec2.exception import EC2Error |
900 | from txaws.service import AWSServiceRegion |
901 | |
902 | @@ -53,13 +54,17 @@ |
903 | def start_machine(self, machine_data, master=False): |
904 | """Start a machine in the provider. |
905 | |
906 | - @param machine_data: a dictionary of data to pass along to the newly |
907 | - launched machine. |
908 | + `machine_data`: a dictionary describing the machine to be launched. |
909 | |
910 | - @param master if True, machine will initialize the ensemble admin |
911 | + @param master: if True, machine will initialize the ensemble admin |
912 | and run a provisioning agent. |
913 | """ |
914 | - return EC2LaunchMachine(self, master).run(machine_data) |
915 | + if "machine-id" not in machine_data: |
916 | + return fail(ProviderError( |
917 | + "Cannot launch a machine without specifying a machine-id")) |
918 | + machine_id = machine_data["machine-id"] |
919 | + constraints = machine_data.get("constraints", {}) |
920 | + return EC2LaunchMachine(self, master, constraints).run(machine_id) |
921 | |
922 | @inlineCallbacks |
923 | def get_machines(self, instance_ids=()): |
924 | |
925 | === modified file 'ensemble/providers/ec2/launch.py' |
926 | --- ensemble/providers/ec2/launch.py 2011-08-23 09:59:59 +0000 |
927 | +++ ensemble/providers/ec2/launch.py 2011-09-13 10:56:30 +0000 |
928 | @@ -1,67 +1,52 @@ |
929 | from twisted.internet.defer import inlineCallbacks, returnValue |
930 | + |
931 | from txaws.ec2.exception import EC2Error |
932 | |
933 | from ensemble.errors import ProviderInteractionError |
934 | from ensemble.providers.common.launch import LaunchMachine |
935 | |
936 | from .machine import machine_from_instance |
937 | -from .utils import log, get_launch_options |
938 | +from .utils import get_image_id, log |
939 | |
940 | |
941 | class EC2LaunchMachine(LaunchMachine): |
942 | """Amazon EC2 operation for launching an instance""" |
943 | |
944 | @inlineCallbacks |
945 | - def start_machine(self, machine_variables, machine_id): |
946 | + def start_machine(self, machine_id, zookeepers): |
947 | """Actually launch an instance on EC2. |
948 | |
949 | - `machine_variables`: non-provider-specific data, sufficient to |
950 | - define the machine's behaviour once it exists. |
951 | + `machine_id`: the ensemble machine ID |
952 | |
953 | - `machine_id`: the external machine ID (also exists in the |
954 | - above bag of data) |
955 | + `zookeepers`: a list of `ProviderMachine`s already running zookeeper, |
956 | + which the machine will need to connect to. May be empty. |
957 | |
958 | Returns a singe-entry list containing a ProviderMachine for the |
959 | new instance |
960 | """ |
961 | - # Retrieves standard ec2 run_instances arguments, of note |
962 | - # image id, and instance type. |
963 | - machine_options = yield get_launch_options( |
964 | - self._provider.config, |
965 | - **machine_variables) |
966 | - |
967 | - # Ensure the ensemble security groups exist (both the overall |
968 | - # group and the machine-specific group) and are included in |
969 | - # the machine instance launch options. |
970 | - machine_options["security_groups"] = yield self._ensure_group( |
971 | - machine_options["security_groups"], |
972 | - machine_id) |
973 | - |
974 | - # Launch the instance. |
975 | - instances = yield self._provider.ec2.run_instances(**machine_options) |
976 | - returnValue([machine_from_instance(instance) |
977 | - for instance in instances]) |
978 | - |
979 | - def get_instance_id_command(self): |
980 | - """Snippet to discover local instance-id |
981 | - |
982 | - @return: a snippet of shell script that evaluates to the |
983 | - local machine's instance_id. |
984 | - """ |
985 | - return "$(curl http://169.254.169.254/1.0/meta-data/instance-id)" |
986 | - |
987 | - def get_machine_variables(self, vars_in): |
988 | - vars_deferred = super( |
989 | - EC2LaunchMachine, self).get_machine_variables(vars_in) |
990 | - |
991 | - def set_provider_type(vars): |
992 | - vars["ensemble-provider-type"] = "ec2" |
993 | - return vars |
994 | - vars_deferred.addCallback(set_provider_type) |
995 | - return vars_deferred |
996 | + cloud_init = self._create_cloud_init(machine_id, zookeepers) |
997 | + cloud_init.set_provider_type("ec2") |
998 | + cloud_init.set_instance_id_accessor( |
999 | + "$(curl http://169.254.169.254/1.0/meta-data/instance-id)") |
1000 | + user_data = cloud_init.render() |
1001 | + |
1002 | + instance_type = self._provider.config.get( |
1003 | + "default-instance-type", "m1.small") |
1004 | + image_id = yield get_image_id(self._provider.config, self._constraints) |
1005 | + security_groups = yield self._ensure_groups(machine_id) |
1006 | + |
1007 | + instances = yield self._provider.ec2.run_instances( |
1008 | + image_id=image_id, |
1009 | + instance_type=instance_type, |
1010 | + min_count=1, |
1011 | + max_count=1, |
1012 | + security_groups=security_groups, |
1013 | + user_data=user_data) |
1014 | + |
1015 | + returnValue([machine_from_instance(i) for i in instances]) |
1016 | |
1017 | @inlineCallbacks |
1018 | - def _ensure_group(self, groups, machine_id): |
1019 | + def _ensure_groups(self, machine_id): |
1020 | """Ensure the ensemble group is the machine launch groups. |
1021 | |
1022 | Machines launched by ensemble are tagged with a group so they |
1023 | @@ -71,21 +56,15 @@ |
1024 | specific machine security group is created for each machine, |
1025 | so that its firewall rules can be configured per machine. |
1026 | |
1027 | - `groups`: The configured security groups for a machine instance. |
1028 | - |
1029 | `machine_id`: The external machine ID of this machine instance |
1030 | """ |
1031 | - # Label the instance with the ensemble name. |
1032 | - security_groups = yield self._provider.ec2.describe_security_groups() |
1033 | ensemble_group = "ensemble-%s" % self._provider.environment_name |
1034 | ensemble_machine_group = "ensemble-%s-%s" % ( |
1035 | self._provider.environment_name, machine_id) |
1036 | + |
1037 | + security_groups = yield self._provider.ec2.describe_security_groups() |
1038 | group_ids = [group.name for group in security_groups] |
1039 | |
1040 | - # Ensure the provider group is added to the instance groups. |
1041 | - if not ensemble_group in groups: |
1042 | - groups.append(ensemble_group) |
1043 | - |
1044 | # Create the provider group if doesn't exist. |
1045 | if not ensemble_group in group_ids: |
1046 | log.debug("Creating ensemble provider group %s", ensemble_group) |
1047 | @@ -131,6 +110,6 @@ |
1048 | ensemble_machine_group, |
1049 | "Ensemble group for %s machine %s" % ( |
1050 | self._provider.environment_name, machine_id)) |
1051 | - groups.append(ensemble_machine_group) |
1052 | - |
1053 | - returnValue(groups) |
1054 | + |
1055 | + returnValue([ensemble_group, ensemble_machine_group]) |
1056 | + |
1057 | |
1058 | === modified file 'ensemble/providers/ec2/tests/common.py' |
1059 | --- ensemble/providers/ec2/tests/common.py 2011-08-29 02:23:11 +0000 |
1060 | +++ ensemble/providers/ec2/tests/common.py 2011-09-13 10:56:30 +0000 |
1061 | @@ -12,8 +12,6 @@ |
1062 | from ensemble.providers.ec2 import MachineProvider |
1063 | from ensemble.providers.ec2.machine import EC2ProviderMachine |
1064 | |
1065 | - |
1066 | -MATCH_AMI = MATCH(lambda image_id: image_id.startswith("ami-")) |
1067 | MATCH_GROUP = MATCH(lambda x: x.startswith("ensemble-moon")) |
1068 | |
1069 | |
1070 | @@ -82,7 +80,7 @@ |
1071 | |
1072 | class EC2MachineLaunchMixin(object): |
1073 | |
1074 | - def _mock_launch_utils(self, **get_ami_kwargs): |
1075 | + def _mock_launch_utils(self, ami_name="ami-default", **get_ami_kwargs): |
1076 | get_public_key = self.mocker.replace( |
1077 | "ensemble.providers.common.utils.get_user_authorized_keys") |
1078 | |
1079 | @@ -100,7 +98,7 @@ |
1080 | |
1081 | def check_kwargs(**kwargs): |
1082 | self.assertEquals(kwargs, get_ami_kwargs) |
1083 | - return succeed("ami-714ba518") |
1084 | + return succeed(ami_name) |
1085 | self.mocker.call(check_kwargs) |
1086 | |
1087 | def _mock_create_group(self): |
1088 | |
1089 | === added file 'ensemble/providers/ec2/tests/data/bootstrap_cloud_init' |
1090 | --- ensemble/providers/ec2/tests/data/bootstrap_cloud_init 1970-01-01 00:00:00 +0000 |
1091 | +++ ensemble/providers/ec2/tests/data/bootstrap_cloud_init 2011-09-13 10:56:30 +0000 |
1092 | @@ -0,0 +1,18 @@ |
1093 | +#cloud-config |
1094 | +apt-update: true |
1095 | +apt-upgrade: true |
1096 | +apt_sources: |
1097 | +- {source: 'ppa:ensemble/ppa'} |
1098 | +machine-data: {ensemble-provider-type: ec2, ensemble-zookeeper-hosts: 'localhost:2181', |
1099 | + machine-id: '0'} |
1100 | +output: {all: '| tee -a /var/log/cloud-init-output.log'} |
1101 | +packages: [bzr, byobu, tmux, python-setuptools, python-twisted, python-argparse, python-txaws, |
1102 | + python-zookeeper, default-jre-headless, zookeeper, zookeeperd] |
1103 | +runcmd: [sudo apt-get -y install ensemble, sudo mkdir -p /var/lib/ensemble, sudo mkdir |
1104 | + -p /var/log/ensemble, 'ensemble-admin initialize --instance-id=$(curl http://169.254.169.254/1.0/meta-data/instance-id) |
1105 | + --admin-identity=admin:JbJ6sDGV37EHzbG9FPvttk64cmg=', 'ENSEMBLE_MACHINE_ID=0 ENSEMBLE_ZOOKEEPER=localhost:2181 |
1106 | + python -m ensemble.agents.machine -n --logfile=/var/log/ensemble/machine-agent.log |
1107 | + --pidfile=/var/run/ensemble/machine-agent.pid', 'ENSEMBLE_ZOOKEEPER=localhost:2181 |
1108 | + python -m ensemble.agents.provision -n --logfile=/var/log/ensemble/provision-agent.log |
1109 | + --pidfile=/var/run/ensemble/provision-agent.pid'] |
1110 | +ssh_authorized_keys: [zebra] |
1111 | |
1112 | === added file 'ensemble/providers/ec2/tests/data/launch_cloud_init' |
1113 | --- ensemble/providers/ec2/tests/data/launch_cloud_init 1970-01-01 00:00:00 +0000 |
1114 | +++ ensemble/providers/ec2/tests/data/launch_cloud_init 2011-09-13 10:56:30 +0000 |
1115 | @@ -0,0 +1,15 @@ |
1116 | +#cloud-config |
1117 | +apt-update: true |
1118 | +apt-upgrade: true |
1119 | +apt_sources: |
1120 | +- {source: 'ppa:ensemble/ppa'} |
1121 | +machine-data: {ensemble-provider-type: ec2, ensemble-zookeeper-hosts: 'es.example.internal:2181', |
1122 | + machine-id: '1'} |
1123 | +output: {all: '| tee -a /var/log/cloud-init-output.log'} |
1124 | +packages: [bzr, byobu, tmux, python-setuptools, python-twisted, python-argparse, python-txaws, |
1125 | + python-zookeeper] |
1126 | +runcmd: [sudo apt-get -y install ensemble, sudo mkdir -p /var/lib/ensemble, sudo mkdir |
1127 | + -p /var/log/ensemble, 'ENSEMBLE_MACHINE_ID=1 ENSEMBLE_ZOOKEEPER=es.example.internal:2181 |
1128 | + python -m ensemble.agents.machine -n --logfile=/var/log/ensemble/machine-agent.log |
1129 | + --pidfile=/var/run/ensemble/machine-agent.pid'] |
1130 | +ssh_authorized_keys: [zebra] |
1131 | |
1132 | === modified file 'ensemble/providers/ec2/tests/test_bootstrap.py' |
1133 | --- ensemble/providers/ec2/tests/test_bootstrap.py 2011-08-11 05:59:30 +0000 |
1134 | +++ ensemble/providers/ec2/tests/test_bootstrap.py 2011-09-13 10:56:30 +0000 |
1135 | @@ -1,6 +1,6 @@ |
1136 | import logging |
1137 | - |
1138 | -from yaml import dump, load |
1139 | +import os |
1140 | +from yaml import dump |
1141 | |
1142 | from twisted.internet.defer import succeed |
1143 | |
1144 | @@ -8,12 +8,11 @@ |
1145 | |
1146 | from ensemble.lib.mocker import MATCH |
1147 | from ensemble.lib.testing import TestCase |
1148 | -from ensemble.providers.common.launch import ( |
1149 | - BOOTSTRAP_PACKAGES, DEFAULT_REPOSITORIES, DEFAULT_PACKAGES) |
1150 | from ensemble.providers.ec2.machine import EC2ProviderMachine |
1151 | -from ensemble.state.auth import make_identity |
1152 | - |
1153 | -from .common import EC2TestMixin, EC2MachineLaunchMixin, MATCH_AMI |
1154 | + |
1155 | +from .common import EC2TestMixin, EC2MachineLaunchMixin |
1156 | + |
1157 | +DATA_DIR = os.path.join(os.path.abspath(os.path.dirname(__file__)), "data") |
1158 | |
1159 | |
1160 | class EC2BootstrapTest(EC2TestMixin, EC2MachineLaunchMixin, TestCase): |
1161 | @@ -34,55 +33,21 @@ |
1162 | MATCH(match_string)) |
1163 | self.mocker.result(succeed(True)) |
1164 | |
1165 | - def _mock_launch(self, machine_id): |
1166 | + def _mock_launch(self): |
1167 | """Mock launching a bootstrap machine on ec2.""" |
1168 | - credentials = "admin:%s" % self.get_config()["admin-secret"] |
1169 | - admin_identity = make_identity(credentials) |
1170 | - |
1171 | def verify_user_data(data): |
1172 | - |
1173 | - lines = data.split("\n") |
1174 | - self.assertEqual(lines.pop(0), "#cloud-config") |
1175 | - config = load("\n".join(lines)) |
1176 | - repos = [dict(source=r) for r in DEFAULT_REPOSITORIES] |
1177 | - self.assertEqual(config["apt_sources"], repos) |
1178 | - self.assertEqual( |
1179 | - config["packages"], |
1180 | - list(DEFAULT_PACKAGES) + BOOTSTRAP_PACKAGES) |
1181 | - self.failUnlessIn("admin-identity", config["runcmd"][-3]) |
1182 | - |
1183 | - script = ( |
1184 | - 'ensemble-admin initialize --instance-id=%s' |
1185 | - ' --admin-identity="%s"') % ( |
1186 | - "$(curl http://169.254.169.254/1.0/meta-data/instance-id)", |
1187 | - admin_identity) |
1188 | - |
1189 | - self.assertEqual(config["runcmd"][-3], script) |
1190 | - |
1191 | - script = ( |
1192 | - "ENSEMBLE_MACHINE_ID=0 ENSEMBLE_ZOOKEEPER=localhost:2181 " |
1193 | - "python -m ensemble.agents.machine -n " |
1194 | - "--logfile=/var/log/ensemble/machine-agent.log " |
1195 | - "--pidfile=/var/run/ensemble/machine-agent.pid") |
1196 | - |
1197 | - self.assertEqual(config["runcmd"][-2], script) |
1198 | - |
1199 | - provision_agent_script = ( |
1200 | - "ENSEMBLE_ZOOKEEPER=localhost:2181 " |
1201 | - "python -m ensemble.agents.provision -n " |
1202 | - "--logfile=/var/log/ensemble/provision-agent.log " |
1203 | - "--pidfile=/var/run/ensemble/provision-agent.pid") |
1204 | - self.assertEqual(config["runcmd"][-1], provision_agent_script) |
1205 | + expect_path = os.path.join(DATA_DIR, "bootstrap_cloud_init") |
1206 | + with open(expect_path) as f: |
1207 | + expect_cloud_init = f.read() |
1208 | + self.assertEquals(data, expect_cloud_init) |
1209 | return True |
1210 | |
1211 | self.ec2.run_instances( |
1212 | - image_id=MATCH_AMI, |
1213 | + image_id="ami-default", |
1214 | instance_type="m1.small", |
1215 | max_count=1, |
1216 | min_count=1, |
1217 | - security_groups=[ |
1218 | - "%s-%s" % ("ensemble", self.env_name), |
1219 | - "%s-%s-%s" % ("ensemble", self.env_name, machine_id)], |
1220 | + security_groups=["ensemble-moon", "ensemble-moon-0"], |
1221 | user_data=MATCH(verify_user_data)) |
1222 | |
1223 | def test_launch_bootstrap(self): |
1224 | @@ -98,7 +63,7 @@ |
1225 | self._mock_create_group() |
1226 | self._mock_create_machine_group(0) |
1227 | self._mock_launch_utils(region="us-east-1") |
1228 | - self._mock_launch(0) |
1229 | + self._mock_launch() |
1230 | self.mocker.result(succeed([])) |
1231 | self._mock_save() |
1232 | self.mocker.replay() |
1233 | @@ -128,7 +93,7 @@ |
1234 | SecurityGroup("ensemble-%s" % self.env_name, "")])) |
1235 | self._mock_create_machine_group(0) |
1236 | self._mock_launch_utils(region="us-east-1") |
1237 | - self._mock_launch(0) |
1238 | + self._mock_launch() |
1239 | self.mocker.result(succeed([])) |
1240 | self._mock_save() |
1241 | self.mocker.replay() |
1242 | @@ -177,7 +142,7 @@ |
1243 | SecurityGroup("ensemble-%s" % self.env_name, "")])) |
1244 | self._mock_create_machine_group(0) |
1245 | self._mock_launch_utils(region="us-east-1") |
1246 | - self._mock_launch(0) |
1247 | + self._mock_launch() |
1248 | self.mocker.result(succeed([self.get_instance("i-foobar")])) |
1249 | self._mock_save() |
1250 | self.mocker.replay() |
1251 | |
1252 | === modified file 'ensemble/providers/ec2/tests/test_launch.py' |
1253 | --- ensemble/providers/ec2/tests/test_launch.py 2011-08-26 12:24:00 +0000 |
1254 | +++ ensemble/providers/ec2/tests/test_launch.py 2011-09-13 10:56:30 +0000 |
1255 | @@ -1,51 +1,54 @@ |
1256 | -from yaml import load |
1257 | +import os |
1258 | |
1259 | from twisted.internet.defer import inlineCallbacks, succeed |
1260 | |
1261 | from txaws.ec2.model import Instance, SecurityGroup |
1262 | |
1263 | -from ensemble.errors import EnvironmentNotFound, ProviderInteractionError |
1264 | -from ensemble.providers.common.launch import ( |
1265 | - DEFAULT_REPOSITORIES, DEFAULT_PACKAGES) |
1266 | +from ensemble.errors import ( |
1267 | + EnvironmentNotFound, ProviderError, ProviderInteractionError) |
1268 | from ensemble.providers.ec2.machine import EC2ProviderMachine |
1269 | |
1270 | from ensemble.lib.testing import TestCase |
1271 | from ensemble.lib.mocker import MATCH |
1272 | |
1273 | -from .common import EC2TestMixin, EC2MachineLaunchMixin, MATCH_AMI |
1274 | +from .common import EC2TestMixin, EC2MachineLaunchMixin |
1275 | + |
1276 | +DATA_DIR = os.path.join(os.path.abspath(os.path.dirname(__file__)), "data") |
1277 | |
1278 | |
1279 | class EC2MachineLaunchTest(EC2TestMixin, EC2MachineLaunchMixin, TestCase): |
1280 | |
1281 | - def _mock_launch(self, instance=None, custom_verify=None, machine_id=None): |
1282 | + def _mock_launch(self, instance, expect_ami="ami-default", |
1283 | + expect_instance_type="m1.small"): |
1284 | |
1285 | def verify_user_data(data): |
1286 | - lines = data.split("\n") |
1287 | - self.assertEqual(lines.pop(0), "#cloud-config") |
1288 | - config = load("\n".join(lines)) |
1289 | - repos = [dict(source=r) for r in DEFAULT_REPOSITORIES] |
1290 | - self.assertEqual(config["apt_sources"], repos) |
1291 | - self.assertEqual(config["packages"], DEFAULT_PACKAGES) |
1292 | - self.assertEqual(config["machine-data"]["machine-id"], "machine-1") |
1293 | - |
1294 | - if custom_verify: |
1295 | - custom_verify(config) |
1296 | + expect_path = os.path.join(DATA_DIR, "launch_cloud_init") |
1297 | + with open(expect_path) as f: |
1298 | + expect_cloud_init = f.read() |
1299 | + self.assertEquals(data, expect_cloud_init) |
1300 | return True |
1301 | |
1302 | self.ec2.run_instances( |
1303 | - image_id=MATCH_AMI, |
1304 | - instance_type="m1.small", |
1305 | + image_id=expect_ami, |
1306 | + instance_type=expect_instance_type, |
1307 | max_count=1, |
1308 | min_count=1, |
1309 | - security_groups=[ |
1310 | - "%s-%s" % ("ensemble", self.env_name), |
1311 | - "%s-%s-%s" % ("ensemble", self.env_name, machine_id)], |
1312 | + security_groups=["ensemble-moon", "ensemble-moon-1"], |
1313 | user_data=MATCH(verify_user_data)) |
1314 | |
1315 | - if instance: |
1316 | - self.mocker.result(succeed([instance])) |
1317 | - else: |
1318 | - self.mocker.result(succeed([])) |
1319 | + self.mocker.result(succeed([instance])) |
1320 | + |
1321 | + def test_bad_data(self): |
1322 | + self.mocker.replay() |
1323 | + d = self.get_provider().start_machine({}) |
1324 | + self.assertFailure(d, ProviderError) |
1325 | + |
1326 | + def verify(error): |
1327 | + self.assertEquals( |
1328 | + str(error), |
1329 | + "Cannot launch a machine without specifying a machine-id") |
1330 | + d.addCallback(verify) |
1331 | + return d |
1332 | |
1333 | def test_provider_launch(self): |
1334 | """ |
1335 | @@ -55,18 +58,17 @@ |
1336 | self.ec2.describe_security_groups() |
1337 | self.mocker.result(succeed([])) |
1338 | self._mock_create_group() |
1339 | - self._mock_create_machine_group("machine-1") |
1340 | + self._mock_create_machine_group("1") |
1341 | self._mock_launch_utils(region="us-east-1") |
1342 | self._mock_get_zookeeper_hosts() |
1343 | - self._mock_launch( |
1344 | - self.get_instance("i-foobar"), machine_id="machine-1") |
1345 | + self._mock_launch(self.get_instance("i-foobar")) |
1346 | self.mocker.replay() |
1347 | |
1348 | def verify_result(result): |
1349 | (machine,) = result |
1350 | self.assert_machine(machine, "i-foobar", "") |
1351 | provider = self.get_provider() |
1352 | - d = provider.start_machine({"machine-id": "machine-1"}) |
1353 | + d = provider.start_machine({"machine-id": "1"}) |
1354 | d.addCallback(verify_result) |
1355 | return d |
1356 | |
1357 | @@ -78,15 +80,14 @@ |
1358 | |
1359 | self.ec2.describe_security_groups() |
1360 | self.mocker.result(succeed([security_group])) |
1361 | - self._mock_create_machine_group("machine-1") |
1362 | + self._mock_create_machine_group("1") |
1363 | self._mock_launch_utils(region="us-east-1") |
1364 | self._mock_get_zookeeper_hosts() |
1365 | - self._mock_launch(instance, machine_id="machine-1") |
1366 | + self._mock_launch(instance) |
1367 | self.mocker.replay() |
1368 | |
1369 | provider = self.get_provider() |
1370 | - provided_machines = yield provider.start_machine( |
1371 | - {"machine-id": "machine-1"}) |
1372 | + provided_machines = yield provider.start_machine({"machine-id": "1"}) |
1373 | self.assertEqual(len(provided_machines), 1) |
1374 | self.assertTrue(isinstance(provided_machines[0], EC2ProviderMachine)) |
1375 | self.assertEqual( |
1376 | @@ -97,21 +98,20 @@ |
1377 | """Verify that the launch works if the machine security group exists""" |
1378 | instance = Instance("i-foobar", "running", dns_name="x1.example.com") |
1379 | machine_group = SecurityGroup( |
1380 | - "ensemble-moon-machine-1", "some description") |
1381 | + "ensemble-moon-1", "some description") |
1382 | |
1383 | self.ec2.describe_security_groups() |
1384 | self.mocker.result(succeed([machine_group])) |
1385 | self._mock_create_group() |
1386 | - self._mock_delete_machine_group("machine-1") # delete existing sg |
1387 | - self._mock_create_machine_group("machine-1") # then recreate |
1388 | + self._mock_delete_machine_group("1") # delete existing sg |
1389 | + self._mock_create_machine_group("1") # then recreate |
1390 | self._mock_launch_utils(region="us-east-1") |
1391 | self._mock_get_zookeeper_hosts() |
1392 | - self._mock_launch(instance, machine_id="machine-1") |
1393 | + self._mock_launch(instance) |
1394 | self.mocker.replay() |
1395 | |
1396 | provider = self.get_provider() |
1397 | - provided_machines = yield provider.start_machine( |
1398 | - {"machine-id": "machine-1"}) |
1399 | + provided_machines = yield provider.start_machine({"machine-id": "1"}) |
1400 | self.assertEqual(len(provided_machines), 1) |
1401 | self.assertTrue(isinstance(provided_machines[0], EC2ProviderMachine)) |
1402 | self.assertEqual( |
1403 | @@ -125,49 +125,27 @@ |
1404 | that security group, generally because it is still shutting |
1405 | down.""" |
1406 | machine_group = SecurityGroup( |
1407 | - "ensemble-moon-machine-1", "some description") |
1408 | + "ensemble-moon-1", "some description") |
1409 | |
1410 | self.ec2.describe_security_groups() |
1411 | self.mocker.result(succeed([machine_group])) |
1412 | self._mock_create_group() |
1413 | - self._mock_delete_machine_group_was_deleted("machine-1") # sg is gone! |
1414 | + self._mock_delete_machine_group_was_deleted("1") # sg is gone! |
1415 | self._mock_launch_utils(region="us-east-1") |
1416 | self._mock_get_zookeeper_hosts() |
1417 | self.mocker.replay() |
1418 | |
1419 | provider = self.get_provider() |
1420 | ex = yield self.assertFailure( |
1421 | - provider.start_machine({"machine-id": "machine-1"}), |
1422 | + provider.start_machine({"machine-id": "1"}), |
1423 | ProviderInteractionError) |
1424 | self.assertEqual( |
1425 | str(ex), |
1426 | "Unexpected EC2Error deleting security group " |
1427 | - "ensemble-moon-machine-1: There are active instances using " |
1428 | - "security group 'ensemble-moon-machine-1'") |
1429 | - |
1430 | - def test_provider_type_machine_variable(self): |
1431 | - """The provider type is available via cloud-init.""" |
1432 | - self.ec2.describe_security_groups() |
1433 | - self.mocker.result(succeed([])) |
1434 | - self._mock_create_group() |
1435 | - self._mock_create_machine_group("machine-1") |
1436 | - self._mock_launch_utils(region="us-east-1") |
1437 | - self._mock_get_zookeeper_hosts() |
1438 | - |
1439 | - def verify_provider_type(data): |
1440 | - self.assertEqual( |
1441 | - data["machine-data"]["ensemble-provider-type"], "ec2") |
1442 | - |
1443 | - self._mock_launch(custom_verify=verify_provider_type, |
1444 | - machine_id="machine-1") |
1445 | - self.mocker.replay() |
1446 | - |
1447 | - provider = self.get_provider() |
1448 | - d = provider.start_machine({"machine-id": "machine-1"}) |
1449 | - return d |
1450 | + "ensemble-moon-1: There are active instances using security group " |
1451 | + "'ensemble-moon-1'") |
1452 | |
1453 | def test_launch_with_no_ensemble_s3_state(self): |
1454 | - # XXX not really testing anything EC2-specific |
1455 | """ |
1456 | Attempting to launch without any ensemble saved state, means |
1457 | we can't provide a way for a launched instance to connect |
1458 | @@ -178,12 +156,11 @@ |
1459 | self.mocker.replay() |
1460 | |
1461 | provider = self.get_provider() |
1462 | - d = provider.start_machine({"machine-id": "machine-1"}) |
1463 | + d = provider.start_machine({"machine-id": "1"}) |
1464 | self.assertFailure(d, EnvironmentNotFound) |
1465 | return d |
1466 | |
1467 | def test_launch_with_no_ensemble_zookeeper_hosts(self): |
1468 | - # XXX not really testing anything EC2-specific |
1469 | """ |
1470 | Attempting to launch without any ensemble zookeeper hosts, means |
1471 | we can't provide a way for a launched instance to connect |
1472 | @@ -194,55 +171,42 @@ |
1473 | self.mocker.replay() |
1474 | |
1475 | provider = self.get_provider() |
1476 | - d = provider.start_machine({"machine-id": "machine-1"}) |
1477 | + d = provider.start_machine({"machine-id": "1"}) |
1478 | self.assertFailure(d, EnvironmentNotFound) |
1479 | return d |
1480 | |
1481 | - def test_launch_starts_machine_agent(self): |
1482 | - # XXX not really testing anything EC2-specific |
1483 | - """A launch'd machine should have a machine agent.""" |
1484 | + def test_launch_options_known_instance_type(self): |
1485 | self.ec2.describe_security_groups() |
1486 | self.mocker.result(succeed([])) |
1487 | self._mock_create_group() |
1488 | - self._mock_create_machine_group("machine-1") |
1489 | + self._mock_create_machine_group("1") |
1490 | self._mock_launch_utils(region="us-east-1") |
1491 | self._mock_get_zookeeper_hosts() |
1492 | - |
1493 | - def verify_machine_agent(data): |
1494 | - script = ("ENSEMBLE_MACHINE_ID=machine-1 " |
1495 | - "ENSEMBLE_ZOOKEEPER=es.example.internal:2181 " |
1496 | - "python -m ensemble.agents.machine " |
1497 | - "-n --logfile=/var/log/ensemble/machine-agent.log " |
1498 | - "--pidfile=/var/run/ensemble/machine-agent.pid") |
1499 | - self.assertEqual(script, data["runcmd"][-1]) |
1500 | - |
1501 | - self._mock_launch(custom_verify=verify_machine_agent, |
1502 | - machine_id="machine-1") |
1503 | - self.mocker.replay() |
1504 | - |
1505 | - provider = self.get_provider() |
1506 | - d = provider.start_machine({"machine-id": "machine-1"}) |
1507 | - return d |
1508 | - |
1509 | - def _mock_launch_with_ami_params(self, get_ami_kwargs): |
1510 | - self.ec2.describe_security_groups() |
1511 | - self.mocker.result(succeed([])) |
1512 | - self._mock_create_group() |
1513 | - self._mock_create_machine_group("machine-1") |
1514 | - self._mock_launch_utils(**get_ami_kwargs) |
1515 | - self._mock_get_zookeeper_hosts() |
1516 | self._mock_launch( |
1517 | - self.get_instance("i-foobar"), machine_id="machine-1") |
1518 | + self.get_instance("i-foobar"), expect_instance_type="m1.ginormous") |
1519 | + self.mocker.replay() |
1520 | + |
1521 | + provider = self.get_provider() |
1522 | + provider.config["default-instance-type"] = "m1.ginormous" |
1523 | + return provider.start_machine({"machine-id": "1"}) |
1524 | + |
1525 | + def _mock_launch_with_ami_params(self, get_ami_kwargs, expect_ami=None): |
1526 | + expect_ami = expect_ami or "ami-default" |
1527 | + self.ec2.describe_security_groups() |
1528 | + self.mocker.result(succeed([])) |
1529 | + self._mock_create_group() |
1530 | + self._mock_create_machine_group("1") |
1531 | + self._mock_launch_utils(ami_name=expect_ami, **get_ami_kwargs) |
1532 | + self._mock_get_zookeeper_hosts() |
1533 | + self._mock_launch(self.get_instance("i-foobar"), expect_ami) |
1534 | |
1535 | def test_launch_options_known_ami(self): |
1536 | - self._mock_launch_with_ami_params({}) |
1537 | + self._mock_launch_with_ami_params({}, expect_ami="ami-different") |
1538 | self.mocker.replay() |
1539 | |
1540 | provider = self.get_provider() |
1541 | - # TODO shouldn't something be testing we actually launch |
1542 | - # an instance of the expected AMI? |
1543 | - provider.config["default-image-id"] = "ami-XXXXXX" |
1544 | - return provider.start_machine({"machine-id": "machine-1"}) |
1545 | + provider.config["default-image-id"] = "ami-different" |
1546 | + return provider.start_machine({"machine-id": "1"}) |
1547 | |
1548 | def test_launch_options_region(self): |
1549 | self._mock_launch_with_ami_params({"region": "somewhere-else-1"}) |
1550 | @@ -250,21 +214,22 @@ |
1551 | |
1552 | provider = self.get_provider() |
1553 | provider.config["region"] = "somewhere-else-1" |
1554 | - return provider.start_machine({"machine-id": "machine-1"}) |
1555 | + return provider.start_machine({"machine-id": "1"}) |
1556 | |
1557 | - def test_launch_options_other(self): |
1558 | + def test_launch_options_custom_image_options(self): |
1559 | self._mock_launch_with_ami_params({ |
1560 | "region": "us-east-1", |
1561 | - "ubuntu_release_name": "choleric", |
1562 | - "arch": "quantum86", |
1563 | - "ebs": "maybe", |
1564 | + "ubuntu_release": "choleric", |
1565 | + "architecture": "quantum86", |
1566 | + "persistent_storage": "maybe", |
1567 | "daily": "perhaps"}) |
1568 | self.mocker.replay() |
1569 | |
1570 | provider = self.get_provider() |
1571 | - return provider.start_machine({ |
1572 | - "machine-id": "machine-1", |
1573 | - "image_release_name": "choleric", |
1574 | - "image_arch": "quantum86", |
1575 | - "image_ebs": "maybe", |
1576 | - "image_daily": "perhaps"}) |
1577 | + constraints = { |
1578 | + "ubuntu_release": "choleric", |
1579 | + "architecture": "quantum86", |
1580 | + "persistent_storage": "maybe", |
1581 | + "daily": "perhaps"} |
1582 | + return provider.start_machine({"machine-id": "1", |
1583 | + "constraints": constraints}) |
1584 | |
1585 | === modified file 'ensemble/providers/ec2/tests/test_utils.py' |
1586 | --- ensemble/providers/ec2/tests/test_utils.py 2011-08-12 19:54:54 +0000 |
1587 | +++ ensemble/providers/ec2/tests/test_utils.py 2011-09-13 10:56:30 +0000 |
1588 | @@ -1,14 +1,12 @@ |
1589 | import inspect |
1590 | import os |
1591 | |
1592 | - |
1593 | from twisted.internet.defer import succeed |
1594 | |
1595 | +from ensemble.lib.testing import TestCase |
1596 | from ensemble.providers import ec2 |
1597 | -from ensemble.providers.common.utils import format_cloud_init |
1598 | -from ensemble.providers.ec2.utils import get_current_ami, get_launch_options |
1599 | +from ensemble.providers.ec2.utils import get_current_ami, get_image_id |
1600 | |
1601 | -from ensemble.lib.testing import TestCase |
1602 | |
1603 | IMAGE_URI_TEMPLATE = "\ |
1604 | http://uec-images.ubuntu.com/query/%s/server/released.current.txt" |
1605 | @@ -17,7 +15,7 @@ |
1606 | os.path.dirname(inspect.getabsfile(ec2)), "tests", "data") |
1607 | |
1608 | |
1609 | -class EC2UtilsTest(TestCase): |
1610 | +class GetCurrentAmiTest(TestCase): |
1611 | |
1612 | def test_umatched_ami(self): |
1613 | """ |
1614 | @@ -28,7 +26,7 @@ |
1615 | page(IMAGE_URI_TEMPLATE % "lucid") |
1616 | self.mocker.result(succeed("")) |
1617 | self.mocker.replay() |
1618 | - d = get_current_ami(ubuntu_release_name="lucid") |
1619 | + d = get_current_ami(ubuntu_release="lucid") |
1620 | self.failUnlessFailure(d, LookupError) |
1621 | return d |
1622 | |
1623 | @@ -40,7 +38,7 @@ |
1624 | open(os.path.join(IMAGE_DATA_DIR, "lucid.txt")).read())) |
1625 | |
1626 | self.mocker.replay() |
1627 | - d = get_current_ami(ubuntu_release_name="lucid") |
1628 | + d = get_current_ami(ubuntu_release="lucid") |
1629 | |
1630 | def verify_result(result): |
1631 | self.assertEqual(result, "ami-714ba518") |
1632 | @@ -56,7 +54,7 @@ |
1633 | succeed(open( |
1634 | os.path.join(IMAGE_DATA_DIR, "lucid.txt")).read())) |
1635 | self.mocker.replay() |
1636 | - d = get_current_ami(ubuntu_release_name="lucid", region="us-west-1") |
1637 | + d = get_current_ami(ubuntu_release="lucid", region="us-west-1") |
1638 | |
1639 | def verify_result(result): |
1640 | self.assertEqual(result, "ami-cb97c68e") |
1641 | @@ -74,7 +72,7 @@ |
1642 | self.mocker.result(succeed( |
1643 | open(os.path.join(IMAGE_DATA_DIR, "lucid.txt")).read())) |
1644 | self.mocker.replay() |
1645 | - d = get_current_ami(ubuntu_release_name="lucid", ebs=False) |
1646 | + d = get_current_ami(ubuntu_release="lucid", persistent_storage=False) |
1647 | |
1648 | def verify_result(result): |
1649 | self.assertEqual(result, "ami-2d4aa444") |
1650 | @@ -82,101 +80,37 @@ |
1651 | d.addCallback(verify_result) |
1652 | return d |
1653 | |
1654 | - def test_get_launch_options(self): |
1655 | - """ |
1656 | - F{get_launch_options}, returns a dictionary of arguments for the ec2 |
1657 | - client's run instance method. |
1658 | - """ |
1659 | - config = {} |
1660 | - config["default-image-id"] = "ami-foobar" |
1661 | - config["default-instance-type"] = "m1.medium" |
1662 | - authorized_keys = "zebra" |
1663 | - repositories = ["ppa:ensemble/ppa"] |
1664 | - packages = ["python-zookeeper", "zookeeperd"] |
1665 | - |
1666 | - d = get_launch_options( |
1667 | - config, |
1668 | - authorized_keys=authorized_keys, |
1669 | - packages=packages, |
1670 | - repositories=repositories, |
1671 | - magic=[1, 2, 3]) |
1672 | - |
1673 | - cloud_init_config = format_cloud_init( |
1674 | - authorized_keys, packages, repositories, |
1675 | - data={"magic": [1, 2, 3]}) |
1676 | - |
1677 | - def verify_result(result): |
1678 | - self.assertTrue(isinstance(result, dict)) |
1679 | - self.assertEqual(result["image_id"], "ami-foobar") |
1680 | - self.assertEqual(result["user_data"], cloud_init_config) |
1681 | - self.assertEqual(result["security_groups"], []) |
1682 | - |
1683 | - d.addCallback(verify_result) |
1684 | - return d |
1685 | - |
1686 | - def test_get_launch_options_defaults(self): |
1687 | - """ |
1688 | - F{get_launch_options}, returns a dictionary of arguments for |
1689 | - the ec2 client's run instance method. A known set of default |
1690 | - is utilized. |
1691 | - """ |
1692 | - |
1693 | - page = self.mocker.replace("twisted.web.client.getPage") |
1694 | - page(IMAGE_URI_TEMPLATE % "natty") |
1695 | - self.mocker.result( |
1696 | - succeed(open( |
1697 | - os.path.join(IMAGE_DATA_DIR, "natty.txt")).read())) |
1698 | - self.mocker.replay() |
1699 | - |
1700 | - config = {} |
1701 | - d = get_launch_options(config, "zebra") |
1702 | - |
1703 | - def verify_result(result): |
1704 | - self.assertTrue(isinstance(result, dict)) |
1705 | - self.assertEqual(result["image_id"], "ami-06ad526f") |
1706 | - self.assertEqual(result["instance_type"], "m1.small") |
1707 | - |
1708 | - d.addCallback(verify_result) |
1709 | - return d |
1710 | - |
1711 | - def test_get_launch_options_with_params(self): |
1712 | - """ |
1713 | - Image parameters can be specified to determine which ami is to |
1714 | - be launched. Image paramters specified are not passed to the |
1715 | - instance via user data. |
1716 | - """ |
1717 | - page = self.mocker.replace("twisted.web.client.getPage") |
1718 | - page(IMAGE_URI_TEMPLATE % "natty") |
1719 | - self.mocker.result( |
1720 | - succeed(open( |
1721 | - os.path.join(IMAGE_DATA_DIR, "natty.txt")).read())) |
1722 | - self.mocker.replay() |
1723 | - |
1724 | - d = get_launch_options( |
1725 | - {"default-instance-type": "m1.large"}, |
1726 | - "zebra", image_arch="amd64", image_ebs=False) |
1727 | - |
1728 | - def verify_result(result): |
1729 | - self.assertTrue(isinstance(result, dict)) |
1730 | - self.assertEqual(result["image_id"], "ami-68ad5201") |
1731 | - self.assertEqual(result["instance_type"], "m1.large") |
1732 | - self.assertNotIn( |
1733 | - "arch", result["user_data"]) |
1734 | - self.assertNotIn( |
1735 | - "ebs", result["user_data"]) |
1736 | - |
1737 | - d.addCallback(verify_result) |
1738 | - return d |
1739 | - |
1740 | - def test_configured_default_image_has_precedence_over_region(self): |
1741 | - """If the server machine image is specified it takes precendence over |
1742 | - region.""" |
1743 | - d = get_launch_options( |
1744 | - {"region": "eu-west-1", "default-image-id": "21down"}, |
1745 | - authorized_keys="") |
1746 | - |
1747 | - def verify_result(result): |
1748 | - self.assertEqual(result["image_id"], "21down") |
1749 | - |
1750 | - d.addCallback(verify_result) |
1751 | + |
1752 | +class GetImageIdTest(TestCase): |
1753 | + |
1754 | + def test_default_image_id(self): |
1755 | + d = get_image_id({"default-image-id": "ami-burble"}, {}) |
1756 | + d.addCallback(self.assertEquals, "ami-burble") |
1757 | + return d |
1758 | + |
1759 | + def test_no_constraints(self): |
1760 | + get_current_ami_m = self.mocker.replace(get_current_ami) |
1761 | + get_current_ami_m(region="us-east-1") |
1762 | + self.mocker.result(succeed("ami-giggle")) |
1763 | + self.mocker.replay() |
1764 | + |
1765 | + d = get_image_id({}, {}) |
1766 | + d.addCallback(self.assertEquals, "ami-giggle") |
1767 | + return d |
1768 | + |
1769 | + def test_uses_constraints(self): |
1770 | + get_current_ami_m = self.mocker.replace(get_current_ami) |
1771 | + get_current_ami_m(ubuntu_release="serendipitous", architecture="x512", |
1772 | + daily=False, persistent_storage=True, |
1773 | + region="blah-north-6") |
1774 | + self.mocker.result(succeed("ami-tinkle")) |
1775 | + self.mocker.replay() |
1776 | + |
1777 | + constraints = { |
1778 | + "architecture": "x512", |
1779 | + "ubuntu_release": "serendipitous", |
1780 | + "persistent_storage": True, |
1781 | + "daily": False} |
1782 | + d = get_image_id({"region": "blah-north-6"}, constraints) |
1783 | + d.addCallback(self.assertEquals, "ami-tinkle") |
1784 | return d |
1785 | |
1786 | === modified file 'ensemble/providers/ec2/utils.py' |
1787 | --- ensemble/providers/ec2/utils.py 2011-08-14 19:24:03 +0000 |
1788 | +++ ensemble/providers/ec2/utils.py 2011-09-13 10:56:30 +0000 |
1789 | @@ -5,22 +5,11 @@ |
1790 | from twisted.web.client import getPage |
1791 | from twisted.internet.defer import succeed |
1792 | |
1793 | -from ensemble.providers.common.utils import format_cloud_init |
1794 | - |
1795 | log = logging.getLogger("ensemble.ec2") |
1796 | |
1797 | -# default image for each region. |
1798 | -AMI_REGION_MAP = { |
1799 | - "us-east-1": "ami-d2ba45bb", |
1800 | - "us-west-1": "ami-15693a50", |
1801 | - "ap-northeast-1": "ami-80bb1181", |
1802 | - "ap-southeast-1": "ami-b4671ee6", |
1803 | - "eu-west-1": "ami-0991a67d" |
1804 | - } |
1805 | - |
1806 | -BASE_IMAGE_URI = "http://uec-images.ubuntu.com/query/" |
1807 | -CURRENT_IMAGE_URI = BASE_IMAGE_URI + \ |
1808 | - "%(ubuntu_release_name)s/%(variant)s/%(version)s.current.txt" |
1809 | +_CURRENT_IMAGE_URI_TEMPLATE = ( |
1810 | + "http://uec-images.ubuntu.com/query/" |
1811 | + "%(ubuntu_release_name)s/%(variant)s/%(version)s.current.txt") |
1812 | |
1813 | |
1814 | def get_region_uri(region): |
1815 | @@ -28,17 +17,18 @@ |
1816 | return "https://ec2.%s.amazonaws.com" % region |
1817 | |
1818 | |
1819 | -def get_current_ami(ubuntu_release_name="natty", arch="i386", ebs=True, |
1820 | - region="us-east-1", daily=False, desktop=False, |
1821 | - url_fetch=None): |
1822 | +# XXX ideally should come from latest available or client release name. |
1823 | +def get_current_ami(ubuntu_release="natty", architecture="i386", |
1824 | + persistent_storage=True, region="us-east-1", daily=False, |
1825 | + desktop=False, url_fetch=None): |
1826 | """Get the latest ami for the last release of ubuntu.""" |
1827 | data = {} |
1828 | - data["ubuntu_release_name"] = ubuntu_release_name |
1829 | + data["ubuntu_release_name"] = ubuntu_release |
1830 | data["version"] = daily and "daily" or "released" |
1831 | data["variant"] = desktop and "desktop" or "server" |
1832 | - ebs_match = ebs and "ebs" or "instance-store" |
1833 | + ebs_match = persistent_storage and "ebs" or "instance-store" |
1834 | |
1835 | - url = CURRENT_IMAGE_URI % data |
1836 | + url = _CURRENT_IMAGE_URI_TEMPLATE % data |
1837 | url_fetch = url_fetch or getPage |
1838 | |
1839 | d = url_fetch(url) |
1840 | @@ -48,60 +38,18 @@ |
1841 | for tokens in csv.reader(data_stream, "excel-tab"): |
1842 | if tokens[4] != ebs_match: |
1843 | continue |
1844 | - if tokens[5] == arch and tokens[6] == region: |
1845 | + if tokens[5] == architecture and tokens[6] == region: |
1846 | return tokens[7] |
1847 | - raise LookupError((ubuntu_release_name, arch, region, |
1848 | + raise LookupError((ubuntu_release, architecture, region, |
1849 | data["version"], data["variant"])) |
1850 | |
1851 | d.addCallback(extract_ami) |
1852 | return d |
1853 | |
1854 | |
1855 | -def get_launch_options(config, authorized_keys, packages=None, |
1856 | - repositories=None, scripts=None, **kw): |
1857 | - """Determine launch options for the machine. |
1858 | - |
1859 | - Given an ensemble environment configuration, extract the proper machine |
1860 | - information to be used for launching the machine via the ec2 api. |
1861 | - """ |
1862 | - |
1863 | - instance_type = config.get("default-instance-type", "m1.small") |
1864 | - |
1865 | - # XXX ideally should come from latest available or client release name. |
1866 | +def get_image_id(config, constraints): |
1867 | + image_id = config.get("default-image-id", None) |
1868 | + if image_id: |
1869 | + return succeed(image_id) |
1870 | region = config.get("region", "us-east-1") |
1871 | - image_id = config.get("default-image-id", None) |
1872 | - |
1873 | - # Lookup a standard ami by region, if none is specified, use any |
1874 | - # specified image specification to locate an image, and remove the |
1875 | - # image specification so it doesn't propogate to user data. |
1876 | - if not image_id: |
1877 | - params = {} |
1878 | - for a_name, p_name in ( |
1879 | - ("image_release_name", "ubuntu_release_name"), |
1880 | - ("image_arch", "arch"), |
1881 | - ("image_ebs", "ebs"), |
1882 | - ("image_daily", "daily")): |
1883 | - if a_name in kw: |
1884 | - params[p_name] = kw[a_name] |
1885 | - del kw[a_name] |
1886 | - params["region"] = region |
1887 | - d = get_current_ami(**params) |
1888 | - else: |
1889 | - d = succeed(image_id) |
1890 | - |
1891 | - user_data = format_cloud_init( |
1892 | - authorized_keys=authorized_keys, |
1893 | - packages=packages, |
1894 | - repositories=repositories, |
1895 | - scripts=scripts, |
1896 | - data=kw) |
1897 | - |
1898 | - def on_machine_image(current_image_id): |
1899 | - return {"image_id": current_image_id, |
1900 | - "min_count": 1, |
1901 | - "max_count": 1, |
1902 | - "instance_type": instance_type, |
1903 | - "user_data": user_data, |
1904 | - "security_groups": []} |
1905 | - d.addCallback(on_machine_image) |
1906 | - return d |
1907 | + return get_current_ami(region=region, **constraints) |
1908 | |
1909 | === modified file 'ensemble/providers/orchestra/__init__.py' |
1910 | --- ensemble/providers/orchestra/__init__.py 2011-08-25 16:41:03 +0000 |
1911 | +++ ensemble/providers/orchestra/__init__.py 2011-09-13 10:56:30 +0000 |
1912 | @@ -1,6 +1,6 @@ |
1913 | import logging |
1914 | |
1915 | -from twisted.internet.defer import inlineCallbacks, returnValue, succeed |
1916 | +from twisted.internet.defer import fail, inlineCallbacks, returnValue, succeed |
1917 | |
1918 | from ensemble.errors import ProviderError |
1919 | from ensemble.providers.common.base import MachineProviderBase |
1920 | @@ -29,13 +29,16 @@ |
1921 | def start_machine(self, machine_data, master=False): |
1922 | """Start a machine in the provider. |
1923 | |
1924 | - @param machine_data: a dictionary of data to pass along to the newly |
1925 | - launched machine. |
1926 | + `machine_data`: a dictionary describing the machine to be launched. |
1927 | |
1928 | - @param master: if True, machine will initialize the ensemble admin |
1929 | - and run a provisioning agent. |
1930 | + `master`: if True, machine will initialize the ensemble admin and run |
1931 | + a provisioning agent, in addition to running a machine agent |
1932 | """ |
1933 | - return OrchestraLaunchMachine(self, master).run(machine_data) |
1934 | + if "machine-id" not in machine_data: |
1935 | + return fail(ProviderError( |
1936 | + "Cannot launch a machine without specifying a machine-id")) |
1937 | + machine_id = machine_data["machine-id"] |
1938 | + return OrchestraLaunchMachine(self, master).run(machine_id) |
1939 | |
1940 | @inlineCallbacks |
1941 | def get_machines(self, instance_ids=()): |
1942 | |
1943 | === modified file 'ensemble/providers/orchestra/launch.py' |
1944 | --- ensemble/providers/orchestra/launch.py 2011-08-11 23:27:45 +0000 |
1945 | +++ ensemble/providers/orchestra/launch.py 2011-09-13 10:56:30 +0000 |
1946 | @@ -5,14 +5,8 @@ |
1947 | from twisted.internet.defer import inlineCallbacks, returnValue |
1948 | |
1949 | from ensemble.providers.common.launch import LaunchMachine |
1950 | -from ensemble.providers.common.utils import format_cloud_init |
1951 | -from ensemble.providers.orchestra.machine import machine_from_dict |
1952 | - |
1953 | -_INSTANCE_ID_VAR = "ENSEMBLE_INSTANCE_ID" |
1954 | - |
1955 | -_SET_INSTANCE_ID_TEMPLATE = "export %s=%%s" % _INSTANCE_ID_VAR |
1956 | - |
1957 | -_CLOUD_INIT_KEYS = "authorized_keys packages repositories scripts".split() |
1958 | + |
1959 | +from .machine import machine_from_dict |
1960 | |
1961 | _LATE_COMMAND_TEMPLATE = """ |
1962 | seed_d=/var/lib/cloud/seed/nocloud-net |
1963 | @@ -36,17 +30,6 @@ |
1964 | "' %s /root/late-command") |
1965 | |
1966 | |
1967 | -def _cloud_init_vars(raw_vars): |
1968 | - # XXX this is nasty; see bug#820892 |
1969 | - vars = raw_vars.copy() |
1970 | - data = {} |
1971 | - for key in vars.keys(): |
1972 | - if key not in _CLOUD_INIT_KEYS: |
1973 | - data[key] = vars.pop(key) |
1974 | - vars["data"] = data |
1975 | - return vars |
1976 | - |
1977 | - |
1978 | def _base64_gzip(content): |
1979 | gzipped = StringIO() |
1980 | gzip_file = GzipFile(fileobj=gzipped, mode="wb", compresslevel=9) |
1981 | @@ -62,28 +45,34 @@ |
1982 | class OrchestraLaunchMachine(LaunchMachine): |
1983 | |
1984 | @inlineCallbacks |
1985 | - def start_machine(self, variables, machine_id): |
1986 | + def start_machine(self, machine_id, zookeepers): |
1987 | + """Actually launch a Cobbler system. |
1988 | + |
1989 | + `machine_id`: the ensemble machine ID |
1990 | + |
1991 | + `zookeepers`: a list of `ProviderMachine`s already running zookeeper, |
1992 | + which the machine will need to connect to. May be empty. |
1993 | + |
1994 | + Returns a singe-entry list containing a ProviderMachine for the |
1995 | + new instance |
1996 | + """ |
1997 | cobbler = self._provider.cobbler |
1998 | instance_id = yield cobbler.acquire_system() |
1999 | - ks_meta = self._build_ks_meta(instance_id, variables) |
2000 | + |
2001 | + cloud_init = self._create_cloud_init(machine_id, zookeepers) |
2002 | + cloud_init.set_provider_type("orchestra") |
2003 | + cloud_init.set_instance_id_accessor(instance_id) |
2004 | + ks_meta = self._build_ks_meta( |
2005 | + cloud_init.render(), machine_id, instance_id) |
2006 | + |
2007 | yield cobbler.set_on_system(instance_id, "ks_meta", ks_meta) |
2008 | yield cobbler.start_system(instance_id) |
2009 | (info,) = yield cobbler.describe_systems(instance_id) |
2010 | returnValue([machine_from_dict(info)]) |
2011 | |
2012 | - def get_instance_id_command(self): |
2013 | - return "$" + _INSTANCE_ID_VAR |
2014 | - |
2015 | - def _build_ks_meta(self, instance_id, variables): |
2016 | - # XXX it's a little bit dirty to modify scripts at this stage; |
2017 | - # but we don't actually know the instance_id at get_variables time. |
2018 | - set_instance_id = _SET_INSTANCE_ID_TEMPLATE % instance_id |
2019 | - variables["scripts"].insert(0, set_instance_id) |
2020 | - |
2021 | - cloud_init_vars = _cloud_init_vars(variables) |
2022 | - user_data = format_cloud_init(**cloud_init_vars) |
2023 | + def _build_ks_meta(self, user_data, machine_id, instance_id): |
2024 | late_command = _LATE_COMMAND_TEMPLATE % (instance_id, user_data) |
2025 | encoded = _base64_gzip(late_command) |
2026 | ensemble_late_command = _KSMETA_LATE_COMMAND_TEMPLATE % encoded |
2027 | - return _ks_meta(MACHINE_ID=variables["machine-id"], |
2028 | - ENSEMBLE_LATE_COMMAND=ensemble_late_command) |
2029 | + return _ks_meta( |
2030 | + MACHINE_ID=machine_id, ENSEMBLE_LATE_COMMAND=ensemble_late_command) |
2031 | |
2032 | === modified file 'ensemble/providers/orchestra/tests/common.py' |
2033 | --- ensemble/providers/orchestra/tests/common.py 2011-08-11 23:27:45 +0000 |
2034 | +++ ensemble/providers/orchestra/tests/common.py 2011-09-13 10:56:30 +0000 |
2035 | @@ -108,7 +108,8 @@ |
2036 | late_command = _extract_late_command(leftover) |
2037 | |
2038 | expect_path = os.path.join(DATA_DIR, late_command_filename) |
2039 | - expect_late_command = open(expect_path).read() |
2040 | + with open(expect_path) as f: |
2041 | + expect_late_command = f.read() |
2042 | self.assertEquals(late_command, expect_late_command) |
2043 | return True |
2044 | return verify |
2045 | |
2046 | === modified file 'ensemble/providers/orchestra/tests/data/bootstrap_late_command' |
2047 | --- ensemble/providers/orchestra/tests/data/bootstrap_late_command 2011-09-07 22:06:57 +0000 |
2048 | +++ ensemble/providers/orchestra/tests/data/bootstrap_late_command 2011-09-13 10:56:30 +0000 |
2049 | @@ -11,13 +11,13 @@ |
2050 | apt-upgrade: true |
2051 | apt_sources: |
2052 | - {source: 'ppa:ensemble/ppa'} |
2053 | -machine-data: {ensemble-zookeeper-hosts: 'localhost:2181', machine-id: '0'} |
2054 | +machine-data: {ensemble-provider-type: orchestra, ensemble-zookeeper-hosts: 'localhost:2181', |
2055 | + machine-id: '0'} |
2056 | output: {all: '| tee -a /var/log/cloud-init-output.log'} |
2057 | packages: [bzr, byobu, tmux, python-setuptools, python-twisted, python-argparse, python-txaws, |
2058 | - python-zookeeper, bzr, default-jre-headless, zookeeper, zookeeperd] |
2059 | -runcmd: [export ENSEMBLE_INSTANCE_ID=winston-uid, sudo apt-get -y install ensemble, |
2060 | - sudo mkdir -p /var/lib/ensemble, sudo mkdir -p /var/log/ensemble, 'ensemble-admin |
2061 | - initialize --instance-id=$ENSEMBLE_INSTANCE_ID --admin-identity="admin:qRBXC1ubEEUqRL6wcBhgmc9xkaY="', |
2062 | + python-zookeeper, default-jre-headless, zookeeper, zookeeperd] |
2063 | +runcmd: [sudo apt-get -y install ensemble, sudo mkdir -p /var/lib/ensemble, sudo mkdir |
2064 | + -p /var/log/ensemble, 'ensemble-admin initialize --instance-id=winston-uid --admin-identity=admin:qRBXC1ubEEUqRL6wcBhgmc9xkaY=', |
2065 | 'ENSEMBLE_MACHINE_ID=0 ENSEMBLE_ZOOKEEPER=localhost:2181 python -m ensemble.agents.machine |
2066 | -n --logfile=/var/log/ensemble/machine-agent.log --pidfile=/var/run/ensemble/machine-agent.pid', |
2067 | 'ENSEMBLE_ZOOKEEPER=localhost:2181 python -m ensemble.agents.provision -n --logfile=/var/log/ensemble/provision-agent.log |
2068 | |
2069 | === modified file 'ensemble/providers/orchestra/tests/data/launch_late_command' |
2070 | --- ensemble/providers/orchestra/tests/data/launch_late_command 2011-09-07 22:06:57 +0000 |
2071 | +++ ensemble/providers/orchestra/tests/data/launch_late_command 2011-09-13 10:56:30 +0000 |
2072 | @@ -11,13 +11,14 @@ |
2073 | apt-upgrade: true |
2074 | apt_sources: |
2075 | - {source: 'ppa:ensemble/ppa'} |
2076 | -machine-data: {ensemble-zookeeper-hosts: 'jennifer:2181', machine-id: '42'} |
2077 | +machine-data: {ensemble-provider-type: orchestra, ensemble-zookeeper-hosts: 'jennifer:2181', |
2078 | + machine-id: '42'} |
2079 | output: {all: '| tee -a /var/log/cloud-init-output.log'} |
2080 | packages: [bzr, byobu, tmux, python-setuptools, python-twisted, python-argparse, python-txaws, |
2081 | python-zookeeper] |
2082 | -runcmd: [export ENSEMBLE_INSTANCE_ID=winston-uid, sudo apt-get -y install ensemble, |
2083 | - sudo mkdir -p /var/lib/ensemble, sudo mkdir -p /var/log/ensemble, 'ENSEMBLE_MACHINE_ID=42 |
2084 | - ENSEMBLE_ZOOKEEPER=jennifer:2181 python -m ensemble.agents.machine -n --logfile=/var/log/ensemble/machine-agent.log |
2085 | +runcmd: [sudo apt-get -y install ensemble, sudo mkdir -p /var/lib/ensemble, sudo mkdir |
2086 | + -p /var/log/ensemble, 'ENSEMBLE_MACHINE_ID=42 ENSEMBLE_ZOOKEEPER=jennifer:2181 |
2087 | + python -m ensemble.agents.machine -n --logfile=/var/log/ensemble/machine-agent.log |
2088 | --pidfile=/var/run/ensemble/machine-agent.pid'] |
2089 | ssh_authorized_keys: [this-is-a-public-key] |
2090 | |
2091 | |
2092 | === modified file 'ensemble/providers/orchestra/tests/test_launch.py' |
2093 | --- ensemble/providers/orchestra/tests/test_launch.py 2011-08-26 12:24:00 +0000 |
2094 | +++ ensemble/providers/orchestra/tests/test_launch.py 2011-09-13 10:56:30 +0000 |
2095 | @@ -11,6 +11,12 @@ |
2096 | def test_bad_data(self): |
2097 | d = self.get_provider().start_machine({}) |
2098 | self.assertFailure(d, ProviderError) |
2099 | + |
2100 | + def verify(error): |
2101 | + self.assertEquals( |
2102 | + str(error), |
2103 | + "Cannot launch a machine without specifying a machine-id") |
2104 | + d.addCallback(verify) |
2105 | return d |
2106 | |
2107 | def test_no_zookeeper(self): |
This looks _very_ nice. Thanks William!
+1, with the following sorted:
[1]
+ self._zookeeper _names = [m.private_dns_name for m in machines] hosts(self) : _names[ :]
(...)
+ def _zookeeper_
+ all_names = self._zookeeper
The branch has two different names for the same data in different
formats. The better approach is the opposite: the same name with
different actions that give a hint about the undelying operation.
I suggest using "zookeeper_hosts" for both, and naming the last hosts or something similar.
function as join_zookeeper_
[2]
+ def _validate(self):
+ missing = []
Very nice logic there.
[3]
+ def _create_ cloud_init( self, machine_id, zookeepers): config add_ssh_ key(get_ user_authorized _keys(config) ) set_machine_ id(machine_ id) set_zookeeper_ machines( zookeepers) get("ensemble- branch" ): set_ensemble_ source( config[ "ensemble- branch" ]) enable_ bootstrap( ) set_zookeeper_ secret( config[ "admin- secret" ])
+ config = self._provider.
+ cloud_init = CloudInit()
+ cloud_init.
+ cloud_init.
+ cloud_init.
+ if config.
+ cloud_init.
+ if self._master:
+ cloud_init.
+ cloud_init.
+ return cloud_init
Beautiful! :-)
[4]
+ instance_type = self._provider. config. get( instance- type", "m1.small") id(self. _provider. config, self._constraints) groups( machine_ id) ec2.run_ instances( type=instance_ type, groups= security_ groups, user_data)
+ "default-
+ image_id = yield get_image_
+ security_groups = yield self._ensure_
+
+ instances = yield self._provider.
+ image_id=image_id,
+ instance_
+ min_count=1,
+ max_count=1,
+ security_
+ user_data=
Very clean as well, thanks!
[5]
+def get_image_ id(config, constraints): release_ name", storage" : "ebs"} constraints( constraints) : NAMES.items( ): constraint_ name] constraints( constraints)
(...)
+_KWARG_NAMES = {
+ "ubuntu_release": "ubuntu_
+ "architecture": "arch",
+ "daily": "daily",
+ "persistent_
(...)
+def _convert_
+ kwargs = {}
+ for (constraint_name, kwarg_name) in _KWARG_
+ if constraint_name in constraints:
+ kwargs[kwarg_name] = constraints[
+ return kwargs
(...)
+ kwargs = _convert_
Why are we doing this? These names in the right-hand-side of the
_KWARG_NAMES are keyword arguments of get_current_ami, which is a function
on the same file (!). The left-hand-side ones are not even used in
any sensible way in our current implementation.
Let's just unify the naming and avoid any conversions entirely. Feel
free to pick sensible ones.
[6]
Again, this looks very nice. Please just make sure you merge trunk into
this, and do one final real-world test on EC2 before committing back
into trunk.