Merge lp:~fwereade/pyjuju/cloud-init-class-used into lp:pyjuju
- cloud-init-class-used
- Merge into trunk
Status: | Merged | ||||
---|---|---|---|---|---|
Approved by: | Gustavo Niemeyer | ||||
Approved revision: | 341 | ||||
Merged at revision: | 342 | ||||
Proposed branch: | lp:~fwereade/pyjuju/cloud-init-class-used | ||||
Merge into: | lp:pyjuju | ||||
Prerequisite: | lp:~fwereade/pyjuju/cloud-init-class-prepare | ||||
Diff against target: |
2107 lines (+514/-880) 28 files modified
ensemble/providers/common/base.py (+3/-4) ensemble/providers/common/cloudinit.py (+12/-13) ensemble/providers/common/launch.py (+46/-170) ensemble/providers/common/tests/data/cloud_init_bootstrap (+1/-2) ensemble/providers/common/tests/data/cloud_init_bootstrap_zookeepers (+1/-2) ensemble/providers/common/tests/data/cloud_init_branch (+1/-2) ensemble/providers/common/tests/data/cloud_init_distro (+1/-2) ensemble/providers/common/tests/data/cloud_init_normal (+1/-2) ensemble/providers/common/tests/test_bootstrap.py (+102/-0) ensemble/providers/common/tests/test_cloudinit.py (+2/-2) ensemble/providers/common/tests/test_launch.py (+61/-224) ensemble/providers/common/tests/test_utils.py (+1/-2) ensemble/providers/common/utils.py (+3/-3) ensemble/providers/ec2/__init__.py (+10/-5) ensemble/providers/ec2/launch.py (+32/-53) ensemble/providers/ec2/tests/common.py (+2/-4) ensemble/providers/ec2/tests/data/bootstrap_cloud_init (+18/-0) ensemble/providers/ec2/tests/data/launch_cloud_init (+15/-0) ensemble/providers/ec2/tests/test_bootstrap.py (+16/-51) ensemble/providers/ec2/tests/test_launch.py (+79/-114) ensemble/providers/ec2/tests/test_utils.py (+40/-106) ensemble/providers/ec2/utils.py (+17/-69) ensemble/providers/orchestra/__init__.py (+9/-6) ensemble/providers/orchestra/launch.py (+23/-34) ensemble/providers/orchestra/tests/common.py (+2/-1) ensemble/providers/orchestra/tests/data/bootstrap_late_command (+5/-5) ensemble/providers/orchestra/tests/data/launch_late_command (+5/-4) ensemble/providers/orchestra/tests/test_launch.py (+6/-0) |
||||
To merge this branch: | bzr merge lp:~fwereade/pyjuju/cloud-init-class-used | ||||
Related bugs: |
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Kapil Thangavelu (community) | Approve | ||
Gustavo Niemeyer | Approve | ||
Review via email: mp+73634@code.launchpad.net |
Commit message
Description of the change
Another attempt at a cleaner cloud-init/
- 339. By William Reade
-
merge parent
- 340. By William Reade
-
renaming in CloudInit
- 341. By William Reade
-
get_current_ami now uses same keys as machine_data constraints dict
Gustavo Niemeyer (niemeyer) wrote : | # |
Thanks for the changes William.
[7]
+# XXX ideally should come from latest available or client release name.
FWIW, it should come from the formula Ubuntu series instead.
Gustavo Niemeyer (niemeyer) wrote : | # |
Please go ahead and merge this William. It's been in review for 2 weeks already.
Kapil Thangavelu (hazmat) wrote : | # |
looks great, bye-bye anonymous blobs. some comments.
[0]
- @param machine_data: a dictionary of data to pass along to the newly
- launched machine.
+ `machine_data`: a dictionary describing the newly launched machine.
'describing the machine to be launched.'
[1]
+ def set_provider_
+ self._provider_type = type_
type isn't a keyword
[2]
+ def set_machine_
+ self._machine_id = id
+
+ def set_instance_
+ self._instance_id = expr
+ if self._zookeeper:
+ require(
+ require(
what's the distinction between the two variables. instance id comes from the provider
implementation as a consequence of actually running the machine, its an output value, not
an input value to cloud init afaics.
its apparent a little bit latter into the diff, this is actually a shell command for retrieval
of instance id.. the naming should be reflective of that. perhaps set_instance_
what's the case for the instance id not being set? is that just for testing ease, if so it should probably be commented as such.
William Reade (fwereade) wrote : | # |
> looks great, bye-bye anonymous blobs. some comments.
>
> [0]
>
> - @param machine_data: a dictionary of data to pass along to the newly
> - launched machine.
> + `machine_data`: a dictionary describing the newly launched machine.
>
> 'describing the machine to be launched.'
Thanks
> [1]
>
> + def set_provider_
> + self._provider_type = type_
>
> type isn't a keyword
But it is a builtin, and I prefer not to shadow them unless I absopositively have to.
> [2]
> + def set_machine_
> + self._machine_id = id
> +
> + def set_instance_
> + self._instance_id = expr
>
> + if self._zookeeper:
> + require(
> + require(
>
> what's the distinction between the two variables. instance id comes from the
> provider
> implementation as a consequence of actually running the machine, its an output
> value, not
> an input value to cloud init afaics.
>
> its apparent a little bit latter into the diff, this is actually a shell
> command for retrieval
> of instance id.. the naming should be reflective of that. perhaps
> set_instance_
Good idea
> what's the case for the instance id not being set? is that just for testing
> ease, if so it should probably be commented as such.
It's just that it's not actually used, except when we're running a bootstrap node. This could well change in the future.
- 342. By William Reade
-
merge parent
- 343. By William Reade
-
fixes per hazmat's review
- 344. By William Reade
-
cloud-inits now contain lists of ssh keys, not list-of-lists of ssh keys
Preview Diff
1 | === modified file 'ensemble/providers/common/base.py' | |||
2 | --- ensemble/providers/common/base.py 2011-08-29 02:23:11 +0000 | |||
3 | +++ ensemble/providers/common/base.py 2011-09-13 10:56:30 +0000 | |||
4 | @@ -65,11 +65,10 @@ | |||
5 | 65 | def start_machine(self, machine_data, master=False): | 65 | def start_machine(self, machine_data, master=False): |
6 | 66 | """Start a machine in the provider. | 66 | """Start a machine in the provider. |
7 | 67 | 67 | ||
10 | 68 | @param machine_data: a dictionary of data to pass along to the newly | 68 | `machine_data`: a dictionary describing the machine to be launched. |
9 | 69 | launched machine. | ||
11 | 70 | 69 | ||
14 | 71 | @param master: if True, machine will initialize the ensemble admin | 70 | `master`: if True, machine will initialize the ensemble admin and run |
15 | 72 | and run a provisioning agent. | 71 | a provisioning agent, in addition to running a machine agent |
16 | 73 | """ | 72 | """ |
17 | 74 | raise NotImplementedError() | 73 | raise NotImplementedError() |
18 | 75 | 74 | ||
19 | 76 | 75 | ||
20 | === modified file 'ensemble/providers/common/cloudinit.py' | |||
21 | --- ensemble/providers/common/cloudinit.py 2011-09-13 10:56:30 +0000 | |||
22 | +++ ensemble/providers/common/cloudinit.py 2011-09-13 10:56:30 +0000 | |||
23 | @@ -60,7 +60,7 @@ | |||
24 | 60 | self._ssh_keys = [] | 60 | self._ssh_keys = [] |
25 | 61 | self._provision = False | 61 | self._provision = False |
26 | 62 | self._zookeeper = False | 62 | self._zookeeper = False |
28 | 63 | self._zookeeper_names = [] | 63 | self._zookeeper_hosts = [] |
29 | 64 | self._zookeeper_secret = None | 64 | self._zookeeper_secret = None |
30 | 65 | 65 | ||
31 | 66 | def add_ssh_key(self, key): | 66 | def add_ssh_key(self, key): |
32 | @@ -83,14 +83,14 @@ | |||
33 | 83 | def set_machine_id(self, id): | 83 | def set_machine_id(self, id): |
34 | 84 | self._machine_id = id | 84 | self._machine_id = id |
35 | 85 | 85 | ||
37 | 86 | def set_instance_id(self, expr): | 86 | def set_instance_id_accessor(self, expr): |
38 | 87 | self._instance_id = expr | 87 | self._instance_id = expr |
39 | 88 | 88 | ||
40 | 89 | def set_provider_type(self, type_): | 89 | def set_provider_type(self, type_): |
41 | 90 | self._provider_type = type_ | 90 | self._provider_type = type_ |
42 | 91 | 91 | ||
43 | 92 | def set_zookeeper_machines(self, machines): | 92 | def set_zookeeper_machines(self, machines): |
45 | 93 | self._zookeeper_names = [m.private_dns_name for m in machines] | 93 | self._zookeeper_hosts = [m.private_dns_name for m in machines] |
46 | 94 | 94 | ||
47 | 95 | def set_zookeeper_secret(self, secret): | 95 | def set_zookeeper_secret(self, secret): |
48 | 96 | self._zookeeper_secret = secret | 96 | self._zookeeper_secret = secret |
49 | @@ -115,20 +115,19 @@ | |||
50 | 115 | require("_machine_id", "set_machine_id") | 115 | require("_machine_id", "set_machine_id") |
51 | 116 | require("_provider_type", "set_provider_type") | 116 | require("_provider_type", "set_provider_type") |
52 | 117 | if self._zookeeper: | 117 | if self._zookeeper: |
54 | 118 | require("_instance_id", "set_instance_id") | 118 | require("_instance_id", "set_instance_id_accessor") |
55 | 119 | require("_zookeeper_secret", "set_zookeeper_secret") | 119 | require("_zookeeper_secret", "set_zookeeper_secret") |
56 | 120 | else: | 120 | else: |
58 | 121 | require("_zookeeper_names", "set_zookeeper_machines") | 121 | require("_zookeeper_hosts", "set_zookeeper_machines") |
59 | 122 | if missing: | 122 | if missing: |
60 | 123 | raise CloudInitError("Incomplete cloud-init: you need to call %s" | 123 | raise CloudInitError("Incomplete cloud-init: you need to call %s" |
61 | 124 | % ", ".join(missing)) | 124 | % ", ".join(missing)) |
62 | 125 | 125 | ||
66 | 126 | @property | 126 | def _join_zookeeper_hosts(self): |
67 | 127 | def _zookeeper_hosts(self): | 127 | all_hosts = self._zookeeper_hosts[:] |
65 | 128 | all_names = self._zookeeper_names[:] | ||
68 | 129 | if self._zookeeper: | 128 | if self._zookeeper: |
71 | 130 | all_names.append("localhost") | 129 | all_hosts.append("localhost") |
72 | 131 | return ",".join(["%s:2181" % name for name in all_names]) | 130 | return ",".join(["%s:2181" % host for host in all_hosts]) |
73 | 132 | 131 | ||
74 | 133 | def _collect_packages(self): | 132 | def _collect_packages(self): |
75 | 134 | packages = [ | 133 | packages = [ |
76 | @@ -150,13 +149,13 @@ | |||
77 | 150 | scripts.extend(_zookeeper_scripts( | 149 | scripts.extend(_zookeeper_scripts( |
78 | 151 | self._instance_id, self._zookeeper_secret)) | 150 | self._instance_id, self._zookeeper_secret)) |
79 | 152 | scripts.extend(_machine_scripts( | 151 | scripts.extend(_machine_scripts( |
81 | 153 | self._machine_id, self._zookeeper_hosts)) | 152 | self._machine_id, self._join_zookeeper_hosts())) |
82 | 154 | if self._provision: | 153 | if self._provision: |
84 | 155 | scripts.extend(_provision_scripts(self._zookeeper_hosts)) | 154 | scripts.extend(_provision_scripts(self._join_zookeeper_hosts())) |
85 | 156 | return scripts | 155 | return scripts |
86 | 157 | 156 | ||
87 | 158 | def _collect_machine_data(self): | 157 | def _collect_machine_data(self): |
88 | 159 | return { | 158 | return { |
89 | 160 | "machine-id": self._machine_id, | 159 | "machine-id": self._machine_id, |
90 | 161 | "ensemble-provider-type": self._provider_type, | 160 | "ensemble-provider-type": self._provider_type, |
92 | 162 | "ensemble-zookeeper-hosts": self._zookeeper_hosts} | 161 | "ensemble-zookeeper-hosts": self._join_zookeeper_hosts()} |
93 | 163 | 162 | ||
94 | === modified file 'ensemble/providers/common/launch.py' | |||
95 | --- ensemble/providers/common/launch.py 2011-08-17 16:01:43 +0000 | |||
96 | +++ ensemble/providers/common/launch.py 2011-09-13 10:56:30 +0000 | |||
97 | @@ -1,195 +1,71 @@ | |||
98 | 1 | import copy | ||
99 | 2 | |||
100 | 3 | from twisted.internet.defer import inlineCallbacks, returnValue | 1 | from twisted.internet.defer import inlineCallbacks, returnValue |
101 | 4 | 2 | ||
105 | 5 | from ensemble.errors import ProviderError | 3 | from .cloudinit import CloudInit |
103 | 6 | from ensemble.state.auth import make_identity | ||
104 | 7 | |||
106 | 8 | from .utils import get_user_authorized_keys | 4 | from .utils import get_user_authorized_keys |
107 | 9 | 5 | ||
108 | 10 | DEFAULT_REPOSITORIES = [ | ||
109 | 11 | "ppa:ensemble/ppa"] | ||
110 | 12 | |||
111 | 13 | DEFAULT_PACKAGES = [ | ||
112 | 14 | "bzr", | ||
113 | 15 | "byobu", | ||
114 | 16 | "tmux", | ||
115 | 17 | "python-setuptools", | ||
116 | 18 | "python-twisted", | ||
117 | 19 | "python-argparse", | ||
118 | 20 | "python-txaws", | ||
119 | 21 | "python-zookeeper"] | ||
120 | 22 | |||
121 | 23 | BOOTSTRAP_PACKAGES = [ | ||
122 | 24 | "bzr", | ||
123 | 25 | "default-jre-headless", | ||
124 | 26 | "zookeeper", | ||
125 | 27 | "zookeeperd"] | ||
126 | 28 | |||
127 | 29 | |||
128 | 30 | def _get_initialize_script(instance_id_command, admin_secret): | ||
129 | 31 | admin_identity = make_identity("admin:%s" % admin_secret) | ||
130 | 32 | template = ('ensemble-admin initialize ' | ||
131 | 33 | '--instance-id=%s --admin-identity="%s"') | ||
132 | 34 | return template % (instance_id_command, admin_identity) | ||
133 | 35 | |||
134 | 36 | |||
135 | 37 | def _get_machine_agent_script(vars): | ||
136 | 38 | template = ( | ||
137 | 39 | "ENSEMBLE_MACHINE_ID=%(machine-id)s " | ||
138 | 40 | "ENSEMBLE_ZOOKEEPER=%(ensemble-zookeeper-hosts)s " | ||
139 | 41 | "python -m ensemble.agents.machine -n " | ||
140 | 42 | "--logfile=/var/log/ensemble/machine-agent.log " | ||
141 | 43 | "--pidfile=/var/run/ensemble/machine-agent.pid") | ||
142 | 44 | return template % vars | ||
143 | 45 | |||
144 | 46 | |||
145 | 47 | def _get_provision_agent_script(vars): | ||
146 | 48 | template = ( | ||
147 | 49 | "ENSEMBLE_ZOOKEEPER=%(ensemble-zookeeper-hosts)s " | ||
148 | 50 | "python -m ensemble.agents.provision -n " | ||
149 | 51 | "--logfile=/var/log/ensemble/provision-agent.log " | ||
150 | 52 | "--pidfile=/var/run/ensemble/provision-agent.pid") | ||
151 | 53 | return template % vars | ||
152 | 54 | |||
153 | 55 | 6 | ||
154 | 56 | class LaunchMachine(object): | 7 | class LaunchMachine(object): |
155 | 57 | """Abstract class with generic instance-launching logic. | 8 | """Abstract class with generic instance-launching logic. |
156 | 58 | 9 | ||
157 | 59 | Constructing with master=True will cause the run method to | ||
158 | 60 | construct a machine which is also running a zookeeper for the | ||
159 | 61 | cluster, and a provisioning agent, as well as the usual machine | ||
160 | 62 | agent. | ||
161 | 63 | |||
162 | 64 | To create your own subclass, you will certainly need to override | 10 | To create your own subclass, you will certainly need to override |
165 | 65 | C{start_machine} and C{get_instance_id_command}, and may find it | 11 | `start_machine`, and will very probably want to implement it with the aid |
166 | 66 | convenient to override C{get_machine_variables} as well. | 12 | of a provider-specific subclass of `ProviderCloudInit`. |
167 | 13 | |||
168 | 14 | `provider`: the necessary `MachineProvider` | ||
169 | 15 | |||
170 | 16 | `master`: if True, the machine will run a zookeeper and a provisioning | ||
171 | 17 | agent, in addition to the machine agent that every machine runs | ||
172 | 67 | """ | 18 | """ |
173 | 68 | 19 | ||
175 | 69 | def __init__(self, provider, master=False): | 20 | def __init__(self, provider, master=False, constraints=None): |
176 | 70 | self._provider = provider | 21 | self._provider = provider |
177 | 71 | self._master = master | 22 | self._master = master |
178 | 23 | self._constraints = constraints or {} | ||
179 | 72 | 24 | ||
180 | 73 | @inlineCallbacks | 25 | @inlineCallbacks |
182 | 74 | def run(self, machine_data): | 26 | def run(self, machine_id): |
183 | 75 | """Launch an instance node within the machine provider environment. | 27 | """Launch an instance node within the machine provider environment. |
184 | 76 | 28 | ||
189 | 77 | `machine_data`: a dictionary of data that is passed along to | 29 | `machine_id`: the ensemble machine ID |
186 | 78 | provided to the machine as serialized yaml. 'machine-id' is a | ||
187 | 79 | required key, denoting the machine's zookeeper node for its | ||
188 | 80 | machine agent. | ||
190 | 81 | """ | 30 | """ |
203 | 82 | if "machine-id" not in machine_data: | 31 | # XXX at some point, we'll want to start multiple zookeepers |
204 | 83 | raise ProviderError( | 32 | # that know about each other; for now, this'll do |
205 | 84 | "Machine state `machine-id` not provided in machine_data.") | 33 | if self._master: |
206 | 85 | 34 | zookeepers = [] | |
207 | 86 | machine_variables = yield self.get_machine_variables(machine_data) | 35 | else: |
208 | 87 | provider_machines = yield self.start_machine( | 36 | zookeepers = yield self._provider.get_zookeeper_machines() |
209 | 88 | machine_variables, machine_data["machine-id"]) | 37 | |
210 | 89 | if self._master: | 38 | machines = yield self.start_machine(machine_id, zookeepers) |
211 | 90 | yield self._on_master_launched(provider_machines) | 39 | if self._master: |
212 | 91 | returnValue(provider_machines) | 40 | yield self._on_new_zookeepers(machines) |
213 | 92 | 41 | returnValue(machines) | |
214 | 93 | def start_machine(self, machine_variables, machine_id): | 42 | |
215 | 43 | def start_machine(self, machine_id, zookeepers): | ||
216 | 94 | """Actually launch a machine for the appropriate provider. | 44 | """Actually launch a machine for the appropriate provider. |
217 | 95 | 45 | ||
225 | 96 | `machine_variables`: non-provider-specific data, sufficient to | 46 | `machine_id`: the ensemble machine ID |
226 | 97 | define the machine's behaviour once it exists. | 47 | |
227 | 98 | 48 | `zookeepers`: a list of `ProviderMachine`s already running zookeeper, | |
228 | 99 | `machine_id`: the external machine ID (also exists in the | 49 | which the machine will need to connect to. May be empty. |
229 | 100 | above bag of data) | 50 | |
230 | 101 | 51 | Returns a singe-entry list containing a `ProviderMachine` for the | |
224 | 102 | Returns a singe-entry list containing a ProviderMachine for the | ||
231 | 103 | new instance | 52 | new instance |
232 | 104 | """ | 53 | """ |
233 | 105 | raise NotImplementedError() | 54 | raise NotImplementedError() |
234 | 106 | 55 | ||
320 | 107 | def get_instance_id_command(self): | 56 | def _create_cloud_init(self, machine_id, zookeepers): |
321 | 108 | """Snippet to discover local instance-id | 57 | config = self._provider.config |
322 | 109 | 58 | cloud_init = CloudInit() | |
323 | 110 | @return: a snippet of shell script that evaluates to the | 59 | cloud_init.add_ssh_key(get_user_authorized_keys(config)) |
324 | 111 | local machine's instance_id. | 60 | cloud_init.set_machine_id(machine_id) |
325 | 112 | """ | 61 | cloud_init.set_zookeeper_machines(zookeepers) |
326 | 113 | raise NotImplementedError() | 62 | if config.get("ensemble-branch"): |
327 | 114 | 63 | cloud_init.set_ensemble_source(config["ensemble-branch"]) | |
328 | 115 | @inlineCallbacks | 64 | if self._master: |
329 | 116 | def get_machine_variables(self, vars_in): | 65 | cloud_init.enable_bootstrap() |
330 | 117 | """Get the variables used for launching a machine and its cloud-init""" | 66 | cloud_init.set_zookeeper_secret(config["admin-secret"]) |
331 | 118 | vars_out = copy.deepcopy(vars_in) | 67 | return cloud_init |
332 | 119 | vars_out[ | 68 | |
333 | 120 | "ensemble-zookeeper-hosts"] = yield self._get_zookeeper_hosts() | 69 | def _on_new_zookeepers(self, machines): |
249 | 121 | |||
250 | 122 | authorized_keys = get_user_authorized_keys(self._provider.config) | ||
251 | 123 | vars_out["authorized_keys"] = authorized_keys | ||
252 | 124 | vars_out["packages"] = self._get_packages() | ||
253 | 125 | vars_out["repositories"] = list(DEFAULT_REPOSITORIES) | ||
254 | 126 | vars_out["scripts"] = self._get_launch_scripts(vars_out) | ||
255 | 127 | returnValue(vars_out) | ||
256 | 128 | |||
257 | 129 | @inlineCallbacks | ||
258 | 130 | def _get_zookeeper_hosts(self): | ||
259 | 131 | if self._master: | ||
260 | 132 | returnValue("localhost:2181") | ||
261 | 133 | machines = yield self._provider.get_zookeeper_machines() | ||
262 | 134 | hosts = [m.private_dns_name for m in machines] | ||
263 | 135 | # format multiple hosts in string suitable for client | ||
264 | 136 | returnValue(",".join(["%s:2181" % h for h in hosts])) | ||
265 | 137 | |||
266 | 138 | def _get_packages(self): | ||
267 | 139 | result = list(DEFAULT_PACKAGES) | ||
268 | 140 | if self._master: | ||
269 | 141 | result.extend(BOOTSTRAP_PACKAGES) | ||
270 | 142 | return result | ||
271 | 143 | |||
272 | 144 | def _get_launch_scripts(self, machine_data): | ||
273 | 145 | """Get the scripts that will execute after the machine has launched.""" | ||
274 | 146 | launch_scripts = self._get_install_scripts() | ||
275 | 147 | |||
276 | 148 | # These should be in the deb | ||
277 | 149 | launch_scripts.extend([ | ||
278 | 150 | "sudo mkdir -p /var/lib/ensemble", | ||
279 | 151 | "sudo mkdir -p /var/log/ensemble"]) | ||
280 | 152 | |||
281 | 153 | if self._master: | ||
282 | 154 | launch_scripts.append(self._get_initialize_script()) | ||
283 | 155 | |||
284 | 156 | # Every machine has its own agent. | ||
285 | 157 | machine_agent_script_template = ( | ||
286 | 158 | "ENSEMBLE_MACHINE_ID=%(machine-id)s " | ||
287 | 159 | "ENSEMBLE_ZOOKEEPER=%(ensemble-zookeeper-hosts)s " | ||
288 | 160 | "python -m ensemble.agents.machine -n " | ||
289 | 161 | "--logfile=/var/log/ensemble/machine-agent.log " | ||
290 | 162 | "--pidfile=/var/run/ensemble/machine-agent.pid") | ||
291 | 163 | launch_scripts.append(machine_agent_script_template % machine_data) | ||
292 | 164 | |||
293 | 165 | if self._master: | ||
294 | 166 | launch_scripts.append(_get_provision_agent_script(machine_data)) | ||
295 | 167 | |||
296 | 168 | return launch_scripts | ||
297 | 169 | |||
298 | 170 | def _get_install_scripts(self): | ||
299 | 171 | branch = self._provider.config.get("ensemble-branch") | ||
300 | 172 | if branch is None: | ||
301 | 173 | return ["sudo apt-get -y install ensemble"] | ||
302 | 174 | # If we're using a custom branch, pass it via cloud-init to the agents, | ||
303 | 175 | # so the entire cluster uses it. | ||
304 | 176 | return [ | ||
305 | 177 | "sudo apt-get install -y python-txzookeeper", | ||
306 | 178 | "sudo mkdir -p /usr/lib/ensemble", | ||
307 | 179 | "cd /usr/lib/ensemble && sudo /usr/bin/bzr co %s ensemble" % \ | ||
308 | 180 | branch, | ||
309 | 181 | "cd /usr/lib/ensemble/ensemble && sudo python setup.py develop"] | ||
310 | 182 | |||
311 | 183 | def _get_initialize_script(self): | ||
312 | 184 | admin_secret = self._provider.config["admin-secret"] | ||
313 | 185 | return _get_initialize_script( | ||
314 | 186 | self.get_instance_id_command(), admin_secret) | ||
315 | 187 | |||
316 | 188 | def _on_master_launched(self, machines): | ||
317 | 189 | # TODO this should be part of Bootstrap (and should really extend, | ||
318 | 190 | # rather than effectively replace, the result of | ||
319 | 191 | # self._provider.get_zookeeper_machines) | ||
334 | 192 | instance_ids = [m.instance_id for m in machines] | 70 | instance_ids = [m.instance_id for m in machines] |
338 | 193 | d = self._provider.save_state({"zookeeper-instances": instance_ids}) | 71 | return self._provider.save_state({"zookeeper-instances": instance_ids}) |
336 | 194 | d.addCallback(lambda _: machines) | ||
337 | 195 | return d | ||
339 | 196 | 72 | ||
340 | === modified file 'ensemble/providers/common/tests/data/cloud_init_bootstrap' | |||
341 | --- ensemble/providers/common/tests/data/cloud_init_bootstrap 2011-09-13 10:56:30 +0000 | |||
342 | +++ ensemble/providers/common/tests/data/cloud_init_bootstrap 2011-09-13 10:56:30 +0000 | |||
343 | @@ -14,5 +14,4 @@ | |||
344 | 14 | -n --logfile=/var/log/ensemble/machine-agent.log --pidfile=/var/run/ensemble/machine-agent.pid', | 14 | -n --logfile=/var/log/ensemble/machine-agent.log --pidfile=/var/run/ensemble/machine-agent.pid', |
345 | 15 | 'ENSEMBLE_ZOOKEEPER=localhost:2181 python -m ensemble.agents.provision -n --logfile=/var/log/ensemble/provision-agent.log | 15 | 'ENSEMBLE_ZOOKEEPER=localhost:2181 python -m ensemble.agents.provision -n --logfile=/var/log/ensemble/provision-agent.log |
346 | 16 | --pidfile=/var/run/ensemble/provision-agent.pid'] | 16 | --pidfile=/var/run/ensemble/provision-agent.pid'] |
349 | 17 | ssh_authorized_keys: | 17 | ssh_authorized_keys: [chubb] |
348 | 18 | - [chubb] | ||
350 | 19 | 18 | ||
351 | === modified file 'ensemble/providers/common/tests/data/cloud_init_bootstrap_zookeepers' | |||
352 | --- ensemble/providers/common/tests/data/cloud_init_bootstrap_zookeepers 2011-09-13 10:56:30 +0000 | |||
353 | +++ ensemble/providers/common/tests/data/cloud_init_bootstrap_zookeepers 2011-09-13 10:56:30 +0000 | |||
354 | @@ -15,5 +15,4 @@ | |||
355 | 15 | --pidfile=/var/run/ensemble/machine-agent.pid', 'ENSEMBLE_ZOOKEEPER=cotswold:2181,longleat:2181,localhost:2181 | 15 | --pidfile=/var/run/ensemble/machine-agent.pid', 'ENSEMBLE_ZOOKEEPER=cotswold:2181,longleat:2181,localhost:2181 |
356 | 16 | python -m ensemble.agents.provision -n --logfile=/var/log/ensemble/provision-agent.log | 16 | python -m ensemble.agents.provision -n --logfile=/var/log/ensemble/provision-agent.log |
357 | 17 | --pidfile=/var/run/ensemble/provision-agent.pid'] | 17 | --pidfile=/var/run/ensemble/provision-agent.pid'] |
360 | 18 | ssh_authorized_keys: | 18 | ssh_authorized_keys: [chubb] |
359 | 19 | - [chubb] | ||
361 | 20 | 19 | ||
362 | === modified file 'ensemble/providers/common/tests/data/cloud_init_branch' | |||
363 | --- ensemble/providers/common/tests/data/cloud_init_branch 2011-09-13 10:56:30 +0000 | |||
364 | +++ ensemble/providers/common/tests/data/cloud_init_branch 2011-09-13 10:56:30 +0000 | |||
365 | @@ -14,5 +14,4 @@ | |||
366 | 14 | sudo mkdir -p /var/log/ensemble, 'ENSEMBLE_MACHINE_ID=passport ENSEMBLE_ZOOKEEPER=cotswold:2181,longleat:2181 | 14 | sudo mkdir -p /var/log/ensemble, 'ENSEMBLE_MACHINE_ID=passport ENSEMBLE_ZOOKEEPER=cotswold:2181,longleat:2181 |
367 | 15 | python -m ensemble.agents.machine -n --logfile=/var/log/ensemble/machine-agent.log | 15 | python -m ensemble.agents.machine -n --logfile=/var/log/ensemble/machine-agent.log |
368 | 16 | --pidfile=/var/run/ensemble/machine-agent.pid'] | 16 | --pidfile=/var/run/ensemble/machine-agent.pid'] |
371 | 17 | ssh_authorized_keys: | 17 | ssh_authorized_keys: [chubb] |
370 | 18 | - [chubb] | ||
372 | 19 | 18 | ||
373 | === modified file 'ensemble/providers/common/tests/data/cloud_init_distro' | |||
374 | --- ensemble/providers/common/tests/data/cloud_init_distro 2011-09-13 10:56:30 +0000 | |||
375 | +++ ensemble/providers/common/tests/data/cloud_init_distro 2011-09-13 10:56:30 +0000 | |||
376 | @@ -10,5 +10,4 @@ | |||
377 | 10 | -p /var/log/ensemble, 'ENSEMBLE_MACHINE_ID=passport ENSEMBLE_ZOOKEEPER=cotswold:2181,longleat:2181 | 10 | -p /var/log/ensemble, 'ENSEMBLE_MACHINE_ID=passport ENSEMBLE_ZOOKEEPER=cotswold:2181,longleat:2181 |
378 | 11 | python -m ensemble.agents.machine -n --logfile=/var/log/ensemble/machine-agent.log | 11 | python -m ensemble.agents.machine -n --logfile=/var/log/ensemble/machine-agent.log |
379 | 12 | --pidfile=/var/run/ensemble/machine-agent.pid'] | 12 | --pidfile=/var/run/ensemble/machine-agent.pid'] |
382 | 13 | ssh_authorized_keys: | 13 | ssh_authorized_keys: [chubb] |
381 | 14 | - [chubb] | ||
383 | 15 | 14 | ||
384 | === modified file 'ensemble/providers/common/tests/data/cloud_init_normal' | |||
385 | --- ensemble/providers/common/tests/data/cloud_init_normal 2011-09-13 10:56:30 +0000 | |||
386 | +++ ensemble/providers/common/tests/data/cloud_init_normal 2011-09-13 10:56:30 +0000 | |||
387 | @@ -12,5 +12,4 @@ | |||
388 | 12 | -p /var/log/ensemble, 'ENSEMBLE_MACHINE_ID=passport ENSEMBLE_ZOOKEEPER=cotswold:2181,longleat:2181 | 12 | -p /var/log/ensemble, 'ENSEMBLE_MACHINE_ID=passport ENSEMBLE_ZOOKEEPER=cotswold:2181,longleat:2181 |
389 | 13 | python -m ensemble.agents.machine -n --logfile=/var/log/ensemble/machine-agent.log | 13 | python -m ensemble.agents.machine -n --logfile=/var/log/ensemble/machine-agent.log |
390 | 14 | --pidfile=/var/run/ensemble/machine-agent.pid'] | 14 | --pidfile=/var/run/ensemble/machine-agent.pid'] |
393 | 15 | ssh_authorized_keys: | 15 | ssh_authorized_keys: [chubb] |
392 | 16 | - [chubb] | ||
394 | 17 | 16 | ||
395 | === added file 'ensemble/providers/common/tests/test_bootstrap.py' | |||
396 | --- ensemble/providers/common/tests/test_bootstrap.py 1970-01-01 00:00:00 +0000 | |||
397 | +++ ensemble/providers/common/tests/test_bootstrap.py 2011-09-13 10:56:30 +0000 | |||
398 | @@ -0,0 +1,102 @@ | |||
399 | 1 | import logging | ||
400 | 2 | import tempfile | ||
401 | 3 | |||
402 | 4 | from twisted.internet.defer import fail, succeed | ||
403 | 5 | |||
404 | 6 | from ensemble.errors import EnvironmentNotFound, ProviderError | ||
405 | 7 | from ensemble.lib.testing import TestCase | ||
406 | 8 | from ensemble.providers.common.base import MachineProviderBase | ||
407 | 9 | from ensemble.providers.dummy import DummyMachine, FileStorage | ||
408 | 10 | |||
409 | 11 | |||
410 | 12 | class SomeError(Exception): | ||
411 | 13 | pass | ||
412 | 14 | |||
413 | 15 | |||
414 | 16 | class WorkingFileStorage(FileStorage): | ||
415 | 17 | |||
416 | 18 | def __init__(self): | ||
417 | 19 | super(WorkingFileStorage, self).__init__(tempfile.mkdtemp()) | ||
418 | 20 | |||
419 | 21 | |||
420 | 22 | class UnwritableFileStorage(object): | ||
421 | 23 | |||
422 | 24 | def put(self, name, f): | ||
423 | 25 | return fail(Exception("oh noes")) | ||
424 | 26 | |||
425 | 27 | |||
426 | 28 | class DummyProvider(MachineProviderBase): | ||
427 | 29 | |||
428 | 30 | def __init__(self, file_storage, zookeeper): | ||
429 | 31 | self._file_storage = file_storage | ||
430 | 32 | self._zookeeper = zookeeper | ||
431 | 33 | |||
432 | 34 | def get_file_storage(self): | ||
433 | 35 | return self._file_storage | ||
434 | 36 | |||
435 | 37 | def get_zookeeper_machines(self): | ||
436 | 38 | if isinstance(self._zookeeper, Exception): | ||
437 | 39 | return fail(self._zookeeper) | ||
438 | 40 | if self._zookeeper: | ||
439 | 41 | return succeed([self._zookeeper]) | ||
440 | 42 | return fail(EnvironmentNotFound()) | ||
441 | 43 | |||
442 | 44 | def start_machine(self, machine_id, master=False): | ||
443 | 45 | assert master is True | ||
444 | 46 | return [DummyMachine("i-keepzoos")] | ||
445 | 47 | |||
446 | 48 | |||
447 | 49 | class BootstrapTest(TestCase): | ||
448 | 50 | |||
449 | 51 | def test_unknown_error(self): | ||
450 | 52 | provider = DummyProvider(None, SomeError()) | ||
451 | 53 | d = provider.bootstrap() | ||
452 | 54 | self.assertFailure(d, SomeError) | ||
453 | 55 | return d | ||
454 | 56 | |||
455 | 57 | def test_zookeeper_exists(self): | ||
456 | 58 | log = self.capture_logging("ensemble.common", level=logging.DEBUG) | ||
457 | 59 | provider = DummyProvider( | ||
458 | 60 | WorkingFileStorage(), DummyMachine("i-alreadykeepzoos")) | ||
459 | 61 | d = provider.bootstrap() | ||
460 | 62 | |||
461 | 63 | def verify(machines): | ||
462 | 64 | (machine,) = machines | ||
463 | 65 | self.assertTrue(isinstance(machine, DummyMachine)) | ||
464 | 66 | self.assertEquals(machine.instance_id, "i-alreadykeepzoos") | ||
465 | 67 | |||
466 | 68 | log_text = log.getvalue() | ||
467 | 69 | self.assertIn( | ||
468 | 70 | "Ensemble environment previously bootstrapped", log_text) | ||
469 | 71 | self.assertNotIn("Launching", log_text) | ||
470 | 72 | d.addCallback(verify) | ||
471 | 73 | return d | ||
472 | 74 | |||
473 | 75 | def test_bad_storage(self): | ||
474 | 76 | provider = DummyProvider(UnwritableFileStorage(), None) | ||
475 | 77 | d = provider.bootstrap() | ||
476 | 78 | self.assertFailure(d, ProviderError) | ||
477 | 79 | |||
478 | 80 | def verify(error): | ||
479 | 81 | self.assertEquals( | ||
480 | 82 | str(error), | ||
481 | 83 | "Bootstrap aborted because file storage is not writable: " | ||
482 | 84 | "oh noes") | ||
483 | 85 | d.addCallback(verify) | ||
484 | 86 | return d | ||
485 | 87 | |||
486 | 88 | def test_create_zookeeper(self): | ||
487 | 89 | log = self.capture_logging("ensemble.common", level=logging.DEBUG) | ||
488 | 90 | provider = DummyProvider(WorkingFileStorage(), None) | ||
489 | 91 | d = provider.bootstrap() | ||
490 | 92 | |||
491 | 93 | def verify(machines): | ||
492 | 94 | (machine,) = machines | ||
493 | 95 | self.assertTrue(isinstance(machine, DummyMachine)) | ||
494 | 96 | self.assertEquals(machine.instance_id, "i-keepzoos") | ||
495 | 97 | |||
496 | 98 | log_text = log.getvalue() | ||
497 | 99 | self.assertIn("Launching Ensemble bootstrap instance", log_text) | ||
498 | 100 | self.assertNotIn("previously bootstrapped", log_text) | ||
499 | 101 | d.addCallback(verify) | ||
500 | 102 | return d | ||
501 | 0 | 103 | ||
502 | === modified file 'ensemble/providers/common/tests/test_cloudinit.py' | |||
503 | --- ensemble/providers/common/tests/test_cloudinit.py 2011-09-13 10:56:30 +0000 | |||
504 | +++ ensemble/providers/common/tests/test_cloudinit.py 2011-09-13 10:56:30 +0000 | |||
505 | @@ -26,7 +26,7 @@ | |||
506 | 26 | cloud_init.add_ssh_key("chubb") | 26 | cloud_init.add_ssh_key("chubb") |
507 | 27 | cloud_init.set_machine_id("passport") | 27 | cloud_init.set_machine_id("passport") |
508 | 28 | cloud_init.set_provider_type("dummy") | 28 | cloud_init.set_provider_type("dummy") |
510 | 29 | cloud_init.set_instance_id("token") | 29 | cloud_init.set_instance_id_accessor("token") |
511 | 30 | cloud_init.set_zookeeper_secret("seekrit") | 30 | cloud_init.set_zookeeper_secret("seekrit") |
512 | 31 | if with_zookeepers: | 31 | if with_zookeepers: |
513 | 32 | cloud_init.set_zookeeper_machines([ | 32 | cloud_init.set_zookeeper_machines([ |
514 | @@ -54,7 +54,7 @@ | |||
515 | 54 | self.assertEquals( | 54 | self.assertEquals( |
516 | 55 | str(error), | 55 | str(error), |
517 | 56 | "Incomplete cloud-init: you need to call add_ssh_key, " | 56 | "Incomplete cloud-init: you need to call add_ssh_key, " |
519 | 57 | "set_machine_id, set_provider_type, set_instance_id, " | 57 | "set_machine_id, set_provider_type, set_instance_id_accessor, " |
520 | 58 | "set_zookeeper_secret") | 58 | "set_zookeeper_secret") |
521 | 59 | 59 | ||
522 | 60 | def test_source_validate(self): | 60 | def test_source_validate(self): |
523 | 61 | 61 | ||
524 | === modified file 'ensemble/providers/common/tests/test_launch.py' | |||
525 | --- ensemble/providers/common/tests/test_launch.py 2011-08-17 12:04:56 +0000 | |||
526 | +++ ensemble/providers/common/tests/test_launch.py 2011-09-13 10:56:30 +0000 | |||
527 | @@ -1,243 +1,80 @@ | |||
528 | 1 | import logging | ||
529 | 2 | import tempfile | 1 | import tempfile |
530 | 3 | 2 | ||
532 | 4 | from twisted.internet.defer import fail, inlineCallbacks, succeed | 3 | from twisted.internet.defer import fail, succeed |
533 | 5 | 4 | ||
535 | 6 | from ensemble.errors import EnvironmentNotFound, ProviderError | 5 | from ensemble.errors import EnvironmentNotFound |
536 | 7 | from ensemble.lib.testing import TestCase | 6 | from ensemble.lib.testing import TestCase |
537 | 8 | from ensemble.providers.common.base import MachineProviderBase | 7 | from ensemble.providers.common.base import MachineProviderBase |
540 | 9 | from ensemble.providers.common.launch import ( | 8 | from ensemble.providers.common.launch import LaunchMachine |
539 | 10 | BOOTSTRAP_PACKAGES, DEFAULT_PACKAGES, DEFAULT_REPOSITORIES, LaunchMachine) | ||
541 | 11 | from ensemble.providers.dummy import DummyMachine, FileStorage | 9 | from ensemble.providers.dummy import DummyMachine, FileStorage |
542 | 12 | 10 | ||
575 | 13 | DEFAULT_INSTALL = ["sudo apt-get -y install ensemble"] | 11 | |
576 | 14 | CREATE_DIRS = ["sudo mkdir -p /var/lib/ensemble", | 12 | def launch_machine(test, master, zookeeper): |
545 | 15 | "sudo mkdir -p /var/log/ensemble"] | ||
546 | 16 | |||
547 | 17 | |||
548 | 18 | class DummyLaunchMachine(LaunchMachine): | ||
549 | 19 | |||
550 | 20 | def start_machine(self, variables, data): | ||
551 | 21 | if self._master: | ||
552 | 22 | name = "bootstrapped-instance-id" | ||
553 | 23 | else: | ||
554 | 24 | name = "some-instance-id" | ||
555 | 25 | return [DummyMachine(name)] | ||
556 | 26 | |||
557 | 27 | def get_instance_id_command(self): | ||
558 | 28 | return "$(magic)" | ||
559 | 29 | |||
560 | 30 | |||
561 | 31 | class WorkingFileStorage(FileStorage): | ||
562 | 32 | |||
563 | 33 | def __init__(self): | ||
564 | 34 | super(WorkingFileStorage, self).__init__(tempfile.mkdtemp()) | ||
565 | 35 | |||
566 | 36 | |||
567 | 37 | class UnwritableFileStorage(object): | ||
568 | 38 | |||
569 | 39 | def put(self, name, f): | ||
570 | 40 | return fail(Exception("oh noes")) | ||
571 | 41 | |||
572 | 42 | |||
573 | 43 | def get_provider(launch_class=DummyLaunchMachine, config=None, zookeeper=True, | ||
574 | 44 | file_storage_class=WorkingFileStorage): | ||
577 | 45 | 13 | ||
578 | 46 | class DummyProvider(MachineProviderBase): | 14 | class DummyProvider(MachineProviderBase): |
579 | 47 | 15 | ||
591 | 48 | def __init__(self, config): | 16 | def __init__(self): |
592 | 49 | super(DummyProvider, self).__init__("venus", config) | 17 | self.config = {"admin-secret": "BLAH"} |
593 | 50 | self._file_storage = file_storage_class() | 18 | self._file_storage = FileStorage(tempfile.mkdtemp()) |
583 | 51 | |||
584 | 52 | def get_zookeeper_machines(self): | ||
585 | 53 | # this is mocked out to avoid insane complexity | ||
586 | 54 | if isinstance(zookeeper, Exception): | ||
587 | 55 | return fail(zookeeper) | ||
588 | 56 | return succeed( | ||
589 | 57 | [DummyMachine("zookeeper-instance-id", | ||
590 | 58 | private_dns_name="zookeeper.internal")]) | ||
594 | 59 | 19 | ||
595 | 60 | def get_file_storage(self): | 20 | def get_file_storage(self): |
596 | 61 | return self._file_storage | 21 | return self._file_storage |
597 | 62 | 22 | ||
611 | 63 | def start_machine(self, machine_data, master=False): | 23 | def get_zookeeper_machines(self): |
612 | 64 | return launch_class(self, master).run(machine_data) | 24 | if zookeeper: |
613 | 65 | 25 | return succeed([zookeeper]) | |
614 | 66 | create_config = {"authorized-keys": "abc"} | 26 | return fail(EnvironmentNotFound()) |
615 | 67 | if config is not None: | 27 | |
616 | 68 | create_config.update(config) | 28 | class DummyLaunchMachine(LaunchMachine): |
617 | 69 | return DummyProvider(create_config) | 29 | |
618 | 70 | 30 | def start_machine(self, machine_id, zookeepers): | |
619 | 71 | 31 | test.assertEquals(machine_id, "1234") | |
620 | 72 | def get_launch(launch_class=DummyLaunchMachine, config=None, | 32 | test.assertEquals(zookeepers, filter(None, [zookeeper])) |
621 | 73 | zookeeper=True, master=False): | 33 | test.assertEquals(self._master, master) |
622 | 74 | provider = get_provider(launch_class, config, zookeeper) | 34 | test.assertEquals(self._constraints, {}) |
623 | 75 | return launch_class(provider, master=master) | 35 | return succeed([DummyMachine("i-malive")]) |
624 | 36 | |||
625 | 37 | provider = DummyProvider() | ||
626 | 38 | d = DummyLaunchMachine(provider, master).run("1234") | ||
627 | 39 | return provider, d | ||
628 | 76 | 40 | ||
629 | 77 | 41 | ||
630 | 78 | class LaunchMachineTest(TestCase): | 42 | class LaunchMachineTest(TestCase): |
631 | 79 | 43 | ||
795 | 80 | def _verify_machines(self, machines): | 44 | def assert_success(self, master, zookeeper): |
796 | 81 | (machine,) = machines | 45 | provider, d = launch_machine(self, master, zookeeper) |
797 | 82 | self.assertTrue(isinstance(machine, DummyMachine)) | 46 | |
798 | 83 | self.assertEquals(machine.instance_id, "some-instance-id") | 47 | def verify(machines): |
799 | 84 | 48 | (machine,) = machines | |
800 | 85 | def test_no_machine_id(self): | 49 | self.assertTrue(isinstance(machine, DummyMachine)) |
801 | 86 | launch = get_launch() | 50 | self.assertEquals(machine.instance_id, "i-malive") |
802 | 87 | d = launch.run({}) | 51 | return provider |
803 | 88 | self.assertFailure(d, ProviderError) | 52 | d.addCallback(verify) |
804 | 89 | return d | 53 | return d |
805 | 90 | 54 | ||
806 | 91 | def test_basic_run(self): | 55 | def test_start_nonzookeeper_no_zookeepers(self): |
807 | 92 | launch = get_launch() | 56 | """Starting a non-zookeeper without a zookeeper is an error""" |
808 | 93 | d = launch.run({"machine-id": "machine-77"}) | 57 | unused, d = launch_machine(self, False, None) |
809 | 94 | d.addCallback(self._verify_machines) | 58 | self.assertFailure(d, EnvironmentNotFound) |
810 | 95 | return d | 59 | return d |
811 | 96 | 60 | ||
812 | 97 | @inlineCallbacks | 61 | def test_start_zookeeper_no_zookeepers(self): |
813 | 98 | def test_bootstrap_run(self): | 62 | """A new zookeeper should be recorded in provider state""" |
814 | 99 | provider = get_provider(config={"admin-secret": "whatever"}) | 63 | d = self.assert_success(True, None) |
815 | 100 | launch = DummyLaunchMachine(provider, master=True) | 64 | |
816 | 101 | yield launch.run({"machine-id": "machine-32767"}) | 65 | def verify(provider): |
817 | 102 | saved_state = yield provider.load_state() | 66 | provider_state = yield provider.load_state() |
818 | 103 | self.assertEquals( | 67 | self.assertEquals( |
819 | 104 | saved_state, {"zookeeper-instances": ["bootstrapped-instance-id"]}) | 68 | provider_state, {"zookeeper-instances": ["i-malive"]}) |
820 | 105 | 69 | d.addCallback(verify) | |
821 | 106 | def test_get_machine_variables_normal(self): | 70 | return d |
822 | 107 | launch = get_launch() | 71 | |
823 | 108 | d = launch.get_machine_variables({"machine-id": "machine-999"}) | 72 | def test_works_with_zookeeper(self): |
824 | 109 | 73 | """Creating a non-zookeeper machine should not alter provider state""" | |
825 | 110 | def verify_vars(vars): | 74 | d = self.assert_success(False, DummyMachine("i-keepzoos")) |
826 | 111 | expect_vars = { | 75 | |
827 | 112 | "authorized_keys": "abc", | 76 | def verify(provider): |
828 | 113 | "ensemble-zookeeper-hosts": "zookeeper.internal:2181", | 77 | provider_state = yield provider.load_state() |
829 | 114 | "machine-id": "machine-999", | 78 | self.assertEquals(provider_state, False) |
830 | 115 | "packages": DEFAULT_PACKAGES, | 79 | d.addCallback(verify) |
668 | 116 | "repositories": DEFAULT_REPOSITORIES, | ||
669 | 117 | "scripts": DEFAULT_INSTALL + CREATE_DIRS + | ||
670 | 118 | ["ENSEMBLE_MACHINE_ID=machine-999 " | ||
671 | 119 | "ENSEMBLE_ZOOKEEPER=zookeeper.internal:2181 " | ||
672 | 120 | "python -m ensemble.agents.machine -n " | ||
673 | 121 | "--logfile=/var/log/ensemble/machine-agent.log " | ||
674 | 122 | "--pidfile=/var/run/ensemble/machine-agent.pid"]} | ||
675 | 123 | self.assertEquals(vars, expect_vars) | ||
676 | 124 | d.addCallback(verify_vars) | ||
677 | 125 | return d | ||
678 | 126 | |||
679 | 127 | def test_get_machine_variables_with_ensemble_branch(self): | ||
680 | 128 | branch_name = "lp:~my/ensemble/branch" | ||
681 | 129 | launch = get_launch(config={"ensemble-branch": branch_name}) | ||
682 | 130 | d = launch.get_machine_variables({"machine-id": "machine-303"}) | ||
683 | 131 | |||
684 | 132 | def verify_vars(vars): | ||
685 | 133 | scripts = vars["scripts"] | ||
686 | 134 | install_scripts = ["sudo apt-get install -y python-txzookeeper", | ||
687 | 135 | "sudo mkdir -p /usr/lib/ensemble", | ||
688 | 136 | "cd /usr/lib/ensemble && " | ||
689 | 137 | "sudo /usr/bin/bzr co " | ||
690 | 138 | + branch_name + " ensemble", | ||
691 | 139 | "cd /usr/lib/ensemble/ensemble && " | ||
692 | 140 | "sudo python setup.py develop"] | ||
693 | 141 | install_count = len(install_scripts) | ||
694 | 142 | self.assertEquals(scripts[:install_count], install_scripts) | ||
695 | 143 | |||
696 | 144 | other_scripts = scripts[install_count:] | ||
697 | 145 | self.assertEquals(other_scripts, | ||
698 | 146 | CREATE_DIRS + [ | ||
699 | 147 | "ENSEMBLE_MACHINE_ID=machine-303 " | ||
700 | 148 | "ENSEMBLE_ZOOKEEPER=zookeeper.internal:2181 " | ||
701 | 149 | "python -m ensemble.agents.machine -n " | ||
702 | 150 | "--logfile=/var/log/ensemble/machine-agent.log " | ||
703 | 151 | "--pidfile=/var/run/ensemble/machine-agent.pid"]) | ||
704 | 152 | d.addCallback(verify_vars) | ||
705 | 153 | return d | ||
706 | 154 | |||
707 | 155 | def test_get_machine_variables_bootstrap(self): | ||
708 | 156 | launch = get_launch(config={"admin-secret": "SEEKRIT"}, master=True) | ||
709 | 157 | d = launch.get_machine_variables({"machine-id": "machine-757"}) | ||
710 | 158 | |||
711 | 159 | def verify_vars(vars): | ||
712 | 160 | expect_vars = { | ||
713 | 161 | "authorized_keys": "abc", | ||
714 | 162 | "ensemble-zookeeper-hosts": "localhost:2181", | ||
715 | 163 | "machine-id": "machine-757", | ||
716 | 164 | "packages": DEFAULT_PACKAGES + BOOTSTRAP_PACKAGES, | ||
717 | 165 | "repositories": DEFAULT_REPOSITORIES} | ||
718 | 166 | for key, value in expect_vars.items(): | ||
719 | 167 | self.assertEquals(vars[key], value) | ||
720 | 168 | |||
721 | 169 | scripts = vars["scripts"] | ||
722 | 170 | self.assertEquals(scripts[:3], DEFAULT_INSTALL + CREATE_DIRS) | ||
723 | 171 | admin_init, machine_agent, provision_agent = scripts[3:] | ||
724 | 172 | |||
725 | 173 | self.assertTrue(admin_init.startswith( | ||
726 | 174 | "ensemble-admin initialize " | ||
727 | 175 | '--instance-id=$(magic) --admin-identity="admin:')) | ||
728 | 176 | self.assertFalse("SEEKRIT" in admin_init) | ||
729 | 177 | |||
730 | 178 | expect_machine = ( | ||
731 | 179 | "ENSEMBLE_MACHINE_ID=machine-757 " | ||
732 | 180 | "ENSEMBLE_ZOOKEEPER=localhost:2181 " | ||
733 | 181 | "python -m ensemble.agents.machine -n " | ||
734 | 182 | "--logfile=/var/log/ensemble/machine-agent.log " | ||
735 | 183 | "--pidfile=/var/run/ensemble/machine-agent.pid") | ||
736 | 184 | self.assertEquals(machine_agent, expect_machine) | ||
737 | 185 | |||
738 | 186 | expect_provision = ( | ||
739 | 187 | "ENSEMBLE_ZOOKEEPER=localhost:2181 " | ||
740 | 188 | "python -m ensemble.agents.provision -n " | ||
741 | 189 | "--logfile=/var/log/ensemble/provision-agent.log " | ||
742 | 190 | "--pidfile=/var/run/ensemble/provision-agent.pid") | ||
743 | 191 | self.assertEquals(provision_agent, expect_provision) | ||
744 | 192 | d.addCallback(verify_vars) | ||
745 | 193 | return d | ||
746 | 194 | |||
747 | 195 | |||
748 | 196 | class BootstrapTest(TestCase): | ||
749 | 197 | |||
750 | 198 | def test_bootstrap_unknown_error(self): | ||
751 | 199 | class BBQError(Exception): | ||
752 | 200 | pass | ||
753 | 201 | provider = get_provider(zookeeper=BBQError()) | ||
754 | 202 | d = provider.bootstrap() | ||
755 | 203 | self.assertFailure(d, BBQError) | ||
756 | 204 | return d | ||
757 | 205 | |||
758 | 206 | def test_bootstrap_unwritable_storage(self): | ||
759 | 207 | provider = get_provider(zookeeper=EnvironmentNotFound(), | ||
760 | 208 | file_storage_class=UnwritableFileStorage) | ||
761 | 209 | d = provider.bootstrap() | ||
762 | 210 | self.assertFailure(d, ProviderError) | ||
763 | 211 | return d | ||
764 | 212 | |||
765 | 213 | def test_bootstrap_no_launch(self): | ||
766 | 214 | log = self.capture_logging("ensemble.common", level=logging.DEBUG) | ||
767 | 215 | provider = get_provider() | ||
768 | 216 | d = provider.bootstrap() | ||
769 | 217 | |||
770 | 218 | def verify_machines(machines): | ||
771 | 219 | (machine,) = machines | ||
772 | 220 | self.assertTrue(isinstance(machine, DummyMachine)) | ||
773 | 221 | self.assertEquals(machine.instance_id, "zookeeper-instance-id") | ||
774 | 222 | log_text = log.getvalue() | ||
775 | 223 | self.assertIn( | ||
776 | 224 | "Ensemble environment previously bootstrapped", log_text) | ||
777 | 225 | self.assertNotIn("Launching", log_text) | ||
778 | 226 | d.addCallback(verify_machines) | ||
779 | 227 | return d | ||
780 | 228 | |||
781 | 229 | def test_bootstrap_launch(self): | ||
782 | 230 | log = self.capture_logging("ensemble.common", level=logging.DEBUG) | ||
783 | 231 | provider = get_provider(config={"admin-secret": "SEEKRIT"}, | ||
784 | 232 | zookeeper=EnvironmentNotFound()) | ||
785 | 233 | d = provider.bootstrap() | ||
786 | 234 | |||
787 | 235 | def verify_machines(machines): | ||
788 | 236 | (machine,) = machines | ||
789 | 237 | self.assertTrue(isinstance(machine, DummyMachine)) | ||
790 | 238 | self.assertEquals(machine.instance_id, "bootstrapped-instance-id") | ||
791 | 239 | log_text = log.getvalue() | ||
792 | 240 | self.assertIn("Launching Ensemble bootstrap instance", log_text) | ||
793 | 241 | self.assertNotIn("previously bootstrapped", log_text) | ||
794 | 242 | d.addCallback(verify_machines) | ||
831 | 243 | return d | 80 | return d |
832 | 244 | 81 | ||
833 | === modified file 'ensemble/providers/common/tests/test_utils.py' | |||
834 | --- ensemble/providers/common/tests/test_utils.py 2011-08-18 12:19:22 +0000 | |||
835 | +++ ensemble/providers/common/tests/test_utils.py 2011-09-13 10:56:30 +0000 | |||
836 | @@ -1,5 +1,4 @@ | |||
837 | 1 | import os | 1 | import os |
838 | 2 | |||
839 | 3 | from yaml import load | 2 | from yaml import load |
840 | 4 | 3 | ||
841 | 5 | from twisted.python.failure import Failure | 4 | from twisted.python.failure import Failure |
842 | @@ -100,7 +99,7 @@ | |||
843 | 100 | scripts = ["wget http://lwn.net > /tmp/out"] | 99 | scripts = ["wget http://lwn.net > /tmp/out"] |
844 | 101 | repositories = ["ppa:ensemble/ppa"] | 100 | repositories = ["ppa:ensemble/ppa"] |
845 | 102 | output = format_cloud_init( | 101 | output = format_cloud_init( |
847 | 103 | "zebra", | 102 | ["zebra"], |
848 | 104 | packages=packages, | 103 | packages=packages, |
849 | 105 | scripts=scripts, | 104 | scripts=scripts, |
850 | 106 | repositories=repositories, | 105 | repositories=repositories, |
851 | 107 | 106 | ||
852 | === modified file 'ensemble/providers/common/utils.py' | |||
853 | --- ensemble/providers/common/utils.py 2011-08-31 21:12:40 +0000 | |||
854 | +++ ensemble/providers/common/utils.py 2011-09-13 10:56:30 +0000 | |||
855 | @@ -1,6 +1,5 @@ | |||
856 | 1 | import logging | 1 | import logging |
857 | 2 | import os | 2 | import os |
858 | 3 | |||
859 | 4 | from yaml import safe_dump | 3 | from yaml import safe_dump |
860 | 5 | 4 | ||
861 | 6 | from twisted.python.failure import Failure | 5 | from twisted.python.failure import Failure |
862 | @@ -40,6 +39,7 @@ | |||
863 | 40 | raise error | 39 | raise error |
864 | 41 | 40 | ||
865 | 42 | 41 | ||
866 | 42 | # XXX this should either be singular, or return a list | ||
867 | 43 | def get_user_authorized_keys(config): | 43 | def get_user_authorized_keys(config): |
868 | 44 | """Locate a public key for the user. | 44 | """Locate a public key for the user. |
869 | 45 | 45 | ||
870 | @@ -85,7 +85,7 @@ | |||
871 | 85 | 85 | ||
872 | 86 | @param authorized_keys The authorized SSH keys to be used when populating | 86 | @param authorized_keys The authorized SSH keys to be used when populating |
873 | 87 | the newly launched machine's. | 87 | the newly launched machine's. |
875 | 88 | @type string | 88 | @type list of strings |
876 | 89 | 89 | ||
877 | 90 | @param packages A list of packages to be installed on a machine. | 90 | @param packages A list of packages to be installed on a machine. |
878 | 91 | @type list of strings | 91 | @type list of strings |
879 | @@ -104,7 +104,7 @@ | |||
880 | 104 | cloud_config = { | 104 | cloud_config = { |
881 | 105 | "apt-update": True, | 105 | "apt-update": True, |
882 | 106 | "apt-upgrade": True, | 106 | "apt-upgrade": True, |
884 | 107 | "ssh_authorized_keys": [authorized_keys], | 107 | "ssh_authorized_keys": authorized_keys, |
885 | 108 | "packages": [], | 108 | "packages": [], |
886 | 109 | "output": { "all": "| tee -a /var/log/cloud-init-output.log" }} | 109 | "output": { "all": "| tee -a /var/log/cloud-init-output.log" }} |
887 | 110 | 110 | ||
888 | 111 | 111 | ||
889 | === modified file 'ensemble/providers/ec2/__init__.py' | |||
890 | --- ensemble/providers/ec2/__init__.py 2011-08-29 18:34:02 +0000 | |||
891 | +++ ensemble/providers/ec2/__init__.py 2011-09-13 10:56:30 +0000 | |||
892 | @@ -1,7 +1,8 @@ | |||
893 | 1 | import os | 1 | import os |
894 | 2 | import re | 2 | import re |
895 | 3 | 3 | ||
897 | 4 | from twisted.internet.defer import inlineCallbacks, returnValue | 4 | from twisted.internet.defer import fail, inlineCallbacks, returnValue |
898 | 5 | |||
899 | 5 | from txaws.ec2.exception import EC2Error | 6 | from txaws.ec2.exception import EC2Error |
900 | 6 | from txaws.service import AWSServiceRegion | 7 | from txaws.service import AWSServiceRegion |
901 | 7 | 8 | ||
902 | @@ -53,13 +54,17 @@ | |||
903 | 53 | def start_machine(self, machine_data, master=False): | 54 | def start_machine(self, machine_data, master=False): |
904 | 54 | """Start a machine in the provider. | 55 | """Start a machine in the provider. |
905 | 55 | 56 | ||
908 | 56 | @param machine_data: a dictionary of data to pass along to the newly | 57 | `machine_data`: a dictionary describing the machine to be launched. |
907 | 57 | launched machine. | ||
909 | 58 | 58 | ||
911 | 59 | @param master if True, machine will initialize the ensemble admin | 59 | @param master: if True, machine will initialize the ensemble admin |
912 | 60 | and run a provisioning agent. | 60 | and run a provisioning agent. |
913 | 61 | """ | 61 | """ |
915 | 62 | return EC2LaunchMachine(self, master).run(machine_data) | 62 | if "machine-id" not in machine_data: |
916 | 63 | return fail(ProviderError( | ||
917 | 64 | "Cannot launch a machine without specifying a machine-id")) | ||
918 | 65 | machine_id = machine_data["machine-id"] | ||
919 | 66 | constraints = machine_data.get("constraints", {}) | ||
920 | 67 | return EC2LaunchMachine(self, master, constraints).run(machine_id) | ||
921 | 63 | 68 | ||
922 | 64 | @inlineCallbacks | 69 | @inlineCallbacks |
923 | 65 | def get_machines(self, instance_ids=()): | 70 | def get_machines(self, instance_ids=()): |
924 | 66 | 71 | ||
925 | === modified file 'ensemble/providers/ec2/launch.py' | |||
926 | --- ensemble/providers/ec2/launch.py 2011-08-23 09:59:59 +0000 | |||
927 | +++ ensemble/providers/ec2/launch.py 2011-09-13 10:56:30 +0000 | |||
928 | @@ -1,67 +1,52 @@ | |||
929 | 1 | from twisted.internet.defer import inlineCallbacks, returnValue | 1 | from twisted.internet.defer import inlineCallbacks, returnValue |
930 | 2 | |||
931 | 2 | from txaws.ec2.exception import EC2Error | 3 | from txaws.ec2.exception import EC2Error |
932 | 3 | 4 | ||
933 | 4 | from ensemble.errors import ProviderInteractionError | 5 | from ensemble.errors import ProviderInteractionError |
934 | 5 | from ensemble.providers.common.launch import LaunchMachine | 6 | from ensemble.providers.common.launch import LaunchMachine |
935 | 6 | 7 | ||
936 | 7 | from .machine import machine_from_instance | 8 | from .machine import machine_from_instance |
938 | 8 | from .utils import log, get_launch_options | 9 | from .utils import get_image_id, log |
939 | 9 | 10 | ||
940 | 10 | 11 | ||
941 | 11 | class EC2LaunchMachine(LaunchMachine): | 12 | class EC2LaunchMachine(LaunchMachine): |
942 | 12 | """Amazon EC2 operation for launching an instance""" | 13 | """Amazon EC2 operation for launching an instance""" |
943 | 13 | 14 | ||
944 | 14 | @inlineCallbacks | 15 | @inlineCallbacks |
946 | 15 | def start_machine(self, machine_variables, machine_id): | 16 | def start_machine(self, machine_id, zookeepers): |
947 | 16 | """Actually launch an instance on EC2. | 17 | """Actually launch an instance on EC2. |
948 | 17 | 18 | ||
951 | 18 | `machine_variables`: non-provider-specific data, sufficient to | 19 | `machine_id`: the ensemble machine ID |
950 | 19 | define the machine's behaviour once it exists. | ||
952 | 20 | 20 | ||
955 | 21 | `machine_id`: the external machine ID (also exists in the | 21 | `zookeepers`: a list of `ProviderMachine`s already running zookeeper, |
956 | 22 | above bag of data) | 22 | which the machine will need to connect to. May be empty. |
957 | 23 | 23 | ||
958 | 24 | Returns a singe-entry list containing a ProviderMachine for the | 24 | Returns a singe-entry list containing a ProviderMachine for the |
959 | 25 | new instance | 25 | new instance |
960 | 26 | """ | 26 | """ |
996 | 27 | # Retrieves standard ec2 run_instances arguments, of note | 27 | cloud_init = self._create_cloud_init(machine_id, zookeepers) |
997 | 28 | # image id, and instance type. | 28 | cloud_init.set_provider_type("ec2") |
998 | 29 | machine_options = yield get_launch_options( | 29 | cloud_init.set_instance_id_accessor( |
999 | 30 | self._provider.config, | 30 | "$(curl http://169.254.169.254/1.0/meta-data/instance-id)") |
1000 | 31 | **machine_variables) | 31 | user_data = cloud_init.render() |
1001 | 32 | 32 | ||
1002 | 33 | # Ensure the ensemble security groups exist (both the overall | 33 | instance_type = self._provider.config.get( |
1003 | 34 | # group and the machine-specific group) and are included in | 34 | "default-instance-type", "m1.small") |
1004 | 35 | # the machine instance launch options. | 35 | image_id = yield get_image_id(self._provider.config, self._constraints) |
1005 | 36 | machine_options["security_groups"] = yield self._ensure_group( | 36 | security_groups = yield self._ensure_groups(machine_id) |
1006 | 37 | machine_options["security_groups"], | 37 | |
1007 | 38 | machine_id) | 38 | instances = yield self._provider.ec2.run_instances( |
1008 | 39 | 39 | image_id=image_id, | |
1009 | 40 | # Launch the instance. | 40 | instance_type=instance_type, |
1010 | 41 | instances = yield self._provider.ec2.run_instances(**machine_options) | 41 | min_count=1, |
1011 | 42 | returnValue([machine_from_instance(instance) | 42 | max_count=1, |
1012 | 43 | for instance in instances]) | 43 | security_groups=security_groups, |
1013 | 44 | 44 | user_data=user_data) | |
1014 | 45 | def get_instance_id_command(self): | 45 | |
1015 | 46 | """Snippet to discover local instance-id | 46 | returnValue([machine_from_instance(i) for i in instances]) |
981 | 47 | |||
982 | 48 | @return: a snippet of shell script that evaluates to the | ||
983 | 49 | local machine's instance_id. | ||
984 | 50 | """ | ||
985 | 51 | return "$(curl http://169.254.169.254/1.0/meta-data/instance-id)" | ||
986 | 52 | |||
987 | 53 | def get_machine_variables(self, vars_in): | ||
988 | 54 | vars_deferred = super( | ||
989 | 55 | EC2LaunchMachine, self).get_machine_variables(vars_in) | ||
990 | 56 | |||
991 | 57 | def set_provider_type(vars): | ||
992 | 58 | vars["ensemble-provider-type"] = "ec2" | ||
993 | 59 | return vars | ||
994 | 60 | vars_deferred.addCallback(set_provider_type) | ||
995 | 61 | return vars_deferred | ||
1016 | 62 | 47 | ||
1017 | 63 | @inlineCallbacks | 48 | @inlineCallbacks |
1019 | 64 | def _ensure_group(self, groups, machine_id): | 49 | def _ensure_groups(self, machine_id): |
1020 | 65 | """Ensure the ensemble group is the machine launch groups. | 50 | """Ensure the ensemble group is the machine launch groups. |
1021 | 66 | 51 | ||
1022 | 67 | Machines launched by ensemble are tagged with a group so they | 52 | Machines launched by ensemble are tagged with a group so they |
1023 | @@ -71,21 +56,15 @@ | |||
1024 | 71 | specific machine security group is created for each machine, | 56 | specific machine security group is created for each machine, |
1025 | 72 | so that its firewall rules can be configured per machine. | 57 | so that its firewall rules can be configured per machine. |
1026 | 73 | 58 | ||
1027 | 74 | `groups`: The configured security groups for a machine instance. | ||
1028 | 75 | |||
1029 | 76 | `machine_id`: The external machine ID of this machine instance | 59 | `machine_id`: The external machine ID of this machine instance |
1030 | 77 | """ | 60 | """ |
1031 | 78 | # Label the instance with the ensemble name. | ||
1032 | 79 | security_groups = yield self._provider.ec2.describe_security_groups() | ||
1033 | 80 | ensemble_group = "ensemble-%s" % self._provider.environment_name | 61 | ensemble_group = "ensemble-%s" % self._provider.environment_name |
1034 | 81 | ensemble_machine_group = "ensemble-%s-%s" % ( | 62 | ensemble_machine_group = "ensemble-%s-%s" % ( |
1035 | 82 | self._provider.environment_name, machine_id) | 63 | self._provider.environment_name, machine_id) |
1036 | 64 | |||
1037 | 65 | security_groups = yield self._provider.ec2.describe_security_groups() | ||
1038 | 83 | group_ids = [group.name for group in security_groups] | 66 | group_ids = [group.name for group in security_groups] |
1039 | 84 | 67 | ||
1040 | 85 | # Ensure the provider group is added to the instance groups. | ||
1041 | 86 | if not ensemble_group in groups: | ||
1042 | 87 | groups.append(ensemble_group) | ||
1043 | 88 | |||
1044 | 89 | # Create the provider group if doesn't exist. | 68 | # Create the provider group if doesn't exist. |
1045 | 90 | if not ensemble_group in group_ids: | 69 | if not ensemble_group in group_ids: |
1046 | 91 | log.debug("Creating ensemble provider group %s", ensemble_group) | 70 | log.debug("Creating ensemble provider group %s", ensemble_group) |
1047 | @@ -131,6 +110,6 @@ | |||
1048 | 131 | ensemble_machine_group, | 110 | ensemble_machine_group, |
1049 | 132 | "Ensemble group for %s machine %s" % ( | 111 | "Ensemble group for %s machine %s" % ( |
1050 | 133 | self._provider.environment_name, machine_id)) | 112 | self._provider.environment_name, machine_id)) |
1054 | 134 | groups.append(ensemble_machine_group) | 113 | |
1055 | 135 | 114 | returnValue([ensemble_group, ensemble_machine_group]) | |
1056 | 136 | returnValue(groups) | 115 | |
1057 | 137 | 116 | ||
1058 | === modified file 'ensemble/providers/ec2/tests/common.py' | |||
1059 | --- ensemble/providers/ec2/tests/common.py 2011-08-29 02:23:11 +0000 | |||
1060 | +++ ensemble/providers/ec2/tests/common.py 2011-09-13 10:56:30 +0000 | |||
1061 | @@ -12,8 +12,6 @@ | |||
1062 | 12 | from ensemble.providers.ec2 import MachineProvider | 12 | from ensemble.providers.ec2 import MachineProvider |
1063 | 13 | from ensemble.providers.ec2.machine import EC2ProviderMachine | 13 | from ensemble.providers.ec2.machine import EC2ProviderMachine |
1064 | 14 | 14 | ||
1065 | 15 | |||
1066 | 16 | MATCH_AMI = MATCH(lambda image_id: image_id.startswith("ami-")) | ||
1067 | 17 | MATCH_GROUP = MATCH(lambda x: x.startswith("ensemble-moon")) | 15 | MATCH_GROUP = MATCH(lambda x: x.startswith("ensemble-moon")) |
1068 | 18 | 16 | ||
1069 | 19 | 17 | ||
1070 | @@ -82,7 +80,7 @@ | |||
1071 | 82 | 80 | ||
1072 | 83 | class EC2MachineLaunchMixin(object): | 81 | class EC2MachineLaunchMixin(object): |
1073 | 84 | 82 | ||
1075 | 85 | def _mock_launch_utils(self, **get_ami_kwargs): | 83 | def _mock_launch_utils(self, ami_name="ami-default", **get_ami_kwargs): |
1076 | 86 | get_public_key = self.mocker.replace( | 84 | get_public_key = self.mocker.replace( |
1077 | 87 | "ensemble.providers.common.utils.get_user_authorized_keys") | 85 | "ensemble.providers.common.utils.get_user_authorized_keys") |
1078 | 88 | 86 | ||
1079 | @@ -100,7 +98,7 @@ | |||
1080 | 100 | 98 | ||
1081 | 101 | def check_kwargs(**kwargs): | 99 | def check_kwargs(**kwargs): |
1082 | 102 | self.assertEquals(kwargs, get_ami_kwargs) | 100 | self.assertEquals(kwargs, get_ami_kwargs) |
1084 | 103 | return succeed("ami-714ba518") | 101 | return succeed(ami_name) |
1085 | 104 | self.mocker.call(check_kwargs) | 102 | self.mocker.call(check_kwargs) |
1086 | 105 | 103 | ||
1087 | 106 | def _mock_create_group(self): | 104 | def _mock_create_group(self): |
1088 | 107 | 105 | ||
1089 | === added file 'ensemble/providers/ec2/tests/data/bootstrap_cloud_init' | |||
1090 | --- ensemble/providers/ec2/tests/data/bootstrap_cloud_init 1970-01-01 00:00:00 +0000 | |||
1091 | +++ ensemble/providers/ec2/tests/data/bootstrap_cloud_init 2011-09-13 10:56:30 +0000 | |||
1092 | @@ -0,0 +1,18 @@ | |||
1093 | 1 | #cloud-config | ||
1094 | 2 | apt-update: true | ||
1095 | 3 | apt-upgrade: true | ||
1096 | 4 | apt_sources: | ||
1097 | 5 | - {source: 'ppa:ensemble/ppa'} | ||
1098 | 6 | machine-data: {ensemble-provider-type: ec2, ensemble-zookeeper-hosts: 'localhost:2181', | ||
1099 | 7 | machine-id: '0'} | ||
1100 | 8 | output: {all: '| tee -a /var/log/cloud-init-output.log'} | ||
1101 | 9 | packages: [bzr, byobu, tmux, python-setuptools, python-twisted, python-argparse, python-txaws, | ||
1102 | 10 | python-zookeeper, default-jre-headless, zookeeper, zookeeperd] | ||
1103 | 11 | runcmd: [sudo apt-get -y install ensemble, sudo mkdir -p /var/lib/ensemble, sudo mkdir | ||
1104 | 12 | -p /var/log/ensemble, 'ensemble-admin initialize --instance-id=$(curl http://169.254.169.254/1.0/meta-data/instance-id) | ||
1105 | 13 | --admin-identity=admin:JbJ6sDGV37EHzbG9FPvttk64cmg=', 'ENSEMBLE_MACHINE_ID=0 ENSEMBLE_ZOOKEEPER=localhost:2181 | ||
1106 | 14 | python -m ensemble.agents.machine -n --logfile=/var/log/ensemble/machine-agent.log | ||
1107 | 15 | --pidfile=/var/run/ensemble/machine-agent.pid', 'ENSEMBLE_ZOOKEEPER=localhost:2181 | ||
1108 | 16 | python -m ensemble.agents.provision -n --logfile=/var/log/ensemble/provision-agent.log | ||
1109 | 17 | --pidfile=/var/run/ensemble/provision-agent.pid'] | ||
1110 | 18 | ssh_authorized_keys: [zebra] | ||
1111 | 0 | 19 | ||
1112 | === added file 'ensemble/providers/ec2/tests/data/launch_cloud_init' | |||
1113 | --- ensemble/providers/ec2/tests/data/launch_cloud_init 1970-01-01 00:00:00 +0000 | |||
1114 | +++ ensemble/providers/ec2/tests/data/launch_cloud_init 2011-09-13 10:56:30 +0000 | |||
1115 | @@ -0,0 +1,15 @@ | |||
1116 | 1 | #cloud-config | ||
1117 | 2 | apt-update: true | ||
1118 | 3 | apt-upgrade: true | ||
1119 | 4 | apt_sources: | ||
1120 | 5 | - {source: 'ppa:ensemble/ppa'} | ||
1121 | 6 | machine-data: {ensemble-provider-type: ec2, ensemble-zookeeper-hosts: 'es.example.internal:2181', | ||
1122 | 7 | machine-id: '1'} | ||
1123 | 8 | output: {all: '| tee -a /var/log/cloud-init-output.log'} | ||
1124 | 9 | packages: [bzr, byobu, tmux, python-setuptools, python-twisted, python-argparse, python-txaws, | ||
1125 | 10 | python-zookeeper] | ||
1126 | 11 | runcmd: [sudo apt-get -y install ensemble, sudo mkdir -p /var/lib/ensemble, sudo mkdir | ||
1127 | 12 | -p /var/log/ensemble, 'ENSEMBLE_MACHINE_ID=1 ENSEMBLE_ZOOKEEPER=es.example.internal:2181 | ||
1128 | 13 | python -m ensemble.agents.machine -n --logfile=/var/log/ensemble/machine-agent.log | ||
1129 | 14 | --pidfile=/var/run/ensemble/machine-agent.pid'] | ||
1130 | 15 | ssh_authorized_keys: [zebra] | ||
1131 | 0 | 16 | ||
1132 | === modified file 'ensemble/providers/ec2/tests/test_bootstrap.py' | |||
1133 | --- ensemble/providers/ec2/tests/test_bootstrap.py 2011-08-11 05:59:30 +0000 | |||
1134 | +++ ensemble/providers/ec2/tests/test_bootstrap.py 2011-09-13 10:56:30 +0000 | |||
1135 | @@ -1,6 +1,6 @@ | |||
1136 | 1 | import logging | 1 | import logging |
1139 | 2 | 2 | import os | |
1140 | 3 | from yaml import dump, load | 3 | from yaml import dump |
1141 | 4 | 4 | ||
1142 | 5 | from twisted.internet.defer import succeed | 5 | from twisted.internet.defer import succeed |
1143 | 6 | 6 | ||
1144 | @@ -8,12 +8,11 @@ | |||
1145 | 8 | 8 | ||
1146 | 9 | from ensemble.lib.mocker import MATCH | 9 | from ensemble.lib.mocker import MATCH |
1147 | 10 | from ensemble.lib.testing import TestCase | 10 | from ensemble.lib.testing import TestCase |
1148 | 11 | from ensemble.providers.common.launch import ( | ||
1149 | 12 | BOOTSTRAP_PACKAGES, DEFAULT_REPOSITORIES, DEFAULT_PACKAGES) | ||
1150 | 13 | from ensemble.providers.ec2.machine import EC2ProviderMachine | 11 | from ensemble.providers.ec2.machine import EC2ProviderMachine |
1154 | 14 | from ensemble.state.auth import make_identity | 12 | |
1155 | 15 | 13 | from .common import EC2TestMixin, EC2MachineLaunchMixin | |
1156 | 16 | from .common import EC2TestMixin, EC2MachineLaunchMixin, MATCH_AMI | 14 | |
1157 | 15 | DATA_DIR = os.path.join(os.path.abspath(os.path.dirname(__file__)), "data") | ||
1158 | 17 | 16 | ||
1159 | 18 | 17 | ||
1160 | 19 | class EC2BootstrapTest(EC2TestMixin, EC2MachineLaunchMixin, TestCase): | 18 | class EC2BootstrapTest(EC2TestMixin, EC2MachineLaunchMixin, TestCase): |
1161 | @@ -34,55 +33,21 @@ | |||
1162 | 34 | MATCH(match_string)) | 33 | MATCH(match_string)) |
1163 | 35 | self.mocker.result(succeed(True)) | 34 | self.mocker.result(succeed(True)) |
1164 | 36 | 35 | ||
1166 | 37 | def _mock_launch(self, machine_id): | 36 | def _mock_launch(self): |
1167 | 38 | """Mock launching a bootstrap machine on ec2.""" | 37 | """Mock launching a bootstrap machine on ec2.""" |
1168 | 39 | credentials = "admin:%s" % self.get_config()["admin-secret"] | ||
1169 | 40 | admin_identity = make_identity(credentials) | ||
1170 | 41 | |||
1171 | 42 | def verify_user_data(data): | 38 | def verify_user_data(data): |
1205 | 43 | 39 | expect_path = os.path.join(DATA_DIR, "bootstrap_cloud_init") | |
1206 | 44 | lines = data.split("\n") | 40 | with open(expect_path) as f: |
1207 | 45 | self.assertEqual(lines.pop(0), "#cloud-config") | 41 | expect_cloud_init = f.read() |
1208 | 46 | config = load("\n".join(lines)) | 42 | self.assertEquals(data, expect_cloud_init) |
1176 | 47 | repos = [dict(source=r) for r in DEFAULT_REPOSITORIES] | ||
1177 | 48 | self.assertEqual(config["apt_sources"], repos) | ||
1178 | 49 | self.assertEqual( | ||
1179 | 50 | config["packages"], | ||
1180 | 51 | list(DEFAULT_PACKAGES) + BOOTSTRAP_PACKAGES) | ||
1181 | 52 | self.failUnlessIn("admin-identity", config["runcmd"][-3]) | ||
1182 | 53 | |||
1183 | 54 | script = ( | ||
1184 | 55 | 'ensemble-admin initialize --instance-id=%s' | ||
1185 | 56 | ' --admin-identity="%s"') % ( | ||
1186 | 57 | "$(curl http://169.254.169.254/1.0/meta-data/instance-id)", | ||
1187 | 58 | admin_identity) | ||
1188 | 59 | |||
1189 | 60 | self.assertEqual(config["runcmd"][-3], script) | ||
1190 | 61 | |||
1191 | 62 | script = ( | ||
1192 | 63 | "ENSEMBLE_MACHINE_ID=0 ENSEMBLE_ZOOKEEPER=localhost:2181 " | ||
1193 | 64 | "python -m ensemble.agents.machine -n " | ||
1194 | 65 | "--logfile=/var/log/ensemble/machine-agent.log " | ||
1195 | 66 | "--pidfile=/var/run/ensemble/machine-agent.pid") | ||
1196 | 67 | |||
1197 | 68 | self.assertEqual(config["runcmd"][-2], script) | ||
1198 | 69 | |||
1199 | 70 | provision_agent_script = ( | ||
1200 | 71 | "ENSEMBLE_ZOOKEEPER=localhost:2181 " | ||
1201 | 72 | "python -m ensemble.agents.provision -n " | ||
1202 | 73 | "--logfile=/var/log/ensemble/provision-agent.log " | ||
1203 | 74 | "--pidfile=/var/run/ensemble/provision-agent.pid") | ||
1204 | 75 | self.assertEqual(config["runcmd"][-1], provision_agent_script) | ||
1209 | 76 | return True | 43 | return True |
1210 | 77 | 44 | ||
1211 | 78 | self.ec2.run_instances( | 45 | self.ec2.run_instances( |
1213 | 79 | image_id=MATCH_AMI, | 46 | image_id="ami-default", |
1214 | 80 | instance_type="m1.small", | 47 | instance_type="m1.small", |
1215 | 81 | max_count=1, | 48 | max_count=1, |
1216 | 82 | min_count=1, | 49 | min_count=1, |
1220 | 83 | security_groups=[ | 50 | security_groups=["ensemble-moon", "ensemble-moon-0"], |
1218 | 84 | "%s-%s" % ("ensemble", self.env_name), | ||
1219 | 85 | "%s-%s-%s" % ("ensemble", self.env_name, machine_id)], | ||
1221 | 86 | user_data=MATCH(verify_user_data)) | 51 | user_data=MATCH(verify_user_data)) |
1222 | 87 | 52 | ||
1223 | 88 | def test_launch_bootstrap(self): | 53 | def test_launch_bootstrap(self): |
1224 | @@ -98,7 +63,7 @@ | |||
1225 | 98 | self._mock_create_group() | 63 | self._mock_create_group() |
1226 | 99 | self._mock_create_machine_group(0) | 64 | self._mock_create_machine_group(0) |
1227 | 100 | self._mock_launch_utils(region="us-east-1") | 65 | self._mock_launch_utils(region="us-east-1") |
1229 | 101 | self._mock_launch(0) | 66 | self._mock_launch() |
1230 | 102 | self.mocker.result(succeed([])) | 67 | self.mocker.result(succeed([])) |
1231 | 103 | self._mock_save() | 68 | self._mock_save() |
1232 | 104 | self.mocker.replay() | 69 | self.mocker.replay() |
1233 | @@ -128,7 +93,7 @@ | |||
1234 | 128 | SecurityGroup("ensemble-%s" % self.env_name, "")])) | 93 | SecurityGroup("ensemble-%s" % self.env_name, "")])) |
1235 | 129 | self._mock_create_machine_group(0) | 94 | self._mock_create_machine_group(0) |
1236 | 130 | self._mock_launch_utils(region="us-east-1") | 95 | self._mock_launch_utils(region="us-east-1") |
1238 | 131 | self._mock_launch(0) | 96 | self._mock_launch() |
1239 | 132 | self.mocker.result(succeed([])) | 97 | self.mocker.result(succeed([])) |
1240 | 133 | self._mock_save() | 98 | self._mock_save() |
1241 | 134 | self.mocker.replay() | 99 | self.mocker.replay() |
1242 | @@ -177,7 +142,7 @@ | |||
1243 | 177 | SecurityGroup("ensemble-%s" % self.env_name, "")])) | 142 | SecurityGroup("ensemble-%s" % self.env_name, "")])) |
1244 | 178 | self._mock_create_machine_group(0) | 143 | self._mock_create_machine_group(0) |
1245 | 179 | self._mock_launch_utils(region="us-east-1") | 144 | self._mock_launch_utils(region="us-east-1") |
1247 | 180 | self._mock_launch(0) | 145 | self._mock_launch() |
1248 | 181 | self.mocker.result(succeed([self.get_instance("i-foobar")])) | 146 | self.mocker.result(succeed([self.get_instance("i-foobar")])) |
1249 | 182 | self._mock_save() | 147 | self._mock_save() |
1250 | 183 | self.mocker.replay() | 148 | self.mocker.replay() |
1251 | 184 | 149 | ||
1252 | === modified file 'ensemble/providers/ec2/tests/test_launch.py' | |||
1253 | --- ensemble/providers/ec2/tests/test_launch.py 2011-08-26 12:24:00 +0000 | |||
1254 | +++ ensemble/providers/ec2/tests/test_launch.py 2011-09-13 10:56:30 +0000 | |||
1255 | @@ -1,51 +1,54 @@ | |||
1257 | 1 | from yaml import load | 1 | import os |
1258 | 2 | 2 | ||
1259 | 3 | from twisted.internet.defer import inlineCallbacks, succeed | 3 | from twisted.internet.defer import inlineCallbacks, succeed |
1260 | 4 | 4 | ||
1261 | 5 | from txaws.ec2.model import Instance, SecurityGroup | 5 | from txaws.ec2.model import Instance, SecurityGroup |
1262 | 6 | 6 | ||
1266 | 7 | from ensemble.errors import EnvironmentNotFound, ProviderInteractionError | 7 | from ensemble.errors import ( |
1267 | 8 | from ensemble.providers.common.launch import ( | 8 | EnvironmentNotFound, ProviderError, ProviderInteractionError) |
1265 | 9 | DEFAULT_REPOSITORIES, DEFAULT_PACKAGES) | ||
1268 | 10 | from ensemble.providers.ec2.machine import EC2ProviderMachine | 9 | from ensemble.providers.ec2.machine import EC2ProviderMachine |
1269 | 11 | 10 | ||
1270 | 12 | from ensemble.lib.testing import TestCase | 11 | from ensemble.lib.testing import TestCase |
1271 | 13 | from ensemble.lib.mocker import MATCH | 12 | from ensemble.lib.mocker import MATCH |
1272 | 14 | 13 | ||
1274 | 15 | from .common import EC2TestMixin, EC2MachineLaunchMixin, MATCH_AMI | 14 | from .common import EC2TestMixin, EC2MachineLaunchMixin |
1275 | 15 | |||
1276 | 16 | DATA_DIR = os.path.join(os.path.abspath(os.path.dirname(__file__)), "data") | ||
1277 | 16 | 17 | ||
1278 | 17 | 18 | ||
1279 | 18 | class EC2MachineLaunchTest(EC2TestMixin, EC2MachineLaunchMixin, TestCase): | 19 | class EC2MachineLaunchTest(EC2TestMixin, EC2MachineLaunchMixin, TestCase): |
1280 | 19 | 20 | ||
1282 | 20 | def _mock_launch(self, instance=None, custom_verify=None, machine_id=None): | 21 | def _mock_launch(self, instance, expect_ami="ami-default", |
1283 | 22 | expect_instance_type="m1.small"): | ||
1284 | 21 | 23 | ||
1285 | 22 | def verify_user_data(data): | 24 | def verify_user_data(data): |
1296 | 23 | lines = data.split("\n") | 25 | expect_path = os.path.join(DATA_DIR, "launch_cloud_init") |
1297 | 24 | self.assertEqual(lines.pop(0), "#cloud-config") | 26 | with open(expect_path) as f: |
1298 | 25 | config = load("\n".join(lines)) | 27 | expect_cloud_init = f.read() |
1299 | 26 | repos = [dict(source=r) for r in DEFAULT_REPOSITORIES] | 28 | self.assertEquals(data, expect_cloud_init) |
1290 | 27 | self.assertEqual(config["apt_sources"], repos) | ||
1291 | 28 | self.assertEqual(config["packages"], DEFAULT_PACKAGES) | ||
1292 | 29 | self.assertEqual(config["machine-data"]["machine-id"], "machine-1") | ||
1293 | 30 | |||
1294 | 31 | if custom_verify: | ||
1295 | 32 | custom_verify(config) | ||
1300 | 33 | return True | 29 | return True |
1301 | 34 | 30 | ||
1302 | 35 | self.ec2.run_instances( | 31 | self.ec2.run_instances( |
1305 | 36 | image_id=MATCH_AMI, | 32 | image_id=expect_ami, |
1306 | 37 | instance_type="m1.small", | 33 | instance_type=expect_instance_type, |
1307 | 38 | max_count=1, | 34 | max_count=1, |
1308 | 39 | min_count=1, | 35 | min_count=1, |
1312 | 40 | security_groups=[ | 36 | security_groups=["ensemble-moon", "ensemble-moon-1"], |
1310 | 41 | "%s-%s" % ("ensemble", self.env_name), | ||
1311 | 42 | "%s-%s-%s" % ("ensemble", self.env_name, machine_id)], | ||
1313 | 43 | user_data=MATCH(verify_user_data)) | 37 | user_data=MATCH(verify_user_data)) |
1314 | 44 | 38 | ||
1319 | 45 | if instance: | 39 | self.mocker.result(succeed([instance])) |
1320 | 46 | self.mocker.result(succeed([instance])) | 40 | |
1321 | 47 | else: | 41 | def test_bad_data(self): |
1322 | 48 | self.mocker.result(succeed([])) | 42 | self.mocker.replay() |
1323 | 43 | d = self.get_provider().start_machine({}) | ||
1324 | 44 | self.assertFailure(d, ProviderError) | ||
1325 | 45 | |||
1326 | 46 | def verify(error): | ||
1327 | 47 | self.assertEquals( | ||
1328 | 48 | str(error), | ||
1329 | 49 | "Cannot launch a machine without specifying a machine-id") | ||
1330 | 50 | d.addCallback(verify) | ||
1331 | 51 | return d | ||
1332 | 49 | 52 | ||
1333 | 50 | def test_provider_launch(self): | 53 | def test_provider_launch(self): |
1334 | 51 | """ | 54 | """ |
1335 | @@ -55,18 +58,17 @@ | |||
1336 | 55 | self.ec2.describe_security_groups() | 58 | self.ec2.describe_security_groups() |
1337 | 56 | self.mocker.result(succeed([])) | 59 | self.mocker.result(succeed([])) |
1338 | 57 | self._mock_create_group() | 60 | self._mock_create_group() |
1340 | 58 | self._mock_create_machine_group("machine-1") | 61 | self._mock_create_machine_group("1") |
1341 | 59 | self._mock_launch_utils(region="us-east-1") | 62 | self._mock_launch_utils(region="us-east-1") |
1342 | 60 | self._mock_get_zookeeper_hosts() | 63 | self._mock_get_zookeeper_hosts() |
1345 | 61 | self._mock_launch( | 64 | self._mock_launch(self.get_instance("i-foobar")) |
1344 | 62 | self.get_instance("i-foobar"), machine_id="machine-1") | ||
1346 | 63 | self.mocker.replay() | 65 | self.mocker.replay() |
1347 | 64 | 66 | ||
1348 | 65 | def verify_result(result): | 67 | def verify_result(result): |
1349 | 66 | (machine,) = result | 68 | (machine,) = result |
1350 | 67 | self.assert_machine(machine, "i-foobar", "") | 69 | self.assert_machine(machine, "i-foobar", "") |
1351 | 68 | provider = self.get_provider() | 70 | provider = self.get_provider() |
1353 | 69 | d = provider.start_machine({"machine-id": "machine-1"}) | 71 | d = provider.start_machine({"machine-id": "1"}) |
1354 | 70 | d.addCallback(verify_result) | 72 | d.addCallback(verify_result) |
1355 | 71 | return d | 73 | return d |
1356 | 72 | 74 | ||
1357 | @@ -78,15 +80,14 @@ | |||
1358 | 78 | 80 | ||
1359 | 79 | self.ec2.describe_security_groups() | 81 | self.ec2.describe_security_groups() |
1360 | 80 | self.mocker.result(succeed([security_group])) | 82 | self.mocker.result(succeed([security_group])) |
1362 | 81 | self._mock_create_machine_group("machine-1") | 83 | self._mock_create_machine_group("1") |
1363 | 82 | self._mock_launch_utils(region="us-east-1") | 84 | self._mock_launch_utils(region="us-east-1") |
1364 | 83 | self._mock_get_zookeeper_hosts() | 85 | self._mock_get_zookeeper_hosts() |
1366 | 84 | self._mock_launch(instance, machine_id="machine-1") | 86 | self._mock_launch(instance) |
1367 | 85 | self.mocker.replay() | 87 | self.mocker.replay() |
1368 | 86 | 88 | ||
1369 | 87 | provider = self.get_provider() | 89 | provider = self.get_provider() |
1372 | 88 | provided_machines = yield provider.start_machine( | 90 | provided_machines = yield provider.start_machine({"machine-id": "1"}) |
1371 | 89 | {"machine-id": "machine-1"}) | ||
1373 | 90 | self.assertEqual(len(provided_machines), 1) | 91 | self.assertEqual(len(provided_machines), 1) |
1374 | 91 | self.assertTrue(isinstance(provided_machines[0], EC2ProviderMachine)) | 92 | self.assertTrue(isinstance(provided_machines[0], EC2ProviderMachine)) |
1375 | 92 | self.assertEqual( | 93 | self.assertEqual( |
1376 | @@ -97,21 +98,20 @@ | |||
1377 | 97 | """Verify that the launch works if the machine security group exists""" | 98 | """Verify that the launch works if the machine security group exists""" |
1378 | 98 | instance = Instance("i-foobar", "running", dns_name="x1.example.com") | 99 | instance = Instance("i-foobar", "running", dns_name="x1.example.com") |
1379 | 99 | machine_group = SecurityGroup( | 100 | machine_group = SecurityGroup( |
1381 | 100 | "ensemble-moon-machine-1", "some description") | 101 | "ensemble-moon-1", "some description") |
1382 | 101 | 102 | ||
1383 | 102 | self.ec2.describe_security_groups() | 103 | self.ec2.describe_security_groups() |
1384 | 103 | self.mocker.result(succeed([machine_group])) | 104 | self.mocker.result(succeed([machine_group])) |
1385 | 104 | self._mock_create_group() | 105 | self._mock_create_group() |
1388 | 105 | self._mock_delete_machine_group("machine-1") # delete existing sg | 106 | self._mock_delete_machine_group("1") # delete existing sg |
1389 | 106 | self._mock_create_machine_group("machine-1") # then recreate | 107 | self._mock_create_machine_group("1") # then recreate |
1390 | 107 | self._mock_launch_utils(region="us-east-1") | 108 | self._mock_launch_utils(region="us-east-1") |
1391 | 108 | self._mock_get_zookeeper_hosts() | 109 | self._mock_get_zookeeper_hosts() |
1393 | 109 | self._mock_launch(instance, machine_id="machine-1") | 110 | self._mock_launch(instance) |
1394 | 110 | self.mocker.replay() | 111 | self.mocker.replay() |
1395 | 111 | 112 | ||
1396 | 112 | provider = self.get_provider() | 113 | provider = self.get_provider() |
1399 | 113 | provided_machines = yield provider.start_machine( | 114 | provided_machines = yield provider.start_machine({"machine-id": "1"}) |
1398 | 114 | {"machine-id": "machine-1"}) | ||
1400 | 115 | self.assertEqual(len(provided_machines), 1) | 115 | self.assertEqual(len(provided_machines), 1) |
1401 | 116 | self.assertTrue(isinstance(provided_machines[0], EC2ProviderMachine)) | 116 | self.assertTrue(isinstance(provided_machines[0], EC2ProviderMachine)) |
1402 | 117 | self.assertEqual( | 117 | self.assertEqual( |
1403 | @@ -125,49 +125,27 @@ | |||
1404 | 125 | that security group, generally because it is still shutting | 125 | that security group, generally because it is still shutting |
1405 | 126 | down.""" | 126 | down.""" |
1406 | 127 | machine_group = SecurityGroup( | 127 | machine_group = SecurityGroup( |
1408 | 128 | "ensemble-moon-machine-1", "some description") | 128 | "ensemble-moon-1", "some description") |
1409 | 129 | 129 | ||
1410 | 130 | self.ec2.describe_security_groups() | 130 | self.ec2.describe_security_groups() |
1411 | 131 | self.mocker.result(succeed([machine_group])) | 131 | self.mocker.result(succeed([machine_group])) |
1412 | 132 | self._mock_create_group() | 132 | self._mock_create_group() |
1414 | 133 | self._mock_delete_machine_group_was_deleted("machine-1") # sg is gone! | 133 | self._mock_delete_machine_group_was_deleted("1") # sg is gone! |
1415 | 134 | self._mock_launch_utils(region="us-east-1") | 134 | self._mock_launch_utils(region="us-east-1") |
1416 | 135 | self._mock_get_zookeeper_hosts() | 135 | self._mock_get_zookeeper_hosts() |
1417 | 136 | self.mocker.replay() | 136 | self.mocker.replay() |
1418 | 137 | 137 | ||
1419 | 138 | provider = self.get_provider() | 138 | provider = self.get_provider() |
1420 | 139 | ex = yield self.assertFailure( | 139 | ex = yield self.assertFailure( |
1422 | 140 | provider.start_machine({"machine-id": "machine-1"}), | 140 | provider.start_machine({"machine-id": "1"}), |
1423 | 141 | ProviderInteractionError) | 141 | ProviderInteractionError) |
1424 | 142 | self.assertEqual( | 142 | self.assertEqual( |
1425 | 143 | str(ex), | 143 | str(ex), |
1426 | 144 | "Unexpected EC2Error deleting security group " | 144 | "Unexpected EC2Error deleting security group " |
1450 | 145 | "ensemble-moon-machine-1: There are active instances using " | 145 | "ensemble-moon-1: There are active instances using security group " |
1451 | 146 | "security group 'ensemble-moon-machine-1'") | 146 | "'ensemble-moon-1'") |
1429 | 147 | |||
1430 | 148 | def test_provider_type_machine_variable(self): | ||
1431 | 149 | """The provider type is available via cloud-init.""" | ||
1432 | 150 | self.ec2.describe_security_groups() | ||
1433 | 151 | self.mocker.result(succeed([])) | ||
1434 | 152 | self._mock_create_group() | ||
1435 | 153 | self._mock_create_machine_group("machine-1") | ||
1436 | 154 | self._mock_launch_utils(region="us-east-1") | ||
1437 | 155 | self._mock_get_zookeeper_hosts() | ||
1438 | 156 | |||
1439 | 157 | def verify_provider_type(data): | ||
1440 | 158 | self.assertEqual( | ||
1441 | 159 | data["machine-data"]["ensemble-provider-type"], "ec2") | ||
1442 | 160 | |||
1443 | 161 | self._mock_launch(custom_verify=verify_provider_type, | ||
1444 | 162 | machine_id="machine-1") | ||
1445 | 163 | self.mocker.replay() | ||
1446 | 164 | |||
1447 | 165 | provider = self.get_provider() | ||
1448 | 166 | d = provider.start_machine({"machine-id": "machine-1"}) | ||
1449 | 167 | return d | ||
1452 | 168 | 147 | ||
1453 | 169 | def test_launch_with_no_ensemble_s3_state(self): | 148 | def test_launch_with_no_ensemble_s3_state(self): |
1454 | 170 | # XXX not really testing anything EC2-specific | ||
1455 | 171 | """ | 149 | """ |
1456 | 172 | Attempting to launch without any ensemble saved state, means | 150 | Attempting to launch without any ensemble saved state, means |
1457 | 173 | we can't provide a way for a launched instance to connect | 151 | we can't provide a way for a launched instance to connect |
1458 | @@ -178,12 +156,11 @@ | |||
1459 | 178 | self.mocker.replay() | 156 | self.mocker.replay() |
1460 | 179 | 157 | ||
1461 | 180 | provider = self.get_provider() | 158 | provider = self.get_provider() |
1463 | 181 | d = provider.start_machine({"machine-id": "machine-1"}) | 159 | d = provider.start_machine({"machine-id": "1"}) |
1464 | 182 | self.assertFailure(d, EnvironmentNotFound) | 160 | self.assertFailure(d, EnvironmentNotFound) |
1465 | 183 | return d | 161 | return d |
1466 | 184 | 162 | ||
1467 | 185 | def test_launch_with_no_ensemble_zookeeper_hosts(self): | 163 | def test_launch_with_no_ensemble_zookeeper_hosts(self): |
1468 | 186 | # XXX not really testing anything EC2-specific | ||
1469 | 187 | """ | 164 | """ |
1470 | 188 | Attempting to launch without any ensemble zookeeper hosts, means | 165 | Attempting to launch without any ensemble zookeeper hosts, means |
1471 | 189 | we can't provide a way for a launched instance to connect | 166 | we can't provide a way for a launched instance to connect |
1472 | @@ -194,55 +171,42 @@ | |||
1473 | 194 | self.mocker.replay() | 171 | self.mocker.replay() |
1474 | 195 | 172 | ||
1475 | 196 | provider = self.get_provider() | 173 | provider = self.get_provider() |
1477 | 197 | d = provider.start_machine({"machine-id": "machine-1"}) | 174 | d = provider.start_machine({"machine-id": "1"}) |
1478 | 198 | self.assertFailure(d, EnvironmentNotFound) | 175 | self.assertFailure(d, EnvironmentNotFound) |
1479 | 199 | return d | 176 | return d |
1480 | 200 | 177 | ||
1484 | 201 | def test_launch_starts_machine_agent(self): | 178 | def test_launch_options_known_instance_type(self): |
1482 | 202 | # XXX not really testing anything EC2-specific | ||
1483 | 203 | """A launch'd machine should have a machine agent.""" | ||
1485 | 204 | self.ec2.describe_security_groups() | 179 | self.ec2.describe_security_groups() |
1486 | 205 | self.mocker.result(succeed([])) | 180 | self.mocker.result(succeed([])) |
1487 | 206 | self._mock_create_group() | 181 | self._mock_create_group() |
1489 | 207 | self._mock_create_machine_group("machine-1") | 182 | self._mock_create_machine_group("1") |
1490 | 208 | self._mock_launch_utils(region="us-east-1") | 183 | self._mock_launch_utils(region="us-east-1") |
1491 | 209 | self._mock_get_zookeeper_hosts() | 184 | self._mock_get_zookeeper_hosts() |
1492 | 210 | |||
1493 | 211 | def verify_machine_agent(data): | ||
1494 | 212 | script = ("ENSEMBLE_MACHINE_ID=machine-1 " | ||
1495 | 213 | "ENSEMBLE_ZOOKEEPER=es.example.internal:2181 " | ||
1496 | 214 | "python -m ensemble.agents.machine " | ||
1497 | 215 | "-n --logfile=/var/log/ensemble/machine-agent.log " | ||
1498 | 216 | "--pidfile=/var/run/ensemble/machine-agent.pid") | ||
1499 | 217 | self.assertEqual(script, data["runcmd"][-1]) | ||
1500 | 218 | |||
1501 | 219 | self._mock_launch(custom_verify=verify_machine_agent, | ||
1502 | 220 | machine_id="machine-1") | ||
1503 | 221 | self.mocker.replay() | ||
1504 | 222 | |||
1505 | 223 | provider = self.get_provider() | ||
1506 | 224 | d = provider.start_machine({"machine-id": "machine-1"}) | ||
1507 | 225 | return d | ||
1508 | 226 | |||
1509 | 227 | def _mock_launch_with_ami_params(self, get_ami_kwargs): | ||
1510 | 228 | self.ec2.describe_security_groups() | ||
1511 | 229 | self.mocker.result(succeed([])) | ||
1512 | 230 | self._mock_create_group() | ||
1513 | 231 | self._mock_create_machine_group("machine-1") | ||
1514 | 232 | self._mock_launch_utils(**get_ami_kwargs) | ||
1515 | 233 | self._mock_get_zookeeper_hosts() | ||
1516 | 234 | self._mock_launch( | 185 | self._mock_launch( |
1518 | 235 | self.get_instance("i-foobar"), machine_id="machine-1") | 186 | self.get_instance("i-foobar"), expect_instance_type="m1.ginormous") |
1519 | 187 | self.mocker.replay() | ||
1520 | 188 | |||
1521 | 189 | provider = self.get_provider() | ||
1522 | 190 | provider.config["default-instance-type"] = "m1.ginormous" | ||
1523 | 191 | return provider.start_machine({"machine-id": "1"}) | ||
1524 | 192 | |||
1525 | 193 | def _mock_launch_with_ami_params(self, get_ami_kwargs, expect_ami=None): | ||
1526 | 194 | expect_ami = expect_ami or "ami-default" | ||
1527 | 195 | self.ec2.describe_security_groups() | ||
1528 | 196 | self.mocker.result(succeed([])) | ||
1529 | 197 | self._mock_create_group() | ||
1530 | 198 | self._mock_create_machine_group("1") | ||
1531 | 199 | self._mock_launch_utils(ami_name=expect_ami, **get_ami_kwargs) | ||
1532 | 200 | self._mock_get_zookeeper_hosts() | ||
1533 | 201 | self._mock_launch(self.get_instance("i-foobar"), expect_ami) | ||
1534 | 236 | 202 | ||
1535 | 237 | def test_launch_options_known_ami(self): | 203 | def test_launch_options_known_ami(self): |
1537 | 238 | self._mock_launch_with_ami_params({}) | 204 | self._mock_launch_with_ami_params({}, expect_ami="ami-different") |
1538 | 239 | self.mocker.replay() | 205 | self.mocker.replay() |
1539 | 240 | 206 | ||
1540 | 241 | provider = self.get_provider() | 207 | provider = self.get_provider() |
1545 | 242 | # TODO shouldn't something be testing we actually launch | 208 | provider.config["default-image-id"] = "ami-different" |
1546 | 243 | # an instance of the expected AMI? | 209 | return provider.start_machine({"machine-id": "1"}) |
1543 | 244 | provider.config["default-image-id"] = "ami-XXXXXX" | ||
1544 | 245 | return provider.start_machine({"machine-id": "machine-1"}) | ||
1547 | 246 | 210 | ||
1548 | 247 | def test_launch_options_region(self): | 211 | def test_launch_options_region(self): |
1549 | 248 | self._mock_launch_with_ami_params({"region": "somewhere-else-1"}) | 212 | self._mock_launch_with_ami_params({"region": "somewhere-else-1"}) |
1550 | @@ -250,21 +214,22 @@ | |||
1551 | 250 | 214 | ||
1552 | 251 | provider = self.get_provider() | 215 | provider = self.get_provider() |
1553 | 252 | provider.config["region"] = "somewhere-else-1" | 216 | provider.config["region"] = "somewhere-else-1" |
1555 | 253 | return provider.start_machine({"machine-id": "machine-1"}) | 217 | return provider.start_machine({"machine-id": "1"}) |
1556 | 254 | 218 | ||
1558 | 255 | def test_launch_options_other(self): | 219 | def test_launch_options_custom_image_options(self): |
1559 | 256 | self._mock_launch_with_ami_params({ | 220 | self._mock_launch_with_ami_params({ |
1560 | 257 | "region": "us-east-1", | 221 | "region": "us-east-1", |
1564 | 258 | "ubuntu_release_name": "choleric", | 222 | "ubuntu_release": "choleric", |
1565 | 259 | "arch": "quantum86", | 223 | "architecture": "quantum86", |
1566 | 260 | "ebs": "maybe", | 224 | "persistent_storage": "maybe", |
1567 | 261 | "daily": "perhaps"}) | 225 | "daily": "perhaps"}) |
1568 | 262 | self.mocker.replay() | 226 | self.mocker.replay() |
1569 | 263 | 227 | ||
1570 | 264 | provider = self.get_provider() | 228 | provider = self.get_provider() |
1577 | 265 | return provider.start_machine({ | 229 | constraints = { |
1578 | 266 | "machine-id": "machine-1", | 230 | "ubuntu_release": "choleric", |
1579 | 267 | "image_release_name": "choleric", | 231 | "architecture": "quantum86", |
1580 | 268 | "image_arch": "quantum86", | 232 | "persistent_storage": "maybe", |
1581 | 269 | "image_ebs": "maybe", | 233 | "daily": "perhaps"} |
1582 | 270 | "image_daily": "perhaps"}) | 234 | return provider.start_machine({"machine-id": "1", |
1583 | 235 | "constraints": constraints}) | ||
1584 | 271 | 236 | ||
1585 | === modified file 'ensemble/providers/ec2/tests/test_utils.py' | |||
1586 | --- ensemble/providers/ec2/tests/test_utils.py 2011-08-12 19:54:54 +0000 | |||
1587 | +++ ensemble/providers/ec2/tests/test_utils.py 2011-09-13 10:56:30 +0000 | |||
1588 | @@ -1,14 +1,12 @@ | |||
1589 | 1 | import inspect | 1 | import inspect |
1590 | 2 | import os | 2 | import os |
1591 | 3 | 3 | ||
1592 | 4 | |||
1593 | 5 | from twisted.internet.defer import succeed | 4 | from twisted.internet.defer import succeed |
1594 | 6 | 5 | ||
1595 | 6 | from ensemble.lib.testing import TestCase | ||
1596 | 7 | from ensemble.providers import ec2 | 7 | from ensemble.providers import ec2 |
1599 | 8 | from ensemble.providers.common.utils import format_cloud_init | 8 | from ensemble.providers.ec2.utils import get_current_ami, get_image_id |
1598 | 9 | from ensemble.providers.ec2.utils import get_current_ami, get_launch_options | ||
1600 | 10 | 9 | ||
1601 | 11 | from ensemble.lib.testing import TestCase | ||
1602 | 12 | 10 | ||
1603 | 13 | IMAGE_URI_TEMPLATE = "\ | 11 | IMAGE_URI_TEMPLATE = "\ |
1604 | 14 | http://uec-images.ubuntu.com/query/%s/server/released.current.txt" | 12 | http://uec-images.ubuntu.com/query/%s/server/released.current.txt" |
1605 | @@ -17,7 +15,7 @@ | |||
1606 | 17 | os.path.dirname(inspect.getabsfile(ec2)), "tests", "data") | 15 | os.path.dirname(inspect.getabsfile(ec2)), "tests", "data") |
1607 | 18 | 16 | ||
1608 | 19 | 17 | ||
1610 | 20 | class EC2UtilsTest(TestCase): | 18 | class GetCurrentAmiTest(TestCase): |
1611 | 21 | 19 | ||
1612 | 22 | def test_umatched_ami(self): | 20 | def test_umatched_ami(self): |
1613 | 23 | """ | 21 | """ |
1614 | @@ -28,7 +26,7 @@ | |||
1615 | 28 | page(IMAGE_URI_TEMPLATE % "lucid") | 26 | page(IMAGE_URI_TEMPLATE % "lucid") |
1616 | 29 | self.mocker.result(succeed("")) | 27 | self.mocker.result(succeed("")) |
1617 | 30 | self.mocker.replay() | 28 | self.mocker.replay() |
1619 | 31 | d = get_current_ami(ubuntu_release_name="lucid") | 29 | d = get_current_ami(ubuntu_release="lucid") |
1620 | 32 | self.failUnlessFailure(d, LookupError) | 30 | self.failUnlessFailure(d, LookupError) |
1621 | 33 | return d | 31 | return d |
1622 | 34 | 32 | ||
1623 | @@ -40,7 +38,7 @@ | |||
1624 | 40 | open(os.path.join(IMAGE_DATA_DIR, "lucid.txt")).read())) | 38 | open(os.path.join(IMAGE_DATA_DIR, "lucid.txt")).read())) |
1625 | 41 | 39 | ||
1626 | 42 | self.mocker.replay() | 40 | self.mocker.replay() |
1628 | 43 | d = get_current_ami(ubuntu_release_name="lucid") | 41 | d = get_current_ami(ubuntu_release="lucid") |
1629 | 44 | 42 | ||
1630 | 45 | def verify_result(result): | 43 | def verify_result(result): |
1631 | 46 | self.assertEqual(result, "ami-714ba518") | 44 | self.assertEqual(result, "ami-714ba518") |
1632 | @@ -56,7 +54,7 @@ | |||
1633 | 56 | succeed(open( | 54 | succeed(open( |
1634 | 57 | os.path.join(IMAGE_DATA_DIR, "lucid.txt")).read())) | 55 | os.path.join(IMAGE_DATA_DIR, "lucid.txt")).read())) |
1635 | 58 | self.mocker.replay() | 56 | self.mocker.replay() |
1637 | 59 | d = get_current_ami(ubuntu_release_name="lucid", region="us-west-1") | 57 | d = get_current_ami(ubuntu_release="lucid", region="us-west-1") |
1638 | 60 | 58 | ||
1639 | 61 | def verify_result(result): | 59 | def verify_result(result): |
1640 | 62 | self.assertEqual(result, "ami-cb97c68e") | 60 | self.assertEqual(result, "ami-cb97c68e") |
1641 | @@ -74,7 +72,7 @@ | |||
1642 | 74 | self.mocker.result(succeed( | 72 | self.mocker.result(succeed( |
1643 | 75 | open(os.path.join(IMAGE_DATA_DIR, "lucid.txt")).read())) | 73 | open(os.path.join(IMAGE_DATA_DIR, "lucid.txt")).read())) |
1644 | 76 | self.mocker.replay() | 74 | self.mocker.replay() |
1646 | 77 | d = get_current_ami(ubuntu_release_name="lucid", ebs=False) | 75 | d = get_current_ami(ubuntu_release="lucid", persistent_storage=False) |
1647 | 78 | 76 | ||
1648 | 79 | def verify_result(result): | 77 | def verify_result(result): |
1649 | 80 | self.assertEqual(result, "ami-2d4aa444") | 78 | self.assertEqual(result, "ami-2d4aa444") |
1650 | @@ -82,101 +80,37 @@ | |||
1651 | 82 | d.addCallback(verify_result) | 80 | d.addCallback(verify_result) |
1652 | 83 | return d | 81 | return d |
1653 | 84 | 82 | ||
1751 | 85 | def test_get_launch_options(self): | 83 | |
1752 | 86 | """ | 84 | class GetImageIdTest(TestCase): |
1753 | 87 | F{get_launch_options}, returns a dictionary of arguments for the ec2 | 85 | |
1754 | 88 | client's run instance method. | 86 | def test_default_image_id(self): |
1755 | 89 | """ | 87 | d = get_image_id({"default-image-id": "ami-burble"}, {}) |
1756 | 90 | config = {} | 88 | d.addCallback(self.assertEquals, "ami-burble") |
1757 | 91 | config["default-image-id"] = "ami-foobar" | 89 | return d |
1758 | 92 | config["default-instance-type"] = "m1.medium" | 90 | |
1759 | 93 | authorized_keys = "zebra" | 91 | def test_no_constraints(self): |
1760 | 94 | repositories = ["ppa:ensemble/ppa"] | 92 | get_current_ami_m = self.mocker.replace(get_current_ami) |
1761 | 95 | packages = ["python-zookeeper", "zookeeperd"] | 93 | get_current_ami_m(region="us-east-1") |
1762 | 96 | 94 | self.mocker.result(succeed("ami-giggle")) | |
1763 | 97 | d = get_launch_options( | 95 | self.mocker.replay() |
1764 | 98 | config, | 96 | |
1765 | 99 | authorized_keys=authorized_keys, | 97 | d = get_image_id({}, {}) |
1766 | 100 | packages=packages, | 98 | d.addCallback(self.assertEquals, "ami-giggle") |
1767 | 101 | repositories=repositories, | 99 | return d |
1768 | 102 | magic=[1, 2, 3]) | 100 | |
1769 | 103 | 101 | def test_uses_constraints(self): | |
1770 | 104 | cloud_init_config = format_cloud_init( | 102 | get_current_ami_m = self.mocker.replace(get_current_ami) |
1771 | 105 | authorized_keys, packages, repositories, | 103 | get_current_ami_m(ubuntu_release="serendipitous", architecture="x512", |
1772 | 106 | data={"magic": [1, 2, 3]}) | 104 | daily=False, persistent_storage=True, |
1773 | 107 | 105 | region="blah-north-6") | |
1774 | 108 | def verify_result(result): | 106 | self.mocker.result(succeed("ami-tinkle")) |
1775 | 109 | self.assertTrue(isinstance(result, dict)) | 107 | self.mocker.replay() |
1776 | 110 | self.assertEqual(result["image_id"], "ami-foobar") | 108 | |
1777 | 111 | self.assertEqual(result["user_data"], cloud_init_config) | 109 | constraints = { |
1778 | 112 | self.assertEqual(result["security_groups"], []) | 110 | "architecture": "x512", |
1779 | 113 | 111 | "ubuntu_release": "serendipitous", | |
1780 | 114 | d.addCallback(verify_result) | 112 | "persistent_storage": True, |
1781 | 115 | return d | 113 | "daily": False} |
1782 | 116 | 114 | d = get_image_id({"region": "blah-north-6"}, constraints) | |
1783 | 117 | def test_get_launch_options_defaults(self): | 115 | d.addCallback(self.assertEquals, "ami-tinkle") |
1687 | 118 | """ | ||
1688 | 119 | F{get_launch_options}, returns a dictionary of arguments for | ||
1689 | 120 | the ec2 client's run instance method. A known set of default | ||
1690 | 121 | is utilized. | ||
1691 | 122 | """ | ||
1692 | 123 | |||
1693 | 124 | page = self.mocker.replace("twisted.web.client.getPage") | ||
1694 | 125 | page(IMAGE_URI_TEMPLATE % "natty") | ||
1695 | 126 | self.mocker.result( | ||
1696 | 127 | succeed(open( | ||
1697 | 128 | os.path.join(IMAGE_DATA_DIR, "natty.txt")).read())) | ||
1698 | 129 | self.mocker.replay() | ||
1699 | 130 | |||
1700 | 131 | config = {} | ||
1701 | 132 | d = get_launch_options(config, "zebra") | ||
1702 | 133 | |||
1703 | 134 | def verify_result(result): | ||
1704 | 135 | self.assertTrue(isinstance(result, dict)) | ||
1705 | 136 | self.assertEqual(result["image_id"], "ami-06ad526f") | ||
1706 | 137 | self.assertEqual(result["instance_type"], "m1.small") | ||
1707 | 138 | |||
1708 | 139 | d.addCallback(verify_result) | ||
1709 | 140 | return d | ||
1710 | 141 | |||
1711 | 142 | def test_get_launch_options_with_params(self): | ||
1712 | 143 | """ | ||
1713 | 144 | Image parameters can be specified to determine which ami is to | ||
1714 | 145 | be launched. Image paramters specified are not passed to the | ||
1715 | 146 | instance via user data. | ||
1716 | 147 | """ | ||
1717 | 148 | page = self.mocker.replace("twisted.web.client.getPage") | ||
1718 | 149 | page(IMAGE_URI_TEMPLATE % "natty") | ||
1719 | 150 | self.mocker.result( | ||
1720 | 151 | succeed(open( | ||
1721 | 152 | os.path.join(IMAGE_DATA_DIR, "natty.txt")).read())) | ||
1722 | 153 | self.mocker.replay() | ||
1723 | 154 | |||
1724 | 155 | d = get_launch_options( | ||
1725 | 156 | {"default-instance-type": "m1.large"}, | ||
1726 | 157 | "zebra", image_arch="amd64", image_ebs=False) | ||
1727 | 158 | |||
1728 | 159 | def verify_result(result): | ||
1729 | 160 | self.assertTrue(isinstance(result, dict)) | ||
1730 | 161 | self.assertEqual(result["image_id"], "ami-68ad5201") | ||
1731 | 162 | self.assertEqual(result["instance_type"], "m1.large") | ||
1732 | 163 | self.assertNotIn( | ||
1733 | 164 | "arch", result["user_data"]) | ||
1734 | 165 | self.assertNotIn( | ||
1735 | 166 | "ebs", result["user_data"]) | ||
1736 | 167 | |||
1737 | 168 | d.addCallback(verify_result) | ||
1738 | 169 | return d | ||
1739 | 170 | |||
1740 | 171 | def test_configured_default_image_has_precedence_over_region(self): | ||
1741 | 172 | """If the server machine image is specified it takes precendence over | ||
1742 | 173 | region.""" | ||
1743 | 174 | d = get_launch_options( | ||
1744 | 175 | {"region": "eu-west-1", "default-image-id": "21down"}, | ||
1745 | 176 | authorized_keys="") | ||
1746 | 177 | |||
1747 | 178 | def verify_result(result): | ||
1748 | 179 | self.assertEqual(result["image_id"], "21down") | ||
1749 | 180 | |||
1750 | 181 | d.addCallback(verify_result) | ||
1784 | 182 | return d | 116 | return d |
1785 | 183 | 117 | ||
1786 | === modified file 'ensemble/providers/ec2/utils.py' | |||
1787 | --- ensemble/providers/ec2/utils.py 2011-08-14 19:24:03 +0000 | |||
1788 | +++ ensemble/providers/ec2/utils.py 2011-09-13 10:56:30 +0000 | |||
1789 | @@ -5,22 +5,11 @@ | |||
1790 | 5 | from twisted.web.client import getPage | 5 | from twisted.web.client import getPage |
1791 | 6 | from twisted.internet.defer import succeed | 6 | from twisted.internet.defer import succeed |
1792 | 7 | 7 | ||
1793 | 8 | from ensemble.providers.common.utils import format_cloud_init | ||
1794 | 9 | |||
1795 | 10 | log = logging.getLogger("ensemble.ec2") | 8 | log = logging.getLogger("ensemble.ec2") |
1796 | 11 | 9 | ||
1809 | 12 | # default image for each region. | 10 | _CURRENT_IMAGE_URI_TEMPLATE = ( |
1810 | 13 | AMI_REGION_MAP = { | 11 | "http://uec-images.ubuntu.com/query/" |
1811 | 14 | "us-east-1": "ami-d2ba45bb", | 12 | "%(ubuntu_release_name)s/%(variant)s/%(version)s.current.txt") |
1800 | 15 | "us-west-1": "ami-15693a50", | ||
1801 | 16 | "ap-northeast-1": "ami-80bb1181", | ||
1802 | 17 | "ap-southeast-1": "ami-b4671ee6", | ||
1803 | 18 | "eu-west-1": "ami-0991a67d" | ||
1804 | 19 | } | ||
1805 | 20 | |||
1806 | 21 | BASE_IMAGE_URI = "http://uec-images.ubuntu.com/query/" | ||
1807 | 22 | CURRENT_IMAGE_URI = BASE_IMAGE_URI + \ | ||
1808 | 23 | "%(ubuntu_release_name)s/%(variant)s/%(version)s.current.txt" | ||
1812 | 24 | 13 | ||
1813 | 25 | 14 | ||
1814 | 26 | def get_region_uri(region): | 15 | def get_region_uri(region): |
1815 | @@ -28,17 +17,18 @@ | |||
1816 | 28 | return "https://ec2.%s.amazonaws.com" % region | 17 | return "https://ec2.%s.amazonaws.com" % region |
1817 | 29 | 18 | ||
1818 | 30 | 19 | ||
1822 | 31 | def get_current_ami(ubuntu_release_name="natty", arch="i386", ebs=True, | 20 | # XXX ideally should come from latest available or client release name. |
1823 | 32 | region="us-east-1", daily=False, desktop=False, | 21 | def get_current_ami(ubuntu_release="natty", architecture="i386", |
1824 | 33 | url_fetch=None): | 22 | persistent_storage=True, region="us-east-1", daily=False, |
1825 | 23 | desktop=False, url_fetch=None): | ||
1826 | 34 | """Get the latest ami for the last release of ubuntu.""" | 24 | """Get the latest ami for the last release of ubuntu.""" |
1827 | 35 | data = {} | 25 | data = {} |
1829 | 36 | data["ubuntu_release_name"] = ubuntu_release_name | 26 | data["ubuntu_release_name"] = ubuntu_release |
1830 | 37 | data["version"] = daily and "daily" or "released" | 27 | data["version"] = daily and "daily" or "released" |
1831 | 38 | data["variant"] = desktop and "desktop" or "server" | 28 | data["variant"] = desktop and "desktop" or "server" |
1833 | 39 | ebs_match = ebs and "ebs" or "instance-store" | 29 | ebs_match = persistent_storage and "ebs" or "instance-store" |
1834 | 40 | 30 | ||
1836 | 41 | url = CURRENT_IMAGE_URI % data | 31 | url = _CURRENT_IMAGE_URI_TEMPLATE % data |
1837 | 42 | url_fetch = url_fetch or getPage | 32 | url_fetch = url_fetch or getPage |
1838 | 43 | 33 | ||
1839 | 44 | d = url_fetch(url) | 34 | d = url_fetch(url) |
1840 | @@ -48,60 +38,18 @@ | |||
1841 | 48 | for tokens in csv.reader(data_stream, "excel-tab"): | 38 | for tokens in csv.reader(data_stream, "excel-tab"): |
1842 | 49 | if tokens[4] != ebs_match: | 39 | if tokens[4] != ebs_match: |
1843 | 50 | continue | 40 | continue |
1845 | 51 | if tokens[5] == arch and tokens[6] == region: | 41 | if tokens[5] == architecture and tokens[6] == region: |
1846 | 52 | return tokens[7] | 42 | return tokens[7] |
1848 | 53 | raise LookupError((ubuntu_release_name, arch, region, | 43 | raise LookupError((ubuntu_release, architecture, region, |
1849 | 54 | data["version"], data["variant"])) | 44 | data["version"], data["variant"])) |
1850 | 55 | 45 | ||
1851 | 56 | d.addCallback(extract_ami) | 46 | d.addCallback(extract_ami) |
1852 | 57 | return d | 47 | return d |
1853 | 58 | 48 | ||
1854 | 59 | 49 | ||
1866 | 60 | def get_launch_options(config, authorized_keys, packages=None, | 50 | def get_image_id(config, constraints): |
1867 | 61 | repositories=None, scripts=None, **kw): | 51 | image_id = config.get("default-image-id", None) |
1868 | 62 | """Determine launch options for the machine. | 52 | if image_id: |
1869 | 63 | 53 | return succeed(image_id) | |
1859 | 64 | Given an ensemble environment configuration, extract the proper machine | ||
1860 | 65 | information to be used for launching the machine via the ec2 api. | ||
1861 | 66 | """ | ||
1862 | 67 | |||
1863 | 68 | instance_type = config.get("default-instance-type", "m1.small") | ||
1864 | 69 | |||
1865 | 70 | # XXX ideally should come from latest available or client release name. | ||
1870 | 71 | region = config.get("region", "us-east-1") | 54 | region = config.get("region", "us-east-1") |
1907 | 72 | image_id = config.get("default-image-id", None) | 55 | return get_current_ami(region=region, **constraints) |
1872 | 73 | |||
1873 | 74 | # Lookup a standard ami by region, if none is specified, use any | ||
1874 | 75 | # specified image specification to locate an image, and remove the | ||
1875 | 76 | # image specification so it doesn't propogate to user data. | ||
1876 | 77 | if not image_id: | ||
1877 | 78 | params = {} | ||
1878 | 79 | for a_name, p_name in ( | ||
1879 | 80 | ("image_release_name", "ubuntu_release_name"), | ||
1880 | 81 | ("image_arch", "arch"), | ||
1881 | 82 | ("image_ebs", "ebs"), | ||
1882 | 83 | ("image_daily", "daily")): | ||
1883 | 84 | if a_name in kw: | ||
1884 | 85 | params[p_name] = kw[a_name] | ||
1885 | 86 | del kw[a_name] | ||
1886 | 87 | params["region"] = region | ||
1887 | 88 | d = get_current_ami(**params) | ||
1888 | 89 | else: | ||
1889 | 90 | d = succeed(image_id) | ||
1890 | 91 | |||
1891 | 92 | user_data = format_cloud_init( | ||
1892 | 93 | authorized_keys=authorized_keys, | ||
1893 | 94 | packages=packages, | ||
1894 | 95 | repositories=repositories, | ||
1895 | 96 | scripts=scripts, | ||
1896 | 97 | data=kw) | ||
1897 | 98 | |||
1898 | 99 | def on_machine_image(current_image_id): | ||
1899 | 100 | return {"image_id": current_image_id, | ||
1900 | 101 | "min_count": 1, | ||
1901 | 102 | "max_count": 1, | ||
1902 | 103 | "instance_type": instance_type, | ||
1903 | 104 | "user_data": user_data, | ||
1904 | 105 | "security_groups": []} | ||
1905 | 106 | d.addCallback(on_machine_image) | ||
1906 | 107 | return d | ||
1908 | 108 | 56 | ||
1909 | === modified file 'ensemble/providers/orchestra/__init__.py' | |||
1910 | --- ensemble/providers/orchestra/__init__.py 2011-08-25 16:41:03 +0000 | |||
1911 | +++ ensemble/providers/orchestra/__init__.py 2011-09-13 10:56:30 +0000 | |||
1912 | @@ -1,6 +1,6 @@ | |||
1913 | 1 | import logging | 1 | import logging |
1914 | 2 | 2 | ||
1916 | 3 | from twisted.internet.defer import inlineCallbacks, returnValue, succeed | 3 | from twisted.internet.defer import fail, inlineCallbacks, returnValue, succeed |
1917 | 4 | 4 | ||
1918 | 5 | from ensemble.errors import ProviderError | 5 | from ensemble.errors import ProviderError |
1919 | 6 | from ensemble.providers.common.base import MachineProviderBase | 6 | from ensemble.providers.common.base import MachineProviderBase |
1920 | @@ -29,13 +29,16 @@ | |||
1921 | 29 | def start_machine(self, machine_data, master=False): | 29 | def start_machine(self, machine_data, master=False): |
1922 | 30 | """Start a machine in the provider. | 30 | """Start a machine in the provider. |
1923 | 31 | 31 | ||
1926 | 32 | @param machine_data: a dictionary of data to pass along to the newly | 32 | `machine_data`: a dictionary describing the machine to be launched. |
1925 | 33 | launched machine. | ||
1927 | 34 | 33 | ||
1930 | 35 | @param master: if True, machine will initialize the ensemble admin | 34 | `master`: if True, machine will initialize the ensemble admin and run |
1931 | 36 | and run a provisioning agent. | 35 | a provisioning agent, in addition to running a machine agent |
1932 | 37 | """ | 36 | """ |
1934 | 38 | return OrchestraLaunchMachine(self, master).run(machine_data) | 37 | if "machine-id" not in machine_data: |
1935 | 38 | return fail(ProviderError( | ||
1936 | 39 | "Cannot launch a machine without specifying a machine-id")) | ||
1937 | 40 | machine_id = machine_data["machine-id"] | ||
1938 | 41 | return OrchestraLaunchMachine(self, master).run(machine_id) | ||
1939 | 39 | 42 | ||
1940 | 40 | @inlineCallbacks | 43 | @inlineCallbacks |
1941 | 41 | def get_machines(self, instance_ids=()): | 44 | def get_machines(self, instance_ids=()): |
1942 | 42 | 45 | ||
1943 | === modified file 'ensemble/providers/orchestra/launch.py' | |||
1944 | --- ensemble/providers/orchestra/launch.py 2011-08-11 23:27:45 +0000 | |||
1945 | +++ ensemble/providers/orchestra/launch.py 2011-09-13 10:56:30 +0000 | |||
1946 | @@ -5,14 +5,8 @@ | |||
1947 | 5 | from twisted.internet.defer import inlineCallbacks, returnValue | 5 | from twisted.internet.defer import inlineCallbacks, returnValue |
1948 | 6 | 6 | ||
1949 | 7 | from ensemble.providers.common.launch import LaunchMachine | 7 | from ensemble.providers.common.launch import LaunchMachine |
1958 | 8 | from ensemble.providers.common.utils import format_cloud_init | 8 | |
1959 | 9 | from ensemble.providers.orchestra.machine import machine_from_dict | 9 | from .machine import machine_from_dict |
1952 | 10 | |||
1953 | 11 | _INSTANCE_ID_VAR = "ENSEMBLE_INSTANCE_ID" | ||
1954 | 12 | |||
1955 | 13 | _SET_INSTANCE_ID_TEMPLATE = "export %s=%%s" % _INSTANCE_ID_VAR | ||
1956 | 14 | |||
1957 | 15 | _CLOUD_INIT_KEYS = "authorized_keys packages repositories scripts".split() | ||
1960 | 16 | 10 | ||
1961 | 17 | _LATE_COMMAND_TEMPLATE = """ | 11 | _LATE_COMMAND_TEMPLATE = """ |
1962 | 18 | seed_d=/var/lib/cloud/seed/nocloud-net | 12 | seed_d=/var/lib/cloud/seed/nocloud-net |
1963 | @@ -36,17 +30,6 @@ | |||
1964 | 36 | "' %s /root/late-command") | 30 | "' %s /root/late-command") |
1965 | 37 | 31 | ||
1966 | 38 | 32 | ||
1967 | 39 | def _cloud_init_vars(raw_vars): | ||
1968 | 40 | # XXX this is nasty; see bug#820892 | ||
1969 | 41 | vars = raw_vars.copy() | ||
1970 | 42 | data = {} | ||
1971 | 43 | for key in vars.keys(): | ||
1972 | 44 | if key not in _CLOUD_INIT_KEYS: | ||
1973 | 45 | data[key] = vars.pop(key) | ||
1974 | 46 | vars["data"] = data | ||
1975 | 47 | return vars | ||
1976 | 48 | |||
1977 | 49 | |||
1978 | 50 | def _base64_gzip(content): | 33 | def _base64_gzip(content): |
1979 | 51 | gzipped = StringIO() | 34 | gzipped = StringIO() |
1980 | 52 | gzip_file = GzipFile(fileobj=gzipped, mode="wb", compresslevel=9) | 35 | gzip_file = GzipFile(fileobj=gzipped, mode="wb", compresslevel=9) |
1981 | @@ -62,28 +45,34 @@ | |||
1982 | 62 | class OrchestraLaunchMachine(LaunchMachine): | 45 | class OrchestraLaunchMachine(LaunchMachine): |
1983 | 63 | 46 | ||
1984 | 64 | @inlineCallbacks | 47 | @inlineCallbacks |
1986 | 65 | def start_machine(self, variables, machine_id): | 48 | def start_machine(self, machine_id, zookeepers): |
1987 | 49 | """Actually launch a Cobbler system. | ||
1988 | 50 | |||
1989 | 51 | `machine_id`: the ensemble machine ID | ||
1990 | 52 | |||
1991 | 53 | `zookeepers`: a list of `ProviderMachine`s already running zookeeper, | ||
1992 | 54 | which the machine will need to connect to. May be empty. | ||
1993 | 55 | |||
1994 | 56 | Returns a singe-entry list containing a ProviderMachine for the | ||
1995 | 57 | new instance | ||
1996 | 58 | """ | ||
1997 | 66 | cobbler = self._provider.cobbler | 59 | cobbler = self._provider.cobbler |
1998 | 67 | instance_id = yield cobbler.acquire_system() | 60 | instance_id = yield cobbler.acquire_system() |
2000 | 68 | ks_meta = self._build_ks_meta(instance_id, variables) | 61 | |
2001 | 62 | cloud_init = self._create_cloud_init(machine_id, zookeepers) | ||
2002 | 63 | cloud_init.set_provider_type("orchestra") | ||
2003 | 64 | cloud_init.set_instance_id_accessor(instance_id) | ||
2004 | 65 | ks_meta = self._build_ks_meta( | ||
2005 | 66 | cloud_init.render(), machine_id, instance_id) | ||
2006 | 67 | |||
2007 | 69 | yield cobbler.set_on_system(instance_id, "ks_meta", ks_meta) | 68 | yield cobbler.set_on_system(instance_id, "ks_meta", ks_meta) |
2008 | 70 | yield cobbler.start_system(instance_id) | 69 | yield cobbler.start_system(instance_id) |
2009 | 71 | (info,) = yield cobbler.describe_systems(instance_id) | 70 | (info,) = yield cobbler.describe_systems(instance_id) |
2010 | 72 | returnValue([machine_from_dict(info)]) | 71 | returnValue([machine_from_dict(info)]) |
2011 | 73 | 72 | ||
2023 | 74 | def get_instance_id_command(self): | 73 | def _build_ks_meta(self, user_data, machine_id, instance_id): |
2013 | 75 | return "$" + _INSTANCE_ID_VAR | ||
2014 | 76 | |||
2015 | 77 | def _build_ks_meta(self, instance_id, variables): | ||
2016 | 78 | # XXX it's a little bit dirty to modify scripts at this stage; | ||
2017 | 79 | # but we don't actually know the instance_id at get_variables time. | ||
2018 | 80 | set_instance_id = _SET_INSTANCE_ID_TEMPLATE % instance_id | ||
2019 | 81 | variables["scripts"].insert(0, set_instance_id) | ||
2020 | 82 | |||
2021 | 83 | cloud_init_vars = _cloud_init_vars(variables) | ||
2022 | 84 | user_data = format_cloud_init(**cloud_init_vars) | ||
2024 | 85 | late_command = _LATE_COMMAND_TEMPLATE % (instance_id, user_data) | 74 | late_command = _LATE_COMMAND_TEMPLATE % (instance_id, user_data) |
2025 | 86 | encoded = _base64_gzip(late_command) | 75 | encoded = _base64_gzip(late_command) |
2026 | 87 | ensemble_late_command = _KSMETA_LATE_COMMAND_TEMPLATE % encoded | 76 | ensemble_late_command = _KSMETA_LATE_COMMAND_TEMPLATE % encoded |
2029 | 88 | return _ks_meta(MACHINE_ID=variables["machine-id"], | 77 | return _ks_meta( |
2030 | 89 | ENSEMBLE_LATE_COMMAND=ensemble_late_command) | 78 | MACHINE_ID=machine_id, ENSEMBLE_LATE_COMMAND=ensemble_late_command) |
2031 | 90 | 79 | ||
2032 | === modified file 'ensemble/providers/orchestra/tests/common.py' | |||
2033 | --- ensemble/providers/orchestra/tests/common.py 2011-08-11 23:27:45 +0000 | |||
2034 | +++ ensemble/providers/orchestra/tests/common.py 2011-09-13 10:56:30 +0000 | |||
2035 | @@ -108,7 +108,8 @@ | |||
2036 | 108 | late_command = _extract_late_command(leftover) | 108 | late_command = _extract_late_command(leftover) |
2037 | 109 | 109 | ||
2038 | 110 | expect_path = os.path.join(DATA_DIR, late_command_filename) | 110 | expect_path = os.path.join(DATA_DIR, late_command_filename) |
2040 | 111 | expect_late_command = open(expect_path).read() | 111 | with open(expect_path) as f: |
2041 | 112 | expect_late_command = f.read() | ||
2042 | 112 | self.assertEquals(late_command, expect_late_command) | 113 | self.assertEquals(late_command, expect_late_command) |
2043 | 113 | return True | 114 | return True |
2044 | 114 | return verify | 115 | return verify |
2045 | 115 | 116 | ||
2046 | === modified file 'ensemble/providers/orchestra/tests/data/bootstrap_late_command' | |||
2047 | --- ensemble/providers/orchestra/tests/data/bootstrap_late_command 2011-09-07 22:06:57 +0000 | |||
2048 | +++ ensemble/providers/orchestra/tests/data/bootstrap_late_command 2011-09-13 10:56:30 +0000 | |||
2049 | @@ -11,13 +11,13 @@ | |||
2050 | 11 | apt-upgrade: true | 11 | apt-upgrade: true |
2051 | 12 | apt_sources: | 12 | apt_sources: |
2052 | 13 | - {source: 'ppa:ensemble/ppa'} | 13 | - {source: 'ppa:ensemble/ppa'} |
2054 | 14 | machine-data: {ensemble-zookeeper-hosts: 'localhost:2181', machine-id: '0'} | 14 | machine-data: {ensemble-provider-type: orchestra, ensemble-zookeeper-hosts: 'localhost:2181', |
2055 | 15 | machine-id: '0'} | ||
2056 | 15 | output: {all: '| tee -a /var/log/cloud-init-output.log'} | 16 | output: {all: '| tee -a /var/log/cloud-init-output.log'} |
2057 | 16 | packages: [bzr, byobu, tmux, python-setuptools, python-twisted, python-argparse, python-txaws, | 17 | packages: [bzr, byobu, tmux, python-setuptools, python-twisted, python-argparse, python-txaws, |
2062 | 17 | python-zookeeper, bzr, default-jre-headless, zookeeper, zookeeperd] | 18 | python-zookeeper, default-jre-headless, zookeeper, zookeeperd] |
2063 | 18 | runcmd: [export ENSEMBLE_INSTANCE_ID=winston-uid, sudo apt-get -y install ensemble, | 19 | runcmd: [sudo apt-get -y install ensemble, sudo mkdir -p /var/lib/ensemble, sudo mkdir |
2064 | 19 | sudo mkdir -p /var/lib/ensemble, sudo mkdir -p /var/log/ensemble, 'ensemble-admin | 20 | -p /var/log/ensemble, 'ensemble-admin initialize --instance-id=winston-uid --admin-identity=admin:qRBXC1ubEEUqRL6wcBhgmc9xkaY=', |
2061 | 20 | initialize --instance-id=$ENSEMBLE_INSTANCE_ID --admin-identity="admin:qRBXC1ubEEUqRL6wcBhgmc9xkaY="', | ||
2065 | 21 | 'ENSEMBLE_MACHINE_ID=0 ENSEMBLE_ZOOKEEPER=localhost:2181 python -m ensemble.agents.machine | 21 | 'ENSEMBLE_MACHINE_ID=0 ENSEMBLE_ZOOKEEPER=localhost:2181 python -m ensemble.agents.machine |
2066 | 22 | -n --logfile=/var/log/ensemble/machine-agent.log --pidfile=/var/run/ensemble/machine-agent.pid', | 22 | -n --logfile=/var/log/ensemble/machine-agent.log --pidfile=/var/run/ensemble/machine-agent.pid', |
2067 | 23 | 'ENSEMBLE_ZOOKEEPER=localhost:2181 python -m ensemble.agents.provision -n --logfile=/var/log/ensemble/provision-agent.log | 23 | 'ENSEMBLE_ZOOKEEPER=localhost:2181 python -m ensemble.agents.provision -n --logfile=/var/log/ensemble/provision-agent.log |
2068 | 24 | 24 | ||
2069 | === modified file 'ensemble/providers/orchestra/tests/data/launch_late_command' | |||
2070 | --- ensemble/providers/orchestra/tests/data/launch_late_command 2011-09-07 22:06:57 +0000 | |||
2071 | +++ ensemble/providers/orchestra/tests/data/launch_late_command 2011-09-13 10:56:30 +0000 | |||
2072 | @@ -11,13 +11,14 @@ | |||
2073 | 11 | apt-upgrade: true | 11 | apt-upgrade: true |
2074 | 12 | apt_sources: | 12 | apt_sources: |
2075 | 13 | - {source: 'ppa:ensemble/ppa'} | 13 | - {source: 'ppa:ensemble/ppa'} |
2077 | 14 | machine-data: {ensemble-zookeeper-hosts: 'jennifer:2181', machine-id: '42'} | 14 | machine-data: {ensemble-provider-type: orchestra, ensemble-zookeeper-hosts: 'jennifer:2181', |
2078 | 15 | machine-id: '42'} | ||
2079 | 15 | output: {all: '| tee -a /var/log/cloud-init-output.log'} | 16 | output: {all: '| tee -a /var/log/cloud-init-output.log'} |
2080 | 16 | packages: [bzr, byobu, tmux, python-setuptools, python-twisted, python-argparse, python-txaws, | 17 | packages: [bzr, byobu, tmux, python-setuptools, python-twisted, python-argparse, python-txaws, |
2081 | 17 | python-zookeeper] | 18 | python-zookeeper] |
2085 | 18 | runcmd: [export ENSEMBLE_INSTANCE_ID=winston-uid, sudo apt-get -y install ensemble, | 19 | runcmd: [sudo apt-get -y install ensemble, sudo mkdir -p /var/lib/ensemble, sudo mkdir |
2086 | 19 | sudo mkdir -p /var/lib/ensemble, sudo mkdir -p /var/log/ensemble, 'ENSEMBLE_MACHINE_ID=42 | 20 | -p /var/log/ensemble, 'ENSEMBLE_MACHINE_ID=42 ENSEMBLE_ZOOKEEPER=jennifer:2181 |
2087 | 20 | ENSEMBLE_ZOOKEEPER=jennifer:2181 python -m ensemble.agents.machine -n --logfile=/var/log/ensemble/machine-agent.log | 21 | python -m ensemble.agents.machine -n --logfile=/var/log/ensemble/machine-agent.log |
2088 | 21 | --pidfile=/var/run/ensemble/machine-agent.pid'] | 22 | --pidfile=/var/run/ensemble/machine-agent.pid'] |
2089 | 22 | ssh_authorized_keys: [this-is-a-public-key] | 23 | ssh_authorized_keys: [this-is-a-public-key] |
2090 | 23 | 24 | ||
2091 | 24 | 25 | ||
2092 | === modified file 'ensemble/providers/orchestra/tests/test_launch.py' | |||
2093 | --- ensemble/providers/orchestra/tests/test_launch.py 2011-08-26 12:24:00 +0000 | |||
2094 | +++ ensemble/providers/orchestra/tests/test_launch.py 2011-09-13 10:56:30 +0000 | |||
2095 | @@ -11,6 +11,12 @@ | |||
2096 | 11 | def test_bad_data(self): | 11 | def test_bad_data(self): |
2097 | 12 | d = self.get_provider().start_machine({}) | 12 | d = self.get_provider().start_machine({}) |
2098 | 13 | self.assertFailure(d, ProviderError) | 13 | self.assertFailure(d, ProviderError) |
2099 | 14 | |||
2100 | 15 | def verify(error): | ||
2101 | 16 | self.assertEquals( | ||
2102 | 17 | str(error), | ||
2103 | 18 | "Cannot launch a machine without specifying a machine-id") | ||
2104 | 19 | d.addCallback(verify) | ||
2105 | 14 | return d | 20 | return d |
2106 | 15 | 21 | ||
2107 | 16 | def test_no_zookeeper(self): | 22 | def test_no_zookeeper(self): |
This looks _very_ nice. Thanks William!
+1, with the following sorted:
[1]
+ self._zookeeper _names = [m.private_dns_name for m in machines] hosts(self) : _names[ :]
(...)
+ def _zookeeper_
+ all_names = self._zookeeper
The branch has two different names for the same data in different
formats. The better approach is the opposite: the same name with
different actions that give a hint about the undelying operation.
I suggest using "zookeeper_hosts" for both, and naming the last hosts or something similar.
function as join_zookeeper_
[2]
+ def _validate(self):
+ missing = []
Very nice logic there.
[3]
+ def _create_ cloud_init( self, machine_id, zookeepers): config add_ssh_ key(get_ user_authorized _keys(config) ) set_machine_ id(machine_ id) set_zookeeper_ machines( zookeepers) get("ensemble- branch" ): set_ensemble_ source( config[ "ensemble- branch" ]) enable_ bootstrap( ) set_zookeeper_ secret( config[ "admin- secret" ])
+ config = self._provider.
+ cloud_init = CloudInit()
+ cloud_init.
+ cloud_init.
+ cloud_init.
+ if config.
+ cloud_init.
+ if self._master:
+ cloud_init.
+ cloud_init.
+ return cloud_init
Beautiful! :-)
[4]
+ instance_type = self._provider. config. get( instance- type", "m1.small") id(self. _provider. config, self._constraints) groups( machine_ id) ec2.run_ instances( type=instance_ type, groups= security_ groups, user_data)
+ "default-
+ image_id = yield get_image_
+ security_groups = yield self._ensure_
+
+ instances = yield self._provider.
+ image_id=image_id,
+ instance_
+ min_count=1,
+ max_count=1,
+ security_
+ user_data=
Very clean as well, thanks!
[5]
+def get_image_ id(config, constraints): release_ name", storage" : "ebs"} constraints( constraints) : NAMES.items( ): constraint_ name] constraints( constraints)
(...)
+_KWARG_NAMES = {
+ "ubuntu_release": "ubuntu_
+ "architecture": "arch",
+ "daily": "daily",
+ "persistent_
(...)
+def _convert_
+ kwargs = {}
+ for (constraint_name, kwarg_name) in _KWARG_
+ if constraint_name in constraints:
+ kwargs[kwarg_name] = constraints[
+ return kwargs
(...)
+ kwargs = _convert_
Why are we doing this? These names in the right-hand-side of the
_KWARG_NAMES are keyword arguments of get_current_ami, which is a function
on the same file (!). The left-hand-side ones are not even used in
any sensible way in our current implementation.
Let's just unify the naming and avoid any conversions entirely. Feel
free to pick sensible ones.
[6]
Again, this looks very nice. Please just make sure you merge trunk into
this, and do one final real-world test on EC2 before committing back
into trunk.