Merge lp:~hazmat/pyjuju/unit-agent-resolved into lp:pyjuju
- unit-agent-resolved
- Merge into trunk
Status: | Merged |
---|---|
Approved by: | Gustavo Niemeyer |
Approved revision: | 275 |
Merged at revision: | 224 |
Proposed branch: | lp:~hazmat/pyjuju/unit-agent-resolved |
Merge into: | lp:pyjuju |
Prerequisite: | lp:~hazmat/pyjuju/ensemble-resolved |
Diff against target: |
1276 lines (+718/-94) 7 files modified
ensemble/control/tests/test_resolved.py (+0/-1) ensemble/state/service.py (+18/-5) ensemble/state/tests/test_service.py (+71/-1) ensemble/unit/lifecycle.py (+85/-21) ensemble/unit/tests/test_lifecycle.py (+242/-3) ensemble/unit/tests/test_workflow.py (+154/-12) ensemble/unit/workflow.py (+148/-51) |
To merge this branch: | bzr merge lp:~hazmat/pyjuju/unit-agent-resolved |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Gustavo Niemeyer | Approve | ||
Review via email: mp+59713@code.launchpad.net |
This proposal supersedes a proposal from 2011-05-02.
Commit message
Description of the change
This implements the unit lifecycle resolving unit relations, and additional changes to enable resolution (transition actions receiving hook execution flags).
The branch is large enough that i'm going to split the remaining unit agent resolving unit relations into another branch.
Gustavo Niemeyer (niemeyer) wrote : | # |
Gustavo Niemeyer (niemeyer) wrote : | # |
[1]
+ def do_retry_
+ return self._invoke_
+ self._lifecycle
We've already debated this live over a voice call, and had already
covered the topic in a previous conversation: I think it is a mistake
to introduce variables which define how the transition should work.
This is increasing the complexity of actions in an unpredictable way
(a single action with parameters could handle *all* the possible
transitions in the state machine).
Kapil Thangavelu (hazmat) wrote : | # |
Changed over to using additional transitions, its a bit nicer now, thanks.
Gustavo Niemeyer (niemeyer) wrote : | # |
Nice, thanks for the changes. Looking good!
+1, taking these in consideration:
[2]
+ def do_retry_
The fire_hooks seems to be a left over.
[3]
+ return self._invoke_
+ self._lifecycle
+ lambda x: self._invoke_
+ self._lifecycle
(...)
+ return self._invoke_
+ self._lifecycle
+ lambda x: self._invoke_
+ self._lifecycle
Breaking these down would make them more readable than the huge one-liners.
[4]
+# Another interesting issue, process recovery using the on disk state,
+# is complicated by consistency to the the in memory state, which
+# won't be directly recoverable anymore without some state specific
Sweet. Thanks for capturing those ideas.
[5]
+ # If the unit lifecycle isn't running we shouldn't process
+ # any relation resolutions.
+ if not self._running:
+ self._log.
+ self._watching_
+ raise StopWatcher()
It's not clear what's the intention here, and test coverage also
looks a bit poor in this area (e.g. replacing everything under the "if" by
"return" still passes tests fine).
What do we want to happen in this case, and what's the rationale
behind it?
[6]
+ if self._client.
+ yield self._process_
What's the "if connected" test about? It feels like pretty much every
interaction with zk could have such a test. Why is this location
special?
Kapil Thangavelu (hazmat) wrote : | # |
Excerpts from Gustavo Niemeyer's message of Thu May 05 02:01:15 UTC 2011:
> Review: Approve
> Nice, thanks for the changes. Looking good!
>
> +1, taking these in consideration:
>
> [5]
>
> + # If the unit lifecycle isn't running we shouldn't process
> + # any relation resolutions.
> + if not self._running:
> + self._log.
> + self._watching_
> + raise StopWatcher()
>
> It's not clear what's the intention here, and test coverage also
> looks a bit poor in this area (e.g. replacing everything under the "if" by
> "return" still passes tests fine).
>
> What do we want to happen in this case, and what's the rationale
> behind it?
>
The notion is if we're not running, the watcher terminates, when we start watching
again the watcher is restarted, if it doesn't already exist. Its designed to
avoid concurrent watchers, and to allow for watch termination.
The design rationale being if a unit is not running, it needs to be resolved before
its unit relations can be fixed.
Being able to correctly tests runs into another behavior on the unit lifecycle,
where we automatically repair unit relations when the unit is started. I've
introduced a separate error state for relations in this branch, so we can choose
not to have unit relation errors automatically recovered, in which case the logic
in question can be tested easily, by bringing the unit back online, and the verifying
the resolve action was performed.
>
> [6]
>
> + if self._client.
> + yield self._process_
>
> What's the "if connected" test about? It feels like pretty much every
> interaction with zk could have such a test. Why is this location
> special?
>
Its typically done in watch callbacks, such that an async background execution while the test
is closing, does not trigger a zookeeper closing exception, and minimizes errors.
- 276. By Kapil Thangavelu
-
Merged ensemble-resolved into unit-agent-
resolved. - 277. By Kapil Thangavelu
-
Merged ensemble-resolved into unit-agent-
resolved. - 278. By Kapil Thangavelu
-
Merged ensemble-resolved into unit-agent-
resolved. - 279. By Kapil Thangavelu
-
address review comments re formatting
Kapil Thangavelu (hazmat) wrote : | # |
Excerpts from Kapil Thangavelu's message of Thu May 05 13:26:02 -0400 2011:
> Excerpts from Gustavo Niemeyer's message of Thu May 05 02:01:15 UTC 2011:
> > Review: Approve
> > Nice, thanks for the changes. Looking good!
> >
> > +1, taking these in consideration:
> >
> > [5]
> >
> > + # If the unit lifecycle isn't running we shouldn't process
> > + # any relation resolutions.
> > + if not self._running:
> > + self._log.
> > + self._watching_
> > + raise StopWatcher()
> >
> > It's not clear what's the intention here, and test coverage also
> > looks a bit poor in this area (e.g. replacing everything under the "if" by
> > "return" still passes tests fine).
> >
> > What do we want to happen in this case, and what's the rationale
> > behind it?
> >
>
> The notion is if we're not running, the watcher terminates, when we start watching
> again the watcher is restarted, if it doesn't already exist. Its designed to
> avoid concurrent watchers, and to allow for watch termination.
>
> The design rationale being if a unit is not running, it needs to be resolved before
> its unit relations can be fixed.
>
> Being able to correctly tests runs into another behavior on the unit lifecycle,
> where we automatically repair unit relations when the unit is started. I've
> introduced a separate error state for relations in this branch, so we can choose
> not to have unit relation errors automatically recovered, in which case the logic
> in question can be tested easily, by bringing the unit back online, and the verifying
> the resolve action was performed.
Just to be clear this is a test for the overall functionality in
test_resolved_
But it because of the above constraint regarding automatic recover on start, it can
only verify that the watcher has stopped, not that the recovery proceeds normally
after restart, so it instead verifies that the resolved setting has persisted
past the end of the watcher.
Preview Diff
1 | === modified file 'ensemble/control/tests/test_resolved.py' | |||
2 | --- ensemble/control/tests/test_resolved.py 2011-04-28 17:09:36 +0000 | |||
3 | +++ ensemble/control/tests/test_resolved.py 2011-05-05 17:43:23 +0000 | |||
4 | @@ -2,7 +2,6 @@ | |||
5 | 2 | from yaml import dump | 2 | from yaml import dump |
6 | 3 | 3 | ||
7 | 4 | from ensemble.control import main | 4 | from ensemble.control import main |
8 | 5 | from ensemble.control.resolved import resolved | ||
9 | 6 | from ensemble.control.tests.common import ControlToolTest | 5 | from ensemble.control.tests.common import ControlToolTest |
10 | 7 | from ensemble.formula.tests.test_repository import RepositoryTestBase | 6 | from ensemble.formula.tests.test_repository import RepositoryTestBase |
11 | 8 | 7 | ||
12 | 9 | 8 | ||
13 | === modified file 'ensemble/state/service.py' | |||
14 | --- ensemble/state/service.py 2011-05-05 16:18:34 +0000 | |||
15 | +++ ensemble/state/service.py 2011-05-05 17:43:23 +0000 | |||
16 | @@ -16,11 +16,10 @@ | |||
17 | 16 | ServiceUnitStateMachineAlreadyAssigned, ServiceStateNameInUse, | 16 | ServiceUnitStateMachineAlreadyAssigned, ServiceStateNameInUse, |
18 | 17 | BadDescriptor, BadServiceStateName, NoUnusedMachines, | 17 | BadDescriptor, BadServiceStateName, NoUnusedMachines, |
19 | 18 | ServiceUnitDebugAlreadyEnabled, ServiceUnitResolvedAlreadyEnabled, | 18 | ServiceUnitDebugAlreadyEnabled, ServiceUnitResolvedAlreadyEnabled, |
21 | 19 | ServiceUnitRelationResolvedAlreadyEnabled) | 19 | ServiceUnitRelationResolvedAlreadyEnabled, StopWatcher) |
22 | 20 | from ensemble.state.formula import FormulaStateManager | 20 | from ensemble.state.formula import FormulaStateManager |
23 | 21 | from ensemble.state.relation import ServiceRelationState, RelationStateManager | 21 | from ensemble.state.relation import ServiceRelationState, RelationStateManager |
24 | 22 | from ensemble.state.machine import _public_machine_id, MachineState | 22 | from ensemble.state.machine import _public_machine_id, MachineState |
25 | 23 | |||
26 | 24 | from ensemble.state.utils import remove_tree, dict_merge, YAMLState | 23 | from ensemble.state.utils import remove_tree, dict_merge, YAMLState |
27 | 25 | 24 | ||
28 | 26 | RETRY_HOOKS = 1000 | 25 | RETRY_HOOKS = 1000 |
29 | @@ -516,7 +515,7 @@ | |||
30 | 516 | its `write` method invoked to publish the state to Zookeeper. | 515 | its `write` method invoked to publish the state to Zookeeper. |
31 | 517 | """ | 516 | """ |
32 | 518 | config_node = YAMLState(self._client, | 517 | config_node = YAMLState(self._client, |
34 | 519 | "/services/%s/config" % self._internal_id ) | 518 | "/services/%s/config" % self._internal_id) |
35 | 520 | yield config_node.read() | 519 | yield config_node.read() |
36 | 521 | returnValue(config_node) | 520 | returnValue(config_node) |
37 | 522 | 521 | ||
38 | @@ -944,9 +943,13 @@ | |||
39 | 944 | def watcher(change_event): | 943 | def watcher(change_event): |
40 | 945 | if not self._client.connected: | 944 | if not self._client.connected: |
41 | 946 | returnValue(None) | 945 | returnValue(None) |
42 | 946 | |||
43 | 947 | exists_d, watch_d = self._client.exists_and_watch( | 947 | exists_d, watch_d = self._client.exists_and_watch( |
44 | 948 | self._unit_resolve_path) | 948 | self._unit_resolve_path) |
46 | 949 | yield callback(change_event) | 949 | try: |
47 | 950 | yield callback(change_event) | ||
48 | 951 | except StopWatcher: | ||
49 | 952 | returnValue(None) | ||
50 | 950 | watch_d.addCallback(watcher) | 953 | watch_d.addCallback(watcher) |
51 | 951 | 954 | ||
52 | 952 | exists_d, watch_d = self._client.exists_and_watch( | 955 | exists_d, watch_d = self._client.exists_and_watch( |
53 | @@ -958,6 +961,8 @@ | |||
54 | 958 | callback_d = maybeDeferred(callback, bool(exists)) | 961 | callback_d = maybeDeferred(callback, bool(exists)) |
55 | 959 | callback_d.addCallback( | 962 | callback_d.addCallback( |
56 | 960 | lambda x: watch_d.addCallback(watcher) and x) | 963 | lambda x: watch_d.addCallback(watcher) and x) |
57 | 964 | callback_d.addErrback( | ||
58 | 965 | lambda failure: failure.trap(StopWatcher)) | ||
59 | 961 | 966 | ||
60 | 962 | @property | 967 | @property |
61 | 963 | def _relation_resolved_path(self): | 968 | def _relation_resolved_path(self): |
62 | @@ -1033,6 +1038,8 @@ | |||
63 | 1033 | 1038 | ||
64 | 1034 | @inlineCallbacks | 1039 | @inlineCallbacks |
65 | 1035 | def clear_relation_resolved(self): | 1040 | def clear_relation_resolved(self): |
66 | 1041 | """ Clear the relation resolved setting. | ||
67 | 1042 | """ | ||
68 | 1036 | try: | 1043 | try: |
69 | 1037 | yield self._client.delete(self._relation_resolved_path) | 1044 | yield self._client.delete(self._relation_resolved_path) |
70 | 1038 | except zookeeper.NoNodeException: | 1045 | except zookeeper.NoNodeException: |
71 | @@ -1055,7 +1062,11 @@ | |||
72 | 1055 | returnValue(None) | 1062 | returnValue(None) |
73 | 1056 | exists_d, watch_d = self._client.exists_and_watch( | 1063 | exists_d, watch_d = self._client.exists_and_watch( |
74 | 1057 | self._relation_resolved_path) | 1064 | self._relation_resolved_path) |
76 | 1058 | yield callback(change_event) | 1065 | try: |
77 | 1066 | yield callback(change_event) | ||
78 | 1067 | except StopWatcher: | ||
79 | 1068 | returnValue(None) | ||
80 | 1069 | |||
81 | 1059 | watch_d.addCallback(watcher) | 1070 | watch_d.addCallback(watcher) |
82 | 1060 | 1071 | ||
83 | 1061 | exists_d, watch_d = self._client.exists_and_watch( | 1072 | exists_d, watch_d = self._client.exists_and_watch( |
84 | @@ -1068,6 +1079,8 @@ | |||
85 | 1068 | callback_d = maybeDeferred(callback, bool(exists)) | 1079 | callback_d = maybeDeferred(callback, bool(exists)) |
86 | 1069 | callback_d.addCallback( | 1080 | callback_d.addCallback( |
87 | 1070 | lambda x: watch_d.addCallback(watcher) and x) | 1081 | lambda x: watch_d.addCallback(watcher) and x) |
88 | 1082 | callback_d.addErrback( | ||
89 | 1083 | lambda failure: failure.trap(StopWatcher)) | ||
90 | 1071 | 1084 | ||
91 | 1072 | 1085 | ||
92 | 1073 | def _parse_unit_name(unit_name): | 1086 | def _parse_unit_name(unit_name): |
93 | 1074 | 1087 | ||
94 | === modified file 'ensemble/state/tests/test_service.py' | |||
95 | --- ensemble/state/tests/test_service.py 2011-05-05 14:31:41 +0000 | |||
96 | +++ ensemble/state/tests/test_service.py 2011-05-05 17:43:23 +0000 | |||
97 | @@ -17,7 +17,7 @@ | |||
98 | 17 | ServiceUnitStateMachineAlreadyAssigned, ServiceStateNameInUse, | 17 | ServiceUnitStateMachineAlreadyAssigned, ServiceStateNameInUse, |
99 | 18 | BadDescriptor, BadServiceStateName, ServiceUnitDebugAlreadyEnabled, | 18 | BadDescriptor, BadServiceStateName, ServiceUnitDebugAlreadyEnabled, |
100 | 19 | MachineStateNotFound, NoUnusedMachines, ServiceUnitResolvedAlreadyEnabled, | 19 | MachineStateNotFound, NoUnusedMachines, ServiceUnitResolvedAlreadyEnabled, |
102 | 20 | ServiceUnitRelationResolvedAlreadyEnabled) | 20 | ServiceUnitRelationResolvedAlreadyEnabled, StopWatcher) |
103 | 21 | 21 | ||
104 | 22 | 22 | ||
105 | 23 | from ensemble.state.tests.common import StateTestBase | 23 | from ensemble.state.tests.common import StateTestBase |
106 | @@ -790,6 +790,42 @@ | |||
107 | 790 | {"retry": NO_HOOKS}) | 790 | {"retry": NO_HOOKS}) |
108 | 791 | 791 | ||
109 | 792 | @inlineCallbacks | 792 | @inlineCallbacks |
110 | 793 | def test_stop_watch_resolved(self): | ||
111 | 794 | """A unit resolved watch can be instituted on a permanent basis. | ||
112 | 795 | |||
113 | 796 | However the callback can raise StopWatcher at anytime to stop the watch | ||
114 | 797 | """ | ||
115 | 798 | unit_state = yield self.get_unit_state() | ||
116 | 799 | |||
117 | 800 | results = [] | ||
118 | 801 | |||
119 | 802 | def callback(value): | ||
120 | 803 | results.append(value) | ||
121 | 804 | if len(results) == 1: | ||
122 | 805 | raise StopWatcher() | ||
123 | 806 | if len(results) == 3: | ||
124 | 807 | raise StopWatcher() | ||
125 | 808 | |||
126 | 809 | unit_state.watch_resolved(callback) | ||
127 | 810 | yield unit_state.set_resolved(RETRY_HOOKS) | ||
128 | 811 | yield unit_state.clear_resolved() | ||
129 | 812 | yield self.poke_zk() | ||
130 | 813 | |||
131 | 814 | unit_state.watch_resolved(callback) | ||
132 | 815 | yield unit_state.set_resolved(NO_HOOKS) | ||
133 | 816 | yield unit_state.clear_resolved() | ||
134 | 817 | |||
135 | 818 | yield self.poke_zk() | ||
136 | 819 | |||
137 | 820 | self.assertEqual(len(results), 3) | ||
138 | 821 | self.assertIdentical(results.pop(0), False) | ||
139 | 822 | self.assertIdentical(results.pop(0), False) | ||
140 | 823 | self.assertEqual(results.pop(0).type_name, "created") | ||
141 | 824 | |||
142 | 825 | self.assertEqual( | ||
143 | 826 | (yield unit_state.get_resolved()), None) | ||
144 | 827 | |||
145 | 828 | @inlineCallbacks | ||
146 | 793 | def test_get_set_clear_relation_resolved(self): | 829 | def test_get_set_clear_relation_resolved(self): |
147 | 794 | """The a unit's realtions can be set to resolved to mark a | 830 | """The a unit's realtions can be set to resolved to mark a |
148 | 795 | future transition, with an optional retry flag.""" | 831 | future transition, with an optional retry flag.""" |
149 | @@ -850,6 +886,40 @@ | |||
150 | 850 | {"0": NO_HOOKS}) | 886 | {"0": NO_HOOKS}) |
151 | 851 | 887 | ||
152 | 852 | @inlineCallbacks | 888 | @inlineCallbacks |
153 | 889 | def test_stop_watch_relation_resolved(self): | ||
154 | 890 | """A unit resolved watch can be instituted on a permanent basis.""" | ||
155 | 891 | unit_state = yield self.get_unit_state() | ||
156 | 892 | |||
157 | 893 | results = [] | ||
158 | 894 | |||
159 | 895 | def callback(value): | ||
160 | 896 | results.append(value) | ||
161 | 897 | |||
162 | 898 | if len(results) == 1: | ||
163 | 899 | raise StopWatcher() | ||
164 | 900 | |||
165 | 901 | if len(results) == 3: | ||
166 | 902 | raise StopWatcher() | ||
167 | 903 | |||
168 | 904 | unit_state.watch_relation_resolved(callback) | ||
169 | 905 | yield unit_state.set_relation_resolved({"0": RETRY_HOOKS}) | ||
170 | 906 | yield unit_state.clear_relation_resolved() | ||
171 | 907 | yield self.poke_zk() | ||
172 | 908 | self.assertEqual(len(results), 1) | ||
173 | 909 | |||
174 | 910 | unit_state.watch_relation_resolved(callback) | ||
175 | 911 | yield unit_state.set_relation_resolved({"0": RETRY_HOOKS}) | ||
176 | 912 | yield unit_state.clear_relation_resolved() | ||
177 | 913 | yield self.poke_zk() | ||
178 | 914 | self.assertEqual(len(results), 3) | ||
179 | 915 | self.assertIdentical(results.pop(0), False) | ||
180 | 916 | self.assertIdentical(results.pop(0), False) | ||
181 | 917 | self.assertEqual(results.pop(0).type_name, "created") | ||
182 | 918 | |||
183 | 919 | self.assertEqual( | ||
184 | 920 | (yield unit_state.get_relation_resolved()), None) | ||
185 | 921 | |||
186 | 922 | @inlineCallbacks | ||
187 | 853 | def test_watch_resolved_slow_callback(self): | 923 | def test_watch_resolved_slow_callback(self): |
188 | 854 | """A slow watch callback is still invoked serially.""" | 924 | """A slow watch callback is still invoked serially.""" |
189 | 855 | unit_state = yield self.get_unit_state() | 925 | unit_state = yield self.get_unit_state() |
190 | 856 | 926 | ||
191 | === modified file 'ensemble/unit/lifecycle.py' | |||
192 | --- ensemble/unit/lifecycle.py 2011-05-05 14:31:43 +0000 | |||
193 | +++ ensemble/unit/lifecycle.py 2011-05-05 17:43:23 +0000 | |||
194 | @@ -2,7 +2,7 @@ | |||
195 | 2 | import logging | 2 | import logging |
196 | 3 | 3 | ||
197 | 4 | from twisted.internet.defer import ( | 4 | from twisted.internet.defer import ( |
199 | 5 | inlineCallbacks, DeferredLock) | 5 | inlineCallbacks, DeferredLock, returnValue) |
200 | 6 | 6 | ||
201 | 7 | from ensemble.hooks.invoker import Invoker | 7 | from ensemble.hooks.invoker import Invoker |
202 | 8 | from ensemble.hooks.scheduler import HookScheduler | 8 | from ensemble.hooks.scheduler import HookScheduler |
203 | @@ -32,7 +32,8 @@ | |||
204 | 32 | self._unit_path = unit_path | 32 | self._unit_path = unit_path |
205 | 33 | self._relations = {} | 33 | self._relations = {} |
206 | 34 | self._running = False | 34 | self._running = False |
208 | 35 | self._watching = False | 35 | self._watching_relation_memberships = False |
209 | 36 | self._watching_relation_resolved = False | ||
210 | 36 | self._run_lock = DeferredLock() | 37 | self._run_lock = DeferredLock() |
211 | 37 | self._log = logging.getLogger("unit.lifecycle") | 38 | self._log = logging.getLogger("unit.lifecycle") |
212 | 38 | 39 | ||
213 | @@ -45,21 +46,23 @@ | |||
214 | 45 | return self._relations[relation_id] | 46 | return self._relations[relation_id] |
215 | 46 | 47 | ||
216 | 47 | @inlineCallbacks | 48 | @inlineCallbacks |
218 | 48 | def install(self): | 49 | def install(self, fire_hooks=True): |
219 | 49 | """Invoke the unit's install hook. | 50 | """Invoke the unit's install hook. |
220 | 50 | """ | 51 | """ |
222 | 51 | yield self._execute_hook("install") | 52 | if fire_hooks: |
223 | 53 | yield self._execute_hook("install") | ||
224 | 52 | 54 | ||
225 | 53 | @inlineCallbacks | 55 | @inlineCallbacks |
227 | 54 | def upgrade_formula(self): | 56 | def upgrade_formula(self, fire_hooks=True): |
228 | 55 | """Invoke the unit's upgrade-formula hook. | 57 | """Invoke the unit's upgrade-formula hook. |
229 | 56 | """ | 58 | """ |
231 | 57 | yield self._execute_hook("upgrade-formula", now=True) | 59 | if fire_hooks: |
232 | 60 | yield self._execute_hook("upgrade-formula", now=True) | ||
233 | 58 | # Restart hook queued hook execution. | 61 | # Restart hook queued hook execution. |
234 | 59 | self._executor.start() | 62 | self._executor.start() |
235 | 60 | 63 | ||
236 | 61 | @inlineCallbacks | 64 | @inlineCallbacks |
238 | 62 | def start(self): | 65 | def start(self, fire_hooks=True): |
239 | 63 | """Invoke the start hook, and setup relation watching. | 66 | """Invoke the start hook, and setup relation watching. |
240 | 64 | """ | 67 | """ |
241 | 65 | self._log.debug("pre-start acquire, running:%s", self._running) | 68 | self._log.debug("pre-start acquire, running:%s", self._running) |
242 | @@ -70,22 +73,29 @@ | |||
243 | 70 | assert not self._running, "Already started" | 73 | assert not self._running, "Already started" |
244 | 71 | 74 | ||
245 | 72 | # Execute the start hook | 75 | # Execute the start hook |
247 | 73 | yield self._execute_hook("start") | 76 | if fire_hooks: |
248 | 77 | yield self._execute_hook("start") | ||
249 | 74 | 78 | ||
250 | 75 | # If we have any existing relations in memory, start them. | 79 | # If we have any existing relations in memory, start them. |
251 | 76 | if self._relations: | 80 | if self._relations: |
252 | 77 | self._log.debug("starting relation lifecycles") | 81 | self._log.debug("starting relation lifecycles") |
253 | 78 | 82 | ||
254 | 79 | for workflow in self._relations.values(): | 83 | for workflow in self._relations.values(): |
255 | 80 | # should not transition an | ||
256 | 81 | yield workflow.transition_state("up") | 84 | yield workflow.transition_state("up") |
257 | 82 | 85 | ||
258 | 83 | # Establish a watch on the existing relations. | 86 | # Establish a watch on the existing relations. |
260 | 84 | if not self._watching: | 87 | if not self._watching_relation_memberships: |
261 | 85 | self._log.debug("starting service relation watch") | 88 | self._log.debug("starting service relation watch") |
262 | 86 | yield self._service.watch_relation_states( | 89 | yield self._service.watch_relation_states( |
263 | 87 | self._on_service_relation_changes) | 90 | self._on_service_relation_changes) |
265 | 88 | self._watching = True | 91 | self._watching_relation_memberships = True |
266 | 92 | |||
267 | 93 | # Establish a watch for resolved relations | ||
268 | 94 | if not self._watching_relation_resolved: | ||
269 | 95 | self._log.debug("starting unit relation resolved watch") | ||
270 | 96 | yield self._unit.watch_relation_resolved( | ||
271 | 97 | self._on_relation_resolved_changes) | ||
272 | 98 | self._watching_relation_resolved = True | ||
273 | 89 | 99 | ||
274 | 90 | # Set current status | 100 | # Set current status |
275 | 91 | self._running = True | 101 | self._running = True |
276 | @@ -94,7 +104,7 @@ | |||
277 | 94 | self._log.debug("started unit lifecycle") | 104 | self._log.debug("started unit lifecycle") |
278 | 95 | 105 | ||
279 | 96 | @inlineCallbacks | 106 | @inlineCallbacks |
281 | 97 | def stop(self): | 107 | def stop(self, fire_hooks=True): |
282 | 98 | """Stop the unit, executes the stop hook, and stops relation watching. | 108 | """Stop the unit, executes the stop hook, and stops relation watching. |
283 | 99 | """ | 109 | """ |
284 | 100 | self._log.debug("pre-stop acquire, running:%s", self._running) | 110 | self._log.debug("pre-stop acquire, running:%s", self._running) |
285 | @@ -110,7 +120,8 @@ | |||
286 | 110 | for workflow in self._relations.values(): | 120 | for workflow in self._relations.values(): |
287 | 111 | yield workflow.transition_state("down") | 121 | yield workflow.transition_state("down") |
288 | 112 | 122 | ||
290 | 113 | yield self._execute_hook("stop") | 123 | if fire_hooks: |
291 | 124 | yield self._execute_hook("stop") | ||
292 | 114 | 125 | ||
293 | 115 | # Set current status | 126 | # Set current status |
294 | 116 | self._running = False | 127 | self._running = False |
295 | @@ -119,9 +130,11 @@ | |||
296 | 119 | self._log.debug("stopped unit lifecycle") | 130 | self._log.debug("stopped unit lifecycle") |
297 | 120 | 131 | ||
298 | 121 | @inlineCallbacks | 132 | @inlineCallbacks |
300 | 122 | def configure(self): | 133 | def configure(self, fire_hooks=True): |
301 | 123 | """Inform the unit that its service config has changed. | 134 | """Inform the unit that its service config has changed. |
302 | 124 | """ | 135 | """ |
303 | 136 | if not fire_hooks: | ||
304 | 137 | returnValue(None) | ||
305 | 125 | yield self._run_lock.acquire() | 138 | yield self._run_lock.acquire() |
306 | 126 | try: | 139 | try: |
307 | 127 | # Verify State | 140 | # Verify State |
308 | @@ -134,6 +147,49 @@ | |||
309 | 134 | self._log.debug("configured unit") | 147 | self._log.debug("configured unit") |
310 | 135 | 148 | ||
311 | 136 | @inlineCallbacks | 149 | @inlineCallbacks |
312 | 150 | def _on_relation_resolved_changes(self, event): | ||
313 | 151 | """Callback for unit relation resolved watching. | ||
314 | 152 | |||
315 | 153 | The callback is invoked whenever the relation resolved | ||
316 | 154 | settings change. | ||
317 | 155 | """ | ||
318 | 156 | self._log.debug("relation resolved changed") | ||
319 | 157 | # Acquire the run lock, and process the changes. | ||
320 | 158 | yield self._run_lock.acquire() | ||
321 | 159 | |||
322 | 160 | try: | ||
323 | 161 | # If the unit lifecycle isn't running we shouldn't process | ||
324 | 162 | # any relation resolutions. | ||
325 | 163 | if not self._running: | ||
326 | 164 | self._log.debug("stop watch relation resolved changes") | ||
327 | 165 | self._watching_relation_resolved = False | ||
328 | 166 | raise StopWatcher() | ||
329 | 167 | |||
330 | 168 | self._log.info("processing relation resolved changed") | ||
331 | 169 | if self._client.connected: | ||
332 | 170 | yield self._process_relation_resolved_changes() | ||
333 | 171 | finally: | ||
334 | 172 | yield self._run_lock.release() | ||
335 | 173 | |||
336 | 174 | @inlineCallbacks | ||
337 | 175 | def _process_relation_resolved_changes(self): | ||
338 | 176 | """Invoke retry transitions on relations if their not running. | ||
339 | 177 | """ | ||
340 | 178 | relation_resolved = yield self._unit.get_relation_resolved() | ||
341 | 179 | if relation_resolved is None: | ||
342 | 180 | returnValue(None) | ||
343 | 181 | else: | ||
344 | 182 | yield self._unit.clear_relation_resolved() | ||
345 | 183 | |||
346 | 184 | keys = set(relation_resolved).intersection(self._relations) | ||
347 | 185 | for rel_id in keys: | ||
348 | 186 | relation_workflow = self._relations[rel_id] | ||
349 | 187 | relation_state = yield relation_workflow.get_state() | ||
350 | 188 | if relation_state == "up": | ||
351 | 189 | continue | ||
352 | 190 | yield relation_workflow.transition_state("up") | ||
353 | 191 | |||
354 | 192 | @inlineCallbacks | ||
355 | 137 | def _on_service_relation_changes(self, old_relations, new_relations): | 193 | def _on_service_relation_changes(self, old_relations, new_relations): |
356 | 138 | """Callback for service relation watching. | 194 | """Callback for service relation watching. |
357 | 139 | 195 | ||
358 | @@ -153,7 +209,7 @@ | |||
359 | 153 | # If the lifecycle is not running, then stop the watcher | 209 | # If the lifecycle is not running, then stop the watcher |
360 | 154 | if not self._running: | 210 | if not self._running: |
361 | 155 | self._log.debug("stop service-rel watcher, discarding changes") | 211 | self._log.debug("stop service-rel watcher, discarding changes") |
363 | 156 | self._watching = False | 212 | self._watching_relation_memberships = False |
364 | 157 | raise StopWatcher() | 213 | raise StopWatcher() |
365 | 158 | 214 | ||
366 | 159 | self._log.debug("processing relations changed") | 215 | self._log.debug("processing relations changed") |
367 | @@ -163,9 +219,9 @@ | |||
368 | 163 | 219 | ||
369 | 164 | @inlineCallbacks | 220 | @inlineCallbacks |
370 | 165 | def _process_service_changes(self, old_relations, new_relations): | 221 | def _process_service_changes(self, old_relations, new_relations): |
372 | 166 | """Add and remove unit lifecycles per the service relations changes. | 222 | """Add and remove unit lifecycles per the service relations Determine. |
373 | 167 | """ | 223 | """ |
375 | 168 | # Determine relation delta of global zk state with our memory state. | 224 | # changes relation delta of global zk state with our memory state. |
376 | 169 | new_relations = dict([(service_relation.internal_relation_id, | 225 | new_relations = dict([(service_relation.internal_relation_id, |
377 | 170 | service_relation) for | 226 | service_relation) for |
378 | 171 | service_relation in new_relations]) | 227 | service_relation in new_relations]) |
379 | @@ -352,8 +408,11 @@ | |||
380 | 352 | self._error_handler = handler | 408 | self._error_handler = handler |
381 | 353 | 409 | ||
382 | 354 | @inlineCallbacks | 410 | @inlineCallbacks |
384 | 355 | def start(self): | 411 | def start(self, watches=True): |
385 | 356 | """Start watching related units and executing change hooks. | 412 | """Start watching related units and executing change hooks. |
386 | 413 | |||
387 | 414 | @param watches: boolean parameter denoting if relation watches | ||
388 | 415 | should be started. | ||
389 | 357 | """ | 416 | """ |
390 | 358 | yield self._run_lock.acquire() | 417 | yield self._run_lock.acquire() |
391 | 359 | try: | 418 | try: |
392 | @@ -364,19 +423,24 @@ | |||
393 | 364 | self._watcher = yield self._unit_relation.watch_related_units( | 423 | self._watcher = yield self._unit_relation.watch_related_units( |
394 | 365 | self._scheduler.notify_change) | 424 | self._scheduler.notify_change) |
395 | 366 | # And start the watcher. | 425 | # And start the watcher. |
397 | 367 | yield self._watcher.start() | 426 | if watches: |
398 | 427 | yield self._watcher.start() | ||
399 | 368 | finally: | 428 | finally: |
400 | 369 | self._run_lock.release() | 429 | self._run_lock.release() |
401 | 370 | self._log.debug( | 430 | self._log.debug( |
402 | 371 | "started relation:%s lifecycle", self._relation_name) | 431 | "started relation:%s lifecycle", self._relation_name) |
403 | 372 | 432 | ||
404 | 373 | @inlineCallbacks | 433 | @inlineCallbacks |
406 | 374 | def stop(self): | 434 | def stop(self, watches=True): |
407 | 375 | """Stop watching changes and stop executing relation change hooks. | 435 | """Stop watching changes and stop executing relation change hooks. |
408 | 436 | |||
409 | 437 | @param watches: boolean parameter denoting if relation watches | ||
410 | 438 | should be stopped. | ||
411 | 376 | """ | 439 | """ |
412 | 377 | yield self._run_lock.acquire() | 440 | yield self._run_lock.acquire() |
413 | 378 | try: | 441 | try: |
415 | 379 | self._watcher.stop() | 442 | if watches and self._watcher: |
416 | 443 | self._watcher.stop() | ||
417 | 380 | self._scheduler.stop() | 444 | self._scheduler.stop() |
418 | 381 | finally: | 445 | finally: |
419 | 382 | yield self._run_lock.release() | 446 | yield self._run_lock.release() |
420 | 383 | 447 | ||
421 | === modified file 'ensemble/unit/tests/test_lifecycle.py' | |||
422 | --- ensemble/unit/tests/test_lifecycle.py 2011-05-05 09:41:11 +0000 | |||
423 | +++ ensemble/unit/tests/test_lifecycle.py 2011-05-05 17:43:23 +0000 | |||
424 | @@ -12,6 +12,8 @@ | |||
425 | 12 | from ensemble.unit.lifecycle import ( | 12 | from ensemble.unit.lifecycle import ( |
426 | 13 | UnitLifecycle, UnitRelationLifecycle, RelationInvoker) | 13 | UnitLifecycle, UnitRelationLifecycle, RelationInvoker) |
427 | 14 | 14 | ||
428 | 15 | from ensemble.unit.workflow import RelationWorkflowState | ||
429 | 16 | |||
430 | 15 | from ensemble.hooks.invoker import Invoker | 17 | from ensemble.hooks.invoker import Invoker |
431 | 16 | from ensemble.hooks.executor import HookExecutor | 18 | from ensemble.hooks.executor import HookExecutor |
432 | 17 | 19 | ||
433 | @@ -19,9 +21,11 @@ | |||
434 | 19 | 21 | ||
435 | 20 | from ensemble.state.endpoint import RelationEndpoint | 22 | from ensemble.state.endpoint import RelationEndpoint |
436 | 21 | from ensemble.state.relation import ClientServerUnitWatcher | 23 | from ensemble.state.relation import ClientServerUnitWatcher |
437 | 24 | from ensemble.state.service import NO_HOOKS | ||
438 | 22 | from ensemble.state.tests.test_relation import RelationTestBase | 25 | from ensemble.state.tests.test_relation import RelationTestBase |
439 | 23 | from ensemble.state.hook import RelationChange | 26 | from ensemble.state.hook import RelationChange |
440 | 24 | 27 | ||
441 | 28 | |||
442 | 25 | from ensemble.lib.testing import TestCase | 29 | from ensemble.lib.testing import TestCase |
443 | 26 | from ensemble.lib.mocker import MATCH | 30 | from ensemble.lib.mocker import MATCH |
444 | 27 | 31 | ||
445 | @@ -97,6 +101,8 @@ | |||
446 | 97 | results.append(hook_name) | 101 | results.append(hook_name) |
447 | 98 | if debug: | 102 | if debug: |
448 | 99 | print "-> exec hook", hook_name | 103 | print "-> exec hook", hook_name |
449 | 104 | if d.called: | ||
450 | 105 | return | ||
451 | 100 | if results == sequence: | 106 | if results == sequence: |
452 | 101 | d.callback(True) | 107 | d.callback(True) |
453 | 102 | if hook_name == name and count is None: | 108 | if hook_name == name and count is None: |
454 | @@ -145,6 +151,182 @@ | |||
455 | 145 | return output | 151 | return output |
456 | 146 | 152 | ||
457 | 147 | 153 | ||
458 | 154 | class LifecycleResolvedTest(LifecycleTestBase): | ||
459 | 155 | |||
460 | 156 | @inlineCallbacks | ||
461 | 157 | def setUp(self): | ||
462 | 158 | yield super(LifecycleResolvedTest, self).setUp() | ||
463 | 159 | yield self.setup_default_test_relation() | ||
464 | 160 | self.lifecycle = UnitLifecycle( | ||
465 | 161 | self.client, self.states["unit"], self.states["service"], | ||
466 | 162 | self.unit_directory, self.executor) | ||
467 | 163 | |||
468 | 164 | def get_unit_relation_workflow(self, states): | ||
469 | 165 | state_dir = os.path.join(self.ensemble_directory, "state") | ||
470 | 166 | lifecycle = UnitRelationLifecycle( | ||
471 | 167 | self.client, | ||
472 | 168 | states["unit_relation"], | ||
473 | 169 | states["service_relation"].relation_name, | ||
474 | 170 | self.unit_directory, | ||
475 | 171 | self.executor) | ||
476 | 172 | |||
477 | 173 | workflow = RelationWorkflowState( | ||
478 | 174 | self.client, | ||
479 | 175 | states["unit_relation"], | ||
480 | 176 | lifecycle, | ||
481 | 177 | state_dir) | ||
482 | 178 | |||
483 | 179 | return (workflow, lifecycle) | ||
484 | 180 | |||
485 | 181 | @inlineCallbacks | ||
486 | 182 | def test_resolved_relation_watch_unit_lifecycle_not_running(self): | ||
487 | 183 | """If the unit is not running then no relation resolving is performed. | ||
488 | 184 | However the resolution value remains the same. | ||
489 | 185 | """ | ||
490 | 186 | # Start the unit. | ||
491 | 187 | yield self.lifecycle.start() | ||
492 | 188 | |||
493 | 189 | # Wait for the relation to be started.... TODO: async background work | ||
494 | 190 | yield self.sleep(0.1) | ||
495 | 191 | |||
496 | 192 | # Simulate relation down on an individual unit relation | ||
497 | 193 | workflow = self.lifecycle.get_relation_workflow( | ||
498 | 194 | self.states["unit_relation"].internal_relation_id) | ||
499 | 195 | self.assertEqual("up", (yield workflow.get_state())) | ||
500 | 196 | |||
501 | 197 | yield workflow.transition_state("down") | ||
502 | 198 | resolved = self.wait_on_state(workflow, "up") | ||
503 | 199 | |||
504 | 200 | # Stop the unit lifecycle | ||
505 | 201 | yield self.lifecycle.stop() | ||
506 | 202 | |||
507 | 203 | # Set the relation to resolved | ||
508 | 204 | yield self.states["unit"].set_relation_resolved( | ||
509 | 205 | {self.states["unit_relation"].internal_relation_id: NO_HOOKS}) | ||
510 | 206 | |||
511 | 207 | # Give a moment for the watch to fire erroneously | ||
512 | 208 | yield self.sleep(0.2) | ||
513 | 209 | |||
514 | 210 | # Ensure we didn't attempt a transition. | ||
515 | 211 | self.assertFalse(resolved.called) | ||
516 | 212 | self.assertEqual( | ||
517 | 213 | {self.states["unit_relation"].internal_relation_id: NO_HOOKS}, | ||
518 | 214 | (yield self.states["unit"].get_relation_resolved())) | ||
519 | 215 | |||
520 | 216 | # If the unit is restarted start, we currently have the | ||
521 | 217 | # behavior that the unit relation workflow will automatically | ||
522 | 218 | # be transitioned back to running, as part of the normal state | ||
523 | 219 | # transition. Sigh.. we should have a separate error | ||
524 | 220 | # state for relation hooks then down with state variable usage. | ||
525 | 221 | |||
526 | 222 | @inlineCallbacks | ||
527 | 223 | def test_resolved_relation_watch_relation_up(self): | ||
528 | 224 | """If a relation marked as to be resolved is already running, | ||
529 | 225 | then no work is performed. | ||
530 | 226 | """ | ||
531 | 227 | # Start the unit. | ||
532 | 228 | yield self.lifecycle.start() | ||
533 | 229 | |||
534 | 230 | # Wait for the relation to be started.... TODO: async background work | ||
535 | 231 | yield self.sleep(0.1) | ||
536 | 232 | |||
537 | 233 | # get a hold of the unit relation and verify state | ||
538 | 234 | workflow = self.lifecycle.get_relation_workflow( | ||
539 | 235 | self.states["unit_relation"].internal_relation_id) | ||
540 | 236 | self.assertEqual("up", (yield workflow.get_state())) | ||
541 | 237 | |||
542 | 238 | # Set the relation to resolved | ||
543 | 239 | yield self.states["unit"].set_relation_resolved( | ||
544 | 240 | {self.states["unit_relation"].internal_relation_id: NO_HOOKS}) | ||
545 | 241 | |||
546 | 242 | # Give a moment for the async background work. | ||
547 | 243 | yield self.sleep(0.1) | ||
548 | 244 | |||
549 | 245 | # Ensure we're still up and the relation resolved setting has been | ||
550 | 246 | # cleared. | ||
551 | 247 | self.assertEqual( | ||
552 | 248 | None, (yield self.states["unit"].get_relation_resolved())) | ||
553 | 249 | self.assertEqual("up", (yield workflow.get_state())) | ||
554 | 250 | |||
555 | 251 | @inlineCallbacks | ||
556 | 252 | def test_resolved_relation_watch_from_error(self): | ||
557 | 253 | """Unit lifecycle's will process a unit relation resolved | ||
558 | 254 | setting, and transition a down relation back to a running | ||
559 | 255 | state. | ||
560 | 256 | """ | ||
561 | 257 | log_output = self.capture_logging( | ||
562 | 258 | "unit.lifecycle", level=logging.DEBUG) | ||
563 | 259 | |||
564 | 260 | # Start the unit. | ||
565 | 261 | yield self.lifecycle.start() | ||
566 | 262 | |||
567 | 263 | # Wait for the relation to be started... TODO: async background work | ||
568 | 264 | yield self.sleep(0.1) | ||
569 | 265 | |||
570 | 266 | # Simulate an error condition | ||
571 | 267 | workflow = self.lifecycle.get_relation_workflow( | ||
572 | 268 | self.states["unit_relation"].internal_relation_id) | ||
573 | 269 | self.assertEqual("up", (yield workflow.get_state())) | ||
574 | 270 | yield workflow.fire_transition("error") | ||
575 | 271 | |||
576 | 272 | resolved = self.wait_on_state(workflow, "up") | ||
577 | 273 | |||
578 | 274 | # Set the relation to resolved | ||
579 | 275 | yield self.states["unit"].set_relation_resolved( | ||
580 | 276 | {self.states["unit_relation"].internal_relation_id: NO_HOOKS}) | ||
581 | 277 | |||
582 | 278 | # Wait for the relation to come back up | ||
583 | 279 | value = yield self.states["unit"].get_relation_resolved() | ||
584 | 280 | |||
585 | 281 | yield resolved | ||
586 | 282 | |||
587 | 283 | # Verify state | ||
588 | 284 | value = yield workflow.get_state() | ||
589 | 285 | self.assertEqual(value, "up") | ||
590 | 286 | |||
591 | 287 | self.assertIn( | ||
592 | 288 | "processing relation resolved changed", log_output.getvalue()) | ||
593 | 289 | |||
594 | 290 | @inlineCallbacks | ||
595 | 291 | def test_resolved_relation_watch(self): | ||
596 | 292 | """Unit lifecycle's will process a unit relation resolved | ||
597 | 293 | setting, and transition a down relation back to a running | ||
598 | 294 | state. | ||
599 | 295 | """ | ||
600 | 296 | log_output = self.capture_logging( | ||
601 | 297 | "unit.lifecycle", level=logging.DEBUG) | ||
602 | 298 | |||
603 | 299 | # Start the unit. | ||
604 | 300 | yield self.lifecycle.start() | ||
605 | 301 | |||
606 | 302 | # Wait for the relation to be started... TODO: async background work | ||
607 | 303 | yield self.sleep(0.1) | ||
608 | 304 | |||
609 | 305 | # Simulate an error condition | ||
610 | 306 | workflow = self.lifecycle.get_relation_workflow( | ||
611 | 307 | self.states["unit_relation"].internal_relation_id) | ||
612 | 308 | self.assertEqual("up", (yield workflow.get_state())) | ||
613 | 309 | yield workflow.transition_state("down") | ||
614 | 310 | |||
615 | 311 | resolved = self.wait_on_state(workflow, "up") | ||
616 | 312 | |||
617 | 313 | # Set the relation to resolved | ||
618 | 314 | yield self.states["unit"].set_relation_resolved( | ||
619 | 315 | {self.states["unit_relation"].internal_relation_id: NO_HOOKS}) | ||
620 | 316 | |||
621 | 317 | # Wait for the relation to come back up | ||
622 | 318 | value = yield self.states["unit"].get_relation_resolved() | ||
623 | 319 | |||
624 | 320 | yield resolved | ||
625 | 321 | |||
626 | 322 | # Verify state | ||
627 | 323 | value = yield workflow.get_state() | ||
628 | 324 | self.assertEqual(value, "up") | ||
629 | 325 | |||
630 | 326 | self.assertIn( | ||
631 | 327 | "processing relation resolved changed", log_output.getvalue()) | ||
632 | 328 | |||
633 | 329 | |||
634 | 148 | class UnitLifecycleTest(LifecycleTestBase): | 330 | class UnitLifecycleTest(LifecycleTestBase): |
635 | 149 | 331 | ||
636 | 150 | @inlineCallbacks | 332 | @inlineCallbacks |
637 | @@ -187,6 +369,45 @@ | |||
638 | 187 | # verify the sockets are cleaned up. | 369 | # verify the sockets are cleaned up. |
639 | 188 | self.assertEqual(os.listdir(self.unit_directory), ["formula"]) | 370 | self.assertEqual(os.listdir(self.unit_directory), ["formula"]) |
640 | 189 | 371 | ||
641 | 372 | @inlineCallbacks | ||
642 | 373 | def test_start_sans_hook(self): | ||
643 | 374 | """The lifecycle start can be invoked without firing hooks.""" | ||
644 | 375 | self.write_hook("start", "#!/bin/sh\n exit 1") | ||
645 | 376 | start_executed = self.wait_on_hook("start") | ||
646 | 377 | yield self.lifecycle.start(fire_hooks=False) | ||
647 | 378 | # Wait for unit relation background processing.... | ||
648 | 379 | yield self.sleep(0.1) | ||
649 | 380 | self.assertFalse(start_executed.called) | ||
650 | 381 | |||
651 | 382 | @inlineCallbacks | ||
652 | 383 | def test_stop_sans_hook(self): | ||
653 | 384 | """The lifecycle stop can be invoked without firing hooks.""" | ||
654 | 385 | self.write_hook("stop", "#!/bin/sh\n exit 1") | ||
655 | 386 | stop_executed = self.wait_on_hook("stop") | ||
656 | 387 | yield self.lifecycle.start() | ||
657 | 388 | yield self.lifecycle.stop(fire_hooks=False) | ||
658 | 389 | # Wait for unit relation background processing.... | ||
659 | 390 | yield self.sleep(0.1) | ||
660 | 391 | self.assertFalse(stop_executed.called) | ||
661 | 392 | |||
662 | 393 | @inlineCallbacks | ||
663 | 394 | def test_install_sans_hook(self): | ||
664 | 395 | """The lifecycle install can be invoked without firing hooks.""" | ||
665 | 396 | self.write_hook("install", "#!/bin/sh\n exit 1") | ||
666 | 397 | install_executed = self.wait_on_hook("install") | ||
667 | 398 | yield self.lifecycle.install(fire_hooks=False) | ||
668 | 399 | self.assertFalse(install_executed.called) | ||
669 | 400 | |||
670 | 401 | @inlineCallbacks | ||
671 | 402 | def test_upgrade_sans_hook(self): | ||
672 | 403 | """The lifecycle upgrade can be invoked without firing hooks.""" | ||
673 | 404 | self.executor.stop() | ||
674 | 405 | self.write_hook("upgrade-formula", "#!/bin/sh\n exit 1") | ||
675 | 406 | upgrade_executed = self.wait_on_hook("upgrade-formula") | ||
676 | 407 | yield self.lifecycle.upgrade_formula(fire_hooks=False) | ||
677 | 408 | self.assertFalse(upgrade_executed.called) | ||
678 | 409 | self.assertTrue(self.executor.running) | ||
679 | 410 | |||
680 | 190 | def test_hook_error(self): | 411 | def test_hook_error(self): |
681 | 191 | """Verify hook execution error, raises an exception.""" | 412 | """Verify hook execution error, raises an exception.""" |
682 | 192 | self.write_hook("install", '#!/bin/sh\n exit 1') | 413 | self.write_hook("install", '#!/bin/sh\n exit 1') |
683 | @@ -196,14 +417,12 @@ | |||
684 | 196 | def test_hook_not_executable(self): | 417 | def test_hook_not_executable(self): |
685 | 197 | """A hook not executable, raises an exception.""" | 418 | """A hook not executable, raises an exception.""" |
686 | 198 | self.write_hook("install", '#!/bin/sh\n exit 0', no_exec=True) | 419 | self.write_hook("install", '#!/bin/sh\n exit 0', no_exec=True) |
687 | 199 | # It would be preferrable if this was also a formulainvocation error. | ||
688 | 200 | return self.failUnlessFailure( | 420 | return self.failUnlessFailure( |
689 | 201 | self.lifecycle.install(), FormulaError) | 421 | self.lifecycle.install(), FormulaError) |
690 | 202 | 422 | ||
691 | 203 | def test_hook_not_formatted_correctly(self): | 423 | def test_hook_not_formatted_correctly(self): |
692 | 204 | """Hook execution error, raises an exception.""" | 424 | """Hook execution error, raises an exception.""" |
693 | 205 | self.write_hook("install", '!/bin/sh\n exit 0') | 425 | self.write_hook("install", '!/bin/sh\n exit 0') |
694 | 206 | # It would be preferrable if this was also a formulainvocation error. | ||
695 | 207 | return self.failUnlessFailure( | 426 | return self.failUnlessFailure( |
696 | 208 | self.lifecycle.install(), FormulaInvocationError) | 427 | self.lifecycle.install(), FormulaInvocationError) |
697 | 209 | 428 | ||
698 | @@ -532,7 +751,7 @@ | |||
699 | 532 | @inlineCallbacks | 751 | @inlineCallbacks |
700 | 533 | def test_initial_start_lifecycle_no_related_no_exec(self): | 752 | def test_initial_start_lifecycle_no_related_no_exec(self): |
701 | 534 | """ | 753 | """ |
703 | 535 | If there are no related units on startup, the relation changed hook | 754 | If there are no related units on startup, the relation joined hook |
704 | 536 | is not invoked. | 755 | is not invoked. |
705 | 537 | """ | 756 | """ |
706 | 538 | file_path = self.makeFile() | 757 | file_path = self.makeFile() |
707 | @@ -545,6 +764,26 @@ | |||
708 | 545 | self.assertFalse(os.path.exists(file_path)) | 764 | self.assertFalse(os.path.exists(file_path)) |
709 | 546 | 765 | ||
710 | 547 | @inlineCallbacks | 766 | @inlineCallbacks |
711 | 767 | def test_stop_can_continue_watching(self): | ||
712 | 768 | """ | ||
713 | 769 | """ | ||
714 | 770 | file_path = self.makeFile() | ||
715 | 771 | self.write_hook( | ||
716 | 772 | "%s-relation-changed" % self.relation_name, | ||
717 | 773 | ("#!/bin/bash\n" "echo executed >> %s\n" % file_path)) | ||
718 | 774 | rel_states = yield self.add_opposite_service_unit(self.states) | ||
719 | 775 | yield self.lifecycle.start() | ||
720 | 776 | yield self.wait_on_hook( | ||
721 | 777 | sequence=["app-relation-joined", "app-relation-changed"]) | ||
722 | 778 | changed_executed = self.wait_on_hook("app-relation-changed") | ||
723 | 779 | yield self.lifecycle.stop(watches=False) | ||
724 | 780 | rel_states["unit_relation"].set_data(yaml.dump(dict(hello="world"))) | ||
725 | 781 | yield self.sleep(0.1) | ||
726 | 782 | self.assertFalse(changed_executed.called) | ||
727 | 783 | yield self.lifecycle.start(watches=False) | ||
728 | 784 | yield changed_executed | ||
729 | 785 | |||
730 | 786 | @inlineCallbacks | ||
731 | 548 | def test_initial_start_lifecycle_with_related(self): | 787 | def test_initial_start_lifecycle_with_related(self): |
732 | 549 | """ | 788 | """ |
733 | 550 | If there are related units on startup, the relation changed hook | 789 | If there are related units on startup, the relation changed hook |
734 | 551 | 790 | ||
735 | === modified file 'ensemble/unit/tests/test_workflow.py' | |||
736 | --- ensemble/unit/tests/test_workflow.py 2011-05-05 14:31:43 +0000 | |||
737 | +++ ensemble/unit/tests/test_workflow.py 2011-05-05 17:43:23 +0000 | |||
738 | @@ -83,11 +83,30 @@ | |||
739 | 83 | self.assertFalse(result) | 83 | self.assertFalse(result) |
740 | 84 | current_state = yield self.workflow.get_state() | 84 | current_state = yield self.workflow.get_state() |
741 | 85 | yield self.assertEqual(current_state, "install_error") | 85 | yield self.assertEqual(current_state, "install_error") |
742 | 86 | self.write_hook("install", "#!/bin/bash\necho hello\n") | ||
743 | 87 | result = yield self.workflow.fire_transition("retry_install") | 86 | result = yield self.workflow.fire_transition("retry_install") |
744 | 88 | yield self.assertState(self.workflow, "installed") | 87 | yield self.assertState(self.workflow, "installed") |
745 | 89 | 88 | ||
746 | 90 | @inlineCallbacks | 89 | @inlineCallbacks |
747 | 90 | def test_install_error_with_retry_hook(self): | ||
748 | 91 | """If the install hook fails, the workflow is transition to the | ||
749 | 92 | install_error state. | ||
750 | 93 | """ | ||
751 | 94 | self.write_hook("install", "#!/bin/bash\nexit 1") | ||
752 | 95 | result = yield self.workflow.fire_transition("install") | ||
753 | 96 | self.assertFalse(result) | ||
754 | 97 | current_state = yield self.workflow.get_state() | ||
755 | 98 | yield self.assertEqual(current_state, "install_error") | ||
756 | 99 | |||
757 | 100 | result = yield self.workflow.fire_transition("retry_install_hook") | ||
758 | 101 | yield self.assertState(self.workflow, "install_error") | ||
759 | 102 | |||
760 | 103 | self.write_hook("install", "#!/bin/bash\necho hello\n") | ||
761 | 104 | hook_deferred = self.wait_on_hook("install") | ||
762 | 105 | result = yield self.workflow.fire_transition_alias("retry_hook") | ||
763 | 106 | yield hook_deferred | ||
764 | 107 | yield self.assertState(self.workflow, "installed") | ||
765 | 108 | |||
766 | 109 | @inlineCallbacks | ||
767 | 91 | def test_start(self): | 110 | def test_start(self): |
768 | 92 | file_path = self.makeFile() | 111 | file_path = self.makeFile() |
769 | 93 | self.write_hook( | 112 | self.write_hook( |
770 | @@ -131,12 +150,41 @@ | |||
771 | 131 | current_state = yield self.workflow.get_state() | 150 | current_state = yield self.workflow.get_state() |
772 | 132 | self.assertEqual(current_state, "start_error") | 151 | self.assertEqual(current_state, "start_error") |
773 | 133 | 152 | ||
774 | 134 | self.write_hook("start", "#!/bin/bash\necho hello\n") | ||
775 | 135 | result = yield self.workflow.fire_transition("retry_start") | 153 | result = yield self.workflow.fire_transition("retry_start") |
776 | 136 | yield self.assertState(self.workflow, "started") | 154 | yield self.assertState(self.workflow, "started") |
780 | 137 | # If we don't stop, we'll end up with the relation lifecycle | 155 | |
781 | 138 | # watches firing in the background when the test stops. | 156 | # If we don't stop, we'll end up with the relation lifecycle |
782 | 139 | self.write_hook("stop", "#!/bin/bash\necho hello\n") | 157 | # watches firing in the background when the test stops. |
783 | 158 | result = yield self.workflow.fire_transition("stop") | ||
784 | 159 | yield self.assertState(self.workflow, "stopped") | ||
785 | 160 | |||
786 | 161 | @inlineCallbacks | ||
787 | 162 | def test_start_error_with_retry_hook(self): | ||
788 | 163 | """Executing the start transition with a hook error, results in the | ||
789 | 164 | workflow going to the start_error state. The start can be retried. | ||
790 | 165 | """ | ||
791 | 166 | self.write_hook("install", "#!/bin/bash\necho hello\n") | ||
792 | 167 | result = yield self.workflow.fire_transition("install") | ||
793 | 168 | self.assertTrue(result) | ||
794 | 169 | self.write_hook("start", "#!/bin/bash\nexit 1") | ||
795 | 170 | result = yield self.workflow.fire_transition("start") | ||
796 | 171 | self.assertFalse(result) | ||
797 | 172 | current_state = yield self.workflow.get_state() | ||
798 | 173 | self.assertEqual(current_state, "start_error") | ||
799 | 174 | |||
800 | 175 | hook_deferred = self.wait_on_hook("start") | ||
801 | 176 | result = yield self.workflow.fire_transition("retry_start_hook") | ||
802 | 177 | yield hook_deferred | ||
803 | 178 | yield self.assertState(self.workflow, "start_error") | ||
804 | 179 | |||
805 | 180 | self.write_hook("start", "#!/bin/bash\nexit 0") | ||
806 | 181 | hook_deferred = self.wait_on_hook("start") | ||
807 | 182 | result = yield self.workflow.fire_transition_alias("retry_hook") | ||
808 | 183 | yield hook_deferred | ||
809 | 184 | yield self.assertState(self.workflow, "started") | ||
810 | 185 | |||
811 | 186 | # If we don't stop, we'll end up with the relation lifecycle | ||
812 | 187 | # watches firing in the background when the test stops. | ||
813 | 140 | result = yield self.workflow.fire_transition("stop") | 188 | result = yield self.workflow.fire_transition("stop") |
814 | 141 | yield self.assertState(self.workflow, "stopped") | 189 | yield self.assertState(self.workflow, "stopped") |
815 | 142 | 190 | ||
816 | @@ -193,13 +241,49 @@ | |||
817 | 193 | yield self.assertState(self.workflow, "configure_error") | 241 | yield self.assertState(self.workflow, "configure_error") |
818 | 194 | 242 | ||
819 | 195 | # Verify recovery from error state | 243 | # Verify recovery from error state |
820 | 244 | result = yield self.workflow.fire_transition_alias("retry") | ||
821 | 245 | self.assertTrue(result) | ||
822 | 246 | yield self.assertState(self.workflow, "started") | ||
823 | 247 | |||
824 | 248 | # Stop any background processing | ||
825 | 249 | yield self.workflow.fire_transition("stop") | ||
826 | 250 | |||
827 | 251 | @inlineCallbacks | ||
828 | 252 | def test_configure_error_and_retry_hook(self): | ||
829 | 253 | """An error while configuring, transitions the unit and | ||
830 | 254 | stops the lifecycle.""" | ||
831 | 255 | #self.capture_output() | ||
832 | 256 | yield self.workflow.fire_transition("install") | ||
833 | 257 | result = yield self.workflow.fire_transition("start") | ||
834 | 258 | self.assertTrue(result) | ||
835 | 259 | self.assertState(self.workflow, "started") | ||
836 | 260 | |||
837 | 261 | # Verify transition to error state | ||
838 | 262 | hook_deferred = self.wait_on_hook("config-changed") | ||
839 | 263 | self.write_hook("config-changed", "#!/bin/bash\nexit 1") | ||
840 | 264 | result = yield self.workflow.fire_transition("reconfigure") | ||
841 | 265 | yield hook_deferred | ||
842 | 266 | self.assertFalse(result) | ||
843 | 267 | yield self.assertState(self.workflow, "configure_error") | ||
844 | 268 | |||
845 | 269 | # Verify retry hook with hook error stays in error state | ||
846 | 270 | hook_deferred = self.wait_on_hook("config-changed") | ||
847 | 271 | result = yield self.workflow.fire_transition("retry_configure_hook") | ||
848 | 272 | |||
849 | 273 | self.assertFalse(result) | ||
850 | 274 | yield hook_deferred | ||
851 | 275 | yield self.assertState(self.workflow, "configure_error") | ||
852 | 276 | |||
853 | 196 | hook_deferred = self.wait_on_hook("config-changed") | 277 | hook_deferred = self.wait_on_hook("config-changed") |
854 | 197 | self.write_hook("config-changed", "#!/bin/bash\nexit 0") | 278 | self.write_hook("config-changed", "#!/bin/bash\nexit 0") |
856 | 198 | result = yield self.workflow.fire_transition("retry_configure") | 279 | result = yield self.workflow.fire_transition_alias("retry_hook") |
857 | 199 | yield hook_deferred | 280 | yield hook_deferred |
858 | 200 | self.assertTrue(result) | ||
859 | 201 | yield self.assertState(self.workflow, "started") | 281 | yield self.assertState(self.workflow, "started") |
860 | 202 | 282 | ||
861 | 283 | # Stop any background processing | ||
862 | 284 | yield self.workflow.fire_transition("stop") | ||
863 | 285 | yield self.sleep(0.1) | ||
864 | 286 | |||
865 | 203 | @inlineCallbacks | 287 | @inlineCallbacks |
866 | 204 | def test_upgrade(self): | 288 | def test_upgrade(self): |
867 | 205 | """Upgrading a workflow results in the upgrade hook being | 289 | """Upgrading a workflow results in the upgrade hook being |
868 | @@ -235,7 +319,7 @@ | |||
869 | 235 | yield self.workflow.fire_transition("stop") | 319 | yield self.workflow.fire_transition("stop") |
870 | 236 | 320 | ||
871 | 237 | @inlineCallbacks | 321 | @inlineCallbacks |
873 | 238 | def test_upgrade_error_state(self): | 322 | def test_upgrade_error_retry(self): |
874 | 239 | """A hook error during an upgrade transitions to | 323 | """A hook error during an upgrade transitions to |
875 | 240 | upgrade_error. | 324 | upgrade_error. |
876 | 241 | """ | 325 | """ |
877 | @@ -259,6 +343,41 @@ | |||
878 | 259 | yield self.workflow.fire_transition("retry_upgrade_formula") | 343 | yield self.workflow.fire_transition("retry_upgrade_formula") |
879 | 260 | current_state = yield self.workflow.get_state() | 344 | current_state = yield self.workflow.get_state() |
880 | 261 | self.assertEqual(current_state, "started") | 345 | self.assertEqual(current_state, "started") |
881 | 346 | |||
882 | 347 | # Stop any background activity | ||
883 | 348 | yield self.workflow.fire_transition("stop") | ||
884 | 349 | |||
885 | 350 | @inlineCallbacks | ||
886 | 351 | def test_upgrade_error_retry_hook(self): | ||
887 | 352 | """A hook error during an upgrade transitions to | ||
888 | 353 | upgrade_error, and can be re-tried with hook execution. | ||
889 | 354 | """ | ||
890 | 355 | yield self.workflow.fire_transition("install") | ||
891 | 356 | yield self.workflow.fire_transition("start") | ||
892 | 357 | current_state = yield self.workflow.get_state() | ||
893 | 358 | self.assertEqual(current_state, "started") | ||
894 | 359 | |||
895 | 360 | # Agent prepares this. | ||
896 | 361 | self.executor.stop() | ||
897 | 362 | |||
898 | 363 | self.write_hook("upgrade-formula", "#!/bin/bash\nexit 1") | ||
899 | 364 | hook_deferred = self.wait_on_hook("upgrade-formula") | ||
900 | 365 | yield self.workflow.fire_transition("upgrade_formula") | ||
901 | 366 | yield hook_deferred | ||
902 | 367 | current_state = yield self.workflow.get_state() | ||
903 | 368 | self.assertEqual(current_state, "formula_upgrade_error") | ||
904 | 369 | |||
905 | 370 | hook_deferred = self.wait_on_hook("upgrade-formula") | ||
906 | 371 | self.write_hook("upgrade-formula", "#!/bin/bash\nexit 0") | ||
907 | 372 | # The upgrade error hook should ensure that the executor is stoppped. | ||
908 | 373 | self.assertFalse(self.executor.running) | ||
909 | 374 | yield self.workflow.fire_transition_alias("retry_hook") | ||
910 | 375 | yield hook_deferred | ||
911 | 376 | current_state = yield self.workflow.get_state() | ||
912 | 377 | self.assertEqual(current_state, "started") | ||
913 | 378 | self.assertTrue(self.executor.running) | ||
914 | 379 | |||
915 | 380 | # Stop any background activity | ||
916 | 262 | yield self.workflow.fire_transition("stop") | 381 | yield self.workflow.fire_transition("stop") |
917 | 263 | 382 | ||
918 | 264 | @inlineCallbacks | 383 | @inlineCallbacks |
919 | @@ -301,13 +420,36 @@ | |||
920 | 301 | self.assertTrue(result) | 420 | self.assertTrue(result) |
921 | 302 | result = yield self.workflow.fire_transition("start") | 421 | result = yield self.workflow.fire_transition("start") |
922 | 303 | self.assertTrue(result) | 422 | self.assertTrue(result) |
923 | 423 | |||
924 | 304 | self.write_hook("stop", "#!/bin/bash\nexit 1") | 424 | self.write_hook("stop", "#!/bin/bash\nexit 1") |
925 | 305 | result = yield self.workflow.fire_transition("stop") | 425 | result = yield self.workflow.fire_transition("stop") |
926 | 306 | self.assertFalse(result) | 426 | self.assertFalse(result) |
929 | 307 | current_state = yield self.workflow.get_state() | 427 | |
930 | 308 | self.assertEqual(current_state, "stop_error") | 428 | yield self.assertState(self.workflow, "stop_error") |
931 | 309 | self.write_hook("stop", "#!/bin/bash\necho hello\n") | 429 | self.write_hook("stop", "#!/bin/bash\necho hello\n") |
932 | 310 | result = yield self.workflow.fire_transition("retry_stop") | 430 | result = yield self.workflow.fire_transition("retry_stop") |
933 | 431 | |||
934 | 432 | yield self.assertState(self.workflow, "stopped") | ||
935 | 433 | |||
936 | 434 | @inlineCallbacks | ||
937 | 435 | def test_stop_error_with_retry_hook(self): | ||
938 | 436 | self.write_hook("install", "#!/bin/bash\necho hello\n") | ||
939 | 437 | self.write_hook("start", "#!/bin/bash\necho hello\n") | ||
940 | 438 | result = yield self.workflow.fire_transition("install") | ||
941 | 439 | self.assertTrue(result) | ||
942 | 440 | result = yield self.workflow.fire_transition("start") | ||
943 | 441 | self.assertTrue(result) | ||
944 | 442 | |||
945 | 443 | self.write_hook("stop", "#!/bin/bash\nexit 1") | ||
946 | 444 | result = yield self.workflow.fire_transition("stop") | ||
947 | 445 | self.assertFalse(result) | ||
948 | 446 | yield self.assertState(self.workflow, "stop_error") | ||
949 | 447 | |||
950 | 448 | result = yield self.workflow.fire_transition_alias("retry_hook") | ||
951 | 449 | yield self.assertState(self.workflow, "stop_error") | ||
952 | 450 | |||
953 | 451 | self.write_hook("stop", "#!/bin/bash\nexit 0") | ||
954 | 452 | result = yield self.workflow.fire_transition_alias("retry_hook") | ||
955 | 311 | yield self.assertState(self.workflow, "stopped") | 453 | yield self.assertState(self.workflow, "stopped") |
956 | 312 | 454 | ||
957 | 313 | @inlineCallbacks | 455 | @inlineCallbacks |
958 | @@ -447,7 +589,7 @@ | |||
959 | 447 | # Add a new unit, and wait for the broken hook to result in | 589 | # Add a new unit, and wait for the broken hook to result in |
960 | 448 | # the transition to the down state. | 590 | # the transition to the down state. |
961 | 449 | yield self.add_opposite_service_unit(self.states) | 591 | yield self.add_opposite_service_unit(self.states) |
963 | 450 | yield self.wait_on_state(self.workflow, "down") | 592 | yield self.wait_on_state(self.workflow, "error") |
964 | 451 | 593 | ||
965 | 452 | f_state, history, zk_state = yield self.read_persistent_state( | 594 | f_state, history, zk_state = yield self.read_persistent_state( |
966 | 453 | history_id=self.workflow.zk_state_id) | 595 | history_id=self.workflow.zk_state_id) |
967 | @@ -458,7 +600,7 @@ | |||
968 | 458 | "formula", "hooks", "app-relation-changed")) | 600 | "formula", "hooks", "app-relation-changed")) |
969 | 459 | 601 | ||
970 | 460 | self.assertEqual(f_state, | 602 | self.assertEqual(f_state, |
972 | 461 | {"state": "down", | 603 | {"state": "error", |
973 | 462 | "state_variables": { | 604 | "state_variables": { |
974 | 463 | "change_type": "joined", | 605 | "change_type": "joined", |
975 | 464 | "error_message": error}}) | 606 | "error_message": error}}) |
976 | 465 | 607 | ||
977 | === modified file 'ensemble/unit/workflow.py' | |||
978 | --- ensemble/unit/workflow.py 2011-05-05 14:31:43 +0000 | |||
979 | +++ ensemble/unit/workflow.py 2011-05-05 17:43:23 +0000 | |||
980 | @@ -14,31 +14,56 @@ | |||
981 | 14 | 14 | ||
982 | 15 | 15 | ||
983 | 16 | UnitWorkflow = Workflow( | 16 | UnitWorkflow = Workflow( |
984 | 17 | # Install transitions | ||
985 | 17 | Transition("install", "Install", None, "installed", | 18 | Transition("install", "Install", None, "installed", |
986 | 18 | error_transition_id="error_install"), | 19 | error_transition_id="error_install"), |
989 | 19 | Transition("error_install", "Install Error", None, "install_error"), | 20 | Transition("error_install", "Install error", None, "install_error"), |
990 | 20 | Transition("retry_install", "Retry Install", "install_error", "installed"), | 21 | |
991 | 22 | Transition("retry_install", "Retry install", "install_error", "installed", | ||
992 | 23 | alias="retry"), | ||
993 | 24 | Transition("retry_install_hook", "Retry install with hook", | ||
994 | 25 | "install_error", "installed", alias="retry_hook"), | ||
995 | 26 | |||
996 | 27 | # Start transitions | ||
997 | 21 | Transition("start", "Start", "installed", "started", | 28 | Transition("start", "Start", "installed", "started", |
998 | 22 | error_transition_id="error_start"), | 29 | error_transition_id="error_start"), |
1001 | 23 | Transition("error_start", "Start Error", "installed", "start_error"), | 30 | Transition("error_start", "Start error", "installed", "start_error"), |
1002 | 24 | Transition("retry_start", "Retry Start", "start_error", "started"), | 31 | Transition("retry_start", "Retry start", "start_error", "started", |
1003 | 32 | alias="retry"), | ||
1004 | 33 | Transition("retry_start_hook", "Retry start with hook", | ||
1005 | 34 | "start_error", "started", alias="retry_hook"), | ||
1006 | 35 | |||
1007 | 36 | # Stop transitions | ||
1008 | 25 | Transition("stop", "Stop", "started", "stopped", | 37 | Transition("stop", "Stop", "started", "stopped", |
1009 | 26 | error_transition_id="error_stop"), | 38 | error_transition_id="error_stop"), |
1014 | 27 | Transition("error_stop", "Stop Error", "started", "stop_error"), | 39 | Transition("error_stop", "Stop error", "started", "stop_error"), |
1015 | 28 | Transition("retry_stop", "Retry Stop", "stop_error", "stopped"), | 40 | Transition("retry_stop", "Retry stop", "stop_error", "stopped", |
1016 | 29 | 41 | alias="retry"), | |
1017 | 30 | # Upgrade Transitions (stay in state, with success transitition) | 42 | Transition("retry_stop_hook", "Retry stop with hook", |
1018 | 43 | "stop_error", "stopped", alias="retry_hook"), | ||
1019 | 44 | |||
1020 | 45 | # Restart transitions | ||
1021 | 46 | Transition("restart", "Restart", "stop", "start", | ||
1022 | 47 | error_transition_id="error_start", alias="retry"), | ||
1023 | 48 | Transition("restart_with_hook", "Restart with hook", | ||
1024 | 49 | "stop", "start", alias="retry_hook", | ||
1025 | 50 | error_transition_id="error_start"), | ||
1026 | 51 | |||
1027 | 52 | # Upgrade transitions | ||
1028 | 31 | Transition( | 53 | Transition( |
1029 | 32 | "upgrade_formula", "Upgrade", "started", "started", | 54 | "upgrade_formula", "Upgrade", "started", "started", |
1030 | 33 | error_transition_id="upgrade_formula_error"), | 55 | error_transition_id="upgrade_formula_error"), |
1031 | 34 | Transition( | 56 | Transition( |
1033 | 35 | "upgrade_formula_error", "On upgrade error", | 57 | "upgrade_formula_error", "Upgrade from stop error", |
1034 | 36 | "started", "formula_upgrade_error"), | 58 | "started", "formula_upgrade_error"), |
1035 | 37 | Transition( | 59 | Transition( |
1038 | 38 | "retry_upgrade_formula", "Retry failed upgrade", | 60 | "retry_upgrade_formula", "Upgrade from stop error", |
1039 | 39 | "formula_upgrade_error", "started"), | 61 | "formula_upgrade_error", "started", alias="retry"), |
1040 | 62 | Transition( | ||
1041 | 63 | "retry_upgrade_formula_hook", "Upgrade from stop error with hook", | ||
1042 | 64 | "formula_upgrade_error", "started", alias="retry_hook"), | ||
1043 | 40 | 65 | ||
1045 | 41 | # Configuration Transitions (stay in state, with success transition) | 66 | # Configuration Transitions |
1046 | 42 | Transition( | 67 | Transition( |
1047 | 43 | "reconfigure", "Reconfigure", "started", "started", | 68 | "reconfigure", "Reconfigure", "started", "started", |
1048 | 44 | error_transition_id="error_configure"), | 69 | error_transition_id="error_configure"), |
1049 | @@ -46,28 +71,56 @@ | |||
1050 | 46 | "error_configure", "On configure error", | 71 | "error_configure", "On configure error", |
1051 | 47 | "started", "configure_error"), | 72 | "started", "configure_error"), |
1052 | 48 | Transition( | 73 | Transition( |
1053 | 74 | "retry_error", "On retry configure error", | ||
1054 | 75 | "configure_error", "configure_error"), | ||
1055 | 76 | Transition( | ||
1056 | 49 | "retry_configure", "Retry configure", | 77 | "retry_configure", "Retry configure", |
1058 | 50 | "configure_error", "started") | 78 | "configure_error", "started", alias="retry", |
1059 | 79 | error_transition_id="retry_error"), | ||
1060 | 80 | Transition( | ||
1061 | 81 | "retry_configure_hook", "Retry configure with hooks", | ||
1062 | 82 | "configure_error", "started", alias="retry_hook", | ||
1063 | 83 | error_transition_id="retry_error") | ||
1064 | 51 | ) | 84 | ) |
1065 | 52 | 85 | ||
1066 | 53 | 86 | ||
1079 | 54 | # There's been some discussion, if we should have per change type error states | 87 | # Unit relation error states |
1080 | 55 | # here, corresponding to the different changes that the relation-changed hook | 88 | # |
1081 | 56 | # is invoked for. The important aspects to capture are both observability of | 89 | # There's been some discussion, if we should have per change type |
1082 | 57 | # error type locally and globally (zk), and per error type and instance | 90 | # error states here, corresponding to the different changes that the |
1083 | 58 | # recovery of the same. To provide for this functionality without additional | 91 | # relation-changed hook is invoked for. The important aspects to |
1084 | 59 | # states, the error information (change type, and error message) are captured | 92 | # capture are both observability of error type locally and globally |
1085 | 60 | # in state variables which are locally and globally observable. Future | 93 | # (zk), and per error type and instance recovery of the same. To |
1086 | 61 | # extension of the restart transition action, will allow for customized | 94 | # provide for this functionality without additional states, the error |
1087 | 62 | # recovery based on the change type state variable. Effectively this | 95 | # information (change type, and error message) are captured in state |
1088 | 63 | # differs from the unit definition, in that it collapses three possible error | 96 | # variables which are locally and globally observable. Future |
1089 | 64 | # states, into a behavior off switch. A separate state will be needed to | 97 | # extension of the restart transition action, will allow for |
1090 | 65 | # denote departing. | 98 | # customized recovery based on the change type state |
1091 | 99 | # variable. Effectively this differs from the unit definition, in that | ||
1092 | 100 | # it collapses three possible error states, into a behavior off | ||
1093 | 101 | # switch. A separate state will be needed to denote departing. | ||
1094 | 102 | |||
1095 | 103 | |||
1096 | 104 | # Process recovery using on disk workflow state | ||
1097 | 105 | # | ||
1098 | 106 | # Another interesting issue, process recovery using the on disk state, | ||
1099 | 107 | # is complicated by consistency to the the in memory state, which | ||
1100 | 108 | # won't be directly recoverable anymore without some state specific | ||
1101 | 109 | # semantics to recovering from on disk state, ie a restarted unit | ||
1102 | 110 | # agent, with a relation in an error state would require special | ||
1103 | 111 | # semantics around loading from disk to ensure that the in-memory | ||
1104 | 112 | # process state (watching and scheduling but not executing) matches | ||
1105 | 113 | # the recovery transition actions (which just restart hook execution, | ||
1106 | 114 | # but assume the watch continues).. this functionality added to better | ||
1107 | 115 | # allow for the behavior that while down due to a hook error, the | ||
1108 | 116 | # relation would continues to schedule pending hooks | ||
1109 | 66 | 117 | ||
1110 | 67 | RelationWorkflow = Workflow( | 118 | RelationWorkflow = Workflow( |
1111 | 68 | Transition("start", "Start", None, "up"), | 119 | Transition("start", "Start", None, "up"), |
1112 | 69 | Transition("stop", "Stop", "up", "down"), | 120 | Transition("stop", "Stop", "up", "down"), |
1114 | 70 | Transition("restart", "Restart", "down", "up"), | 121 | Transition("restart", "Restart", "down", "up", alias="retry"), |
1115 | 122 | Transition("error", "Relation hook error", "up", "error"), | ||
1116 | 123 | Transition("reset", "Recover from hook error", "error", "up"), | ||
1117 | 71 | Transition("depart", "Relation broken", "up", "departed"), | 124 | Transition("depart", "Relation broken", "up", "departed"), |
1118 | 72 | Transition("down_depart", "Relation broken", "down", "departed"), | 125 | Transition("down_depart", "Relation broken", "down", "departed"), |
1119 | 73 | ) | 126 | ) |
1120 | @@ -223,7 +276,6 @@ | |||
1121 | 223 | row per entry with CSV escaping. | 276 | row per entry with CSV escaping. |
1122 | 224 | """ | 277 | """ |
1123 | 225 | state_serialized = yaml.safe_dump(state_dict) | 278 | state_serialized = yaml.safe_dump(state_dict) |
1124 | 226 | |||
1125 | 227 | # State File | 279 | # State File |
1126 | 228 | with open(self.state_file_path, "w") as handle: | 280 | with open(self.state_file_path, "w") as handle: |
1127 | 229 | handle.write(state_serialized) | 281 | handle.write(state_serialized) |
1128 | @@ -243,6 +295,7 @@ | |||
1129 | 243 | return {"state": None} | 295 | return {"state": None} |
1130 | 244 | with open(self.state_file_path, "r") as handle: | 296 | with open(self.state_file_path, "r") as handle: |
1131 | 245 | content = handle.read() | 297 | content = handle.read() |
1132 | 298 | |||
1133 | 246 | return yaml.load(content) | 299 | return yaml.load(content) |
1134 | 247 | 300 | ||
1135 | 248 | 301 | ||
1136 | @@ -282,51 +335,77 @@ | |||
1137 | 282 | self._lifecycle = lifecycle | 335 | self._lifecycle = lifecycle |
1138 | 283 | 336 | ||
1139 | 284 | @inlineCallbacks | 337 | @inlineCallbacks |
1141 | 285 | def _invoke_lifecycle(self, method): | 338 | def _invoke_lifecycle(self, method, *args, **kw): |
1142 | 286 | try: | 339 | try: |
1144 | 287 | result = yield method() | 340 | result = yield method(*args, **kw) |
1145 | 288 | except (FileNotFound, FormulaError, FormulaInvocationError), e: | 341 | except (FileNotFound, FormulaError, FormulaInvocationError), e: |
1146 | 289 | raise TransitionError(e) | 342 | raise TransitionError(e) |
1147 | 290 | returnValue(result) | 343 | returnValue(result) |
1148 | 291 | 344 | ||
1150 | 292 | # Transition Actions | 345 | # Install transitions |
1151 | 293 | def do_install(self): | 346 | def do_install(self): |
1152 | 294 | return self._invoke_lifecycle(self._lifecycle.install) | 347 | return self._invoke_lifecycle(self._lifecycle.install) |
1153 | 295 | 348 | ||
1154 | 349 | def do_retry_install(self): | ||
1155 | 350 | return self._invoke_lifecycle(self._lifecycle.install, | ||
1156 | 351 | fire_hooks=False) | ||
1157 | 352 | |||
1158 | 353 | def do_retry_install_hook(self): | ||
1159 | 354 | return self._invoke_lifecycle(self._lifecycle.install) | ||
1160 | 355 | |||
1161 | 356 | # Start transitions | ||
1162 | 296 | def do_start(self): | 357 | def do_start(self): |
1163 | 297 | return self._invoke_lifecycle(self._lifecycle.start) | 358 | return self._invoke_lifecycle(self._lifecycle.start) |
1164 | 298 | 359 | ||
1165 | 299 | def do_stop(self): | ||
1166 | 300 | return self._invoke_lifecycle(self._lifecycle.stop) | ||
1167 | 301 | |||
1168 | 302 | def do_retry_start(self): | 360 | def do_retry_start(self): |
1169 | 361 | return self._invoke_lifecycle(self._lifecycle.start, | ||
1170 | 362 | fire_hooks=False) | ||
1171 | 363 | |||
1172 | 364 | def do_retry_start_hook(self): | ||
1173 | 303 | return self._invoke_lifecycle(self._lifecycle.start) | 365 | return self._invoke_lifecycle(self._lifecycle.start) |
1174 | 304 | 366 | ||
1175 | 367 | # Stop transitions | ||
1176 | 368 | def do_stop(self): | ||
1177 | 369 | return self._invoke_lifecycle(self._lifecycle.stop) | ||
1178 | 370 | |||
1179 | 305 | def do_retry_stop(self): | 371 | def do_retry_stop(self): |
1184 | 306 | self._invoke_lifecycle(self._lifecycle.stop) | 372 | return self._invoke_lifecycle(self._lifecycle.stop, |
1185 | 307 | 373 | fire_hooks=False) | |
1186 | 308 | def do_retry_install(self): | 374 | |
1187 | 309 | return self._invoke_lifecycle(self._lifecycle.install) | 375 | def do_retry_stop_hook(self): |
1188 | 376 | return self._invoke_lifecycle(self._lifecycle.stop) | ||
1189 | 377 | |||
1190 | 378 | # Upgrade transititions | ||
1191 | 379 | def do_upgrade_formula(self): | ||
1192 | 380 | return self._invoke_lifecycle(self._lifecycle.upgrade_formula) | ||
1193 | 310 | 381 | ||
1194 | 311 | def do_retry_upgrade_formula(self): | 382 | def do_retry_upgrade_formula(self): |
1202 | 312 | return self._invoke_lifecycle(self._lifecycle.upgrade_formula) | 383 | return self._invoke_lifecycle(self._lifecycle.upgrade_formula, |
1203 | 313 | 384 | fire_hooks=False) | |
1204 | 314 | def do_upgrade_formula(self): | 385 | |
1205 | 315 | return self._invoke_lifecycle(self._lifecycle.upgrade_formula) | 386 | def do_retry_upgrade_formula_hook(self): |
1206 | 316 | 387 | return self._invoke_lifecycle(self._lifecycle.upgrade_formula) | |
1207 | 317 | # Some of this needs support from the resolved branches, as we | 388 | |
1208 | 318 | # want to fire some of these lifecycle methods sans hooks. | 389 | # Config transitions |
1209 | 319 | def do_error_configure(self): | 390 | def do_error_configure(self): |
1212 | 320 | # self._invoke_lifecycle(self._lifecycle.stop, fire_hooks=False) | 391 | return self._invoke_lifecycle(self._lifecycle.stop, fire_hooks=False) |
1211 | 321 | pass | ||
1213 | 322 | 392 | ||
1214 | 323 | def do_reconfigure(self): | 393 | def do_reconfigure(self): |
1215 | 324 | return self._invoke_lifecycle(self._lifecycle.configure) | 394 | return self._invoke_lifecycle(self._lifecycle.configure) |
1216 | 325 | 395 | ||
1217 | 396 | def do_retry_error(self): | ||
1218 | 397 | return self._invoke_lifecycle(self._lifecycle.stop, fire_hooks=False) | ||
1219 | 398 | |||
1220 | 399 | @inlineCallbacks | ||
1221 | 326 | def do_retry_configure(self): | 400 | def do_retry_configure(self): |
1225 | 327 | # self._invoke_lifecycle(self._lifecycle.start, fire_hooks=False) | 401 | yield self._invoke_lifecycle(self._lifecycle.start, fire_hooks=False) |
1226 | 328 | self._invoke_lifecycle( | 402 | yield self._invoke_lifecycle(self._lifecycle.configure, |
1227 | 329 | self._lifecycle.configure) # fire_hooks=False) | 403 | fire_hooks=False) |
1228 | 404 | |||
1229 | 405 | @inlineCallbacks | ||
1230 | 406 | def do_retry_configure_hook(self): | ||
1231 | 407 | yield self._invoke_lifecycle(self._lifecycle.start, fire_hooks=False) | ||
1232 | 408 | yield self._invoke_lifecycle(self._lifecycle.configure) | ||
1233 | 330 | 409 | ||
1234 | 331 | 410 | ||
1235 | 332 | class RelationWorkflowState(DiskWorkflowState): | 411 | class RelationWorkflowState(DiskWorkflowState): |
1236 | @@ -360,7 +439,7 @@ | |||
1237 | 360 | 439 | ||
1238 | 361 | @param: error: The error from hook invocation. | 440 | @param: error: The error from hook invocation. |
1239 | 362 | """ | 441 | """ |
1241 | 363 | yield self.fire_transition("stop", | 442 | yield self.fire_transition("error", |
1242 | 364 | change_type=relation_change.change_type, | 443 | change_type=relation_change.change_type, |
1243 | 365 | error_message=str(error)) | 444 | error_message=str(error)) |
1244 | 366 | 445 | ||
1245 | @@ -369,12 +448,30 @@ | |||
1246 | 369 | """Transition the workflow to the 'down' state. | 448 | """Transition the workflow to the 'down' state. |
1247 | 370 | 449 | ||
1248 | 371 | Turns off the unit-relation lifecycle monitoring and hook execution. | 450 | Turns off the unit-relation lifecycle monitoring and hook execution. |
1249 | 451 | |||
1250 | 452 | :param error_info: If called on relation hook error, contains | ||
1251 | 453 | error variables. | ||
1252 | 372 | """ | 454 | """ |
1253 | 373 | yield self._lifecycle.stop() | 455 | yield self._lifecycle.stop() |
1254 | 374 | 456 | ||
1255 | 375 | @inlineCallbacks | 457 | @inlineCallbacks |
1256 | 458 | def do_reset(self): | ||
1257 | 459 | """Transition the workflow to the 'up' state from an error state. | ||
1258 | 460 | |||
1259 | 461 | Turns on the unit-relation lifecycle monitoring and hook execution. | ||
1260 | 462 | """ | ||
1261 | 463 | yield self._lifecycle.start(watches=False) | ||
1262 | 464 | |||
1263 | 465 | @inlineCallbacks | ||
1264 | 466 | def do_error(self, **error_info): | ||
1265 | 467 | """A relation hook error, stops further execution hooks but | ||
1266 | 468 | continues to watch for changes. | ||
1267 | 469 | """ | ||
1268 | 470 | yield self._lifecycle.stop(watches=False) | ||
1269 | 471 | |||
1270 | 472 | @inlineCallbacks | ||
1271 | 376 | def do_restart(self): | 473 | def do_restart(self): |
1273 | 377 | """Transition the workflow to the 'up' state. | 474 | """Transition the workflow to the 'up' state from the down state. |
1274 | 378 | 475 | ||
1275 | 379 | Turns on the unit-relation lifecycle monitoring and hook execution. | 476 | Turns on the unit-relation lifecycle monitoring and hook execution. |
1276 | 380 | """ | 477 | """ |
Kapil mentioned he's still working on this one.