Merge lp:~bigdata-dev/charms/trusty/apache-hadoop-compute-slave/status into lp:~bigdata-dev/charms/trusty/apache-hadoop-compute-slave/trunk

Proposed by Cory Johns on 2015-06-24
Status: Merged
Merged at revision: 85
Proposed branch: lp:~bigdata-dev/charms/trusty/apache-hadoop-compute-slave/status
Merge into: lp:~bigdata-dev/charms/trusty/apache-hadoop-compute-slave/trunk
Diff against target: 238 lines (+85/-46)
4 files modified
hooks/callbacks.py (+43/-0)
hooks/common.py (+33/-31)
hooks/setup.py (+8/-10)
resources.yaml (+1/-5)
To merge this branch: bzr merge lp:~bigdata-dev/charms/trusty/apache-hadoop-compute-slave/status
Reviewer Review Type Date Requested Status
amir sanjar (community) Approve on 2015-06-25
Kevin W Monroe 2015-06-24 Approve on 2015-06-25
Review via email: mp+262927@code.launchpad.net

Description of the change

Added extended status support.

Example status (tabular) outputs:

http://pastebin.ubuntu.com/11769979/ (in progress)
http://pastebin.ubuntu.com/11770196/ (complete)

Example status-history outputs:

http://pastebin.ubuntu.com/11770192/ (hdfs-master)
http://pastebin.ubuntu.com/11770195/ (compute-slave)

To post a comment you must log in.
Cory Johns (johnsca) wrote :

I also bundled charmhelpers and its dependencies, since there is almost no churn on it now and it made the status handling code cleaner as I could depend on hookenv.status_set() always being there.

Kevin W Monroe (kwmonroe) wrote :

Looks really good Cory! I deployed to AWS and all the status changes looked legit (i like the inclusion of '/etc/hosts registration' in addition to the normal boring enhanced status).

I have a beef, though. This feels heavy:

+ archives = glob('resources/*')
+ subprocess.check_call(['pip', 'install'] + archives)

What if we want to shove things in ./resources that aren't pip'able? We can either cross that bridge when we get to it, or shove pip stuffs in ./resources/pip. Thoughts?

Cory Johns (johnsca) wrote :

> Looks really good Cory! I deployed to AWS and all the status changes looked
> legit (i like the inclusion of '/etc/hosts registration' in addition to the
> normal boring enhanced status).

What /etc/hosts registration are you referring to?

> I have a beef, though. This feels heavy:
>
> + archives = glob('resources/*')
> + subprocess.check_call(['pip', 'install'] + archives)
>
> What if we want to shove things in ./resources that aren't pip'able? We can
> either cross that bridge when we get to it, or shove pip stuffs in
> ./resources/pip. Thoughts?

Yeah. Honestly, I just got tired of adding things there that I had missed. Could do a python subdirectory, though.

86. By Cory Johns on 2015-06-25

Nested python resources to avoid attempting to pip install future non-python resources

Kevin W Monroe (kwmonroe) wrote :

+1 Cory! LGTM, merged into r85.

review: Approve
amir sanjar (asanjar) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'hooks/callbacks.py'
2--- hooks/callbacks.py 2015-05-12 21:56:13 +0000
3+++ hooks/callbacks.py 2015-06-25 13:44:06 +0000
4@@ -15,3 +15,46 @@
5 Add any additional tasks / setup here. If a callback is used by mutliple
6 charms, consider refactoring it up to the jujubigdata library.
7 """
8+
9+from charmhelpers.core import hookenv
10+from charmhelpers.core import unitdata
11+from jujubigdata.relations import NameNodeMaster, ResourceManagerMaster
12+
13+
14+def update_blocked_status():
15+ if unitdata.kv().get('charm.active', False):
16+ return
17+ rels = (
18+ ('Yarn', 'ResourceManager', ResourceManagerMaster()),
19+ ('HDFS', 'NameNode', NameNodeMaster()),
20+ )
21+ missing_rel = [rel for rel, res, impl in rels if not impl.connected_units()]
22+ missing_hosts = [rel for rel, res, impl in rels if not impl.am_i_registered()]
23+ not_ready = [(rel, res) for rel, res, impl in rels if not impl.is_ready()]
24+ if missing_rel:
25+ hookenv.status_set('blocked', 'Waiting for relation to %s master%s' % (
26+ ' and '.join(missing_rel),
27+ 's' if len(missing_rel) > 1 else '',
28+ )),
29+ elif missing_hosts:
30+ hookenv.status_set('waiting', 'Waiting for /etc/hosts registration on %s' % (
31+ ' and '.join(missing_hosts),
32+ ))
33+ elif not_ready:
34+ unready_rels, unready_ress = zip(*not_ready)
35+ hookenv.status_set('waiting', 'Waiting for %s to provide %s' % (
36+ ' and '.join(unready_rels),
37+ ' and '.join(unready_ress),
38+ ))
39+
40+
41+def update_working_status():
42+ if unitdata.kv().get('charm.active', False):
43+ hookenv.status_set('maintenance', 'Updating configuration')
44+ return
45+ hookenv.status_set('maintenance', 'Setting up NodeManager and DataNode')
46+
47+
48+def update_active_status():
49+ unitdata.kv().set('charm.active', True)
50+ hookenv.status_set('active', 'Ready')
51
52=== modified file 'hooks/common.py'
53--- hooks/common.py 2015-06-19 17:24:59 +0000
54+++ hooks/common.py 2015-06-25 13:44:06 +0000
55@@ -15,29 +15,39 @@
56 """
57
58 import jujuresources
59+from charmhelpers.core import hookenv
60+from charmhelpers.core import unitdata
61+from charmhelpers.core import charmframework
62
63
64 def bootstrap_resources():
65 """
66 Install required resources defined in resources.yaml
67 """
68+ if unitdata.kv().get('charm.bootstrapped', False):
69+ return True
70+ hookenv.status_set('maintenance', 'Installing base resources')
71 mirror_url = jujuresources.config_get('resources_mirror')
72 if not jujuresources.fetch(mirror_url=mirror_url):
73- jujuresources.juju_log('Resources unavailable; manual intervention required', 'ERROR')
74+ missing = jujuresources.invalid()
75+ hookenv.status_set('blocked', 'Unable to fetch required resource%s: %s' % (
76+ 's' if len(missing) > 1 else '',
77+ ', '.join(missing),
78+ ))
79 return False
80- jujuresources.install(['pathlib', 'pyaml', 'six', 'charmhelpers', 'jujubigdata'])
81+ jujuresources.install(['pathlib', 'jujubigdata'])
82+ unitdata.kv().set('charm.bootstrapped', True)
83 return True
84
85
86 def manage():
87 if not bootstrap_resources():
88- # defer until resources are available, since charmhelpers, and thus
89- # the framework, (will be) managed by jujuresources
90+ # defer until resources are available, since jujubigdata, and thus the
91+ # classes needed for the requires blocks, (will be) managed by jujuresources
92 return
93
94- from charmhelpers.core import charmframework
95 import jujubigdata
96- import callbacks # noqa (ignore when linting)
97+ import callbacks
98
99 # list of keys required to be in the dist.yaml
100 slave_reqs = ['vendor', 'hadoop_version', 'packages', 'groups', 'users',
101@@ -57,51 +67,43 @@
102 ],
103 'callbacks': [
104 hadoop.install,
105+ callbacks.update_blocked_status,
106 ],
107 },
108 {
109- 'name': 'compute-slave-hdfs',
110+ 'name': 'compute-slave',
111 'provides': [
112 jujubigdata.relations.DataNode(),
113+ jujubigdata.relations.NodeManager(),
114 ],
115 'requires': [
116 hadoop.is_installed,
117 hdfs_relation,
118+ yarn_relation,
119 hdfs_relation.am_i_registered,
120+ yarn_relation.am_i_registered,
121 ],
122 'callbacks': [
123+ callbacks.update_working_status,
124 hdfs_relation.register_provided_hosts,
125+ yarn_relation.register_provided_hosts,
126 jujubigdata.utils.manage_etc_hosts,
127 hdfs_relation.install_ssh_keys,
128+ yarn_relation.install_ssh_keys,
129 hdfs.configure_datanode,
130+ yarn.configure_nodemanager,
131 hdfs.start_datanode,
132- charmframework.helpers.open_ports(dist_config.exposed_ports('compute-slave-hdfs')),
133+ yarn.start_nodemanager,
134+ charmframework.helpers.open_ports(
135+ dist_config.exposed_ports('compute-slave-hdfs') +
136+ dist_config.exposed_ports('compute-slave-yarn')),
137+ callbacks.update_active_status,
138 ],
139 'cleanup': [
140- charmframework.helpers.close_ports(dist_config.exposed_ports('compute-slave-hdfs')),
141+ charmframework.helpers.close_ports(
142+ dist_config.exposed_ports('compute-slave-hdfs') +
143+ dist_config.exposed_ports('compute-slave-yarn')),
144 hdfs.stop_datanode,
145- ],
146- },
147- {
148- 'name': 'compute-slave-yarn',
149- 'provides': [
150- jujubigdata.relations.NodeManager(),
151- ],
152- 'requires': [
153- hadoop.is_installed,
154- yarn_relation,
155- yarn_relation.am_i_registered,
156- ],
157- 'callbacks': [
158- yarn_relation.register_provided_hosts,
159- jujubigdata.utils.manage_etc_hosts,
160- yarn_relation.install_ssh_keys,
161- yarn.configure_nodemanager,
162- yarn.start_nodemanager,
163- charmframework.helpers.open_ports(dist_config.exposed_ports('compute-slave-yarn')),
164- ],
165- 'cleanup': [
166- charmframework.helpers.close_ports(dist_config.exposed_ports('compute-slave-yarn')),
167 yarn.stop_nodemanager,
168 ],
169 },
170
171=== modified file 'hooks/setup.py'
172--- hooks/setup.py 2015-03-03 22:38:32 +0000
173+++ hooks/setup.py 2015-06-25 13:44:06 +0000
174@@ -12,24 +12,22 @@
175 import subprocess
176 from glob import glob
177
178+
179 def pre_install():
180 """
181 Do any setup required before the install hook.
182 """
183 install_pip()
184- install_jujuresources()
185+ install_bundled_resources()
186
187
188 def install_pip():
189 subprocess.check_call(['apt-get', 'install', '-yq', 'python-pip', 'bzr'])
190
191
192-def install_jujuresources():
193- """
194- Install the bundled jujuresources library, if not present.
195- """
196- try:
197- import jujuresources # noqa
198- except ImportError:
199- jr_archive = glob('resources/jujuresources-*.tar.gz')[0]
200- subprocess.check_call(['pip', 'install', jr_archive])
201+def install_bundled_resources():
202+ """
203+ Install the bundled resources libraries.
204+ """
205+ archives = glob('resources/python/*')
206+ subprocess.check_call(['pip', 'install'] + archives)
207
208=== modified file 'resources.yaml'
209--- resources.yaml 2015-06-19 17:24:59 +0000
210+++ resources.yaml 2015-06-25 13:44:06 +0000
211@@ -3,12 +3,8 @@
212 resources:
213 pathlib:
214 pypi: path.py>=7.0
215- pyaml:
216- pypi: pyaml
217- six:
218- pypi: six
219 jujubigdata:
220- pypi: jujubigdata>=2.0.0,<3.0.0
221+ pypi: jujubigdata>=2.0.2,<3.0.0
222 charmhelpers:
223 pypi: http://bazaar.launchpad.net/~bigdata-dev/bigdata-data/trunk/download/kevin.monroe%40canonical.com-20150610215108-gq4ud2rriy1sp1h6/charmhelpers0.3.1.ta-20150610215050-vxrqhizdsrqwnmcm-1/charmhelpers-0.3.1.tar.gz
224 hash: 061fe7204289b96fab5d3ca02883040ea3026526ebf0ad38e21457527a18d139
225
226=== removed file 'resources/jujuresources-0.2.5.tar.gz'
227Binary files resources/jujuresources-0.2.5.tar.gz 2015-03-03 22:38:32 +0000 and resources/jujuresources-0.2.5.tar.gz 1970-01-01 00:00:00 +0000 differ
228=== added directory 'resources/python'
229=== added file 'resources/python/PyYAML-3.11.tar.gz'
230Binary files resources/python/PyYAML-3.11.tar.gz 1970-01-01 00:00:00 +0000 and resources/python/PyYAML-3.11.tar.gz 2015-06-25 13:44:06 +0000 differ
231=== added file 'resources/python/charmhelpers-0.3.1.tar.gz'
232Binary files resources/python/charmhelpers-0.3.1.tar.gz 1970-01-01 00:00:00 +0000 and resources/python/charmhelpers-0.3.1.tar.gz 2015-06-25 13:44:06 +0000 differ
233=== added file 'resources/python/jujuresources-0.2.8.tar.gz'
234Binary files resources/python/jujuresources-0.2.8.tar.gz 1970-01-01 00:00:00 +0000 and resources/python/jujuresources-0.2.8.tar.gz 2015-06-25 13:44:06 +0000 differ
235=== added file 'resources/python/pyaml-15.5.7.tar.gz'
236Binary files resources/python/pyaml-15.5.7.tar.gz 1970-01-01 00:00:00 +0000 and resources/python/pyaml-15.5.7.tar.gz 2015-06-25 13:44:06 +0000 differ
237=== added file 'resources/python/six-1.9.0-py2.py3-none-any.whl'
238Binary files resources/python/six-1.9.0-py2.py3-none-any.whl 1970-01-01 00:00:00 +0000 and resources/python/six-1.9.0-py2.py3-none-any.whl 2015-06-25 13:44:06 +0000 differ

Subscribers

People subscribed via source and target branches

to all changes: