Merge lp:~mattyw/charms/trusty/mongodb/mongodb-backup into lp:charms/trusty/mongodb
- Trusty Tahr (14.04)
- mongodb-backup
- Merge into trunk
Status: | Merged |
---|---|
Merged at revision: | 78 |
Proposed branch: | lp:~mattyw/charms/trusty/mongodb/mongodb-backup |
Merge into: | lp:charms/trusty/mongodb |
Diff against target: |
502 lines (+323/-28) 12 files modified
Makefile (+3/-1) actions.yaml (+22/-0) actions/backup.py (+56/-0) actions/backup_test.py (+52/-0) actions/dump (+4/-0) actions/restore (+4/-0) charm-helpers-sync.yaml (+1/-1) charmhelpers/core/hookenv.py (+43/-9) charmhelpers/core/host.py (+32/-16) charmhelpers/core/hugepage.py (+8/-1) charmhelpers/core/kernel.py (+68/-0) charmhelpers/core/strutils.py (+30/-0) |
To merge this branch: | bzr merge lp:~mattyw/charms/trusty/mongodb/mongodb-backup |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Charles Butler (community) | Approve | ||
Domas Monkus (community) | Approve | ||
Marco Ceppi (community) | Needs Fixing | ||
Ryan Beisner (community) | Needs Fixing | ||
Review via email: mp+273544@code.launchpad.net |
Commit message
Added actions/dump and actions/restore
These are actions that run mongodump and mongorestore.
Both actions take two parameters, the list of arguments and the workingDirectory. The simplest use case is just
juju action do mongodb/0 dump
# SSH into mongodb - drop all the data
juju action do mongodb/0 restore
# SSH into mongo to confirm the data is restored
Description of the change
Added actions/dump and actions/restore
These are actions that run mongodump and mongorestore.
Both actions take two parameters, the list of arguments and the workingDirectory. The simplest use case is just
juju action do mongodb/0 dump
# SSH into mongodb - drop all the data
juju action do mongodb/0 restore
# SSH into mongo to confirm the data is restored
uosci-testing-bot (uosci-testing-bot) wrote : | # |
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_unit_test #10582 mongodb for mattyw mp273544
UNIT OK: passed
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_amulet_test #7136 mongodb for mattyw mp273544
AMULET FAIL: amulet-test failed
AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.
Full amulet test output: http://
Build: http://
Ryan Beisner (1chb1n) wrote : | # |
Looks like a random interwebs failure during the apt process in the install hook on one of the mongo units. Re-triggering amulet test...
Ryan Beisner (1chb1n) wrote : | # |
FYI, for the curious, this was where it stumbled (seemingly unrelated to the proposed changes):
2015-10-06 15:51:38 INFO install W: Failed to fetch http://
Ryan Beisner (1chb1n) wrote : | # |
Please adjust the Makefile in your proposal to add lint test coverage of the new actions dir:
- .venv/bin/flake8 --exclude hooks/charmhelpers hooks tests unit_tests
+ .venv/bin/flake8 --exclude hooks/charmhelpers actions hooks tests unit_tests
Also: By default, flake8 will not check files in a directory unless they have a .py extension. Please use the .py extension on the action files.
Thank you
Ryan Beisner (1chb1n) wrote : | # |
@mattw
RE: irc discussion, yes, if you can adjust the existing action along with your new actions, that would be great.
I would bzr mv the action files to add the .py extensions, then add a symlink of the old action filename to the new .py filename.
Here is an example of this approach in different charm:
openstack_upgrade --> symlinks to --> openstack_
Thanks again!
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_amulet_test #7139 mongodb for mattyw mp273544
AMULET OK: passed
Build: http://
Marco Ceppi (marcoceppi) wrote : | # |
I've got a few suggestions.
First, we have action-get, action-set, and action-fail in charmhelpers. I'd consider re-syncing charm-helpers in hooks/charmhelpers to include these updated methods then importing them in the actions files.
Also, symlinking to a .py file is a bit tedious, instead to address the lack of linting coverage, I'd recommend the following patch: http://
Ryan Beisner (1chb1n) wrote : | # |
@mattw
From the test perspective: as long as the action code gets test coverage, I'm agnostic to how that is achieved in this charm.
Marco's example of adding file inspection logic to the Makefile is quite clever and a totally acceptable approach. Thanks @marcoceppi!
- 80. By Matthew Williams
-
Makefile: Added lint support for the actions, fixed linting issues found
- 81. By Matthew Williams
-
charmhelpers: Resync
- 82. By Matthew Williams
-
charmhelpers: sync
- 83. By Matthew Williams
-
actions/backup: Refactored the backup actions to allow unit testing
- 84. By Matthew Williams
-
actions/backup: unit tests for actions
- 85. By Matthew Williams
-
action/backup: Use the action functions from charmhelpers
Matthew Williams (mattyw) wrote : | # |
@beisner @marcoceppi PTAL
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_lint_check #11483 mongodb for mattyw mp273544
LINT FAIL: lint-test failed
LINT Results (max last 2 lines):
make: *** [lint] Error 1
ERROR:root:Make target returned non-zero.
Full lint test output: http://
Build: http://
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_unit_test #10676 mongodb for mattyw mp273544
UNIT FAIL: unit-test failed
UNIT Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.
Full unit test output: http://
Build: http://
- 86. By Matthew Williams
-
actions/backup: Don't need this import anymore
Domas Monkus (tasdomas) : | # |
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_lint_check #11485 mongodb for mattyw mp273544
LINT OK: passed
Build: http://
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_unit_test #10677 mongodb for mattyw mp273544
UNIT FAIL: unit-test failed
UNIT Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.
Full unit test output: http://
Build: http://
Matthew Williams (mattyw) wrote : | # |
Sorry for the large diff. charmhelpers has been moved to location accessible by hooks and actions.
PTAL
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_lint_check #11487 mongodb for mattyw mp273544
LINT OK: passed
Build: http://
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_unit_test #10679 mongodb for mattyw mp273544
UNIT OK: passed
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_amulet_test #7186 mongodb for mattyw mp273544
AMULET OK: passed
Build: http://
Ryan Beisner (1chb1n) wrote : | # |
Tip:
If you'd like to drastically reduce the diff, while achieving the same effect:
Revert back to 86: bzr uncommit && bzr revert
Then bzr mv hooks/charmhelpers instead of mv, and do your linking after that. This will keep the version control side of things happier.
Then commit and bzr push with --overwrite.
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_amulet_test #7188 mongodb for mattyw mp273544
AMULET OK: passed
Build: http://
- 87. By Matthew Williams
-
charmhelpers: Moved to common location
- 88. By Matthew Williams
-
charmhelpers: symlinks
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_lint_check #11489 mongodb for mattyw mp273544
LINT OK: passed
Build: http://
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_unit_test #10680 mongodb for mattyw mp273544
UNIT OK: passed
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_amulet_test #7190 mongodb for mattyw mp273544
AMULET FAIL: amulet-test failed
AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.
Full amulet test output: http://
Build: http://
Ryan Beisner (1chb1n) wrote : | # |
^ that amulet test failure was unrelated to the proposed changes, rerunning...
2015-10-07 17:05:59 INFO install W: Failed to fetch http://
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_amulet_test #7206 mongodb for mattyw mp273544
AMULET FAIL: amulet-test failed
AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.
Full amulet test output: http://
Build: http://
Domas Monkus (tasdomas) wrote : | # |
A few comments inline with the code.
- 89. By Matthew Williams
-
actions/backups.py: Fixed comments from domas' review
Matthew Williams (mattyw) wrote : | # |
> charm_amulet_test #7206 mongodb for mattyw mp273544
> AMULET FAIL: amulet-test failed
>
> AMULET Results (max last 2 lines):
> make: *** [functional_test] Error 1
> ERROR:root:Make target returned non-zero.
>
> Full amulet test output: http://
> Build: http://
The output of this amulet tests seems to show that when three mongodb units were started in a replica set we ended up in a situation where there were two primaries. This is a known problem with the existing charm right? Seems like the tests should be altered?
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_lint_check #11504 mongodb for mattyw mp273544
LINT OK: passed
Build: http://
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_unit_test #10696 mongodb for mattyw mp273544
UNIT OK: passed
uosci-testing-bot (uosci-testing-bot) wrote : | # |
charm_amulet_test #7208 mongodb for mattyw mp273544
AMULET OK: passed
Build: http://
Domas Monkus (tasdomas) wrote : | # |
looks good to me
Charles Butler (lazypower) wrote : | # |
+1 LGTM
Approved and merged. Thanks for the contribution Mattyw
Preview Diff
1 | === modified file 'Makefile' |
2 | --- Makefile 2014-12-11 06:29:02 +0000 |
3 | +++ Makefile 2015-10-08 08:12:47 +0000 |
4 | @@ -14,6 +14,7 @@ |
5 | # along with this program. If not, see <http://www.gnu.org/licenses/>. |
6 | |
7 | PYTHON := /usr/bin/env python |
8 | +ACTIONS := $(shell grep -slE '\#!(.*)python' actions/*) |
9 | |
10 | clean: |
11 | rm -f .coverage |
12 | @@ -27,11 +28,12 @@ |
13 | .venv/bin/pip install -I -r test_requirements.txt |
14 | |
15 | lint: .venv |
16 | - .venv/bin/flake8 --exclude hooks/charmhelpers hooks tests unit_tests |
17 | + .venv/bin/flake8 --exclude hooks/charmhelpers actions $(ACTIONS) hooks tests unit_tests |
18 | |
19 | test: .venv |
20 | @echo Starting unit tests... |
21 | .venv/bin/nosetests -s --nologcapture --with-coverage $(EXTRA) unit_tests/ |
22 | + .venv/bin/nosetests -s --nologcapture --with-coverage $(EXTRA) actions/ |
23 | |
24 | functional_test: |
25 | @echo Starting amulet tests... |
26 | |
27 | === modified file 'actions.yaml' |
28 | --- actions.yaml 2015-05-07 20:42:32 +0000 |
29 | +++ actions.yaml 2015-10-08 08:12:47 +0000 |
30 | @@ -47,3 +47,25 @@ |
31 | Only use syncDelay in conjunction with mmf set to true. |
32 | type: integer |
33 | default: 0 |
34 | +dump: |
35 | + description: Runs the mongodump command |
36 | + params: |
37 | + args: |
38 | + description: The arguments to pass to the mongodump command |
39 | + type: string |
40 | + default: "" |
41 | + workingDir: |
42 | + description: The directory to run the mongodump command from. |
43 | + type: string |
44 | + default: "/tmp/mongodump" |
45 | +restore: |
46 | + description: Runs the mongorestore command |
47 | + params: |
48 | + args: |
49 | + description: The arguments to pass to the mongorestore command |
50 | + type: string |
51 | + default: "" |
52 | + workingDir: |
53 | + description: The directory to run the mongorestore command from. |
54 | + type: string |
55 | + default: "/tmp/mongodump" |
56 | |
57 | === added file 'actions/backup.py' |
58 | --- actions/backup.py 1970-01-01 00:00:00 +0000 |
59 | +++ actions/backup.py 2015-10-08 08:12:47 +0000 |
60 | @@ -0,0 +1,56 @@ |
61 | +import subprocess |
62 | +import os |
63 | +try: |
64 | + from charmhelpers.core.hookenv import action_get, action_set, action_fail |
65 | +except ImportError: |
66 | + subprocess.check_call(['apt-get', 'install', '-y', 'python-pip']) |
67 | + subprocess.check_call(['pip', 'install', 'charmhelpers']) |
68 | + from charmhelpers.core.hookenv import action_get, action_set, action_fail |
69 | + |
70 | + |
71 | +def mkdir(dir): |
72 | + return os.makedirs(dir) |
73 | + |
74 | + |
75 | +def execute(command, current_working_directory): |
76 | + return subprocess.check_output( |
77 | + " ".join(command), |
78 | + cwd=current_working_directory, |
79 | + ) |
80 | + |
81 | + |
82 | +def dump(): |
83 | + args = str(action_get("args")).strip() |
84 | + dir = str(action_get("workingDir")) |
85 | + backup_command("mongodump", args, dir) |
86 | + |
87 | + |
88 | +def restore(): |
89 | + args = str(action_get("args")).strip() |
90 | + dir = str(action_get("workingDir")) |
91 | + backup_command("mongorestore", args, dir) |
92 | + |
93 | + |
94 | +def backup_command(cmd, args, dir): |
95 | + try: |
96 | + mkdir(dir) |
97 | + except OSError, e: |
98 | + pass # Ignoring, the directory already exists |
99 | + except Exception, e: |
100 | + action_set({"directory creation exception": e}) |
101 | + action_fail(e) |
102 | + return |
103 | + |
104 | + command = cmd + " " + args |
105 | + command = command.strip() |
106 | + action_set({"command": command, "working-dir": dir}) |
107 | + try: |
108 | + output = execute(command, dir) |
109 | + action_set({"output": output}) |
110 | + except subprocess.CalledProcessError, e: |
111 | + action_set({"error_code": e.returncode, |
112 | + "exception": e, "output": e.output}) |
113 | + action_fail(e) |
114 | + except Exception, e: |
115 | + action_set({"exception": e}) |
116 | + action_fail(e) |
117 | |
118 | === added file 'actions/backup_test.py' |
119 | --- actions/backup_test.py 1970-01-01 00:00:00 +0000 |
120 | +++ actions/backup_test.py 2015-10-08 08:12:47 +0000 |
121 | @@ -0,0 +1,52 @@ |
122 | +import unittest |
123 | +from mock import create_autospec, call |
124 | +import backup |
125 | + |
126 | + |
127 | +def mock_action_get(key): |
128 | + return {"workingDir": "this/dir", "args": "-arg1 -arg2"}[key] |
129 | + |
130 | + |
131 | +class TestBackups(unittest.TestCase): |
132 | + |
133 | + def test_dump(self): |
134 | + mkdir = create_autospec(backup.mkdir, return_value=True) |
135 | + execute = create_autospec(backup.execute, return_value="output") |
136 | + action_set = create_autospec(backup.action_set, return_value=None) |
137 | + action_fail = create_autospec(backup.action_fail, return_value=None) |
138 | + backup.mkdir = mkdir |
139 | + backup.execute = execute |
140 | + backup.action_set = action_set |
141 | + backup.action_fail = action_fail |
142 | + backup.action_get = mock_action_get |
143 | + |
144 | + backup.dump() |
145 | + |
146 | + mkdir.assert_called_once_with("this/dir") |
147 | + execute.assert_called_once_with("mongodump -arg1 -arg2", "this/dir") |
148 | + action_set.assert_has_calls([ |
149 | + call({"command": "mongodump -arg1 -arg2", |
150 | + "working-dir": "this/dir"}), |
151 | + call({"output": "output"})]) |
152 | + |
153 | + def test_restore(self): |
154 | + mkdir = create_autospec(backup.mkdir, return_value=True) |
155 | + execute = create_autospec(backup.execute, return_value="output") |
156 | + action_set = create_autospec(backup.action_set, return_value=None) |
157 | + action_fail = create_autospec(backup.action_fail, return_value=None) |
158 | + backup.mkdir = mkdir |
159 | + backup.execute = execute |
160 | + backup.action_set = action_set |
161 | + backup.action_fail = action_fail |
162 | + |
163 | + backup.restore() |
164 | + |
165 | + mkdir.assert_called_once_with("this/dir") |
166 | + execute.assert_called_once_with("mongorestore -arg1 -arg2", "this/dir") |
167 | + action_set.assert_has_calls([ |
168 | + call({"command": "mongodump -arg1 -arg2", |
169 | + "working-dir": "this/dir"}), |
170 | + call({"output": "output"})]) |
171 | + |
172 | +if __name__ == '__main__': |
173 | + unittest.main() |
174 | |
175 | === added symlink 'actions/charmhelpers' |
176 | === target is u'../charmhelpers' |
177 | === added file 'actions/dump' |
178 | --- actions/dump 1970-01-01 00:00:00 +0000 |
179 | +++ actions/dump 2015-10-08 08:12:47 +0000 |
180 | @@ -0,0 +1,4 @@ |
181 | +#!/usr/bin/env python |
182 | +import backup |
183 | + |
184 | +backup.dump() |
185 | |
186 | === added file 'actions/restore' |
187 | --- actions/restore 1970-01-01 00:00:00 +0000 |
188 | +++ actions/restore 2015-10-08 08:12:47 +0000 |
189 | @@ -0,0 +1,4 @@ |
190 | +#!/usr/bin/env python |
191 | +import backup |
192 | + |
193 | +backup.restore() |
194 | |
195 | === modified file 'charm-helpers-sync.yaml' |
196 | --- charm-helpers-sync.yaml 2015-02-25 00:24:56 +0000 |
197 | +++ charm-helpers-sync.yaml 2015-10-08 08:12:47 +0000 |
198 | @@ -1,5 +1,5 @@ |
199 | branch: lp:charm-helpers |
200 | -destination: hooks/charmhelpers |
201 | +destination: charmhelpers |
202 | include: |
203 | - core |
204 | - fetch |
205 | |
206 | === renamed directory 'hooks/charmhelpers' => 'charmhelpers' |
207 | === modified file 'charmhelpers/core/hookenv.py' |
208 | --- hooks/charmhelpers/core/hookenv.py 2015-08-19 13:58:52 +0000 |
209 | +++ charmhelpers/core/hookenv.py 2015-10-08 08:12:47 +0000 |
210 | @@ -623,6 +623,38 @@ |
211 | return unit_get('private-address') |
212 | |
213 | |
214 | +@cached |
215 | +def storage_get(attribute="", storage_id=""): |
216 | + """Get storage attributes""" |
217 | + _args = ['storage-get', '--format=json'] |
218 | + if storage_id: |
219 | + _args.extend(('-s', storage_id)) |
220 | + if attribute: |
221 | + _args.append(attribute) |
222 | + try: |
223 | + return json.loads(subprocess.check_output(_args).decode('UTF-8')) |
224 | + except ValueError: |
225 | + return None |
226 | + |
227 | + |
228 | +@cached |
229 | +def storage_list(storage_name=""): |
230 | + """List the storage IDs for the unit""" |
231 | + _args = ['storage-list', '--format=json'] |
232 | + if storage_name: |
233 | + _args.append(storage_name) |
234 | + try: |
235 | + return json.loads(subprocess.check_output(_args).decode('UTF-8')) |
236 | + except ValueError: |
237 | + return None |
238 | + except OSError as e: |
239 | + import errno |
240 | + if e.errno == errno.ENOENT: |
241 | + # storage-list does not exist |
242 | + return [] |
243 | + raise |
244 | + |
245 | + |
246 | class UnregisteredHookError(Exception): |
247 | """Raised when an undefined hook is called""" |
248 | pass |
249 | @@ -767,21 +799,23 @@ |
250 | |
251 | |
252 | def status_get(): |
253 | - """Retrieve the previously set juju workload state |
254 | - |
255 | - If the status-set command is not found then assume this is juju < 1.23 and |
256 | - return 'unknown' |
257 | + """Retrieve the previously set juju workload state and message |
258 | + |
259 | + If the status-get command is not found then assume this is juju < 1.23 and |
260 | + return 'unknown', "" |
261 | + |
262 | """ |
263 | - cmd = ['status-get'] |
264 | + cmd = ['status-get', "--format=json", "--include-data"] |
265 | try: |
266 | - raw_status = subprocess.check_output(cmd, universal_newlines=True) |
267 | - status = raw_status.rstrip() |
268 | - return status |
269 | + raw_status = subprocess.check_output(cmd) |
270 | except OSError as e: |
271 | if e.errno == errno.ENOENT: |
272 | - return 'unknown' |
273 | + return ('unknown', "") |
274 | else: |
275 | raise |
276 | + else: |
277 | + status = json.loads(raw_status.decode("UTF-8")) |
278 | + return (status["status"], status["message"]) |
279 | |
280 | |
281 | def translate_exc(from_exc, to_exc): |
282 | |
283 | === modified file 'charmhelpers/core/host.py' |
284 | --- hooks/charmhelpers/core/host.py 2015-08-19 13:58:52 +0000 |
285 | +++ charmhelpers/core/host.py 2015-10-08 08:12:47 +0000 |
286 | @@ -63,32 +63,48 @@ |
287 | return service_result |
288 | |
289 | |
290 | -def service_pause(service_name, init_dir=None): |
291 | +def service_pause(service_name, init_dir="/etc/init", initd_dir="/etc/init.d"): |
292 | """Pause a system service. |
293 | |
294 | Stop it, and prevent it from starting again at boot.""" |
295 | - if init_dir is None: |
296 | - init_dir = "/etc/init" |
297 | stopped = service_stop(service_name) |
298 | - # XXX: Support systemd too |
299 | - override_path = os.path.join( |
300 | - init_dir, '{}.override'.format(service_name)) |
301 | - with open(override_path, 'w') as fh: |
302 | - fh.write("manual\n") |
303 | + upstart_file = os.path.join(init_dir, "{}.conf".format(service_name)) |
304 | + sysv_file = os.path.join(initd_dir, service_name) |
305 | + if os.path.exists(upstart_file): |
306 | + override_path = os.path.join( |
307 | + init_dir, '{}.override'.format(service_name)) |
308 | + with open(override_path, 'w') as fh: |
309 | + fh.write("manual\n") |
310 | + elif os.path.exists(sysv_file): |
311 | + subprocess.check_call(["update-rc.d", service_name, "disable"]) |
312 | + else: |
313 | + # XXX: Support SystemD too |
314 | + raise ValueError( |
315 | + "Unable to detect {0} as either Upstart {1} or SysV {2}".format( |
316 | + service_name, upstart_file, sysv_file)) |
317 | return stopped |
318 | |
319 | |
320 | -def service_resume(service_name, init_dir=None): |
321 | +def service_resume(service_name, init_dir="/etc/init", |
322 | + initd_dir="/etc/init.d"): |
323 | """Resume a system service. |
324 | |
325 | Reenable starting again at boot. Start the service""" |
326 | - # XXX: Support systemd too |
327 | - if init_dir is None: |
328 | - init_dir = "/etc/init" |
329 | - override_path = os.path.join( |
330 | - init_dir, '{}.override'.format(service_name)) |
331 | - if os.path.exists(override_path): |
332 | - os.unlink(override_path) |
333 | + upstart_file = os.path.join(init_dir, "{}.conf".format(service_name)) |
334 | + sysv_file = os.path.join(initd_dir, service_name) |
335 | + if os.path.exists(upstart_file): |
336 | + override_path = os.path.join( |
337 | + init_dir, '{}.override'.format(service_name)) |
338 | + if os.path.exists(override_path): |
339 | + os.unlink(override_path) |
340 | + elif os.path.exists(sysv_file): |
341 | + subprocess.check_call(["update-rc.d", service_name, "enable"]) |
342 | + else: |
343 | + # XXX: Support SystemD too |
344 | + raise ValueError( |
345 | + "Unable to detect {0} as either Upstart {1} or SysV {2}".format( |
346 | + service_name, upstart_file, sysv_file)) |
347 | + |
348 | started = service_start(service_name) |
349 | return started |
350 | |
351 | |
352 | === modified file 'charmhelpers/core/hugepage.py' |
353 | --- hooks/charmhelpers/core/hugepage.py 2015-08-19 13:58:52 +0000 |
354 | +++ charmhelpers/core/hugepage.py 2015-10-08 08:12:47 +0000 |
355 | @@ -25,11 +25,13 @@ |
356 | fstab_mount, |
357 | mkdir, |
358 | ) |
359 | +from charmhelpers.core.strutils import bytes_from_string |
360 | +from subprocess import check_output |
361 | |
362 | |
363 | def hugepage_support(user, group='hugetlb', nr_hugepages=256, |
364 | max_map_count=65536, mnt_point='/run/hugepages/kvm', |
365 | - pagesize='2MB', mount=True): |
366 | + pagesize='2MB', mount=True, set_shmmax=False): |
367 | """Enable hugepages on system. |
368 | |
369 | Args: |
370 | @@ -49,6 +51,11 @@ |
371 | 'vm.max_map_count': max_map_count, |
372 | 'vm.hugetlb_shm_group': gid, |
373 | } |
374 | + if set_shmmax: |
375 | + shmmax_current = int(check_output(['sysctl', '-n', 'kernel.shmmax'])) |
376 | + shmmax_minsize = bytes_from_string(pagesize) * nr_hugepages |
377 | + if shmmax_minsize > shmmax_current: |
378 | + sysctl_settings['kernel.shmmax'] = shmmax_minsize |
379 | sysctl.create(yaml.dump(sysctl_settings), '/etc/sysctl.d/10-hugepage.conf') |
380 | mkdir(mnt_point, owner='root', group='root', perms=0o755, force=False) |
381 | lfstab = fstab.Fstab() |
382 | |
383 | === added file 'charmhelpers/core/kernel.py' |
384 | --- charmhelpers/core/kernel.py 1970-01-01 00:00:00 +0000 |
385 | +++ charmhelpers/core/kernel.py 2015-10-08 08:12:47 +0000 |
386 | @@ -0,0 +1,68 @@ |
387 | +#!/usr/bin/env python |
388 | +# -*- coding: utf-8 -*- |
389 | + |
390 | +# Copyright 2014-2015 Canonical Limited. |
391 | +# |
392 | +# This file is part of charm-helpers. |
393 | +# |
394 | +# charm-helpers is free software: you can redistribute it and/or modify |
395 | +# it under the terms of the GNU Lesser General Public License version 3 as |
396 | +# published by the Free Software Foundation. |
397 | +# |
398 | +# charm-helpers is distributed in the hope that it will be useful, |
399 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
400 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
401 | +# GNU Lesser General Public License for more details. |
402 | +# |
403 | +# You should have received a copy of the GNU Lesser General Public License |
404 | +# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. |
405 | + |
406 | +__author__ = "Jorge Niedbalski <jorge.niedbalski@canonical.com>" |
407 | + |
408 | +from charmhelpers.core.hookenv import ( |
409 | + log, |
410 | + INFO |
411 | +) |
412 | + |
413 | +from subprocess import check_call, check_output |
414 | +import re |
415 | + |
416 | + |
417 | +def modprobe(module, persist=True): |
418 | + """Load a kernel module and configure for auto-load on reboot.""" |
419 | + cmd = ['modprobe', module] |
420 | + |
421 | + log('Loading kernel module %s' % module, level=INFO) |
422 | + |
423 | + check_call(cmd) |
424 | + if persist: |
425 | + with open('/etc/modules', 'r+') as modules: |
426 | + if module not in modules.read(): |
427 | + modules.write(module) |
428 | + |
429 | + |
430 | +def rmmod(module, force=False): |
431 | + """Remove a module from the linux kernel""" |
432 | + cmd = ['rmmod'] |
433 | + if force: |
434 | + cmd.append('-f') |
435 | + cmd.append(module) |
436 | + log('Removing kernel module %s' % module, level=INFO) |
437 | + return check_call(cmd) |
438 | + |
439 | + |
440 | +def lsmod(): |
441 | + """Shows what kernel modules are currently loaded""" |
442 | + return check_output(['lsmod'], |
443 | + universal_newlines=True) |
444 | + |
445 | + |
446 | +def is_module_loaded(module): |
447 | + """Checks if a kernel module is already loaded""" |
448 | + matches = re.findall('^%s[ ]+' % module, lsmod(), re.M) |
449 | + return len(matches) > 0 |
450 | + |
451 | + |
452 | +def update_initramfs(version='all'): |
453 | + """Updates an initramfs image""" |
454 | + return check_call(["update-initramfs", "-k", version, "-u"]) |
455 | |
456 | === modified file 'charmhelpers/core/strutils.py' |
457 | --- hooks/charmhelpers/core/strutils.py 2015-08-19 13:58:52 +0000 |
458 | +++ charmhelpers/core/strutils.py 2015-10-08 08:12:47 +0000 |
459 | @@ -18,6 +18,7 @@ |
460 | # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>. |
461 | |
462 | import six |
463 | +import re |
464 | |
465 | |
466 | def bool_from_string(value): |
467 | @@ -40,3 +41,32 @@ |
468 | |
469 | msg = "Unable to interpret string value '%s' as boolean" % (value) |
470 | raise ValueError(msg) |
471 | + |
472 | + |
473 | +def bytes_from_string(value): |
474 | + """Interpret human readable string value as bytes. |
475 | + |
476 | + Returns int |
477 | + """ |
478 | + BYTE_POWER = { |
479 | + 'K': 1, |
480 | + 'KB': 1, |
481 | + 'M': 2, |
482 | + 'MB': 2, |
483 | + 'G': 3, |
484 | + 'GB': 3, |
485 | + 'T': 4, |
486 | + 'TB': 4, |
487 | + 'P': 5, |
488 | + 'PB': 5, |
489 | + } |
490 | + if isinstance(value, six.string_types): |
491 | + value = six.text_type(value) |
492 | + else: |
493 | + msg = "Unable to interpret non-string value '%s' as boolean" % (value) |
494 | + raise ValueError(msg) |
495 | + matches = re.match("([0-9]+)([a-zA-Z]+)", value) |
496 | + if not matches: |
497 | + msg = "Unable to interpret string value '%s' as bytes" % (value) |
498 | + raise ValueError(msg) |
499 | + return int(matches.group(1)) * (1024 ** BYTE_POWER[matches.group(2)]) |
500 | |
501 | === added symlink 'hooks/charmhelpers' |
502 | === target is u'../charmhelpers' |
charm_lint_check #11389 mongodb for mattyw mp273544
LINT OK: passed
Build: http:// 10.245. 162.77: 8080/job/ charm_lint_ check/11389/