Merge lp:~lihuiguo/landscape-charm/bug-1934816 into lp:~landscape/landscape-charm/trunk
- bug-1934816
- Merge into trunk
Status: | Merged |
---|---|
Approved by: | Simon Poirier |
Approved revision: | 409 |
Merged at revision: | 407 |
Proposed branch: | lp:~lihuiguo/landscape-charm/bug-1934816 |
Merge into: | lp:~landscape/landscape-charm/trunk |
Diff against target: |
3080 lines (+1978/-192) 29 files modified
charm-helpers.yaml (+1/-0) charmhelpers/__init__.py (+6/-4) charmhelpers/contrib/charmsupport/__init__.py (+13/-0) charmhelpers/contrib/charmsupport/nrpe.py (+522/-0) charmhelpers/contrib/hahelpers/apache.py (+5/-1) charmhelpers/contrib/hahelpers/cluster.py (+47/-2) charmhelpers/core/decorators.py (+38/-0) charmhelpers/core/hookenv.py (+184/-35) charmhelpers/core/host.py (+262/-60) charmhelpers/core/host_factory/ubuntu.py (+13/-5) charmhelpers/core/services/base.py (+7/-2) charmhelpers/core/strutils.py (+7/-4) charmhelpers/core/sysctl.py (+12/-2) charmhelpers/core/unitdata.py (+3/-3) charmhelpers/fetch/__init__.py (+7/-2) charmhelpers/fetch/python/packages.py (+6/-4) charmhelpers/fetch/snap.py (+3/-3) charmhelpers/fetch/ubuntu.py (+341/-59) charmhelpers/fetch/ubuntu_apt_pkg.py (+312/-0) charmhelpers/osplatform.py (+27/-3) config.yaml (+16/-0) hooks/nrpe-external-master-relation-changed (+9/-0) hooks/nrpe-external-master-relation-joined (+9/-0) lib/callbacks/nrpe.py (+51/-0) lib/callbacks/tests/test_nrpe.py (+36/-0) lib/services.py (+5/-1) lib/tests/stubs.py (+24/-0) lib/tests/test_services.py (+9/-2) metadata.yaml (+3/-0) |
To merge this branch: | bzr merge lp:~lihuiguo/landscape-charm/bug-1934816 |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
🤖 Landscape Builder | test results | Approve | |
Simon Poirier (community) | Approve | ||
James Troup (community) | Approve | ||
Review via email:
|
Commit message
Sync charm-helpers
Add relation: nrpe-external-
Add nrpe checks check_systemd for landscape services
Description of the change
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
🤖 Landscape Builder (landscape-builder) : | # |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Linda Guo (lihuiguo) wrote : | # |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
🤖 Landscape Builder (landscape-builder) wrote : | # |
Command: make ci-test
Result: Fail
Revno: 407
Branch: lp:~lihuiguo/landscape-charm/bug-1934816
Jenkins: https:/
- 408. By Linda Guo <email address hidden>
-
Add charmhelpers.
contrib. charmsupport dependency
to charm-helpers.yaml to get nrpe.py
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
🤖 Landscape Builder (landscape-builder) : | # |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
🤖 Landscape Builder (landscape-builder) wrote : | # |
Command: make ci-test
Result: Success
Revno: 408
Branch: lp:~lihuiguo/landscape-charm/bug-1934816
Jenkins: https:/
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
James Troup (elmo) wrote : | # |
I didn't review the charmhelpers changes, and my only comments are nitpick of docstrings (see inline comments). Other than those, this LGTM.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Simon Poirier (simpoir) wrote : | # |
+1 with inline comment
The test cases are a bit thin for my taste (only adding checks is covered, while the hooks handle add/remove/change)
- 409. By Linda Guo <email address hidden>
-
Fixed docstring
Added unit test to check remove nrpe
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
🤖 Landscape Builder (landscape-builder) : | # |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
🤖 Landscape Builder (landscape-builder) wrote : | # |
Command: make ci-test
Result: Success
Revno: 409
Branch: lp:~lihuiguo/landscape-charm/bug-1934816
Jenkins: https:/
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Linda Guo (lihuiguo) wrote : | # |
> +1 with inline comment
>
> The test cases are a bit thin for my taste (only adding checks is covered,
> while the hooks handle add/remove/change)
I have added more test cases to cover the nrpe check remove
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Simon Poirier (simpoir) wrote : | # |
Thanks for that extra test.
Preview Diff
1 | === modified file 'charm-helpers.yaml' |
2 | --- charm-helpers.yaml 2017-03-04 02:41:39 +0000 |
3 | +++ charm-helpers.yaml 2021-11-10 05:36:20 +0000 |
4 | @@ -6,3 +6,4 @@ |
5 | - fetch |
6 | - osplatform |
7 | - contrib.hahelpers |
8 | + - contrib.charmsupport.nrpe |
9 | |
10 | === modified file 'charmhelpers/__init__.py' |
11 | --- charmhelpers/__init__.py 2019-05-24 12:41:48 +0000 |
12 | +++ charmhelpers/__init__.py 2021-11-10 05:36:20 +0000 |
13 | @@ -49,7 +49,8 @@ |
14 | |
15 | def deprecate(warning, date=None, log=None): |
16 | """Add a deprecation warning the first time the function is used. |
17 | - The date, which is a string in semi-ISO8660 format indicate the year-month |
18 | + |
19 | + The date which is a string in semi-ISO8660 format indicates the year-month |
20 | that the function is officially going to be removed. |
21 | |
22 | usage: |
23 | @@ -62,10 +63,11 @@ |
24 | The reason for passing the logging function (log) is so that hookenv.log |
25 | can be used for a charm if needed. |
26 | |
27 | - :param warning: String to indicat where it has moved ot. |
28 | - :param date: optional sting, in YYYY-MM format to indicate when the |
29 | + :param warning: String to indicate what is to be used instead. |
30 | + :param date: Optional string in YYYY-MM format to indicate when the |
31 | function will definitely (probably) be removed. |
32 | - :param log: The log function to call to log. If not, logs to stdout |
33 | + :param log: The log function to call in order to log. If None, logs to |
34 | + stdout |
35 | """ |
36 | def wrap(f): |
37 | |
38 | |
39 | === added directory 'charmhelpers/contrib/charmsupport' |
40 | === added file 'charmhelpers/contrib/charmsupport/__init__.py' |
41 | --- charmhelpers/contrib/charmsupport/__init__.py 1970-01-01 00:00:00 +0000 |
42 | +++ charmhelpers/contrib/charmsupport/__init__.py 2021-11-10 05:36:20 +0000 |
43 | @@ -0,0 +1,13 @@ |
44 | +# Copyright 2014-2015 Canonical Limited. |
45 | +# |
46 | +# Licensed under the Apache License, Version 2.0 (the "License"); |
47 | +# you may not use this file except in compliance with the License. |
48 | +# You may obtain a copy of the License at |
49 | +# |
50 | +# http://www.apache.org/licenses/LICENSE-2.0 |
51 | +# |
52 | +# Unless required by applicable law or agreed to in writing, software |
53 | +# distributed under the License is distributed on an "AS IS" BASIS, |
54 | +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
55 | +# See the License for the specific language governing permissions and |
56 | +# limitations under the License. |
57 | |
58 | === added file 'charmhelpers/contrib/charmsupport/nrpe.py' |
59 | --- charmhelpers/contrib/charmsupport/nrpe.py 1970-01-01 00:00:00 +0000 |
60 | +++ charmhelpers/contrib/charmsupport/nrpe.py 2021-11-10 05:36:20 +0000 |
61 | @@ -0,0 +1,522 @@ |
62 | +# Copyright 2012-2021 Canonical Limited. |
63 | +# |
64 | +# Licensed under the Apache License, Version 2.0 (the "License"); |
65 | +# you may not use this file except in compliance with the License. |
66 | +# You may obtain a copy of the License at |
67 | +# |
68 | +# http://www.apache.org/licenses/LICENSE-2.0 |
69 | +# |
70 | +# Unless required by applicable law or agreed to in writing, software |
71 | +# distributed under the License is distributed on an "AS IS" BASIS, |
72 | +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
73 | +# See the License for the specific language governing permissions and |
74 | +# limitations under the License. |
75 | + |
76 | +"""Compatibility with the nrpe-external-master charm""" |
77 | +# |
78 | +# Authors: |
79 | +# Matthew Wedgwood <matthew.wedgwood@canonical.com> |
80 | + |
81 | +import glob |
82 | +import grp |
83 | +import os |
84 | +import pwd |
85 | +import re |
86 | +import shlex |
87 | +import shutil |
88 | +import subprocess |
89 | +import yaml |
90 | + |
91 | +from charmhelpers.core.hookenv import ( |
92 | + config, |
93 | + hook_name, |
94 | + local_unit, |
95 | + log, |
96 | + relation_get, |
97 | + relation_ids, |
98 | + relation_set, |
99 | + relations_of_type, |
100 | +) |
101 | + |
102 | +from charmhelpers.core.host import service |
103 | +from charmhelpers.core import host |
104 | + |
105 | +# This module adds compatibility with the nrpe-external-master and plain nrpe |
106 | +# subordinate charms. To use it in your charm: |
107 | +# |
108 | +# 1. Update metadata.yaml |
109 | +# |
110 | +# provides: |
111 | +# (...) |
112 | +# nrpe-external-master: |
113 | +# interface: nrpe-external-master |
114 | +# scope: container |
115 | +# |
116 | +# and/or |
117 | +# |
118 | +# provides: |
119 | +# (...) |
120 | +# local-monitors: |
121 | +# interface: local-monitors |
122 | +# scope: container |
123 | + |
124 | +# |
125 | +# 2. Add the following to config.yaml |
126 | +# |
127 | +# nagios_context: |
128 | +# default: "juju" |
129 | +# type: string |
130 | +# description: | |
131 | +# Used by the nrpe subordinate charms. |
132 | +# A string that will be prepended to instance name to set the host name |
133 | +# in nagios. So for instance the hostname would be something like: |
134 | +# juju-myservice-0 |
135 | +# If you're running multiple environments with the same services in them |
136 | +# this allows you to differentiate between them. |
137 | +# nagios_servicegroups: |
138 | +# default: "" |
139 | +# type: string |
140 | +# description: | |
141 | +# A comma-separated list of nagios servicegroups. |
142 | +# If left empty, the nagios_context will be used as the servicegroup |
143 | +# |
144 | +# 3. Add custom checks (Nagios plugins) to files/nrpe-external-master |
145 | +# |
146 | +# 4. Update your hooks.py with something like this: |
147 | +# |
148 | +# from charmsupport.nrpe import NRPE |
149 | +# (...) |
150 | +# def update_nrpe_config(): |
151 | +# nrpe_compat = NRPE() |
152 | +# nrpe_compat.add_check( |
153 | +# shortname = "myservice", |
154 | +# description = "Check MyService", |
155 | +# check_cmd = "check_http -w 2 -c 10 http://localhost" |
156 | +# ) |
157 | +# nrpe_compat.add_check( |
158 | +# "myservice_other", |
159 | +# "Check for widget failures", |
160 | +# check_cmd = "/srv/myapp/scripts/widget_check" |
161 | +# ) |
162 | +# nrpe_compat.write() |
163 | +# |
164 | +# def config_changed(): |
165 | +# (...) |
166 | +# update_nrpe_config() |
167 | +# |
168 | +# def nrpe_external_master_relation_changed(): |
169 | +# update_nrpe_config() |
170 | +# |
171 | +# def local_monitors_relation_changed(): |
172 | +# update_nrpe_config() |
173 | +# |
174 | +# 4.a If your charm is a subordinate charm set primary=False |
175 | +# |
176 | +# from charmsupport.nrpe import NRPE |
177 | +# (...) |
178 | +# def update_nrpe_config(): |
179 | +# nrpe_compat = NRPE(primary=False) |
180 | +# |
181 | +# 5. ln -s hooks.py nrpe-external-master-relation-changed |
182 | +# ln -s hooks.py local-monitors-relation-changed |
183 | + |
184 | + |
185 | +class CheckException(Exception): |
186 | + pass |
187 | + |
188 | + |
189 | +class Check(object): |
190 | + shortname_re = '[A-Za-z0-9-_.@]+$' |
191 | + service_template = (""" |
192 | +#--------------------------------------------------- |
193 | +# This file is Juju managed |
194 | +#--------------------------------------------------- |
195 | +define service {{ |
196 | + use active-service |
197 | + host_name {nagios_hostname} |
198 | + service_description {nagios_hostname}[{shortname}] """ |
199 | + """{description} |
200 | + check_command check_nrpe!{command} |
201 | + servicegroups {nagios_servicegroup} |
202 | +{service_config_overrides} |
203 | +}} |
204 | +""") |
205 | + |
206 | + def __init__(self, shortname, description, check_cmd, max_check_attempts=None): |
207 | + super(Check, self).__init__() |
208 | + # XXX: could be better to calculate this from the service name |
209 | + if not re.match(self.shortname_re, shortname): |
210 | + raise CheckException("shortname must match {}".format( |
211 | + Check.shortname_re)) |
212 | + self.shortname = shortname |
213 | + self.command = "check_{}".format(shortname) |
214 | + # Note: a set of invalid characters is defined by the |
215 | + # Nagios server config |
216 | + # The default is: illegal_object_name_chars=`~!$%^&*"|'<>?,()= |
217 | + self.description = description |
218 | + self.check_cmd = self._locate_cmd(check_cmd) |
219 | + self.max_check_attempts = max_check_attempts |
220 | + |
221 | + def _get_check_filename(self): |
222 | + return os.path.join(NRPE.nrpe_confdir, '{}.cfg'.format(self.command)) |
223 | + |
224 | + def _get_service_filename(self, hostname): |
225 | + return os.path.join(NRPE.nagios_exportdir, |
226 | + 'service__{}_{}.cfg'.format(hostname, self.command)) |
227 | + |
228 | + def _locate_cmd(self, check_cmd): |
229 | + search_path = ( |
230 | + '/usr/lib/nagios/plugins', |
231 | + '/usr/local/lib/nagios/plugins', |
232 | + ) |
233 | + parts = shlex.split(check_cmd) |
234 | + for path in search_path: |
235 | + if os.path.exists(os.path.join(path, parts[0])): |
236 | + command = os.path.join(path, parts[0]) |
237 | + if len(parts) > 1: |
238 | + command += " " + " ".join(parts[1:]) |
239 | + return command |
240 | + log('Check command not found: {}'.format(parts[0])) |
241 | + return '' |
242 | + |
243 | + def _remove_service_files(self): |
244 | + if not os.path.exists(NRPE.nagios_exportdir): |
245 | + return |
246 | + for f in os.listdir(NRPE.nagios_exportdir): |
247 | + if f.endswith('_{}.cfg'.format(self.command)): |
248 | + os.remove(os.path.join(NRPE.nagios_exportdir, f)) |
249 | + |
250 | + def remove(self, hostname): |
251 | + nrpe_check_file = self._get_check_filename() |
252 | + if os.path.exists(nrpe_check_file): |
253 | + os.remove(nrpe_check_file) |
254 | + self._remove_service_files() |
255 | + |
256 | + def write(self, nagios_context, hostname, nagios_servicegroups): |
257 | + nrpe_check_file = self._get_check_filename() |
258 | + with open(nrpe_check_file, 'w') as nrpe_check_config: |
259 | + nrpe_check_config.write("# check {}\n".format(self.shortname)) |
260 | + if nagios_servicegroups: |
261 | + nrpe_check_config.write( |
262 | + "# The following header was added automatically by juju\n") |
263 | + nrpe_check_config.write( |
264 | + "# Modifying it will affect nagios monitoring and alerting\n") |
265 | + nrpe_check_config.write( |
266 | + "# servicegroups: {}\n".format(nagios_servicegroups)) |
267 | + nrpe_check_config.write("command[{}]={}\n".format( |
268 | + self.command, self.check_cmd)) |
269 | + |
270 | + if not os.path.exists(NRPE.nagios_exportdir): |
271 | + log('Not writing service config as {} is not accessible'.format( |
272 | + NRPE.nagios_exportdir)) |
273 | + else: |
274 | + self.write_service_config(nagios_context, hostname, |
275 | + nagios_servicegroups) |
276 | + |
277 | + def write_service_config(self, nagios_context, hostname, |
278 | + nagios_servicegroups): |
279 | + self._remove_service_files() |
280 | + |
281 | + if self.max_check_attempts: |
282 | + service_config_overrides = ' max_check_attempts {}'.format( |
283 | + self.max_check_attempts |
284 | + ) # Note indentation is here rather than in the template to avoid trailing spaces |
285 | + else: |
286 | + service_config_overrides = '' # empty string to avoid printing 'None' |
287 | + templ_vars = { |
288 | + 'nagios_hostname': hostname, |
289 | + 'nagios_servicegroup': nagios_servicegroups, |
290 | + 'description': self.description, |
291 | + 'shortname': self.shortname, |
292 | + 'command': self.command, |
293 | + 'service_config_overrides': service_config_overrides, |
294 | + } |
295 | + nrpe_service_text = Check.service_template.format(**templ_vars) |
296 | + nrpe_service_file = self._get_service_filename(hostname) |
297 | + with open(nrpe_service_file, 'w') as nrpe_service_config: |
298 | + nrpe_service_config.write(str(nrpe_service_text)) |
299 | + |
300 | + def run(self): |
301 | + subprocess.call(self.check_cmd) |
302 | + |
303 | + |
304 | +class NRPE(object): |
305 | + nagios_logdir = '/var/log/nagios' |
306 | + nagios_exportdir = '/var/lib/nagios/export' |
307 | + nrpe_confdir = '/etc/nagios/nrpe.d' |
308 | + homedir = '/var/lib/nagios' # home dir provided by nagios-nrpe-server |
309 | + |
310 | + def __init__(self, hostname=None, primary=True): |
311 | + super(NRPE, self).__init__() |
312 | + self.config = config() |
313 | + self.primary = primary |
314 | + self.nagios_context = self.config['nagios_context'] |
315 | + if 'nagios_servicegroups' in self.config and self.config['nagios_servicegroups']: |
316 | + self.nagios_servicegroups = self.config['nagios_servicegroups'] |
317 | + else: |
318 | + self.nagios_servicegroups = self.nagios_context |
319 | + self.unit_name = local_unit().replace('/', '-') |
320 | + if hostname: |
321 | + self.hostname = hostname |
322 | + else: |
323 | + nagios_hostname = get_nagios_hostname() |
324 | + if nagios_hostname: |
325 | + self.hostname = nagios_hostname |
326 | + else: |
327 | + self.hostname = "{}-{}".format(self.nagios_context, self.unit_name) |
328 | + self.checks = [] |
329 | + # Iff in an nrpe-external-master relation hook, set primary status |
330 | + relation = relation_ids('nrpe-external-master') |
331 | + if relation: |
332 | + log("Setting charm primary status {}".format(primary)) |
333 | + for rid in relation: |
334 | + relation_set(relation_id=rid, relation_settings={'primary': self.primary}) |
335 | + self.remove_check_queue = set() |
336 | + |
337 | + @classmethod |
338 | + def does_nrpe_conf_dir_exist(cls): |
339 | + """Return True if th nrpe_confdif directory exists.""" |
340 | + return os.path.isdir(cls.nrpe_confdir) |
341 | + |
342 | + def add_check(self, *args, **kwargs): |
343 | + shortname = None |
344 | + if kwargs.get('shortname') is None: |
345 | + if len(args) > 0: |
346 | + shortname = args[0] |
347 | + else: |
348 | + shortname = kwargs['shortname'] |
349 | + |
350 | + self.checks.append(Check(*args, **kwargs)) |
351 | + try: |
352 | + self.remove_check_queue.remove(shortname) |
353 | + except KeyError: |
354 | + pass |
355 | + |
356 | + def remove_check(self, *args, **kwargs): |
357 | + if kwargs.get('shortname') is None: |
358 | + raise ValueError('shortname of check must be specified') |
359 | + |
360 | + # Use sensible defaults if they're not specified - these are not |
361 | + # actually used during removal, but they're required for constructing |
362 | + # the Check object; check_disk is chosen because it's part of the |
363 | + # nagios-plugins-basic package. |
364 | + if kwargs.get('check_cmd') is None: |
365 | + kwargs['check_cmd'] = 'check_disk' |
366 | + if kwargs.get('description') is None: |
367 | + kwargs['description'] = '' |
368 | + |
369 | + check = Check(*args, **kwargs) |
370 | + check.remove(self.hostname) |
371 | + self.remove_check_queue.add(kwargs['shortname']) |
372 | + |
373 | + def write(self): |
374 | + try: |
375 | + nagios_uid = pwd.getpwnam('nagios').pw_uid |
376 | + nagios_gid = grp.getgrnam('nagios').gr_gid |
377 | + except Exception: |
378 | + log("Nagios user not set up, nrpe checks not updated") |
379 | + return |
380 | + |
381 | + if not os.path.exists(NRPE.nagios_logdir): |
382 | + os.mkdir(NRPE.nagios_logdir) |
383 | + os.chown(NRPE.nagios_logdir, nagios_uid, nagios_gid) |
384 | + |
385 | + nrpe_monitors = {} |
386 | + monitors = {"monitors": {"remote": {"nrpe": nrpe_monitors}}} |
387 | + |
388 | + # check that the charm can write to the conf dir. If not, then nagios |
389 | + # probably isn't installed, and we can defer. |
390 | + if not self.does_nrpe_conf_dir_exist(): |
391 | + return |
392 | + |
393 | + for nrpecheck in self.checks: |
394 | + nrpecheck.write(self.nagios_context, self.hostname, |
395 | + self.nagios_servicegroups) |
396 | + nrpe_monitors[nrpecheck.shortname] = { |
397 | + "command": nrpecheck.command, |
398 | + } |
399 | + # If we were passed max_check_attempts, add that to the relation data |
400 | + if nrpecheck.max_check_attempts is not None: |
401 | + nrpe_monitors[nrpecheck.shortname]['max_check_attempts'] = nrpecheck.max_check_attempts |
402 | + |
403 | + # update-status hooks are configured to firing every 5 minutes by |
404 | + # default. When nagios-nrpe-server is restarted, the nagios server |
405 | + # reports checks failing causing unnecessary alerts. Let's not restart |
406 | + # on update-status hooks. |
407 | + if not hook_name() == 'update-status': |
408 | + service('restart', 'nagios-nrpe-server') |
409 | + |
410 | + monitor_ids = relation_ids("local-monitors") + \ |
411 | + relation_ids("nrpe-external-master") |
412 | + for rid in monitor_ids: |
413 | + reldata = relation_get(unit=local_unit(), rid=rid) |
414 | + if 'monitors' in reldata: |
415 | + # update the existing set of monitors with the new data |
416 | + old_monitors = yaml.safe_load(reldata['monitors']) |
417 | + old_nrpe_monitors = old_monitors['monitors']['remote']['nrpe'] |
418 | + # remove keys that are in the remove_check_queue |
419 | + old_nrpe_monitors = {k: v for k, v in old_nrpe_monitors.items() |
420 | + if k not in self.remove_check_queue} |
421 | + # update/add nrpe_monitors |
422 | + old_nrpe_monitors.update(nrpe_monitors) |
423 | + old_monitors['monitors']['remote']['nrpe'] = old_nrpe_monitors |
424 | + # write back to the relation |
425 | + relation_set(relation_id=rid, monitors=yaml.dump(old_monitors)) |
426 | + else: |
427 | + # write a brand new set of monitors, as no existing ones. |
428 | + relation_set(relation_id=rid, monitors=yaml.dump(monitors)) |
429 | + |
430 | + self.remove_check_queue.clear() |
431 | + |
432 | + |
433 | +def get_nagios_hostcontext(relation_name='nrpe-external-master'): |
434 | + """ |
435 | + Query relation with nrpe subordinate, return the nagios_host_context |
436 | + |
437 | + :param str relation_name: Name of relation nrpe sub joined to |
438 | + """ |
439 | + for rel in relations_of_type(relation_name): |
440 | + if 'nagios_host_context' in rel: |
441 | + return rel['nagios_host_context'] |
442 | + |
443 | + |
444 | +def get_nagios_hostname(relation_name='nrpe-external-master'): |
445 | + """ |
446 | + Query relation with nrpe subordinate, return the nagios_hostname |
447 | + |
448 | + :param str relation_name: Name of relation nrpe sub joined to |
449 | + """ |
450 | + for rel in relations_of_type(relation_name): |
451 | + if 'nagios_hostname' in rel: |
452 | + return rel['nagios_hostname'] |
453 | + |
454 | + |
455 | +def get_nagios_unit_name(relation_name='nrpe-external-master'): |
456 | + """ |
457 | + Return the nagios unit name prepended with host_context if needed |
458 | + |
459 | + :param str relation_name: Name of relation nrpe sub joined to |
460 | + """ |
461 | + host_context = get_nagios_hostcontext(relation_name) |
462 | + if host_context: |
463 | + unit = "%s:%s" % (host_context, local_unit()) |
464 | + else: |
465 | + unit = local_unit() |
466 | + return unit |
467 | + |
468 | + |
469 | +def add_init_service_checks(nrpe, services, unit_name, immediate_check=True): |
470 | + """ |
471 | + Add checks for each service in list |
472 | + |
473 | + :param NRPE nrpe: NRPE object to add check to |
474 | + :param list services: List of services to check |
475 | + :param str unit_name: Unit name to use in check description |
476 | + :param bool immediate_check: For sysv init, run the service check immediately |
477 | + """ |
478 | + for svc in services: |
479 | + # Don't add a check for these services from neutron-gateway |
480 | + if svc in ['ext-port', 'os-charm-phy-nic-mtu']: |
481 | + next |
482 | + |
483 | + upstart_init = '/etc/init/%s.conf' % svc |
484 | + sysv_init = '/etc/init.d/%s' % svc |
485 | + |
486 | + if host.init_is_systemd(service_name=svc): |
487 | + nrpe.add_check( |
488 | + shortname=svc, |
489 | + description='process check {%s}' % unit_name, |
490 | + check_cmd='check_systemd.py %s' % svc |
491 | + ) |
492 | + elif os.path.exists(upstart_init): |
493 | + nrpe.add_check( |
494 | + shortname=svc, |
495 | + description='process check {%s}' % unit_name, |
496 | + check_cmd='check_upstart_job %s' % svc |
497 | + ) |
498 | + elif os.path.exists(sysv_init): |
499 | + cronpath = '/etc/cron.d/nagios-service-check-%s' % svc |
500 | + checkpath = '%s/service-check-%s.txt' % (nrpe.homedir, svc) |
501 | + croncmd = ( |
502 | + '/usr/local/lib/nagios/plugins/check_exit_status.pl ' |
503 | + '-e -s /etc/init.d/%s status' % svc |
504 | + ) |
505 | + cron_file = '*/5 * * * * root %s > %s\n' % (croncmd, checkpath) |
506 | + f = open(cronpath, 'w') |
507 | + f.write(cron_file) |
508 | + f.close() |
509 | + nrpe.add_check( |
510 | + shortname=svc, |
511 | + description='service check {%s}' % unit_name, |
512 | + check_cmd='check_status_file.py -f %s' % checkpath, |
513 | + ) |
514 | + # if /var/lib/nagios doesn't exist open(checkpath, 'w') will fail |
515 | + # (LP: #1670223). |
516 | + if immediate_check and os.path.isdir(nrpe.homedir): |
517 | + f = open(checkpath, 'w') |
518 | + subprocess.call( |
519 | + croncmd.split(), |
520 | + stdout=f, |
521 | + stderr=subprocess.STDOUT |
522 | + ) |
523 | + f.close() |
524 | + os.chmod(checkpath, 0o644) |
525 | + |
526 | + |
527 | +def copy_nrpe_checks(nrpe_files_dir=None): |
528 | + """ |
529 | + Copy the nrpe checks into place |
530 | + |
531 | + """ |
532 | + NAGIOS_PLUGINS = '/usr/local/lib/nagios/plugins' |
533 | + if nrpe_files_dir is None: |
534 | + # determine if "charmhelpers" is in CHARMDIR or CHARMDIR/hooks |
535 | + for segment in ['.', 'hooks']: |
536 | + nrpe_files_dir = os.path.abspath(os.path.join( |
537 | + os.getenv('CHARM_DIR'), |
538 | + segment, |
539 | + 'charmhelpers', |
540 | + 'contrib', |
541 | + 'openstack', |
542 | + 'files')) |
543 | + if os.path.isdir(nrpe_files_dir): |
544 | + break |
545 | + else: |
546 | + raise RuntimeError("Couldn't find charmhelpers directory") |
547 | + if not os.path.exists(NAGIOS_PLUGINS): |
548 | + os.makedirs(NAGIOS_PLUGINS) |
549 | + for fname in glob.glob(os.path.join(nrpe_files_dir, "check_*")): |
550 | + if os.path.isfile(fname): |
551 | + shutil.copy2(fname, |
552 | + os.path.join(NAGIOS_PLUGINS, os.path.basename(fname))) |
553 | + |
554 | + |
555 | +def add_haproxy_checks(nrpe, unit_name): |
556 | + """ |
557 | + Add checks for each service in list |
558 | + |
559 | + :param NRPE nrpe: NRPE object to add check to |
560 | + :param str unit_name: Unit name to use in check description |
561 | + """ |
562 | + nrpe.add_check( |
563 | + shortname='haproxy_servers', |
564 | + description='Check HAProxy {%s}' % unit_name, |
565 | + check_cmd='check_haproxy.sh') |
566 | + nrpe.add_check( |
567 | + shortname='haproxy_queue', |
568 | + description='Check HAProxy queue depth {%s}' % unit_name, |
569 | + check_cmd='check_haproxy_queue_depth.sh') |
570 | + |
571 | + |
572 | +def remove_deprecated_check(nrpe, deprecated_services): |
573 | + """ |
574 | + Remove checks for deprecated services in list |
575 | + |
576 | + :param nrpe: NRPE object to remove check from |
577 | + :type nrpe: NRPE |
578 | + :param deprecated_services: List of deprecated services that are removed |
579 | + :type deprecated_services: list |
580 | + """ |
581 | + for dep_svc in deprecated_services: |
582 | + log('Deprecated service: {}'.format(dep_svc)) |
583 | + nrpe.remove_check(shortname=dep_svc) |
584 | |
585 | === modified file 'charmhelpers/contrib/hahelpers/apache.py' |
586 | --- charmhelpers/contrib/hahelpers/apache.py 2019-05-24 12:41:48 +0000 |
587 | +++ charmhelpers/contrib/hahelpers/apache.py 2021-11-10 05:36:20 +0000 |
588 | @@ -34,6 +34,10 @@ |
589 | INFO, |
590 | ) |
591 | |
592 | +# This file contains the CA cert from the charms ssl_ca configuration |
593 | +# option, in future the file name should be updated reflect that. |
594 | +CONFIG_CA_CERT_FILE = 'keystone_juju_ca_cert' |
595 | + |
596 | |
597 | def get_cert(cn=None): |
598 | # TODO: deal with multiple https endpoints via charm config |
599 | @@ -83,4 +87,4 @@ |
600 | |
601 | |
602 | def install_ca_cert(ca_cert): |
603 | - host.install_ca_cert(ca_cert, 'keystone_juju_ca_cert') |
604 | + host.install_ca_cert(ca_cert, CONFIG_CA_CERT_FILE) |
605 | |
606 | === modified file 'charmhelpers/contrib/hahelpers/cluster.py' |
607 | --- charmhelpers/contrib/hahelpers/cluster.py 2019-05-24 12:41:48 +0000 |
608 | +++ charmhelpers/contrib/hahelpers/cluster.py 2021-11-10 05:36:20 +0000 |
609 | @@ -1,4 +1,4 @@ |
610 | -# Copyright 2014-2015 Canonical Limited. |
611 | +# Copyright 2014-2021 Canonical Limited. |
612 | # |
613 | # Licensed under the Apache License, Version 2.0 (the "License"); |
614 | # you may not use this file except in compliance with the License. |
615 | @@ -25,6 +25,7 @@ |
616 | clustering-related helpers. |
617 | """ |
618 | |
619 | +import functools |
620 | import subprocess |
621 | import os |
622 | import time |
623 | @@ -85,7 +86,7 @@ |
624 | 2. If the charm is part of a corosync cluster, call corosync to |
625 | determine leadership. |
626 | 3. If the charm is not part of a corosync cluster, the leader is |
627 | - determined as being "the alive unit with the lowest unit numer". In |
628 | + determined as being "the alive unit with the lowest unit number". In |
629 | other words, the oldest surviving unit. |
630 | """ |
631 | try: |
632 | @@ -281,6 +282,10 @@ |
633 | return public_port - (i * 10) |
634 | |
635 | |
636 | +determine_apache_port_single = functools.partial( |
637 | + determine_apache_port, singlenode_mode=True) |
638 | + |
639 | + |
640 | def get_hacluster_config(exclude_keys=None): |
641 | ''' |
642 | Obtains all relevant configuration from charm configuration required |
643 | @@ -404,3 +409,43 @@ |
644 | log(msg, DEBUG) |
645 | status_set('maintenance', msg) |
646 | time.sleep(calculated_wait) |
647 | + |
648 | + |
649 | +def get_managed_services_and_ports(services, external_ports, |
650 | + external_services=None, |
651 | + port_conv_f=determine_apache_port_single): |
652 | + """Get the services and ports managed by this charm. |
653 | + |
654 | + Return only the services and corresponding ports that are managed by this |
655 | + charm. This excludes haproxy when there is a relation with hacluster. This |
656 | + is because this charm passes responsibility for stopping and starting |
657 | + haproxy to hacluster. |
658 | + |
659 | + Similarly, if a relation with hacluster exists then the ports returned by |
660 | + this method correspond to those managed by the apache server rather than |
661 | + haproxy. |
662 | + |
663 | + :param services: List of services. |
664 | + :type services: List[str] |
665 | + :param external_ports: List of ports managed by external services. |
666 | + :type external_ports: List[int] |
667 | + :param external_services: List of services to be removed if ha relation is |
668 | + present. |
669 | + :type external_services: List[str] |
670 | + :param port_conv_f: Function to apply to ports to calculate the ports |
671 | + managed by services controlled by this charm. |
672 | + :type port_convert_func: f() |
673 | + :returns: A tuple containing a list of services first followed by a list of |
674 | + ports. |
675 | + :rtype: Tuple[List[str], List[int]] |
676 | + """ |
677 | + if external_services is None: |
678 | + external_services = ['haproxy'] |
679 | + if relation_ids('ha'): |
680 | + for svc in external_services: |
681 | + try: |
682 | + services.remove(svc) |
683 | + except ValueError: |
684 | + pass |
685 | + external_ports = [port_conv_f(p) for p in external_ports] |
686 | + return services, external_ports |
687 | |
688 | === modified file 'charmhelpers/core/decorators.py' |
689 | --- charmhelpers/core/decorators.py 2017-03-03 21:03:14 +0000 |
690 | +++ charmhelpers/core/decorators.py 2021-11-10 05:36:20 +0000 |
691 | @@ -53,3 +53,41 @@ |
692 | return _retry_on_exception_inner_2 |
693 | |
694 | return _retry_on_exception_inner_1 |
695 | + |
696 | + |
697 | +def retry_on_predicate(num_retries, predicate_fun, base_delay=0): |
698 | + """Retry based on return value |
699 | + |
700 | + The return value of the decorated function is passed to the given predicate_fun. If the |
701 | + result of the predicate is False, retry the decorated function up to num_retries times |
702 | + |
703 | + An exponential backoff up to base_delay^num_retries seconds can be introduced by setting |
704 | + base_delay to a nonzero value. The default is to run with a zero (i.e. no) delay |
705 | + |
706 | + :param num_retries: Max. number of retries to perform |
707 | + :type num_retries: int |
708 | + :param predicate_fun: Predicate function to determine if a retry is necessary |
709 | + :type predicate_fun: callable |
710 | + :param base_delay: Starting value in seconds for exponential delay, defaults to 0 (no delay) |
711 | + :type base_delay: float |
712 | + """ |
713 | + def _retry_on_pred_inner_1(f): |
714 | + def _retry_on_pred_inner_2(*args, **kwargs): |
715 | + retries = num_retries |
716 | + multiplier = 1 |
717 | + delay = base_delay |
718 | + while True: |
719 | + result = f(*args, **kwargs) |
720 | + if predicate_fun(result) or retries <= 0: |
721 | + return result |
722 | + delay *= multiplier |
723 | + multiplier += 1 |
724 | + log("Result {}, retrying '{}' {} more times (delay={})".format( |
725 | + result, f.__name__, retries, delay), level=INFO) |
726 | + retries -= 1 |
727 | + if delay: |
728 | + time.sleep(delay) |
729 | + |
730 | + return _retry_on_pred_inner_2 |
731 | + |
732 | + return _retry_on_pred_inner_1 |
733 | |
734 | === modified file 'charmhelpers/core/hookenv.py' |
735 | --- charmhelpers/core/hookenv.py 2019-05-24 12:41:48 +0000 |
736 | +++ charmhelpers/core/hookenv.py 2021-11-10 05:36:20 +0000 |
737 | @@ -1,4 +1,4 @@ |
738 | -# Copyright 2014-2015 Canonical Limited. |
739 | +# Copyright 2013-2021 Canonical Limited. |
740 | # |
741 | # Licensed under the Apache License, Version 2.0 (the "License"); |
742 | # you may not use this file except in compliance with the License. |
743 | @@ -13,7 +13,6 @@ |
744 | # limitations under the License. |
745 | |
746 | "Interactions with the Juju environment" |
747 | -# Copyright 2013 Canonical Ltd. |
748 | # |
749 | # Authors: |
750 | # Charm Helpers Developers <juju@lists.ubuntu.com> |
751 | @@ -21,6 +20,7 @@ |
752 | from __future__ import print_function |
753 | import copy |
754 | from distutils.version import LooseVersion |
755 | +from enum import Enum |
756 | from functools import wraps |
757 | from collections import namedtuple |
758 | import glob |
759 | @@ -34,6 +34,8 @@ |
760 | import tempfile |
761 | from subprocess import CalledProcessError |
762 | |
763 | +from charmhelpers import deprecate |
764 | + |
765 | import six |
766 | if not six.PY3: |
767 | from UserDict import UserDict |
768 | @@ -55,6 +57,14 @@ |
769 | 'This may not be compatible with software you are ' |
770 | 'running in your shell.') |
771 | |
772 | + |
773 | +class WORKLOAD_STATES(Enum): |
774 | + ACTIVE = 'active' |
775 | + BLOCKED = 'blocked' |
776 | + MAINTENANCE = 'maintenance' |
777 | + WAITING = 'waiting' |
778 | + |
779 | + |
780 | cache = {} |
781 | |
782 | |
783 | @@ -119,6 +129,24 @@ |
784 | raise |
785 | |
786 | |
787 | +def function_log(message): |
788 | + """Write a function progress message""" |
789 | + command = ['function-log'] |
790 | + if not isinstance(message, six.string_types): |
791 | + message = repr(message) |
792 | + command += [message[:SH_MAX_ARG]] |
793 | + # Missing function-log should not cause failures in unit tests |
794 | + # Send function_log output to stderr |
795 | + try: |
796 | + subprocess.call(command) |
797 | + except OSError as e: |
798 | + if e.errno == errno.ENOENT: |
799 | + message = "function-log: {}".format(message) |
800 | + print(message, file=sys.stderr) |
801 | + else: |
802 | + raise |
803 | + |
804 | + |
805 | class Serializable(UserDict): |
806 | """Wrapper, an object that can be serialized to yaml or json""" |
807 | |
808 | @@ -197,6 +225,17 @@ |
809 | raise ValueError('Must specify neither or both of relation_name and service_or_unit') |
810 | |
811 | |
812 | +def departing_unit(): |
813 | + """The departing unit for the current relation hook. |
814 | + |
815 | + Available since juju 2.8. |
816 | + |
817 | + :returns: the departing unit, or None if the information isn't available. |
818 | + :rtype: Optional[str] |
819 | + """ |
820 | + return os.environ.get('JUJU_DEPARTING_UNIT', None) |
821 | + |
822 | + |
823 | def local_unit(): |
824 | """Local unit ID""" |
825 | return os.environ['JUJU_UNIT_NAME'] |
826 | @@ -343,8 +382,10 @@ |
827 | try: |
828 | self._prev_dict = json.load(f) |
829 | except ValueError as e: |
830 | - log('Unable to parse previous config data - {}'.format(str(e)), |
831 | - level=ERROR) |
832 | + log('Found but was unable to parse previous config data, ' |
833 | + 'ignoring which will report all values as changed - {}' |
834 | + .format(str(e)), level=ERROR) |
835 | + return |
836 | for k, v in copy.deepcopy(self._prev_dict).items(): |
837 | if k not in self: |
838 | self[k] = v |
839 | @@ -426,15 +467,20 @@ |
840 | |
841 | |
842 | @cached |
843 | -def relation_get(attribute=None, unit=None, rid=None): |
844 | +def relation_get(attribute=None, unit=None, rid=None, app=None): |
845 | """Get relation information""" |
846 | _args = ['relation-get', '--format=json'] |
847 | + if app is not None: |
848 | + if unit is not None: |
849 | + raise ValueError("Cannot use both 'unit' and 'app'") |
850 | + _args.append('--app') |
851 | if rid: |
852 | _args.append('-r') |
853 | _args.append(rid) |
854 | _args.append(attribute or '-') |
855 | - if unit: |
856 | - _args.append(unit) |
857 | + # unit or application name |
858 | + if unit or app: |
859 | + _args.append(unit or app) |
860 | try: |
861 | return json.loads(subprocess.check_output(_args).decode('UTF-8')) |
862 | except ValueError: |
863 | @@ -445,12 +491,14 @@ |
864 | raise |
865 | |
866 | |
867 | -def relation_set(relation_id=None, relation_settings=None, **kwargs): |
868 | +def relation_set(relation_id=None, relation_settings=None, app=False, **kwargs): |
869 | """Set relation information for the current unit""" |
870 | relation_settings = relation_settings if relation_settings else {} |
871 | relation_cmd_line = ['relation-set'] |
872 | accepts_file = "--file" in subprocess.check_output( |
873 | relation_cmd_line + ["--help"], universal_newlines=True) |
874 | + if app: |
875 | + relation_cmd_line.append('--app') |
876 | if relation_id is not None: |
877 | relation_cmd_line.extend(('-r', relation_id)) |
878 | settings = relation_settings.copy() |
879 | @@ -561,7 +609,7 @@ |
880 | relation_type())) |
881 | |
882 | :param reltype: Relation type to list data for, default is to list data for |
883 | - the realtion type we are currently executing a hook for. |
884 | + the relation type we are currently executing a hook for. |
885 | :type reltype: str |
886 | :returns: iterator |
887 | :rtype: types.GeneratorType |
888 | @@ -578,7 +626,7 @@ |
889 | |
890 | @cached |
891 | def relation_for_unit(unit=None, rid=None): |
892 | - """Get the json represenation of a unit's relation""" |
893 | + """Get the json representation of a unit's relation""" |
894 | unit = unit or remote_unit() |
895 | relation = relation_get(unit=unit, rid=rid) |
896 | for key in relation: |
897 | @@ -946,9 +994,23 @@ |
898 | return os.environ.get('CHARM_DIR') |
899 | |
900 | |
901 | +def cmd_exists(cmd): |
902 | + """Return True if the specified cmd exists in the path""" |
903 | + return any( |
904 | + os.access(os.path.join(path, cmd), os.X_OK) |
905 | + for path in os.environ["PATH"].split(os.pathsep) |
906 | + ) |
907 | + |
908 | + |
909 | @cached |
910 | +@deprecate("moved to function_get()", log=log) |
911 | def action_get(key=None): |
912 | - """Gets the value of an action parameter, or all key/value param pairs""" |
913 | + """ |
914 | + .. deprecated:: 0.20.7 |
915 | + Alias for :func:`function_get`. |
916 | + |
917 | + Gets the value of an action parameter, or all key/value param pairs. |
918 | + """ |
919 | cmd = ['action-get'] |
920 | if key is not None: |
921 | cmd.append(key) |
922 | @@ -957,52 +1019,130 @@ |
923 | return action_data |
924 | |
925 | |
926 | +@cached |
927 | +def function_get(key=None): |
928 | + """Gets the value of an action parameter, or all key/value param pairs""" |
929 | + cmd = ['function-get'] |
930 | + # Fallback for older charms. |
931 | + if not cmd_exists('function-get'): |
932 | + cmd = ['action-get'] |
933 | + |
934 | + if key is not None: |
935 | + cmd.append(key) |
936 | + cmd.append('--format=json') |
937 | + function_data = json.loads(subprocess.check_output(cmd).decode('UTF-8')) |
938 | + return function_data |
939 | + |
940 | + |
941 | +@deprecate("moved to function_set()", log=log) |
942 | def action_set(values): |
943 | - """Sets the values to be returned after the action finishes""" |
944 | + """ |
945 | + .. deprecated:: 0.20.7 |
946 | + Alias for :func:`function_set`. |
947 | + |
948 | + Sets the values to be returned after the action finishes. |
949 | + """ |
950 | cmd = ['action-set'] |
951 | for k, v in list(values.items()): |
952 | cmd.append('{}={}'.format(k, v)) |
953 | subprocess.check_call(cmd) |
954 | |
955 | |
956 | +def function_set(values): |
957 | + """Sets the values to be returned after the function finishes""" |
958 | + cmd = ['function-set'] |
959 | + # Fallback for older charms. |
960 | + if not cmd_exists('function-get'): |
961 | + cmd = ['action-set'] |
962 | + |
963 | + for k, v in list(values.items()): |
964 | + cmd.append('{}={}'.format(k, v)) |
965 | + subprocess.check_call(cmd) |
966 | + |
967 | + |
968 | +@deprecate("moved to function_fail()", log=log) |
969 | def action_fail(message): |
970 | - """Sets the action status to failed and sets the error message. |
971 | - |
972 | - The results set by action_set are preserved.""" |
973 | + """ |
974 | + .. deprecated:: 0.20.7 |
975 | + Alias for :func:`function_fail`. |
976 | + |
977 | + Sets the action status to failed and sets the error message. |
978 | + |
979 | + The results set by action_set are preserved. |
980 | + """ |
981 | subprocess.check_call(['action-fail', message]) |
982 | |
983 | |
984 | +def function_fail(message): |
985 | + """Sets the function status to failed and sets the error message. |
986 | + |
987 | + The results set by function_set are preserved.""" |
988 | + cmd = ['function-fail'] |
989 | + # Fallback for older charms. |
990 | + if not cmd_exists('function-fail'): |
991 | + cmd = ['action-fail'] |
992 | + cmd.append(message) |
993 | + |
994 | + subprocess.check_call(cmd) |
995 | + |
996 | + |
997 | def action_name(): |
998 | """Get the name of the currently executing action.""" |
999 | return os.environ.get('JUJU_ACTION_NAME') |
1000 | |
1001 | |
1002 | +def function_name(): |
1003 | + """Get the name of the currently executing function.""" |
1004 | + return os.environ.get('JUJU_FUNCTION_NAME') or action_name() |
1005 | + |
1006 | + |
1007 | def action_uuid(): |
1008 | """Get the UUID of the currently executing action.""" |
1009 | return os.environ.get('JUJU_ACTION_UUID') |
1010 | |
1011 | |
1012 | +def function_id(): |
1013 | + """Get the ID of the currently executing function.""" |
1014 | + return os.environ.get('JUJU_FUNCTION_ID') or action_uuid() |
1015 | + |
1016 | + |
1017 | def action_tag(): |
1018 | """Get the tag for the currently executing action.""" |
1019 | return os.environ.get('JUJU_ACTION_TAG') |
1020 | |
1021 | |
1022 | -def status_set(workload_state, message): |
1023 | +def function_tag(): |
1024 | + """Get the tag for the currently executing function.""" |
1025 | + return os.environ.get('JUJU_FUNCTION_TAG') or action_tag() |
1026 | + |
1027 | + |
1028 | +def status_set(workload_state, message, application=False): |
1029 | """Set the workload state with a message |
1030 | |
1031 | Use status-set to set the workload state with a message which is visible |
1032 | to the user via juju status. If the status-set command is not found then |
1033 | - assume this is juju < 1.23 and juju-log the message unstead. |
1034 | + assume this is juju < 1.23 and juju-log the message instead. |
1035 | |
1036 | - workload_state -- valid juju workload state. |
1037 | - message -- status update message |
1038 | + workload_state -- valid juju workload state. str or WORKLOAD_STATES |
1039 | + message -- status update message |
1040 | + application -- Whether this is an application state set |
1041 | """ |
1042 | - valid_states = ['maintenance', 'blocked', 'waiting', 'active'] |
1043 | - if workload_state not in valid_states: |
1044 | - raise ValueError( |
1045 | - '{!r} is not a valid workload state'.format(workload_state) |
1046 | - ) |
1047 | - cmd = ['status-set', workload_state, message] |
1048 | + bad_state_msg = '{!r} is not a valid workload state' |
1049 | + |
1050 | + if isinstance(workload_state, str): |
1051 | + try: |
1052 | + # Convert string to enum. |
1053 | + workload_state = WORKLOAD_STATES[workload_state.upper()] |
1054 | + except KeyError: |
1055 | + raise ValueError(bad_state_msg.format(workload_state)) |
1056 | + |
1057 | + if workload_state not in WORKLOAD_STATES: |
1058 | + raise ValueError(bad_state_msg.format(workload_state)) |
1059 | + |
1060 | + cmd = ['status-set'] |
1061 | + if application: |
1062 | + cmd.append('--application') |
1063 | + cmd.extend([workload_state.value, message]) |
1064 | try: |
1065 | ret = subprocess.call(cmd) |
1066 | if ret == 0: |
1067 | @@ -1010,7 +1150,7 @@ |
1068 | except OSError as e: |
1069 | if e.errno != errno.ENOENT: |
1070 | raise |
1071 | - log_message = 'status-set failed: {} {}'.format(workload_state, |
1072 | + log_message = 'status-set failed: {} {}'.format(workload_state.value, |
1073 | message) |
1074 | log(log_message, level='INFO') |
1075 | |
1076 | @@ -1425,13 +1565,13 @@ |
1077 | """Get proxy settings from process environment variables. |
1078 | |
1079 | Get charm proxy settings from environment variables that correspond to |
1080 | - juju-http-proxy, juju-https-proxy and juju-no-proxy (available as of 2.4.2, |
1081 | - see lp:1782236) in a format suitable for passing to an application that |
1082 | - reacts to proxy settings passed as environment variables. Some applications |
1083 | - support lowercase or uppercase notation (e.g. curl), some support only |
1084 | - lowercase (e.g. wget), there are also subjectively rare cases of only |
1085 | - uppercase notation support. no_proxy CIDR and wildcard support also varies |
1086 | - between runtimes and applications as there is no enforced standard. |
1087 | + juju-http-proxy, juju-https-proxy juju-no-proxy (available as of 2.4.2, see |
1088 | + lp:1782236) and juju-ftp-proxy in a format suitable for passing to an |
1089 | + application that reacts to proxy settings passed as environment variables. |
1090 | + Some applications support lowercase or uppercase notation (e.g. curl), some |
1091 | + support only lowercase (e.g. wget), there are also subjectively rare cases |
1092 | + of only uppercase notation support. no_proxy CIDR and wildcard support also |
1093 | + varies between runtimes and applications as there is no enforced standard. |
1094 | |
1095 | Some applications may connect to multiple destinations and expose config |
1096 | options that would affect only proxy settings for a specific destination |
1097 | @@ -1473,11 +1613,11 @@ |
1098 | def _contains_range(addresses): |
1099 | """Check for cidr or wildcard domain in a string. |
1100 | |
1101 | - Given a string comprising a comma seperated list of ip addresses |
1102 | + Given a string comprising a comma separated list of ip addresses |
1103 | and domain names, determine whether the string contains IP ranges |
1104 | or wildcard domains. |
1105 | |
1106 | - :param addresses: comma seperated list of domains and ip addresses. |
1107 | + :param addresses: comma separated list of domains and ip addresses. |
1108 | :type addresses: str |
1109 | """ |
1110 | return ( |
1111 | @@ -1488,3 +1628,12 @@ |
1112 | addresses.startswith(".") or |
1113 | ",." in addresses or |
1114 | " ." in addresses) |
1115 | + |
1116 | + |
1117 | +def is_subordinate(): |
1118 | + """Check whether charm is subordinate in unit metadata. |
1119 | + |
1120 | + :returns: True if unit is subordniate, False otherwise. |
1121 | + :rtype: bool |
1122 | + """ |
1123 | + return metadata().get('subordinate') is True |
1124 | |
1125 | === modified file 'charmhelpers/core/host.py' |
1126 | --- charmhelpers/core/host.py 2019-05-24 12:41:48 +0000 |
1127 | +++ charmhelpers/core/host.py 2021-11-10 05:36:20 +0000 |
1128 | @@ -1,4 +1,4 @@ |
1129 | -# Copyright 2014-2015 Canonical Limited. |
1130 | +# Copyright 2014-2021 Canonical Limited. |
1131 | # |
1132 | # Licensed under the Apache License, Version 2.0 (the "License"); |
1133 | # you may not use this file except in compliance with the License. |
1134 | @@ -19,6 +19,7 @@ |
1135 | # Nick Moffitt <nick.moffitt@canonical.com> |
1136 | # Matthew Wedgwood <matthew.wedgwood@canonical.com> |
1137 | |
1138 | +import errno |
1139 | import os |
1140 | import re |
1141 | import pwd |
1142 | @@ -33,7 +34,7 @@ |
1143 | import six |
1144 | |
1145 | from contextlib import contextmanager |
1146 | -from collections import OrderedDict |
1147 | +from collections import OrderedDict, defaultdict |
1148 | from .hookenv import log, INFO, DEBUG, local_unit, charm_name |
1149 | from .fstab import Fstab |
1150 | from charmhelpers.osplatform import get_platform |
1151 | @@ -59,6 +60,7 @@ |
1152 | ) # flake8: noqa -- ignore F401 for this import |
1153 | |
1154 | UPDATEDB_PATH = '/etc/updatedb.conf' |
1155 | +CA_CERT_DIR = '/usr/local/share/ca-certificates' |
1156 | |
1157 | |
1158 | def service_start(service_name, **kwargs): |
1159 | @@ -193,7 +195,7 @@ |
1160 | stopped = service_stop(service_name, **kwargs) |
1161 | upstart_file = os.path.join(init_dir, "{}.conf".format(service_name)) |
1162 | sysv_file = os.path.join(initd_dir, service_name) |
1163 | - if init_is_systemd(): |
1164 | + if init_is_systemd(service_name=service_name): |
1165 | service('disable', service_name) |
1166 | service('mask', service_name) |
1167 | elif os.path.exists(upstart_file): |
1168 | @@ -215,7 +217,7 @@ |
1169 | initd_dir="/etc/init.d", **kwargs): |
1170 | """Resume a system service. |
1171 | |
1172 | - Reenable starting again at boot. Start the service. |
1173 | + Re-enable starting again at boot. Start the service. |
1174 | |
1175 | :param service_name: the name of the service to resume |
1176 | :param init_dir: the path to the init dir |
1177 | @@ -227,7 +229,7 @@ |
1178 | """ |
1179 | upstart_file = os.path.join(init_dir, "{}.conf".format(service_name)) |
1180 | sysv_file = os.path.join(initd_dir, service_name) |
1181 | - if init_is_systemd(): |
1182 | + if init_is_systemd(service_name=service_name): |
1183 | service('unmask', service_name) |
1184 | service('enable', service_name) |
1185 | elif os.path.exists(upstart_file): |
1186 | @@ -257,7 +259,7 @@ |
1187 | :param **kwargs: additional params to be passed to the service command in |
1188 | the form of key=value. |
1189 | """ |
1190 | - if init_is_systemd(): |
1191 | + if init_is_systemd(service_name=service_name): |
1192 | cmd = ['systemctl', action, service_name] |
1193 | else: |
1194 | cmd = ['service', service_name, action] |
1195 | @@ -281,7 +283,7 @@ |
1196 | units (e.g. service ceph-osd status id=2). The kwargs |
1197 | are ignored in systemd services. |
1198 | """ |
1199 | - if init_is_systemd(): |
1200 | + if init_is_systemd(service_name=service_name): |
1201 | return service('is-active', service_name) |
1202 | else: |
1203 | if os.path.exists(_UPSTART_CONF.format(service_name)): |
1204 | @@ -311,8 +313,14 @@ |
1205 | SYSTEMD_SYSTEM = '/run/systemd/system' |
1206 | |
1207 | |
1208 | -def init_is_systemd(): |
1209 | - """Return True if the host system uses systemd, False otherwise.""" |
1210 | +def init_is_systemd(service_name=None): |
1211 | + """ |
1212 | + Returns whether the host uses systemd for the specified service. |
1213 | + |
1214 | + @param Optional[str] service_name: specific name of service |
1215 | + """ |
1216 | + if str(service_name).startswith("snap."): |
1217 | + return True |
1218 | if lsb_release()['DISTRIB_CODENAME'] == 'trusty': |
1219 | return False |
1220 | return os.path.isdir(SYSTEMD_SYSTEM) |
1221 | @@ -671,7 +679,7 @@ |
1222 | |
1223 | :param str checksum: Value of the checksum used to validate the file. |
1224 | :param str hash_type: Hash algorithm used to generate `checksum`. |
1225 | - Can be any hash alrgorithm supported by :mod:`hashlib`, |
1226 | + Can be any hash algorithm supported by :mod:`hashlib`, |
1227 | such as md5, sha1, sha256, sha512, etc. |
1228 | :raises ChecksumError: If the file fails the checksum |
1229 | |
1230 | @@ -686,78 +694,227 @@ |
1231 | pass |
1232 | |
1233 | |
1234 | -def restart_on_change(restart_map, stopstart=False, restart_functions=None): |
1235 | - """Restart services based on configuration files changing |
1236 | - |
1237 | - This function is used a decorator, for example:: |
1238 | - |
1239 | - @restart_on_change({ |
1240 | - '/etc/ceph/ceph.conf': [ 'cinder-api', 'cinder-volume' ] |
1241 | - '/etc/apache/sites-enabled/*': [ 'apache2' ] |
1242 | - }) |
1243 | - def config_changed(): |
1244 | - pass # your code here |
1245 | - |
1246 | - In this example, the cinder-api and cinder-volume services |
1247 | - would be restarted if /etc/ceph/ceph.conf is changed by the |
1248 | - ceph_client_changed function. The apache2 service would be |
1249 | - restarted if any file matching the pattern got changed, created |
1250 | - or removed. Standard wildcards are supported, see documentation |
1251 | - for the 'glob' module for more information. |
1252 | - |
1253 | - @param restart_map: {path_file_name: [service_name, ...] |
1254 | - @param stopstart: DEFAULT false; whether to stop, start OR restart |
1255 | - @param restart_functions: nonstandard functions to use to restart services |
1256 | - {svc: func, ...} |
1257 | - @returns result from decorated function |
1258 | +class restart_on_change(object): |
1259 | + """Decorator and context manager to handle restarts. |
1260 | + |
1261 | + Usage: |
1262 | + |
1263 | + @restart_on_change(restart_map, ...) |
1264 | + def function_that_might_trigger_a_restart(...) |
1265 | + ... |
1266 | + |
1267 | + Or: |
1268 | + |
1269 | + with restart_on_change(restart_map, ...): |
1270 | + do_stuff_that_might_trigger_a_restart() |
1271 | + ... |
1272 | """ |
1273 | - def wrap(f): |
1274 | + |
1275 | + def __init__(self, restart_map, stopstart=False, restart_functions=None, |
1276 | + can_restart_now_f=None, post_svc_restart_f=None, |
1277 | + pre_restarts_wait_f=None): |
1278 | + """ |
1279 | + :param restart_map: {file: [service, ...]} |
1280 | + :type restart_map: Dict[str, List[str,]] |
1281 | + :param stopstart: whether to stop, start or restart a service |
1282 | + :type stopstart: booleean |
1283 | + :param restart_functions: nonstandard functions to use to restart |
1284 | + services {svc: func, ...} |
1285 | + :type restart_functions: Dict[str, Callable[[str], None]] |
1286 | + :param can_restart_now_f: A function used to check if the restart is |
1287 | + permitted. |
1288 | + :type can_restart_now_f: Callable[[str, List[str]], boolean] |
1289 | + :param post_svc_restart_f: A function run after a service has |
1290 | + restarted. |
1291 | + :type post_svc_restart_f: Callable[[str], None] |
1292 | + :param pre_restarts_wait_f: A function called before any restarts. |
1293 | + :type pre_restarts_wait_f: Callable[None, None] |
1294 | + """ |
1295 | + self.restart_map = restart_map |
1296 | + self.stopstart = stopstart |
1297 | + self.restart_functions = restart_functions |
1298 | + self.can_restart_now_f = can_restart_now_f |
1299 | + self.post_svc_restart_f = post_svc_restart_f |
1300 | + self.pre_restarts_wait_f = pre_restarts_wait_f |
1301 | + |
1302 | + def __call__(self, f): |
1303 | + """Work like a decorator. |
1304 | + |
1305 | + Returns a wrapped function that performs the restart if triggered. |
1306 | + |
1307 | + :param f: The function that is being wrapped. |
1308 | + :type f: Callable[[Any], Any] |
1309 | + :returns: the wrapped function |
1310 | + :rtype: Callable[[Any], Any] |
1311 | + """ |
1312 | @functools.wraps(f) |
1313 | def wrapped_f(*args, **kwargs): |
1314 | return restart_on_change_helper( |
1315 | - (lambda: f(*args, **kwargs)), restart_map, stopstart, |
1316 | - restart_functions) |
1317 | + (lambda: f(*args, **kwargs)), |
1318 | + self.restart_map, |
1319 | + stopstart=self.stopstart, |
1320 | + restart_functions=self.restart_functions, |
1321 | + can_restart_now_f=self.can_restart_now_f, |
1322 | + post_svc_restart_f=self.post_svc_restart_f, |
1323 | + pre_restarts_wait_f=self.pre_restarts_wait_f) |
1324 | return wrapped_f |
1325 | - return wrap |
1326 | + |
1327 | + def __enter__(self): |
1328 | + """Enter the runtime context related to this object. """ |
1329 | + self.checksums = _pre_restart_on_change_helper(self.restart_map) |
1330 | + |
1331 | + def __exit__(self, exc_type, exc_val, exc_tb): |
1332 | + """Exit the runtime context related to this object. |
1333 | + |
1334 | + The parameters describe the exception that caused the context to be |
1335 | + exited. If the context was exited without an exception, all three |
1336 | + arguments will be None. |
1337 | + """ |
1338 | + if exc_type is None: |
1339 | + _post_restart_on_change_helper( |
1340 | + self.checksums, |
1341 | + self.restart_map, |
1342 | + stopstart=self.stopstart, |
1343 | + restart_functions=self.restart_functions, |
1344 | + can_restart_now_f=self.can_restart_now_f, |
1345 | + post_svc_restart_f=self.post_svc_restart_f, |
1346 | + pre_restarts_wait_f=self.pre_restarts_wait_f) |
1347 | + # All is good, so return False; any exceptions will propagate. |
1348 | + return False |
1349 | |
1350 | |
1351 | def restart_on_change_helper(lambda_f, restart_map, stopstart=False, |
1352 | - restart_functions=None): |
1353 | + restart_functions=None, |
1354 | + can_restart_now_f=None, |
1355 | + post_svc_restart_f=None, |
1356 | + pre_restarts_wait_f=None): |
1357 | """Helper function to perform the restart_on_change function. |
1358 | |
1359 | This is provided for decorators to restart services if files described |
1360 | in the restart_map have changed after an invocation of lambda_f(). |
1361 | |
1362 | - @param lambda_f: function to call. |
1363 | - @param restart_map: {file: [service, ...]} |
1364 | - @param stopstart: whether to stop, start or restart a service |
1365 | - @param restart_functions: nonstandard functions to use to restart services |
1366 | - {svc: func, ...} |
1367 | - @returns result of lambda_f() |
1368 | + This functions allows for a number of helper functions to be passed. |
1369 | + |
1370 | + `restart_functions` is a map with a service as the key and the |
1371 | + corresponding value being the function to call to restart the service. For |
1372 | + example if `restart_functions={'some-service': my_restart_func}` then |
1373 | + `my_restart_func` should a function which takes one argument which is the |
1374 | + service name to be retstarted. |
1375 | + |
1376 | + `can_restart_now_f` is a function which checks that a restart is permitted. |
1377 | + It should return a bool which indicates if a restart is allowed and should |
1378 | + take a service name (str) and a list of changed files (List[str]) as |
1379 | + arguments. |
1380 | + |
1381 | + `post_svc_restart_f` is a function which runs after a service has been |
1382 | + restarted. It takes the service name that was restarted as an argument. |
1383 | + |
1384 | + `pre_restarts_wait_f` is a function which is called before any restarts |
1385 | + occur. The use case for this is an application which wants to try and |
1386 | + stagger restarts between units. |
1387 | + |
1388 | + :param lambda_f: function to call. |
1389 | + :type lambda_f: Callable[[], ANY] |
1390 | + :param restart_map: {file: [service, ...]} |
1391 | + :type restart_map: Dict[str, List[str,]] |
1392 | + :param stopstart: whether to stop, start or restart a service |
1393 | + :type stopstart: booleean |
1394 | + :param restart_functions: nonstandard functions to use to restart services |
1395 | + {svc: func, ...} |
1396 | + :type restart_functions: Dict[str, Callable[[str], None]] |
1397 | + :param can_restart_now_f: A function used to check if the restart is |
1398 | + permitted. |
1399 | + :type can_restart_now_f: Callable[[str, List[str]], boolean] |
1400 | + :param post_svc_restart_f: A function run after a service has |
1401 | + restarted. |
1402 | + :type post_svc_restart_f: Callable[[str], None] |
1403 | + :param pre_restarts_wait_f: A function called before any restarts. |
1404 | + :type pre_restarts_wait_f: Callable[None, None] |
1405 | + :returns: result of lambda_f() |
1406 | + :rtype: ANY |
1407 | + """ |
1408 | + checksums = _pre_restart_on_change_helper(restart_map) |
1409 | + r = lambda_f() |
1410 | + _post_restart_on_change_helper(checksums, |
1411 | + restart_map, |
1412 | + stopstart, |
1413 | + restart_functions, |
1414 | + can_restart_now_f, |
1415 | + post_svc_restart_f, |
1416 | + pre_restarts_wait_f) |
1417 | + return r |
1418 | + |
1419 | + |
1420 | +def _pre_restart_on_change_helper(restart_map): |
1421 | + """Take a snapshot of file hashes. |
1422 | + |
1423 | + :param restart_map: {file: [service, ...]} |
1424 | + :type restart_map: Dict[str, List[str,]] |
1425 | + :returns: Dictionary of file paths and the files checksum. |
1426 | + :rtype: Dict[str, str] |
1427 | + """ |
1428 | + return {path: path_hash(path) for path in restart_map} |
1429 | + |
1430 | + |
1431 | +def _post_restart_on_change_helper(checksums, |
1432 | + restart_map, |
1433 | + stopstart=False, |
1434 | + restart_functions=None, |
1435 | + can_restart_now_f=None, |
1436 | + post_svc_restart_f=None, |
1437 | + pre_restarts_wait_f=None): |
1438 | + """Check whether files have changed. |
1439 | + |
1440 | + :param checksums: Dictionary of file paths and the files checksum. |
1441 | + :type checksums: Dict[str, str] |
1442 | + :param restart_map: {file: [service, ...]} |
1443 | + :type restart_map: Dict[str, List[str,]] |
1444 | + :param stopstart: whether to stop, start or restart a service |
1445 | + :type stopstart: booleean |
1446 | + :param restart_functions: nonstandard functions to use to restart services |
1447 | + {svc: func, ...} |
1448 | + :type restart_functions: Dict[str, Callable[[str], None]] |
1449 | + :param can_restart_now_f: A function used to check if the restart is |
1450 | + permitted. |
1451 | + :type can_restart_now_f: Callable[[str, List[str]], boolean] |
1452 | + :param post_svc_restart_f: A function run after a service has |
1453 | + restarted. |
1454 | + :type post_svc_restart_f: Callable[[str], None] |
1455 | + :param pre_restarts_wait_f: A function called before any restarts. |
1456 | + :type pre_restarts_wait_f: Callable[None, None] |
1457 | """ |
1458 | if restart_functions is None: |
1459 | restart_functions = {} |
1460 | - checksums = {path: path_hash(path) for path in restart_map} |
1461 | - r = lambda_f() |
1462 | + changed_files = defaultdict(list) |
1463 | + restarts = [] |
1464 | # create a list of lists of the services to restart |
1465 | - restarts = [restart_map[path] |
1466 | - for path in restart_map |
1467 | - if path_hash(path) != checksums[path]] |
1468 | + for path, services in restart_map.items(): |
1469 | + if path_hash(path) != checksums[path]: |
1470 | + restarts.append(services) |
1471 | + for svc in services: |
1472 | + changed_files[svc].append(path) |
1473 | # create a flat list of ordered services without duplicates from lists |
1474 | services_list = list(OrderedDict.fromkeys(itertools.chain(*restarts))) |
1475 | if services_list: |
1476 | + if pre_restarts_wait_f: |
1477 | + pre_restarts_wait_f() |
1478 | actions = ('stop', 'start') if stopstart else ('restart',) |
1479 | for service_name in services_list: |
1480 | + if can_restart_now_f: |
1481 | + if not can_restart_now_f(service_name, |
1482 | + changed_files[service_name]): |
1483 | + continue |
1484 | if service_name in restart_functions: |
1485 | restart_functions[service_name](service_name) |
1486 | else: |
1487 | for action in actions: |
1488 | service(action, service_name) |
1489 | - return r |
1490 | + if post_svc_restart_f: |
1491 | + post_svc_restart_f(service_name) |
1492 | |
1493 | |
1494 | def pwgen(length=None): |
1495 | - """Generate a random pasword.""" |
1496 | + """Generate a random password.""" |
1497 | if length is None: |
1498 | # A random length is ok to use a weak PRNG |
1499 | length = random.choice(range(35, 45)) |
1500 | @@ -819,7 +976,8 @@ |
1501 | if nic_type: |
1502 | for int_type in int_types: |
1503 | cmd = ['ip', 'addr', 'show', 'label', int_type + '*'] |
1504 | - ip_output = subprocess.check_output(cmd).decode('UTF-8') |
1505 | + ip_output = subprocess.check_output( |
1506 | + cmd).decode('UTF-8', errors='replace') |
1507 | ip_output = ip_output.split('\n') |
1508 | ip_output = (line for line in ip_output if line) |
1509 | for line in ip_output: |
1510 | @@ -835,7 +993,8 @@ |
1511 | interfaces.append(iface) |
1512 | else: |
1513 | cmd = ['ip', 'a'] |
1514 | - ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n') |
1515 | + ip_output = subprocess.check_output( |
1516 | + cmd).decode('UTF-8', errors='replace').split('\n') |
1517 | ip_output = (line.strip() for line in ip_output if line) |
1518 | |
1519 | key = re.compile(r'^[0-9]+:\s+(.+):') |
1520 | @@ -859,7 +1018,8 @@ |
1521 | def get_nic_mtu(nic): |
1522 | """Return the Maximum Transmission Unit (MTU) for a network interface.""" |
1523 | cmd = ['ip', 'addr', 'show', nic] |
1524 | - ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n') |
1525 | + ip_output = subprocess.check_output( |
1526 | + cmd).decode('UTF-8', errors='replace').split('\n') |
1527 | mtu = "" |
1528 | for line in ip_output: |
1529 | words = line.split() |
1530 | @@ -871,7 +1031,7 @@ |
1531 | def get_nic_hwaddr(nic): |
1532 | """Return the Media Access Control (MAC) for a network interface.""" |
1533 | cmd = ['ip', '-o', '-0', 'addr', 'show', nic] |
1534 | - ip_output = subprocess.check_output(cmd).decode('UTF-8') |
1535 | + ip_output = subprocess.check_output(cmd).decode('UTF-8', errors='replace') |
1536 | hwaddr = "" |
1537 | words = ip_output.split() |
1538 | if 'link/ether' in words: |
1539 | @@ -883,7 +1043,7 @@ |
1540 | def chdir(directory): |
1541 | """Change the current working directory to a different directory for a code |
1542 | block and return the previous directory after the block exits. Useful to |
1543 | - run commands from a specificed directory. |
1544 | + run commands from a specified directory. |
1545 | |
1546 | :param str directory: The directory path to change to for this context. |
1547 | """ |
1548 | @@ -918,9 +1078,13 @@ |
1549 | for root, dirs, files in os.walk(path, followlinks=follow_links): |
1550 | for name in dirs + files: |
1551 | full = os.path.join(root, name) |
1552 | - broken_symlink = os.path.lexists(full) and not os.path.exists(full) |
1553 | - if not broken_symlink: |
1554 | + try: |
1555 | chown(full, uid, gid) |
1556 | + except (IOError, OSError) as e: |
1557 | + # Intended to ignore "file not found". Catching both to be |
1558 | + # compatible with both Python 2.7 and 3.x. |
1559 | + if e.errno == errno.ENOENT: |
1560 | + pass |
1561 | |
1562 | |
1563 | def lchownr(path, owner, group): |
1564 | @@ -1053,6 +1217,17 @@ |
1565 | return calculated_wait_time |
1566 | |
1567 | |
1568 | +def ca_cert_absolute_path(basename_without_extension): |
1569 | + """Returns absolute path to CA certificate. |
1570 | + |
1571 | + :param basename_without_extension: Filename without extension |
1572 | + :type basename_without_extension: str |
1573 | + :returns: Absolute full path |
1574 | + :rtype: str |
1575 | + """ |
1576 | + return '{}/{}.crt'.format(CA_CERT_DIR, basename_without_extension) |
1577 | + |
1578 | + |
1579 | def install_ca_cert(ca_cert, name=None): |
1580 | """ |
1581 | Install the given cert as a trusted CA. |
1582 | @@ -1068,10 +1243,37 @@ |
1583 | ca_cert = ca_cert.encode('utf8') |
1584 | if not name: |
1585 | name = 'juju-{}'.format(charm_name()) |
1586 | - cert_file = '/usr/local/share/ca-certificates/{}.crt'.format(name) |
1587 | + cert_file = ca_cert_absolute_path(name) |
1588 | new_hash = hashlib.md5(ca_cert).hexdigest() |
1589 | if file_hash(cert_file) == new_hash: |
1590 | return |
1591 | log("Installing new CA cert at: {}".format(cert_file), level=INFO) |
1592 | write_file(cert_file, ca_cert) |
1593 | subprocess.check_call(['update-ca-certificates', '--fresh']) |
1594 | + |
1595 | + |
1596 | +def get_system_env(key, default=None): |
1597 | + """Get data from system environment as represented in ``/etc/environment``. |
1598 | + |
1599 | + :param key: Key to look up |
1600 | + :type key: str |
1601 | + :param default: Value to return if key is not found |
1602 | + :type default: any |
1603 | + :returns: Value for key if found or contents of default parameter |
1604 | + :rtype: any |
1605 | + :raises: subprocess.CalledProcessError |
1606 | + """ |
1607 | + env_file = '/etc/environment' |
1608 | + # use the shell and env(1) to parse the global environments file. This is |
1609 | + # done to get the correct result even if the user has shell variable |
1610 | + # substitutions or other shell logic in that file. |
1611 | + output = subprocess.check_output( |
1612 | + ['env', '-i', '/bin/bash', '-c', |
1613 | + 'set -a && source {} && env'.format(env_file)], |
1614 | + universal_newlines=True) |
1615 | + for k, v in (line.split('=', 1) |
1616 | + for line in output.splitlines() if '=' in line): |
1617 | + if k == key: |
1618 | + return v |
1619 | + else: |
1620 | + return default |
1621 | |
1622 | === modified file 'charmhelpers/core/host_factory/ubuntu.py' |
1623 | --- charmhelpers/core/host_factory/ubuntu.py 2019-05-24 12:41:48 +0000 |
1624 | +++ charmhelpers/core/host_factory/ubuntu.py 2021-11-10 05:36:20 +0000 |
1625 | @@ -24,6 +24,12 @@ |
1626 | 'bionic', |
1627 | 'cosmic', |
1628 | 'disco', |
1629 | + 'eoan', |
1630 | + 'focal', |
1631 | + 'groovy', |
1632 | + 'hirsute', |
1633 | + 'impish', |
1634 | + 'jammy', |
1635 | ) |
1636 | |
1637 | |
1638 | @@ -93,12 +99,14 @@ |
1639 | the pkgcache argument is None. Be sure to add charmhelpers.fetch if |
1640 | you call this function, or pass an apt_pkg.Cache() instance. |
1641 | """ |
1642 | - import apt_pkg |
1643 | + from charmhelpers.fetch import apt_pkg, get_installed_version |
1644 | if not pkgcache: |
1645 | - from charmhelpers.fetch import apt_cache |
1646 | - pkgcache = apt_cache() |
1647 | - pkg = pkgcache[package] |
1648 | - return apt_pkg.version_compare(pkg.current_ver.ver_str, revno) |
1649 | + current_ver = get_installed_version(package) |
1650 | + else: |
1651 | + pkg = pkgcache[package] |
1652 | + current_ver = pkg.current_ver |
1653 | + |
1654 | + return apt_pkg.version_compare(current_ver.ver_str, revno) |
1655 | |
1656 | |
1657 | @cached |
1658 | |
1659 | === modified file 'charmhelpers/core/services/base.py' |
1660 | --- charmhelpers/core/services/base.py 2019-05-24 12:41:48 +0000 |
1661 | +++ charmhelpers/core/services/base.py 2021-11-10 05:36:20 +0000 |
1662 | @@ -14,9 +14,11 @@ |
1663 | |
1664 | import os |
1665 | import json |
1666 | -from inspect import getargspec |
1667 | +import inspect |
1668 | from collections import Iterable, OrderedDict |
1669 | |
1670 | +import six |
1671 | + |
1672 | from charmhelpers.core import host |
1673 | from charmhelpers.core import hookenv |
1674 | |
1675 | @@ -169,7 +171,10 @@ |
1676 | if not units: |
1677 | continue |
1678 | remote_service = units[0].split('/')[0] |
1679 | - argspec = getargspec(provider.provide_data) |
1680 | + if six.PY2: |
1681 | + argspec = inspect.getargspec(provider.provide_data) |
1682 | + else: |
1683 | + argspec = inspect.getfullargspec(provider.provide_data) |
1684 | if len(argspec.args) > 1: |
1685 | data = provider.provide_data(remote_service, service_ready) |
1686 | else: |
1687 | |
1688 | === modified file 'charmhelpers/core/strutils.py' |
1689 | --- charmhelpers/core/strutils.py 2019-05-24 12:41:48 +0000 |
1690 | +++ charmhelpers/core/strutils.py 2021-11-10 05:36:20 +0000 |
1691 | @@ -18,8 +18,11 @@ |
1692 | import six |
1693 | import re |
1694 | |
1695 | - |
1696 | -def bool_from_string(value): |
1697 | +TRUTHY_STRINGS = {'y', 'yes', 'true', 't', 'on'} |
1698 | +FALSEY_STRINGS = {'n', 'no', 'false', 'f', 'off'} |
1699 | + |
1700 | + |
1701 | +def bool_from_string(value, truthy_strings=TRUTHY_STRINGS, falsey_strings=FALSEY_STRINGS, assume_false=False): |
1702 | """Interpret string value as boolean. |
1703 | |
1704 | Returns True if value translates to True otherwise False. |
1705 | @@ -32,9 +35,9 @@ |
1706 | |
1707 | value = value.strip().lower() |
1708 | |
1709 | - if value in ['y', 'yes', 'true', 't', 'on']: |
1710 | + if value in truthy_strings: |
1711 | return True |
1712 | - elif value in ['n', 'no', 'false', 'f', 'off']: |
1713 | + elif value in falsey_strings or assume_false: |
1714 | return False |
1715 | |
1716 | msg = "Unable to interpret string value '%s' as boolean" % (value) |
1717 | |
1718 | === modified file 'charmhelpers/core/sysctl.py' |
1719 | --- charmhelpers/core/sysctl.py 2019-05-24 12:41:48 +0000 |
1720 | +++ charmhelpers/core/sysctl.py 2021-11-10 05:36:20 +0000 |
1721 | @@ -17,14 +17,17 @@ |
1722 | |
1723 | import yaml |
1724 | |
1725 | -from subprocess import check_call |
1726 | +from subprocess import check_call, CalledProcessError |
1727 | |
1728 | from charmhelpers.core.hookenv import ( |
1729 | log, |
1730 | DEBUG, |
1731 | ERROR, |
1732 | + WARNING, |
1733 | ) |
1734 | |
1735 | +from charmhelpers.core.host import is_container |
1736 | + |
1737 | __author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>' |
1738 | |
1739 | |
1740 | @@ -62,4 +65,11 @@ |
1741 | if ignore: |
1742 | call.append("-e") |
1743 | |
1744 | - check_call(call) |
1745 | + try: |
1746 | + check_call(call) |
1747 | + except CalledProcessError as e: |
1748 | + if is_container(): |
1749 | + log("Error setting some sysctl keys in this container: {}".format(e.output), |
1750 | + level=WARNING) |
1751 | + else: |
1752 | + raise e |
1753 | |
1754 | === modified file 'charmhelpers/core/unitdata.py' |
1755 | --- charmhelpers/core/unitdata.py 2019-05-24 12:41:48 +0000 |
1756 | +++ charmhelpers/core/unitdata.py 2021-11-10 05:36:20 +0000 |
1757 | @@ -1,7 +1,7 @@ |
1758 | #!/usr/bin/env python |
1759 | # -*- coding: utf-8 -*- |
1760 | # |
1761 | -# Copyright 2014-2015 Canonical Limited. |
1762 | +# Copyright 2014-2021 Canonical Limited. |
1763 | # |
1764 | # Licensed under the Apache License, Version 2.0 (the "License"); |
1765 | # you may not use this file except in compliance with the License. |
1766 | @@ -61,7 +61,7 @@ |
1767 | 'previous value', prev, |
1768 | 'current value', cur) |
1769 | |
1770 | - # Get some unit specific bookeeping |
1771 | + # Get some unit specific bookkeeping |
1772 | if not db.get('pkg_key'): |
1773 | key = urllib.urlopen('https://example.com/pkg_key').read() |
1774 | db.set('pkg_key', key) |
1775 | @@ -449,7 +449,7 @@ |
1776 | 'previous value', prev, |
1777 | 'current value', cur) |
1778 | |
1779 | - # Get some unit specific bookeeping |
1780 | + # Get some unit specific bookkeeping |
1781 | if not db.get('pkg_key'): |
1782 | key = urllib.urlopen('https://example.com/pkg_key').read() |
1783 | db.set('pkg_key', key) |
1784 | |
1785 | === modified file 'charmhelpers/fetch/__init__.py' |
1786 | --- charmhelpers/fetch/__init__.py 2019-05-24 12:41:48 +0000 |
1787 | +++ charmhelpers/fetch/__init__.py 2021-11-10 05:36:20 +0000 |
1788 | @@ -1,4 +1,4 @@ |
1789 | -# Copyright 2014-2015 Canonical Limited. |
1790 | +# Copyright 2014-2021 Canonical Limited. |
1791 | # |
1792 | # Licensed under the Apache License, Version 2.0 (the "License"); |
1793 | # you may not use this file except in compliance with the License. |
1794 | @@ -103,6 +103,11 @@ |
1795 | apt_unhold = fetch.apt_unhold |
1796 | import_key = fetch.import_key |
1797 | get_upstream_version = fetch.get_upstream_version |
1798 | + apt_pkg = fetch.ubuntu_apt_pkg |
1799 | + get_apt_dpkg_env = fetch.get_apt_dpkg_env |
1800 | + get_installed_version = fetch.get_installed_version |
1801 | + OPENSTACK_RELEASES = fetch.OPENSTACK_RELEASES |
1802 | + UBUNTU_OPENSTACK_RELEASE = fetch.UBUNTU_OPENSTACK_RELEASE |
1803 | elif __platform__ == "centos": |
1804 | yum_search = fetch.yum_search |
1805 | |
1806 | @@ -200,7 +205,7 @@ |
1807 | classname) |
1808 | plugin_list.append(handler_class()) |
1809 | except NotImplementedError: |
1810 | - # Skip missing plugins so that they can be ommitted from |
1811 | + # Skip missing plugins so that they can be omitted from |
1812 | # installation if desired |
1813 | log("FetchHandler {} not found, skipping plugin".format( |
1814 | handler_name)) |
1815 | |
1816 | === modified file 'charmhelpers/fetch/python/packages.py' |
1817 | --- charmhelpers/fetch/python/packages.py 2019-05-24 12:41:48 +0000 |
1818 | +++ charmhelpers/fetch/python/packages.py 2021-11-10 05:36:20 +0000 |
1819 | @@ -1,7 +1,7 @@ |
1820 | #!/usr/bin/env python |
1821 | # coding: utf-8 |
1822 | |
1823 | -# Copyright 2014-2015 Canonical Limited. |
1824 | +# Copyright 2014-2021 Canonical Limited. |
1825 | # |
1826 | # Licensed under the Apache License, Version 2.0 (the "License"); |
1827 | # you may not use this file except in compliance with the License. |
1828 | @@ -27,7 +27,7 @@ |
1829 | |
1830 | |
1831 | def pip_execute(*args, **kwargs): |
1832 | - """Overriden pip_execute() to stop sys.path being changed. |
1833 | + """Overridden pip_execute() to stop sys.path being changed. |
1834 | |
1835 | The act of importing main from the pip module seems to cause add wheels |
1836 | from the /usr/share/python-wheels which are installed by various tools. |
1837 | @@ -142,8 +142,10 @@ |
1838 | """Create an isolated Python environment.""" |
1839 | if six.PY2: |
1840 | apt_install('python-virtualenv') |
1841 | + extra_flags = [] |
1842 | else: |
1843 | - apt_install('python3-virtualenv') |
1844 | + apt_install(['python3-virtualenv', 'virtualenv']) |
1845 | + extra_flags = ['--python=python3'] |
1846 | |
1847 | if path: |
1848 | venv_path = path |
1849 | @@ -151,4 +153,4 @@ |
1850 | venv_path = os.path.join(charm_dir(), 'venv') |
1851 | |
1852 | if not os.path.exists(venv_path): |
1853 | - subprocess.check_call(['virtualenv', venv_path]) |
1854 | + subprocess.check_call(['virtualenv', venv_path] + extra_flags) |
1855 | |
1856 | === modified file 'charmhelpers/fetch/snap.py' |
1857 | --- charmhelpers/fetch/snap.py 2019-05-24 12:41:48 +0000 |
1858 | +++ charmhelpers/fetch/snap.py 2021-11-10 05:36:20 +0000 |
1859 | @@ -1,4 +1,4 @@ |
1860 | -# Copyright 2014-2017 Canonical Limited. |
1861 | +# Copyright 2014-2021 Canonical Limited. |
1862 | # |
1863 | # Licensed under the Apache License, Version 2.0 (the "License"); |
1864 | # you may not use this file except in compliance with the License. |
1865 | @@ -65,11 +65,11 @@ |
1866 | retry_count += + 1 |
1867 | if retry_count > SNAP_NO_LOCK_RETRY_COUNT: |
1868 | raise CouldNotAcquireLockException( |
1869 | - 'Could not aquire lock after {} attempts' |
1870 | + 'Could not acquire lock after {} attempts' |
1871 | .format(SNAP_NO_LOCK_RETRY_COUNT)) |
1872 | return_code = e.returncode |
1873 | log('Snap failed to acquire lock, trying again in {} seconds.' |
1874 | - .format(SNAP_NO_LOCK_RETRY_DELAY, level='WARN')) |
1875 | + .format(SNAP_NO_LOCK_RETRY_DELAY), level='WARN') |
1876 | sleep(SNAP_NO_LOCK_RETRY_DELAY) |
1877 | |
1878 | return return_code |
1879 | |
1880 | === modified file 'charmhelpers/fetch/ubuntu.py' |
1881 | --- charmhelpers/fetch/ubuntu.py 2019-05-24 12:41:48 +0000 |
1882 | +++ charmhelpers/fetch/ubuntu.py 2021-11-10 05:36:20 +0000 |
1883 | @@ -1,4 +1,4 @@ |
1884 | -# Copyright 2014-2015 Canonical Limited. |
1885 | +# Copyright 2014-2021 Canonical Limited. |
1886 | # |
1887 | # Licensed under the Apache License, Version 2.0 (the "License"); |
1888 | # you may not use this file except in compliance with the License. |
1889 | @@ -17,10 +17,12 @@ |
1890 | import platform |
1891 | import re |
1892 | import six |
1893 | +import subprocess |
1894 | +import sys |
1895 | import time |
1896 | -import subprocess |
1897 | |
1898 | -from charmhelpers.core.host import get_distrib_codename |
1899 | +from charmhelpers import deprecate |
1900 | +from charmhelpers.core.host import get_distrib_codename, get_system_env |
1901 | |
1902 | from charmhelpers.core.hookenv import ( |
1903 | log, |
1904 | @@ -29,6 +31,7 @@ |
1905 | env_proxy_settings, |
1906 | ) |
1907 | from charmhelpers.fetch import SourceConfigError, GPGKeyError |
1908 | +from charmhelpers.fetch import ubuntu_apt_pkg |
1909 | |
1910 | PROPOSED_POCKET = ( |
1911 | "# Proposed\n" |
1912 | @@ -173,12 +176,112 @@ |
1913 | 'stein/proposed': 'bionic-proposed/stein', |
1914 | 'bionic-stein/proposed': 'bionic-proposed/stein', |
1915 | 'bionic-proposed/stein': 'bionic-proposed/stein', |
1916 | + # Train |
1917 | + 'train': 'bionic-updates/train', |
1918 | + 'bionic-train': 'bionic-updates/train', |
1919 | + 'bionic-train/updates': 'bionic-updates/train', |
1920 | + 'bionic-updates/train': 'bionic-updates/train', |
1921 | + 'train/proposed': 'bionic-proposed/train', |
1922 | + 'bionic-train/proposed': 'bionic-proposed/train', |
1923 | + 'bionic-proposed/train': 'bionic-proposed/train', |
1924 | + # Ussuri |
1925 | + 'ussuri': 'bionic-updates/ussuri', |
1926 | + 'bionic-ussuri': 'bionic-updates/ussuri', |
1927 | + 'bionic-ussuri/updates': 'bionic-updates/ussuri', |
1928 | + 'bionic-updates/ussuri': 'bionic-updates/ussuri', |
1929 | + 'ussuri/proposed': 'bionic-proposed/ussuri', |
1930 | + 'bionic-ussuri/proposed': 'bionic-proposed/ussuri', |
1931 | + 'bionic-proposed/ussuri': 'bionic-proposed/ussuri', |
1932 | + # Victoria |
1933 | + 'victoria': 'focal-updates/victoria', |
1934 | + 'focal-victoria': 'focal-updates/victoria', |
1935 | + 'focal-victoria/updates': 'focal-updates/victoria', |
1936 | + 'focal-updates/victoria': 'focal-updates/victoria', |
1937 | + 'victoria/proposed': 'focal-proposed/victoria', |
1938 | + 'focal-victoria/proposed': 'focal-proposed/victoria', |
1939 | + 'focal-proposed/victoria': 'focal-proposed/victoria', |
1940 | + # Wallaby |
1941 | + 'wallaby': 'focal-updates/wallaby', |
1942 | + 'focal-wallaby': 'focal-updates/wallaby', |
1943 | + 'focal-wallaby/updates': 'focal-updates/wallaby', |
1944 | + 'focal-updates/wallaby': 'focal-updates/wallaby', |
1945 | + 'wallaby/proposed': 'focal-proposed/wallaby', |
1946 | + 'focal-wallaby/proposed': 'focal-proposed/wallaby', |
1947 | + 'focal-proposed/wallaby': 'focal-proposed/wallaby', |
1948 | + # Xena |
1949 | + 'xena': 'focal-updates/xena', |
1950 | + 'focal-xena': 'focal-updates/xena', |
1951 | + 'focal-xena/updates': 'focal-updates/xena', |
1952 | + 'focal-updates/xena': 'focal-updates/xena', |
1953 | + 'xena/proposed': 'focal-proposed/xena', |
1954 | + 'focal-xena/proposed': 'focal-proposed/xena', |
1955 | + 'focal-proposed/xena': 'focal-proposed/xena', |
1956 | + # Yoga |
1957 | + 'yoga': 'focal-updates/yoga', |
1958 | + 'focal-yoga': 'focal-updates/yoga', |
1959 | + 'focal-yoga/updates': 'focal-updates/yoga', |
1960 | + 'focal-updates/yoga': 'focal-updates/yoga', |
1961 | + 'yoga/proposed': 'focal-proposed/yoga', |
1962 | + 'focal-yoga/proposed': 'focal-proposed/yoga', |
1963 | + 'focal-proposed/yoga': 'focal-proposed/yoga', |
1964 | } |
1965 | |
1966 | |
1967 | +OPENSTACK_RELEASES = ( |
1968 | + 'diablo', |
1969 | + 'essex', |
1970 | + 'folsom', |
1971 | + 'grizzly', |
1972 | + 'havana', |
1973 | + 'icehouse', |
1974 | + 'juno', |
1975 | + 'kilo', |
1976 | + 'liberty', |
1977 | + 'mitaka', |
1978 | + 'newton', |
1979 | + 'ocata', |
1980 | + 'pike', |
1981 | + 'queens', |
1982 | + 'rocky', |
1983 | + 'stein', |
1984 | + 'train', |
1985 | + 'ussuri', |
1986 | + 'victoria', |
1987 | + 'wallaby', |
1988 | + 'xena', |
1989 | + 'yoga', |
1990 | +) |
1991 | + |
1992 | + |
1993 | +UBUNTU_OPENSTACK_RELEASE = OrderedDict([ |
1994 | + ('oneiric', 'diablo'), |
1995 | + ('precise', 'essex'), |
1996 | + ('quantal', 'folsom'), |
1997 | + ('raring', 'grizzly'), |
1998 | + ('saucy', 'havana'), |
1999 | + ('trusty', 'icehouse'), |
2000 | + ('utopic', 'juno'), |
2001 | + ('vivid', 'kilo'), |
2002 | + ('wily', 'liberty'), |
2003 | + ('xenial', 'mitaka'), |
2004 | + ('yakkety', 'newton'), |
2005 | + ('zesty', 'ocata'), |
2006 | + ('artful', 'pike'), |
2007 | + ('bionic', 'queens'), |
2008 | + ('cosmic', 'rocky'), |
2009 | + ('disco', 'stein'), |
2010 | + ('eoan', 'train'), |
2011 | + ('focal', 'ussuri'), |
2012 | + ('groovy', 'victoria'), |
2013 | + ('hirsute', 'wallaby'), |
2014 | + ('impish', 'xena'), |
2015 | + ('jammy', 'yoga'), |
2016 | +]) |
2017 | + |
2018 | + |
2019 | APT_NO_LOCK = 100 # The return code for "couldn't acquire lock" in APT. |
2020 | CMD_RETRY_DELAY = 10 # Wait 10 seconds between command retries. |
2021 | -CMD_RETRY_COUNT = 3 # Retry a failing fatal command X times. |
2022 | +CMD_RETRY_COUNT = 10 # Retry a failing fatal command X times. |
2023 | |
2024 | |
2025 | def filter_installed_packages(packages): |
2026 | @@ -208,18 +311,50 @@ |
2027 | ) |
2028 | |
2029 | |
2030 | -def apt_cache(in_memory=True, progress=None): |
2031 | - """Build and return an apt cache.""" |
2032 | - from apt import apt_pkg |
2033 | - apt_pkg.init() |
2034 | - if in_memory: |
2035 | - apt_pkg.config.set("Dir::Cache::pkgcache", "") |
2036 | - apt_pkg.config.set("Dir::Cache::srcpkgcache", "") |
2037 | - return apt_pkg.Cache(progress) |
2038 | - |
2039 | - |
2040 | -def apt_install(packages, options=None, fatal=False): |
2041 | - """Install one or more packages.""" |
2042 | +def apt_cache(*_, **__): |
2043 | + """Shim returning an object simulating the apt_pkg Cache. |
2044 | + |
2045 | + :param _: Accept arguments for compatibility, not used. |
2046 | + :type _: any |
2047 | + :param __: Accept keyword arguments for compatibility, not used. |
2048 | + :type __: any |
2049 | + :returns:Object used to interrogate the system apt and dpkg databases. |
2050 | + :rtype:ubuntu_apt_pkg.Cache |
2051 | + """ |
2052 | + if 'apt_pkg' in sys.modules: |
2053 | + # NOTE(fnordahl): When our consumer use the upstream ``apt_pkg`` module |
2054 | + # in conjunction with the apt_cache helper function, they may expect us |
2055 | + # to call ``apt_pkg.init()`` for them. |
2056 | + # |
2057 | + # Detect this situation, log a warning and make the call to |
2058 | + # ``apt_pkg.init()`` to avoid the consumer Python interpreter from |
2059 | + # crashing with a segmentation fault. |
2060 | + @deprecate( |
2061 | + 'Support for use of upstream ``apt_pkg`` module in conjunction' |
2062 | + 'with charm-helpers is deprecated since 2019-06-25', |
2063 | + date=None, log=lambda x: log(x, level=WARNING)) |
2064 | + def one_shot_log(): |
2065 | + pass |
2066 | + |
2067 | + one_shot_log() |
2068 | + sys.modules['apt_pkg'].init() |
2069 | + return ubuntu_apt_pkg.Cache() |
2070 | + |
2071 | + |
2072 | +def apt_install(packages, options=None, fatal=False, quiet=False): |
2073 | + """Install one or more packages. |
2074 | + |
2075 | + :param packages: Package(s) to install |
2076 | + :type packages: Option[str, List[str]] |
2077 | + :param options: Options to pass on to apt-get |
2078 | + :type options: Option[None, List[str]] |
2079 | + :param fatal: Whether the command's output should be checked and |
2080 | + retried. |
2081 | + :type fatal: bool |
2082 | + :param quiet: if True (default), suppress log message to stdout/stderr |
2083 | + :type quiet: bool |
2084 | + :raises: subprocess.CalledProcessError |
2085 | + """ |
2086 | if options is None: |
2087 | options = ['--option=Dpkg::Options::=--force-confold'] |
2088 | |
2089 | @@ -230,13 +365,24 @@ |
2090 | cmd.append(packages) |
2091 | else: |
2092 | cmd.extend(packages) |
2093 | - log("Installing {} with options: {}".format(packages, |
2094 | - options)) |
2095 | - _run_apt_command(cmd, fatal) |
2096 | + if not quiet: |
2097 | + log("Installing {} with options: {}" |
2098 | + .format(packages, options)) |
2099 | + _run_apt_command(cmd, fatal, quiet=quiet) |
2100 | |
2101 | |
2102 | def apt_upgrade(options=None, fatal=False, dist=False): |
2103 | - """Upgrade all packages.""" |
2104 | + """Upgrade all packages. |
2105 | + |
2106 | + :param options: Options to pass on to apt-get |
2107 | + :type options: Option[None, List[str]] |
2108 | + :param fatal: Whether the command's output should be checked and |
2109 | + retried. |
2110 | + :type fatal: bool |
2111 | + :param dist: Whether ``dist-upgrade`` should be used over ``upgrade`` |
2112 | + :type dist: bool |
2113 | + :raises: subprocess.CalledProcessError |
2114 | + """ |
2115 | if options is None: |
2116 | options = ['--option=Dpkg::Options::=--force-confold'] |
2117 | |
2118 | @@ -257,7 +403,15 @@ |
2119 | |
2120 | |
2121 | def apt_purge(packages, fatal=False): |
2122 | - """Purge one or more packages.""" |
2123 | + """Purge one or more packages. |
2124 | + |
2125 | + :param packages: Package(s) to install |
2126 | + :type packages: Option[str, List[str]] |
2127 | + :param fatal: Whether the command's output should be checked and |
2128 | + retried. |
2129 | + :type fatal: bool |
2130 | + :raises: subprocess.CalledProcessError |
2131 | + """ |
2132 | cmd = ['apt-get', '--assume-yes', 'purge'] |
2133 | if isinstance(packages, six.string_types): |
2134 | cmd.append(packages) |
2135 | @@ -268,7 +422,14 @@ |
2136 | |
2137 | |
2138 | def apt_autoremove(purge=True, fatal=False): |
2139 | - """Purge one or more packages.""" |
2140 | + """Purge one or more packages. |
2141 | + :param purge: Whether the ``--purge`` option should be passed on or not. |
2142 | + :type purge: bool |
2143 | + :param fatal: Whether the command's output should be checked and |
2144 | + retried. |
2145 | + :type fatal: bool |
2146 | + :raises: subprocess.CalledProcessError |
2147 | + """ |
2148 | cmd = ['apt-get', '--assume-yes', 'autoremove'] |
2149 | if purge: |
2150 | cmd.append('--purge') |
2151 | @@ -304,7 +465,7 @@ |
2152 | A Radix64 format keyid is also supported for backwards |
2153 | compatibility. In this case Ubuntu keyserver will be |
2154 | queried for a key via HTTPS by its keyid. This method |
2155 | - is less preferrable because https proxy servers may |
2156 | + is less preferable because https proxy servers may |
2157 | require traffic decryption which is equivalent to a |
2158 | man-in-the-middle attack (a proxy server impersonates |
2159 | keyserver TLS certificates and has to be explicitly |
2160 | @@ -481,6 +642,10 @@ |
2161 | with be used. If staging is NOT used then the cloud archive [3] will be |
2162 | added, and the 'ubuntu-cloud-keyring' package will be added for the |
2163 | current distro. |
2164 | + '<openstack-version>': translate to cloud:<release> based on the current |
2165 | + distro version (i.e. for 'ussuri' this will either be 'bionic-ussuri' or |
2166 | + 'distro'. |
2167 | + '<openstack-version>/proposed': as above, but for proposed. |
2168 | |
2169 | Otherwise the source is not recognised and this is logged to the juju log. |
2170 | However, no error is raised, unless sys_error_on_exit is True. |
2171 | @@ -499,7 +664,7 @@ |
2172 | id may also be used, but be aware that only insecure protocols are |
2173 | available to retrieve the actual public key from a public keyserver |
2174 | placing your Juju environment at risk. ppa and cloud archive keys |
2175 | - are securely added automtically, so sould not be provided. |
2176 | + are securely added automatically, so should not be provided. |
2177 | |
2178 | @param fail_invalid: (boolean) if True, then the function raises a |
2179 | SourceConfigError is there is no matching installation source. |
2180 | @@ -507,6 +672,12 @@ |
2181 | @raises SourceConfigError() if for cloud:<pocket>, the <pocket> is not a |
2182 | valid pocket in CLOUD_ARCHIVE_POCKETS |
2183 | """ |
2184 | + # extract the OpenStack versions from the CLOUD_ARCHIVE_POCKETS; can't use |
2185 | + # the list in contrib.openstack.utils as it might not be included in |
2186 | + # classic charms and would break everything. Having OpenStack specific |
2187 | + # code in this file is a bit of an antipattern, anyway. |
2188 | + os_versions_regex = "({})".format("|".join(OPENSTACK_RELEASES)) |
2189 | + |
2190 | _mapping = OrderedDict([ |
2191 | (r"^distro$", lambda: None), # This is a NOP |
2192 | (r"^(?:proposed|distro-proposed)$", _add_proposed), |
2193 | @@ -516,6 +687,9 @@ |
2194 | (r"^cloud:(.*)-(.*)$", _add_cloud_distro_check), |
2195 | (r"^cloud:(.*)$", _add_cloud_pocket), |
2196 | (r"^snap:.*-(.*)-(.*)$", _add_cloud_distro_check), |
2197 | + (r"^{}\/proposed$".format(os_versions_regex), |
2198 | + _add_bare_openstack_proposed), |
2199 | + (r"^{}$".format(os_versions_regex), _add_bare_openstack), |
2200 | ]) |
2201 | if source is None: |
2202 | source = '' |
2203 | @@ -547,7 +721,7 @@ |
2204 | Uses get_distrib_codename to determine the correct stanza for |
2205 | the deb line. |
2206 | |
2207 | - For intel architecutres PROPOSED_POCKET is used for the release, but for |
2208 | + For Intel architectures PROPOSED_POCKET is used for the release, but for |
2209 | other architectures PROPOSED_PORTS_POCKET is used for the release. |
2210 | """ |
2211 | release = get_distrib_codename() |
2212 | @@ -568,11 +742,9 @@ |
2213 | if '{series}' in spec: |
2214 | series = get_distrib_codename() |
2215 | spec = spec.replace('{series}', series) |
2216 | - # software-properties package for bionic properly reacts to proxy settings |
2217 | - # passed as environment variables (See lp:1433761). This is not the case |
2218 | - # LTS and non-LTS releases below bionic. |
2219 | _run_with_retries(['add-apt-repository', '--yes', spec], |
2220 | - cmd_env=env_proxy_settings(['https'])) |
2221 | + cmd_env=env_proxy_settings(['https', 'http', 'no_proxy']) |
2222 | + ) |
2223 | |
2224 | |
2225 | def _add_cloud_pocket(pocket): |
2226 | @@ -648,25 +820,102 @@ |
2227 | 'version ({})'.format(release, os_release, ubuntu_rel)) |
2228 | |
2229 | |
2230 | +def _add_bare_openstack(openstack_release): |
2231 | + """Add cloud or distro based on the release given. |
2232 | + |
2233 | + The spec given is, say, 'ussuri', but this could apply cloud:bionic-ussuri |
2234 | + or 'distro' depending on whether the ubuntu release is bionic or focal. |
2235 | + |
2236 | + :param openstack_release: the OpenStack codename to determine the release |
2237 | + for. |
2238 | + :type openstack_release: str |
2239 | + :raises: SourceConfigError |
2240 | + """ |
2241 | + # TODO(ajkavanagh) - surely this means we should be removing cloud archives |
2242 | + # if they exist? |
2243 | + __add_bare_helper(openstack_release, "{}-{}", lambda: None) |
2244 | + |
2245 | + |
2246 | +def _add_bare_openstack_proposed(openstack_release): |
2247 | + """Add cloud of distro but with proposed. |
2248 | + |
2249 | + The spec given is, say, 'ussuri' but this could apply |
2250 | + cloud:bionic-ussuri/proposed or 'distro/proposed' depending on whether the |
2251 | + ubuntu release is bionic or focal. |
2252 | + |
2253 | + :param openstack_release: the OpenStack codename to determine the release |
2254 | + for. |
2255 | + :type openstack_release: str |
2256 | + :raises: SourceConfigError |
2257 | + """ |
2258 | + __add_bare_helper(openstack_release, "{}-{}/proposed", _add_proposed) |
2259 | + |
2260 | + |
2261 | +def __add_bare_helper(openstack_release, pocket_format, final_function): |
2262 | + """Helper for _add_bare_openstack[_proposed] |
2263 | + |
2264 | + The bulk of the work between the two functions is exactly the same except |
2265 | + for the pocket format and the function that is run if it's the distro |
2266 | + version. |
2267 | + |
2268 | + :param openstack_release: the OpenStack codename. e.g. ussuri |
2269 | + :type openstack_release: str |
2270 | + :param pocket_format: the pocket formatter string to construct a pocket str |
2271 | + from the openstack_release and the current ubuntu version. |
2272 | + :type pocket_format: str |
2273 | + :param final_function: the function to call if it is the distro version. |
2274 | + :type final_function: Callable |
2275 | + :raises SourceConfigError on error |
2276 | + """ |
2277 | + ubuntu_version = get_distrib_codename() |
2278 | + possible_pocket = pocket_format.format(ubuntu_version, openstack_release) |
2279 | + if possible_pocket in CLOUD_ARCHIVE_POCKETS: |
2280 | + _add_cloud_pocket(possible_pocket) |
2281 | + return |
2282 | + # Otherwise it's almost certainly the distro version; verify that it |
2283 | + # exists. |
2284 | + try: |
2285 | + assert UBUNTU_OPENSTACK_RELEASE[ubuntu_version] == openstack_release |
2286 | + except KeyError: |
2287 | + raise SourceConfigError( |
2288 | + "Invalid ubuntu version {} isn't known to this library" |
2289 | + .format(ubuntu_version)) |
2290 | + except AssertionError: |
2291 | + raise SourceConfigError( |
2292 | + 'Invalid OpenStack release specified: {} for Ubuntu version {}' |
2293 | + .format(openstack_release, ubuntu_version)) |
2294 | + final_function() |
2295 | + |
2296 | + |
2297 | def _run_with_retries(cmd, max_retries=CMD_RETRY_COUNT, retry_exitcodes=(1,), |
2298 | - retry_message="", cmd_env=None): |
2299 | + retry_message="", cmd_env=None, quiet=False): |
2300 | """Run a command and retry until success or max_retries is reached. |
2301 | |
2302 | - :param: cmd: str: The apt command to run. |
2303 | - :param: max_retries: int: The number of retries to attempt on a fatal |
2304 | - command. Defaults to CMD_RETRY_COUNT. |
2305 | - :param: retry_exitcodes: tuple: Optional additional exit codes to retry. |
2306 | - Defaults to retry on exit code 1. |
2307 | - :param: retry_message: str: Optional log prefix emitted during retries. |
2308 | - :param: cmd_env: dict: Environment variables to add to the command run. |
2309 | + :param cmd: The apt command to run. |
2310 | + :type cmd: str |
2311 | + :param max_retries: The number of retries to attempt on a fatal |
2312 | + command. Defaults to CMD_RETRY_COUNT. |
2313 | + :type max_retries: int |
2314 | + :param retry_exitcodes: Optional additional exit codes to retry. |
2315 | + Defaults to retry on exit code 1. |
2316 | + :type retry_exitcodes: tuple |
2317 | + :param retry_message: Optional log prefix emitted during retries. |
2318 | + :type retry_message: str |
2319 | + :param: cmd_env: Environment variables to add to the command run. |
2320 | + :type cmd_env: Option[None, Dict[str, str]] |
2321 | + :param quiet: if True, silence the output of the command from stdout and |
2322 | + stderr |
2323 | + :type quiet: bool |
2324 | """ |
2325 | + env = get_apt_dpkg_env() |
2326 | + if cmd_env: |
2327 | + env.update(cmd_env) |
2328 | |
2329 | - env = None |
2330 | kwargs = {} |
2331 | - if cmd_env: |
2332 | - env = os.environ.copy() |
2333 | - env.update(cmd_env) |
2334 | - kwargs['env'] = env |
2335 | + if quiet: |
2336 | + devnull = os.devnull if six.PY2 else subprocess.DEVNULL |
2337 | + kwargs['stdout'] = devnull |
2338 | + kwargs['stderr'] = devnull |
2339 | |
2340 | if not retry_message: |
2341 | retry_message = "Failed executing '{}'".format(" ".join(cmd)) |
2342 | @@ -678,8 +927,7 @@ |
2343 | retry_results = (None,) + retry_exitcodes |
2344 | while result in retry_results: |
2345 | try: |
2346 | - # result = subprocess.check_call(cmd, env=env) |
2347 | - result = subprocess.check_call(cmd, **kwargs) |
2348 | + result = subprocess.check_call(cmd, env=env, **kwargs) |
2349 | except subprocess.CalledProcessError as e: |
2350 | retry_count = retry_count + 1 |
2351 | if retry_count > max_retries: |
2352 | @@ -689,25 +937,30 @@ |
2353 | time.sleep(CMD_RETRY_DELAY) |
2354 | |
2355 | |
2356 | -def _run_apt_command(cmd, fatal=False): |
2357 | +def _run_apt_command(cmd, fatal=False, quiet=False): |
2358 | """Run an apt command with optional retries. |
2359 | |
2360 | - :param: cmd: str: The apt command to run. |
2361 | - :param: fatal: bool: Whether the command's output should be checked and |
2362 | - retried. |
2363 | + :param cmd: The apt command to run. |
2364 | + :type cmd: str |
2365 | + :param fatal: Whether the command's output should be checked and |
2366 | + retried. |
2367 | + :type fatal: bool |
2368 | + :param quiet: if True, silence the output of the command from stdout and |
2369 | + stderr |
2370 | + :type quiet: bool |
2371 | """ |
2372 | - # Provide DEBIAN_FRONTEND=noninteractive if not present in the environment. |
2373 | - cmd_env = { |
2374 | - 'DEBIAN_FRONTEND': os.environ.get('DEBIAN_FRONTEND', 'noninteractive')} |
2375 | - |
2376 | if fatal: |
2377 | _run_with_retries( |
2378 | - cmd, cmd_env=cmd_env, retry_exitcodes=(1, APT_NO_LOCK,), |
2379 | - retry_message="Couldn't acquire DPKG lock") |
2380 | + cmd, retry_exitcodes=(1, APT_NO_LOCK,), |
2381 | + retry_message="Couldn't acquire DPKG lock", |
2382 | + quiet=quiet) |
2383 | else: |
2384 | - env = os.environ.copy() |
2385 | - env.update(cmd_env) |
2386 | - subprocess.call(cmd, env=env) |
2387 | + kwargs = {} |
2388 | + if quiet: |
2389 | + devnull = os.devnull if six.PY2 else subprocess.DEVNULL |
2390 | + kwargs['stdout'] = devnull |
2391 | + kwargs['stderr'] = devnull |
2392 | + subprocess.call(cmd, env=get_apt_dpkg_env(), **kwargs) |
2393 | |
2394 | |
2395 | def get_upstream_version(package): |
2396 | @@ -715,7 +968,6 @@ |
2397 | |
2398 | @returns None (if not installed) or the upstream version |
2399 | """ |
2400 | - import apt_pkg |
2401 | cache = apt_cache() |
2402 | try: |
2403 | pkg = cache[package] |
2404 | @@ -727,4 +979,34 @@ |
2405 | # package is known, but no version is currently installed. |
2406 | return None |
2407 | |
2408 | - return apt_pkg.upstream_version(pkg.current_ver.ver_str) |
2409 | + return ubuntu_apt_pkg.upstream_version(pkg.current_ver.ver_str) |
2410 | + |
2411 | + |
2412 | +def get_installed_version(package): |
2413 | + """Determine installed version of a package |
2414 | + |
2415 | + @returns None (if not installed) or the installed version as |
2416 | + Version object |
2417 | + """ |
2418 | + cache = apt_cache() |
2419 | + dpkg_result = cache._dpkg_list([package]).get(package, {}) |
2420 | + current_ver = None |
2421 | + installed_version = dpkg_result.get('version') |
2422 | + |
2423 | + if installed_version: |
2424 | + current_ver = ubuntu_apt_pkg.Version({'ver_str': installed_version}) |
2425 | + return current_ver |
2426 | + |
2427 | + |
2428 | +def get_apt_dpkg_env(): |
2429 | + """Get environment suitable for execution of APT and DPKG tools. |
2430 | + |
2431 | + We keep this in a helper function instead of in a global constant to |
2432 | + avoid execution on import of the library. |
2433 | + :returns: Environment suitable for execution of APT and DPKG tools. |
2434 | + :rtype: Dict[str, str] |
2435 | + """ |
2436 | + # The fallback is used in the event of ``/etc/environment`` not containing |
2437 | + # avalid PATH variable. |
2438 | + return {'DEBIAN_FRONTEND': 'noninteractive', |
2439 | + 'PATH': get_system_env('PATH', '/usr/sbin:/usr/bin:/sbin:/bin')} |
2440 | |
2441 | === added file 'charmhelpers/fetch/ubuntu_apt_pkg.py' |
2442 | --- charmhelpers/fetch/ubuntu_apt_pkg.py 1970-01-01 00:00:00 +0000 |
2443 | +++ charmhelpers/fetch/ubuntu_apt_pkg.py 2021-11-10 05:36:20 +0000 |
2444 | @@ -0,0 +1,312 @@ |
2445 | +# Copyright 2019-2021 Canonical Ltd |
2446 | +# |
2447 | +# Licensed under the Apache License, Version 2.0 (the "License"); |
2448 | +# you may not use this file except in compliance with the License. |
2449 | +# You may obtain a copy of the License at |
2450 | +# |
2451 | +# http://www.apache.org/licenses/LICENSE-2.0 |
2452 | +# |
2453 | +# Unless required by applicable law or agreed to in writing, software |
2454 | +# distributed under the License is distributed on an "AS IS" BASIS, |
2455 | +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
2456 | +# See the License for the specific language governing permissions and |
2457 | +# limitations under the License. |
2458 | + |
2459 | +"""Provide a subset of the ``python-apt`` module API. |
2460 | + |
2461 | +Data collection is done through subprocess calls to ``apt-cache`` and |
2462 | +``dpkg-query`` commands. |
2463 | + |
2464 | +The main purpose for this module is to avoid dependency on the |
2465 | +``python-apt`` python module. |
2466 | + |
2467 | +The indicated python module is a wrapper around the ``apt`` C++ library |
2468 | +which is tightly connected to the version of the distribution it was |
2469 | +shipped on. It is not developed in a backward/forward compatible manner. |
2470 | + |
2471 | +This in turn makes it incredibly hard to distribute as a wheel for a piece |
2472 | +of python software that supports a span of distro releases [0][1]. |
2473 | + |
2474 | +Upstream feedback like [2] does not give confidence in this ever changing, |
2475 | +so with this we get rid of the dependency. |
2476 | + |
2477 | +0: https://github.com/juju-solutions/layer-basic/pull/135 |
2478 | +1: https://bugs.launchpad.net/charm-octavia/+bug/1824112 |
2479 | +2: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=845330#10 |
2480 | +""" |
2481 | + |
2482 | +import locale |
2483 | +import os |
2484 | +import subprocess |
2485 | +import sys |
2486 | + |
2487 | + |
2488 | +class _container(dict): |
2489 | + """Simple container for attributes.""" |
2490 | + __getattr__ = dict.__getitem__ |
2491 | + __setattr__ = dict.__setitem__ |
2492 | + |
2493 | + |
2494 | +class Package(_container): |
2495 | + """Simple container for package attributes.""" |
2496 | + |
2497 | + |
2498 | +class Version(_container): |
2499 | + """Simple container for version attributes.""" |
2500 | + |
2501 | + |
2502 | +class Cache(object): |
2503 | + """Simulation of ``apt_pkg`` Cache object.""" |
2504 | + def __init__(self, progress=None): |
2505 | + pass |
2506 | + |
2507 | + def __contains__(self, package): |
2508 | + try: |
2509 | + pkg = self.__getitem__(package) |
2510 | + return pkg is not None |
2511 | + except KeyError: |
2512 | + return False |
2513 | + |
2514 | + def __getitem__(self, package): |
2515 | + """Get information about a package from apt and dpkg databases. |
2516 | + |
2517 | + :param package: Name of package |
2518 | + :type package: str |
2519 | + :returns: Package object |
2520 | + :rtype: object |
2521 | + :raises: KeyError, subprocess.CalledProcessError |
2522 | + """ |
2523 | + apt_result = self._apt_cache_show([package])[package] |
2524 | + apt_result['name'] = apt_result.pop('package') |
2525 | + pkg = Package(apt_result) |
2526 | + dpkg_result = self._dpkg_list([package]).get(package, {}) |
2527 | + current_ver = None |
2528 | + installed_version = dpkg_result.get('version') |
2529 | + if installed_version: |
2530 | + current_ver = Version({'ver_str': installed_version}) |
2531 | + pkg.current_ver = current_ver |
2532 | + pkg.architecture = dpkg_result.get('architecture') |
2533 | + return pkg |
2534 | + |
2535 | + def _dpkg_list(self, packages): |
2536 | + """Get data from system dpkg database for package. |
2537 | + |
2538 | + :param packages: Packages to get data from |
2539 | + :type packages: List[str] |
2540 | + :returns: Structured data about installed packages, keys like |
2541 | + ``dpkg-query --list`` |
2542 | + :rtype: dict |
2543 | + :raises: subprocess.CalledProcessError |
2544 | + """ |
2545 | + pkgs = {} |
2546 | + cmd = ['dpkg-query', '--list'] |
2547 | + cmd.extend(packages) |
2548 | + if locale.getlocale() == (None, None): |
2549 | + # subprocess calls out to locale.getpreferredencoding(False) to |
2550 | + # determine encoding. Workaround for Trusty where the |
2551 | + # environment appears to not be set up correctly. |
2552 | + locale.setlocale(locale.LC_ALL, 'en_US.UTF-8') |
2553 | + try: |
2554 | + output = subprocess.check_output(cmd, |
2555 | + stderr=subprocess.STDOUT, |
2556 | + universal_newlines=True) |
2557 | + except subprocess.CalledProcessError as cp: |
2558 | + # ``dpkg-query`` may return error and at the same time have |
2559 | + # produced useful output, for example when asked for multiple |
2560 | + # packages where some are not installed |
2561 | + if cp.returncode != 1: |
2562 | + raise |
2563 | + output = cp.output |
2564 | + headings = [] |
2565 | + for line in output.splitlines(): |
2566 | + if line.startswith('||/'): |
2567 | + headings = line.split() |
2568 | + headings.pop(0) |
2569 | + continue |
2570 | + elif (line.startswith('|') or line.startswith('+') or |
2571 | + line.startswith('dpkg-query:')): |
2572 | + continue |
2573 | + else: |
2574 | + data = line.split(None, 4) |
2575 | + status = data.pop(0) |
2576 | + if status not in ('ii', 'hi'): |
2577 | + continue |
2578 | + pkg = {} |
2579 | + pkg.update({k.lower(): v for k, v in zip(headings, data)}) |
2580 | + if 'name' in pkg: |
2581 | + pkgs.update({pkg['name']: pkg}) |
2582 | + return pkgs |
2583 | + |
2584 | + def _apt_cache_show(self, packages): |
2585 | + """Get data from system apt cache for package. |
2586 | + |
2587 | + :param packages: Packages to get data from |
2588 | + :type packages: List[str] |
2589 | + :returns: Structured data about package, keys like |
2590 | + ``apt-cache show`` |
2591 | + :rtype: dict |
2592 | + :raises: subprocess.CalledProcessError |
2593 | + """ |
2594 | + pkgs = {} |
2595 | + cmd = ['apt-cache', 'show', '--no-all-versions'] |
2596 | + cmd.extend(packages) |
2597 | + if locale.getlocale() == (None, None): |
2598 | + # subprocess calls out to locale.getpreferredencoding(False) to |
2599 | + # determine encoding. Workaround for Trusty where the |
2600 | + # environment appears to not be set up correctly. |
2601 | + locale.setlocale(locale.LC_ALL, 'en_US.UTF-8') |
2602 | + try: |
2603 | + output = subprocess.check_output(cmd, |
2604 | + stderr=subprocess.STDOUT, |
2605 | + universal_newlines=True) |
2606 | + previous = None |
2607 | + pkg = {} |
2608 | + for line in output.splitlines(): |
2609 | + if not line: |
2610 | + if 'package' in pkg: |
2611 | + pkgs.update({pkg['package']: pkg}) |
2612 | + pkg = {} |
2613 | + continue |
2614 | + if line.startswith(' '): |
2615 | + if previous and previous in pkg: |
2616 | + pkg[previous] += os.linesep + line.lstrip() |
2617 | + continue |
2618 | + if ':' in line: |
2619 | + kv = line.split(':', 1) |
2620 | + key = kv[0].lower() |
2621 | + if key == 'n': |
2622 | + continue |
2623 | + previous = key |
2624 | + pkg.update({key: kv[1].lstrip()}) |
2625 | + except subprocess.CalledProcessError as cp: |
2626 | + # ``apt-cache`` returns 100 if none of the packages asked for |
2627 | + # exist in the apt cache. |
2628 | + if cp.returncode != 100: |
2629 | + raise |
2630 | + return pkgs |
2631 | + |
2632 | + |
2633 | +class Config(_container): |
2634 | + def __init__(self): |
2635 | + super(Config, self).__init__(self._populate()) |
2636 | + |
2637 | + def _populate(self): |
2638 | + cfgs = {} |
2639 | + cmd = ['apt-config', 'dump'] |
2640 | + output = subprocess.check_output(cmd, |
2641 | + stderr=subprocess.STDOUT, |
2642 | + universal_newlines=True) |
2643 | + for line in output.splitlines(): |
2644 | + if not line.startswith("CommandLine"): |
2645 | + k, v = line.split(" ", 1) |
2646 | + cfgs[k] = v.strip(";").strip("\"") |
2647 | + |
2648 | + return cfgs |
2649 | + |
2650 | + |
2651 | +# Backwards compatibility with old apt_pkg module |
2652 | +sys.modules[__name__].config = Config() |
2653 | + |
2654 | + |
2655 | +def init(): |
2656 | + """Compatibility shim that does nothing.""" |
2657 | + pass |
2658 | + |
2659 | + |
2660 | +def upstream_version(version): |
2661 | + """Extracts upstream version from a version string. |
2662 | + |
2663 | + Upstream reference: https://salsa.debian.org/apt-team/apt/blob/master/ |
2664 | + apt-pkg/deb/debversion.cc#L259 |
2665 | + |
2666 | + :param version: Version string |
2667 | + :type version: str |
2668 | + :returns: Upstream version |
2669 | + :rtype: str |
2670 | + """ |
2671 | + if version: |
2672 | + version = version.split(':')[-1] |
2673 | + version = version.split('-')[0] |
2674 | + return version |
2675 | + |
2676 | + |
2677 | +def version_compare(a, b): |
2678 | + """Compare the given versions. |
2679 | + |
2680 | + Call out to ``dpkg`` to make sure the code doing the comparison is |
2681 | + compatible with what the ``apt`` library would do. Mimic the return |
2682 | + values. |
2683 | + |
2684 | + Upstream reference: |
2685 | + https://apt-team.pages.debian.net/python-apt/library/apt_pkg.html |
2686 | + ?highlight=version_compare#apt_pkg.version_compare |
2687 | + |
2688 | + :param a: version string |
2689 | + :type a: str |
2690 | + :param b: version string |
2691 | + :type b: str |
2692 | + :returns: >0 if ``a`` is greater than ``b``, 0 if a equals b, |
2693 | + <0 if ``a`` is smaller than ``b`` |
2694 | + :rtype: int |
2695 | + :raises: subprocess.CalledProcessError, RuntimeError |
2696 | + """ |
2697 | + for op in ('gt', 1), ('eq', 0), ('lt', -1): |
2698 | + try: |
2699 | + subprocess.check_call(['dpkg', '--compare-versions', |
2700 | + a, op[0], b], |
2701 | + stderr=subprocess.STDOUT, |
2702 | + universal_newlines=True) |
2703 | + return op[1] |
2704 | + except subprocess.CalledProcessError as cp: |
2705 | + if cp.returncode == 1: |
2706 | + continue |
2707 | + raise |
2708 | + else: |
2709 | + raise RuntimeError('Unable to compare "{}" and "{}", according to ' |
2710 | + 'our logic they are neither greater, equal nor ' |
2711 | + 'less than each other.'.format(a, b)) |
2712 | + |
2713 | + |
2714 | +class PkgVersion(): |
2715 | + """Allow package versions to be compared. |
2716 | + |
2717 | + For example:: |
2718 | + |
2719 | + >>> import charmhelpers.fetch as fetch |
2720 | + >>> (fetch.apt_pkg.PkgVersion('2:20.4.0') < |
2721 | + ... fetch.apt_pkg.PkgVersion('2:20.5.0')) |
2722 | + True |
2723 | + >>> pkgs = [fetch.apt_pkg.PkgVersion('2:20.4.0'), |
2724 | + ... fetch.apt_pkg.PkgVersion('2:21.4.0'), |
2725 | + ... fetch.apt_pkg.PkgVersion('2:17.4.0')] |
2726 | + >>> pkgs.sort() |
2727 | + >>> pkgs |
2728 | + [2:17.4.0, 2:20.4.0, 2:21.4.0] |
2729 | + """ |
2730 | + |
2731 | + def __init__(self, version): |
2732 | + self.version = version |
2733 | + |
2734 | + def __lt__(self, other): |
2735 | + return version_compare(self.version, other.version) == -1 |
2736 | + |
2737 | + def __le__(self, other): |
2738 | + return self.__lt__(other) or self.__eq__(other) |
2739 | + |
2740 | + def __gt__(self, other): |
2741 | + return version_compare(self.version, other.version) == 1 |
2742 | + |
2743 | + def __ge__(self, other): |
2744 | + return self.__gt__(other) or self.__eq__(other) |
2745 | + |
2746 | + def __eq__(self, other): |
2747 | + return version_compare(self.version, other.version) == 0 |
2748 | + |
2749 | + def __ne__(self, other): |
2750 | + return not self.__eq__(other) |
2751 | + |
2752 | + def __repr__(self): |
2753 | + return self.version |
2754 | + |
2755 | + def __hash__(self): |
2756 | + return hash(repr(self)) |
2757 | |
2758 | === modified file 'charmhelpers/osplatform.py' |
2759 | --- charmhelpers/osplatform.py 2017-03-04 02:42:23 +0000 |
2760 | +++ charmhelpers/osplatform.py 2021-11-10 05:36:20 +0000 |
2761 | @@ -1,4 +1,5 @@ |
2762 | import platform |
2763 | +import os |
2764 | |
2765 | |
2766 | def get_platform(): |
2767 | @@ -9,9 +10,13 @@ |
2768 | This string is used to decide which platform module should be imported. |
2769 | """ |
2770 | # linux_distribution is deprecated and will be removed in Python 3.7 |
2771 | - # Warings *not* disabled, as we certainly need to fix this. |
2772 | - tuple_platform = platform.linux_distribution() |
2773 | - current_platform = tuple_platform[0] |
2774 | + # Warnings *not* disabled, as we certainly need to fix this. |
2775 | + if hasattr(platform, 'linux_distribution'): |
2776 | + tuple_platform = platform.linux_distribution() |
2777 | + current_platform = tuple_platform[0] |
2778 | + else: |
2779 | + current_platform = _get_platform_from_fs() |
2780 | + |
2781 | if "Ubuntu" in current_platform: |
2782 | return "ubuntu" |
2783 | elif "CentOS" in current_platform: |
2784 | @@ -20,6 +25,25 @@ |
2785 | # Stock Python does not detect Ubuntu and instead returns debian. |
2786 | # Or at least it does in some build environments like Travis CI |
2787 | return "ubuntu" |
2788 | + elif "elementary" in current_platform: |
2789 | + # ElementaryOS fails to run tests locally without this. |
2790 | + return "ubuntu" |
2791 | + elif "Pop!_OS" in current_platform: |
2792 | + # Pop!_OS also fails to run tests locally without this. |
2793 | + return "ubuntu" |
2794 | else: |
2795 | raise RuntimeError("This module is not supported on {}." |
2796 | .format(current_platform)) |
2797 | + |
2798 | + |
2799 | +def _get_platform_from_fs(): |
2800 | + """Get Platform from /etc/os-release.""" |
2801 | + with open(os.path.join(os.sep, 'etc', 'os-release')) as fin: |
2802 | + content = dict( |
2803 | + line.split('=', 1) |
2804 | + for line in fin.read().splitlines() |
2805 | + if '=' in line |
2806 | + ) |
2807 | + for k, v in content.items(): |
2808 | + content[k] = v.strip('"') |
2809 | + return content["NAME"] |
2810 | |
2811 | === modified file 'config.yaml' |
2812 | --- config.yaml 2021-10-07 00:38:56 +0000 |
2813 | +++ config.yaml 2021-11-10 05:36:20 +0000 |
2814 | @@ -121,3 +121,19 @@ |
2815 | type: string |
2816 | default: '' |
2817 | description: An unique site name for Landscape deployment |
2818 | + nagios_context: |
2819 | + default: "juju" |
2820 | + type: string |
2821 | + description: | |
2822 | + Used by the nrpe subordinate charms. |
2823 | + A string that will be prepended to instance name to set the host name |
2824 | + in nagios. So for instance the hostname would be something like: |
2825 | + juju-myservice-0 |
2826 | + If you're running multiple environments with the same services in them |
2827 | + this allows you to differentiate between them. |
2828 | + nagios_servicegroups: |
2829 | + default: "" |
2830 | + type: string |
2831 | + description: | |
2832 | + A comma-separated list of nagios servicegroups. |
2833 | + If left empty, the nagios_context will be used as the servicegroup |
2834 | |
2835 | === added file 'hooks/nrpe-external-master-relation-changed' |
2836 | --- hooks/nrpe-external-master-relation-changed 1970-01-01 00:00:00 +0000 |
2837 | +++ hooks/nrpe-external-master-relation-changed 2021-11-10 05:36:20 +0000 |
2838 | @@ -0,0 +1,9 @@ |
2839 | +#!/usr/bin/python |
2840 | +import sys |
2841 | + |
2842 | +from lib.services import ServicesHook |
2843 | + |
2844 | + |
2845 | +if __name__ == "__main__": |
2846 | + hook = ServicesHook() |
2847 | + sys.exit(hook()) |
2848 | |
2849 | === added file 'hooks/nrpe-external-master-relation-joined' |
2850 | --- hooks/nrpe-external-master-relation-joined 1970-01-01 00:00:00 +0000 |
2851 | +++ hooks/nrpe-external-master-relation-joined 2021-11-10 05:36:20 +0000 |
2852 | @@ -0,0 +1,9 @@ |
2853 | +#!/usr/bin/python |
2854 | +import sys |
2855 | + |
2856 | +from lib.services import ServicesHook |
2857 | + |
2858 | + |
2859 | +if __name__ == "__main__": |
2860 | + hook = ServicesHook() |
2861 | + sys.exit(hook()) |
2862 | |
2863 | === added file 'lib/callbacks/nrpe.py' |
2864 | --- lib/callbacks/nrpe.py 1970-01-01 00:00:00 +0000 |
2865 | +++ lib/callbacks/nrpe.py 2021-11-10 05:36:20 +0000 |
2866 | @@ -0,0 +1,51 @@ |
2867 | +from charmhelpers.core import hookenv |
2868 | +from charmhelpers.core.services.base import ManagerCallback |
2869 | +from charmhelpers.contrib.charmsupport import nrpe |
2870 | + |
2871 | + |
2872 | +# services running on all nodes |
2873 | +DEFAULT_SERVICES = ['landscape-api', 'landscape-appserver', |
2874 | + 'landscape-async-frontend', 'landscape-job-handler', |
2875 | + 'landscape-msgserver', 'landscape-pingserver'] |
2876 | + |
2877 | +# services running only on the leader |
2878 | +LEADER_SERVICES = ['landscape-package-search', 'landscape-package-upload'] |
2879 | + |
2880 | + |
2881 | +class ConfigureNRPE(ManagerCallback): |
2882 | + """Configure service checks if nrpe-external-master relation exists""" |
2883 | + |
2884 | + def __init__(self, hookenv=hookenv, nrpe_config=None): |
2885 | + self._hookenv = hookenv |
2886 | + self._unit = self._hookenv.local_unit() |
2887 | + if nrpe_config: |
2888 | + self._nrpe_config = nrpe_config |
2889 | + else: |
2890 | + self._nrpe_config = nrpe.NRPE() |
2891 | + |
2892 | + def __call__(self, manager, service_name, event_name): |
2893 | + self._hookenv.log('Configuring NRPE checks') |
2894 | + if self._hookenv.relations_of_type('nrpe-external-master'): |
2895 | + if self._hookenv.is_leader(): |
2896 | + self._add_checks(DEFAULT_SERVICES + LEADER_SERVICES) |
2897 | + else: |
2898 | + self._add_checks(DEFAULT_SERVICES) |
2899 | + self._remove_checks(LEADER_SERVICES) |
2900 | + else: |
2901 | + self._remove_checks(DEFAULT_SERVICES + LEADER_SERVICES) |
2902 | + self._nrpe_config.write() |
2903 | + |
2904 | + def _add_checks(self, services): |
2905 | + """ Add a service check """ |
2906 | + for service in services: |
2907 | + hookenv.log('Adding nrpe check: %s' % service, hookenv.DEBUG) |
2908 | + self._nrpe_config.add_check( |
2909 | + shortname='%s' % service, |
2910 | + description='process check {%s}' % self._unit, |
2911 | + check_cmd='check_systemd.py %s' % service) |
2912 | + |
2913 | + def _remove_checks(self, services): |
2914 | + """ Remove a service check """ |
2915 | + for service in services: |
2916 | + hookenv.log('Removing nrpe check: %s' % service, hookenv.DEBUG) |
2917 | + self._nrpe_config.remove_check(shortname=service) |
2918 | |
2919 | === added file 'lib/callbacks/tests/test_nrpe.py' |
2920 | --- lib/callbacks/tests/test_nrpe.py 1970-01-01 00:00:00 +0000 |
2921 | +++ lib/callbacks/tests/test_nrpe.py 2021-11-10 05:36:20 +0000 |
2922 | @@ -0,0 +1,36 @@ |
2923 | +from charmhelpers.core.services.base import ServiceManager |
2924 | + |
2925 | +from lib.tests.helpers import HookenvTest |
2926 | +from lib.tests.stubs import NrpeConfigStub |
2927 | +from lib.callbacks.nrpe import ( |
2928 | + ConfigureNRPE, |
2929 | + DEFAULT_SERVICES, |
2930 | + LEADER_SERVICES) |
2931 | + |
2932 | + |
2933 | +class ConfigureNRPETest(HookenvTest): |
2934 | + def setUp(self): |
2935 | + super(ConfigureNRPETest, self).setUp() |
2936 | + self.manager = ServiceManager([]) |
2937 | + self.fake_nrpe = NrpeConfigStub() |
2938 | + self.callback = ConfigureNRPE(hookenv=self.hookenv, |
2939 | + nrpe_config=self.fake_nrpe) |
2940 | + |
2941 | + def test_add_nrpe_check(self): |
2942 | + """Test adding NRPE checks.""" |
2943 | + config = self.hookenv.config() |
2944 | + config["nagios_context"] = "juju" |
2945 | + self.hookenv.relations['nrpe-external-master'] = {"id": "1"} |
2946 | + self.callback(self.manager, None, None) |
2947 | + nrpe_checks = self.fake_nrpe.get_nrpe_checks() |
2948 | + for svc in DEFAULT_SERVICES: |
2949 | + self.assertIn(svc, nrpe_checks) |
2950 | + for svc in LEADER_SERVICES: |
2951 | + self.assertIn(svc, nrpe_checks) |
2952 | + |
2953 | + def test_remove_nrpe_check(self): |
2954 | + config = self.hookenv.config() |
2955 | + config["nagios_context"] = "juju" |
2956 | + self.callback(self.manager, None, None) |
2957 | + nrpe_checks = self.fake_nrpe.get_nrpe_checks() |
2958 | + self.assertTrue(len(nrpe_checks) == 0) |
2959 | |
2960 | === modified file 'lib/services.py' |
2961 | --- lib/services.py 2021-10-29 00:55:43 +0000 |
2962 | +++ lib/services.py 2021-11-10 05:36:20 +0000 |
2963 | @@ -21,6 +21,7 @@ |
2964 | from lib.callbacks.filesystem import ( |
2965 | EnsureConfigDir, WriteCustomSSLCertificate, WriteLicenseFile) |
2966 | from lib.callbacks.apt import SetAPTSources |
2967 | +from lib.callbacks.nrpe import ConfigureNRPE |
2968 | |
2969 | |
2970 | class ServicesHook(Hook): |
2971 | @@ -32,7 +33,7 @@ |
2972 | """ |
2973 | def __init__(self, hookenv=hookenv, host=host, |
2974 | subprocess=subprocess, paths=default_paths, fetch=fetch, |
2975 | - psutil=psutil): |
2976 | + psutil=psutil, nrpe_config=None): |
2977 | super(ServicesHook, self).__init__(hookenv=hookenv) |
2978 | self._hookenv = hookenv |
2979 | self._host = host |
2980 | @@ -40,6 +41,7 @@ |
2981 | self._psutil = psutil |
2982 | self._subprocess = subprocess |
2983 | self._fetch = fetch |
2984 | + self._nrpe_config = nrpe_config |
2985 | |
2986 | def _run(self): |
2987 | |
2988 | @@ -88,6 +90,8 @@ |
2989 | WriteLicenseFile(host=self._host, paths=self._paths), |
2990 | ConfigureSMTP( |
2991 | hookenv=self._hookenv, subprocess=self._subprocess), |
2992 | + ConfigureNRPE(hookenv=self._hookenv, |
2993 | + nrpe_config=self._nrpe_config), |
2994 | ], |
2995 | "start": LSCtl(subprocess=self._subprocess, hookenv=self._hookenv), |
2996 | }]) |
2997 | |
2998 | === modified file 'lib/tests/stubs.py' |
2999 | --- lib/tests/stubs.py 2019-07-15 20:01:06 +0000 |
3000 | +++ lib/tests/stubs.py 2021-11-10 05:36:20 +0000 |
3001 | @@ -33,6 +33,9 @@ |
3002 | def config(self): |
3003 | return self._config |
3004 | |
3005 | + def relations_of_type(self, reltype): |
3006 | + return self.relations.get(reltype, None) |
3007 | + |
3008 | def log(self, message, level=None): |
3009 | self.messages.append((message, level)) |
3010 | |
3011 | @@ -264,3 +267,24 @@ |
3012 | |
3013 | def virtual_memory(self): |
3014 | return PsutilUsageStub(self._physical_memory) |
3015 | + |
3016 | + |
3017 | +class NrpeConfigStub(object): |
3018 | + def __init__(self): |
3019 | + self._nrpe_checks = {} |
3020 | + |
3021 | + def write(self): |
3022 | + pass |
3023 | + |
3024 | + def add_check(self, shortname, description, check_cmd): |
3025 | + self._nrpe_checks[shortname] = { |
3026 | + "description": description, |
3027 | + "command": check_cmd, |
3028 | + } |
3029 | + |
3030 | + def remove_check(self, shortname): |
3031 | + if self._nrpe_checks.get(shortname): |
3032 | + del self._nrpe_checks[shortname] |
3033 | + |
3034 | + def get_nrpe_checks(self): |
3035 | + return self._nrpe_checks |
3036 | |
3037 | === modified file 'lib/tests/test_services.py' |
3038 | --- lib/tests/test_services.py 2021-04-13 19:03:22 +0000 |
3039 | +++ lib/tests/test_services.py 2021-11-10 05:36:20 +0000 |
3040 | @@ -5,7 +5,12 @@ |
3041 | from charmhelpers.core import templating |
3042 | |
3043 | from lib.tests.helpers import HookenvTest |
3044 | -from lib.tests.stubs import HostStub, PsutilStub, SubprocessStub, FetchStub |
3045 | +from lib.tests.stubs import ( |
3046 | + HostStub, |
3047 | + PsutilStub, |
3048 | + SubprocessStub, |
3049 | + FetchStub, |
3050 | + NrpeConfigStub) |
3051 | from lib.tests.sample import ( |
3052 | SAMPLE_DB_UNIT_DATA, SAMPLE_LEADER_DATA, SAMPLE_AMQP_UNIT_DATA, |
3053 | SAMPLE_CONFIG_LICENSE_DATA, SAMPLE_CONFIG_OPENID_DATA, |
3054 | @@ -34,9 +39,11 @@ |
3055 | self.root_dir = self.useFixture(RootDir()) |
3056 | self.fetch = FetchStub(self.hookenv.config) |
3057 | self.psutil = PsutilStub(num_cpus=2, physical_memory=1*1024**3) |
3058 | + self.nrpe_config = NrpeConfigStub() |
3059 | self.hook = ServicesHook( |
3060 | hookenv=self.hookenv, host=self.host, subprocess=self.subprocess, |
3061 | - paths=self.paths, fetch=self.fetch, psutil=self.psutil) |
3062 | + paths=self.paths, fetch=self.fetch, psutil=self.psutil, |
3063 | + nrpe_config=self.nrpe_config) |
3064 | |
3065 | # XXX Monkey patch the templating API, charmhelpers doesn't sport |
3066 | # any dependency injection here as well. |
3067 | |
3068 | === modified file 'metadata.yaml' |
3069 | --- metadata.yaml 2021-10-05 08:24:58 +0000 |
3070 | +++ metadata.yaml 2021-11-10 05:36:20 +0000 |
3071 | @@ -21,6 +21,9 @@ |
3072 | hosted: |
3073 | interface: landscape-hosted |
3074 | scope: container |
3075 | + nrpe-external-master: |
3076 | + interface: nrpe-external-master |
3077 | + scope: container |
3078 | series: |
3079 | - bionic |
3080 | - xenial |
This merge proposal is to supersede this patch
https:/ /code.launchpad .net/~stephanpa mpel/landscape- charm/systemd- monitoring/ +merge/ 411083