Merge lp:~chad.smith/landscape-charm/add-apt-repository-retries into lp:landscape-charm

Proposed by Chad Smith
Status: Merged
Approved by: Chad Smith
Approved revision: 386
Merged at revision: 383
Proposed branch: lp:~chad.smith/landscape-charm/add-apt-repository-retries
Merge into: lp:landscape-charm
Diff against target: 3080 lines (+1633/-771)
34 files modified
charm-helpers.yaml (+1/-0)
charmhelpers/__init__.py (+11/-13)
charmhelpers/contrib/__init__.py (+11/-13)
charmhelpers/contrib/hahelpers/__init__.py (+11/-13)
charmhelpers/contrib/hahelpers/apache.py (+30/-17)
charmhelpers/contrib/hahelpers/cluster.py (+70/-23)
charmhelpers/core/__init__.py (+11/-13)
charmhelpers/core/decorators.py (+11/-13)
charmhelpers/core/files.py (+11/-13)
charmhelpers/core/fstab.py (+11/-13)
charmhelpers/core/hookenv.py (+73/-14)
charmhelpers/core/host.py (+327/-126)
charmhelpers/core/host_factory/centos.py (+56/-0)
charmhelpers/core/host_factory/ubuntu.py (+56/-0)
charmhelpers/core/hugepage.py (+11/-13)
charmhelpers/core/kernel.py (+34/-30)
charmhelpers/core/kernel_factory/centos.py (+17/-0)
charmhelpers/core/kernel_factory/ubuntu.py (+13/-0)
charmhelpers/core/services/__init__.py (+11/-13)
charmhelpers/core/services/base.py (+11/-13)
charmhelpers/core/services/helpers.py (+11/-13)
charmhelpers/core/strutils.py (+11/-13)
charmhelpers/core/sysctl.py (+11/-13)
charmhelpers/core/templating.py (+19/-16)
charmhelpers/core/unitdata.py (+11/-14)
charmhelpers/fetch/__init__.py (+42/-309)
charmhelpers/fetch/archiveurl.py (+11/-13)
charmhelpers/fetch/bzrurl.py (+31/-23)
charmhelpers/fetch/centos.py (+171/-0)
charmhelpers/fetch/giturl.py (+15/-16)
charmhelpers/fetch/snap.py (+122/-0)
charmhelpers/fetch/ubuntu.py (+364/-0)
charmhelpers/osplatform.py (+25/-0)
lib/apt.py (+2/-1)
To merge this branch: bzr merge lp:~chad.smith/landscape-charm/add-apt-repository-retries
Reviewer Review Type Date Requested Status
Landscape Builder test results Approve
Eric Snow (community) Approve
Review via email: mp+318962@code.launchpad.net

Commit message

Sync of charmhelpers to get updated fetch.add_source which handles 30 retries upon exit 1

Add charmhelpers.osplatform dependency to charm-helpers.yaml

Description of the change

Sync of charmhelpers to get updated fetch.add_source which handles 30 retries upon exit 1. Generally this exit code is returned upon network failure like bug https://bugs.launchpad.net/charm-helpers/+bug/1370933.

Retry fixes are in charmhelpers/fetch/ubuntu.py

To post a comment you must log in.
Revision history for this message
Landscape Builder (landscape-builder) :
review: Abstain (executing tests)
Revision history for this message
Landscape Builder (landscape-builder) wrote :
review: Needs Fixing (test results)
Revision history for this message
Eric Snow (ericsnowcurrently) wrote :

As this doesn't introduce any changes, LGTM. :)

review: Approve
384. By Chad Smith

add charmhelpers.osplatform dependency for charmhelpers.host get_platform

385. By Chad Smith

make sync pulling in osplatform

Revision history for this message
Landscape Builder (landscape-builder) :
review: Abstain (executing tests)
Revision history for this message
Landscape Builder (landscape-builder) wrote :
review: Needs Fixing (test results)
386. By Chad Smith

lint me

Revision history for this message
Landscape Builder (landscape-builder) :
review: Abstain (executing tests)
Revision history for this message
Landscape Builder (landscape-builder) wrote :
review: Approve (test results)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'charm-helpers.yaml'
--- charm-helpers.yaml 2015-05-07 10:26:45 +0000
+++ charm-helpers.yaml 2017-03-04 02:50:20 +0000
@@ -4,4 +4,5 @@
4 - __init__4 - __init__
5 - core5 - core
6 - fetch6 - fetch
7 - osplatform
7 - contrib.hahelpers8 - contrib.hahelpers
89
=== modified file 'charmhelpers/__init__.py'
--- charmhelpers/__init__.py 2015-01-28 08:59:02 +0000
+++ charmhelpers/__init__.py 2017-03-04 02:50:20 +0000
@@ -1,18 +1,16 @@
1# Copyright 2014-2015 Canonical Limited.1# Copyright 2014-2015 Canonical Limited.
2#2#
3# This file is part of charm-helpers.3# Licensed under the Apache License, Version 2.0 (the "License");
4#4# you may not use this file except in compliance with the License.
5# charm-helpers is free software: you can redistribute it and/or modify5# You may obtain a copy of the License at
6# it under the terms of the GNU Lesser General Public License version 3 as6#
7# published by the Free Software Foundation.7# http://www.apache.org/licenses/LICENSE-2.0
8#8#
9# charm-helpers is distributed in the hope that it will be useful,9# Unless required by applicable law or agreed to in writing, software
10# but WITHOUT ANY WARRANTY; without even the implied warranty of10# distributed under the License is distributed on an "AS IS" BASIS,
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# GNU Lesser General Public License for more details.12# See the License for the specific language governing permissions and
13#13# limitations under the License.
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1614
17# Bootstrap charm-helpers, installing its dependencies if necessary using15# Bootstrap charm-helpers, installing its dependencies if necessary using
18# only standard libraries.16# only standard libraries.
1917
=== modified file 'charmhelpers/contrib/__init__.py'
--- charmhelpers/contrib/__init__.py 2015-01-30 11:16:09 +0000
+++ charmhelpers/contrib/__init__.py 2017-03-04 02:50:20 +0000
@@ -1,15 +1,13 @@
1# Copyright 2014-2015 Canonical Limited.1# Copyright 2014-2015 Canonical Limited.
2#2#
3# This file is part of charm-helpers.3# Licensed under the Apache License, Version 2.0 (the "License");
4#4# you may not use this file except in compliance with the License.
5# charm-helpers is free software: you can redistribute it and/or modify5# You may obtain a copy of the License at
6# it under the terms of the GNU Lesser General Public License version 3 as6#
7# published by the Free Software Foundation.7# http://www.apache.org/licenses/LICENSE-2.0
8#8#
9# charm-helpers is distributed in the hope that it will be useful,9# Unless required by applicable law or agreed to in writing, software
10# but WITHOUT ANY WARRANTY; without even the implied warranty of10# distributed under the License is distributed on an "AS IS" BASIS,
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# GNU Lesser General Public License for more details.12# See the License for the specific language governing permissions and
13#13# limitations under the License.
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1614
=== modified file 'charmhelpers/contrib/hahelpers/__init__.py'
--- charmhelpers/contrib/hahelpers/__init__.py 2015-01-30 11:16:09 +0000
+++ charmhelpers/contrib/hahelpers/__init__.py 2017-03-04 02:50:20 +0000
@@ -1,15 +1,13 @@
1# Copyright 2014-2015 Canonical Limited.1# Copyright 2014-2015 Canonical Limited.
2#2#
3# This file is part of charm-helpers.3# Licensed under the Apache License, Version 2.0 (the "License");
4#4# you may not use this file except in compliance with the License.
5# charm-helpers is free software: you can redistribute it and/or modify5# You may obtain a copy of the License at
6# it under the terms of the GNU Lesser General Public License version 3 as6#
7# published by the Free Software Foundation.7# http://www.apache.org/licenses/LICENSE-2.0
8#8#
9# charm-helpers is distributed in the hope that it will be useful,9# Unless required by applicable law or agreed to in writing, software
10# but WITHOUT ANY WARRANTY; without even the implied warranty of10# distributed under the License is distributed on an "AS IS" BASIS,
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# GNU Lesser General Public License for more details.12# See the License for the specific language governing permissions and
13#13# limitations under the License.
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1614
=== modified file 'charmhelpers/contrib/hahelpers/apache.py'
--- charmhelpers/contrib/hahelpers/apache.py 2015-01-30 11:16:09 +0000
+++ charmhelpers/contrib/hahelpers/apache.py 2017-03-04 02:50:20 +0000
@@ -1,18 +1,16 @@
1# Copyright 2014-2015 Canonical Limited.1# Copyright 2014-2015 Canonical Limited.
2#2#
3# This file is part of charm-helpers.3# Licensed under the Apache License, Version 2.0 (the "License");
4#4# you may not use this file except in compliance with the License.
5# charm-helpers is free software: you can redistribute it and/or modify5# You may obtain a copy of the License at
6# it under the terms of the GNU Lesser General Public License version 3 as6#
7# published by the Free Software Foundation.7# http://www.apache.org/licenses/LICENSE-2.0
8#8#
9# charm-helpers is distributed in the hope that it will be useful,9# Unless required by applicable law or agreed to in writing, software
10# but WITHOUT ANY WARRANTY; without even the implied warranty of10# distributed under the License is distributed on an "AS IS" BASIS,
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# GNU Lesser General Public License for more details.12# See the License for the specific language governing permissions and
13#13# limitations under the License.
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1614
17#15#
18# Copyright 2012 Canonical Ltd.16# Copyright 2012 Canonical Ltd.
@@ -24,6 +22,7 @@
24# Adam Gandelman <adamg@ubuntu.com>22# Adam Gandelman <adamg@ubuntu.com>
25#23#
2624
25import os
27import subprocess26import subprocess
2827
29from charmhelpers.core.hookenv import (28from charmhelpers.core.hookenv import (
@@ -74,9 +73,23 @@
74 return ca_cert73 return ca_cert
7574
7675
76def retrieve_ca_cert(cert_file):
77 cert = None
78 if os.path.isfile(cert_file):
79 with open(cert_file, 'r') as crt:
80 cert = crt.read()
81 return cert
82
83
77def install_ca_cert(ca_cert):84def install_ca_cert(ca_cert):
78 if ca_cert:85 if ca_cert:
79 with open('/usr/local/share/ca-certificates/keystone_juju_ca_cert.crt',86 cert_file = ('/usr/local/share/ca-certificates/'
80 'w') as crt:87 'keystone_juju_ca_cert.crt')
81 crt.write(ca_cert)88 old_cert = retrieve_ca_cert(cert_file)
82 subprocess.check_call(['update-ca-certificates', '--fresh'])89 if old_cert and old_cert == ca_cert:
90 log("CA cert is the same as installed version", level=INFO)
91 else:
92 log("Installing new CA cert", level=INFO)
93 with open(cert_file, 'w') as crt:
94 crt.write(ca_cert)
95 subprocess.check_call(['update-ca-certificates', '--fresh'])
8396
=== modified file 'charmhelpers/contrib/hahelpers/cluster.py'
--- charmhelpers/contrib/hahelpers/cluster.py 2015-07-03 09:13:26 +0000
+++ charmhelpers/contrib/hahelpers/cluster.py 2017-03-04 02:50:20 +0000
@@ -1,18 +1,16 @@
1# Copyright 2014-2015 Canonical Limited.1# Copyright 2014-2015 Canonical Limited.
2#2#
3# This file is part of charm-helpers.3# Licensed under the Apache License, Version 2.0 (the "License");
4#4# you may not use this file except in compliance with the License.
5# charm-helpers is free software: you can redistribute it and/or modify5# You may obtain a copy of the License at
6# it under the terms of the GNU Lesser General Public License version 3 as6#
7# published by the Free Software Foundation.7# http://www.apache.org/licenses/LICENSE-2.0
8#8#
9# charm-helpers is distributed in the hope that it will be useful,9# Unless required by applicable law or agreed to in writing, software
10# but WITHOUT ANY WARRANTY; without even the implied warranty of10# distributed under the License is distributed on an "AS IS" BASIS,
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# GNU Lesser General Public License for more details.12# See the License for the specific language governing permissions and
13#13# limitations under the License.
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1614
17#15#
18# Copyright 2012 Canonical Ltd.16# Copyright 2012 Canonical Ltd.
@@ -41,10 +39,11 @@
41 relation_get,39 relation_get,
42 config as config_get,40 config as config_get,
43 INFO,41 INFO,
44 ERROR,42 DEBUG,
45 WARNING,43 WARNING,
46 unit_get,44 unit_get,
47 is_leader as juju_is_leader45 is_leader as juju_is_leader,
46 status_set,
48)47)
49from charmhelpers.core.decorators import (48from charmhelpers.core.decorators import (
50 retry_on_exception,49 retry_on_exception,
@@ -60,6 +59,10 @@
60 pass59 pass
6160
6261
62class HAIncorrectConfig(Exception):
63 pass
64
65
63class CRMResourceNotFound(Exception):66class CRMResourceNotFound(Exception):
64 pass67 pass
6568
@@ -274,27 +277,71 @@
274 Obtains all relevant configuration from charm configuration required277 Obtains all relevant configuration from charm configuration required
275 for initiating a relation to hacluster:278 for initiating a relation to hacluster:
276279
277 ha-bindiface, ha-mcastport, vip280 ha-bindiface, ha-mcastport, vip, os-internal-hostname,
281 os-admin-hostname, os-public-hostname, os-access-hostname
278282
279 param: exclude_keys: list of setting key(s) to be excluded.283 param: exclude_keys: list of setting key(s) to be excluded.
280 returns: dict: A dict containing settings keyed by setting name.284 returns: dict: A dict containing settings keyed by setting name.
281 raises: HAIncompleteConfig if settings are missing.285 raises: HAIncompleteConfig if settings are missing or incorrect.
282 '''286 '''
283 settings = ['ha-bindiface', 'ha-mcastport', 'vip']287 settings = ['ha-bindiface', 'ha-mcastport', 'vip', 'os-internal-hostname',
288 'os-admin-hostname', 'os-public-hostname', 'os-access-hostname']
284 conf = {}289 conf = {}
285 for setting in settings:290 for setting in settings:
286 if exclude_keys and setting in exclude_keys:291 if exclude_keys and setting in exclude_keys:
287 continue292 continue
288293
289 conf[setting] = config_get(setting)294 conf[setting] = config_get(setting)
290 missing = []295
291 [missing.append(s) for s, v in six.iteritems(conf) if v is None]296 if not valid_hacluster_config():
292 if missing:297 raise HAIncorrectConfig('Insufficient or incorrect config data to '
293 log('Insufficient config data to configure hacluster.', level=ERROR)298 'configure hacluster.')
294 raise HAIncompleteConfig
295 return conf299 return conf
296300
297301
302def valid_hacluster_config():
303 '''
304 Check that either vip or dns-ha is set. If dns-ha then one of os-*-hostname
305 must be set.
306
307 Note: ha-bindiface and ha-macastport both have defaults and will always
308 be set. We only care that either vip or dns-ha is set.
309
310 :returns: boolean: valid config returns true.
311 raises: HAIncompatibileConfig if settings conflict.
312 raises: HAIncompleteConfig if settings are missing.
313 '''
314 vip = config_get('vip')
315 dns = config_get('dns-ha')
316 if not(bool(vip) ^ bool(dns)):
317 msg = ('HA: Either vip or dns-ha must be set but not both in order to '
318 'use high availability')
319 status_set('blocked', msg)
320 raise HAIncorrectConfig(msg)
321
322 # If dns-ha then one of os-*-hostname must be set
323 if dns:
324 dns_settings = ['os-internal-hostname', 'os-admin-hostname',
325 'os-public-hostname', 'os-access-hostname']
326 # At this point it is unknown if one or all of the possible
327 # network spaces are in HA. Validate at least one is set which is
328 # the minimum required.
329 for setting in dns_settings:
330 if config_get(setting):
331 log('DNS HA: At least one hostname is set {}: {}'
332 ''.format(setting, config_get(setting)),
333 level=DEBUG)
334 return True
335
336 msg = ('DNS HA: At least one os-*-hostname(s) must be set to use '
337 'DNS HA')
338 status_set('blocked', msg)
339 raise HAIncompleteConfig(msg)
340
341 log('VIP HA: VIP is set {}'.format(vip), level=DEBUG)
342 return True
343
344
298def canonical_url(configs, vip_setting='vip'):345def canonical_url(configs, vip_setting='vip'):
299 '''346 '''
300 Returns the correct HTTP URL to this host given the state of HTTPS347 Returns the correct HTTP URL to this host given the state of HTTPS
301348
=== modified file 'charmhelpers/core/__init__.py'
--- charmhelpers/core/__init__.py 2015-01-28 08:59:02 +0000
+++ charmhelpers/core/__init__.py 2017-03-04 02:50:20 +0000
@@ -1,15 +1,13 @@
1# Copyright 2014-2015 Canonical Limited.1# Copyright 2014-2015 Canonical Limited.
2#2#
3# This file is part of charm-helpers.3# Licensed under the Apache License, Version 2.0 (the "License");
4#4# you may not use this file except in compliance with the License.
5# charm-helpers is free software: you can redistribute it and/or modify5# You may obtain a copy of the License at
6# it under the terms of the GNU Lesser General Public License version 3 as6#
7# published by the Free Software Foundation.7# http://www.apache.org/licenses/LICENSE-2.0
8#8#
9# charm-helpers is distributed in the hope that it will be useful,9# Unless required by applicable law or agreed to in writing, software
10# but WITHOUT ANY WARRANTY; without even the implied warranty of10# distributed under the License is distributed on an "AS IS" BASIS,
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# GNU Lesser General Public License for more details.12# See the License for the specific language governing permissions and
13#13# limitations under the License.
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1614
=== modified file 'charmhelpers/core/decorators.py'
--- charmhelpers/core/decorators.py 2015-01-28 08:59:02 +0000
+++ charmhelpers/core/decorators.py 2017-03-04 02:50:20 +0000
@@ -1,18 +1,16 @@
1# Copyright 2014-2015 Canonical Limited.1# Copyright 2014-2015 Canonical Limited.
2#2#
3# This file is part of charm-helpers.3# Licensed under the Apache License, Version 2.0 (the "License");
4#4# you may not use this file except in compliance with the License.
5# charm-helpers is free software: you can redistribute it and/or modify5# You may obtain a copy of the License at
6# it under the terms of the GNU Lesser General Public License version 3 as6#
7# published by the Free Software Foundation.7# http://www.apache.org/licenses/LICENSE-2.0
8#8#
9# charm-helpers is distributed in the hope that it will be useful,9# Unless required by applicable law or agreed to in writing, software
10# but WITHOUT ANY WARRANTY; without even the implied warranty of10# distributed under the License is distributed on an "AS IS" BASIS,
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# GNU Lesser General Public License for more details.12# See the License for the specific language governing permissions and
13#13# limitations under the License.
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1614
17#15#
18# Copyright 2014 Canonical Ltd.16# Copyright 2014 Canonical Ltd.
1917
=== modified file 'charmhelpers/core/files.py'
--- charmhelpers/core/files.py 2015-12-11 15:23:38 +0000
+++ charmhelpers/core/files.py 2017-03-04 02:50:20 +0000
@@ -3,19 +3,17 @@
33
4# Copyright 2014-2015 Canonical Limited.4# Copyright 2014-2015 Canonical Limited.
5#5#
6# This file is part of charm-helpers.6# Licensed under the Apache License, Version 2.0 (the "License");
7#7# you may not use this file except in compliance with the License.
8# charm-helpers is free software: you can redistribute it and/or modify8# You may obtain a copy of the License at
9# it under the terms of the GNU Lesser General Public License version 3 as9#
10# published by the Free Software Foundation.10# http://www.apache.org/licenses/LICENSE-2.0
11#11#
12# charm-helpers is distributed in the hope that it will be useful,12# Unless required by applicable law or agreed to in writing, software
13# but WITHOUT ANY WARRANTY; without even the implied warranty of13# distributed under the License is distributed on an "AS IS" BASIS,
14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the14# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15# GNU Lesser General Public License for more details.15# See the License for the specific language governing permissions and
16#16# limitations under the License.
17# You should have received a copy of the GNU Lesser General Public License
18# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1917
20__author__ = 'Jorge Niedbalski <niedbalski@ubuntu.com>'18__author__ = 'Jorge Niedbalski <niedbalski@ubuntu.com>'
2119
2220
=== modified file 'charmhelpers/core/fstab.py'
--- charmhelpers/core/fstab.py 2015-03-12 11:42:26 +0000
+++ charmhelpers/core/fstab.py 2017-03-04 02:50:20 +0000
@@ -3,19 +3,17 @@
33
4# Copyright 2014-2015 Canonical Limited.4# Copyright 2014-2015 Canonical Limited.
5#5#
6# This file is part of charm-helpers.6# Licensed under the Apache License, Version 2.0 (the "License");
7#7# you may not use this file except in compliance with the License.
8# charm-helpers is free software: you can redistribute it and/or modify8# You may obtain a copy of the License at
9# it under the terms of the GNU Lesser General Public License version 3 as9#
10# published by the Free Software Foundation.10# http://www.apache.org/licenses/LICENSE-2.0
11#11#
12# charm-helpers is distributed in the hope that it will be useful,12# Unless required by applicable law or agreed to in writing, software
13# but WITHOUT ANY WARRANTY; without even the implied warranty of13# distributed under the License is distributed on an "AS IS" BASIS,
14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the14# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15# GNU Lesser General Public License for more details.15# See the License for the specific language governing permissions and
16#16# limitations under the License.
17# You should have received a copy of the GNU Lesser General Public License
18# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1917
20import io18import io
21import os19import os
2220
=== modified file 'charmhelpers/core/hookenv.py'
--- charmhelpers/core/hookenv.py 2016-05-06 07:12:40 +0000
+++ charmhelpers/core/hookenv.py 2017-03-04 02:50:20 +0000
@@ -1,18 +1,16 @@
1# Copyright 2014-2015 Canonical Limited.1# Copyright 2014-2015 Canonical Limited.
2#2#
3# This file is part of charm-helpers.3# Licensed under the Apache License, Version 2.0 (the "License");
4#4# you may not use this file except in compliance with the License.
5# charm-helpers is free software: you can redistribute it and/or modify5# You may obtain a copy of the License at
6# it under the terms of the GNU Lesser General Public License version 3 as6#
7# published by the Free Software Foundation.7# http://www.apache.org/licenses/LICENSE-2.0
8#8#
9# charm-helpers is distributed in the hope that it will be useful,9# Unless required by applicable law or agreed to in writing, software
10# but WITHOUT ANY WARRANTY; without even the implied warranty of10# distributed under the License is distributed on an "AS IS" BASIS,
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# GNU Lesser General Public License for more details.12# See the License for the specific language governing permissions and
13#13# limitations under the License.
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1614
17"Interactions with the Juju environment"15"Interactions with the Juju environment"
18# Copyright 2013 Canonical Ltd.16# Copyright 2013 Canonical Ltd.
@@ -334,6 +332,8 @@
334 config_cmd_line = ['config-get']332 config_cmd_line = ['config-get']
335 if scope is not None:333 if scope is not None:
336 config_cmd_line.append(scope)334 config_cmd_line.append(scope)
335 else:
336 config_cmd_line.append('--all')
337 config_cmd_line.append('--format=json')337 config_cmd_line.append('--format=json')
338 try:338 try:
339 config_data = json.loads(339 config_data = json.loads(
@@ -616,6 +616,20 @@
616 subprocess.check_call(_args)616 subprocess.check_call(_args)
617617
618618
619def open_ports(start, end, protocol="TCP"):
620 """Opens a range of service network ports"""
621 _args = ['open-port']
622 _args.append('{}-{}/{}'.format(start, end, protocol))
623 subprocess.check_call(_args)
624
625
626def close_ports(start, end, protocol="TCP"):
627 """Close a range of service network ports"""
628 _args = ['close-port']
629 _args.append('{}-{}/{}'.format(start, end, protocol))
630 subprocess.check_call(_args)
631
632
619@cached633@cached
620def unit_get(attribute):634def unit_get(attribute):
621 """Get the unit ID for the remote unit"""635 """Get the unit ID for the remote unit"""
@@ -845,6 +859,20 @@
845 return inner_translate_exc1859 return inner_translate_exc1
846860
847861
862def application_version_set(version):
863 """Charm authors may trigger this command from any hook to output what
864 version of the application is running. This could be a package version,
865 for instance postgres version 9.5. It could also be a build number or
866 version control revision identifier, for instance git sha 6fb7ba68. """
867
868 cmd = ['application-version-set']
869 cmd.append(version)
870 try:
871 subprocess.check_call(cmd)
872 except OSError:
873 log("Application Version: {}".format(version))
874
875
848@translate_exc(from_exc=OSError, to_exc=NotImplementedError)876@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
849def is_leader():877def is_leader():
850 """Does the current unit hold the juju leadership878 """Does the current unit hold the juju leadership
@@ -1006,4 +1034,35 @@
1006 :raise: NotImplementedError if run on Juju < 2.01034 :raise: NotImplementedError if run on Juju < 2.0
1007 '''1035 '''
1008 cmd = ['network-get', '--primary-address', binding]1036 cmd = ['network-get', '--primary-address', binding]
1009 return subprocess.check_output(cmd).strip()1037 return subprocess.check_output(cmd).decode('UTF-8').strip()
1038
1039
1040def add_metric(*args, **kwargs):
1041 """Add metric values. Values may be expressed with keyword arguments. For
1042 metric names containing dashes, these may be expressed as one or more
1043 'key=value' positional arguments. May only be called from the collect-metrics
1044 hook."""
1045 _args = ['add-metric']
1046 _kvpairs = []
1047 _kvpairs.extend(args)
1048 _kvpairs.extend(['{}={}'.format(k, v) for k, v in kwargs.items()])
1049 _args.extend(sorted(_kvpairs))
1050 try:
1051 subprocess.check_call(_args)
1052 return
1053 except EnvironmentError as e:
1054 if e.errno != errno.ENOENT:
1055 raise
1056 log_message = 'add-metric failed: {}'.format(' '.join(_kvpairs))
1057 log(log_message, level='INFO')
1058
1059
1060def meter_status():
1061 """Get the meter status, if running in the meter-status-changed hook."""
1062 return os.environ.get('JUJU_METER_STATUS')
1063
1064
1065def meter_info():
1066 """Get the meter status information, if running in the meter-status-changed
1067 hook."""
1068 return os.environ.get('JUJU_METER_INFO')
10101069
=== modified file 'charmhelpers/core/host.py'
--- charmhelpers/core/host.py 2016-05-06 07:12:40 +0000
+++ charmhelpers/core/host.py 2017-03-04 02:50:20 +0000
@@ -1,18 +1,16 @@
1# Copyright 2014-2015 Canonical Limited.1# Copyright 2014-2015 Canonical Limited.
2#2#
3# This file is part of charm-helpers.3# Licensed under the Apache License, Version 2.0 (the "License");
4#4# you may not use this file except in compliance with the License.
5# charm-helpers is free software: you can redistribute it and/or modify5# You may obtain a copy of the License at
6# it under the terms of the GNU Lesser General Public License version 3 as6#
7# published by the Free Software Foundation.7# http://www.apache.org/licenses/LICENSE-2.0
8#8#
9# charm-helpers is distributed in the hope that it will be useful,9# Unless required by applicable law or agreed to in writing, software
10# but WITHOUT ANY WARRANTY; without even the implied warranty of10# distributed under the License is distributed on an "AS IS" BASIS,
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# GNU Lesser General Public License for more details.12# See the License for the specific language governing permissions and
13#13# limitations under the License.
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1614
17"""Tools for working with the host system"""15"""Tools for working with the host system"""
18# Copyright 2012 Canonical Ltd.16# Copyright 2012 Canonical Ltd.
@@ -32,46 +30,162 @@
32import hashlib30import hashlib
33import functools31import functools
34import itertools32import itertools
33import six
34
35from contextlib import contextmanager35from contextlib import contextmanager
36from collections import OrderedDict36from collections import OrderedDict
37
38import six
39
40from .hookenv import log37from .hookenv import log
41from .fstab import Fstab38from .fstab import Fstab
4239from charmhelpers.osplatform import get_platform
4340
44def service_start(service_name):41__platform__ = get_platform()
45 """Start a system service"""42if __platform__ == "ubuntu":
46 return service('start', service_name)43 from charmhelpers.core.host_factory.ubuntu import (
4744 service_available,
4845 add_new_group,
49def service_stop(service_name):46 lsb_release,
50 """Stop a system service"""47 cmp_pkgrevno,
51 return service('stop', service_name)48 ) # flake8: noqa -- ignore F401 for this import
5249elif __platform__ == "centos":
5350 from charmhelpers.core.host_factory.centos import (
54def service_restart(service_name):51 service_available,
55 """Restart a system service"""52 add_new_group,
53 lsb_release,
54 cmp_pkgrevno,
55 ) # flake8: noqa -- ignore F401 for this import
56
57UPDATEDB_PATH = '/etc/updatedb.conf'
58
59def service_start(service_name, **kwargs):
60 """Start a system service.
61
62 The specified service name is managed via the system level init system.
63 Some init systems (e.g. upstart) require that additional arguments be
64 provided in order to directly control service instances whereas other init
65 systems allow for addressing instances of a service directly by name (e.g.
66 systemd).
67
68 The kwargs allow for the additional parameters to be passed to underlying
69 init systems for those systems which require/allow for them. For example,
70 the ceph-osd upstart script requires the id parameter to be passed along
71 in order to identify which running daemon should be reloaded. The follow-
72 ing example stops the ceph-osd service for instance id=4:
73
74 service_stop('ceph-osd', id=4)
75
76 :param service_name: the name of the service to stop
77 :param **kwargs: additional parameters to pass to the init system when
78 managing services. These will be passed as key=value
79 parameters to the init system's commandline. kwargs
80 are ignored for systemd enabled systems.
81 """
82 return service('start', service_name, **kwargs)
83
84
85def service_stop(service_name, **kwargs):
86 """Stop a system service.
87
88 The specified service name is managed via the system level init system.
89 Some init systems (e.g. upstart) require that additional arguments be
90 provided in order to directly control service instances whereas other init
91 systems allow for addressing instances of a service directly by name (e.g.
92 systemd).
93
94 The kwargs allow for the additional parameters to be passed to underlying
95 init systems for those systems which require/allow for them. For example,
96 the ceph-osd upstart script requires the id parameter to be passed along
97 in order to identify which running daemon should be reloaded. The follow-
98 ing example stops the ceph-osd service for instance id=4:
99
100 service_stop('ceph-osd', id=4)
101
102 :param service_name: the name of the service to stop
103 :param **kwargs: additional parameters to pass to the init system when
104 managing services. These will be passed as key=value
105 parameters to the init system's commandline. kwargs
106 are ignored for systemd enabled systems.
107 """
108 return service('stop', service_name, **kwargs)
109
110
111def service_restart(service_name, **kwargs):
112 """Restart a system service.
113
114 The specified service name is managed via the system level init system.
115 Some init systems (e.g. upstart) require that additional arguments be
116 provided in order to directly control service instances whereas other init
117 systems allow for addressing instances of a service directly by name (e.g.
118 systemd).
119
120 The kwargs allow for the additional parameters to be passed to underlying
121 init systems for those systems which require/allow for them. For example,
122 the ceph-osd upstart script requires the id parameter to be passed along
123 in order to identify which running daemon should be restarted. The follow-
124 ing example restarts the ceph-osd service for instance id=4:
125
126 service_restart('ceph-osd', id=4)
127
128 :param service_name: the name of the service to restart
129 :param **kwargs: additional parameters to pass to the init system when
130 managing services. These will be passed as key=value
131 parameters to the init system's commandline. kwargs
132 are ignored for init systems not allowing additional
133 parameters via the commandline (systemd).
134 """
56 return service('restart', service_name)135 return service('restart', service_name)
57136
58137
59def service_reload(service_name, restart_on_failure=False):138def service_reload(service_name, restart_on_failure=False, **kwargs):
60 """Reload a system service, optionally falling back to restart if139 """Reload a system service, optionally falling back to restart if
61 reload fails"""140 reload fails.
62 service_result = service('reload', service_name)141
142 The specified service name is managed via the system level init system.
143 Some init systems (e.g. upstart) require that additional arguments be
144 provided in order to directly control service instances whereas other init
145 systems allow for addressing instances of a service directly by name (e.g.
146 systemd).
147
148 The kwargs allow for the additional parameters to be passed to underlying
149 init systems for those systems which require/allow for them. For example,
150 the ceph-osd upstart script requires the id parameter to be passed along
151 in order to identify which running daemon should be reloaded. The follow-
152 ing example restarts the ceph-osd service for instance id=4:
153
154 service_reload('ceph-osd', id=4)
155
156 :param service_name: the name of the service to reload
157 :param restart_on_failure: boolean indicating whether to fallback to a
158 restart if the reload fails.
159 :param **kwargs: additional parameters to pass to the init system when
160 managing services. These will be passed as key=value
161 parameters to the init system's commandline. kwargs
162 are ignored for init systems not allowing additional
163 parameters via the commandline (systemd).
164 """
165 service_result = service('reload', service_name, **kwargs)
63 if not service_result and restart_on_failure:166 if not service_result and restart_on_failure:
64 service_result = service('restart', service_name)167 service_result = service('restart', service_name, **kwargs)
65 return service_result168 return service_result
66169
67170
68def service_pause(service_name, init_dir="/etc/init", initd_dir="/etc/init.d"):171def service_pause(service_name, init_dir="/etc/init", initd_dir="/etc/init.d",
172 **kwargs):
69 """Pause a system service.173 """Pause a system service.
70174
71 Stop it, and prevent it from starting again at boot."""175 Stop it, and prevent it from starting again at boot.
176
177 :param service_name: the name of the service to pause
178 :param init_dir: path to the upstart init directory
179 :param initd_dir: path to the sysv init directory
180 :param **kwargs: additional parameters to pass to the init system when
181 managing services. These will be passed as key=value
182 parameters to the init system's commandline. kwargs
183 are ignored for init systems which do not support
184 key=value arguments via the commandline.
185 """
72 stopped = True186 stopped = True
73 if service_running(service_name):187 if service_running(service_name, **kwargs):
74 stopped = service_stop(service_name)188 stopped = service_stop(service_name, **kwargs)
75 upstart_file = os.path.join(init_dir, "{}.conf".format(service_name))189 upstart_file = os.path.join(init_dir, "{}.conf".format(service_name))
76 sysv_file = os.path.join(initd_dir, service_name)190 sysv_file = os.path.join(initd_dir, service_name)
77 if init_is_systemd():191 if init_is_systemd():
@@ -92,10 +206,19 @@
92206
93207
94def service_resume(service_name, init_dir="/etc/init",208def service_resume(service_name, init_dir="/etc/init",
95 initd_dir="/etc/init.d"):209 initd_dir="/etc/init.d", **kwargs):
96 """Resume a system service.210 """Resume a system service.
97211
98 Reenable starting again at boot. Start the service"""212 Reenable starting again at boot. Start the service.
213
214 :param service_name: the name of the service to resume
215 :param init_dir: the path to the init dir
216 :param initd dir: the path to the initd dir
217 :param **kwargs: additional parameters to pass to the init system when
218 managing services. These will be passed as key=value
219 parameters to the init system's commandline. kwargs
220 are ignored for systemd enabled systems.
221 """
99 upstart_file = os.path.join(init_dir, "{}.conf".format(service_name))222 upstart_file = os.path.join(init_dir, "{}.conf".format(service_name))
100 sysv_file = os.path.join(initd_dir, service_name)223 sysv_file = os.path.join(initd_dir, service_name)
101 if init_is_systemd():224 if init_is_systemd():
@@ -112,62 +235,70 @@
112 "Unable to detect {0} as SystemD, Upstart {1} or"235 "Unable to detect {0} as SystemD, Upstart {1} or"
113 " SysV {2}".format(236 " SysV {2}".format(
114 service_name, upstart_file, sysv_file))237 service_name, upstart_file, sysv_file))
238 started = service_running(service_name, **kwargs)
115239
116 started = service_running(service_name)
117 if not started:240 if not started:
118 started = service_start(service_name)241 started = service_start(service_name, **kwargs)
119 return started242 return started
120243
121244
122def service(action, service_name):245def service(action, service_name, **kwargs):
123 """Control a system service"""246 """Control a system service.
247
248 :param action: the action to take on the service
249 :param service_name: the name of the service to perform th action on
250 :param **kwargs: additional params to be passed to the service command in
251 the form of key=value.
252 """
124 if init_is_systemd():253 if init_is_systemd():
125 cmd = ['systemctl', action, service_name]254 cmd = ['systemctl', action, service_name]
126 else:255 else:
127 cmd = ['service', service_name, action]256 cmd = ['service', service_name, action]
257 for key, value in six.iteritems(kwargs):
258 parameter = '%s=%s' % (key, value)
259 cmd.append(parameter)
128 return subprocess.call(cmd) == 0260 return subprocess.call(cmd) == 0
129261
130262
131def systemv_services_running():263_UPSTART_CONF = "/etc/init/{}.conf"
132 output = subprocess.check_output(264_INIT_D_CONF = "/etc/init.d/{}"
133 ['service', '--status-all'],265
134 stderr=subprocess.STDOUT).decode('UTF-8')266
135 return [row.split()[-1] for row in output.split('\n') if '[ + ]' in row]267def service_running(service_name, **kwargs):
136268 """Determine whether a system service is running.
137269
138def service_running(service_name):270 :param service_name: the name of the service
139 """Determine whether a system service is running"""271 :param **kwargs: additional args to pass to the service command. This is
272 used to pass additional key=value arguments to the
273 service command line for managing specific instance
274 units (e.g. service ceph-osd status id=2). The kwargs
275 are ignored in systemd services.
276 """
140 if init_is_systemd():277 if init_is_systemd():
141 return service('is-active', service_name)278 return service('is-active', service_name)
142 else:279 else:
143 try:280 if os.path.exists(_UPSTART_CONF.format(service_name)):
144 output = subprocess.check_output(281 try:
145 ['service', service_name, 'status'],282 cmd = ['status', service_name]
146 stderr=subprocess.STDOUT).decode('UTF-8')283 for key, value in six.iteritems(kwargs):
147 except subprocess.CalledProcessError:284 parameter = '%s=%s' % (key, value)
148 return False285 cmd.append(parameter)
149 else:286 output = subprocess.check_output(cmd,
150 # This works for upstart scripts where the 'service' command287 stderr=subprocess.STDOUT).decode('UTF-8')
151 # returns a consistent string to represent running 'start/running'288 except subprocess.CalledProcessError:
152 if ("start/running" in output or "is running" in output or289 return False
153 "up and running" in output):290 else:
154 return True291 # This works for upstart scripts where the 'service' command
292 # returns a consistent string to represent running
293 # 'start/running'
294 if ("start/running" in output or
295 "is running" in output or
296 "up and running" in output):
297 return True
298 elif os.path.exists(_INIT_D_CONF.format(service_name)):
155 # Check System V scripts init script return codes299 # Check System V scripts init script return codes
156 if service_name in systemv_services_running():300 return service('status', service_name)
157 return True301 return False
158 return False
159
160
161def service_available(service_name):
162 """Determine whether a system service is available"""
163 try:
164 subprocess.check_output(
165 ['service', service_name, 'status'],
166 stderr=subprocess.STDOUT).decode('UTF-8')
167 except subprocess.CalledProcessError as e:
168 return b'unrecognized service' not in e.output
169 else:
170 return True
171302
172303
173SYSTEMD_SYSTEM = '/run/systemd/system'304SYSTEMD_SYSTEM = '/run/systemd/system'
@@ -178,8 +309,9 @@
178 return os.path.isdir(SYSTEMD_SYSTEM)309 return os.path.isdir(SYSTEMD_SYSTEM)
179310
180311
181def adduser(username, password=None, shell='/bin/bash', system_user=False,312def adduser(username, password=None, shell='/bin/bash',
182 primary_group=None, secondary_groups=None):313 system_user=False, primary_group=None,
314 secondary_groups=None, uid=None, home_dir=None):
183 """Add a user to the system.315 """Add a user to the system.
184316
185 Will log but otherwise succeed if the user already exists.317 Will log but otherwise succeed if the user already exists.
@@ -190,15 +322,24 @@
190 :param bool system_user: Whether to create a login or system user322 :param bool system_user: Whether to create a login or system user
191 :param str primary_group: Primary group for user; defaults to username323 :param str primary_group: Primary group for user; defaults to username
192 :param list secondary_groups: Optional list of additional groups324 :param list secondary_groups: Optional list of additional groups
325 :param int uid: UID for user being created
326 :param str home_dir: Home directory for user
193327
194 :returns: The password database entry struct, as returned by `pwd.getpwnam`328 :returns: The password database entry struct, as returned by `pwd.getpwnam`
195 """329 """
196 try:330 try:
197 user_info = pwd.getpwnam(username)331 user_info = pwd.getpwnam(username)
198 log('user {0} already exists!'.format(username))332 log('user {0} already exists!'.format(username))
333 if uid:
334 user_info = pwd.getpwuid(int(uid))
335 log('user with uid {0} already exists!'.format(uid))
199 except KeyError:336 except KeyError:
200 log('creating user {0}'.format(username))337 log('creating user {0}'.format(username))
201 cmd = ['useradd']338 cmd = ['useradd']
339 if uid:
340 cmd.extend(['--uid', str(uid)])
341 if home_dir:
342 cmd.extend(['--home', str(home_dir)])
202 if system_user or password is None:343 if system_user or password is None:
203 cmd.append('--system')344 cmd.append('--system')
204 else:345 else:
@@ -233,22 +374,56 @@
233 return user_exists374 return user_exists
234375
235376
236def add_group(group_name, system_group=False):377def uid_exists(uid):
237 """Add a group to the system"""378 """Check if a uid exists"""
379 try:
380 pwd.getpwuid(uid)
381 uid_exists = True
382 except KeyError:
383 uid_exists = False
384 return uid_exists
385
386
387def group_exists(groupname):
388 """Check if a group exists"""
389 try:
390 grp.getgrnam(groupname)
391 group_exists = True
392 except KeyError:
393 group_exists = False
394 return group_exists
395
396
397def gid_exists(gid):
398 """Check if a gid exists"""
399 try:
400 grp.getgrgid(gid)
401 gid_exists = True
402 except KeyError:
403 gid_exists = False
404 return gid_exists
405
406
407def add_group(group_name, system_group=False, gid=None):
408 """Add a group to the system
409
410 Will log but otherwise succeed if the group already exists.
411
412 :param str group_name: group to create
413 :param bool system_group: Create system group
414 :param int gid: GID for user being created
415
416 :returns: The password database entry struct, as returned by `grp.getgrnam`
417 """
238 try:418 try:
239 group_info = grp.getgrnam(group_name)419 group_info = grp.getgrnam(group_name)
240 log('group {0} already exists!'.format(group_name))420 log('group {0} already exists!'.format(group_name))
421 if gid:
422 group_info = grp.getgrgid(gid)
423 log('group with gid {0} already exists!'.format(gid))
241 except KeyError:424 except KeyError:
242 log('creating group {0}'.format(group_name))425 log('creating group {0}'.format(group_name))
243 cmd = ['addgroup']426 add_new_group(group_name, system_group, gid)
244 if system_group:
245 cmd.append('--system')
246 else:
247 cmd.extend([
248 '--group',
249 ])
250 cmd.append(group_name)
251 subprocess.check_call(cmd)
252 group_info = grp.getgrnam(group_name)427 group_info = grp.getgrnam(group_name)
253 return group_info428 return group_info
254429
@@ -260,15 +435,17 @@
260 subprocess.check_call(cmd)435 subprocess.check_call(cmd)
261436
262437
263def rsync(from_path, to_path, flags='-r', options=None):438def rsync(from_path, to_path, flags='-r', options=None, timeout=None):
264 """Replicate the contents of a path"""439 """Replicate the contents of a path"""
265 options = options or ['--delete', '--executability']440 options = options or ['--delete', '--executability']
266 cmd = ['/usr/bin/rsync', flags]441 cmd = ['/usr/bin/rsync', flags]
442 if timeout:
443 cmd = ['timeout', str(timeout)] + cmd
267 cmd.extend(options)444 cmd.extend(options)
268 cmd.append(from_path)445 cmd.append(from_path)
269 cmd.append(to_path)446 cmd.append(to_path)
270 log(" ".join(cmd))447 log(" ".join(cmd))
271 return subprocess.check_output(cmd).decode('UTF-8').strip()448 return subprocess.check_output(cmd, stderr=subprocess.STDOUT).decode('UTF-8').strip()
272449
273450
274def symlink(source, destination):451def symlink(source, destination):
@@ -493,16 +670,6 @@
493 return r670 return r
494671
495672
496def lsb_release():
497 """Return /etc/lsb-release in a dict"""
498 d = {}
499 with open('/etc/lsb-release', 'r') as lsb:
500 for l in lsb:
501 k, v = l.split('=')
502 d[k.strip()] = v.strip()
503 return d
504
505
506def pwgen(length=None):673def pwgen(length=None):
507 """Generate a random pasword."""674 """Generate a random pasword."""
508 if length is None:675 if length is None:
@@ -626,25 +793,6 @@
626 return hwaddr793 return hwaddr
627794
628795
629def cmp_pkgrevno(package, revno, pkgcache=None):
630 """Compare supplied revno with the revno of the installed package
631
632 * 1 => Installed revno is greater than supplied arg
633 * 0 => Installed revno is the same as supplied arg
634 * -1 => Installed revno is less than supplied arg
635
636 This function imports apt_cache function from charmhelpers.fetch if
637 the pkgcache argument is None. Be sure to add charmhelpers.fetch if
638 you call this function, or pass an apt_pkg.Cache() instance.
639 """
640 import apt_pkg
641 if not pkgcache:
642 from charmhelpers.fetch import apt_cache
643 pkgcache = apt_cache()
644 pkg = pkgcache[package]
645 return apt_pkg.version_compare(pkg.current_ver.ver_str, revno)
646
647
648@contextmanager796@contextmanager
649def chdir(directory):797def chdir(directory):
650 """Change the current working directory to a different directory for a code798 """Change the current working directory to a different directory for a code
@@ -667,7 +815,7 @@
667 :param str path: The string path to start changing ownership.815 :param str path: The string path to start changing ownership.
668 :param str owner: The owner string to use when looking up the uid.816 :param str owner: The owner string to use when looking up the uid.
669 :param str group: The group string to use when looking up the gid.817 :param str group: The group string to use when looking up the gid.
670 :param bool follow_links: Also Chown links if True818 :param bool follow_links: Also follow and chown links if True
671 :param bool chowntopdir: Also chown path itself if True819 :param bool chowntopdir: Also chown path itself if True
672 """820 """
673 uid = pwd.getpwnam(owner).pw_uid821 uid = pwd.getpwnam(owner).pw_uid
@@ -681,7 +829,7 @@
681 broken_symlink = os.path.lexists(path) and not os.path.exists(path)829 broken_symlink = os.path.lexists(path) and not os.path.exists(path)
682 if not broken_symlink:830 if not broken_symlink:
683 chown(path, uid, gid)831 chown(path, uid, gid)
684 for root, dirs, files in os.walk(path):832 for root, dirs, files in os.walk(path, followlinks=follow_links):
685 for name in dirs + files:833 for name in dirs + files:
686 full = os.path.join(root, name)834 full = os.path.join(root, name)
687 broken_symlink = os.path.lexists(full) and not os.path.exists(full)835 broken_symlink = os.path.lexists(full) and not os.path.exists(full)
@@ -701,6 +849,20 @@
701 chownr(path, owner, group, follow_links=False)849 chownr(path, owner, group, follow_links=False)
702850
703851
852def owner(path):
853 """Returns a tuple containing the username & groupname owning the path.
854
855 :param str path: the string path to retrieve the ownership
856 :return tuple(str, str): A (username, groupname) tuple containing the
857 name of the user and group owning the path.
858 :raises OSError: if the specified path does not exist
859 """
860 stat = os.stat(path)
861 username = pwd.getpwuid(stat.st_uid)[0]
862 groupname = grp.getgrgid(stat.st_gid)[0]
863 return username, groupname
864
865
704def get_total_ram():866def get_total_ram():
705 """The total amount of system RAM in bytes.867 """The total amount of system RAM in bytes.
706868
@@ -715,3 +877,42 @@
715 assert unit == 'kB', 'Unknown unit'877 assert unit == 'kB', 'Unknown unit'
716 return int(value) * 1024 # Classic, not KiB.878 return int(value) * 1024 # Classic, not KiB.
717 raise NotImplementedError()879 raise NotImplementedError()
880
881
882UPSTART_CONTAINER_TYPE = '/run/container_type'
883
884
885def is_container():
886 """Determine whether unit is running in a container
887
888 @return: boolean indicating if unit is in a container
889 """
890 if init_is_systemd():
891 # Detect using systemd-detect-virt
892 return subprocess.call(['systemd-detect-virt',
893 '--container']) == 0
894 else:
895 # Detect using upstart container file marker
896 return os.path.exists(UPSTART_CONTAINER_TYPE)
897
898
899def add_to_updatedb_prunepath(path, updatedb_path=UPDATEDB_PATH):
900 with open(updatedb_path, 'r+') as f_id:
901 updatedb_text = f_id.read()
902 output = updatedb(updatedb_text, path)
903 f_id.seek(0)
904 f_id.write(output)
905 f_id.truncate()
906
907
908def updatedb(updatedb_text, new_path):
909 lines = [line for line in updatedb_text.split("\n")]
910 for i, line in enumerate(lines):
911 if line.startswith("PRUNEPATHS="):
912 paths_line = line.split("=")[1].replace('"', '')
913 paths = paths_line.split(" ")
914 if new_path not in paths:
915 paths.append(new_path)
916 lines[i] = 'PRUNEPATHS="{}"'.format(' '.join(paths))
917 output = "\n".join(lines)
918 return output
718919
=== added directory 'charmhelpers/core/host_factory'
=== added file 'charmhelpers/core/host_factory/__init__.py'
=== added file 'charmhelpers/core/host_factory/centos.py'
--- charmhelpers/core/host_factory/centos.py 1970-01-01 00:00:00 +0000
+++ charmhelpers/core/host_factory/centos.py 2017-03-04 02:50:20 +0000
@@ -0,0 +1,56 @@
1import subprocess
2import yum
3import os
4
5
6def service_available(service_name):
7 # """Determine whether a system service is available."""
8 if os.path.isdir('/run/systemd/system'):
9 cmd = ['systemctl', 'is-enabled', service_name]
10 else:
11 cmd = ['service', service_name, 'is-enabled']
12 return subprocess.call(cmd) == 0
13
14
15def add_new_group(group_name, system_group=False, gid=None):
16 cmd = ['groupadd']
17 if gid:
18 cmd.extend(['--gid', str(gid)])
19 if system_group:
20 cmd.append('-r')
21 cmd.append(group_name)
22 subprocess.check_call(cmd)
23
24
25def lsb_release():
26 """Return /etc/os-release in a dict."""
27 d = {}
28 with open('/etc/os-release', 'r') as lsb:
29 for l in lsb:
30 s = l.split('=')
31 if len(s) != 2:
32 continue
33 d[s[0].strip()] = s[1].strip()
34 return d
35
36
37def cmp_pkgrevno(package, revno, pkgcache=None):
38 """Compare supplied revno with the revno of the installed package.
39
40 * 1 => Installed revno is greater than supplied arg
41 * 0 => Installed revno is the same as supplied arg
42 * -1 => Installed revno is less than supplied arg
43
44 This function imports YumBase function if the pkgcache argument
45 is None.
46 """
47 if not pkgcache:
48 y = yum.YumBase()
49 packages = y.doPackageLists()
50 pkgcache = {i.Name: i.version for i in packages['installed']}
51 pkg = pkgcache[package]
52 if pkg > revno:
53 return 1
54 if pkg < revno:
55 return -1
56 return 0
057
=== added file 'charmhelpers/core/host_factory/ubuntu.py'
--- charmhelpers/core/host_factory/ubuntu.py 1970-01-01 00:00:00 +0000
+++ charmhelpers/core/host_factory/ubuntu.py 2017-03-04 02:50:20 +0000
@@ -0,0 +1,56 @@
1import subprocess
2
3
4def service_available(service_name):
5 """Determine whether a system service is available"""
6 try:
7 subprocess.check_output(
8 ['service', service_name, 'status'],
9 stderr=subprocess.STDOUT).decode('UTF-8')
10 except subprocess.CalledProcessError as e:
11 return b'unrecognized service' not in e.output
12 else:
13 return True
14
15
16def add_new_group(group_name, system_group=False, gid=None):
17 cmd = ['addgroup']
18 if gid:
19 cmd.extend(['--gid', str(gid)])
20 if system_group:
21 cmd.append('--system')
22 else:
23 cmd.extend([
24 '--group',
25 ])
26 cmd.append(group_name)
27 subprocess.check_call(cmd)
28
29
30def lsb_release():
31 """Return /etc/lsb-release in a dict"""
32 d = {}
33 with open('/etc/lsb-release', 'r') as lsb:
34 for l in lsb:
35 k, v = l.split('=')
36 d[k.strip()] = v.strip()
37 return d
38
39
40def cmp_pkgrevno(package, revno, pkgcache=None):
41 """Compare supplied revno with the revno of the installed package.
42
43 * 1 => Installed revno is greater than supplied arg
44 * 0 => Installed revno is the same as supplied arg
45 * -1 => Installed revno is less than supplied arg
46
47 This function imports apt_cache function from charmhelpers.fetch if
48 the pkgcache argument is None. Be sure to add charmhelpers.fetch if
49 you call this function, or pass an apt_pkg.Cache() instance.
50 """
51 import apt_pkg
52 if not pkgcache:
53 from charmhelpers.fetch import apt_cache
54 pkgcache = apt_cache()
55 pkg = pkgcache[package]
56 return apt_pkg.version_compare(pkg.current_ver.ver_str, revno)
057
=== modified file 'charmhelpers/core/hugepage.py'
--- charmhelpers/core/hugepage.py 2015-12-11 15:23:38 +0000
+++ charmhelpers/core/hugepage.py 2017-03-04 02:50:20 +0000
@@ -2,19 +2,17 @@
22
3# Copyright 2014-2015 Canonical Limited.3# Copyright 2014-2015 Canonical Limited.
4#4#
5# This file is part of charm-helpers.5# Licensed under the Apache License, Version 2.0 (the "License");
6#6# you may not use this file except in compliance with the License.
7# charm-helpers is free software: you can redistribute it and/or modify7# You may obtain a copy of the License at
8# it under the terms of the GNU Lesser General Public License version 3 as8#
9# published by the Free Software Foundation.9# http://www.apache.org/licenses/LICENSE-2.0
10#10#
11# charm-helpers is distributed in the hope that it will be useful,11# Unless required by applicable law or agreed to in writing, software
12# but WITHOUT ANY WARRANTY; without even the implied warranty of12# distributed under the License is distributed on an "AS IS" BASIS,
13# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the13# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14# GNU Lesser General Public License for more details.14# See the License for the specific language governing permissions and
15#15# limitations under the License.
16# You should have received a copy of the GNU Lesser General Public License
17# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1816
19import yaml17import yaml
20from charmhelpers.core import fstab18from charmhelpers.core import fstab
2119
=== modified file 'charmhelpers/core/kernel.py'
--- charmhelpers/core/kernel.py 2015-12-11 15:23:38 +0000
+++ charmhelpers/core/kernel.py 2017-03-04 02:50:20 +0000
@@ -3,29 +3,40 @@
33
4# Copyright 2014-2015 Canonical Limited.4# Copyright 2014-2015 Canonical Limited.
5#5#
6# This file is part of charm-helpers.6# Licensed under the Apache License, Version 2.0 (the "License");
7#7# you may not use this file except in compliance with the License.
8# charm-helpers is free software: you can redistribute it and/or modify8# You may obtain a copy of the License at
9# it under the terms of the GNU Lesser General Public License version 3 as9#
10# published by the Free Software Foundation.10# http://www.apache.org/licenses/LICENSE-2.0
11#11#
12# charm-helpers is distributed in the hope that it will be useful,12# Unless required by applicable law or agreed to in writing, software
13# but WITHOUT ANY WARRANTY; without even the implied warranty of13# distributed under the License is distributed on an "AS IS" BASIS,
14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the14# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15# GNU Lesser General Public License for more details.15# See the License for the specific language governing permissions and
16#16# limitations under the License.
17# You should have received a copy of the GNU Lesser General Public License17
18# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.18import re
1919import subprocess
20__author__ = "Jorge Niedbalski <jorge.niedbalski@canonical.com>"20
2121from charmhelpers.osplatform import get_platform
22from charmhelpers.core.hookenv import (22from charmhelpers.core.hookenv import (
23 log,23 log,
24 INFO24 INFO
25)25)
2626
27from subprocess import check_call, check_output27__platform__ = get_platform()
28import re28if __platform__ == "ubuntu":
29 from charmhelpers.core.kernel_factory.ubuntu import (
30 persistent_modprobe,
31 update_initramfs,
32 ) # flake8: noqa -- ignore F401 for this import
33elif __platform__ == "centos":
34 from charmhelpers.core.kernel_factory.centos import (
35 persistent_modprobe,
36 update_initramfs,
37 ) # flake8: noqa -- ignore F401 for this import
38
39__author__ = "Jorge Niedbalski <jorge.niedbalski@canonical.com>"
2940
3041
31def modprobe(module, persist=True):42def modprobe(module, persist=True):
@@ -34,11 +45,9 @@
3445
35 log('Loading kernel module %s' % module, level=INFO)46 log('Loading kernel module %s' % module, level=INFO)
3647
37 check_call(cmd)48 subprocess.check_call(cmd)
38 if persist:49 if persist:
39 with open('/etc/modules', 'r+') as modules:50 persistent_modprobe(module)
40 if module not in modules.read():
41 modules.write(module)
4251
4352
44def rmmod(module, force=False):53def rmmod(module, force=False):
@@ -48,21 +57,16 @@
48 cmd.append('-f')57 cmd.append('-f')
49 cmd.append(module)58 cmd.append(module)
50 log('Removing kernel module %s' % module, level=INFO)59 log('Removing kernel module %s' % module, level=INFO)
51 return check_call(cmd)60 return subprocess.check_call(cmd)
5261
5362
54def lsmod():63def lsmod():
55 """Shows what kernel modules are currently loaded"""64 """Shows what kernel modules are currently loaded"""
56 return check_output(['lsmod'],65 return subprocess.check_output(['lsmod'],
57 universal_newlines=True)66 universal_newlines=True)
5867
5968
60def is_module_loaded(module):69def is_module_loaded(module):
61 """Checks if a kernel module is already loaded"""70 """Checks if a kernel module is already loaded"""
62 matches = re.findall('^%s[ ]+' % module, lsmod(), re.M)71 matches = re.findall('^%s[ ]+' % module, lsmod(), re.M)
63 return len(matches) > 072 return len(matches) > 0
64
65
66def update_initramfs(version='all'):
67 """Updates an initramfs image"""
68 return check_call(["update-initramfs", "-k", version, "-u"])
6973
=== added directory 'charmhelpers/core/kernel_factory'
=== added file 'charmhelpers/core/kernel_factory/__init__.py'
=== added file 'charmhelpers/core/kernel_factory/centos.py'
--- charmhelpers/core/kernel_factory/centos.py 1970-01-01 00:00:00 +0000
+++ charmhelpers/core/kernel_factory/centos.py 2017-03-04 02:50:20 +0000
@@ -0,0 +1,17 @@
1import subprocess
2import os
3
4
5def persistent_modprobe(module):
6 """Load a kernel module and configure for auto-load on reboot."""
7 if not os.path.exists('/etc/rc.modules'):
8 open('/etc/rc.modules', 'a')
9 os.chmod('/etc/rc.modules', 111)
10 with open('/etc/rc.modules', 'r+') as modules:
11 if module not in modules.read():
12 modules.write('modprobe %s\n' % module)
13
14
15def update_initramfs(version='all'):
16 """Updates an initramfs image."""
17 return subprocess.check_call(["dracut", "-f", version])
018
=== added file 'charmhelpers/core/kernel_factory/ubuntu.py'
--- charmhelpers/core/kernel_factory/ubuntu.py 1970-01-01 00:00:00 +0000
+++ charmhelpers/core/kernel_factory/ubuntu.py 2017-03-04 02:50:20 +0000
@@ -0,0 +1,13 @@
1import subprocess
2
3
4def persistent_modprobe(module):
5 """Load a kernel module and configure for auto-load on reboot."""
6 with open('/etc/modules', 'r+') as modules:
7 if module not in modules.read():
8 modules.write(module + "\n")
9
10
11def update_initramfs(version='all'):
12 """Updates an initramfs image."""
13 return subprocess.check_call(["update-initramfs", "-k", version, "-u"])
014
=== modified file 'charmhelpers/core/services/__init__.py'
--- charmhelpers/core/services/__init__.py 2015-01-28 08:59:02 +0000
+++ charmhelpers/core/services/__init__.py 2017-03-04 02:50:20 +0000
@@ -1,18 +1,16 @@
1# Copyright 2014-2015 Canonical Limited.1# Copyright 2014-2015 Canonical Limited.
2#2#
3# This file is part of charm-helpers.3# Licensed under the Apache License, Version 2.0 (the "License");
4#4# you may not use this file except in compliance with the License.
5# charm-helpers is free software: you can redistribute it and/or modify5# You may obtain a copy of the License at
6# it under the terms of the GNU Lesser General Public License version 3 as6#
7# published by the Free Software Foundation.7# http://www.apache.org/licenses/LICENSE-2.0
8#8#
9# charm-helpers is distributed in the hope that it will be useful,9# Unless required by applicable law or agreed to in writing, software
10# but WITHOUT ANY WARRANTY; without even the implied warranty of10# distributed under the License is distributed on an "AS IS" BASIS,
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# GNU Lesser General Public License for more details.12# See the License for the specific language governing permissions and
13#13# limitations under the License.
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1614
17from .base import * # NOQA15from .base import * # NOQA
18from .helpers import * # NOQA16from .helpers import * # NOQA
1917
=== modified file 'charmhelpers/core/services/base.py'
--- charmhelpers/core/services/base.py 2015-07-03 09:13:26 +0000
+++ charmhelpers/core/services/base.py 2017-03-04 02:50:20 +0000
@@ -1,18 +1,16 @@
1# Copyright 2014-2015 Canonical Limited.1# Copyright 2014-2015 Canonical Limited.
2#2#
3# This file is part of charm-helpers.3# Licensed under the Apache License, Version 2.0 (the "License");
4#4# you may not use this file except in compliance with the License.
5# charm-helpers is free software: you can redistribute it and/or modify5# You may obtain a copy of the License at
6# it under the terms of the GNU Lesser General Public License version 3 as6#
7# published by the Free Software Foundation.7# http://www.apache.org/licenses/LICENSE-2.0
8#8#
9# charm-helpers is distributed in the hope that it will be useful,9# Unless required by applicable law or agreed to in writing, software
10# but WITHOUT ANY WARRANTY; without even the implied warranty of10# distributed under the License is distributed on an "AS IS" BASIS,
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# GNU Lesser General Public License for more details.12# See the License for the specific language governing permissions and
13#13# limitations under the License.
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1614
17import os15import os
18import json16import json
1917
=== modified file 'charmhelpers/core/services/helpers.py'
--- charmhelpers/core/services/helpers.py 2015-12-11 15:23:38 +0000
+++ charmhelpers/core/services/helpers.py 2017-03-04 02:50:20 +0000
@@ -1,18 +1,16 @@
1# Copyright 2014-2015 Canonical Limited.1# Copyright 2014-2015 Canonical Limited.
2#2#
3# This file is part of charm-helpers.3# Licensed under the Apache License, Version 2.0 (the "License");
4#4# you may not use this file except in compliance with the License.
5# charm-helpers is free software: you can redistribute it and/or modify5# You may obtain a copy of the License at
6# it under the terms of the GNU Lesser General Public License version 3 as6#
7# published by the Free Software Foundation.7# http://www.apache.org/licenses/LICENSE-2.0
8#8#
9# charm-helpers is distributed in the hope that it will be useful,9# Unless required by applicable law or agreed to in writing, software
10# but WITHOUT ANY WARRANTY; without even the implied warranty of10# distributed under the License is distributed on an "AS IS" BASIS,
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# GNU Lesser General Public License for more details.12# See the License for the specific language governing permissions and
13#13# limitations under the License.
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1614
17import os15import os
18import yaml16import yaml
1917
=== modified file 'charmhelpers/core/strutils.py'
--- charmhelpers/core/strutils.py 2015-12-11 15:23:38 +0000
+++ charmhelpers/core/strutils.py 2017-03-04 02:50:20 +0000
@@ -3,19 +3,17 @@
33
4# Copyright 2014-2015 Canonical Limited.4# Copyright 2014-2015 Canonical Limited.
5#5#
6# This file is part of charm-helpers.6# Licensed under the Apache License, Version 2.0 (the "License");
7#7# you may not use this file except in compliance with the License.
8# charm-helpers is free software: you can redistribute it and/or modify8# You may obtain a copy of the License at
9# it under the terms of the GNU Lesser General Public License version 3 as9#
10# published by the Free Software Foundation.10# http://www.apache.org/licenses/LICENSE-2.0
11#11#
12# charm-helpers is distributed in the hope that it will be useful,12# Unless required by applicable law or agreed to in writing, software
13# but WITHOUT ANY WARRANTY; without even the implied warranty of13# distributed under the License is distributed on an "AS IS" BASIS,
14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the14# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15# GNU Lesser General Public License for more details.15# See the License for the specific language governing permissions and
16#16# limitations under the License.
17# You should have received a copy of the GNU Lesser General Public License
18# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1917
20import six18import six
21import re19import re
2220
=== modified file 'charmhelpers/core/sysctl.py'
--- charmhelpers/core/sysctl.py 2015-03-12 11:42:26 +0000
+++ charmhelpers/core/sysctl.py 2017-03-04 02:50:20 +0000
@@ -3,19 +3,17 @@
33
4# Copyright 2014-2015 Canonical Limited.4# Copyright 2014-2015 Canonical Limited.
5#5#
6# This file is part of charm-helpers.6# Licensed under the Apache License, Version 2.0 (the "License");
7#7# you may not use this file except in compliance with the License.
8# charm-helpers is free software: you can redistribute it and/or modify8# You may obtain a copy of the License at
9# it under the terms of the GNU Lesser General Public License version 3 as9#
10# published by the Free Software Foundation.10# http://www.apache.org/licenses/LICENSE-2.0
11#11#
12# charm-helpers is distributed in the hope that it will be useful,12# Unless required by applicable law or agreed to in writing, software
13# but WITHOUT ANY WARRANTY; without even the implied warranty of13# distributed under the License is distributed on an "AS IS" BASIS,
14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the14# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15# GNU Lesser General Public License for more details.15# See the License for the specific language governing permissions and
16#16# limitations under the License.
17# You should have received a copy of the GNU Lesser General Public License
18# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1917
20import yaml18import yaml
2119
2220
=== modified file 'charmhelpers/core/templating.py'
--- charmhelpers/core/templating.py 2015-12-11 15:23:38 +0000
+++ charmhelpers/core/templating.py 2017-03-04 02:50:20 +0000
@@ -1,20 +1,19 @@
1# Copyright 2014-2015 Canonical Limited.1# Copyright 2014-2015 Canonical Limited.
2#2#
3# This file is part of charm-helpers.3# Licensed under the Apache License, Version 2.0 (the "License");
4#4# you may not use this file except in compliance with the License.
5# charm-helpers is free software: you can redistribute it and/or modify5# You may obtain a copy of the License at
6# it under the terms of the GNU Lesser General Public License version 3 as6#
7# published by the Free Software Foundation.7# http://www.apache.org/licenses/LICENSE-2.0
8#8#
9# charm-helpers is distributed in the hope that it will be useful,9# Unless required by applicable law or agreed to in writing, software
10# but WITHOUT ANY WARRANTY; without even the implied warranty of10# distributed under the License is distributed on an "AS IS" BASIS,
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# GNU Lesser General Public License for more details.12# See the License for the specific language governing permissions and
13#13# limitations under the License.
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1614
17import os15import os
16import sys
1817
19from charmhelpers.core import host18from charmhelpers.core import host
20from charmhelpers.core import hookenv19from charmhelpers.core import hookenv
@@ -40,8 +39,9 @@
40 The rendered template will be written to the file as well as being returned39 The rendered template will be written to the file as well as being returned
41 as a string.40 as a string.
4241
43 Note: Using this requires python-jinja2; if it is not installed, calling42 Note: Using this requires python-jinja2 or python3-jinja2; if it is not
44 this will attempt to use charmhelpers.fetch.apt_install to install it.43 installed, calling this will attempt to use charmhelpers.fetch.apt_install
44 to install it.
45 """45 """
46 try:46 try:
47 from jinja2 import FileSystemLoader, Environment, exceptions47 from jinja2 import FileSystemLoader, Environment, exceptions
@@ -53,7 +53,10 @@
53 'charmhelpers.fetch to install it',53 'charmhelpers.fetch to install it',
54 level=hookenv.ERROR)54 level=hookenv.ERROR)
55 raise55 raise
56 apt_install('python-jinja2', fatal=True)56 if sys.version_info.major == 2:
57 apt_install('python-jinja2', fatal=True)
58 else:
59 apt_install('python3-jinja2', fatal=True)
57 from jinja2 import FileSystemLoader, Environment, exceptions60 from jinja2 import FileSystemLoader, Environment, exceptions
5861
59 if template_loader:62 if template_loader:
6063
=== modified file 'charmhelpers/core/unitdata.py'
--- charmhelpers/core/unitdata.py 2015-12-11 15:23:38 +0000
+++ charmhelpers/core/unitdata.py 2017-03-04 02:50:20 +0000
@@ -3,20 +3,17 @@
3#3#
4# Copyright 2014-2015 Canonical Limited.4# Copyright 2014-2015 Canonical Limited.
5#5#
6# This file is part of charm-helpers.6# Licensed under the Apache License, Version 2.0 (the "License");
7#7# you may not use this file except in compliance with the License.
8# charm-helpers is free software: you can redistribute it and/or modify8# You may obtain a copy of the License at
9# it under the terms of the GNU Lesser General Public License version 3 as9#
10# published by the Free Software Foundation.10# http://www.apache.org/licenses/LICENSE-2.0
11#11#
12# charm-helpers is distributed in the hope that it will be useful,12# Unless required by applicable law or agreed to in writing, software
13# but WITHOUT ANY WARRANTY; without even the implied warranty of13# distributed under the License is distributed on an "AS IS" BASIS,
14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the14# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15# GNU Lesser General Public License for more details.15# See the License for the specific language governing permissions and
16#16# limitations under the License.
17# You should have received a copy of the GNU Lesser General Public License
18# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
19#
20#17#
21# Authors:18# Authors:
22# Kapil Thangavelu <kapil.foss@gmail.com>19# Kapil Thangavelu <kapil.foss@gmail.com>
2320
=== modified file 'charmhelpers/fetch/__init__.py'
--- charmhelpers/fetch/__init__.py 2016-05-06 07:12:40 +0000
+++ charmhelpers/fetch/__init__.py 2017-03-04 02:50:20 +0000
@@ -1,32 +1,24 @@
1# Copyright 2014-2015 Canonical Limited.1# Copyright 2014-2015 Canonical Limited.
2#2#
3# This file is part of charm-helpers.3# Licensed under the Apache License, Version 2.0 (the "License");
4#4# you may not use this file except in compliance with the License.
5# charm-helpers is free software: you can redistribute it and/or modify5# You may obtain a copy of the License at
6# it under the terms of the GNU Lesser General Public License version 3 as6#
7# published by the Free Software Foundation.7# http://www.apache.org/licenses/LICENSE-2.0
8#8#
9# charm-helpers is distributed in the hope that it will be useful,9# Unless required by applicable law or agreed to in writing, software
10# but WITHOUT ANY WARRANTY; without even the implied warranty of10# distributed under the License is distributed on an "AS IS" BASIS,
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# GNU Lesser General Public License for more details.12# See the License for the specific language governing permissions and
13#13# limitations under the License.
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1614
17import importlib15import importlib
18from tempfile import NamedTemporaryFile16from charmhelpers.osplatform import get_platform
19import time
20from yaml import safe_load17from yaml import safe_load
21from charmhelpers.core.host import (
22 lsb_release
23)
24import subprocess
25from charmhelpers.core.hookenv import (18from charmhelpers.core.hookenv import (
26 config,19 config,
27 log,20 log,
28)21)
29import os
3022
31import six23import six
32if six.PY3:24if six.PY3:
@@ -35,79 +27,6 @@
35 from urlparse import urlparse, urlunparse27 from urlparse import urlparse, urlunparse
3628
3729
38CLOUD_ARCHIVE = """# Ubuntu Cloud Archive
39deb http://ubuntu-cloud.archive.canonical.com/ubuntu {} main
40"""
41PROPOSED_POCKET = """# Proposed
42deb http://archive.ubuntu.com/ubuntu {}-proposed main universe multiverse restricted
43"""
44CLOUD_ARCHIVE_POCKETS = {
45 # Folsom
46 'folsom': 'precise-updates/folsom',
47 'precise-folsom': 'precise-updates/folsom',
48 'precise-folsom/updates': 'precise-updates/folsom',
49 'precise-updates/folsom': 'precise-updates/folsom',
50 'folsom/proposed': 'precise-proposed/folsom',
51 'precise-folsom/proposed': 'precise-proposed/folsom',
52 'precise-proposed/folsom': 'precise-proposed/folsom',
53 # Grizzly
54 'grizzly': 'precise-updates/grizzly',
55 'precise-grizzly': 'precise-updates/grizzly',
56 'precise-grizzly/updates': 'precise-updates/grizzly',
57 'precise-updates/grizzly': 'precise-updates/grizzly',
58 'grizzly/proposed': 'precise-proposed/grizzly',
59 'precise-grizzly/proposed': 'precise-proposed/grizzly',
60 'precise-proposed/grizzly': 'precise-proposed/grizzly',
61 # Havana
62 'havana': 'precise-updates/havana',
63 'precise-havana': 'precise-updates/havana',
64 'precise-havana/updates': 'precise-updates/havana',
65 'precise-updates/havana': 'precise-updates/havana',
66 'havana/proposed': 'precise-proposed/havana',
67 'precise-havana/proposed': 'precise-proposed/havana',
68 'precise-proposed/havana': 'precise-proposed/havana',
69 # Icehouse
70 'icehouse': 'precise-updates/icehouse',
71 'precise-icehouse': 'precise-updates/icehouse',
72 'precise-icehouse/updates': 'precise-updates/icehouse',
73 'precise-updates/icehouse': 'precise-updates/icehouse',
74 'icehouse/proposed': 'precise-proposed/icehouse',
75 'precise-icehouse/proposed': 'precise-proposed/icehouse',
76 'precise-proposed/icehouse': 'precise-proposed/icehouse',
77 # Juno
78 'juno': 'trusty-updates/juno',
79 'trusty-juno': 'trusty-updates/juno',
80 'trusty-juno/updates': 'trusty-updates/juno',
81 'trusty-updates/juno': 'trusty-updates/juno',
82 'juno/proposed': 'trusty-proposed/juno',
83 'trusty-juno/proposed': 'trusty-proposed/juno',
84 'trusty-proposed/juno': 'trusty-proposed/juno',
85 # Kilo
86 'kilo': 'trusty-updates/kilo',
87 'trusty-kilo': 'trusty-updates/kilo',
88 'trusty-kilo/updates': 'trusty-updates/kilo',
89 'trusty-updates/kilo': 'trusty-updates/kilo',
90 'kilo/proposed': 'trusty-proposed/kilo',
91 'trusty-kilo/proposed': 'trusty-proposed/kilo',
92 'trusty-proposed/kilo': 'trusty-proposed/kilo',
93 # Liberty
94 'liberty': 'trusty-updates/liberty',
95 'trusty-liberty': 'trusty-updates/liberty',
96 'trusty-liberty/updates': 'trusty-updates/liberty',
97 'trusty-updates/liberty': 'trusty-updates/liberty',
98 'liberty/proposed': 'trusty-proposed/liberty',
99 'trusty-liberty/proposed': 'trusty-proposed/liberty',
100 'trusty-proposed/liberty': 'trusty-proposed/liberty',
101 # Mitaka
102 'mitaka': 'trusty-updates/mitaka',
103 'trusty-mitaka': 'trusty-updates/mitaka',
104 'trusty-mitaka/updates': 'trusty-updates/mitaka',
105 'trusty-updates/mitaka': 'trusty-updates/mitaka',
106 'mitaka/proposed': 'trusty-proposed/mitaka',
107 'trusty-mitaka/proposed': 'trusty-proposed/mitaka',
108 'trusty-proposed/mitaka': 'trusty-proposed/mitaka',
109}
110
111# The order of this list is very important. Handlers should be listed in from30# The order of this list is very important. Handlers should be listed in from
112# least- to most-specific URL matching.31# least- to most-specific URL matching.
113FETCH_HANDLERS = (32FETCH_HANDLERS = (
@@ -116,10 +35,6 @@
116 'charmhelpers.fetch.giturl.GitUrlFetchHandler',35 'charmhelpers.fetch.giturl.GitUrlFetchHandler',
117)36)
11837
119APT_NO_LOCK = 100 # The return code for "couldn't acquire lock" in APT.
120APT_NO_LOCK_RETRY_DELAY = 10 # Wait 10 seconds between apt lock checks.
121APT_NO_LOCK_RETRY_COUNT = 30 # Retry to acquire the lock X times.
122
12338
124class SourceConfigError(Exception):39class SourceConfigError(Exception):
125 pass40 pass
@@ -157,180 +72,38 @@
157 return urlunparse(parts)72 return urlunparse(parts)
15873
15974
160def filter_installed_packages(packages):75__platform__ = get_platform()
161 """Returns a list of packages that require installation"""76module = "charmhelpers.fetch.%s" % __platform__
162 cache = apt_cache()77fetch = importlib.import_module(module)
163 _pkgs = []78
164 for package in packages:79filter_installed_packages = fetch.filter_installed_packages
165 try:80install = fetch.install
166 p = cache[package]81upgrade = fetch.upgrade
167 p.current_ver or _pkgs.append(package)82update = fetch.update
168 except KeyError:83purge = fetch.purge
169 log('Package {} has no installation candidate.'.format(package),84add_source = fetch.add_source
170 level='WARNING')85
171 _pkgs.append(package)86if __platform__ == "ubuntu":
172 return _pkgs87 apt_cache = fetch.apt_cache
17388 apt_install = fetch.install
17489 apt_update = fetch.update
175def apt_cache(in_memory=True):90 apt_upgrade = fetch.upgrade
176 """Build and return an apt cache"""91 apt_purge = fetch.purge
177 from apt import apt_pkg92 apt_mark = fetch.apt_mark
178 apt_pkg.init()93 apt_hold = fetch.apt_hold
179 if in_memory:94 apt_unhold = fetch.apt_unhold
180 apt_pkg.config.set("Dir::Cache::pkgcache", "")95 get_upstream_version = fetch.get_upstream_version
181 apt_pkg.config.set("Dir::Cache::srcpkgcache", "")96elif __platform__ == "centos":
182 return apt_pkg.Cache()97 yum_search = fetch.yum_search
183
184
185def apt_install(packages, options=None, fatal=False):
186 """Install one or more packages"""
187 if options is None:
188 options = ['--option=Dpkg::Options::=--force-confold']
189
190 cmd = ['apt-get', '--assume-yes']
191 cmd.extend(options)
192 cmd.append('install')
193 if isinstance(packages, six.string_types):
194 cmd.append(packages)
195 else:
196 cmd.extend(packages)
197 log("Installing {} with options: {}".format(packages,
198 options))
199 _run_apt_command(cmd, fatal)
200
201
202def apt_upgrade(options=None, fatal=False, dist=False):
203 """Upgrade all packages"""
204 if options is None:
205 options = ['--option=Dpkg::Options::=--force-confold']
206
207 cmd = ['apt-get', '--assume-yes']
208 cmd.extend(options)
209 if dist:
210 cmd.append('dist-upgrade')
211 else:
212 cmd.append('upgrade')
213 log("Upgrading with options: {}".format(options))
214 _run_apt_command(cmd, fatal)
215
216
217def apt_update(fatal=False):
218 """Update local apt cache"""
219 cmd = ['apt-get', 'update']
220 _run_apt_command(cmd, fatal)
221
222
223def apt_purge(packages, fatal=False):
224 """Purge one or more packages"""
225 cmd = ['apt-get', '--assume-yes', 'purge']
226 if isinstance(packages, six.string_types):
227 cmd.append(packages)
228 else:
229 cmd.extend(packages)
230 log("Purging {}".format(packages))
231 _run_apt_command(cmd, fatal)
232
233
234def apt_mark(packages, mark, fatal=False):
235 """Flag one or more packages using apt-mark"""
236 log("Marking {} as {}".format(packages, mark))
237 cmd = ['apt-mark', mark]
238 if isinstance(packages, six.string_types):
239 cmd.append(packages)
240 else:
241 cmd.extend(packages)
242
243 if fatal:
244 subprocess.check_call(cmd, universal_newlines=True)
245 else:
246 subprocess.call(cmd, universal_newlines=True)
247
248
249def apt_hold(packages, fatal=False):
250 return apt_mark(packages, 'hold', fatal=fatal)
251
252
253def apt_unhold(packages, fatal=False):
254 return apt_mark(packages, 'unhold', fatal=fatal)
255
256
257def add_source(source, key=None):
258 """Add a package source to this system.
259
260 @param source: a URL or sources.list entry, as supported by
261 add-apt-repository(1). Examples::
262
263 ppa:charmers/example
264 deb https://stub:key@private.example.com/ubuntu trusty main
265
266 In addition:
267 'proposed:' may be used to enable the standard 'proposed'
268 pocket for the release.
269 'cloud:' may be used to activate official cloud archive pockets,
270 such as 'cloud:icehouse'
271 'distro' may be used as a noop
272
273 @param key: A key to be added to the system's APT keyring and used
274 to verify the signatures on packages. Ideally, this should be an
275 ASCII format GPG public key including the block headers. A GPG key
276 id may also be used, but be aware that only insecure protocols are
277 available to retrieve the actual public key from a public keyserver
278 placing your Juju environment at risk. ppa and cloud archive keys
279 are securely added automtically, so sould not be provided.
280 """
281 if source is None:
282 log('Source is not present. Skipping')
283 return
284
285 if (source.startswith('ppa:') or
286 source.startswith('http') or
287 source.startswith('deb ') or
288 source.startswith('cloud-archive:')):
289 subprocess.check_call(['add-apt-repository', '--yes', source])
290 elif source.startswith('cloud:'):
291 apt_install(filter_installed_packages(['ubuntu-cloud-keyring']),
292 fatal=True)
293 pocket = source.split(':')[-1]
294 if pocket not in CLOUD_ARCHIVE_POCKETS:
295 raise SourceConfigError(
296 'Unsupported cloud: source option %s' %
297 pocket)
298 actual_pocket = CLOUD_ARCHIVE_POCKETS[pocket]
299 with open('/etc/apt/sources.list.d/cloud-archive.list', 'w') as apt:
300 apt.write(CLOUD_ARCHIVE.format(actual_pocket))
301 elif source == 'proposed':
302 release = lsb_release()['DISTRIB_CODENAME']
303 with open('/etc/apt/sources.list.d/proposed.list', 'w') as apt:
304 apt.write(PROPOSED_POCKET.format(release))
305 elif source == 'distro':
306 pass
307 else:
308 log("Unknown source: {!r}".format(source))
309
310 if key:
311 if '-----BEGIN PGP PUBLIC KEY BLOCK-----' in key:
312 with NamedTemporaryFile('w+') as key_file:
313 key_file.write(key)
314 key_file.flush()
315 key_file.seek(0)
316 subprocess.check_call(['apt-key', 'add', '-'], stdin=key_file)
317 else:
318 # Note that hkp: is in no way a secure protocol. Using a
319 # GPG key id is pointless from a security POV unless you
320 # absolutely trust your network and DNS.
321 subprocess.check_call(['apt-key', 'adv', '--keyserver',
322 'hkp://keyserver.ubuntu.com:80', '--recv',
323 key])
32498
32599
326def configure_sources(update=False,100def configure_sources(update=False,
327 sources_var='install_sources',101 sources_var='install_sources',
328 keys_var='install_keys'):102 keys_var='install_keys'):
329 """103 """Configure multiple sources from charm configuration.
330 Configure multiple sources from charm configuration.
331104
332 The lists are encoded as yaml fragments in the configuration.105 The lists are encoded as yaml fragments in the configuration.
333 The frament needs to be included as a string. Sources and their106 The fragment needs to be included as a string. Sources and their
334 corresponding keys are of the types supported by add_source().107 corresponding keys are of the types supported by add_source().
335108
336 Example config:109 Example config:
@@ -362,12 +135,11 @@
362 for source, key in zip(sources, keys):135 for source, key in zip(sources, keys):
363 add_source(source, key)136 add_source(source, key)
364 if update:137 if update:
365 apt_update(fatal=True)138 fetch.update(fatal=True)
366139
367140
368def install_remote(source, *args, **kwargs):141def install_remote(source, *args, **kwargs):
369 """142 """Install a file tree from a remote source.
370 Install a file tree from a remote source
371143
372 The specified source should be a url of the form:144 The specified source should be a url of the form:
373 scheme://[host]/path[#[option=value][&...]]145 scheme://[host]/path[#[option=value][&...]]
@@ -390,19 +162,17 @@
390 # We ONLY check for True here because can_handle may return a string162 # We ONLY check for True here because can_handle may return a string
391 # explaining why it can't handle a given source.163 # explaining why it can't handle a given source.
392 handlers = [h for h in plugins() if h.can_handle(source) is True]164 handlers = [h for h in plugins() if h.can_handle(source) is True]
393 installed_to = None
394 for handler in handlers:165 for handler in handlers:
395 try:166 try:
396 installed_to = handler.install(source, *args, **kwargs)167 return handler.install(source, *args, **kwargs)
397 except UnhandledSource as e:168 except UnhandledSource as e:
398 log('Install source attempt unsuccessful: {}'.format(e),169 log('Install source attempt unsuccessful: {}'.format(e),
399 level='WARNING')170 level='WARNING')
400 if not installed_to:171 raise UnhandledSource("No handler found for source {}".format(source))
401 raise UnhandledSource("No handler found for source {}".format(source))
402 return installed_to
403172
404173
405def install_from_config(config_var_name):174def install_from_config(config_var_name):
175 """Install a file from config."""
406 charm_config = config()176 charm_config = config()
407 source = charm_config[config_var_name]177 source = charm_config[config_var_name]
408 return install_remote(source)178 return install_remote(source)
@@ -425,40 +195,3 @@
425 log("FetchHandler {} not found, skipping plugin".format(195 log("FetchHandler {} not found, skipping plugin".format(
426 handler_name))196 handler_name))
427 return plugin_list197 return plugin_list
428
429
430def _run_apt_command(cmd, fatal=False):
431 """
432 Run an APT command, checking output and retrying if the fatal flag is set
433 to True.
434
435 :param: cmd: str: The apt command to run.
436 :param: fatal: bool: Whether the command's output should be checked and
437 retried.
438 """
439 env = os.environ.copy()
440
441 if 'DEBIAN_FRONTEND' not in env:
442 env['DEBIAN_FRONTEND'] = 'noninteractive'
443
444 if fatal:
445 retry_count = 0
446 result = None
447
448 # If the command is considered "fatal", we need to retry if the apt
449 # lock was not acquired.
450
451 while result is None or result == APT_NO_LOCK:
452 try:
453 result = subprocess.check_call(cmd, env=env)
454 except subprocess.CalledProcessError as e:
455 retry_count = retry_count + 1
456 if retry_count > APT_NO_LOCK_RETRY_COUNT:
457 raise
458 result = e.returncode
459 log("Couldn't acquire DPKG lock. Will retry in {} seconds."
460 "".format(APT_NO_LOCK_RETRY_DELAY))
461 time.sleep(APT_NO_LOCK_RETRY_DELAY)
462
463 else:
464 subprocess.call(cmd, env=env)
465198
=== modified file 'charmhelpers/fetch/archiveurl.py'
--- charmhelpers/fetch/archiveurl.py 2015-12-11 15:23:38 +0000
+++ charmhelpers/fetch/archiveurl.py 2017-03-04 02:50:20 +0000
@@ -1,18 +1,16 @@
1# Copyright 2014-2015 Canonical Limited.1# Copyright 2014-2015 Canonical Limited.
2#2#
3# This file is part of charm-helpers.3# Licensed under the Apache License, Version 2.0 (the "License");
4#4# you may not use this file except in compliance with the License.
5# charm-helpers is free software: you can redistribute it and/or modify5# You may obtain a copy of the License at
6# it under the terms of the GNU Lesser General Public License version 3 as6#
7# published by the Free Software Foundation.7# http://www.apache.org/licenses/LICENSE-2.0
8#8#
9# charm-helpers is distributed in the hope that it will be useful,9# Unless required by applicable law or agreed to in writing, software
10# but WITHOUT ANY WARRANTY; without even the implied warranty of10# distributed under the License is distributed on an "AS IS" BASIS,
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# GNU Lesser General Public License for more details.12# See the License for the specific language governing permissions and
13#13# limitations under the License.
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1614
17import os15import os
18import hashlib16import hashlib
1917
=== modified file 'charmhelpers/fetch/bzrurl.py'
--- charmhelpers/fetch/bzrurl.py 2016-05-06 07:12:40 +0000
+++ charmhelpers/fetch/bzrurl.py 2017-03-04 02:50:20 +0000
@@ -1,18 +1,16 @@
1# Copyright 2014-2015 Canonical Limited.1# Copyright 2014-2015 Canonical Limited.
2#2#
3# This file is part of charm-helpers.3# Licensed under the Apache License, Version 2.0 (the "License");
4#4# you may not use this file except in compliance with the License.
5# charm-helpers is free software: you can redistribute it and/or modify5# You may obtain a copy of the License at
6# it under the terms of the GNU Lesser General Public License version 3 as6#
7# published by the Free Software Foundation.7# http://www.apache.org/licenses/LICENSE-2.0
8#8#
9# charm-helpers is distributed in the hope that it will be useful,9# Unless required by applicable law or agreed to in writing, software
10# but WITHOUT ANY WARRANTY; without even the implied warranty of10# distributed under the License is distributed on an "AS IS" BASIS,
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# GNU Lesser General Public License for more details.12# See the License for the specific language governing permissions and
13#13# limitations under the License.
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1614
17import os15import os
18from subprocess import check_call16from subprocess import check_call
@@ -20,19 +18,20 @@
20 BaseFetchHandler,18 BaseFetchHandler,
21 UnhandledSource,19 UnhandledSource,
22 filter_installed_packages,20 filter_installed_packages,
23 apt_install,21 install,
24)22)
25from charmhelpers.core.host import mkdir23from charmhelpers.core.host import mkdir
2624
2725
28if filter_installed_packages(['bzr']) != []:26if filter_installed_packages(['bzr']) != []:
29 apt_install(['bzr'])27 install(['bzr'])
30 if filter_installed_packages(['bzr']) != []:28 if filter_installed_packages(['bzr']) != []:
31 raise NotImplementedError('Unable to install bzr')29 raise NotImplementedError('Unable to install bzr')
3230
3331
34class BzrUrlFetchHandler(BaseFetchHandler):32class BzrUrlFetchHandler(BaseFetchHandler):
35 """Handler for bazaar branches via generic and lp URLs"""33 """Handler for bazaar branches via generic and lp URLs."""
34
36 def can_handle(self, source):35 def can_handle(self, source):
37 url_parts = self.parse_url(source)36 url_parts = self.parse_url(source)
38 if url_parts.scheme not in ('bzr+ssh', 'lp', ''):37 if url_parts.scheme not in ('bzr+ssh', 'lp', ''):
@@ -42,15 +41,23 @@
42 else:41 else:
43 return True42 return True
4443
45 def branch(self, source, dest):44 def branch(self, source, dest, revno=None):
46 if not self.can_handle(source):45 if not self.can_handle(source):
47 raise UnhandledSource("Cannot handle {}".format(source))46 raise UnhandledSource("Cannot handle {}".format(source))
47 cmd_opts = []
48 if revno:
49 cmd_opts += ['-r', str(revno)]
48 if os.path.exists(dest):50 if os.path.exists(dest):
49 check_call(['bzr', 'pull', '--overwrite', '-d', dest, source])51 cmd = ['bzr', 'pull']
52 cmd += cmd_opts
53 cmd += ['--overwrite', '-d', dest, source]
50 else:54 else:
51 check_call(['bzr', 'branch', source, dest])55 cmd = ['bzr', 'branch']
56 cmd += cmd_opts
57 cmd += [source, dest]
58 check_call(cmd)
5259
53 def install(self, source, dest=None):60 def install(self, source, dest=None, revno=None):
54 url_parts = self.parse_url(source)61 url_parts = self.parse_url(source)
55 branch_name = url_parts.path.strip("/").split("/")[-1]62 branch_name = url_parts.path.strip("/").split("/")[-1]
56 if dest:63 if dest:
@@ -59,10 +66,11 @@
59 dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",66 dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
60 branch_name)67 branch_name)
6168
62 if not os.path.exists(dest_dir):69 if dest and not os.path.exists(dest):
63 mkdir(dest_dir, perms=0o755)70 mkdir(dest, perms=0o755)
71
64 try:72 try:
65 self.branch(source, dest_dir)73 self.branch(source, dest_dir, revno)
66 except OSError as e:74 except OSError as e:
67 raise UnhandledSource(e.strerror)75 raise UnhandledSource(e.strerror)
68 return dest_dir76 return dest_dir
6977
=== added file 'charmhelpers/fetch/centos.py'
--- charmhelpers/fetch/centos.py 1970-01-01 00:00:00 +0000
+++ charmhelpers/fetch/centos.py 2017-03-04 02:50:20 +0000
@@ -0,0 +1,171 @@
1# Copyright 2014-2015 Canonical Limited.
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
15import subprocess
16import os
17import time
18import six
19import yum
20
21from tempfile import NamedTemporaryFile
22from charmhelpers.core.hookenv import log
23
24YUM_NO_LOCK = 1 # The return code for "couldn't acquire lock" in YUM.
25YUM_NO_LOCK_RETRY_DELAY = 10 # Wait 10 seconds between apt lock checks.
26YUM_NO_LOCK_RETRY_COUNT = 30 # Retry to acquire the lock X times.
27
28
29def filter_installed_packages(packages):
30 """Return a list of packages that require installation."""
31 yb = yum.YumBase()
32 package_list = yb.doPackageLists()
33 temp_cache = {p.base_package_name: 1 for p in package_list['installed']}
34
35 _pkgs = [p for p in packages if not temp_cache.get(p, False)]
36 return _pkgs
37
38
39def install(packages, options=None, fatal=False):
40 """Install one or more packages."""
41 cmd = ['yum', '--assumeyes']
42 if options is not None:
43 cmd.extend(options)
44 cmd.append('install')
45 if isinstance(packages, six.string_types):
46 cmd.append(packages)
47 else:
48 cmd.extend(packages)
49 log("Installing {} with options: {}".format(packages,
50 options))
51 _run_yum_command(cmd, fatal)
52
53
54def upgrade(options=None, fatal=False, dist=False):
55 """Upgrade all packages."""
56 cmd = ['yum', '--assumeyes']
57 if options is not None:
58 cmd.extend(options)
59 cmd.append('upgrade')
60 log("Upgrading with options: {}".format(options))
61 _run_yum_command(cmd, fatal)
62
63
64def update(fatal=False):
65 """Update local yum cache."""
66 cmd = ['yum', '--assumeyes', 'update']
67 log("Update with fatal: {}".format(fatal))
68 _run_yum_command(cmd, fatal)
69
70
71def purge(packages, fatal=False):
72 """Purge one or more packages."""
73 cmd = ['yum', '--assumeyes', 'remove']
74 if isinstance(packages, six.string_types):
75 cmd.append(packages)
76 else:
77 cmd.extend(packages)
78 log("Purging {}".format(packages))
79 _run_yum_command(cmd, fatal)
80
81
82def yum_search(packages):
83 """Search for a package."""
84 output = {}
85 cmd = ['yum', 'search']
86 if isinstance(packages, six.string_types):
87 cmd.append(packages)
88 else:
89 cmd.extend(packages)
90 log("Searching for {}".format(packages))
91 result = subprocess.check_output(cmd)
92 for package in list(packages):
93 output[package] = package in result
94 return output
95
96
97def add_source(source, key=None):
98 """Add a package source to this system.
99
100 @param source: a URL with a rpm package
101
102 @param key: A key to be added to the system's keyring and used
103 to verify the signatures on packages. Ideally, this should be an
104 ASCII format GPG public key including the block headers. A GPG key
105 id may also be used, but be aware that only insecure protocols are
106 available to retrieve the actual public key from a public keyserver
107 placing your Juju environment at risk.
108 """
109 if source is None:
110 log('Source is not present. Skipping')
111 return
112
113 if source.startswith('http'):
114 directory = '/etc/yum.repos.d/'
115 for filename in os.listdir(directory):
116 with open(directory + filename, 'r') as rpm_file:
117 if source in rpm_file.read():
118 break
119 else:
120 log("Add source: {!r}".format(source))
121 # write in the charms.repo
122 with open(directory + 'Charms.repo', 'a') as rpm_file:
123 rpm_file.write('[%s]\n' % source[7:].replace('/', '_'))
124 rpm_file.write('name=%s\n' % source[7:])
125 rpm_file.write('baseurl=%s\n\n' % source)
126 else:
127 log("Unknown source: {!r}".format(source))
128
129 if key:
130 if '-----BEGIN PGP PUBLIC KEY BLOCK-----' in key:
131 with NamedTemporaryFile('w+') as key_file:
132 key_file.write(key)
133 key_file.flush()
134 key_file.seek(0)
135 subprocess.check_call(['rpm', '--import', key_file])
136 else:
137 subprocess.check_call(['rpm', '--import', key])
138
139
140def _run_yum_command(cmd, fatal=False):
141 """Run an YUM command.
142
143 Checks the output and retry if the fatal flag is set to True.
144
145 :param: cmd: str: The yum command to run.
146 :param: fatal: bool: Whether the command's output should be checked and
147 retried.
148 """
149 env = os.environ.copy()
150
151 if fatal:
152 retry_count = 0
153 result = None
154
155 # If the command is considered "fatal", we need to retry if the yum
156 # lock was not acquired.
157
158 while result is None or result == YUM_NO_LOCK:
159 try:
160 result = subprocess.check_call(cmd, env=env)
161 except subprocess.CalledProcessError as e:
162 retry_count = retry_count + 1
163 if retry_count > YUM_NO_LOCK_RETRY_COUNT:
164 raise
165 result = e.returncode
166 log("Couldn't acquire YUM lock. Will retry in {} seconds."
167 "".format(YUM_NO_LOCK_RETRY_DELAY))
168 time.sleep(YUM_NO_LOCK_RETRY_DELAY)
169
170 else:
171 subprocess.call(cmd, env=env)
0172
=== modified file 'charmhelpers/fetch/giturl.py'
--- charmhelpers/fetch/giturl.py 2016-05-06 07:12:40 +0000
+++ charmhelpers/fetch/giturl.py 2017-03-04 02:50:20 +0000
@@ -1,18 +1,16 @@
1# Copyright 2014-2015 Canonical Limited.1# Copyright 2014-2015 Canonical Limited.
2#2#
3# This file is part of charm-helpers.3# Licensed under the Apache License, Version 2.0 (the "License");
4#4# you may not use this file except in compliance with the License.
5# charm-helpers is free software: you can redistribute it and/or modify5# You may obtain a copy of the License at
6# it under the terms of the GNU Lesser General Public License version 3 as6#
7# published by the Free Software Foundation.7# http://www.apache.org/licenses/LICENSE-2.0
8#8#
9# charm-helpers is distributed in the hope that it will be useful,9# Unless required by applicable law or agreed to in writing, software
10# but WITHOUT ANY WARRANTY; without even the implied warranty of10# distributed under the License is distributed on an "AS IS" BASIS,
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# GNU Lesser General Public License for more details.12# See the License for the specific language governing permissions and
13#13# limitations under the License.
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1614
17import os15import os
18from subprocess import check_call, CalledProcessError16from subprocess import check_call, CalledProcessError
@@ -20,17 +18,18 @@
20 BaseFetchHandler,18 BaseFetchHandler,
21 UnhandledSource,19 UnhandledSource,
22 filter_installed_packages,20 filter_installed_packages,
23 apt_install,21 install,
24)22)
2523
26if filter_installed_packages(['git']) != []:24if filter_installed_packages(['git']) != []:
27 apt_install(['git'])25 install(['git'])
28 if filter_installed_packages(['git']) != []:26 if filter_installed_packages(['git']) != []:
29 raise NotImplementedError('Unable to install git')27 raise NotImplementedError('Unable to install git')
3028
3129
32class GitUrlFetchHandler(BaseFetchHandler):30class GitUrlFetchHandler(BaseFetchHandler):
33 """Handler for git branches via generic and github URLs"""31 """Handler for git branches via generic and github URLs."""
32
34 def can_handle(self, source):33 def can_handle(self, source):
35 url_parts = self.parse_url(source)34 url_parts = self.parse_url(source)
36 # TODO (mattyw) no support for ssh git@ yet35 # TODO (mattyw) no support for ssh git@ yet
3736
=== added file 'charmhelpers/fetch/snap.py'
--- charmhelpers/fetch/snap.py 1970-01-01 00:00:00 +0000
+++ charmhelpers/fetch/snap.py 2017-03-04 02:50:20 +0000
@@ -0,0 +1,122 @@
1# Copyright 2014-2017 Canonical Limited.
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14"""
15Charm helpers snap for classic charms.
16
17If writing reactive charms, use the snap layer:
18https://lists.ubuntu.com/archives/snapcraft/2016-September/001114.html
19"""
20import subprocess
21from os import environ
22from time import sleep
23from charmhelpers.core.hookenv import log
24
25__author__ = 'Joseph Borg <joseph.borg@canonical.com>'
26
27SNAP_NO_LOCK = 1 # The return code for "couldn't acquire lock" in Snap (hopefully this will be improved).
28SNAP_NO_LOCK_RETRY_DELAY = 10 # Wait X seconds between Snap lock checks.
29SNAP_NO_LOCK_RETRY_COUNT = 30 # Retry to acquire the lock X times.
30
31
32class CouldNotAcquireLockException(Exception):
33 pass
34
35
36def _snap_exec(commands):
37 """
38 Execute snap commands.
39
40 :param commands: List commands
41 :return: Integer exit code
42 """
43 assert type(commands) == list
44
45 retry_count = 0
46 return_code = None
47
48 while return_code is None or return_code == SNAP_NO_LOCK:
49 try:
50 return_code = subprocess.check_call(['snap'] + commands, env=environ)
51 except subprocess.CalledProcessError as e:
52 retry_count += + 1
53 if retry_count > SNAP_NO_LOCK_RETRY_COUNT:
54 raise CouldNotAcquireLockException('Could not aquire lock after %s attempts' % SNAP_NO_LOCK_RETRY_COUNT)
55 return_code = e.returncode
56 log('Snap failed to acquire lock, trying again in %s seconds.' % SNAP_NO_LOCK_RETRY_DELAY, level='WARN')
57 sleep(SNAP_NO_LOCK_RETRY_DELAY)
58
59 return return_code
60
61
62def snap_install(packages, *flags):
63 """
64 Install a snap package.
65
66 :param packages: String or List String package name
67 :param flags: List String flags to pass to install command
68 :return: Integer return code from snap
69 """
70 if type(packages) is not list:
71 packages = [packages]
72
73 flags = list(flags)
74
75 message = 'Installing snap(s) "%s"' % ', '.join(packages)
76 if flags:
77 message += ' with option(s) "%s"' % ', '.join(flags)
78
79 log(message, level='INFO')
80 return _snap_exec(['install'] + flags + packages)
81
82
83def snap_remove(packages, *flags):
84 """
85 Remove a snap package.
86
87 :param packages: String or List String package name
88 :param flags: List String flags to pass to remove command
89 :return: Integer return code from snap
90 """
91 if type(packages) is not list:
92 packages = [packages]
93
94 flags = list(flags)
95
96 message = 'Removing snap(s) "%s"' % ', '.join(packages)
97 if flags:
98 message += ' with options "%s"' % ', '.join(flags)
99
100 log(message, level='INFO')
101 return _snap_exec(['remove'] + flags + packages)
102
103
104def snap_refresh(packages, *flags):
105 """
106 Refresh / Update snap package.
107
108 :param packages: String or List String package name
109 :param flags: List String flags to pass to refresh command
110 :return: Integer return code from snap
111 """
112 if type(packages) is not list:
113 packages = [packages]
114
115 flags = list(flags)
116
117 message = 'Refreshing snap(s) "%s"' % ', '.join(packages)
118 if flags:
119 message += ' with options "%s"' % ', '.join(flags)
120
121 log(message, level='INFO')
122 return _snap_exec(['refresh'] + flags + packages)
0123
=== added file 'charmhelpers/fetch/ubuntu.py'
--- charmhelpers/fetch/ubuntu.py 1970-01-01 00:00:00 +0000
+++ charmhelpers/fetch/ubuntu.py 2017-03-04 02:50:20 +0000
@@ -0,0 +1,364 @@
1# Copyright 2014-2015 Canonical Limited.
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
15import os
16import six
17import time
18import subprocess
19
20from tempfile import NamedTemporaryFile
21from charmhelpers.core.host import (
22 lsb_release
23)
24from charmhelpers.core.hookenv import log
25from charmhelpers.fetch import SourceConfigError
26
27CLOUD_ARCHIVE = """# Ubuntu Cloud Archive
28deb http://ubuntu-cloud.archive.canonical.com/ubuntu {} main
29"""
30
31PROPOSED_POCKET = """# Proposed
32deb http://archive.ubuntu.com/ubuntu {}-proposed main universe multiverse restricted
33"""
34
35CLOUD_ARCHIVE_POCKETS = {
36 # Folsom
37 'folsom': 'precise-updates/folsom',
38 'precise-folsom': 'precise-updates/folsom',
39 'precise-folsom/updates': 'precise-updates/folsom',
40 'precise-updates/folsom': 'precise-updates/folsom',
41 'folsom/proposed': 'precise-proposed/folsom',
42 'precise-folsom/proposed': 'precise-proposed/folsom',
43 'precise-proposed/folsom': 'precise-proposed/folsom',
44 # Grizzly
45 'grizzly': 'precise-updates/grizzly',
46 'precise-grizzly': 'precise-updates/grizzly',
47 'precise-grizzly/updates': 'precise-updates/grizzly',
48 'precise-updates/grizzly': 'precise-updates/grizzly',
49 'grizzly/proposed': 'precise-proposed/grizzly',
50 'precise-grizzly/proposed': 'precise-proposed/grizzly',
51 'precise-proposed/grizzly': 'precise-proposed/grizzly',
52 # Havana
53 'havana': 'precise-updates/havana',
54 'precise-havana': 'precise-updates/havana',
55 'precise-havana/updates': 'precise-updates/havana',
56 'precise-updates/havana': 'precise-updates/havana',
57 'havana/proposed': 'precise-proposed/havana',
58 'precise-havana/proposed': 'precise-proposed/havana',
59 'precise-proposed/havana': 'precise-proposed/havana',
60 # Icehouse
61 'icehouse': 'precise-updates/icehouse',
62 'precise-icehouse': 'precise-updates/icehouse',
63 'precise-icehouse/updates': 'precise-updates/icehouse',
64 'precise-updates/icehouse': 'precise-updates/icehouse',
65 'icehouse/proposed': 'precise-proposed/icehouse',
66 'precise-icehouse/proposed': 'precise-proposed/icehouse',
67 'precise-proposed/icehouse': 'precise-proposed/icehouse',
68 # Juno
69 'juno': 'trusty-updates/juno',
70 'trusty-juno': 'trusty-updates/juno',
71 'trusty-juno/updates': 'trusty-updates/juno',
72 'trusty-updates/juno': 'trusty-updates/juno',
73 'juno/proposed': 'trusty-proposed/juno',
74 'trusty-juno/proposed': 'trusty-proposed/juno',
75 'trusty-proposed/juno': 'trusty-proposed/juno',
76 # Kilo
77 'kilo': 'trusty-updates/kilo',
78 'trusty-kilo': 'trusty-updates/kilo',
79 'trusty-kilo/updates': 'trusty-updates/kilo',
80 'trusty-updates/kilo': 'trusty-updates/kilo',
81 'kilo/proposed': 'trusty-proposed/kilo',
82 'trusty-kilo/proposed': 'trusty-proposed/kilo',
83 'trusty-proposed/kilo': 'trusty-proposed/kilo',
84 # Liberty
85 'liberty': 'trusty-updates/liberty',
86 'trusty-liberty': 'trusty-updates/liberty',
87 'trusty-liberty/updates': 'trusty-updates/liberty',
88 'trusty-updates/liberty': 'trusty-updates/liberty',
89 'liberty/proposed': 'trusty-proposed/liberty',
90 'trusty-liberty/proposed': 'trusty-proposed/liberty',
91 'trusty-proposed/liberty': 'trusty-proposed/liberty',
92 # Mitaka
93 'mitaka': 'trusty-updates/mitaka',
94 'trusty-mitaka': 'trusty-updates/mitaka',
95 'trusty-mitaka/updates': 'trusty-updates/mitaka',
96 'trusty-updates/mitaka': 'trusty-updates/mitaka',
97 'mitaka/proposed': 'trusty-proposed/mitaka',
98 'trusty-mitaka/proposed': 'trusty-proposed/mitaka',
99 'trusty-proposed/mitaka': 'trusty-proposed/mitaka',
100 # Newton
101 'newton': 'xenial-updates/newton',
102 'xenial-newton': 'xenial-updates/newton',
103 'xenial-newton/updates': 'xenial-updates/newton',
104 'xenial-updates/newton': 'xenial-updates/newton',
105 'newton/proposed': 'xenial-proposed/newton',
106 'xenial-newton/proposed': 'xenial-proposed/newton',
107 'xenial-proposed/newton': 'xenial-proposed/newton',
108 # Ocata
109 'ocata': 'xenial-updates/ocata',
110 'xenial-ocata': 'xenial-updates/ocata',
111 'xenial-ocata/updates': 'xenial-updates/ocata',
112 'xenial-updates/ocata': 'xenial-updates/ocata',
113 'ocata/proposed': 'xenial-proposed/ocata',
114 'xenial-ocata/proposed': 'xenial-proposed/ocata',
115 'xenial-ocata/newton': 'xenial-proposed/ocata',
116}
117
118APT_NO_LOCK = 100 # The return code for "couldn't acquire lock" in APT.
119CMD_RETRY_DELAY = 10 # Wait 10 seconds between command retries.
120CMD_RETRY_COUNT = 30 # Retry a failing fatal command X times.
121
122
123def filter_installed_packages(packages):
124 """Return a list of packages that require installation."""
125 cache = apt_cache()
126 _pkgs = []
127 for package in packages:
128 try:
129 p = cache[package]
130 p.current_ver or _pkgs.append(package)
131 except KeyError:
132 log('Package {} has no installation candidate.'.format(package),
133 level='WARNING')
134 _pkgs.append(package)
135 return _pkgs
136
137
138def apt_cache(in_memory=True, progress=None):
139 """Build and return an apt cache."""
140 from apt import apt_pkg
141 apt_pkg.init()
142 if in_memory:
143 apt_pkg.config.set("Dir::Cache::pkgcache", "")
144 apt_pkg.config.set("Dir::Cache::srcpkgcache", "")
145 return apt_pkg.Cache(progress)
146
147
148def install(packages, options=None, fatal=False):
149 """Install one or more packages."""
150 if options is None:
151 options = ['--option=Dpkg::Options::=--force-confold']
152
153 cmd = ['apt-get', '--assume-yes']
154 cmd.extend(options)
155 cmd.append('install')
156 if isinstance(packages, six.string_types):
157 cmd.append(packages)
158 else:
159 cmd.extend(packages)
160 log("Installing {} with options: {}".format(packages,
161 options))
162 _run_apt_command(cmd, fatal)
163
164
165def upgrade(options=None, fatal=False, dist=False):
166 """Upgrade all packages."""
167 if options is None:
168 options = ['--option=Dpkg::Options::=--force-confold']
169
170 cmd = ['apt-get', '--assume-yes']
171 cmd.extend(options)
172 if dist:
173 cmd.append('dist-upgrade')
174 else:
175 cmd.append('upgrade')
176 log("Upgrading with options: {}".format(options))
177 _run_apt_command(cmd, fatal)
178
179
180def update(fatal=False):
181 """Update local apt cache."""
182 cmd = ['apt-get', 'update']
183 _run_apt_command(cmd, fatal)
184
185
186def purge(packages, fatal=False):
187 """Purge one or more packages."""
188 cmd = ['apt-get', '--assume-yes', 'purge']
189 if isinstance(packages, six.string_types):
190 cmd.append(packages)
191 else:
192 cmd.extend(packages)
193 log("Purging {}".format(packages))
194 _run_apt_command(cmd, fatal)
195
196
197def apt_mark(packages, mark, fatal=False):
198 """Flag one or more packages using apt-mark."""
199 log("Marking {} as {}".format(packages, mark))
200 cmd = ['apt-mark', mark]
201 if isinstance(packages, six.string_types):
202 cmd.append(packages)
203 else:
204 cmd.extend(packages)
205
206 if fatal:
207 subprocess.check_call(cmd, universal_newlines=True)
208 else:
209 subprocess.call(cmd, universal_newlines=True)
210
211
212def apt_hold(packages, fatal=False):
213 return apt_mark(packages, 'hold', fatal=fatal)
214
215
216def apt_unhold(packages, fatal=False):
217 return apt_mark(packages, 'unhold', fatal=fatal)
218
219
220def add_source(source, key=None):
221 """Add a package source to this system.
222
223 @param source: a URL or sources.list entry, as supported by
224 add-apt-repository(1). Examples::
225
226 ppa:charmers/example
227 deb https://stub:key@private.example.com/ubuntu trusty main
228
229 In addition:
230 'proposed:' may be used to enable the standard 'proposed'
231 pocket for the release.
232 'cloud:' may be used to activate official cloud archive pockets,
233 such as 'cloud:icehouse'
234 'distro' may be used as a noop
235
236 @param key: A key to be added to the system's APT keyring and used
237 to verify the signatures on packages. Ideally, this should be an
238 ASCII format GPG public key including the block headers. A GPG key
239 id may also be used, but be aware that only insecure protocols are
240 available to retrieve the actual public key from a public keyserver
241 placing your Juju environment at risk. ppa and cloud archive keys
242 are securely added automtically, so sould not be provided.
243 """
244 if source is None:
245 log('Source is not present. Skipping')
246 return
247
248 if (source.startswith('ppa:') or
249 source.startswith('http') or
250 source.startswith('deb ') or
251 source.startswith('cloud-archive:')):
252 cmd = ['add-apt-repository', '--yes', source]
253 _run_with_retries(cmd)
254 elif source.startswith('cloud:'):
255 install(filter_installed_packages(['ubuntu-cloud-keyring']),
256 fatal=True)
257 pocket = source.split(':')[-1]
258 if pocket not in CLOUD_ARCHIVE_POCKETS:
259 raise SourceConfigError(
260 'Unsupported cloud: source option %s' %
261 pocket)
262 actual_pocket = CLOUD_ARCHIVE_POCKETS[pocket]
263 with open('/etc/apt/sources.list.d/cloud-archive.list', 'w') as apt:
264 apt.write(CLOUD_ARCHIVE.format(actual_pocket))
265 elif source == 'proposed':
266 release = lsb_release()['DISTRIB_CODENAME']
267 with open('/etc/apt/sources.list.d/proposed.list', 'w') as apt:
268 apt.write(PROPOSED_POCKET.format(release))
269 elif source == 'distro':
270 pass
271 else:
272 log("Unknown source: {!r}".format(source))
273
274 if key:
275 if '-----BEGIN PGP PUBLIC KEY BLOCK-----' in key:
276 with NamedTemporaryFile('w+') as key_file:
277 key_file.write(key)
278 key_file.flush()
279 key_file.seek(0)
280 subprocess.check_call(['apt-key', 'add', '-'], stdin=key_file)
281 else:
282 # Note that hkp: is in no way a secure protocol. Using a
283 # GPG key id is pointless from a security POV unless you
284 # absolutely trust your network and DNS.
285 subprocess.check_call(['apt-key', 'adv', '--keyserver',
286 'hkp://keyserver.ubuntu.com:80', '--recv',
287 key])
288
289
290def _run_with_retries(cmd, max_retries=CMD_RETRY_COUNT, retry_exitcodes=(1,),
291 retry_message="", cmd_env=None):
292 """Run a command and retry until success or max_retries is reached.
293
294 :param: cmd: str: The apt command to run.
295 :param: max_retries: int: The number of retries to attempt on a fatal
296 command. Defaults to CMD_RETRY_COUNT.
297 :param: retry_exitcodes: tuple: Optional additional exit codes to retry.
298 Defaults to retry on exit code 1.
299 :param: retry_message: str: Optional log prefix emitted during retries.
300 :param: cmd_env: dict: Environment variables to add to the command run.
301 """
302
303 env = os.environ.copy()
304 if cmd_env:
305 env.update(cmd_env)
306
307 if not retry_message:
308 retry_message = "Failed executing '{}'".format(" ".join(cmd))
309 retry_message += ". Will retry in {} seconds".format(CMD_RETRY_DELAY)
310
311 retry_count = 0
312 result = None
313
314 retry_results = (None,) + retry_exitcodes
315 while result in retry_results:
316 try:
317 result = subprocess.check_call(cmd, env=env)
318 except subprocess.CalledProcessError as e:
319 retry_count = retry_count + 1
320 if retry_count > max_retries:
321 raise
322 result = e.returncode
323 log(retry_message)
324 time.sleep(CMD_RETRY_DELAY)
325
326
327def _run_apt_command(cmd, fatal=False):
328 """Run an apt command with optional retries.
329
330 :param: fatal: bool: Whether the command's output should be checked and
331 retried.
332 """
333 # Provide DEBIAN_FRONTEND=noninteractive if not present in the environment.
334 cmd_env = {
335 'DEBIAN_FRONTEND': os.environ.get('DEBIAN_FRONTEND', 'noninteractive')}
336
337 if fatal:
338 _run_with_retries(
339 cmd, cmd_env=cmd_env, retry_exitcodes=(1, APT_NO_LOCK,),
340 retry_message="Couldn't acquire DPKG lock")
341 else:
342 env = os.environ.copy()
343 env.update(cmd_env)
344 subprocess.call(cmd, env=env)
345
346
347def get_upstream_version(package):
348 """Determine upstream version based on installed package
349
350 @returns None (if not installed) or the upstream version
351 """
352 import apt_pkg
353 cache = apt_cache()
354 try:
355 pkg = cache[package]
356 except:
357 # the package is unknown to the current apt cache.
358 return None
359
360 if not pkg.current_ver:
361 # package is known, but no version is currently installed.
362 return None
363
364 return apt_pkg.upstream_version(pkg.current_ver.ver_str)
0365
=== added file 'charmhelpers/osplatform.py'
--- charmhelpers/osplatform.py 1970-01-01 00:00:00 +0000
+++ charmhelpers/osplatform.py 2017-03-04 02:50:20 +0000
@@ -0,0 +1,25 @@
1import platform
2
3
4def get_platform():
5 """Return the current OS platform.
6
7 For example: if current os platform is Ubuntu then a string "ubuntu"
8 will be returned (which is the name of the module).
9 This string is used to decide which platform module should be imported.
10 """
11 # linux_distribution is deprecated and will be removed in Python 3.7
12 # Warings *not* disabled, as we certainly need to fix this.
13 tuple_platform = platform.linux_distribution()
14 current_platform = tuple_platform[0]
15 if "Ubuntu" in current_platform:
16 return "ubuntu"
17 elif "CentOS" in current_platform:
18 return "centos"
19 elif "debian" in current_platform:
20 # Stock Python does not detect Ubuntu and instead returns debian.
21 # Or at least it does in some build environments like Travis CI
22 return "ubuntu"
23 else:
24 raise RuntimeError("This module is not supported on {}."
25 .format(current_platform))
026
=== modified file 'lib/apt.py'
--- lib/apt.py 2017-03-01 15:05:23 +0000
+++ lib/apt.py 2017-03-04 02:50:20 +0000
@@ -14,7 +14,8 @@
1414
15LANDSCAPE_PACKAGES = ("landscape-server", "landscape-hashids")15LANDSCAPE_PACKAGES = ("landscape-server", "landscape-hashids")
16INSTALL_PACKAGES = LANDSCAPE_PACKAGES + ("python-minimal", "python-psutil")16INSTALL_PACKAGES = LANDSCAPE_PACKAGES + ("python-minimal", "python-psutil")
17PACKAGES_DEV = ("dpkg-dev", "devscripts", "pbuilder", "aptitude", "build-essential")17PACKAGES_DEV = (
18 "dpkg-dev", "devscripts", "pbuilder", "aptitude", "build-essential")
18TARBALL = "landscape-server_*.tar.gz"19TARBALL = "landscape-server_*.tar.gz"
1920
20# XXX Eventually we'll want to use a dedicated PPA, populated by Jenkins.21# XXX Eventually we'll want to use a dedicated PPA, populated by Jenkins.

Subscribers

People subscribed via source and target branches