Merge lp:~danilo/charms/trusty/glance-simplestreams-sync/invert-failure-mode into lp:~landscape/charms/trusty/glance-simplestreams-sync/landscape

Proposed by Данило Шеган
Status: Merged
Merged at revision: 66
Proposed branch: lp:~danilo/charms/trusty/glance-simplestreams-sync/invert-failure-mode
Merge into: lp:~landscape/charms/trusty/glance-simplestreams-sync/landscape
Diff against target: 96 lines (+27/-46)
1 file modified
scripts/glance-simplestreams-sync.py (+27/-46)
To merge this branch: bzr merge lp:~danilo/charms/trusty/glance-simplestreams-sync/invert-failure-mode
Reviewer Review Type Date Requested Status
Geoff Teale (community) Approve
Bogdana Vereha (community) Approve
Review via email: mp+294635@code.launchpad.net

Description of the change

We've hit a number of cases where glance-simplestreams-sync script would fail due to swift, glance or something else not being fully ready, and had to extend it a number of times.

That has long pointed as at the fact that it'd be better to invert the logic there: only drop the per-minute cronjob on actual success.

This will mean that it will indefinitely retry even with fatal failures, but that's not a big deal for us, since our installation can take forever as well.

I'll do another test run before merging this in and after addressing all the review comments, fwiw.

To post a comment you must log in.
Revision history for this message
Bogdana Vereha (bogdana) wrote :

Looks good (considering the circumstances) :)

review: Approve
Revision history for this message
Geoff Teale (tealeg) wrote :

+1 - I guess it was inevitable ;-)

review: Approve
Revision history for this message
Данило Шеган (danilo) wrote :

Merged and pushed into cs:~landscape-charmers/trusty/glance-simplestreams-sync-8 and cs:~landscape-charmers/xenial/glance-simplestreams-sync-2

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'scripts/glance-simplestreams-sync.py'
2--- scripts/glance-simplestreams-sync.py 2016-05-10 19:43:36 +0000
3+++ scripts/glance-simplestreams-sync.py 2016-05-13 12:24:50 +0000
4@@ -414,28 +414,21 @@
5 charm_conf['use_swift'],
6 swift_exists))
7
8- should_delete_cron_poll = True
9-
10- if not swift_exists and charm_conf['use_swift']:
11- # If use_swift is set, we need to wait for swift to become available.
12- log.info("Swift not yet ready.")
13- should_delete_cron_poll = False
14-
15- if ps_service_exists and charm_conf['use_swift'] and swift_exists:
16- log.info("Updating product streams service.")
17- try:
18- update_product_streams_service(ksc, services,
19- charm_conf['region'])
20- except:
21- log.exception("Exception during update_product_streams_service")
22- should_delete_cron_poll = False
23-
24- else:
25- log.info("Not updating product streams service.")
26-
27- status_exchange = StatusExchange()
28-
29 try:
30+ if not swift_exists and charm_conf['use_swift']:
31+ # If use_swift is set, we need to wait for swift to become
32+ # available.
33+ log.info("Swift not yet ready.")
34+ return
35+
36+ if ps_service_exists and charm_conf['use_swift'] and swift_exists:
37+ log.info("Updating product streams service.")
38+ update_product_streams_service(ksc, services, charm_conf['region'])
39+ else:
40+ log.info("Not updating product streams service.")
41+
42+ status_exchange = StatusExchange()
43+
44 log.info("Beginning image sync")
45 status_set('maintenance', 'Synchronising images')
46
47@@ -447,38 +440,26 @@
48 status_exchange.send_message({"status": "Done",
49 "message": completed_msg})
50 status_set('active', completed_msg)
51+
52+ status_exchange.close()
53+
54+ # If this is an initial per-minute sync attempt, delete it on success.
55+ if os.path.exists(CRON_POLL_FILENAME):
56+ os.unlink(CRON_POLL_FILENAME)
57+ log.info("Initial sync attempt done. every-minute cronjob removed.")
58+
59 except keystone_exceptions.EndpointNotFound as e:
60 # matching string "{PublicURL} endpoint for {type}{region} not
61 # found". where {type} is 'image' and {region} is potentially
62 # not empty so we only match on this substring:
63 if 'endpoint for image' in e.message:
64- should_delete_cron_poll = False
65 log.info("Glance endpoint not found, will continue polling.")
66-
67- except (glanceclient.exc.ClientException, glanceclient.exc.CommunicationError) as e:
68- log.exception("Glance Client exception during do_sync:\n%s\n"
69- "Will continue polling." % e.message)
70- should_delete_cron_poll = False
71- except swiftclient.ClientException as e:
72- log.exception("Swift Client exception during do_sync:\n%s\n"
73- "Will continue polling." % e.message)
74- should_delete_cron_poll = False
75- except requests.exceptions.RequestException as e:
76- log.exception("Requests exception during do_sync:\n%s\n"
77- "Will continue polling." % e.message)
78- should_delete_cron_poll = False
79-
80 except Exception as e:
81- log.exception("Exception during do_sync")
82- status_exchange.send_message({"status": "Error",
83- "message": traceback.format_exc()})
84- status_set('blocked', 'Image sync failed')
85-
86- status_exchange.close()
87-
88- if os.path.exists(CRON_POLL_FILENAME) and should_delete_cron_poll:
89- os.unlink(CRON_POLL_FILENAME)
90- log.info("Initial sync attempt done. every-minute cronjob removed.")
91+ log.exception("Exception during syncing:\n%s")
92+ status_exchange.send_message(
93+ {"status": "Error", "message": traceback.format_exc()})
94+ status_set('blocked', 'Image sync failed, retrying soon.')
95+
96 log.info("sync done.")
97
98

Subscribers

People subscribed via source and target branches