Ubuntu Mirror Charm

Merge ~pjdc/ubuntu-mirror-charm/+git/ubuntu-mirror-charm:mirror-trigger-scripts into ubuntu-mirror-charm:master

  1. Git
  2. lp:~pjdc/ubuntu-mirror-charm/+git/ubuntu-mirror-charm
  3. mirror-trigger-scripts
  4. Merge into master
Proposed by Paul Collins
Status: Merged
Approved by: Paul Collins
Approved revision: de9a16e393aa185bd3b7d0d54a53ba11ea5a6254
Merged at revision: d59c2e2dd778973359b7d9114a79cd48a392444d
Proposed branch: ~pjdc/ubuntu-mirror-charm/+git/ubuntu-mirror-charm:mirror-trigger-scripts
Merge into: ubuntu-mirror-charm:master
Diff against target: 368 lines (+188/-13)
9 files modified
config.yaml (+117/-0)
files/check-mirror.sh (+2/-0)
files/check-updates.sh (+2/-0)
files/mirror-1stage.sh (+1/-1)
files/mirror-2stage.sh (+8/-1)
files/trigger-downstream-mirrors.sh (+23/-0)
hooks/Config.py (+20/-4)
hooks/hooks.py (+12/-7)
templates/mirror-conf.tmpl (+3/-0)
Reviewer Review Type Date Requested Status
Haw Loeung +1 Approve
David Lawson (community) Approve
Review via email: mp+391668@code.launchpad.net

Commit message

add downstream_mirrors and trigger_keyfile per role, and wire them up

To post a comment you must log in.
Revision history for this message
David Lawson (deej) wrote :

+1

review: Approve
Revision history for this message
🤖 Canonical IS Merge Bot (canonical-is-mergebot) wrote :

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

Revision history for this message
🤖 Canonical IS Merge Bot (canonical-is-mergebot) wrote :

Unable to determine commit message from repository - please click "Set commit message" and enter the commit message manually.

Revision history for this message
Haw Loeung (hloeung) wrote :

LGTM, +1

review: Approve (+1)
Revision history for this message
🤖 Canonical IS Merge Bot (canonical-is-mergebot) wrote :

Change successfully merged at revision d59c2e2dd778973359b7d9114a79cd48a392444d

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/config.yaml b/config.yaml
2index b9c9f43..dccc75f 100644
3--- a/config.yaml
4+++ b/config.yaml
5@@ -95,6 +95,19 @@ options:
6 default: ""
7 type: string
8 description: "A base64 string containing apache configuration options to be included late in the config file"
9+ mirror_ubuntu_downstream_mirrors:
10+ default: ""
11+ type: "string"
12+ description: >
13+ A space-separated list of mirrors to trigger following a sync.
14+ Username defaults to the mirror_user config setting, and
15+ otherwise may be specified by "user@host" syntax.
16+ mirror_ubuntu_trigger_keyfile:
17+ default: ""
18+ type: "string"
19+ description: >
20+ Local path of the key file to use when triggering downstream
21+ mirrors. If empty, downstream mirrors are not triggered.
22 mirror_ubuntu-cloud-archive_name:
23 default: "ubuntu-cloud.archive.canonical.com"
24 type: string
25@@ -147,6 +160,19 @@ options:
26 default: ""
27 type: string
28 description: "A base64 string containing apache configuration options to be included late in the config file"
29+ mirror_ubuntu-cloud-archive_downstream_mirrors:
30+ default: ""
31+ type: "string"
32+ description: >
33+ A space-separated list of mirrors to trigger following a sync.
34+ Username defaults to the mirror_user config setting, and
35+ otherwise may be specified by "user@host" syntax.
36+ mirror_ubuntu-cloud-archive_trigger_keyfile:
37+ default: ""
38+ type: "string"
39+ description: >
40+ Local path of the key file to use when triggering downstream
41+ mirrors. If empty, downstream mirrors are not triggered.
42 mirror_releases_name:
43 default: "releases.ubuntu.com"
44 type: string
45@@ -199,6 +225,19 @@ options:
46 default: ""
47 type: string
48 description: "A base64 string containing apache configuration options to be included late in the config file"
49+ mirror_releases_downstream_mirrors:
50+ default: ""
51+ type: "string"
52+ description: >
53+ A space-separated list of mirrors to trigger following a sync.
54+ Username defaults to the mirror_user config setting, and
55+ otherwise may be specified by "user@host" syntax.
56+ mirror_releases_trigger_keyfile:
57+ default: ""
58+ type: "string"
59+ description: >
60+ Local path of the key file to use when triggering downstream
61+ mirrors. If empty, downstream mirrors are not triggered.
62 mirror_cloud-images_name:
63 default: "cloud-images.ubuntu.com"
64 type: string
65@@ -251,6 +290,19 @@ options:
66 default: ""
67 type: string
68 description: "A base64 string containing apache configuration options to be included late in the config file"
69+ mirror_cloud-images_downstream_mirrors:
70+ default: ""
71+ type: "string"
72+ description: >
73+ A space-separated list of mirrors to trigger following a sync.
74+ Username defaults to the mirror_user config setting, and
75+ otherwise may be specified by "user@host" syntax.
76+ mirror_cloud-images_trigger_keyfile:
77+ default: ""
78+ type: "string"
79+ description: >
80+ Local path of the key file to use when triggering downstream
81+ mirrors. If empty, downstream mirrors are not triggered.
82 mirror_simple-streams_name:
83 default: "streams.canonical.com"
84 type: string
85@@ -303,6 +355,19 @@ options:
86 default: ""
87 type: string
88 description: "A base64 string containing apache configuration options to be included late in the config file"
89+ mirror_simple-streams_downstream_mirrors:
90+ default: ""
91+ type: "string"
92+ description: >
93+ A space-separated list of mirrors to trigger following a sync.
94+ Username defaults to the mirror_user config setting, and
95+ otherwise may be specified by "user@host" syntax.
96+ mirror_simple-streams_trigger_keyfile:
97+ default: ""
98+ type: "string"
99+ description: >
100+ Local path of the key file to use when triggering downstream
101+ mirrors. If empty, downstream mirrors are not triggered.
102 mirror_cdimage_name:
103 default: "cdimage.ubuntu.com"
104 type: string
105@@ -355,6 +420,19 @@ options:
106 default: ""
107 type: string
108 description: "A base64 string containing apache configuration options to be included late in the config file"
109+ mirror_cdimage_downstream_mirrors:
110+ default: ""
111+ type: "string"
112+ description: >
113+ A space-separated list of mirrors to trigger following a sync.
114+ Username defaults to the mirror_user config setting, and
115+ otherwise may be specified by "user@host" syntax.
116+ mirror_cdimage_trigger_keyfile:
117+ default: ""
118+ type: "string"
119+ description: >
120+ Local path of the key file to use when triggering downstream
121+ mirrors. If empty, downstream mirrors are not triggered.
122 mirror_ports_name:
123 default: "ports.ubuntu.com"
124 type: string
125@@ -407,6 +485,19 @@ options:
126 default: ""
127 type: string
128 description: "A base64 string containing apache configuration options to be included late in the config file"
129+ mirror_ports_downstream_mirrors:
130+ default: ""
131+ type: "string"
132+ description: >
133+ A space-separated list of mirrors to trigger following a sync.
134+ Username defaults to the mirror_user config setting, and
135+ otherwise may be specified by "user@host" syntax.
136+ mirror_ports_trigger_keyfile:
137+ default: ""
138+ type: "string"
139+ description: >
140+ Local path of the key file to use when triggering downstream
141+ mirrors. If empty, downstream mirrors are not triggered.
142 mirror_maas-images_name:
143 default: "images.maas.io"
144 type: string
145@@ -459,6 +550,19 @@ options:
146 default: ""
147 type: string
148 description: "A base64 string containing apache configuration options to be included late in the config file"
149+ mirror_maas-images_downstream_mirrors:
150+ default: ""
151+ type: "string"
152+ description: >
153+ A space-separated list of mirrors to trigger following a sync.
154+ Username defaults to the mirror_user config setting, and
155+ otherwise may be specified by "user@host" syntax.
156+ mirror_maas-images_trigger_keyfile:
157+ default: ""
158+ type: "string"
159+ description: >
160+ Local path of the key file to use when triggering downstream
161+ mirrors. If empty, downstream mirrors are not triggered.
162 mirror_old-releases_name:
163 default: "old-releases.ubuntu.com"
164 type: string
165@@ -511,6 +615,19 @@ options:
166 default: ""
167 type: string
168 description: "A base64 string containing apache configuration options to be included late in the config file"
169+ mirror_old-releases_downstream_mirrors:
170+ default: ""
171+ type: "string"
172+ description: >
173+ A space-separated list of mirrors to trigger following a sync.
174+ Username defaults to the mirror_user config setting, and
175+ otherwise may be specified by "user@host" syntax.
176+ mirror_old-releases_trigger_keyfile:
177+ default: ""
178+ type: "string"
179+ description: >
180+ Local path of the key file to use when triggering downstream
181+ mirrors. If empty, downstream mirrors are not triggered.
182 ftp_max_instances:
183 default: 200
184 type: int
185diff --git a/files/check-mirror.sh b/files/check-mirror.sh
186index 979e851..dde03f6 100755
187--- a/files/check-mirror.sh
188+++ b/files/check-mirror.sh
189@@ -4,6 +4,8 @@
190 # This file is Juju managed - do not make local changes #
191 #-------------------------------------------------------#
192 #
193+# Copyright 2014 Canonical Ltd.
194+#
195 # Check the validity of a mirror's files
196
197 set -u
198diff --git a/files/check-updates.sh b/files/check-updates.sh
199index faf2cdc..92b963b 100755
200--- a/files/check-updates.sh
201+++ b/files/check-updates.sh
202@@ -4,6 +4,8 @@
203 # This file is Juju managed - do not make local changes #
204 #-------------------------------------------------------#
205 #
206+# Copyright 2014 Canonical Ltd.
207+#
208 # Check when the last mirror update was and warn if too old
209
210 set -u
211diff --git a/files/mirror-1stage.sh b/files/mirror-1stage.sh
212index 201d4c4..6348aa1 100755
213--- a/files/mirror-1stage.sh
214+++ b/files/mirror-1stage.sh
215@@ -5,7 +5,7 @@
216 #-------------------------------------------------------#
217 #
218 # Author: Chris Stratford <chris.stratford@canonical.com>
219-# Copyright: Canonical Ltd. 2014
220+# Copyright 2014 Canonical Ltd.
221 #
222 # Single-pass rsync mirror
223 #
224diff --git a/files/mirror-2stage.sh b/files/mirror-2stage.sh
225index af0c628..3a4ebc9 100755
226--- a/files/mirror-2stage.sh
227+++ b/files/mirror-2stage.sh
228@@ -5,7 +5,7 @@
229 #-------------------------------------------------------#
230 #
231 # Author: Chris Stratford <chris.stratford@canonical.com>
232-# Copyright: Canonical Ltd. 2014
233+# Copyright 2014 Canonical Ltd.
234 #
235 # Two-pass rsync mirror (for archives)
236 #
237@@ -91,5 +91,12 @@ find ${DEST_DIR} -type f ! -perm /o=r -ls -exec chmod go+r '{}' +
238
239 install -d "${DEST_DIR}/.trace/"
240 date -u > "${DEST_DIR}/.trace/${role}-${myhostname}"
241+
242+if [ -n "${DOWNSTREAM_MIRRORS}" -a -n "${TRIGGER_KEYFILE}" ]; then
243+ t=15m
244+ log "== Triggering downstream mirrors (will give up after $t) =="
245+ timeout $t ${SCRIPT_DIR}/trigger-downstream-mirrors.sh $role $TRIGGER_KEYFILE $DOWNSTREAM_MIRRORS
246+fi
247+
248 savelog ${logfile} > /dev/null 2>&1
249 rm -f ${lockpath} > /dev/null 2>&1
250diff --git a/files/trigger-downstream-mirrors.sh b/files/trigger-downstream-mirrors.sh
251new file mode 100755
252index 0000000..69b0973
253--- /dev/null
254+++ b/files/trigger-downstream-mirrors.sh
255@@ -0,0 +1,23 @@
256+#!/bin/sh
257+#
258+#-------------------------------------------------------#
259+# This file is Juju managed - do not make local changes #
260+#-------------------------------------------------------#
261+#
262+# Copyright 2020 Canonical Ltd.
263+#
264+
265+set -eu
266+
267+role=$1 ; shift
268+keyfile=$1 ; shift
269+downstream="$@"
270+
271+for host in $downstream; do
272+ echo Triggering $host ...
273+ timeout 1m ssh -i "${keyfile}" $host sync-${role} &
274+done
275+
276+echo Waiting for triggers to complete ...
277+wait
278+echo All triggers exited. All done.
279diff --git a/hooks/Config.py b/hooks/Config.py
280index 6e3c072..95c1ff2 100755
281--- a/hooks/Config.py
282+++ b/hooks/Config.py
283@@ -10,6 +10,25 @@ import os.path
284 from charmhelpers.core.hookenv import config, log, in_relation_hook, relation_get, relation_type
285
286
287+ROLE_CONFIG_KEYS = [
288+ "name",
289+ "aliases",
290+ "path",
291+ "description",
292+ "rsync_log",
293+ "command",
294+ "source_url",
295+ "rsync_auth",
296+ "rsync_module",
297+ "sync_time",
298+ "trigger",
299+ "apache_late_extra",
300+ "apache_early_extra",
301+ "downstream_mirrors",
302+ "trigger_keyfile",
303+]
304+
305+
306 class Config:
307 _mirror_map = {}
308 _relation_data = {}
309@@ -124,10 +143,7 @@ class Config:
310 'unsafe': role.endswith('-unsafe'),
311 'tmpl_file': str("apache-{}.tmpl".format(base_role)),
312 }
313- for key in ["name", "aliases", "path", "description", "rsync_log",
314- "command", "source_url", "rsync_auth", "rsync_module",
315- "sync_time", "trigger", "apache_late_extra",
316- "apache_early_extra"]:
317+ for key in ROLE_CONFIG_KEYS:
318 config_key = "mirror_%s_%s" % (base_role, key)
319 try:
320 value = config(config_key)
321diff --git a/hooks/hooks.py b/hooks/hooks.py
322index 1f56257..824fc28 100755
323--- a/hooks/hooks.py
324+++ b/hooks/hooks.py
325@@ -350,19 +350,24 @@ def configure_rsync_client(conf, hostname): # noqa: C901
326 except ValueError:
327 log("CHARM: Missing details - skipping {}".format(role))
328 else:
329- tmpl_data = {}
330- sync_time = mirror["sync_time"]
331- sync_cmd = mirror["command"]
332 auth_credentials = conf.auth_credentials(role, hostname)
333- tmpl_data["source_url"] = mirror["source_url"]
334- tmpl_data["dest_dir"] = mirror["path"]
335- tmpl_data["user"] = auth_credentials["user"]
336- tmpl_data["password"] = auth_credentials["password"]
337+ tmpl_data = {
338+ "source_url": mirror["source_url"],
339+ "dest_dir": mirror["path"],
340+ "user": auth_credentials["user"],
341+ "password": auth_credentials["password"],
342+ "script_dir": conf.script_dir(),
343+ "downstream_mirrors": mirror["downstream_mirrors"],
344+ "trigger_keyfile": mirror["trigger_keyfile"],
345+ }
346 config_path = os.path.join(mirror_confdir, config_file)
347 file_from_template("mirror-conf.tmpl", config_path, tmpl_data)
348 os.chmod(config_path, 0o440)
349 os.chown(config_path, mirror_userinfo.pw_uid, mirror_userinfo.pw_gid)
350+
351+ sync_time = mirror["sync_time"]
352 if sync_time != 'ondemand':
353+ sync_cmd = mirror["command"]
354 line = "%s %s %s/%s %s\n" % (sync_time, conf.mirror_user(), conf.script_dir(), sync_cmd, role)
355 cron += line
356
357diff --git a/templates/mirror-conf.tmpl b/templates/mirror-conf.tmpl
358index 3ca03d8..5d94d70 100644
359--- a/templates/mirror-conf.tmpl
360+++ b/templates/mirror-conf.tmpl
361@@ -1,4 +1,7 @@
362+SCRIPT_DIR=${script_dir}
363 DEST_DIR=${dest_dir}
364 SOURCE_URL=${source_url}
365 RSYNC_USER=${user}
366 RSYNC_PASSWORD=${password}
367+DOWNSTREAM_MIRRORS=${downstream_mirrors}
368+TRIGGER_KEYFILE=${trigger_keyfile}

Subscribers

People subscribed via source and target branches