Merge lp:~sil2100/ubuntu-system-image/server_script_testability into lp:ubuntu-system-image/server
- server_script_testability
- Merge into server
Status: | Needs review |
---|---|
Proposed branch: | lp:~sil2100/ubuntu-system-image/server_script_testability |
Merge into: | lp:ubuntu-system-image/server |
Diff against target: |
2040 lines (+1340/-531) 12 files modified
bin/copy-image (+11/-277) bin/set-phased-percentage (+8/-72) bin/tag-image (+4/-178) lib/systemimage/script.py (+269/-0) lib/systemimage/scripts/script_copy_image.py (+187/-0) lib/systemimage/scripts/script_set_phased_percentage.py (+83/-0) lib/systemimage/scripts/script_tag_image.py (+127/-0) lib/systemimage/tests/test_scripts.py (+590/-0) lib/systemimage/tests/test_tools.py (+41/-0) lib/systemimage/tests/test_tree.py (+1/-1) lib/systemimage/tools.py (+15/-0) lib/systemimage/tree.py (+4/-3) |
To merge this branch: | bzr merge lp:~sil2100/ubuntu-system-image/server_script_testability |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Barry Warsaw (community) | Approve | ||
Review via email: mp+285041@code.launchpad.net |
Commit message
Make scripts implemented as part of a Script class, making them more easily testable. Add some basic tests for the first two migrated - copy-image and tag-image.
Description of the change
The general script-testability branch. It's something I'm working on part-time between all other things - so far only two scripts are being migrated. Submitting for early review.
Barry Warsaw (barry) wrote : | # |
In general I really like the direction you're going in with this. More tests == better!
- 287. By Łukasz Zemczak
-
Add test for the script execute for tag-image. Fix a bug in setting the phased percentage after tagging (detected thanks to the unit test).
- 288. By Łukasz Zemczak
-
Add more tests for scripts.
- 289. By Łukasz Zemczak
-
Move the script classes to a separate script directory. Update copyright.
- 290. By Łukasz Zemczak
-
Fix script tests. Port the phased percentage script to the new framework, add tests.
- 291. By Łukasz Zemczak
-
Actually use the new script in the bin/ script of phased percentage.
- 292. By Łukasz Zemczak
-
Merge trunk.
- 293. By Łukasz Zemczak
-
Backport the new per-device redirect checks to scripts, add tests for those cases. Fix a few existing test failures.
- 294. By Łukasz Zemczak
-
pep8 fixes.
Unmerged revisions
- 294. By Łukasz Zemczak
-
pep8 fixes.
- 293. By Łukasz Zemczak
-
Backport the new per-device redirect checks to scripts, add tests for those cases. Fix a few existing test failures.
- 292. By Łukasz Zemczak
-
Merge trunk.
- 291. By Łukasz Zemczak
-
Actually use the new script in the bin/ script of phased percentage.
- 290. By Łukasz Zemczak
-
Fix script tests. Port the phased percentage script to the new framework, add tests.
- 289. By Łukasz Zemczak
-
Move the script classes to a separate script directory. Update copyright.
- 288. By Łukasz Zemczak
-
Add more tests for scripts.
- 287. By Łukasz Zemczak
-
Add test for the script execute for tag-image. Fix a bug in setting the phased percentage after tagging (detected thanks to the unit test).
- 286. By Łukasz Zemczak
-
PEP8 fixes to scripts.py.
- 285. By Łukasz Zemczak
-
Add arg parsing tests for the tag script as well. Do some PEP8 syntax fixes.
Preview Diff
1 | === modified file 'bin/copy-image' (properties changed: +x to -x) | |||
2 | --- bin/copy-image 2016-06-24 21:02:18 +0000 | |||
3 | +++ bin/copy-image 2016-07-20 12:34:06 +0000 | |||
4 | @@ -1,8 +1,8 @@ | |||
5 | 1 | #!/usr/bin/python | 1 | #!/usr/bin/python |
6 | 2 | # -*- coding: utf-8 -*- | 2 | # -*- coding: utf-8 -*- |
7 | 3 | 3 | ||
10 | 4 | # Copyright (C) 2013 Canonical Ltd. | 4 | # Copyright (C) 2015-2016 Canonical Ltd. |
11 | 5 | # Author: Stéphane Graber <stgraber@ubuntu.com> | 5 | # Author: Łukasz 'sil2100' Zemczak <lukasz.zemczak@ubuntu.com> |
12 | 6 | 6 | ||
13 | 7 | # This program is free software: you can redistribute it and/or modify | 7 | # This program is free software: you can redistribute it and/or modify |
14 | 8 | # it under the terms of the GNU General Public License as published by | 8 | # it under the terms of the GNU General Public License as published by |
15 | @@ -16,284 +16,18 @@ | |||
16 | 16 | # You should have received a copy of the GNU General Public License | 16 | # You should have received a copy of the GNU General Public License |
17 | 17 | # along with this program. If not, see <http://www.gnu.org/licenses/>. | 17 | # along with this program. If not, see <http://www.gnu.org/licenses/>. |
18 | 18 | 18 | ||
19 | 19 | import json | ||
20 | 20 | import os | 19 | import os |
21 | 21 | import sys | 20 | import sys |
22 | 21 | |||
23 | 22 | sys.path.insert(0, os.path.join(sys.path[0], os.pardir, "lib")) | 22 | sys.path.insert(0, os.path.join(sys.path[0], os.pardir, "lib")) |
24 | 23 | 23 | ||
30 | 24 | from systemimage import config, generators, tools, tree | 24 | from systemimage.scripts import script_copy_image |
31 | 25 | 25 | ||
32 | 26 | import argparse | 26 | |
33 | 27 | import fcntl | 27 | def main(): |
34 | 28 | import logging | 28 | script = script_copy_image.CopyImageScript() |
35 | 29 | script.run(sys.argv) | ||
36 | 30 | |||
37 | 29 | 31 | ||
38 | 30 | if __name__ == '__main__': | 32 | if __name__ == '__main__': |
308 | 31 | parser = argparse.ArgumentParser(description="image copier") | 33 | main() |
40 | 32 | parser.add_argument("source_channel", metavar="SOURCE-CHANNEL") | ||
41 | 33 | parser.add_argument("destination_channel", metavar="DESTINATION-CHANNEL") | ||
42 | 34 | parser.add_argument("device", metavar="DEVICE") | ||
43 | 35 | parser.add_argument("version", metavar="VERSION", type=int) | ||
44 | 36 | parser.add_argument("-k", "--keep-version", action="store_true", | ||
45 | 37 | help="Keep the original version number") | ||
46 | 38 | parser.add_argument("-p", "--phased-percentage", type=int, | ||
47 | 39 | help="Set the phased percentage for the copied image", | ||
48 | 40 | default=100) | ||
49 | 41 | parser.add_argument("-t", "--tag", type=str, | ||
50 | 42 | help="Set a version tag on the new image") | ||
51 | 43 | parser.add_argument("--verbose", "-v", action="count", default=0) | ||
52 | 44 | |||
53 | 45 | args = parser.parse_args() | ||
54 | 46 | |||
55 | 47 | # Setup logging | ||
56 | 48 | formatter = logging.Formatter( | ||
57 | 49 | "%(asctime)s %(levelname)s %(message)s") | ||
58 | 50 | |||
59 | 51 | levels = {1: logging.ERROR, | ||
60 | 52 | 2: logging.WARNING, | ||
61 | 53 | 3: logging.INFO, | ||
62 | 54 | 4: logging.DEBUG} | ||
63 | 55 | |||
64 | 56 | if args.verbose > 0: | ||
65 | 57 | stdoutlogger = logging.StreamHandler(sys.stdout) | ||
66 | 58 | stdoutlogger.setFormatter(formatter) | ||
67 | 59 | logging.root.setLevel(levels[min(4, args.verbose)]) | ||
68 | 60 | logging.root.addHandler(stdoutlogger) | ||
69 | 61 | else: | ||
70 | 62 | logging.root.addHandler(logging.NullHandler()) | ||
71 | 63 | |||
72 | 64 | # Load the configuration | ||
73 | 65 | conf = config.Config() | ||
74 | 66 | |||
75 | 67 | # Try to acquire a global lock | ||
76 | 68 | lock_file = os.path.join(conf.state_path, "global.lock") | ||
77 | 69 | lock_fd = open(lock_file, 'w') | ||
78 | 70 | |||
79 | 71 | try: | ||
80 | 72 | fcntl.lockf(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) | ||
81 | 73 | except IOError: | ||
82 | 74 | print("Something else holds the global lock. exiting.") | ||
83 | 75 | sys.exit(0) | ||
84 | 76 | |||
85 | 77 | # Load the tree | ||
86 | 78 | pub = tree.Tree(conf) | ||
87 | 79 | |||
88 | 80 | # Do some checks | ||
89 | 81 | channels = pub.list_channels() | ||
90 | 82 | if args.source_channel not in channels: | ||
91 | 83 | parser.error("Invalid source channel: %s" % args.source_channel) | ||
92 | 84 | |||
93 | 85 | if args.destination_channel not in channels: | ||
94 | 86 | parser.error("Invalid destination channel: %s" % | ||
95 | 87 | args.destination_channel) | ||
96 | 88 | |||
97 | 89 | if args.device not in channels[args.source_channel]['devices']: | ||
98 | 90 | parser.error("Invalid device for source channel: %s" % | ||
99 | 91 | args.device) | ||
100 | 92 | |||
101 | 93 | if args.device not in \ | ||
102 | 94 | channels[args.destination_channel]['devices']: | ||
103 | 95 | parser.error("Invalid device for destination channel: %s" % | ||
104 | 96 | args.device) | ||
105 | 97 | |||
106 | 98 | if "alias" in channels[args.source_channel] and \ | ||
107 | 99 | channels[args.source_channel]['alias'] \ | ||
108 | 100 | != args.source_channel: | ||
109 | 101 | parser.error("Source channel is an alias.") | ||
110 | 102 | |||
111 | 103 | if "alias" in channels[args.destination_channel] and \ | ||
112 | 104 | channels[args.destination_channel]['alias'] \ | ||
113 | 105 | != args.destination_channel: | ||
114 | 106 | parser.error("Destination channel is an alias.") | ||
115 | 107 | |||
116 | 108 | if "redirect" in channels[args.source_channel]: | ||
117 | 109 | parser.error("Source channel is a redirect.") | ||
118 | 110 | |||
119 | 111 | if "redirect" in channels[args.destination_channel]: | ||
120 | 112 | parser.error("Destination channel is a redirect.") | ||
121 | 113 | |||
122 | 114 | src_device = channels[args.source_channel]['devices'][args.device] | ||
123 | 115 | if "redirect" in src_device: | ||
124 | 116 | parser.error("Source device is a redirect. Use original channel " | ||
125 | 117 | "%s instead." % src_device['redirect']) | ||
126 | 118 | |||
127 | 119 | dest_device = channels[args.destination_channel]['devices'][args.device] | ||
128 | 120 | if "redirect" in dest_device: | ||
129 | 121 | parser.error("Destination device is a redirect. Use original channel " | ||
130 | 122 | "%s instead." % dest_device['redirect']) | ||
131 | 123 | |||
132 | 124 | if args.tag and "," in args.tag: | ||
133 | 125 | parser.error("Image tag cannot include the character ','.") | ||
134 | 126 | |||
135 | 127 | source_device = pub.get_device(args.source_channel, args.device) | ||
136 | 128 | destination_device = pub.get_device(args.destination_channel, args.device) | ||
137 | 129 | |||
138 | 130 | if args.keep_version: | ||
139 | 131 | images = [image for image in destination_device.list_images() | ||
140 | 132 | if image['version'] == args.version] | ||
141 | 133 | if images: | ||
142 | 134 | parser.error("Version number is already used: %s" % args.version) | ||
143 | 135 | |||
144 | 136 | # Assign a new version number | ||
145 | 137 | new_version = args.version | ||
146 | 138 | if not args.keep_version: | ||
147 | 139 | # Find the next available version | ||
148 | 140 | new_version = 1 | ||
149 | 141 | for image in destination_device.list_images(): | ||
150 | 142 | if image['version'] >= new_version: | ||
151 | 143 | new_version = image['version'] + 1 | ||
152 | 144 | logging.debug("Version for next image: %s" % new_version) | ||
153 | 145 | |||
154 | 146 | # Extract the build we want to copy | ||
155 | 147 | images = [image for image in source_device.list_images() | ||
156 | 148 | if image['type'] == "full" and image['version'] == args.version] | ||
157 | 149 | if not images: | ||
158 | 150 | parser.error("Can't find version: %s" % args.version) | ||
159 | 151 | source_image = images[0] | ||
160 | 152 | |||
161 | 153 | # Extract the list of existing full images | ||
162 | 154 | full_images = {image['version']: image | ||
163 | 155 | for image in destination_device.list_images() | ||
164 | 156 | if image['type'] == "full"} | ||
165 | 157 | |||
166 | 158 | # Check that the last full and the new image aren't one and the same | ||
167 | 159 | source_files = [entry['path'].split("/")[-1] | ||
168 | 160 | for entry in source_image['files'] | ||
169 | 161 | if not entry['path'].split("/")[-1].startswith("version-")] | ||
170 | 162 | destination_files = [] | ||
171 | 163 | if full_images: | ||
172 | 164 | latest_full = sorted(full_images.values(), | ||
173 | 165 | key=lambda image: image['version'])[-1] | ||
174 | 166 | destination_files = [entry['path'].split("/")[-1] | ||
175 | 167 | for entry in latest_full['files'] | ||
176 | 168 | if not entry['path'].split( | ||
177 | 169 | "/")[-1].startswith("version-")] | ||
178 | 170 | if source_files == destination_files: | ||
179 | 171 | parser.error("Source image is already latest full in " | ||
180 | 172 | "destination channel.") | ||
181 | 173 | |||
182 | 174 | # Generate a list of required deltas | ||
183 | 175 | delta_base = tools.get_required_deltas( | ||
184 | 176 | conf, pub, args.destination_channel, args.device) | ||
185 | 177 | |||
186 | 178 | # Create new empty entries | ||
187 | 179 | new_images = {'full': {'files': []}} | ||
188 | 180 | for delta in delta_base: | ||
189 | 181 | new_images['delta_%s' % delta['version']] = {'files': []} | ||
190 | 182 | |||
191 | 183 | # Extract current version_detail and files | ||
192 | 184 | version_detail = tools.extract_files_and_version( | ||
193 | 185 | conf, source_image['files'], args.version, new_images['full']['files']) | ||
194 | 186 | |||
195 | 187 | # Generate new version tarball | ||
196 | 188 | environment = {} | ||
197 | 189 | environment['channel_name'] = args.destination_channel | ||
198 | 190 | environment['device'] = destination_device | ||
199 | 191 | environment['device_name'] = args.device | ||
200 | 192 | environment['version'] = new_version | ||
201 | 193 | environment['version_detail'] = [entry | ||
202 | 194 | for entry in version_detail.split(",") | ||
203 | 195 | if not entry.startswith("version=")] | ||
204 | 196 | environment['new_files'] = new_images['full']['files'] | ||
205 | 197 | |||
206 | 198 | # Add new tag if requested | ||
207 | 199 | if args.tag: | ||
208 | 200 | tools.set_tag_on_version_detail( | ||
209 | 201 | environment['version_detail'], args.tag) | ||
210 | 202 | logging.info("Setting tag for image to '%s'" % args.tag) | ||
211 | 203 | |||
212 | 204 | logging.info("Generating new version tarball for '%s' (%s)" | ||
213 | 205 | % (new_version, ",".join(environment['version_detail']))) | ||
214 | 206 | version_path = generators.generate_file(conf, "version", [], environment) | ||
215 | 207 | if version_path: | ||
216 | 208 | new_images['full']['files'].append(version_path) | ||
217 | 209 | |||
218 | 210 | # Generate deltas | ||
219 | 211 | for abspath in new_images['full']['files']: | ||
220 | 212 | prefix = abspath.split("/")[-1].rsplit("-", 1)[0] | ||
221 | 213 | for delta in delta_base: | ||
222 | 214 | # Extract the source | ||
223 | 215 | src_path = None | ||
224 | 216 | for file_dict in delta['files']: | ||
225 | 217 | if (file_dict['path'].split("/")[-1] | ||
226 | 218 | .startswith(prefix)): | ||
227 | 219 | src_path = "%s/%s" % (conf.publish_path, | ||
228 | 220 | file_dict['path']) | ||
229 | 221 | break | ||
230 | 222 | |||
231 | 223 | # Check that it's not the current file | ||
232 | 224 | if src_path: | ||
233 | 225 | src_path = os.path.realpath(src_path) | ||
234 | 226 | |||
235 | 227 | # FIXME: the keyring- is a big hack... | ||
236 | 228 | if src_path == abspath and "keyring-" not in src_path: | ||
237 | 229 | continue | ||
238 | 230 | |||
239 | 231 | # Generators are allowed to return None when no delta | ||
240 | 232 | # exists at all. | ||
241 | 233 | logging.info("Generating delta from '%s' for '%s'" % | ||
242 | 234 | (delta['version'], | ||
243 | 235 | prefix)) | ||
244 | 236 | delta_path = generators.generate_delta(conf, src_path, | ||
245 | 237 | abspath) | ||
246 | 238 | else: | ||
247 | 239 | delta_path = abspath | ||
248 | 240 | |||
249 | 241 | if not delta_path: | ||
250 | 242 | continue | ||
251 | 243 | |||
252 | 244 | # Get the full and relative paths | ||
253 | 245 | delta_abspath, delta_relpath = tools.expand_path( | ||
254 | 246 | delta_path, conf.publish_path) | ||
255 | 247 | |||
256 | 248 | new_images['delta_%s' % delta['version']]['files'] \ | ||
257 | 249 | .append(delta_abspath) | ||
258 | 250 | |||
259 | 251 | # Add full image | ||
260 | 252 | logging.info("Publishing new image '%s' (%s) with %s files." | ||
261 | 253 | % (new_version, ",".join(environment['version_detail']), | ||
262 | 254 | len(new_images['full']['files']))) | ||
263 | 255 | destination_device.create_image( | ||
264 | 256 | "full", new_version, ",".join(environment['version_detail']), | ||
265 | 257 | new_images['full']['files'], | ||
266 | 258 | version_detail=",".join(environment['version_detail'])) | ||
267 | 259 | |||
268 | 260 | # Add delta images | ||
269 | 261 | for delta in delta_base: | ||
270 | 262 | files = new_images['delta_%s' % delta['version']]['files'] | ||
271 | 263 | logging.info("Publishing new delta from '%s' (%s)" | ||
272 | 264 | " to '%s' (%s) with %s files" % | ||
273 | 265 | (delta['version'], delta.get("description", ""), | ||
274 | 266 | new_version, ",".join(environment['version_detail']), | ||
275 | 267 | len(files))) | ||
276 | 268 | |||
277 | 269 | destination_device.create_image( | ||
278 | 270 | "delta", new_version, | ||
279 | 271 | ",".join(environment['version_detail']), | ||
280 | 272 | files, | ||
281 | 273 | base=delta['version'], | ||
282 | 274 | version_detail=",".join(environment['version_detail'])) | ||
283 | 275 | |||
284 | 276 | # Set the phased percentage value for the new image | ||
285 | 277 | logging.info("Setting phased percentage of the new image to %s%%" % | ||
286 | 278 | args.phased_percentage) | ||
287 | 279 | destination_device.set_phased_percentage( | ||
288 | 280 | new_version, args.phased_percentage) | ||
289 | 281 | |||
290 | 282 | # Expire images | ||
291 | 283 | if args.destination_channel in conf.channels: | ||
292 | 284 | if conf.channels[args.destination_channel].fullcount > 0: | ||
293 | 285 | logging.info("Expiring old images") | ||
294 | 286 | destination_device.expire_images( | ||
295 | 287 | conf.channels[args.destination_channel].fullcount) | ||
296 | 288 | |||
297 | 289 | # Sync all channel aliases | ||
298 | 290 | logging.info("Syncing any existing alias") | ||
299 | 291 | pub.sync_aliases(args.destination_channel) | ||
300 | 292 | |||
301 | 293 | # Remove any orphaned file | ||
302 | 294 | logging.info("Removing orphaned files from the pool") | ||
303 | 295 | pub.cleanup_tree() | ||
304 | 296 | |||
305 | 297 | # Sync the mirrors | ||
306 | 298 | logging.info("Triggering a mirror sync") | ||
307 | 299 | tools.sync_mirrors(conf) | ||
309 | 300 | 34 | ||
310 | === modified file 'bin/set-phased-percentage' (properties changed: +x to -x) | |||
311 | --- bin/set-phased-percentage 2016-06-24 21:02:18 +0000 | |||
312 | +++ bin/set-phased-percentage 2016-07-20 12:34:06 +0000 | |||
313 | @@ -20,77 +20,13 @@ | |||
314 | 20 | import sys | 20 | import sys |
315 | 21 | sys.path.insert(0, os.path.join(sys.path[0], os.pardir, "lib")) | 21 | sys.path.insert(0, os.path.join(sys.path[0], os.pardir, "lib")) |
316 | 22 | 22 | ||
321 | 23 | from systemimage import config, tools, tree | 23 | from systemimage.scripts import script_set_phased_percentage |
322 | 24 | 24 | ||
323 | 25 | import argparse | 25 | |
324 | 26 | import logging | 26 | def main(): |
325 | 27 | script = script_set_phased_percentage.SetPhasedPercentageScript() | ||
326 | 28 | script.run(sys.argv) | ||
327 | 29 | |||
328 | 27 | 30 | ||
329 | 28 | if __name__ == '__main__': | 31 | if __name__ == '__main__': |
398 | 29 | parser = argparse.ArgumentParser(description="set phased percentage") | 32 | main() |
331 | 30 | parser.add_argument("channel", metavar="CHANNEL") | ||
332 | 31 | parser.add_argument("device", metavar="DEVICE") | ||
333 | 32 | parser.add_argument("version", metavar="VERSION", type=int) | ||
334 | 33 | parser.add_argument("percentage", metavar="PERCENTAGE", type=int) | ||
335 | 34 | parser.add_argument("--verbose", "-v", action="count") | ||
336 | 35 | |||
337 | 36 | args = parser.parse_args() | ||
338 | 37 | |||
339 | 38 | # Setup logging | ||
340 | 39 | formatter = logging.Formatter( | ||
341 | 40 | "%(asctime)s %(levelname)s %(message)s") | ||
342 | 41 | |||
343 | 42 | levels = {1: logging.ERROR, | ||
344 | 43 | 2: logging.WARNING, | ||
345 | 44 | 3: logging.INFO, | ||
346 | 45 | 4: logging.DEBUG} | ||
347 | 46 | |||
348 | 47 | if args.verbose > 0: | ||
349 | 48 | stdoutlogger = logging.StreamHandler(sys.stdout) | ||
350 | 49 | stdoutlogger.setFormatter(formatter) | ||
351 | 50 | logging.root.setLevel(levels[min(4, args.verbose)]) | ||
352 | 51 | logging.root.addHandler(stdoutlogger) | ||
353 | 52 | else: | ||
354 | 53 | logging.root.addHandler(logging.NullHandler()) | ||
355 | 54 | |||
356 | 55 | # Load the configuration | ||
357 | 56 | conf = config.Config() | ||
358 | 57 | |||
359 | 58 | # Load the tree | ||
360 | 59 | pub = tree.Tree(conf) | ||
361 | 60 | |||
362 | 61 | # Do some checks | ||
363 | 62 | channels = pub.list_channels() | ||
364 | 63 | if args.channel not in channels: | ||
365 | 64 | parser.error("Invalid channel: %s" % args.channel) | ||
366 | 65 | |||
367 | 66 | if args.device not in channels[args.channel]['devices']: | ||
368 | 67 | parser.error("Invalid device for source channel: %s" % | ||
369 | 68 | args.device) | ||
370 | 69 | |||
371 | 70 | if args.percentage < 0 or args.percentage > 100: | ||
372 | 71 | parser.error("Invalid value: %s" % args.percentage) | ||
373 | 72 | |||
374 | 73 | if "alias" in channels[args.channel] and \ | ||
375 | 74 | channels[args.channel]['alias'] != args.channel: | ||
376 | 75 | parser.error("Channel is an alias.") | ||
377 | 76 | |||
378 | 77 | if "redirect" in channels[args.channel]: | ||
379 | 78 | parser.error("Channel is a redirect.") | ||
380 | 79 | |||
381 | 80 | target_device = channels[args.channel]['devices'][args.device] | ||
382 | 81 | if "redirect" in target_device: | ||
383 | 82 | parser.error("Target device is a redirect. Use original channel " | ||
384 | 83 | "%s instead." % target_device['redirect']) | ||
385 | 84 | |||
386 | 85 | dev = pub.get_device(args.channel, args.device) | ||
387 | 86 | logging.info("Setting phased-percentage of '%s' to %s%%" % | ||
388 | 87 | (args.version, args.percentage)) | ||
389 | 88 | dev.set_phased_percentage(args.version, args.percentage) | ||
390 | 89 | |||
391 | 90 | # Sync all channel aliases | ||
392 | 91 | logging.info("Syncing any existing alias") | ||
393 | 92 | pub.sync_aliases(args.channel) | ||
394 | 93 | |||
395 | 94 | # Sync the mirrors | ||
396 | 95 | logging.info("Triggering a mirror sync") | ||
397 | 96 | tools.sync_mirrors(conf) | ||
399 | 97 | 33 | ||
400 | === modified file 'bin/tag-image' (properties changed: +x to -x) | |||
401 | --- bin/tag-image 2016-06-24 21:02:18 +0000 | |||
402 | +++ bin/tag-image 2016-07-20 12:34:06 +0000 | |||
403 | @@ -1,7 +1,7 @@ | |||
404 | 1 | #!/usr/bin/python | 1 | #!/usr/bin/python |
405 | 2 | # -*- coding: utf-8 -*- | 2 | # -*- coding: utf-8 -*- |
406 | 3 | 3 | ||
408 | 4 | # Copyright (C) 2015 Canonical Ltd. | 4 | # Copyright (C) 2015-2016 Canonical Ltd. |
409 | 5 | # Author: Łukasz 'sil2100' Zemczak <lukasz.zemczak@ubuntu.com> | 5 | # Author: Łukasz 'sil2100' Zemczak <lukasz.zemczak@ubuntu.com> |
410 | 6 | 6 | ||
411 | 7 | # This program is free software: you can redistribute it and/or modify | 7 | # This program is free software: you can redistribute it and/or modify |
412 | @@ -16,191 +16,17 @@ | |||
413 | 16 | # You should have received a copy of the GNU General Public License | 16 | # You should have received a copy of the GNU General Public License |
414 | 17 | # along with this program. If not, see <http://www.gnu.org/licenses/>. | 17 | # along with this program. If not, see <http://www.gnu.org/licenses/>. |
415 | 18 | 18 | ||
416 | 19 | import json | ||
417 | 20 | import os | 19 | import os |
418 | 21 | import sys | 20 | import sys |
419 | 22 | import argparse | ||
420 | 23 | import fcntl | ||
421 | 24 | import logging | ||
422 | 25 | 21 | ||
423 | 26 | sys.path.insert(0, os.path.join(sys.path[0], os.pardir, "lib")) | 22 | sys.path.insert(0, os.path.join(sys.path[0], os.pardir, "lib")) |
424 | 27 | 23 | ||
427 | 28 | from operator import itemgetter | 24 | from systemimage.scripts import script_tag_image |
426 | 29 | from systemimage import config, generators, tools, tree | ||
428 | 30 | 25 | ||
429 | 31 | 26 | ||
430 | 32 | def main(): | 27 | def main(): |
602 | 33 | parser = argparse.ArgumentParser(description="image tagger") | 28 | script = script_tag_image.TagImageScript() |
603 | 34 | parser.add_argument("channel", metavar="CHANNEL") | 29 | script.run(sys.argv) |
433 | 35 | parser.add_argument("device", metavar="DEVICE") | ||
434 | 36 | parser.add_argument("version", metavar="VERSION", type=int) | ||
435 | 37 | parser.add_argument("tag", metavar="TAG") | ||
436 | 38 | parser.add_argument("--verbose", "-v", action="count", default=0) | ||
437 | 39 | |||
438 | 40 | args = parser.parse_args() | ||
439 | 41 | |||
440 | 42 | # Setup logging | ||
441 | 43 | formatter = logging.Formatter( | ||
442 | 44 | "%(asctime)s %(levelname)s %(message)s") | ||
443 | 45 | |||
444 | 46 | levels = {1: logging.ERROR, | ||
445 | 47 | 2: logging.WARNING, | ||
446 | 48 | 3: logging.INFO, | ||
447 | 49 | 4: logging.DEBUG} | ||
448 | 50 | |||
449 | 51 | if args.verbose > 0: | ||
450 | 52 | stdoutlogger = logging.StreamHandler(sys.stdout) | ||
451 | 53 | stdoutlogger.setFormatter(formatter) | ||
452 | 54 | logging.root.setLevel(levels[min(4, args.verbose)]) | ||
453 | 55 | logging.root.addHandler(stdoutlogger) | ||
454 | 56 | else: | ||
455 | 57 | logging.root.addHandler(logging.NullHandler()) | ||
456 | 58 | |||
457 | 59 | # Load the configuration | ||
458 | 60 | conf = config.Config() | ||
459 | 61 | |||
460 | 62 | # Try to acquire a global lock | ||
461 | 63 | lock_file = os.path.join(conf.state_path, "global.lock") | ||
462 | 64 | lock_fd = open(lock_file, 'w') | ||
463 | 65 | |||
464 | 66 | try: | ||
465 | 67 | fcntl.lockf(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) | ||
466 | 68 | except IOError: | ||
467 | 69 | print("Something else holds the global lock. exiting.") | ||
468 | 70 | sys.exit(0) | ||
469 | 71 | |||
470 | 72 | # Load the tree | ||
471 | 73 | pub = tree.Tree(conf) | ||
472 | 74 | |||
473 | 75 | # Do some checks | ||
474 | 76 | channels = pub.list_channels() | ||
475 | 77 | if args.channel not in channels: | ||
476 | 78 | parser.error("Invalid channel: %s" % args.channel) | ||
477 | 79 | |||
478 | 80 | if args.device not in channels[args.channel]['devices']: | ||
479 | 81 | parser.error("Invalid device for channel: %s" % args.device) | ||
480 | 82 | |||
481 | 83 | if "alias" in channels[args.channel] and \ | ||
482 | 84 | channels[args.channel]['alias'] \ | ||
483 | 85 | != args.channel: | ||
484 | 86 | parser.error("Channel is an alias.") | ||
485 | 87 | |||
486 | 88 | if "redirect" in channels[args.channel]: | ||
487 | 89 | parser.error("Channel is a redirect.") | ||
488 | 90 | |||
489 | 91 | target_device = channels[args.channel]['devices'][args.device] | ||
490 | 92 | if "redirect" in target_device: | ||
491 | 93 | parser.error("Target device is a redirect. Use original channel " | ||
492 | 94 | "%s instead." % target_device['redirect']) | ||
493 | 95 | |||
494 | 96 | if "," in args.tag: | ||
495 | 97 | parser.error("Image tag cannot include the character ','.") | ||
496 | 98 | |||
497 | 99 | device = pub.get_device(args.channel, args.device) | ||
498 | 100 | |||
499 | 101 | device_images = device.list_images() | ||
500 | 102 | if not device_images: | ||
501 | 103 | parser.error("No images in selected channel/device.") | ||
502 | 104 | |||
503 | 105 | sorted_images = sorted(device_images, key=itemgetter('version')) | ||
504 | 106 | if args.version != sorted_images[-1]['version']: | ||
505 | 107 | parser.error("You can only tag the latest image.") | ||
506 | 108 | |||
507 | 109 | # Extract the build we want to copy | ||
508 | 110 | images = [image for image in device_images | ||
509 | 111 | if image['type'] == "full" and image['version'] == args.version] | ||
510 | 112 | if not images: | ||
511 | 113 | parser.error("Can't find version: %s" % args.version) | ||
512 | 114 | image = images[0] | ||
513 | 115 | |||
514 | 116 | # Assign a new version number | ||
515 | 117 | new_version = args.version + 1 | ||
516 | 118 | |||
517 | 119 | # Generate a list of required deltas | ||
518 | 120 | delta_base = tools.get_required_deltas( | ||
519 | 121 | conf, pub, args.channel, args.device) | ||
520 | 122 | |||
521 | 123 | # Create a new empty file list | ||
522 | 124 | new_files = [] | ||
523 | 125 | |||
524 | 126 | # Extract current version_detail and image files | ||
525 | 127 | version_detail = tools.extract_files_and_version( | ||
526 | 128 | conf, image['files'], args.version, new_files) | ||
527 | 129 | logging.debug("Original version_detail is: %s" % version_detail) | ||
528 | 130 | |||
529 | 131 | # Generate new version tarball environment | ||
530 | 132 | environment = {} | ||
531 | 133 | environment['channel_name'] = args.channel | ||
532 | 134 | environment['device'] = device | ||
533 | 135 | environment['device_name'] = args.device | ||
534 | 136 | environment['version'] = new_version | ||
535 | 137 | environment['version_detail'] = [entry | ||
536 | 138 | for entry in version_detail.split(",") | ||
537 | 139 | if not entry.startswith("version=")] | ||
538 | 140 | environment['new_files'] = new_files | ||
539 | 141 | |||
540 | 142 | # Set the tag for the image | ||
541 | 143 | tools.set_tag_on_version_detail(environment['version_detail'], args.tag) | ||
542 | 144 | logging.info("Setting tag for image to '%s'" % args.tag) | ||
543 | 145 | |||
544 | 146 | # Generate the new version tarball | ||
545 | 147 | logging.info("Generating new version tarball for '%s' (%s)" | ||
546 | 148 | % (new_version, ",".join(environment['version_detail']))) | ||
547 | 149 | version_path = generators.generate_file(conf, "version", [], environment) | ||
548 | 150 | if version_path: | ||
549 | 151 | new_files.append(version_path) | ||
550 | 152 | |||
551 | 153 | # Add full image | ||
552 | 154 | logging.info("Publishing new image '%s' (%s) with %s files." | ||
553 | 155 | % (new_version, ",".join(environment['version_detail']), | ||
554 | 156 | len(new_files))) | ||
555 | 157 | device.create_image( | ||
556 | 158 | "full", new_version, ",".join(environment['version_detail']), | ||
557 | 159 | new_files, version_detail=",".join(environment['version_detail'])) | ||
558 | 160 | |||
559 | 161 | # Add delta images | ||
560 | 162 | delta_files = [] | ||
561 | 163 | for path in new_files: | ||
562 | 164 | filename = path.split("/")[-1] | ||
563 | 165 | if filename.startswith("version-") or filename.startswith("keyring-"): | ||
564 | 166 | delta_files.append(path) | ||
565 | 167 | |||
566 | 168 | for delta in delta_base: | ||
567 | 169 | logging.info("Publishing new delta from '%s' (%s)" | ||
568 | 170 | " to '%s' (%s)" % | ||
569 | 171 | (delta['version'], delta.get("description", ""), | ||
570 | 172 | new_version, ",".join(environment['version_detail']))) | ||
571 | 173 | |||
572 | 174 | device.create_image( | ||
573 | 175 | "delta", new_version, | ||
574 | 176 | ",".join(environment['version_detail']), | ||
575 | 177 | delta_files, | ||
576 | 178 | base=delta['version'], | ||
577 | 179 | version_detail=",".join(environment['version_detail'])) | ||
578 | 180 | |||
579 | 181 | # Set the phased percentage value for the new image | ||
580 | 182 | percentage = device.get_phased_percentage(args.version) | ||
581 | 183 | logging.info("Setting phased percentage of the new image to %s%%" % | ||
582 | 184 | percentage) | ||
583 | 185 | device.set_phased_percentage(new_version, percentage) | ||
584 | 186 | |||
585 | 187 | # Expire images | ||
586 | 188 | if args.channel in conf.channels: | ||
587 | 189 | if conf.channels[args.channel].fullcount > 0: | ||
588 | 190 | logging.info("Expiring old images") | ||
589 | 191 | device.expire_images(conf.channels[args.channel].fullcount) | ||
590 | 192 | |||
591 | 193 | # Sync all channel aliases | ||
592 | 194 | logging.info("Syncing any existing alias") | ||
593 | 195 | pub.sync_aliases(args.channel) | ||
594 | 196 | |||
595 | 197 | # Remove any orphaned file | ||
596 | 198 | logging.info("Removing orphaned files from the pool") | ||
597 | 199 | pub.cleanup_tree() | ||
598 | 200 | |||
599 | 201 | # Sync the mirrors | ||
600 | 202 | logging.info("Triggering a mirror sync") | ||
601 | 203 | tools.sync_mirrors(conf) | ||
604 | 204 | 30 | ||
605 | 205 | 31 | ||
606 | 206 | if __name__ == '__main__': | 32 | if __name__ == '__main__': |
607 | 207 | 33 | ||
608 | === added file 'lib/systemimage/script.py' | |||
609 | --- lib/systemimage/script.py 1970-01-01 00:00:00 +0000 | |||
610 | +++ lib/systemimage/script.py 2016-07-20 12:34:06 +0000 | |||
611 | @@ -0,0 +1,269 @@ | |||
612 | 1 | # -*- coding: utf-8 -*- | ||
613 | 2 | |||
614 | 3 | # Copyright (C) 2015-2016 Canonical Ltd. | ||
615 | 4 | # Authors: Łukasz 'sil2100' Zemczak <lukasz.zemczak@ubuntu.com> | ||
616 | 5 | # Stéphane Graber <stgraber@ubuntu.com> | ||
617 | 6 | |||
618 | 7 | # This program is free software: you can redistribute it and/or modify | ||
619 | 8 | # it under the terms of the GNU General Public License as published by | ||
620 | 9 | # the Free Software Foundation; version 3 of the License. | ||
621 | 10 | # | ||
622 | 11 | # This program is distributed in the hope that it will be useful, | ||
623 | 12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
624 | 13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
625 | 14 | # GNU General Public License for more details. | ||
626 | 15 | # | ||
627 | 16 | # You should have received a copy of the GNU General Public License | ||
628 | 17 | # along with this program. If not, see <http://www.gnu.org/licenses/>. | ||
629 | 18 | |||
630 | 19 | import json | ||
631 | 20 | import os | ||
632 | 21 | import sys | ||
633 | 22 | import argparse | ||
634 | 23 | import fcntl | ||
635 | 24 | import logging | ||
636 | 25 | |||
637 | 26 | sys.path.insert(0, os.path.join(sys.path[0], os.pardir, "lib")) | ||
638 | 27 | |||
639 | 28 | from operator import itemgetter | ||
640 | 29 | from systemimage import config, generators, tools, tree | ||
641 | 30 | |||
642 | 31 | |||
643 | 32 | class ScriptError(Exception): | ||
644 | 33 | """Exception used to indicate error on any stage of a script.""" | ||
645 | 34 | pass | ||
646 | 35 | |||
647 | 36 | |||
648 | 37 | class Script: | ||
649 | 38 | """Base common class for all system-image scripts.""" | ||
650 | 39 | |||
651 | 40 | def __init__(self, path=None): | ||
652 | 41 | # Load the configuration | ||
653 | 42 | self.conf = config.Config(path) if path else config.Config() | ||
654 | 43 | self.pub = None # system-image tree, defined during run() | ||
655 | 44 | |||
656 | 45 | # Variables to be defined during setup()/execute() | ||
657 | 46 | self.channel = None # Channel name | ||
658 | 47 | self.device = None # Device object | ||
659 | 48 | |||
660 | 49 | self.lock_fd = None | ||
661 | 50 | |||
662 | 51 | def setup_logging(self, verbosity): | ||
663 | 52 | """Helper function setting up common logging scheme.""" | ||
664 | 53 | |||
665 | 54 | formatter = logging.Formatter( | ||
666 | 55 | "%(asctime)s %(levelname)s %(message)s") | ||
667 | 56 | |||
668 | 57 | levels = {1: logging.ERROR, | ||
669 | 58 | 2: logging.WARNING, | ||
670 | 59 | 3: logging.INFO, | ||
671 | 60 | 4: logging.DEBUG} | ||
672 | 61 | |||
673 | 62 | if verbosity > 0: | ||
674 | 63 | stdoutlogger = logging.StreamHandler(sys.stdout) | ||
675 | 64 | stdoutlogger.setFormatter(formatter) | ||
676 | 65 | logging.root.setLevel(levels[min(4, verbosity)]) | ||
677 | 66 | logging.root.addHandler(stdoutlogger) | ||
678 | 67 | else: | ||
679 | 68 | logging.root.addHandler(logging.NullHandler()) | ||
680 | 69 | |||
681 | 70 | def sync_trees(self): | ||
682 | 71 | """Helper function for syncing changes to all trees.""" | ||
683 | 72 | # Sync all channel aliases | ||
684 | 73 | logging.info("Syncing any existing alias") | ||
685 | 74 | self.pub.sync_aliases(self.channel) | ||
686 | 75 | |||
687 | 76 | # Sync the mirrors | ||
688 | 77 | logging.info("Triggering a mirror sync") | ||
689 | 78 | tools.sync_mirrors(self.conf) | ||
690 | 79 | |||
691 | 80 | def lock(self): | ||
692 | 81 | """Aquire the global lock, protecting from races.""" | ||
693 | 82 | |||
694 | 83 | # Try to acquire a global lock | ||
695 | 84 | lock_file = os.path.join(self.conf.state_path, "global.lock") | ||
696 | 85 | self.lock_fd = open(lock_file, 'w') | ||
697 | 86 | |||
698 | 87 | try: | ||
699 | 88 | fcntl.lockf(self.lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) | ||
700 | 89 | except IOError: | ||
701 | 90 | raise ScriptError("Something else holds the global lock. exiting.") | ||
702 | 91 | |||
703 | 92 | def presetup(self): | ||
704 | 93 | """All low-level pre-setup to be done before the script can be run.""" | ||
705 | 94 | # Load the tree | ||
706 | 95 | self.pub = tree.Tree(self.conf) | ||
707 | 96 | |||
708 | 97 | def setup(self, argv): | ||
709 | 98 | """Implementable in script. Perform script setup, parse cmdline.""" | ||
710 | 99 | pass | ||
711 | 100 | |||
712 | 101 | def execute(self): | ||
713 | 102 | """Implementable in script. All script execution code goes here.""" | ||
714 | 103 | pass | ||
715 | 104 | |||
716 | 105 | def cleanup(self): | ||
717 | 106 | """Called at the end - by default provides image cleanup.""" | ||
718 | 107 | |||
719 | 108 | # Expire images, if a device was used in the script | ||
720 | 109 | if self.device and self.channel in self.conf.channels: | ||
721 | 110 | if self.conf.channels[self.channel].fullcount > 0: | ||
722 | 111 | logging.info("Expiring old images") | ||
723 | 112 | self.device.expire_images( | ||
724 | 113 | self.conf.channels[ | ||
725 | 114 | self.channel].fullcount) | ||
726 | 115 | |||
727 | 116 | # Remove any orphaned file | ||
728 | 117 | logging.info("Removing orphaned files from the pool") | ||
729 | 118 | self.pub.cleanup_tree() | ||
730 | 119 | |||
731 | 120 | self.sync_trees() | ||
732 | 121 | |||
733 | 122 | def run(self, argv): | ||
734 | 123 | """Run the script.""" | ||
735 | 124 | |||
736 | 125 | try: | ||
737 | 126 | self.lock() | ||
738 | 127 | self.presetup() | ||
739 | 128 | self.setup(argv) | ||
740 | 129 | self.execute() | ||
741 | 130 | self.cleanup() | ||
742 | 131 | except ScriptError as err: | ||
743 | 132 | print(str(err)) | ||
744 | 133 | return False | ||
745 | 134 | |||
746 | 135 | return True | ||
747 | 136 | |||
748 | 137 | |||
749 | 138 | class CommonImageManipulationScript(Script): | ||
750 | 139 | """A common base for scripts that manipulate images, providing helpers.""" | ||
751 | 140 | |||
752 | 141 | def generate_version(self, source_files, version, tag=None): | ||
753 | 142 | """ | ||
754 | 143 | Generate the version tarball for the version and files. | ||
755 | 144 | Requires the self.target_version, self.channel and self.device | ||
756 | 145 | variables: for the new version number, destination channel and | ||
757 | 146 | device object respectively. | ||
758 | 147 | """ | ||
759 | 148 | |||
760 | 149 | # Extract current version_detail and files | ||
761 | 150 | version_detail = tools.extract_files_and_version( | ||
762 | 151 | self.conf, | ||
763 | 152 | source_files, | ||
764 | 153 | version, | ||
765 | 154 | self.new_images['full']['files']) | ||
766 | 155 | |||
767 | 156 | # Generate new version tarball | ||
768 | 157 | self.environment = {} | ||
769 | 158 | self.environment['channel_name'] = self.channel | ||
770 | 159 | self.environment['device'] = self.device | ||
771 | 160 | self.environment['device_name'] = self.device.name | ||
772 | 161 | self.environment['version'] = self.target_version | ||
773 | 162 | self.environment['version_detail'] = [ | ||
774 | 163 | entry for entry in version_detail.split(",") | ||
775 | 164 | if not entry.startswith("version=")] | ||
776 | 165 | self.environment['new_files'] = self.new_images['full']['files'] | ||
777 | 166 | |||
778 | 167 | # Add new tag if requested | ||
779 | 168 | if tag: | ||
780 | 169 | tools.set_tag_on_version_detail( | ||
781 | 170 | self.environment['version_detail'], tag) | ||
782 | 171 | logging.info("Setting tag for image to '%s'" % tag) | ||
783 | 172 | |||
784 | 173 | logging.info("Generating new version tarball for '%s' (%s)" | ||
785 | 174 | % (self.target_version, ",".join( | ||
786 | 175 | self.environment['version_detail']))) | ||
787 | 176 | version_path = generators.generate_file( | ||
788 | 177 | self.conf, "version", [], self.environment) | ||
789 | 178 | if version_path: | ||
790 | 179 | self.new_images['full']['files'].append(version_path) | ||
791 | 180 | |||
792 | 181 | def generate_deltas(self): | ||
793 | 182 | """ | ||
794 | 183 | Generates deltas from the list in self.delta_base. | ||
795 | 184 | To generate the deltas the self.new_images dictionary is used. All | ||
796 | 185 | the newly generated delta files are then appended to the new_images | ||
797 | 186 | structure. | ||
798 | 187 | """ | ||
799 | 188 | |||
800 | 189 | for delta in self.delta_base: | ||
801 | 190 | self.new_images['delta_%s' % delta['version']] = {'files': []} | ||
802 | 191 | |||
803 | 192 | # Generate deltas | ||
804 | 193 | for abspath in self.new_images['full']['files']: | ||
805 | 194 | prefix = abspath.split("/")[-1].rsplit("-", 1)[0] | ||
806 | 195 | for delta in self.delta_base: | ||
807 | 196 | # Extract the source | ||
808 | 197 | src_path = None | ||
809 | 198 | for file_dict in delta['files']: | ||
810 | 199 | if (file_dict['path'].split("/")[-1] | ||
811 | 200 | .startswith(prefix)): | ||
812 | 201 | src_path = "%s/%s" % (self.conf.publish_path, | ||
813 | 202 | file_dict['path']) | ||
814 | 203 | break | ||
815 | 204 | |||
816 | 205 | # Check that it's not the current file | ||
817 | 206 | if src_path: | ||
818 | 207 | src_path = os.path.realpath(src_path) | ||
819 | 208 | |||
820 | 209 | # FIXME: the keyring- is a big hack... | ||
821 | 210 | if src_path == abspath and "keyring-" not in src_path: | ||
822 | 211 | continue | ||
823 | 212 | |||
824 | 213 | # Generators are allowed to return None when no delta | ||
825 | 214 | # exists at all. | ||
826 | 215 | logging.info("Generating delta from '%s' for '%s'" % | ||
827 | 216 | (delta['version'], | ||
828 | 217 | prefix)) | ||
829 | 218 | delta_path = generators.generate_delta(self.conf, src_path, | ||
830 | 219 | abspath) | ||
831 | 220 | else: | ||
832 | 221 | delta_path = abspath | ||
833 | 222 | |||
834 | 223 | if not delta_path: | ||
835 | 224 | continue | ||
836 | 225 | |||
837 | 226 | # Get the full and relative paths | ||
838 | 227 | delta_abspath, delta_relpath = tools.expand_path( | ||
839 | 228 | delta_path, self.conf.publish_path) | ||
840 | 229 | |||
841 | 230 | self.new_images['delta_%s' % delta['version']]['files'] \ | ||
842 | 231 | .append(delta_abspath) | ||
843 | 232 | |||
844 | 233 | def add_images_to_index(self): | ||
845 | 234 | """ | ||
846 | 235 | Adds all newly created image files to the index. | ||
847 | 236 | This basically takes all the new entries from the self.new_images | ||
848 | 237 | dictionary (full and delta entries) and adds them, calling | ||
849 | 238 | create_image on the device object. | ||
850 | 239 | """ | ||
851 | 240 | |||
852 | 241 | # Add full image | ||
853 | 242 | logging.info("Publishing new image '%s' (%s) with %s files." | ||
854 | 243 | % (self.target_version, | ||
855 | 244 | ",".join(self.environment['version_detail']), | ||
856 | 245 | len(self.new_images['full']['files']))) | ||
857 | 246 | self.device.create_image( | ||
858 | 247 | "full", | ||
859 | 248 | self.target_version, | ||
860 | 249 | ",".join(self.environment['version_detail']), | ||
861 | 250 | self.new_images['full']['files'], | ||
862 | 251 | version_detail=",".join(self.environment['version_detail'])) | ||
863 | 252 | |||
864 | 253 | # Add delta images | ||
865 | 254 | for delta in self.delta_base: | ||
866 | 255 | files = self.new_images['delta_%s' % delta['version']]['files'] | ||
867 | 256 | logging.info("Publishing new delta from '%s' (%s)" | ||
868 | 257 | " to '%s' (%s) with %s files" % | ||
869 | 258 | (delta['version'], | ||
870 | 259 | delta.get("description", ""), | ||
871 | 260 | self.target_version, | ||
872 | 261 | ",".join(self.environment['version_detail']), | ||
873 | 262 | len(files))) | ||
874 | 263 | |||
875 | 264 | self.device.create_image( | ||
876 | 265 | "delta", self.target_version, | ||
877 | 266 | ",".join(self.environment['version_detail']), | ||
878 | 267 | files, | ||
879 | 268 | base=delta['version'], | ||
880 | 269 | version_detail=",".join(self.environment['version_detail'])) | ||
881 | 0 | 270 | ||
882 | === added directory 'lib/systemimage/scripts' | |||
883 | === added file 'lib/systemimage/scripts/__init__.py' | |||
884 | === added file 'lib/systemimage/scripts/script_copy_image.py' | |||
885 | --- lib/systemimage/scripts/script_copy_image.py 1970-01-01 00:00:00 +0000 | |||
886 | +++ lib/systemimage/scripts/script_copy_image.py 2016-07-20 12:34:06 +0000 | |||
887 | @@ -0,0 +1,187 @@ | |||
888 | 1 | # -*- coding: utf-8 -*- | ||
889 | 2 | |||
890 | 3 | # Copyright (C) 2016 Canonical Ltd. | ||
891 | 4 | # Authors: Łukasz 'sil2100' Zemczak <lukasz.zemczak@ubuntu.com> | ||
892 | 5 | # Stéphane Graber <stgraber@ubuntu.com> | ||
893 | 6 | |||
894 | 7 | # This program is free software: you can redistribute it and/or modify | ||
895 | 8 | # it under the terms of the GNU General Public License as published by | ||
896 | 9 | # the Free Software Foundation; version 3 of the License. | ||
897 | 10 | # | ||
898 | 11 | # This program is distributed in the hope that it will be useful, | ||
899 | 12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
900 | 13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
901 | 14 | # GNU General Public License for more details. | ||
902 | 15 | # | ||
903 | 16 | # You should have received a copy of the GNU General Public License | ||
904 | 17 | # along with this program. If not, see <http://www.gnu.org/licenses/>. | ||
905 | 18 | |||
906 | 19 | import json | ||
907 | 20 | import os | ||
908 | 21 | import sys | ||
909 | 22 | import argparse | ||
910 | 23 | import fcntl | ||
911 | 24 | import logging | ||
912 | 25 | |||
913 | 26 | from systemimage import config, generators, tools, tree, script | ||
914 | 27 | from systemimage.script import ScriptError | ||
915 | 28 | |||
916 | 29 | |||
917 | 30 | class CopyImageScript(script.CommonImageManipulationScript): | ||
918 | 31 | """Script used to copy images from channel to channel.""" | ||
919 | 32 | |||
920 | 33 | def setup(self, argv): | ||
921 | 34 | parser = argparse.ArgumentParser(description="image copier") | ||
922 | 35 | parser.add_argument("source_channel", metavar="SOURCE-CHANNEL") | ||
923 | 36 | parser.add_argument("destination_channel", | ||
924 | 37 | metavar="DESTINATION-CHANNEL") | ||
925 | 38 | parser.add_argument("device", metavar="DEVICE") | ||
926 | 39 | parser.add_argument("version", metavar="VERSION", type=int) | ||
927 | 40 | parser.add_argument("-k", "--keep-version", action="store_true", | ||
928 | 41 | help="Keep the original version number") | ||
929 | 42 | parser.add_argument("-p", "--phased-percentage", type=int, | ||
930 | 43 | help="Set the phased percentage for the copied " | ||
931 | 44 | "image", | ||
932 | 45 | default=100) | ||
933 | 46 | parser.add_argument("-t", "--tag", type=str, | ||
934 | 47 | help="Set a version tag on the new image") | ||
935 | 48 | parser.add_argument("--verbose", "-v", action="count", default=0) | ||
936 | 49 | |||
937 | 50 | argv.pop(0) # To fix issues with parse_args() shifting arguments | ||
938 | 51 | args = parser.parse_args(argv) | ||
939 | 52 | |||
940 | 53 | # Setup logging | ||
941 | 54 | self.setup_logging(args.verbose) | ||
942 | 55 | |||
943 | 56 | # Do some checks | ||
944 | 57 | channels = self.pub.list_channels() | ||
945 | 58 | if args.source_channel not in channels: | ||
946 | 59 | raise ScriptError("Invalid source channel: %s" % | ||
947 | 60 | args.source_channel) | ||
948 | 61 | |||
949 | 62 | if args.destination_channel not in channels: | ||
950 | 63 | raise ScriptError("Invalid destination channel: %s" % | ||
951 | 64 | args.destination_channel) | ||
952 | 65 | |||
953 | 66 | if args.device not in \ | ||
954 | 67 | channels[args.source_channel]['devices']: | ||
955 | 68 | raise ScriptError("Invalid device for source channel: %s" % | ||
956 | 69 | args.device) | ||
957 | 70 | |||
958 | 71 | if args.device not in \ | ||
959 | 72 | channels[args.destination_channel]['devices']: | ||
960 | 73 | raise ScriptError("Invalid device for destination channel: %s" % | ||
961 | 74 | args.device) | ||
962 | 75 | |||
963 | 76 | if "alias" in channels[args.source_channel] and \ | ||
964 | 77 | channels[args.source_channel]['alias'] \ | ||
965 | 78 | != args.source_channel: | ||
966 | 79 | raise ScriptError("Source channel is an alias.") | ||
967 | 80 | |||
968 | 81 | if "alias" in channels[args.destination_channel] and \ | ||
969 | 82 | channels[args.destination_channel]['alias'] \ | ||
970 | 83 | != args.destination_channel: | ||
971 | 84 | raise ScriptError("Destination channel is an alias.") | ||
972 | 85 | |||
973 | 86 | if "redirect" in channels[args.source_channel]: | ||
974 | 87 | raise ScriptError("Source channel is a redirect.") | ||
975 | 88 | |||
976 | 89 | if "redirect" in channels[args.destination_channel]: | ||
977 | 90 | raise ScriptError("Destination channel is a redirect.") | ||
978 | 91 | |||
979 | 92 | src_device = channels[args.source_channel]['devices'][args.device] | ||
980 | 93 | if "redirect" in src_device: | ||
981 | 94 | raise ScriptError("Source device is a redirect. Use original " | ||
982 | 95 | "channel %s instead." % src_device['redirect']) | ||
983 | 96 | |||
984 | 97 | dest_device = \ | ||
985 | 98 | channels[args.destination_channel]['devices'][args.device] | ||
986 | 99 | if "redirect" in dest_device: | ||
987 | 100 | raise ScriptError("Destination device is a redirect. Use original " | ||
988 | 101 | "channel %s instead." % dest_device['redirect']) | ||
989 | 102 | |||
990 | 103 | if args.tag and "," in args.tag: | ||
991 | 104 | raise ScriptError("Image tag cannot include the character ','.") | ||
992 | 105 | |||
993 | 106 | self.args = args | ||
994 | 107 | self.channel = args.destination_channel | ||
995 | 108 | |||
996 | 109 | def assign_new_version(self, version, keep_version=False): | ||
997 | 110 | if keep_version: | ||
998 | 111 | images = [image for image in self.device.list_images() | ||
999 | 112 | if image['version'] == version] | ||
1000 | 113 | if images: | ||
1001 | 114 | raise ScriptError( | ||
1002 | 115 | "Version number is already used: %s" % version) | ||
1003 | 116 | |||
1004 | 117 | # Assign a new version number | ||
1005 | 118 | new_version = version | ||
1006 | 119 | if not keep_version: | ||
1007 | 120 | # Find the next available version | ||
1008 | 121 | new_version = 1 | ||
1009 | 122 | for image in self.device.list_images(): | ||
1010 | 123 | if image['version'] >= new_version: | ||
1011 | 124 | new_version = image['version'] + 1 | ||
1012 | 125 | logging.debug("Version for next image: %s" % new_version) | ||
1013 | 126 | |||
1014 | 127 | self.target_version = new_version | ||
1015 | 128 | |||
1016 | 129 | def execute(self): | ||
1017 | 130 | source_device = self.pub.get_device( | ||
1018 | 131 | self.args.source_channel, self.args.device) | ||
1019 | 132 | self.device = self.pub.get_device( | ||
1020 | 133 | self.channel, self.args.device) | ||
1021 | 134 | |||
1022 | 135 | # Assign a version number to the new image | ||
1023 | 136 | self.assign_new_version(self.args.version, self.args.keep_version) | ||
1024 | 137 | |||
1025 | 138 | # Extract the build we want to copy | ||
1026 | 139 | source_image = tools.extract_image( | ||
1027 | 140 | source_device.list_images(), self.args.version) | ||
1028 | 141 | if not source_image: | ||
1029 | 142 | raise ScriptError("Can't find version: %s" % version) | ||
1030 | 143 | |||
1031 | 144 | # Extract the list of existing full images | ||
1032 | 145 | full_images = {image['version']: image | ||
1033 | 146 | for image in self.device.list_images() | ||
1034 | 147 | if image['type'] == "full"} | ||
1035 | 148 | |||
1036 | 149 | # Check that the last full and the new image aren't one and the same | ||
1037 | 150 | source_files = [entry['path'].split("/")[-1] | ||
1038 | 151 | for entry in source_image['files'] | ||
1039 | 152 | if not entry['path'].split("/")[-1].startswith( | ||
1040 | 153 | "version-")] | ||
1041 | 154 | destination_files = [] | ||
1042 | 155 | if full_images: | ||
1043 | 156 | latest_full = sorted(full_images.values(), | ||
1044 | 157 | key=lambda image: image['version'])[-1] | ||
1045 | 158 | destination_files = [entry['path'].split("/")[-1] | ||
1046 | 159 | for entry in latest_full['files'] | ||
1047 | 160 | if not entry['path'].split( | ||
1048 | 161 | "/")[-1].startswith("version-")] | ||
1049 | 162 | if source_files == destination_files: | ||
1050 | 163 | raise ScriptError("Source image is already latest full in " | ||
1051 | 164 | "destination channel.") | ||
1052 | 165 | |||
1053 | 166 | # Generate a list of required deltas | ||
1054 | 167 | self.delta_base = tools.get_required_deltas( | ||
1055 | 168 | self.conf, self.pub, self.channel, self.args.device) | ||
1056 | 169 | |||
1057 | 170 | # Create new empty entries | ||
1058 | 171 | self.new_images = {'full': {'files': []}} | ||
1059 | 172 | |||
1060 | 173 | # Generate the version tarball | ||
1061 | 174 | self.generate_version( | ||
1062 | 175 | source_image['files'], self.args.version, self.args.tag) | ||
1063 | 176 | |||
1064 | 177 | # Generate all required deltas | ||
1065 | 178 | self.generate_deltas() | ||
1066 | 179 | |||
1067 | 180 | # Add both full and delta images to the index | ||
1068 | 181 | self.add_images_to_index() | ||
1069 | 182 | |||
1070 | 183 | # Set the phased percentage value for the new image | ||
1071 | 184 | logging.info("Setting phased percentage of the new image to %s%%" % | ||
1072 | 185 | self.args.phased_percentage) | ||
1073 | 186 | self.device.set_phased_percentage( | ||
1074 | 187 | self.target_version, self.args.phased_percentage) | ||
1075 | 0 | 188 | ||
1076 | === added file 'lib/systemimage/scripts/script_set_phased_percentage.py' | |||
1077 | --- lib/systemimage/scripts/script_set_phased_percentage.py 1970-01-01 00:00:00 +0000 | |||
1078 | +++ lib/systemimage/scripts/script_set_phased_percentage.py 2016-07-20 12:34:06 +0000 | |||
1079 | @@ -0,0 +1,83 @@ | |||
1080 | 1 | # -*- coding: utf-8 -*- | ||
1081 | 2 | |||
1082 | 3 | # Copyright (C) 2016 Canonical Ltd. | ||
1083 | 4 | # Authors: Łukasz 'sil2100' Zemczak <lukasz.zemczak@ubuntu.com> | ||
1084 | 5 | # Stéphane Graber <stgraber@ubuntu.com> | ||
1085 | 6 | |||
1086 | 7 | # This program is free software: you can redistribute it and/or modify | ||
1087 | 8 | # it under the terms of the GNU General Public License as published by | ||
1088 | 9 | # the Free Software Foundation; version 3 of the License. | ||
1089 | 10 | # | ||
1090 | 11 | # This program is distributed in the hope that it will be useful, | ||
1091 | 12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
1092 | 13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
1093 | 14 | # GNU General Public License for more details. | ||
1094 | 15 | # | ||
1095 | 16 | # You should have received a copy of the GNU General Public License | ||
1096 | 17 | # along with this program. If not, see <http://www.gnu.org/licenses/>. | ||
1097 | 18 | |||
1098 | 19 | import json | ||
1099 | 20 | import os | ||
1100 | 21 | import sys | ||
1101 | 22 | import argparse | ||
1102 | 23 | import fcntl | ||
1103 | 24 | import logging | ||
1104 | 25 | |||
1105 | 26 | from systemimage import config, generators, tools, tree, script | ||
1106 | 27 | from systemimage.script import ScriptError | ||
1107 | 28 | |||
1108 | 29 | |||
1109 | 30 | class SetPhasedPercentageScript(script.CommonImageManipulationScript): | ||
1110 | 31 | """Script used to set phased percentages of images.""" | ||
1111 | 32 | |||
1112 | 33 | def setup(self, argv): | ||
1113 | 34 | parser = argparse.ArgumentParser(description="set phased percentage") | ||
1114 | 35 | parser.add_argument("channel", metavar="CHANNEL") | ||
1115 | 36 | parser.add_argument("device", metavar="DEVICE") | ||
1116 | 37 | parser.add_argument("version", metavar="VERSION", type=int) | ||
1117 | 38 | parser.add_argument("percentage", metavar="PERCENTAGE", type=int) | ||
1118 | 39 | parser.add_argument("--verbose", "-v", action="count", default=0) | ||
1119 | 40 | |||
1120 | 41 | argv.pop(0) # To fix issues with parse_args() shifting arguments | ||
1121 | 42 | args = parser.parse_args(argv) | ||
1122 | 43 | |||
1123 | 44 | # Setup logging | ||
1124 | 45 | self.setup_logging(args.verbose) | ||
1125 | 46 | |||
1126 | 47 | # Do some checks | ||
1127 | 48 | channels = self.pub.list_channels() | ||
1128 | 49 | if args.channel not in channels: | ||
1129 | 50 | raise ScriptError("Invalid channel: %s" % args.channel) | ||
1130 | 51 | |||
1131 | 52 | if args.device not in channels[args.channel]['devices']: | ||
1132 | 53 | raise ScriptError("Invalid device for source channel: %s" % | ||
1133 | 54 | args.device) | ||
1134 | 55 | |||
1135 | 56 | if args.percentage < 0 or args.percentage > 100: | ||
1136 | 57 | raise ScriptError("Invalid value: %s" % args.percentage) | ||
1137 | 58 | |||
1138 | 59 | if "alias" in channels[args.channel] and \ | ||
1139 | 60 | channels[args.channel]['alias'] != args.channel: | ||
1140 | 61 | raise ScriptError("Channel is an alias.") | ||
1141 | 62 | |||
1142 | 63 | if "redirect" in channels[args.channel]: | ||
1143 | 64 | raise ScriptError("Channel is a redirect.") | ||
1144 | 65 | |||
1145 | 66 | target_device = channels[args.channel]['devices'][args.device] | ||
1146 | 67 | if "redirect" in target_device: | ||
1147 | 68 | raise ScriptError( | ||
1148 | 69 | "Target device is a redirect. Use original channel %s " | ||
1149 | 70 | "instead." % target_device['redirect']) | ||
1150 | 71 | |||
1151 | 72 | self.args = args | ||
1152 | 73 | |||
1153 | 74 | def execute(self): | ||
1154 | 75 | dev = self.pub.get_device(self.args.channel, self.args.device) | ||
1155 | 76 | logging.info("Setting phased-percentage of '%s' to %s%%" % | ||
1156 | 77 | (self.args.version, self.args.percentage)) | ||
1157 | 78 | dev.set_phased_percentage(self.args.version, self.args.percentage) | ||
1158 | 79 | |||
1159 | 80 | def cleanup(self): | ||
1160 | 81 | # We're overwriting this method as there's no need to expire images | ||
1161 | 82 | # after running this command | ||
1162 | 83 | self.sync_trees() | ||
1163 | 0 | 84 | ||
1164 | === added file 'lib/systemimage/scripts/script_tag_image.py' | |||
1165 | --- lib/systemimage/scripts/script_tag_image.py 1970-01-01 00:00:00 +0000 | |||
1166 | +++ lib/systemimage/scripts/script_tag_image.py 2016-07-20 12:34:06 +0000 | |||
1167 | @@ -0,0 +1,127 @@ | |||
1168 | 1 | # -*- coding: utf-8 -*- | ||
1169 | 2 | |||
1170 | 3 | # Copyright (C) 2016 Canonical Ltd. | ||
1171 | 4 | # Authors: Łukasz 'sil2100' Zemczak <lukasz.zemczak@ubuntu.com> | ||
1172 | 5 | # Stéphane Graber <stgraber@ubuntu.com> | ||
1173 | 6 | |||
1174 | 7 | # This program is free software: you can redistribute it and/or modify | ||
1175 | 8 | # it under the terms of the GNU General Public License as published by | ||
1176 | 9 | # the Free Software Foundation; version 3 of the License. | ||
1177 | 10 | # | ||
1178 | 11 | # This program is distributed in the hope that it will be useful, | ||
1179 | 12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
1180 | 13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
1181 | 14 | # GNU General Public License for more details. | ||
1182 | 15 | # | ||
1183 | 16 | # You should have received a copy of the GNU General Public License | ||
1184 | 17 | # along with this program. If not, see <http://www.gnu.org/licenses/>. | ||
1185 | 18 | |||
1186 | 19 | import json | ||
1187 | 20 | import os | ||
1188 | 21 | import sys | ||
1189 | 22 | import argparse | ||
1190 | 23 | import fcntl | ||
1191 | 24 | import logging | ||
1192 | 25 | |||
1193 | 26 | from operator import itemgetter | ||
1194 | 27 | from systemimage import config, generators, tools, tree, script | ||
1195 | 28 | from systemimage.script import ScriptError | ||
1196 | 29 | |||
1197 | 30 | |||
1198 | 31 | class TagImageScript(script.CommonImageManipulationScript): | ||
1199 | 32 | """Script used to append/modify tags in images.""" | ||
1200 | 33 | |||
1201 | 34 | def setup(self, argv): | ||
1202 | 35 | parser = argparse.ArgumentParser(description="image tagger") | ||
1203 | 36 | parser.add_argument("channel", metavar="CHANNEL") | ||
1204 | 37 | parser.add_argument("device", metavar="DEVICE") | ||
1205 | 38 | parser.add_argument("version", metavar="VERSION", type=int) | ||
1206 | 39 | parser.add_argument("tag", metavar="TAG") | ||
1207 | 40 | parser.add_argument("--verbose", "-v", action="count", default=0) | ||
1208 | 41 | |||
1209 | 42 | argv.pop(0) # To fix issues with parse_args() shifting arguments | ||
1210 | 43 | args = parser.parse_args(argv) | ||
1211 | 44 | |||
1212 | 45 | # Setup logging | ||
1213 | 46 | self.setup_logging(args.verbose) | ||
1214 | 47 | |||
1215 | 48 | # Do some checks | ||
1216 | 49 | channels = self.pub.list_channels() | ||
1217 | 50 | if args.channel not in channels: | ||
1218 | 51 | raise ScriptError("Invalid channel: %s" % args.channel) | ||
1219 | 52 | |||
1220 | 53 | if args.device not in \ | ||
1221 | 54 | channels[args.channel]['devices']: | ||
1222 | 55 | raise ScriptError("Invalid device for channel: %s" % args.device) | ||
1223 | 56 | |||
1224 | 57 | if "alias" in channels[args.channel] and \ | ||
1225 | 58 | channels[args.channel]['alias'] \ | ||
1226 | 59 | != args.channel: | ||
1227 | 60 | raise ScriptError("Channel is an alias.") | ||
1228 | 61 | |||
1229 | 62 | if "redirect" in channels[args.channel]: | ||
1230 | 63 | raise ScriptError("Channel is a redirect.") | ||
1231 | 64 | |||
1232 | 65 | target_device = channels[args.channel]['devices'][args.device] | ||
1233 | 66 | if "redirect" in target_device: | ||
1234 | 67 | raise ScriptError( | ||
1235 | 68 | "Target device is a redirect. Use original channel %s " | ||
1236 | 69 | "instead." % target_device['redirect']) | ||
1237 | 70 | |||
1238 | 71 | if "," in args.tag: | ||
1239 | 72 | raise ScriptError("Image tag cannot include the character ','.") | ||
1240 | 73 | |||
1241 | 74 | self.args = args | ||
1242 | 75 | self.channel = args.channel | ||
1243 | 76 | |||
1244 | 77 | def execute(self): | ||
1245 | 78 | self.device = self.pub.get_device(self.args.channel, self.args.device) | ||
1246 | 79 | |||
1247 | 80 | device_images = self.device.list_images() | ||
1248 | 81 | if not device_images: | ||
1249 | 82 | raise ScriptError("No images in selected channel/device.") | ||
1250 | 83 | |||
1251 | 84 | sorted_images = sorted(device_images, key=itemgetter('version')) | ||
1252 | 85 | if self.args.version != sorted_images[-1]['version']: | ||
1253 | 86 | raise ScriptError("You can only tag the latest image.") | ||
1254 | 87 | |||
1255 | 88 | # Extract the build we want to copy | ||
1256 | 89 | self.image = tools.extract_image(device_images, self.args.version) | ||
1257 | 90 | if not self.image: | ||
1258 | 91 | raise ScriptError("Can't find version: %s" % version) | ||
1259 | 92 | |||
1260 | 93 | # Remember the current image's phased percentage | ||
1261 | 94 | percentage = self.device.get_phased_percentage(self.args.version) | ||
1262 | 95 | |||
1263 | 96 | # Assign a new version number | ||
1264 | 97 | self.target_version = self.args.version + 1 | ||
1265 | 98 | |||
1266 | 99 | # Generate a list of required deltas | ||
1267 | 100 | self.delta_base = tools.get_required_deltas( | ||
1268 | 101 | self.conf, self.pub, self.channel, self.args.device) | ||
1269 | 102 | |||
1270 | 103 | # Create new empty entries | ||
1271 | 104 | self.new_images = {'full': {'files': []}} | ||
1272 | 105 | |||
1273 | 106 | self.generate_version( | ||
1274 | 107 | self.image['files'], self.args.version, self.args.tag) | ||
1275 | 108 | |||
1276 | 109 | # Add delta images | ||
1277 | 110 | delta_files = [] | ||
1278 | 111 | for path in self.new_images['full']['files']: | ||
1279 | 112 | filename = path.split("/")[-1] | ||
1280 | 113 | if (filename.startswith("version-") or | ||
1281 | 114 | filename.startswith("keyring-")): | ||
1282 | 115 | delta_files.append(path) | ||
1283 | 116 | |||
1284 | 117 | for delta in self.delta_base: | ||
1285 | 118 | self.new_images['delta_%s' % delta['version']] = \ | ||
1286 | 119 | {'files': delta_files} | ||
1287 | 120 | |||
1288 | 121 | # Add both full and delta images to the index | ||
1289 | 122 | self.add_images_to_index() | ||
1290 | 123 | |||
1291 | 124 | # Set the phased percentage of the image to the previous value | ||
1292 | 125 | logging.info("Setting phased percentage of the new image to %s%%" % | ||
1293 | 126 | percentage) | ||
1294 | 127 | self.device.set_phased_percentage(self.target_version, percentage) | ||
1295 | 0 | 128 | ||
1296 | === added file 'lib/systemimage/tests/test_scripts.py' | |||
1297 | --- lib/systemimage/tests/test_scripts.py 1970-01-01 00:00:00 +0000 | |||
1298 | +++ lib/systemimage/tests/test_scripts.py 2016-07-20 12:34:06 +0000 | |||
1299 | @@ -0,0 +1,590 @@ | |||
1300 | 1 | # -*- coding: utf-8 -*- | ||
1301 | 2 | |||
1302 | 3 | # Copyright (C) 2015-2016 Canonical Ltd. | ||
1303 | 4 | # Author: Łukasz 'sil2100' Zemczak <lukasz.zemczak@ubuntu.com> | ||
1304 | 5 | |||
1305 | 6 | # This program is free software: you can redistribute it and/or modify | ||
1306 | 7 | # it under the terms of the GNU General Public License as published by | ||
1307 | 8 | # the Free Software Foundation; version 3 of the License. | ||
1308 | 9 | # | ||
1309 | 10 | # This program is distributed in the hope that it will be useful, | ||
1310 | 11 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
1311 | 12 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
1312 | 13 | # GNU General Public License for more details. | ||
1313 | 14 | # | ||
1314 | 15 | # You should have received a copy of the GNU General Public License | ||
1315 | 16 | # along with this program. If not, see <http://www.gnu.org/licenses/>. | ||
1316 | 17 | |||
1317 | 18 | import os | ||
1318 | 19 | import shutil | ||
1319 | 20 | import stat | ||
1320 | 21 | import subprocess | ||
1321 | 22 | import tarfile | ||
1322 | 23 | import tempfile | ||
1323 | 24 | import unittest | ||
1324 | 25 | import json | ||
1325 | 26 | import six | ||
1326 | 27 | import fcntl | ||
1327 | 28 | |||
1328 | 29 | from mock import Mock, patch | ||
1329 | 30 | from systemimage import config, tools, tree, gpg | ||
1330 | 31 | from systemimage import script as script_base | ||
1331 | 32 | from systemimage.scripts import ( | ||
1332 | 33 | script_copy_image, | ||
1333 | 34 | script_tag_image, | ||
1334 | 35 | script_set_phased_percentage, | ||
1335 | 36 | ) | ||
1336 | 37 | from systemimage.testing.helpers import HAS_TEST_KEYS, MISSING_KEYS_WARNING | ||
1337 | 38 | from systemimage.tools import xz_uncompress | ||
1338 | 39 | |||
1339 | 40 | |||
1340 | 41 | class ScriptTests(unittest.TestCase): | ||
1341 | 42 | def setUp(self): | ||
1342 | 43 | temp_directory = tempfile.mkdtemp() | ||
1343 | 44 | self.temp_directory = temp_directory | ||
1344 | 45 | self.old_path = os.environ.get("PATH", None) | ||
1345 | 46 | |||
1346 | 47 | os.mkdir(os.path.join(self.temp_directory, "etc")) | ||
1347 | 48 | state_dir = os.path.join(self.temp_directory, "state") | ||
1348 | 49 | os.mkdir(state_dir) | ||
1349 | 50 | lock_file = os.path.join(state_dir, "global.lock") | ||
1350 | 51 | open(lock_file, 'w+') | ||
1351 | 52 | |||
1352 | 53 | def tearDown(self): | ||
1353 | 54 | shutil.rmtree(self.temp_directory) | ||
1354 | 55 | if self.old_path: | ||
1355 | 56 | os.environ['PATH'] = self.old_path | ||
1356 | 57 | |||
1357 | 58 | |||
1358 | 59 | class BaseScriptTests(ScriptTests): | ||
1359 | 60 | def setUp(self): | ||
1360 | 61 | super().setUp() | ||
1361 | 62 | config_path = os.path.join(self.temp_directory, "etc", "config") | ||
1362 | 63 | with open(config_path, "w+") as fd: | ||
1363 | 64 | fd.write("""[global] | ||
1364 | 65 | base_path = %s | ||
1365 | 66 | gpg_key_path = %s | ||
1366 | 67 | public_fqdn = system-image.example.net | ||
1367 | 68 | public_http_port = 880 | ||
1368 | 69 | public_https_port = 8443 | ||
1369 | 70 | """ % (self.temp_directory, os.path.join(os.getcwd(), "tools", "keys"))) | ||
1370 | 71 | self.config_path = config_path | ||
1371 | 72 | |||
1372 | 73 | def test_lock(self): | ||
1373 | 74 | """Make sure locking works.""" | ||
1374 | 75 | script = script_base.Script(self.config_path) | ||
1375 | 76 | script.lock() | ||
1376 | 77 | lock_file = os.path.join(script.conf.state_path, "global.lock") | ||
1377 | 78 | lock_fd = open(lock_file, 'r') | ||
1378 | 79 | with self.assertRaises(IOError): | ||
1379 | 80 | fcntl.lockf(lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) | ||
1380 | 81 | |||
1381 | 82 | def test_lock_already_locked(self): | ||
1382 | 83 | """Make sure locking twice rises an error.""" | ||
1383 | 84 | script = script_base.Script(self.config_path) | ||
1384 | 85 | script2 = script_base.Script(self.config_path) | ||
1385 | 86 | script.lock() | ||
1386 | 87 | with self.assertRaises(script_base.ScriptError): | ||
1387 | 88 | script2.lock() | ||
1388 | 89 | |||
1389 | 90 | @patch('systemimage.tools.sync_mirrors') | ||
1390 | 91 | def test_sync_trees(self, sync_mock): | ||
1391 | 92 | """Check if calling sync_trees synces what we need.""" | ||
1392 | 93 | script = script_base.Script(self.config_path) | ||
1393 | 94 | script.pub = Mock() | ||
1394 | 95 | script.sync_trees() | ||
1395 | 96 | self.assertEqual(script.pub.sync_aliases.call_count, 1) | ||
1396 | 97 | self.assertEqual(sync_mock.call_count, 1) | ||
1397 | 98 | |||
1398 | 99 | @patch('systemimage.tools.sync_mirrors') | ||
1399 | 100 | def test_cleanup(self, sync_mock): | ||
1400 | 101 | """Check if cleanup performs proper steps.""" | ||
1401 | 102 | script = script_base.Script(self.config_path) | ||
1402 | 103 | script.pub = Mock() | ||
1403 | 104 | script.cleanup() | ||
1404 | 105 | self.assertEqual(script.pub.sync_aliases.call_count, 1) | ||
1405 | 106 | self.assertEqual(script.pub.cleanup_tree.call_count, 1) | ||
1406 | 107 | self.assertEqual(sync_mock.call_count, 1) | ||
1407 | 108 | |||
1408 | 109 | def test_run(self): | ||
1409 | 110 | """Check if all required class methods are called.""" | ||
1410 | 111 | script = script_base.Script(self.config_path) | ||
1411 | 112 | script.lock = Mock() | ||
1412 | 113 | script.presetup = Mock() | ||
1413 | 114 | script.setup = Mock() | ||
1414 | 115 | script.execute = Mock() | ||
1415 | 116 | script.cleanup = Mock() | ||
1416 | 117 | script.run([]) | ||
1417 | 118 | self.assertEqual(script.lock.call_count, 1) | ||
1418 | 119 | self.assertEqual(script.presetup.call_count, 1) | ||
1419 | 120 | self.assertEqual(script.setup.call_count, 1) | ||
1420 | 121 | self.assertEqual(script.execute.call_count, 1) | ||
1421 | 122 | self.assertEqual(script.cleanup.call_count, 1) | ||
1422 | 123 | |||
1423 | 124 | def test_error_handled(self): | ||
1424 | 125 | """Check if error on any stage causes the script to fail.""" | ||
1425 | 126 | script = script_base.Script(self.config_path) | ||
1426 | 127 | script.lock = Mock() | ||
1427 | 128 | script.presetup = Mock() | ||
1428 | 129 | script.setup = Mock() | ||
1429 | 130 | script.execute = Mock() | ||
1430 | 131 | script.cleanup = Mock() | ||
1431 | 132 | |||
1432 | 133 | for name in ["lock", "presetup", "setup", "execute", "cleanup"]: | ||
1433 | 134 | setattr(script, name, Mock( | ||
1434 | 135 | side_effect=script_base.ScriptError('foo'))) | ||
1435 | 136 | self.assertFalse(script.run([])) | ||
1436 | 137 | setattr(script, name, Mock()) | ||
1437 | 138 | |||
1438 | 139 | |||
1439 | 140 | class CommonScriptTests(ScriptTests): | ||
1440 | 141 | def setUp(self): | ||
1441 | 142 | super().setUp() | ||
1442 | 143 | config_path = os.path.join(self.temp_directory, "etc", "config") | ||
1443 | 144 | with open(config_path, "w+") as fd: | ||
1444 | 145 | fd.write("""[global] | ||
1445 | 146 | base_path = %s | ||
1446 | 147 | gpg_key_path = %s | ||
1447 | 148 | public_fqdn = system-image.example.net | ||
1448 | 149 | public_http_port = 880 | ||
1449 | 150 | public_https_port = 8443 | ||
1450 | 151 | """ % (self.temp_directory, os.path.join(os.getcwd(), "tools", "keys"))) | ||
1451 | 152 | self.config_path = config_path | ||
1452 | 153 | |||
1453 | 154 | os.mkdir(os.path.join(self.temp_directory, "www")) | ||
1454 | 155 | script = script_base.CommonImageManipulationScript(self.config_path) | ||
1455 | 156 | script.presetup() | ||
1456 | 157 | script.pub.create_channel("test") | ||
1457 | 158 | script.pub.create_device("test", "test") | ||
1458 | 159 | self.script = script | ||
1459 | 160 | self.device = script.pub.get_device("test", "test") | ||
1460 | 161 | |||
1461 | 162 | # An image helper file that can be used for contents | ||
1462 | 163 | self.image_file = os.path.join( | ||
1463 | 164 | self.script.conf.publish_path, "test_file") | ||
1464 | 165 | open(self.image_file, "w+").close() | ||
1465 | 166 | gpg.sign_file(self.script.conf, "image-signing", self.image_file) | ||
1466 | 167 | |||
1467 | 168 | @unittest.skipUnless(HAS_TEST_KEYS, MISSING_KEYS_WARNING) | ||
1468 | 169 | @patch('systemimage.tools.extract_files_and_version') | ||
1469 | 170 | def test_generate_version(self, extract_mock): | ||
1470 | 171 | """Check if script parts for generating version tarballs work""" | ||
1471 | 172 | |||
1472 | 173 | version = 1 | ||
1473 | 174 | tag = "Test-1" | ||
1474 | 175 | |||
1475 | 176 | # Emulate environment of the script run | ||
1476 | 177 | self.script.target_version = version + 1 | ||
1477 | 178 | self.script.channel = "ubuntu-test/test" | ||
1478 | 179 | self.script.device = self.device | ||
1479 | 180 | self.script.new_images = {'full': {'files': ["some-file.tar.xz"]}} | ||
1480 | 181 | |||
1481 | 182 | self.script.generate_version([], version, tag) | ||
1482 | 183 | |||
1483 | 184 | self.assertEqual( | ||
1484 | 185 | self.script.new_images['full']['files'], | ||
1485 | 186 | ["some-file.tar.xz", | ||
1486 | 187 | os.path.realpath(os.path.join( | ||
1487 | 188 | self.device.path, | ||
1488 | 189 | "version-%s.tar.xz" % self.script.target_version))]) | ||
1489 | 190 | |||
1490 | 191 | target_obj = None | ||
1491 | 192 | xz_path = self.script.new_images['full']['files'][-1] | ||
1492 | 193 | unxz_path = os.path.join(self.temp_directory, "temp-unpack.tar") | ||
1493 | 194 | channel_path = "system/etc/system-image/channel.ini" | ||
1494 | 195 | extracted_path = os.path.join(self.temp_directory, channel_path) | ||
1495 | 196 | try: | ||
1496 | 197 | xz_uncompress(xz_path, unxz_path) | ||
1497 | 198 | target_obj = tarfile.open(unxz_path, "r") | ||
1498 | 199 | target_obj.extract( | ||
1499 | 200 | channel_path, self.temp_directory) | ||
1500 | 201 | with open(extracted_path, "r") as f: | ||
1501 | 202 | contents = f.read() | ||
1502 | 203 | self.assertIn( | ||
1503 | 204 | "tag=%s" % tag, contents) | ||
1504 | 205 | self.assertIn( | ||
1505 | 206 | "channel: %s" % self.script.channel, contents) | ||
1506 | 207 | self.assertIn( | ||
1507 | 208 | "build_number: %s" % self.script.target_version, contents) | ||
1508 | 209 | finally: | ||
1509 | 210 | target_obj.close() | ||
1510 | 211 | os.remove(unxz_path) | ||
1511 | 212 | os.remove(extracted_path) | ||
1512 | 213 | |||
1513 | 214 | @unittest.skipUnless(HAS_TEST_KEYS, MISSING_KEYS_WARNING) | ||
1514 | 215 | @patch('systemimage.generators.generate_delta') | ||
1515 | 216 | def test_generate_deltas(self, delta_mock): | ||
1516 | 217 | """Check if we generate proper delta files.""" | ||
1517 | 218 | |||
1518 | 219 | # Emulate environment of the script run | ||
1519 | 220 | self.script.version = 2 | ||
1520 | 221 | self.script.channel = "ubuntu-test/test" | ||
1521 | 222 | self.script.device = self.device | ||
1522 | 223 | self.script.new_images = {'full': {'files': [ | ||
1523 | 224 | os.path.join(self.script.conf.publish_path, "file-source.tar.xz"), | ||
1524 | 225 | os.path.join(self.script.conf.publish_path, "file2-source.tar.xz"), | ||
1525 | 226 | os.path.join(self.script.conf.publish_path, | ||
1526 | 227 | "keyring-source.tar.xz")]}} | ||
1527 | 228 | self.script.delta_base = [ | ||
1528 | 229 | {'version': 1, 'files': [{'path': "file-other.tar.xz"}]}, | ||
1529 | 230 | {'version': 2, 'files': [{'path': "file-other2.tar.xz"}, | ||
1530 | 231 | {'path': "file2-other3.tar.xz"}]}, | ||
1531 | 232 | {'version': 3, 'files': [{'path': "file-other4.tar.xz"}, | ||
1532 | 233 | {'path': "keyring-source.tar.xz"}, | ||
1533 | 234 | {'path': "file2-other5.tar.xz"}]}, | ||
1534 | 235 | {'version': 4, 'files': [{'path': "file-other6.tar.xz"}, | ||
1535 | 236 | {'path': "keyring-other.tar.xz"}, | ||
1536 | 237 | {'path': "file-source.tar.xz"}]}, ] | ||
1537 | 238 | |||
1538 | 239 | expected_lengths = {1: 1, 2: 2, 3: 3, 4: 2} | ||
1539 | 240 | delta_mock.return_value = "delta.tar.xz" | ||
1540 | 241 | |||
1541 | 242 | self.script.generate_deltas() | ||
1542 | 243 | |||
1543 | 244 | for delta in self.script.delta_base: | ||
1544 | 245 | version = delta['version'] | ||
1545 | 246 | self.assertIn('delta_%s' % version, self.script.new_images) | ||
1546 | 247 | delta_files = self.script.new_images['delta_%s' % version]['files'] | ||
1547 | 248 | self.assertEqual( | ||
1548 | 249 | len(delta_files), | ||
1549 | 250 | len(self.script.new_images['full']['files'])) | ||
1550 | 251 | self.assertEqual( | ||
1551 | 252 | len([x for x in delta_files if "delta" in x]), | ||
1552 | 253 | expected_lengths[version]) | ||
1553 | 254 | |||
1554 | 255 | @unittest.skipUnless(HAS_TEST_KEYS, MISSING_KEYS_WARNING) | ||
1555 | 256 | def test_add_images_to_index(self): | ||
1556 | 257 | """Check if a full image upload is properly added to the index.""" | ||
1557 | 258 | |||
1558 | 259 | self.script.target_version = 1 | ||
1559 | 260 | self.script.environment = {'version_detail': [ | ||
1560 | 261 | "ubuntu=1", "device=1", "tag=Foo"]} | ||
1561 | 262 | self.script.new_images = {'full': {'files': [self.image_file]}} | ||
1562 | 263 | self.script.delta_base = [] | ||
1563 | 264 | self.script.device = self.device | ||
1564 | 265 | |||
1565 | 266 | self.script.add_images_to_index() | ||
1566 | 267 | image = tools.extract_image(self.script.device.list_images(), 1) | ||
1567 | 268 | |||
1568 | 269 | self.assertIn('version', image) | ||
1569 | 270 | self.assertEqual(image['version'], 1) | ||
1570 | 271 | self.assertIn('version_detail', image) | ||
1571 | 272 | self.assertEqual(image['version_detail'], "ubuntu=1,device=1,tag=Foo") | ||
1572 | 273 | |||
1573 | 274 | @unittest.skipUnless(HAS_TEST_KEYS, MISSING_KEYS_WARNING) | ||
1574 | 275 | def test_add_images_to_index_with_delta(self): | ||
1575 | 276 | """Check if a full + delta upload is properly added.""" | ||
1576 | 277 | |||
1577 | 278 | self.script.target_version = 1 | ||
1578 | 279 | self.script.environment = {'version_detail': [ | ||
1579 | 280 | "ubuntu=1", "device=1", "tag=Foo"]} | ||
1580 | 281 | self.script.new_images = { | ||
1581 | 282 | 'full': {'files': [self.image_file]}, | ||
1582 | 283 | 'delta_4': {'files': [self.image_file]}} | ||
1583 | 284 | self.script.delta_base = [{ | ||
1584 | 285 | 'version': 4, 'files': [{'path': "other-file.tar.xz"}]}] | ||
1585 | 286 | self.script.device = self.device | ||
1586 | 287 | |||
1587 | 288 | self.script.add_images_to_index() | ||
1588 | 289 | images = self.script.device.list_images() | ||
1589 | 290 | |||
1590 | 291 | self.assertEqual(len(images), 2) | ||
1591 | 292 | |||
1592 | 293 | has_delta = False | ||
1593 | 294 | has_full = False | ||
1594 | 295 | for image in images: | ||
1595 | 296 | self.assertIn('type', image) | ||
1596 | 297 | if image['type'] == "delta": | ||
1597 | 298 | has_delta = True | ||
1598 | 299 | self.assertIn('base', image) | ||
1599 | 300 | self.assertEqual(image['base'], 4) | ||
1600 | 301 | elif image['type'] == "full": | ||
1601 | 302 | has_full = True | ||
1602 | 303 | self.assertIn('version', image) | ||
1603 | 304 | self.assertEqual(image['version'], 1) | ||
1604 | 305 | self.assertIn('version_detail', image) | ||
1605 | 306 | self.assertEqual( | ||
1606 | 307 | image['version_detail'], "ubuntu=1,device=1,tag=Foo") | ||
1607 | 308 | |||
1608 | 309 | self.assertTrue(has_delta) | ||
1609 | 310 | self.assertTrue(has_full) | ||
1610 | 311 | |||
1611 | 312 | |||
1612 | 313 | class ImageScriptBaseTests(ScriptTests): | ||
1613 | 314 | scriptClass = script_base.CommonImageManipulationScript | ||
1614 | 315 | |||
1615 | 316 | def setUp(self): | ||
1616 | 317 | super().setUp() | ||
1617 | 318 | config_path = os.path.join(self.temp_directory, "etc", "config") | ||
1618 | 319 | with open(config_path, "w+") as fd: | ||
1619 | 320 | fd.write("""[global] | ||
1620 | 321 | base_path = %s | ||
1621 | 322 | gpg_key_path = %s | ||
1622 | 323 | public_fqdn = system-image.example.net | ||
1623 | 324 | public_http_port = 880 | ||
1624 | 325 | public_https_port = 8443 | ||
1625 | 326 | """ % (self.temp_directory, os.path.join(os.getcwd(), "tools", "keys"))) | ||
1626 | 327 | self.config_path = config_path | ||
1627 | 328 | |||
1628 | 329 | os.mkdir(os.path.join(self.temp_directory, "www")) | ||
1629 | 330 | script = self.scriptClass(self.config_path) | ||
1630 | 331 | script.presetup() | ||
1631 | 332 | script.pub.create_channel("empty") | ||
1632 | 333 | script.pub.create_channel("from") | ||
1633 | 334 | script.pub.create_channel("to") | ||
1634 | 335 | script.pub.create_device("empty", "test") | ||
1635 | 336 | script.pub.create_device("from", "test") | ||
1636 | 337 | script.pub.create_device("to", "test") | ||
1637 | 338 | |||
1638 | 339 | self.image_file = os.path.join(script.conf.publish_path, "test_file") | ||
1639 | 340 | open(self.image_file, "w+").close() | ||
1640 | 341 | gpg.sign_file(script.conf, "image-signing", self.image_file) | ||
1641 | 342 | |||
1642 | 343 | device = script.pub.get_device("from", "test") | ||
1643 | 344 | device.create_image("full", 1, "abc", [self.image_file]) | ||
1644 | 345 | device.create_image("full", 2, "def", [self.image_file]) | ||
1645 | 346 | device = script.pub.get_device("to", "test") | ||
1646 | 347 | device.create_image("full", 1, "abc", [self.image_file]) | ||
1647 | 348 | |||
1648 | 349 | self.script = script | ||
1649 | 350 | |||
1650 | 351 | |||
1651 | 352 | class CopyImageScriptTests(ImageScriptBaseTests): | ||
1652 | 353 | scriptClass = script_copy_image.CopyImageScript | ||
1653 | 354 | |||
1654 | 355 | def test_copy_setup_correct(self): | ||
1655 | 356 | """Check if setup doesn't fail with good arguments.""" | ||
1656 | 357 | argv = ["copy-image", "from", "to", "test", "1"] | ||
1657 | 358 | self.script.setup(argv) | ||
1658 | 359 | |||
1659 | 360 | def test_copy_setup_invalid_channel(self): | ||
1660 | 361 | """Check if setup fails on invalid channel arguments.""" | ||
1661 | 362 | argv = ["copy-image", "none", "to", "test", "1"] | ||
1662 | 363 | with self.assertRaises(script_base.ScriptError): | ||
1663 | 364 | self.script.setup(argv) | ||
1664 | 365 | argv = ["copy-image", "from", "none", "test", "1"] | ||
1665 | 366 | with self.assertRaises(script_base.ScriptError): | ||
1666 | 367 | self.script.setup(argv) | ||
1667 | 368 | |||
1668 | 369 | self.script.pub.create_channel_alias("alias1", "from") | ||
1669 | 370 | argv = ["copy-image", "alias1", "to", "test", "1"] | ||
1670 | 371 | with self.assertRaises(script_base.ScriptError): | ||
1671 | 372 | self.script.setup(argv) | ||
1672 | 373 | self.script.pub.create_channel_alias("alias2", "to") | ||
1673 | 374 | argv = ["copy-image", "from", "alias2", "test", "1"] | ||
1674 | 375 | with self.assertRaises(script_base.ScriptError): | ||
1675 | 376 | self.script.setup(argv) | ||
1676 | 377 | |||
1677 | 378 | self.script.pub.create_channel_redirect("re1", "from") | ||
1678 | 379 | argv = ["copy-image", "re1", "to", "test", "1"] | ||
1679 | 380 | with self.assertRaises(script_base.ScriptError): | ||
1680 | 381 | self.script.setup(argv) | ||
1681 | 382 | self.script.pub.create_channel_alias("re2", "to") | ||
1682 | 383 | argv = ["copy-image", "from", "re2", "test", "1"] | ||
1683 | 384 | with self.assertRaises(script_base.ScriptError): | ||
1684 | 385 | self.script.setup(argv) | ||
1685 | 386 | |||
1686 | 387 | self.script.pub.create_channel("redirects") | ||
1687 | 388 | self.script.pub.create_per_device_channel_redirect( | ||
1688 | 389 | "test", "redirects", "empty") | ||
1689 | 390 | argv = ["copy-image", "redirects", "to", "test", "1"] | ||
1690 | 391 | with self.assertRaises(script_base.ScriptError): | ||
1691 | 392 | self.script.setup(argv) | ||
1692 | 393 | |||
1693 | 394 | argv = ["copy-image", "from", "redirects", "test", "1"] | ||
1694 | 395 | with self.assertRaises(script_base.ScriptError): | ||
1695 | 396 | self.script.setup(argv) | ||
1696 | 397 | |||
1697 | 398 | def test_copy_setup_invalid_tag(self): | ||
1698 | 399 | """Check if setup fails on invalid tag arguments.""" | ||
1699 | 400 | argv = ["copy-image", "from", "to", "test", "1", "--tag=\"tag,tag\""] | ||
1700 | 401 | with self.assertRaises(script_base.ScriptError): | ||
1701 | 402 | self.script.setup(argv) | ||
1702 | 403 | |||
1703 | 404 | def test_copy_setup_invalid_device(self): | ||
1704 | 405 | """Check if setup fails on invalid device selection.""" | ||
1705 | 406 | argv = ["copy-image", "from", "to", "none", "1"] | ||
1706 | 407 | with self.assertRaises(script_base.ScriptError): | ||
1707 | 408 | self.script.setup(argv) | ||
1708 | 409 | |||
1709 | 410 | self.script.pub.create_device("to", "to_only") | ||
1710 | 411 | argv = ["copy-image", "from", "to", "to_only", "1"] | ||
1711 | 412 | with self.assertRaises(script_base.ScriptError): | ||
1712 | 413 | self.script.setup(argv) | ||
1713 | 414 | |||
1714 | 415 | self.script.pub.create_device("from", "from_only") | ||
1715 | 416 | device = self.script.pub.get_device("from", "from_only") | ||
1716 | 417 | device.create_image("full", 1, "abc", [self.image_file]) | ||
1717 | 418 | argv = ["copy-image", "from", "to", "from_only", "1"] | ||
1718 | 419 | with self.assertRaises(script_base.ScriptError): | ||
1719 | 420 | self.script.setup(argv) | ||
1720 | 421 | |||
1721 | 422 | def test_assign_new_version(self): | ||
1722 | 423 | """Check if new version assignment logic is correct.""" | ||
1723 | 424 | self.script.device = self.script.pub.get_device("to", "test") | ||
1724 | 425 | |||
1725 | 426 | self.script.assign_new_version(1, False) | ||
1726 | 427 | self.assertEqual(self.script.target_version, 2) | ||
1727 | 428 | self.script.assign_new_version(4, False) | ||
1728 | 429 | self.assertEqual(self.script.target_version, 2) | ||
1729 | 430 | self.script.assign_new_version(4, True) | ||
1730 | 431 | self.assertEqual(self.script.target_version, 4) | ||
1731 | 432 | with self.assertRaises(script_base.ScriptError): | ||
1732 | 433 | self.script.assign_new_version(1, True) | ||
1733 | 434 | |||
1734 | 435 | |||
1735 | 436 | class TagImageScriptTests(ImageScriptBaseTests): | ||
1736 | 437 | scriptClass = script_tag_image.TagImageScript | ||
1737 | 438 | |||
1738 | 439 | def test_tag_setup_correct(self): | ||
1739 | 440 | """Check if setup doesn't fail with good arguments.""" | ||
1740 | 441 | argv = ["tag-image", "to", "test", "1", "TAG1"] | ||
1741 | 442 | self.script.setup(argv) | ||
1742 | 443 | |||
1743 | 444 | def test_tag_setup_invalid_channel(self): | ||
1744 | 445 | """Check if setup fails on invalid channel arguments.""" | ||
1745 | 446 | argv = ["tag-image", "none", "test", "1", "TAG1"] | ||
1746 | 447 | with self.assertRaises(script_base.ScriptError): | ||
1747 | 448 | self.script.setup(argv) | ||
1748 | 449 | |||
1749 | 450 | self.script.pub.create_channel_alias("alias", "from") | ||
1750 | 451 | argv = ["tag-image", "alias", "test", "1", "TAG1"] | ||
1751 | 452 | with self.assertRaises(script_base.ScriptError): | ||
1752 | 453 | self.script.setup(argv) | ||
1753 | 454 | |||
1754 | 455 | self.script.pub.create_channel_redirect("re", "from") | ||
1755 | 456 | argv = ["tag-image", "re", "test", "1", "TAG1"] | ||
1756 | 457 | with self.assertRaises(script_base.ScriptError): | ||
1757 | 458 | self.script.setup(argv) | ||
1758 | 459 | |||
1759 | 460 | self.script.pub.create_channel("redirects") | ||
1760 | 461 | self.script.pub.create_per_device_channel_redirect( | ||
1761 | 462 | "test", "redirects", "empty") | ||
1762 | 463 | argv = ["tag-image", "redirects", "test", "1", "TAG1"] | ||
1763 | 464 | with self.assertRaises(script_base.ScriptError): | ||
1764 | 465 | self.script.setup(argv) | ||
1765 | 466 | |||
1766 | 467 | def test_tag_setup_invalid_tag(self): | ||
1767 | 468 | """Check if setup fails on invalid tag arguments.""" | ||
1768 | 469 | argv = ["tag-image", "to", "test", "1", "TAG1,TAG2"] | ||
1769 | 470 | with self.assertRaises(script_base.ScriptError): | ||
1770 | 471 | self.script.setup(argv) | ||
1771 | 472 | |||
1772 | 473 | def test_tag_setup_invalid_device(self): | ||
1773 | 474 | """Check if setup fails on invalid device selection.""" | ||
1774 | 475 | argv = ["tag-image", "to", "none", "1", "TAG1"] | ||
1775 | 476 | with self.assertRaises(script_base.ScriptError): | ||
1776 | 477 | self.script.setup(argv) | ||
1777 | 478 | |||
1778 | 479 | def test_tag_execute_invalid_image(self): | ||
1779 | 480 | """Make sure execute fails on invalid image scenarios.""" | ||
1780 | 481 | argv = ["tag-image", "empty", "test", "1", "TAG1"] | ||
1781 | 482 | self.script.setup(argv) | ||
1782 | 483 | with self.assertRaises(script_base.ScriptError): | ||
1783 | 484 | self.script.execute() | ||
1784 | 485 | |||
1785 | 486 | argv = ["tag-image", "from", "test", "1", "TAG1"] | ||
1786 | 487 | self.script.setup(argv) | ||
1787 | 488 | with self.assertRaises(script_base.ScriptError): | ||
1788 | 489 | self.script.execute() | ||
1789 | 490 | |||
1790 | 491 | argv = ["tag-image", "from", "test", "3", "TAG1"] | ||
1791 | 492 | self.script.setup(argv) | ||
1792 | 493 | with self.assertRaises(script_base.ScriptError): | ||
1793 | 494 | self.script.execute() | ||
1794 | 495 | |||
1795 | 496 | def test_tag_execute(self): | ||
1796 | 497 | """Check that we can correctly tag an image.""" | ||
1797 | 498 | device = self.script.pub.get_device("from", "test") | ||
1798 | 499 | device.set_phased_percentage(2, 60) | ||
1799 | 500 | |||
1800 | 501 | argv = ["tag-image", "from", "test", "2", "TAG1"] | ||
1801 | 502 | self.script.setup(argv) | ||
1802 | 503 | self.script.execute() | ||
1803 | 504 | |||
1804 | 505 | image = tools.extract_image(self.script.device.list_images(), 3) | ||
1805 | 506 | self.assertTrue(image) | ||
1806 | 507 | self.assertIn("version_detail", image) | ||
1807 | 508 | self.assertIn("tag=TAG1", image['version_detail']) | ||
1808 | 509 | self.assertEqual(device.get_phased_percentage(3), 60) | ||
1809 | 510 | |||
1810 | 511 | previous = tools.extract_image(self.script.device.list_images(), 2) | ||
1811 | 512 | self.assertIn("files", image) | ||
1812 | 513 | for f in previous['files']: | ||
1813 | 514 | self.assertIn("path", f) | ||
1814 | 515 | if "version-" not in f['path']: | ||
1815 | 516 | self.assertIn(f, image['files']) | ||
1816 | 517 | |||
1817 | 518 | |||
1818 | 519 | class SetPhasedPercentageScriptTests(ImageScriptBaseTests): | ||
1819 | 520 | scriptClass = script_set_phased_percentage.SetPhasedPercentageScript | ||
1820 | 521 | |||
1821 | 522 | def test_phased_setup_correct(self): | ||
1822 | 523 | """Check if setup doesn't fail with good arguments.""" | ||
1823 | 524 | argv = ["set-phased-percentage", "to", "test", "1", "50"] | ||
1824 | 525 | self.script.setup(argv) | ||
1825 | 526 | |||
1826 | 527 | def test_phased_setup_invalid_percentage(self): | ||
1827 | 528 | """Check if setup fails with an invalid phase percentage value.""" | ||
1828 | 529 | argv = ["set-phased-percentage", "to", "test", "1", "foo"] | ||
1829 | 530 | with self.assertRaises(SystemExit): | ||
1830 | 531 | self.script.setup(argv) | ||
1831 | 532 | |||
1832 | 533 | argv = ["set-phased-percentage", "to", "test", "1", "9000"] | ||
1833 | 534 | with self.assertRaises(script_base.ScriptError): | ||
1834 | 535 | self.script.setup(argv) | ||
1835 | 536 | |||
1836 | 537 | argv = ["set-phased-percentage", "to", "test", "1", "-10"] | ||
1837 | 538 | with self.assertRaises(Exception): | ||
1838 | 539 | self.script.setup(argv) | ||
1839 | 540 | |||
1840 | 541 | def test_phased_setup_invalid_channel(self): | ||
1841 | 542 | """Check if setup fails on invalid channel arguments.""" | ||
1842 | 543 | argv = ["set-phased-percentage", "none", "test", "1", "50"] | ||
1843 | 544 | with self.assertRaises(script_base.ScriptError): | ||
1844 | 545 | self.script.setup(argv) | ||
1845 | 546 | |||
1846 | 547 | self.script.pub.create_channel_alias("alias", "from") | ||
1847 | 548 | argv = ["set-phased-percentage", "alias", "test", "1", "50"] | ||
1848 | 549 | with self.assertRaises(script_base.ScriptError): | ||
1849 | 550 | self.script.setup(argv) | ||
1850 | 551 | |||
1851 | 552 | self.script.pub.create_channel_redirect("re", "from") | ||
1852 | 553 | argv = ["set-phased-percentage", "re", "test", "1", "50"] | ||
1853 | 554 | with self.assertRaises(script_base.ScriptError): | ||
1854 | 555 | self.script.setup(argv) | ||
1855 | 556 | |||
1856 | 557 | self.script.pub.create_channel("redirects") | ||
1857 | 558 | self.script.pub.create_per_device_channel_redirect( | ||
1858 | 559 | "test", "redirects", "empty") | ||
1859 | 560 | argv = ["set-phased-percentage", "redirects", "test", "1", "50"] | ||
1860 | 561 | with self.assertRaises(script_base.ScriptError): | ||
1861 | 562 | self.script.setup(argv) | ||
1862 | 563 | |||
1863 | 564 | def test_phased_execute(self): | ||
1864 | 565 | """Check if we can change the percentage forwards and backwards.""" | ||
1865 | 566 | device = self.script.pub.get_device("to", "test") | ||
1866 | 567 | |||
1867 | 568 | argv = ["set-phased-percentage", "to", "test", "1", "50"] | ||
1868 | 569 | self.script.setup(argv) | ||
1869 | 570 | self.script.execute() | ||
1870 | 571 | self.assertEqual(device.get_phased_percentage(1), 50) | ||
1871 | 572 | |||
1872 | 573 | argv = ["set-phased-percentage", "to", "test", "1", "60"] | ||
1873 | 574 | self.script.setup(argv) | ||
1874 | 575 | self.script.execute() | ||
1875 | 576 | self.assertEqual(device.get_phased_percentage(1), 60) | ||
1876 | 577 | |||
1877 | 578 | argv = ["set-phased-percentage", "to", "test", "1", "15"] | ||
1878 | 579 | self.script.setup(argv) | ||
1879 | 580 | self.script.execute() | ||
1880 | 581 | self.assertEqual(device.get_phased_percentage(1), 15) | ||
1881 | 582 | |||
1882 | 583 | def test_phased_execute_invalid_image(self): | ||
1883 | 584 | """Check if we fail on trying to phase an older image.""" | ||
1884 | 585 | device = self.script.pub.get_device("to", "test") | ||
1885 | 586 | |||
1886 | 587 | argv = ["set-phased-percentage", "from", "test", "1", "50"] | ||
1887 | 588 | with self.assertRaises(Exception): | ||
1888 | 589 | self.script.setup(argv) | ||
1889 | 590 | self.script.execute() | ||
1890 | 0 | 591 | ||
1891 | === modified file 'lib/systemimage/tests/test_tools.py' | |||
1892 | --- lib/systemimage/tests/test_tools.py 2016-03-16 18:52:39 +0000 | |||
1893 | +++ lib/systemimage/tests/test_tools.py 2016-07-20 12:34:06 +0000 | |||
1894 | @@ -432,6 +432,7 @@ | |||
1895 | 432 | 432 | ||
1896 | 433 | def test_set_tag_on_version_detail(self): | 433 | def test_set_tag_on_version_detail(self): |
1897 | 434 | """Set a basic tag.""" | 434 | """Set a basic tag.""" |
1898 | 435 | |||
1899 | 435 | version_detail_list = [ | 436 | version_detail_list = [ |
1900 | 436 | "device=20150821-736d127", | 437 | "device=20150821-736d127", |
1901 | 437 | "custom=20150925-901-35-40-vivid", | 438 | "custom=20150925-901-35-40-vivid", |
1902 | @@ -444,6 +445,7 @@ | |||
1903 | 444 | 445 | ||
1904 | 445 | def test_set_tag_on_version_detail_rewrite(self): | 446 | def test_set_tag_on_version_detail_rewrite(self): |
1905 | 446 | """Make sure tags can be rewritten.""" | 447 | """Make sure tags can be rewritten.""" |
1906 | 448 | |||
1907 | 447 | version_detail_list = [ | 449 | version_detail_list = [ |
1908 | 448 | "device=20150821-736d127", | 450 | "device=20150821-736d127", |
1909 | 449 | "custom=20150925-901-35-40-vivid", | 451 | "custom=20150925-901-35-40-vivid", |
1910 | @@ -458,6 +460,7 @@ | |||
1911 | 458 | 460 | ||
1912 | 459 | def test_set_tag_on_version_detail_clear(self): | 461 | def test_set_tag_on_version_detail_clear(self): |
1913 | 460 | """Clear the tag.""" | 462 | """Clear the tag.""" |
1914 | 463 | |||
1915 | 461 | version_detail_list = [ | 464 | version_detail_list = [ |
1916 | 462 | "device=20150821-736d127", | 465 | "device=20150821-736d127", |
1917 | 463 | "custom=20150925-901-35-40-vivid", | 466 | "custom=20150925-901-35-40-vivid", |
1918 | @@ -470,6 +473,7 @@ | |||
1919 | 470 | 473 | ||
1920 | 471 | def test_extract_files_and_version(self): | 474 | def test_extract_files_and_version(self): |
1921 | 472 | """Check if version_detail is correctly extracted""" | 475 | """Check if version_detail is correctly extracted""" |
1922 | 476 | |||
1923 | 473 | os.mkdir(self.config.publish_path) | 477 | os.mkdir(self.config.publish_path) |
1924 | 474 | 478 | ||
1925 | 475 | version = 12 | 479 | version = 12 |
1926 | @@ -501,6 +505,7 @@ | |||
1927 | 501 | @unittest.skip("Current deltabase handling is broken") | 505 | @unittest.skip("Current deltabase handling is broken") |
1928 | 502 | def test_get_required_deltas(self): | 506 | def test_get_required_deltas(self): |
1929 | 503 | """Check if a proper list of valid deltabases is found.""" | 507 | """Check if a proper list of valid deltabases is found.""" |
1930 | 508 | |||
1931 | 504 | config_path = os.path.join(self.temp_directory, "etc", "config") | 509 | config_path = os.path.join(self.temp_directory, "etc", "config") |
1932 | 505 | with open(config_path, "w+") as fd: | 510 | with open(config_path, "w+") as fd: |
1933 | 506 | fd.write("""[global] | 511 | fd.write("""[global] |
1934 | @@ -550,6 +555,42 @@ | |||
1935 | 550 | six.assertCountEqual( | 555 | six.assertCountEqual( |
1936 | 551 | self, [base_image1, base_image2], delta_base) | 556 | self, [base_image1, base_image2], delta_base) |
1937 | 552 | 557 | ||
1938 | 558 | def test_extract_image(self): | ||
1939 | 559 | """Extracting single images from devices.""" | ||
1940 | 560 | |||
1941 | 561 | config_path = os.path.join(self.temp_directory, "etc", "config") | ||
1942 | 562 | with open(config_path, "w+") as fd: | ||
1943 | 563 | fd.write("""[global] | ||
1944 | 564 | base_path = %s | ||
1945 | 565 | gpg_key_path = %s | ||
1946 | 566 | public_fqdn = system-image.example.net | ||
1947 | 567 | public_http_port = 880 | ||
1948 | 568 | public_https_port = 8443 | ||
1949 | 569 | """ % (self.temp_directory, os.path.join(os.getcwd(), "tools", "keys"))) | ||
1950 | 570 | test_config = config.Config(config_path) | ||
1951 | 571 | os.makedirs(test_config.publish_path) | ||
1952 | 572 | |||
1953 | 573 | test_tree = tree.Tree(test_config) | ||
1954 | 574 | test_tree.create_channel("testing") | ||
1955 | 575 | test_tree.create_device("testing", "test") | ||
1956 | 576 | |||
1957 | 577 | image_file = os.path.join(self.config.publish_path, "test_file") | ||
1958 | 578 | open(image_file, "w+").close() | ||
1959 | 579 | gpg.sign_file(test_config, "image-signing", image_file) | ||
1960 | 580 | |||
1961 | 581 | device = test_tree.get_device("testing", "test") | ||
1962 | 582 | device.create_image("full", 1, "abc", [image_file]) | ||
1963 | 583 | device.create_image("delta", 2, "abc", [image_file], base=1) | ||
1964 | 584 | device.create_image("full", 4, "abc", [image_file]) | ||
1965 | 585 | |||
1966 | 586 | image = tools.extract_image(device.list_images(), 1) | ||
1967 | 587 | self.assertIsNotNone(image) | ||
1968 | 588 | self.assertEqual(image['version'], 1) | ||
1969 | 589 | self.assertIsNone(tools.extract_image(device.list_images(), 2)) | ||
1970 | 590 | image = tools.extract_image(device.list_images(), 4) | ||
1971 | 591 | self.assertIsNotNone(image) | ||
1972 | 592 | self.assertEqual(image['version'], 4) | ||
1973 | 593 | |||
1974 | 553 | def test_guess_file_compression(self): | 594 | def test_guess_file_compression(self): |
1975 | 554 | """Check if we can correctly guess compression algorithms.""" | 595 | """Check if we can correctly guess compression algorithms.""" |
1976 | 555 | test_string = "test-string" | 596 | test_string = "test-string" |
1977 | 556 | 597 | ||
1978 | === modified file 'lib/systemimage/tests/test_tree.py' | |||
1979 | --- lib/systemimage/tests/test_tree.py 2016-06-24 19:10:46 +0000 | |||
1980 | +++ lib/systemimage/tests/test_tree.py 2016-07-20 12:34:06 +0000 | |||
1981 | @@ -648,7 +648,7 @@ | |||
1982 | 648 | 648 | ||
1983 | 649 | @unittest.skipUnless(HAS_TEST_KEYS, MISSING_KEYS_WARNING) | 649 | @unittest.skipUnless(HAS_TEST_KEYS, MISSING_KEYS_WARNING) |
1984 | 650 | def test_index(self): | 650 | def test_index(self): |
1986 | 651 | device = tree.Device(self.config, self.temp_directory) | 651 | device = tree.Device("test", self.config, self.temp_directory) |
1987 | 652 | 652 | ||
1988 | 653 | # Check without files | 653 | # Check without files |
1989 | 654 | self.assertRaises(Exception, device.create_image, "full", 1234, | 654 | self.assertRaises(Exception, device.create_image, "full", 1234, |
1990 | 655 | 655 | ||
1991 | === modified file 'lib/systemimage/tools.py' | |||
1992 | --- lib/systemimage/tools.py 2016-03-16 18:52:39 +0000 | |||
1993 | +++ lib/systemimage/tools.py 2016-07-20 12:34:06 +0000 | |||
1994 | @@ -603,6 +603,21 @@ | |||
1995 | 603 | return version_detail | 603 | return version_detail |
1996 | 604 | 604 | ||
1997 | 605 | 605 | ||
1998 | 606 | def extract_image(images, version): | ||
1999 | 607 | """ | ||
2000 | 608 | Extracts the image file from an device image list for the selected | ||
2001 | 609 | version number. | ||
2002 | 610 | """ | ||
2003 | 611 | |||
2004 | 612 | images = [image for image in images | ||
2005 | 613 | if image['type'] == "full" and | ||
2006 | 614 | image['version'] == version] | ||
2007 | 615 | if not images: | ||
2008 | 616 | return None | ||
2009 | 617 | |||
2010 | 618 | return images[0] | ||
2011 | 619 | |||
2012 | 620 | |||
2013 | 606 | def set_tag_on_version_detail(version_detail_list, tag): | 621 | def set_tag_on_version_detail(version_detail_list, tag): |
2014 | 607 | """ | 622 | """ |
2015 | 608 | Append a tag to the version_detail array. | 623 | Append a tag to the version_detail array. |
2016 | 609 | 624 | ||
2017 | === modified file 'lib/systemimage/tree.py' | |||
2018 | --- lib/systemimage/tree.py 2016-06-24 19:10:46 +0000 | |||
2019 | +++ lib/systemimage/tree.py 2016-07-20 12:34:06 +0000 | |||
2020 | @@ -429,8 +429,8 @@ | |||
2021 | 429 | device_path = os.path.dirname(channels[channel_name]['devices'] | 429 | device_path = os.path.dirname(channels[channel_name]['devices'] |
2022 | 430 | [device_name]['index']) | 430 | [device_name]['index']) |
2023 | 431 | 431 | ||
2026 | 432 | return Device(self.config, os.path.normpath("%s/%s" % (self.path, | 432 | return Device(device_name, self.config, |
2027 | 433 | device_path))) | 433 | os.path.normpath("%s/%s" % (self.path, device_path))) |
2028 | 434 | 434 | ||
2029 | 435 | def hide_channel(self, channel_name): | 435 | def hide_channel(self, channel_name): |
2030 | 436 | """ | 436 | """ |
2031 | @@ -874,7 +874,8 @@ | |||
2032 | 874 | 874 | ||
2033 | 875 | 875 | ||
2034 | 876 | class Device: | 876 | class Device: |
2036 | 877 | def __init__(self, config, path): | 877 | def __init__(self, name, config, path): |
2037 | 878 | self.name = name | ||
2038 | 878 | self.config = config | 879 | self.config = config |
2039 | 879 | self.pub_path = self.config.publish_path | 880 | self.pub_path = self.config.publish_path |
2040 | 880 | self.path = path | 881 | self.path = path |
I wouldn't land this right now even without writing more tests, but I'd like to gather some feedback in the meantime when I'm busy with other duties.