Merge ~ltrager/maas-images:version_remove_copy into maas-images:master
- Git
- lp:~ltrager/maas-images
- version_remove_copy
- Merge into master
Proposed by
Lee Trager
Status: | Merged |
---|---|
Merge reported by: | Lee Trager |
Merged at revision: | 0c3ba47197755743bd936360f60c07c43ce41624 |
Proposed branch: | ~ltrager/maas-images:version_remove_copy |
Merge into: | maas-images:master |
Diff against target: |
268 lines (+184/-9) 5 files modified
doc/copying-product-version.txt (+36/-0) doc/removing-product-version.txt (+51/-0) meph2/commands/flags.py (+23/-4) meph2/commands/meph2_util.py (+65/-0) meph2/util.py (+9/-5) |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Newell Jensen (community) | Approve | ||
Review via email: mp+374577@code.launchpad.net |
Commit message
Add the ability to remove or copy versions from a stream.
Description of the change
Example - remove a version from all Bionic images
./maas-
Example - copy a version for all amd64 Bionic images
/maas-images/
To post a comment you must log in.
- 0c3ba47... by Lee Trager
-
Add documentation
Revision history for this message
Robert C Jennings (rcj) wrote : | # |
Lee, the docs look good, thank you. Should we copy an image forward from $serial to $serial.1 to test this out? Or at least, how did you test this?
Revision history for this message
Lee Trager (ltrager) wrote : | # |
For development and testing I created a local mirror of the images and ran remove-version and copy-version on it. I verified with diff that the modifications happened as expected.
Preview Diff
[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1 | diff --git a/doc/copying-product-version.txt b/doc/copying-product-version.txt | |||
2 | 0 | new file mode 100644 | 0 | new file mode 100644 |
3 | index 0000000..6008d0b | |||
4 | --- /dev/null | |||
5 | +++ b/doc/copying-product-version.txt | |||
6 | @@ -0,0 +1,36 @@ | |||
7 | 1 | In the case that a bad image is published it may be quicker to copy a working | ||
8 | 2 | version to a new version than wait for a fixed version to appear at | ||
9 | 3 | cloud-images.ubuntu.com. This process is safer than just removing a broken | ||
10 | 4 | version as it mimics a new version being added to the stream. This guarantees | ||
11 | 5 | all clients, not just MAAS, will get the fixed image. | ||
12 | 6 | |||
13 | 7 | It is also suggested to create a backup of the stream metadata in | ||
14 | 8 | maas-v3-images/streams/v1 so you can use diff to verify only versions you wish | ||
15 | 9 | to remove were removed. | ||
16 | 10 | |||
17 | 11 | Basic usage: | ||
18 | 12 | meph2-util copy-version data_d from_version to_version | ||
19 | 13 | |||
20 | 14 | data_d - The path to the directory containing the stream you wish to modify. | ||
21 | 15 | from_version - The version you wish to copy from | ||
22 | 16 | to_version - The version you wish to copy to | ||
23 | 17 | |||
24 | 18 | Example - Copy 20191004 to 20191022 on all products | ||
25 | 19 | meph2-util copy-version /path/to/stream 20191004 20191022 | ||
26 | 20 | |||
27 | 21 | Filters: | ||
28 | 22 | You may also add filters to the copy-version command. Only products matching | ||
29 | 23 | those filters will have the specified version copied. Filters may be any field | ||
30 | 24 | described in the product. | ||
31 | 25 | |||
32 | 26 | Example - Copy the version 20191004 to 20191022 on all AMD64 Bionic and Xenial | ||
33 | 27 | products. | ||
34 | 28 | meph2-util copy-version /path/to/stream 20191004 20191022 \ | ||
35 | 29 | 'release~(bionic|xenial)' arch=amd64 | ||
36 | 30 | |||
37 | 31 | Optional Arguments: | ||
38 | 32 | -n, --dry-run - Only show what will be copied, do not modify the stream. | ||
39 | 33 | -u, --no-sign - Do not sign the stream when done. A stream can be signed later | ||
40 | 34 | with the meph2-util sign command. | ||
41 | 35 | --keyring - Specify the keyring to use when verifying the stream. Defaults | ||
42 | 36 | to /usr/share/keyrings/ubuntu-cloudimage-keyring.gpg | ||
43 | diff --git a/doc/removing-product-version.txt b/doc/removing-product-version.txt | |||
44 | 0 | new file mode 100644 | 37 | new file mode 100644 |
45 | index 0000000..df8dfe6 | |||
46 | --- /dev/null | |||
47 | +++ b/doc/removing-product-version.txt | |||
48 | @@ -0,0 +1,51 @@ | |||
49 | 1 | In the case that a bad image is published it may be quicker to remove it from | ||
50 | 2 | the stream than wait for a fixed version to appear at cloud-images.ubuntu.com. | ||
51 | 3 | MAAS will import the latest version available from the stream, if a version is | ||
52 | 4 | removed the previous version will be automatically retrieved. | ||
53 | 5 | |||
54 | 6 | Before beginning make sure any import process which imports from | ||
55 | 7 | cloud-images.ubuntu.com is disabled so the image isn't recreated. | ||
56 | 8 | |||
57 | 9 | It is also suggested to create a backup of the stream metadata in | ||
58 | 10 | maas-v3-images/streams/v1 so you can use diff to verify only versions you wish | ||
59 | 11 | to remove were removed. | ||
60 | 12 | |||
61 | 13 | Basic usage: | ||
62 | 14 | meph2-util remove-version data_d version | ||
63 | 15 | |||
64 | 16 | data_d - The path to the directory containing the stream you wish to modify. | ||
65 | 17 | version - The version of the product you wish to remove. | ||
66 | 18 | |||
67 | 19 | Example - Remove the version 20191021 from all products | ||
68 | 20 | meph2-util remove-version /path/to/stream 20191021 | ||
69 | 21 | |||
70 | 22 | Filters: | ||
71 | 23 | You may also add filters to the remove-version command. Only products matching | ||
72 | 24 | those filters will have the specified version removed. Filters may be any field | ||
73 | 25 | described in the product. | ||
74 | 26 | |||
75 | 27 | Example - Remove the version 20191021 from all AMD64 Bionic and Xenial | ||
76 | 28 | products. | ||
77 | 29 | meph2-util remove-version /path/to/stream 20191021 \ | ||
78 | 30 | 'release~(bionic|xenial)' arch=amd64 | ||
79 | 31 | |||
80 | 32 | Optional Arguments: | ||
81 | 33 | -n, --dry-run - Only show what will be removed, do not modify the stream. | ||
82 | 34 | -u, --no-sign - Do not sign the stream when done. A stream can be signed later | ||
83 | 35 | with the meph2-util sign command. | ||
84 | 36 | --keyring - Specify the keyring to use when verifying the stream. Defaults | ||
85 | 37 | to /usr/share/keyrings/ubuntu-cloudimage-keyring.gpg | ||
86 | 38 | |||
87 | 39 | Clean up: | ||
88 | 40 | meph2-util remove-version only modifies the stream metadata. The image files | ||
89 | 41 | themselves are left on the filesystem. To automatically remove them run the | ||
90 | 42 | following commands | ||
91 | 43 | |||
92 | 44 | Find all orphaned files and write the list to /tmp/orphans: | ||
93 | 45 | meph2-util find-orphans /tmp/orphans /path/to/stream | ||
94 | 46 | |||
95 | 47 | Delete all orphaned files: | ||
96 | 48 | meph2-util reap-orphans /tmp/orphans /path/to/stream --older 0 | ||
97 | 49 | |||
98 | 50 | Remove orphan file: | ||
99 | 51 | rm /tmp/orphans | ||
100 | diff --git a/meph2/commands/flags.py b/meph2/commands/flags.py | |||
101 | index e2cffa5..d0c3d61 100644 | |||
102 | --- a/meph2/commands/flags.py | |||
103 | +++ b/meph2/commands/flags.py | |||
104 | @@ -24,6 +24,8 @@ COMMON_FLAGS = { | |||
105 | 24 | 'keyring': (('--keyring',), | 24 | 'keyring': (('--keyring',), |
106 | 25 | {'help': 'gpg keyring to check sjson', | 25 | {'help': 'gpg keyring to check sjson', |
107 | 26 | 'default': DEF_KEYRING}), | 26 | 'default': DEF_KEYRING}), |
108 | 27 | 'filters': ('filters', {'nargs': '*', 'default': []}), | ||
109 | 28 | 'version': ('version', {'help': 'the version_id to promote.'}), | ||
110 | 27 | } | 29 | } |
111 | 28 | 30 | ||
112 | 29 | SUBCOMMANDS = { | 31 | SUBCOMMANDS = { |
113 | @@ -33,7 +35,7 @@ SUBCOMMANDS = { | |||
114 | 33 | COMMON_FLAGS['dry-run'], COMMON_FLAGS['no-sign'], | 35 | COMMON_FLAGS['dry-run'], COMMON_FLAGS['no-sign'], |
115 | 34 | COMMON_FLAGS['keyring'], | 36 | COMMON_FLAGS['keyring'], |
116 | 35 | COMMON_FLAGS['src'], COMMON_FLAGS['target'], | 37 | COMMON_FLAGS['src'], COMMON_FLAGS['target'], |
118 | 36 | ('filters', {'nargs': '*', 'default': []}), | 38 | COMMON_FLAGS['filters'], |
119 | 37 | ] | 39 | ] |
120 | 38 | }, | 40 | }, |
121 | 39 | 'import': { | 41 | 'import': { |
122 | @@ -68,8 +70,7 @@ SUBCOMMANDS = { | |||
123 | 68 | {'help': 'do not copy files, only metadata [TEST_ONLY]', | 70 | {'help': 'do not copy files, only metadata [TEST_ONLY]', |
124 | 69 | 'action': 'store_true', 'default': False}), | 71 | 'action': 'store_true', 'default': False}), |
125 | 70 | COMMON_FLAGS['src'], COMMON_FLAGS['target'], | 72 | COMMON_FLAGS['src'], COMMON_FLAGS['target'], |
128 | 71 | ('version', {'help': 'the version_id to promote.'}), | 73 | COMMON_FLAGS['version'], COMMON_FLAGS['filters'], |
127 | 72 | ('filters', {'nargs': '+', 'default': []}), | ||
129 | 73 | ] | 74 | ] |
130 | 74 | }, | 75 | }, |
131 | 75 | 'clean-md': { | 76 | 'clean-md': { |
132 | @@ -78,7 +79,7 @@ SUBCOMMANDS = { | |||
133 | 78 | COMMON_FLAGS['dry-run'], COMMON_FLAGS['no-sign'], | 79 | COMMON_FLAGS['dry-run'], COMMON_FLAGS['no-sign'], |
134 | 79 | COMMON_FLAGS['keyring'], | 80 | COMMON_FLAGS['keyring'], |
135 | 80 | ('max', {'type': int}), ('target', {}), | 81 | ('max', {'type': int}), ('target', {}), |
137 | 81 | ('filters', {'nargs': '*', 'default': []}), | 82 | COMMON_FLAGS['filters'], |
138 | 82 | ] | 83 | ] |
139 | 83 | }, | 84 | }, |
140 | 84 | 'find-orphans': { | 85 | 'find-orphans': { |
141 | @@ -106,6 +107,24 @@ SUBCOMMANDS = { | |||
142 | 106 | COMMON_FLAGS['data_d'], COMMON_FLAGS['no-sign'], | 107 | COMMON_FLAGS['data_d'], COMMON_FLAGS['no-sign'], |
143 | 107 | ], | 108 | ], |
144 | 108 | }, | 109 | }, |
145 | 110 | 'remove-version': { | ||
146 | 111 | 'help': 'Remove a version from a product', | ||
147 | 112 | 'opts': [ | ||
148 | 113 | COMMON_FLAGS['dry-run'], COMMON_FLAGS['no-sign'], | ||
149 | 114 | COMMON_FLAGS['keyring'], COMMON_FLAGS['data_d'], | ||
150 | 115 | COMMON_FLAGS['version'], COMMON_FLAGS['filters'], | ||
151 | 116 | ], | ||
152 | 117 | }, | ||
153 | 118 | 'copy-version': { | ||
154 | 119 | 'help': 'Copy a version of a product to a new version', | ||
155 | 120 | 'opts': [ | ||
156 | 121 | COMMON_FLAGS['dry-run'], COMMON_FLAGS['no-sign'], | ||
157 | 122 | COMMON_FLAGS['keyring'], COMMON_FLAGS['data_d'], | ||
158 | 123 | ('from_version', {'help': 'the version_id to copy from.'}), | ||
159 | 124 | ('to_version', {'help': 'the version_id to copy to.'}), | ||
160 | 125 | COMMON_FLAGS['filters'] | ||
161 | 126 | ], | ||
162 | 127 | }, | ||
163 | 109 | } | 128 | } |
164 | 110 | 129 | ||
165 | 111 | # vi: ts=4 expandtab syntax=python | 130 | # vi: ts=4 expandtab syntax=python |
166 | diff --git a/meph2/commands/meph2_util.py b/meph2/commands/meph2_util.py | |||
167 | index 4109634..0b114dc 100755 | |||
168 | --- a/meph2/commands/meph2_util.py | |||
169 | +++ b/meph2/commands/meph2_util.py | |||
170 | @@ -403,6 +403,71 @@ def main_sign(args): | |||
171 | 403 | return 0 | 403 | return 0 |
172 | 404 | 404 | ||
173 | 405 | 405 | ||
174 | 406 | def main_remove_version(args): | ||
175 | 407 | filter_list = filters.get_filters(args.filters) | ||
176 | 408 | product_streams = util.load_product_streams(args.data_d) | ||
177 | 409 | resign = False | ||
178 | 410 | |||
179 | 411 | for product_stream in product_streams: | ||
180 | 412 | product_stream_path = os.path.join(args.data_d, product_stream) | ||
181 | 413 | content = util.load_content(product_stream_path) | ||
182 | 414 | products = content['products'] | ||
183 | 415 | write_stream = False | ||
184 | 416 | for product, data in products.items(): | ||
185 | 417 | if ( | ||
186 | 418 | filters.filter_dict(filter_list, data) and | ||
187 | 419 | args.version in data['versions']): | ||
188 | 420 | print('Removing %s from %s' % (args.version, product)) | ||
189 | 421 | if not args.dry_run: | ||
190 | 422 | del data['versions'][args.version] | ||
191 | 423 | resign = write_stream = True | ||
192 | 424 | if write_stream: | ||
193 | 425 | with open(product_stream_path, 'wb') as f: | ||
194 | 426 | f.write(util.dump_data(content).strip()) | ||
195 | 427 | if resign: | ||
196 | 428 | util.gen_index_and_sign(args.data_d, not args.no_sign) | ||
197 | 429 | return 0 | ||
198 | 430 | |||
199 | 431 | |||
200 | 432 | def main_copy_version(args): | ||
201 | 433 | filter_list = filters.get_filters(args.filters) | ||
202 | 434 | product_streams = util.load_product_streams(args.data_d) | ||
203 | 435 | resign = False | ||
204 | 436 | |||
205 | 437 | for product_stream in product_streams: | ||
206 | 438 | product_stream_path = os.path.join(args.data_d, product_stream) | ||
207 | 439 | content = util.load_content(product_stream_path) | ||
208 | 440 | products = content['products'] | ||
209 | 441 | write_stream = False | ||
210 | 442 | for product, data in products.items(): | ||
211 | 443 | if ( | ||
212 | 444 | filters.filter_dict(filter_list, data) and | ||
213 | 445 | args.from_version in data['versions']): | ||
214 | 446 | print('Copying %s to %s in %s' % ( | ||
215 | 447 | args.from_version, args.to_version, product)) | ||
216 | 448 | if not args.dry_run: | ||
217 | 449 | new_version = copy.deepcopy( | ||
218 | 450 | data['versions'][args.from_version]) | ||
219 | 451 | for item in new_version['items'].values(): | ||
220 | 452 | old_path = os.path.join(args.data_d, item['path']) | ||
221 | 453 | item['path'] = item['path'].replace( | ||
222 | 454 | args.from_version, args.to_version) | ||
223 | 455 | new_path = os.path.join(args.data_d, item['path']) | ||
224 | 456 | if not os.path.exists(new_path): | ||
225 | 457 | os.makedirs( | ||
226 | 458 | os.path.dirname(new_path), exist_ok=True) | ||
227 | 459 | shutil.copy( | ||
228 | 460 | old_path, new_path, follow_symlinks=False) | ||
229 | 461 | data['versions'][args.to_version] = new_version | ||
230 | 462 | resign = write_stream = True | ||
231 | 463 | if write_stream: | ||
232 | 464 | with open(product_stream_path, 'wb') as f: | ||
233 | 465 | f.write(util.dump_data(content).strip()) | ||
234 | 466 | if resign: | ||
235 | 467 | util.gen_index_and_sign(args.data_d, not args.no_sign) | ||
236 | 468 | return 0 | ||
237 | 469 | |||
238 | 470 | |||
239 | 406 | def main_import(args): | 471 | def main_import(args): |
240 | 407 | """meph2-util import wraps the preferred command 'meph2-import'. | 472 | """meph2-util import wraps the preferred command 'meph2-import'. |
241 | 408 | 473 | ||
242 | diff --git a/meph2/util.py b/meph2/util.py | |||
243 | index 017e5ef..f990eb9 100644 | |||
244 | --- a/meph2/util.py | |||
245 | +++ b/meph2/util.py | |||
246 | @@ -264,15 +264,19 @@ def dump_data(data, end_cr=True): | |||
247 | 264 | return bytestr | 264 | return bytestr |
248 | 265 | 265 | ||
249 | 266 | 266 | ||
250 | 267 | def load_content(path): | ||
251 | 268 | if not os.path.exists(path): | ||
252 | 269 | return {} | ||
253 | 270 | with scontentsource.UrlContentSource(path) as tcs: | ||
254 | 271 | return sutil.load_content(tcs.read()) | ||
255 | 272 | |||
256 | 273 | |||
257 | 267 | def load_products(path, product_streams): | 274 | def load_products(path, product_streams): |
258 | 268 | products = {} | 275 | products = {} |
259 | 269 | for product_stream in product_streams: | 276 | for product_stream in product_streams: |
260 | 270 | product_stream_path = os.path.join(path, product_stream) | 277 | product_stream_path = os.path.join(path, product_stream) |
266 | 271 | if os.path.exists(product_stream_path): | 278 | product_listing = load_content(product_stream_path) |
267 | 272 | with scontentsource.UrlContentSource( | 279 | products.update(product_listing['products']) |
263 | 273 | product_stream_path) as tcs: | ||
264 | 274 | product_listing = sutil.load_content(tcs.read()) | ||
265 | 275 | products.update(product_listing['products']) | ||
268 | 276 | return products | 280 | return products |
269 | 277 | 281 | ||
270 | 278 | 282 |
LGTM