Merge lp:~paelzer/cloud-init/test-apt-source into lp:~cloud-init-dev/cloud-init/trunk

Proposed by Christian Ehrhardt  on 2016-05-12
Status: Merged
Merged at revision: 1224
Proposed branch: lp:~paelzer/cloud-init/test-apt-source
Merge into: lp:~cloud-init-dev/cloud-init/trunk
Diff against target: 1134 lines (+945/-56)
6 files modified
cloudinit/config/cc_apt_configure.py (+82/-32)
cloudinit/templater.py (+5/-0)
cloudinit/util.py (+10/-0)
doc/examples/cloud-config.txt (+131/-24)
tests/unittests/test_handler/test_handler_apt_configure_sources_list.py (+166/-0)
tests/unittests/test_handler/test_handler_apt_source.py (+551/-0)
To merge this branch: bzr merge lp:~paelzer/cloud-init/test-apt-source
Reviewer Review Type Date Requested Status
Scott Moser 2016-05-12 Needs Fixing on 2016-05-24
Review via email: mp+294521@code.launchpad.net

Commit message

Apt sources configuration improvements

- keyid-only (no source statement)
- key only (no source statement)
- custom source.list template
- support long gpg key fingerprints with spaces
- fix issue with key's that were already in the local gpg keyring
- allowing a new format to specify apt_sources in a dictionary instead of a list to allow merging of configurations

Along the way to ensure things are working this series also added several tests. Related to the new format and the realization of corner cases related to "no" or "multiple" specified filenames further unittests were added.

Testing various combinations of the execution as it was so far:
- test_apt_source_basic Test Fix deb source string, has to overwrite mirror conf in params
- test_apt_source_replace Test Autoreplacement of MIRROR and RELEASE in source specs
- test_apt_source_keyid Test specification of a source + keyid
- test_apt_source_key Test specification of a source + key
- test_apt_source_keyonly Test specification key without source
- test_apt_source_keyidonly Test specification of a keyid without source
- test_apt_source_ppa Test specification of a ppa
- test_apt_source_list_debian Test rendering of a source.list from template for debian
- test_apt_source_list_ubuntu Test rendering of a source.list from template for ubuntu
- test_apt_source_list_debian_mirrorfail Test rendering of a source.list from template for debian
- test_apt_source_list_ubuntu_mirrorfail Test rendering of a source.list from template for ubuntu
- test_apt_srcl_custom Test rendering from a custom source.list template
- test_apt_source_keyid_real Test specification of a keyid without source incl real addition (no mock) of the add
- test_apt_source_longkeyid_real Test specification of a long key fingerprint without source incl real addition (no mock) of the add

Testing variations of above cases but without filename being specified
- test_apt_src_basic_nofn
- test_apt_src_replace_nofn
- test_apt_src_keyid_nofn
- test_apt_src_key_nofn

Testing variations of above cases with multiple specifications in one call:
test_apt_src_basic_tri
test_apt_src_replace_tri
test_apt_src_keyid_tri
test_apt_src_ppa_tri

Testing the new format:
test_apt_src_basic_dict
test_apt_src_basic_dict_tri
test_apt_src_replace_dict_tri
test_convert_to_new_format

Description of the change

As discussed, the cloud-init portion of key-only (no source) and alternative source.list template.

I refused to do all the major changes you indicated as "might" in your mail.
Especially since IIRC the thought is to SRU it back.
The impact on backward compatibility could be too big.

Instead I tried to take a safe approach.
For none of the already existing features that touched existed unit tests.
So I wrote these, then added functionality and verified at least nothing on the old tests broke.
Finally I added tests for the new functions and also documentations in the example files.

To post a comment you must log in.
Scott Moser (smoser) wrote :

looks really good. a couple things to tweak.

Scott Moser (smoser) wrote :

I *do* want the dictionary format supported though.
dictionaries just merge so much more easily.

so this MP doesn't have to have it, but we want dictionary 'apt_sources' to support being a dictionary to fix that bug, and want that in curtin.

1246. By Christian Ehrhardt  on 2016-05-12

remove no more applicable "not supported" statements

1247. By Christian Ehrhardt  on 2016-05-12

move errorlist.append out of add_key

1248. By Christian Ehrhardt  on 2016-05-12

remove Unnecessary parens in add_key

1249. By Christian Ehrhardt  on 2016-05-12

fix EXPORT_GPG_KEYID for long key fingerprints

1250. By Christian Ehrhardt  on 2016-05-12

split add_key and add_key_raw fior better testability

1251. By Christian Ehrhardt  on 2016-05-12

Adding test_apt_source_keyid_real and test_apt_source_longkeyid_real

This now ensures that the stack of fetching IDs from keyservers and adding them
really works by comparing against known good keys that are expected.

1252. By Christian Ehrhardt  on 2016-05-12

remove superfluous import

1253. By Christian Ehrhardt  on 2016-05-12

alphabetical import order

1254. By Christian Ehrhardt  on 2016-05-12

alphabetical order on imports

1255. By Christian Ehrhardt  on 2016-05-12

fix old typo in example

1256. By Christian Ehrhardt  on 2016-05-12

improve spacing in apt_source_list test

1257. By Christian Ehrhardt  on 2016-05-12

streamline code and sanitize expected result string definition

1258. By Christian Ehrhardt  on 2016-05-12

make pep8 happy with a few spaces

1259. By Christian Ehrhardt  on 2016-05-23

generalize test_apt_source_basic to be reusable across more testcases

1260. By Christian Ehrhardt  on 2016-05-23

test_apt_source_basic_nofn check for non-specified filename

Cloud-inint uses a default fallback, we want to ensure no code change modfies
this behaviour.

1261. By Christian Ehrhardt  on 2016-05-23

drop unused mockappsubp

1262. By Christian Ehrhardt  on 2016-05-23

extend test_apt_source_replace by a no-filename case

1263. By Christian Ehrhardt  on 2016-05-23

extend test_apt_source_keyid by no filename case

1264. By Christian Ehrhardt  on 2016-05-23

put fallbackfn to init

This was now used by multiple methods, no need to duplicate code.

1265. By Christian Ehrhardt  on 2016-05-23

extend test_apt_source_key by nofn case

1266. By Christian Ehrhardt  on 2016-05-23

support apt_sources to be a dictionary

key is the filename, and "old" input shall be handled as it was all the time.
For compatibility this will (continue to) overwrite the file of multiple
options that did not specify an output file (they all get the same default).
Yet it will process them all - as it always did - e.g. to add the keys of all
of them.

Any users of the new format won't have these issues, as they will always have
a key.

1267. By Christian Ehrhardt  on 2016-05-23

warn about multiple colliding apt_source without filenames

1268. By Christian Ehrhardt  on 2016-05-23

fix function names in inline doc

1269. By Christian Ehrhardt  on 2016-05-23

testcases with multiple source list entries

1270. By Christian Ehrhardt  on 2016-05-23

add triple case for test_apt_source_keyid_triple incl triple key check

1271. By Christian Ehrhardt  on 2016-05-23

make checkers happy about unused loop index

1272. By Christian Ehrhardt  on 2016-05-23

add triple test for ppa adding

1273. By Christian Ehrhardt  on 2016-05-23

fix issue with dictionary style apt_sources handling filenames

1274. By Christian Ehrhardt  on 2016-05-23

add test_apt_source_basic_dict

This is the basic testcase but in the new dictionary format

1275. By Christian Ehrhardt  on 2016-05-23

unify basic triple check and add test_apt_src_basic_dict_triple based on it

1276. By Christian Ehrhardt  on 2016-05-23

make sure we only handle list or dict apt_sources and bail out for others

1277. By Christian Ehrhardt  on 2016-05-23

shorten method names to follow python rules

1278. By Christian Ehrhardt  on 2016-05-23

add test_apt_src_replace_dict_tri

This includes a test for the weird but valid case in the new dictionary syntax
that one sets a key (which is the filename) but overwrites the filename value
inside of it.

1279. By Christian Ehrhardt  on 2016-05-23

modify cloud-config examples to match the new apt_source format

1280. By Christian Ehrhardt  on 2016-05-23

final pep8 check fixups

Christian Ehrhardt  (paelzer) wrote :

Following the discussion with smoser I added the requested new dictionary based format to apt_sources. This goes along with a collection of extra testcases to ensure the behavior of new (and old) configuration works.

Scott Moser (smoser) wrote :

so, some comments inline . that might seem like a lot, and that i'm nit picking. Sorry if it seems like that.

review: Needs Fixing
1281. By Christian Ehrhardt  on 2016-05-24

fix typo in examples doc

1282. By Christian Ehrhardt  on 2016-05-24

improve examples of ap_source

Christian Ehrhardt  (paelzer) wrote :

> so, some comments inline . that might seem like a lot, and that i'm nit
> picking. Sorry if it seems like that.

You are not, I'm happy about a good review and that it is!
If the file would be free of warnings in general it would be easier to see the new ones :-)

I usually found a lot of missing docstrings and invalid constant/variable name issues of pylint littering my view of checker warnings probably too much.

Anyway - most of the (not nit) picks are stuff I copied or moved and now you just look at it again - so I don't feel offended but assisted to make the code better.

I found even more minor things on my own in the meantime that I can merge in given that changes of a respin.

Christian Ehrhardt  (paelzer) wrote :

On Tue, May 24, 2016 at 4:01 PM, Scott Moser <email address hidden> wrote:

> so, some comments inline . that might seem like a lot, and that i'm nit
> picking. Sorry if it seems like that.
>

Having worked on the list of feedback I really found that they are mostly
picks on older code I moved - so I feel good :-)

And for my English sometimes I think I just shouldn't try to race to a new
MP at the end of a day :-)

> > + """
> > + if 'keyid' in ent and 'key' not in ent:
> > + keyserver = "keyserver.ubuntu.com"
> > + if 'keyserver' in ent:
> > + keyserver = ent['keyserver']
> > + try:
> > + ent['key'] = getkeybyid(ent['keyid'], keyserver)
> > + except:
> > + raise Exception('failed to get key from %s' % keyserver)
>
> KeyError ? or LookupError maybe? might seem reasonable.
> catch the subp.ProcessExecutionError specifically or even just let it go
> through would be better than swallowing it with a more generic error.
>
>
Yeah, given a second thought I like not catching it at that level the most.

> >
> > errorlist = []
> > - for ent in srclist:
> > + # convert old list format to new dict based format
>
> converting the array to a dictionary seems like a method itself that would
> be stand alone be easily unit tested.
>

 I first thought to refuse that since so many older tests are testing that
implicitly but ten decided to add it (refusing to go home to the family -
remote this week for better internet access).
Uploading updated MP now ...

1283. By Christian Ehrhardt  on 2016-05-24

rebased with upstream and reolved merge conflicts

1284. By Christian Ehrhardt  on 2016-05-24

integrate further smaller review feedback

1285. By Christian Ehrhardt  on 2016-05-24

pacify pep8 regarding the new changes

1286. By Christian Ehrhardt  on 2016-05-24

add test for the now isolated convert_to_new_format function

Christian Ehrhardt  (paelzer) wrote :

Review feedback integrated and new test test_convert_to_new_format added as requested.
Ready for another review or merging.

1287. By Christian Ehrhardt  on 2016-05-25

make test_apt_srcl_custom independent to where it is executed

Christian Ehrhardt  (paelzer) wrote :

by ongoing testing I found that one of the unittests depended on the system being xenial, this last commit fixes that dependency

1288. By Christian Ehrhardt  on 2016-05-25

fix inline doc of test_apt_src_longkeyid_real

Scott Moser (smoser) wrote :

one more, but i think we're there.

1289. By Christian Ehrhardt  on 2016-05-30

drop errorlist from convert_to_new_format

1290. By Christian Ehrhardt  on 2016-05-30

add test for wrong apt_source format

1291. By Christian Ehrhardt  on 2016-05-30

improve wording in the examples

1292. By Christian Ehrhardt  on 2016-05-30

fix EXPORT_GPG_KEYID for existing keys

This was broken for keys already existing in the local keyring.
There instead of the keycontent it reported the header like:
 pub 1024R/03683F77 2009-10-27
 uid Launchpad PPA for Scott Moser

1293. By Christian Ehrhardt  on 2016-05-30

merge with last upstream to avoid merging conflicts on MP

Christian Ehrhardt  (paelzer) wrote :

Thanks for the repeated great review, due to that I changed:
- warning/errors in convert_to_new_format
- improved example wording

Further on I:
- extended test_convert_to_new_format to test wrong format
- fixed an issue identified the key* tests depending if it was already imported in gpg (that turned out to be a real issue with the implementation)
- rebased again to avoid merge conflicts

Will update commit message now ...

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'cloudinit/config/cc_apt_configure.py'
2--- cloudinit/config/cc_apt_configure.py 2016-05-12 17:56:26 +0000
3+++ cloudinit/config/cc_apt_configure.py 2016-05-30 11:10:57 +0000
4@@ -40,9 +40,9 @@
5 k=${1} ks=${2};
6 exec 2>/dev/null
7 [ -n "$k" ] || exit 1;
8- armour=$(gpg --list-keys --armour "${k}")
9+ armour=$(gpg --export --armour "${k}")
10 if [ -z "${armour}" ]; then
11- gpg --keyserver ${ks} --recv $k >/dev/null &&
12+ gpg --keyserver ${ks} --recv "${k}" >/dev/null &&
13 armour=$(gpg --export --armour "${k}") &&
14 gpg --batch --yes --delete-keys "${k}"
15 fi
16@@ -70,7 +70,7 @@
17
18 if not util.get_cfg_option_bool(cfg,
19 'apt_preserve_sources_list', False):
20- generate_sources_list(release, mirrors, cloud, log)
21+ generate_sources_list(cfg, release, mirrors, cloud, log)
22 old_mirrors = cfg.get('apt_old_mirrors',
23 {"primary": "archive.ubuntu.com/ubuntu",
24 "security": "security.ubuntu.com/ubuntu"})
25@@ -149,7 +149,17 @@
26 return stdout.strip()
27
28
29-def generate_sources_list(codename, mirrors, cloud, log):
30+def generate_sources_list(cfg, codename, mirrors, cloud, log):
31+ params = {'codename': codename}
32+ for k in mirrors:
33+ params[k] = mirrors[k]
34+
35+ custtmpl = cfg.get('apt_custom_sources_list', None)
36+ if custtmpl is not None:
37+ templater.render_string_to_file(custtmpl,
38+ '/etc/apt/sources.list', params)
39+ return
40+
41 template_fn = cloud.get_template_filename('sources.list.%s' %
42 (cloud.distro.name))
43 if not template_fn:
44@@ -158,12 +168,60 @@
45 log.warn("No template found, not rendering /etc/apt/sources.list")
46 return
47
48- params = {'codename': codename}
49- for k in mirrors:
50- params[k] = mirrors[k]
51 templater.render_to_file(template_fn, '/etc/apt/sources.list', params)
52
53
54+def add_key_raw(key):
55+ """
56+ actual adding of a key as defined in key argument
57+ to the system
58+ """
59+ try:
60+ util.subp(('apt-key', 'add', '-'), key)
61+ except util.ProcessExecutionError:
62+ raise Exception('failed add key')
63+
64+
65+def add_key(ent):
66+ """
67+ add key to the system as defined in ent (if any)
68+ supports raw keys or keyid's
69+ The latter will as a first step fetch the raw key from a keyserver
70+ """
71+ if 'keyid' in ent and 'key' not in ent:
72+ keyserver = "keyserver.ubuntu.com"
73+ if 'keyserver' in ent:
74+ keyserver = ent['keyserver']
75+ ent['key'] = getkeybyid(ent['keyid'], keyserver)
76+
77+ if 'key' in ent:
78+ add_key_raw(ent['key'])
79+
80+
81+def convert_to_new_format(srclist):
82+ """ convert_to_new_format
83+ convert the old list based format to the new dict based one
84+ """
85+ srcdict = {}
86+ if isinstance(srclist, list):
87+ for srcent in srclist:
88+ if 'filename' not in srcent:
89+ # file collides for multiple !filename cases for compatibility
90+ # yet we need them all processed, so not same dictionary key
91+ srcent['filename'] = "cloud_config_sources.list"
92+ key = util.rand_dict_key(srcdict, "cloud_config_sources.list")
93+ else:
94+ # all with filename use that as key (matching new format)
95+ key = srcent['filename']
96+ srcdict[key] = srcent
97+ elif isinstance(srclist, dict):
98+ srcdict = srclist
99+ else:
100+ raise ValueError("unknown apt_sources format")
101+
102+ return srcdict
103+
104+
105 def add_sources(srclist, template_params=None, aa_repo_match=None):
106 """
107 add entries in /etc/apt/sources.list.d for each abbreviated
108@@ -178,14 +236,29 @@
109 return False
110
111 errorlist = []
112- for ent in srclist:
113+ srcdict = convert_to_new_format(srclist)
114+
115+ for filename in srcdict:
116+ ent = srcdict[filename]
117+ if 'filename' not in ent:
118+ ent['filename'] = filename
119+
120+ # keys can be added without specifying a source
121+ try:
122+ add_key(ent)
123+ except Exception as detail:
124+ errorlist.append([ent, detail])
125+
126 if 'source' not in ent:
127 errorlist.append(["", "missing source"])
128 continue
129-
130 source = ent['source']
131 source = templater.render_string(source, template_params)
132
133+ if not ent['filename'].startswith(os.path.sep):
134+ ent['filename'] = os.path.join("/etc/apt/sources.list.d/",
135+ ent['filename'])
136+
137 if aa_repo_match(source):
138 try:
139 util.subp(["add-apt-repository", source])
140@@ -194,29 +267,6 @@
141 ("add-apt-repository failed. " + str(e))])
142 continue
143
144- if 'filename' not in ent:
145- ent['filename'] = 'cloud_config_sources.list'
146-
147- if not ent['filename'].startswith("/"):
148- ent['filename'] = os.path.join("/etc/apt/sources.list.d/",
149- ent['filename'])
150-
151- if ('keyid' in ent and 'key' not in ent):
152- ks = "keyserver.ubuntu.com"
153- if 'keyserver' in ent:
154- ks = ent['keyserver']
155- try:
156- ent['key'] = getkeybyid(ent['keyid'], ks)
157- except Exception:
158- errorlist.append([source, "failed to get key from %s" % ks])
159- continue
160-
161- if 'key' in ent:
162- try:
163- util.subp(('apt-key', 'add', '-'), ent['key'])
164- except Exception:
165- errorlist.append([source, "failed add key"])
166-
167 try:
168 contents = "%s\n" % (source)
169 util.write_file(ent['filename'], contents, omode="ab")
170
171=== modified file 'cloudinit/templater.py'
172--- cloudinit/templater.py 2015-01-23 02:21:04 +0000
173+++ cloudinit/templater.py 2016-05-30 11:10:57 +0000
174@@ -142,6 +142,11 @@
175 util.write_file(outfn, contents, mode=mode)
176
177
178+def render_string_to_file(content, outfn, params, mode=0o644):
179+ contents = render_string(content, params)
180+ util.write_file(outfn, contents, mode=mode)
181+
182+
183 def render_string(content, params):
184 if not params:
185 params = {}
186
187=== modified file 'cloudinit/util.py'
188--- cloudinit/util.py 2016-05-12 17:56:26 +0000
189+++ cloudinit/util.py 2016-05-30 11:10:57 +0000
190@@ -336,6 +336,16 @@
191 return "".join([random.choice(select_from) for _x in range(0, strlen)])
192
193
194+def rand_dict_key(dictionary, postfix=None):
195+ if not postfix:
196+ postfix = ""
197+ while True:
198+ newkey = rand_str(strlen=8) + "_" + postfix
199+ if newkey not in dictionary:
200+ break
201+ return newkey
202+
203+
204 def read_conf(fname):
205 try:
206 return load_yaml(load_file(fname), default={})
207
208=== modified file 'doc/examples/cloud-config.txt'
209--- doc/examples/cloud-config.txt 2015-03-04 17:42:34 +0000
210+++ doc/examples/cloud-config.txt 2016-05-30 11:10:57 +0000
211@@ -72,14 +72,87 @@
212 # then apt_mirror above will have no effect
213 apt_preserve_sources_list: true
214
215+# Provide a custom template for rendering sources.list
216+# Default: a default template for Ubuntu/Debain will be used as packaged in
217+# Ubuntu: /etc/cloud/templates/sources.list.ubuntu.tmpl
218+# Debian: /etc/cloud/templates/sources.list.debian.tmpl
219+# Others: n/a
220+# This will follow the normal mirror/codename replacement rules before
221+# being written to disk.
222+apt_custom_sources_list: |
223+ ## template:jinja
224+ ## Note, this file is written by cloud-init on first boot of an instance
225+ ## modifications made here will not survive a re-bundle.
226+ ## if you wish to make changes you can:
227+ ## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg
228+ ## or do the same in user-data
229+ ## b.) add sources in /etc/apt/sources.list.d
230+ ## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl
231+ deb {{mirror}} {{codename}} main restricted
232+ deb-src {{mirror}} {{codename}} main restricted
233+
234+ # could drop some of the usually used entries
235+
236+ # could refer to other mirrors
237+ deb http://ddebs.ubuntu.com {{codename}} main restricted universe multiverse
238+ deb http://ddebs.ubuntu.com {{codename}}-updates main restricted universe multiverse
239+ deb http://ddebs.ubuntu.com {{codename}}-proposed main restricted universe multiverse
240+
241+ # or even more uncommon examples like local or NFS mounted repos,
242+ # eventually whatever is compatible with sources.list syntax
243+ deb file:/home/apt/debian unstable main contrib non-free
244+
245 # 'source' entries in apt-sources that match this python regex
246 # expression will be passed to add-apt-repository
247 add_apt_repo_match: '^[\w-]+:\w'
248
249+# 'apt_sources' is a dictionary
250+# The key is the filename and will be prepended by /etc/apt/sources.list.d/ if
251+# it doesn't start with a '/'.
252+# There are certain cases - where no content is written into a source.list file
253+# where the filename will be ignored - yet it can still be used as index for
254+# merging.
255+# The value it maps to is a dictionary with the following optional entries:
256+# source: a sources.list entry (some variable replacements apply)
257+# keyid: providing a key to import via shortid or fingerprint
258+# key: providing a raw PGP key
259+# keyserver: keyserver to fetch keys from, default is keyserver.ubuntu.com
260+# filename: for compatibility with the older format (now the key to this
261+# dictionary is the filename). If specified this overwrites the
262+# filename given as key.
263+
264+# the new "filename: {specification-dictionary}, filename2: ..." format allows
265+# better merging between multiple input files than a list like:
266+# cloud-config1
267+# sources:
268+ s1: {'key': 'key1', 'source': 'source1'}
269+# cloud-config2
270+# sources:
271+ s2: {'key': 'key2'}
272+ s1: {filename: 'foo'}
273+# this would be merged to
274+#sources:
275+# s1:
276+# filename: foo
277+# key: key1
278+# source: source1
279+# s2:
280+# key: key2
281+# Be aware that this style of merging is not the default (for backward
282+# compatibility reasons). You should specify the following merge_how to get
283+# this more complete and modern merging behaviour:
284+# merge_how: "list()+dict()+str()"
285+# This would then also be equivalent to the config merging used in curtin
286+# (https://launchpad.net/curtin).
287+
288+# for more details see below in the various examples
289+
290 apt_sources:
291- - source: "deb http://ppa.launchpad.net/byobu/ppa/ubuntu karmic main"
292+ byobu-ppa.list:
293+ source: "deb http://ppa.launchpad.net/byobu/ppa/ubuntu karmic main"
294 keyid: F430BBA5 # GPG key ID published on a key server
295- filename: byobu-ppa.list
296+ # adding a source.list line, importing a gpg key for a given key id and
297+ # storing it in the file /etc/apt/sources.list.d/byobu-ppa.list
298
299 # PPA shortcut:
300 # * Setup correct apt sources.list line
301@@ -87,7 +160,9 @@
302 #
303 # See https://help.launchpad.net/Packaging/PPA for more information
304 # this requires 'add-apt-repository'
305- - source: "ppa:smoser/ppa" # Quote the string
306+ # due to that the filename key is ignored in this case
307+ ignored1:
308+ source: "ppa:smoser/ppa" # Quote the string
309
310 # Custom apt repository:
311 # * all that is required is 'source'
312@@ -95,42 +170,74 @@
313 # * [optional] Import the apt signing key from the keyserver
314 # * Defaults:
315 # + keyserver: keyserver.ubuntu.com
316- # + filename: cloud_config_sources.list
317 #
318 # See sources.list man page for more information about the format
319- - source: deb http://archive.ubuntu.com/ubuntu karmic-backports main universe multiverse restricted
320+ my-repo.list:
321+ source: deb http://archive.ubuntu.com/ubuntu karmic-backports main universe multiverse restricted
322
323 # sources can use $MIRROR and $RELEASE and they will be replaced
324 # with the local mirror for this cloud, and the running release
325 # the entry below would be possibly turned into:
326- # - source: deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu natty multiverse
327- - source: deb $MIRROR $RELEASE multiverse
328+ # source: deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu natty multiverse
329+ my-repo.list:
330+ source: deb $MIRROR $RELEASE multiverse
331
332 # this would have the same end effect as 'ppa:byobu/ppa'
333- - source: "deb http://ppa.launchpad.net/byobu/ppa/ubuntu karmic main"
334+ my-repo.list:
335+ source: "deb http://ppa.launchpad.net/byobu/ppa/ubuntu karmic main"
336 keyid: F430BBA5 # GPG key ID published on a key server
337 filename: byobu-ppa.list
338
339+ # this would only import the key without adding a ppa or other source spec
340+ # since this doesn't generate a source.list file the filename key is ignored
341+ ignored2:
342+ keyid: F430BBA5 # GPG key ID published on a key server
343+
344+ # In general keyid's can also be specified via their long fingerprints
345+ # since this doesn't generate a source.list file the filename key is ignored
346+ ignored3:
347+ keyid: B59D 5F15 97A5 04B7 E230 6DCA 0620 BBCF 0368 3F77
348+
349 # Custom apt repository:
350 # * The apt signing key can also be specified
351 # by providing a pgp public key block
352- # * Providing the PBG key here is the most robust method for
353+ # * Providing the PGP key here is the most robust method for
354 # specifying a key, as it removes dependency on a remote key server
355-
356- - source: deb http://ppa.launchpad.net/alestic/ppa/ubuntu karmic main
357- key: | # The value needs to start with -----BEGIN PGP PUBLIC KEY BLOCK-----
358- -----BEGIN PGP PUBLIC KEY BLOCK-----
359- Version: SKS 1.0.10
360-
361- mI0ESpA3UQEEALdZKVIMq0j6qWAXAyxSlF63SvPVIgxHPb9Nk0DZUixn+akqytxG4zKCONz6
362- qLjoBBfHnynyVLfT4ihg9an1PqxRnTO+JKQxl8NgKGz6Pon569GtAOdWNKw15XKinJTDLjnj
363- 9y96ljJqRcpV9t/WsIcdJPcKFR5voHTEoABE2aEXABEBAAG0GUxhdW5jaHBhZCBQUEEgZm9y
364- IEFsZXN0aWOItgQTAQIAIAUCSpA3UQIbAwYLCQgHAwIEFQIIAwQWAgMBAh4BAheAAAoJEA7H
365- 5Qi+CcVxWZ8D/1MyYvfj3FJPZUm2Yo1zZsQ657vHI9+pPouqflWOayRR9jbiyUFIn0VdQBrP
366- t0FwvnOFArUovUWoKAEdqR8hPy3M3APUZjl5K4cMZR/xaMQeQRZ5CHpS4DBKURKAHC0ltS5o
367- uBJKQOZm5iltJp15cgyIkBkGe8Mx18VFyVglAZey
368- =Y2oI
369- -----END PGP PUBLIC KEY BLOCK-----
370+ my-repo.list:
371+ source: deb http://ppa.launchpad.net/alestic/ppa/ubuntu karmic main
372+ key: | # The value needs to start with -----BEGIN PGP PUBLIC KEY BLOCK-----
373+ -----BEGIN PGP PUBLIC KEY BLOCK-----
374+ Version: SKS 1.0.10
375+
376+ mI0ESpA3UQEEALdZKVIMq0j6qWAXAyxSlF63SvPVIgxHPb9Nk0DZUixn+akqytxG4zKCONz6
377+ qLjoBBfHnynyVLfT4ihg9an1PqxRnTO+JKQxl8NgKGz6Pon569GtAOdWNKw15XKinJTDLjnj
378+ 9y96ljJqRcpV9t/WsIcdJPcKFR5voHTEoABE2aEXABEBAAG0GUxhdW5jaHBhZCBQUEEgZm9y
379+ IEFsZXN0aWOItgQTAQIAIAUCSpA3UQIbAwYLCQgHAwIEFQIIAwQWAgMBAh4BAheAAAoJEA7H
380+ 5Qi+CcVxWZ8D/1MyYvfj3FJPZUm2Yo1zZsQ657vHI9+pPouqflWOayRR9jbiyUFIn0VdQBrP
381+ t0FwvnOFArUovUWoKAEdqR8hPy3M3APUZjl5K4cMZR/xaMQeQRZ5CHpS4DBKURKAHC0ltS5o
382+ uBJKQOZm5iltJp15cgyIkBkGe8Mx18VFyVglAZey
383+ =Y2oI
384+ -----END PGP PUBLIC KEY BLOCK-----
385+
386+ # Custom gpg key:
387+ # * As with keyid, a key may also be specified without a related source.
388+ # * all other facts mentioned above still apply
389+ # since this doesn't generate a source.list file the filename key is ignored
390+ ignored4:
391+ key: | # The value needs to start with -----BEGIN PGP PUBLIC KEY BLOCK-----
392+ -----BEGIN PGP PUBLIC KEY BLOCK-----
393+ Version: SKS 1.0.10
394+
395+ mI0ESpA3UQEEALdZKVIMq0j6qWAXAyxSlF63SvPVIgxHPb9Nk0DZUixn+akqytxG4zKCONz6
396+ qLjoBBfHnynyVLfT4ihg9an1PqxRnTO+JKQxl8NgKGz6Pon569GtAOdWNKw15XKinJTDLjnj
397+ 9y96ljJqRcpV9t/WsIcdJPcKFR5voHTEoABE2aEXABEBAAG0GUxhdW5jaHBhZCBQUEEgZm9y
398+ IEFsZXN0aWOItgQTAQIAIAUCSpA3UQIbAwYLCQgHAwIEFQIIAwQWAgMBAh4BAheAAAoJEA7H
399+ 5Qi+CcVxWZ8D/1MyYvfj3FJPZUm2Yo1zZsQ657vHI9+pPouqflWOayRR9jbiyUFIn0VdQBrP
400+ t0FwvnOFArUovUWoKAEdqR8hPy3M3APUZjl5K4cMZR/xaMQeQRZ5CHpS4DBKURKAHC0ltS5o
401+ uBJKQOZm5iltJp15cgyIkBkGe8Mx18VFyVglAZey
402+ =Y2oI
403+ -----END PGP PUBLIC KEY BLOCK-----
404+
405
406 ## apt config via system_info:
407 # under the 'system_info', you can further customize cloud-init's interaction
408
409=== added file 'tests/unittests/test_handler/test_handler_apt_configure_sources_list.py'
410--- tests/unittests/test_handler/test_handler_apt_configure_sources_list.py 1970-01-01 00:00:00 +0000
411+++ tests/unittests/test_handler/test_handler_apt_configure_sources_list.py 2016-05-30 11:10:57 +0000
412@@ -0,0 +1,166 @@
413+""" test_handler_apt_configure_sources_list
414+Test templating of sources list
415+"""
416+import logging
417+import os
418+import re
419+import shutil
420+import tempfile
421+
422+try:
423+ from unittest import mock
424+except ImportError:
425+ import mock
426+
427+from cloudinit import cloud
428+from cloudinit import distros
429+from cloudinit import helpers
430+from cloudinit import templater
431+from cloudinit import util
432+
433+from cloudinit.config import cc_apt_configure
434+from cloudinit.sources import DataSourceNone
435+
436+from .. import helpers as t_help
437+
438+LOG = logging.getLogger(__name__)
439+
440+YAML_TEXT_CUSTOM_SL = """
441+apt_mirror: http://archive.ubuntu.com/ubuntu/
442+apt_custom_sources_list: |
443+ ## template:jinja
444+ ## Note, this file is written by cloud-init on first boot of an instance
445+ ## modifications made here will not survive a re-bundle.
446+ ## if you wish to make changes you can:
447+ ## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg
448+ ## or do the same in user-data
449+ ## b.) add sources in /etc/apt/sources.list.d
450+ ## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl
451+
452+ # See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
453+ # newer versions of the distribution.
454+ deb {{mirror}} {{codename}} main restricted
455+ deb-src {{mirror}} {{codename}} main restricted
456+ # FIND_SOMETHING_SPECIAL
457+"""
458+
459+EXPECTED_CONVERTED_CONTENT = (
460+ """## Note, this file is written by cloud-init on first boot of an instance
461+## modifications made here will not survive a re-bundle.
462+## if you wish to make changes you can:
463+## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg
464+## or do the same in user-data
465+## b.) add sources in /etc/apt/sources.list.d
466+## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl
467+
468+# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
469+# newer versions of the distribution.
470+deb http://archive.ubuntu.com/ubuntu/ fakerelease main restricted
471+deb-src http://archive.ubuntu.com/ubuntu/ fakerelease main restricted
472+# FIND_SOMETHING_SPECIAL
473+""")
474+
475+
476+def load_tfile_or_url(*args, **kwargs):
477+ """ load_tfile_or_url
478+ load file and return content after decoding
479+ """
480+ return util.decode_binary(util.read_file_or_url(*args, **kwargs).contents)
481+
482+
483+class TestAptSourceConfigSourceList(t_help.FilesystemMockingTestCase):
484+ """ TestAptSourceConfigSourceList
485+ Main Class to test sources list rendering
486+ """
487+ def setUp(self):
488+ super(TestAptSourceConfigSourceList, self).setUp()
489+ self.subp = util.subp
490+ self.new_root = tempfile.mkdtemp()
491+ self.addCleanup(shutil.rmtree, self.new_root)
492+
493+ def _get_cloud(self, distro, metadata=None):
494+ self.patchUtils(self.new_root)
495+ paths = helpers.Paths({})
496+ cls = distros.fetch(distro)
497+ mydist = cls(distro, {}, paths)
498+ myds = DataSourceNone.DataSourceNone({}, mydist, paths)
499+ if metadata:
500+ myds.metadata.update(metadata)
501+ return cloud.Cloud(myds, paths, {}, mydist, None)
502+
503+ def apt_source_list(self, distro, mirror, mirrorcheck=None):
504+ """ apt_source_list
505+ Test rendering of a source.list from template for a given distro
506+ """
507+ if mirrorcheck is None:
508+ mirrorcheck = mirror
509+
510+ if isinstance(mirror, list):
511+ cfg = {'apt_mirror_search': mirror}
512+ else:
513+ cfg = {'apt_mirror': mirror}
514+ mycloud = self._get_cloud(distro)
515+
516+ with mock.patch.object(templater, 'render_to_file') as mocktmpl:
517+ with mock.patch.object(os.path, 'isfile',
518+ return_value=True) as mockisfile:
519+ cc_apt_configure.handle("notimportant", cfg, mycloud,
520+ LOG, None)
521+
522+ mockisfile.assert_any_call(
523+ ('/etc/cloud/templates/sources.list.%s.tmpl' % distro))
524+ mocktmpl.assert_called_once_with(
525+ ('/etc/cloud/templates/sources.list.%s.tmpl' % distro),
526+ '/etc/apt/sources.list',
527+ {'codename': '', 'primary': mirrorcheck, 'mirror': mirrorcheck})
528+
529+ def test_apt_source_list_debian(self):
530+ """ test_apt_source_list_debian
531+ Test rendering of a source.list from template for debian
532+ """
533+ self.apt_source_list('debian', 'http://httpredir.debian.org/debian')
534+
535+ def test_apt_source_list_ubuntu(self):
536+ """ test_apt_source_list_ubuntu
537+ Test rendering of a source.list from template for ubuntu
538+ """
539+ self.apt_source_list('ubuntu', 'http://archive.ubuntu.com/ubuntu/')
540+
541+ def test_apt_srcl_debian_mirrorfail(self):
542+ """ test_apt_source_list_debian_mirrorfail
543+ Test rendering of a source.list from template for debian
544+ """
545+ self.apt_source_list('debian', ['http://does.not.exist',
546+ 'http://httpredir.debian.org/debian'],
547+ 'http://httpredir.debian.org/debian')
548+
549+ def test_apt_srcl_ubuntu_mirrorfail(self):
550+ """ test_apt_source_list_ubuntu_mirrorfail
551+ Test rendering of a source.list from template for ubuntu
552+ """
553+ self.apt_source_list('ubuntu', ['http://does.not.exist',
554+ 'http://archive.ubuntu.com/ubuntu/'],
555+ 'http://archive.ubuntu.com/ubuntu/')
556+
557+ def test_apt_srcl_custom(self):
558+ """ test_apt_srcl_custom
559+ Test rendering from a custom source.list template
560+ """
561+ cfg = util.load_yaml(YAML_TEXT_CUSTOM_SL)
562+ mycloud = self._get_cloud('ubuntu')
563+
564+ # the second mock restores the original subp
565+ with mock.patch.object(util, 'write_file') as mockwrite:
566+ with mock.patch.object(util, 'subp', self.subp):
567+ with mock.patch.object(cc_apt_configure, 'get_release',
568+ return_value='fakerelease'):
569+ cc_apt_configure.handle("notimportant", cfg, mycloud,
570+ LOG, None)
571+
572+ mockwrite.assert_called_once_with(
573+ '/etc/apt/sources.list',
574+ EXPECTED_CONVERTED_CONTENT,
575+ mode=420)
576+
577+
578+# vi: ts=4 expandtab
579
580=== added file 'tests/unittests/test_handler/test_handler_apt_source.py'
581--- tests/unittests/test_handler/test_handler_apt_source.py 1970-01-01 00:00:00 +0000
582+++ tests/unittests/test_handler/test_handler_apt_source.py 2016-05-30 11:10:57 +0000
583@@ -0,0 +1,551 @@
584+""" test_handler_apt_source
585+Testing various config variations of the apt_source config
586+"""
587+import os
588+import re
589+import shutil
590+import tempfile
591+
592+try:
593+ from unittest import mock
594+except ImportError:
595+ import mock
596+from mock import call
597+
598+from cloudinit.config import cc_apt_configure
599+from cloudinit import util
600+
601+from ..helpers import TestCase
602+
603+EXPECTEDKEY = """-----BEGIN PGP PUBLIC KEY BLOCK-----
604+Version: GnuPG v1
605+
606+mI0ESuZLUgEEAKkqq3idtFP7g9hzOu1a8+v8ImawQN4TrvlygfScMU1TIS1eC7UQ
607+NUA8Qqgr9iUaGnejb0VciqftLrU9D6WYHSKz+EITefgdyJ6SoQxjoJdsCpJ7o9Jy
608+8PQnpRttiFm4qHu6BVnKnBNxw/z3ST9YMqW5kbMQpfxbGe+obRox59NpABEBAAG0
609+HUxhdW5jaHBhZCBQUEEgZm9yIFNjb3R0IE1vc2VyiLYEEwECACAFAkrmS1ICGwMG
610+CwkIBwMCBBUCCAMEFgIDAQIeAQIXgAAKCRAGILvPA2g/d3aEA/9tVjc10HOZwV29
611+OatVuTeERjjrIbxflO586GLA8cp0C9RQCwgod/R+cKYdQcHjbqVcP0HqxveLg0RZ
612+FJpWLmWKamwkABErwQLGlM/Hwhjfade8VvEQutH5/0JgKHmzRsoqfR+LMO6OS+Sm
613+S0ORP6HXET3+jC8BMG4tBWCTK/XEZw==
614+=ACB2
615+-----END PGP PUBLIC KEY BLOCK-----"""
616+
617+
618+def load_tfile_or_url(*args, **kwargs):
619+ """ load_tfile_or_url
620+ load file and return content after decoding
621+ """
622+ return util.decode_binary(util.read_file_or_url(*args, **kwargs).contents)
623+
624+
625+class TestAptSourceConfig(TestCase):
626+ """ TestAptSourceConfig
627+ Main Class to test apt_source configs
628+ """
629+ def setUp(self):
630+ super(TestAptSourceConfig, self).setUp()
631+ self.tmp = tempfile.mkdtemp()
632+ self.addCleanup(shutil.rmtree, self.tmp)
633+ self.aptlistfile = os.path.join(self.tmp, "single-deb.list")
634+ self.aptlistfile2 = os.path.join(self.tmp, "single-deb2.list")
635+ self.aptlistfile3 = os.path.join(self.tmp, "single-deb3.list")
636+ self.join = os.path.join
637+ # mock fallback filename into writable tmp dir
638+ self.fallbackfn = os.path.join(self.tmp, "etc/apt/sources.list.d/",
639+ "cloud_config_sources.list")
640+
641+ @staticmethod
642+ def _get_default_params():
643+ """ get_default_params
644+ Get the most basic default mrror and release info to be used in tests
645+ """
646+ params = {}
647+ params['RELEASE'] = cc_apt_configure.get_release()
648+ params['MIRROR'] = "http://archive.ubuntu.com/ubuntu"
649+ return params
650+
651+ def myjoin(self, *args, **kwargs):
652+ """ myjoin - redir into writable tmpdir"""
653+ if (args[0] == "/etc/apt/sources.list.d/" and
654+ args[1] == "cloud_config_sources.list" and
655+ len(args) == 2):
656+ return self.join(self.tmp, args[0].lstrip("/"), args[1])
657+ else:
658+ return self.join(*args, **kwargs)
659+
660+ def apt_src_basic(self, filename, cfg):
661+ """ apt_src_basic
662+ Test Fix deb source string, has to overwrite mirror conf in params
663+ """
664+ params = self._get_default_params()
665+
666+ cc_apt_configure.add_sources(cfg, params)
667+
668+ self.assertTrue(os.path.isfile(filename))
669+
670+ contents = load_tfile_or_url(filename)
671+ self.assertTrue(re.search(r"%s %s %s %s\n" %
672+ ("deb", "http://archive.ubuntu.com/ubuntu",
673+ "karmic-backports",
674+ "main universe multiverse restricted"),
675+ contents, flags=re.IGNORECASE))
676+
677+ def test_apt_src_basic(self):
678+ """ test_apt_src_basic
679+ Test Fix deb source string, has to overwrite mirror conf in params.
680+ Test with a filename provided in config.
681+ """
682+ cfg = {'source': ('deb http://archive.ubuntu.com/ubuntu'
683+ ' karmic-backports'
684+ ' main universe multiverse restricted'),
685+ 'filename': self.aptlistfile}
686+ self.apt_src_basic(self.aptlistfile, [cfg])
687+
688+ def test_apt_src_basic_dict(self):
689+ """ test_apt_src_basic_dict
690+ Test Fix deb source string, has to overwrite mirror conf in params.
691+ Test with a filename provided in config.
692+ Provided in a dictionary with filename being the key (new format)
693+ """
694+ cfg = {self.aptlistfile: {'source':
695+ ('deb http://archive.ubuntu.com/ubuntu'
696+ ' karmic-backports'
697+ ' main universe multiverse restricted')}}
698+ self.apt_src_basic(self.aptlistfile, cfg)
699+
700+ def apt_src_basic_tri(self, cfg):
701+ """ apt_src_basic_tri
702+ Test Fix three deb source string, has to overwrite mirror conf in
703+ params. Test with filenames provided in config.
704+ generic part to check three files with different content
705+ """
706+ self.apt_src_basic(self.aptlistfile, cfg)
707+
708+ # extra verify on two extra files of this test
709+ contents = load_tfile_or_url(self.aptlistfile2)
710+ self.assertTrue(re.search(r"%s %s %s %s\n" %
711+ ("deb", "http://archive.ubuntu.com/ubuntu",
712+ "precise-backports",
713+ "main universe multiverse restricted"),
714+ contents, flags=re.IGNORECASE))
715+ contents = load_tfile_or_url(self.aptlistfile3)
716+ self.assertTrue(re.search(r"%s %s %s %s\n" %
717+ ("deb", "http://archive.ubuntu.com/ubuntu",
718+ "lucid-backports",
719+ "main universe multiverse restricted"),
720+ contents, flags=re.IGNORECASE))
721+
722+ def test_apt_src_basic_tri(self):
723+ """ test_apt_src_basic_tri
724+ Test Fix three deb source string, has to overwrite mirror conf in
725+ params. Test with filenames provided in config.
726+ """
727+ cfg1 = {'source': ('deb http://archive.ubuntu.com/ubuntu'
728+ ' karmic-backports'
729+ ' main universe multiverse restricted'),
730+ 'filename': self.aptlistfile}
731+ cfg2 = {'source': ('deb http://archive.ubuntu.com/ubuntu'
732+ ' precise-backports'
733+ ' main universe multiverse restricted'),
734+ 'filename': self.aptlistfile2}
735+ cfg3 = {'source': ('deb http://archive.ubuntu.com/ubuntu'
736+ ' lucid-backports'
737+ ' main universe multiverse restricted'),
738+ 'filename': self.aptlistfile3}
739+ self.apt_src_basic_tri([cfg1, cfg2, cfg3])
740+
741+ def test_apt_src_basic_dict_tri(self):
742+ """ test_apt_src_basic_dict_tri
743+ Test Fix three deb source string, has to overwrite mirror conf in
744+ params. Test with filenames provided in config.
745+ Provided in a dictionary with filename being the key (new format)
746+ """
747+ cfg = {self.aptlistfile: {'source':
748+ ('deb http://archive.ubuntu.com/ubuntu'
749+ ' karmic-backports'
750+ ' main universe multiverse restricted')},
751+ self.aptlistfile2: {'source':
752+ ('deb http://archive.ubuntu.com/ubuntu'
753+ ' precise-backports'
754+ ' main universe multiverse restricted')},
755+ self.aptlistfile3: {'source':
756+ ('deb http://archive.ubuntu.com/ubuntu'
757+ ' lucid-backports'
758+ ' main universe multiverse restricted')}}
759+ self.apt_src_basic_tri(cfg)
760+
761+ def test_apt_src_basic_nofn(self):
762+ """ test_apt_src_basic_nofn
763+ Test Fix deb source string, has to overwrite mirror conf in params.
764+ Test without a filename provided in config and test for known fallback.
765+ """
766+ cfg = {'source': ('deb http://archive.ubuntu.com/ubuntu'
767+ ' karmic-backports'
768+ ' main universe multiverse restricted')}
769+ with mock.patch.object(os.path, 'join', side_effect=self.myjoin):
770+ self.apt_src_basic(self.fallbackfn, [cfg])
771+
772+ def apt_src_replacement(self, filename, cfg):
773+ """ apt_src_replace
774+ Test Autoreplacement of MIRROR and RELEASE in source specs
775+ """
776+ params = self._get_default_params()
777+ cc_apt_configure.add_sources(cfg, params)
778+
779+ self.assertTrue(os.path.isfile(filename))
780+
781+ contents = load_tfile_or_url(filename)
782+ self.assertTrue(re.search(r"%s %s %s %s\n" %
783+ ("deb", params['MIRROR'], params['RELEASE'],
784+ "multiverse"),
785+ contents, flags=re.IGNORECASE))
786+
787+ def test_apt_src_replace(self):
788+ """ test_apt_src_replace
789+ Test Autoreplacement of MIRROR and RELEASE in source specs with
790+ Filename being set
791+ """
792+ cfg = {'source': 'deb $MIRROR $RELEASE multiverse',
793+ 'filename': self.aptlistfile}
794+ self.apt_src_replacement(self.aptlistfile, [cfg])
795+
796+ def apt_src_replace_tri(self, cfg):
797+ """ apt_src_replace_tri
798+ Test three autoreplacements of MIRROR and RELEASE in source specs with
799+ generic part
800+ """
801+ self.apt_src_replacement(self.aptlistfile, cfg)
802+
803+ # extra verify on two extra files of this test
804+ params = self._get_default_params()
805+ contents = load_tfile_or_url(self.aptlistfile2)
806+ self.assertTrue(re.search(r"%s %s %s %s\n" %
807+ ("deb", params['MIRROR'], params['RELEASE'],
808+ "main"),
809+ contents, flags=re.IGNORECASE))
810+ contents = load_tfile_or_url(self.aptlistfile3)
811+ self.assertTrue(re.search(r"%s %s %s %s\n" %
812+ ("deb", params['MIRROR'], params['RELEASE'],
813+ "universe"),
814+ contents, flags=re.IGNORECASE))
815+
816+ def test_apt_src_replace_tri(self):
817+ """ test_apt_src_replace_tri
818+ Test three autoreplacements of MIRROR and RELEASE in source specs with
819+ Filename being set
820+ """
821+ cfg1 = {'source': 'deb $MIRROR $RELEASE multiverse',
822+ 'filename': self.aptlistfile}
823+ cfg2 = {'source': 'deb $MIRROR $RELEASE main',
824+ 'filename': self.aptlistfile2}
825+ cfg3 = {'source': 'deb $MIRROR $RELEASE universe',
826+ 'filename': self.aptlistfile3}
827+ self.apt_src_replace_tri([cfg1, cfg2, cfg3])
828+
829+ def test_apt_src_replace_dict_tri(self):
830+ """ test_apt_src_replace_dict_tri
831+ Test three autoreplacements of MIRROR and RELEASE in source specs with
832+ Filename being set
833+ Provided in a dictionary with filename being the key (new format)
834+ We also test a new special conditions of the new format that allows
835+ filenames to be overwritten inside the directory entry.
836+ """
837+ cfg = {self.aptlistfile: {'source': 'deb $MIRROR $RELEASE multiverse'},
838+ 'notused': {'source': 'deb $MIRROR $RELEASE main',
839+ 'filename': self.aptlistfile2},
840+ self.aptlistfile3: {'source': 'deb $MIRROR $RELEASE universe'}}
841+ self.apt_src_replace_tri(cfg)
842+
843+ def test_apt_src_replace_nofn(self):
844+ """ test_apt_src_replace_nofn
845+ Test Autoreplacement of MIRROR and RELEASE in source specs with
846+ No filename being set
847+ """
848+ cfg = {'source': 'deb $MIRROR $RELEASE multiverse'}
849+ with mock.patch.object(os.path, 'join', side_effect=self.myjoin):
850+ self.apt_src_replacement(self.fallbackfn, [cfg])
851+
852+ def apt_src_keyid(self, filename, cfg, keynum):
853+ """ apt_src_keyid
854+ Test specification of a source + keyid
855+ """
856+ params = self._get_default_params()
857+
858+ with mock.patch.object(util, 'subp',
859+ return_value=('fakekey 1234', '')) as mockobj:
860+ cc_apt_configure.add_sources(cfg, params)
861+
862+ # check if it added the right ammount of keys
863+ calls = []
864+ for _ in range(keynum):
865+ calls.append(call(('apt-key', 'add', '-'), 'fakekey 1234'))
866+ mockobj.assert_has_calls(calls, any_order=True)
867+
868+ self.assertTrue(os.path.isfile(filename))
869+
870+ contents = load_tfile_or_url(filename)
871+ self.assertTrue(re.search(r"%s %s %s %s\n" %
872+ ("deb",
873+ ('http://ppa.launchpad.net/smoser/'
874+ 'cloud-init-test/ubuntu'),
875+ "xenial", "main"),
876+ contents, flags=re.IGNORECASE))
877+
878+ def test_apt_src_keyid(self):
879+ """ test_apt_src_keyid
880+ Test specification of a source + keyid with filename being set
881+ """
882+ cfg = {'source': ('deb '
883+ 'http://ppa.launchpad.net/'
884+ 'smoser/cloud-init-test/ubuntu'
885+ ' xenial main'),
886+ 'keyid': "03683F77",
887+ 'filename': self.aptlistfile}
888+ self.apt_src_keyid(self.aptlistfile, [cfg], 1)
889+
890+ def test_apt_src_keyid_tri(self):
891+ """ test_apt_src_keyid_tri
892+ Test specification of a source + keyid with filename being set
893+ Setting three of such, check for content and keys
894+ """
895+ cfg1 = {'source': ('deb '
896+ 'http://ppa.launchpad.net/'
897+ 'smoser/cloud-init-test/ubuntu'
898+ ' xenial main'),
899+ 'keyid': "03683F77",
900+ 'filename': self.aptlistfile}
901+ cfg2 = {'source': ('deb '
902+ 'http://ppa.launchpad.net/'
903+ 'smoser/cloud-init-test/ubuntu'
904+ ' xenial universe'),
905+ 'keyid': "03683F77",
906+ 'filename': self.aptlistfile2}
907+ cfg3 = {'source': ('deb '
908+ 'http://ppa.launchpad.net/'
909+ 'smoser/cloud-init-test/ubuntu'
910+ ' xenial multiverse'),
911+ 'keyid': "03683F77",
912+ 'filename': self.aptlistfile3}
913+
914+ self.apt_src_keyid(self.aptlistfile, [cfg1, cfg2, cfg3], 3)
915+ contents = load_tfile_or_url(self.aptlistfile2)
916+ self.assertTrue(re.search(r"%s %s %s %s\n" %
917+ ("deb",
918+ ('http://ppa.launchpad.net/smoser/'
919+ 'cloud-init-test/ubuntu'),
920+ "xenial", "universe"),
921+ contents, flags=re.IGNORECASE))
922+ contents = load_tfile_or_url(self.aptlistfile3)
923+ self.assertTrue(re.search(r"%s %s %s %s\n" %
924+ ("deb",
925+ ('http://ppa.launchpad.net/smoser/'
926+ 'cloud-init-test/ubuntu'),
927+ "xenial", "multiverse"),
928+ contents, flags=re.IGNORECASE))
929+
930+ def test_apt_src_keyid_nofn(self):
931+ """ test_apt_src_keyid_nofn
932+ Test specification of a source + keyid without filename being set
933+ """
934+ cfg = {'source': ('deb '
935+ 'http://ppa.launchpad.net/'
936+ 'smoser/cloud-init-test/ubuntu'
937+ ' xenial main'),
938+ 'keyid': "03683F77"}
939+ with mock.patch.object(os.path, 'join', side_effect=self.myjoin):
940+ self.apt_src_keyid(self.fallbackfn, [cfg], 1)
941+
942+ def apt_src_key(self, filename, cfg):
943+ """ apt_src_key
944+ Test specification of a source + key
945+ """
946+ params = self._get_default_params()
947+
948+ with mock.patch.object(util, 'subp') as mockobj:
949+ cc_apt_configure.add_sources([cfg], params)
950+
951+ mockobj.assert_called_with(('apt-key', 'add', '-'), 'fakekey 4321')
952+
953+ self.assertTrue(os.path.isfile(filename))
954+
955+ contents = load_tfile_or_url(filename)
956+ self.assertTrue(re.search(r"%s %s %s %s\n" %
957+ ("deb",
958+ ('http://ppa.launchpad.net/smoser/'
959+ 'cloud-init-test/ubuntu'),
960+ "xenial", "main"),
961+ contents, flags=re.IGNORECASE))
962+
963+ def test_apt_src_key(self):
964+ """ test_apt_src_key
965+ Test specification of a source + key with filename being set
966+ """
967+ cfg = {'source': ('deb '
968+ 'http://ppa.launchpad.net/'
969+ 'smoser/cloud-init-test/ubuntu'
970+ ' xenial main'),
971+ 'key': "fakekey 4321",
972+ 'filename': self.aptlistfile}
973+ self.apt_src_key(self.aptlistfile, cfg)
974+
975+ def test_apt_src_key_nofn(self):
976+ """ test_apt_src_key_nofn
977+ Test specification of a source + key without filename being set
978+ """
979+ cfg = {'source': ('deb '
980+ 'http://ppa.launchpad.net/'
981+ 'smoser/cloud-init-test/ubuntu'
982+ ' xenial main'),
983+ 'key': "fakekey 4321"}
984+ with mock.patch.object(os.path, 'join', side_effect=self.myjoin):
985+ self.apt_src_key(self.fallbackfn, cfg)
986+
987+ def test_apt_src_keyonly(self):
988+ """ test_apt_src_keyonly
989+ Test specification key without source
990+ """
991+ params = self._get_default_params()
992+ cfg = {'key': "fakekey 4242",
993+ 'filename': self.aptlistfile}
994+
995+ with mock.patch.object(util, 'subp') as mockobj:
996+ cc_apt_configure.add_sources([cfg], params)
997+
998+ mockobj.assert_called_once_with(('apt-key', 'add', '-'),
999+ 'fakekey 4242')
1000+
1001+ # filename should be ignored on key only
1002+ self.assertFalse(os.path.isfile(self.aptlistfile))
1003+
1004+ def test_apt_src_keyidonly(self):
1005+ """ test_apt_src_keyidonly
1006+ Test specification of a keyid without source
1007+ """
1008+ params = self._get_default_params()
1009+ cfg = {'keyid': "03683F77",
1010+ 'filename': self.aptlistfile}
1011+
1012+ with mock.patch.object(util, 'subp',
1013+ return_value=('fakekey 1212', '')) as mockobj:
1014+ cc_apt_configure.add_sources([cfg], params)
1015+
1016+ mockobj.assert_called_with(('apt-key', 'add', '-'), 'fakekey 1212')
1017+
1018+ # filename should be ignored on key only
1019+ self.assertFalse(os.path.isfile(self.aptlistfile))
1020+
1021+ def test_apt_src_keyid_real(self):
1022+ """ test_apt_src_keyid_real
1023+ Test specification of a keyid without source incl
1024+ up to addition of the key (nothing but add_key_raw mocked)
1025+ """
1026+ keyid = "03683F77"
1027+ params = self._get_default_params()
1028+ cfg = {'keyid': keyid,
1029+ 'filename': self.aptlistfile}
1030+
1031+ with mock.patch.object(cc_apt_configure, 'add_key_raw') as mockobj:
1032+ cc_apt_configure.add_sources([cfg], params)
1033+
1034+ mockobj.assert_called_with(EXPECTEDKEY)
1035+
1036+ # filename should be ignored on key only
1037+ self.assertFalse(os.path.isfile(self.aptlistfile))
1038+
1039+ def test_apt_src_longkeyid_real(self):
1040+ """ test_apt_src_longkeyid_real
1041+ Test specification of a long key fingerprint without source incl
1042+ up to addition of the key (nothing but add_key_raw mocked)
1043+ """
1044+ keyid = "B59D 5F15 97A5 04B7 E230 6DCA 0620 BBCF 0368 3F77"
1045+ params = self._get_default_params()
1046+ cfg = {'keyid': keyid,
1047+ 'filename': self.aptlistfile}
1048+
1049+ with mock.patch.object(cc_apt_configure, 'add_key_raw') as mockobj:
1050+ cc_apt_configure.add_sources([cfg], params)
1051+
1052+ mockobj.assert_called_with(EXPECTEDKEY)
1053+
1054+ # filename should be ignored on key only
1055+ self.assertFalse(os.path.isfile(self.aptlistfile))
1056+
1057+ def test_apt_src_ppa(self):
1058+ """ test_apt_src_ppa
1059+ Test specification of a ppa
1060+ """
1061+ params = self._get_default_params()
1062+ cfg = {'source': 'ppa:smoser/cloud-init-test',
1063+ 'filename': self.aptlistfile}
1064+
1065+ # default matcher needed for ppa
1066+ matcher = re.compile(r'^[\w-]+:\w').search
1067+
1068+ with mock.patch.object(util, 'subp') as mockobj:
1069+ cc_apt_configure.add_sources([cfg], params, aa_repo_match=matcher)
1070+ mockobj.assert_called_once_with(['add-apt-repository',
1071+ 'ppa:smoser/cloud-init-test'])
1072+
1073+ # adding ppa should ignore filename (uses add-apt-repository)
1074+ self.assertFalse(os.path.isfile(self.aptlistfile))
1075+
1076+ def test_apt_src_ppa_tri(self):
1077+ """ test_apt_src_ppa_tri
1078+ Test specification of a ppa
1079+ """
1080+ params = self._get_default_params()
1081+ cfg1 = {'source': 'ppa:smoser/cloud-init-test',
1082+ 'filename': self.aptlistfile}
1083+ cfg2 = {'source': 'ppa:smoser/cloud-init-test2',
1084+ 'filename': self.aptlistfile2}
1085+ cfg3 = {'source': 'ppa:smoser/cloud-init-test3',
1086+ 'filename': self.aptlistfile3}
1087+
1088+ # default matcher needed for ppa
1089+ matcher = re.compile(r'^[\w-]+:\w').search
1090+
1091+ with mock.patch.object(util, 'subp') as mockobj:
1092+ cc_apt_configure.add_sources([cfg1, cfg2, cfg3], params,
1093+ aa_repo_match=matcher)
1094+ calls = [call(['add-apt-repository', 'ppa:smoser/cloud-init-test']),
1095+ call(['add-apt-repository', 'ppa:smoser/cloud-init-test2']),
1096+ call(['add-apt-repository', 'ppa:smoser/cloud-init-test3'])]
1097+ mockobj.assert_has_calls(calls, any_order=True)
1098+
1099+ # adding ppa should ignore all filenames (uses add-apt-repository)
1100+ self.assertFalse(os.path.isfile(self.aptlistfile))
1101+ self.assertFalse(os.path.isfile(self.aptlistfile2))
1102+ self.assertFalse(os.path.isfile(self.aptlistfile3))
1103+
1104+ def test_convert_to_new_format(self):
1105+ """ test_convert_to_new_format
1106+ Test the conversion of old to new format
1107+ And the noop conversion of new to new format as well
1108+ """
1109+ cfg1 = {'source': 'deb $MIRROR $RELEASE multiverse',
1110+ 'filename': self.aptlistfile}
1111+ cfg2 = {'source': 'deb $MIRROR $RELEASE main',
1112+ 'filename': self.aptlistfile2}
1113+ cfg3 = {'source': 'deb $MIRROR $RELEASE universe',
1114+ 'filename': self.aptlistfile3}
1115+ checkcfg = {self.aptlistfile: {'filename': self.aptlistfile,
1116+ 'source': 'deb $MIRROR $RELEASE '
1117+ 'multiverse'},
1118+ self.aptlistfile2: {'filename': self.aptlistfile2,
1119+ 'source': 'deb $MIRROR $RELEASE main'},
1120+ self.aptlistfile3: {'filename': self.aptlistfile3,
1121+ 'source': 'deb $MIRROR $RELEASE '
1122+ 'universe'}}
1123+
1124+ newcfg = cc_apt_configure.convert_to_new_format([cfg1, cfg2, cfg3])
1125+ self.assertEqual(newcfg, checkcfg)
1126+
1127+ newcfg2 = cc_apt_configure.convert_to_new_format(newcfg)
1128+ self.assertEqual(newcfg2, checkcfg)
1129+
1130+ with self.assertRaises(ValueError):
1131+ cc_apt_configure.convert_to_new_format(5)
1132+
1133+
1134+# vi: ts=4 expandtab