Merge lp:~paelzer/curtin/bug-1574113-derived-repositories into lp:~curtin-dev/curtin/trunk

Proposed by Christian Ehrhardt  on 2016-05-31
Status: Merged
Merged at revision: 410
Proposed branch: lp:~paelzer/curtin/bug-1574113-derived-repositories
Merge into: lp:~curtin-dev/curtin/trunk
Diff against target: 4008 lines (+3551/-146)
25 files modified
curtin/__init__.py (+2/-0)
curtin/commands/apt_config.py (+691/-0)
curtin/commands/curthooks.py (+11/-135)
curtin/commands/main.py (+1/-1)
curtin/gpg.py (+74/-0)
curtin/util.py (+144/-2)
doc/devel/README-vmtest.txt (+3/-3)
doc/topics/apt_source.rst (+152/-0)
examples/apt-source.yaml (+239/-0)
examples/tests/apt_config_command.yaml (+85/-0)
examples/tests/apt_source_custom.yaml (+97/-0)
examples/tests/apt_source_modify.yaml (+92/-0)
examples/tests/apt_source_modify_arches.yaml (+102/-0)
examples/tests/apt_source_modify_disable_suite.yaml (+92/-0)
examples/tests/apt_source_preserve.yaml (+98/-0)
examples/tests/apt_source_search.yaml (+97/-0)
examples/tests/apt_source_search_dns.yaml (+21/-0)
examples/tests/test_old_apt_features.yaml (+10/-0)
examples/tests/test_old_apt_features_ports.yaml (+10/-0)
tests/unittests/test_apt_custom_sources_list.py (+172/-0)
tests/unittests/test_apt_source.py (+927/-0)
tests/vmtests/__init__.py (+19/-5)
tests/vmtests/test_apt_config_cmd.py (+55/-0)
tests/vmtests/test_apt_source.py (+277/-0)
tests/vmtests/test_old_apt_features.py (+80/-0)
To merge this branch: bzr merge lp:~paelzer/curtin/bug-1574113-derived-repositories
Reviewer Review Type Date Requested Status
Server Team CI bot continuous-integration Approve on 2016-07-26
Ryan Harper 2016-05-31 Approve on 2016-06-14
Review via email: mp+296118@code.launchpad.net

Commit message

Adding apt features to curtin.

This allows to define derived repositories by being able to control various apt configurations, proxies, mirrors, ppa's, sources.list content and pgp keys trusted by apt.
See examples/apt-source.yaml for examples and explanation of the individual capabilities.

Along the actual features a huge list of related unit- and vm-tests is added.

Description of the change

Adding apt_source features for the install stage of curtin.
This is inherited from cloud-init, but simplified (as we don't have to care about some features and old compatibility).
And then added plenty of unit- and vm-tests for the functionality.

To post a comment you must log in.
review: Needs Fixing (continuous-integration)
442. By Christian Ehrhardt  on 2016-06-01

test_mir_apt_list_rename fix assertion of call

depending on the unittest environment the rename calls can differ in order
and amount. Convert the check to any_call

443. By Christian Ehrhardt  on 2016-06-01

test_mir_apt_list_rename mockup source filename

Depending on the unittest environment the source file could be wrong.
E.g. if a developer runs with a non default mirror.
Mock up the check for this file and by that, fixup the test in various
environments.

444. By Christian Ehrhardt  on 2016-06-01

fix premission error in unit test

depending on the environment it could have had the side effect of really trying
to rename files. That was bad for either the permission error or the actual
rename.
So we mock the rename to keep it free of side effects.

445. By Christian Ehrhardt  on 2016-06-01

drop reinstalling the mocked subp as it is not needed anymore

In the simplified test subp is not mocked to begin with, so there is no reason
to restore it.

446. By Christian Ehrhardt  on 2016-06-01

convert all apt_custom_sources_list tests to fakerelease

That makes them independent to the execution environment

review: Approve (continuous-integration)
Christian Ehrhardt  (paelzer) wrote :

The jenkins bot had identified a few issues with the unittests in Trusty environments which are now fixed.

Ryan Harper (raharper) wrote :

This is really nice.

A couple of nits in the code.

Super happy to see unittests AND vmtests +1

One last request, might be a doc/topics/apt_source.rst

I think most of it can be copied out of your example files which included lots of
documentation, but in general, I'd like to see some documentation that we'd hand
to the MAAS team about how to use the new dictionary config to pass through the
landscape derived repo.

review: Needs Fixing
Christian Ehrhardt  (paelzer) wrote :

Hi,
thanks again for the review.

I added some comments to your feedback where appropriate.
ALmost all is accepted and integrated, just sometimes in slightly modified ways and the comments state the reasons.
Any non commented feedback is acked and integrated as suggested.

My own unit and vmtests are fine still, I'll run a full suite of vmtests and if successful push to the MP for the next round of review or acceptance.

447. By Christian Ehrhardt  on 2016-06-02

clarify docstring for mirror2lists_fileprefix

448. By Christian Ehrhardt  on 2016-06-02

make lists_d a global constant in the form of APT_LISTS

449. By Christian Ehrhardt  on 2016-06-02

fix whitespace issue

450. By Christian Ehrhardt  on 2016-06-02

fix docstrings to always use triple quotes

451. By Christian Ehrhardt  on 2016-06-02

degrade test_apt_srcl_custom to static method

452. By Christian Ehrhardt  on 2016-06-02

handle potential exception on os.rename

453. By Christian Ehrhardt  on 2016-06-02

replace get_release usage with already existing lsb_release

454. By Christian Ehrhardt  on 2016-06-02

add sanity checks and early exit to render_string_to_file

455. By Christian Ehrhardt  on 2016-06-02

move BASIC_MATCHER up to global variables

456. By Christian Ehrhardt  on 2016-06-02

improve error message if add_key_raw fails

457. By Christian Ehrhardt  on 2016-06-02

improve docstring of method add_key

458. By Christian Ehrhardt  on 2016-06-02

make default mirror and keyserver global variables

459. By Christian Ehrhardt  on 2016-06-02

add unittest for specifying a custom keyserver

460. By Christian Ehrhardt  on 2016-06-02

simplify find_apt_mirror_info by using DEFAULT_MIRRORS

This now shares a single default definition with rename_apt_lists

461. By Christian Ehrhardt  on 2016-06-02

dropping unrelated changes

FYI - rharper will carry those in his docu update branch

462. By Christian Ehrhardt  on 2016-06-02

optimize apt_source unittest imports

463. By Christian Ehrhardt  on 2016-06-02

apt_source: add doc in rst format and improve example yaml

review: Approve (continuous-integration)
464. By Christian Ehrhardt  on 2016-06-02

improve rst formatting

Christian Ehrhardt  (paelzer) wrote :

Ok, I found an rst format checker and was able to improve that as well.
Should be available for another round of review towards acceptance now.

review: Approve (continuous-integration)
Ryan Harper (raharper) :
Christian Ehrhardt  (paelzer) wrote :

Hi,
thanks again for all the feedback.
Every section not explicitly mentiones is acked and integrated.

For everything else see the comment why/how I have modified the suggestions.

Running a full round of tests again before next upload.

465. By Christian Ehrhardt  on 2016-06-03

convert gpg key handling to pure python

466. By Christian Ehrhardt  on 2016-06-03

rename mirror2lists_fileprefix to mirrorurl_to_apt_fileprefix

This much more reflects what it does, so it helps code readability

467. By Christian Ehrhardt  on 2016-06-03

include exception info on the OSError exception for file renaming

468. By Christian Ehrhardt  on 2016-06-03

move generic gpg functions to util

469. By Christian Ehrhardt  on 2016-06-03

move generic part of template rendering to util

470. By Christian Ehrhardt  on 2016-06-03

add apt name portion to name add_key[_raw]

471. By Christian Ehrhardt  on 2016-06-03

backup sources.list before writing to it

472. By Christian Ehrhardt  on 2016-06-03

rename add_sources to the more appropriate add_apt_sources

473. By Christian Ehrhardt  on 2016-06-03

simply control-flow in apt_source

474. By Christian Ehrhardt  on 2016-06-03

simplify error handling in handle_apt_source

475. By Christian Ehrhardt  on 2016-06-03

remove unused import

review: Approve (continuous-integration)
476. By Christian Ehrhardt  on 2016-06-06

unify key tests for real keys in apt_src_keyid_real

477. By Christian Ehrhardt  on 2016-06-06

fix gpg key based unitests to work in networkless environments as well

478. By Christian Ehrhardt  on 2016-06-06

add test for alternative keyserver

479. By Christian Ehrhardt  on 2016-06-06

capture output of gpg calls to avoid messing up stdout/stderr

480. By Christian Ehrhardt  on 2016-06-06

fix fallback key import

481. By Christian Ehrhardt  on 2016-06-07

drop gpg activity from apt-source key tests

While the unittest gets a bit less real by that change, it will work in
protected environment (e.g. sbuild) and leave the developers .gpg keyring

Christian Ehrhardt  (paelzer) wrote :

This now also includes some extensions/fixes created and discussed on the cloud-init side of things.
There is no reason to wait and find them again in the curtin code.

I could build fine in sbuild, but was unable to test in a ppa for now (gets rejected for bin/source mismatch for some reason).

I started to mess around with the changes file trying to fix it, but I guess smoser has a much nicer and faster way to throw that at a ppa - I'd be happy about a discussion what was missing.

Ready for another round of review for getting merged.

review: Approve (continuous-integration)
Ryan Harper (raharper) wrote :

This is really nice. Just a couple of minor issues with naming and logging.

review: Needs Fixing
482. By Christian Ehrhardt  on 2016-06-07

improve debug message

483. By Christian Ehrhardt  on 2016-06-07

report stacktrace on high level exception handler

484. By Christian Ehrhardt  on 2016-06-07

typo fix fule->file

485. By Christian Ehrhardt  on 2016-06-07

lower debug level of renaming apt list message

486. By Christian Ehrhardt  on 2016-06-07

fix exception handler that had a unreferenced %s

487. By Christian Ehrhardt  on 2016-06-07

prefix getkeybyid with gpg_ as the other related functions

488. By Christian Ehrhardt  on 2016-06-07

rename basic_render to basic_template_render for readability

Christian Ehrhardt  (paelzer) wrote :

All integrated, except one that I discussed on IRC

Christian Ehrhardt  (paelzer) wrote :

Ready for another review or acceptance

review: Approve (continuous-integration)
489. By Christian Ehrhardt  on 2016-06-09

improve error handling of gpg functions

Christian Ehrhardt  (paelzer) wrote :

Added another fix that came up when working on the cloud-init portion.
Ready for review towards merging again.

review: Approve (continuous-integration)
Ryan Harper (raharper) wrote :

Looks great. Should get this merged today or tomorrow.

review: Approve
Scott Moser (smoser) wrote :

mostly nitpicks
some to make it like what is in cloud-init trunk.

Christian Ehrhardt  (paelzer) wrote :

Hi,
thanks for the review - all accepted and integrated except one.
I commented below and catch you on IRC for that one.

490. By Christian Ehrhardt  on 2016-06-14

move docsting after copyright

491. By Christian Ehrhardt  on 2016-06-14

move gpg functions to their own file

492. By Christian Ehrhardt  on 2016-06-14

drop unused self.orig_gpg_recv_key

493. By Christian Ehrhardt  on 2016-06-14

add curtin/gpg.py

494. By Christian Ehrhardt  on 2016-06-14

drop rstrip from gpg function

Christian Ehrhardt  (paelzer) wrote :

That is the integration of the last feedback, ready for review/merge again

review: Approve (continuous-integration)
Scott Moser (smoser) wrote :

further reading and thinking.
I think that the implementation doens't actually solve the thing we need solved.

apt_sources configuration to curtin declares what the target's apt should be configured with (much the same as the storage_config or 'kernel' configuration is about the installed environment not the host).

So, the apt_sources needs to apply to the target.
For maas, to get all of this working together, it has to
a.) declare to cloud-init that certain apt sources are necessary in the ephemeral environment
b.) declare to curtin that the target environment must include apt configured with a set of apt sources.

'a' is done in cloud-init user-data
'b' is done in curtin config.

Christian Ehrhardt  (paelzer) wrote :

Hi Scott,
I've implemented it the "new" way now and adapted all around that was needed.
Please read the comments I added below as answer to your feedback - to see if my assumptions when implementing made sense.

Everything else in the next round of code review (just doing a full round of tests and build before to be sure).

495. By Christian Ehrhardt  on 2016-06-16

Merge with upstream to avoid conflicts

pending merges:
  Ryan Harper 2016-06-14 [merge] reporting: set webhook handler level to DEBUG, no filtering
    Ryan Harper 2016-06-14 [merge] merge from trunk
    Ryan Harper 2016-06-14 reporting: default webhook handler level to DEBUG, no filtering
    Scott Moser 2016-06-14 [merge] improve net-meta network configuration
    Scott Moser 2016-06-13 spelling
    Scott Moser 2016-06-13 log cleanup
    Scott Moser 2016-06-13 improve net-meta network configuration
    Ryan Harper 2016-06-03 [merge] curtin/net: fix inet value for subnets, don't add interface attributes to alias
    Ryan Harper 2016-06-03 [merge] curtin/net: calculate the inet value for each subnet independently
    Mike Pontillo 2016-06-03 Address review comment.
    Mike Pontillo 2016-06-02 Don't append the '6' to IPv4 aliases if an IPv6 alias preceded it.
    Mike Pontillo 2016-06-02 Add failing test case for bug #1588547
    Ryan Harper 2016-06-03 [merge] curtin/net: Don't add interface attributes to alias interfaces
    Mike Pontillo 2016-06-03 Don't add additional parameters to alias interfaces. (Rebase from trunk and address review comments.)

496. By Christian Ehrhardt  on 2016-06-16

add docstring to apt_source.py's POPULATE_SUBCMD

497. By Christian Ehrhardt  on 2016-06-16

avoid python checker warnings by making POPULATE_SUBCMD lower case

498. By Christian Ehrhardt  on 2016-06-16

add --target parameter to apt-source command

This allows to modify the target environment as needed.
But given it is an argment can work in arbitrary chroots as long as the tools
like gpg, apt-key as well as the files it modifies are there.

499. By Christian Ehrhardt  on 2016-06-16

adapt apt-source tests to check the target environment

500. By Christian Ehrhardt  on 2016-06-16

report checked files in output_files_exist

This can help to quickly identify which file is missing in case a long list
of files is checked.

501. By Christian Ehrhardt  on 2016-06-16

lsb_release to support target

502. By Christian Ehrhardt  on 2016-06-16

propagate and use target throughout apt-source handling

503. By Christian Ehrhardt  on 2016-06-16

fix issue in ChrootableTarget if resolv.conf didn't yet exist in target

504. By Christian Ehrhardt  on 2016-06-16

ensure errorlist contains only strings

505. By Christian Ehrhardt  on 2016-06-17

improve error handling to not silently ignore program execution issues

506. By Christian Ehrhardt  on 2016-06-17

order config builtin in execution order

507. By Christian Ehrhardt  on 2016-06-17

extend builtin key of CONFIG_BUILTIN

It is a single namespace per stage - just "builtin" is too ambiguous.

508. By Christian Ehrhardt  on 2016-06-17

insert apt-source after extract

509. By Christian Ehrhardt  on 2016-06-17

remove useless warn if no source is specified

This can be a quite common case since we allo keys without surce now.
No need to flood the log with warnings that didn't even tell you much.

510. By Christian Ehrhardt  on 2016-06-17

revert rev 503: fix issue in ChrootableTarget if resolv.conf

511. By Christian Ehrhardt  on 2016-06-17

update apt data after changing configuration and/or source.lists

512. By Christian Ehrhardt  on 2016-06-17

mock out apt_update in apt-source unit tests

513. By Christian Ehrhardt  on 2016-06-20

allow a testcase to provide custom cloud-init data to the booted system

514. By Christian Ehrhardt  on 2016-06-20

apt-source vmtest now uses boot_cloudconf

This allows the vmtest to avoid having cloud-init to rewrite its just laid
down sources list. Just as a final user is expected to set this conf when
using curtin apt-source feature the test is using itnow as well.

515. By Christian Ehrhardt  on 2016-06-20

doc: update for apt-source now having a target specification

516. By Christian Ehrhardt  on 2016-06-20

don't skip cloud-init if we run in preserved mode

517. By Christian Ehrhardt  on 2016-06-20

take back individual builtin naming

518. By Christian Ehrhardt  on 2016-06-20

hook apt-source handling at curthooks

519. By Christian Ehrhardt  on 2016-06-20

doc: update new placement of apt-source in curthooks

520. By Christian Ehrhardt  on 2016-06-20

harden apt-source unittests for more test environments

In some environments the lower levels run a few extra udevadm settle.
This makes the unittests resilent against those slightly different, but
acceptable changes in behavior.

Christian Ehrhardt  (paelzer) wrote :

Ran full testsuite, tested build in sbuild and on ppa.
Ready for review and inclusion again.

review: Approve (continuous-integration)
521. By Christian Ehrhardt  on 2016-06-23

remove gpg_ prefix not that related functions are in a gpg.py

522. By Christian Ehrhardt  on 2016-06-24

drop builtin sources.list

In case there is no custom source list specified we will fall back to modify
the one we find on the given target. That is done with as few dependencies to
the source.list content as possible, to avoid issues in case that changes.
If expected content isn't found curtin will warn about it but not fail.
That means we have time to adapt/fix in case of changes and in the worst case
one can always fix things with a custom template.

523. By Christian Ehrhardt  on 2016-06-24

drop unused method load_tfile_or_url in tests

524. By Christian Ehrhardt  on 2016-06-24

adapt documentation and examples

Christian Ehrhardt  (paelzer) wrote :

Integrated the changes due to our discussion about having no "builtin" source.list.

If no custom template is provided we now fall back to modify whatever we find in the target.
All tests that should be affected passed already as well as an sbuild.

I'm running a ppa build and a full test-suite just to be sure and give this thread another ping then.

review: Approve (continuous-integration)
Christian Ehrhardt  (paelzer) wrote :

All tests and ppa build went well, ready for re-review and inclusion.

525. By Christian Ehrhardt  on 2016-07-13

merge with latest trunk to avoid later merge conflicts

526. By Christian Ehrhardt  on 2016-07-13

restore populate_subcmd to old coding style, ignoring pylint complaint

This should keep the MP free of unrelated changes

527. By Christian Ehrhardt  on 2016-07-13

remove unrelated change in curtin/commands/install.py

That was needed/intended when we still wired it up as extra builtin command.
But no more now that we move to curthooks integration.

528. By Christian Ehrhardt  on 2016-07-13

remove stale merge artifacts

529. By Christian Ehrhardt  on 2016-07-13

integrate apt in curthooks

This makes the apt feature run by curthooks (but a noop of no config provided).
And additionally keeps the standalone command for other configs/targets.

530. By Christian Ehrhardt  on 2016-07-13

fix apt handling if no config is provided

531. By Christian Ehrhardt  on 2016-07-13

fix debconf if only provided for unconfigurable packages

532. By Christian Ehrhardt  on 2016-07-13

log apt_proxy configuration

533. By Christian Ehrhardt  on 2016-07-13

fix apt conf to match upper/lower case of apt-config

534. By Christian Ehrhardt  on 2016-07-13

add Testcases for old curtin apt features

535. By Christian Ehrhardt  on 2016-07-13

match aptproxy config path as specified

536. By Christian Ehrhardt  on 2016-07-13

translate old curtin apt features to new format

This includes (re-)moving of old curtin code.
All apt related configuration is handled in apt.py now.
In a config that specified old&new content it fails with a proper error
message as they are mutually exclusive (curtin can't decide which one to keep).

537. By Christian Ehrhardt  on 2016-07-13

modify existing functionality and tests to follow the new apt config format

538. By Christian Ehrhardt  on 2016-07-13

improve mirror and source_list handling

Replace the full string found in config, set the full specified string.
This gets rid of // or no / in testcases.
Set the mirror to what was specified, that is what has to be done.
Also fixup some references to the old custom_source_list to source_list

539. By Christian Ehrhardt  on 2016-07-14

add search and search_dns features

This inherits the cloud-init capabilities for these two features.
But it follows the new apt config format with split options for
primary/secondary for that as well.
Code could be simplified and unified a lot so that despite the extra split
primary/secondary it is mre readable now.
As discussed some fallbacks of cloud-init that rely on cloud-datasource are
dropped in the curtin handling of that configuration.

540. By Christian Ehrhardt  on 2016-07-14

add unittest test_mirror_search

541. By Christian Ehrhardt  on 2016-07-14

add unittest test_mirror_search_dns

542. By Christian Ehrhardt  on 2016-07-14

add unittests test_mirror_search_many2 and test_mirror_search_many3

543. By Christian Ehrhardt  on 2016-07-14

drop second subcheck that is already covered by test_mirror_search_many3

Would require some extra mocking as now search is hit for dns and non_dns cases.
Since it is covered already drop that part.

544. By Scott Moser on 2016-07-15

list feature APT_CONFIG_V1

545. By Christian Ehrhardt  on 2016-07-18

fixed urlparse call path

546. By Christian Ehrhardt  on 2016-07-18

fix py2/py3 compatibility import of urlparse

547. By Christian Ehrhardt  on 2016-07-18

add unittests for is_resolvable_url

548. By Christian Ehrhardt  on 2016-07-18

add vmtest for mirror_search features

549. By Christian Ehrhardt  on 2016-07-18

make dns search resilent against fqdn not being avavilable

550. By Christian Ehrhardt  on 2016-07-18

add vmtest for dns_search feature

551. By Christian Ehrhardt  on 2016-07-18

adapt example to cover search and search_dns features

552. By Christian Ehrhardt  on 2016-07-19

move primary/security mirror specifications into a dict - code

553. By Christian Ehrhardt  on 2016-07-19

move primary/security mirror specifications into a dict - examples

554. By Christian Ehrhardt  on 2016-07-19

move primary/security mirror specifications into a dict - testcases

555. By Christian Ehrhardt  on 2016-07-19

fix TestAptSrcModify reference to test yaml

556. By Christian Ehrhardt  on 2016-07-19

make mirror specs per acrhitecture - code

557. By Christian Ehrhardt  on 2016-07-19

make mirror specs per acrhitecture - examples

558. By Christian Ehrhardt  on 2016-07-19

make mirror specs per acrhitecture - testcases

559. By Christian Ehrhardt  on 2016-07-19

arches and repositioned mirrors adapted in doc

560. By Christian Ehrhardt  on 2016-07-19

add unittest test_mirror

561. By Christian Ehrhardt  on 2016-07-19

add unittest test_mirror_default

562. By Christian Ehrhardt  on 2016-07-19

add unittest test_mirror_arches

563. By Christian Ehrhardt  on 2016-07-19

add unittest test_mirror_arches_default

564. By Christian Ehrhardt  on 2016-07-19

add unittest test_mirror_arches_sysdefault

565. By Christian Ehrhardt  on 2016-07-19

add vmtest TestAptSrcModifyArches

566. By Christian Ehrhardt  on 2016-07-20

make replaced archive arch dependent

567. By Christian Ehrhardt  on 2016-07-20

make tetsing old apt features work cross platforms

568. By Christian Ehrhardt  on 2016-07-20

make use of new arch-dependent mirrors in unittests

This should also make the tests more likely to work on non amd64 architectures.

569. By Christian Ehrhardt  on 2016-07-20

make find apt mirror info tests work on ubuntu ports

570. By Christian Ehrhardt  on 2016-07-20

disable_pockets feature - code

571. By Christian Ehrhardt  on 2016-07-20

disable_pockets feature - example

572. By Christian Ehrhardt  on 2016-07-20

disable_pockets feature - unittest

573. By Christian Ehrhardt  on 2016-07-20

disable_pockets feature - vmtest

574. By Christian Ehrhardt  on 2016-07-20

update example in rst doc with disable_pockets feature

575. By Christian Ehrhardt  on 2016-07-20

update docu with some more use cases

576. By Christian Ehrhardt  on 2016-07-20

adapt old apt feature entry path to new mirrr config

Christian Ehrhardt  (paelzer) wrote :

Ready for review - curtin implementation of the new definition as in the gdoc.
I need to update the commit message thou ..

review: Needs Fixing (continuous-integration)
review: Needs Fixing (continuous-integration)
Christian Ehrhardt  (paelzer) wrote :

That seems to be stuff that only triggers in an environment even more
restricted than I have tested on horse.
I'll try to fix and push over the day, ...

Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

On Wed, Jul 20, 2016 at 8:08 PM, Server Team CI bot <
<email address hidden>> wrote:

> Review: Needs Fixing continuous-integration
>
> FAILED: Continuous integration, rev:576
> https://server-team-jenkins.canonical.com/job/curtin-ci/318/
> Executed test runs:
> None:
> https://server-team-jenkins.canonical.com/job/generic-update-mp/332/console
>
> Click here to trigger a rebuild:
> https://server-team-jenkins.canonical.com/job/curtin-ci/318/rebuild
>
> --
>
> https://code.launchpad.net/~paelzer/curtin/bug-1574113-derived-repositories/+merge/296118
> You are the owner of lp:~paelzer/curtin/bug-1574113-derived-repositories.
>

Christian Ehrhardt  (paelzer) wrote :

Hmm,
seems to be about mocking out an object that it hard fetched not from mock
context or so.
What might be interesting for CI in general is that I can reproduce with:
"nosetests tests/unittests" (as called by jenkins), while "make check" runs
just fine.

Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

On Thu, Jul 21, 2016 at 11:16 AM, Christian Ehrhardt <
<email address hidden>> wrote:

> That seems to be stuff that only triggers in an environment even more
> restricted than I have tested on horse.
> I'll try to fix and push over the day, ...
>
> Christian Ehrhardt
> Software Engineer, Ubuntu Server
> Canonical Ltd
>
> On Wed, Jul 20, 2016 at 8:08 PM, Server Team CI bot <
> <email address hidden>> wrote:
>
>> Review: Needs Fixing continuous-integration
>>
>> FAILED: Continuous integration, rev:576
>> https://server-team-jenkins.canonical.com/job/curtin-ci/318/
>> Executed test runs:
>> None:
>> https://server-team-jenkins.canonical.com/job/generic-update-mp/332/console
>>
>> Click here to trigger a rebuild:
>> https://server-team-jenkins.canonical.com/job/curtin-ci/318/rebuild
>>
>> --
>>
>> https://code.launchpad.net/~paelzer/curtin/bug-1574113-derived-repositories/+merge/296118
>> You are the owner of lp:~paelzer/curtin/bug-1574113-derived-repositories.
>>
>
>

Christian Ehrhardt  (paelzer) wrote :

The magic switch seems to be "--nologcapture" which is kind of unexpected
...
With it enabled things work, without it seems that the mocked call to
get_mirror like:
pmirror = get_mirror(cfg, "primary", arch)
Gets "mocked wrong"

Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

On Thu, Jul 21, 2016 at 11:24 AM, Christian Ehrhardt <
<email address hidden>> wrote:

> Hmm,
> seems to be about mocking out an object that it hard fetched not from mock
> context or so.
> What might be interesting for CI in general is that I can reproduce with:
> "nosetests tests/unittests" (as called by jenkins), while "make check" runs
> just fine.
>
> Christian Ehrhardt
> Software Engineer, Ubuntu Server
> Canonical Ltd
>
> On Thu, Jul 21, 2016 at 11:16 AM, Christian Ehrhardt <
> <email address hidden>> wrote:
>
>> That seems to be stuff that only triggers in an environment even more
>> restricted than I have tested on horse.
>> I'll try to fix and push over the day, ...
>>
>> Christian Ehrhardt
>> Software Engineer, Ubuntu Server
>> Canonical Ltd
>>
>> On Wed, Jul 20, 2016 at 8:08 PM, Server Team CI bot <
>> <email address hidden>> wrote:
>>
>>> Review: Needs Fixing continuous-integration
>>>
>>> FAILED: Continuous integration, rev:576
>>> https://server-team-jenkins.canonical.com/job/curtin-ci/318/
>>> Executed test runs:
>>> None:
>>> https://server-team-jenkins.canonical.com/job/generic-update-mp/332/console
>>>
>>> Click here to trigger a rebuild:
>>> https://server-team-jenkins.canonical.com/job/curtin-ci/318/rebuild
>>>
>>> --
>>>
>>> https://code.launchpad.net/~paelzer/curtin/bug-1574113-derived-repositories/+merge/296118
>>> You are the owner of lp:~paelzer/curtin/bug-1574113-derived-repositories.
>>>
>>
>>
>

Christian Ehrhardt  (paelzer) wrote :

Ok, things make sense now.
Mocking without retvale sets pmirror to the mock.
That without disabled log leads to "LOG.info("got primary mirror: %s",
pmirror)" be a constructor call.

Ok, that should be fixable ...

Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

On Thu, Jul 21, 2016 at 11:54 AM, Christian Ehrhardt <
<email address hidden>> wrote:

> The magic switch seems to be "--nologcapture" which is kind of unexpected
> ...
> With it enabled things work, without it seems that the mocked call to
> get_mirror like:
> pmirror = get_mirror(cfg, "primary", arch)
> Gets "mocked wrong"
>
> Christian Ehrhardt
> Software Engineer, Ubuntu Server
> Canonical Ltd
>
> On Thu, Jul 21, 2016 at 11:24 AM, Christian Ehrhardt <
> <email address hidden>> wrote:
>
>> Hmm,
>> seems to be about mocking out an object that it hard fetched not from
>> mock context or so.
>> What might be interesting for CI in general is that I can reproduce with:
>> "nosetests tests/unittests" (as called by jenkins), while "make check" runs
>> just fine.
>>
>> Christian Ehrhardt
>> Software Engineer, Ubuntu Server
>> Canonical Ltd
>>
>> On Thu, Jul 21, 2016 at 11:16 AM, Christian Ehrhardt <
>> <email address hidden>> wrote:
>>
>>> That seems to be stuff that only triggers in an environment even more
>>> restricted than I have tested on horse.
>>> I'll try to fix and push over the day, ...
>>>
>>> Christian Ehrhardt
>>> Software Engineer, Ubuntu Server
>>> Canonical Ltd
>>>
>>> On Wed, Jul 20, 2016 at 8:08 PM, Server Team CI bot <
>>> <email address hidden>> wrote:
>>>
>>>> Review: Needs Fixing continuous-integration
>>>>
>>>> FAILED: Continuous integration, rev:576
>>>> https://server-team-jenkins.canonical.com/job/curtin-ci/318/
>>>> Executed test runs:
>>>> None:
>>>> https://server-team-jenkins.canonical.com/job/generic-update-mp/332/console
>>>>
>>>> Click here to trigger a rebuild:
>>>> https://server-team-jenkins.canonical.com/job/curtin-ci/318/rebuild
>>>>
>>>> --
>>>>
>>>> https://code.launchpad.net/~paelzer/curtin/bug-1574113-derived-repositories/+merge/296118
>>>> You are the owner of
>>>> lp:~paelzer/curtin/bug-1574113-derived-repositories.
>>>>
>>>
>>>
>>
>

577. By Christian Ehrhardt  on 2016-07-21

make metohd of apt commandline handling less ambiguous

578. By Christian Ehrhardt  on 2016-07-21

add docstring to clean_cloud_init

579. By Christian Ehrhardt  on 2016-07-21

follow the "logging-not-lazy" warning of checkers and change string extension

580. By Christian Ehrhardt  on 2016-07-21

fix unittests for cases without --nologcapture

review: Needs Fixing (continuous-integration)
Christian Ehrhardt  (paelzer) wrote :

Hmm, that is tox now with a special case on urllib which changed py2->py3.
General question - should we make a tox run part of make check ?

581. By Christian Ehrhardt  on 2016-07-21

avoid py2 pylint issue due to renamed urllib import

review: Approve (continuous-integration)
review: Approve (continuous-integration)
Scott Moser (smoser) wrote :

Started a vmtest run at
https://server-team-jenkins.canonical.com/job/curtin-vmtest-venonat-devel/33/

I dont think we have 'preserve_sources_list' implemented as intended.

Our doc says:
|  preserve_sources_list: <true|false|keep>
|    # For cloud-init this defaults to false. By default, cloud-init renders
|    # /etc/apt/sources.list from its provided templates on first boot.
|    # For Curtin this defaults to false. By default, curtin will always attempt to
|    # replace mirrors inside the sources.list it finds in the image.
|    # Curtin will configure cloud-init in the target via
|    # /etc/cloud/cloud.cfg.d/curtin-preserve-sources.list with True,
|    # which causes cloud-init to not touch /etc/apt/sources.list by default.
|    # Unless explicitly  requested by cloud-init userdata (overwriting to False).
|    # Effectively this changes the default from False to True

I thought that in discussion you and I had agreed that this was in fact 'boolean', and that curtin would *always* replace /etc/apt/sources.list . I'm not terribly opposed to a tri-state, where it can be told to leave it untouched though.

That said, i do not see where curtin is configuring cloud-init in the target to set. curtin should always set cloud-init to preserve sources.list unless there is another use case that i'm missing.

I'm not done with the review yet, but wanted to hit 'submit' before losing context and losing what i had done.

Scott Moser (smoser) wrote :

for disable_pockets, it looks to me like the string provided always has the release prepended.
for example:
 disable_pockets = [updates]
will disable 'trusty-updates'.

that makes sense.
but for:
  disable_pockets = ["mypocket", "unstable"]
this doesnt make so much sense.
I dont know how easily to do this other than special casing the words:
 updates, release, security, backports

also, might want to just check that this content from my /etc/apt/sources.list.d/google-chrome.list works as expected:
 deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main

Scott Moser (smoser) wrote :

I thought that 'uri' would override 'search', does it?

Christian Ehrhardt  (paelzer) wrote :

> Started a vmtest run at
> https://server-team-jenkins.canonical.com/job/curtin-vmtest-venonat-devel/33/

Thanks, that one worked at least.

> I dont think we have 'preserve_sources_list' implemented as intended.

Yeah the tristate was dropped in our discussion yet the doc needs update (DONE now).
The placing of the config for cloud-init was missing (I missed to add to our todo's after the discussion).
Pushed in the next update to the MP.

Also all vmtests that take care to check.
So far I provided "preserve_sources_list" via the vmtest, now that it is implemented I dropped it there and it is still working the same way.

It seems I need something to make cloud-init to pick up the config thou - will contact you later today as I guess that is talked in 3 or searched in 15 minutes :-)

Christian Ehrhardt  (paelzer) wrote :

> I thought that 'uri' would override 'search', does it?

Yes it does:
see curtin/commands/apt.py:get_mirror

1. uri
2. search
3. search_dns

As soon as one of 1-3 hits the others aren't checked anymore.

Also see the following unittests that ensure this is true:
test_mirror_search_many3 (uri beats search & search_dns)
test_mirror_search_many2 (search beats search_dns)

Christian Ehrhardt  (paelzer) wrote :

> for disable_pockets, it looks to me like the string provided always has the
> release prepended.
> for example:
> disable_pockets = [updates]
> will disable 'trusty-updates'.
>
> that makes sense.
> but for:
> disable_pockets = ["mypocket", "unstable"]
> this doesnt make so much sense.
> I dont know how easily to do this other than special casing the words:
> updates, release, security, backports

I didn't think of "custom" pockets yet, but I see no other easy way than "special casing" them.

> also, might want to just check that this content from my
> /etc/apt/sources.list.d/google-chrome.list works as expected:
> deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main

Hrm, yeah man sources.list states this (options) in the section "ONE-LINE-STYLE FORMAT".
I need to adapt to work with them as well.
Although it is highly unlikely to find some in a source.list (disabling doesn't care about other files in sources.list.d) these days we should look ahead and be prepared.
Working on it ...

Christian Ehrhardt  (paelzer) wrote :

> > I dont know how easily to do this other than special casing the words:
> > updates, release, security, backports
>
> I didn't think of "custom" pockets yet, but I see no other easy way than
> "special casing" them.

I was missing an "... either" here.

But giving it some thought - if we go as far as to allow custom pocket names.
We need to cover custom pocket names that match the known names.
So what if someone has a "updates" pocket and it shall not become xenial-updates.

We already have a way to specify these and I think we should stick to that.
We can allow $RELEASE in there and replace it, just as we do when rendering sources.list in the first place.

That allows one to differentiate between: "$RELEASE-updates" and "updates"

That also covers the sepcial case of the main release pocket being "".
One can just specify "$RELEASE" and gets e.g. "xenial" which is just what we need in that case as well.

Christian Ehrhardt  (paelzer) wrote :

Reading more into it we should think about:
- what we called "pockets" is actually defined as "suite" - should we rename our option?
- long term (not for this bug & MP) think about supporting deb822 format as well

582. By Christian Ehrhardt  on 2016-07-25

leave cloud-init config to prevent clobbering sources.list

583. By Christian Ehrhardt  on 2016-07-25

add $RELEASE replacement to disable_pockets

584. By Christian Ehrhardt  on 2016-07-25

improve doc wording

585. By Christian Ehrhardt  on 2016-07-25

check for old apt_preserve_list config as we select th eold for compatibility

586. By Christian Ehrhardt  on 2016-07-25

fix typo

587. By Christian Ehrhardt  on 2016-07-25

support source.list lines with options

588. By Christian Ehrhardt  on 2016-07-25

cover options in source.list lines in unittests

589. By Christian Ehrhardt  on 2016-07-25

fix test_old_apt_features vmtest

590. By Christian Ehrhardt  on 2016-07-25

switch apt_preserve_sources_list to True

Now that all tests have succeeded we switch the apt_preserve_sources_list
that is passed to cloud-init to True to keep curtin rendered sources.list
intact unless overwritten by user data.

591. By Christian Ehrhardt  on 2016-07-25

rename pocket to suite following man sources.list

592. By Christian Ehrhardt  on 2016-07-25

rename curtin-preserve-sources.list to curtin-preserve-sources.cfg

The example in the gdoc was uncommon, clout-init usually has .cfg files

593. By Christian Ehrhardt  on 2016-07-25

rename vmtets yaml following the pockets to suite rename

594. By Christian Ehrhardt  on 2016-07-25

fix preserve check in vmtest TestAptSrcPreserve

In this path the preserving of the target is not triggered, so the respective
checks need to be adapted.

Christian Ehrhardt  (paelzer) wrote :

All Feedback handled and a bit more (according to my comments to the review feedback before).
Ready again.

review: Needs Fixing (continuous-integration)
review: Needs Fixing (continuous-integration)
595. By Christian Ehrhardt  on 2016-07-25

make unittests runnable as non-privileged user in tox

596. By Christian Ehrhardt  on 2016-07-25

let curtin exit with failure if protecting sources.list didn't work

597. By Christian Ehrhardt  on 2016-07-25

make trusty pep8 happy which even complains about raw string length

Christian Ehrhardt  (paelzer) wrote :

de-toxed the unitests for disable_suites

review: Approve (continuous-integration)
review: Approve (continuous-integration)
Scott Moser (smoser) wrote :

> > > I dont know how easily to do this other than special casing the words:
> > > updates, release, security, backports
> >
> > I didn't think of "custom" pockets yet, but I see no other easy way than
> > "special casing" them.
>
> I was missing an "... either" here.
>
> But giving it some thought - if we go as far as to allow custom pocket names.
> We need to cover custom pocket names that match the known names.
> So what if someone has a "updates" pocket and it shall not become xenial-
> updates.
>
> We already have a way to specify these and I think we should stick to that.
> We can allow $RELEASE in there and replace it, just as we do when rendering
> sources.list in the first place.
>
> That allows one to differentiate between: "$RELEASE-updates" and "updates"
>
> That also covers the sepcial case of the main release pocket being "".
> One can just specify "$RELEASE" and gets e.g. "xenial" which is just what we
> need in that case as well.

I'd really prefer to handle let the user specify 'updates', 'release' literally rather than '$RELEASE-updates'.

that does mean that these "well known names" block the user from using pockets literally named 'release'. I usually dont like such holes in design, but i think it might be worth it here for readability. The pocket names 'release', 'updates', 'proposed' and 'backports' are well known in Ubuntu (example https://wiki.ubuntu.com/SecurityTeam/FAQ).

given the same url there, i use that for justification of the name 'pockets'. If you'd like we can also support 'suites', but i think pocket is fine.

Scott Moser (smoser) wrote :

I think you must not have seen my in-line comments in my review that starts with
'for disable_pockets, '

598. By Christian Ehrhardt  on 2016-07-26

auto extend known ubunut suites by $RELEASE-

599. By Christian Ehrhardt  on 2016-07-26

reduce default verbosity of apt feature by moving some info to debug

600. By Christian Ehrhardt  on 2016-07-26

make old apt features less verbose

601. By Christian Ehrhardt  on 2016-07-26

move old apt feature config conversion into own method and out of curthooks

602. By Christian Ehrhardt  on 2016-07-26

rework apt standalone command entry path

This is dropping a lot of old artifacts from the command entry path.

603. By Christian Ehrhardt  on 2016-07-26

minimal commandline call test for apt features

604. By Christian Ehrhardt  on 2016-07-26

add spaced around string concat plus operators

605. By Christian Ehrhardt  on 2016-07-26

merge rev 400: move uefi boot knowledge from launch and vmtest to xkvm

Christian Ehrhardt  (paelzer) wrote :

All we discussed yesterday night is in now, ready for review/merge

606. By Christian Ehrhardt  on 2016-07-26

fix config of ChrootableTarget for apt command

607. By Christian Ehrhardt  on 2016-07-26

adapt standalong command example to new invocation

608. By Christian Ehrhardt  on 2016-07-26

change standalong apt command test to be vmtest based

609. By Christian Ehrhardt  on 2016-07-26

rename apt to apt_config for stanalone command

That implies to rename the module name as well.

610. By Christian Ehrhardt  on 2016-07-26

change apt to apt-config in vmtest

611. By Christian Ehrhardt  on 2016-07-26

change ppa's and keys used to something under our control

612. By Christian Ehrhardt  on 2016-07-26

change the preserve_sources_list default to True in curtin

613. By Christian Ehrhardt  on 2016-07-26

rework testing of standalone command

614. By Christian Ehrhardt  on 2016-07-26

make copy in testcases release independent

615. By Christian Ehrhardt  on 2016-07-26

add fallback if user did not add .list to source keys

Includes modification to one of the testcastes to cover that.

616. By Christian Ehrhardt  on 2016-07-26

fixup test to check for content added by apt-add-repository

Christian Ehrhardt  (paelzer) wrote :

Since we co-work on a branch that we can jointly push to now there is a new MP at https://code.launchpad.net/~ubuntu-server/curtin/bug-1574113-derived-repositories/+merge/301362

This old MP here is kept open to have the discussion history available (not sure if it would go away when rejecting the old MP).

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'curtin/__init__.py'
2--- curtin/__init__.py 2015-10-26 17:35:29 +0000
3+++ curtin/__init__.py 2016-07-26 19:53:43 +0000
4@@ -33,6 +33,8 @@
5 'SUBCOMMAND_SYSTEM_INSTALL',
6 # subcommand 'system-upgrade' is present
7 'SUBCOMMAND_SYSTEM_UPGRADE',
8+ # supports new format of apt configuration
9+ 'APT_CONFIG_V1',
10 ]
11
12 # vi: ts=4 expandtab syntax=python
13
14=== added file 'curtin/commands/apt_config.py'
15--- curtin/commands/apt_config.py 1970-01-01 00:00:00 +0000
16+++ curtin/commands/apt_config.py 2016-07-26 19:53:43 +0000
17@@ -0,0 +1,691 @@
18+# Copyright (C) 2016 Canonical Ltd.
19+#
20+# Author: Christian Ehrhardt <christian.ehrhardt@canonical.com>
21+#
22+# Curtin is free software: you can redistribute it and/or modify it under
23+# the terms of the GNU Affero General Public License as published by the
24+# Free Software Foundation, either version 3 of the License, or (at your
25+# option) any later version.
26+#
27+# Curtin is distributed in the hope that it will be useful, but WITHOUT ANY
28+# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
29+# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for
30+# more details.
31+#
32+# You should have received a copy of the GNU Affero General Public License
33+# along with Curtin. If not, see <http://www.gnu.org/licenses/>.
34+"""
35+apt.py
36+Handling the setup of apt related tasks like proxies, PGP keys, repositories.
37+"""
38+
39+import argparse
40+import glob
41+import os
42+import re
43+import sys
44+import yaml
45+
46+from curtin.log import LOG
47+from curtin import (config, util, gpg)
48+
49+from . import populate_one_subcmd
50+
51+# this will match 'XXX:YYY' (ie, 'cloud-archive:foo' or 'ppa:bar')
52+ADD_APT_REPO_MATCH = r"^[\w-]+:\w"
53+
54+# place where apt stores cached repository data
55+APT_LISTS = "/var/lib/apt/lists"
56+
57+# Files to store proxy information
58+APT_CONFIG_FN = "/etc/apt/apt.conf.d/94curtin-config"
59+APT_PROXY_FN = "/etc/apt/apt.conf.d/90curtin-aptproxy"
60+
61+# Default keyserver to use
62+DEFAULT_KEYSERVER = "keyserver.ubuntu.com"
63+
64+# Default archive mirrors
65+PRIMARY_ARCH_MIRRORS = {"PRIMARY": "http://archive.ubuntu.com/ubuntu/",
66+ "SECURITY": "http://security.ubuntu.com/ubuntu/"}
67+PORTS_MIRRORS = {"PRIMARY": "http://ports.ubuntu.com/ubuntu-ports",
68+ "SECURITY": "http://ports.ubuntu.com/ubuntu-ports"}
69+PRIMARY_ARCHES = ['amd64', 'i386']
70+PORTS_ARCHES = ['s390x', 'arm64', 'armhf', 'powerpc', 'ppc64el']
71+
72+
73+def get_default_mirrors(target=None):
74+ """returns the default mirrors for the target. These depend on the
75+ architecture, for more see:
76+ https://wiki.ubuntu.com/UbuntuDevelopment/PackageArchive#Ports"""
77+ arch = util.get_architecture(target)
78+ if arch in PRIMARY_ARCHES:
79+ return PRIMARY_ARCH_MIRRORS
80+ if arch in PORTS_ARCHES:
81+ return PORTS_MIRRORS
82+ raise ValueError("No default mirror known for arch %s" % arch)
83+
84+
85+def handle_apt(cfg, target):
86+ """ handle_apt
87+ process the config for apt_config. This can be called from
88+ curthooks if a global apt config was provided or via the "apt"
89+ standalone command.
90+ """
91+ release = util.lsb_release(target=target)['codename']
92+ mirrors = find_apt_mirror_info(cfg, target)
93+ LOG.debug("Apt Mirror info: %s", mirrors)
94+
95+ apply_debconf_selections(cfg, target)
96+
97+ if not config.value_as_boolean(cfg.get('preserve_sources_list',
98+ True)):
99+ generate_sources_list(cfg, release, mirrors, target)
100+ rename_apt_lists(mirrors, target)
101+
102+ try:
103+ apply_apt_proxy_config(cfg, target + APT_PROXY_FN,
104+ target + APT_CONFIG_FN)
105+ except (IOError, OSError):
106+ LOG.exception("Failed to apply proxy or apt config info:")
107+
108+ # Process 'apt_source -> sources {dict}'
109+ if 'sources' in cfg:
110+ params = mirrors
111+ params['RELEASE'] = release
112+ params['MIRROR'] = mirrors["MIRROR"]
113+
114+ matcher = None
115+ matchcfg = cfg.get('add_apt_repo_match', ADD_APT_REPO_MATCH)
116+ if matchcfg:
117+ matcher = re.compile(matchcfg).search
118+
119+ add_apt_sources(cfg['sources'], target,
120+ template_params=params, aa_repo_match=matcher)
121+
122+
123+def apply_debconf_selections(cfg, target):
124+ """apply_debconf_selections - push content to debconf"""
125+ # debconf_selections:
126+ # set1: |
127+ # cloud-init cloud-init/datasources multiselect MAAS
128+ # set2: pkg pkg/value string bar
129+ selsets = cfg.get('debconf_selections')
130+ if not selsets:
131+ LOG.debug("debconf_selections was not set in config")
132+ return
133+
134+ # for each entry in selections, chroot and apply them.
135+ # keep a running total of packages we've seen.
136+ pkgs_cfgd = set()
137+ for key, content in selsets.items():
138+ LOG.debug("setting for %s, %s", key, content)
139+ util.subp(['chroot', target, 'debconf-set-selections'],
140+ data=content.encode())
141+ for line in content.splitlines():
142+ if line.startswith("#"):
143+ continue
144+ pkg = re.sub(r"[:\s].*", "", line)
145+ pkgs_cfgd.add(pkg)
146+
147+ pkgs_installed = util.get_installed_packages(target)
148+
149+ LOG.debug("pkgs_cfgd: %s", pkgs_cfgd)
150+ LOG.debug("pkgs_installed: %s", pkgs_installed)
151+ need_reconfig = pkgs_cfgd.intersection(pkgs_installed)
152+
153+ if len(need_reconfig) == 0:
154+ LOG.debug("no need for reconfig")
155+ return
156+
157+ # For any packages that are already installed, but have preseed data
158+ # we populate the debconf database, but the filesystem configuration
159+ # would be preferred on a subsequent dpkg-reconfigure.
160+ # so, what we have to do is "know" information about certain packages
161+ # to unconfigure them.
162+ unhandled = []
163+ to_config = []
164+ for pkg in need_reconfig:
165+ if pkg in CONFIG_CLEANERS:
166+ LOG.debug("unconfiguring %s", pkg)
167+ CONFIG_CLEANERS[pkg](target)
168+ to_config.append(pkg)
169+ else:
170+ unhandled.append(pkg)
171+
172+ if len(unhandled):
173+ LOG.warn("The following packages were installed and preseeded, "
174+ "but cannot be unconfigured: %s", unhandled)
175+
176+ if len(to_config):
177+ util.subp(['chroot', target, 'dpkg-reconfigure',
178+ '--frontend=noninteractive'] +
179+ list(to_config), data=None)
180+
181+
182+def clean_cloud_init(target):
183+ """clean out any local cloud-init config"""
184+ flist = glob.glob(
185+ os.path.sep.join([target, "/etc/cloud/cloud.cfg.d/*dpkg*"]))
186+
187+ LOG.debug("cleaning cloud-init config from: %s", flist)
188+ for dpkg_cfg in flist:
189+ os.unlink(dpkg_cfg)
190+
191+
192+def mirrorurl_to_apt_fileprefix(mirror):
193+ """ mirrorurl_to_apt_fileprefix
194+ Convert a mirror url to the file prefix used by apt on disk to
195+ store cache information for that mirror.
196+ To do so do:
197+ - take off ???://
198+ - drop tailing /
199+ - convert in string / to _
200+ """
201+ string = mirror
202+ if string.endswith("/"):
203+ string = string[0:-1]
204+ pos = string.find("://")
205+ if pos >= 0:
206+ string = string[pos + 3:]
207+ string = string.replace("/", "_")
208+ return string
209+
210+
211+def rename_apt_lists(new_mirrors, target):
212+ """rename_apt_lists - rename apt lists to preserve old cache data"""
213+ default_mirrors = get_default_mirrors(target)
214+ for (name, omirror) in default_mirrors.items():
215+ nmirror = new_mirrors.get(name)
216+ if not nmirror:
217+ continue
218+ oprefix = os.path.join(target, APT_LISTS,
219+ mirrorurl_to_apt_fileprefix(omirror))
220+ nprefix = os.path.join(target, APT_LISTS,
221+ mirrorurl_to_apt_fileprefix(nmirror))
222+ if oprefix == nprefix:
223+ continue
224+ olen = len(oprefix)
225+ for filename in glob.glob("%s_*" % oprefix):
226+ newname = "%s%s" % (nprefix, filename[olen:])
227+ LOG.debug("Renaming apt list %s to %s", filename, newname)
228+ try:
229+ os.rename(filename, newname)
230+ except OSError:
231+ # since this is a best effort task, warn with but don't fail
232+ LOG.warn("Failed to rename apt list:", exc_info=True)
233+
234+
235+def mirror_to_placeholder(tmpl, mirror, placeholder):
236+ """ mirror_to_placeholder
237+ replace the specified mirror in a template with a placeholder string
238+ Checks for existance of the expected mirror and warns if not found
239+ """
240+ if mirror not in tmpl:
241+ LOG.warn("Expected mirror '%s' not found in: %s", mirror, tmpl)
242+ return tmpl.replace(mirror, placeholder)
243+
244+
245+def map_known_suites(suite):
246+ """there are a few default names which will be auto-extended.
247+ This comes at the inability to use those names literally as suites,
248+ but on the other hand increases readability of the cfg quite a lot"""
249+ mapping = {'updates': '$RELEASE-updates',
250+ 'backports': '$RELEASE-backports',
251+ 'security': '$RELEASE-security',
252+ 'proposed': '$RELEASE-proposed',
253+ 'release': '$RELEASE'}
254+ try:
255+ retsuite = mapping[suite]
256+ except KeyError:
257+ retsuite = suite
258+ return retsuite
259+
260+
261+def disable_suites(cfg, src, release):
262+ """reads the config for suites to be disabled and removes those
263+ from the template"""
264+ retsrc = src
265+ suites_to_disable = cfg.get('disable_suites', None)
266+ if suites_to_disable is not None:
267+ for suite in suites_to_disable:
268+ suite = map_known_suites(suite)
269+ releasesuite = util.render_string(suite, {'RELEASE': release})
270+ LOG.debug("Disabling suite %s as %s", suite, releasesuite)
271+
272+ newsrc = ""
273+ for line in retsrc.splitlines(True):
274+ if line.startswith("#"):
275+ newsrc += line
276+ continue
277+
278+ # sources.list allow options in cols[1] which can have spaces
279+ # so the actual suite can be [2] or later
280+ cols = line.split()
281+ pcol = 2
282+ if cols[1].startswith("["):
283+ for col in cols[1:]:
284+ pcol += 1
285+ if col.endswith("]"):
286+ break
287+
288+ if cols[pcol] == releasesuite:
289+ line = '# suite disabled by curtin: %s' % line
290+ newsrc += line
291+ retsrc = newsrc
292+
293+ return retsrc
294+
295+
296+def generate_sources_list(cfg, release, mirrors, target):
297+ """ generate_sources_list
298+ create a source.list file based on a custom or default template
299+ by replacing mirrors and release in the template
300+ """
301+ default_mirrors = get_default_mirrors(target)
302+ aptsrc = "/etc/apt/sources.list"
303+ params = {'RELEASE': release}
304+ for k in mirrors:
305+ params[k] = mirrors[k]
306+
307+ tmpl = cfg.get('sources_list', None)
308+ if tmpl is None:
309+ LOG.info("No custom template provided, fall back to modify"
310+ "mirrors in %s on the target system", aptsrc)
311+ tmpl = util.load_file(target + aptsrc)
312+ # Strategy if no custom template was provided:
313+ # - Only replacing mirrors
314+ # - no reason to replace "release" as it is from target anyway
315+ # - The less we depend upon, the more stable this is against changes
316+ # - warn if expected original content wasn't found
317+ tmpl = mirror_to_placeholder(tmpl, default_mirrors['PRIMARY'],
318+ "$MIRROR")
319+ tmpl = mirror_to_placeholder(tmpl, default_mirrors['SECURITY'],
320+ "$SECURITY")
321+ try:
322+ os.rename(target + aptsrc,
323+ target + aptsrc + ".curtin")
324+ except OSError:
325+ LOG.exception("failed to backup %s/%s", target, aptsrc)
326+
327+ rendered = util.render_string(tmpl, params)
328+ disabled = disable_suites(cfg, rendered, release)
329+ util.write_file(target + aptsrc, disabled, mode=0o644)
330+
331+ # protect the just generated sources.list from cloud-init
332+ clouddir = "/etc/cloud/cloud.cfg.d"
333+ cloudfile = clouddir + "/" + "curtin-preserve-sources.cfg"
334+ # this has to work with older cloud-init as well, so use old key
335+ cloudconf = yaml.dump({'apt_preserve_sources_list': True}, indent=1)
336+ util.subp(['mkdir', '-p', target + clouddir], rcs=[0, 1])
337+ try:
338+ util.write_file(target + cloudfile, cloudconf, mode=0o644)
339+ except IOError:
340+ LOG.exception("Failed to protect source.list from cloud-init in (%s)",
341+ target + cloudfile)
342+ raise
343+
344+
345+def add_apt_key_raw(key, target):
346+ """
347+ actual adding of a key as defined in key argument
348+ to the system
349+ """
350+ LOG.debug("Adding key:\n'%s'", key)
351+ try:
352+ with util.RunInChroot(target, allow_daemons=True) as in_chroot:
353+ in_chroot(['apt-key', 'add', '-'], data=key.encode())
354+ except util.ProcessExecutionError:
355+ LOG.exception("failed to add apt GPG Key to apt keyring")
356+ raise
357+
358+
359+def add_apt_key(ent, target):
360+ """
361+ Add key to the system as defined in ent (if any).
362+ Supports raw keys or keyid's
363+ The latter will as a first step fetched to get the raw key
364+ """
365+ if 'keyid' in ent and 'key' not in ent:
366+ keyserver = DEFAULT_KEYSERVER
367+ if 'keyserver' in ent:
368+ keyserver = ent['keyserver']
369+
370+ ent['key'] = gpg.getkeybyid(ent['keyid'], keyserver)
371+
372+ if 'key' in ent:
373+ add_apt_key_raw(ent['key'], target)
374+
375+
376+def add_apt_sources(srcdict, target, template_params=None, aa_repo_match=None):
377+ """
378+ add entries in /etc/apt/sources.list.d for each abbreviated
379+ sources.list entry in 'srcdict'. When rendering template, also
380+ include the values in dictionary searchList
381+ """
382+ if template_params is None:
383+ template_params = {}
384+
385+ if aa_repo_match is None:
386+ raise ValueError('did not get a valid repo matcher')
387+
388+ if not isinstance(srcdict, dict):
389+ raise TypeError('unknown apt format: %s' % (srcdict))
390+
391+ for filename in srcdict:
392+ ent = srcdict[filename]
393+ if 'filename' not in ent:
394+ ent['filename'] = filename
395+
396+ add_apt_key(ent, target)
397+
398+ if 'source' not in ent:
399+ continue
400+ source = ent['source']
401+ source = util.render_string(source, template_params)
402+
403+ if not ent['filename'].startswith("/"):
404+ ent['filename'] = os.path.join("/etc/apt/sources.list.d/",
405+ ent['filename'])
406+ if not ent['filename'].endswith(".list"):
407+ ent['filename'] += ".list"
408+
409+ if aa_repo_match(source):
410+ try:
411+ with util.RunInChroot(target, allow_daemons=True) as in_chroot:
412+ in_chroot(["add-apt-repository", source])
413+ except util.ProcessExecutionError:
414+ LOG.exception("add-apt-repository failed.")
415+ raise
416+ continue
417+
418+ sourcefn = target + ent['filename']
419+ try:
420+ contents = "%s\n" % (source)
421+ util.write_file(sourcefn, contents, omode="a")
422+ except IOError as detail:
423+ LOG.exception("failed write to file %s: %s", sourcefn, detail)
424+ raise
425+
426+ util.apt_update(target=target, force=True,
427+ comment="apt-source changed config")
428+
429+ return
430+
431+
432+def search_for_mirror(candidates):
433+ """
434+ Search through a list of mirror urls for one that works
435+ This needs to return quickly.
436+ """
437+ if candidates is None:
438+ return None
439+
440+ for cand in candidates:
441+ try:
442+ if util.is_resolvable_url(cand):
443+ return cand
444+ except Exception:
445+ pass
446+ return None
447+
448+
449+def search_for_mirror_dns(enabled, mirrortext):
450+ "builds a list of potential mirror to check"
451+ if enabled is None or not enabled:
452+ return None
453+
454+ mydom = ""
455+ doms = []
456+
457+ # curtin has no fqdn/hostname in config as cloud-init
458+ # but if we got a hostname by dhcp, then search its domain portion first
459+ try:
460+ (fqdn, _) = util.subp(["hostname", "--fqdn"], rcs=[0], capture=True)
461+ mydom = ".".join(fqdn.split(".")[1:])
462+ if mydom:
463+ doms.append(".%s" % mydom)
464+ except util.ProcessExecutionError:
465+ # this can happen if /etc/hostname isn't set up properly yet
466+ # so log, but don't fail
467+ LOG.exception("failed to get fqdn")
468+
469+ doms.extend((".localdomain", "",))
470+
471+ potential_mirror_list = []
472+ # for curtin just ubuntu instead of fetching from datasource
473+ distro = "ubuntu"
474+ mirrorfmt = "http://%s-%s%s/%s" % (distro, mirrortext, "%s", distro)
475+ for post in doms:
476+ potential_mirror_list.append(mirrorfmt % (post))
477+
478+ return search_for_mirror(potential_mirror_list)
479+
480+
481+def update_mirror_info(pmirror, smirror, target=None):
482+ """sets security mirror to primary if not defined.
483+ returns defaults if no mirrors are defined"""
484+ if pmirror is not None:
485+ if smirror is None:
486+ smirror = pmirror
487+ return {'PRIMARY': pmirror,
488+ 'SECURITY': smirror}
489+ return get_default_mirrors(target)
490+
491+
492+def get_arch_mirrorconfig(cfg, mirrortype, arch):
493+ """out of a list of potential mirror configurations select
494+ and return the one matching the architecture (or default)"""
495+ # select the mirror specification (if-any)
496+ mirror_cfg_list = cfg.get(mirrortype, None)
497+ if mirror_cfg_list is None:
498+ return None
499+
500+ # select the specification matching the target arch
501+ default = None
502+ for mirror_cfg_elem in mirror_cfg_list:
503+ arches = mirror_cfg_elem.get("arches")
504+ if arch in arches:
505+ return mirror_cfg_elem
506+ if "default" in arches:
507+ default = mirror_cfg_elem
508+ return default
509+
510+
511+def get_mirror(cfg, mirrortype, arch):
512+ """pass the three potential stages of mirror specification
513+ returns None is neither of them found anything otherwise the first
514+ hit is returned"""
515+ mcfg = get_arch_mirrorconfig(cfg, mirrortype, arch)
516+ if mcfg is None:
517+ return None
518+
519+ # directly specified
520+ mirror = mcfg.get("uri", None)
521+ if mirror is None:
522+ # list of mirrors to try to resolve
523+ mirror = search_for_mirror(mcfg.get("search", None))
524+
525+ if mirror is None:
526+ # search for predfined dns patterns
527+ if mirrortype == "primary":
528+ pattern = "mirror"
529+ else:
530+ pattern = "%s-mirror" % mirrortype
531+ mirror = search_for_mirror_dns(mcfg.get("search_dns", None), pattern)
532+
533+ return mirror
534+
535+
536+def find_apt_mirror_info(cfg, target=None):
537+ """find_apt_mirror_info
538+ find an apt_mirror given the cfg provided.
539+ It can check for separate config of primary and security mirrors
540+ If only primary is given security is assumed to be equal to primary
541+ If the generic apt_mirror is given that is defining for both
542+ """
543+
544+ arch = util.get_architecture(target)
545+ LOG.debug("got arch for mirror selection: %s", arch)
546+ pmirror = get_mirror(cfg, "primary", arch)
547+ LOG.debug("got primary mirror: %s", pmirror)
548+ smirror = get_mirror(cfg, "security", arch)
549+ LOG.debug("got security mirror: %s", smirror)
550+
551+ # Note: curtin has no cloud-datasource fallback
552+
553+ mirror_info = update_mirror_info(pmirror, smirror, target)
554+
555+ # less complex replacements use only MIRROR, derive from primary
556+ mirror_info["MIRROR"] = mirror_info["PRIMARY"]
557+
558+ return mirror_info
559+
560+
561+def apply_apt_proxy_config(cfg, proxy_fname, config_fname):
562+ """apply_apt_proxy_config
563+ Applies any apt*proxy config from if specified
564+ """
565+ # Set up any apt proxy
566+ cfgs = (('proxy', 'Acquire::http::Proxy "%s";'),
567+ ('http_proxy', 'Acquire::http::Proxy "%s";'),
568+ ('ftp_proxy', 'Acquire::ftp::Proxy "%s";'),
569+ ('https_proxy', 'Acquire::https::Proxy "%s";'))
570+
571+ proxies = [fmt % cfg.get(name) for (name, fmt) in cfgs if cfg.get(name)]
572+ if len(proxies):
573+ LOG.debug("write apt proxy info to %s", proxy_fname)
574+ util.write_file(proxy_fname, '\n'.join(proxies) + '\n')
575+ elif os.path.isfile(proxy_fname):
576+ util.del_file(proxy_fname)
577+ LOG.debug("no apt proxy configured, removed %s", proxy_fname)
578+
579+ if cfg.get('conf', None):
580+ LOG.debug("write apt config info to %s", config_fname)
581+ util.write_file(config_fname, cfg.get('conf'))
582+ elif os.path.isfile(config_fname):
583+ util.del_file(config_fname)
584+ LOG.debug("no apt config configured, removed %s", config_fname)
585+
586+
587+def apt_command(args):
588+ """ Main entry point for curtin apt-config standalone command
589+ This does not read the global config as handled by curthooks, but
590+ instead one can specify a different "target" and a new cfg via --config
591+ """
592+ cfg = config.load_command_config(args, {})
593+
594+ if args.target is not None:
595+ target = args.target
596+ else:
597+ state = util.load_command_environment()
598+ target = state['target']
599+
600+ if target is None:
601+ sys.stderr.write("Unable to find target. "
602+ "Use --target or set TARGET_MOUNT_POINT\n")
603+ sys.exit(2)
604+
605+ apt_cfg = cfg.get("apt")
606+ # if no apt config section is available, do nothing
607+ if apt_cfg is not None:
608+ LOG.debug("Handling apt to target %s with config %s",
609+ target, apt_cfg)
610+ try:
611+ with util.ChrootableTarget(target, allow_daemons=False,
612+ sys_resolvconf=True):
613+ handle_apt(apt_cfg, target)
614+ except (RuntimeError, TypeError, ValueError, IOError):
615+ LOG.exception("Failed to configure apt features '%s'", apt_cfg)
616+ sys.exit(1)
617+ else:
618+ LOG.info("No apt config provided, skipping")
619+
620+ sys.exit(0)
621+
622+
623+def translate_old_apt_features(cfg):
624+ """translate the few old apt related features into the new config format"""
625+ predef_apt_cfg = cfg.get("apt")
626+ if predef_apt_cfg is None:
627+ cfg['apt'] = {}
628+ predef_apt_cfg = cfg.get("apt")
629+
630+ if cfg.get('apt_proxy') is not None:
631+ if predef_apt_cfg.get('proxy') is not None:
632+ msg = ("Error in apt_proxy configuration: "
633+ "old and new format of apt features "
634+ "are mutually exclusive")
635+ LOG.error(msg)
636+ raise ValueError(msg)
637+
638+ cfg['apt']['proxy'] = cfg.get('apt_proxy')
639+ LOG.debug("Transferred %s into new format: %s", cfg.get('apt_proxy'),
640+ cfg.get('apte'))
641+ del cfg['apt_proxy']
642+
643+ if cfg.get('apt_mirrors') is not None:
644+ if predef_apt_cfg.get('mirrors') is not None:
645+ msg = ("Error in apt_mirror configuration: "
646+ "old and new format of apt features "
647+ "are mutually exclusive")
648+ LOG.error(msg)
649+ raise ValueError(msg)
650+
651+ old = cfg.get('apt_mirrors')
652+ cfg['apt']['primary'] = [{"arches": ["default"],
653+ "uri": old.get('ubuntu_archive')}]
654+ cfg['apt']['security'] = [{"arches": ["default"],
655+ "uri": old.get('ubuntu_security')}]
656+ LOG.debug("Transferred %s into new format: %s", cfg.get('apt_mirror'),
657+ cfg.get('apt'))
658+ del cfg['apt_mirrors']
659+ # to work this also needs to disable the default protection
660+ psl = predef_apt_cfg.get('preserve_sources_list')
661+ if psl is not None:
662+ if config.value_as_boolean(psl) is True:
663+ msg = ("Error in apt_mirror configuration: "
664+ "apt_mirrors and preserve_sources_list: True "
665+ "are mutually exclusive")
666+ LOG.error(msg)
667+ raise ValueError(msg)
668+ cfg['apt']['preserve_sources_list'] = False
669+
670+ if cfg.get('debconf_selections') is not None:
671+ if predef_apt_cfg.get('debconf_selections') is not None:
672+ msg = ("Error in debconf_selections configuration: "
673+ "old and new format of apt features "
674+ "are mutually exclusive")
675+ LOG.error(msg)
676+ raise ValueError(msg)
677+
678+ selsets = cfg.get('debconf_selections')
679+ cfg['apt']['debconf_selections'] = selsets
680+ LOG.info("Transferred %s into new format: %s",
681+ cfg.get('debconf_selections'),
682+ cfg.get('apt'))
683+ del cfg['debconf_selections']
684+
685+ return cfg
686+
687+
688+CMD_ARGUMENTS = (
689+ ((('-c', '--config'),
690+ {'help': 'read configuration from cfg', 'action': util.MergedCmdAppend,
691+ 'metavar': 'FILE', 'type': argparse.FileType("rb"),
692+ 'dest': 'cfgopts', 'default': []}),
693+ (('-t', '--target'),
694+ {'help': 'chroot to target. default is env[TARGET_MOUNT_POINT]',
695+ 'action': 'store', 'metavar': 'TARGET',
696+ 'default': os.environ.get('TARGET_MOUNT_POINT')}),)
697+)
698+
699+
700+def POPULATE_SUBCMD(parser):
701+ """Populate subcommand option parsing for apt-config"""
702+ populate_one_subcmd(parser, CMD_ARGUMENTS, apt_command)
703+
704+CONFIG_CLEANERS = {
705+ 'cloud-init': clean_cloud_init,
706+}
707+
708+# vi: ts=4 expandtab syntax=python
709
710=== modified file 'curtin/commands/curthooks.py'
711--- curtin/commands/curthooks.py 2016-06-24 19:27:52 +0000
712+++ curtin/commands/curthooks.py 2016-07-26 19:53:43 +0000
713@@ -16,10 +16,8 @@
714 # along with Curtin. If not, see <http://www.gnu.org/licenses/>.
715
716 import copy
717-import glob
718 import os
719 import platform
720-import re
721 import sys
722 import shutil
723 import textwrap
724@@ -32,6 +30,7 @@
725 from curtin import util
726 from curtin import net
727 from curtin.reporter import events
728+from curtin.commands import apt_config
729
730 from . import populate_one_subcmd
731
732@@ -90,45 +89,15 @@
733 info.get('perms', "0644")))
734
735
736-def apt_config(cfg, target):
737- # cfg['apt_proxy']
738-
739- proxy_cfg_path = os.path.sep.join(
740- [target, '/etc/apt/apt.conf.d/90curtin-aptproxy'])
741- if cfg.get('apt_proxy'):
742- util.write_file(
743- proxy_cfg_path,
744- content='Acquire::HTTP::Proxy "%s";\n' % cfg['apt_proxy'])
745+def do_apt_config(cfg, target):
746+ cfg = apt_config.translate_old_apt_features(cfg)
747+ apt_cfg = cfg.get("apt")
748+ if apt_cfg is not None:
749+ LOG.info("curthooks handling apt to target %s with config %s",
750+ target, apt_cfg)
751+ apt_config.handle_apt(apt_cfg, target)
752 else:
753- if os.path.isfile(proxy_cfg_path):
754- os.unlink(proxy_cfg_path)
755-
756- # cfg['apt_mirrors']
757- # apt_mirrors:
758- # ubuntu_archive: http://local.archive/ubuntu
759- # ubuntu_security: http://local.archive/ubuntu
760- sources_list = os.path.sep.join([target, '/etc/apt/sources.list'])
761- if (isinstance(cfg.get('apt_mirrors'), dict) and
762- os.path.isfile(sources_list)):
763- repls = [
764- ('ubuntu_archive', r'http://\S*[.]*archive.ubuntu.com/\S*'),
765- ('ubuntu_security', r'http://security.ubuntu.com/\S*'),
766- ]
767- content = None
768- for name, regex in repls:
769- mirror = cfg['apt_mirrors'].get(name)
770- if not mirror:
771- continue
772-
773- if content is None:
774- with open(sources_list) as fp:
775- content = fp.read()
776- util.write_file(sources_list + ".dist", content)
777-
778- content = re.sub(regex, mirror + " ", content)
779-
780- if content is not None:
781- util.write_file(sources_list, content)
782+ LOG.info("No apt config provided, skipping")
783
784
785 def disable_overlayroot(cfg, target):
786@@ -140,15 +109,6 @@
787 shutil.move(local_conf, local_conf + ".old")
788
789
790-def clean_cloud_init(target):
791- flist = glob.glob(
792- os.path.sep.join([target, "/etc/cloud/cloud.cfg.d/*dpkg*"]))
793-
794- LOG.debug("cleaning cloud-init config from: %s" % flist)
795- for dpkg_cfg in flist:
796- os.unlink(dpkg_cfg)
797-
798-
799 def _maybe_remove_legacy_eth0(target,
800 path="/etc/network/interfaces.d/eth0.cfg"):
801 """Ubuntu cloud images previously included a 'eth0.cfg' that had
802@@ -293,85 +253,6 @@
803 " System may not boot.", package)
804
805
806-def apply_debconf_selections(cfg, target):
807- # debconf_selections:
808- # set1: |
809- # cloud-init cloud-init/datasources multiselect MAAS
810- # set2: pkg pkg/value string bar
811- selsets = cfg.get('debconf_selections')
812- if not selsets:
813- LOG.debug("debconf_selections was not set in config")
814- return
815-
816- # for each entry in selections, chroot and apply them.
817- # keep a running total of packages we've seen.
818- pkgs_cfgd = set()
819- for key, content in selsets.items():
820- LOG.debug("setting for %s, %s" % (key, content))
821- util.subp(['chroot', target, 'debconf-set-selections'],
822- data=content.encode())
823- for line in content.splitlines():
824- if line.startswith("#"):
825- continue
826- pkg = re.sub(r"[:\s].*", "", line)
827- pkgs_cfgd.add(pkg)
828-
829- pkgs_installed = get_installed_packages(target)
830-
831- LOG.debug("pkgs_cfgd: %s" % pkgs_cfgd)
832- LOG.debug("pkgs_installed: %s" % pkgs_installed)
833- need_reconfig = pkgs_cfgd.intersection(pkgs_installed)
834-
835- if len(need_reconfig) == 0:
836- LOG.debug("no need for reconfig")
837- return
838-
839- # For any packages that are already installed, but have preseed data
840- # we populate the debconf database, but the filesystem configuration
841- # would be preferred on a subsequent dpkg-reconfigure.
842- # so, what we have to do is "know" information about certain packages
843- # to unconfigure them.
844- unhandled = []
845- to_config = []
846- for pkg in need_reconfig:
847- if pkg in CONFIG_CLEANERS:
848- LOG.debug("unconfiguring %s" % pkg)
849- CONFIG_CLEANERS[pkg](target)
850- to_config.append(pkg)
851- else:
852- unhandled.append(pkg)
853-
854- if len(unhandled):
855- LOG.warn("The following packages were installed and preseeded, "
856- "but cannot be unconfigured: %s", unhandled)
857-
858- util.subp(['chroot', target, 'dpkg-reconfigure',
859- '--frontend=noninteractive'] +
860- list(to_config), data=None)
861-
862-
863-def get_installed_packages(target=None):
864- cmd = []
865- if target is not None:
866- cmd = ['chroot', target]
867- cmd.extend(['dpkg-query', '--list'])
868-
869- (out, _err) = util.subp(cmd, capture=True)
870- if isinstance(out, bytes):
871- out = out.decode()
872-
873- pkgs_inst = set()
874- for line in out.splitlines():
875- try:
876- (state, pkg, other) = line.split(None, 2)
877- except ValueError:
878- continue
879- if state.startswith("hi") or state.startswith("ii"):
880- pkgs_inst.add(re.sub(":.*", "", pkg))
881-
882- return pkgs_inst
883-
884-
885 def setup_grub(cfg, target):
886 # target is the path to the mounted filesystem
887
888@@ -740,7 +621,7 @@
889 }
890
891 needed_packages = []
892- installed_packages = get_installed_packages(target)
893+ installed_packages = util.get_installed_packages(target)
894 for cust_cfg, pkg_reqs in custom_configs.items():
895 if cust_cfg not in cfg:
896 continue
897@@ -820,7 +701,7 @@
898 name=stack_prefix, reporting_enabled=True, level="INFO",
899 description="writing config files and configuring apt"):
900 write_files(cfg, target)
901- apt_config(cfg, target)
902+ do_apt_config(cfg, target)
903 disable_overlayroot(cfg, target)
904
905 # packages may be needed prior to installing kernel
906@@ -843,7 +724,6 @@
907 setup_zipl(cfg, target)
908 install_kernel(cfg, target)
909 run_zipl(cfg, target)
910- apply_debconf_selections(cfg, target)
911
912 restore_dist_interfaces(cfg, target)
913
914@@ -906,8 +786,4 @@
915 populate_one_subcmd(parser, CMD_ARGUMENTS, curthooks)
916
917
918-CONFIG_CLEANERS = {
919- 'cloud-init': clean_cloud_init,
920-}
921-
922 # vi: ts=4 expandtab syntax=python
923
924=== modified file 'curtin/commands/main.py'
925--- curtin/commands/main.py 2016-04-04 20:12:01 +0000
926+++ curtin/commands/main.py 2016-07-26 19:53:43 +0000
927@@ -27,7 +27,7 @@
928
929 SUB_COMMAND_MODULES = [
930 'apply_net', 'block-meta', 'block-wipe', 'curthooks', 'extract',
931- 'hook', 'in-target', 'install', 'mkfs', 'net-meta',
932+ 'hook', 'in-target', 'install', 'mkfs', 'net-meta', 'apt-config',
933 'pack', 'swap', 'system-install', 'system-upgrade']
934
935
936
937=== added file 'curtin/gpg.py'
938--- curtin/gpg.py 1970-01-01 00:00:00 +0000
939+++ curtin/gpg.py 2016-07-26 19:53:43 +0000
940@@ -0,0 +1,74 @@
941+# Copyright (C) 2016 Canonical Ltd.
942+#
943+# Author: Scott Moser <scott.moser@canonical.com>
944+# Christian Ehrhardt <christian.ehrhardt@canonical.com>
945+#
946+# Curtin is free software: you can redistribute it and/or modify it under
947+# the terms of the GNU Affero General Public License as published by the
948+# Free Software Foundation, either version 3 of the License, or (at your
949+# option) any later version.
950+#
951+# Curtin is distributed in the hope that it will be useful, but WITHOUT ANY
952+# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
953+# FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for
954+# more details.
955+#
956+# You should have received a copy of the GNU Affero General Public License
957+# along with Curtin. If not, see <http://www.gnu.org/licenses/>.
958+""" gpg.py
959+gpg related utilities to get raw keys data by their id
960+"""
961+
962+from curtin import util
963+
964+from .log import LOG
965+
966+
967+def export_armour(key):
968+ """Export gpg key, armoured key gets returned"""
969+ try:
970+ (armour, _) = util.subp(["gpg", "--export", "--armour", key],
971+ capture=True)
972+ except util.ProcessExecutionError as error:
973+ # debug, since it happens for any key not on the system initially
974+ LOG.debug('Failed to export armoured key "%s": %s', key, error)
975+ armour = None
976+ return armour
977+
978+
979+def recv_key(key, keyserver):
980+ """Receive gpg key from the specified keyserver"""
981+ LOG.debug('Receive gpg key "%s"', key)
982+ try:
983+ util.subp(["gpg", "--keyserver", keyserver, "--recv", key],
984+ capture=True)
985+ except util.ProcessExecutionError as error:
986+ raise ValueError(('Failed to import key "%s" '
987+ 'from server "%s" - error %s') %
988+ (key, keyserver, error))
989+
990+
991+def delete_key(key):
992+ """Delete the specified key from the local gpg ring"""
993+ try:
994+ util.subp(["gpg", "--batch", "--yes", "--delete-keys", key],
995+ capture=True)
996+ except util.ProcessExecutionError as error:
997+ LOG.warn('Failed delete key "%s": %s', key, error)
998+
999+
1000+def getkeybyid(keyid, keyserver='keyserver.ubuntu.com'):
1001+ """get gpg keyid from keyserver"""
1002+ armour = export_armour(keyid)
1003+ if not armour:
1004+ try:
1005+ recv_key(keyid, keyserver=keyserver)
1006+ armour = export_armour(keyid)
1007+ except ValueError:
1008+ LOG.exception('Failed to obtain gpg key %s', keyid)
1009+ raise
1010+ finally:
1011+ # delete just imported key to leave environment as it was before
1012+ delete_key(keyid)
1013+
1014+ return armour
1015
1016=== modified file 'curtin/util.py'
1017--- curtin/util.py 2016-06-24 19:27:52 +0000
1018+++ curtin/util.py 2016-07-26 19:53:43 +0000
1019@@ -16,18 +16,30 @@
1020 # along with Curtin. If not, see <http://www.gnu.org/licenses/>.
1021
1022 import argparse
1023+import collections
1024 import errno
1025 import glob
1026 import json
1027 import os
1028 import platform
1029+import re
1030 import shutil
1031+import socket
1032 import subprocess
1033 import stat
1034 import sys
1035 import tempfile
1036 import time
1037
1038+# avoid the dependency to python3-six as used in cloud-init
1039+try:
1040+ from urlparse import urlparse
1041+except ImportError:
1042+ # python3
1043+ # avoid triggering pylint, https://github.com/PyCQA/pylint/issues/769
1044+ # pylint:disable=import-error,no-name-in-module
1045+ from urllib.parse import urlparse
1046+
1047 from .log import LOG
1048
1049 _INSTALLED_HELPERS_PATH = '/usr/lib/curtin/helpers'
1050@@ -35,6 +47,11 @@
1051
1052 _LSB_RELEASE = {}
1053
1054+_DNS_REDIRECT_IP = None
1055+
1056+# matcher used in template rendering functions
1057+BASIC_MATCHER = re.compile(r'\$\{([A-Za-z0-9_.]+)\}|\$([A-Za-z0-9_.]+)')
1058+
1059
1060 def _subp(args, data=None, rcs=None, env=None, capture=False, shell=False,
1061 logstring=False, decode="replace"):
1062@@ -486,6 +503,28 @@
1063 return False
1064
1065
1066+def get_installed_packages(target=None):
1067+ cmd = []
1068+ if target is not None:
1069+ cmd = ['chroot', target]
1070+ cmd.extend(['dpkg-query', '--list'])
1071+
1072+ (out, _) = subp(cmd, capture=True)
1073+ if isinstance(out, bytes):
1074+ out = out.decode()
1075+
1076+ pkgs_inst = set()
1077+ for line in out.splitlines():
1078+ try:
1079+ (state, pkg, _) = line.split(None, 2)
1080+ except ValueError:
1081+ continue
1082+ if state.startswith("hi") or state.startswith("ii"):
1083+ pkgs_inst.add(re.sub(":.*", "", pkg))
1084+
1085+ return pkgs_inst
1086+
1087+
1088 def has_pkg_installed(pkg, target=None):
1089 chroot = []
1090 if target is not None:
1091@@ -846,14 +885,17 @@
1092 return (isinstance(exc, IOError) and exc.errno == errno.ENOENT)
1093
1094
1095-def lsb_release():
1096+def lsb_release(target=None):
1097 fmap = {'Codename': 'codename', 'Description': 'description',
1098 'Distributor ID': 'id', 'Release': 'release'}
1099+ chroot = []
1100+ if target is not None:
1101+ chroot = ['chroot', target]
1102 global _LSB_RELEASE
1103 if not _LSB_RELEASE:
1104 data = {}
1105 try:
1106- out, err = subp(['lsb_release', '--all'], capture=True)
1107+ out, err = subp(chroot + ['lsb_release', '--all'], capture=True)
1108 for line in out.splitlines():
1109 fname, tok, val = line.partition(":")
1110 if fname in fmap:
1111@@ -895,4 +937,104 @@
1112 }
1113 return platform2arch.get(platform.machine(), platform.machine())
1114
1115+
1116+def basic_template_render(content, params):
1117+ """This does simple replacement of bash variable like templates.
1118+
1119+ It identifies patterns like ${a} or $a and can also identify patterns like
1120+ ${a.b} or $a.b which will look for a key 'b' in the dictionary rooted
1121+ by key 'a'.
1122+ """
1123+
1124+ def replacer(match):
1125+ """ replacer
1126+ replacer used in regex match to replace content
1127+ """
1128+ # Only 1 of the 2 groups will actually have a valid entry.
1129+ name = match.group(1)
1130+ if name is None:
1131+ name = match.group(2)
1132+ if name is None:
1133+ raise RuntimeError("Match encountered but no valid group present")
1134+ path = collections.deque(name.split("."))
1135+ selected_params = params
1136+ while len(path) > 1:
1137+ key = path.popleft()
1138+ if not isinstance(selected_params, dict):
1139+ raise TypeError("Can not traverse into"
1140+ " non-dictionary '%s' of type %s while"
1141+ " looking for subkey '%s'"
1142+ % (selected_params,
1143+ selected_params.__class__.__name__,
1144+ key))
1145+ selected_params = selected_params[key]
1146+ key = path.popleft()
1147+ if not isinstance(selected_params, dict):
1148+ raise TypeError("Can not extract key '%s' from non-dictionary"
1149+ " '%s' of type %s"
1150+ % (key, selected_params,
1151+ selected_params.__class__.__name__))
1152+ return str(selected_params[key])
1153+
1154+ return BASIC_MATCHER.sub(replacer, content)
1155+
1156+
1157+def render_string(content, params):
1158+ """ render_string
1159+ render a string following replacement rules as defined in
1160+ basic_template_render returning the string
1161+ """
1162+ if not params:
1163+ params = {}
1164+ return basic_template_render(content, params)
1165+
1166+
1167+def is_resolvable(name):
1168+ """determine if a url is resolvable, return a boolean
1169+ This also attempts to be resilent against dns redirection.
1170+
1171+ Note, that normal nsswitch resolution is used here. So in order
1172+ to avoid any utilization of 'search' entries in /etc/resolv.conf
1173+ we have to append '.'.
1174+
1175+ The top level 'invalid' domain is invalid per RFC. And example.com
1176+ should also not exist. The random entry will be resolved inside
1177+ the search list.
1178+ """
1179+ global _DNS_REDIRECT_IP
1180+ if _DNS_REDIRECT_IP is None:
1181+ badips = set()
1182+ badnames = ("does-not-exist.example.com.", "example.invalid.")
1183+ badresults = {}
1184+ for iname in badnames:
1185+ try:
1186+ result = socket.getaddrinfo(iname, None, 0, 0,
1187+ socket.SOCK_STREAM,
1188+ socket.AI_CANONNAME)
1189+ badresults[iname] = []
1190+ for (_, _, _, cname, sockaddr) in result:
1191+ badresults[iname].append("%s: %s" % (cname, sockaddr[0]))
1192+ badips.add(sockaddr[0])
1193+ except (socket.gaierror, socket.error):
1194+ pass
1195+ _DNS_REDIRECT_IP = badips
1196+ if badresults:
1197+ LOG.debug("detected dns redirection: %s", badresults)
1198+
1199+ try:
1200+ result = socket.getaddrinfo(name, None)
1201+ # check first result's sockaddr field
1202+ addr = result[0][4][0]
1203+ if addr in _DNS_REDIRECT_IP:
1204+ return False
1205+ return True
1206+ except (socket.gaierror, socket.error):
1207+ return False
1208+
1209+
1210+def is_resolvable_url(url):
1211+ """determine if this url is resolvable (existing or ip)."""
1212+ return is_resolvable(urlparse(url).hostname)
1213+
1214+
1215 # vi: ts=4 expandtab syntax=python
1216
1217=== modified file 'doc/devel/README-vmtest.txt'
1218--- doc/devel/README-vmtest.txt 2016-01-08 17:37:01 +0000
1219+++ doc/devel/README-vmtest.txt 2016-07-26 19:53:43 +0000
1220@@ -90,13 +90,13 @@
1221 The tests themselves don't actually have to run as root, but the
1222 test setup does.
1223 * the 'tools' directory must be in your path.
1224- * test will set apt_proxy in the guests to the value of
1225- 'apt_proxy' environment variable. If that is not set it will
1226+ * test will set apt: { proxy } in the guests to the value of
1227+ 'apt_proxy' environment variable. If that is not set it will
1228 look at the host's apt config and read 'Acquire::HTTP::Proxy'
1229
1230 == Environment Variables ==
1231 Some environment variables affect the running of vmtest
1232- * apt_proxy:
1233+ * apt_proxy:
1234 test will set apt_proxy in the guests to the value of 'apt_proxy'.
1235 If that is not set it will look at the host's apt config and read
1236 'Acquire::HTTP::Proxy'
1237
1238=== added file 'doc/topics/apt_source.rst'
1239--- doc/topics/apt_source.rst 1970-01-01 00:00:00 +0000
1240+++ doc/topics/apt_source.rst 2016-07-26 19:53:43 +0000
1241@@ -0,0 +1,152 @@
1242+==========
1243+APT Source
1244+==========
1245+
1246+This part of curtin is meant to allow influencing the apt behaviour and configuration.
1247+
1248+By default - if no apt config is provided - it does nothing. That keeps behavior compatible on upgrades.
1249+
1250+The feature has a target argument which - by default - is used to modify the environment that curtin currently installs (@TARGET_MOUNT_POINT).
1251+
1252+Features
1253+--------
1254+
1255+* Add PGP keys to the APT trusted keyring
1256+
1257+ - add via short keyid
1258+
1259+ - add via long key fingerprint
1260+
1261+ - specify a custom keyserver to pull from
1262+
1263+ - add raw keys (which makes you independent of keyservers)
1264+
1265+* Influence global apt configuration
1266+
1267+ - adding ppa's
1268+
1269+ - replacing mirror, security mirror and release in sources.list
1270+
1271+ - able to provide a fully custom template for sources.list
1272+
1273+ - add arbitrary apt.conf settings
1274+
1275+
1276+Configuration
1277+-------------
1278+
1279+The general configuration of the apt feature is under an element called ``apt``.
1280+
1281+This can have various "global" subelements as listed in the examples below.
1282+These global configurations are valid throughput all of the apt feature.
1283+So for exmaple a global specification of ``primary`` for a mirror will apply to all rendered sources entries.
1284+
1285+Then there is a section ``sources`` which can hold a number of subelements itself.
1286+The key is the filename and will be prepended by /etc/apt/sources.list.d/ if it doesn't start with a ``/``.
1287+There are certain cases - where no content is written into a source.list file where the filename will be ignored - yet it can still be used as index for merging.
1288+
1289+The values inside the entries consist of the following optional entries::
1290+* ``source``: a sources.list entry (some variable replacements apply)
1291+
1292+* ``keyid``: providing a key to import via shortid or fingerprint
1293+
1294+* ``key``: providing a raw PGP key
1295+
1296+* ``keyserver``: specify an alternate keyserver to pull keys from that were specified by keyid
1297+
1298+* ``filename``: for compatibility with the older format (now the key to this dictionary is the filename). If specified this overwrites the filename given as key.
1299+
1300+The section "sources" is is a dictionary (unlike most block/net configs which are lists). This format allows merging between multiple input files than a list like::
1301+ sources:
1302+ s1: {'key': 'key1', 'source': 'source1'}
1303+
1304+ sources:
1305+ s2: {'key': 'key2'}
1306+ s1: {'filename': 'foo'}
1307+
1308+ This would be merged into
1309+ s1: {'key': 'key1', 'source': 'source1', filename: 'foo'}
1310+ s2: {'key': 'key2'}
1311+
1312+Here is just one of the most common examples that could be used to install with curtin in a closed environment (derived repository):
1313+
1314+What do we need for that:
1315+* insert the PGP key of the local repository to be trusted
1316+
1317+ - since you are locked down you can't pull from keyserver.ubuntu.com
1318+
1319+ - if you have an internal keyserver you could pull from there, but let us assume you don't even have that; so you have to provide the raw key
1320+
1321+ - in the example I'll use the key of the "Ubuntu CD Image Automatic Signing Key" which makes no sense as it is in the trusted keyring anyway, but it is a good example. (Also the key is shortened to stay readable)
1322+
1323+::
1324+
1325+ -----BEGIN PGP PUBLIC KEY BLOCK-----
1326+ Version: GnuPG v1
1327+ mQGiBEFEnz8RBAC7LstGsKD7McXZgd58oN68KquARLBl6rjA2vdhwl77KkPPOr3O
1328+ RwIbDAAKCRBAl26vQ30FtdxYAJsFjU+xbex7gevyGQ2/mhqidES4MwCggqQyo+w1
1329+ Twx6DKLF+3rF5nf1F3Q=
1330+ =PBAe
1331+ -----END PGP PUBLIC KEY BLOCK-----
1332+
1333+* replace the mirror from apt pulls repository data
1334+
1335+ - lets consider we have a local mirror at ``mymirror.local`` but otherwise following the usual paths
1336+
1337+ - make an example with a partial mirror that doesn't mirror the backports suite, so backports have to be disabled
1338+
1339+That would be specified as
1340+::
1341+
1342+ apt:
1343+ primary:
1344+ - arches [default]
1345+ uri: http://mymirror.local/ubuntu/
1346+ disable_suites: [backports]
1347+ sources:
1348+ localrepokey:
1349+ key: | # full key as block
1350+ -----BEGIN PGP PUBLIC KEY BLOCK-----
1351+ Version: GnuPG v1
1352+
1353+ mQGiBEFEnz8RBAC7LstGsKD7McXZgd58oN68KquARLBl6rjA2vdhwl77KkPPOr3O
1354+ RwIbDAAKCRBAl26vQ30FtdxYAJsFjU+xbex7gevyGQ2/mhqidES4MwCggqQyo+w1
1355+ Twx6DKLF+3rF5nf1F3Q=
1356+ =PBAe
1357+ -----END PGP PUBLIC KEY BLOCK-----
1358+
1359+Please also read the section ``Dependencies`` below to avoid loosing some of the configuration content on first boot.
1360+
1361+The file examples/apt-source.yaml holds various further examples that can be configured with this feature.
1362+
1363+Common snippets
1364+---------------
1365+This is a collection of additional ideas people can use the feature for customizing their to-be-installed system.
1366+
1367+* enable proposed on installing
1368+ apt:
1369+ sources:
1370+ proposed.list: deb $MIRROR $RELEASE-proposed main restricted universe multiverse
1371+
1372+* Make debug symbols available
1373+ apt:
1374+  sources:
1375+   ddebs.list: |
1376+   deb http://ddebs.ubuntu.com $RELEASE main restricted universe multiverse
1377+   deb http://ddebs.ubuntu.com $RELEASE-updates main restricted universe multiverse
1378+  deb http://ddebs.ubuntu.com $RELEASE-security main restricted universe multiverse
1379+ deb http://ddebs.ubuntu.com $RELEASE-proposed main restricted universe multiverse
1380+
1381+Timing
1382+------
1383+The feature is implemented at the stage of curthooks_commands, after which runs just after curtin has extracted the image to the target.
1384+It can be ran as standalong command "curtin -v --config <yourconfigfile> apt-config".
1385+
1386+This will pick up the target from the environment variable that is set by curtin, if you want to use it to a different target or outside of usual curtin handling you can add ``--target <path>`` to it to overwrite the target path.
1387+This target should have at least a minimal system with apt and dpkg installed for the functionality to work.
1388+
1389+
1390+Dependencies
1391+------------
1392+Cloud-init might need to resolve dependencies and install packages in the ephemeral environment to run curtin.
1393+Therefore it is recommended to not only configure curtin for the target, but also the install environment with proper apt configuration via cloud-init.
1394
1395=== added file 'examples/apt-source.yaml'
1396--- examples/apt-source.yaml 1970-01-01 00:00:00 +0000
1397+++ examples/apt-source.yaml 2016-07-26 19:53:43 +0000
1398@@ -0,0 +1,239 @@
1399+# YAML example of an apt config.
1400+apt:
1401+ sources:
1402+ # This is a dictionary (unlike most block/net which are lists)
1403+ # The key is the filename and will be prepended by /etc/apt/sources.list.d/
1404+ # if it doesn't start with a '/'.
1405+ # There are certain cases - where no content is written into a source.list
1406+ # file where the filename will be ignored - yet it can still be used as
1407+ # index for merging.
1408+ # The values inside the entries consost of the following optional entries:
1409+ # 'source': a sources.list entry (some variable replacements apply)
1410+ # 'keyid': providing a key to import via shortid or fingerprint
1411+ # 'key': providing a raw PGP key
1412+ # 'keyserver': specify an alternate keyserver to pull keys from that
1413+ # were specified by keyid
1414+ # 'filename': for compatibility with the older format (now the key to
1415+ # this dictionary is the filename). If specified this
1416+ # overwrites the filename given as key.
1417+
1418+ # The new format allows merging between multiple input files than a list
1419+ # like:
1420+ # cloud-config1
1421+ # sources:
1422+ # s1: {'key': 'key1', 'source': 'source1'}
1423+ # cloud-config2
1424+ # sources:
1425+ # s2: {'key': 'key2'}
1426+ # s1: {'filename': 'foo'}
1427+ # This would be merged to
1428+ # sources:
1429+ # s1:
1430+ # filename: foo
1431+ # key: key1
1432+ # source: source1
1433+ # s2:
1434+ # key: key2
1435+
1436+ curtin-dev-ppa.list:
1437+ source: "deb http://ppa.launchpad.net/curtin-dev/test-archive/ubuntu xenial main"
1438+ keyid: F430BBA5 # GPG key ID published on a key server
1439+ # adding a source.list line, importing a gpg key for a given key id and
1440+ # storing the deb entry in /etc/apt/sources.list.d/curtin-dev-ppa.list
1441+
1442+ # PPA shortcut:
1443+ # * Setup correct apt sources.list line
1444+ # * Import the signing key from LP
1445+ #
1446+ # See https://help.launchpad.net/Packaging/PPA for more information
1447+ # this requires 'add-apt-repository'
1448+ # due to that the filename key is ignored in this case
1449+ ignored1:
1450+ source: "ppa:curtin-dev/test-archive" # Quote the string
1451+
1452+ # additional custom deb source config:
1453+ # * Creates a file in /etc/apt/sources.list.d/ for the sources list entry
1454+ # * [optional] Import the apt signing key from the keyserver
1455+ # * Defaults:
1456+ # + keyserver: keyserver.ubuntu.com
1457+ #
1458+ # See sources.list man page for more information about the format
1459+ my-repo1.list:
1460+ source: deb http://archive.ubuntu.com/ubuntu xenial-backports main universe multiverse restricted
1461+
1462+ # sources can use $MIRROR and $RELEASE and they will be replaced
1463+ # with the default or specified mirror and the running release.
1464+ # The entry below would be possibly turned into:
1465+ # source: deb http://archive.ubuntu.com/ubuntu xenial multiverse
1466+ # see below at apt*mirror for more
1467+ my-repo2.list:
1468+ source: deb $MIRROR $RELEASE multiverse
1469+
1470+ # this would have the same end effect as 'ppa:curtin-dev/test-archive'
1471+ my-repo3.list:
1472+ source: "deb http://ppa.launchpad.net/curtin-dev/test-archive/ubuntu xenial main"
1473+ keyid: F430BBA5 # GPG key ID published on a key server
1474+ filename: curtin-dev-ppa.list
1475+
1476+ # this would only import the key without adding a ppa or other source spec
1477+ # since this doesn't generate a source.list file the filename key is ignored
1478+ ignored2:
1479+ keyid: F430BBA5 # GPG key ID published on a key server
1480+
1481+ # In general keyid's can also be specified via their long fingerprints
1482+ # since this doesn't generate a source.list file the filename key is ignored
1483+ ignored3:
1484+ keyid: B59D 5F15 97A5 04B7 E230 6DCA 0620 BBCF 0368 3F77
1485+
1486+ # Custom apt repository:
1487+ # * The apt signing key can also be specified
1488+ # by providing a pgp public key block
1489+ # * Providing the PGP key here is the most robust method for
1490+ # specifying a key, as it removes dependency on a remote key server
1491+ my-repo4.list:
1492+ source: deb http://ppa.launchpad.net/curtin-dev/test-archive/ubuntu xenial main
1493+ key: | # The value needs to start with -----BEGIN PGP PUBLIC KEY BLOCK-----
1494+ -----BEGIN PGP PUBLIC KEY BLOCK-----
1495+ Version: SKS 1.0.10
1496+
1497+ mI0ESpA3UQEEALdZKVIMq0j6qWAXAyxSlF63SvPVIgxHPb9Nk0DZUixn+akqytxG4zKCONz6
1498+ qLjoBBfHnynyVLfT4ihg9an1PqxRnTO+JKQxl8NgKGz6Pon569GtAOdWNKw15XKinJTDLjnj
1499+ 9y96ljJqRcpV9t/WsIcdJPcKFR5voHTEoABE2aEXABEBAAG0GUxhdW5jaHBhZCBQUEEgZm9y
1500+ IEFsZXN0aWOItgQTAQIAIAUCSpA3UQIbAwYLCQgHAwIEFQIIAwQWAgMBAh4BAheAAAoJEA7H
1501+ 5Qi+CcVxWZ8D/1MyYvfj3FJPZUm2Yo1zZsQ657vHI9+pPouqflWOayRR9jbiyUFIn0VdQBrP
1502+ t0FwvnOFArUovUWoKAEdqR8hPy3M3APUZjl5K4cMZR/xaMQeQRZ5CHpS4DBKURKAHC0ltS5o
1503+ uBJKQOZm5iltJp15cgyIkBkGe8Mx18VFyVglAZey
1504+ =Y2oI
1505+ -----END PGP PUBLIC KEY BLOCK-----
1506+
1507+ # Custom gpg key:
1508+ # * As with keyid, a key may also be specified without a related source.
1509+ # * all other facts mentioned above still apply
1510+ # since this doesn't generate a source.list file the filename key is ignored
1511+ ignored4:
1512+ key: | # The value needs to start with -----BEGIN PGP PUBLIC KEY BLOCK-----
1513+ -----BEGIN PGP PUBLIC KEY BLOCK-----
1514+ Version: SKS 1.0.10
1515+
1516+ mI0ESpA3UQEEALdZKVIMq0j6qWAXAyxSlF63SvPVIgxHPb9Nk0DZUixn+akqytxG4zKCONz6
1517+ qLjoBBfHnynyVLfT4ihg9an1PqxRnTO+JKQxl8NgKGz6Pon569GtAOdWNKw15XKinJTDLjnj
1518+ 9y96ljJqRcpV9t/WsIcdJPcKFR5voHTEoABE2aEXABEBAAG0GUxhdW5jaHBhZCBQUEEgZm9y
1519+ IEFsZXN0aWOItgQTAQIAIAUCSpA3UQIbAwYLCQgHAwIEFQIIAwQWAgMBAh4BAheAAAoJEA7H
1520+ 5Qi+CcVxWZ8D/1MyYvfj3FJPZUm2Yo1zZsQ657vHI9+pPouqflWOayRR9jbiyUFIn0VdQBrP
1521+ t0FwvnOFArUovUWoKAEdqR8hPy3M3APUZjl5K4cMZR/xaMQeQRZ5CHpS4DBKURKAHC0ltS5o
1522+ uBJKQOZm5iltJp15cgyIkBkGe8Mx18VFyVglAZey
1523+ =Y2oI
1524+ -----END PGP PUBLIC KEY BLOCK-----
1525+ # end of apt dictionary
1526+
1527+
1528+ #
1529+ # All of the following are auxillary configurations to the apt handling
1530+ #
1531+
1532+ # Preserve existing /etc/apt/sources.list
1533+ # Default: True - do not overwrite sources_list. If staying at true
1534+ # then any "mirrors" configuration above will have no effect
1535+ # Set to False to affect sources.list with the configuration. Without only
1536+ # extra source specifications will be written into /etc/apt/sources.list.d/.
1537+ preserve_sources_list: false
1538+
1539+ # disable_suites by default it is an empty list, so nothing is removed.
1540+ # If given, those suites are removed from source.list after all other
1541+ # modifications have been made.
1542+ # suites are even disabled if no other modification was made,
1543+ # but not if is preserve_sources_list is active.
1544+ # There is a special alias “$RELEASE” as in the sources that will be replace
1545+ # by the matching release.
1546+ # To ease configuration and improve readability the following common ubuntu
1547+ # suites will be automatically mapped to their full definition.
1548+ # updates => $RELEASE-updates
1549+ # backports => $RELEASE-backports
1550+ # security => $RELEASE-security
1551+ # proposed => $RELEASE-proposed
1552+ # release => $RELEASE
1553+ # Note: Lines don’t get deleted, but disabled by being converted to a comment.
1554+ # The following example disables all defaults except $RELEASE-security. On top
1555+ # it disables a custom suite called "mysuite"
1556+ disable_suites: [$RELEASE-updates, backports, $RELEASE, mysuite]
1557+
1558+ # a custom (e.g. localized) mirror that will be set in sources.list and
1559+ # deb / deb-src lines
1560+ # one can set primary and security mirror to different uri's
1561+ # the child elements to the keys primary and secondary are equivalent
1562+ primary:
1563+ # arches is list of architectures the following config applies to
1564+ # the special keyword "default" applies to any architecture not explicitly
1565+ # listed.
1566+ - arches: [amd64, i386, default]
1567+ # uri is just defining the target as-is
1568+ uri: http://us.archive.ubuntu.com/ubuntu
1569+ #
1570+ # via search one can define lists that are
1571+ # tried one by one. The first with a working DNS resolution (or if it is an
1572+ # IP) will be picked. That way one can keep one configuration for multiple
1573+ # subenvironments that select the working one.
1574+ search:
1575+ - http://cool.but-sometimes-unreachable.com/ubuntu
1576+ - http://us.archive.ubuntu.com/ubuntu
1577+ #
1578+ # This will search for <distro>-mirror locally and at the fqdn of the system.
1579+ # If resolving that will be used as archive. That allows configuring local
1580+ # mirrors via providing certin DNS names via a local nameserver.
1581+ # These can even be set to resolve to the public names of defaults like
1582+ # archive.ubuntu.com as long as they are reachable from the target.
1583+ # For security it will search <distro>-security-mirror
1584+ search_dns: True
1585+ #
1586+ # If multiple of a category are given
1587+ # 1. uri
1588+ # 2. search
1589+ # 3. search_dns
1590+ # are given the first defining a valid mirror wins (in the
1591+ # order as defined here, not the one it is listed in the config).
1592+ - arches: [s390x, arm64]
1593+ # as above, allowing to have one config for different per arch mirrors
1594+ # security is optional, if not defined it is set to the value of primary
1595+ security:
1596+ uri: http://security.ubuntu.com/ubuntu
1597+ [...]
1598+ #
1599+ # if no mirrors are specified at all, or all lookups fail it will use:
1600+ # primary: http://archive.ubuntu.com/ubuntu
1601+ # security: http://security.ubuntu.com/ubuntu
1602+
1603+ # Provide a custom template for rendering sources.list
1604+ # without one provided curtin will try to modify the sources.list it find
1605+ # in the target at /etc/apt/.
1606+ # Within these source.list templates you can use the following replacement
1607+ # variables (all have sane ubuntu defaults, see above for details):
1608+ # $RELEASE, $MIRROR, $PRIMARY, $SECURITY
1609+ sources_list: | # written by curtin custom template
1610+ deb $MIRROR $RELEASE main restricted
1611+ deb-src $MIRROR $RELEASE main restricted
1612+ deb $PRIMARY $RELEASE universe restricted
1613+ deb $SECURITY $RELEASE-security multiverse
1614+
1615+ # any apt config string that will be made available to apt
1616+ # see the APT.CONF(5) man page for details what can be specified
1617+ conf: | # APT config
1618+ APT {
1619+ Get {
1620+ Assume-Yes "true";
1621+ Fix-Broken "true";
1622+ };
1623+ };
1624+
1625+ # proxies are the most common conf option that is set there is a
1626+ # shortcut via apt*proxy. Those get automatically translated into the
1627+ # correct Acquire::*::Proxy statements.
1628+ proxy: http://[[user][:pass]@]host[:port]/
1629+ http_proxy: http://[[user][:pass]@]host[:port]/
1630+ ftp_proxy: ftp://[[user][:pass]@]host[:port]/
1631+ https_proxy: https://[[user][:pass]@]host[:port]/
1632+ # note: proxy actually being a short synonym to http_proxy
1633+
1634+ # 'source' entries in apt-sources that match this python regex
1635+ # expression will be passed to add-apt-repository
1636+ # The following example is also the builtin default if nothing is specified
1637+ add_apt_repo_match: '^[\w-]+:\w'
1638
1639=== added file 'examples/tests/apt_config_command.yaml'
1640--- examples/tests/apt_config_command.yaml 1970-01-01 00:00:00 +0000
1641+++ examples/tests/apt_config_command.yaml 2016-07-26 19:53:43 +0000
1642@@ -0,0 +1,85 @@
1643+# This pushes curtin through a automatic installation
1644+# where no storage configuration is necessary.
1645+# exercising the standalong curtin apt-config command
1646+-placeholder_simple_install: unused
1647+bucket:
1648+ - &run_with_stdin |
1649+ #!/bin/sh
1650+ input="$1"
1651+ shift
1652+ printf "%s\n" "$input" | "$@"
1653+
1654+ - &run_apt_config_file |
1655+ #!/bin/sh
1656+ # take the first argument, write it to a tmp file and execute
1657+ # curtin apt --config=<tmpfile> "$@"
1658+ set -e
1659+ config="$1"
1660+ shift
1661+ TEMP_D=$(mktemp -d "${TMPDIR:-/tmp}/${0##*/}.XXXXXX")
1662+ trap cleanup EXIT
1663+ cleanup() { [ -z "${TEMP_D}" || rm -Rf "${TEMP_D}"; }
1664+ cfg_file="${TEMP_D}/curtin-apt.conf"
1665+ printf "%s\n" "$config" > "${cfg_file}"
1666+ curtin apt-config "--config=${cfg_file}" "$@"
1667+
1668+ - &apt_config_ppa |
1669+ # this is just a large string
1670+ apt:
1671+ sources:
1672+ ignored:
1673+ source: "ppa:curtin-dev/test-archive"
1674+ curtin-test1.list:
1675+ source: "deb $MIRROR $RELEASE-proposed main"
1676+
1677+ - &apt_config_source |
1678+ apt:
1679+ preserve_sources_list: false
1680+ sources:
1681+ ignored:
1682+ source: "ppa:curtin-dev/test-archive"
1683+ # apt-add-repositroy adds the key anyway, but lets pass that code
1684+ # path of adding once more in this scope
1685+ key: |
1686+ -----BEGIN PGP PUBLIC KEY BLOCK-----
1687+ Version: GnuPG v1
1688+
1689+ mQINBFazYtEBEADXrW53tDOvwcnHwchLapTKK89+wBWR2qQKXx5Mymtjkrb688Fs
1690+ ciXcCsvClnNGJ9bEhrJTucyb7WF0KcDVQcvOd0C4HOSEAc0DANBu1Mdp/tmCWuiW
1691+ 1TbbhomyHAcHNdbuSZeMDh5xi9M3DYPVq72PwYwjrE4lotVxHeX5nYEH304U+5nJ
1692+ tBNpVon91k3ItymQ6Jii+9gVoQ7ujiH1/Gw4/J/1/5zQ3C1mOjq68vLunz5iw1Kn
1693+ 7TMVyID6qwq2UFEgudpseLfFZcb/p7KgI0m3S/OViwzSc44m63ggTPMmbeHW51xA
1694+ 1rpUChSU+cm0cJ4tNtAcYHRYRltWAo/3J1OzB6Ut5P7vIC5r+QcCyyMbku9NjYaw
1695+ dWX4DDKqW3is3qJ/7EeOKPL4N8wuKwuWUC7s2wqsIZL8EmsvR+ZOnTJ3bHZFvsLg
1696+ p/OKqmhxMGYXiXOWDOEJ+vwboPxrvhD90JZl8weNGPnpla+EkxRDBSpEb31Vgt5X
1697+ AIoxE7XxwfuXS3MGMA7fSqkGPGHfSLYQFFk+CAIeTUV+ypKW94hIxXKgqRxa7dxz
1698+ Ymqs+wgIGaWJCnx7z1Kpd3HD9iTAYjyWyhlQ/Tjt43kwUBdALhTL0vYUTGQyTgKt
1699+ tAriVf5bqHb6Hj5PS5YZQ/+YoCUI2OTrAWWNyH9rIEZGsFc30oJFPHj3fQARAQAB
1700+ tCNMYXVuY2hwYWQgUFBBIGZvciBjdXJ0aW4gZGV2ZWxvcGVyc4kCOAQTAQIAIgUC
1701+ VrNi0QIbAwYLCQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQVf58jAFlAT4TGg//
1702+ SV7vWmkJqr5TSlT9JqCBfmtFjgudxTGG8XM2zwnta+m/3YVOMo0ZjyGL4fUKjCmN
1703+ eh6eYihwpRtfdawziaEOydDxNfdjwscV4Qcy7FjHX+DQnNzQyzK+WgWRJwNWloCw
1704+ skg2tF+EDRajalTRjHJAn+5zAilXVn71T/hhOCxkF0PBiH9s/e7pW/KcgBEC1MYV
1705+ Fs0fLST8SYhsIxttVRWuRkJDrtEY1zeVhkvk+PN6UuCY6/gyRSQ1rhhBF3ePqiba
1706+ CmLiUjnJMEm1OJOkuD33IMNPKQi99TZhr8y3AGCcrmAQtJsYLvVDPcsOsjGQHXP4
1707+ 2qQXK+jE/AAUycCQ6tgrAqCcUNQiClP8xUPkZOiDNvVMiPvIj/s79ShkoRaWLMb7
1708+ n9jyDOhs3L7dtmKQwHWq9qJ56fzx1L0/jxSanzm+ZJ/Q7t6E/GFxY1RsAk7xtI1C
1709+ SzSmrGKmtlbWlOyqqQb6zhULIJpaXvh/GaYyo0xI3rA+QvPDt/fgUJEBiSidwabW
1710+ Q8JU9iI5HXQxbVq1gSdy/z31fue5JuZSqjnjCjgho/UrXa4i1RPtqsY3FoTk7Hmo
1711+ C1z2cJc8HQI8JnEX/4qJXvPMRM2JsMD9DqvgsUJG5M9Qchy8cymYY+xeiBVYzJI+
1712+ WHCq6LHqnVxYZ+RM858lSsD6wetN44vguIjL3qJJ+wU=
1713+ curtin-test1.list:
1714+ source: "deb $MIRROR $RELEASE-proposed main"
1715+
1716+# into ephemeral environment
1717+early_commands:
1718+ 00_add_archive: [sh, -c, *run_with_stdin, "curtin-apt",
1719+ *apt_config_ppa, curtin, apt-config, --config=-, --target=/]
1720+ # tests itself by installing a packet only available in that ppa
1721+ 00_install_package: [apt-get, install, --assume-yes, smello]
1722+
1723+# into target environment
1724+late_commands:
1725+ 00_add_archive: [sh, -c, *run_apt_config_file, "curtin-apt-file",
1726+ *apt_config_source]
1727+ 00_install_package: [curtin, in-target, --, apt-get, install, --assume-yes, smello]
1728
1729=== added file 'examples/tests/apt_source_custom.yaml'
1730--- examples/tests/apt_source_custom.yaml 1970-01-01 00:00:00 +0000
1731+++ examples/tests/apt_source_custom.yaml 2016-07-26 19:53:43 +0000
1732@@ -0,0 +1,97 @@
1733+showtrace: true
1734+apt:
1735+ preserve_sources_list: false
1736+ primary:
1737+ - arches: [default]
1738+ uri: http://us.archive.ubuntu.com/ubuntu
1739+ security:
1740+ - arches: [default]
1741+ uri: http://security.ubuntu.com/ubuntu
1742+ sources_list: | # written by curtin custom template
1743+ deb $MIRROR $RELEASE main restricted
1744+ deb-src $MIRROR $RELEASE main restricted
1745+ deb $PRIMARY $RELEASE universe restricted
1746+ deb $SECURITY $RELEASE-security multiverse
1747+ # nice line to check in test
1748+ conf: | # APT config
1749+ ACQUIRE {
1750+ Retries "3";
1751+ };
1752+ sources:
1753+ curtin-dev-ppa.list:
1754+ source: "deb http://ppa.launchpad.net/curtin-dev/test-archive/ubuntu xenial main"
1755+ keyid: F430BBA5
1756+ ignored1:
1757+ source: "ppa:curtin-dev/test-archive"
1758+ my-repo2.list:
1759+ source: deb $MIRROR $RELEASE multiverse
1760+ ignored3:
1761+ keyid: 0E72 9061 0D2F 6DC4 D65E A921 9A31 4EC5 F470 A0AC
1762+ my-repo4.list:
1763+ source: deb http://ppa.launchpad.net/curtin-dev/test-archive/ubuntu xenial main
1764+ key: |
1765+ -----BEGIN PGP PUBLIC KEY BLOCK-----
1766+ Version: GnuPG v1
1767+
1768+ mQINBFXJ3NcBEAC85PMdaKdItkdjCT1vRJrdwNqj4lN5mu6z4dDVfeZlmozRDBGb
1769+ ENSOWCiYz3meANO7bKthQQCqAETSBV72rrDCqFZUpXeyG3zCN98Z/UdJ8zpQD9uw
1770+ mq2CaAqWMk6ty+PkHQ4gtIc390lGfRbHNoZ5HaWJNVOK7FCB2hBmnTZW7AViYiYa
1771+ YswOjYxaCkwQ/DsMOPD7S5OjwbLucs2YGjkBm7YF1nnXNzyt+BwieKQW/sQ2+ga1
1772+ mkgLW1BTQN3+JreBpeHy/yrRdK4dOZZUar4WPZitZzOW2eNpaaf6hKNA14LB/96a
1773+ tEguK8VazoqSQGvNV/R3PjIYmurVP3/Z9bEVgOKhMCflgwKCYgx+tBUypN3zFWv9
1774+ pgVq3iHx1MFCvoP9FsNB7I6jzOxlQP4z25BzR3ympx/QexkFw5CBFXhdrU+qNVBl
1775+ SSnz69aLEjCRXqBOnQEr0irs/e/35+yLJdEuw89vSwWwrzbV5r1Y7uxinEGWSydT
1776+ qddj97uKOWeMmnp20Be4+nhDDW/BMiTFI4Y3bYeDTrftbWMaSEmtSTw5HHxtAFtg
1777+ X9Hyx0Q3eN1w3gRZgIdm0xYTe7bNTofFRdfXzB/9wtNIcaW10+IlODShFHPCnh+d
1778+ i56a8LCdZcXiiLfCIhEcnqmM37BVvhjIQKSyOU1eMEgX148aVEz36OVuMwARAQAB
1779+ tCdDaHJpc3RpYW4gRWhyaGFyZHQgPGNwYWVsemVyQGdtYWlsLmNvbT6JAjgEEwEC
1780+ ACIFAlXJ3NcCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJELo+KTOCgLJC
1781+ BugP/ir0ES3wCzvHMnkz2UlXt9FR4KqY0L9uFmwu9VYpmfAploEVIOi2HcuxpcRp
1782+ hgoQlUtkz3lRhUeZzCxuB1ljM2JKTJiezP1tFTTGCbVYhPyA0LmUiHDWylG7FzPb
1783+ TX96HY/G0jf+m4CfR8q3HNHjeDi4VeA2ppBxdHcVE5I7HihDgRPJd+CvCa3nYdAb
1784+ nXDKlQZz5aZc7AgrRVamr4mshkzWuwNNCwOt3AIgHDkU/HzA5xlXfwHxOoP6scWH
1785+ /T7vFsd/vOikBphGseWPgKm6w1zyQ5Dk/wjRL8UeSJZW+Rh4PuBMbxg01lAZpPTq
1786+ tu/bePeNty3g5bhwO6oHMpWhprn3dO37R680qo6UnBPzICeuBUnSYgpPnsQC9maz
1787+ FEjiBtMsXSanU5vww7TpxY1JHjk5KFcmKx4sBeablznsm+GuVaDFN8R4eDjrM14r
1788+ SOzA9cV0bSQr4dMqA9fZFSx6qLTacIeMfptybW3zaDX/pJOeBBWRAtoAfZIFbBnu
1789+ /ZxDDgiQtZzpVK4UkYk5rjjtV/CPVXx64AnTHi35YfUn14KkE+k3odHdvPfBiv9+
1790+ NxfkTuV/koOgpD3+lTIYXyVHS9gwvhfRD/YfdrnVGl7bRZe68j7bfWDuQuSqIhSA
1791+ jpeJslJCawnqv6fVB6buj6jjcgHIxqCVn99chaPFSblEIPfXtDVDaHJpc3RpYW4g
1792+ RWhyaGFyZHQgPGNocmlzdGlhbi5laHJoYXJkdEBjYW5vbmljYWwuY29tPokCOAQT
1793+ AQIAIgUCVsbUOgIbAwYLCQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQuj4pM4KA
1794+ skJNPg/7BF/iXHHdSBigWKXCCvQz58uInoc/R4beIegxRCMq7wkYEey4B7Fd35zY
1795+ zv9CBOTV3hZePMCg9jxl4ki2kSsrZSCIEJw4L/aXDtJtx3HT18uTW0QKoU3nK/ro
1796+ OtthVqBqmiSEi40UUU+5MGrUjwLSm+PjaaSapjK/lddf0KbXBB78/BtR/XT0gxWM
1797+ +o68Oei9Nj1S3h6UndJwNAQ1xaDWmU2T7CRJet3F+cXZd3aDuS2axOTSTZbraSq7
1798+ zdl1xUiKtzXZIp8X1ewne+dzkewZuWj7DOwOBEFK26UhxCjKd5mUr7jpWQ4ampFX
1799+ 6xfd/MK8SJFY+iHOBKyzq9po40tE23dqWuaHB+T3MxOgQ9JHCo9x22XNvEuKZW/V
1800+ 4WaoGHVkR+jtWNC8Qv/xCMHL3CEvAklKJR68WDhozwUYTgNt5vCoJOviMlbhDSwf
1801+ 0zVXpQwMR//4c0QSA0+BPpIEPDnx5vTIHBVXHy4bBBHU2Vi87QIDS0AtiBpNcspN
1802+ 6AG0ktuldkE/pqfSTJ2A9HpHZyU+8boagRS5/z102Pjtmf/mzUkcHmfRb9o0DE15
1803+ X5fqpA3lYyx9eHIAgH4eaB1+G20Ez/EY5hr8IMS2nNBSem491UW6DXDYRu6eBLrR
1804+ sRmtrJ6DlTZFRFlqVZ47bce/SbeM/xljvRkBxWG6RtDRsTyNVI65Ag0EVcnc1wEQ
1805+ ANzk9W058tSHqf05UEtJGrN0K8DLriCvPd7QdFA8yVIZM3WD+m0AMBGXjd8BT5c2
1806+ lt0GmhB8klonHZvPiVLTRTLcSsc3NBopr1HL1bWsgOczwWiXSrc62oGAHUOQT/bv
1807+ vS6KIkZgez+qtCo/DCOGJrADaoJBiBCLSsZgowpzazZZDPUF7rAsfcryVCFvftK0
1808+ wAe1OdvUG77NHrMrE1oX3zh82hTqR5azBre6Y81lNwxxug/Xl/RHjNhEOYohcsLS
1809+ /xl0m2X831fHzcGGpoISRgrfel+M4RoC7KsLrwVhrF8koCD/ZQlevfLpuRl5LNpO
1810+ s1ZtEi8ZvLliih+H+BOlBD0zUc3zZrrks/NCpm1eZba0Z6L48r4TIHW08SGlHx7o
1811+ SrXgkq3mtoM8C4uDiLwjav5KxiF7n68s/9LF82aAr7YjNXd+xYZNjsmmFlYj9CGI
1812+ lL4jVt4v4EtTONa6pbtCNv5ezOLDZ6BBcQ36xdkrWzdpjQjL2mnh3sqIAGIPu7tH
1813+ N8euQ5L1zIvIjVqYlR1eJssp96QYPWYxF7TosfML4BUhCP631IWfuD9X/K2LzDmv
1814+ B2gVZo9fbhSC+P7GYVG+tV4VLAMbspAxRXXL69+j98aeV5g59f8OFQPbGpKE/SAY
1815+ eIXtq8DD+PYUXXq3VUI2brVLv42LBVdSJpKNKG3decIBABEBAAGJAh8EGAECAAkF
1816+ AlXJ3NcCGwwACgkQuj4pM4KAskKzeg/9FxXJLV3eWwY4nn2VhwYTHnHtSUpi8usk
1817+ RzIa3Mcj6OEVjU2LZaT3UQF8h6dLM9y+CemcwyjMqm1RQ5+ogfrItby1AaBXwCvm
1818+ XCUGw2zFOAnyzSHHoDFj27sllFxDmfSiBY5KP8M+/ywHKZDkRb6EjzMPx5oKFeGW
1819+ Hmqaj5FDmTeWChSIHd1ZxobashFauOZDbS/ijRRMsVGFulU2Nb/4QJK73g3orfhY
1820+ 5mq1TMkQ5Kcbqh4OmYYYayLtJQcpa6ZVopaRhAJFe30P83zW9pM5LQDpP9JIyY+S
1821+ DjasEY4ekYtw6oCKAjpqlwaaNDjl27OkJ7R7laFKy4grZ2TSB/2KTjn/Ea3CH/pA
1822+ SrpVis1LvC90XytbBnsEKYXU55H943wmBc6oj+itQhx4WyIiv+UgtHI/DbnYbUru
1823+ 71wpfapqGBXYfu/zAra8PITngOFuizeYu+idemu55ANO3keJPKr3ZBUSBBpNFauT
1824+ VUUCSnrLt+kpSLopYESiNdsPW/aQTFgFvA4BkBJTIMQsQZXicuXUePYlg5xFzXOv
1825+ XgiqkjRA9xBI5JAIUgLRk3ulVFt2bIsTG9XgtGyphEs86Q0MOIMo0WbZGtAYDrZO
1826+ DITbm2KzVLGVLn/ZJiW11RSHPNiwgg66/puKdFWrSogYYDJdDEUJtLIhypZ+ORxe
1827+ 7oh88hTkC1w=
1828+ =UNSw
1829+ -----END PGP PUBLIC KEY BLOCK-----
1830
1831=== added file 'examples/tests/apt_source_modify.yaml'
1832--- examples/tests/apt_source_modify.yaml 1970-01-01 00:00:00 +0000
1833+++ examples/tests/apt_source_modify.yaml 2016-07-26 19:53:43 +0000
1834@@ -0,0 +1,92 @@
1835+showtrace: true
1836+apt:
1837+ preserve_sources_list: false
1838+ primary:
1839+ - arches: [default]
1840+ uri: http://us.archive.ubuntu.com/ubuntu
1841+ security:
1842+ - arches: [default]
1843+ uri: http://security.ubuntu.com/ubuntu
1844+ conf: | # APT config
1845+ ACQUIRE {
1846+ Retries "3";
1847+ };
1848+ sources:
1849+ curtin-dev-ppa.list:
1850+ source: "deb http://ppa.launchpad.net/curtin-dev/test-archive/ubuntu xenial main"
1851+ keyid: F430BBA5
1852+ ignored1:
1853+ source: "ppa:curtin-dev/test-archive"
1854+ # intentionally dropped the .list here, has to be added by the code
1855+ my-repo2:
1856+ source: deb $MIRROR $RELEASE multiverse
1857+ ignored3:
1858+ keyid: 0E72 9061 0D2F 6DC4 D65E A921 9A31 4EC5 F470 A0AC
1859+ my-repo4.list:
1860+ source: deb http://ppa.launchpad.net/curtin-dev/test-archive/ubuntu xenial main
1861+ key: |
1862+ -----BEGIN PGP PUBLIC KEY BLOCK-----
1863+ Version: GnuPG v1
1864+
1865+ mQINBFXJ3NcBEAC85PMdaKdItkdjCT1vRJrdwNqj4lN5mu6z4dDVfeZlmozRDBGb
1866+ ENSOWCiYz3meANO7bKthQQCqAETSBV72rrDCqFZUpXeyG3zCN98Z/UdJ8zpQD9uw
1867+ mq2CaAqWMk6ty+PkHQ4gtIc390lGfRbHNoZ5HaWJNVOK7FCB2hBmnTZW7AViYiYa
1868+ YswOjYxaCkwQ/DsMOPD7S5OjwbLucs2YGjkBm7YF1nnXNzyt+BwieKQW/sQ2+ga1
1869+ mkgLW1BTQN3+JreBpeHy/yrRdK4dOZZUar4WPZitZzOW2eNpaaf6hKNA14LB/96a
1870+ tEguK8VazoqSQGvNV/R3PjIYmurVP3/Z9bEVgOKhMCflgwKCYgx+tBUypN3zFWv9
1871+ pgVq3iHx1MFCvoP9FsNB7I6jzOxlQP4z25BzR3ympx/QexkFw5CBFXhdrU+qNVBl
1872+ SSnz69aLEjCRXqBOnQEr0irs/e/35+yLJdEuw89vSwWwrzbV5r1Y7uxinEGWSydT
1873+ qddj97uKOWeMmnp20Be4+nhDDW/BMiTFI4Y3bYeDTrftbWMaSEmtSTw5HHxtAFtg
1874+ X9Hyx0Q3eN1w3gRZgIdm0xYTe7bNTofFRdfXzB/9wtNIcaW10+IlODShFHPCnh+d
1875+ i56a8LCdZcXiiLfCIhEcnqmM37BVvhjIQKSyOU1eMEgX148aVEz36OVuMwARAQAB
1876+ tCdDaHJpc3RpYW4gRWhyaGFyZHQgPGNwYWVsemVyQGdtYWlsLmNvbT6JAjgEEwEC
1877+ ACIFAlXJ3NcCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJELo+KTOCgLJC
1878+ BugP/ir0ES3wCzvHMnkz2UlXt9FR4KqY0L9uFmwu9VYpmfAploEVIOi2HcuxpcRp
1879+ hgoQlUtkz3lRhUeZzCxuB1ljM2JKTJiezP1tFTTGCbVYhPyA0LmUiHDWylG7FzPb
1880+ TX96HY/G0jf+m4CfR8q3HNHjeDi4VeA2ppBxdHcVE5I7HihDgRPJd+CvCa3nYdAb
1881+ nXDKlQZz5aZc7AgrRVamr4mshkzWuwNNCwOt3AIgHDkU/HzA5xlXfwHxOoP6scWH
1882+ /T7vFsd/vOikBphGseWPgKm6w1zyQ5Dk/wjRL8UeSJZW+Rh4PuBMbxg01lAZpPTq
1883+ tu/bePeNty3g5bhwO6oHMpWhprn3dO37R680qo6UnBPzICeuBUnSYgpPnsQC9maz
1884+ FEjiBtMsXSanU5vww7TpxY1JHjk5KFcmKx4sBeablznsm+GuVaDFN8R4eDjrM14r
1885+ SOzA9cV0bSQr4dMqA9fZFSx6qLTacIeMfptybW3zaDX/pJOeBBWRAtoAfZIFbBnu
1886+ /ZxDDgiQtZzpVK4UkYk5rjjtV/CPVXx64AnTHi35YfUn14KkE+k3odHdvPfBiv9+
1887+ NxfkTuV/koOgpD3+lTIYXyVHS9gwvhfRD/YfdrnVGl7bRZe68j7bfWDuQuSqIhSA
1888+ jpeJslJCawnqv6fVB6buj6jjcgHIxqCVn99chaPFSblEIPfXtDVDaHJpc3RpYW4g
1889+ RWhyaGFyZHQgPGNocmlzdGlhbi5laHJoYXJkdEBjYW5vbmljYWwuY29tPokCOAQT
1890+ AQIAIgUCVsbUOgIbAwYLCQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQuj4pM4KA
1891+ skJNPg/7BF/iXHHdSBigWKXCCvQz58uInoc/R4beIegxRCMq7wkYEey4B7Fd35zY
1892+ zv9CBOTV3hZePMCg9jxl4ki2kSsrZSCIEJw4L/aXDtJtx3HT18uTW0QKoU3nK/ro
1893+ OtthVqBqmiSEi40UUU+5MGrUjwLSm+PjaaSapjK/lddf0KbXBB78/BtR/XT0gxWM
1894+ +o68Oei9Nj1S3h6UndJwNAQ1xaDWmU2T7CRJet3F+cXZd3aDuS2axOTSTZbraSq7
1895+ zdl1xUiKtzXZIp8X1ewne+dzkewZuWj7DOwOBEFK26UhxCjKd5mUr7jpWQ4ampFX
1896+ 6xfd/MK8SJFY+iHOBKyzq9po40tE23dqWuaHB+T3MxOgQ9JHCo9x22XNvEuKZW/V
1897+ 4WaoGHVkR+jtWNC8Qv/xCMHL3CEvAklKJR68WDhozwUYTgNt5vCoJOviMlbhDSwf
1898+ 0zVXpQwMR//4c0QSA0+BPpIEPDnx5vTIHBVXHy4bBBHU2Vi87QIDS0AtiBpNcspN
1899+ 6AG0ktuldkE/pqfSTJ2A9HpHZyU+8boagRS5/z102Pjtmf/mzUkcHmfRb9o0DE15
1900+ X5fqpA3lYyx9eHIAgH4eaB1+G20Ez/EY5hr8IMS2nNBSem491UW6DXDYRu6eBLrR
1901+ sRmtrJ6DlTZFRFlqVZ47bce/SbeM/xljvRkBxWG6RtDRsTyNVI65Ag0EVcnc1wEQ
1902+ ANzk9W058tSHqf05UEtJGrN0K8DLriCvPd7QdFA8yVIZM3WD+m0AMBGXjd8BT5c2
1903+ lt0GmhB8klonHZvPiVLTRTLcSsc3NBopr1HL1bWsgOczwWiXSrc62oGAHUOQT/bv
1904+ vS6KIkZgez+qtCo/DCOGJrADaoJBiBCLSsZgowpzazZZDPUF7rAsfcryVCFvftK0
1905+ wAe1OdvUG77NHrMrE1oX3zh82hTqR5azBre6Y81lNwxxug/Xl/RHjNhEOYohcsLS
1906+ /xl0m2X831fHzcGGpoISRgrfel+M4RoC7KsLrwVhrF8koCD/ZQlevfLpuRl5LNpO
1907+ s1ZtEi8ZvLliih+H+BOlBD0zUc3zZrrks/NCpm1eZba0Z6L48r4TIHW08SGlHx7o
1908+ SrXgkq3mtoM8C4uDiLwjav5KxiF7n68s/9LF82aAr7YjNXd+xYZNjsmmFlYj9CGI
1909+ lL4jVt4v4EtTONa6pbtCNv5ezOLDZ6BBcQ36xdkrWzdpjQjL2mnh3sqIAGIPu7tH
1910+ N8euQ5L1zIvIjVqYlR1eJssp96QYPWYxF7TosfML4BUhCP631IWfuD9X/K2LzDmv
1911+ B2gVZo9fbhSC+P7GYVG+tV4VLAMbspAxRXXL69+j98aeV5g59f8OFQPbGpKE/SAY
1912+ eIXtq8DD+PYUXXq3VUI2brVLv42LBVdSJpKNKG3decIBABEBAAGJAh8EGAECAAkF
1913+ AlXJ3NcCGwwACgkQuj4pM4KAskKzeg/9FxXJLV3eWwY4nn2VhwYTHnHtSUpi8usk
1914+ RzIa3Mcj6OEVjU2LZaT3UQF8h6dLM9y+CemcwyjMqm1RQ5+ogfrItby1AaBXwCvm
1915+ XCUGw2zFOAnyzSHHoDFj27sllFxDmfSiBY5KP8M+/ywHKZDkRb6EjzMPx5oKFeGW
1916+ Hmqaj5FDmTeWChSIHd1ZxobashFauOZDbS/ijRRMsVGFulU2Nb/4QJK73g3orfhY
1917+ 5mq1TMkQ5Kcbqh4OmYYYayLtJQcpa6ZVopaRhAJFe30P83zW9pM5LQDpP9JIyY+S
1918+ DjasEY4ekYtw6oCKAjpqlwaaNDjl27OkJ7R7laFKy4grZ2TSB/2KTjn/Ea3CH/pA
1919+ SrpVis1LvC90XytbBnsEKYXU55H943wmBc6oj+itQhx4WyIiv+UgtHI/DbnYbUru
1920+ 71wpfapqGBXYfu/zAra8PITngOFuizeYu+idemu55ANO3keJPKr3ZBUSBBpNFauT
1921+ VUUCSnrLt+kpSLopYESiNdsPW/aQTFgFvA4BkBJTIMQsQZXicuXUePYlg5xFzXOv
1922+ XgiqkjRA9xBI5JAIUgLRk3ulVFt2bIsTG9XgtGyphEs86Q0MOIMo0WbZGtAYDrZO
1923+ DITbm2KzVLGVLn/ZJiW11RSHPNiwgg66/puKdFWrSogYYDJdDEUJtLIhypZ+ORxe
1924+ 7oh88hTkC1w=
1925+ =UNSw
1926+ -----END PGP PUBLIC KEY BLOCK-----
1927
1928=== added file 'examples/tests/apt_source_modify_arches.yaml'
1929--- examples/tests/apt_source_modify_arches.yaml 1970-01-01 00:00:00 +0000
1930+++ examples/tests/apt_source_modify_arches.yaml 2016-07-26 19:53:43 +0000
1931@@ -0,0 +1,102 @@
1932+showtrace: true
1933+apt:
1934+ preserve_sources_list: false
1935+ primary:
1936+ # we don't know on which arch this will run, so we can't put the "right"
1937+ # config in an arch, but we can provide various confusing alternatives
1938+ # and orders and it has to pick default out of them
1939+ - arches: [x86_2048, x86_4096, x86_8192, amd18.5, "foobar"]
1940+ uri: http://notthis.com/ubuntu
1941+ - arches: ["*"]
1942+ uri: http://notthis.com/ubuntu
1943+ - arches: [default]
1944+ uri: http://us.archive.ubuntu.com/ubuntu
1945+ - arches: []
1946+ uri: http://notthis.com/ubuntu
1947+ security:
1948+ - arches: [default]
1949+ uri: http://security.ubuntu.com/ubuntu
1950+ - arches: ["supersecurearchthatdoesnexist"]
1951+ uri: http://notthat.com/ubuntu
1952+ conf: | # APT config
1953+ ACQUIRE {
1954+ Retries "3";
1955+ };
1956+ sources:
1957+ curtin-dev-ppa.list:
1958+ source: "deb http://ppa.launchpad.net/curtin-dev/test-archive/ubuntu xenial main"
1959+ keyid: F430BBA5
1960+ ignored1:
1961+ source: "ppa:curtin-dev/test-archive"
1962+ my-repo2.list:
1963+ source: deb $MIRROR $RELEASE multiverse
1964+ ignored3:
1965+ keyid: 0E72 9061 0D2F 6DC4 D65E A921 9A31 4EC5 F470 A0AC
1966+ my-repo4.list:
1967+ source: deb http://ppa.launchpad.net/curtin-dev/test-archive/ubuntu xenial main
1968+ key: |
1969+ -----BEGIN PGP PUBLIC KEY BLOCK-----
1970+ Version: GnuPG v1
1971+
1972+ mQINBFXJ3NcBEAC85PMdaKdItkdjCT1vRJrdwNqj4lN5mu6z4dDVfeZlmozRDBGb
1973+ ENSOWCiYz3meANO7bKthQQCqAETSBV72rrDCqFZUpXeyG3zCN98Z/UdJ8zpQD9uw
1974+ mq2CaAqWMk6ty+PkHQ4gtIc390lGfRbHNoZ5HaWJNVOK7FCB2hBmnTZW7AViYiYa
1975+ YswOjYxaCkwQ/DsMOPD7S5OjwbLucs2YGjkBm7YF1nnXNzyt+BwieKQW/sQ2+ga1
1976+ mkgLW1BTQN3+JreBpeHy/yrRdK4dOZZUar4WPZitZzOW2eNpaaf6hKNA14LB/96a
1977+ tEguK8VazoqSQGvNV/R3PjIYmurVP3/Z9bEVgOKhMCflgwKCYgx+tBUypN3zFWv9
1978+ pgVq3iHx1MFCvoP9FsNB7I6jzOxlQP4z25BzR3ympx/QexkFw5CBFXhdrU+qNVBl
1979+ SSnz69aLEjCRXqBOnQEr0irs/e/35+yLJdEuw89vSwWwrzbV5r1Y7uxinEGWSydT
1980+ qddj97uKOWeMmnp20Be4+nhDDW/BMiTFI4Y3bYeDTrftbWMaSEmtSTw5HHxtAFtg
1981+ X9Hyx0Q3eN1w3gRZgIdm0xYTe7bNTofFRdfXzB/9wtNIcaW10+IlODShFHPCnh+d
1982+ i56a8LCdZcXiiLfCIhEcnqmM37BVvhjIQKSyOU1eMEgX148aVEz36OVuMwARAQAB
1983+ tCdDaHJpc3RpYW4gRWhyaGFyZHQgPGNwYWVsemVyQGdtYWlsLmNvbT6JAjgEEwEC
1984+ ACIFAlXJ3NcCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJELo+KTOCgLJC
1985+ BugP/ir0ES3wCzvHMnkz2UlXt9FR4KqY0L9uFmwu9VYpmfAploEVIOi2HcuxpcRp
1986+ hgoQlUtkz3lRhUeZzCxuB1ljM2JKTJiezP1tFTTGCbVYhPyA0LmUiHDWylG7FzPb
1987+ TX96HY/G0jf+m4CfR8q3HNHjeDi4VeA2ppBxdHcVE5I7HihDgRPJd+CvCa3nYdAb
1988+ nXDKlQZz5aZc7AgrRVamr4mshkzWuwNNCwOt3AIgHDkU/HzA5xlXfwHxOoP6scWH
1989+ /T7vFsd/vOikBphGseWPgKm6w1zyQ5Dk/wjRL8UeSJZW+Rh4PuBMbxg01lAZpPTq
1990+ tu/bePeNty3g5bhwO6oHMpWhprn3dO37R680qo6UnBPzICeuBUnSYgpPnsQC9maz
1991+ FEjiBtMsXSanU5vww7TpxY1JHjk5KFcmKx4sBeablznsm+GuVaDFN8R4eDjrM14r
1992+ SOzA9cV0bSQr4dMqA9fZFSx6qLTacIeMfptybW3zaDX/pJOeBBWRAtoAfZIFbBnu
1993+ /ZxDDgiQtZzpVK4UkYk5rjjtV/CPVXx64AnTHi35YfUn14KkE+k3odHdvPfBiv9+
1994+ NxfkTuV/koOgpD3+lTIYXyVHS9gwvhfRD/YfdrnVGl7bRZe68j7bfWDuQuSqIhSA
1995+ jpeJslJCawnqv6fVB6buj6jjcgHIxqCVn99chaPFSblEIPfXtDVDaHJpc3RpYW4g
1996+ RWhyaGFyZHQgPGNocmlzdGlhbi5laHJoYXJkdEBjYW5vbmljYWwuY29tPokCOAQT
1997+ AQIAIgUCVsbUOgIbAwYLCQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQuj4pM4KA
1998+ skJNPg/7BF/iXHHdSBigWKXCCvQz58uInoc/R4beIegxRCMq7wkYEey4B7Fd35zY
1999+ zv9CBOTV3hZePMCg9jxl4ki2kSsrZSCIEJw4L/aXDtJtx3HT18uTW0QKoU3nK/ro
2000+ OtthVqBqmiSEi40UUU+5MGrUjwLSm+PjaaSapjK/lddf0KbXBB78/BtR/XT0gxWM
2001+ +o68Oei9Nj1S3h6UndJwNAQ1xaDWmU2T7CRJet3F+cXZd3aDuS2axOTSTZbraSq7
2002+ zdl1xUiKtzXZIp8X1ewne+dzkewZuWj7DOwOBEFK26UhxCjKd5mUr7jpWQ4ampFX
2003+ 6xfd/MK8SJFY+iHOBKyzq9po40tE23dqWuaHB+T3MxOgQ9JHCo9x22XNvEuKZW/V
2004+ 4WaoGHVkR+jtWNC8Qv/xCMHL3CEvAklKJR68WDhozwUYTgNt5vCoJOviMlbhDSwf
2005+ 0zVXpQwMR//4c0QSA0+BPpIEPDnx5vTIHBVXHy4bBBHU2Vi87QIDS0AtiBpNcspN
2006+ 6AG0ktuldkE/pqfSTJ2A9HpHZyU+8boagRS5/z102Pjtmf/mzUkcHmfRb9o0DE15
2007+ X5fqpA3lYyx9eHIAgH4eaB1+G20Ez/EY5hr8IMS2nNBSem491UW6DXDYRu6eBLrR
2008+ sRmtrJ6DlTZFRFlqVZ47bce/SbeM/xljvRkBxWG6RtDRsTyNVI65Ag0EVcnc1wEQ
2009+ ANzk9W058tSHqf05UEtJGrN0K8DLriCvPd7QdFA8yVIZM3WD+m0AMBGXjd8BT5c2
2010+ lt0GmhB8klonHZvPiVLTRTLcSsc3NBopr1HL1bWsgOczwWiXSrc62oGAHUOQT/bv
2011+ vS6KIkZgez+qtCo/DCOGJrADaoJBiBCLSsZgowpzazZZDPUF7rAsfcryVCFvftK0
2012+ wAe1OdvUG77NHrMrE1oX3zh82hTqR5azBre6Y81lNwxxug/Xl/RHjNhEOYohcsLS
2013+ /xl0m2X831fHzcGGpoISRgrfel+M4RoC7KsLrwVhrF8koCD/ZQlevfLpuRl5LNpO
2014+ s1ZtEi8ZvLliih+H+BOlBD0zUc3zZrrks/NCpm1eZba0Z6L48r4TIHW08SGlHx7o
2015+ SrXgkq3mtoM8C4uDiLwjav5KxiF7n68s/9LF82aAr7YjNXd+xYZNjsmmFlYj9CGI
2016+ lL4jVt4v4EtTONa6pbtCNv5ezOLDZ6BBcQ36xdkrWzdpjQjL2mnh3sqIAGIPu7tH
2017+ N8euQ5L1zIvIjVqYlR1eJssp96QYPWYxF7TosfML4BUhCP631IWfuD9X/K2LzDmv
2018+ B2gVZo9fbhSC+P7GYVG+tV4VLAMbspAxRXXL69+j98aeV5g59f8OFQPbGpKE/SAY
2019+ eIXtq8DD+PYUXXq3VUI2brVLv42LBVdSJpKNKG3decIBABEBAAGJAh8EGAECAAkF
2020+ AlXJ3NcCGwwACgkQuj4pM4KAskKzeg/9FxXJLV3eWwY4nn2VhwYTHnHtSUpi8usk
2021+ RzIa3Mcj6OEVjU2LZaT3UQF8h6dLM9y+CemcwyjMqm1RQ5+ogfrItby1AaBXwCvm
2022+ XCUGw2zFOAnyzSHHoDFj27sllFxDmfSiBY5KP8M+/ywHKZDkRb6EjzMPx5oKFeGW
2023+ Hmqaj5FDmTeWChSIHd1ZxobashFauOZDbS/ijRRMsVGFulU2Nb/4QJK73g3orfhY
2024+ 5mq1TMkQ5Kcbqh4OmYYYayLtJQcpa6ZVopaRhAJFe30P83zW9pM5LQDpP9JIyY+S
2025+ DjasEY4ekYtw6oCKAjpqlwaaNDjl27OkJ7R7laFKy4grZ2TSB/2KTjn/Ea3CH/pA
2026+ SrpVis1LvC90XytbBnsEKYXU55H943wmBc6oj+itQhx4WyIiv+UgtHI/DbnYbUru
2027+ 71wpfapqGBXYfu/zAra8PITngOFuizeYu+idemu55ANO3keJPKr3ZBUSBBpNFauT
2028+ VUUCSnrLt+kpSLopYESiNdsPW/aQTFgFvA4BkBJTIMQsQZXicuXUePYlg5xFzXOv
2029+ XgiqkjRA9xBI5JAIUgLRk3ulVFt2bIsTG9XgtGyphEs86Q0MOIMo0WbZGtAYDrZO
2030+ DITbm2KzVLGVLn/ZJiW11RSHPNiwgg66/puKdFWrSogYYDJdDEUJtLIhypZ+ORxe
2031+ 7oh88hTkC1w=
2032+ =UNSw
2033+ -----END PGP PUBLIC KEY BLOCK-----
2034
2035=== added file 'examples/tests/apt_source_modify_disable_suite.yaml'
2036--- examples/tests/apt_source_modify_disable_suite.yaml 1970-01-01 00:00:00 +0000
2037+++ examples/tests/apt_source_modify_disable_suite.yaml 2016-07-26 19:53:43 +0000
2038@@ -0,0 +1,92 @@
2039+showtrace: true
2040+apt:
2041+ preserve_sources_list: false
2042+ primary:
2043+ - arches: [default]
2044+ uri: http://us.archive.ubuntu.com/ubuntu
2045+ security:
2046+ - arches: [default]
2047+ uri: http://security.ubuntu.com/ubuntu
2048+ disable_suites: [$RELEASE-updates]
2049+ conf: | # APT config
2050+ ACQUIRE {
2051+ Retries "3";
2052+ };
2053+ sources:
2054+ curtin-dev-ppa.list:
2055+ source: "deb http://ppa.launchpad.net/curtin-dev/test-archive/ubuntu xenial main"
2056+ keyid: F430BBA5
2057+ ignored1:
2058+ source: "ppa:curtin-dev/test-archive"
2059+ my-repo2.list:
2060+ source: deb $MIRROR $RELEASE multiverse
2061+ ignored3:
2062+ keyid: 0E72 9061 0D2F 6DC4 D65E A921 9A31 4EC5 F470 A0AC
2063+ my-repo4.list:
2064+ source: deb http://ppa.launchpad.net/curtin-dev/test-archive/ubuntu xenial main
2065+ key: |
2066+ -----BEGIN PGP PUBLIC KEY BLOCK-----
2067+ Version: GnuPG v1
2068+
2069+ mQINBFXJ3NcBEAC85PMdaKdItkdjCT1vRJrdwNqj4lN5mu6z4dDVfeZlmozRDBGb
2070+ ENSOWCiYz3meANO7bKthQQCqAETSBV72rrDCqFZUpXeyG3zCN98Z/UdJ8zpQD9uw
2071+ mq2CaAqWMk6ty+PkHQ4gtIc390lGfRbHNoZ5HaWJNVOK7FCB2hBmnTZW7AViYiYa
2072+ YswOjYxaCkwQ/DsMOPD7S5OjwbLucs2YGjkBm7YF1nnXNzyt+BwieKQW/sQ2+ga1
2073+ mkgLW1BTQN3+JreBpeHy/yrRdK4dOZZUar4WPZitZzOW2eNpaaf6hKNA14LB/96a
2074+ tEguK8VazoqSQGvNV/R3PjIYmurVP3/Z9bEVgOKhMCflgwKCYgx+tBUypN3zFWv9
2075+ pgVq3iHx1MFCvoP9FsNB7I6jzOxlQP4z25BzR3ympx/QexkFw5CBFXhdrU+qNVBl
2076+ SSnz69aLEjCRXqBOnQEr0irs/e/35+yLJdEuw89vSwWwrzbV5r1Y7uxinEGWSydT
2077+ qddj97uKOWeMmnp20Be4+nhDDW/BMiTFI4Y3bYeDTrftbWMaSEmtSTw5HHxtAFtg
2078+ X9Hyx0Q3eN1w3gRZgIdm0xYTe7bNTofFRdfXzB/9wtNIcaW10+IlODShFHPCnh+d
2079+ i56a8LCdZcXiiLfCIhEcnqmM37BVvhjIQKSyOU1eMEgX148aVEz36OVuMwARAQAB
2080+ tCdDaHJpc3RpYW4gRWhyaGFyZHQgPGNwYWVsemVyQGdtYWlsLmNvbT6JAjgEEwEC
2081+ ACIFAlXJ3NcCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJELo+KTOCgLJC
2082+ BugP/ir0ES3wCzvHMnkz2UlXt9FR4KqY0L9uFmwu9VYpmfAploEVIOi2HcuxpcRp
2083+ hgoQlUtkz3lRhUeZzCxuB1ljM2JKTJiezP1tFTTGCbVYhPyA0LmUiHDWylG7FzPb
2084+ TX96HY/G0jf+m4CfR8q3HNHjeDi4VeA2ppBxdHcVE5I7HihDgRPJd+CvCa3nYdAb
2085+ nXDKlQZz5aZc7AgrRVamr4mshkzWuwNNCwOt3AIgHDkU/HzA5xlXfwHxOoP6scWH
2086+ /T7vFsd/vOikBphGseWPgKm6w1zyQ5Dk/wjRL8UeSJZW+Rh4PuBMbxg01lAZpPTq
2087+ tu/bePeNty3g5bhwO6oHMpWhprn3dO37R680qo6UnBPzICeuBUnSYgpPnsQC9maz
2088+ FEjiBtMsXSanU5vww7TpxY1JHjk5KFcmKx4sBeablznsm+GuVaDFN8R4eDjrM14r
2089+ SOzA9cV0bSQr4dMqA9fZFSx6qLTacIeMfptybW3zaDX/pJOeBBWRAtoAfZIFbBnu
2090+ /ZxDDgiQtZzpVK4UkYk5rjjtV/CPVXx64AnTHi35YfUn14KkE+k3odHdvPfBiv9+
2091+ NxfkTuV/koOgpD3+lTIYXyVHS9gwvhfRD/YfdrnVGl7bRZe68j7bfWDuQuSqIhSA
2092+ jpeJslJCawnqv6fVB6buj6jjcgHIxqCVn99chaPFSblEIPfXtDVDaHJpc3RpYW4g
2093+ RWhyaGFyZHQgPGNocmlzdGlhbi5laHJoYXJkdEBjYW5vbmljYWwuY29tPokCOAQT
2094+ AQIAIgUCVsbUOgIbAwYLCQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQuj4pM4KA
2095+ skJNPg/7BF/iXHHdSBigWKXCCvQz58uInoc/R4beIegxRCMq7wkYEey4B7Fd35zY
2096+ zv9CBOTV3hZePMCg9jxl4ki2kSsrZSCIEJw4L/aXDtJtx3HT18uTW0QKoU3nK/ro
2097+ OtthVqBqmiSEi40UUU+5MGrUjwLSm+PjaaSapjK/lddf0KbXBB78/BtR/XT0gxWM
2098+ +o68Oei9Nj1S3h6UndJwNAQ1xaDWmU2T7CRJet3F+cXZd3aDuS2axOTSTZbraSq7
2099+ zdl1xUiKtzXZIp8X1ewne+dzkewZuWj7DOwOBEFK26UhxCjKd5mUr7jpWQ4ampFX
2100+ 6xfd/MK8SJFY+iHOBKyzq9po40tE23dqWuaHB+T3MxOgQ9JHCo9x22XNvEuKZW/V
2101+ 4WaoGHVkR+jtWNC8Qv/xCMHL3CEvAklKJR68WDhozwUYTgNt5vCoJOviMlbhDSwf
2102+ 0zVXpQwMR//4c0QSA0+BPpIEPDnx5vTIHBVXHy4bBBHU2Vi87QIDS0AtiBpNcspN
2103+ 6AG0ktuldkE/pqfSTJ2A9HpHZyU+8boagRS5/z102Pjtmf/mzUkcHmfRb9o0DE15
2104+ X5fqpA3lYyx9eHIAgH4eaB1+G20Ez/EY5hr8IMS2nNBSem491UW6DXDYRu6eBLrR
2105+ sRmtrJ6DlTZFRFlqVZ47bce/SbeM/xljvRkBxWG6RtDRsTyNVI65Ag0EVcnc1wEQ
2106+ ANzk9W058tSHqf05UEtJGrN0K8DLriCvPd7QdFA8yVIZM3WD+m0AMBGXjd8BT5c2
2107+ lt0GmhB8klonHZvPiVLTRTLcSsc3NBopr1HL1bWsgOczwWiXSrc62oGAHUOQT/bv
2108+ vS6KIkZgez+qtCo/DCOGJrADaoJBiBCLSsZgowpzazZZDPUF7rAsfcryVCFvftK0
2109+ wAe1OdvUG77NHrMrE1oX3zh82hTqR5azBre6Y81lNwxxug/Xl/RHjNhEOYohcsLS
2110+ /xl0m2X831fHzcGGpoISRgrfel+M4RoC7KsLrwVhrF8koCD/ZQlevfLpuRl5LNpO
2111+ s1ZtEi8ZvLliih+H+BOlBD0zUc3zZrrks/NCpm1eZba0Z6L48r4TIHW08SGlHx7o
2112+ SrXgkq3mtoM8C4uDiLwjav5KxiF7n68s/9LF82aAr7YjNXd+xYZNjsmmFlYj9CGI
2113+ lL4jVt4v4EtTONa6pbtCNv5ezOLDZ6BBcQ36xdkrWzdpjQjL2mnh3sqIAGIPu7tH
2114+ N8euQ5L1zIvIjVqYlR1eJssp96QYPWYxF7TosfML4BUhCP631IWfuD9X/K2LzDmv
2115+ B2gVZo9fbhSC+P7GYVG+tV4VLAMbspAxRXXL69+j98aeV5g59f8OFQPbGpKE/SAY
2116+ eIXtq8DD+PYUXXq3VUI2brVLv42LBVdSJpKNKG3decIBABEBAAGJAh8EGAECAAkF
2117+ AlXJ3NcCGwwACgkQuj4pM4KAskKzeg/9FxXJLV3eWwY4nn2VhwYTHnHtSUpi8usk
2118+ RzIa3Mcj6OEVjU2LZaT3UQF8h6dLM9y+CemcwyjMqm1RQ5+ogfrItby1AaBXwCvm
2119+ XCUGw2zFOAnyzSHHoDFj27sllFxDmfSiBY5KP8M+/ywHKZDkRb6EjzMPx5oKFeGW
2120+ Hmqaj5FDmTeWChSIHd1ZxobashFauOZDbS/ijRRMsVGFulU2Nb/4QJK73g3orfhY
2121+ 5mq1TMkQ5Kcbqh4OmYYYayLtJQcpa6ZVopaRhAJFe30P83zW9pM5LQDpP9JIyY+S
2122+ DjasEY4ekYtw6oCKAjpqlwaaNDjl27OkJ7R7laFKy4grZ2TSB/2KTjn/Ea3CH/pA
2123+ SrpVis1LvC90XytbBnsEKYXU55H943wmBc6oj+itQhx4WyIiv+UgtHI/DbnYbUru
2124+ 71wpfapqGBXYfu/zAra8PITngOFuizeYu+idemu55ANO3keJPKr3ZBUSBBpNFauT
2125+ VUUCSnrLt+kpSLopYESiNdsPW/aQTFgFvA4BkBJTIMQsQZXicuXUePYlg5xFzXOv
2126+ XgiqkjRA9xBI5JAIUgLRk3ulVFt2bIsTG9XgtGyphEs86Q0MOIMo0WbZGtAYDrZO
2127+ DITbm2KzVLGVLn/ZJiW11RSHPNiwgg66/puKdFWrSogYYDJdDEUJtLIhypZ+ORxe
2128+ 7oh88hTkC1w=
2129+ =UNSw
2130+ -----END PGP PUBLIC KEY BLOCK-----
2131
2132=== added file 'examples/tests/apt_source_preserve.yaml'
2133--- examples/tests/apt_source_preserve.yaml 1970-01-01 00:00:00 +0000
2134+++ examples/tests/apt_source_preserve.yaml 2016-07-26 19:53:43 +0000
2135@@ -0,0 +1,98 @@
2136+showtrace: true
2137+apt:
2138+ # this is like the other apt_source test but with preserve true
2139+ # this is the default now preserve_sources_list: true
2140+ primary:
2141+ - arches: [default]
2142+ uri: http://us.archive.ubuntu.com/ubuntu
2143+ security:
2144+ - arches: [default]
2145+ uri: http://security.ubuntu.com/ubuntu
2146+ sources_list: | # written by curtin custom template
2147+ deb $MIRROR $RELEASE main restricted
2148+ deb-src $MIRROR $RELEASE main restricted
2149+ deb $PRIMARY $RELEASE universe restricted
2150+ deb $SECURITY $RELEASE-security multiverse
2151+ # nice line to check in test
2152+ conf: | # APT config
2153+ ACQUIRE {
2154+ Retries "3";
2155+ };
2156+ sources:
2157+ curtin-dev-ppa.list:
2158+ source: "deb http://ppa.launchpad.net/curtin-dev/test-archive/ubuntu xenial main"
2159+ keyid: F430BBA5
2160+ ignored1:
2161+ source: "ppa:curtin-dev/test-archive"
2162+ my-repo2.list:
2163+ source: deb $MIRROR $RELEASE multiverse
2164+ ignored3:
2165+ keyid: 0E72 9061 0D2F 6DC4 D65E A921 9A31 4EC5 F470 A0AC
2166+ my-repo4.list:
2167+ source: deb http://ppa.launchpad.net/curtin-dev/test-archive/ubuntu xenial main
2168+ key: |
2169+ -----BEGIN PGP PUBLIC KEY BLOCK-----
2170+ Version: GnuPG v1
2171+
2172+ mQINBFXJ3NcBEAC85PMdaKdItkdjCT1vRJrdwNqj4lN5mu6z4dDVfeZlmozRDBGb
2173+ ENSOWCiYz3meANO7bKthQQCqAETSBV72rrDCqFZUpXeyG3zCN98Z/UdJ8zpQD9uw
2174+ mq2CaAqWMk6ty+PkHQ4gtIc390lGfRbHNoZ5HaWJNVOK7FCB2hBmnTZW7AViYiYa
2175+ YswOjYxaCkwQ/DsMOPD7S5OjwbLucs2YGjkBm7YF1nnXNzyt+BwieKQW/sQ2+ga1
2176+ mkgLW1BTQN3+JreBpeHy/yrRdK4dOZZUar4WPZitZzOW2eNpaaf6hKNA14LB/96a
2177+ tEguK8VazoqSQGvNV/R3PjIYmurVP3/Z9bEVgOKhMCflgwKCYgx+tBUypN3zFWv9
2178+ pgVq3iHx1MFCvoP9FsNB7I6jzOxlQP4z25BzR3ympx/QexkFw5CBFXhdrU+qNVBl
2179+ SSnz69aLEjCRXqBOnQEr0irs/e/35+yLJdEuw89vSwWwrzbV5r1Y7uxinEGWSydT
2180+ qddj97uKOWeMmnp20Be4+nhDDW/BMiTFI4Y3bYeDTrftbWMaSEmtSTw5HHxtAFtg
2181+ X9Hyx0Q3eN1w3gRZgIdm0xYTe7bNTofFRdfXzB/9wtNIcaW10+IlODShFHPCnh+d
2182+ i56a8LCdZcXiiLfCIhEcnqmM37BVvhjIQKSyOU1eMEgX148aVEz36OVuMwARAQAB
2183+ tCdDaHJpc3RpYW4gRWhyaGFyZHQgPGNwYWVsemVyQGdtYWlsLmNvbT6JAjgEEwEC
2184+ ACIFAlXJ3NcCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJELo+KTOCgLJC
2185+ BugP/ir0ES3wCzvHMnkz2UlXt9FR4KqY0L9uFmwu9VYpmfAploEVIOi2HcuxpcRp
2186+ hgoQlUtkz3lRhUeZzCxuB1ljM2JKTJiezP1tFTTGCbVYhPyA0LmUiHDWylG7FzPb
2187+ TX96HY/G0jf+m4CfR8q3HNHjeDi4VeA2ppBxdHcVE5I7HihDgRPJd+CvCa3nYdAb
2188+ nXDKlQZz5aZc7AgrRVamr4mshkzWuwNNCwOt3AIgHDkU/HzA5xlXfwHxOoP6scWH
2189+ /T7vFsd/vOikBphGseWPgKm6w1zyQ5Dk/wjRL8UeSJZW+Rh4PuBMbxg01lAZpPTq
2190+ tu/bePeNty3g5bhwO6oHMpWhprn3dO37R680qo6UnBPzICeuBUnSYgpPnsQC9maz
2191+ FEjiBtMsXSanU5vww7TpxY1JHjk5KFcmKx4sBeablznsm+GuVaDFN8R4eDjrM14r
2192+ SOzA9cV0bSQr4dMqA9fZFSx6qLTacIeMfptybW3zaDX/pJOeBBWRAtoAfZIFbBnu
2193+ /ZxDDgiQtZzpVK4UkYk5rjjtV/CPVXx64AnTHi35YfUn14KkE+k3odHdvPfBiv9+
2194+ NxfkTuV/koOgpD3+lTIYXyVHS9gwvhfRD/YfdrnVGl7bRZe68j7bfWDuQuSqIhSA
2195+ jpeJslJCawnqv6fVB6buj6jjcgHIxqCVn99chaPFSblEIPfXtDVDaHJpc3RpYW4g
2196+ RWhyaGFyZHQgPGNocmlzdGlhbi5laHJoYXJkdEBjYW5vbmljYWwuY29tPokCOAQT
2197+ AQIAIgUCVsbUOgIbAwYLCQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQuj4pM4KA
2198+ skJNPg/7BF/iXHHdSBigWKXCCvQz58uInoc/R4beIegxRCMq7wkYEey4B7Fd35zY
2199+ zv9CBOTV3hZePMCg9jxl4ki2kSsrZSCIEJw4L/aXDtJtx3HT18uTW0QKoU3nK/ro
2200+ OtthVqBqmiSEi40UUU+5MGrUjwLSm+PjaaSapjK/lddf0KbXBB78/BtR/XT0gxWM
2201+ +o68Oei9Nj1S3h6UndJwNAQ1xaDWmU2T7CRJet3F+cXZd3aDuS2axOTSTZbraSq7
2202+ zdl1xUiKtzXZIp8X1ewne+dzkewZuWj7DOwOBEFK26UhxCjKd5mUr7jpWQ4ampFX
2203+ 6xfd/MK8SJFY+iHOBKyzq9po40tE23dqWuaHB+T3MxOgQ9JHCo9x22XNvEuKZW/V
2204+ 4WaoGHVkR+jtWNC8Qv/xCMHL3CEvAklKJR68WDhozwUYTgNt5vCoJOviMlbhDSwf
2205+ 0zVXpQwMR//4c0QSA0+BPpIEPDnx5vTIHBVXHy4bBBHU2Vi87QIDS0AtiBpNcspN
2206+ 6AG0ktuldkE/pqfSTJ2A9HpHZyU+8boagRS5/z102Pjtmf/mzUkcHmfRb9o0DE15
2207+ X5fqpA3lYyx9eHIAgH4eaB1+G20Ez/EY5hr8IMS2nNBSem491UW6DXDYRu6eBLrR
2208+ sRmtrJ6DlTZFRFlqVZ47bce/SbeM/xljvRkBxWG6RtDRsTyNVI65Ag0EVcnc1wEQ
2209+ ANzk9W058tSHqf05UEtJGrN0K8DLriCvPd7QdFA8yVIZM3WD+m0AMBGXjd8BT5c2
2210+ lt0GmhB8klonHZvPiVLTRTLcSsc3NBopr1HL1bWsgOczwWiXSrc62oGAHUOQT/bv
2211+ vS6KIkZgez+qtCo/DCOGJrADaoJBiBCLSsZgowpzazZZDPUF7rAsfcryVCFvftK0
2212+ wAe1OdvUG77NHrMrE1oX3zh82hTqR5azBre6Y81lNwxxug/Xl/RHjNhEOYohcsLS
2213+ /xl0m2X831fHzcGGpoISRgrfel+M4RoC7KsLrwVhrF8koCD/ZQlevfLpuRl5LNpO
2214+ s1ZtEi8ZvLliih+H+BOlBD0zUc3zZrrks/NCpm1eZba0Z6L48r4TIHW08SGlHx7o
2215+ SrXgkq3mtoM8C4uDiLwjav5KxiF7n68s/9LF82aAr7YjNXd+xYZNjsmmFlYj9CGI
2216+ lL4jVt4v4EtTONa6pbtCNv5ezOLDZ6BBcQ36xdkrWzdpjQjL2mnh3sqIAGIPu7tH
2217+ N8euQ5L1zIvIjVqYlR1eJssp96QYPWYxF7TosfML4BUhCP631IWfuD9X/K2LzDmv
2218+ B2gVZo9fbhSC+P7GYVG+tV4VLAMbspAxRXXL69+j98aeV5g59f8OFQPbGpKE/SAY
2219+ eIXtq8DD+PYUXXq3VUI2brVLv42LBVdSJpKNKG3decIBABEBAAGJAh8EGAECAAkF
2220+ AlXJ3NcCGwwACgkQuj4pM4KAskKzeg/9FxXJLV3eWwY4nn2VhwYTHnHtSUpi8usk
2221+ RzIa3Mcj6OEVjU2LZaT3UQF8h6dLM9y+CemcwyjMqm1RQ5+ogfrItby1AaBXwCvm
2222+ XCUGw2zFOAnyzSHHoDFj27sllFxDmfSiBY5KP8M+/ywHKZDkRb6EjzMPx5oKFeGW
2223+ Hmqaj5FDmTeWChSIHd1ZxobashFauOZDbS/ijRRMsVGFulU2Nb/4QJK73g3orfhY
2224+ 5mq1TMkQ5Kcbqh4OmYYYayLtJQcpa6ZVopaRhAJFe30P83zW9pM5LQDpP9JIyY+S
2225+ DjasEY4ekYtw6oCKAjpqlwaaNDjl27OkJ7R7laFKy4grZ2TSB/2KTjn/Ea3CH/pA
2226+ SrpVis1LvC90XytbBnsEKYXU55H943wmBc6oj+itQhx4WyIiv+UgtHI/DbnYbUru
2227+ 71wpfapqGBXYfu/zAra8PITngOFuizeYu+idemu55ANO3keJPKr3ZBUSBBpNFauT
2228+ VUUCSnrLt+kpSLopYESiNdsPW/aQTFgFvA4BkBJTIMQsQZXicuXUePYlg5xFzXOv
2229+ XgiqkjRA9xBI5JAIUgLRk3ulVFt2bIsTG9XgtGyphEs86Q0MOIMo0WbZGtAYDrZO
2230+ DITbm2KzVLGVLn/ZJiW11RSHPNiwgg66/puKdFWrSogYYDJdDEUJtLIhypZ+ORxe
2231+ 7oh88hTkC1w=
2232+ =UNSw
2233+ -----END PGP PUBLIC KEY BLOCK-----
2234
2235=== added file 'examples/tests/apt_source_search.yaml'
2236--- examples/tests/apt_source_search.yaml 1970-01-01 00:00:00 +0000
2237+++ examples/tests/apt_source_search.yaml 2016-07-26 19:53:43 +0000
2238@@ -0,0 +1,97 @@
2239+showtrace: true
2240+apt:
2241+ preserve_sources_list: false
2242+ primary:
2243+ - arches: [default]
2244+ search:
2245+ - http://does.not.exist/ubuntu
2246+ - http://does.also.not.exist/ubuntu
2247+ - http://us.archive.ubuntu.com/ubuntu
2248+ security:
2249+ - arches: [default]
2250+ search:
2251+ - http://does.not.exist/ubuntu
2252+ - http://does.also.not.exist/ubuntu
2253+ - http://security.ubuntu.com/ubuntu
2254+ conf: | # APT config
2255+ ACQUIRE {
2256+ Retries "3";
2257+ };
2258+ sources:
2259+ curtin-dev-ppa.list:
2260+ source: "deb http://ppa.launchpad.net/curtin-dev/test-archive/ubuntu xenial main"
2261+ keyid: F430BBA5
2262+ ignored1:
2263+ source: "ppa:curtin-dev/test-archive"
2264+ my-repo2.list:
2265+ source: deb $MIRROR $RELEASE multiverse
2266+ ignored3:
2267+ keyid: 0E72 9061 0D2F 6DC4 D65E A921 9A31 4EC5 F470 A0AC
2268+ my-repo4.list:
2269+ source: deb http://ppa.launchpad.net/curtin-dev/test-archive/ubuntu xenial main
2270+ key: |
2271+ -----BEGIN PGP PUBLIC KEY BLOCK-----
2272+ Version: GnuPG v1
2273+
2274+ mQINBFXJ3NcBEAC85PMdaKdItkdjCT1vRJrdwNqj4lN5mu6z4dDVfeZlmozRDBGb
2275+ ENSOWCiYz3meANO7bKthQQCqAETSBV72rrDCqFZUpXeyG3zCN98Z/UdJ8zpQD9uw
2276+ mq2CaAqWMk6ty+PkHQ4gtIc390lGfRbHNoZ5HaWJNVOK7FCB2hBmnTZW7AViYiYa
2277+ YswOjYxaCkwQ/DsMOPD7S5OjwbLucs2YGjkBm7YF1nnXNzyt+BwieKQW/sQ2+ga1
2278+ mkgLW1BTQN3+JreBpeHy/yrRdK4dOZZUar4WPZitZzOW2eNpaaf6hKNA14LB/96a
2279+ tEguK8VazoqSQGvNV/R3PjIYmurVP3/Z9bEVgOKhMCflgwKCYgx+tBUypN3zFWv9
2280+ pgVq3iHx1MFCvoP9FsNB7I6jzOxlQP4z25BzR3ympx/QexkFw5CBFXhdrU+qNVBl
2281+ SSnz69aLEjCRXqBOnQEr0irs/e/35+yLJdEuw89vSwWwrzbV5r1Y7uxinEGWSydT
2282+ qddj97uKOWeMmnp20Be4+nhDDW/BMiTFI4Y3bYeDTrftbWMaSEmtSTw5HHxtAFtg
2283+ X9Hyx0Q3eN1w3gRZgIdm0xYTe7bNTofFRdfXzB/9wtNIcaW10+IlODShFHPCnh+d
2284+ i56a8LCdZcXiiLfCIhEcnqmM37BVvhjIQKSyOU1eMEgX148aVEz36OVuMwARAQAB
2285+ tCdDaHJpc3RpYW4gRWhyaGFyZHQgPGNwYWVsemVyQGdtYWlsLmNvbT6JAjgEEwEC
2286+ ACIFAlXJ3NcCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJELo+KTOCgLJC
2287+ BugP/ir0ES3wCzvHMnkz2UlXt9FR4KqY0L9uFmwu9VYpmfAploEVIOi2HcuxpcRp
2288+ hgoQlUtkz3lRhUeZzCxuB1ljM2JKTJiezP1tFTTGCbVYhPyA0LmUiHDWylG7FzPb
2289+ TX96HY/G0jf+m4CfR8q3HNHjeDi4VeA2ppBxdHcVE5I7HihDgRPJd+CvCa3nYdAb
2290+ nXDKlQZz5aZc7AgrRVamr4mshkzWuwNNCwOt3AIgHDkU/HzA5xlXfwHxOoP6scWH
2291+ /T7vFsd/vOikBphGseWPgKm6w1zyQ5Dk/wjRL8UeSJZW+Rh4PuBMbxg01lAZpPTq
2292+ tu/bePeNty3g5bhwO6oHMpWhprn3dO37R680qo6UnBPzICeuBUnSYgpPnsQC9maz
2293+ FEjiBtMsXSanU5vww7TpxY1JHjk5KFcmKx4sBeablznsm+GuVaDFN8R4eDjrM14r
2294+ SOzA9cV0bSQr4dMqA9fZFSx6qLTacIeMfptybW3zaDX/pJOeBBWRAtoAfZIFbBnu
2295+ /ZxDDgiQtZzpVK4UkYk5rjjtV/CPVXx64AnTHi35YfUn14KkE+k3odHdvPfBiv9+
2296+ NxfkTuV/koOgpD3+lTIYXyVHS9gwvhfRD/YfdrnVGl7bRZe68j7bfWDuQuSqIhSA
2297+ jpeJslJCawnqv6fVB6buj6jjcgHIxqCVn99chaPFSblEIPfXtDVDaHJpc3RpYW4g
2298+ RWhyaGFyZHQgPGNocmlzdGlhbi5laHJoYXJkdEBjYW5vbmljYWwuY29tPokCOAQT
2299+ AQIAIgUCVsbUOgIbAwYLCQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQuj4pM4KA
2300+ skJNPg/7BF/iXHHdSBigWKXCCvQz58uInoc/R4beIegxRCMq7wkYEey4B7Fd35zY
2301+ zv9CBOTV3hZePMCg9jxl4ki2kSsrZSCIEJw4L/aXDtJtx3HT18uTW0QKoU3nK/ro
2302+ OtthVqBqmiSEi40UUU+5MGrUjwLSm+PjaaSapjK/lddf0KbXBB78/BtR/XT0gxWM
2303+ +o68Oei9Nj1S3h6UndJwNAQ1xaDWmU2T7CRJet3F+cXZd3aDuS2axOTSTZbraSq7
2304+ zdl1xUiKtzXZIp8X1ewne+dzkewZuWj7DOwOBEFK26UhxCjKd5mUr7jpWQ4ampFX
2305+ 6xfd/MK8SJFY+iHOBKyzq9po40tE23dqWuaHB+T3MxOgQ9JHCo9x22XNvEuKZW/V
2306+ 4WaoGHVkR+jtWNC8Qv/xCMHL3CEvAklKJR68WDhozwUYTgNt5vCoJOviMlbhDSwf
2307+ 0zVXpQwMR//4c0QSA0+BPpIEPDnx5vTIHBVXHy4bBBHU2Vi87QIDS0AtiBpNcspN
2308+ 6AG0ktuldkE/pqfSTJ2A9HpHZyU+8boagRS5/z102Pjtmf/mzUkcHmfRb9o0DE15
2309+ X5fqpA3lYyx9eHIAgH4eaB1+G20Ez/EY5hr8IMS2nNBSem491UW6DXDYRu6eBLrR
2310+ sRmtrJ6DlTZFRFlqVZ47bce/SbeM/xljvRkBxWG6RtDRsTyNVI65Ag0EVcnc1wEQ
2311+ ANzk9W058tSHqf05UEtJGrN0K8DLriCvPd7QdFA8yVIZM3WD+m0AMBGXjd8BT5c2
2312+ lt0GmhB8klonHZvPiVLTRTLcSsc3NBopr1HL1bWsgOczwWiXSrc62oGAHUOQT/bv
2313+ vS6KIkZgez+qtCo/DCOGJrADaoJBiBCLSsZgowpzazZZDPUF7rAsfcryVCFvftK0
2314+ wAe1OdvUG77NHrMrE1oX3zh82hTqR5azBre6Y81lNwxxug/Xl/RHjNhEOYohcsLS
2315+ /xl0m2X831fHzcGGpoISRgrfel+M4RoC7KsLrwVhrF8koCD/ZQlevfLpuRl5LNpO
2316+ s1ZtEi8ZvLliih+H+BOlBD0zUc3zZrrks/NCpm1eZba0Z6L48r4TIHW08SGlHx7o
2317+ SrXgkq3mtoM8C4uDiLwjav5KxiF7n68s/9LF82aAr7YjNXd+xYZNjsmmFlYj9CGI
2318+ lL4jVt4v4EtTONa6pbtCNv5ezOLDZ6BBcQ36xdkrWzdpjQjL2mnh3sqIAGIPu7tH
2319+ N8euQ5L1zIvIjVqYlR1eJssp96QYPWYxF7TosfML4BUhCP631IWfuD9X/K2LzDmv
2320+ B2gVZo9fbhSC+P7GYVG+tV4VLAMbspAxRXXL69+j98aeV5g59f8OFQPbGpKE/SAY
2321+ eIXtq8DD+PYUXXq3VUI2brVLv42LBVdSJpKNKG3decIBABEBAAGJAh8EGAECAAkF
2322+ AlXJ3NcCGwwACgkQuj4pM4KAskKzeg/9FxXJLV3eWwY4nn2VhwYTHnHtSUpi8usk
2323+ RzIa3Mcj6OEVjU2LZaT3UQF8h6dLM9y+CemcwyjMqm1RQ5+ogfrItby1AaBXwCvm
2324+ XCUGw2zFOAnyzSHHoDFj27sllFxDmfSiBY5KP8M+/ywHKZDkRb6EjzMPx5oKFeGW
2325+ Hmqaj5FDmTeWChSIHd1ZxobashFauOZDbS/ijRRMsVGFulU2Nb/4QJK73g3orfhY
2326+ 5mq1TMkQ5Kcbqh4OmYYYayLtJQcpa6ZVopaRhAJFe30P83zW9pM5LQDpP9JIyY+S
2327+ DjasEY4ekYtw6oCKAjpqlwaaNDjl27OkJ7R7laFKy4grZ2TSB/2KTjn/Ea3CH/pA
2328+ SrpVis1LvC90XytbBnsEKYXU55H943wmBc6oj+itQhx4WyIiv+UgtHI/DbnYbUru
2329+ 71wpfapqGBXYfu/zAra8PITngOFuizeYu+idemu55ANO3keJPKr3ZBUSBBpNFauT
2330+ VUUCSnrLt+kpSLopYESiNdsPW/aQTFgFvA4BkBJTIMQsQZXicuXUePYlg5xFzXOv
2331+ XgiqkjRA9xBI5JAIUgLRk3ulVFt2bIsTG9XgtGyphEs86Q0MOIMo0WbZGtAYDrZO
2332+ DITbm2KzVLGVLn/ZJiW11RSHPNiwgg66/puKdFWrSogYYDJdDEUJtLIhypZ+ORxe
2333+ 7oh88hTkC1w=
2334+ =UNSw
2335+ -----END PGP PUBLIC KEY BLOCK-----
2336
2337=== added file 'examples/tests/apt_source_search_dns.yaml'
2338--- examples/tests/apt_source_search_dns.yaml 1970-01-01 00:00:00 +0000
2339+++ examples/tests/apt_source_search_dns.yaml 2016-07-26 19:53:43 +0000
2340@@ -0,0 +1,21 @@
2341+showtrace: true
2342+# fake up something that works in vmtets environment without dependency to external dns setup
2343+early_commands:
2344+ xx_prep_fake_dns_search0: 'echo foo.bar > /etc/hostname'
2345+ xx_prep_fake_dns_search1: 'echo "$(getent ahosts us.archive.ubuntu.com | cut --fields=1 --delimiter " " | grep -v : | head -n 1) ubuntu-mirror" >> /etc/hosts'
2346+ xx_prep_fake_dns_search2: 'echo "$(getent ahosts security.ubuntu.com | cut --fields=1 --delimiter " " | grep -v : | head -n 1) ubuntu-security-mirror" >> /etc/hosts'
2347+apt:
2348+ primary:
2349+ - arches: [default]
2350+ search_dns: True
2351+ security:
2352+ - arches: [default]
2353+ search_dns: True
2354+ # avoiding issues with these fake mirrors we only set them in a disabled list
2355+ # this is the default now preserve_sources_list: true
2356+ sources:
2357+ dnssearch.list.disabled:
2358+ source: |
2359+ deb $MIRROR $RELEASE multiverse
2360+ deb $PRIMARY $RELEASE universe
2361+ deb $SECURITY $RELEASE main
2362
2363=== added file 'examples/tests/test_old_apt_features.yaml'
2364--- examples/tests/test_old_apt_features.yaml 1970-01-01 00:00:00 +0000
2365+++ examples/tests/test_old_apt_features.yaml 2016-07-26 19:53:43 +0000
2366@@ -0,0 +1,10 @@
2367+showtrace: true
2368+# apt_proxy gets configured by tools/launch and tests/vmtests/__init__.py
2369+apt_mirrors:
2370+# we need a mirror that works (even in CI) but isn't the default
2371+ ubuntu_archive: http://us.archive.ubuntu.com/ubuntu
2372+ ubuntu_security: http://archive.ubuntu.com/ubuntu
2373+# set some key that surely is available to a non-default value
2374+debconf_selections:
2375+ set1: |
2376+ debconf debconf/priority select low
2377
2378=== added file 'examples/tests/test_old_apt_features_ports.yaml'
2379--- examples/tests/test_old_apt_features_ports.yaml 1970-01-01 00:00:00 +0000
2380+++ examples/tests/test_old_apt_features_ports.yaml 2016-07-26 19:53:43 +0000
2381@@ -0,0 +1,10 @@
2382+showtrace: true
2383+# apt_proxy gets configured by tools/launch and tests/vmtests/__init__.py
2384+apt_mirrors:
2385+# For ports there is no non-default alternative we could use
2386+ ubuntu_archive: http://ports.ubuntu.com/ubuntu-ports
2387+ ubuntu_security: http://ports.ubuntu.com/ubuntu-ports
2388+# set some key that surely is available to a non-default value
2389+debconf_selections:
2390+ set1: |
2391+ debconf debconf/priority select low
2392
2393=== added file 'tests/unittests/test_apt_custom_sources_list.py'
2394--- tests/unittests/test_apt_custom_sources_list.py 1970-01-01 00:00:00 +0000
2395+++ tests/unittests/test_apt_custom_sources_list.py 2016-07-26 19:53:43 +0000
2396@@ -0,0 +1,172 @@
2397+""" test_apt_custom_sources_list
2398+Test templating of custom sources list
2399+"""
2400+import logging
2401+import os
2402+import shutil
2403+import tempfile
2404+
2405+from unittest import TestCase
2406+
2407+import yaml
2408+import mock
2409+from mock import call
2410+
2411+from curtin import util
2412+from curtin.commands import apt_config
2413+
2414+LOG = logging.getLogger(__name__)
2415+
2416+TARGET = "/"
2417+
2418+# Input and expected output for the custom template
2419+YAML_TEXT_CUSTOM_SL = """
2420+preserve_sources_list: false
2421+primary:
2422+ - arches: [default]
2423+ uri: http://test.ubuntu.com/ubuntu/
2424+security:
2425+ - arches: [default]
2426+ uri: http://testsec.ubuntu.com/ubuntu/
2427+sources_list: |
2428+
2429+ ## Note, this file is written by curtin at install time. It should not end
2430+ ## up on the installed system itself.
2431+ #
2432+ # See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
2433+ # newer versions of the distribution.
2434+ deb $MIRROR $RELEASE main restricted
2435+ deb-src $MIRROR $RELEASE main restricted
2436+ deb $PRIMARY $RELEASE universe restricted
2437+ deb $SECURITY $RELEASE-security multiverse
2438+ # FIND_SOMETHING_SPECIAL
2439+"""
2440+
2441+EXPECTED_CONVERTED_CONTENT = """
2442+## Note, this file is written by curtin at install time. It should not end
2443+## up on the installed system itself.
2444+#
2445+# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
2446+# newer versions of the distribution.
2447+deb http://test.ubuntu.com/ubuntu/ fakerel main restricted
2448+deb-src http://test.ubuntu.com/ubuntu/ fakerel main restricted
2449+deb http://test.ubuntu.com/ubuntu/ fakerel universe restricted
2450+deb http://testsec.ubuntu.com/ubuntu/ fakerel-security multiverse
2451+# FIND_SOMETHING_SPECIAL
2452+"""
2453+
2454+# mocked to be independent to the unittest system
2455+MOCKED_APT_SRC_LIST = """
2456+deb http://test.ubuntu.com/ubuntu/ notouched main restricted
2457+deb-src http://test.ubuntu.com/ubuntu/ notouched main restricted
2458+deb http://test.ubuntu.com/ubuntu/ notouched-updates main restricted
2459+deb http://testsec.ubuntu.com/ubuntu/ notouched-security main restricted
2460+"""
2461+
2462+EXPECTED_BASE_CONTENT = ("""
2463+deb http://test.ubuntu.com/ubuntu/ notouched main restricted
2464+deb-src http://test.ubuntu.com/ubuntu/ notouched main restricted
2465+deb http://test.ubuntu.com/ubuntu/ notouched-updates main restricted
2466+deb http://testsec.ubuntu.com/ubuntu/ notouched-security main restricted
2467+""")
2468+
2469+EXPECTED_MIRROR_CONTENT = ("""
2470+deb http://test.ubuntu.com/ubuntu/ notouched main restricted
2471+deb-src http://test.ubuntu.com/ubuntu/ notouched main restricted
2472+deb http://test.ubuntu.com/ubuntu/ notouched-updates main restricted
2473+deb http://test.ubuntu.com/ubuntu/ notouched-security main restricted
2474+""")
2475+
2476+EXPECTED_PRIMSEC_CONTENT = ("""
2477+deb http://test.ubuntu.com/ubuntu/ notouched main restricted
2478+deb-src http://test.ubuntu.com/ubuntu/ notouched main restricted
2479+deb http://test.ubuntu.com/ubuntu/ notouched-updates main restricted
2480+deb http://testsec.ubuntu.com/ubuntu/ notouched-security main restricted
2481+""")
2482+
2483+
2484+class TestAptSourceConfigSourceList(TestCase):
2485+ """TestAptSourceConfigSourceList - Class to test sources list rendering"""
2486+ def setUp(self):
2487+ super(TestAptSourceConfigSourceList, self).setUp()
2488+ self.new_root = tempfile.mkdtemp()
2489+ self.addCleanup(shutil.rmtree, self.new_root)
2490+ # self.patchUtils(self.new_root)
2491+
2492+ @staticmethod
2493+ def _apt_source_list(cfg, expected):
2494+ "_apt_source_list - Test rendering from template (generic)"
2495+
2496+ arch = util.get_architecture()
2497+ # would fail inside the unittest context
2498+ with mock.patch.object(util, 'get_architecture',
2499+ return_value=arch) as mockga:
2500+ with mock.patch.object(util, 'write_file') as mockwrite:
2501+ # keep it side effect free and avoid permission errors
2502+ with mock.patch.object(os, 'rename'):
2503+ # make test independent to executing system
2504+ with mock.patch.object(util, 'load_file',
2505+ return_value=MOCKED_APT_SRC_LIST):
2506+ with mock.patch.object(util, 'lsb_release',
2507+ return_value={'codename':
2508+ 'fakerel'}):
2509+ apt_config.handle_apt(cfg, TARGET)
2510+
2511+ mockga.assert_called_with("/")
2512+
2513+ cloudfile = '/etc/cloud/cloud.cfg.d/curtin-preserve-sources.cfg'
2514+ cloudconf = yaml.dump({'apt_preserve_sources_list': True}, indent=1)
2515+ calls = [call(TARGET + '/etc/apt/sources.list',
2516+ expected,
2517+ mode=0o644),
2518+ call(TARGET + cloudfile,
2519+ cloudconf,
2520+ mode=0o644)]
2521+ mockwrite.assert_has_calls(calls)
2522+
2523+ def test_apt_source_list(self):
2524+ """test_apt_source_list - Test with neither custom sources nor parms"""
2525+ cfg = {'preserve_sources_list': False}
2526+
2527+ self._apt_source_list(cfg, EXPECTED_BASE_CONTENT)
2528+
2529+ def test_apt_source_list_psm(self):
2530+ """test_apt_source_list_psm - Test specifying prim+sec mirrors"""
2531+ cfg = {'preserve_sources_list': False,
2532+ 'primary': [{'arches': ["default"],
2533+ 'uri': 'http://test.ubuntu.com/ubuntu/'}],
2534+ 'security': [{'arches': ["default"],
2535+ 'uri': 'http://testsec.ubuntu.com/ubuntu/'}]}
2536+
2537+ self._apt_source_list(cfg, EXPECTED_PRIMSEC_CONTENT)
2538+
2539+ @staticmethod
2540+ def test_apt_srcl_custom():
2541+ """test_apt_srcl_custom - Test rendering a custom source template"""
2542+ cfg = yaml.safe_load(YAML_TEXT_CUSTOM_SL)
2543+
2544+ arch = util.get_architecture()
2545+ # would fail inside the unittest context
2546+ with mock.patch.object(util, 'get_architecture',
2547+ return_value=arch) as mockga:
2548+ with mock.patch.object(util, 'write_file') as mockwrite:
2549+ # keep it side effect free and avoid permission errors
2550+ with mock.patch.object(os, 'rename'):
2551+ with mock.patch.object(util, 'lsb_release',
2552+ return_value={'codename':
2553+ 'fakerel'}):
2554+ apt_config.handle_apt(cfg, TARGET)
2555+
2556+ mockga.assert_called_with("/")
2557+ cloudfile = '/etc/cloud/cloud.cfg.d/curtin-preserve-sources.cfg'
2558+ cloudconf = yaml.dump({'apt_preserve_sources_list': True}, indent=1)
2559+ calls = [call(TARGET + '/etc/apt/sources.list',
2560+ EXPECTED_CONVERTED_CONTENT,
2561+ mode=0o644),
2562+ call(TARGET + cloudfile,
2563+ cloudconf,
2564+ mode=0o644)]
2565+ mockwrite.assert_has_calls(calls)
2566+
2567+
2568+# vi: ts=4 expandtab
2569
2570=== added file 'tests/unittests/test_apt_source.py'
2571--- tests/unittests/test_apt_source.py 1970-01-01 00:00:00 +0000
2572+++ tests/unittests/test_apt_source.py 2016-07-26 19:53:43 +0000
2573@@ -0,0 +1,927 @@
2574+""" test_apt_source
2575+Testing various config variations of the apt_source custom config
2576+"""
2577+import glob
2578+import os
2579+import re
2580+import shutil
2581+import socket
2582+import tempfile
2583+
2584+from unittest import TestCase
2585+
2586+import mock
2587+from mock import call
2588+
2589+from curtin import util
2590+from curtin import gpg
2591+from curtin.commands import apt_config
2592+
2593+
2594+EXPECTEDKEY = u"""-----BEGIN PGP PUBLIC KEY BLOCK-----
2595+Version: GnuPG v1
2596+
2597+mI0ESuZLUgEEAKkqq3idtFP7g9hzOu1a8+v8ImawQN4TrvlygfScMU1TIS1eC7UQ
2598+NUA8Qqgr9iUaGnejb0VciqftLrU9D6WYHSKz+EITefgdyJ6SoQxjoJdsCpJ7o9Jy
2599+8PQnpRttiFm4qHu6BVnKnBNxw/z3ST9YMqW5kbMQpfxbGe+obRox59NpABEBAAG0
2600+HUxhdW5jaHBhZCBQUEEgZm9yIFNjb3R0IE1vc2VyiLYEEwECACAFAkrmS1ICGwMG
2601+CwkIBwMCBBUCCAMEFgIDAQIeAQIXgAAKCRAGILvPA2g/d3aEA/9tVjc10HOZwV29
2602+OatVuTeERjjrIbxflO586GLA8cp0C9RQCwgod/R+cKYdQcHjbqVcP0HqxveLg0RZ
2603+FJpWLmWKamwkABErwQLGlM/Hwhjfade8VvEQutH5/0JgKHmzRsoqfR+LMO6OS+Sm
2604+S0ORP6HXET3+jC8BMG4tBWCTK/XEZw==
2605+=ACB2
2606+-----END PGP PUBLIC KEY BLOCK-----"""
2607+
2608+ADD_APT_REPO_MATCH = r"^[\w-]+:\w"
2609+
2610+TARGET = "/"
2611+
2612+
2613+def load_tfile(filename):
2614+ """ load_tfile
2615+ load file and return content after decoding
2616+ """
2617+ try:
2618+ content = util.load_file(filename, mode="r")
2619+ except Exception as error:
2620+ print('failed to load file content for test: %s' % error)
2621+ raise
2622+
2623+ return content
2624+
2625+
2626+class TestAptSourceConfig(TestCase):
2627+ """ TestAptSourceConfig
2628+ Main Class to test apt configs
2629+ """
2630+ def setUp(self):
2631+ super(TestAptSourceConfig, self).setUp()
2632+ self.tmp = tempfile.mkdtemp()
2633+ self.addCleanup(shutil.rmtree, self.tmp)
2634+ self.aptlistfile = os.path.join(self.tmp, "single-deb.list")
2635+ self.aptlistfile2 = os.path.join(self.tmp, "single-deb2.list")
2636+ self.aptlistfile3 = os.path.join(self.tmp, "single-deb3.list")
2637+ self.join = os.path.join
2638+ self.matcher = re.compile(ADD_APT_REPO_MATCH).search
2639+
2640+ @staticmethod
2641+ def _add_apt_sources(*args, **kwargs):
2642+ with mock.patch.object(util, 'apt_update'):
2643+ apt_config.add_apt_sources(*args, **kwargs)
2644+
2645+ @staticmethod
2646+ def _get_default_params():
2647+ """ get_default_params
2648+ Get the most basic default mrror and release info to be used in tests
2649+ """
2650+ params = {}
2651+ params['RELEASE'] = util.lsb_release()['codename']
2652+ params['MIRROR'] = apt_config.get_default_mirrors()["PRIMARY"]
2653+ return params
2654+
2655+ def _myjoin(self, *args, **kwargs):
2656+ """ _myjoin - redir into writable tmpdir"""
2657+ if (args[0] == "/etc/apt/sources.list.d/" and
2658+ args[1] == "cloud_config_sources.list" and
2659+ len(args) == 2):
2660+ return self.join(self.tmp, args[0].lstrip("/"), args[1])
2661+ else:
2662+ return self.join(*args, **kwargs)
2663+
2664+ def _apt_src_basic(self, filename, cfg):
2665+ """ _apt_src_basic
2666+ Test Fix deb source string, has to overwrite mirror conf in params
2667+ """
2668+ params = self._get_default_params()
2669+
2670+ self._add_apt_sources(cfg, TARGET, template_params=params,
2671+ aa_repo_match=self.matcher)
2672+
2673+ self.assertTrue(os.path.isfile(filename))
2674+
2675+ contents = load_tfile(filename)
2676+ self.assertTrue(re.search(r"%s %s %s %s\n" %
2677+ ("deb", "http://test.ubuntu.com/ubuntu",
2678+ "karmic-backports",
2679+ "main universe multiverse restricted"),
2680+ contents, flags=re.IGNORECASE))
2681+
2682+ def test_apt_src_basic(self):
2683+ """test_apt_src_basic - Test fix deb source string"""
2684+ cfg = {self.aptlistfile: {'source':
2685+ ('deb http://test.ubuntu.com/ubuntu'
2686+ ' karmic-backports'
2687+ ' main universe multiverse restricted')}}
2688+ self._apt_src_basic(self.aptlistfile, cfg)
2689+
2690+ def test_apt_src_basic_tri(self):
2691+ """test_apt_src_basic_tri - Test multiple fix deb source strings"""
2692+ cfg = {self.aptlistfile: {'source':
2693+ ('deb http://test.ubuntu.com/ubuntu'
2694+ ' karmic-backports'
2695+ ' main universe multiverse restricted')},
2696+ self.aptlistfile2: {'source':
2697+ ('deb http://test.ubuntu.com/ubuntu'
2698+ ' precise-backports'
2699+ ' main universe multiverse restricted')},
2700+ self.aptlistfile3: {'source':
2701+ ('deb http://test.ubuntu.com/ubuntu'
2702+ ' lucid-backports'
2703+ ' main universe multiverse restricted')}}
2704+ self._apt_src_basic(self.aptlistfile, cfg)
2705+
2706+ # extra verify on two extra files of this test
2707+ contents = load_tfile(self.aptlistfile2)
2708+ self.assertTrue(re.search(r"%s %s %s %s\n" %
2709+ ("deb", "http://test.ubuntu.com/ubuntu",
2710+ "precise-backports",
2711+ "main universe multiverse restricted"),
2712+ contents, flags=re.IGNORECASE))
2713+ contents = load_tfile(self.aptlistfile3)
2714+ self.assertTrue(re.search(r"%s %s %s %s\n" %
2715+ ("deb", "http://test.ubuntu.com/ubuntu",
2716+ "lucid-backports",
2717+ "main universe multiverse restricted"),
2718+ contents, flags=re.IGNORECASE))
2719+
2720+ def _apt_src_replacement(self, filename, cfg):
2721+ """ apt_src_replace
2722+ Test Autoreplacement of MIRROR and RELEASE in source specs
2723+ """
2724+ params = self._get_default_params()
2725+ self._add_apt_sources(cfg, TARGET, template_params=params,
2726+ aa_repo_match=self.matcher)
2727+
2728+ self.assertTrue(os.path.isfile(filename))
2729+
2730+ contents = load_tfile(filename)
2731+ self.assertTrue(re.search(r"%s %s %s %s\n" %
2732+ ("deb", params['MIRROR'], params['RELEASE'],
2733+ "multiverse"),
2734+ contents, flags=re.IGNORECASE))
2735+
2736+ def test_apt_src_replace(self):
2737+ """test_apt_src_replace - Test Autoreplacement of MIRROR and RELEASE"""
2738+ cfg = {self.aptlistfile: {'source': 'deb $MIRROR $RELEASE multiverse'}}
2739+ self._apt_src_replacement(self.aptlistfile, cfg)
2740+
2741+ def test_apt_src_replace_fn(self):
2742+ """test_apt_src_replace_fn - Test filename being overwritten in dict"""
2743+ cfg = {'ignored': {'source': 'deb $MIRROR $RELEASE multiverse',
2744+ 'filename': self.aptlistfile}}
2745+ # second file should overwrite the dict key
2746+ self._apt_src_replacement(self.aptlistfile, cfg)
2747+
2748+ def _apt_src_replace_tri(self, cfg):
2749+ """ _apt_src_replace_tri
2750+ Test three autoreplacements of MIRROR and RELEASE in source specs with
2751+ generic part
2752+ """
2753+ self._apt_src_replacement(self.aptlistfile, cfg)
2754+
2755+ # extra verify on two extra files of this test
2756+ params = self._get_default_params()
2757+ contents = load_tfile(self.aptlistfile2)
2758+ self.assertTrue(re.search(r"%s %s %s %s\n" %
2759+ ("deb", params['MIRROR'], params['RELEASE'],
2760+ "main"),
2761+ contents, flags=re.IGNORECASE))
2762+ contents = load_tfile(self.aptlistfile3)
2763+ self.assertTrue(re.search(r"%s %s %s %s\n" %
2764+ ("deb", params['MIRROR'], params['RELEASE'],
2765+ "universe"),
2766+ contents, flags=re.IGNORECASE))
2767+
2768+ def test_apt_src_replace_tri(self):
2769+ """test_apt_src_replace_tri - Test multiple replacements/overwrites"""
2770+ cfg = {self.aptlistfile: {'source': 'deb $MIRROR $RELEASE multiverse'},
2771+ 'notused': {'source': 'deb $MIRROR $RELEASE main',
2772+ 'filename': self.aptlistfile2},
2773+ self.aptlistfile3: {'source': 'deb $MIRROR $RELEASE universe'}}
2774+ self._apt_src_replace_tri(cfg)
2775+
2776+ def _apt_src_keyid(self, filename, cfg, keynum):
2777+ """ _apt_src_keyid
2778+ Test specification of a source + keyid
2779+ """
2780+ params = self._get_default_params()
2781+
2782+ with mock.patch.object(util, 'subp',
2783+ return_value=('fakekey 1234', '')) as mockobj:
2784+ self._add_apt_sources(cfg, TARGET, template_params=params,
2785+ aa_repo_match=self.matcher)
2786+
2787+ # check if it added the right ammount of keys
2788+ calls = []
2789+ for _ in range(keynum):
2790+ calls.append(call(['apt-key', 'add', '-'], data=b'fakekey 1234'))
2791+ mockobj.assert_has_calls(calls, any_order=True)
2792+
2793+ self.assertTrue(os.path.isfile(filename))
2794+
2795+ contents = load_tfile(filename)
2796+ self.assertTrue(re.search(r"%s %s %s %s\n" %
2797+ ("deb",
2798+ ('http://ppa.launchpad.net/smoser/'
2799+ 'cloud-init-test/ubuntu'),
2800+ "xenial", "main"),
2801+ contents, flags=re.IGNORECASE))
2802+
2803+ def test_apt_src_keyid(self):
2804+ """test_apt_src_keyid - Test source + keyid with filename being set"""
2805+ cfg = {self.aptlistfile: {'source': ('deb '
2806+ 'http://ppa.launchpad.net/'
2807+ 'smoser/cloud-init-test/ubuntu'
2808+ ' xenial main'),
2809+ 'keyid': "03683F77"}}
2810+ self._apt_src_keyid(self.aptlistfile, cfg, 1)
2811+
2812+ def test_apt_src_keyid_tri(self):
2813+ """test_apt_src_keyid_tri - Test multiple src+keyid+filen overwrites"""
2814+ cfg = {self.aptlistfile: {'source': ('deb '
2815+ 'http://ppa.launchpad.net/'
2816+ 'smoser/cloud-init-test/ubuntu'
2817+ ' xenial main'),
2818+ 'keyid': "03683F77"},
2819+ 'ignored': {'source': ('deb '
2820+ 'http://ppa.launchpad.net/'
2821+ 'smoser/cloud-init-test/ubuntu'
2822+ ' xenial universe'),
2823+ 'keyid': "03683F77",
2824+ 'filename': self.aptlistfile2},
2825+ self.aptlistfile3: {'source': ('deb '
2826+ 'http://ppa.launchpad.net/'
2827+ 'smoser/cloud-init-test/ubuntu'
2828+ ' xenial multiverse'),
2829+ 'keyid': "03683F77"}}
2830+
2831+ self._apt_src_keyid(self.aptlistfile, cfg, 3)
2832+ contents = load_tfile(self.aptlistfile2)
2833+ self.assertTrue(re.search(r"%s %s %s %s\n" %
2834+ ("deb",
2835+ ('http://ppa.launchpad.net/smoser/'
2836+ 'cloud-init-test/ubuntu'),
2837+ "xenial", "universe"),
2838+ contents, flags=re.IGNORECASE))
2839+ contents = load_tfile(self.aptlistfile3)
2840+ self.assertTrue(re.search(r"%s %s %s %s\n" %
2841+ ("deb",
2842+ ('http://ppa.launchpad.net/smoser/'
2843+ 'cloud-init-test/ubuntu'),
2844+ "xenial", "multiverse"),
2845+ contents, flags=re.IGNORECASE))
2846+
2847+ def test_apt_src_key(self):
2848+ """test_apt_src_key - Test source + key"""
2849+ params = self._get_default_params()
2850+ cfg = {self.aptlistfile: {'source': ('deb '
2851+ 'http://ppa.launchpad.net/'
2852+ 'smoser/cloud-init-test/ubuntu'
2853+ ' xenial main'),
2854+ 'key': "fakekey 4321"}}
2855+
2856+ with mock.patch.object(util, 'subp') as mockobj:
2857+ self._add_apt_sources(cfg, TARGET, template_params=params,
2858+ aa_repo_match=self.matcher)
2859+
2860+ mockobj.assert_any_call(['apt-key', 'add', '-'], data=b'fakekey 4321')
2861+
2862+ self.assertTrue(os.path.isfile(self.aptlistfile))
2863+
2864+ contents = load_tfile(self.aptlistfile)
2865+ self.assertTrue(re.search(r"%s %s %s %s\n" %
2866+ ("deb",
2867+ ('http://ppa.launchpad.net/smoser/'
2868+ 'cloud-init-test/ubuntu'),
2869+ "xenial", "main"),
2870+ contents, flags=re.IGNORECASE))
2871+
2872+ def test_apt_src_keyonly(self):
2873+ """test_apt_src_keyonly - Test key without source"""
2874+ params = self._get_default_params()
2875+ cfg = {self.aptlistfile: {'key': "fakekey 4242"}}
2876+
2877+ with mock.patch.object(util, 'subp') as mockobj:
2878+ self._add_apt_sources(cfg, TARGET, template_params=params,
2879+ aa_repo_match=self.matcher)
2880+
2881+ mockobj.assert_any_call(['apt-key', 'add', '-'], data=b'fakekey 4242')
2882+
2883+ # filename should be ignored on key only
2884+ self.assertFalse(os.path.isfile(self.aptlistfile))
2885+
2886+ def test_apt_src_keyidonly(self):
2887+ """test_apt_src_keyidonly - Test keyid without source"""
2888+ params = self._get_default_params()
2889+ cfg = {self.aptlistfile: {'keyid': "03683F77"}}
2890+
2891+ with mock.patch.object(util, 'subp',
2892+ return_value=('fakekey 1212', '')) as mockobj:
2893+ self._add_apt_sources(cfg, TARGET, template_params=params,
2894+ aa_repo_match=self.matcher)
2895+
2896+ mockobj.assert_any_call(['apt-key', 'add', '-'], data=b'fakekey 1212')
2897+
2898+ # filename should be ignored on key only
2899+ self.assertFalse(os.path.isfile(self.aptlistfile))
2900+
2901+ def apt_src_keyid_real(self, cfg, expectedkey):
2902+ """apt_src_keyid_real
2903+ Test specification of a keyid without source including
2904+ up to addition of the key (add_apt_key_raw mocked to keep the
2905+ environment as is)
2906+ """
2907+ params = self._get_default_params()
2908+
2909+ with mock.patch.object(apt_config, 'add_apt_key_raw') as mockkey:
2910+ with mock.patch.object(gpg, 'getkeybyid',
2911+ return_value=expectedkey) as mockgetkey:
2912+ self._add_apt_sources(cfg, TARGET, template_params=params,
2913+ aa_repo_match=self.matcher)
2914+
2915+ keycfg = cfg[self.aptlistfile]
2916+ mockgetkey.assert_called_with(keycfg['keyid'],
2917+ keycfg.get('keyserver',
2918+ 'keyserver.ubuntu.com'))
2919+ mockkey.assert_called_with(expectedkey, TARGET)
2920+
2921+ # filename should be ignored on key only
2922+ self.assertFalse(os.path.isfile(self.aptlistfile))
2923+
2924+ def test_apt_src_keyid_real(self):
2925+ """test_apt_src_keyid_real - Test keyid including key add"""
2926+ keyid = "03683F77"
2927+ cfg = {self.aptlistfile: {'keyid': keyid}}
2928+
2929+ self.apt_src_keyid_real(cfg, EXPECTEDKEY)
2930+
2931+ def test_apt_src_longkeyid_real(self):
2932+ """test_apt_src_longkeyid_real Test long keyid including key add"""
2933+ keyid = "B59D 5F15 97A5 04B7 E230 6DCA 0620 BBCF 0368 3F77"
2934+ cfg = {self.aptlistfile: {'keyid': keyid}}
2935+
2936+ self.apt_src_keyid_real(cfg, EXPECTEDKEY)
2937+
2938+ def test_apt_src_longkeyid_ks_real(self):
2939+ """test_apt_src_longkeyid_ks_real Test long keyid from other ks"""
2940+ keyid = "B59D 5F15 97A5 04B7 E230 6DCA 0620 BBCF 0368 3F77"
2941+ cfg = {self.aptlistfile: {'keyid': keyid,
2942+ 'keyserver': 'keys.gnupg.net'}}
2943+
2944+ self.apt_src_keyid_real(cfg, EXPECTEDKEY)
2945+
2946+ def test_apt_src_keyid_keyserver(self):
2947+ """test_apt_src_keyid_keyserver - Test custom keyserver"""
2948+ keyid = "03683F77"
2949+ params = self._get_default_params()
2950+ cfg = {self.aptlistfile: {'keyid': keyid,
2951+ 'keyserver': 'test.random.com'}}
2952+
2953+ # in some test environments only *.ubuntu.com is reachable
2954+ # so mock the call and check if the config got there
2955+ with mock.patch.object(gpg, 'getkeybyid',
2956+ return_value="fakekey") as mockgetkey:
2957+ with mock.patch.object(apt_config, 'add_apt_key_raw') as mockadd:
2958+ self._add_apt_sources(cfg, TARGET, template_params=params,
2959+ aa_repo_match=self.matcher)
2960+
2961+ mockgetkey.assert_called_with('03683F77', 'test.random.com')
2962+ mockadd.assert_called_with('fakekey', TARGET)
2963+
2964+ # filename should be ignored on key only
2965+ self.assertFalse(os.path.isfile(self.aptlistfile))
2966+
2967+ def test_apt_src_ppa(self):
2968+ """test_apt_src_ppa - Test specification of a ppa"""
2969+ params = self._get_default_params()
2970+ cfg = {self.aptlistfile: {'source': 'ppa:smoser/cloud-init-test'}}
2971+
2972+ with mock.patch.object(util, 'subp') as mockobj:
2973+ self._add_apt_sources(cfg, TARGET, template_params=params,
2974+ aa_repo_match=self.matcher)
2975+ mockobj.assert_any_call(['add-apt-repository',
2976+ 'ppa:smoser/cloud-init-test'])
2977+
2978+ # adding ppa should ignore filename (uses add-apt-repository)
2979+ self.assertFalse(os.path.isfile(self.aptlistfile))
2980+
2981+ def test_apt_src_ppa_tri(self):
2982+ """test_apt_src_ppa_tri - Test specification of multiple ppa's"""
2983+ params = self._get_default_params()
2984+ cfg = {self.aptlistfile: {'source': 'ppa:smoser/cloud-init-test'},
2985+ self.aptlistfile2: {'source': 'ppa:smoser/cloud-init-test2'},
2986+ self.aptlistfile3: {'source': 'ppa:smoser/cloud-init-test3'}}
2987+
2988+ with mock.patch.object(util, 'subp') as mockobj:
2989+ self._add_apt_sources(cfg, TARGET, template_params=params,
2990+ aa_repo_match=self.matcher)
2991+ calls = [call(['add-apt-repository', 'ppa:smoser/cloud-init-test']),
2992+ call(['add-apt-repository', 'ppa:smoser/cloud-init-test2']),
2993+ call(['add-apt-repository', 'ppa:smoser/cloud-init-test3'])]
2994+ mockobj.assert_has_calls(calls, any_order=True)
2995+
2996+ # adding ppa should ignore all filenames (uses add-apt-repository)
2997+ self.assertFalse(os.path.isfile(self.aptlistfile))
2998+ self.assertFalse(os.path.isfile(self.aptlistfile2))
2999+ self.assertFalse(os.path.isfile(self.aptlistfile3))
3000+
3001+ def test_mir_apt_list_rename(self):
3002+ """test_mir_apt_list_rename - Test find mirror and apt list renaming"""
3003+ pre = "/var/lib/apt/lists"
3004+ # filenames are archive dependent
3005+ arch = util.get_architecture()
3006+ if arch in apt_config.PRIMARY_ARCHES:
3007+ component = "ubuntu"
3008+ archive = "archive.ubuntu.com"
3009+ else:
3010+ component = "ubuntu-ports"
3011+ archive = "ports.ubuntu.com"
3012+
3013+ cfg = {'primary': [{'arches': ["default"],
3014+ 'uri':
3015+ 'http://test.ubuntu.com/%s/' % component}],
3016+ 'security': [{'arches': ["default"],
3017+ 'uri':
3018+ 'http://testsec.ubuntu.com/%s/' % component}]}
3019+ post = ("%s_dists_%s-updates_InRelease" %
3020+ (component, util.lsb_release()['codename']))
3021+ fromfn = ("%s/%s_%s" % (pre, archive, post))
3022+ tofn = ("%s/test.ubuntu.com_%s" % (pre, post))
3023+
3024+ mirrors = apt_config.find_apt_mirror_info(cfg)
3025+
3026+ self.assertEqual(mirrors['MIRROR'],
3027+ "http://test.ubuntu.com/%s/" % component)
3028+ self.assertEqual(mirrors['PRIMARY'],
3029+ "http://test.ubuntu.com/%s/" % component)
3030+ self.assertEqual(mirrors['SECURITY'],
3031+ "http://testsec.ubuntu.com/%s/" % component)
3032+
3033+ # get_architecture would fail inside the unittest context
3034+ with mock.patch.object(util, 'get_architecture', return_value=arch):
3035+ with mock.patch.object(os, 'rename') as mockren:
3036+ with mock.patch.object(glob, 'glob',
3037+ return_value=[fromfn]):
3038+ apt_config.rename_apt_lists(mirrors, TARGET)
3039+
3040+ mockren.assert_any_call(fromfn, tofn)
3041+
3042+ @staticmethod
3043+ def test_apt_proxy():
3044+ """test_mir_apt_list_rename - Test apt_*proxy configuration"""
3045+ cfg = {"proxy": "foobar1",
3046+ "http_proxy": "foobar2",
3047+ "ftp_proxy": "foobar3",
3048+ "https_proxy": "foobar4"}
3049+
3050+ with mock.patch.object(util, 'write_file') as mockobj:
3051+ apt_config.apply_apt_proxy_config(cfg, "proxyfn", "notused")
3052+
3053+ mockobj.assert_called_with('proxyfn',
3054+ ('Acquire::http::Proxy "foobar1";\n'
3055+ 'Acquire::http::Proxy "foobar2";\n'
3056+ 'Acquire::ftp::Proxy "foobar3";\n'
3057+ 'Acquire::https::Proxy "foobar4";\n'))
3058+
3059+ def test_mirror(self):
3060+ """test_mirror - Test defining a mirror"""
3061+ pmir = "http://us.archive.ubuntu.com/ubuntu/"
3062+ smir = "http://security.ubuntu.com/ubuntu/"
3063+ cfg = {"primary": [{'arches': ["default"],
3064+ "uri": pmir}],
3065+ "security": [{'arches': ["default"],
3066+ "uri": smir}]}
3067+
3068+ mirrors = apt_config.find_apt_mirror_info(cfg)
3069+
3070+ self.assertEqual(mirrors['MIRROR'],
3071+ pmir)
3072+ self.assertEqual(mirrors['PRIMARY'],
3073+ pmir)
3074+ self.assertEqual(mirrors['SECURITY'],
3075+ smir)
3076+
3077+ def test_mirror_default(self):
3078+ """test_mirror_default - Test without defining a mirror"""
3079+ default_mirrors = apt_config.get_default_mirrors()
3080+ pmir = default_mirrors["PRIMARY"]
3081+ smir = default_mirrors["SECURITY"]
3082+ mirrors = apt_config.find_apt_mirror_info({})
3083+
3084+ self.assertEqual(mirrors['MIRROR'],
3085+ pmir)
3086+ self.assertEqual(mirrors['PRIMARY'],
3087+ pmir)
3088+ self.assertEqual(mirrors['SECURITY'],
3089+ smir)
3090+
3091+ def test_mirror_arches(self):
3092+ """test_mirror_arches - Test arches selection of mirror"""
3093+ pmir = "http://us.archive.ubuntu.com/ubuntu/"
3094+ smir = "http://security.ubuntu.com/ubuntu/"
3095+ cfg = {"primary": [{'arches': ["default"],
3096+ "uri": "notthis"},
3097+ {'arches': [util.get_architecture()],
3098+ "uri": pmir}],
3099+ "security": [{'arches': [util.get_architecture()],
3100+ "uri": smir},
3101+ {'arches': ["default"],
3102+ "uri": "nothat"}]}
3103+
3104+ mirrors = apt_config.find_apt_mirror_info(cfg)
3105+
3106+ self.assertEqual(mirrors['MIRROR'],
3107+ pmir)
3108+ self.assertEqual(mirrors['PRIMARY'],
3109+ pmir)
3110+ self.assertEqual(mirrors['SECURITY'],
3111+ smir)
3112+
3113+ def test_mirror_arches_default(self):
3114+ """test_mirror_arches - Test falling back to default arch"""
3115+ pmir = "http://us.archive.ubuntu.com/ubuntu/"
3116+ smir = "http://security.ubuntu.com/ubuntu/"
3117+ cfg = {"primary": [{'arches': ["default"],
3118+ "uri": pmir},
3119+ {'arches': ["thisarchdoesntexist"],
3120+ "uri": "notthis"}],
3121+ "security": [{'arches': ["thisarchdoesntexist"],
3122+ "uri": "nothat"},
3123+ {'arches': ["default"],
3124+ "uri": smir}]}
3125+
3126+ mirrors = apt_config.find_apt_mirror_info(cfg)
3127+
3128+ self.assertEqual(mirrors['MIRROR'],
3129+ pmir)
3130+ self.assertEqual(mirrors['PRIMARY'],
3131+ pmir)
3132+ self.assertEqual(mirrors['SECURITY'],
3133+ smir)
3134+
3135+ def test_mirror_arches_sysdefault(self):
3136+ """test_mirror_arches - Test arches falling back to sys default"""
3137+ default_mirrors = apt_config.get_default_mirrors()
3138+ pmir = default_mirrors["PRIMARY"]
3139+ smir = default_mirrors["SECURITY"]
3140+ cfg = {"primary": [{'arches': ["thisarchdoesntexist_64"],
3141+ "uri": "notthis"},
3142+ {'arches': ["thisarchdoesntexist"],
3143+ "uri": "notthiseither"}],
3144+ "security": [{'arches': ["thisarchdoesntexist"],
3145+ "uri": "nothat"},
3146+ {'arches': ["thisarchdoesntexist_64"],
3147+ "uri": "nothateither"}]}
3148+
3149+ mirrors = apt_config.find_apt_mirror_info(cfg)
3150+
3151+ self.assertEqual(mirrors['MIRROR'],
3152+ pmir)
3153+ self.assertEqual(mirrors['PRIMARY'],
3154+ pmir)
3155+ self.assertEqual(mirrors['SECURITY'],
3156+ smir)
3157+
3158+ def test_mirror_search(self):
3159+ """test_mirror_search - Test searching mirrors in a list
3160+ mock checks to avoid relying on network connectivity"""
3161+ pmir = "http://us.archive.ubuntu.com/ubuntu/"
3162+ smir = "http://security.ubuntu.com/ubuntu/"
3163+ cfg = {"primary": [{'arches': ["default"],
3164+ "search": ["pfailme", pmir]}],
3165+ "security": [{'arches': ["default"],
3166+ "search": ["sfailme", smir]}]}
3167+
3168+ with mock.patch.object(apt_config, 'search_for_mirror',
3169+ side_effect=[pmir, smir]) as mocksearch:
3170+ mirrors = apt_config.find_apt_mirror_info(cfg)
3171+
3172+ calls = [call(["pfailme", pmir]),
3173+ call(["sfailme", smir])]
3174+ mocksearch.assert_has_calls(calls)
3175+
3176+ self.assertEqual(mirrors['MIRROR'],
3177+ pmir)
3178+ self.assertEqual(mirrors['PRIMARY'],
3179+ pmir)
3180+ self.assertEqual(mirrors['SECURITY'],
3181+ smir)
3182+
3183+ def test_mirror_search_dns(self):
3184+ """test_mirror_search_dns - Test searching dns patterns"""
3185+ pmir = "phit"
3186+ smir = "shit"
3187+ cfg = {"primary": [{'arches': ["default"],
3188+ "search_dns": True}],
3189+ "security": [{'arches': ["default"],
3190+ "search_dns": True}]}
3191+
3192+ with mock.patch.object(apt_config, 'get_mirror',
3193+ return_value="http://mocked/foo") as mockgm:
3194+ mirrors = apt_config.find_apt_mirror_info(cfg)
3195+ calls = [call(cfg, 'primary', util.get_architecture()),
3196+ call(cfg, 'security', util.get_architecture())]
3197+ mockgm.assert_has_calls(calls)
3198+
3199+ with mock.patch.object(apt_config, 'search_for_mirror_dns',
3200+ return_value="http://mocked/foo") as mocksdns:
3201+ mirrors = apt_config.find_apt_mirror_info(cfg)
3202+ calls = [call(True, 'mirror'),
3203+ call(True, 'security-mirror')]
3204+ mocksdns.assert_has_calls(calls)
3205+
3206+ # first return is for the non-dns call before
3207+ with mock.patch.object(apt_config, 'search_for_mirror',
3208+ side_effect=[None, pmir, None, smir]) as mockse:
3209+ with mock.patch.object(util, 'subp',
3210+ return_value=('host.sub.com', '')) as mocks:
3211+ mirrors = apt_config.find_apt_mirror_info(cfg)
3212+
3213+ calls = [call(None),
3214+ call(['http://ubuntu-mirror.sub.com/ubuntu',
3215+ 'http://ubuntu-mirror.localdomain/ubuntu',
3216+ 'http://ubuntu-mirror/ubuntu']),
3217+ call(None),
3218+ call(['http://ubuntu-security-mirror.sub.com/ubuntu',
3219+ 'http://ubuntu-security-mirror.localdomain/ubuntu',
3220+ 'http://ubuntu-security-mirror/ubuntu'])]
3221+ mockse.assert_has_calls(calls)
3222+ mocks.assert_called_with(['hostname', '--fqdn'], capture=True, rcs=[0])
3223+
3224+ self.assertEqual(mirrors['MIRROR'],
3225+ pmir)
3226+ self.assertEqual(mirrors['PRIMARY'],
3227+ pmir)
3228+ self.assertEqual(mirrors['SECURITY'],
3229+ smir)
3230+
3231+ def test_mirror_search_many3(self):
3232+ """test_mirror_search_many3 - Test all three mirrors specs at once"""
3233+ pmir = "http://us.archive.ubuntu.com/ubuntu/"
3234+ smir = "http://security.ubuntu.com/ubuntu/"
3235+ cfg = {"primary": [{'arches': ["default"],
3236+ "uri": pmir,
3237+ "search_dns": True,
3238+ "search": ["pfailme", "foo"]}],
3239+ "security": [{'arches': ["default"],
3240+ "uri": smir,
3241+ "search_dns": True,
3242+ "search": ["sfailme", "bar"]}]}
3243+
3244+ # should be called once per type, despite three configs each
3245+ with mock.patch.object(apt_config, 'get_mirror',
3246+ return_value="http://mocked/foo") as mockgm:
3247+ mirrors = apt_config.find_apt_mirror_info(cfg)
3248+ calls = [call(cfg, 'primary', util.get_architecture()),
3249+ call(cfg, 'security', util.get_architecture())]
3250+ mockgm.assert_has_calls(calls)
3251+
3252+ # should not be called, since primary is specified
3253+ with mock.patch.object(apt_config, 'search_for_mirror_dns') as mockdns:
3254+ mirrors = apt_config.find_apt_mirror_info(cfg)
3255+ mockdns.assert_not_called()
3256+
3257+ # should not be called, since primary is specified
3258+ with mock.patch.object(apt_config, 'search_for_mirror') as mockse:
3259+ mirrors = apt_config.find_apt_mirror_info(cfg)
3260+ mockse.assert_not_called()
3261+
3262+ self.assertEqual(mirrors['MIRROR'],
3263+ pmir)
3264+ self.assertEqual(mirrors['PRIMARY'],
3265+ pmir)
3266+ self.assertEqual(mirrors['SECURITY'],
3267+ smir)
3268+
3269+ def test_mirror_search_many2(self):
3270+ """test_mirror_search_many2 - Test the two search specs at once"""
3271+ pmir = "http://us.archive.ubuntu.com/ubuntu/"
3272+ smir = "http://security.ubuntu.com/ubuntu/"
3273+ cfg = {"primary": [{'arches': ["default"],
3274+ "search_dns": True,
3275+ "search": ["pfailme", pmir]}],
3276+ "security": [{'arches': ["default"],
3277+ "search_dns": True,
3278+ "search": ["sfailme", smir]}]}
3279+
3280+ # should be called once per type, despite three configs each
3281+ with mock.patch.object(apt_config, 'get_mirror',
3282+ return_value="http://mocked/foo") as mockgm:
3283+ mirrors = apt_config.find_apt_mirror_info(cfg)
3284+ calls = [call(cfg, 'primary', util.get_architecture()),
3285+ call(cfg, 'security', util.get_architecture())]
3286+ mockgm.assert_has_calls(calls)
3287+
3288+ # this should be the winner by priority, despite config order
3289+ with mock.patch.object(apt_config, 'search_for_mirror',
3290+ side_effect=[pmir, smir]) as mocksearch:
3291+ mirrors = apt_config.find_apt_mirror_info(cfg)
3292+ calls = [call(["pfailme", pmir]),
3293+ call(["sfailme", smir])]
3294+ mocksearch.assert_has_calls(calls)
3295+
3296+ self.assertEqual(mirrors['MIRROR'],
3297+ pmir)
3298+ self.assertEqual(mirrors['PRIMARY'],
3299+ pmir)
3300+ self.assertEqual(mirrors['SECURITY'],
3301+ smir)
3302+
3303+ def test_url_resolvable(self):
3304+ """test_url_resolvable - Test resolving urls"""
3305+
3306+ with mock.patch.object(util, 'is_resolvable') as mockresolve:
3307+ util.is_resolvable_url("http://1.2.3.4/ubuntu")
3308+ mockresolve.assert_called_with("1.2.3.4")
3309+
3310+ with mock.patch.object(util, 'is_resolvable') as mockresolve:
3311+ util.is_resolvable_url("http://us.archive.ubuntu.com/ubuntu")
3312+ mockresolve.assert_called_with("us.archive.ubuntu.com")
3313+
3314+ bad = [(None, None, None, "badname", ["10.3.2.1"])]
3315+ good = [(None, None, None, "goodname", ["10.2.3.4"])]
3316+ with mock.patch.object(socket, 'getaddrinfo',
3317+ side_effect=[bad, bad, good,
3318+ good]) as mocksock:
3319+ ret = util.is_resolvable_url("http://us.archive.ubuntu.com/ubuntu")
3320+ ret2 = util.is_resolvable_url("http://1.2.3.4/ubuntu")
3321+ calls = [call('does-not-exist.example.com.', None, 0, 0, 1, 2),
3322+ call('example.invalid.', None, 0, 0, 1, 2),
3323+ call('us.archive.ubuntu.com', None),
3324+ call('1.2.3.4', None)]
3325+ mocksock.assert_has_calls(calls)
3326+ self.assertTrue(ret)
3327+ self.assertTrue(ret2)
3328+
3329+ # side effect need only bad ret after initial call
3330+ with mock.patch.object(socket, 'getaddrinfo',
3331+ side_effect=[bad]) as mocksock:
3332+ ret3 = util.is_resolvable_url("http://failme.com/ubuntu")
3333+ calls = [call('failme.com', None)]
3334+ mocksock.assert_has_calls(calls)
3335+ self.assertFalse(ret3)
3336+
3337+ def test_disable_suites(self):
3338+ """test_disable_suites - disable_suites with many configurations"""
3339+ release = "xenial"
3340+ orig = """deb http://ubuntu.com//ubuntu xenial main
3341+deb http://ubuntu.com//ubuntu xenial-updates main
3342+deb http://ubuntu.com//ubuntu xenial-security main
3343+deb-src http://ubuntu.com//ubuntu universe multiverse
3344+deb http://ubuntu.com/ubuntu/ xenial-proposed main"""
3345+
3346+ # disable nothing
3347+ cfg = {"disable_suites": []}
3348+ expect = """deb http://ubuntu.com//ubuntu xenial main
3349+deb http://ubuntu.com//ubuntu xenial-updates main
3350+deb http://ubuntu.com//ubuntu xenial-security main
3351+deb-src http://ubuntu.com//ubuntu universe multiverse
3352+deb http://ubuntu.com/ubuntu/ xenial-proposed main"""
3353+ result = apt_config.disable_suites(cfg, orig, release)
3354+ self.assertEqual(expect, result)
3355+
3356+ # single disable release suite
3357+ cfg = {"disable_suites": ["$RELEASE"]}
3358+ expect = """\
3359+# suite disabled by curtin: deb http://ubuntu.com//ubuntu xenial main
3360+deb http://ubuntu.com//ubuntu xenial-updates main
3361+deb http://ubuntu.com//ubuntu xenial-security main
3362+deb-src http://ubuntu.com//ubuntu universe multiverse
3363+deb http://ubuntu.com/ubuntu/ xenial-proposed main"""
3364+ result = apt_config.disable_suites(cfg, orig, release)
3365+ self.assertEqual(expect, result)
3366+
3367+ # single disable other suite
3368+ cfg = {"disable_suites": ["$RELEASE-updates"]}
3369+ expect = """deb http://ubuntu.com//ubuntu xenial main
3370+# suite disabled by curtin: deb http://ubuntu.com//ubuntu xenial-updates main
3371+deb http://ubuntu.com//ubuntu xenial-security main
3372+deb-src http://ubuntu.com//ubuntu universe multiverse
3373+deb http://ubuntu.com/ubuntu/ xenial-proposed main"""
3374+ result = apt_config.disable_suites(cfg, orig, release)
3375+ self.assertEqual(expect, result)
3376+
3377+ # multi disable
3378+ cfg = {"disable_suites": ["$RELEASE-updates", "$RELEASE-security"]}
3379+ expect = """deb http://ubuntu.com//ubuntu xenial main
3380+# suite disabled by curtin: deb http://ubuntu.com//ubuntu xenial-updates main
3381+# suite disabled by curtin: deb http://ubuntu.com//ubuntu xenial-security main
3382+deb-src http://ubuntu.com//ubuntu universe multiverse
3383+deb http://ubuntu.com/ubuntu/ xenial-proposed main"""
3384+ result = apt_config.disable_suites(cfg, orig, release)
3385+ self.assertEqual(expect, result)
3386+
3387+ # multi line disable (same suite multiple times in input)
3388+ cfg = {"disable_suites": ["$RELEASE-updates", "$RELEASE-security"]}
3389+ orig = """deb http://ubuntu.com//ubuntu xenial main
3390+deb http://ubuntu.com//ubuntu xenial-updates main
3391+deb http://ubuntu.com//ubuntu xenial-security main
3392+deb-src http://ubuntu.com//ubuntu universe multiverse
3393+deb http://UBUNTU.com//ubuntu xenial-updates main
3394+deb http://UBUNTU.COM//ubuntu xenial-updates main
3395+deb http://ubuntu.com/ubuntu/ xenial-proposed main"""
3396+ expect = """deb http://ubuntu.com//ubuntu xenial main
3397+# suite disabled by curtin: deb http://ubuntu.com//ubuntu xenial-updates main
3398+# suite disabled by curtin: deb http://ubuntu.com//ubuntu xenial-security main
3399+deb-src http://ubuntu.com//ubuntu universe multiverse
3400+# suite disabled by curtin: deb http://UBUNTU.com//ubuntu xenial-updates main
3401+# suite disabled by curtin: deb http://UBUNTU.COM//ubuntu xenial-updates main
3402+deb http://ubuntu.com/ubuntu/ xenial-proposed main"""
3403+ result = apt_config.disable_suites(cfg, orig, release)
3404+ self.assertEqual(expect, result)
3405+
3406+ # comment in input
3407+ cfg = {"disable_suites": ["$RELEASE-updates", "$RELEASE-security"]}
3408+ orig = """deb http://ubuntu.com//ubuntu xenial main
3409+deb http://ubuntu.com//ubuntu xenial-updates main
3410+deb http://ubuntu.com//ubuntu xenial-security main
3411+deb-src http://ubuntu.com//ubuntu universe multiverse
3412+#foo
3413+#deb http://UBUNTU.com//ubuntu xenial-updates main
3414+deb http://UBUNTU.COM//ubuntu xenial-updates main
3415+deb http://ubuntu.com/ubuntu/ xenial-proposed main"""
3416+ expect = """deb http://ubuntu.com//ubuntu xenial main
3417+# suite disabled by curtin: deb http://ubuntu.com//ubuntu xenial-updates main
3418+# suite disabled by curtin: deb http://ubuntu.com//ubuntu xenial-security main
3419+deb-src http://ubuntu.com//ubuntu universe multiverse
3420+#foo
3421+#deb http://UBUNTU.com//ubuntu xenial-updates main
3422+# suite disabled by curtin: deb http://UBUNTU.COM//ubuntu xenial-updates main
3423+deb http://ubuntu.com/ubuntu/ xenial-proposed main"""
3424+ result = apt_config.disable_suites(cfg, orig, release)
3425+ self.assertEqual(expect, result)
3426+
3427+ # single disable custom suite
3428+ cfg = {"disable_suites": ["foobar"]}
3429+ orig = """deb http://ubuntu.com//ubuntu xenial main
3430+deb http://ubuntu.com//ubuntu xenial-updates main
3431+deb http://ubuntu.com//ubuntu xenial-security main
3432+deb http://ubuntu.com/ubuntu/ foobar main"""
3433+ expect = """deb http://ubuntu.com//ubuntu xenial main
3434+deb http://ubuntu.com//ubuntu xenial-updates main
3435+deb http://ubuntu.com//ubuntu xenial-security main
3436+# suite disabled by curtin: deb http://ubuntu.com/ubuntu/ foobar main"""
3437+ result = apt_config.disable_suites(cfg, orig, release)
3438+ self.assertEqual(expect, result)
3439+
3440+ # single disable non existing suite
3441+ cfg = {"disable_suites": ["foobar"]}
3442+ orig = """deb http://ubuntu.com//ubuntu xenial main
3443+deb http://ubuntu.com//ubuntu xenial-updates main
3444+deb http://ubuntu.com//ubuntu xenial-security main
3445+deb http://ubuntu.com/ubuntu/ notfoobar main"""
3446+ expect = """deb http://ubuntu.com//ubuntu xenial main
3447+deb http://ubuntu.com//ubuntu xenial-updates main
3448+deb http://ubuntu.com//ubuntu xenial-security main
3449+deb http://ubuntu.com/ubuntu/ notfoobar main"""
3450+ result = apt_config.disable_suites(cfg, orig, release)
3451+ self.assertEqual(expect, result)
3452+
3453+ # single disable suite with option
3454+ cfg = {"disable_suites": ["$RELEASE-updates"]}
3455+ orig = """deb http://ubuntu.com//ubuntu xenial main
3456+deb [a=b] http://ubu.com//ubu xenial-updates main
3457+deb http://ubuntu.com//ubuntu xenial-security main
3458+deb-src http://ubuntu.com//ubuntu universe multiverse
3459+deb http://ubuntu.com/ubuntu/ xenial-proposed main"""
3460+ expect = """deb http://ubuntu.com//ubuntu xenial main
3461+# suite disabled by curtin: deb [a=b] http://ubu.com//ubu xenial-updates main
3462+deb http://ubuntu.com//ubuntu xenial-security main
3463+deb-src http://ubuntu.com//ubuntu universe multiverse
3464+deb http://ubuntu.com/ubuntu/ xenial-proposed main"""
3465+ result = apt_config.disable_suites(cfg, orig, release)
3466+ self.assertEqual(expect, result)
3467+
3468+ # single disable suite with more options and auto $RELEASE expansion
3469+ cfg = {"disable_suites": ["updates"]}
3470+ orig = """deb http://ubuntu.com//ubuntu xenial main
3471+deb [a=b c=d] http://ubu.com//ubu xenial-updates main
3472+deb http://ubuntu.com//ubuntu xenial-security main
3473+deb-src http://ubuntu.com//ubuntu universe multiverse
3474+deb http://ubuntu.com/ubuntu/ xenial-proposed main"""
3475+ expect = """deb http://ubuntu.com//ubuntu xenial main
3476+# suite disabled by curtin: deb [a=b c=d] \
3477+http://ubu.com//ubu xenial-updates main
3478+deb http://ubuntu.com//ubuntu xenial-security main
3479+deb-src http://ubuntu.com//ubuntu universe multiverse
3480+deb http://ubuntu.com/ubuntu/ xenial-proposed main"""
3481+ result = apt_config.disable_suites(cfg, orig, release)
3482+ self.assertEqual(expect, result)
3483+
3484+ # single disable suite while options at others
3485+ cfg = {"disable_suites": ["$RELEASE-security"]}
3486+ orig = """deb http://ubuntu.com//ubuntu xenial main
3487+deb [arch=foo] http://ubuntu.com//ubuntu xenial-updates main
3488+deb http://ubuntu.com//ubuntu xenial-security main
3489+deb-src http://ubuntu.com//ubuntu universe multiverse
3490+deb http://ubuntu.com/ubuntu/ xenial-proposed main"""
3491+ expect = """deb http://ubuntu.com//ubuntu xenial main
3492+deb [arch=foo] http://ubuntu.com//ubuntu xenial-updates main
3493+# suite disabled by curtin: deb http://ubuntu.com//ubuntu xenial-security main
3494+deb-src http://ubuntu.com//ubuntu universe multiverse
3495+deb http://ubuntu.com/ubuntu/ xenial-proposed main"""
3496+ result = apt_config.disable_suites(cfg, orig, release)
3497+ self.assertEqual(expect, result)
3498+
3499+#
3500+# vi: ts=4 expandtab
3501
3502=== modified file 'tests/vmtests/__init__.py'
3503--- tests/vmtests/__init__.py 2016-07-19 19:25:08 +0000
3504+++ tests/vmtests/__init__.py 2016-07-26 19:53:43 +0000
3505@@ -355,6 +355,7 @@
3506 extra_kern_args = None
3507 fstab_expected = {}
3508 image_store_class = ImageStore
3509+ boot_cloudconf = None
3510 install_timeout = 3000
3511 interactive = False
3512 multipath = False
3513@@ -363,6 +364,7 @@
3514 recorded_errors = 0
3515 recorded_failures = 0
3516 uefi = False
3517+ proxy = None
3518
3519 # these get set from base_vm_classes
3520 release = None
3521@@ -392,7 +394,8 @@
3522 # set up tempdir
3523 cls.td = TempDir(
3524 name=cls.__name__,
3525- user_data=generate_user_data(collect_scripts=cls.collect_scripts))
3526+ user_data=generate_user_data(collect_scripts=cls.collect_scripts,
3527+ boot_cloudconf=cls.boot_cloudconf))
3528 logger.info('Using tempdir: %s , Image: %s', cls.td.tmpdir,
3529 img_verstr)
3530 cls.install_log = os.path.join(cls.td.logs, 'install-serial.log')
3531@@ -492,11 +495,11 @@
3532
3533 # proxy config
3534 configs = [cls.conf_file]
3535- proxy = get_apt_proxy()
3536- if get_apt_proxy is not None:
3537+ cls.proxy = get_apt_proxy()
3538+ if cls.proxy is not None:
3539 proxy_config = os.path.join(cls.td.install, 'proxy.cfg')
3540 with open(proxy_config, "w") as fp:
3541- fp.write(json.dumps({'apt_proxy': proxy}) + "\n")
3542+ fp.write(json.dumps({'apt_proxy': cls.proxy}) + "\n")
3543 configs.append(proxy_config)
3544
3545 uefi_flags = []
3546@@ -731,8 +734,14 @@
3547 # Misc functions that are useful for many tests
3548 def output_files_exist(self, files):
3549 for f in files:
3550+ logger.debug('checking file %s', f)
3551 self.assertTrue(os.path.exists(os.path.join(self.td.collect, f)))
3552
3553+ def output_files_dont_exist(self, files):
3554+ for f in files:
3555+ logger.debug('checking file %s', f)
3556+ self.assertFalse(os.path.exists(os.path.join(self.td.collect, f)))
3557+
3558 def check_file_strippedline(self, filename, search):
3559 with open(os.path.join(self.td.collect, filename), "r") as fp:
3560 data = list(i.strip() for i in fp.readlines())
3561@@ -961,7 +970,8 @@
3562 return None
3563
3564
3565-def generate_user_data(collect_scripts=None, apt_proxy=None):
3566+def generate_user_data(collect_scripts=None, apt_proxy=None,
3567+ boot_cloudconf=None):
3568 # this returns the user data for the *booted* system
3569 # its a cloud-config-archive type, which is
3570 # just a list of parts. the 'x-shellscript' parts
3571@@ -986,6 +996,10 @@
3572 'content': yaml.dump(base_cloudconfig, indent=1)},
3573 {'type': 'text/cloud-config', 'content': ssh_keys}]
3574
3575+ if boot_cloudconf is not None:
3576+ parts.append({'type': 'text/cloud-config', 'content':
3577+ yaml.dump(boot_cloudconf, indent=1)})
3578+
3579 output_dir = '/mnt/output'
3580 output_dir_macro = 'OUTPUT_COLLECT_D'
3581 output_device = '/dev/disk/by-id/virtio-%s' % OUTPUT_DISK_NAME
3582
3583=== added file 'tests/vmtests/test_apt_config_cmd.py'
3584--- tests/vmtests/test_apt_config_cmd.py 1970-01-01 00:00:00 +0000
3585+++ tests/vmtests/test_apt_config_cmd.py 2016-07-26 19:53:43 +0000
3586@@ -0,0 +1,55 @@
3587+""" test_apt_config_cmd
3588+ Collection of tests for the apt configuration features when called via the
3589+ apt-config standalone command.
3590+"""
3591+import textwrap
3592+
3593+from . import VMBaseClass
3594+from .releases import base_vm_classes as relbase
3595+
3596+
3597+class TestAptConfigCMD(VMBaseClass):
3598+ """TestAptConfigCMD - test standalone command"""
3599+ conf_file = "examples/tests/apt_config_command.yaml"
3600+ interactive = False
3601+ extra_disks = []
3602+ fstab_expected = {}
3603+ disk_to_check = []
3604+ collect_scripts = [textwrap.dedent("""
3605+ cd OUTPUT_COLLECT_D
3606+ cat /etc/fstab > fstab
3607+ ls /dev/disk/by-dname > ls_dname
3608+ find /etc/network/interfaces.d > find_interfacesd
3609+ cp /etc/apt/sources.list.d/curtin-dev-ubuntu-test-archive-*.list .
3610+ cp /etc/cloud/cloud.cfg.d/curtin-preserve-sources.cfg .
3611+ apt-cache policy | grep proposed > proposed-enabled
3612+ """)]
3613+
3614+ def test_cmd_proposed_enabled(self):
3615+ """check if proposed was enabled"""
3616+ self.output_files_exist(["proposed-enabled"])
3617+ self.check_file_regex("proposed-enabled",
3618+ r"500.*%s-proposed" % self.release)
3619+
3620+ def test_cmd_ppa_enabled(self):
3621+ """check if specified curtin-dev ppa was enabled"""
3622+ self.output_files_exist(
3623+ ["curtin-dev-ubuntu-test-archive-%s.list" % self.release])
3624+ self.check_file_regex("curtin-dev-ubuntu-test-archive-%s.list" %
3625+ self.release,
3626+ (r"http://ppa.launchpad.net/"
3627+ r"curtin-dev/test-archive/ubuntu"
3628+ r" %s main" % self.release))
3629+
3630+ def test_cmd_preserve_source(self):
3631+ """check if cloud-init was prevented from overwriting"""
3632+ self.output_files_exist(["curtin-preserve-sources.cfg"])
3633+ self.check_file_regex("curtin-preserve-sources.cfg",
3634+ "apt_preserve_sources_list.*true")
3635+
3636+
3637+class XenialTestAptConfigCMDCMD(relbase.xenial, TestAptConfigCMD):
3638+ """ XenialTestAptSrcModifyCMD
3639+ apt feature Test for Xenial using the standalone command
3640+ """
3641+ __test__ = True
3642
3643=== added file 'tests/vmtests/test_apt_source.py'
3644--- tests/vmtests/test_apt_source.py 1970-01-01 00:00:00 +0000
3645+++ tests/vmtests/test_apt_source.py 2016-07-26 19:53:43 +0000
3646@@ -0,0 +1,277 @@
3647+""" test_apt_source
3648+ Collection of tests for the apt configuration features
3649+"""
3650+import textwrap
3651+
3652+from . import VMBaseClass
3653+from .releases import base_vm_classes as relbase
3654+
3655+from unittest import SkipTest
3656+from curtin import util
3657+
3658+
3659+class TestAptSrcAbs(VMBaseClass):
3660+ """TestAptSrcAbs - Basic tests for apt features of curtin"""
3661+ interactive = False
3662+ extra_disks = []
3663+ fstab_expected = {}
3664+ disk_to_check = []
3665+ collect_scripts = [textwrap.dedent("""
3666+ cd OUTPUT_COLLECT_D
3667+ cat /etc/fstab > fstab
3668+ ls /dev/disk/by-dname > ls_dname
3669+ find /etc/network/interfaces.d > find_interfacesd
3670+ apt-key list "F430BBA5" > keyid-F430BBA5
3671+ apt-key list "0165013E" > keyppa-0165013E
3672+ apt-key list "F470A0AC" > keylongid-F470A0AC
3673+ apt-key list "8280B242" > keyraw-8280B242
3674+ ls -laF /etc/apt/sources.list.d/ > sources.list.d
3675+ cp /etc/apt/sources.list.d/curtin-dev-ppa.list .
3676+ cp /etc/apt/sources.list.d/my-repo2.list .
3677+ cp /etc/apt/sources.list.d/my-repo4.list .
3678+ cp /etc/apt/sources.list.d/curtin-dev-ubuntu-test-archive-*.list .
3679+ find /etc/apt/sources.list.d/ -maxdepth 1 -name "*ignore*" | wc -l > ic
3680+ apt-config dump | grep Retries > aptconf
3681+ cp /etc/apt/sources.list sources.list
3682+ cp /etc/cloud/cloud.cfg.d/curtin-preserve-sources.cfg .
3683+ """)]
3684+ mirror = "http://us.archive.ubuntu.com/ubuntu"
3685+ secmirror = "http://security.ubuntu.com/ubuntu"
3686+
3687+ def test_output_files_exist(self):
3688+ """test_output_files_exist - Check if all output files exist"""
3689+ self.output_files_exist(
3690+ ["fstab", "ic", "keyid-F430BBA5", "keylongid-F470A0AC",
3691+ "keyraw-8280B242", "keyppa-0165013E", "aptconf", "sources.list",
3692+ "curtin-dev-ppa.list", "my-repo2.list", "my-repo4.list"])
3693+ self.output_files_exist(
3694+ ["curtin-dev-ubuntu-test-archive-%s.list" % self.release])
3695+
3696+ def test_keys_imported(self):
3697+ """test_keys_imported - Check if all keys are imported correctly"""
3698+ self.check_file_regex("keyid-F430BBA5",
3699+ r"Launchpad PPA for Ubuntu Screen Profile")
3700+ self.check_file_regex("keylongid-F470A0AC",
3701+ r"Ryan Harper")
3702+ self.check_file_regex("keyppa-0165013E",
3703+ r"Launchpad PPA for curtin developers")
3704+ self.check_file_regex("keyraw-8280B242",
3705+ r"Christian Ehrhardt")
3706+
3707+ def test_preserve_source(self):
3708+ """test_preserve_source - no clobbering sources.list by cloud-init"""
3709+ self.output_files_exist(["curtin-preserve-sources.cfg"])
3710+ self.check_file_regex("curtin-preserve-sources.cfg",
3711+ "apt_preserve_sources_list.*true")
3712+
3713+ def test_source_files(self):
3714+ """test_source_files - Check generated .lists for correct content"""
3715+ # hard coded deb lines
3716+ self.check_file_strippedline("curtin-dev-ppa.list",
3717+ ("deb http://ppa.launchpad.net/curtin-dev"
3718+ "/test-archive/ubuntu xenial main"))
3719+ self.check_file_strippedline("my-repo4.list",
3720+ ("deb http://ppa.launchpad.net/curtin-dev"
3721+ "/test-archive/ubuntu xenial main"))
3722+ # mirror and release replacement in deb line
3723+ self.check_file_strippedline("my-repo2.list", "deb %s %s multiverse" %
3724+ (self.mirror, self.release))
3725+ # auto creation by apt-add-repository
3726+ self.check_file_regex("curtin-dev-ubuntu-test-archive-%s.list" %
3727+ self.release,
3728+ (r"http://ppa.launchpad.net/"
3729+ r"curtin-dev/test-archive/ubuntu"
3730+ r" %s main" % self.release))
3731+
3732+ def test_ignore_count(self):
3733+ """test_ignore_count - Check for files that should not be created"""
3734+ self.check_file_strippedline("ic", "0")
3735+
3736+ def test_apt_conf(self):
3737+ """test_apt_conf - Check if the selected apt conf was set"""
3738+ self.check_file_strippedline("aptconf", 'Acquire::Retries "3";')
3739+
3740+
3741+class TestAptSrcCustom(TestAptSrcAbs):
3742+ """TestAptSrcNormal - tests valid in the custom sources.list case"""
3743+ conf_file = "examples/tests/apt_source_custom.yaml"
3744+
3745+ def test_custom_source_list(self):
3746+ """test_custom_source_list - Check custom sources with replacement"""
3747+ # check that all replacements happened
3748+ self.check_file_strippedline("sources.list",
3749+ "deb %s %s main restricted" %
3750+ (self.mirror, self.release))
3751+ self.check_file_strippedline("sources.list",
3752+ "deb-src %s %s main restricted" %
3753+ (self.mirror, self.release))
3754+ self.check_file_strippedline("sources.list",
3755+ "deb %s %s universe restricted" %
3756+ (self.mirror, self.release))
3757+ self.check_file_strippedline("sources.list",
3758+ "deb %s %s-security multiverse" %
3759+ (self.secmirror, self.release))
3760+ # check for something that guarantees us to come from our test
3761+ self.check_file_strippedline("sources.list",
3762+ "# nice line to check in test")
3763+
3764+
3765+class TestAptSrcPreserve(TestAptSrcAbs):
3766+ """TestAptSrcPreserve - tests valid in the preserved sources.list case"""
3767+ conf_file = "examples/tests/apt_source_preserve.yaml"
3768+ boot_cloudconf = None
3769+
3770+ def test_preserved_source_list(self):
3771+ """test_preserved_source_list - Check sources to be preserved as-is"""
3772+ # curtin didn't touch it, so we should find what curtin set as default
3773+ self.check_file_regex("sources.list",
3774+ r"this file is written by cloud-init")
3775+
3776+ # overwrite inherited check to match situation here
3777+ def test_preserve_source(self):
3778+ """test_preserve_source - check apt_preserve_sources_list not set"""
3779+ self.output_files_dont_exist(["curtin-preserve-sources.cfg"])
3780+
3781+
3782+class TestAptSrcModify(TestAptSrcAbs):
3783+ """TestAptSrcModify - tests modifying sources.list"""
3784+ conf_file = "examples/tests/apt_source_modify.yaml"
3785+
3786+ def test_modified_source_list(self):
3787+ """test_modified_source_list - Check sources with replacement"""
3788+ # we set us.archive which is non default, check for that
3789+ # this will catch if a target ever changes the expected defaults we
3790+ # have to replace in case there is no custom template
3791+ self.check_file_regex("sources.list",
3792+ r"us.archive.ubuntu.com")
3793+ self.check_file_regex("sources.list",
3794+ r"security.ubuntu.com")
3795+
3796+
3797+class TestAptSrcDisablePockets(TestAptSrcAbs):
3798+ """TestAptSrcDisablePockets - tests disabling a suite in sources.list"""
3799+ conf_file = "examples/tests/apt_source_modify_disable_suite.yaml"
3800+
3801+ def test_disabled_suite(self):
3802+ """test_disabled_suite - Check if suites were disabled"""
3803+ # two not disabled
3804+ self.check_file_regex("sources.list",
3805+ r"deb.*us.archive.ubuntu.com")
3806+ self.check_file_regex("sources.list",
3807+ r"deb.*security.ubuntu.com")
3808+ # updates disabled
3809+ self.check_file_regex("sources.list",
3810+ r"# suite disabled by curtin:.*-updates")
3811+
3812+
3813+class TestAptSrcModifyArches(TestAptSrcModify):
3814+ """TestAptSrcModify - tests modifying sources.list with per arch mirror"""
3815+ # same test, just different yaml to specify the mirrors per arch
3816+ conf_file = "examples/tests/apt_source_modify_arches.yaml"
3817+
3818+
3819+class TestAptSrcSearch(TestAptSrcAbs):
3820+ """TestAptSrcSearch - tests checking a list of mirror options"""
3821+ conf_file = "examples/tests/apt_source_search.yaml"
3822+
3823+ def test_mirror_search(self):
3824+ """test_mirror_search
3825+ Check searching through a mirror list
3826+ This is checked in the test (late) intentionally.
3827+ No matter if resolution worked or failed it shouldn't fail
3828+ fatally (python error and trace).
3829+ We just can't rely on the content to be found in that case
3830+ so we skip the check then."""
3831+ res1 = util.is_resolvable_url("http://does.not.exist/ubuntu")
3832+ res2 = util.is_resolvable_url("http://does.also.not.exist/ubuntu")
3833+ res3 = util.is_resolvable_url("http://us.archive.ubuntu.com/ubuntu")
3834+ res4 = util.is_resolvable_url("http://security.ubuntu.com/ubuntu")
3835+ if res1 or res2 or not res3 or not res4:
3836+ raise SkipTest(("Name resolution not as required"
3837+ "(%s, %s, %s, %s)" % (res1, res2, res3, res4)))
3838+
3839+ self.check_file_regex("sources.list",
3840+ r"us.archive.ubuntu.com")
3841+ self.check_file_regex("sources.list",
3842+ r"security.ubuntu.com")
3843+
3844+
3845+class TestAptSrcSearchDNS(VMBaseClass):
3846+ """TestAptSrcSearchDNS - tests checking for predefined DNS names"""
3847+ interactive = False
3848+ extra_disks = []
3849+ fstab_expected = {}
3850+ conf_file = "examples/tests/apt_source_search_dns.yaml"
3851+ disk_to_check = []
3852+ collect_scripts = [textwrap.dedent("""
3853+ cd OUTPUT_COLLECT_D
3854+ cat /etc/fstab > fstab
3855+ ls /dev/disk/by-dname > ls_dname
3856+ find /etc/network/interfaces.d > find_interfacesd
3857+ cp /etc/apt/sources.list.d/dnssearch.list.disabled .
3858+ """)]
3859+
3860+ def test_output_files_exist(self):
3861+ """test_output_files_exist - Check if all output files exist"""
3862+ self.output_files_exist(["fstab", "dnssearch.list.disabled"])
3863+
3864+ def test_mirror_search_dns(self):
3865+ """test_mirror_search_dns - tests checking for predefined DNS names"""
3866+ # these should be the first it got resolved, so they should be in the
3867+ # sources.list file. We want to see that .lcoaldomain was not picked
3868+ # but instead what we added to the temp /etc/hosts
3869+ self.check_file_regex("dnssearch.list.disabled",
3870+ r"ubuntu-mirror/ubuntu.*multiverse")
3871+ self.check_file_regex("dnssearch.list.disabled",
3872+ r"ubuntu-mirror/ubuntu.*universe")
3873+ self.check_file_regex("dnssearch.list.disabled",
3874+ r"ubuntu-security-mirror/ubuntu.*main")
3875+
3876+
3877+class XenialTestAptSrcCustom(relbase.xenial, TestAptSrcCustom):
3878+ """ XenialTestAptSrcCustom
3879+ apt feature Test for Xenial with a custom template
3880+ """
3881+ __test__ = True
3882+
3883+
3884+class XenialTestAptSrcPreserve(relbase.xenial, TestAptSrcPreserve):
3885+ """ XenialTestAptSrcPreserve
3886+ apt feature Test for Xenial with apt_preserve_sources_list enabled
3887+ """
3888+ __test__ = True
3889+
3890+
3891+class XenialTestAptSrcModify(relbase.xenial, TestAptSrcModify):
3892+ """ XenialTestAptSrcModify
3893+ apt feature Test for Xenial modifying the sources.list of the image
3894+ """
3895+ __test__ = True
3896+
3897+
3898+class XenialTestAptSrcSearch(relbase.xenial, TestAptSrcSearch):
3899+ """ XenialTestAptSrcModify
3900+ apt feature Test for Xenial searching for mirrors
3901+ """
3902+ __test__ = True
3903+
3904+
3905+class XenialTestAptSrcSearchDNS(relbase.xenial, TestAptSrcSearchDNS):
3906+ """ XenialTestAptSrcModify
3907+ apt feature Test for Xenial searching for predefined DNS names
3908+ """
3909+ __test__ = True
3910+
3911+
3912+class XenialTestAptSrcModifyArches(relbase.xenial, TestAptSrcModifyArches):
3913+ """ XenialTestAptSrcModifyArches
3914+ apt feature Test for Xenial checking per arch mirror specification
3915+ """
3916+ __test__ = True
3917+
3918+
3919+class XenialTestAptSrcDisablePockets(relbase.xenial, TestAptSrcDisablePockets):
3920+ """ XenialTestAptSrcDisablePockets
3921+ apt feature Test for Xenial disabling a suite
3922+ """
3923+ __test__ = True
3924
3925=== added file 'tests/vmtests/test_old_apt_features.py'
3926--- tests/vmtests/test_old_apt_features.py 1970-01-01 00:00:00 +0000
3927+++ tests/vmtests/test_old_apt_features.py 2016-07-26 19:53:43 +0000
3928@@ -0,0 +1,80 @@
3929+""" testold_apt_features
3930+ Testing the former minimal apt features of curtin
3931+"""
3932+import re
3933+import textwrap
3934+
3935+from . import VMBaseClass
3936+from .releases import base_vm_classes as relbase
3937+
3938+from curtin import util
3939+
3940+
3941+class TestOldAptAbs(VMBaseClass):
3942+ """TestOldAptAbs - Basic tests for old apt features of curtin"""
3943+ interactive = False
3944+ extra_disks = []
3945+ fstab_expected = {}
3946+ disk_to_check = []
3947+ collect_scripts = [textwrap.dedent("""
3948+ cd OUTPUT_COLLECT_D
3949+ cat /etc/fstab > fstab
3950+ ls /dev/disk/by-dname > ls_dname
3951+ find /etc/network/interfaces.d > find_interfacesd
3952+ grep -A 3 "Name: debconf/priority" /var/cache/debconf/config.dat > debc
3953+ apt-config dump > aptconf
3954+ cp /etc/apt/apt.conf.d/90curtin-aptproxy .
3955+ cp /etc/apt/sources.list .
3956+ cp /etc/cloud/cloud.cfg.d/curtin-preserve-sources.cfg .
3957+ """)]
3958+ arch = util.get_architecture()
3959+ if arch in ['amd64', 'i386']:
3960+ conf_file = "examples/tests/test_old_apt_features.yaml"
3961+ exp_mirror = "http://us.archive.ubuntu.com/ubuntu"
3962+ exp_secmirror = "http://archive.ubuntu.com/ubuntu"
3963+ if arch in ['s390x', 'arm64', 'armhf', 'powerpc', 'ppc64el']:
3964+ conf_file = "examples/tests/test_old_apt_features_ports.yaml"
3965+ exp_mirror = "http://ports.ubuntu.com/ubuntu-ports"
3966+ exp_secmirror = "http://ports.ubuntu.com/ubuntu-ports"
3967+
3968+ def test_output_files_exist(self):
3969+ """test_output_files_exist - Check if all output files exist"""
3970+ self.output_files_exist(
3971+ ["debc", "aptconf", "sources.list", "90curtin-aptproxy",
3972+ "curtin-preserve-sources.cfg"])
3973+
3974+ def test_preserve_source(self):
3975+ """test_preserve_source - no clobbering sources.list by cloud-init"""
3976+ self.check_file_regex("curtin-preserve-sources.cfg",
3977+ "apt_preserve_sources_list.*true")
3978+
3979+ def test_debconf(self):
3980+ """test_debconf - Check if debconf is in place"""
3981+ self.check_file_strippedline("debc", "Value: low")
3982+
3983+ def test_aptconf(self):
3984+ """test_aptconf - Check if apt conf for proxy is in place"""
3985+ # this gets configured by tools/launch and get_apt_proxy in
3986+ # tests/vmtests/__init__.py, so compare with those
3987+ rproxy = r"Acquire::http::Proxy \"" + re.escape(self.proxy) + r"\";"
3988+ self.check_file_regex("aptconf", rproxy)
3989+ self.check_file_regex("90curtin-aptproxy", rproxy)
3990+
3991+ def test_mirrors(self):
3992+ """test_mirrors - Check for mirrors placed in source.list"""
3993+
3994+ self.check_file_strippedline("sources.list",
3995+ "deb %s %s" %
3996+ (self.exp_mirror, self.release) +
3997+ " main restricted universe multiverse")
3998+ self.check_file_strippedline("sources.list",
3999+ "deb %s %s-security" %
4000+ (self.exp_secmirror, self.release) +
4001+ " main restricted universe multiverse")
4002+
4003+
4004+class XenialTestOldApt(relbase.xenial, TestOldAptAbs):
4005+ """ XenialTestOldApt
4006+ Old apt features for Xenial
4007+ """
4008+ __test__ = True

Subscribers

People subscribed via source and target branches