Merge lp:~fginther/adt-cloud-worker/uci-nova-mirror into lp:~canonical-ci-engineering/adt-cloud-worker/uci-nova

Proposed by Francis Ginther on 2015-04-22
Status: Merged
Approved by: Francis Ginther on 2015-04-23
Approved revision: 15
Merged at revision: 11
Proposed branch: lp:~fginther/adt-cloud-worker/uci-nova-mirror
Merge into: lp:~canonical-ci-engineering/adt-cloud-worker/uci-nova
Prerequisite: lp:~fginther/adt-cloud-worker/uci-nova-wait-for-active
Diff against target: 73 lines (+25/-3)
1 file modified
uci-nova (+25/-3)
To merge this branch: bzr merge lp:~fginther/adt-cloud-worker/uci-nova-mirror
Reviewer Review Type Date Requested Status
Celso Providelo (community) Approve on 2015-04-23
Evan 2015-04-22 Pending
Review via email: mp+257024@code.launchpad.net

This proposal supersedes a proposal from 2015-04-21.

Commit message

Add support for an optional archive mirror and add multiverse if it's specified.

Description of the change

To post a comment you must log in.
Evan (ev) wrote : Posted in a previous version of this proposal

I know where the multiverse requirement is coming from, but why do we need this optional archive mirror?

review: Needs Information
Francis Ginther (fginther) wrote : Posted in a previous version of this proposal

Added some comments for reviewers.

Francis Ginther (fginther) wrote : Posted in a previous version of this proposal

> I know where the multiverse requirement is coming from, but why do we need
> this optional archive mirror?

The existing adt VM tests use ftpmaster.internal as the archive source in order to have access to packages as soon as they are published. Without this, tests end up running against older versions of packages and proposed-migration can't detect this until it gets to britney (and then a manual retry is required). We can't force the use of ftpmaster.internal because it's not publicly available and we want to maintain the ability to use uci-nova in other environments.

Providing --mirror as an option was the only solution I've found so far for satisfying the above.

Celso Providelo (cprov) wrote :

Francis,

Thanks for implementing this feature. I agree it's required to be optional because it depends on the environment it's operating, i.e. the same repository used for triggering britney analysis.

Although it seems entirely unfortunate that instead of dealing with possible/transient inconsistencies (needs to test version X, but it's not available, will retry later) the adt-jenkins system relies on external arrangements to make sure it never happens (internal mirrors, apt-caches, extra load on ftpmaster, etc).

I think we can do a lot better, if we start requesting and monitoring tests in adt-cloud direct from britney (I don't mean direct amqp access as uci-britney, but instead the synchronous rest api we already have).

That said, let's see how the uci-nova workaround performs in production.

review: Approve
Francis Ginther (fginther) wrote :

Celso,

I fully agree. During the Austin sprint, cjwatson and jibel mentioned that there is an alternate means to ensure that a testbed has the correct version of the archive by comparing timestamps. But there is additional work in other parts of the proposed-migration process to make this functional. I believing 'fixing' that problem is outside the scope of what we need to accomplish for this story. But it should be part of a more thorough update when the time comes.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'uci-nova'
2--- uci-nova 2015-04-22 03:06:12 +0000
3+++ uci-nova 2015-04-22 03:06:12 +0000
4@@ -17,6 +17,8 @@
5 # UUID of the network that should be used for the instance
6 # -n name | --name=name
7 # Name for the new server. A name will be generated if not specified.
8+# -m mirror | --mirror=mirror
9+# Optional ubuntu archive mirror to override the default.
10 # -c console-log | --console=file-name
11 # Save the nova console-log of the server to the specified file.
12 #
13@@ -56,6 +58,7 @@
14 SRVNAME=""
15 NET_ID=""
16 CONSOLE=""
17+MIRROR=""
18 DEBUG=""
19
20
21@@ -75,8 +78,8 @@
22 parse_args() {
23 # Parse command line argument and populate environment
24
25- SHORTOPTS="f:,i:,N:,n:,c:,d"
26- LONGOPTS="flavor:,image:,net-id:,name:,console:,debug"
27+ SHORTOPTS="f:,i:,N:,n:,m:,c:,d"
28+ LONGOPTS="flavor:,image:,net-id:,name:,mirror:,console:,debug"
29
30 TEMP=$(getopt -o $SHORTOPTS --long $LONGOPTS -- "$@")
31 eval set -- "$TEMP"
32@@ -95,6 +98,9 @@
33 -n|--name)
34 SRVNAME=$2
35 shift 2;;
36+ -m|--mirror)
37+ MIRROR=$2
38+ shift 2;;
39 -c|--console)
40 CONSOLE=$2
41 shift 2;;
42@@ -193,7 +199,21 @@
43 else
44 security_setup_nova
45 fi
46-
47+
48+ # Generate a new apt sources.list using either the specified mirror
49+ # or the test client's currently configured archive mirror. This includes
50+ # adding multiverse which is not enabled in our cloud images by default.
51+ #
52+ # The following archive_setup uses an awk command to determine the release
53+ # and archive mirror configured on the test client. $2 will be set to the
54+ # archive mirror and $3 to the release. These are used before overwriting
55+ # the test client's /etc/apt/sources.list file.
56+ if [ -n "${MIRROR}" ]; then
57+ archive_setup="mirror=${MIRROR}; release=\`awk '/^deb .*(debian|ubuntu)/ { print \$3; exit }' \"\$root/etc/apt/sources.list\"\`; echo \"deb \${mirror} \${release} main restricted universe multiverse\ndeb \${mirror} \${release}-updates main restricted universe multiverse\ndeb-src \${mirror} \${release} main restricted universe multiverse\ndeb-src \${mirror} \${release}-updates main restricted universe multiverse\" > /etc/apt/sources.list"
58+ else
59+ archive_setup="mirror_release=\`awk '/^deb .*(debian|ubuntu)/ { print \$2,\$3; exit }' \"\$root/etc/apt/sources.list\"\`; echo \"deb \${mirror_release} main restricted universe multiverse\ndeb \${mirror_release}-updates main restricted universe multiverse\ndeb-src \${mirror_release} main restricted universe multiverse\ndeb-src \${mirror_release}-updates main restricted universe multiverse\" > /etc/apt/sources.list"
60+ fi
61+
62 # generate cloud-init user data; mostly for manage_etc_hosts, but also get
63 # rid of some unnecessary stuff in the VM
64 #
65@@ -211,6 +231,8 @@
66 ssh_enabled: True
67
68 runcmd:
69+ # Setup archive
70+ - ${archive_setup}
71 # Make apt faster:
72 - echo 'Acquire::Languages "none";' > /etc/apt/apt.conf.d/90nolanguages
73 - echo 'force-unsafe-io' > /etc/dpkg/dpkg.cfg.d/autopkgtest

Subscribers

People subscribed via source and target branches