cloud-init

Merge ~powersj/cloud-init:create-centos-tests into cloud-init:master

  1. Git
  2. lp:~powersj/cloud-init
  3. create-centos-tests
  4. Merge into master
Proposed by Joshua Powers on 2017-06-01
Status: Merged
Merged at revision: ee324391bcb436b1d3a1c44951aa1aa673005cf6
Proposed branch: ~powersj/cloud-init:create-centos-tests
Merge into: cloud-init:master
Diff against target: 276 lines (+264/-0)
2 files modified
tools/run-centos (+215/-0)
tools/setup-centos (+49/-0)
Reviewer Review Type Date Requested Status
Server Team CI bot continuous-integration Approve on 2017-06-08
Ryan Harper 2017-06-01 Needs Fixing on 2017-06-07
Review via email: mp+324982@code.launchpad.net

Commit Message

tools: add centos scripts to build and test

* Creates centos 6 or 7 lxd container
    * Sets http_proxy variable for yum if set locally
    * Creates centos user
* Push local tree
    * Tar's up working directory
    * Pushes to container and untars
* Installs pip and yum dependencies
* As user centos it can then based on flags:
    * runs unittests
    * run ./packages/brpm
    * run ./packages/brpm --srpm
    * artifact the built *.rpm

Description of the Change

Read for review

To post a comment you must log in.
Ryan Harper (raharper) :
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Scott Moser (smoser) wrote :

i use 'newscript' to create scripts once they start needing options.
make sure you use it with '--spaces'

http://smoser.brickies.net/git/?p=tildabin.git;a=blob;f=newscript;h=494d31a8f8fa0c83d7c8b03d3d63deb641709b99;hb=HEAD

Joshua Powers (powersj) wrote :

@smoser: thanks for the template!

@rharper: responses below two big points:

1) Knowning when network is up is a better solution than the sleep. May try to see if cloud-init is finished if cloud-init is in the images or some other solution.
2) Knowing when we need to use the proxy. When this is run in the test env. we need to have the proxy set for yum to work properly.

Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Ryan Harper (raharper) wrote :

See some more inline comments.

review: Needs Fixing
~powersj/cloud-init:create-centos-tests updated on 2017-06-07
950151f... by Joshua Powers on 2017-06-07

Changing network check to do a getent

This will run getent instead of random sleeps to wait for the networking
to come up.

Joshua Powers (powersj) wrote :

Updated to use getnet to check if the network is up, removed extra sleeps, and added missing args.

Don't use make-tarball because we need the .git directory when we push into the container. When we are in the container doing the build, the build command itself will run the make-tarball command and needs the git log and values to know what version to call the tarball.

Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
~powersj/cloud-init:create-centos-tests updated on 2017-06-08
d7a80bb... by Joshua Powers on 2017-06-08

Clean up and fixes from smoser.

Joshua Powers (powersj) wrote :

Now, new and improved with smoser's fixes

Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/tools/run-centos b/tools/run-centos
2new file mode 100755
3index 0000000..fbc736d
4--- /dev/null
5+++ b/tools/run-centos
6@@ -0,0 +1,215 @@
7+#!/bin/bash
8+# This file is part of cloud-init. See LICENSE file for license information.
9+
10+set -u
11+
12+VERBOSITY=0
13+TEMP_D=""
14+KEEP=false
15+CONTAINER=""
16+
17+error() { echo "$@" 1>&2; }
18+fail() { [ $# -eq 0 ] || error "$@"; exit 1; }
19+errorrc() { local r=$?; error "$@" "ret=$r"; return $r; }
20+
21+Usage() {
22+ cat <<EOF
23+Usage: ${0##*/} [ options ] CentOS version
24+
25+ This utility can makes it easier to run tests, build rpm and source rpm
26+ generation inside a LXC of the specified version of CentOS.
27+
28+ options:
29+ -a | --artifact keep .rpm artifacts
30+ -k | --keep keep container after tests
31+ -r | --rpm build .rpm
32+ -s | --srpm build .src.rpm
33+ -u | --unittest run unit tests
34+EOF
35+}
36+
37+bad_Usage() { Usage 1>&2; [ $# -eq 0 ] || error "$@"; return 1; }
38+cleanup() {
39+ if [ -n "$CONTAINER" -a "$KEEP" = "false" ]; then
40+ delete_container "$CONTAINER"
41+ fi
42+ [ -z "${TEMP_D}" -o ! -d "${TEMP_D}" ] || rm -Rf "${TEMP_D}"
43+}
44+
45+debug() {
46+ local level=${1}; shift;
47+ [ "${level}" -gt "${VERBOSITY}" ] && return
48+ error "${@}"
49+}
50+
51+
52+inside_as() {
53+ local name="$1" user="$2"
54+ shift 2
55+ if [ $# -gt 1 ]; then
56+ error "inside_as only can take one arg. sorry."
57+ return 1
58+ fi
59+ inside "$name" su "$user" -c "$1"
60+}
61+
62+inside() {
63+ local name="$1"
64+ shift
65+ lxc exec "$name" -- "$@"
66+}
67+
68+inject_cloud_init(){
69+ local name="$1"
70+ tarball_name='cloud-init.tar.gz'
71+ top_d=$(git rev-parse --show-toplevel) ||
72+ fail "failed to get top level"
73+ cd "$top_d" ||
74+ fail "failed to cd to git top dir"
75+ tar_folder=${PWD##*/}
76+ cd ..
77+ tar -czf "$TEMP_D/$tarball_name" "$tar_folder" ||
78+ fail "failed: creating tarball_name"
79+ cd "$tar_folder" ||
80+ fail "failed: changing directory"
81+
82+ user='centos'
83+ tarball="/home/$user/$tarball_name"
84+ inside "$name" useradd "$user"
85+ lxc file push "$TEMP_D/$tarball_name" "$name/home/$user"/
86+ inside "$name" chown "$user:$user" "$tarball"
87+ inside_as "$name" "$user" "tar -C /home/$user -xzf $tarball" ||
88+ fail "failed: extracting tarball"
89+}
90+
91+start_container() {
92+ local src="$1" name="$2"
93+ debug 1 "starting container $name from '$src'"
94+ lxc launch "$src" "$name" || {
95+ errorrc "failed to start container '$name' from '$src'";
96+ return
97+ }
98+ CONTAINER=$name
99+
100+ local out="" ret=""
101+ debug 1 "waiting for networking"
102+ out=$(inside "$name" sh -c '
103+ i=0
104+ while [ $i -lt 60 ]; do
105+ getent hosts mirrorlist.centos.org && exit 0
106+ sleep 2
107+ done' 2>&1)
108+ ret=$?
109+ if [ $ret -ne 0 ]; then
110+ error "waiting for network in container '$name' failed. [$ret]"
111+ error "$out"
112+ return $ret
113+ fi
114+
115+ if [ ! -z "${http_proxy-}" ]; then
116+ debug 1 "configuring proxy ${http_proxy}"
117+ inside "$name" sh -c "echo proxy=$http_proxy >> /etc/yum.conf"
118+ fi
119+}
120+
121+delete_container() {
122+ debug 1 "removing container $1 [--keep to keep]"
123+ lxc delete --force "$1"
124+}
125+
126+main() {
127+ local short_opts="ahkrsuv:"
128+ local long_opts="artifact,help,keep,rpm,srpm,unittest,verbose:"
129+ local getopt_out=""
130+ getopt_out=$(getopt --name "${0##*/}" \
131+ --options "${short_opts}" --long "${long_opts}" -- "$@") &&
132+ eval set -- "${getopt_out}" ||
133+ { bad_Usage; return; }
134+
135+ local cur="" next=""
136+ local artifact="" keep="" rpm="" srpm="" unittest="" version=""
137+
138+ while [ $# -ne 0 ]; do
139+ cur="${1:-}"; next="${2:-}";
140+ case "$cur" in
141+ -a|--artifact) artifact=1;;
142+ -h|--help) Usage ; exit 0;;
143+ -k|--keep) KEEP=true;;
144+ -r|--rpm) rpm=1;;
145+ -s|--srpm) srpm=1;;
146+ -u|--unittest) unittest=1;;
147+ -v|--verbose) VERBOSITY=$((${VERBOSITY}+1));;
148+ --) shift; break;;
149+ esac
150+ shift;
151+ done
152+
153+ [ $# -eq 1 ] || { bad_Usage "ERROR: must provide version!"; return; }
154+ version="$1"
155+
156+ TEMP_D=$(mktemp -d "${TMPDIR:-/tmp}/${0##*/}.XXXXXX") ||
157+ fail "failed to make tempdir"
158+ trap cleanup EXIT
159+
160+ # program starts here
161+ local uuid="" name=""
162+ uuid=$(uuidgen -t) || { error "no uuidgen"; return 1; }
163+ name="cloud-init-centos-${uuid%%-*}"
164+
165+ start_container "images:centos/$version" "$name"
166+ # CentOS 6 does not come with tar
167+ if [ "$version" = "6" ]; then
168+ inside "$name" yum install --assumeyes tar || {
169+ errorrc "FAIL: yum install tar failed";
170+ }
171+ fi
172+
173+ debug 1 "inserting cloud-init"
174+ inject_cloud_init "$name" || {
175+ errorrc "FAIL: injecting cloud-init into $name failed."
176+ return
177+ }
178+
179+ # install dependencies
180+ debug 1 "installing dependencies"
181+ inside "$name" /bin/sh <tools/setup-centos ||
182+ fail "failed: setting up container $name"
183+
184+ local errors=0 do_cd="cd /home/$user/$tar_folder"
185+ inside_as "$name" "$user" "$do_cd; git checkout .; git status" ||
186+ { errorrc "git checkout failed."; errors=$(($errors+1)); }
187+
188+ if [ -n "$unittest" ]; then
189+ debug 1 "running unit tests."
190+ inside_as "$name" "$user" "$do_cd; nosetests tests/unittests" ||
191+ { errorrc "nosetests failed."; errors=$(($errors+1)); }
192+ fi
193+
194+ if [ -n "$srpm" ]; then
195+ debug 1 "building srpm."
196+ inside_as "$name" "$user" "$do_cd; ./packages/brpm --srpm" ||
197+ { errorrc "brpm --srpm."; errors=$(($errors+1)); }
198+ fi
199+
200+ if [ -n "$rpm" ]; then
201+ debug 1 "building rpm."
202+ inside_as "$name" "$user" "$do_cd; ./packages/brpm" ||
203+ { errorrc "brpm failed."; errors=$(($errors+1)); }
204+ fi
205+
206+ if [ -n "$artifact" ]; then
207+ cmd="ls /home/$user/$tar_folder/*.rpm"
208+ for built_rpm in $(lxc exec "$name" -- sh -c "$cmd"); do
209+ lxc file pull "$name/$built_rpm" .
210+ done
211+ fi
212+
213+ if [ "$errors" != "0" ]; then
214+ error "there were $errors errors."
215+ return 1
216+ fi
217+ return 0
218+}
219+
220+main "$@"
221+# vi: ts=4 expandtab
222diff --git a/tools/setup-centos b/tools/setup-centos
223new file mode 100755
224index 0000000..bc5da8a
225--- /dev/null
226+++ b/tools/setup-centos
227@@ -0,0 +1,49 @@
228+#!/bin/sh
229+# This file is part of cloud-init. See LICENSE file for license information.
230+set -fux
231+export LANG=C
232+
233+packages="
234+ file
235+ git
236+ pyserial
237+ python-argparse
238+ python-cheetah
239+ python-configobj
240+ python-devel
241+ python-jinja2
242+ python-jsonpatch
243+ python-oauthlib
244+ python-pip
245+ python-prettytable
246+ python-requests
247+ python-six
248+ PyYAML
249+ rpm-build
250+"
251+
252+pips="
253+ contextlib2
254+ httpretty
255+ mock
256+ nose
257+ pep8
258+ unittest2
259+"
260+
261+error() { echo "$@" 1>&2; }
262+fail() { [ $# -eq 0 ] || error "$@"; exit 1; }
263+info() { echo "$@"; }
264+
265+pips=$(for p in $pips; do echo "$p"; done | sort -u)
266+packages=$(for p in $packages; do echo "$p"; done | sort -u)
267+
268+if ! rpm -q epel-release >/dev/null; then
269+ yum install --assumeyes epel-release ||
270+ fail "failed: yum install epel-release"
271+fi
272+yum install --assumeyes $packages ||
273+ fail "failed: yum install" "$packages"
274+
275+pip install --upgrade $pips ||
276+ fail "failed: pip install $pips"

Subscribers

People subscribed via source and target branches