Merge lp:~deeptik/linaro-ci/fix-bug1013611 into lp:linaro-ci

[Deepti B. Kalakeri]

[Milo Casagrande (milo)]

Hello Deepti,

thanks for working on this!
Merge proposal looks good to me, the only thing I will add is or a README file with a little bit of docs/description of the various scripts, or small comments per script that describe the script intents (even if they are not that complex, at least it will be easier for other to pick up faster what has been done).

Also, I see that two files have license headers, others do not: what is the policy in these cases? Should all source code files have a license header?

Another thing, but definitely minor: files use underscores or dashes in their names. There are file names with the latter and other with the former. Should we use just one?

review: Needs Fixing

[lp:~deeptik/linaro-ci/fix-bug1013611]

73. By Deepti B. Kalakeri on 2012-06-18

Address review comments, added comments in every script, modified script names

[Deepti B. Kalakeri (deeptik)]

On Mon, Jun 18, 2012 at 2:47 PM, Milo Casagrande
<email address hidden>wrote:

> Review: Needs Fixing
>
> Hello Deepti,
>
> thanks for working on this!
> Merge proposal looks good to me, the only thing I will add is or a README
> file with a little bit of docs/description of the various scripts, or small
> comments per script that describe the script intents (even if they are not
> that complex, at least it will be easier for other to pick up faster what
> has been done).
>
> Thanks for the quick review.
I included lines in each of the scripts as its more easy to understand the
usage of the scripts.

> Also, I see that two files have license headers, others do not: what is
> the policy in these cases? Should all source code files have a license
> header?
>

The license header is for 2011 I will remove it for now, and would speak
with Danilo and the whole team on what is the appropriate header to be
included in the scripts.

>
> Another thing, but definitely minor: files use underscores or dashes in
> their names. There are file names with the latter and other with the
> former. Should we use just one?
>

AFAIK, there is no particular naming convention that I came across in
various tools in infrastructure.

>
> --
> https://code.launchpad.net/~deeptik/linaro-ci/fix-bug1013611/+merge/110740<https://code.launchpad.net/%7Edeeptik/linaro-ci/fix-bug1013611/+merge/110740>
> You are the owner of lp:~deeptik/linaro-ci/fix-bug1013611.
>

--
Thanks and Regards,
Deepti

[Milo Casagrande (milo)]

Hello Deepti,

thanks for adding some descriptions to the scritp!
As discussed on IRC, we can merge and in case fix later the other bits that need to be discussed.

review: Approve

[Paul Sokolovsky (pfalcon)]

This is definitely looks good, and thanks for quick turnaround! For custom AMI BP, we with Stevan might need to do further tweaks, but having this will allow us to concentrate on our code first of all instead of spreading thin.

I also appreciate flexibility on Milo's side - it's true that our codebases lack on consistency side quite a bunch, but fixing that in adhoc manner oftentimes only complicates and procrastinates functional work. What I dream of is of us getting a blueprint "Clean up our codebases consistently", settle on extra rules we currently miss (like '_' vs '-' dichotomy) have a good swipe over all our projects, and they maintain them on that level.

review: Approve

=== added directory 'node'
=== added file 'node/init-fifo'
--- node/init-fifo	1970-01-01 00:00:00 +0000
+++ node/init-fifo	2012-06-18 11:17:42 +0000
@@ -0,0 +1,19 @@
+#! /bin/sh
+
+# This script creates and acquires a fifo. 
+# This needs to be called in the slave init script
+# before calling any of the setup-*-node scripts
+
+set -x
+
+if ! test -e /tmp/fifo2; then
+        mkfifo /tmp/fifo2 || true
+fi
+if ! test -e /tmp/fifo3; then
+        mkfifo /tmp/fifo3 || true
+fi
+
+echo 'Acquire::http::Proxy "http://ci.linaro.org:3128";' > /etc/apt/apt.conf.d/ci_linaro_org_http_proxy
+
+echo "acquiring fifo lock"
+echo "start init" > /tmp/fifo2

=== added file 'node/setup-natty-node'
--- node/setup-natty-node	1970-01-01 00:00:00 +0000
+++ node/setup-natty-node	2012-06-18 11:17:42 +0000
@@ -0,0 +1,23 @@
+#!/bin/sh
+
+# This script contains the configurations required specific to the
+# Natty node, which is mainly used by kernel builds on ci.linaro.org
+
+set -xe
+
+. ./setup_lib
+
+export DEBIAN_FRONTEND=noninteractive
+
+# Install packages
+# ia32-libs is required to be able to run 32 bit gcc4.7 on 64 bit OS
+apt-get-retry install -y  build-essential \
+              default-jre \
+              git \
+              gcc-arm-linux-gnueabi \
+              uboot-mkimage \
+              python-beautifulsoup \
+              python-html2text \
+              ia32-libs 
+
+git_config

=== added file 'node/setup-oneiric-node'
--- node/setup-oneiric-node	1970-01-01 00:00:00 +0000
+++ node/setup-oneiric-node	2012-06-18 11:17:42 +0000
@@ -0,0 +1,32 @@
+#!/bin/sh
+
+# This script contains the configurations required specific to the
+# Oneiric node, which is mainly used by oneiric hwpack/image builds on ci.linaro.org
+
+set -xe
+
+. ./setup_lib
+
+export DEBIAN_FRONTEND=noninteractive
+
+# Install packages
+apt-get-retry install -y  build-essential \
+              debootstrap \
+              schroot  \
+              default-jre \
+              eatmydata  \
+              git \
+              gcc-arm-linux-gnueabi \
+              linaro-image-tools \
+              qemu-user-static \
+              uboot-mkimage 
+
+live_build_install
+git_config
+
+
+if test -e /tmp/fifo3; then
+    echo "end init" > /tmp/fifo3
+fi
+
+fifo_cleanup

=== added file 'node/setup-precise-node'
--- node/setup-precise-node	1970-01-01 00:00:00 +0000
+++ node/setup-precise-node	2012-06-18 11:17:42 +0000
@@ -0,0 +1,38 @@
+#!/bin/sh
+
+# This script contains the configurations required specific to the
+# Precise node, which is mainly used by precise hwpack/image and 
+# kernel builds on ci.linaro.org
+
+set -xe
+
+. ./setup_lib
+apt-get-retry install -y python-software-properties 
+add-apt-repository 'deb http://us.archive.ubuntu.com/ubuntu/ precise-backports main restricted universe'
+apt-get-retry --non-fatal update
+
+export DEBIAN_FRONTEND=noninteractive
+
+# Install packages
+apt-get-retry install -y   --target-release precise-backports debootstrap 
+apt-get-retry install -y  build-essential \
+              git \
+              schroot  \
+              default-jre \
+              eatmydata  \
+              gcc-arm-linux-gnueabi \
+              linaro-image-tools \
+              qemu-user-static \
+              uboot-mkimage \
+              python-beautifulsoup \
+              python-html2text \
+              gcc-arm-linux-gnueabihf 
+
+live_build_install
+git_config
+
+if test -e /tmp/fifo3; then
+    echo "end init" > /tmp/fifo3
+fi
+
+fifo_cleanup

=== added file 'node/setup_lib'
--- node/setup_lib	1970-01-01 00:00:00 +0000
+++ node/setup_lib	2012-06-18 11:17:42 +0000
@@ -0,0 +1,59 @@
+#!/bin/sh
+
+# This file contains all the common routines that are used across 
+# different node setup.
+
+# Stubborn apt-get, retrying on errors
+apt-get-retry () 
+{
+    local fatal="yes";
+    if [ "$1" == "--non-fatal" ]; then
+        fatal=""
+        shift
+    fi
+    local delay=1;
+    while [ $delay -lt 100 ]; do
+        if apt-get "$@"; then
+            return
+        fi
+        echo "apt-get failed, sleeping ${delay}s before retrying"
+        sleep $delay
+        delay=$((delay * 2))
+    done
+    if [ -n "$fatal" ]; then
+        echo "apt-get failed after several attempts, aborting"
+        exit 1
+    fi
+}
+
+#fix_bug_932088()
+#{
+    # Adding the s3 mirror to use instead to over come the 404, 403 errors reported in bug 932088
+    #sed -i.bk 's,^\(.*://[^.]*.ec2.archive.ubuntu.com\)/,\1.s3.amazonaws.com/,' /etc/apt/sources.list
+    #apt-get-retry --non-fatal update
+#}
+
+live_build_install() 
+{
+     wget "http://git.linaro.org/gitweb?p=ubuntu/ubuntu-build-service.git;a=blob_plain;f=packages/live-build_3.0~a45-1_all.deb" -O live-build_3.0~a45-1_all.deb
+     dpkg -i live-build_3.0~a45-1_all.deb
+}
+
+git_config() 
+{
+   git config --system --unset-all http.proxy || true
+   git config --system http.proxy http://ci.linaro.org:3128
+}
+
+
+
+fifo_cleanup()
+{
+    if test -e /tmp/fifo3 ; then
+        rm -f /tmp/fifo3 
+    fi
+    
+    if test -e /tmp/fifo2 ; then
+        rm -f /tmp/fifo2
+    fi 
+}

=== added file 'node/userdata-fifo'
--- node/userdata-fifo	1970-01-01 00:00:00 +0000
+++ node/userdata-fifo	2012-06-18 11:17:42 +0000
@@ -0,0 +1,21 @@
+#! /bin/sh
+
+# This script waits for the slave init script to finish execution
+# This script needs to be called from the UserData part of the slave init
+
+set -x
+
+if ! test -e /tmp/fifo2; then
+        mkfifo /tmp/fifo2 || true
+fi
+if ! test -e /tmp/fifo3; then
+        mkfifo /tmp/fifo3 || true
+fi
+
+echo "waiting for init script to start"
+cat /tmp/fifo2 > /dev/null
+echo "init script running"
+
+# wait for init script to signal "finish"
+cat /tmp/fifo3 > /dev/null
+echo "init script ended ... continuing with cloud-init"