Merge lp:~linaro-automation/linaro-ci/one-time-submission-job into lp:linaro-ci

[Deepti B. Kalakeri]

Adding script for one time job submission.

[Alexander Sack (asac)]

1. + if test -z "$git_branch_ref" ; then
 => I think this and all other sanity checks of input parameters should be moved to the top.

2. since we have quite a few variables and input parameters here itmight make sense to use special syntax for input parameters (like all uppercase, or param_... prefix etc.)

3. + git remote rm "$remote_tree_alias_name"
 => I think this cleanup should be moved to a block that gets always executed (even in case of error). there is a signal syntax to catch errors and jump to a cleanup code block in sh we could use.

4. i am not sure if the cleanup code should be going back to master branch; I dont think it is always the right thing; there can be trees that have no master branch etc. iirc jenkins creates its own branch names. check what branch you are on if you enter the script and switch back to that maybe.

5. I feel we duplicate loads of code here that could be reused from the "normal" kenrel CI job; either refactor things now, or add a blueprint about code-reuse-effort-for-jenkins-scripts.

review: Needs Resubmitting

[Alexander Sack (asac)]

>
> 3. + git remote rm "$remote_tree_alias_name"
> => I think this cleanup should be moved to a block that gets always executed
> (even in case of error). there is a signal syntax to catch errors and jump to
> a cleanup code block in sh we could use.

here background on this: http://steve-parker.org/sh/trap.shtml

we wantt he cleanup code to be run in case of error and if things go smooth.

[Alexander Sack (asac)]

haven't checked one by one, but all the parameters provided by the kernel job should be submitted to the dashboard too.

[Deepti B. Kalakeri (deeptik)]

>
> 1. + if test -z "$git_branch_ref" ; then
> => I think this and all other sanity checks of input parameters should be
> moved to the top.
>
> 2. since we have quite a few variables and input parameters here itmight make
> sense to use special syntax for input parameters (like all uppercase, or
> param_... prefix etc.)
>
> 3. + git remote rm "$remote_tree_alias_name"
> => I think this cleanup should be moved to a block that gets always executed
> (even in case of error). there is a signal syntax to catch errors and jump to
> a cleanup code block in sh we could use.
>

Added a cleanup function using the trap command.
> 4. i am not sure if the cleanup code should be going back to master branch; I
> dont think it is always the right thing; there can be trees that have no
> master branch etc. iirc jenkins creates its own branch names. check what
> branch you are on if you enter the script and switch back to that maybe.
>
DONE
> 5. I feel we duplicate loads of code here that could be reused from the
> "normal" kenrel CI job; either refactor things now, or add a blueprint about
> code-reuse-effort-for-jenkins-scripts.

I would prefer a BP on this and would like this to be done in 2011.11.

[lp:~linaro-automation/linaro-ci/one-time-submission-job]

40. By Deepti B. Kalakeri on 2011-10-25

Addressing review comments on script for one time job submission

=== added file 'jenkins_maintainers_kernel_build_inst'
--- jenkins_maintainers_kernel_build_inst	1970-01-01 00:00:00 +0000
+++ jenkins_maintainers_kernel_build_inst	2011-10-25 13:56:23 +0000
@@ -0,0 +1,118 @@
+#! /bin/sh
+# Example env variables and its values
+# kernel_config='omap2plus_defconfig'
+# export board_type='panda'
+# kernel_flavour='omap4'
+# rootfs_type='nano'
+set -xe
+
+trap cleanup EXIT
+
+cleanup()
+{
+
+if test "$matching_remote_trees" = "$remote_tree_alias_name" ; then
+   # Cleaning up remote tree that was added
+   git remote rm "$remote_tree_alias_name"
+fi
+if test "$matching_remote_branch" = "$remote_branch_name" ; then
+   # Cleaning up remote branch that was added
+   git checkout $GIT_BRANCH
+   git branch -D $remote_branch_name
+fi
+}
+
+if test -z "$GIT_REMOTE_URL" ; then
+   echo "Please specify the GIT_REMOTE_URL"
+   exit 1
+fi
+if test -z "$GIT_REMOTE_REF" ; then
+   echo "Please specify the GIT_REMOTE_REF"
+   exit 1
+fi
+if test -z "$rootfs_type"; then
+  rootfs_type='nano'
+fi
+
+remote_tree_name=`basename $GIT_REMOTE_URL  | awk -F '.git' '{print $1}'`
+remote_tree_alias_name="$remote_tree_name-alias" 
+remote_branch_name="$remote_tree_name-branch"
+git remote add "$remote_tree_alias_name" $GIT_REMOTE_URL
+git fetch $remote_tree_alias_name
+matching_remote_trees=`git remote -v | grep "$remote_tree_alias_name" | awk '{ print $1}' | uniq`
+git checkout -b $remote_branch_name $GIT_REMOTE_REF
+matching_remote_branch=`git branch | grep "$remote_branch_name" | awk -F ' ' '{ print $2}'`
+
+bundle_stream_name=`echo $JOB_NAME | sed -e 's/_.*//' -e 's/\./_/g'`
+kernel_config_name=`echo ${kernel_config} | sed -e 's/_defconfig//g'`
+kernel_version=`git describe --match='v*' | sed -e 's/^v//'``echo -${kernel_config_name}-linaro-${kernel_flavour} | sed -e 's/_/-/g'`
+
+# Below is the list of values that will be used in the json files
+log_info=$BUILD_URL"consoleText"
+bundle_stream_name_path="/anonymous/ci-$bundle_stream_name/"
+echo > hwpack_info # start with empty
+echo KERNEL_CONFIG=$kernel_config >> hwpack_info
+echo OS_INFO=`cat /etc/lsb-release | grep DISTRIB_DESCRIPTION | cut -d '=' -f 2` >> hwpack_info
+echo LOG=$log_info >> hwpack_info
+echo BUILD_ID=$BUILD_NUMBER >> hwpack_info
+echo GCC_VERSION=`dpkg -s gcc | grep Version | cut -d ' ' -f2` >> hwpack_info
+echo GCC_CROSS_COMPILER=`dpkg -s gcc-arm-linux-gnueabi | grep Version | cut -d ' ' -f2` >> hwpack_info
+echo BOARD_TYPE=$board_type >> hwpack_info
+echo HWPACK_TYPE=$hwpack_type >> hwpack_info
+echo ROOTFS_TYPE=$rootfs_type >> hwpack_info
+echo BUNDLE_STREAM_NAME=$bundle_stream_name_path >> hwpack_info
+echo JOB_URL=http://ci.linaro.org/kernel_hwpack/ >> hwpack_info
+echo KERNEL_VERSION=`git describe --match='v*'` >> hwpack_info
+echo KERNEL_GIT=$git_remote_url  >> hwpack_info
+echo KERNEL_COMMIT=`git rev-parse HEAD`  >> hwpack_info
+echo COMMIT_TIME=`git show -s --format=%cD HEAD` >> hwpack_info
+echo KERNEL_NAME=$bundle_stream_name >> hwpack_info
+echo SUBMIT_LAVA_JOB=$submit_job >> hwpack_info
+echo KERNEL_BUILD_URL=$BUILD_URL >> hwpack_info
+echo KERNEL_BRANCH=`git branch | grep -e '*'  | cut -d '*' -f 2` >> hwpack_info
+echo KERNEL_JOB_NAME=$JOB_NAME >> hwpack_info
+echo GIT_LOG=`git log -n1 --pretty=oneline` >> hwpack_info
+
+rm -f *.deb
+cp scripts/package/builddeb scripts/package/builddeb.orig
+cp builddeb scripts/package/builddeb
+cpu_count=`cat /proc/cpuinfo  | grep processor | wc -l`
+if test x$cpu_count = x; then
+  cpu_count=1
+fi
+j_count=`expr $cpu_count '*' 2`
+
+pkg_dir=$kernel_version"_output_dir"
+rm -rf $pkg_dir
+mkdir $pkg_dir
+
+echo "DEBUG: The remote branch: $remote_branch_name will be built now for the commit id $KERNEL_VERSION"
+START=$(date +%s)
+make ARCH=arm clean mrproper
+make ARCH=arm O=$pkg_dir KERNELVERSION="$kernel_version" KERNELRELEASE="$kernel_version" CROSS_COMPILE=arm-linux-gnueabi- $kernel_config
+yes "" | make ARCH=arm O=$pkg_dir KERNELVERSION="$kernel_version" KERNELRELEASE="$kernel_version" CROSS_COMPILE=arm-linux-gnueabi- oldconfig
+
+# Enabling the THUMB2 kernel option as the THUMB kernel does not seem to work
+# well with THUMB2 userspace. This fixes the reboot issue(bug: 860556) we have seen
+# for THUMB Kernel working with THUMB2 userspace.
+
+echo "CONFIG_ARCH_OMAP2=n" >> $WORKSPACE/$pkg_dir/.config
+echo "CONFIG_THUMB2_KERNEL=y" >> $WORKSPACE/$pkg_dir/.config
+yes "" | make ARCH=arm O=$pkg_dir KERNELVERSION="$kernel_version" KERNELRELEASE="$kernel_version" CROSS_COMPILE=arm-linux-gnueabi- oldconfig	
+make ARCH=arm O=$pkg_dir KERNELVERSION="$kernel_version" KERNELRELEASE="$kernel_version" CROSS_COMPILE=arm-linux-gnueabi- -j$j_count uImage
+make ARCH=arm O=$pkg_dir KERNELVERSION="$kernel_version" KERNELRELEASE="$kernel_version" CROSS_COMPILE=arm-linux-gnueabi- -j$j_count modules
+make ARCH=arm O=$pkg_dir KERNELVERSION="$kernel_version" KERNELRELEASE="$kernel_version" CROSS_COMPILE=arm-linux-gnueabi- KBUILD_DEBARCH=armel V=1 deb-pkg
+END=$(date +%s)
+EXECUTION_TIME_IN_SEC=$(( $END - $START ))
+
+latest_kernel=`python lci-build-tools/get_latest_slo_hwpack`
+latest_hwpack_name=`basename $latest_kernel`
+wget -c $latest_kernel
+rm -rf linaro-image-tools
+bzr branch lp:linaro-image-tools
+new_hwpack_name=`python linaro-image-tools/linaro-hwpack-replace -t $latest_hwpack_name -p ./linux-image*${kernel_version}*.deb -r linux-image -n $BUILD_NUMBER`
+use_hwpack_name=`basename $new_hwpack_name`
+echo HWPACK_NAME=$use_hwpack_name >> hwpack_info
+echo URL=$JENKINS_HOME/jobs/$JOB_NAME/workspace/$use_hwpack_name >> hwpack_info
+echo EXECUTION_TIME_IN_SEC=$(( $END - $START )) >> hwpack_info
+echo HWPACK_BUILD_DATE=`echo $use_hwpack_name | cut -d '_' -f3` >>hwpack_info