Linaro CI

Merge lp:~milo/linaro-ci/bug1090416 into lp:linaro-ci

  1. bug1090416
  2. Merge into lci-build-tools
Proposed by Milo Casagrande
Status: Merged
Approved by: James Tunnicliffe
Approved revision: 123
Merged at revision: 128
Proposed branch: lp:~milo/linaro-ci/bug1090416
Merge into: lp:linaro-ci
Diff against target: 344 lines (+141/-122)
5 files modified
get_latest_ci_hwpack (+122/-111)
jenkins_common_lib (+1/-1)
jenkins_kernel_build_inst (+9/-4)
jenkins_lava_job_submissions (+6/-3)
jenkins_packaged_kernel_build_inst (+3/-3)
To merge this branch: bzr merge lp:~milo/linaro-ci/bug1090416
Reviewer Review Type Date Requested Status
James Tunnicliffe (community) Approve
Deepti B. Kalakeri (community) Needs Fixing
Review via email: mp+140215@code.launchpad.net

Commit message

Support sending LAVA jobs to multiple boards.

Description of the change

The branch adds support for sending multiple jobs to LAVA to different boards, from the same CI job.

It introduces a new env variable: board_types.
The old one can still be used in the CI jobs, but the new one will be exported, and it will be used looping through the defined boards.

Multiple boards have to be written comma separated.

To post a comment you must log in.
Revision history for this message
Deepti B. Kalakeri (deeptik) wrote :

283 +elif test -n "$boards" -a -n "$board_type"; then
284 + board_types="$board_types,$board_type"

Typo error: elseif needs to be "elif test -n "$board_types" -a -n "$board_type"; than the above
between, I don't know why would anyone want to define board_types and board_type together in a job, instead if someone needed to define multiple board_types then they could just use board_types param or if they needed single board testing then they can use only board_type. I feel they are compliment params.

review: Needs Fixing
Revision history for this message
Milo Casagrande (milo) wrote :

On Fri, Dec 21, 2012 at 12:34 PM, Deepti B. Kalakeri
<email address hidden> wrote:
> Review: Needs Fixing
>
> 283 +elif test -n "$boards" -a -n "$board_type"; then
> 284 + board_types="$board_types,$board_type"
>
> Typo error: elseif needs to be "elif test -n "$board_types" -a -n "$board_type"; than the above
> between, I don't know why would anyone want to define board_types and board_type together in a job, instead if someone needed to define multiple board_types then they could just use board_types param or if they needed single board testing then they can use only board_type. I feel they are compliment params.

They are, or well should be: just one of them should be defined in the
job. Actually, it has been done in that way, introducing
'board_types', not to drop support for all the jobs that are already
using 'board_type'. Or we could just use 'board_type' even for
multiple boards.
All the if-elif code is just for using 'board_types' in the script,
and looping other it, instead of exporting both of them and having to
test which one is defined or not later.

--
Milo Casagrande
Infrastructure Engineer
Linaro.org <www.linaro.org> │ Open source software for ARM SoCs

lp:~milo/linaro-ci/bug1090416 updated
122. By Milo Casagrande

Fixed wrong env variable name.

123. By Milo Casagrande

Merged from trunk.

Revision history for this message
James Tunnicliffe (dooferlad) wrote :

Looks good. I wish the diff viewer could show "the indentation changed on this bit" though!

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'get_latest_ci_hwpack'
--- get_latest_ci_hwpack 2012-08-27 08:23:44 +0000
+++ get_latest_ci_hwpack 2012-12-21 15:41:20 +0000
@@ -3,119 +3,130 @@
3Generate a job definition from a template using the latest hwpack and rootfs3Generate a job definition from a template using the latest hwpack and rootfs
4of specified types.4of specified types.
5"""5"""
6import os, sys6import os
7import sys
7import re8import re
8import json9import json
9from find_latest import find_latest_rootfs, find_ci_latest10from find_latest import find_latest_rootfs, find_ci_latest
1011
11hwpacktype = os.getenv("HWPACK_TYPE", "unknown")12
12imagetype = os.getenv("ROOTFS_TYPE", "unknown")13def main(board):
13job_name = os.getenv("KERNEL_JOB_NAME", "unknown") + '_' + \14 if not board:
14 os.getenv("HWPACK_BUILD_DATE", "unknown") + '_daily test'15 board = 'unknown'
1516
16testplan = os.getenv("LAVA_TEST_PLAN", "ltp, pwrmgmt")17 hwpacktype = os.getenv("HWPACK_TYPE", "unknown")
17osdevicetags = os.getenv("DEVICE_TAGS", "none")18 imagetype = os.getenv("ROOTFS_TYPE", "unknown")
1819 job_name = os.getenv("KERNEL_JOB_NAME", "unknown") + '_' + \
19template = {20 os.getenv("HWPACK_BUILD_DATE", "unknown") + '_daily test'
20 "timeout": 20000,21
21 "actions": [22 testplan = os.getenv("LAVA_TEST_PLAN", "ltp, pwrmgmt")
22 {23 osdevicetags = os.getenv("DEVICE_TAGS", "none")
23 "command": "deploy_linaro_image"24
24 },25 template = {
25 {26 "timeout": 20000,
26 "command": "boot_linaro_image"27 "actions": [
27 },28 {
28 {29 "command": "deploy_linaro_image"
29 "command": "submit_results",30 },
30 "parameters": {31 {
31 "stream": os.getenv("BUNDLE_STREAM_NAME", "/anonymous/plars/"),32 "command": "boot_linaro_image"
32 "server": "http://validation.linaro.org/lava-server/RPC2/"33 },
33 }34 {
35 "command": "submit_results",
36 "parameters": {
37 "stream": os.getenv("BUNDLE_STREAM_NAME", "/anonymous/plars/"),
38 "server": "http://validation.linaro.org/lava-server/RPC2/"
39 }
40 }
41 ],
42 "job_name": job_name,
43 "device_type": board
34 }44 }
35 ],45
36 "job_name": job_name,46 if osdevicetags != "none":
37 "device_type": os.getenv("BOARD_TYPE", "unknown")47 devicetags = osdevicetags.split(',')
38}48 template["device_tags"] = devicetags
3949
40if osdevicetags != "none":50 url = os.getenv("JOB_URL", "http://snapshots.linaro.org/kernel-hwpack/")
41 devicetags = osdevicetags.split(',')51 hwpack_parent_url = os.getenv("KERNEL_NAME", "unknown")
42 template["device_tags"] = devicetags52 hwpack_dir_url = os.getenv("KERNEL_JOB_NAME", "unknown")
4353 hwpack_url_path = os.path.join(url, hwpack_parent_url)
44url=os.getenv("JOB_URL", "http://snapshots.linaro.org/kernel-hwpack/")54 hwpack_url_path = os.path.join(hwpack_url_path, hwpack_dir_url)
45hwpack_parent_url=os.getenv("KERNEL_NAME","unknown")55
46hwpack_dir_url=os.getenv("KERNEL_JOB_NAME", "unknown")56 latest_hwpack = os.getenv('HWPACK_NAME', 'unknown')
47hwpack_url_path = os.path.join(url, hwpack_parent_url)57 hwpack_url = find_ci_latest(hwpack_url_path, latest_hwpack)
48hwpack_url_path = os.path.join(hwpack_url_path, hwpack_dir_url)58 if hwpack_url == None:
4959 sys.exit(1)
50latest_hwpack = os.getenv('HWPACK_NAME', 'unknown')60 rootfs_url = find_latest_rootfs(imagetype)
51hwpack_url = find_ci_latest(hwpack_url_path, latest_hwpack)61 rootfs_name, rootfsdate, rootfsbuild = os.path.basename(rootfs_url).\
52if hwpack_url == None:62 rsplit(".", 2)[0].rsplit("-", 2)
53 sys.exit(1)63
54rootfs_url = find_latest_rootfs(imagetype)64 deploy_params = {'hwpack': hwpack_url, 'rootfs': rootfs_url}
55rootfs_name, rootfsdate, rootfsbuild = os.path.basename(rootfs_url).\65 template['actions'][0]['parameters'] = deploy_params
56 rsplit(".", 2)[0].rsplit("-", 2)66
5767 metadata = {}
58deploy_params = { 'hwpack':hwpack_url, 'rootfs':rootfs_url }68 hwpack_filename = os.path.basename(hwpack_url)
59template['actions'][0]['parameters'] = deploy_params69 hwpackdate, hwpackbuild = hwpack_filename.rsplit('_', 4)[1].split('-')
6070 metadata['git_url'] = os.getenv("KERNEL_GIT", "git://git.linaro.org/kernel/unknown")
61metadata = {}71 metadata['kernel.gitweb_url'] = os.getenv("GIT_WEB_URL", "unknown")
62hwpack_filename = os.path.basename(hwpack_url)72 metadata['git_commitid'] = os.getenv("KERNEL_COMMIT", "unknown")
63hwpackdate, hwpackbuild = hwpack_filename.rsplit('_', 4)[1].split('-')73 metadata['kernel_name'] = os.getenv("KERNEL_NAME", "unknown")
64metadata['git_url'] = os.getenv("KERNEL_GIT", "git://git.linaro.org/kernel/unknown")74 metadata['kernel_version'] = os.getenv("KERNEL_VERSION", "unknown")
65metadata['kernel.gitweb_url'] = os.getenv("GIT_WEB_URL", "unknown")75 metadata['build.id'] = os.getenv("BUILD_ID", "unknown")
66metadata['git_commitid'] = os.getenv("KERNEL_COMMIT", "unknown")76 metadata['kernel.config'] = os.getenv('KERNEL_CONFIG', 'unknown')
67metadata['kernel_name'] = os.getenv("KERNEL_NAME", "unknown")77 metadata['kernel.build_url'] = os.getenv("KERNEL_BUILD_URL", "unknown")
68metadata['kernel_version'] = os.getenv("KERNEL_VERSION", "unknown")78 metadata['kernel.git_branch_name'] = os.getenv("KERNEL_BRANCH", "unknown")
69metadata['build.id'] = os.getenv("BUILD_ID", "unknown")79 metadata['kernel.git_log_info'] = os.getenv("GIT_LOG", "unknown")
70metadata['kernel.config'] = os.getenv('KERNEL_CONFIG', 'unknown')80 metadata['hwpack.type'] = hwpacktype
71metadata['kernel.build_url'] = os.getenv("KERNEL_BUILD_URL", "unknown")81 metadata['hwpack.date'] = hwpackdate
72metadata['kernel.git_branch_name'] = os.getenv("KERNEL_BRANCH", "unknown")82
73metadata['kernel.git_log_info'] = os.getenv("GIT_LOG", "unknown")83 metadata['rootfs.type'] = imagetype
74metadata['hwpack.type'] = hwpacktype84 metadata['rootfs.date'] = rootfsdate
75metadata['hwpack.date'] = hwpackdate85 metadata['rootfs.build'] = rootfsbuild
7686
77metadata['rootfs.type'] = imagetype87 template['actions'][0]['metadata'] = metadata
78metadata['rootfs.date'] = rootfsdate88
79metadata['rootfs.build'] = rootfsbuild89 testcases = testplan.split(',')
8090 testcases = [tc.strip() for tc in testplan.split(',')]
81template['actions'][0]['metadata'] = metadata91 regex = re.compile("(?P<testcase>^\w+)(\((?P<options>[^\)]+)\))?"\
8292 "(:(?P<timeout>\d+))?")
83testcases = testplan.split(',')93
84testcases = [tc.strip() for tc in testplan.split(',')]94 # Dict for lava_test_install
85regex = re.compile("(?P<testcase>^\w+)(\((?P<options>[^\)]+)\))?"\95 lava_test_install_command = {}
86 "(:(?P<timeout>\d+))?")96 lava_test_install_command["command"] = "lava_test_install"
8797 lava_test_install_command["parameters"] = {"tests": [], "timeout": 4000}
88# Dict for lava_test_install 98
89lava_test_install_command = {}99 # The positions in the template dict to insert test install/run commands
90lava_test_install_command["command"] = "lava_test_install"100 install_cmd_pos = 1
91lava_test_install_command["parameters"] = { "tests" : [], "timeout" : 4000 }101 run_cmd_pos = 2
92102
93# The positions in the template dict to insert test install/run commands103 for testcase in testcases:
94install_cmd_pos = 1104 r = regex.search(testcase)
95run_cmd_pos = 2105 if r.group("testcase") != None:
96106 lava_test_install_command["parameters"]["tests"].\
97for testcase in testcases:107 append(r.group("testcase"))
98 r = regex.search(testcase)108
99 if r.group("testcase") != None:109 # append lava_test_run commands
100 lava_test_install_command["parameters"]["tests"].\110 timeout = 4800
101 append(r.group("testcase"))111 if r.group('timeout'):
102112 timeout = r.group('timeout')
103 # append lava_test_run commands113
104 timeout = 4800114 lava_test_run_command = {}
105 if r.group('timeout'):115 lava_test_run_command["command"] = "lava_test_run"
106 timeout = r.group('timeout')116 lava_test_run_command["parameters"] = {"test_name": r.group('testcase'),
107117 "timeout": timeout}
108 lava_test_run_command = {}118
109 lava_test_run_command["command"] = "lava_test_run"119 if r.group("options"):
110 lava_test_run_command["parameters"] = {"test_name" : r.group('testcase'),120 lava_test_run_command["parameters"]["test_options"] = r.group("options")
111 "timeout" : timeout }121
112122 template['actions'].insert(run_cmd_pos, lava_test_run_command)
113 if r.group("options"):123 run_cmd_pos = run_cmd_pos + 1
114 lava_test_run_command["parameters"]["test_options"] = r.group("options")124
115125 # append the lava test install command
116 template['actions'].insert(run_cmd_pos, lava_test_run_command)126 template['actions'].insert(install_cmd_pos, lava_test_install_command)
117 run_cmd_pos = run_cmd_pos + 1127 print json.dumps(template, indent=2)
118128
119# append the lava test install command129
120template['actions'].insert(install_cmd_pos, lava_test_install_command)130if __name__ == '__main__':
121print json.dumps(template, indent=2)131 # We need only one argument, the board type.
132 main(sys.argv[1])
122133
=== modified file 'jenkins_common_lib'
--- jenkins_common_lib 2012-12-21 15:18:27 +0000
+++ jenkins_common_lib 2012-12-21 15:41:20 +0000
@@ -15,7 +15,7 @@
15echo BUILD_ID=$BUILD_NUMBER >> hwpack_info15echo BUILD_ID=$BUILD_NUMBER >> hwpack_info
16echo GCC_VERSION=`dpkg -s gcc | grep Version | cut -d ' ' -f2` >> hwpack_info16echo GCC_VERSION=`dpkg -s gcc | grep Version | cut -d ' ' -f2` >> hwpack_info
17echo GCC_CROSS_COMPILER=$gcc_cc_version >> hwpack_info17echo GCC_CROSS_COMPILER=$gcc_cc_version >> hwpack_info
18echo BOARD_TYPE=$board_type >> hwpack_info18echo BOARD_TYPES=$board_types >> hwpack_info
19echo HWPACK_TYPE=$hwpack_type >> hwpack_info19echo HWPACK_TYPE=$hwpack_type >> hwpack_info
20echo ROOTFS_TYPE=$rootfs_type >> hwpack_info20echo ROOTFS_TYPE=$rootfs_type >> hwpack_info
21echo BUNDLE_STREAM_NAME=$bundle_stream_name_path >> hwpack_info21echo BUNDLE_STREAM_NAME=$bundle_stream_name_path >> hwpack_info
2222
=== modified file 'jenkins_kernel_build_inst'
--- jenkins_kernel_build_inst 2012-12-21 15:15:51 +0000
+++ jenkins_kernel_build_inst 2012-12-21 15:41:20 +0000
@@ -1,7 +1,7 @@
1#! /bin/bash1#! /bin/bash
2# Example env variables and its values2# Example env variables and its values
3# kernel_config='omap2plus_defconfig'3# kernel_config='omap2plus_defconfig'
4# export board_type='panda'4# export board_types='panda'
5# kernel_flavour='omap4'5# kernel_flavour='omap4'
6# rootfs_type='nano'6# rootfs_type='nano'
7#7#
@@ -32,8 +32,13 @@
32if test -z "$hwpack_type"; then32if test -z "$hwpack_type"; then
33 hwpack_type='panda'33 hwpack_type='panda'
34fi34fi
35if test -z "$board_type"; then35# Use the new board_types variable to define multiple boards for LAVA testing.
36 board_type='panda'36if test -z "$board_types" -a -z "$board_type"; then
37 board_types="panda"
38elif test -z "$board_types" -a -n "$board_type"; then
39 board_types="$board_type"
40elif test -n "$board_types" -a -n "$board_type"; then
41 board_types="$board_types,$board_type"
37fi42fi
38if test -z "$kernel_flavour"; then43if test -z "$kernel_flavour"; then
39 kernel_flavour='omap'44 kernel_flavour='omap'
@@ -78,7 +83,7 @@
78export conf_external83export conf_external
79export kernel_config84export kernel_config
80export hwpack_type85export hwpack_type
81export board_type86export board_types
82export kernel_flavour87export kernel_flavour
83export rootfs_type88export rootfs_type
84export git_web_url89export git_web_url
8590
=== modified file 'jenkins_lava_job_submissions'
--- jenkins_lava_job_submissions 2012-05-28 03:38:56 +0000
+++ jenkins_lava_job_submissions 2012-12-21 15:41:20 +0000
@@ -15,7 +15,10 @@
15python $WORKSPACE/lci-build-tools/jenkins_kernel_ci_bundle > $build_json_fname15python $WORKSPACE/lci-build-tools/jenkins_kernel_ci_bundle > $build_json_fname
16lava-dashboard-tool put --dashboard-url "$LAVA_ENDPOINT" $build_json_fname "$BUILD_STREAM_NAME"16lava-dashboard-tool put --dashboard-url "$LAVA_ENDPOINT" $build_json_fname "$BUILD_STREAM_NAME"
17if test $SUBMIT_LAVA_JOB -eq 1; then17if test $SUBMIT_LAVA_JOB -eq 1; then
18 python $WORKSPACE/lci-build-tools/get_latest_ci_hwpack > $kernel_test_json_fname18 for board in `printf "$BOARD_TYPES" | tr ',' ' '`
19 lava_job_id=`lava-tool submit-job "$LAVA_ENDPOINT" $kernel_test_json_fname | grep -v EXPERIMENTAL | cut -d ":" -f 2 | tr -d ' '`19 do
20 echo "LAVA Job Id: $lava_job_id, URL: http://$LAVA_SERVER/lava-server/scheduler/job/$lava_job_id"20 python $WORKSPACE/lci-build-tools/get_latest_ci_hwpack "$board" > $kernel_test_json_fname
21 lava_job_id=`lava-tool submit-job "$LAVA_ENDPOINT" $kernel_test_json_fname | grep -v EXPERIMENTAL | cut -d ":" -f 2 | tr -d ' '`
22 echo "LAVA Job Id: $lava_job_id, URL: http://$LAVA_SERVER/lava-server/scheduler/job/$lava_job_id"
23 done
21fi24fi
2225
=== modified file 'jenkins_packaged_kernel_build_inst'
--- jenkins_packaged_kernel_build_inst 2012-03-23 11:27:18 +0000
+++ jenkins_packaged_kernel_build_inst 2012-12-21 15:41:20 +0000
@@ -1,6 +1,6 @@
1#! /bin/sh1#! /bin/sh
2# Example env variables and its values2# Example env variables and its values
3# export board_type='panda'3# export board_types='panda'
4# rootfs_type='nano'4# rootfs_type='nano'
5set -xe5set -xe
66
@@ -9,7 +9,7 @@
9#9#
10: ${kernel_config:='packaged_kernel_config'}10: ${kernel_config:='packaged_kernel_config'}
11: ${hwpack_type:='panda'}11: ${hwpack_type:='panda'}
12: ${board_type:='panda'}12: ${board_types:='panda'}
13: ${rootfs_type:='nano'}13: ${rootfs_type:='nano'}
14: ${submit_job:=1}14: ${submit_job:=1}
15: ${upstream_git_tree:='git://git.linaro.org/kernel/linux-linaro-3.0.git'}15: ${upstream_git_tree:='git://git.linaro.org/kernel/linux-linaro-3.0.git'}
@@ -337,7 +337,7 @@
337 BUILD_ID=$BUILD_NUMBER337 BUILD_ID=$BUILD_NUMBER
338 GCC_VERSION=$(dpkg -s gcc | grep Version | cut -d ' ' -f2)338 GCC_VERSION=$(dpkg -s gcc | grep Version | cut -d ' ' -f2)
339 GCC_CROSS_COMPILER=$(dpkg -s gcc-arm-linux-gnueabi | grep Version | cut -d ' ' -f2)339 GCC_CROSS_COMPILER=$(dpkg -s gcc-arm-linux-gnueabi | grep Version | cut -d ' ' -f2)
340 BOARD_TYPE=$board_type340 BOARD_TYPES=$board_types
341 HWPACK_TYPE=$hwpack_type341 HWPACK_TYPE=$hwpack_type
342 ROOTFS_TYPE=$rootfs_type342 ROOTFS_TYPE=$rootfs_type
343 BUNDLE_STREAM_NAME=$bundle_stream_name_path343 BUNDLE_STREAM_NAME=$bundle_stream_name_path

Subscribers

People subscribed via source and target branches