Merge lp:~linaro-infrastructure/cbuild/cbuild-scheduler_cbuild-lava into lp:~linaro-toolchain-dev/cbuild/scheduler

[Stevan Radaković]

This branch include changes in the CBuild scheduler configuration.
Not all of these are viable for merge yet, waiting for infra team review.

[Milo Casagrande (milo)]

Only comment I have is for lines like these:

64 + - apt-get install -y wget
65 + - if grep -q Beagle /proc/cpuinfo; then
66 + - " # if it's supposed QEMU, download from local server"
67 + - url="http://10.0.2.2/images/tmp/gcc-linaro-4.6-2012.09-results.tar.gz"
68 + - else
69 + - url="http://people.linaro.org/~pfalcon/cbuild-lava/gcc-linaro-4.6-2012.09-results.tar.gz"
70 + - fi

I see that there is a mix of syntax: some steps are quoted, some are not. It definitely will work on both cases, it is more from a consistency point of view. Was puzzled actually by how the yaml parser will handle that, but it handles them correctly anyway.

The other thing is including if-else bash structures in the yaml file: can we have a separate bash script and run that one? Or do we need to do other things before it is possible to achieve that: like sending the file to lava, etc. I know it's a couple of lines and is just the template... (oddily we might even move all the steps in a bash script and just execute that one).

[Stevan Radaković (stevanr)]

> Only comment I have is for lines like these:
>
> 64 + - apt-get install -y wget
> 65 + - if grep -q Beagle /proc/cpuinfo; then
> 66 + - " # if it's supposed QEMU, download from local server"
> 67 + - url="http://10.0.2.2/images/tmp/gcc-
> linaro-4.6-2012.09-results.tar.gz"
> 68 + - else
> 69 + - url="http://people.linaro.org/~pfalcon/cbuild-lava/gcc-
> linaro-4.6-2012.09-results.tar.gz"
> 70 + - fi
>
> I see that there is a mix of syntax: some steps are quoted, some are not. It
> definitely will work on both cases, it is more from a consistency point of
> view. Was puzzled actually by how the yaml parser will handle that, but it
> handles them correctly anyway.

Yes, quoted and unquoted lines are parsed in the same way it seems. It may be logical to remove the quotes from that one line we have for consistency reasons.

>
> The other thing is including if-else bash structures in the yaml file: can we
> have a separate bash script and run that one? Or do we need to do other things
> before it is possible to achieve that: like sending the file to lava, etc. I
> know it's a couple of lines and is just the template... (oddily we might even
> move all the steps in a bash script and just execute that one).

I'm not sure if this can be done, but anyway I fail to see the difference between the current approach and the one you described. Only if we can extract some piece of shell code that would be used in multiple templates scripts would this make sense. But if it makes sense to try and play around with this approach we can certainly do so.

[Milo Casagrande (milo)]

On Wed, Jan 9, 2013 at 4:41 PM, Stevan Radaković
<email address hidden> wrote:
>> The other thing is including if-else bash structures in the yaml file: can we
>> have a separate bash script and run that one? Or do we need to do other things
>> before it is possible to achieve that: like sending the file to lava, etc. I
>> know it's a couple of lines and is just the template... (oddily we might even
>> move all the steps in a bash script and just execute that one).
>
> I'm not sure if this can be done, but anyway I fail to see the difference between the current approach and the one you described. Only if we can extract some piece of shell code that would be used in multiple templates scripts would this make sense. But if it makes sense to try and play around with this approach we can certainly do so.

Basically, it changes nothing, apart from the re-use point of view.
The only difference I could see is that it looks clumsy to write
multiple lines of bash shell in a yaml-like syntax (that gets
converted into a shell script anyway after that).

--
Milo Casagrande | Infrastructure Team
Linaro.org <www.linaro.org> │ Open source software for ARM SoCs

[Paul Sokolovsky (pfalcon)]

> I see that there is a mix of syntax: some steps are quoted, some are not. It definitely will work on both cases, it is more from a consistency point of view. Was puzzled actually by how the yaml parser will handle that, but it handles them correctly anyway.

In YAML, string quotes can be omitted, if it can be parsed unambiguously. Yes, to be on safe side (and be consistent) we can quote everything. But that's just character noise which affects readability. So, I'd personally not quoted unless really required (that would be less consistently-looking, but see better than arrays of stray quotes imho). Current content may not adhere to the above is it was cut&pasted originally. If you agree with that idea (quote only if needed), let's fix it.

[Paul Sokolovsky (pfalcon)]

> The other thing is including if-else bash structures in the yaml file: can we have a separate bash script and run that one?

As oftentimes happens, a viable solution, IMHO, lies somewhere between these these two extremes: 1) put everything in YAML, and 2) host everything in separate repo(s) and pull it from YAML. If we host somewhere elsewhere, there're number of questions to consider: 1) where to host it; 2) how easy it will be to find/change it; 3) how easy to pull it where needed (for example, we could put each small script into own repo, but fetch dozen of repos is boring when doing local test). When there were good answers to those questions, I put it outside, for example "./workaround-multiarch.sh" which is standalone, reusable script. But setup/wrapper code is best in YAML, imho.

Btw, that if you quoted can be removed now - it remnant from previous work (I originally wanted to make a common YAML testdef for all builds, but for flexibility, now they're per queue, and something live in "panda" queue is not supposed to run qemu).

[Paul Sokolovsky (pfalcon)]

Another semantical (non-cleanup) thing: this waits for Stevan's feedback that build log layout structure as made by lava-panda-mock/lava-test-shell-template.yaml is ok to deal with in cronjob fetcher. If so, it should be propagated into lava-panda/*. In either case, those 2 should be consistent regarding log attaching steps.

[Milo Casagrande (milo)]

On Thu, Jan 10, 2013 at 2:52 PM, Paul Sokolovsky
<email address hidden> wrote:
> If you agree with that idea (quote only if needed), let's fix it.

For me it's OK.

--
Milo Casagrande | Infrastructure Team
Linaro.org <www.linaro.org> │ Open source software for ARM SoCs

[lp:~linaro-infrastructure/cbuild/cbuild-scheduler_cbuild-lava]

15. By Paul Sokolovsky on 2013-01-11

Update to use cleaned up build scripts branch.

16. By Paul Sokolovsky on 2013-01-11

Switch back to known good image, use symlinks for switching.

17. By Paul Sokolovsky on 2013-01-11

Call workaround-multiarch.sh to simulate production build better.

18. By Paul Sokolovsky on 2013-01-11

Try empty line in YAML.

19. By Paul Sokolovsky on 2013-01-11

Update lava build dir and further cleanup.

20. By Paul Sokolovsky on 2013-01-11

Cannot have empty list elements in YAML shell script after all.

21. By Paul Sokolovsky on 2013-01-11

Try TCWG image on panda-es.

22. By Paul Sokolovsky on 2013-01-11

LAVA lab doesn't have panda-es with usb-flash-drive currently, so try w/o.

23. By Paul Sokolovsky on 2013-01-13

Record logs for real build.

24. By Paul Sokolovsky on 2013-01-14

Add set -x and explicit start marker.

25. By Milo Casagrande on 2013-01-15

Create tarball and send it as an attachment.

26. By Paul Sokolovsky on 2013-01-16

Bump timeout to 15h, as it seems to trigger in testsuite.

27. By Paul Sokolovsky on 2013-01-17

Re-enable "usb-flash-drive" device tag.

28. By Paul Sokolovsky on 2013-01-17

Use single log tarball, propagate from lava-panda-mock.

29. By Paul Sokolovsky on 2013-01-17

Create separate queue for PandaES with Usb Drive.

There's currently one such machine, and if it hangs, the testing blocked.

30. By Paul Sokolovsky on 2013-01-17

Add hosts.txt for lava-pandaes-usbdrive.

31. By Paul Sokolovsky on 2013-01-17

Don't tie LAVA testcase names to GCC.

32. By Paul Sokolovsky on 2013-01-17

Create toplevel.txt the same way as native CBuild does.

33. By Paul Sokolovsky on 2013-01-17

We also need to pass job content down to builder, as it contains extra config.

34. By Paul Sokolovsky on 2013-01-18

Debug what's inside .job file produced on build slave.

35. By Paul Sokolovsky on 2013-01-18

Use dynamic lava_test_shell timeout as calculated by webapp.

36. By Paul Sokolovsky on 2013-01-18

Cut down global job timeout too.

TODO: Figure out what it actually means.

37. By Paul Sokolovsky on 2013-01-18

Actually enable USB drive in lava-pandaes-usbdrive.

38. By Paul Sokolovsky on 2013-01-18

dash strikes again, try to use /bin/echo

39. By Paul Sokolovsky on 2013-01-18

Settle on using /bin/echo which support -e switch.

40. By Paul Sokolovsky on 2013-01-18

Record actual build status in lava.

41. By Paul Sokolovsky on 2013-01-18

Add lava-panda-usbdrive config while lava-pandaes-usbdrive is down.

42. By Paul Sokolovsky on 2013-01-18

Use standard rootfs/hwpack for lava-panda-usbdrive

43. By Paul Sokolovsky on 2013-01-18

workaround-multiarch.sh should run after host gcc and libs were installed.

44. By Paul Sokolovsky on 2013-01-18

Propagate cbuild-global build result handling.

45. By Paul Sokolovsky on 2013-01-21

Don't create fake configure/build/install test cases for now.

46. By Paul Sokolovsky on 2013-01-21

Add extra output to debug result outpit dir for LAVA builds.

47. By Paul Sokolovsky on 2013-01-21

File was renamed. Also, dump hostname early.

48. By Paul Sokolovsky on 2013-01-21

Rsync is a must.

But how did it work before?

Unmerged revisions

=== added directory 'queue/lava-panda'
=== added directory 'queue/lava-panda-mock'
=== added file 'queue/lava-panda-mock/hosts.txt'
--- queue/lava-panda-mock/hosts.txt	1970-01-01 00:00:00 +0000
+++ queue/lava-panda-mock/hosts.txt	2013-01-15 12:59:22 +0000
@@ -0,0 +1,1 @@
+lava

=== added file 'queue/lava-panda-mock/lava-job-template.json'
--- queue/lava-panda-mock/lava-job-template.json	1970-01-01 00:00:00 +0000
+++ queue/lava-panda-mock/lava-job-template.json	2013-01-15 12:59:22 +0000
@@ -0,0 +1,29 @@
+{
+  "timeout": 1800,
+  "job_name": "%(job)s",
+  "device_type": "panda",
+  "device_tags": ["usb-flash-drive"],
+  "actions": [
+    {
+        "command": "deploy_linaro_image",
+        "parameters": {
+            "rootfs": "http://releases.linaro.org/12.02/ubuntu/oneiric-images/nano/linaro-o-nano-tar-20120221-0.tar.gz",
+            "hwpack": "http://releases.linaro.org/12.02/ubuntu/oneiric-images/nano/hwpack_linaro-lt-panda_20120221-1_armel_supported.tar.gz"
+        }
+    },
+    {
+        "command": "lava_test_shell",
+        "parameters": {
+            "testdef_urls": ["http://%(SERVER_NAME)s:%(SERVER_PORT)s/helpers/scheduler/lava/testdef/%(queue)s/%(job)s"],
+            "timeout": 36000
+        }
+    },
+    {
+         "command": "submit_results",
+         "parameters": {
+             "stream": "/anonymous/cbuild/",
+             "server": "https://validation.linaro.org/lava-server/RPC2/"
+         }
+    }
+  ]
+}

=== added file 'queue/lava-panda-mock/lava-test-shell-template.yaml'
--- queue/lava-panda-mock/lava-test-shell-template.yaml	1970-01-01 00:00:00 +0000
+++ queue/lava-panda-mock/lava-test-shell-template.yaml	2013-01-15 12:59:22 +0000
@@ -0,0 +1,45 @@
+metadata:
+    format: "Lava-Test Test Definition 1.0"
+    name: "%(job)s"
+
+install:
+    deps:
+        - bzr
+    git-repos:
+        - git://git.linaro.org/people/davelong/lt_ti_lava.git
+    steps:
+        - echo In Install
+        - lt_ti_lava/utils/lava-usb-drive.sh /dev/sda
+        - mkdir -p cbuild-lava
+        - mount /dev/sda1 cbuild-lava
+        - df
+        - bzr branch --use-existing-dir lp:~linaro-infrastructure/cbuild/cbuild-lava2 cbuild-lava
+        - cd cbuild-lava/slaves/lava
+        #-  Can't run workaround-multiarch.sh because we don't have gcc installed
+        #- ./workaround-multiarch.sh
+
+        #- Mock build core, download tarball of previously done build
+        #- and expand it to expected place.
+        - apt-get install -y wget
+        - if grep -q Beagle /proc/cpuinfo; then
+        - "    # if it's supposed QEMU, download from local server"
+        -     url="http://10.0.2.2/images/tmp/gcc-linaro-4.6-2012.09-results.tar.gz"
+        - else
+        -     url="http://people.linaro.org/~pfalcon/cbuild-lava/gcc-linaro-4.6-2012.09-results.tar.gz"
+        - fi
+        - wget -q -O- $url | gzip -d -c | tar -x
+
+run:
+    steps:
+        - set -x
+        - echo In run
+        - cd cbuild-lava/slaves/lava
+        #- make -f ../../lib/build.mk final/gcc-linaro-4.6-2012.09.stamp
+        - echo CBuild make finished, attaching logs to LAVA results
+        - lava-test-case gcc-configure --result pass
+        - lava-test-case gcc-build --result pass
+        - lava-test-case gcc-install --result pass
+        - lava-test-case cbuild-global --result pass
+        - tar -cf cbuild-global.tar.gz -z --atime-preserve -C results/build/*/logs/* .
+        - lava-test-case-attach cbuild-global cbuild-global.tar.gz
+        - echo Attaching logs done

=== added file 'queue/lava-panda/hosts.txt'
--- queue/lava-panda/hosts.txt	1970-01-01 00:00:00 +0000
+++ queue/lava-panda/hosts.txt	2013-01-15 12:59:22 +0000
@@ -0,0 +1,1 @@
+lava

=== added symlink 'queue/lava-panda/lava-job-template.json'
=== target is u'lava-job-template_tcwg-image.json'
=== added file 'queue/lava-panda/lava-job-template_hwpack.json'
--- queue/lava-panda/lava-job-template_hwpack.json	1970-01-01 00:00:00 +0000
+++ queue/lava-panda/lava-job-template_hwpack.json	2013-01-15 12:59:22 +0000
@@ -0,0 +1,29 @@
+{
+  "timeout": 1800,
+  "job_name": "%(job)s",
+  "device_type": "panda",
+  "device_tags": ["usb-flash-drive"],
+  "actions": [
+    {
+        "command": "deploy_linaro_image",
+        "parameters": {
+            "rootfs": "http://releases.linaro.org/12.02/ubuntu/oneiric-images/nano/linaro-o-nano-tar-20120221-0.tar.gz",
+            "hwpack": "http://releases.linaro.org/12.02/ubuntu/oneiric-images/nano/hwpack_linaro-lt-panda_20120221-1_armel_supported.tar.gz"
+        }
+    },
+    {
+        "command": "lava_test_shell",
+        "parameters": {
+            "testdef_urls": ["http://%(SERVER_NAME)s:%(SERVER_PORT)s/helpers/scheduler/lava/testdef/%(queue)s/%(job)s"],
+            "timeout": 36000
+        }
+    },
+    {
+         "command": "submit_results",
+         "parameters": {
+             "stream": "/anonymous/cbuild/",
+             "server": "https://validation.linaro.org/lava-server/RPC2/"
+         }
+    }
+  ]
+}

=== added file 'queue/lava-panda/lava-job-template_tcwg-image.json'
--- queue/lava-panda/lava-job-template_tcwg-image.json	1970-01-01 00:00:00 +0000
+++ queue/lava-panda/lava-job-template_tcwg-image.json	2013-01-15 12:59:22 +0000
@@ -0,0 +1,28 @@
+{
+  "timeout": 1800,
+  "job_name": "%(job)s",
+  "device_type": "panda-es",
+  "device_tags": [],
+  "actions": [
+    {
+        "command": "deploy_linaro_image",
+        "parameters": {
+            "image": "http://people.linaro.org/~pfalcon/cbuild-lava/cbuild-auto-build-precise-r1-ascii.img.gz"
+        }
+    },
+    {
+        "command": "lava_test_shell",
+        "parameters": {
+            "testdef_urls": ["http://%(SERVER_NAME)s:%(SERVER_PORT)s/helpers/scheduler/lava/testdef/%(queue)s/%(job)s"],
+            "timeout": 36000
+        }
+    },
+    {
+         "command": "submit_results",
+         "parameters": {
+             "stream": "/anonymous/cbuild/",
+             "server": "https://validation.linaro.org/lava-server/RPC2/"
+         }
+    }
+  ]
+}

=== added file 'queue/lava-panda/lava-test-shell-template.yaml'
--- queue/lava-panda/lava-test-shell-template.yaml	1970-01-01 00:00:00 +0000
+++ queue/lava-panda/lava-test-shell-template.yaml	2013-01-15 12:59:22 +0000
@@ -0,0 +1,52 @@
+metadata:
+    format: "Lava-Test Test Definition 1.0"
+    name: "%(job)s"
+
+install:
+    deps:
+        - bzr
+    git-repos:
+        - git://git.linaro.org/people/davelong/lt_ti_lava.git
+    steps:
+        - echo In Install
+        - apt-get update
+        # LAVA lab doesn't have panda-es with usb-flash-drive currently
+        #- lt_ti_lava/utils/lava-usb-drive.sh /dev/sda
+        #- mkdir -p cbuild-lava
+        #- mount /dev/sda1 cbuild-lava
+        #- df
+
+        # LAVA Debugging
+        #- apt-get install -y openssh-server
+
+        #- Fetch build script first for "fail fast" approach
+        - bzr branch --use-existing-dir lp:~linaro-infrastructure/cbuild/cbuild-lava2 cbuild-lava
+        - cd cbuild-lava/slaves/lava
+        - ./workaround-multiarch.sh
+
+        #- Install build dependencies
+        #- apt-get install -y build-essential gdb lsb-release time python-irclib wget gnupg ccrypt rsync coreutils sendemail bzr xdelta3 xz-utils unzip openssh-client libio-socket-ssl-perl libnet-ssleay-perl
+        - apt-get install -y build-essential gdb lsb-release time wget coreutils bzr xz-utils unzip openssh-client
+        #- apt-get build-dep -y gcc gcc-4.5 gdb eglibc
+        - apt-get build-dep -y gcc gcc-4.5
+
+run:
+    steps:
+        - set -x
+        - echo In Run
+        - cd cbuild-lava/slaves/lava
+        - make -f ../../lib/build.mk final/%(job)s.stamp
+        - echo CBuild make finished, attaching logs to LAVA results
+        - lava-test-case gcc-configure --result pass
+        - lava-test-case gcc-build --result pass
+        - lava-test-case gcc-install --result pass
+        - lava-test-case cbuild-global --result pass
+        - lava-test-case-attach gcc-configure results/build/*/logs/*/gcc-configure.txt
+        - lava-test-case-attach gcc-build results/build/*/logs/*/gcc-build.txt
+        - lava-test-case-attach gcc-install results/build/*/logs/*/gcc-install.txt
+        - lava-test-case-attach cbuild-global results/build/*/logs/*/gcc-build.time
+        - lava-test-case-attach cbuild-global results/build/*/logs/*/gcc-archive.time
+        - lava-test-case-attach cbuild-global results/build/*/logs/*/host.txt
+        - lava-test-case-attach cbuild-global results/build/*/logs/*/md5sums.txt
+        - lava-test-case-attach cbuild-global results/build/*/logs/*/finished.txt
+        - echo Attaching logs done

=== added directory 'queue/lava-qemu-minimal'
=== added file 'queue/lava-qemu-minimal/hosts.txt'
--- queue/lava-qemu-minimal/hosts.txt	1970-01-01 00:00:00 +0000
+++ queue/lava-qemu-minimal/hosts.txt	2013-01-15 12:59:22 +0000
@@ -0,0 +1,1 @@
+lava

=== added file 'queue/lava-qemu-minimal/lava-job-template.json'
--- queue/lava-qemu-minimal/lava-job-template.json	1970-01-01 00:00:00 +0000
+++ queue/lava-qemu-minimal/lava-job-template.json	2013-01-15 12:59:22 +0000
@@ -0,0 +1,27 @@
+{
+  "timeout": 1800,
+  "job_name": "%(job)s",
+  "device_type": "qemu",
+  "actions": [
+    {
+        "command": "deploy_linaro_image",
+        "parameters": {
+            "image": "file:///tmp/lava/beagle-nano.img.gz"
+        }
+    },
+    {
+        "command": "lava_test_shell",
+        "parameters": {
+            "testdef_urls": ["http://%(SERVER_NAME)s:%(SERVER_PORT)s/helpers/scheduler/lava/testdef/%(queue)s/%(job)s"],
+            "timeout": 36000
+        }
+    },
+    {
+         "command": "submit_results",
+         "parameters": {
+             "stream": "/anonymous/cbuild/",
+             "server": "http://localhost/cbuild/RPC2/"
+         }
+    }
+  ]
+}

=== added file 'queue/lava-qemu-minimal/lava-test-shell-template.yaml'
--- queue/lava-qemu-minimal/lava-test-shell-template.yaml	1970-01-01 00:00:00 +0000
+++ queue/lava-qemu-minimal/lava-test-shell-template.yaml	2013-01-15 12:59:22 +0000
@@ -0,0 +1,12 @@
+metadata:
+    format: "Lava-Test Test Definition 1.0"
+    name: "%(job)s"
+
+install:
+        bzr-repos:
+            - lp:~linaro-infrastructure/cbuild/cbuild-lava2
+        steps:
+            - echo In Install for job %(job)s
+run:
+    steps:
+        - echo In run for job %(job)s

=== added directory 'spawn/lava-test'
=== added symlink 'spawn/lava-test/lava-qemu-minimal'
=== target is u'../../queue/lava-qemu-minimal'