Merge lp:~pfalcon/linaro-android-build-tools/disk-space-monitor into lp:linaro-android-build-tools

[Paul Sokolovsky]

This bumps ramdisk size to 11.75Gb (previous abs maximum), adds df at the end of build, and allow to specify RAMDISK_SIZE for a build (0 disables it).

Well, as Andy Doan hinted that current cascade of failures is likely related to kernel objects drop into build/out/ , I don't hold my breath that 11.75Gb will help, likely will need to limit ramdisk to build/out/target/ . But so far running as such: https://android-build.linaro.org/jenkins/job/pfalcon_slave-disk-space/6/

[Данило Шеган (danilo)]

As discussed on IRC, when RAMDISK_SIZE is not set, we keep the previous behaviour. It would be nice to do a drive-by fix to make code make sense when read in the second `test .. -gt 10 gig` to actually be `test -gt 12582912` (i.e. 12gb).

review: Approve

[lp:~pfalcon/linaro-android-build-tools/disk-space-monitor]

400. By Paul Sokolovsky on 2012-01-06

Make sure that RAM thresholds are bigger than ramdisk sizes created for them.

401. By Paul Sokolovsky on 2012-01-06

Use ramdisk for just build/out/target , build/out no longer reasonably fits.

402. By Paul Sokolovsky on 2012-01-06

umount build/out/target on cleanup.

403. By Paul Sokolovsky on 2012-01-06

Initial chown jenkins-build should be recursive.

=== modified file 'build-scripts/build-android'
--- build-scripts/build-android	2011-12-20 19:51:31 +0000
+++ build-scripts/build-android	2012-01-06 18:17:39 +0000
@@ -86,6 +86,8 @@
 
 time -p make -j$MAKE_JOBS $MAKE_TARGETS $SHOWCOMMANDS HOST_CC=gcc-4.5 HOST_CXX=g++-4.5 HOST_CPP=cpp-4.5
 
+df
+
 if [ -n "$BUILD_SYSROOT" ]; then
     git clone git://git.linaro.org/android/toolchain/build.git toolchain-build
     rm -rf /tmp/sysroot

=== modified file 'node/build'
--- node/build	2012-01-04 18:33:13 +0000
+++ node/build	2012-01-06 18:17:39 +0000
@@ -7,35 +7,57 @@
 # which accompanies this distribution, and is available at
 # http://www.eclipse.org/legal/epl-v10.html
 ###############################################################################
+#
+# $1 = mirror service host
+# $2 = build CONFIG, base64-encoded
+#
 
 set -xe
 
+# We need ramdisk size when executing under root, but still don't want
+# to evaluate build config (== arbitrary code) as such.
+function get_ramdisk_size () {
+    sudo -E -H -u jenkins-build bash -es "$1" <<\EOF
+    export CONFIGURATION="$(echo "$1" | base64 -id)"
+    set -a
+    eval "$CONFIGURATION"
+    set +a
+    echo $RAMDISK_SIZE
+EOF
+}
+
 BUILD_SCRIPT_ROOT=$(readlink -f "$(dirname "${0}")/../build-scripts")
 
 # Stopgap measure to cleanup environment on instances reused for different jobs
 mount | grep -E "^tmpfs on .+workspace/" | awk ' {print $3}' | xargs --no-run-if-empty -n1 umount
 
+umount build/out/target || true
 umount build/out || true
 umount build || true
 rm -rf build
 mkdir build
 
-# XXX mounting build/out on a tmpfs is probably a bit android specific...
-mem_total=`cat /proc/meminfo | grep MemTotal | sed -s 's/[^0-9]*\([0-9]*\) kB/\1/'`
-if [ "$mem_total" -gt 15680064 ]
-then
-  echo using 11.75G tmpfs for build
-  mount -t tmpfs -o size=11750M tmpfs build
-elif [ "$mem_total" -gt 10485760 ]
-then
-  echo using 11.25G tmpfs for build/out
-  mkdir build/out
-  mount -t tmpfs -o size=11250M tmpfs build/out
-else
-  mkdir build/out
+ramdisk_size=$(get_ramdisk_size $2)
+
+# Put build/* on a ramdisk to speed up build
+if [ -z "$ramdisk_size" -o "$ramdisk_size" != "0" ]; then
+    mem_total=`cat /proc/meminfo | grep MemTotal | sed -s 's/[^0-9]*\([0-9]*\) kB/\1/'`
+    if [ "$mem_total" -gt 15680064 ]; then
+      ramdisk_size=${ramdisk_size:-11750M}
+      echo "Using $ramdisk_size tmpfs for build"
+      mount -t tmpfs -o size=$ramdisk_size tmpfs build
+    elif [ "$mem_total" -gt 12485760 ]; then
+      # XXX mounting build/out/target on a tmpfs is probably a bit android specific...
+      ramdisk_size=${ramdisk_size:-11G}
+      echo "Using $ramdisk_size tmpfs for build/out/target"
+      mkdir -p build/out/target
+      mount -t tmpfs -o size=$ramdisk_size tmpfs build/out/target
+    else
+      mkdir build/out
+    fi
 fi
 
-chown jenkins-build:nogroup build
+chown jenkins-build:nogroup -R build
 cd build
 sudo -E -H -u jenkins-build bash -xes "${BUILD_SCRIPT_ROOT}" "$@" <<\EOF
 export BUILD_SCRIPT_ROOT="${1}"