Merge lp:~danilo/linaro-android-build-tools/use-http-proxy into lp:linaro-android-build-tools

[Данило Шеган]

Make use of a http_proxy inside Amazon EC2 us-east zone. Instead of setting
up another squid on android-build.linaro.org, I've decided to simply increase
the disk cache storage for Squid on ci.linaro.org to 30GB (up from 10GB) and
point http_proxy at it.

After a few builds, disk cache usage is at about 16GB on ci.linaro.org (it
is already set up on partition of 400GB).

[Данило Шеган (danilo)]

FWIW, without the seed file we got down to 28 mins for "repo sync" (from 44 mins with seed and no proxy, and 41 for initial run with the proxy) on https://android-build.linaro.org/jenkins/job/danilo_galaxynexus-jb-gcc47-aosp-blob/

[Paul Sokolovsky (pfalcon)]

Changes look good, thumbs up for checking proxy availability first.

But I'm not sure about "repo sync" times - do you have a ling for that. "Standard" seed dl + repo sync time is 20min, I took first random build and see 16min: https://android-build.linaro.org/jenkins/job/linaro-android_panda-ics-gcc47-tilt-tracking-blob/176/parsed_console/? , expand "Info" section:

2 TIME: Seed download and uncompress: 8m44.897s

3 TIME: Repo sync (using seed as reference): 7m37.108s
https://android-build.linaro.org/jenkins/job/linaro-android_panda-ics-gcc47-tilt-tracking-blob/176/parsed_console/?

(warning! that page is huge (100mb), and will certainly crash Firefox and make Chromium thrash swap.)

review: Approve

[Данило Шеган (danilo)]

Thanks for the review Paul.

Now on to your questions.

https://android-build.linaro.org/jenkins/job/danilo_galaxynexus-jb-gcc47-aosp-blob/4/consoleText doesn't show the time as "repo sync (using seed as reference)", but it does output:

real 28m46.952s
user 18m19.650s
sys 6m6.020s

Similarly, no-seed build without proxy (https://android-build.linaro.org/jenkins/job/danilo_galaxynexus-jb-gcc47-aosp-blob/3/consoleText) shows:

real 41m18.811s
user 18m36.370s
sys 6m14.210s

And finally, build with the seed and no proxy (https://android-build.linaro.org/jenkins/job/danilo_galaxynexus-jb-gcc47-aosp-blob/1/consoleText) shows:

TIME: Seed download and uncompress: 8m53.905s
...
TIME: Repo sync (using seed as reference): 44m39.969s

This was all done on the same configuration so results are more comparable. It's quite likely that galaxynexus-jb build is not a good reference point, but I wanted something that's relatively stable and this was just an arbitrary pick (and comparing one to the other is good enough).

This is generally very hard to compare, since it will depend on network conditions, time since seed was updated, and similar, but I wanted to ensure we introduce no big slowdowns, and I think the numbers back that. The point of http proxy is to avoid the need to use the network bandwidth more than anything else.

=== modified file 'build-scripts/build-android'
--- build-scripts/build-android	2012-11-06 06:43:21 +0000
+++ build-scripts/build-android	2012-11-29 12:00:27 +0000
@@ -18,6 +18,7 @@
 
 else
 
+    set_proxy_if_accessible
     if [ -n "$REPO_SEED_URL" ]; then
         repo-sync-from-seed "${1}"
     else

=== modified file 'build-scripts/helpers'
--- build-scripts/helpers	2012-11-22 09:26:42 +0000
+++ build-scripts/helpers	2012-11-29 12:00:27 +0000
@@ -181,3 +181,15 @@
             echo -n "$1";;
     esac
 }
+
+# Check if a proxy is accessible using wget.
+set_proxy_if_accessible() {
+    proxy_address=http://ci.linaro.org:3128/
+    export http_proxy=$proxy_address
+    wget -q -O /dev/null --timeout=10 http://android.git.linaro.org/ || unset http_proxy
+    if [ -z "$http_proxy" ]; then
+        echo "WARNING: Proxy at $proxy_address doesn't work."
+    else
+        echo "INFO: http_proxy set to '$http_proxy'."
+    fi
+}