Merge lp:~dooferlad/linaro-android-frontend/proxy_lava-job-info into lp:linaro-android-frontend

[James Tunnicliffe]

Adds a proxy for lava-job-info files for the front end so results files will be loaded and parsed without being blocked by the license request web page.

django: Added proxy to the API. Checks to see if file is called lava-job-info so it can't just be used to proxy anything. Possibly should tie down a list of acceptable server names as well at some point.

JavaScript: Added a new URL to try to download lava-job-info from and push all requests through above proxy.

[James Tunnicliffe (dooferlad)]

Updated: Rev 245 locks down the proxy so only snapshots.linaro.org will be proxied, and only lava-job-info files on it. Front end understands this and will only use the proxy to access snapshots.linaro.org.

[lp:~dooferlad/linaro-android-frontend/proxy_lava-job-info]

245. By James Tunnicliffe on 2012-01-09

Locked down proxy so only snapshot.linaro.org is proxied. Front end will only use proxy for URLs with snapshots.linaro.org within them.

[Michael Hudson-Doyle (mwhudson)]

I don't understand all the new_style_url/old_style_url gyrations. In particular, it sure looks as if both could go via the proxy, in which case both will succeed at the http level (/api/get-lava-job-info always returns http 200) and then its a race as to which response gets displayed.

The other bits look fine.

[lp:~dooferlad/linaro-android-frontend/proxy_lava-job-info]

246. By James Tunnicliffe on 2012-01-10

Removed unncessarey mangling of old_style_url.
Protect against having two calls to receivedLavaJobInfo that contain parsable data.

[James Tunnicliffe (dooferlad)]

On 9 January 2012 22:24, Michael Hudson-Doyle
<email address hidden> wrote:
> I don't understand all the new_style_url/old_style_url gyrations.  In
> particular, it sure looks as if both could go via the proxy, in which
> case both will succeed at the http level (/api/get-lava-job-info always
> returns http 200) and then its a race as to which response gets displayed.

I did have both go via the proxy at one point, but it does leave the
proxy open to anyone wanting to proxy a file that they can call
lava-job-info. It seemed like restricting the domain to be proxied to
snapshots.linaro.org was reasonable.

I have got rid of the code that makes old_style_url always absolute,
since it has gone back to a non-proxied Y.io call.

The reason for there being two URLs and there being no protection
about which gets loaded is that old_style_url will only match jobs
that have jenkins native archiving enabled and my understanding is
that those jobs will be going away shortly and don't account for any
jobs with artefacts published to snapshots.linaro.org.

It is simple enough to stop receivedLavaJobInfo from modifying the
page twice if you think it is worth protecting against. I have checked
in this change to do so:

    function receivedLavaJobInfo (e, response) {
        if (response.response.match(/^Server error: 404$/))
        {
            return; // Nothing to parse
        }

        if (typeof receivedLavaJobInfo.got_info != "undefined")
        {
            alert("Two resutls files found. Please report this error
by submitting a bug report against" +
                  "
https://code.launchpad.net/~linaro-infrastructure/linaro-android-frontend/trunk")
            return;
        }

        receivedLavaJobInfo.got_info = true;

I don't think there is a right location to get lava-job-info from if
it is in more than one location, so this should be sufficient. I added
an alert because it would be nice to know if getting two files ever
happens.

--
James Tunnicliffe

[Paul Sokolovsky (pfalcon)]

Ok, looks good for v1.0. I'm sure we'd need more tweaks to it, but let's proceed with integration so we know we do the right tweaks.

review: Approve

=== modified file 'android_build/frontend/api.py'
--- android_build/frontend/api.py	2011-11-17 22:53:32 +0000
+++ android_build/frontend/api.py	2012-01-10 11:42:24 +0000
@@ -254,3 +254,32 @@
     if resp:
         return resp
     return HttpResponse('foo')
+
+def get_lava_job_info(request):
+    """Give Web UI a proxy to fetch lava-job-info for it to parse.
+
+    This is required because the android files are now protected by the need
+    for the user to have accepted a license. If a URL isn't for a file called
+    lava-job-info, we return a 404. We could lock down to just
+    snapshots.linaro.org, but that would break old builds unless we complicate
+    the UI javaScript.
+    """
+
+    url = ""
+    for thing in request.GET.getlist("url"):
+        url = thing
+
+    if not (re.search("/lava-job-info$", url) and
+            re.search("^https?://snapshots.linaro.org/", url)):
+        raise Http404
+
+    try:
+        response = urllib2.urlopen(url)
+    except urllib2.HTTPError, e:
+        message = "Server error: {}".format(e.code)
+        return HttpResponse(message)
+    except urllib2.URLError, e:
+        message = "We failed to reach a server. Reason{}".format(e.reason)
+        return HttpResponse(message)
+    else:
+        return HttpResponse(response.read())

=== modified file 'android_build/urls.py'
--- android_build/urls.py	2011-03-21 02:19:57 +0000
+++ android_build/urls.py	2012-01-10 11:42:24 +0000
@@ -27,6 +27,7 @@
     (r'^api/is-daily$', 'android_build.frontend.api.is_daily'),
     (r'^api/new$', 'android_build.frontend.api.new'),
     (r'^api/set-daily$', 'android_build.frontend.api.set_daily'),
+    (r'^api/get-lava-job-info$', 'android_build.frontend.api.get_lava_job_info'),
 
     (r'^openid/', include('django_openid_auth.urls')),
     (r'^logout$', 'django.contrib.auth.views.logout'),

=== modified file 'static/buildDetails.js'
--- static/buildDetails.js	2011-12-16 16:33:24 +0000
+++ static/buildDetails.js	2012-01-10 11:42:24 +0000
@@ -95,6 +95,20 @@
     }
 
     function receivedLavaJobInfo (e, response) {
+        if (response.response.match(/^Server error: 404$/))
+        {
+            return; // Nothing to parse
+        }
+
+        if (typeof receivedLavaJobInfo.got_info != "undefined")
+        {
+            alert("Two resutls files found. Please report this error by submitting a bug report against" +
+                  " https://code.launchpad.net/~linaro-infrastructure/linaro-android-frontend/trunk")
+            return;
+        }
+
+        receivedLavaJobInfo.got_info = true;
+
         var job_info = Y.JSON.parse(response.responseText);
         var job_info_json_url = job_info.lava_url + 'scheduler/job/' + job_info.job_id;
         Y.jsonp(
@@ -144,20 +158,37 @@
               + results.buildUrl + '/console">Tail</a> - <a href="' + results.buildUrl + '/consoleText">Raw</a>';
       },
       testresults: function (results) {
-          if (results.artifacts.length) {
-              for (var i = 0; i < results.artifacts.length; i++) {
-                  var artifact = results.artifacts[i];
-                  if (artifact.displayPath == 'lava-job-info') {
-                      Y.io(
-                          results.buildUrl + '/artifact/' + artifact.relativePath,
-                          {
-                              on: { success: receivedLavaJobInfo }
-                          });
-                      return 'Loading...';
-                  }
-              }
+
+          /* The build result can exist in one of two locations depending
+           on how old the test result is. We create both URLs and fire
+           off fetches to both locations. The one that works will be
+           used to display the results.
+           */
+          var old_style_url = results.buildUrl + '/artifact/build/out/lava-job-info';
+
+          /* Check URL. If it points to snapshot.linaro.org, proxy it through
+             the master, which doesn't get stopped to ask to accept a license. */
+          if (old_style_url.match(/https:\/\/snapshots.linaro.org\//))
+          {
+              old_style_url = '/api/get-lava-job-info?url=' + old_style_url
           }
-          return 'none found';
+
+          var new_style_url = '/api/get-lava-job-info?url=http://snapshots.linaro.org/android/'
+              + rawBuildName + '/' + results.number.toString() + '/lava-job-info';
+
+          // Try both locations. File should exist only in one location...
+          Y.io(
+              old_style_url,
+              {
+                  on: { success: receivedLavaJobInfo }
+              });
+          Y.io(
+              new_style_url,
+              {
+                  on: { success: receivedLavaJobInfo }
+              });
+
+          return 'Loading...';
       },
       output: function (results) {
         var downloadAreaUrl = 'http://snapshots.linaro.org/android/' + rawBuildName + '/' + results.number.toString() + '/';