Merge lp:~dooferlad/linaro-license-protection/add_api into lp:~linaro-automation/linaro-license-protection/trunk

[James Tunnicliffe]

Adds an API for clients to use to download files. For sample usage see scripts/download.py.

[Milo Casagrande (milo)]

Hi James, thanks for working on this.

It looks to go for me.
A couple of things that can be fixed during merge:

216 +# Generate the URL that will return the license information. This is the URL
217 +# if the file with /api/license prepended to the path.

I guess there is a small type there: s/if the file/of the file

Another thing that might be good is to add something in the HACKING file about the new changes/API.

review: Approve

[lp:~dooferlad/linaro-license-protection/add_api]

178. By James Tunnicliffe on 2013-03-12

Added information about the API to the HACKING document.
Added test for the listing API.
Updated download.py to use all parts of the API, demonstrating how to download all files in a directory, perform interactive checks with the user for if they accept a license and storing which licenses the user accepts so they don't have to re-accept a license they have alread read.

179. By James Tunnicliffe on 2013-03-12

Added tests to check failure mode of API.

=== modified file 'HACKING'
--- HACKING	2012-08-17 15:05:44 +0000
+++ HACKING	2013-03-12 15:05:47 +0000
@@ -94,3 +94,31 @@
 
   staging.snapshots.linaro.org
   staging.releases.linaro.org
+
+API
+---
+An API is provided to allow scripts to easily interact with the service without
+having to scrape the web interface. It is designed as a RESTful service and is
+demonstrated in scripts/download.py, which:
+
+ * Gets a directory listing using /api/ls
+   - <server>/api/ls/<path to file> returns a JSON document containing all the
+     data shown by a file listing on the web interface.
+ * Downloads each file, using the type key from /api/ls to avoid directories
+ * Displays the license that protects the file to the user by fetching it
+   using /api/license.
+   - <server>/api/license/<path to file> returns a JSON document containing
+     the licence information for the file pointed to. Both the license text
+     and the digest used to accept the license are returned. Including the
+     digest means that the choice of digest used internally can change without
+     re-writing clients - to the client this is just a magic string and they
+     don't need to care how it is generated.
+ * Stores the digest for each license that the user accepts to avoid asking
+   them to accept the same license twice.
+ * Downloads each file in the directory by providing the appropriate license
+   accept header.
+   - The existing web service stores which licenses have been accepted in a
+     cookie. This is incompatible with a stateless API and is also
+     inconvenient to manage in scripts. The license accept check function now
+     checks for a header containing accepted license digests, which scripts
+     should use.

=== modified file 'license_protected_downloads/tests/test_views.py'
--- license_protected_downloads/tests/test_views.py	2013-02-28 11:20:23 +0000
+++ license_protected_downloads/tests/test_views.py	2013-03-12 15:05:47 +0000
@@ -1,5 +1,6 @@
 __author__ = 'dooferlad'
 
+from datetime import datetime
 from django.conf import settings
 from django.test import Client, TestCase
 import hashlib
@@ -8,6 +9,7 @@
 import unittest
 import urllib2
 import urlparse
+import json
 
 from license_protected_downloads import bzr_version
 from license_protected_downloads.buildinfo import BuildInfo
@@ -191,6 +193,108 @@
         file_path = os.path.join(TESTSERVER_ROOT, target_file)
         self.assertEqual(response['X-Sendfile'], file_path)
 
+    def test_api_get_license_list(self):
+        target_file = "build-info/snowball-blob.txt"
+        digest = self.set_up_license(target_file)
+
+        license_url = "/api/license/" + target_file
+
+        # Download JSON containing license information
+        response = self.client.get(license_url)
+        data = json.loads(response.content)["licenses"]
+
+        # Extract digests
+        digests = [d["digest"] for d in data]
+
+        # Make sure digests match what is in the database
+        self.assertIn(digest, digests)
+        self.assertEqual(len(digests), 1)
+
+    def test_api_get_license_list_multi_license(self):
+        target_file = "build-info/multi-license.txt"
+        digest_1 = self.set_up_license(target_file)
+        digest_2 = self.set_up_license(target_file, 1)
+
+        license_url = "/api/license/" + target_file
+
+        # Download JSON containing license information
+        response = self.client.get(license_url)
+        data = json.loads(response.content)["licenses"]
+
+        # Extract digests
+        digests = [d["digest"] for d in data]
+
+        # Make sure digests match what is in the database
+        self.assertIn(digest_1, digests)
+        self.assertIn(digest_2, digests)
+        self.assertEqual(len(digests), 2)
+
+    def test_api_get_license_list_404(self):
+        target_file = "build-info/snowball-b"
+        license_url = "/api/license/" + target_file
+
+        # Download JSON containing license information
+        response = self.client.get(license_url)
+        self.assertEqual(response.status_code, 404)
+
+    def test_api_download_file(self):
+        target_file = "build-info/snowball-blob.txt"
+        digest = self.set_up_license(target_file)
+
+        url = urlparse.urljoin("http://testserver/", target_file)
+        response = self.client.get(url, follow=True,
+                                   HTTP_LICENSE_ACCEPTED=digest)
+        self.assertEqual(response.status_code, 200)
+        file_path = os.path.join(TESTSERVER_ROOT, target_file)
+        self.assertEqual(response['X-Sendfile'], file_path)
+
+    def test_api_download_file_multi_license(self):
+        target_file = "build-info/multi-license.txt"
+        digest_1 = self.set_up_license(target_file)
+        digest_2 = self.set_up_license(target_file, 1)
+
+        url = urlparse.urljoin("http://testserver/", target_file)
+        response = self.client.get(url, follow=True,
+                        HTTP_LICENSE_ACCEPTED=" ".join([digest_1, digest_2]))
+        self.assertEqual(response.status_code, 200)
+        file_path = os.path.join(TESTSERVER_ROOT, target_file)
+        self.assertEqual(response['X-Sendfile'], file_path)
+
+    def test_api_download_file_404(self):
+        target_file = "build-info/snowball-blob.txt"
+        digest = self.set_up_license(target_file)
+
+        url = urlparse.urljoin("http://testserver/", target_file[:-2])
+        response = self.client.get(url, follow=True,
+                                   HTTP_LICENSE_ACCEPTED=digest)
+        self.assertEqual(response.status_code, 404)
+
+    def test_api_get_listing(self):
+        url = "/api/ls/build-info"
+        response = self.client.get(url)
+        self.assertEqual(response.status_code, 200)
+
+        data = json.loads(response.content)["files"]
+
+        # For each file listed, check some key attributes
+        for file_info in data:
+            file_path = os.path.join(TESTSERVER_ROOT,
+                                     file_info["url"].lstrip("/"))
+            if file_info["type"] == "folder":
+                self.assertTrue(os.path.isdir(file_path))
+            else:
+                self.assertTrue(os.path.isfile(file_path))
+
+            mtime = datetime.fromtimestamp(
+                os.path.getmtime(file_path)).strftime('%d-%b-%Y %H:%M')
+
+            self.assertEqual(mtime, file_info["mtime"])
+
+    def test_api_get_listing_404(self):
+        url = "/api/ls/buld-info"
+        response = self.client.get(url)
+        self.assertEqual(response.status_code, 404)
+
     def test_OPEN_EULA_txt(self):
         target_file = '~linaro-android/staging-vexpress-a9/test.txt'
         url = urlparse.urljoin("http://testserver/", target_file)

=== modified file 'license_protected_downloads/views.py'
--- license_protected_downloads/views.py	2013-02-28 10:05:14 +0000
+++ license_protected_downloads/views.py	2013-03-12 15:05:47 +0000
@@ -83,14 +83,14 @@
             # example), it doesn't have a mtime.
             mtime = 0
 
-        type = "other"
+        target_type = "other"
         if os.path.isdir(file):
-            type = "folder"
+            target_type = "folder"
         else:
             type_tuple = guess_type(name)
             if type_tuple and type_tuple[0]:
                 if type_tuple[0].split('/')[0] == "text":
-                    type = "text"
+                    target_type = "text"
 
         if os.path.exists(file):
             size = os.path.getsize(file)
@@ -111,7 +111,7 @@
         license_list = License.objects.all_with_hashes(license_digest_list)
         listing.append({'name': name,
                         'size': _sizeof_fmt(size),
-                        'type': type,
+                        'type': target_type,
                         'mtime': mtime,
                         'license_digest_list': license_digest_list,
                         'license_list': license_list,
@@ -298,6 +298,11 @@
 
 
 def license_accepted(request, digest):
+    license_header = "HTTP_LICENSE_ACCEPTED"
+    if license_header in request.META:
+        if digest in request.META[license_header].split():
+            return True
+
     return 'license_accepted_' + digest in request.COOKIES
 
 
@@ -387,7 +392,7 @@
     if not result:
         raise Http404
 
-    type = result[0]
+    target_type = result[0]
     path = result[1]
 
     if get_client_ip(request) in config.INTERNAL_HOSTS:
@@ -411,7 +416,7 @@
             if openid_response:
                 return openid_response
 
-    if type == "dir":
+    if target_type == "dir":
         # Generate a link to the parent directory (if one exists)
         if url != '/' and url != '':
             up_dir = "/" + os.path.split(url)[0]
@@ -496,3 +501,59 @@
         raise
 
     return HttpResponse(data)
+
+
+def list_files_api(request, path):
+    url = path
+    result = test_path(path)
+    if not result:
+        raise Http404
+
+    target_type = result[0]
+    path = result[1]
+
+    if target_type == "dir":
+        listing = dir_list(url, path)
+
+        clean_listing = []
+        for entry in listing:
+            if len(entry["license_list"]) == 0:
+                entry["license_list"] = ["Open"]
+
+            clean_listing.append({
+                "name": entry["name"],
+                "size": entry["size"],
+                "type": entry["type"],
+                "mtime": entry["mtime"],
+                "url": entry["url"],
+            })
+
+        data = json.dumps({"files": clean_listing})
+    else:
+        data = json.dumps({"files": ["File not found."]})
+
+    return HttpResponse(data, mimetype='application/json')
+
+
+def get_license_api(request, path):
+    result = test_path(path)
+    if not result:
+        raise Http404
+
+    target_type = result[0]
+    path = result[1]
+
+    if target_type == "dir":
+        data = json.dumps({"licenses":
+                           ["File not found."]})
+    else:
+        license_digest_list = is_protected(path)
+        license_list = License.objects.all_with_hashes(license_digest_list)
+        if len(license_list) == 0:
+            license_list = ["Open"]
+        else:
+            license_list = [{"text": l.text, "digest": l.digest}
+                            for l in license_list]
+        data = json.dumps({"licenses": license_list})
+
+    return HttpResponse(data, mimetype='application/json')

=== added file 'scripts/download.py'
--- scripts/download.py	1970-01-01 00:00:00 +0000
+++ scripts/download.py	2013-03-12 15:05:47 +0000
@@ -0,0 +1,89 @@
+#!/usr/bin/python
+
+import json
+import urlparse
+import shutil
+import urllib2
+import os
+from html2text import html2text
+
+# Example of how to use the API to download all files in a directory. This is
+# written as one procedural script without functions
+directory_url = "http://localhost:8001/build-info"
+
+# Generate the URL that will return the license information. This is the URL
+# of the file with /api/license prepended to the path.
+
+# Unfortunately urlsplit returns an immutable object. Convert it to an array
+# so we can modify the path section (index 2)
+parsed_url = [c for c in urlparse.urlsplit(directory_url)]
+url_path_section = parsed_url[2]
+
+parsed_url[2] = "/api/ls" + url_path_section
+listing_url = urlparse.urlunsplit(parsed_url)
+
+u = urllib2.urlopen(listing_url)
+data = json.loads(u.read())["files"]
+
+for file_info in data:
+    if file_info["type"] == "folder":
+        # Skip folders...
+        continue
+
+    parsed_url[2] = "/api/license" + file_info["url"]
+    license_url = urlparse.urlunsplit(parsed_url)
+
+    parsed_url[2] = file_info["url"]
+    file_url = urlparse.urlunsplit(parsed_url)
+
+    # Get the licenses. They are returned as a JSON document in the form:
+    # {"licenses":
+    #  [{"text": "<license text>", "digest": "<digest of license>"},
+    #   {"text": "<license text>", "digest": "<digest of license>"},
+    #   ...
+    # ]}
+    # Each license has a digest associated with it.
+    u = urllib2.urlopen(license_url)
+    data = json.loads(u.read())["licenses"]
+
+    if data[0] == "Open":
+        headers = {}
+    else:
+        # If this were a command line client designed to ask the user to accept
+        # each license, you could use this code to ask the user to accept each
+        # license in turn. In this example we store which licenses are accepted
+        # so the user only has to accept them once.
+        if os.path.isfile("accepted_licenses"):
+            with open("accepted_licenses") as accepted_licenses_file:
+                accepted_licenses = accepted_licenses_file.read().split()
+        else:
+            accepted_licenses = []
+
+        # Present each license to the user...
+        for d in data:
+            if d["digest"] not in accepted_licenses:
+                # Licenses are stored as HTML. Convert them to markdown (text)
+                # and print it to the terminal.
+                print html2text(d["text"])
+
+                # Ask the user if they accept the license. If they don't we
+                # terminate the script.
+                user_response = raw_input("Do you accept this license? (y/N)")
+                if user_response != "y":
+                    exit(1)
+
+                accepted_licenses.append(d["digest"])
+
+        # Store the licenses that the user accepted
+        with open("accepted_licenses", "w") as accepted_licenses_file:
+            accepted_licenses_file.write(" ".join(accepted_licenses))
+
+        # To accept a license, place the digest in the LICENSE_ACCEPTED header.
+        # For multiple licenses, they are stored space separated.
+        digests = [d["digest"] for d in data]
+        headers = {"LICENSE_ACCEPTED": " ".join(digests)}
+
+    # Once the header has been generated, just download the file.
+    req = urllib2.urlopen(urllib2.Request(file_url, headers=headers))
+    with open(os.path.basename(parsed_url[2]), 'wb') as fp:
+        shutil.copyfileobj(req, fp)

=== modified file 'urls.py'
--- urls.py	2013-02-26 19:38:30 +0000
+++ urls.py	2013-03-12 15:05:47 +0000
@@ -46,6 +46,12 @@
         'license_protected_downloads.views.get_textile_files',
         name='get_textile_files'),
 
+    url(r'^api/ls/(?P<path>.*)$',
+        'license_protected_downloads.views.list_files_api'),
+
+    url(r'^api/license/(?P<path>.*)$',
+        'license_protected_downloads.views.get_license_api'),
+
     # Catch-all. We always return a file (or try to) if it exists.
     # This handler does that.
     url(r'(?P<path>.*)', 'license_protected_downloads.views.file_server'),