Merge lp:~dooferlad/linaro-license-protection/push_support into lp:~linaro-automation/linaro-license-protection/trunk

[James Tunnicliffe]

Adds to the API to allow file uploads.

settings.py has a blank master key by default, which prevents any API keys from being issued, thus disabling the new functionality. The master key should, of course, be generated on the server that this is installed on and only shared with trusted parties.

[lp:~dooferlad/linaro-license-protection/push_support]

199. By James Tunnicliffe on 2013-06-11

Make full dir path for uploads
Expect local_settings.py
If user supplies a valid key, default protection to OPEN.

200. By James Tunnicliffe on 2013-06-12

Uploads are now downloadable if the upload path isn't the same as one of the shared directories!

201. By James Tunnicliffe on 2013-06-14

Updated to make downloads work from inside the upload directory.

[Milo Casagrande (milo)]

Hi James!

Awesome progress! Everything looks good, there are only two definitely minor comments and one question from me.

First comes the question.
To upload a file there is no "api/" URL, we just use the catch-all urls.py rule and differentiate on GET/POST. Is that right?

So, an imaginary URL like 'https://snapshots.l.o/a_big_file.zip', with my API key and file contents as data, and a POST request, should upload the file. Is it correct?

=== added file 'license_protected_downloads/common.py'
--- license_protected_downloads/common.py 1970-01-01 00:00:00 +0000
+++ license_protected_downloads/common.py 2013-06-14 12:41:40 +0000
@@ -0,0 +1,18 @@
+import os
+
+def safe_path_join(base_path, *paths):
+ """os.path.join with with check that result is inside base_path.

s/with with/with

=== modified file 'license_protected_downloads/views.py'
+
+ # if key is in request.GET["key"] then need to mod path and give
+ # access to a per-key directory.
+ if "key" in request.GET :

Extra space before the colon?

=== modified file 'settings.py'
--- settings.py 2013-06-10 16:03:50 +0000
+++ settings.py 2013-06-14 12:41:40 +0000
 TEMPLATE_CONTEXT_PROCESSORS = (
     'django.contrib.messages.context_processors.messages',
@@ -224,3 +227,7 @@
     "init.css": "http://www.linaro.org/remote/css/init.css",
     "remote.css": "http://www.linaro.org/remote/css/remote.css",
     }
+
+MASTER_API_KEY = ""
+
+from local_settings import *

Maybe is worth adding some comments here about the MASTER_API_KEY, and also update the installation/deployment notes.

Approving since those things can be fixed during merge (and the questions are just to make sure I understood the implementation correctly).
Thanks.

review: Approve

[lp:~dooferlad/linaro-license-protection/push_support]

202. By James Tunnicliffe on 2013-06-14

Added auto-generation of local_settings (if needed) and updated README

203. By James Tunnicliffe on 2013-06-14

Cosmetic fixes

[Milo Casagrande (milo)]

Looks good to go for me!
Thanks James!

review: Approve

=== modified file 'README'
--- README	2013-06-03 14:05:59 +0000
+++ README	2013-06-14 17:43:26 +0000
@@ -21,6 +21,7 @@
    (as used by Launchpad.net) and Atlassian Crowd are supported).
  * post-processing of all uploads (a script that manages uploads)
  * per-IP pass-through (for automatic services like a test framework)
+ * key based file uploads into a private area.
 
 Background
 ----------
@@ -79,6 +80,32 @@
 so it may be inaccessible to the general public.
 
 
+Publishing API
+--------------
+To upload a file using the publishing API you need a key. This can be obtained
+by the query <server>/api/request_key?<MASTER_API_KEY> where MASTER_API_KEY
+is stored in local_settings.py.
+
+Now you can use HTTP PUT requests to push up files:
+curl -F file=@<file> -F key=<key> <server>/path/to/store/file
+
+And this file can be downloaded from:
+<server>/path/to/store/file?key=<key>
+
+ * By default API key protected files have no further license protection.
+ * You can upload multiple files with the same key
+ * Directories are automatically created
+ * File listings still work (as long as you append ?key=<key> to the URL)
+
+Once you have finished with the files, you can delete all files associated
+with a temporary API key by deleting the key:
+
+curl <server>/api/delete_key?key=<key>
+
+Uploading files to a public directory is not supported. This is planned for
+the future.
+
+
 Build-Info support
 ------------------
 

=== added file 'license_protected_downloads/common.py'
--- license_protected_downloads/common.py	1970-01-01 00:00:00 +0000
+++ license_protected_downloads/common.py	2013-06-14 17:43:26 +0000
@@ -0,0 +1,18 @@
+import os
+
+def safe_path_join(base_path, *paths):
+    """os.path.join with check that result is inside base_path.
+
+    Checks that the generated path doesn't end up outside the target
+    directory, so server accesses stay where we expect them.
+    """
+
+    target_path = os.path.join(base_path, *paths)
+
+    if not target_path.startswith(base_path):
+        return None
+
+    if not os.path.normpath(target_path) == target_path.rstrip("/"):
+        return None
+
+    return target_path

=== modified file 'license_protected_downloads/models.py'
--- license_protected_downloads/models.py	2012-08-06 11:53:16 +0000
+++ license_protected_downloads/models.py	2013-06-14 17:43:26 +0000
@@ -26,3 +26,7 @@
 
     def __unicode__(self):
         return self.digest
+
+
+class APIKeyStore(models.Model):
+    key = models.CharField(max_length=80)

=== modified file 'license_protected_downloads/tests/test_views.py'
--- license_protected_downloads/tests/test_views.py	2013-06-05 13:12:53 +0000
+++ license_protected_downloads/tests/test_views.py	2013-06-14 17:43:26 +0000
@@ -10,7 +10,8 @@
 import urllib2
 import urlparse
 import json
-
+import random
+import shutil
 from mock import Mock
 
 from license_protected_downloads import bzr_version
@@ -34,9 +35,19 @@
         self.old_served_paths = settings.SERVED_PATHS
         settings.SERVED_PATHS = [os.path.join(THIS_DIRECTORY,
                                              "testserver_root")]
+        self.old_upload_path = settings.UPLOAD_PATH
+        settings.UPLOAD_PATH = os.path.join(THIS_DIRECTORY,
+                                            "test_upload_root")
+        if not os.path.isdir(settings.UPLOAD_PATH):
+            os.makedirs(settings.UPLOAD_PATH)
+        self.old_master_api_key = settings.MASTER_API_KEY
+        settings.MASTER_API_KEY = "1234abcd"
 
     def tearDown(self):
         settings.SERVED_PATHS = self.old_served_paths
+        settings.MASTER_API_KEY = self.old_master_api_key
+        os.rmdir(settings.UPLOAD_PATH)
+        settings.UPLOAD_PATH = self.old_upload_path
 
 
 class ViewTests(BaseServeViewTest):
@@ -839,6 +850,138 @@
         # Shouldn't be able to escape served paths...
         self.assertEqual(response.status_code, 404)
 
+    def test_get_key(self):
+        response = self.client.get("http://testserver/api/request_key",
+                                   data={"key": settings.MASTER_API_KEY})
+
+        self.assertEqual(response.status_code, 200)
+        # Don't care what the key is, as long as it isn't blank
+        self.assertRegexpMatches(response.content, "\S+")
+
+    def test_get_key_api_disabled(self):
+        settings.MASTER_API_KEY = ""
+        response = self.client.get("http://testserver/api/request_key",
+                                   data={"key": settings.MASTER_API_KEY})
+
+        self.assertEqual(response.status_code, 403)
+
+    def test_get_key_post_and_get_file(self):
+        response = self.client.get("http://testserver/api/request_key",
+                                   data={"key": settings.MASTER_API_KEY})
+
+        self.assertEqual(response.status_code, 200)
+        # Don't care what the key is, as long as it isn't blank
+        self.assertRegexpMatches(response.content, "\S+")
+        key = response.content
+
+        # Now write a file so we can upload it
+        file_content = "test_get_key_post_and_get_file"
+        file_root = "/tmp"
+
+        tmp_file_name = os.path.join(
+            file_root,
+            self.make_temporary_file(file_content))
+
+        # Send the file
+        with open(tmp_file_name) as f:
+            response = self.client.post("http://testserver/file_name",
+                             data={"key": key, "file": f})
+            self.assertEqual(response.status_code, 200)
+
+        # Check the upload worked by reading the file back from its uploaded
+        # location
+        uploaded_file_path = os.path.join(settings.UPLOAD_PATH,
+                                                    key,
+                                                    "file_name")
+        with open(uploaded_file_path) as f:
+            self.assertEqual(f.read(), file_content)
+
+        # Test we can fetch the newly uploaded file if we present the key
+        response = self.client.get("http://testserver/file_name",
+                                   data={"key": key})
+        self.assertEqual(response.status_code, 200)
+
+        response = self.client.get("http://testserver/file_name")
+        self.assertNotEqual(response.status_code, 200)
+
+        # Delete the files generated by the test
+        shutil.rmtree(os.path.join(settings.UPLOAD_PATH, key))
+
+    def test_post_file_no_key(self):
+        file_content = "test_post_file_no_key"
+        file_root = "/tmp"
+
+        tmp_file_name = os.path.join(
+            file_root,
+            self.make_temporary_file(file_content))
+
+        # Try to upload a file without a key.
+        with open(tmp_file_name) as f:
+            response = self.client.post("http://testserver/file_name",
+                             data={"file": f})
+            self.assertEqual(response.status_code, 500)
+
+        # Make sure the file didn't get created.
+        self.assertFalse(os.path.isfile(os.path.join(settings.UPLOAD_PATH,
+                                                    "file_name")))
+
+    def test_post_file_random_key(self):
+        key = "%030x" % random.randrange(256**15)
+        file_content = "test_post_file_random_key"
+        file_root = "/tmp"
+
+        tmp_file_name = os.path.join(
+            file_root,
+            self.make_temporary_file(file_content))
+
+        # Try to upload a file with a randomly generated key.
+        with open(tmp_file_name) as f:
+            response = self.client.post("http://testserver/file_name",
+                             data={"key": key, "file": f})
+            self.assertEqual(response.status_code, 500)
+
+        # Make sure the file didn't get created.
+        self.assertFalse(os.path.isfile(os.path.join(settings.UPLOAD_PATH,
+                                                    key,
+                                                    "file_name")))
+
+    def test_api_delete_key(self):
+        response = self.client.get("http://testserver/api/request_key",
+                                   data={"key": settings.MASTER_API_KEY})
+
+        self.assertEqual(response.status_code, 200)
+        # Don't care what the key is, as long as it isn't blank
+        self.assertRegexpMatches(response.content, "\S+")
+        key = response.content
+        file_content = "test_api_delete_key"
+        file_root = "/tmp"
+
+        tmp_file_name = os.path.join(
+            file_root,
+            self.make_temporary_file(file_content))
+
+        with open(tmp_file_name) as f:
+            response = self.client.post("http://testserver/file_name",
+                             data={"key": key, "file": f})
+            self.assertEqual(response.status_code, 200)
+
+        self.assertTrue(os.path.isfile(os.path.join(settings.UPLOAD_PATH,
+                                                    key,
+                                                    "file_name")))
+
+        # Release the key, the files should be deleted
+        response = self.client.get("http://testserver/api/delete_key",
+                                   data={"key": key})
+        self.assertEqual(response.status_code, 200)
+        self.assertFalse(os.path.isfile(os.path.join(settings.UPLOAD_PATH,
+                                                    key,
+                                                    "file_name")))
+
+        # Key shouldn't work after released
+        response = self.client.get("http://testserver/file_name",
+                                   data={"key": key})
+        self.assertNotEqual(response.status_code, 200)
+
 
 class HowtoViewTests(BaseServeViewTest):
     def test_no_howtos(self):

=== added file 'license_protected_downloads/uploads.py'
--- license_protected_downloads/uploads.py	1970-01-01 00:00:00 +0000
+++ license_protected_downloads/uploads.py	2013-06-14 17:43:26 +0000
@@ -0,0 +1,117 @@
+from django.views.decorators.csrf import csrf_exempt
+from django.http import (
+    HttpResponse,
+    HttpResponseForbidden,
+    HttpResponseServerError
+)
+from django import forms
+import random
+from django.conf import settings
+import os
+import shutil
+from models import APIKeyStore
+from common import safe_path_join
+
+
+class UploadFileForm(forms.Form):
+    file = forms.FileField()
+
+
+def upload_target_path(path, key):
+    """Quick path handling function.
+
+    Checks that the generated path doesn't end up outside the target directory,
+    so you can't set path to start with "/" and upload to anywhere.
+    """
+    base_path = os.path.join(settings.UPLOAD_PATH, key)
+    return safe_path_join(base_path, path)
+
+
+@csrf_exempt
+def file_server_post(request, path):
+    """ Handle post requests.
+
+    All post requests must be accompanied by a valid key. If not, the upload
+    will be ignored.
+
+    Files are stored in a private directory that can not be accessed via the
+    web interface unless you have the same key.
+    """
+    if not ("key" in request.POST and
+        APIKeyStore.objects.filter(key=request.POST["key"])):
+        return HttpResponseServerError("Invalid key")
+
+    form = UploadFileForm(request.POST, request.FILES)
+    if not form.is_valid() or not path:
+        return HttpResponseServerError("Invalid call")
+
+    path = upload_target_path(path, request.POST["key"])
+
+    # Create directory if required
+    dirname = os.path.dirname(path)
+    if not os.path.isdir(dirname):
+        os.makedirs(dirname)
+
+    with open(path, "wb") as destination:
+        for chunk in request.FILES["file"].chunks():
+            destination.write(chunk)
+
+    return HttpResponse("OK")
+
+
+def api_request_key(request):
+    if("key" in request.GET and
+       request.GET["key"] == settings.MASTER_API_KEY and
+       settings.MASTER_API_KEY):
+
+        # Generate a new, random key.
+        key = "%030x" % random.randrange(256**15)
+        while APIKeyStore.objects.filter(key=key):
+            key = "%030x" % random.randrange(256**15)
+
+        api_key = APIKeyStore(key=key)
+        api_key.save()
+        return HttpResponse(key)
+
+    return HttpResponseForbidden()
+
+def api_delete_key(request):
+    if "key" not in request.GET:
+        return HttpResponseServerError("Invalid key")
+
+    key = request.GET["key"]
+    api_key = APIKeyStore.objects.filter(key=key)
+
+    if not api_key:
+        return HttpResponseServerError("Invalid key")
+
+    # Delete key from database and all files associated with it
+    api_key.delete()
+    shutil.rmtree(os.path.join(settings.UPLOAD_PATH, key))
+
+    return HttpResponse("OK")
+
+def api_push_to_server(request):
+    # TODO: Upload files from this machine to another linaro-licence-protection
+    # node.
+    """
+    Something like:
+
+    if request.GET["target"] in settings.REMOTE_SERVERS:
+        remote_server = settings.REMOTE_SERVERS[request.GET["target"]]
+
+        remote_server should contain:
+        {
+            "key": "...",
+            "url": "...",
+        }
+
+        now just POST files from this machine to the specified URL/KEY.
+
+        Possibly add some magic to POST endpoint (file_server_post) to allow
+        (some users??) uploads to a public path:
+
+        POST snapshots.linaro.org/path/to/file?key="key"&public=true
+
+    """
+    pass

=== modified file 'license_protected_downloads/views.py'
--- license_protected_downloads/views.py	2013-06-06 13:24:17 +0000
+++ license_protected_downloads/views.py	2013-06-14 17:43:26 +0000
@@ -19,6 +19,7 @@
 from django.shortcuts import render_to_response, redirect
 from django.template import RequestContext
 from django.utils.encoding import smart_str, iri_to_uri
+from django.views.decorators.csrf import csrf_exempt
 
 import bzr_version
 from buildinfo import BuildInfo, IncorrectDataFormatException
@@ -28,7 +29,9 @@
 import importlib
 group_auth_modules = [importlib.import_module(m) for m in settings.GROUP_AUTH_MODULES]
 from BeautifulSoup import BeautifulSoup
+from uploads import file_server_post
 import config
+from common import safe_path_join
 from group_auth_common import GroupAuthError
 
 
@@ -132,28 +135,22 @@
     return listing
 
 
-def safe_path_join(base_path, *paths):
-    """os.path.join with check that result is inside base_path.
+def test_path(path, served_paths=None):
+    """Check that path points to something we can serve up.
 
-    Checks that the generated path doesn't end up outside the target
-    directory, so server accesses stay where we expect them.
+    served_paths can be provided to overwrite settings.SERVED_PATHS. This is
+    used for uploaded files, which may not be shared in the server root.
     """
 
-    target_path = os.path.join(base_path, *paths)
-
-    if not target_path.startswith(base_path):
-        return None
-
-    if not os.path.normpath(target_path) == target_path.rstrip("/"):
-        return None
-
-    return target_path
-
-
-def test_path(path):
-
-    for basepath in settings.SERVED_PATHS:
+    if served_paths is None:
+        served_paths = settings.SERVED_PATHS
+    else:
+        if not isinstance(served_paths, list):
+            served_paths = [served_paths]
+
+    for basepath in served_paths:
         fullpath = safe_path_join(basepath, path)
+
         if fullpath is None:
             return None
 
@@ -443,11 +440,32 @@
     return response
 
 
+@csrf_exempt
 def file_server(request, path):
     """Serve up a file / directory listing or license page as required"""
     path = iri_to_uri(path)
+
+    # Intercept post requests and send them to file_server_post.
+    if request.method == "POST":
+        return file_server_post(request, path)
+
+    # GET requests are handled by file_server_get
+    elif request.method == "GET":
+        return file_server_get(request, path)
+
+
+def file_server_get(request, path):
+
     url = path
-    result = test_path(path)
+
+    # if key is in request.GET["key"] then need to mod path and give
+    # access to a per-key directory.
+    if "key" in request.GET:
+        path = os.path.join(request.GET["key"], path)
+        result = test_path(path, settings.UPLOAD_PATH)
+    else:
+        result = test_path(path)
+
     if not result:
         raise Http404
 
@@ -522,8 +540,9 @@
     if not file_listed(path, url):
         raise Http404
 
-    if get_client_ip(request) in config.INTERNAL_HOSTS or\
-       is_whitelisted(os.path.join('/', url)):
+    if (get_client_ip(request) in config.INTERNAL_HOSTS or
+        is_whitelisted(os.path.join('/', url)) or
+        "key" in request.GET): # If user has a key, default to open
         digests = 'OPEN'
     else:
         digests = is_protected(path)

=== modified file 'settings.py'
--- settings.py	2013-06-10 16:03:50 +0000
+++ settings.py	2013-06-14 17:43:26 +0000
@@ -30,6 +30,8 @@
     }
 }
 
+FILE_UPLOAD_PERMISSIONS = 0644
+
 TIME_ZONE = None
 
 # Language code for this installation. All choices can be found here:
@@ -80,9 +82,6 @@
 #    'django.contrib.staticfiles.finders.DefaultStorageFinder',
 )
 
-# Make this unique, and don't share it with anybody.
-SECRET_KEY = 'lkye^=q_i(#jies7^cz#anqq(1g0k$luy5^1jr2nk=g#inet(n'
-
 # List of callables that know how to import templates from various sources.
 TEMPLATE_LOADERS = (
     'django.template.loaders.filesystem.Loader',
@@ -183,6 +182,7 @@
 }
 
 SERVED_PATHS = [os.path.join(PROJECT_ROOT, "sampleroot")]
+UPLOAD_PATH = os.path.join(PROJECT_ROOT, "sample_upload_root")
 
 TEMPLATE_CONTEXT_PROCESSORS = (
     'django.contrib.messages.context_processors.messages',
@@ -224,3 +224,34 @@
     "init.css": "http://www.linaro.org/remote/css/init.css",
     "remote.css": "http://www.linaro.org/remote/css/remote.css",
     }
+
+MASTER_API_KEY = ""
+
+# Try to import local_settings. If it doesn't exist, generate it. It contains
+# SECRET_KEY (to keep it secret).
+try:
+    from local_settings import *
+except ImportError:
+    import random
+
+    # Create local_settings with random SECRET_KEY and MASTER_API_KEY
+    char_selection = '0123456789abcdefghijklmnopqrstuvwxyz' \
+                     'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
+    char_selection_with_punctuation = char_selection + '!@#$%^&*(-_=+)'
+
+    # SECRET_KEY contains anything but whitespace
+    secret_key = ''.join(random.sample(char_selection_with_punctuation, 50))
+    local_settings_content = "SECRET_KEY = '%s'\n" % secret_key
+
+    # At the moment the publishing API is still in development so it is
+    # disabled...
+    if False:
+        # MASTER_API_KEY contains characters that don't have to be % encoded
+        # in an HTTP URL.
+        master_api_key = ''.join(random.sample(char_selection, 50))
+        local_settings_content += "MASTER_API_KEY = '%s'\n" % master_api_key
+
+    with open(os.path.join(PROJECT_ROOT, "local_settings.py"), "w") as f:
+        f.write(local_settings_content)
+
+    from local_settings import *

=== modified file 'urls.py'
--- urls.py	2013-03-11 15:31:31 +0000
+++ urls.py	2013-06-14 17:43:26 +0000
@@ -52,6 +52,12 @@
     url(r'^api/license/(?P<path>.*)$',
         'license_protected_downloads.views.get_license_api'),
 
+    url(r'^api/request_key$',
+        'license_protected_downloads.uploads.api_request_key'),
+
+    url(r'^api/delete_key$',
+        'license_protected_downloads.uploads.api_delete_key'),
+
     # Catch-all. We always return a file (or try to) if it exists.
     # This handler does that.
     url(r'(?P<path>.*)', 'license_protected_downloads.views.file_server'),