Merge lp:~salgado/offspring/private-projects into lp:~linaro-automation/offspring/linaro

[Guilherme Salgado]

This is a work in progress

[lp:~salgado/offspring/private-projects]

30. By Guilherme Salgado on 2011-09-23

Fix one final place which was using offspring-builder-script instead of offspring-builder-config

31. By Guilherme Salgado on 2011-11-03

merge all-python-config

32. By Guilherme Salgado on 2011-11-03

merge all-python-config

33. By Guilherme Salgado on 2011-11-03

Update one remaining place which was still running offspring-builder-config

34. By Guilherme Salgado on 2011-11-18

merge trunk

35. By Guilherme Salgado on 2011-11-18

Some minor tweaks

36. By Guilherme Salgado on 2011-11-18

revert a few unnecessary changes

37. By Guilherme Salgado on 2011-11-18

remove a .bzrignore change that's no longer needed

38. By Guilherme Salgado on 2011-11-18

Add an object factory to create anonymous django model instances in tests

39. By Guilherme Salgado on 2011-11-18

Remove makeAccessGroup() from ObjectFactory as we don't want to depend on django-group-access yet. Also add a makeProjectGroup() method to ObjectFactory

40. By Guilherme Salgado on 2011-11-18

Rename the object factory methods to follow PEP-8 naming standard rather than camelCase

41. By Guilherme Salgado on 2011-11-18

Add support for private projects on models of web.queuemanager

42. By Guilherme Salgado on 2011-11-23

merge object-factory branch

43. By Guilherme Salgado on 2011-11-23

Fix a typo on a factory method call

44. By Guilherme Salgado on 2011-11-23

Rename makeAccessGroup to make_access_group

45. By Guilherme Salgado on 2011-11-30

merge trunk

46. By Guilherme Salgado on 2011-11-30

Ammend the 001-project-privacy.sql migration script to create the projects_access_groups table

47. By Guilherme Salgado on 2011-11-30

Revert the path-independence changes

48. By Guilherme Salgado on 2011-11-30

merge trunk

49. By Guilherme Salgado on 2011-12-01

Lots of docstrings for tests

Unmerged revisions

49. By Guilherme Salgado on 2011-12-01

Lots of docstrings for tests

48. By Guilherme Salgado on 2011-11-30

merge trunk

47. By Guilherme Salgado on 2011-11-30

Revert the path-independence changes

46. By Guilherme Salgado on 2011-11-30

Ammend the 001-project-privacy.sql migration script to create the projects_access_groups table

45. By Guilherme Salgado on 2011-11-30

merge trunk

44. By Guilherme Salgado on 2011-11-23

Rename makeAccessGroup to make_access_group

43. By Guilherme Salgado on 2011-11-23

Fix a typo on a factory method call

42. By Guilherme Salgado on 2011-11-23

merge object-factory branch

41. By Guilherme Salgado on 2011-11-18

Add support for private projects on models of web.queuemanager

40. By Guilherme Salgado on 2011-11-18

Rename the object factory methods to follow PEP-8 naming standard rather than camelCase

=== modified file 'Makefile'
--- Makefile	2011-09-21 23:22:28 +0000
+++ Makefile	2011-09-22 20:33:19 +0000
@@ -46,7 +46,10 @@
 	./.virtualenv/bin/nosetests offspring.master
 
 test-web: web install-test-runner
-	./bin/offspring-web test --settings=offspring.web.settings_test
+	./bin/offspring-web test queuemanager --settings=offspring.web.settings_test
+
+test-web-postgres: web install-test-runner
+	./bin/offspring-web test queuemanager --settings=offspring.web.settings_devel
 
 test: master slave web install-test-runner test-web
 	./.virtualenv/bin/nosetests -e 'queuemanager.*'

=== modified file 'lib/offspring/web/queuemanager/models.py'
--- lib/offspring/web/queuemanager/models.py	2011-03-24 21:30:51 +0000
+++ lib/offspring/web/queuemanager/models.py	2011-09-22 20:33:19 +0000
@@ -12,14 +12,16 @@
 )
 import math
 
-from django.conf import settings
-from django.contrib.auth.models import User
+from django.contrib.auth.models import AnonymousUser, User
 from django.db import (
     connection,
     models
 )
 
 from offspring import config
+from django_group_access.models import (
+    AccessGroupMixin,
+    AccessManager)
 from offspring.enums import ProjectBuildStates
 
 
@@ -130,7 +132,44 @@
             return u""
 
 
-class Project(models.Model):
+class PrivacyAwareManagerMixin(object):
+
+    @property
+    def _privacy_attr(self):
+        privacy_attr = 'is_private'
+        if getattr(self.model, 'access_relation', None) is not None:
+            privacy_attr = self.model.access_relation + '__' + privacy_attr
+        return privacy_attr
+
+
+class PublicObjectsManager(models.Manager, PrivacyAwareManagerMixin):
+
+    def get_query_set(self):
+        return super(PublicObjectsManager, self).get_query_set().filter(
+            models.Q(**{self._privacy_attr: False}))
+
+
+class PrivateObjectsManager(AccessManager, PrivacyAwareManagerMixin):
+
+    def _get_accessible_by_user_filter_rules(self, user):
+        rules = super(
+            PrivateObjectsManager, self)._get_accessible_by_user_filter_rules(
+                user)
+        return rules | models.Q(**{self._privacy_attr: False})
+
+    def accessible_by_user(self, user):
+        if isinstance(user, AnonymousUser):
+            # If we're passed in an AnonymousUser we must not attempt to apply
+            # the usual constraints because they rely on the given user
+            # existing in the DB and that's not the case with AnonymousUsers.
+            return super(PrivateObjectsManager, self).get_query_set().filter(
+                models.Q(**{self._privacy_attr: False}))
+        else:
+            return super(PrivateObjectsManager, self).accessible_by_user(
+                user)
+
+
+class Project(AccessGroupMixin):
     #XXX: This should probably be managed in the database.
     STATUS_CHOICES = (
         (u'devel', u'Active Development'),
@@ -140,7 +179,15 @@
         (u'experimental', u'Experimental'),
         (u'obsolete', u'Obsolete'),
     )
+    all_objects = PrivateObjectsManager()
+    # Using PublicObjectsManager here we ensure that any oversights when
+    # updating existing uses of Project.objects won't leak private projects.
+    objects = PublicObjectsManager()
 
+    is_private = models.BooleanField(default=False)
+    # We allow projects without an owner for backwards compatibility but
+    # there's a DB constraint to ensure private projects always have an owner.
+    owner = models.ForeignKey(User, blank=True, null=True)
     name = models.SlugField('codename', max_length=200, primary_key=True, unique=True)
     title = models.CharField('project title', max_length=30)
     project_group = models.ForeignKey(ProjectGroup, blank=True, null=True)
@@ -229,6 +276,13 @@
     current_job = models.ForeignKey("BuildResult", db_column="current_job_id", editable=False, blank=True, null=True)
     notes = models.TextField('whiteboard', blank=True, null=True)
 
+    # Using PublicObjectsManager here we ensure that any oversights when
+    # updating existing uses of Lexbuilder.objects won't leak anything related
+    # to private projects.
+    access_relation = 'current_job__project'
+    objects = PublicObjectsManager()
+    all_objects = PrivateObjectsManager()
+
     class Meta:
         db_table = "lexbuilders"
 
@@ -303,6 +357,13 @@
     dispatched_at = models.DateTimeField('date dispatched', editable=False, null=True, blank=True) 
     notes = models.CharField(max_length=200, null=True, blank=True)
 
+    # Using PublicObjectsManager here we ensure that any oversights when
+    # updating existing uses of BuildResult.objects won't leak anything
+    # related to private projects.
+    access_relation = 'project'
+    objects = PublicObjectsManager()
+    all_objects = PrivateObjectsManager()
+
     class Meta:
         db_table = "buildresults"
         get_latest_by = "started_at"
@@ -357,6 +418,13 @@
     score = models.IntegerField(default=10)
     reason = models.TextField('request reason', blank=True, max_length=200)
 
+    # Using PublicObjectsManager here we ensure that any oversights when
+    # updating existing uses of BuildRequest.objects won't leak anything
+    # related to private projects.
+    access_relation = 'project'
+    objects = PublicObjectsManager()
+    all_objects = PrivateObjectsManager()
+
     class Meta:
         db_table = "buildrequests"
 
@@ -492,6 +560,12 @@
         ("RC", "Release Candidate"),
         ("GM", "Gold Master"),
     )
+    # Using PublicObjectsManager here we ensure that any oversights when
+    # updating existing uses of Release.objects won't leak anything related to
+    # private projects.
+    access_relation = 'build__project'
+    objects = PublicObjectsManager()
+    all_objects = PrivateObjectsManager()
 
     build = models.OneToOneField(BuildResult, primary_key=True, unique=True)
     name = models.CharField('release name', max_length=200)

=== added directory 'lib/offspring/web/queuemanager/sql'
=== added file 'lib/offspring/web/queuemanager/sql/project.postgresql_psycopg2.sql'
--- lib/offspring/web/queuemanager/sql/project.postgresql_psycopg2.sql	1970-01-01 00:00:00 +0000
+++ lib/offspring/web/queuemanager/sql/project.postgresql_psycopg2.sql	2011-09-22 20:33:19 +0000
@@ -0,0 +1,5 @@
+-- XXX: This doesn't apply when running tests because sqlite3 doesn't support
+-- addition of new constraints to existing tables.  Need to decide whether
+-- it's preferred to just ignore this in our tests or use postgresql to run
+-- the tests as well.
+ALTER TABLE projects ADD CONSTRAINT "private_project_requires_owner" CHECK (is_private IS FALSE OR owner_id IS NOT NULL);

=== added directory 'lib/offspring/web/queuemanager/tests'
=== added file 'lib/offspring/web/queuemanager/tests/__init__.py'
--- lib/offspring/web/queuemanager/tests/__init__.py	1970-01-01 00:00:00 +0000
+++ lib/offspring/web/queuemanager/tests/__init__.py	2011-09-22 20:33:19 +0000
@@ -0,0 +1,2 @@
+from .test_models import *
+

=== added file 'lib/offspring/web/queuemanager/tests/factory.py'
--- lib/offspring/web/queuemanager/tests/factory.py	1970-01-01 00:00:00 +0000
+++ lib/offspring/web/queuemanager/tests/factory.py	2011-09-22 20:33:19 +0000
@@ -0,0 +1,108 @@
+from itertools import count
+
+from django.contrib.auth.models import User
+from django_group_access.models import AccessGroup
+
+from offspring.web.queuemanager.models import (
+    BuildRequest,
+    BuildResult,
+    Lexbuilder,
+    Project,
+    Release)
+
+
+class ObjectFactory(object):
+
+    def __init__(self):
+        self.counter = count()
+
+    def getUniqueEmailAddress(self):
+        return "[e-mail]" % self.getUniqueString('email')
+
+    def getUniqueString(self, prefix=None):
+        """Return a string unique to this factory instance.
+
+        :param prefix: Used as a prefix for the unique string. If unspecified,
+            defaults to 'generic-string'.
+        """
+        if prefix is None:
+            prefix = "generic-string"
+        string = "%s%s" % (prefix, self.getUniqueInteger())
+        return string.lower()
+
+    def getUniqueInteger(self):
+        """Return an integer unique to this factory instance."""
+        return self.counter.next()
+
+    def makeAccessGroup(self, users):
+        name = self.getUniqueString()
+        group = AccessGroup(name=name)
+        group.save()
+        for user in users:
+            group.members.add(user)
+        group.save()
+        return group
+
+    def makeUser(self):
+        userid = password = self.getUniqueString()
+        user = User.objects.create_user(
+            userid, self.getUniqueEmailAddress(), password)
+        user.save()
+        return user
+
+    def makeProject(self, name=None, title=None, is_private=False,
+                    owner=None, access_groups=[]):
+        if name is None:
+            name = self.getUniqueString()
+        if title is None:
+            title = self.getUniqueString()
+        if owner is None:
+            owner = self.makeUser()
+        project = Project(
+            name=name, title=title, is_private=is_private, owner=owner)
+        project.save()
+        for group in access_groups:
+            project.access_groups.add(group)
+        project.save()
+        return project
+
+    def makeBuildResult(self, project=None):
+        if project is None:
+            project = self.makeProject()
+            project.save()
+        result = BuildResult(project=project)
+        result.save()
+        return result
+
+    def makeRelease(self, build=None, name=None, creator=None):
+        if build is None:
+            build = self.makeBuildResult()
+        if name is None:
+            name = self.getUniqueString()
+        if creator is None:
+            creator = self.makeUser()
+        release = Release(build=build, name=name, creator=creator)
+        release.save()
+        return release
+
+    def makeLexbuilder(self, name=None, uri=None, current_job=None):
+        if name is None:
+            name = self.getUniqueString()
+        if uri is None:
+            uri = 'htt://example.com/%s' % name
+        if current_job is None:
+            current_job = self.makeBuildResult()
+        builder = Lexbuilder(name=name, uri=uri, current_job=current_job)
+        builder.save()
+        return builder
+
+    def makeBuildRequest(self, project=None):
+        if project is None:
+            project = self.makeProject()
+            project.save()
+        request = BuildRequest(project=project)
+        request.save()
+        return request
+
+
+factory = ObjectFactory()

=== added file 'lib/offspring/web/queuemanager/tests/test_models.py'
--- lib/offspring/web/queuemanager/tests/test_models.py	1970-01-01 00:00:00 +0000
+++ lib/offspring/web/queuemanager/tests/test_models.py	2011-09-22 20:33:19 +0000
@@ -0,0 +1,167 @@
+from django.contrib.auth.models import AnonymousUser
+from django.test import TestCase
+
+from offspring.web.queuemanager.models import (
+    BuildRequest,
+    BuildResult,
+    Lexbuilder,
+    Project,
+    Release)
+from offspring.web.queuemanager.tests.factory import factory
+
+
+class ProjectTests(TestCase):
+
+    def test_default_model_manager_only_returns_public_projects(self):
+        project = factory.makeProject(is_private=False)
+        project2 = factory.makeProject(is_private=True)
+        self.assertEqual([project], list(Project.objects.all()))
+
+    def test_all_objects_model_manager(self):
+        # Project.all_objects() is a separate model manager which returns
+        # private objects as well. Its .all() method will return public and
+        # private objects regardless of whether or not the current user has
+        # access to them. It is to be used with caution.
+        project = factory.makeProject(is_private=False)
+        project2 = factory.makeProject(is_private=True)
+        self.assertEqual([project, project2], list(Project.all_objects.all()))
+
+    def test_accessible_by_user_with_anonymous_user(self):
+        project = factory.makeProject(is_private=True)
+        project2 = factory.makeProject(is_private=False)
+        self.assertEqual(True, project.is_private)
+        self.assertEqual(False, project2.is_private)
+        objects = Project.all_objects.accessible_by_user(AnonymousUser())
+        self.assertEqual([project2], list(objects))
+
+    def test_accessible_by_user_with_public_project(self):
+        user = factory.makeUser()
+        project = factory.makeProject(is_private=False)
+        self.assertEqual(False, project.is_private)
+        objects = Project.all_objects.accessible_by_user(user)
+        self.assertEqual([project], list(objects))
+
+    def test_accessible_by_user_with_private_project(self):
+        user = factory.makeUser()
+        group = factory.makeAccessGroup([user])
+        project = factory.makeProject(is_private=True, access_groups=[group])
+        # This second project is private but has no access groups set up, so
+        # the user cannot see it.
+        project2 = factory.makeProject(is_private=True)
+        self.assertEqual(True, project.is_private)
+        self.assertEqual(True, project2.is_private)
+        objects = Project.all_objects.accessible_by_user(user)
+        self.assertEqual([project], list(objects))
+
+
+class ReleaseOrLexbuilderTestsMixin(object):
+    """A mixin with tests that work for either Release or Lexbuilder.
+
+    You just need to mix this with TestCase in your subclass and define
+    factoryMethod and model.
+    """
+    model = None
+
+    def factoryMethod(self, build):
+        raise NotImplementedError()
+
+    def test_default_model_manager_only_returns_public_objects(self):
+        private_project = factory.makeProject(is_private=True)
+        private_object = self.factoryMethod(
+            build=factory.makeBuildResult(project=private_project))
+        public_project = factory.makeProject(is_private=False)
+        public_object = self.factoryMethod(
+            build=factory.makeBuildResult(project=public_project))
+        self.assertEqual([public_object], list(self.model.objects.all()))
+
+    def test_all_objects_model_manager(self):
+        # self.model.all_objects() is a separate model manager which returns
+        # private objects as well. Its .all() method will return public and
+        # private objects regardless of whether or not the current user has
+        # access to them. It is to be used with caution.
+        public_object = self.factoryMethod(build=factory.makeBuildResult(
+            project=factory.makeProject(is_private=False)))
+        private_object = self.factoryMethod(build=factory.makeBuildResult(
+            project=factory.makeProject(is_private=True)))
+        self.assertEqual([public_object, private_object],
+                         list(self.model.all_objects.all()))
+
+    def test_accessible_by_user(self):
+        user = factory.makeUser()
+        group = factory.makeAccessGroup([user])
+        project = factory.makeProject(is_private=True, access_groups=[group])
+        visible_object = self.factoryMethod(
+            build=factory.makeBuildResult(project=project))
+        # This release/builder is from a private project which has no access
+        # groups set up, so the user cannot see it.
+        invisible_object = self.factoryMethod(build=factory.makeBuildResult(
+            project=factory.makeProject(is_private=True)))
+        self.assertEqual(
+            [visible_object],
+            list(self.model.all_objects.accessible_by_user(user)))
+
+
+class ReleaseTests(TestCase, ReleaseOrLexbuilderTestsMixin):
+    model = Release
+
+    def factoryMethod(self, build):
+        return factory.makeRelease(build=build)
+
+
+class LexbuilderTests(TestCase, ReleaseOrLexbuilderTestsMixin):
+    model = Lexbuilder
+
+    def factoryMethod(self, build):
+        return factory.makeLexbuilder(current_job=build)
+
+
+class BuildResultOrRequestsTestsMixin(object):
+    """A mixin with tests that work for either BuildResult or BuildRequest.
+
+    You just need to mix this with TestCase in your subclass and define
+    factoryMethod and model.
+    """
+    factoryMethod = None
+    model = None
+
+    def test_default_model_manager_only_returns_public_builds(self):
+        private_object = self.factoryMethod(
+            project=factory.makeProject(is_private=True))
+        public_object = self.factoryMethod(
+            project=factory.makeProject(is_private=False))
+        self.assertEqual([public_object], list(self.model.objects.all()))
+
+    def test_all_objects_model_manager(self):
+        # self.model.all_objects() is a separate model manager which returns
+        # private objects as well. Its .all() method will return public and
+        # private objects regardless of whether or not the current user has
+        # access to them. It is to be used with caution.
+        public_object = self.factoryMethod(
+                project=factory.makeProject(is_private=False))
+        private_object = self.factoryMethod(
+                project=factory.makeProject(is_private=True))
+        self.assertEqual([public_object, private_object],
+                         list(self.model.all_objects.all()))
+
+    def test_accessible_by_user(self):
+        user = factory.makeUser()
+        group = factory.makeAccessGroup([user])
+        project = factory.makeProject(is_private=True, access_groups=[group])
+        visible_object = self.factoryMethod(project=project)
+        # This release is from a private project which has no access groups
+        # set up, so the user cannot see it.
+        invisible_object = self.factoryMethod(
+                project=factory.makeProject(is_private=True))
+        self.assertEqual(
+            [visible_object],
+            list(self.model.all_objects.accessible_by_user(user)))
+
+
+class BuildResultTests(TestCase, BuildResultOrRequestsTestsMixin):
+    model = BuildResult
+    factoryMethod = factory.makeBuildResult
+
+
+class BuildRequestTests(TestCase, BuildResultOrRequestsTestsMixin):
+    model = BuildRequest
+    factoryMethod = factory.makeBuildRequest

=== modified file 'lib/offspring/web/settings.py'
--- lib/offspring/web/settings.py	2011-09-20 01:13:12 +0000
+++ lib/offspring/web/settings.py	2011-09-22 20:33:19 +0000
@@ -96,6 +96,7 @@
     'djcelery',
     'djkombu',
     'piston',
+    'django_group_access',
     # Offspring Apps
     'offspring.web.queuemanager',
 )

=== added directory 'migration'
=== added file 'migration/001-project-privacy.sql'
--- migration/001-project-privacy.sql	1970-01-01 00:00:00 +0000
+++ migration/001-project-privacy.sql	2011-09-22 20:33:19 +0000
@@ -0,0 +1,3 @@
+ALTER TABLE projects ADD COLUMN "is_private" boolean NOT NULL default FALSE;
+ALTER TABLE projects ADD COLUMN "owner_id" integer REFERENCES "auth_user" ("id") DEFERRABLE INITIALLY DEFERRED;
+ALTER TABLE projects ADD CONSTRAINT "private_project_requires_owner" CHECK (is_private IS FALSE OR owner_id IS NOT NULL);

=== modified file 'requirements/requirements.web.txt'
--- requirements/requirements.web.txt	2011-08-11 00:16:09 +0000
+++ requirements/requirements.web.txt	2011-09-22 20:33:19 +0000
@@ -11,3 +11,4 @@
 celery>=2.3.1
 django-celery>=2.3.0
 django-kombu>=0.9.4
+django-group-access>=0.0.1