Merge lp:~dholbach/harvest/packagesets into lp:harvest

[Daniel Holbach]

[Paul Hummer (rockstar)]

=== added file 'harvest/common/launchpad.py'
--- harvest/common/launchpad.py 1970-01-01 00:00:00 +0000
+++ harvest/common/launchpad.py 2010-01-22 17:07:20 +0000
@@ -0,0 +1,38 @@
+from launchpadlib.launchpad import Launchpad, EDGE_SERVICE_ROOT
+from launchpadlib.credentials import Credentials
+from launchpadlib.errors import HTTPError
+
+from django.conf import settings
+
+import sys
+import os
+
+def lp_login(lp_instance=EDGE_SERVICE_ROOT):

A docstring here would be really helpful. Something like:

    """Return a logged in launchpad object."""

+ cachedir = os.path.join(settings.PROJECT_PATH, 'lp_data/cache')
+ creddir = os.path.join(settings.PROJECT_PATH, 'lp_data/lp_credentials')
+ project = settings.PROJECT_NAME.strip()
+ if not os.path.isdir(creddir):
+ os.makedirs(creddir)
+ cred = os.path.join(creddir, '%s.credentials' % project)
+
+ if os.path.exists(cred):
+ credentials = Credentials()
+ credentials.load(open(cred))
+ launchpad = Launchpad(credentials, lp_instance, cachedir)
+ else:
+ try:
+ launchpad = Launchpad.get_token_and_login(project, lp_instance,
+ cachedir)
+ except HTTPError, e:
+ print >> sys.stderr, 'Error connecting to Launchpad: %s' % str(e)
+ sys.exit(1)

It would be better to raise an exception here, instead of printing to standard error and exiting. Sometimes the e part of the except is rather unhelpful.

+ f = open(cred, 'w')
+ os.chmod(cred, 0600)
+ launchpad.credentials.save(f)
+ f.close()
+ return launchpad
+
+def get_packagesets(lp):
+ current_series = lp.distributions['ubuntu'].current_series
+ packagesets = filter(lambda a: a.distroseries == current_series, lp.packagesets)
+ return packagesets

=== modified file 'harvest/opportunities/models.py'
--- harvest/opportunities/models.py 2010-01-22 17:07:20 +0000
+++ harvest/opportunities/models.py 2010-01-22 17:07:20 +0000
@@ -7,9 +7,12 @@
 TYPE_GREEN_THRESHOLD = 200
 TYPE_RED_THRESHOLD = 1000

+class PackageSet(models.Model):
+ name = models.SlugField(_("Name"), max_length=40)

 class SourcePackage(models.Model):
     name = models.SlugField(_("Name"), max_length=70)
+ packagesets = models.ManyToManyField(PackageSet, null=True)

     class Meta:
         ordering = ['name']

So, by this model definition, is it safe to say that packages can be in many different package sets? Have you tested this at all? I think we need a property on packagesets that can link back to the SourcePackages as well.

[Paul Hummer (rockstar)]

review: Needs Information

[Daniel Holbach (dholbach)]

On 26.01.2010 04:28, Paul Hummer wrote:
> A docstring here would be really helpful. Something like:
>
> """Return a logged in launchpad object."""

I'll have a look at that later on. Thanks. :)

> It would be better to raise an exception here, instead of printing to standard error and exiting. Sometimes the e part of the except is rather unhelpful.

Ok. I'll have a look and see what I can improve there.

> So, by this model definition, is it safe to say that packages can be in many different package sets? Have you tested this at all? I think we need a property on packagesets that can link back to the SourcePackages as well.

Have a look at http://people.canonical.com/~cjwatson/packagesets and
search for "mousepad", it ended up in various packagesets. It sure makes
our life more complicated, but it was deemed the best solution for the
reality we are facing.

[Daniel Holbach (dholbach)]

The only thing that's missing is the property for the packagesets that you mentioned. Do you think you can help with that?

[Daniel Holbach (dholbach)]

Just let me know what you think.

[James Westby (james-w)]

Looks ok to me.

review: Approve

=== modified file 'INSTALL'
--- INSTALL	2009-12-07 11:07:52 +0000
+++ INSTALL	2010-01-27 12:44:16 +0000
@@ -1,63 +1,23 @@
-To begin, we have two options to get started with harvest: first one is using sqlite3 and the other one is using postgres
-
-First one is using sqlite3, this is probably the easier one:
-
-1. 
-
-sudo apt-get install python-django python-launchpadlib
-
-- install python-django-openid-auth from karmic or from:
-http://archive.ubuntu.com/ubuntu/pool/universe/p/python-django-openid-auth/
-
-2. 
-
-- cd harvest
-- cp settings.py.sample settings.py
-
-3. 
-
-Now to edit settings.py
-
-set DATABASE_ENGINE = 'sqlite3'
-set DATABASE_NAME = 'dev.db'
-
-4.
-
-- ./manage.py syncdb
-- ./manage.py compilemessages
-- ./manage.py updatelists
-- ./manage.py runserver
-
---------------------------------------------------------------------------------------------------
-
-Second one is using postgres:
-
-1.
-
- - sudo apt-get install postgresql-8.3 python-django python-psycopg2 python-launchpadlib 
-
-2.
-
- - cd harvest
- - cp settings.py.sample settings.py
+
+1. sudo apt-get install python-django python-launchpadlib python-django-openid-auth bzr
+2. cd harvest; cp settings.py.sample settings.py
+
+---
+Optional for postgres usage:
+  - sudo apt-get install postgresql-8.3 python-psycopg2
+  - Edit settings.py
+  - set DATABASE_* according to the appropiate settings 
+  - Not sure if necessary:
+    - sudo passwd postgres
+    change password to 'password' or whatever the password is in harvest/settings.py
+    - sudo -u postgres psql template1
+    ALTER USER postgres WITH ENCRYPTED PASSWORD 'password';
+    - sudo -u postgres createdb -O postgres harvest
+---
+
 
 3.
-
-Edit settings.py
-
-set SECRET_KEY, DATABASE_USER and DATABASE_PASSWORD according to the appropiate settings
-
-* Not sure if necessary
-
-- sudo passwd postgres
-change password to 'password' or whatever the password is in harvest/settings.py
-- sudo -u postgres psql template1  
-ALTER USER postgres WITH ENCRYPTED PASSWORD 'password';
-- sudo -u postgres createdb -O postgres harvest
-
-4.
-
-- ./manage.py syncdb
-- ./manage.py compilemessages
-- ./manage.py updatelists
-- ./manage.py runserver
+ - ./manage.py syncdb
+ - ./manage.py init-harvest
+ - ./manage.py updatelists
+ - ./manage.py runserver
 
=== added file 'harvest/common/launchpad.py'
--- harvest/common/launchpad.py	1970-01-01 00:00:00 +0000
+++ harvest/common/launchpad.py	2010-01-27 12:44:16 +0000
@@ -0,0 +1,46 @@
+from launchpadlib.launchpad import Launchpad, EDGE_SERVICE_ROOT
+from launchpadlib.credentials import Credentials
+from launchpadlib.errors import HTTPError
+
+from django.conf import settings
+
+import sys
+import os
+
+def lp_login(lp_instance=EDGE_SERVICE_ROOT):
+    """
+    Return a logged in Launchpad object. 
+    
+    XXX: Once https://bugs.launchpad.net/soyuz/+bug/510180 is fixed we can 
+    probably use "anonymous login" everywhere.
+    """
+    cachedir = os.path.join(settings.PROJECT_PATH, 'lp_data/cache')
+    creddir = os.path.join(settings.PROJECT_PATH, 'lp_data/lp_credentials')
+    project = settings.PROJECT_NAME.strip()
+    if not os.path.isdir(creddir):
+        os.makedirs(creddir)
+    cred = os.path.join(creddir, '%s.credentials' % project)
+
+    if os.path.exists(cred):
+        credentials = Credentials()
+        credentials.load(open(cred))
+        launchpad = Launchpad(credentials, lp_instance, cachedir)
+    else:
+        try:
+            launchpad = Launchpad.get_token_and_login(project, lp_instance, 
+                                                      cachedir)
+        except HTTPError as e:
+            raise 'Error connecting to Launchpad: %s' % str(e.value)
+        f = open(cred, 'w')
+        os.chmod(cred, 0600)
+        launchpad.credentials.save(f)
+        f.close()
+    return launchpad
+
+def get_packagesets(lp):
+    """
+    Get list of packagesets for current development release of Ubuntu.
+    """
+    current_series = lp.distributions['ubuntu'].current_series
+    packagesets = filter(lambda a: a.distroseries == current_series, lp.packagesets)
+    return packagesets
 
=== removed file 'harvest/common/utils.py'
--- harvest/common/utils.py	2009-07-08 11:31:32 +0000
+++ harvest/common/utils.py	1970-01-01 00:00:00 +0000
@@ -1,9 +0,0 @@
-from django.utils.translation import ugettext as _
-
-def trans_sort_object_list(lst, tr_field):
-    """ Sort an object list with translated_name """
-    for l in lst:
-        l.translated_name = _(getattr(l, tr_field))
-    templist = [(obj_.translated_name.lower(), obj_) for obj_ in lst]
-    templist.sort()
-    return [obj_ for (key1, obj_) in templist]
 
=== added file 'harvest/opportunities/management/commands/init-harvest.py'
--- harvest/opportunities/management/commands/init-harvest.py	1970-01-01 00:00:00 +0000
+++ harvest/opportunities/management/commands/init-harvest.py	2010-01-27 12:44:16 +0000
@@ -0,0 +1,24 @@
+#!/usr/bin/python
+
+from django.core.management.base import NoArgsCommand
+
+import settings
+
+import subprocess
+import os
+
+class Command(NoArgsCommand):
+    help = "Make sure Harvest is set up properly."
+
+    def handle_noargs(self, **options):
+        print " * Compiling messages."
+        subprocess.call(["./manage.py", "compilemessages"])
+
+        path = settings.PROJECT_PATH
+        
+        if os.path.exists(os.path.join(path, "settings.py.sample")) and \
+           os.path.exists(os.path.join(path, "settings.py")):
+            print " * Showing diff between settings.py.sample and settings.py."
+            subprocess.call(["diff", "-u", 
+                             os.path.join(path, "settings.py.sample"),
+                             os.path.join(path, "settings.py")])
 
=== added file 'harvest/opportunities/management/commands/updatepackagesets.py'
--- harvest/opportunities/management/commands/updatepackagesets.py	1970-01-01 00:00:00 +0000
+++ harvest/opportunities/management/commands/updatepackagesets.py	2010-01-27 12:44:16 +0000
@@ -0,0 +1,41 @@
+#!/usr/bin/python
+
+from django.core.management.base import NoArgsCommand
+
+from common import launchpad
+from opportunities.models import SourcePackage, PackageSet
+
+import sys
+
+class Command(NoArgsCommand):
+    help = "Pull packageset information from Launchpad."
+
+    def handle_noargs(self, **options):
+        lp = launchpad.lp_login()
+        if not lp:
+            sys.exit(1)
+        lp_packagesets = launchpad.get_packagesets(lp)
+        package_mapping = {}
+        for lp_packageset in lp_packagesets:
+            packageset, created = PackageSet.objects.get_or_create(name=lp_packageset.name)
+            if created:
+                packageset.save()
+            for package_name in lp_packageset.getSourcesIncluded():
+                if not package_mapping.has_key(package_name):
+                    package_mapping[package_name] = set()
+                package_mapping[package_name].add((lp_packageset.name))
+        for package_name in package_mapping.keys():
+            try:
+                package = SourcePackage.objects.get(name=package_name)
+            except SourcePackage.DoesNotExist:
+                package = None
+            if package:
+                for packageset_name in package_mapping[package_name]:
+                    try:
+                        package.packagesets.get(name=packageset_name)
+                    except PackageSet.DoesNotExist:
+                        package.packagesets.add(PackageSet.objects.get(name=packageset_name))
+                for packageset in package.packagesets.all():
+                    if packageset.name not in package_mapping[package_name]:
+                        package.packagesets.delete(PackageSet.objects.get(name=packageset))
+                package.save()
 
=== modified file 'harvest/opportunities/models.py'
--- harvest/opportunities/models.py	2010-01-15 11:49:19 +0000
+++ harvest/opportunities/models.py	2010-01-27 12:44:16 +0000
@@ -7,9 +7,12 @@
 TYPE_GREEN_THRESHOLD = 200
 TYPE_RED_THRESHOLD = 1000
 
+class PackageSet(models.Model):
+    name = models.SlugField(_("Name"), max_length=40)
 
 class SourcePackage(models.Model):
     name = models.SlugField(_("Name"), max_length=70)
+    packagesets = models.ManyToManyField(PackageSet, null=True)
 
     class Meta:
         ordering = ['name']
 
=== modified file 'harvest/opportunities/views.py'
--- harvest/opportunities/views.py	2010-01-05 01:39:53 +0000
+++ harvest/opportunities/views.py	2010-01-27 12:44:16 +0000
@@ -1,6 +1,5 @@
 from django.core.paginator import Paginator, InvalidPage, EmptyPage
 from django.db.models import Count
-from django.http import HttpResponse
 from django.shortcuts import get_object_or_404
 from django.shortcuts import render_to_response as render
 
@@ -8,7 +7,6 @@
 from django.views.generic import list_detail
 
 import models
-from common import utils
 
 def opportunity_index(request):
     sources_list = models.SourcePackage.objects.all()
 
=== modified file 'harvest/settings.py.sample'
--- harvest/settings.py.sample	2010-01-22 16:49:24 +0000
+++ harvest/settings.py.sample	2010-01-27 12:44:16 +0000
@@ -6,6 +6,7 @@
 DEBUG = True
 TEMPLATE_DEBUG = DEBUG
 STATIC_SERVE = True
+PROJECT_NAME = 'harvest'
 
 ADMINS = ('Daniel Holbach', '[e-mail]')
 MANAGERS = ADMINS