Merge lp:~jamalta/launchpad/bug-127489 into lp:launchpad

[Jamal Fanaian]

[Stuart Bishop (stub)]

This is going to need a page test to catch future regressions.

[Gavin Panella (allenap)]

Hi Jamal,

Thank you for working on this bug!

As Stuart commented, this will need a pagetest. In fact, there is
already one that is probably broken by this change. Have a look at the
end of lib/lp/registry/stories/foaf/xx-person-home.txt.

Did you have a mentor for this branch, or a pre-implementation
discussion? Perhaps that might have spotted the need to update the
pagetest. Have a look at https://dev.launchpad.net/PatchSubmission if
you haven't already. It's the same process as all the Launchpad devs
from Canonical use.

I have a comment about the code, because the cmp argument to sorted()
is deprecated (and I think it's gone entirely in Python 3).

Thanks again!

Gavin.

> === modified file 'lib/lp/registry/browser/person.py'
> --- lib/lp/registry/browser/person.py 2009-07-29 15:34:46 +0000
> +++ lib/lp/registry/browser/person.py 2009-07-31 21:12:09 +0000
> @@ -2517,7 +2517,10 @@
> categories = set()
> for contrib in self.contributions:
> categories.update(category for category in contrib['categories'])
> - return sorted(categories, key=attrgetter('title'))
> + sort = {'code': 0, 'bugs': 1, 'blueprints': 2, 'translations': 3,
> + 'answers': 4, 'specs': 5, 'soyuz': 6}
> + return sorted(categories, key=attrgetter('name'),
> + cmp=lambda x,y: cmp(sort[x], sort[y]))

Because cmp is deprecated, I think this would be better as:

       return sorted(categories, key=lambda category: sort[category.name])

>
> @cachedproperty
> def context_is_probably_a_team(self):

review: Needs Fixing

[Jamal Fanaian (jamalta)]

Hi Gavin,

Thanks for your input regarding this bug. I have actually been working on a test for this. I did not have a mentor when I started working on this issue, but went on #launchpad-dev to get some help. Salgado had recommended that I write a unittest to call the view, and make sure the data was returned in the right order, using a LaunchpadObjectFactory. I had been struggling trying to figure out how to do that exactly, and haven't gotten back to it in a few days now. I will take a look at the test you mentioned and make sure I fix it. Also, I appreciate your comment regarding the depracated cmp() function. I am fairly new to Python and was not aware of this. I will fix that shortly as well, using your recommendation.

> Hi Jamal,
>
> Thank you for working on this bug!
>
> As Stuart commented, this will need a pagetest. In fact, there is
> already one that is probably broken by this change. Have a look at the
> end of lib/lp/registry/stories/foaf/xx-person-home.txt.
>
> Did you have a mentor for this branch, or a pre-implementation
> discussion? Perhaps that might have spotted the need to update the
> pagetest. Have a look at https://dev.launchpad.net/PatchSubmission if
> you haven't already. It's the same process as all the Launchpad devs
> from Canonical use.
>
> I have a comment about the code, because the cmp argument to sorted()
> is deprecated (and I think it's gone entirely in Python 3).
>
> Thanks again!
>
> Gavin.
>
>
> > === modified file 'lib/lp/registry/browser/person.py'
> > --- lib/lp/registry/browser/person.py 2009-07-29 15:34:46 +0000
> > +++ lib/lp/registry/browser/person.py 2009-07-31 21:12:09 +0000
> > @@ -2517,7 +2517,10 @@
> > categories = set()
> > for contrib in self.contributions:
> > categories.update(category for category in
> contrib['categories'])
> > - return sorted(categories, key=attrgetter('title'))
> > + sort = {'code': 0, 'bugs': 1, 'blueprints': 2, 'translations': 3,
> > + 'answers': 4, 'specs': 5, 'soyuz': 6}
> > + return sorted(categories, key=attrgetter('name'),
> > + cmp=lambda x,y: cmp(sort[x], sort[y]))
>
> Because cmp is deprecated, I think this would be better as:
>
> return sorted(categories, key=lambda category: sort[category.name])
>
> >
> > @cachedproperty
> > def context_is_probably_a_team(self):

[Jamal Fanaian (jamalta)]

I went through the pagetest you referenced, and it will actually not fail. It is also not possible to test my changes with this because there is not enough data. The problem is that the only two icons displayed are bugs and translations. Sorted alphabetically, bugs will obviously always be first, which is the case with the replacement sort.

To test this I need to be able to add a third entry, say, Answers for example. I have this in my local environment (which I should really revert since it's going to break tests in the future), but I don't know how to properly write a pagetest for it because of this limitation. I think this is the reason salgado recommended writing a unittest.

Let me give the unittests another try to see if I can get them done. Thanks for your help so far!

> Hi Gavin,
>
> Thanks for your input regarding this bug. I have actually been working on a
> test for this. I did not have a mentor when I started working on this issue,
> but went on #launchpad-dev to get some help. Salgado had recommended that I
> write a unittest to call the view, and make sure the data was returned in the
> right order, using a LaunchpadObjectFactory. I had been struggling trying to
> figure out how to do that exactly, and haven't gotten back to it in a few days
> now. I will take a look at the test you mentioned and make sure I fix it.
> Also, I appreciate your comment regarding the depracated cmp() function. I am
> fairly new to Python and was not aware of this. I will fix that shortly as
> well, using your recommendation.
>
> > Hi Jamal,
> >
> > Thank you for working on this bug!
> >
> > As Stuart commented, this will need a pagetest. In fact, there is
> > already one that is probably broken by this change. Have a look at the
> > end of lib/lp/registry/stories/foaf/xx-person-home.txt.
> >
> > Did you have a mentor for this branch, or a pre-implementation
> > discussion? Perhaps that might have spotted the need to update the
> > pagetest. Have a look at https://dev.launchpad.net/PatchSubmission if
> > you haven't already. It's the same process as all the Launchpad devs
> > from Canonical use.
> >
> > I have a comment about the code, because the cmp argument to sorted()
> > is deprecated (and I think it's gone entirely in Python 3).
> >
> > Thanks again!
> >
> > Gavin.
> >
> >
> > > === modified file 'lib/lp/registry/browser/person.py'
> > > --- lib/lp/registry/browser/person.py 2009-07-29 15:34:46 +0000
> > > +++ lib/lp/registry/browser/person.py 2009-07-31 21:12:09 +0000
> > > @@ -2517,7 +2517,10 @@
> > > categories = set()
> > > for contrib in self.contributions:
> > > categories.update(category for category in
> > contrib['categories'])
> > > - return sorted(categories, key=attrgetter('title'))
> > > + sort = {'code': 0, 'bugs': 1, 'blueprints': 2, 'translations': 3,
> > > + 'answers': 4, 'specs': 5, 'soyuz': 6}
> > > + return sorted(categories, key=attrgetter('name'),
> > > + cmp=lambda x,y: cmp(sort[x], sort[y]))
> >
> > Because cmp is deprecated, I think this would be better as:
> >
> > return sorted(categories, key=lambda category: sort[categ...

[Gavin Panella (allenap)]

Have a look at using lp.testing.factory.LaunchpadObjectFactory. You can use this to quickly create, say, branches, so that there are code artifacts for your user. That will probably be enough to test this code.

[Guilherme Salgado (salgado)]

On Fri, 2009-08-07 at 09:19 +0000, Gavin Panella wrote:
> Have a look at using lp.testing.factory.LaunchpadObjectFactory. You can use this to quickly create, say, branches, so that there are code artifacts for your user. That will probably be enough to test this code.

I don't think so. That list of categories is generated based on the kind
of karma the person has on the KarmaCache table, and that table is
populated by a script. Even though the person would get karma when
factory.makeBranch() is called, it wouldn't end up in the KarmaCache
table until we ran the script. We could run the script in the test, but
that'd make the test *a lot* slower.

That's why I suggested a new method on LPObjectFactory to create the
KarmaCache entries directly.

Jamal, I remember you were able to write the new factory method, so you
should be quite close to getting a test for your fix. If you need any
help, just drop by #launchpad-reviewers and I'll help you.

Cheers,

--
Guilherme Salgado <email address hidden>

[Jamal Fanaian (jamalta)]

I have added a test that makes sure the categories returned by PersonView.contributed_categories are sorted correctly. Instead of running the script to update Karma I am using the test factories and creating the KarmaCache records based on how the Karma update script does it.

I may not be doing this the best way, so I am requesting a review and hopefully I can chat with someone about it come Monday to figure out what needs to be improved or changed.

Thanks again for all your help and sorry this took so long!

> On Fri, 2009-08-07 at 09:19 +0000, Gavin Panella wrote:
> > Have a look at using lp.testing.factory.LaunchpadObjectFactory. You can use
> this to quickly create, say, branches, so that there are code artifacts for
> your user. That will probably be enough to test this code.
>
> I don't think so. That list of categories is generated based on the kind
> of karma the person has on the KarmaCache table, and that table is
> populated by a script. Even though the person would get karma when
> factory.makeBranch() is called, it wouldn't end up in the KarmaCache
> table until we ran the script. We could run the script in the test, but
> that'd make the test *a lot* slower.
>
> That's why I suggested a new method on LPObjectFactory to create the
> KarmaCache entries directly.
>
> Jamal, I remember you were able to write the new factory method, so you
> should be quite close to getting a test for your fix. If you need any
> help, just drop by #launchpad-reviewers and I'll help you.
>
> Cheers,
>
> --
> Guilherme Salgado <email address hidden>

review: Needs Resubmitting

[Jamal Fanaian (jamalta)]

Sorry forgot to mention this on my last comment. I will try to ping someone in #launchpad-reviews come Monday to go over my branch, that way we don't have to go back and forth over email.

Thanks!

> I have added a test that makes sure the categories returned by
> PersonView.contributed_categories are sorted correctly. Instead of running the
> script to update Karma I am using the test factories and creating the
> KarmaCache records based on how the Karma update script does it.
>
> I may not be doing this the best way, so I am requesting a review and
> hopefully I can chat with someone about it come Monday to figure out what
> needs to be improved or changed.
>
> Thanks again for all your help and sorry this took so long!
>
> > On Fri, 2009-08-07 at 09:19 +0000, Gavin Panella wrote:
> > > Have a look at using lp.testing.factory.LaunchpadObjectFactory. You can
> use
> > this to quickly create, say, branches, so that there are code artifacts for
> > your user. That will probably be enough to test this code.
> >
> > I don't think so. That list of categories is generated based on the kind
> > of karma the person has on the KarmaCache table, and that table is
> > populated by a script. Even though the person would get karma when
> > factory.makeBranch() is called, it wouldn't end up in the KarmaCache
> > table until we ran the script. We could run the script in the test, but
> > that'd make the test *a lot* slower.
> >
> > That's why I suggested a new method on LPObjectFactory to create the
> > KarmaCache entries directly.
> >
> > Jamal, I remember you were able to write the new factory method, so you
> > should be quite close to getting a test for your fix. If you need any
> > help, just drop by #launchpad-reviewers and I'll help you.
> >
> > Cheers,
> >
> > --
> > Guilherme Salgado <email address hidden>

[Jamal Fanaian (jamalta)]

After taking to salgado in #launchpad-reviews, he recommended that the makeKarmaCache() be modified so that it creates the total records (with category=NULL), instead of processing all entries in KarmaCache to create the sums.

review: Needs Fixing

[lp:~jamalta/launchpad/bug-127489]

9023. By Jamal Fanaian <jamal@jfvm1> on 2009-08-27

Simplified the karma test by only creating totals of the specified products

9024. By Jamal Fanaian <jamal@jfvm1> on 2009-08-30

Cleaned up the code to match the style guide. Modified makeKarmaCache to take category and product objects instead of their ids. Switching back to the launchpad db user after making changes to karma. Removed the raw sql query and replaced it with calls to the karma cache manager.a

9025. By Jamal Fanaian <jamal@jfvm1> on 2009-09-13

Person is no longer an instance variable. makeKarmaCache now requires a product be passed. Organized imports and a few other things according to the style guidelines and recommendations from the merge proposal.

9026. By Jamal Fanaian <jamal@jfvm1> on 2009-09-18

Removed the statement to switch DB user back to launchpad in makeKarmaCache. Renamed to _makeKarmaCache as it is more of an internal call. Added function docs to explain that the DB user will be switched.

9027. By Jamal Fanaian <jamal@jfvm1> on 2009-09-23

replaced the 'blueprints' category with it's correct name 'specs'

Unmerged revisions

9027. By Jamal Fanaian <jamal@jfvm1> on 2009-09-23

replaced the 'blueprints' category with it's correct name 'specs'

=== modified file 'lib/lp/registry/browser/person.py'
--- lib/lp/registry/browser/person.py	2009-07-30 22:35:16 +0000
+++ lib/lp/registry/browser/person.py	2009-08-05 19:55:13 +0000
@@ -2517,7 +2517,9 @@
         categories = set()
         for contrib in self.contributions:
             categories.update(category for category in contrib['categories'])
-        return sorted(categories, key=attrgetter('title'))
+        sort = {'code': 0, 'bugs': 1, 'blueprints': 2, 'translations': 3,
+                'answers': 4, 'specs': 5, 'soyuz': 6}
+        return sorted(categories, key=lambda category: sort[category.name])
 
     @cachedproperty
     def context_is_probably_a_team(self):

=== added file 'lib/lp/registry/browser/tests/test_person_view.py'
--- lib/lp/registry/browser/tests/test_person_view.py	1970-01-01 00:00:00 +0000
+++ lib/lp/registry/browser/tests/test_person_view.py	2009-08-22 23:03:21 +0000
@@ -0,0 +1,138 @@
+# Copyright 2009 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+__metaclass__ = type
+
+import unittest
+
+from canonical.config import config
+from canonical.launchpad.webapp.servers import LaunchpadTestRequest
+from canonical.testing import LaunchpadZopelessLayer
+from canonical.database.sqlbase import cursor
+from lp.registry.browser.person import PersonView
+from lp.testing import TestCaseWithFactory
+from zope.component import getUtility
+from canonical.launchpad.interfaces import IKarmaCacheManager
+
+class TestPerson(TestCaseWithFactory):
+    """Test Person view."""
+
+    layer = LaunchpadZopelessLayer
+
+    def setUp(self):
+        # Create a person
+        TestCaseWithFactory.setUp(self)
+        self.person = self.factory.makePerson()
+        self.view = PersonView(self.person,
+                               LaunchpadTestRequest())
+
+    def test_karma_category_sort(self):
+        # Add karma to some categories for the user
+        cur = cursor()
+        karmacache = self.makeKarmaCache(person=self.person, category_id=2)
+        karmacache = self.makeKarmaCache(person=self.person, category_id=7)
+        karmacache = self.makeKarmaCache(person=self.person, category_id=8) 
+
+        # Update Karma totals
+        self.updateKarmaTotals()
+
+        # Get contributed categories for the user
+        categories = self.view.contributed_categories
+        category_names = []
+        for category in categories:
+            category_names.append(category.name)
+
+        self.assertEqual(category_names, [u'code', u'bugs', u'answers'], 
+                         'Categories are not sorted correctly')
+
+    def makeKarmaCache(self, person=None, value=10, category_id=2,
+                       product_id=5):
+
+        LaunchpadZopelessLayer.switchDbUser(config.karmacacheupdater.dbuser)
+        karmacache = getUtility(IKarmaCacheManager).new(
+            person_id=person.id, value=value, category_id=category_id,
+            product_id=product_id)
+        LaunchpadZopelessLayer.commit()
+
+        return karmacache
+
+    def updateKarmaTotals(self):
+        LaunchpadZopelessLayer.switchDbUser(config.karmacacheupdater.dbuser)
+
+        cur = cursor()
+        cur.execute("DELETE FROM KarmaCache WHERE category IS NULL")
+        cur.execute("""
+            DELETE FROM KarmaCache
+            WHERE project IS NOT NULL AND product IS NULL""")
+        cur.execute("""
+            DELETE FROM KarmaCache
+            WHERE category IS NOT NULL AND project IS NULL AND product IS NULL
+                  AND distribution IS NULL AND sourcepackagename IS NULL""")
+        cur.execute("DELETE FROM KarmaCache WHERE karmavalue <= 0")
+
+        # - All actions with a specific category of a person.
+        cur.execute("""
+            INSERT INTO KarmaCache 
+                (person, category, karmavalue, product, distribution,
+                 sourcepackagename, project)
+            SELECT person, category, SUM(karmavalue), NULL, NULL, NULL, NULL
+            FROM KarmaCache
+            WHERE category IS NOT NULL
+            GROUP BY person, category
+            """)
+
+        # - All actions of a person on a given product.
+        cur.execute("""
+            INSERT INTO KarmaCache 
+                (person, category, karmavalue, product, distribution,
+                 sourcepackagename, project)
+            SELECT person, NULL, SUM(karmavalue), product, NULL, NULL, NULL
+            FROM KarmaCache
+            WHERE product IS NOT NULL
+            GROUP BY person, product
+            """)
+
+        # - All actions of a person on a given distribution.
+        cur.execute("""
+            INSERT INTO KarmaCache 
+                (person, category, karmavalue, product, distribution,
+                 sourcepackagename, project)
+            SELECT person, NULL, SUM(karmavalue), NULL, distribution, NULL, NULL
+            FROM KarmaCache
+            WHERE distribution IS NOT NULL
+            GROUP BY person, distribution
+            """)
+
+        # - All actions of a person on a given project.
+        cur.execute("""
+            INSERT INTO KarmaCache 
+                (person, category, karmavalue, product, distribution,
+                 sourcepackagename, project)
+            SELECT person, NULL, SUM(karmavalue), NULL, NULL, NULL,
+                   Product.project
+            FROM KarmaCache
+            JOIN Product ON product = Product.id
+            WHERE Product.project IS NOT NULL AND product IS NOT NULL
+                  AND category IS NOT NULL
+            GROUP BY person, Product.project
+            """)
+
+        # - All actions with a specific category of a person on a given project
+        # IMPORTANT: This has to be the latest step; otherwise the rows
+        # inserted here will be included in the calculation of the overall
+        # karma of a person on a given project.
+        cur.execute("""
+            INSERT INTO KarmaCache 
+                (person, category, karmavalue, product, distribution,
+                 sourcepackagename, project)
+            SELECT person, category, SUM(karmavalue), NULL, NULL, NULL,
+                   Product.project
+            FROM KarmaCache
+            JOIN Product ON product = Product.id
+            WHERE Product.project IS NOT NULL AND product IS NOT NULL
+                  AND category IS NOT NULL
+            GROUP BY person, category, Product.project
+            """)
+
+def test_suite():
+    return unittest.TestLoader().loadTestsFromName(__name__)