Merge lp:~gary-lasker/software-center/catalog-published-date-lp803028 into lp:software-center

[Gary Lasker]

This branch implements the new 'date_published" field for for-purchase applications from the software-center-agent and incorporates it into catalog_times so that we display the for-purchase items in the What's New section correctly and in the correct order per their date of publication. This branch fixes bug 803028 and bug 86698.

The corresponding unit tests are added/updated. Note that currently the unit test points to the vpn test server because this functionality has not been deployed to the staging server yet. This is simply to prevent the unit tests failure before the staging server deployment is complete. I've marked this with a TODO to update it to point to the staging server once the support is there.

Finally, also note that I've left in the previous cataloged_time functionality as a fallback for for-purchase items so that these applications will not fall off the What's New list completely in the interim before the corresponding agent code is deployed on the production server. Once the agent is deployed, however, this fallback code will no longer be called and will have no effect. Nevertheless, I have marked it clearly with a TODO to remove it since it will serve no useful purpose after the production deployment is in place.

Should we also consider this fix as a candidate for an SRU to Oneiric?

Thanks!

=== modified file 'debian/changelog'
--- debian/changelog	2011-12-12 12:45:41 +0000
+++ debian/changelog	2011-12-12 22:09:24 +0000
@@ -15,7 +15,14 @@
     - Add a translator comment to the 'Top Rated %s' string, name the 
      variable. LP: #868971
 
- -- Michael Vogt <[e-mail]>  Fri, 09 Dec 2011 18:26:59 +0100
+  [ Gary Lasker ]
+  * lp:~gary-lasker/software-center/catalog-published-date-lp803028:
+    - add date_published value from the software-center-agent server
+      for use with cataloged time so that for-purchase items appear
+      correctly and in the proper order in What's New; include unit
+      tests for the new functionality (LP: #803028, LP: #886698
+
+ -- Gary Lasker <[e-mail]>  Mon, 12 Dec 2011 16:45:32 -0500
 
 software-center (5.1.3.1) precise; urgency=low
 
 
=== modified file 'softwarecenter/db/application.py'
--- softwarecenter/db/application.py	2011-11-28 23:28:52 +0000
+++ softwarecenter/db/application.py	2011-12-12 22:09:24 +0000
@@ -339,7 +339,7 @@
         from softwarecenter.db.history import get_pkg_history
         self._history = get_pkg_history()
         return self._history.get_installed_date(self.pkgname)
-        
+
     @property
     def purchase_date(self):
         if self._doc:
@@ -359,6 +359,11 @@
             return self._distro.get_license_text(self.component)
 
     @property
+    def date_published(self):
+        if self._doc:
+            return self._doc.get_value(XapianValues.DATE_PUBLISHED)
+
+    @property
     def maintenance_status(self):
         return self._distro.get_maintenance_status(
             self._cache, self.display_name, self.pkgname, self.component, 
@@ -671,6 +676,7 @@
         details.append("             license: %s" % self.license)
         details.append("         license_key: %s" % self.license_key[0:3] + len(self.license_key)*"*")
         details.append("    license_key_path: %s" % self.license_key_path)
+        details.append("      date_published: %s" % self.date_published)
         details.append("  maintenance_status: %s" % self.maintenance_status)
         details.append("           pkg_state: %s" % self.pkg_state)
         details.append("               price: %s" % self.price)
 
=== modified file 'softwarecenter/db/update.py'
--- softwarecenter/db/update.py	2011-12-12 09:15:52 +0000
+++ softwarecenter/db/update.py	2011-12-12 22:09:24 +0000
@@ -23,8 +23,8 @@
 import json
 import string
 import shutil
+import xapian
 import time
-import xapian
 
 from gi.repository import GObject
 
@@ -144,6 +144,7 @@
                 'Deb-Line'   : 'deb_line',
                 'Signing-Key-Id' : 'signing_key_id',
                 'License'    : 'license',
+                'Date-Published' : 'date_published',
                 'Purchased-Date' : 'purchase_date',
                 'License-Key' : 'license_key',
                 'License-Key-Path' : 'license_key_path',
@@ -544,7 +545,7 @@
         try:
             # magic channel
             entry.channel = AVAILABLE_FOR_PURCHASE_MAGIC_CHANNEL_NAME
-            # icon is transmited inline
+            # icon is transmitted inline
             if hasattr(entry, "icon_data") and entry.icon_data:
                 icondata = base64.b64decode(entry.icon_data)
             elif hasattr(entry, "icon_64_data") and entry.icon_64_data:
@@ -637,9 +638,15 @@
                 doc.add_value(axi_values["catalogedtime"], 
                               xapian.sortable_serialise(cataloged_times[pkgname]))
             else:
-                # also catalog apps not found in axi (e.g. for-purchase apps)
+                #####################################################
+                # TODO: This is just a fallback so that we keep our current
+                #       behavior of having new items for-purchase appear in
+                #       what's new...THIS SHOULD BE REMOVED after support
+                #       for date_purchased in the agent has been deployed
+                #       to the production server
                 doc.add_value(axi_values["catalogedtime"], 
                               xapian.sortable_serialise(time.time()))
+                #####################################################
         # pocket (main, restricted, ...)
         if parser.has_option_desktop("X-AppInstall-Section"):
             archive_section = parser.get_desktop("X-AppInstall-Section")
@@ -662,6 +669,23 @@
         if parser.has_option_desktop("X-AppInstall-License"):
             license = parser.get_desktop("X-AppInstall-License")
             doc.add_value(XapianValues.LICENSE, license)
+        # date published
+        if parser.has_option_desktop("X-AppInstall-Date-Published"):
+            date_published = parser.get_desktop("X-AppInstall-Date-Published")
+            # strip the subseconds from the end of the published date string
+            date_published = str(date_published).split(".")[0]
+            doc.add_value(XapianValues.DATE_PUBLISHED,
+                          date_published)
+            # we use the date published value for the cataloged time as well
+            if "catalogedtime" in axi_values:
+                LOG.debug(
+                        ("pkgname: %s, date_published cataloged time is: %s" %
+                             (pkgname, parser.get_desktop("date_published"))))
+                date_published_sec = time.mktime(
+                                        time.strptime(date_published,
+                                                      "%Y-%m-%d  %H:%M:%S"))
+                doc.add_value(axi_values["catalogedtime"], 
+                              xapian.sortable_serialise(date_published_sec))
         # purchased date
         if parser.has_option_desktop("X-AppInstall-Purchased-Date"):
             date = parser.get_desktop("X-AppInstall-Purchased-Date")
 
=== modified file 'softwarecenter/enums.py'
--- softwarecenter/enums.py	2011-11-11 03:05:45 +0000
+++ softwarecenter/enums.py	2011-12-12 22:09:24 +0000
@@ -129,6 +129,7 @@
     LICENSE_KEY_PATH = 193           # no longer used
     LICENSE = 194
     VIDEO_URL = 195
+    DATE_PUBLISHED = 196
 
 # fake channels
 PURCHASED_NEEDS_REINSTALL_MAGIC_CHANNEL_NAME = "for-pay-needs-reinstall"
 
=== modified file 'test/test_database.py'
--- test/test_database.py	2011-10-07 14:48:05 +0000
+++ test/test_database.py	2011-12-12 22:09:24 +0000
@@ -15,7 +15,10 @@
 from softwarecenter.db.enquire import AppEnquire
 from softwarecenter.db.database import parse_axi_values_file
 from softwarecenter.db.pkginfo import get_pkg_info
-from softwarecenter.db.update import update_from_app_install_data, update_from_var_lib_apt_lists, update_from_appstream_xml
+from softwarecenter.db.update import (update_from_app_install_data,
+                                      update_from_var_lib_apt_lists,
+                                      update_from_appstream_xml,
+                                      update_from_software_center_agent)
 from softwarecenter.enums import (
     XapianValues,
     PkgStates,
@@ -94,7 +97,6 @@
         self.assertEqual(db.get_doccount(), 1)
 
     def test_build_from_software_center_agent(self):
-        from softwarecenter.db.update import update_from_software_center_agent
         db = xapian.WritableDatabase("./data/test.db", 
                                      xapian.DB_CREATE_OR_OVERWRITE)
         cache = apt.Cache()
@@ -120,7 +122,6 @@
                 doc.get_value(XapianValues.ICON).startswith("sc-agent"))
 
     def test_license_string_data_from_software_center_agent(self):
-        from softwarecenter.db.update import update_from_software_center_agent
         from softwarecenter.testutils import get_test_pkg_info
         #os.environ["SOFTWARE_CENTER_DEBUG_HTTP"] = "1"
         os.environ["SOFTWARE_CENTER_BUY_HOST"] = "http://sc.staging.ubuntu.com/"
@@ -279,9 +280,45 @@
             doc.get_value(value_time) >= last_time
             last_time = doc.get_value(value_time)
             
-    def test_non_axi_apps_cataloged_time(self):
-        db = xapian.WritableDatabase("./data/test.db", 
-                                     xapian.DB_CREATE_OR_OVERWRITE)
+    def test_for_purchase_apps_date_published(self):
+        from softwarecenter.testutils import get_test_pkg_info
+        #os.environ["SOFTWARE_CENTER_DEBUG_HTTP"] = "1"
+        ####################################################################
+        # TODO: Point the following back to http://sc.staging.ubuntu.com/ once
+        #       the support for published date is deployed there
+        ####################################################################
+        os.environ["SOFTWARE_CENTER_BUY_HOST"] = "http://sca.razorgirl.info"
+        # staging does not have a valid cert
+        os.environ["PISTON_MINI_CLIENT_DISABLE_SSL_VALIDATION"] = "1"
+        cache = get_test_pkg_info()
+        db = xapian.WritableDatabase("./data/test.db", 
+                                     xapian.DB_CREATE_OR_OVERWRITE)
+        res = update_from_software_center_agent(db, cache, ignore_cache=True)
+        self.assertTrue(res)
+        
+        for p in db.postlist(""):
+            doc = db.get_document(p.docid)
+            date_published = doc.get_value(XapianValues.DATE_PUBLISHED)
+            # make sure that a date_published value is provided
+            self.assertNotEqual(date_published, "")
+            self.assertNotEqual(date_published, None)
+        del os.environ["SOFTWARE_CENTER_BUY_HOST"]
+        
+    def test_for_purchase_apps_cataloged_time(self):
+        from softwarecenter.testutils import get_test_pkg_info
+        #os.environ["SOFTWARE_CENTER_DEBUG_HTTP"] = "1"
+        ####################################################################
+        # TODO: Point the following back to http://sc.staging.ubuntu.com/ once
+        #       the support for published date is deployed there
+        ####################################################################
+        os.environ["SOFTWARE_CENTER_BUY_HOST"] = "http://sca.razorgirl.info"
+        # staging does not have a valid cert
+        os.environ["PISTON_MINI_CLIENT_DISABLE_SSL_VALIDATION"] = "1"
+        cache = get_test_pkg_info()
+        db = xapian.WritableDatabase("./data/test.db", 
+                                     xapian.DB_CREATE_OR_OVERWRITE)
+        res = update_from_software_center_agent(db, cache, ignore_cache=True)
+        self.assertTrue(res)
         res = update_from_app_install_data(db, self.cache, datadir="./data/desktop")
         self.assertTrue(res)
         db = StoreDatabase("./data/test.db", self.cache)
@@ -291,12 +328,15 @@
         sc_app = Application("Ubuntu Software Center Test", "software-center")
         sc_doc = db.get_xapian_document(sc_app.appname, sc_app.pkgname)
         sc_cataloged_time = sc_doc.get_value(axi_value_time)
-        so_app = Application("Scintillant Orange", "scintillant-orange")
-        so_doc = db.get_xapian_document(so_app.appname, so_app.pkgname)
-        so_cataloged_time = so_doc.get_value(axi_value_time)
-        # the test package Scintillant Orange should be cataloged at a
+        for_purch_app = Application("For Purchase Test App", "hellox")
+        for_purch_doc = db.get_xapian_document(for_purch_app.appname,
+                                               for_purch_app.pkgname)
+        for_purch_cataloged_time = for_purch_doc.get_value(axi_value_time)
+        # the for-purchase test package should be cataloged at a
         # later time than axi package Ubuntu Software Center
-        self.assertTrue(so_cataloged_time > sc_cataloged_time)
+        self.assertTrue(for_purch_cataloged_time > sc_cataloged_time)
+
+        del os.environ["SOFTWARE_CENTER_BUY_HOST"]
 
     def test_parse_axi_values_file(self):
         s = """