Merge lp:~julian-edwards/launchpad/metadata-type-bug-595038 into lp:launchpad

[Julian Edwards]

= Summary =
Add meta-data custom upload type for use by the Software Center.

== Implementation details ==
This branch defines a new custom upload type, META_DATA, which is defined by a
file's section being "meta-data".

The file's format is of no concern to Soyuz, it simply lives in the librarian
and is published to an area outside of the repo so that if the repo is
private, the meta-data is still accessible.

== Tests ==
bin/test -cvvt publishing-meta-data-files.txt -t meta-data-custom-files.txt

== Demo and Q/A ==
We need to make a source package with a meta-data custom file attached and
upload. It should get accepted as normal and published appropriately.

= Launchpad lint =

Checking for conflicts. and issues in doctests and templates.
Running jslint, xmllint, pyflakes, and pylint.
Using normal rules.

Linting changed files:
  lib/lp/soyuz/model/queue.py
  lib/lp/soyuz/interfaces/queue.py
  lib/lp/archivepublisher/tests/publishing-meta-data-files.txt
  lib/lp/archiveuploader/nascentuploadfile.py
  lib/lp/archiveuploader/tests/meta-data-custom-files.txt

[Jelmer Vernooij (jelmer)]

lib/lp/archiveuploader/tests/meta-data-custom-files.txt uses the British spelling of "Software Centre".

[Jelmer Vernooij (jelmer)]

Nice, surprising to see how easy it is to add another custom file type.

The enum value "META_DATA" doesn't really describe to me what this data is about to me, as everything that is not the package itself (changes, dsc files) could be considered metadata. CATALOGUE perhaps, or something that explicitly refers to software center?

review: Needs Information (code)

[Julian Edwards (julian-edwards)]

Thanks for the review Jelmer.

I think CATALOGUE is another can of worms, because the US spelling is CATALOG. I also don't think changing the enum name itself will help a lot, to be honest, but the doctest and the enum text should be very clear and easy to understand. Is there anything in those you think can be improved?

J

[Jelmer Vernooij (jelmer)]

Oops, I hadn't realized catalogue was also UK English. :-) The enum value name was a minor issue, I think it's sufficiently documented and fine to land as is.

review: Approve (code)

=== added file 'lib/lp/archivepublisher/tests/publishing-meta-data-files.txt'
--- lib/lp/archivepublisher/tests/publishing-meta-data-files.txt	1970-01-01 00:00:00 +0000
+++ lib/lp/archivepublisher/tests/publishing-meta-data-files.txt	2010-06-17 14:00:53 +0000
@@ -0,0 +1,65 @@
+Publishing Meta-Data Custom Files
+=================================
+
+Meta-data custom files are published, unmodified, to a special area
+outside the actual archive directory.  This is so that the files can be
+seen even when the archive is private, and allows commercial customers
+to browse contents for potential later purchase.
+
+We can demonstrate this publishing behaviour by creating a
+PackageUploadCustom object for a meta-data upload:
+
+    >>> from canonical.launchpad.interfaces.librarian import (
+    ...     ILibraryFileAliasSet)
+    >>> from lp.archiveuploader.tests import mock_logger
+    >>> from lp.registry.interfaces.distribution import IDistributionSet
+    >>> from lp.soyuz.interfaces.publishing import PackagePublishingPocket
+    >>> from lp.soyuz.interfaces.queue import PackageUploadCustomFormat
+    >>> from lp.soyuz.model.queue import PackageUploadCustom
+    >>> from cStringIO import StringIO
+
+    >>> bat = getUtility(IDistributionSet)['ubuntutest']['breezy-autotest']
+    >>> ppa = factory.makeArchive(distribution=bat.distribution)
+    >>> package_upload = bat.createQueueEntry(
+    ...     pocket=PackagePublishingPocket.RELEASE, changesfilename="test",
+    ...     changesfilecontent="test",
+    ...     archive=ppa)
+    >>> content = "test"
+    >>> test_file = getUtility(
+    ...     ILibraryFileAliasSet).create(
+    ...         "testmeta", len(content), StringIO(content), "text/plain")
+    >>> custom_upload = PackageUploadCustom()
+    >>> custom_upload.customformat = PackageUploadCustomFormat.META_DATA
+    >>> custom_upload.packageupload = package_upload
+    >>> custom_upload.libraryfilealias = test_file
+    >>> # commit so that the file is put in the librarian.
+    >>> import transaction
+    >>> transaction.commit()
+
+Now we can publish the custom upload:
+
+    >>> custom_upload.publish(logger=mock_logger)
+    DEBUG: Publishing custom testmeta to ubuntutest/breezy-autotest
+
+The custom file is just called "testmeta" with the contents "test".  It will
+be published in a location of the scheme:
+
+/<person_name>/meta/<ppa_name>/<filename>
+
+    >>> import os
+    >>> from lp.archivepublisher.config import getPubConfig
+    >>> pub_config = getPubConfig(ppa)
+    >>> ppa_root = pub_config.distroroot
+    >>> final_destination = os.path.join(
+    ...     ppa_root, ppa.owner.name, "meta", ppa.name)
+    >>> published_file = os.path.join(
+    ...     final_destination, "testmeta")
+
+    >>> os.path.exists(published_file)
+    True
+
+    >>> with open(published_file, 'rb') as fp:
+    ...     content = fp.read()
+
+    >>> print content
+    test

=== modified file 'lib/lp/archiveuploader/nascentuploadfile.py'
--- lib/lp/archiveuploader/nascentuploadfile.py	2010-05-12 08:17:20 +0000
+++ lib/lp/archiveuploader/nascentuploadfile.py	2010-06-17 14:00:53 +0000
@@ -260,6 +260,8 @@
         'raw-ddtp-tarball': PackageUploadCustomFormat.DDTP_TARBALL,
         'raw-translations-static':
             PackageUploadCustomFormat.STATIC_TRANSLATIONS,
+        'meta-data' :
+            PackageUploadCustomFormat.META_DATA,
         }
 
     @property

=== added file 'lib/lp/archiveuploader/tests/meta-data-custom-files.txt'
--- lib/lp/archiveuploader/tests/meta-data-custom-files.txt	1970-01-01 00:00:00 +0000
+++ lib/lp/archiveuploader/tests/meta-data-custom-files.txt	2010-06-17 14:00:53 +0000
@@ -0,0 +1,21 @@
+Meta-Data Custom files
+======================
+
+Meta-data custom files are files that contain more information about the
+source package they're being uploaded with, such as descriptions and
+pricing.  This information is used by the Software Centre to display
+better descriptions about packages than would normally be found in the
+package itself.
+
+When the CustomUploadFile object is created with the right section name,
+its custom_type property returns the right DBEnum,
+PackageUploadCustomFormat.META_DATA.
+
+    >>> from lp.archiveuploader.nascentuploadfile import CustomUploadFile
+    >>> custom_upload_file = CustomUploadFile(
+    ...     filepath="", digest="", size=1, priority_name="", policy=None,
+    ...     component_and_section="main/meta-data", logger=None)
+
+    >>> print custom_upload_file.custom_type.name
+    META_DATA
+

=== modified file 'lib/lp/soyuz/interfaces/queue.py'
--- lib/lp/soyuz/interfaces/queue.py	2010-03-17 12:14:36 +0000
+++ lib/lp/soyuz/interfaces/queue.py	2010-06-17 14:00:53 +0000
@@ -605,6 +605,17 @@
         reside in the librarian for later retrieval using the webservice.
         """
 
+    def publish_META_DATA(logger):
+        """Publish this custom item as a meta-data file.
+
+        This method writes the meta-data custom file to the archive in
+        the location matching this schema:
+        /<person>/meta/<ppa_name>/<filename>
+
+        It's not written to the main archive location because that could be
+        protected by htaccess in the case of private archives.
+        """
+
 
 class IPackageUploadSet(Interface):
     """Represents a set of IPackageUploads"""
@@ -742,3 +753,10 @@
 
         A tarball containing raw (Gnome) help file translations.
         """)
+
+    META_DATA = DBItem(5, """
+        meta-data
+
+        A file containing meta-data about the package, mainly for use in
+        the Software Center.
+        """)

=== modified file 'lib/lp/soyuz/model/queue.py'
--- lib/lp/soyuz/model/queue.py	2010-05-14 04:51:42 +0000
+++ lib/lp/soyuz/model/queue.py	2010-06-17 14:00:53 +0000
@@ -1747,6 +1747,33 @@
         debug(logger, "Skipping publishing of static translations.")
         return
 
+    def publish_META_DATA(self, logger=None):
+        """See `IPackageUploadCustom`."""
+        # In the future this could use the existing custom upload file
+        # processing which deals with versioning, etc., but that's too
+        # complicated for our needs right now.  Also, the existing code
+        # assumes that everything is a tarball and tries to unpack it.
+
+        archive = self.packageupload.archive
+        # See the XXX near the import for getPubConfig.
+        archive_config = getPubConfig(archive)
+        meta_root = os.path.join(
+            archive_config.distroroot, archive.owner.name)
+        dest_dir = os.path.join(
+            meta_root, "meta", archive.name)
+        dest_file = os.path.join(
+            dest_dir, self.libraryfilealias.filename)
+        if not os.path.isdir(dest_dir):
+            os.makedirs(dest_dir, 0755)
+
+        # At this point we now have a directory of the format:
+        # <person_name>/meta/<ppa_name>
+        # We're ready to copy the file out of the librarian into it.
+
+        file_obj = file(dest_file, "wb")
+        self.libraryfilealias.open()
+        copy_and_close(self.libraryfilealias, file_obj)
+
 
 class PackageUploadSet:
     """See `IPackageUploadSet`"""