Merge lp:~inspirated/launchpad/implement-Bug-findAttachments into lp:launchpad

[Kamran Riaz Khan]

Implemented Bug.findAttachments()

Summary:
As part of my Google Summer of Code project, I had to implement attachment searching functionality in Arsenal. The end-product would allow user to specify a search string which would be searched in all the bug attachments for a project.

Doing this efficiently required two modifications in Launchpad:
    * Exposing a bug collection for a particular product
    * Implement a attachment search method for a particular bug (this branch)

Proposed fix:
Export a read operation findAttachments in IBug which returns a collection of IBugAttachment.

Pre-implementation Notes:
    * The search operation is essentially a text-only grep.

Implementation Details:
    * Zope configuration had to be updated to export the method
    * Multi-line searches break the GET request with a “HTTP/1.1 400 Bad Request” response
    * The attachments are all opened as regular file objects and then searched for the target pattern.

Tests:
bin/test -vv -m lp.bugs.tests.test_bug_find_attachment

Demo and Q/A:
Open any Launchpad bug in a browser:
 https://launchpad.dev/bugs/15
Create an attachment and upload any text file containing the string ‘char buf’.
Create a Launchpad instance:
 >>> from launchpadlib.launchpad import Launchpad
 >>> launchpad = Launchpad.login_with('just testing', 'https://api.launchpad.dev/beta/')
 The authorization page:
    (https://launchpad.dev/+authorize-token?oauth_token=ppd2bRZDnN6VX94lRqrq)
 should be opening in your browser. After you have authorized
 this program to access Launchpad on your behalf you should come
 back here and press <Enter> to finish the authentication process.
Load the bug:
 >>> bug = launchpad.bugs[15]
Search for the attachment containing the string ‘char buf’:
 >>> results = bug.findAttachments(text=u'char buf')
 >>> for attachment in results:
 ... print attachment.title
 ...
 Buffer Overflow Intro.txt

lint:
The changes are lint clean.

[Graham Binns (gmb)]

After a lengthy discussion in #launchpad-reviews we've agreed that there should be a different solution to this problem.

review: Disapprove (code)

[lp:~inspirated/launchpad/implement-Bug-findAttachments]

11013. By Kamran Riaz Khan on 2010-06-15

Fixed unicode searches in Bug.findAttachments()

11014. By Kamran Riaz Khan on 2010-06-15

Changes in Bug.findAttachments():
 * Limited the size of attachments that can be searched to 8 MB
 * Modified searching to go-through attachments line-wise

11015. By Kamran Riaz Khan on 2010-06-15

Fixed Bug.findAttachments() to use preferred encoding instead of UTF-8

11016. By Kamran Riaz Khan on 2010-06-16

Added support for multi-line searches in Bug.findAttachments() using
is_encoded parameter

11017. By Kamran Riaz Khan on 2010-06-16

Changes in Bug.findAttachments():
 * Use Boyer-Moore-Horspool algorithm to search attachments in chunks
 * Fixed unicode and multiline unittests

11018. By Kamran Riaz Khan on 2010-06-17

Use encode('unicode_escape') instead of urllib.quote() for multiline
searches

11019. By Kamran Riaz Khan on 2010-06-17

Minor changes to improve code readability.

11020. By Kamran Riaz Khan on 2010-06-17

Removed urllib2 usage in Bug.findAttachments() by opening attachments
directly.

11021. By Kamran Riaz Khan on 2010-06-17

Minor changes to improve unittest readability.

Unmerged revisions

11021. By Kamran Riaz Khan on 2010-06-17

Minor changes to improve unittest readability.

11020. By Kamran Riaz Khan on 2010-06-17

Removed urllib2 usage in Bug.findAttachments() by opening attachments
directly.

11019. By Kamran Riaz Khan on 2010-06-17

Minor changes to improve code readability.

11018. By Kamran Riaz Khan on 2010-06-17

Use encode('unicode_escape') instead of urllib.quote() for multiline
searches

11017. By Kamran Riaz Khan on 2010-06-16

Changes in Bug.findAttachments():
 * Use Boyer-Moore-Horspool algorithm to search attachments in chunks
 * Fixed unicode and multiline unittests

11016. By Kamran Riaz Khan on 2010-06-16

Added support for multi-line searches in Bug.findAttachments() using
is_encoded parameter

11015. By Kamran Riaz Khan on 2010-06-15

Fixed Bug.findAttachments() to use preferred encoding instead of UTF-8

11014. By Kamran Riaz Khan on 2010-06-15

Changes in Bug.findAttachments():
 * Limited the size of attachments that can be searched to 8 MB
 * Modified searching to go-through attachments line-wise

11013. By Kamran Riaz Khan on 2010-06-15

Fixed unicode searches in Bug.findAttachments()

11012. By Kamran Riaz Khan on 2010-06-15

Added unit tests for Bug.findAttachments() method

=== modified file 'lib/lp/bugs/configure.zcml'
--- lib/lp/bugs/configure.zcml	2010-06-04 09:31:21 +0000
+++ lib/lp/bugs/configure.zcml	2010-06-15 00:23:30 +0000
@@ -644,6 +644,7 @@
                     getBugWatch
                     canBeNominatedFor
                     getNominationFor
+                    findAttachments
                     getNominations
                     date_last_message
                     number_of_duplicates

=== modified file 'lib/lp/bugs/interfaces/bug.py'
--- lib/lp/bugs/interfaces/bug.py	2010-06-10 18:55:22 +0000
+++ lib/lp/bugs/interfaces/bug.py	2010-06-15 00:23:30 +0000
@@ -361,6 +361,14 @@
     latest_patch = Attribute("The most recent patch of this bug.")
 
     @operation_parameters(
+        text=TextLine(title=_("Search text"), default=u""))
+    @operation_returns_collection_of(IBugAttachment)
+    @export_read_operation()
+    def findAttachments(text=""):
+        """Return all attachments related to this bug which match
+        <text>."""
+
+    @operation_parameters(
         subject=optional_message_subject_field(),
         content=copy_field(IMessage['content']))
     @call_with(owner=REQUEST_USER)

=== modified file 'lib/lp/bugs/model/bug.py'
--- lib/lp/bugs/model/bug.py	2010-06-10 18:55:22 +0000
+++ lib/lp/bugs/model/bug.py	2010-06-15 00:23:30 +0000
@@ -812,6 +812,24 @@
             notification.date_emailed = UTC_NOW
             notification.syncUpdate()
 
+    def findAttachments(self, text=u""):
+        """See `IBugAttachment`."""
+        if not isinstance(text, unicode):
+            raise TypeError, "`text` must be a unicode string"
+        if not text:
+            return EmptyResultSet()
+
+        store = IStore(Bug)
+
+        result = store.find(BugAttachment, BugAttachment.bug == self.id)
+        matches = []
+        for attachment in result:
+            data = attachment.data.read()
+            if text in data:
+                matches.append(attachment)
+
+        return matches
+
     def newMessage(self, owner=None, subject=None,
                    content=None, parent=None, bugwatch=None,
                    remote_comment_id=None):

=== added file 'lib/lp/bugs/tests/test_bug_find_attachment.py'
--- lib/lp/bugs/tests/test_bug_find_attachment.py	1970-01-01 00:00:00 +0000
+++ lib/lp/bugs/tests/test_bug_find_attachment.py	2010-06-15 00:23:30 +0000
@@ -0,0 +1,68 @@
+# Copyright 2010 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+__metaclass__ = type
+
+import os
+import transaction
+import unittest
+
+from canonical.config import config
+from canonical.testing import LaunchpadZopelessLayer
+
+from lp.bugs.interfaces.bugjob import BugJobType
+from lp.bugs.model.bugjob import BugJob, BugJobDerived
+from lp.testing import TestCaseWithFactory
+
+from storm.store import EmptyResultSet
+
+from zope.security.proxy import isinstance
+
+class BugFindAttachmentTestCase(TestCaseWithFactory):
+    """Test case for bug attachment searches."""
+
+    layer = LaunchpadZopelessLayer
+
+    def setUp(self):
+        super(BugFindAttachmentTestCase, self).setUp()
+
+        dir = os.path.join(config.root, 'lib/lp/bugs/tests/testfiles')
+        filenames = [
+            'sample-attachment-alsa.diff',
+            'sample-attachment-pidgin.diff',
+        ]
+        self.searches = [
+            u'CONFIG_SND_HDA_POWER_SAVE',
+            u'PURPLE_CONNECTION_ERROR_NETWORK_ERROR',
+        ]
+        filepaths = [
+            os.path.join(dir, filename) for
+            filename in filenames
+        ]
+        person = self.factory.makePerson()
+        self.bug = self.factory.makeBug()
+
+        self.attachments = []
+        for filename, filepath in zip(filenames, filepaths):
+            attachment = self.factory.makeBugAttachment(
+                bug=self.bug, owner=person, data=file(filepath),
+                comment=filename, filename=filename)
+            self.attachments.append(attachment)
+
+        transaction.commit()
+
+    def test_text_type_check(self):
+        self.assertRaises(TypeError, self.bug.findAttachments, 42)
+
+    def test_empty_result_set(self):
+        result = self.bug.findAttachments(u'')
+        self.assertTrue(isinstance(result, EmptyResultSet))
+
+    def test_result_consistency(self):
+        for (search, attachment) in zip(self.searches, self.attachments):
+            result = self.bug.findAttachments(search)
+            self.assertEqual(1, len(result))
+            self.assertEqual(attachment, result[0])
+
+def test_suite():
+    return unittest.TestLoader().loadTestsFromName(__name__)

=== added file 'lib/lp/bugs/tests/testfiles/sample-attachment-alsa.diff'
--- lib/lp/bugs/tests/testfiles/sample-attachment-alsa.diff	1970-01-01 00:00:00 +0000
+++ lib/lp/bugs/tests/testfiles/sample-attachment-alsa.diff	2010-06-15 00:23:30 +0000
@@ -0,0 +1,15 @@
+Signed-off-by: Linuxant Support <[e-mail]>
+
+--- alsa-driver-1.0.16-lnxt/alsa-kernel/pci/hda/hda_codec.h	2008/04/04 19:28:32	1.1
++++ alsa-driver-1.0.16-lnxt/alsa-kernel/pci/hda/hda_codec.h	2008/04/04 19:34:36
+@@ -464,6 +464,10 @@
+ 		       unsigned int verb, unsigned int parm);
+ 	/* get a response from the last command */
+ 	unsigned int (*get_response)(struct hda_codec *codec);
++	/* get the wall clock counter */
++	u32 (*get_wallclock)(struct hda_codec *codec);
++	/* get the link position counter */
++	u32 (*get_linkpos)(struct snd_pcm_substream *substream);
+ 	/* free the private data */
+ 	void (*private_free)(struct hda_bus *);
+ #ifdef CONFIG_SND_HDA_POWER_SAVE

=== added file 'lib/lp/bugs/tests/testfiles/sample-attachment-pidgin.diff'
--- lib/lp/bugs/tests/testfiles/sample-attachment-pidgin.diff	1970-01-01 00:00:00 +0000
+++ lib/lp/bugs/tests/testfiles/sample-attachment-pidgin.diff	2010-06-15 00:23:30 +0000
@@ -0,0 +1,22 @@
+--- libpurple/protocols/jabber/jabber.c	e98b3c84ef06f15c947b3190304799abef8f30b3
++++ libpurple/protocols/jabber/jabber.c	f005b3dede749de2f533f11c1c70a8575f2ef47a
+@@ -430,12 +430,17 @@ jabber_recv_cb_ssl(gpointer data, Purple
+ 			jabber_stream_init(js);
+ 	}
+ 
+-	if(errno == EAGAIN)
++	if(len < 0 && errno == EAGAIN)
+ 		return;
+-	else
++	else {
++		if (len == 0)
++			purple_debug_info("jabber", "Server closed the connection.\n");
++		else
++			purple_debug_info("jabber", "Disconnected: %s\n", g_strerror(errno));
+ 		purple_connection_error_reason (js->gc,
+ 			PURPLE_CONNECTION_ERROR_NETWORK_ERROR,
+ 			_("Read Error"));
++	}
+ }
+ 
+ static void