Merge lp:~benji/launchpad/bug-580035 into lp:launchpad

[Benji York]

Commands can be sent to bugs via GPG signed email. The commands don't contain
a nonce, so if someone were able to get ahold of an email from someone they
could resend it later (bug 580035).

This branch greatly reduces the effective window for such an attack by
rejecting any email message containing commands if the signature was generated
too long ago (more than 48 hours).

To help avoid attacks that arise from people accidentally having their clocks
set too far in the future and thus creating messages that can be reused for a
long time, the branch also rejects emails with commands that were signed more
than 10 minutes in the future.

This approach is suggested fix number 3 in the bug description.

I had a pre-implementation discussion with Gary and he had one with Francis.

The bug email tests had a natural place to add tests for this behavior. To run
the tests:

    bin/test -ct bugs-emailinterface.txt

I fixed a large amount of lint (but I don't think it pollutes the diff too
much). Here's the lint report that I made go away:

Linting changed files:
  lib/canonical/launchpad/mail/handlers.py
  lib/canonical/launchpad/mail/helpers.py
  lib/canonical/launchpad/mail/errortemplates/old-signature.txt
  lib/canonical/launchpad/utilities/gpghandler.py
  lib/lp/bugs/tests/bugs-emailinterface.txt

./lib/canonical/launchpad/mail/handlers.py
     353: W191 indentation contains tabs
     353: E101 indentation contains mixed spaces and tabs
     353: Line contains a tab character.
./lib/canonical/launchpad/mail/helpers.py
      78: E202 whitespace before ']'
     138: E302 expected 2 blank lines, found 1
./lib/canonical/launchpad/utilities/gpghandler.py
      78: E211 whitespace before '('
     288: E202 whitespace before ')'
     470: E202 whitespace before '}'
./lib/lp/bugs/tests/bugs-emailinterface.txt
      79: source exceeds 78 characters.
     341: source exceeds 78 characters.
     575: narrative exceeds 78 characters.
     967: source exceeds 78 characters.
    1226: source exceeds 78 characters.
    1230: source has bad indentation.
    1447: want exceeds 78 characters.
    1473: narrative exceeds 78 characters.
    1489: want exceeds 78 characters.
    1502: want exceeds 78 characters.
    1894: source exceeds 78 characters.
    1895: source exceeds 78 characters.
    2344: narrative has trailing whitespace.
    2583: narrative has trailing whitespace.
    2584: narrative has trailing whitespace.
    2585: narrative has trailing whitespace.
    2696: source exceeds 78 characters.
    2895: want exceeds 78 characters.

[Māris Fogels (mars)]

Hi Benji,

Once again, these changes look good. I really like the use of named variables for time durations in ensure_sane_signature_timestamp(). It really improves the readability.

One question I did have: did you have a pre-implementation call with the bugs team about this? IIRC at the last EPIC gmb was in the process of killing all doctests in bugs. I would think the bugs team members would have asked for unit tests of this new feature.

If it turns out that the bugs team does in fact want unit tests, then I am sure you can land this as-is and land a follow-up branch soon after.

Looks good, r=mars.

Maris

review: Approve

[Deryck Hodge (deryck)]

Benji and I chatted on IRC. I would indeed prefer this be converted to a unit test. I would prefer it before it lands (best laid plans and all that... ;)), but I won't block on that point. Thanks for pointing that out, Maris!

Cheers,
deryck

[Deryck Hodge (deryck)]

The updates to make this a unit test look good to me, Benji. Thanks!

review: Approve (code)

=== added file 'lib/canonical/launchpad/mail/errortemplates/old-signature.txt'
--- lib/canonical/launchpad/mail/errortemplates/old-signature.txt	1970-01-01 00:00:00 +0000
+++ lib/canonical/launchpad/mail/errortemplates/old-signature.txt	2010-08-18 18:27:52 +0000
@@ -0,0 +1,2 @@
+The message you sent included commands to modify the %(context)s, but the
+signature was (apparently) generated too far in the past or future.

=== modified file 'lib/canonical/launchpad/mail/handlers.py'
--- lib/canonical/launchpad/mail/handlers.py	2009-07-24 12:54:07 +0000
+++ lib/canonical/launchpad/mail/handlers.py	2010-08-18 18:27:52 +0000
@@ -27,7 +27,8 @@
     BugEmailCommands, get_error_message)
 from canonical.launchpad.mail.helpers import (
     ensure_not_weakly_authenticated, get_main_body, guess_bugtask,
-    IncomingEmailError, parse_commands, reformat_wiki_text)
+    IncomingEmailError, parse_commands, reformat_wiki_text,
+    ensure_sane_signature_timestamp)
 from lp.services.mail.sendmail import sendmail, simple_sendmail
 from canonical.launchpad.mail.specexploder import get_spec_url_from_moin_mail
 from canonical.launchpad.mailnotification import (
@@ -62,15 +63,19 @@
         commands = self.getCommands(signed_msg)
         user, host = to_addr.split('@')
         add_comment_to_bug = False
+        signature = signed_msg.signature
 
         try:
             if len(commands) > 0:
-                ensure_not_weakly_authenticated(signed_msg, 'bug report')
+                CONTEXT = 'bug report'
+                ensure_not_weakly_authenticated(signed_msg, CONTEXT)
+                if signature is not None:
+                    ensure_sane_signature_timestamp(signature, CONTEXT)
 
             if user.lower() == 'new':
                 # A submit request.
                 commands.insert(0, BugEmailCommands.get('bug', ['new']))
-                if signed_msg.signature is None:
+                if signature is None:
                     raise IncomingEmailError(
                         get_error_message('not-gpg-signed.txt'))
             elif user.isdigit():
@@ -345,7 +350,7 @@
         """
         if question.status in [
             QuestionStatus.OPEN, QuestionStatus.NEEDSINFO,
-	    QuestionStatus.ANSWERED]:
+        QuestionStatus.ANSWERED]:
             question.giveAnswer(message.owner, message)
         else:
             # In the other states, only a comment can be added.

=== modified file 'lib/canonical/launchpad/mail/helpers.py'
--- lib/canonical/launchpad/mail/helpers.py	2010-08-16 13:50:28 +0000
+++ lib/canonical/launchpad/mail/helpers.py	2010-08-18 18:27:52 +0000
@@ -5,6 +5,7 @@
 
 import os.path
 import re
+import time
 
 from zope.component import getUtility
 
@@ -74,7 +75,9 @@
         <...IDistroSeriesBugTask>
     """
     bugtask_interfaces = [
-        IUpstreamBugTask, IDistroBugTask, IDistroSeriesBugTask
+        IUpstreamBugTask,
+        IDistroBugTask,
+        IDistroSeriesBugTask,
         ]
     for interface in bugtask_interfaces:
         if interface.providedBy(bugtask):
@@ -134,6 +137,7 @@
 
     return text
 
+
 def parse_commands(content, command_names):
     """Extract indented commands from email body.
 
@@ -220,3 +224,18 @@
                 no_key_template, import_url=import_url,
                 context=context)
         raise IncomingEmailError(error_message)
+
+
+def ensure_sane_signature_timestamp(signature, context,
+                                    error_template='old-signature.txt'):
+    """Ensure the signature was generated recently but not in the future."""
+    fourty_eight_hours = 48 * 60 * 60
+    ten_minutes = 10 * 60
+    now = time.time()
+    fourty_eight_hours_ago = now - fourty_eight_hours
+    ten_minutes_in_the_future = now + ten_minutes
+
+    if (signature.timestamp < fourty_eight_hours_ago
+            or signature.timestamp > ten_minutes_in_the_future):
+        error_message = get_error_message(error_template, context=context)
+        raise IncomingEmailError(error_message)

=== modified file 'lib/canonical/launchpad/mail/tests/test_handlers.py'
--- lib/canonical/launchpad/mail/tests/test_handlers.py	2010-07-14 14:11:15 +0000
+++ lib/canonical/launchpad/mail/tests/test_handlers.py	2010-08-18 18:27:52 +0000
@@ -3,14 +3,18 @@
 
 __metaclass__ = type
 
+import email
+import time
 import unittest
 
 from doctest import DocTestSuite
 
+from canonical.database.sqlbase import commit
 from canonical.launchpad.mail.commands import BugEmailCommand
 from canonical.launchpad.mail.handlers import MaloneHandler
-from lp.testing import TestCaseWithFactory
 from canonical.testing import LaunchpadFunctionalLayer
+from lp.services.mail import stub
+from lp.testing import TestCaseWithFactory, person_logged_in
 
 
 class TestMaloneHandler(TestCaseWithFactory):
@@ -35,6 +39,88 @@
         self.assertEqual(['foo'], commands[0].string_args)
 
 
+class FakeSignature:
+
+    def __init__(self, timestamp):
+        self.timestamp = timestamp
+
+
+def get_last_email():
+    from_addr, to_addrs, raw_message = stub.test_emails[-1]
+    sent_msg = email.message_from_string(raw_message)
+    error_mail, original_mail = sent_msg.get_payload()
+    # clear the emails so we don't accidentally get one from a previous test
+    return dict(
+        subject=sent_msg['Subject'],
+        body=error_mail.get_payload(decode=True))
+
+
+BAD_SIGNATURE_TIMESTAMP_MESSAGE = (
+    'The message you sent included commands to modify the bug '
+    'report, but the\nsignature was (apparently) generated too far '
+    'in the past or future.')
+
+
+class TestSignatureTimestampValidation(TestCaseWithFactory):
+    """GPG signature timestamps are checked for emails containing commands."""
+
+    layer = LaunchpadFunctionalLayer
+
+    def test_far_past_gpg_signature_timestamp(self):
+        # If an email message's GPG signature's timestamp is too far in the
+        # past and the message contains a command, the command will fail and
+        # send the user an email telling them what happened.
+
+        msg = self.factory.makeSignedMessage(body=' security no')
+        now = time.time()
+        one_week = 60 * 60 * 24 * 7
+        msg.signature = FakeSignature(timestamp=now-one_week)
+        handler = MaloneHandler()
+        with person_logged_in(self.factory.makePerson()):
+            success = handler.process(msg, msg['To'])
+        commit()
+        email = get_last_email()
+        self.assertEqual(email['subject'], 'Submit Request Failure')
+        self.assertIn(BAD_SIGNATURE_TIMESTAMP_MESSAGE, email['body'])
+
+    def test_far_future_gpg_signature_timestamp(self):
+        # If an email message's GPG signature's timestamp is too far in the
+        # future and the message contains a command, the command will fail and
+        # send the user an email telling them what happened.
+
+        msg = self.factory.makeSignedMessage(body=' security no')
+        now = time.time()
+        one_week = 60 * 60 * 24 * 7
+        msg.signature = FakeSignature(timestamp=now+one_week)
+        handler = MaloneHandler()
+        with person_logged_in(self.factory.makePerson()):
+            success = handler.process(msg, msg['To'])
+        commit()
+        email = get_last_email()
+        self.assertEqual(email['subject'], 'Submit Request Failure')
+        self.assertIn(BAD_SIGNATURE_TIMESTAMP_MESSAGE, email['body'])
+
+    def test_bad_timestamp_but_no_commands(self):
+        # If an email message's GPG signature's timestamp is too far in the
+        # future or past but it doesn't contain any commands, the email is
+        # processed anyway.
+
+        msg = self.factory.makeSignedMessage(
+            body='I really hope this bug gets fixed.')
+        now = time.time()
+        one_week = 60 * 60 * 24 * 7
+        msg.signature = FakeSignature(timestamp=now+one_week)
+        handler = MaloneHandler()
+        # Clear old emails before potentially generating more.
+        del stub.test_emails[:]
+        with person_logged_in(self.factory.makePerson()):
+            success = handler.process(msg, msg['To'])
+        commit()
+        # Since there were no commands in the poorly-timestamped message, no
+        # error emails were generated.
+        self.assertEqual(stub.test_emails, [])
+
+
 def test_suite():
     suite = unittest.TestSuite()
     suite.addTests(DocTestSuite('canonical.launchpad.mail.handlers'))

=== modified file 'lib/canonical/launchpad/mail/tests/test_helpers.py'
--- lib/canonical/launchpad/mail/tests/test_helpers.py	2010-07-14 14:11:15 +0000
+++ lib/canonical/launchpad/mail/tests/test_helpers.py	2010-08-18 18:27:52 +0000
@@ -4,6 +4,7 @@
 __metaclass__ = type
 
 from doctest import DocTestSuite
+import time
 import unittest
 
 from zope.interface import directlyProvides, directlyProvidedBy
@@ -12,7 +13,7 @@
     EmailProcessingError, IWeaklyAuthenticatedPrincipal)
 from canonical.launchpad.mail.helpers import (
     ensure_not_weakly_authenticated, get_person_or_team,
-    IncomingEmailError, parse_commands)
+    IncomingEmailError, parse_commands, ensure_sane_signature_timestamp)
 from lp.testing import login_person, TestCase, TestCaseWithFactory
 from canonical.testing import DatabaseFunctionalLayer
 from canonical.launchpad.webapp.interaction import get_current_principal
@@ -77,6 +78,50 @@
             parse_commands(' command:', ['command']))
 
 
+class FakeSignature:
+
+    def __init__(self, timestamp):
+        self.timestamp = timestamp
+
+
+class TestEnsureSaneSignatureTimestamp(unittest.TestCase):
+    """Tests for ensure_sane_signature_timestamp"""
+
+    def test_too_old_timestamp(self):
+        # signature timestamps shouldn't be too old
+        now = time.time()
+        one_week = 60 * 60 * 24 * 7
+        signature = FakeSignature(timestamp=now-one_week)
+        self.assertRaises(
+            IncomingEmailError, ensure_sane_signature_timestamp,
+            signature, 'bug report')
+
+    def test_future_timestamp(self):
+        # signature timestamps shouldn't be (far) in the future
+        now = time.time()
+        one_week = 60 * 60 * 24 * 7
+        signature = FakeSignature(timestamp=now+one_week)
+        self.assertRaises(
+            IncomingEmailError, ensure_sane_signature_timestamp,
+            signature, 'bug report')
+
+    def test_near_future_timestamp(self):
+        # signature timestamps in the near future are OK
+        now = time.time()
+        one_minute = 60
+        signature = FakeSignature(timestamp=now+one_minute)
+        # this should not raise an exception
+        ensure_sane_signature_timestamp(signature, 'bug report')
+
+    def test_recent_timestamp(self):
+        # signature timestamps in the recent past are OK
+        now = time.time()
+        one_hour = 60 * 60
+        signature = FakeSignature(timestamp=now-one_hour)
+        # this should not raise an exception
+        ensure_sane_signature_timestamp(signature, 'bug report')
+
+
 class TestEnsureNotWeaklyAuthenticated(TestCaseWithFactory):
     """Test the ensure_not_weakly_authenticated function."""
 

=== modified file 'lib/canonical/launchpad/utilities/gpghandler.py'
--- lib/canonical/launchpad/utilities/gpghandler.py	2010-08-10 16:17:12 +0000
+++ lib/canonical/launchpad/utilities/gpghandler.py	2010-08-18 18:27:52 +0000
@@ -75,9 +75,9 @@
         # automatically retrieve from the keyserver unknown key when
         # verifying signatures and 'no-auto-check-trustdb' avoid wasting
         # time verifying the local keyring consistence.
-        conf.write ('keyserver hkp://%s\n'
-                    'keyserver-options auto-key-retrieve\n'
-                    'no-auto-check-trustdb\n' % config.gpghandler.host)
+        conf.write('keyserver hkp://%s\n'
+                   'keyserver-options auto-key-retrieve\n'
+                   'no-auto-check-trustdb\n' % config.gpghandler.host)
         conf.close()
         # create a local atexit handler to remove the configuration directory
         # on normal termination.
@@ -156,35 +156,26 @@
             sig = StringIO(signature)
             # store the content
             plain = StringIO(content)
-            # process it
-            try:
-                signatures = ctx.verify(sig, plain, None)
-            except gpgme.GpgmeError, e:
-                # XXX: 2010-04-26, Salgado, bug=570244: This hack is needed
-                # for python2.5 compatibility. We should remove it when we no
-                # longer need to run on python2.5.
-                if hasattr(e, 'strerror'):
-                    msg = e.strerror
-                else:
-                    msg = e.message
-                raise GPGVerificationError(msg)
+            args = (sig, plain, None)
         else:
             # store clearsigned signature
             sig = StringIO(content)
             # writeable content
             plain = StringIO()
-            # process it
-            try:
-                signatures = ctx.verify(sig, None, plain)
-            except gpgme.GpgmeError, e:
-                # XXX: 2010-04-26, Salgado, bug=570244: This hack is needed
-                # for python2.5 compatibility. We should remove it when we no
-                # longer need to run on python2.5.
-                if hasattr(e, 'strerror'):
-                    msg = e.strerror
-                else:
-                    msg = e.message
-                raise GPGVerificationError(msg)
+            args = (sig, None, plain)
+
+        # process it
+        try:
+            signatures = ctx.verify(*args)
+        except gpgme.GpgmeError, e:
+            # XXX: 2010-04-26, Salgado, bug=570244: This hack is needed
+            # for python2.5 compatibility. We should remove it when we no
+            # longer need to run on python2.5.
+            if hasattr(e, 'strerror'):
+                msg = e.strerror
+            else:
+                msg = e.message
+            raise GPGVerificationError(msg)
 
         # XXX jamesh 2006-01-31:
         # We raise an exception if we don't get exactly one signature.
@@ -204,7 +195,6 @@
                                        'found multiple signatures')
 
         signature = signatures[0]
-
         # signature.status == 0 means "Ok"
         if signature.status is not None:
             raise GPGVerificationError(signature.status.args)
@@ -295,8 +285,7 @@
         # See more information at:
         # http://pyme.sourceforge.net/doc/gpgme/Generating-Keys.html
         result = context.genkey(
-            signing_only_param % {'name': name.encode('utf-8')}
-            )
+            signing_only_param % {'name': name.encode('utf-8')})
 
         # Right, it might seem paranoid to have this many assertions,
         # but we have to take key generation very seriously.
@@ -477,8 +466,8 @@
         """See IGPGHandler"""
         params = {
             'search': '0x%s' % fingerprint,
-            'op': action
-        }
+            'op': action,
+            }
         if public:
             host = config.gpghandler.public_host
         else:

=== modified file 'lib/lp/bugs/tests/bugs-emailinterface.txt'
--- lib/lp/bugs/tests/bugs-emailinterface.txt	2010-08-02 17:48:13 +0000
+++ lib/lp/bugs/tests/bugs-emailinterface.txt	2010-08-18 18:27:52 +0000
@@ -1,11 +1,13 @@
-= Launchpad Bugs e-mail interface =
+Launchpad Bugs e-mail interface
+===============================
 
 Launchpad's bugtracker has an e-mail interface, with which you may report new
 bugs, add comments, and change the details of existing bug reports. Commands
 can be interleaved within a comment, so to distinguish them from the comment,
 they must be indented with at least one space or tab character.
 
-== Submit a new bug ==
+Submit a new bug
+----------------
 
 To report a bug, you send an OpenPGP-signed e-mail message to
 [e-mail]-domain. You must have registered your key in
@@ -48,12 +50,19 @@
 signed, so that the system can verify the sender. But to avoid having
 to sign each email, we'll create a class which fakes a signed email:
 
-    >>> import email.Utils
+    >>> import time
+    >>> class MockSignature(object):
+    ...     def __init__(self):
+    ...         self.timestamp = time.time()
+
+    >>> import email.Message
     >>> class MockSignedMessage(email.Message.Message):
+    ...     def __init__(self, *args, **kws):
+    ...         email.Message.Message.__init__(self, *args, **kws)
+    ...         self.signature = MockSignature()
     ...     @property
     ...     def signedMessage(self):
     ...         return self
-    ...     signature = object()
 
 And since we'll pass the email directly to the correct handler,
 we'll have to authenticate the user manually:
@@ -66,10 +75,15 @@
 
     >>> from canonical.launchpad.mail.handlers import MaloneHandler
     >>> handler = MaloneHandler()
-    >>> def process_email(raw_mail):
-    ...     msg = email.message_from_string(raw_mail, _class=MockSignedMessage)
+    >>> def construct_email(raw_mail):
+    ...     msg = email.message_from_string(
+    ...         raw_mail, _class=MockSignedMessage)
     ...     if not msg.has_key('Message-Id'):
     ...         msg['Message-Id'] = factory.makeUniqueRFC822MsgId()
+    ...     return msg
+
+    >>> def process_email(raw_mail):
+    ...     msg = construct_email(raw_mail)
     ...     handler.process(msg, msg['To'])
 
     >>> process_email(submit_mail)
@@ -224,7 +238,8 @@
     u'A folded email subject'
 
 
-== Add a comment ==
+Add a comment
+-------------
 
 After a bug has been submitted a notification is sent out. The reply-to
 address is set to the bug address, $bugid@malone-domain. We can send
@@ -263,7 +278,8 @@
     True
 
 
-== Edit bugs ==
+Edit bugs
+---------
 
 Sometimes you may want to simply edit a bug, without adding a comment.
 For that you can send mails to edit@malone-domain.
@@ -304,7 +320,8 @@
     Nicer summary
 
 
-== GPG signing and adding comments ==
+GPG signing and adding comments
+-------------------------------
 
 In order to include commands in the comment, the email has to be GPG
 signed. The key used to sign the email has to be associated with the
@@ -322,7 +339,8 @@
 will provide IWeaklyAuthenticatedPrincipal. Let's mark the current
 principal with that.
 
-    >>> from canonical.launchpad.interfaces import IWeaklyAuthenticatedPrincipal
+    >>> from canonical.launchpad.interfaces import (
+    ...     IWeaklyAuthenticatedPrincipal)
     >>> from zope.interface import directlyProvides, directlyProvidedBy
     >>> from zope.security.management import queryInteraction
     >>> participations = queryInteraction().participations
@@ -446,13 +464,14 @@
     ...     provided_interfaces - IWeaklyAuthenticatedPrincipal)
 
 
-== Commands == 
+Commands
+--------
 
 Now let's take a closer look at all the commands that are available for
 us to play with. First we define a function to easily submit commands
 to edit bug 4:
 
-    >>> def submit_commands(bug, *commands):
+    >>> def construct_command_email(bug, *commands):
     ...     edit_mail = ("From: [e-mail]\n"
     ...                  "To: edit@malone-domain\n"
     ...                  "Date: Fri Jun 17 10:10:23 BST 2005\n"
@@ -460,12 +479,20 @@
     ...                  "\n"
     ...                  " bug %d\n" % bug.id)
     ...     edit_mail += ' ' + '\n '.join(commands)
-    ...     process_email(edit_mail)
+    ...     return construct_email(edit_mail)
+
+    >>> def submit_command_email(msg):
+    ...     handler.process(msg, msg['To'])
     ...     commit()
     ...     sync(bug)
 
-
-=== bug $bugid ===
+    >>> def submit_commands(bug, *commands):
+    ...     msg = construct_command_email(bug, *commands)
+    ...     submit_command_email(msg)
+
+
+bug $bugid
+~~~~~~~~~~
 
 Switches what bug you want to edit. Example:
 
@@ -509,7 +536,8 @@
     ...
 
 
-=== summary "$summary" ===
+summary "$summary"
+~~~~~~~~~~~~~~~~~~
 
 Changes the summary of the bug. The title has to be enclosed in
 quotes. Example:
@@ -541,12 +569,13 @@
     ...
 
 
-=== private yes|no ===
+private yes|no
+~~~~~~~~~~~~~~
 
 Changes the visibility of the bug. Example:
 
-(We'll subscribe Sample Person to this bug before marking it private, otherwise
-permission to complete the operation will be denied.)
+(We'll subscribe Sample Person to this bug before marking it private,
+otherwise permission to complete the operation will be denied.)
 
     >>> subscription = bug_four.subscribe(bug_four.owner, bug_four.owner)
 
@@ -598,7 +627,8 @@
     ...
 
 
-=== security yes|no ===
+security yes|no
+~~~~~~~~~~~~~~~
 
 Changes the security flag of the bug. Example:
 
@@ -647,7 +677,8 @@
         security yes
     ...
 
-=== subscribe [$name|$email] ===
+subscribe [$name|$email]
+~~~~~~~~~~~~~~~~~~~~~~~~
 
 Subscribes yourself or someone else to the bug. All arguments are
 optional. If you don't specify a name, the sender of the email will
@@ -688,7 +719,8 @@
     [e-mail]
     ...
 
-=== unsubscribe [$name|$email] ===
+unsubscribe [$name|$email]
+~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Unsubscribes yourself or someone else from the bug.  If you don't
 specify a name or email, the sender of the email will be
@@ -782,7 +814,8 @@
     >>> submit_commands(bug_four, 'subscribe [e-mail]')
 
 
-=== tag $tag ===
+tag $tag
+~~~~~~~~
 
 The 'tag' command assigns a tag to a bug. Using this command we will add the
 tags foo and bar to the bug. Adding a single tag multiple times should
@@ -865,7 +898,8 @@
     with-hyphen+period.
 
 
-=== duplicate $bug_id ===
+duplicate $bug_id
+~~~~~~~~~~~~~~~~~
 
 The 'duplicate' command marks a bug as a duplicate of another bug.
 
@@ -927,11 +961,13 @@
     ...
 
 
-=== cve $cve ===
+cve $cve
+~~~~~~~~
 
 The 'cve' command associates a bug with a CVE reference.
 
-    >>> from canonical.launchpad.interfaces import CreateBugParams, IProductSet
+    >>> from canonical.launchpad.interfaces import (CreateBugParams,
+    ...     IProductSet)
     >>> def new_firefox_bug():
     ...     firefox = getUtility(IProductSet).getByName('firefox')
     ...     return firefox.createBug(CreateBugParams(
@@ -962,7 +998,8 @@
     Launchpad can't find the CVE "no-such-cve".
     ...
 
-=== affects, assignee, status, importance, milestone ===
+affects, assignee, status, importance, milestone
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 affects $path [assignee $name|$email|nobody]
               [status $status]
@@ -1189,11 +1226,12 @@
     >>> LaunchpadZopelessLayer.switchDbUser('launchpad')
     >>> login('[e-mail]')
     >>> ubuntu = getUtility(IDistributionSet).getByName('ubuntu')
-    >>> ubuntu.driver = getUtility(IPersonSet).getByEmail('[e-mail]')
+    >>> ubuntu.driver = getUtility(IPersonSet).getByEmail(
+    ...     '[e-mail]')
     >>> commit()
     >>> LaunchpadZopelessLayer.switchDbUser(config.processmail.dbuser)
     >>> login('[e-mail]')
-     >>> sync(bug)
+    >>> sync(bug)
 
 Now a new bugtask for the series will be create directly.
 
@@ -1328,7 +1366,8 @@
     Sample Person
 
 
-=== Restricted bug statuses ===
+Restricted bug statuses
+~~~~~~~~~~~~~~~~~~~~~~~
 
     >>> email_user = getUtility(ILaunchBag).user
 
@@ -1409,7 +1448,8 @@
         status foo
     ...
     The 'status' command expects any of the following arguments:
-    new, incomplete, opinion, invalid, wontfix, expired, confirmed, triaged, inprogress, fixcommitted, fixreleased
+    new, incomplete, opinion, invalid, wontfix, expired, confirmed, triaged,
+    inprogress, fixcommitted, fixreleased
     <BLANKLINE>
     For example:
     <BLANKLINE>
@@ -1435,8 +1475,8 @@
         importance critical
     ...
 
-XXX mpt 20060516: "importance undecided" is a silly example, but customizing it
-to a realistic value is difficult (see convertArguments in
+XXX mpt 20060516: "importance undecided" is a silly example, but customizing
+it to a realistic value is difficult (see convertArguments in
 launchpad/mail/commands.py).
 
 Trying to use the obsolete "severity" or "priority" commands:
@@ -1451,8 +1491,8 @@
     Failing command:
         severity major
     ...
-    To make life a little simpler, Malone no longer has "priority" and "severity"
-    fields. There is now an "importance" field...
+    To make life a little simpler, Malone no longer has "priority" and
+    "severity" fields. There is now an "importance" field...
     ...
 
     >>> submit_commands(bug_four, 'affects firefox', 'priority low')
@@ -1464,8 +1504,8 @@
     Failing command:
         priority low
     ...
-    To make life a little simpler, Malone no longer has "priority" and "severity"
-    fields. There is now an "importance" field...
+    To make life a little simpler, Malone no longer has "priority" and
+    "severity" fields. There is now an "importance" field...
     ...
 
 Invalid assignee:
@@ -1486,7 +1526,8 @@
     >>> stub.test_emails = []
 
 
-== Multiple Commands ==
+Multiple Commands
+-----------------
 
 An email can contain multiple commands, even for different bugs.
 
@@ -1541,7 +1582,8 @@
     >>> bugtask_created_listener.unregister()
 
 
-== Default 'affects' target ==
+Default 'affects' target
+------------------------
 
 Most of the time it's not necessary to give the 'affects' command. If
 you omit it, the email interface  tries to guess which bug task you
@@ -1571,7 +1613,8 @@
 the user wanted to edit. We apply the following heuristics for choosing
 which bug task to edit:
 
-=== The user is a bug supervisors of the upstream product ===
+The user is a bug supervisors of the upstream product
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
     >>> login('[e-mail]')
     >>> bug_one = getUtility(IBugSet).get(1)
@@ -1598,7 +1641,8 @@
            Status: New => Confirmed...
 
 
-=== The user is a package bug supervisor ===
+The user is a package bug supervisor
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
     >>> from canonical.launchpad.interfaces import (
     ...     IDistributionSet, ISourcePackageNameSet)
@@ -1638,12 +1682,14 @@
     ** Changed in: mozilla-firefox (Ubuntu)
            Status: New => Confirmed
 
-=== The user is a bug supervisor of a distribution ===
+The user is a bug supervisor of a distribution
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 XXX: TBD after InitialBugContacts is implemented.
      -- Bjorn Tillenius, 2005-11-30
 
-=== The user is a distribution member ===
+The user is a distribution member
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
     >>> login('[e-mail]')
     >>> submit_commands(
@@ -1668,7 +1714,8 @@
            Status: Confirmed => New
 
 
-=== No matching bug task ===
+No matching bug task
+~~~~~~~~~~~~~~~~~~~~
 
 If none of the bug tasks can be chosen, an error message is sent to the
 user, telling him that he has to use the 'affects' command.
@@ -1696,7 +1743,8 @@
     ...
 
 
-== More About Error Handling ==
+More About Error Handling
+-------------------------
 
 If an error is encountered, an email is sent to the sender informing
 him about the error. Let's start with trying to submit a bug without
@@ -1706,6 +1754,7 @@
 
     >>> from canonical.launchpad.mail import signed_message_from_string
     >>> msg = signed_message_from_string(submit_mail)
+    >>> import email.Utils
     >>> msg['Message-Id'] = email.Utils.make_msgid()
     >>> handler.process(msg, msg['To'])
     True
@@ -1788,9 +1837,10 @@
     ... Subject: A bug with no affects
     ...
     ... I'm abusing ltsp-build-client to build a diskless fat client, but dint
-    ... of --late-packages ubuntu-desktop. The dpkg --configure step for eg. HAL
-    ... will try to start the daemon and failing, due to the lack of /proc. This
-    ... is just the tip of the iceberg; I'll file more bugs as I go along.
+    ... of --late-packages ubuntu-desktop. The dpkg --configure step for eg.
+    ... HAL will try to start the daemon and failing, due to the lack of
+    ... /proc.  This is just the tip of the iceberg; I'll file more bugs as I
+    ... go along.
     ... """
 
     >>> process_email(submit_mail)
@@ -1982,7 +2032,8 @@
     Original message body.
 
 
-== Error handling ==
+Error handling
+--------------
 
 When creating a new task and assigning it to a team, it is possible
 that the team will not have a contact address. This is not generally
@@ -2023,7 +2074,8 @@
     landscape-developers
 
 
-== Recovering from errors ==
+Recovering from errors
+----------------------
 
 When a user sends an email with multiple commands, some of them might
 fail (because of bad arguments, for example). Some commands, namely
@@ -2139,7 +2191,8 @@
     or send an email to [e-mail]
 
 
-== Terminating command input ==
+Terminating command input
+-------------------------
 
 To make it possible to submit emails with lines that look like commands
 (but aren't), a 'done' statement is provided. When the email parser
@@ -2176,7 +2229,8 @@
     CONFIRMED
 
 
-== Requesting help ==
+Requesting help
+---------------
 
 It's possible to ask for the help document for the email interface via
 email too. Just send an email to `[e-mail]`.
@@ -2231,10 +2285,11 @@
     See you in {{{#launchpad}}}.
 
 
-== Email attachments ==
+Email attachments
+-----------------
 
-Email attachments are stored as bug attachments (provided that they 
-match the criteria described below).
+Email attachments are stored as bug attachments (provided that they match the
+criteria described below).
 
     >>> def print_attachments(attachments):
     ...     if attachments.count() == 0:
@@ -2479,9 +2534,9 @@
     >>> process_email(submit_mail)
     >>> print_attachments(get_latest_added_bug().attachments)
     No attachments
- 
-If an attachment has one of the content types application/applefile 
-(the resource fork of a MacOS file), application/pgp-signature, 
+
+If an attachment has one of the content types application/applefile
+(the resource fork of a MacOS file), application/pgp-signature,
 application/pkcs7-signature, application/x-pkcs7-signature,
 text/x-vcard, application/ms-tnef, it is not stored.
 
@@ -2504,7 +2559,7 @@
     ...
     ... -----BEGIN PGP SIGNATURE-----
     ... Version: GnuPG v1.4.6 (GNU/Linux)
-    ... 
+    ...
     ... 123eetsdtdgdg43e4
     ... -----END PGP SIGNATURE-----
     ... --BOUNDARY
@@ -2550,12 +2605,12 @@
     ...  affects firefox
     ... --BOUNDARY
     ... Content-type: multipart/appledouble; boundary="SUBBOUNDARY"
-    ... 
+    ...
     ... --SUBBOUNDARY
     ... Content-type: application/applefile
     ... Content-disposition: attachment; filename="sampledata"
     ... Content-tranfer-encoding: 7bit
-    ... 
+    ...
     ... qwert
     ... --SUBBOUNDARY
     ... Content-type: text/plain
@@ -2595,7 +2650,8 @@
     ... --BOUNDARY"""
     >>>
     >>> process_email(submit_mail)
-    >>> new_message = getUtility(IMessageSet).get('comment-with-attachment')[0]
+    >>> new_message = getUtility(IMessageSet).get(
+    ...     'comment-with-attachment')[0]
     >>> new_message in bug_one.messages
     True
     >>> print_attachments(new_message.bugattachments)
@@ -2741,7 +2797,7 @@
     ... Content-Transfer-Encoding: base64
     ... X-Attachment-Id: f_fcuhv1fz0
     ... Content-Disposition: attachment; filename=image.jpg
-    ... 
+    ...
     ... dGhpcyBpcyBub3QgYSByZWFsIEpQRyBmaWxl==
     ... --BOUNDARY"""
     >>>
@@ -2773,7 +2829,7 @@
     ... Content-Transfer-Encoding: base64
     ... X-Attachment-Id: f_fcuhv1fz0
     ... Content-Disposition: attachment
-    ... 
+    ...
     ... dGhpcyBpcyBub3QgYSByZWFsIEpQRyBmaWxl==
     ... --BOUNDARY
     ... Content-type: text/x-diff; name="sourcefile1.diff"
@@ -2799,7 +2855,7 @@
     >>> print_attachments(get_latest_added_bug().attachments)
     LibraryFileAlias ... image/jpeg; name="image.jpg" UNSPECIFIED
     this is not a real JPG file
-    LibraryFileAlias sourcefile.diff text/x-diff; name="sourcefile1.diff" PATCH
+    LibraryFileAlias ... text/x-diff; name="sourcefile1.diff" PATCH
     this should be diff output.
 
 
@@ -2807,7 +2863,8 @@
      -- Bjorn Tillenius, 2005-05-20
 
 
-== Reply to a comment on a remote bug ==
+Reply to a comment on a remote bug
+----------------------------------
 
 If someone uses the email interface to reply to a comment which was
 imported into Launchpad from a remote bugtracker their reply will be