Merge lp:~sinzui/launchpad/discard-list-spam-0 into lp:launchpad

[Curtis Hovey]

This is my branch to move mailman from canonical.launchpad to lp.services

    lp:~sinzui/launchpad/discard-list-spam-0
    Diff size: 417
    Launchpad bug:
          https://bugs.launchpad.net/bugs/641526
    Test command: ./bin/test -vv --layer=Mailman (may need to run 3 times)
        ./bin/test -vv lp.services.mailman
    Pre-implementation: no one
    Target release: 10.10

Automatically discard messages that claim to be from the list
-------------------------------------------------------------

The mailman config and monekypatch rules can be moved to lp.services.mailman,
which is the only package that uses them. The monkeypatch tests are already
in lp.services.mailman.

This should be done before any monkeypatch work is needed.

Rules
-----

    * Move the remaining code into lp.services/mailman
    ADDENDUM
    * Fix the logging test because the LaunchapdErrorUtility changed in
      July
    * Fix the staging text because the cached property is not needed.

QA
--

    * On staging verify mail arrives to the archive
    * On staging verify mail goes in to the moderation queue.

Lint
----

Linting changed files:
  buildmailman.py
  lib/canonical/launchpad/scripts/runlaunchpad.py
  lib/lp/scripts/utilities/killservice.py
  lib/lp/services/mailman/runmailman.py
  lib/lp/services/mailman/config/tests/test_config.py
  lib/lp/services/mailman/doc/basic-integration.txt
  lib/lp/services/mailman/doc/batching.txt
  lib/lp/services/mailman/doc/logging.txt
  lib/lp/services/mailman/doc/staging.txt
  lib/lp/services/mailman/monkeypatches/__init__.py
  lib/lp/services/mailman/monkeypatches/mm_cfg.py.in
  lib/lp/services/mailman/testing/layers.py
  lib/lp/services/mailman/testing/withlist_1.py
  lib/lp/services/mailman/testing/withlist_2.py
  scripts/mlist-sync.py

Lint reports some issue in some modules about the imports, but the code
is correct for how we are building mailman

Test
----

Run the tests many times to see them pass. The layer is fatally plagues
by timeouts--the entire testing strategy needs to be rethought.

Implementation
--------------

No code changes.

[Brad Crittenden (bac)]

review: Approve (code)

=== modified file 'buildmailman.py'
--- buildmailman.py	2010-04-27 19:48:39 +0000
+++ buildmailman.py	2010-09-18 03:14:43 +0000
@@ -13,9 +13,9 @@
 import subprocess
 
 from canonical.config import config
-from canonical.launchpad.mailman.config import (
+from lp.services.mailman.config import (
     configure_prefix, configure_siteowner)
-from canonical.launchpad.mailman.monkeypatches import monkey_patch
+from lp.services.mailman.monkeypatches import monkey_patch
 from lazr.config import as_username_groupname
 
 basepath = [part for part in sys.path if part]
@@ -42,9 +42,9 @@
         return 0
 
     # sys.path_importer_cache is a mapping of elements of sys.path to importer
-    # objects used to handle them. In Python2.5+ when an element of sys.path is
-    # found to not exist on disk, a NullImporter is created and cached - this
-    # causes Python to never bother re-inspecting the disk for that path
+    # objects used to handle them. In Python2.5+ when an element of sys.path
+    # is found to not exist on disk, a NullImporter is created and cached -
+    # this causes Python to never bother re-inspecting the disk for that path
     # element. We must clear that cache element so that our second attempt to
     # import MailMan after building it will actually check the disk.
     del sys.path_importer_cache[mailman_path]
@@ -102,7 +102,7 @@
         print >> sys.stderr, 'Could not configure Mailman:'
         sys.exit(retcode)
     # Make.
-    retcode = subprocess.call(('make',), cwd=mailman_source)
+    retcode = subprocess.call(('make', ), cwd=mailman_source)
     if retcode:
         print >> sys.stderr, 'Could not make Mailman.'
         sys.exit(retcode)
@@ -116,8 +116,8 @@
     # installation will call Mailman's ``bin/update``, which is a script
     # that needs this fix.
     build_dir = os.path.join(mailman_source, 'build')
-    original = '#! %s\n' % (executable,)
-    modified = '#! /usr/bin/env %s\n' % (executable,)
+    original = '#! %s\n' % (executable, )
+    modified = '#! /usr/bin/env %s\n' % (executable, )
     for (dirpath, dirnames, filenames) in os.walk(build_dir):
         for filename in filenames:
             filename = os.path.join(dirpath, filename)
@@ -191,7 +191,8 @@
 def configure_site_list(mailman_bin, site_list_name):
     """Configure the site list.
 
-    Currently, the only thing we want to set is to not advertise the site list.
+    Currently, the only thing we want to set is to not advertise the
+    site list.
     """
     fd, config_file_name = tempfile.mkstemp()
     try:
@@ -219,7 +220,6 @@
     return build_mailman()
 
 
-
 if __name__ == '__main__':
     return_code = main()
     sys.exit(return_code)

=== removed directory 'lib/canonical/launchpad/mailman'
=== removed file 'lib/canonical/launchpad/mailman/__init__.py'
=== removed directory 'lib/canonical/launchpad/mailman/doc'
=== modified file 'lib/canonical/launchpad/scripts/runlaunchpad.py'
--- lib/canonical/launchpad/scripts/runlaunchpad.py	2010-08-20 20:31:18 +0000
+++ lib/canonical/launchpad/scripts/runlaunchpad.py	2010-09-18 03:14:43 +0000
@@ -16,7 +16,7 @@
 from zope.app.server.main import main
 
 from canonical.config import config
-from canonical.launchpad.mailman import runmailman
+from lp.services.mailman import runmailman
 from canonical.launchpad.testing import googletestservice
 from canonical.lazr.pidfile import (
     make_pidfile,
@@ -32,6 +32,7 @@
 
 
 class Service(object):
+
     @property
     def should_launch(self):
         """Return true if this service should be launched."""
@@ -52,7 +53,8 @@
             script.
         :param section_name: The config section name that provides the
             launch, logfile and spew options.
-        :param pre_launch: A callable that is called before the launch process.
+        :param pre_launch: A callable that is called before the launch
+            process.
         """
         # No point calling super's __init__.
         # pylint: disable-msg=W0231
@@ -119,6 +121,7 @@
 
 
 class MailmanService(Service):
+
     @property
     def should_launch(self):
         return config.mailman.launch
@@ -132,6 +135,7 @@
 
 
 class CodebrowseService(Service):
+
     @property
     def should_launch(self):
         return False
@@ -158,11 +162,11 @@
 
 class MemcachedService(Service):
     """A local memcached service for developer environments."""
+
     @property
     def should_launch(self):
         return config.memcached.launch
 
-
     def launch(self):
         cmd = [
             'memcached',
@@ -187,6 +191,7 @@
 
     :param process: An instance of subprocess.Popen.
     """
+
     def stop_process():
         if process.poll() is None:
             os.kill(process.pid, signal.SIGTERM)
@@ -227,10 +232,10 @@
     """Split the given command-line arguments into services to start and Zope
     arguments.
 
-    The runlaunchpad script can take an optional '-r services,...' argument. If
-    this argument is present, then the value is returned as the first element
-    of the return tuple. The rest of the arguments are returned as the second
-    element of the return tuple.
+    The runlaunchpad script can take an optional '-r services,...' argument.
+    If this argument is present, then the value is returned as the first
+    element of the return tuple. The rest of the arguments are returned as the
+    second element of the return tuple.
 
     Returns a tuple of the form ([service_name, ...], remaining_argv).
     """

=== modified file 'lib/lp/scripts/utilities/killservice.py'
--- lib/lp/scripts/utilities/killservice.py	2010-08-20 20:31:18 +0000
+++ lib/lp/scripts/utilities/killservice.py	2010-09-18 03:14:43 +0000
@@ -17,7 +17,7 @@
 import time
 
 from canonical.config import config
-from canonical.launchpad.mailman.runmailman import stop_mailman
+from lp.services.mailman.runmailman import stop_mailman
 from canonical.launchpad.scripts import (
     logger,
     logger_options,

=== renamed directory 'lib/canonical/launchpad/mailman/config' => 'lib/lp/services/mailman/config'
=== modified file 'lib/lp/services/mailman/config/tests/test_config.py'
--- lib/canonical/launchpad/mailman/config/tests/test_config.py	2010-08-20 20:31:18 +0000
+++ lib/lp/services/mailman/config/tests/test_config.py	2010-09-18 03:14:43 +0000
@@ -10,9 +10,6 @@
 
 def test_suite():
     suite = DocTestSuite(
-            'canonical.launchpad.mailman.config',
-            optionflags=NORMALIZE_WHITESPACE | ELLIPSIS
-            )
+            'lp.services.mailman.config',
+            optionflags=NORMALIZE_WHITESPACE | ELLIPSIS)
     return suite
-
-

=== modified file 'lib/lp/services/mailman/doc/basic-integration.txt'
--- lib/lp/services/mailman/doc/basic-integration.txt	2009-09-24 20:14:42 +0000
+++ lib/lp/services/mailman/doc/basic-integration.txt	2010-09-18 03:14:43 +0000
@@ -9,7 +9,7 @@
 
     >>> import subprocess, os
     >>> from canonical.config import config
-    >>> from canonical.launchpad.mailman.config import configure_prefix
+    >>> from lp.services.mailman.config import configure_prefix
     >>> def withlist(callable, *options):
     ...     callable_path = 'lp.services.mailman.testing.' + callable
     ...     args = ' '.join(options)

=== modified file 'lib/lp/services/mailman/doc/batching.txt'
--- lib/lp/services/mailman/doc/batching.txt	2009-06-23 17:51:24 +0000
+++ lib/lp/services/mailman/doc/batching.txt	2010-09-18 03:14:43 +0000
@@ -18,7 +18,7 @@
 
 Now we'll restart Mailman with a batch size of 1.
 
-    >>> from canonical.launchpad.mailman import runmailman
+    >>> from lp.services.mailman import runmailman
     >>> from canonical.testing.layers import LayerProcessController
     >>> runmailman.stop_mailman(
     ...     quiet=True, config=LayerProcessController.appserver_config)

=== modified file 'lib/lp/services/mailman/doc/logging.txt'
--- lib/lp/services/mailman/doc/logging.txt	2009-09-24 18:51:29 +0000
+++ lib/lp/services/mailman/doc/logging.txt	2010-09-18 03:14:43 +0000
@@ -12,19 +12,19 @@
 in the [mailman] section of config.
 
     >>> from canonical.config import config
-    >>> from canonical.launchpad.mailman.monkeypatches.xmlrpcrunner import (
+    >>> from lp.services.mailman.monkeypatches.xmlrpcrunner import (
     ...     MailmanErrorUtility)
 
     >>> error_utility = MailmanErrorUtility()
     >>> print error_utility._default_config_section
     mailman
 
-    >>> print error_utility.prefix
+    >>> print error_utility.oops_prefix
     TMMX
     >>> print config.mailman.oops_prefix
     TMMX
 
-    >>> print error_utility.error_dir
+    >>> print error_utility.log_namer._output_root
     /var/tmp/mailman-xmlrpc.test
     >>> print config.mailman.error_dir
     /var/tmp/mailman-xmlrpc.test
@@ -36,7 +36,7 @@
 The log_exception() function is used by XMLRPCRunner to report known
 exceptions. It reports the OOPS.
 
-    >>> from canonical.launchpad.mailman.monkeypatches.xmlrpcrunner import (
+    >>> from lp.services.mailman.monkeypatches.xmlrpcrunner import (
     ...     log_exception)
 
     >>> def test_log_exception():

=== modified file 'lib/lp/services/mailman/doc/staging.txt'
--- lib/lp/services/mailman/doc/staging.txt	2010-09-01 09:57:15 +0000
+++ lib/lp/services/mailman/doc/staging.txt	2010-09-18 03:14:43 +0000
@@ -30,10 +30,8 @@
     >>> logout()
 
     >>> from canonical.launchpad.testing.pages import setupBrowser
-    >>> from lp.services.propertycache import IPropertyCache
     >>> browser = setupBrowser('Basic %s:%s' %
-    ...     (IPropertyCache(owner).preferredemail.email,
-    ...      owner._password_cleartext_cached))
+    ...     (owner.preferredemail.email, owner._password_cleartext_cached))
 
     >>> from lp.services.mailman.testing import helpers
     >>> helpers.apply_for_list(browser, 'staging-one')
@@ -140,7 +138,7 @@
 Now we need to simulate the difference between the staging and production
 databases.  To do this, we'll use a helper, but first we need to stop Mailman.
 
-    >>> from canonical.launchpad.mailman import runmailman
+    >>> from lp.services.mailman import runmailman
     >>> from canonical.testing.layers import LayerProcessController
     >>> runmailman.stop_mailman(
     ...     quiet=True, config=LayerProcessController.appserver_config)
@@ -194,7 +192,7 @@
 
     >>> import os
     >>> import sys
-    >>> from subprocess import *
+    >>> from subprocess import Popen, PIPE
     >>> proc = Popen(
     ...     ('scripts/mlist-sync.py', '--hostname',
     ...      'lists.prod.launchpad.dev', sync_details.source_dir),

=== renamed directory 'lib/canonical/launchpad/mailman/monkeypatches' => 'lib/lp/services/mailman/monkeypatches'
=== modified file 'lib/lp/services/mailman/monkeypatches/__init__.py'
--- lib/canonical/launchpad/mailman/monkeypatches/__init__.py	2009-09-24 20:14:42 +0000
+++ lib/lp/services/mailman/monkeypatches/__init__.py	2010-09-18 03:14:43 +0000
@@ -37,7 +37,7 @@
     # Calculate the parent directory of the canonical package.  This directory
     # will get appended to Mailman's sys.path.
     import canonical
-    from canonical.launchpad.mailman.config import configure_siteowner
+    from lp.services.mailman.config import configure_siteowner
     launchpad_top = os.path.dirname(
         os.path.dirname(os.path.dirname(canonical.__file__)))
     # Read the email footer template for all Launchpad messages.
@@ -88,7 +88,7 @@
     runner_file = open(runner_path, 'w')
     try:
         print >> runner_file, (
-            'from canonical.launchpad.mailman.monkeypatches.xmlrpcrunner '
+            'from lp.services.mailman.monkeypatches.xmlrpcrunner '
             'import *')
     finally:
         runner_file.close()
@@ -104,7 +104,7 @@
             mailman_path, 'Mailman', 'Handlers', mm_name + '.py')
         handler_file = open(handler_path, 'w')
         try:
-            package = 'canonical.launchpad.mailman.monkeypatches'
+            package = 'lp.services.mailman.monkeypatches'
             module = package + '.' + lp_name
             print >> handler_file, 'from', module, 'import *'
         finally:

=== modified file 'lib/lp/services/mailman/monkeypatches/mm_cfg.py.in'
--- lib/canonical/launchpad/mailman/monkeypatches/mm_cfg.py.in	2009-09-24 20:10:01 +0000
+++ lib/lp/services/mailman/monkeypatches/mm_cfg.py.in	2010-09-18 03:14:43 +0000
@@ -17,7 +17,7 @@
 
 # Pick up Launchpad static overrides.  This will also pick up the standard
 # Mailman.Defaults.* variables.
-from canonical.launchpad.mailman.monkeypatches.defaults import *
+from lp.services.mailman.monkeypatches.defaults import *
 
 # Our dynamic overrides of all the static defaults.
 SMTPHOST = '%(smtp_host)s'

=== renamed file 'lib/canonical/launchpad/mailman/runmailman.py' => 'lib/lp/services/mailman/runmailman.py'
--- lib/canonical/launchpad/mailman/runmailman.py	2010-08-20 20:31:18 +0000
+++ lib/lp/services/mailman/runmailman.py	2010-09-18 03:14:43 +0000
@@ -17,8 +17,8 @@
 import sys
 
 import canonical
-from canonical.launchpad.mailman.config import configure_prefix
-from canonical.launchpad.mailman.monkeypatches import monkey_patch
+from lp.services.mailman.config import configure_prefix
+from lp.services.mailman.monkeypatches import monkey_patch
 
 
 def mailmanctl(command, quiet=False, config=None, *additional_arguments):
@@ -36,7 +36,7 @@
     if config is None:
         config = canonical.config.config
     mailman_path = configure_prefix(config.mailman.build_prefix)
-    mailman_bin  = os.path.join(mailman_path, 'bin')
+    mailman_bin = os.path.join(mailman_path, 'bin')
     args = ['./mailmanctl']
     args.extend(additional_arguments)
     args.append(command)
@@ -117,7 +117,7 @@
     # We need the Mailman bin directory so we can run some of Mailman's
     # command line scripts.
     mailman_path = configure_prefix(config.mailman.build_prefix)
-    mailman_bin  = os.path.join(mailman_path, 'bin')
+    mailman_bin = os.path.join(mailman_path, 'bin')
 
     # Monkey-patch the installed Mailman 2.1 tree.
     monkey_patch(mailman_path, config)

=== modified file 'lib/lp/services/mailman/testing/layers.py'
--- lib/lp/services/mailman/testing/layers.py	2010-08-20 20:31:18 +0000
+++ lib/lp/services/mailman/testing/layers.py	2010-09-18 03:14:43 +0000
@@ -12,7 +12,7 @@
 import atexit
 import os
 
-from canonical.launchpad.mailman.runmailman import (
+from lp.services.mailman.runmailman import (
     start_mailman,
     stop_mailman,
     )

=== modified file 'lib/lp/services/mailman/testing/withlist_1.py'
--- lib/lp/services/mailman/testing/withlist_1.py	2010-08-20 20:31:18 +0000
+++ lib/lp/services/mailman/testing/withlist_1.py	2010-09-18 03:14:43 +0000
@@ -9,7 +9,7 @@
 # This script must be called like so:
 #
 # bin/withlist -r \
-#   canonical.launchpad.mailman.tests.withlist_1.test_site_list mailman
+#   lp.services.mailman.tests.withlist_1.test_site_list mailman
 #
 # where the final argument is the name of the site list, which must exist.
 

=== modified file 'lib/lp/services/mailman/testing/withlist_2.py'
--- lib/lp/services/mailman/testing/withlist_2.py	2010-08-20 20:31:18 +0000
+++ lib/lp/services/mailman/testing/withlist_2.py	2010-09-18 03:14:43 +0000
@@ -8,14 +8,14 @@
 #
 # This script must be called like so:
 #
-# bin/withlist -r canonical.launchpad.mailman.tests.withlist_2.can_import
+# bin/withlist -r lp.services.mailman.tests.withlist_2.can_import
 
 import sys
 
 
 def can_import(mlist):
     try:
-        import canonical.launchpad.mailman
+        import lp.services.mailman
     except ImportError:
         sys.exit(1)
     else:

=== modified file 'scripts/mlist-sync.py'
--- scripts/mlist-sync.py	2010-04-27 19:48:39 +0000
+++ scripts/mlist-sync.py	2010-09-18 03:14:43 +0000
@@ -39,7 +39,7 @@
 from canonical.config import config
 from canonical.launchpad.interfaces import (
     IEmailAddressSet, IMailingListSet, IPersonSet)
-from canonical.launchpad.mailman.config import configure_prefix
+from lp.services.mailman.config import configure_prefix
 from lp.services.scripts.base import LaunchpadCronScript
 
 

=== modified file 'utilities/launchpad-database-setup'
--- utilities/launchpad-database-setup	2010-09-17 11:59:56 +0000
+++ utilities/launchpad-database-setup	2010-09-18 03:14:43 +0000
@@ -41,7 +41,12 @@
   echo "ensure postgres is running on port 5432."
 fi;
 
-sudo /etc/init.d/postgresql stop $pgversion
+if [ -e /etc/init.d/postgresql-$pgversion ]; then
+    sudo /etc/init.d/postgresql-$pgversion stop
+else
+    # This is Maverick.
+    sudo /etc/init.d/postgresql stop $pgversion
+fi
 
 echo Purging postgresql data...
 sudo pg_dropcluster $pgversion main --stop-server
@@ -99,7 +104,12 @@
 EOF
 fi
 
-sudo /etc/init.d/postgresql start $pgversion
+if [ -e /etc/init.d/postgresql-$pgversion ]; then
+    sudo /etc/init.d/postgresql-$pgversion start
+else
+    # This is Maverick.
+    sudo /etc/init.d/postgresql start $pgversion
+fi
 
 echo Waiting 10 seconds for postgresql to come up...
 sleep 10