Merge lp:~abompard/mailman/sqlalchemy into lp:~raj-abhilash1/mailman/sqlalchemy

[Aurélien Bompard]

A small import fix

[lp:~abompard/mailman/sqlalchemy]

7287. By Aurélien Bompard on 2014-10-30

Fix doctests with PostgreSQL

- drop the tables after the whole testsuite has run
- don't expose DB ids in the doctests
- sort lists by list-id in the ListManager

------------- This line and the following will be ignored --------------

modified:
  src/mailman/app/docs/moderator.rst
  src/mailman/commands/docs/withlist.rst
  src/mailman/database/base.py
  src/mailman/model/docs/mailinglist.rst
  src/mailman/model/domain.py
  src/mailman/model/listmanager.py
  src/mailman/rest/docs/moderation.rst
  src/mailman/testing/layers.py

7288. By Aurélien Bompard on 2014-10-31

Add a cleanup instruction in a doctest

7289. By Aurélien Bompard on 2014-10-31

One more sorting issue

Unmerged revisions

7289. By Aurélien Bompard on 2014-10-31

One more sorting issue

7288. By Aurélien Bompard on 2014-10-31

Add a cleanup instruction in a doctest

7287. By Aurélien Bompard on 2014-10-30

Fix doctests with PostgreSQL

- drop the tables after the whole testsuite has run
- don't expose DB ids in the doctests
- sort lists by list-id in the ListManager

------------- This line and the following will be ignored --------------

modified:
  src/mailman/app/docs/moderator.rst
  src/mailman/commands/docs/withlist.rst
  src/mailman/database/base.py
  src/mailman/model/docs/mailinglist.rst
  src/mailman/model/domain.py
  src/mailman/model/listmanager.py
  src/mailman/rest/docs/moderation.rst
  src/mailman/testing/layers.py

7286. By Aurélien Bompard on 2014-10-29

Importer: encode_ascii_prefixes need to be converted to bool

7285. By Barry Warsaw on 2014-10-14

Use print() to smooth over the SA return of Python longs in PostgreSQL.

7284. By Barry Warsaw on 2014-10-13

Remove some unnecessary code.

7283. By Barry Warsaw on 2014-10-13

Move alembic settings to a separate alembic.cfg.

7282. By Barry Warsaw on 2014-10-13

Merge Aurélien Bompard's latest merge branch, with some cleaning up by Barry.

7281. By Barry Warsaw on 2014-10-12

Add the [logging.database] section and use it to configure the SQLAlchemy and
Alembic loggers.

7280. By Barry Warsaw on 2014-10-12

Remove some unused stuff.

=== modified file 'MANIFEST.in'
--- MANIFEST.in	2014-10-07 10:06:44 +0000
+++ MANIFEST.in	2014-10-31 13:23:25 +0000
@@ -1,4 +1,4 @@
-include *.py *.rc
+include *.py *.rc *.mako
 include COPYING
 recursive-include .buildout *
 recursive-include contrib *
@@ -6,12 +6,9 @@
 recursive-include data *
 global-include *.txt *.rst *.po *.mo *.cfg *.sql *.zcml *.html
 global-exclude *.egg-info
-exclude MANIFEST.in
 prune src/attic
 prune src/web
 prune eggs
 prune parts
 include MANIFEST.in
 include src/mailman/testing/config.pck
-include src/mailman/database/alembic/script.py.mako
-include src/mailman/database/alembic/versions/*.py

=== modified file 'src/mailman/app/docs/moderator.rst'
--- src/mailman/app/docs/moderator.rst	2014-04-28 15:23:35 +0000
+++ src/mailman/app/docs/moderator.rst	2014-10-31 13:23:25 +0000
@@ -211,9 +211,8 @@
     ...
     ... Here's something important about our mailing list.
     ... """)
-    >>> hold_message(mlist, msg, {}, 'Needs approval')
-    2
-    >>> handle_message(mlist, 2, Action.discard, forward=['[e-mail]'])
+    >>> req_id = hold_message(mlist, msg, {}, 'Needs approval')
+    >>> handle_message(mlist, req_id, Action.discard, forward=['[e-mail]'])
 
 The forwarded message is in the virgin queue, destined for the moderator.
 ::
@@ -243,10 +242,9 @@
 
     >>> from mailman.app.moderator import hold_subscription
     >>> from mailman.interfaces.member import DeliveryMode
-    >>> hold_subscription(mlist,
+    >>> req_id = hold_subscription(mlist,
     ...     '[e-mail]', 'Fred Person',
     ...     '{NONE}abcxyz', DeliveryMode.regular, 'en')
-    2
 
 
 Disposing of membership change requests
@@ -257,26 +255,26 @@
 simply defer a decision for now.
 
     >>> from mailman.app.moderator import handle_subscription
-    >>> handle_subscription(mlist, 2, Action.defer)
-    >>> requests.get_request(2) is not None
+    >>> handle_subscription(mlist, req_id, Action.defer)
+    >>> requests.get_request(req_id) is not None
     True
 
 The held subscription can also be discarded.
 
-    >>> handle_subscription(mlist, 2, Action.discard)
-    >>> print(requests.get_request(2))
+    >>> handle_subscription(mlist, req_id, Action.discard)
+    >>> print(requests.get_request(req_id))
     None
 
 Gwen tries to subscribe to the mailing list, but...
 
-    >>> hold_subscription(mlist,
+    >>> req_id = hold_subscription(mlist,
     ...     '[e-mail]', 'Gwen Person',
     ...     '{NONE}zyxcba', DeliveryMode.regular, 'en')
-    2
+
 
 ...her request is rejected...
 
-    >>> handle_subscription(mlist, 2, Action.reject, 'This is a closed list')
+    >>> handle_subscription(mlist, req_id, Action.reject, 'This is a closed list')
     >>> messages = get_queue_messages('virgin')
     >>> len(messages)
     1
@@ -304,14 +302,13 @@
 mailing list.
 
     >>> mlist.send_welcome_message = False
-    >>> hold_subscription(mlist,
+    >>> req_id = hold_subscription(mlist,
     ...     '[e-mail]', 'Herb Person',
     ...     'abcxyz', DeliveryMode.regular, 'en')
-    2
 
 The moderators accept the subscription request.
 
-    >>> handle_subscription(mlist, 2, Action.accept)
+    >>> handle_subscription(mlist, req_id, Action.accept)
 
 And now Herb is a member of the mailing list.
 
@@ -328,29 +325,27 @@
 Herb now wants to leave the mailing list, but his request must be approved.
 
     >>> from mailman.app.moderator import hold_unsubscription
-    >>> hold_unsubscription(mlist, '[e-mail]')
-    2
+    >>> req_id = hold_unsubscription(mlist, '[e-mail]')
 
 As with subscription requests, the unsubscription request can be deferred.
 
     >>> from mailman.app.moderator import handle_unsubscription
-    >>> handle_unsubscription(mlist, 2, Action.defer)
+    >>> handle_unsubscription(mlist, req_id, Action.defer)
     >>> print(mlist.members.get_member('[e-mail]').address)
     Herb Person <[e-mail]>
 
 The held unsubscription can also be discarded, and the member will remain
 subscribed.
 
-    >>> handle_unsubscription(mlist, 2, Action.discard)
+    >>> handle_unsubscription(mlist, req_id, Action.discard)
     >>> print(mlist.members.get_member('[e-mail]').address)
     Herb Person <[e-mail]>
 
 The request can be rejected, in which case a message is sent to the member,
 and the person remains a member of the mailing list.
 
-    >>> hold_unsubscription(mlist, '[e-mail]')
-    2
-    >>> handle_unsubscription(mlist, 2, Action.reject, 'No can do')
+    >>> req_id = hold_unsubscription(mlist, '[e-mail]')
+    >>> handle_unsubscription(mlist, req_id, Action.reject, 'No can do')
     >>> print(mlist.members.get_member('[e-mail]').address)
     Herb Person <[e-mail]>
 
@@ -381,10 +376,9 @@
 The unsubscription request can also be accepted.  This removes the member from
 the mailing list.
 
-    >>> hold_unsubscription(mlist, '[e-mail]')
-    2
+    >>> req_id = hold_unsubscription(mlist, '[e-mail]')
     >>> mlist.send_goodbye_message = False
-    >>> handle_unsubscription(mlist, 2, Action.accept)
+    >>> handle_unsubscription(mlist, req_id, Action.accept)
     >>> print(mlist.members.get_member('[e-mail]'))
     None
 
@@ -403,9 +397,8 @@
 
 Iris tries to subscribe to the mailing list.
 
-    >>> hold_subscription(mlist, '[e-mail]', 'Iris Person',
+    >>> req_id = hold_subscription(mlist, '[e-mail]', 'Iris Person',
     ...                   'password', DeliveryMode.regular, 'en')
-    2
 
 There's now a message in the virgin queue, destined for the list owner.
 
@@ -429,8 +422,7 @@
 Similarly, the administrator gets notifications on unsubscription requests.
 Jeff is a member of the mailing list, and chooses to unsubscribe.
 
-    >>> hold_unsubscription(mlist, '[e-mail]')
-    3
+    >>> unsub_req_id = hold_unsubscription(mlist, '[e-mail]')
     >>> messages = get_queue_messages('virgin')
     >>> len(messages)
     1
@@ -457,7 +449,7 @@
 
     >>> mlist.admin_notify_mchanges = True
     >>> mlist.admin_immed_notify = False
-    >>> handle_subscription(mlist, 2, Action.accept)
+    >>> handle_subscription(mlist, req_id, Action.accept)
     >>> messages = get_queue_messages('virgin')
     >>> len(messages)
     1
@@ -474,9 +466,8 @@
 Similarly when an unsubscription request is accepted, the administrators can
 get a notification.
 
-    >>> hold_unsubscription(mlist, '[e-mail]')
-    4
-    >>> handle_unsubscription(mlist, 4, Action.accept)
+    >>> req_id = hold_unsubscription(mlist, '[e-mail]')
+    >>> handle_unsubscription(mlist, req_id, Action.accept)
     >>> messages = get_queue_messages('virgin')
     >>> len(messages)
     1
@@ -498,10 +489,9 @@
 
     >>> mlist.admin_notify_mchanges = False
     >>> mlist.send_welcome_message = True
-    >>> hold_subscription(mlist, '[e-mail]', 'Kate Person',
-    ...                   'password', DeliveryMode.regular, 'en')
-    4
-    >>> handle_subscription(mlist, 4, Action.accept)
+    >>> req_id = hold_subscription(mlist, '[e-mail]', 'Kate Person',
+    ...                            'password', DeliveryMode.regular, 'en')
+    >>> handle_subscription(mlist, req_id, Action.accept)
     >>> messages = get_queue_messages('virgin')
     >>> len(messages)
     1
@@ -523,9 +513,8 @@
 goodbye message.
 
     >>> mlist.send_goodbye_message = True
-    >>> hold_unsubscription(mlist, '[e-mail]')
-    4
-    >>> handle_unsubscription(mlist, 4, Action.accept)
+    >>> req_id = hold_unsubscription(mlist, '[e-mail]')
+    >>> handle_unsubscription(mlist, req_id, Action.accept)
     >>> messages = get_queue_messages('virgin')
     >>> len(messages)
     1

=== modified file 'src/mailman/commands/docs/conf.rst'
--- src/mailman/commands/docs/conf.rst	2014-10-10 04:59:43 +0000
+++ src/mailman/commands/docs/conf.rst	2014-10-31 13:23:25 +0000
@@ -22,7 +22,7 @@
 command without any options.
 
     >>> command.process(FakeArgs)
-    [alembic] script_location: mailman.database:alembic
+    [logging.archiver] path: mailman.log
     ...
     [passwords] password_length: 8
     ...
@@ -43,7 +43,6 @@
     >>> FakeArgs.section = None
     >>> FakeArgs.key = 'path'
     >>> command.process(FakeArgs)
-    [logging.dbmigration] path: mailman.log
     [logging.archiver] path: mailman.log
     [logging.locks] path: mailman.log
     [logging.mischief] path: mailman.log

=== modified file 'src/mailman/commands/docs/withlist.rst'
--- src/mailman/commands/docs/withlist.rst	2014-04-28 15:23:35 +0000
+++ src/mailman/commands/docs/withlist.rst	2014-10-31 13:23:25 +0000
@@ -90,13 +90,13 @@
     >>> args.listname = '^.*example.com'
     >>> command.process(args)
     The list's display name is Aardvark
+    The list's display name is Badboys
     The list's display name is Badger
-    The list's display name is Badboys
 
     >>> args.listname = '^bad.*'
     >>> command.process(args)
+    The list's display name is Badboys
     The list's display name is Badger
-    The list's display name is Badboys
 
     >>> args.listname = '^foo'
     >>> command.process(args)

=== added file 'src/mailman/config/alembic.cfg'
--- src/mailman/config/alembic.cfg	1970-01-01 00:00:00 +0000
+++ src/mailman/config/alembic.cfg	2014-10-31 13:23:25 +0000
@@ -0,0 +1,20 @@
+# Copyright (C) 2014 by the Free Software Foundation, Inc.
+#
+# This file is part of GNU Mailman.
+#
+# GNU Mailman is free software: you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option)
+# any later version.
+#
+# GNU Mailman is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+# more details.
+#
+# You should have received a copy of the GNU General Public License along with
+# GNU Mailman.  If not, see <http://www.gnu.org/licenses/>.
+
+[alembic]
+# Path to Alembic migration scripts.
+script_location: mailman.database:alembic

=== modified file 'src/mailman/config/config.py'
--- src/mailman/config/config.py	2014-10-02 14:46:00 +0000
+++ src/mailman/config/config.py	2014-10-31 13:23:25 +0000
@@ -87,7 +87,6 @@
         self.pipelines = {}
         self.commands = {}
         self.password_context = None
-        self.initialized = False
 
     def _clear(self):
         """Clear the cached configuration variables."""
@@ -137,7 +136,6 @@
         # Expand and set up all directories.
         self._expand_paths()
         self.ensure_directories_exist()
-        self.initialized = True
         notify(ConfigurationUpdatedEvent(self))
 
     def _expand_paths(self):

=== modified file 'src/mailman/config/schema.cfg'
--- src/mailman/config/schema.cfg	2014-10-10 04:59:43 +0000
+++ src/mailman/config/schema.cfg	2014-10-31 13:23:25 +0000
@@ -226,6 +226,7 @@
 # - archiver        --  All archiver output
 # - bounce          --  All bounce processing logs go here
 # - config          --  Configuration issues
+# - database        --  Database logging (SQLAlchemy and Alembic)
 # - debug           --  Only used for development
 # - error           --  All exceptions go to this log
 # - fromusenet      --  Information related to the Usenet to Mailman gateway
@@ -237,8 +238,6 @@
 # - smtp-failure    --  Unsuccessful SMTP activity
 # - subscribe       --  Information about leaves/joins
 # - vette           --  Message vetting information
-# - database        --  Database activity
-# - dbmigration     --  Database migrations
 format: %(asctime)s (%(process)d) %(message)s
 datefmt: %b %d %H:%M:%S %Y
 propagate: no
@@ -254,6 +253,8 @@
 
 [logging.config]
 
+[logging.database]
+
 [logging.debug]
 path: debug.log
 level: info
@@ -306,9 +307,6 @@
 [logging.database]
 level: warn
 
-[logging.dbmigration]
-level: warn
-
 
 [webservice]
 # The hostname at which admin web service resources are exposed.
@@ -645,7 +643,3 @@
     CC X-Original-CC
     Content-Transfer-Encoding X-Original-Content-Transfer-Encoding
     MIME-Version X-MIME-Version
-
-[alembic]
-# path to migration scripts
-script_location = mailman.database:alembic

=== modified file 'src/mailman/core/docs/runner.rst'
--- src/mailman/core/docs/runner.rst	2014-04-28 15:23:35 +0000
+++ src/mailman/core/docs/runner.rst	2014-10-31 13:23:25 +0000
@@ -73,3 +73,5 @@
     version  : 3
 
 XXX More of the Runner API should be tested.
+
+    >>> config.pop('test-runner')

=== modified file 'src/mailman/core/logging.py'
--- src/mailman/core/logging.py	2014-10-07 09:19:20 +0000
+++ src/mailman/core/logging.py	2014-10-31 13:23:25 +0000
@@ -104,6 +104,27 @@
 
 
 

+def _init_logger(propagate, sub_name, log, logger_config):
+    # Get settings from log configuration file (or defaults).
+    log_format = logger_config.format
+    log_datefmt = logger_config.datefmt
+    # Propagation to the root logger is how we handle logging to stderr
+    # when the runners are not run as a subprocess of 'bin/mailman start'.
+    log.propagate = (as_boolean(logger_config.propagate)
+                     if propagate is None else propagate)
+    # Set the logger's level.
+    log.setLevel(as_log_level(logger_config.level))
+    # Create a formatter for this logger, then a handler, and link the
+    # formatter to the handler.
+    formatter = logging.Formatter(fmt=log_format, datefmt=log_datefmt)
+    path_str = logger_config.path
+    path_abs = os.path.normpath(os.path.join(config.LOG_DIR, path_str))
+    handler = ReopenableFileHandler(sub_name, path_abs)
+    _handlers[sub_name] = handler
+    handler.setFormatter(formatter)
+    log.addHandler(handler)
+
+
 def initialize(propagate=None):
     """Initialize all logs.
 
@@ -126,32 +147,18 @@
             continue
         if sub_name == 'locks':
             log = logging.getLogger('flufl.lock')
-        elif sub_name == 'database':
+        if sub_name == 'database':
+            # Set both the SQLAlchemy and Alembic logs to the mailman.database
+            # log configuration, essentially ignoring the alembic.cfg
+            # settings.  Do the SQLAlchemy one first, then let the Alembic one
+            # fall through to the common code path.
             log = logging.getLogger('sqlalchemy')
-        elif sub_name == 'dbmigration':
+            _init_logger(propagate, sub_name, log, logger_config)
             log = logging.getLogger('alembic')
         else:
             logger_name = 'mailman.' + sub_name
             log = logging.getLogger(logger_name)
-        # Get settings from log configuration file (or defaults).
-        log_format = logger_config.format
-        log_datefmt = logger_config.datefmt
-        # Propagation to the root logger is how we handle logging to stderr
-        # when the runners are not run as a subprocess of 'bin/mailman start'.
-        log.propagate = (as_boolean(logger_config.propagate)
-                         if propagate is None else propagate)
-        # Set the logger's level.
-        log.setLevel(as_log_level(logger_config.level))
-        # Create a formatter for this logger, then a handler, and link the
-        # formatter to the handler.
-        formatter = logging.Formatter(fmt=log_format, datefmt=log_datefmt)
-        path_str = logger_config.path
-        path_abs = os.path.normpath(os.path.join(config.LOG_DIR, path_str))
-        handler = ReopenableFileHandler(sub_name, path_abs)
-        _handlers[sub_name] = handler
-        handler.setFormatter(formatter)
-        log.addHandler(handler)
-
+        _init_logger(propagate, sub_name, log, logger_config)
 
 

 def reopen():

=== modified file 'src/mailman/database/alembic/__init__.py'
--- src/mailman/database/alembic/__init__.py	2014-10-10 04:59:43 +0000
+++ src/mailman/database/alembic/__init__.py	2014-10-31 13:23:25 +0000
@@ -15,18 +15,18 @@
 # You should have received a copy of the GNU General Public License along with
 # GNU Mailman.  If not, see <http://www.gnu.org/licenses/>.
 
-"Alembic config init."
+"""Alembic configuration initization."""
 
 from __future__ import absolute_import, print_function, unicode_literals
 
 __metaclass__ = type
 __all__ = [
-    'alembic_cfg'
-]
+    'alembic_cfg',
+    ]
 
 
 from alembic.config import Config
 from mailman.utilities.modules import expand_path
 
 
-alembic_cfg=Config(expand_path("python:mailman.config.schema"))
+alembic_cfg = Config(expand_path('python:mailman.config.alembic'))

=== modified file 'src/mailman/database/alembic/env.py'
--- src/mailman/database/alembic/env.py	2014-10-10 04:59:43 +0000
+++ src/mailman/database/alembic/env.py	2014-10-31 13:23:25 +0000
@@ -30,15 +30,10 @@
 from contextlib import closing
 from sqlalchemy import create_engine
 
-from mailman.core import initialize
 from mailman.config import config
-from mailman.database.alembic import alembic_cfg
 from mailman.database.model import Model
 from mailman.utilities.string import expand
 
-if not config.initialized:
-    initialize.initialize_1(context.config.config_file_name)
-
 
 

 def run_migrations_offline():

=== modified file 'src/mailman/database/alembic/versions/51b7f92bd06c_initial.py'
--- src/mailman/database/alembic/versions/51b7f92bd06c_initial.py	2014-10-10 16:58:48 +0000
+++ src/mailman/database/alembic/versions/51b7f92bd06c_initial.py	2014-10-31 13:23:25 +0000
@@ -1,34 +1,66 @@
-"""initial
+# Copyright (C) 2014 by the Free Software Foundation, Inc.
+#
+# This file is part of GNU Mailman.
+#
+# GNU Mailman is free software: you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free
+# Software Foundation, either version 3 of the License, or (at your option)
+# any later version.
+#
+# GNU Mailman is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+# more details.
+#
+# You should have received a copy of the GNU General Public License along with
+# GNU Mailman.  If not, see <http://www.gnu.org/licenses/>.
+
+"""Initial migration.
+
+This empty migration file makes sure there is always an alembic_version
+in the database.  As a consequence, if the database version is reported
+as None, it means the database needs to be created from scratch with
+SQLAlchemy itself.
+
+It also removes schema items left over from Storm.
 
 Revision ID: 51b7f92bd06c
 Revises: None
 Create Date: 2014-10-10 09:53:35.624472
-
 """
 
-# revision identifiers, used by Alembic.
+from __future__ import absolute_import, print_function, unicode_literals
+
+__metaclass__ = type
+__all__ = [
+    'downgrade',
+    'upgrade',
+    ]
+
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# Revision identifiers, used by Alembic.
 revision = '51b7f92bd06c'
 down_revision = None
 
-from alembic import op
-import sqlalchemy as sa
-
 
 def upgrade():
-    ### commands auto generated by Alembic - please adjust! ###
     op.drop_table('version')
-    if op.get_bind().dialect.name != "sqlite":
-        # SQLite does not support dropping columns
+    if op.get_bind().dialect.name != 'sqlite':
+        # SQLite does not support dropping columns.
         op.drop_column('mailinglist', 'acceptable_aliases_id')
-    op.create_index(op.f('ix_user__user_id'), 'user', ['_user_id'], unique=False)
+    op.create_index(op.f('ix_user__user_id'), 'user',
+                    ['_user_id'], unique=False)
     op.drop_index('ix_user_user_id', table_name='user')
-    ### end Alembic commands ###
 
 
 def downgrade():
-    ### commands auto generated by Alembic - please adjust! ###
     op.create_table('version')
     op.create_index('ix_user_user_id', 'user', ['_user_id'], unique=False)
     op.drop_index(op.f('ix_user__user_id'), table_name='user')
-    op.add_column('mailinglist', sa.Column('acceptable_aliases_id', sa.INTEGER(), nullable=True))
-    ### end Alembic commands ###
+    op.add_column(
+        'mailinglist',
+        sa.Column('acceptable_aliases_id', sa.INTEGER(), nullable=True))

=== modified file 'src/mailman/database/base.py'
--- src/mailman/database/base.py	2014-10-06 13:58:58 +0000
+++ src/mailman/database/base.py	2014-10-31 13:23:25 +0000
@@ -25,7 +25,6 @@
 
 import logging
 
-from alembic import command
 from sqlalchemy import create_engine
 from sqlalchemy.orm import sessionmaker
 from zope.interface import implementer
@@ -115,3 +114,8 @@
         session = sessionmaker(bind=self.engine)
         self.store = session()
         self.store.commit()
+
+    def destroy(self):
+        """Drop all database tables"""
+        from mailman.database.model import Model
+        Model.metadata.drop_all(self.engine)

=== modified file 'src/mailman/database/factory.py'
--- src/mailman/database/factory.py	2014-10-10 16:44:01 +0000
+++ src/mailman/database/factory.py	2014-10-31 13:23:25 +0000
@@ -28,8 +28,8 @@
 
 import os
 import types
+import alembic.command
 
-from alembic import command
 from alembic.migration import MigrationContext
 from alembic.script import ScriptDirectory
 from flufl.lock import Lock
@@ -38,10 +38,14 @@
 from zope.interface.verify import verifyObject
 
 from mailman.config import config
+from mailman.database.alembic import alembic_cfg
 from mailman.database.model import Model
-from mailman.database.alembic import alembic_cfg
-from mailman.interfaces.database import IDatabase, IDatabaseFactory
-from mailman.utilities.modules import call_name, expand_path
+from mailman.interfaces.database import (
+    DatabaseError, IDatabase, IDatabaseFactory)
+from mailman.utilities.modules import call_name
+
+
+LAST_STORM_SCHEMA_VERSION = '20130406000000'
 
 
 

@@ -57,67 +61,58 @@
             database = call_name(database_class)
             verifyObject(IDatabase, database)
             database.initialize()
-            schema_mgr = SchemaManager(database)
-            schema_mgr.setup_db()
+            SchemaManager(database).setup_database()
             database.commit()
             return database
 
 
 

 class SchemaManager:
-
-    LAST_STORM_SCHEMA_VERSION = '20130406000000'
+    "Manage schema migrations."""
 
     def __init__(self, database):
-        self.database = database
-        self.script = ScriptDirectory.from_config(alembic_cfg)
+        self._database = database
+        self._script = ScriptDirectory.from_config(alembic_cfg)
 
-    def get_storm_schema_version(self):
-        md = MetaData()
-        md.reflect(bind=self.database.engine)
-        if "version" not in md.tables:
+    def _get_storm_schema_version(self):
+        metadata = MetaData()
+        metadata.reflect(bind=self._database.engine)
+        if 'version' not in metadata.tables:
+            # There are no Storm artifacts left.
             return None
-        Version = md.tables["version"]
-        last_version = self.database.store.query(Version.c.version).filter(
-                Version.c.component == "schema"
-                ).order_by(Version.c.version.desc()).first()
-        # Don't leave open transactions or they will block any schema change
-        self.database.commit()
+        Version = metadata.tables['version']
+        last_version = self._database.store.query(Version.c.version).filter(
+            Version.c.component == 'schema'
+            ).order_by(Version.c.version.desc()).first()
+        # Don't leave open transactions or they will block any schema change.
+        self._database.commit()
         return last_version
 
-    def _create(self):
-        # initial DB creation
-        Model.metadata.create_all(self.database.engine)
-        self.database.commit()
-        command.stamp(alembic_cfg, "head")
-
-    def _upgrade(self):
-        command.upgrade(alembic_cfg, "head")
-
-    def setup_db(self):
-        context = MigrationContext.configure(self.database.store.connection())
+    def setup_database(self):
+        context = MigrationContext.configure(self._database.store.connection())
         current_rev = context.get_current_revision()
-        head_rev = self.script.get_current_head()
+        head_rev = self._script.get_current_head()
         if current_rev == head_rev:
-            return head_rev # already at the latest revision, nothing to do
-        if current_rev == None:
-            # no alembic information
-            storm_version = self.get_storm_schema_version()
+             # We're already at the latest revision so there's nothing to do.
+            return head_rev
+        if current_rev is None:
+            # No Alembic information is available.
+            storm_version = self._get_storm_schema_version()
             if storm_version is None:
-                # initial DB creation
-                self._create()
+                # Initial database creation.
+                Model.metadata.create_all(self._database.engine)
+                self._database.commit()
+                alembic.command.stamp(alembic_cfg, 'head')
             else:
-                # DB from a previous version managed by Storm
-                if storm_version.version < self.LAST_STORM_SCHEMA_VERSION:
-                    raise RuntimeError(
-                            "Upgrading while skipping beta version is "
-                            "unsupported, please install the previous "
-                            "Mailman beta release")
-                # Run migrations to remove the Storm-specific table and
-                # upgrade to SQLAlchemy & Alembic
-                self._upgrade()
+                # The database was previously managed by Storm.
+                if storm_version.version < LAST_STORM_SCHEMA_VERSION:
+                    raise DatabaseError(
+                        'Upgrades skipping beta versions is not supported.')
+                # Run migrations to remove the Storm-specific table and upgrade
+                # to SQLAlchemy and Alembic.
+                alembic.command.upgrade(alembic_cfg, 'head')
         elif current_rev != head_rev:
-            self._upgrade()
+            alembic.command.upgrade(alembic_cfg, 'head')
         return head_rev
 
 

=== modified file 'src/mailman/database/tests/test_factory.py'
--- src/mailman/database/tests/test_factory.py	2014-10-10 16:58:48 +0000
+++ src/mailman/database/tests/test_factory.py	2014-10-31 13:23:25 +0000
@@ -21,24 +21,24 @@
 
 __metaclass__ = type
 __all__ = [
+    'TestSchemaManager',
     ]
 
 
 import unittest
-import types
-
 import alembic.command
-from mock import Mock
+
+from mock import patch
 from sqlalchemy import MetaData, Table, Column, Integer, Unicode
+from sqlalchemy.exc import ProgrammingError, OperationalError
 from sqlalchemy.schema import Index
-from sqlalchemy.exc import ProgrammingError, OperationalError
 
 from mailman.config import config
-from mailman.testing.layers import ConfigLayer
-from mailman.database.factory import SchemaManager, _reset
-from mailman.database.sqlite import SQLiteDatabase
 from mailman.database.alembic import alembic_cfg
+from mailman.database.factory import LAST_STORM_SCHEMA_VERSION, SchemaManager
 from mailman.database.model import Model
+from mailman.interfaces.database import DatabaseError
+from mailman.testing.layers import ConfigLayer
 
 
 

@@ -47,116 +47,114 @@
     layer = ConfigLayer
 
     def setUp(self):
-        # Drop the existing database
+        # Drop the existing database.
         Model.metadata.drop_all(config.db.engine)
         md = MetaData()
         md.reflect(bind=config.db.engine)
-        for tablename in ("alembic_version", "version"):
+        for tablename in ('alembic_version', 'version'):
             if tablename in md.tables:
                 md.tables[tablename].drop(config.db.engine)
         self.schema_mgr = SchemaManager(config.db)
 
     def tearDown(self):
         self._drop_storm_database()
-        # Restore a virgin DB
+        # Restore a virgin database.
         Model.metadata.create_all(config.db.engine)
 
-
     def _table_exists(self, tablename):
         md = MetaData()
         md.reflect(bind=config.db.engine)
         return tablename in md.tables
 
     def _create_storm_database(self, revision):
-        version_table = Table("version", Model.metadata,
-                Column("id", Integer, primary_key=True),
-                Column("component", Unicode),
-                Column("version", Unicode),
-                )
+        version_table = Table(
+            'version', Model.metadata,
+            Column('id', Integer, primary_key=True),
+            Column('component', Unicode),
+            Column('version', Unicode),
+            )
         version_table.create(config.db.engine)
         config.db.store.execute(version_table.insert().values(
-                component='schema', version=revision))
+            component='schema', version=revision))
         config.db.commit()
         # Other Storm specific changes, those SQL statements hopefully work on
         # all DB engines...
         config.db.engine.execute(
-            "ALTER TABLE mailinglist ADD COLUMN acceptable_aliases_id INT")
-        Index("ix_user__user_id").drop(bind=config.db.engine)
-        # Don't pollute our main metadata object, create a new one
+            'ALTER TABLE mailinglist ADD COLUMN acceptable_aliases_id INT')
+        Index('ix_user__user_id').drop(bind=config.db.engine)
+        # Don't pollute our main metadata object, create a new one.
         md = MetaData()
-        user_table = Model.metadata.tables["user"].tometadata(md)
-        Index("ix_user_user_id", user_table.c._user_id
-              ).create(bind=config.db.engine)
+        user_table = Model.metadata.tables['user'].tometadata(md)
+        Index('ix_user_user_id', user_table.c._user_id).create(
+            bind=config.db.engine)
         config.db.commit()
 
     def _drop_storm_database(self):
-        """
-        Remove the leftovers from a Storm DB.
-        (you must issue a drop_all() afterwards)
-        """
-        if "version" in Model.metadata.tables:
-            version = Model.metadata.tables["version"]
+        """Remove the leftovers from a Storm DB.
+
+        A drop_all() must be issued afterwards.
+        """
+        if 'version' in Model.metadata.tables:
+            version = Model.metadata.tables['version']
             version.drop(config.db.engine, checkfirst=True)
             Model.metadata.remove(version)
         try:
-            Index("ix_user_user_id").drop(bind=config.db.engine)
-        except (ProgrammingError, OperationalError) as e:
-            # non-existant (PGSQL raises a ProgrammingError, while SQLite
-            # raises an OperationalError)
+            Index('ix_user_user_id').drop(bind=config.db.engine)
+        except (ProgrammingError, OperationalError):
+            # Nonexistent.  PostgreSQL raises a ProgrammingError, while SQLite
+            # raises an OperationalError.
             pass
         config.db.commit()
 
-
-    def test_current_db(self):
-        """The database is already at the latest version"""
-        alembic.command.stamp(alembic_cfg, "head")
-        self.schema_mgr._create = Mock()
-        self.schema_mgr._upgrade = Mock()
-        self.schema_mgr.setup_db()
-        self.assertFalse(self.schema_mgr._create.called)
-        self.assertFalse(self.schema_mgr._upgrade.called)
-
-    def test_initial(self):
-        """No existing database"""
-        self.assertFalse(self._table_exists("mailinglist"))
-        self.assertFalse(self._table_exists("alembic_version"))
-        self.schema_mgr._upgrade = Mock()
-        self.schema_mgr.setup_db()
-        self.assertFalse(self.schema_mgr._upgrade.called)
-        self.assertTrue(self._table_exists("mailinglist"))
-        self.assertTrue(self._table_exists("alembic_version"))
-
-    def test_storm(self):
-        """Existing Storm database"""
-        Model.metadata.create_all(config.db.engine)
-        self._create_storm_database(
-                self.schema_mgr.LAST_STORM_SCHEMA_VERSION)
-        self.schema_mgr._create = Mock()
-        self.schema_mgr.setup_db()
-        self.assertFalse(self.schema_mgr._create.called)
-        self.assertTrue(self._table_exists("mailinglist")
-                    and self._table_exists("alembic_version")
-                    and not self._table_exists("version"))
-
-    def test_old_storm(self):
-        """Existing Storm database in an old version"""
-        Model.metadata.create_all(config.db.engine)
-        self._create_storm_database("001")
-        self.schema_mgr._create = Mock()
-        self.assertRaises(RuntimeError, self.schema_mgr.setup_db)
-        self.assertFalse(self.schema_mgr._create.called)
+    def test_current_database(self):
+        # The database is already at the latest version.
+        alembic.command.stamp(alembic_cfg, 'head')
+        with patch('alembic.command') as alembic_command:
+            self.schema_mgr.setup_database()
+            self.assertFalse(alembic_command.stamp.called)
+            self.assertFalse(alembic_command.upgrade.called)
+
+    @patch('alembic.command')
+    def test_initial(self, alembic_command):
+        # No existing database.
+        self.assertFalse(self._table_exists('mailinglist'))
+        self.assertFalse(self._table_exists('alembic_version'))
+        self.schema_mgr.setup_database()
+        self.assertFalse(alembic_command.upgrade.called)
+        self.assertTrue(self._table_exists('mailinglist'))
+        self.assertTrue(self._table_exists('alembic_version'))
+
+    @patch('alembic.command.stamp')
+    def test_storm(self, alembic_command_stamp):
+        # Existing Storm database.
+        Model.metadata.create_all(config.db.engine)
+        self._create_storm_database(LAST_STORM_SCHEMA_VERSION)
+        self.schema_mgr.setup_database()
+        self.assertFalse(alembic_command_stamp.called)
+        self.assertTrue(
+            self._table_exists('mailinglist')
+            and self._table_exists('alembic_version')
+            and not self._table_exists('version'))
+
+    @patch('alembic.command')
+    def test_old_storm(self, alembic_command):
+        # Existing Storm database in an old version.
+        Model.metadata.create_all(config.db.engine)
+        self._create_storm_database('001')
+        self.assertRaises(DatabaseError, self.schema_mgr.setup_database)
+        self.assertFalse(alembic_command.stamp.called)
+        self.assertFalse(alembic_command.upgrade.called)
 
     def test_old_db(self):
-        """The database is in an old revision, must upgrade"""
-        alembic.command.stamp(alembic_cfg, "head")
+        # The database is in an old revision, must upgrade.
+        alembic.command.stamp(alembic_cfg, 'head')
         md = MetaData()
         md.reflect(bind=config.db.engine)
-        config.db.store.execute(md.tables["alembic_version"].delete())
-        config.db.store.execute(md.tables["alembic_version"].insert().values(
-                version_num="dummyrevision"))
+        config.db.store.execute(md.tables['alembic_version'].delete())
+        config.db.store.execute(md.tables['alembic_version'].insert().values(
+            version_num='dummyrevision'))
         config.db.commit()
-        self.schema_mgr._create = Mock()
-        self.schema_mgr._upgrade = Mock()
-        self.schema_mgr.setup_db()
-        self.assertFalse(self.schema_mgr._create.called)
-        self.assertTrue(self.schema_mgr._upgrade.called)
+        with patch('alembic.command') as alembic_command:
+            self.schema_mgr.setup_database()
+            self.assertFalse(alembic_command.stamp.called)
+            self.assertTrue(alembic_command.upgrade.called)

=== modified file 'src/mailman/handlers/docs/owner-recips.rst'
--- src/mailman/handlers/docs/owner-recips.rst	2012-03-23 20:34:54 +0000
+++ src/mailman/handlers/docs/owner-recips.rst	2014-10-31 13:23:25 +0000
@@ -41,7 +41,7 @@
     >>> handler.process(mlist_1, msg, msgdata)
     >>> dump_list(msgdata['recipients'])
     [e-mail]
-    
+
 If Bart also disables his owner delivery, then no one could contact the list's
 owners.  Since this is unacceptable, the site owner is used as a fallback.
 
@@ -55,7 +55,7 @@
 a fallback.
 
     >>> mlist_2 = create_list('[e-mail]')
-    >>> mlist_2.administrators.member_count
+    >>> print(mlist_2.administrators.member_count)
     0
     >>> msgdata = {}
     >>> handler.process(mlist_2, msg, msgdata)

=== modified file 'src/mailman/model/docs/autorespond.rst'
--- src/mailman/model/docs/autorespond.rst	2014-04-28 15:23:35 +0000
+++ src/mailman/model/docs/autorespond.rst	2014-10-31 13:23:25 +0000
@@ -37,34 +37,34 @@
     ...     '[e-mail]')
 
     >>> from mailman.interfaces.autorespond import Response
-    >>> response_set.todays_count(address, Response.hold)
+    >>> print(response_set.todays_count(address, Response.hold))
     0
-    >>> response_set.todays_count(address, Response.command)
+    >>> print(response_set.todays_count(address, Response.command))
     0
 
 Using the response set, we can record that a hold response is sent to the
 address.
 
     >>> response_set.response_sent(address, Response.hold)
-    >>> response_set.todays_count(address, Response.hold)
+    >>> print(response_set.todays_count(address, Response.hold))
     1
-    >>> response_set.todays_count(address, Response.command)
+    >>> print(response_set.todays_count(address, Response.command))
     0
 
 We can also record that a command response was sent.
 
     >>> response_set.response_sent(address, Response.command)
-    >>> response_set.todays_count(address, Response.hold)
+    >>> print(response_set.todays_count(address, Response.hold))
     1
-    >>> response_set.todays_count(address, Response.command)
+    >>> print(response_set.todays_count(address, Response.command))
     1
 
 Let's send one more.
 
     >>> response_set.response_sent(address, Response.command)
-    >>> response_set.todays_count(address, Response.hold)
+    >>> print(response_set.todays_count(address, Response.hold))
     1
-    >>> response_set.todays_count(address, Response.command)
+    >>> print(response_set.todays_count(address, Response.command))
     2
 
 Now the day flips over and all the counts reset.
@@ -73,9 +73,9 @@
     >>> from mailman.utilities.datetime import factory
     >>> factory.fast_forward()
 
-    >>> response_set.todays_count(address, Response.hold)
+    >>> print(response_set.todays_count(address, Response.hold))
     0
-    >>> response_set.todays_count(address, Response.command)
+    >>> print(response_set.todays_count(address, Response.command))
     0
 
 
@@ -110,7 +110,7 @@
 
     >>> address = getUtility(IUserManager).create_address(
     ...     '[e-mail]')
-    >>> response_set.todays_count(address, Response.command)
+    >>> print(response_set.todays_count(address, Response.command))
     0
     >>> print(response_set.last_response(address, Response.command))
     None

=== modified file 'src/mailman/model/docs/mailinglist.rst'
--- src/mailman/model/docs/mailinglist.rst	2014-04-28 15:23:35 +0000
+++ src/mailman/model/docs/mailinglist.rst	2014-10-31 13:23:25 +0000
@@ -50,7 +50,7 @@
 
 Both addresses appear on the roster of members.
 
-    >>> for member in mlist.members.members:
+    >>> for member in sorted(mlist.members.members, key=lambda m: m.address.email):
     ...     print(member)
     <Member: [e-mail] on [e-mail] as MemberRole.member>
     <Member: [e-mail] on [e-mail] as MemberRole.member>
@@ -72,7 +72,7 @@
 an owner and a moderator.
 ::
 
-    >>> for member in mlist.owners.members:
+    >>> for member in sorted(mlist.owners.members, key=lambda m: m.address.email):
     ...     print(member)
     <Member: [e-mail] on [e-mail] as MemberRole.owner>
     <Member: [e-mail] on [e-mail] as MemberRole.owner>
@@ -87,13 +87,13 @@
 ::
 
     >>> roster = mlist.get_roster(MemberRole.member)
-    >>> for member in roster.members:
+    >>> for member in sorted(roster.members, key=lambda m: m.address.email):
     ...     print(member)
     <Member: [e-mail] on [e-mail] as MemberRole.member>
     <Member: [e-mail] on [e-mail] as MemberRole.member>
 
     >>> roster = mlist.get_roster(MemberRole.owner)
-    >>> for member in roster.members:
+    >>> for member in sorted(roster.members, key=lambda m: m.address.email):
     ...     print(member)
     <Member: [e-mail] on [e-mail] as MemberRole.owner>
     <Member: [e-mail] on [e-mail] as MemberRole.owner>
@@ -122,7 +122,7 @@
     >>> mlist.subscribe(user)
     <Member: Dave Person <[e-mail]> on [e-mail]
              as MemberRole.member>
-    >>> for member in mlist.members.members:
+    >>> for member in sorted(mlist.members.members, key=lambda m: m.address.email):
     ...     print(member)
     <Member: [e-mail] on [e-mail] as MemberRole.member>
     <Member: [e-mail] on [e-mail] as MemberRole.member>
@@ -133,7 +133,7 @@
     >>> new_address.verified_on = now()
     >>> user.preferred_address = new_address
 
-    >>> for member in mlist.members.members:
+    >>> for member in sorted(mlist.members.members, key=lambda m: m.address.email):
     ...     print(member)
     <Member: [e-mail] on [e-mail] as MemberRole.member>
     <Member: [e-mail] on [e-mail] as MemberRole.member>

=== modified file 'src/mailman/model/docs/requests.rst'
--- src/mailman/model/docs/requests.rst	2014-04-28 15:23:35 +0000
+++ src/mailman/model/docs/requests.rst	2014-10-31 13:23:25 +0000
@@ -35,7 +35,7 @@
 
 The list's requests database starts out empty.
 
-    >>> requests.count
+    >>> print(requests.count)
     0
     >>> dump_list(requests.held_requests)
     *Empty*
@@ -68,21 +68,21 @@
 
 We can see the total number of requests being held.
 
-    >>> requests.count
+    >>> print(requests.count)
     3
 
 We can also see the number of requests being held by request type.
 
-    >>> requests.count_of(RequestType.subscription)
+    >>> print(requests.count_of(RequestType.subscription))
     1
-    >>> requests.count_of(RequestType.unsubscription)
+    >>> print(requests.count_of(RequestType.unsubscription))
     1
 
 We can also see when there are multiple held requests of a particular type.
 
-    >>> requests.hold_request(RequestType.held_message, 'hold_4')
+    >>> print(requests.hold_request(RequestType.held_message, 'hold_4'))
     4
-    >>> requests.count_of(RequestType.held_message)
+    >>> print(requests.count_of(RequestType.held_message))
     2
 
 We can ask the requests database for a specific request, by providing the id
@@ -132,7 +132,7 @@
 To make it easier to find specific requests, the list requests can be iterated
 over by type.
 
-    >>> requests.count_of(RequestType.held_message)
+    >>> print(requests.count_of(RequestType.held_message))
     3
     >>> for request in requests.of_type(RequestType.held_message):
     ...     key, data = requests.get_request(request.id)
@@ -154,10 +154,10 @@
 Once a specific request has been handled, it can be deleted from the requests
 database.
 
-    >>> requests.count
+    >>> print(requests.count)
     5
     >>> requests.delete_request(2)
-    >>> requests.count
+    >>> print(requests.count)
     4
 
 Request 2 is no longer in the database.
@@ -167,5 +167,5 @@
 
     >>> for request in requests.held_requests:
     ...     requests.delete_request(request.id)
-    >>> requests.count
+    >>> print(requests.count)
     0

=== modified file 'src/mailman/model/domain.py'
--- src/mailman/model/domain.py	2014-09-22 18:47:02 +0000
+++ src/mailman/model/domain.py	2014-10-31 13:23:25 +0000
@@ -48,7 +48,7 @@
 
     id = Column(Integer, primary_key=True)
 
-    mail_host = Column(Unicode)
+    mail_host = Column(Unicode) # TODO: add index?
     base_url = Column(Unicode)
     description = Column(Unicode)
     contact_address = Column(Unicode)
@@ -95,7 +95,8 @@
     def mailing_lists(self, store):
         """See `IDomain`."""
         mailing_lists = store.query(MailingList).filter(
-            MailingList.mail_host == self.mail_host)
+            MailingList.mail_host == self.mail_host
+            ).order_by(MailingList._list_id)
         for mlist in mailing_lists:
             yield mlist
 
@@ -170,7 +171,7 @@
     @dbconnection
     def __iter__(self, store):
         """See `IDomainManager`."""
-        for domain in store.query(Domain).all():
+        for domain in store.query(Domain).order_by(Domain.mail_host).all():
             yield domain
 
     @dbconnection

=== modified file 'src/mailman/model/listmanager.py'
--- src/mailman/model/listmanager.py	2014-09-21 21:06:40 +0000
+++ src/mailman/model/listmanager.py	2014-10-31 13:23:25 +0000
@@ -86,7 +86,8 @@
     @dbconnection
     def mailing_lists(self, store):
         """See `IListManager`."""
-        for mlist in store.query(MailingList).all():
+        for mlist in store.query(MailingList).order_by(
+            MailingList._list_id).all():
             yield mlist
 
     @dbconnection

=== modified file 'src/mailman/model/mailinglist.py'
--- src/mailman/model/mailinglist.py	2014-10-10 04:59:43 +0000
+++ src/mailman/model/mailinglist.py	2014-10-31 13:23:25 +0000
@@ -504,10 +504,9 @@
 
     id = Column(Integer, primary_key=True)
 
-    mailing_list_id = Column(Integer,
-                             ForeignKey('mailinglist.id'),
-                             index=True,
-                             nullable=False)
+    mailing_list_id = Column(
+        Integer, ForeignKey('mailinglist.id'),
+        index=True, nullable=False)
     mailing_list = relationship('MailingList', backref='acceptable_alias')
     alias = Column(Unicode, index=True, nullable=False)
 
@@ -561,9 +560,11 @@
 
     id = Column(Integer, primary_key=True)
 
-    mailing_list_id = Column(Integer, ForeignKey('mailinglist.id'),
-                             index=True, nullable=False)
+    mailing_list_id = Column(
+        Integer, ForeignKey('mailinglist.id'),
+        index=True, nullable=False)
     mailing_list = relationship('MailingList')
+
     name = Column(Unicode, nullable=False)
     _is_enabled = Column(Boolean)
 

=== modified file 'src/mailman/model/user.py'
--- src/mailman/model/user.py	2014-10-10 04:59:43 +0000
+++ src/mailman/model/user.py	2014-10-31 13:23:25 +0000
@@ -56,7 +56,7 @@
 
     id = Column(Integer, primary_key=True)
     display_name = Column(Unicode)
-    _password = Column('password', LargeBinary) # TODO : was RawStr()
+    _password = Column('password', LargeBinary)
     _user_id = Column(UUID, index=True)
     _created_on = Column(DateTime)
 
@@ -68,7 +68,7 @@
         Integer,
         ForeignKey('address.id', use_alter=True,
                    name='_preferred_address',
-                   ondelete="SET NULL"))
+                   ondelete='SET NULL'))
 
     _preferred_address = relationship(
         'Address', primaryjoin=(_preferred_address_id==Address.id),

=== modified file 'src/mailman/rest/docs/moderation.rst'
--- src/mailman/rest/docs/moderation.rst	2014-04-28 15:23:35 +0000
+++ src/mailman/rest/docs/moderation.rst	2014-10-31 13:23:25 +0000
@@ -226,10 +226,9 @@
 
     >>> from mailman.app.moderator import hold_subscription
     >>> from mailman.interfaces.member import DeliveryMode
-    >>> hold_subscription(
+    >>> sub_req_id = hold_subscription(
     ...     ant, '[e-mail]', 'Anne Person',
     ...     'password', DeliveryMode.regular, 'en')
-    1
     >>> transaction.commit()
 
 The subscription request is available from the mailing list.
@@ -242,7 +241,7 @@
         http_etag: "..."
         language: en
         password: password
-        request_id: 1
+        request_id: ...
         type: subscription
         when: 2005-08-01T07:49:23
     http_etag: "..."
@@ -259,8 +258,7 @@
     >>> from mailman.app.moderator import hold_unsubscription
     >>> bart = add_member(ant, '[e-mail]', 'Bart Person',
     ...     'password', DeliveryMode.regular, 'en')
-    >>> hold_unsubscription(ant, '[e-mail]')
-    2
+    >>> unsub_req_id = hold_unsubscription(ant, '[e-mail]')
     >>> transaction.commit()
 
 The unsubscription request is also available from the mailing list.
@@ -273,13 +271,13 @@
         http_etag: "..."
         language: en
         password: password
-        request_id: 1
+        request_id: ...
         type: subscription
         when: 2005-08-01T07:49:23
     entry 1:
         address: [e-mail]
         http_etag: "..."
-        request_id: 2
+        request_id: ...
         type: unsubscription
     http_etag: "..."
     start: 0
@@ -292,23 +290,25 @@
 You can view an individual membership change request by providing the
 request id.  Anne's subscription request looks like this.
 
-    >>> dump_json('http://localhost:9001/3.0/lists/[e-mail]/requests/1')
+    >>> dump_json('http://localhost:9001/3.0/lists/[e-mail]/'
+    ...           'requests/{}'.format(sub_req_id))
     address: [e-mail]
     delivery_mode: regular
     display_name: Anne Person
     http_etag: "..."
     language: en
     password: password
-    request_id: 1
+    request_id: ...
     type: subscription
     when: 2005-08-01T07:49:23
 
 Bart's unsubscription request looks like this.
 
-    >>> dump_json('http://localhost:9001/3.0/lists/[e-mail]/requests/2')
+    >>> dump_json('http://localhost:9001/3.0/lists/[e-mail]/'
+    ...           'requests/{}'.format(unsub_req_id))
     address: [e-mail]
     http_etag: "..."
-    request_id: 2
+    request_id: ...
     type: unsubscription
 
 
@@ -328,9 +328,8 @@
 Anne's subscription request is accepted.
 
     >>> dump_json('http://localhost:9001/3.0/lists/'
-    ...           '[e-mail]/requests/1', {
-    ...           'action': 'accept',
-    ...           })
+    ...           '[e-mail]/requests/{}'.format(sub_req_id),
+    ...           {'action': 'accept'})
     content-length: 0
     date: ...
     server: ...
@@ -347,9 +346,8 @@
 Bart's unsubscription request is discarded.
 
     >>> dump_json('http://localhost:9001/3.0/lists/'
-    ...           '[e-mail]/requests/2', {
-    ...           'action': 'discard',
-    ...           })
+    ...           '[e-mail]/requests/{}'.format(unsub_req_id),
+    ...           {'action': 'discard'})
     content-length: 0
     date: ...
     server: ...

=== modified file 'src/mailman/testing/layers.py'
--- src/mailman/testing/layers.py	2014-10-06 17:17:50 +0000
+++ src/mailman/testing/layers.py	2014-10-31 13:23:25 +0000
@@ -47,16 +47,13 @@
 
 from lazr.config import as_boolean
 from pkg_resources import resource_string
-from sqlalchemy import MetaData
 from textwrap import dedent
-from zope import event
 from zope.component import getUtility
 
 from mailman.config import config
 from mailman.core import initialize
 from mailman.core.initialize import INHIBIT_CONFIG_FILE
 from mailman.core.logging import get_handler
-from mailman.database.model import Model
 from mailman.database.transaction import transaction
 from mailman.interfaces.domain import IDomainManager
 from mailman.testing.helpers import (
@@ -101,9 +98,7 @@
         # Set up the basic configuration stuff.  Turn off path creation until
         # we've pushed the testing config.
         config.create_paths = False
-        if not event.subscribers:
-            # only if not yet initialized by another layer
-            initialize.initialize_1(INHIBIT_CONFIG_FILE)
+        initialize.initialize_1(INHIBIT_CONFIG_FILE)
         assert cls.var_dir is None, 'Layer already set up'
         # Calculate a temporary VAR_DIR directory so that run-time artifacts
         # of the tests won't tread on the installation's data.  This also
@@ -195,10 +190,11 @@
     @classmethod
     def tearDown(cls):
         assert cls.var_dir is not None, 'Layer not set up'
-        # Reset the test database after the tests are done so that there is no
+        reset_the_world()
+        # Destroy the test database after the tests are done so that there is no
         # data in case the tests are rerun with a database layer like mysql or
         # postgresql which are not deleted in teardown.
-        reset_the_world()
+        config.db.destroy()
         config.pop('test config')
         shutil.rmtree(cls.var_dir)
         cls.var_dir = None

=== modified file 'src/mailman/testing/testing.cfg'
--- src/mailman/testing/testing.cfg	2014-10-10 04:59:43 +0000
+++ src/mailman/testing/testing.cfg	2014-10-31 13:23:25 +0000
@@ -20,7 +20,7 @@
 # For testing against PostgreSQL.
 # [database]
 # class: mailman.database.postgresql.PostgreSQLDatabase
-# url: postgresql://maxking:maxking@localhost/mailman_test
+# url: postgresql://$USER:$USER@localhost/mailman_test
 
 [mailman]
 site_owner: [e-mail]

=== modified file 'src/mailman/utilities/importer.py'
--- src/mailman/utilities/importer.py	2014-09-28 00:17:05 +0000
+++ src/mailman/utilities/importer.py	2014-10-31 13:23:25 +0000
@@ -175,6 +175,7 @@
     allow_list_posts=bool,
     include_rfc2369_headers=bool,
     nntp_prefix_subject_too=bool,
+    encode_ascii_prefixes=bool,
     )
 
 

=== modified file 'src/mailman/utilities/tests/test_import.py'
--- src/mailman/utilities/tests/test_import.py	2014-04-28 15:23:35 +0000
+++ src/mailman/utilities/tests/test_import.py	2014-10-31 13:23:25 +0000
@@ -34,6 +34,7 @@
 from datetime import timedelta, datetime
 from enum import Enum
 from pkg_resources import resource_filename
+from sqlalchemy.exc import IntegrityError
 from zope.component import getUtility
 
 from mailman.app.lifecycle import create_list
@@ -291,6 +292,15 @@
         else:
             self.fail('Import21Error was not raised')
 
+    def test_encode_ascii_prefixes(self):
+        self._pckdict['encode_ascii_prefixes'] = 2
+        self.assertEqual(self._mlist.encode_ascii_prefixes, False)
+        try:
+            self._import()
+        except IntegrityError as e:
+            self.fail(e)
+        self.assertEqual(self._mlist.encode_ascii_prefixes, True)
+
 
 

 class TestArchiveImport(unittest.TestCase):