Merge lp:~raj-abhilash1/mailman/sqlalchemy into lp:mailman

Proposed by Abhilash Raj
Status: Merged
Merged at revision: 7253
Proposed branch: lp:~raj-abhilash1/mailman/sqlalchemy
Merge into: lp:mailman
Diff against target: 6388 lines (+1161/-3567)
66 files modified
MANIFEST.in (+2/-0)
setup.py (+2/-1)
src/mailman/app/subscriptions.py (+7/-9)
src/mailman/bin/tests/test_master.py (+1/-1)
src/mailman/commands/docs/conf.rst (+3/-1)
src/mailman/config/config.py (+5/-8)
src/mailman/config/configure.zcml (+0/-20)
src/mailman/config/schema.cfg (+13/-4)
src/mailman/core/logging.py (+4/-0)
src/mailman/database/alembic/__init__.py (+32/-0)
src/mailman/database/alembic/env.py (+80/-0)
src/mailman/database/alembic/script.py.mako (+22/-0)
src/mailman/database/alembic/versions/51b7f92bd06c_initial.py (+34/-0)
src/mailman/database/base.py (+14/-126)
src/mailman/database/docs/migration.rst (+0/-207)
src/mailman/database/factory.py (+72/-25)
src/mailman/database/model.py (+30/-40)
src/mailman/database/postgresql.py (+6/-59)
src/mailman/database/schema/helpers.py (+0/-43)
src/mailman/database/schema/mm_00000000000000_base.py (+0/-35)
src/mailman/database/schema/mm_20120407000000.py (+0/-212)
src/mailman/database/schema/mm_20121015000000.py (+0/-95)
src/mailman/database/schema/mm_20130406000000.py (+0/-65)
src/mailman/database/schema/postgres.sql (+0/-349)
src/mailman/database/schema/sqlite.sql (+0/-327)
src/mailman/database/schema/sqlite_20120407000000_01.sql (+0/-280)
src/mailman/database/schema/sqlite_20121015000000_01.sql (+0/-230)
src/mailman/database/schema/sqlite_20130406000000_01.sql (+0/-46)
src/mailman/database/sqlite.py (+5/-41)
src/mailman/database/tests/data/migration_postgres_1.sql (+0/-133)
src/mailman/database/tests/data/migration_sqlite_1.sql (+0/-133)
src/mailman/database/tests/test_factory.py (+162/-0)
src/mailman/database/tests/test_migrations.py (+0/-506)
src/mailman/database/types.py (+57/-30)
src/mailman/interfaces/database.py (+1/-7)
src/mailman/interfaces/messages.py (+1/-1)
src/mailman/model/address.py (+17/-12)
src/mailman/model/autorespond.py (+25/-23)
src/mailman/model/bans.py (+16/-13)
src/mailman/model/bounce.py (+13/-10)
src/mailman/model/digests.py (+13/-10)
src/mailman/model/docs/messagestore.rst (+3/-2)
src/mailman/model/domain.py (+15/-14)
src/mailman/model/language.py (+7/-5)
src/mailman/model/listmanager.py (+12/-13)
src/mailman/model/mailinglist.py (+230/-216)
src/mailman/model/member.py (+19/-17)
src/mailman/model/message.py (+7/-5)
src/mailman/model/messagestore.py (+15/-14)
src/mailman/model/mime.py (+11/-8)
src/mailman/model/pending.py (+34/-27)
src/mailman/model/preferences.py (+11/-9)
src/mailman/model/requests.py (+26/-17)
src/mailman/model/roster.py (+19/-26)
src/mailman/model/tests/test_listmanager.py (+12/-4)
src/mailman/model/tests/test_requests.py (+2/-2)
src/mailman/model/uid.py (+9/-5)
src/mailman/model/user.py (+32/-17)
src/mailman/model/usermanager.py (+9/-9)
src/mailman/model/version.py (+0/-44)
src/mailman/rest/validator.py (+1/-1)
src/mailman/styles/base.py (+2/-6)
src/mailman/testing/layers.py (+10/-1)
src/mailman/testing/testing.cfg (+1/-1)
src/mailman/utilities/importer.py (+23/-1)
src/mailman/utilities/modules.py (+14/-1)
To merge this branch: bzr merge lp:~raj-abhilash1/mailman/sqlalchemy
Reviewer Review Type Date Requested Status
Mailman Coders Pending
Review via email: mp+235329@code.launchpad.net

Description of the change

Replace storm with sqlalchemy.

To post a comment you must log in.
lp:~raj-abhilash1/mailman/sqlalchemy updated
7266. By Abhilash Raj

merge test fix from Barry

7267. By Abhilash Raj

merge updates from to sqlalchemy branch

7268. By Abhilash Raj

merge updated with some removal from barry

7269. By Abhilash Raj

added support for migrations via alembic

7270. By Abhilash Raj

added license block for the new file

7271. By Abhilash Raj

no need to stamp the testing db

7272. By Abhilash Raj

add new command `mailman migrate` to migrate the new schema on the old database

7273. By Abhilash Raj

add autogenerate switch that generates to create migration scripts automatically

7274. By Abhilash Raj

* fixed a bug where alemnic could not find its migrations directory
* add a new method in base database to stamp with latest alembic version

7275. By Abhilash Raj

Add support for postgresql

* revert changes in message_id_has encoding by barry
* Change message_id_hash column to LargeBinary
  (from previously mistaken one i.e.unicode)
* add missing import in database/types.py
* fix a bug in database/Model.py, transaction has no
  method abort(), instead it is rollback()

7276. By Abhilash Raj

Merge alembic setup from abompard

7277. By Abhilash Raj

Merge barry\'s branch with test fixes and clean code

7278. By Abhilash Raj

add central alembic config

7279. By Abhilash Raj

fix database reset error due to foreign key constraint between user and address tables

7280. By Abhilash Raj

merge branch from abompard

7281. By Abhilash Raj

* remove migrate command
* remove alembic.cfg, move contents to schema.cfg
* fix import errors in src/mailman/model/language.py
* add indexes
* change the previously wrong written tablename autoresponserecord
* change alembic_cfg to use schema.cfg instead of alembic.cfg

7282. By Abhilash Raj

changes from abompard to fix the unit tests

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'MANIFEST.in'
2--- MANIFEST.in 2011-06-11 18:16:04 +0000
3+++ MANIFEST.in 2014-10-11 02:14:38 +0000
4@@ -13,3 +13,5 @@
5 prune parts
6 include MANIFEST.in
7 include src/mailman/testing/config.pck
8+include src/mailman/database/alembic/script.py.mako
9+include src/mailman/database/alembic/versions/*.py
10
11=== modified file 'setup.py'
12--- setup.py 2014-04-15 16:06:01 +0000
13+++ setup.py 2014-10-11 02:14:38 +0000
14@@ -93,6 +93,7 @@
15 'console_scripts' : list(scripts),
16 },
17 install_requires = [
18+ 'alembic',
19 'enum34',
20 'flufl.bounce',
21 'flufl.i18n',
22@@ -104,7 +105,7 @@
23 'nose2',
24 'passlib',
25 'restish',
26- 'storm',
27+ 'sqlalchemy',
28 'zope.component',
29 'zope.configuration',
30 'zope.event',
31
32=== modified file 'src/mailman/app/subscriptions.py'
33--- src/mailman/app/subscriptions.py 2014-04-15 14:03:39 +0000
34+++ src/mailman/app/subscriptions.py 2014-10-11 02:14:38 +0000
35@@ -28,7 +28,7 @@
36
37 from operator import attrgetter
38 from passlib.utils import generate_password as generate
39-from storm.expr import And, Or
40+from sqlalchemy import and_, or_
41 from uuid import UUID
42 from zope.component import getUtility
43 from zope.interface import implementer
44@@ -88,9 +88,7 @@
45 @dbconnection
46 def get_member(self, store, member_id):
47 """See `ISubscriptionService`."""
48- members = store.find(
49- Member,
50- Member._member_id == member_id)
51+ members = store.query(Member).filter(Member._member_id == member_id)
52 if members.count() == 0:
53 return None
54 else:
55@@ -117,8 +115,8 @@
56 # This probably could be made more efficient.
57 if address is None or user is None:
58 return []
59- query.append(Or(Member.address_id == address.id,
60- Member.user_id == user.id))
61+ query.append(or_(Member.address_id == address.id,
62+ Member.user_id == user.id))
63 else:
64 # subscriber is a user id.
65 user = user_manager.get_user_by_id(subscriber)
66@@ -126,15 +124,15 @@
67 if address.id is not None)
68 if len(address_ids) == 0 or user is None:
69 return []
70- query.append(Or(Member.user_id == user.id,
71- Member.address_id.is_in(address_ids)))
72+ query.append(or_(Member.user_id == user.id,
73+ Member.address_id.in_(address_ids)))
74 # Calculate the rest of the query expression, which will get And'd
75 # with the Or clause above (if there is one).
76 if list_id is not None:
77 query.append(Member.list_id == list_id)
78 if role is not None:
79 query.append(Member.role == role)
80- results = store.find(Member, And(*query))
81+ results = store.query(Member).filter(and_(*query))
82 return sorted(results, key=_membership_sort_key)
83
84 def __iter__(self):
85
86=== modified file 'src/mailman/bin/tests/test_master.py'
87--- src/mailman/bin/tests/test_master.py 2014-04-28 15:23:35 +0000
88+++ src/mailman/bin/tests/test_master.py 2014-10-11 02:14:38 +0000
89@@ -55,7 +55,7 @@
90 lock = master.acquire_lock_1(False, self.lock_file)
91 is_locked = lock.is_locked
92 lock.unlock()
93- self.failUnless(is_locked)
94+ self.assertTrue(is_locked)
95
96 def test_master_state(self):
97 my_lock = Lock(self.lock_file)
98
99=== modified file 'src/mailman/commands/docs/conf.rst'
100--- src/mailman/commands/docs/conf.rst 2013-09-01 15:08:46 +0000
101+++ src/mailman/commands/docs/conf.rst 2014-10-11 02:14:38 +0000
102@@ -22,7 +22,7 @@
103 command without any options.
104
105 >>> command.process(FakeArgs)
106- [logging.archiver] path: mailman.log
107+ [alembic] script_location: mailman.database:alembic
108 ...
109 [passwords] password_length: 8
110 ...
111@@ -43,12 +43,14 @@
112 >>> FakeArgs.section = None
113 >>> FakeArgs.key = 'path'
114 >>> command.process(FakeArgs)
115+ [logging.dbmigration] path: mailman.log
116 [logging.archiver] path: mailman.log
117 [logging.locks] path: mailman.log
118 [logging.mischief] path: mailman.log
119 [logging.config] path: mailman.log
120 [logging.error] path: mailman.log
121 [logging.smtp] path: smtp.log
122+ [logging.database] path: mailman.log
123 [logging.http] path: mailman.log
124 [logging.root] path: mailman.log
125 [logging.fromusenet] path: mailman.log
126
127=== modified file 'src/mailman/config/config.py'
128--- src/mailman/config/config.py 2014-03-02 22:59:30 +0000
129+++ src/mailman/config/config.py 2014-10-11 02:14:38 +0000
130@@ -33,7 +33,7 @@
131 from ConfigParser import SafeConfigParser
132 from flufl.lock import Lock
133 from lazr.config import ConfigSchema, as_boolean
134-from pkg_resources import resource_filename, resource_stream, resource_string
135+from pkg_resources import resource_stream, resource_string
136 from string import Template
137 from zope.component import getUtility
138 from zope.event import notify
139@@ -46,7 +46,7 @@
140 ConfigurationUpdatedEvent, IConfiguration, MissingConfigurationFileError)
141 from mailman.interfaces.languages import ILanguageManager
142 from mailman.utilities.filesystem import makedirs
143-from mailman.utilities.modules import call_name
144+from mailman.utilities.modules import call_name, expand_path
145
146
147 SPACE = ' '
148@@ -87,6 +87,7 @@
149 self.pipelines = {}
150 self.commands = {}
151 self.password_context = None
152+ self.initialized = False
153
154 def _clear(self):
155 """Clear the cached configuration variables."""
156@@ -136,6 +137,7 @@
157 # Expand and set up all directories.
158 self._expand_paths()
159 self.ensure_directories_exist()
160+ self.initialized = True
161 notify(ConfigurationUpdatedEvent(self))
162
163 def _expand_paths(self):
164@@ -304,12 +306,7 @@
165 :return: A `ConfigParser` instance.
166 """
167 # Is the context coming from a file system or Python path?
168- if path.startswith('python:'):
169- resource_path = path[7:]
170- package, dot, resource = resource_path.rpartition('.')
171- cfg_path = resource_filename(package, resource + '.cfg')
172- else:
173- cfg_path = path
174+ cfg_path = expand_path(path)
175 parser = SafeConfigParser()
176 files = parser.read(cfg_path)
177 if files != [cfg_path]:
178
179=== modified file 'src/mailman/config/configure.zcml'
180--- src/mailman/config/configure.zcml 2013-11-26 02:26:15 +0000
181+++ src/mailman/config/configure.zcml 2014-10-11 02:14:38 +0000
182@@ -40,20 +40,6 @@
183 factory="mailman.model.requests.ListRequests"
184 />
185
186- <adapter
187- for="mailman.interfaces.database.IDatabase"
188- provides="mailman.interfaces.database.ITemporaryDatabase"
189- factory="mailman.database.sqlite.make_temporary"
190- name="sqlite"
191- />
192-
193- <adapter
194- for="mailman.interfaces.database.IDatabase"
195- provides="mailman.interfaces.database.ITemporaryDatabase"
196- factory="mailman.database.postgresql.make_temporary"
197- name="postgres"
198- />
199-
200 <utility
201 provides="mailman.interfaces.bounce.IBounceProcessor"
202 factory="mailman.model.bounce.BounceProcessor"
203@@ -72,12 +58,6 @@
204 />
205
206 <utility
207- provides="mailman.interfaces.database.IDatabaseFactory"
208- factory="mailman.database.factory.DatabaseTemporaryFactory"
209- name="temporary"
210- />
211-
212- <utility
213 provides="mailman.interfaces.domain.IDomainManager"
214 factory="mailman.model.domain.DomainManager"
215 />
216
217=== modified file 'src/mailman/config/schema.cfg'
218--- src/mailman/config/schema.cfg 2014-01-01 14:59:42 +0000
219+++ src/mailman/config/schema.cfg 2014-10-11 02:14:38 +0000
220@@ -204,9 +204,6 @@
221 url: sqlite:///$DATA_DIR/mailman.db
222 debug: no
223
224-# The module path to the migrations modules.
225-migrations_path: mailman.database.schema
226-
227 [logging.template]
228 # This defines various log settings. The options available are:
229 #
230@@ -240,6 +237,8 @@
231 # - smtp-failure -- Unsuccessful SMTP activity
232 # - subscribe -- Information about leaves/joins
233 # - vette -- Message vetting information
234+# - database -- Database activity
235+# - dbmigration -- Database migrations
236 format: %(asctime)s (%(process)d) %(message)s
237 datefmt: %b %d %H:%M:%S %Y
238 propagate: no
239@@ -304,6 +303,12 @@
240
241 [logging.vette]
242
243+[logging.database]
244+level: warn
245+
246+[logging.dbmigration]
247+level: warn
248+
249
250 [webservice]
251 # The hostname at which admin web service resources are exposed.
252@@ -532,7 +537,7 @@
253 # following values.
254
255 # The class implementing the IArchiver interface.
256-class:
257+class:
258
259 # Set this to 'yes' to enable the archiver.
260 enable: no
261@@ -640,3 +645,7 @@
262 CC X-Original-CC
263 Content-Transfer-Encoding X-Original-Content-Transfer-Encoding
264 MIME-Version X-MIME-Version
265+
266+[alembic]
267+# path to migration scripts
268+script_location = mailman.database:alembic
269
270=== modified file 'src/mailman/core/logging.py'
271--- src/mailman/core/logging.py 2014-04-28 15:23:35 +0000
272+++ src/mailman/core/logging.py 2014-10-11 02:14:38 +0000
273@@ -126,6 +126,10 @@
274 continue
275 if sub_name == 'locks':
276 log = logging.getLogger('flufl.lock')
277+ elif sub_name == 'database':
278+ log = logging.getLogger('sqlalchemy')
279+ elif sub_name == 'dbmigration':
280+ log = logging.getLogger('alembic')
281 else:
282 logger_name = 'mailman.' + sub_name
283 log = logging.getLogger(logger_name)
284
285=== added directory 'src/mailman/database/alembic'
286=== added file 'src/mailman/database/alembic/__init__.py'
287--- src/mailman/database/alembic/__init__.py 1970-01-01 00:00:00 +0000
288+++ src/mailman/database/alembic/__init__.py 2014-10-11 02:14:38 +0000
289@@ -0,0 +1,32 @@
290+# Copyright (C) 2014 by the Free Software Foundation, Inc.
291+#
292+# This file is part of GNU Mailman.
293+#
294+# GNU Mailman is free software: you can redistribute it and/or modify it under
295+# the terms of the GNU General Public License as published by the Free
296+# Software Foundation, either version 3 of the License, or (at your option)
297+# any later version.
298+#
299+# GNU Mailman is distributed in the hope that it will be useful, but WITHOUT
300+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
301+# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
302+# more details.
303+#
304+# You should have received a copy of the GNU General Public License along with
305+# GNU Mailman. If not, see <http://www.gnu.org/licenses/>.
306+
307+"Alembic config init."
308+
309+from __future__ import absolute_import, print_function, unicode_literals
310+
311+__metaclass__ = type
312+__all__ = [
313+ 'alembic_cfg'
314+]
315+
316+
317+from alembic.config import Config
318+from mailman.utilities.modules import expand_path
319+
320+
321+alembic_cfg=Config(expand_path("python:mailman.config.schema"))
322
323=== added file 'src/mailman/database/alembic/env.py'
324--- src/mailman/database/alembic/env.py 1970-01-01 00:00:00 +0000
325+++ src/mailman/database/alembic/env.py 2014-10-11 02:14:38 +0000
326@@ -0,0 +1,80 @@
327+# Copyright (C) 2014 by the Free Software Foundation, Inc.
328+#
329+# This file is part of GNU Mailman.
330+#
331+# GNU Mailman is free software: you can redistribute it and/or modify it under
332+# the terms of the GNU General Public License as published by the Free
333+# Software Foundation, either version 3 of the License, or (at your option)
334+# any later version.
335+#
336+# GNU Mailman is distributed in the hope that it will be useful, but WITHOUT
337+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
338+# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
339+# more details.
340+#
341+# You should have received a copy of the GNU General Public License along with
342+# GNU Mailman. If not, see <http://www.gnu.org/licenses/>.
343+
344+"""Alembic migration environment."""
345+
346+from __future__ import absolute_import, print_function, unicode_literals
347+
348+__metaclass__ = type
349+__all__ = [
350+ 'run_migrations_offline',
351+ 'run_migrations_online',
352+ ]
353+
354+
355+from alembic import context
356+from contextlib import closing
357+from sqlalchemy import create_engine
358+
359+from mailman.core import initialize
360+from mailman.config import config
361+from mailman.database.alembic import alembic_cfg
362+from mailman.database.model import Model
363+from mailman.utilities.string import expand
364+
365+if not config.initialized:
366+ initialize.initialize_1(context.config.config_file_name)
367+
368+
369+
370
371+def run_migrations_offline():
372+ """Run migrations in 'offline' mode.
373+
374+ This configures the context with just a URL and not an Engine,
375+ though an Engine is acceptable here as well. By skipping the Engine
376+ creation we don't even need a DBAPI to be available.
377+
378+ Calls to context.execute() here emit the given string to the script
379+ output.
380+ """
381+ url = expand(config.database.url, config.paths)
382+ context.configure(url=url, target_metadata=Model.metadata)
383+ with context.begin_transaction():
384+ context.run_migrations()
385+
386+
387+def run_migrations_online():
388+ """Run migrations in 'online' mode.
389+
390+ In this scenario we need to create an Engine and associate a
391+ connection with the context.
392+ """
393+ url = expand(config.database.url, config.paths)
394+ engine = create_engine(url)
395+
396+ connection = engine.connect()
397+ with closing(connection):
398+ context.configure(
399+ connection=connection, target_metadata=Model.metadata)
400+ with context.begin_transaction():
401+ context.run_migrations()
402+
403+
404+if context.is_offline_mode():
405+ run_migrations_offline()
406+else:
407+ run_migrations_online()
408
409=== added file 'src/mailman/database/alembic/script.py.mako'
410--- src/mailman/database/alembic/script.py.mako 1970-01-01 00:00:00 +0000
411+++ src/mailman/database/alembic/script.py.mako 2014-10-11 02:14:38 +0000
412@@ -0,0 +1,22 @@
413+"""${message}
414+
415+Revision ID: ${up_revision}
416+Revises: ${down_revision}
417+Create Date: ${create_date}
418+
419+"""
420+
421+# revision identifiers, used by Alembic.
422+revision = ${repr(up_revision)}
423+down_revision = ${repr(down_revision)}
424+
425+from alembic import op
426+import sqlalchemy as sa
427+${imports if imports else ""}
428+
429+def upgrade():
430+ ${upgrades if upgrades else "pass"}
431+
432+
433+def downgrade():
434+ ${downgrades if downgrades else "pass"}
435
436=== added directory 'src/mailman/database/alembic/versions'
437=== added file 'src/mailman/database/alembic/versions/51b7f92bd06c_initial.py'
438--- src/mailman/database/alembic/versions/51b7f92bd06c_initial.py 1970-01-01 00:00:00 +0000
439+++ src/mailman/database/alembic/versions/51b7f92bd06c_initial.py 2014-10-11 02:14:38 +0000
440@@ -0,0 +1,34 @@
441+"""initial
442+
443+Revision ID: 51b7f92bd06c
444+Revises: None
445+Create Date: 2014-10-10 09:53:35.624472
446+
447+"""
448+
449+# revision identifiers, used by Alembic.
450+revision = '51b7f92bd06c'
451+down_revision = None
452+
453+from alembic import op
454+import sqlalchemy as sa
455+
456+
457+def upgrade():
458+ ### commands auto generated by Alembic - please adjust! ###
459+ op.drop_table('version')
460+ if op.get_bind().dialect.name != "sqlite":
461+ # SQLite does not support dropping columns
462+ op.drop_column('mailinglist', 'acceptable_aliases_id')
463+ op.create_index(op.f('ix_user__user_id'), 'user', ['_user_id'], unique=False)
464+ op.drop_index('ix_user_user_id', table_name='user')
465+ ### end Alembic commands ###
466+
467+
468+def downgrade():
469+ ### commands auto generated by Alembic - please adjust! ###
470+ op.create_table('version')
471+ op.create_index('ix_user_user_id', 'user', ['_user_id'], unique=False)
472+ op.drop_index(op.f('ix_user__user_id'), table_name='user')
473+ op.add_column('mailinglist', sa.Column('acceptable_aliases_id', sa.INTEGER(), nullable=True))
474+ ### end Alembic commands ###
475
476=== modified file 'src/mailman/database/base.py'
477--- src/mailman/database/base.py 2014-01-01 14:59:42 +0000
478+++ src/mailman/database/base.py 2014-10-11 02:14:38 +0000
479@@ -19,49 +19,40 @@
480
481 __metaclass__ = type
482 __all__ = [
483- 'StormBaseDatabase',
484+ 'SABaseDatabase',
485 ]
486
487
488-import os
489-import sys
490 import logging
491
492-from lazr.config import as_boolean
493-from pkg_resources import resource_listdir, resource_string
494-from storm.cache import GenerationalCache
495-from storm.locals import create_database, Store
496+from alembic import command
497+from sqlalchemy import create_engine
498+from sqlalchemy.orm import sessionmaker
499 from zope.interface import implementer
500
501 from mailman.config import config
502 from mailman.interfaces.database import IDatabase
503-from mailman.model.version import Version
504 from mailman.utilities.string import expand
505
506+
507 log = logging.getLogger('mailman.config')
508-
509 NL = '\n'
510
511
512
513
514 @implementer(IDatabase)
515-class StormBaseDatabase:
516- """The database base class for use with the Storm ORM.
517+class SABaseDatabase:
518+ """The database base class for use with SQLAlchemy.
519
520- Use this as a base class for your DB-specific derived classes.
521+ Use this as a base class for your DB-Specific derived classes.
522 """
523-
524- # Tag used to distinguish the database being used. Override this in base
525- # classes.
526- TAG = ''
527-
528 def __init__(self):
529 self.url = None
530 self.store = None
531
532 def begin(self):
533 """See `IDatabase`."""
534- # Storm takes care of this for us.
535+ # SQLAlchemy does this for us.
536 pass
537
538 def commit(self):
539@@ -72,16 +63,6 @@
540 """See `IDatabase`."""
541 self.store.rollback()
542
543- def _database_exists(self):
544- """Return True if the database exists and is initialized.
545-
546- Return False when Mailman needs to create and initialize the
547- underlying database schema.
548-
549- Base classes *must* override this.
550- """
551- raise NotImplementedError
552-
553 def _pre_reset(self, store):
554 """Clean up method for testing.
555
556@@ -113,6 +94,7 @@
557 """See `IDatabase`."""
558 # Calculate the engine url.
559 url = expand(config.database.url, config.paths)
560+ self._prepare(url)
561 log.debug('Database url: %s', url)
562 # XXX By design of SQLite, database file creation does not honor
563 # umask. See their ticket #1193:
564@@ -129,101 +111,7 @@
565 # engines, and yes, we could have chmod'd the file after the fact, but
566 # half dozen and all...
567 self.url = url
568- self._prepare(url)
569- database = create_database(url)
570- store = Store(database, GenerationalCache())
571- database.DEBUG = (as_boolean(config.database.debug)
572- if debug is None else debug)
573- self.store = store
574- store.commit()
575-
576- def load_migrations(self, until=None):
577- """Load schema migrations.
578-
579- :param until: Load only the migrations up to the specified timestamp.
580- With default value of None, load all migrations.
581- :type until: string
582- """
583- migrations_path = config.database.migrations_path
584- if '.' in migrations_path:
585- parent, dot, child = migrations_path.rpartition('.')
586- else:
587- parent = migrations_path
588- child = ''
589- # If the database does not yet exist, load the base schema.
590- filenames = sorted(resource_listdir(parent, child))
591- # Find out which schema migrations have already been loaded.
592- if self._database_exists(self.store):
593- versions = set(version.version for version in
594- self.store.find(Version, component='schema'))
595- else:
596- versions = set()
597- for filename in filenames:
598- module_fn, extension = os.path.splitext(filename)
599- if extension != '.py':
600- continue
601- parts = module_fn.split('_')
602- if len(parts) < 2:
603- continue
604- version = parts[1].strip()
605- if len(version) == 0:
606- # Not a schema migration file.
607- continue
608- if version in versions:
609- log.debug('already migrated to %s', version)
610- continue
611- if until is not None and version > until:
612- # We're done.
613- break
614- module_path = migrations_path + '.' + module_fn
615- __import__(module_path)
616- upgrade = getattr(sys.modules[module_path], 'upgrade', None)
617- if upgrade is None:
618- continue
619- log.debug('migrating db to %s: %s', version, module_path)
620- upgrade(self, self.store, version, module_path)
621- self.commit()
622-
623- def load_sql(self, store, sql):
624- """Load the given SQL into the store.
625-
626- :param store: The Storm store to load the schema into.
627- :type store: storm.locals.Store`
628- :param sql: The possibly multi-line SQL to load.
629- :type sql: string
630- """
631- # Discard all blank and comment lines.
632- lines = (line for line in sql.splitlines()
633- if line.strip() != '' and line.strip()[:2] != '--')
634- sql = NL.join(lines)
635- for statement in sql.split(';'):
636- if statement.strip() != '':
637- store.execute(statement + ';')
638-
639- def load_schema(self, store, version, filename, module_path):
640- """Load the schema from a file.
641-
642- This is a helper method for migration classes to call.
643-
644- :param store: The Storm store to load the schema into.
645- :type store: storm.locals.Store`
646- :param version: The schema version identifier of the form
647- YYYYMMDDHHMMSS.
648- :type version: string
649- :param filename: The file name containing the schema to load. Pass
650- `None` if there is no schema file to load.
651- :type filename: string
652- :param module_path: The fully qualified Python module path to the
653- migration module being loaded. This is used to record information
654- for use by the test suite.
655- :type module_path: string
656- """
657- if filename is not None:
658- contents = resource_string('mailman.database.schema', filename)
659- self.load_sql(store, contents)
660- # Add a marker that indicates the migration version being applied.
661- store.add(Version(component='schema', version=version))
662-
663- @staticmethod
664- def _make_temporary():
665- raise NotImplementedError
666+ self.engine = create_engine(url)
667+ session = sessionmaker(bind=self.engine)
668+ self.store = session()
669+ self.store.commit()
670
671=== removed directory 'src/mailman/database/docs'
672=== removed file 'src/mailman/database/docs/__init__.py'
673=== removed file 'src/mailman/database/docs/migration.rst'
674--- src/mailman/database/docs/migration.rst 2014-04-28 15:23:35 +0000
675+++ src/mailman/database/docs/migration.rst 1970-01-01 00:00:00 +0000
676@@ -1,207 +0,0 @@
677-=================
678-Schema migrations
679-=================
680-
681-The SQL database schema will over time require upgrading to support new
682-features. This is supported via schema migration.
683-
684-Migrations are embodied in individual Python classes, which themselves may
685-load SQL into the database. The naming scheme for migration files is:
686-
687- mm_YYYYMMDDHHMMSS_comment.py
688-
689-where `YYYYMMDDHHMMSS` is a required numeric year, month, day, hour, minute,
690-and second specifier providing unique ordering for processing. Only this
691-component of the file name is used to determine the ordering. The prefix is
692-required due to Python module naming requirements, but it is actually
693-ignored. `mm_` is reserved for Mailman's own use.
694-
695-The optional `comment` part of the file name can be used as a short
696-description for the migration, although comments and docstrings in the
697-migration files should be used for more detailed descriptions.
698-
699-Migrations are applied automatically when Mailman starts up, but can also be
700-applied at any time by calling in the API directly. Once applied, a
701-migration's version string is registered so it will not be applied again.
702-
703-We see that the base migration, as well as subsequent standard migrations, are
704-already applied.
705-
706- >>> from mailman.model.version import Version
707- >>> results = config.db.store.find(Version, component='schema')
708- >>> results.count()
709- 4
710- >>> versions = sorted(result.version for result in results)
711- >>> for version in versions:
712- ... print(version)
713- 00000000000000
714- 20120407000000
715- 20121015000000
716- 20130406000000
717-
718-
719-Migrations
720-==========
721-
722-Migrations can be loaded at any time, and can be found in the migrations path
723-specified in the configuration file.
724-
725-.. Create a temporary directory for the migrations::
726-
727- >>> import os, sys, tempfile
728- >>> tempdir = tempfile.mkdtemp()
729- >>> path = os.path.join(tempdir, 'migrations')
730- >>> os.makedirs(path)
731- >>> sys.path.append(tempdir)
732- >>> config.push('migrations', """
733- ... [database]
734- ... migrations_path: migrations
735- ... """)
736-
737-.. Clean this up at the end of the doctest.
738- >>> def cleanup():
739- ... import shutil
740- ... from mailman.config import config
741- ... config.pop('migrations')
742- ... shutil.rmtree(tempdir)
743- >>> cleanups.append(cleanup)
744-
745-Here is an example migrations module. The key part of this interface is the
746-``upgrade()`` method, which takes four arguments:
747-
748- * `database` - The database class, as derived from `StormBaseDatabase`
749- * `store` - The Storm `Store` object.
750- * `version` - The version string as derived from the migrations module's file
751- name. This will include only the `YYYYMMDDHHMMSS` string.
752- * `module_path` - The dotted module path to the migrations module, suitable
753- for lookup in `sys.modules`.
754-
755-This migration module just adds a marker to the `version` table.
756-
757- >>> with open(os.path.join(path, '__init__.py'), 'w') as fp:
758- ... pass
759- >>> with open(os.path.join(path, 'mm_20159999000000.py'), 'w') as fp:
760- ... print("""
761- ... from __future__ import unicode_literals
762- ... from mailman.model.version import Version
763- ... def upgrade(database, store, version, module_path):
764- ... v = Version(component='test', version=version)
765- ... store.add(v)
766- ... database.load_schema(store, version, None, module_path)
767- ... """, file=fp)
768-
769-This will load the new migration, since it hasn't been loaded before.
770-
771- >>> config.db.load_migrations()
772- >>> results = config.db.store.find(Version, component='schema')
773- >>> for result in sorted(result.version for result in results):
774- ... print(result)
775- 00000000000000
776- 20120407000000
777- 20121015000000
778- 20130406000000
779- 20159999000000
780- >>> test = config.db.store.find(Version, component='test').one()
781- >>> print(test.version)
782- 20159999000000
783-
784-Migrations will only be loaded once.
785-
786- >>> with open(os.path.join(path, 'mm_20159999000001.py'), 'w') as fp:
787- ... print("""
788- ... from __future__ import unicode_literals
789- ... from mailman.model.version import Version
790- ... _marker = 801
791- ... def upgrade(database, store, version, module_path):
792- ... global _marker
793- ... # Pad enough zeros on the left to reach 14 characters wide.
794- ... marker = '{0:=#014d}'.format(_marker)
795- ... _marker += 1
796- ... v = Version(component='test', version=marker)
797- ... store.add(v)
798- ... database.load_schema(store, version, None, module_path)
799- ... """, file=fp)
800-
801-The first time we load this new migration, we'll get the 801 marker.
802-
803- >>> config.db.load_migrations()
804- >>> results = config.db.store.find(Version, component='schema')
805- >>> for result in sorted(result.version for result in results):
806- ... print(result)
807- 00000000000000
808- 20120407000000
809- 20121015000000
810- 20130406000000
811- 20159999000000
812- 20159999000001
813- >>> test = config.db.store.find(Version, component='test')
814- >>> for marker in sorted(marker.version for marker in test):
815- ... print(marker)
816- 00000000000801
817- 20159999000000
818-
819-We do not get an 802 marker because the migration has already been loaded.
820-
821- >>> config.db.load_migrations()
822- >>> results = config.db.store.find(Version, component='schema')
823- >>> for result in sorted(result.version for result in results):
824- ... print(result)
825- 00000000000000
826- 20120407000000
827- 20121015000000
828- 20130406000000
829- 20159999000000
830- 20159999000001
831- >>> test = config.db.store.find(Version, component='test')
832- >>> for marker in sorted(marker.version for marker in test):
833- ... print(marker)
834- 00000000000801
835- 20159999000000
836-
837-
838-Partial upgrades
839-================
840-
841-It's possible (mostly for testing purposes) to only do a partial upgrade, by
842-providing a timestamp to `load_migrations()`. To demonstrate this, we add two
843-additional migrations, intended to be applied in sequential order.
844-
845- >>> from shutil import copyfile
846- >>> from mailman.testing.helpers import chdir
847- >>> with chdir(path):
848- ... copyfile('mm_20159999000000.py', 'mm_20159999000002.py')
849- ... copyfile('mm_20159999000000.py', 'mm_20159999000003.py')
850- ... copyfile('mm_20159999000000.py', 'mm_20159999000004.py')
851-
852-Now, only migrate to the ...03 timestamp.
853-
854- >>> config.db.load_migrations('20159999000003')
855-
856-You'll notice that the ...04 version is not present.
857-
858- >>> results = config.db.store.find(Version, component='schema')
859- >>> for result in sorted(result.version for result in results):
860- ... print(result)
861- 00000000000000
862- 20120407000000
863- 20121015000000
864- 20130406000000
865- 20159999000000
866- 20159999000001
867- 20159999000002
868- 20159999000003
869-
870-
871-.. cleanup:
872- Because the Version table holds schema migration data, it will not be
873- cleaned up by the standard test suite. This is generally not a problem
874- for SQLite since each test gets a new database file, but for PostgreSQL,
875- this will cause migration.rst to fail on subsequent runs. So let's just
876- clean up the database explicitly.
877-
878- >>> if config.db.TAG != 'sqlite':
879- ... results = config.db.store.execute("""
880- ... DELETE FROM version WHERE version.version >= '201299990000'
881- ... OR version.component = 'test';
882- ... """)
883- ... config.db.commit()
884
885=== modified file 'src/mailman/database/factory.py'
886--- src/mailman/database/factory.py 2014-01-01 14:59:42 +0000
887+++ src/mailman/database/factory.py 2014-10-11 02:14:38 +0000
888@@ -22,7 +22,6 @@
889 __metaclass__ = type
890 __all__ = [
891 'DatabaseFactory',
892- 'DatabaseTemporaryFactory',
893 'DatabaseTestingFactory',
894 ]
895
896@@ -30,15 +29,19 @@
897 import os
898 import types
899
900+from alembic import command
901+from alembic.migration import MigrationContext
902+from alembic.script import ScriptDirectory
903 from flufl.lock import Lock
904-from zope.component import getAdapter
905+from sqlalchemy import MetaData
906 from zope.interface import implementer
907 from zope.interface.verify import verifyObject
908
909 from mailman.config import config
910-from mailman.interfaces.database import (
911- IDatabase, IDatabaseFactory, ITemporaryDatabase)
912-from mailman.utilities.modules import call_name
913+from mailman.database.model import Model
914+from mailman.database.alembic import alembic_cfg
915+from mailman.interfaces.database import IDatabase, IDatabaseFactory
916+from mailman.utilities.modules import call_name, expand_path
917
918
919
920
921@@ -54,18 +57,78 @@
922 database = call_name(database_class)
923 verifyObject(IDatabase, database)
924 database.initialize()
925- database.load_migrations()
926+ schema_mgr = SchemaManager(database)
927+ schema_mgr.setup_db()
928 database.commit()
929 return database
930
931
932
933
934+class SchemaManager:
935+
936+ LAST_STORM_SCHEMA_VERSION = '20130406000000'
937+
938+ def __init__(self, database):
939+ self.database = database
940+ self.script = ScriptDirectory.from_config(alembic_cfg)
941+
942+ def get_storm_schema_version(self):
943+ md = MetaData()
944+ md.reflect(bind=self.database.engine)
945+ if "version" not in md.tables:
946+ return None
947+ Version = md.tables["version"]
948+ last_version = self.database.store.query(Version.c.version).filter(
949+ Version.c.component == "schema"
950+ ).order_by(Version.c.version.desc()).first()
951+ # Don't leave open transactions or they will block any schema change
952+ self.database.commit()
953+ return last_version
954+
955+ def _create(self):
956+ # initial DB creation
957+ Model.metadata.create_all(self.database.engine)
958+ self.database.commit()
959+ command.stamp(alembic_cfg, "head")
960+
961+ def _upgrade(self):
962+ command.upgrade(alembic_cfg, "head")
963+
964+ def setup_db(self):
965+ context = MigrationContext.configure(self.database.store.connection())
966+ current_rev = context.get_current_revision()
967+ head_rev = self.script.get_current_head()
968+ if current_rev == head_rev:
969+ return head_rev # already at the latest revision, nothing to do
970+ if current_rev == None:
971+ # no alembic information
972+ storm_version = self.get_storm_schema_version()
973+ if storm_version is None:
974+ # initial DB creation
975+ self._create()
976+ else:
977+ # DB from a previous version managed by Storm
978+ if storm_version.version < self.LAST_STORM_SCHEMA_VERSION:
979+ raise RuntimeError(
980+ "Upgrading while skipping beta version is "
981+ "unsupported, please install the previous "
982+ "Mailman beta release")
983+ # Run migrations to remove the Storm-specific table and
984+ # upgrade to SQLAlchemy & Alembic
985+ self._upgrade()
986+ elif current_rev != head_rev:
987+ self._upgrade()
988+ return head_rev
989+
990+
991+
992
993 def _reset(self):
994 """See `IDatabase`."""
995- from mailman.database.model import ModelMeta
996+ # Avoid a circular import at module level.
997+ from mailman.database.model import Model
998 self.store.rollback()
999 self._pre_reset(self.store)
1000- ModelMeta._reset(self.store)
1001+ Model._reset(self)
1002 self._post_reset(self.store)
1003 self.store.commit()
1004
1005@@ -81,24 +144,8 @@
1006 database = call_name(database_class)
1007 verifyObject(IDatabase, database)
1008 database.initialize()
1009- database.load_migrations()
1010+ Model.metadata.create_all(database.engine)
1011 database.commit()
1012 # Make _reset() a bound method of the database instance.
1013 database._reset = types.MethodType(_reset, database)
1014 return database
1015-
1016-
1017-
1018
1019-@implementer(IDatabaseFactory)
1020-class DatabaseTemporaryFactory:
1021- """Create a temporary database for some of the migration tests."""
1022-
1023- @staticmethod
1024- def create():
1025- """See `IDatabaseFactory`."""
1026- database_class_name = config.database['class']
1027- database = call_name(database_class_name)
1028- verifyObject(IDatabase, database)
1029- adapted_database = getAdapter(
1030- database, ITemporaryDatabase, database.TAG)
1031- return adapted_database
1032
1033=== modified file 'src/mailman/database/model.py'
1034--- src/mailman/database/model.py 2014-01-01 14:59:42 +0000
1035+++ src/mailman/database/model.py 2014-10-11 02:14:38 +0000
1036@@ -25,44 +25,34 @@
1037 ]
1038
1039
1040-from operator import attrgetter
1041-
1042-from storm.properties import PropertyPublisherMeta
1043-
1044-
1045-
1046
1047-class ModelMeta(PropertyPublisherMeta):
1048- """Do more magic on table classes."""
1049-
1050- _class_registry = set()
1051-
1052- def __init__(self, name, bases, dict):
1053- # Before we let the base class do it's thing, force an __storm_table__
1054- # property to enforce our table naming convention.
1055- self.__storm_table__ = name.lower()
1056- super(ModelMeta, self).__init__(name, bases, dict)
1057- # Register the model class so that it can be more easily cleared.
1058- # This is required by the test framework so that the corresponding
1059- # table can be reset between tests.
1060- #
1061- # The PRESERVE flag indicates whether the table should be reset or
1062- # not. We have to handle the actual Model base class explicitly
1063- # because it does not correspond to a table in the database.
1064- if not getattr(self, 'PRESERVE', False) and name != 'Model':
1065- ModelMeta._class_registry.add(self)
1066-
1067+import contextlib
1068+
1069+from sqlalchemy.ext.declarative import declarative_base
1070+
1071+from mailman.config import config
1072+
1073+
1074+class ModelMeta:
1075+ """The custom metaclass for all model base classes.
1076+
1077+ This is used in the test suite to quickly reset the database after each
1078+ test. It works by iterating over all the tables, deleting each. The test
1079+ suite will then recreate the tables before each test.
1080+ """
1081 @staticmethod
1082- def _reset(store):
1083- from mailman.config import config
1084- config.db._pre_reset(store)
1085- # Make sure this is deterministic, by sorting on the storm table name.
1086- classes = sorted(ModelMeta._class_registry,
1087- key=attrgetter('__storm_table__'))
1088- for model_class in classes:
1089- store.find(model_class).remove()
1090-
1091-
1092-
1093
1094-class Model:
1095- """Like Storm's `Storm` subclass, but with a bit extra."""
1096- __metaclass__ = ModelMeta
1097+ def _reset(db):
1098+ with contextlib.closing(config.db.engine.connect()) as connection:
1099+ transaction = connection.begin()
1100+ try:
1101+ # Delete all the tables in reverse foreign key dependency
1102+ # order. http://tinyurl.com/on8dy6f
1103+ for table in reversed(Model.metadata.sorted_tables):
1104+ connection.execute(table.delete())
1105+ except:
1106+ transaction.rollback()
1107+ raise
1108+ else:
1109+ transaction.commit()
1110+
1111+
1112+Model = declarative_base(cls=ModelMeta)
1113
1114=== modified file 'src/mailman/database/postgresql.py'
1115--- src/mailman/database/postgresql.py 2014-01-01 14:59:42 +0000
1116+++ src/mailman/database/postgresql.py 2014-10-11 02:14:38 +0000
1117@@ -22,34 +22,17 @@
1118 __metaclass__ = type
1119 __all__ = [
1120 'PostgreSQLDatabase',
1121- 'make_temporary',
1122 ]
1123
1124
1125-import types
1126-
1127-from functools import partial
1128-from operator import attrgetter
1129-from urlparse import urlsplit, urlunsplit
1130-
1131-from mailman.database.base import StormBaseDatabase
1132-from mailman.testing.helpers import configuration
1133+from mailman.database.base import SABaseDatabase
1134+from mailman.database.model import Model
1135
1136
1137
1138
1139-class PostgreSQLDatabase(StormBaseDatabase):
1140+class PostgreSQLDatabase(SABaseDatabase):
1141 """Database class for PostgreSQL."""
1142
1143- TAG = 'postgres'
1144-
1145- def _database_exists(self, store):
1146- """See `BaseDatabase`."""
1147- table_query = ('SELECT table_name FROM information_schema.tables '
1148- "WHERE table_schema = 'public'")
1149- results = store.execute(table_query)
1150- table_names = set(item[0] for item in results)
1151- return 'version' in table_names
1152-
1153 def _post_reset(self, store):
1154 """PostgreSQL-specific test suite cleanup.
1155
1156@@ -57,49 +40,13 @@
1157 restart from zero for new tests.
1158 """
1159 super(PostgreSQLDatabase, self)._post_reset(store)
1160- from mailman.database.model import ModelMeta
1161- classes = sorted(ModelMeta._class_registry,
1162- key=attrgetter('__storm_table__'))
1163+ tables = reversed(Model.metadata.sorted_tables)
1164 # Recipe adapted from
1165 # http://stackoverflow.com/questions/544791/
1166 # django-postgresql-how-to-reset-primary-key
1167- for model_class in classes:
1168+ for table in tables:
1169 store.execute("""\
1170 SELECT setval('"{0}_id_seq"', coalesce(max("id"), 1),
1171 max("id") IS NOT null)
1172 FROM "{0}";
1173- """.format(model_class.__storm_table__))
1174-
1175-
1176-
1177
1178-# Test suite adapter for ITemporaryDatabase.
1179-
1180-def _cleanup(self, store, tempdb_name):
1181- from mailman.config import config
1182- store.rollback()
1183- store.close()
1184- # From the original database connection, drop the now unused database.
1185- config.db.store.execute('DROP DATABASE {0}'.format(tempdb_name))
1186-
1187-
1188-def make_temporary(database):
1189- """Adapts by monkey patching an existing PostgreSQL IDatabase."""
1190- from mailman.config import config
1191- parts = urlsplit(config.database.url)
1192- assert parts.scheme == 'postgres'
1193- new_parts = list(parts)
1194- new_parts[2] = '/mmtest'
1195- url = urlunsplit(new_parts)
1196- # Use the existing database connection to create a new testing
1197- # database.
1198- config.db.store.execute('ABORT;')
1199- config.db.store.execute('CREATE DATABASE mmtest;')
1200- with configuration('database', url=url):
1201- database.initialize()
1202- database._cleanup = types.MethodType(
1203- partial(_cleanup, store=database.store, tempdb_name='mmtest'),
1204- database)
1205- # bool column values in PostgreSQL.
1206- database.FALSE = 'False'
1207- database.TRUE = 'True'
1208- return database
1209+ """.format(table))
1210
1211=== removed directory 'src/mailman/database/schema'
1212=== removed file 'src/mailman/database/schema/__init__.py'
1213=== removed file 'src/mailman/database/schema/helpers.py'
1214--- src/mailman/database/schema/helpers.py 2014-01-01 14:59:42 +0000
1215+++ src/mailman/database/schema/helpers.py 1970-01-01 00:00:00 +0000
1216@@ -1,43 +0,0 @@
1217-# Copyright (C) 2013-2014 by the Free Software Foundation, Inc.
1218-#
1219-# This file is part of GNU Mailman.
1220-#
1221-# GNU Mailman is free software: you can redistribute it and/or modify it under
1222-# the terms of the GNU General Public License as published by the Free
1223-# Software Foundation, either version 3 of the License, or (at your option)
1224-# any later version.
1225-#
1226-# GNU Mailman is distributed in the hope that it will be useful, but WITHOUT
1227-# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
1228-# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
1229-# more details.
1230-#
1231-# You should have received a copy of the GNU General Public License along with
1232-# GNU Mailman. If not, see <http://www.gnu.org/licenses/>.
1233-
1234-"""Schema migration helpers."""
1235-
1236-from __future__ import absolute_import, print_function, unicode_literals
1237-
1238-__metaclass__ = type
1239-__all__ = [
1240- 'make_listid',
1241- ]
1242-
1243-
1244-
1245
1246-def make_listid(fqdn_listname):
1247- """Turn a FQDN list name into a List-ID."""
1248- list_name, at, mail_host = fqdn_listname.partition('@')
1249- if at == '':
1250- # If there is no @ sign in the value, assume it already contains the
1251- # list-id.
1252- return fqdn_listname
1253- return '{0}.{1}'.format(list_name, mail_host)
1254-
1255-
1256-
1257
1258-def pivot(store, table_name):
1259- """Pivot a backup table into the real table name."""
1260- store.execute('DROP TABLE {}'.format(table_name))
1261- store.execute('ALTER TABLE {0}_backup RENAME TO {0}'.format(table_name))
1262
1263=== removed file 'src/mailman/database/schema/mm_00000000000000_base.py'
1264--- src/mailman/database/schema/mm_00000000000000_base.py 2014-01-01 14:59:42 +0000
1265+++ src/mailman/database/schema/mm_00000000000000_base.py 1970-01-01 00:00:00 +0000
1266@@ -1,35 +0,0 @@
1267-# Copyright (C) 2012-2014 by the Free Software Foundation, Inc.
1268-#
1269-# This file is part of GNU Mailman.
1270-#
1271-# GNU Mailman is free software: you can redistribute it and/or modify it under
1272-# the terms of the GNU General Public License as published by the Free
1273-# Software Foundation, either version 3 of the License, or (at your option)
1274-# any later version.
1275-#
1276-# GNU Mailman is distributed in the hope that it will be useful, but WITHOUT
1277-# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
1278-# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
1279-# more details.
1280-#
1281-# You should have received a copy of the GNU General Public License along with
1282-# GNU Mailman. If not, see <http://www.gnu.org/licenses/>.
1283-
1284-"""Load the base schema."""
1285-
1286-from __future__ import absolute_import, print_function, unicode_literals
1287-
1288-__metaclass__ = type
1289-__all__ = [
1290- 'upgrade',
1291- ]
1292-
1293-
1294-VERSION = '00000000000000'
1295-_helper = None
1296-
1297-
1298-
1299
1300-def upgrade(database, store, version, module_path):
1301- filename = '{0}.sql'.format(database.TAG)
1302- database.load_schema(store, version, filename, module_path)
1303
1304=== removed file 'src/mailman/database/schema/mm_20120407000000.py'
1305--- src/mailman/database/schema/mm_20120407000000.py 2014-01-01 14:59:42 +0000
1306+++ src/mailman/database/schema/mm_20120407000000.py 1970-01-01 00:00:00 +0000
1307@@ -1,212 +0,0 @@
1308-# Copyright (C) 2012-2014 by the Free Software Foundation, Inc.
1309-#
1310-# This file is part of GNU Mailman.
1311-#
1312-# GNU Mailman is free software: you can redistribute it and/or modify it under
1313-# the terms of the GNU General Public License as published by the Free
1314-# Software Foundation, either version 3 of the License, or (at your option)
1315-# any later version.
1316-#
1317-# GNU Mailman is distributed in the hope that it will be useful, but WITHOUT
1318-# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
1319-# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
1320-# more details.
1321-#
1322-# You should have received a copy of the GNU General Public License along with
1323-# GNU Mailman. If not, see <http://www.gnu.org/licenses/>.
1324-
1325-"""3.0b1 -> 3.0b2 schema migrations.
1326-
1327-All column changes are in the `mailinglist` table.
1328-
1329-* Renames:
1330- - news_prefix_subject_too -> nntp_prefix_subject_too
1331- - news_moderation -> newsgroup_moderation
1332-
1333-* Collapsing:
1334- - archive, archive_private -> archive_policy
1335-
1336-* Remove:
1337- - archive_volume_frequency
1338- - generic_nonmember_action
1339- - nntp_host
1340-
1341-* Added:
1342- - list_id
1343-
1344-* Changes:
1345- member.mailing_list holds the list_id not the fqdn_listname
1346-
1347-See https://bugs.launchpad.net/mailman/+bug/971013 for details.
1348-"""
1349-
1350-from __future__ import absolute_import, print_function, unicode_literals
1351-
1352-__metaclass__ = type
1353-__all__ = [
1354- 'upgrade',
1355- ]
1356-
1357-
1358-from mailman.database.schema.helpers import pivot
1359-from mailman.interfaces.archiver import ArchivePolicy
1360-
1361-
1362-VERSION = '20120407000000'
1363-
1364-
1365-
1366
1367-def upgrade(database, store, version, module_path):
1368- if database.TAG == 'sqlite':
1369- upgrade_sqlite(database, store, version, module_path)
1370- else:
1371- upgrade_postgres(database, store, version, module_path)
1372-
1373-
1374-
1375
1376-def archive_policy(archive, archive_private):
1377- """Convert archive and archive_private to archive_policy."""
1378- if archive == 0:
1379- return ArchivePolicy.never.value
1380- elif archive_private == 1:
1381- return ArchivePolicy.private.value
1382- else:
1383- return ArchivePolicy.public.value
1384-
1385-
1386-
1387
1388-def upgrade_sqlite(database, store, version, module_path):
1389- # Load the first part of the migration. This creates a temporary table to
1390- # hold the new mailinglist table columns. The problem is that some of the
1391- # changes must be performed in Python, so after the first part is loaded,
1392- # we do the Python changes, drop the old mailing list table, and then
1393- # rename the temporary table to its place.
1394- database.load_schema(
1395- store, version, 'sqlite_{0}_01.sql'.format(version), module_path)
1396- results = store.execute("""
1397- SELECT id, include_list_post_header,
1398- news_prefix_subject_too, news_moderation,
1399- archive, archive_private, list_name, mail_host
1400- FROM mailinglist;
1401- """)
1402- for value in results:
1403- (id, list_post,
1404- news_prefix, news_moderation,
1405- archive, archive_private,
1406- list_name, mail_host) = value
1407- # Figure out what the new archive_policy column value should be.
1408- list_id = '{0}.{1}'.format(list_name, mail_host)
1409- fqdn_listname = '{0}@{1}'.format(list_name, mail_host)
1410- store.execute("""
1411- UPDATE mailinglist_backup SET
1412- allow_list_posts = {0},
1413- newsgroup_moderation = {1},
1414- nntp_prefix_subject_too = {2},
1415- archive_policy = {3},
1416- list_id = '{4}'
1417- WHERE id = {5};
1418- """.format(
1419- list_post,
1420- news_moderation,
1421- news_prefix,
1422- archive_policy(archive, archive_private),
1423- list_id,
1424- id))
1425- # Also update the member.mailing_list column to hold the list_id
1426- # instead of the fqdn_listname.
1427- store.execute("""
1428- UPDATE member SET
1429- mailing_list = '{0}'
1430- WHERE mailing_list = '{1}';
1431- """.format(list_id, fqdn_listname))
1432- # Pivot the backup table to the real thing.
1433- pivot(store, 'mailinglist')
1434- # Now add some indexes that were previously missing.
1435- store.execute(
1436- 'CREATE INDEX ix_mailinglist_list_id ON mailinglist (list_id);')
1437- store.execute(
1438- 'CREATE INDEX ix_mailinglist_fqdn_listname '
1439- 'ON mailinglist (list_name, mail_host);')
1440- # Now, do the member table.
1441- results = store.execute('SELECT id, mailing_list FROM member;')
1442- for id, mailing_list in results:
1443- list_name, at, mail_host = mailing_list.partition('@')
1444- if at == '':
1445- list_id = mailing_list
1446- else:
1447- list_id = '{0}.{1}'.format(list_name, mail_host)
1448- store.execute("""
1449- UPDATE member_backup SET list_id = '{0}'
1450- WHERE id = {1};
1451- """.format(list_id, id))
1452- # Pivot the backup table to the real thing.
1453- pivot(store, 'member')
1454-
1455-
1456-
1457
1458-def upgrade_postgres(database, store, version, module_path):
1459- # Get the old values from the mailinglist table.
1460- results = store.execute("""
1461- SELECT id, archive, archive_private, list_name, mail_host
1462- FROM mailinglist;
1463- """)
1464- # Do the simple renames first.
1465- store.execute("""
1466- ALTER TABLE mailinglist
1467- RENAME COLUMN news_prefix_subject_too TO nntp_prefix_subject_too;
1468- """)
1469- store.execute("""
1470- ALTER TABLE mailinglist
1471- RENAME COLUMN news_moderation TO newsgroup_moderation;
1472- """)
1473- store.execute("""
1474- ALTER TABLE mailinglist
1475- RENAME COLUMN include_list_post_header TO allow_list_posts;
1476- """)
1477- # Do the easy column drops next.
1478- for column in ('archive_volume_frequency',
1479- 'generic_nonmember_action',
1480- 'nntp_host'):
1481- store.execute(
1482- 'ALTER TABLE mailinglist DROP COLUMN {0};'.format(column))
1483- # Now do the trickier collapsing of values. Add the new columns.
1484- store.execute('ALTER TABLE mailinglist ADD COLUMN archive_policy INTEGER;')
1485- store.execute('ALTER TABLE mailinglist ADD COLUMN list_id TEXT;')
1486- # Query the database for the old values of archive and archive_private in
1487- # each column. Then loop through all the results and update the new
1488- # archive_policy from the old values.
1489- for value in results:
1490- id, archive, archive_private, list_name, mail_host = value
1491- list_id = '{0}.{1}'.format(list_name, mail_host)
1492- store.execute("""
1493- UPDATE mailinglist SET
1494- archive_policy = {0},
1495- list_id = '{1}'
1496- WHERE id = {2};
1497- """.format(archive_policy(archive, archive_private), list_id, id))
1498- # Now drop the old columns.
1499- for column in ('archive', 'archive_private'):
1500- store.execute(
1501- 'ALTER TABLE mailinglist DROP COLUMN {0};'.format(column))
1502- # Now add some indexes that were previously missing.
1503- store.execute(
1504- 'CREATE INDEX ix_mailinglist_list_id ON mailinglist (list_id);')
1505- store.execute(
1506- 'CREATE INDEX ix_mailinglist_fqdn_listname '
1507- 'ON mailinglist (list_name, mail_host);')
1508- # Now, do the member table.
1509- results = store.execute('SELECT id, mailing_list FROM member;')
1510- store.execute('ALTER TABLE member ADD COLUMN list_id TEXT;')
1511- for id, mailing_list in results:
1512- list_name, at, mail_host = mailing_list.partition('@')
1513- if at == '':
1514- list_id = mailing_list
1515- else:
1516- list_id = '{0}.{1}'.format(list_name, mail_host)
1517- store.execute("""
1518- UPDATE member SET list_id = '{0}'
1519- WHERE id = {1};
1520- """.format(list_id, id))
1521- store.execute('ALTER TABLE member DROP COLUMN mailing_list;')
1522- # Record the migration in the version table.
1523- database.load_schema(store, version, None, module_path)
1524
1525=== removed file 'src/mailman/database/schema/mm_20121015000000.py'
1526--- src/mailman/database/schema/mm_20121015000000.py 2014-01-01 14:59:42 +0000
1527+++ src/mailman/database/schema/mm_20121015000000.py 1970-01-01 00:00:00 +0000
1528@@ -1,95 +0,0 @@
1529-# Copyright (C) 2012-2014 by the Free Software Foundation, Inc.
1530-#
1531-# This file is part of GNU Mailman.
1532-#
1533-# GNU Mailman is free software: you can redistribute it and/or modify it under
1534-# the terms of the GNU General Public License as published by the Free
1535-# Software Foundation, either version 3 of the License, or (at your option)
1536-# any later version.
1537-#
1538-# GNU Mailman is distributed in the hope that it will be useful, but WITHOUT
1539-# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
1540-# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
1541-# more details.
1542-#
1543-# You should have received a copy of the GNU General Public License along with
1544-# GNU Mailman. If not, see <http://www.gnu.org/licenses/>.
1545-
1546-"""3.0b2 -> 3.0b3 schema migrations.
1547-
1548-Renamed:
1549- * bans.mailing_list -> bans.list_id
1550-
1551-Removed:
1552- * mailinglist.new_member_options
1553- * mailinglist.send_remindersn
1554-"""
1555-
1556-from __future__ import absolute_import, print_function, unicode_literals
1557-
1558-__metaclass__ = type
1559-__all__ = [
1560- 'upgrade',
1561- ]
1562-
1563-
1564-from mailman.database.schema.helpers import make_listid, pivot
1565-
1566-
1567-VERSION = '20121015000000'
1568-
1569-
1570-
1571
1572-def upgrade(database, store, version, module_path):
1573- if database.TAG == 'sqlite':
1574- upgrade_sqlite(database, store, version, module_path)
1575- else:
1576- upgrade_postgres(database, store, version, module_path)
1577-
1578-
1579-
1580
1581-def upgrade_sqlite(database, store, version, module_path):
1582- database.load_schema(
1583- store, version, 'sqlite_{}_01.sql'.format(version), module_path)
1584- results = store.execute("""
1585- SELECT id, mailing_list
1586- FROM ban;
1587- """)
1588- for id, mailing_list in results:
1589- # Skip global bans since there's nothing to update.
1590- if mailing_list is None:
1591- continue
1592- store.execute("""
1593- UPDATE ban_backup SET list_id = '{}'
1594- WHERE id = {};
1595- """.format(make_listid(mailing_list), id))
1596- # Pivot the bans backup table to the real thing.
1597- pivot(store, 'ban')
1598- pivot(store, 'mailinglist')
1599-
1600-
1601-
1602
1603-def upgrade_postgres(database, store, version, module_path):
1604- # Get the old values from the ban table.
1605- results = store.execute('SELECT id, mailing_list FROM ban;')
1606- store.execute('ALTER TABLE ban ADD COLUMN list_id TEXT;')
1607- for id, mailing_list in results:
1608- # Skip global bans since there's nothing to update.
1609- if mailing_list is None:
1610- continue
1611- store.execute("""
1612- UPDATE ban SET list_id = '{0}'
1613- WHERE id = {1};
1614- """.format(make_listid(mailing_list), id))
1615- store.execute('ALTER TABLE ban DROP COLUMN mailing_list;')
1616- store.execute('ALTER TABLE mailinglist DROP COLUMN new_member_options;')
1617- store.execute('ALTER TABLE mailinglist DROP COLUMN send_reminders;')
1618- store.execute('ALTER TABLE mailinglist DROP COLUMN subscribe_policy;')
1619- store.execute('ALTER TABLE mailinglist DROP COLUMN unsubscribe_policy;')
1620- store.execute(
1621- 'ALTER TABLE mailinglist DROP COLUMN subscribe_auto_approval;')
1622- store.execute('ALTER TABLE mailinglist DROP COLUMN private_roster;')
1623- store.execute(
1624- 'ALTER TABLE mailinglist DROP COLUMN admin_member_chunksize;')
1625- # Record the migration in the version table.
1626- database.load_schema(store, version, None, module_path)
1627
1628=== removed file 'src/mailman/database/schema/mm_20130406000000.py'
1629--- src/mailman/database/schema/mm_20130406000000.py 2014-01-01 14:59:42 +0000
1630+++ src/mailman/database/schema/mm_20130406000000.py 1970-01-01 00:00:00 +0000
1631@@ -1,65 +0,0 @@
1632-# Copyright (C) 2013-2014 by the Free Software Foundation, Inc.
1633-#
1634-# This file is part of GNU Mailman.
1635-#
1636-# GNU Mailman is free software: you can redistribute it and/or modify it under
1637-# the terms of the GNU General Public License as published by the Free
1638-# Software Foundation, either version 3 of the License, or (at your option)
1639-# any later version.
1640-#
1641-# GNU Mailman is distributed in the hope that it will be useful, but WITHOUT
1642-# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
1643-# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
1644-# more details.
1645-#
1646-# You should have received a copy of the GNU General Public License along with
1647-# GNU Mailman. If not, see <http://www.gnu.org/licenses/>.
1648-
1649-"""3.0b3 -> 3.0b4 schema migrations.
1650-
1651-Renamed:
1652- * bounceevent.list_name -> bounceevent.list_id
1653-"""
1654-
1655-
1656-from __future__ import absolute_import, print_function, unicode_literals
1657-
1658-__metaclass__ = type
1659-__all__ = [
1660- 'upgrade'
1661- ]
1662-
1663-
1664-from mailman.database.schema.helpers import make_listid, pivot
1665-
1666-
1667-VERSION = '20130406000000'
1668-
1669-
1670-
1671
1672-def upgrade(database, store, version, module_path):
1673- if database.TAG == 'sqlite':
1674- upgrade_sqlite(database, store, version, module_path)
1675- else:
1676- upgrade_postgres(database, store, version, module_path)
1677-
1678-
1679-
1680
1681-def upgrade_sqlite(database, store, version, module_path):
1682- database.load_schema(
1683- store, version, 'sqlite_{}_01.sql'.format(version), module_path)
1684- results = store.execute("""
1685- SELECT id, list_name
1686- FROM bounceevent;
1687- """)
1688- for id, list_name in results:
1689- store.execute("""
1690- UPDATE bounceevent_backup SET list_id = '{}'
1691- WHERE id = {};
1692- """.format(make_listid(list_name), id))
1693- pivot(store, 'bounceevent')
1694-
1695-
1696-
1697
1698-def upgrade_postgres(database, store, version, module_path):
1699- pass
1700
1701=== removed file 'src/mailman/database/schema/postgres.sql'
1702--- src/mailman/database/schema/postgres.sql 2012-07-23 14:40:53 +0000
1703+++ src/mailman/database/schema/postgres.sql 1970-01-01 00:00:00 +0000
1704@@ -1,349 +0,0 @@
1705-CREATE TABLE mailinglist (
1706- id SERIAL NOT NULL,
1707- -- List identity
1708- list_name TEXT,
1709- mail_host TEXT,
1710- include_list_post_header BOOLEAN,
1711- include_rfc2369_headers BOOLEAN,
1712- -- Attributes not directly modifiable via the web u/i
1713- created_at TIMESTAMP,
1714- admin_member_chunksize INTEGER,
1715- next_request_id INTEGER,
1716- next_digest_number INTEGER,
1717- digest_last_sent_at TIMESTAMP,
1718- volume INTEGER,
1719- last_post_at TIMESTAMP,
1720- accept_these_nonmembers BYTEA,
1721- acceptable_aliases_id INTEGER,
1722- admin_immed_notify BOOLEAN,
1723- admin_notify_mchanges BOOLEAN,
1724- administrivia BOOLEAN,
1725- advertised BOOLEAN,
1726- anonymous_list BOOLEAN,
1727- archive BOOLEAN,
1728- archive_private BOOLEAN,
1729- archive_volume_frequency INTEGER,
1730- -- Automatic responses.
1731- autorespond_owner INTEGER,
1732- autoresponse_owner_text TEXT,
1733- autorespond_postings INTEGER,
1734- autoresponse_postings_text TEXT,
1735- autorespond_requests INTEGER,
1736- autoresponse_request_text TEXT,
1737- autoresponse_grace_period TEXT,
1738- -- Bounces.
1739- forward_unrecognized_bounces_to INTEGER,
1740- process_bounces BOOLEAN,
1741- bounce_info_stale_after TEXT,
1742- bounce_matching_headers TEXT,
1743- bounce_notify_owner_on_disable BOOLEAN,
1744- bounce_notify_owner_on_removal BOOLEAN,
1745- bounce_score_threshold INTEGER,
1746- bounce_you_are_disabled_warnings INTEGER,
1747- bounce_you_are_disabled_warnings_interval TEXT,
1748- -- Content filtering.
1749- filter_action INTEGER,
1750- filter_content BOOLEAN,
1751- collapse_alternatives BOOLEAN,
1752- convert_html_to_plaintext BOOLEAN,
1753- default_member_action INTEGER,
1754- default_nonmember_action INTEGER,
1755- description TEXT,
1756- digest_footer_uri TEXT,
1757- digest_header_uri TEXT,
1758- digest_is_default BOOLEAN,
1759- digest_send_periodic BOOLEAN,
1760- digest_size_threshold REAL,
1761- digest_volume_frequency INTEGER,
1762- digestable BOOLEAN,
1763- discard_these_nonmembers BYTEA,
1764- emergency BOOLEAN,
1765- encode_ascii_prefixes BOOLEAN,
1766- first_strip_reply_to BOOLEAN,
1767- footer_uri TEXT,
1768- forward_auto_discards BOOLEAN,
1769- gateway_to_mail BOOLEAN,
1770- gateway_to_news BOOLEAN,
1771- generic_nonmember_action INTEGER,
1772- goodbye_message_uri TEXT,
1773- header_matches BYTEA,
1774- header_uri TEXT,
1775- hold_these_nonmembers BYTEA,
1776- info TEXT,
1777- linked_newsgroup TEXT,
1778- max_days_to_hold INTEGER,
1779- max_message_size INTEGER,
1780- max_num_recipients INTEGER,
1781- member_moderation_notice TEXT,
1782- mime_is_default_digest BOOLEAN,
1783- moderator_password TEXT,
1784- new_member_options INTEGER,
1785- news_moderation INTEGER,
1786- news_prefix_subject_too BOOLEAN,
1787- nntp_host TEXT,
1788- nondigestable BOOLEAN,
1789- nonmember_rejection_notice TEXT,
1790- obscure_addresses BOOLEAN,
1791- owner_chain TEXT,
1792- owner_pipeline TEXT,
1793- personalize INTEGER,
1794- post_id INTEGER,
1795- posting_chain TEXT,
1796- posting_pipeline TEXT,
1797- preferred_language TEXT,
1798- private_roster BOOLEAN,
1799- display_name TEXT,
1800- reject_these_nonmembers BYTEA,
1801- reply_goes_to_list INTEGER,
1802- reply_to_address TEXT,
1803- require_explicit_destination BOOLEAN,
1804- respond_to_post_requests BOOLEAN,
1805- scrub_nondigest BOOLEAN,
1806- send_goodbye_message BOOLEAN,
1807- send_reminders BOOLEAN,
1808- send_welcome_message BOOLEAN,
1809- subject_prefix TEXT,
1810- subscribe_auto_approval BYTEA,
1811- subscribe_policy INTEGER,
1812- topics BYTEA,
1813- topics_bodylines_limit INTEGER,
1814- topics_enabled BOOLEAN,
1815- unsubscribe_policy INTEGER,
1816- welcome_message_uri TEXT,
1817- -- This was accidentally added by the PostgreSQL porter.
1818- -- moderation_callback TEXT,
1819- PRIMARY KEY (id)
1820- );
1821-
1822-CREATE TABLE _request (
1823- id SERIAL NOT NULL,
1824- "key" TEXT,
1825- request_type INTEGER,
1826- data_hash BYTEA,
1827- mailing_list_id INTEGER,
1828- PRIMARY KEY (id)
1829- -- XXX: config.db_reset() triggers IntegrityError
1830- -- ,
1831- -- CONSTRAINT _request_mailing_list_id_fk
1832- -- FOREIGN KEY (mailing_list_id) REFERENCES mailinglist (id)
1833- );
1834-
1835-CREATE TABLE acceptablealias (
1836- id SERIAL NOT NULL,
1837- "alias" TEXT NOT NULL,
1838- mailing_list_id INTEGER NOT NULL,
1839- PRIMARY KEY (id)
1840- -- XXX: config.db_reset() triggers IntegrityError
1841- -- ,
1842- -- CONSTRAINT acceptablealias_mailing_list_id_fk
1843- -- FOREIGN KEY (mailing_list_id) REFERENCES mailinglist (id)
1844- );
1845-CREATE INDEX ix_acceptablealias_mailing_list_id
1846- ON acceptablealias (mailing_list_id);
1847-CREATE INDEX ix_acceptablealias_alias ON acceptablealias ("alias");
1848-
1849-CREATE TABLE preferences (
1850- id SERIAL NOT NULL,
1851- acknowledge_posts BOOLEAN,
1852- hide_address BOOLEAN,
1853- preferred_language TEXT,
1854- receive_list_copy BOOLEAN,
1855- receive_own_postings BOOLEAN,
1856- delivery_mode INTEGER,
1857- delivery_status INTEGER,
1858- PRIMARY KEY (id)
1859- );
1860-
1861-CREATE TABLE address (
1862- id SERIAL NOT NULL,
1863- email TEXT,
1864- _original TEXT,
1865- display_name TEXT,
1866- verified_on TIMESTAMP,
1867- registered_on TIMESTAMP,
1868- user_id INTEGER,
1869- preferences_id INTEGER,
1870- PRIMARY KEY (id)
1871- -- XXX: config.db_reset() triggers IntegrityError
1872- -- ,
1873- -- CONSTRAINT address_preferences_id_fk
1874- -- FOREIGN KEY (preferences_id) REFERENCES preferences (id)
1875- );
1876-
1877-CREATE TABLE "user" (
1878- id SERIAL NOT NULL,
1879- display_name TEXT,
1880- password BYTEA,
1881- _user_id UUID,
1882- _created_on TIMESTAMP,
1883- _preferred_address_id INTEGER,
1884- preferences_id INTEGER,
1885- PRIMARY KEY (id)
1886- -- XXX: config.db_reset() triggers IntegrityError
1887- -- ,
1888- -- CONSTRAINT user_preferences_id_fk
1889- -- FOREIGN KEY (preferences_id) REFERENCES preferences (id),
1890- -- XXX: config.db_reset() triggers IntegrityError
1891- -- CONSTRAINT _preferred_address_id_fk
1892- -- FOREIGN KEY (_preferred_address_id) REFERENCES address (id)
1893- );
1894-CREATE INDEX ix_user_user_id ON "user" (_user_id);
1895-
1896--- since user and address have circular foreign key refs, the
1897--- constraint on the address table has to be added after
1898--- the user table is created
1899---
1900--- XXX: users.rst triggers an IntegrityError
1901--- ALTER TABLE address ADD
1902--- CONSTRAINT address_user_id_fk
1903--- FOREIGN KEY (user_id) REFERENCES "user" (id);
1904-
1905-CREATE TABLE autoresponserecord (
1906- id SERIAL NOT NULL,
1907- address_id INTEGER,
1908- mailing_list_id INTEGER,
1909- response_type INTEGER,
1910- date_sent TIMESTAMP,
1911- PRIMARY KEY (id)
1912- -- XXX: config.db_reset() triggers IntegrityError
1913- -- ,
1914- -- CONSTRAINT autoresponserecord_address_id_fk
1915- -- FOREIGN KEY (address_id) REFERENCES address (id)
1916- -- XXX: config.db_reset() triggers IntegrityError
1917- -- ,
1918- -- CONSTRAINT autoresponserecord_mailing_list_id
1919- -- FOREIGN KEY (mailing_list_id) REFERENCES mailinglist (id)
1920- );
1921-CREATE INDEX ix_autoresponserecord_address_id
1922- ON autoresponserecord (address_id);
1923-CREATE INDEX ix_autoresponserecord_mailing_list_id
1924- ON autoresponserecord (mailing_list_id);
1925-
1926-CREATE TABLE bounceevent (
1927- id SERIAL NOT NULL,
1928- list_name TEXT,
1929- email TEXT,
1930- "timestamp" TIMESTAMP,
1931- message_id TEXT,
1932- context INTEGER,
1933- processed BOOLEAN,
1934- PRIMARY KEY (id)
1935- );
1936-
1937-CREATE TABLE contentfilter (
1938- id SERIAL NOT NULL,
1939- mailing_list_id INTEGER,
1940- filter_pattern TEXT,
1941- filter_type INTEGER,
1942- PRIMARY KEY (id),
1943- CONSTRAINT contentfilter_mailing_list_id
1944- FOREIGN KEY (mailing_list_id) REFERENCES mailinglist (id)
1945- );
1946-CREATE INDEX ix_contentfilter_mailing_list_id
1947- ON contentfilter (mailing_list_id);
1948-
1949-CREATE TABLE domain (
1950- id SERIAL NOT NULL,
1951- mail_host TEXT,
1952- base_url TEXT,
1953- description TEXT,
1954- contact_address TEXT,
1955- PRIMARY KEY (id)
1956- );
1957-
1958-CREATE TABLE language (
1959- id SERIAL NOT NULL,
1960- code TEXT,
1961- PRIMARY KEY (id)
1962- );
1963-
1964-CREATE TABLE member (
1965- id SERIAL NOT NULL,
1966- _member_id UUID,
1967- role INTEGER,
1968- mailing_list TEXT,
1969- moderation_action INTEGER,
1970- address_id INTEGER,
1971- preferences_id INTEGER,
1972- user_id INTEGER,
1973- PRIMARY KEY (id)
1974- -- XXX: config.db_reset() triggers IntegrityError
1975- -- ,
1976- -- CONSTRAINT member_address_id_fk
1977- -- FOREIGN KEY (address_id) REFERENCES address (id),
1978- -- XXX: config.db_reset() triggers IntegrityError
1979- -- CONSTRAINT member_preferences_id_fk
1980- -- FOREIGN KEY (preferences_id) REFERENCES preferences (id),
1981- -- CONSTRAINT member_user_id_fk
1982- -- FOREIGN KEY (user_id) REFERENCES "user" (id)
1983- );
1984-CREATE INDEX ix_member__member_id ON member (_member_id);
1985-CREATE INDEX ix_member_address_id ON member (address_id);
1986-CREATE INDEX ix_member_preferences_id ON member (preferences_id);
1987-
1988-CREATE TABLE message (
1989- id SERIAL NOT NULL,
1990- message_id_hash BYTEA,
1991- path BYTEA,
1992- message_id TEXT,
1993- PRIMARY KEY (id)
1994- );
1995-
1996-CREATE TABLE onelastdigest (
1997- id SERIAL NOT NULL,
1998- mailing_list_id INTEGER,
1999- address_id INTEGER,
2000- delivery_mode INTEGER,
2001- PRIMARY KEY (id),
2002- CONSTRAINT onelastdigest_mailing_list_id_fk
2003- FOREIGN KEY (mailing_list_id) REFERENCES mailinglist(id),
2004- CONSTRAINT onelastdigest_address_id_fk
2005- FOREIGN KEY (address_id) REFERENCES address(id)
2006- );
2007-
2008-CREATE TABLE pended (
2009- id SERIAL NOT NULL,
2010- token BYTEA,
2011- expiration_date TIMESTAMP,
2012- PRIMARY KEY (id)
2013- );
2014-
2015-CREATE TABLE pendedkeyvalue (
2016- id SERIAL NOT NULL,
2017- "key" TEXT,
2018- value TEXT,
2019- pended_id INTEGER,
2020- PRIMARY KEY (id)
2021- -- ,
2022- -- XXX: config.db_reset() triggers IntegrityError
2023- -- CONSTRAINT pendedkeyvalue_pended_id_fk
2024- -- FOREIGN KEY (pended_id) REFERENCES pended (id)
2025- );
2026-
2027-CREATE TABLE version (
2028- id SERIAL NOT NULL,
2029- component TEXT,
2030- version TEXT,
2031- PRIMARY KEY (id)
2032- );
2033-
2034-CREATE INDEX ix__request_mailing_list_id ON _request (mailing_list_id);
2035-CREATE INDEX ix_address_preferences_id ON address (preferences_id);
2036-CREATE INDEX ix_address_user_id ON address (user_id);
2037-CREATE INDEX ix_pendedkeyvalue_pended_id ON pendedkeyvalue (pended_id);
2038-CREATE INDEX ix_user_preferences_id ON "user" (preferences_id);
2039-
2040-CREATE TABLE ban (
2041- id SERIAL NOT NULL,
2042- email TEXT,
2043- mailing_list TEXT,
2044- PRIMARY KEY (id)
2045- );
2046-
2047-CREATE TABLE uid (
2048- -- Keep track of all assigned unique ids to prevent re-use.
2049- id SERIAL NOT NULL,
2050- uid UUID,
2051- PRIMARY KEY (id)
2052- );
2053-CREATE INDEX ix_uid_uid ON uid (uid);
2054
2055=== removed file 'src/mailman/database/schema/sqlite.sql'
2056--- src/mailman/database/schema/sqlite.sql 2012-04-08 16:15:29 +0000
2057+++ src/mailman/database/schema/sqlite.sql 1970-01-01 00:00:00 +0000
2058@@ -1,327 +0,0 @@
2059--- THIS FILE HAS BEEN FROZEN AS OF 3.0b1
2060--- SEE THE SCHEMA MIGRATIONS FOR DIFFERENCES.
2061-
2062-PRAGMA foreign_keys = ON;
2063-
2064-CREATE TABLE _request (
2065- id INTEGER NOT NULL,
2066- "key" TEXT,
2067- request_type INTEGER,
2068- data_hash TEXT,
2069- mailing_list_id INTEGER,
2070- PRIMARY KEY (id),
2071- CONSTRAINT _request_mailing_list_id_fk
2072- FOREIGN KEY (mailing_list_id) REFERENCES mailinglist (id)
2073- );
2074-
2075-CREATE TABLE acceptablealias (
2076- id INTEGER NOT NULL,
2077- "alias" TEXT NOT NULL,
2078- mailing_list_id INTEGER NOT NULL,
2079- PRIMARY KEY (id),
2080- CONSTRAINT acceptablealias_mailing_list_id_fk
2081- FOREIGN KEY (mailing_list_id) REFERENCES mailinglist (id)
2082- );
2083-CREATE INDEX ix_acceptablealias_mailing_list_id
2084- ON acceptablealias (mailing_list_id);
2085-CREATE INDEX ix_acceptablealias_alias ON acceptablealias ("alias");
2086-
2087-CREATE TABLE address (
2088- id INTEGER NOT NULL,
2089- email TEXT,
2090- _original TEXT,
2091- display_name TEXT,
2092- verified_on TIMESTAMP,
2093- registered_on TIMESTAMP,
2094- user_id INTEGER,
2095- preferences_id INTEGER,
2096- PRIMARY KEY (id),
2097- CONSTRAINT address_user_id_fk
2098- FOREIGN KEY (user_id) REFERENCES user (id),
2099- CONSTRAINT address_preferences_id_fk
2100- FOREIGN KEY (preferences_id) REFERENCES preferences (id)
2101- );
2102-
2103-CREATE TABLE autoresponserecord (
2104- id INTEGER NOT NULL,
2105- address_id INTEGER,
2106- mailing_list_id INTEGER,
2107- response_type INTEGER,
2108- date_sent TIMESTAMP,
2109- PRIMARY KEY (id),
2110- CONSTRAINT autoresponserecord_address_id_fk
2111- FOREIGN KEY (address_id) REFERENCES address (id),
2112- CONSTRAINT autoresponserecord_mailing_list_id
2113- FOREIGN KEY (mailing_list_id) REFERENCES mailinglist (id)
2114- );
2115-CREATE INDEX ix_autoresponserecord_address_id
2116- ON autoresponserecord (address_id);
2117-CREATE INDEX ix_autoresponserecord_mailing_list_id
2118- ON autoresponserecord (mailing_list_id);
2119-
2120-CREATE TABLE bounceevent (
2121- id INTEGER NOT NULL,
2122- list_name TEXT,
2123- email TEXT,
2124- 'timestamp' TIMESTAMP,
2125- message_id TEXT,
2126- context INTEGER,
2127- processed BOOLEAN,
2128- PRIMARY KEY (id)
2129- );
2130-
2131-CREATE TABLE contentfilter (
2132- id INTEGER NOT NULL,
2133- mailing_list_id INTEGER,
2134- filter_pattern TEXT,
2135- filter_type INTEGER,
2136- PRIMARY KEY (id),
2137- CONSTRAINT contentfilter_mailing_list_id
2138- FOREIGN KEY (mailing_list_id) REFERENCES mailinglist (id)
2139- );
2140-CREATE INDEX ix_contentfilter_mailing_list_id
2141- ON contentfilter (mailing_list_id);
2142-
2143-CREATE TABLE domain (
2144- id INTEGER NOT NULL,
2145- mail_host TEXT,
2146- base_url TEXT,
2147- description TEXT,
2148- contact_address TEXT,
2149- PRIMARY KEY (id)
2150- );
2151-
2152-CREATE TABLE language (
2153- id INTEGER NOT NULL,
2154- code TEXT,
2155- PRIMARY KEY (id)
2156- );
2157-
2158-CREATE TABLE mailinglist (
2159- id INTEGER NOT NULL,
2160- -- List identity
2161- list_name TEXT,
2162- mail_host TEXT,
2163- include_list_post_header BOOLEAN,
2164- include_rfc2369_headers BOOLEAN,
2165- -- Attributes not directly modifiable via the web u/i
2166- created_at TIMESTAMP,
2167- admin_member_chunksize INTEGER,
2168- next_request_id INTEGER,
2169- next_digest_number INTEGER,
2170- digest_last_sent_at TIMESTAMP,
2171- volume INTEGER,
2172- last_post_at TIMESTAMP,
2173- accept_these_nonmembers BLOB,
2174- acceptable_aliases_id INTEGER,
2175- admin_immed_notify BOOLEAN,
2176- admin_notify_mchanges BOOLEAN,
2177- administrivia BOOLEAN,
2178- advertised BOOLEAN,
2179- anonymous_list BOOLEAN,
2180- archive BOOLEAN,
2181- archive_private BOOLEAN,
2182- archive_volume_frequency INTEGER,
2183- -- Automatic responses.
2184- autorespond_owner INTEGER,
2185- autoresponse_owner_text TEXT,
2186- autorespond_postings INTEGER,
2187- autoresponse_postings_text TEXT,
2188- autorespond_requests INTEGER,
2189- autoresponse_request_text TEXT,
2190- autoresponse_grace_period TEXT,
2191- -- Bounces.
2192- forward_unrecognized_bounces_to INTEGER,
2193- process_bounces BOOLEAN,
2194- bounce_info_stale_after TEXT,
2195- bounce_matching_headers TEXT,
2196- bounce_notify_owner_on_disable BOOLEAN,
2197- bounce_notify_owner_on_removal BOOLEAN,
2198- bounce_score_threshold INTEGER,
2199- bounce_you_are_disabled_warnings INTEGER,
2200- bounce_you_are_disabled_warnings_interval TEXT,
2201- -- Content filtering.
2202- filter_action INTEGER,
2203- filter_content BOOLEAN,
2204- collapse_alternatives BOOLEAN,
2205- convert_html_to_plaintext BOOLEAN,
2206- default_member_action INTEGER,
2207- default_nonmember_action INTEGER,
2208- description TEXT,
2209- digest_footer_uri TEXT,
2210- digest_header_uri TEXT,
2211- digest_is_default BOOLEAN,
2212- digest_send_periodic BOOLEAN,
2213- digest_size_threshold FLOAT,
2214- digest_volume_frequency INTEGER,
2215- digestable BOOLEAN,
2216- discard_these_nonmembers BLOB,
2217- emergency BOOLEAN,
2218- encode_ascii_prefixes BOOLEAN,
2219- first_strip_reply_to BOOLEAN,
2220- footer_uri TEXT,
2221- forward_auto_discards BOOLEAN,
2222- gateway_to_mail BOOLEAN,
2223- gateway_to_news BOOLEAN,
2224- generic_nonmember_action INTEGER,
2225- goodbye_message_uri TEXT,
2226- header_matches BLOB,
2227- header_uri TEXT,
2228- hold_these_nonmembers BLOB,
2229- info TEXT,
2230- linked_newsgroup TEXT,
2231- max_days_to_hold INTEGER,
2232- max_message_size INTEGER,
2233- max_num_recipients INTEGER,
2234- member_moderation_notice TEXT,
2235- mime_is_default_digest BOOLEAN,
2236- moderator_password TEXT,
2237- new_member_options INTEGER,
2238- news_moderation INTEGER,
2239- news_prefix_subject_too BOOLEAN,
2240- nntp_host TEXT,
2241- nondigestable BOOLEAN,
2242- nonmember_rejection_notice TEXT,
2243- obscure_addresses BOOLEAN,
2244- owner_chain TEXT,
2245- owner_pipeline TEXT,
2246- personalize INTEGER,
2247- post_id INTEGER,
2248- posting_chain TEXT,
2249- posting_pipeline TEXT,
2250- preferred_language TEXT,
2251- private_roster BOOLEAN,
2252- display_name TEXT,
2253- reject_these_nonmembers BLOB,
2254- reply_goes_to_list INTEGER,
2255- reply_to_address TEXT,
2256- require_explicit_destination BOOLEAN,
2257- respond_to_post_requests BOOLEAN,
2258- scrub_nondigest BOOLEAN,
2259- send_goodbye_message BOOLEAN,
2260- send_reminders BOOLEAN,
2261- send_welcome_message BOOLEAN,
2262- subject_prefix TEXT,
2263- subscribe_auto_approval BLOB,
2264- subscribe_policy INTEGER,
2265- topics BLOB,
2266- topics_bodylines_limit INTEGER,
2267- topics_enabled BOOLEAN,
2268- unsubscribe_policy INTEGER,
2269- welcome_message_uri TEXT,
2270- PRIMARY KEY (id)
2271- );
2272-
2273-CREATE TABLE member (
2274- id INTEGER NOT NULL,
2275- _member_id TEXT,
2276- role INTEGER,
2277- mailing_list TEXT,
2278- moderation_action INTEGER,
2279- address_id INTEGER,
2280- preferences_id INTEGER,
2281- user_id INTEGER,
2282- PRIMARY KEY (id),
2283- CONSTRAINT member_address_id_fk
2284- FOREIGN KEY (address_id) REFERENCES address (id),
2285- CONSTRAINT member_preferences_id_fk
2286- FOREIGN KEY (preferences_id) REFERENCES preferences (id)
2287- CONSTRAINT member_user_id_fk
2288- FOREIGN KEY (user_id) REFERENCES user (id)
2289- );
2290-CREATE INDEX ix_member__member_id ON member (_member_id);
2291-CREATE INDEX ix_member_address_id ON member (address_id);
2292-CREATE INDEX ix_member_preferences_id ON member (preferences_id);
2293-
2294-CREATE TABLE message (
2295- id INTEGER NOT NULL,
2296- message_id_hash TEXT,
2297- path TEXT,
2298- message_id TEXT,
2299- PRIMARY KEY (id)
2300- );
2301-
2302-CREATE TABLE onelastdigest (
2303- id INTEGER NOT NULL,
2304- mailing_list_id INTEGER,
2305- address_id INTEGER,
2306- delivery_mode INTEGER,
2307- PRIMARY KEY (id),
2308- CONSTRAINT onelastdigest_mailing_list_id_fk
2309- FOREIGN KEY (mailing_list_id) REFERENCES mailinglist(id),
2310- CONSTRAINT onelastdigest_address_id_fk
2311- FOREIGN KEY (address_id) REFERENCES address(id)
2312- );
2313-
2314-CREATE TABLE pended (
2315- id INTEGER NOT NULL,
2316- token TEXT,
2317- expiration_date TIMESTAMP,
2318- PRIMARY KEY (id)
2319- );
2320-
2321-CREATE TABLE pendedkeyvalue (
2322- id INTEGER NOT NULL,
2323- "key" TEXT,
2324- value TEXT,
2325- pended_id INTEGER,
2326- PRIMARY KEY (id),
2327- CONSTRAINT pendedkeyvalue_pended_id_fk
2328- FOREIGN KEY (pended_id) REFERENCES pended (id)
2329- );
2330-
2331-CREATE TABLE preferences (
2332- id INTEGER NOT NULL,
2333- acknowledge_posts BOOLEAN,
2334- hide_address BOOLEAN,
2335- preferred_language TEXT,
2336- receive_list_copy BOOLEAN,
2337- receive_own_postings BOOLEAN,
2338- delivery_mode INTEGER,
2339- delivery_status INTEGER,
2340- PRIMARY KEY (id)
2341- );
2342-
2343-CREATE TABLE user (
2344- id INTEGER NOT NULL,
2345- display_name TEXT,
2346- password BINARY,
2347- _user_id TEXT,
2348- _created_on TIMESTAMP,
2349- _preferred_address_id INTEGER,
2350- preferences_id INTEGER,
2351- PRIMARY KEY (id),
2352- CONSTRAINT user_preferences_id_fk
2353- FOREIGN KEY (preferences_id) REFERENCES preferences (id),
2354- CONSTRAINT _preferred_address_id_fk
2355- FOREIGN KEY (_preferred_address_id) REFERENCES address (id)
2356- );
2357-CREATE INDEX ix_user_user_id ON user (_user_id);
2358-
2359-CREATE TABLE version (
2360- id INTEGER NOT NULL,
2361- component TEXT,
2362- version TEXT,
2363- PRIMARY KEY (id)
2364- );
2365-
2366-CREATE INDEX ix__request_mailing_list_id ON _request (mailing_list_id);
2367-CREATE INDEX ix_address_preferences_id ON address (preferences_id);
2368-CREATE INDEX ix_address_user_id ON address (user_id);
2369-CREATE INDEX ix_pendedkeyvalue_pended_id ON pendedkeyvalue (pended_id);
2370-CREATE INDEX ix_user_preferences_id ON user (preferences_id);
2371-
2372-CREATE TABLE ban (
2373- id INTEGER NOT NULL,
2374- email TEXT,
2375- mailing_list TEXT,
2376- PRIMARY KEY (id)
2377- );
2378-
2379-CREATE TABLE uid (
2380- -- Keep track of all assigned unique ids to prevent re-use.
2381- id INTEGER NOT NULL,
2382- uid TEXT,
2383- PRIMARY KEY (id)
2384- );
2385-CREATE INDEX ix_uid_uid ON uid (uid);
2386
2387=== removed file 'src/mailman/database/schema/sqlite_20120407000000_01.sql'
2388--- src/mailman/database/schema/sqlite_20120407000000_01.sql 2013-09-01 15:15:08 +0000
2389+++ src/mailman/database/schema/sqlite_20120407000000_01.sql 1970-01-01 00:00:00 +0000
2390@@ -1,280 +0,0 @@
2391--- This file contains the sqlite3 schema migration from
2392--- 3.0b1 TO 3.0b2
2393---
2394--- 3.0b2 has been released thus you MAY NOT edit this file.
2395-
2396--- For SQLite3 migration strategy, see
2397--- http://sqlite.org/faq.html#q11
2398-
2399--- REMOVALS from the mailinglist table:
2400--- REM archive
2401--- REM archive_private
2402--- REM archive_volume_frequency
2403--- REM include_list_post_header
2404--- REM news_moderation
2405--- REM news_prefix_subject_too
2406--- REM nntp_host
2407---
2408--- ADDS to the mailing list table:
2409--- ADD allow_list_posts
2410--- ADD archive_policy
2411--- ADD list_id
2412--- ADD newsgroup_moderation
2413--- ADD nntp_prefix_subject_too
2414-
2415--- LP: #971013
2416--- LP: #967238
2417-
2418--- REMOVALS from the member table:
2419--- REM mailing_list
2420-
2421--- ADDS to the member table:
2422--- ADD list_id
2423-
2424--- LP: #1024509
2425-
2426-
2427-CREATE TABLE mailinglist_backup (
2428- id INTEGER NOT NULL,
2429- -- List identity
2430- list_name TEXT,
2431- mail_host TEXT,
2432- allow_list_posts BOOLEAN,
2433- include_rfc2369_headers BOOLEAN,
2434- -- Attributes not directly modifiable via the web u/i
2435- created_at TIMESTAMP,
2436- admin_member_chunksize INTEGER,
2437- next_request_id INTEGER,
2438- next_digest_number INTEGER,
2439- digest_last_sent_at TIMESTAMP,
2440- volume INTEGER,
2441- last_post_at TIMESTAMP,
2442- accept_these_nonmembers BLOB,
2443- acceptable_aliases_id INTEGER,
2444- admin_immed_notify BOOLEAN,
2445- admin_notify_mchanges BOOLEAN,
2446- administrivia BOOLEAN,
2447- advertised BOOLEAN,
2448- anonymous_list BOOLEAN,
2449- -- Automatic responses.
2450- autorespond_owner INTEGER,
2451- autoresponse_owner_text TEXT,
2452- autorespond_postings INTEGER,
2453- autoresponse_postings_text TEXT,
2454- autorespond_requests INTEGER,
2455- autoresponse_request_text TEXT,
2456- autoresponse_grace_period TEXT,
2457- -- Bounces.
2458- forward_unrecognized_bounces_to INTEGER,
2459- process_bounces BOOLEAN,
2460- bounce_info_stale_after TEXT,
2461- bounce_matching_headers TEXT,
2462- bounce_notify_owner_on_disable BOOLEAN,
2463- bounce_notify_owner_on_removal BOOLEAN,
2464- bounce_score_threshold INTEGER,
2465- bounce_you_are_disabled_warnings INTEGER,
2466- bounce_you_are_disabled_warnings_interval TEXT,
2467- -- Content filtering.
2468- filter_action INTEGER,
2469- filter_content BOOLEAN,
2470- collapse_alternatives BOOLEAN,
2471- convert_html_to_plaintext BOOLEAN,
2472- default_member_action INTEGER,
2473- default_nonmember_action INTEGER,
2474- description TEXT,
2475- digest_footer_uri TEXT,
2476- digest_header_uri TEXT,
2477- digest_is_default BOOLEAN,
2478- digest_send_periodic BOOLEAN,
2479- digest_size_threshold FLOAT,
2480- digest_volume_frequency INTEGER,
2481- digestable BOOLEAN,
2482- discard_these_nonmembers BLOB,
2483- emergency BOOLEAN,
2484- encode_ascii_prefixes BOOLEAN,
2485- first_strip_reply_to BOOLEAN,
2486- footer_uri TEXT,
2487- forward_auto_discards BOOLEAN,
2488- gateway_to_mail BOOLEAN,
2489- gateway_to_news BOOLEAN,
2490- goodbye_message_uri TEXT,
2491- header_matches BLOB,
2492- header_uri TEXT,
2493- hold_these_nonmembers BLOB,
2494- info TEXT,
2495- linked_newsgroup TEXT,
2496- max_days_to_hold INTEGER,
2497- max_message_size INTEGER,
2498- max_num_recipients INTEGER,
2499- member_moderation_notice TEXT,
2500- mime_is_default_digest BOOLEAN,
2501- moderator_password TEXT,
2502- new_member_options INTEGER,
2503- nondigestable BOOLEAN,
2504- nonmember_rejection_notice TEXT,
2505- obscure_addresses BOOLEAN,
2506- owner_chain TEXT,
2507- owner_pipeline TEXT,
2508- personalize INTEGER,
2509- post_id INTEGER,
2510- posting_chain TEXT,
2511- posting_pipeline TEXT,
2512- preferred_language TEXT,
2513- private_roster BOOLEAN,
2514- display_name TEXT,
2515- reject_these_nonmembers BLOB,
2516- reply_goes_to_list INTEGER,
2517- reply_to_address TEXT,
2518- require_explicit_destination BOOLEAN,
2519- respond_to_post_requests BOOLEAN,
2520- scrub_nondigest BOOLEAN,
2521- send_goodbye_message BOOLEAN,
2522- send_reminders BOOLEAN,
2523- send_welcome_message BOOLEAN,
2524- subject_prefix TEXT,
2525- subscribe_auto_approval BLOB,
2526- subscribe_policy INTEGER,
2527- topics BLOB,
2528- topics_bodylines_limit INTEGER,
2529- topics_enabled BOOLEAN,
2530- unsubscribe_policy INTEGER,
2531- welcome_message_uri TEXT,
2532- PRIMARY KEY (id)
2533- );
2534-
2535-INSERT INTO mailinglist_backup SELECT
2536- id,
2537- -- List identity
2538- list_name,
2539- mail_host,
2540- include_list_post_header,
2541- include_rfc2369_headers,
2542- -- Attributes not directly modifiable via the web u/i
2543- created_at,
2544- admin_member_chunksize,
2545- next_request_id,
2546- next_digest_number,
2547- digest_last_sent_at,
2548- volume,
2549- last_post_at,
2550- accept_these_nonmembers,
2551- acceptable_aliases_id,
2552- admin_immed_notify,
2553- admin_notify_mchanges,
2554- administrivia,
2555- advertised,
2556- anonymous_list,
2557- -- Automatic responses.
2558- autorespond_owner,
2559- autoresponse_owner_text,
2560- autorespond_postings,
2561- autoresponse_postings_text,
2562- autorespond_requests,
2563- autoresponse_request_text,
2564- autoresponse_grace_period,
2565- -- Bounces.
2566- forward_unrecognized_bounces_to,
2567- process_bounces,
2568- bounce_info_stale_after,
2569- bounce_matching_headers,
2570- bounce_notify_owner_on_disable,
2571- bounce_notify_owner_on_removal,
2572- bounce_score_threshold,
2573- bounce_you_are_disabled_warnings,
2574- bounce_you_are_disabled_warnings_interval,
2575- -- Content filtering.
2576- filter_action,
2577- filter_content,
2578- collapse_alternatives,
2579- convert_html_to_plaintext,
2580- default_member_action,
2581- default_nonmember_action,
2582- description,
2583- digest_footer_uri,
2584- digest_header_uri,
2585- digest_is_default,
2586- digest_send_periodic,
2587- digest_size_threshold,
2588- digest_volume_frequency,
2589- digestable,
2590- discard_these_nonmembers,
2591- emergency,
2592- encode_ascii_prefixes,
2593- first_strip_reply_to,
2594- footer_uri,
2595- forward_auto_discards,
2596- gateway_to_mail,
2597- gateway_to_news,
2598- goodbye_message_uri,
2599- header_matches,
2600- header_uri,
2601- hold_these_nonmembers,
2602- info,
2603- linked_newsgroup,
2604- max_days_to_hold,
2605- max_message_size,
2606- max_num_recipients,
2607- member_moderation_notice,
2608- mime_is_default_digest,
2609- moderator_password,
2610- new_member_options,
2611- nondigestable,
2612- nonmember_rejection_notice,
2613- obscure_addresses,
2614- owner_chain,
2615- owner_pipeline,
2616- personalize,
2617- post_id,
2618- posting_chain,
2619- posting_pipeline,
2620- preferred_language,
2621- private_roster,
2622- display_name,
2623- reject_these_nonmembers,
2624- reply_goes_to_list,
2625- reply_to_address,
2626- require_explicit_destination,
2627- respond_to_post_requests,
2628- scrub_nondigest,
2629- send_goodbye_message,
2630- send_reminders,
2631- send_welcome_message,
2632- subject_prefix,
2633- subscribe_auto_approval,
2634- subscribe_policy,
2635- topics,
2636- topics_bodylines_limit,
2637- topics_enabled,
2638- unsubscribe_policy,
2639- welcome_message_uri
2640- FROM mailinglist;
2641-
2642-CREATE TABLE member_backup(
2643- id INTEGER NOT NULL,
2644- _member_id TEXT,
2645- role INTEGER,
2646- moderation_action INTEGER,
2647- address_id INTEGER,
2648- preferences_id INTEGER,
2649- user_id INTEGER,
2650- PRIMARY KEY (id)
2651- );
2652-
2653-INSERT INTO member_backup SELECT
2654- id,
2655- _member_id,
2656- role,
2657- moderation_action,
2658- address_id,
2659- preferences_id,
2660- user_id
2661- FROM member;
2662-
2663-
2664--- Add the new columns. They'll get inserted at the Python layer.
2665-ALTER TABLE mailinglist_backup ADD COLUMN archive_policy INTEGER;
2666-ALTER TABLE mailinglist_backup ADD COLUMN list_id TEXT;
2667-ALTER TABLE mailinglist_backup ADD COLUMN nntp_prefix_subject_too INTEGER;
2668-ALTER TABLE mailinglist_backup ADD COLUMN newsgroup_moderation INTEGER;
2669-
2670-ALTER TABLE member_backup ADD COLUMN list_id TEXT;
2671
2672=== removed file 'src/mailman/database/schema/sqlite_20121015000000_01.sql'
2673--- src/mailman/database/schema/sqlite_20121015000000_01.sql 2013-09-01 15:15:08 +0000
2674+++ src/mailman/database/schema/sqlite_20121015000000_01.sql 1970-01-01 00:00:00 +0000
2675@@ -1,230 +0,0 @@
2676--- This file contains the sqlite3 schema migration from
2677--- 3.0b2 TO 3.0b3
2678---
2679--- 3.0b3 has been released thus you MAY NOT edit this file.
2680-
2681--- REMOVALS from the ban table:
2682--- REM mailing_list
2683-
2684--- ADDS to the ban table:
2685--- ADD list_id
2686-
2687-CREATE TABLE ban_backup (
2688- id INTEGER NOT NULL,
2689- email TEXT,
2690- PRIMARY KEY (id)
2691- );
2692-
2693-INSERT INTO ban_backup SELECT
2694- id, email
2695- FROM ban;
2696-
2697-ALTER TABLE ban_backup ADD COLUMN list_id TEXT;
2698-
2699--- REMOVALS from the mailinglist table.
2700--- REM new_member_options
2701--- REM send_reminders
2702--- REM subscribe_policy
2703--- REM unsubscribe_policy
2704--- REM subscribe_auto_approval
2705--- REM private_roster
2706--- REM admin_member_chunksize
2707-
2708-CREATE TABLE mailinglist_backup (
2709- id INTEGER NOT NULL,
2710- list_name TEXT,
2711- mail_host TEXT,
2712- allow_list_posts BOOLEAN,
2713- include_rfc2369_headers BOOLEAN,
2714- created_at TIMESTAMP,
2715- next_request_id INTEGER,
2716- next_digest_number INTEGER,
2717- digest_last_sent_at TIMESTAMP,
2718- volume INTEGER,
2719- last_post_at TIMESTAMP,
2720- accept_these_nonmembers BLOB,
2721- acceptable_aliases_id INTEGER,
2722- admin_immed_notify BOOLEAN,
2723- admin_notify_mchanges BOOLEAN,
2724- administrivia BOOLEAN,
2725- advertised BOOLEAN,
2726- anonymous_list BOOLEAN,
2727- autorespond_owner INTEGER,
2728- autoresponse_owner_text TEXT,
2729- autorespond_postings INTEGER,
2730- autoresponse_postings_text TEXT,
2731- autorespond_requests INTEGER,
2732- autoresponse_request_text TEXT,
2733- autoresponse_grace_period TEXT,
2734- forward_unrecognized_bounces_to INTEGER,
2735- process_bounces BOOLEAN,
2736- bounce_info_stale_after TEXT,
2737- bounce_matching_headers TEXT,
2738- bounce_notify_owner_on_disable BOOLEAN,
2739- bounce_notify_owner_on_removal BOOLEAN,
2740- bounce_score_threshold INTEGER,
2741- bounce_you_are_disabled_warnings INTEGER,
2742- bounce_you_are_disabled_warnings_interval TEXT,
2743- filter_action INTEGER,
2744- filter_content BOOLEAN,
2745- collapse_alternatives BOOLEAN,
2746- convert_html_to_plaintext BOOLEAN,
2747- default_member_action INTEGER,
2748- default_nonmember_action INTEGER,
2749- description TEXT,
2750- digest_footer_uri TEXT,
2751- digest_header_uri TEXT,
2752- digest_is_default BOOLEAN,
2753- digest_send_periodic BOOLEAN,
2754- digest_size_threshold FLOAT,
2755- digest_volume_frequency INTEGER,
2756- digestable BOOLEAN,
2757- discard_these_nonmembers BLOB,
2758- emergency BOOLEAN,
2759- encode_ascii_prefixes BOOLEAN,
2760- first_strip_reply_to BOOLEAN,
2761- footer_uri TEXT,
2762- forward_auto_discards BOOLEAN,
2763- gateway_to_mail BOOLEAN,
2764- gateway_to_news BOOLEAN,
2765- goodbye_message_uri TEXT,
2766- header_matches BLOB,
2767- header_uri TEXT,
2768- hold_these_nonmembers BLOB,
2769- info TEXT,
2770- linked_newsgroup TEXT,
2771- max_days_to_hold INTEGER,
2772- max_message_size INTEGER,
2773- max_num_recipients INTEGER,
2774- member_moderation_notice TEXT,
2775- mime_is_default_digest BOOLEAN,
2776- moderator_password TEXT,
2777- nondigestable BOOLEAN,
2778- nonmember_rejection_notice TEXT,
2779- obscure_addresses BOOLEAN,
2780- owner_chain TEXT,
2781- owner_pipeline TEXT,
2782- personalize INTEGER,
2783- post_id INTEGER,
2784- posting_chain TEXT,
2785- posting_pipeline TEXT,
2786- preferred_language TEXT,
2787- display_name TEXT,
2788- reject_these_nonmembers BLOB,
2789- reply_goes_to_list INTEGER,
2790- reply_to_address TEXT,
2791- require_explicit_destination BOOLEAN,
2792- respond_to_post_requests BOOLEAN,
2793- scrub_nondigest BOOLEAN,
2794- send_goodbye_message BOOLEAN,
2795- send_welcome_message BOOLEAN,
2796- subject_prefix TEXT,
2797- topics BLOB,
2798- topics_bodylines_limit INTEGER,
2799- topics_enabled BOOLEAN,
2800- welcome_message_uri TEXT,
2801- archive_policy INTEGER,
2802- list_id TEXT,
2803- nntp_prefix_subject_too INTEGER,
2804- newsgroup_moderation INTEGER,
2805- PRIMARY KEY (id)
2806- );
2807-
2808-INSERT INTO mailinglist_backup SELECT
2809- id,
2810- list_name,
2811- mail_host,
2812- allow_list_posts,
2813- include_rfc2369_headers,
2814- created_at,
2815- next_request_id,
2816- next_digest_number,
2817- digest_last_sent_at,
2818- volume,
2819- last_post_at,
2820- accept_these_nonmembers,
2821- acceptable_aliases_id,
2822- admin_immed_notify,
2823- admin_notify_mchanges,
2824- administrivia,
2825- advertised,
2826- anonymous_list,
2827- autorespond_owner,
2828- autoresponse_owner_text,
2829- autorespond_postings,
2830- autoresponse_postings_text,
2831- autorespond_requests,
2832- autoresponse_request_text,
2833- autoresponse_grace_period,
2834- forward_unrecognized_bounces_to,
2835- process_bounces,
2836- bounce_info_stale_after,
2837- bounce_matching_headers,
2838- bounce_notify_owner_on_disable,
2839- bounce_notify_owner_on_removal,
2840- bounce_score_threshold,
2841- bounce_you_are_disabled_warnings,
2842- bounce_you_are_disabled_warnings_interval,
2843- filter_action,
2844- filter_content,
2845- collapse_alternatives,
2846- convert_html_to_plaintext,
2847- default_member_action,
2848- default_nonmember_action,
2849- description,
2850- digest_footer_uri,
2851- digest_header_uri,
2852- digest_is_default,
2853- digest_send_periodic,
2854- digest_size_threshold,
2855- digest_volume_frequency,
2856- digestable,
2857- discard_these_nonmembers,
2858- emergency,
2859- encode_ascii_prefixes,
2860- first_strip_reply_to,
2861- footer_uri,
2862- forward_auto_discards,
2863- gateway_to_mail,
2864- gateway_to_news,
2865- goodbye_message_uri,
2866- header_matches,
2867- header_uri,
2868- hold_these_nonmembers,
2869- info,
2870- linked_newsgroup,
2871- max_days_to_hold,
2872- max_message_size,
2873- max_num_recipients,
2874- member_moderation_notice,
2875- mime_is_default_digest,
2876- moderator_password,
2877- nondigestable,
2878- nonmember_rejection_notice,
2879- obscure_addresses,
2880- owner_chain,
2881- owner_pipeline,
2882- personalize,
2883- post_id,
2884- posting_chain,
2885- posting_pipeline,
2886- preferred_language,
2887- display_name,
2888- reject_these_nonmembers,
2889- reply_goes_to_list,
2890- reply_to_address,
2891- require_explicit_destination,
2892- respond_to_post_requests,
2893- scrub_nondigest,
2894- send_goodbye_message,
2895- send_welcome_message,
2896- subject_prefix,
2897- topics,
2898- topics_bodylines_limit,
2899- topics_enabled,
2900- welcome_message_uri,
2901- archive_policy,
2902- list_id,
2903- nntp_prefix_subject_too,
2904- newsgroup_moderation
2905- FROM mailinglist;
2906
2907=== removed file 'src/mailman/database/schema/sqlite_20130406000000_01.sql'
2908--- src/mailman/database/schema/sqlite_20130406000000_01.sql 2013-11-26 02:26:15 +0000
2909+++ src/mailman/database/schema/sqlite_20130406000000_01.sql 1970-01-01 00:00:00 +0000
2910@@ -1,46 +0,0 @@
2911--- This file contains the SQLite schema migration from
2912--- 3.0b3 to 3.0b4
2913---
2914--- After 3.0b4 is released you may not edit this file.
2915-
2916--- For SQLite3 migration strategy, see
2917--- http://sqlite.org/faq.html#q11
2918-
2919--- ADD listarchiver table.
2920-
2921--- REMOVALs from the bounceevent table:
2922--- REM list_name
2923-
2924--- ADDs to the bounceevent table:
2925--- ADD list_id
2926-
2927--- ADDs to the mailinglist table:
2928--- ADD archiver_id
2929-
2930-CREATE TABLE bounceevent_backup (
2931- id INTEGER NOT NULL,
2932- email TEXT,
2933- 'timestamp' TIMESTAMP,
2934- message_id TEXT,
2935- context INTEGER,
2936- processed BOOLEAN,
2937- PRIMARY KEY (id)
2938- );
2939-
2940-INSERT INTO bounceevent_backup SELECT
2941- id, email, "timestamp", message_id,
2942- context, processed
2943- FROM bounceevent;
2944-
2945-ALTER TABLE bounceevent_backup ADD COLUMN list_id TEXT;
2946-
2947-CREATE TABLE listarchiver (
2948- id INTEGER NOT NULL,
2949- mailing_list_id INTEGER NOT NULL,
2950- name TEXT NOT NULL,
2951- _is_enabled BOOLEAN,
2952- PRIMARY KEY (id)
2953- );
2954-
2955-CREATE INDEX ix_listarchiver_mailing_list_id
2956- ON listarchiver(mailing_list_id);
2957
2958=== modified file 'src/mailman/database/sqlite.py'
2959--- src/mailman/database/sqlite.py 2014-01-01 14:59:42 +0000
2960+++ src/mailman/database/sqlite.py 2014-10-11 02:14:38 +0000
2961@@ -22,63 +22,27 @@
2962 __metaclass__ = type
2963 __all__ = [
2964 'SQLiteDatabase',
2965- 'make_temporary',
2966 ]
2967
2968
2969 import os
2970-import types
2971-import shutil
2972-import tempfile
2973
2974-from functools import partial
2975+from mailman.database.base import SABaseDatabase
2976 from urlparse import urlparse
2977
2978-from mailman.database.base import StormBaseDatabase
2979-from mailman.testing.helpers import configuration
2980-
2981
2982
2983
2984-class SQLiteDatabase(StormBaseDatabase):
2985+class SQLiteDatabase(SABaseDatabase):
2986 """Database class for SQLite."""
2987
2988- TAG = 'sqlite'
2989-
2990- def _database_exists(self, store):
2991- """See `BaseDatabase`."""
2992- table_query = 'select tbl_name from sqlite_master;'
2993- table_names = set(item[0] for item in
2994- store.execute(table_query))
2995- return 'version' in table_names
2996-
2997 def _prepare(self, url):
2998 parts = urlparse(url)
2999 assert parts.scheme == 'sqlite', (
3000 'Database url mismatch (expected sqlite prefix): {0}'.format(url))
3001+ # Ensure that the SQLite database file has the proper permissions,
3002+ # since SQLite doesn't play nice with umask.
3003 path = os.path.normpath(parts.path)
3004- fd = os.open(path, os.O_WRONLY | os.O_NONBLOCK | os.O_CREAT, 0666)
3005+ fd = os.open(path, os.O_WRONLY | os.O_NONBLOCK | os.O_CREAT, 0o666)
3006 # Ignore errors
3007 if fd > 0:
3008 os.close(fd)
3009-
3010-
3011-
3012
3013-# Test suite adapter for ITemporaryDatabase.
3014-
3015-def _cleanup(self, tempdir):
3016- shutil.rmtree(tempdir)
3017-
3018-
3019-def make_temporary(database):
3020- """Adapts by monkey patching an existing SQLite IDatabase."""
3021- tempdir = tempfile.mkdtemp()
3022- url = 'sqlite:///' + os.path.join(tempdir, 'mailman.db')
3023- with configuration('database', url=url):
3024- database.initialize()
3025- database._cleanup = types.MethodType(
3026- partial(_cleanup, tempdir=tempdir),
3027- database)
3028- # bool column values in SQLite must be integers.
3029- database.FALSE = 0
3030- database.TRUE = 1
3031- return database
3032
3033=== added directory 'src/mailman/database/tests'
3034=== removed directory 'src/mailman/database/tests'
3035=== added file 'src/mailman/database/tests/__init__.py'
3036=== removed file 'src/mailman/database/tests/__init__.py'
3037=== removed directory 'src/mailman/database/tests/data'
3038=== removed file 'src/mailman/database/tests/data/__init__.py'
3039=== removed file 'src/mailman/database/tests/data/mailman_01.db'
3040Binary files src/mailman/database/tests/data/mailman_01.db 2012-04-20 21:32:27 +0000 and src/mailman/database/tests/data/mailman_01.db 1970-01-01 00:00:00 +0000 differ
3041=== removed file 'src/mailman/database/tests/data/migration_postgres_1.sql'
3042--- src/mailman/database/tests/data/migration_postgres_1.sql 2012-07-26 04:22:19 +0000
3043+++ src/mailman/database/tests/data/migration_postgres_1.sql 1970-01-01 00:00:00 +0000
3044@@ -1,133 +0,0 @@
3045-INSERT INTO "acceptablealias" VALUES(1,'foo@example.com',1);
3046-INSERT INTO "acceptablealias" VALUES(2,'bar@example.com',1);
3047-
3048-INSERT INTO "address" VALUES(
3049- 1,'anne@example.com',NULL,'Anne Person',
3050- '2012-04-19 00:52:24.826432','2012-04-19 00:49:42.373769',1,2);
3051-INSERT INTO "address" VALUES(
3052- 2,'bart@example.com',NULL,'Bart Person',
3053- '2012-04-19 00:53:25.878800','2012-04-19 00:49:52.882050',2,4);
3054-
3055-INSERT INTO "domain" VALUES(
3056- 1,'example.com','http://example.com',NULL,'postmaster@example.com');
3057-
3058-INSERT INTO "mailinglist" VALUES(
3059- -- id,list_name,mail_host,include_list_post_header,include_rfc2369_headers
3060- 1,'test','example.com',True,True,
3061- -- created_at,admin_member_chunksize,next_request_id,next_digest_number
3062- '2012-04-19 00:46:13.173844',30,1,1,
3063- -- digest_last_sent_at,volume,last_post_at,accept_these_nonmembers
3064- NULL,1,NULL,E'\\x80025D71012E',
3065- -- acceptable_aliases_id,admin_immed_notify,admin_notify_mchanges
3066- NULL,True,False,
3067- -- administrivia,advertised,anonymous_list,archive,archive_private
3068- True,True,False,True,False,
3069- -- archive_volume_frequency
3070- 1,
3071- --autorespond_owner,autoresponse_owner_text
3072- 0,'',
3073- -- autorespond_postings,autoresponse_postings_text
3074- 0,'',
3075- -- autorespond_requests,authoresponse_requests_text
3076- 0,'',
3077- -- autoresponse_grace_period
3078- '90 days, 0:00:00',
3079- -- forward_unrecognized_bounces_to,process_bounces
3080- 1,True,
3081- -- bounce_info_stale_after,bounce_matching_headers
3082- '7 days, 0:00:00','
3083-# Lines that *start* with a ''#'' are comments.
3084-to: friend@public.com
3085-message-id: relay.comanche.denmark.eu
3086-from: list@listme.com
3087-from: .*@uplinkpro.com
3088-',
3089- -- bounce_notify_owner_on_disable,bounce_notify_owner_on_removal
3090- True,True,
3091- -- bounce_score_threshold,bounce_you_are_disabled_warnings
3092- 5,3,
3093- -- bounce_you_are_disabled_warnings_interval
3094- '7 days, 0:00:00',
3095- -- filter_action,filter_content,collapse_alternatives
3096- 2,False,True,
3097- -- convert_html_to_plaintext,default_member_action,default_nonmember_action
3098- False,4,0,
3099- -- description
3100- '',
3101- -- digest_footer_uri
3102- 'mailman:///$listname/$language/footer-generic.txt',
3103- -- digest_header_uri
3104- NULL,
3105- -- digest_is_default,digest_send_periodic,digest_size_threshold
3106- False,True,30.0,
3107- -- digest_volume_frequency,digestable,discard_these_nonmembers
3108- 1,True,E'\\x80025D71012E',
3109- -- emergency,encode_ascii_prefixes,first_strip_reply_to
3110- False,False,False,
3111- -- footer_uri
3112- 'mailman:///$listname/$language/footer-generic.txt',
3113- -- forward_auto_discards,gateway_to_mail,gateway_to_news
3114- True,False,FAlse,
3115- -- generic_nonmember_action,goodby_message_uri
3116- 1,'',
3117- -- header_matches,header_uri,hold_these_nonmembers,info,linked_newsgroup
3118- E'\\x80025D71012E',NULL,E'\\x80025D71012E','','',
3119- -- max_days_to_hold,max_message_size,max_num_recipients
3120- 0,40,10,
3121- -- member_moderation_notice,mime_is_default_digest,moderator_password
3122- '',False,NULL,
3123- -- new_member_options,news_moderation,news_prefix_subject_too
3124- 256,0,True,
3125- -- nntp_host,nondigestable,nonmember_rejection_notice,obscure_addresses
3126- '',True,'',True,
3127- -- owner_chain,owner_pipeline,personalize,post_id
3128- 'default-owner-chain','default-owner-pipeline',0,1,
3129- -- posting_chain,posting_pipeline,preferred_language,private_roster
3130- 'default-posting-chain','default-posting-pipeline','en',True,
3131- -- display_name,reject_these_nonmembers
3132- 'Test',E'\\x80025D71012E',
3133- -- reply_goes_to_list,reply_to_address
3134- 0,'',
3135- -- require_explicit_destination,respond_to_post_requests
3136- True,True,
3137- -- scrub_nondigest,send_goodbye_message,send_reminders,send_welcome_message
3138- False,True,True,True,
3139- -- subject_prefix,subscribe_auto_approval
3140- '[Test] ',E'\\x80025D71012E',
3141- -- subscribe_policy,topics,topics_bodylines_limit,topics_enabled
3142- 1,E'\\x80025D71012E',5,False,
3143- -- unsubscribe_policy,welcome_message_uri
3144- 0,'mailman:///welcome.txt');
3145-
3146-INSERT INTO "member" VALUES(
3147- 1,'d1243f4d-e604-4f6b-af52-98d0a7bce0f1',1,'test@example.com',4,NULL,5,1);
3148-INSERT INTO "member" VALUES(
3149- 2,'dccc3851-fdfb-4afa-90cf-bdcbf80ad0fd',2,'test@example.com',3,NULL,6,1);
3150-INSERT INTO "member" VALUES(
3151- 3,'479be431-45f2-473d-bc3c-7eac614030ac',3,'test@example.com',3,NULL,7,2);
3152-INSERT INTO "member" VALUES(
3153- 4,'e2dc604c-d93a-4b91-b5a8-749e3caade36',1,'test@example.com',4,NULL,8,2);
3154-
3155-INSERT INTO "preferences" VALUES(1,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
3156-INSERT INTO "preferences" VALUES(2,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
3157-INSERT INTO "preferences" VALUES(3,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
3158-INSERT INTO "preferences" VALUES(4,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
3159-INSERT INTO "preferences" VALUES(5,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
3160-INSERT INTO "preferences" VALUES(6,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
3161-INSERT INTO "preferences" VALUES(7,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
3162-INSERT INTO "preferences" VALUES(8,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
3163-
3164-INSERT INTO "user" VALUES(
3165- 1,'Anne Person',NULL,'0adf3caa-6f26-46f8-a11d-5256c8148592',
3166- '2012-04-19 00:49:42.370493',1,1);
3167-INSERT INTO "user" VALUES(
3168- 2,'Bart Person',NULL,'63f5d1a2-e533-4055-afe4-475dec3b1163',
3169- '2012-04-19 00:49:52.868746',2,3);
3170-
3171-INSERT INTO "uid" VALUES(1,'8bf9a615-f23e-4980-b7d1-90ac0203c66f');
3172-INSERT INTO "uid" VALUES(2,'0adf3caa-6f26-46f8-a11d-5256c8148592');
3173-INSERT INTO "uid" VALUES(3,'63f5d1a2-e533-4055-afe4-475dec3b1163');
3174-INSERT INTO "uid" VALUES(4,'d1243f4d-e604-4f6b-af52-98d0a7bce0f1');
3175-INSERT INTO "uid" VALUES(5,'dccc3851-fdfb-4afa-90cf-bdcbf80ad0fd');
3176-INSERT INTO "uid" VALUES(6,'479be431-45f2-473d-bc3c-7eac614030ac');
3177-INSERT INTO "uid" VALUES(7,'e2dc604c-d93a-4b91-b5a8-749e3caade36');
3178
3179=== removed file 'src/mailman/database/tests/data/migration_sqlite_1.sql'
3180--- src/mailman/database/tests/data/migration_sqlite_1.sql 2012-07-26 04:22:19 +0000
3181+++ src/mailman/database/tests/data/migration_sqlite_1.sql 1970-01-01 00:00:00 +0000
3182@@ -1,133 +0,0 @@
3183-INSERT INTO "acceptablealias" VALUES(1,'foo@example.com',1);
3184-INSERT INTO "acceptablealias" VALUES(2,'bar@example.com',1);
3185-
3186-INSERT INTO "address" VALUES(
3187- 1,'anne@example.com',NULL,'Anne Person',
3188- '2012-04-19 00:52:24.826432','2012-04-19 00:49:42.373769',1,2);
3189-INSERT INTO "address" VALUES(
3190- 2,'bart@example.com',NULL,'Bart Person',
3191- '2012-04-19 00:53:25.878800','2012-04-19 00:49:52.882050',2,4);
3192-
3193-INSERT INTO "domain" VALUES(
3194- 1,'example.com','http://example.com',NULL,'postmaster@example.com');
3195-
3196-INSERT INTO "mailinglist" VALUES(
3197- -- id,list_name,mail_host,include_list_post_header,include_rfc2369_headers
3198- 1,'test','example.com',1,1,
3199- -- created_at,admin_member_chunksize,next_request_id,next_digest_number
3200- '2012-04-19 00:46:13.173844',30,1,1,
3201- -- digest_last_sent_at,volume,last_post_at,accept_these_nonmembers
3202- NULL,1,NULL,X'80025D71012E',
3203- -- acceptable_aliases_id,admin_immed_notify,admin_notify_mchanges
3204- NULL,1,0,
3205- -- administrivia,advertised,anonymous_list,archive,archive_private
3206- 1,1,0,1,0,
3207- -- archive_volume_frequency
3208- 1,
3209- --autorespond_owner,autoresponse_owner_text
3210- 0,'',
3211- -- autorespond_postings,autoresponse_postings_text
3212- 0,'',
3213- -- autorespond_requests,authoresponse_requests_text
3214- 0,'',
3215- -- autoresponse_grace_period
3216- '90 days, 0:00:00',
3217- -- forward_unrecognized_bounces_to,process_bounces
3218- 1,1,
3219- -- bounce_info_stale_after,bounce_matching_headers
3220- '7 days, 0:00:00','
3221-# Lines that *start* with a ''#'' are comments.
3222-to: friend@public.com
3223-message-id: relay.comanche.denmark.eu
3224-from: list@listme.com
3225-from: .*@uplinkpro.com
3226-',
3227- -- bounce_notify_owner_on_disable,bounce_notify_owner_on_removal
3228- 1,1,
3229- -- bounce_score_threshold,bounce_you_are_disabled_warnings
3230- 5,3,
3231- -- bounce_you_are_disabled_warnings_interval
3232- '7 days, 0:00:00',
3233- -- filter_action,filter_content,collapse_alternatives
3234- 2,0,1,
3235- -- convert_html_to_plaintext,default_member_action,default_nonmember_action
3236- 0,4,0,
3237- -- description
3238- '',
3239- -- digest_footer_uri
3240- 'mailman:///$listname/$language/footer-generic.txt',
3241- -- digest_header_uri
3242- NULL,
3243- -- digest_is_default,digest_send_periodic,digest_size_threshold
3244- 0,1,30.0,
3245- -- digest_volume_frequency,digestable,discard_these_nonmembers
3246- 1,1,X'80025D71012E',
3247- -- emergency,encode_ascii_prefixes,first_strip_reply_to
3248- 0,0,0,
3249- -- footer_uri
3250- 'mailman:///$listname/$language/footer-generic.txt',
3251- -- forward_auto_discards,gateway_to_mail,gateway_to_news
3252- 1,0,0,
3253- -- generic_nonmember_action,goodby_message_uri
3254- 1,'',
3255- -- header_matches,header_uri,hold_these_nonmembers,info,linked_newsgroup
3256- X'80025D71012E',NULL,X'80025D71012E','','',
3257- -- max_days_to_hold,max_message_size,max_num_recipients
3258- 0,40,10,
3259- -- member_moderation_notice,mime_is_default_digest,moderator_password
3260- '',0,NULL,
3261- -- new_member_options,news_moderation,news_prefix_subject_too
3262- 256,0,1,
3263- -- nntp_host,nondigestable,nonmember_rejection_notice,obscure_addresses
3264- '',1,'',1,
3265- -- owner_chain,owner_pipeline,personalize,post_id
3266- 'default-owner-chain','default-owner-pipeline',0,1,
3267- -- posting_chain,posting_pipeline,preferred_language,private_roster
3268- 'default-posting-chain','default-posting-pipeline','en',1,
3269- -- display_name,reject_these_nonmembers
3270- 'Test',X'80025D71012E',
3271- -- reply_goes_to_list,reply_to_address
3272- 0,'',
3273- -- require_explicit_destination,respond_to_post_requests
3274- 1,1,
3275- -- scrub_nondigest,send_goodbye_message,send_reminders,send_welcome_message
3276- 0,1,1,1,
3277- -- subject_prefix,subscribe_auto_approval
3278- '[Test] ',X'80025D71012E',
3279- -- subscribe_policy,topics,topics_bodylines_limit,topics_enabled
3280- 1,X'80025D71012E',5,0,
3281- -- unsubscribe_policy,welcome_message_uri
3282- 0,'mailman:///welcome.txt');
3283-
3284-INSERT INTO "member" VALUES(
3285- 1,'d1243f4d-e604-4f6b-af52-98d0a7bce0f1',1,'test@example.com',4,NULL,5,1);
3286-INSERT INTO "member" VALUES(
3287- 2,'dccc3851-fdfb-4afa-90cf-bdcbf80ad0fd',2,'test@example.com',3,NULL,6,1);
3288-INSERT INTO "member" VALUES(
3289- 3,'479be431-45f2-473d-bc3c-7eac614030ac',3,'test@example.com',3,NULL,7,2);
3290-INSERT INTO "member" VALUES(
3291- 4,'e2dc604c-d93a-4b91-b5a8-749e3caade36',1,'test@example.com',4,NULL,8,2);
3292-
3293-INSERT INTO "preferences" VALUES(1,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
3294-INSERT INTO "preferences" VALUES(2,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
3295-INSERT INTO "preferences" VALUES(3,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
3296-INSERT INTO "preferences" VALUES(4,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
3297-INSERT INTO "preferences" VALUES(5,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
3298-INSERT INTO "preferences" VALUES(6,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
3299-INSERT INTO "preferences" VALUES(7,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
3300-INSERT INTO "preferences" VALUES(8,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
3301-
3302-INSERT INTO "user" VALUES(
3303- 1,'Anne Person',NULL,'0adf3caa-6f26-46f8-a11d-5256c8148592',
3304- '2012-04-19 00:49:42.370493',1,1);
3305-INSERT INTO "user" VALUES(
3306- 2,'Bart Person',NULL,'63f5d1a2-e533-4055-afe4-475dec3b1163',
3307- '2012-04-19 00:49:52.868746',2,3);
3308-
3309-INSERT INTO "uid" VALUES(1,'8bf9a615-f23e-4980-b7d1-90ac0203c66f');
3310-INSERT INTO "uid" VALUES(2,'0adf3caa-6f26-46f8-a11d-5256c8148592');
3311-INSERT INTO "uid" VALUES(3,'63f5d1a2-e533-4055-afe4-475dec3b1163');
3312-INSERT INTO "uid" VALUES(4,'d1243f4d-e604-4f6b-af52-98d0a7bce0f1');
3313-INSERT INTO "uid" VALUES(5,'dccc3851-fdfb-4afa-90cf-bdcbf80ad0fd');
3314-INSERT INTO "uid" VALUES(6,'479be431-45f2-473d-bc3c-7eac614030ac');
3315-INSERT INTO "uid" VALUES(7,'e2dc604c-d93a-4b91-b5a8-749e3caade36');
3316
3317=== added file 'src/mailman/database/tests/test_factory.py'
3318--- src/mailman/database/tests/test_factory.py 1970-01-01 00:00:00 +0000
3319+++ src/mailman/database/tests/test_factory.py 2014-10-11 02:14:38 +0000
3320@@ -0,0 +1,162 @@
3321+# Copyright (C) 2013-2014 by the Free Software Foundation, Inc.
3322+#
3323+# This file is part of GNU Mailman.
3324+#
3325+# GNU Mailman is free software: you can redistribute it and/or modify it under
3326+# the terms of the GNU General Public License as published by the Free
3327+# Software Foundation, either version 3 of the License, or (at your option)
3328+# any later version.
3329+#
3330+# GNU Mailman is distributed in the hope that it will be useful, but WITHOUT
3331+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
3332+# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
3333+# more details.
3334+#
3335+# You should have received a copy of the GNU General Public License along with
3336+# GNU Mailman. If not, see <http://www.gnu.org/licenses/>.
3337+
3338+"""Test database schema migrations"""
3339+
3340+from __future__ import absolute_import, print_function, unicode_literals
3341+
3342+__metaclass__ = type
3343+__all__ = [
3344+ ]
3345+
3346+
3347+import unittest
3348+import types
3349+
3350+import alembic.command
3351+from mock import Mock
3352+from sqlalchemy import MetaData, Table, Column, Integer, Unicode
3353+from sqlalchemy.schema import Index
3354+from sqlalchemy.exc import ProgrammingError, OperationalError
3355+
3356+from mailman.config import config
3357+from mailman.testing.layers import ConfigLayer
3358+from mailman.database.factory import SchemaManager, _reset
3359+from mailman.database.sqlite import SQLiteDatabase
3360+from mailman.database.alembic import alembic_cfg
3361+from mailman.database.model import Model
3362+
3363+
3364+
3365
3366+class TestSchemaManager(unittest.TestCase):
3367+
3368+ layer = ConfigLayer
3369+
3370+ def setUp(self):
3371+ # Drop the existing database
3372+ Model.metadata.drop_all(config.db.engine)
3373+ md = MetaData()
3374+ md.reflect(bind=config.db.engine)
3375+ for tablename in ("alembic_version", "version"):
3376+ if tablename in md.tables:
3377+ md.tables[tablename].drop(config.db.engine)
3378+ self.schema_mgr = SchemaManager(config.db)
3379+
3380+ def tearDown(self):
3381+ self._drop_storm_database()
3382+ # Restore a virgin DB
3383+ Model.metadata.create_all(config.db.engine)
3384+
3385+
3386+ def _table_exists(self, tablename):
3387+ md = MetaData()
3388+ md.reflect(bind=config.db.engine)
3389+ return tablename in md.tables
3390+
3391+ def _create_storm_database(self, revision):
3392+ version_table = Table("version", Model.metadata,
3393+ Column("id", Integer, primary_key=True),
3394+ Column("component", Unicode),
3395+ Column("version", Unicode),
3396+ )
3397+ version_table.create(config.db.engine)
3398+ config.db.store.execute(version_table.insert().values(
3399+ component='schema', version=revision))
3400+ config.db.commit()
3401+ # Other Storm specific changes, those SQL statements hopefully work on
3402+ # all DB engines...
3403+ config.db.engine.execute(
3404+ "ALTER TABLE mailinglist ADD COLUMN acceptable_aliases_id INT")
3405+ Index("ix_user__user_id").drop(bind=config.db.engine)
3406+ # Don't pollute our main metadata object, create a new one
3407+ md = MetaData()
3408+ user_table = Model.metadata.tables["user"].tometadata(md)
3409+ Index("ix_user_user_id", user_table.c._user_id
3410+ ).create(bind=config.db.engine)
3411+ config.db.commit()
3412+
3413+ def _drop_storm_database(self):
3414+ """
3415+ Remove the leftovers from a Storm DB.
3416+ (you must issue a drop_all() afterwards)
3417+ """
3418+ if "version" in Model.metadata.tables:
3419+ version = Model.metadata.tables["version"]
3420+ version.drop(config.db.engine, checkfirst=True)
3421+ Model.metadata.remove(version)
3422+ try:
3423+ Index("ix_user_user_id").drop(bind=config.db.engine)
3424+ except (ProgrammingError, OperationalError) as e:
3425+ # non-existant (PGSQL raises a ProgrammingError, while SQLite
3426+ # raises an OperationalError)
3427+ pass
3428+ config.db.commit()
3429+
3430+
3431+ def test_current_db(self):
3432+ """The database is already at the latest version"""
3433+ alembic.command.stamp(alembic_cfg, "head")
3434+ self.schema_mgr._create = Mock()
3435+ self.schema_mgr._upgrade = Mock()
3436+ self.schema_mgr.setup_db()
3437+ self.assertFalse(self.schema_mgr._create.called)
3438+ self.assertFalse(self.schema_mgr._upgrade.called)
3439+
3440+ def test_initial(self):
3441+ """No existing database"""
3442+ self.assertFalse(self._table_exists("mailinglist"))
3443+ self.assertFalse(self._table_exists("alembic_version"))
3444+ self.schema_mgr._upgrade = Mock()
3445+ self.schema_mgr.setup_db()
3446+ self.assertFalse(self.schema_mgr._upgrade.called)
3447+ self.assertTrue(self._table_exists("mailinglist"))
3448+ self.assertTrue(self._table_exists("alembic_version"))
3449+
3450+ def test_storm(self):
3451+ """Existing Storm database"""
3452+ Model.metadata.create_all(config.db.engine)
3453+ self._create_storm_database(
3454+ self.schema_mgr.LAST_STORM_SCHEMA_VERSION)
3455+ self.schema_mgr._create = Mock()
3456+ self.schema_mgr.setup_db()
3457+ self.assertFalse(self.schema_mgr._create.called)
3458+ self.assertTrue(self._table_exists("mailinglist")
3459+ and self._table_exists("alembic_version")
3460+ and not self._table_exists("version"))
3461+
3462+ def test_old_storm(self):
3463+ """Existing Storm database in an old version"""
3464+ Model.metadata.create_all(config.db.engine)
3465+ self._create_storm_database("001")
3466+ self.schema_mgr._create = Mock()
3467+ self.assertRaises(RuntimeError, self.schema_mgr.setup_db)
3468+ self.assertFalse(self.schema_mgr._create.called)
3469+
3470+ def test_old_db(self):
3471+ """The database is in an old revision, must upgrade"""
3472+ alembic.command.stamp(alembic_cfg, "head")
3473+ md = MetaData()
3474+ md.reflect(bind=config.db.engine)
3475+ config.db.store.execute(md.tables["alembic_version"].delete())
3476+ config.db.store.execute(md.tables["alembic_version"].insert().values(
3477+ version_num="dummyrevision"))
3478+ config.db.commit()
3479+ self.schema_mgr._create = Mock()
3480+ self.schema_mgr._upgrade = Mock()
3481+ self.schema_mgr.setup_db()
3482+ self.assertFalse(self.schema_mgr._create.called)
3483+ self.assertTrue(self.schema_mgr._upgrade.called)
3484
3485=== removed file 'src/mailman/database/tests/test_migrations.py'
3486--- src/mailman/database/tests/test_migrations.py 2014-01-01 14:59:42 +0000
3487+++ src/mailman/database/tests/test_migrations.py 1970-01-01 00:00:00 +0000
3488@@ -1,506 +0,0 @@
3489-# Copyright (C) 2012-2014 by the Free Software Foundation, Inc.
3490-#
3491-# This file is part of GNU Mailman.
3492-#
3493-# GNU Mailman is free software: you can redistribute it and/or modify it under
3494-# the terms of the GNU General Public License as published by the Free
3495-# Software Foundation, either version 3 of the License, or (at your option)
3496-# any later version.
3497-#
3498-# GNU Mailman is distributed in the hope that it will be useful, but WITHOUT
3499-# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
3500-# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
3501-# more details.
3502-#
3503-# You should have received a copy of the GNU General Public License along with
3504-# GNU Mailman. If not, see <http://www.gnu.org/licenses/>.
3505-
3506-"""Test schema migrations."""
3507-
3508-from __future__ import absolute_import, print_function, unicode_literals
3509-
3510-__metaclass__ = type
3511-__all__ = [
3512- 'TestMigration20120407MigratedData',
3513- 'TestMigration20120407Schema',
3514- 'TestMigration20120407UnchangedData',
3515- 'TestMigration20121015MigratedData',
3516- 'TestMigration20121015Schema',
3517- 'TestMigration20130406MigratedData',
3518- 'TestMigration20130406Schema',
3519- ]
3520-
3521-
3522-import unittest
3523-
3524-from datetime import datetime
3525-from operator import attrgetter
3526-from pkg_resources import resource_string
3527-from sqlite3 import OperationalError
3528-from storm.exceptions import DatabaseError
3529-from zope.component import getUtility
3530-
3531-from mailman.interfaces.database import IDatabaseFactory
3532-from mailman.interfaces.domain import IDomainManager
3533-from mailman.interfaces.archiver import ArchivePolicy
3534-from mailman.interfaces.bounce import BounceContext
3535-from mailman.interfaces.listmanager import IListManager
3536-from mailman.interfaces.mailinglist import IAcceptableAliasSet
3537-from mailman.interfaces.nntp import NewsgroupModeration
3538-from mailman.interfaces.subscriptions import ISubscriptionService
3539-from mailman.model.bans import Ban
3540-from mailman.model.bounce import BounceEvent
3541-from mailman.testing.helpers import temporary_db
3542-from mailman.testing.layers import ConfigLayer
3543-
3544-
3545-
3546
3547-class MigrationTestBase(unittest.TestCase):
3548- """Test database migrations."""
3549-
3550- layer = ConfigLayer
3551-
3552- def setUp(self):
3553- self._database = getUtility(IDatabaseFactory, 'temporary').create()
3554-
3555- def tearDown(self):
3556- self._database._cleanup()
3557-
3558- def _table_missing_present(self, migrations, missing, present):
3559- """The appropriate migrations leave some tables missing and present.
3560-
3561- :param migrations: Sequence of migrations to load.
3562- :param missing: Tables which should be missing.
3563- :param present: Tables which should be present.
3564- """
3565- for migration in migrations:
3566- self._database.load_migrations(migration)
3567- self._database.store.commit()
3568- for table in missing:
3569- self.assertRaises(OperationalError,
3570- self._database.store.execute,
3571- 'select * from {};'.format(table))
3572- for table in present:
3573- self._database.store.execute('select * from {};'.format(table))
3574-
3575- def _missing_present(self, table, migrations, missing, present):
3576- """The appropriate migrations leave columns missing and present.
3577-
3578- :param table: The table to test columns from.
3579- :param migrations: Sequence of migrations to load.
3580- :param missing: Set of columns which should be missing after the
3581- migrations are loaded.
3582- :param present: Set of columns which should be present after the
3583- migrations are loaded.
3584- """
3585- for migration in migrations:
3586- self._database.load_migrations(migration)
3587- self._database.store.commit()
3588- for column in missing:
3589- self.assertRaises(DatabaseError,
3590- self._database.store.execute,
3591- 'select {0} from {1};'.format(column, table))
3592- self._database.store.rollback()
3593- for column in present:
3594- # This should not produce an exception. Is there some better test
3595- # that we can perform?
3596- self._database.store.execute(
3597- 'select {0} from {1};'.format(column, table))
3598-
3599-
3600-
3601
3602-class TestMigration20120407Schema(MigrationTestBase):
3603- """Test column migrations."""
3604-
3605- def test_pre_upgrade_columns_migration(self):
3606- # Test that before the migration, the old table columns are present
3607- # and the new database columns are not.
3608- self._missing_present('mailinglist',
3609- ['20120406999999'],
3610- # New columns are missing.
3611- ('allow_list_posts',
3612- 'archive_policy',
3613- 'list_id',
3614- 'nntp_prefix_subject_too'),
3615- # Old columns are present.
3616- ('archive',
3617- 'archive_private',
3618- 'archive_volume_frequency',
3619- 'generic_nonmember_action',
3620- 'include_list_post_header',
3621- 'news_moderation',
3622- 'news_prefix_subject_too',
3623- 'nntp_host'))
3624- self._missing_present('member',
3625- ['20120406999999'],
3626- ('list_id',),
3627- ('mailing_list',))
3628-
3629- def test_post_upgrade_columns_migration(self):
3630- # Test that after the migration, the old table columns are missing
3631- # and the new database columns are present.
3632- self._missing_present('mailinglist',
3633- ['20120406999999',
3634- '20120407000000'],
3635- # The old columns are missing.
3636- ('archive',
3637- 'archive_private',
3638- 'archive_volume_frequency',
3639- 'generic_nonmember_action',
3640- 'include_list_post_header',
3641- 'news_moderation',
3642- 'news_prefix_subject_too',
3643- 'nntp_host'),
3644- # The new columns are present.
3645- ('allow_list_posts',
3646- 'archive_policy',
3647- 'list_id',
3648- 'nntp_prefix_subject_too'))
3649- self._missing_present('member',
3650- ['20120406999999',
3651- '20120407000000'],
3652- ('mailing_list',),
3653- ('list_id',))
3654-
3655-
3656-
3657
3658-class TestMigration20120407UnchangedData(MigrationTestBase):
3659- """Test non-migrated data."""
3660-
3661- def setUp(self):
3662- MigrationTestBase.setUp(self)
3663- # Load all the migrations to just before the one we're testing.
3664- self._database.load_migrations('20120406999999')
3665- # Load the previous schema's sample data.
3666- sample_data = resource_string(
3667- 'mailman.database.tests.data',
3668- 'migration_{0}_1.sql'.format(self._database.TAG))
3669- self._database.load_sql(self._database.store, sample_data)
3670- # XXX 2012-12-28: We have to load the last migration defined in the
3671- # system, otherwise the ORM model will not match the SQL table
3672- # definitions and we'll get OperationalErrors from SQLite.
3673- self._database.load_migrations('20121015000000')
3674-
3675- def test_migration_domains(self):
3676- # Test that the domains table, which isn't touched, doesn't change.
3677- with temporary_db(self._database):
3678- # Check that the domains survived the migration. This table
3679- # was not touched so it should be fine.
3680- domains = list(getUtility(IDomainManager))
3681- self.assertEqual(len(domains), 1)
3682- self.assertEqual(domains[0].mail_host, 'example.com')
3683-
3684- def test_migration_mailing_lists(self):
3685- # Test that the mailing lists survive migration.
3686- with temporary_db(self._database):
3687- # There should be exactly one mailing list defined.
3688- mlists = list(getUtility(IListManager).mailing_lists)
3689- self.assertEqual(len(mlists), 1)
3690- self.assertEqual(mlists[0].fqdn_listname, 'test@example.com')
3691-
3692- def test_migration_acceptable_aliases(self):
3693- # Test that the mailing list's acceptable aliases survive migration.
3694- # This proves that foreign key references are migrated properly.
3695- with temporary_db(self._database):
3696- mlist = getUtility(IListManager).get('test@example.com')
3697- aliases_set = IAcceptableAliasSet(mlist)
3698- self.assertEqual(set(aliases_set.aliases),
3699- set(['foo@example.com', 'bar@example.com']))
3700-
3701- def test_migration_members(self):
3702- # Test that the members of a mailing list all survive migration.
3703- with temporary_db(self._database):
3704- mlist = getUtility(IListManager).get('test@example.com')
3705- # Test that all the members we expect are still there. Start with
3706- # the two list delivery members.
3707- addresses = set(address.email
3708- for address in mlist.members.addresses)
3709- self.assertEqual(addresses,
3710- set(['anne@example.com', 'bart@example.com']))
3711- # There is one owner.
3712- owners = set(address.email for address in mlist.owners.addresses)
3713- self.assertEqual(len(owners), 1)
3714- self.assertEqual(owners.pop(), 'anne@example.com')
3715- # There is one moderator.
3716- moderators = set(address.email
3717- for address in mlist.moderators.addresses)
3718- self.assertEqual(len(moderators), 1)
3719- self.assertEqual(moderators.pop(), 'bart@example.com')
3720-
3721-
3722-
3723
3724-class TestMigration20120407MigratedData(MigrationTestBase):
3725- """Test affected migration data."""
3726-
3727- def setUp(self):
3728- MigrationTestBase.setUp(self)
3729- # Load all the migrations to just before the one we're testing.
3730- self._database.load_migrations('20120406999999')
3731- # Load the previous schema's sample data.
3732- sample_data = resource_string(
3733- 'mailman.database.tests.data',
3734- 'migration_{0}_1.sql'.format(self._database.TAG))
3735- self._database.load_sql(self._database.store, sample_data)
3736-
3737- def _upgrade(self):
3738- # XXX 2012-12-28: We have to load the last migration defined in the
3739- # system, otherwise the ORM model will not match the SQL table
3740- # definitions and we'll get OperationalErrors from SQLite.
3741- self._database.load_migrations('20121015000000')
3742-
3743- def test_migration_archive_policy_never_0(self):
3744- # Test that the new archive_policy value is updated correctly. In the
3745- # case of old column archive=0, the archive_private column is
3746- # ignored. This test sets it to 0 to ensure it's ignored.
3747- self._database.store.execute(
3748- 'UPDATE mailinglist SET archive = {0}, archive_private = {0} '
3749- 'WHERE id = 1;'.format(self._database.FALSE))
3750- # Complete the migration
3751- self._upgrade()
3752- with temporary_db(self._database):
3753- mlist = getUtility(IListManager).get('test@example.com')
3754- self.assertEqual(mlist.archive_policy, ArchivePolicy.never)
3755-
3756- def test_migration_archive_policy_never_1(self):
3757- # Test that the new archive_policy value is updated correctly. In the
3758- # case of old column archive=0, the archive_private column is
3759- # ignored. This test sets it to 1 to ensure it's ignored.
3760- self._database.store.execute(
3761- 'UPDATE mailinglist SET archive = {0}, archive_private = {1} '
3762- 'WHERE id = 1;'.format(self._database.FALSE,
3763- self._database.TRUE))
3764- # Complete the migration
3765- self._upgrade()
3766- with temporary_db(self._database):
3767- mlist = getUtility(IListManager).get('test@example.com')
3768- self.assertEqual(mlist.archive_policy, ArchivePolicy.never)
3769-
3770- def test_archive_policy_private(self):
3771- # Test that the new archive_policy value is updated correctly for
3772- # private archives.
3773- self._database.store.execute(
3774- 'UPDATE mailinglist SET archive = {0}, archive_private = {0} '
3775- 'WHERE id = 1;'.format(self._database.TRUE))
3776- # Complete the migration
3777- self._upgrade()
3778- with temporary_db(self._database):
3779- mlist = getUtility(IListManager).get('test@example.com')
3780- self.assertEqual(mlist.archive_policy, ArchivePolicy.private)
3781-
3782- def test_archive_policy_public(self):
3783- # Test that the new archive_policy value is updated correctly for
3784- # public archives.
3785- self._database.store.execute(
3786- 'UPDATE mailinglist SET archive = {1}, archive_private = {0} '
3787- 'WHERE id = 1;'.format(self._database.FALSE,
3788- self._database.TRUE))
3789- # Complete the migration
3790- self._upgrade()
3791- with temporary_db(self._database):
3792- mlist = getUtility(IListManager).get('test@example.com')
3793- self.assertEqual(mlist.archive_policy, ArchivePolicy.public)
3794-
3795- def test_list_id(self):
3796- # Test that the mailinglist table gets a list_id column.
3797- self._upgrade()
3798- with temporary_db(self._database):
3799- mlist = getUtility(IListManager).get('test@example.com')
3800- self.assertEqual(mlist.list_id, 'test.example.com')
3801-
3802- def test_list_id_member(self):
3803- # Test that the member table's mailing_list column becomes list_id.
3804- self._upgrade()
3805- with temporary_db(self._database):
3806- service = getUtility(ISubscriptionService)
3807- members = list(service.find_members(list_id='test.example.com'))
3808- self.assertEqual(len(members), 4)
3809-
3810- def test_news_moderation_none(self):
3811- # Test that news_moderation becomes newsgroup_moderation.
3812- self._database.store.execute(
3813- 'UPDATE mailinglist SET news_moderation = 0 '
3814- 'WHERE id = 1;')
3815- self._upgrade()
3816- with temporary_db(self._database):
3817- mlist = getUtility(IListManager).get('test@example.com')
3818- self.assertEqual(mlist.newsgroup_moderation,
3819- NewsgroupModeration.none)
3820-
3821- def test_news_moderation_open_moderated(self):
3822- # Test that news_moderation becomes newsgroup_moderation.
3823- self._database.store.execute(
3824- 'UPDATE mailinglist SET news_moderation = 1 '
3825- 'WHERE id = 1;')
3826- self._upgrade()
3827- with temporary_db(self._database):
3828- mlist = getUtility(IListManager).get('test@example.com')
3829- self.assertEqual(mlist.newsgroup_moderation,
3830- NewsgroupModeration.open_moderated)
3831-
3832- def test_news_moderation_moderated(self):
3833- # Test that news_moderation becomes newsgroup_moderation.
3834- self._database.store.execute(
3835- 'UPDATE mailinglist SET news_moderation = 2 '
3836- 'WHERE id = 1;')
3837- self._upgrade()
3838- with temporary_db(self._database):
3839- mlist = getUtility(IListManager).get('test@example.com')
3840- self.assertEqual(mlist.newsgroup_moderation,
3841- NewsgroupModeration.moderated)
3842-
3843- def test_nntp_prefix_subject_too_false(self):
3844- # Test that news_prefix_subject_too becomes nntp_prefix_subject_too.
3845- self._database.store.execute(
3846- 'UPDATE mailinglist SET news_prefix_subject_too = {0} '
3847- 'WHERE id = 1;'.format(self._database.FALSE))
3848- self._upgrade()
3849- with temporary_db(self._database):
3850- mlist = getUtility(IListManager).get('test@example.com')
3851- self.assertFalse(mlist.nntp_prefix_subject_too)
3852-
3853- def test_nntp_prefix_subject_too_true(self):
3854- # Test that news_prefix_subject_too becomes nntp_prefix_subject_too.
3855- self._database.store.execute(
3856- 'UPDATE mailinglist SET news_prefix_subject_too = {0} '
3857- 'WHERE id = 1;'.format(self._database.TRUE))
3858- self._upgrade()
3859- with temporary_db(self._database):
3860- mlist = getUtility(IListManager).get('test@example.com')
3861- self.assertTrue(mlist.nntp_prefix_subject_too)
3862-
3863- def test_allow_list_posts_false(self):
3864- # Test that include_list_post_header -> allow_list_posts.
3865- self._database.store.execute(
3866- 'UPDATE mailinglist SET include_list_post_header = {0} '
3867- 'WHERE id = 1;'.format(self._database.FALSE))
3868- self._upgrade()
3869- with temporary_db(self._database):
3870- mlist = getUtility(IListManager).get('test@example.com')
3871- self.assertFalse(mlist.allow_list_posts)
3872-
3873- def test_allow_list_posts_true(self):
3874- # Test that include_list_post_header -> allow_list_posts.
3875- self._database.store.execute(
3876- 'UPDATE mailinglist SET include_list_post_header = {0} '
3877- 'WHERE id = 1;'.format(self._database.TRUE))
3878- self._upgrade()
3879- with temporary_db(self._database):
3880- mlist = getUtility(IListManager).get('test@example.com')
3881- self.assertTrue(mlist.allow_list_posts)
3882-
3883-
3884-
3885
3886-class TestMigration20121015Schema(MigrationTestBase):
3887- """Test column migrations."""
3888-
3889- def test_pre_upgrade_column_migrations(self):
3890- self._missing_present('ban',
3891- ['20121014999999'],
3892- ('list_id',),
3893- ('mailing_list',))
3894- self._missing_present('mailinglist',
3895- ['20121014999999'],
3896- (),
3897- ('new_member_options', 'send_reminders',
3898- 'subscribe_policy', 'unsubscribe_policy',
3899- 'subscribe_auto_approval', 'private_roster',
3900- 'admin_member_chunksize'),
3901- )
3902-
3903- def test_post_upgrade_column_migrations(self):
3904- self._missing_present('ban',
3905- ['20121014999999',
3906- '20121015000000'],
3907- ('mailing_list',),
3908- ('list_id',))
3909- self._missing_present('mailinglist',
3910- ['20121014999999',
3911- '20121015000000'],
3912- ('new_member_options', 'send_reminders',
3913- 'subscribe_policy', 'unsubscribe_policy',
3914- 'subscribe_auto_approval', 'private_roster',
3915- 'admin_member_chunksize'),
3916- ())
3917-
3918-
3919-
3920
3921-class TestMigration20121015MigratedData(MigrationTestBase):
3922- """Test non-migrated data."""
3923-
3924- def test_migration_bans(self):
3925- # Load all the migrations to just before the one we're testing.
3926- self._database.load_migrations('20121014999999')
3927- # Insert a list-specific ban.
3928- self._database.store.execute("""
3929- INSERT INTO ban VALUES (
3930- 1, 'anne@example.com', 'test@example.com');
3931- """)
3932- # Insert a global ban.
3933- self._database.store.execute("""
3934- INSERT INTO ban VALUES (
3935- 2, 'bart@example.com', NULL);
3936- """)
3937- # Update to the current migration we're testing.
3938- self._database.load_migrations('20121015000000')
3939- # Now both the local and global bans should still be present.
3940- bans = sorted(self._database.store.find(Ban),
3941- key=attrgetter('email'))
3942- self.assertEqual(bans[0].email, 'anne@example.com')
3943- self.assertEqual(bans[0].list_id, 'test.example.com')
3944- self.assertEqual(bans[1].email, 'bart@example.com')
3945- self.assertEqual(bans[1].list_id, None)
3946-
3947-
3948-
3949
3950-class TestMigration20130406Schema(MigrationTestBase):
3951- """Test column migrations."""
3952-
3953- def test_pre_upgrade_column_migrations(self):
3954- self._missing_present('bounceevent',
3955- ['20130405999999'],
3956- ('list_id',),
3957- ('list_name',))
3958-
3959- def test_post_upgrade_column_migrations(self):
3960- self._missing_present('bounceevent',
3961- ['20130405999999',
3962- '20130406000000'],
3963- ('list_name',),
3964- ('list_id',))
3965-
3966- def test_pre_listarchiver_table(self):
3967- self._table_missing_present(['20130405999999'], ('listarchiver',), ())
3968-
3969- def test_post_listarchiver_table(self):
3970- self._table_missing_present(['20130405999999',
3971- '20130406000000'],
3972- (),
3973- ('listarchiver',))
3974-
3975-
3976-
3977
3978-class TestMigration20130406MigratedData(MigrationTestBase):
3979- """Test migrated data."""
3980-
3981- def test_migration_bounceevent(self):
3982- # Load all migrations to just before the one we're testing.
3983- self._database.load_migrations('20130405999999')
3984- # Insert a bounce event.
3985- self._database.store.execute("""
3986- INSERT INTO bounceevent VALUES (
3987- 1, 'test@example.com', 'anne@example.com',
3988- '2013-04-06 21:12:00', '<abc@example.com>',
3989- 1, 0);
3990- """)
3991- # Update to the current migration we're testing
3992- self._database.load_migrations('20130406000000')
3993- # The bounce event should exist, but with a list-id instead of a fqdn
3994- # list name.
3995- events = list(self._database.store.find(BounceEvent))
3996- self.assertEqual(len(events), 1)
3997- self.assertEqual(events[0].list_id, 'test.example.com')
3998- self.assertEqual(events[0].email, 'anne@example.com')
3999- self.assertEqual(events[0].timestamp, datetime(2013, 4, 6, 21, 12))
4000- self.assertEqual(events[0].message_id, '<abc@example.com>')
4001- self.assertEqual(events[0].context, BounceContext.normal)
4002- self.assertFalse(events[0].processed)
4003
4004=== modified file 'src/mailman/database/types.py'
4005--- src/mailman/database/types.py 2014-04-28 15:23:35 +0000
4006+++ src/mailman/database/types.py 2014-10-11 02:14:38 +0000
4007@@ -23,43 +23,70 @@
4008 __metaclass__ = type
4009 __all__ = [
4010 'Enum',
4011+ 'UUID',
4012 ]
4013
4014+import uuid
4015
4016-from storm.properties import SimpleProperty
4017-from storm.variables import Variable
4018+from sqlalchemy import Integer
4019+from sqlalchemy.dialects import postgresql
4020+from sqlalchemy.types import TypeDecorator, CHAR
4021
4022
4023
4024
4025-class _EnumVariable(Variable):
4026- """Storm variable for supporting enum types.
4027+class Enum(TypeDecorator):
4028+ """Handle Python 3.4 style enums.
4029
4030- To use this, make the database column a INTEGER.
4031+ Stores an integer-based Enum as an integer in the database, and
4032+ converts it on-the-fly.
4033 """
4034-
4035- def __init__(self, *args, **kws):
4036- self._enum = kws.pop('enum')
4037- super(_EnumVariable, self).__init__(*args, **kws)
4038-
4039- def parse_set(self, value, from_db):
4040- if value is None:
4041- return None
4042- if not from_db:
4043- return value
4044- return self._enum(value)
4045-
4046- def parse_get(self, value, to_db):
4047- if value is None:
4048- return None
4049- if not to_db:
4050- return value
4051+ impl = Integer
4052+
4053+ def __init__(self, enum, *args, **kw):
4054+ self.enum = enum
4055+ super(Enum, self).__init__(*args, **kw)
4056+
4057+ def process_bind_param(self, value, dialect):
4058+ if value is None:
4059+ return None
4060 return value.value
4061
4062-
4063-class Enum(SimpleProperty):
4064- """Custom type for Storm supporting enums."""
4065-
4066- variable_class = _EnumVariable
4067-
4068- def __init__(self, enum=None):
4069- super(Enum, self).__init__(enum=enum)
4070+ def process_result_value(self, value, dialect):
4071+ if value is None:
4072+ return None
4073+ return self.enum(value)
4074+
4075+
4076+
4077
4078+class UUID(TypeDecorator):
4079+ """Platform-independent GUID type.
4080+
4081+ Uses Postgresql's UUID type, otherwise uses
4082+ CHAR(32), storing as stringified hex values.
4083+
4084+ """
4085+ impl = CHAR
4086+
4087+ def load_dialect_impl(self, dialect):
4088+ if dialect.name == 'postgresql':
4089+ return dialect.type_descriptor(postgresql.UUID())
4090+ else:
4091+ return dialect.type_descriptor(CHAR(32))
4092+
4093+ def process_bind_param(self, value, dialect):
4094+ if value is None:
4095+ return value
4096+ elif dialect.name == 'postgresql':
4097+ return str(value)
4098+ else:
4099+ if not isinstance(value, uuid.UUID):
4100+ return "%.32x" % uuid.UUID(value)
4101+ else:
4102+ # hexstring
4103+ return "%.32x" % value
4104+
4105+ def process_result_value(self, value, dialect):
4106+ if value is None:
4107+ return value
4108+ else:
4109+ return uuid.UUID(value)
4110
4111=== modified file 'src/mailman/interfaces/database.py'
4112--- src/mailman/interfaces/database.py 2014-01-01 14:59:42 +0000
4113+++ src/mailman/interfaces/database.py 2014-10-11 02:14:38 +0000
4114@@ -24,7 +24,6 @@
4115 'DatabaseError',
4116 'IDatabase',
4117 'IDatabaseFactory',
4118- 'ITemporaryDatabase',
4119 ]
4120
4121
4122@@ -61,12 +60,7 @@
4123 """Abort the current transaction."""
4124
4125 store = Attribute(
4126- """The underlying Storm store on which you can do queries.""")
4127-
4128-
4129-
4130
4131-class ITemporaryDatabase(Interface):
4132- """Marker interface for test suite adaptation."""
4133+ """The underlying database object on which you can do queries.""")
4134
4135
4136
4137
4138
4139=== modified file 'src/mailman/interfaces/messages.py'
4140--- src/mailman/interfaces/messages.py 2014-04-28 15:23:35 +0000
4141+++ src/mailman/interfaces/messages.py 2014-10-11 02:14:38 +0000
4142@@ -83,7 +83,7 @@
4143
4144 def get_message_by_hash(message_id_hash):
4145 """Return the message with the matching X-Message-ID-Hash.
4146-
4147+
4148 :param message_id_hash: The X-Message-ID-Hash header contents to
4149 search for.
4150 :returns: The message, or None if no matching message was found.
4151
4152=== modified file 'src/mailman/model/address.py'
4153--- src/mailman/model/address.py 2014-04-15 03:00:41 +0000
4154+++ src/mailman/model/address.py 2014-10-11 02:14:38 +0000
4155@@ -26,7 +26,9 @@
4156
4157
4158 from email.utils import formataddr
4159-from storm.locals import DateTime, Int, Reference, Unicode
4160+from sqlalchemy import (
4161+ Column, DateTime, ForeignKey, Integer, Unicode)
4162+from sqlalchemy.orm import relationship, backref
4163 from zope.component import getUtility
4164 from zope.event import notify
4165 from zope.interface import implementer
4166@@ -42,17 +44,20 @@
4167 class Address(Model):
4168 """See `IAddress`."""
4169
4170- id = Int(primary=True)
4171- email = Unicode()
4172- _original = Unicode()
4173- display_name = Unicode()
4174- _verified_on = DateTime(name='verified_on')
4175- registered_on = DateTime()
4176-
4177- user_id = Int()
4178- user = Reference(user_id, 'User.id')
4179- preferences_id = Int()
4180- preferences = Reference(preferences_id, 'Preferences.id')
4181+ __tablename__ = 'address'
4182+
4183+ id = Column(Integer, primary_key=True)
4184+ email = Column(Unicode)
4185+ _original = Column(Unicode)
4186+ display_name = Column(Unicode)
4187+ _verified_on = Column('verified_on', DateTime)
4188+ registered_on = Column(DateTime)
4189+
4190+ user_id = Column(Integer, ForeignKey('user.id'), index=True)
4191+
4192+ preferences_id = Column(Integer, ForeignKey('preferences.id'), index=True)
4193+ preferences = relationship(
4194+ 'Preferences', backref=backref('address', uselist=False))
4195
4196 def __init__(self, email, display_name):
4197 super(Address, self).__init__()
4198
4199=== modified file 'src/mailman/model/autorespond.py'
4200--- src/mailman/model/autorespond.py 2014-01-01 14:59:42 +0000
4201+++ src/mailman/model/autorespond.py 2014-10-11 02:14:38 +0000
4202@@ -26,7 +26,9 @@
4203 ]
4204
4205
4206-from storm.locals import And, Date, Desc, Int, Reference
4207+from sqlalchemy import Column, Date, ForeignKey, Integer
4208+from sqlalchemy import desc
4209+from sqlalchemy.orm import relationship
4210 from zope.interface import implementer
4211
4212 from mailman.database.model import Model
4213@@ -42,16 +44,18 @@
4214 class AutoResponseRecord(Model):
4215 """See `IAutoResponseRecord`."""
4216
4217- id = Int(primary=True)
4218-
4219- address_id = Int()
4220- address = Reference(address_id, 'Address.id')
4221-
4222- mailing_list_id = Int()
4223- mailing_list = Reference(mailing_list_id, 'MailingList.id')
4224-
4225- response_type = Enum(Response)
4226- date_sent = Date()
4227+ __tablename__ = 'autoresponserecord'
4228+
4229+ id = Column(Integer, primary_key=True)
4230+
4231+ address_id = Column(Integer, ForeignKey('address.id'), index=True)
4232+ address = relationship('Address')
4233+
4234+ mailing_list_id = Column(Integer, ForeignKey('mailinglist.id'), index=True)
4235+ mailing_list = relationship('MailingList')
4236+
4237+ response_type = Column(Enum(Response))
4238+ date_sent = Column(Date)
4239
4240 def __init__(self, mailing_list, address, response_type):
4241 self.mailing_list = mailing_list
4242@@ -71,12 +75,11 @@
4243 @dbconnection
4244 def todays_count(self, store, address, response_type):
4245 """See `IAutoResponseSet`."""
4246- return store.find(
4247- AutoResponseRecord,
4248- And(AutoResponseRecord.address == address,
4249- AutoResponseRecord.mailing_list == self._mailing_list,
4250- AutoResponseRecord.response_type == response_type,
4251- AutoResponseRecord.date_sent == today())).count()
4252+ return store.query(AutoResponseRecord).filter_by(
4253+ address=address,
4254+ mailing_list=self._mailing_list,
4255+ response_type=response_type,
4256+ date_sent=today()).count()
4257
4258 @dbconnection
4259 def response_sent(self, store, address, response_type):
4260@@ -88,10 +91,9 @@
4261 @dbconnection
4262 def last_response(self, store, address, response_type):
4263 """See `IAutoResponseSet`."""
4264- results = store.find(
4265- AutoResponseRecord,
4266- And(AutoResponseRecord.address == address,
4267- AutoResponseRecord.mailing_list == self._mailing_list,
4268- AutoResponseRecord.response_type == response_type)
4269- ).order_by(Desc(AutoResponseRecord.date_sent))
4270+ results = store.query(AutoResponseRecord).filter_by(
4271+ address=address,
4272+ mailing_list=self._mailing_list,
4273+ response_type=response_type
4274+ ).order_by(desc(AutoResponseRecord.date_sent))
4275 return (None if results.count() == 0 else results.first())
4276
4277=== modified file 'src/mailman/model/bans.py'
4278--- src/mailman/model/bans.py 2014-01-01 14:59:42 +0000
4279+++ src/mailman/model/bans.py 2014-10-11 02:14:38 +0000
4280@@ -27,7 +27,7 @@
4281
4282 import re
4283
4284-from storm.locals import Int, Unicode
4285+from sqlalchemy import Column, Integer, Unicode
4286 from zope.interface import implementer
4287
4288 from mailman.database.model import Model
4289@@ -40,9 +40,11 @@
4290 class Ban(Model):
4291 """See `IBan`."""
4292
4293- id = Int(primary=True)
4294- email = Unicode()
4295- list_id = Unicode()
4296+ __tablename__ = 'ban'
4297+
4298+ id = Column(Integer, primary_key=True)
4299+ email = Column(Unicode)
4300+ list_id = Column(Unicode)
4301
4302 def __init__(self, email, list_id):
4303 super(Ban, self).__init__()
4304@@ -62,7 +64,7 @@
4305 @dbconnection
4306 def ban(self, store, email):
4307 """See `IBanManager`."""
4308- bans = store.find(Ban, email=email, list_id=self._list_id)
4309+ bans = store.query(Ban).filter_by(email=email, list_id=self._list_id)
4310 if bans.count() == 0:
4311 ban = Ban(email, self._list_id)
4312 store.add(ban)
4313@@ -70,9 +72,10 @@
4314 @dbconnection
4315 def unban(self, store, email):
4316 """See `IBanManager`."""
4317- ban = store.find(Ban, email=email, list_id=self._list_id).one()
4318+ ban = store.query(Ban).filter_by(
4319+ email=email, list_id=self._list_id).first()
4320 if ban is not None:
4321- store.remove(ban)
4322+ store.delete(ban)
4323
4324 @dbconnection
4325 def is_banned(self, store, email):
4326@@ -81,32 +84,32 @@
4327 if list_id is None:
4328 # The client is asking for global bans. Look up bans on the
4329 # specific email address first.
4330- bans = store.find(Ban, email=email, list_id=None)
4331+ bans = store.query(Ban).filter_by(email=email, list_id=None)
4332 if bans.count() > 0:
4333 return True
4334 # And now look for global pattern bans.
4335- bans = store.find(Ban, list_id=None)
4336+ bans = store.query(Ban).filter_by(list_id=None)
4337 for ban in bans:
4338 if (ban.email.startswith('^') and
4339 re.match(ban.email, email, re.IGNORECASE) is not None):
4340 return True
4341 else:
4342 # This is a list-specific ban.
4343- bans = store.find(Ban, email=email, list_id=list_id)
4344+ bans = store.query(Ban).filter_by(email=email, list_id=list_id)
4345 if bans.count() > 0:
4346 return True
4347 # Try global bans next.
4348- bans = store.find(Ban, email=email, list_id=None)
4349+ bans = store.query(Ban).filter_by(email=email, list_id=None)
4350 if bans.count() > 0:
4351 return True
4352 # Now try specific mailing list bans, but with a pattern.
4353- bans = store.find(Ban, list_id=list_id)
4354+ bans = store.query(Ban).filter_by(list_id=list_id)
4355 for ban in bans:
4356 if (ban.email.startswith('^') and
4357 re.match(ban.email, email, re.IGNORECASE) is not None):
4358 return True
4359 # And now try global pattern bans.
4360- bans = store.find(Ban, list_id=None)
4361+ bans = store.query(Ban).filter_by(list_id=None)
4362 for ban in bans:
4363 if (ban.email.startswith('^') and
4364 re.match(ban.email, email, re.IGNORECASE) is not None):
4365
4366=== modified file 'src/mailman/model/bounce.py'
4367--- src/mailman/model/bounce.py 2014-01-01 14:59:42 +0000
4368+++ src/mailman/model/bounce.py 2014-10-11 02:14:38 +0000
4369@@ -26,7 +26,8 @@
4370 ]
4371
4372
4373-from storm.locals import Bool, Int, DateTime, Unicode
4374+
4375+from sqlalchemy import Boolean, Column, DateTime, Integer, Unicode
4376 from zope.interface import implementer
4377
4378 from mailman.database.model import Model
4379@@ -42,13 +43,15 @@
4380 class BounceEvent(Model):
4381 """See `IBounceEvent`."""
4382
4383- id = Int(primary=True)
4384- list_id = Unicode()
4385- email = Unicode()
4386- timestamp = DateTime()
4387- message_id = Unicode()
4388- context = Enum(BounceContext)
4389- processed = Bool()
4390+ __tablename__ = 'bounceevent'
4391+
4392+ id = Column(Integer, primary_key=True)
4393+ list_id = Column(Unicode)
4394+ email = Column(Unicode)
4395+ timestamp = Column(DateTime)
4396+ message_id = Column(Unicode)
4397+ context = Column(Enum(BounceContext))
4398+ processed = Column(Boolean)
4399
4400 def __init__(self, list_id, email, msg, context=None):
4401 self.list_id = list_id
4402@@ -75,12 +78,12 @@
4403 @dbconnection
4404 def events(self, store):
4405 """See `IBounceProcessor`."""
4406- for event in store.find(BounceEvent):
4407+ for event in store.query(BounceEvent).all():
4408 yield event
4409
4410 @property
4411 @dbconnection
4412 def unprocessed(self, store):
4413 """See `IBounceProcessor`."""
4414- for event in store.find(BounceEvent, BounceEvent.processed == False):
4415+ for event in store.query(BounceEvent).filter_by(processed=False):
4416 yield event
4417
4418=== modified file 'src/mailman/model/digests.py'
4419--- src/mailman/model/digests.py 2014-01-01 14:59:42 +0000
4420+++ src/mailman/model/digests.py 2014-10-11 02:14:38 +0000
4421@@ -25,7 +25,8 @@
4422 ]
4423
4424
4425-from storm.locals import Int, Reference
4426+from sqlalchemy import Column, Integer, ForeignKey
4427+from sqlalchemy.orm import relationship
4428 from zope.interface import implementer
4429
4430 from mailman.database.model import Model
4431@@ -39,15 +40,17 @@
4432 class OneLastDigest(Model):
4433 """See `IOneLastDigest`."""
4434
4435- id = Int(primary=True)
4436-
4437- mailing_list_id = Int()
4438- mailing_list = Reference(mailing_list_id, 'MailingList.id')
4439-
4440- address_id = Int()
4441- address = Reference(address_id, 'Address.id')
4442-
4443- delivery_mode = Enum(DeliveryMode)
4444+ __tablename__ = 'onelastdigest'
4445+
4446+ id = Column(Integer, primary_key=True)
4447+
4448+ mailing_list_id = Column(Integer, ForeignKey('mailinglist.id'))
4449+ mailing_list = relationship('MailingList')
4450+
4451+ address_id = Column(Integer, ForeignKey('address.id'))
4452+ address = relationship('Address')
4453+
4454+ delivery_mode = Column(Enum(DeliveryMode))
4455
4456 def __init__(self, mailing_list, address, delivery_mode):
4457 self.mailing_list = mailing_list
4458
4459=== modified file 'src/mailman/model/docs/messagestore.rst'
4460--- src/mailman/model/docs/messagestore.rst 2014-04-28 15:23:35 +0000
4461+++ src/mailman/model/docs/messagestore.rst 2014-10-11 02:14:38 +0000
4462@@ -28,8 +28,9 @@
4463 However, if the message has a ``Message-ID`` header, it can be stored.
4464
4465 >>> msg['Message-ID'] = '<87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>'
4466- >>> message_store.add(msg)
4467- 'AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35'
4468+ >>> x_message_id_hash = message_store.add(msg)
4469+ >>> print(x_message_id_hash)
4470+ AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35
4471 >>> print(msg.as_string())
4472 Subject: An important message
4473 Message-ID: <87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>
4474
4475=== modified file 'src/mailman/model/domain.py'
4476--- src/mailman/model/domain.py 2014-01-01 14:59:42 +0000
4477+++ src/mailman/model/domain.py 2014-10-11 02:14:38 +0000
4478@@ -26,8 +26,8 @@
4479 ]
4480
4481
4482+from sqlalchemy import Column, Integer, Unicode
4483 from urlparse import urljoin, urlparse
4484-from storm.locals import Int, Unicode
4485 from zope.event import notify
4486 from zope.interface import implementer
4487
4488@@ -44,12 +44,14 @@
4489 class Domain(Model):
4490 """Domains."""
4491
4492- id = Int(primary=True)
4493-
4494- mail_host = Unicode()
4495- base_url = Unicode()
4496- description = Unicode()
4497- contact_address = Unicode()
4498+ __tablename__ = 'domain'
4499+
4500+ id = Column(Integer, primary_key=True)
4501+
4502+ mail_host = Column(Unicode)
4503+ base_url = Column(Unicode)
4504+ description = Column(Unicode)
4505+ contact_address = Column(Unicode)
4506
4507 def __init__(self, mail_host,
4508 description=None,
4509@@ -92,8 +94,7 @@
4510 @dbconnection
4511 def mailing_lists(self, store):
4512 """See `IDomain`."""
4513- mailing_lists = store.find(
4514- MailingList,
4515+ mailing_lists = store.query(MailingList).filter(
4516 MailingList.mail_host == self.mail_host)
4517 for mlist in mailing_lists:
4518 yield mlist
4519@@ -140,14 +141,14 @@
4520 def remove(self, store, mail_host):
4521 domain = self[mail_host]
4522 notify(DomainDeletingEvent(domain))
4523- store.remove(domain)
4524+ store.delete(domain)
4525 notify(DomainDeletedEvent(mail_host))
4526 return domain
4527
4528 @dbconnection
4529 def get(self, store, mail_host, default=None):
4530 """See `IDomainManager`."""
4531- domains = store.find(Domain, mail_host=mail_host)
4532+ domains = store.query(Domain).filter_by(mail_host=mail_host)
4533 if domains.count() < 1:
4534 return default
4535 assert domains.count() == 1, (
4536@@ -164,15 +165,15 @@
4537
4538 @dbconnection
4539 def __len__(self, store):
4540- return store.find(Domain).count()
4541+ return store.query(Domain).count()
4542
4543 @dbconnection
4544 def __iter__(self, store):
4545 """See `IDomainManager`."""
4546- for domain in store.find(Domain):
4547+ for domain in store.query(Domain).all():
4548 yield domain
4549
4550 @dbconnection
4551 def __contains__(self, store, mail_host):
4552 """See `IDomainManager`."""
4553- return store.find(Domain, mail_host=mail_host).count() > 0
4554+ return store.query(Domain).filter_by(mail_host=mail_host).count() > 0
4555
4556=== modified file 'src/mailman/model/language.py'
4557--- src/mailman/model/language.py 2014-01-01 14:59:42 +0000
4558+++ src/mailman/model/language.py 2014-10-11 02:14:38 +0000
4559@@ -25,11 +25,11 @@
4560 ]
4561
4562
4563-from storm.locals import Int, Unicode
4564+from sqlalchemy import Column, Integer, Unicode
4565 from zope.interface import implementer
4566
4567-from mailman.database import Model
4568-from mailman.interfaces import ILanguage
4569+from mailman.database.model import Model
4570+from mailman.interfaces.languages import ILanguage
4571
4572
4573
4574
4575@@ -37,5 +37,7 @@
4576 class Language(Model):
4577 """See `ILanguage`."""
4578
4579- id = Int(primary=True)
4580- code = Unicode()
4581+ __tablename__ = 'language'
4582+
4583+ id = Column(Integer, primary_key=True)
4584+ code = Column(Unicode)
4585
4586=== modified file 'src/mailman/model/listmanager.py'
4587--- src/mailman/model/listmanager.py 2014-04-14 16:14:13 +0000
4588+++ src/mailman/model/listmanager.py 2014-10-11 02:14:38 +0000
4589@@ -52,9 +52,7 @@
4590 raise InvalidEmailAddressError(fqdn_listname)
4591 list_id = '{0}.{1}'.format(listname, hostname)
4592 notify(ListCreatingEvent(fqdn_listname))
4593- mlist = store.find(
4594- MailingList,
4595- MailingList._list_id == list_id).one()
4596+ mlist = store.query(MailingList).filter_by(_list_id=list_id).first()
4597 if mlist:
4598 raise ListAlreadyExistsError(fqdn_listname)
4599 mlist = MailingList(fqdn_listname)
4600@@ -68,40 +66,40 @@
4601 """See `IListManager`."""
4602 listname, at, hostname = fqdn_listname.partition('@')
4603 list_id = '{0}.{1}'.format(listname, hostname)
4604- return store.find(MailingList, MailingList._list_id == list_id).one()
4605+ return store.query(MailingList).filter_by(_list_id=list_id).first()
4606
4607 @dbconnection
4608 def get_by_list_id(self, store, list_id):
4609 """See `IListManager`."""
4610- return store.find(MailingList, MailingList._list_id == list_id).one()
4611+ return store.query(MailingList).filter_by(_list_id=list_id).first()
4612
4613 @dbconnection
4614 def delete(self, store, mlist):
4615 """See `IListManager`."""
4616 fqdn_listname = mlist.fqdn_listname
4617 notify(ListDeletingEvent(mlist))
4618- store.find(ContentFilter, ContentFilter.mailing_list == mlist).remove()
4619- store.remove(mlist)
4620+ store.query(ContentFilter).filter_by(mailing_list=mlist).delete()
4621+ store.delete(mlist)
4622 notify(ListDeletedEvent(fqdn_listname))
4623
4624 @property
4625 @dbconnection
4626 def mailing_lists(self, store):
4627 """See `IListManager`."""
4628- for mlist in store.find(MailingList):
4629+ for mlist in store.query(MailingList).all():
4630 yield mlist
4631
4632 @dbconnection
4633 def __iter__(self, store):
4634 """See `IListManager`."""
4635- for mlist in store.find(MailingList):
4636+ for mlist in store.query(MailingList).all():
4637 yield mlist
4638
4639 @property
4640 @dbconnection
4641 def names(self, store):
4642 """See `IListManager`."""
4643- result_set = store.find(MailingList)
4644+ result_set = store.query(MailingList)
4645 for mail_host, list_name in result_set.values(MailingList.mail_host,
4646 MailingList.list_name):
4647 yield '{0}@{1}'.format(list_name, mail_host)
4648@@ -110,15 +108,16 @@
4649 @dbconnection
4650 def list_ids(self, store):
4651 """See `IListManager`."""
4652- result_set = store.find(MailingList)
4653+ result_set = store.query(MailingList)
4654 for list_id in result_set.values(MailingList._list_id):
4655- yield list_id
4656+ assert isinstance(list_id, tuple) and len(list_id) == 1
4657+ yield list_id[0]
4658
4659 @property
4660 @dbconnection
4661 def name_components(self, store):
4662 """See `IListManager`."""
4663- result_set = store.find(MailingList)
4664+ result_set = store.query(MailingList)
4665 for mail_host, list_name in result_set.values(MailingList.mail_host,
4666 MailingList.list_name):
4667 yield list_name, mail_host
4668
4669=== modified file 'src/mailman/model/mailinglist.py'
4670--- src/mailman/model/mailinglist.py 2014-04-14 16:14:13 +0000
4671+++ src/mailman/model/mailinglist.py 2014-10-11 02:14:38 +0000
4672@@ -27,9 +27,11 @@
4673
4674 import os
4675
4676-from storm.locals import (
4677- And, Bool, DateTime, Float, Int, Pickle, RawStr, Reference, Store,
4678- TimeDelta, Unicode)
4679+from sqlalchemy import (
4680+ Boolean, Column, DateTime, Float, ForeignKey, Integer, Interval,
4681+ LargeBinary, PickleType, Unicode)
4682+from sqlalchemy.event import listen
4683+from sqlalchemy.orm import relationship
4684 from urlparse import urljoin
4685 from zope.component import getUtility
4686 from zope.event import notify
4687@@ -37,6 +39,7 @@
4688
4689 from mailman.config import config
4690 from mailman.database.model import Model
4691+from mailman.database.transaction import dbconnection
4692 from mailman.database.types import Enum
4693 from mailman.interfaces.action import Action, FilterAction
4694 from mailman.interfaces.address import IAddress
4695@@ -73,121 +76,121 @@
4696 class MailingList(Model):
4697 """See `IMailingList`."""
4698
4699- id = Int(primary=True)
4700+ __tablename__ = 'mailinglist'
4701+
4702+ id = Column(Integer, primary_key=True)
4703
4704 # XXX denotes attributes that should be part of the public interface but
4705 # are currently missing.
4706
4707 # List identity
4708- list_name = Unicode()
4709- mail_host = Unicode()
4710- _list_id = Unicode(name='list_id')
4711- allow_list_posts = Bool()
4712- include_rfc2369_headers = Bool()
4713- advertised = Bool()
4714- anonymous_list = Bool()
4715+ list_name = Column(Unicode)
4716+ mail_host = Column(Unicode)
4717+ _list_id = Column('list_id', Unicode)
4718+ allow_list_posts = Column(Boolean)
4719+ include_rfc2369_headers = Column(Boolean)
4720+ advertised = Column(Boolean)
4721+ anonymous_list = Column(Boolean)
4722 # Attributes not directly modifiable via the web u/i
4723- created_at = DateTime()
4724- # Attributes which are directly modifiable via the web u/i. The more
4725- # complicated attributes are currently stored as pickles, though that
4726- # will change as the schema and implementation is developed.
4727- next_request_id = Int()
4728- next_digest_number = Int()
4729- digest_last_sent_at = DateTime()
4730- volume = Int()
4731- last_post_at = DateTime()
4732- # Implicit destination.
4733- acceptable_aliases_id = Int()
4734- acceptable_alias = Reference(acceptable_aliases_id, 'AcceptableAlias.id')
4735- # Attributes which are directly modifiable via the web u/i. The more
4736- # complicated attributes are currently stored as pickles, though that
4737- # will change as the schema and implementation is developed.
4738- accept_these_nonmembers = Pickle() # XXX
4739- admin_immed_notify = Bool()
4740- admin_notify_mchanges = Bool()
4741- administrivia = Bool()
4742- archive_policy = Enum(ArchivePolicy)
4743+ created_at = Column(DateTime)
4744+ # Attributes which are directly modifiable via the web u/i. The more
4745+ # complicated attributes are currently stored as pickles, though that
4746+ # will change as the schema and implementation is developed.
4747+ next_request_id = Column(Integer)
4748+ next_digest_number = Column(Integer)
4749+ digest_last_sent_at = Column(DateTime)
4750+ volume = Column(Integer)
4751+ last_post_at = Column(DateTime)
4752+ # Attributes which are directly modifiable via the web u/i. The more
4753+ # complicated attributes are currently stored as pickles, though that
4754+ # will change as the schema and implementation is developed.
4755+ accept_these_nonmembers = Column(PickleType) # XXX
4756+ admin_immed_notify = Column(Boolean)
4757+ admin_notify_mchanges = Column(Boolean)
4758+ administrivia = Column(Boolean)
4759+ archive_policy = Column(Enum(ArchivePolicy))
4760 # Automatic responses.
4761- autoresponse_grace_period = TimeDelta()
4762- autorespond_owner = Enum(ResponseAction)
4763- autoresponse_owner_text = Unicode()
4764- autorespond_postings = Enum(ResponseAction)
4765- autoresponse_postings_text = Unicode()
4766- autorespond_requests = Enum(ResponseAction)
4767- autoresponse_request_text = Unicode()
4768+ autoresponse_grace_period = Column(Interval)
4769+ autorespond_owner = Column(Enum(ResponseAction))
4770+ autoresponse_owner_text = Column(Unicode)
4771+ autorespond_postings = Column(Enum(ResponseAction))
4772+ autoresponse_postings_text = Column(Unicode)
4773+ autorespond_requests = Column(Enum(ResponseAction))
4774+ autoresponse_request_text = Column(Unicode)
4775 # Content filters.
4776- filter_action = Enum(FilterAction)
4777- filter_content = Bool()
4778- collapse_alternatives = Bool()
4779- convert_html_to_plaintext = Bool()
4780+ filter_action = Column(Enum(FilterAction))
4781+ filter_content = Column(Boolean)
4782+ collapse_alternatives = Column(Boolean)
4783+ convert_html_to_plaintext = Column(Boolean)
4784 # Bounces.
4785- bounce_info_stale_after = TimeDelta() # XXX
4786- bounce_matching_headers = Unicode() # XXX
4787- bounce_notify_owner_on_disable = Bool() # XXX
4788- bounce_notify_owner_on_removal = Bool() # XXX
4789- bounce_score_threshold = Int() # XXX
4790- bounce_you_are_disabled_warnings = Int() # XXX
4791- bounce_you_are_disabled_warnings_interval = TimeDelta() # XXX
4792- forward_unrecognized_bounces_to = Enum(UnrecognizedBounceDisposition)
4793- process_bounces = Bool()
4794+ bounce_info_stale_after = Column(Interval) # XXX
4795+ bounce_matching_headers = Column(Unicode) # XXX
4796+ bounce_notify_owner_on_disable = Column(Boolean) # XXX
4797+ bounce_notify_owner_on_removal = Column(Boolean) # XXX
4798+ bounce_score_threshold = Column(Integer) # XXX
4799+ bounce_you_are_disabled_warnings = Column(Integer) # XXX
4800+ bounce_you_are_disabled_warnings_interval = Column(Interval) # XXX
4801+ forward_unrecognized_bounces_to = Column(
4802+ Enum(UnrecognizedBounceDisposition))
4803+ process_bounces = Column(Boolean)
4804 # Miscellaneous
4805- default_member_action = Enum(Action)
4806- default_nonmember_action = Enum(Action)
4807- description = Unicode()
4808- digest_footer_uri = Unicode()
4809- digest_header_uri = Unicode()
4810- digest_is_default = Bool()
4811- digest_send_periodic = Bool()
4812- digest_size_threshold = Float()
4813- digest_volume_frequency = Enum(DigestFrequency)
4814- digestable = Bool()
4815- discard_these_nonmembers = Pickle()
4816- emergency = Bool()
4817- encode_ascii_prefixes = Bool()
4818- first_strip_reply_to = Bool()
4819- footer_uri = Unicode()
4820- forward_auto_discards = Bool()
4821- gateway_to_mail = Bool()
4822- gateway_to_news = Bool()
4823- goodbye_message_uri = Unicode()
4824- header_matches = Pickle()
4825- header_uri = Unicode()
4826- hold_these_nonmembers = Pickle()
4827- info = Unicode()
4828- linked_newsgroup = Unicode()
4829- max_days_to_hold = Int()
4830- max_message_size = Int()
4831- max_num_recipients = Int()
4832- member_moderation_notice = Unicode()
4833- mime_is_default_digest = Bool()
4834+ default_member_action = Column(Enum(Action))
4835+ default_nonmember_action = Column(Enum(Action))
4836+ description = Column(Unicode)
4837+ digest_footer_uri = Column(Unicode)
4838+ digest_header_uri = Column(Unicode)
4839+ digest_is_default = Column(Boolean)
4840+ digest_send_periodic = Column(Boolean)
4841+ digest_size_threshold = Column(Float)
4842+ digest_volume_frequency = Column(Enum(DigestFrequency))
4843+ digestable = Column(Boolean)
4844+ discard_these_nonmembers = Column(PickleType)
4845+ emergency = Column(Boolean)
4846+ encode_ascii_prefixes = Column(Boolean)
4847+ first_strip_reply_to = Column(Boolean)
4848+ footer_uri = Column(Unicode)
4849+ forward_auto_discards = Column(Boolean)
4850+ gateway_to_mail = Column(Boolean)
4851+ gateway_to_news = Column(Boolean)
4852+ goodbye_message_uri = Column(Unicode)
4853+ header_matches = Column(PickleType)
4854+ header_uri = Column(Unicode)
4855+ hold_these_nonmembers = Column(PickleType)
4856+ info = Column(Unicode)
4857+ linked_newsgroup = Column(Unicode)
4858+ max_days_to_hold = Column(Integer)
4859+ max_message_size = Column(Integer)
4860+ max_num_recipients = Column(Integer)
4861+ member_moderation_notice = Column(Unicode)
4862+ mime_is_default_digest = Column(Boolean)
4863 # FIXME: There should be no moderator_password
4864- moderator_password = RawStr()
4865- newsgroup_moderation = Enum(NewsgroupModeration)
4866- nntp_prefix_subject_too = Bool()
4867- nondigestable = Bool()
4868- nonmember_rejection_notice = Unicode()
4869- obscure_addresses = Bool()
4870- owner_chain = Unicode()
4871- owner_pipeline = Unicode()
4872- personalize = Enum(Personalization)
4873- post_id = Int()
4874- posting_chain = Unicode()
4875- posting_pipeline = Unicode()
4876- _preferred_language = Unicode(name='preferred_language')
4877- display_name = Unicode()
4878- reject_these_nonmembers = Pickle()
4879- reply_goes_to_list = Enum(ReplyToMunging)
4880- reply_to_address = Unicode()
4881- require_explicit_destination = Bool()
4882- respond_to_post_requests = Bool()
4883- scrub_nondigest = Bool()
4884- send_goodbye_message = Bool()
4885- send_welcome_message = Bool()
4886- subject_prefix = Unicode()
4887- topics = Pickle()
4888- topics_bodylines_limit = Int()
4889- topics_enabled = Bool()
4890- welcome_message_uri = Unicode()
4891+ moderator_password = Column(LargeBinary) # TODO : was RawStr()
4892+ newsgroup_moderation = Column(Enum(NewsgroupModeration))
4893+ nntp_prefix_subject_too = Column(Boolean)
4894+ nondigestable = Column(Boolean)
4895+ nonmember_rejection_notice = Column(Unicode)
4896+ obscure_addresses = Column(Boolean)
4897+ owner_chain = Column(Unicode)
4898+ owner_pipeline = Column(Unicode)
4899+ personalize = Column(Enum(Personalization))
4900+ post_id = Column(Integer)
4901+ posting_chain = Column(Unicode)
4902+ posting_pipeline = Column(Unicode)
4903+ _preferred_language = Column('preferred_language', Unicode)
4904+ display_name = Column(Unicode)
4905+ reject_these_nonmembers = Column(PickleType)
4906+ reply_goes_to_list = Column(Enum(ReplyToMunging))
4907+ reply_to_address = Column(Unicode)
4908+ require_explicit_destination = Column(Boolean)
4909+ respond_to_post_requests = Column(Boolean)
4910+ scrub_nondigest = Column(Boolean)
4911+ send_goodbye_message = Column(Boolean)
4912+ send_welcome_message = Column(Boolean)
4913+ subject_prefix = Column(Unicode)
4914+ topics = Column(PickleType)
4915+ topics_bodylines_limit = Column(Integer)
4916+ topics_enabled = Column(Boolean)
4917+ welcome_message_uri = Column(Unicode)
4918
4919 def __init__(self, fqdn_listname):
4920 super(MailingList, self).__init__()
4921@@ -198,14 +201,15 @@
4922 self._list_id = '{0}.{1}'.format(listname, hostname)
4923 # For the pending database
4924 self.next_request_id = 1
4925- # We need to set up the rosters. Normally, this method will get
4926- # called when the MailingList object is loaded from the database, but
4927- # that's not the case when the constructor is called. So, set up the
4928- # rosters explicitly.
4929- self.__storm_loaded__()
4930+ # We need to set up the rosters. Normally, this method will get called
4931+ # when the MailingList object is loaded from the database, but when the
4932+ # constructor is called, SQLAlchemy's `load` event isn't triggered.
4933+ # Thus we need to set up the rosters explicitly.
4934+ self._post_load()
4935 makedirs(self.data_path)
4936
4937- def __storm_loaded__(self):
4938+ def _post_load(self, *args):
4939+ # This hooks up to SQLAlchemy's `load` event.
4940 self.owners = roster.OwnerRoster(self)
4941 self.moderators = roster.ModeratorRoster(self)
4942 self.administrators = roster.AdministratorRoster(self)
4943@@ -215,6 +219,13 @@
4944 self.subscribers = roster.Subscribers(self)
4945 self.nonmembers = roster.NonmemberRoster(self)
4946
4947+ @classmethod
4948+ def __declare_last__(cls):
4949+ # SQLAlchemy special directive hook called after mappings are assumed
4950+ # to be complete. Use this to connect the roster instance creation
4951+ # method with the SA `load` event.
4952+ listen(cls, 'load', cls._post_load)
4953+
4954 def __repr__(self):
4955 return '<mailing list "{0}" at {1:#x}>'.format(
4956 self.fqdn_listname, id(self))
4957@@ -323,42 +334,42 @@
4958 except AttributeError:
4959 self._preferred_language = language
4960
4961- def send_one_last_digest_to(self, address, delivery_mode):
4962+ @dbconnection
4963+ def send_one_last_digest_to(self, store, address, delivery_mode):
4964 """See `IMailingList`."""
4965 digest = OneLastDigest(self, address, delivery_mode)
4966- Store.of(self).add(digest)
4967+ store.add(digest)
4968
4969 @property
4970- def last_digest_recipients(self):
4971+ @dbconnection
4972+ def last_digest_recipients(self, store):
4973 """See `IMailingList`."""
4974- results = Store.of(self).find(
4975- OneLastDigest,
4976+ results = store.query(OneLastDigest).filter(
4977 OneLastDigest.mailing_list == self)
4978 recipients = [(digest.address, digest.delivery_mode)
4979 for digest in results]
4980- results.remove()
4981+ results.delete()
4982 return recipients
4983
4984 @property
4985- def filter_types(self):
4986+ @dbconnection
4987+ def filter_types(self, store):
4988 """See `IMailingList`."""
4989- results = Store.of(self).find(
4990- ContentFilter,
4991- And(ContentFilter.mailing_list == self,
4992- ContentFilter.filter_type == FilterType.filter_mime))
4993+ results = store.query(ContentFilter).filter(
4994+ ContentFilter.mailing_list == self,
4995+ ContentFilter.filter_type == FilterType.filter_mime)
4996 for content_filter in results:
4997 yield content_filter.filter_pattern
4998
4999 @filter_types.setter
5000- def filter_types(self, sequence):
The diff has been truncated for viewing.