Merge lp:~raj-abhilash1/mailman/sqlalchemy into lp:mailman
- sqlalchemy
- Merge into 3.0
Status: | Merged |
---|---|
Merged at revision: | 7253 |
Proposed branch: | lp:~raj-abhilash1/mailman/sqlalchemy |
Merge into: | lp:mailman |
Diff against target: |
6388 lines (+1161/-3567) 66 files modified
MANIFEST.in (+2/-0) setup.py (+2/-1) src/mailman/app/subscriptions.py (+7/-9) src/mailman/bin/tests/test_master.py (+1/-1) src/mailman/commands/docs/conf.rst (+3/-1) src/mailman/config/config.py (+5/-8) src/mailman/config/configure.zcml (+0/-20) src/mailman/config/schema.cfg (+13/-4) src/mailman/core/logging.py (+4/-0) src/mailman/database/alembic/__init__.py (+32/-0) src/mailman/database/alembic/env.py (+80/-0) src/mailman/database/alembic/script.py.mako (+22/-0) src/mailman/database/alembic/versions/51b7f92bd06c_initial.py (+34/-0) src/mailman/database/base.py (+14/-126) src/mailman/database/docs/migration.rst (+0/-207) src/mailman/database/factory.py (+72/-25) src/mailman/database/model.py (+30/-40) src/mailman/database/postgresql.py (+6/-59) src/mailman/database/schema/helpers.py (+0/-43) src/mailman/database/schema/mm_00000000000000_base.py (+0/-35) src/mailman/database/schema/mm_20120407000000.py (+0/-212) src/mailman/database/schema/mm_20121015000000.py (+0/-95) src/mailman/database/schema/mm_20130406000000.py (+0/-65) src/mailman/database/schema/postgres.sql (+0/-349) src/mailman/database/schema/sqlite.sql (+0/-327) src/mailman/database/schema/sqlite_20120407000000_01.sql (+0/-280) src/mailman/database/schema/sqlite_20121015000000_01.sql (+0/-230) src/mailman/database/schema/sqlite_20130406000000_01.sql (+0/-46) src/mailman/database/sqlite.py (+5/-41) src/mailman/database/tests/data/migration_postgres_1.sql (+0/-133) src/mailman/database/tests/data/migration_sqlite_1.sql (+0/-133) src/mailman/database/tests/test_factory.py (+162/-0) src/mailman/database/tests/test_migrations.py (+0/-506) src/mailman/database/types.py (+57/-30) src/mailman/interfaces/database.py (+1/-7) src/mailman/interfaces/messages.py (+1/-1) src/mailman/model/address.py (+17/-12) src/mailman/model/autorespond.py (+25/-23) src/mailman/model/bans.py (+16/-13) src/mailman/model/bounce.py (+13/-10) src/mailman/model/digests.py (+13/-10) src/mailman/model/docs/messagestore.rst (+3/-2) src/mailman/model/domain.py (+15/-14) src/mailman/model/language.py (+7/-5) src/mailman/model/listmanager.py (+12/-13) src/mailman/model/mailinglist.py (+230/-216) src/mailman/model/member.py (+19/-17) src/mailman/model/message.py (+7/-5) src/mailman/model/messagestore.py (+15/-14) src/mailman/model/mime.py (+11/-8) src/mailman/model/pending.py (+34/-27) src/mailman/model/preferences.py (+11/-9) src/mailman/model/requests.py (+26/-17) src/mailman/model/roster.py (+19/-26) src/mailman/model/tests/test_listmanager.py (+12/-4) src/mailman/model/tests/test_requests.py (+2/-2) src/mailman/model/uid.py (+9/-5) src/mailman/model/user.py (+32/-17) src/mailman/model/usermanager.py (+9/-9) src/mailman/model/version.py (+0/-44) src/mailman/rest/validator.py (+1/-1) src/mailman/styles/base.py (+2/-6) src/mailman/testing/layers.py (+10/-1) src/mailman/testing/testing.cfg (+1/-1) src/mailman/utilities/importer.py (+23/-1) src/mailman/utilities/modules.py (+14/-1) |
To merge this branch: | bzr merge lp:~raj-abhilash1/mailman/sqlalchemy |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Mailman Coders | Pending | ||
Review via email: mp+235329@code.launchpad.net |
Commit message
Description of the change
Replace storm with sqlalchemy.
- 7266. By Abhilash Raj
-
merge test fix from Barry
- 7267. By Abhilash Raj
-
merge updates from to sqlalchemy branch
- 7268. By Abhilash Raj
-
merge updated with some removal from barry
- 7269. By Abhilash Raj
-
added support for migrations via alembic
- 7270. By Abhilash Raj
-
added license block for the new file
- 7271. By Abhilash Raj
-
no need to stamp the testing db
- 7272. By Abhilash Raj
-
add new command `mailman migrate` to migrate the new schema on the old database
- 7273. By Abhilash Raj
-
add autogenerate switch that generates to create migration scripts automatically
- 7274. By Abhilash Raj
-
* fixed a bug where alemnic could not find its migrations directory
* add a new method in base database to stamp with latest alembic version - 7275. By Abhilash Raj
-
Add support for postgresql
* revert changes in message_id_has encoding by barry
* Change message_id_hash column to LargeBinary
(from previously mistaken one i.e.unicode)
* add missing import in database/types.py
* fix a bug in database/Model.py, transaction has no
method abort(), instead it is rollback() - 7276. By Abhilash Raj
-
Merge alembic setup from abompard
- 7277. By Abhilash Raj
-
Merge barry\'s branch with test fixes and clean code
- 7278. By Abhilash Raj
-
add central alembic config
- 7279. By Abhilash Raj
-
fix database reset error due to foreign key constraint between user and address tables
- 7280. By Abhilash Raj
-
merge branch from abompard
- 7281. By Abhilash Raj
-
* remove migrate command
* remove alembic.cfg, move contents to schema.cfg
* fix import errors in src/mailman/model/language. py
* add indexes
* change the previously wrong written tablename autoresponserecord
* change alembic_cfg to use schema.cfg instead of alembic.cfg - 7282. By Abhilash Raj
-
changes from abompard to fix the unit tests
Preview Diff
1 | === modified file 'MANIFEST.in' | |||
2 | --- MANIFEST.in 2011-06-11 18:16:04 +0000 | |||
3 | +++ MANIFEST.in 2014-10-11 02:14:38 +0000 | |||
4 | @@ -13,3 +13,5 @@ | |||
5 | 13 | prune parts | 13 | prune parts |
6 | 14 | include MANIFEST.in | 14 | include MANIFEST.in |
7 | 15 | include src/mailman/testing/config.pck | 15 | include src/mailman/testing/config.pck |
8 | 16 | include src/mailman/database/alembic/script.py.mako | ||
9 | 17 | include src/mailman/database/alembic/versions/*.py | ||
10 | 16 | 18 | ||
11 | === modified file 'setup.py' | |||
12 | --- setup.py 2014-04-15 16:06:01 +0000 | |||
13 | +++ setup.py 2014-10-11 02:14:38 +0000 | |||
14 | @@ -93,6 +93,7 @@ | |||
15 | 93 | 'console_scripts' : list(scripts), | 93 | 'console_scripts' : list(scripts), |
16 | 94 | }, | 94 | }, |
17 | 95 | install_requires = [ | 95 | install_requires = [ |
18 | 96 | 'alembic', | ||
19 | 96 | 'enum34', | 97 | 'enum34', |
20 | 97 | 'flufl.bounce', | 98 | 'flufl.bounce', |
21 | 98 | 'flufl.i18n', | 99 | 'flufl.i18n', |
22 | @@ -104,7 +105,7 @@ | |||
23 | 104 | 'nose2', | 105 | 'nose2', |
24 | 105 | 'passlib', | 106 | 'passlib', |
25 | 106 | 'restish', | 107 | 'restish', |
27 | 107 | 'storm', | 108 | 'sqlalchemy', |
28 | 108 | 'zope.component', | 109 | 'zope.component', |
29 | 109 | 'zope.configuration', | 110 | 'zope.configuration', |
30 | 110 | 'zope.event', | 111 | 'zope.event', |
31 | 111 | 112 | ||
32 | === modified file 'src/mailman/app/subscriptions.py' | |||
33 | --- src/mailman/app/subscriptions.py 2014-04-15 14:03:39 +0000 | |||
34 | +++ src/mailman/app/subscriptions.py 2014-10-11 02:14:38 +0000 | |||
35 | @@ -28,7 +28,7 @@ | |||
36 | 28 | 28 | ||
37 | 29 | from operator import attrgetter | 29 | from operator import attrgetter |
38 | 30 | from passlib.utils import generate_password as generate | 30 | from passlib.utils import generate_password as generate |
40 | 31 | from storm.expr import And, Or | 31 | from sqlalchemy import and_, or_ |
41 | 32 | from uuid import UUID | 32 | from uuid import UUID |
42 | 33 | from zope.component import getUtility | 33 | from zope.component import getUtility |
43 | 34 | from zope.interface import implementer | 34 | from zope.interface import implementer |
44 | @@ -88,9 +88,7 @@ | |||
45 | 88 | @dbconnection | 88 | @dbconnection |
46 | 89 | def get_member(self, store, member_id): | 89 | def get_member(self, store, member_id): |
47 | 90 | """See `ISubscriptionService`.""" | 90 | """See `ISubscriptionService`.""" |
51 | 91 | members = store.find( | 91 | members = store.query(Member).filter(Member._member_id == member_id) |
49 | 92 | Member, | ||
50 | 93 | Member._member_id == member_id) | ||
52 | 94 | if members.count() == 0: | 92 | if members.count() == 0: |
53 | 95 | return None | 93 | return None |
54 | 96 | else: | 94 | else: |
55 | @@ -117,8 +115,8 @@ | |||
56 | 117 | # This probably could be made more efficient. | 115 | # This probably could be made more efficient. |
57 | 118 | if address is None or user is None: | 116 | if address is None or user is None: |
58 | 119 | return [] | 117 | return [] |
61 | 120 | query.append(Or(Member.address_id == address.id, | 118 | query.append(or_(Member.address_id == address.id, |
62 | 121 | Member.user_id == user.id)) | 119 | Member.user_id == user.id)) |
63 | 122 | else: | 120 | else: |
64 | 123 | # subscriber is a user id. | 121 | # subscriber is a user id. |
65 | 124 | user = user_manager.get_user_by_id(subscriber) | 122 | user = user_manager.get_user_by_id(subscriber) |
66 | @@ -126,15 +124,15 @@ | |||
67 | 126 | if address.id is not None) | 124 | if address.id is not None) |
68 | 127 | if len(address_ids) == 0 or user is None: | 125 | if len(address_ids) == 0 or user is None: |
69 | 128 | return [] | 126 | return [] |
72 | 129 | query.append(Or(Member.user_id == user.id, | 127 | query.append(or_(Member.user_id == user.id, |
73 | 130 | Member.address_id.is_in(address_ids))) | 128 | Member.address_id.in_(address_ids))) |
74 | 131 | # Calculate the rest of the query expression, which will get And'd | 129 | # Calculate the rest of the query expression, which will get And'd |
75 | 132 | # with the Or clause above (if there is one). | 130 | # with the Or clause above (if there is one). |
76 | 133 | if list_id is not None: | 131 | if list_id is not None: |
77 | 134 | query.append(Member.list_id == list_id) | 132 | query.append(Member.list_id == list_id) |
78 | 135 | if role is not None: | 133 | if role is not None: |
79 | 136 | query.append(Member.role == role) | 134 | query.append(Member.role == role) |
81 | 137 | results = store.find(Member, And(*query)) | 135 | results = store.query(Member).filter(and_(*query)) |
82 | 138 | return sorted(results, key=_membership_sort_key) | 136 | return sorted(results, key=_membership_sort_key) |
83 | 139 | 137 | ||
84 | 140 | def __iter__(self): | 138 | def __iter__(self): |
85 | 141 | 139 | ||
86 | === modified file 'src/mailman/bin/tests/test_master.py' | |||
87 | --- src/mailman/bin/tests/test_master.py 2014-04-28 15:23:35 +0000 | |||
88 | +++ src/mailman/bin/tests/test_master.py 2014-10-11 02:14:38 +0000 | |||
89 | @@ -55,7 +55,7 @@ | |||
90 | 55 | lock = master.acquire_lock_1(False, self.lock_file) | 55 | lock = master.acquire_lock_1(False, self.lock_file) |
91 | 56 | is_locked = lock.is_locked | 56 | is_locked = lock.is_locked |
92 | 57 | lock.unlock() | 57 | lock.unlock() |
94 | 58 | self.failUnless(is_locked) | 58 | self.assertTrue(is_locked) |
95 | 59 | 59 | ||
96 | 60 | def test_master_state(self): | 60 | def test_master_state(self): |
97 | 61 | my_lock = Lock(self.lock_file) | 61 | my_lock = Lock(self.lock_file) |
98 | 62 | 62 | ||
99 | === modified file 'src/mailman/commands/docs/conf.rst' | |||
100 | --- src/mailman/commands/docs/conf.rst 2013-09-01 15:08:46 +0000 | |||
101 | +++ src/mailman/commands/docs/conf.rst 2014-10-11 02:14:38 +0000 | |||
102 | @@ -22,7 +22,7 @@ | |||
103 | 22 | command without any options. | 22 | command without any options. |
104 | 23 | 23 | ||
105 | 24 | >>> command.process(FakeArgs) | 24 | >>> command.process(FakeArgs) |
107 | 25 | [logging.archiver] path: mailman.log | 25 | [alembic] script_location: mailman.database:alembic |
108 | 26 | ... | 26 | ... |
109 | 27 | [passwords] password_length: 8 | 27 | [passwords] password_length: 8 |
110 | 28 | ... | 28 | ... |
111 | @@ -43,12 +43,14 @@ | |||
112 | 43 | >>> FakeArgs.section = None | 43 | >>> FakeArgs.section = None |
113 | 44 | >>> FakeArgs.key = 'path' | 44 | >>> FakeArgs.key = 'path' |
114 | 45 | >>> command.process(FakeArgs) | 45 | >>> command.process(FakeArgs) |
115 | 46 | [logging.dbmigration] path: mailman.log | ||
116 | 46 | [logging.archiver] path: mailman.log | 47 | [logging.archiver] path: mailman.log |
117 | 47 | [logging.locks] path: mailman.log | 48 | [logging.locks] path: mailman.log |
118 | 48 | [logging.mischief] path: mailman.log | 49 | [logging.mischief] path: mailman.log |
119 | 49 | [logging.config] path: mailman.log | 50 | [logging.config] path: mailman.log |
120 | 50 | [logging.error] path: mailman.log | 51 | [logging.error] path: mailman.log |
121 | 51 | [logging.smtp] path: smtp.log | 52 | [logging.smtp] path: smtp.log |
122 | 53 | [logging.database] path: mailman.log | ||
123 | 52 | [logging.http] path: mailman.log | 54 | [logging.http] path: mailman.log |
124 | 53 | [logging.root] path: mailman.log | 55 | [logging.root] path: mailman.log |
125 | 54 | [logging.fromusenet] path: mailman.log | 56 | [logging.fromusenet] path: mailman.log |
126 | 55 | 57 | ||
127 | === modified file 'src/mailman/config/config.py' | |||
128 | --- src/mailman/config/config.py 2014-03-02 22:59:30 +0000 | |||
129 | +++ src/mailman/config/config.py 2014-10-11 02:14:38 +0000 | |||
130 | @@ -33,7 +33,7 @@ | |||
131 | 33 | from ConfigParser import SafeConfigParser | 33 | from ConfigParser import SafeConfigParser |
132 | 34 | from flufl.lock import Lock | 34 | from flufl.lock import Lock |
133 | 35 | from lazr.config import ConfigSchema, as_boolean | 35 | from lazr.config import ConfigSchema, as_boolean |
135 | 36 | from pkg_resources import resource_filename, resource_stream, resource_string | 36 | from pkg_resources import resource_stream, resource_string |
136 | 37 | from string import Template | 37 | from string import Template |
137 | 38 | from zope.component import getUtility | 38 | from zope.component import getUtility |
138 | 39 | from zope.event import notify | 39 | from zope.event import notify |
139 | @@ -46,7 +46,7 @@ | |||
140 | 46 | ConfigurationUpdatedEvent, IConfiguration, MissingConfigurationFileError) | 46 | ConfigurationUpdatedEvent, IConfiguration, MissingConfigurationFileError) |
141 | 47 | from mailman.interfaces.languages import ILanguageManager | 47 | from mailman.interfaces.languages import ILanguageManager |
142 | 48 | from mailman.utilities.filesystem import makedirs | 48 | from mailman.utilities.filesystem import makedirs |
144 | 49 | from mailman.utilities.modules import call_name | 49 | from mailman.utilities.modules import call_name, expand_path |
145 | 50 | 50 | ||
146 | 51 | 51 | ||
147 | 52 | SPACE = ' ' | 52 | SPACE = ' ' |
148 | @@ -87,6 +87,7 @@ | |||
149 | 87 | self.pipelines = {} | 87 | self.pipelines = {} |
150 | 88 | self.commands = {} | 88 | self.commands = {} |
151 | 89 | self.password_context = None | 89 | self.password_context = None |
152 | 90 | self.initialized = False | ||
153 | 90 | 91 | ||
154 | 91 | def _clear(self): | 92 | def _clear(self): |
155 | 92 | """Clear the cached configuration variables.""" | 93 | """Clear the cached configuration variables.""" |
156 | @@ -136,6 +137,7 @@ | |||
157 | 136 | # Expand and set up all directories. | 137 | # Expand and set up all directories. |
158 | 137 | self._expand_paths() | 138 | self._expand_paths() |
159 | 138 | self.ensure_directories_exist() | 139 | self.ensure_directories_exist() |
160 | 140 | self.initialized = True | ||
161 | 139 | notify(ConfigurationUpdatedEvent(self)) | 141 | notify(ConfigurationUpdatedEvent(self)) |
162 | 140 | 142 | ||
163 | 141 | def _expand_paths(self): | 143 | def _expand_paths(self): |
164 | @@ -304,12 +306,7 @@ | |||
165 | 304 | :return: A `ConfigParser` instance. | 306 | :return: A `ConfigParser` instance. |
166 | 305 | """ | 307 | """ |
167 | 306 | # Is the context coming from a file system or Python path? | 308 | # Is the context coming from a file system or Python path? |
174 | 307 | if path.startswith('python:'): | 309 | cfg_path = expand_path(path) |
169 | 308 | resource_path = path[7:] | ||
170 | 309 | package, dot, resource = resource_path.rpartition('.') | ||
171 | 310 | cfg_path = resource_filename(package, resource + '.cfg') | ||
172 | 311 | else: | ||
173 | 312 | cfg_path = path | ||
175 | 313 | parser = SafeConfigParser() | 310 | parser = SafeConfigParser() |
176 | 314 | files = parser.read(cfg_path) | 311 | files = parser.read(cfg_path) |
177 | 315 | if files != [cfg_path]: | 312 | if files != [cfg_path]: |
178 | 316 | 313 | ||
179 | === modified file 'src/mailman/config/configure.zcml' | |||
180 | --- src/mailman/config/configure.zcml 2013-11-26 02:26:15 +0000 | |||
181 | +++ src/mailman/config/configure.zcml 2014-10-11 02:14:38 +0000 | |||
182 | @@ -40,20 +40,6 @@ | |||
183 | 40 | factory="mailman.model.requests.ListRequests" | 40 | factory="mailman.model.requests.ListRequests" |
184 | 41 | /> | 41 | /> |
185 | 42 | 42 | ||
186 | 43 | <adapter | ||
187 | 44 | for="mailman.interfaces.database.IDatabase" | ||
188 | 45 | provides="mailman.interfaces.database.ITemporaryDatabase" | ||
189 | 46 | factory="mailman.database.sqlite.make_temporary" | ||
190 | 47 | name="sqlite" | ||
191 | 48 | /> | ||
192 | 49 | |||
193 | 50 | <adapter | ||
194 | 51 | for="mailman.interfaces.database.IDatabase" | ||
195 | 52 | provides="mailman.interfaces.database.ITemporaryDatabase" | ||
196 | 53 | factory="mailman.database.postgresql.make_temporary" | ||
197 | 54 | name="postgres" | ||
198 | 55 | /> | ||
199 | 56 | |||
200 | 57 | <utility | 43 | <utility |
201 | 58 | provides="mailman.interfaces.bounce.IBounceProcessor" | 44 | provides="mailman.interfaces.bounce.IBounceProcessor" |
202 | 59 | factory="mailman.model.bounce.BounceProcessor" | 45 | factory="mailman.model.bounce.BounceProcessor" |
203 | @@ -72,12 +58,6 @@ | |||
204 | 72 | /> | 58 | /> |
205 | 73 | 59 | ||
206 | 74 | <utility | 60 | <utility |
207 | 75 | provides="mailman.interfaces.database.IDatabaseFactory" | ||
208 | 76 | factory="mailman.database.factory.DatabaseTemporaryFactory" | ||
209 | 77 | name="temporary" | ||
210 | 78 | /> | ||
211 | 79 | |||
212 | 80 | <utility | ||
213 | 81 | provides="mailman.interfaces.domain.IDomainManager" | 61 | provides="mailman.interfaces.domain.IDomainManager" |
214 | 82 | factory="mailman.model.domain.DomainManager" | 62 | factory="mailman.model.domain.DomainManager" |
215 | 83 | /> | 63 | /> |
216 | 84 | 64 | ||
217 | === modified file 'src/mailman/config/schema.cfg' | |||
218 | --- src/mailman/config/schema.cfg 2014-01-01 14:59:42 +0000 | |||
219 | +++ src/mailman/config/schema.cfg 2014-10-11 02:14:38 +0000 | |||
220 | @@ -204,9 +204,6 @@ | |||
221 | 204 | url: sqlite:///$DATA_DIR/mailman.db | 204 | url: sqlite:///$DATA_DIR/mailman.db |
222 | 205 | debug: no | 205 | debug: no |
223 | 206 | 206 | ||
224 | 207 | # The module path to the migrations modules. | ||
225 | 208 | migrations_path: mailman.database.schema | ||
226 | 209 | |||
227 | 210 | [logging.template] | 207 | [logging.template] |
228 | 211 | # This defines various log settings. The options available are: | 208 | # This defines various log settings. The options available are: |
229 | 212 | # | 209 | # |
230 | @@ -240,6 +237,8 @@ | |||
231 | 240 | # - smtp-failure -- Unsuccessful SMTP activity | 237 | # - smtp-failure -- Unsuccessful SMTP activity |
232 | 241 | # - subscribe -- Information about leaves/joins | 238 | # - subscribe -- Information about leaves/joins |
233 | 242 | # - vette -- Message vetting information | 239 | # - vette -- Message vetting information |
234 | 240 | # - database -- Database activity | ||
235 | 241 | # - dbmigration -- Database migrations | ||
236 | 243 | format: %(asctime)s (%(process)d) %(message)s | 242 | format: %(asctime)s (%(process)d) %(message)s |
237 | 244 | datefmt: %b %d %H:%M:%S %Y | 243 | datefmt: %b %d %H:%M:%S %Y |
238 | 245 | propagate: no | 244 | propagate: no |
239 | @@ -304,6 +303,12 @@ | |||
240 | 304 | 303 | ||
241 | 305 | [logging.vette] | 304 | [logging.vette] |
242 | 306 | 305 | ||
243 | 306 | [logging.database] | ||
244 | 307 | level: warn | ||
245 | 308 | |||
246 | 309 | [logging.dbmigration] | ||
247 | 310 | level: warn | ||
248 | 311 | |||
249 | 307 | 312 | ||
250 | 308 | [webservice] | 313 | [webservice] |
251 | 309 | # The hostname at which admin web service resources are exposed. | 314 | # The hostname at which admin web service resources are exposed. |
252 | @@ -532,7 +537,7 @@ | |||
253 | 532 | # following values. | 537 | # following values. |
254 | 533 | 538 | ||
255 | 534 | # The class implementing the IArchiver interface. | 539 | # The class implementing the IArchiver interface. |
257 | 535 | class: | 540 | class: |
258 | 536 | 541 | ||
259 | 537 | # Set this to 'yes' to enable the archiver. | 542 | # Set this to 'yes' to enable the archiver. |
260 | 538 | enable: no | 543 | enable: no |
261 | @@ -640,3 +645,7 @@ | |||
262 | 640 | CC X-Original-CC | 645 | CC X-Original-CC |
263 | 641 | Content-Transfer-Encoding X-Original-Content-Transfer-Encoding | 646 | Content-Transfer-Encoding X-Original-Content-Transfer-Encoding |
264 | 642 | MIME-Version X-MIME-Version | 647 | MIME-Version X-MIME-Version |
265 | 648 | |||
266 | 649 | [alembic] | ||
267 | 650 | # path to migration scripts | ||
268 | 651 | script_location = mailman.database:alembic | ||
269 | 643 | 652 | ||
270 | === modified file 'src/mailman/core/logging.py' | |||
271 | --- src/mailman/core/logging.py 2014-04-28 15:23:35 +0000 | |||
272 | +++ src/mailman/core/logging.py 2014-10-11 02:14:38 +0000 | |||
273 | @@ -126,6 +126,10 @@ | |||
274 | 126 | continue | 126 | continue |
275 | 127 | if sub_name == 'locks': | 127 | if sub_name == 'locks': |
276 | 128 | log = logging.getLogger('flufl.lock') | 128 | log = logging.getLogger('flufl.lock') |
277 | 129 | elif sub_name == 'database': | ||
278 | 130 | log = logging.getLogger('sqlalchemy') | ||
279 | 131 | elif sub_name == 'dbmigration': | ||
280 | 132 | log = logging.getLogger('alembic') | ||
281 | 129 | else: | 133 | else: |
282 | 130 | logger_name = 'mailman.' + sub_name | 134 | logger_name = 'mailman.' + sub_name |
283 | 131 | log = logging.getLogger(logger_name) | 135 | log = logging.getLogger(logger_name) |
284 | 132 | 136 | ||
285 | === added directory 'src/mailman/database/alembic' | |||
286 | === added file 'src/mailman/database/alembic/__init__.py' | |||
287 | --- src/mailman/database/alembic/__init__.py 1970-01-01 00:00:00 +0000 | |||
288 | +++ src/mailman/database/alembic/__init__.py 2014-10-11 02:14:38 +0000 | |||
289 | @@ -0,0 +1,32 @@ | |||
290 | 1 | # Copyright (C) 2014 by the Free Software Foundation, Inc. | ||
291 | 2 | # | ||
292 | 3 | # This file is part of GNU Mailman. | ||
293 | 4 | # | ||
294 | 5 | # GNU Mailman is free software: you can redistribute it and/or modify it under | ||
295 | 6 | # the terms of the GNU General Public License as published by the Free | ||
296 | 7 | # Software Foundation, either version 3 of the License, or (at your option) | ||
297 | 8 | # any later version. | ||
298 | 9 | # | ||
299 | 10 | # GNU Mailman is distributed in the hope that it will be useful, but WITHOUT | ||
300 | 11 | # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or | ||
301 | 12 | # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for | ||
302 | 13 | # more details. | ||
303 | 14 | # | ||
304 | 15 | # You should have received a copy of the GNU General Public License along with | ||
305 | 16 | # GNU Mailman. If not, see <http://www.gnu.org/licenses/>. | ||
306 | 17 | |||
307 | 18 | "Alembic config init." | ||
308 | 19 | |||
309 | 20 | from __future__ import absolute_import, print_function, unicode_literals | ||
310 | 21 | |||
311 | 22 | __metaclass__ = type | ||
312 | 23 | __all__ = [ | ||
313 | 24 | 'alembic_cfg' | ||
314 | 25 | ] | ||
315 | 26 | |||
316 | 27 | |||
317 | 28 | from alembic.config import Config | ||
318 | 29 | from mailman.utilities.modules import expand_path | ||
319 | 30 | |||
320 | 31 | |||
321 | 32 | alembic_cfg=Config(expand_path("python:mailman.config.schema")) | ||
322 | 0 | 33 | ||
323 | === added file 'src/mailman/database/alembic/env.py' | |||
324 | --- src/mailman/database/alembic/env.py 1970-01-01 00:00:00 +0000 | |||
325 | +++ src/mailman/database/alembic/env.py 2014-10-11 02:14:38 +0000 | |||
326 | @@ -0,0 +1,80 @@ | |||
327 | 1 | # Copyright (C) 2014 by the Free Software Foundation, Inc. | ||
328 | 2 | # | ||
329 | 3 | # This file is part of GNU Mailman. | ||
330 | 4 | # | ||
331 | 5 | # GNU Mailman is free software: you can redistribute it and/or modify it under | ||
332 | 6 | # the terms of the GNU General Public License as published by the Free | ||
333 | 7 | # Software Foundation, either version 3 of the License, or (at your option) | ||
334 | 8 | # any later version. | ||
335 | 9 | # | ||
336 | 10 | # GNU Mailman is distributed in the hope that it will be useful, but WITHOUT | ||
337 | 11 | # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or | ||
338 | 12 | # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for | ||
339 | 13 | # more details. | ||
340 | 14 | # | ||
341 | 15 | # You should have received a copy of the GNU General Public License along with | ||
342 | 16 | # GNU Mailman. If not, see <http://www.gnu.org/licenses/>. | ||
343 | 17 | |||
344 | 18 | """Alembic migration environment.""" | ||
345 | 19 | |||
346 | 20 | from __future__ import absolute_import, print_function, unicode_literals | ||
347 | 21 | |||
348 | 22 | __metaclass__ = type | ||
349 | 23 | __all__ = [ | ||
350 | 24 | 'run_migrations_offline', | ||
351 | 25 | 'run_migrations_online', | ||
352 | 26 | ] | ||
353 | 27 | |||
354 | 28 | |||
355 | 29 | from alembic import context | ||
356 | 30 | from contextlib import closing | ||
357 | 31 | from sqlalchemy import create_engine | ||
358 | 32 | |||
359 | 33 | from mailman.core import initialize | ||
360 | 34 | from mailman.config import config | ||
361 | 35 | from mailman.database.alembic import alembic_cfg | ||
362 | 36 | from mailman.database.model import Model | ||
363 | 37 | from mailman.utilities.string import expand | ||
364 | 38 | |||
365 | 39 | if not config.initialized: | ||
366 | 40 | initialize.initialize_1(context.config.config_file_name) | ||
367 | 41 | |||
368 | 42 | |||
369 | 43 | |||
370 | 0 | 44 | ||
371 | 45 | def run_migrations_offline(): | ||
372 | 46 | """Run migrations in 'offline' mode. | ||
373 | 47 | |||
374 | 48 | This configures the context with just a URL and not an Engine, | ||
375 | 49 | though an Engine is acceptable here as well. By skipping the Engine | ||
376 | 50 | creation we don't even need a DBAPI to be available. | ||
377 | 51 | |||
378 | 52 | Calls to context.execute() here emit the given string to the script | ||
379 | 53 | output. | ||
380 | 54 | """ | ||
381 | 55 | url = expand(config.database.url, config.paths) | ||
382 | 56 | context.configure(url=url, target_metadata=Model.metadata) | ||
383 | 57 | with context.begin_transaction(): | ||
384 | 58 | context.run_migrations() | ||
385 | 59 | |||
386 | 60 | |||
387 | 61 | def run_migrations_online(): | ||
388 | 62 | """Run migrations in 'online' mode. | ||
389 | 63 | |||
390 | 64 | In this scenario we need to create an Engine and associate a | ||
391 | 65 | connection with the context. | ||
392 | 66 | """ | ||
393 | 67 | url = expand(config.database.url, config.paths) | ||
394 | 68 | engine = create_engine(url) | ||
395 | 69 | |||
396 | 70 | connection = engine.connect() | ||
397 | 71 | with closing(connection): | ||
398 | 72 | context.configure( | ||
399 | 73 | connection=connection, target_metadata=Model.metadata) | ||
400 | 74 | with context.begin_transaction(): | ||
401 | 75 | context.run_migrations() | ||
402 | 76 | |||
403 | 77 | |||
404 | 78 | if context.is_offline_mode(): | ||
405 | 79 | run_migrations_offline() | ||
406 | 80 | else: | ||
407 | 81 | run_migrations_online() | ||
408 | 1 | 82 | ||
409 | === added file 'src/mailman/database/alembic/script.py.mako' | |||
410 | --- src/mailman/database/alembic/script.py.mako 1970-01-01 00:00:00 +0000 | |||
411 | +++ src/mailman/database/alembic/script.py.mako 2014-10-11 02:14:38 +0000 | |||
412 | @@ -0,0 +1,22 @@ | |||
413 | 1 | """${message} | ||
414 | 2 | |||
415 | 3 | Revision ID: ${up_revision} | ||
416 | 4 | Revises: ${down_revision} | ||
417 | 5 | Create Date: ${create_date} | ||
418 | 6 | |||
419 | 7 | """ | ||
420 | 8 | |||
421 | 9 | # revision identifiers, used by Alembic. | ||
422 | 10 | revision = ${repr(up_revision)} | ||
423 | 11 | down_revision = ${repr(down_revision)} | ||
424 | 12 | |||
425 | 13 | from alembic import op | ||
426 | 14 | import sqlalchemy as sa | ||
427 | 15 | ${imports if imports else ""} | ||
428 | 16 | |||
429 | 17 | def upgrade(): | ||
430 | 18 | ${upgrades if upgrades else "pass"} | ||
431 | 19 | |||
432 | 20 | |||
433 | 21 | def downgrade(): | ||
434 | 22 | ${downgrades if downgrades else "pass"} | ||
435 | 0 | 23 | ||
436 | === added directory 'src/mailman/database/alembic/versions' | |||
437 | === added file 'src/mailman/database/alembic/versions/51b7f92bd06c_initial.py' | |||
438 | --- src/mailman/database/alembic/versions/51b7f92bd06c_initial.py 1970-01-01 00:00:00 +0000 | |||
439 | +++ src/mailman/database/alembic/versions/51b7f92bd06c_initial.py 2014-10-11 02:14:38 +0000 | |||
440 | @@ -0,0 +1,34 @@ | |||
441 | 1 | """initial | ||
442 | 2 | |||
443 | 3 | Revision ID: 51b7f92bd06c | ||
444 | 4 | Revises: None | ||
445 | 5 | Create Date: 2014-10-10 09:53:35.624472 | ||
446 | 6 | |||
447 | 7 | """ | ||
448 | 8 | |||
449 | 9 | # revision identifiers, used by Alembic. | ||
450 | 10 | revision = '51b7f92bd06c' | ||
451 | 11 | down_revision = None | ||
452 | 12 | |||
453 | 13 | from alembic import op | ||
454 | 14 | import sqlalchemy as sa | ||
455 | 15 | |||
456 | 16 | |||
457 | 17 | def upgrade(): | ||
458 | 18 | ### commands auto generated by Alembic - please adjust! ### | ||
459 | 19 | op.drop_table('version') | ||
460 | 20 | if op.get_bind().dialect.name != "sqlite": | ||
461 | 21 | # SQLite does not support dropping columns | ||
462 | 22 | op.drop_column('mailinglist', 'acceptable_aliases_id') | ||
463 | 23 | op.create_index(op.f('ix_user__user_id'), 'user', ['_user_id'], unique=False) | ||
464 | 24 | op.drop_index('ix_user_user_id', table_name='user') | ||
465 | 25 | ### end Alembic commands ### | ||
466 | 26 | |||
467 | 27 | |||
468 | 28 | def downgrade(): | ||
469 | 29 | ### commands auto generated by Alembic - please adjust! ### | ||
470 | 30 | op.create_table('version') | ||
471 | 31 | op.create_index('ix_user_user_id', 'user', ['_user_id'], unique=False) | ||
472 | 32 | op.drop_index(op.f('ix_user__user_id'), table_name='user') | ||
473 | 33 | op.add_column('mailinglist', sa.Column('acceptable_aliases_id', sa.INTEGER(), nullable=True)) | ||
474 | 34 | ### end Alembic commands ### | ||
475 | 0 | 35 | ||
476 | === modified file 'src/mailman/database/base.py' | |||
477 | --- src/mailman/database/base.py 2014-01-01 14:59:42 +0000 | |||
478 | +++ src/mailman/database/base.py 2014-10-11 02:14:38 +0000 | |||
479 | @@ -19,49 +19,40 @@ | |||
480 | 19 | 19 | ||
481 | 20 | __metaclass__ = type | 20 | __metaclass__ = type |
482 | 21 | __all__ = [ | 21 | __all__ = [ |
484 | 22 | 'StormBaseDatabase', | 22 | 'SABaseDatabase', |
485 | 23 | ] | 23 | ] |
486 | 24 | 24 | ||
487 | 25 | 25 | ||
488 | 26 | import os | ||
489 | 27 | import sys | ||
490 | 28 | import logging | 26 | import logging |
491 | 29 | 27 | ||
496 | 30 | from lazr.config import as_boolean | 28 | from alembic import command |
497 | 31 | from pkg_resources import resource_listdir, resource_string | 29 | from sqlalchemy import create_engine |
498 | 32 | from storm.cache import GenerationalCache | 30 | from sqlalchemy.orm import sessionmaker |
495 | 33 | from storm.locals import create_database, Store | ||
499 | 34 | from zope.interface import implementer | 31 | from zope.interface import implementer |
500 | 35 | 32 | ||
501 | 36 | from mailman.config import config | 33 | from mailman.config import config |
502 | 37 | from mailman.interfaces.database import IDatabase | 34 | from mailman.interfaces.database import IDatabase |
503 | 38 | from mailman.model.version import Version | ||
504 | 39 | from mailman.utilities.string import expand | 35 | from mailman.utilities.string import expand |
505 | 40 | 36 | ||
506 | 37 | |||
507 | 41 | log = logging.getLogger('mailman.config') | 38 | log = logging.getLogger('mailman.config') |
508 | 42 | |||
509 | 43 | NL = '\n' | 39 | NL = '\n' |
510 | 44 | 40 | ||
511 | 45 | 41 | ||
512 | 46 | 42 | ||
513 | 47 | 43 | ||
514 | 48 | @implementer(IDatabase) | 44 | @implementer(IDatabase) |
517 | 49 | class StormBaseDatabase: | 45 | class SABaseDatabase: |
518 | 50 | """The database base class for use with the Storm ORM. | 46 | """The database base class for use with SQLAlchemy. |
519 | 51 | 47 | ||
521 | 52 | Use this as a base class for your DB-specific derived classes. | 48 | Use this as a base class for your DB-Specific derived classes. |
522 | 53 | """ | 49 | """ |
523 | 54 | |||
524 | 55 | # Tag used to distinguish the database being used. Override this in base | ||
525 | 56 | # classes. | ||
526 | 57 | TAG = '' | ||
527 | 58 | |||
528 | 59 | def __init__(self): | 50 | def __init__(self): |
529 | 60 | self.url = None | 51 | self.url = None |
530 | 61 | self.store = None | 52 | self.store = None |
531 | 62 | 53 | ||
532 | 63 | def begin(self): | 54 | def begin(self): |
533 | 64 | """See `IDatabase`.""" | 55 | """See `IDatabase`.""" |
535 | 65 | # Storm takes care of this for us. | 56 | # SQLAlchemy does this for us. |
536 | 66 | pass | 57 | pass |
537 | 67 | 58 | ||
538 | 68 | def commit(self): | 59 | def commit(self): |
539 | @@ -72,16 +63,6 @@ | |||
540 | 72 | """See `IDatabase`.""" | 63 | """See `IDatabase`.""" |
541 | 73 | self.store.rollback() | 64 | self.store.rollback() |
542 | 74 | 65 | ||
543 | 75 | def _database_exists(self): | ||
544 | 76 | """Return True if the database exists and is initialized. | ||
545 | 77 | |||
546 | 78 | Return False when Mailman needs to create and initialize the | ||
547 | 79 | underlying database schema. | ||
548 | 80 | |||
549 | 81 | Base classes *must* override this. | ||
550 | 82 | """ | ||
551 | 83 | raise NotImplementedError | ||
552 | 84 | |||
553 | 85 | def _pre_reset(self, store): | 66 | def _pre_reset(self, store): |
554 | 86 | """Clean up method for testing. | 67 | """Clean up method for testing. |
555 | 87 | 68 | ||
556 | @@ -113,6 +94,7 @@ | |||
557 | 113 | """See `IDatabase`.""" | 94 | """See `IDatabase`.""" |
558 | 114 | # Calculate the engine url. | 95 | # Calculate the engine url. |
559 | 115 | url = expand(config.database.url, config.paths) | 96 | url = expand(config.database.url, config.paths) |
560 | 97 | self._prepare(url) | ||
561 | 116 | log.debug('Database url: %s', url) | 98 | log.debug('Database url: %s', url) |
562 | 117 | # XXX By design of SQLite, database file creation does not honor | 99 | # XXX By design of SQLite, database file creation does not honor |
563 | 118 | # umask. See their ticket #1193: | 100 | # umask. See their ticket #1193: |
564 | @@ -129,101 +111,7 @@ | |||
565 | 129 | # engines, and yes, we could have chmod'd the file after the fact, but | 111 | # engines, and yes, we could have chmod'd the file after the fact, but |
566 | 130 | # half dozen and all... | 112 | # half dozen and all... |
567 | 131 | self.url = url | 113 | self.url = url |
666 | 132 | self._prepare(url) | 114 | self.engine = create_engine(url) |
667 | 133 | database = create_database(url) | 115 | session = sessionmaker(bind=self.engine) |
668 | 134 | store = Store(database, GenerationalCache()) | 116 | self.store = session() |
669 | 135 | database.DEBUG = (as_boolean(config.database.debug) | 117 | self.store.commit() |
572 | 136 | if debug is None else debug) | ||
573 | 137 | self.store = store | ||
574 | 138 | store.commit() | ||
575 | 139 | |||
576 | 140 | def load_migrations(self, until=None): | ||
577 | 141 | """Load schema migrations. | ||
578 | 142 | |||
579 | 143 | :param until: Load only the migrations up to the specified timestamp. | ||
580 | 144 | With default value of None, load all migrations. | ||
581 | 145 | :type until: string | ||
582 | 146 | """ | ||
583 | 147 | migrations_path = config.database.migrations_path | ||
584 | 148 | if '.' in migrations_path: | ||
585 | 149 | parent, dot, child = migrations_path.rpartition('.') | ||
586 | 150 | else: | ||
587 | 151 | parent = migrations_path | ||
588 | 152 | child = '' | ||
589 | 153 | # If the database does not yet exist, load the base schema. | ||
590 | 154 | filenames = sorted(resource_listdir(parent, child)) | ||
591 | 155 | # Find out which schema migrations have already been loaded. | ||
592 | 156 | if self._database_exists(self.store): | ||
593 | 157 | versions = set(version.version for version in | ||
594 | 158 | self.store.find(Version, component='schema')) | ||
595 | 159 | else: | ||
596 | 160 | versions = set() | ||
597 | 161 | for filename in filenames: | ||
598 | 162 | module_fn, extension = os.path.splitext(filename) | ||
599 | 163 | if extension != '.py': | ||
600 | 164 | continue | ||
601 | 165 | parts = module_fn.split('_') | ||
602 | 166 | if len(parts) < 2: | ||
603 | 167 | continue | ||
604 | 168 | version = parts[1].strip() | ||
605 | 169 | if len(version) == 0: | ||
606 | 170 | # Not a schema migration file. | ||
607 | 171 | continue | ||
608 | 172 | if version in versions: | ||
609 | 173 | log.debug('already migrated to %s', version) | ||
610 | 174 | continue | ||
611 | 175 | if until is not None and version > until: | ||
612 | 176 | # We're done. | ||
613 | 177 | break | ||
614 | 178 | module_path = migrations_path + '.' + module_fn | ||
615 | 179 | __import__(module_path) | ||
616 | 180 | upgrade = getattr(sys.modules[module_path], 'upgrade', None) | ||
617 | 181 | if upgrade is None: | ||
618 | 182 | continue | ||
619 | 183 | log.debug('migrating db to %s: %s', version, module_path) | ||
620 | 184 | upgrade(self, self.store, version, module_path) | ||
621 | 185 | self.commit() | ||
622 | 186 | |||
623 | 187 | def load_sql(self, store, sql): | ||
624 | 188 | """Load the given SQL into the store. | ||
625 | 189 | |||
626 | 190 | :param store: The Storm store to load the schema into. | ||
627 | 191 | :type store: storm.locals.Store` | ||
628 | 192 | :param sql: The possibly multi-line SQL to load. | ||
629 | 193 | :type sql: string | ||
630 | 194 | """ | ||
631 | 195 | # Discard all blank and comment lines. | ||
632 | 196 | lines = (line for line in sql.splitlines() | ||
633 | 197 | if line.strip() != '' and line.strip()[:2] != '--') | ||
634 | 198 | sql = NL.join(lines) | ||
635 | 199 | for statement in sql.split(';'): | ||
636 | 200 | if statement.strip() != '': | ||
637 | 201 | store.execute(statement + ';') | ||
638 | 202 | |||
639 | 203 | def load_schema(self, store, version, filename, module_path): | ||
640 | 204 | """Load the schema from a file. | ||
641 | 205 | |||
642 | 206 | This is a helper method for migration classes to call. | ||
643 | 207 | |||
644 | 208 | :param store: The Storm store to load the schema into. | ||
645 | 209 | :type store: storm.locals.Store` | ||
646 | 210 | :param version: The schema version identifier of the form | ||
647 | 211 | YYYYMMDDHHMMSS. | ||
648 | 212 | :type version: string | ||
649 | 213 | :param filename: The file name containing the schema to load. Pass | ||
650 | 214 | `None` if there is no schema file to load. | ||
651 | 215 | :type filename: string | ||
652 | 216 | :param module_path: The fully qualified Python module path to the | ||
653 | 217 | migration module being loaded. This is used to record information | ||
654 | 218 | for use by the test suite. | ||
655 | 219 | :type module_path: string | ||
656 | 220 | """ | ||
657 | 221 | if filename is not None: | ||
658 | 222 | contents = resource_string('mailman.database.schema', filename) | ||
659 | 223 | self.load_sql(store, contents) | ||
660 | 224 | # Add a marker that indicates the migration version being applied. | ||
661 | 225 | store.add(Version(component='schema', version=version)) | ||
662 | 226 | |||
663 | 227 | @staticmethod | ||
664 | 228 | def _make_temporary(): | ||
665 | 229 | raise NotImplementedError | ||
670 | 230 | 118 | ||
671 | === removed directory 'src/mailman/database/docs' | |||
672 | === removed file 'src/mailman/database/docs/__init__.py' | |||
673 | === removed file 'src/mailman/database/docs/migration.rst' | |||
674 | --- src/mailman/database/docs/migration.rst 2014-04-28 15:23:35 +0000 | |||
675 | +++ src/mailman/database/docs/migration.rst 1970-01-01 00:00:00 +0000 | |||
676 | @@ -1,207 +0,0 @@ | |||
677 | 1 | ================= | ||
678 | 2 | Schema migrations | ||
679 | 3 | ================= | ||
680 | 4 | |||
681 | 5 | The SQL database schema will over time require upgrading to support new | ||
682 | 6 | features. This is supported via schema migration. | ||
683 | 7 | |||
684 | 8 | Migrations are embodied in individual Python classes, which themselves may | ||
685 | 9 | load SQL into the database. The naming scheme for migration files is: | ||
686 | 10 | |||
687 | 11 | mm_YYYYMMDDHHMMSS_comment.py | ||
688 | 12 | |||
689 | 13 | where `YYYYMMDDHHMMSS` is a required numeric year, month, day, hour, minute, | ||
690 | 14 | and second specifier providing unique ordering for processing. Only this | ||
691 | 15 | component of the file name is used to determine the ordering. The prefix is | ||
692 | 16 | required due to Python module naming requirements, but it is actually | ||
693 | 17 | ignored. `mm_` is reserved for Mailman's own use. | ||
694 | 18 | |||
695 | 19 | The optional `comment` part of the file name can be used as a short | ||
696 | 20 | description for the migration, although comments and docstrings in the | ||
697 | 21 | migration files should be used for more detailed descriptions. | ||
698 | 22 | |||
699 | 23 | Migrations are applied automatically when Mailman starts up, but can also be | ||
700 | 24 | applied at any time by calling in the API directly. Once applied, a | ||
701 | 25 | migration's version string is registered so it will not be applied again. | ||
702 | 26 | |||
703 | 27 | We see that the base migration, as well as subsequent standard migrations, are | ||
704 | 28 | already applied. | ||
705 | 29 | |||
706 | 30 | >>> from mailman.model.version import Version | ||
707 | 31 | >>> results = config.db.store.find(Version, component='schema') | ||
708 | 32 | >>> results.count() | ||
709 | 33 | 4 | ||
710 | 34 | >>> versions = sorted(result.version for result in results) | ||
711 | 35 | >>> for version in versions: | ||
712 | 36 | ... print(version) | ||
713 | 37 | 00000000000000 | ||
714 | 38 | 20120407000000 | ||
715 | 39 | 20121015000000 | ||
716 | 40 | 20130406000000 | ||
717 | 41 | |||
718 | 42 | |||
719 | 43 | Migrations | ||
720 | 44 | ========== | ||
721 | 45 | |||
722 | 46 | Migrations can be loaded at any time, and can be found in the migrations path | ||
723 | 47 | specified in the configuration file. | ||
724 | 48 | |||
725 | 49 | .. Create a temporary directory for the migrations:: | ||
726 | 50 | |||
727 | 51 | >>> import os, sys, tempfile | ||
728 | 52 | >>> tempdir = tempfile.mkdtemp() | ||
729 | 53 | >>> path = os.path.join(tempdir, 'migrations') | ||
730 | 54 | >>> os.makedirs(path) | ||
731 | 55 | >>> sys.path.append(tempdir) | ||
732 | 56 | >>> config.push('migrations', """ | ||
733 | 57 | ... [database] | ||
734 | 58 | ... migrations_path: migrations | ||
735 | 59 | ... """) | ||
736 | 60 | |||
737 | 61 | .. Clean this up at the end of the doctest. | ||
738 | 62 | >>> def cleanup(): | ||
739 | 63 | ... import shutil | ||
740 | 64 | ... from mailman.config import config | ||
741 | 65 | ... config.pop('migrations') | ||
742 | 66 | ... shutil.rmtree(tempdir) | ||
743 | 67 | >>> cleanups.append(cleanup) | ||
744 | 68 | |||
745 | 69 | Here is an example migrations module. The key part of this interface is the | ||
746 | 70 | ``upgrade()`` method, which takes four arguments: | ||
747 | 71 | |||
748 | 72 | * `database` - The database class, as derived from `StormBaseDatabase` | ||
749 | 73 | * `store` - The Storm `Store` object. | ||
750 | 74 | * `version` - The version string as derived from the migrations module's file | ||
751 | 75 | name. This will include only the `YYYYMMDDHHMMSS` string. | ||
752 | 76 | * `module_path` - The dotted module path to the migrations module, suitable | ||
753 | 77 | for lookup in `sys.modules`. | ||
754 | 78 | |||
755 | 79 | This migration module just adds a marker to the `version` table. | ||
756 | 80 | |||
757 | 81 | >>> with open(os.path.join(path, '__init__.py'), 'w') as fp: | ||
758 | 82 | ... pass | ||
759 | 83 | >>> with open(os.path.join(path, 'mm_20159999000000.py'), 'w') as fp: | ||
760 | 84 | ... print(""" | ||
761 | 85 | ... from __future__ import unicode_literals | ||
762 | 86 | ... from mailman.model.version import Version | ||
763 | 87 | ... def upgrade(database, store, version, module_path): | ||
764 | 88 | ... v = Version(component='test', version=version) | ||
765 | 89 | ... store.add(v) | ||
766 | 90 | ... database.load_schema(store, version, None, module_path) | ||
767 | 91 | ... """, file=fp) | ||
768 | 92 | |||
769 | 93 | This will load the new migration, since it hasn't been loaded before. | ||
770 | 94 | |||
771 | 95 | >>> config.db.load_migrations() | ||
772 | 96 | >>> results = config.db.store.find(Version, component='schema') | ||
773 | 97 | >>> for result in sorted(result.version for result in results): | ||
774 | 98 | ... print(result) | ||
775 | 99 | 00000000000000 | ||
776 | 100 | 20120407000000 | ||
777 | 101 | 20121015000000 | ||
778 | 102 | 20130406000000 | ||
779 | 103 | 20159999000000 | ||
780 | 104 | >>> test = config.db.store.find(Version, component='test').one() | ||
781 | 105 | >>> print(test.version) | ||
782 | 106 | 20159999000000 | ||
783 | 107 | |||
784 | 108 | Migrations will only be loaded once. | ||
785 | 109 | |||
786 | 110 | >>> with open(os.path.join(path, 'mm_20159999000001.py'), 'w') as fp: | ||
787 | 111 | ... print(""" | ||
788 | 112 | ... from __future__ import unicode_literals | ||
789 | 113 | ... from mailman.model.version import Version | ||
790 | 114 | ... _marker = 801 | ||
791 | 115 | ... def upgrade(database, store, version, module_path): | ||
792 | 116 | ... global _marker | ||
793 | 117 | ... # Pad enough zeros on the left to reach 14 characters wide. | ||
794 | 118 | ... marker = '{0:=#014d}'.format(_marker) | ||
795 | 119 | ... _marker += 1 | ||
796 | 120 | ... v = Version(component='test', version=marker) | ||
797 | 121 | ... store.add(v) | ||
798 | 122 | ... database.load_schema(store, version, None, module_path) | ||
799 | 123 | ... """, file=fp) | ||
800 | 124 | |||
801 | 125 | The first time we load this new migration, we'll get the 801 marker. | ||
802 | 126 | |||
803 | 127 | >>> config.db.load_migrations() | ||
804 | 128 | >>> results = config.db.store.find(Version, component='schema') | ||
805 | 129 | >>> for result in sorted(result.version for result in results): | ||
806 | 130 | ... print(result) | ||
807 | 131 | 00000000000000 | ||
808 | 132 | 20120407000000 | ||
809 | 133 | 20121015000000 | ||
810 | 134 | 20130406000000 | ||
811 | 135 | 20159999000000 | ||
812 | 136 | 20159999000001 | ||
813 | 137 | >>> test = config.db.store.find(Version, component='test') | ||
814 | 138 | >>> for marker in sorted(marker.version for marker in test): | ||
815 | 139 | ... print(marker) | ||
816 | 140 | 00000000000801 | ||
817 | 141 | 20159999000000 | ||
818 | 142 | |||
819 | 143 | We do not get an 802 marker because the migration has already been loaded. | ||
820 | 144 | |||
821 | 145 | >>> config.db.load_migrations() | ||
822 | 146 | >>> results = config.db.store.find(Version, component='schema') | ||
823 | 147 | >>> for result in sorted(result.version for result in results): | ||
824 | 148 | ... print(result) | ||
825 | 149 | 00000000000000 | ||
826 | 150 | 20120407000000 | ||
827 | 151 | 20121015000000 | ||
828 | 152 | 20130406000000 | ||
829 | 153 | 20159999000000 | ||
830 | 154 | 20159999000001 | ||
831 | 155 | >>> test = config.db.store.find(Version, component='test') | ||
832 | 156 | >>> for marker in sorted(marker.version for marker in test): | ||
833 | 157 | ... print(marker) | ||
834 | 158 | 00000000000801 | ||
835 | 159 | 20159999000000 | ||
836 | 160 | |||
837 | 161 | |||
838 | 162 | Partial upgrades | ||
839 | 163 | ================ | ||
840 | 164 | |||
841 | 165 | It's possible (mostly for testing purposes) to only do a partial upgrade, by | ||
842 | 166 | providing a timestamp to `load_migrations()`. To demonstrate this, we add two | ||
843 | 167 | additional migrations, intended to be applied in sequential order. | ||
844 | 168 | |||
845 | 169 | >>> from shutil import copyfile | ||
846 | 170 | >>> from mailman.testing.helpers import chdir | ||
847 | 171 | >>> with chdir(path): | ||
848 | 172 | ... copyfile('mm_20159999000000.py', 'mm_20159999000002.py') | ||
849 | 173 | ... copyfile('mm_20159999000000.py', 'mm_20159999000003.py') | ||
850 | 174 | ... copyfile('mm_20159999000000.py', 'mm_20159999000004.py') | ||
851 | 175 | |||
852 | 176 | Now, only migrate to the ...03 timestamp. | ||
853 | 177 | |||
854 | 178 | >>> config.db.load_migrations('20159999000003') | ||
855 | 179 | |||
856 | 180 | You'll notice that the ...04 version is not present. | ||
857 | 181 | |||
858 | 182 | >>> results = config.db.store.find(Version, component='schema') | ||
859 | 183 | >>> for result in sorted(result.version for result in results): | ||
860 | 184 | ... print(result) | ||
861 | 185 | 00000000000000 | ||
862 | 186 | 20120407000000 | ||
863 | 187 | 20121015000000 | ||
864 | 188 | 20130406000000 | ||
865 | 189 | 20159999000000 | ||
866 | 190 | 20159999000001 | ||
867 | 191 | 20159999000002 | ||
868 | 192 | 20159999000003 | ||
869 | 193 | |||
870 | 194 | |||
871 | 195 | .. cleanup: | ||
872 | 196 | Because the Version table holds schema migration data, it will not be | ||
873 | 197 | cleaned up by the standard test suite. This is generally not a problem | ||
874 | 198 | for SQLite since each test gets a new database file, but for PostgreSQL, | ||
875 | 199 | this will cause migration.rst to fail on subsequent runs. So let's just | ||
876 | 200 | clean up the database explicitly. | ||
877 | 201 | |||
878 | 202 | >>> if config.db.TAG != 'sqlite': | ||
879 | 203 | ... results = config.db.store.execute(""" | ||
880 | 204 | ... DELETE FROM version WHERE version.version >= '201299990000' | ||
881 | 205 | ... OR version.component = 'test'; | ||
882 | 206 | ... """) | ||
883 | 207 | ... config.db.commit() | ||
884 | 208 | 0 | ||
885 | === modified file 'src/mailman/database/factory.py' | |||
886 | --- src/mailman/database/factory.py 2014-01-01 14:59:42 +0000 | |||
887 | +++ src/mailman/database/factory.py 2014-10-11 02:14:38 +0000 | |||
888 | @@ -22,7 +22,6 @@ | |||
889 | 22 | __metaclass__ = type | 22 | __metaclass__ = type |
890 | 23 | __all__ = [ | 23 | __all__ = [ |
891 | 24 | 'DatabaseFactory', | 24 | 'DatabaseFactory', |
892 | 25 | 'DatabaseTemporaryFactory', | ||
893 | 26 | 'DatabaseTestingFactory', | 25 | 'DatabaseTestingFactory', |
894 | 27 | ] | 26 | ] |
895 | 28 | 27 | ||
896 | @@ -30,15 +29,19 @@ | |||
897 | 30 | import os | 29 | import os |
898 | 31 | import types | 30 | import types |
899 | 32 | 31 | ||
900 | 32 | from alembic import command | ||
901 | 33 | from alembic.migration import MigrationContext | ||
902 | 34 | from alembic.script import ScriptDirectory | ||
903 | 33 | from flufl.lock import Lock | 35 | from flufl.lock import Lock |
905 | 34 | from zope.component import getAdapter | 36 | from sqlalchemy import MetaData |
906 | 35 | from zope.interface import implementer | 37 | from zope.interface import implementer |
907 | 36 | from zope.interface.verify import verifyObject | 38 | from zope.interface.verify import verifyObject |
908 | 37 | 39 | ||
909 | 38 | from mailman.config import config | 40 | from mailman.config import config |
913 | 39 | from mailman.interfaces.database import ( | 41 | from mailman.database.model import Model |
914 | 40 | IDatabase, IDatabaseFactory, ITemporaryDatabase) | 42 | from mailman.database.alembic import alembic_cfg |
915 | 41 | from mailman.utilities.modules import call_name | 43 | from mailman.interfaces.database import IDatabase, IDatabaseFactory |
916 | 44 | from mailman.utilities.modules import call_name, expand_path | ||
917 | 42 | 45 | ||
918 | 43 | 46 | ||
919 | 44 | 47 | ||
920 | 45 | 48 | ||
921 | @@ -54,18 +57,78 @@ | |||
922 | 54 | database = call_name(database_class) | 57 | database = call_name(database_class) |
923 | 55 | verifyObject(IDatabase, database) | 58 | verifyObject(IDatabase, database) |
924 | 56 | database.initialize() | 59 | database.initialize() |
926 | 57 | database.load_migrations() | 60 | schema_mgr = SchemaManager(database) |
927 | 61 | schema_mgr.setup_db() | ||
928 | 58 | database.commit() | 62 | database.commit() |
929 | 59 | return database | 63 | return database |
930 | 60 | 64 | ||
931 | 61 | 65 | ||
932 | 62 | 66 | ||
933 | 63 | 67 | ||
934 | 68 | class SchemaManager: | ||
935 | 69 | |||
936 | 70 | LAST_STORM_SCHEMA_VERSION = '20130406000000' | ||
937 | 71 | |||
938 | 72 | def __init__(self, database): | ||
939 | 73 | self.database = database | ||
940 | 74 | self.script = ScriptDirectory.from_config(alembic_cfg) | ||
941 | 75 | |||
942 | 76 | def get_storm_schema_version(self): | ||
943 | 77 | md = MetaData() | ||
944 | 78 | md.reflect(bind=self.database.engine) | ||
945 | 79 | if "version" not in md.tables: | ||
946 | 80 | return None | ||
947 | 81 | Version = md.tables["version"] | ||
948 | 82 | last_version = self.database.store.query(Version.c.version).filter( | ||
949 | 83 | Version.c.component == "schema" | ||
950 | 84 | ).order_by(Version.c.version.desc()).first() | ||
951 | 85 | # Don't leave open transactions or they will block any schema change | ||
952 | 86 | self.database.commit() | ||
953 | 87 | return last_version | ||
954 | 88 | |||
955 | 89 | def _create(self): | ||
956 | 90 | # initial DB creation | ||
957 | 91 | Model.metadata.create_all(self.database.engine) | ||
958 | 92 | self.database.commit() | ||
959 | 93 | command.stamp(alembic_cfg, "head") | ||
960 | 94 | |||
961 | 95 | def _upgrade(self): | ||
962 | 96 | command.upgrade(alembic_cfg, "head") | ||
963 | 97 | |||
964 | 98 | def setup_db(self): | ||
965 | 99 | context = MigrationContext.configure(self.database.store.connection()) | ||
966 | 100 | current_rev = context.get_current_revision() | ||
967 | 101 | head_rev = self.script.get_current_head() | ||
968 | 102 | if current_rev == head_rev: | ||
969 | 103 | return head_rev # already at the latest revision, nothing to do | ||
970 | 104 | if current_rev == None: | ||
971 | 105 | # no alembic information | ||
972 | 106 | storm_version = self.get_storm_schema_version() | ||
973 | 107 | if storm_version is None: | ||
974 | 108 | # initial DB creation | ||
975 | 109 | self._create() | ||
976 | 110 | else: | ||
977 | 111 | # DB from a previous version managed by Storm | ||
978 | 112 | if storm_version.version < self.LAST_STORM_SCHEMA_VERSION: | ||
979 | 113 | raise RuntimeError( | ||
980 | 114 | "Upgrading while skipping beta version is " | ||
981 | 115 | "unsupported, please install the previous " | ||
982 | 116 | "Mailman beta release") | ||
983 | 117 | # Run migrations to remove the Storm-specific table and | ||
984 | 118 | # upgrade to SQLAlchemy & Alembic | ||
985 | 119 | self._upgrade() | ||
986 | 120 | elif current_rev != head_rev: | ||
987 | 121 | self._upgrade() | ||
988 | 122 | return head_rev | ||
989 | 123 | |||
990 | 124 | |||
991 | 125 | |||
992 | 64 | 126 | ||
993 | 65 | def _reset(self): | 127 | def _reset(self): |
994 | 66 | """See `IDatabase`.""" | 128 | """See `IDatabase`.""" |
996 | 67 | from mailman.database.model import ModelMeta | 129 | # Avoid a circular import at module level. |
997 | 130 | from mailman.database.model import Model | ||
998 | 68 | self.store.rollback() | 131 | self.store.rollback() |
999 | 69 | self._pre_reset(self.store) | 132 | self._pre_reset(self.store) |
1001 | 70 | ModelMeta._reset(self.store) | 133 | Model._reset(self) |
1002 | 71 | self._post_reset(self.store) | 134 | self._post_reset(self.store) |
1003 | 72 | self.store.commit() | 135 | self.store.commit() |
1004 | 73 | 136 | ||
1005 | @@ -81,24 +144,8 @@ | |||
1006 | 81 | database = call_name(database_class) | 144 | database = call_name(database_class) |
1007 | 82 | verifyObject(IDatabase, database) | 145 | verifyObject(IDatabase, database) |
1008 | 83 | database.initialize() | 146 | database.initialize() |
1010 | 84 | database.load_migrations() | 147 | Model.metadata.create_all(database.engine) |
1011 | 85 | database.commit() | 148 | database.commit() |
1012 | 86 | # Make _reset() a bound method of the database instance. | 149 | # Make _reset() a bound method of the database instance. |
1013 | 87 | database._reset = types.MethodType(_reset, database) | 150 | database._reset = types.MethodType(_reset, database) |
1014 | 88 | return database | 151 | return database |
1015 | 89 | |||
1016 | 90 | |||
1017 | 91 | |||
1018 | 92 | 152 | ||
1019 | 93 | @implementer(IDatabaseFactory) | ||
1020 | 94 | class DatabaseTemporaryFactory: | ||
1021 | 95 | """Create a temporary database for some of the migration tests.""" | ||
1022 | 96 | |||
1023 | 97 | @staticmethod | ||
1024 | 98 | def create(): | ||
1025 | 99 | """See `IDatabaseFactory`.""" | ||
1026 | 100 | database_class_name = config.database['class'] | ||
1027 | 101 | database = call_name(database_class_name) | ||
1028 | 102 | verifyObject(IDatabase, database) | ||
1029 | 103 | adapted_database = getAdapter( | ||
1030 | 104 | database, ITemporaryDatabase, database.TAG) | ||
1031 | 105 | return adapted_database | ||
1032 | 106 | 153 | ||
1033 | === modified file 'src/mailman/database/model.py' | |||
1034 | --- src/mailman/database/model.py 2014-01-01 14:59:42 +0000 | |||
1035 | +++ src/mailman/database/model.py 2014-10-11 02:14:38 +0000 | |||
1036 | @@ -25,44 +25,34 @@ | |||
1037 | 25 | ] | 25 | ] |
1038 | 26 | 26 | ||
1039 | 27 | 27 | ||
1040 | 28 | from operator import attrgetter | ||
1041 | 29 | |||
1042 | 30 | from storm.properties import PropertyPublisherMeta | ||
1043 | 31 | |||
1044 | 32 | |||
1045 | 33 | |||
1046 | 34 | 28 | ||
1067 | 35 | class ModelMeta(PropertyPublisherMeta): | 29 | import contextlib |
1068 | 36 | """Do more magic on table classes.""" | 30 | |
1069 | 37 | 31 | from sqlalchemy.ext.declarative import declarative_base | |
1070 | 38 | _class_registry = set() | 32 | |
1071 | 39 | 33 | from mailman.config import config | |
1072 | 40 | def __init__(self, name, bases, dict): | 34 | |
1073 | 41 | # Before we let the base class do it's thing, force an __storm_table__ | 35 | |
1074 | 42 | # property to enforce our table naming convention. | 36 | class ModelMeta: |
1075 | 43 | self.__storm_table__ = name.lower() | 37 | """The custom metaclass for all model base classes. |
1076 | 44 | super(ModelMeta, self).__init__(name, bases, dict) | 38 | |
1077 | 45 | # Register the model class so that it can be more easily cleared. | 39 | This is used in the test suite to quickly reset the database after each |
1078 | 46 | # This is required by the test framework so that the corresponding | 40 | test. It works by iterating over all the tables, deleting each. The test |
1079 | 47 | # table can be reset between tests. | 41 | suite will then recreate the tables before each test. |
1080 | 48 | # | 42 | """ |
1061 | 49 | # The PRESERVE flag indicates whether the table should be reset or | ||
1062 | 50 | # not. We have to handle the actual Model base class explicitly | ||
1063 | 51 | # because it does not correspond to a table in the database. | ||
1064 | 52 | if not getattr(self, 'PRESERVE', False) and name != 'Model': | ||
1065 | 53 | ModelMeta._class_registry.add(self) | ||
1066 | 54 | |||
1081 | 55 | @staticmethod | 43 | @staticmethod |
1082 | 56 | def _reset(store): | ||
1083 | 57 | from mailman.config import config | ||
1084 | 58 | config.db._pre_reset(store) | ||
1085 | 59 | # Make sure this is deterministic, by sorting on the storm table name. | ||
1086 | 60 | classes = sorted(ModelMeta._class_registry, | ||
1087 | 61 | key=attrgetter('__storm_table__')) | ||
1088 | 62 | for model_class in classes: | ||
1089 | 63 | store.find(model_class).remove() | ||
1090 | 64 | |||
1091 | 65 | |||
1092 | 66 | |||
1093 | 67 | 44 | ||
1097 | 68 | class Model: | 45 | def _reset(db): |
1098 | 69 | """Like Storm's `Storm` subclass, but with a bit extra.""" | 46 | with contextlib.closing(config.db.engine.connect()) as connection: |
1099 | 70 | __metaclass__ = ModelMeta | 47 | transaction = connection.begin() |
1100 | 48 | try: | ||
1101 | 49 | # Delete all the tables in reverse foreign key dependency | ||
1102 | 50 | # order. http://tinyurl.com/on8dy6f | ||
1103 | 51 | for table in reversed(Model.metadata.sorted_tables): | ||
1104 | 52 | connection.execute(table.delete()) | ||
1105 | 53 | except: | ||
1106 | 54 | transaction.rollback() | ||
1107 | 55 | raise | ||
1108 | 56 | else: | ||
1109 | 57 | transaction.commit() | ||
1110 | 58 | |||
1111 | 59 | |||
1112 | 60 | Model = declarative_base(cls=ModelMeta) | ||
1113 | 71 | 61 | ||
1114 | === modified file 'src/mailman/database/postgresql.py' | |||
1115 | --- src/mailman/database/postgresql.py 2014-01-01 14:59:42 +0000 | |||
1116 | +++ src/mailman/database/postgresql.py 2014-10-11 02:14:38 +0000 | |||
1117 | @@ -22,34 +22,17 @@ | |||
1118 | 22 | __metaclass__ = type | 22 | __metaclass__ = type |
1119 | 23 | __all__ = [ | 23 | __all__ = [ |
1120 | 24 | 'PostgreSQLDatabase', | 24 | 'PostgreSQLDatabase', |
1121 | 25 | 'make_temporary', | ||
1122 | 26 | ] | 25 | ] |
1123 | 27 | 26 | ||
1124 | 28 | 27 | ||
1133 | 29 | import types | 28 | from mailman.database.base import SABaseDatabase |
1134 | 30 | 29 | from mailman.database.model import Model | |
1127 | 31 | from functools import partial | ||
1128 | 32 | from operator import attrgetter | ||
1129 | 33 | from urlparse import urlsplit, urlunsplit | ||
1130 | 34 | |||
1131 | 35 | from mailman.database.base import StormBaseDatabase | ||
1132 | 36 | from mailman.testing.helpers import configuration | ||
1135 | 37 | 30 | ||
1136 | 38 | 31 | ||
1137 | 39 | 32 | ||
1138 | 40 | 33 | ||
1140 | 41 | class PostgreSQLDatabase(StormBaseDatabase): | 34 | class PostgreSQLDatabase(SABaseDatabase): |
1141 | 42 | """Database class for PostgreSQL.""" | 35 | """Database class for PostgreSQL.""" |
1142 | 43 | 36 | ||
1143 | 44 | TAG = 'postgres' | ||
1144 | 45 | |||
1145 | 46 | def _database_exists(self, store): | ||
1146 | 47 | """See `BaseDatabase`.""" | ||
1147 | 48 | table_query = ('SELECT table_name FROM information_schema.tables ' | ||
1148 | 49 | "WHERE table_schema = 'public'") | ||
1149 | 50 | results = store.execute(table_query) | ||
1150 | 51 | table_names = set(item[0] for item in results) | ||
1151 | 52 | return 'version' in table_names | ||
1152 | 53 | |||
1153 | 54 | def _post_reset(self, store): | 37 | def _post_reset(self, store): |
1154 | 55 | """PostgreSQL-specific test suite cleanup. | 38 | """PostgreSQL-specific test suite cleanup. |
1155 | 56 | 39 | ||
1156 | @@ -57,49 +40,13 @@ | |||
1157 | 57 | restart from zero for new tests. | 40 | restart from zero for new tests. |
1158 | 58 | """ | 41 | """ |
1159 | 59 | super(PostgreSQLDatabase, self)._post_reset(store) | 42 | super(PostgreSQLDatabase, self)._post_reset(store) |
1163 | 60 | from mailman.database.model import ModelMeta | 43 | tables = reversed(Model.metadata.sorted_tables) |
1161 | 61 | classes = sorted(ModelMeta._class_registry, | ||
1162 | 62 | key=attrgetter('__storm_table__')) | ||
1164 | 63 | # Recipe adapted from | 44 | # Recipe adapted from |
1165 | 64 | # http://stackoverflow.com/questions/544791/ | 45 | # http://stackoverflow.com/questions/544791/ |
1166 | 65 | # django-postgresql-how-to-reset-primary-key | 46 | # django-postgresql-how-to-reset-primary-key |
1168 | 66 | for model_class in classes: | 47 | for table in tables: |
1169 | 67 | store.execute("""\ | 48 | store.execute("""\ |
1170 | 68 | SELECT setval('"{0}_id_seq"', coalesce(max("id"), 1), | 49 | SELECT setval('"{0}_id_seq"', coalesce(max("id"), 1), |
1171 | 69 | max("id") IS NOT null) | 50 | max("id") IS NOT null) |
1172 | 70 | FROM "{0}"; | 51 | FROM "{0}"; |
1173 | 71 | """.format(model_class.__storm_table__)) | ||
1174 | 72 | |||
1175 | 73 | |||
1176 | 74 | |||
1177 | 75 | 52 | ||
1209 | 76 | # Test suite adapter for ITemporaryDatabase. | 53 | """.format(table)) |
1179 | 77 | |||
1180 | 78 | def _cleanup(self, store, tempdb_name): | ||
1181 | 79 | from mailman.config import config | ||
1182 | 80 | store.rollback() | ||
1183 | 81 | store.close() | ||
1184 | 82 | # From the original database connection, drop the now unused database. | ||
1185 | 83 | config.db.store.execute('DROP DATABASE {0}'.format(tempdb_name)) | ||
1186 | 84 | |||
1187 | 85 | |||
1188 | 86 | def make_temporary(database): | ||
1189 | 87 | """Adapts by monkey patching an existing PostgreSQL IDatabase.""" | ||
1190 | 88 | from mailman.config import config | ||
1191 | 89 | parts = urlsplit(config.database.url) | ||
1192 | 90 | assert parts.scheme == 'postgres' | ||
1193 | 91 | new_parts = list(parts) | ||
1194 | 92 | new_parts[2] = '/mmtest' | ||
1195 | 93 | url = urlunsplit(new_parts) | ||
1196 | 94 | # Use the existing database connection to create a new testing | ||
1197 | 95 | # database. | ||
1198 | 96 | config.db.store.execute('ABORT;') | ||
1199 | 97 | config.db.store.execute('CREATE DATABASE mmtest;') | ||
1200 | 98 | with configuration('database', url=url): | ||
1201 | 99 | database.initialize() | ||
1202 | 100 | database._cleanup = types.MethodType( | ||
1203 | 101 | partial(_cleanup, store=database.store, tempdb_name='mmtest'), | ||
1204 | 102 | database) | ||
1205 | 103 | # bool column values in PostgreSQL. | ||
1206 | 104 | database.FALSE = 'False' | ||
1207 | 105 | database.TRUE = 'True' | ||
1208 | 106 | return database | ||
1210 | 107 | 54 | ||
1211 | === removed directory 'src/mailman/database/schema' | |||
1212 | === removed file 'src/mailman/database/schema/__init__.py' | |||
1213 | === removed file 'src/mailman/database/schema/helpers.py' | |||
1214 | --- src/mailman/database/schema/helpers.py 2014-01-01 14:59:42 +0000 | |||
1215 | +++ src/mailman/database/schema/helpers.py 1970-01-01 00:00:00 +0000 | |||
1216 | @@ -1,43 +0,0 @@ | |||
1217 | 1 | # Copyright (C) 2013-2014 by the Free Software Foundation, Inc. | ||
1218 | 2 | # | ||
1219 | 3 | # This file is part of GNU Mailman. | ||
1220 | 4 | # | ||
1221 | 5 | # GNU Mailman is free software: you can redistribute it and/or modify it under | ||
1222 | 6 | # the terms of the GNU General Public License as published by the Free | ||
1223 | 7 | # Software Foundation, either version 3 of the License, or (at your option) | ||
1224 | 8 | # any later version. | ||
1225 | 9 | # | ||
1226 | 10 | # GNU Mailman is distributed in the hope that it will be useful, but WITHOUT | ||
1227 | 11 | # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or | ||
1228 | 12 | # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for | ||
1229 | 13 | # more details. | ||
1230 | 14 | # | ||
1231 | 15 | # You should have received a copy of the GNU General Public License along with | ||
1232 | 16 | # GNU Mailman. If not, see <http://www.gnu.org/licenses/>. | ||
1233 | 17 | |||
1234 | 18 | """Schema migration helpers.""" | ||
1235 | 19 | |||
1236 | 20 | from __future__ import absolute_import, print_function, unicode_literals | ||
1237 | 21 | |||
1238 | 22 | __metaclass__ = type | ||
1239 | 23 | __all__ = [ | ||
1240 | 24 | 'make_listid', | ||
1241 | 25 | ] | ||
1242 | 26 | |||
1243 | 27 | |||
1244 | 28 | |||
1245 | 29 | 0 | ||
1246 | 30 | def make_listid(fqdn_listname): | ||
1247 | 31 | """Turn a FQDN list name into a List-ID.""" | ||
1248 | 32 | list_name, at, mail_host = fqdn_listname.partition('@') | ||
1249 | 33 | if at == '': | ||
1250 | 34 | # If there is no @ sign in the value, assume it already contains the | ||
1251 | 35 | # list-id. | ||
1252 | 36 | return fqdn_listname | ||
1253 | 37 | return '{0}.{1}'.format(list_name, mail_host) | ||
1254 | 38 | |||
1255 | 39 | |||
1256 | 40 | |||
1257 | 41 | 1 | ||
1258 | 42 | def pivot(store, table_name): | ||
1259 | 43 | """Pivot a backup table into the real table name.""" | ||
1260 | 44 | store.execute('DROP TABLE {}'.format(table_name)) | ||
1261 | 45 | store.execute('ALTER TABLE {0}_backup RENAME TO {0}'.format(table_name)) | ||
1262 | 46 | 2 | ||
1263 | === removed file 'src/mailman/database/schema/mm_00000000000000_base.py' | |||
1264 | --- src/mailman/database/schema/mm_00000000000000_base.py 2014-01-01 14:59:42 +0000 | |||
1265 | +++ src/mailman/database/schema/mm_00000000000000_base.py 1970-01-01 00:00:00 +0000 | |||
1266 | @@ -1,35 +0,0 @@ | |||
1267 | 1 | # Copyright (C) 2012-2014 by the Free Software Foundation, Inc. | ||
1268 | 2 | # | ||
1269 | 3 | # This file is part of GNU Mailman. | ||
1270 | 4 | # | ||
1271 | 5 | # GNU Mailman is free software: you can redistribute it and/or modify it under | ||
1272 | 6 | # the terms of the GNU General Public License as published by the Free | ||
1273 | 7 | # Software Foundation, either version 3 of the License, or (at your option) | ||
1274 | 8 | # any later version. | ||
1275 | 9 | # | ||
1276 | 10 | # GNU Mailman is distributed in the hope that it will be useful, but WITHOUT | ||
1277 | 11 | # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or | ||
1278 | 12 | # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for | ||
1279 | 13 | # more details. | ||
1280 | 14 | # | ||
1281 | 15 | # You should have received a copy of the GNU General Public License along with | ||
1282 | 16 | # GNU Mailman. If not, see <http://www.gnu.org/licenses/>. | ||
1283 | 17 | |||
1284 | 18 | """Load the base schema.""" | ||
1285 | 19 | |||
1286 | 20 | from __future__ import absolute_import, print_function, unicode_literals | ||
1287 | 21 | |||
1288 | 22 | __metaclass__ = type | ||
1289 | 23 | __all__ = [ | ||
1290 | 24 | 'upgrade', | ||
1291 | 25 | ] | ||
1292 | 26 | |||
1293 | 27 | |||
1294 | 28 | VERSION = '00000000000000' | ||
1295 | 29 | _helper = None | ||
1296 | 30 | |||
1297 | 31 | |||
1298 | 32 | |||
1299 | 33 | 0 | ||
1300 | 34 | def upgrade(database, store, version, module_path): | ||
1301 | 35 | filename = '{0}.sql'.format(database.TAG) | ||
1302 | 36 | database.load_schema(store, version, filename, module_path) | ||
1303 | 37 | 1 | ||
1304 | === removed file 'src/mailman/database/schema/mm_20120407000000.py' | |||
1305 | --- src/mailman/database/schema/mm_20120407000000.py 2014-01-01 14:59:42 +0000 | |||
1306 | +++ src/mailman/database/schema/mm_20120407000000.py 1970-01-01 00:00:00 +0000 | |||
1307 | @@ -1,212 +0,0 @@ | |||
1308 | 1 | # Copyright (C) 2012-2014 by the Free Software Foundation, Inc. | ||
1309 | 2 | # | ||
1310 | 3 | # This file is part of GNU Mailman. | ||
1311 | 4 | # | ||
1312 | 5 | # GNU Mailman is free software: you can redistribute it and/or modify it under | ||
1313 | 6 | # the terms of the GNU General Public License as published by the Free | ||
1314 | 7 | # Software Foundation, either version 3 of the License, or (at your option) | ||
1315 | 8 | # any later version. | ||
1316 | 9 | # | ||
1317 | 10 | # GNU Mailman is distributed in the hope that it will be useful, but WITHOUT | ||
1318 | 11 | # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or | ||
1319 | 12 | # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for | ||
1320 | 13 | # more details. | ||
1321 | 14 | # | ||
1322 | 15 | # You should have received a copy of the GNU General Public License along with | ||
1323 | 16 | # GNU Mailman. If not, see <http://www.gnu.org/licenses/>. | ||
1324 | 17 | |||
1325 | 18 | """3.0b1 -> 3.0b2 schema migrations. | ||
1326 | 19 | |||
1327 | 20 | All column changes are in the `mailinglist` table. | ||
1328 | 21 | |||
1329 | 22 | * Renames: | ||
1330 | 23 | - news_prefix_subject_too -> nntp_prefix_subject_too | ||
1331 | 24 | - news_moderation -> newsgroup_moderation | ||
1332 | 25 | |||
1333 | 26 | * Collapsing: | ||
1334 | 27 | - archive, archive_private -> archive_policy | ||
1335 | 28 | |||
1336 | 29 | * Remove: | ||
1337 | 30 | - archive_volume_frequency | ||
1338 | 31 | - generic_nonmember_action | ||
1339 | 32 | - nntp_host | ||
1340 | 33 | |||
1341 | 34 | * Added: | ||
1342 | 35 | - list_id | ||
1343 | 36 | |||
1344 | 37 | * Changes: | ||
1345 | 38 | member.mailing_list holds the list_id not the fqdn_listname | ||
1346 | 39 | |||
1347 | 40 | See https://bugs.launchpad.net/mailman/+bug/971013 for details. | ||
1348 | 41 | """ | ||
1349 | 42 | |||
1350 | 43 | from __future__ import absolute_import, print_function, unicode_literals | ||
1351 | 44 | |||
1352 | 45 | __metaclass__ = type | ||
1353 | 46 | __all__ = [ | ||
1354 | 47 | 'upgrade', | ||
1355 | 48 | ] | ||
1356 | 49 | |||
1357 | 50 | |||
1358 | 51 | from mailman.database.schema.helpers import pivot | ||
1359 | 52 | from mailman.interfaces.archiver import ArchivePolicy | ||
1360 | 53 | |||
1361 | 54 | |||
1362 | 55 | VERSION = '20120407000000' | ||
1363 | 56 | |||
1364 | 57 | |||
1365 | 58 | |||
1366 | 59 | 0 | ||
1367 | 60 | def upgrade(database, store, version, module_path): | ||
1368 | 61 | if database.TAG == 'sqlite': | ||
1369 | 62 | upgrade_sqlite(database, store, version, module_path) | ||
1370 | 63 | else: | ||
1371 | 64 | upgrade_postgres(database, store, version, module_path) | ||
1372 | 65 | |||
1373 | 66 | |||
1374 | 67 | |||
1375 | 68 | 1 | ||
1376 | 69 | def archive_policy(archive, archive_private): | ||
1377 | 70 | """Convert archive and archive_private to archive_policy.""" | ||
1378 | 71 | if archive == 0: | ||
1379 | 72 | return ArchivePolicy.never.value | ||
1380 | 73 | elif archive_private == 1: | ||
1381 | 74 | return ArchivePolicy.private.value | ||
1382 | 75 | else: | ||
1383 | 76 | return ArchivePolicy.public.value | ||
1384 | 77 | |||
1385 | 78 | |||
1386 | 79 | |||
1387 | 80 | 2 | ||
1388 | 81 | def upgrade_sqlite(database, store, version, module_path): | ||
1389 | 82 | # Load the first part of the migration. This creates a temporary table to | ||
1390 | 83 | # hold the new mailinglist table columns. The problem is that some of the | ||
1391 | 84 | # changes must be performed in Python, so after the first part is loaded, | ||
1392 | 85 | # we do the Python changes, drop the old mailing list table, and then | ||
1393 | 86 | # rename the temporary table to its place. | ||
1394 | 87 | database.load_schema( | ||
1395 | 88 | store, version, 'sqlite_{0}_01.sql'.format(version), module_path) | ||
1396 | 89 | results = store.execute(""" | ||
1397 | 90 | SELECT id, include_list_post_header, | ||
1398 | 91 | news_prefix_subject_too, news_moderation, | ||
1399 | 92 | archive, archive_private, list_name, mail_host | ||
1400 | 93 | FROM mailinglist; | ||
1401 | 94 | """) | ||
1402 | 95 | for value in results: | ||
1403 | 96 | (id, list_post, | ||
1404 | 97 | news_prefix, news_moderation, | ||
1405 | 98 | archive, archive_private, | ||
1406 | 99 | list_name, mail_host) = value | ||
1407 | 100 | # Figure out what the new archive_policy column value should be. | ||
1408 | 101 | list_id = '{0}.{1}'.format(list_name, mail_host) | ||
1409 | 102 | fqdn_listname = '{0}@{1}'.format(list_name, mail_host) | ||
1410 | 103 | store.execute(""" | ||
1411 | 104 | UPDATE mailinglist_backup SET | ||
1412 | 105 | allow_list_posts = {0}, | ||
1413 | 106 | newsgroup_moderation = {1}, | ||
1414 | 107 | nntp_prefix_subject_too = {2}, | ||
1415 | 108 | archive_policy = {3}, | ||
1416 | 109 | list_id = '{4}' | ||
1417 | 110 | WHERE id = {5}; | ||
1418 | 111 | """.format( | ||
1419 | 112 | list_post, | ||
1420 | 113 | news_moderation, | ||
1421 | 114 | news_prefix, | ||
1422 | 115 | archive_policy(archive, archive_private), | ||
1423 | 116 | list_id, | ||
1424 | 117 | id)) | ||
1425 | 118 | # Also update the member.mailing_list column to hold the list_id | ||
1426 | 119 | # instead of the fqdn_listname. | ||
1427 | 120 | store.execute(""" | ||
1428 | 121 | UPDATE member SET | ||
1429 | 122 | mailing_list = '{0}' | ||
1430 | 123 | WHERE mailing_list = '{1}'; | ||
1431 | 124 | """.format(list_id, fqdn_listname)) | ||
1432 | 125 | # Pivot the backup table to the real thing. | ||
1433 | 126 | pivot(store, 'mailinglist') | ||
1434 | 127 | # Now add some indexes that were previously missing. | ||
1435 | 128 | store.execute( | ||
1436 | 129 | 'CREATE INDEX ix_mailinglist_list_id ON mailinglist (list_id);') | ||
1437 | 130 | store.execute( | ||
1438 | 131 | 'CREATE INDEX ix_mailinglist_fqdn_listname ' | ||
1439 | 132 | 'ON mailinglist (list_name, mail_host);') | ||
1440 | 133 | # Now, do the member table. | ||
1441 | 134 | results = store.execute('SELECT id, mailing_list FROM member;') | ||
1442 | 135 | for id, mailing_list in results: | ||
1443 | 136 | list_name, at, mail_host = mailing_list.partition('@') | ||
1444 | 137 | if at == '': | ||
1445 | 138 | list_id = mailing_list | ||
1446 | 139 | else: | ||
1447 | 140 | list_id = '{0}.{1}'.format(list_name, mail_host) | ||
1448 | 141 | store.execute(""" | ||
1449 | 142 | UPDATE member_backup SET list_id = '{0}' | ||
1450 | 143 | WHERE id = {1}; | ||
1451 | 144 | """.format(list_id, id)) | ||
1452 | 145 | # Pivot the backup table to the real thing. | ||
1453 | 146 | pivot(store, 'member') | ||
1454 | 147 | |||
1455 | 148 | |||
1456 | 149 | |||
1457 | 150 | 3 | ||
1458 | 151 | def upgrade_postgres(database, store, version, module_path): | ||
1459 | 152 | # Get the old values from the mailinglist table. | ||
1460 | 153 | results = store.execute(""" | ||
1461 | 154 | SELECT id, archive, archive_private, list_name, mail_host | ||
1462 | 155 | FROM mailinglist; | ||
1463 | 156 | """) | ||
1464 | 157 | # Do the simple renames first. | ||
1465 | 158 | store.execute(""" | ||
1466 | 159 | ALTER TABLE mailinglist | ||
1467 | 160 | RENAME COLUMN news_prefix_subject_too TO nntp_prefix_subject_too; | ||
1468 | 161 | """) | ||
1469 | 162 | store.execute(""" | ||
1470 | 163 | ALTER TABLE mailinglist | ||
1471 | 164 | RENAME COLUMN news_moderation TO newsgroup_moderation; | ||
1472 | 165 | """) | ||
1473 | 166 | store.execute(""" | ||
1474 | 167 | ALTER TABLE mailinglist | ||
1475 | 168 | RENAME COLUMN include_list_post_header TO allow_list_posts; | ||
1476 | 169 | """) | ||
1477 | 170 | # Do the easy column drops next. | ||
1478 | 171 | for column in ('archive_volume_frequency', | ||
1479 | 172 | 'generic_nonmember_action', | ||
1480 | 173 | 'nntp_host'): | ||
1481 | 174 | store.execute( | ||
1482 | 175 | 'ALTER TABLE mailinglist DROP COLUMN {0};'.format(column)) | ||
1483 | 176 | # Now do the trickier collapsing of values. Add the new columns. | ||
1484 | 177 | store.execute('ALTER TABLE mailinglist ADD COLUMN archive_policy INTEGER;') | ||
1485 | 178 | store.execute('ALTER TABLE mailinglist ADD COLUMN list_id TEXT;') | ||
1486 | 179 | # Query the database for the old values of archive and archive_private in | ||
1487 | 180 | # each column. Then loop through all the results and update the new | ||
1488 | 181 | # archive_policy from the old values. | ||
1489 | 182 | for value in results: | ||
1490 | 183 | id, archive, archive_private, list_name, mail_host = value | ||
1491 | 184 | list_id = '{0}.{1}'.format(list_name, mail_host) | ||
1492 | 185 | store.execute(""" | ||
1493 | 186 | UPDATE mailinglist SET | ||
1494 | 187 | archive_policy = {0}, | ||
1495 | 188 | list_id = '{1}' | ||
1496 | 189 | WHERE id = {2}; | ||
1497 | 190 | """.format(archive_policy(archive, archive_private), list_id, id)) | ||
1498 | 191 | # Now drop the old columns. | ||
1499 | 192 | for column in ('archive', 'archive_private'): | ||
1500 | 193 | store.execute( | ||
1501 | 194 | 'ALTER TABLE mailinglist DROP COLUMN {0};'.format(column)) | ||
1502 | 195 | # Now add some indexes that were previously missing. | ||
1503 | 196 | store.execute( | ||
1504 | 197 | 'CREATE INDEX ix_mailinglist_list_id ON mailinglist (list_id);') | ||
1505 | 198 | store.execute( | ||
1506 | 199 | 'CREATE INDEX ix_mailinglist_fqdn_listname ' | ||
1507 | 200 | 'ON mailinglist (list_name, mail_host);') | ||
1508 | 201 | # Now, do the member table. | ||
1509 | 202 | results = store.execute('SELECT id, mailing_list FROM member;') | ||
1510 | 203 | store.execute('ALTER TABLE member ADD COLUMN list_id TEXT;') | ||
1511 | 204 | for id, mailing_list in results: | ||
1512 | 205 | list_name, at, mail_host = mailing_list.partition('@') | ||
1513 | 206 | if at == '': | ||
1514 | 207 | list_id = mailing_list | ||
1515 | 208 | else: | ||
1516 | 209 | list_id = '{0}.{1}'.format(list_name, mail_host) | ||
1517 | 210 | store.execute(""" | ||
1518 | 211 | UPDATE member SET list_id = '{0}' | ||
1519 | 212 | WHERE id = {1}; | ||
1520 | 213 | """.format(list_id, id)) | ||
1521 | 214 | store.execute('ALTER TABLE member DROP COLUMN mailing_list;') | ||
1522 | 215 | # Record the migration in the version table. | ||
1523 | 216 | database.load_schema(store, version, None, module_path) | ||
1524 | 217 | 4 | ||
1525 | === removed file 'src/mailman/database/schema/mm_20121015000000.py' | |||
1526 | --- src/mailman/database/schema/mm_20121015000000.py 2014-01-01 14:59:42 +0000 | |||
1527 | +++ src/mailman/database/schema/mm_20121015000000.py 1970-01-01 00:00:00 +0000 | |||
1528 | @@ -1,95 +0,0 @@ | |||
1529 | 1 | # Copyright (C) 2012-2014 by the Free Software Foundation, Inc. | ||
1530 | 2 | # | ||
1531 | 3 | # This file is part of GNU Mailman. | ||
1532 | 4 | # | ||
1533 | 5 | # GNU Mailman is free software: you can redistribute it and/or modify it under | ||
1534 | 6 | # the terms of the GNU General Public License as published by the Free | ||
1535 | 7 | # Software Foundation, either version 3 of the License, or (at your option) | ||
1536 | 8 | # any later version. | ||
1537 | 9 | # | ||
1538 | 10 | # GNU Mailman is distributed in the hope that it will be useful, but WITHOUT | ||
1539 | 11 | # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or | ||
1540 | 12 | # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for | ||
1541 | 13 | # more details. | ||
1542 | 14 | # | ||
1543 | 15 | # You should have received a copy of the GNU General Public License along with | ||
1544 | 16 | # GNU Mailman. If not, see <http://www.gnu.org/licenses/>. | ||
1545 | 17 | |||
1546 | 18 | """3.0b2 -> 3.0b3 schema migrations. | ||
1547 | 19 | |||
1548 | 20 | Renamed: | ||
1549 | 21 | * bans.mailing_list -> bans.list_id | ||
1550 | 22 | |||
1551 | 23 | Removed: | ||
1552 | 24 | * mailinglist.new_member_options | ||
1553 | 25 | * mailinglist.send_remindersn | ||
1554 | 26 | """ | ||
1555 | 27 | |||
1556 | 28 | from __future__ import absolute_import, print_function, unicode_literals | ||
1557 | 29 | |||
1558 | 30 | __metaclass__ = type | ||
1559 | 31 | __all__ = [ | ||
1560 | 32 | 'upgrade', | ||
1561 | 33 | ] | ||
1562 | 34 | |||
1563 | 35 | |||
1564 | 36 | from mailman.database.schema.helpers import make_listid, pivot | ||
1565 | 37 | |||
1566 | 38 | |||
1567 | 39 | VERSION = '20121015000000' | ||
1568 | 40 | |||
1569 | 41 | |||
1570 | 42 | |||
1571 | 43 | 0 | ||
1572 | 44 | def upgrade(database, store, version, module_path): | ||
1573 | 45 | if database.TAG == 'sqlite': | ||
1574 | 46 | upgrade_sqlite(database, store, version, module_path) | ||
1575 | 47 | else: | ||
1576 | 48 | upgrade_postgres(database, store, version, module_path) | ||
1577 | 49 | |||
1578 | 50 | |||
1579 | 51 | |||
1580 | 52 | 1 | ||
1581 | 53 | def upgrade_sqlite(database, store, version, module_path): | ||
1582 | 54 | database.load_schema( | ||
1583 | 55 | store, version, 'sqlite_{}_01.sql'.format(version), module_path) | ||
1584 | 56 | results = store.execute(""" | ||
1585 | 57 | SELECT id, mailing_list | ||
1586 | 58 | FROM ban; | ||
1587 | 59 | """) | ||
1588 | 60 | for id, mailing_list in results: | ||
1589 | 61 | # Skip global bans since there's nothing to update. | ||
1590 | 62 | if mailing_list is None: | ||
1591 | 63 | continue | ||
1592 | 64 | store.execute(""" | ||
1593 | 65 | UPDATE ban_backup SET list_id = '{}' | ||
1594 | 66 | WHERE id = {}; | ||
1595 | 67 | """.format(make_listid(mailing_list), id)) | ||
1596 | 68 | # Pivot the bans backup table to the real thing. | ||
1597 | 69 | pivot(store, 'ban') | ||
1598 | 70 | pivot(store, 'mailinglist') | ||
1599 | 71 | |||
1600 | 72 | |||
1601 | 73 | |||
1602 | 74 | 2 | ||
1603 | 75 | def upgrade_postgres(database, store, version, module_path): | ||
1604 | 76 | # Get the old values from the ban table. | ||
1605 | 77 | results = store.execute('SELECT id, mailing_list FROM ban;') | ||
1606 | 78 | store.execute('ALTER TABLE ban ADD COLUMN list_id TEXT;') | ||
1607 | 79 | for id, mailing_list in results: | ||
1608 | 80 | # Skip global bans since there's nothing to update. | ||
1609 | 81 | if mailing_list is None: | ||
1610 | 82 | continue | ||
1611 | 83 | store.execute(""" | ||
1612 | 84 | UPDATE ban SET list_id = '{0}' | ||
1613 | 85 | WHERE id = {1}; | ||
1614 | 86 | """.format(make_listid(mailing_list), id)) | ||
1615 | 87 | store.execute('ALTER TABLE ban DROP COLUMN mailing_list;') | ||
1616 | 88 | store.execute('ALTER TABLE mailinglist DROP COLUMN new_member_options;') | ||
1617 | 89 | store.execute('ALTER TABLE mailinglist DROP COLUMN send_reminders;') | ||
1618 | 90 | store.execute('ALTER TABLE mailinglist DROP COLUMN subscribe_policy;') | ||
1619 | 91 | store.execute('ALTER TABLE mailinglist DROP COLUMN unsubscribe_policy;') | ||
1620 | 92 | store.execute( | ||
1621 | 93 | 'ALTER TABLE mailinglist DROP COLUMN subscribe_auto_approval;') | ||
1622 | 94 | store.execute('ALTER TABLE mailinglist DROP COLUMN private_roster;') | ||
1623 | 95 | store.execute( | ||
1624 | 96 | 'ALTER TABLE mailinglist DROP COLUMN admin_member_chunksize;') | ||
1625 | 97 | # Record the migration in the version table. | ||
1626 | 98 | database.load_schema(store, version, None, module_path) | ||
1627 | 99 | 3 | ||
1628 | === removed file 'src/mailman/database/schema/mm_20130406000000.py' | |||
1629 | --- src/mailman/database/schema/mm_20130406000000.py 2014-01-01 14:59:42 +0000 | |||
1630 | +++ src/mailman/database/schema/mm_20130406000000.py 1970-01-01 00:00:00 +0000 | |||
1631 | @@ -1,65 +0,0 @@ | |||
1632 | 1 | # Copyright (C) 2013-2014 by the Free Software Foundation, Inc. | ||
1633 | 2 | # | ||
1634 | 3 | # This file is part of GNU Mailman. | ||
1635 | 4 | # | ||
1636 | 5 | # GNU Mailman is free software: you can redistribute it and/or modify it under | ||
1637 | 6 | # the terms of the GNU General Public License as published by the Free | ||
1638 | 7 | # Software Foundation, either version 3 of the License, or (at your option) | ||
1639 | 8 | # any later version. | ||
1640 | 9 | # | ||
1641 | 10 | # GNU Mailman is distributed in the hope that it will be useful, but WITHOUT | ||
1642 | 11 | # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or | ||
1643 | 12 | # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for | ||
1644 | 13 | # more details. | ||
1645 | 14 | # | ||
1646 | 15 | # You should have received a copy of the GNU General Public License along with | ||
1647 | 16 | # GNU Mailman. If not, see <http://www.gnu.org/licenses/>. | ||
1648 | 17 | |||
1649 | 18 | """3.0b3 -> 3.0b4 schema migrations. | ||
1650 | 19 | |||
1651 | 20 | Renamed: | ||
1652 | 21 | * bounceevent.list_name -> bounceevent.list_id | ||
1653 | 22 | """ | ||
1654 | 23 | |||
1655 | 24 | |||
1656 | 25 | from __future__ import absolute_import, print_function, unicode_literals | ||
1657 | 26 | |||
1658 | 27 | __metaclass__ = type | ||
1659 | 28 | __all__ = [ | ||
1660 | 29 | 'upgrade' | ||
1661 | 30 | ] | ||
1662 | 31 | |||
1663 | 32 | |||
1664 | 33 | from mailman.database.schema.helpers import make_listid, pivot | ||
1665 | 34 | |||
1666 | 35 | |||
1667 | 36 | VERSION = '20130406000000' | ||
1668 | 37 | |||
1669 | 38 | |||
1670 | 39 | |||
1671 | 40 | 0 | ||
1672 | 41 | def upgrade(database, store, version, module_path): | ||
1673 | 42 | if database.TAG == 'sqlite': | ||
1674 | 43 | upgrade_sqlite(database, store, version, module_path) | ||
1675 | 44 | else: | ||
1676 | 45 | upgrade_postgres(database, store, version, module_path) | ||
1677 | 46 | |||
1678 | 47 | |||
1679 | 48 | |||
1680 | 49 | 1 | ||
1681 | 50 | def upgrade_sqlite(database, store, version, module_path): | ||
1682 | 51 | database.load_schema( | ||
1683 | 52 | store, version, 'sqlite_{}_01.sql'.format(version), module_path) | ||
1684 | 53 | results = store.execute(""" | ||
1685 | 54 | SELECT id, list_name | ||
1686 | 55 | FROM bounceevent; | ||
1687 | 56 | """) | ||
1688 | 57 | for id, list_name in results: | ||
1689 | 58 | store.execute(""" | ||
1690 | 59 | UPDATE bounceevent_backup SET list_id = '{}' | ||
1691 | 60 | WHERE id = {}; | ||
1692 | 61 | """.format(make_listid(list_name), id)) | ||
1693 | 62 | pivot(store, 'bounceevent') | ||
1694 | 63 | |||
1695 | 64 | |||
1696 | 65 | |||
1697 | 66 | 2 | ||
1698 | 67 | def upgrade_postgres(database, store, version, module_path): | ||
1699 | 68 | pass | ||
1700 | 69 | 3 | ||
1701 | === removed file 'src/mailman/database/schema/postgres.sql' | |||
1702 | --- src/mailman/database/schema/postgres.sql 2012-07-23 14:40:53 +0000 | |||
1703 | +++ src/mailman/database/schema/postgres.sql 1970-01-01 00:00:00 +0000 | |||
1704 | @@ -1,349 +0,0 @@ | |||
1705 | 1 | CREATE TABLE mailinglist ( | ||
1706 | 2 | id SERIAL NOT NULL, | ||
1707 | 3 | -- List identity | ||
1708 | 4 | list_name TEXT, | ||
1709 | 5 | mail_host TEXT, | ||
1710 | 6 | include_list_post_header BOOLEAN, | ||
1711 | 7 | include_rfc2369_headers BOOLEAN, | ||
1712 | 8 | -- Attributes not directly modifiable via the web u/i | ||
1713 | 9 | created_at TIMESTAMP, | ||
1714 | 10 | admin_member_chunksize INTEGER, | ||
1715 | 11 | next_request_id INTEGER, | ||
1716 | 12 | next_digest_number INTEGER, | ||
1717 | 13 | digest_last_sent_at TIMESTAMP, | ||
1718 | 14 | volume INTEGER, | ||
1719 | 15 | last_post_at TIMESTAMP, | ||
1720 | 16 | accept_these_nonmembers BYTEA, | ||
1721 | 17 | acceptable_aliases_id INTEGER, | ||
1722 | 18 | admin_immed_notify BOOLEAN, | ||
1723 | 19 | admin_notify_mchanges BOOLEAN, | ||
1724 | 20 | administrivia BOOLEAN, | ||
1725 | 21 | advertised BOOLEAN, | ||
1726 | 22 | anonymous_list BOOLEAN, | ||
1727 | 23 | archive BOOLEAN, | ||
1728 | 24 | archive_private BOOLEAN, | ||
1729 | 25 | archive_volume_frequency INTEGER, | ||
1730 | 26 | -- Automatic responses. | ||
1731 | 27 | autorespond_owner INTEGER, | ||
1732 | 28 | autoresponse_owner_text TEXT, | ||
1733 | 29 | autorespond_postings INTEGER, | ||
1734 | 30 | autoresponse_postings_text TEXT, | ||
1735 | 31 | autorespond_requests INTEGER, | ||
1736 | 32 | autoresponse_request_text TEXT, | ||
1737 | 33 | autoresponse_grace_period TEXT, | ||
1738 | 34 | -- Bounces. | ||
1739 | 35 | forward_unrecognized_bounces_to INTEGER, | ||
1740 | 36 | process_bounces BOOLEAN, | ||
1741 | 37 | bounce_info_stale_after TEXT, | ||
1742 | 38 | bounce_matching_headers TEXT, | ||
1743 | 39 | bounce_notify_owner_on_disable BOOLEAN, | ||
1744 | 40 | bounce_notify_owner_on_removal BOOLEAN, | ||
1745 | 41 | bounce_score_threshold INTEGER, | ||
1746 | 42 | bounce_you_are_disabled_warnings INTEGER, | ||
1747 | 43 | bounce_you_are_disabled_warnings_interval TEXT, | ||
1748 | 44 | -- Content filtering. | ||
1749 | 45 | filter_action INTEGER, | ||
1750 | 46 | filter_content BOOLEAN, | ||
1751 | 47 | collapse_alternatives BOOLEAN, | ||
1752 | 48 | convert_html_to_plaintext BOOLEAN, | ||
1753 | 49 | default_member_action INTEGER, | ||
1754 | 50 | default_nonmember_action INTEGER, | ||
1755 | 51 | description TEXT, | ||
1756 | 52 | digest_footer_uri TEXT, | ||
1757 | 53 | digest_header_uri TEXT, | ||
1758 | 54 | digest_is_default BOOLEAN, | ||
1759 | 55 | digest_send_periodic BOOLEAN, | ||
1760 | 56 | digest_size_threshold REAL, | ||
1761 | 57 | digest_volume_frequency INTEGER, | ||
1762 | 58 | digestable BOOLEAN, | ||
1763 | 59 | discard_these_nonmembers BYTEA, | ||
1764 | 60 | emergency BOOLEAN, | ||
1765 | 61 | encode_ascii_prefixes BOOLEAN, | ||
1766 | 62 | first_strip_reply_to BOOLEAN, | ||
1767 | 63 | footer_uri TEXT, | ||
1768 | 64 | forward_auto_discards BOOLEAN, | ||
1769 | 65 | gateway_to_mail BOOLEAN, | ||
1770 | 66 | gateway_to_news BOOLEAN, | ||
1771 | 67 | generic_nonmember_action INTEGER, | ||
1772 | 68 | goodbye_message_uri TEXT, | ||
1773 | 69 | header_matches BYTEA, | ||
1774 | 70 | header_uri TEXT, | ||
1775 | 71 | hold_these_nonmembers BYTEA, | ||
1776 | 72 | info TEXT, | ||
1777 | 73 | linked_newsgroup TEXT, | ||
1778 | 74 | max_days_to_hold INTEGER, | ||
1779 | 75 | max_message_size INTEGER, | ||
1780 | 76 | max_num_recipients INTEGER, | ||
1781 | 77 | member_moderation_notice TEXT, | ||
1782 | 78 | mime_is_default_digest BOOLEAN, | ||
1783 | 79 | moderator_password TEXT, | ||
1784 | 80 | new_member_options INTEGER, | ||
1785 | 81 | news_moderation INTEGER, | ||
1786 | 82 | news_prefix_subject_too BOOLEAN, | ||
1787 | 83 | nntp_host TEXT, | ||
1788 | 84 | nondigestable BOOLEAN, | ||
1789 | 85 | nonmember_rejection_notice TEXT, | ||
1790 | 86 | obscure_addresses BOOLEAN, | ||
1791 | 87 | owner_chain TEXT, | ||
1792 | 88 | owner_pipeline TEXT, | ||
1793 | 89 | personalize INTEGER, | ||
1794 | 90 | post_id INTEGER, | ||
1795 | 91 | posting_chain TEXT, | ||
1796 | 92 | posting_pipeline TEXT, | ||
1797 | 93 | preferred_language TEXT, | ||
1798 | 94 | private_roster BOOLEAN, | ||
1799 | 95 | display_name TEXT, | ||
1800 | 96 | reject_these_nonmembers BYTEA, | ||
1801 | 97 | reply_goes_to_list INTEGER, | ||
1802 | 98 | reply_to_address TEXT, | ||
1803 | 99 | require_explicit_destination BOOLEAN, | ||
1804 | 100 | respond_to_post_requests BOOLEAN, | ||
1805 | 101 | scrub_nondigest BOOLEAN, | ||
1806 | 102 | send_goodbye_message BOOLEAN, | ||
1807 | 103 | send_reminders BOOLEAN, | ||
1808 | 104 | send_welcome_message BOOLEAN, | ||
1809 | 105 | subject_prefix TEXT, | ||
1810 | 106 | subscribe_auto_approval BYTEA, | ||
1811 | 107 | subscribe_policy INTEGER, | ||
1812 | 108 | topics BYTEA, | ||
1813 | 109 | topics_bodylines_limit INTEGER, | ||
1814 | 110 | topics_enabled BOOLEAN, | ||
1815 | 111 | unsubscribe_policy INTEGER, | ||
1816 | 112 | welcome_message_uri TEXT, | ||
1817 | 113 | -- This was accidentally added by the PostgreSQL porter. | ||
1818 | 114 | -- moderation_callback TEXT, | ||
1819 | 115 | PRIMARY KEY (id) | ||
1820 | 116 | ); | ||
1821 | 117 | |||
1822 | 118 | CREATE TABLE _request ( | ||
1823 | 119 | id SERIAL NOT NULL, | ||
1824 | 120 | "key" TEXT, | ||
1825 | 121 | request_type INTEGER, | ||
1826 | 122 | data_hash BYTEA, | ||
1827 | 123 | mailing_list_id INTEGER, | ||
1828 | 124 | PRIMARY KEY (id) | ||
1829 | 125 | -- XXX: config.db_reset() triggers IntegrityError | ||
1830 | 126 | -- , | ||
1831 | 127 | -- CONSTRAINT _request_mailing_list_id_fk | ||
1832 | 128 | -- FOREIGN KEY (mailing_list_id) REFERENCES mailinglist (id) | ||
1833 | 129 | ); | ||
1834 | 130 | |||
1835 | 131 | CREATE TABLE acceptablealias ( | ||
1836 | 132 | id SERIAL NOT NULL, | ||
1837 | 133 | "alias" TEXT NOT NULL, | ||
1838 | 134 | mailing_list_id INTEGER NOT NULL, | ||
1839 | 135 | PRIMARY KEY (id) | ||
1840 | 136 | -- XXX: config.db_reset() triggers IntegrityError | ||
1841 | 137 | -- , | ||
1842 | 138 | -- CONSTRAINT acceptablealias_mailing_list_id_fk | ||
1843 | 139 | -- FOREIGN KEY (mailing_list_id) REFERENCES mailinglist (id) | ||
1844 | 140 | ); | ||
1845 | 141 | CREATE INDEX ix_acceptablealias_mailing_list_id | ||
1846 | 142 | ON acceptablealias (mailing_list_id); | ||
1847 | 143 | CREATE INDEX ix_acceptablealias_alias ON acceptablealias ("alias"); | ||
1848 | 144 | |||
1849 | 145 | CREATE TABLE preferences ( | ||
1850 | 146 | id SERIAL NOT NULL, | ||
1851 | 147 | acknowledge_posts BOOLEAN, | ||
1852 | 148 | hide_address BOOLEAN, | ||
1853 | 149 | preferred_language TEXT, | ||
1854 | 150 | receive_list_copy BOOLEAN, | ||
1855 | 151 | receive_own_postings BOOLEAN, | ||
1856 | 152 | delivery_mode INTEGER, | ||
1857 | 153 | delivery_status INTEGER, | ||
1858 | 154 | PRIMARY KEY (id) | ||
1859 | 155 | ); | ||
1860 | 156 | |||
1861 | 157 | CREATE TABLE address ( | ||
1862 | 158 | id SERIAL NOT NULL, | ||
1863 | 159 | email TEXT, | ||
1864 | 160 | _original TEXT, | ||
1865 | 161 | display_name TEXT, | ||
1866 | 162 | verified_on TIMESTAMP, | ||
1867 | 163 | registered_on TIMESTAMP, | ||
1868 | 164 | user_id INTEGER, | ||
1869 | 165 | preferences_id INTEGER, | ||
1870 | 166 | PRIMARY KEY (id) | ||
1871 | 167 | -- XXX: config.db_reset() triggers IntegrityError | ||
1872 | 168 | -- , | ||
1873 | 169 | -- CONSTRAINT address_preferences_id_fk | ||
1874 | 170 | -- FOREIGN KEY (preferences_id) REFERENCES preferences (id) | ||
1875 | 171 | ); | ||
1876 | 172 | |||
1877 | 173 | CREATE TABLE "user" ( | ||
1878 | 174 | id SERIAL NOT NULL, | ||
1879 | 175 | display_name TEXT, | ||
1880 | 176 | password BYTEA, | ||
1881 | 177 | _user_id UUID, | ||
1882 | 178 | _created_on TIMESTAMP, | ||
1883 | 179 | _preferred_address_id INTEGER, | ||
1884 | 180 | preferences_id INTEGER, | ||
1885 | 181 | PRIMARY KEY (id) | ||
1886 | 182 | -- XXX: config.db_reset() triggers IntegrityError | ||
1887 | 183 | -- , | ||
1888 | 184 | -- CONSTRAINT user_preferences_id_fk | ||
1889 | 185 | -- FOREIGN KEY (preferences_id) REFERENCES preferences (id), | ||
1890 | 186 | -- XXX: config.db_reset() triggers IntegrityError | ||
1891 | 187 | -- CONSTRAINT _preferred_address_id_fk | ||
1892 | 188 | -- FOREIGN KEY (_preferred_address_id) REFERENCES address (id) | ||
1893 | 189 | ); | ||
1894 | 190 | CREATE INDEX ix_user_user_id ON "user" (_user_id); | ||
1895 | 191 | |||
1896 | 192 | -- since user and address have circular foreign key refs, the | ||
1897 | 193 | -- constraint on the address table has to be added after | ||
1898 | 194 | -- the user table is created | ||
1899 | 195 | -- | ||
1900 | 196 | -- XXX: users.rst triggers an IntegrityError | ||
1901 | 197 | -- ALTER TABLE address ADD | ||
1902 | 198 | -- CONSTRAINT address_user_id_fk | ||
1903 | 199 | -- FOREIGN KEY (user_id) REFERENCES "user" (id); | ||
1904 | 200 | |||
1905 | 201 | CREATE TABLE autoresponserecord ( | ||
1906 | 202 | id SERIAL NOT NULL, | ||
1907 | 203 | address_id INTEGER, | ||
1908 | 204 | mailing_list_id INTEGER, | ||
1909 | 205 | response_type INTEGER, | ||
1910 | 206 | date_sent TIMESTAMP, | ||
1911 | 207 | PRIMARY KEY (id) | ||
1912 | 208 | -- XXX: config.db_reset() triggers IntegrityError | ||
1913 | 209 | -- , | ||
1914 | 210 | -- CONSTRAINT autoresponserecord_address_id_fk | ||
1915 | 211 | -- FOREIGN KEY (address_id) REFERENCES address (id) | ||
1916 | 212 | -- XXX: config.db_reset() triggers IntegrityError | ||
1917 | 213 | -- , | ||
1918 | 214 | -- CONSTRAINT autoresponserecord_mailing_list_id | ||
1919 | 215 | -- FOREIGN KEY (mailing_list_id) REFERENCES mailinglist (id) | ||
1920 | 216 | ); | ||
1921 | 217 | CREATE INDEX ix_autoresponserecord_address_id | ||
1922 | 218 | ON autoresponserecord (address_id); | ||
1923 | 219 | CREATE INDEX ix_autoresponserecord_mailing_list_id | ||
1924 | 220 | ON autoresponserecord (mailing_list_id); | ||
1925 | 221 | |||
1926 | 222 | CREATE TABLE bounceevent ( | ||
1927 | 223 | id SERIAL NOT NULL, | ||
1928 | 224 | list_name TEXT, | ||
1929 | 225 | email TEXT, | ||
1930 | 226 | "timestamp" TIMESTAMP, | ||
1931 | 227 | message_id TEXT, | ||
1932 | 228 | context INTEGER, | ||
1933 | 229 | processed BOOLEAN, | ||
1934 | 230 | PRIMARY KEY (id) | ||
1935 | 231 | ); | ||
1936 | 232 | |||
1937 | 233 | CREATE TABLE contentfilter ( | ||
1938 | 234 | id SERIAL NOT NULL, | ||
1939 | 235 | mailing_list_id INTEGER, | ||
1940 | 236 | filter_pattern TEXT, | ||
1941 | 237 | filter_type INTEGER, | ||
1942 | 238 | PRIMARY KEY (id), | ||
1943 | 239 | CONSTRAINT contentfilter_mailing_list_id | ||
1944 | 240 | FOREIGN KEY (mailing_list_id) REFERENCES mailinglist (id) | ||
1945 | 241 | ); | ||
1946 | 242 | CREATE INDEX ix_contentfilter_mailing_list_id | ||
1947 | 243 | ON contentfilter (mailing_list_id); | ||
1948 | 244 | |||
1949 | 245 | CREATE TABLE domain ( | ||
1950 | 246 | id SERIAL NOT NULL, | ||
1951 | 247 | mail_host TEXT, | ||
1952 | 248 | base_url TEXT, | ||
1953 | 249 | description TEXT, | ||
1954 | 250 | contact_address TEXT, | ||
1955 | 251 | PRIMARY KEY (id) | ||
1956 | 252 | ); | ||
1957 | 253 | |||
1958 | 254 | CREATE TABLE language ( | ||
1959 | 255 | id SERIAL NOT NULL, | ||
1960 | 256 | code TEXT, | ||
1961 | 257 | PRIMARY KEY (id) | ||
1962 | 258 | ); | ||
1963 | 259 | |||
1964 | 260 | CREATE TABLE member ( | ||
1965 | 261 | id SERIAL NOT NULL, | ||
1966 | 262 | _member_id UUID, | ||
1967 | 263 | role INTEGER, | ||
1968 | 264 | mailing_list TEXT, | ||
1969 | 265 | moderation_action INTEGER, | ||
1970 | 266 | address_id INTEGER, | ||
1971 | 267 | preferences_id INTEGER, | ||
1972 | 268 | user_id INTEGER, | ||
1973 | 269 | PRIMARY KEY (id) | ||
1974 | 270 | -- XXX: config.db_reset() triggers IntegrityError | ||
1975 | 271 | -- , | ||
1976 | 272 | -- CONSTRAINT member_address_id_fk | ||
1977 | 273 | -- FOREIGN KEY (address_id) REFERENCES address (id), | ||
1978 | 274 | -- XXX: config.db_reset() triggers IntegrityError | ||
1979 | 275 | -- CONSTRAINT member_preferences_id_fk | ||
1980 | 276 | -- FOREIGN KEY (preferences_id) REFERENCES preferences (id), | ||
1981 | 277 | -- CONSTRAINT member_user_id_fk | ||
1982 | 278 | -- FOREIGN KEY (user_id) REFERENCES "user" (id) | ||
1983 | 279 | ); | ||
1984 | 280 | CREATE INDEX ix_member__member_id ON member (_member_id); | ||
1985 | 281 | CREATE INDEX ix_member_address_id ON member (address_id); | ||
1986 | 282 | CREATE INDEX ix_member_preferences_id ON member (preferences_id); | ||
1987 | 283 | |||
1988 | 284 | CREATE TABLE message ( | ||
1989 | 285 | id SERIAL NOT NULL, | ||
1990 | 286 | message_id_hash BYTEA, | ||
1991 | 287 | path BYTEA, | ||
1992 | 288 | message_id TEXT, | ||
1993 | 289 | PRIMARY KEY (id) | ||
1994 | 290 | ); | ||
1995 | 291 | |||
1996 | 292 | CREATE TABLE onelastdigest ( | ||
1997 | 293 | id SERIAL NOT NULL, | ||
1998 | 294 | mailing_list_id INTEGER, | ||
1999 | 295 | address_id INTEGER, | ||
2000 | 296 | delivery_mode INTEGER, | ||
2001 | 297 | PRIMARY KEY (id), | ||
2002 | 298 | CONSTRAINT onelastdigest_mailing_list_id_fk | ||
2003 | 299 | FOREIGN KEY (mailing_list_id) REFERENCES mailinglist(id), | ||
2004 | 300 | CONSTRAINT onelastdigest_address_id_fk | ||
2005 | 301 | FOREIGN KEY (address_id) REFERENCES address(id) | ||
2006 | 302 | ); | ||
2007 | 303 | |||
2008 | 304 | CREATE TABLE pended ( | ||
2009 | 305 | id SERIAL NOT NULL, | ||
2010 | 306 | token BYTEA, | ||
2011 | 307 | expiration_date TIMESTAMP, | ||
2012 | 308 | PRIMARY KEY (id) | ||
2013 | 309 | ); | ||
2014 | 310 | |||
2015 | 311 | CREATE TABLE pendedkeyvalue ( | ||
2016 | 312 | id SERIAL NOT NULL, | ||
2017 | 313 | "key" TEXT, | ||
2018 | 314 | value TEXT, | ||
2019 | 315 | pended_id INTEGER, | ||
2020 | 316 | PRIMARY KEY (id) | ||
2021 | 317 | -- , | ||
2022 | 318 | -- XXX: config.db_reset() triggers IntegrityError | ||
2023 | 319 | -- CONSTRAINT pendedkeyvalue_pended_id_fk | ||
2024 | 320 | -- FOREIGN KEY (pended_id) REFERENCES pended (id) | ||
2025 | 321 | ); | ||
2026 | 322 | |||
2027 | 323 | CREATE TABLE version ( | ||
2028 | 324 | id SERIAL NOT NULL, | ||
2029 | 325 | component TEXT, | ||
2030 | 326 | version TEXT, | ||
2031 | 327 | PRIMARY KEY (id) | ||
2032 | 328 | ); | ||
2033 | 329 | |||
2034 | 330 | CREATE INDEX ix__request_mailing_list_id ON _request (mailing_list_id); | ||
2035 | 331 | CREATE INDEX ix_address_preferences_id ON address (preferences_id); | ||
2036 | 332 | CREATE INDEX ix_address_user_id ON address (user_id); | ||
2037 | 333 | CREATE INDEX ix_pendedkeyvalue_pended_id ON pendedkeyvalue (pended_id); | ||
2038 | 334 | CREATE INDEX ix_user_preferences_id ON "user" (preferences_id); | ||
2039 | 335 | |||
2040 | 336 | CREATE TABLE ban ( | ||
2041 | 337 | id SERIAL NOT NULL, | ||
2042 | 338 | email TEXT, | ||
2043 | 339 | mailing_list TEXT, | ||
2044 | 340 | PRIMARY KEY (id) | ||
2045 | 341 | ); | ||
2046 | 342 | |||
2047 | 343 | CREATE TABLE uid ( | ||
2048 | 344 | -- Keep track of all assigned unique ids to prevent re-use. | ||
2049 | 345 | id SERIAL NOT NULL, | ||
2050 | 346 | uid UUID, | ||
2051 | 347 | PRIMARY KEY (id) | ||
2052 | 348 | ); | ||
2053 | 349 | CREATE INDEX ix_uid_uid ON uid (uid); | ||
2054 | 350 | 0 | ||
2055 | === removed file 'src/mailman/database/schema/sqlite.sql' | |||
2056 | --- src/mailman/database/schema/sqlite.sql 2012-04-08 16:15:29 +0000 | |||
2057 | +++ src/mailman/database/schema/sqlite.sql 1970-01-01 00:00:00 +0000 | |||
2058 | @@ -1,327 +0,0 @@ | |||
2059 | 1 | -- THIS FILE HAS BEEN FROZEN AS OF 3.0b1 | ||
2060 | 2 | -- SEE THE SCHEMA MIGRATIONS FOR DIFFERENCES. | ||
2061 | 3 | |||
2062 | 4 | PRAGMA foreign_keys = ON; | ||
2063 | 5 | |||
2064 | 6 | CREATE TABLE _request ( | ||
2065 | 7 | id INTEGER NOT NULL, | ||
2066 | 8 | "key" TEXT, | ||
2067 | 9 | request_type INTEGER, | ||
2068 | 10 | data_hash TEXT, | ||
2069 | 11 | mailing_list_id INTEGER, | ||
2070 | 12 | PRIMARY KEY (id), | ||
2071 | 13 | CONSTRAINT _request_mailing_list_id_fk | ||
2072 | 14 | FOREIGN KEY (mailing_list_id) REFERENCES mailinglist (id) | ||
2073 | 15 | ); | ||
2074 | 16 | |||
2075 | 17 | CREATE TABLE acceptablealias ( | ||
2076 | 18 | id INTEGER NOT NULL, | ||
2077 | 19 | "alias" TEXT NOT NULL, | ||
2078 | 20 | mailing_list_id INTEGER NOT NULL, | ||
2079 | 21 | PRIMARY KEY (id), | ||
2080 | 22 | CONSTRAINT acceptablealias_mailing_list_id_fk | ||
2081 | 23 | FOREIGN KEY (mailing_list_id) REFERENCES mailinglist (id) | ||
2082 | 24 | ); | ||
2083 | 25 | CREATE INDEX ix_acceptablealias_mailing_list_id | ||
2084 | 26 | ON acceptablealias (mailing_list_id); | ||
2085 | 27 | CREATE INDEX ix_acceptablealias_alias ON acceptablealias ("alias"); | ||
2086 | 28 | |||
2087 | 29 | CREATE TABLE address ( | ||
2088 | 30 | id INTEGER NOT NULL, | ||
2089 | 31 | email TEXT, | ||
2090 | 32 | _original TEXT, | ||
2091 | 33 | display_name TEXT, | ||
2092 | 34 | verified_on TIMESTAMP, | ||
2093 | 35 | registered_on TIMESTAMP, | ||
2094 | 36 | user_id INTEGER, | ||
2095 | 37 | preferences_id INTEGER, | ||
2096 | 38 | PRIMARY KEY (id), | ||
2097 | 39 | CONSTRAINT address_user_id_fk | ||
2098 | 40 | FOREIGN KEY (user_id) REFERENCES user (id), | ||
2099 | 41 | CONSTRAINT address_preferences_id_fk | ||
2100 | 42 | FOREIGN KEY (preferences_id) REFERENCES preferences (id) | ||
2101 | 43 | ); | ||
2102 | 44 | |||
2103 | 45 | CREATE TABLE autoresponserecord ( | ||
2104 | 46 | id INTEGER NOT NULL, | ||
2105 | 47 | address_id INTEGER, | ||
2106 | 48 | mailing_list_id INTEGER, | ||
2107 | 49 | response_type INTEGER, | ||
2108 | 50 | date_sent TIMESTAMP, | ||
2109 | 51 | PRIMARY KEY (id), | ||
2110 | 52 | CONSTRAINT autoresponserecord_address_id_fk | ||
2111 | 53 | FOREIGN KEY (address_id) REFERENCES address (id), | ||
2112 | 54 | CONSTRAINT autoresponserecord_mailing_list_id | ||
2113 | 55 | FOREIGN KEY (mailing_list_id) REFERENCES mailinglist (id) | ||
2114 | 56 | ); | ||
2115 | 57 | CREATE INDEX ix_autoresponserecord_address_id | ||
2116 | 58 | ON autoresponserecord (address_id); | ||
2117 | 59 | CREATE INDEX ix_autoresponserecord_mailing_list_id | ||
2118 | 60 | ON autoresponserecord (mailing_list_id); | ||
2119 | 61 | |||
2120 | 62 | CREATE TABLE bounceevent ( | ||
2121 | 63 | id INTEGER NOT NULL, | ||
2122 | 64 | list_name TEXT, | ||
2123 | 65 | email TEXT, | ||
2124 | 66 | 'timestamp' TIMESTAMP, | ||
2125 | 67 | message_id TEXT, | ||
2126 | 68 | context INTEGER, | ||
2127 | 69 | processed BOOLEAN, | ||
2128 | 70 | PRIMARY KEY (id) | ||
2129 | 71 | ); | ||
2130 | 72 | |||
2131 | 73 | CREATE TABLE contentfilter ( | ||
2132 | 74 | id INTEGER NOT NULL, | ||
2133 | 75 | mailing_list_id INTEGER, | ||
2134 | 76 | filter_pattern TEXT, | ||
2135 | 77 | filter_type INTEGER, | ||
2136 | 78 | PRIMARY KEY (id), | ||
2137 | 79 | CONSTRAINT contentfilter_mailing_list_id | ||
2138 | 80 | FOREIGN KEY (mailing_list_id) REFERENCES mailinglist (id) | ||
2139 | 81 | ); | ||
2140 | 82 | CREATE INDEX ix_contentfilter_mailing_list_id | ||
2141 | 83 | ON contentfilter (mailing_list_id); | ||
2142 | 84 | |||
2143 | 85 | CREATE TABLE domain ( | ||
2144 | 86 | id INTEGER NOT NULL, | ||
2145 | 87 | mail_host TEXT, | ||
2146 | 88 | base_url TEXT, | ||
2147 | 89 | description TEXT, | ||
2148 | 90 | contact_address TEXT, | ||
2149 | 91 | PRIMARY KEY (id) | ||
2150 | 92 | ); | ||
2151 | 93 | |||
2152 | 94 | CREATE TABLE language ( | ||
2153 | 95 | id INTEGER NOT NULL, | ||
2154 | 96 | code TEXT, | ||
2155 | 97 | PRIMARY KEY (id) | ||
2156 | 98 | ); | ||
2157 | 99 | |||
2158 | 100 | CREATE TABLE mailinglist ( | ||
2159 | 101 | id INTEGER NOT NULL, | ||
2160 | 102 | -- List identity | ||
2161 | 103 | list_name TEXT, | ||
2162 | 104 | mail_host TEXT, | ||
2163 | 105 | include_list_post_header BOOLEAN, | ||
2164 | 106 | include_rfc2369_headers BOOLEAN, | ||
2165 | 107 | -- Attributes not directly modifiable via the web u/i | ||
2166 | 108 | created_at TIMESTAMP, | ||
2167 | 109 | admin_member_chunksize INTEGER, | ||
2168 | 110 | next_request_id INTEGER, | ||
2169 | 111 | next_digest_number INTEGER, | ||
2170 | 112 | digest_last_sent_at TIMESTAMP, | ||
2171 | 113 | volume INTEGER, | ||
2172 | 114 | last_post_at TIMESTAMP, | ||
2173 | 115 | accept_these_nonmembers BLOB, | ||
2174 | 116 | acceptable_aliases_id INTEGER, | ||
2175 | 117 | admin_immed_notify BOOLEAN, | ||
2176 | 118 | admin_notify_mchanges BOOLEAN, | ||
2177 | 119 | administrivia BOOLEAN, | ||
2178 | 120 | advertised BOOLEAN, | ||
2179 | 121 | anonymous_list BOOLEAN, | ||
2180 | 122 | archive BOOLEAN, | ||
2181 | 123 | archive_private BOOLEAN, | ||
2182 | 124 | archive_volume_frequency INTEGER, | ||
2183 | 125 | -- Automatic responses. | ||
2184 | 126 | autorespond_owner INTEGER, | ||
2185 | 127 | autoresponse_owner_text TEXT, | ||
2186 | 128 | autorespond_postings INTEGER, | ||
2187 | 129 | autoresponse_postings_text TEXT, | ||
2188 | 130 | autorespond_requests INTEGER, | ||
2189 | 131 | autoresponse_request_text TEXT, | ||
2190 | 132 | autoresponse_grace_period TEXT, | ||
2191 | 133 | -- Bounces. | ||
2192 | 134 | forward_unrecognized_bounces_to INTEGER, | ||
2193 | 135 | process_bounces BOOLEAN, | ||
2194 | 136 | bounce_info_stale_after TEXT, | ||
2195 | 137 | bounce_matching_headers TEXT, | ||
2196 | 138 | bounce_notify_owner_on_disable BOOLEAN, | ||
2197 | 139 | bounce_notify_owner_on_removal BOOLEAN, | ||
2198 | 140 | bounce_score_threshold INTEGER, | ||
2199 | 141 | bounce_you_are_disabled_warnings INTEGER, | ||
2200 | 142 | bounce_you_are_disabled_warnings_interval TEXT, | ||
2201 | 143 | -- Content filtering. | ||
2202 | 144 | filter_action INTEGER, | ||
2203 | 145 | filter_content BOOLEAN, | ||
2204 | 146 | collapse_alternatives BOOLEAN, | ||
2205 | 147 | convert_html_to_plaintext BOOLEAN, | ||
2206 | 148 | default_member_action INTEGER, | ||
2207 | 149 | default_nonmember_action INTEGER, | ||
2208 | 150 | description TEXT, | ||
2209 | 151 | digest_footer_uri TEXT, | ||
2210 | 152 | digest_header_uri TEXT, | ||
2211 | 153 | digest_is_default BOOLEAN, | ||
2212 | 154 | digest_send_periodic BOOLEAN, | ||
2213 | 155 | digest_size_threshold FLOAT, | ||
2214 | 156 | digest_volume_frequency INTEGER, | ||
2215 | 157 | digestable BOOLEAN, | ||
2216 | 158 | discard_these_nonmembers BLOB, | ||
2217 | 159 | emergency BOOLEAN, | ||
2218 | 160 | encode_ascii_prefixes BOOLEAN, | ||
2219 | 161 | first_strip_reply_to BOOLEAN, | ||
2220 | 162 | footer_uri TEXT, | ||
2221 | 163 | forward_auto_discards BOOLEAN, | ||
2222 | 164 | gateway_to_mail BOOLEAN, | ||
2223 | 165 | gateway_to_news BOOLEAN, | ||
2224 | 166 | generic_nonmember_action INTEGER, | ||
2225 | 167 | goodbye_message_uri TEXT, | ||
2226 | 168 | header_matches BLOB, | ||
2227 | 169 | header_uri TEXT, | ||
2228 | 170 | hold_these_nonmembers BLOB, | ||
2229 | 171 | info TEXT, | ||
2230 | 172 | linked_newsgroup TEXT, | ||
2231 | 173 | max_days_to_hold INTEGER, | ||
2232 | 174 | max_message_size INTEGER, | ||
2233 | 175 | max_num_recipients INTEGER, | ||
2234 | 176 | member_moderation_notice TEXT, | ||
2235 | 177 | mime_is_default_digest BOOLEAN, | ||
2236 | 178 | moderator_password TEXT, | ||
2237 | 179 | new_member_options INTEGER, | ||
2238 | 180 | news_moderation INTEGER, | ||
2239 | 181 | news_prefix_subject_too BOOLEAN, | ||
2240 | 182 | nntp_host TEXT, | ||
2241 | 183 | nondigestable BOOLEAN, | ||
2242 | 184 | nonmember_rejection_notice TEXT, | ||
2243 | 185 | obscure_addresses BOOLEAN, | ||
2244 | 186 | owner_chain TEXT, | ||
2245 | 187 | owner_pipeline TEXT, | ||
2246 | 188 | personalize INTEGER, | ||
2247 | 189 | post_id INTEGER, | ||
2248 | 190 | posting_chain TEXT, | ||
2249 | 191 | posting_pipeline TEXT, | ||
2250 | 192 | preferred_language TEXT, | ||
2251 | 193 | private_roster BOOLEAN, | ||
2252 | 194 | display_name TEXT, | ||
2253 | 195 | reject_these_nonmembers BLOB, | ||
2254 | 196 | reply_goes_to_list INTEGER, | ||
2255 | 197 | reply_to_address TEXT, | ||
2256 | 198 | require_explicit_destination BOOLEAN, | ||
2257 | 199 | respond_to_post_requests BOOLEAN, | ||
2258 | 200 | scrub_nondigest BOOLEAN, | ||
2259 | 201 | send_goodbye_message BOOLEAN, | ||
2260 | 202 | send_reminders BOOLEAN, | ||
2261 | 203 | send_welcome_message BOOLEAN, | ||
2262 | 204 | subject_prefix TEXT, | ||
2263 | 205 | subscribe_auto_approval BLOB, | ||
2264 | 206 | subscribe_policy INTEGER, | ||
2265 | 207 | topics BLOB, | ||
2266 | 208 | topics_bodylines_limit INTEGER, | ||
2267 | 209 | topics_enabled BOOLEAN, | ||
2268 | 210 | unsubscribe_policy INTEGER, | ||
2269 | 211 | welcome_message_uri TEXT, | ||
2270 | 212 | PRIMARY KEY (id) | ||
2271 | 213 | ); | ||
2272 | 214 | |||
2273 | 215 | CREATE TABLE member ( | ||
2274 | 216 | id INTEGER NOT NULL, | ||
2275 | 217 | _member_id TEXT, | ||
2276 | 218 | role INTEGER, | ||
2277 | 219 | mailing_list TEXT, | ||
2278 | 220 | moderation_action INTEGER, | ||
2279 | 221 | address_id INTEGER, | ||
2280 | 222 | preferences_id INTEGER, | ||
2281 | 223 | user_id INTEGER, | ||
2282 | 224 | PRIMARY KEY (id), | ||
2283 | 225 | CONSTRAINT member_address_id_fk | ||
2284 | 226 | FOREIGN KEY (address_id) REFERENCES address (id), | ||
2285 | 227 | CONSTRAINT member_preferences_id_fk | ||
2286 | 228 | FOREIGN KEY (preferences_id) REFERENCES preferences (id) | ||
2287 | 229 | CONSTRAINT member_user_id_fk | ||
2288 | 230 | FOREIGN KEY (user_id) REFERENCES user (id) | ||
2289 | 231 | ); | ||
2290 | 232 | CREATE INDEX ix_member__member_id ON member (_member_id); | ||
2291 | 233 | CREATE INDEX ix_member_address_id ON member (address_id); | ||
2292 | 234 | CREATE INDEX ix_member_preferences_id ON member (preferences_id); | ||
2293 | 235 | |||
2294 | 236 | CREATE TABLE message ( | ||
2295 | 237 | id INTEGER NOT NULL, | ||
2296 | 238 | message_id_hash TEXT, | ||
2297 | 239 | path TEXT, | ||
2298 | 240 | message_id TEXT, | ||
2299 | 241 | PRIMARY KEY (id) | ||
2300 | 242 | ); | ||
2301 | 243 | |||
2302 | 244 | CREATE TABLE onelastdigest ( | ||
2303 | 245 | id INTEGER NOT NULL, | ||
2304 | 246 | mailing_list_id INTEGER, | ||
2305 | 247 | address_id INTEGER, | ||
2306 | 248 | delivery_mode INTEGER, | ||
2307 | 249 | PRIMARY KEY (id), | ||
2308 | 250 | CONSTRAINT onelastdigest_mailing_list_id_fk | ||
2309 | 251 | FOREIGN KEY (mailing_list_id) REFERENCES mailinglist(id), | ||
2310 | 252 | CONSTRAINT onelastdigest_address_id_fk | ||
2311 | 253 | FOREIGN KEY (address_id) REFERENCES address(id) | ||
2312 | 254 | ); | ||
2313 | 255 | |||
2314 | 256 | CREATE TABLE pended ( | ||
2315 | 257 | id INTEGER NOT NULL, | ||
2316 | 258 | token TEXT, | ||
2317 | 259 | expiration_date TIMESTAMP, | ||
2318 | 260 | PRIMARY KEY (id) | ||
2319 | 261 | ); | ||
2320 | 262 | |||
2321 | 263 | CREATE TABLE pendedkeyvalue ( | ||
2322 | 264 | id INTEGER NOT NULL, | ||
2323 | 265 | "key" TEXT, | ||
2324 | 266 | value TEXT, | ||
2325 | 267 | pended_id INTEGER, | ||
2326 | 268 | PRIMARY KEY (id), | ||
2327 | 269 | CONSTRAINT pendedkeyvalue_pended_id_fk | ||
2328 | 270 | FOREIGN KEY (pended_id) REFERENCES pended (id) | ||
2329 | 271 | ); | ||
2330 | 272 | |||
2331 | 273 | CREATE TABLE preferences ( | ||
2332 | 274 | id INTEGER NOT NULL, | ||
2333 | 275 | acknowledge_posts BOOLEAN, | ||
2334 | 276 | hide_address BOOLEAN, | ||
2335 | 277 | preferred_language TEXT, | ||
2336 | 278 | receive_list_copy BOOLEAN, | ||
2337 | 279 | receive_own_postings BOOLEAN, | ||
2338 | 280 | delivery_mode INTEGER, | ||
2339 | 281 | delivery_status INTEGER, | ||
2340 | 282 | PRIMARY KEY (id) | ||
2341 | 283 | ); | ||
2342 | 284 | |||
2343 | 285 | CREATE TABLE user ( | ||
2344 | 286 | id INTEGER NOT NULL, | ||
2345 | 287 | display_name TEXT, | ||
2346 | 288 | password BINARY, | ||
2347 | 289 | _user_id TEXT, | ||
2348 | 290 | _created_on TIMESTAMP, | ||
2349 | 291 | _preferred_address_id INTEGER, | ||
2350 | 292 | preferences_id INTEGER, | ||
2351 | 293 | PRIMARY KEY (id), | ||
2352 | 294 | CONSTRAINT user_preferences_id_fk | ||
2353 | 295 | FOREIGN KEY (preferences_id) REFERENCES preferences (id), | ||
2354 | 296 | CONSTRAINT _preferred_address_id_fk | ||
2355 | 297 | FOREIGN KEY (_preferred_address_id) REFERENCES address (id) | ||
2356 | 298 | ); | ||
2357 | 299 | CREATE INDEX ix_user_user_id ON user (_user_id); | ||
2358 | 300 | |||
2359 | 301 | CREATE TABLE version ( | ||
2360 | 302 | id INTEGER NOT NULL, | ||
2361 | 303 | component TEXT, | ||
2362 | 304 | version TEXT, | ||
2363 | 305 | PRIMARY KEY (id) | ||
2364 | 306 | ); | ||
2365 | 307 | |||
2366 | 308 | CREATE INDEX ix__request_mailing_list_id ON _request (mailing_list_id); | ||
2367 | 309 | CREATE INDEX ix_address_preferences_id ON address (preferences_id); | ||
2368 | 310 | CREATE INDEX ix_address_user_id ON address (user_id); | ||
2369 | 311 | CREATE INDEX ix_pendedkeyvalue_pended_id ON pendedkeyvalue (pended_id); | ||
2370 | 312 | CREATE INDEX ix_user_preferences_id ON user (preferences_id); | ||
2371 | 313 | |||
2372 | 314 | CREATE TABLE ban ( | ||
2373 | 315 | id INTEGER NOT NULL, | ||
2374 | 316 | email TEXT, | ||
2375 | 317 | mailing_list TEXT, | ||
2376 | 318 | PRIMARY KEY (id) | ||
2377 | 319 | ); | ||
2378 | 320 | |||
2379 | 321 | CREATE TABLE uid ( | ||
2380 | 322 | -- Keep track of all assigned unique ids to prevent re-use. | ||
2381 | 323 | id INTEGER NOT NULL, | ||
2382 | 324 | uid TEXT, | ||
2383 | 325 | PRIMARY KEY (id) | ||
2384 | 326 | ); | ||
2385 | 327 | CREATE INDEX ix_uid_uid ON uid (uid); | ||
2386 | 328 | 0 | ||
2387 | === removed file 'src/mailman/database/schema/sqlite_20120407000000_01.sql' | |||
2388 | --- src/mailman/database/schema/sqlite_20120407000000_01.sql 2013-09-01 15:15:08 +0000 | |||
2389 | +++ src/mailman/database/schema/sqlite_20120407000000_01.sql 1970-01-01 00:00:00 +0000 | |||
2390 | @@ -1,280 +0,0 @@ | |||
2391 | 1 | -- This file contains the sqlite3 schema migration from | ||
2392 | 2 | -- 3.0b1 TO 3.0b2 | ||
2393 | 3 | -- | ||
2394 | 4 | -- 3.0b2 has been released thus you MAY NOT edit this file. | ||
2395 | 5 | |||
2396 | 6 | -- For SQLite3 migration strategy, see | ||
2397 | 7 | -- http://sqlite.org/faq.html#q11 | ||
2398 | 8 | |||
2399 | 9 | -- REMOVALS from the mailinglist table: | ||
2400 | 10 | -- REM archive | ||
2401 | 11 | -- REM archive_private | ||
2402 | 12 | -- REM archive_volume_frequency | ||
2403 | 13 | -- REM include_list_post_header | ||
2404 | 14 | -- REM news_moderation | ||
2405 | 15 | -- REM news_prefix_subject_too | ||
2406 | 16 | -- REM nntp_host | ||
2407 | 17 | -- | ||
2408 | 18 | -- ADDS to the mailing list table: | ||
2409 | 19 | -- ADD allow_list_posts | ||
2410 | 20 | -- ADD archive_policy | ||
2411 | 21 | -- ADD list_id | ||
2412 | 22 | -- ADD newsgroup_moderation | ||
2413 | 23 | -- ADD nntp_prefix_subject_too | ||
2414 | 24 | |||
2415 | 25 | -- LP: #971013 | ||
2416 | 26 | -- LP: #967238 | ||
2417 | 27 | |||
2418 | 28 | -- REMOVALS from the member table: | ||
2419 | 29 | -- REM mailing_list | ||
2420 | 30 | |||
2421 | 31 | -- ADDS to the member table: | ||
2422 | 32 | -- ADD list_id | ||
2423 | 33 | |||
2424 | 34 | -- LP: #1024509 | ||
2425 | 35 | |||
2426 | 36 | |||
2427 | 37 | CREATE TABLE mailinglist_backup ( | ||
2428 | 38 | id INTEGER NOT NULL, | ||
2429 | 39 | -- List identity | ||
2430 | 40 | list_name TEXT, | ||
2431 | 41 | mail_host TEXT, | ||
2432 | 42 | allow_list_posts BOOLEAN, | ||
2433 | 43 | include_rfc2369_headers BOOLEAN, | ||
2434 | 44 | -- Attributes not directly modifiable via the web u/i | ||
2435 | 45 | created_at TIMESTAMP, | ||
2436 | 46 | admin_member_chunksize INTEGER, | ||
2437 | 47 | next_request_id INTEGER, | ||
2438 | 48 | next_digest_number INTEGER, | ||
2439 | 49 | digest_last_sent_at TIMESTAMP, | ||
2440 | 50 | volume INTEGER, | ||
2441 | 51 | last_post_at TIMESTAMP, | ||
2442 | 52 | accept_these_nonmembers BLOB, | ||
2443 | 53 | acceptable_aliases_id INTEGER, | ||
2444 | 54 | admin_immed_notify BOOLEAN, | ||
2445 | 55 | admin_notify_mchanges BOOLEAN, | ||
2446 | 56 | administrivia BOOLEAN, | ||
2447 | 57 | advertised BOOLEAN, | ||
2448 | 58 | anonymous_list BOOLEAN, | ||
2449 | 59 | -- Automatic responses. | ||
2450 | 60 | autorespond_owner INTEGER, | ||
2451 | 61 | autoresponse_owner_text TEXT, | ||
2452 | 62 | autorespond_postings INTEGER, | ||
2453 | 63 | autoresponse_postings_text TEXT, | ||
2454 | 64 | autorespond_requests INTEGER, | ||
2455 | 65 | autoresponse_request_text TEXT, | ||
2456 | 66 | autoresponse_grace_period TEXT, | ||
2457 | 67 | -- Bounces. | ||
2458 | 68 | forward_unrecognized_bounces_to INTEGER, | ||
2459 | 69 | process_bounces BOOLEAN, | ||
2460 | 70 | bounce_info_stale_after TEXT, | ||
2461 | 71 | bounce_matching_headers TEXT, | ||
2462 | 72 | bounce_notify_owner_on_disable BOOLEAN, | ||
2463 | 73 | bounce_notify_owner_on_removal BOOLEAN, | ||
2464 | 74 | bounce_score_threshold INTEGER, | ||
2465 | 75 | bounce_you_are_disabled_warnings INTEGER, | ||
2466 | 76 | bounce_you_are_disabled_warnings_interval TEXT, | ||
2467 | 77 | -- Content filtering. | ||
2468 | 78 | filter_action INTEGER, | ||
2469 | 79 | filter_content BOOLEAN, | ||
2470 | 80 | collapse_alternatives BOOLEAN, | ||
2471 | 81 | convert_html_to_plaintext BOOLEAN, | ||
2472 | 82 | default_member_action INTEGER, | ||
2473 | 83 | default_nonmember_action INTEGER, | ||
2474 | 84 | description TEXT, | ||
2475 | 85 | digest_footer_uri TEXT, | ||
2476 | 86 | digest_header_uri TEXT, | ||
2477 | 87 | digest_is_default BOOLEAN, | ||
2478 | 88 | digest_send_periodic BOOLEAN, | ||
2479 | 89 | digest_size_threshold FLOAT, | ||
2480 | 90 | digest_volume_frequency INTEGER, | ||
2481 | 91 | digestable BOOLEAN, | ||
2482 | 92 | discard_these_nonmembers BLOB, | ||
2483 | 93 | emergency BOOLEAN, | ||
2484 | 94 | encode_ascii_prefixes BOOLEAN, | ||
2485 | 95 | first_strip_reply_to BOOLEAN, | ||
2486 | 96 | footer_uri TEXT, | ||
2487 | 97 | forward_auto_discards BOOLEAN, | ||
2488 | 98 | gateway_to_mail BOOLEAN, | ||
2489 | 99 | gateway_to_news BOOLEAN, | ||
2490 | 100 | goodbye_message_uri TEXT, | ||
2491 | 101 | header_matches BLOB, | ||
2492 | 102 | header_uri TEXT, | ||
2493 | 103 | hold_these_nonmembers BLOB, | ||
2494 | 104 | info TEXT, | ||
2495 | 105 | linked_newsgroup TEXT, | ||
2496 | 106 | max_days_to_hold INTEGER, | ||
2497 | 107 | max_message_size INTEGER, | ||
2498 | 108 | max_num_recipients INTEGER, | ||
2499 | 109 | member_moderation_notice TEXT, | ||
2500 | 110 | mime_is_default_digest BOOLEAN, | ||
2501 | 111 | moderator_password TEXT, | ||
2502 | 112 | new_member_options INTEGER, | ||
2503 | 113 | nondigestable BOOLEAN, | ||
2504 | 114 | nonmember_rejection_notice TEXT, | ||
2505 | 115 | obscure_addresses BOOLEAN, | ||
2506 | 116 | owner_chain TEXT, | ||
2507 | 117 | owner_pipeline TEXT, | ||
2508 | 118 | personalize INTEGER, | ||
2509 | 119 | post_id INTEGER, | ||
2510 | 120 | posting_chain TEXT, | ||
2511 | 121 | posting_pipeline TEXT, | ||
2512 | 122 | preferred_language TEXT, | ||
2513 | 123 | private_roster BOOLEAN, | ||
2514 | 124 | display_name TEXT, | ||
2515 | 125 | reject_these_nonmembers BLOB, | ||
2516 | 126 | reply_goes_to_list INTEGER, | ||
2517 | 127 | reply_to_address TEXT, | ||
2518 | 128 | require_explicit_destination BOOLEAN, | ||
2519 | 129 | respond_to_post_requests BOOLEAN, | ||
2520 | 130 | scrub_nondigest BOOLEAN, | ||
2521 | 131 | send_goodbye_message BOOLEAN, | ||
2522 | 132 | send_reminders BOOLEAN, | ||
2523 | 133 | send_welcome_message BOOLEAN, | ||
2524 | 134 | subject_prefix TEXT, | ||
2525 | 135 | subscribe_auto_approval BLOB, | ||
2526 | 136 | subscribe_policy INTEGER, | ||
2527 | 137 | topics BLOB, | ||
2528 | 138 | topics_bodylines_limit INTEGER, | ||
2529 | 139 | topics_enabled BOOLEAN, | ||
2530 | 140 | unsubscribe_policy INTEGER, | ||
2531 | 141 | welcome_message_uri TEXT, | ||
2532 | 142 | PRIMARY KEY (id) | ||
2533 | 143 | ); | ||
2534 | 144 | |||
2535 | 145 | INSERT INTO mailinglist_backup SELECT | ||
2536 | 146 | id, | ||
2537 | 147 | -- List identity | ||
2538 | 148 | list_name, | ||
2539 | 149 | mail_host, | ||
2540 | 150 | include_list_post_header, | ||
2541 | 151 | include_rfc2369_headers, | ||
2542 | 152 | -- Attributes not directly modifiable via the web u/i | ||
2543 | 153 | created_at, | ||
2544 | 154 | admin_member_chunksize, | ||
2545 | 155 | next_request_id, | ||
2546 | 156 | next_digest_number, | ||
2547 | 157 | digest_last_sent_at, | ||
2548 | 158 | volume, | ||
2549 | 159 | last_post_at, | ||
2550 | 160 | accept_these_nonmembers, | ||
2551 | 161 | acceptable_aliases_id, | ||
2552 | 162 | admin_immed_notify, | ||
2553 | 163 | admin_notify_mchanges, | ||
2554 | 164 | administrivia, | ||
2555 | 165 | advertised, | ||
2556 | 166 | anonymous_list, | ||
2557 | 167 | -- Automatic responses. | ||
2558 | 168 | autorespond_owner, | ||
2559 | 169 | autoresponse_owner_text, | ||
2560 | 170 | autorespond_postings, | ||
2561 | 171 | autoresponse_postings_text, | ||
2562 | 172 | autorespond_requests, | ||
2563 | 173 | autoresponse_request_text, | ||
2564 | 174 | autoresponse_grace_period, | ||
2565 | 175 | -- Bounces. | ||
2566 | 176 | forward_unrecognized_bounces_to, | ||
2567 | 177 | process_bounces, | ||
2568 | 178 | bounce_info_stale_after, | ||
2569 | 179 | bounce_matching_headers, | ||
2570 | 180 | bounce_notify_owner_on_disable, | ||
2571 | 181 | bounce_notify_owner_on_removal, | ||
2572 | 182 | bounce_score_threshold, | ||
2573 | 183 | bounce_you_are_disabled_warnings, | ||
2574 | 184 | bounce_you_are_disabled_warnings_interval, | ||
2575 | 185 | -- Content filtering. | ||
2576 | 186 | filter_action, | ||
2577 | 187 | filter_content, | ||
2578 | 188 | collapse_alternatives, | ||
2579 | 189 | convert_html_to_plaintext, | ||
2580 | 190 | default_member_action, | ||
2581 | 191 | default_nonmember_action, | ||
2582 | 192 | description, | ||
2583 | 193 | digest_footer_uri, | ||
2584 | 194 | digest_header_uri, | ||
2585 | 195 | digest_is_default, | ||
2586 | 196 | digest_send_periodic, | ||
2587 | 197 | digest_size_threshold, | ||
2588 | 198 | digest_volume_frequency, | ||
2589 | 199 | digestable, | ||
2590 | 200 | discard_these_nonmembers, | ||
2591 | 201 | emergency, | ||
2592 | 202 | encode_ascii_prefixes, | ||
2593 | 203 | first_strip_reply_to, | ||
2594 | 204 | footer_uri, | ||
2595 | 205 | forward_auto_discards, | ||
2596 | 206 | gateway_to_mail, | ||
2597 | 207 | gateway_to_news, | ||
2598 | 208 | goodbye_message_uri, | ||
2599 | 209 | header_matches, | ||
2600 | 210 | header_uri, | ||
2601 | 211 | hold_these_nonmembers, | ||
2602 | 212 | info, | ||
2603 | 213 | linked_newsgroup, | ||
2604 | 214 | max_days_to_hold, | ||
2605 | 215 | max_message_size, | ||
2606 | 216 | max_num_recipients, | ||
2607 | 217 | member_moderation_notice, | ||
2608 | 218 | mime_is_default_digest, | ||
2609 | 219 | moderator_password, | ||
2610 | 220 | new_member_options, | ||
2611 | 221 | nondigestable, | ||
2612 | 222 | nonmember_rejection_notice, | ||
2613 | 223 | obscure_addresses, | ||
2614 | 224 | owner_chain, | ||
2615 | 225 | owner_pipeline, | ||
2616 | 226 | personalize, | ||
2617 | 227 | post_id, | ||
2618 | 228 | posting_chain, | ||
2619 | 229 | posting_pipeline, | ||
2620 | 230 | preferred_language, | ||
2621 | 231 | private_roster, | ||
2622 | 232 | display_name, | ||
2623 | 233 | reject_these_nonmembers, | ||
2624 | 234 | reply_goes_to_list, | ||
2625 | 235 | reply_to_address, | ||
2626 | 236 | require_explicit_destination, | ||
2627 | 237 | respond_to_post_requests, | ||
2628 | 238 | scrub_nondigest, | ||
2629 | 239 | send_goodbye_message, | ||
2630 | 240 | send_reminders, | ||
2631 | 241 | send_welcome_message, | ||
2632 | 242 | subject_prefix, | ||
2633 | 243 | subscribe_auto_approval, | ||
2634 | 244 | subscribe_policy, | ||
2635 | 245 | topics, | ||
2636 | 246 | topics_bodylines_limit, | ||
2637 | 247 | topics_enabled, | ||
2638 | 248 | unsubscribe_policy, | ||
2639 | 249 | welcome_message_uri | ||
2640 | 250 | FROM mailinglist; | ||
2641 | 251 | |||
2642 | 252 | CREATE TABLE member_backup( | ||
2643 | 253 | id INTEGER NOT NULL, | ||
2644 | 254 | _member_id TEXT, | ||
2645 | 255 | role INTEGER, | ||
2646 | 256 | moderation_action INTEGER, | ||
2647 | 257 | address_id INTEGER, | ||
2648 | 258 | preferences_id INTEGER, | ||
2649 | 259 | user_id INTEGER, | ||
2650 | 260 | PRIMARY KEY (id) | ||
2651 | 261 | ); | ||
2652 | 262 | |||
2653 | 263 | INSERT INTO member_backup SELECT | ||
2654 | 264 | id, | ||
2655 | 265 | _member_id, | ||
2656 | 266 | role, | ||
2657 | 267 | moderation_action, | ||
2658 | 268 | address_id, | ||
2659 | 269 | preferences_id, | ||
2660 | 270 | user_id | ||
2661 | 271 | FROM member; | ||
2662 | 272 | |||
2663 | 273 | |||
2664 | 274 | -- Add the new columns. They'll get inserted at the Python layer. | ||
2665 | 275 | ALTER TABLE mailinglist_backup ADD COLUMN archive_policy INTEGER; | ||
2666 | 276 | ALTER TABLE mailinglist_backup ADD COLUMN list_id TEXT; | ||
2667 | 277 | ALTER TABLE mailinglist_backup ADD COLUMN nntp_prefix_subject_too INTEGER; | ||
2668 | 278 | ALTER TABLE mailinglist_backup ADD COLUMN newsgroup_moderation INTEGER; | ||
2669 | 279 | |||
2670 | 280 | ALTER TABLE member_backup ADD COLUMN list_id TEXT; | ||
2671 | 281 | 0 | ||
2672 | === removed file 'src/mailman/database/schema/sqlite_20121015000000_01.sql' | |||
2673 | --- src/mailman/database/schema/sqlite_20121015000000_01.sql 2013-09-01 15:15:08 +0000 | |||
2674 | +++ src/mailman/database/schema/sqlite_20121015000000_01.sql 1970-01-01 00:00:00 +0000 | |||
2675 | @@ -1,230 +0,0 @@ | |||
2676 | 1 | -- This file contains the sqlite3 schema migration from | ||
2677 | 2 | -- 3.0b2 TO 3.0b3 | ||
2678 | 3 | -- | ||
2679 | 4 | -- 3.0b3 has been released thus you MAY NOT edit this file. | ||
2680 | 5 | |||
2681 | 6 | -- REMOVALS from the ban table: | ||
2682 | 7 | -- REM mailing_list | ||
2683 | 8 | |||
2684 | 9 | -- ADDS to the ban table: | ||
2685 | 10 | -- ADD list_id | ||
2686 | 11 | |||
2687 | 12 | CREATE TABLE ban_backup ( | ||
2688 | 13 | id INTEGER NOT NULL, | ||
2689 | 14 | email TEXT, | ||
2690 | 15 | PRIMARY KEY (id) | ||
2691 | 16 | ); | ||
2692 | 17 | |||
2693 | 18 | INSERT INTO ban_backup SELECT | ||
2694 | 19 | id, email | ||
2695 | 20 | FROM ban; | ||
2696 | 21 | |||
2697 | 22 | ALTER TABLE ban_backup ADD COLUMN list_id TEXT; | ||
2698 | 23 | |||
2699 | 24 | -- REMOVALS from the mailinglist table. | ||
2700 | 25 | -- REM new_member_options | ||
2701 | 26 | -- REM send_reminders | ||
2702 | 27 | -- REM subscribe_policy | ||
2703 | 28 | -- REM unsubscribe_policy | ||
2704 | 29 | -- REM subscribe_auto_approval | ||
2705 | 30 | -- REM private_roster | ||
2706 | 31 | -- REM admin_member_chunksize | ||
2707 | 32 | |||
2708 | 33 | CREATE TABLE mailinglist_backup ( | ||
2709 | 34 | id INTEGER NOT NULL, | ||
2710 | 35 | list_name TEXT, | ||
2711 | 36 | mail_host TEXT, | ||
2712 | 37 | allow_list_posts BOOLEAN, | ||
2713 | 38 | include_rfc2369_headers BOOLEAN, | ||
2714 | 39 | created_at TIMESTAMP, | ||
2715 | 40 | next_request_id INTEGER, | ||
2716 | 41 | next_digest_number INTEGER, | ||
2717 | 42 | digest_last_sent_at TIMESTAMP, | ||
2718 | 43 | volume INTEGER, | ||
2719 | 44 | last_post_at TIMESTAMP, | ||
2720 | 45 | accept_these_nonmembers BLOB, | ||
2721 | 46 | acceptable_aliases_id INTEGER, | ||
2722 | 47 | admin_immed_notify BOOLEAN, | ||
2723 | 48 | admin_notify_mchanges BOOLEAN, | ||
2724 | 49 | administrivia BOOLEAN, | ||
2725 | 50 | advertised BOOLEAN, | ||
2726 | 51 | anonymous_list BOOLEAN, | ||
2727 | 52 | autorespond_owner INTEGER, | ||
2728 | 53 | autoresponse_owner_text TEXT, | ||
2729 | 54 | autorespond_postings INTEGER, | ||
2730 | 55 | autoresponse_postings_text TEXT, | ||
2731 | 56 | autorespond_requests INTEGER, | ||
2732 | 57 | autoresponse_request_text TEXT, | ||
2733 | 58 | autoresponse_grace_period TEXT, | ||
2734 | 59 | forward_unrecognized_bounces_to INTEGER, | ||
2735 | 60 | process_bounces BOOLEAN, | ||
2736 | 61 | bounce_info_stale_after TEXT, | ||
2737 | 62 | bounce_matching_headers TEXT, | ||
2738 | 63 | bounce_notify_owner_on_disable BOOLEAN, | ||
2739 | 64 | bounce_notify_owner_on_removal BOOLEAN, | ||
2740 | 65 | bounce_score_threshold INTEGER, | ||
2741 | 66 | bounce_you_are_disabled_warnings INTEGER, | ||
2742 | 67 | bounce_you_are_disabled_warnings_interval TEXT, | ||
2743 | 68 | filter_action INTEGER, | ||
2744 | 69 | filter_content BOOLEAN, | ||
2745 | 70 | collapse_alternatives BOOLEAN, | ||
2746 | 71 | convert_html_to_plaintext BOOLEAN, | ||
2747 | 72 | default_member_action INTEGER, | ||
2748 | 73 | default_nonmember_action INTEGER, | ||
2749 | 74 | description TEXT, | ||
2750 | 75 | digest_footer_uri TEXT, | ||
2751 | 76 | digest_header_uri TEXT, | ||
2752 | 77 | digest_is_default BOOLEAN, | ||
2753 | 78 | digest_send_periodic BOOLEAN, | ||
2754 | 79 | digest_size_threshold FLOAT, | ||
2755 | 80 | digest_volume_frequency INTEGER, | ||
2756 | 81 | digestable BOOLEAN, | ||
2757 | 82 | discard_these_nonmembers BLOB, | ||
2758 | 83 | emergency BOOLEAN, | ||
2759 | 84 | encode_ascii_prefixes BOOLEAN, | ||
2760 | 85 | first_strip_reply_to BOOLEAN, | ||
2761 | 86 | footer_uri TEXT, | ||
2762 | 87 | forward_auto_discards BOOLEAN, | ||
2763 | 88 | gateway_to_mail BOOLEAN, | ||
2764 | 89 | gateway_to_news BOOLEAN, | ||
2765 | 90 | goodbye_message_uri TEXT, | ||
2766 | 91 | header_matches BLOB, | ||
2767 | 92 | header_uri TEXT, | ||
2768 | 93 | hold_these_nonmembers BLOB, | ||
2769 | 94 | info TEXT, | ||
2770 | 95 | linked_newsgroup TEXT, | ||
2771 | 96 | max_days_to_hold INTEGER, | ||
2772 | 97 | max_message_size INTEGER, | ||
2773 | 98 | max_num_recipients INTEGER, | ||
2774 | 99 | member_moderation_notice TEXT, | ||
2775 | 100 | mime_is_default_digest BOOLEAN, | ||
2776 | 101 | moderator_password TEXT, | ||
2777 | 102 | nondigestable BOOLEAN, | ||
2778 | 103 | nonmember_rejection_notice TEXT, | ||
2779 | 104 | obscure_addresses BOOLEAN, | ||
2780 | 105 | owner_chain TEXT, | ||
2781 | 106 | owner_pipeline TEXT, | ||
2782 | 107 | personalize INTEGER, | ||
2783 | 108 | post_id INTEGER, | ||
2784 | 109 | posting_chain TEXT, | ||
2785 | 110 | posting_pipeline TEXT, | ||
2786 | 111 | preferred_language TEXT, | ||
2787 | 112 | display_name TEXT, | ||
2788 | 113 | reject_these_nonmembers BLOB, | ||
2789 | 114 | reply_goes_to_list INTEGER, | ||
2790 | 115 | reply_to_address TEXT, | ||
2791 | 116 | require_explicit_destination BOOLEAN, | ||
2792 | 117 | respond_to_post_requests BOOLEAN, | ||
2793 | 118 | scrub_nondigest BOOLEAN, | ||
2794 | 119 | send_goodbye_message BOOLEAN, | ||
2795 | 120 | send_welcome_message BOOLEAN, | ||
2796 | 121 | subject_prefix TEXT, | ||
2797 | 122 | topics BLOB, | ||
2798 | 123 | topics_bodylines_limit INTEGER, | ||
2799 | 124 | topics_enabled BOOLEAN, | ||
2800 | 125 | welcome_message_uri TEXT, | ||
2801 | 126 | archive_policy INTEGER, | ||
2802 | 127 | list_id TEXT, | ||
2803 | 128 | nntp_prefix_subject_too INTEGER, | ||
2804 | 129 | newsgroup_moderation INTEGER, | ||
2805 | 130 | PRIMARY KEY (id) | ||
2806 | 131 | ); | ||
2807 | 132 | |||
2808 | 133 | INSERT INTO mailinglist_backup SELECT | ||
2809 | 134 | id, | ||
2810 | 135 | list_name, | ||
2811 | 136 | mail_host, | ||
2812 | 137 | allow_list_posts, | ||
2813 | 138 | include_rfc2369_headers, | ||
2814 | 139 | created_at, | ||
2815 | 140 | next_request_id, | ||
2816 | 141 | next_digest_number, | ||
2817 | 142 | digest_last_sent_at, | ||
2818 | 143 | volume, | ||
2819 | 144 | last_post_at, | ||
2820 | 145 | accept_these_nonmembers, | ||
2821 | 146 | acceptable_aliases_id, | ||
2822 | 147 | admin_immed_notify, | ||
2823 | 148 | admin_notify_mchanges, | ||
2824 | 149 | administrivia, | ||
2825 | 150 | advertised, | ||
2826 | 151 | anonymous_list, | ||
2827 | 152 | autorespond_owner, | ||
2828 | 153 | autoresponse_owner_text, | ||
2829 | 154 | autorespond_postings, | ||
2830 | 155 | autoresponse_postings_text, | ||
2831 | 156 | autorespond_requests, | ||
2832 | 157 | autoresponse_request_text, | ||
2833 | 158 | autoresponse_grace_period, | ||
2834 | 159 | forward_unrecognized_bounces_to, | ||
2835 | 160 | process_bounces, | ||
2836 | 161 | bounce_info_stale_after, | ||
2837 | 162 | bounce_matching_headers, | ||
2838 | 163 | bounce_notify_owner_on_disable, | ||
2839 | 164 | bounce_notify_owner_on_removal, | ||
2840 | 165 | bounce_score_threshold, | ||
2841 | 166 | bounce_you_are_disabled_warnings, | ||
2842 | 167 | bounce_you_are_disabled_warnings_interval, | ||
2843 | 168 | filter_action, | ||
2844 | 169 | filter_content, | ||
2845 | 170 | collapse_alternatives, | ||
2846 | 171 | convert_html_to_plaintext, | ||
2847 | 172 | default_member_action, | ||
2848 | 173 | default_nonmember_action, | ||
2849 | 174 | description, | ||
2850 | 175 | digest_footer_uri, | ||
2851 | 176 | digest_header_uri, | ||
2852 | 177 | digest_is_default, | ||
2853 | 178 | digest_send_periodic, | ||
2854 | 179 | digest_size_threshold, | ||
2855 | 180 | digest_volume_frequency, | ||
2856 | 181 | digestable, | ||
2857 | 182 | discard_these_nonmembers, | ||
2858 | 183 | emergency, | ||
2859 | 184 | encode_ascii_prefixes, | ||
2860 | 185 | first_strip_reply_to, | ||
2861 | 186 | footer_uri, | ||
2862 | 187 | forward_auto_discards, | ||
2863 | 188 | gateway_to_mail, | ||
2864 | 189 | gateway_to_news, | ||
2865 | 190 | goodbye_message_uri, | ||
2866 | 191 | header_matches, | ||
2867 | 192 | header_uri, | ||
2868 | 193 | hold_these_nonmembers, | ||
2869 | 194 | info, | ||
2870 | 195 | linked_newsgroup, | ||
2871 | 196 | max_days_to_hold, | ||
2872 | 197 | max_message_size, | ||
2873 | 198 | max_num_recipients, | ||
2874 | 199 | member_moderation_notice, | ||
2875 | 200 | mime_is_default_digest, | ||
2876 | 201 | moderator_password, | ||
2877 | 202 | nondigestable, | ||
2878 | 203 | nonmember_rejection_notice, | ||
2879 | 204 | obscure_addresses, | ||
2880 | 205 | owner_chain, | ||
2881 | 206 | owner_pipeline, | ||
2882 | 207 | personalize, | ||
2883 | 208 | post_id, | ||
2884 | 209 | posting_chain, | ||
2885 | 210 | posting_pipeline, | ||
2886 | 211 | preferred_language, | ||
2887 | 212 | display_name, | ||
2888 | 213 | reject_these_nonmembers, | ||
2889 | 214 | reply_goes_to_list, | ||
2890 | 215 | reply_to_address, | ||
2891 | 216 | require_explicit_destination, | ||
2892 | 217 | respond_to_post_requests, | ||
2893 | 218 | scrub_nondigest, | ||
2894 | 219 | send_goodbye_message, | ||
2895 | 220 | send_welcome_message, | ||
2896 | 221 | subject_prefix, | ||
2897 | 222 | topics, | ||
2898 | 223 | topics_bodylines_limit, | ||
2899 | 224 | topics_enabled, | ||
2900 | 225 | welcome_message_uri, | ||
2901 | 226 | archive_policy, | ||
2902 | 227 | list_id, | ||
2903 | 228 | nntp_prefix_subject_too, | ||
2904 | 229 | newsgroup_moderation | ||
2905 | 230 | FROM mailinglist; | ||
2906 | 231 | 0 | ||
2907 | === removed file 'src/mailman/database/schema/sqlite_20130406000000_01.sql' | |||
2908 | --- src/mailman/database/schema/sqlite_20130406000000_01.sql 2013-11-26 02:26:15 +0000 | |||
2909 | +++ src/mailman/database/schema/sqlite_20130406000000_01.sql 1970-01-01 00:00:00 +0000 | |||
2910 | @@ -1,46 +0,0 @@ | |||
2911 | 1 | -- This file contains the SQLite schema migration from | ||
2912 | 2 | -- 3.0b3 to 3.0b4 | ||
2913 | 3 | -- | ||
2914 | 4 | -- After 3.0b4 is released you may not edit this file. | ||
2915 | 5 | |||
2916 | 6 | -- For SQLite3 migration strategy, see | ||
2917 | 7 | -- http://sqlite.org/faq.html#q11 | ||
2918 | 8 | |||
2919 | 9 | -- ADD listarchiver table. | ||
2920 | 10 | |||
2921 | 11 | -- REMOVALs from the bounceevent table: | ||
2922 | 12 | -- REM list_name | ||
2923 | 13 | |||
2924 | 14 | -- ADDs to the bounceevent table: | ||
2925 | 15 | -- ADD list_id | ||
2926 | 16 | |||
2927 | 17 | -- ADDs to the mailinglist table: | ||
2928 | 18 | -- ADD archiver_id | ||
2929 | 19 | |||
2930 | 20 | CREATE TABLE bounceevent_backup ( | ||
2931 | 21 | id INTEGER NOT NULL, | ||
2932 | 22 | email TEXT, | ||
2933 | 23 | 'timestamp' TIMESTAMP, | ||
2934 | 24 | message_id TEXT, | ||
2935 | 25 | context INTEGER, | ||
2936 | 26 | processed BOOLEAN, | ||
2937 | 27 | PRIMARY KEY (id) | ||
2938 | 28 | ); | ||
2939 | 29 | |||
2940 | 30 | INSERT INTO bounceevent_backup SELECT | ||
2941 | 31 | id, email, "timestamp", message_id, | ||
2942 | 32 | context, processed | ||
2943 | 33 | FROM bounceevent; | ||
2944 | 34 | |||
2945 | 35 | ALTER TABLE bounceevent_backup ADD COLUMN list_id TEXT; | ||
2946 | 36 | |||
2947 | 37 | CREATE TABLE listarchiver ( | ||
2948 | 38 | id INTEGER NOT NULL, | ||
2949 | 39 | mailing_list_id INTEGER NOT NULL, | ||
2950 | 40 | name TEXT NOT NULL, | ||
2951 | 41 | _is_enabled BOOLEAN, | ||
2952 | 42 | PRIMARY KEY (id) | ||
2953 | 43 | ); | ||
2954 | 44 | |||
2955 | 45 | CREATE INDEX ix_listarchiver_mailing_list_id | ||
2956 | 46 | ON listarchiver(mailing_list_id); | ||
2957 | 47 | 0 | ||
2958 | === modified file 'src/mailman/database/sqlite.py' | |||
2959 | --- src/mailman/database/sqlite.py 2014-01-01 14:59:42 +0000 | |||
2960 | +++ src/mailman/database/sqlite.py 2014-10-11 02:14:38 +0000 | |||
2961 | @@ -22,63 +22,27 @@ | |||
2962 | 22 | __metaclass__ = type | 22 | __metaclass__ = type |
2963 | 23 | __all__ = [ | 23 | __all__ = [ |
2964 | 24 | 'SQLiteDatabase', | 24 | 'SQLiteDatabase', |
2965 | 25 | 'make_temporary', | ||
2966 | 26 | ] | 25 | ] |
2967 | 27 | 26 | ||
2968 | 28 | 27 | ||
2969 | 29 | import os | 28 | import os |
2970 | 30 | import types | ||
2971 | 31 | import shutil | ||
2972 | 32 | import tempfile | ||
2973 | 33 | 29 | ||
2975 | 34 | from functools import partial | 30 | from mailman.database.base import SABaseDatabase |
2976 | 35 | from urlparse import urlparse | 31 | from urlparse import urlparse |
2977 | 36 | 32 | ||
2978 | 37 | from mailman.database.base import StormBaseDatabase | ||
2979 | 38 | from mailman.testing.helpers import configuration | ||
2980 | 39 | |||
2981 | 40 | 33 | ||
2982 | 41 | 34 | ||
2983 | 42 | 35 | ||
2985 | 43 | class SQLiteDatabase(StormBaseDatabase): | 36 | class SQLiteDatabase(SABaseDatabase): |
2986 | 44 | """Database class for SQLite.""" | 37 | """Database class for SQLite.""" |
2987 | 45 | 38 | ||
2988 | 46 | TAG = 'sqlite' | ||
2989 | 47 | |||
2990 | 48 | def _database_exists(self, store): | ||
2991 | 49 | """See `BaseDatabase`.""" | ||
2992 | 50 | table_query = 'select tbl_name from sqlite_master;' | ||
2993 | 51 | table_names = set(item[0] for item in | ||
2994 | 52 | store.execute(table_query)) | ||
2995 | 53 | return 'version' in table_names | ||
2996 | 54 | |||
2997 | 55 | def _prepare(self, url): | 39 | def _prepare(self, url): |
2998 | 56 | parts = urlparse(url) | 40 | parts = urlparse(url) |
2999 | 57 | assert parts.scheme == 'sqlite', ( | 41 | assert parts.scheme == 'sqlite', ( |
3000 | 58 | 'Database url mismatch (expected sqlite prefix): {0}'.format(url)) | 42 | 'Database url mismatch (expected sqlite prefix): {0}'.format(url)) |
3001 | 43 | # Ensure that the SQLite database file has the proper permissions, | ||
3002 | 44 | # since SQLite doesn't play nice with umask. | ||
3003 | 59 | path = os.path.normpath(parts.path) | 45 | path = os.path.normpath(parts.path) |
3005 | 60 | fd = os.open(path, os.O_WRONLY | os.O_NONBLOCK | os.O_CREAT, 0666) | 46 | fd = os.open(path, os.O_WRONLY | os.O_NONBLOCK | os.O_CREAT, 0o666) |
3006 | 61 | # Ignore errors | 47 | # Ignore errors |
3007 | 62 | if fd > 0: | 48 | if fd > 0: |
3008 | 63 | os.close(fd) | 49 | os.close(fd) |
3009 | 64 | |||
3010 | 65 | |||
3011 | 66 | |||
3012 | 67 | 50 | ||
3013 | 68 | # Test suite adapter for ITemporaryDatabase. | ||
3014 | 69 | |||
3015 | 70 | def _cleanup(self, tempdir): | ||
3016 | 71 | shutil.rmtree(tempdir) | ||
3017 | 72 | |||
3018 | 73 | |||
3019 | 74 | def make_temporary(database): | ||
3020 | 75 | """Adapts by monkey patching an existing SQLite IDatabase.""" | ||
3021 | 76 | tempdir = tempfile.mkdtemp() | ||
3022 | 77 | url = 'sqlite:///' + os.path.join(tempdir, 'mailman.db') | ||
3023 | 78 | with configuration('database', url=url): | ||
3024 | 79 | database.initialize() | ||
3025 | 80 | database._cleanup = types.MethodType( | ||
3026 | 81 | partial(_cleanup, tempdir=tempdir), | ||
3027 | 82 | database) | ||
3028 | 83 | # bool column values in SQLite must be integers. | ||
3029 | 84 | database.FALSE = 0 | ||
3030 | 85 | database.TRUE = 1 | ||
3031 | 86 | return database | ||
3032 | 87 | 51 | ||
3033 | === added directory 'src/mailman/database/tests' | |||
3034 | === removed directory 'src/mailman/database/tests' | |||
3035 | === added file 'src/mailman/database/tests/__init__.py' | |||
3036 | === removed file 'src/mailman/database/tests/__init__.py' | |||
3037 | === removed directory 'src/mailman/database/tests/data' | |||
3038 | === removed file 'src/mailman/database/tests/data/__init__.py' | |||
3039 | === removed file 'src/mailman/database/tests/data/mailman_01.db' | |||
3040 | 88 | Binary files src/mailman/database/tests/data/mailman_01.db 2012-04-20 21:32:27 +0000 and src/mailman/database/tests/data/mailman_01.db 1970-01-01 00:00:00 +0000 differ | 52 | Binary files src/mailman/database/tests/data/mailman_01.db 2012-04-20 21:32:27 +0000 and src/mailman/database/tests/data/mailman_01.db 1970-01-01 00:00:00 +0000 differ |
3041 | === removed file 'src/mailman/database/tests/data/migration_postgres_1.sql' | |||
3042 | --- src/mailman/database/tests/data/migration_postgres_1.sql 2012-07-26 04:22:19 +0000 | |||
3043 | +++ src/mailman/database/tests/data/migration_postgres_1.sql 1970-01-01 00:00:00 +0000 | |||
3044 | @@ -1,133 +0,0 @@ | |||
3045 | 1 | INSERT INTO "acceptablealias" VALUES(1,'foo@example.com',1); | ||
3046 | 2 | INSERT INTO "acceptablealias" VALUES(2,'bar@example.com',1); | ||
3047 | 3 | |||
3048 | 4 | INSERT INTO "address" VALUES( | ||
3049 | 5 | 1,'anne@example.com',NULL,'Anne Person', | ||
3050 | 6 | '2012-04-19 00:52:24.826432','2012-04-19 00:49:42.373769',1,2); | ||
3051 | 7 | INSERT INTO "address" VALUES( | ||
3052 | 8 | 2,'bart@example.com',NULL,'Bart Person', | ||
3053 | 9 | '2012-04-19 00:53:25.878800','2012-04-19 00:49:52.882050',2,4); | ||
3054 | 10 | |||
3055 | 11 | INSERT INTO "domain" VALUES( | ||
3056 | 12 | 1,'example.com','http://example.com',NULL,'postmaster@example.com'); | ||
3057 | 13 | |||
3058 | 14 | INSERT INTO "mailinglist" VALUES( | ||
3059 | 15 | -- id,list_name,mail_host,include_list_post_header,include_rfc2369_headers | ||
3060 | 16 | 1,'test','example.com',True,True, | ||
3061 | 17 | -- created_at,admin_member_chunksize,next_request_id,next_digest_number | ||
3062 | 18 | '2012-04-19 00:46:13.173844',30,1,1, | ||
3063 | 19 | -- digest_last_sent_at,volume,last_post_at,accept_these_nonmembers | ||
3064 | 20 | NULL,1,NULL,E'\\x80025D71012E', | ||
3065 | 21 | -- acceptable_aliases_id,admin_immed_notify,admin_notify_mchanges | ||
3066 | 22 | NULL,True,False, | ||
3067 | 23 | -- administrivia,advertised,anonymous_list,archive,archive_private | ||
3068 | 24 | True,True,False,True,False, | ||
3069 | 25 | -- archive_volume_frequency | ||
3070 | 26 | 1, | ||
3071 | 27 | --autorespond_owner,autoresponse_owner_text | ||
3072 | 28 | 0,'', | ||
3073 | 29 | -- autorespond_postings,autoresponse_postings_text | ||
3074 | 30 | 0,'', | ||
3075 | 31 | -- autorespond_requests,authoresponse_requests_text | ||
3076 | 32 | 0,'', | ||
3077 | 33 | -- autoresponse_grace_period | ||
3078 | 34 | '90 days, 0:00:00', | ||
3079 | 35 | -- forward_unrecognized_bounces_to,process_bounces | ||
3080 | 36 | 1,True, | ||
3081 | 37 | -- bounce_info_stale_after,bounce_matching_headers | ||
3082 | 38 | '7 days, 0:00:00',' | ||
3083 | 39 | # Lines that *start* with a ''#'' are comments. | ||
3084 | 40 | to: friend@public.com | ||
3085 | 41 | message-id: relay.comanche.denmark.eu | ||
3086 | 42 | from: list@listme.com | ||
3087 | 43 | from: .*@uplinkpro.com | ||
3088 | 44 | ', | ||
3089 | 45 | -- bounce_notify_owner_on_disable,bounce_notify_owner_on_removal | ||
3090 | 46 | True,True, | ||
3091 | 47 | -- bounce_score_threshold,bounce_you_are_disabled_warnings | ||
3092 | 48 | 5,3, | ||
3093 | 49 | -- bounce_you_are_disabled_warnings_interval | ||
3094 | 50 | '7 days, 0:00:00', | ||
3095 | 51 | -- filter_action,filter_content,collapse_alternatives | ||
3096 | 52 | 2,False,True, | ||
3097 | 53 | -- convert_html_to_plaintext,default_member_action,default_nonmember_action | ||
3098 | 54 | False,4,0, | ||
3099 | 55 | -- description | ||
3100 | 56 | '', | ||
3101 | 57 | -- digest_footer_uri | ||
3102 | 58 | 'mailman:///$listname/$language/footer-generic.txt', | ||
3103 | 59 | -- digest_header_uri | ||
3104 | 60 | NULL, | ||
3105 | 61 | -- digest_is_default,digest_send_periodic,digest_size_threshold | ||
3106 | 62 | False,True,30.0, | ||
3107 | 63 | -- digest_volume_frequency,digestable,discard_these_nonmembers | ||
3108 | 64 | 1,True,E'\\x80025D71012E', | ||
3109 | 65 | -- emergency,encode_ascii_prefixes,first_strip_reply_to | ||
3110 | 66 | False,False,False, | ||
3111 | 67 | -- footer_uri | ||
3112 | 68 | 'mailman:///$listname/$language/footer-generic.txt', | ||
3113 | 69 | -- forward_auto_discards,gateway_to_mail,gateway_to_news | ||
3114 | 70 | True,False,FAlse, | ||
3115 | 71 | -- generic_nonmember_action,goodby_message_uri | ||
3116 | 72 | 1,'', | ||
3117 | 73 | -- header_matches,header_uri,hold_these_nonmembers,info,linked_newsgroup | ||
3118 | 74 | E'\\x80025D71012E',NULL,E'\\x80025D71012E','','', | ||
3119 | 75 | -- max_days_to_hold,max_message_size,max_num_recipients | ||
3120 | 76 | 0,40,10, | ||
3121 | 77 | -- member_moderation_notice,mime_is_default_digest,moderator_password | ||
3122 | 78 | '',False,NULL, | ||
3123 | 79 | -- new_member_options,news_moderation,news_prefix_subject_too | ||
3124 | 80 | 256,0,True, | ||
3125 | 81 | -- nntp_host,nondigestable,nonmember_rejection_notice,obscure_addresses | ||
3126 | 82 | '',True,'',True, | ||
3127 | 83 | -- owner_chain,owner_pipeline,personalize,post_id | ||
3128 | 84 | 'default-owner-chain','default-owner-pipeline',0,1, | ||
3129 | 85 | -- posting_chain,posting_pipeline,preferred_language,private_roster | ||
3130 | 86 | 'default-posting-chain','default-posting-pipeline','en',True, | ||
3131 | 87 | -- display_name,reject_these_nonmembers | ||
3132 | 88 | 'Test',E'\\x80025D71012E', | ||
3133 | 89 | -- reply_goes_to_list,reply_to_address | ||
3134 | 90 | 0,'', | ||
3135 | 91 | -- require_explicit_destination,respond_to_post_requests | ||
3136 | 92 | True,True, | ||
3137 | 93 | -- scrub_nondigest,send_goodbye_message,send_reminders,send_welcome_message | ||
3138 | 94 | False,True,True,True, | ||
3139 | 95 | -- subject_prefix,subscribe_auto_approval | ||
3140 | 96 | '[Test] ',E'\\x80025D71012E', | ||
3141 | 97 | -- subscribe_policy,topics,topics_bodylines_limit,topics_enabled | ||
3142 | 98 | 1,E'\\x80025D71012E',5,False, | ||
3143 | 99 | -- unsubscribe_policy,welcome_message_uri | ||
3144 | 100 | 0,'mailman:///welcome.txt'); | ||
3145 | 101 | |||
3146 | 102 | INSERT INTO "member" VALUES( | ||
3147 | 103 | 1,'d1243f4d-e604-4f6b-af52-98d0a7bce0f1',1,'test@example.com',4,NULL,5,1); | ||
3148 | 104 | INSERT INTO "member" VALUES( | ||
3149 | 105 | 2,'dccc3851-fdfb-4afa-90cf-bdcbf80ad0fd',2,'test@example.com',3,NULL,6,1); | ||
3150 | 106 | INSERT INTO "member" VALUES( | ||
3151 | 107 | 3,'479be431-45f2-473d-bc3c-7eac614030ac',3,'test@example.com',3,NULL,7,2); | ||
3152 | 108 | INSERT INTO "member" VALUES( | ||
3153 | 109 | 4,'e2dc604c-d93a-4b91-b5a8-749e3caade36',1,'test@example.com',4,NULL,8,2); | ||
3154 | 110 | |||
3155 | 111 | INSERT INTO "preferences" VALUES(1,NULL,NULL,NULL,NULL,NULL,NULL,NULL); | ||
3156 | 112 | INSERT INTO "preferences" VALUES(2,NULL,NULL,NULL,NULL,NULL,NULL,NULL); | ||
3157 | 113 | INSERT INTO "preferences" VALUES(3,NULL,NULL,NULL,NULL,NULL,NULL,NULL); | ||
3158 | 114 | INSERT INTO "preferences" VALUES(4,NULL,NULL,NULL,NULL,NULL,NULL,NULL); | ||
3159 | 115 | INSERT INTO "preferences" VALUES(5,NULL,NULL,NULL,NULL,NULL,NULL,NULL); | ||
3160 | 116 | INSERT INTO "preferences" VALUES(6,NULL,NULL,NULL,NULL,NULL,NULL,NULL); | ||
3161 | 117 | INSERT INTO "preferences" VALUES(7,NULL,NULL,NULL,NULL,NULL,NULL,NULL); | ||
3162 | 118 | INSERT INTO "preferences" VALUES(8,NULL,NULL,NULL,NULL,NULL,NULL,NULL); | ||
3163 | 119 | |||
3164 | 120 | INSERT INTO "user" VALUES( | ||
3165 | 121 | 1,'Anne Person',NULL,'0adf3caa-6f26-46f8-a11d-5256c8148592', | ||
3166 | 122 | '2012-04-19 00:49:42.370493',1,1); | ||
3167 | 123 | INSERT INTO "user" VALUES( | ||
3168 | 124 | 2,'Bart Person',NULL,'63f5d1a2-e533-4055-afe4-475dec3b1163', | ||
3169 | 125 | '2012-04-19 00:49:52.868746',2,3); | ||
3170 | 126 | |||
3171 | 127 | INSERT INTO "uid" VALUES(1,'8bf9a615-f23e-4980-b7d1-90ac0203c66f'); | ||
3172 | 128 | INSERT INTO "uid" VALUES(2,'0adf3caa-6f26-46f8-a11d-5256c8148592'); | ||
3173 | 129 | INSERT INTO "uid" VALUES(3,'63f5d1a2-e533-4055-afe4-475dec3b1163'); | ||
3174 | 130 | INSERT INTO "uid" VALUES(4,'d1243f4d-e604-4f6b-af52-98d0a7bce0f1'); | ||
3175 | 131 | INSERT INTO "uid" VALUES(5,'dccc3851-fdfb-4afa-90cf-bdcbf80ad0fd'); | ||
3176 | 132 | INSERT INTO "uid" VALUES(6,'479be431-45f2-473d-bc3c-7eac614030ac'); | ||
3177 | 133 | INSERT INTO "uid" VALUES(7,'e2dc604c-d93a-4b91-b5a8-749e3caade36'); | ||
3178 | 134 | 0 | ||
3179 | === removed file 'src/mailman/database/tests/data/migration_sqlite_1.sql' | |||
3180 | --- src/mailman/database/tests/data/migration_sqlite_1.sql 2012-07-26 04:22:19 +0000 | |||
3181 | +++ src/mailman/database/tests/data/migration_sqlite_1.sql 1970-01-01 00:00:00 +0000 | |||
3182 | @@ -1,133 +0,0 @@ | |||
3183 | 1 | INSERT INTO "acceptablealias" VALUES(1,'foo@example.com',1); | ||
3184 | 2 | INSERT INTO "acceptablealias" VALUES(2,'bar@example.com',1); | ||
3185 | 3 | |||
3186 | 4 | INSERT INTO "address" VALUES( | ||
3187 | 5 | 1,'anne@example.com',NULL,'Anne Person', | ||
3188 | 6 | '2012-04-19 00:52:24.826432','2012-04-19 00:49:42.373769',1,2); | ||
3189 | 7 | INSERT INTO "address" VALUES( | ||
3190 | 8 | 2,'bart@example.com',NULL,'Bart Person', | ||
3191 | 9 | '2012-04-19 00:53:25.878800','2012-04-19 00:49:52.882050',2,4); | ||
3192 | 10 | |||
3193 | 11 | INSERT INTO "domain" VALUES( | ||
3194 | 12 | 1,'example.com','http://example.com',NULL,'postmaster@example.com'); | ||
3195 | 13 | |||
3196 | 14 | INSERT INTO "mailinglist" VALUES( | ||
3197 | 15 | -- id,list_name,mail_host,include_list_post_header,include_rfc2369_headers | ||
3198 | 16 | 1,'test','example.com',1,1, | ||
3199 | 17 | -- created_at,admin_member_chunksize,next_request_id,next_digest_number | ||
3200 | 18 | '2012-04-19 00:46:13.173844',30,1,1, | ||
3201 | 19 | -- digest_last_sent_at,volume,last_post_at,accept_these_nonmembers | ||
3202 | 20 | NULL,1,NULL,X'80025D71012E', | ||
3203 | 21 | -- acceptable_aliases_id,admin_immed_notify,admin_notify_mchanges | ||
3204 | 22 | NULL,1,0, | ||
3205 | 23 | -- administrivia,advertised,anonymous_list,archive,archive_private | ||
3206 | 24 | 1,1,0,1,0, | ||
3207 | 25 | -- archive_volume_frequency | ||
3208 | 26 | 1, | ||
3209 | 27 | --autorespond_owner,autoresponse_owner_text | ||
3210 | 28 | 0,'', | ||
3211 | 29 | -- autorespond_postings,autoresponse_postings_text | ||
3212 | 30 | 0,'', | ||
3213 | 31 | -- autorespond_requests,authoresponse_requests_text | ||
3214 | 32 | 0,'', | ||
3215 | 33 | -- autoresponse_grace_period | ||
3216 | 34 | '90 days, 0:00:00', | ||
3217 | 35 | -- forward_unrecognized_bounces_to,process_bounces | ||
3218 | 36 | 1,1, | ||
3219 | 37 | -- bounce_info_stale_after,bounce_matching_headers | ||
3220 | 38 | '7 days, 0:00:00',' | ||
3221 | 39 | # Lines that *start* with a ''#'' are comments. | ||
3222 | 40 | to: friend@public.com | ||
3223 | 41 | message-id: relay.comanche.denmark.eu | ||
3224 | 42 | from: list@listme.com | ||
3225 | 43 | from: .*@uplinkpro.com | ||
3226 | 44 | ', | ||
3227 | 45 | -- bounce_notify_owner_on_disable,bounce_notify_owner_on_removal | ||
3228 | 46 | 1,1, | ||
3229 | 47 | -- bounce_score_threshold,bounce_you_are_disabled_warnings | ||
3230 | 48 | 5,3, | ||
3231 | 49 | -- bounce_you_are_disabled_warnings_interval | ||
3232 | 50 | '7 days, 0:00:00', | ||
3233 | 51 | -- filter_action,filter_content,collapse_alternatives | ||
3234 | 52 | 2,0,1, | ||
3235 | 53 | -- convert_html_to_plaintext,default_member_action,default_nonmember_action | ||
3236 | 54 | 0,4,0, | ||
3237 | 55 | -- description | ||
3238 | 56 | '', | ||
3239 | 57 | -- digest_footer_uri | ||
3240 | 58 | 'mailman:///$listname/$language/footer-generic.txt', | ||
3241 | 59 | -- digest_header_uri | ||
3242 | 60 | NULL, | ||
3243 | 61 | -- digest_is_default,digest_send_periodic,digest_size_threshold | ||
3244 | 62 | 0,1,30.0, | ||
3245 | 63 | -- digest_volume_frequency,digestable,discard_these_nonmembers | ||
3246 | 64 | 1,1,X'80025D71012E', | ||
3247 | 65 | -- emergency,encode_ascii_prefixes,first_strip_reply_to | ||
3248 | 66 | 0,0,0, | ||
3249 | 67 | -- footer_uri | ||
3250 | 68 | 'mailman:///$listname/$language/footer-generic.txt', | ||
3251 | 69 | -- forward_auto_discards,gateway_to_mail,gateway_to_news | ||
3252 | 70 | 1,0,0, | ||
3253 | 71 | -- generic_nonmember_action,goodby_message_uri | ||
3254 | 72 | 1,'', | ||
3255 | 73 | -- header_matches,header_uri,hold_these_nonmembers,info,linked_newsgroup | ||
3256 | 74 | X'80025D71012E',NULL,X'80025D71012E','','', | ||
3257 | 75 | -- max_days_to_hold,max_message_size,max_num_recipients | ||
3258 | 76 | 0,40,10, | ||
3259 | 77 | -- member_moderation_notice,mime_is_default_digest,moderator_password | ||
3260 | 78 | '',0,NULL, | ||
3261 | 79 | -- new_member_options,news_moderation,news_prefix_subject_too | ||
3262 | 80 | 256,0,1, | ||
3263 | 81 | -- nntp_host,nondigestable,nonmember_rejection_notice,obscure_addresses | ||
3264 | 82 | '',1,'',1, | ||
3265 | 83 | -- owner_chain,owner_pipeline,personalize,post_id | ||
3266 | 84 | 'default-owner-chain','default-owner-pipeline',0,1, | ||
3267 | 85 | -- posting_chain,posting_pipeline,preferred_language,private_roster | ||
3268 | 86 | 'default-posting-chain','default-posting-pipeline','en',1, | ||
3269 | 87 | -- display_name,reject_these_nonmembers | ||
3270 | 88 | 'Test',X'80025D71012E', | ||
3271 | 89 | -- reply_goes_to_list,reply_to_address | ||
3272 | 90 | 0,'', | ||
3273 | 91 | -- require_explicit_destination,respond_to_post_requests | ||
3274 | 92 | 1,1, | ||
3275 | 93 | -- scrub_nondigest,send_goodbye_message,send_reminders,send_welcome_message | ||
3276 | 94 | 0,1,1,1, | ||
3277 | 95 | -- subject_prefix,subscribe_auto_approval | ||
3278 | 96 | '[Test] ',X'80025D71012E', | ||
3279 | 97 | -- subscribe_policy,topics,topics_bodylines_limit,topics_enabled | ||
3280 | 98 | 1,X'80025D71012E',5,0, | ||
3281 | 99 | -- unsubscribe_policy,welcome_message_uri | ||
3282 | 100 | 0,'mailman:///welcome.txt'); | ||
3283 | 101 | |||
3284 | 102 | INSERT INTO "member" VALUES( | ||
3285 | 103 | 1,'d1243f4d-e604-4f6b-af52-98d0a7bce0f1',1,'test@example.com',4,NULL,5,1); | ||
3286 | 104 | INSERT INTO "member" VALUES( | ||
3287 | 105 | 2,'dccc3851-fdfb-4afa-90cf-bdcbf80ad0fd',2,'test@example.com',3,NULL,6,1); | ||
3288 | 106 | INSERT INTO "member" VALUES( | ||
3289 | 107 | 3,'479be431-45f2-473d-bc3c-7eac614030ac',3,'test@example.com',3,NULL,7,2); | ||
3290 | 108 | INSERT INTO "member" VALUES( | ||
3291 | 109 | 4,'e2dc604c-d93a-4b91-b5a8-749e3caade36',1,'test@example.com',4,NULL,8,2); | ||
3292 | 110 | |||
3293 | 111 | INSERT INTO "preferences" VALUES(1,NULL,NULL,NULL,NULL,NULL,NULL,NULL); | ||
3294 | 112 | INSERT INTO "preferences" VALUES(2,NULL,NULL,NULL,NULL,NULL,NULL,NULL); | ||
3295 | 113 | INSERT INTO "preferences" VALUES(3,NULL,NULL,NULL,NULL,NULL,NULL,NULL); | ||
3296 | 114 | INSERT INTO "preferences" VALUES(4,NULL,NULL,NULL,NULL,NULL,NULL,NULL); | ||
3297 | 115 | INSERT INTO "preferences" VALUES(5,NULL,NULL,NULL,NULL,NULL,NULL,NULL); | ||
3298 | 116 | INSERT INTO "preferences" VALUES(6,NULL,NULL,NULL,NULL,NULL,NULL,NULL); | ||
3299 | 117 | INSERT INTO "preferences" VALUES(7,NULL,NULL,NULL,NULL,NULL,NULL,NULL); | ||
3300 | 118 | INSERT INTO "preferences" VALUES(8,NULL,NULL,NULL,NULL,NULL,NULL,NULL); | ||
3301 | 119 | |||
3302 | 120 | INSERT INTO "user" VALUES( | ||
3303 | 121 | 1,'Anne Person',NULL,'0adf3caa-6f26-46f8-a11d-5256c8148592', | ||
3304 | 122 | '2012-04-19 00:49:42.370493',1,1); | ||
3305 | 123 | INSERT INTO "user" VALUES( | ||
3306 | 124 | 2,'Bart Person',NULL,'63f5d1a2-e533-4055-afe4-475dec3b1163', | ||
3307 | 125 | '2012-04-19 00:49:52.868746',2,3); | ||
3308 | 126 | |||
3309 | 127 | INSERT INTO "uid" VALUES(1,'8bf9a615-f23e-4980-b7d1-90ac0203c66f'); | ||
3310 | 128 | INSERT INTO "uid" VALUES(2,'0adf3caa-6f26-46f8-a11d-5256c8148592'); | ||
3311 | 129 | INSERT INTO "uid" VALUES(3,'63f5d1a2-e533-4055-afe4-475dec3b1163'); | ||
3312 | 130 | INSERT INTO "uid" VALUES(4,'d1243f4d-e604-4f6b-af52-98d0a7bce0f1'); | ||
3313 | 131 | INSERT INTO "uid" VALUES(5,'dccc3851-fdfb-4afa-90cf-bdcbf80ad0fd'); | ||
3314 | 132 | INSERT INTO "uid" VALUES(6,'479be431-45f2-473d-bc3c-7eac614030ac'); | ||
3315 | 133 | INSERT INTO "uid" VALUES(7,'e2dc604c-d93a-4b91-b5a8-749e3caade36'); | ||
3316 | 134 | 0 | ||
3317 | === added file 'src/mailman/database/tests/test_factory.py' | |||
3318 | --- src/mailman/database/tests/test_factory.py 1970-01-01 00:00:00 +0000 | |||
3319 | +++ src/mailman/database/tests/test_factory.py 2014-10-11 02:14:38 +0000 | |||
3320 | @@ -0,0 +1,162 @@ | |||
3321 | 1 | # Copyright (C) 2013-2014 by the Free Software Foundation, Inc. | ||
3322 | 2 | # | ||
3323 | 3 | # This file is part of GNU Mailman. | ||
3324 | 4 | # | ||
3325 | 5 | # GNU Mailman is free software: you can redistribute it and/or modify it under | ||
3326 | 6 | # the terms of the GNU General Public License as published by the Free | ||
3327 | 7 | # Software Foundation, either version 3 of the License, or (at your option) | ||
3328 | 8 | # any later version. | ||
3329 | 9 | # | ||
3330 | 10 | # GNU Mailman is distributed in the hope that it will be useful, but WITHOUT | ||
3331 | 11 | # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or | ||
3332 | 12 | # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for | ||
3333 | 13 | # more details. | ||
3334 | 14 | # | ||
3335 | 15 | # You should have received a copy of the GNU General Public License along with | ||
3336 | 16 | # GNU Mailman. If not, see <http://www.gnu.org/licenses/>. | ||
3337 | 17 | |||
3338 | 18 | """Test database schema migrations""" | ||
3339 | 19 | |||
3340 | 20 | from __future__ import absolute_import, print_function, unicode_literals | ||
3341 | 21 | |||
3342 | 22 | __metaclass__ = type | ||
3343 | 23 | __all__ = [ | ||
3344 | 24 | ] | ||
3345 | 25 | |||
3346 | 26 | |||
3347 | 27 | import unittest | ||
3348 | 28 | import types | ||
3349 | 29 | |||
3350 | 30 | import alembic.command | ||
3351 | 31 | from mock import Mock | ||
3352 | 32 | from sqlalchemy import MetaData, Table, Column, Integer, Unicode | ||
3353 | 33 | from sqlalchemy.schema import Index | ||
3354 | 34 | from sqlalchemy.exc import ProgrammingError, OperationalError | ||
3355 | 35 | |||
3356 | 36 | from mailman.config import config | ||
3357 | 37 | from mailman.testing.layers import ConfigLayer | ||
3358 | 38 | from mailman.database.factory import SchemaManager, _reset | ||
3359 | 39 | from mailman.database.sqlite import SQLiteDatabase | ||
3360 | 40 | from mailman.database.alembic import alembic_cfg | ||
3361 | 41 | from mailman.database.model import Model | ||
3362 | 42 | |||
3363 | 43 | |||
3364 | 44 | |||
3365 | 0 | 45 | ||
3366 | 46 | class TestSchemaManager(unittest.TestCase): | ||
3367 | 47 | |||
3368 | 48 | layer = ConfigLayer | ||
3369 | 49 | |||
3370 | 50 | def setUp(self): | ||
3371 | 51 | # Drop the existing database | ||
3372 | 52 | Model.metadata.drop_all(config.db.engine) | ||
3373 | 53 | md = MetaData() | ||
3374 | 54 | md.reflect(bind=config.db.engine) | ||
3375 | 55 | for tablename in ("alembic_version", "version"): | ||
3376 | 56 | if tablename in md.tables: | ||
3377 | 57 | md.tables[tablename].drop(config.db.engine) | ||
3378 | 58 | self.schema_mgr = SchemaManager(config.db) | ||
3379 | 59 | |||
3380 | 60 | def tearDown(self): | ||
3381 | 61 | self._drop_storm_database() | ||
3382 | 62 | # Restore a virgin DB | ||
3383 | 63 | Model.metadata.create_all(config.db.engine) | ||
3384 | 64 | |||
3385 | 65 | |||
3386 | 66 | def _table_exists(self, tablename): | ||
3387 | 67 | md = MetaData() | ||
3388 | 68 | md.reflect(bind=config.db.engine) | ||
3389 | 69 | return tablename in md.tables | ||
3390 | 70 | |||
3391 | 71 | def _create_storm_database(self, revision): | ||
3392 | 72 | version_table = Table("version", Model.metadata, | ||
3393 | 73 | Column("id", Integer, primary_key=True), | ||
3394 | 74 | Column("component", Unicode), | ||
3395 | 75 | Column("version", Unicode), | ||
3396 | 76 | ) | ||
3397 | 77 | version_table.create(config.db.engine) | ||
3398 | 78 | config.db.store.execute(version_table.insert().values( | ||
3399 | 79 | component='schema', version=revision)) | ||
3400 | 80 | config.db.commit() | ||
3401 | 81 | # Other Storm specific changes, those SQL statements hopefully work on | ||
3402 | 82 | # all DB engines... | ||
3403 | 83 | config.db.engine.execute( | ||
3404 | 84 | "ALTER TABLE mailinglist ADD COLUMN acceptable_aliases_id INT") | ||
3405 | 85 | Index("ix_user__user_id").drop(bind=config.db.engine) | ||
3406 | 86 | # Don't pollute our main metadata object, create a new one | ||
3407 | 87 | md = MetaData() | ||
3408 | 88 | user_table = Model.metadata.tables["user"].tometadata(md) | ||
3409 | 89 | Index("ix_user_user_id", user_table.c._user_id | ||
3410 | 90 | ).create(bind=config.db.engine) | ||
3411 | 91 | config.db.commit() | ||
3412 | 92 | |||
3413 | 93 | def _drop_storm_database(self): | ||
3414 | 94 | """ | ||
3415 | 95 | Remove the leftovers from a Storm DB. | ||
3416 | 96 | (you must issue a drop_all() afterwards) | ||
3417 | 97 | """ | ||
3418 | 98 | if "version" in Model.metadata.tables: | ||
3419 | 99 | version = Model.metadata.tables["version"] | ||
3420 | 100 | version.drop(config.db.engine, checkfirst=True) | ||
3421 | 101 | Model.metadata.remove(version) | ||
3422 | 102 | try: | ||
3423 | 103 | Index("ix_user_user_id").drop(bind=config.db.engine) | ||
3424 | 104 | except (ProgrammingError, OperationalError) as e: | ||
3425 | 105 | # non-existant (PGSQL raises a ProgrammingError, while SQLite | ||
3426 | 106 | # raises an OperationalError) | ||
3427 | 107 | pass | ||
3428 | 108 | config.db.commit() | ||
3429 | 109 | |||
3430 | 110 | |||
3431 | 111 | def test_current_db(self): | ||
3432 | 112 | """The database is already at the latest version""" | ||
3433 | 113 | alembic.command.stamp(alembic_cfg, "head") | ||
3434 | 114 | self.schema_mgr._create = Mock() | ||
3435 | 115 | self.schema_mgr._upgrade = Mock() | ||
3436 | 116 | self.schema_mgr.setup_db() | ||
3437 | 117 | self.assertFalse(self.schema_mgr._create.called) | ||
3438 | 118 | self.assertFalse(self.schema_mgr._upgrade.called) | ||
3439 | 119 | |||
3440 | 120 | def test_initial(self): | ||
3441 | 121 | """No existing database""" | ||
3442 | 122 | self.assertFalse(self._table_exists("mailinglist")) | ||
3443 | 123 | self.assertFalse(self._table_exists("alembic_version")) | ||
3444 | 124 | self.schema_mgr._upgrade = Mock() | ||
3445 | 125 | self.schema_mgr.setup_db() | ||
3446 | 126 | self.assertFalse(self.schema_mgr._upgrade.called) | ||
3447 | 127 | self.assertTrue(self._table_exists("mailinglist")) | ||
3448 | 128 | self.assertTrue(self._table_exists("alembic_version")) | ||
3449 | 129 | |||
3450 | 130 | def test_storm(self): | ||
3451 | 131 | """Existing Storm database""" | ||
3452 | 132 | Model.metadata.create_all(config.db.engine) | ||
3453 | 133 | self._create_storm_database( | ||
3454 | 134 | self.schema_mgr.LAST_STORM_SCHEMA_VERSION) | ||
3455 | 135 | self.schema_mgr._create = Mock() | ||
3456 | 136 | self.schema_mgr.setup_db() | ||
3457 | 137 | self.assertFalse(self.schema_mgr._create.called) | ||
3458 | 138 | self.assertTrue(self._table_exists("mailinglist") | ||
3459 | 139 | and self._table_exists("alembic_version") | ||
3460 | 140 | and not self._table_exists("version")) | ||
3461 | 141 | |||
3462 | 142 | def test_old_storm(self): | ||
3463 | 143 | """Existing Storm database in an old version""" | ||
3464 | 144 | Model.metadata.create_all(config.db.engine) | ||
3465 | 145 | self._create_storm_database("001") | ||
3466 | 146 | self.schema_mgr._create = Mock() | ||
3467 | 147 | self.assertRaises(RuntimeError, self.schema_mgr.setup_db) | ||
3468 | 148 | self.assertFalse(self.schema_mgr._create.called) | ||
3469 | 149 | |||
3470 | 150 | def test_old_db(self): | ||
3471 | 151 | """The database is in an old revision, must upgrade""" | ||
3472 | 152 | alembic.command.stamp(alembic_cfg, "head") | ||
3473 | 153 | md = MetaData() | ||
3474 | 154 | md.reflect(bind=config.db.engine) | ||
3475 | 155 | config.db.store.execute(md.tables["alembic_version"].delete()) | ||
3476 | 156 | config.db.store.execute(md.tables["alembic_version"].insert().values( | ||
3477 | 157 | version_num="dummyrevision")) | ||
3478 | 158 | config.db.commit() | ||
3479 | 159 | self.schema_mgr._create = Mock() | ||
3480 | 160 | self.schema_mgr._upgrade = Mock() | ||
3481 | 161 | self.schema_mgr.setup_db() | ||
3482 | 162 | self.assertFalse(self.schema_mgr._create.called) | ||
3483 | 163 | self.assertTrue(self.schema_mgr._upgrade.called) | ||
3484 | 1 | 164 | ||
3485 | === removed file 'src/mailman/database/tests/test_migrations.py' | |||
3486 | --- src/mailman/database/tests/test_migrations.py 2014-01-01 14:59:42 +0000 | |||
3487 | +++ src/mailman/database/tests/test_migrations.py 1970-01-01 00:00:00 +0000 | |||
3488 | @@ -1,506 +0,0 @@ | |||
3489 | 1 | # Copyright (C) 2012-2014 by the Free Software Foundation, Inc. | ||
3490 | 2 | # | ||
3491 | 3 | # This file is part of GNU Mailman. | ||
3492 | 4 | # | ||
3493 | 5 | # GNU Mailman is free software: you can redistribute it and/or modify it under | ||
3494 | 6 | # the terms of the GNU General Public License as published by the Free | ||
3495 | 7 | # Software Foundation, either version 3 of the License, or (at your option) | ||
3496 | 8 | # any later version. | ||
3497 | 9 | # | ||
3498 | 10 | # GNU Mailman is distributed in the hope that it will be useful, but WITHOUT | ||
3499 | 11 | # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or | ||
3500 | 12 | # FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for | ||
3501 | 13 | # more details. | ||
3502 | 14 | # | ||
3503 | 15 | # You should have received a copy of the GNU General Public License along with | ||
3504 | 16 | # GNU Mailman. If not, see <http://www.gnu.org/licenses/>. | ||
3505 | 17 | |||
3506 | 18 | """Test schema migrations.""" | ||
3507 | 19 | |||
3508 | 20 | from __future__ import absolute_import, print_function, unicode_literals | ||
3509 | 21 | |||
3510 | 22 | __metaclass__ = type | ||
3511 | 23 | __all__ = [ | ||
3512 | 24 | 'TestMigration20120407MigratedData', | ||
3513 | 25 | 'TestMigration20120407Schema', | ||
3514 | 26 | 'TestMigration20120407UnchangedData', | ||
3515 | 27 | 'TestMigration20121015MigratedData', | ||
3516 | 28 | 'TestMigration20121015Schema', | ||
3517 | 29 | 'TestMigration20130406MigratedData', | ||
3518 | 30 | 'TestMigration20130406Schema', | ||
3519 | 31 | ] | ||
3520 | 32 | |||
3521 | 33 | |||
3522 | 34 | import unittest | ||
3523 | 35 | |||
3524 | 36 | from datetime import datetime | ||
3525 | 37 | from operator import attrgetter | ||
3526 | 38 | from pkg_resources import resource_string | ||
3527 | 39 | from sqlite3 import OperationalError | ||
3528 | 40 | from storm.exceptions import DatabaseError | ||
3529 | 41 | from zope.component import getUtility | ||
3530 | 42 | |||
3531 | 43 | from mailman.interfaces.database import IDatabaseFactory | ||
3532 | 44 | from mailman.interfaces.domain import IDomainManager | ||
3533 | 45 | from mailman.interfaces.archiver import ArchivePolicy | ||
3534 | 46 | from mailman.interfaces.bounce import BounceContext | ||
3535 | 47 | from mailman.interfaces.listmanager import IListManager | ||
3536 | 48 | from mailman.interfaces.mailinglist import IAcceptableAliasSet | ||
3537 | 49 | from mailman.interfaces.nntp import NewsgroupModeration | ||
3538 | 50 | from mailman.interfaces.subscriptions import ISubscriptionService | ||
3539 | 51 | from mailman.model.bans import Ban | ||
3540 | 52 | from mailman.model.bounce import BounceEvent | ||
3541 | 53 | from mailman.testing.helpers import temporary_db | ||
3542 | 54 | from mailman.testing.layers import ConfigLayer | ||
3543 | 55 | |||
3544 | 56 | |||
3545 | 57 | |||
3546 | 58 | 0 | ||
3547 | 59 | class MigrationTestBase(unittest.TestCase): | ||
3548 | 60 | """Test database migrations.""" | ||
3549 | 61 | |||
3550 | 62 | layer = ConfigLayer | ||
3551 | 63 | |||
3552 | 64 | def setUp(self): | ||
3553 | 65 | self._database = getUtility(IDatabaseFactory, 'temporary').create() | ||
3554 | 66 | |||
3555 | 67 | def tearDown(self): | ||
3556 | 68 | self._database._cleanup() | ||
3557 | 69 | |||
3558 | 70 | def _table_missing_present(self, migrations, missing, present): | ||
3559 | 71 | """The appropriate migrations leave some tables missing and present. | ||
3560 | 72 | |||
3561 | 73 | :param migrations: Sequence of migrations to load. | ||
3562 | 74 | :param missing: Tables which should be missing. | ||
3563 | 75 | :param present: Tables which should be present. | ||
3564 | 76 | """ | ||
3565 | 77 | for migration in migrations: | ||
3566 | 78 | self._database.load_migrations(migration) | ||
3567 | 79 | self._database.store.commit() | ||
3568 | 80 | for table in missing: | ||
3569 | 81 | self.assertRaises(OperationalError, | ||
3570 | 82 | self._database.store.execute, | ||
3571 | 83 | 'select * from {};'.format(table)) | ||
3572 | 84 | for table in present: | ||
3573 | 85 | self._database.store.execute('select * from {};'.format(table)) | ||
3574 | 86 | |||
3575 | 87 | def _missing_present(self, table, migrations, missing, present): | ||
3576 | 88 | """The appropriate migrations leave columns missing and present. | ||
3577 | 89 | |||
3578 | 90 | :param table: The table to test columns from. | ||
3579 | 91 | :param migrations: Sequence of migrations to load. | ||
3580 | 92 | :param missing: Set of columns which should be missing after the | ||
3581 | 93 | migrations are loaded. | ||
3582 | 94 | :param present: Set of columns which should be present after the | ||
3583 | 95 | migrations are loaded. | ||
3584 | 96 | """ | ||
3585 | 97 | for migration in migrations: | ||
3586 | 98 | self._database.load_migrations(migration) | ||
3587 | 99 | self._database.store.commit() | ||
3588 | 100 | for column in missing: | ||
3589 | 101 | self.assertRaises(DatabaseError, | ||
3590 | 102 | self._database.store.execute, | ||
3591 | 103 | 'select {0} from {1};'.format(column, table)) | ||
3592 | 104 | self._database.store.rollback() | ||
3593 | 105 | for column in present: | ||
3594 | 106 | # This should not produce an exception. Is there some better test | ||
3595 | 107 | # that we can perform? | ||
3596 | 108 | self._database.store.execute( | ||
3597 | 109 | 'select {0} from {1};'.format(column, table)) | ||
3598 | 110 | |||
3599 | 111 | |||
3600 | 112 | |||
3601 | 113 | 1 | ||
3602 | 114 | class TestMigration20120407Schema(MigrationTestBase): | ||
3603 | 115 | """Test column migrations.""" | ||
3604 | 116 | |||
3605 | 117 | def test_pre_upgrade_columns_migration(self): | ||
3606 | 118 | # Test that before the migration, the old table columns are present | ||
3607 | 119 | # and the new database columns are not. | ||
3608 | 120 | self._missing_present('mailinglist', | ||
3609 | 121 | ['20120406999999'], | ||
3610 | 122 | # New columns are missing. | ||
3611 | 123 | ('allow_list_posts', | ||
3612 | 124 | 'archive_policy', | ||
3613 | 125 | 'list_id', | ||
3614 | 126 | 'nntp_prefix_subject_too'), | ||
3615 | 127 | # Old columns are present. | ||
3616 | 128 | ('archive', | ||
3617 | 129 | 'archive_private', | ||
3618 | 130 | 'archive_volume_frequency', | ||
3619 | 131 | 'generic_nonmember_action', | ||
3620 | 132 | 'include_list_post_header', | ||
3621 | 133 | 'news_moderation', | ||
3622 | 134 | 'news_prefix_subject_too', | ||
3623 | 135 | 'nntp_host')) | ||
3624 | 136 | self._missing_present('member', | ||
3625 | 137 | ['20120406999999'], | ||
3626 | 138 | ('list_id',), | ||
3627 | 139 | ('mailing_list',)) | ||
3628 | 140 | |||
3629 | 141 | def test_post_upgrade_columns_migration(self): | ||
3630 | 142 | # Test that after the migration, the old table columns are missing | ||
3631 | 143 | # and the new database columns are present. | ||
3632 | 144 | self._missing_present('mailinglist', | ||
3633 | 145 | ['20120406999999', | ||
3634 | 146 | '20120407000000'], | ||
3635 | 147 | # The old columns are missing. | ||
3636 | 148 | ('archive', | ||
3637 | 149 | 'archive_private', | ||
3638 | 150 | 'archive_volume_frequency', | ||
3639 | 151 | 'generic_nonmember_action', | ||
3640 | 152 | 'include_list_post_header', | ||
3641 | 153 | 'news_moderation', | ||
3642 | 154 | 'news_prefix_subject_too', | ||
3643 | 155 | 'nntp_host'), | ||
3644 | 156 | # The new columns are present. | ||
3645 | 157 | ('allow_list_posts', | ||
3646 | 158 | 'archive_policy', | ||
3647 | 159 | 'list_id', | ||
3648 | 160 | 'nntp_prefix_subject_too')) | ||
3649 | 161 | self._missing_present('member', | ||
3650 | 162 | ['20120406999999', | ||
3651 | 163 | '20120407000000'], | ||
3652 | 164 | ('mailing_list',), | ||
3653 | 165 | ('list_id',)) | ||
3654 | 166 | |||
3655 | 167 | |||
3656 | 168 | |||
3657 | 169 | 2 | ||
3658 | 170 | class TestMigration20120407UnchangedData(MigrationTestBase): | ||
3659 | 171 | """Test non-migrated data.""" | ||
3660 | 172 | |||
3661 | 173 | def setUp(self): | ||
3662 | 174 | MigrationTestBase.setUp(self) | ||
3663 | 175 | # Load all the migrations to just before the one we're testing. | ||
3664 | 176 | self._database.load_migrations('20120406999999') | ||
3665 | 177 | # Load the previous schema's sample data. | ||
3666 | 178 | sample_data = resource_string( | ||
3667 | 179 | 'mailman.database.tests.data', | ||
3668 | 180 | 'migration_{0}_1.sql'.format(self._database.TAG)) | ||
3669 | 181 | self._database.load_sql(self._database.store, sample_data) | ||
3670 | 182 | # XXX 2012-12-28: We have to load the last migration defined in the | ||
3671 | 183 | # system, otherwise the ORM model will not match the SQL table | ||
3672 | 184 | # definitions and we'll get OperationalErrors from SQLite. | ||
3673 | 185 | self._database.load_migrations('20121015000000') | ||
3674 | 186 | |||
3675 | 187 | def test_migration_domains(self): | ||
3676 | 188 | # Test that the domains table, which isn't touched, doesn't change. | ||
3677 | 189 | with temporary_db(self._database): | ||
3678 | 190 | # Check that the domains survived the migration. This table | ||
3679 | 191 | # was not touched so it should be fine. | ||
3680 | 192 | domains = list(getUtility(IDomainManager)) | ||
3681 | 193 | self.assertEqual(len(domains), 1) | ||
3682 | 194 | self.assertEqual(domains[0].mail_host, 'example.com') | ||
3683 | 195 | |||
3684 | 196 | def test_migration_mailing_lists(self): | ||
3685 | 197 | # Test that the mailing lists survive migration. | ||
3686 | 198 | with temporary_db(self._database): | ||
3687 | 199 | # There should be exactly one mailing list defined. | ||
3688 | 200 | mlists = list(getUtility(IListManager).mailing_lists) | ||
3689 | 201 | self.assertEqual(len(mlists), 1) | ||
3690 | 202 | self.assertEqual(mlists[0].fqdn_listname, 'test@example.com') | ||
3691 | 203 | |||
3692 | 204 | def test_migration_acceptable_aliases(self): | ||
3693 | 205 | # Test that the mailing list's acceptable aliases survive migration. | ||
3694 | 206 | # This proves that foreign key references are migrated properly. | ||
3695 | 207 | with temporary_db(self._database): | ||
3696 | 208 | mlist = getUtility(IListManager).get('test@example.com') | ||
3697 | 209 | aliases_set = IAcceptableAliasSet(mlist) | ||
3698 | 210 | self.assertEqual(set(aliases_set.aliases), | ||
3699 | 211 | set(['foo@example.com', 'bar@example.com'])) | ||
3700 | 212 | |||
3701 | 213 | def test_migration_members(self): | ||
3702 | 214 | # Test that the members of a mailing list all survive migration. | ||
3703 | 215 | with temporary_db(self._database): | ||
3704 | 216 | mlist = getUtility(IListManager).get('test@example.com') | ||
3705 | 217 | # Test that all the members we expect are still there. Start with | ||
3706 | 218 | # the two list delivery members. | ||
3707 | 219 | addresses = set(address.email | ||
3708 | 220 | for address in mlist.members.addresses) | ||
3709 | 221 | self.assertEqual(addresses, | ||
3710 | 222 | set(['anne@example.com', 'bart@example.com'])) | ||
3711 | 223 | # There is one owner. | ||
3712 | 224 | owners = set(address.email for address in mlist.owners.addresses) | ||
3713 | 225 | self.assertEqual(len(owners), 1) | ||
3714 | 226 | self.assertEqual(owners.pop(), 'anne@example.com') | ||
3715 | 227 | # There is one moderator. | ||
3716 | 228 | moderators = set(address.email | ||
3717 | 229 | for address in mlist.moderators.addresses) | ||
3718 | 230 | self.assertEqual(len(moderators), 1) | ||
3719 | 231 | self.assertEqual(moderators.pop(), 'bart@example.com') | ||
3720 | 232 | |||
3721 | 233 | |||
3722 | 234 | |||
3723 | 235 | 3 | ||
3724 | 236 | class TestMigration20120407MigratedData(MigrationTestBase): | ||
3725 | 237 | """Test affected migration data.""" | ||
3726 | 238 | |||
3727 | 239 | def setUp(self): | ||
3728 | 240 | MigrationTestBase.setUp(self) | ||
3729 | 241 | # Load all the migrations to just before the one we're testing. | ||
3730 | 242 | self._database.load_migrations('20120406999999') | ||
3731 | 243 | # Load the previous schema's sample data. | ||
3732 | 244 | sample_data = resource_string( | ||
3733 | 245 | 'mailman.database.tests.data', | ||
3734 | 246 | 'migration_{0}_1.sql'.format(self._database.TAG)) | ||
3735 | 247 | self._database.load_sql(self._database.store, sample_data) | ||
3736 | 248 | |||
3737 | 249 | def _upgrade(self): | ||
3738 | 250 | # XXX 2012-12-28: We have to load the last migration defined in the | ||
3739 | 251 | # system, otherwise the ORM model will not match the SQL table | ||
3740 | 252 | # definitions and we'll get OperationalErrors from SQLite. | ||
3741 | 253 | self._database.load_migrations('20121015000000') | ||
3742 | 254 | |||
3743 | 255 | def test_migration_archive_policy_never_0(self): | ||
3744 | 256 | # Test that the new archive_policy value is updated correctly. In the | ||
3745 | 257 | # case of old column archive=0, the archive_private column is | ||
3746 | 258 | # ignored. This test sets it to 0 to ensure it's ignored. | ||
3747 | 259 | self._database.store.execute( | ||
3748 | 260 | 'UPDATE mailinglist SET archive = {0}, archive_private = {0} ' | ||
3749 | 261 | 'WHERE id = 1;'.format(self._database.FALSE)) | ||
3750 | 262 | # Complete the migration | ||
3751 | 263 | self._upgrade() | ||
3752 | 264 | with temporary_db(self._database): | ||
3753 | 265 | mlist = getUtility(IListManager).get('test@example.com') | ||
3754 | 266 | self.assertEqual(mlist.archive_policy, ArchivePolicy.never) | ||
3755 | 267 | |||
3756 | 268 | def test_migration_archive_policy_never_1(self): | ||
3757 | 269 | # Test that the new archive_policy value is updated correctly. In the | ||
3758 | 270 | # case of old column archive=0, the archive_private column is | ||
3759 | 271 | # ignored. This test sets it to 1 to ensure it's ignored. | ||
3760 | 272 | self._database.store.execute( | ||
3761 | 273 | 'UPDATE mailinglist SET archive = {0}, archive_private = {1} ' | ||
3762 | 274 | 'WHERE id = 1;'.format(self._database.FALSE, | ||
3763 | 275 | self._database.TRUE)) | ||
3764 | 276 | # Complete the migration | ||
3765 | 277 | self._upgrade() | ||
3766 | 278 | with temporary_db(self._database): | ||
3767 | 279 | mlist = getUtility(IListManager).get('test@example.com') | ||
3768 | 280 | self.assertEqual(mlist.archive_policy, ArchivePolicy.never) | ||
3769 | 281 | |||
3770 | 282 | def test_archive_policy_private(self): | ||
3771 | 283 | # Test that the new archive_policy value is updated correctly for | ||
3772 | 284 | # private archives. | ||
3773 | 285 | self._database.store.execute( | ||
3774 | 286 | 'UPDATE mailinglist SET archive = {0}, archive_private = {0} ' | ||
3775 | 287 | 'WHERE id = 1;'.format(self._database.TRUE)) | ||
3776 | 288 | # Complete the migration | ||
3777 | 289 | self._upgrade() | ||
3778 | 290 | with temporary_db(self._database): | ||
3779 | 291 | mlist = getUtility(IListManager).get('test@example.com') | ||
3780 | 292 | self.assertEqual(mlist.archive_policy, ArchivePolicy.private) | ||
3781 | 293 | |||
3782 | 294 | def test_archive_policy_public(self): | ||
3783 | 295 | # Test that the new archive_policy value is updated correctly for | ||
3784 | 296 | # public archives. | ||
3785 | 297 | self._database.store.execute( | ||
3786 | 298 | 'UPDATE mailinglist SET archive = {1}, archive_private = {0} ' | ||
3787 | 299 | 'WHERE id = 1;'.format(self._database.FALSE, | ||
3788 | 300 | self._database.TRUE)) | ||
3789 | 301 | # Complete the migration | ||
3790 | 302 | self._upgrade() | ||
3791 | 303 | with temporary_db(self._database): | ||
3792 | 304 | mlist = getUtility(IListManager).get('test@example.com') | ||
3793 | 305 | self.assertEqual(mlist.archive_policy, ArchivePolicy.public) | ||
3794 | 306 | |||
3795 | 307 | def test_list_id(self): | ||
3796 | 308 | # Test that the mailinglist table gets a list_id column. | ||
3797 | 309 | self._upgrade() | ||
3798 | 310 | with temporary_db(self._database): | ||
3799 | 311 | mlist = getUtility(IListManager).get('test@example.com') | ||
3800 | 312 | self.assertEqual(mlist.list_id, 'test.example.com') | ||
3801 | 313 | |||
3802 | 314 | def test_list_id_member(self): | ||
3803 | 315 | # Test that the member table's mailing_list column becomes list_id. | ||
3804 | 316 | self._upgrade() | ||
3805 | 317 | with temporary_db(self._database): | ||
3806 | 318 | service = getUtility(ISubscriptionService) | ||
3807 | 319 | members = list(service.find_members(list_id='test.example.com')) | ||
3808 | 320 | self.assertEqual(len(members), 4) | ||
3809 | 321 | |||
3810 | 322 | def test_news_moderation_none(self): | ||
3811 | 323 | # Test that news_moderation becomes newsgroup_moderation. | ||
3812 | 324 | self._database.store.execute( | ||
3813 | 325 | 'UPDATE mailinglist SET news_moderation = 0 ' | ||
3814 | 326 | 'WHERE id = 1;') | ||
3815 | 327 | self._upgrade() | ||
3816 | 328 | with temporary_db(self._database): | ||
3817 | 329 | mlist = getUtility(IListManager).get('test@example.com') | ||
3818 | 330 | self.assertEqual(mlist.newsgroup_moderation, | ||
3819 | 331 | NewsgroupModeration.none) | ||
3820 | 332 | |||
3821 | 333 | def test_news_moderation_open_moderated(self): | ||
3822 | 334 | # Test that news_moderation becomes newsgroup_moderation. | ||
3823 | 335 | self._database.store.execute( | ||
3824 | 336 | 'UPDATE mailinglist SET news_moderation = 1 ' | ||
3825 | 337 | 'WHERE id = 1;') | ||
3826 | 338 | self._upgrade() | ||
3827 | 339 | with temporary_db(self._database): | ||
3828 | 340 | mlist = getUtility(IListManager).get('test@example.com') | ||
3829 | 341 | self.assertEqual(mlist.newsgroup_moderation, | ||
3830 | 342 | NewsgroupModeration.open_moderated) | ||
3831 | 343 | |||
3832 | 344 | def test_news_moderation_moderated(self): | ||
3833 | 345 | # Test that news_moderation becomes newsgroup_moderation. | ||
3834 | 346 | self._database.store.execute( | ||
3835 | 347 | 'UPDATE mailinglist SET news_moderation = 2 ' | ||
3836 | 348 | 'WHERE id = 1;') | ||
3837 | 349 | self._upgrade() | ||
3838 | 350 | with temporary_db(self._database): | ||
3839 | 351 | mlist = getUtility(IListManager).get('test@example.com') | ||
3840 | 352 | self.assertEqual(mlist.newsgroup_moderation, | ||
3841 | 353 | NewsgroupModeration.moderated) | ||
3842 | 354 | |||
3843 | 355 | def test_nntp_prefix_subject_too_false(self): | ||
3844 | 356 | # Test that news_prefix_subject_too becomes nntp_prefix_subject_too. | ||
3845 | 357 | self._database.store.execute( | ||
3846 | 358 | 'UPDATE mailinglist SET news_prefix_subject_too = {0} ' | ||
3847 | 359 | 'WHERE id = 1;'.format(self._database.FALSE)) | ||
3848 | 360 | self._upgrade() | ||
3849 | 361 | with temporary_db(self._database): | ||
3850 | 362 | mlist = getUtility(IListManager).get('test@example.com') | ||
3851 | 363 | self.assertFalse(mlist.nntp_prefix_subject_too) | ||
3852 | 364 | |||
3853 | 365 | def test_nntp_prefix_subject_too_true(self): | ||
3854 | 366 | # Test that news_prefix_subject_too becomes nntp_prefix_subject_too. | ||
3855 | 367 | self._database.store.execute( | ||
3856 | 368 | 'UPDATE mailinglist SET news_prefix_subject_too = {0} ' | ||
3857 | 369 | 'WHERE id = 1;'.format(self._database.TRUE)) | ||
3858 | 370 | self._upgrade() | ||
3859 | 371 | with temporary_db(self._database): | ||
3860 | 372 | mlist = getUtility(IListManager).get('test@example.com') | ||
3861 | 373 | self.assertTrue(mlist.nntp_prefix_subject_too) | ||
3862 | 374 | |||
3863 | 375 | def test_allow_list_posts_false(self): | ||
3864 | 376 | # Test that include_list_post_header -> allow_list_posts. | ||
3865 | 377 | self._database.store.execute( | ||
3866 | 378 | 'UPDATE mailinglist SET include_list_post_header = {0} ' | ||
3867 | 379 | 'WHERE id = 1;'.format(self._database.FALSE)) | ||
3868 | 380 | self._upgrade() | ||
3869 | 381 | with temporary_db(self._database): | ||
3870 | 382 | mlist = getUtility(IListManager).get('test@example.com') | ||
3871 | 383 | self.assertFalse(mlist.allow_list_posts) | ||
3872 | 384 | |||
3873 | 385 | def test_allow_list_posts_true(self): | ||
3874 | 386 | # Test that include_list_post_header -> allow_list_posts. | ||
3875 | 387 | self._database.store.execute( | ||
3876 | 388 | 'UPDATE mailinglist SET include_list_post_header = {0} ' | ||
3877 | 389 | 'WHERE id = 1;'.format(self._database.TRUE)) | ||
3878 | 390 | self._upgrade() | ||
3879 | 391 | with temporary_db(self._database): | ||
3880 | 392 | mlist = getUtility(IListManager).get('test@example.com') | ||
3881 | 393 | self.assertTrue(mlist.allow_list_posts) | ||
3882 | 394 | |||
3883 | 395 | |||
3884 | 396 | |||
3885 | 397 | 4 | ||
3886 | 398 | class TestMigration20121015Schema(MigrationTestBase): | ||
3887 | 399 | """Test column migrations.""" | ||
3888 | 400 | |||
3889 | 401 | def test_pre_upgrade_column_migrations(self): | ||
3890 | 402 | self._missing_present('ban', | ||
3891 | 403 | ['20121014999999'], | ||
3892 | 404 | ('list_id',), | ||
3893 | 405 | ('mailing_list',)) | ||
3894 | 406 | self._missing_present('mailinglist', | ||
3895 | 407 | ['20121014999999'], | ||
3896 | 408 | (), | ||
3897 | 409 | ('new_member_options', 'send_reminders', | ||
3898 | 410 | 'subscribe_policy', 'unsubscribe_policy', | ||
3899 | 411 | 'subscribe_auto_approval', 'private_roster', | ||
3900 | 412 | 'admin_member_chunksize'), | ||
3901 | 413 | ) | ||
3902 | 414 | |||
3903 | 415 | def test_post_upgrade_column_migrations(self): | ||
3904 | 416 | self._missing_present('ban', | ||
3905 | 417 | ['20121014999999', | ||
3906 | 418 | '20121015000000'], | ||
3907 | 419 | ('mailing_list',), | ||
3908 | 420 | ('list_id',)) | ||
3909 | 421 | self._missing_present('mailinglist', | ||
3910 | 422 | ['20121014999999', | ||
3911 | 423 | '20121015000000'], | ||
3912 | 424 | ('new_member_options', 'send_reminders', | ||
3913 | 425 | 'subscribe_policy', 'unsubscribe_policy', | ||
3914 | 426 | 'subscribe_auto_approval', 'private_roster', | ||
3915 | 427 | 'admin_member_chunksize'), | ||
3916 | 428 | ()) | ||
3917 | 429 | |||
3918 | 430 | |||
3919 | 431 | |||
3920 | 432 | 5 | ||
3921 | 433 | class TestMigration20121015MigratedData(MigrationTestBase): | ||
3922 | 434 | """Test non-migrated data.""" | ||
3923 | 435 | |||
3924 | 436 | def test_migration_bans(self): | ||
3925 | 437 | # Load all the migrations to just before the one we're testing. | ||
3926 | 438 | self._database.load_migrations('20121014999999') | ||
3927 | 439 | # Insert a list-specific ban. | ||
3928 | 440 | self._database.store.execute(""" | ||
3929 | 441 | INSERT INTO ban VALUES ( | ||
3930 | 442 | 1, 'anne@example.com', 'test@example.com'); | ||
3931 | 443 | """) | ||
3932 | 444 | # Insert a global ban. | ||
3933 | 445 | self._database.store.execute(""" | ||
3934 | 446 | INSERT INTO ban VALUES ( | ||
3935 | 447 | 2, 'bart@example.com', NULL); | ||
3936 | 448 | """) | ||
3937 | 449 | # Update to the current migration we're testing. | ||
3938 | 450 | self._database.load_migrations('20121015000000') | ||
3939 | 451 | # Now both the local and global bans should still be present. | ||
3940 | 452 | bans = sorted(self._database.store.find(Ban), | ||
3941 | 453 | key=attrgetter('email')) | ||
3942 | 454 | self.assertEqual(bans[0].email, 'anne@example.com') | ||
3943 | 455 | self.assertEqual(bans[0].list_id, 'test.example.com') | ||
3944 | 456 | self.assertEqual(bans[1].email, 'bart@example.com') | ||
3945 | 457 | self.assertEqual(bans[1].list_id, None) | ||
3946 | 458 | |||
3947 | 459 | |||
3948 | 460 | |||
3949 | 461 | 6 | ||
3950 | 462 | class TestMigration20130406Schema(MigrationTestBase): | ||
3951 | 463 | """Test column migrations.""" | ||
3952 | 464 | |||
3953 | 465 | def test_pre_upgrade_column_migrations(self): | ||
3954 | 466 | self._missing_present('bounceevent', | ||
3955 | 467 | ['20130405999999'], | ||
3956 | 468 | ('list_id',), | ||
3957 | 469 | ('list_name',)) | ||
3958 | 470 | |||
3959 | 471 | def test_post_upgrade_column_migrations(self): | ||
3960 | 472 | self._missing_present('bounceevent', | ||
3961 | 473 | ['20130405999999', | ||
3962 | 474 | '20130406000000'], | ||
3963 | 475 | ('list_name',), | ||
3964 | 476 | ('list_id',)) | ||
3965 | 477 | |||
3966 | 478 | def test_pre_listarchiver_table(self): | ||
3967 | 479 | self._table_missing_present(['20130405999999'], ('listarchiver',), ()) | ||
3968 | 480 | |||
3969 | 481 | def test_post_listarchiver_table(self): | ||
3970 | 482 | self._table_missing_present(['20130405999999', | ||
3971 | 483 | '20130406000000'], | ||
3972 | 484 | (), | ||
3973 | 485 | ('listarchiver',)) | ||
3974 | 486 | |||
3975 | 487 | |||
3976 | 488 | |||
3977 | 489 | 7 | ||
3978 | 490 | class TestMigration20130406MigratedData(MigrationTestBase): | ||
3979 | 491 | """Test migrated data.""" | ||
3980 | 492 | |||
3981 | 493 | def test_migration_bounceevent(self): | ||
3982 | 494 | # Load all migrations to just before the one we're testing. | ||
3983 | 495 | self._database.load_migrations('20130405999999') | ||
3984 | 496 | # Insert a bounce event. | ||
3985 | 497 | self._database.store.execute(""" | ||
3986 | 498 | INSERT INTO bounceevent VALUES ( | ||
3987 | 499 | 1, 'test@example.com', 'anne@example.com', | ||
3988 | 500 | '2013-04-06 21:12:00', '<abc@example.com>', | ||
3989 | 501 | 1, 0); | ||
3990 | 502 | """) | ||
3991 | 503 | # Update to the current migration we're testing | ||
3992 | 504 | self._database.load_migrations('20130406000000') | ||
3993 | 505 | # The bounce event should exist, but with a list-id instead of a fqdn | ||
3994 | 506 | # list name. | ||
3995 | 507 | events = list(self._database.store.find(BounceEvent)) | ||
3996 | 508 | self.assertEqual(len(events), 1) | ||
3997 | 509 | self.assertEqual(events[0].list_id, 'test.example.com') | ||
3998 | 510 | self.assertEqual(events[0].email, 'anne@example.com') | ||
3999 | 511 | self.assertEqual(events[0].timestamp, datetime(2013, 4, 6, 21, 12)) | ||
4000 | 512 | self.assertEqual(events[0].message_id, '<abc@example.com>') | ||
4001 | 513 | self.assertEqual(events[0].context, BounceContext.normal) | ||
4002 | 514 | self.assertFalse(events[0].processed) | ||
4003 | 515 | 8 | ||
4004 | === modified file 'src/mailman/database/types.py' | |||
4005 | --- src/mailman/database/types.py 2014-04-28 15:23:35 +0000 | |||
4006 | +++ src/mailman/database/types.py 2014-10-11 02:14:38 +0000 | |||
4007 | @@ -23,43 +23,70 @@ | |||
4008 | 23 | __metaclass__ = type | 23 | __metaclass__ = type |
4009 | 24 | __all__ = [ | 24 | __all__ = [ |
4010 | 25 | 'Enum', | 25 | 'Enum', |
4011 | 26 | 'UUID', | ||
4012 | 26 | ] | 27 | ] |
4013 | 27 | 28 | ||
4014 | 29 | import uuid | ||
4015 | 28 | 30 | ||
4018 | 29 | from storm.properties import SimpleProperty | 31 | from sqlalchemy import Integer |
4019 | 30 | from storm.variables import Variable | 32 | from sqlalchemy.dialects import postgresql |
4020 | 33 | from sqlalchemy.types import TypeDecorator, CHAR | ||
4021 | 31 | 34 | ||
4022 | 32 | 35 | ||
4023 | 33 | 36 | ||
4024 | 34 | 37 | ||
4027 | 35 | class _EnumVariable(Variable): | 38 | class Enum(TypeDecorator): |
4028 | 36 | """Storm variable for supporting enum types. | 39 | """Handle Python 3.4 style enums. |
4029 | 37 | 40 | ||
4031 | 38 | To use this, make the database column a INTEGER. | 41 | Stores an integer-based Enum as an integer in the database, and |
4032 | 42 | converts it on-the-fly. | ||
4033 | 39 | """ | 43 | """ |
4051 | 40 | 44 | impl = Integer | |
4052 | 41 | def __init__(self, *args, **kws): | 45 | |
4053 | 42 | self._enum = kws.pop('enum') | 46 | def __init__(self, enum, *args, **kw): |
4054 | 43 | super(_EnumVariable, self).__init__(*args, **kws) | 47 | self.enum = enum |
4055 | 44 | 48 | super(Enum, self).__init__(*args, **kw) | |
4056 | 45 | def parse_set(self, value, from_db): | 49 | |
4057 | 46 | if value is None: | 50 | def process_bind_param(self, value, dialect): |
4058 | 47 | return None | 51 | if value is None: |
4059 | 48 | if not from_db: | 52 | return None |
4043 | 49 | return value | ||
4044 | 50 | return self._enum(value) | ||
4045 | 51 | |||
4046 | 52 | def parse_get(self, value, to_db): | ||
4047 | 53 | if value is None: | ||
4048 | 54 | return None | ||
4049 | 55 | if not to_db: | ||
4050 | 56 | return value | ||
4060 | 57 | return value.value | 53 | return value.value |
4061 | 58 | 54 | ||
4070 | 59 | 55 | def process_result_value(self, value, dialect): | |
4071 | 60 | class Enum(SimpleProperty): | 56 | if value is None: |
4072 | 61 | """Custom type for Storm supporting enums.""" | 57 | return None |
4073 | 62 | 58 | return self.enum(value) | |
4074 | 63 | variable_class = _EnumVariable | 59 | |
4075 | 64 | 60 | ||
4076 | 65 | def __init__(self, enum=None): | 61 | |
4069 | 66 | super(Enum, self).__init__(enum=enum) | ||
4077 | 67 | 62 | ||
4078 | 63 | class UUID(TypeDecorator): | ||
4079 | 64 | """Platform-independent GUID type. | ||
4080 | 65 | |||
4081 | 66 | Uses Postgresql's UUID type, otherwise uses | ||
4082 | 67 | CHAR(32), storing as stringified hex values. | ||
4083 | 68 | |||
4084 | 69 | """ | ||
4085 | 70 | impl = CHAR | ||
4086 | 71 | |||
4087 | 72 | def load_dialect_impl(self, dialect): | ||
4088 | 73 | if dialect.name == 'postgresql': | ||
4089 | 74 | return dialect.type_descriptor(postgresql.UUID()) | ||
4090 | 75 | else: | ||
4091 | 76 | return dialect.type_descriptor(CHAR(32)) | ||
4092 | 77 | |||
4093 | 78 | def process_bind_param(self, value, dialect): | ||
4094 | 79 | if value is None: | ||
4095 | 80 | return value | ||
4096 | 81 | elif dialect.name == 'postgresql': | ||
4097 | 82 | return str(value) | ||
4098 | 83 | else: | ||
4099 | 84 | if not isinstance(value, uuid.UUID): | ||
4100 | 85 | return "%.32x" % uuid.UUID(value) | ||
4101 | 86 | else: | ||
4102 | 87 | # hexstring | ||
4103 | 88 | return "%.32x" % value | ||
4104 | 89 | |||
4105 | 90 | def process_result_value(self, value, dialect): | ||
4106 | 91 | if value is None: | ||
4107 | 92 | return value | ||
4108 | 93 | else: | ||
4109 | 94 | return uuid.UUID(value) | ||
4110 | 68 | 95 | ||
4111 | === modified file 'src/mailman/interfaces/database.py' | |||
4112 | --- src/mailman/interfaces/database.py 2014-01-01 14:59:42 +0000 | |||
4113 | +++ src/mailman/interfaces/database.py 2014-10-11 02:14:38 +0000 | |||
4114 | @@ -24,7 +24,6 @@ | |||
4115 | 24 | 'DatabaseError', | 24 | 'DatabaseError', |
4116 | 25 | 'IDatabase', | 25 | 'IDatabase', |
4117 | 26 | 'IDatabaseFactory', | 26 | 'IDatabaseFactory', |
4118 | 27 | 'ITemporaryDatabase', | ||
4119 | 28 | ] | 27 | ] |
4120 | 29 | 28 | ||
4121 | 30 | 29 | ||
4122 | @@ -61,12 +60,7 @@ | |||
4123 | 61 | """Abort the current transaction.""" | 60 | """Abort the current transaction.""" |
4124 | 62 | 61 | ||
4125 | 63 | store = Attribute( | 62 | store = Attribute( |
4126 | 64 | """The underlying Storm store on which you can do queries.""") | ||
4127 | 65 | |||
4128 | 66 | |||
4129 | 67 | |||
4130 | 68 | 63 | ||
4133 | 69 | class ITemporaryDatabase(Interface): | 64 | """The underlying database object on which you can do queries.""") |
4132 | 70 | """Marker interface for test suite adaptation.""" | ||
4134 | 71 | 65 | ||
4135 | 72 | 66 | ||
4136 | 73 | 67 | ||
4137 | 74 | 68 | ||
4138 | 75 | 69 | ||
4139 | === modified file 'src/mailman/interfaces/messages.py' | |||
4140 | --- src/mailman/interfaces/messages.py 2014-04-28 15:23:35 +0000 | |||
4141 | +++ src/mailman/interfaces/messages.py 2014-10-11 02:14:38 +0000 | |||
4142 | @@ -83,7 +83,7 @@ | |||
4143 | 83 | 83 | ||
4144 | 84 | def get_message_by_hash(message_id_hash): | 84 | def get_message_by_hash(message_id_hash): |
4145 | 85 | """Return the message with the matching X-Message-ID-Hash. | 85 | """Return the message with the matching X-Message-ID-Hash. |
4147 | 86 | 86 | ||
4148 | 87 | :param message_id_hash: The X-Message-ID-Hash header contents to | 87 | :param message_id_hash: The X-Message-ID-Hash header contents to |
4149 | 88 | search for. | 88 | search for. |
4150 | 89 | :returns: The message, or None if no matching message was found. | 89 | :returns: The message, or None if no matching message was found. |
4151 | 90 | 90 | ||
4152 | === modified file 'src/mailman/model/address.py' | |||
4153 | --- src/mailman/model/address.py 2014-04-15 03:00:41 +0000 | |||
4154 | +++ src/mailman/model/address.py 2014-10-11 02:14:38 +0000 | |||
4155 | @@ -26,7 +26,9 @@ | |||
4156 | 26 | 26 | ||
4157 | 27 | 27 | ||
4158 | 28 | from email.utils import formataddr | 28 | from email.utils import formataddr |
4160 | 29 | from storm.locals import DateTime, Int, Reference, Unicode | 29 | from sqlalchemy import ( |
4161 | 30 | Column, DateTime, ForeignKey, Integer, Unicode) | ||
4162 | 31 | from sqlalchemy.orm import relationship, backref | ||
4163 | 30 | from zope.component import getUtility | 32 | from zope.component import getUtility |
4164 | 31 | from zope.event import notify | 33 | from zope.event import notify |
4165 | 32 | from zope.interface import implementer | 34 | from zope.interface import implementer |
4166 | @@ -42,17 +44,20 @@ | |||
4167 | 42 | class Address(Model): | 44 | class Address(Model): |
4168 | 43 | """See `IAddress`.""" | 45 | """See `IAddress`.""" |
4169 | 44 | 46 | ||
4181 | 45 | id = Int(primary=True) | 47 | __tablename__ = 'address' |
4182 | 46 | email = Unicode() | 48 | |
4183 | 47 | _original = Unicode() | 49 | id = Column(Integer, primary_key=True) |
4184 | 48 | display_name = Unicode() | 50 | email = Column(Unicode) |
4185 | 49 | _verified_on = DateTime(name='verified_on') | 51 | _original = Column(Unicode) |
4186 | 50 | registered_on = DateTime() | 52 | display_name = Column(Unicode) |
4187 | 51 | 53 | _verified_on = Column('verified_on', DateTime) | |
4188 | 52 | user_id = Int() | 54 | registered_on = Column(DateTime) |
4189 | 53 | user = Reference(user_id, 'User.id') | 55 | |
4190 | 54 | preferences_id = Int() | 56 | user_id = Column(Integer, ForeignKey('user.id'), index=True) |
4191 | 55 | preferences = Reference(preferences_id, 'Preferences.id') | 57 | |
4192 | 58 | preferences_id = Column(Integer, ForeignKey('preferences.id'), index=True) | ||
4193 | 59 | preferences = relationship( | ||
4194 | 60 | 'Preferences', backref=backref('address', uselist=False)) | ||
4195 | 56 | 61 | ||
4196 | 57 | def __init__(self, email, display_name): | 62 | def __init__(self, email, display_name): |
4197 | 58 | super(Address, self).__init__() | 63 | super(Address, self).__init__() |
4198 | 59 | 64 | ||
4199 | === modified file 'src/mailman/model/autorespond.py' | |||
4200 | --- src/mailman/model/autorespond.py 2014-01-01 14:59:42 +0000 | |||
4201 | +++ src/mailman/model/autorespond.py 2014-10-11 02:14:38 +0000 | |||
4202 | @@ -26,7 +26,9 @@ | |||
4203 | 26 | ] | 26 | ] |
4204 | 27 | 27 | ||
4205 | 28 | 28 | ||
4207 | 29 | from storm.locals import And, Date, Desc, Int, Reference | 29 | from sqlalchemy import Column, Date, ForeignKey, Integer |
4208 | 30 | from sqlalchemy import desc | ||
4209 | 31 | from sqlalchemy.orm import relationship | ||
4210 | 30 | from zope.interface import implementer | 32 | from zope.interface import implementer |
4211 | 31 | 33 | ||
4212 | 32 | from mailman.database.model import Model | 34 | from mailman.database.model import Model |
4213 | @@ -42,16 +44,18 @@ | |||
4214 | 42 | class AutoResponseRecord(Model): | 44 | class AutoResponseRecord(Model): |
4215 | 43 | """See `IAutoResponseRecord`.""" | 45 | """See `IAutoResponseRecord`.""" |
4216 | 44 | 46 | ||
4227 | 45 | id = Int(primary=True) | 47 | __tablename__ = 'autoresponserecord' |
4228 | 46 | 48 | ||
4229 | 47 | address_id = Int() | 49 | id = Column(Integer, primary_key=True) |
4230 | 48 | address = Reference(address_id, 'Address.id') | 50 | |
4231 | 49 | 51 | address_id = Column(Integer, ForeignKey('address.id'), index=True) | |
4232 | 50 | mailing_list_id = Int() | 52 | address = relationship('Address') |
4233 | 51 | mailing_list = Reference(mailing_list_id, 'MailingList.id') | 53 | |
4234 | 52 | 54 | mailing_list_id = Column(Integer, ForeignKey('mailinglist.id'), index=True) | |
4235 | 53 | response_type = Enum(Response) | 55 | mailing_list = relationship('MailingList') |
4236 | 54 | date_sent = Date() | 56 | |
4237 | 57 | response_type = Column(Enum(Response)) | ||
4238 | 58 | date_sent = Column(Date) | ||
4239 | 55 | 59 | ||
4240 | 56 | def __init__(self, mailing_list, address, response_type): | 60 | def __init__(self, mailing_list, address, response_type): |
4241 | 57 | self.mailing_list = mailing_list | 61 | self.mailing_list = mailing_list |
4242 | @@ -71,12 +75,11 @@ | |||
4243 | 71 | @dbconnection | 75 | @dbconnection |
4244 | 72 | def todays_count(self, store, address, response_type): | 76 | def todays_count(self, store, address, response_type): |
4245 | 73 | """See `IAutoResponseSet`.""" | 77 | """See `IAutoResponseSet`.""" |
4252 | 74 | return store.find( | 78 | return store.query(AutoResponseRecord).filter_by( |
4253 | 75 | AutoResponseRecord, | 79 | address=address, |
4254 | 76 | And(AutoResponseRecord.address == address, | 80 | mailing_list=self._mailing_list, |
4255 | 77 | AutoResponseRecord.mailing_list == self._mailing_list, | 81 | response_type=response_type, |
4256 | 78 | AutoResponseRecord.response_type == response_type, | 82 | date_sent=today()).count() |
4251 | 79 | AutoResponseRecord.date_sent == today())).count() | ||
4257 | 80 | 83 | ||
4258 | 81 | @dbconnection | 84 | @dbconnection |
4259 | 82 | def response_sent(self, store, address, response_type): | 85 | def response_sent(self, store, address, response_type): |
4260 | @@ -88,10 +91,9 @@ | |||
4261 | 88 | @dbconnection | 91 | @dbconnection |
4262 | 89 | def last_response(self, store, address, response_type): | 92 | def last_response(self, store, address, response_type): |
4263 | 90 | """See `IAutoResponseSet`.""" | 93 | """See `IAutoResponseSet`.""" |
4270 | 91 | results = store.find( | 94 | results = store.query(AutoResponseRecord).filter_by( |
4271 | 92 | AutoResponseRecord, | 95 | address=address, |
4272 | 93 | And(AutoResponseRecord.address == address, | 96 | mailing_list=self._mailing_list, |
4273 | 94 | AutoResponseRecord.mailing_list == self._mailing_list, | 97 | response_type=response_type |
4274 | 95 | AutoResponseRecord.response_type == response_type) | 98 | ).order_by(desc(AutoResponseRecord.date_sent)) |
4269 | 96 | ).order_by(Desc(AutoResponseRecord.date_sent)) | ||
4275 | 97 | return (None if results.count() == 0 else results.first()) | 99 | return (None if results.count() == 0 else results.first()) |
4276 | 98 | 100 | ||
4277 | === modified file 'src/mailman/model/bans.py' | |||
4278 | --- src/mailman/model/bans.py 2014-01-01 14:59:42 +0000 | |||
4279 | +++ src/mailman/model/bans.py 2014-10-11 02:14:38 +0000 | |||
4280 | @@ -27,7 +27,7 @@ | |||
4281 | 27 | 27 | ||
4282 | 28 | import re | 28 | import re |
4283 | 29 | 29 | ||
4285 | 30 | from storm.locals import Int, Unicode | 30 | from sqlalchemy import Column, Integer, Unicode |
4286 | 31 | from zope.interface import implementer | 31 | from zope.interface import implementer |
4287 | 32 | 32 | ||
4288 | 33 | from mailman.database.model import Model | 33 | from mailman.database.model import Model |
4289 | @@ -40,9 +40,11 @@ | |||
4290 | 40 | class Ban(Model): | 40 | class Ban(Model): |
4291 | 41 | """See `IBan`.""" | 41 | """See `IBan`.""" |
4292 | 42 | 42 | ||
4296 | 43 | id = Int(primary=True) | 43 | __tablename__ = 'ban' |
4297 | 44 | email = Unicode() | 44 | |
4298 | 45 | list_id = Unicode() | 45 | id = Column(Integer, primary_key=True) |
4299 | 46 | email = Column(Unicode) | ||
4300 | 47 | list_id = Column(Unicode) | ||
4301 | 46 | 48 | ||
4302 | 47 | def __init__(self, email, list_id): | 49 | def __init__(self, email, list_id): |
4303 | 48 | super(Ban, self).__init__() | 50 | super(Ban, self).__init__() |
4304 | @@ -62,7 +64,7 @@ | |||
4305 | 62 | @dbconnection | 64 | @dbconnection |
4306 | 63 | def ban(self, store, email): | 65 | def ban(self, store, email): |
4307 | 64 | """See `IBanManager`.""" | 66 | """See `IBanManager`.""" |
4309 | 65 | bans = store.find(Ban, email=email, list_id=self._list_id) | 67 | bans = store.query(Ban).filter_by(email=email, list_id=self._list_id) |
4310 | 66 | if bans.count() == 0: | 68 | if bans.count() == 0: |
4311 | 67 | ban = Ban(email, self._list_id) | 69 | ban = Ban(email, self._list_id) |
4312 | 68 | store.add(ban) | 70 | store.add(ban) |
4313 | @@ -70,9 +72,10 @@ | |||
4314 | 70 | @dbconnection | 72 | @dbconnection |
4315 | 71 | def unban(self, store, email): | 73 | def unban(self, store, email): |
4316 | 72 | """See `IBanManager`.""" | 74 | """See `IBanManager`.""" |
4318 | 73 | ban = store.find(Ban, email=email, list_id=self._list_id).one() | 75 | ban = store.query(Ban).filter_by( |
4319 | 76 | email=email, list_id=self._list_id).first() | ||
4320 | 74 | if ban is not None: | 77 | if ban is not None: |
4322 | 75 | store.remove(ban) | 78 | store.delete(ban) |
4323 | 76 | 79 | ||
4324 | 77 | @dbconnection | 80 | @dbconnection |
4325 | 78 | def is_banned(self, store, email): | 81 | def is_banned(self, store, email): |
4326 | @@ -81,32 +84,32 @@ | |||
4327 | 81 | if list_id is None: | 84 | if list_id is None: |
4328 | 82 | # The client is asking for global bans. Look up bans on the | 85 | # The client is asking for global bans. Look up bans on the |
4329 | 83 | # specific email address first. | 86 | # specific email address first. |
4331 | 84 | bans = store.find(Ban, email=email, list_id=None) | 87 | bans = store.query(Ban).filter_by(email=email, list_id=None) |
4332 | 85 | if bans.count() > 0: | 88 | if bans.count() > 0: |
4333 | 86 | return True | 89 | return True |
4334 | 87 | # And now look for global pattern bans. | 90 | # And now look for global pattern bans. |
4336 | 88 | bans = store.find(Ban, list_id=None) | 91 | bans = store.query(Ban).filter_by(list_id=None) |
4337 | 89 | for ban in bans: | 92 | for ban in bans: |
4338 | 90 | if (ban.email.startswith('^') and | 93 | if (ban.email.startswith('^') and |
4339 | 91 | re.match(ban.email, email, re.IGNORECASE) is not None): | 94 | re.match(ban.email, email, re.IGNORECASE) is not None): |
4340 | 92 | return True | 95 | return True |
4341 | 93 | else: | 96 | else: |
4342 | 94 | # This is a list-specific ban. | 97 | # This is a list-specific ban. |
4344 | 95 | bans = store.find(Ban, email=email, list_id=list_id) | 98 | bans = store.query(Ban).filter_by(email=email, list_id=list_id) |
4345 | 96 | if bans.count() > 0: | 99 | if bans.count() > 0: |
4346 | 97 | return True | 100 | return True |
4347 | 98 | # Try global bans next. | 101 | # Try global bans next. |
4349 | 99 | bans = store.find(Ban, email=email, list_id=None) | 102 | bans = store.query(Ban).filter_by(email=email, list_id=None) |
4350 | 100 | if bans.count() > 0: | 103 | if bans.count() > 0: |
4351 | 101 | return True | 104 | return True |
4352 | 102 | # Now try specific mailing list bans, but with a pattern. | 105 | # Now try specific mailing list bans, but with a pattern. |
4354 | 103 | bans = store.find(Ban, list_id=list_id) | 106 | bans = store.query(Ban).filter_by(list_id=list_id) |
4355 | 104 | for ban in bans: | 107 | for ban in bans: |
4356 | 105 | if (ban.email.startswith('^') and | 108 | if (ban.email.startswith('^') and |
4357 | 106 | re.match(ban.email, email, re.IGNORECASE) is not None): | 109 | re.match(ban.email, email, re.IGNORECASE) is not None): |
4358 | 107 | return True | 110 | return True |
4359 | 108 | # And now try global pattern bans. | 111 | # And now try global pattern bans. |
4361 | 109 | bans = store.find(Ban, list_id=None) | 112 | bans = store.query(Ban).filter_by(list_id=None) |
4362 | 110 | for ban in bans: | 113 | for ban in bans: |
4363 | 111 | if (ban.email.startswith('^') and | 114 | if (ban.email.startswith('^') and |
4364 | 112 | re.match(ban.email, email, re.IGNORECASE) is not None): | 115 | re.match(ban.email, email, re.IGNORECASE) is not None): |
4365 | 113 | 116 | ||
4366 | === modified file 'src/mailman/model/bounce.py' | |||
4367 | --- src/mailman/model/bounce.py 2014-01-01 14:59:42 +0000 | |||
4368 | +++ src/mailman/model/bounce.py 2014-10-11 02:14:38 +0000 | |||
4369 | @@ -26,7 +26,8 @@ | |||
4370 | 26 | ] | 26 | ] |
4371 | 27 | 27 | ||
4372 | 28 | 28 | ||
4374 | 29 | from storm.locals import Bool, Int, DateTime, Unicode | 29 | |
4375 | 30 | from sqlalchemy import Boolean, Column, DateTime, Integer, Unicode | ||
4376 | 30 | from zope.interface import implementer | 31 | from zope.interface import implementer |
4377 | 31 | 32 | ||
4378 | 32 | from mailman.database.model import Model | 33 | from mailman.database.model import Model |
4379 | @@ -42,13 +43,15 @@ | |||
4380 | 42 | class BounceEvent(Model): | 43 | class BounceEvent(Model): |
4381 | 43 | """See `IBounceEvent`.""" | 44 | """See `IBounceEvent`.""" |
4382 | 44 | 45 | ||
4390 | 45 | id = Int(primary=True) | 46 | __tablename__ = 'bounceevent' |
4391 | 46 | list_id = Unicode() | 47 | |
4392 | 47 | email = Unicode() | 48 | id = Column(Integer, primary_key=True) |
4393 | 48 | timestamp = DateTime() | 49 | list_id = Column(Unicode) |
4394 | 49 | message_id = Unicode() | 50 | email = Column(Unicode) |
4395 | 50 | context = Enum(BounceContext) | 51 | timestamp = Column(DateTime) |
4396 | 51 | processed = Bool() | 52 | message_id = Column(Unicode) |
4397 | 53 | context = Column(Enum(BounceContext)) | ||
4398 | 54 | processed = Column(Boolean) | ||
4399 | 52 | 55 | ||
4400 | 53 | def __init__(self, list_id, email, msg, context=None): | 56 | def __init__(self, list_id, email, msg, context=None): |
4401 | 54 | self.list_id = list_id | 57 | self.list_id = list_id |
4402 | @@ -75,12 +78,12 @@ | |||
4403 | 75 | @dbconnection | 78 | @dbconnection |
4404 | 76 | def events(self, store): | 79 | def events(self, store): |
4405 | 77 | """See `IBounceProcessor`.""" | 80 | """See `IBounceProcessor`.""" |
4407 | 78 | for event in store.find(BounceEvent): | 81 | for event in store.query(BounceEvent).all(): |
4408 | 79 | yield event | 82 | yield event |
4409 | 80 | 83 | ||
4410 | 81 | @property | 84 | @property |
4411 | 82 | @dbconnection | 85 | @dbconnection |
4412 | 83 | def unprocessed(self, store): | 86 | def unprocessed(self, store): |
4413 | 84 | """See `IBounceProcessor`.""" | 87 | """See `IBounceProcessor`.""" |
4415 | 85 | for event in store.find(BounceEvent, BounceEvent.processed == False): | 88 | for event in store.query(BounceEvent).filter_by(processed=False): |
4416 | 86 | yield event | 89 | yield event |
4417 | 87 | 90 | ||
4418 | === modified file 'src/mailman/model/digests.py' | |||
4419 | --- src/mailman/model/digests.py 2014-01-01 14:59:42 +0000 | |||
4420 | +++ src/mailman/model/digests.py 2014-10-11 02:14:38 +0000 | |||
4421 | @@ -25,7 +25,8 @@ | |||
4422 | 25 | ] | 25 | ] |
4423 | 26 | 26 | ||
4424 | 27 | 27 | ||
4426 | 28 | from storm.locals import Int, Reference | 28 | from sqlalchemy import Column, Integer, ForeignKey |
4427 | 29 | from sqlalchemy.orm import relationship | ||
4428 | 29 | from zope.interface import implementer | 30 | from zope.interface import implementer |
4429 | 30 | 31 | ||
4430 | 31 | from mailman.database.model import Model | 32 | from mailman.database.model import Model |
4431 | @@ -39,15 +40,17 @@ | |||
4432 | 39 | class OneLastDigest(Model): | 40 | class OneLastDigest(Model): |
4433 | 40 | """See `IOneLastDigest`.""" | 41 | """See `IOneLastDigest`.""" |
4434 | 41 | 42 | ||
4444 | 42 | id = Int(primary=True) | 43 | __tablename__ = 'onelastdigest' |
4445 | 43 | 44 | ||
4446 | 44 | mailing_list_id = Int() | 45 | id = Column(Integer, primary_key=True) |
4447 | 45 | mailing_list = Reference(mailing_list_id, 'MailingList.id') | 46 | |
4448 | 46 | 47 | mailing_list_id = Column(Integer, ForeignKey('mailinglist.id')) | |
4449 | 47 | address_id = Int() | 48 | mailing_list = relationship('MailingList') |
4450 | 48 | address = Reference(address_id, 'Address.id') | 49 | |
4451 | 49 | 50 | address_id = Column(Integer, ForeignKey('address.id')) | |
4452 | 50 | delivery_mode = Enum(DeliveryMode) | 51 | address = relationship('Address') |
4453 | 52 | |||
4454 | 53 | delivery_mode = Column(Enum(DeliveryMode)) | ||
4455 | 51 | 54 | ||
4456 | 52 | def __init__(self, mailing_list, address, delivery_mode): | 55 | def __init__(self, mailing_list, address, delivery_mode): |
4457 | 53 | self.mailing_list = mailing_list | 56 | self.mailing_list = mailing_list |
4458 | 54 | 57 | ||
4459 | === modified file 'src/mailman/model/docs/messagestore.rst' | |||
4460 | --- src/mailman/model/docs/messagestore.rst 2014-04-28 15:23:35 +0000 | |||
4461 | +++ src/mailman/model/docs/messagestore.rst 2014-10-11 02:14:38 +0000 | |||
4462 | @@ -28,8 +28,9 @@ | |||
4463 | 28 | However, if the message has a ``Message-ID`` header, it can be stored. | 28 | However, if the message has a ``Message-ID`` header, it can be stored. |
4464 | 29 | 29 | ||
4465 | 30 | >>> msg['Message-ID'] = '<87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>' | 30 | >>> msg['Message-ID'] = '<87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>' |
4468 | 31 | >>> message_store.add(msg) | 31 | >>> x_message_id_hash = message_store.add(msg) |
4469 | 32 | 'AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35' | 32 | >>> print(x_message_id_hash) |
4470 | 33 | AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35 | ||
4471 | 33 | >>> print(msg.as_string()) | 34 | >>> print(msg.as_string()) |
4472 | 34 | Subject: An important message | 35 | Subject: An important message |
4473 | 35 | Message-ID: <87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp> | 36 | Message-ID: <87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp> |
4474 | 36 | 37 | ||
4475 | === modified file 'src/mailman/model/domain.py' | |||
4476 | --- src/mailman/model/domain.py 2014-01-01 14:59:42 +0000 | |||
4477 | +++ src/mailman/model/domain.py 2014-10-11 02:14:38 +0000 | |||
4478 | @@ -26,8 +26,8 @@ | |||
4479 | 26 | ] | 26 | ] |
4480 | 27 | 27 | ||
4481 | 28 | 28 | ||
4482 | 29 | from sqlalchemy import Column, Integer, Unicode | ||
4483 | 29 | from urlparse import urljoin, urlparse | 30 | from urlparse import urljoin, urlparse |
4484 | 30 | from storm.locals import Int, Unicode | ||
4485 | 31 | from zope.event import notify | 31 | from zope.event import notify |
4486 | 32 | from zope.interface import implementer | 32 | from zope.interface import implementer |
4487 | 33 | 33 | ||
4488 | @@ -44,12 +44,14 @@ | |||
4489 | 44 | class Domain(Model): | 44 | class Domain(Model): |
4490 | 45 | """Domains.""" | 45 | """Domains.""" |
4491 | 46 | 46 | ||
4498 | 47 | id = Int(primary=True) | 47 | __tablename__ = 'domain' |
4499 | 48 | 48 | ||
4500 | 49 | mail_host = Unicode() | 49 | id = Column(Integer, primary_key=True) |
4501 | 50 | base_url = Unicode() | 50 | |
4502 | 51 | description = Unicode() | 51 | mail_host = Column(Unicode) |
4503 | 52 | contact_address = Unicode() | 52 | base_url = Column(Unicode) |
4504 | 53 | description = Column(Unicode) | ||
4505 | 54 | contact_address = Column(Unicode) | ||
4506 | 53 | 55 | ||
4507 | 54 | def __init__(self, mail_host, | 56 | def __init__(self, mail_host, |
4508 | 55 | description=None, | 57 | description=None, |
4509 | @@ -92,8 +94,7 @@ | |||
4510 | 92 | @dbconnection | 94 | @dbconnection |
4511 | 93 | def mailing_lists(self, store): | 95 | def mailing_lists(self, store): |
4512 | 94 | """See `IDomain`.""" | 96 | """See `IDomain`.""" |
4515 | 95 | mailing_lists = store.find( | 97 | mailing_lists = store.query(MailingList).filter( |
4514 | 96 | MailingList, | ||
4516 | 97 | MailingList.mail_host == self.mail_host) | 98 | MailingList.mail_host == self.mail_host) |
4517 | 98 | for mlist in mailing_lists: | 99 | for mlist in mailing_lists: |
4518 | 99 | yield mlist | 100 | yield mlist |
4519 | @@ -140,14 +141,14 @@ | |||
4520 | 140 | def remove(self, store, mail_host): | 141 | def remove(self, store, mail_host): |
4521 | 141 | domain = self[mail_host] | 142 | domain = self[mail_host] |
4522 | 142 | notify(DomainDeletingEvent(domain)) | 143 | notify(DomainDeletingEvent(domain)) |
4524 | 143 | store.remove(domain) | 144 | store.delete(domain) |
4525 | 144 | notify(DomainDeletedEvent(mail_host)) | 145 | notify(DomainDeletedEvent(mail_host)) |
4526 | 145 | return domain | 146 | return domain |
4527 | 146 | 147 | ||
4528 | 147 | @dbconnection | 148 | @dbconnection |
4529 | 148 | def get(self, store, mail_host, default=None): | 149 | def get(self, store, mail_host, default=None): |
4530 | 149 | """See `IDomainManager`.""" | 150 | """See `IDomainManager`.""" |
4532 | 150 | domains = store.find(Domain, mail_host=mail_host) | 151 | domains = store.query(Domain).filter_by(mail_host=mail_host) |
4533 | 151 | if domains.count() < 1: | 152 | if domains.count() < 1: |
4534 | 152 | return default | 153 | return default |
4535 | 153 | assert domains.count() == 1, ( | 154 | assert domains.count() == 1, ( |
4536 | @@ -164,15 +165,15 @@ | |||
4537 | 164 | 165 | ||
4538 | 165 | @dbconnection | 166 | @dbconnection |
4539 | 166 | def __len__(self, store): | 167 | def __len__(self, store): |
4541 | 167 | return store.find(Domain).count() | 168 | return store.query(Domain).count() |
4542 | 168 | 169 | ||
4543 | 169 | @dbconnection | 170 | @dbconnection |
4544 | 170 | def __iter__(self, store): | 171 | def __iter__(self, store): |
4545 | 171 | """See `IDomainManager`.""" | 172 | """See `IDomainManager`.""" |
4547 | 172 | for domain in store.find(Domain): | 173 | for domain in store.query(Domain).all(): |
4548 | 173 | yield domain | 174 | yield domain |
4549 | 174 | 175 | ||
4550 | 175 | @dbconnection | 176 | @dbconnection |
4551 | 176 | def __contains__(self, store, mail_host): | 177 | def __contains__(self, store, mail_host): |
4552 | 177 | """See `IDomainManager`.""" | 178 | """See `IDomainManager`.""" |
4554 | 178 | return store.find(Domain, mail_host=mail_host).count() > 0 | 179 | return store.query(Domain).filter_by(mail_host=mail_host).count() > 0 |
4555 | 179 | 180 | ||
4556 | === modified file 'src/mailman/model/language.py' | |||
4557 | --- src/mailman/model/language.py 2014-01-01 14:59:42 +0000 | |||
4558 | +++ src/mailman/model/language.py 2014-10-11 02:14:38 +0000 | |||
4559 | @@ -25,11 +25,11 @@ | |||
4560 | 25 | ] | 25 | ] |
4561 | 26 | 26 | ||
4562 | 27 | 27 | ||
4564 | 28 | from storm.locals import Int, Unicode | 28 | from sqlalchemy import Column, Integer, Unicode |
4565 | 29 | from zope.interface import implementer | 29 | from zope.interface import implementer |
4566 | 30 | 30 | ||
4569 | 31 | from mailman.database import Model | 31 | from mailman.database.model import Model |
4570 | 32 | from mailman.interfaces import ILanguage | 32 | from mailman.interfaces.languages import ILanguage |
4571 | 33 | 33 | ||
4572 | 34 | 34 | ||
4573 | 35 | 35 | ||
4574 | 36 | 36 | ||
4575 | @@ -37,5 +37,7 @@ | |||
4576 | 37 | class Language(Model): | 37 | class Language(Model): |
4577 | 38 | """See `ILanguage`.""" | 38 | """See `ILanguage`.""" |
4578 | 39 | 39 | ||
4581 | 40 | id = Int(primary=True) | 40 | __tablename__ = 'language' |
4582 | 41 | code = Unicode() | 41 | |
4583 | 42 | id = Column(Integer, primary_key=True) | ||
4584 | 43 | code = Column(Unicode) | ||
4585 | 42 | 44 | ||
4586 | === modified file 'src/mailman/model/listmanager.py' | |||
4587 | --- src/mailman/model/listmanager.py 2014-04-14 16:14:13 +0000 | |||
4588 | +++ src/mailman/model/listmanager.py 2014-10-11 02:14:38 +0000 | |||
4589 | @@ -52,9 +52,7 @@ | |||
4590 | 52 | raise InvalidEmailAddressError(fqdn_listname) | 52 | raise InvalidEmailAddressError(fqdn_listname) |
4591 | 53 | list_id = '{0}.{1}'.format(listname, hostname) | 53 | list_id = '{0}.{1}'.format(listname, hostname) |
4592 | 54 | notify(ListCreatingEvent(fqdn_listname)) | 54 | notify(ListCreatingEvent(fqdn_listname)) |
4596 | 55 | mlist = store.find( | 55 | mlist = store.query(MailingList).filter_by(_list_id=list_id).first() |
4594 | 56 | MailingList, | ||
4595 | 57 | MailingList._list_id == list_id).one() | ||
4597 | 58 | if mlist: | 56 | if mlist: |
4598 | 59 | raise ListAlreadyExistsError(fqdn_listname) | 57 | raise ListAlreadyExistsError(fqdn_listname) |
4599 | 60 | mlist = MailingList(fqdn_listname) | 58 | mlist = MailingList(fqdn_listname) |
4600 | @@ -68,40 +66,40 @@ | |||
4601 | 68 | """See `IListManager`.""" | 66 | """See `IListManager`.""" |
4602 | 69 | listname, at, hostname = fqdn_listname.partition('@') | 67 | listname, at, hostname = fqdn_listname.partition('@') |
4603 | 70 | list_id = '{0}.{1}'.format(listname, hostname) | 68 | list_id = '{0}.{1}'.format(listname, hostname) |
4605 | 71 | return store.find(MailingList, MailingList._list_id == list_id).one() | 69 | return store.query(MailingList).filter_by(_list_id=list_id).first() |
4606 | 72 | 70 | ||
4607 | 73 | @dbconnection | 71 | @dbconnection |
4608 | 74 | def get_by_list_id(self, store, list_id): | 72 | def get_by_list_id(self, store, list_id): |
4609 | 75 | """See `IListManager`.""" | 73 | """See `IListManager`.""" |
4611 | 76 | return store.find(MailingList, MailingList._list_id == list_id).one() | 74 | return store.query(MailingList).filter_by(_list_id=list_id).first() |
4612 | 77 | 75 | ||
4613 | 78 | @dbconnection | 76 | @dbconnection |
4614 | 79 | def delete(self, store, mlist): | 77 | def delete(self, store, mlist): |
4615 | 80 | """See `IListManager`.""" | 78 | """See `IListManager`.""" |
4616 | 81 | fqdn_listname = mlist.fqdn_listname | 79 | fqdn_listname = mlist.fqdn_listname |
4617 | 82 | notify(ListDeletingEvent(mlist)) | 80 | notify(ListDeletingEvent(mlist)) |
4620 | 83 | store.find(ContentFilter, ContentFilter.mailing_list == mlist).remove() | 81 | store.query(ContentFilter).filter_by(mailing_list=mlist).delete() |
4621 | 84 | store.remove(mlist) | 82 | store.delete(mlist) |
4622 | 85 | notify(ListDeletedEvent(fqdn_listname)) | 83 | notify(ListDeletedEvent(fqdn_listname)) |
4623 | 86 | 84 | ||
4624 | 87 | @property | 85 | @property |
4625 | 88 | @dbconnection | 86 | @dbconnection |
4626 | 89 | def mailing_lists(self, store): | 87 | def mailing_lists(self, store): |
4627 | 90 | """See `IListManager`.""" | 88 | """See `IListManager`.""" |
4629 | 91 | for mlist in store.find(MailingList): | 89 | for mlist in store.query(MailingList).all(): |
4630 | 92 | yield mlist | 90 | yield mlist |
4631 | 93 | 91 | ||
4632 | 94 | @dbconnection | 92 | @dbconnection |
4633 | 95 | def __iter__(self, store): | 93 | def __iter__(self, store): |
4634 | 96 | """See `IListManager`.""" | 94 | """See `IListManager`.""" |
4636 | 97 | for mlist in store.find(MailingList): | 95 | for mlist in store.query(MailingList).all(): |
4637 | 98 | yield mlist | 96 | yield mlist |
4638 | 99 | 97 | ||
4639 | 100 | @property | 98 | @property |
4640 | 101 | @dbconnection | 99 | @dbconnection |
4641 | 102 | def names(self, store): | 100 | def names(self, store): |
4642 | 103 | """See `IListManager`.""" | 101 | """See `IListManager`.""" |
4644 | 104 | result_set = store.find(MailingList) | 102 | result_set = store.query(MailingList) |
4645 | 105 | for mail_host, list_name in result_set.values(MailingList.mail_host, | 103 | for mail_host, list_name in result_set.values(MailingList.mail_host, |
4646 | 106 | MailingList.list_name): | 104 | MailingList.list_name): |
4647 | 107 | yield '{0}@{1}'.format(list_name, mail_host) | 105 | yield '{0}@{1}'.format(list_name, mail_host) |
4648 | @@ -110,15 +108,16 @@ | |||
4649 | 110 | @dbconnection | 108 | @dbconnection |
4650 | 111 | def list_ids(self, store): | 109 | def list_ids(self, store): |
4651 | 112 | """See `IListManager`.""" | 110 | """See `IListManager`.""" |
4653 | 113 | result_set = store.find(MailingList) | 111 | result_set = store.query(MailingList) |
4654 | 114 | for list_id in result_set.values(MailingList._list_id): | 112 | for list_id in result_set.values(MailingList._list_id): |
4656 | 115 | yield list_id | 113 | assert isinstance(list_id, tuple) and len(list_id) == 1 |
4657 | 114 | yield list_id[0] | ||
4658 | 116 | 115 | ||
4659 | 117 | @property | 116 | @property |
4660 | 118 | @dbconnection | 117 | @dbconnection |
4661 | 119 | def name_components(self, store): | 118 | def name_components(self, store): |
4662 | 120 | """See `IListManager`.""" | 119 | """See `IListManager`.""" |
4664 | 121 | result_set = store.find(MailingList) | 120 | result_set = store.query(MailingList) |
4665 | 122 | for mail_host, list_name in result_set.values(MailingList.mail_host, | 121 | for mail_host, list_name in result_set.values(MailingList.mail_host, |
4666 | 123 | MailingList.list_name): | 122 | MailingList.list_name): |
4667 | 124 | yield list_name, mail_host | 123 | yield list_name, mail_host |
4668 | 125 | 124 | ||
4669 | === modified file 'src/mailman/model/mailinglist.py' | |||
4670 | --- src/mailman/model/mailinglist.py 2014-04-14 16:14:13 +0000 | |||
4671 | +++ src/mailman/model/mailinglist.py 2014-10-11 02:14:38 +0000 | |||
4672 | @@ -27,9 +27,11 @@ | |||
4673 | 27 | 27 | ||
4674 | 28 | import os | 28 | import os |
4675 | 29 | 29 | ||
4679 | 30 | from storm.locals import ( | 30 | from sqlalchemy import ( |
4680 | 31 | And, Bool, DateTime, Float, Int, Pickle, RawStr, Reference, Store, | 31 | Boolean, Column, DateTime, Float, ForeignKey, Integer, Interval, |
4681 | 32 | TimeDelta, Unicode) | 32 | LargeBinary, PickleType, Unicode) |
4682 | 33 | from sqlalchemy.event import listen | ||
4683 | 34 | from sqlalchemy.orm import relationship | ||
4684 | 33 | from urlparse import urljoin | 35 | from urlparse import urljoin |
4685 | 34 | from zope.component import getUtility | 36 | from zope.component import getUtility |
4686 | 35 | from zope.event import notify | 37 | from zope.event import notify |
4687 | @@ -37,6 +39,7 @@ | |||
4688 | 37 | 39 | ||
4689 | 38 | from mailman.config import config | 40 | from mailman.config import config |
4690 | 39 | from mailman.database.model import Model | 41 | from mailman.database.model import Model |
4691 | 42 | from mailman.database.transaction import dbconnection | ||
4692 | 40 | from mailman.database.types import Enum | 43 | from mailman.database.types import Enum |
4693 | 41 | from mailman.interfaces.action import Action, FilterAction | 44 | from mailman.interfaces.action import Action, FilterAction |
4694 | 42 | from mailman.interfaces.address import IAddress | 45 | from mailman.interfaces.address import IAddress |
4695 | @@ -73,121 +76,121 @@ | |||
4696 | 73 | class MailingList(Model): | 76 | class MailingList(Model): |
4697 | 74 | """See `IMailingList`.""" | 77 | """See `IMailingList`.""" |
4698 | 75 | 78 | ||
4700 | 76 | id = Int(primary=True) | 79 | __tablename__ = 'mailinglist' |
4701 | 80 | |||
4702 | 81 | id = Column(Integer, primary_key=True) | ||
4703 | 77 | 82 | ||
4704 | 78 | # XXX denotes attributes that should be part of the public interface but | 83 | # XXX denotes attributes that should be part of the public interface but |
4705 | 79 | # are currently missing. | 84 | # are currently missing. |
4706 | 80 | 85 | ||
4707 | 81 | # List identity | 86 | # List identity |
4715 | 82 | list_name = Unicode() | 87 | list_name = Column(Unicode) |
4716 | 83 | mail_host = Unicode() | 88 | mail_host = Column(Unicode) |
4717 | 84 | _list_id = Unicode(name='list_id') | 89 | _list_id = Column('list_id', Unicode) |
4718 | 85 | allow_list_posts = Bool() | 90 | allow_list_posts = Column(Boolean) |
4719 | 86 | include_rfc2369_headers = Bool() | 91 | include_rfc2369_headers = Column(Boolean) |
4720 | 87 | advertised = Bool() | 92 | advertised = Column(Boolean) |
4721 | 88 | anonymous_list = Bool() | 93 | anonymous_list = Column(Boolean) |
4722 | 89 | # Attributes not directly modifiable via the web u/i | 94 | # Attributes not directly modifiable via the web u/i |
4743 | 90 | created_at = DateTime() | 95 | created_at = Column(DateTime) |
4744 | 91 | # Attributes which are directly modifiable via the web u/i. The more | 96 | # Attributes which are directly modifiable via the web u/i. The more |
4745 | 92 | # complicated attributes are currently stored as pickles, though that | 97 | # complicated attributes are currently stored as pickles, though that |
4746 | 93 | # will change as the schema and implementation is developed. | 98 | # will change as the schema and implementation is developed. |
4747 | 94 | next_request_id = Int() | 99 | next_request_id = Column(Integer) |
4748 | 95 | next_digest_number = Int() | 100 | next_digest_number = Column(Integer) |
4749 | 96 | digest_last_sent_at = DateTime() | 101 | digest_last_sent_at = Column(DateTime) |
4750 | 97 | volume = Int() | 102 | volume = Column(Integer) |
4751 | 98 | last_post_at = DateTime() | 103 | last_post_at = Column(DateTime) |
4752 | 99 | # Implicit destination. | 104 | # Attributes which are directly modifiable via the web u/i. The more |
4753 | 100 | acceptable_aliases_id = Int() | 105 | # complicated attributes are currently stored as pickles, though that |
4754 | 101 | acceptable_alias = Reference(acceptable_aliases_id, 'AcceptableAlias.id') | 106 | # will change as the schema and implementation is developed. |
4755 | 102 | # Attributes which are directly modifiable via the web u/i. The more | 107 | accept_these_nonmembers = Column(PickleType) # XXX |
4756 | 103 | # complicated attributes are currently stored as pickles, though that | 108 | admin_immed_notify = Column(Boolean) |
4757 | 104 | # will change as the schema and implementation is developed. | 109 | admin_notify_mchanges = Column(Boolean) |
4758 | 105 | accept_these_nonmembers = Pickle() # XXX | 110 | administrivia = Column(Boolean) |
4759 | 106 | admin_immed_notify = Bool() | 111 | archive_policy = Column(Enum(ArchivePolicy)) |
4740 | 107 | admin_notify_mchanges = Bool() | ||
4741 | 108 | administrivia = Bool() | ||
4742 | 109 | archive_policy = Enum(ArchivePolicy) | ||
4760 | 110 | # Automatic responses. | 112 | # Automatic responses. |
4768 | 111 | autoresponse_grace_period = TimeDelta() | 113 | autoresponse_grace_period = Column(Interval) |
4769 | 112 | autorespond_owner = Enum(ResponseAction) | 114 | autorespond_owner = Column(Enum(ResponseAction)) |
4770 | 113 | autoresponse_owner_text = Unicode() | 115 | autoresponse_owner_text = Column(Unicode) |
4771 | 114 | autorespond_postings = Enum(ResponseAction) | 116 | autorespond_postings = Column(Enum(ResponseAction)) |
4772 | 115 | autoresponse_postings_text = Unicode() | 117 | autoresponse_postings_text = Column(Unicode) |
4773 | 116 | autorespond_requests = Enum(ResponseAction) | 118 | autorespond_requests = Column(Enum(ResponseAction)) |
4774 | 117 | autoresponse_request_text = Unicode() | 119 | autoresponse_request_text = Column(Unicode) |
4775 | 118 | # Content filters. | 120 | # Content filters. |
4780 | 119 | filter_action = Enum(FilterAction) | 121 | filter_action = Column(Enum(FilterAction)) |
4781 | 120 | filter_content = Bool() | 122 | filter_content = Column(Boolean) |
4782 | 121 | collapse_alternatives = Bool() | 123 | collapse_alternatives = Column(Boolean) |
4783 | 122 | convert_html_to_plaintext = Bool() | 124 | convert_html_to_plaintext = Column(Boolean) |
4784 | 123 | # Bounces. | 125 | # Bounces. |
4794 | 124 | bounce_info_stale_after = TimeDelta() # XXX | 126 | bounce_info_stale_after = Column(Interval) # XXX |
4795 | 125 | bounce_matching_headers = Unicode() # XXX | 127 | bounce_matching_headers = Column(Unicode) # XXX |
4796 | 126 | bounce_notify_owner_on_disable = Bool() # XXX | 128 | bounce_notify_owner_on_disable = Column(Boolean) # XXX |
4797 | 127 | bounce_notify_owner_on_removal = Bool() # XXX | 129 | bounce_notify_owner_on_removal = Column(Boolean) # XXX |
4798 | 128 | bounce_score_threshold = Int() # XXX | 130 | bounce_score_threshold = Column(Integer) # XXX |
4799 | 129 | bounce_you_are_disabled_warnings = Int() # XXX | 131 | bounce_you_are_disabled_warnings = Column(Integer) # XXX |
4800 | 130 | bounce_you_are_disabled_warnings_interval = TimeDelta() # XXX | 132 | bounce_you_are_disabled_warnings_interval = Column(Interval) # XXX |
4801 | 131 | forward_unrecognized_bounces_to = Enum(UnrecognizedBounceDisposition) | 133 | forward_unrecognized_bounces_to = Column( |
4802 | 132 | process_bounces = Bool() | 134 | Enum(UnrecognizedBounceDisposition)) |
4803 | 135 | process_bounces = Column(Boolean) | ||
4804 | 133 | # Miscellaneous | 136 | # Miscellaneous |
4834 | 134 | default_member_action = Enum(Action) | 137 | default_member_action = Column(Enum(Action)) |
4835 | 135 | default_nonmember_action = Enum(Action) | 138 | default_nonmember_action = Column(Enum(Action)) |
4836 | 136 | description = Unicode() | 139 | description = Column(Unicode) |
4837 | 137 | digest_footer_uri = Unicode() | 140 | digest_footer_uri = Column(Unicode) |
4838 | 138 | digest_header_uri = Unicode() | 141 | digest_header_uri = Column(Unicode) |
4839 | 139 | digest_is_default = Bool() | 142 | digest_is_default = Column(Boolean) |
4840 | 140 | digest_send_periodic = Bool() | 143 | digest_send_periodic = Column(Boolean) |
4841 | 141 | digest_size_threshold = Float() | 144 | digest_size_threshold = Column(Float) |
4842 | 142 | digest_volume_frequency = Enum(DigestFrequency) | 145 | digest_volume_frequency = Column(Enum(DigestFrequency)) |
4843 | 143 | digestable = Bool() | 146 | digestable = Column(Boolean) |
4844 | 144 | discard_these_nonmembers = Pickle() | 147 | discard_these_nonmembers = Column(PickleType) |
4845 | 145 | emergency = Bool() | 148 | emergency = Column(Boolean) |
4846 | 146 | encode_ascii_prefixes = Bool() | 149 | encode_ascii_prefixes = Column(Boolean) |
4847 | 147 | first_strip_reply_to = Bool() | 150 | first_strip_reply_to = Column(Boolean) |
4848 | 148 | footer_uri = Unicode() | 151 | footer_uri = Column(Unicode) |
4849 | 149 | forward_auto_discards = Bool() | 152 | forward_auto_discards = Column(Boolean) |
4850 | 150 | gateway_to_mail = Bool() | 153 | gateway_to_mail = Column(Boolean) |
4851 | 151 | gateway_to_news = Bool() | 154 | gateway_to_news = Column(Boolean) |
4852 | 152 | goodbye_message_uri = Unicode() | 155 | goodbye_message_uri = Column(Unicode) |
4853 | 153 | header_matches = Pickle() | 156 | header_matches = Column(PickleType) |
4854 | 154 | header_uri = Unicode() | 157 | header_uri = Column(Unicode) |
4855 | 155 | hold_these_nonmembers = Pickle() | 158 | hold_these_nonmembers = Column(PickleType) |
4856 | 156 | info = Unicode() | 159 | info = Column(Unicode) |
4857 | 157 | linked_newsgroup = Unicode() | 160 | linked_newsgroup = Column(Unicode) |
4858 | 158 | max_days_to_hold = Int() | 161 | max_days_to_hold = Column(Integer) |
4859 | 159 | max_message_size = Int() | 162 | max_message_size = Column(Integer) |
4860 | 160 | max_num_recipients = Int() | 163 | max_num_recipients = Column(Integer) |
4861 | 161 | member_moderation_notice = Unicode() | 164 | member_moderation_notice = Column(Unicode) |
4862 | 162 | mime_is_default_digest = Bool() | 165 | mime_is_default_digest = Column(Boolean) |
4863 | 163 | # FIXME: There should be no moderator_password | 166 | # FIXME: There should be no moderator_password |
4891 | 164 | moderator_password = RawStr() | 167 | moderator_password = Column(LargeBinary) # TODO : was RawStr() |
4892 | 165 | newsgroup_moderation = Enum(NewsgroupModeration) | 168 | newsgroup_moderation = Column(Enum(NewsgroupModeration)) |
4893 | 166 | nntp_prefix_subject_too = Bool() | 169 | nntp_prefix_subject_too = Column(Boolean) |
4894 | 167 | nondigestable = Bool() | 170 | nondigestable = Column(Boolean) |
4895 | 168 | nonmember_rejection_notice = Unicode() | 171 | nonmember_rejection_notice = Column(Unicode) |
4896 | 169 | obscure_addresses = Bool() | 172 | obscure_addresses = Column(Boolean) |
4897 | 170 | owner_chain = Unicode() | 173 | owner_chain = Column(Unicode) |
4898 | 171 | owner_pipeline = Unicode() | 174 | owner_pipeline = Column(Unicode) |
4899 | 172 | personalize = Enum(Personalization) | 175 | personalize = Column(Enum(Personalization)) |
4900 | 173 | post_id = Int() | 176 | post_id = Column(Integer) |
4901 | 174 | posting_chain = Unicode() | 177 | posting_chain = Column(Unicode) |
4902 | 175 | posting_pipeline = Unicode() | 178 | posting_pipeline = Column(Unicode) |
4903 | 176 | _preferred_language = Unicode(name='preferred_language') | 179 | _preferred_language = Column('preferred_language', Unicode) |
4904 | 177 | display_name = Unicode() | 180 | display_name = Column(Unicode) |
4905 | 178 | reject_these_nonmembers = Pickle() | 181 | reject_these_nonmembers = Column(PickleType) |
4906 | 179 | reply_goes_to_list = Enum(ReplyToMunging) | 182 | reply_goes_to_list = Column(Enum(ReplyToMunging)) |
4907 | 180 | reply_to_address = Unicode() | 183 | reply_to_address = Column(Unicode) |
4908 | 181 | require_explicit_destination = Bool() | 184 | require_explicit_destination = Column(Boolean) |
4909 | 182 | respond_to_post_requests = Bool() | 185 | respond_to_post_requests = Column(Boolean) |
4910 | 183 | scrub_nondigest = Bool() | 186 | scrub_nondigest = Column(Boolean) |
4911 | 184 | send_goodbye_message = Bool() | 187 | send_goodbye_message = Column(Boolean) |
4912 | 185 | send_welcome_message = Bool() | 188 | send_welcome_message = Column(Boolean) |
4913 | 186 | subject_prefix = Unicode() | 189 | subject_prefix = Column(Unicode) |
4914 | 187 | topics = Pickle() | 190 | topics = Column(PickleType) |
4915 | 188 | topics_bodylines_limit = Int() | 191 | topics_bodylines_limit = Column(Integer) |
4916 | 189 | topics_enabled = Bool() | 192 | topics_enabled = Column(Boolean) |
4917 | 190 | welcome_message_uri = Unicode() | 193 | welcome_message_uri = Column(Unicode) |
4918 | 191 | 194 | ||
4919 | 192 | def __init__(self, fqdn_listname): | 195 | def __init__(self, fqdn_listname): |
4920 | 193 | super(MailingList, self).__init__() | 196 | super(MailingList, self).__init__() |
4921 | @@ -198,14 +201,15 @@ | |||
4922 | 198 | self._list_id = '{0}.{1}'.format(listname, hostname) | 201 | self._list_id = '{0}.{1}'.format(listname, hostname) |
4923 | 199 | # For the pending database | 202 | # For the pending database |
4924 | 200 | self.next_request_id = 1 | 203 | self.next_request_id = 1 |
4930 | 201 | # We need to set up the rosters. Normally, this method will get | 204 | # We need to set up the rosters. Normally, this method will get called |
4931 | 202 | # called when the MailingList object is loaded from the database, but | 205 | # when the MailingList object is loaded from the database, but when the |
4932 | 203 | # that's not the case when the constructor is called. So, set up the | 206 | # constructor is called, SQLAlchemy's `load` event isn't triggered. |
4933 | 204 | # rosters explicitly. | 207 | # Thus we need to set up the rosters explicitly. |
4934 | 205 | self.__storm_loaded__() | 208 | self._post_load() |
4935 | 206 | makedirs(self.data_path) | 209 | makedirs(self.data_path) |
4936 | 207 | 210 | ||
4938 | 208 | def __storm_loaded__(self): | 211 | def _post_load(self, *args): |
4939 | 212 | # This hooks up to SQLAlchemy's `load` event. | ||
4940 | 209 | self.owners = roster.OwnerRoster(self) | 213 | self.owners = roster.OwnerRoster(self) |
4941 | 210 | self.moderators = roster.ModeratorRoster(self) | 214 | self.moderators = roster.ModeratorRoster(self) |
4942 | 211 | self.administrators = roster.AdministratorRoster(self) | 215 | self.administrators = roster.AdministratorRoster(self) |
4943 | @@ -215,6 +219,13 @@ | |||
4944 | 215 | self.subscribers = roster.Subscribers(self) | 219 | self.subscribers = roster.Subscribers(self) |
4945 | 216 | self.nonmembers = roster.NonmemberRoster(self) | 220 | self.nonmembers = roster.NonmemberRoster(self) |
4946 | 217 | 221 | ||
4947 | 222 | @classmethod | ||
4948 | 223 | def __declare_last__(cls): | ||
4949 | 224 | # SQLAlchemy special directive hook called after mappings are assumed | ||
4950 | 225 | # to be complete. Use this to connect the roster instance creation | ||
4951 | 226 | # method with the SA `load` event. | ||
4952 | 227 | listen(cls, 'load', cls._post_load) | ||
4953 | 228 | |||
4954 | 218 | def __repr__(self): | 229 | def __repr__(self): |
4955 | 219 | return '<mailing list "{0}" at {1:#x}>'.format( | 230 | return '<mailing list "{0}" at {1:#x}>'.format( |
4956 | 220 | self.fqdn_listname, id(self)) | 231 | self.fqdn_listname, id(self)) |
4957 | @@ -323,42 +334,42 @@ | |||
4958 | 323 | except AttributeError: | 334 | except AttributeError: |
4959 | 324 | self._preferred_language = language | 335 | self._preferred_language = language |
4960 | 325 | 336 | ||
4962 | 326 | def send_one_last_digest_to(self, address, delivery_mode): | 337 | @dbconnection |
4963 | 338 | def send_one_last_digest_to(self, store, address, delivery_mode): | ||
4964 | 327 | """See `IMailingList`.""" | 339 | """See `IMailingList`.""" |
4965 | 328 | digest = OneLastDigest(self, address, delivery_mode) | 340 | digest = OneLastDigest(self, address, delivery_mode) |
4967 | 329 | Store.of(self).add(digest) | 341 | store.add(digest) |
4968 | 330 | 342 | ||
4969 | 331 | @property | 343 | @property |
4971 | 332 | def last_digest_recipients(self): | 344 | @dbconnection |
4972 | 345 | def last_digest_recipients(self, store): | ||
4973 | 333 | """See `IMailingList`.""" | 346 | """See `IMailingList`.""" |
4976 | 334 | results = Store.of(self).find( | 347 | results = store.query(OneLastDigest).filter( |
4975 | 335 | OneLastDigest, | ||
4977 | 336 | OneLastDigest.mailing_list == self) | 348 | OneLastDigest.mailing_list == self) |
4978 | 337 | recipients = [(digest.address, digest.delivery_mode) | 349 | recipients = [(digest.address, digest.delivery_mode) |
4979 | 338 | for digest in results] | 350 | for digest in results] |
4981 | 339 | results.remove() | 351 | results.delete() |
4982 | 340 | return recipients | 352 | return recipients |
4983 | 341 | 353 | ||
4984 | 342 | @property | 354 | @property |
4986 | 343 | def filter_types(self): | 355 | @dbconnection |
4987 | 356 | def filter_types(self, store): | ||
4988 | 344 | """See `IMailingList`.""" | 357 | """See `IMailingList`.""" |
4993 | 345 | results = Store.of(self).find( | 358 | results = store.query(ContentFilter).filter( |
4994 | 346 | ContentFilter, | 359 | ContentFilter.mailing_list == self, |
4995 | 347 | And(ContentFilter.mailing_list == self, | 360 | ContentFilter.filter_type == FilterType.filter_mime) |
4992 | 348 | ContentFilter.filter_type == FilterType.filter_mime)) | ||
4996 | 349 | for content_filter in results: | 361 | for content_filter in results: |
4997 | 350 | yield content_filter.filter_pattern | 362 | yield content_filter.filter_pattern |
4998 | 351 | 363 | ||
4999 | 352 | @filter_types.setter | 364 | @filter_types.setter |
5000 | 353 | def filter_types(self, sequence): |