Merge lp:~danci-emanuel/mailman/dlist_runner into lp:~danci-emanuel/mailman/mm_3.0b2

[Emanuel Danci]

[Robin J (robin-jeffries)]

What are you doing with this stuff about "test_field"? It's clearly something not finished yet (which is OK), but what is it a proxy for?

in sqlite.sql, lets make this more clear. Instead of
   Is the Dynamic sublists feature enabled for this list?

How about:
   Allow people to (un)subscribe to individual conversations?

What does the change to maxsize.py mean? (looks like you took out part of the copyright notice)

I don't see that you have the actual Runner functionality. Did I miss that or does it go somewhere else?

 Could you for the next round (since it is part of the runner), look at what Sneha Priscilla did to split the dlistRunner into an early and a late part (the reason was to deal with emails with a to: address of the form <email address hidden> that also have 'unsubscribe' in the subject line. The actual dlist work had to go late, but shunting these off to keep them from being picked up by the administrivia processor had to happen early. It could be that things are done in a different order here, and you might be able to show that this isn't needed, but please look into it as part of the Runner work.

[Emanuel Danci (danci-emanuel)]

> What are you doing with this stuff about "test_field"? It's clearly something
> not finished yet (which is OK), but what is it a proxy for?

 I added it for some testing, but I will remove it as soon as the DlistUtils will be completed.

> in sqlite.sql, lets make this more clear. Instead of
> Is the Dynamic sublists feature enabled for this list?
>
> How about:
> Allow people to (un)subscribe to individual conversations?

Done, I have changed it.

> What does the change to maxsize.py mean? (looks like you took out part of the
> copyright notice)

Due to some quick editing in vim, I deleted it by mistake. I placed the missing row back.

> I don't see that you have the actual Runner functionality. Did I miss that or
> does it go somewhere else?
>
> Could you for the next round (since it is part of the runner), look at what
> Sneha Priscilla did to split the dlistRunner into an early and a late part
> (the reason was to deal with emails with a to: address of the form
> <email address hidden> that also have 'unsubscribe' in the subject
> line. The actual dlist work had to go late, but shunting these off to keep
> them from being picked up by the administrivia processor had to happen early.
> It could be that things are done in a different order here, and you might be
> able to show that this isn't needed, but please look into it as part of the
> Runner work.

Ok, I will look at what Sneha Priscilla did and see how I can adapt that here.

Unmerged revisions

7164. By Emanuel Danci on 2012-09-11

The DlistRunner was added.

7163. By Emanuel Danci on 2012-09-11

DlistRunner added

7162. By Emanuel Danci on 2012-09-11

Merge from the main branch

7161. By Emanuel Danci on 2012-09-11

Add the dlist runner

7160. By Emanuel Danci on 2012-07-10

Contains the changes needed for supporting Dynamic sublists

7159. By Emanuel Danci on 2012-07-10

Contains the changes needed for supporting Dynamic sublists

7158. By Emanuel Danci on 2012-07-10

Dynamic sublists support added

7157. By Emanuel Danci on 2012-07-09

Contains the changes needed for supporting Dynamic sublists

7156. By Emanuel Danci on 2012-07-09

contains UI changes needed for supporing Dynamic sublists

=== modified file 'src/mailman/config/config.py'
--- src/mailman/config/config.py	2012-07-02 20:56:35 +0000
+++ src/mailman/config/config.py	2012-09-11 20:41:22 +0000
@@ -154,6 +154,7 @@
             messages_dir            = category.messages_dir,
             archive_dir             = category.archive_dir,
             queue_dir               = category.queue_dir,
+            dlist_queue_dir         = category.dlist_queue_dir,
             var_dir                 = var_dir,
             template_dir            = (
                 os.path.dirname(mailman.templates.__file__)

=== modified file 'src/mailman/config/mailman.cfg'
--- src/mailman/config/mailman.cfg	2012-03-31 10:51:42 +0000
+++ src/mailman/config/mailman.cfg	2012-09-11 20:41:22 +0000
@@ -88,4 +88,12 @@
 [runner.digest]
 class: mailman.runners.digest.DigestRunner
 
+# add the dlist runner 
+[runner.dlist]
+class: mailman.runners.dlist.DlistRunner
+
 [style.default]
+
+[database]
+class: mailman.database.postgresql.PostgreSQLDatabase
+url: postgres://mailman:mailman@localhost/mailman

=== modified file 'src/mailman/config/schema.cfg'
--- src/mailman/config/schema.cfg	2012-08-16 01:08:39 +0000
+++ src/mailman/config/schema.cfg	2012-09-11 20:41:22 +0000
@@ -98,6 +98,9 @@
 var_dir: /var/tmp/mailman
 # This is where the Mailman queue files directories will be created.
 queue_dir: $var_dir/queue
+
+dlist_queue_dir: $queue_dir/dlist
+
 # This is the directory containing the Mailman 'runner' and 'master' commands
 # if set to the string '$argv', it will be taken as the directory containing
 # the 'bin/mailman' command.

=== modified file 'src/mailman/core/pipelines.py'
--- src/mailman/core/pipelines.py	2012-04-26 02:08:22 +0000
+++ src/mailman/core/pipelines.py	2012-09-11 20:41:22 +0000
@@ -111,6 +111,8 @@
     name = 'default-posting-pipeline'
     description = _('The built-in posting pipeline.')
 
+    ## dlist_custom ##
+    ## TODO: add the dlist handler to the default-posting-pipeline ## 
     _default_handlers = (
         'mime-delete',
         'tagger',

=== modified file 'src/mailman/database/schema/postgres.sql'
--- src/mailman/database/schema/postgres.sql	2012-07-23 14:40:53 +0000
+++ src/mailman/database/schema/postgres.sql	2012-09-11 20:41:22 +0000
@@ -19,6 +19,7 @@
     admin_notify_mchanges BOOLEAN,
     administrivia BOOLEAN,
     advertised BOOLEAN,
+    dlists_enabled BOOLEAN,
     anonymous_list BOOLEAN,
     archive BOOLEAN,
     archive_private BOOLEAN,
@@ -174,6 +175,10 @@
     id SERIAL NOT NULL,
     display_name TEXT,
     password BYTEA,
+
+    --- custom ----
+    test_field TEXT,
+    ---------------
     _user_id UUID,
     _created_on TIMESTAMP,
     _preferred_address_id INTEGER,

=== modified file 'src/mailman/database/schema/sqlite.sql'
--- src/mailman/database/schema/sqlite.sql	2012-04-08 16:15:29 +0000
+++ src/mailman/database/schema/sqlite.sql	2012-09-11 20:41:22 +0000
@@ -118,6 +118,7 @@
     admin_notify_mchanges BOOLEAN,
     administrivia BOOLEAN,
     advertised BOOLEAN,
+    dlists_enabled BOOLEAN,
     anonymous_list BOOLEAN,
     archive BOOLEAN,
     archive_private BOOLEAN,

=== modified file 'src/mailman/interfaces/mailinglist.py'
--- src/mailman/interfaces/mailinglist.py	2012-09-05 01:31:50 +0000
+++ src/mailman/interfaces/mailinglist.py	2012-09-11 20:41:22 +0000
@@ -132,6 +132,9 @@
         """Advertise this mailing list when people ask for an overview of the
         available mailing lists.""")
 
+    dlists_enabled = Attribute(
+	    """Is the Dynamic sublists feature enabled for this list?""")
+	
     # Contact addresses
 
     posting_address = Attribute(
@@ -205,8 +208,8 @@
     administrators = Attribute(
         """The IUser administrators of this mailing list.
 
-        This includes the IUsers who are both owners and moderators of the
-        mailing list.""")
+		This includes the IUsers who are both owners and moderators of the
+		mailing list.""")
 
     members = Attribute(
         """An iterator over all the members of the mailing list, regardless of

=== modified file 'src/mailman/interfaces/user.py'
--- src/mailman/interfaces/user.py	2012-04-07 21:59:30 +0000
+++ src/mailman/interfaces/user.py	2012-09-11 20:41:22 +0000
@@ -53,6 +53,11 @@
 class IUser(Interface):
     """A basic user."""
 
+    ## dlist custom ##
+    test_field = Attribute(
+            """Test field""")
+    #######################
+
     display_name = Attribute(
         """This user's display name.""")
 

=== modified file 'src/mailman/model/mailinglist.py'
--- src/mailman/model/mailinglist.py	2012-09-05 01:31:50 +0000
+++ src/mailman/model/mailinglist.py	2012-09-11 20:41:22 +0000
@@ -84,6 +84,7 @@
     allow_list_posts = Bool()
     include_rfc2369_headers = Bool()
     advertised = Bool()
+    dlists_enabled = Bool()
     anonymous_list = Bool()
     # Attributes not directly modifiable via the web u/i
     created_at = DateTime()

=== modified file 'src/mailman/model/user.py'
--- src/mailman/model/user.py	2012-04-26 02:08:22 +0000
+++ src/mailman/model/user.py	2012-09-11 20:41:22 +0000
@@ -52,6 +52,10 @@
     """Mailman users."""
 
     id = Int(primary=True)
+
+    #####dlist custom#######
+    test_field = Unicode()
+    ################
     display_name = Unicode()
     _password = RawStr(name='password')
     _user_id = UUID()
@@ -73,6 +77,11 @@
         self._user_id = user_id
         self.display_name = ('' if display_name is None else display_name)
         self.preferences = preferences
+
+
+        ### dlist custom ###
+        #init test_field
+        ###############################
         store.add(self)
 
     def __repr__(self):

=== modified file 'src/mailman/rest/configuration.py'
--- src/mailman/rest/configuration.py	2012-08-21 00:55:24 +0000
+++ src/mailman/rest/configuration.py	2012-09-11 20:41:22 +0000
@@ -163,6 +163,7 @@
     admin_notify_mchanges=GetterSetter(as_boolean),
     administrivia=GetterSetter(as_boolean),
     advertised=GetterSetter(as_boolean),
+    dlists_enabled = GetterSetter(as_boolean),
     anonymous_list=GetterSetter(as_boolean),
     autorespond_owner=GetterSetter(enum_validator(ResponseAction)),
     autorespond_postings=GetterSetter(enum_validator(ResponseAction)),
@@ -206,11 +207,10 @@
     welcome_message_uri=GetterSetter(unicode),
     )
 
-
 VALIDATORS = ATTRIBUTES.copy()
 for attribute, gettersetter in VALIDATORS.items():
     if gettersetter.decoder is None:
-        del VALIDATORS[attribute]
+   		del VALIDATORS[attribute]
 
 
 

@@ -234,7 +234,7 @@
             return http.bad_request(
                 [], b'Unknown attribute: {0}'.format(self._attribute))
         else:
-            attribute = self._attribute
+            attribute = self._attribute       
             value = ATTRIBUTES[attribute].get(self._mlist, attribute)
             resource[attribute] = value
         return http.ok([], etag(resource))

=== modified file 'src/mailman/rules/max_size.py'
--- src/mailman/rules/max_size.py	2012-04-26 02:08:22 +0000
+++ src/mailman/rules/max_size.py	2012-09-11 20:41:22 +0000
@@ -3,7 +3,6 @@
 # This file is part of GNU Mailman.
 #
 # GNU Mailman is free software: you can redistribute it and/or modify it under
-# the terms of the GNU General Public License as published by the Free
 # Software Foundation, either version 3 of the License, or (at your option)
 # any later version.
 #

=== added file 'src/mailman/runners/dlist.py'
--- src/mailman/runners/dlist.py	1970-01-01 00:00:00 +0000
+++ src/mailman/runners/dlist.py	2012-09-11 20:41:22 +0000
@@ -0,0 +1,34 @@
+# Copyright (C) 1998,1999,2000,2001 by the Free Software Foundation,     Inc.
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+ # as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-130    7, USA.
+ 
+"""Dynamic sublist processor."""
+
+from mailman.config import config
+from mailman.core.runner import Runner
+
+__metaclass__ = type
+__all__ = [
+    'DlistRunner'
+    ]
+
+class DlistRunner(Runner):
+    """The dlist runner."""
+
+    def _dispose(self, mlist, msg, msgdata):
+        """See 'IRunner'."""
+        
+        config.switchboards['out'].enqueue(msg, msgdata)
+        return False
+

=== modified file 'src/mailman/styles/default.py'
--- src/mailman/styles/default.py	2012-09-05 01:31:50 +0000
+++ src/mailman/styles/default.py	2012-09-11 20:41:22 +0000
@@ -24,12 +24,9 @@
     'DefaultStyle',
     ]
 
-
 # XXX Styles need to be reconciled with lazr.config.
-
 from datetime import timedelta
 from zope.interface import implementer
-
 from mailman.core.i18n import _
 from mailman.interfaces.action import Action, FilterAction
 from mailman.interfaces.archiver import ArchivePolicy
@@ -63,6 +60,7 @@
         mlist.new_member_options = 256
         mlist.respond_to_post_requests = True
         mlist.advertised = True
+        mlist.dlists_enabled = False 
         mlist.max_num_recipients = 10
         mlist.max_message_size = 40 # KB
         mlist.reply_goes_to_list = ReplyToMunging.no_munging

=== modified file 'src/mailman/testing/helpers.py'
--- src/mailman/testing/helpers.py	2012-07-21 22:06:43 +0000
+++ src/mailman/testing/helpers.py	2012-09-11 20:41:22 +0000
@@ -38,6 +38,14 @@
     'subscribe',
     'temporary_db',
     'wait_for_webservice',
+    'print_clog',
+    'enq_messages',
+    'CLS',
+    'print_lists',
+    'check_qfiles',
+    'check_dlist_qfiles',
+    'check_out_qfiles',
+    'check_shunt_qfiles',
     ]
 
 
@@ -531,3 +539,70 @@
         bytes_payload = payload.encode('utf-8')
         for line in bytes_payload.splitlines():
             yield line
+
+#######################################
+# dlist_custom classes/functions 
+# XXX: remove the classes/functions
+# that are no longer needed after 
+# development is complete
+#######################################
+
+# print a custom message into the testing log file 
+def print_clog(string, filename=None):
+    
+     if filename is None:
+         filename = "/home/zalman/Desktop/testing_log.txt"
+
+     with open(filename, 'w+') as f:
+         f.write(string)
+
+# enqueue test messages into a particular queue
+def enq_messages(queue, messages):
+
+    switchboard = config.switchboards[queue]
+
+    for i in range(len(messages)):
+        switchboard.enqueue(messages[i].msg, messages[i].msgdata)
+
+# used to clear the python console while testing code
+class CLS(object):
+    def __repr__(self):
+        os.system("clear")
+        return ''
+
+# print existing lists
+def print_lists():
+
+    from mailman.interfaces.listmanager import IListManager
+    from zope.component import getUtility
+
+    list_manager = getUtility(IListManager)
+    for name in sorted(list_manager.names):
+        print (name)
+
+# check the number of files from a given queue
+def check_qfiles(qdir):
+    files = {}
+    for qfile in os.listdir(qdir):
+        root, ext = os.path.splitext(qfile)
+        files[ext] = files.get(ext, 0) + 1
+    if len(files) == 0:
+        print ('empty')
+    for ext in sorted(files):
+        print ('{0}: {1}'.format(ext, files[ext]))
+
+# check the number of files from the dlist qdir
+def check_dlist_qfiles():
+    check_qfiles(config.DLIST_QUEUE_DIR)
+
+# check the number of files from the out qdir
+def check_out_qfiles():
+    qdir = os.path.join(config.QUEUE_DIR, 'out')
+    check_qfiles(qdir)
+
+# check the number of files from the shunt qdir
+def check_shunt_qfiles():
+    qdir = os.path.join(config.QUEUE_DIR, 'shunt')
+    check_qfiles(qdir)
+
+

=== modified file 'src/mailman/testing/testing.cfg'
--- src/mailman/testing/testing.cfg	2012-08-16 01:08:39 +0000
+++ src/mailman/testing/testing.cfg	2012-09-11 20:41:22 +0000
@@ -88,3 +88,7 @@
 [language.fr]
 description: French
 charset: iso-8859-1
+
+[database]
+class: mailman.database.postgresql.PostgreSQLDatabase
+url: postgres://mailman:mailman@local:/mailman