Merge lp:~jimpop/mailman/dmarc-moderation-addresses into lp:mailman/2.1

[Jim Popovitch]

Patch for dmarc_moderation_addresses to automatically apply dmarc_moderation_action against a regexp of addresses.

This patch adds the new variable dmarc_moderation_addresses which accepts a list of regexp addresses that will automatically apply dmarc_moderation_action. This is for circumstances where a domain's internal policies reject external emails From their own domain (looking at you hydro.qc.ca). Even though there is no DMARC record, the dmarc_moderation_action is the perfect solution for this problem.

[Jim Popovitch (jimpop)]

The first feedback should include recommendations for improving the logical OR on Mailman/Handlers/SpamDetect.py:32 :-)

[Mark Sapiro (msapiro)]

Regarding line 32 in the diff, I don't think there should be an at_list argument in the call to GetPattern. Specifying a value for this says that one can use the @listname syntax to specify a listname, all of whose members will be included in this list's dmarc_moderation_addresses. It doesn't seem this is useful, as the entries would likely be regexps matching the domain rather than individual addresses. In any case, I would write

            if (mlist.GetPattern(addr,
                                 mlist.dmarc_moderation_addresses,
                                 at_list='dmarc_moderation_addresses'
                                ) or
                Utils.IsDMARCProhibited(mlist, addr)):

with the at_list argument or

            if (mlist.GetPattern(addr, mlist.dmarc_moderation_addresses) or
                Utils.IsDMARCProhibited(mlist, addr)):

without it.

I understand the motivation for this from your description and I think it's valid, but I think some explanation of the intent is also required in the VARHELP for the setting.

Also, I'll be traveling for the next week and probably won't be able to do more until I return.

[lp:~jimpop/mailman/dmarc-moderation-addresses]

1805. By Jim Popovitch on 2018-12-01

Updated SpamDetect.py and VARHELP based on feedback from Mark S.

[Mark Sapiro (msapiro)]

I made a minor change to VARHELP and added a NEWS item.

Thanks for your contribution.

review: Approve (code)

=== modified file 'Mailman/Gui/Privacy.py'
--- Mailman/Gui/Privacy.py	2018-07-15 03:18:00 +0000
+++ Mailman/Gui/Privacy.py	2018-12-01 04:14:39 +0000
@@ -356,6 +356,20 @@
              be sent to anyone who posts to this list from a domain
              with a DMARC Reject%(quarantine)s Policy.""")),
              
+            ('dmarc_moderation_addresses', mm_cfg.EmailListEx, (10, WIDTH), 1,
+             _("""List of addresses (or regexps) whose posts should always apply
+             <a href="?VARHELP=privacy/sender/dmarc_moderation_action"
+             >dmarc_moderation_action</a> 
+             regardless of any domain specific DMARC Policy."""),
+
+             _("""Postings from any of these addresses will automatically
+             apply any DMARC action mitigation.  This can be utilized to
+             automatically wrap or munge postings from known addresses or
+             domains such as internal domains.
+
+             <p>Add member addresses one per line; start the line with a ^
+             character to designate a regular expression match.""")),
+             
             ('dmarc_wrapped_message_text', mm_cfg.Text, (10, WIDTH), 1,
              _("""If dmarc_moderation_action applies and is Wrap Message,
              and this text is provided, the text will be placed in a

=== modified file 'Mailman/Handlers/SpamDetect.py'
--- Mailman/Handlers/SpamDetect.py	2018-06-17 23:47:34 +0000
+++ Mailman/Handlers/SpamDetect.py	2018-12-01 04:14:39 +0000
@@ -109,7 +109,8 @@
         msgdata['from_is_list'] = 0
         dn, addr = parseaddr(msg.get('from'))
         if addr and mlist.dmarc_moderation_action > 0:
-            if Utils.IsDMARCProhibited(mlist, addr):
+            if (mlist.GetPattern(addr, mlist.dmarc_moderation_addresses) or
+                Utils.IsDMARCProhibited(mlist, addr)):
                 # Note that for dmarc_moderation_action, 0 = Accept, 
                 #    1 = Munge, 2 = Wrap, 3 = Reject, 4 = Discard
                 if mlist.dmarc_moderation_action == 1:

=== modified file 'Mailman/MailList.py'
--- Mailman/MailList.py	2018-06-21 16:23:09 +0000
+++ Mailman/MailList.py	2018-12-01 04:14:39 +0000
@@ -424,6 +424,7 @@
         self.dmarc_none_moderation_action = (
             mm_cfg.DEFAULT_DMARC_NONE_MODERATION_ACTION)
         self.dmarc_moderation_notice = ''
+	self.dmarc_moderation_addresses = []
         self.dmarc_wrapped_message_text = (
             mm_cfg.DEFAULT_DMARC_WRAPPED_MESSAGE_TEXT)
         self.equivalent_domains = (

=== modified file 'Mailman/Version.py'
--- Mailman/Version.py	2018-07-24 21:58:31 +0000
+++ Mailman/Version.py	2018-12-01 04:14:39 +0000
@@ -37,7 +37,7 @@
                (REL_LEVEL << 4)  | (REL_SERIAL << 0))
 
 # config.pck schema version number
-DATA_FILE_VERSION = 110
+DATA_FILE_VERSION = 111
 
 # qfile/*.db schema version number
 QFILE_SCHEMA_VERSION = 3

=== modified file 'Mailman/versions.py'
--- Mailman/versions.py	2018-06-17 23:47:34 +0000
+++ Mailman/versions.py	2018-12-01 04:14:39 +0000
@@ -497,6 +497,7 @@
     add_only_if_missing('dmarc_none_moderation_action',
                        mm_cfg.DEFAULT_DMARC_NONE_MODERATION_ACTION)
     add_only_if_missing('dmarc_moderation_notice', '')
+    add_only_if_missing('dmarc_moderation_addresses', [])
     add_only_if_missing('dmarc_wrapped_message_text',
                        mm_cfg.DEFAULT_DMARC_WRAPPED_MESSAGE_TEXT)
     add_only_if_missing('member_verbosity_threshold',