Merge lp:~jimpop/mailman/dmarc-reject into lp:mailman/2.1

[Barry Warsaw]

[Mark Sapiro (msapiro)]

The merge is clean. I will be reviewing/testing further tomorrow.

Thanks greatly Jim, and I'm sorry it took the recent "wake-up" call to get action on this. It would have been better if I'd done this sooner.

review: Approve

[Jim Popovitch (jimpop)]

> The merge is clean. I will be reviewing/testing further tomorrow.
>
> Thanks greatly Jim, and I'm sorry it took the recent "wake-up" call to get
> action on this. It would have been better if I'd done this sooner.

Awesome, I am glad to see this headed in this direction. Mark, no need for any regret, I'm just glad to contribute something that is hopefully helpful.

[Mark Sapiro (msapiro)]

I have merged this branch as a starting point. I am refactoring a few things including removing the Hold option and adding Munge From and Wrap Message options so the choices are "Accept, Wrap Message, Munge From, Reject and Discard" and the list owner is not allowed to set an option with a lower value than the site default.

I am also keeping the general Wrap Message and Munge From list options which will apply to messages From: domains without DMARC reject/quarantine and all messages if this setting is Accept.

I want to thank you again for this work Jim and also point out one problem I found.

The code in Moderate includes

    if sender:
        if Utils.IsDmarcProhibited(sender):

but sender may not be the From: address. I made this

    dn, addr = email.Utils.parseaddr(msg.get('from'))
    if addr:
        if Utils.IsDmarcProhibited(addr)

[Jim Popovitch (jimpop)]

Thanks for feedback Mark. Should I push that change to dmarc-reject, or let that branch die now that the code is being merged?

[Mark Sapiro (msapiro)]

Wait until I publish the merged 2.1 branch. Then if you think this branch is still useful, fix it. Otherwise let it die.

[Jim Popovitch (jimpop)]

IMO, the dmarc-reject branch is useless after this merge, so I will let it die.

Thanks!

=== modified file 'Mailman/Gui/Privacy.py'
--- Mailman/Gui/Privacy.py	2009-01-11 16:06:13 +0000
+++ Mailman/Gui/Privacy.py	2014-04-14 01:04:55 +0000
@@ -235,6 +235,30 @@
              >rejection notice</a> to
              be sent to moderated members who post to this list.""")),
 
+            ('dmarc_moderation_action', mm_cfg.Radio,
+             (_('Accept'), _('Hold'), _('Reject'), _('Discard')), 0,
+             _("""Action to take when anyone posts to the
+             list from a domain with a DMARC Reject/Quarantine Policy."""),
+             _("""<ul><li><b>Hold</b> -- this holds the message for approval
+             by the list moderators.
+
+             <p><li><b>Reject</b> -- this automatically rejects the message by
+             sending a bounce notice to the post's author.  The text of the
+             bounce notice can be <a
+             href="?VARHELP=privacy/sender/dmarc_moderation_notice"
+             >configured by you</a>.
+
+             <p><li><b>Discard</b> -- this simply discards the message, with
+             no notice sent to the post's author.
+             </ul>""")),
+
+            ('dmarc_moderation_notice', mm_cfg.Text, (10, WIDTH), 1,
+             _("""Text to include in any
+             <a href="?VARHELP/privacy/sender/dmarc_moderation_action"
+             >rejection notice</a> to
+             be sent to anyone who posts to this list from a domain
+             with DMARC Reject/Quarantine Policy.""")),
+
             _('Non-member filters'),
 
             ('accept_these_nonmembers', mm_cfg.EmailListEx, (10, WIDTH), 1,
 
=== modified file 'Mailman/Handlers/Moderate.py'
--- Mailman/Handlers/Moderate.py	2013-11-19 04:52:17 +0000
+++ Mailman/Handlers/Moderate.py	2014-04-14 01:04:55 +0000
@@ -56,6 +56,25 @@
     else:
         sender = None
     if sender:
+        if Utils.IsDmarcProhibited(sender):
+            # Note that for dmarc_moderation_action, 0 = Accept, 
+            #    1 = Hold, 2 = Reject, 3 = Discard
+            if mlist.dmarc_moderation_action == 1:
+                msgdata['sender'] = sender
+                Hold.hold_for_approval(mlist, msg, msgdata,
+                                       ModeratedMemberPost)
+            elif mlist.dmarc_moderation_action == 2:
+                # Reject
+                text = mlist.dmarc_moderation_notice
+                if text:
+                    text = Utils.wrap(text)
+                else:
+                    # Use the default RejectMessage notice string
+                    text = None
+                raise Errors.RejectMessage, text
+            elif mlist.dmarc_moderation_action == 3:
+                raise Errors.DiscardMessage
+
         # If the member's moderation flag is on, then perform the moderation
         # action.
         if mlist.getMemberOption(sender, mm_cfg.Moderate):
 
=== modified file 'Mailman/MailList.py'
--- Mailman/MailList.py	2013-09-28 23:08:15 +0000
+++ Mailman/MailList.py	2014-04-14 01:04:55 +0000
@@ -389,6 +389,8 @@
         # 2==Discard
         self.member_moderation_action = 0
         self.member_moderation_notice = ''
+        self.dmarc_moderation_action = mm_cfg.DEFAULT_DMARC_MODERATION_ACTION
+        self.dmarc_moderation_notice = ''
         self.accept_these_nonmembers = []
         self.hold_these_nonmembers = []
         self.reject_these_nonmembers = []
 
=== modified file 'Mailman/Utils.py'
--- Mailman/Utils.py	2013-12-07 01:19:28 +0000
+++ Mailman/Utils.py	2014-04-14 01:04:55 +0000
@@ -35,6 +35,7 @@
 import base64
 import random
 import urlparse
+import collections
 import htmlentitydefs
 import email.Header
 import email.Iterators
@@ -71,6 +72,13 @@
     True = 1
     False = 0
 
+try:
+    import dns.resolver
+    from dns.exception import DNSException
+    dns_resolver = True
+except ImportError:
+    dns_resolver = False
+
 EMPTYSTRING = ''
 UEMPTYSTRING = u''
 NL = '\n'
@@ -1058,3 +1066,78 @@
     else:
         return False
 
+
+# This takes an email address, and returns True if DMARC policy is p=reject
+def IsDmarcProhibited(email):
+    if not dns_resolver:
+         return False
+
+    email = email.lower()
+    at_sign = email.find('@')
+    if at_sign < 1:
+        return False
+    dmarc_domain = '_dmarc.' + email[at_sign+1:]
+
+    try:
+        resolver = dns.resolver.Resolver()
+        resolver.timeout = 3
+        resolver.lifetime = 5
+        txt_recs = resolver.query(dmarc_domain, dns.rdatatype.TXT)
+    except (dns.resolver.NXDOMAIN, dns.resolver.NoAnswer):
+        return False
+    except DNSException, e:
+        syslog('error', 'DNSException: Unable to query DMARC policy for %s (%s). %s',
+              email, dmarc_domain, e.__class__)
+        return False
+    else:
+# people are already being dumb, don't trust them to provide honest DNS
+# where the answer section only contains what was asked for, nor to include
+# CNAMEs before the values they point to.
+        full_record = ""
+        results_by_name = collections.defaultdict(list)
+        cnames = {}
+        want_names = set([dmarc_domain + '.'])
+        for txt_rec in txt_recs.response.answer:
+            if txt_rec.rdtype == dns.rdatatype.CNAME:
+                cnames[txt_rec.name.to_text()] = txt_rec.items[0].target.to_text()
+            if txt_rec.rdtype != dns.rdatatype.TXT:
+                continue
+            results_by_name[txt_rec.name.to_text()].append("".join(txt_rec.items[0].strings))
+        expands = list(want_names)
+        seen = set(expands)
+        while expands:
+            item = expands.pop(0)
+            if item in cnames:
+                if cnames[item] in seen:
+                    continue # cname loop
+                expands.append(cnames[item])
+                seen.add(cnames[item])
+                want_names.add(cnames[item])
+                want_names.discard(item)
+
+        if len(want_names) != 1:
+            syslog('error', 'multiple DMARC entries in results for %s, processing each to be strict',
+                    dmarc_domain)
+        for name in want_names:
+            if name not in results_by_name:
+                continue
+            dmarcs = filter(lambda n: n.startswith('v=DMARC1;'), results_by_name[name])
+            if len(dmarcs) == 0:
+                return False
+            if len(dmarcs) > 1:
+                syslog('error', 'RRset of TXT records for %s has %d v=DMARC1 entries; testing them all',
+                        dmarc_domain, len(dmarc))
+            for entry in dmarcs:
+                if re.search(r'\bp=reject\b', entry, re.IGNORECASE):
+                    syslog('info', 'DMARC lookup for %s (%s) found p=reject in %s = %s',
+                            email, dmarc_domain, name, entry)
+                    return True
+
+                if re.search(r'\bp=quarantine\b', entry, re.IGNORECASE):
+                    syslog('info', 'DMARC lookup for %s (%s) found p=quarantine in %s = %s',
+                            email, dmarc_domain, name, entry)
+                    return True
+
+    return False
+
+
 
=== modified file 'Mailman/versions.py'
--- Mailman/versions.py	2013-09-28 23:08:15 +0000
+++ Mailman/versions.py	2014-04-14 01:04:55 +0000
@@ -388,6 +388,9 @@
     # the current GUI description model.  So, 0==Hold, 1==Reject, 2==Discard
     add_only_if_missing('member_moderation_action', 0)
     add_only_if_missing('member_moderation_notice', '')
+    add_only_if_missing('dmarc_moderation_action', 
+                        mm_cfg.DEFAULT_DMARC_MODERATION_ACTION)
+    add_only_if_missing('dmarc_moderation_notice', '')
     add_only_if_missing('new_member_options',
                         mm_cfg.DEFAULT_NEW_MEMBER_OPTIONS)
     # Emergency moderation flag