Merge lp:~jimpop/mailman/mailman-auto-mod-verbose-members into lp:mailman/2.1

[Jim Popovitch]

Auto-Moderate verbose members

This is a simple patch to enable the following two Privacy/Sender features:

  member_verbosity_threshold
     Ceiling on acceptable number of member posts, per interval, before automatic moderation.

  member_verbosity_interval
     Number of seconds to use in determining whether or not to automatically moderate a member.

The code tracks member posts in a simple python dictionary, and then checks the dictionary each time a post is processed via Mailman/Handlers/SpamDetect.py. If a member exceeds X posts in Y time, then the member is automatically moderated.

[lp:~jimpop/mailman/mailman-auto-mod-verbose-members]

1587. By <email address hidden> on 2015-11-04

Removed 2 development debugging lines

[Mark Sapiro (msapiro)]

It is not clear why you have

 add_only_if_missing('member_verbosity_threshold', 0)

in versions.py instead of

 add_only_if_missing('member_verbosity_threshold',
                     mm_cfg.DEFAULT_MEMBER_VERBOSITY_THRESHOLD)

Oversight perhaps. Also I think Defaults.py.in should probably have

DEFAULT_MEMBER_VERBOSITY_THRESHOLD = 0

Also, the first few lines of Utils.IsVerboseMember() seem unnecessary, and I'm not completely comfortable with recentMemberPostings being a global in Utils.py rather than a list attribute.

And, do you really see 5 posts from the same member in 5 minutes?

Finally, I'm curious if you ever saw <https://www.msapiro.net/scripts/PostLimit.py>. I guess it is very different in detail, but the goal may be similar.

I've been reluctant to actually implement that on my production server, although it has been suggested as a way to control one member (I'd rather have a conversation than hit him with a hammer, although hammers have their place.)

review: Needs Information

[Jim Popovitch (jimpop)]

Hi Mark, thank you. I have made the following improvements based on your feedback:

Mailman/versions.py:
    set mm_cfg.DEFAULT_MEMBER_VERBOSITY_THRESHOLD rather than hardcoded "0".

Mailman/Defaults.py.in:
    DEFAULT_MEMBER_VERBOSITY_THRESHOLD now defaults to 0

Mailman/Utils.py:
    Removed unnecessary value checks from the top of Utils.IsVerboseMember()

Mailman/Handlers/SpamDetect.py
    Added "mlist.member_verbosity_threshold > 0" test before call to Utils.IsVerboseMember()

I set recentMemberPostings as a global because I felt it was important to measure a members posting pattern across all lists, not a single list. Although I currently only moderated the user on the single impacted list, ideally I'd like to find a way to moderate the user on all subscribed lists.

NANOG recently had a problem of hundreds of spam posts in a matter of minutes. Search for "Fw: new messsage" on http://mailman.nanog.org/pipermail/nanog/2015-October/thread.html (DO NOT CLICK ON ANY OF THE EMAILED LINKS IN THOSE NANOG POSTS). There were so many posts it clogged their outbound queues for at least 1 day.

I have not seen PostLimit before, thanks. I honestly don't care which version makes it into Mailman, but there is clear evidence that spamvertised postings, using a known member address, is going to be a huge problem going forward.

[lp:~jimpop/mailman/mailman-auto-mod-verbose-members]

1588. By <email address hidden> on 2015-11-04

Improvements based on feedback from Mark Sapiro
   https://code.launchpad.net/~jimpop/mailman/mailman-auto-mod-verbose-members/+merge/276706/comments/699744

[Mark Sapiro (msapiro)]

Thanks for your replies and changes. I now have a much clearer understanding of the issue that prompts this. I will probably do a little tweaking, but it looks good and I will get it in the next release.

It really has different goals from PostLimit which I did as a response to some request on Mailman-users.

Actually, we could have used something like this for a flood of bogus subscription requests a while back, but the addresses were predictable enough that I was able to handle it with a regexp in the new GLOBAL_BAN_LIST (and variable enough that matching on the address wouldn't work).

Anyway, You've done good work here. Thank you. I will get the feature in the next release (and probably aim for a release soon, but not before December as I a traveling the next 3 weeks)

review: Approve

[Mark Sapiro (msapiro)]

I have merged this branch and made some changes. The most significant is your code to drop old times from the recentMemberPostings dictionary is wrong. Deleting items from a list while processing 'for item in list' results in skipping items.

I also allow for different lists to have different member_verbosity_interval settings and process that I hope reasonably.

Finally, I don't attempt to moderate a non-member, and I look at more than the From: of the message to find a member address.

[Jim Popovitch (jimpop)]

Sounds good Mark, Thanks

=== modified file 'Mailman/Defaults.py.in'
--- Mailman/Defaults.py.in	2015-09-19 02:18:34 +0000
+++ Mailman/Defaults.py.in	2015-11-05 00:11:03 +0000
@@ -1118,6 +1118,12 @@
 # The total time to spend trying to get an answer to the question.
 DMARC_RESOLVER_LIFETIME = seconds(5)
 
+# Should the list server auto-moderate members who post too frequently
+# DEFAULT_MEMBER_VERBOSITY_INTERVAL = number of seconds to track posts
+# DEFAULT_MEMBER_VERBOSITY_THRESHOLD = number of allowed posts per interval (0 to disable).
+DEFAULT_MEMBER_VERBOSITY_INTERVAL = 300
+DEFAULT_MEMBER_VERBOSITY_THRESHOLD = 0
+
 # What domains should be considered equivalent when testing list membership
 # for posting/moderation.
 # If two poster addresses with the same local part but

=== modified file 'Mailman/Gui/Privacy.py'
--- Mailman/Gui/Privacy.py	2015-02-04 04:48:25 +0000
+++ Mailman/Gui/Privacy.py	2015-11-05 00:11:03 +0000
@@ -229,6 +229,18 @@
              <a href="%(adminurl)s/members">membership management
              screens</a>.""")),
 
+            ('member_verbosity_threshold', mm_cfg.Number, 5, 0,
+             _('Ceiling on acceptable number of member posts, per interval, before automatic moderation.'),
+
+             _('''If a member posts this many times, within sender_verbosity_interval,
+             the member is automatically moderated.  Use 0 to disable.''')),
+
+            ('member_verbosity_interval', mm_cfg.Number, 5, 300,
+             _('Number of seconds to use in determining whether or not to automatically moderate a member.'),
+
+             _('''If a member posts exceed member_verbosity_threshold, within sender_verbosity_interval,
+             the member is automatically moderated.''')),
+
             ('member_moderation_action', mm_cfg.Radio,
              (_('Hold'), _('Reject'), _('Discard')), 0,
              _("""Action to take when a moderated member posts to the

=== modified file 'Mailman/Handlers/SpamDetect.py'
--- Mailman/Handlers/SpamDetect.py	2015-03-02 21:50:24 +0000
+++ Mailman/Handlers/SpamDetect.py	2015-11-05 00:11:03 +0000
@@ -122,6 +122,12 @@
                     raise Errors.RejectMessage, text
                 elif mlist.dmarc_moderation_action == 4:
                     raise Errors.DiscardMessage
+
+        if mlist.member_verbosity_threshold > 0 and Utils.IsVerboseMember(mlist, addr):
+             mlist.setMemberOption(addr, mm_cfg.Moderate, 1)
+             syslog('vette', '%s: Automatically Moderated %s for verbose postings.',
+                   mlist.real_name, addr) 
+
     if msgdata.get('approved'):
         return
     # First do site hard coded header spam checks
@@ -169,3 +175,5 @@
                     hold_for_approval(mlist, msg, msgdata, HeaderMatchHold)
                 if action == mm_cfg.ACCEPT:
                     return
+
+

=== modified file 'Mailman/Utils.py'
--- Mailman/Utils.py	2015-09-17 00:39:34 +0000
+++ Mailman/Utils.py	2015-11-05 00:11:03 +0000
@@ -1246,6 +1246,25 @@
     return False
 
 
+# Check a known list in order to auto-moderate verbose members
+recentMemberPostings = {};
+def IsVerboseMember(mlist, email):
+
+    if mlist.member_verbosity_threshold == 0:
+        return False
+
+    email = email.lower()
+
+    t = time.time()
+    recentMemberPostings.setdefault(email,[]).append(t)
+
+    for t in recentMemberPostings[email]:
+        if t < time.time() - float(mlist.member_verbosity_interval):
+            recentMemberPostings[email].remove(t)
+
+    return len(recentMemberPostings[email]) >= mlist.member_verbosity_threshold
+
+
 def check_eq_domains(email, domains_list):
     """The arguments are an email address and a string representing a
     list of lists in a form like 'a,b,c;1,2' representing [['a', 'b',

=== modified file 'Mailman/Version.py'
--- Mailman/Version.py	2015-03-31 17:18:39 +0000
+++ Mailman/Version.py	2015-11-05 00:11:03 +0000
@@ -37,7 +37,7 @@
                (REL_LEVEL << 4)  | (REL_SERIAL << 0))
 
 # config.pck schema version number
-DATA_FILE_VERSION = 108
+DATA_FILE_VERSION = 109
 
 # qfile/*.db schema version number
 QFILE_SCHEMA_VERSION = 3

=== modified file 'Mailman/versions.py'
--- Mailman/versions.py	2015-02-08 07:54:26 +0000
+++ Mailman/versions.py	2015-11-05 00:11:03 +0000
@@ -497,6 +497,10 @@
     add_only_if_missing('dmarc_moderation_notice', '')
     add_only_if_missing('dmarc_wrapped_message_text',
                        mm_cfg.DEFAULT_DMARC_WRAPPED_MESSAGE_TEXT)
+    add_only_if_missing('member_verbosity_threshold', 
+                       mm_cfg.DEFAULT_MEMBER_VERBOSITY_THRESHOLD)
+    add_only_if_missing('member_verbosity_interval',
+                       mm_cfg.DEFAULT_MEMBER_VERBOSITY_INTERVAL)
     add_only_if_missing('equivalent_domains', 
                        mm_cfg.DEFAULT_EQUIVALENT_DOMAINS)
     add_only_if_missing('new_member_options',