Merge lp:~ralfjung-e/mailman/2.1 into lp:mailman/2.1

[Ralf Jung]

Implement a simple CAPTCHA scheme based on questions and answers configured by the site admin.

Fixes https://bugs.launchpad.net/mailman/+bug/1774826.

I have manually edited listinfo.html for a few languages for testing; is there a way to automate that or do I have to manually do that for all languages?

[Jim Popovitch (jimpop)]

I like this solution, it's simple and easily customizable, doesn't rely on 3rd-party integrations, and the CAPTCHA questions can be easily updated. Nice work Ralf.

One thing that I would like to see in the future is storing the CAPTCHAs in pck or even sql, etc.

[Mark Sapiro (msapiro)]

Thanks for this. I am currently traveling and I will review it in detail as I have time.

[Jim Popovitch (jimpop)]

If line 20 (in the patch below) sets "captcha_idx = 0" python throws an error:

   TypeError: cannot concatenate 'str' and 'int' objects

I think a better solution would be to remove the captcha_idx from that section of code if it is None, (and set it to None at line 20).

[lp:~ralfjung-e/mailman/2.1]

1816. By Ralf Jung <email address hidden> on 2019-06-10

fix computing the form hash when there is no CAPTCHA

[Ralf Jung (ralfjung-e)]

Ah good catch, I forgot to test this without CAPTCHAS set.

However, the variable is used later to compute the hash for the form secret. Removing it conditionally will mean we have to mirror the same logic on the other side in subscribe.py. So for now I changed it to use the empty string instead. I don't know if this will automatically get included in this MR?

[lp:~ralfjung-e/mailman/2.1]

1817. By Ralf Jung <email address hidden> on 2019-06-10

Mention in the docs that 'en' is used as the default key

1818. By Ralf Jung <email address hidden> on 2019-06-10

Don't enable CAPTCHA if 'en' key is not set

[Jim Popovitch (jimpop)]

The recent change will be included in the MR, you can confirm this by looking at the diff down below these comments on webform for the MR.

[Mark Sapiro (msapiro)]

Thanks very much Ralf. I made a few minor style changes and added a NEWS item, but this is committed at https://bazaar.launchpad.net/~mailman-coders/mailman/2.1/revision/1815

review: Approve

[Ralf Jung (ralfjung-e)]

Great. :) I see you added <mm-captcha-ui> to all the templates, thanks!

Oh and could you please correct my name in NEWS? It's Ralf with an "f". :)

[Mark Sapiro (msapiro)]

Sorry about the misspelling of your name (I did get it right in the commit message). Fixed now.

=== modified file 'Mailman/Cgi/listinfo.py'
--- Mailman/Cgi/listinfo.py	2018-06-21 16:23:09 +0000
+++ Mailman/Cgi/listinfo.py	2019-06-10 20:07:09 +0000
@@ -216,10 +216,25 @@
             #        drop one : resulting in an invalid format, but it's only
             #        for our hash so it doesn't matter.
             remote = remote.rsplit(':', 1)[0]
+        # render CAPTCHA, if configured
+        if isinstance(mm_cfg.CAPTCHAS, dict) and 'en' in mm_cfg.CAPTCHAS:
+            (captcha_question, captcha_box, captcha_idx) = \
+                Utils.captcha_display(mlist, lang, mm_cfg.CAPTCHAS)
+            pre_question = _(
+                    '''Please answer the following question to prove that
+                    you are not a bot:'''
+                )
+            replacements['<mm-captcha-ui>'] = (
+                """<tr><td BGCOLOR="#dddddd">%s<br>%s</td><td>%s</td></tr>"""
+                % (pre_question, captcha_question, captcha_box))
+        else:
+            captcha_idx = "" # just to have something to include in the hash below
+        # fill form
         replacements['<mm-subscribe-form-start>'] += (
-                '<input type="hidden" name="sub_form_token" value="%s:%s">\n'
-                % (now, Utils.sha_new(mm_cfg.SUBSCRIBE_FORM_SECRET + ":" +
+                '<input type="hidden" name="sub_form_token" value="%s:%s:%s">\n'
+                % (now, captcha_idx, Utils.sha_new(mm_cfg.SUBSCRIBE_FORM_SECRET + ":" +
                           now + ":" +
+                          captcha_idx + ":" +
                           mlist.internal_name() + ":" +
                           remote
                           ).hexdigest()

=== modified file 'Mailman/Cgi/subscribe.py'
--- Mailman/Cgi/subscribe.py	2018-06-17 23:47:34 +0000
+++ Mailman/Cgi/subscribe.py	2019-06-10 20:07:09 +0000
@@ -168,13 +168,14 @@
             #        for our hash so it doesn't matter.
             remote1 = remote.rsplit(':', 1)[0]
         try:
-            ftime, fhash = cgidata.getfirst('sub_form_token', '').split(':')
+            ftime, fcaptcha_idx, fhash = cgidata.getfirst('sub_form_token', '').split(':')
             then = int(ftime)
         except ValueError:
-            ftime = fhash = ''
+            ftime = fcaptcha_idx = fhash = ''
             then = 0
         token = Utils.sha_new(mm_cfg.SUBSCRIBE_FORM_SECRET + ":" +
                               ftime + ":" +
+                              fcaptcha_idx + ":" +
                               mlist.internal_name() + ":" +
                               remote1).hexdigest()
         if ftime and now - then > mm_cfg.FORM_LIFETIME:
@@ -189,6 +190,11 @@
             results.append(
     _('There was no hidden token in your submission or it was corrupted.'))
             results.append(_('You must GET the form before submitting it.'))
+        # Check captcha
+        if isinstance(mm_cfg.CAPTCHAS, dict):
+            captcha_answer = cgidata.getvalue('captcha_answer', '')
+            if not Utils.captcha_verify(fcaptcha_idx, captcha_answer, mm_cfg.CAPTCHAS):
+                results.append(_('This was not the right answer to the CAPTCHA question.'))
     # Was an attempt made to subscribe the list to itself?
     if email == mlist.GetListEmail():
         syslog('mischief', 'Attempt to self subscribe %s: %s', email, remote)

=== modified file 'Mailman/Defaults.py.in'
--- Mailman/Defaults.py.in	2019-05-22 00:10:40 +0000
+++ Mailman/Defaults.py.in	2019-06-10 20:07:09 +0000
@@ -131,6 +131,25 @@
 # test.
 SUBSCRIBE_FORM_MIN_TIME = seconds(5)
 
+# Use a custom question-answer CAPTCHA to protect against subscription spam.
+# Has no effect unless SUBSCRIBE_FORM_SECRET is set.
+# Should be set to a dict mapping language keys to a list of pairs
+# of questions and regexes for the answers, e.g.
+# CAPTCHAS = {
+#   'en': [
+#     ('What is two times six?', '(12|twelve)'),
+#     ('What is this mailing list software called?', '[Mm]ailman'),
+#   ],
+#   'de': [
+#     ('Was ist 3 mal 6?', '(18|achtzehn)'),
+#   ],
+# }
+# The regular expression must match the full string, i.e., it is implicitly
+# acting as if it had "^" in the beginning and "$" at the end.
+# An 'en' key must be present and is used as fall-back if there are no questions
+# for the currently set language.
+CAPTCHAS = None
+
 # Use Google reCAPTCHA to protect the subscription form from spam bots.  The
 # following must be set to a pair of keys issued by the reCAPTCHA service at
 # https://www.google.com/recaptcha/admin

=== modified file 'Mailman/Utils.py'
--- Mailman/Utils.py	2019-03-02 02:24:14 +0000
+++ Mailman/Utils.py	2019-06-10 20:07:09 +0000
@@ -1576,3 +1576,32 @@
         if not re.search(r'127\.0\.1\.255$', text, re.MULTILINE):
             return True
     return False
+
+
+def captcha_display(mlist, lang, captchas):
+    """Returns a CAPTCHA question, the HTML for the answer box, and
+    the data to be put into the CSRF token"""
+    if not lang in captchas:
+        lang = 'en'
+    captchas = captchas[lang]
+    idx = random.randrange(len(captchas))
+    question = captchas[idx][0]
+    box_html = mlist.FormatBox('captcha_answer', size=30)
+    # Remember to encode the language in the index so that we can get it out again!
+    return (websafe(question), box_html, lang + "-" + str(idx))
+
+def captcha_verify(idx, given_answer, captchas):
+    try:
+        (lang, idx) = idx.split("-")
+        idx = int(idx)
+    except ValueError:
+        return False
+    if not lang in captchas:
+        return False
+    captchas = captchas[lang]
+    if not idx in range(len(captchas)):
+        return False
+    # Check the given answer.
+    # We append a `$` to emulate `re.fullmatch`.
+    correct_answer_pattern = captchas[idx][1] + "$"
+    return re.match(correct_answer_pattern, given_answer)

=== modified file 'templates/de/listinfo.html'
--- templates/de/listinfo.html	2018-01-29 12:58:42 +0000
+++ templates/de/listinfo.html	2019-06-10 20:07:09 +0000
@@ -115,6 +115,7 @@
       </tr>
       <mm-digest-question-end>
       <mm-recaptcha-ui>
+      <mm-captcha-ui>
       <tr>
 	<td colspan="3">
 	  <center><MM-Subscribe-Button></center>

=== modified file 'templates/en/listinfo.html'
--- templates/en/listinfo.html	2018-01-29 12:58:42 +0000
+++ templates/en/listinfo.html	2019-06-10 20:07:09 +0000
@@ -116,6 +116,7 @@
       </tr>
       <mm-digest-question-end>
       <mm-recaptcha-ui>
+      <mm-captcha-ui>
       <tr>
 	<td colspan="3">
 	  <center><MM-Subscribe-Button></center>

=== modified file 'templates/fr/listinfo.html'
--- templates/fr/listinfo.html	2018-01-29 12:58:42 +0000
+++ templates/fr/listinfo.html	2019-06-10 20:07:09 +0000
@@ -119,6 +119,7 @@
        </tr>
 	<mm-digest-question-end>
        <mm-recaptcha-ui>
+       <mm-captcha-ui>
        <tr>
  	<td colspan="3"> 	  
               <center><MM-Subscribe-Button></center>