Merge lp:~widelands-dev/widelands-website/anti_spam into lp:widelands-website

[kaputtnik]

Hide all posts/topics which are potentially spam using a keyword filter.

Hide all posts/topics which are potentially spam using a keyword filter.

Add a boolean 'hided' field to pybb.Post.

Post filter: Applies if a post contains 'vashikaran' AND 'baba'. This catches each case insentive occurrence of this strings, so 'VaShIkaRan' is caught as well as 'babaco' or 'blabababla'. Because 'Baba' is also known as 'Bye-bye' these keywords must occur both.

Topic name filter: Applies if ' baba ' (as a word) OR 'ji' is used. Also case insensitive. Should catch ' baBaJi' as 'baBa ji' or ' baba '

The filters are set in pybb/views.py

When commiting on the server, ./manage.py migrate must be run before the website is restartet. This adds the 'hided' field to all columns in the database with default value 'False'.

[kaputtnik (franku)]

Hm, the diff is truncated... maybe remove the akismet api?

[SirVer (sirver)]

That Is Fine. Well review offline then. I hope I get around to it tomorrow. Maybe it will be Saturday.

> Am 06.10.2016 um 20:02 schrieb kaputtnik <email address hidden>:
>
> Hm, the diff is truncated... maybe remove the akismet api?
> --
> https://code.launchpad.net/~widelands-dev/widelands-website/anti_spam/+merge/307869
> You are subscribed to branch lp:widelands-website.

[kaputtnik (franku)]

Just a thought of last night:
Change the Text of the redirect page saying that EACH Post is hided now and will be moderated.

So if the spammers ever read this, they should assume that spamming has no chance here. If a non spammer read this (i guess this would be very rare, if it ever comes up) we could explain him the text.

And of course, it would be better to have the lists of keywords in local_settings.py.

I am very busy these days with normal work, so on Saturday i am working the whole day and in the evening my son is visiting me.

[GunChleoc (gunchleoc)]

Just a quick note: past tense of "hide" is "hid" ;)

[lp:~widelands-dev/widelands-website/anti_spam]

429. By kaputtnik on 2016-10-07

changed text of redirect page

430. By GunChleoc on 2016-10-08

Proofreading.

[GunChleoc (gunchleoc)]

I have decided to rename hided -> hidden right now, because it affects the models.

Not tested but code LGTM as far as I can tell.

I'd say do a final text to make sure that I didn't mess anything up and then go live.

review: Approve

[lp:~widelands-dev/widelands-website/anti_spam]

431. By SirVer on 2016-10-08

Minor nits.

[SirVer (sirver)]

1) Is it necessary that we ship the askimet client in our repo, i.e. can it not be installed wia pip_requirements.txt? We definitively should remove the .svn directory if possible. Also, for this branch it is completely unused, so I actually now agree with #1 and suggest removing the API for now.

2) I was also changing hided -> hidden, at the very same time than Gun - but she beat me to submitting :).

3) You should also check in the title of the post, not only the body. We had some spam that only had data in the title. You can do that cheaply by just concatenating: text = title + body and then working over text.

4) It might make sense to add a regular expression checking for international phone numbers - I do not think such a post can ever be legit.

5) You are marking any post of a user with 'ji' in its name as SPAM. Is that not a bit too broad?

6) There is no admin notification right now - so we will not know if posts are in the queue for moderation. I think that is fine for now, but we will need to remember to check daily once this is deployed until we have that. I am all for deploying this soon though, this SPAM is annoying.

review: Approve

[lp:~widelands-dev/widelands-website/anti_spam]

432. By kaputtnik on 2016-10-08

removed akismet api

[kaputtnik (franku)]

Just short answers because i am in hurry:

> 1) Is it necessary that we ship the askimet client in our repo, i.e. can it
> not be installed wia pip_requirements.txt?

I don't why i am downloaded the complete zip file, whereas one could also download the py-file (containing all classes and functions) I have removed the complete zip file now in this branch. Regarding to pip_requirements: I actually don't know, sorry. But a single file shouldn't be a problem for our repo.

> 3) You should also check in the title of the post, not only the body. We had
> some spam that only had data in the title. You can do that cheaply by just
> concatenating: text = title + body and then working over text.

This is already in there. See the description of this merge proposal "Topic name filter: "

> 4) It might make sense to add a regular expression checking for international
> phone numbers - I do not think such a post can ever be legit.

Yes, but i am not a regular expression freak :-D Will take some time and tests for me to do something like that.

> 5) You are marking any post of a user with 'ji' in its name as SPAM. Is that
> not a bit too broad?

Yes, maybe change it into ' ji ' or ' ji'

> 6) There is no admin notification right now - so we will not know if posts are
> in the queue for moderation. I think that is fine for now, but we will need to
> remember to check daily once this is deployed until we have that. I am all for
> deploying this soon though, this SPAM is annoying.

Yes, if we add admin notification, we should also prevent notification of normal users (who has 'Inform me on new topics" activated).

Thanks for the changes :-)

[kaputtnik (franku)]

There is a akismet Pypi package, so we could easily add this to pip_requirements.txt.

> > 5) You are marking any post of a user with 'ji' in its name as SPAM. Is that
> > not a bit too broad?
>
> Yes, maybe change it into ' ji ' or ' ji'

Correcting: 'ji' is only used for topic names (subject). And i think it is better to have a difference between topic name and post text.

Looking at my gathered spam posts, we should also add ' molvi' to the keywords. Both in topic name and post text.

I will make the following changes tomorrow:
- Add ' molvi' to keywords
- Using of keyword lists in local_settings.py (instead of hardcoded in pybb/views.py)
- merge into trunk and deploy

For me this is quite an interesting problem and i would be interested to make an own 'anti_spam' app of it (at least because there is a lot to learn). But since akismet makes his/here job, for widelands it would be better to use this service, because it is maybe more effective in a shorter time. But if we use this we may need additional views and buttons to submit ham or spam to akismet.

[lp:~widelands-dev/widelands-website/anti_spam]

433. By kaputtnik on 2016-10-09

moved list of keywords to local_settings.py

[kaputtnik (franku)]

Merged and deployed.

=== modified file 'pybb/admin.py'
--- pybb/admin.py	2012-03-18 21:06:49 +0000
+++ pybb/admin.py	2016-10-09 11:17:41 +0000
@@ -39,7 +39,7 @@
          ),
         (_('Additional options'), {
                 'classes': ('collapse',),
-                'fields': (('views',), ('sticky', 'closed'), 'subscribers')
+                'fields': (('views',), ('sticky', 'closed', 'hidden'), 'subscribers')
                 }
          ),
         )
@@ -57,7 +57,7 @@
          ),
         (_('Additional options'), {
                 'classes': ('collapse',),
-                'fields' : (('created', 'updated'), 'user_ip')
+                'fields' : (('created', 'updated'), 'user_ip', 'hidden')
                 }
          ),
         (_('Message'), {

=== modified file 'pybb/forms.py'
--- pybb/forms.py	2016-06-22 21:02:53 +0000
+++ pybb/forms.py	2016-10-09 11:17:41 +0000
@@ -66,7 +66,6 @@
         post = Post(topic=topic, user=self.user, user_ip=self.ip,
                     markup=self.cleaned_data['markup'],
                     body=self.cleaned_data['body'])
-
         post.save(*args, **kwargs)
 
         if pybb_settings.ATTACHMENT_ENABLE:

=== added file 'pybb/migrations/0002_auto_20161001_2046.py'
--- pybb/migrations/0002_auto_20161001_2046.py	1970-01-01 00:00:00 +0000
+++ pybb/migrations/0002_auto_20161001_2046.py	2016-10-09 11:17:41 +0000
@@ -0,0 +1,19 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import models, migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('pybb', '0001_initial'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='post',
+            name='hidden',
+            field=models.BooleanField(default=False, verbose_name='Hidden'),
+        ),
+    ]

=== modified file 'pybb/models.py'
--- pybb/models.py	2016-06-22 21:02:53 +0000
+++ pybb/models.py	2016-10-09 11:17:41 +0000
@@ -95,6 +95,13 @@
         except IndexError:
             return None
 
+    @property
+    def last_nonhidden_post(self):
+        posts = self.posts.order_by('-created').filter(hidden=False).select_related()
+        try:
+            return posts[0]
+        except IndexError:
+            return None
 
 class Topic(models.Model):
     forum = models.ForeignKey(Forum, related_name='topics', verbose_name=_('Forum'))
@@ -132,6 +139,22 @@
         return self.posts.all().order_by('-created').select_related()[0]
 
     @property
+    def last_nonhidden_post(self):
+        try:
+            return self.posts.all().order_by('-created').filter(hidden=False).select_related()[0]
+        except IndexError:
+            return self.posts.all().order_by('-created').select_related()[0]
+
+    # If the first post of this topic is hidden, the topic is hidden
+    @property
+    def is_hidden(self):
+        try:
+            p = self.posts.all().order_by('created').filter(hidden=False).select_related()[0]
+        except IndexError:
+            return True
+        return False
+
+    @property
     def post_count(self):
         return Post.objects.filter(topic=self).count()
 
@@ -193,6 +216,7 @@
     body_html = models.TextField(_('HTML version'))
     body_text = models.TextField(_('Text version'))
     user_ip = models.GenericIPAddressField(_('User IP'), default='')
+    hidden = models.BooleanField(_('Hidden'), blank=True, default=False)
 
     # Django sphinx
     if settings.USE_SPHINX:

=== modified file 'pybb/urls.py'
--- pybb/urls.py	2016-06-04 14:17:40 +0000
+++ pybb/urls.py	2016-10-09 11:17:41 +0000
@@ -30,7 +30,8 @@
     url('^post/(?P<post_id>\d+)/$', views.show_post, name='pybb_post'),
     url('^post/(?P<post_id>\d+)/edit/$', views.edit_post, name='pybb_edit_post'),
     url('^post/(?P<post_id>\d+)/delete/$', views.delete_post, name='pybb_delete_post'),
-
+    url('pybb_moderate_info/$', views.pybb_moderate_info),
+    
     # Attachment
     url('^attachment/(?P<hash>\w+)/$', views.show_attachment, name='pybb_attachment'),
 

=== modified file 'pybb/views.py'
--- pybb/views.py	2016-06-15 19:20:24 +0000
+++ pybb/views.py	2016-10-09 11:17:41 +0000
@@ -1,6 +1,6 @@
 import math
 from mainpage.templatetags.wl_markdown import do_wl_markdown
-from pybb.markups import mypostmarkup 
+from pybb.markups import mypostmarkup
 
 from django.shortcuts import get_object_or_404
 from django.http import HttpResponseRedirect, HttpResponse, HttpResponseNotFound, Http404
@@ -10,13 +10,15 @@
 from django.core.urlresolvers import reverse
 from django.db import connection
 from django.utils import translation
+from django.shortcuts import render
 
 from pybb.util import render_to, paged, build_form, quote_text, paginate, set_language, ajax, urlize
 from pybb.models import Category, Forum, Topic, Post, PrivateMessage, Attachment,\
                         MARKUP_CHOICES
-from pybb.forms import AddPostForm, EditPostForm, UserSearchForm 
+from pybb.forms import AddPostForm, EditPostForm, UserSearchForm
 from pybb import settings as pybb_settings
 from pybb.orm import load_related
+from django.conf import settings
 
 try:
     from notification import models as notification
@@ -74,7 +76,7 @@
             }
 show_forum = render_to('pybb/forum.html')(show_forum_ctx)
 
-    
+
 def show_topic_ctx(request, topic_id):
 
     try:
@@ -112,7 +114,7 @@
     # profiles = Profile.objects.filter(user__pk__in=
     #     set(x.user.id for x in page.object_list))
     # profiles = dict((x.user_id, x) for x in profiles)
-    
+
     # for post in page.object_list:
     #     post.user.pybb_profile = profiles[post.user.id]
 
@@ -159,7 +161,29 @@
                       initial={'markup': "markdown", 'body': quote})
 
     if form.is_valid():
-        post = form.save();
+        # TODO: Add akismet check here
+        spam = False
+
+        # Check in post text.
+        text = form.cleaned_data['body']
+        if any(x in text.lower() for x in settings.ANTI_SPAM_BODY):
+            spam = True
+        
+        # Check in topic subject ('name' is empty if this a post to an existing topic)
+        text = form.cleaned_data['name']
+        if text != '':
+            # This is a new topic
+            if any(x in text.lower() for x in settings.ANTI_SPAM_TOPIC):
+                spam = True
+
+        post = form.save()
+        if spam:
+            # Hide the post against normal users
+            post.hidden = True
+            post.save()
+            # Redirect to an info page to inform the user
+            return HttpResponseRedirect('pybb_moderate_info')
+
         if not topic:
             post.topic.subscribers.add(request.user)
         return HttpResponseRedirect(post.get_absolute_url())
@@ -353,3 +377,6 @@
 
     html = urlize(html)
     return {'content': html}
+
+def pybb_moderate_info(request):
+    return render(request, 'pybb/pybb_moderate_info.html')

=== modified file 'templates/pybb/forum.html'
--- templates/pybb/forum.html	2016-04-29 07:28:16 +0000
+++ templates/pybb/forum.html	2016-10-09 11:17:41 +0000
@@ -40,7 +40,8 @@
 			</tr>
 		</thead>
 		<tbody>
-	{% for topic in topics %}
+		{% for topic in topics %}
+		{% if not topic.is_hidden %}
 		<tr class="{% cycle 'odd' 'even' %}">
 			<td class="forumIcon center">
 				{% if topic|pybb_has_unreads:user %}
@@ -60,13 +61,43 @@
 				Views: {{ topic.views }}
 			</td>
 			<td class="lastPost">
-				{%if topic.last_post %}
+				{% if user.is_superuser %}
+					{% if topic.last_post %}
+						{{ topic.last_post.user|user_link }} <a href="{{ topic.last_post.get_absolute_url }}">&#187;</a><br />
+						<span class="small">on {{ topic.last_post.created|custom_date:user }}</span>
+					{% endif %}
+				{% else %}
+					{{ topic.last_nonhidden_post.user|user_link }} <a href="{{ topic.last_nonhidden_post.get_absolute_url }}">&#187;</a><br />
+					<span class="small">on {{ topic.last_nonhidden_post.created|custom_date:user }}</span>
+				{% endif %}
+			</td>
+		</tr>
+		{% elif user.is_superuser %}
+		<tr class="{% cycle 'odd' 'even' %}">
+			<td class="forumIcon center">
+				{% if topic|pybb_has_unreads:user %}
+				<img src="{{ MEDIA_URL }}forum/img/doc_big_work_star.png" style="margin: 0px;" alt="" class="middle" />
+				{% else %}
+					<img src="{{ MEDIA_URL }}forum/img/doc_big_work.png" style="margin: 0px;" alt="" class="middle" />
+				{% endif %}
+			</td>
+			<td class="forumTitle">
+				{% if topic.sticky %}<img src="{{ MEDIA_URL }}forum/img/sticky.png" alt="Sticky" title="Sticky" />{% endif %}
+				{% if topic.closed %}<img src="{{ MEDIA_URL }}forum/img/closed.png" alt="Closed" title="Closed" />{% endif %}
+				<a href="{{ topic.get_absolute_url }}">{{ topic.name }}</a><br />
+				<span class="small">Created by {{ topic.user|user_link }} on {{ topic.created|custom_date:user }}</span>
+			</td>
+			<td class="forumCount center small" style="width: 120px;">
+				Posts: {{ topic.post_count }}<br/>
+				Views: {{ topic.views }}
+			</td>
+			<td class="lastPost">
 				{{ topic.last_post.user|user_link }} <a href="{{ topic.last_post.get_absolute_url }}">&#187;</a><br />
 				<span class="small">on {{ topic.last_post.created|custom_date:user }}</span>
-				{% endif %}
 			</td>
 		</tr>
-	{% endfor %}
+		{% endif %} {# topic.is_hidden #}
+		{% endfor %} {# topic #}
 		</tbody>
 	</table>
 

=== modified file 'templates/pybb/inlines/display_category.html'
--- templates/pybb/inlines/display_category.html	2016-03-02 21:02:38 +0000
+++ templates/pybb/inlines/display_category.html	2016-10-09 11:17:41 +0000
@@ -29,13 +29,22 @@
 			Topics: {{ forum.topics.count }}<br/>
 			Posts: {{ forum.posts.count }}
 		</td>
-		<td class="lastPost">
-		{%if forum.last_post %}
-			<a href="{{forum.last_post.get_absolute_url}}">{{ forum.last_post.topic.name }}</a><br />
-			<span class="small">by {{ forum.last_post.user|user_link }}<br />
-			on {{ forum.last_post.created|custom_date:user}}</span>
-		{% else %}
-			&nbsp;
+		{% if user.is_superuser %} {# Show all to superuser #}
+			{% if forum.last_post %}
+			<td class="lastPost">
+				<a href="{{forum.last_post.get_absolute_url}}">{{ forum.last_post.topic.name }}</a><br />
+				<span class="small">by {{ forum.last_post.user|user_link }}<br />
+				on {{ forum.last_post.created|custom_date:user}}</span>
+			</td>
+			{% endif %}
+		{% else %} {# no super_user: Show only nonhidden posts#}
+			{% if forum.last_nonhidden_post %}
+			<td class="lastPost">
+				<a href="{{forum.last_nonhidden_post.get_absolute_url}}">{{ forum.last_nonhidden_post.topic.name }}</a><br />
+				<span class="small">by {{ forum.last_nonhidden_post.user|user_link }}<br />
+				on {{ forum.last_nonhidden_post.created|custom_date:user}}</span>
+			</td>
+			{% endif %}
 		{% endif %}
 		</td>
 	</tr>

=== modified file 'templates/pybb/last_posts.html'
--- templates/pybb/last_posts.html	2016-03-02 21:02:38 +0000
+++ templates/pybb/last_posts.html	2016-10-09 11:17:41 +0000
@@ -9,11 +9,13 @@
 	<div class="columnModuleBox">
 		<ul>
 			{% for post in posts %}
-			<li>
-				{{ post.topic.forum.name }}<br />
-				<a href="{{ post.get_absolute_url }}" title="{{ post.topic.name }}">{{ post.topic.name|pybb_cut_string:30 }}</a><br />
-				by <a href="{% url 'profile_view' post.user %}">{{post.user.username}}</a> {{ post.created|minutes }} ago
-			</li>
+				{% if not post.hidden %}
+				<li>
+					{{ post.topic.forum.name }}<br />
+					<a href="{{ post.get_absolute_url }}" title="{{ post.topic.name }}">{{ post.topic.name|pybb_cut_string:30 }}</a><br />
+					by <a href="{% url 'profile_view' post.user %}">{{post.user.username}}</a> {{ post.created|minutes }} ago
+				</li>
+				{% endif %}
 			{% endfor %}
 		</ul>
 	</div>

=== added file 'templates/pybb/pybb_moderate_info.html'
--- templates/pybb/pybb_moderate_info.html	1970-01-01 00:00:00 +0000
+++ templates/pybb/pybb_moderate_info.html	2016-10-09 11:17:41 +0000
@@ -0,0 +1,12 @@
+{% extends 'pybb/base.html' %}
+
+{% block content %}
+
+<h1>All comments have to be moderated</h1>
+
+<div class="blogEntry">
+	<p>Your comment has been saved but hidden to normal users. A moderator
+	will take a look at it and review it as soon as possible.</p>
+</div>
+	
+{% endblock %}

=== modified file 'templates/pybb/topic.html'
--- templates/pybb/topic.html	2016-07-31 08:44:48 +0000
+++ templates/pybb/topic.html	2016-10-09 11:17:41 +0000
@@ -157,6 +157,7 @@
 	<table class="forum">
 		<tbody>
 	{% for post in posts %}
+		{% if not post.hidden or user.is_superuser %}
 		<tr class="{% cycle 'odd' 'even' %}">
 			<td class="author">
 				{{ post.user|user_link }}<br />
@@ -228,6 +229,7 @@
 				{% endif %}
 			</td>
 		</tr>
+		{% endif %}
 		<tr class="spacer">
 			<td></td>
 			<td></td>