Merge lp:~widelands-dev/widelands-website/static_robots_txt into lp:widelands-website

[kaputtnik]

Add a robots.txt.

The content is discussed with janus and should be ok.

[lp:~widelands-dev/widelands-website/static_robots_txt]

449. By kaputtnik on 2016-12-15

allow polls and screenshots

[SirVer (sirver)]

I feel really uncomfortable about the black listing, I think it should be a whitelist.

Otherwise some nits.

review: Needs Information

[lp:~widelands-dev/widelands-website/static_robots_txt]

450. By kaputtnik on 2016-12-16

reverted to r446

451. By kaputtnik on 2016-12-16

added whitespace; cleanup robots.txt

[kaputtnik (franku)]

So the first version was better than the adjusted version.

Thanks for looking into it :-)

review: Needs Resubmitting

[SirVer (sirver)]

> So the first version was better than the adjusted version.

Well - we did not know then though :). Software development is iterative in nature, you sometimes have to take the roundabout way to get back were you started - but only with the new knowledge acquired on the way you can truly judge that the new/old solution is the best.

lgtm.

review: Approve

[kaputtnik (franku)]

Thanks :-)

Merged and deployed.

=== added file 'templates/robots.txt'
--- templates/robots.txt	1970-01-01 00:00:00 +0000
+++ templates/robots.txt	2016-12-16 08:27:03 +0000
@@ -0,0 +1,10 @@
+# robots.txt for wl.widelands.org
+
+# These things should never be crawled
+User-agent: *
+Disallow: /profile
+Disallow: /admin
+Disallow: /accounts
+
+# url to sitemap
+Sitemap: https://wl.widelands.org/sitemap.xml/

=== modified file 'urls.py'
--- urls.py	2016-12-13 18:28:51 +0000
+++ urls.py	2016-12-16 08:27:03 +0000
@@ -7,6 +7,7 @@
 from mainpage.views import mainpage
 from news.feeds import NewsPostsFeed
 from django.views.generic.base import RedirectView
+from django.views.generic import TemplateView
 from django.contrib.syndication.views import Feed
 from registration.backends.hmac.views import RegistrationView
 from mainpage.forms import RegistrationWithCaptchaForm
@@ -15,6 +16,8 @@
 urlpatterns = [
     # Creating a sitemap.xml
     url(r'^sitemap\.xml/', include('sitemap_urls')),
+    # Static view of robots.txt
+    url(r'^robots\.txt/', TemplateView.as_view(template_name='robots.txt', content_type="text/plain")),
 
     # Uncomment the next line to enable the admin:
     url(r'^admin/', admin.site.urls),