Merge lp:~kfogel/launchpad/582977-api-in-robots.txt into lp:launchpad

[Karl Fogel]

Disallow access under /api/ in robots.txt.

(Is there anything more we'd need to do -- i.e., does lib/canonical/launchpad/templates/robotstxt.pt get automatically propagated out to /robots.txt on the production servers?)

[Francis J. Lacoste (flacoste)]

I have no idea if the deployment of robots.txt is automatic to production. Maybe that file is only used for the local instance.

Better to ask a LOSA for confirmation. If that file is automatically deployed, we should cherry-pick this.

review: Approve (code release-critical)

[Karl Fogel (kfogel)]

Asking them now.

[Karl Fogel (kfogel)]

Okay, it turns out that filing an RT is the way to go. And we should probably remove that template, or at least add a comment noting that it is not used in production. I'll make a separate MP about that.

[Karl Fogel (kfogel)]

Sorry, meant to mention: RT 39388.

=== modified file 'lib/canonical/launchpad/templates/robotstxt.pt'
--- lib/canonical/launchpad/templates/robotstxt.pt	2009-07-17 17:59:07 +0000
+++ lib/canonical/launchpad/templates/robotstxt.pt	2010-05-25 17:02:29 +0000
@@ -6,6 +6,6 @@
 <tal:dummy define="dummy python:request.response.setHeader('content-type',
 'text/plain')" />
 User-agent: *
-Disallow:
+Disallow: /api/
 </tal:root>