Merge lp:~maxiberta/canonical-identity-provider/update-robots.txt into lp:canonical-identity-provider/release

[Maximiliano Bertacchini]

Update robots.txt.

- Limit URLs with an ending '$' where appropriate.
- Duplicate rules with an encoded '+' (currently cowboyed in prod as per https://portal.admin.canonical.com/C114092). Should not be needed but Bing seems happier with it; and it won't hurt anyway.
- Add /saml endpoint exclusion.

[Daniel Manrique (roadmr)]

This is the current robots.txt on production (cowboy):

User-agent: *
Disallow: /+bad-token
Disallow: /+deactivated
Disallow: /+logout
Disallow: /+logout-to-confirm
Disallow: /+openid
Disallow: /+saml
Disallow: /+suspended

# 2018-09-06 maxiberta cowboy (lp:1787823)
Disallow: /%2Bbad-token$
Disallow: /%2Bdeactivated$
Disallow: /%2Blogout$
Disallow: /%2Blogout-to-confirm$
Disallow: /%2Bopenid$
Disallow: /%2Bsaml$
Disallow: /%2Bsuspended$

I spot a few differences, see below, but +1'd in case I'm just misunderstanding where the $ ending is needed.

review: Approve

[Maximiliano Bertacchini (maxiberta)]

Currently in production, all rules with a '+' are open-ended (i.e. with an implicit '*' at the end); and otoh, all rules with '%2B' are all exact matches. The proposed unified rules are exact matching where possible, or open-ended if URL variations ending in /* exist. Makes sense? Thanks!

[Daniel Manrique (roadmr)]

Makes sense. Let's go then!

review: Approve

[Otto Co-Pilot (otto-copilot)]

Running landing tests failed
https://jenkins.ols.canonical.com/online-services/job/canonical-identity-provider/194/

=== modified file 'src/webui/templates/static/robots.txt'
--- src/webui/templates/static/robots.txt	2018-08-24 16:15:01 +0000
+++ src/webui/templates/static/robots.txt	2018-09-12 14:54:50 +0000
@@ -1,8 +1,17 @@
 User-agent: *
 Disallow: /+bad-token
-Disallow: /+deactivated
-Disallow: /+logout
+Disallow: /+deactivated$
+Disallow: /+logout$
 Disallow: /+logout-to-confirm
-Disallow: /+openid
+Disallow: /+openid$
 Disallow: /+saml
-Disallow: /+suspended
+Disallow: /+suspended$
+Disallow: /saml
+
+Disallow: /%2Bbad-token
+Disallow: /%2Bdeactivated$
+Disallow: /%2Blogout$
+Disallow: /%2Blogout-to-confirm
+Disallow: /%2Bopenid$
+Disallow: /%2Bsaml
+Disallow: /%2Bsuspended$