robots.txt

Bug #1836505 reported by janus
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Widelands Website
Fix Released
High
Unassigned

Bug Description

aus aktuellem anlass müsste die robots.txt angepasst werden. ein crawler schafte es, seit dem wir auf dem neuem server sind, auf 38k anfragen zumeist auf /accounts/login/... welches eigentlich ausgeschlossen ist aber nicht wirklich.

aus:

# robots.txt for wl.widelands.org

# These things should never be crawled
User-agent: *
Disallow: /profile
Disallow: /admin
Disallow: /accounts

# url to sitemap
Sitemap: https://wl.widelands.org/sitemap.xml/

müsste werden:

# robots.txt for www.widelands.org

# These things should never be crawled
User-agent: *
Disallow: /profile/
Disallow: /admin/
Disallow: /accounts/

Crawl-delay: 10

# url to sitemap
Sitemap: https://www.widelands.org/sitemap.xml/

Revision history for this message
kaputtnik (franku) wrote :
Changed in widelands-website:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.