Merge lp:~jpds/launchpad/fix_565345-mirror-prober-agent into lp:launchpad

[Jonathan Davies]

Added "User-Agent" header to the mirror prober.

= Summary =

This branch adds a "User-Agent" header to the mirror prober so that probes can be identified easily in HTTPd logs.

[Jeroen T. Vermeulen (jtv)]

A good idea, especially since we've traditionally asked bots accessing our site to provide meaningful user agent identification. Shouldn't ours include a URL as well, by the way?

It's a shame you have to mess with sendHeader in this way, but the only alternative I see is to make FakeTransport log its header and that's probably more effort than it's worth.

A few points remain though:
 * Don't return self.assertEquals(...); just invoke it. It'll raise an exception if it fails.
 * Don't test "headers[1]"—a hard-coded index is far too fragile. Why not make "headers" a dict and test headers['User-Agent']?

Jeroen

review: Approve (code)

[Jeroen T. Vermeulen (jtv)]

review: Needs Fixing (code)

[Jeroen T. Vermeulen (jtv)]

Thanks for the changes. You're good to go!

review: Approve (code)

=== modified file 'lib/lp/registry/scripts/distributionmirror_prober.py'
--- lib/lp/registry/scripts/distributionmirror_prober.py	2009-12-23 16:54:57 +0000
+++ lib/lp/registry/scripts/distributionmirror_prober.py	2010-04-20 15:05:40 +0000
@@ -122,6 +122,8 @@
         """
         self.sendCommand('HEAD', self.factory.connect_path)
         self.sendHeader('HOST', self.factory.connect_host)
+        self.sendHeader('User-Agent',
+            'Launchpad Mirror Prober ( https://launchpad.net/ )')
         self.endHeaders()
 
     def handleStatus(self, version, status, message):
 
=== modified file 'lib/lp/registry/tests/test_distributionmirror_prober.py'
--- lib/lp/registry/tests/test_distributionmirror_prober.py	2009-12-23 16:39:52 +0000
+++ lib/lp/registry/tests/test_distributionmirror_prober.py	2010-04-20 15:05:40 +0000
@@ -12,7 +12,6 @@
 import httplib
 import logging
 import os
-import re
 from StringIO import StringIO
 import unittest
 
@@ -213,6 +212,25 @@
         return self.assertFailure(d, ProberTimeout)
 
 
+    def test_prober_user_agent(self):
+        protocol = RedirectAwareProberProtocol()
+
+        orig_sendHeader = protocol.sendHeader
+        headers = {}
+
+        def mySendHeader(header, value):
+            orig_sendHeader(header, value)
+            headers[header] = value
+
+        protocol.sendHeader = mySendHeader
+
+        protocol.factory = FakeFactory('http://foo.bar/')
+        protocol.makeConnection(FakeTransport())
+        self.assertEquals(
+            'Launchpad Mirror Prober ( https://launchpad.net/ )',
+            headers['User-Agent'])
+
+
 class FakeTimeOutCall:
     resetCalled = False
 
@@ -811,7 +829,8 @@
         logger.logMessage("Ubuntu Warty Released")
         logger.log_file.seek(0)
         message = logger.log_file.read()
-        self.failUnlessEqual('Wed Oct 20 12:00:00 2004: Ubuntu Warty Released',
+        self.failUnlessEqual(
+            'Wed Oct 20 12:00:00 2004: Ubuntu Warty Released',
             message)
 
     def test_logMessage_integration(self):