Merge lp:~smoser/maas/trunk.maas-signal-clockskew into lp:~maas-committers/maas/trunk

[Scott Moser]

make maas-signal adjust clock and retry on 401 or 403 errors.

oauth failures due to incorrect clock by maas return error 401.
If we see this error, then maas-signal will read the timestamp in the
response header and update future requests based on that time.

This is the same approach fix that went into cloud-init.

make maas-signal adjust clock and retry on 401 or 403 errors.

oauth failures due to incorrect clock by maas return error 401.
If we see this error, then maas-signal will read the timestamp in the
response header and update future requests based on that time.

This is the same approach fix that went into cloud-init.

[Robie Basak (racb)]

Looks good! I've tested this and it appears to work. Commissioning does not set the RTC as expected, but the install step does.

A couple of comments, although I understand the criticality of this fix and the imminent deadline so we don't necessarily need to consider this now. Go ahead and land it.

1. Instead of hardcoding 401 and 403, we should use httplib constants.

2. I'm not sure about the use of exc to carry through the exception. In the catch clause, does that exc stay in scope for after it is set, or is there a new scope that hides the outer exc? I'm not sure. For readability, it might be worth using a different name for the inner scope and explicitly setting the one defined in the outer scope. I'm not sure about the whole mechanism, actually, but I won't think about it any longer. It'll work for now.

review: Approve

[Scott Moser (smoser)]

Reguarding '2', the exc works as expected:
#!/usr/bin/python
import sys
def main():
    print "HI WORLD"
    exc = Exception("unexpected error")
    for i in (1,2,3,4):
       try:
          raise(TypeError("foobar: %d" % i))
       except TypeError as exc:
          sys.stderr.write("caught exc: %s\n" % exc)
       except Exception as exc:
          sys.stderr.write("caught EXC: %s\n" % exc)
    raise exc
main()

=== modified file 'etc/maas/commissioning-user-data'
--- etc/maas/commissioning-user-data	2012-10-08 01:51:07 +0000
+++ etc/maas/commissioning-user-data	2012-10-09 19:01:25 +0000
@@ -305,6 +305,7 @@
 add_bin "maas-signal" <<"END_MAAS_SIGNAL"
 #!/usr/bin/python
 
+from email.utils import parsedate
 import mimetypes
 import oauth.oauth as oauth
 import os.path
@@ -369,13 +370,17 @@
     return body, headers
 
 
-def oauth_headers(url, consumer_key, token_key, token_secret, consumer_secret):
+def oauth_headers(url, consumer_key, token_key, token_secret, consumer_secret,
+                  clockskew=0):
     consumer = oauth.OAuthConsumer(consumer_key, consumer_secret)
     token = oauth.OAuthToken(token_key, token_secret)
+
+    timestamp = int(time.time()) + clockskew
+
     params = {
         'oauth_version': "1.0",
         'oauth_nonce': oauth.generate_nonce(),
-        'oauth_timestamp': int(time.time()),
+        'oauth_timestamp': timestamp,
         'oauth_token': token.key,
         'oauth_consumer_key': consumer.key,
     }
@@ -393,13 +398,43 @@
     else:
         headers = dict(headers)
 
-    if creds.get('consumer_key', None) != None:
-        headers.update(oauth_headers(url,
-            consumer_key=creds['consumer_key'], token_key=creds['token_key'],
-            token_secret=creds['token_secret'],
-            consumer_secret=creds['consumer_secret']))
-    req = urllib2.Request(url=url, data=data, headers=headers)
-    return(urllib2.urlopen(req).read())
+    clockskew = 0
+
+    def warn(msg):
+        sys.stderr.write(msg + "\n")
+
+    exc = Exception("Unexpected Error")
+    for naptime in (1, 1, 2, 4, 8, 16, 32):
+        if creds.get('consumer_key', None) != None:
+            headers.update(oauth_headers(url,
+                consumer_key=creds['consumer_key'],
+                token_key=creds['token_key'],
+                token_secret=creds['token_secret'],
+                consumer_secret=creds['consumer_secret'],
+                clockskew=clockskew))
+        try:
+            req = urllib2.Request(url=url, data=data, headers=headers)
+            return(urllib2.urlopen(req).read())
+        except urllib2.HTTPError as exc:
+            if 'date' not in exc.headers:
+                warn("date field not in %d headers" % exc.code)
+                pass
+            elif (exc.code == 401 or exc.code == 403):
+                date = exc.headers['date']
+                try:
+                    ret_time = time.mktime(parsedate(date))
+                    clockskew = int(ret_time - time.time())
+                    warn("updated clock skew to %d" % clockskew)
+                except:
+                    warn("failed to convert date '%s'" % date)
+        except Exception as exc:
+            pass
+
+        warn("request to %s failed. sleeping %d.: %s" % (url, naptime, exc))
+        time.sleep(naptime)
+
+    raise exc
+
 
 def read_config(url, creds):
     if url.startswith("http://") or url.startswith("https://"):