Merge ~pwlars/testflinger-cli:dont-let-poll-die into testflinger-cli:master

[Paul Larson]

I noticed that in one of the test runs today, we saw a crash while testflinger-cli was polling [1]. The job continues to run in the background, but because jenkins loses it's monitor of the status, it tries to continue and fails. So even though the test might run, we don't get the email at the end with the summary and all that. This should make testflinger-cli retry if it hits spurious errors like this, rather than just dying.

[Paul Larson (pwlars)]

[1]
Timeout while trying to communicate with the server.
Traceback (most recent call last):
  File "/srv/mnt/jenkins/jobs/stlouis-stlouis-kernel-edge/workspace/testflinger-cli/env/lib/python3.5/site-packages/urllib3-1.21.1-py3.5.egg/urllib3/connectionpool.py", line 386, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/srv/mnt/jenkins/jobs/stlouis-stlouis-kernel-edge/workspace/testflinger-cli/env/lib/python3.5/site-packages/urllib3-1.21.1-py3.5.egg/urllib3/connectionpool.py", line 382, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.5/http/client.py", line 1197, in getresponse
    response.begin()
  File "/usr/lib/python3.5/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.5/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/mnt/jenkins/jobs/stlouis-stlouis-kernel-edge/workspace/testflinger-cli/env/lib/python3.5/site-packages/requests-2.18.1-py3.5.egg/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/srv/mnt/jenkins/jobs/stlouis-stlouis-kernel-edge/workspace/testflinger-cli/env/lib/python3.5/site-packages/urllib3-1.21.1-py3.5.egg/urllib3/connectionpool.py", line 649, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/srv/mnt/jenkins/jobs/stlouis-stlouis-kernel-edge/workspace/testflinger-cli/env/lib/python3.5/site-packages/urllib3-1.21.1-py3.5.egg/urllib3/util/retry.py", line 357, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/srv/mnt/jenkins/jobs/stlouis-stlouis-kernel-edge/workspace/testflinger-cli/env/lib/python3.5/site-packages/urllib3-1.21.1-py3.5.egg/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/srv/mnt/jenkins/jobs/stlouis-stlouis-kernel-edge/workspace/testflinger-cli/env/lib/python3.5/site-packages/urllib3-1.21.1-py3.5.egg/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/srv/mnt/jenkins/jobs/stlouis-stlouis-kernel-edge/workspace/testflinger-cli/env/lib/python3.5/site-packages/urllib3-1.21.1-py3.5.egg/urllib3/connectionpool.py", line 388, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/srv/mnt/jenkins/jobs/stlouis-stlouis-kernel-edge/workspace/testflinger-cli/env/lib/python3.5/site-packages/urllib3-1.21.1-py3.5.egg/urllib3/connectionpool.py", line 308, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='testflinger.canonical.com', port=80): Read timed out. (read timeout=15)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/mnt/jenkins/jobs/stlouis-stlouis-kernel-edge/workspace/testflinger-cli/env/bin/testflinger-cli", line 4, in <module>
    __import__('pkg_resources').run_...

[Paul Larson (pwlars)]

I've tried this locally from a virtualenv, and seems to be working or at least not causing problems. I'd like to land it in trunk at least, so that we can safely kick off some runs and have better hope for not hitting that problem again

review: Approve

diff --git a/testflinger-cli b/testflinger-cli
index d51b4f2..4625ac5 100755
--- a/testflinger-cli
+++ b/testflinger-cli
@@ -194,7 +194,12 @@ def poll(ctx, job_id):
             continue
         if output:
             print(output, end='', flush=True)
-        job_state = conn.get_status(job_id)
+        try:
+            job_state = conn.get_status(job_id)
+        except:
+            # If something breaks here, just retry so we don't affect
+            # a running test monitor that relies on poll
+            continue
         if job_state == 'complete':
             break
         time.sleep(10)