Merge lp:~fginther/jenkins-launchpad-plugin/wait-for-node into lp:~private-ps-quality-team/jenkins-launchpad-plugin/trunk

[Francis Ginther]

Adds a script, waitForNodeIdle.py to wait for a jenkins slave node to become idle. This can be used to automate slave node maintenance tasks which require the node to be offline and idle.

Adds a script to wait for a jenkins slave node to become idle. This can be used to automate slave node maintenance tasks which require the node to be offline and idle. See ps-qa-pbuilder-clean for an example.

Testing:
 - Includes unit tests for wait_for_node_idle()
 - Tested with jenkins job ps-qa-pbuilder-clean to perform a pbuilder --clean.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:80
http://s-jenkins:8080/job/jenkins-launchpad-plugin-ci/220/
Executed test runs:
    SUCCESS: http://s-jenkins:8080/job/jenkins-launchpad-plugin-ci/./distribution=quantal,flavor=amd64/220/console

Click here to trigger a rebuild:
http://s-jenkins:8080/job/jenkins-launchpad-plugin-ci/220//rebuild/?

review: Approve (continuous-integration)

[Martin Mrazik (mrazik)]

Few things:

1. Can we make the timeout arg optional with some reasonable default value? It will make the use of this easier as I won't need to think of what "reasonable timeout" is. Not quite sure what the default should be but I would expect something like 2 hours.

2. I would prefer an "--mark-offline" option (maybe on by default) which would put the node to maintenance so no new jobs will be scheduled. You will need to do this anyway and if we do it directly in this script we will shorten the window where a race condition can occur. After the node is in maintenance I would do the check for idle again just to make sure nothing was scheduled while we were doing the lock. Once --mark-offline is there it would be nice to have --mark-online which would put the node back online again without any additional checks. That way all I need is this python script and the rest can be done in shell as a separate build step.

3. I wonder if "def wait_for_node_idle" shouldn't be in jenkinsutils.py (as well as any helpers for making the node online/offline).

review: Needs Fixing

[Martin Mrazik (mrazik)]

btw. I just tried to put a node offline while a test was running and the test continues to run. Maybe with the --mark-offline option the script can first mark the node offline and then wait for being idle.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:81
http://s-jenkins:8080/job/jenkins-launchpad-plugin-ci/222/
Executed test runs:
    UNSTABLE: http://s-jenkins:8080/job/jenkins-launchpad-plugin-ci/./distribution=quantal,flavor=amd64/222/console

Click here to trigger a rebuild:
http://s-jenkins:8080/job/jenkins-launchpad-plugin-ci/222//rebuild/?

review: Needs Fixing (continuous-integration)

[Francis Ginther (fginther)]

This doesn't work right due to some authorization issue:
Traceback (most recent call last):
  File "/var/lib/jenkins/jenkins-launchpad-plugin-wfn/waitForNodeIdle.py", line 95, in <module>
    sys.exit(main())
  File "/var/lib/jenkins/jenkins-launchpad-plugin-wfn/waitForNodeIdle.py", line 87, in main
    mark_offline(json_jenkins, args.url, args.node)
  File "/var/lib/jenkins/jenkins-launchpad-plugin-wfn/waitForNodeIdle.py", line 46, in mark_offline
    data = json_jenkins.get_json_data(offline_url, False)
  File "/var/lib/jenkins/jenkins-launchpad-plugin-wfn/jsonjenkins.py", line 19, in get_json_data
    data = urllib2.urlopen(url)
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 444, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden

It's the call to load http://{server}/computer/{node}/toggleOffline which works fine when triggered from a browser. Yes, I know that get_json_data isn't the right call for this, but it's failing on the urlopen method, not the json.loads method.

The alternative solution is to call jenkins-cli from within the python script. I don't want to do that as it's mixing two APIs when a pure web based solution should work.

[Martin Mrazik (mrazik)]

using PreemptiveBasicAuthHandler class from[1] instead of urllib2.HTTPBasicAuthHandler seems to fix the issue (well, then it fails on the json.load call but that seems to be straightforward).

What about adding PreemptiveBasicAuthHandler and renaming JSONJenkins to something else (JenkinsURLlib?) providing two different methods -- get_json_data (with the json.load as its right now) and get_data (returning raw data)?

[1]
http://pythonhosted.org/jenkinsapi/_modules/jenkinsapi/utils/urlopener.html

[Francis Ginther (fginther)]

Address the earlier comments. This version was tested on a local jenkins server.

mark-offline will mark the node offline, then wait for the timeout (currently 2 hours) for the node to become idle. When a node is offline, jenkins will not schedule new jobs, but does wait for any running jobs to complete. This allows for a graceful shutdown of a node without impacting any running jobs.

[Francis Ginther (fginther)]

Now ready for review.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:85
http://s-jenkins:8080/job/jenkins-launchpad-plugin-ci/235/
Executed test runs:
    UNSTABLE: http://s-jenkins:8080/job/jenkins-launchpad-plugin-ci/./distribution=quantal,flavor=amd64/235/console

Click here to trigger a rebuild:
http://s-jenkins:8080/job/jenkins-launchpad-plugin-ci/235//rebuild/?

review: Needs Fixing (continuous-integration)

[Martin Mrazik (mrazik)]

Tests are failing.

review: Needs Fixing

Unmerged revisions

85. By Francis Ginther on 2013-03-01

Added logging messages.

84. By Francis Ginther on 2013-03-01

Merge to trunk

83. By Francis Ginther on 2013-02-22

Updated waitForNodeIdle test cases.

82. By Francis Ginther on 2013-02-21

Refactored and renamed JSONJenkins to JenkinsUrl and fixed authorization to allow setting nodes offline. Moved code to manipulate nodes into jenkinsutils.

81. By Francis Ginther on 2013-02-19

Add options to mark the node offline and online.

80. By Francis Ginther on 2013-02-17

Added test_waitForNodeIdle.py for unit tests.

79. By Francis Ginther on 2013-02-15

Fixed timeout issue found in testing

78. By Francis Ginther on 2013-02-15

Add wait_for_node_idle.py to determine when a jenkins node is idle and ready for clean-up actions.

=== modified file 'jenkinsutils.py'
--- jenkinsutils.py	2013-02-23 13:27:39 +0000
+++ jenkinsutils.py	2013-03-01 03:45:24 +0000
@@ -1,11 +1,34 @@
+import datetime
+import jenkins
+import json
 import re
+import time
+import urllib2
+from jenkinsapi.utils.urlopener import get_jenkins_auth_handler
 from xml.dom.minidom import parseString
-import jenkins
+
 from settings import (JENKINS_URL,
                       JENKINS_PASSWORD, JENKINS_USER,
                       PUBLIC_JENKINS_URL, URLS_TO_HIDE, logger)
 
 
+class JenkinsUrl():
+    urllib_opener = None
+
+    def __init__(self, jenkins_url, username, password):
+        handlers = get_jenkins_auth_handler(username, password, jenkins_url)
+        self.urllib_opener = urllib2.build_opener(*handlers)
+        urllib2.install_opener(self.urllib_opener)
+
+    def get_json_data(self, url, append_api=True):
+        if append_api:
+            url = url + '/api/json'
+        return json.load(self.get_data(url))
+
+    def get_data(self, url):
+        return urllib2.urlopen(url)
+
+
 def is_job_published(job):
     if not job:
         logger.debug('is_job_published: no job specified')
@@ -217,3 +240,52 @@
         raise KeyError("Multiple merge jobs found: %s" %
                        (', '.join(merge_jobs)))
     return merge_jobs[0]
+
+
+def is_node_offline(jenkins, url, node):
+    '''Returns the offline state of the node'''
+    node_url = '/'.join([url, 'computer', node])
+    data = jenkins.get_json_data(node_url)
+    return data['temporarilyOffline']
+
+
+def is_node_idle(jenkins, url, node):
+    '''Returns the idle state of the node'''
+    node_url = '/'.join([url, 'computer', node])
+    data = jenkins.get_json_data(node_url)
+    return data['idle']
+
+
+def set_node_offline(jenkins, url, node, message):
+    '''Sets the node offline'''
+    node_url = '/'.join([url, 'computer', node])
+    if is_node_offline(jenkins, url, node):
+        return
+    offline_url = node_url + "/toggleOffline?offlineMessage=" + message
+    jenkins.get_data(offline_url)
+    if not is_node_offline(jenkins, url, node):
+        raise EnvironmentError("Failed to set node offline")
+
+
+def set_node_online(jenkins, url, node):
+    '''Sets the node online'''
+    node_url = '/'.join([url, 'computer', node])
+    if not is_node_offline(jenkins, url, node):
+        return
+    online_url = node_url + "/toggleOffline"
+    jenkins.get_data(online_url)
+    if is_node_offline(jenkins, url, node):
+        raise EnvironmentError("Failed to set node online")
+
+
+def wait_for_node_idle(jenkins, url, node, timeout):
+    '''Waits timeout seconds for a jenkins slave node to become idle'''
+    node_url = '/'.join([url, 'computer', node])
+    start_time = current_time = datetime.datetime.now()
+    end_time = start_time + datetime.timedelta(seconds=timeout)
+    while current_time < end_time:
+        if is_node_idle(jenkins, url, node):
+            return 0
+        current_time = datetime.datetime.now()
+        time.sleep(10)
+    raise EnvironmentError("Node failed to become idle")

=== removed file 'jsonjenkins.py'
--- jsonjenkins.py	2013-02-06 10:01:05 +0000
+++ jsonjenkins.py	1970-01-01 00:00:00 +0000
@@ -1,19 +0,0 @@
-import json
-import urllib2
-
-
-class JSONJenkins():
-    urllib_opener = None
-
-    def __init__(self, jenkins_url, username, password):
-        passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
-        passman.add_password(None, jenkins_url, username, password)
-        authhandler = urllib2.HTTPBasicAuthHandler(passman)
-        self.urllib_opener = urllib2.build_opener(authhandler)
-        urllib2.install_opener(self.urllib_opener)
-
-    def get_json_data(self, url, append_api=True):
-        if append_api:
-            url = url + '/api/json'
-        data = urllib2.urlopen(url)
-        return json.load(data)

=== modified file 'launchpad.py'
--- launchpad.py	2013-02-23 17:25:38 +0000
+++ launchpad.py	2013-03-01 03:45:24 +0000
@@ -8,7 +8,7 @@
                       LP_ENV, LP_APP, logger)
 from lazr.restfulclient.errors import Unauthorized
 import re
-from jsonjenkins import JSONJenkins
+from jenkinsutils import JenkinsUrl
 from textwrap import dedent
 import jenkinsutils
 import jenkins
@@ -189,8 +189,8 @@
 
     def _get_json_jenkins(self, url):
         if not self._json_jenkins:
-            self._json_jenkins = JSONJenkins(url, JENKINS_USER,
-                                             JENKINS_PASSWORD)
+            self._json_jenkins = JenkinsUrl(url, JENKINS_USER,
+                                            JENKINS_PASSWORD)
         return self._json_jenkins
 
     def format_message_for_mp_update(self, build_url, message=None,

=== modified file 'tests/test_jenkinsutils.py'
--- tests/test_jenkinsutils.py	2013-02-23 19:57:22 +0000
+++ tests/test_jenkinsutils.py	2013-03-01 03:45:24 +0000
@@ -1,3 +1,4 @@
+import datetime
 from mock import MagicMock, patch
 import jenkinsutils
 import unittest
@@ -306,3 +307,136 @@
                                        'missing_branch')
         self.assertIn("jenkins job parameter [missing_branch]: not found",
                       "%s" % (err.exception))
+
+
+class NodeTestCase(unittest.TestCase):
+
+    def test_is_node_idle(self):
+        jenkins_mock = MagicMock()
+        jenkins_mock.get_json_data = MagicMock(return_value={'idle': True})
+        ret = jenkinsutils.is_node_idle(jenkins_mock, 'http://ip',
+                                        'pbuilder-node')
+        self.assertTrue(ret)
+
+    def test_is_node_busy(self):
+        jenkins_mock = MagicMock()
+        jenkins_mock.get_json_data = MagicMock(return_value={'idle': False})
+        ret = jenkinsutils.is_node_idle(jenkins_mock, 'http://ip',
+                                        'pbuilder-node')
+        self.assertFalse(ret)
+
+    def test_is_node_offline(self):
+        jenkins_mock = MagicMock()
+        jenkins_mock.get_json_data = MagicMock(return_value={'temporarilyOffline': True})
+        ret = jenkinsutils.is_node_offline(jenkins_mock, 'http://ip',
+                                           'pbuilder-node')
+        self.assertTrue(ret)
+
+    def test_is_node_online(self):
+        jenkins_mock = MagicMock()
+        jenkins_mock.get_json_data = MagicMock(return_value={'temporarilyOffline': False})
+        ret = jenkinsutils.is_node_offline(jenkins_mock, 'http://ip',
+                                           'pbuilder-node')
+        self.assertFalse(ret)
+
+    def test_set_node_online(self):
+        jenkins_mock = MagicMock()
+        node_data = [{'temporarilyOffline': False},
+                     {'temporarilyOffline': True}]
+        jenkins_mock.get_json_data = MagicMock(side_effect=node_data)
+        jenkins_mock.get_data = MagicMock()
+        jenkinsutils.set_node_offline(jenkins_mock, 'http://ip',
+                                      'pbuilder-node', 'message')
+        jenkins_mock.get_data.assert_called_with('http://ip/computer/pbuilder-node/toggleOffline?offlineMessage=message')
+
+    def test_set_node_online_already_online(self):
+        jenkins_mock = MagicMock()
+        node_data = [{'temporarilyOffline': True},
+                     {'temporarilyOffline': True}]
+        jenkins_mock.get_json_data = MagicMock(side_effect=node_data)
+        jenkins_mock.get_data = MagicMock()
+        jenkinsutils.set_node_offline(jenkins_mock, 'http://ip',
+                                      'pbuilder-node', 'message')
+        jenkins_mock.get_data.assert_not_called()
+
+    def test_set_node_online_fail(self):
+        jenkins_mock = MagicMock()
+        node_data = [{'temporarilyOffline': False},
+                     {'temporarilyOffline': False}]
+        jenkins_mock.get_json_data = MagicMock(side_effect=node_data)
+        jenkins_mock.get_data = MagicMock()
+        with self.assertRaises(EnvironmentError) as err:
+            jenkinsutils.set_node_offline(jenkins_mock, 'http://ip',
+                                          'pbuilder-node', 'message')
+        self.assertEqual('Failed to set node offline', '%s' % err.exception)
+
+    def test_set_node_offline(self):
+        jenkins_mock = MagicMock()
+        node_data = [{'temporarilyOffline': True},
+                     {'temporarilyOffline': False}]
+        jenkins_mock.get_json_data = MagicMock(side_effect=node_data)
+        jenkins_mock.get_data = MagicMock()
+        jenkinsutils.set_node_online(jenkins_mock, 'http://ip',
+                                      'pbuilder-node')
+        jenkins_mock.get_data.assert_called_with('http://ip/computer/pbuilder-node/toggleOffline')
+
+    def test_set_node_offline_already_offline(self):
+        jenkins_mock = MagicMock()
+        node_data = [{'temporarilyOffline': False},
+                     {'temporarilyOffline': False}]
+        jenkins_mock.get_json_data = MagicMock(side_effect=node_data)
+        jenkins_mock.get_data = MagicMock()
+        jenkinsutils.set_node_online(jenkins_mock, 'http://ip',
+                                      'pbuilder-node')
+        jenkins_mock.get_data.assert_not_called()
+
+    def test_set_node_offline_fail(self):
+        jenkins_mock = MagicMock()
+        node_data = [{'temporarilyOffline': True},
+                     {'temporarilyOffline': True}]
+        jenkins_mock.get_json_data = MagicMock(side_effect=node_data)
+        jenkins_mock.get_data = MagicMock()
+        with self.assertRaises(EnvironmentError) as err:
+            jenkinsutils.set_node_online(jenkins_mock, 'http://ip',
+                                         'pbuilder-node')
+        self.assertEqual('Failed to set node online', '%s' % err.exception)
+
+
+class WaitForNodeIdle(unittest.TestCase):
+
+    def setUp(self):
+        self.times = [datetime.datetime.now(),
+                      datetime.datetime.now() + datetime.timedelta(seconds=10),
+                      datetime.datetime.now() + datetime.timedelta(seconds=20),
+                      datetime.datetime.now() + datetime.timedelta(seconds=30)]
+        self.datetime_mock = MagicMock()
+        self.datetime_mock.now = MagicMock(side_effect=self.times)
+        self.datetime_patch = patch('datetime.datetime', self.datetime_mock)
+        self.datetime_patch.start()
+        self.sleep_patch = patch('time.sleep', MagicMock())
+        self.sleep_patch.start()
+
+    def tearDown(self):
+        self.datetime_patch.stop()
+        self.sleep_patch.stop()
+
+    def test_wait_for_node_idle(self):
+        '''Verifies that an idle node returns 0'''
+        jenkins_mock = MagicMock()
+        jenkins_mock.get_json_data = MagicMock(return_value={'idle': True})
+        ret = jenkinsutils.wait_for_node_idle(jenkins_mock,
+                                              'http://ip',
+                                              'pbuilder-node',
+                                              20)
+        self.assertEqual(ret, 0)
+
+    def test_wait_for_node_idle_timeout(self):
+        '''Verifies that a busy node returns 1 (timeout hit)'''
+        jenkins_mock = MagicMock()
+        jenkins_mock.get_json_data = MagicMock(return_value={'idle': False})
+        with self.assertRaises(EnvironmentError) as err:
+            jenkinsutils.wait_for_node_idle(jenkins_mock,
+                                            'http://ip',
+                                            'pbuilder-node',
+                                            24)
+        self.assertEqual('Node failed to become idle', '%s' % err.exception)

=== added file 'tests/test_waitForNodeIdle.py'
--- tests/test_waitForNodeIdle.py	1970-01-01 00:00:00 +0000
+++ tests/test_waitForNodeIdle.py	2013-03-01 03:45:24 +0000
@@ -0,0 +1,60 @@
+import datetime
+from mock import patch, MagicMock
+import unittest
+
+import jenkinsutils
+import waitForNodeIdle
+
+
+class TestWaitForNodeIdle(unittest.TestCase):
+
+    def setUp(self):
+        self.args_mock = MagicMock()
+        self.args_mock.url = 'http://ip'
+        self.args_mock.node = 'pbuilder-node'
+        self.args_patch = patch('waitForNodeIdle.parse_arguments',
+                                MagicMock(return_value=self.args_mock))
+        self.args_patch.start()
+
+    def tearDown(self):
+        self.args_patch.stop()
+
+    def test_mark_offline(self):
+        '''Verifies that a successful node offline request returns 0'''
+        set_offline_mock = MagicMock()
+        wait_mock = MagicMock()
+        self.args_mock.mark_offline = True
+        self.args_mock.mark_online = False
+        with patch('jenkinsutils.set_node_offline', set_offline_mock):
+            with patch('jenkinsutils.wait_for_node_idle', wait_mock):
+                ret = waitForNodeIdle.main()
+                self.assertEqual(ret, 0)
+
+    def test_mark_offline_fail(self):
+        '''Verifies that a failed node offline request returns 1'''
+        set_offline_mock = MagicMock()
+        wait_mock = MagicMock(side_effect=EnvironmentError())
+        self.args_mock.mark_offline = True
+        self.args_mock.mark_online = False
+        with patch('jenkinsutils.set_node_offline', set_offline_mock):
+            with patch('jenkinsutils.wait_for_node_idle', wait_mock):
+                ret = waitForNodeIdle.main()
+                self.assertEqual(ret, 1)
+
+    def test_mark_online(self):
+        '''Verifies that a successful node online request returns 0'''
+        self.args_mock.mark_offline = False
+        self.args_mock.mark_online = True
+        set_online_mock = MagicMock()
+        with patch('jenkinsutils.set_node_online', set_online_mock):
+            ret = waitForNodeIdle.main()
+            self.assertEqual(ret, 0)
+
+    def test_mark_online_fail(self):
+        '''Verifies that a failed node online request returns 1'''
+        self.args_mock.mark_offline = False
+        self.args_mock.mark_online = True
+        set_online_mock = MagicMock(side_effect=EnvironmentError())
+        with patch('jenkinsutils.set_node_online', set_online_mock):
+            ret = waitForNodeIdle.main()
+            self.assertEqual(ret, 1)

=== added file 'waitForNodeIdle.py'
--- waitForNodeIdle.py	1970-01-01 00:00:00 +0000
+++ waitForNodeIdle.py	2013-03-01 03:45:24 +0000
@@ -0,0 +1,63 @@
+#!/usr/bin/env python
+import argparse
+import sys
+import time
+
+import jenkinsutils
+from settings import (JENKINS_USER, JENKINS_PASSWORD, logger)
+
+
+def parse_arguments():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-u', '--url',
+                        required=True,
+                        help='URL of the jenkins server')
+    parser.add_argument('-n', '--node',
+                        required=True,
+                        help='Node to query')
+    parser.add_argument('-t', '--timeout',
+                        required=False,
+                        default=7200,
+                        type=int,
+                        help='Length of time in seconds to wait')
+    parser.add_argument('--mark-offline',
+                        default=False,
+                        action='store_true',
+                        help='Mark the node offline and wait')
+    parser.add_argument('--mark-online',
+                        default=False,
+                        action='store_true',
+                        help='Mark the node online')
+    args = parser.parse_args()
+    if not (args.mark_online or args.mark_offline):
+        parser.error('Must specify --mark-offline or --mark-online')
+    return args
+
+
+def main():
+    args = parse_arguments()
+    jenkins = jenkinsutils.JenkinsUrl(args.url, JENKINS_USER, JENKINS_PASSWORD)
+    if args.mark_offline:
+        try:
+            jenkinsutils.set_node_offline(jenkins, args.url, args.node,
+                                          'waitForNodeIdle')
+            jenkinsutils.wait_for_node_idle(jenkins, args.url, args.node,
+                                            args.timeout)
+            logger.debug('Node is offline')
+        except EnvironmentError:
+            logger.debug('Failed to fully offline node')
+            if jenkinsutils.is_node_offline(jenkins, args.url, args.node):
+                logger.debug('Node is offline, but may be busy')
+            return 1
+    elif args.mark_online:
+        try:
+            jenkinsutils.set_node_online(jenkins, args.url, args.node)
+            logger.debug('Node is online')
+        except EnvironmentError:
+            logger.debug('Failed to online node')
+            return 1
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())