Merge lp:~pfalcon/linaro-android-build-tools/cmdline-client into lp:linaro-android-build-tools

[Paul Sokolovsky]

android-build command line client. This implements https://cards.linaro.org/browse/LAVA-540 . See README for usage instructions - worth given a try to it, if possible.

[Milo Casagrande (milo)]

Hey Paul,

following my comments, there's some typos that need fixing and a couple of suggestion/thoughts.
Marking as fix needed, but once you cleaned the typos it should be ready to go.

+Basic android-build.linaro.org command-line client
+==================================================
+
+This tool provides proof-of-concept command-line client for Linaro Android
+Build Jenkins master (android-build.linaro.org), as an alternative for
+existing Web UI interface. One particular feature the command-line client
+has and Web UI lacks is support for creating completely private builds (such
+builds are not availabel via Web UI at all). At the same time, this tool is
+so far in the proof of concept stage and provides only basic job management
+actions (namely, create a job and schedule its build). If you find this tools

s/this tools/this tool

+useful, please share you comments and suggestion using this bugtracker:

s/you comments/your comments

+
+https://bugs.launchpad.net/linaro-android-infrastructure
+
+to help its improvement and evolution.
+
+
+Quick start
+-----------
+
+1. You should have access to Jenkins at:
+
+http://android-build.linaro.org/jenkins/
+
+(Generally available for Linaro Android team members).
+
+2. Download android-build-client tool. As it is written to depend only

s/to depend/it depends

+on Python standard library, an easy way to get it is to download seperate
+file via Launchpad BZR viewer:
+
+wget <fill in link once branch lands>
+chmod +x android-build-client
+
+3. Look up your Jenkins API token by visiting
+https://android-build.linaro.org/jenkins/me/configure and clicking
+"Show API Token..." button.
+
+4. Run:
+
+./android-build-client authorize
+
+Enter you Jenkins username and API token. Note that this will cache these

s/you Jenkins/your Jenkins

+credentials in your home dir, so use this only on your personal well-protected
+workstation. The alternative is to use --user switch and input API key
+interactively.
+
+5. To create a new job, prepare an Android job config (see
+https://wiki.linaro.org/Platform/Android/LinaroAndroidBuildService for more
+info) in a file. Run:
+
+./android-build-client create <job_name> <job_config_file>
+
+Note that job name should conform to Android Build Service naming
+converntions, see documentation link above for more info. To create a job
+private for particular group, pass --private=<group> switch to create command.

s/job private/private job

+
+6. To schedule a build, run:
+
+./android-build-client build <job_name>
+
+
+Run ./android-build-client --help to see all options.

I tried it, but it does not show me all the available "commands", just the normale options.

It's early in the development stage, but it might be worth to take a look at how lava-tool is handling the commands definition.

There's a lot more code to write to achieve that, but it is a well structured pattern. Just food for thoughts, and something to keep in mind.

=== added file 'utils/cmdline-client/android-build-client'
--- utils/cmdline-client/android-build-client 1970-01-01 00:00:00 +0000
+++ utils/cmdline-client/android-build-client 2013-08-29 17:39:17 +0000
@@ -0,0 +1,199 @@
+#!/usr/bin/env py...

review: Needs Fixing

[Paul Sokolovsky (pfalcon)]

Hello Milo,

Thanks for comments!

On Fri, 30 Aug 2013 14:31:38 -0000
Milo Casagrande <email address hidden> wrote:

> Review: Needs Fixing
>
> Hey Paul,
>
> following my comments, there's some typos that need fixing and a
> couple of suggestion/thoughts. Marking as fix needed, but once you
> cleaned the typos it should be ready to go.
>
> +Basic android-build.linaro.org command-line client
> +==================================================
> +
> +This tool provides proof-of-concept command-line client for Linaro
> Android +Build Jenkins master (android-build.linaro.org), as an
> alternative for +existing Web UI interface. One particular feature
> the command-line client +has and Web UI lacks is support for creating
> completely private builds (such +builds are not availabel via Web UI
> at all). At the same time, this tool is +so far in the proof of
> concept stage and provides only basic job management +actions
> (namely, create a job and schedule its build). If you find this tools
>
> s/this tools/this tool

fixed

>
> +useful, please share you comments and suggestion using this
> bugtracker:
>
> s/you comments/your comments

fixed

>
> +
> +https://bugs.launchpad.net/linaro-android-infrastructure
> +
> +to help its improvement and evolution.
> +
> +
> +Quick start
> +-----------
> +
> +1. You should have access to Jenkins at:
> +
> +http://android-build.linaro.org/jenkins/
> +
> +(Generally available for Linaro Android team members).
> +
> +2. Download android-build-client tool. As it is written to depend
> only
>
> s/to depend/it depends

You probably mean replacing the whole starting phrase? Nope, that
loses meaning. It not just randomly depends only on stdlib, it's
purposedly written to ;-).

>
> +on Python standard library, an easy way to get it is to download
> seperate +file via Launchpad BZR viewer:
> +
> +wget <fill in link once branch lands>
> +chmod +x android-build-client
> +
> +3. Look up your Jenkins API token by visiting
> +https://android-build.linaro.org/jenkins/me/configure and clicking
> +"Show API Token..." button.
> +
> +4. Run:
> +
> +./android-build-client authorize
> +
> +Enter you Jenkins username and API token. Note that this will cache
> these
>
> s/you Jenkins/your Jenkins

fixed

>
> +credentials in your home dir, so use this only on your personal
> well-protected +workstation. The alternative is to use --user switch
> and input API key +interactively.
> +
> +5. To create a new job, prepare an Android job config (see
> +https://wiki.linaro.org/Platform/Android/LinaroAndroidBuildService
> for more +info) in a file. Run:
> +
> +./android-build-client create <job_name> <job_config_file>
> +
> +Note that job name should conform to Android Build Service naming
> +converntions, see documentation link above for more info. To create
> a job +private for particular group, pass --private=<group> switch to
> create command.
>
> s/job private/private job

Nope, again changes meaning ;-). "job private for particular group" ==
"job which is private for particular group"

>
> +
> +6. To schedule a build, run:
> +
> +./android-build-client build <job_name>
> +
> +
> +Run ./android-build-client --help to s...

[lp:~pfalcon/linaro-android-build-tools/cmdline-client]

663. By Paul Sokolovsky on 2013-09-03

Fixed typos, thanks Milo for review.

664. By Paul Sokolovsky on 2013-09-03

Make --help show which commands are available.

665. By Paul Sokolovsky on 2013-09-03

Print error messages to stderr.

666. By Paul Sokolovsky on 2013-09-03

Fix typo.

[Paul Sokolovsky (pfalcon)]

<pfalcon> Gwaihir: thanks for review on https://code.launchpad.net/~pfalcon/linaro-android-build-tools/cmdline-client/+merge/182969 . I acted on most comments. The only serious one IMHO is licensing, but as I replied, it's the license of linaro-android-build-tools ;-I. So, unless you strongly suggest to use diff. lic. just for that script, it could be merged
<Gwaihir> pfalcon, sure, lets not block on that, it is an open source license after all, non GPL compatible, but OSI approved... will raise this with Alan and see if we need to use just one license for all our projects or if it is fine to use others as long as they are open source

review: Approve

=== added directory 'utils/cmdline-client'
=== added file 'utils/cmdline-client/README'
--- utils/cmdline-client/README	1970-01-01 00:00:00 +0000
+++ utils/cmdline-client/README	2013-09-03 10:45:47 +0000
@@ -0,0 +1,62 @@
+Basic android-build.linaro.org command-line client
+==================================================
+
+This tool provides proof-of-concept command-line client for Linaro Android
+Build Jenkins master (android-build.linaro.org), as an alternative for
+existing Web UI interface. One particular feature the command-line client
+has and Web UI lacks is support for creating completely private builds (such
+builds are not availabel via Web UI at all). At the same time, this tool is
+so far in the proof of concept stage and provides only basic job management
+actions (namely, create a job and schedule its build). If you find this tool
+useful, please share your comments and suggestion using this bugtracker:
+
+https://bugs.launchpad.net/linaro-android-infrastructure
+
+to help its improvement and evolution.
+
+
+Quick start
+-----------
+
+1. You should have access to Jenkins at:
+
+http://android-build.linaro.org/jenkins/
+
+(Generally available for Linaro Android team members).
+
+2. Download android-build-client tool. As it is written to depend only
+on Python standard library, an easy way to get it is to download seperate
+file via Launchpad BZR viewer:
+
+wget <fill in link once branch lands>
+chmod +x android-build-client
+
+3. Look up your Jenkins API token by visiting
+https://android-build.linaro.org/jenkins/me/configure and clicking
+"Show API Token..." button.
+
+4. Run:
+
+./android-build-client authorize
+
+Enter yous Jenkins username and API token. Note that this will cache these
+credentials in your home dir, so use this only on your personal well-protected
+workstation. The alternative is to use --user switch and input API key
+interactively.
+
+5. To create a new job, prepare an Android job config (see
+https://wiki.linaro.org/Platform/Android/LinaroAndroidBuildService for more
+info) in a file. Run:
+
+./android-build-client create <job_name> <job_config_file>
+
+Note that job name should conform to Android Build Service naming
+converntions, see documentation link above for more info. To create a job
+private for a particular group, pass --private=<group> switch to create command.
+
+6. To schedule a build, run:
+
+./android-build-client build <job_name>
+
+
+Run ./android-build-client --help to see all options.

=== added file 'utils/cmdline-client/android-build-client'
--- utils/cmdline-client/android-build-client	1970-01-01 00:00:00 +0000
+++ utils/cmdline-client/android-build-client	2013-09-03 10:45:47 +0000
@@ -0,0 +1,199 @@
+#!/usr/bin/env python
+###############################################################################
+# Copyright (c) 2013 Linaro
+# All rights reserved. This program and the accompanying materials
+# are made available under the terms of the Eclipse Public License v1.0
+# which accompanies this distribution, and is available at
+# http://www.eclipse.org/legal/epl-v10.html
+###############################################################################
+
+import base64
+import json
+import os
+import sys
+import urllib2
+import optparse
+import getpass
+from xml.dom import minidom
+
+PRIVATE_ACL = """\
+    <hudson.security.AuthorizationMatrixProperty>
+      <permission>hudson.model.Item.Build:%(group)s</permission>
+      <permission>hudson.model.Item.Configure:%(group)s</permission>
+      <permission>hudson.model.Item.Read:%(group)s</permission>
+      <permission>hudson.model.Item.Cancel:%(group)s</permission>
+      <permission>hudson.model.Item.Discover:%(group)s</permission>
+    </hudson.security.AuthorizationMatrixProperty>
+"""
+
+
+class Jenkins(object):
+
+    def __init__(self, base_url, username, passwd):
+        self.base = base_url
+        self.auth_headers = {
+            'Authorization': 'Basic %s' % (
+                base64.b64encode('%s:%s' % (username, passwd)))}
+        self.csrf = None
+
+    def jenkins_rest(self, jenkins_path, data=None, extra_headers=None):
+        """Make an authenticated request to jenkins.
+
+        @param jenkins_path: The path on the Jenkins instance to make the
+            request to.
+        @param data: Data to include in the request (if this is not None the
+            request will be a POST).
+        @param extra_headers: A dictionary of extra headers that will passed in
+            addition to Authorization.
+        @raises urllib2.HTTPError: If the response is not a HTTP 200.
+        @returns: the body of the response.
+        """
+        headers = self.auth_headers.copy()
+        if extra_headers:
+            headers.update(extra_headers)
+        url = self.base + jenkins_path
+        print url
+        req = urllib2.Request(
+            url, data, headers)
+        resp = urllib2.urlopen(req)
+        data = resp.read()
+        resp.close()
+        return data
+
+    def get_csrf_headers(self):
+        if self.csrf is None:
+            try:
+                crumb_data = self.jenkins_rest('crumbIssuer/api/json')
+                data = json.loads(crumb_data)
+                self.csrf = {data['crumbRequestField']: data['crumb']}
+            except urllib2.HTTPError:
+                # Ignore errors in case CSRF protection is not enabled
+                self.csrf = {}
+        return self.csrf
+
+    def jenkins_rest_post(self, jenkins_path, data, extra_headers=None):
+        if extra_headers is None:
+            extra_headers = {}
+        extra_headers.update(self.get_csrf_headers())
+        return self.jenkins_rest(jenkins_path, data, extra_headers)
+
+    def get_job_config(self, job_name):
+        return self.jenkins_rest('job/' + job_name + '/config.xml')
+
+    def post_config(self, url, config_xml):
+        return self.jenkins_rest_post(url, config_xml,
+            {'Content-Type': 'text/xml'})
+
+    def set_job_config(self, job_name, config_xml):
+        return self.post_config('job/' + job_name + '/config.xml', config_xml)
+
+    def create_job(self, job_name, config_xml):
+        return self.post_config('createItem?name=' + job_name, config_xml)
+
+    def build_job(self, job_name):
+        self.jenkins_rest_post(
+            'job/' + job_name + '/buildWithParameters?delay=0sec', '')
+
+
+def job2user_group(job_group):
+    if job_group == "linaro-android":
+        return "linaro-android-builders"
+    return job_group
+
+
+def error(msg):
+    print >>sys.stderr, msg
+    sys.exit(1)
+
+
+def main():
+    global options
+    optparser = optparse.OptionParser(usage="%prog authorize|create|build <args>...")
+    optparser.add_option("--url",
+        default="https://android-build.linaro.org/jenkins",
+        help="Jenkins base url, default: %default")
+    optparser.add_option("--user",
+                         help="Jenkins username, default: $USER")
+    optparser.add_option("--apikey-file", metavar="FILE",
+                         help="File holding Jenkins API key")
+    optparser.add_option("--cred-file", metavar="FILE",
+                         help="File holding Jenkins username:API key pair")
+    optparser.add_option("--private", metavar="GROUP",
+        help="Create private job accessible to GROUP")
+
+    options, args = optparser.parse_args(sys.argv[1:])
+    if len(args) < 1:
+        optparser.error("Wrong number of arguments")
+
+    config_dir = os.path.expanduser("~/.config/android-build-client")
+
+    password = None
+    if options.cred_file:
+        options.user, password = open(options.cred_file).read().strip().split(":")
+    elif options.apikey_file:
+        password = open(options.passwd_file).read().strip()
+    elif os.path.exists(config_dir + "/cred"):
+        options.user, password = open(config_dir + "/cred").read().strip().split(":")
+        print "INFO: Using cached authorization for: %s" % options.user
+    elif args[0] != "authorize":
+        password = getpass.getpass("API Token:")
+
+    if options.url[-1] != '/':
+        options.url += '/'
+
+    j = Jenkins(options.url, options.user, password)
+
+    if args[0] == "authorize":
+        if len(args) != 1:
+            optparser.error("Usage: authorize")
+        import getpass
+        user = raw_input("Enter user name: ")
+        while True:
+            key = getpass.getpass("API key: ")
+            if len(key) == 32:
+                break
+            print "This does not look like Jenkisn API key, please try again"
+        try:
+            os.makedirs(config_dir)
+        except OSError:
+            pass
+        f = open(config_dir + "/cred", "w")
+        print >>f, "%s:%s" % (user, key)
+        f.close()
+        print "Credentials cached for future use"
+    elif args[0] == "create":
+        if len(args) != 3:
+            optparser.error("Usage: create <job_name> <build_config_file>")
+        job_group, job_subname = args[1].split("/")
+        template = j.get_job_config("template_" + job2user_group(job_group))
+        dom = minidom.parseString(template)
+        nodes = dom.getElementsByTagName("hudson.model.StringParameterDefinition")
+        assert len(nodes) == 1
+        n = nodes[0].getElementsByTagName("defaultValue")[0]
+        build_config = open(args[2]).read()
+        n.childNodes[0].data = base64.encodestring(build_config)
+
+        if options.private:
+            n = dom.getElementsByTagName("properties")[0]
+            c = n.getElementsByTagName("hudson.security.AuthorizationMatrixProperty")[0]
+            acl_dom = minidom.parseString(PRIVATE_ACL % {"group": options.private})
+            n.replaceChild(acl_dom.documentElement, c)
+
+        job_conf = dom.toxml()
+        jenkins_job = args[1].replace("/", "_")
+        try:
+            j.create_job(jenkins_job, job_conf)
+        except urllib2.HTTPError:
+            error("Error creating job '%s' (job exists?)" % jenkins_job)
+        print "Job created successfully: %s/job/%s/" % (options.url, jenkins_job)
+    elif args[0] == "build":
+        if len(args) != 2:
+            optparser.error("Usage: build <job_name>")
+        j.build_job(args[1].replace("/", "_"))
+        print "Build queued"
+    else:
+        optparser.error("Unknown command '%s'" % args[0])
+
+
+if __name__ == "__main__":
+    main()