Merge ~dannf/charms/+source/scalebot-jenkins:go-stop into ~dannf/charms/+source/scalebot-jenkins:master

[dann frazier]

Unmerged commits

13888a4... by dann frazier on 2021-04-15

Block unnecessary reactions when the charm is stopping

If a user removes the scalebot application (`juju remove-application`), juju
will trigger the stop hook. We provide a stop hook which destroys the lab
controller, freeing up resources in the cloud. However, the charm continues
to react to other events, which can lead to it *re*-bootstrapping a
controller. This can lead to orphaned resources in the cloud.

Introduce a new pair of states[*], "scalebot.go" and "scalebot.stop", which
are intended to be mutually exclusive. The charm will start out and remain
in the "scalebot.go" state until the stop hook is called, at which point
we'll transition to the "scalebot.stop" state. Add guards to prevent
various reactions from executing in "scalebot.stop".

[*] The "state" API has been apparently deprecated for a "flag" API for
    some time. We should switch over to the "flag" API - but I'll leave
    that for another time.

79a5200... by dann frazier on 2021-03-30

Fix typo and simplify log message

The main goal here is to correct spelling of "bootstrapping", but I also
decided to shorten the message because I'm not sure the second sentence
was adding to the first.

5f7aec1... by dann frazier on 2021-03-30

Fix reference-before-assignment issue

Seen in debug-log:

unit-scalebot-0: 12:01:15 WARNING unit.scalebot/0.stop UnboundLocalError: local variable 'modeldefaultsfile' referenced before assignment
unit-scalebot-0: 12:01:16 ERROR juju.worker.uniter.operation hook "stop" (via explicit, bespoke hook script) failed: exit status 1

fe90de4... by dann frazier on 2021-03-24

Add focal support

This charm deploys fine on Ubuntu 20.04 ('focal'), so designate it to
be a supported series.

5e2ad36... by dann frazier on 2021-03-24

Remove the python3-maas-client dependency

The scalebot-jenkins charm itself does not know anything about MAAS, it is
cloud-agnostic by design. If a specific scalebot config needs to talk to
a MAAS directly, it should install the necessary dependencies itself
(e.g. using scalebot.d/init). python3-maas-client is no longer available
in Ubuntu 20.04 ('focal'), so this is necessary to support that series.

1bd653d... by Taihsiang Ho on 2020-10-08

Merge branch 'mr-remote-lxd' into master

1454aa4... by Taihsiang Ho on 2020-09-30

refactor(config.yaml): rm useless lxd parameters

676404c... by Taihsiang Ho on 2020-09-29

feat: integrate juju yaml of remote lxd

By integrating the clouds.yaml and credentials.yaml for remote lxd, we could
merge the bootstraping methods of remote lxd and maas clouds.

BREAKING CHANGE: LP merge request: #391092

72c3e58... by Taihsiang Ho on 2020-09-21

feat(log): provide more explicit failure messages for debugging of importjenkins

ca70db6... by Taihsiang Ho on 2020-09-21

style(pre-commit): style fix

diff --git a/.flake8 b/.flake8
new file mode 100644
index 0000000..b5039a2
--- /dev/null
+++ b/.flake8
@@ -0,0 +1,2 @@
+[flake8]
+ignore = W503, E501
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
new file mode 100644
index 0000000..5f7fa9a
--- /dev/null
+++ b/.pre-commit-config.yaml
@@ -0,0 +1,10 @@
+repos:
+  - repo: https://github.com/psf/black
+    rev: stable
+    hooks:
+      - id: black
+        language_version: python3.8
+  - repo: https://gitlab.com/pycqa/flake8
+    rev: master
+    hooks:
+      - id: flake8
diff --git a/README.devel b/README.devel
new file mode 100644
index 0000000..fbaa6ff
--- /dev/null
+++ b/README.devel
@@ -0,0 +1,12 @@
+# Pushing changes to the charm store
+```
+$ charm build
+$ charm push /tmp/charm-builds/scalebot-jenkins cs:~ce-hyperscale/scalebot-jenkins
+```
+This will return a URL with a new version, e.g.:
+url: cs:~ce-hyperscale/scalebot-jenkins-7
+Use that new URL in the following commands:
+```
+$ charm release <url>
+$ charm grant <url> everyone
+```
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..43e5e2f
--- /dev/null
+++ b/README.md
@@ -0,0 +1,92 @@
+# What is ScaleBot?
+ScaleBot is a framework for running scheduled jobs against a cloud. It is built upon the Jenkins CI and Juju modeling tools.
+
+# QuickStart (using Amazon Web Services)
+Clone a local copy of the sample ScaleBot configuration repository:
+```
+$ git clone git://git.launchpad.net/~scalebot-team/+git/scalebot-config
+```
+Make a copy of the aws credentials template, and add your account keys:
+```
+$ cd scalebot-config
+$ cp labs/aws/credentials.yaml.template labs/aws/credentials.yaml
+$ vi labs/aws/credentials.yaml
+```
+Deploy the ScaleBot bundle. In this example, we'll deploy to local LXD containers:
+```
+$ juju bootstrap localhost
+Creating Juju controller "localhost-localhost" on localhost/localhost
+Looking for packaged Juju agent version 2.2.4 for amd64
+To configure your system to better support LXD containers, please see: https://github.com/lxc/lxd/blob/master/doc/production-setup.md
+Launching controller instance(s) on localhost/localhost...
+ - juju-689661-0 (arch=amd64)
+Fetching Juju GUI 2.9.2
+Waiting for address
+Attempting to connect to 10.98.200.35:22
+Bootstrap agent now started
+Contacting Juju controller at 10.98.200.35 to verify accessibility...
+Bootstrap complete, "localhost-localhost" controller now available.
+Controller machines are in the "controller" model.
+Initial model "default" added.
+$ juju deploy bundle.yaml
+Deploying charm "cs:xenial/jenkins-4"
+Deploying charm "cs:~ce-hyperscale/scalebot-jenkins-3"
+Related "jenkins:extension" and "scalebot:extension"
+Deploy of bundle completed.
+```
+Once deployment has completed, use ```juju status``` to find the IP of your scalebot jenkins server. You can connect to it at http://ip-of-server:8080. The sample configuration includes an example "Hello World" job. Building this job will deploy a node in the Lab cloud and run ```echo "Hello World"``` on it.
+
+# Architecture
+ScaleBot is deployed from a Juju charm that builds a running instance of Jenkins CI, initialized with a git repository that provides job definitions and a target cloud to run those jobs against. We refer to this target cloud as a "lab". ScaleBot will use the target cloud ("lab") configuration to automatically setup an *internal* Juju controller which Jenkins jobs can use to deploy systems and workloads as necessary within the lab.
+
+<!-- To edit this image, modify the following image and select
+     "Publish to the web..."
+     https://docs.google.com/drawings/d/1BPKUyUMAy8PfAf81397eAKD8shFQT3D9uuqDW4yfRCg/edit -->
+
+![alt text](https://docs.google.com/drawings/d/e/2PACX-1vS0RwIg_CyMTKzJlnRJTf9wrH4e4v3W8wbMsI4RwEcB1FuvclU8Q7X4NPVOLSzYWtGBcXERMVI-PY7p/pub?w=935&h=527 "ScaleBot Diagram A")
+
+Of course, since scalebot-jenkins is a charm itself, it also is deployed in a cloud. This can just be a LXD cloud though, as the controller and Jenkins server can easily run in containers. The important thing is that the jenkins server in the ScaleBot cloud is on a network that can access the Lab cloud.
+
+<!-- To edit this image, modify the following image and select
+     "Publish to the web..."
+     https://docs.google.com/drawings/d/1mb8oea6fmZtTmnC-gpMZgmsGnsaMPaO4XELZv0jNRFo/edit -->
+
+![alt text](https://docs.google.com/drawings/d/e/2PACX-1vRNzqlTKIU57N-cdSA1L8M-20zxUbwMxteOAQEs321NrER-EkSEhbb1P-qZc8V_p-dUq1lTmsTl5IHq/pub?w=1016&h=508 "ScaleBot Diagram B")
+
+# Configuration Repository
+The scalebot-jenkins charm accepts configuration that tells ScaleBot where to fetch Jenkins job and Lab Cloud definitions. This data must be in a git repository with the following layout:
+
+```
+jobs/<jobfoo>.yaml
+jobs/<jobbar>.yaml
+...
+jobs/<jobbaz>.yaml
+
+labs/<labname>/<labname>.yaml
+labs/<labname>/clouds.yaml
+labs/<labname>/credentials.yaml
+labs/<labname>/model-defaults.yaml
+scalebot.d/init
+scalebot.d/refresh
+```
+
+Jobs must be in the YAML format used by [Jenkins Job Builder](https://docs.openstack.org/infra/jenkins-job-builder), as that is the tool used to load them into Jenkins. The clouds.yaml, credentials.yaml, and model-defaults.yaml files are passed directly to Juju when bootsrapping the Lab controller, so see [Juju documentation](http://jujucharms.com/docs) for details on their contents.
+
+You may also want to store code in this repository and call it from your jobs.
+
+
+# Job Environment
+ScaleBot will automatically create an empty model prior to building a jenkins job. This model will have a unique name, based on the $BUILD_ID. Jobs can assume that this model exists and is in-focus prior to executing, so your build code can just begin deploying charms immediately on startup.
+
+Since it is expected that builds will deploy systems using Juju, the python [bindings for juju](https://github.com/juju/python-libjuju) are pre-installed on the system.
+
+## Environment Variables
+In addition to the environment variables set by Jenkins, the following variable is also available:
+
+**$SCALEBOT_REPO** will be set to the path of the locally cloned copy of the Scalebot configuration repository.
+
+## scalebot.d/init hook
+If it exists, ```$SCALEBOT_REPO/scalebot.d/init``` will be executed each time the configuration repository is refreshed. You can use this to install any additional dependencies needed for your jobs.
+
+## scalebot.d/refresh hook
+If it exists, ```$SCALEBOT_REPO/scalebot.d/refresh``` will be executed after the repository is updated, and before loading jobs. This is useful in case something job yaml files need to be dynamically generated.
diff --git a/actions.yaml b/actions.yaml
new file mode 100644
index 0000000..6d6949f
--- /dev/null
+++ b/actions.yaml
@@ -0,0 +1,2 @@
+refreshjobs:
+  descripton: Perform a Git Pull and Refresh the Jenkins Jobs if any.
diff --git a/actions/refreshjobs b/actions/refreshjobs
new file mode 100755
index 0000000..21e3870
--- /dev/null
+++ b/actions/refreshjobs
@@ -0,0 +1,6 @@
+#!/bin/sh
+
+set -e
+set -x
+
+su - jenkins -c "${JUJU_CHARM_DIR}/bin/pull-and-reload-jobs"
diff --git a/bin/jobsync.py b/bin/jobsync.py
new file mode 100755
index 0000000..108c8ac
--- /dev/null
+++ b/bin/jobsync.py
@@ -0,0 +1,100 @@
+#!/usr/bin/env python3
+import argparse
+import os
+import logging
+import subprocess
+import sys
+
+SCALEBOT_HOME = "/srv/scalebot"
+SCALEBOT_REPO = os.path.join(SCALEBOT_HOME, "repo")
+
+
+def syncjobs(JOBSET):
+    jjb = "/usr/local/bin/jenkins-jobs"
+    # Load jenkinsjobs config
+    config_ini = "/etc/jenkins_jobs/jenkins_jobs.ini"
+    # common directory to add jjb Macro
+    COMMON = SCALEBOT_REPO + "/jobs/common"
+    cmd = [jjb, "--flush-cache", "--conf", config_ini, "update"]
+    flag = []
+    if os.path.exists(COMMON):
+        jobs = "{}/jobs/{}:{}".format(SCALEBOT_REPO, JOBSET, COMMON)
+        flag = ["-r", jobs]
+    else:
+        jobs = "{}/jobs/{}".format(SCALEBOT_REPO, JOBSET)
+        flag = [jobs]
+    subprocess.run(cmd + flag, check=True)
+
+
+def main():
+
+    parser = argparse.ArgumentParser()
+
+    parser.add_argument(
+        "-s",
+        "--syncjobs",
+        help=(
+            "This will import jenkins jobs via yaml from \
+                                 $SCALEBOT_REPO/jobs/"
+        ),
+        required=False,
+        action="store_true",
+    )
+    parser.add_argument(
+        "-d",
+        "--deletejobs",
+        help=(
+            "This will Delete all jobs from Hudson, useful \
+                              or refreshing job list after branch change."
+        ),
+        required=False,
+        action="store_true",
+    )
+    parser.add_argument(
+        "-j",
+        "--jobset",
+        help=(
+            "Name of the desired jobset from which to import \
+                                 $SCALEBOT_REPO/jobs/$JOBSET"
+        ),
+        required=True,
+        type=str,
+    )
+
+    args = parser.parse_args()
+
+    # If no args specified upon calling tool, dump help and exit
+    if len(sys.argv[1:]) == 0:
+        parser.print_help()
+        parser.exit()
+
+    try:
+        if args.deletejobs:
+            # Delete the jobs completely from Jenkins
+            logger.info("Removing Jenkins Jobs from Hudson")
+            cmd = subprocess.Popen(
+                ["jenkins-jobs", "delete-all"],
+                stdin=subprocess.PIPE,
+                stdout=subprocess.PIPE,
+            )
+            cmd.communicate(input="Y".encode())
+            logger.info("Hudson Jobs Deleted")
+
+        if args.syncjobs:
+            JOBSET = args.jobset
+            logger.info("Importing Jenkins Jobs")
+            syncjobs(JOBSET)
+
+            logger.info("Import Jobs Complete")
+
+    except Exception as e:
+        logger.info("Problem with job import")
+        logger.info(e)
+        return "-1"
+
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/bin/pull-and-reload-jobs b/bin/pull-and-reload-jobs
new file mode 100755
index 0000000..330f824
--- /dev/null
+++ b/bin/pull-and-reload-jobs
@@ -0,0 +1,18 @@
+#!/bin/sh
+
+# Pull the git repo and refresh jobs
+
+set -e
+set -x
+
+[ -n "$SCALEBOT_HOME" ]
+[ -n "$SCALEBOT_JOBSET" ]
+[ -n "$SCALEBOT_REPO" ]
+
+git -C ${SCALEBOT_REPO} pull --ff-only
+refresh="${SCALEBOT_REPO}/scalebot.d/refresh"
+if [ -x "$refresh" ]; then
+    $refresh
+fi
+jobsync="${SCALEBOT_HOME}/bin/jobsync.py"
+${jobsync} -j ${SCALEBOT_JOBSET} -s
diff --git a/config.yaml b/config.yaml
new file mode 100644
index 0000000..98588c7
--- /dev/null
+++ b/config.yaml
@@ -0,0 +1,47 @@
+options:
+  scalebot_private_ssh_key:
+    description: A null-passphrase SSH private key with access to $scalebot_repo
+    type: "string"
+    default:
+  scalebot_config_repo:
+    description: URI to a ScaleBot config git repository
+    type: "string"
+    default: git://git.launchpad.net/~scalebot-team/+git/scalebot-config
+  scalebot_config_branch:
+    description: The branch of $scalebot_repo to checkout
+    type: "string"
+    default: master
+  scalebot_juju_clouds:
+    description: Juju cloud config YAML for target cloud
+    type: "string"
+    default:
+  scalebot_juju_credentials:
+    description: Credentials YAML for Juju cloud defined in scalebot_juju_clouds
+    type: "string"
+    default:
+  scalebot_juju_model_defaults:
+    description: Juju model defaults configuration YAML
+    type: "string"
+    default:
+  scalebot_juju_bootstrap_constraints:
+    description: Bootstrap constraints for Juju controller
+    type: "string"
+    default:
+  scalebot_lab_type:
+    description: Bootstrap lab type for Juju controller
+    type: "string"
+    default: maas
+  scalebot_jobset:
+    description: Desired jobset to import jenkins jobs from
+    type: "string"
+    default:
+  scalebot_production:
+    description: Is this a production (vs. a development) deployment? When set to true, tests will find SCALEBOT_PRODUCTION=1 in their environment. This gives tests an option to behave differently in production vs. development environments.
+    type: "boolean"
+    default: false
+  install_sources:
+    default: |
+      - ppa:juju/stable
+  install_keys:
+    default: |
+      - null
diff --git a/files/known_hosts.lp b/files/known_hosts.lp
new file mode 100644
index 0000000..6c51a1e
--- /dev/null
+++ b/files/known_hosts.lp
@@ -0,0 +1,2 @@
+|1|dHt6eHIAjzK73nzI0mQEboOUmsg=|dngXa3BXrbOi2e8ATMU9VUD6A9E= ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEFREwBD2ye2Xrc2SVcUmmJ44MF1BCB3W11NTaiqzVj7XZnQmgWZk9UadHVY2wBXvelcDO51MPN5ozJjFAknw09rP7XMRJMlAOLSIVoU6DRF1u1j8kJVY+dfiDHheS7+siADnrmb8HGn2xQQ6EJDjAXrw1x58x5eZjQ0PFWdI+pRTdYGvWkpHdXKFO6a9/lDx4uo9MCnePEGi/QnkCmKqLCBUlYNZYRiB8nVee2tMF0mjV8xk1rJ+/UP+897+FXFR9w/B1EPRjiQ35ZNQZKPP4isxPtyMuCQkZY7ckWr5YsylNfvNcyGDnO1XazZhJ71rzDpi1RmnFXBW5i+2dm2y7
+|1|CWLqG45/xn+VEnXUvak6ST5YuiQ=|ZwtFngsyHwu3XLlLeu/JGjQijis= ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEFREwBD2ye2Xrc2SVcUmmJ44MF1BCB3W11NTaiqzVj7XZnQmgWZk9UadHVY2wBXvelcDO51MPN5ozJjFAknw09rP7XMRJMlAOLSIVoU6DRF1u1j8kJVY+dfiDHheS7+siADnrmb8HGn2xQQ6EJDjAXrw1x58x5eZjQ0PFWdI+pRTdYGvWkpHdXKFO6a9/lDx4uo9MCnePEGi/QnkCmKqLCBUlYNZYRiB8nVee2tMF0mjV8xk1rJ+/UP+897+FXFR9w/B1EPRjiQ35ZNQZKPP4isxPtyMuCQkZY7ckWr5YsylNfvNcyGDnO1XazZhJ71rzDpi1RmnFXBW5i+2dm2y7
diff --git a/layer.yaml b/layer.yaml
index 7623e14..413895f 100644
--- a/layer.yaml
+++ b/layer.yaml
@@ -1 +1,22 @@
-includes: ['layer:jenkins-charm']
+includes:
+  - layer:basic
+  - layer:apt
+  - interface:jenkins-extension
+repo: git+ssh://git.launchpad.net/~ce-hyperscale/charms/+source/scalebot-jenkins
+options:
+  basic:
+    use_venv: false
+    python_packages:
+      - python-jenkins==0.4.15
+      - jenkins-job-builder
+      - async-timeout
+      - juju
+    packages:
+      - libssl-dev
+      - libffi-dev
+      - python3-cffi
+      - python3-nacl
+  apt:
+    packages:
+      - git
+      - juju
diff --git a/lib/charms/layer/scalebot_jenkins.py b/lib/charms/layer/scalebot_jenkins.py
new file mode 100644
index 0000000..6a2d81f
--- /dev/null
+++ b/lib/charms/layer/scalebot_jenkins.py
@@ -0,0 +1,44 @@
+from subprocess import CalledProcessError
+from subprocess import check_call
+
+
+__all__ = ["get_bootstrap_failure_hint"]
+
+
+def get_bootstrap_failure_hint(msg):
+    """
+    Check input message and return a guess of the root cause of error.
+
+    Parameters
+    ----------
+    msg : string
+        A message string. It is often stdout or stderr returned by a subprocess command.
+
+    Returns
+    -------
+    string
+        The description of possible failure root cause.
+
+    """
+    rtn_msg_prefix = "Failed for:"
+    if (
+        "cannot start bootstrap instance: failed to acquire node" in msg
+        and "No such tag" in msg
+    ):
+        return f"{rtn_msg_prefix} not finding nodes with specified tags. You may want to check your bootstrap constraint."
+    elif "bootstrap instance started but did not change to Deployed state" in msg:
+        return f"{rtn_msg_prefix} bootstrap instance started but did not change to Deployed state."
+    elif "failed to acquire node: No available machine matches constraints.":
+        return f"{rtn_msg_prefix} failed to acquire node: No available machine matches constraints."
+    else:
+        return f"{rtn_msg_prefix} unknown reason."
+
+
+def if_controller():
+    cmd = ["/bin/su", "-", "jenkins", "-c", "juju controllers"]
+    try:
+        check_call(cmd)
+        return True
+
+    except CalledProcessError:
+        return False
diff --git a/metadata.yaml b/metadata.yaml
index b57c212..7451207 100644
--- a/metadata.yaml
+++ b/metadata.yaml
@@ -1,5 +1,16 @@
 name: scalebot-jenkins
 summary: Automation Server for ScaleBot
-maintainer: dann frazier <[e-mail]>
+maintainer: [e-mail]
 description: |
   Automates jobs in a ScaleBot Lab.
+tags:
+ - ops
+series:
+ - focal
+ - bionic
+subordinate: true
+requires:
+  extension:
+    interface: jenkins-extension
+    scope: container
+
diff --git a/reactive/scalebot-jenkins.py b/reactive/scalebot-jenkins.py
new file mode 100644
index 0000000..9a8a2b6
--- /dev/null
+++ b/reactive/scalebot-jenkins.py
@@ -0,0 +1,550 @@
+import os
+import shutil
+import stat
+import subprocess
+import tempfile
+import requests
+import jenkins
+import io
+import yaml
+
+from subprocess import CalledProcessError
+from charmhelpers.core import hookenv
+from charmhelpers.core.host import service_restart
+from charmhelpers.core.hookenv import (
+    log,
+    status_set,
+    ERROR,
+    WARNING,
+    DEBUG,
+)
+from charms.reactive import (
+    hook,
+    when,
+    when_all,
+    when_none,
+    when_not,
+    set_state,
+    get_flags,
+    remove_state,
+)
+from charms.reactive.helpers import data_changed
+from charms.layer import scalebot_jenkins
+
+SCALEBOT_URL = "http://localhost:8080"
+SCALEBOT_HOME = "/srv/scalebot"
+SCALEBOT_REPO = os.path.join(SCALEBOT_HOME, "repo")
+PYTHONPATH = os.path.join(SCALEBOT_REPO, "lib")
+
+
+def _run_cmd_yaml(args):
+    """Runs a command, assuming it will send some valid YAML data to stdout,
+    and parses its output. All the output will be loaded in memory.
+    """
+
+    proc = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    try:
+        rstdout, rstderr = proc.communicate(timeout=15)
+    except subprocess.TimeoutExpired as exc:
+        proc.kill()
+        raise exc
+    else:
+        if proc.returncode != 0:
+            raise Exception("Command error: " + str(rstderr))
+        return yaml.load(io.BytesIO(rstdout), Loader=yaml.SafeLoader)
+
+
+@when_not("scalebot.jjb.workaround.installed")
+def scalebot_install_jjb_workaround():
+    cfgfile = "/etc/default/jenkins"
+    tmpfile = "%s.scalebot.tmp" % (cfgfile)
+
+    snippet = [
+        "# ScaleBot Workaround for https://storyboard.openstack.org/#!/story/2006489\n",
+        'JAVA_ARGS+=" -Dhudson.security.csrf.DefaultCrumbIssuer.EXCLUDE_SESSION_ID=true"\n',
+    ]
+
+    with open(cfgfile, "r") as f:
+        cfg = f.readlines()
+    with open(tmpfile, "w+") as f:
+        for line in cfg:
+            if line in snippet:
+                continue
+            f.write("%s" % (line))
+        f.writelines(snippet)
+    shutil.copymode(cfgfile, tmpfile)
+    os.rename(tmpfile, cfgfile)
+
+    # Do a full service restart for Jenkins. We can't do a simple safe
+    # restart yet because we need this workaround to connect to the API.
+    # We only can do this on deployment, otherwise jobs can be disrupted.
+    service_restart("jenkins")
+
+    set_state("scalebot.jjb.workaround.installed")
+
+
+@when_all("scalebot.go", "extension.available")
+def configure_jenkins_jobbuilder(extension):
+    remove_state("scalebot.configured.jobbuilder")
+    status_set("maintenance", "Configuring JobBuilder")
+    if not os.path.isdir("/etc/jenkins_jobs"):
+        os.mkdir("/etc/jenkins_jobs")
+
+    jj_ini = "/etc/jenkins_jobs/jenkins_jobs.ini"
+    if os.path.isfile(jj_ini):
+        os.remove(jj_ini)
+
+    config = extension.get_connection_info()
+    username = config["admin_username"]
+    password = config["admin_password"]
+
+    jenkinsini = open(jj_ini, "w")
+    jenkinsini.write(
+        """[jenkins]\nuser=%s\n
+password=%s
+url=%s
+"""
+        % (username, password, SCALEBOT_URL)
+    )
+    jenkinsini.close()
+
+    status_set(
+        "maintenance", "Configured JobBuilder. Waiting for Scalebot Juju controller."
+    )
+
+    log(scalebot_jenkins.if_controller(), DEBUG)
+
+    if scalebot_jenkins.if_controller():
+        status_set(
+            "maintenance", "Configured JobBuilder and found Scalebot Juju controllers."
+        )
+        set_state("scalebot.configured.jobbuilder")
+        log("scalebot.configured.jobbuilder is set.")
+
+        flags = get_flags()
+        log(f"current flags: {flags}", DEBUG)
+    else:
+        status_set(
+            "waiting", "Configured JobBuilder. Waiting for Scalebot Juju controller."
+        )
+
+
+@when("extension.changed")
+def unconfigure_jobbuilder():
+    remove_state("scalebot.configured.jobbuilder")
+
+
+@hook("config-changed")
+def configure_scalebot_env():
+    config = hookenv.config()
+    jobset = config.get("scalebot_jobset")
+    sbprofile = "/etc/profile.d/scalebot.sh"
+
+    with open(sbprofile, "w") as f:
+        f.write("#Export Scalebot required paths\n")
+        f.write("export PYTHONPATH=%s\n" % (PYTHONPATH))
+        f.write('export SCALEBOT_HOME="%s"\n' % (SCALEBOT_HOME))
+        f.write('export SCALEBOT_REPO="%s"\n' % (SCALEBOT_REPO))
+        f.write('export SCALEBOT_JOBSET="%s"\n' % (jobset))
+        prod = int(config.get("scalebot_production"))
+        f.write("export SCALEBOT_PRODUCTION=%d\n" % (prod))
+
+    # We need to restart to get this environment.
+    remove_state("scalebot.configured.env")
+    set_state("scalebot.written.env")
+
+
+def scalebot_link_files_into_home():
+    # Link files used outside of charm context into $SCALEBOT_HOME
+    src = os.path.join(hookenv.charm_dir(), "bin")
+    dest = os.path.join(SCALEBOT_HOME, "bin")
+    if os.path.islink(dest):
+        os.unlink(dest)
+    os.symlink(src, dest)
+
+
+def scalebot_git_config_check():
+    required_configs = ["scalebot_config_repo", "scalebot_config_branch"]
+    missing_configs = []
+    config = hookenv.config()
+    for c in required_configs:
+        if not config.get(c):
+            missing_configs.append(c)
+    if len(missing_configs) > 0:
+        status_set(
+            "blocked",
+            "Unable to continue due to missing configs: %s"
+            % ", ".join(missing_configs),
+        )
+        return False
+    return True
+
+
+@when_all("scalebot.go", "apt.installed.git")
+def scalebot_git_clone():
+    if not scalebot_git_config_check():
+        return
+
+    config = hookenv.config()
+    repo = config.get("scalebot_config_repo")
+    branch = config.get("scalebot_config_branch")
+    ssh_key = config.get("scalebot_private_ssh_key")
+
+    if not data_changed("scalebot.git.config", [repo, branch, ssh_key]):
+        log("Repo config has not changed, nothing to do.")
+        return True
+
+    log("Scalebot repo config has changed, (re-)cloning.")
+    remove_state("scalebot.configured.repo")
+    status_set("maintenance", "Cloning Scalebot repository")
+
+    try:
+        shutil.rmtree(SCALEBOT_HOME)
+    except FileNotFoundError:
+        pass
+
+    os.makedirs(SCALEBOT_HOME)
+    shutil.chown(SCALEBOT_HOME, user="jenkins", group="jenkins")
+    scalebot_link_files_into_home()
+
+    known_hosts = os.path.join(hookenv.charm_dir(), "files", "known_hosts.lp")
+
+    git_ssh = ""
+    keypath = os.path.join(SCALEBOT_HOME, "key")
+    if ssh_key:
+        log("Writing ssh key to %s" % (keypath))
+        with os.fdopen(
+            os.open(keypath, os.O_WRONLY | os.O_CREAT, mode=0o600), "w"
+        ) as keyfile:
+            shutil.chown(keypath, user="jenkins", group="jenkins")
+            keyfile.write(ssh_key)
+        ssh_cmd = "/usr/bin/ssh -v -i %s -o UserKnownHostsFile=%s" % (
+            keypath,
+            known_hosts,
+        )
+        git_ssh = '-c core.sshCommand="%s"' % (ssh_cmd)
+    else:
+        try:
+            os.unlink(keypath)
+        except FileNotFoundError:
+            pass
+
+    git_cmd = "/usr/bin/git clone %s -b %s %s %s" % (
+        git_ssh,
+        branch,
+        repo,
+        os.path.join(SCALEBOT_HOME, "repo"),
+    )
+
+    su_git_cmd = ["/bin/su", "-", "jenkins", "-c", git_cmd]
+    log("Beginning git clone")
+    log(su_git_cmd)
+    subprocess.run(su_git_cmd, check=True)
+
+    set_state("scalebot.configured.repo")
+    remove_state("scalebot.repo.init")
+
+
+@when_not("scalebot.repo.init", "scalebot.stop")
+def scalebot_repo_init():
+    init_path = os.path.join(SCALEBOT_REPO, "scalebot.d", "init")
+    if os.path.exists(init_path):
+        status_set("maintenance", "Executing %s" % (init_path))
+        subprocess.check_call(init_path)
+    set_state("scalebot.repo.init")
+
+
+@hook("config-changed")
+def scalebot_git_reconfig():
+    config = hookenv.config()
+    required_configs = [
+        "scalebot_private_ssh_key",
+        "scalebot_config_repo",
+        "scalebot_config_branch",
+    ]
+    if any(config.changed(key) for key in required_configs):
+        scalebot_git_clone()
+
+
+@when_all(
+    "scalebot.configured.repo",
+    "scalebot.configured.jobbuilder",
+    "scalebot.configured.juju",
+    "scalebot.go",
+)
+def scalebot_ready():
+    status_set("active", "ScaleBot is Ready")
+    log("Congratulations!! Scalebot is ready. Go testing go!!")
+
+
+@when_all(
+    "scalebot.configured.repo",
+    "scalebot.configured.jobbuilder",
+    "scalebot.configured.env",
+    "scalebot.go",
+    "scalebot.jjb.workaround.installed",
+    "scalebot.jenkins-http-ready",
+)
+def importjenkins():
+    cmd = os.path.join(hookenv.charm_dir(), "actions", "refreshjobs")
+    log(cmd, DEBUG)
+    try:
+        subprocess.run(
+            [cmd], check=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+        )
+    except CalledProcessError as e:
+        msg_stdout = e.stdout.decode()
+        log(msg_stdout, level=ERROR)
+
+
+@hook("config-changed")
+def scalebot_juju_reconfig():
+    config = hookenv.config()
+    required_configs = [
+        "scalebot_juju_clouds",
+        "scalebot_juju_credentials",
+        "scalebot_juju_model_defaults",
+    ]
+    if any(config.changed(key) for key in required_configs):
+        scalebot_configure_juju()
+
+
+@when_all("apt.installed.juju", "scalebot.go")
+def scalebot_configure_juju():
+    config = hookenv.config()
+
+    clouds = config.get("scalebot_juju_clouds")
+    creds = config.get("scalebot_juju_credentials")
+    modeldefaults = config.get("scalebot_juju_model_defaults")
+    bootstrapconstraints = config.get("scalebot_juju_bootstrap_constraints")
+
+    if (
+        not data_changed("scalebot.juju.config", [clouds, creds])
+        and scalebot_jenkins.if_controller()
+    ):
+        log(
+            "Juju controller is available and its config has not changed, nothing to do."
+        )
+
+        return True
+
+    if data_changed("scalebot.juju.config", [clouds, creds]):
+        log("Juju config has changed, (re-)bootstrapping.")
+    elif not scalebot_jenkins.if_controller():
+        log("Scalebot Juju controller is not ready, (re-)bootstrapping.")
+    else:
+        log("Scalebot Juju controller is going to be deployed, (re-)bootstrapping.")
+
+    remove_state("scalebot.configured.juju")
+    status_set("maintenance", "Bootstrapping Juju controller")
+
+    # In case we're reconfiguring, destroy any pre-existing
+    # controller before re-bootstrapping
+    subprocess.run(
+        ["/bin/su", "-", "jenkins", "-c", "juju destroy-controller jenkins -y"],
+        check=False,
+    )
+
+    scalebot_configure_juju_for_lab(clouds, creds, modeldefaults, bootstrapconstraints)
+
+
+def scalebot_configure_juju_for_lab(clouds, creds, modeldefaults, bootstrapconstraints):
+    if clouds:
+        cloudsfile = tempfile.NamedTemporaryFile(mode="w", delete=False)
+        cloudsfile.write(clouds)
+        # Make it readable by user jenkins
+        os.chmod(
+            cloudsfile.name, stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH
+        )
+        cloudsfile.close()
+        curr_clouds = _run_cmd_yaml(
+            ["/bin/su", "-", "jenkins", "-c", "juju list-clouds --format yaml"]
+        )
+        if curr_clouds and ("scalebot" in curr_clouds):
+            command = "juju update-cloud --client scalebot -f %s" % (cloudsfile.name)
+        else:
+            command = "juju add-cloud --client scalebot %s" % (cloudsfile.name)
+        subprocess.run(["/bin/su", "-", "jenkins", "-c", command], check=True)
+        os.unlink(cloudsfile.name)
+
+    if creds:
+        credsfile = tempfile.NamedTemporaryFile(mode="w", delete=False)
+        credsfile.write(creds)
+        # Make it readable by user jenkins
+        os.chmod(
+            credsfile.name, stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH
+        )
+        credsfile.close()
+        curr_creds = _run_cmd_yaml(
+            ["/bin/su", "-", "jenkins", "-c", "juju show-credentials --format yaml"]
+        )
+        if (
+            curr_creds
+            and ("client-credentials" in curr_creds)
+            and ("scalebot" in curr_creds["client-credentials"])
+        ):
+            command = "juju update-credential --client -f %s scalebot" % (
+                credsfile.name
+            )
+        else:
+            command = "juju add-credential --client -f %s scalebot" % (credsfile.name)
+        subprocess.run(["/bin/su", "-", "jenkins", "-c", command], check=True)
+        os.unlink(credsfile.name)
+
+    modeldefaultsparam = ""
+    if modeldefaults:
+        modeldefaultsfile = tempfile.NamedTemporaryFile(mode="w", delete=False)
+        modeldefaultsfile.write(modeldefaults)
+        os.chmod(
+            modeldefaultsfile.name,
+            stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH,
+        )
+        modeldefaultsfile.close()
+        modeldefaultsparam = " --model-default %s" % (modeldefaultsfile.name)
+
+    bconstraints = ""
+    if bootstrapconstraints:
+        bconstraints = ' --bootstrap-constraints "%s"' % (bootstrapconstraints)
+
+    try:
+        subprocess.run(
+            [
+                "/bin/su",
+                "-",
+                "jenkins",
+                "-c",
+                "juju bootstrap --debug scalebot jenkins"
+                + modeldefaultsparam
+                + bconstraints,
+            ],
+            check=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.STDOUT,
+        )
+
+        if modeldefaults:
+            log(f"Removing {modeldefaultsfile.name}", DEBUG)
+            os.unlink(modeldefaultsfile.name)
+            log(f"Removed {modeldefaultsfile.name}", DEBUG)
+
+        log(
+            "Bootstrapping of Scalebot Juju controller looks good."
+        )
+
+    except CalledProcessError as e:
+        msg_stdout = e.stdout.decode()
+        log(msg_stdout, level=ERROR)
+
+        msg_guess = scalebot_jenkins.get_bootstrap_failure_hint(msg_stdout)
+        msg = f"Fail to bootstrap a controller. It may be because: {msg_guess}"
+        log(msg, level=ERROR)
+
+        # fail to bootstrap a controller is devastating
+        # by setting status as "blocked" to help stop the deployment process
+        status_set("blocked", msg)
+
+    if scalebot_jenkins.if_controller():
+        set_state("scalebot.configured.juju")
+
+
+@when_not("scalebot.stop")
+def scalebot_start():
+    set_state("scalebot.go")
+
+
+@hook("stop")
+def scalebot_stop():
+    set_state("scalebot.stop")
+    remove_state("scalebot.go")
+    log("Stopping Scalebot.")
+    status_set("maintenance", "Stopping ScaleBot")
+
+
+@when("scalebot.stop")
+def scalebot_juju_destroy_controller():
+    status_set("maintenance", "Destroying Juju controller")
+    subprocess.run(
+        [
+            "/bin/su",
+            "-",
+            "jenkins",
+            "-c",
+            "juju destroy-controller --destroy-all-models -y jenkins",
+        ]
+    )
+
+
+@when_not("scalebot.jenkins-http-ready")
+def jenkins_http_ready():
+    rc = 500
+    try:
+        rc = requests.get(SCALEBOT_URL, timeout=5).status_code
+    except (
+        requests.HTTPError,
+        requests.exceptions.Timeout,
+        requests.exceptions.ConnectionError,
+    ):
+        pass
+
+    # Valid status codes (we can get redirects, login forms, etc.)
+    if rc in (200, 300, 301, 302, 403):
+        set_state("scalebot.jenkins-http-ready")
+
+
+# Count the number of jobs currently running on Jenkins and lock it for new
+# jobs if zero. This is required to prevent a race condition if new jobs are
+# started between running this script and restarting the server.
+
+JK_RESTART_CHECK_SCRIPT = """
+import jenkins.model.*
+import hudson.model.*
+
+lst = Jenkins.instance.getView('All').getBuilds().findAll() {
+    it.getResult().equals(null)
+}
+if (lst.size() == 0) {
+    Jenkins.instance.doQuietDown();
+}
+println(lst.size());
+"""
+
+
+@when_none("scalebot.configured.env", "scalebot.stop")
+@when_all(
+    "scalebot.go",
+    "scalebot.jenkins-http-ready",
+    "scalebot.jjb.workaround.installed",
+    "scalebot.written.env",
+)
+@when_all("extension.available", "scalebot.go")
+def safe_jenkins_service_restart(extension):
+    """Restart Jenkins service if no jobs are running.
+
+    Environment variables required by some jobs will not be available until
+    we do a full service restart (i.e. finishing Java process and starting
+    it again instead of just reloading Jenkins, as done by safe-restart).
+
+    Tests can be lost if we just restart so we need to get the number of
+    running jobs (using the Rest API), restart the service and ensure that
+    Jenkins is back online before proceeding.
+    """
+
+    config = extension.get_connection_info()
+    try:
+        server = jenkins.Jenkins(
+            SCALEBOT_URL,
+            username=config["admin_username"],
+            password=config["admin_password"],
+            timeout=10,
+        )
+        ret = server.run_script(JK_RESTART_CHECK_SCRIPT)
+        if ret and int(ret) == 0:
+            log("Restarting Jenkins service")
+            service_restart("jenkins")
+            remove_state("scalebot.jenkins-http-ready")
+            set_state("scalebot.configured.env")
+        else:
+            log("Jenkins is running some jobs, will restart later")
+
+    except jenkins.JenkinsException as exc:
+        log("Jenkins API exception:" + str(exc), level=WARNING)