Merge ~travisghansen/maas:master into maas:master

[Travis Glenn Hansen]

Webhook Power Driver

This is a simple power driver that POSTs to an http(s) endpoint of the user's choosing. The endpoint receives a json payload which looks like:

{"action":"power_on","system_id":"<id>"}
OR
{"action":"power_off","system_id":"<id>"}
OR
{"action":"power_query","system_id":"<id>"}

The "power_query" POST expects a response of the following:
{"power_state": "on|off|unknown"}

SSL validation can be disabled to work with self-signed certs.

Authentication can be done by either setting a username/password for basic auth OR bearer auth by setting a token. Preference if given to the token if both token and username/password are set.

Of note, the URL can include query params.

I've tested this out by hacking the files into a snap and prototyping. In my simple use-case I'm configuring the endpoint to point to a Node-RED instance I have and then I wire everything up to control some crude/cheap iot power plugs (gosund devices with tasmota firmware flashed).

I'm completely new to maas (and python is not my strength) so I could be missing something entirely here so any tips are appreciated :)

[MAAS Lander (maas-lander)]

UNIT TESTS
-b master lp:~travisghansen/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/maas/job/branch-tester/8968/console
COMMIT: 64e77f06124644a038f2705826989b41be66580d

review: Needs Fixing

[MAAS Lander (maas-lander)]

UNIT TESTS
-b master lp:~travisghansen/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/maas/job/branch-tester/8969/console
COMMIT: 23f40223494313dc3a298987b622604ef0387e98

review: Needs Fixing

[Adam Collard (adam-collard)]

Thanks for the contribution!

I haven't looked at the code per se but wanted to share output from the CI job:

ERROR: /run/build/maas/src/provisioningserver/drivers/power/webhook.py Imports are incorrectly sorted.
 --- /run/build/maas/src/provisioningserver/drivers/power/webhook.py:before 2021-01-04 19:19:49.596325
 +++ /run/build/maas/src/provisioningserver/drivers/power/webhook.py:after 2021-01-04 19:22:47.812771
 @@ -11,13 +11,13 @@
  import urllib.request

  from provisioningserver.drivers import make_ip_extractor, make_setting_field
 -from provisioningserver.logger import get_maas_logger
  from provisioningserver.drivers.power import (
      PowerActionError,
      PowerConnError,
      PowerDriver,
      PowerFatalError,
  )
 +from provisioningserver.logger import get_maas_logger

  maaslog = get_maas_logger("drivers.power.webhook")

 make: *** [Makefile:213: lint-py] Error 1

[Travis Glenn Hansen (travisghansen)]

Fun. Do you have a standard linter/code formatter I can use to sort it out (pun intended).

[Adam Collard (adam-collard)]

We do indeed!

`make format-py` is the easiest way to get it, else `bin/isort src/provisioningserver/drivers/power/webhook.py` for a more targetted fix.

In terms of formatting, we use https://github.com/psf/black

[Travis Glenn Hansen (travisghansen)]

I tried the make format-py and it updated basically everything, so I went with the isort command :)

[MAAS Lander (maas-lander)]

UNIT TESTS
-b master lp:~travisghansen/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/maas/job/branch-tester/8992/console
COMMIT: a90321b475d79e7812d7a516e142bb3b7ddad855

review: Needs Fixing

[Lee Trager (ltrager)]

Thanks for the patch! Couple of things

1. I think this should have an optional "id" field. This will allow one webhook service to handle multiple machines. For example if you create a wake on LAN webhook the id would be the MAC address.
2. We require unit tests for all branches to land.
3. MAAS uses Twisted[1] for HTTP calls. This allows MAAS to perform other tasks while waiting for the response.
4. See other comments below

If Twisted looks to complicated you can skip 2 and 3 and I'll Twistify it and add unit tests in a follow up branch.

[1] https://twistedmatrix.com/documents/current/web/howto/client.html

review: Needs Fixing

[MAAS Lander (maas-lander)]

UNIT TESTS
-b master lp:~travisghansen/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/maas/job/branch-tester/8993/console
COMMIT: 13a6cadde0c989e676cc98f6d25264d1885b6509

review: Needs Fixing

[Travis Glenn Hansen (travisghansen)]

Thanks for the review!

1. I considered this but figured it could be managed via the GET/query params within the URL itself.
2. I'll see what I can put together for that. Trying to setup a development environment has been...challenging. Even worse is trying to inject the code into a snap :( so testing for me generally has been quite challenging.
3. I based it on the code from recs.py which is using urllib

It would be great if you could twistify it as I'm sure I'll mess something up with the async/await stuff especially given how hard it is for me to *really* test changes. Is there a doc somewhere with instructions for running locally and what services are needed etc? Preferably something that is OS independent since I don't run ubuntu on my workstation.

[Travis Glenn Hansen (travisghansen)]

Any further guidance on this? I'd hate to see bitrot :)

[Lee Trager (ltrager)]

I created a follow on branch[1] which uses Twisted for the web calls and makes this a bit more extendable so it will work with multiple services. If that works I can land both.

[1] https://code.launchpad.net/~ltrager/maas/+git/maas/+merge/396049

[Travis Glenn Hansen (travisghansen)]

Awesome! I've commented over there to continue the conversation!

[Lee Trager (ltrager)]

review: Approve

[MAAS Lander (maas-lander)]

LANDING
-b master lp:~travisghansen/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: FAILED BUILD
LOG: http://maas-ci.internal:8080/job/maas/job/branch-tester/9048/consoleText

diff --git a/src/provisioningserver/drivers/power/registry.py b/src/provisioningserver/drivers/power/registry.py
index 93db204..91baa68 100644
--- a/src/provisioningserver/drivers/power/registry.py
+++ b/src/provisioningserver/drivers/power/registry.py
@@ -25,6 +25,7 @@ from provisioningserver.drivers.power.redfish import RedfishPowerDriver
 from provisioningserver.drivers.power.seamicro import SeaMicroPowerDriver
 from provisioningserver.drivers.power.ucsm import UCSMPowerDriver
 from provisioningserver.drivers.power.vmware import VMwarePowerDriver
+from provisioningserver.drivers.power.webhook import WebhookPowerDriver
 from provisioningserver.drivers.power.wedge import WedgePowerDriver
 from provisioningserver.utils.registry import Registry
 
@@ -65,6 +66,7 @@ power_drivers = [
     SeaMicroPowerDriver(),
     UCSMPowerDriver(),
     VMwarePowerDriver(),
+    WebhookPowerDriver(),
     WedgePowerDriver(),
 ]
 for driver in power_drivers:
diff --git a/src/provisioningserver/drivers/power/webhook.py b/src/provisioningserver/drivers/power/webhook.py
new file mode 100644
index 0000000..ad7e5b6
--- /dev/null
+++ b/src/provisioningserver/drivers/power/webhook.py
@@ -0,0 +1,121 @@
+# Copyright 2016 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Generic Webhook Power Driver."""
+
+import base64
+import json
+import ssl
+import urllib.error
+import urllib.parse
+import urllib.request
+
+from provisioningserver.drivers import make_ip_extractor, make_setting_field
+from provisioningserver.drivers.power import (
+    PowerActionError,
+    PowerConnError,
+    PowerDriver,
+    PowerFatalError,
+)
+
+SSL_INSECURE_YES="y"
+SSL_INSECURE_NO="n"
+
+SSL_INSECURE_CHOICES = [
+    [SSL_INSECURE_NO, "No"],
+    [SSL_INSECURE_YES, "Yes"]]
+
+
+class WebhookPowerDriver(PowerDriver):
+
+    name = "webhook"
+    chassis = False
+    can_probe = False
+    description = "Webhook"
+    settings = [
+        make_setting_field("power_url", "URL", required=True),
+        make_setting_field("power_insecure_ssl", "Allow insecure SSL connections", field_type='choice', required=True,
+        choices=SSL_INSECURE_CHOICES, default=SSL_INSECURE_NO),
+        make_setting_field("power_user", "Power user"),
+        make_setting_field(
+            "power_pass", "Power password", field_type="password"
+        ),
+        make_setting_field(
+            "power_token", "Power token", field_type="password"
+        ),
+    ]
+
+    #ip_extractor = make_ip_extractor("power_address")
+    ip_extractor = None
+
+    def post(self, context, params={}):
+        """Dispatch a POST request to webhook."""
+        url = context.get("power_url")
+        data = json.dumps(params).encode('utf-8')
+        req = urllib.request.Request(url, data, method="POST")
+
+        if context.get("power_token") and len(context.get("power_token")) > 0:
+          authString = "{token}".format(token = context.get("power_token")).encode("utf-8")
+          authString = str(authString, "utf-8")
+          req.add_header("Authorization", "Bearer %s" % authString)
+        elif context.get("power_user") and len(context.get("power_user")) > 0 and context.get("power_pass") and len(context.get("power_pass")) > 0:
+          authString = base64.urlsafe_b64encode("{username}:{password}".format(username = context.get("power_user"), password = context.get("power_pass")).encode("utf-8"))
+          authString = str(authString, "utf-8")
+          req.add_header("Authorization", "Basic %s" % authString)
+
+        req.add_header('content-type', 'application/json')
+
+        myssl = None
+        if context.get("power_insecure_ssl") and context.get("power_insecure_ssl") == SSL_INSECURE_YES:
+            myssl = ssl.create_default_context();
+            myssl.check_hostname=False
+            myssl.verify_mode=ssl.CERT_NONE
+        try:
+            res = urllib.request.urlopen(req, context=myssl)
+        except urllib.error.HTTPError as e:
+            raise PowerConnError(
+                "Could not make proper connection to Webhook."
+                " HTTP error code: %s" % e.code
+            )
+        except urllib.error.URLError as e:
+            raise PowerConnError(
+                "Could not make proper connection to Webhook."
+                " Server could not be reached: %s" % e.reason
+            )
+        else:
+            body = res.read();
+            if not body or len(body) == 0:
+                return dict()
+            return json.loads(body)
+
+    def detect_missing_packages(self):
+        # uses pure-python http client - nothing to look for!
+        return []
+
+    def power_on(self, system_id, context):
+        """Power on webhook."""
+        try:
+            self.post(context, {"action": "power_on", "system_id": system_id})
+        except PowerConnError:
+            raise PowerActionError("Webhook Driver unable to power on")
+
+    def power_off(self, system_id, context):
+        """Power off webhook."""
+        try:
+            self.post(context, {"action": "power_off", "system_id": system_id})
+        except PowerConnError:
+            raise PowerActionError("Webhook Driver unable to power off")
+
+    def power_query(self, system_id, context):
+        """Power query webhook."""
+        try:
+            res = self.post(context, {"action": "power_query", "system_id": system_id})
+        except PowerConnError:
+            raise PowerActionError("Webhook Driver unable to power query")
+        else:
+            if res["power_state"] == "off":
+                return "off"
+            elif res["power_state"] == "on":
+                return "on"
+            else:
+                return "unknown"