Merge ~peter-sabaini/charm-sudo-pair:feature/add-pd-logging into charm-sudo-pair:master

[Peter Sabaini]

Pagerduty alerting, action logging

Ability to have pagerduty alerts triggered on auto-approve. Revamp logging, add logging for remove-sudopair action. Use https proxy to contact the PD events endpoint if set in model config.

[Giuseppe Petralia (peppepetra)]

Small comment on the pagerduty_proxy that may be removed in favor of the juju model-config if any.

Comments in line.

Other than that looks good to me.

review: Needs Fixing

[Paul Goins (vultaire)]

I agree with Giuseppe. I think we should pull the proxy from the env's JUJU_CHARM_HTTPS_PROXY variable to avoid proliferating proxy settings. Other than that I'm +1.

review: Needs Fixing

[Peter Sabaini (peter-sabaini)]

Thanks -- proxy config updated as requested. I've resubmitted the branch as there was some code reorg interim.

[Celia Wang (ziyiwang)]

LGTM.

[Paul Goins (vultaire)]

Sorry for the delay here - It looks like this is close to if not identical to https://code.launchpad.net/~peter-sabaini/charm-sudo-pair/+git/sudo-pair-charm/+merge/377884, except that it might be based off newer code. I don't see any changes re: proxy variable use; this code is still using pagerduty_proxy rather than e.g. JUJU_CHARM_HTTPS_PROXY.

review: Needs Fixing

[Peter Sabaini (peter-sabaini)]

Apologies, appears I superseded with the wrong branch here. Trying this again

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[Tom Haddon (mthaddon)]

Abstaining, just claimed the review to take it off the canonical-is-reviewers dashboard, mergebot has been switched to add bootstack-reviewers for future MPs.

review: Abstain

[Paul Goins (vultaire)]

Looks good to me!

review: Approve

[James Troup (elmo)]

Couple of comments inline; I consider the config file permissions blocking unless I'm wrong about the perms?

review: Needs Fixing

7b10f8a... by Peter Sabaini on 2021-12-28

Address review comments

[Peter Sabaini (peter-sabaini)]

Should be better now, please take a look

Unmerged commits

7b10f8a... by Peter Sabaini on 2021-12-28

Address review comments

3681b10... by Peter Sabaini on 2020-11-19

Pagerduty alerting, action logging

Ability to have pagerduty alerts triggered on auto-approve. Revamp
logging, add logging for remove-sudopair action. Use https proxy to
contact the PD events endpoint if set in model config.

diff --git a/src/actions/actions.py b/src/actions/actions.py
index 9ff1ec5..65c5a13 100755
--- a/src/actions/actions.py
+++ b/src/actions/actions.py
@@ -14,7 +14,10 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import logging
+import logging.handlers
 import os
+import subprocess
 import sys
 
 from charmhelpers.core.hookenv import action_fail, action_set, status_set
@@ -25,11 +28,21 @@ sys.path.append("lib")
 import libsudopair  # NOQA
 
 
+logger = logging.getLogger("sudopair")
+logger.setLevel(logging.INFO)
+handler = logging.handlers.SysLogHandler(
+    address="/dev/log", facility=logging.handlers.SysLogHandler.LOG_AUTH
+)
+logger.addHandler(handler)
+
+
 def remove():
     """Remove sudo-pair config and binaries."""
     sph = libsudopair.SudoPairHelper()
     sph.deconfigure()
     msg = "Successfully removed sudo-pair config and binaries"
+    logger.warning(msg)
+    subprocess.run(["/usr/local/bin/pagerdutyevent", msg])
     action_set({"message": msg})
     status_set("active", msg)
 
diff --git a/src/config.yaml b/src/config.yaml
index 795269b..6089ac9 100644
--- a/src/config.yaml
+++ b/src/config.yaml
@@ -19,3 +19,11 @@ options:
     type: boolean
     default: true
     description: "If true, auto approval is permitted."
+  pagerduty_key:
+    type: string
+    default: ''
+    description: "If set, a pagerduty event will be triggered upon auto-approving"
+  pagerduty_context:
+    type: string
+    default: ''
+    description: "Prefix to add to pagerduty events"
diff --git a/src/files/pagerdutyevent.py b/src/files/pagerdutyevent.py
new file mode 100644
index 0000000..f33be42
--- /dev/null
+++ b/src/files/pagerdutyevent.py
@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+"""Send events to pagerduty."""
+
+import argparse
+import configparser
+import json
+import logging
+import logging.handlers
+
+import requests
+
+logger = logging.getLogger("sudopair")
+logger.setLevel(logging.INFO)
+handler = logging.handlers.SysLogHandler(
+    address="/dev/log", facility=logging.handlers.SysLogHandler.LOG_AUTH
+)
+logger.addHandler(handler)
+
+
+def args():
+    """Get cli args."""
+    parser = argparse.ArgumentParser()
+    parser.add_argument("summary")
+    return parser.parse_args()
+
+
+def trigger_incident(pd_key, summary, source, https_proxy=None):
+    """Send an event to the PD events endpoint."""
+    header = {"Content-Type": "application/json"}
+
+    payload = {
+        "routing_key": pd_key,
+        "event_action": "trigger",
+        "payload": {"summary": summary, "source": source, "severity": "info"},
+    }
+    if https_proxy:
+        proxies = {"https": https_proxy}
+    else:
+        proxies = None
+    response = requests.post(
+        "https://events.pagerduty.com/v2/enqueue",
+        data=json.dumps(payload),
+        proxies=proxies,
+        headers=header,
+    )
+
+    if response.json()["status"] == "success":
+        logger.info("Triggered alert with key {}".format(response.json()["dedup_key"]))
+    else:
+        logger.warning("Failed to send pagerduty alert: {}".format(response.text))
+
+
+def get_conf():
+    """Return config options from file."""
+    config = configparser.ConfigParser()
+    config.read("/etc/sudo_pair.pagerduty.ini")
+    sec = config["DEFAULT"]
+    return sec["pagerduty_key"], sec["pagerduty_source"], sec.get("https_proxy")
+
+
+def main():
+    """Run the script."""
+    pd_key, source, https_proxy = get_conf()
+    trigger_incident(pd_key, args().summary, source, https_proxy)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/src/lib/libsudopair.py b/src/lib/libsudopair.py
index 2f7c45b..24dfcf2 100644
--- a/src/lib/libsudopair.py
+++ b/src/lib/libsudopair.py
@@ -67,6 +67,9 @@ class SudoPairHelper(object):
         self.sudoers_perms = 0o440
         self.sudo_conf_perms = 0o644
         self.sudo_approve_perms = 0o755
+        self.pagerduty_conf_path = "/etc/sudo_pair.pagerduty.ini"
+        self.pagerduty_path = "/usr/local/bin/pagerdutyevent"
+        self.pagerduty_perms = 0o755
 
     def get_config(self):
         """Return config as a dict."""
@@ -81,8 +84,11 @@ class SudoPairHelper(object):
             "gids_exempted": group_names_to_group_ids(
                 self.charm_config["groups_exempted"]
             ),
+            "pagerduty_key": self.charm_config.get("pagerduty_key"),
         }
-
+        proxy_settings = hookenv.env_proxy_settings()
+        if proxy_settings:
+            config["https_proxy"] = proxy_settings.get("https_proxy")
         config.update(self.charm_config)
         return config
 
@@ -90,6 +96,13 @@ class SudoPairHelper(object):
         """Update configuration."""
         self.charm_config = charm_config
 
+    def copy_pagerduty(self):
+        """Copy PD script in place."""
+        pd_file = os.path.join(hookenv.charm_dir(), "files", "pagerdutyevent")
+        copy_file(
+            pd_file, self.pagerduty_path, self.owner, self.group, self.pagerduty_perms
+        )
+
     def render_sudo_conf(self):
         """Render sudo.conf file."""
         return templating.render(
@@ -184,6 +197,25 @@ class SudoPairHelper(object):
             )
         return None
 
+    def render_pagerduty_conf(self):
+        """Create the PD script configuration."""
+        pd_source = "{}-{}".format(
+            self.charm_config.get("pagerduty_context", "juju"), hookenv.principal_unit()
+        )
+        cfg = self.get_config()
+        cfg["pagerduty_source"] = pd_source
+        hookenv.log(
+            "Rendering pagerduty configuration to {}".format(self.pagerduty_conf_path)
+        )
+        return templating.render(
+            "sudo_pair.pagerduty.tmpl",
+            self.pagerduty_conf_path,
+            cfg,
+            owner=self.owner,
+            group=self.group,
+            perms=0o400,
+        )
+
     def deconfigure(self):
         """Remove sudo-pair configuration."""
         paths = [
diff --git a/src/reactive/sudo_pair.py b/src/reactive/sudo_pair.py
index caab64f..392383b 100644
--- a/src/reactive/sudo_pair.py
+++ b/src/reactive/sudo_pair.py
@@ -32,6 +32,10 @@ def install_sudo_pair():
     # Add "Defaults log_output to /etc/sudoers
     sph.copy_sudoers()
 
+    # Add pagerduty script and config
+    sph.copy_pagerduty()
+    sph.render_pagerduty_conf()
+
     # If there are cmds to bypass sudo pairing create file unders sudoers.d
     sph.render_bypass_cmds()
 
diff --git a/src/templates/sudo_approve.tmpl b/src/templates/sudo_approve.tmpl
index 7164b5a..5ebace4 100755
--- a/src/templates/sudo_approve.tmpl
+++ b/src/templates/sudo_approve.tmpl
@@ -88,7 +88,7 @@ main() {
     declare -r username
 
     declare log_line
-    log_line="$(date "+[%b %d %H:%M:%S] WARNING: ${username} approved is own sudo session.")"
+    log_line="WARNING: ${username} approved own sudo session."
     declare -r log_line
 
     if [[ "${uid}" -eq "${ruid}" ]]; then
@@ -97,7 +97,10 @@ main() {
         exit 1
         {% else %}
         echo "You are approving your own session. The incident will be logged."
-        echo ${log_line} >> /var/log/sudo_pair.log
+        logger -p auth.warn $log_line
+        {% if pagerduty_key %}
+          /usr/local/bin/pagerdutyevent "$log_line"
+        {% endif %}
         {% endif %}
     fi
 
diff --git a/src/templates/sudo_pair.pagerduty.tmpl b/src/templates/sudo_pair.pagerduty.tmpl
new file mode 100644
index 0000000..2185cf9
--- /dev/null
+++ b/src/templates/sudo_pair.pagerduty.tmpl
@@ -0,0 +1,6 @@
+[DEFAULT]
+pagerduty_key: {{ pagerduty_key }}
+pagerduty_source: {{ pagerduty_source }}
+{% if https_proxy -%}
+https_proxy: {{ https_proxy }}
+{% endif %}
\ No newline at end of file
diff --git a/src/tests/unit/conftest.py b/src/tests/unit/conftest.py
index c1df1d0..bc65d15 100644
--- a/src/tests/unit/conftest.py
+++ b/src/tests/unit/conftest.py
@@ -16,7 +16,7 @@ def mock_hookenv_config(monkeypatch):
 
     def mock_config():
         cfg = {}
-        yml = yaml.load(open("./config.yaml"))
+        yml = yaml.safe_load(open("./config.yaml"))
 
         # Load all defaults
         for key, value in yml["options"].items():
@@ -54,4 +54,5 @@ def sph(mock_hookenv_config, mock_charm_dir, tmpdir):
     sph.sudoers_bypass_path = tmpdir.join(sph.sudoers_bypass_path)
     sph.socket_dir_perms = 0o775
     sph.sudo_conf_perms = 0o644
+    sph.pagerduty_path = tmpdir.join(sph.pagerduty_path)
     return sph
diff --git a/src/tests/unit/test_actions.py b/src/tests/unit/test_actions.py
index ddf6003..9c194f3 100644
--- a/src/tests/unit/test_actions.py
+++ b/src/tests/unit/test_actions.py
@@ -12,10 +12,12 @@ import actions  # NOQA
 @mock.patch("libsudopair.SudoPairHelper")
 @mock.patch("actions.action_set")
 @mock.patch("actions.status_set")
-def test_remove_action(status_set, action_set, sudo_pair_helper):
[e-mail]("actions.subprocess.run")
+def test_remove_action(subprocess_run, status_set, action_set, sudo_pair_helper):
     """Verify remove action."""
     actions.remove()
     msg = "Successfully removed sudo-pair config and binaries"
     action_set.assert_called_with({"message": msg})
     status_set.assert_called_with("active", msg)
     sudo_pair_helper().deconfigure.assert_called()
+    subprocess_run.assert_called_with(["/usr/local/bin/pagerdutyevent", msg])
diff --git a/src/tests/unit/test_libsudopair.py b/src/tests/unit/test_libsudopair.py
index 860d8db..8d730ad 100644
--- a/src/tests/unit/test_libsudopair.py
+++ b/src/tests/unit/test_libsudopair.py
@@ -144,7 +144,7 @@ class TestSudoPairHelper:
     def test_render_sudo_approve(self, sph, tmpdir):
         """Check that sudo_approve file is rendered correctly."""
         # Auto Approve true
-        expected_content = "echo ${log_line} >> /var/log/sudo_pair.log"
+        expected_content = "logger -p auth.warn $log_line"
         socket_dir = tmpdir.join("/var/run/sudo_pair")
         expected_content_socket_dir = 'declare -r SUDO_SOCKET_PATH="{}"'.format(
             socket_dir