Merge ~mitchburton/landscape-charm:merge-beta into landscape-charm:stable

[Mitch Burton]

Merge beta into stable

Follow up on https://code.launchpad.net/~mitchburton/landscape-charm/+git/landscape-charm/+merge/462351, with essentially the same changes.

Summary:
- License update to newer version of GPLv2
- Update Makefile and README
- Add Landscape PPA key and secret token support
- Add Grafana machine agent library
- Fix bug causing package installation to hang
- Clarify inline documentation
- Bootstrap account on schema migration
- Fix bug where haproxy fails if leader goes down due to missing backend service keys
- Ensure haproxy cert is properly encoded
- Add application dashboard relation joined handler for LMA support
- Only set website relations in haproxy for the leader
- Update haproxy config to use `http-request` for path replacement instead of regex
- Update haproxy config to use hashid backend if present

[Spencer Runde (spencerrunde)]

LGTM

review: Approve

diff --git a/LICENSE b/LICENSE
index 5b6e7c6..d159169 100644
--- a/LICENSE
+++ b/LICENSE
@@ -1,12 +1,12 @@
-		    GNU GENERAL PUBLIC LICENSE
-		       Version 2, June 1991
+                    GNU GENERAL PUBLIC LICENSE
+                       Version 2, June 1991
 
- Copyright (C) 1989, 1991 Free Software Foundation, Inc.
-                       59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+ Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
+ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
  Everyone is permitted to copy and distribute verbatim copies
  of this license document, but changing it is not allowed.
 
-			    Preamble
+                            Preamble
 
   The licenses for most software are designed to take away your
 freedom to share and change it.  By contrast, the GNU General Public
@@ -15,7 +15,7 @@ software--to make sure the software is free for all its users.  This
 General Public License applies to most of the Free Software
 Foundation's software and to any other program whose authors commit to
 using it.  (Some other Free Software Foundation software is covered by
-the GNU Library General Public License instead.)  You can apply it to
+the GNU Lesser General Public License instead.)  You can apply it to
 your programs, too.
 
   When we speak of free software, we are referring to freedom, not
@@ -55,8 +55,8 @@ patent must be licensed for everyone's free use or not licensed at all.
 
   The precise terms and conditions for copying, distribution and
 modification follow.
-

-		    GNU GENERAL PUBLIC LICENSE
+
+                    GNU GENERAL PUBLIC LICENSE
    TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
 
   0. This License applies to any program or other work which contains
@@ -110,7 +110,7 @@ above, provided that you also meet all of these conditions:
     License.  (Exception: if the Program itself is interactive but
     does not normally print such an announcement, your work based on
     the Program is not required to print an announcement.)
-

+
 These requirements apply to the modified work as a whole.  If
 identifiable sections of that work are not derived from the Program,
 and can be reasonably considered independent and separate works in
@@ -168,7 +168,7 @@ access to copy from a designated place, then offering equivalent
 access to copy the source code from the same place counts as
 distribution of the source code, even though third parties are not
 compelled to copy the source along with the object code.
-

+
   4. You may not copy, modify, sublicense, or distribute the Program
 except as expressly provided under this License.  Any attempt
 otherwise to copy, modify, sublicense or distribute the Program is
@@ -225,7 +225,7 @@ impose that choice.
 
 This section is intended to make thoroughly clear what is believed to
 be a consequence of the rest of this License.
-

+
   8. If the distribution and/or use of the Program is restricted in
 certain countries either by patents or by copyrighted interfaces, the
 original copyright holder who places the Program under this License
@@ -255,7 +255,7 @@ make exceptions for this.  Our decision will be guided by the two goals
 of preserving the free status of all derivatives of our free software and
 of promoting the sharing and reuse of software generally.
 
-			    NO WARRANTY
+                            NO WARRANTY
 
   11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
 FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN
@@ -277,9 +277,9 @@ YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
 PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
 POSSIBILITY OF SUCH DAMAGES.
 
-		     END OF TERMS AND CONDITIONS
-

-	    How to Apply These Terms to Your New Programs
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
 
   If you develop a new program, and you want it to be of the greatest
 possible use to the public, the best way to achieve this is to make it
@@ -303,10 +303,9 @@ the "copyright" line and a pointer to where the full notice is found.
     MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
     GNU General Public License for more details.
 
-    You should have received a copy of the GNU General Public License
-    along with this program; if not, write to the Free Software
-    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
-
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 Also add information on how to contact you by electronic and paper mail.
 
@@ -336,5 +335,5 @@ necessary.  Here is a sample; alter the names:
 This General Public License does not permit incorporating your program into
 proprietary programs.  If your program is a subroutine library, you may
 consider it more useful to permit linking proprietary applications with the
-library.  If this is what you want to do, use the GNU Library General
+library.  If this is what you want to do, use the GNU Lesser General
 Public License instead of this License.
diff --git a/Makefile b/Makefile
index 6fb245f..2827c47 100644
--- a/Makefile
+++ b/Makefile
@@ -1,8 +1,11 @@
+DIRNAME = $(notdir $(shell pwd))
+DIRNAME := $(addsuffix -auto, $(DIRNAME))
+
 build: clean
 	charmcraft pack
-	juju add-model testserver
+	juju add-model $(DIRNAME)
 	juju deploy ./bundle.yaml
 
 clean:
 	-rm *.charm
-	-juju destroy-model -y testserver --force
+	-juju destroy-model -y $(DIRNAME) --force
diff --git a/README.md b/README.md
index 86763ce..5832b12 100644
--- a/README.md
+++ b/README.md
@@ -69,6 +69,5 @@ When developing the charm, here's a quick way to test out changes as
 they would be deployed by `landscape-scalable`:
 
 ```bash
-charmcraft pack
-juju deploy ./bundle.yaml
-```
\ No newline at end of file
+make build
+```
diff --git a/bundle.yaml b/bundle.yaml
index 0e7f386..7af5829 100644
--- a/bundle.yaml
+++ b/bundle.yaml
@@ -1,19 +1,17 @@
 description: Landscape Scalable
+series: jammy
 applications:
   postgresql:
-    series: focal
+    channel: 14/beta
     charm: ch:postgresql
     num_units: 1
-    options:
-      extra_packages: python3-apt postgresql-contrib postgresql-.*-debversion postgresql-plpython3-*
-      max_connections: 500
-      max_prepared_transactions: 500
   rabbitmq-server:
-    series: focal
+    channel: latest/edge
     charm: ch:rabbitmq-server
     num_units: 1
   haproxy:
-    series: focal
+    series: jammy
+    channel: edge
     charm: ch:haproxy
     num_units: 1
     expose: true
@@ -23,7 +21,6 @@ applications:
       ssl_cert: SELFSIGNED
       global_default_bind_options: "no-tlsv10"
   landscape-server:
-    series: jammy
     charm: ./landscape-server_ubuntu-22.04-amd64-arm64_ubuntu-20.04-amd64-arm64.charm
     num_units: 1
 relations:
diff --git a/config.yaml b/config.yaml
index c08640f..cc1e765 100644
--- a/config.yaml
+++ b/config.yaml
@@ -6,6 +6,11 @@ options:
     type: string
     default: "ppa:landscape/self-hosted-23.03"
     description: The PPA from which Landscape Server will be installed.
+  landscape_ppa_key:
+    type: string
+    default: ""
+    description: |
+      Full ASCII-armoured GPG public key for the Landscape PPA source.
   worker_counts:
     type: int
     default: 2
@@ -182,3 +187,9 @@ options:
     description: |
       Additional service.conf settings to be merged with the default
       configuration.
+  secret_token:
+    type: string
+    default:
+    description: |
+      A secret token for the landscape service. If not set one will be
+      generated securely.
diff --git a/lib/charms/grafana_agent/LICENSE b/lib/charms/grafana_agent/LICENSE
new file mode 100644
index 0000000..a76e8a4
--- /dev/null
+++ b/lib/charms/grafana_agent/LICENSE
@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2024 Canonical Ltd.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
\ No newline at end of file
diff --git a/lib/charms/grafana_agent/v0/cos_agent.py b/lib/charms/grafana_agent/v0/cos_agent.py
new file mode 100644
index 0000000..259a901
--- /dev/null
+++ b/lib/charms/grafana_agent/v0/cos_agent.py
@@ -0,0 +1,819 @@
+# Copyright 2023 Canonical Ltd.
+# See LICENSE file for licensing details.
+
+r"""## Overview.
+
+This library can be used to manage the cos_agent relation interface:
+
+- `COSAgentProvider`: Use in machine charms that need to have a workload's metrics
+  or logs scraped, or forward rule files or dashboards to Prometheus, Loki or Grafana through
+  the Grafana Agent machine charm.
+
+- `COSAgentConsumer`: Used in the Grafana Agent machine charm to manage the requirer side of
+  the `cos_agent` interface.
+
+
+## COSAgentProvider Library Usage
+
+Grafana Agent machine Charmed Operator interacts with its clients using the cos_agent library.
+Charms seeking to send telemetry, must do so using the `COSAgentProvider` object from
+this charm library.
+
+Using the `COSAgentProvider` object only requires instantiating it,
+typically in the `__init__` method of your charm (the one which sends telemetry).
+
+The constructor of `COSAgentProvider` has only one required and nine optional parameters:
+
+```python
+    def __init__(
+        self,
+        charm: CharmType,
+        relation_name: str = DEFAULT_RELATION_NAME,
+        metrics_endpoints: Optional[List[_MetricsEndpointDict]] = None,
+        metrics_rules_dir: str = "./src/prometheus_alert_rules",
+        logs_rules_dir: str = "./src/loki_alert_rules",
+        recurse_rules_dirs: bool = False,
+        log_slots: Optional[List[str]] = None,
+        dashboard_dirs: Optional[List[str]] = None,
+        refresh_events: Optional[List] = None,
+        scrape_configs: Optional[Union[List[Dict], Callable]] = None,
+    ):
+```
+
+### Parameters
+
+- `charm`: The instance of the charm that instantiates `COSAgentProvider`, typically `self`.
+
+- `relation_name`: If your charmed operator uses a relation name other than `cos-agent` to use
+    the `cos_agent` interface, this is where you have to specify that.
+
+- `metrics_endpoints`: In this parameter you can specify the metrics endpoints that Grafana Agent
+    machine Charmed Operator will scrape. The configs of this list will be merged with the configs
+    from `scrape_configs`.
+
+- `metrics_rules_dir`: The directory in which the Charmed Operator stores its metrics alert rules
+  files.
+
+- `logs_rules_dir`: The directory in which the Charmed Operator stores its logs alert rules files.
+
+- `recurse_rules_dirs`: This parameters set whether Grafana Agent machine Charmed Operator has to
+  search alert rules files recursively in the previous two directories or not.
+
+- `log_slots`: Snap slots to connect to for scraping logs in the form ["snap-name:slot", ...].
+
+- `dashboard_dirs`: List of directories where the dashboards are stored in the Charmed Operator.
+
+- `refresh_events`: List of events on which to refresh relation data.
+
+- `scrape_configs`: List of standard scrape_configs dicts or a callable that returns the list in
+    case the configs need to be generated dynamically. The contents of this list will be merged
+    with the configs from `metrics_endpoints`.
+
+
+### Example 1 - Minimal instrumentation:
+
+In order to use this object the following should be in the `charm.py` file.
+
+```python
+from charms.grafana_agent.v0.cos_agent import COSAgentProvider
+...
+class TelemetryProviderCharm(CharmBase):
+    def __init__(self, *args):
+        ...
+        self._grafana_agent = COSAgentProvider(self)
+```
+
+### Example 2 - Full instrumentation:
+
+In order to use this object the following should be in the `charm.py` file.
+
+```python
+from charms.grafana_agent.v0.cos_agent import COSAgentProvider
+...
+class TelemetryProviderCharm(CharmBase):
+    def __init__(self, *args):
+        ...
+        self._grafana_agent = COSAgentProvider(
+            self,
+            relation_name="custom-cos-agent",
+            metrics_endpoints=[
+                # specify "path" and "port" to scrape from localhost
+                {"path": "/metrics", "port": 9000},
+                {"path": "/metrics", "port": 9001},
+                {"path": "/metrics", "port": 9002},
+            ],
+            metrics_rules_dir="./src/alert_rules/prometheus",
+            logs_rules_dir="./src/alert_rules/loki",
+            recursive_rules_dir=True,
+            log_slots=["my-app:slot"],
+            dashboard_dirs=["./src/dashboards_1", "./src/dashboards_2"],
+            refresh_events=["update-status", "upgrade-charm"],
+            scrape_configs=[
+                {
+                    "job_name": "custom_job",
+                    "metrics_path": "/metrics",
+                    "authorization": {"credentials": "bearer-token"},
+                    "static_configs": [
+                        {
+                            "targets": ["localhost:9003"]},
+                            "labels": {"key": "value"},
+                        },
+                    ],
+                },
+            ]
+        )
+```
+
+### Example 3 - Dynamic scrape configs generation:
+
+Pass a function to the `scrape_configs` to decouple the generation of the configs
+from the instantiation of the COSAgentProvider object.
+
+```python
+from charms.grafana_agent.v0.cos_agent import COSAgentProvider
+...
+
+class TelemetryProviderCharm(CharmBase):
+    def generate_scrape_configs(self):
+        return [
+            {
+                "job_name": "custom",
+                "metrics_path": "/metrics",
+                "static_configs": [{"targets": ["localhost:9000"]}],
+            },
+        ]
+
+    def __init__(self, *args):
+        ...
+        self._grafana_agent = COSAgentProvider(
+            self,
+            scrape_configs=self.generate_scrape_configs,
+        )
+```
+
+## COSAgentConsumer Library Usage
+
+This object may be used by any Charmed Operator which gathers telemetry data by
+implementing the consumer side of the `cos_agent` interface.
+For instance Grafana Agent machine Charmed Operator.
+
+For this purpose the charm needs to instantiate the `COSAgentConsumer` object with one mandatory
+and two optional arguments.
+
+### Parameters
+
+- `charm`: A reference to the parent (Grafana Agent machine) charm.
+
+- `relation_name`: The name of the relation that the charm uses to interact
+  with its clients that provides telemetry data using the `COSAgentProvider` object.
+
+  If provided, this relation name must match a provided relation in metadata.yaml with the
+  `cos_agent` interface.
+  The default value of this argument is "cos-agent".
+
+- `refresh_events`: List of events on which to refresh relation data.
+
+
+### Example 1 - Minimal instrumentation:
+
+In order to use this object the following should be in the `charm.py` file.
+
+```python
+from charms.grafana_agent.v0.cos_agent import COSAgentConsumer
+...
+class GrafanaAgentMachineCharm(GrafanaAgentCharm)
+    def __init__(self, *args):
+        ...
+        self._cos = COSAgentRequirer(self)
+```
+
+
+### Example 2 - Full instrumentation:
+
+In order to use this object the following should be in the `charm.py` file.
+
+```python
+from charms.grafana_agent.v0.cos_agent import COSAgentConsumer
+...
+class GrafanaAgentMachineCharm(GrafanaAgentCharm)
+    def __init__(self, *args):
+        ...
+        self._cos = COSAgentRequirer(
+            self,
+            relation_name="cos-agent-consumer",
+            refresh_events=["update-status", "upgrade-charm"],
+        )
+```
+"""
+
+import json
+import logging
+from collections import namedtuple
+from itertools import chain
+from pathlib import Path
+from typing import TYPE_CHECKING, Any, Callable, ClassVar, Dict, List, Optional, Set, Union
+
+import pydantic
+from cosl import GrafanaDashboard, JujuTopology
+from cosl.rules import AlertRules
+from ops.charm import RelationChangedEvent
+from ops.framework import EventBase, EventSource, Object, ObjectEvents
+from ops.model import Relation, Unit
+from ops.testing import CharmType
+
+if TYPE_CHECKING:
+    try:
+        from typing import TypedDict
+
+        class _MetricsEndpointDict(TypedDict):
+            path: str
+            port: int
+
+    except ModuleNotFoundError:
+        _MetricsEndpointDict = Dict  # pyright: ignore
+
+LIBID = "dc15fa84cef84ce58155fb84f6c6213a"
+LIBAPI = 0
+LIBPATCH = 7
+
+PYDEPS = ["cosl", "pydantic < 2"]
+
+DEFAULT_RELATION_NAME = "cos-agent"
+DEFAULT_PEER_RELATION_NAME = "peers"
+DEFAULT_SCRAPE_CONFIG = {
+    "static_configs": [{"targets": ["localhost:80"]}],
+    "metrics_path": "/metrics",
+}
+
+logger = logging.getLogger(__name__)
+SnapEndpoint = namedtuple("SnapEndpoint", "owner, name")
+
+
+class CosAgentProviderUnitData(pydantic.BaseModel):
+    """Unit databag model for `cos-agent` relation."""
+
+    # The following entries are the same for all units of the same principal.
+    # Note that the same grafana agent subordinate may be related to several apps.
+    # this needs to make its way to the gagent leader
+    metrics_alert_rules: dict
+    log_alert_rules: dict
+    dashboards: List[GrafanaDashboard]
+    subordinate: Optional[bool]
+
+    # The following entries may vary across units of the same principal app.
+    # this data does not need to be forwarded to the gagent leader
+    metrics_scrape_jobs: List[Dict]
+    log_slots: List[str]
+
+    # when this whole datastructure is dumped into a databag, it will be nested under this key.
+    # while not strictly necessary (we could have it 'flattened out' into the databag),
+    # this simplifies working with the model.
+    KEY: ClassVar[str] = "config"
+
+
+class CosAgentPeersUnitData(pydantic.BaseModel):
+    """Unit databag model for `peers` cos-agent machine charm peer relation."""
+
+    # We need the principal unit name and relation metadata to be able to render identifiers
+    # (e.g. topology) on the leader side, after all the data moves into peer data (the grafana
+    # agent leader can only see its own principal, because it is a subordinate charm).
+    principal_unit_name: str
+    principal_relation_id: str
+    principal_relation_name: str
+
+    # The only data that is forwarded to the leader is data that needs to go into the app databags
+    # of the outgoing o11y relations.
+    metrics_alert_rules: Optional[dict]
+    log_alert_rules: Optional[dict]
+    dashboards: Optional[List[GrafanaDashboard]]
+
+    # when this whole datastructure is dumped into a databag, it will be nested under this key.
+    # while not strictly necessary (we could have it 'flattened out' into the databag),
+    # this simplifies working with the model.
+    KEY: ClassVar[str] = "config"
+
+    @property
+    def app_name(self) -> str:
+        """Parse out the app name from the unit name.
+
+        TODO: Switch to using `model_post_init` when pydantic v2 is released?
+          https://github.com/pydantic/pydantic/issues/1729#issuecomment-1300576214
+        """
+        return self.principal_unit_name.split("/")[0]
+
+
+class COSAgentProvider(Object):
+    """Integration endpoint wrapper for the provider side of the cos_agent interface."""
+
+    def __init__(
+        self,
+        charm: CharmType,
+        relation_name: str = DEFAULT_RELATION_NAME,
+        metrics_endpoints: Optional[List["_MetricsEndpointDict"]] = None,
+        metrics_rules_dir: str = "./src/prometheus_alert_rules",
+        logs_rules_dir: str = "./src/loki_alert_rules",
+        recurse_rules_dirs: bool = False,
+        log_slots: Optional[List[str]] = None,
+        dashboard_dirs: Optional[List[str]] = None,
+        refresh_events: Optional[List] = None,
+        *,
+        scrape_configs: Optional[Union[List[dict], Callable]] = None,
+    ):
+        """Create a COSAgentProvider instance.
+
+        Args:
+            charm: The `CharmBase` instance that is instantiating this object.
+            relation_name: The name of the relation to communicate over.
+            metrics_endpoints: List of endpoints in the form [{"path": path, "port": port}, ...].
+                This argument is a simplified form of the `scrape_configs`.
+                The contents of this list will be merged with the contents of `scrape_configs`.
+            metrics_rules_dir: Directory where the metrics rules are stored.
+            logs_rules_dir: Directory where the logs rules are stored.
+            recurse_rules_dirs: Whether to recurse into rule paths.
+            log_slots: Snap slots to connect to for scraping logs
+                in the form ["snap-name:slot", ...].
+            dashboard_dirs: Directory where the dashboards are stored.
+            refresh_events: List of events on which to refresh relation data.
+            scrape_configs: List of standard scrape_configs dicts or a callable
+                that returns the list in case the configs need to be generated dynamically.
+                The contents of this list will be merged with the contents of `metrics_endpoints`.
+        """
+        super().__init__(charm, relation_name)
+        dashboard_dirs = dashboard_dirs or ["./src/grafana_dashboards"]
+
+        self._charm = charm
+        self._relation_name = relation_name
+        self._metrics_endpoints = metrics_endpoints or []
+        self._scrape_configs = scrape_configs or []
+        self._metrics_rules = metrics_rules_dir
+        self._logs_rules = logs_rules_dir
+        self._recursive = recurse_rules_dirs
+        self._log_slots = log_slots or []
+        self._dashboard_dirs = dashboard_dirs
+        self._refresh_events = refresh_events or [self._charm.on.config_changed]
+
+        events = self._charm.on[relation_name]
+        self.framework.observe(events.relation_joined, self._on_refresh)
+        self.framework.observe(events.relation_changed, self._on_refresh)
+        for event in self._refresh_events:
+            self.framework.observe(event, self._on_refresh)
+
+    def _on_refresh(self, event):
+        """Trigger the class to update relation data."""
+        relations = self._charm.model.relations[self._relation_name]
+
+        for relation in relations:
+            # Before a principal is related to the grafana-agent subordinate, we'd get
+            # ModelError: ERROR cannot read relation settings: unit "zk/2": settings not found
+            # Add a guard to make sure it doesn't happen.
+            if relation.data and self._charm.unit in relation.data:
+                # Subordinate relations can communicate only over unit data.
+                try:
+                    data = CosAgentProviderUnitData(
+                        metrics_alert_rules=self._metrics_alert_rules,
+                        log_alert_rules=self._log_alert_rules,
+                        dashboards=self._dashboards,
+                        metrics_scrape_jobs=self._scrape_jobs,
+                        log_slots=self._log_slots,
+                        subordinate=self._charm.meta.subordinate,
+                    )
+                    relation.data[self._charm.unit][data.KEY] = data.json()
+                except (
+                    pydantic.ValidationError,
+                    json.decoder.JSONDecodeError,
+                ) as e:
+                    logger.error("Invalid relation data provided: %s", e)
+
+    @property
+    def _scrape_jobs(self) -> List[Dict]:
+        """Return a prometheus_scrape-like data structure for jobs.
+
+        https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
+        """
+        if callable(self._scrape_configs):
+            scrape_configs = self._scrape_configs()
+        else:
+            # Create a copy of the user scrape_configs, since we will mutate this object
+            scrape_configs = self._scrape_configs.copy()
+
+        # Convert "metrics_endpoints" to standard scrape_configs, and add them in
+        for endpoint in self._metrics_endpoints:
+            scrape_configs.append(
+                {
+                    "metrics_path": endpoint["path"],
+                    "static_configs": [{"targets": [f"localhost:{endpoint['port']}"]}],
+                }
+            )
+
+        scrape_configs = scrape_configs or [DEFAULT_SCRAPE_CONFIG]
+
+        # Augment job name to include the app name and a unique id (index)
+        for idx, scrape_config in enumerate(scrape_configs):
+            scrape_config["job_name"] = "_".join(
+                [self._charm.app.name, str(idx), scrape_config.get("job_name", "default")]
+            )
+
+        return scrape_configs
+
+    @property
+    def _metrics_alert_rules(self) -> Dict:
+        """Use (for now) the prometheus_scrape AlertRules to initialize this."""
+        alert_rules = AlertRules(
+            query_type="promql", topology=JujuTopology.from_charm(self._charm)
+        )
+        alert_rules.add_path(self._metrics_rules, recursive=self._recursive)
+        return alert_rules.as_dict()
+
+    @property
+    def _log_alert_rules(self) -> Dict:
+        """Use (for now) the loki_push_api AlertRules to initialize this."""
+        alert_rules = AlertRules(query_type="logql", topology=JujuTopology.from_charm(self._charm))
+        alert_rules.add_path(self._logs_rules, recursive=self._recursive)
+        return alert_rules.as_dict()
+
+    @property
+    def _dashboards(self) -> List[GrafanaDashboard]:
+        dashboards: List[GrafanaDashboard] = []
+        for d in self._dashboard_dirs:
+            for path in Path(d).glob("*"):
+                dashboard = GrafanaDashboard._serialize(path.read_bytes())
+                dashboards.append(dashboard)
+        return dashboards
+
+
+class COSAgentDataChanged(EventBase):
+    """Event emitted by `COSAgentRequirer` when relation data changes."""
+
+
+class COSAgentValidationError(EventBase):
+    """Event emitted by `COSAgentRequirer` when there is an error in the relation data."""
+
+    def __init__(self, handle, message: str = ""):
+        super().__init__(handle)
+        self.message = message
+
+    def snapshot(self) -> Dict:
+        """Save COSAgentValidationError source information."""
+        return {"message": self.message}
+
+    def restore(self, snapshot):
+        """Restore COSAgentValidationError source information."""
+        self.message = snapshot["message"]
+
+
+class COSAgentRequirerEvents(ObjectEvents):
+    """`COSAgentRequirer` events."""
+
+    data_changed = EventSource(COSAgentDataChanged)
+    validation_error = EventSource(COSAgentValidationError)
+
+
+class MultiplePrincipalsError(Exception):
+    """Custom exception for when there are multiple principal applications."""
+
+    pass
+
+
+class COSAgentRequirer(Object):
+    """Integration endpoint wrapper for the Requirer side of the cos_agent interface."""
+
+    on = COSAgentRequirerEvents()  # pyright: ignore
+
+    def __init__(
+        self,
+        charm: CharmType,
+        *,
+        relation_name: str = DEFAULT_RELATION_NAME,
+        peer_relation_name: str = DEFAULT_PEER_RELATION_NAME,
+        refresh_events: Optional[List[str]] = None,
+    ):
+        """Create a COSAgentRequirer instance.
+
+        Args:
+            charm: The `CharmBase` instance that is instantiating this object.
+            relation_name: The name of the relation to communicate over.
+            peer_relation_name: The name of the peer relation to communicate over.
+            refresh_events: List of events on which to refresh relation data.
+        """
+        super().__init__(charm, relation_name)
+        self._charm = charm
+        self._relation_name = relation_name
+        self._peer_relation_name = peer_relation_name
+        self._refresh_events = refresh_events or [self._charm.on.config_changed]
+
+        events = self._charm.on[relation_name]
+        self.framework.observe(
+            events.relation_joined, self._on_relation_data_changed
+        )  # TODO: do we need this?
+        self.framework.observe(events.relation_changed, self._on_relation_data_changed)
+        for event in self._refresh_events:
+            self.framework.observe(event, self.trigger_refresh)  # pyright: ignore
+
+        # Peer relation events
+        # A peer relation is needed as it is the only mechanism for exchanging data across
+        # subordinate units.
+        # self.framework.observe(
+        #     self.on[self._peer_relation_name].relation_joined, self._on_peer_relation_joined
+        # )
+        peer_events = self._charm.on[peer_relation_name]
+        self.framework.observe(peer_events.relation_changed, self._on_peer_relation_changed)
+
+    @property
+    def peer_relation(self) -> Optional["Relation"]:
+        """Helper function for obtaining the peer relation object.
+
+        Returns: peer relation object
+        (NOTE: would return None if called too early, e.g. during install).
+        """
+        return self.model.get_relation(self._peer_relation_name)
+
+    def _on_peer_relation_changed(self, _):
+        # Peer data is used for forwarding data from principal units to the grafana agent
+        # subordinate leader, for updating the app data of the outgoing o11y relations.
+        if self._charm.unit.is_leader():
+            self.on.data_changed.emit()  # pyright: ignore
+
+    def _on_relation_data_changed(self, event: RelationChangedEvent):
+        # Peer data is the only means of communication between subordinate units.
+        if not self.peer_relation:
+            event.defer()
+            return
+
+        cos_agent_relation = event.relation
+        if not event.unit or not cos_agent_relation.data.get(event.unit):
+            return
+        principal_unit = event.unit
+
+        # Coherence check
+        units = cos_agent_relation.units
+        if len(units) > 1:
+            # should never happen
+            raise ValueError(
+                f"unexpected error: subordinate relation {cos_agent_relation} "
+                f"should have exactly one unit"
+            )
+
+        if not (raw := cos_agent_relation.data[principal_unit].get(CosAgentProviderUnitData.KEY)):
+            return
+
+        if not (provider_data := self._validated_provider_data(raw)):
+            return
+
+        # Copy data from the principal relation to the peer relation, so the leader could
+        # follow up.
+        # Save the originating unit name, so it could be used for topology later on by the leader.
+        data = CosAgentPeersUnitData(  # peer relation databag model
+            principal_unit_name=event.unit.name,
+            principal_relation_id=str(event.relation.id),
+            principal_relation_name=event.relation.name,
+            metrics_alert_rules=provider_data.metrics_alert_rules,
+            log_alert_rules=provider_data.log_alert_rules,
+            dashboards=provider_data.dashboards,
+        )
+        self.peer_relation.data[self._charm.unit][
+            f"{CosAgentPeersUnitData.KEY}-{event.unit.name}"
+        ] = data.json()
+
+        # We can't easily tell if the data that was changed is limited to only the data
+        # that goes into peer relation (in which case, if this is not a leader unit, we wouldn't
+        # need to emit `on.data_changed`), so we're emitting `on.data_changed` either way.
+        self.on.data_changed.emit()  # pyright: ignore
+
+    def _validated_provider_data(self, raw) -> Optional[CosAgentProviderUnitData]:
+        try:
+            return CosAgentProviderUnitData(**json.loads(raw))
+        except (pydantic.ValidationError, json.decoder.JSONDecodeError) as e:
+            self.on.validation_error.emit(message=str(e))  # pyright: ignore
+            return None
+
+    def trigger_refresh(self, _):
+        """Trigger a refresh of relation data."""
+        # FIXME: Figure out what we should do here
+        self.on.data_changed.emit()  # pyright: ignore
+
+    @property
+    def _principal_unit(self) -> Optional[Unit]:
+        """Return the principal unit for a relation.
+
+        Assumes that the relation is of type subordinate.
+        Relies on the fact that, for subordinate relations, the only remote unit visible to
+        *this unit* is the principal unit that this unit is attached to.
+        """
+        if relations := self._principal_relations:
+            # Technically it's a list, but for subordinates there can only be one relation
+            principal_relation = next(iter(relations))
+            if units := principal_relation.units:
+                # Technically it's a list, but for subordinates there can only be one
+                return next(iter(units))
+
+        return None
+
+    @property
+    def _principal_relations(self):
+        relations = []
+        for relation in self._charm.model.relations[self._relation_name]:
+            if not json.loads(relation.data[next(iter(relation.units))]["config"]).get(
+                ["subordinate"], False
+            ):
+                relations.append(relation)
+        if len(relations) > 1:
+            logger.error(
+                "Multiple applications claiming to be principal. Update the cos-agent library in the client application charms."
+            )
+            raise MultiplePrincipalsError("Multiple principal applications.")
+        return relations
+
+    @property
+    def _remote_data(self) -> List[CosAgentProviderUnitData]:
+        """Return a list of remote data from each of the related units.
+
+        Assumes that the relation is of type subordinate.
+        Relies on the fact that, for subordinate relations, the only remote unit visible to
+        *this unit* is the principal unit that this unit is attached to.
+        """
+        all_data = []
+
+        for relation in self._charm.model.relations[self._relation_name]:
+            if not relation.units:
+                continue
+            unit = next(iter(relation.units))
+            if not (raw := relation.data[unit].get(CosAgentProviderUnitData.KEY)):
+                continue
+            if not (provider_data := self._validated_provider_data(raw)):
+                continue
+            all_data.append(provider_data)
+
+        return all_data
+
+    def _gather_peer_data(self) -> List[CosAgentPeersUnitData]:
+        """Collect data from the peers.
+
+        Returns a trimmed-down list of CosAgentPeersUnitData.
+        """
+        relation = self.peer_relation
+
+        # Ensure that whatever context we're running this in, we take the necessary precautions:
+        if not relation or not relation.data or not relation.app:
+            return []
+
+        # Iterate over all peer unit data and only collect every principal once.
+        peer_data: List[CosAgentPeersUnitData] = []
+        app_names: Set[str] = set()
+
+        for unit in chain((self._charm.unit,), relation.units):
+            if not relation.data.get(unit):
+                continue
+
+            for unit_name in relation.data.get(unit):  # pyright: ignore
+                if not unit_name.startswith(CosAgentPeersUnitData.KEY):
+                    continue
+                raw = relation.data[unit].get(unit_name)
+                if raw is None:
+                    continue
+                data = CosAgentPeersUnitData(**json.loads(raw))
+                # Have we already seen this principal app?
+                if (app_name := data.app_name) in app_names:
+                    continue
+                peer_data.append(data)
+                app_names.add(app_name)
+
+        return peer_data
+
+    @property
+    def metrics_alerts(self) -> Dict[str, Any]:
+        """Fetch metrics alerts."""
+        alert_rules = {}
+
+        seen_apps: List[str] = []
+        for data in self._gather_peer_data():
+            if rules := data.metrics_alert_rules:
+                app_name = data.app_name
+                if app_name in seen_apps:
+                    continue  # dedup!
+                seen_apps.append(app_name)
+                # This is only used for naming the file, so be as specific as we can be
+                identifier = JujuTopology(
+                    model=self._charm.model.name,
+                    model_uuid=self._charm.model.uuid,
+                    application=app_name,
+                    # For the topology unit, we could use `data.principal_unit_name`, but that unit
+                    # name may not be very stable: `_gather_peer_data` de-duplicates by app name so
+                    # the exact unit name that turns up first in the iterator may vary from time to
+                    # time. So using the grafana-agent unit name instead.
+                    unit=self._charm.unit.name,
+                ).identifier
+
+                alert_rules[identifier] = rules
+
+        return alert_rules
+
+    @property
+    def metrics_jobs(self) -> List[Dict]:
+        """Parse the relation data contents and extract the metrics jobs."""
+        scrape_jobs = []
+        for data in self._remote_data:
+            for job in data.metrics_scrape_jobs:
+                # In #220, relation schema changed from a simplified dict to the standard
+                # `scrape_configs`.
+                # This is to ensure backwards compatibility with Providers older than v0.5.
+                if "path" in job and "port" in job and "job_name" in job:
+                    job = {
+                        "job_name": job["job_name"],
+                        "metrics_path": job["path"],
+                        "static_configs": [{"targets": [f"localhost:{job['port']}"]}],
+                        # We include insecure_skip_verify because we are always scraping localhost.
+                        # Even if we have the certs for the scrape targets, we'd rather specify the scrape
+                        # jobs with localhost rather than the SAN DNS the cert was issued for.
+                        "tls_config": {"insecure_skip_verify": True},
+                    }
+
+                scrape_jobs.append(job)
+
+        return scrape_jobs
+
+    @property
+    def snap_log_endpoints(self) -> List[SnapEndpoint]:
+        """Fetch logging endpoints exposed by related snaps."""
+        plugs = []
+        for data in self._remote_data:
+            targets = data.log_slots
+            if targets:
+                for target in targets:
+                    if target in plugs:
+                        logger.warning(
+                            f"plug {target} already listed. "
+                            "The same snap is being passed from multiple "
+                            "endpoints; this should not happen."
+                        )
+                    else:
+                        plugs.append(target)
+
+        endpoints = []
+        for plug in plugs:
+            if ":" not in plug:
+                logger.error(f"invalid plug definition received: {plug}. Ignoring...")
+            else:
+                endpoint = SnapEndpoint(*plug.split(":"))
+                endpoints.append(endpoint)
+        return endpoints
+
+    @property
+    def logs_alerts(self) -> Dict[str, Any]:
+        """Fetch log alerts."""
+        alert_rules = {}
+        seen_apps: List[str] = []
+
+        for data in self._gather_peer_data():
+            if rules := data.log_alert_rules:
+                # This is only used for naming the file, so be as specific as we can be
+                app_name = data.app_name
+                if app_name in seen_apps:
+                    continue  # dedup!
+                seen_apps.append(app_name)
+
+                identifier = JujuTopology(
+                    model=self._charm.model.name,
+                    model_uuid=self._charm.model.uuid,
+                    application=app_name,
+                    # For the topology unit, we could use `data.principal_unit_name`, but that unit
+                    # name may not be very stable: `_gather_peer_data` de-duplicates by app name so
+                    # the exact unit name that turns up first in the iterator may vary from time to
+                    # time. So using the grafana-agent unit name instead.
+                    unit=self._charm.unit.name,
+                ).identifier
+
+                alert_rules[identifier] = rules
+
+        return alert_rules
+
+    @property
+    def dashboards(self) -> List[Dict[str, str]]:
+        """Fetch dashboards as encoded content.
+
+        Dashboards are assumed not to vary across units of the same primary.
+        """
+        dashboards: List[Dict[str, Any]] = []
+
+        seen_apps: List[str] = []
+        for data in self._gather_peer_data():
+            app_name = data.app_name
+            if app_name in seen_apps:
+                continue  # dedup!
+            seen_apps.append(app_name)
+
+            for encoded_dashboard in data.dashboards or ():
+                content = GrafanaDashboard(encoded_dashboard)._deserialize()
+
+                title = content.get("title", "no_title")
+
+                dashboards.append(
+                    {
+                        "relation_id": data.principal_relation_id,
+                        # We have the remote charm name - use it for the identifier
+                        "charm": f"{data.principal_relation_name}-{app_name}",
+                        "content": content,
+                        "title": title,
+                    }
+                )
+
+        return dashboards
diff --git a/lib/charms/operator_libs_linux/v0/apt.py b/lib/charms/operator_libs_linux/v0/apt.py
index 2f921f0..1400df7 100644
--- a/lib/charms/operator_libs_linux/v0/apt.py
+++ b/lib/charms/operator_libs_linux/v0/apt.py
@@ -78,7 +78,6 @@ Keys are constructed as `{repo_type}-{}-{release}` in order to uniquely identify
 Repositories can be added with explicit values through a Python constructor.
 
 Example:
-
 ```python
 repositories = apt.RepositoryMapping()
 
@@ -91,7 +90,6 @@ Alternatively, any valid `sources.list` line may be used to construct a new
 `DebianRepository`.
 
 Example:
-
 ```python
 repositories = apt.RepositoryMapping()
 
@@ -110,7 +108,7 @@ import re
 import subprocess
 from collections.abc import Mapping
 from enum import Enum
-from subprocess import PIPE, CalledProcessError, check_call, check_output
+from subprocess import PIPE, CalledProcessError, check_output
 from typing import Iterable, List, Optional, Tuple, Union
 from urllib.parse import urlparse
 
@@ -124,7 +122,7 @@ LIBAPI = 0
 
 # Increment this PATCH version before using `charmcraft publish-lib` or reset
 # to 0 if you are raising the major API version
-LIBPATCH = 8
+LIBPATCH = 13
 
 
 VALID_SOURCE_TYPES = ("deb", "deb-src")
@@ -135,7 +133,7 @@ class Error(Exception):
     """Base class of most errors raised by this library."""
 
     def __repr__(self):
-        """String representation of Error."""
+        """Represent the Error."""
         return "<{}.{} {}>".format(type(self).__module__, type(self).__name__, self.args)
 
     @property
@@ -212,15 +210,15 @@ class DebianPackage:
         ) == (other._name, other._version.number)
 
     def __hash__(self):
-        """A basic hash so this class can be used in Mappings and dicts."""
+        """Return a hash of this package."""
         return hash((self._name, self._version.number))
 
     def __repr__(self):
-        """A representation of the package."""
+        """Represent the package."""
         return "<{}.{}: {}>".format(self.__module__, self.__class__.__name__, self.__dict__)
 
     def __str__(self):
-        """A human-readable representation of the package."""
+        """Return a human-readable representation of the package."""
         return "<{}: {}-{}.{} -- {}>".format(
             self.__class__.__name__,
             self._name,
@@ -250,11 +248,12 @@ class DebianPackage:
             package_names = [package_names]
         _cmd = ["apt-get", "-y", *optargs, command, *package_names]
         try:
-            env = {"DEBIAN_FRONTEND": "noninteractive"}
-            check_call(_cmd, env=env, stderr=PIPE, stdout=PIPE)
+            env = os.environ.copy()
+            env["DEBIAN_FRONTEND"] = "noninteractive"
+            subprocess.run(_cmd, capture_output=True, check=True, text=True, env=env)
         except CalledProcessError as e:
             raise PackageError(
-                "Could not {} package(s) [{}]: {}".format(command, [*package_names], e.output)
+                "Could not {} package(s) [{}]: {}".format(command, [*package_names], e.stderr)
             ) from None
 
     def _add(self) -> None:
@@ -266,7 +265,7 @@ class DebianPackage:
         )
 
     def _remove(self) -> None:
-        """Removes a package from the system. Implementation-specific."""
+        """Remove a package from the system. Implementation-specific."""
         return self._apt("remove", "{}={}".format(self.name, self.version))
 
     @property
@@ -275,7 +274,7 @@ class DebianPackage:
         return self._name
 
     def ensure(self, state: PackageState):
-        """Ensures that a package is in a given state.
+        """Ensure that a package is in a given state.
 
         Args:
           state: a `PackageState` to reconcile the package to
@@ -307,7 +306,7 @@ class DebianPackage:
 
     @state.setter
     def state(self, state: PackageState) -> None:
-        """Sets the package state to a given value.
+        """Set the package state to a given value.
 
         Args:
           state: a `PackageState` to reconcile the package to
@@ -356,7 +355,7 @@ class DebianPackage:
 
         Args:
             package: a string representing the package
-            version: an optional string if a specific version isr equested
+            version: an optional string if a specific version is requested
             arch: an optional architecture, defaulting to `dpkg --print-architecture`. If an
                 architecture is not specified, this will be used for selection.
 
@@ -389,7 +388,7 @@ class DebianPackage:
 
         Args:
             package: a string representing the package
-            version: an optional string if a specific version isr equested
+            version: an optional string if a specific version is requested
             arch: an optional architecture, defaulting to `dpkg --print-architecture`.
                 If an architecture is not specified, this will be used for selection.
         """
@@ -459,7 +458,7 @@ class DebianPackage:
 
         Args:
             package: a string representing the package
-            version: an optional string if a specific version isr equested
+            version: an optional string if a specific version is requested
             arch: an optional architecture, defaulting to `dpkg --print-architecture`.
                 If an architecture is not specified, this will be used for selection.
         """
@@ -477,7 +476,7 @@ class DebianPackage:
             )
         except CalledProcessError as e:
             raise PackageError(
-                "Could not list packages in apt-cache: {}".format(e.output)
+                "Could not list packages in apt-cache: {}".format(e.stderr)
             ) from None
 
         pkg_groups = output.strip().split("\n\n")
@@ -515,7 +514,7 @@ class Version:
     """An abstraction around package versions.
 
     This seems like it should be strictly unnecessary, except that `apt_pkg` is not usable inside a
-    venv, and wedging version comparisions into `DebianPackage` would overcomplicate it.
+    venv, and wedging version comparisons into `DebianPackage` would overcomplicate it.
 
     This class implements the algorithm found here:
     https://www.debian.org/doc/debian-policy/ch-controlfields.html#version
@@ -526,11 +525,11 @@ class Version:
         self._epoch = epoch or ""
 
     def __repr__(self):
-        """A representation of the package."""
+        """Represent the package."""
         return "<{}.{}: {}>".format(self.__module__, self.__class__.__name__, self.__dict__)
 
     def __str__(self):
-        """A human-readable representation of the package."""
+        """Return human-readable representation of the package."""
         return "{}{}".format("{}:".format(self._epoch) if self._epoch else "", self._version)
 
     @property
@@ -731,13 +730,16 @@ def add_package(
     """Add a package or list of packages to the system.
 
     Args:
+        package_names: single package name, or list of package names
         name: the name(s) of the package(s)
         version: an (Optional) version as a string. Defaults to the latest known
         arch: an optional architecture for the package
         update_cache: whether or not to run `apt-get update` prior to operating
 
     Raises:
+        TypeError if no package name is given, or explicit version is set for multiple packages
         PackageNotFoundError if the package is not in the cache.
+        PackageError if packages fail to install
     """
     cache_refreshed = False
     if update_cache:
@@ -746,7 +748,7 @@ def add_package(
 
     packages = {"success": [], "retry": [], "failed": []}
 
-    package_names = [package_names] if type(package_names) is str else package_names
+    package_names = [package_names] if isinstance(package_names, str) else package_names
     if not package_names:
         raise TypeError("Expected at least one package name to add, received zero!")
 
@@ -785,7 +787,7 @@ def _add(
     version: Optional[str] = "",
     arch: Optional[str] = "",
 ) -> Tuple[Union[DebianPackage, str], bool]:
-    """Adds a package.
+    """Add a package to the system.
 
     Args:
         name: the name(s) of the package(s)
@@ -806,7 +808,7 @@ def _add(
 def remove_package(
     package_names: Union[str, List[str]]
 ) -> Union[DebianPackage, List[DebianPackage]]:
-    """Removes a package from the system.
+    """Remove package(s) from the system.
 
     Args:
         package_names: the name of a package
@@ -816,7 +818,7 @@ def remove_package(
     """
     packages = []
 
-    package_names = [package_names] if type(package_names) is str else package_names
+    package_names = [package_names] if isinstance(package_names, str) else package_names
     if not package_names:
         raise TypeError("Expected at least one package name to add, received zero!")
 
@@ -834,8 +836,70 @@ def remove_package(
 
 
 def update() -> None:
-    """Updates the apt cache via `apt-get update`."""
-    check_call(["apt-get", "update"], stderr=PIPE, stdout=PIPE)
+    """Update the apt cache via `apt-get update`."""
+    subprocess.run(["apt-get", "update"], capture_output=True, check=True)
+
+
+def import_key(key: str) -> str:
+    """Import an ASCII Armor key.
+
+    A Radix64 format keyid is also supported for backwards
+    compatibility. In this case Ubuntu keyserver will be
+    queried for a key via HTTPS by its keyid. This method
+    is less preferable because https proxy servers may
+    require traffic decryption which is equivalent to a
+    man-in-the-middle attack (a proxy server impersonates
+    keyserver TLS certificates and has to be explicitly
+    trusted by the system).
+
+    Args:
+        key: A GPG key in ASCII armor format, including BEGIN
+            and END markers or a keyid.
+
+    Returns:
+        The GPG key filename written.
+
+    Raises:
+        GPGKeyError if the key could not be imported
+    """
+    key = key.strip()
+    if "-" in key or "\n" in key:
+        # Send everything not obviously a keyid to GPG to import, as
+        # we trust its validation better than our own. eg. handling
+        # comments before the key.
+        logger.debug("PGP key found (looks like ASCII Armor format)")
+        if (
+            "-----BEGIN PGP PUBLIC KEY BLOCK-----" in key
+            and "-----END PGP PUBLIC KEY BLOCK-----" in key
+        ):
+            logger.debug("Writing provided PGP key in the binary format")
+            key_bytes = key.encode("utf-8")
+            key_name = DebianRepository._get_keyid_by_gpg_key(key_bytes)
+            key_gpg = DebianRepository._dearmor_gpg_key(key_bytes)
+            gpg_key_filename = "/etc/apt/trusted.gpg.d/{}.gpg".format(key_name)
+            DebianRepository._write_apt_gpg_keyfile(
+                key_name=gpg_key_filename, key_material=key_gpg
+            )
+            return gpg_key_filename
+        else:
+            raise GPGKeyError("ASCII armor markers missing from GPG key")
+    else:
+        logger.warning(
+            "PGP key found (looks like Radix64 format). "
+            "SECURELY importing PGP key from keyserver; "
+            "full key not provided."
+        )
+        # as of bionic add-apt-repository uses curl with an HTTPS keyserver URL
+        # to retrieve GPG keys. `apt-key adv` command is deprecated as is
+        # apt-key in general as noted in its manpage. See lp:1433761 for more
+        # history. Instead, /etc/apt/trusted.gpg.d is used directly to drop
+        # gpg
+        key_asc = DebianRepository._get_key_by_keyid(key)
+        # write the key in GPG format so that apt-key list shows it
+        key_gpg = DebianRepository._dearmor_gpg_key(key_asc.encode("utf-8"))
+        gpg_key_filename = "/etc/apt/trusted.gpg.d/{}.gpg".format(key)
+        DebianRepository._write_apt_gpg_keyfile(key_name=gpg_key_filename, key_material=key_gpg)
+        return gpg_key_filename
 
 
 class InvalidSourceError(Error):
@@ -901,7 +965,7 @@ class DebianRepository:
 
     @filename.setter
     def filename(self, fname: str) -> None:
-        """Sets the filename used when a repo is written back to diskself.
+        """Set the filename used when a repo is written back to disk.
 
         Args:
             fname: a filename to write the repository information to.
@@ -1004,7 +1068,7 @@ class DebianRepository:
         A Radix64 format keyid is also supported for backwards
         compatibility. In this case Ubuntu keyserver will be
         queried for a key via HTTPS by its keyid. This method
-        is less preferrable because https proxy servers may
+        is less preferable because https proxy servers may
         require traffic decryption which is equivalent to a
         man-in-the-middle attack (a proxy server impersonates
         keyserver TLS certificates and has to be explicitly
@@ -1017,40 +1081,7 @@ class DebianRepository:
         Raises:
           GPGKeyError if the key could not be imported
         """
-        key = key.strip()
-        if "-" in key or "\n" in key:
-            # Send everything not obviously a keyid to GPG to import, as
-            # we trust its validation better than our own. eg. handling
-            # comments before the key.
-            logger.debug("PGP key found (looks like ASCII Armor format)")
-            if (
-                "-----BEGIN PGP PUBLIC KEY BLOCK-----" in key
-                and "-----END PGP PUBLIC KEY BLOCK-----" in key
-            ):
-                logger.debug("Writing provided PGP key in the binary format")
-                key_bytes = key.encode("utf-8")
-                key_name = self._get_keyid_by_gpg_key(key_bytes)
-                key_gpg = self._dearmor_gpg_key(key_bytes)
-                self._gpg_key_filename = "/etc/apt/trusted.gpg.d/{}.gpg".format(key_name)
-                self._write_apt_gpg_keyfile(key_name=self._gpg_key_filename, key_material=key_gpg)
-            else:
-                raise GPGKeyError("ASCII armor markers missing from GPG key")
-        else:
-            logger.warning(
-                "PGP key found (looks like Radix64 format). "
-                "SECURELY importing PGP key from keyserver; "
-                "full key not provided."
-            )
-            # as of bionic add-apt-repository uses curl with an HTTPS keyserver URL
-            # to retrieve GPG keys. `apt-key adv` command is deprecated as is
-            # apt-key in general as noted in its manpage. See lp:1433761 for more
-            # history. Instead, /etc/apt/trusted.gpg.d is used directly to drop
-            # gpg
-            key_asc = self._get_key_by_keyid(key)
-            # write the key in GPG format so that apt-key list shows it
-            key_gpg = self._dearmor_gpg_key(key_asc.encode("utf-8"))
-            self._gpg_key_filename = "/etc/apt/trusted.gpg.d/{}.gpg".format(key)
-            self._write_apt_gpg_keyfile(key_name=key, key_material=key_gpg)
+        self._gpg_key_filename = import_key(key)
 
     @staticmethod
     def _get_keyid_by_gpg_key(key_material: bytes) -> str:
@@ -1116,7 +1147,7 @@ class DebianRepository:
 
     @staticmethod
     def _dearmor_gpg_key(key_asc: bytes) -> bytes:
-        """Converts a GPG key in the ASCII armor format to the binary format.
+        """Convert a GPG key in the ASCII armor format to the binary format.
 
         Args:
           key_asc: A GPG key in ASCII armor format.
@@ -1140,7 +1171,7 @@ class DebianRepository:
 
     @staticmethod
     def _write_apt_gpg_keyfile(key_name: str, key_material: bytes) -> None:
-        """Writes GPG key material into a file at a provided path.
+        """Write GPG key material into a file at a provided path.
 
         Args:
           key_name: A key name to use for a key file (could be a fingerprint)
@@ -1188,7 +1219,7 @@ class RepositoryMapping(Mapping):
         return len(self._repository_map)
 
     def __iter__(self) -> Iterable[DebianRepository]:
-        """Iterator magic method for RepositoryMapping."""
+        """Return iterator for RepositoryMapping."""
         return iter(self._repository_map.values())
 
     def __getitem__(self, repository_uri: str) -> DebianRepository:
diff --git a/lib/charms/operator_libs_linux/v0/passwd.py b/lib/charms/operator_libs_linux/v0/passwd.py
index b692e70..ed5a058 100644
--- a/lib/charms/operator_libs_linux/v0/passwd.py
+++ b/lib/charms/operator_libs_linux/v0/passwd.py
@@ -45,7 +45,7 @@ LIBAPI = 0
 
 # Increment this PATCH version before using `charmcraft publish-lib` or reset
 # to 0 if you are raising the major API version
-LIBPATCH = 3
+LIBPATCH = 4
 
 
 def user_exists(user: Union[str, int]) -> Optional[pwd.struct_passwd]:
@@ -99,6 +99,7 @@ def add_user(
     secondary_groups: List[str] = None,
     uid: int = None,
     home_dir: str = None,
+    create_home: bool = True,
 ) -> str:
     """Add a user to the system.
 
@@ -113,6 +114,7 @@ def add_user(
         secondary_groups: Optional list of additional groups
         uid: UID for user being created
         home_dir: Home directory for user
+        create_home: Force home directory creation
 
     Returns:
         The password database entry struct, as returned by `pwd.getpwnam`
@@ -135,7 +137,9 @@ def add_user(
     if home_dir:
         cmd.extend(["--home", str(home_dir)])
     if password:
-        cmd.extend(["--password", password, "--create-home"])
+        cmd.extend(["--password", password])
+    if create_home:
+        cmd.append("--create-home")
     if system_user or password is None:
         cmd.append("--system")
 
diff --git a/metadata.yaml b/metadata.yaml
index efc7563..61ddaad 100644
--- a/metadata.yaml
+++ b/metadata.yaml
@@ -31,6 +31,8 @@ provides:
   nrpe-external-master:
     interface: nrpe-external-master
     scope: container
+  cos-agent:
+    interface: cos_agent
 
 peers:
   replicas:
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 4f2a3f5..671f33c 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -1,3 +1,7 @@
 -r requirements.txt
 coverage
 flake8
+
+# Grafana Agent Library
+cosl
+pydantic < 2
diff --git a/src/charm.py b/src/charm.py
index 29885b8..533fc40 100755
--- a/src/charm.py
+++ b/src/charm.py
@@ -21,31 +21,50 @@ from subprocess import CalledProcessError, check_call
 import yaml
 
 from charms.operator_libs_linux.v0 import apt
-from charms.operator_libs_linux.v0.apt import (
-    PackageError, PackageNotFoundError)
+from charms.operator_libs_linux.v0.apt import PackageError, PackageNotFoundError
 from charms.operator_libs_linux.v0.passwd import group_exists, user_exists
 from charms.operator_libs_linux.v0.systemd import service_reload
+from charms.grafana_agent.v0.cos_agent import COSAgentProvider
 
 from ops.charm import (
-    ActionEvent, CharmBase, InstallEvent, LeaderElectedEvent,
-    LeaderSettingsChangedEvent, RelationChangedEvent, RelationJoinedEvent,
-    UpdateStatusEvent)
+    ActionEvent,
+    CharmBase,
+    InstallEvent,
+    LeaderElectedEvent,
+    LeaderSettingsChangedEvent,
+    RelationChangedEvent,
+    RelationDepartedEvent,
+    RelationJoinedEvent,
+    UpdateStatusEvent,
+)
 from ops.framework import StoredState
 from ops.main import main
 from ops.model import (
-    ActiveStatus, BlockedStatus, Relation, MaintenanceStatus, WaitingStatus)
+    ActiveStatus,
+    BlockedStatus,
+    Relation,
+    MaintenanceStatus,
+    WaitingStatus,
+)
 
 from settings_files import (
-    DEFAULT_POSTGRES_PORT, configure_for_deployment_mode, merge_service_conf,
-    prepend_default_settings, update_db_conf, update_default_settings, update_service_conf,
-    write_license_file, write_ssl_cert)
+    DEFAULT_POSTGRES_PORT,
+    configure_for_deployment_mode,
+    generate_secret_token,
+    merge_service_conf,
+    prepend_default_settings,
+    update_db_conf,
+    update_default_settings,
+    update_service_conf,
+    write_license_file,
+    write_ssl_cert,
+)
 
 logger = logging.getLogger(__name__)
 
 DEBCONF_SET_SELECTIONS = "/usr/bin/debconf-set-selections"
 DPKG_RECONFIGURE = "/usr/sbin/dpkg-reconfigure"
-HAPROXY_CONFIG_FILE = os.path.join(os.path.dirname(__file__),
-                                   "haproxy-config.yaml")
+HAPROXY_CONFIG_FILE = os.path.join(os.path.dirname(__file__), "haproxy-config.yaml")
 LSCTL = "/usr/bin/lsctl"
 NRPE_D_DIR = "/etc/nagios/nrpe.d"
 POSTFIX_CF = "/etc/postfix/main.cf"
@@ -53,11 +72,11 @@ SCHEMA_SCRIPT = "/usr/bin/landscape-schema"
 BOOTSTRAP_ACCOUNT_SCRIPT = "/opt/canonical/landscape/bootstrap-account"
 HASH_ID_DATABASES = "/opt/canonical/landscape/hash-id-databases-ignore-maintenance"
 
+LANDSCAPE_SERVER = "landscape-server"
 LANDSCAPE_PACKAGES = (
-    "landscape-server",
+    LANDSCAPE_SERVER,
     "landscape-client",
     "landscape-common",
-    "landscape-hashids"
 )
 
 DEFAULT_SERVICES = (
@@ -100,56 +119,71 @@ class LandscapeServerCharm(CharmBase):
         self.framework.observe(self.on.update_status, self._update_status)
 
         # Relations
-        self.framework.observe(self.on.db_relation_joined,
-                               self._db_relation_changed)
-        self.framework.observe(self.on.db_relation_changed,
-                               self._db_relation_changed)
-        self.framework.observe(self.on.amqp_relation_joined,
-                               self._amqp_relation_joined)
-        self.framework.observe(self.on.amqp_relation_changed,
-                               self._amqp_relation_changed)
-        self.framework.observe(self.on.website_relation_joined,
-                               self._website_relation_joined)
-        self.framework.observe(self.on.website_relation_changed,
-                               self._website_relation_changed)
-        self.framework.observe(self.on.nrpe_external_master_relation_joined,
-                               self._nrpe_external_master_relation_joined)
-        self.framework.observe(self.on.application_dashboard_relation_joined,
-                               self._application_dashboard_relation_joined)
+        self.framework.observe(self.on.db_relation_joined, self._db_relation_changed)
+        self.framework.observe(self.on.db_relation_changed, self._db_relation_changed)
+        self.framework.observe(self.on.amqp_relation_joined, self._amqp_relation_joined)
+        self.framework.observe(
+            self.on.amqp_relation_changed, self._amqp_relation_changed
+        )
+        self.framework.observe(
+            self.on.website_relation_joined, self._website_relation_joined
+        )
+        self.framework.observe(
+            self.on.website_relation_changed, self._website_relation_changed
+        )
+        self.framework.observe(
+            self.on.website_relation_departed, self._website_relation_departed
+        )
+        self.framework.observe(
+            self.on.nrpe_external_master_relation_joined,
+            self._nrpe_external_master_relation_joined,
+        )
+        self.framework.observe(
+            self.on.application_dashboard_relation_joined,
+            self._application_dashboard_relation_joined,
+        )
 
         # Leadership/peering
         self.framework.observe(self.on.leader_elected, self._leader_elected)
-        self.framework.observe(self.on.leader_settings_changed,
-                               self._leader_settings_changed)
-        self.framework.observe(self.on.replicas_relation_joined,
-                               self._on_replicas_relation_joined)
-        self.framework.observe(self.on.replicas_relation_changed,
-                               self._on_replicas_relation_changed)
+        self.framework.observe(
+            self.on.leader_settings_changed, self._leader_settings_changed
+        )
+        self.framework.observe(
+            self.on.replicas_relation_joined, self._on_replicas_relation_joined
+        )
+        self.framework.observe(
+            self.on.replicas_relation_changed, self._on_replicas_relation_changed
+        )
 
         # Actions
         self.framework.observe(self.on.pause_action, self._pause)
         self.framework.observe(self.on.resume_action, self._resume)
         self.framework.observe(self.on.upgrade_action, self._upgrade)
-        self.framework.observe(self.on.migrate_schema_action,
-                               self._migrate_schema)
-        self.framework.observe(self.on.hash_id_databases_action,
-                               self._hash_id_databases)
+        self.framework.observe(self.on.migrate_schema_action, self._migrate_schema)
+        self.framework.observe(
+            self.on.hash_id_databases_action, self._hash_id_databases
+        )
 
         # State
-        self._stored.set_default(ready={
-            "db": False,
-            "amqp": False,
-            "haproxy": False,
-        })
+        self._stored.set_default(
+            ready={
+                "db": False,
+                "amqp": False,
+                "haproxy": False,
+            }
+        )
         self._stored.set_default(leader_ip="")
         self._stored.set_default(running=False)
         self._stored.set_default(paused=False)
         self._stored.set_default(default_root_url="")
         self._stored.set_default(account_bootstrapped=False)
+        self._stored.set_default(secret_token=None)
 
         self.landscape_uid = user_exists("landscape").pw_uid
         self.root_gid = group_exists("root").gr_gid
 
+        self._grafana_agent = COSAgentProvider(self)
+
     def _on_config_changed(self, _) -> None:
         prev_status = self.unit.status
 
@@ -173,10 +207,8 @@ class LandscapeServerCharm(CharmBase):
         # Write the license file, if it exists.
         license_file = self.model.config.get("license_file")
         if license_file:
-            self.unit.status = MaintenanceStatus(
-                "Writing Landscape license file")
-            write_license_file(
-                license_file, self.landscape_uid, self.root_gid)
+            self.unit.status = MaintenanceStatus("Writing Landscape license file")
+            write_license_file(license_file, self.landscape_uid, self.root_gid)
             self.unit.status = WaitingStatus("Waiting on relations")
 
         smtp_relay_host = self.model.config.get("smtp_relay_host")
@@ -189,10 +221,12 @@ class LandscapeServerCharm(CharmBase):
         for relation in haproxy_relations:
             self._update_haproxy_connection(relation)
 
-        if any(self.model.config.get(v) for v in OPENID_CONFIG_VALS) \
-                and any(self.model.config.get(v) for v in OIDC_CONFIG_VALS):
+        if any(self.model.config.get(v) for v in OPENID_CONFIG_VALS) and any(
+            self.model.config.get(v) for v in OIDC_CONFIG_VALS
+        ):
             self.unit.status = BlockedStatus(
-                "OpenID and OIDC configurations are mutually exclusive")
+                "OpenID and OIDC configurations are mutually exclusive"
+            )
         else:
             self._configure_openid()
             self._configure_oidc()
@@ -200,13 +234,13 @@ class LandscapeServerCharm(CharmBase):
         # Update root_url, if provided
         root_url = self.model.config.get("root_url")
         if root_url:
-            update_service_conf({
-                "global": {"root-url": root_url},
-                "api": {"root-url": root_url},
-                "package-upload": {"root-url": root_url},
-            })
-
-        self._bootstrap_account()
+            update_service_conf(
+                {
+                    "global": {"root-url": root_url},
+                    "api": {"root-url": root_url},
+                    "package-upload": {"root-url": root_url},
+                }
+            )
 
         config_host = self.model.config.get("db_host")
         schema_password = self.model.config.get("db_schema_password")
@@ -232,37 +266,63 @@ class LandscapeServerCharm(CharmBase):
             else:
                 return
 
+        self._bootstrap_account()
+
+        secret_token = self._get_secret_token()
+        if self.unit.is_leader():
+            if not secret_token:
+                # If the secret token wasn't in the config, and we don't have one
+                # in the peer relation data, then the leader needs to generate one
+                # for all of the units to use.
+                logger.info("Generating new random secret token")
+                secret_token = generate_secret_token()
+                peer_relation = self.model.get_relation("replicas")
+                peer_relation.data[self.app].update({"secret-token": secret_token})
+        if (secret_token) and (secret_token != self._stored.secret_token):
+            self._write_secret_token(secret_token)
+            self._stored.secret_token = secret_token
+
         if isinstance(prev_status, BlockedStatus):
             self.unit.status = prev_status
 
         self._update_ready_status(restart_services=True)
 
+    def _get_secret_token(self):
+        secret_token = self.model.config.get("secret_token")
+        if not secret_token:
+            peer_relation = self.model.get_relation("replicas")
+            secret_token = peer_relation.data[self.app].get("secret-token", None)
+        return secret_token
+
+    def _write_secret_token(self, secret_token):
+        logger.info("Writing secret token")
+        update_service_conf({"landscape": {"secret-token": secret_token}})
+
     def _on_install(self, event: InstallEvent) -> None:
         """Handle the install event."""
         self.unit.status = MaintenanceStatus("Installing apt packages")
 
+        landscape_ppa_key = self.model.config["landscape_ppa_key"]
+        if landscape_ppa_key != "":
+            try:
+                landscape_key_file = apt.import_key(landscape_ppa_key)
+                logger.info(f"Imported Landscape PPA key at {landscape_key_file}")
+            except apt.GPGKeyError:
+                logger.error("Failed to import Landscape PPA key")
+
         landscape_ppa = self.model.config["landscape_ppa"]
 
         try:
+            # This package is responsible for the hanging installs and ignores env vars
+            apt.remove_package(["needrestart"])
             # Add the Landscape Server PPA and install via apt.
             check_call(["add-apt-repository", "-y", landscape_ppa])
-            apt.add_package(["landscape-server", "landscape-hashids"])
-        except PackageNotFoundError:
-            logger.error("Landscape package not found in package cache "
-                         "or on system")
-            self.unit.status = BlockedStatus("Failed to install packages")
-            return
-        except PackageError as e:
-            logger.error(
-                "Could not install landscape-server package. Reason: %s",
-                e.message)
-            self.unit.status = BlockedStatus("Failed to install packages")
-            return
-        except CalledProcessError as e:
-            logger.error("Package install failed with return code %d",
-                         e.returncode)
-            self.unit.status = BlockedStatus("Failed to install packages")
-            return
+            # Explicitly ensure cache is up-to-date after adding the PPA.
+            apt.add_package([LANDSCAPE_SERVER, "landscape-hashids"], update_cache=True)
+            check_call(["apt-mark", "hold", "landscape-hashids", LANDSCAPE_SERVER])
+        except (PackageNotFoundError, PackageError, CalledProcessError) as exc:
+            logger.error("Failed to install packages")
+            raise exc  # This will trigger juju's exponential retry
 
         # Write the config-provided SSL certificate, if it exists.
         config_ssl_cert = self.model.config["ssl_cert"]
@@ -275,8 +335,7 @@ class LandscapeServerCharm(CharmBase):
         license_file = self.model.config.get("license_file")
 
         if license_file:
-            self.unit.status = MaintenanceStatus(
-                "Writing Landscape license file")
+            self.unit.status = MaintenanceStatus("Writing Landscape license file")
             write_license_file(license_file, self.landscape_uid, self.root_gid)
 
         self.unit.status = ActiveStatus("Unit is ready")
@@ -296,10 +355,10 @@ class LandscapeServerCharm(CharmBase):
             return
 
         if not all(self._stored.ready.values()):
-            waiting_on = [
-                rel for rel, ready in self._stored.ready.items() if not ready]
+            waiting_on = [rel for rel, ready in self._stored.ready.items() if not ready]
             self.unit.status = WaitingStatus(
-                "Waiting on relations: {}".format(", ".join(waiting_on)))
+                "Waiting on relations: {}".format(", ".join(waiting_on))
+            )
             return
 
         if self._stored.running and not restart_services:
@@ -322,19 +381,23 @@ class LandscapeServerCharm(CharmBase):
         deployment_mode = self.model.config.get("deployment_mode")
         is_standalone = deployment_mode == "standalone"
 
-        update_default_settings({
-            "RUN_ALL": "no",
-            "RUN_APISERVER": str(self.model.config["worker_counts"]),
-            "RUN_ASYNC_FRONTEND": "yes",
-            "RUN_JOBHANDLER": "yes",
-            "RUN_APPSERVER": str(self.model.config["worker_counts"]),
-            "RUN_MSGSERVER": str(self.model.config["worker_counts"]),
-            "RUN_PINGSERVER": str(self.model.config["worker_counts"]),
-            "RUN_CRON": "yes" if is_leader else "no",
-            "RUN_PACKAGESEARCH": "yes" if is_leader else "no",
-            "RUN_PACKAGEUPLOADSERVER": "yes" if is_leader and is_standalone else "no",
-            "RUN_PPPA_PROXY": "no",
-        })
+        update_default_settings(
+            {
+                "RUN_ALL": "no",
+                "RUN_APISERVER": str(self.model.config["worker_counts"]),
+                "RUN_ASYNC_FRONTEND": "yes",
+                "RUN_JOBHANDLER": "yes",
+                "RUN_APPSERVER": str(self.model.config["worker_counts"]),
+                "RUN_MSGSERVER": str(self.model.config["worker_counts"]),
+                "RUN_PINGSERVER": str(self.model.config["worker_counts"]),
+                "RUN_CRON": "yes" if is_leader else "no",
+                "RUN_PACKAGESEARCH": "yes" if is_leader else "no",
+                "RUN_PACKAGEUPLOADSERVER": "yes"
+                if is_leader and is_standalone
+                else "no",
+                "RUN_PPPA_PROXY": "no",
+            }
+        )
 
         logger.info("Starting services")
 
@@ -366,7 +429,7 @@ class LandscapeServerCharm(CharmBase):
 
         allowed_units = unit_data["allowed-units"].split()
         if self.unit.name not in allowed_units:
-            logger.info("%s not in allowed_units")
+            logger.info(f"{self.unit.name} not in allowed_units")
             self.unit.status = ActiveStatus("Unit is ready")
             self._update_ready_status()
             return
@@ -406,8 +469,13 @@ class LandscapeServerCharm(CharmBase):
         else:
             user = unit_data["user"]
 
-        update_db_conf(host=host, port=port, user=user, password=password,
-                       schema_password=schema_password)
+        update_db_conf(
+            host=host,
+            port=port,
+            user=user,
+            password=password,
+            schema_password=schema_password,
+        )
 
         if not self._migrate_schema_bootstrap():
             return
@@ -419,10 +487,12 @@ class LandscapeServerCharm(CharmBase):
     def _migrate_schema_bootstrap(self):
         """
         Migrates schema along with the bootstrap command which ensures that the
-        databases along with the landscape user exists. Returns True on success
+        databases along with the landscape user exists. In addition creates
+        admin if configured. Returns True on success
         """
         try:
             check_call([SCHEMA_SCRIPT, "--bootstrap"])
+            self._bootstrap_account()
             return True
         except CalledProcessError as e:
             logger.error(
@@ -435,10 +505,12 @@ class LandscapeServerCharm(CharmBase):
         self._stored.ready["amqp"] = False
         self.unit.status = MaintenanceStatus("Setting up amqp connection")
 
-        event.relation.data[self.unit].update({
-            "username": "landscape",
-            "vhost": "landscape",
-        })
+        event.relation.data[self.unit].update(
+            {
+                "username": "landscape",
+                "vhost": "landscape",
+            }
+        )
 
     def _amqp_relation_changed(self, event):
         unit_data = event.relation.data[event.unit]
@@ -453,12 +525,14 @@ class LandscapeServerCharm(CharmBase):
         if isinstance(hostname, list):
             hostname = ",".join(hostname)
 
-        update_service_conf({
-            "broker": {
-                "host": hostname,
-                "password": password,
+        update_service_conf(
+            {
+                "broker": {
+                    "host": hostname,
+                    "password": password,
+                }
             }
-        })
+        )
 
         self._stored.ready["amqp"] = True
         self.unit.status = ActiveStatus("Unit is ready")
@@ -471,11 +545,13 @@ class LandscapeServerCharm(CharmBase):
         if not self.model.config.get("root_url"):
             url = f'https://{event.relation.data[event.unit]["public-address"]}/'
             self._stored.default_root_url = url
-            update_service_conf({
-                "global": {"root-url": url},
-                "api": {"root-url": url},
-                "package-upload": {"root-url": url},
-            })
+            update_service_conf(
+                {
+                    "global": {"root-url": url},
+                    "api": {"root-url": url},
+                    "package-upload": {"root-url": url},
+                }
+            )
 
         self._update_ready_status()
 
@@ -491,7 +567,8 @@ class LandscapeServerCharm(CharmBase):
         if ssl_cert != "DEFAULT" and ssl_key == "":
             # We have a cert but no key, this is an error.
             self.unit.status = BlockedStatus(
-                "`ssl_cert` is specified but `ssl_key` is missing")
+                "`ssl_cert` is specified but `ssl_key` is missing"
+            )
             return
 
         if ssl_cert != "DEFAULT":
@@ -501,8 +578,8 @@ class LandscapeServerCharm(CharmBase):
                 ssl_cert = b64encode(ssl_cert + b"\n" + ssl_key)
             except binascii.Error:
                 self.unit.status = BlockedStatus(
-                    "Unable to decode `ssl_cert` or `ssl_key` - must be "
-                    "b64-encoded")
+                    "Unable to decode `ssl_cert` or `ssl_key` - must be b64-encoded"
+                )
                 return
 
         with open(HAPROXY_CONFIG_FILE) as haproxy_config_file:
@@ -512,68 +589,83 @@ class LandscapeServerCharm(CharmBase):
         https_service = haproxy_config["https_service"]
         https_service["crts"] = [ssl_cert]
 
-        if self.unit.is_leader():
-            https_service["service_options"].extend(
-                haproxy_config["leader_service_options"])
-
         server_ip = relation.data[self.unit]["private-address"]
         unit_name = self.unit.name.replace("/", "-")
         worker_counts = self.model.config["worker_counts"]
 
         (appservers, pingservers, message_servers, api_servers) = [
-            [(
-                f"landscape-{name}-{unit_name}-{i}",
-                server_ip,
-                haproxy_config["ports"][name] + i,
-                haproxy_config["server_options"],
-            ) for i in range(worker_counts)]
+            [
+                (
+                    f"landscape-{name}-{unit_name}-{i}",
+                    server_ip,
+                    haproxy_config["ports"][name] + i,
+                    haproxy_config["server_options"],
+                )
+                for i in range(worker_counts)
+            ]
             for name in ("appserver", "pingserver", "message-server", "api")
         ]
 
         # There should only ever be one package-upload-server service.
-        package_upload_servers = [(
-            f"landscape-package-upload-{unit_name}-0",
-            server_ip,
-            haproxy_config["ports"]["package-upload"],
-            haproxy_config["server_options"],
-        )]
+        package_upload_servers = [
+            (
+                f"landscape-package-upload-{unit_name}-0",
+                server_ip,
+                haproxy_config["ports"]["package-upload"],
+                haproxy_config["server_options"],
+            )
+        ]
 
         http_service["servers"] = appservers
-        http_service["backends"] = [{
-            "backend_name": "landscape-ping",
-            "servers": pingservers,
-        }]
+        http_service["backends"] = [
+            {
+                "backend_name": "landscape-ping",
+                "servers": pingservers,
+            }
+        ]
         https_service["servers"] = appservers
-        https_service["backends"] = [{
-            "backend_name": "landscape-message",
-            "servers": message_servers,
-        }, {
-            "backend_name": "landscape-api",
-            "servers": api_servers,
-        }]
-
-        if self.unit.is_leader():
-            https_service["backends"].append({
+        https_service["backends"] = [
+            {
+                "backend_name": "landscape-message",
+                "servers": message_servers,
+            },
+            {
+                "backend_name": "landscape-api",
+                "servers": api_servers,
+            },
+            # Only the leader should have servers for the landscape-package-upload
+            # and landscape-hashid-databases backends. However, when the leader
+            # is lost, haproxy will fail as the service options will reference
+            # a (no longer) existing backend. To prevent that, all units should
+            # declare all backends, even if a unit should not have any servers on
+            # a specific backend.
+            {
                 "backend_name": "landscape-package-upload",
-                "servers": package_upload_servers,
-            })
+                "servers": package_upload_servers if self.unit.is_leader() else [],
+            },
+            {
+                "backend_name": "landscape-hashid-databases",
+                "servers": appservers if self.unit.is_leader() else [],
+            },
+        ]
 
         error_files_location = haproxy_config["error_files"]["location"]
         error_files = []
         for code, filename in haproxy_config["error_files"]["files"].items():
             error_file_path = os.path.join(error_files_location, filename)
             with open(error_file_path, "rb") as error_file:
-                error_files.append({
-                    "http_status": code,
-                    "content": b64encode(error_file.read())
-                })
+                error_files.append(
+                    {"http_status": code, "content": b64encode(error_file.read())}
+                )
 
         http_service["error_files"] = error_files
         https_service["error_files"] = error_files
 
-        relation.data[self.unit].update({
-            "services": yaml.safe_dump([http_service, https_service])
-        })
+        relation.data[self.unit].update(
+            {
+                "services": yaml.safe_dump([http_service, https_service]),
+            }
+        )
 
         self._stored.ready["haproxy"] = True
 
@@ -596,19 +688,24 @@ class LandscapeServerCharm(CharmBase):
         self.unit.status = MaintenanceStatus("Configuring HAProxy")
         haproxy_ssl_cert = event.relation.data[event.unit]["ssl_cert"]
 
+        # Sometimes the data has not been encoded properly in the HA charm
+        if haproxy_ssl_cert.startswith("b'"):
+            haproxy_ssl_cert = haproxy_ssl_cert.strip('b').strip("'")
+
         if haproxy_ssl_cert != "DEFAULT":
             # If DEFAULT, cert is being managed by a third party,
             # possibly a subordinate charm.
             write_ssl_cert(haproxy_ssl_cert)
 
         self.unit.status = ActiveStatus("Unit is ready")
-
         self._update_haproxy_connection(event.relation)
 
         self._update_ready_status()
 
-    def _nrpe_external_master_relation_joined(
-            self, event: RelationJoinedEvent) -> None:
+    def _website_relation_departed(self, event: RelationDepartedEvent) -> None:
+        event.relation.data[self.unit].update({"services": ""})
+
+    def _nrpe_external_master_relation_joined(self, event: RelationJoinedEvent) -> None:
         self._update_nrpe_checks(event.relation)
 
     def _update_nrpe_checks(self, relation: Relation):
@@ -624,16 +721,16 @@ class LandscapeServerCharm(CharmBase):
         monitors = {
             "monitors": {
                 "remote": {
-                    "nrpe": {
-                        s: {"command": f"check_{s}"} for s in services_to_add
-                    },
+                    "nrpe": {s: {"command": f"check_{s}"} for s in services_to_add},
                 },
             },
         }
 
-        relation.data[self.unit].update({
-            "monitors": yaml.safe_dump(monitors),
-        })
+        relation.data[self.unit].update(
+            {
+                "monitors": yaml.safe_dump(monitors),
+            }
+        )
 
         if not os.path.exists(NRPE_D_DIR):
             logger.debug("NRPE directories not ready")
@@ -647,12 +744,14 @@ class LandscapeServerCharm(CharmBase):
                 continue
 
             with open(cfg_filename, "w") as cfg_fp:
-                cfg_fp.write(f"""# check {service}
+                cfg_fp.write(
+                    f"""# check {service}
 # The following header was added by the landscape-server charm
 # Modifying it will affect nagios monitoring and alerting
 # servicegroups: juju
 command[check_{service}]=/usr/local/lib/nagios/plugins/check_systemd.py {service}
-""")
+"""
+                )
 
         for service in services_to_remove:
             service_cfg = service.replace("-", "_")
@@ -673,7 +772,8 @@ command[check_{service}]=/usr/local/lib/nagios/plugins/check_systemd.py {service
 
         if not root_url:
             root_url = "https://" + str(
-                self.model.get_binding(event.relation).network.bind_address)
+                self.model.get_binding(event.relation).network.bind_address
+            )
 
         site_name = self.model.config.get("site_name")
         if site_name:
@@ -690,13 +790,15 @@ command[check_{service}]=/usr/local/lib/nagios/plugins/check_systemd.py {service
         else:
             icon_data = None
 
-        event.relation.data[self.app].update({
-            "name": "Landscape",
-            "url": root_url,
-            "subtitle": subtitle,
-            "group": group,
-            "icon": icon_data,
-        })
+        event.relation.data[self.app].update(
+            {
+                "name": "Landscape",
+                "url": root_url,
+                "subtitle": subtitle,
+                "group": group,
+                "icon": icon_data,
+            }
+        )
 
     def _leader_elected(self, event: LeaderElectedEvent) -> None:
         # Just because we received this event does not mean we are
@@ -709,16 +811,17 @@ command[check_{service}]=/usr/local/lib/nagios/plugins/check_systemd.py {service
             ip = str(self.model.get_binding(peer_relation).network.bind_address)
             peer_relation.data[self.app].update({"leader-ip": ip})
 
-            update_service_conf({
-                "package-search": {
-                    "host": "localhost",
-                },
-            })
+            update_service_conf(
+                {
+                    "package-search": {
+                        "host": "localhost",
+                    },
+                }
+            )
 
         self._leader_changed()
 
-    def _leader_settings_changed(
-            self, event: LeaderSettingsChangedEvent) -> None:
+    def _leader_settings_changed(self, event: LeaderSettingsChangedEvent) -> None:
         # Just because we received this event does not mean we are
         # guaranteed to be a follower by the time we process it. See
         # https://juju.is/docs/sdk/leader-elected-event
@@ -728,11 +831,13 @@ command[check_{service}]=/usr/local/lib/nagios/plugins/check_systemd.py {service
             leader_ip = peer_relation.data[self.app].get("leader-ip")
 
             if leader_ip:
-                update_service_conf({
-                    "package-search": {
-                        "host": leader_ip,
-                    },
-                })
+                update_service_conf(
+                    {
+                        "package-search": {
+                            "host": leader_ip,
+                        },
+                    }
+                )
 
         self._leader_changed()
 
@@ -747,9 +852,10 @@ command[check_{service}]=/usr/local/lib/nagios/plugins/check_systemd.py {service
         for relation in nrpe_relations:
             self._update_nrpe_checks(relation)
 
-        haproxy_relations = self.model.relations.get("website", [])
-        for relation in haproxy_relations:
-            self._update_haproxy_connection(relation)
+        if self.unit.is_leader():
+            haproxy_relations = self.model.relations.get("website", [])
+            for relation in haproxy_relations:
+                self._update_haproxy_connection(relation)
 
         self._update_ready_status(restart_services=True)
 
@@ -760,13 +866,23 @@ command[check_{service}]=/usr/local/lib/nagios/plugins/check_systemd.py {service
 
         event.relation.data[self.unit].update({"unit-data": self.unit.name})
 
-    def _on_replicas_relation_changed(
-            self, event: RelationChangedEvent) -> None:
+    def _on_replicas_relation_changed(self, event: RelationChangedEvent) -> None:
         leader_ip_value = event.relation.data[self.app].get("leader-ip")
 
         if leader_ip_value and leader_ip_value != self._stored.leader_ip:
             self._stored.leader_ip = leader_ip_value
 
+        if self.unit.is_leader():
+            haproxy_relations = self.model.relations.get("website", [])
+            for relation in haproxy_relations:
+                self._update_haproxy_connection(relation)
+
+        secret_token = self._get_secret_token()
+        if (secret_token) and (secret_token != self._stored.secret_token):
+            self._write_secret_token(secret_token)
+            self._stored.secret_token = secret_token
+            self._update_ready_status(restart_services=True)
+
     def _configure_smtp(self, relay_host: str) -> None:
 
         # Rewrite postfix config.
@@ -800,27 +916,32 @@ command[check_{service}]=/usr/local/lib/nagios/plugins/check_systemd.py {service
         none_count = oidc_vals.count(None)
 
         if none_count == 0:
-            update_service_conf({
-                "landscape": {
-                    "oidc-issuer": oidc_issuer,
-                    "oidc-client-id": oidc_client_id,
-                    "oidc-client-secret": oidc_client_secret,
-                    "oidc-logout-url": oidc_logout_url,
-                },
-            })
+            update_service_conf(
+                {
+                    "landscape": {
+                        "oidc-issuer": oidc_issuer,
+                        "oidc-client-id": oidc_client_id,
+                        "oidc-client-secret": oidc_client_secret,
+                        "oidc-logout-url": oidc_logout_url,
+                    },
+                }
+            )
         elif none_count == 1 and oidc_logout_url is None:
             # Only the logout url is optional.
-            update_service_conf({
-                "landscape": {
-                    "oidc-issuer": oidc_issuer,
-                    "oidc-client-id": oidc_client_id,
-                    "oidc-client-secret": oidc_client_secret,
-                },
-            })
+            update_service_conf(
+                {
+                    "landscape": {
+                        "oidc-issuer": oidc_issuer,
+                        "oidc-client-id": oidc_client_id,
+                        "oidc-client-secret": oidc_client_secret,
+                    },
+                }
+            )
         elif none_count < 4:
             self.unit.status = BlockedStatus(
                 "OIDC connect config requires at least 'oidc_issuer', "
-                "'oidc_client_id', and 'oidc_client_secret' values")
+                "'oidc_client_id', and 'oidc_client_secret' values"
+            )
             return
 
         self.unit.status = WaitingStatus("Waiting on relations")
@@ -832,17 +953,20 @@ command[check_{service}]=/usr/local/lib/nagios/plugins/check_systemd.py {service
         openid_logout_url = self.model.config.get("openid_logout_url")
 
         if openid_provider_url and openid_logout_url:
-            update_service_conf({
-                "landscape": {
-                    "openid-provider-url": openid_provider_url,
-                    "openid-logout-url": openid_logout_url,
-                },
-            })
+            update_service_conf(
+                {
+                    "landscape": {
+                        "openid-provider-url": openid_provider_url,
+                        "openid-logout-url": openid_logout_url,
+                    },
+                }
+            )
             self.unit.status = WaitingStatus("Waiting on relations")
         elif openid_provider_url or openid_logout_url:
             self.unit.status = BlockedStatus(
                 "OpenID configuration requires both 'openid_provider_url' and "
-                "'openid_logout_url'")
+                "'openid_logout_url'"
+            )
 
     def _bootstrap_account(self):
         """If admin account details are provided, create admin"""
@@ -906,8 +1030,7 @@ command[check_{service}]=/usr/local/lib/nagios/plugins/check_systemd.py {service
         try:
             check_call([LSCTL, "stop"])
         except CalledProcessError as e:
-            logger.error("Stopping services failed with return code %d",
-                         e.returncode)
+            logger.error("Stopping services failed with return code %d", e.returncode)
             self.unit.status = BlockedStatus("Failed to stop services")
             event.fail("Failed to stop services")
         else:
@@ -919,14 +1042,12 @@ command[check_{service}]=/usr/local/lib/nagios/plugins/check_systemd.py {service
         self.unit.status = MaintenanceStatus("Starting services")
         event.log("Starting services")
 
-        start_result = subprocess.run([LSCTL, "start"], capture_output=True,
-                                      text=True)
+        start_result = subprocess.run([LSCTL, "start"], capture_output=True, text=True)
 
         try:
             check_call([LSCTL, "status"])
         except CalledProcessError as e:
-            logger.error("Starting services failed with return code %d",
-                         e.returncode)
+            logger.error("Starting services failed with return code %d", e.returncode)
             logger.error("Failed to start services: %s", start_result.stdout)
             self.unit.status = MaintenanceStatus("Stopping services")
             subprocess.run([LSCTL, "stop"])
@@ -940,8 +1061,10 @@ command[check_{service}]=/usr/local/lib/nagios/plugins/check_systemd.py {service
 
     def _upgrade(self, event: ActionEvent) -> None:
         if self._stored.running:
-            event.fail("Cannot upgrade while running. Please run action "
-                       "'pause' prior to upgrade")
+            event.fail(
+                "Cannot upgrade while running. Please run action "
+                "'pause' prior to upgrade"
+            )
             return
 
         prev_status = self.unit.status
@@ -953,12 +1076,18 @@ command[check_{service}]=/usr/local/lib/nagios/plugins/check_systemd.py {service
         for package in LANDSCAPE_PACKAGES:
             try:
                 event.log(f"Upgrading {package}...")
+                if package == LANDSCAPE_SERVER:
+                    check_call(["apt-mark", "unhold", LANDSCAPE_SERVER])
                 pkg = apt.DebianPackage.from_apt_cache(package)
                 pkg.ensure(state=apt.PackageState.Latest)
                 installed = apt.DebianPackage.from_installed_package(package)
                 event.log(f"Upgraded to {installed.version}...")
+                if package == LANDSCAPE_SERVER:
+                    check_call(["apt-mark", "hold", LANDSCAPE_SERVER])
             except PackageNotFoundError as e:
-                logger.error(f"Could not upgrade package {package}. Reason: {e.message}")
+                logger.error(
+                    f"Could not upgrade package {package}. Reason: {e.message}"
+                )
                 event.fail(f"Could not upgrade package {package}. Reason: {e.message}")
                 self.unit.status = BlockedStatus("Failed to upgrade packages")
                 return
@@ -967,8 +1096,10 @@ command[check_{service}]=/usr/local/lib/nagios/plugins/check_systemd.py {service
 
     def _migrate_schema(self, event: ActionEvent) -> None:
         if self._stored.running:
-            event.fail("Cannot migrate schema while running. Please run action"
-                       " 'pause' prior to migration")
+            event.fail(
+                "Cannot migrate schema while running. Please run action"
+                " 'pause' prior to migration"
+            )
             return
 
         prev_status = self.unit.status
@@ -978,10 +1109,8 @@ command[check_{service}]=/usr/local/lib/nagios/plugins/check_systemd.py {service
         try:
             subprocess.run([SCHEMA_SCRIPT], check=True, text=True)
         except CalledProcessError as e:
-            logger.error("Schema migration failed with error code %s",
-                         e.returncode)
-            event.fail("Schema migration failed with error code %s",
-                       e.returncode)
+            logger.error("Schema migration failed with error code %s", e.returncode)
+            event.fail("Schema migration failed with error code %s", e.returncode)
             self.unit.status = BlockedStatus("Failed schema migration")
         else:
             self.unit.status = prev_status
@@ -992,7 +1121,9 @@ command[check_{service}]=/usr/local/lib/nagios/plugins/check_systemd.py {service
         event.log("Running hash_id_databases")
 
         try:
-            subprocess.run(["sudo", "-u", "landscape", HASH_ID_DATABASES], check=True, text=True)
+            subprocess.run(
+                ["sudo", "-u", "landscape", HASH_ID_DATABASES], check=True, text=True
+            )
         except CalledProcessError as e:
             logger.error("Hashing ID databases failed with error code %s", e.returncode)
             event.fail("Hashing ID databases failed with error code %s", e.returncode)
diff --git a/src/haproxy-config.yaml b/src/haproxy-config.yaml
index e986f53..030fc6e 100644
--- a/src/haproxy-config.yaml
+++ b/src/haproxy-config.yaml
@@ -34,11 +34,11 @@ https_service:
     - use_backend landscape-message if attachment
     - use_backend landscape-api if api
     - use_backend landscape-ping if ping
-
-leader_service_options:
-  - acl package-upload path_beg -i /upload
-  - use_backend landscape-package-upload if package-upload
-  - reqrep ^([^\ ]*)\ /upload/(.*) \1\ /\2
+    - acl hashids path_beg -i /hash-id-databases
+    - use_backend landscape-hashid-databases if hashids
+    - acl package-upload path_beg -i /upload
+    - use_backend landscape-package-upload if package-upload
+    - http-request replace-path ^([^\ ]*)\ /upload/(.*) \1\ /\2
 
 error_files:
   location: /opt/canonical/landscape/canonical/landscape/offline
@@ -55,11 +55,10 @@ ports:
   message-server: 8090
   api: 9080
   package-upload: 9100
-  pppa-proxy: 9298
 
 server_options:
   - check
   - inter 5000
   - rise 2
   - fall 5
-  - maxconn 50
\ No newline at end of file
+  - maxconn 50
diff --git a/src/settings_files.py b/src/settings_files.py
index 6eb7231..b4d1189 100644
--- a/src/settings_files.py
+++ b/src/settings_files.py
@@ -9,6 +9,8 @@ import os
 from base64 import b64decode, binascii
 from collections import defaultdict
 from configparser import ConfigParser
+import secrets
+from string import ascii_letters, digits
 from urllib.request import urlopen
 from urllib.error import URLError
 
@@ -37,6 +39,14 @@ class SSLCertReadException(Exception):
     pass
 
 
+class ServiceConfMissing(Exception):
+    pass
+
+
+class SecretTokenMissing(Exception):
+    pass
+
+
 def configure_for_deployment_mode(mode: str) -> None:
     """
     Places files where Landscape expects to find them for different deployment
@@ -111,6 +121,11 @@ def update_service_conf(updates: dict) -> None:
     `updates` is a mapping of {section => {key => value}}, to be applied
         to the config file.
     """
+    if not os.path.isfile(SERVICE_CONF):
+        # Landscape server will not overwrite this file on install, so we
+        # cannot get the default values if we create it here
+        raise ServiceConfMissing("Landscape server install failed!")
+
     config = ConfigParser()
     config.read(SERVICE_CONF)
 
@@ -125,6 +140,11 @@ def update_service_conf(updates: dict) -> None:
         config.write(config_fp)
 
 
+def generate_secret_token():
+    alphanumerics = ascii_letters + digits
+    return "".join(secrets.choice(alphanumerics) for _ in range(172))
+
+
 def write_license_file(license_file: str, uid: int, gid: int) -> None:
     """
     Reads or decodes `license_file` to LICENSE_FILE and sets it up
diff --git a/tests/test_charm.py b/tests/test_charm.py
index 08a4119..7ee1671 100644
--- a/tests/test_charm.py
+++ b/tests/test_charm.py
@@ -24,7 +24,8 @@ from charms.operator_libs_linux.v0.apt import (
 
 from charm import (
     DEFAULT_SERVICES, HAPROXY_CONFIG_FILE, LANDSCAPE_PACKAGES, LEADER_SERVICES, LSCTL,
-    NRPE_D_DIR, SCHEMA_SCRIPT, HASH_ID_DATABASES, LandscapeServerCharm)
+    NRPE_D_DIR, SCHEMA_SCRIPT, HASH_ID_DATABASES, LandscapeServerCharm,
+    )
 
 
 class TestCharm(unittest.TestCase):
@@ -71,10 +72,13 @@ class TestCharm(unittest.TestCase):
         with patches as mocks:
             harness.begin_with_initial_hooks()
 
-        mocks["check_call"].assert_called_once_with(
+        mocks["check_call"].assert_any_call(
             ["add-apt-repository", "-y", ppa])
-        mocks["apt"].add_package.assert_called_once_with(["landscape-server", 
-            "landscape-hashids"])
+        mocks["check_call"].assert_any_call(
+            ["apt-mark", "hold", "landscape-hashids", "landscape-server"])
+        mocks["apt"].add_package.assert_called_once_with(
+            ["landscape-server", "landscape-hashids"], update_cache=True,
+        )
         status = harness.charm.unit.status
         self.assertIsInstance(status, WaitingStatus)
         self.assertEqual(status.message,
@@ -95,11 +99,8 @@ class TestCharm(unittest.TestCase):
 
         with patches as mocks:
             mocks["apt"].add_package.side_effect = PackageNotFoundError
-            harness.begin_with_initial_hooks()
-
-        status = harness.charm.unit.status
-        self.assertIsInstance(status, BlockedStatus)
-        self.assertEqual(status.message, "Failed to install packages")
+            self.assertRaises(PackageNotFoundError,
+                harness.begin_with_initial_hooks)
 
     def test_install_package_error(self):
         harness = Harness(LandscapeServerCharm)
@@ -116,11 +117,7 @@ class TestCharm(unittest.TestCase):
 
         with patches as mocks:
             mocks["apt"].add_package.side_effect = PackageError("ouch")
-            harness.begin_with_initial_hooks()
-
-        status = harness.charm.unit.status
-        self.assertIsInstance(status, BlockedStatus)
-        self.assertEqual(status.message, "Failed to install packages")
+            self.assertRaises(PackageError, harness.begin_with_initial_hooks)
 
     def test_install_called_process_error(self):
         harness = Harness(LandscapeServerCharm)
@@ -131,11 +128,8 @@ class TestCharm(unittest.TestCase):
         with patch("charm.check_call") as mock:
             with patch("charm.update_service_conf"):
                 mock.side_effect = CalledProcessError(127, Mock())
-                harness.begin_with_initial_hooks()
-
-        status = harness.charm.unit.status
-        self.assertIsInstance(status, BlockedStatus)
-        self.assertEqual(status.message, "Failed to install packages")
+                self.assertRaises(CalledProcessError, 
+                    harness.begin_with_initial_hooks)
 
     def test_install_ssl_cert(self):
         harness = Harness(LandscapeServerCharm)
@@ -188,20 +182,28 @@ class TestCharm(unittest.TestCase):
 
     def test_install_license_file_b64(self):
         harness = Harness(LandscapeServerCharm)
-        harness.update_config({"license_file": "VEhJUyBJUyBBIExJQ0VOU0U="})
+        license_text = "VEhJUyBJUyBBIExJQ0VOU0U"
+        harness.update_config({"license_file": license_text})
         relation_id = harness.add_relation("replicas", "landscape-server")
         harness.update_relation_data(
             relation_id, "landscape-server", {"leader-ip": "test"})
 
         with patch.multiple(
                 "charm",
+                apt=DEFAULT,
+                check_call=DEFAULT,
                 update_service_conf=DEFAULT,
+                prepend_default_settings=DEFAULT,
                 write_license_file=DEFAULT,
         ) as mocks:
             harness.begin_with_initial_hooks()
 
-        mocks["write_license_file"].assert_called_once_with(
-            "VEhJUyBJUyBBIExJQ0VOU0U=", 1000, 1000)
+        mock_write = mocks["write_license_file"]
+        self.assertEqual(len(mock_write.mock_calls), 2)
+        self.assertEqual(mock_write.mock_calls[0].args,
+            (license_text, 1000, 1000))
+        self.assertEqual(mock_write.mock_calls[1].args,
+            (license_text, 1000, 1000))
 
     def test_update_ready_status_not_running(self):
         self.harness.charm.unit.status = WaitingStatus()
@@ -505,6 +507,7 @@ class TestCharm(unittest.TestCase):
                 "password": "testpass",
             },
         }
+        self.harness.add_relation("replicas", "landscape-server")
 
         with patch("charm.check_call"):
             with patch(
@@ -809,10 +812,34 @@ class TestCharm(unittest.TestCase):
         self.assertIsInstance(status, WaitingStatus)
         write_cert_mock.assert_called_once_with("FANCYNEWCERT")
 
+    def test_website_relation_changed_strip_b_char(self):
+        self.harness.charm._update_haproxy_connection = Mock()
+        mock_event = Mock()
+        mock_event.relation.data = {
+            mock_event.unit: {"ssl_cert": "b'FANCYNEWCERT'"},
+            self.harness.charm.unit: {
+                "private-address": "test",
+                "public-address": "test2",
+            },
+        }
+
+        with patch.multiple(
+                "charm",
+                write_ssl_cert=DEFAULT,
+                update_service_conf=DEFAULT,
+        ) as mocks:
+            write_cert_mock = mocks["write_ssl_cert"]
+            self.harness.charm._website_relation_changed(mock_event)
+
+        status = self.harness.charm.unit.status
+        self.assertIsInstance(status, WaitingStatus)
+        write_cert_mock.assert_called_once_with("FANCYNEWCERT")
+
     @patch("charm.update_service_conf")
     def test_on_config_changed_no_smtp_change(self, _):
         self.harness.charm._update_ready_status = Mock()
         self.harness.charm._configure_smtp = Mock()
+        self.harness.add_relation("replicas", "landscape-server")
         self.harness.update_config({"smtp_relay_host": ""})
 
         self.harness.charm._configure_smtp.assert_not_called()
@@ -822,6 +849,7 @@ class TestCharm(unittest.TestCase):
     def test_on_config_changed_smtp_change(self, _):
         self.harness.charm._update_ready_status = Mock()
         self.harness.charm._configure_smtp = Mock()
+        self.harness.add_relation("replicas", "landscape-server")
         self.harness.update_config({"smtp_relay_host": "smtp.example.com"})
 
         self.harness.charm._configure_smtp.assert_called_once_with(
@@ -932,11 +960,12 @@ class TestCharm(unittest.TestCase):
         prev_status = self.harness.charm.unit.status
 
         with patch("charm.apt", spec_set=apt) as apt_mock:
-            pkg_mock = Mock()
-            apt_mock.DebianPackage.from_apt_cache.return_value = pkg_mock
-            self.harness.charm._upgrade(event)
+            with patch("charm.check_call"):
+                pkg_mock = Mock()
+                apt_mock.DebianPackage.from_apt_cache.return_value = pkg_mock
+                self.harness.charm._upgrade(event)
 
-        self.assertEqual(event.log.call_count, 9)
+        self.assertGreaterEqual(event.log.call_count, 5)
         self.assertEqual(
             apt_mock.DebianPackage.from_apt_cache.call_count,
             len(LANDSCAPE_PACKAGES)
@@ -963,10 +992,11 @@ class TestCharm(unittest.TestCase):
         self.harness.charm._stored.running = False
 
         with patch("charm.apt", spec_set=apt) as apt_mock:
-            pkg_mock = Mock()
-            apt_mock.DebianPackage.from_apt_cache.return_value = pkg_mock
-            pkg_mock.ensure.side_effect = PackageNotFoundError("ouch")
-            self.harness.charm._upgrade(event)
+            with patch("charm.check_call"):
+                pkg_mock = Mock()
+                apt_mock.DebianPackage.from_apt_cache.return_value = pkg_mock
+                pkg_mock.ensure.side_effect = PackageNotFoundError("ouch")
+                self.harness.charm._upgrade(event)
 
         self.assertEqual(event.log.call_count, 2)
         event.fail.assert_called_once()
@@ -1149,6 +1179,13 @@ class TestBootstrapAccount(unittest.TestCase):
         )
         self.harness.add_relation("replicas", "landscape-server")
         self.harness.set_leader()
+        
+        pwd_mock = patch("charm.user_exists").start()
+        pwd_mock.return_value = Mock(
+            spec_set=struct_passwd, pw_uid=1000)
+        grp_mock = patch("charm.group_exists").start()
+        grp_mock.return_value = Mock(
+            spec_set=struct_group, gr_gid=1000)
 
         self.process_mock = patch("subprocess.run").start()
         self.log_mock = patch("charm.logger.error").start()
diff --git a/tests/test_settings_files.py b/tests/test_settings_files.py
index e16f8f7..84b4ffc 100644
--- a/tests/test_settings_files.py
+++ b/tests/test_settings_files.py
@@ -210,9 +210,11 @@ class UpdateServiceConfTestCase(TestCase):
             i += 1
             return retval
 
-        with patch("builtins.open") as open_mock:
-            open_mock.side_effect = return_conf
-            update_service_conf({"test": {"new": "yes"}})
+        with patch("os.path.isfile") as mock_isfile:
+            with patch("builtins.open") as open_mock:
+                mock_isfile.return_value = True
+                open_mock.side_effect = return_conf
+                update_service_conf({"test": {"new": "yes"}})
 
         self.assertEqual(outfile.captured,
                          "[fixed]\nold = no\n\n[test]\nnew = yes\n\n")
@@ -230,9 +232,11 @@ class UpdateServiceConfTestCase(TestCase):
             i += 1
             return retval
 
-        with patch("builtins.open") as open_mock:
-            open_mock.side_effect = return_conf
-            update_service_conf({"fixed": {"old": "yes"}})
+        with patch("os.path.isfile") as mock_isfile:
+            with patch("builtins.open") as open_mock:
+                mock_isfile.return_value = True
+                open_mock.side_effect = return_conf
+                update_service_conf({"fixed": {"old": "yes"}})
 
         self.assertEqual(outfile.captured, "[fixed]\nold = yes\n\n")