Merge charm-storage-connector:defer-service-restarts-integration into charm-storage-connector:master

[Tianqi Xiao]

Add defer methods for better handling service restarts

This MP fixes LP#2003241. The following changes have been made:

- Make use of charmhelpers policy_rcd script to block automatic stop, restarts, try-restarts action for all charm managed services, including iscsid open-iscsi, and multipathd.
- Handles deferred restarts triggered by both charm and non-charm events
- Display the queue of deferred service restarts in charm status message
and periodic updates
- Add `enable-auto-restarts` config option to give user option to allow automatic restarts action if needed (the default value is False)
- Add `restart-services` action to restart charm managed service. The
operator can choose to restart either all deferred services or a specific
set of services.
- Add `show-deferred-restarts` action to show all deferred restart events
with the timestamp and reason
- Add `iscsi-discovery-and-login` action that allow operator to manually
run iscsiadm discovery and login when needed
- Remove `restart-iscsi-services` action in favor of the new
`restart-services` action
- Separate charm status logic from the state of its underlying services
- Add unittests and functional tests for the new and modified methods
- Rework functional tests logic to make each test case independent of each other.

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:37b83ef6eab6b315e36e14bc53961e4af918d089
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/17/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/268/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/96/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/17//rebuild

review: Approve (continuous-integration)

[Andrea Ieri (aieri)]

Can you please link to the original MPs that generated the integration branch you're trying to merge? I think it would be useful to note them for tracking purposes.

[Tianqi Xiao (txiao)]

@aieri Sure, thanks for the suggestion.

The original MPs that generated the integration branch are #438042 [1] and #438419 [2].

[1] https://code.launchpad.net/~txiao/charm-storage-connector/+git/charm-storage-connector/+merge/438042
[2] https://code.launchpad.net/~txiao/charm-storage-connector/+git/charm-storage-connector/+merge/438419

[Andrea Ieri (aieri)]

lgtm, thanks

review: Approve

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change successfully merged at revision 0ac891951896d6133ed80e7f2759f7f403b99918

diff --git a/README.md b/README.md
index 247b96d..2802aab 100644
--- a/README.md
+++ b/README.md
@@ -2,10 +2,10 @@
 
 ## Overview
 
-This charm configures a unit to connect to a storage endpoint, either iSCSI or Fibre Channel. 
-It acts as a subordinate charm, which can be deployed on any bare metal or virtual machine, 
-alongside a main charm. It is not supported in containers due to lack of ability for lxd 
-containers to access iSCSI and Fibre Channel hardware of the underlying kernel. 
+This charm configures a unit to connect to a storage endpoint, either iSCSI or Fibre Channel.
+It acts as a subordinate charm, which can be deployed on any bare metal or virtual machine,
+alongside a main charm. It is not supported in containers due to lack of ability for lxd
+containers to access iSCSI and Fibre Channel hardware of the underlying kernel.
 
 If you configure this charm for iSCSI, this charm will:
 - Generate an iSCSI initiator name and put it in /etc/iscsi/initiatorname.iscsi
@@ -23,11 +23,11 @@ If you configure it for Fibre Channel, this charm will:
 - Configure multipath under /etc/multipath/conf.d directory
 - Reload and restart multipathd.service
 
-Please note that the Fiber Channel option currently supports only one device. 
+Please note that the Fiber Channel option currently supports only one device.
 
 If iSCSI, the user can input a initiator name dictionary in config.yaml if they wish to use a
 specific IQN for a specific unit. Also, the target IP and port are needed to perform
-the discovery and login with iscsiadm. 
+the discovery and login with iscsiadm.
 
 For Fibre Channel, the user can choose the device alias to be used when mapping the disk.
 See the option "fc-lun-alias" for further details.
@@ -85,9 +85,9 @@ juju relate ubuntu storage-connector
 
 ## Scaling
 
-This charm will scale with the units it is related to. For example, if you scale the 
-ubuntu application, and the storage-connector is related to it, it will be deployed on each 
-newly deployed ubuntu unit. 
+This charm will scale with the units it is related to. For example, if you scale the
+ubuntu application, and the storage-connector is related to it, it will be deployed on each
+newly deployed ubuntu unit.
 ```
 juju add-unit ubuntu
 juju remove-unit ubuntu/1
@@ -104,4 +104,3 @@ cause of the error.
  - Author: Camille Rodriguez <[e-mail]>
  - Maintainers: BootStack Charmers <[e-mail]>
  - Bug Tracker: [here](https://bugs.launchpad.net/charm-storage-connector)
- 
\ No newline at end of file
diff --git a/actions.yaml b/actions.yaml
index 998a670..79eed26 100644
--- a/actions.yaml
+++ b/actions.yaml
@@ -1,4 +1,33 @@
-restart-iscsi-services:
-  description: "Restart iscsid and open-iscsi services"
+restart-services:
+  description: |
+      Restart services managed by the charm.
+  params:
+    deferred-only:
+      type: boolean
+      default: false
+      description: |
+        Restart all deferred services. To check which services has been deferred,
+        run the `show-deferred-events` action.
+        NOTE: deferred-only and services params are mutually exclusive. Providing
+        both options will cause the action to fail.
+    services:
+      type: string
+      default: ""
+      description: |
+        List of services to restart. Supporting services includes iscsi,
+        open-iscsi, and multipathd.
+        This param accepts space-separated services in string format,
+        e.g. "iscsid open-iscsi"
+        NOTE: deferred-only and services params are mutually exclusive. Providing
+        both options will cause the action to fail.
+show-deferred-restarts:
+  descrpition: |
+    Show the outstanding service restarts
 reload-multipathd-service:
-  description: "Reload multipathd service"
+  description: |
+    Reload multipathd service
+iscsi-discovery-and-login:
+  description: |
+    Run discovery and login against iscsi target. This action is needed when
+    changes are made to iscsi configuration and the iscsi services
+    are restarted to apply these changes.
diff --git a/config.yaml b/config.yaml
index 739ef42..c962a7f 100644
--- a/config.yaml
+++ b/config.yaml
@@ -8,7 +8,7 @@ options:
         type: string
         default: data1
         description: |
-            LUN alias to give to the WWID for the mapping of the device. It is used in the path of the new mounted device, i.e /dev/mapper/<FC-LUN-ALIAS>. 
+            LUN alias to give to the WWID for the mapping of the device. It is used in the path of the new mounted device, i.e /dev/mapper/<FC-LUN-ALIAS>.
     iscsi-target:
         type: string
         default:
@@ -21,42 +21,49 @@ options:
         type: boolean
         default: True
         description: |
-            If set to True (default) and storage-type is iscsi, the charm will configure 
-            the iscsid and multipath configuration files, and then run an iscsiadm 
-            discovery and login against the target. If set to False, the charm will simply 
-            configure the unit but not run the discovery and login against the target. 
+            If set to True (default) and storage-type is iscsi, the charm will run an
+            iscsiadm discovery and login against the target after charm-managed iscsi
+            services restart (triggered by either config changes or by the restart-services
+            action). If set to False, the charm will not automatically run the discovery
+            and login against the target.
     initiator-dictionary:
         type: string
         default: '{}'
         description: |
-            Dictionary of hostnames (FQDN) and initiator names, surrounded by 
-            single quotes. The charm compares the machine hostname to 
-            the list, and provide the correct initiator associated to 
-            it. If not defined, the initiator name will be provided 
-            randomly. Format : 
+            Dictionary of hostnames (FQDN) and initiator names, surrounded by
+            single quotes. The charm compares the machine hostname to
+            the list, and provide the correct initiator associated to
+            it. If not defined, the initiator name will be provided
+            randomly. Format :
             '{"hostname1": "iqn.yyyy-mm.naming-authority:uniquename1",
               "hostname2": "iqn.yyyy-mm.naming-authority:uniquename2}'
+    enable-auto-restarts:
+        type: boolean
+        default: False
+        description: |
+            Allow the charm and packages to restart services automatically when
+            required.
     multipath-defaults:
         type: string
         default: '{"user_friendly_names": "yes"}'
         description: |
             In multipath config, sets the defaults configuration. String should be of JSON dictionary format.
             Double quotes are essential to the correct format of this JSON string.
-            Example: 
+            Example:
                 value : '{"user_friendly_names":"yes", "find_multipaths":"yes", "polling_interval":"10"}'
                 Which will produce this configuration:
-                defaults { 
-                    user_friendly_names yes 
-                    find_multipaths yes 
+                defaults {
+                    user_friendly_names yes
+                    find_multipaths yes
                     polling_interval 10
-                }                       
+                }
     multipath-devices:
         type: string
         default:
         description: |
             In multipath config, add a device specific configuration. String should be of JSON dictionary format.
             Double quotes are essential to the correct format of this JSON string.
-            Example: 
+            Example:
                 value : '{"vendor":"PURE","product": "FlashArray","fast_io_fail_tmo": "10", "path_grouping_policy":"group_by_prio"}'
                 Which will produce this configuration:
                 device {
@@ -67,9 +74,9 @@ options:
                 }
     multipath-blacklist:
         type: string
-        default: 
+        default:
         description: |
-            In multipath config, add a blacklist device section to exlude local disks from being handled by multipath-tools. It is possible
+            In multipath config, add a blacklist device section to exclude local disks from being handled by multipath-tools. It is possible
             to blacklist by vendor/product, by devnode or WWID. String should be of JSON dictionary format. Double quotes are essential to the correct format of this JSON string.
             Example:
                 value: '{"vendor": "QEMU", "product": "*"}'
diff --git a/metadata.yaml b/metadata.yaml
index 81f963d..1ab3f3d 100644
--- a/metadata.yaml
+++ b/metadata.yaml
@@ -5,8 +5,8 @@ maintainers:
 description: |
     The Storage Connector charm is an open-source charm that can be installed as a
     subordinate to any VM or bare metal machine. It has two modes: iSCSI or Fibre Channel.
-    In the iSCSI mode, it will connect the host to an iSCSI target and configure multipath 
-    accordingly. For Fibre Channel, this charm detects the WWID of the FC LUNs and 
+    In the iSCSI mode, it will connect the host to an iSCSI target and configure multipath
+    accordingly. For Fibre Channel, this charm detects the WWID of the FC LUNs and
     configures multipathd so that the device is ready to be used.
 tags:
     - iscsi
diff --git a/requirements.txt b/requirements.txt
index 814be4a..49af55c 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,3 +1,5 @@
 jinja2
-ops
-charmhelpers
\ No newline at end of file
+charmhelpers
+
+# Pin ops version to support bionic deployments
+ops < 2.0
\ No newline at end of file
diff --git a/src/charm.py b/src/charm.py
index d921131..21ebdb2 100755
--- a/src/charm.py
+++ b/src/charm.py
@@ -2,15 +2,24 @@
 
 """Iscsi Connector Charm."""
 
+
 import json
 import logging
+import os
 import re
 import socket
 import subprocess
+import time
+from datetime import datetime
+from functools import wraps
 from pathlib import Path
+from typing import Any, Callable
 
 import apt
 
+import charmhelpers.contrib.openstack.deferred_events as deferred_events
+import charmhelpers.contrib.openstack.policy_rcd as policy_rcd
+
 from jinja2 import Environment, FileSystemLoader
 
 from ops.charm import CharmBase
@@ -23,9 +32,42 @@ from charms.prometheus_k8s.v0.prometheus_scrape import MetricsEndpointProvider  
 from storage_connector import metrics_utils, nrpe_utils
 import utils  # noqa
 
+import yaml
+
 logger = logging.getLogger(__name__)
 
 
+def check_deferred_restarts_queue(func: Callable) -> Callable:
+    """Check the queue of deferred restarts and set correct status message.
+
+    This wrapper can be used to decorate a method (primarily an event handler) from
+    `StorageConnectorCharm` class. After decorated function is executed, this
+    wrapper will clear the expired deferred restarts and attach correct status
+    message if the unit is active.
+    """
+
+    @wraps(func)
+    def wrapper(self: "StorageConnectorCharm", *args: Any, **kwargs: Any) -> Any:
+        """Execute wrapped method and perform status assessment."""
+        result = func(self, *args, **kwargs)
+
+        # Handle manual service restarts outside the charm (e.g. run
+        # `systemctl restart {service}` in the unit). It compares
+        # each service's start time with its deferred events timestamp
+        # and clears any deferred events which happened prior to this time
+        try:
+            deferred_events.check_restart_timestamps()
+        except ValueError as err:
+            logging.error("Cannot retrieve services' start time: %s", err)
+
+        if isinstance(self.unit.status, ActiveStatus):
+            self.unit.status = ActiveStatus(self._get_status_message())
+
+        return result
+
+    return wrapper
+
+
 class StorageConnectorCharm(CharmBase):
     """Class representing this Operator charm."""
 
@@ -41,6 +83,7 @@ class StorageConnectorCharm(CharmBase):
 
     ISCSI_SERVICES = ['iscsid', 'open-iscsi']
     MULTIPATHD_SERVICE = 'multipathd'
+    DEFERRED_SERVICES = ISCSI_SERVICES + [MULTIPATHD_SERVICE]
 
     VALID_STORAGE_TYPES = ["fc", "iscsi"]
     MANDATORY_CONFIG = {
@@ -62,15 +105,19 @@ class StorageConnectorCharm(CharmBase):
         super().__init__(*args)
 
         # -- standard hook observation
-        self.framework.observe(self.on.install, self.on_install)
-        self.framework.observe(self.on.install, self.render_config)
-        self.framework.observe(self.on.start, self.on_start)
-        self.framework.observe(self.on.config_changed, self.render_config)
-        self.framework.observe(self.on.config_changed, self.on_config_changed)
-        self.framework.observe(self.on.restart_iscsi_services_action,
-                               self.on_restart_iscsi_services_action)
+        self.framework.observe(self.on.install, self._on_install)
+        self.framework.observe(self.on.start, self._on_start)
+        self.framework.observe(self.on.config_changed, self._render_config)
+        self.framework.observe(self.on.config_changed, self._on_config_changed)
+        self.framework.observe(self.on.update_status, self._on_update_status)
+        self.framework.observe(self.on.restart_services_action,
+                               self._on_restart_services_action)
+        self.framework.observe(self.on.show_deferred_restarts_action,
+                               self._on_show_deferred_restarts_action)
         self.framework.observe(self.on.reload_multipathd_service_action,
-                               self.on_reload_multipathd_service_action)
+                               self._on_reload_multipathd_service_action)
+        self.framework.observe(self.on.iscsi_discovery_and_login_action,
+                               self._on_iscsi_discovery_and_login_action)
         self.framework.observe(self.on.metrics_endpoint_relation_created,
                                self._on_metrics_endpoint_relation_created)
         self.framework.observe(self.on.metrics_endpoint_relation_broken,
@@ -105,13 +152,14 @@ class StorageConnectorCharm(CharmBase):
         )
         self.mp_path = self.MULTIPATH_CONF_PATH / self._stored.mp_conf_name
 
-    def on_install(self, event):
+    def _on_install(self, _):
         """Handle install state."""
         self.unit.status = MaintenanceStatus("Installing charm software")
         if self._check_if_container():
             return
 
-        if not self._check_mandatory_config():
+        self._check_mandatory_config()
+        if isinstance(self.unit.status, BlockedStatus):
             return
 
         # install packages
@@ -137,21 +185,24 @@ class StorageConnectorCharm(CharmBase):
         logging.info("Install of software complete")
         self._stored.installed = True
 
-    def on_config_changed(self, event):
+    def _on_config_changed(self, _):
         """Config-changed event handler."""
         if self._stored.nrpe_related:
             nrpe_utils.update_nrpe_config(self.model.config)
 
-    def render_config(self, event):
+    def _render_config(self, _):
         """Render configuration templates upon config change."""
         if self._check_if_container():
             return
 
-        if not self._check_mandatory_config():
+        self.unit.status = MaintenanceStatus("Validating charm configuration")
+        self._check_mandatory_config()
+        if isinstance(self.unit.status, BlockedStatus):
             return
 
         if self._stored.storage_type == 'fc' and not self._stored.fc_scan_ran_once:
-            if not self._fc_scan_host_successfully():
+            self._fc_scan_host()
+            if isinstance(self.unit.status, BlockedStatus):
                 return
 
         self.unit.status = MaintenanceStatus("Rendering charm configuration")
@@ -159,19 +210,19 @@ class StorageConnectorCharm(CharmBase):
 
         tenv = Environment(loader=FileSystemLoader('templates'))
 
-        if self._stored.storage_type == 'iscsi':
-            self._iscsi_initiator(tenv)
-            self._iscsid_configuration(tenv)
-            self._restart_iscsi_services()
+        self._configure_deferred_restarts()
 
-            if self.model.config.get('iscsi-discovery-and-login'):
-                logging.info('Launching iscsiadm discovery and login')
-                self._iscsiadm_discovery()
-                self._iscsiadm_login()
+        if self._stored.storage_type == 'iscsi':
+            self._configure_iscsi(tenv=tenv, event_name="config changed")
+            if isinstance(self.unit.status, BlockedStatus):
+                return
 
-        if not self._multipath_configuration(tenv):
+        self._multipath_configuration(tenv)
+        if isinstance(self.unit.status, BlockedStatus):
             return
-        if not self._validate_multipath_config():
+
+        self._validate_multipath_config()
+        if isinstance(self.unit.status, BlockedStatus):
             return
 
         self._reload_multipathd_service()
@@ -179,9 +230,9 @@ class StorageConnectorCharm(CharmBase):
         logging.info("Setting started state")
         self._stored.started = True
         self._stored.configured = True
-        self.unit.status = ActiveStatus("Unit is ready")
+        self.unit.status = ActiveStatus(self._get_status_message())
 
-    def on_start(self, event):
+    def _on_start(self, event):
         """Handle start state."""
         if not self._stored.configured:
             logging.warning("Start called before configuration complete, " +
@@ -190,32 +241,124 @@ class StorageConnectorCharm(CharmBase):
             return
         self.unit.status = MaintenanceStatus("Starting charm software")
         # Start software
-        self.unit.status = ActiveStatus("Unit is ready")
+        self.unit.status = ActiveStatus(self._get_status_message())
         self._stored.started = True
         logging.info("Started")
 
+    @check_deferred_restarts_queue
+    def _on_update_status(self, _):
+        """Assess unit's status."""
+
     # Actions
-    def on_restart_iscsi_services_action(self, event):
-        """Restart iscsid and open-iscsi services."""
-        event.log('Restarting iscsi services')
-        self._restart_iscsi_services()
+    @check_deferred_restarts_queue
+    def _on_restart_services_action(self, event):
+        """Restart charm managed services."""
+        deferred_only = event.params["deferred-only"]
+        services = event.params["services"].split()
+        if deferred_only and services:
+            event.set_results({
+                "failed": "deferred-only and services are mutually exclusive"
+            })
+            return
+        if not (deferred_only or services):
+            event.set_results({
+                "failed": "Please specify either deferred-only or services"
+            })
+            return
+        if deferred_only:
+            deferred_services = list(
+                {deferred_event.service for deferred_event
+                    in deferred_events.get_deferred_restarts()})
+            if not deferred_services:
+                event.set_results({"failed": "No deferred services to restart"})
+                return
+            event.log('Restarting the following deferred services: {}'.format(
+                ', '.join(deferred_services)))
+            self._restart_services(services=deferred_services)
+        else:
+            specified_services = list(
+                {service for service in services if service in
+                    self.DEFERRED_SERVICES})
+            if not specified_services:
+                event.set_results({"failed": "No valid services are specified."})
+                return
+            event.log('Restarting the following services: {}'.format(
+                ', '.join(specified_services)))
+            self._restart_services(services=specified_services)
+
         event.set_results({"success": "True"})
 
-    def on_reload_multipathd_service_action(self, event):
+    def _on_show_deferred_restarts_action(self, event):
+        """Get and display the list of service multipathd service."""
+        output = []
+        for deferred_event in deferred_events.get_deferred_restarts():
+            output.append('{} {} {} {}'.format(
+                str(datetime.utcfromtimestamp(deferred_event.timestamp)),
+                "+0000 UTC",
+                deferred_event.service.ljust(40),
+                deferred_event.reason))
+        output.sort()
+        event.set_results({"deferred-restarts": "{}".format(
+            yaml.dump(output, default_flow_style=False)
+        )})
+
+    def _on_reload_multipathd_service_action(self, event):
         """Reload multipathd service."""
         event.log('Reloading multipathd service')
         self._reload_multipathd_service()
         event.set_results({"success": "True"})
 
+    def _on_iscsi_discovery_and_login_action(self, event):
+        """Run discovery and login against iscsi target(s)."""
+        self._iscsi_discovery_and_login()
+        event.set_results({"success": "True"})
+
     # Additional functions
-    def _restart_iscsi_services(self):
+    def _get_status_message(self):
+        """Set unit status to active with correct status message.
+
+        Check if any services need restarts. If yes, append the queue of
+        such services to the status message.
+        """
+        status_message = "Unit is ready"
+        if not self.model.config.get('enable-auto-restarts'):
+            deferred_restarts = list(
+                {event.service for event in deferred_events.get_deferred_restarts()
+                    if event.policy_requestor_name == self.app.name})
+            if deferred_restarts:
+                svc_msg = "Services queued for restart: {}".format(
+                    ', '.join(sorted(deferred_restarts)))
+                status_message = "{}. {}".format(status_message, svc_msg)
+
+        return status_message
+
+    def _defer_service_restart(self, services=None, reason=None):
+        """Defer service restarts and record this event."""
+        for service in services:
+            deferred_events.save_event(deferred_events.ServiceEvent(
+                timestamp=round(time.time()),
+                service=service,
+                reason='Charm event: {}'.format(reason),
+                action='restart'))
+
+    def _restart_services(self, services=None):
         """Restart iscsid and open-iscsi services."""
-        for service in self.ISCSI_SERVICES:
-            logging.info('Restarting %s service', service)
-            try:
-                subprocess.check_call(['systemctl', 'restart', service])
-            except subprocess.CalledProcessError:
-                logging.exception('An error occured while restarting %s.', service)
+        if services:
+            for service in services:
+                logging.info('Restarting %s service', service)
+                try:
+                    subprocess.check_call(['systemctl', 'restart', service])
+                except subprocess.CalledProcessError:
+                    logging.exception('An error occured while restarting %s.', service)
+
+            # Clear deferred restart events
+            deferred_events.clear_deferred_restarts(services)
+
+            # If any iscsi services restarted and iscsi-discovery-and-login config is
+            # set to true, run iscsiadm discovery and login.
+            if (any(svc in self.ISCSI_SERVICES for svc in services) and
+                    self.model.config.get('iscsi-discovery-and-login')):
+                self._iscsi_discovery_and_login()
 
     def _reload_multipathd_service(self):
         """Reload multipathd service."""
@@ -223,15 +366,25 @@ class StorageConnectorCharm(CharmBase):
         try:
             subprocess.check_call(['systemctl', 'reload', self.MULTIPATHD_SERVICE])
         except subprocess.CalledProcessError:
-            self.unit.status = BlockedStatus(
-                "An error occured while reloading the multipathd service."
-            )
             logging.exception(
                 '%s',
                 "An error occured while reloading the multipathd service."
             )
 
+    def _configure_iscsi(self, tenv, event_name):
+        self._iscsi_initiator(tenv)
+        self._iscsid_configuration(tenv)
+
+        charm_config = self.model.config
+        if charm_config.get('enable-auto-restarts') or not self._stored.started:
+            self._restart_services(services=self.ISCSI_SERVICES)
+        else:
+            self._defer_service_restart(
+                services=self.ISCSI_SERVICES,
+                reason=event_name)
+
     def _check_mandatory_config(self):
+        """Check whether mandatory configs are provided."""
         charm_config = self.model.config
 
         if charm_config['storage-type'] in self.VALID_STORAGE_TYPES:
@@ -246,12 +399,12 @@ class StorageConnectorCharm(CharmBase):
                 self.unit.status = BlockedStatus(
                     "Storage type cannot be changed after deployment."
                 )
-                return False
+                return
         else:
             self.unit.status = BlockedStatus(
                 "Missing/Invalid storage type. Valid options are 'iscsi' or 'fc'."
             )
-            return False
+            return
 
         mandatory_config = self.MANDATORY_CONFIG[self._stored.storage_type]
         missing_config = []
@@ -261,8 +414,6 @@ class StorageConnectorCharm(CharmBase):
         if missing_config:
             self.unit.status = BlockedStatus("Missing mandatory configuration " +
                                              "option(s) {}".format(missing_config))
-            return False
-        return True
 
     def _defer_once(self, event):
         """Defer the given event, but only once."""
@@ -352,7 +503,7 @@ class StorageConnectorCharm(CharmBase):
                         "Exception occured during the multipath \
                         configuration. Please check logs."
                     )
-                    return False
+                    return
             else:
                 logging.debug("multipath-" + section + " is empty.")
 
@@ -362,7 +513,7 @@ class StorageConnectorCharm(CharmBase):
                 self.unit.status = BlockedStatus(
                     "No WWID was found. Please check multipath status and logs."
                 )
-                return False
+                return
             alias = charm_config.get('fc-lun-alias')
             ctxt['multipaths'] = {'wwid': wwid, 'alias': alias}
 
@@ -371,33 +522,29 @@ class StorageConnectorCharm(CharmBase):
         rendered_content = template.render(ctxt)
         self.mp_path.write_text(rendered_content)
         self.mp_path.chmod(0o600)
-        return True
 
-    def _iscsiadm_discovery(self):
+    def _iscsi_discovery_and_login(self):
+        """Run iscsiadm discovery and login against targets."""
         charm_config = self.model.config
         target = charm_config.get('iscsi-target')
         port = charm_config.get('iscsi-port')
-        logging.info('Launching iscsiadm discovery against target')
+        logging.info('Launching iscsiadm discovery and login against targets')
+
         try:
             subprocess.check_call(['iscsiadm', '-m', 'discovery', '-t', 'sendtargets',
                                   '-p', target + ':' + port])
         except subprocess.CalledProcessError:
             logging.exception('Iscsi discovery failed.')
-            self.unit.status = BlockedStatus(
-                'Iscsi discovery failed against target'
-            )
+            return
 
-    def _iscsiadm_login(self):
-        # add check if already logged in, no error if it is.
         try:
-            subprocess.check_call(['iscsiadm', '-m', 'node', '--login'])
-        except subprocess.CalledProcessError:
-            logging.exception('Iscsi login failed.')
-            self.unit.status = BlockedStatus(
-                'Iscsi login failed against target'
-            )
+            subprocess.check_output(
+                ['iscsiadm', '-m', 'node', '--login'],
+                stderr=subprocess.STDOUT)
+        except subprocess.CalledProcessError as err:
+            logging.exception('Iscsi login failed. \n%s', err.output.decode("utf-8"))
 
-    def _fc_scan_host_successfully(self):
+    def _fc_scan_host(self):
         hba_adapters = subprocess.getoutput('ls /sys/class/scsi_host')
         logging.debug('hba_adapters: {}'.format(hba_adapters))
         if not hba_adapters:
@@ -405,7 +552,7 @@ class StorageConnectorCharm(CharmBase):
             self.unit.status = BlockedStatus(
                 "No scsi devices were found. Scan aborted"
             )
-            return False
+            return
 
         for adapter in hba_adapters.split('\n'):
             try:
@@ -419,9 +566,8 @@ class StorageConnectorCharm(CharmBase):
                 self.unit.status = BlockedStatus(
                     'Scan of the HBA adapters failed on the host.'
                 )
-                return False
+                return
         self._stored.fc_scan_ran_once = True
-        return True
 
     def _retrieve_multipath_wwid(self):
         logging.info('Retrive device WWID via multipath -ll')
@@ -440,8 +586,6 @@ class StorageConnectorCharm(CharmBase):
             self.unit.status = BlockedStatus(
                 'Multipath conf error: {}', error
             )
-            return False
-        return True
 
     def _check_if_container(self):
         """Check if the charm is being deployed on a container host."""
@@ -455,13 +599,29 @@ class StorageConnectorCharm(CharmBase):
             return True
         return False
 
+    def _configure_deferred_restarts(self):
+        """Set up deferred restarts in policy-rc.d."""
+        policy_rcd.install_policy_rcd()
+        os.chmod(
+            '/var/lib/charm/{}/policy-rc.d'.format(
+                self.app.name),
+            0o755)
+
+        charm_config = self.model.config
+        if charm_config.get('enable-auto-restarts'):
+            policy_rcd.remove_policy_file()
+        else:
+            blocked_actions = ['stop', 'restart', 'try-restart']
+            for svc in self.DEFERRED_SERVICES:
+                policy_rcd.add_policy_block(svc, blocked_actions)
+
     def _on_metrics_endpoint_relation_created(self, event):
         """Relation-created event handler for metrics-endpoint."""
         self.unit.status = MaintenanceStatus("Installing exporter")
         metrics_utils.install_exporter(self.model.resources)
 
         self._stored.prometheus_related = True
-        self.unit.status = ActiveStatus("Unit is ready")
+        self.unit.status = ActiveStatus(self._get_status_message())
 
     def _on_metrics_endpoint_relation_broken(self, event):
         """Relation-broken event handler for metrics-endpoint."""
@@ -470,7 +630,7 @@ class StorageConnectorCharm(CharmBase):
             metrics_utils.uninstall_exporter()
 
         self._stored.prometheus_related = False
-        self.unit.status = ActiveStatus("Unit is ready")
+        self.unit.status = ActiveStatus(self._get_status_message())
 
     def _on_nrpe_external_master_relation_created(self, event):
         """Relation-created event handler for nrpe-external-master."""
@@ -478,7 +638,7 @@ class StorageConnectorCharm(CharmBase):
         metrics_utils.install_exporter(self.model.resources)
 
         self._stored.nrpe_related = True
-        self.unit.status = ActiveStatus("Unit is ready")
+        self.unit.status = ActiveStatus(self._get_status_message())
 
     def _on_nrpe_external_master_relation_changed(self, event):
         """Relation-changed event handler for nrpe-external-master."""
@@ -494,7 +654,7 @@ class StorageConnectorCharm(CharmBase):
         nrpe_utils.unsync_nrpe_files()
 
         self._stored.nrpe_related = False
-        self.unit.status = ActiveStatus("Unit is ready")
+        self.unit.status = ActiveStatus(self._get_status_message())
 
 
 if __name__ == "__main__":
diff --git a/tests/functional/storage-connector-func-tests/setup.py b/tests/functional/storage-connector-func-tests/setup.py
index 3f8d801..01e3eb8 100644
--- a/tests/functional/storage-connector-func-tests/setup.py
+++ b/tests/functional/storage-connector-func-tests/setup.py
@@ -14,6 +14,7 @@
 
 """Code for setting up storage-connector tests."""
 
+import json
 import logging
 
 import zaza.model
@@ -58,3 +59,50 @@ def configure_iscsi_target():
     # Restart tgt to load new config
     restart_tgt = "systemctl restart tgt"
     zaza.model.run_on_unit('ubuntu-target/0', restart_tgt)
+
+
+def get_unit_full_hostname(unit_name):
+    """Retrieve the full hostname of a unit."""
+    for unit in zaza.model.get_units(unit_name):
+        result = zaza.model.run_on_unit(unit.entity_id, 'hostname -f')
+        hostname = result['Stdout'].rstrip()
+    return hostname
+
+
+def configure_iscsi_connector():
+    """Configure iscsi connector."""
+    iqn = 'iqn.2020-07.canonical.com:lun1'
+    unit_fqdn = get_unit_full_hostname('ubuntu')
+    iscsi_target = zaza.model.get_app_ips('ubuntu-target')[0]
+    initiator_dictionary = json.dumps({unit_fqdn: iqn})
+    conf = {
+        'storage-type': 'iscsi',
+        'initiator-dictionary': initiator_dictionary,
+        'iscsi-target': iscsi_target,
+        'iscsi-port': '3260',
+        'iscsi-node-session-auth-authmethod': 'CHAP',
+        'iscsi-node-session-auth-username': 'iscsi-user',
+        'iscsi-node-session-auth-password': 'password123',
+        'iscsi-node-session-auth-username-in': 'iscsi-target',
+        'iscsi-node-session-auth-password-in': 'secretpass',
+        'multipath-devices': '{}',
+        'enable-auto-restarts': "false"
+    }
+
+    zaza.model.set_application_config('storage-connector', conf)
+    zaza.model.wait_for_application_states(
+        states={
+            "storage-connector": {
+                "workload-status": "active",
+                "workload-status-message-prefix": "Unit is ready"
+            },
+            "ubuntu": {
+                "workload-status": "active",
+                "workload-status-message-regex": "^$"
+            },
+            "ubuntu-target": {
+                "workload-status": "active",
+                "workload-status-message-regex": "^$"
+            }
+        }
+    )
diff --git a/tests/functional/storage-connector-func-tests/tests.py b/tests/functional/storage-connector-func-tests/tests.py
index bc6c7e9..21a7d25 100644
--- a/tests/functional/storage-connector-func-tests/tests.py
+++ b/tests/functional/storage-connector-func-tests/tests.py
@@ -14,7 +14,6 @@
 
 """Encapsulate storage-connector testing."""
 
-import json
 import logging
 
 import zaza.model
@@ -29,42 +28,37 @@ class StorageConnectorTest(test_utils.BaseCharmTest):
         """Run class setup for running glance tests."""
         super(StorageConnectorTest, cls).setUpClass()
 
-    def configure_iscsi_connector(self):
-        """Configure iscsi connector."""
-        iqn = 'iqn.2020-07.canonical.com:lun1'
-        unit_fqdn = self.get_unit_full_hostname('ubuntu')
-        target_ip = zaza.model.get_app_ips('ubuntu-target')[0]
-        initiator_dictionary = json.dumps({unit_fqdn: iqn})
-        conf = {
-            'storage-type': 'iscsi',
-            'initiator-dictionary': initiator_dictionary,
-            'iscsi-target': target_ip,
-            'iscsi-port': '3260',
-            'iscsi-node-session-auth-authmethod': 'CHAP',
-            'iscsi-node-session-auth-username': 'iscsi-user',
-            'iscsi-node-session-auth-password': 'password123',
-            'iscsi-node-session-auth-username-in': 'iscsi-target',
-            'iscsi-node-session-auth-password-in': 'secretpass',
-            'multipath-devices': '{}'
-        }
+    def test_iscsi_connector_config_changed(self):
+        """Test iscsi configuration changes and wait for idle status."""
+        conf = zaza.model.get_application_config('storage-connector')
+        conf["storage-type"] = "null"
         zaza.model.set_application_config('storage-connector', conf)
+        logging.info('Wait for block status...')
+        zaza.model.wait_for_application_states(
+            states={
+                "storage-connector": {
+                    "workload-status": "blocked",
+                    "workload-status-message-prefix": "Missing"
+                },
+                "ubuntu": {
+                    "workload-status": "active",
+                    "workload-status-message-regex": "^$"
+                },
+                "ubuntu-target": {
+                    "workload-status": "active",
+                    "workload-status-message-regex": "^$"
+                }
+            }
+        )
 
-    def get_unit_full_hostname(self, unit_name):
-        """Retrieve the full hostname of a unit."""
-        for unit in zaza.model.get_units(unit_name):
-            result = zaza.model.run_on_unit(unit.entity_id, 'hostname -f')
-            hostname = result['Stdout'].rstrip()
-        return hostname
-
-    def test_iscsi_connector(self):
-        """Test iscsi configuration and wait for idle status."""
-        self.configure_iscsi_connector()
+        conf["storage-type"] = "iscsi"
+        zaza.model.set_application_config('storage-connector', conf)
         logging.info('Wait for idle/ready status...')
         zaza.model.wait_for_application_states(
             states={
                 "storage-connector": {
                     "workload-status": "active",
-                    "workload-status-message-prefix": "Unit is ready"
+                    "workload-status-message-prefix": ""
                 },
                 "ubuntu": {
                     "workload-status": "active",
@@ -79,7 +73,7 @@ class StorageConnectorTest(test_utils.BaseCharmTest):
 
     def test_validate_iscsi_session(self):
         """Validate that the iscsi session is active."""
-        unit = zaza.model.get_units('ubuntu')[0]
+        unit = zaza.model.get_units('storage-connector')[0]
         logging.info('Checking if iscsi session is active.')
         run = zaza.model.run_on_unit(unit.entity_id, 'iscsiadm -m session')
         logging.info("""iscsiadm -m session: Stdout: {}, Stderr: {}, """
@@ -87,3 +81,76 @@ class StorageConnectorTest(test_utils.BaseCharmTest):
                                            run['Stderr'],
                                            run['Code']))
         assert run['Code'] == '0'
+
+    def test_defer_service_restarts(self):
+        """Validate that the iscsi services are deferred on config-changed."""
+        unit = zaza.model.get_units('storage-connector')[0]
+
+        active_time_pre_check = zaza.model.get_systemd_service_active_time(
+            unit.entity_id, 'iscsid.service'
+        )
+        logging.info("""Service start time before config change: {}""".format(
+            active_time_pre_check))
+
+        # Modify the value of a random config option to trigger config-changed event
+        conf = zaza.model.get_application_config('storage-connector')
+        conf["iscsi-node-session-scan"] = "manual"
+        zaza.model.set_application_config('storage-connector', conf)
+
+        zaza.model.wait_for_application_states(
+            states={
+                "storage-connector": {
+                    "workload-status": "active",
+                    "workload-status-message-regex": ".*Services queued for restart.*"
+                },
+                "ubuntu": {
+                    "workload-status": "active",
+                    "workload-status-message-regex": "^$"
+                },
+                "ubuntu-target": {
+                    "workload-status": "active",
+                    "workload-status-message-regex": "^$"
+                }
+            }
+        )
+
+        active_time_post_check = zaza.model.get_systemd_service_active_time(
+            unit.entity_id, 'iscsid.service'
+        )
+        logging.info("""Service start time after config change: {}""".format(
+            active_time_post_check))
+
+        # Check if service restart is blocked
+        assert active_time_pre_check == active_time_post_check
+
+        # Run restart-iscsi-services to restart the deferred services
+        zaza.model.run_action(
+            unit.entity_id,
+            "restart-services",
+            action_params={
+                'deferred-only': True})
+        zaza.model.wait_for_application_states(
+            states={
+                "storage-connector": {
+                    "workload-status": "active",
+                    "workload-status-message-prefix": "Unit is ready"
+                },
+                "ubuntu": {
+                    "workload-status": "active",
+                    "workload-status-message-regex": "^$"
+                },
+                "ubuntu-target": {
+                    "workload-status": "active",
+                    "workload-status-message-regex": "^$"
+                }
+            }
+        )
+
+        active_time_after_restart = zaza.model.get_systemd_service_active_time(
+            unit.entity_id, 'iscsid.service'
+        )
+        logging.info("""Service start time after service restart action: {}""".format(
+            active_time_after_restart))
+
+        # Check if service restart was successful
+        assert active_time_post_check != active_time_after_restart
diff --git a/tests/functional/tests/tests.yaml b/tests/functional/tests/tests.yaml
index 6a43b9b..57551f4 100644
--- a/tests/functional/tests/tests.yaml
+++ b/tests/functional/tests/tests.yaml
@@ -11,6 +11,7 @@ gate_bundles:
 configure:
  - storage-connector-func-tests.setup.basic_target_setup
  - storage-connector-func-tests.setup.configure_iscsi_target
+ - storage-connector-func-tests.setup.configure_iscsi_connector
 
 tests:
  - storage-connector-func-tests.tests.StorageConnectorTest
diff --git a/tests/unit/test_charm.py b/tests/unit/test_charm.py
index 499e35d..18daa18 100644
--- a/tests/unit/test_charm.py
+++ b/tests/unit/test_charm.py
@@ -10,7 +10,10 @@ from unittest.mock import PropertyMock, call, mock_open, patch
 
 import charm
 
+import charmhelpers.contrib.openstack.deferred_events as deferred_events
+
 from ops.framework import EventBase
+from ops.model import ActiveStatus
 from ops.testing import Harness
 
 
@@ -40,6 +43,7 @@ class TestCharm(unittest.TestCase):
 
     @patch("storage_connector.nrpe_utils.update_nrpe_config")
     @patch("charm.subprocess.check_call")
+    @patch("charm.subprocess.check_output")
     @patch("charm.subprocess.getoutput")
     @patch("charm.utils.is_container")
     @patch("charm.StorageConnectorCharm.MULTIPATH_CONF_PATH", new_callable=PropertyMock)
@@ -49,7 +53,9 @@ class TestCharm(unittest.TestCase):
     )
     @patch("charm.StorageConnectorCharm.ISCSI_CONF", new_callable=PropertyMock)
     @patch("charm.StorageConnectorCharm.ISCSI_CONF_PATH", new_callable=PropertyMock)
+    @patch("charm.StorageConnectorCharm._configure_deferred_restarts")
     def test_on_iscsi_install(self,
+                              m_configure_deferred_restarts,
                               m_iscsi_conf_path,
                               m_iscsi_conf,
                               m_iscsi_initiator_name,
@@ -57,6 +63,7 @@ class TestCharm(unittest.TestCase):
                               m_multipath_conf_path,
                               m_is_container,
                               m_getoutput,
+                              m_check_output,
                               m_check_call,
                               _):
         """Test installation."""
@@ -86,11 +93,17 @@ class TestCharm(unittest.TestCase):
             [
                 call("systemctl restart iscsid".split()),
                 call("systemctl restart open-iscsi".split()),
-                call("iscsiadm -m discovery -t sendtargets -p abc:443".split()),
-                call("iscsiadm -m node --login".split())
+                call("iscsiadm -m discovery -t sendtargets -p abc:443".split())
+            ],
+            any_order=False
+        )
+        m_check_output.assert_has_calls(
+            [
+                call("iscsiadm -m node --login".split(), stderr=-2)
             ],
             any_order=False
         )
+        m_configure_deferred_restarts.assert_called_once()
         self.assertTrue(self.harness.charm._stored.installed)
 
     @patch("storage_connector.nrpe_utils.update_nrpe_config")
@@ -101,7 +114,9 @@ class TestCharm(unittest.TestCase):
     @patch("charm.StorageConnectorCharm.MULTIPATH_CONF_DIR", new_callable=PropertyMock)
     @patch("charm.StorageConnectorCharm.ISCSI_CONF", new_callable=PropertyMock)
     @patch("charm.StorageConnectorCharm.ISCSI_CONF_PATH", new_callable=PropertyMock)
+    @patch("charm.StorageConnectorCharm._configure_deferred_restarts")
     def test_on_fiberchannel_install(self,
+                                     m_configure_deferred_restarts,
                                      m_iscsi_conf_path,
                                      m_iscsi_conf,
                                      m_multipath_conf_dir,
@@ -129,6 +144,7 @@ class TestCharm(unittest.TestCase):
         self.assertTrue(os.path.exists(self.harness.charm.MULTIPATH_CONF_PATH))
         self.assertTrue(self.harness.charm._stored.installed)
         m_open.assert_called_once_with("/sys/class/scsi_host/host0/scan", "w")
+        m_configure_deferred_restarts.assert_called_once()
         self.assertTrue(
             call(self.harness.charm.ISCSI_CONF, "w") not in m_open.mock_calls
         )
@@ -144,29 +160,314 @@ class TestCharm(unittest.TestCase):
         self.harness.charm.on.start.emit()
         self.assertTrue(self.harness.charm._stored.started)
 
+    def test_on_restart_services_action_mutually_exclusive_params(self):
+        """Test on restart servcices action with both deferred-only and services."""
+        action_event = FakeActionEvent(params={
+            "deferred-only": True, "services": "test_service"})
+        self.harness.charm._on_restart_services_action(action_event)
+        self.assertEqual(
+            action_event.results['failed'],
+            'deferred-only and services are mutually exclusive')
+
+    def test_on_restart_services_action_no_params(self):
+        """Test on restart servcices action with no param."""
+        action_event = FakeActionEvent(params={
+            "deferred-only": False, "services": ""})
+        self.harness.charm._on_restart_services_action(action_event)
+        self.assertEqual(
+            action_event.results['failed'],
+            'Please specify either deferred-only or services')
+
     @patch("charm.subprocess.check_call")
-    def test_on_restart_iscsi_services_action(self, m_check_call):
-        """Test on restart action."""
-        action_event = FakeActionEvent()
-        m_check_call.return_value = True
-        self.harness.charm.on_restart_iscsi_services_action(action_event)
+    @patch("charm.deferred_events.get_deferred_restarts")
+    @patch("charm.deferred_events.check_restart_timestamps")
+    def test_on_restart_services_action_deferred_only_failed(self,
+                                                             m_check_rtimestamps,
+                                                             m_get_deferred_restarts,
+                                                             m_check_call):
+        """Test on restart servcices action empty list of deferred restarts."""
+        m_get_deferred_restarts.return_value = []
+        action_event = FakeActionEvent(params={
+            "deferred-only": True, "services": ""})
+        self.harness.charm._on_restart_services_action(action_event)
+        self.assertEqual(
+            action_event.results['failed'], 'No deferred services to restart')
+
+    @patch("charm.subprocess.check_call")
+    @patch("charm.deferred_events.get_deferred_restarts")
+    @patch("charm.deferred_events.check_restart_timestamps")
+    def test_on_restart_services_action_deferred_only_success(self,
+                                                              m_check_rtimestamps,
+                                                              m_get_deferred_restarts,
+                                                              m_check_call):
+        """Test on restart servcices action with deferred only param."""
+        m_get_deferred_restarts.return_value = [deferred_events.ServiceEvent(
+            timestamp=123456,
+            service='svc',
+            reason='Reason',
+            action='restart',
+            policy_requestor_name='myapp',
+            policy_requestor_type='charm')]
+        action_event = FakeActionEvent(params={
+            "deferred-only": True, "services": ""})
+        self.harness.charm._on_restart_services_action(action_event)
         m_check_call.assert_has_calls(
             [
-                call(["systemctl", "restart", "iscsid"]),
-                call(["systemctl", "restart", "open-iscsi"])
+                call(["systemctl", "restart", "svc"])
             ],
             any_order=False
         )
         self.assertEqual(action_event.results['success'], 'True')
 
+    def test_on_restart_services_action_services_failed(self):
+        """Test on restart servcices action failed with invalid service input."""
+        action_event = FakeActionEvent(params={
+            "deferred-only": False, "services": "non_valid_service"})
+        self.harness.charm._on_restart_services_action(action_event)
+        self.assertEqual(
+            action_event.results['failed'], 'No valid services are specified.')
+
+    @patch("charm.subprocess.check_call")
+    @patch("charm.StorageConnectorCharm._iscsi_discovery_and_login")
+    def test_on_restart_services_action_services_success(self,
+                                                         m_iscsi_discovery_and_login,
+                                                         m_check_call):
+        """Test on restart servcices action successfully run with services param."""
+        action_event = FakeActionEvent(params={
+            "deferred-only": False, "services": "iscsid open-iscsi multipathd"})
+        self.harness.charm._on_restart_services_action(action_event)
+        m_check_call.assert_has_calls(
+            [
+                call(["systemctl", "restart", "iscsid"]),
+                call(["systemctl", "restart", "open-iscsi"]),
+                call(["systemctl", "restart", "multipathd"])
+            ],
+            any_order=True
+        )
+        self.assertEqual(action_event.results['success'], 'True')
+        m_iscsi_discovery_and_login.assert_called_once()
+
+    @patch("charm.deferred_events.get_deferred_restarts")
+    def test_on_show_deferred_restarts_action(self, m_get_deferred_restarts):
+        """Test on show deferred restarts action."""
+        m_get_deferred_restarts.return_value = [deferred_events.ServiceEvent(
+            timestamp=123456,
+            service='svc',
+            reason='Reason',
+            action='restart',
+            policy_requestor_name='myapp',
+            policy_requestor_type='charm')]
+
+        action_event = FakeActionEvent()
+        self.harness.charm._on_show_deferred_restarts_action(action_event)
+        self.assertEqual(
+            action_event.results['deferred-restarts'],
+            '- 1970-01-02 10:17:36 +0000 UTC svc' +
+            '                                      Reason\n')
+
     @patch("charm.subprocess.check_call")
     def test_on_reload_multipathd_service_action(self, m_check_call):
-        """Test on reload action."""
+        """Test on reload multipathd action."""
         action_event = FakeActionEvent()
-        self.harness.charm.on_reload_multipathd_service_action(action_event)
+        self.harness.charm._on_reload_multipathd_service_action(action_event)
         m_check_call.assert_called_once_with(["systemctl", "reload", "multipathd"])
         self.assertEqual(action_event.results['success'], 'True')
 
+    @patch("charm.subprocess.check_call")
+    @patch("charm.subprocess.check_output")
+    def test_on_iscsi_discovery_and_login_action(self, m_check_output, m_check_call):
+        """Test on iscsi discovery and login action."""
+        self.harness.update_config({
+            "iscsi-target": "abc",
+            "iscsi-port": "443"
+        })
+        action_event = FakeActionEvent()
+        self.harness.charm._on_iscsi_discovery_and_login_action(action_event)
+
+        m_check_call.assert_has_calls(
+            [
+                call(['iscsiadm', '-m', 'discovery', '-t', 'sendtargets',
+                                  '-p', "abc" + ':' + "443"]),
+            ],
+            any_order=False
+        )
+        m_check_output.assert_has_calls(
+            [
+                call(['iscsiadm', '-m', 'node', '--login'], stderr=-2)
+            ],
+            any_order=False
+        )
+        self.assertEqual(action_event.results['success'], 'True')
+
+    @patch("charm.deferred_events.get_deferred_restarts")
+    def test_get_status_message(self, m_get_deferred_restarts):
+        """Test on setting active status with correct status message."""
+        m_get_deferred_restarts.return_value = []
+        self.assertEqual(self.harness.charm._get_status_message(), "Unit is ready")
+
+        m_get_deferred_restarts.return_value = [deferred_events.ServiceEvent(
+            timestamp=123456,
+            service='svc1',
+            reason='Reason1',
+            action='restart',
+            policy_requestor_name='other_app',
+            policy_requestor_type='charm'), deferred_events.ServiceEvent(
+            timestamp=234567,
+            service='svc2',
+            reason='Reason2',
+            action='restart',
+            policy_requestor_name='storage-connector',
+            policy_requestor_type='charm')]
+        self.assertEqual(
+            self.harness.charm._get_status_message(),
+            "Unit is ready. Services queued for restart: svc2"
+        )
+
+    @patch("charm.deferred_events.check_restart_timestamps")
+    @patch("charm.deferred_events.get_deferred_restarts")
+    def test_check_deferred_restarts_queue(self,
+                                           m_get_deferred_restarts,
+                                           m_check_restart_timestamps):
+        """Test check_deferred_restarts_queue decorator function."""
+        self.harness.charm.unit.status = ActiveStatus("Unit is ready")
+        m_get_deferred_restarts.return_value = [deferred_events.ServiceEvent(
+            timestamp=234567,
+            service='svc',
+            reason='Reason',
+            action='restart',
+            policy_requestor_name='storage-connector',
+            policy_requestor_type='charm')]
+
+        # Trigger decorator function by emitting update_status event
+        self.harness.charm.on.update_status.emit()
+
+        m_check_restart_timestamps.assert_called_once()
+        self.assertEqual(
+            self.harness.charm.unit.status,
+            ActiveStatus("Unit is ready. Services queued for restart: svc")
+        )
+
+    @patch("charm.policy_rcd.install_policy_rcd")
+    @patch("charm.policy_rcd.remove_policy_file")
+    @patch("charm.policy_rcd.add_policy_block")
+    @patch("charm.os.chmod")
+    def test_configure_deferred_restarts(self,
+                                         m_chmod,
+                                         m_add_policy_block,
+                                         m_remove_policy_file,
+                                         install_policy_rcd):
+        """Test on setting up deferred restarts in policy-rc.d."""
+        self.harness.update_config({'enable-auto-restarts': True})
+        self.harness.charm._configure_deferred_restarts()
+        install_policy_rcd.assert_called_once()
+        m_remove_policy_file.assert_called_once()
+        m_chmod.assert_called_once()
+
+        self.harness.update_config({'enable-auto-restarts': False})
+        self.harness.charm._configure_deferred_restarts()
+        m_add_policy_block.assert_has_calls(
+            [
+                call('iscsid', ['stop', 'restart', 'try-restart']),
+                call('open-iscsi', ['stop', 'restart', 'try-restart'])
+            ],
+            any_order=False
+        )
+
+    @patch("charm.subprocess.check_call")
+    @patch("charm.StorageConnectorCharm._iscsi_discovery_and_login")
+    def test_on_restart_non_iscsi_services(self,
+                                           m_iscsi_discovery_and_login,
+                                           m_check_call):
+        """Test on restarting non-iscsi services."""
+        self.harness.charm._restart_services(services=["multipathd"])
+        m_check_call.assert_has_calls(
+            [
+                call(["systemctl", "restart", "multipathd"])
+            ],
+            any_order=True
+        )
+        m_iscsi_discovery_and_login.assert_not_called()
+
+    @patch("charm.subprocess.check_call")
+    @patch("charm.StorageConnectorCharm._iscsi_discovery_and_login")
+    def test_on_restart_iscsi_services_with_discovery_login(
+        self,
+        m_iscsi_discovery_and_login,
+        m_check_call
+    ):
+        """Test on restarting a iscsi service with discovery and login."""
+        self.harness.update_config({'iscsi-discovery-and-login': True})
+        self.harness.charm._restart_services(services=["iscsid"])
+        m_check_call.assert_has_calls(
+            [
+                call(["systemctl", "restart", "iscsid"])
+            ],
+            any_order=True
+        )
+        m_iscsi_discovery_and_login.assert_called_once()
+
+    @patch("charm.subprocess.check_call")
+    @patch("charm.StorageConnectorCharm._iscsi_discovery_and_login")
+    def test_on_restart_iscsi_services_without_discovery_login(
+        self,
+        m_iscsi_discovery_and_login,
+        m_check_call
+    ):
+        """Test on restarting a iscsi service without running discovery and login."""
+        self.harness.update_config({'iscsi-discovery-and-login': False})
+        self.harness.charm._restart_services(services=["iscsid"])
+        m_check_call.assert_has_calls(
+            [
+                call(["systemctl", "restart", "iscsid"])
+            ],
+            any_order=True
+        )
+        m_iscsi_discovery_and_login.assert_not_called()
+
+    @patch("charm.subprocess.check_output")
+    @patch("charm.subprocess.check_call")
+    @patch("charm.logging.exception")
+    def test_iscsiadm_discovery_failed(self,
+                                       m_log_exception,
+                                       m_check_call,
+                                       m_check_output):
+        """Test response to iscsiadm discovery failure."""
+        self.harness.update_config({
+            "storage-type": "iscsi",
+            "iscsi-target": "abc",
+            "iscsi-port": "443"
+        })
+        self.harness.charm.unit.status = ActiveStatus("Unit is ready")
+        m_check_call.side_effect = charm.subprocess.CalledProcessError(
+            returncode=15,
+            cmd=['iscsiadm', '-m', 'discovery', '-t', 'sendtargets',
+                 '-p', "abc" + ':' + "443"],
+        )
+        self.harness.charm._iscsi_discovery_and_login()
+        m_log_exception.assert_called_once()
+
+    @patch("charm.subprocess.check_output")
+    @patch("charm.subprocess.check_call")
+    @patch("charm.logging.exception")
+    def test_iscsiadm_login_failed(self,
+                                   m_log_exception,
+                                   m_check_call,
+                                   m_check_output):
+        """Test response to iscsiadm login failure."""
+        self.harness.update_config({
+            "storage-type": "iscsi",
+            "iscsi-target": "abc",
+            "iscsi-port": "443"
+        })
+        self.harness.charm.unit.status = ActiveStatus("Unit is ready")
+        m_check_output.side_effect = charm.subprocess.CalledProcessError(
+            returncode=15,
+            cmd=['iscsiadm', '-m', 'node', '--login'],
+            output=b'iscsiadm: Could not log into all portals'
+        )
+        self.harness.charm._iscsi_discovery_and_login()
+        m_log_exception.assert_called_once()
+
     @patch("storage_connector.metrics_utils.uninstall_exporter")
     @patch("storage_connector.metrics_utils.install_exporter")
     def test_on_metrics_endpoint_handlers(
diff --git a/tox.ini b/tox.ini
index 5c8e284..a401667 100644
--- a/tox.ini
+++ b/tox.ini
@@ -8,14 +8,14 @@ skip_missing_interpreters = False
 basepython = python3
 setenv =
   PYTHONPATH = {toxinidir}/lib/:{toxinidir}
-passenv = 
+passenv =
     HOME
     CHARM_BUILD_DIR
     OS_*
 whitelist_externals = charmcraft
 
 [testenv:unit]
-commands = 
+commands =
     coverage erase
     stestr run --slowest {posargs}
     coverage combine
@@ -24,7 +24,7 @@ commands =
     coverage report -m --omit lib/charms/*,templates/*
 deps = -r{toxinidir}/tests/unit/requirements.txt
        -r{toxinidir}/requirements.txt
-setenv = 
+setenv =
      {[testenv]setenv}
      PYTHON=coverage run
 
@@ -32,9 +32,9 @@ setenv =
 branch = True
 concurrency = multiprocessing
 parallel = True
-source = 
+source =
     .
-omit = 
+omit =
     .tox/*
     tests/*