Merge ~txiao/charm-storage-connector:defer_restart into charm-storage-connector:defer-service-restarts-integration

[Tianqi Xiao]

Add defer methods for better handling service restarts

This MP fixes LP#2003241. The following changes have been made:

- Make use of charmhelpers policy_rcd script to block automatic stop, restarts, try-restarts action for all charm managed services, including iscsid open-iscsi, and multipathd.
- Handles deferred restarts triggered by both charm and non-charm events
- Display the queue of deferred service restarts in charm status message
and periodic updates
- Add `enable-auto-restarts` config option to give user option to allow automatic restarts action if needed (the default value is False)
- Add `restart-services` action to restart charm managed service. The
operator can choose to restart either all deferred services or a specific
set of services.
- Add `show-deferred-restarts` action to show all deferred restart events
with the timestamp and reason
- Add `iscsi-discovery-and-login` action that allow operator to manually
run iscsiadm discovery and login when needed
- Remove `restart-iscsi-services` action in favor of the new
`restart-services` action
- Better handling different types of iscsi discovery and login failures.
If failed because session already present, ignore the error; otherwise
change charm status to blocked.
- Add unittests and functional tests for the new and modified methods
- Rework functional tests logic to make each test case independent of each other.

[Erhan Sunar (esunar)]

How did you test this?

[Tianqi Xiao (txiao)]

@martin-kalcok Thanks for the review. I have applied the suggestions and replied to your inquiry in-line.

[Tianqi Xiao (txiao)]

> How did you test this?

I've deployed the jammy bundle used in functional tests, performed config changes and run restart actions to observe its behavior. I'm also working on writing the unittests and functional tests, which will be pushed upon completion.

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:60bb5f05f95a475a74dec7a53657b3b1222e806c
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/18/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/2183/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/1990/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/18//rebuild

review: Approve (continuous-integration)

[Paul Goins (vultaire)]

I haven't reviewed the tests, but I did review the main implementation. It's quite interesting, and I see some things I like there.

I have a few areas I have some concerns, or where I think we may have some subtle bugs. Please see my comments.

Also (as I think we've also discussed internally), we probably want to block restarts of the multipathd service as well.

I am not sure if a policy-based block on restarting is the best solution here, but I simply haven't seen this done elsewhere. It's interesting. A local state flag may be another idea, just to determine whether we've already initialized the node once and to block subsequent restarts unless explicitly desired... But, that's *only* an idea; the policy-based solution may be better. I just don't know since it's not something I'm familiar with.

Anyway, please take a look at my comments. Thank you.

review: Needs Fixing

[Eric Chen (eric-chen)]

Please inline comment below

review: Needs Information

[Eric Chen (eric-chen)]

Please check inline comment.

review: Needs Information

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:7c3607952e097ce96df475e1d7d3b11cc2f527a3
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/1/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/2/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/2/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/1//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:918505fdff4adfef4c73e0f06e4efcf1963af49b
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/2/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/3/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/3/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/2//rebuild

review: Approve (continuous-integration)

[Tianqi Xiao (txiao)]

Hi reviewers. I have made major changes to the code implementation to facilitate new functionalities as requested. Initially I wanted to open a separate MP for the new changes, but it feels wrong to get this merged and then overwrite the methods created here. I apologize for increasing the workload needed to review this MP.

In hope of better illustrating new changes, I have kept the commits separate. The latest commit (918505f) contains the updates, including addressing the previous comments.

Feedback and suggestions are highly appreciated :)

[Andrea Ieri (aieri)]

Comments in-line. I'm actually not done with the review but I'm a bit out of day and I wanted to start sending these comments.

I have also dropped some pre-existing trailing whitespace in https://code.launchpad.net/~aieri/charm-storage-connector/+git/charm-storage-connector/+merge/437433

The lint target does not include black. Please align the tox and/or pyproject.toml configs to the one from bootstack-charms-spec (in a separate MP).

review: Needs Fixing

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:1540ff0713b56d2625424a71ae38de8219db5d42
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/3/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/55/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/22/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/3//rebuild

review: Approve (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:47e23872d87092f53917acaa54c55ba1c4774aaa
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/4/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/56/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/23/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/4//rebuild

review: Approve (continuous-integration)

[Tianqi Xiao (txiao)]

@aieri Thanks a lot for the review. I have addressed your suggestions in the latest commit and answered your questions in-line.

[Andrea Ieri (aieri)]

Added replies to previous comments for commit 918505f. My review is not complete.

[Andrea Ieri (aieri)]

Comments in-line

review: Needs Fixing

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:f3c025427b69127c8ffe5b3250413e92cebc294c
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/5/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/115/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/44/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/5//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:a026647ae5f39839cfbc1a4140edcd1409e5aae2
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/6/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/116/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/45/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/6//rebuild

review: Approve (continuous-integration)

[Tianqi Xiao (txiao)]

@aieri Thanks for your review. I have made changes addressing your comments.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:ae8936b5932ae690af8f97e5e1b797789f8ab60a
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/7/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/117/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/46/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/7//rebuild

review: Approve (continuous-integration)

[JamesLin (jneo8)]

Only one unit tests request.
Others LGTM.

review: Needs Fixing

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:a5df52ea49e89e98153ea683937b7d50aace6d48
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/8/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/122/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/51/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/8//rebuild

review: Needs Fixing (continuous-integration)

[Tianqi Xiao (txiao)]

@jneo8 Thanks for the review. I've added the requested unit test case.

[Andrea Ieri (aieri)]

saving some in-line comments

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:7d4ddba400fce37aac74b85c2da37deca78f9e89
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/9/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/123/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/52/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/9//rebuild

review: Needs Fixing (continuous-integration)

[Paul Goins (vultaire)]

This took some time to review... but I think I'm *almost* a +1 now.

I did find one bug, verified locally on a VM, which requires attention. Please see my comment; this would have been a hard one to catch.

Once this is resolved, I'm ready to +1 this review.

review: Needs Fixing

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:7d221cbcbdddc3e72b85ed9d2a25f05a2fe8484e
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/10/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/124/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/53/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/10//rebuild

review: Needs Fixing (continuous-integration)

[Paul Goins (vultaire)]

+1 from me. Not sure what is up with CI; the error there doesn't look like anything related to the actual tests.

review: Approve

[JamesLin (jneo8)]

LGTM

review: Approve

[Tianqi Xiao (txiao)]

I re-run the CI pipeline and it has passed. Since it was not automatically attached to this MP, I'm putting the link here: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/125/

[Andrea Ieri (aieri)]

Overall I'm +1 as well, but please don't merge until we have a chance to demo the functionality and discuss whether the iscsi login action is really desirable from a usability perspective.

Another option would be to merge into a separate integration branch and create a new MP if we decide to change how actions behave.

review: Needs Information

[Eric Chen (eric-chen)]

I try to merge it this morning and fortunately I found Andrea's message in time and restore the status.
Therefore, I change the status to WIP until we decide to merge it.

[Paul Goins (vultaire)]

LGTM

review: Approve

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:784d3b30e706e6ae5e65550b745bdf51311217c1
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/11/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/177/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/73/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-storage-connector-ci/11//rebuild

review: Approve (continuous-integration)

[Andrea Ieri (aieri)]

LGTM

review: Approve

diff --git a/README.md b/README.md
index 247b96d..2802aab 100644
--- a/README.md
+++ b/README.md
@@ -2,10 +2,10 @@
 
 ## Overview
 
-This charm configures a unit to connect to a storage endpoint, either iSCSI or Fibre Channel. 
-It acts as a subordinate charm, which can be deployed on any bare metal or virtual machine, 
-alongside a main charm. It is not supported in containers due to lack of ability for lxd 
-containers to access iSCSI and Fibre Channel hardware of the underlying kernel. 
+This charm configures a unit to connect to a storage endpoint, either iSCSI or Fibre Channel.
+It acts as a subordinate charm, which can be deployed on any bare metal or virtual machine,
+alongside a main charm. It is not supported in containers due to lack of ability for lxd
+containers to access iSCSI and Fibre Channel hardware of the underlying kernel.
 
 If you configure this charm for iSCSI, this charm will:
 - Generate an iSCSI initiator name and put it in /etc/iscsi/initiatorname.iscsi
@@ -23,11 +23,11 @@ If you configure it for Fibre Channel, this charm will:
 - Configure multipath under /etc/multipath/conf.d directory
 - Reload and restart multipathd.service
 
-Please note that the Fiber Channel option currently supports only one device. 
+Please note that the Fiber Channel option currently supports only one device.
 
 If iSCSI, the user can input a initiator name dictionary in config.yaml if they wish to use a
 specific IQN for a specific unit. Also, the target IP and port are needed to perform
-the discovery and login with iscsiadm. 
+the discovery and login with iscsiadm.
 
 For Fibre Channel, the user can choose the device alias to be used when mapping the disk.
 See the option "fc-lun-alias" for further details.
@@ -85,9 +85,9 @@ juju relate ubuntu storage-connector
 
 ## Scaling
 
-This charm will scale with the units it is related to. For example, if you scale the 
-ubuntu application, and the storage-connector is related to it, it will be deployed on each 
-newly deployed ubuntu unit. 
+This charm will scale with the units it is related to. For example, if you scale the
+ubuntu application, and the storage-connector is related to it, it will be deployed on each
+newly deployed ubuntu unit.
 ```
 juju add-unit ubuntu
 juju remove-unit ubuntu/1
@@ -104,4 +104,3 @@ cause of the error.
  - Author: Camille Rodriguez <[e-mail]>
  - Maintainers: BootStack Charmers <[e-mail]>
  - Bug Tracker: [here](https://bugs.launchpad.net/charm-storage-connector)
- 
\ No newline at end of file
diff --git a/actions.yaml b/actions.yaml
index 998a670..79eed26 100644
--- a/actions.yaml
+++ b/actions.yaml
@@ -1,4 +1,33 @@
-restart-iscsi-services:
-  description: "Restart iscsid and open-iscsi services"
+restart-services:
+  description: |
+      Restart services managed by the charm.
+  params:
+    deferred-only:
+      type: boolean
+      default: false
+      description: |
+        Restart all deferred services. To check which services has been deferred,
+        run the `show-deferred-events` action.
+        NOTE: deferred-only and services params are mutually exclusive. Providing
+        both options will cause the action to fail.
+    services:
+      type: string
+      default: ""
+      description: |
+        List of services to restart. Supporting services includes iscsi,
+        open-iscsi, and multipathd.
+        This param accepts space-separated services in string format,
+        e.g. "iscsid open-iscsi"
+        NOTE: deferred-only and services params are mutually exclusive. Providing
+        both options will cause the action to fail.
+show-deferred-restarts:
+  descrpition: |
+    Show the outstanding service restarts
 reload-multipathd-service:
-  description: "Reload multipathd service"
+  description: |
+    Reload multipathd service
+iscsi-discovery-and-login:
+  description: |
+    Run discovery and login against iscsi target. This action is needed when
+    changes are made to iscsi configuration and the iscsi services
+    are restarted to apply these changes.
diff --git a/config.yaml b/config.yaml
index 739ef42..c283b6f 100644
--- a/config.yaml
+++ b/config.yaml
@@ -8,7 +8,7 @@ options:
         type: string
         default: data1
         description: |
-            LUN alias to give to the WWID for the mapping of the device. It is used in the path of the new mounted device, i.e /dev/mapper/<FC-LUN-ALIAS>. 
+            LUN alias to give to the WWID for the mapping of the device. It is used in the path of the new mounted device, i.e /dev/mapper/<FC-LUN-ALIAS>.
     iscsi-target:
         type: string
         default:
@@ -21,42 +21,48 @@ options:
         type: boolean
         default: True
         description: |
-            If set to True (default) and storage-type is iscsi, the charm will configure 
-            the iscsid and multipath configuration files, and then run an iscsiadm 
-            discovery and login against the target. If set to False, the charm will simply 
-            configure the unit but not run the discovery and login against the target. 
+            If set to True (default) and storage-type is iscsi, the charm will configure
+            the iscsid and multipath configuration files, and then run an iscsiadm
+            discovery and login against the target. If set to False, the charm will simply
+            configure the unit but not run the discovery and login against the target.
     initiator-dictionary:
         type: string
         default: '{}'
         description: |
-            Dictionary of hostnames (FQDN) and initiator names, surrounded by 
-            single quotes. The charm compares the machine hostname to 
-            the list, and provide the correct initiator associated to 
-            it. If not defined, the initiator name will be provided 
-            randomly. Format : 
+            Dictionary of hostnames (FQDN) and initiator names, surrounded by
+            single quotes. The charm compares the machine hostname to
+            the list, and provide the correct initiator associated to
+            it. If not defined, the initiator name will be provided
+            randomly. Format :
             '{"hostname1": "iqn.yyyy-mm.naming-authority:uniquename1",
               "hostname2": "iqn.yyyy-mm.naming-authority:uniquename2}'
+    enable-auto-restarts:
+        type: boolean
+        default: False
+        description: |
+            Allow the charm and packages to restart services automatically when
+            required.
     multipath-defaults:
         type: string
         default: '{"user_friendly_names": "yes"}'
         description: |
             In multipath config, sets the defaults configuration. String should be of JSON dictionary format.
             Double quotes are essential to the correct format of this JSON string.
-            Example: 
+            Example:
                 value : '{"user_friendly_names":"yes", "find_multipaths":"yes", "polling_interval":"10"}'
                 Which will produce this configuration:
-                defaults { 
-                    user_friendly_names yes 
-                    find_multipaths yes 
+                defaults {
+                    user_friendly_names yes
+                    find_multipaths yes
                     polling_interval 10
-                }                       
+                }
     multipath-devices:
         type: string
         default:
         description: |
             In multipath config, add a device specific configuration. String should be of JSON dictionary format.
             Double quotes are essential to the correct format of this JSON string.
-            Example: 
+            Example:
                 value : '{"vendor":"PURE","product": "FlashArray","fast_io_fail_tmo": "10", "path_grouping_policy":"group_by_prio"}'
                 Which will produce this configuration:
                 device {
@@ -67,9 +73,9 @@ options:
                 }
     multipath-blacklist:
         type: string
-        default: 
+        default:
         description: |
-            In multipath config, add a blacklist device section to exlude local disks from being handled by multipath-tools. It is possible
+            In multipath config, add a blacklist device section to exclude local disks from being handled by multipath-tools. It is possible
             to blacklist by vendor/product, by devnode or WWID. String should be of JSON dictionary format. Double quotes are essential to the correct format of this JSON string.
             Example:
                 value: '{"vendor": "QEMU", "product": "*"}'
diff --git a/metadata.yaml b/metadata.yaml
index 81f963d..1ab3f3d 100644
--- a/metadata.yaml
+++ b/metadata.yaml
@@ -5,8 +5,8 @@ maintainers:
 description: |
     The Storage Connector charm is an open-source charm that can be installed as a
     subordinate to any VM or bare metal machine. It has two modes: iSCSI or Fibre Channel.
-    In the iSCSI mode, it will connect the host to an iSCSI target and configure multipath 
-    accordingly. For Fibre Channel, this charm detects the WWID of the FC LUNs and 
+    In the iSCSI mode, it will connect the host to an iSCSI target and configure multipath
+    accordingly. For Fibre Channel, this charm detects the WWID of the FC LUNs and
     configures multipathd so that the device is ready to be used.
 tags:
     - iscsi
diff --git a/src/charm.py b/src/charm.py
index d921131..8f680a4 100755
--- a/src/charm.py
+++ b/src/charm.py
@@ -2,15 +2,24 @@
 
 """Iscsi Connector Charm."""
 
+
 import json
 import logging
+import os
 import re
 import socket
 import subprocess
+import time
+from datetime import datetime
+from functools import wraps
 from pathlib import Path
+from typing import Any, Callable
 
 import apt
 
+import charmhelpers.contrib.openstack.deferred_events as deferred_events
+import charmhelpers.contrib.openstack.policy_rcd as policy_rcd
+
 from jinja2 import Environment, FileSystemLoader
 
 from ops.charm import CharmBase
@@ -23,9 +32,42 @@ from charms.prometheus_k8s.v0.prometheus_scrape import MetricsEndpointProvider  
 from storage_connector import metrics_utils, nrpe_utils
 import utils  # noqa
 
+import yaml
+
 logger = logging.getLogger(__name__)
 
 
+def check_deferred_restarts_queue(func: Callable) -> Callable:
+    """Check the queue of deferred restarts and set correct status message.
+
+    This wrapper can be used to decorate a method (primarily an event handler) from
+    `StorageConnectorCharm` class. After decorated function is executed, this
+    wrapper will clear the expired deferred restarts and attach correct status
+    message if the unit is active.
+    """
+
+    @wraps(func)
+    def wrapper(self: "StorageConnectorCharm", *args: Any, **kwargs: Any) -> Any:
+        """Execute wrapped method and perform status assessment."""
+        result = func(self, *args, **kwargs)
+
+        # Handle manual service restarts outside the charm (e.g. run
+        # `systemctl restart {service}` in the unit). It compares
+        # each service's start time with its deferred events timestamp
+        # and clears any deferred events which happened prior to this time
+        try:
+            deferred_events.check_restart_timestamps()
+        except ValueError as err:
+            logging.error("Cannot retrieve services' start time: %s", err)
+
+        if isinstance(self.unit.status, ActiveStatus):
+            self.unit.status = ActiveStatus(self._get_status_message())
+
+        return result
+
+    return wrapper
+
+
 class StorageConnectorCharm(CharmBase):
     """Class representing this Operator charm."""
 
@@ -41,6 +83,7 @@ class StorageConnectorCharm(CharmBase):
 
     ISCSI_SERVICES = ['iscsid', 'open-iscsi']
     MULTIPATHD_SERVICE = 'multipathd'
+    DEFERRED_SERVICES = ISCSI_SERVICES + [MULTIPATHD_SERVICE]
 
     VALID_STORAGE_TYPES = ["fc", "iscsi"]
     MANDATORY_CONFIG = {
@@ -62,15 +105,19 @@ class StorageConnectorCharm(CharmBase):
         super().__init__(*args)
 
         # -- standard hook observation
-        self.framework.observe(self.on.install, self.on_install)
-        self.framework.observe(self.on.install, self.render_config)
-        self.framework.observe(self.on.start, self.on_start)
-        self.framework.observe(self.on.config_changed, self.render_config)
-        self.framework.observe(self.on.config_changed, self.on_config_changed)
-        self.framework.observe(self.on.restart_iscsi_services_action,
-                               self.on_restart_iscsi_services_action)
+        self.framework.observe(self.on.install, self._on_install)
+        self.framework.observe(self.on.start, self._on_start)
+        self.framework.observe(self.on.config_changed, self._render_config)
+        self.framework.observe(self.on.config_changed, self._on_config_changed)
+        self.framework.observe(self.on.update_status, self._on_update_status)
+        self.framework.observe(self.on.restart_services_action,
+                               self._on_restart_services_action)
+        self.framework.observe(self.on.show_deferred_restarts_action,
+                               self._on_show_deferred_restarts_action)
         self.framework.observe(self.on.reload_multipathd_service_action,
-                               self.on_reload_multipathd_service_action)
+                               self._on_reload_multipathd_service_action)
+        self.framework.observe(self.on.iscsi_discovery_and_login_action,
+                               self._on_iscsi_discovery_and_login_action)
         self.framework.observe(self.on.metrics_endpoint_relation_created,
                                self._on_metrics_endpoint_relation_created)
         self.framework.observe(self.on.metrics_endpoint_relation_broken,
@@ -105,13 +152,14 @@ class StorageConnectorCharm(CharmBase):
         )
         self.mp_path = self.MULTIPATH_CONF_PATH / self._stored.mp_conf_name
 
-    def on_install(self, event):
+    def _on_install(self, _):
         """Handle install state."""
         self.unit.status = MaintenanceStatus("Installing charm software")
         if self._check_if_container():
             return
 
-        if not self._check_mandatory_config():
+        self._check_mandatory_config()
+        if isinstance(self.unit.status, BlockedStatus):
             return
 
         # install packages
@@ -137,21 +185,24 @@ class StorageConnectorCharm(CharmBase):
         logging.info("Install of software complete")
         self._stored.installed = True
 
-    def on_config_changed(self, event):
+    def _on_config_changed(self, _):
         """Config-changed event handler."""
         if self._stored.nrpe_related:
             nrpe_utils.update_nrpe_config(self.model.config)
 
-    def render_config(self, event):
+    def _render_config(self, _):
         """Render configuration templates upon config change."""
         if self._check_if_container():
             return
 
-        if not self._check_mandatory_config():
+        self.unit.status = MaintenanceStatus("Validating charm configuration")
+        self._check_mandatory_config()
+        if isinstance(self.unit.status, BlockedStatus):
             return
 
         if self._stored.storage_type == 'fc' and not self._stored.fc_scan_ran_once:
-            if not self._fc_scan_host_successfully():
+            self._fc_scan_host()
+            if isinstance(self.unit.status, BlockedStatus):
                 return
 
         self.unit.status = MaintenanceStatus("Rendering charm configuration")
@@ -159,19 +210,19 @@ class StorageConnectorCharm(CharmBase):
 
         tenv = Environment(loader=FileSystemLoader('templates'))
 
-        if self._stored.storage_type == 'iscsi':
-            self._iscsi_initiator(tenv)
-            self._iscsid_configuration(tenv)
-            self._restart_iscsi_services()
+        self._configure_deferred_restarts()
 
-            if self.model.config.get('iscsi-discovery-and-login'):
-                logging.info('Launching iscsiadm discovery and login')
-                self._iscsiadm_discovery()
-                self._iscsiadm_login()
+        if self._stored.storage_type == 'iscsi':
+            self._configure_iscsi(tenv=tenv, event_name="config changed")
+            if isinstance(self.unit.status, BlockedStatus):
+                return
 
-        if not self._multipath_configuration(tenv):
+        self._multipath_configuration(tenv)
+        if isinstance(self.unit.status, BlockedStatus):
             return
-        if not self._validate_multipath_config():
+
+        self._validate_multipath_config()
+        if isinstance(self.unit.status, BlockedStatus):
             return
 
         self._reload_multipathd_service()
@@ -179,9 +230,9 @@ class StorageConnectorCharm(CharmBase):
         logging.info("Setting started state")
         self._stored.started = True
         self._stored.configured = True
-        self.unit.status = ActiveStatus("Unit is ready")
+        self.unit.status = ActiveStatus(self._get_status_message())
 
-    def on_start(self, event):
+    def _on_start(self, event):
         """Handle start state."""
         if not self._stored.configured:
             logging.warning("Start called before configuration complete, " +
@@ -190,32 +241,117 @@ class StorageConnectorCharm(CharmBase):
             return
         self.unit.status = MaintenanceStatus("Starting charm software")
         # Start software
-        self.unit.status = ActiveStatus("Unit is ready")
+        self.unit.status = ActiveStatus(self._get_status_message())
         self._stored.started = True
         logging.info("Started")
 
+    @check_deferred_restarts_queue
+    def _on_update_status(self, _):
+        """Assess unit's status."""
+
     # Actions
-    def on_restart_iscsi_services_action(self, event):
-        """Restart iscsid and open-iscsi services."""
-        event.log('Restarting iscsi services')
-        self._restart_iscsi_services()
+    @check_deferred_restarts_queue
+    def _on_restart_services_action(self, event):
+        """Restart charm managed services."""
+        deferred_only = event.params["deferred-only"]
+        services = event.params["services"].split()
+        if deferred_only and services:
+            event.set_results({
+                "failed": "deferred-only and services are mutually exclusive"
+            })
+            return
+        if not (deferred_only or services):
+            event.set_results({
+                "failed": "Please specify either deferred-only or services"
+            })
+            return
+        if deferred_only:
+            deferred_services = list(
+                {deferred_event.service for deferred_event
+                    in deferred_events.get_deferred_restarts()})
+            if not deferred_services:
+                event.set_results({"failed": "No deferred services to restart"})
+                return
+            event.log('Restarting the following deferred services: {}'.format(
+                ', '.join(deferred_services)))
+            self._restart_services(services=deferred_services)
+        else:
+            specified_services = list(
+                {service for service in services if service in
+                    self.DEFERRED_SERVICES})
+            if not specified_services:
+                event.set_results({"failed": "No valid services are specified."})
+                return
+            event.log('Restarting the following services: {}'.format(
+                ', '.join(specified_services)))
+            self._restart_services(services=specified_services)
+
         event.set_results({"success": "True"})
 
-    def on_reload_multipathd_service_action(self, event):
+    def _on_show_deferred_restarts_action(self, event):
+        """Get and display the list of service multipathd service."""
+        output = []
+        for deferred_event in deferred_events.get_deferred_restarts():
+            output.append('{} {} {} {}'.format(
+                str(datetime.utcfromtimestamp(deferred_event.timestamp)),
+                "+0000 UTC",
+                deferred_event.service.ljust(40),
+                deferred_event.reason))
+        output.sort()
+        event.set_results({"deferred-restarts": "{}".format(
+            yaml.dump(output, default_flow_style=False)
+        )})
+
+    def _on_reload_multipathd_service_action(self, event):
         """Reload multipathd service."""
         event.log('Reloading multipathd service')
         self._reload_multipathd_service()
         event.set_results({"success": "True"})
 
+    def _on_iscsi_discovery_and_login_action(self, event):
+        """Run discovery and login against iscsi target(s)."""
+        self._iscsi_discovery_and_login()
+        event.set_results({"success": "True"})
+
     # Additional functions
-    def _restart_iscsi_services(self):
+    def _get_status_message(self):
+        """Set unit status to active with correct status message.
+
+        Check if any services need restarts. If yes, append the queue of
+        such services to the status message.
+        """
+        status_message = "Unit is ready"
+        if not self.model.config.get('enable-auto-restarts'):
+            deferred_restarts = list(
+                {event.service for event in deferred_events.get_deferred_restarts()
+                    if event.policy_requestor_name == self.app.name})
+            if deferred_restarts:
+                svc_msg = "Services queued for restart: {}".format(
+                    ', '.join(sorted(deferred_restarts)))
+                status_message = "{}. {}".format(status_message, svc_msg)
+
+        return status_message
+
+    def _defer_service_restart(self, services=None, reason=None):
+        """Defer service restarts and record this event."""
+        for service in services:
+            deferred_events.save_event(deferred_events.ServiceEvent(
+                timestamp=round(time.time()),
+                service=service,
+                reason='Charm event: {}'.format(reason),
+                action='restart'))
+
+    def _restart_services(self, services=None):
         """Restart iscsid and open-iscsi services."""
-        for service in self.ISCSI_SERVICES:
-            logging.info('Restarting %s service', service)
-            try:
-                subprocess.check_call(['systemctl', 'restart', service])
-            except subprocess.CalledProcessError:
-                logging.exception('An error occured while restarting %s.', service)
+        if services:
+            for service in services:
+                logging.info('Restarting %s service', service)
+                try:
+                    subprocess.check_call(['systemctl', 'restart', service])
+                except subprocess.CalledProcessError:
+                    logging.exception('An error occured while restarting %s.', service)
+
+            deferred_events.clear_deferred_restarts(services)
 
     def _reload_multipathd_service(self):
         """Reload multipathd service."""
@@ -231,7 +367,22 @@ class StorageConnectorCharm(CharmBase):
                 "An error occured while reloading the multipathd service."
             )
 
+    def _configure_iscsi(self, tenv, event_name):
+        self._iscsi_initiator(tenv)
+        self._iscsid_configuration(tenv)
+
+        charm_config = self.model.config
+        if charm_config.get('enable-auto-restarts') or not self._stored.started:
+            self._restart_services(services=self.ISCSI_SERVICES)
+            if charm_config.get('iscsi-discovery-and-login'):
+                self._iscsi_discovery_and_login()
+        else:
+            self._defer_service_restart(
+                services=self.ISCSI_SERVICES,
+                reason=event_name)
+
     def _check_mandatory_config(self):
+        """Check whether mandatory configs are provided."""
         charm_config = self.model.config
 
         if charm_config['storage-type'] in self.VALID_STORAGE_TYPES:
@@ -246,12 +397,12 @@ class StorageConnectorCharm(CharmBase):
                 self.unit.status = BlockedStatus(
                     "Storage type cannot be changed after deployment."
                 )
-                return False
+                return
         else:
             self.unit.status = BlockedStatus(
                 "Missing/Invalid storage type. Valid options are 'iscsi' or 'fc'."
             )
-            return False
+            return
 
         mandatory_config = self.MANDATORY_CONFIG[self._stored.storage_type]
         missing_config = []
@@ -261,8 +412,6 @@ class StorageConnectorCharm(CharmBase):
         if missing_config:
             self.unit.status = BlockedStatus("Missing mandatory configuration " +
                                              "option(s) {}".format(missing_config))
-            return False
-        return True
 
     def _defer_once(self, event):
         """Defer the given event, but only once."""
@@ -352,7 +501,7 @@ class StorageConnectorCharm(CharmBase):
                         "Exception occured during the multipath \
                         configuration. Please check logs."
                     )
-                    return False
+                    return
             else:
                 logging.debug("multipath-" + section + " is empty.")
 
@@ -362,7 +511,7 @@ class StorageConnectorCharm(CharmBase):
                 self.unit.status = BlockedStatus(
                     "No WWID was found. Please check multipath status and logs."
                 )
-                return False
+                return
             alias = charm_config.get('fc-lun-alias')
             ctxt['multipaths'] = {'wwid': wwid, 'alias': alias}
 
@@ -371,13 +520,14 @@ class StorageConnectorCharm(CharmBase):
         rendered_content = template.render(ctxt)
         self.mp_path.write_text(rendered_content)
         self.mp_path.chmod(0o600)
-        return True
 
-    def _iscsiadm_discovery(self):
+    def _iscsi_discovery_and_login(self):
+        """Run iscsiadm discovery and login against targets."""
         charm_config = self.model.config
         target = charm_config.get('iscsi-target')
         port = charm_config.get('iscsi-port')
-        logging.info('Launching iscsiadm discovery against target')
+        logging.info('Launching iscsiadm discovery and login against targets')
+
         try:
             subprocess.check_call(['iscsiadm', '-m', 'discovery', '-t', 'sendtargets',
                                   '-p', target + ':' + port])
@@ -386,18 +536,23 @@ class StorageConnectorCharm(CharmBase):
             self.unit.status = BlockedStatus(
                 'Iscsi discovery failed against target'
             )
+            return
 
-    def _iscsiadm_login(self):
-        # add check if already logged in, no error if it is.
         try:
-            subprocess.check_call(['iscsiadm', '-m', 'node', '--login'])
-        except subprocess.CalledProcessError:
-            logging.exception('Iscsi login failed.')
-            self.unit.status = BlockedStatus(
-                'Iscsi login failed against target'
-            )
+            subprocess.check_output(
+                ['iscsiadm', '-m', 'node', '--login'],
+                stderr=subprocess.STDOUT)
+        except subprocess.CalledProcessError as err:
+            logging.exception('Iscsi login failed. \n%s', err.output.decode("utf-8"))
+
+            # Check if exception occurred because session already present.
+            # No error if it is, otherwise set block status.
+            if "already present" not in err.output.decode("utf-8"):
+                self.unit.status = BlockedStatus(
+                    'Iscsi login failed against target'
+                )
 
-    def _fc_scan_host_successfully(self):
+    def _fc_scan_host(self):
         hba_adapters = subprocess.getoutput('ls /sys/class/scsi_host')
         logging.debug('hba_adapters: {}'.format(hba_adapters))
         if not hba_adapters:
@@ -405,7 +560,7 @@ class StorageConnectorCharm(CharmBase):
             self.unit.status = BlockedStatus(
                 "No scsi devices were found. Scan aborted"
             )
-            return False
+            return
 
         for adapter in hba_adapters.split('\n'):
             try:
@@ -419,9 +574,8 @@ class StorageConnectorCharm(CharmBase):
                 self.unit.status = BlockedStatus(
                     'Scan of the HBA adapters failed on the host.'
                 )
-                return False
+                return
         self._stored.fc_scan_ran_once = True
-        return True
 
     def _retrieve_multipath_wwid(self):
         logging.info('Retrive device WWID via multipath -ll')
@@ -440,8 +594,6 @@ class StorageConnectorCharm(CharmBase):
             self.unit.status = BlockedStatus(
                 'Multipath conf error: {}', error
             )
-            return False
-        return True
 
     def _check_if_container(self):
         """Check if the charm is being deployed on a container host."""
@@ -455,13 +607,29 @@ class StorageConnectorCharm(CharmBase):
             return True
         return False
 
+    def _configure_deferred_restarts(self):
+        """Set up deferred restarts in policy-rc.d."""
+        policy_rcd.install_policy_rcd()
+        os.chmod(
+            '/var/lib/charm/{}/policy-rc.d'.format(
+                self.app.name),
+            0o755)
+
+        charm_config = self.model.config
+        if charm_config.get('enable-auto-restarts'):
+            policy_rcd.remove_policy_file()
+        else:
+            blocked_actions = ['stop', 'restart', 'try-restart']
+            for svc in self.DEFERRED_SERVICES:
+                policy_rcd.add_policy_block(svc, blocked_actions)
+
     def _on_metrics_endpoint_relation_created(self, event):
         """Relation-created event handler for metrics-endpoint."""
         self.unit.status = MaintenanceStatus("Installing exporter")
         metrics_utils.install_exporter(self.model.resources)
 
         self._stored.prometheus_related = True
-        self.unit.status = ActiveStatus("Unit is ready")
+        self.unit.status = ActiveStatus(self._get_status_message())
 
     def _on_metrics_endpoint_relation_broken(self, event):
         """Relation-broken event handler for metrics-endpoint."""
@@ -470,7 +638,7 @@ class StorageConnectorCharm(CharmBase):
             metrics_utils.uninstall_exporter()
 
         self._stored.prometheus_related = False
-        self.unit.status = ActiveStatus("Unit is ready")
+        self.unit.status = ActiveStatus(self._get_status_message())
 
     def _on_nrpe_external_master_relation_created(self, event):
         """Relation-created event handler for nrpe-external-master."""
@@ -478,7 +646,7 @@ class StorageConnectorCharm(CharmBase):
         metrics_utils.install_exporter(self.model.resources)
 
         self._stored.nrpe_related = True
-        self.unit.status = ActiveStatus("Unit is ready")
+        self.unit.status = ActiveStatus(self._get_status_message())
 
     def _on_nrpe_external_master_relation_changed(self, event):
         """Relation-changed event handler for nrpe-external-master."""
@@ -494,7 +662,7 @@ class StorageConnectorCharm(CharmBase):
         nrpe_utils.unsync_nrpe_files()
 
         self._stored.nrpe_related = False
-        self.unit.status = ActiveStatus("Unit is ready")
+        self.unit.status = ActiveStatus(self._get_status_message())
 
 
 if __name__ == "__main__":
diff --git a/tests/functional/storage-connector-func-tests/setup.py b/tests/functional/storage-connector-func-tests/setup.py
index 3f8d801..01e3eb8 100644
--- a/tests/functional/storage-connector-func-tests/setup.py
+++ b/tests/functional/storage-connector-func-tests/setup.py
@@ -14,6 +14,7 @@
 
 """Code for setting up storage-connector tests."""
 
+import json
 import logging
 
 import zaza.model
@@ -58,3 +59,50 @@ def configure_iscsi_target():
     # Restart tgt to load new config
     restart_tgt = "systemctl restart tgt"
     zaza.model.run_on_unit('ubuntu-target/0', restart_tgt)
+
+
+def get_unit_full_hostname(unit_name):
+    """Retrieve the full hostname of a unit."""
+    for unit in zaza.model.get_units(unit_name):
+        result = zaza.model.run_on_unit(unit.entity_id, 'hostname -f')
+        hostname = result['Stdout'].rstrip()
+    return hostname
+
+
+def configure_iscsi_connector():
+    """Configure iscsi connector."""
+    iqn = 'iqn.2020-07.canonical.com:lun1'
+    unit_fqdn = get_unit_full_hostname('ubuntu')
+    iscsi_target = zaza.model.get_app_ips('ubuntu-target')[0]
+    initiator_dictionary = json.dumps({unit_fqdn: iqn})
+    conf = {
+        'storage-type': 'iscsi',
+        'initiator-dictionary': initiator_dictionary,
+        'iscsi-target': iscsi_target,
+        'iscsi-port': '3260',
+        'iscsi-node-session-auth-authmethod': 'CHAP',
+        'iscsi-node-session-auth-username': 'iscsi-user',
+        'iscsi-node-session-auth-password': 'password123',
+        'iscsi-node-session-auth-username-in': 'iscsi-target',
+        'iscsi-node-session-auth-password-in': 'secretpass',
+        'multipath-devices': '{}',
+        'enable-auto-restarts': "false"
+    }
+
+    zaza.model.set_application_config('storage-connector', conf)
+    zaza.model.wait_for_application_states(
+        states={
+            "storage-connector": {
+                "workload-status": "active",
+                "workload-status-message-prefix": "Unit is ready"
+            },
+            "ubuntu": {
+                "workload-status": "active",
+                "workload-status-message-regex": "^$"
+            },
+            "ubuntu-target": {
+                "workload-status": "active",
+                "workload-status-message-regex": "^$"
+            }
+        }
+    )
diff --git a/tests/functional/storage-connector-func-tests/tests.py b/tests/functional/storage-connector-func-tests/tests.py
index bc6c7e9..21a7d25 100644
--- a/tests/functional/storage-connector-func-tests/tests.py
+++ b/tests/functional/storage-connector-func-tests/tests.py
@@ -14,7 +14,6 @@
 
 """Encapsulate storage-connector testing."""
 
-import json
 import logging
 
 import zaza.model
@@ -29,42 +28,37 @@ class StorageConnectorTest(test_utils.BaseCharmTest):
         """Run class setup for running glance tests."""
         super(StorageConnectorTest, cls).setUpClass()
 
-    def configure_iscsi_connector(self):
-        """Configure iscsi connector."""
-        iqn = 'iqn.2020-07.canonical.com:lun1'
-        unit_fqdn = self.get_unit_full_hostname('ubuntu')
-        target_ip = zaza.model.get_app_ips('ubuntu-target')[0]
-        initiator_dictionary = json.dumps({unit_fqdn: iqn})
-        conf = {
-            'storage-type': 'iscsi',
-            'initiator-dictionary': initiator_dictionary,
-            'iscsi-target': target_ip,
-            'iscsi-port': '3260',
-            'iscsi-node-session-auth-authmethod': 'CHAP',
-            'iscsi-node-session-auth-username': 'iscsi-user',
-            'iscsi-node-session-auth-password': 'password123',
-            'iscsi-node-session-auth-username-in': 'iscsi-target',
-            'iscsi-node-session-auth-password-in': 'secretpass',
-            'multipath-devices': '{}'
-        }
+    def test_iscsi_connector_config_changed(self):
+        """Test iscsi configuration changes and wait for idle status."""
+        conf = zaza.model.get_application_config('storage-connector')
+        conf["storage-type"] = "null"
         zaza.model.set_application_config('storage-connector', conf)
+        logging.info('Wait for block status...')
+        zaza.model.wait_for_application_states(
+            states={
+                "storage-connector": {
+                    "workload-status": "blocked",
+                    "workload-status-message-prefix": "Missing"
+                },
+                "ubuntu": {
+                    "workload-status": "active",
+                    "workload-status-message-regex": "^$"
+                },
+                "ubuntu-target": {
+                    "workload-status": "active",
+                    "workload-status-message-regex": "^$"
+                }
+            }
+        )
 
-    def get_unit_full_hostname(self, unit_name):
-        """Retrieve the full hostname of a unit."""
-        for unit in zaza.model.get_units(unit_name):
-            result = zaza.model.run_on_unit(unit.entity_id, 'hostname -f')
-            hostname = result['Stdout'].rstrip()
-        return hostname
-
-    def test_iscsi_connector(self):
-        """Test iscsi configuration and wait for idle status."""
-        self.configure_iscsi_connector()
+        conf["storage-type"] = "iscsi"
+        zaza.model.set_application_config('storage-connector', conf)
         logging.info('Wait for idle/ready status...')
         zaza.model.wait_for_application_states(
             states={
                 "storage-connector": {
                     "workload-status": "active",
-                    "workload-status-message-prefix": "Unit is ready"
+                    "workload-status-message-prefix": ""
                 },
                 "ubuntu": {
                     "workload-status": "active",
@@ -79,7 +73,7 @@ class StorageConnectorTest(test_utils.BaseCharmTest):
 
     def test_validate_iscsi_session(self):
         """Validate that the iscsi session is active."""
-        unit = zaza.model.get_units('ubuntu')[0]
+        unit = zaza.model.get_units('storage-connector')[0]
         logging.info('Checking if iscsi session is active.')
         run = zaza.model.run_on_unit(unit.entity_id, 'iscsiadm -m session')
         logging.info("""iscsiadm -m session: Stdout: {}, Stderr: {}, """
@@ -87,3 +81,76 @@ class StorageConnectorTest(test_utils.BaseCharmTest):
                                            run['Stderr'],
                                            run['Code']))
         assert run['Code'] == '0'
+
+    def test_defer_service_restarts(self):
+        """Validate that the iscsi services are deferred on config-changed."""
+        unit = zaza.model.get_units('storage-connector')[0]
+
+        active_time_pre_check = zaza.model.get_systemd_service_active_time(
+            unit.entity_id, 'iscsid.service'
+        )
+        logging.info("""Service start time before config change: {}""".format(
+            active_time_pre_check))
+
+        # Modify the value of a random config option to trigger config-changed event
+        conf = zaza.model.get_application_config('storage-connector')
+        conf["iscsi-node-session-scan"] = "manual"
+        zaza.model.set_application_config('storage-connector', conf)
+
+        zaza.model.wait_for_application_states(
+            states={
+                "storage-connector": {
+                    "workload-status": "active",
+                    "workload-status-message-regex": ".*Services queued for restart.*"
+                },
+                "ubuntu": {
+                    "workload-status": "active",
+                    "workload-status-message-regex": "^$"
+                },
+                "ubuntu-target": {
+                    "workload-status": "active",
+                    "workload-status-message-regex": "^$"
+                }
+            }
+        )
+
+        active_time_post_check = zaza.model.get_systemd_service_active_time(
+            unit.entity_id, 'iscsid.service'
+        )
+        logging.info("""Service start time after config change: {}""".format(
+            active_time_post_check))
+
+        # Check if service restart is blocked
+        assert active_time_pre_check == active_time_post_check
+
+        # Run restart-iscsi-services to restart the deferred services
+        zaza.model.run_action(
+            unit.entity_id,
+            "restart-services",
+            action_params={
+                'deferred-only': True})
+        zaza.model.wait_for_application_states(
+            states={
+                "storage-connector": {
+                    "workload-status": "active",
+                    "workload-status-message-prefix": "Unit is ready"
+                },
+                "ubuntu": {
+                    "workload-status": "active",
+                    "workload-status-message-regex": "^$"
+                },
+                "ubuntu-target": {
+                    "workload-status": "active",
+                    "workload-status-message-regex": "^$"
+                }
+            }
+        )
+
+        active_time_after_restart = zaza.model.get_systemd_service_active_time(
+            unit.entity_id, 'iscsid.service'
+        )
+        logging.info("""Service start time after service restart action: {}""".format(
+            active_time_after_restart))
+
+        # Check if service restart was successful
+        assert active_time_post_check != active_time_after_restart
diff --git a/tests/functional/tests/tests.yaml b/tests/functional/tests/tests.yaml
index 6a43b9b..57551f4 100644
--- a/tests/functional/tests/tests.yaml
+++ b/tests/functional/tests/tests.yaml
@@ -11,6 +11,7 @@ gate_bundles:
 configure:
  - storage-connector-func-tests.setup.basic_target_setup
  - storage-connector-func-tests.setup.configure_iscsi_target
+ - storage-connector-func-tests.setup.configure_iscsi_connector
 
 tests:
  - storage-connector-func-tests.tests.StorageConnectorTest
diff --git a/tests/unit/test_charm.py b/tests/unit/test_charm.py
index 499e35d..37f78a5 100644
--- a/tests/unit/test_charm.py
+++ b/tests/unit/test_charm.py
@@ -10,7 +10,10 @@ from unittest.mock import PropertyMock, call, mock_open, patch
 
 import charm
 
+import charmhelpers.contrib.openstack.deferred_events as deferred_events
+
 from ops.framework import EventBase
+from ops.model import ActiveStatus, BlockedStatus
 from ops.testing import Harness
 
 
@@ -40,6 +43,7 @@ class TestCharm(unittest.TestCase):
 
     @patch("storage_connector.nrpe_utils.update_nrpe_config")
     @patch("charm.subprocess.check_call")
+    @patch("charm.subprocess.check_output")
     @patch("charm.subprocess.getoutput")
     @patch("charm.utils.is_container")
     @patch("charm.StorageConnectorCharm.MULTIPATH_CONF_PATH", new_callable=PropertyMock)
@@ -49,7 +53,9 @@ class TestCharm(unittest.TestCase):
     )
     @patch("charm.StorageConnectorCharm.ISCSI_CONF", new_callable=PropertyMock)
     @patch("charm.StorageConnectorCharm.ISCSI_CONF_PATH", new_callable=PropertyMock)
+    @patch("charm.StorageConnectorCharm._configure_deferred_restarts")
     def test_on_iscsi_install(self,
+                              m_configure_deferred_restarts,
                               m_iscsi_conf_path,
                               m_iscsi_conf,
                               m_iscsi_initiator_name,
@@ -57,6 +63,7 @@ class TestCharm(unittest.TestCase):
                               m_multipath_conf_path,
                               m_is_container,
                               m_getoutput,
+                              m_check_output,
                               m_check_call,
                               _):
         """Test installation."""
@@ -86,11 +93,17 @@ class TestCharm(unittest.TestCase):
             [
                 call("systemctl restart iscsid".split()),
                 call("systemctl restart open-iscsi".split()),
-                call("iscsiadm -m discovery -t sendtargets -p abc:443".split()),
-                call("iscsiadm -m node --login".split())
+                call("iscsiadm -m discovery -t sendtargets -p abc:443".split())
+            ],
+            any_order=False
+        )
+        m_check_output.assert_has_calls(
+            [
+                call("iscsiadm -m node --login".split(), stderr=-2)
             ],
             any_order=False
         )
+        m_configure_deferred_restarts.assert_called_once()
         self.assertTrue(self.harness.charm._stored.installed)
 
     @patch("storage_connector.nrpe_utils.update_nrpe_config")
@@ -101,7 +114,9 @@ class TestCharm(unittest.TestCase):
     @patch("charm.StorageConnectorCharm.MULTIPATH_CONF_DIR", new_callable=PropertyMock)
     @patch("charm.StorageConnectorCharm.ISCSI_CONF", new_callable=PropertyMock)
     @patch("charm.StorageConnectorCharm.ISCSI_CONF_PATH", new_callable=PropertyMock)
+    @patch("charm.StorageConnectorCharm._configure_deferred_restarts")
     def test_on_fiberchannel_install(self,
+                                     m_configure_deferred_restarts,
                                      m_iscsi_conf_path,
                                      m_iscsi_conf,
                                      m_multipath_conf_dir,
@@ -129,6 +144,7 @@ class TestCharm(unittest.TestCase):
         self.assertTrue(os.path.exists(self.harness.charm.MULTIPATH_CONF_PATH))
         self.assertTrue(self.harness.charm._stored.installed)
         m_open.assert_called_once_with("/sys/class/scsi_host/host0/scan", "w")
+        m_configure_deferred_restarts.assert_called_once()
         self.assertTrue(
             call(self.harness.charm.ISCSI_CONF, "w") not in m_open.mock_calls
         )
@@ -144,29 +160,281 @@ class TestCharm(unittest.TestCase):
         self.harness.charm.on.start.emit()
         self.assertTrue(self.harness.charm._stored.started)
 
+    def test_on_restart_services_action_mutually_exclusive_params(self):
+        """Test on restart servcices action with both deferred-only and services."""
+        action_event = FakeActionEvent(params={
+            "deferred-only": True, "services": "test_service"})
+        self.harness.charm._on_restart_services_action(action_event)
+        self.assertEqual(
+            action_event.results['failed'],
+            'deferred-only and services are mutually exclusive')
+
+    def test_on_restart_services_action_no_params(self):
+        """Test on restart servcices action with no param."""
+        action_event = FakeActionEvent(params={
+            "deferred-only": False, "services": ""})
+        self.harness.charm._on_restart_services_action(action_event)
+        self.assertEqual(
+            action_event.results['failed'],
+            'Please specify either deferred-only or services')
+
     @patch("charm.subprocess.check_call")
-    def test_on_restart_iscsi_services_action(self, m_check_call):
-        """Test on restart action."""
-        action_event = FakeActionEvent()
-        m_check_call.return_value = True
-        self.harness.charm.on_restart_iscsi_services_action(action_event)
+    @patch("charm.deferred_events.get_deferred_restarts")
+    @patch("charm.deferred_events.check_restart_timestamps")
+    def test_on_restart_services_action_deferred_only_failed(self,
+                                                             m_check_rtimestamps,
+                                                             m_get_deferred_restarts,
+                                                             m_check_call):
+        """Test on restart servcices action empty list of deferred restarts."""
+        m_get_deferred_restarts.return_value = []
+        action_event = FakeActionEvent(params={
+            "deferred-only": True, "services": ""})
+        self.harness.charm._on_restart_services_action(action_event)
+        self.assertEqual(
+            action_event.results['failed'], 'No deferred services to restart')
+
+    @patch("charm.subprocess.check_call")
+    @patch("charm.deferred_events.get_deferred_restarts")
+    @patch("charm.deferred_events.check_restart_timestamps")
+    def test_on_restart_services_action_deferred_only_success(self,
+                                                              m_check_rtimestamps,
+                                                              m_get_deferred_restarts,
+                                                              m_check_call):
+        """Test on restart servcices action with deferred only param."""
+        m_get_deferred_restarts.return_value = [deferred_events.ServiceEvent(
+            timestamp=123456,
+            service='svc',
+            reason='Reason',
+            action='restart',
+            policy_requestor_name='myapp',
+            policy_requestor_type='charm')]
+        action_event = FakeActionEvent(params={
+            "deferred-only": True, "services": ""})
+        self.harness.charm._on_restart_services_action(action_event)
         m_check_call.assert_has_calls(
             [
-                call(["systemctl", "restart", "iscsid"]),
-                call(["systemctl", "restart", "open-iscsi"])
+                call(["systemctl", "restart", "svc"])
             ],
             any_order=False
         )
         self.assertEqual(action_event.results['success'], 'True')
 
+    def test_on_restart_services_action_services_failed(self):
+        """Test on restart servcices action failed with invalid service input."""
+        action_event = FakeActionEvent(params={
+            "deferred-only": False, "services": "non_valid_service"})
+        self.harness.charm._on_restart_services_action(action_event)
+        self.assertEqual(
+            action_event.results['failed'], 'No valid services are specified.')
+
+    @patch("charm.subprocess.check_call")
+    def test_on_restart_services_actionservices_success(self,
+                                                        m_check_call):
+        """Test on restart servcices action successfully run with services param."""
+        action_event = FakeActionEvent(params={
+            "deferred-only": False, "services": "iscsid open-iscsi multipathd"})
+        self.harness.charm._on_restart_services_action(action_event)
+        m_check_call.assert_has_calls(
+            [
+                call(["systemctl", "restart", "iscsid"]),
+                call(["systemctl", "restart", "open-iscsi"]),
+                call(["systemctl", "restart", "multipathd"])
+            ],
+            any_order=True
+        )
+        self.assertEqual(action_event.results['success'], 'True')
+
+    @patch("charm.deferred_events.get_deferred_restarts")
+    def test_on_show_deferred_restarts_action(self, m_get_deferred_restarts):
+        """Test on show deferred restarts action."""
+        m_get_deferred_restarts.return_value = [deferred_events.ServiceEvent(
+            timestamp=123456,
+            service='svc',
+            reason='Reason',
+            action='restart',
+            policy_requestor_name='myapp',
+            policy_requestor_type='charm')]
+
+        action_event = FakeActionEvent()
+        self.harness.charm._on_show_deferred_restarts_action(action_event)
+        self.assertEqual(
+            action_event.results['deferred-restarts'],
+            '- 1970-01-02 10:17:36 +0000 UTC svc' +
+            '                                      Reason\n')
+
     @patch("charm.subprocess.check_call")
     def test_on_reload_multipathd_service_action(self, m_check_call):
-        """Test on reload action."""
+        """Test on reload multipathd action."""
         action_event = FakeActionEvent()
-        self.harness.charm.on_reload_multipathd_service_action(action_event)
+        self.harness.charm._on_reload_multipathd_service_action(action_event)
         m_check_call.assert_called_once_with(["systemctl", "reload", "multipathd"])
         self.assertEqual(action_event.results['success'], 'True')
 
+    @patch("charm.subprocess.check_call")
+    @patch("charm.subprocess.check_output")
+    def test_on_iscsi_discovery_and_login_action(self, m_check_output, m_check_call):
+        """Test on iscsi discovery and login action."""
+        self.harness.update_config({
+            "iscsi-target": "abc",
+            "iscsi-port": "443"
+        })
+        action_event = FakeActionEvent()
+        self.harness.charm._on_iscsi_discovery_and_login_action(action_event)
+
+        m_check_call.assert_has_calls(
+            [
+                call(['iscsiadm', '-m', 'discovery', '-t', 'sendtargets',
+                                  '-p', "abc" + ':' + "443"]),
+            ],
+            any_order=False
+        )
+        m_check_output.assert_has_calls(
+            [
+                call(['iscsiadm', '-m', 'node', '--login'], stderr=-2)
+            ],
+            any_order=False
+        )
+        self.assertEqual(action_event.results['success'], 'True')
+
+    @patch("charm.deferred_events.get_deferred_restarts")
+    def test_get_status_message(self, m_get_deferred_restarts):
+        """Test on setting active status with correct status message."""
+        m_get_deferred_restarts.return_value = []
+        self.assertEqual(self.harness.charm._get_status_message(), "Unit is ready")
+
+        m_get_deferred_restarts.return_value = [deferred_events.ServiceEvent(
+            timestamp=123456,
+            service='svc1',
+            reason='Reason1',
+            action='restart',
+            policy_requestor_name='other_app',
+            policy_requestor_type='charm'), deferred_events.ServiceEvent(
+            timestamp=234567,
+            service='svc2',
+            reason='Reason2',
+            action='restart',
+            policy_requestor_name='storage-connector',
+            policy_requestor_type='charm')]
+        self.assertEqual(
+            self.harness.charm._get_status_message(),
+            "Unit is ready. Services queued for restart: svc2"
+        )
+
+    @patch("charm.deferred_events.check_restart_timestamps")
+    @patch("charm.deferred_events.get_deferred_restarts")
+    def test_check_deferred_restarts_queue(self,
+                                           m_get_deferred_restarts,
+                                           m_check_restart_timestamps):
+        """Test check_deferred_restarts_queue decorator function."""
+        self.harness.charm.unit.status = ActiveStatus("Unit is ready")
+        m_get_deferred_restarts.return_value = [deferred_events.ServiceEvent(
+            timestamp=234567,
+            service='svc',
+            reason='Reason',
+            action='restart',
+            policy_requestor_name='storage-connector',
+            policy_requestor_type='charm')]
+
+        # Trigger decorator function by emitting update_status event
+        self.harness.charm.on.update_status.emit()
+
+        m_check_restart_timestamps.assert_called_once()
+        self.assertEqual(
+            self.harness.charm.unit.status,
+            ActiveStatus("Unit is ready. Services queued for restart: svc")
+        )
+
+    @patch("charm.policy_rcd.install_policy_rcd")
+    @patch("charm.policy_rcd.remove_policy_file")
+    @patch("charm.policy_rcd.add_policy_block")
+    @patch("charm.os.chmod")
+    def test_configure_deferred_restarts(self,
+                                         m_chmod,
+                                         m_add_policy_block,
+                                         m_remove_policy_file,
+                                         install_policy_rcd):
+        """Test on setting up deferred restarts in policy-rc.d."""
+        self.harness.update_config({'enable-auto-restarts': True})
+        self.harness.charm._configure_deferred_restarts()
+        install_policy_rcd.assert_called_once()
+        m_remove_policy_file.assert_called_once()
+        m_chmod.assert_called_once()
+
+        self.harness.update_config({'enable-auto-restarts': False})
+        self.harness.charm._configure_deferred_restarts()
+        m_add_policy_block.assert_has_calls(
+            [
+                call('iscsid', ['stop', 'restart', 'try-restart']),
+                call('open-iscsi', ['stop', 'restart', 'try-restart'])
+            ],
+            any_order=False
+        )
+
+    @patch("charm.subprocess.check_output")
+    @patch("charm.subprocess.check_call")
+    def test_iscsiadm_discovery_failed(self, m_check_call, m_check_output):
+        """Test response to iscsiadm discovery failure."""
+        self.harness.update_config({
+            "storage-type": "iscsi",
+            "iscsi-target": "abc",
+            "iscsi-port": "443"
+        })
+        self.harness.charm.unit.status = ActiveStatus("Unit is ready")
+        m_check_call.side_effect = charm.subprocess.CalledProcessError(
+            returncode=15,
+            cmd=['iscsiadm', '-m', 'discovery', '-t', 'sendtargets',
+                 '-p', "abc" + ':' + "443"],
+        )
+        self.harness.charm._iscsi_discovery_and_login()
+        self.assertEqual(
+            self.harness.charm.unit.status,
+            BlockedStatus('Iscsi discovery failed against target')
+        )
+
+    @patch("charm.subprocess.check_output")
+    @patch("charm.subprocess.check_call")
+    def test_iscsiadm_login_failed(self, m_check_call, m_check_output):
+        """Test response to iscsiadm login failure."""
+        self.harness.update_config({
+            "storage-type": "iscsi",
+            "iscsi-target": "abc",
+            "iscsi-port": "443"
+        })
+        self.harness.charm.unit.status = ActiveStatus("Unit is ready")
+        m_check_output.side_effect = charm.subprocess.CalledProcessError(
+            returncode=15,
+            cmd=['iscsiadm', '-m', 'node', '--login'],
+            output=b'iscsiadm: Could not log into all portals'
+        )
+        self.harness.charm._iscsi_discovery_and_login()
+        self.assertEqual(
+            self.harness.charm.unit.status,
+            BlockedStatus('Iscsi login failed against target')
+        )
+
+    @patch("charm.subprocess.check_output")
+    @patch("charm.subprocess.check_call")
+    def test_iscsiadm_login_exist_session(self, m_check_call, m_check_output):
+        """Test response to iscsiadm login to exist session."""
+        self.harness.update_config({
+            "storage-type": "iscsi",
+            "iscsi-target": "abc",
+            "iscsi-port": "443"
+        })
+        self.harness.charm.unit.status = ActiveStatus("Unit is ready")
+        m_check_output.side_effect = charm.subprocess.CalledProcessError(
+            returncode=15,
+            cmd=['iscsiadm', '-m', 'node', '--login'],
+            output=b'iscsiadm: default: 1 session requested, ' +
+                   b'but 1 already present.\n' +
+                   b'iscsiadm: Could not log into all portals'
+        )
+        self.harness.charm._iscsi_discovery_and_login()
+        self.assertEqual(
+            self.harness.charm.unit.status,
+            ActiveStatus("Unit is ready")
+        )
+
     @patch("storage_connector.metrics_utils.uninstall_exporter")
     @patch("storage_connector.metrics_utils.install_exporter")
     def test_on_metrics_endpoint_handlers(
diff --git a/tox.ini b/tox.ini
index 5c8e284..a401667 100644
--- a/tox.ini
+++ b/tox.ini
@@ -8,14 +8,14 @@ skip_missing_interpreters = False
 basepython = python3
 setenv =
   PYTHONPATH = {toxinidir}/lib/:{toxinidir}
-passenv = 
+passenv =
     HOME
     CHARM_BUILD_DIR
     OS_*
 whitelist_externals = charmcraft
 
 [testenv:unit]
-commands = 
+commands =
     coverage erase
     stestr run --slowest {posargs}
     coverage combine
@@ -24,7 +24,7 @@ commands =
     coverage report -m --omit lib/charms/*,templates/*
 deps = -r{toxinidir}/tests/unit/requirements.txt
        -r{toxinidir}/requirements.txt
-setenv = 
+setenv =
      {[testenv]setenv}
      PYTHON=coverage run
 
@@ -32,9 +32,9 @@ setenv =
 branch = True
 concurrency = multiprocessing
 parallel = True
-source = 
+source =
     .
-omit = 
+omit =
     .tox/*
     tests/*