Merge ~aluria/charm-grafana:bug/1822329-tests into ~prometheus-charmers/charm-grafana:rewrite

[Alvaro Uria]

* lint: passed (make lint)
* unit tests: 30 passed, 19 warnings in 1.17 seconds (make unittest)
* Func tests: 13 passed (a bootstrapped controller needs to exist; functest-grafana model will be created or used, if it already exists) - (make functional) - py36+ needs to be used

Both snap and apt install_methods are tested, as well as a fake install method ("wrong-method") to verify that the unit gets blocked.

715d850... by Alvaro Uria on 2019-07-03

Func tests: Render the overlay template

a360580... by Alvaro Uria on 2019-07-03

Fix wrong install_method output

[Chris Sanders (chris.sanders)]

A few comments inline. I notice that the git-log correctly states that unit testing are being run from the built charm directory. I'm curious that I don't see that change in this MR, maybe it was already that way? The git log doesn't seem to support that, are there changes that are somehow introduced and not shown on the MR?

Back to the unit testing. Charm building can be quite slow, unit testing needs to be very fast and those are at odds. Requiring a charm-build before a unit test will take the test time for unit tests from several seconds (or less) to several minutes (or more) per run. Whatever dependencies are causing this should be mocked instead.

If the dependency issue is another layer, the template includes an example of patching charm layers and options on charm layers. The unit testing shouldn't extend to testing code in included layers so mocking is sufficient.

A second item I noticed when looking at the repository. Several of these files do an import, catch, and apt install. That really should not happen, dependencies should be installed with the install hook not at run time during an import.

review: Needs Fixing

[Stuart Bishop (stub)]

Seems fine from my POV. Code looks fine. Mostly adding tests, which I can't really comment on as they are using what is essentially now a proprietary charm test framework and quite alien. The intents seem fine, but I can't comment on correctness or recommend alternate approaches. The extent of new tests will likely cement this as a BootStack maintained charm. Over here we seem to be proceeding with the plan to use zaza.

Unrelated to this particular branch, I think the charm would benefit from using layer:status. Reasoning about status messages becomes much easier, with layer:status buffering them to ensure that the most important one is what is presented to the end user and you stop needing to worry about a 'maintenance' message obliterating an important 'blocked' or 'active'. It should just drop in, monkey patching hookenv.status_set if you don't want to switch to the slightly nicer call syntax.

Also unrelated to this particular branch, as charms are supposed to encode best practice, the charm should really decide what that is and only support one installation method. This of course might be difficult for an upgraded deployment, but it would be good if it could be considered a goal. Choice here means more complexity (like this test suite, needing to run many more combinations of options), and a good proportion of installations we need to support not being best practice.

review: Approve

[Xav Paice (xavpaice)]

Can this please be rebased against the https://code.launchpad.net/~llama-charmers/charm-grafana/+git/charm-grafana/+ref/master and resubmitted?

review: Needs Fixing

Unmerged commits

a360580... by Alvaro Uria on 2019-07-03

Fix wrong install_method output

ea07818... by Nicolas Pochet on 2019-07-01

Move to new default repository

* According to Grafana's documentation, one needs to use the new
repository.
* This change replaces the old repository and key by the new ones.

Closes-Bug: 1814303

715d850... by Alvaro Uria on 2019-07-03

Func tests: Render the overlay template

bdc7c06... by Alvaro Uria on 2019-07-02

Func test to verify if dashboards were imported

7611dba... by Alvaro Uria on 2019-07-01

Add initial func tests

86ac5e1... by Alvaro Uria on 2019-07-01

Start adding lib_grafana.db unit tests

1e11f71... by Alvaro Uria on 2019-07-01

Add more unit tests on the helpers lib

3193cee... by Alvaro Uria on 2019-06-28

Add extensive unit test coverage

5bd2d0d... by Alvaro Uria on 2019-06-28

Unit tests: remove unittest.mock usage (only pytest)

diff --git a/.gitignore b/.gitignore
index 6a23b4b..d5218f7 100644
--- a/.gitignore
+++ b/.gitignore
@@ -20,3 +20,7 @@ repo-info
 
 # reports
 reports/*
+
+# included layers or interfaces
+interfaces/
+layers/
diff --git a/actions/grafana_utils.py b/actions/grafana_utils.py
index 0079dec..dfd4f57 100644
--- a/actions/grafana_utils.py
+++ b/actions/grafana_utils.py
@@ -1,4 +1,4 @@
-#!/usr/bin/python3
+#!/usr/local/sbin/charm-env python3
 
 from charmhelpers.core.hookenv import (
     config,
diff --git a/config.yaml b/config.yaml
index 5db3f1a..c4dcd96 100644
--- a/config.yaml
+++ b/config.yaml
@@ -1,7 +1,7 @@
 options:
   install_sources:
     default: |
-      - 'deb https://packagecloud.io/grafana/stable/debian/ stretch main'
+      - 'deb https://packages.grafana.com/oss/deb stable main'
     type: string
     description: |
       YAML list of additional installation sources, as a string. The number of
@@ -13,68 +13,34 @@ options:
     default: |
       - |
           -----BEGIN PGP PUBLIC KEY BLOCK-----
-          Version: GnuPG v1.4.11 (GNU/Linux)
+          Version: GnuPG v1
 
-          mQINBFu7jn8BEAC+f2xaHm8VnvpsoK2mD9dQAPDf9Pslvv0EH0Rhs6D54LpkF7hj
-          VjeUH+cpNha7Gcr4vTubcVVAsq5mHplatn54UmHUg1inuNbe32vVcoDF/UPtg4tg
-          nq9CbGFQvCRX4gEGVfiOsoipKhu50hv09LEyN7y8bYoWSrYcjPL2fswr94Bm4+Kq
-          IFdHwKBKYrJQ390NuIrP+ncBDIJ5ubdMtM5S2gpKEW4daO/4pnfr7YmgfQr34+Xe
-          5SkVcOix3CqPXwQ/OyTPwhZJssXljxcWbcOM09hCZBRqADAshuFJwlvn/meXRWK4
-          5Mlnr3BaCEDhLgaLHlBRdT0jF6LYl4K0qY2o60NUlnPFWLRYB+jJFFISFrhiwpoQ
-          mo+6erbBa2AHB7MoZmzpXzWDz8UQSuxw1UYKQYU21f4aLiu14Q2bvaICLY518+JQ
-          z8lgmT25MKiMNIEcx015gvuxSfbXDMRG3piGZmB4UCMHqkCY5dp/UtTfHnv1V9OB
-          T+jOaSoVc9qsSwzUPK3ThhfGkUTtCwR+MY/tnKj4BZdpQLdwv7KIrENm9Dqwg/8I
-          T2fm7xA2DyXSRtqOrQfEXLMeI2A9r+bUSBWM8CqjwPVjeYzz1Fghs/twRlnQdB5W
-          /6bJS7u/abwnSU3GikUG1iW153XEEEinvU/7Q7Ehy6abHfWRlb5dEtt+SQARAQAB
-          tGtodHRwczovL3BhY2thZ2VjbG91ZC5pby9ncmFmYW5hL3N0YWJsZSAoaHR0cHM6
-          Ly9wYWNrYWdlY2xvdWQuaW8vZG9jcyNncGdfc2lnbmluZykgPHN1cHBvcnRAcGFj
-          a2FnZWNsb3VkLmlvPokCOAQTAQIAIgUCW7uOfwIbLwYLCQgHAwIGFQgCCQoLBBYC
-          AwECHgECF4AACgkQSZ5/GSNGcgEVqg//d1LG+T2ouY6myTZMuiGOPeks1KXSb+DX
-          KH2zajrLjp4fCjXTAEZVFawi90G77elAhdxi6bqmP68ZMBb6W8DfZH+x2evYjH/g
-          zgqYbawSRwJQPwNpQgRY4vXwqlgExl6CFfv7IyoSGY+ZQZ6kmslhcmte7f8h43Qq
-          GBfXhKg2yz4Wyl9g+0+aUr9tr9soLfudh1nYq7Zh+0KCGtV39/bLy08vQNeFJBnH
-          ZN41kJP9rdAgpnCyLEBw9Rm1K5JNCy3uihM460xG0Jp8otNJXT2tbxhh2A+q+reH
-          EHCrezvhNpzu+egYv3F/2iJssOZCw+f/3FFXAMy0RJtdTp/3NpjYcP+OxiM51ghe
-          NZGyOwvhcr3XK7SWyFekNKlCAOkLJ0s+PJrqCS/LRQYd0JcjsbTZ3eCElNl7CXzF
-          Uce3jnQ9vpOJpFsTZmBoaWuvedkYTIC6BdrkSd8yRHVmnRlP8dg9rHXWobATgVwT
-          XSl/zM3xlvgzFSgrYpVfQ5d8A6D+YrvA9nuC84mRf60fg+rORrwQed6M1M4YcdwB
-          jsVHyevY4oDMkkwQ0mOLbbQeTalm5RxrRvqocOfgwTTVrBxJuQIt51TC8KP1vcny
-          fkuE+hq8k1MWJlTGAMg2w/MJFwc1yE4+op+SJk8qB9vJ81RbOGlUaY2D4CCC4MLF
-          rrFFmWyR6By5Ag0EW7uOfwEQAOHSuWMTXEjN7THgi1zhWWlolcunuAYGas8hlB6U
-          PaV8oTMEII3xwR4STZcpIsEd31cuq3lwwRS8y7HpDhYfhY8uuIXOQ7cCXzEkXJ1H
-          YX+3WMnCnzGe+k2u3sL9TGdcNludFHEMiNJrRIY1RvZGCsT+r1FE2T0P8t4zvog6
-          986wwKXu4TUk0nyGW6CWP+3mZOIu5BLSKvusWex6c2sbYUuSDPYwpq4atsk+NDmY
-          e0bZq4SPFzluRs6QI20rxZLimmkplBoatblhOIefG7vfIvRBaitFlaVoHJW15Yos
-          s88eJZTfkXN9WIDGPj7mZwkgxTrCX/8/aAiBgVNro4x2tFRNJlTqGLY5eZzNoved
-          rxcJbJSag0fP3WxNH2TTAadBhTCSN3v2W+Tg1QNM+j0z4le23YYNRHausdIrXgct
-          fMFZMbR0so7ROuw+RI/ZuvyAu5vzvdtaRaUkWaDCWk5t/bzWtfsl80uvdi7AIqWp
-          TYsTTLz0h/B6Kz0l36i3S0xHDBQPIeGNAOHR/7oTbrpt0C0d4lwT8CTOeekFWCpk
-          Qxh4NRIf7NzV2HXCh464nO+NCT7arB1Hy8GRgtw92ldOQfFu083B0617DqiV9Mqz
-          lE4/kuG0khNNTW7GqqHf8BTLcE/pJx83rxrkvP+WOPs+P4csJ1dWPMkULUvXDX3F
-          aChZABEBAAGJBD4EGAECAAkFAlu7jn8CGy4CKQkQSZ5/GSNGcgHBXSAEGQECAAYF
-          Alu7jn8ACgkQQPNwofkIG2SO4BAAoFBrV2a2Dxpl7OJL7nefLUCWCIZeEMV5sQ2q
-          JMStELizea/qbndBYCdSQQJG3j2E2rbWafKIOJxIrcOGNDs1ufxIknWvjUY4AaGC
-          Eo2EQ2iIuQQjwfWJ0vz5nsYuaWmdRSMdeHjpMvnJ78CFebbBQy3n0xSWF1XH7Y5+
-          n835NYfMdeIXlXQvx/6Hbli3zqM4dKm3+aOFmR1h5s5tBk8Off9G7huN8DfJ6Q/L
-          i8nZMY01hFIXV9sjozsyFLsqEckpXRIN9FeA6nKMNqo/XAsgjgaZWMCzzL9yENh7
-          r7V7hc0XonC1Fp+ET2my1DvP74Nr9vnKdynZVxIKnufSh7AIAGulCpygcGXZuK7Y
-          W6SWUKMVGwqZyNSy5WMF4useayejebEHO/X9BBXfvhyZYKDmniYLFp4PpiNzQhrG
-          cpkitEF6iRW2jx7Bcce0AuHIZQmaycOzVGG7CV4xU9NAIopa6HMMMIEDnUBY/qm4
-          v4HAGLmy4Qw1p7b0u2LzE1k3jsZ5Kum4qUQVNzZEM+5O6Ok0d4CcZb29DkhLHUvh
-          8T6Yc9MxvHstuY5vnqZLnPTfqbsaeTQAUoUnRmKqxxsur0j69riDYaoqSJwZEGSn
-          9ggepFMbFDq0Kvw28jxKm9CGVSkZ7EsBYXtIDGHD+2MqktkKR4lveU5069toFAyv
-          kh41yekOWhAAmID+8sSRNtSmrNAGuRWxtLW6VNG8jthz1NBgsgLq+aTtGebM0uEU
-          bEJgM8JKG3YkWSAr4kNyXiCJeaylSCrYGCbBCFvp1xR5w3sJz61UnVdDtV0KstpM
-          YehEVQC8erAS6dywztkW7MaTUKWQ16Rxqsenkw7rks08iTeWF3AxKU4fxjq8S6Uc
-          qGJ7eqdNIbJpAOCRWX66SVd02kieEg71yBvM7f35j80ruD+EaqG+5QqhNhoO5H9n
-          srsLH+X3IUQw7F16j3/NIumxUygbgsioA4ZEBKdsuXU/2eHJ1ywJ52YB0ZHGlYqA
-          toejg/dvFGdtT24hKo4hedEc8ymBLujeHpZe3x3u/dvPftwBAyrx+txOTC5luv1q
-          Ttqd5NRqsnw2KArA8ZM7iPJinvUaoIZVDPJghh7b+0PrYjM+fpFmUeAAN7jgttvO
-          DK11MKLqaRvk4njE//vfxjZpSJaktlUJTFpzwXuYQbuTdwDJNJcuFVmjfMWT5tKf
-          2cy8N9cgPhrulZDbYU/S8ZOFJUQ4qpHpf+q+NDGnucM3kCNkOMgqeBfBvC5wJP5C
-          ZHoaKVW9+o1CKmFKYz+1woY6qugYB/8Uy7gy3C9qGbi7UZwMFUJUCYxu5htHuB/a
-          tTUJ4nM//ichv9TCsTA5X/tYJBD0USEVl0bMV4CtS+qP2il17D846bA=
-          =ewdt
+          mQENBFiHXVIBCADr3VDEAGpq9Sg/xrPVu1GGqWGXdbnTbbNKeveCtFHZz7/GSATW
+          iwiY1skvlAOBiIKCqJEji0rZZgd8WxuhdfugiCBk1hDTMWCpjI0P+YymV77jHjYB
+          jHrKNlhb+aLjEd9Gf2EtbKUT1fvGUkzlVrcRGSX/XR9MBZlgja7NIyuVbn3uwZQ4
+          jflWSNSlvMpohNxTFkrBFTRrCJXhbDLfCS46+so22CP3+1VQyqJ7/6RWK9v9KYdS
+          AVNgILXMggSrMqha4WA1a/ktczVQXNtP8IuPxTdp9pNYsklOTmrFVeq3mXsvWh9Q
+          lIhpYHIZlTZ5wVBq4wTRchsXC5MubIhz+ASDABEBAAG0GkdyYWZhbmEgPGluZm9A
+          Z3JhZmFuYS5jb20+iQE4BBMBAgAiBQJYh11SAhsDBgsJCAcDAgYVCAIJCgsEFgID
+          AQIeAQIXgAAKCRCMjDTFJAmMthxJB/9Id6JrwqRkJW+eSBb71FGQmRsJvNFR8J+3
+          NPVhJNkTFFOM7TnjAMUIv+LYEURqGcceTNAN1aHq/7n/8ybXucCS0CnDYyNYpyVs
+          tWJ3FOQK3jPrmziDCWPQATqMM/Z2auXVFWrDFqfh2xKZNjuix0w2nyuWB8U0CG2U
+          89w+ksPJblGGU5xLPPzDQoAqyZXY3gpGGTkCuohMq2RWYbp/QJSQagYhQkKZoJhr
+          XJlnw4At6R1A5UUPzDw6WJqMRkGrkieE6ApIgf1vZSmnLRpXkqquRTAEyGT8Pugg
+          ee6YkD19/LK6ED6gn32StY770U9ti560U7oRjrOPK/Kjp4+qBtkQuQENBFiHXVIB
+          CACz4hO1g/4fKO9QWLcbSWpB75lbNgt1kHXP0UcW8TE0DIgqrifod09lC85adIz0
+          zdhs+00lLqckM5wNbp2r+pd5rRaxOsMw2V+c/y1Pt3qZxupmPc5l5lL6jzbEVR9g
+          ygPaE+iabTk9Np2OZQ7Qv5gIDzivqK2mRHXaHTzoQn2dA/3xpFcxnen9dvu7LCpA
+          CdScSj9/UIRKk9PHIgr2RJhcjzLx0u1PxN9MEqfIsIJUUgZOoDsr8oCs44PGGIMm
+          cK1CKALLLiC4ZM58B56jRyXo18MqB6VYsC1X9wkcIs72thL3tThXO70oDGcoXzoo
+          ywAHBH63EzEyduInOhecDIKlABEBAAGJAR8EGAECAAkFAliHXVICGwwACgkQjIw0
+          xSQJjLbWSwf/VIM5wEFBY4QLGUAfqfjDyfGXpcha58Y24Vv3n6MwJqnCIbTAaeWf
+          30CZ/wHg3NNIMB7I31vgmMOEbHQdv0LPTi9TG205VQeehcpNtZRZQ0D8TIetbxyi
+          Emmn9osig9U3/7jaAWBabE/9bGx4TF3eLlEH9wmFrNYeXvgRqmyqVoqhIMCNAAOY
+          REYyHyy9mzr9ywkwl0aroBqhzKIPyFlatZy9oRKllY/CCKO9RJy4DZidLphuwzqU
+          ymdQ1sqe5nKvwG5GvcncPc3O7LMevDBWnpNNkgERnVxCqpm90TuE3ONbirnU4+/S
+          tUsVU1DERc1fjOCnAm4pKIlNYphISIE7OQ==
+          =0pMC
           -----END PGP PUBLIC KEY BLOCK-----
     type: string
     description: |
diff --git a/lib/lib_grafana.py b/lib/lib_grafana.py
deleted file mode 100644
index 8a4405f..0000000
--- a/lib/lib_grafana.py
+++ /dev/null
@@ -1,43 +0,0 @@
-from charmhelpers.core import hookenv, host, unitdata
-from charms.reactive import remove_state
-
-
-APT_COMMON = '/var/lib/grafana'
-SNAP_NAME = 'grafana'
-SNAP_COMMON = '/var/snap/{}/common/data'.format(SNAP_NAME)
-
-
-class GrafanaHelper():
-    def __init__(self):
-        self.charm_config = hookenv.config()
-
-    @property
-    def password(self):
-        if self.charm_config.get('admin_password', False):
-            return self.charm_config.get('admin_password')
-
-        kv = unitdata.kv()
-        if kv.get('grafana.admin_password', False):
-            return kv.get('grafana.admin_password')
-
-        passwd = host.pwgen(16)
-        kv.set('grafana.admin_password', passwd)
-        return passwd
-
-    @property
-    def nagios_context(self):
-        nag_ctxt = self.charm_config.get('nagios_context', False)
-        if not nag_ctxt:
-            nag_ctxt = 'UNKNOWN'
-        return nag_ctxt
-
-
-def data_path():
-    data_dir = {'snap': SNAP_COMMON, 'apt': APT_COMMON}
-    kv = unitdata.kv()
-    source = kv.get('install_method')
-    if source in ('snap', 'apt'):
-        return data_dir[source]
-    else:
-        hookenv.status_set('blocked', 'Unsupported install_method')
-        remove_state('grafana.installed')
diff --git a/lib/lib_grafana/db.py b/lib/lib_grafana/db.py
index ad8a573..4abe3fd 100644
--- a/lib/lib_grafana/db.py
+++ b/lib/lib_grafana/db.py
@@ -149,7 +149,7 @@ class DbManagement():
                 stmt = ('UPDATE DATA_SOURCE SET basic_auth_user = ?,'
                         ' basic_auth_password = ?, basic_auth = 0')
                 values = ('', '')
-        return (stmt, values)
+        return [stmt, values]
 
     def delete_from_datasource(self, type, url):
         query = 'DELETE FROM DATA_SOURCE WHERE type=? AND url=?'
diff --git a/lib/lib_grafana/helpers.py b/lib/lib_grafana/helpers.py
index eb2ecfb..8e258d8 100644
--- a/lib/lib_grafana/helpers.py
+++ b/lib/lib_grafana/helpers.py
@@ -306,9 +306,13 @@ class GrafanaHelper():
         # Since there are deployments configured before the SNAP support, we need
         # to leave APT as the default install_method
         source = self.charm_config.get('install_method', 'apt')
+        if source not in ('snap', 'apt'):
+            raise GrafanaBlockedError('Unsupported install_method')
+
         kv = unitdata.kv()
         if not kv.get('install_method', False):
             kv.set('install_method', source)
+
         elif kv.get('install_method') != source:
             raise GrafanaBlockedError(
                 'install_method changes unsupported,'
diff --git a/reactive/grafana.py b/reactive/grafana.py
index 17eda8d..fa5969c 100644
--- a/reactive/grafana.py
+++ b/reactive/grafana.py
@@ -22,7 +22,7 @@ def install_packages():
     try:
         source = helper.verify_install_method()
     except GrafanaBlockedError as error:
-        hookenv.status_set('blocked', error)
+        hookenv.status_set('blocked', str(error))
         return
 
     if source == 'apt':
@@ -34,10 +34,8 @@ def install_packages():
         # NOTE(aluria): precise is the last supported Ubuntu release, so
         # anything below 'p' is actually newer than xenial (systemd support)
         helper.install_method_snap()
-        hookenv.status_set('blocked', 'Missing relations: grafana-source')
-    else:
-        hookenv.status_set('blocked', 'Unsupported install_method')
-        return
+
+    hookenv.status_set('blocked', 'Missing relations: grafana-source')
     set_flag('grafana.installed')
 
 
@@ -47,7 +45,7 @@ def install_plugins():
     try:
         plugins_dir = helper.plugins_dir
     except GrafanaBlockedError as error:
-        hookenv.status_set('blocked', error)
+        hookenv.status_set('blocked', str(error))
         clear_flag('grafana.installed')
         return
 
@@ -90,7 +88,7 @@ def setup_grafana():
         clear_flag('grafana.started')
         hookenv.status_set('active', 'Completed configuring grafana')
     except GrafanaBlockedError as error:
-        hookenv.status_set('blocked', error)
+        hookenv.status_set('blocked', str(error))
 
 
 @when('grafana.started')
@@ -122,7 +120,7 @@ def restart_grafana():
         hookenv.status_set('active', 'Grafana is running')
         set_flag('grafana.started')
     except GrafanaBlockedError as error:
-        hookenv.status_set('blocked', error)
+        hookenv.status_set('blocked', str(error))
 
 
 @when('grafana.started')
@@ -134,7 +132,7 @@ def update_nrpe_config(svc):
     try:
         helper.update_nrpe_config()
     except GrafanaBlockedError as error:
-        hookenv.status_set('blocked', error)
+        hookenv.status_set('blocked', str(error))
 
 
 @when_not('nrpe-external-master.available')
@@ -187,7 +185,7 @@ def sources_gone(relation):
     hookenv.log('Removing datasource: {}'.format(ds))
 
     if ds:
-        db.delete_from_datasource()
+        db.delete_from_datasource(ds['type'], ds['url'])
     relation.set_local = None
 
 
diff --git a/tests/functional/bundle.yaml b/tests/functional/bundle.yaml
new file mode 100644
index 0000000..381d91d
--- /dev/null
+++ b/tests/functional/bundle.yaml
@@ -0,0 +1,189 @@
+series: bionic
+applications:
+  ceph-mon:
+    annotations:
+      gui-x: '750'
+      gui-y: '500'
+    charm: cs:ceph-mon
+    num_units: 3
+    options:
+      expected-osd-count: 3
+  ceph-osd:
+    annotations:
+      gui-x: '1000'
+      gui-y: '500'
+    charm: cs:ceph-osd
+    num_units: 3
+    options:
+      osd-devices: /srv/osd
+      use-direct-io: False
+      bluestore: False
+  cinder:
+    annotations:
+      gui-x: '750'
+      gui-y: '0'
+    charm: cs:cinder
+    num_units: 1
+    options:
+      worker-multiplier: 0.1
+      block-device: None
+      glance-api-version: 2
+  cinder-ceph:
+    annotations:
+      gui-x: '750'
+      gui-y: '250'
+    charm: cs:cinder-ceph
+    num_units: 0
+  glance:
+    annotations:
+      gui-x: '250'
+      gui-y: '0'
+    charm: cs:glance
+    num_units: 1
+    options:
+      worker-multiplier: 0.1
+  keystone:
+    annotations:
+      gui-x: '500'
+      gui-y: '0'
+    charm: cs:keystone
+    num_units: 1
+    options:
+      worker-multiplier: 0.1
+  mysql:
+    annotations:
+      gui-x: '0'
+      gui-y: '250'
+    charm: cs:percona-cluster
+    num_units: 1
+    options:
+      max-connections: 1000
+      innodb-buffer-pool-size: 256M
+      tuning-level: fast
+  neutron-api:
+    annotations:
+      gui-x: '500'
+      gui-y: '500'
+    charm: cs:neutron-api
+    num_units: 1
+    options:
+      worker-multiplier: 0.1
+      neutron-security-groups: true
+      overlay-network-type: "gre vxlan"
+      flat-network-providers: physnet1
+  neutron-openvswitch:
+    annotations:
+      gui-x: '250'
+      gui-y: '500'
+    charm: cs:neutron-openvswitch
+    num_units: 0
+  nova-cloud-controller:
+    annotations:
+      gui-x: '0'
+      gui-y: '500'
+    charm: cs:nova-cloud-controller
+    num_units: 1
+    options:
+      worker-multiplier: 0.1
+      network-manager: Neutron
+      ram-allocation-ratio: '64'
+      cpu-allocation-ratio: '64'
+  nova-compute:
+    annotations:
+      gui-x: '250'
+      gui-y: '250'
+    charm: cs:nova-compute
+    num_units: 1
+    options:
+      enable-live-migration: False
+      enable-resize: False
+      migration-auth-type: ssh
+      force-raw-images: False
+  rabbitmq-server:
+    annotations:
+      gui-x: '500'
+      gui-y: '250'
+    charm: cs:rabbitmq-server
+    num_units: 1
+
+  prometheus:
+    charm: cs:prometheus2
+    num_units: 1
+    options:
+      label-juju-units: true
+  telegraf:
+    charm: cs:telegraf
+    options:
+      hostname: '{unit}'
+      prometheus_output_port: default
+  prometheus-ceph-exporter:
+    charm: cs:prometheus-ceph-exporter
+    num_units: 1
+
+relations:
+- - nova-compute:ceph-access
+  - cinder-ceph:ceph-access
+- - nova-compute:amqp
+  - rabbitmq-server:amqp
+- - keystone:shared-db
+  - mysql:shared-db
+- - nova-cloud-controller:identity-service
+  - keystone:identity-service
+- - glance:identity-service
+  - keystone:identity-service
+- - neutron-api:identity-service
+  - keystone:identity-service
+- - neutron-openvswitch:neutron-plugin-api
+  - neutron-api:neutron-plugin-api
+- - neutron-api:shared-db
+  - mysql:shared-db
+- - neutron-api:amqp
+  - rabbitmq-server:amqp
+- - glance:shared-db
+  - mysql:shared-db
+- - glance:amqp
+  - rabbitmq-server:amqp
+- - nova-cloud-controller:image-service
+  - glance:image-service
+- - nova-compute:image-service
+  - glance:image-service
+- - nova-cloud-controller:cloud-compute
+  - nova-compute:cloud-compute
+- - nova-cloud-controller:amqp
+  - rabbitmq-server:amqp
+- - nova-compute:neutron-plugin
+  - neutron-openvswitch:neutron-plugin
+- - neutron-openvswitch:amqp
+  - rabbitmq-server:amqp
+- - nova-cloud-controller:shared-db
+  - mysql:shared-db
+- - nova-cloud-controller:neutron-api
+  - neutron-api:neutron-api
+- - cinder:image-service
+  - glance:image-service
+- - cinder:amqp
+  - rabbitmq-server:amqp
+- - cinder:identity-service
+  - keystone:identity-service
+- - cinder:cinder-volume-service
+  - nova-cloud-controller:cinder-volume-service
+- - cinder-ceph:storage-backend
+  - cinder:storage-backend
+- - ceph-mon:client
+  - nova-compute:ceph
+- - cinder:shared-db
+  - mysql:shared-db
+- - ceph-mon:client
+  - cinder-ceph:ceph
+- - ceph-mon:client
+  - glance:ceph
+- - ceph-osd:mon
+  - ceph-mon:osd
+- - prometheus-ceph-exporter
+  - ceph-mon:client
+- - prometheus:target
+  - telegraf:prometheus-client
+- - prometheus:target
+  - prometheus-ceph-exporter
+- - telegraf
+  - ceph-mon
diff --git a/tests/functional/conftest.py b/tests/functional/conftest.py
index dac4c73..e0f6205 100644
--- a/tests/functional/conftest.py
+++ b/tests/functional/conftest.py
@@ -1,4 +1,3 @@
-#!/usr/bin/python3
 '''
 Reusable pytest fixtures for functional testing
 
@@ -14,10 +13,10 @@ import json
 import os
 import subprocess
 import uuid
+
 import juju
 from juju.controller import Controller
 from juju.errors import JujuError
-
 import pytest
 
 STAT_CMD = '''python3 - <<EOF
diff --git a/tests/functional/overlay.yaml.j2 b/tests/functional/overlay.yaml.j2
new file mode 100644
index 0000000..cab782a
--- /dev/null
+++ b/tests/functional/overlay.yaml.j2
@@ -0,0 +1,11 @@
+applications:
+  {{ appname }}:
+    series: {{ series }}
+    charm: {{ charm_path }}
+    num_units: 1
+    options:
+      install_method: {{ install_method }}
+
+relations:
+- - prometheus:grafana-source
+  - {{ appname }}:grafana-source
diff --git a/tests/functional/requirements.txt b/tests/functional/requirements.txt
index f76bfbb..b2e6ce6 100644
--- a/tests/functional/requirements.txt
+++ b/tests/functional/requirements.txt
@@ -1,4 +1,5 @@
 flake8
+Jinja2
 juju
 mock
 pytest
diff --git a/tests/functional/test_deploy.py b/tests/functional/test_deploy.py
index ea7436e..47794ef 100644
--- a/tests/functional/test_deploy.py
+++ b/tests/functional/test_deploy.py
@@ -1,50 +1,200 @@
 import os
+import subprocess
+
+import asyncio
+import json
 import pytest
+import requests
+import yaml
+import jinja2
 
 # Treat all tests as coroutines
 pytestmark = pytest.mark.asyncio
 
-series = ['xenial', 'bionic']
-charm_build_dir = os.getenv('CHARM_BUILD_DIR', '.').rstrip('/')
+INSTALL_METHOD = ['apt', 'snap', 'wrong-method']
+FAKE_INSTALL_METHOD = 'wrong-method'
+SERIES = ['xenial', 'bionic']
+CHARM_BUILD_DIR = os.getenv('CHARM_BUILD_DIR', '.').rstrip('/')
+
+################
+# Helper funcs #
+################
+
 
+def render(templates_dir, template_name, context):
+    templates = jinja2.Environment(
+        loader=jinja2.FileSystemLoader(templates_dir))
+    template = templates.get_template(template_name)
+    return template.render(context)
 
 ###################
 # Custom fixtures #
 ###################
 
 
[e-mail]
-async def apps(model):
-    apps = []
-    for entry in series:
-        app = model.applications['grafana-{}'.format(entry)]
-        apps.append(app)
[e-mail](scope='module', params=SERIES)
+async def apps(request, model):
+    series = request.param
+    apps = model.applications.get('grafana-{}'.format(series))
     return apps
 
 
[e-mail]
-async def units(apps):
-    units = []
-    for app in apps:
-        units.extend(app.units)
-    return units
[e-mail](scope='module', params=SERIES)
+async def units(request, apps):
+    return apps.units
 
 
[e-mail]('series', series)
-async def test_grafana_deploy(model, series):
-    # Starts a deploy for each series
-    await model.deploy('{}/grafana'.format(charm_build_dir), series=series,
-                       application_name='grafana-{}'.format(series))
-    assert True
[e-mail](scope='module')
+async def model(controller):
+    model_name = 'functest-grafana'
+    if model_name not in await controller.list_models():
+        _model = await controller.add_model(model_name,
+                                            cloud_name=os.getenv('PYTEST_CLOUD_NAME'),
+                                            region=os.getenv('PYTEST_CLOUD_REGION'),
+                                            )
+    else:
+        _model = await controller.get_model(model_name)
+
+    # https://github.com/juju/python-libjuju/issues/267
+    subprocess.check_call(['juju', 'models'])
+    while model_name not in await controller.list_models():
+        await asyncio.sleep(1)
+    yield _model
+    await _model.disconnect()
+
+
[e-mail](scope='module')
+async def deploy_openstack(model):
+    dir_path = os.path.dirname(os.path.realpath(__file__))
+    bundle_path = os.path.join(dir_path, 'bundle.yaml')
+
+    if os.path.exists(bundle_path):
+        # Note(aluria): using await model.deploy(bundle_path) will raise an error
+        # because ceph-mon application already exists and cannot be added
+        # OTOH, 'juju deploy bundle.yaml' will try to finalize a previous run
+        subprocess.check_call([
+            'juju',
+            'deploy',
+            '-m',
+            model.info.name,
+            bundle_path,
+        ])
+        assert True
+    else:
+        assert False
+
+
[e-mail](scope='module',
+                params=[[series, install_method]
+                        for series in SERIES for install_method in INSTALL_METHOD],
+                ids=['{}-{}'.format(series, install_method)
+                     for series in SERIES for install_method in INSTALL_METHOD])
+async def deploy_app(request, model):
+    series, install_method = request.param
+    app_name = 'grafana-{series}-{imethod}'.format(series=series, imethod=install_method)
+    dir_path = os.path.dirname(os.path.realpath(__file__))
+    bundle_path = os.path.join(dir_path, 'bundle.yaml')
+    overlay_path = os.path.join(CHARM_BUILD_DIR, 'overlay.yaml')
+    context = {
+        'appname': app_name,
+        'series': series,
+        'install_method': install_method,
+        'charm_path': os.path.join(CHARM_BUILD_DIR, 'grafana'),
+    }
+    # render overlay config file
+    rendered = render(dir_path, 'overlay.yaml.j2', context)
+    with open(overlay_path, 'w') as fd:
+        fd.write(rendered)
 
+    subprocess.check_call([
+        'juju',
+        'deploy',
+        '-m',
+        model.info.name,
+        bundle_path,
+        '--overlay',
+        overlay_path,
+    ])
+    while True:
+        try:
+            grafana_app = model.applications[app_name]
+            break
+        except KeyError:
+            await asyncio.sleep(5)
+    yield grafana_app
 
 #########
 # Tests #
 #########
 
 
-async def test_grafana_status(apps, model):
-    # Verifies status for all deployed series of the charm
-    for app in apps:
-        await model.block_until(lambda: app.status == 'active')
+async def test_openstack_deploy(deploy_openstack, model):
+    dir_path = os.path.dirname(os.path.realpath(__file__))
+    bundle_path = os.path.join(dir_path, 'bundle.yaml')
+    with open(bundle_path, 'r') as fd:
+        num_apps = len(yaml.safe_load(fd)["applications"].keys())
+
+    await model.block_until(lambda: (len(model.applications) >= num_apps
+                                     and all(app.status == 'active'
+                                             for name, app in model.applications.items()
+                                             if name.find(FAKE_INSTALL_METHOD) == -1)),
+                            timeout=3600)
+    assert True
+
+
+async def test_grafana_status(deploy_app, model):
+    config = await deploy_app.get_config()
+    install_method = config['install_method']['value']
+    if install_method == FAKE_INSTALL_METHOD:
+        status = 'blocked'
+        message = 'Unsupported install_method'
+    else:
+        status = 'active'
+        message = 'Grafana is running'
+
+    await model.block_until(lambda: (deploy_app.status == status
+                                     and all(unit.workload_status_message == message
+                                             for unit in deploy_app.units)),
+                            timeout=900)
     assert True
+
+
+async def test_grafana_imported_dashboards(deploy_app, model):
+    config = await deploy_app.get_config()
+    install_method = config['install_method']['value']
+    if install_method == FAKE_INSTALL_METHOD:
+        status = 'blocked'
+        message = 'Unsupported install_method'
+    else:
+        status = 'active'
+        message = 'Grafana is running'
+
+    port = config['port']['value']
+    units = deploy_app.units
+    for grafana_unit in units:
+        await model.block_until(lambda: (grafana_unit.workload_status == status
+                                         and grafana_unit.agent_status == 'idle'
+                                         and grafana_unit.workload_status_message == message),
+                                timeout=900)
+        if install_method == FAKE_INSTALL_METHOD:
+            # no further checks
+            continue
+
+        action = await grafana_unit.run_action('get-admin-password')
+        action = await action.wait()
+        passwd = action.results['password']
+        host = grafana_unit.public_address
+        req = requests.get('http://{host}:{port}'
+                           '/api/search?dashboardIds'.format(host=host, port=port),
+                           auth=('admin', passwd))
+        if req.status_code == 200:
+            dashboards = json.loads(req.text)
+            if len(dashboards) >= 3:
+                assert all(dash.get('type', '') == 'dash-db'
+                           and dash.get('uri', '').startswith('db/juju-')
+                           for dash in dashboards)
+                continue
+            assert len(dashboards) >= 3
+        assert req.status_code == 200
+
+    assert len(units) > 0
diff --git a/tests/unit/conftest.py b/tests/unit/conftest.py
index b7c019d..18273ef 100644
--- a/tests/unit/conftest.py
+++ b/tests/unit/conftest.py
@@ -1,8 +1,10 @@
-#!/usr/bin/python3
+import datetime
 import unittest.mock as mock
 
 import pytest
 
+import lib_grafana
+
 
 # If layer options are used, add this to grafana
 # and import layer in lib_grafana
@@ -44,6 +46,28 @@ def mock_hookenv_config(monkeypatch):
 
 
 @pytest.fixture
+def mock_unitdata_admin_password(monkeypatch):
+    class TestUnitdataAdminPwd(dict):
+        def __init__(self, d={}):
+            self._d = d
+
+        def __get__(self, name):
+            return self._d[name]
+
+        def get(self, name, altvalue=None):
+            return self._d.get(name, altvalue)
+
+        def set(self, name, value):
+            self._d[name] = value
+
+    def _unitdata_admin_password(arg={}):
+        obj = TestUnitdataAdminPwd(arg)
+        return obj
+
+    return _unitdata_admin_password
+
+
[e-mail]
 def mock_remote_unit(monkeypatch):
     monkeypatch.setattr('lib_grafana.helpers.hookenv.remote_unit', lambda: 'unit-mock/0')
 
@@ -55,7 +79,7 @@ def mock_charm_dir(monkeypatch):
 
 
 @pytest.fixture
-def grafana(tmpdir, mock_hookenv_config, mock_charm_dir, monkeypatch):
+def grafana_helper(tmpdir, mock_hookenv_config, mock_charm_dir, monkeypatch):
     from lib_grafana.helpers import GrafanaHelper
     helper = GrafanaHelper()
 
@@ -63,3 +87,29 @@ def grafana(tmpdir, mock_hookenv_config, mock_charm_dir, monkeypatch):
     monkeypatch.setattr('lib_grafana.helpers.GrafanaHelper', lambda: helper)
 
     return helper
+
+
[e-mail]
+def grafana_db(tmpdir, mock_hookenv_config, mock_charm_dir, monkeypatch):
+    class TestSqlite3(object):
+        def cursor(cls):
+            pass
+
+        def close(cls):
+            pass
+
+    class NewDate(datetime.datetime):
+        @classmethod
+        def today(cls):
+            return cls(1971, 1, 1)
+
+    from lib_grafana.db import DbManagement
+
+    monkeypatch.setattr('lib_grafana.db.sqlite3.connect', lambda filename, timeout: TestSqlite3())
+    monkeypatch.setattr('lib_grafana.db.hookenv.log', lambda message, level=None: None)
+    helper = DbManagement(lib_grafana.SNAP_COMMON)
+    # Any other functions that load helper will get this version
+    monkeypatch.setattr('lib_grafana.db.DbManagement', lambda: helper)
+    monkeypatch.setattr('lib_grafana.db.datetime.datetime', NewDate)
+
+    return helper
diff --git a/tests/unit/test_lib.py b/tests/unit/test_lib.py
index 8cba983..bb799ce 100644
--- a/tests/unit/test_lib.py
+++ b/tests/unit/test_lib.py
@@ -1,20 +1,36 @@
-#!/usr/bin/python3
-import unittest.mock as mock
-
 import pytest
 
 import lib_grafana
 import lib_grafana.helpers
 
 
[e-mail]('admin_password', 'password')
-def test_grafana(admin_password, grafana):
+def test_grafana_constants(grafana_helper):
     """See if the helper fixture works to load charm configs"""
-    grafana.charm_config['admin_password'] = admin_password
+    assert isinstance(grafana_helper.charm_config, dict)
+    assert grafana_helper.nagios_context == 'juju'
+    assert grafana_helper.snap_name == 'grafana'
+    assert grafana_helper.plugins_exist == ''
+
+
+def test_admin_password_config(grafana_helper):
+    admin_password = 'password'
+    grafana_helper.charm_config['admin_password'] = admin_password
+    assert grafana_helper.password == admin_password
+
+
+def test_admin_password_unitdata(monkeypatch, grafana_helper):
+    admin_password = 'password'
+    monkeypatch.setattr('lib_grafana.helpers.unitdata.kv',
+                        lambda: {'grafana.admin_password': admin_password})
+    assert grafana_helper.password == admin_password
+
 
-    assert isinstance(grafana.charm_config, dict)
-    assert grafana.password == admin_password
-    assert grafana.nagios_context == 'juju'
+def test_admin_password_pwgen(monkeypatch, mock_unitdata_admin_password, grafana_helper):
+    admin_password = 'password'
+    monkeypatch.setattr('lib_grafana.helpers.unitdata.kv',
+                        mock_unitdata_admin_password)
+    monkeypatch.setattr('lib_grafana.helpers.host.pwgen', lambda n: admin_password)
+    assert grafana_helper.password == admin_password
 
 
 @pytest.mark.parametrize(
@@ -25,12 +41,103 @@ def test_grafana(admin_password, grafana):
         ('wrong_method', lib_grafana.helpers.GrafanaBlockedError),
     ]
 )
[e-mail]('lib_grafana.helpers.unitdata.kv')
-def test_data_path(mock_unitdata, install_method, result, grafana):
-    mock_unitdata.return_value = {'install_method': install_method}
+def test_data_path(install_method, result, monkeypatch, grafana_helper):
+    monkeypatch.setattr('lib_grafana.helpers.unitdata.kv',
+                        lambda: {'install_method': install_method})
+    if install_method != 'wrong_method':
+        assert grafana_helper.data_path == result
+    else:
+        with pytest.raises(result) as excinfo:
+            grafana_helper.data_path
+        assert str(excinfo.value) == 'Unsupported install_method'
+
+
[e-mail](
+    'install_method,result',
+    [
+        ('apt', '/etc/grafana/grafana.ini'),
+        ('snap', '{}/conf/grafana.ini'.format(lib_grafana.SNAP_DATA)),
+        ('wrong_method', lib_grafana.helpers.GrafanaBlockedError),
+    ]
+)
+def test_grafana_ini(install_method, result, monkeypatch, grafana_helper):
+    monkeypatch.setattr('lib_grafana.helpers.unitdata.kv',
+                        lambda: {'install_method': install_method})
     if install_method != 'wrong_method':
-        assert grafana.data_path == result
+        assert grafana_helper.grafana_ini == result
     else:
         with pytest.raises(result) as excinfo:
-            grafana.data_path
+            grafana_helper.grafana_ini
         assert str(excinfo.value) == 'Unsupported install_method'
+
+
[e-mail](
+    'install_method,result',
+    [
+        ('apt', 'apt'),
+        ('snap', 'snap'),
+        ('wrong_method', lib_grafana.helpers.GrafanaBlockedError),
+    ]
+)
+def test_source(install_method, result, monkeypatch, grafana_helper):
+    monkeypatch.setattr('lib_grafana.helpers.unitdata.kv',
+                        lambda: {'install_method': install_method})
+    if install_method != 'wrong_method':
+        assert grafana_helper.source == result
+    else:
+        with pytest.raises(result) as excinfo:
+            grafana_helper.data_path
+        assert str(excinfo.value) == 'Unsupported install_method'
+
+
[e-mail](
+    'datasources,result',
+    [
+        (False, False),
+        ('type,name,access,url,password,user,database', False),
+        ('prometheus,name,proxy,url,password,user,database', True),
+        ('prometheus,name,proxy,missing_elements', False)
+    ]
+)
+def test_validate_datasources(datasources, result, grafana_helper):
+    grafana_helper.charm_config['datasources'] = datasources
+    assert grafana_helper.validate_datasources() == result
+
+
[e-mail](
+    'jujuconfig,stored,result',
+    [
+        ('apt', 'apt', 'apt'),
+        ('apt', 'snap', lib_grafana.helpers.GrafanaBlockedError),
+        ('snap', 'snap', 'snap'),
+        ('snap', 'apt', lib_grafana.helpers.GrafanaBlockedError),
+        (None, 'apt', 'apt'),
+        (None, 'snap', lib_grafana.helpers.GrafanaBlockedError),
+        ('wrong_method', None, lib_grafana.helpers.GrafanaBlockedError),
+        ('wrong_method', 'apt', lib_grafana.helpers.GrafanaBlockedError),
+    ]
+)
+def test_verify_install_method(jujuconfig, stored, result, mock_unitdata_admin_password,
+                               monkeypatch, grafana_helper):
+    grafana_helper.charm_config['install_method'] = jujuconfig
+    if jujuconfig is None:
+        grafana_helper.charm_config['install_method'] = 'apt'
+
+    if stored is None:
+        monkeypatch.setattr('lib_grafana.helpers.unitdata.kv',
+                            mock_unitdata_admin_password)
+    else:
+        monkeypatch.setattr('lib_grafana.helpers.unitdata.kv',
+                            lambda: mock_unitdata_admin_password({'install_method': stored}))
+
+    if isinstance(result, str):
+        assert grafana_helper.verify_install_method() == result
+    else:
+        if jujuconfig == 'wrong_method':
+            errormsg = 'Unsupported install_method'
+        else:
+            errormsg = ('install_method changes unsupported,'
+                        ' revert install_method config to previous value')
+        with pytest.raises(result) as excinfo:
+            grafana_helper.verify_install_method()
+        assert str(excinfo.value) == errormsg
diff --git a/tests/unit/test_libdb.py b/tests/unit/test_libdb.py
new file mode 100644
index 0000000..006c8ed
--- /dev/null
+++ b/tests/unit/test_libdb.py
@@ -0,0 +1,66 @@
+import pytest
+
+
[e-mail](
+    'ds,url,exists',
+    [
+        ('prometheus', 'http://10.10.20.20:9090', True),
+        ('prometheus', 'http://wrong.url', False),
+        ('wrong_ds', 'http://10.10.20.20:9090', False),
+        ('wrong_ds', 'http://wrong.url', False),
+    ]
+)
+def test_check_datasource(ds, url, exists, grafana_db):
+    datasource = {
+        'service_name': 'prometheus2',
+        'type': 'prometheus',
+        'url': 'http://10.10.20.20:9090',
+        'description': 'Juju generated source',
+    }
+    grafana_db.select_query = lambda x: [
+        [
+            'item0', ds, 'item2', url, 'item4',
+        ],
+    ]
+    grafana_db.update_db = lambda x, y: None
+    if exists:
+        grafana_db.generate_query = lambda x, y, z: ['stmt', ['values']]
+    else:
+        grafana_db.generate_query = lambda x, y: ['stmt', ['values']]
+
+    grafana_db.check_datasource(datasource)
+    assert True
+
+
+def test_generate_query(monkeypatch, grafana_db):
+    date_str = '1971-01-01 00:00:00'
+    datasource = {
+        'service_name': 'prometheus2',
+        'type': 'prometheus',
+        'url': 'http://10.10.20.20:9090',
+        'description': 'Juju generated source',
+    }
+    expected_stmt = ('INSERT INTO DATA_SOURCE (org_id'
+                     ', version'
+                     ', type'
+                     ', name'
+                     ', access'
+                     ', url'
+                     ', is_default'
+                     ', created'
+                     ', updated'
+                     ', basic_auth)'
+                     ' VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)')
+    expected_values = [
+        1,
+        0,
+        'prometheus',
+        'prometheus2 - Juju generated source',
+        'proxy',
+        'http://10.10.20.20:9090',
+        0,
+        date_str,
+        date_str,
+        0,
+    ]
+    assert grafana_db.generate_query(datasource, 0) == [expected_stmt, expected_values]
diff --git a/tox.ini b/tox.ini
index 3a54b37..42cf001 100644
--- a/tox.ini
+++ b/tox.ini
@@ -41,5 +41,6 @@ exclude =
     .git,
     __pycache__,
     .tox,
+    layers,
 max-line-length = 120
 max-complexity = 10