Merge ~karimsye/charm-grafana:k8s-model-filter-template into ~prometheus-charmers/charm-grafana:master

[Syed Mohammad Adnan Karim]

Add variables to filter k8s clusters by model

Added juju_model variable to Telegraf.json.j2 dashboard template for
filtering k8s clusters by model name. Also renamed "non-juju hosts"
variable to host, and set its value to hostnames that are part of the
selected juju_model variable. Also added {% raw %} and {% endraw %} to
escape jinja's '{{ }}' syntax. This enables the legendFormat variable
to properly display varables such as host.

Add variables to filter k8s clusters by model

Added juju_model variable to Telegraf.json.j2 dashboard template for
filtering k8s clusters by model name. Also renamed "non-juju hosts"
variable to host, and set its value to hostnames that are part of the
selected juju_model variable. Also added {% raw %} and {% endraw %} to
escape jinja's '{{ }}' syntax. This enables the legendFormat variable
to properly display varables such as host.

Example Prometheus target metrics can be found here:
- https://pastebin.canonical.com/p/jRQRf36psn/
- https://pastebin.canonical.com/p/FSMmkBg9bQ/
- https://pastebin.canonical.com/p/CGK6Nz2QJj/

The 'juju_model' variable was derived by using the following in Grafana:
- Query: system_load1{}
- Regex: .*host="([^:]+).*$

The 'non-juju hosts' variable was changed to host and derived using the following in Grafana:
- Query: system_load1{host=~".*^($juju_model).*"}
- Regex: .*host="([^"]+).*

Both variables support multi-value and auto include all selection options.

I also added {% raw %} and {% endraw %} to escape jinja's '{{ }}' syntax.
For example, Jinja will evaluate {{host}} as "" and when the dashboard is imported, it will not render {{host}} in the legend section of the graphs. The escaping enables the "legendFormat" variable to properly display variables such as {{host}} and {{interface}}.

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[Chris Sanders (chris.sanders)]

This is adding an entirely new dashboard which, as much as I can tell from the .json, is a duplicate of the current Telegraf with some modifications.
 * Is there a reason this isn't a modification of the Telegraf dashboard rather than a new template?
 * Won't this result in *both* dashboards being installed (if they have the same metrics) which we wouldn't want.
 * Assuming you can modify the Telegraf dashboard rather than put a duplicate in the charm, please make minimal changes so the diff is as readable as a json diff can be. It looks like you've run this dashboard through a json formatter that is different from the Telegraf json that Grafana produced. Small changes in white space, with no functional change to the json, only serve to make the diffs harder to read.

Also looking through the template, I notice you used {% raw %} in several places. Why has that been added to the template when it appears to wrap sections that were already present and didn't have it before?

review: Needs Information

[Syed Mohammad Adnan Karim (karimsye)]

> This is adding an entirely new dashboard which, as much as I can tell from the
> .json, is a duplicate of the current Telegraf with some modifications.
> * Is there a reason this isn't a modification of the Telegraf dashboard
> rather than a new template?
I created the dashboard based on https://grafana.admin.canonical.com/d/000000044/telegraf-host?orgId=1. Acknowledged. I will update the existing Telegraf.json.j2 dashboard to include the changes to filter by model.

> * Won't this result in *both* dashboards being installed (if they have the
> same metrics) which we wouldn't want.
Correct, I will update the existing Telegraf.json.j2 dashboard to include these changes.
I will also remove TelegrafKubernetes.json.j2 dashboard.

> * Assuming you can modify the Telegraf dashboard rather than put a duplicate
> in the charm, please make minimal changes so the diff is as readable as a json
> diff can be. It looks like you've run this dashboard through a json formatter
> that is different from the Telegraf json that Grafana produced. Small changes
> in white space, with no functional change to the json, only serve to make the
> diffs harder to read.
>
Acknowledged.

> Also looking through the template, I notice you used {% raw %} in several
> places. Why has that been added to the template when it appears to wrap
> sections that were already present and didn't have it before?
I used {% raw %} to escape {{ variable_name }} in the legendFormat variable.
Jinja will evaluate it as "" and when the dashboard is imported, it will not render
variable_name (typically {{host}}) in the legend section of the graphs.

de20009... by Syed Mohammad Adnan Karim on 2019-04-25

Add variables to filter k8s clusters by model

Added juju_model variable to Telegraf.json.j2 dashboard template for
filtering k8s clusters by model name. Also renamed "non-juju hosts"
variable to host, and set its value to hostnames that are part of the
selected juju_model variable. Also added {% raw %} and {% endraw %} to
escape jinja's '{{ }}' syntax. This enables the legendFormat variable
to properly display varables such as host.

[Stuart Bishop (stub)]

All good from ~canonical-is-reviews, but still needs a signoff from ~prometheus-charmers by someone who can confirm this is what we want and won't make our existing deployments fail.

review: Approve (is-reviews)

[Chris Sanders (chris.sanders)]

Thanks, the diff is significantly easier to read. I believe this address all of my previous concerns.

The only remaining comment is, with out testing in the charm someone should verify that this branch will still install expected dashboards on a standard non-k8s install. It doesn't look like any metrics actually changed so I think it's safe to say it will be ok.

Syed, can you confirm that there was no change to the metrics, only the new raw values?
Do you by chance already try this on an openstack w/o K8s to see that it still works as previously expected?

[Syed Mohammad Adnan Karim (karimsye)]

> Thanks, the diff is significantly easier to read. I believe this address all
> of my previous concerns.

No problem.

> The only remaining comment is, with out testing in the charm someone should
> verify that this branch will still install expected dashboards on a standard
> non-k8s install. It doesn't look like any metrics actually changed so I think
> it's safe to say it will be ok.
>
> Syed, can you confirm that there was no change to the metrics, only the new
> raw values?

Yes I can confirm that is the case.

> Do you by chance already try this on an openstack w/o K8s to see that it still
> works as previously expected?

I did not test it on openstack w/o k8s yet but I can give that a go and report mt findings.

[Syed Mohammad Adnan Karim (karimsye)]

I can confirm that on openstack deployments w/o k8s, the graphs work as expected, the only differences are in the Telegraf Host Metrics Dashboard. The diffrence is that now filtering by juju models is possible, and that unit hostnames show up properly on the legend of the graphs.

I saved screenshots comparing the changes between my update and the charmstore's version 28:
https://drive.google.com/drive/folders/19gr2loO2k3LEk3jfp8Pae3l5yAhBVtQZ?usp=sharing
https://drive.google.com/drive/folders/14KBXkFpTIDqdDt1u-8Nql7ugAHU4Vn9t?usp=sharing

[Xav Paice (xavpaice)]

Thanks for this work.

Please note that the charm-grafana code has moved to https://code.launchpad.net/~llama-charmers/charm-grafana/+git/charm-grafana/+ref/master and this change should be rebased against that, then re-submitted.

review: Needs Fixing

[Alvaro Uria (aluria)]

Moved the MP to WIP status because charm-grafana is now under ~llama-charmers namespace and charm-telegraf is the one maintaining the dashboard that will be pushed to grafana via the grafana-dashboards interface.

review: Needs Fixing

Unmerged commits

de20009... by Syed Mohammad Adnan Karim on 2019-04-25

Add variables to filter k8s clusters by model

Added juju_model variable to Telegraf.json.j2 dashboard template for
filtering k8s clusters by model name. Also renamed "non-juju hosts"
variable to host, and set its value to hostnames that are part of the
selected juju_model variable. Also added {% raw %} and {% endraw %} to
escape jinja's '{{ }}' syntax. This enables the legendFormat variable
to properly display varables such as host.

2dad259... by Syed Mohammad Adnan Karim on 2019-04-25

Add dashboard for filtering k8s clusters by model

Exisiting Telegraf.json.j2 dashboard only filters by unit name, making
it difficult to monitor multiple k8s clusters. Added dashboard called
TelegrafKubernetes.json.j2 to templates/dashboards/prometheus/ for
filtering k8s clusters by model and unit names. The dashboard uses
juju_model and juju_unit variables to achieve this. These variables
use regex to parse the hostname coming from a Prometheus data source.
The variables juju_model and juju_unit have been added to the
ignore_metrics variable because they are not real Prometheus metrics.

diff --git a/reactive/grafana.py b/reactive/grafana.py
index f98e11c..2ceb441 100644
--- a/reactive/grafana.py
+++ b/reactive/grafana.py
@@ -522,7 +522,8 @@ def generate_prometheus_dashboards(gf_adminpasswd, ds):
     ignore_metrics = ['by_cloud_status', 'group_left', 'count_values',
                       'nova_service_status', 'hypervisor_hostname', 'external_network',
                       'ip_type', 'instance_state', 'network_name', 'subnet_name',
-                      'ip_status', 'neutron_net', config['external_network']
+                      'ip_status', 'neutron_net', config['external_network'],
+                      'juju_model'
                       ]
     prom_metrics.extend(ignore_metrics)
     for filename in os.listdir(templates_dir):
diff --git a/templates/dashboards/prometheus/Telegraf.json.j2 b/templates/dashboards/prometheus/Telegraf.json.j2
index a58f03e..8e7f071 100644
--- a/templates/dashboards/prometheus/Telegraf.json.j2
+++ b/templates/dashboards/prometheus/Telegraf.json.j2
@@ -202,7 +202,9 @@
               "expr": "system_load1{host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "{{host}} 1min",
+              {% endraw %}
               "refId": "A",
               "step": 2
             },
@@ -210,7 +212,9 @@
               "expr": "system_load5{host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "{{host}} 5min",
+              {% endraw %}
               "refId": "B",
               "step": 2
             },
@@ -218,7 +222,9 @@
               "expr": "system_load15{host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "{{host}} 15min",
+              {% endraw %}
               "refId": "C",
               "step": 2
             }
@@ -296,7 +302,9 @@
               "expr": "mem_used{host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Used {{host}}",
+              {% endraw %}
               "refId": "A",
               "step": 2
             },
@@ -304,7 +312,9 @@
               "expr": "mem_buffered{host=~\"$host\"} + mem_cached{host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Buffers+cache {{host}}",
+              {% endraw %}
               "refId": "B",
               "step": 2
             },
@@ -312,7 +322,9 @@
               "expr": "mem_free{host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Free {{host}}",
+              {% endraw %}
               "refId": "C",
               "step": 2
             }
@@ -394,7 +406,9 @@
               "expr": "cpu_usage_user{cpu=\"cpu-total\",host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "user {{host}}",
+              {% endraw %}
               "refId": "A",
               "step": 2
             },
@@ -402,7 +416,9 @@
               "expr": "cpu_usage_iowait{cpu=\"cpu-total\", host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "iowait {{host}}",
+              {% endraw %}
               "refId": "C",
               "step": 2
             },
@@ -410,7 +426,9 @@
               "expr": "cpu_usage_nice{cpu=\"cpu-total\", host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "nice {{host}}",
+              {% endraw %}
               "refId": "D",
               "step": 2
             },
@@ -418,7 +436,9 @@
               "expr": "cpu_usage_softirq{cpu=\"cpu-total\",host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "softirq {{host}}",
+              {% endraw %}
               "refId": "E",
               "step": 2
             },
@@ -426,7 +446,9 @@
               "expr": "cpu_usage_irq{cpu=\"cpu-total\", host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "irq {{host}}",
+              {% endraw %}
               "refId": "F",
               "step": 2
             },
@@ -434,7 +456,9 @@
               "expr": "cpu_usage_system{cpu=\"cpu-total\", host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "system {{host}}",
+              {% endraw %}
               "refId": "G",
               "step": 2
             },
@@ -442,7 +466,9 @@
               "expr": "cpu_usage_idle{cpu=\"cpu-total\", host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "idle {{host}}",
+              {% endraw %}
               "refId": "B",
               "step": 2
             }
@@ -522,7 +548,9 @@
               "expr": "system_n_users{host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Users {{host}}",
+              {% endraw %}
               "refId": "A",
               "step": 2
             }
@@ -600,7 +628,9 @@
               "expr": "rate(swap_in{host=~\"$host\"}[2m])",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Swap in {{host}}",
+              {% endraw %}
               "refId": "A",
               "step": 2
             },
@@ -608,7 +638,9 @@
               "expr": "rate(swap_out{host=~\"$host\"}[2m])",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Swap out {{host}}",
+              {% endraw %}
               "refId": "B",
               "step": 2
             }
@@ -713,7 +745,9 @@
               "expr": "netstat_tcp_established{host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Established {{host}}",
+              {% endraw %}
               "refId": "A",
               "step": 4
             },
@@ -721,7 +755,9 @@
               "expr": "netstat_tcp_syn_sent{host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Syn sent {{host}}",
+              {% endraw %}
               "refId": "B",
               "step": 4
             },
@@ -729,7 +765,9 @@
               "expr": "netstat_tcp_syn_recv{host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Syn recv {{host}}",
+              {% endraw %}
               "refId": "C",
               "step": 4
             },
@@ -737,7 +775,9 @@
               "expr": "netstat_tcp_fin_wait1{host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Fin wait1 {{host}}",
+              {% endraw %}
               "refId": "D",
               "step": 4
             },
@@ -745,7 +785,9 @@
               "expr": "netstat_tcp_fin_wait2{host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Fin wait2 {{host}}",
+              {% endraw %}
               "refId": "E",
               "step": 4
             },
@@ -753,7 +795,9 @@
               "expr": "netstat_tcp_time_wait{host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Time wait {{host}}",
+              {% endraw %}
               "refId": "F",
               "step": 4
             },
@@ -761,7 +805,9 @@
               "expr": "netstat_tcp_close{host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Close {{host}}",
+              {% endraw %}
               "refId": "G",
               "step": 4
             },
@@ -769,7 +815,9 @@
               "expr": "netstat_tcp_close_wait{host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Close wait {{host}}",
+              {% endraw %}
               "refId": "H",
               "step": 4
             },
@@ -777,7 +825,9 @@
               "expr": "netstat_tcp_last_ack{host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Last ack {{host}}",
+              {% endraw %}
               "refId": "I",
               "step": 4
             },
@@ -785,7 +835,9 @@
               "expr": "netstat_tcp_listen{host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Listen {{host}}",
+              {% endraw %}
               "refId": "J",
               "step": 4
             },
@@ -793,7 +845,9 @@
               "expr": "netstat_tcp_closing{host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Closing {{host}}",
+              {% endraw %}
               "refId": "K",
               "step": 4
             },
@@ -801,7 +855,9 @@
               "expr": "netstat_tcp_none{host=~\"$host\"}",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "None {{host}}",
+              {% endraw %}
               "refId": "L",
               "step": 4
             }
@@ -897,7 +953,9 @@
               "instant": false,
               "interval": "1m",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "In {{host}} {{interface}}",
+              {% endraw %}
               "metric": "net_by",
               "refId": "A",
               "step": 4
@@ -908,7 +966,9 @@
               "hide": false,
               "interval": "1m",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Out {{host}} {{interface}}",
+              {% endraw %}
               "refId": "B",
               "step": 4
             }
@@ -991,7 +1051,9 @@
               "expr": "rate(net_packets_recv{host=~\"$host\",interface=~\"^bond.*|^eth.*|^en.*\"}[5m])",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "In {{interface}} {{host}}",
+              {% endraw %}
               "refId": "A",
               "step": 4
             },
@@ -999,7 +1061,9 @@
               "expr": "rate(net_packets_sent{host=~\"$host\",interface=~\"^bond.*|^eth.*|^en.*\"}[5m])",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Out {{interface}} {{host}}",
+              {% endraw %}
               "refId": "B",
               "step": 4
             },
@@ -1007,7 +1071,9 @@
               "expr": "rate(net_drop_in{host=~\"$host\",interface=~\"^bond.*|^eth.*|^en.*\"}[5m])",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Drop In {{interface}} {{host}}",
+              {% endraw %}
               "refId": "C",
               "step": 4
             },
@@ -1015,7 +1081,9 @@
               "expr": "rate(net_drop_out{host=~\"$host\",interface=~\"^bond.*|^eth.*|^en.*\"}[5m])",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Drop Out {{interface}} {{host}}",
+              {% endraw %}
               "refId": "D",
               "step": 4
             }
@@ -1096,7 +1164,9 @@
               "format": "time_series",
               "interval": "",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Err in {{interface}} {{host}}",
+              {% endraw %}
               "metric": "net_err_in",
               "refId": "A",
               "step": 4
@@ -1105,7 +1175,9 @@
               "expr": "rate(net_err_out{host=~\"$host\",interface=~\"^bond.*|^eth.*|^en.*|^br.*\"}[5m])",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Err out {{interface}} {{host}}",
+              {% endraw %}
               "metric": "net_d",
               "refId": "B",
               "step": 4
@@ -1115,7 +1187,9 @@
               "format": "time_series",
               "interval": "",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Drop in {{interface}} {{host}}",
+              {% endraw %}
               "refId": "C",
               "step": 4
             },
@@ -1123,7 +1197,9 @@
               "expr": "rate(net_drop_in{host=~\"$host\",interface=~\"^bond.*|^eth.*|^en.*|^br.*\"}[5m])",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "Drop out {{interface}} {{host}}",
+              {% endraw %}
               "refId": "D",
               "step": 4
             }
@@ -1206,7 +1282,9 @@
               "instant": false,
               "interval": "1m",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "{{host}} - {{bond}} - {{interface}}",
+              {% endraw %}
               "metric": "bond_slave_failures",
               "refId": "A",
               "step": 4
@@ -1286,7 +1364,9 @@
               "format": "time_series",
               "interval": "",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "{{host}} - {{interface}}",
+              {% endraw %}
               "metric": "bond_slave_status",
               "refId": "A",
               "step": 4
@@ -1365,7 +1445,9 @@
               "format": "time_series",
               "interval": "",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "{{host}} - {{bond}}",
+              {% endraw %}
               "metric": "",
               "refId": "A",
               "step": 4
@@ -1462,7 +1544,9 @@
               "expr": "irate(diskio_reads{name=~\"bcache.*|[a-z]+\",host=~\"$host\"}[2m])",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "{{host}} {{name}}",
+              {% endraw %}
               "refId": "B",
               "step": 2
             }
@@ -1543,7 +1627,9 @@
               "expr": "irate(diskio_writes{name=~\"bcache.*|[a-z]+\",host=~\"$host\"}[2m])",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "{{host}} {{name}}",
+              {% endraw %}
               "refId": "B",
               "step": 2
             }
@@ -1624,7 +1710,9 @@
               "expr": "irate(diskio_read_bytes{name=~\"bcache.*|[a-z]+\",host=~\"$host\"}[2m])",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "{{host}} {{name}}",
+              {% endraw %}
               "refId": "B",
               "step": 2
             }
@@ -1705,7 +1793,9 @@
               "expr": "irate(diskio_write_bytes{name=~\"bcache.*|[a-z]+\",host=~\"$host\"}[2m])",
               "format": "time_series",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "{{host}} {{name}}",
+              {% endraw %}
               "refId": "B",
               "step": 2
             }
@@ -1797,7 +1887,9 @@
               "format": "time_series",
               "interval": "",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "{{host}} {{bcache_dev}}/{{backing_dev}}",
+              {% endraw %}
               "metric": "bcache_dirty_data",
               "refId": "A",
               "step": 4
@@ -1877,7 +1969,9 @@
               "format": "time_series",
               "interval": "",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "{{host}} {{bcache_dev}}/{{backing_dev}}",
+              {% endraw %}
               "metric": "bcache_cache_hit_ratio",
               "refId": "A",
               "step": 4
@@ -1956,7 +2050,9 @@
               "format": "time_series",
               "interval": "",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "{{host}} {{bcache_dev}}/{{backing_dev}}",
+              {% endraw %}
               "metric": "bcache_bypassed",
               "refId": "A",
               "step": 4
@@ -2048,7 +2144,9 @@
               "format": "time_series",
               "interval": "",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "{{host}}",
+              {% endraw %}
               "metric": "exec",
               "refId": "A",
               "step": 2
@@ -2127,7 +2225,9 @@
               "format": "time_series",
               "interval": "",
               "intervalFactor": 2,
+              {% raw %}
               "legendFormat": "{{host}} - Count",
+              {% endraw %}
               "metric": "conntrack_ip_conntrack_count",
               "refId": "A",
               "step": 2
@@ -2184,18 +2284,38 @@
   "templating": {
     "list": [
       {
-        "allValue": ".*",
+        "allValue": "",
         "current": {},
         "datasource": "{{ datasource }}",
         "hide": 0,
-        "includeAll": false,
-        "label": "non-juju hosts",
+        "includeAll": true,
+        "label": null,
+        "multi": true,
+        "name": "juju_model",
+        "options": [],
+        "query": "system_load1{}",
+        "refresh": 1,
+        "regex": ".*host=\"([^:]+).*$",
+        "sort": 1,
+        "tagValuesQuery": "",
+        "tags": [],
+        "tagsQuery": "",
+        "type": "query",
+        "useTags": false
+      },
+      {
+        "allValue": "",
+        "current": {},
+        "datasource": "{{ datasource }}",
+        "hide": 0,
+        "includeAll": true,
+        "label": null,
         "multi": true,
         "name": "host",
         "options": [],
-        "query": "system_load1",
+        "query": "system_load1{host=~\".*^($juju_model).*\"}",
         "refresh": 1,
-        "regex": ".*host=\"([^\"]+)\".*$",
+        "regex": ".*host=\"([^\"]+).*",
         "sort": 1,
         "tagValuesQuery": "",
         "tags": [],