Juju Lint

Merge ~jfguedez/juju-lint:bug/1896551 into juju-lint:master

  1. Git
  2. lp:~jfguedez/juju-lint
  3. bug/1896551
  4. Merge into master
Proposed by Jose Guedez
Status: Merged
Approved by: James Troup
Approved revision: 31d208a66a156e1f279f9d8026840b9d2de1a097
Merged at revision: 737b1c02c3a8cbfe2100d9f7d8f275e53c34d9e5
Proposed branch: ~jfguedez/juju-lint:bug/1896551
Merge into: juju-lint:master
Diff against target: 611 lines (+427/-7)
22 files modified
README.md (+3/-2)
contrib/canonical-rules.yaml (+1/-0)
contrib/includes/aggregator-openstack.yaml (+20/-0)
contrib/includes/base.yaml (+40/-0)
contrib/includes/database/mysql.yaml (+10/-0)
contrib/includes/database/percona-cluster.yaml (+9/-0)
contrib/includes/kubernetes.yaml (+24/-0)
contrib/includes/networking/ovn.yaml (+9/-0)
contrib/includes/networking/ovs.yaml (+4/-0)
contrib/includes/openstack.yaml (+105/-0)
contrib/includes/operations.yaml (+49/-0)
contrib/includes/operations/bionic.yaml (+2/-0)
contrib/includes/operations/focal.yaml (+1/-0)
contrib/includes/saas.yaml (+7/-0)
contrib/kubernetes.yaml (+22/-0)
contrib/openstack-bionic-ovn.yaml (+13/-0)
contrib/openstack-bionic-ovs.yaml (+13/-0)
contrib/openstack-focal-ovn.yaml (+13/-0)
contrib/openstack-focal-ovs.yaml (+13/-0)
jujulint/lint.py (+38/-5)
jujulint/util.py (+4/-0)
tests/test_jujulint.py (+27/-0)
Reviewer Review Type Date Requested Status
Juju Lint maintainers Pending
Review via email: mp+408765@code.launchpad.net

Commit message

Add support for includes, as well as several reference rules files

To post a comment you must log in.
Revision history for this message
🤖 Canonical IS Merge Bot (canonical-is-mergebot) wrote :

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

Revision history for this message
🤖 Canonical IS Merge Bot (canonical-is-mergebot) wrote :

Merge proposal is approved, but source revision has changed, setting status to needs review.

Revision history for this message
🤖 Canonical IS Merge Bot (canonical-is-mergebot) wrote :

Change successfully merged at revision 737b1c02c3a8cbfe2100d9f7d8f275e53c34d9e5

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/README.md b/README.md
index 00d737d..d7ef4c7 100644
--- a/README.md
+++ b/README.md
@@ -2,7 +2,7 @@
22
3== Introduction ==3== Introduction ==
44
5This is intended to be run against a yaml dump of Juju status, a YAML dump of 5This is intended to be run against a yaml dump of Juju status, a YAML dump of
6a juju bundle or a remote cloud or clouds via SSH.6a juju bundle or a remote cloud or clouds via SSH.
77
8To generate a status if you just want to audit placement:8To generate a status if you just want to audit placement:
@@ -49,7 +49,8 @@ Supported top-level options for your rules file:
49 3. `operations [mandatory|optional|subordinate]`49 3. `operations [mandatory|optional|subordinate]`
50 4. `openstack [mandatory|optional|subordinate]`50 4. `openstack [mandatory|optional|subordinate]`
51 5. `config` - application configuration auditing51 5. `config` - application configuration auditing
52 5. `[openstack|kubernetes] config` - config auditing for specific cloud types.52 6. `[openstack|kubernetes] config` - config auditing for specific cloud types.
53 7. `!include <relative path>` - Extension to yaml to include files.
5354
54== License ==55== License ==
5556
diff --git a/contrib/canonical-rules.yaml b/contrib/canonical-rules.yaml
index 6c1a101..b655f34 100644
--- a/contrib/canonical-rules.yaml
+++ b/contrib/canonical-rules.yaml
@@ -1,4 +1,5 @@
1---1---
2# legacy rules file, there are also more specific configurations available
2kubernetes config:3kubernetes config:
3 kubernetes-master:4 kubernetes-master:
4 authorization-mode:5 authorization-mode:
diff --git a/contrib/includes/aggregator-openstack.yaml b/contrib/includes/aggregator-openstack.yaml
5new file mode 1006446new file mode 100644
index 0000000..6982a91
--- /dev/null
+++ b/contrib/includes/aggregator-openstack.yaml
@@ -0,0 +1,20 @@
1operations charms: &operations-charms
2 - *operations-mandatory-charms
3 - *operations-mandatory-deps
4 - *operations-mandatory-subs
5 - *operations-optional-charms
6 - *operations-optional-subs
7 - *operations-openstack-mandatory-charms
8
9openstack charms: &openstack-charms
10 - *openstack-mandatory-charms
11 - *openstack-mandatory-deps
12 - *openstack-mandatory-subs
13 - *openstack-optional-charms
14 - *cisco-aci-charms
15 - *trilio-charms
16
17known charms:
18 - ubuntu
19 - *openstack-charms
20 - *operations-charms
diff --git a/contrib/includes/base.yaml b/contrib/includes/base.yaml
0new file mode 10064421new file mode 100644
index 0000000..1815ee2
--- /dev/null
+++ b/contrib/includes/base.yaml
@@ -0,0 +1,40 @@
1config:
2 hacluster:
3 cluster_count:
4 gte: 3
5 ntp:
6 auto_peers:
7 eq: false
8 nrpe:
9 lacp_bonds:
10 neq: ""
11 netlinks:
12 neq: ""
13 landscape-client:
14 disable-unattended-upgrades:
15 eq: true
16
17subordinates:
18 telegraf:
19 where: all except prometheus
20 # and prometheus-ceph-exporter and prometheus-openstack-exporter
21 landscape-client:
22 where: all except landscape-server
23 filebeat:
24 where: all except graylog
25 canonical-livepatch:
26 where: host only
27 nrpe:
28 where: container aware
29 host-suffixes: [host, physical, guest]
30 container-suffixes: [lxd, container]
31 exceptions: [nagios]
32 ntp:
33 # You don't want NTP in a container dueling with ntp in the host
34 where: host only
35 thruk-agent:
36 where: on nagios
37 hw-health:
38 where: host only
39 logrotated:
40 where: all
diff --git a/contrib/includes/database/mysql.yaml b/contrib/includes/database/mysql.yaml
0new file mode 10064441new file mode 100644
index 0000000..ef827e3
--- /dev/null
+++ b/contrib/includes/database/mysql.yaml
@@ -0,0 +1,10 @@
1openstack database config: &openstack-config-database
2 mysql-innodb-cluster:
3 innodb-buffer-pool-size:
4 gte: 6G
5 max-connections:
6 gte: 2000
7
8openstack mandatory deps database: &openstack-mandatory-deps-database
9 - mysql-innodb-cluster
10 - mysql-router
diff --git a/contrib/includes/database/percona-cluster.yaml b/contrib/includes/database/percona-cluster.yaml
0new file mode 10064411new file mode 100644
index 0000000..7102f52
--- /dev/null
+++ b/contrib/includes/database/percona-cluster.yaml
@@ -0,0 +1,9 @@
1openstack database config: &openstack-config-database
2 percona-cluster:
3 innodb-buffer-pool-size:
4 gte: 6G
5 max-connections:
6 gte: 2000
7
8openstack mandatory deps database: &openstack-mandatory-deps-database
9 - percona-cluster
diff --git a/contrib/includes/kubernetes.yaml b/contrib/includes/kubernetes.yaml
0new file mode 10064410new file mode 100644
index 0000000..02d5fa0
--- /dev/null
+++ b/contrib/includes/kubernetes.yaml
@@ -0,0 +1,24 @@
1kubernetes config:
2 kubernetes-master:
3 authorization-mode:
4 eq: "RBAC,Node"
5 canal:
6 cidr:
7 isset: false
8
9kubernetes mandatory: &kubernetes-mandatory-charms
10 - containerd
11 - kubeapi-load-balancer
12 - kubernetes-master
13 - kubernetes-worker
14
15kubernetes optional charms: &kubernetes-optional-charms
16 - calico
17 - canal
18 - coredns
19 - easyrsa
20 - etcd
21 - flannel
22 - kubernetes-dashboard
23 - openstack-integrator
24 - vsphere-integrator
diff --git a/contrib/includes/networking/ovn.yaml b/contrib/includes/networking/ovn.yaml
0new file mode 10064425new file mode 100644
index 0000000..b4f4140
--- /dev/null
+++ b/contrib/includes/networking/ovn.yaml
@@ -0,0 +1,9 @@
1openstack config networking: &openstack-config-networking
2 ovn-central:
3 ovsdb-server-election-timer:
4 gte: 4
5
6openstack mandatory charms networking: &openstack-mandatory-charms-networking
7 - ovn-central
8 - ovn-chassis
9 - neutron-api-plugin-ovn
diff --git a/contrib/includes/networking/ovs.yaml b/contrib/includes/networking/ovs.yaml
0new file mode 10064410new file mode 100644
index 0000000..493195e
--- /dev/null
+++ b/contrib/includes/networking/ovs.yaml
@@ -0,0 +1,4 @@
1openstack config networking: &openstack-config-networking {}
2
3openstack mandatory charms networking: &openstack-mandatory-charms-networking
4 - neutron-openvswitch
diff --git a/contrib/includes/openstack.yaml b/contrib/includes/openstack.yaml
0new file mode 1006445new file mode 100644
index 0000000..5235aea
--- /dev/null
+++ b/contrib/includes/openstack.yaml
@@ -0,0 +1,105 @@
1openstack config base: &openstack-config-base
2 ceph-radosgw:
3 ceph-osd-replication-count:
4 eq: 3
5 cinder-ceph:
6 ceph-osd-replication-count:
7 eq: 3
8 glance:
9 ceph-osd-replication-count:
10 eq: 3
11 neutron-api:
12 path-mtu:
13 eq: 9000
14 global-physnet-mtu:
15 eq: 9000
16 nova-compute:
17 cpu-model:
18 neq: ""
19 ceph-osd-replication-count:
20 eq: 3
21 rabbitmq-server:
22 cluster-partition-handling:
23 eq: "pause_minority"
24 keystone:
25 token-expiration:
26 gte: 86400
27 sysconfig:
28 governor:
29 eq: "performance"
30
31openstack config:
32 << : [ *openstack-config-base, *openstack-config-networking, *openstack-config-database ]
33
34openstack mandatory: &openstack-mandatory-charms
35 - ceilometer
36 - ceilometer-agent
37 - ceph-mon
38 - ceph-osd
39 - cinder
40 - cinder-ceph
41 - glance
42 - heat
43 - keystone
44 - neutron-api
45 - nova-cloud-controller
46 - nova-compute
47 - openstack-dashboard
48 - *openstack-mandatory-charms-networking
49
50openstack mandatory deps base: &openstack-mandatory-deps-base
51 - haproxy
52 - memcached
53 - rabbitmq-server
54
55openstack mandatory deps: &openstack-mandatory-deps
56 - *openstack-mandatory-deps-base
57 - *openstack-mandatory-deps-database
58
59openstack mandatory subordinates: &openstack-mandatory-subs
60 - hacluster
61
62openstack optional charms: &openstack-optional-charms
63 - aodh
64 - barbican
65 - barbican-vault
66 - ceph-fs
67 - ceph-radosgw
68 - cinder-backup
69 - designate
70 - designate-bind
71 - easyrsa
72 - etcd
73 - glance-simplestreams-sync
74 - glance-sync-slave
75 - gnocchi
76 - ironic-api
77 - ironic-conductor
78 - keystone-ldap
79 - manila
80 - manila-dashboard
81 - manila-ganesha
82 - masakari
83 - masakari-monitors
84 - mongodb # Optional since Gnocchi
85 - neutron-gateway
86 - neutron-api-plugin-ironic
87 - octavia
88 - octavia-dashboard
89 - octavia-diskimage-retrofit
90 - pacemaker-remote
91 - placement
92 - swift-proxy
93 - swift-storage
94 - vault
95 - cinder-lvm
96
97cisco-aci-charms: &cisco-aci-charms
98 - neutron-api-plugin-aci
99 - openstack-dashboard-plugin-gbp
100
101trilio-charms: &trilio-charms
102 - trilio-dm-api
103 - trilio-horizon-plugin
104 - trilio-data-mover
105 - trilio-wlm
diff --git a/contrib/includes/operations.yaml b/contrib/includes/operations.yaml
0new file mode 100644106new file mode 100644
index 0000000..40b628b
--- /dev/null
+++ b/contrib/includes/operations.yaml
@@ -0,0 +1,49 @@
1operations mandatory: &operations-mandatory-charms
2 - elasticsearch
3 - grafana
4 - graylog
5 - landscape-server
6 - nagios
7 - prometheus2
8
9operations optional: &operations-optional-charms
10 - infra-node
11 - cloudstats
12 - juju-lint
13
14operations openstack mandatory base: &operations-openstack-mandatory-base
15 - openstack-service-checks
16 - prometheus-libvirt-exporter
17 - prometheus-openstack-exporter
18 - prometheus-grok-exporter
19
20operations openstack mandatory: &operations-openstack-mandatory-charms
21 - *operations-openstack-mandatory-base
22 - *operations-openstack-mandatory-series
23
24operations mandatory dependencies: &operations-mandatory-deps
25 - postgresql
26
27operations subordinates: &operations-mandatory-subs
28 - canonical-livepatch
29 - filebeat
30 - ksplice
31 - landscape-client
32 - lldpd
33 - nrpe
34 - ntp
35 - telegraf
36 - thruk-agent
37 - hw-health
38 - logrotated
39
40operations optional subordinates: &operations-optional-subs
41 - policy-routing
42 - bcache-tuning
43 - sysconfig
44 - logrotate-charm
45 - advanced-routing
46 - rsyslog-forwarder-ha
47
48operations kubernetes mandatory: &operations-kubernetes-mandatory-charms
49 - kubernetes-service-checks
diff --git a/contrib/includes/operations/bionic.yaml b/contrib/includes/operations/bionic.yaml
0new file mode 10064450new file mode 100644
index 0000000..a6baf9e
--- /dev/null
+++ b/contrib/includes/operations/bionic.yaml
@@ -0,0 +1,2 @@
1operations openstack mandatory series: &operations-openstack-mandatory-series
2 - prometheus-ceph-exporter
diff --git a/contrib/includes/operations/focal.yaml b/contrib/includes/operations/focal.yaml
0new file mode 1006443new file mode 100644
index 0000000..d658661
--- /dev/null
+++ b/contrib/includes/operations/focal.yaml
@@ -0,0 +1 @@
1operations openstack mandatory series: &operations-openstack-mandatory-series []
diff --git a/contrib/includes/saas.yaml b/contrib/includes/saas.yaml
0new file mode 1006442new file mode 100644
index 0000000..a3ed08f
--- /dev/null
+++ b/contrib/includes/saas.yaml
@@ -0,0 +1,7 @@
1saas:
2 - elasticsearch
3 - grafana
4 - graylog
5 - landscape-server
6 - nagios
7 - prometheus2
diff --git a/contrib/kubernetes.yaml b/contrib/kubernetes.yaml
0new file mode 1006448new file mode 100644
index 0000000..c7460b1
--- /dev/null
+++ b/contrib/kubernetes.yaml
@@ -0,0 +1,22 @@
1---
2!include includes/base.yaml
3!include includes/operations.yaml
4!include includes/kubernetes.yaml
5!include includes/saas.yaml
6
7operations charms: &operations-charms
8 - *operations-kubernetes-mandatory-charms
9 - *operations-mandatory-charms
10 - *operations-mandatory-deps
11 - *operations-mandatory-subs
12 - *operations-optional-charms
13 - *operations-optional-subs
14
15kubernetes charms: &kubernetes-charms
16 - *kubernetes-mandatory-charms
17 - *kubernetes-optional-charms
18
19known charms:
20 - ubuntu
21 - *operations-charms
22 - *kubernetes-charms
diff --git a/contrib/openstack-bionic-ovn.yaml b/contrib/openstack-bionic-ovn.yaml
0new file mode 10064423new file mode 100644
index 0000000..d1a600c
--- /dev/null
+++ b/contrib/openstack-bionic-ovn.yaml
@@ -0,0 +1,13 @@
1---
2# Openstack rule set for a Bionic OVN Cloud
3# * Use OVN, not OVS
4# * Uses percona-cluster, not mysql-innodb-cluster/router
5# * Includes prometheus-ceph-exporter
6!include includes/base.yaml
7!include includes/networking/ovn.yaml
8!include includes/database/percona-cluster.yaml
9!include includes/operations/bionic.yaml
10!include includes/openstack.yaml
11!include includes/operations.yaml
12!include includes/saas.yaml
13!include includes/aggregator-openstack.yaml
diff --git a/contrib/openstack-bionic-ovs.yaml b/contrib/openstack-bionic-ovs.yaml
0new file mode 10064414new file mode 100644
index 0000000..60304e3
--- /dev/null
+++ b/contrib/openstack-bionic-ovs.yaml
@@ -0,0 +1,13 @@
1---
2# Openstack rule set for a Bionic OVS Cloud
3# * Use OVS, not OVN
4# * Uses percona-cluster, not mysql-innodb-cluster/router
5# * Includes prometheus-ceph-exporter
6!include includes/base.yaml
7!include includes/networking/ovs.yaml
8!include includes/database/percona-cluster.yaml
9!include includes/operations/bionic.yaml
10!include includes/openstack.yaml
11!include includes/operations.yaml
12!include includes/saas.yaml
13!include includes/aggregator-openstack.yaml
diff --git a/contrib/openstack-focal-ovn.yaml b/contrib/openstack-focal-ovn.yaml
0new file mode 10064414new file mode 100644
index 0000000..3c0a562
--- /dev/null
+++ b/contrib/openstack-focal-ovn.yaml
@@ -0,0 +1,13 @@
1---
2# Openstack rule set for a Focal OVN Cloud
3# * Use OVN, not OVS
4# * Uses mysql-innodb-cluster/router, not percona-cluster
5# * No prometheus-ceph-exporter
6!include includes/base.yaml
7!include includes/networking/ovn.yaml
8!include includes/database/mysql.yaml
9!include includes/operations/focal.yaml
10!include includes/openstack.yaml
11!include includes/operations.yaml
12!include includes/saas.yaml
13!include includes/aggregator-openstack.yaml
diff --git a/contrib/openstack-focal-ovs.yaml b/contrib/openstack-focal-ovs.yaml
0new file mode 10064414new file mode 100644
index 0000000..e1039cb
--- /dev/null
+++ b/contrib/openstack-focal-ovs.yaml
@@ -0,0 +1,13 @@
1---
2# Openstack rule set for a Focal OVS Cloud
3# * Use OVS, not OVN
4# * Uses mysql-innodb-cluster/router, not percona-cluster
5# * No prometheus-ceph-exporter
6!include includes/base.yaml
7!include includes/networking/ovs.yaml
8!include includes/database/mysql.yaml
9!include includes/operations/focal.yaml
10!include includes/openstack.yaml
11!include includes/operations.yaml
12!include includes/saas.yaml
13!include includes/aggregator-openstack.yaml
diff --git a/jujulint/lint.py b/jujulint/lint.py
index 3add8bf..38dfc00 100755
--- a/jujulint/lint.py
+++ b/jujulint/lint.py
@@ -118,8 +118,11 @@ class Linter:
118 def read_rules(self):118 def read_rules(self):
119 """Read and parse rules from YAML, optionally processing provided overrides."""119 """Read and parse rules from YAML, optionally processing provided overrides."""
120 if os.path.isfile(self.filename):120 if os.path.isfile(self.filename):
121 with open(self.filename, "r") as yaml_file:121 with open(self.filename, "r") as rules_file:
122 self.lint_rules = yaml.safe_load(yaml_file)122 raw_rules_txt = rules_file.read()
123
124 self.lint_rules = self._process_includes_in_rules(raw_rules_txt)
125
123 if self.overrides:126 if self.overrides:
124 for override in self.overrides.split("#"):127 for override in self.overrides.split("#"):
125 (name, where) = override.split(":")128 (name, where) = override.split(":")
@@ -133,9 +136,10 @@ class Linter:
133 )136 )
134 )137 )
135 self.lint_rules["subordinates"][name] = dict(where=where)138 self.lint_rules["subordinates"][name] = dict(where=where)
136 self.lint_rules["known charms"] = flatten_list(139
137 self.lint_rules["known charms"]140 # Flatten all entries (to account for nesting due to YAML anchors (templating)
138 )141 self.lint_rules = {k: flatten_list(v) for k, v in self.lint_rules.items()}
142
139 self.logger.debug(143 self.logger.debug(
140 "[{}] [{}/{}] Lint Rules: {}".format(144 "[{}] [{}/{}] Lint Rules: {}".format(
141 self.cloud_name,145 self.cloud_name,
@@ -1163,3 +1167,32 @@ class Linter:
1163 )1167 )
1164 if self.collect_errors:1168 if self.collect_errors:
1165 self.collect(error)1169 self.collect(error)
1170
1171 def _process_includes_in_rules(self, yaml_txt):
1172 """
1173 Process any includes in the rules file.
1174
1175 Only top level includes are supported (without recursion), with relative paths.
1176
1177 Example syntax:
1178
1179 !include foo.yaml
1180 """
1181 collector = []
1182 for line in yaml_txt.splitlines():
1183 if line.startswith("!include"):
1184 try:
1185 _, rel_path = line.split()
1186 except ValueError:
1187 self.logger.warn("invalid include in rules, ignored: '{}'".format(line))
1188 continue
1189
1190 include_path = os.path.join(os.path.dirname(self.filename), rel_path)
1191
1192 if os.path.isfile(include_path):
1193 with open(include_path, "r") as f:
1194 collector.append(f.read())
1195 else:
1196 collector.append(line)
1197
1198 return yaml.safe_load("\n".join(collector))
diff --git a/jujulint/util.py b/jujulint/util.py
index b8c1359..c6eb80e 100644
--- a/jujulint/util.py
+++ b/jujulint/util.py
@@ -29,6 +29,10 @@ class InvalidCharmNameError(Exception):
2929
30def flatten_list(lumpy_list):30def flatten_list(lumpy_list):
31 """Flatten a list potentially containing other lists."""31 """Flatten a list potentially containing other lists."""
32 # Ensure we only operate on lists, otherwise will affect other iterables
33 if not isinstance(lumpy_list, list):
34 return lumpy_list
35
32 flat_list = []36 flat_list = []
33 for item in lumpy_list:37 for item in lumpy_list:
34 if not isinstance(item, list):38 if not isinstance(item, list):
diff --git a/tests/test_jujulint.py b/tests/test_jujulint.py
index 1e200d2..ce123ee 100644
--- a/tests/test_jujulint.py
+++ b/tests/test_jujulint.py
@@ -16,6 +16,12 @@ def test_flatten_list(utils):
16 assert flattened_list == utils.flatten_list(unflattened_list)16 assert flattened_list == utils.flatten_list(unflattened_list)
1717
1818
19def test_flatten_list_non_list_iterable(utils):
20 """Test the utils flatten_list function."""
21 iterable = {1: 2}
22 assert iterable == utils.flatten_list(iterable)
23
24
19def test_map_charms(linter, utils):25def test_map_charms(linter, utils):
20 """Test the charm name validation code."""26 """Test the charm name validation code."""
21 applications = {27 applications = {
@@ -434,3 +440,24 @@ class TestLinter:
434 assert error is not None440 assert error is not None
435 assert error["id"] == "ops-charm-missing"441 assert error["id"] == "ops-charm-missing"
436 assert error["charm"] == "grafana"442 assert error["charm"] == "grafana"
443
444 def test_read_rules_plain_yaml(self, linter, tmp_path):
445 """Test that a simple rules YAML is imported as expected."""
446 rules_path = tmp_path / "rules.yaml"
447 rules_path.write_text('---\nkey:\n "value"')
448
449 linter.filename = str(rules_path)
450 linter.read_rules()
451 assert linter.lint_rules == {"key": "value"}
452
453 def test_read_rules_include(self, linter, tmp_path):
454 """Test that rules YAML with an include is imported as expected."""
455 include_path = tmp_path / "include.yaml"
456 include_path.write_text('key-inc:\n "value2"')
457
458 rules_path = tmp_path / "rules.yaml"
459 rules_path.write_text('---\n!include include.yaml\nkey:\n "value"')
460
461 linter.filename = str(rules_path)
462 linter.read_rules()
463 assert linter.lint_rules == {"key": "value", "key-inc": "value2"}

Subscribers

People subscribed via source and target branches