Merge ~lloydwaltersj/maas:oapi-linter into maas:master
- Git
- lp:~lloydwaltersj/maas
- oapi-linter
- Merge into master
Status: | Merged |
---|---|
Approved by: | Jack Lloyd-Walters |
Approved revision: | 1cbbc4976c25603e92dad0e557d623ebe889a17f |
Merge reported by: | MAAS Lander |
Merged at revision: | not available |
Proposed branch: | ~lloydwaltersj/maas:oapi-linter |
Merge into: | maas:master |
Diff against target: |
135 lines (+79/-2) 3 files modified
Makefile (+6/-1) setup.cfg (+10/-1) utilities/check-oapi (+63/-0) |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
MAAS Lander | Approve | ||
Alexsander de Souza | Approve | ||
Adam Collard (community) | Abstain | ||
Review via email: mp+428646@code.launchpad.net |
Commit message
Add a linter for the OpenAPI spec
Description of the change
add the ability to lint the MAAS Http api against OAS3
Alexsander de Souza (alexsander-souza) wrote : | # |
use tox to handle the test environment
Adam Collard (adam-collard) wrote : | # |
Don't think we should consider this a linter in the same way that we do the other linters.
It should be implemented as a test in the api package, alongside the API definition, documentation etc.
If this fails, it's not because we've used a deprecated way of writing Python, or sorted imports differently, it's a failure close to the OpenAPI feature.
Alexsander de Souza (alexsander-souza) wrote : | # |
> If this fails, it's not because we've used a deprecated way of writing Python,
> or sorted imports differently, it's a failure close to the OpenAPI feature.
I think this is real linter, because if this fails it means that we failed to write valid documentation for whatever we are doing, not that the OpenAPI feature has a bug.
- cd623b8... by Jack Lloyd-Walters
-
transition to tox
MAAS Lander (maas-lander) wrote : | # |
UNIT TESTS
-b oapi-linter lp:~lloydwaltersj/maas/+git/maas into -b master lp:~maas-committers/maas
STATUS: FAILED
LOG: http://
COMMIT: cd623b889ddf406
- a90fe6e... by Jack Lloyd-Walters
-
change how oapi doccs are generated for linting
Jack Lloyd-Walters (lloydwaltersj) wrote (last edit ): | # |
> STATUS: FAILED
> LOG: http://
> COMMIT: cd623b889ddf406
This is working exactly as expected, a successfull failure if you will!
MAAS Lander (maas-lander) wrote : | # |
UNIT TESTS
-b oapi-linter lp:~lloydwaltersj/maas/+git/maas into -b master lp:~maas-committers/maas
STATUS: FAILED
LOG: http://
COMMIT: a90fe6e49bee699
Jack Lloyd-Walters (lloydwaltersj) wrote : | # |
jenkins: !test
MAAS Lander (maas-lander) wrote : | # |
UNIT TESTS
-b oapi-linter lp:~lloydwaltersj/maas/+git/maas into -b master lp:~maas-committers/maas
STATUS: FAILED
LOG: http://
COMMIT: a90fe6e49bee699
- 39c02c3... by Jack Lloyd-Walters
-
bring branch up to parity
- c0695f9... by Jack Lloyd-Walters
-
revert yaml.dump change
Adam Collard (adam-collard) : | # |
MAAS Lander (maas-lander) wrote : | # |
UNIT TESTS
-b oapi-linter lp:~lloydwaltersj/maas/+git/maas into -b master lp:~maas-committers/maas
STATUS: FAILED
LOG: http://
COMMIT: c0695f98b1e9ced
- 4eaea74... by Jack Lloyd-Walters
-
remove make docs from lint
MAAS Lander (maas-lander) wrote : | # |
UNIT TESTS
-b oapi-linter lp:~lloydwaltersj/maas/+git/maas into -b master lp:~maas-committers/maas
STATUS: FAILED
LOG: http://
COMMIT: 4eaea745bfebe77
Adam Collard (adam-collard) : | # |
- 2c01771... by Jack Lloyd-Walters
-
remove hardcoded openapi file location
MAAS Lander (maas-lander) wrote : | # |
UNIT TESTS
-b oapi-linter lp:~lloydwaltersj/maas/+git/maas into -b master lp:~maas-committers/maas
STATUS: FAILED
LOG: http://
COMMIT: 2c01771792cbdf7
Adam Collard (adam-collard) : | # |
- 6247138... by Jack Lloyd-Walters
-
bring branch up to parity
MAAS Lander (maas-lander) wrote : | # |
UNIT TESTS
-b oapi-linter lp:~lloydwaltersj/maas/+git/maas into -b master lp:~maas-committers/maas
STATUS: FAILED
LOG: http://
COMMIT: 6247138baf4652b
- b55d865... by Jack Lloyd-Walters
-
bring branch up to parity
MAAS Lander (maas-lander) wrote : | # |
UNIT TESTS
-b oapi-linter lp:~lloydwaltersj/maas/+git/maas into -b master lp:~maas-committers/maas
STATUS: FAILED
LOG: http://
COMMIT: b55d865e978a033
- 8421bf8... by Jack Lloyd-Walters
-
generate spec if missing
- bb8d96d... by Jack Lloyd-Walters
-
make linter depend on spec
MAAS Lander (maas-lander) wrote : | # |
UNIT TESTS
-b oapi-linter lp:~lloydwaltersj/maas/+git/maas into -b master lp:~maas-committers/maas
STATUS: SUCCESS
COMMIT: 8421bf8f4b32f39
MAAS Lander (maas-lander) wrote : | # |
UNIT TESTS
-b oapi-linter lp:~lloydwaltersj/maas/+git/maas into -b master lp:~maas-committers/maas
STATUS: SUCCESS
COMMIT: bb8d96d2bf145e9
Adam Collard (adam-collard) : | # |
Adam Collard (adam-collard) : | # |
- 921393f... by Jack Lloyd-Walters
-
remove help supression
MAAS Lander (maas-lander) wrote : | # |
UNIT TESTS
-b oapi-linter lp:~lloydwaltersj/maas/+git/maas into -b master lp:~maas-committers/maas
STATUS: FAILED
LOG: http://
COMMIT: 921393ffd0cc2f1
Jack Lloyd-Walters (lloydwaltersj) wrote : | # |
jenkins: !test
MAAS Lander (maas-lander) wrote : | # |
UNIT TESTS
-b oapi-linter lp:~lloydwaltersj/maas/+git/maas into -b master lp:~maas-committers/maas
STATUS: SUCCESS
COMMIT: 921393ffd0cc2f1
Alexsander de Souza (alexsander-souza) wrote : | # |
+1
- 46f3269... by Jack Lloyd-Walters
-
remove lint-oapi from lint target
- 1cbbc49... by Jack Lloyd-Walters
-
move oapi linter to test phase
MAAS Lander (maas-lander) wrote : | # |
UNIT TESTS
-b oapi-linter lp:~lloydwaltersj/maas/+git/maas into -b master lp:~maas-committers/maas
STATUS: SUCCESS
COMMIT: 1cbbc4976c25603
Preview Diff
1 | diff --git a/Makefile b/Makefile | |||
2 | index 5a5c2af..903c5af 100644 | |||
3 | --- a/Makefile | |||
4 | +++ b/Makefile | |||
5 | @@ -121,7 +121,7 @@ go-bins: | |||
6 | 121 | $(MAKE) -j -C src/host-info build | 121 | $(MAKE) -j -C src/host-info build |
7 | 122 | .PHONY: go-bins | 122 | .PHONY: go-bins |
8 | 123 | 123 | ||
10 | 124 | test: test-missing-migrations test-py | 124 | test: test-missing-migrations test-py lint-oapi |
11 | 125 | .PHONY: test | 125 | .PHONY: test |
12 | 126 | 126 | ||
13 | 127 | test-missing-migrations: bin/database bin/maas-region | 127 | test-missing-migrations: bin/database bin/maas-region |
14 | @@ -166,6 +166,11 @@ lint-py-linefeeds: | |||
15 | 166 | (echo "Lint check failed; run make format to fix DOS linefeeds."; false) | 166 | (echo "Lint check failed; run make format to fix DOS linefeeds."; false) |
16 | 167 | .PHONY: lint-py-linefeeds | 167 | .PHONY: lint-py-linefeeds |
17 | 168 | 168 | ||
18 | 169 | # Open API Spec | ||
19 | 170 | lint-oapi: openapi.yaml | ||
20 | 171 | @tox -e oapi | ||
21 | 172 | .PHONY: lint-oapi | ||
22 | 173 | |||
23 | 169 | # Go fmt | 174 | # Go fmt |
24 | 170 | lint-go: | 175 | lint-go: |
25 | 171 | @find src/ \( -name pkg -o -name vendor \) -prune -o -name '*.go' -exec gofmt -l {} + | \ | 176 | @find src/ \( -name pkg -o -name vendor \) -prune -o -name '*.go' -exec gofmt -l {} + | \ |
26 | diff --git a/setup.cfg b/setup.cfg | |||
27 | index 5d02873..c4c7faa 100644 | |||
28 | --- a/setup.cfg | |||
29 | +++ b/setup.cfg | |||
30 | @@ -94,6 +94,7 @@ lint_files = | |||
31 | 94 | src/provisioningserver/refresh/50-maas-01-commissioning | 94 | src/provisioningserver/refresh/50-maas-01-commissioning |
32 | 95 | utilities/bin2python.py | 95 | utilities/bin2python.py |
33 | 96 | utilities/check-imports | 96 | utilities/check-imports |
34 | 97 | utilities/check-oapi | ||
35 | 97 | utilities/import-db-dump | 98 | utilities/import-db-dump |
36 | 98 | utilities/python_standard_libs.py | 99 | utilities/python_standard_libs.py |
37 | 99 | utilities/snap-bind-mounts | 100 | utilities/snap-bind-mounts |
38 | @@ -109,6 +110,9 @@ deps_lint = | |||
39 | 109 | cogapp == 3.3.0 | 110 | cogapp == 3.3.0 |
40 | 110 | click == 8.1.3 | 111 | click == 8.1.3 |
41 | 111 | 112 | ||
42 | 113 | deps_oapi = | ||
43 | 114 | openapi-spec-validator == 0.4.0 | ||
44 | 115 | |||
45 | 112 | [flake8] | 116 | [flake8] |
46 | 113 | ignore = E203, E266, E501, W503, W504 | 117 | ignore = E203, E266, E501, W503, W504 |
47 | 114 | exclude = | 118 | exclude = |
48 | @@ -116,7 +120,7 @@ exclude = | |||
49 | 116 | 120 | ||
50 | 117 | [tox:tox] | 121 | [tox:tox] |
51 | 118 | skipsdist = True | 122 | skipsdist = True |
53 | 119 | envlist = format,lint | 123 | envlist = format,lint,oapi |
54 | 120 | 124 | ||
55 | 121 | [testenv:format] | 125 | [testenv:format] |
56 | 122 | deps = {[globals]deps_lint} | 126 | deps = {[globals]deps_lint} |
57 | @@ -132,3 +136,8 @@ commands = | |||
58 | 132 | black --check {[globals]lint_files} | 136 | black --check {[globals]lint_files} |
59 | 133 | flake8 {[globals]lint_files} | 137 | flake8 {[globals]lint_files} |
60 | 134 | cog --check --verbosity=1 {[globals]cog_files} | 138 | cog --check --verbosity=1 {[globals]cog_files} |
61 | 139 | |||
62 | 140 | [testenv:oapi] | ||
63 | 141 | deps = {[globals]deps_oapi} | ||
64 | 142 | commands = | ||
65 | 143 | python3 utilities/check-oapi openapi.yaml | ||
66 | 135 | \ No newline at end of file | 144 | \ No newline at end of file |
67 | diff --git a/utilities/check-oapi b/utilities/check-oapi | |||
68 | 136 | new file mode 100755 | 145 | new file mode 100755 |
69 | index 0000000..d77ae23 | |||
70 | --- /dev/null | |||
71 | +++ b/utilities/check-oapi | |||
72 | @@ -0,0 +1,63 @@ | |||
73 | 1 | #!/usr/bin/env python3 | ||
74 | 2 | |||
75 | 3 | """Script to ensure that an OpenAPI spec matches OAS3""" | ||
76 | 4 | |||
77 | 5 | import argparse | ||
78 | 6 | import sys | ||
79 | 7 | |||
80 | 8 | from openapi_spec_validator import openapi_v3_spec_validator as validator | ||
81 | 9 | import yaml | ||
82 | 10 | |||
83 | 11 | failure = "\033[0;31m" | ||
84 | 12 | success = "\033[0;32m" | ||
85 | 13 | end = "\033[0m" | ||
86 | 14 | |||
87 | 15 | |||
88 | 16 | def fetch_errors(errors): | ||
89 | 17 | """Return all the errors in the oapi spec with | ||
90 | 18 | some nice formatting for the terminal | ||
91 | 19 | """ | ||
92 | 20 | num = len(errors) | ||
93 | 21 | pad = len(str(num)) | ||
94 | 22 | err_ = [f"{num} problems with the spec:"] | ||
95 | 23 | for i, err in enumerate(errors): | ||
96 | 24 | msg, reason, loc = str(err).split("\n\n") | ||
97 | 25 | i = str(i).rjust(pad) | ||
98 | 26 | err_ += [ | ||
99 | 27 | end | ||
100 | 28 | + f"{i}: {msg}\n{failure}{reason}\n{loc}".replace( | ||
101 | 29 | "\n", "\n" + " " * (2 + pad) | ||
102 | 30 | ) | ||
103 | 31 | ] | ||
104 | 32 | return failure + "\n\n".join(err for err in err_) + end | ||
105 | 33 | |||
106 | 34 | |||
107 | 35 | def validate(spec): | ||
108 | 36 | """a wrapper around the spec validator for our purposes""" | ||
109 | 37 | errors = list(validator.iter_errors(spec)) | ||
110 | 38 | if errors: | ||
111 | 39 | error = fetch_errors(errors) | ||
112 | 40 | raise Exception(error) | ||
113 | 41 | else: | ||
114 | 42 | print(success + "No errors found in spec!" + end) | ||
115 | 43 | |||
116 | 44 | |||
117 | 45 | def main(): | ||
118 | 46 | parser = argparse.ArgumentParser( | ||
119 | 47 | description="Check an OpenAPI yaml schema against OAS3" | ||
120 | 48 | ) | ||
121 | 49 | parser.add_argument( | ||
122 | 50 | "api_spec", | ||
123 | 51 | help="path for the file to validate against.", | ||
124 | 52 | type=argparse.FileType("r"), | ||
125 | 53 | ) | ||
126 | 54 | args = parser.parse_args() | ||
127 | 55 | with args.api_spec as f: | ||
128 | 56 | spec = yaml.load(f, Loader=yaml.FullLoader) | ||
129 | 57 | info = spec["info"] | ||
130 | 58 | print(f"Checking {info['title']} version {info['version']}") | ||
131 | 59 | validate(spec) | ||
132 | 60 | |||
133 | 61 | |||
134 | 62 | if __name__ == "__main__": | ||
135 | 63 | sys.exit(main()) |
UNIT TESTS
-b oapi-linter lp:~lloydwaltersj/maas/+git/maas into -b master lp:~maas-committers/maas
STATUS: FAILED maas-ci. internal: 8080/job/ maas-tester/ 355/consoleText 4e22296b5c0e243 6ab1c66120
LOG: http://
COMMIT: 94339310e45dd8f