Merge ~lloydwaltersj/maas:fix_oapi_get_config_description into maas:master

[Jack Lloyd-Walters]

Add formatting fix for https://github.com/canonical/maas.io/issues/806

[MAAS Lander (maas-lander)]

UNIT TESTS
-b fix_oapi_get_config_description lp:~lloydwaltersj/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: SUCCESS
COMMIT: b69a16ef1a7636406227a22973de28048c919be1

review: Approve

dd40ddd... by Jack Lloyd-Walters on 2023-09-13

cleanup and respond to feedback

[MAAS Lander (maas-lander)]

UNIT TESTS
-b fix_oapi_get_config_description lp:~lloydwaltersj/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/maas-tester/3536/console
COMMIT: dd40ddd71042cb5b118c255163dc7fc6cfa24fab

review: Needs Fixing

eb20fea... by Jack Lloyd-Walters on 2023-09-13

update openapi template

[Nick De Villiers (nickdv99)]

Once this has landed, I have a branch ready for maas.io that should fix the formatting on the frontend as well. Let me know once the YAML file we use for production has been updated

[MAAS Lander (maas-lander)]

UNIT TESTS
-b fix_oapi_get_config_description lp:~lloydwaltersj/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/maas-tester/3539/console
COMMIT: eb20fea54c621b94eb1ffb679448ebae8a4beb55

review: Needs Fixing

ada201d... by Jack Lloyd-Walters on 2023-09-13

linting

af06abf... by Jack Lloyd-Walters on 2023-09-13

add offline docs to api page

485e6b2... by Jack Lloyd-Walters on 2023-09-13

add todo

bb74991... by Jack Lloyd-Walters on 2023-09-13

add makefile targets

[MAAS Lander (maas-lander)]

UNIT TESTS
-b fix_oapi_get_config_description lp:~lloydwaltersj/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: SUCCESS
COMMIT: 485e6b28a8a7843d071466e4e50ff192fb85bb05

review: Approve

[MAAS Lander (maas-lander)]

UNIT TESTS
-b fix_oapi_get_config_description lp:~lloydwaltersj/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: SUCCESS
COMMIT: bb749919ff5dec223e523a068475d23b1954fe2d

review: Approve

2e78d34... by Jack Lloyd-Walters on 2023-09-15

fix makefile name

[MAAS Lander (maas-lander)]

UNIT TESTS
-b fix_oapi_get_config_description lp:~lloydwaltersj/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: SUCCESS
COMMIT: 2e78d34bd2eee96abe4ceff798b675b7448f29bd

review: Approve

diff --git a/Makefile b/Makefile
index a22a714..4635d90 100644
--- a/Makefile
+++ b/Makefile
@@ -259,7 +259,10 @@ api-docs.rst: bin/maas-region src/maasserver/api/doc_handler.py syncdb
 openapi.yaml: bin/maas-region src/maasserver/api/doc_handler.py syncdb
 	bin/maas-region generate_oapi_spec > $@
 
-doc: api-docs.rst openapi.yaml swagger-css swagger-js
+openapi-doc: openapi.yaml swagger-css swagger-js
+.PHONY: openapi-doc
+
+doc: api-docs.rst openapi-doc
 .PHONY: doc
 
 clean-ui:
diff --git a/src/maasserver/api/doc_oapi.py b/src/maasserver/api/doc_oapi.py
index 5ee4d55..e8e3f1b 100644
--- a/src/maasserver/api/doc_oapi.py
+++ b/src/maasserver/api/doc_oapi.py
@@ -11,8 +11,10 @@ from inspect import getdoc, signature
 import json
 import re
 from textwrap import dedent
+from typing import Any
 
 from django.http import HttpResponse
+from piston3.resource import Resource
 import yaml
 
 from maasserver.api import support
@@ -29,8 +31,19 @@ PARAM_RE = re.compile(
     r"^{(?P<param>\S+)}$",
 )
 
+# https://github.com/canonical/maas.io/issues/806
+# Match variables enclosed by :, ', ` within a docstring, variables cannot contain spaces or the
+# enclosing character within their definition
+MARKERS = [":", "'", "`"]
+MATCHES = [re.compile(rf" {k}[^ ^{k}]*{k} ") for k in MARKERS]
 
-def landing_page(request):
+NEWLINES = re.compile(r"(?<![:.])[\n]+")
+WHITESPACE = re.compile(r"\n\s+")
+PUNCTUATION = re.compile(r"([^\w\s])\1+")
+POINTS = ["*", "+", "-"]
+
+
+def landing_page(request: str) -> HttpResponse:
     """Render a landing page with pointers for the MAAS API.
 
     :return: An `HttpResponse` containing a JSON page with pointers to both
@@ -46,7 +59,7 @@ def landing_page(request):
     )
 
 
-def endpoint(request):
+def endpoint(request: str) -> HttpResponse:
     """Render the OpenApi endpoint.
 
     :return: An `HttpResponse` containing a YAML document that complies
@@ -61,7 +74,7 @@ def endpoint(request):
     )
 
 
-def get_api_landing_page():
+def get_api_landing_page() -> dict[str, str | Any]:
     """Return the API landing page"""
     description = {
         "title": "MAAS API",
@@ -74,23 +87,29 @@ def get_api_landing_page():
                 "title": "this document",
             },
             {
+                "path": "/MAAS/docs",
+                "rel": "service-doc",
+                "type": "text/html",
+                "title": "offline MAAS documentation",
+            },
+            {
                 "path": f"{settings.API_URL_PREFIX}openapi.yaml",
                 "rel": "service-desc",
                 "type": "application/openapi+yaml",
-                "title": "the API definition",
+                "title": "the OpenAPI definition",
             },
             {
                 "path": "/MAAS/api/docs/",
                 "rel": "service-doc",
                 "type": "text/html",
-                "title": "the API documentation",
+                "title": "OpenAPI documentation",
             },
         ],
     }
     return description
 
 
-def get_api_endpoint():
+def get_api_endpoint() -> dict[str, str | Any]:
     """Return the API endpoint"""
     description = {
         "openapi": "3.0.0",
@@ -112,7 +131,7 @@ def get_api_endpoint():
     return description
 
 
-def _get_maas_servers():
+def _get_maas_servers() -> list[dict[str, str]]:
     """Return a servers defintion of the public-facing MAAS address.
 
     :return: An object describing the MAAS public-facing server.
@@ -129,7 +148,7 @@ def _get_maas_servers():
     ]
 
 
-def _new_path_item(params):
+def _new_path_item(params: list[Any]) -> dict[str, dict[str, Any]]:
     path_item = {}
     for p in params:
         path_item.setdefault("parameters", []).append(
@@ -143,17 +162,65 @@ def _new_path_item(params):
     return path_item
 
 
-def _prettify(doc):
-    """Cleans up text by replacing newlines with spaces, so that sentences are
-    not broken apart prematurely.
-    Respects paragraphing by not replacing newlines that occur after periods.
+def _prettify(doc: str) -> str:
+    """Cleans up text by:
+    - Dedenting text to the same depth.
+    - Respecting paragraphing by not replacing newlines that occur after periods or colons
+    - Removeing duplicate punctuation groups and replaces with singular
     """
-    return re.sub("(?<![.\n])[\n]+", " ", dedent(doc)).strip()
+    doc = dedent(doc)
+    doc = NEWLINES.sub(" ", doc)
+    doc = WHITESPACE.sub("\n", doc)
+    doc = PUNCTUATION.sub(r"\1", doc)
+    for idx, point in enumerate(POINTS):
+        doc = re.sub(rf"\n\s*\{point}", f"\n{' '*2*idx}{point}", doc)
+    return doc.strip()
+
+
+def _contains_variables(doc: str) -> list[str] | None:
+    """Search for any instances of :variable:, ''variable'', or `variable`."""
+    for m in MATCHES:
+        if (v := m.findall(doc[doc.find(":") :])) and len(v) > 1:
+            return v
+
+
+def _parse_enumerable(doc: str) -> tuple[str, list[str]]:
+    """Parse docstring for multiple variables. Clean up and represent as the correct
+    form. (https://github.com/canonical/maas.io/issues/806)"""
+    enumerable = {}
+    if variables := _contains_variables(doc):
+        for idx, var in enumerate(variables):
+            start_pos = doc.find(var) + len(var)
+            end_pos = (
+                len(doc)
+                if idx >= len(variables) - 1
+                else doc.find(variables[idx + 1])
+            )
+            var_string = doc[start_pos:end_pos]
+            doc = doc.replace(var + var_string, "")
+            depth = MARKERS.index(var[1])
+            enumerable[var.strip(var[:2]) or " "] = {
+                "description": _parse_enumerable(var_string)[0],
+                "point": POINTS[depth],
+            }
+        doc = "\n".join(
+            [f"{doc}"]
+            + [
+                f"{v['point']} `{k}` {v['description'].strip('.,')}."
+                for k, v in enumerable.items()
+            ]
+        )
+    return doc, list(enumerable.keys())
 
 
 def _render_oapi_oper_item(
-    http_method, op, doc, uri_params, function, resources
-):
+    http_method: str,
+    op: str,
+    doc: str,
+    uri_params: Any,
+    function: object,
+    resources: set[Resource],
+) -> dict[str, str | Any]:
     oper_id = op or support.OperationsResource.crudmap.get(http_method)
     oper_obj = {
         "operationId": f"{doc.name}_{oper_id}",
@@ -171,9 +238,13 @@ def _render_oapi_oper_item(
 
 
 def _oapi_item_from_docstring(
-    function, http_method, uri_params, doc, resources
-):
-    def _type_to_string(schema):
+    function: object,
+    http_method: str,
+    uri_params: Any,
+    doc: str,
+    resources: set[Resource],
+) -> dict[str, str | Any]:
+    def _type_to_string(schema: str) -> str:
         match schema:
             case "Boolean":
                 return "boolean"
@@ -186,7 +257,7 @@ def _oapi_item_from_docstring(
             case _:
                 return "object"
 
-    def _response_pair(ap_dict):
+    def _response_pair(ap_dict: dict[str, str | Any]) -> list[str]:
         status_code = "HTTP Status Code"
         status = content = {}
         paired = []
@@ -215,11 +286,15 @@ def _oapi_item_from_docstring(
         ap.parse(docstring)
         ap_dict = ap.get_dict()
         oper_obj["summary"] = ap_dict["description_title"].strip()
-        oper_obj["description"] = _prettify(ap_dict["description"])
+
+        oper_obj["description"] = _prettify(
+            _parse_enumerable(ap_dict["description"])[0]
+        )
+
         if "deprecated" in oper_obj["description"].lower():
             oper_obj["deprecated"] = True
         for param in ap_dict["params"]:
-            description = _prettify(param["description_stripped"])
+            description = _parse_enumerable(param["description_stripped"])[0]
             # LP 2009140
             stripped_name = PARAM_RE.match(param["name"])
             name = (
@@ -236,7 +311,7 @@ def _oapi_item_from_docstring(
                 param_dict = {
                     "name": name,
                     "in": "path" if name in uri_params else "query",
-                    "description": description,
+                    "description": _prettify(description),
                     "schema": {
                         "type": _type_to_string(param["type"]),
                     },
@@ -244,10 +319,11 @@ def _oapi_item_from_docstring(
                 }
                 oper_obj.setdefault("parameters", []).append(param_dict)
             else:
-                body.setdefault("properties", {})[name] = {
-                    "description": description,
+                params_dict = {
+                    "description": _prettify(description),
                     "type": _type_to_string(param["type"]),
                 }
+                body.setdefault("properties", {})[name] = params_dict
                 if required:
                     body.setdefault("required", []).append(name)
 
@@ -336,13 +412,15 @@ def _oapi_item_from_docstring(
     return oper_obj
 
 
-def _render_oapi_paths():
+def _render_oapi_paths() -> dict[str, str | Any]:
     from maasserver import urls_api as urlconf
 
-    def _resource_key(resource):
+    def _resource_key(resource: Resource) -> str:
         return resource.handler.__class__.__name__
 
-    def _export_key(export):
+    def _export_key(
+        export: tuple[tuple[str, str], object]
+    ) -> tuple[str, object]:
         (http_method, op), function = export
         return http_method, op or "", function
 
diff --git a/src/maasserver/api/tests/test_oapi.py b/src/maasserver/api/tests/test_oapi.py
index d22c2cc..c28e18d 100644
--- a/src/maasserver/api/tests/test_oapi.py
+++ b/src/maasserver/api/tests/test_oapi.py
@@ -138,12 +138,8 @@ represented in ASCII using ``bsondump example.bson`` and is for
 demonstrative purposes."""
 
         after = """\
-Returns system details -- for example, LLDP and ``lshw`` XML dumps.
-
-
-Returns a ``{detail_type: xml, ...}`` map, where ``detail_type`` is something like "lldp" or "lshw".
-
-
-Note that this is returned as BSON and not JSON. This is for efficiency, but mainly because JSON can''t do binary content without applying additional encoding like base-64. The example output below is represented in ASCII using ``bsondump example.bson`` and is for demonstrative purposes."""
+Returns system details - for example, LLDP and `lshw` XML dumps.
+Returns a `{detail_type: xml, .}` map, where `detail_type` is something like "lldp" or "lshw".
+Note that this is returned as BSON and not JSON. This is for efficiency, but mainly because JSON can't do binary content without applying additional encoding like base-64. The example output below is represented in ASCII using `bsondump example.bson` and is for demonstrative purposes."""
 
         self.assertEqual(_prettify(before), after)
diff --git a/src/maasserver/templates/openapi.html b/src/maasserver/templates/openapi.html
index 4b75523..23186c8 100644
--- a/src/maasserver/templates/openapi.html
+++ b/src/maasserver/templates/openapi.html
@@ -4,7 +4,8 @@
   <head>
     <meta charset="UTF-8">
     <title>MAAS API</title>
-    <link href="/MAAS/r/static/css/main.a4c6517c.css" rel="stylesheet">
+    <!-- TODO: Figure out how to update this programatically -->
+    <link href="/MAAS/r/static/css/main.d1c59af9.css" rel="stylesheet">
     <style>
       {% include "dist/swagger-ui.css" %}
     </style>
@@ -21,6 +22,9 @@
         margin: 0 !important;
         max-width: 100% !important;
       }
+      .swagger-ui .parameters-col_description {
+        width: 85% !important
+      }
     </style>
   </head>