Merge ~artivis/lpci:feature/yaml-alias into lpci:main

[jeremie]

Changes pydantic configuration for extra fields from 'forbid' to 'ignore'. This allows defining extra fields which can be used to e.g. define yaml anchors. Aliases are resolved to anchors by pyyaml before being ignored (filtered out) by pydantic.

[Colin Watson (cjwatson)]

I'm concerned about this approach, because it would mean that people's configuration files might break when we add new official fields.

Is there some way that we could at least confine field names used for anchors to some kind of namespace?

[jeremie (artivis)]

> Is there some way that we could at least confine field names used for anchors
> to some kind of namespace?

Thanks for the prompt review.
Another option would be to implement a pydantic root_validator that simply filters out entries at the root level based on a prefix - see here (https://git.launchpad.net/~artivis/lpcraft/tree/lpcraft/config.py?h=feature/yaml-alias-filter#n269). Note that the prefix I used is a simple dot but it could be more elaborated. For instance Docker Compose as a similar mechanism called extension fields using the prefix 'x-' (https://docs.docker.com/compose/compose-file/compose-file-v3/#extension-fields).

[Jürgen Gmach (jugmac00)]

Some more info about this topic:
https://medium.com/@kinghuang/docker-compose-anchors-aliases-extensions-a1e4105d70bd

[Jürgen Gmach (jugmac00)]

On our internal channels I have proposed a solution which leverages an external tool (or Python one-liner) to convert an enhanced version of a YAML file including alias/includes into a standard YAML file which we already support as of now:
https://chat.canonical.com/canonical/pl/r71cgk3ejtr1zmbfnfitbi187c

Let's keep the conversation on the chat until we come to a conclusion.

Unmerged commits

9e4b6f6... by jeremie on 2023-01-09

couple tests yaml anchor/alias

Signed-off-by: artivis <email address hidden>

	test:0 (build)
	build:0 (build)

1 → 2 of 2 results

First • Previous • Next • Last

e31b622... by jeremie on 2023-01-09

pydantic ignore extra fields in config

Signed-off-by: artivis <email address hidden>

diff --git a/lpcraft/config.py b/lpcraft/config.py
index d228270..eaaf73c 100644
--- a/lpcraft/config.py
+++ b/lpcraft/config.py
@@ -31,7 +31,7 @@ class _Identifier(pydantic.ConstrainedStr):
 
 class ModelConfigDefaults(
     pydantic.BaseModel,
-    extra=pydantic.Extra.forbid,
+    extra=pydantic.Extra.ignore,
     alias_generator=lambda s: s.replace("_", "-"),
     underscore_attrs_are_private=True,
 ):
diff --git a/lpcraft/tests/test_config.py b/lpcraft/tests/test_config.py
index 6c69eab..8695520 100644
--- a/lpcraft/tests/test_config.py
+++ b/lpcraft/tests/test_config.py
@@ -825,3 +825,87 @@ class TestConfig(TestCase):
             Config.load,
             path,
         )
+
+    def test_yaml_anchor_alias_single_value(self):
+        # A yaml alias is resolved to an anchor holding a single value.
+        path = self.create_config(
+            dedent(
+                """
+                pipeline:
+                    - test
+
+                jobs:
+                    test-a:
+                        series: &myseries focal
+                        architectures: [amd64]
+
+                    test-b:
+                        series: *myseries
+                        architectures: [amd64]
+                """
+            )
+        )
+        config = Config.load(path)
+        self.assertEqual(
+            config.jobs["test-a"][0].series, config.jobs["test-b"][0].series
+        )
+        self.assertEqual("focal", config.jobs["test-b"][0].series)
+
+    def test_yaml_anchor_alias_single_value_at_root(self):
+        # A yaml alias is resolved to an anchor holding a single value.
+        path = self.create_config(
+            dedent(
+                """
+                .series: &myseries focal
+
+                pipeline:
+                    - test
+
+                jobs:
+                    test:
+                        series: *myseries
+                        architectures: [amd64]
+                """
+            )
+        )
+        config = Config.load(path)
+        self.assertEqual("focal", config.jobs["test"][0].series)
+
+    def test_yaml_anchor_alias_job(self):
+        # A yaml alias is resolved to an anchor holding a whole job.
+        # The root-level extra filed is filtered out by pydantic.
+        path = self.create_config(
+            dedent(
+                """
+                .test: &mytest
+                    series: focal
+                    architectures: [amd64]
+
+                pipeline:
+                    - test
+
+                jobs:
+                    test: *mytest
+                """
+            )
+        )
+        config = Config.load(path)
+
+        self.assertThat(
+            config,
+            MatchesStructure(
+                pipeline=Equals([["test"]]),
+                jobs=MatchesDict(
+                    {
+                        "test": MatchesListwise(
+                            [
+                                MatchesStructure.byEquality(
+                                    series="focal",
+                                    architectures=["amd64"],
+                                )
+                            ]
+                        )
+                    }
+                ),
+            ),
+        )