Merge ~dashmage/charm-sysconfig:bug-2012581/inherit-existing-kernel-params into charm-sysconfig:master

Proposed by Ashley James
Status: Merged
Approved by: Eric Chen
Approved revision: 0e5531c372b4776fcde2f541565fc17c6d8f9886
Merged at revision: a2e49a8d488bbf5eadcf84dc5934731b5ec44743
Proposed branch: ~dashmage/charm-sysconfig:bug-2012581/inherit-existing-kernel-params
Merge into: charm-sysconfig:master
Diff against target: 149 lines (+71/-7)
5 files modified
src/actions.yaml (+1/-1)
src/config.yaml (+3/-0)
src/lib/lib_sysconfig.py (+26/-1)
src/templates/grub.j2 (+0/-5)
src/tests/unit/test_lib.py (+41/-0)
Reviewer Review Type Date Requested Status
JamesLin Approve
🤖 prod-jenkaas-bootstack (community) continuous-integration Approve
Eric Chen Approve
Erhan Sunar (community) Needs Fixing
BootStack Reviewers Pending
Review via email: mp+441304@code.launchpad.net

Commit message

Inherit pre-existing kernel cmdline parameters by default.

- Also prevents overwrite of existing console parameters by the charm.

Description of the change

Fix for issue #4 in BSENG-1027 [1] for the LP bugs #2012581 [2], #1854135 [3]. Addresses this comment on the LP bug as well [4].

[1]: https://warthogs.atlassian.net/browse/BSENG-1027
[2]: https://bugs.launchpad.net/charm-sysconfig/+bug/2012581
[3]: https://bugs.launchpad.net/charm-sysconfig/+bug/1854135
[4]: https://bugs.launchpad.net/charm-sysconfig/+bug/1854135/comments/4

To post a comment you must log in.
Revision history for this message
JamesLin (jneo8) wrote (last edit ):

It's LGTM for implementation part, but I think the unit test for it is required.
Also please provide more detail to explain why this change works.

review: Needs Fixing
Revision history for this message
Erhan Sunar (esunar) wrote :

Can you link the bug reports to MR

review: Needs Fixing
Revision history for this message
Ashley James (dashmage) wrote :

> It's LGTM for implementation part, but I think the unit test for it is
> required.

There seems to be unit tests already for the `update_grub_file` function [1]. Would I need to add something additionally to test for the change made in the template file?

[1]: https://git.launchpad.net/charm-sysconfig/tree/src/tests/unit/test_lib.py#n250

> Also please provide more detail to explain why this change works.

```
-{% else -%}
-GRUB_CMDLINE_LINUX_DEFAULT=""
```

This is to allow inheriting the pre-existing kernel cmdline parameters the default option. With the charm no longer resetting the value of `GRUB_CMDLINE_LINUX_DEFAULT` to an empty string, it'll only append to the already existing values (ex: coming from MAAS).

```
-GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT console=ttyS0,115200 console=tty0"
+GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,115200 console=tty0 $GRUB_CMDLINE_LINUX_DEFAULT"
```

These parameters have been switched around so that the pre-existing console values (ex: coming from MAAS) will not be overwritten by the charm. According to my understanding, the console parameters (if defined) from `$GRUB_CMDLINE_LINUX_DEFAULT` will take precedence since they are declared the last.

Revision history for this message
🤖 Canonical IS Merge Bot (canonical-is-mergebot) wrote :

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Approve (continuous-integration)
Revision history for this message
Ashley James (dashmage) wrote :

Got feedback from @mfo who implemented the initial logic in the grub template file:

```
It looks like that MR is good.

AFAICT it
1) fixes the case I had wrong (do not erase $GRUB... if charm config option is not set), and
2) also addresses the case there are MAAS kernel options.
I haven't done any testing (I know you would / already have, for both issues), but it looks good from what I see.
```

Revision history for this message
JamesLin (jneo8) wrote (last edit ):

> There seems to be unit tests already for the `update_grub_file` function [1]. Would I need to add something additionally to test for the change made in the template file?

The unit test currently doesn't cover the bug case and it should be.

review: Needs Fixing
Revision history for this message
Eric Chen (eric-chen) wrote :

> > It's LGTM for implementation part, but I think the unit test for it is
> > required.
>
> There seems to be unit tests already for the `update_grub_file` function [1].
> Would I need to add something additionally to test for the change made in the
> template file?
>
> [1]: https://git.launchpad.net/charm-
> sysconfig/tree/src/tests/unit/test_lib.py#n250
>
> > Also please provide more detail to explain why this change works.
>
> ```
> -{% else -%}
> -GRUB_CMDLINE_LINUX_DEFAULT=""
> ```
>
> This is to allow inheriting the pre-existing kernel cmdline parameters the
> default option. With the charm no longer resetting the value of
> `GRUB_CMDLINE_LINUX_DEFAULT` to an empty string, it'll only append to the
> already existing values (ex: coming from MAAS).
>
> ```
> -GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT console=ttyS0,115200
> console=tty0"
> +GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,115200 console=tty0
> $GRUB_CMDLINE_LINUX_DEFAULT"
> ```
>
> These parameters have been switched around so that the pre-existing console
> values (ex: coming from MAAS) will not be overwritten by the charm. According
> to my understanding, the console parameters (if defined) from
> `$GRUB_CMDLINE_LINUX_DEFAULT` will take precedence since they are declared the
> last.

This is a little tricky to me. There already a precedence design under /etc/grub.d/ folder.
If we want to put "console=ttyS0,115200 console=tty0" as a "nice to have" option, we should put it into 00xxxx. But now, we put it into 90sysconfig (almost the biggest number) and use this way to avoid taking precedence.

I don't know if other people have the same feeling. I suggest that we should find the stakeholders to discuss which one should have higher precedence and solve it in a correct way.

Revision history for this message
Ashley James (dashmage) wrote (last edit ):

After discussing with @nobuto, the real issue faced is not that the MAAS console parameters are being overwritten.

```
the original issue was that even if the machine already had ttyS0 and tty, charm added those additionally and it triggered a reboot required flag

as long as the reboot required flag is not triggered, I'm happy with any solutions
```

The charm will certainly trigger a reboot notification at the time of deployment since the template file has minor changes from the original config file on the unit. But this only happens at the time of initial deployment of the charm. Could a possible solution be to have the charm run the `clear-notification` action as part of the install hook handler since the notification is a false positive?

In case the charm does need to support a way to prevent overwriting the MAAS kernel parameters, I've listed down some possible solutions and my proposal in this document here: https://docs.google.com/document/d/19sB_LqBY5xFL0-8JDc1RzgOzQh8quwa3NSsXQIbkfj0/edit?usp=sharing

Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Ashley James (dashmage) wrote :

What this MR does:
- Inherit pre-existing kernel parameters (eg: from MAAS) by default
- Remove hard-coded console parameters
- Introduce new "console" config option

Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Approve (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Approve (continuous-integration)
Revision history for this message
Eric Chen (eric-chen) wrote :

1. No unit test for prepare_console_context
2. some inline comment, please fix it.

review: Needs Fixing
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Ashley James (dashmage) wrote :

Updated MR:
- Inherit pre-existing kernel parameters (eg: from MAAS) by default
- Remove hard-coded console parameters
- Rework config parser function to support console parameters
- Unit tests for config parser function

Revision history for this message
JamesLin (jneo8) wrote :

Please also update the config and doc-string.

review: Needs Fixing
Revision history for this message
Eric Chen (eric-chen) :
review: Needs Fixing
Revision history for this message
Eric Chen (eric-chen) wrote :

Please see inline comment.

review: Needs Fixing
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Ashley James (dashmage) wrote :

Resolved comments by @jneo8 and @eric-chen

Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Eric Chen (eric-chen) wrote :

please check inline comment and fix it

review: Needs Fixing
Revision history for this message
Eric Chen (eric-chen) wrote :

one more inline comment

review: Needs Fixing
Revision history for this message
Ashley James (dashmage) wrote :

> I think a string with "=" assignments is not the key point in case 3. There are also "=" in case 1&2. The major different is double quote. Please mention that the subkey and value can be assigned by double quote.

There are "=" in the prev 2 cases but they do not occur in the values. Here in case 3,
'key1="subkey1=val1 subkey2=val2", key2=val3' -- the double quotes are just to make the assignment between key value pairs more clearer. The key point is that the subkey values themselves contain assignments that include "=". Even if the double quotes aren't present, the parser will correctly parse it into a dictionary.

> This case is weird. Don't we need the double quote?
'key1="subkey1=val1,val2 subkey2=val3"'

Here as well, the double quotes aren't really necessary but just to aid in identifying the key and value clearer. This case is to show how there's "=" assignments and the subkey is assigned multiple values separated by commas unlike the previous case which just contained a single value. The commas between val1 and val2 in "subkey1=val1,val2" is what this case is supposed to capture.

I've also added another unit test where the double quotes aren't present. I've addressed the other inline comments as well.

Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Eric Chen (eric-chen) wrote :

Please check my inline comment.

review: Needs Fixing
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Approve (continuous-integration)
Revision history for this message
Eric Chen (eric-chen) wrote :

Please check inline comment. You miss some of my comments before.

review: Needs Fixing
Revision history for this message
Ashley James (dashmage) wrote :

@eric-chen -- addressed all your latest comments

Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Approve (continuous-integration)
Revision history for this message
Eric Chen (eric-chen) :
review: Approve
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Approve (continuous-integration)
Revision history for this message
JamesLin (jneo8) wrote :

LGTM

review: Approve
Revision history for this message
🤖 Canonical IS Merge Bot (canonical-is-mergebot) wrote :

Change successfully merged at revision a2e49a8d488bbf5eadcf84dc5934731b5ec44743

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/src/actions.yaml b/src/actions.yaml
2index 1dcaedb..93282ae 100644
3--- a/src/actions.yaml
4+++ b/src/actions.yaml
5@@ -6,7 +6,7 @@ clear-notification:
6 By doing so, unit will change the description to 'ready'
7 update-grub:
8 description: |
9- This action will run 'update-grub' to the generate a GRUB configuration file.
10+ This action will run 'update-grub' to generate a GRUB configuration file.
11 check-update-grub:
12 description: |
13 This action will test if there is any GRUB configuration changes available.
14diff --git a/src/config.yaml b/src/config.yaml
15index ba4a0a6..f7455e4 100644
16--- a/src/config.yaml
17+++ b/src/config.yaml
18@@ -121,6 +121,9 @@ options:
19 to avoid them being overwritten, for example
20 grub-config-flags='GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT
21 nvme_core.multipath=0'
22+ Also supports adding values with special characters in them(`$`, `=`, `,` etc.)
23+ but they need to be enclosed in double quotes. For example, grub-config-flags='GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT
24+ console=ttyS0,115200 console=tty0",GRUB_TIMEOUT=0'
25 isolcpus:
26 default: ""
27 type: string
28diff --git a/src/lib/lib_sysconfig.py b/src/lib/lib_sysconfig.py
29index 1a9875a..caf4c00 100644
30--- a/src/lib/lib_sysconfig.py
31+++ b/src/lib/lib_sysconfig.py
32@@ -5,6 +5,7 @@ Manage grub, systemd, coufrequtils and kernel version configuration.
33 import filecmp
34 import hashlib
35 import os
36+import re
37 import subprocess
38 from datetime import datetime, timedelta, timezone
39
40@@ -35,10 +36,34 @@ IRQBALANCE_CONF = "/etc/default/irqbalance"
41 def parse_config_flags(config_flags):
42 """Parse config flags into a dict.
43
44+ This parsing function supports a few different formats
45+ depending on the config flags being provided.
46+
47+ 1. Simple string with key=value pairs. For example,
48+ a string in the format of 'key1=val1, key2=val2' will return a dict of:
49+ {'key1': 'val1', 'key2': 'val2'}.
50+
51+ 2. A string in the above format, but supporting a comma-delimited
52+ list of values for the same key. For example, a string in the format of
53+ 'key1=val1, key2=val2,val3,val4' will return a dict of:
54+ {'key1': 'val1', 'key2': 'val2,val3,val4'}
55+
56+ 3. A string with special characters (`=`, `,`) in the value.
57+ Note: This case requires double quotes to properly separate the key and value pairs
58+ since there are special characters that need to be protected. Adding the double
59+ quotes explicitly ensures that the quotes are present in the dict output as well.
60+ For example, a string in the format of 'key1="subkey1=val1,val2 subkey2=val3"'
61+ will return a dict of: {'key1': '"subkey1=val1,val2 subkey2=val3"'}
62+
63 :param config_flags: key pairs list. Format: key1=value1,key2=value2
64 :return dict: format {'key1': 'value1', 'key2': 'value2'}
65 """
66- key_value_pairs = config_flags.split(",")
67+ # This regular expression is used to split the config flags
68+ # into a list while preserving the content within the
69+ # double quotes even if it contains commas
70+ # For example, it splits 'val1, val2, "val3,val4", val5'
71+ # as ['val1', 'val2', '"val3, val4"', 'val5]
72+ key_value_pairs = re.split(r',(?=(?:[^"]*"[^"]*")*[^"]*$)', config_flags)
73 parsed_config_flags = {}
74 for index, pair in enumerate(key_value_pairs):
75 if "=" in pair:
76diff --git a/src/templates/grub.j2 b/src/templates/grub.j2
77index a76ca57..5d7cf79 100644
78--- a/src/templates/grub.j2
79+++ b/src/templates/grub.j2
80@@ -5,18 +5,13 @@
81 # For full documentation of the options in this file, see:
82 # info -f grub -n 'Simple configuration'
83
84-# Evaluate grub-config-flags first, in case it uses $GRUB_CMDLINE_LINUX_DEFAULT.
85-# However, if grub-config-flags is not specified, ignore that (current behavior.)
86 {% if grub_config_flags is defined and grub_config_flags -%}
87 # config-flags
88 {% for key, value in grub_config_flags.items() -%}
89 {{ key }}={{ value }}
90 {% endfor -%}
91-{% else -%}
92-GRUB_CMDLINE_LINUX_DEFAULT=""
93 {% endif -%}
94
95-GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT console=ttyS0,115200 console=tty0"
96 {% if isolcpus is defined and isolcpus -%}
97 GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT isolcpus={{ isolcpus }}"
98 {% endif -%}
99diff --git a/src/tests/unit/test_lib.py b/src/tests/unit/test_lib.py
100index 9721313..b2b5354 100644
101--- a/src/tests/unit/test_lib.py
102+++ b/src/tests/unit/test_lib.py
103@@ -45,6 +45,47 @@ def test_check_update_grub_unavailable(check_output, cmp_file):
104 assert "No available grub updates found." in message
105
106
107+@pytest.mark.parametrize(
108+ "config_flags, expected",
109+ [
110+ ("key1=val1,key2=val2", {"key1": "val1", "key2": "val2"}),
111+ # whitespace between key value pairs after comma
112+ ("key1=val1, key2=val2", {"key1": "val1", "key2": "val2"}),
113+ # multiple values assigned to key
114+ ("key1=val1,val2,key2=val3", {"key1": "val1,val2", "key2": "val3"}),
115+ ("key1=val1 val2 val3", {"key1": "val1 val2 val3"}),
116+ # Subkey assigned to key has a single value (val) assigned with "="
117+ ('key="$key subkey=val"', {"key": '"$key subkey=val"'}),
118+ (
119+ 'key1="subkey1=val1 val2",key2="$key2 val3"',
120+ {"key1": '"subkey1=val1 val2"', "key2": '"$key2 val3"'},
121+ ),
122+ # Subkey has multiple values (val1, val2) assigned each separated by comma
123+ (
124+ 'key1="subkey1=val1,val2 subkey2=val3",key2=val4',
125+ {"key1": '"subkey1=val1,val2 subkey2=val3"', "key2": "val4"},
126+ ),
127+ (
128+ 'key1="$key1 subkey1=val1,val2 subkey2=val3", key2=val4',
129+ {"key1": '"$key1 subkey1=val1,val2 subkey2=val3"', "key2": "val4"},
130+ ),
131+ (
132+ (
133+ 'key1="$key1 subkey1=val1,val2 subkey2=val3", key2="$key2 '
134+ 'subkey1=val1 subkey2=val2"'
135+ ),
136+ {
137+ "key1": '"$key1 subkey1=val1,val2 subkey2=val3"',
138+ "key2": '"$key2 subkey1=val1 subkey2=val2"',
139+ },
140+ ),
141+ ],
142+)
143+def test_parse_config_flags(config_flags, expected):
144+ """Test parsing function for config flags."""
145+ assert lib_sysconfig.parse_config_flags(config_flags) == expected
146+
147+
148 class TestBootResourceState:
149 """Test BootResourceState class."""
150

Subscribers

People subscribed via source and target branches

to all changes: