Merge ~dashmage/charm-sysconfig:bug-2012581/inherit-existing-kernel-params into charm-sysconfig:master

Proposed by Ashley James
Status: Merged
Approved by: Eric Chen
Approved revision: 0e5531c372b4776fcde2f541565fc17c6d8f9886
Merged at revision: a2e49a8d488bbf5eadcf84dc5934731b5ec44743
Proposed branch: ~dashmage/charm-sysconfig:bug-2012581/inherit-existing-kernel-params
Merge into: charm-sysconfig:master
Diff against target: 149 lines (+71/-7)
5 files modified
src/actions.yaml (+1/-1)
src/config.yaml (+3/-0)
src/lib/lib_sysconfig.py (+26/-1)
src/templates/grub.j2 (+0/-5)
src/tests/unit/test_lib.py (+41/-0)
Reviewer Review Type Date Requested Status
JamesLin Approve
🤖 prod-jenkaas-bootstack (community) continuous-integration Approve
Eric Chen Approve
Erhan Sunar (community) Needs Fixing
BootStack Reviewers Pending
Review via email: mp+441304@code.launchpad.net

Commit message

Inherit pre-existing kernel cmdline parameters by default.

- Also prevents overwrite of existing console parameters by the charm.

Description of the change

Fix for issue #4 in BSENG-1027 [1] for the LP bugs #2012581 [2], #1854135 [3]. Addresses this comment on the LP bug as well [4].

[1]: https://warthogs.atlassian.net/browse/BSENG-1027
[2]: https://bugs.launchpad.net/charm-sysconfig/+bug/2012581
[3]: https://bugs.launchpad.net/charm-sysconfig/+bug/1854135
[4]: https://bugs.launchpad.net/charm-sysconfig/+bug/1854135/comments/4

To post a comment you must log in.
Revision history for this message
JamesLin (jneo8) wrote (last edit ):

It's LGTM for implementation part, but I think the unit test for it is required.
Also please provide more detail to explain why this change works.

review: Needs Fixing
Revision history for this message
Erhan Sunar (esunar) wrote :

Can you link the bug reports to MR

review: Needs Fixing
Revision history for this message
Ashley James (dashmage) wrote :

> It's LGTM for implementation part, but I think the unit test for it is
> required.

There seems to be unit tests already for the `update_grub_file` function [1]. Would I need to add something additionally to test for the change made in the template file?

[1]: https://git.launchpad.net/charm-sysconfig/tree/src/tests/unit/test_lib.py#n250

> Also please provide more detail to explain why this change works.

```
-{% else -%}
-GRUB_CMDLINE_LINUX_DEFAULT=""
```

This is to allow inheriting the pre-existing kernel cmdline parameters the default option. With the charm no longer resetting the value of `GRUB_CMDLINE_LINUX_DEFAULT` to an empty string, it'll only append to the already existing values (ex: coming from MAAS).

```
-GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT console=ttyS0,115200 console=tty0"
+GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,115200 console=tty0 $GRUB_CMDLINE_LINUX_DEFAULT"
```

These parameters have been switched around so that the pre-existing console values (ex: coming from MAAS) will not be overwritten by the charm. According to my understanding, the console parameters (if defined) from `$GRUB_CMDLINE_LINUX_DEFAULT` will take precedence since they are declared the last.

Revision history for this message
🤖 Canonical IS Merge Bot (canonical-is-mergebot) wrote :

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Approve (continuous-integration)
Revision history for this message
Ashley James (dashmage) wrote :

Got feedback from @mfo who implemented the initial logic in the grub template file:

```
It looks like that MR is good.

AFAICT it
1) fixes the case I had wrong (do not erase $GRUB... if charm config option is not set), and
2) also addresses the case there are MAAS kernel options.
I haven't done any testing (I know you would / already have, for both issues), but it looks good from what I see.
```

Revision history for this message
JamesLin (jneo8) wrote (last edit ):

> There seems to be unit tests already for the `update_grub_file` function [1]. Would I need to add something additionally to test for the change made in the template file?

The unit test currently doesn't cover the bug case and it should be.

review: Needs Fixing
Revision history for this message
Eric Chen (eric-chen) wrote :

> > It's LGTM for implementation part, but I think the unit test for it is
> > required.
>
> There seems to be unit tests already for the `update_grub_file` function [1].
> Would I need to add something additionally to test for the change made in the
> template file?
>
> [1]: https://git.launchpad.net/charm-
> sysconfig/tree/src/tests/unit/test_lib.py#n250
>
> > Also please provide more detail to explain why this change works.
>
> ```
> -{% else -%}
> -GRUB_CMDLINE_LINUX_DEFAULT=""
> ```
>
> This is to allow inheriting the pre-existing kernel cmdline parameters the
> default option. With the charm no longer resetting the value of
> `GRUB_CMDLINE_LINUX_DEFAULT` to an empty string, it'll only append to the
> already existing values (ex: coming from MAAS).
>
> ```
> -GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT console=ttyS0,115200
> console=tty0"
> +GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,115200 console=tty0
> $GRUB_CMDLINE_LINUX_DEFAULT"
> ```
>
> These parameters have been switched around so that the pre-existing console
> values (ex: coming from MAAS) will not be overwritten by the charm. According
> to my understanding, the console parameters (if defined) from
> `$GRUB_CMDLINE_LINUX_DEFAULT` will take precedence since they are declared the
> last.

This is a little tricky to me. There already a precedence design under /etc/grub.d/ folder.
If we want to put "console=ttyS0,115200 console=tty0" as a "nice to have" option, we should put it into 00xxxx. But now, we put it into 90sysconfig (almost the biggest number) and use this way to avoid taking precedence.

I don't know if other people have the same feeling. I suggest that we should find the stakeholders to discuss which one should have higher precedence and solve it in a correct way.

Revision history for this message
Ashley James (dashmage) wrote (last edit ):

After discussing with @nobuto, the real issue faced is not that the MAAS console parameters are being overwritten.

```
the original issue was that even if the machine already had ttyS0 and tty, charm added those additionally and it triggered a reboot required flag

as long as the reboot required flag is not triggered, I'm happy with any solutions
```

The charm will certainly trigger a reboot notification at the time of deployment since the template file has minor changes from the original config file on the unit. But this only happens at the time of initial deployment of the charm. Could a possible solution be to have the charm run the `clear-notification` action as part of the install hook handler since the notification is a false positive?

In case the charm does need to support a way to prevent overwriting the MAAS kernel parameters, I've listed down some possible solutions and my proposal in this document here: https://docs.google.com/document/d/19sB_LqBY5xFL0-8JDc1RzgOzQh8quwa3NSsXQIbkfj0/edit?usp=sharing

Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Ashley James (dashmage) wrote :

What this MR does:
- Inherit pre-existing kernel parameters (eg: from MAAS) by default
- Remove hard-coded console parameters
- Introduce new "console" config option

Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Approve (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Approve (continuous-integration)
Revision history for this message
Eric Chen (eric-chen) wrote :

1. No unit test for prepare_console_context
2. some inline comment, please fix it.

review: Needs Fixing
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Ashley James (dashmage) wrote :

Updated MR:
- Inherit pre-existing kernel parameters (eg: from MAAS) by default
- Remove hard-coded console parameters
- Rework config parser function to support console parameters
- Unit tests for config parser function

Revision history for this message
JamesLin (jneo8) wrote :

Please also update the config and doc-string.

review: Needs Fixing
Revision history for this message
Eric Chen (eric-chen) :
review: Needs Fixing
Revision history for this message
Eric Chen (eric-chen) wrote :

Please see inline comment.

review: Needs Fixing
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Ashley James (dashmage) wrote :

Resolved comments by @jneo8 and @eric-chen

Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Eric Chen (eric-chen) wrote :

please check inline comment and fix it

review: Needs Fixing
Revision history for this message
Eric Chen (eric-chen) wrote :

one more inline comment

review: Needs Fixing
Revision history for this message
Ashley James (dashmage) wrote :

> I think a string with "=" assignments is not the key point in case 3. There are also "=" in case 1&2. The major different is double quote. Please mention that the subkey and value can be assigned by double quote.

There are "=" in the prev 2 cases but they do not occur in the values. Here in case 3,
'key1="subkey1=val1 subkey2=val2", key2=val3' -- the double quotes are just to make the assignment between key value pairs more clearer. The key point is that the subkey values themselves contain assignments that include "=". Even if the double quotes aren't present, the parser will correctly parse it into a dictionary.

> This case is weird. Don't we need the double quote?
'key1="subkey1=val1,val2 subkey2=val3"'

Here as well, the double quotes aren't really necessary but just to aid in identifying the key and value clearer. This case is to show how there's "=" assignments and the subkey is assigned multiple values separated by commas unlike the previous case which just contained a single value. The commas between val1 and val2 in "subkey1=val1,val2" is what this case is supposed to capture.

I've also added another unit test where the double quotes aren't present. I've addressed the other inline comments as well.

Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Eric Chen (eric-chen) wrote :

Please check my inline comment.

review: Needs Fixing
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Approve (continuous-integration)
Revision history for this message
Eric Chen (eric-chen) wrote :

Please check inline comment. You miss some of my comments before.

review: Needs Fixing
Revision history for this message
Ashley James (dashmage) wrote :

@eric-chen -- addressed all your latest comments

Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Approve (continuous-integration)
Revision history for this message
Eric Chen (eric-chen) :
review: Approve
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Approve (continuous-integration)
Revision history for this message
JamesLin (jneo8) wrote :

LGTM

review: Approve
Revision history for this message
🤖 Canonical IS Merge Bot (canonical-is-mergebot) wrote :

Change successfully merged at revision a2e49a8d488bbf5eadcf84dc5934731b5ec44743

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/src/actions.yaml b/src/actions.yaml
index 1dcaedb..93282ae 100644
--- a/src/actions.yaml
+++ b/src/actions.yaml
@@ -6,7 +6,7 @@ clear-notification:
6 By doing so, unit will change the description to 'ready'6 By doing so, unit will change the description to 'ready'
7update-grub:7update-grub:
8 description: |8 description: |
9 This action will run 'update-grub' to the generate a GRUB configuration file.9 This action will run 'update-grub' to generate a GRUB configuration file.
10check-update-grub:10check-update-grub:
11 description: |11 description: |
12 This action will test if there is any GRUB configuration changes available.12 This action will test if there is any GRUB configuration changes available.
diff --git a/src/config.yaml b/src/config.yaml
index ba4a0a6..f7455e4 100644
--- a/src/config.yaml
+++ b/src/config.yaml
@@ -121,6 +121,9 @@ options:
121 to avoid them being overwritten, for example121 to avoid them being overwritten, for example
122 grub-config-flags='GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT122 grub-config-flags='GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT
123 nvme_core.multipath=0'123 nvme_core.multipath=0'
124 Also supports adding values with special characters in them(`$`, `=`, `,` etc.)
125 but they need to be enclosed in double quotes. For example, grub-config-flags='GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT
126 console=ttyS0,115200 console=tty0",GRUB_TIMEOUT=0'
124 isolcpus:127 isolcpus:
125 default: ""128 default: ""
126 type: string129 type: string
diff --git a/src/lib/lib_sysconfig.py b/src/lib/lib_sysconfig.py
index 1a9875a..caf4c00 100644
--- a/src/lib/lib_sysconfig.py
+++ b/src/lib/lib_sysconfig.py
@@ -5,6 +5,7 @@ Manage grub, systemd, coufrequtils and kernel version configuration.
5import filecmp5import filecmp
6import hashlib6import hashlib
7import os7import os
8import re
8import subprocess9import subprocess
9from datetime import datetime, timedelta, timezone10from datetime import datetime, timedelta, timezone
1011
@@ -35,10 +36,34 @@ IRQBALANCE_CONF = "/etc/default/irqbalance"
35def parse_config_flags(config_flags):36def parse_config_flags(config_flags):
36 """Parse config flags into a dict.37 """Parse config flags into a dict.
3738
39 This parsing function supports a few different formats
40 depending on the config flags being provided.
41
42 1. Simple string with key=value pairs. For example,
43 a string in the format of 'key1=val1, key2=val2' will return a dict of:
44 {'key1': 'val1', 'key2': 'val2'}.
45
46 2. A string in the above format, but supporting a comma-delimited
47 list of values for the same key. For example, a string in the format of
48 'key1=val1, key2=val2,val3,val4' will return a dict of:
49 {'key1': 'val1', 'key2': 'val2,val3,val4'}
50
51 3. A string with special characters (`=`, `,`) in the value.
52 Note: This case requires double quotes to properly separate the key and value pairs
53 since there are special characters that need to be protected. Adding the double
54 quotes explicitly ensures that the quotes are present in the dict output as well.
55 For example, a string in the format of 'key1="subkey1=val1,val2 subkey2=val3"'
56 will return a dict of: {'key1': '"subkey1=val1,val2 subkey2=val3"'}
57
38 :param config_flags: key pairs list. Format: key1=value1,key2=value258 :param config_flags: key pairs list. Format: key1=value1,key2=value2
39 :return dict: format {'key1': 'value1', 'key2': 'value2'}59 :return dict: format {'key1': 'value1', 'key2': 'value2'}
40 """60 """
41 key_value_pairs = config_flags.split(",")61 # This regular expression is used to split the config flags
62 # into a list while preserving the content within the
63 # double quotes even if it contains commas
64 # For example, it splits 'val1, val2, "val3,val4", val5'
65 # as ['val1', 'val2', '"val3, val4"', 'val5]
66 key_value_pairs = re.split(r',(?=(?:[^"]*"[^"]*")*[^"]*$)', config_flags)
42 parsed_config_flags = {}67 parsed_config_flags = {}
43 for index, pair in enumerate(key_value_pairs):68 for index, pair in enumerate(key_value_pairs):
44 if "=" in pair:69 if "=" in pair:
diff --git a/src/templates/grub.j2 b/src/templates/grub.j2
index a76ca57..5d7cf79 100644
--- a/src/templates/grub.j2
+++ b/src/templates/grub.j2
@@ -5,18 +5,13 @@
5# For full documentation of the options in this file, see:5# For full documentation of the options in this file, see:
6# info -f grub -n 'Simple configuration'6# info -f grub -n 'Simple configuration'
77
8# Evaluate grub-config-flags first, in case it uses $GRUB_CMDLINE_LINUX_DEFAULT.
9# However, if grub-config-flags is not specified, ignore that (current behavior.)
10{% if grub_config_flags is defined and grub_config_flags -%}8{% if grub_config_flags is defined and grub_config_flags -%}
11# config-flags9# config-flags
12{% for key, value in grub_config_flags.items() -%}10{% for key, value in grub_config_flags.items() -%}
13{{ key }}={{ value }}11{{ key }}={{ value }}
14{% endfor -%}12{% endfor -%}
15{% else -%}
16GRUB_CMDLINE_LINUX_DEFAULT=""
17{% endif -%}13{% endif -%}
1814
19GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT console=ttyS0,115200 console=tty0"
20{% if isolcpus is defined and isolcpus -%}15{% if isolcpus is defined and isolcpus -%}
21GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT isolcpus={{ isolcpus }}"16GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT isolcpus={{ isolcpus }}"
22{% endif -%}17{% endif -%}
diff --git a/src/tests/unit/test_lib.py b/src/tests/unit/test_lib.py
index 9721313..b2b5354 100644
--- a/src/tests/unit/test_lib.py
+++ b/src/tests/unit/test_lib.py
@@ -45,6 +45,47 @@ def test_check_update_grub_unavailable(check_output, cmp_file):
45 assert "No available grub updates found." in message45 assert "No available grub updates found." in message
4646
4747
48@pytest.mark.parametrize(
49 "config_flags, expected",
50 [
51 ("key1=val1,key2=val2", {"key1": "val1", "key2": "val2"}),
52 # whitespace between key value pairs after comma
53 ("key1=val1, key2=val2", {"key1": "val1", "key2": "val2"}),
54 # multiple values assigned to key
55 ("key1=val1,val2,key2=val3", {"key1": "val1,val2", "key2": "val3"}),
56 ("key1=val1 val2 val3", {"key1": "val1 val2 val3"}),
57 # Subkey assigned to key has a single value (val) assigned with "="
58 ('key="$key subkey=val"', {"key": '"$key subkey=val"'}),
59 (
60 'key1="subkey1=val1 val2",key2="$key2 val3"',
61 {"key1": '"subkey1=val1 val2"', "key2": '"$key2 val3"'},
62 ),
63 # Subkey has multiple values (val1, val2) assigned each separated by comma
64 (
65 'key1="subkey1=val1,val2 subkey2=val3",key2=val4',
66 {"key1": '"subkey1=val1,val2 subkey2=val3"', "key2": "val4"},
67 ),
68 (
69 'key1="$key1 subkey1=val1,val2 subkey2=val3", key2=val4',
70 {"key1": '"$key1 subkey1=val1,val2 subkey2=val3"', "key2": "val4"},
71 ),
72 (
73 (
74 'key1="$key1 subkey1=val1,val2 subkey2=val3", key2="$key2 '
75 'subkey1=val1 subkey2=val2"'
76 ),
77 {
78 "key1": '"$key1 subkey1=val1,val2 subkey2=val3"',
79 "key2": '"$key2 subkey1=val1 subkey2=val2"',
80 },
81 ),
82 ],
83)
84def test_parse_config_flags(config_flags, expected):
85 """Test parsing function for config flags."""
86 assert lib_sysconfig.parse_config_flags(config_flags) == expected
87
88
48class TestBootResourceState:89class TestBootResourceState:
49 """Test BootResourceState class."""90 """Test BootResourceState class."""
5091

Subscribers

People subscribed via source and target branches

to all changes: