Merge ~dashmage/charm-logrotated:bug/2017798-update-cron-daily-location into charm-logrotated:master

Proposed by Ashley James
Status: Superseded
Proposed branch: ~dashmage/charm-logrotated:bug/2017798-update-cron-daily-location
Merge into: charm-logrotated:master
Diff against target: 201 lines (+62/-26)
4 files modified
src/lib/lib_cron.py (+19/-14)
src/tests/functional/requirements.txt (+1/-0)
src/tests/functional/test_logrotate.py (+13/-5)
src/tests/unit/test_logrotate.py (+29/-7)
Reviewer Review Type Date Requested Status
Eric Chen Needs Fixing
Tianqi Xiao (community) Needs Information
🤖 prod-jenkaas-bootstack (community) continuous-integration Approve
Chi Wai CHAN Approve
Robert Gildein Approve
BootStack Reviewers Pending
JamesLin Pending
Review via email: mp+442506@code.launchpad.net

This proposal has been superseded by a proposal from 2023-05-19.

Commit message

Change from sudo user to system crontab for cron daily

Description of the change

`update-cron-daily-schedule` currently updates the sudo user present in `/var/spool/cron/crontabs/root` instead of the system crontab file (`/etc/crontab`). This change was performed so that the script only runs from a single crontab file and to fix `hookenv.log` which wasn't correctly executed when running with sudo user cronjob (messages did not show up in juju debug-log).

Fixes LP#2017798

To post a comment you must log in.
Revision history for this message
🤖 Canonical IS Merge Bot (canonical-is-mergebot) wrote :

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

Revision history for this message
Ashley James (dashmage) wrote :

I see one potential problem now - in an upgrade scenario, even after implementing this fix, the sudo user crontab would still exist if cron-update-daily-schedule was configured. This would mean that the script would still run twice.

I propose that we add a function that checks whether the `/var/spool/cron/crontabs/root` crontab file exists and remove it in case it is present since we are only managing through `/etc/crontab`.

Revision history for this message
Robert Gildein (rgildein) :
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Ashley James (dashmage) wrote (last edit ):

> Is this really wisely to overwritte system crontab? Why not put new file to /etc/cron.d/<name>?

Even if we add this as a new file to `/etc/cron.d/<name>`, the script (placed in cron.daily) would be run twice - once for `/etc/crontab` and then for `/etc/cron.d/<name>`.

In case we're modifying the time for running cron.daily scripts and want to add it as a separate file in `/etc/cron.d`, we would need to remove the line from `/etc/crontab`. I prefer we make the change directly in `/etc/crontab` itself for changing the time.

Edit: Just to clarify, the `update-cron-daily-schedule` will only change the time for running the cron.daily scripts in the /etc/crontab file.

Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Approve (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Robert Gildein (rgildein) :
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Approve (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Robert Gildein (rgildein) wrote :

LGTM, thanks

review: Approve
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Approve (continuous-integration)
Revision history for this message
Eric Chen (eric-chen) wrote :

I am curious

1. who is the runner when this library being called. (Ubuntu? Juju?) why they have root permission.
2. what's the different between f.flush and f.seek.
3. The logic looks like a magic if we want to copy the temp file. Is there better way to avoid the magic flush() or seek() function? I prefer not to use some implictly ability of flush() or seek() here. I prefer to make it just like a human do here.

#1 create temp file
#2 write data and close
#3 copy file
#4 delete temp file

review: Needs Information
Revision history for this message
Ashley James (dashmage) wrote :

> I am curious
>
> 1. who is the runner when this library being called. (Ubuntu? Juju?) why they
> have root permission.

After looking into this, the script is called in cron.daily (or weekly/monthly depending on config) and it runs as root. Here is the relevant part in the code [1]. Which is why the copy command works without "sudo". I propose we can remove the sudo while copying from the tmp file.

```
$ cat /etc/cron.daily/charm-logrotated
#!/bin/bash
/usr/bin/sudo /usr/bin/juju-run logrotated/1 "/var/lib/juju/agents/unit-logrotated-1/.venv/bin/python3 /var/lib/juju/agents/unit-logrotated-1/charm/lib/lib_cron.py"
```

[1]: https://git.launchpad.net/charm-logrotated/tree/src/lib/lib_cron.py#n66

> 2. what's the different between f.flush and f.seek.

f.seek() would make the file pointer return to zero so that we can read contents from the start of the file. f.flush() is used to clear the internal buffer and put contents into the file explicitly before f.close() or reaching the end of the context manager (where the file is automatically closed). In this situation we could also use f.flush and push the contents to the tmp file before we read it, I just chose to pick f.seek here.

> 3. The logic looks like a magic if we want to copy the temp file. Is there
> better way to avoid the magic flush() or seek() function? I prefer not to use
> some implictly ability of flush() or seek() here. I prefer to make it just
> like a human do here.
>
> #1 create temp file
> #2 write data and close
> #3 copy file
> #4 delete temp file

In case you suggest to avoid using seek/flush, we can do it this way to make it more like how a human would do it:

```
with open(r"/tmp/crontab","w") as tmp_crontab:
    tmp_crontab.write(data)

try:
    cmd = ["cp", "/tmp/crontab", "/etc/crontab"]
    subprocess.run(cmd, check=True)

except subprocess.CalledProcessError as exception:
    hookenv.log(
        f"Writing to system crontab failed with error: {exception}",
        level=hookenv.ERROR,
    )
    raise
os.remove("/tmp/crontab")
```

Revision history for this message
Eric Chen (eric-chen) :
Revision history for this message
Eric Chen (eric-chen) wrote :

Based on the description you provided in the document, f.flush() is more reasonable. There is no clue that f.seek() will flush the data into the disk. Anyway, I prefer we handle it like a human way unless there is any specific reason we must copy the file before we close the file handle.

review: Needs Fixing
Revision history for this message
Eric Chen (eric-chen) wrote :

I'm so surprise that we will execute /var/lib/juju/agents/unit-logrotated-1/charm/lib/lib_cron.py from the cronjob. It is very unreasonable to me. I suppose that every library should be used during

- event handler under src/reactive
- juju actions under src/actions

Could you check the reason why we need to design like this?

BTW, update_cron_daily_schedule is not only used in /etc/cron.daily/charm-logrotated, it may be triggered from

src/actions -> update_cronjob -> cron.install_cronjob -> update_cron_daily_schedule

The runner of these two behaviors may be different. Could you also confirm it?

1. from cron.daily
2. from juju actions

review: Needs Information
Revision history for this message
Chi Wai CHAN (raychan96) wrote :

LGTM

review: Approve
Revision history for this message
Eric Chen (eric-chen) wrote :

Change the status to "Needs Fixing"

review: Needs Fixing
Revision history for this message
Ashley James (dashmage) wrote (last edit ):

> I'm so surprise that we will execute /var/lib/juju/agents/unit-
> logrotated-1/charm/lib/lib_cron.py from the cronjob. It is very unreasonable
> to me. I suppose that every library should be used during
>
> - event handler under src/reactive
> - juju actions under src/actions
>
> Could you check the reason why we need to design like this?

After going through the code, this is what I could infer ->

The cron_lib.py code should be executed by the cronjob because the charm is used to rotate the logrotate.d files at the regular interval that is determined by the time configured in the cronjob. The event handlers under src/reactive defines install/config-changed hook to run logrotate.modify_configs once during that event but to have it run regularly by the charm it would need to be present in the cronjob so that the script can be invoked at the hourly/daily/monthly times. Are you suggesting that instead of invoking the library script from the cronjob, there's another way to trigger the charm functionality at the different intervals?

In my view - the juju action allows to install the cronjob/ update logrotate files manually by the operator if they don't want to wait for the cronjob trigger time.

======

> BTW, update_cron_daily_schedule is not only used in /etc/cron.daily/charm-
> logrotated, it may be triggered from
>
> src/actions -> update_cronjob -> cron.install_cronjob ->
> update_cron_daily_schedule
>
> The runner of these two behaviors may be different. Could you also confirm it?
>
> 1. from cron.daily
> 2. from juju actions

So even if this functionality is triggered from src/actions (with update_cronjob) or during execution of the install hook event handler or config changed handler, they utilize the same `CronHelper.install_cronjob` method which then runs update_cron_daily_schedule`. Therefore, the runner in both cases will be root since sudo is used in the crontab entry to call the library script.

cron.daily is set with the `logrotate-cronjob-frequency` config and is used both by the action and by the other event handlers of the charm to run the same `update_cron_daily_schedule` function.

53248e0... by Ashley James

Fix flaky functional test

- Use tenacity.Retrying while checking for file in
`test_reconfigure_cronjob_frequency`

Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Approve (continuous-integration)
Revision history for this message
🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack) wrote :
review: Approve (continuous-integration)
Revision history for this message
Tianqi Xiao (txiao) wrote :

Have you considered using a mature library (e.g. python-crontab [1]) to interact with crontab? I'm a bit worried that there are some consequences we have not considered when directly overwriting the cron file.

[1]: https://pypi.org/project/python-crontab/

review: Needs Information
Revision history for this message
Eric Chen (eric-chen) wrote :

I cannot fully understand your answer above. I want to know why we need to execute cron_lib.py regularly. But I didn't find the real reason in your explanation.

My major concern is

1. In general case, the library in charm is used under the juju framework. The design in logrotate breaks the common situation and we should know the real reason. Maybe the solution is not to copy the file, we should just avoid this behavior.

2. Sudo is a dangerous command, we should only use it when needed. We should also need to know how the executor gets the permission?

3. subprocess is always my last choice to handle the program. because it implies many assumptions of the environment. So I agree with Tianqi's comment, we should consider the mature library. Even when we copy the file, we should think about how to use shutil.move if possible. (In the previous commit, you didn't use shutil.move when there is no "sudo").

4. "Retry" is also something I don't like a lot. Please write down the reason why we cannot avoid it. If there is a bug that only happen with 30% possibility, it will be concealed by the "Retry" mechanism.

I suggest that you can write a spec to clarify all the things and purpose the proposal again.

review: Needs Fixing
Revision history for this message
Ashley James (dashmage) wrote :

Problem
-------
Currently when the logrotate-cronjob-frequency config option is set to daily, then the configured value for update-cron-daily-schedule (eg: set,08:00) is used to update the job execution time for the logrotated cron.daily job. This is done using the root user’s crontab (with `sudo crontab -u root <file_with_new_crontab_contents>`). But this is in conflict with the job that already exists in the system crontab (/etc/crontab) which means that the cron.daily job would run twice. It also

Fixes LP#2017798 [1]

Proposal
--------
For daily cronjobs, create a new cron file in /etc/cron.d where the charm will be able to manage the execution time without messing with the system crontab (/etc/crontab) or the root user’s crontab. It also prevents the charm from running commands with sudo.

I’m proposing to add a new file to /etc/cron.d instead of modifying the cron.daily execution time directly in the system crontab (/etc/crontab) because this would affect the execution of other scripts in cron.daily.

Changes performed by MP
-----------------------
- Use tenacity.retry while checking whether a file exists in the functional tests. This prevents failure in cases of performance issues on the CI server.

- 2 new files are now managed by the charm --> `cronjob_daily_command_path` which has the juju-run command that executes the lib_cron.py script and is called from the cronjob. `cronjob_daily_file` is the file being added in /etc/cron.d.

- In the `update_cron_daily_schedule` method
    - Renames the function to something more appropriate (`create_cron_daily_job`) since we're creating a new cron job and not updating the cron.daily job.
    - Replaces updating root user crontab with creating a new cron file in /etc/cron.d for daily cronjob

- Replace dangling `open` function calls with a context manager in `install_cronjob`.

- `cleanup_cronjob_files` also cleans up the new daily cronjob files.

I'm still working on these:
- Fix existing unit tests in order to not write directly to the crontab of the system where it's being run. Mock the required files which are being accessed.
- Re-write unit test for `create_cron_daily_job` method.

Other Notes
------------
I couldn’t use the python-crontab [2] library in order to manage the crontab changes since it does not support the random execution time syntax (00~45 08~09). This feature is not supported by the cron package which is used by Ubuntu in the first place [3]. A separate bug report has been raised for this issue [4].

-------------

[1]: https://bugs.launchpad.net/charm-logrotated/+bug/2017798
[2]: https://pypi.org/project/python-crontab/
[3]: https://packages.ubuntu.com/jammy/cron
[4]: https://bugs.launchpad.net/charm-logrotated/+bug/2019990

Revision history for this message
Mert Kirpici (mertkirpici) wrote :

I am a little bit late to the discussion here, but I think I found the reason for the self-installing cronjob design decision in this charm, that Eric, Ray, Ashley and I were wondering at different times.

Taking from the original bug report:

LP #1832296: Create functionality to manage logrotate files
https://bugs.launchpad.net/charm-logrotated/+bug/1832296

"""
- As some packages might change the logrotate configurations - a cronjob option needs to be defined, where it will constantly update the configurations with regards to the retention period defined
"""

So it is for making sure, our retention settings are taking precedence if any other package/operator makes changes to the /etc/logrotate.d/ files. While I am not sure that it would be the right place to do this, but i suppose we could move this functionality to the "update-status" hook, since it is fired periodically.

Unmerged commits

78a0422... by Ashley James

Remove cron daily job from root user crontab.

Currently while modifying cron daily job execution time, it is
updated in the root user's crontab (/var/spool/cron/crontabs/root).
This leads to a duplicate execution along with the execution of
cron.daily scripts in /etc/crontab.

This commit aims to manage the execution of the charms' daily cron job
solely from the system crontab, /etc/crontab. It also fixes the existing
unit test `test_cron_daily_schedule` and mocks the crontab instead of
directly writing to it.

53248e0... by Ashley James

Fix flaky functional test

- Use tenacity.Retrying while checking for file in
`test_reconfigure_cronjob_frequency`

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/src/lib/lib_cron.py b/src/lib/lib_cron.py
2index 1f30ddc..a314a55 100644
3--- a/src/lib/lib_cron.py
4+++ b/src/lib/lib_cron.py
5@@ -1,7 +1,6 @@
6 """Cron helper module."""
7 import os
8 import re
9-import subprocess
10
11 from charmhelpers.core import hookenv
12
13@@ -62,15 +61,14 @@ class CronHelper:
14 logrotate_unit = hookenv.local_unit()
15 python_venv_path = os.getcwd().replace("charm", "") + ".venv/bin/python3"
16 # upgrade to template if logic increases
17- cron_file = open(cron_file_path, "w")
18 cron_job = """#!/bin/bash
19 /usr/bin/sudo /usr/bin/juju-run {} "{} {}"
20 """.format(
21 logrotate_unit, python_venv_path, cronjob_path
22 )
23- cron_file.write(cron_job)
24- cron_file.close()
25- os.chmod(cron_file_path, 700)
26+ with open(cron_file_path, "w") as cron_file:
27+ cron_file.write(cron_job)
28+ os.chmod(cron_file_path, 0o744)
29
30 # update cron.daily schedule if logrotate-cronjob-frequency set to "daily"
31 if self.cronjob_frequency == 1 and self.validate_cron_conf():
32@@ -123,7 +121,16 @@ class CronHelper:
33 else:
34 raise RuntimeError("Unknown daily schedule type: {}".format(schedule_type))
35
36- cron_daily_timestamp = cron_daily_minute + " " + cron_daily_hour + "\t"
37+ cron_daily_timestamp = cron_daily_minute + " " + cron_daily_hour
38+ self.write_to_crontab(cron_daily_timestamp)
39+
40+ return cron_daily_timestamp
41+
42+ def write_to_crontab(self, cron_daily_timestamp, target="/etc/crontab"):
43+ """Write daily cronjob with provided timestamp to target crontab.
44+
45+ Default value for target is system crontab (/etc/crontab).
46+ """
47 cron_pattern = re.compile(r".*\/etc\/cron.daily.*")
48 with open(r"/etc/crontab", "r") as crontab:
49 data = crontab.read()
50@@ -132,15 +139,13 @@ class CronHelper:
51 updated_cron_daily = ""
52 if cron_daily:
53 updated_cron_daily = re.sub(
54- r"\d?\d(~\d\d)?\s\d?\d(~\d\d)?\t", cron_daily_timestamp, cron_daily[0]
55+ r"\d?\d(~\d\d)?\s\d?\d(~\d\d)?\t",
56+ cron_daily_timestamp + "\t",
57+ cron_daily[0],
58 )
59- data = data.replace(cron_daily[0], updated_cron_daily)
60- with open(r"/tmp/crontab", "w") as crontab:
61- crontab.write(data)
62-
63- subprocess.check_output(["sudo", "crontab", "-u", "root", "/tmp/crontab"])
64-
65- return updated_cron_daily
66+ updated_data = data.replace(cron_daily[0], updated_cron_daily)
67+ with open(target, "w") as crontab:
68+ crontab.write(updated_data)
69
70 def validate_cron_conf(self):
71 """Block the unit and exit the hook if there is invalid configuration."""
72diff --git a/src/tests/functional/requirements.txt b/src/tests/functional/requirements.txt
73index 6c4de97..4128a5d 100644
74--- a/src/tests/functional/requirements.txt
75+++ b/src/tests/functional/requirements.txt
76@@ -7,3 +7,4 @@ mock
77 pytest
78 pytest-asyncio
79 requests
80+tenacity
81diff --git a/src/tests/functional/test_logrotate.py b/src/tests/functional/test_logrotate.py
82index 0afbb82..7088371 100644
83--- a/src/tests/functional/test_logrotate.py
84+++ b/src/tests/functional/test_logrotate.py
85@@ -9,6 +9,8 @@ import pytest
86
87 import pytest_asyncio
88
89+import tenacity
90+
91 pytestmark = pytest.mark.asyncio
92 SERIES = ["bionic", "focal", "jammy"]
93
94@@ -118,10 +120,17 @@ async def test_reconfigure_cronjob_frequency(model, deploy_app, unit, jujutools)
95 await model.block_until(lambda: deploy_app.status == "active")
96 config = await deploy_app.get_config()
97
98- result = await jujutools.run_command(
99- "test -f /etc/cron.weekly/charm-logrotate", unit
100- )
101- weekly_cronjob_exists = result["return-code"] == 0
102+ # Retry because test fails sometimes when cron file isn't ready yet
103+ for attempt in tenacity.Retrying(
104+ stop=tenacity.stop_after_attempt(3),
105+ wait=tenacity.wait_exponential(multiplier=1, min=2, max=10),
106+ ):
107+ with attempt:
108+ result = await jujutools.run_command(
109+ "test -f /etc/cron.weekly/charm-logrotate", unit
110+ )
111+ weekly_cronjob_exists = result["return-code"] == 0
112+ assert weekly_cronjob_exists
113
114 result = await jujutools.run_command(
115 "test -f /etc/cron.daily/charm-logrotate", unit
116@@ -130,7 +139,6 @@ async def test_reconfigure_cronjob_frequency(model, deploy_app, unit, jujutools)
117
118 assert config["logrotate-cronjob-frequency"]["value"] == "weekly"
119 assert not daily_cronjob_exists
120- assert weekly_cronjob_exists
121
122
123 async def test_configure_override_01(model, deploy_app, jujutools, unit):
124diff --git a/src/tests/unit/test_logrotate.py b/src/tests/unit/test_logrotate.py
125index 8d329e5..ee136cd 100644
126--- a/src/tests/unit/test_logrotate.py
127+++ b/src/tests/unit/test_logrotate.py
128@@ -1,6 +1,7 @@
129 """Main unit test module."""
130 import json
131 import os
132+import re
133 from textwrap import dedent
134 from unittest import mock
135
136@@ -245,18 +246,40 @@ class TestCronHelper:
137 ("unset", "25 6"),
138 ],
139 )
140- def test_cron_daily_schedule(self, cron, cron_schedule, exp_pattern):
141+ def test_cron_daily_schedule(self, cron, cron_schedule, exp_pattern, mocker):
142 """Test the validate and update random cron.daily schedule."""
143 cron_config = cron()
144 cron_config.cronjob_enabled = True
145 cron_config.cronjob_frequency = 1
146 cron_config.cron_daily_schedule = cron_schedule
147
148- assert cron_config.validate_cron_conf()
149+ mock_method = mocker.Mock()
150+ mocker.patch.object(cron, "write_to_crontab", new=mock_method)
151
152 updated_cron_daily = cron_config.update_cron_daily_schedule()
153
154+ assert cron_config.validate_cron_conf()
155 assert updated_cron_daily.split("\t")[0] == exp_pattern
156+ mock_method.assert_called_once_with(exp_pattern)
157+
158+ @pytest.mark.parametrize("cron_daily_timestamp", ["00 08", "25 6", "00~50 06~07"])
159+ def test_write_to_crontab(self, cron, cron_daily_timestamp, tmp_path):
160+ """Test function that writes updated data to crontab file."""
161+ cron_config = cron()
162+ tmp_cronjob = tmp_path / "cron_file"
163+
164+ # write updated content into temp crontab file
165+ cron_config.write_to_crontab(cron_daily_timestamp, target=tmp_cronjob)
166+
167+ with open(tmp_cronjob) as crontab:
168+ data = crontab.read()
169+
170+ # get cron daily job from crontab data
171+ cron_pattern = re.compile(r".*\/etc\/cron.daily.*")
172+ cron_daily = cron_pattern.findall(data)
173+
174+ # assert if timestamps are equal
175+ assert cron_daily[0].split("\t")[0] == cron_daily_timestamp
176
177 @pytest.mark.parametrize(
178 ("cron_schedule"),
179@@ -292,9 +315,9 @@ class TestCronHelper:
180 return_value=os.path.join(mock_charm_dir, "lib/lib_cron.py"),
181 )
182 mocker.patch("lib_cron.os.getcwd", return_value=mock_charm_dir)
183- mock_open = mocker.patch("lib_cron.open")
184- mock_handle = mock_open.return_value
185-
186+ mock_open = mocker.mock_open()
187+ mocker.patch("lib_cron.open", mock_open)
188+ mock_handle = mock_open.return_value.__enter__.return_value
189 expected_files_to_be_removed = [
190 "/etc/cron.hourly/charm-logrotate",
191 "/etc/cron.daily/charm-logrotate",
192@@ -322,8 +345,7 @@ class TestCronHelper:
193 """ # noqa
194 )
195 )
196- mock_handle.close.assert_called_once()
197- mock_chmod.assert_called_once_with("/etc/cron.weekly/charm-logrotate", 700)
198+ mock_chmod.assert_called_once_with("/etc/cron.weekly/charm-logrotate", 0o744)
199
200 def test_install_cronjob_removes_etc_config_when_cronjob_disabled(
201 self, cron, mocker

Subscribers

People subscribed via source and target branches

to all changes: