Merge ~ogayot/curtin:systemd-offline into curtin:master

Proposed by Olivier Gayot
Status: Merged
Approved by: Olivier Gayot
Approved revision: 568b903a1dbef12519069b7022f541a889d95187
Merge reported by: Server Team CI bot
Merged at revision: not available
Proposed branch: ~ogayot/curtin:systemd-offline
Merge into: curtin:master
Diff against target: 115 lines (+80/-1)
2 files modified
curtin/util.py (+29/-1)
tests/unittests/test_util.py (+51/-0)
Reviewer Review Type Date Requested Status
Dan Bungert Approve
Server Team CI bot continuous-integration Approve
Chris Peterson Approve
Review via email: mp+462140@code.launchpad.net

Commit message

apt: ensure systemd knows it runs in a chroot, when executing postinst

Since we added the --mount-proc option to unshare, the postinst script
for openssh-server (and most likely other packages) started failing with
the following error when `systemctl daemon-reload` was invoked:

  > Failed to connect to bus: No data available

Before the option was added, it would simply do nothing because systemd
rightly understood it was running in a chroot.

To determine if we are running in a chroot, systemd checks if
/proc/1/root (corresponding to the init process) and / are the same
inode. If they are different, systemd assumes we are in a chroot.

However, we are running apt-get in a new PID namespace which means that
in the new namespace, apt-get gets assigned PID 1 and is therefore the
"init" process.

Now that /proc is properly mounted in the chroot, when systemd compares
/proc/1/root and /, it sees they are identical because the init process
(which is apt-get) is actually running inside the chroot.

Without the --mount-proc option, /proc/1 in the chroot would still refer
to the systemd init process (running outside the chroot), so it would
work properly.

With the SYSTEMD_OFFLINE variable, one can "force" systemd to assume
it is running in a chroot. Let's use it when running commands in a
chroot, and when the variable is not already defined.

LP: #2056570

Signed-off-by: Olivier Gayot <email address hidden>

Description of the change

Since we added the --mount-proc option to unshare, the postinst script for openssh-server (and most likely other packages) started failing with the following error when `systemctl daemon-reload` was invoked:

  > Failed to connect to bus: No data available

This would cause failed installations of Ubuntu. This happens because systemd fails to determine that we are running in a chroot when executed in the postinst script context.

To determine if we are running in a chroot, systemd checks if /proc/1/root (corresponding to the init process) and / are the same inode. If they are different, systemd assumes we are in a chroot.

However, we are running apt-get in a new PID namespace which means that in the new namespace, apt-get gets assigned PID 1 and is therefore the "init" process.

Now that /proc is properly mounted in the chroot, when systemd compares /proc/1/root and /, it sees they are identical because the init process (which is apt-get) is actually running inside the chroot.

Without the --mount-proc option, /proc/1 in the chroot would still refer to the systemd init process (running outside the chroot), so it would work properly.

With the SYSTEMD_OFFLINE variable, one can "force" systemd to assume it is running in a chroot. Let's use it.

To post a comment you must log in.
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Dan Bungert (dbungert) :
review: Needs Information
Revision history for this message
Dan Bungert (dbungert) wrote :

Also do we want this set more broadly than the apt-install code? I bet I can run into the same problem in late-commands.

Revision history for this message
Chris Peterson (cpete) wrote :

Nice work! Looks good to me. I was slightly confused to see SYSTEMD_OFFLINE='true' work since the docs says 0 or 1, but it looks like systemd will check for truthy values too.

review: Approve
Revision history for this message
Chris Peterson (cpete) :
Revision history for this message
Olivier Gayot (ogayot) wrote :

Thanks for the comments, Dan and Chris! I updated the MP accordingly.

SYSTEMD_OFFLINE will now take values '0' or '1' instead of 'true'. Originally I found out about this variable while reading the systemd code and I didn't look at the documentation. But, I agree that it is better to follow the doc.

I also liked the idea of `systemd_force_offline` defaulting to None so I implemented the change.

Lastly, I moved the systemd_force_offline variable to a more global location in `curtin.util.subp`. The logic is as follows now:

* if systemd_force_offline is True or False, then SYSTEMD_OFFLINE` is set accordingly in the environment (any previous value is overridden).
* if systemd_force_offline is None, then SYSTEMD_OFFLINE is set to '1' if and only if the two following conditions are met:
  * We are executing the command in a chroot (i.e., target is not /)
  * The SYSTEMD_OFFLINE variable is not present in the environment

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Dan Bungert (dbungert) wrote :

LGTM

yes, 'true'/'false' should be fine and '1'/'0' has a slight edge due to being the documented interface. Thanks!

Revision history for this message
Dan Bungert (dbungert) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/curtin/util.py b/curtin/util.py
index 58ee7e1..dcf9422 100644
--- a/curtin/util.py
+++ b/curtin/util.py
@@ -73,12 +73,36 @@ class NotExclusiveError(OSError):
73def _subp(args, data=None, rcs=None, env=None, capture=False,73def _subp(args, data=None, rcs=None, env=None, capture=False,
74 combine_capture=False, shell=False, logstring=False,74 combine_capture=False, shell=False, logstring=False,
75 decode="replace", target=None, cwd=None, log_captured=False,75 decode="replace", target=None, cwd=None, log_captured=False,
76 unshare_pid=None):76 unshare_pid=None,
77 *, systemd_force_offline: Optional[bool] = None):
77 if rcs is None:78 if rcs is None:
78 rcs = [0]79 rcs = [0]
79 devnull_fp = None80 devnull_fp = None
8081
81 tpath = paths.target_path(target)82 tpath = paths.target_path(target)
83
84 env = env.copy() if env is not None else os.environ.copy()
85 # To determine if we are running in a chroot, systemd checks if
86 # /proc/1/root (corresponding to the init process) and / are the same
87 # inode. If they are different, systemd assumes we are in a chroot.
88 # However, we are running apt-get in a new PID namespace (with /proc
89 # properly mounted). This means that in the new namespace, apt-get gets
90 # assigned PID 1 and is therefore the "init" process.
91 # When systemd compares /proc/1/root and /, it sees they are identical
92 # because the init process is actually running in the chroot.
93 #
94 # Before we started passing the --mount-proc option to unshare, it was
95 # working because /proc/1 in the chroot would still refer to the systemd
96 # init process (running outside the chroot).
97 #
98 # With the SYSTEMD_OFFLINE variable, one can "force" systemd to assume it
99 # is running in a chroot. Let's use it.
100 if systemd_force_offline is not None:
101 # Override the SYSTEMD_OFFLINE variable even if it already exists.
102 env['SYSTEMD_OFFLINE'] = str(int(systemd_force_offline))
103 elif 'SYSTEMD_OFFLINE' not in env and tpath != "/":
104 env['SYSTEMD_OFFLINE'] = '1'
105
82 chroot_args = [] if tpath == "/" else ['chroot', target]106 chroot_args = [] if tpath == "/" else ['chroot', target]
83 sh_args = ['sh', '-c'] if shell else []107 sh_args = ['sh', '-c'] if shell else []
84 if isinstance(args, string_types):108 if isinstance(args, string_types):
@@ -260,6 +284,10 @@ def subp(*args, **kwargs):
260 unshare the pid namespace.284 unshare the pid namespace.
261 default value (None) is to unshare pid namespace if possible285 default value (None) is to unshare pid namespace if possible
262 and target != /286 and target != /
287 :param systemd_force_offline:
288 if not None, will set the SYSTEMD_OFFLINE env variable to '1' or '0'
289 if None, the variable will be set to '1' only if running in a chroot
290 (i.e., target is not /) and the variable is not already set.
263291
264 :return292 :return
265 if not capturing, return is (None, None)293 if not capturing, return is (None, None)
diff --git a/tests/unittests/test_util.py b/tests/unittests/test_util.py
index 2dd63a7..96f260f 100644
--- a/tests/unittests/test_util.py
+++ b/tests/unittests/test_util.py
@@ -350,6 +350,57 @@ class TestSubp(CiTestCase):
350 ['apt-get', 'install'])350 ['apt-get', 'install'])
351 self.assertEqual(expected, args[0])351 self.assertEqual(expected, args[0])
352352
353 @mock.patch.dict(os.environ, clear=True)
354 @mock.patch('curtin.util.subprocess.Popen', side_effect=OSError)
355 def test_systemd_offline_default__no_chroot(self, m_popen):
356 with self.assertRaises(util.ProcessExecutionError):
357 util.subp(['ls', '-l'], target='/')
358 m_popen.assert_called_once()
359 self.assertEqual(m_popen.call_args.kwargs['env'], {})
360
361 @mock.patch('curtin.util.subprocess.Popen', side_effect=OSError)
362 def test_systemd_offline_default__in_chroot(self, m_popen):
363 with mock.patch.dict(os.environ, clear=True):
364 with self.assertRaises(util.ProcessExecutionError):
365 util.subp(['ls', '-l'], target='/target')
366 m_popen.assert_called_once()
367 self.assertEqual(m_popen.call_args.kwargs['env'],
368 {'SYSTEMD_OFFLINE': '1'})
369
370 @mock.patch('curtin.util.subprocess.Popen', side_effect=OSError)
371 def test_systemd_offline_default__no_override(self, m_popen):
372 with mock.patch.dict(os.environ, {'SYSTEMD_OFFLINE': '1'}, clear=True):
373 with self.assertRaises(util.ProcessExecutionError):
374 util.subp(['ls', '-l'], target='/target')
375 m_popen.assert_called_once()
376 self.assertEqual(m_popen.call_args.kwargs['env'],
377 {'SYSTEMD_OFFLINE': '1'})
378
379 m_popen.reset_mock()
380 with mock.patch.dict(os.environ, {'SYSTEMD_OFFLINE': '0'}, clear=True):
381 with self.assertRaises(util.ProcessExecutionError):
382 util.subp(['ls', '-l'], target='/target')
383 m_popen.assert_called_once()
384 self.assertEqual(m_popen.call_args.kwargs['env'],
385 {'SYSTEMD_OFFLINE': '0'})
386
387 @mock.patch('curtin.util.subprocess.Popen', side_effect=OSError)
388 def test_systemd_offline_specified(self, m_popen):
389 with mock.patch.dict(os.environ, {'SYSTEMD_OFFLINE': '0'}, clear=True):
390 with self.assertRaises(util.ProcessExecutionError):
391 util.subp(['ls', '-l'], systemd_force_offline=True)
392 m_popen.assert_called_once()
393 self.assertEqual(m_popen.call_args.kwargs['env'],
394 {'SYSTEMD_OFFLINE': '1'})
395
396 m_popen.reset_mock()
397 with mock.patch.dict(os.environ, {'SYSTEMD_OFFLINE': '1'}, clear=True):
398 with self.assertRaises(util.ProcessExecutionError):
399 util.subp(['ls', '-l'], systemd_force_offline=False)
400 m_popen.assert_called_once()
401 self.assertEqual(m_popen.call_args.kwargs['env'],
402 {'SYSTEMD_OFFLINE': '0'})
403
353404
354class TestGetUnsharePidArgs(CiTestCase):405class TestGetUnsharePidArgs(CiTestCase):
355 """Test the internal implementation for when to unshare."""406 """Test the internal implementation for when to unshare."""

Subscribers

People subscribed via source and target branches