Merge lp:~jmwilson/duplicity/capabilities into lp:~duplicity-team/duplicity/0.7-series

Proposed by James Wilson
Status: Superseded
Proposed branch: lp:~jmwilson/duplicity/capabilities
Merge into: lp:~duplicity-team/duplicity/0.7-series
Diff against target: 72 lines (+22/-5)
4 files modified
README (+1/-0)
bin/duplicity (+13/-5)
debian/control (+1/-0)
debian/duplicity.postinst (+7/-0)
To merge this branch: bzr merge lp:~jmwilson/duplicity/capabilities
Reviewer Review Type Date Requested Status
duplicity-team Pending
Review via email: mp+257488@code.launchpad.net

This proposal has been superseded by a proposal from 2015-08-03.

Description of the change

Proposal is to add an unprivileged "duplicity" user during installation that is used to limit the capabilities when duplicity is run as root. To manage capabilities, I'm using the python bindings for libcap-ng (which needs its own updating since at least the current vivid package is empty for some reason, but is correct when built from source).

I've been interested in using duplicity for system-wide backups, but one thing that is troubling is that it must be run as root. As an interpreted program that communicates on the network, this exposes the host to possible bugs in python or any other packages imported in duplicity.

First, examining the code in bin/duplicity:
    # if python is run setuid, it's only partway set,
    # so make sure to run with euid/egid of root
    if os.geteuid() == 0:
        # make sure uid/gid match euid/egid
        os.setuid(os.geteuid())
        os.setgid(os.getegid())

I'm not sure what this is for; if you're root (i.e., os.geteuid() == 0) then there's no need to switch the real uid. When the machination of "setuid(geteuid())" is used it is typically when the ruid=0 and we want to irrevocably drop privileges to a euid!=0 normal user. That's not the case here, so this doesn't help or harm us.

Then there's the issue of running duplicity as root. Since it's an interpreted program, the normal ways of expanding privileges (SUID executable or setcap on the script) are unavailable, and the other options are to run it as root, or put SUID or file capabilities on the python interpreter.

The only reason to run duplicity as root is get read access to the whole file system. We can safely drop all capabilities other than CAP_DAC_READ_SEARCH. Since we're still root, we'll get all capabilities back if we do execve, so we could also change the bounding set or lock the securebits to prevent the kernel from re-granting privileges on execve. Nonetheless, we're still root, and lots of important files are owned by and writable to root, so the best choice is to change uid to an unprivileged user who maintains only the ability to read the whole file system.

It's hard to test the change directly, since it doesn't change the output or actions of duplicity. The following python script mimics what the code does and demonstrates the reduction of capabilities:

#!/usr/bin/env python

from __future__ import print_function
from capng import *
import fcntl
import os
import pwd
import signal
import sys

if os.geteuid() != 0:
    sys.exit()

user = pwd.getpwnam("nobody")
capng_clear(CAPNG_SELECT_CAPS)
capng_update(CAPNG_ADD, CAPNG_EFFECTIVE | CAPNG_PERMITTED, CAP_DAC_READ_SEARCH)
capng_change_id(user.pw_uid, user.pw_gid, CAPNG_DROP_SUPP_GRP)

print("getuid = {}, geteuid = {}".format(os.getuid(), os.geteuid()))
print("After dropping capabilities:")
capng_get_caps_process()
capng_print_caps_numeric(CAPNG_PRINT_STDOUT, CAPNG_SELECT_BOTH)
print()

pread, pwrite = os.pipe()
pid = os.fork()
if pid == 0:
    os.close(pread)
    fcntl.fcntl(pwrite, fcntl.F_SETFD, fcntl.FD_CLOEXEC)
    os.execl('/usr/bin/tail', '-f', '/dev/null')
else:
    os.close(pwrite)
    os.read(pread, 1)
    capng_setpid(pid)
    capng_get_caps_process()
    print("getuid = {}, geteuid = {}".format(os.getuid(), os.geteuid()))
    print("Capabilities after execve:")
    caps = capng_print_caps_numeric(CAPNG_PRINT_STDOUT, CAPNG_SELECT_BOTH)
    os.kill(pid, signal.SIGTERM)

which when run as root (sudo python script.py) should produce this output:
getuid = 65534, geteuid = 65534
After dropping capabilities:
Effective: 00000000, 00000004
Permitted: 00000000, 00000004
Inheritable: 00000000, 00000000
Bounding Set: 0000003F, FFFFFFFF

getuid = 65534, geteuid = 65534
Capabilities after execve:
Effective: 00000000, 00000000
Permitted: 00000000, 00000000
Inheritable: 00000000, 00000000
Bounding Set: 0000003F, FFFFFFFF

To post a comment you must log in.
Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :
Download full text (7.6 KiB)

If I understand this correctly, then new files will be created by
'duplicity', but older files owned by root will not be deletable.
Correct? I'm thinking a one-time 'chown -R duplicity: ~/cache/duplicity/*'
would fix that problem.

On Sun, Apr 26, 2015 at 7:31 PM, James Wilson <email address hidden> wrote:

> James Wilson has proposed merging lp:~jmwilson/duplicity/capabilities into
> lp:duplicity.
>
> Requested reviews:
> duplicity-team (duplicity-team)
>
> For more details, see:
> https://code.launchpad.net/~jmwilson/duplicity/capabilities/+merge/257488
>
> Proposal is to add an unprivileged "duplicity" user during installation
> that is used to limit the capabilities when duplicity is run as root. To
> manage capabilities, I'm using the python bindings for libcap-ng (which
> needs its own updating since at least the current vivid package is empty
> for some reason, but is correct when built from source).
>
> I've been interested in using duplicity for system-wide backups, but one
> thing that is troubling is that it must be run as root. As an interpreted
> program that communicates on the network, this exposes the host to possible
> bugs in python or any other packages imported in duplicity.
>
> First, examining the code in bin/duplicity:
> # if python is run setuid, it's only partway set,
> # so make sure to run with euid/egid of root
> if os.geteuid() == 0:
> # make sure uid/gid match euid/egid
> os.setuid(os.geteuid())
> os.setgid(os.getegid())
>
> I'm not sure what this is for; if you're root (i.e., os.geteuid() == 0)
> then there's no need to switch the real uid. When the machination of
> "setuid(geteuid())" is used it is typically when the ruid=0 and we want to
> irrevocably drop privileges to a euid!=0 normal user. That's not the case
> here, so this doesn't help or harm us.
>
> Then there's the issue of running duplicity as root. Since it's an
> interpreted program, the normal ways of expanding privileges (SUID
> executable or setcap on the script) are unavailable, and the other options
> are to run it as root, or put SUID or file capabilities on the python
> interpreter.
>
> The only reason to run duplicity as root is get read access to the whole
> file system. We can safely drop all capabilities other than
> CAP_DAC_READ_SEARCH. Since we're still root, we'll get all capabilities
> back if we do execve, so we could also change the bounding set or lock the
> securebits to prevent the kernel from re-granting privileges on execve.
> Nonetheless, we're still root, and lots of important files are owned by and
> writable to root, so the best choice is to change uid to an unprivileged
> user who maintains only the ability to read the whole file system.
>
> It's hard to test the change directly, since it doesn't change the output
> or actions of duplicity. The following python script mimics what the code
> does and demonstrates the reduction of capabilities:
>
> #!/usr/bin/env python
>
> from __future__ import print_function
> from capng import *
> import fcntl
> import os
> import pwd
> import signal
> import sys
>
> if os.geteuid() != 0:
> sys.exit()
>
> user = pwd.getpwnam("nobody")
> capng_clear...

Read more...

Revision history for this message
edso (ed.so) wrote :
Download full text (8.0 KiB)

looks like capng is difficult to retrieve. if so i would hesitate to make duplicity depend on it by default.

does it work with older python? we officially support python 2.6+ still.

..ede/duply.net

On 27.04.2015 15:15, Kenneth Loafman wrote:
> If I understand this correctly, then new files will be created by
> 'duplicity', but older files owned by root will not be deletable.
> Correct? I'm thinking a one-time 'chown -R duplicity: ~/cache/duplicity/*'
> would fix that problem.
>
>
> On Sun, Apr 26, 2015 at 7:31 PM, James Wilson <email address hidden> wrote:
>
>> James Wilson has proposed merging lp:~jmwilson/duplicity/capabilities into
>> lp:duplicity.
>>
>> Requested reviews:
>> duplicity-team (duplicity-team)
>>
>> For more details, see:
>> https://code.launchpad.net/~jmwilson/duplicity/capabilities/+merge/257488
>>
>> Proposal is to add an unprivileged "duplicity" user during installation
>> that is used to limit the capabilities when duplicity is run as root. To
>> manage capabilities, I'm using the python bindings for libcap-ng (which
>> needs its own updating since at least the current vivid package is empty
>> for some reason, but is correct when built from source).
>>
>> I've been interested in using duplicity for system-wide backups, but one
>> thing that is troubling is that it must be run as root. As an interpreted
>> program that communicates on the network, this exposes the host to possible
>> bugs in python or any other packages imported in duplicity.
>>
>> First, examining the code in bin/duplicity:
>> # if python is run setuid, it's only partway set,
>> # so make sure to run with euid/egid of root
>> if os.geteuid() == 0:
>> # make sure uid/gid match euid/egid
>> os.setuid(os.geteuid())
>> os.setgid(os.getegid())
>>
>> I'm not sure what this is for; if you're root (i.e., os.geteuid() == 0)
>> then there's no need to switch the real uid. When the machination of
>> "setuid(geteuid())" is used it is typically when the ruid=0 and we want to
>> irrevocably drop privileges to a euid!=0 normal user. That's not the case
>> here, so this doesn't help or harm us.
>>
>> Then there's the issue of running duplicity as root. Since it's an
>> interpreted program, the normal ways of expanding privileges (SUID
>> executable or setcap on the script) are unavailable, and the other options
>> are to run it as root, or put SUID or file capabilities on the python
>> interpreter.
>>
>> The only reason to run duplicity as root is get read access to the whole
>> file system. We can safely drop all capabilities other than
>> CAP_DAC_READ_SEARCH. Since we're still root, we'll get all capabilities
>> back if we do execve, so we could also change the bounding set or lock the
>> securebits to prevent the kernel from re-granting privileges on execve.
>> Nonetheless, we're still root, and lots of important files are owned by and
>> writable to root, so the best choice is to change uid to an unprivileged
>> user who maintains only the ability to read the whole file system.
>>
>> It's hard to test the change directly, since it doesn't change the output
>> or actions of duplicity. The following python script mimics what t...

Read more...

Revision history for this message
Michael Terry (mterry) wrote :

Also, this should fallback gracefully if the 'duplicity' user doesn't exist.

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :
Download full text (8.7 KiB)

I cannot find python-capng for Ubuntu 14.04, Python 2.7, so I'm guessing
it's a new critter.

On Mon, Apr 27, 2015 at 8:33 AM, edso <email address hidden> wrote:

> looks like capng is difficult to retrieve. if so i would hesitate to make
> duplicity depend on it by default.
>
> does it work with older python? we officially support python 2.6+ still.
>
> ..ede/duply.net
>
> On 27.04.2015 15:15, Kenneth Loafman wrote:
> > If I understand this correctly, then new files will be created by
> > 'duplicity', but older files owned by root will not be deletable.
> > Correct? I'm thinking a one-time 'chown -R duplicity:
> ~/cache/duplicity/*'
> > would fix that problem.
> >
> >
> > On Sun, Apr 26, 2015 at 7:31 PM, James Wilson <email address hidden> wrote:
> >
> >> James Wilson has proposed merging lp:~jmwilson/duplicity/capabilities
> into
> >> lp:duplicity.
> >>
> >> Requested reviews:
> >> duplicity-team (duplicity-team)
> >>
> >> For more details, see:
> >>
> https://code.launchpad.net/~jmwilson/duplicity/capabilities/+merge/257488
> >>
> >> Proposal is to add an unprivileged "duplicity" user during installation
> >> that is used to limit the capabilities when duplicity is run as root. To
> >> manage capabilities, I'm using the python bindings for libcap-ng (which
> >> needs its own updating since at least the current vivid package is empty
> >> for some reason, but is correct when built from source).
> >>
> >> I've been interested in using duplicity for system-wide backups, but one
> >> thing that is troubling is that it must be run as root. As an
> interpreted
> >> program that communicates on the network, this exposes the host to
> possible
> >> bugs in python or any other packages imported in duplicity.
> >>
> >> First, examining the code in bin/duplicity:
> >> # if python is run setuid, it's only partway set,
> >> # so make sure to run with euid/egid of root
> >> if os.geteuid() == 0:
> >> # make sure uid/gid match euid/egid
> >> os.setuid(os.geteuid())
> >> os.setgid(os.getegid())
> >>
> >> I'm not sure what this is for; if you're root (i.e., os.geteuid() == 0)
> >> then there's no need to switch the real uid. When the machination of
> >> "setuid(geteuid())" is used it is typically when the ruid=0 and we want
> to
> >> irrevocably drop privileges to a euid!=0 normal user. That's not the
> case
> >> here, so this doesn't help or harm us.
> >>
> >> Then there's the issue of running duplicity as root. Since it's an
> >> interpreted program, the normal ways of expanding privileges (SUID
> >> executable or setcap on the script) are unavailable, and the other
> options
> >> are to run it as root, or put SUID or file capabilities on the python
> >> interpreter.
> >>
> >> The only reason to run duplicity as root is get read access to the whole
> >> file system. We can safely drop all capabilities other than
> >> CAP_DAC_READ_SEARCH. Since we're still root, we'll get all capabilities
> >> back if we do execve, so we could also change the bounding set or lock
> the
> >> securebits to prevent the kernel from re-granting privileges on execve.
> >> Nonetheless, we're still root, and lots of important files are owned by
...

Read more...

Revision history for this message
James Wilson (jmwilson) wrote :

Oops, the package is python-cap-ng. The problem is that the distro version is missing the goods. I think they forgot to include a dependency in their control file, since if I grab the source it builds fine and I didn't have to do anything. It's tracked in https://bugs.launchpad.net/ubuntu/+source/libcap-ng/+bug/1244384 and yeah it looks like they've been sitting on it for quite a while.

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :

I marked this as a "Work in Progress" because I don't want to forget it, but it sounds like it would be a support nightmare without distro support or pip support. Perhaps someone will put together a PPA with the appropriate packaging, but for now, we'll just have to wait.

Revision history for this message
James Wilson (jmwilson) wrote :

Hey, I wanted to resume this discussion. Since April a couple of things have changed: 1) I helped the maintainer of libcap-ng to fix its build issues as of 0.7.4-3 that is available in sid and stretch, and 2) I found another way to limit root capabilities using file capabilities that has a smaller footprint (it doesn't create a new system user).

The catch for #2 is that file capabilities need a real binary executable, because the kernel ignores file capabilities set on interpreted programs. I found a utility called cx_Freeze that uses a minimal binary loader to start up the python environment and run a bundled script. Then we can set inherited file capabilities on /usr/bin/duplicity in the post-install script. The inherited capabilities cannot be harnessed unless the executing user also has the same capabilities in their inherited set, which the system administrator can take care of by granting inherited capabilities on the backup user. The easiest way to do this is using pam_cap.so. One possible issue is that cx_Freeze is available as an ubuntu package (cx-freeze) but it is not in Debian.

I've been running a custom version of duplicity with this setup for several weeks. It's a cleaner solution since it doesn't require a specific duplicity user to exist. The tradeoff is that system-wide backups require more setup since the system administrator will need to set up the inherited user capabilities. It's worth adding some documentation on how to do all of this.

Are either of these directions something you'd like to consider now?

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :

I've run cx-freeze as well and I like it (gsutil on Freebsd). From the
standpoint of isolation it would give us the ability to supply all versions
of all needed modules as well as the Python version itself (for those that
refuse to upgrade). The cost is that the package becomes huge, but it's
just tradeoff, our support time vs a bit of their storage space.

What we might consider doing is both a frozen and a non-frozen version.
The users still on Python 2.4 (gack!) could upgrade to the latest and
greatest by using the frozen version, while the more advanced could upgrade
normally. I know it works on all Linux distros and Freebsd.

As to file capabilities, that will depend on what kernel version they are
running and whether it's baked into the filesystem. I suppose the code
could check for availability, use if possible and operate as normal if not.

On Fri, Jun 5, 2015 at 12:27 PM, James Wilson <email address hidden> wrote:

> Hey, I wanted to resume this discussion. Since April a couple of things
> have changed: 1) I helped the maintainer of libcap-ng to fix its build
> issues as of 0.7.4-3 that is available in sid and stretch, and 2) I found
> another way to limit root capabilities using file capabilities that has a
> smaller footprint (it doesn't create a new system user).
>
> The catch for #2 is that file capabilities need a real binary executable,
> because the kernel ignores file capabilities set on interpreted programs. I
> found a utility called cx_Freeze that uses a minimal binary loader to start
> up the python environment and run a bundled script. Then we can set
> inherited file capabilities on /usr/bin/duplicity in the post-install
> script. The inherited capabilities cannot be harnessed unless the executing
> user also has the same capabilities in their inherited set, which the
> system administrator can take care of by granting inherited capabilities on
> the backup user. The easiest way to do this is using pam_cap.so. One
> possible issue is that cx_Freeze is available as an ubuntu package
> (cx-freeze) but it is not in Debian.
>
> I've been running a custom version of duplicity with this setup for
> several weeks. It's a cleaner solution since it doesn't require a specific
> duplicity user to exist. The tradeoff is that system-wide backups require
> more setup since the system administrator will need to set up the inherited
> user capabilities. It's worth adding some documentation on how to do all of
> this.
>
> Are either of these directions something you'd like to consider now?
> --
> https://code.launchpad.net/~jmwilson/duplicity/capabilities/+merge/257488
> You are subscribed to branch lp:duplicity.
>

Unmerged revisions

1089. By James Wilson

Drop capabilities and change user when duplicity is run as root.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'README'
2--- README 2014-10-18 19:44:29 +0000
3+++ README 2015-04-27 00:30:33 +0000
4@@ -23,6 +23,7 @@
5 * librsync v0.9.6 or later
6 * GnuPG v1.x for encryption
7 * python-lockfile for concurrency locking
8+ * python-capng for managing capabilities when run as root
9 * for scp/sftp -- python-paramiko and python-pycryptopp
10 * for ftp -- lftp version 3.7.15 or later
11 * Boto 2.0 or later for single-processing S3 or GCS access (default)
12
13=== modified file 'bin/duplicity'
14--- bin/duplicity 2015-03-09 18:50:58 +0000
15+++ bin/duplicity 2015-04-27 00:30:33 +0000
16@@ -38,6 +38,8 @@
17 import resource
18 import re
19 import threading
20+import capng
21+import pwd
22 from datetime import datetime
23 from lockfile import FileLock
24
25@@ -1340,12 +1342,18 @@
26 See https://bugs.launchpad.net/duplicity/+bug/931175
27 """), log.ErrorCode.pythonoptimize_set)
28
29- # if python is run setuid, it's only partway set,
30- # so make sure to run with euid/egid of root
31+ # if python is running as root, then drop all capabilities except
32+ # unrestricted read access and then change user to prevent regaining
33+ # capabilities via execve.
34 if os.geteuid() == 0:
35- # make sure uid/gid match euid/egid
36- os.setuid(os.geteuid())
37- os.setgid(os.getegid())
38+ user = pwd.getpwnam("duplicity")
39+ capng.capng_clear(capng.CAPNG_SELECT_CAPS)
40+ if (capng.capng_update(capng.CAPNG_ADD,
41+ capng.CAPNG_EFFECTIVE | capng.CAPNG_PERMITTED,
42+ capng.CAP_DAC_READ_SEARCH)
43+ or capng.capng_change_id(user.pw_uid, user.pw_gid,
44+ capng.CAPNG_DROP_SUPP_GRP)):
45+ log.FatalError("Unable to drop root privileges.")
46
47 # set the current time strings (make it available for command line processing)
48 dup_time.setcurtime()
49
50=== modified file 'debian/control'
51--- debian/control 2014-10-27 14:15:52 +0000
52+++ debian/control 2015-04-27 00:30:33 +0000
53@@ -27,6 +27,7 @@
54 gnupg,
55 python-lockfile,
56 python-pexpect,
57+ python-capng,
58 Suggests: ncftp,
59 python-boto,
60 python-paramiko,
61
62=== added file 'debian/duplicity.postinst'
63--- debian/duplicity.postinst 1970-01-01 00:00:00 +0000
64+++ debian/duplicity.postinst 2015-04-27 00:30:33 +0000
65@@ -0,0 +1,7 @@
66+#!/bin/sh -e
67+
68+if [ "$1" = "configure" ]; then
69+ if ! getent passwd duplicity >/dev/null; then
70+ adduser --quiet --system --no-create-home --home /nonexistant --shell /usr/sbin/nologin duplicity
71+ fi
72+fi

Subscribers

People subscribed via source and target branches