Merge dkimpy-milter:dkg/socket-activation into dkimpy-milter:master

[dkg]

This series makes it possible to do socket activation. It is written on top of the earlier patch series to make it easier to consider merging in sequence (it is also quite close to several bugfixes)

Unmerged commits

7092874... by dkg on 2019-02-21

Enable sd_listen_fds(3)-style socket-activation support

I've added straightforward systemd unit files in
system/socket-activation/ that make use of this approach, and a
README.md in the same location that describes the tradeoffs.

25fdd3b... by dkg on 2019-02-21

Do not create PidFile by default

By default, avoid creating a PIDFile.

PIDFiles are racy and potentially dangerous. Modern system
supervision systems don't need them, because they manage the process
groups directly.

If the configuration file doesn't specify a PidFile, dkimpy-milter
shouldn't try to create one.

9d5316c... by dkg on 2019-02-21

Handle defaults for Socket differently

We want to be able to select the default for Socket differently in the
future.

This change augments the API for dkimpy_milter.util.own_socketfile()
by adding an optional sockname argument. This is a
backward-compatible change. If we aren't committed to API stability
for this function, we could make a more invasive change that would
probably be a more reasonable API going forward, but this is probably
good enough.

ea09bab... by dkg on 2019-02-21

Convert __init__.py to python3

The main work here is about bytes vs. strings. This work was
confusing for several reasons:

 * pymilter thinks that headers are all strings, but body is bytes

 * dkimpy wants to deal with bytes objects generally (though it
   accepts a string object as an ed25519 secret key for some reason,
   despite requiring bytes as an RSA secret key)

 * authres.AuthenticationResultsHeader object converts easily to a
   string, but has no direct bytes conversion. meanwhile, it wants
   its arguments as strings, but will accept them if they are bytes
   and convert them with something like str(), which leaves weird
   cruft like "header.a=b'ed25519-sha256'"

 * dkimpy_milter/utils.py contains fold() which expects bytes

 * self.fp needs to accumulate the on-the-wire version of the message
   as a whole (so it needs to be bytes). That means converting the
   headers. Header names and values are US-ASCII, per §2.2 of RFC
   5322, so they should be convertible cleanly, but we still have to
   convert them explicitly so that python knows the right thing to do.

At any rate, tests/runtests all passes with these changes, and the
output for both Authentication-Results: and DKIM-Signature headers
looks the same.

391b535... by dkg on 2019-02-20

Convert mostly to python3 (still need strings/bytes conversions)

This covers conversion of the whole project to python3, *except* for
the strings/bytes distinction in __init__.py, which i'm leaving for a
second commit.

The changes in this commit are intended to be relatively
uncontroversial, so that the following commit contains the tricky
bits.

ad8f396... by dkg on 2019-02-21

Expand test suite to cover RSA as well as ed25519

479820a... by dkg on 2019-02-19

tests: test DKIM signing and verification

This test makes use of DNSOverride and the new verifying milter to
ensure that signatures can be verified properly.

It doesn't test the actual interaction with the public DNS, but
getting that kind of test to work on arbitrary platforms might be more
trouble than it's worth.

I note that the DNSOverride only works as long as testkey.dns is a
single line, which is fine for ed25519, but maybe not for RSA.

7bfb87f... by dkg on 2019-02-19

Set up __main__.py, use it in tests

This allows us to invoke dkimpy-milter as "python -m dkimpy_milter
dkimpy-milter.conf", which makes running the test suite easier.

5c1d5d6... by dkg on 2019-02-19

tests: Run a verifying milter as well as a signing milter

Having a verifying milter will come in handy when we want to test both
sides of the DKIM process.

ae31730... by dkg on 2019-02-19

check for actions claimed by the filter

diff --git a/dkimpy_milter/__init__.py b/dkimpy_milter/__init__.py
index bcdf2a7..4c64891 100644
--- a/dkimpy_milter/__init__.py
+++ b/dkimpy_milter/__init__.py
@@ -354,7 +354,18 @@ def main():
     Milter.set_flags(Milter.CHGHDRS + Milter.ADDHDRS)
     miltername = 'dkimpy-filter'
     socketname = milterconfig.get('Socket')
-    own_socketfile(milterconfig)
+    if socketname is None:
+        if int(os.environ.get('LISTEN_PID', '0')) == os.getpid():
+            lfds = os.environ.get('LISTEN_FDS')
+            if lfds is not None:
+                if lfds != '1':
+                    syslog.syslog('LISTEN_FDS is set to "{0}", but we only know how to deal with "1", ignoring it'.
+                                  format(lfds))
+                else:
+                    socketname = 'fd:3'
+        if socketname is None:
+            socketname = 'local:/var/run/dkimpy-milter/dkimpy-milter.sock'
+    own_socketfile(milterconfig, socketname)
     drop_privileges(milterconfig)
     sys.stdout.flush()
     Milter.runmilter(miltername, socketname, 240)
diff --git a/dkimpy_milter/config.py b/dkimpy_milter/config.py
index bf6551a..3e2c736 100644
--- a/dkimpy_milter/config.py
+++ b/dkimpy_milter/config.py
@@ -39,8 +39,8 @@ defaultConfigData = {
     'SyslogFacility': 'mail',
     'UMask': 0o07,
     'Mode': 'sv',
-    'Socket': 'local:/var/run/dkimpy-milter/dkimpy-milter.sock',
-    'PidFile': '/var/run/dkimpy-milter/dkimpy-milter.pid',
+    'Socket': None,
+    'PidFile': None,
     'UserID': 'dkimpy-milter',
     'Canonicalization': 'relaxed/simple',
     'InternalHosts': '127.0.0.1',
diff --git a/dkimpy_milter/util.py b/dkimpy_milter/util.py
index 53d4c0d..bcdd11b 100644
--- a/dkimpy_milter/util.py
+++ b/dkimpy_milter/util.py
@@ -115,43 +115,49 @@ def write_pid(milterconfig):
     """Write PID in pidfile.  Will not overwrite an existing file."""
     import os
     import syslog
-    if not os.path.isfile(milterconfig.get('PidFile')):
+    pidfile = milterconfig.get('PidFile')
+    if pidfile is None:
+        return
+    if not os.path.isfile(pidfile):
         pid = str(os.getpid())
         try:
-            f = open(milterconfig.get('PidFile'), 'w')
+            f = open(pidfile, 'w')
         except IOError as e:
             if str(e)[:35] == '[Errno 2] No such file or directory':
-                piddir = milterconfig.get('PidFile').rsplit('/', 1)[0]
+                piddir = pidfile.rsplit('/', 1)[0]
                 os.mkdir(piddir)
                 user, group = user_group(milterconfig.get('UserID'))
                 os.chown(piddir, user, group)
-                f = open(milterconfig.get('PidFile'), 'w')
+                f = open(pidfile, 'w')
                 if milterconfig.get('Syslog'):
                     syslog.syslog('PID dir created: {0}'.format(piddir))
             else:
                 if milterconfig.get('Syslog'):
                     syslog.syslog('Unable to write pidfle {0}.  IOError: {1}'
-                                  .format(milterconfig.get('PidFile'), e))
+                                  .format(pidfile, e))
                 raise
         f.write(pid)
         f.close()
         user, group = user_group(milterconfig.get('UserID'))
-        os.chown(milterconfig.get('PidFile'), user, group)
+        os.chown(pidfile, user, group)
     else:
         if milterconfig.get('Syslog'):
             syslog.syslog('Unable to write pidfle {0}.  File exists.'
-                          .format(milterconfig.get('PidFile')))
+                          .format(pidfile))
         raise RuntimeError('Unable to write pidfle {0}.  File exists.'
-                           .format(milterconfig.get('PidFile')))
+                           .format(pidfile))
     return pid
 
 
-def own_socketfile(milterconfig):
+def own_socketfile(milterconfig, sockname=None):
     """If socket is Unix socket, chown to UserID before dropping privileges"""
     import os
     user, group = user_group(milterconfig.get('UserID'))
     offset = None
-    sockname = milterconfig.get('Socket')
+    if sockname is None:
+        sockname = milterconfig.get('Socket')
+    if sockname is None:
+        return
     if sockname[:1] == '/':
         offset = 0
     elif sockname[:6] == "local:":
diff --git a/man/dkimpy-milter.conf.5 b/man/dkimpy-milter.conf.5
index d0662c4..ea07593 100644
--- a/man/dkimpy-milter.conf.5
+++ b/man/dkimpy-milter.conf.5
@@ -355,7 +355,7 @@ will be checked. [PeerList NOT IMPLEMENTED - included for reference only]
 .TP
 .I PidFile (string)
 Specifies the path to a file that should be created at process start
-containing the process ID.
+containing the process ID.  If not specified, no such file will be created.
 
 .TP
 .I Selector (string)
diff --git a/system/socket-activation/README.md b/system/socket-activation/README.md
 new file mode 100644
index 0000000..b4410ac
--- /dev/null
+++ b/system/socket-activation/README.md
@@ -0,0 +1,32 @@
+This directory contains example systemd unit files for running a
+supervised, socket-activated instance of dkimpy-milter.
+
+There are several advantages of using socket activation:
+
+- dkimpy-milter never runs with elevated privileges, they are dropped
+  before any dkimpy-milter code is executed.
+   
+- The socket is opened before dkimpy-milter runs.  This means that
+  clients can connect() to the socket immediately.  So even if there
+  is a delay in dkimpy-milter startup, or in libmilter itself, the
+  connection will not fail.
+  
+- You can set the privileges of a listening Unix-domain socket by an
+  override of ListenGroup= in dkimpy-milter.socket (see
+  systemd.unit(5) for how to override).  This lets you control who has
+  access to the daemon with finer granularity than is available with
+  dkimpy-milter on its own.
+  
+- dkimpy-milter will not consume system resources if it is not used.
+
+- A fully-supervised dkimpy-milter needs no PIDFile, UMask, UserID, or
+  Socket configuation.  This eliminates common race conditions and
+  startup failures, and simplifies the resulting configuration file.
+  
+There is one downside to using socket activation:
+
+- it will only work on systems where libmilter can support connection
+  strings like "fd:3".  This has been supported on Debian and derived
+  systems since sendmail 8.14.4-6 (before Debian Jessie, in early
+  2014), see for example:
+  https://sources.debian.org/src/sendmail/8.15.2-8/debian/patches/socket_activation.patch/
diff --git a/system/socket-activation/dkimpy-milter.service b/system/socket-activation/dkimpy-milter.service
 new file mode 100644
index 0000000..2116bb7
--- /dev/null
+++ b/system/socket-activation/dkimpy-milter.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=DKIMpy Milter
+Documentation=man:dkimpy-milter(8) man:dkimpy-milter.conf(5)
+Requires=dkimpy-milter.socket
+
+[Service]
+ExecStart=/usr/bin/dkimpy-milter /etc/dkimpy-milter.conf
+User=dkimpy-milter
+
+[Install]
+Also=dkimpy-milter.socket
diff --git a/system/socket-activation/dkimpy-milter.socket b/system/socket-activation/dkimpy-milter.socket
 new file mode 100644
index 0000000..d4338a3
--- /dev/null
+++ b/system/socket-activation/dkimpy-milter.socket
@@ -0,0 +1,12 @@
+[Unit]
+Description=DKIMpy Milter socket
+Documentation=man:dkimpy-milter(8) man:dkimpy-milter.conf(5)
+
+[Socket]
+ListenStream=/run/dkimpy-milter/dkimpy-milter.sock
+SocketMode=0660
+# override SocketGroup to grant access to members of another system group:
+SocketGroup=dkimpy-milter
+
+[Install]
+WantedBy=sockets.target