Merge lp:~stephen-xemacs/mailman/sprint-2012-overview into lp:mailman

[Stephen Turnbull]

Andrea Crotti's work making python setup.py build_sphinx work, plus my "Overview" document.
This branch contains and supersedes lp:~andrea-crotti-o/mailman/conf_py.

To build the docs on my Mac OS X/Macports system I used

$ python2.6 bootstrap.py # get python 2.6 from MacPorts
$ bin/buildout
$ python2.6 setup.py build_sphinx

The output ends up in src/mailman/docs/build/sphinx/html/README.html. No promises for
other platforms. :-p

New files in src/mailman/docs:
  8-miles-high.rst # ReST file containing notes from Mailman sprint at PyCon 2012
  chains.png # used by 8-miles-high:
  pipeline.png # from Barry Warsaw's Mailman presentation
  msg-flow.png # replaced by in-line graphviz code (will be removed)
Other changes to (1) fix build, and (2) integrate 8-miles-high into README.rst.

[lp:~stephen-xemacs/mailman/sprint-2012-overview]

7117. By Stephen Turnbull on 2012-03-14

Improved layout of graphviz version of chains.

7118. By Stephen Turnbull on 2012-03-15

source, chains, sink subgraphs form

7119. By Stephen Turnbull on 2012-03-15

groups help

7120. By Stephen Turnbull on 2012-03-15

We gots a graphviz! Now WAT?!

7121. By Stephen Turnbull on 2012-03-15

Remove redundant PNGs.

7122. By Stephen Turnbull on 2012-03-15

Use minlen to disentagle the POSTING queue.

=== modified file 'README.rst'
--- README.rst	2012-03-13 01:03:35 +0000
+++ README.rst	2012-03-15 22:29:19 +0000
@@ -37,6 +37,7 @@
     src/mailman/docs/START
     src/mailman/docs/DATABASE
     src/mailman/docs/MTA
+    src/mailman/docs/8-miles-high
     src/mailman/bin/docs/*
     src/mailman/commands/docs/*
     src/mailman/database/docs/*
 
=== removed directory '_static'
=== removed directory '_templates'
=== modified file 'conf.py'
--- conf.py	2012-02-12 15:01:07 +0000
+++ conf.py	2012-03-15 22:29:19 +0000
@@ -25,10 +25,12 @@
 
 # Add any Sphinx extension module names here, as strings. They can be extensions
 # coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
-extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode']
+extensions = ['sphinx.ext.autodoc',
+              'sphinx.ext.viewcode',
+              'sphinx.ext.graphviz']
 
 # Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
+# templates_path = ['_templates']
 
 # The suffix of source filenames.
 source_suffix = '.rst'
@@ -122,7 +124,7 @@
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
+# html_static_path = ['_static']
 
 # If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
 # using the given strftime format.
@@ -222,6 +224,7 @@
     import errno
     cwd = os.getcwd()
     try:
+        os.makedirs('build/sphinx/html')
         os.chdir('build/sphinx/html')
         os.symlink('README.html', 'index.html')
         print 'index.html -> README.html'
 
=== added file 'src/mailman/docs/8-miles-high.rst'
--- src/mailman/docs/8-miles-high.rst	1970-01-01 00:00:00 +0000
+++ src/mailman/docs/8-miles-high.rst	2012-03-15 22:29:19 +0000
@@ -0,0 +1,190 @@
+========================================
+Notes from the PyCon 2012 Mailman Sprint
+========================================
+
+.. authorship
+   The notes are based on Barry Warsaw's description of the Mailman 3
+   pipeline at the PyCon 2012 Mailman sprint on March 13, with
+   diagrams from his "Mailman" presentation at PyCon 2012.
+   Transcribed by Stephen Turnbull.
+
+These are notes from the Mailman sprint at PyCon 2012.  They are not
+terribly well organized, nor fully fleshed out.  Please edit and push
+branches to Launchpad at lp:mailman or post patches to <WHERE?> <URL?>.
+
+The intent of this document is to provide a view of Mailman 3's
+workflow and structures from "eight miles high".
+
+
+Basic Messaging Handling Workflow
+---------------------------------
+
+Mailman accepts a message via the LMTP protocol (RFC 2033).  It
+implements a simple LMTP server internally based on the LMTP server
+provided in the Python stdlib.  The LMTP server's responsibility is to
+parse the message into a tuple (*mlist*, *msg*, *msg_data*).  If the
+parse fails (including messages which Mailman considers to be invalid
+due to lack of Message-Id as strongly recommended by RFC 2822 and RFC
+5322), the message will be rejected, otherwise the tuple is pickled,
+and the resulting *message pickle* added to one of the IN, COMMAND, or
+BOUNCE processing queues.
+
+.. graphviz::
+
+   digraph msgflow {
+     rankdir = LR;
+     node [shape=box, color=lightblue, style=filled];
+     msg [shape=ellipse, color=black, fillcolor=white];
+     lmtpd [label="LMTP\nSERVER"];
+     msg -> MTA [label="SMTP"];
+     MTA -> lmtpd [label="LMTP"];
+     lmtpd -> MTA [label="reject"];
+     lmtpd -> IN -> POSTING [label=".pck"];
+     lmtpd -> BOUNCES [label=".pck"];
+     lmtpd -> COMMAND [label=".pck"];
+   }
+
+The IN queue is processed by *filter chains* (explained below) to
+determine whether the post (or administrative request) will be
+processed.  If not allowed, the message pickle is discarded, rejected
+(returned to sender), or held (added to the MODERATION queue -- not
+shown).  Otherwise the message is added to the POSTING queue.
+
+Each of the COMMAND, BOUNCE, and POSTING queues is processed by a
+*pipeline of handlers* as in Mailman 2's pipeline.  (Some functions
+such as spam detection that were handled in the Mailman 2 pipeline are
+now in the filter chains.)
+
+Handlers may copy messages to other queues (*e.g.*, ARCHIVE), and
+eventually posts for distribution end up in the OUT queue for
+injection into the MTA.
+
+The VIRGIN queue is a special queue for messages created by Mailman.
+
+.. graphviz::
+
+   digraph pipeline {
+   node [shape=box, style=rounded, group=0]
+   { "MIME\ndelete" -> "cleanse headers" -> "add headers" -> \
+     "calculate\nrecipients" -> "to digest" -> "to archive" -> \
+     "to outgoing" }
+   node [shape=box, color=lightblue, style=filled, group=1]
+   { rank=same; POSTING -> "MIME\ndelete" }
+   { rank=same; "to digest" -> DIGEST }
+   { rank=same; "to archive" -> ARCHIVE }
+   { rank=same; "to outgoing" -> OUT }
+   }
+
+
+Message Filtering
+-----------------
+
+Once a message has been classified as a post or administrivia, rules
+are applied to determine whether the message should be distributed or
+acted on.  Rules include things like "if the message's sender is a
+non-member, hold it for moderation", or "if the message contains an
+Approved field with a valid password, distribute it".  A rule may also
+make no decision, in which case the message pickle is passed on to the
+next rule in the filter chain.  The default set of rules looks
+something like this:
+
+.. graphviz::
+
+   digraph chain_rules {
+     rankdir=LR    /* This gives the right orientation of the columns. */
+     rank=same
+     subgraph in { IN [shape=box, color=lightblue, style=filled] }
+     subgraph rules {
+       rankdir=TB
+       rank=same
+       node [shape=record]
+       approved [group=0, label="<f0> approved | {<f1> | <f2>}"]
+       emergency [group=0, label="<f0> emergency | {<f1> | <f2>}"]
+       loop [group=0, label="<f0> loop | {<f1> | <f2>}"]
+       modmember [group=0, label="<f0> member\nmoderated | {<f1> | <f2>}"]
+       administrivia [group=0, label="<f0> administrivia | <f1>"]
+       maxsize [group=0, label="<f0> max\ size | {<f1> | <f2>}"]
+       any [group=0, label="<f0> any | {<f1> | <f2>}"]
+       truth [label="<f0> truth | <f1>"]
+       approved:f1 -> emergency:f0 [weight=100]
+       emergency:f1 -> loop:f0 
+       loop:f1 -> modmember:f0 
+       modmember:f1 -> administrivia:f0 
+       administrivia:f1 -> maxsize:f0 
+       maxsize:f1 -> any:f0 
+       any:f1 -> truth:f0 
+     }
+     subgraph queues {
+       rankdir=TB
+       rank=same
+       node [shape=box, style=filled];
+       DISCARD [shape=invhouse, color=black, style=solid];
+       MODERATION [color=wheat];
+       HOLD [color=wheat];
+     }
+     { POSTING [shape=box, style=filled, color=cyan]; }
+
+     IN -> approved:f0
+     approved:f2 -> POSTING [minlen=2]
+     loop:f2 -> DISCARD
+     modmember:f2 -> MODERATION
+
+     emergency:f2:e -> HOLD
+     maxsize:f2 -> MODERATION
+     any:f2 -> MODERATION
+     truth:f1 -> POSTING [minlen=2]
+   }
+
+
+Configuration
+-------------
+
+Uses lazr.config.
+
+Each Runner's configuration object knows whether it should be started
+when the Mailman daemon starts, and what queue the Runner manages.
+
+
+Shell Commands
+--------------
+
+bin/mailman: This is an ubercommand, with subcommands for all the
+various things admins might want to do, similar to mailmanctl, but
+with more functionality.
+
+bin/master: The runner manager: starts, watches, stops the runner
+daemons.
+
+bin/runner: Individual runner daemons.  Each instance is configured
+with a configure object specified on the command line, and other
+command line options.
+
+
+User Model
+----------
+
+A *user* represents a person.  A user has an *id* and a *display
+name*, and a list of addresses.
+
+Each *address* is a separate object, linked to a user with a user id.
+
+A list *member* is a link from a user to a mailing list.  Each list
+member has a user id, a mailing list name, an address (which may be
+None, which will be replaced by the user's preferred address, a list
+of preferences, and a *role* such as "owner" or "moderator".  Roles
+are used to determine what kinds of mail the user receives via that
+membership.  *Owners* will receive mail to *list*-owner, but not posts
+and moderation traffic, for example.  A user with multiple roles on a
+single list will therefore have multiple memberships in that list, one
+for each role.
+
+Roles are implemented by "magical, invisible" *rosters*.
+
+
+List Styles
+-----------
+
+Each list *style* is a named object.  Its attributes are functions
+used to apply the relevant style settings to the mailing list *at
+creation time*.  Since these are functions, they can be composed in
+various ways, to create substyles, *etc*.