Merge lp:mailman/2.1 into lp:mailman

[Salah hazaa]

[Mark Sapiro (msapiro)]

lp:mailman is a very early version of Mailman 3 and is no longer relevant. Mailman 3 is currently on GitLab and while it currently doesn't support all features of Mailman 2.1 it has diverged so far from Mailman 2.1 that merging all or even selected parts of Mailman 2.1 would be a disaster.

review: Disapprove

Unmerged revisions

1894. By Mark Sapiro on 2024-11-15

Update reCAPTCHA urls for global use.

1893. By Mark Sapiro on 2023-05-22

Improved fix for LP: #2017813.

1892. By Mark Sapiro on 2023-04-26

Fixed yet another possible list membership leak via the user options CGI.

1891. By Mark Sapiro on 2023-04-05

Fixed another possible list membership leak via the user options CGI.

1890. By Mark Sapiro on 2022-07-10

Fixed a possible list membership leak via the user options CGI.

1889. By Mark Sapiro on 2022-03-28

Fix German translation of Esperanto.

1888. By Mark Sapiro on 2022-02-22

Improve fix for lp:1961762 in prior commit.

1887. By Mark Sapiro on 2022-02-22

Avoid 500 Internal Server Error for non-member with private roster.

1886. By Mark Sapiro on 2022-01-11

Fix test for valid header following From_ line.

1885. By Mark Sapiro on 2021-12-13

Bumped branch version to: 2.1.39

=== added file '.bzrignore'
--- .bzrignore	1970-01-01 00:00:00 +0000
+++ .bzrignore	2025-02-01 12:21:11 +0000
@@ -0,0 +1,26 @@
+Mailman/Defaults.py
+Mailman/mm_cfg.py.dist
+build
+config.log
+config.status
+cron/crontab.in
+misc/JapaneseCodecs-1.4.11
+misc/KoreanCodecs-2.0.5
+misc/email-2.5.8
+misc/mailman
+misc/paths.py
+src/admin
+src/admindb
+src/confirm
+src/create
+src/edithtml
+src/listinfo
+src/mailman
+src/options
+src/private
+src/rmlist
+src/roster
+src/subscribe
+Makefile
+*.mo
+autom4te.cache

=== renamed file '.bzrignore' => '.bzrignore.moved'
=== added file 'ACKNOWLEDGMENTS'
--- ACKNOWLEDGMENTS	1970-01-01 00:00:00 +0000
+++ ACKNOWLEDGMENTS	2025-02-01 12:21:11 +0000
@@ -0,0 +1,251 @@
+Mailman - The GNU Mailing List Management System
+Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
+
+The following folks are or have been core developers of Mailman (in reverse
+alphabetical order):
+
+    Barry Warsaw, Mailman's yappy guard dog
+    Thomas Wouters, Mailman's Dutch treat
+    John Viega, Mailman's inventor
+    Mark Sapiro, Mailman's compulsive responder
+    Harald Meland, Norse Mailman
+    Ken Manheimer, Mailman's savior
+    Tokio Kikuchi, Mailman's weatherman
+    Scott Cotton, Cookie-Monster
+
+They can be contacted directly via [e-mail].
+
+Here is the list of other contributors who have donated large bits of
+code, and have assigned copyright for contributions to the FSF:
+
+    Juan Carlos Rey Anaya
+    Richard Barrett
+    Stephan Berndts
+    Norbert Bollow
+    Ben Gertzfield
+    Victoriano Giralt
+    Mads Kiilerich
+    The Dragon De Monsyne
+    Les Niles
+    Terri Oda
+    Simone Piunno
+
+Thanks also go to the following people for their important contributions in
+other aspects of the Mailman project:
+
+    Brad Knowles
+    JC Dill
+
+Thanks also to Dragon for his winning Mailman logo contribution, and
+to Terri Oda for the neat shortcut icon and the member documentation.
+
+Control.com sponsored development of several Mailman 2.1 features,
+including topics filters, external membership sources, and initial
+virtual mailing list support.  My thanks especially to Dan Pierson and
+Ken Crater from Control.com.
+
+Here is the list of other people who have contributed useful ideas,
+suggestions, bug fixes, testing, etc., or who have been very helpful
+in answering questions on mailman-users.
+
+    David Abrahams
+    William Ahern
+    Terry Allen
+    Jose Paulo Moitinho de Almeida
+    Sven Anderson
+    Matthias Andree
+    Anton Antonov
+    Mike Avery
+    Stonewall Ballard
+    Moreno Baricevic
+    Jeff Berliner
+    Stuart Bishop
+    David Blomquist
+    Bojan
+    Søren Bondrup
+    Grant Bowman
+    Alessio Bragadini
+    J. D. Bronson
+    Stan Bubrouski
+    Daniel Buchmann
+    Ben Burnett
+    Ted Cabeen
+    Mentor Cana
+    John Carnes
+    Julio A. Cartaya
+    Claudio Cattazzo
+    Donn Cave
+    David Champion
+    Hye-Shik Chang
+    Eric D. Christensen
+    Tom G. Christensen
+    Paul Cox
+    Stefaniu Criste
+    Robert Daeley
+    Ned Dawes
+    Emilio Delgado
+    John Dennis
+    Stefan Divjak
+    Maximillian Dornseif
+    Fred Drake
+    Maxim Dzumanenko
+    Piarres Beobide Egaña
+    Rob Ellis
+    Kerem Erkan
+    Fil
+    Patrick Finnerty
+    Bob Fleck
+    Erik Forsberg
+    Darrell Fuhriman
+    Robert Garrigós
+    Carson Gaspar
+    Pascal GEORGE
+    Vadim Getmanshchuk
+    David Gibbs
+    Dmitri I GOULIAEV
+    Terry Grace
+    Federico Grau
+    Pekka Haavisto
+    David Habben
+    Stig Hackvan
+    Jeff Hahn
+    Terry Hardie
+    Paul Hebble
+    Tollef Fog Heen
+    Peer Heinlein
+    James Henstridge
+    Walter Hop
+    Bert Hubert
+    Henny Huisman
+    Jeremy Hylton
+    Ikeda Soji
+    Rostyk Ivantsiv
+    Ron Jarrell
+    Matthias Juchem
+    Tamito KAJIYAMA
+    Nino Katic
+    SHIGENO Kazutaka
+    Ashley M. Kirchner
+    Matthias Klose
+    Harald Koch
+    Eddie Kohler
+    Chris Kolar
+    Uros Kositer
+    Andrew Kuchling
+    Ricardo Kustner
+    L'homme Moderne
+    Sylvain Langlade
+    Ed Lau
+    J C Lawrence
+    Greg Lindahl
+    Christopher P. Lindsey
+    Martin von Loewis
+    Dario Lopez-Kästen
+    Tanner Lovelace
+    Jay Luker
+    Gergely Madarasz
+    Luca Maranzano
+    John A. Martin
+    Andrew Martynov
+    Jason R. Mastaler
+    Michael Mclay
+    Michael Meltzer
+    Marc MERLIN
+    Nigel Metheringham
+    Dan Mick
+    Garey Mills
+    Martin Mokrejs
+    Michael Fischer v. Mollard
+    David Martínez Moreno
+    Dirk Mueller
+    Jonas Muerer
+    Erik Myllymaki
+    Balazs Nagy
+    Moritz Naumann
+    Dale Newfield
+    Hrvoje Niksic
+    Les Niles
+    Mike Noyes
+    David B. O'Donnell
+    Timothy O'Malley
+    "office"
+    Dan Ohnesorg
+    Gerald Oskoboiny
+    Eva Österlind
+    Toni Panadès
+    Jon Parise
+    Chris Pepper
+    Tim Peters
+    Joe Peterson
+    PieterB
+    Rodolfo Pilas
+    Skye Poier
+    Martin Pool
+    Don Porter
+    Francesco Potortì
+    Bob Puff
+    Michael Ranner
+    John Read
+    Sean Reifschneider
+    Christian Reis
+    Ademar de Souza Reis, Jr.
+    Bernhard Reiter
+    Stephan Richter
+    Tristan Roddis
+    Heiko Rommel
+    Luigi Rosa
+    Guido van Rossum
+    Nicholas Russo
+    Chris Ryan
+    Cabel Sasser
+    Bartosz Sawicki
+    Kai Schaetzl
+    Karoly Segesdi
+    Gleydson Mazioli da Silva
+    Pasi Sjöholm
+    Chris Snell
+    Mikhail Sobolev
+    Greg Stein
+    Dale Stimson
+    Students of HIT <[e-mail]>
+    Szabolcs Szigeti
+    Vizi Szilard
+    David T-G
+    Owen Taylor
+    Danny Terweij
+    Jim Tittsler
+    Todd (Freedom Lover)
+    Roger Tsang
+    Chuq Von Rospach
+    Jens Vagelpohl
+    Valia V. Vaneeva
+    Anti Veeranna
+    Todd Vierling
+    Bill Wagner
+    Greg Ward
+    Mark Weaver
+    Kathleen Webb
+    Florian Weimer
+    Ousmane Wilane
+    Dan Wilder
+    Seb Wills
+    Dai Xiaoguang
+    Ping Yeh
+    YASUDA Yukihiro
+    Michael Yount
+    Blair Zajac
+    Mikhail Zabaluev
+    Noam Zeilberger
+    Daniel Zeiss
+    Todd Zullinger
+
+And everyone else on [e-mail] and
[e-mail]!  Thank you, all.
+
+
+

+Local Variables:
+mode: indented-text
+indent-tabs-mode: nil
+End:

=== added file 'BUGS'
--- BUGS	1970-01-01 00:00:00 +0000
+++ BUGS	2025-02-01 12:21:11 +0000
@@ -0,0 +1,13 @@
+Mailman - The GNU Mailing List Management System
+Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
+
+The Mailman project is being managed on Launchpad at
+
+    https://launchpad.net/mailman
+
+You should submit bugs to the Launchpad bug manager at
+
+    https://bugs.launchpad.net/mailman
+
+If you have a suggested fix, please attach it to your bug report.

=== added file 'FAQ'
--- FAQ	1970-01-01 00:00:00 +0000
+++ FAQ	2025-02-01 12:21:11 +0000
@@ -0,0 +1,383 @@
+Mailman - The GNU Mailing List Management System
+Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
+
+Note: We've migrated the FAQ to the wiki at http://wiki.list.org/
+      To see the Mailman FAQ go to http://wiki.list.org/x/AgA3
+
+FREQUENTLY ASKED QUESTIONS
+
+Q. How do you spell this program?
+
+A. You spell it "Mailman", with a leading capital "M" and a lowercase
+   second "m".  It is incorrect to spell it "MailMan" (i.e. you should
+   not use StudlyCaps).
+
+Q. I'm getting really terrible performance for outgoing messages.  It
+   seems that if the MTA has trouble resolving DNS for any recipients,
+   qrunner just gets really slow clearing the queue.  Any ideas?
+
+A. What's likely happening is that your MTA is doing DNS resolution on
+   recipients for messages delivered locally (i.e. from Mailman to
+   your MTA via SMTPDirect.py).  This is a Bad Thing.  You need to
+   turn off synchronous DNS resolution for messages originating from
+   the local host.
+
+   In Exim, the value to edit is receiver_verify_hosts.  Consult the
+   Mailman Installation Manual for details.  Other MTAs have (of
+   course) different parameters and defaults that control this.  First
+   check the README file for your MTA and then consult your MTA's own
+   documentation.
+
+Q. My list members are complaining about Mailman's List-* headers!
+   What can I do about this?
+
+A. These headers are described in RFC 2369 and are added by Mailman
+   for the long-term benefit of end-users.  While discouraged, the
+   list admin can disable these via the General Options page.  See
+   also README.USERAGENT for more information.
+
+Q. Can I put the user's address in the footer that Mailman adds to
+   each message?
+
+A. Yes, in Mailman 2.1.  The site admin needs to enable personalization by
+   setting the following variable in the mm_cfg.py file:
+
+    OWNERS_CAN_ENABLE_PERSONALIZATION = Yes
+
+   Once this is done, list admins can enable personalization for regular
+   delivery members (digest deliveries can't be personalized currently).  A
+   personalized list can include the user's address in the footer.
+
+Q. My users hate HTML in their email and for security reasons, I want
+   to strip out all MIME attachments.  How can I do this?
+
+A. Mailman 2.1 has this feature built-in.  See the Content Filtering
+   Options page in the admin interface.
+
+Q. What if I get "document contains no data" from the web server, or
+   mail isn't getting delivered, or I see "Premature end of script
+   headers" or "Mailman CGI error!!!"
+
+A. The most likely cause of this is that the GID that is compiled into
+   the C wrappers does not match the GID that your Web server invokes
+   CGI scripts with.  Note that a similar error could occur if your
+   mail system invokes filter programs under a GID that does not match
+   the one compiled into the C mail wrapper.
+
+   To fix this you will need to re-configure Mailman using the
+   --with-cgi-gid and --with-mail-gid options.  See the INSTALL file
+   for details.
+
+   These errors are logged to syslog and they do not show up in the
+   Mailman log files.  Problems with the CGI wrapper do get reported
+   in the web browser though (unless STEALTH_MODE is enabled), and
+   include the expected GID, so that should help a lot.
+
+   You may want to have syslog running and configured to log the
+   mail.error log class somewhere; on Solaris systems, the line
+
+       mail.debug                /var/log/syslog
+
+   causes the messages to go to them in /var/log/syslog, for example.
+   (The distributed syslog.conf forwards the message to the loghost,
+   when present.  See the syslog man page for more details.)
+
+   If your system is set up like this, and you get a failure trying to
+   visit the mailman/listinfo web page, and it's due to a UID or GID
+   mismatch, then you should get an entry at the end of
+   /var/log/syslog identifying the expected and received values.
+
+   If you are not getting any log messages in syslog, or in Mailman's
+   own log files, but messages are still not being delivered, then it
+   is likely that qrunner is not running (qrunner is the process that
+   handles all mail in the system).  In Mailman 2.0, qrunner was
+   invoked from cron so make sure your crontab entries for the
+   `mailman' user have been installed.  In Mailman 2.1, qrunner is
+   started with the bin/mailmanctl script, which can be invoked
+   manually, or merged with your OS's init scripts.
+
+Q. What should I check periodically?
+
+A. Many of the scripts have their standard error logged to
+   $prefix/logs/error, and some of the modules write caught errors
+   there, as well, so you should check there at least occasionally to
+   look for bugs in the code and problems in your setup.
+
+   You may want to periodically check the other log files in the logs/
+   directory, perhaps occasionally rotating them with something like
+   the Linux logrotate script.
+
+Q. I can't access the public archives.  Why?
+
+A. If you are using Apache, you must make sure that FollowSymLinks is
+   enabled for the path to the public archives.  Note that the actual
+   archives always reside in the private tree, and only when archives
+   are public, is the symlink followed. See this archive message for
+   more details:
+
+   http://mail.python.org/pipermail/mailman-users/1998-November/000150.html
+
+Q. Still having problems?  Running QMail?
+
+A. Make sure that you are using "preline" before calling the "mailman"
+   wrapper:
+
+       |preline /home/mailman/mail/mailman post listname
+
+   "preline" adds a Unix-style "From " header which the archiver requires.
+   You can fix the archive mbox files by adding:
+
+       From somebody Mon Oct  9 12:27:34 MDT 2000
+
+   before every message and re-running the archive command
+   "bin/arch listname".  The archives should now exist.
+
+Q. I want to get rid of some messages in my archive.  How do I do
+   this?
+
+A. David Rocher posts the following recipe:
+
+   * remove $prefix/archives/private/<listname>
+   * edit $prefix/archives/private/<listname>.mbox/<listname>.mbox [optional]
+   * run $prefix/bin/arch <listname>
+
+Q. How secure are the authentication mechanisms used in Mailman's web
+   interface?
+
+A. If your Mailman installation run on an SSL-enabled web server
+   (i.e. you access the Mailman web pages with "https://..." URLs),
+   you should be as safe as SSL itself is.
+
+   However, most Mailman installation run under standard,
+   encryption-unaware servers.  There's nothing wrong with that for
+   most applications, but a sufficiently determined cracker *could*
+   get unauthorized access by:
+
+   * Packet sniffing: The password used to do the initial
+     authentication for any non-public Mailman page is sent as clear
+     text over the net.  If you consider this to be a big problem, you
+     really should use an SSL-enabled server.
+
+   * Stealing a valid cookie: After successful password
+     authentication, Mailman sends a "cookie" back to the user's
+     browser.  This cookie will be used for "automatic" authentication
+     when browsing further within the list's protected pages.  Mailman
+     employs "session cookies" which are set until you quit your
+     browser or explicitly log out.
+
+     Gaining access to the user's cookie (e.g. by being able to read
+     the user's browser cookie database, or by means of packet
+     sniffing, or maybe even by some broken browser offering all it's
+     cookies to any and all sites the user accesses), and at the same
+     time being able to fulfill the other criteria for using the
+     cookie could result in unauthorized access.
+
+     Note that this problem is more easily exploited when users browse
+     the web via proxies -- in that case, the cookie would be valid
+     for any connections made through that proxy, and not just for
+     connections made from the particular machine the user happens to
+     be accessing the proxy from.
+
+   * Getting access to the user's terminal: This is really just
+     another kind of cookie stealing.  The short cookie expiration
+     time is supposed to help defeat this problem.  It can be
+     considered the price to pay for the convenience of not having to
+     type the password in every time.
+
+Q. I want to backup my lists.  What do I need to save?
+
+A. See this FAQ entry: http://wiki.list.org/x/5oA9
+
+Q. How do I rename a list?
+
+A. Renaming a list is currently a bit of a pain to do completely
+   correctly, especially if you want to make sure that the old list
+   contacts are automatically forwarded to the new list.  This ought
+   to be easier. :(
+
+   The biggest problem you have is how to stop mail and web traffic to
+   your list during the transition, and what to do about any mail
+   undelivered to the old list after the move.  I don't think there
+   are any foolproof steps, but here's how you can reduce the risk:
+
+   - Temporarily disable qrunner.  To do this, you need to edit the
+     user `mailman's crontab entry.  Execute the following command,
+     commenting out the qrunner line when you're dropped into your
+     editor.  Then save the file and quit the editor.
+
+     % crontab -u mailman -e
+
+   - Turn off your mail server.  This is mostly harmless since remote
+     MTAs will just keep retrying until you turn it back on, and it's
+     not going to be off for very long.
+
+   - Next turn off your web server if possible.  This of course means
+     your entire site will be off-line while you make the switch and
+     this may not be acceptable to you.  The next best suggestion is
+     to set up your permanent redirects now for the list you're
+     moving.  This means that anybody looking for the list under its
+     old name will be redirected to the new name, but they'll get
+     errors until you've completed the move.
+
+     Let's say the old name is "oldname" and the new name is
+     "newname".  Here are some Apache directives that will do the
+     trick, though YMMV:
+
+     RedirectMatch permanent /mailman/(.*)/oldname(.*) http://www.dom.ain/mailman/$1/newname$2
+     RedirectMatch permanent /pipermail/oldname(.*)    http://www.dom.ain/pipermail/newname$1
+
+     Add these to your httpd.conf file and restart Apache.
+
+   - Now cd to the directory where you've installed Mailman.  Let's
+     say it's /usr/local/mailman:
+
+     % cd /usr/local/mailman
+
+     and cd to the `lists' subdirectory:
+
+     % cd lists
+
+     You should now see the directory `oldname'.  Move this to
+     `newname':
+
+     % mv oldname newname
+
+   - Now cd to the private archives directory:
+
+     % cd ../archives/private
+
+     You will need to move the oldname's .mbox directory, and the
+     .mbox file within that directory.  Don't worry about the public
+     archives; the next few steps will take care of them without
+     requiring you to fiddle around in the file system:
+
+     % mv oldname.mbox newname.mbox
+     % mv newname.mbox/oldname.mbox newname.mbox/newname.mbox
+
+   - You now need to run the `bin/move_list' script to update some of
+     the internal archiver paths.  IMPORTANT: Skip this step if you
+     are using Mailman 2.1!
+
+     % cd ../..
+     % bin/move_list newname
+
+   - You should now regenerate the public archives:
+
+     % bin/arch newname
+
+   - You'll likely need to change some of your list's configuration
+     options, especially if you want to accept postings addressed to
+     the old list on the new list.  Visit the admin interface for your
+     new list:
+
+     o Go to the General options
+
+     o Change the "real_name" option to reflect the new list's name,
+       e.g. "Newname"
+
+     o Change the subject prefix to reflect the new list's name,
+       e.g. "[Newname] " (yes, that's a trailing space character).
+
+     o Optionally, update other configuration fields like info,
+       description, or welcome_msg.  YMMV.
+
+     o Save your changes
+
+     o Go to the Privacy options
+
+     o Add the old list's address to acceptable_aliases.
+       E.g. "[e-mail]".  This way, (after the /etc/aliases
+       changes described below) messages posted to the old list will
+       not be held by the new list for "implicit destination"
+       approval.
+
+     o Save your changes
+
+   - Now you want to update your /etc/aliases file to include the
+     aliases for the new list, and forwards for the old list to the
+     new list.  Note that these instructions are for Sendmail style
+     alias files, adjust to the specifics of how your MTA is set up.
+
+     o Find the lines defining the aliases for your old list's name
+
+     o Copy and paste them just below the originals.
+
+     o Change all the references of "oldname" to "newname" in the
+       pasted stanza.
+
+     o Now change the targets of the original aliases to forward to
+       the new aliases.  When you're done, you will end up with
+       /etc/aliases entries like the following (YMMV):
+
+       XXX This needs updating for MM2.1!
+
+       # Forward the oldname list to the newname list
+       oldname:         [e-mail]
+       oldname-request: [e-mail]
+       oldname-admin:   [e-mail]
+       oldname-owner:   [e-mail]
+
+       newname:         "|/usr/local/mailman/mail/mailman post newname"
+       newname-admin:   "|/usr/local/mailman/mail/mailman mailowner newname"
+       newname-request: "|/usr/local/mailman/mail/mailman mailcmd newname"
+       newname-owner:   newname-admin
+
+     o Run newaliases
+
+   - Before you restart everything, you want to make one last check.
+     You're looking for files in the qfiles/ directory that may have
+     been addressed to the old list but weren't delivered before you
+     renamed the list.  Do something like the following:
+
+     % cd /usr/local/mailman/qfiles
+     % grep oldname *.msg
+
+     If you get no hits, skip to the next step, you've got nothing to
+     worry about.
+
+     If you did get hits, then things get complicated.  I warn you
+     that the rest of this step is untested. :(
+
+     For each of the .msg files that were destined for the old list,
+     you need to change the corresponding .db file.  Unfortunately
+     there's no easy way to do this.  Anyway...
+
+     Save the following Python code in a file called 'hackdb.py':
+
+     -------------------------hackdb.py
+     import sys
+     import marshal
+     fp = open(sys.argv[1])
+     d = marshal.load(fp)
+     fp.close()
+     d['listname'] = sys.argv[2]
+     fp = open(sys.argv[1], 'w')
+     marshal.dump(d, fp)
+     fp.close()
+     -------------------------
+
+     And then for each file that matched your grep above, do the
+     following:
+
+     % python hackdb.py reallylonghexfilenamematch1.db newname
+
+   - It's now safe to turn your MTA back on.
+
+   - Turn your qrunner back on by running
+
+     % crontab -u mailman -e
+
+     again and this time uncommenting the qrunner line.  Save the file
+     and quit your editor.
+
+   - Rejoice, you're done.  Send $100,000 in shiny new pennies to the
+     Mailman cabal as your downpayment toward making this easier for
+     the next list you have to rename. :)
+
+
+

+Local Variables:
+mode: text
+indent-tabs-mode: nil
+End:

=== added file 'INSTALL'
--- INSTALL	1970-01-01 00:00:00 +0000
+++ INSTALL	2025-02-01 12:21:11 +0000
@@ -0,0 +1,25 @@
+Mailman - The GNU Mailing List Management System
+Copyright (C) 1998-2018 Free Software Foundation, Inc.
+51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
+
+The installation and upgrading instructions are now completely contained in
+the Mailman Installation Guide.  Web, PostScript, PDF, and plaintext formats
+for this guide are available both within this source distribution and online.
+
+All manuals within this source distribution are provided in the doc/
+directory:
+
+    HTML        : doc/mailman-install/index.html
+    PostScript  : doc/mailman-install.ps
+    PDF         : doc/mailman-install.pdf
+    plain text  : doc/mailman-install.txt
+
+Or go online at http://www.list.org/site.html to find the online installation
+guide.
+
+
+

+Local Variables:
+mode: indented-text
+indent-tabs-mode: nil
+End:

=== added directory 'Mailman'
=== added directory 'Mailman/Archiver'
=== added file 'Mailman/Archiver/Archiver.py'
--- Mailman/Archiver/Archiver.py	1970-01-01 00:00:00 +0000
+++ Mailman/Archiver/Archiver.py	2025-02-01 12:21:11 +0000
@@ -0,0 +1,244 @@
+# Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+
+
+"""Mixin class for putting new messages in the right place for archival.
+
+Public archives are separated from private ones.  An external archival
+mechanism (eg, pipermail) should be pointed to the right places, to do the
+archival.
+"""
+
+import os
+import errno
+import traceback
+import re
+from cStringIO import StringIO
+
+from Mailman import mm_cfg
+from Mailman import Mailbox
+from Mailman import Utils
+from Mailman import Site
+from Mailman.SafeDict import SafeDict
+from Mailman.Logging.Syslog import syslog
+from Mailman.i18n import _
+
+try:
+    True, False
+except NameError:
+    True = 1
+    False = 0
+
+
+

+def makelink(old, new):
+    try:
+        os.symlink(old, new)
+    except OSError, e:
+        if e.errno <> errno.EEXIST:
+            raise
+
+def breaklink(link):
+    try:
+        os.unlink(link)
+    except OSError, e:
+        if e.errno <> errno.ENOENT:
+            raise
+
+
+

+class Archiver:
+    #
+    # Interface to Pipermail.  HyperArch.py uses this method to get the
+    # archive directory for the mailing list
+    #
+    def InitVars(self):
+        # Configurable
+        self.archive = mm_cfg.DEFAULT_ARCHIVE
+        # 0=public, 1=private:
+        self.archive_private = mm_cfg.DEFAULT_ARCHIVE_PRIVATE
+        self.archive_volume_frequency = \
+                mm_cfg.DEFAULT_ARCHIVE_VOLUME_FREQUENCY
+        # The archive file structure by default is:
+        #
+        # archives/
+        #     private/
+        #         listname.mbox/
+        #             listname.mbox
+        #         listname/
+        #             lots-of-pipermail-stuff
+        #     public/
+        #         listname.mbox@ -> ../private/listname.mbox
+        #         listname@ -> ../private/listname
+        #
+        # IOW, the mbox and pipermail archives are always stored in the
+        # private archive for the list.  This is safe because archives/private
+        # is always set to o-rx.  Public archives have a symlink to get around
+        # the private directory, pointing directly to the private/listname
+        # which has o+rx permissions.  Private archives do not have the
+        # symbolic links.
+        omask = os.umask(0)
+        try:
+            try:
+                os.mkdir(self.archive_dir()+'.mbox', 02775)
+            except OSError, e:
+                if e.errno <> errno.EEXIST: raise
+                # We also create an empty pipermail archive directory into
+                # which we'll drop an empty index.html file into.  This is so
+                # that lists that have not yet received a posting have
+                # /something/ as their index.html, and don't just get a 404.
+            try:
+                os.mkdir(self.archive_dir(), 02775)
+            except OSError, e:
+                if e.errno <> errno.EEXIST: raise
+            # See if there's an index.html file there already and if not,
+            # write in the empty archive notice.
+            indexfile = os.path.join(self.archive_dir(), 'index.html')
+            fp = None
+            try:
+                fp = open(indexfile)
+            except IOError, e:
+                if e.errno <> errno.ENOENT: raise
+                omask = os.umask(002)
+                try:
+                    fp = open(indexfile, 'w')
+                finally:
+                    os.umask(omask)
+                fp.write(Utils.maketext(
+                    'emptyarchive.html',
+                    {'listname': self.real_name,
+                     'listinfo': self.GetScriptURL('listinfo', absolute=1),
+                     }, mlist=self))
+            if fp:
+                fp.close()
+        finally:
+            os.umask(omask)
+
+    def archive_dir(self):
+        return Site.get_archpath(self.internal_name())
+
+    def ArchiveFileName(self):
+        """The mbox name where messages are left for archive construction."""
+        return os.path.join(self.archive_dir() + '.mbox',
+                            self.internal_name() + '.mbox')
+
+    def GetBaseArchiveURL(self):
+        url = self.GetScriptURL('private', absolute=1) + '/'
+        if self.archive_private:
+            return url
+        else:
+            hostname = re.match('[^:]*://([^/]*)/.*', url).group(1)\
+                       or mm_cfg.DEFAULT_URL_HOST
+            url = mm_cfg.PUBLIC_ARCHIVE_URL % {
+                'listname': self.internal_name(),
+                'hostname': hostname
+                }
+            if not url.endswith('/'):
+                url += '/'
+            return url
+
+    def __archive_file(self, afn):
+        """Open (creating, if necessary) the named archive file."""
+        omask = os.umask(002)
+        try:
+            return Mailbox.Mailbox(open(afn, 'a+'))
+        finally:
+            os.umask(omask)
+
+    #
+    # old ArchiveMail function, retained under a new name
+    # for optional archiving to an mbox
+    #
+    def __archive_to_mbox(self, post):
+        """Retain a text copy of the message in an mbox file."""
+        try:
+            afn = self.ArchiveFileName()
+            mbox = self.__archive_file(afn)
+            mbox.AppendMessage(post)
+            mbox.fp.close()
+        except IOError, msg:
+            syslog('error', 'Archive file access failure:\n\t%s %s', afn, msg)
+            raise
+
+    def ExternalArchive(self, ar, txt):
+        d = SafeDict({'listname': self.internal_name(),
+                      'hostname': self.host_name,
+                      })
+        cmd = ar % d
+        extarch = os.popen(cmd, 'w')
+        extarch.write(txt)
+        status = extarch.close()
+        if status:
+            syslog('error', 'external archiver non-zero exit status: %d\n',
+                   (status & 0xff00) >> 8)
+
+    #
+    # archiving in real time  this is called from list.post(msg)
+    #
+    def ArchiveMail(self, msg):
+        """Store postings in mbox and/or pipermail archive, depending."""
+        # Fork so archival errors won't disrupt normal list delivery
+        if mm_cfg.ARCHIVE_TO_MBOX == -1:
+            return
+        #
+        # We don't need an extra archiver lock here because we know the list
+        # itself must be locked.
+        if mm_cfg.ARCHIVE_TO_MBOX in (1, 2):
+            self.__archive_to_mbox(msg)
+            if mm_cfg.ARCHIVE_TO_MBOX == 1:
+                # Archive to mbox only.
+                return
+        txt = str(msg)
+        # should we use the internal or external archiver?
+        private_p = self.archive_private
+        if mm_cfg.PUBLIC_EXTERNAL_ARCHIVER and not private_p:
+            self.ExternalArchive(mm_cfg.PUBLIC_EXTERNAL_ARCHIVER, txt)
+        elif mm_cfg.PRIVATE_EXTERNAL_ARCHIVER and private_p:
+            self.ExternalArchive(mm_cfg.PRIVATE_EXTERNAL_ARCHIVER, txt)
+        else:
+            # use the internal archiver
+            f = StringIO(txt)
+            import HyperArch
+            h = HyperArch.HyperArchive(self)
+            h.processUnixMailbox(f)
+            h.close()
+            f.close()
+
+    #
+    # called from MailList.MailList.Save()
+    #
+    def CheckHTMLArchiveDir(self):
+        # We need to make sure that the archive directory has the right perms
+        # for public vs private.  If it doesn't exist, or some weird
+        # permissions errors prevent us from stating the directory, it's
+        # pointless to try to fix the perms, so we just return -scott
+        if mm_cfg.ARCHIVE_TO_MBOX == -1:
+            # Archiving is completely disabled, don't require the skeleton.
+            return
+        pubdir = Site.get_archpath(self.internal_name(), public=True)
+        privdir = self.archive_dir()
+        pubmbox = pubdir + '.mbox'
+        privmbox = privdir + '.mbox'
+        if self.archive_private:
+            breaklink(pubdir)
+            breaklink(pubmbox)
+        else:
+            # BAW: privdir or privmbox could be nonexistant.  We'd get an
+            # OSError, ENOENT which should be caught and reported properly.
+            makelink(privdir, pubdir)
+            # Only make this link if the site has enabled public mbox files
+            if mm_cfg.PUBLIC_MBOX:
+                makelink(privmbox, pubmbox)

=== added file 'Mailman/Archiver/HyperArch.py'
--- Mailman/Archiver/HyperArch.py	1970-01-01 00:00:00 +0000
+++ Mailman/Archiver/HyperArch.py	2025-02-01 12:21:11 +0000
@@ -0,0 +1,1335 @@
+# Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301,
+# USA.
+
+"""HyperArch: Pipermail archiving for Mailman
+
+     - The Dragon De Monsyne <[e-mail]>
+
+   TODO:
+     - Should be able to force all HTML to be regenerated next time the
+       archive is run, in case a template is changed.
+     - Run a command to generate tarball of html archives for downloading
+       (probably in the 'update_dirty_archives' method).
+"""
+
+from __future__ import nested_scopes
+
+import sys
+import re
+import errno
+import urllib
+import time
+import os
+import types
+import HyperDatabase
+import pipermail
+import weakref
+import binascii
+
+from email.Header import decode_header, make_header
+from email.Errors import HeaderParseError
+from email.Charset import Charset
+
+from Mailman import mm_cfg
+from Mailman import Utils
+from Mailman import Errors
+from Mailman import LockFile
+from Mailman import MailList
+from Mailman import i18n
+from Mailman.SafeDict import SafeDict
+from Mailman.Logging.Syslog import syslog
+from Mailman.Mailbox import ArchiverMailbox
+
+# Set up i18n.  Assume the current language has already been set in the caller.
+_ = i18n._
+C_ = i18n.C_
+
+gzip = None
+if mm_cfg.GZIP_ARCHIVE_TXT_FILES:
+    try:
+        import gzip
+    except ImportError:
+        pass
+
+EMPTYSTRING = ''
+NL = '\n'
+
+# MacOSX has a default stack size that is too small for deeply recursive
+# regular expressions.  We see this as crashes in the Python test suite when
+# running test_re.py and test_sre.py.  The fix is to set the stack limit to
+# 2048; the general recommendation is to do in the shell before running the
+# test suite.  But that's inconvenient for a daemon like the qrunner.
+#
+# AFAIK, this problem only affects the archiver, so we're adding this work
+# around to this file (it'll get imported by the bundled pipermail or by the
+# bin/arch script.  We also only do this on darwin, a.k.a. MacOSX.
+if sys.platform == 'darwin':
+    try:
+        import resource
+    except ImportError:
+        pass
+    else:
+        soft, hard = resource.getrlimit(resource.RLIMIT_STACK)
+        newsoft = min(hard, max(soft, 1024*2048))
+        resource.setrlimit(resource.RLIMIT_STACK, (newsoft, hard))
+
+
+try:
+    True, False
+except NameError:
+    True = 1
+    False = 0
+
+
+

+def html_quote(s, lang=None):
+    repls = ( ('&', '&amp;'),
+              ("<", '&lt;'),
+              (">", '&gt;'),
+              ('"', '&quot;'))
+    for thing, repl in repls:
+        s = s.replace(thing, repl)
+    return Utils.uncanonstr(s, lang)
+
+
+def url_quote(s):
+    return urllib.quote(s)
+
+
+def null_to_space(s):
+    return s.replace('\000', ' ')
+
+
+def sizeof(filename, lang):
+    try:
+        size = os.path.getsize(filename)
+    except OSError, e:
+        # ENOENT can happen if the .mbox file was moved away or deleted, and
+        # an explicit mbox file name was given to bin/arch.
+        if e.errno <> errno.ENOENT: raise
+        return _('size not available')
+    if size < 1000:
+        # Avoid i18n side-effects
+        otrans = i18n.get_translation()
+        try:
+            i18n.set_language(lang)
+            out = _(' %(size)i bytes ')
+        finally:
+            i18n.set_translation(otrans)
+        return out
+    elif size < 1000000:
+        return ' %d KB ' % (size / 1000)
+    # GB?? :-)
+    return ' %d MB ' % (size / 1000000)
+
+
+html_charset = '<META http-equiv="Content-Type" ' \
+               'content="text/html; charset=%s">'
+
+def CGIescape(arg, lang=None):
+    if isinstance(arg, types.UnicodeType):
+        s = Utils.websafe(arg)
+    else:
+        s = Utils.websafe(str(arg))
+    return Utils.uncanonstr(s.replace('"', '&quot;'), lang)
+
+# Parenthesized human name
+paren_name_pat = re.compile(r'([(].*[)])')
+
+# Subject lines preceded with 'Re:'
+REpat = re.compile( r"\s*RE\s*(\[\d+\]\s*)?:\s*", re.IGNORECASE)
+
+# E-mail addresses and URLs in text
+emailpat = re.compile(r'([-+,.\w]+@[-+.\w]+)')
+
+#  Argh!  This pattern is buggy, and will choke on URLs with GET parameters.
+# MAS: Given that people are not constrained in how they write URIs in plain
+# text, it is not possible to have a single regexp to reliably match them.
+# The regexp below is intended to match straightforward cases.  Even humans
+# can't reliably tell whether various punctuation at the end of a URI is part
+# of the URI or not.
+urlpat = re.compile(r'([a-z]+://.*?)(?:_\s|_$|$|[]})>\'"\s])', re.IGNORECASE)
+
+# Blank lines
+blankpat = re.compile(r'^\s*$')
+
+# Starting <html> directive
+htmlpat = re.compile(r'^\s*<HTML>\s*$', re.IGNORECASE)
+# Ending </html> directive
+nohtmlpat = re.compile(r'^\s*</HTML>\s*$', re.IGNORECASE)
+# Match quoted text
+quotedpat = re.compile(r'^([>|:]|&gt;)+')
+
+
+

+# Like Utils.maketext() but with caching to improve performance.
+#
+# _templatefilepathcache is used to associate a (templatefile, lang, listname)
+# key with the file system path to a template file.  This path is the one that
+# the Utils.findtext() function has computed is the one to match the values in
+# the key tuple.
+#
+# _templatecache associate a file system path as key with the text
+# returned after processing the contents of that file by Utils.findtext()
+#
+# We keep two caches to reduce the amount of template text kept in memory,
+# since the _templatefilepathcache is a many->one mapping and _templatecache
+# is a one->one mapping.  Imagine 1000 lists all using the same default
+# English template.
+
+_templatefilepathcache = {}
+_templatecache = {}
+
+def quick_maketext(templatefile, dict=None, lang=None, mlist=None):
+    if mlist is None:
+        listname = ''
+    else:
+        listname = mlist._internal_name
+    if lang is None:
+        if mlist is None:
+            lang = mm_cfg.DEFAULT_SERVER_LANGUAGE
+        else:
+            lang = mlist.preferred_language
+    cachekey = (templatefile, lang, listname)
+    filepath =  _templatefilepathcache.get(cachekey)
+    if filepath:
+        template = _templatecache.get(filepath)
+    if filepath is None or template is None:
+        # Use the basic maketext, with defaults to get the raw template
+        template, filepath = Utils.findtext(templatefile, lang=lang,
+                                            raw=True, mlist=mlist)
+        _templatefilepathcache[cachekey] = filepath
+        _templatecache[filepath] = template
+    # Copied from Utils.maketext()
+    text = template
+    if dict is not None:
+        try:
+            sdict = SafeDict(dict)
+            try:
+                text = sdict.interpolate(template)
+            except UnicodeError:
+                # Try again after coercing the template to unicode
+                utemplate = unicode(template,
+                                    Utils.GetCharSet(lang),
+                                    'replace')
+                text = sdict.interpolate(utemplate)
+        except (TypeError, ValueError), e:
+            # The template is really screwed up
+            syslog('error', 'broken template: %s\n%s', filepath, e)
+    # Make sure the text is in the given character set, or html-ify any bogus
+    # characters.
+    return Utils.uncanonstr(text, lang)
+
+
+

+# Note: I'm overriding most, if not all of the pipermail Article class
+#       here -ddm
+# The Article class encapsulates a single posting.  The attributes are:
+#
+#  sequence : Sequence number, unique for each article in a set of archives
+#  subject  : Subject
+#  datestr  : The posting date, in human-readable format
+#  date     : The posting date, in purely numeric format
+#  fromdate : The posting date, in `unixfrom' format
+#  headers  : Any other headers of interest
+#  author   : The author's name (and possibly organization)
+#  email    : The author's e-mail address
+#  msgid    : A unique message ID
+#  in_reply_to : If !="", this is the msgid of the article being replied to
+#  references: A (possibly empty) list of msgid's of earlier articles in
+#              the thread
+#  body     : A list of strings making up the message body
+
+class Article(pipermail.Article):
+    __super_init = pipermail.Article.__init__
+    __super_set_date = pipermail.Article._set_date
+
+    _last_article_time = time.time()
+
+    def __init__(self, message=None, sequence=0, keepHeaders=[],
+                       lang=mm_cfg.DEFAULT_SERVER_LANGUAGE, mlist=None):
+        self.__super_init(message, sequence, keepHeaders)
+        self.prev = None
+        self.next = None
+        # Trim Re: from the subject line
+        i = 0
+        while i != -1:
+            result = REpat.match(self.subject)
+            if result:
+                i = result.end(0)
+                self.subject = self.subject[i:]
+                if self.subject == '':
+                    self.subject = _('No subject')
+            else:
+                i = -1
+        # Useful to keep around
+        self._lang = lang
+        self._mlist = mlist
+
+        if mm_cfg.ARCHIVER_OBSCURES_EMAILADDRS:
+            # Avoid i18n side-effects.  Note that the language for this
+            # article (for this list) could be different from the site-wide
+            # preferred language, so we need to ensure no side-effects will
+            # occur.  Think what happens when executing bin/arch.
+            otrans = i18n.get_translation()
+            try:
+                i18n.set_language(lang)
+                if self.author == self.email:
+                    self.author = self.email = re.sub('@', _(' at '),
+                                                      self.email)
+                else:
+                    self.email = re.sub('@', _(' at '), self.email)
+            finally:
+                i18n.set_translation(otrans)
+
+        # Snag the content-* headers.  RFC 1521 states that their values are
+        # case insensitive.
+        ctype = message.get('Content-Type', 'text/plain')
+        cenc = message.get('Content-Transfer-Encoding', '')
+        self.ctype = ctype.lower()
+        self.cenc = cenc.lower()
+        self.decoded = {}
+        cset = Utils.GetCharSet(mlist.preferred_language)
+        cset_out = Charset(cset).output_charset or cset
+        if isinstance(cset_out, unicode):
+            # email 3.0.1 (python 2.4) doesn't like unicode
+            cset_out = cset_out.encode('us-ascii')
+        charset = message.get_content_charset(cset_out)
+        if charset:
+            charset = charset.lower().strip()
+            if charset[0]=='"' and charset[-1]=='"':
+                charset = charset[1:-1]
+            if charset[0]=="'" and charset[-1]=="'":
+                charset = charset[1:-1]
+            try:
+                body = message.get_payload(decode=True)
+            except binascii.Error:
+                body = None
+            if body and charset != Utils.GetCharSet(self._lang):
+                # decode body
+                try:
+                    body = unicode(body, charset)
+                except (UnicodeError, LookupError):
+                    body = None
+            if body:
+                self.body = [l + "\n" for l in body.splitlines()]
+
+        self.decode_headers()
+
+    # Mapping of listnames to MailList instances as a weak value dictionary.
+    # This code is copied from Runner.py but there's one important operational
+    # difference.  In Runner.py, we always .Load() the MailList object for
+    # each _dispose() run, otherwise the object retrieved from the cache won't
+    # be up-to-date.  Since we're creating a new HyperArchive instance for
+    # each message being archived, we don't need to worry about that -- but it
+    # does mean there are additional opportunities for optimization.
+    _listcache = weakref.WeakValueDictionary()
+
+    def _open_list(self, listname):
+        # Cache the open list so that any use of the list within this process
+        # uses the same object.  We use a WeakValueDictionary so that when the
+        # list is no longer necessary, its memory is freed.
+        mlist = self._listcache.get(listname)
+        if not mlist:
+            try:
+                mlist = MailList.MailList(listname, lock=0)
+            except Errors.MMListError, e:
+                syslog('error', 'error opening list: %s\n%s', listname, e)
+                return None
+            else:
+                self._listcache[listname] = mlist
+        return mlist
+
+    def __getstate__(self):
+        d = self.__dict__.copy()
+        # We definitely don't want to pickle the MailList instance, so just
+        # pickle a reference to it.
+        if d.has_key('_mlist'):
+            mlist = d['_mlist']
+            del d['_mlist']
+        else:
+            mlist = None
+        if mlist:
+            d['__listname'] = self._mlist.internal_name()
+        else:
+            d['__listname'] = None
+        # Delete a few other things we don't want in the pickle
+        for attr in ('prev', 'next', 'body'):
+            if d.has_key(attr):
+                del d[attr]
+        d['body'] = []
+        return d
+
+    def __setstate__(self, d):
+        # For loading older Articles via pickle.  All this stuff was added
+        # when Simone Piunni and Tokio Kikuchi i18n'ified Pipermail.  See SF
+        # patch #594771.
+        self.__dict__ = d
+        listname = d.get('__listname')
+        if listname:
+            del d['__listname']
+            d['_mlist'] = self._open_list(listname)
+        if not d.has_key('_lang'):
+            if hasattr(self, '_mlist'):
+                self._lang = self._mlist.preferred_language
+            else:
+                self._lang = mm_cfg.DEFAULT_SERVER_LANGUAGE
+        if not d.has_key('cenc'):
+            self.cenc = None
+        if not d.has_key('decoded'):
+            self.decoded = {}
+
+    def setListIfUnset(self, mlist):
+        if getattr(self, '_mlist', None) is None:
+            self._mlist = mlist
+
+    def quote(self, buf):
+        return html_quote(buf, self._lang)
+
+    def decode_headers(self):
+        """MIME-decode headers.
+
+        If the email, subject, or author attributes contain non-ASCII
+        characters using the encoded-word syntax of RFC 2047, decoded versions
+        of those attributes are placed in the self.decoded (a dictionary).
+
+        If the list's charset differs from the header charset, an attempt is
+        made to decode the headers as Unicode.  If that fails, they are left
+        undecoded.
+        """
+        author = self.decode_charset(self.author)
+        subject = self.decode_charset(self.subject)
+        if author:
+            self.decoded['author'] = author
+            email = self.decode_charset(self.email)
+            if email:
+                self.decoded['email'] = email
+        if subject:
+            if mm_cfg.ARCHIVER_OBSCURES_EMAILADDRS:
+                otrans = i18n.get_translation()
+                try:
+                    i18n.set_language(self._lang)
+                    atmark = unicode(_(' at '), Utils.GetCharSet(self._lang))
+                    subject = re.sub(r'([-+,.\w]+)@([-+.\w]+)',
+                              '\g<1>' + atmark + '\g<2>', subject)
+                finally:
+                    i18n.set_translation(otrans)
+            self.decoded['subject'] = subject
+        self.decoded['stripped'] = self.strip_subject(subject or self.subject)
+
+    def strip_subject(self, subject):
+        # Strip subject_prefix and Re: for subject sorting
+        # This part was taken from CookHeaders.py (TK)
+        prefix = self._mlist.subject_prefix.strip()
+        if prefix:
+            prefix_pat = re.escape(prefix)
+            prefix_pat = '%'.join(prefix_pat.split(r'\%'))
+            prefix_pat = re.sub(r'%\d*d', r'\s*\d+\s*', prefix_pat)
+            subject = re.sub(prefix_pat, '', subject)
+        subject = subject.lstrip()
+        # MAS Should we strip FW and FWD too?
+        strip_pat = re.compile('^((RE|AW|SV|VS)(\[\d+\])?:\s*)+', re.I)
+        stripped = strip_pat.sub('', subject)
+        # Also remove whitespace to avoid folding/unfolding differences
+        stripped = re.sub('\s', '', stripped)
+        return stripped
+
+    def decode_charset(self, field):
+        # TK: This function was rewritten for unifying to Unicode.
+        # Convert 'field' into Unicode one line string.
+        try:
+            pairs = decode_header(field)
+            ustr = make_header(pairs).__unicode__()
+        except (LookupError, UnicodeError, ValueError, HeaderParseError):
+            # assume list's language
+            cset = Utils.GetCharSet(self._mlist.preferred_language)
+            if cset == 'us-ascii':
+                cset = 'iso-8859-1' # assume this for English list
+            ustr = unicode(field, cset, 'replace')
+        return u''.join(ustr.splitlines())
+
+    def as_html(self):
+        d = self.__dict__.copy()
+        # avoid i18n side-effects
+        otrans = i18n.get_translation()
+        i18n.set_language(self._lang)
+        try:
+            d["prev"], d["prev_wsubj"] = self._get_prev()
+            d["next"], d["next_wsubj"] = self._get_next()
+
+            d["email_html"] = self.quote(self.email)
+            d["title"] = self.quote(self.subject)
+            d["subject_html"] = self.quote(self.subject)
+            d["message_id"] = self.quote(self._message_id)
+            # TK: These two _url variables are used to compose a response
+            # from the archive web page.  So, ...
+            d["subject_url"] = url_quote('Re: ' + self.subject)
+            d["in_reply_to_url"] = url_quote(self._message_id)
+            if mm_cfg.ARCHIVER_OBSCURES_EMAILADDRS:
+                # Point the mailto url back to the list
+                author = re.sub('@', _(' at '), self.author)
+                emailurl = self._mlist.GetListEmail()
+            else:
+                author = self.author
+                emailurl = self.email
+            d["author_html"] = self.quote(author)
+            d["email_url"] = url_quote(emailurl)
+            d["datestr_html"] = self.quote(i18n.ctime(int(self.date)))
+            d["body"] = self._get_body()
+            d['listurl'] = self._mlist.GetScriptURL('listinfo', absolute=1)
+            d['listname'] = self._mlist.real_name
+            d['encoding'] = ''
+        finally:
+            i18n.set_translation(otrans)
+
+        charset = Utils.GetCharSet(self._lang)
+        d["encoding"] = html_charset % charset
+
+        self._add_decoded(d)
+        return quick_maketext(
+             'article.html', d,
+             lang=self._lang, mlist=self._mlist)
+
+    def _get_prev(self):
+        """Return the href and subject for the previous message"""
+        if self.prev:
+            subject = self._get_subject_enc(self.prev)
+            prev = ('<LINK REL="Previous"  HREF="%s">'
+                    % (url_quote(self.prev.filename)))
+            prev_wsubj = ('<LI>' + _('Previous message (by thread):') +
+                          ' <A HREF="%s">%s\n</A></li>'
+                          % (url_quote(self.prev.filename),
+                             self.quote(subject)))
+        else:
+            prev = prev_wsubj = ""
+        return prev, prev_wsubj
+
+    def _get_subject_enc(self, art):
+        """Return the subject of art, decoded if possible.
+
+        If the charset of the current message and art match and the
+        article's subject is encoded, decode it.
+        """
+        return art.decoded.get('subject', art.subject)
+
+    def _get_next(self):
+        """Return the href and subject for the previous message"""
+        if self.next:
+            subject = self._get_subject_enc(self.next)
+            next = ('<LINK REL="Next"  HREF="%s">'
+                    % (url_quote(self.next.filename)))
+            next_wsubj = ('<LI>' + _('Next message (by thread):') +
+                          ' <A HREF="%s">%s\n</A></li>'
+                          % (url_quote(self.next.filename),
+                             self.quote(subject)))
+        else:
+            next = next_wsubj = ""
+        return next, next_wsubj
+
+    _rx_quote = re.compile('=([A-F0-9][A-F0-9])')
+    _rx_softline = re.compile('=[ \t]*$')
+
+    def _get_body(self):
+        """Return the message body ready for HTML, decoded if necessary"""
+        try:
+            body = self.html_body
+        except AttributeError:
+            body = self.body
+        return null_to_space(EMPTYSTRING.join(body))
+
+    def _add_decoded(self, d):
+        """Add encoded-word keys to HTML output"""
+        for src, dst in (('author', 'author_html'),
+                         ('email', 'email_html'),
+                         ('subject', 'subject_html'),
+                         ('subject', 'title')):
+            if self.decoded.has_key(src):
+                d[dst] = self.quote(self.decoded[src])
+
+    def as_text(self):
+        d = self.__dict__.copy()
+        # We need to guarantee a valid From_ line, even if there are
+        # bososities in the headers.
+        if not d.get('fromdate', '').strip():
+            d['fromdate'] = time.ctime(time.time())
+        if not d.get('email', '').strip():
+            d['email'] = '[e-mail]'
+        if not d.get('datestr', '').strip():
+            d['datestr'] = time.ctime(time.time())
+        #
+        headers = ['From %(email)s  %(fromdate)s',
+                 'From: %(email)s (%(author)s)',
+                 'Date: %(datestr)s',
+                 'Subject: %(subject)s']
+        if d['_in_reply_to']:
+            headers.append('In-Reply-To: %(_in_reply_to)s')
+        if d['_references']:
+            headers.append('References: %(_references)s')
+        if d['_message_id']:
+            headers.append('Message-ID: %(_message_id)s')
+        body = EMPTYSTRING.join(self.body)
+        cset = Utils.GetCharSet(self._lang)
+        # Coerce the body to Unicode and replace any invalid characters.
+        if not isinstance(body, types.UnicodeType):
+            body = unicode(body, cset, 'replace')
+        if mm_cfg.ARCHIVER_OBSCURES_EMAILADDRS:
+            otrans = i18n.get_translation()
+            try:
+                i18n.set_language(self._lang)
+                atmark = unicode(_(' at '), cset)
+                body = re.sub(r'([-+,.\w]+)@([-+.\w]+)',
+                              '\g<1>' + atmark + '\g<2>', body)
+            finally:
+                i18n.set_translation(otrans)
+        # Return body to character set of article.
+        body = body.encode(cset, 'replace')
+        return NL.join(headers) % d + '\n\n' + body + '\n'
+
+    def _set_date(self, message):
+        self.__super_set_date(message)
+        self.fromdate = time.ctime(int(self.date))
+
+    def loadbody_fromHTML(self,fileobj):
+        self.body = []
+        begin = 0
+        while 1:
+            line = fileobj.readline()
+            if not line:
+                break
+            if not begin:
+                if line.strip() == '<!--beginarticle-->':
+                    begin = 1
+                continue
+            if line.strip() == '<!--endarticle-->':
+                break
+            self.body.append(line)
+
+    def finished_update_article(self):
+        self.body = []
+        try:
+            del self.html_body
+        except AttributeError:
+            pass
+
+

+class HyperArchive(pipermail.T):
+    __super_init = pipermail.T.__init__
+    __super_update_archive = pipermail.T.update_archive
+    __super_update_dirty_archives = pipermail.T.update_dirty_archives
+    __super_add_article = pipermail.T.add_article
+
+    # some defaults
+    DIRMODE = 02775
+    FILEMODE = 0660
+
+    VERBOSE = 0
+    DEFAULTINDEX = 'thread'
+    ARCHIVE_PERIOD = 'month'
+
+    THREADLAZY = 0
+    THREADLEVELS = 3
+
+    ALLOWHTML = 1             # "Lines between <html></html>" handled as is.
+    SHOWHTML = 0              # Eg, nuke leading whitespace in html manner.
+    IQUOTES = 1               # Italicize quoted text.
+    SHOWBR = 0                # Add <br> onto every line
+
+    def __init__(self, maillist):
+        # can't init the database while other processes are writing to it!
+        # XXX TODO- implement native locking
+        # with mailman's LockFile module for HyperDatabase.HyperDatabase
+        #
+        dir = maillist.archive_dir()
+        db = HyperDatabase.HyperDatabase(dir, maillist)
+        self.__super_init(dir, reload=1, database=db)
+
+        self.maillist = maillist
+        self._lock_file = None
+        self.lang = maillist.preferred_language
+        self.charset = Utils.GetCharSet(maillist.preferred_language)
+
+        if hasattr(self.maillist,'archive_volume_frequency'):
+            if self.maillist.archive_volume_frequency == 0:
+                self.ARCHIVE_PERIOD='year'
+            elif self.maillist.archive_volume_frequency == 2:
+                self.ARCHIVE_PERIOD='quarter'
+            elif self.maillist.archive_volume_frequency == 3:
+                self.ARCHIVE_PERIOD='week'
+            elif self.maillist.archive_volume_frequency == 4:
+                self.ARCHIVE_PERIOD='day'
+            else:
+                self.ARCHIVE_PERIOD='month'
+
+        yre = r'(?P<year>[0-9]{4,4})'
+        mre = r'(?P<month>[01][0-9])'
+        dre = r'(?P<day>[0123][0-9])'
+        self._volre = {
+            'year':    '^' + yre + '$',
+            'quarter': '^' + yre + r'q(?P<quarter>[1234])$',
+            'month':   '^' + yre + r'-(?P<month>[a-zA-Z]+)$',
+            'week':    r'^Week-of-Mon-' + yre + mre + dre,
+            'day':     '^' + yre + mre + dre + '$'
+            }
+
+    def _makeArticle(self, msg, sequence):
+        return Article(msg, sequence,
+                       lang=self.maillist.preferred_language,
+                       mlist=self.maillist)
+
+    def html_foot(self):
+        # avoid i18n side-effects
+        mlist = self.maillist
+        otrans = i18n.get_translation()
+        i18n.set_language(mlist.preferred_language)
+        # Convenience
+        def quotetime(s):
+            return html_quote(i18n.ctime(s), self.lang)
+        try:
+            d = {"lastdate": quotetime(self.lastdate),
+                 "archivedate": quotetime(self.archivedate),
+                 "listinfo": mlist.GetScriptURL('listinfo', absolute=1),
+                 "version": self.version,
+                 "listname": html_quote(mlist.real_name, self.lang),
+                 }
+            i = {"thread": _("thread"),
+                 "subject": _("subject"),
+                 "author": _("author"),
+                 "date": _("date")
+                 }
+        finally:
+            i18n.set_translation(otrans)
+
+        for t in i.keys():
+            cap = t[0].upper() + t[1:]
+            if self.type == cap:
+                d["%s_ref" % (t)] = ""
+            else:
+                d["%s_ref" % (t)] = ('<a href="%s.html#start">[ %s ]</a>'
+                                     % (t, i[t]))
+        return quick_maketext(
+            'archidxfoot.html', d,
+            mlist=mlist)
+
+    def html_head(self):
+        # avoid i18n side-effects
+        mlist = self.maillist
+        otrans = i18n.get_translation()
+        i18n.set_language(mlist.preferred_language)
+        # Convenience
+        def quotetime(s):
+            return html_quote(i18n.ctime(s), self.lang)
+        try:
+            d = {"listname": html_quote(mlist.real_name, self.lang),
+                 "archtype": self.type,
+                 "archive":  self.volNameToDesc(self.archive),
+                 "listinfo": mlist.GetScriptURL('listinfo', absolute=1),
+                 "firstdate": quotetime(self.firstdate),
+                 "lastdate": quotetime(self.lastdate),
+                 "size": self.size,
+                 }
+            i = {"thread": _("thread"),
+                 "subject": _("subject"),
+                 "author": _("author"),
+                 "date": _("date"),
+                 }
+        finally:
+            i18n.set_translation(otrans)
+
+        for t in i.keys():
+            cap = t[0].upper() + t[1:]
+            if self.type == cap:
+                d["%s_ref" % (t)] = ""
+                d["archtype"] = i[t]
+            else:
+                d["%s_ref" % (t)] = ('<a href="%s.html#start">[ %s ]</a>'
+                                     % (t, i[t]))
+        if self.charset:
+            d["encoding"] = html_charset % self.charset
+        else:
+            d["encoding"] = ""
+        return quick_maketext(
+            'archidxhead.html', d,
+            mlist=mlist)
+
+    def html_TOC(self):
+        mlist = self.maillist
+        listname = mlist.internal_name()
+        mbox = os.path.join(mlist.archive_dir()+'.mbox', listname+'.mbox')
+        d = {"listname": mlist.real_name,
+             "listinfo": mlist.GetScriptURL('listinfo', absolute=1),
+             "fullarch": '../%s.mbox/%s.mbox' % (listname, listname),
+             "size": sizeof(mbox, mlist.preferred_language),
+             'meta': '',
+             }
+        # Avoid i18n side-effects
+        otrans = i18n.get_translation()
+        i18n.set_language(mlist.preferred_language)
+        try:
+            if not self.archives:
+                d["noarchive_msg"] = _(
+                    '<P>Currently, there are no archives. </P>')
+                d["archive_listing_start"] = ""
+                d["archive_listing_end"] = ""
+                d["archive_listing"] = ""
+            else:
+                d["noarchive_msg"] = ""
+                d["archive_listing_start"] = quick_maketext(
+                    'archliststart.html',
+                    lang=mlist.preferred_language,
+                    mlist=mlist)
+                d["archive_listing_end"] = quick_maketext(
+                    'archlistend.html',
+                    mlist=mlist)
+
+                accum = []
+                for a in self.archives:
+                    accum.append(self.html_TOC_entry(a))
+                d["archive_listing"] = EMPTYSTRING.join(accum)
+        finally:
+            i18n.set_translation(otrans)
+        # The TOC is always in the charset of the list's preferred language
+        d['meta'] += html_charset % Utils.GetCharSet(mlist.preferred_language)
+        # The site can disable public access to the mbox file.
+        if mm_cfg.PUBLIC_MBOX:
+            template = 'archtoc.html'
+        else:
+            template = 'archtocnombox.html'
+        return quick_maketext(template, d, mlist=mlist)
+
+    def html_TOC_entry(self, arch):
+        # Check to see if the archive is gzip'd or not
+        txtfile = os.path.join(self.maillist.archive_dir(), arch + '.txt')
+        gzfile = txtfile + '.gz'
+        # which exists?  .txt.gz first, then .txt
+        if os.path.exists(gzfile):
+            file = gzfile
+            url = arch + '.txt.gz'
+            templ = '<td><A href="%(url)s">[ ' + _('Gzip\'d Text%(sz)s') \
+                    + ']</a></td>'
+        elif os.path.exists(txtfile):
+            file = txtfile
+            url = arch + '.txt'
+            templ = '<td><A href="%(url)s">[ ' + _('Text%(sz)s') + ']</a></td>'
+        else:
+            # neither found?
+            file = None
+        # in Python 1.5.2 we have an easy way to get the size
+        if file:
+            textlink = templ % {
+                'url': url,
+                'sz' : sizeof(file, self.maillist.preferred_language)
+                }
+        else:
+            # there's no archive file at all... hmmm.
+            textlink = ''
+        return quick_maketext(
+            'archtocentry.html',
+            {'archive': arch,
+             'archivelabel': self.volNameToDesc(arch),
+             'textlink': textlink
+             },
+            mlist=self.maillist)
+
+    def GetArchLock(self):
+        if self._lock_file:
+            return 1
+        self._lock_file = LockFile.LockFile(
+            os.path.join(mm_cfg.LOCK_DIR,
+                         self.maillist.internal_name() + '-arch.lock'))
+        try:
+            self._lock_file.lock(timeout=0.5)
+        except LockFile.TimeOutError:
+            return 0
+        return 1
+
+    def DropArchLock(self):
+        if self._lock_file:
+            self._lock_file.unlock(unconditionally=1)
+            self._lock_file = None
+
+    def processListArch(self):
+        name = self.maillist.ArchiveFileName()
+        wname= name+'.working'
+        ename= name+'.err_unarchived'
+        try:
+            os.stat(name)
+        except (IOError,os.error):
+            #no archive file, nothin to do -ddm
+            return
+
+        #see if arch is locked here -ddm
+        if not self.GetArchLock():
+            #another archiver is running, nothing to do. -ddm
+            return
+
+        #if the working file is still here, the archiver may have
+        # crashed during archiving. Save it, log an error, and move on.
+        try:
+            wf = open(wname)
+            syslog('error',
+                   'Archive working file %s present.  '
+                   'Check %s for possibly unarchived msgs',
+                   wname, ename)
+            omask = os.umask(007)
+            try:
+                ef = open(ename, 'a+')
+            finally:
+                os.umask(omask)
+            ef.seek(1,2)
+            if ef.read(1) <> '\n':
+                ef.write('\n')
+            ef.write(wf.read())
+            ef.close()
+            wf.close()
+            os.unlink(wname)
+        except IOError:
+            pass
+        os.rename(name,wname)
+        archfile = open(wname)
+        self.processUnixMailbox(archfile)
+        archfile.close()
+        os.unlink(wname)
+        self.DropArchLock()
+
+    def get_filename(self, article):
+        return '%06i.html' % (article.sequence,)
+
+    def get_archives(self, article):
+        """Return a list of indexes where the article should be filed.
+        A string can be returned if the list only contains one entry,
+        and the empty list is legal."""
+        res = self.dateToVolName(float(article.date))
+        self.message(C_("figuring article archives\n"))
+        self.message(res + "\n")
+        return res
+
+    def volNameToDesc(self, volname):
+        volname = volname.strip()
+        # Don't make these module global constants since we have to runtime
+        # translate them anyway.
+        monthdict = [
+            '',
+            _('January'),   _('February'), _('March'),    _('April'),
+            _('May'),       _('June'),     _('July'),     _('August'),
+            _('September'), _('October'),  _('November'), _('December')
+            ]
+        for each in self._volre.keys():
+            match = re.match(self._volre[each], volname)
+            # Let ValueErrors percolate up
+            if match:
+                year = int(match.group('year'))
+                if each == 'quarter':
+                    d =["", _("First"), _("Second"), _("Third"), _("Fourth") ]
+                    ord = d[int(match.group('quarter'))]
+                    return _("%(ord)s quarter %(year)i")
+                elif each == 'month':
+                    monthstr = match.group('month').lower()
+                    for i in range(1, 13):
+                        monthname = time.strftime("%B", (1999,i,1,0,0,0,0,1,0))
+                        if monthstr.lower() == monthname.lower():
+                            month = monthdict[i]
+                            return _("%(month)s %(year)i")
+                    raise ValueError, "%s is not a month!" % monthstr
+                elif each == 'week':
+                    month = monthdict[int(match.group("month"))]
+                    day = int(match.group("day"))
+                    return _("The Week Of Monday %(day)i %(month)s %(year)i")
+                elif each == 'day':
+                    month = monthdict[int(match.group("month"))]
+                    day = int(match.group("day"))
+                    return _("%(day)i %(month)s %(year)i")
+                else:
+                    return match.group('year')
+        raise ValueError, "%s is not a valid volname" % volname
+
+# The following two methods should be inverses of each other. -ddm
+
+    def dateToVolName(self,date):
+        datetuple=time.localtime(date)
+        if self.ARCHIVE_PERIOD=='year':
+            return time.strftime("%Y",datetuple)
+        elif self.ARCHIVE_PERIOD=='quarter':
+            if datetuple[1] in [1,2,3]:
+                return time.strftime("%Yq1",datetuple)
+            elif datetuple[1] in [4,5,6]:
+                return time.strftime("%Yq2",datetuple)
+            elif datetuple[1] in [7,8,9]:
+                return time.strftime("%Yq3",datetuple)
+            else:
+                return time.strftime("%Yq4",datetuple)
+        elif self.ARCHIVE_PERIOD == 'day':
+            return time.strftime("%Y%m%d", datetuple)
+        elif self.ARCHIVE_PERIOD == 'week':
+            # Reconstruct "seconds since epoch", and subtract weekday
+            # multiplied by the number of seconds in a day.
+            monday = time.mktime(datetuple) - datetuple[6] * 24 * 60 * 60
+            # Build a new datetuple from this "seconds since epoch" value
+            datetuple = time.localtime(monday)
+            return time.strftime("Week-of-Mon-%Y%m%d", datetuple)
+        # month. -ddm
+        else:
+            return time.strftime("%Y-%B",datetuple)
+
+
+    def volNameToDate(self, volname):
+        volname = volname.strip()
+        for each in self._volre.keys():
+            match = re.match(self._volre[each],volname)
+            if match:
+                year = int(match.group('year'))
+                month = 1
+                day = 1
+                if each == 'quarter':
+                    q = int(match.group('quarter'))
+                    month = (q * 3) - 2
+                elif each == 'month':
+                    monthstr = match.group('month').lower()
+                    m = []
+                    for i in range(1,13):
+                        m.append(
+                            time.strftime("%B",(1999,i,1,0,0,0,0,1,0)).lower())
+                    try:
+                        month = m.index(monthstr) + 1
+                    except ValueError:
+                        pass
+                elif each == 'week' or each == 'day':
+                    month = int(match.group("month"))
+                    day = int(match.group("day"))
+                try:
+                    return time.mktime((year,month,1,0,0,0,0,1,-1))
+                except OverflowError:
+                    return 0.0
+        return 0.0
+
+    def sortarchives(self):
+        def sf(a, b):
+            al = self.volNameToDate(a)
+            bl = self.volNameToDate(b)
+            if al > bl:
+                return 1
+            elif al < bl:
+                return -1
+            else:
+                return 0
+        if self.ARCHIVE_PERIOD in ('month','year','quarter'):
+            self.archives.sort(sf)
+        else:
+            self.archives.sort()
+        self.archives.reverse()
+
+    def message(self, msg):
+        if self.VERBOSE:
+            f = sys.stderr
+            f.write(msg)
+            if msg[-1:] != '\n':
+                f.write('\n')
+            f.flush()
+
+    def open_new_archive(self, archive, archivedir):
+        index_html = os.path.join(archivedir, 'index.html')
+        try:
+            os.unlink(index_html)
+        except:
+            pass
+        os.symlink(self.DEFAULTINDEX+'.html',index_html)
+
+    def write_index_header(self):
+        self.depth=0
+        print self.html_head()
+        if not self.THREADLAZY and self.type=='Thread':
+            self.message(C_("Computing threaded index\n"))
+            self.updateThreadedIndex()
+
+    def write_index_footer(self):
+        for i in range(self.depth):
+            print '</UL>'
+        print self.html_foot()
+
+    def write_index_entry(self, article):
+        subject = self.get_header("subject", article)
+        author = self.get_header("author", article)
+        if mm_cfg.ARCHIVER_OBSCURES_EMAILADDRS:
+            try:
+                author = re.sub('@', _(' at '), author)
+            except UnicodeError:
+                # Non-ASCII author contains '@' ... no valid email anyway
+                pass
+        subject = CGIescape(subject, self.lang)
+        author = CGIescape(author, self.lang)
+
+        d = {
+            'filename': urllib.quote(article.filename),
+            'subject':  subject,
+            'sequence': article.sequence,
+            'author':   author
+        }
+        print quick_maketext(
+            'archidxentry.html', d,
+            mlist=self.maillist)
+
+    def get_header(self, field, article):
+        # if we have no decoded header, return the encoded one
+        result = article.decoded.get(field)
+        if result is None:
+            return getattr(article, field)
+        # otherwise, the decoded one will be Unicode
+        return result
+
+    def write_threadindex_entry(self, article, depth):
+        if depth < 0:
+            self.message('depth<0')
+            depth = 0
+        if depth > self.THREADLEVELS:
+            depth = self.THREADLEVELS
+        if depth < self.depth:
+            for i in range(self.depth-depth):
+                print '</UL>'
+        elif depth > self.depth:
+            for i in range(depth-self.depth):
+                print '<UL>'
+        print '<!--%i %s -->' % (depth, article.threadKey)
+        self.depth = depth
+        self.write_index_entry(article)
+
+    def write_TOC(self):
+        self.sortarchives()
+        omask = os.umask(002)
+        try:
+            toc = open(os.path.join(self.basedir, 'index.html'), 'w')
+        finally:
+            os.umask(omask)
+        toc.write(self.html_TOC())
+        toc.close()
+
+    def write_article(self, index, article, path):
+        # called by add_article
+        omask = os.umask(002)
+        try:
+            f = open(path, 'w')
+        finally:
+            os.umask(omask)
+        f.write(article.as_html())
+        f.close()
+
+        # Write the text article to the text archive.
+        path = os.path.join(self.basedir, "%s.txt" % index)
+        omask = os.umask(002)
+        try:
+            f = open(path, 'a+')
+        finally:
+            os.umask(omask)
+        f.write(article.as_text())
+        f.close()
+
+    def update_archive(self, archive):
+        self.__super_update_archive(archive)
+        # only do this if the gzip module was imported globally, and
+        # gzip'ing was enabled via mm_cfg.GZIP_ARCHIVE_TXT_FILES.  See
+        # above.
+        if gzip:
+            archz = None
+            archt = None
+            txtfile = os.path.join(self.basedir, '%s.txt' % archive)
+            gzipfile = os.path.join(self.basedir, '%s.txt.gz' % archive)
+            oldgzip = os.path.join(self.basedir, '%s.old.txt.gz' % archive)
+            try:
+                # open the plain text file
+                archt = open(txtfile)
+            except IOError:
+                return
+            try:
+                os.rename(gzipfile, oldgzip)
+                archz = gzip.open(oldgzip)
+            except (IOError, RuntimeError, os.error):
+                pass
+            try:
+                ou = os.umask(002)
+                newz = gzip.open(gzipfile, 'w')
+            finally:
+                # XXX why is this a finally?
+                os.umask(ou)
+            if archz:
+                newz.write(archz.read())
+                archz.close()
+                os.unlink(oldgzip)
+            # XXX do we really need all this in a try/except?
+            try:
+                newz.write(archt.read())
+                newz.close()
+                archt.close()
+            except IOError:
+                pass
+            os.unlink(txtfile)
+
+    _skip_attrs = ('maillist', '_lock_file', 'charset')
+
+    def getstate(self):
+        d={}
+        for each in self.__dict__.keys():
+            if not (each in self._skip_attrs
+                    or each.upper() == each):
+                d[each] = self.__dict__[each]
+        return d
+
+    # Add <A HREF="..."> tags around URLs and e-mail addresses.
+
+    def __processbody_URLquote(self, lines):
+        # XXX a lot to do here:
+        # 1. use lines directly, rather than source and dest
+        # 2. make it clearer
+        # 3. make it faster
+        # TK: Prepare for unicode obscure.
+        atmark = _(' at ')
+        if lines and isinstance(lines[0], types.UnicodeType):
+            atmark = unicode(atmark, Utils.GetCharSet(self.lang), 'replace')
+        source = lines[:]
+        dest = lines
+        last_line_was_quoted = 0
+        for i in xrange(0, len(source)):
+            Lorig = L = source[i]
+            prefix = suffix = ""
+            if L is None:
+                continue
+            # Italicise quoted text
+            if self.IQUOTES:
+                quoted = quotedpat.match(L)
+                if quoted is None:
+                    last_line_was_quoted = 0
+                else:
+                    quoted = quoted.end(0)
+                    prefix = CGIescape(L[:quoted], self.lang) + '<i>'
+                    suffix = '</I>'
+                    if self.SHOWHTML:
+                        suffix += '<BR>'
+                        if not last_line_was_quoted:
+                            prefix = '<BR>' + prefix
+                    L = L[quoted:]
+                    last_line_was_quoted = 1
+            # Check for an e-mail address
+            L2 = ""
+            jr = emailpat.search(L)
+            kr = urlpat.search(L)
+            while jr is not None or kr is not None:
+                if jr == None:
+                    j = -1
+                else:
+                    j = jr.start(0)
+                if kr is None:
+                    k = -1
+                else:
+                    k = kr.start(0)
+                if j != -1 and (j < k or k == -1):
+                    text = jr.group(1)
+                    length = len(text)
+                    if mm_cfg.ARCHIVER_OBSCURES_EMAILADDRS:
+                        text = re.sub('@', atmark, text)
+                        URL = self.maillist.GetScriptURL(
+                            'listinfo', absolute=1)
+                    else:
+                        URL = 'mailto:' + text
+                    pos = j
+                elif k != -1 and (j > k or j == -1):
+                    text = URL = kr.group(1)
+                    length = len(text)
+                    pos = k
+                else: # j==k
+                    raise ValueError, "j==k: This can't happen!"
+                #length = len(text)
+                #self.message("URL: %s %s %s \n"
+                #             % (CGIescape(L[:pos]), URL, CGIescape(text)))
+                L2 += '%s<A HREF="%s">%s</A>' % (
+                    CGIescape(L[:pos], self.lang),
+                    html_quote(URL), CGIescape(text, self.lang))
+                L = L[pos+length:]
+                jr = emailpat.search(L)
+                kr = urlpat.search(L)
+            if jr is None and kr is None:
+                L = CGIescape(L, self.lang)
+            L = prefix + L2 + L + suffix
+            source[i] = None
+            dest[i] = L
+
+    # Perform Hypermail-style processing of <HTML></HTML> directives
+    # in message bodies.  Lines between <HTML> and </HTML> will be written
+    # out precisely as they are; other lines will be passed to func2
+    # for further processing .
+
+    def __processbody_HTML(self, lines):
+        # XXX need to make this method modify in place
+        source = lines[:]
+        dest = lines
+        l = len(source)
+        i = 0
+        while i < l:
+            while i < l and htmlpat.match(source[i]) is None:
+                i = i + 1
+            if i < l:
+                source[i] = None
+                i = i + 1
+            while i < l and nohtmlpat.match(source[i]) is None:
+                dest[i], source[i] = source[i], None
+                i = i + 1
+            if i < l:
+                source[i] = None
+                i = i + 1
+
+    def format_article(self, article):
+        # called from add_article
+        # TBD: Why do the HTML formatting here and keep it in the
+        # pipermail database?  It makes more sense to do the html
+        # formatting as the article is being written as html and toss
+        # the data after it has been written to the archive file.
+        lines = filter(None, article.body)
+        # Handle <HTML> </HTML> directives
+        if self.ALLOWHTML:
+            self.__processbody_HTML(lines)
+        self.__processbody_URLquote(lines)
+        if not self.SHOWHTML and lines:
+            lines.insert(0, '<PRE>')
+            lines.append('</PRE>')
+        else:
+            # Do fancy formatting here
+            if self.SHOWBR:
+                lines = map(lambda x:x + "<BR>", lines)
+            else:
+                for i in range(0, len(lines)):
+                    s = lines[i]
+                    if s[0:1] in ' \t\n':
+                        lines[i] = '<P>' + s
+        article.html_body = lines
+        return article
+
+    def update_article(self, arcdir, article, prev, next):
+        seq = article.sequence
+        filename = os.path.join(arcdir, article.filename)
+        self.message(C_('Updating HTML for article %(seq)s'))
+        try:
+            f = open(filename)
+            article.loadbody_fromHTML(f)
+            f.close()
+        except IOError, e:
+            if e.errno <> errno.ENOENT: raise
+            self.message(C_('article file %(filename)s is missing!'))
+        article.prev = prev
+        article.next = next
+        omask = os.umask(002)
+        try:
+            f = open(filename, 'w')
+        finally:
+            os.umask(omask)
+        f.write(article.as_html())
+        f.close()

=== added file 'Mailman/Archiver/HyperDatabase.py'
--- Mailman/Archiver/HyperDatabase.py	1970-01-01 00:00:00 +0000
+++ Mailman/Archiver/HyperDatabase.py	2025-02-01 12:21:11 +0000
@@ -0,0 +1,344 @@
+# Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301,
+# USA.
+
+#
+# site modules
+#
+import os
+import marshal
+import time
+import errno
+
+#
+# package/project modules
+#
+import pipermail
+from Mailman import LockFile
+
+CACHESIZE = pipermail.CACHESIZE
+
+try:
+    import cPickle
+    pickle = cPickle
+except ImportError:
+    import pickle
+
+#
+# we're using a python dict in place of
+# of bsddb.btree database.  only defining
+# the parts of the interface used by class HyperDatabase
+# only one thing can access this at a time.
+#
+class DumbBTree:
+    """Stores pickles of Article objects
+
+    This dictionary-like object stores pickles of all the Article
+    objects.  The object itself is stored using marshal.  It would be
+    much simpler, and probably faster, to store the actual objects in
+    the DumbBTree and pickle it.
+
+    TBD: Also needs a more sensible name, like IteratableDictionary or
+    SortedDictionary.
+    """
+
+    def __init__(self, path):
+        self.current_index = 0
+        self.path = path
+        self.lockfile = LockFile.LockFile(self.path + ".lock")
+        self.lock()
+        self.__dirty = 0
+        self.dict = {}
+        self.sorted = []
+        self.load()
+
+    def __repr__(self):
+        return "DumbBTree(%s)" % self.path
+
+    def __sort(self, dirty=None):
+        if self.__dirty == 1 or dirty:
+            self.sorted = self.dict.keys()
+            self.sorted.sort()
+            self.__dirty = 0
+
+    def lock(self):
+        self.lockfile.lock()
+
+    def unlock(self):
+        try:
+            self.lockfile.unlock()
+        except LockFile.NotLockedError:
+            pass
+
+    def __delitem__(self, item):
+        # if first hasn't been called, we can skip the sort
+        if self.current_index == 0:
+            del self.dict[item]
+            self.__dirty = 1
+            return
+        try:
+            ci = self.sorted[self.current_index]
+        except IndexError:
+            ci = None
+        if ci == item:
+            try:
+                ci = self.sorted[self.current_index + 1]
+            except IndexError:
+                ci = None
+        del self.dict[item]
+        self.__sort(dirty=1)
+        if ci is not None:
+            self.current_index = self.sorted.index(ci)
+        else:
+            self.current_index = self.current_index + 1
+
+    def clear(self):
+        # bulk clearing much faster than deleting each item, esp. with the
+        # implementation of __delitem__() above :(
+        self.dict = {}
+
+    def first(self):
+        self.__sort() # guarantee that the list is sorted
+        if not self.sorted:
+            raise KeyError
+        else:
+            key = self.sorted[0]
+            self.current_index = 1
+            return key, self.dict[key]
+
+    def last(self):
+        if not self.sorted:
+            raise KeyError
+        else:
+            key = self.sorted[-1]
+            self.current_index = len(self.sorted) - 1
+            return key, self.dict[key]
+
+    def next(self):
+        try:
+            key = self.sorted[self.current_index]
+        except IndexError:
+            raise KeyError
+        self.current_index = self.current_index + 1
+        return key, self.dict[key]
+
+    def has_key(self, key):
+        return self.dict.has_key(key)
+
+    def set_location(self, loc):
+        index = 0
+        self.__sort()
+        for key in self.sorted:
+            if key[0] == loc:
+                self.current_index = index
+                return key,self.dict[key]
+            index = index + 1
+        raise KeyError(loc)
+
+    def __getitem__(self, item):
+        return self.dict[item]
+
+    def __setitem__(self, item, val):
+        # if first hasn't been called, then we don't need to worry
+        # about sorting again
+        if self.current_index == 0:
+            self.dict[item] = val
+            self.__dirty = 1
+            return
+        try:
+            current_item = self.sorted[self.current_index]
+        except IndexError:
+            current_item = item
+        self.dict[item] = val
+        self.__sort(dirty=1)
+        self.current_index = self.sorted.index(current_item)
+
+    def __len__(self):
+        return len(self.sorted)
+
+    def load(self):
+        try:
+            fp = open(self.path)
+            try:
+                self.dict = marshal.load(fp)
+            finally:
+                fp.close()
+        except IOError, e:
+            if e.errno <> errno.ENOENT: raise
+            pass
+        except EOFError:
+            pass
+        else:
+            self.__sort(dirty=1)
+
+    def close(self):
+        omask = os.umask(007)
+        try:
+            fp = open(self.path, 'w')
+        finally:
+            os.umask(omask)
+        fp.write(marshal.dumps(self.dict))
+        fp.close()
+        self.unlock()
+
+

+# this is lifted straight out of pipermail with
+# the bsddb.btree replaced with above class.
+# didn't use inheritance because of all the
+# __internal stuff that needs to be here -scott
+#
+class HyperDatabase(pipermail.Database):
+    __super_addArticle = pipermail.Database.addArticle
+
+    def __init__(self, basedir, mlist):
+        self.__cache = {}
+        self.__currentOpenArchive = None   # The currently open indices
+        self._mlist = mlist
+        self.basedir = os.path.expanduser(basedir)
+        # Recently added articles, indexed only by message ID
+        self.changed={}
+
+    def firstdate(self, archive):
+        self.__openIndices(archive)
+        date = 'None'
+        try:
+            datekey, msgid = self.dateIndex.first()
+            date = time.asctime(time.localtime(float(datekey[0])))
+        except KeyError:
+            pass
+        return date
+
+    def lastdate(self, archive):
+        self.__openIndices(archive)
+        date = 'None'
+        try:
+            datekey, msgid = self.dateIndex.last()
+            date = time.asctime(time.localtime(float(datekey[0])))
+        except KeyError:
+            pass
+        return date
+
+    def numArticles(self, archive):
+        self.__openIndices(archive)
+        return len(self.dateIndex)
+
+    def addArticle(self, archive, article, subject=None, author=None,
+                   date=None):
+        self.__openIndices(archive)
+        self.__super_addArticle(archive, article, subject, author, date)
+
+    def __openIndices(self, archive):
+        if self.__currentOpenArchive == archive:
+            return
+        self.__closeIndices()
+        arcdir = os.path.join(self.basedir, 'database')
+        omask = os.umask(0)
+        try:
+            try:
+                os.mkdir(arcdir, 02770)
+            except OSError, e:
+                if e.errno <> errno.EEXIST: raise
+        finally:
+            os.umask(omask)
+        for i in ('date', 'author', 'subject', 'article', 'thread'):
+            t = DumbBTree(os.path.join(arcdir, archive + '-' + i))
+            setattr(self, i + 'Index', t)
+        self.__currentOpenArchive = archive
+
+    def __closeIndices(self):
+        for i in ('date', 'author', 'subject', 'thread', 'article'):
+            attr = i + 'Index'
+            if hasattr(self, attr):
+                index = getattr(self, attr)
+                if i == 'article':
+                    if not hasattr(self, 'archive_length'):
+                        self.archive_length = {}
+                    l = len(index)
+                    self.archive_length[self.__currentOpenArchive] = l
+                index.close()
+                delattr(self, attr)
+        self.__currentOpenArchive = None
+
+    def close(self):
+        self.__closeIndices()
+
+    def hasArticle(self, archive, msgid):
+        self.__openIndices(archive)
+        return self.articleIndex.has_key(msgid)
+
+    def setThreadKey(self, archive, key, msgid):
+        self.__openIndices(archive)
+        self.threadIndex[key]=msgid
+
+    def getArticle(self, archive, msgid):
+        self.__openIndices(archive)
+        if not self.__cache.has_key(msgid):
+            # get the pickled object out of the DumbBTree
+            buf = self.articleIndex[msgid]
+            article = self.__cache[msgid] = pickle.loads(buf)
+            # For upgrading older archives
+            article.setListIfUnset(self._mlist)
+        else:
+            article = self.__cache[msgid]
+        return article
+
+    def first(self, archive, index):
+        self.__openIndices(archive)
+        index = getattr(self, index + 'Index')
+        try:
+            key, msgid = index.first()
+            return msgid
+        except KeyError:
+            return None
+
+    def next(self, archive, index):
+        self.__openIndices(archive)
+        index = getattr(self, index + 'Index')
+        try:
+            key, msgid = index.next()
+            return msgid
+        except KeyError:
+            return None
+
+    def getOldestArticle(self, archive, subject):
+        self.__openIndices(archive)
+        subject = subject.lower()
+        try:
+            self.subjectIndex.set_location(subject)
+            key, tempid = self.subjectIndex.next()
+            [subject2, date]= key[:2]
+            if subject!=subject2: return None
+            return tempid
+        except KeyError:
+            return None
+
+    def newArchive(self, archive):
+        pass
+
+    def clearIndex(self, archive, index):
+        self.__openIndices(archive)
+        if hasattr(self.threadIndex, 'clear'):
+            self.threadIndex.clear()
+            return
+        finished=0
+        try:
+            key, msgid=self.threadIndex.first()
+        except KeyError: finished=1
+        while not finished:
+            del self.threadIndex[key]
+            try:
+                key, msgid=self.threadIndex.next()
+            except KeyError: finished=1

=== added file 'Mailman/Archiver/Makefile.in'
--- Mailman/Archiver/Makefile.in	1970-01-01 00:00:00 +0000
+++ Mailman/Archiver/Makefile.in	2025-02-01 12:21:11 +0000
@@ -0,0 +1,73 @@
+# Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software 
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+
+# NOTE: Makefile.in is converted into Makefile by the configure script
+# in the parent directory.  Once configure has run, you can recreate
+# the Makefile by running just config.status.
+
+# Variables set by configure
+
+VPATH=		@srcdir@
+srcdir= 	@srcdir@
+bindir= 	@bindir@
+prefix=   	@prefix@
+exec_prefix=	@exec_prefix@
+DESTDIR=
+
+CC=		@CC@
+CHMOD=  	@CHMOD@
+INSTALL=	@INSTALL@
+
+DEFS=   	@DEFS@
+
+# Customizable but not set by configure
+
+OPT=		@OPT@
+CFLAGS=		$(OPT) $(DEFS)
+PACKAGEDIR= 	$(prefix)/Mailman/Archiver
+SHELL=		/bin/sh
+
+MODULES=	__init__.py Archiver.py HyperArch.py HyperDatabase.py \
+pipermail.py
+
+
+# Modes for directories and executables created by the install
+# process.  Default to group-writable directories but
+# user-only-writable for executables.
+DIRMODE=	775
+EXEMODE=	755
+FILEMODE=	644
+INSTALL_PROGRAM=$(INSTALL) -m $(EXEMODE)
+
+
+# Rules
+
+all:
+
+install: 
+	for f in $(MODULES); \
+	do \
+	    $(INSTALL) -m $(FILEMODE) $(srcdir)/$$f $(DESTDIR)$(PACKAGEDIR); \
+	done
+
+finish:
+
+clean:
+
+distclean:
+	-rm *.pyc
+	-rm Makefile
+

=== added file 'Mailman/Archiver/__init__.py'
--- Mailman/Archiver/__init__.py	1970-01-01 00:00:00 +0000
+++ Mailman/Archiver/__init__.py	2025-02-01 12:21:11 +0000
@@ -0,0 +1,17 @@
+# Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software 
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+
+from Archiver import *

=== added file 'Mailman/Archiver/pipermail.py'
--- Mailman/Archiver/pipermail.py	1970-01-01 00:00:00 +0000
+++ Mailman/Archiver/pipermail.py	2025-02-01 12:21:11 +0000
@@ -0,0 +1,908 @@
+#! /usr/bin/env python
+
+from __future__ import nested_scopes
+
+import mailbox
+import os
+import re
+import sys
+import time
+from email.Utils import parseaddr, parsedate_tz, mktime_tz, formatdate
+import cPickle as pickle
+from cStringIO import StringIO
+from string import lowercase
+
+# Work around for some misguided Python packages that add iso-8859-1
+# accented characters to string.lowercase.
+lowercase = lowercase[:26]
+
+__version__ = '0.09 (Mailman edition)'
+VERSION = __version__
+CACHESIZE = 100    # Number of slots in the cache
+
+from Mailman import mm_cfg
+from Mailman import Errors
+from Mailman.Mailbox import ArchiverMailbox
+from Mailman.Logging.Syslog import syslog
+from Mailman.i18n import _, C_
+
+# True/False
+try:
+    True, False
+except NameError:
+    True = 1
+    False = 0
+
+SPACE = ' '
+
+
+

+msgid_pat = re.compile(r'(<.*>)')
+def strip_separators(s):
+    "Remove quotes or parenthesization from a Message-ID string"
+    if not s:
+        return ""
+    if s[0] in '"<([' and s[-1] in '">)]':
+        s = s[1:-1]
+    return s
+
+smallNameParts = ['van', 'von', 'der', 'de']
+
+def fixAuthor(author):
+    "Canonicalize a name into Last, First format"
+    # If there's a comma, guess that it's already in "Last, First" format
+    if ',' in author:
+        return author
+    L = author.split()
+    i = len(L) - 1
+    if i == 0:
+        return author # The string's one word--forget it
+    if author.upper() == author or author.lower() == author:
+        # Damn, the name is all upper- or lower-case.
+        while i > 0 and L[i-1].lower() in smallNameParts:
+            i = i - 1
+    else:
+        # Mixed case; assume that small parts of the last name will be
+        # in lowercase, and check them against the list.
+        while i>0 and (L[i-1][0] in lowercase or
+                       L[i-1].lower() in smallNameParts):
+            i = i - 1
+    author = SPACE.join(L[-1:] + L[i:-1]) + ', ' + SPACE.join(L[:i])
+    return author
+
+# Abstract class for databases
+
+class DatabaseInterface:
+    def __init__(self): pass
+    def close(self): pass
+    def getArticle(self, archive, msgid): pass
+    def hasArticle(self, archive, msgid): pass
+    def addArticle(self, archive, article, subject=None, author=None,
+                   date=None): pass
+    def firstdate(self, archive): pass
+    def lastdate(self, archive): pass
+    def first(self, archive, index): pass
+    def next(self, archive, index): pass
+    def numArticles(self, archive): pass
+    def newArchive(self, archive): pass
+    def setThreadKey(self, archive, key, msgid): pass
+    def getOldestArticle(self, subject): pass
+
+class Database(DatabaseInterface):
+    """Define the basic sorting logic for a database
+
+    Assumes that the database internally uses dateIndex, authorIndex,
+    etc.
+    """
+
+    # TBD Factor out more of the logic shared between BSDDBDatabase
+    # and HyperDatabase and place it in this class.
+
+    def __init__(self):
+        # This method need not be called by subclasses that do their
+        # own initialization.
+        self.dateIndex = {}
+        self.authorIndex = {}
+        self.subjectIndex = {}
+        self.articleIndex = {}
+        self.changed = {}
+
+    def addArticle(self, archive, article, subject=None, author=None,
+                   date=None):
+        # create the keys; always end w/ msgid which will be unique
+        authorkey = (author or article.author, article.date,
+                     article.msgid)
+        subjectkey = (subject or article.subject, article.date,
+                      article.msgid)
+        datekey = date or article.date, article.msgid
+
+        # Add the new article
+        self.dateIndex[datekey] = article.msgid
+        self.authorIndex[authorkey] = article.msgid
+        self.subjectIndex[subjectkey] = article.msgid
+
+        self.store_article(article)
+        self.changed[archive, article.msgid] = None
+
+        parentID = article.parentID
+        if parentID is not None and self.articleIndex.has_key(parentID):
+            parent = self.getArticle(archive, parentID)
+            myThreadKey = (parent.threadKey + article.date + '.'
+                           + str(article.sequence) + '-')
+        else:
+            myThreadKey = article.date + '.' + str(article.sequence) + '-'
+        article.threadKey = myThreadKey
+        key = myThreadKey, article.msgid
+        self.setThreadKey(archive, key, article.msgid)
+
+    def store_article(self, article):
+        """Store article without message body to save space"""
+        # TBD this is not thread safe!
+        temp = article.body
+        temp2 = article.html_body
+        article.body = []
+        del article.html_body
+        self.articleIndex[article.msgid] = pickle.dumps(article)
+        article.body = temp
+        article.html_body = temp2
+
+
+# The Article class encapsulates a single posting.  The attributes
+# are:
+#
+# sequence   : Sequence number, unique for each article in a set of archives
+# subject    : Subject
+# datestr    : The posting date, in human-readable format
+# date       : The posting date, in purely numeric format
+# headers    : Any other headers of interest
+# author     : The author's name (and possibly organization)
+# email      : The author's e-mail address
+# msgid      : A unique message ID
+# in_reply_to: If != "", this is the msgid of the article being replied to
+# references : A (possibly empty) list of msgid's of earlier articles
+#              in the thread
+# body       : A list of strings making up the message body
+
+class Article:
+    _last_article_time = time.time()
+
+    def __init__(self, message = None, sequence = 0, keepHeaders = []):
+        if message is None:
+            return
+        self.sequence = sequence
+
+        self.parentID = None
+        self.threadKey = None
+        # otherwise the current sequence number is used.
+        id = strip_separators(message['Message-Id'])
+        if id == "":
+            self.msgid = str(self.sequence)
+        else: self.msgid = id
+
+        if message.has_key('Subject'):
+            self.subject = str(message['Subject'])
+        else:
+            self.subject = _('No subject')
+        if self.subject == "": self.subject = _('No subject')
+
+        self._set_date(message)
+
+        # Figure out the e-mail address and poster's name.  Use the From:
+        # field first, followed by Reply-To:
+        self.author, self.email = parseaddr(message.get('From', ''))
+        e = message['Reply-To']
+        if not self.email and e is not None:
+            ignoreauthor, self.email = parseaddr(e)
+        self.email = strip_separators(self.email)
+        self.author = strip_separators(self.author)
+
+        if self.author == "":
+            self.author = self.email
+
+        # Save the In-Reply-To:, References:, and Message-ID: lines
+        #
+        # TBD: The original code does some munging on these fields, which
+        # shouldn't be necessary, but changing this may break code.  For
+        # safety, I save the original headers on different attributes for use
+        # in writing the plain text periodic flat files.
+        self._in_reply_to = message['in-reply-to']
+        self._references = message['references']
+        self._message_id = message['message-id']
+
+        i_r_t = message['In-Reply-To']
+        if i_r_t is None:
+            self.in_reply_to = ''
+        else:
+            match = msgid_pat.search(i_r_t)
+            if match is None: self.in_reply_to = ''
+            else: self.in_reply_to = strip_separators(match.group(1))
+
+        references = message['References']
+        if references is None:
+            self.references = []
+        else:
+            self.references = map(strip_separators, references.split())
+
+        # Save any other interesting headers
+        self.headers = {}
+        for i in keepHeaders:
+            if message.has_key(i):
+                self.headers[i] = message[i]
+
+        # Read the message body
+        s = StringIO(message.get_payload(decode=True)\
+                     or message.as_string().split('\n\n',1)[1])
+        self.body = s.readlines()
+
+    def _set_date(self, message):
+        def floatdate(datestr):
+            if not datestr:
+                return None
+            date = parsedate_tz(datestr)
+            try:
+                date = mktime_tz(date)
+                if (date < 0 or
+                    date - time.time() >
+                        mm_cfg.ARCHIVER_ALLOWABLE_SANE_DATE_SKEW
+                   ):
+                    return None
+                return date
+            except (TypeError, ValueError, OverflowError):
+                return None
+        date = floatdate(message.get('date'))
+        if date is None:
+            date = floatdate(message.get('x-list-received-date'))
+        if date is None:
+            rec_re = re.compile(r'^.*;\s*', re.DOTALL)
+            date = floatdate(rec_re.sub('', message.get('received', '')))
+        if date is None:
+            date = floatdate(re.sub(r'From \s*\S+\s+', '',
+                                    message.get_unixfrom() or '' ))
+        if date is None:
+            date = self._last_article_time + 1
+        self._last_article_time = date
+        self.date = '%011i' % date
+        self.datestr = message.get('date') \
+                       or message.get('x-list-received-date') \
+                       or formatdate(date)
+
+    def __repr__(self):
+        return '<Article ID = '+repr(self.msgid)+'>'
+
+    def finished_update_article(self):
+        pass    
+
+# Pipermail formatter class
+
+class T:
+    DIRMODE = 0755      # Mode to give to created directories
+    FILEMODE = 0644     # Mode to give to created files
+    INDEX_EXT = ".html" # Extension for indexes
+
+    def __init__(self, basedir = None, reload = 1, database = None):
+        # If basedir isn't provided, assume the current directory
+        if basedir is None:
+            self.basedir = os.getcwd()
+        else:
+            basedir = os.path.expanduser(basedir)
+            self.basedir = basedir
+        self.database = database
+
+        # If the directory doesn't exist, create it.  This code shouldn't get
+        # run anymore, we create the directory in Archiver.py.  It should only
+        # get used by legacy lists created that are only receiving their first
+        # message in the HTML archive now -- Marc
+        try:
+            os.stat(self.basedir)
+        except os.error, errdata:
+            errno, errmsg = errdata
+            if errno != 2:
+                raise os.error, errdata
+            else:
+                self.message(C_('Creating archive directory ') + self.basedir)
+                omask = os.umask(0)
+                try:
+                    os.mkdir(self.basedir, self.DIRMODE)
+                finally:
+                    os.umask(omask)
+
+        # Try to load previously pickled state
+        try:
+            if not reload:
+                raise IOError
+            f = open(os.path.join(self.basedir, 'pipermail.pck'), 'r')
+            self.message(C_('Reloading pickled archive state'))
+            d = pickle.load(f)
+            f.close()
+            for key, value in d.items():
+                setattr(self, key, value)
+        except (IOError, EOFError):
+            # No pickled version, so initialize various attributes
+            self.archives = []        # Archives
+            self._dirty_archives = [] # Archives that will have to be updated
+            self.sequence = 0         # Sequence variable used for
+                                      #   numbering articles
+            self.update_TOC = 0       # Does the TOC need updating?
+        #
+        # make the basedir variable work when passed in as an __init__ arg
+        # and different from the one in the pickle.  Let the one passed in
+        # as an __init__ arg take precedence if it's stated.  This way, an
+        # archive can be moved from one place to another and still work.
+        #
+        if basedir != self.basedir:
+            self.basedir = basedir
+
+    def close(self):
+        "Close an archive, save its state, and update any changed archives."
+        self.update_dirty_archives()
+        self.update_TOC = 0
+        self.write_TOC()
+        # Save the collective state
+        self.message(C_('Pickling archive state into ')
+                     + os.path.join(self.basedir, 'pipermail.pck'))
+        self.database.close()
+        del self.database
+
+        omask = os.umask(007)
+        try:
+            f = open(os.path.join(self.basedir, 'pipermail.pck'), 'w')
+        finally:
+            os.umask(omask)
+        pickle.dump(self.getstate(), f)
+        f.close()
+
+    def getstate(self):
+        # can override this in subclass
+        return self.__dict__
+
+    #
+    # Private methods
+    #
+    # These will be neither overridden nor called by custom archivers.
+    #
+
+
+    # Create a dictionary of various parameters that will be passed
+    # to the write_index_{header,footer} functions
+    def __set_parameters(self, archive):
+        # Determine the earliest and latest date in the archive
+        firstdate = self.database.firstdate(archive)
+        lastdate = self.database.lastdate(archive)
+
+        # Get the current time
+        now = time.asctime(time.localtime(time.time()))
+        self.firstdate = firstdate
+        self.lastdate = lastdate
+        self.archivedate = now
+        self.size = self.database.numArticles(archive)
+        self.archive = archive
+        self.version = __version__
+
+    # Find the message ID of an article's parent, or return None
+    # if no parent can be found.
+
+    def __findParent(self, article, children = []):
+            parentID = None
+            if article.in_reply_to:
+                parentID = article.in_reply_to
+            elif article.references:
+                # Remove article IDs that aren't in the archive
+                refs = filter(self.articleIndex.has_key, article.references)
+                if not refs:
+                    return None
+                maxdate = self.database.getArticle(self.archive,
+                                                   refs[0])
+                for ref in refs[1:]:
+                    a = self.database.getArticle(self.archive, ref)
+                    if a.date > maxdate.date:
+                        maxdate = a
+                parentID = maxdate.msgid
+            else:
+                # Look for the oldest matching subject
+                try:
+                    key, tempid = \
+                         self.subjectIndex.set_location(article.subject)
+                    print key, tempid
+                    self.subjectIndex.next()
+                    [subject, date] = key.split('\0')
+                    print article.subject, subject, date
+                    if subject == article.subject and tempid not in children:
+                        parentID = tempid
+                except KeyError:
+                    pass
+            return parentID
+
+    # Update the threaded index completely
+    def updateThreadedIndex(self):
+        # Erase the threaded index
+        self.database.clearIndex(self.archive, 'thread')
+
+        # Loop over all the articles
+        msgid = self.database.first(self.archive, 'date')
+        while msgid is not None:
+            try:
+                article = self.database.getArticle(self.archive, msgid)
+            except KeyError:
+                pass
+            else:
+                if article.parentID is None or \
+                   not self.database.hasArticle(self.archive,
+                                                article.parentID):
+                    # then
+                    pass
+                else:
+                    parent = self.database.getArticle(self.archive,
+                                                    article.parentID)
+                    article.threadKey = (parent.threadKey + article.date + '.'
+                                         + str(article.sequence) + '-')
+                self.database.setThreadKey(self.archive,
+                    (article.threadKey, article.msgid),
+                    msgid)
+            msgid = self.database.next(self.archive, 'date')
+
+    #
+    # Public methods:
+    #
+    # These are part of the public interface of the T class, but will
+    # never be overridden (unless you're trying to do something very new).
+
+    # Update a single archive's indices, whether the archive's been
+    # dirtied or not.
+    def update_archive(self, archive):
+        self.archive = archive
+        self.message(C_("Updating index files for archive [%(archive)s]"))
+        arcdir = os.path.join(self.basedir, archive)
+        self.__set_parameters(archive)
+
+        for hdr in ('Date', 'Subject', 'Author'):
+            self._update_simple_index(hdr, archive, arcdir)
+
+        self._update_thread_index(archive, arcdir)
+
+    def _update_simple_index(self, hdr, archive, arcdir):
+        self.message("  " + hdr)
+        self.type = hdr
+        hdr = hdr.lower()
+
+        self._open_index_file_as_stdout(arcdir, hdr)
+        self.write_index_header()
+        count = 0
+        # Loop over the index entries
+        msgid = self.database.first(archive, hdr)
+        while msgid is not None:
+            try:
+                article = self.database.getArticle(self.archive, msgid)
+            except KeyError:
+                pass
+            else:
+                count = count + 1
+                self.write_index_entry(article)
+            msgid = self.database.next(archive, hdr)
+        # Finish up this index
+        self.write_index_footer()
+        self._restore_stdout()
+
+    def _update_thread_index(self, archive, arcdir):
+        self.message(C_("  Thread"))
+        self._open_index_file_as_stdout(arcdir, "thread")
+        self.type = 'Thread'
+        self.write_index_header()
+
+        # To handle the prev./next in thread pointers, we need to
+        # track articles 5 at a time.
+
+        # Get the first 5 articles
+        L = [None] * 5
+        i = 2
+        msgid = self.database.first(self.archive, 'thread')
+
+        while msgid is not None and i < 5:
+            L[i] = self.database.getArticle(self.archive, msgid)
+            i = i + 1
+            msgid = self.database.next(self.archive, 'thread')
+
+        while L[2] is not None:
+            article = L[2]
+            artkey = None
+            if article is not None:
+                artkey = article.threadKey
+            if artkey is not None:
+                self.write_threadindex_entry(article, artkey.count('-') - 1)
+                if self.database.changed.has_key((archive,article.msgid)):
+                    a1 = L[1]
+                    a3 = L[3]
+                    self.update_article(arcdir, article, a1, a3)
+                    if a3 is not None:
+                        self.database.changed[(archive, a3.msgid)] = None
+                    if a1 is not None:
+                        key = archive, a1.msgid
+                        if not self.database.changed.has_key(key):
+                            self.update_article(arcdir, a1, L[0], L[2])
+                        else:
+                            del self.database.changed[key]
+            if L[0]:
+                L[0].finished_update_article()
+            L = L[1:]                   # Rotate the list
+            if msgid is None:
+                L.append(msgid)
+            else:
+                L.append(self.database.getArticle(self.archive, msgid))
+            msgid = self.database.next(self.archive, 'thread')
+
+        self.write_index_footer()
+        self._restore_stdout()
+
+    def _open_index_file_as_stdout(self, arcdir, index_name):
+        path = os.path.join(arcdir, index_name + self.INDEX_EXT)
+        omask = os.umask(002)
+        try:
+            self.__f = open(path, 'w')
+        finally:
+            os.umask(omask)
+        self.__stdout = sys.stdout
+        sys.stdout = self.__f
+
+    def _restore_stdout(self):
+        sys.stdout = self.__stdout
+        self.__f.close()
+        del self.__f
+        del self.__stdout
+
+    # Update only archives that have been marked as "changed".
+    def update_dirty_archives(self):
+        for i in self._dirty_archives:
+            self.update_archive(i)
+        self._dirty_archives = []
+
+    # Read a Unix mailbox file from the file object <input>,
+    # and create a series of Article objects.  Each article
+    # object will then be archived.
+
+    def _makeArticle(self, msg, sequence):
+        return Article(msg, sequence)
+
+    def processUnixMailbox(self, input, start=None, end=None):
+        mbox = ArchiverMailbox(input, self.maillist)
+        if start is None:
+            start = 0
+        counter = 0
+        if start:
+            mbox.skipping(True)
+        while counter < start:
+            try:
+                m = mbox.next()
+            except Errors.DiscardMessage:
+                continue
+            if m is None:
+                return
+            counter += 1
+        if start:
+            mbox.skipping(False)
+        while 1:
+            try:
+                pos = input.tell()
+                m = mbox.next()
+            except Errors.DiscardMessage:
+                continue
+            except Exception:
+                syslog('error', 'uncaught archiver exception at filepos: %s',
+                       pos)
+                raise
+            if m is None:
+                break
+            if m == '':
+                # It was an unparseable message
+                continue
+            msgid = m.get('message-id', 'n/a')
+            self.message(C_('#%(counter)05d %(msgid)s'))
+            a = self._makeArticle(m, self.sequence)
+            self.sequence += 1
+            self.add_article(a)
+            if end is not None and counter >= end:
+               break
+            counter += 1
+
+    def new_archive(self, archive, archivedir):
+        self.archives.append(archive)
+        self.update_TOC = 1
+        self.database.newArchive(archive)
+        # If the archive directory doesn't exist, create it
+        try:
+            os.stat(archivedir)
+        except os.error, errdata:
+            errno, errmsg = errdata
+            if errno == 2:
+                omask = os.umask(0)
+                try:
+                    os.mkdir(archivedir, self.DIRMODE)
+                finally:
+                    os.umask(omask)
+            else:
+                raise os.error, errdata
+        self.open_new_archive(archive, archivedir)
+
+    def add_article(self, article):
+        archives = self.get_archives(article)
+        if not archives:
+            return
+        if type(archives) == type(''):
+            archives = [archives]
+
+        article.filename = filename = self.get_filename(article)
+        temp = self.format_article(article)
+        for arch in archives:
+            self.archive = arch # why do this???
+            archivedir = os.path.join(self.basedir, arch)
+            if arch not in self.archives:
+                self.new_archive(arch, archivedir)
+
+            # Write the HTML-ized article
+            self.write_article(arch, temp, os.path.join(archivedir,
+                                                        filename))
+
+            if article.decoded.has_key('author'):
+                author = fixAuthor(article.decoded['author'])
+            else:
+                author = fixAuthor(article.author)
+            if article.decoded.has_key('stripped'):
+                subject = article.decoded['stripped'].lower()
+            else:
+                subject = article.subject.lower()
+
+            article.parentID = parentID = self.get_parent_info(arch, article)
+            if parentID:
+                parent = self.database.getArticle(arch, parentID)
+                article.threadKey = (parent.threadKey + article.date + '.'
+                                     + str(article.sequence) + '-')
+            else:
+                article.threadKey = (article.date + '.'
+                                     + str(article.sequence) + '-')
+            key = article.threadKey, article.msgid
+
+            self.database.setThreadKey(arch, key, article.msgid)
+            self.database.addArticle(arch, temp, author=author,
+                                     subject=subject)
+
+            if arch not in self._dirty_archives:
+                self._dirty_archives.append(arch)
+
+    def get_parent_info(self, archive, article):
+        parentID = None
+        if article.in_reply_to:
+            if self.database.hasArticle(archive, article.in_reply_to):
+                # Only use In-Reply-To if it's in the archive.
+                parentID = article.in_reply_to
+        if not parentID and article.references:
+            refs = self._remove_external_references(article.references)
+            if refs:
+                maxdate = self.database.getArticle(archive, refs[0])
+                for ref in refs[1:]:
+                    a = self.database.getArticle(archive, ref)
+                    if a.date > maxdate.date:
+                        maxdate = a
+                parentID = maxdate.msgid
+        if not parentID:
+            # Get the oldest article with a matching subject, and
+            # assume this is a follow-up to that article
+            # But, use the subject that's in the database
+            if article.decoded.has_key('stripped'):
+                subject = article.decoded['stripped'].lower()
+            else:
+                subject = article.subject.lower()
+            parentID = self.database.getOldestArticle(archive, subject)
+
+        if parentID and not self.database.hasArticle(archive, parentID):
+            parentID = None
+        return parentID
+
+    def write_article(self, index, article, path):
+        omask = os.umask(002)
+        try:
+            f = open(path, 'w')
+        finally:
+            os.umask(omask)
+        temp_stdout, sys.stdout = sys.stdout, f
+        self.write_article_header(article)
+        sys.stdout.writelines(article.body)
+        self.write_article_footer(article)
+        sys.stdout = temp_stdout
+        f.close()
+
+    def _remove_external_references(self, refs):
+        keep = []
+        for ref in refs:
+            if self.database.hasArticle(self.archive, ref):
+                keep.append(ref)
+        return keep
+
+    # Abstract methods: these will need to be overridden by subclasses
+    # before anything useful can be done.
+
+    def get_filename(self, article):
+        pass
+    def get_archives(self, article):
+        """Return a list of indexes where the article should be filed.
+        A string can be returned if the list only contains one entry,
+        and the empty list is legal."""
+        pass
+    def format_article(self, article):
+        pass
+    def write_index_header(self):
+        pass
+    def write_index_footer(self):
+        pass
+    def write_index_entry(self, article):
+        pass
+    def write_threadindex_entry(self, article, depth):
+        pass
+    def write_article_header(self, article):
+        pass
+    def write_article_footer(self, article):
+        pass
+    def write_article_entry(self, article):
+        pass
+    def update_article(self, archivedir, article, prev, next):
+        pass
+    def write_TOC(self):
+        pass
+    def open_new_archive(self, archive, dir):
+        pass
+    def message(self, msg):
+        pass
+
+
+class BSDDBdatabase(Database):
+    __super_addArticle = Database.addArticle
+
+    def __init__(self, basedir):
+        self.__cachekeys = []
+        self.__cachedict = {}
+        self.__currentOpenArchive = None # The currently open indices
+        self.basedir = os.path.expanduser(basedir)
+        self.changed = {} # Recently added articles, indexed only by
+                          # message ID
+
+    def firstdate(self, archive):
+        self.__openIndices(archive)
+        date = 'None'
+        try:
+            date, msgid = self.dateIndex.first()
+            date = time.asctime(time.localtime(float(date)))
+        except KeyError:
+            pass
+        return date
+
+    def lastdate(self, archive):
+        self.__openIndices(archive)
+        date = 'None'
+        try:
+            date, msgid = self.dateIndex.last()
+            date = time.asctime(time.localtime(float(date)))
+        except KeyError:
+            pass
+        return date
+
+    def numArticles(self, archive):
+        self.__openIndices(archive)
+        return len(self.dateIndex)
+
+    def addArticle(self, archive, article, subject=None, author=None,
+                   date=None):
+        self.__openIndices(archive)
+        self.__super_addArticle(archive, article, subject, author, date)
+
+    # Open the BSDDB files that are being used as indices
+    # (dateIndex, authorIndex, subjectIndex, articleIndex)
+    def __openIndices(self, archive):
+        if self.__currentOpenArchive == archive:
+            return
+
+        import bsddb
+        self.__closeIndices()
+        arcdir = os.path.join(self.basedir, 'database')
+        omask = os.umask(0)
+        try:
+            try:
+                os.mkdir(arcdir, 02775)
+            except OSError:
+                # BAW: Hmm...
+                pass
+        finally:
+            os.umask(omask)
+        for hdr in ('date', 'author', 'subject', 'article', 'thread'):
+            path = os.path.join(arcdir, archive + '-' + hdr)
+            t = bsddb.btopen(path, 'c')
+            setattr(self, hdr + 'Index', t)
+        self.__currentOpenArchive = archive
+
+    # Close the BSDDB files that are being used as indices (if they're
+    # open--this is safe to call if they're already closed)
+    def __closeIndices(self):
+        if self.__currentOpenArchive is not None:
+            pass
+        for hdr in ('date', 'author', 'subject', 'thread', 'article'):
+            attr = hdr + 'Index'
+            if hasattr(self, attr):
+                index = getattr(self, attr)
+                if hdr == 'article':
+                    if not hasattr(self, 'archive_length'):
+                        self.archive_length = {}
+                    self.archive_length[self.__currentOpenArchive] = len(index)
+                index.close()
+                delattr(self,attr)
+        self.__currentOpenArchive = None
+
+    def close(self):
+        self.__closeIndices()
+    def hasArticle(self, archive, msgid):
+        self.__openIndices(archive)
+        return self.articleIndex.has_key(msgid)
+    def setThreadKey(self, archive, key, msgid):
+        self.__openIndices(archive)
+        self.threadIndex[key] = msgid
+    def getArticle(self, archive, msgid):
+        self.__openIndices(archive)
+        if self.__cachedict.has_key(msgid):
+            self.__cachekeys.remove(msgid)
+            self.__cachekeys.append(msgid)
+            return self.__cachedict[msgid]
+        if len(self.__cachekeys) == CACHESIZE:
+            delkey, self.__cachekeys = (self.__cachekeys[0],
+                                        self.__cachekeys[1:])
+            del self.__cachedict[delkey]
+        s = self.articleIndex[msgid]
+        article = pickle.loads(s)
+        self.__cachekeys.append(msgid)
+        self.__cachedict[msgid] = article
+        return article
+
+    def first(self, archive, index):
+        self.__openIndices(archive)
+        index = getattr(self, index+'Index')
+        try:
+            key, msgid = index.first()
+            return msgid
+        except KeyError:
+            return None
+    def next(self, archive, index):
+        self.__openIndices(archive)
+        index = getattr(self, index+'Index')
+        try:
+            key, msgid = index.next()
+        except KeyError:
+            return None
+        else:
+            return msgid
+
+    def getOldestArticle(self, archive, subject):
+        self.__openIndices(archive)
+        subject = subject.lower()
+        try:
+            key, tempid = self.subjectIndex.set_location(subject)
+            self.subjectIndex.next()
+            [subject2, date] = key.split('\0')
+            if subject != subject2:
+                return None
+            return tempid
+        except KeyError: # XXX what line raises the KeyError?
+            return None
+
+    def newArchive(self, archive):
+        pass
+
+    def clearIndex(self, archive, index):
+        self.__openIndices(archive)
+        index = getattr(self, index+'Index')
+        finished = 0
+        try:
+            key, msgid = self.threadIndex.first()
+        except KeyError:
+            finished = 1
+        while not finished:
+            del self.threadIndex[key]
+            try:
+                key, msgid = self.threadIndex.next()
+            except KeyError:
+                finished = 1
+
+

=== added file 'Mailman/Autoresponder.py'
--- Mailman/Autoresponder.py	1970-01-01 00:00:00 +0000
+++ Mailman/Autoresponder.py	2025-02-01 12:21:11 +0000
@@ -0,0 +1,43 @@
+# Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software 
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+
+"""MailList mixin class managing the autoresponder.
+"""
+
+from Mailman import mm_cfg
+from Mailman.i18n import _
+
+
+

+class Autoresponder:
+    def InitVars(self):
+        # configurable
+        self.autorespond_postings = 0
+        self.autorespond_admin = 0
+        # this value can be
+        #  0 - no autoresponse on the -request line
+        #  1 - autorespond, but discard the original message
+        #  2 - autorespond, and forward the message on to be processed
+        self.autorespond_requests = 0
+        self.autoresponse_postings_text = ''
+        self.autoresponse_admin_text = ''
+        self.autoresponse_request_text = ''
+        self.autoresponse_graceperiod = 90 # days
+        # non-configurable
+        self.postings_responses = {}
+        self.admin_responses = {}
+        self.request_responses = {}
+

=== added file 'Mailman/Bouncer.py'
--- Mailman/Bouncer.py	1970-01-01 00:00:00 +0000
+++ Mailman/Bouncer.py	2025-02-01 12:21:11 +0000
@@ -0,0 +1,354 @@
+# Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301,
+# USA.
+
+"""Handle delivery bounces."""
+
+import sys
+import time
+from types import StringType
+
+from email.MIMEText import MIMEText
+from email.MIMEMessage import MIMEMessage
+
+from Mailman import mm_cfg
+from Mailman import Utils
+from Mailman import Message
+from Mailman import MemberAdaptor
+from Mailman import Pending
+from Mailman.Errors import MMUnknownListError
+from Mailman.Logging.Syslog import syslog
+from Mailman import i18n
+
+EMPTYSTRING = ''
+
+# This constant is supposed to represent the day containing the first midnight
+# after the epoch.  We'll add (0,)*6 to this tuple to get a value appropriate
+# for time.mktime().
+ZEROHOUR_PLUSONEDAY = time.localtime(mm_cfg.days(1))[:3]
+
+def D_(s): return s
+_ = D_
+
+REASONS = {MemberAdaptor.BYBOUNCE: _('due to excessive bounces'),
+           MemberAdaptor.BYUSER: _('by yourself'),
+           MemberAdaptor.BYADMIN: _('by the list administrator'),
+           MemberAdaptor.UNKNOWN: _('for unknown reasons'),
+           }
+
+_ = i18n._
+
+
+

+class _BounceInfo:
+    def __init__(self, member, score, date, noticesleft):
+        self.member = member
+        self.cookie = None
+        self.reset(score, date, noticesleft)
+
+    def reset(self, score, date, noticesleft):
+        self.score = score
+        self.date = date
+        self.noticesleft = noticesleft
+        self.lastnotice = ZEROHOUR_PLUSONEDAY
+
+    def __repr__(self):
+        # For debugging
+        return """\
+<bounce info for member %(member)s
+        current score: %(score)s
+        last bounce date: %(date)s
+        email notices left: %(noticesleft)s
+        last notice date: %(lastnotice)s
+        confirmation cookie: %(cookie)s
+        >""" % self.__dict__
+
+
+

+class Bouncer:
+    def InitVars(self):
+        # Configurable...
+        self.bounce_processing = mm_cfg.DEFAULT_BOUNCE_PROCESSING
+        self.bounce_score_threshold = mm_cfg.DEFAULT_BOUNCE_SCORE_THRESHOLD
+        self.bounce_info_stale_after = mm_cfg.DEFAULT_BOUNCE_INFO_STALE_AFTER
+        self.bounce_you_are_disabled_warnings = \
+            mm_cfg.DEFAULT_BOUNCE_YOU_ARE_DISABLED_WARNINGS
+        self.bounce_you_are_disabled_warnings_interval = \
+            mm_cfg.DEFAULT_BOUNCE_YOU_ARE_DISABLED_WARNINGS_INTERVAL
+        self.bounce_unrecognized_goes_to_list_owner = \
+            mm_cfg.DEFAULT_BOUNCE_UNRECOGNIZED_GOES_TO_LIST_OWNER
+        self.bounce_notify_owner_on_bounce_increment = \
+            mm_cfg.DEFAULT_BOUNCE_NOTIFY_OWNER_ON_BOUNCE_INCREMENT
+        self.bounce_notify_owner_on_disable = \
+            mm_cfg.DEFAULT_BOUNCE_NOTIFY_OWNER_ON_DISABLE
+        self.bounce_notify_owner_on_removal = \
+            mm_cfg.DEFAULT_BOUNCE_NOTIFY_OWNER_ON_REMOVAL
+        # Not configurable...
+        #
+        # This holds legacy member related information.  It's keyed by the
+        # member address, and the value is an object containing the bounce
+        # score, the date of the last received bounce, and a count of the
+        # notifications left to send.
+        self.bounce_info = {}
+        # New style delivery status
+        self.delivery_status = {}
+
+    def registerBounce(self, member, msg, weight=1.0, day=None, sibling=False):
+        if not self.isMember(member):
+            # check regular_include_lists, only one level
+            if not self.regular_include_lists or sibling:
+                return
+            from Mailman.MailList import MailList
+            for listaddr in self.regular_include_lists:
+                listname, hostname = listaddr.split('@')
+                listname = listname.lower()
+                if listname == self.internal_name():
+                    syslog('error',
+                           'Bouncer: %s: Include list self reference',
+                           listname)
+                    continue
+                try:
+                    siblist = None
+                    try:
+                        siblist = MailList(listname)
+                    except MMUnknownListError:
+                        syslog('error',
+                               'Bouncer: %s: Include list "%s" not found.',
+                               self.real_name,
+                               listname)
+                        continue
+                    siblist.registerBounce(member, msg, weight, day,
+                                           sibling=True)
+                    siblist.Save()
+                finally:
+                    if siblist and siblist.Locked():
+                        siblist.Unlock()
+            return
+        info = self.getBounceInfo(member)
+        first_today = True
+        if day is None:
+            # Use today's date
+            day = time.localtime()[:3]
+        if not isinstance(info, _BounceInfo):
+            # This is the first bounce we've seen from this member
+            info = _BounceInfo(member, weight, day,
+                               self.bounce_you_are_disabled_warnings)
+            # setBounceInfo is now called below after check phase.
+            syslog('bounce', '%s: %s bounce score: %s', self.internal_name(),
+                   member, info.score)
+            # Continue to the check phase below
+        elif self.getDeliveryStatus(member) <> MemberAdaptor.ENABLED:
+            # The user is already disabled, so we can just ignore subsequent
+            # bounces.  These are likely due to residual messages that were
+            # sent before disabling the member, but took a while to bounce.
+            syslog('bounce', '%s: %s residual bounce received',
+                   self.internal_name(), member)
+            return
+        elif info.date == day:
+            # We've already scored any bounces for this day, so ignore it.
+            first_today = False
+            syslog('bounce', '%s: %s already scored a bounce for date %s',
+                   self.internal_name(), member,
+                   time.strftime('%d-%b-%Y', day + (0,0,0,0,1,0)))
+            # Continue to check phase below
+        else:
+            # See if this member's bounce information is stale.
+            now = Utils.midnight(day)
+            lastbounce = Utils.midnight(info.date)
+            if lastbounce + self.bounce_info_stale_after < now:
+                # Information is stale, so simply reset it
+                info.reset(weight, day, self.bounce_you_are_disabled_warnings)
+                syslog('bounce', '%s: %s has stale bounce info, resetting',
+                       self.internal_name(), member)
+            else:
+                # Nope, the information isn't stale, so add to the bounce
+                # score and take any necessary action.
+                info.score += weight
+                info.date = day
+                syslog('bounce', '%s: %s current bounce score: %s',
+                       self.internal_name(), member, info.score)
+            # Continue to the check phase below
+        #
+        # Now that we've adjusted the bounce score for this bounce, let's
+        # check to see if the disable-by-bounce threshold has been reached.
+        if info.score >= self.bounce_score_threshold:
+            if mm_cfg.VERP_PROBES:
+                syslog('bounce',
+                   'sending %s list probe to: %s (score %s >= %s)',
+                   self.internal_name(), member, info.score,
+                   self.bounce_score_threshold)
+                self.sendProbe(member, msg)
+                info.reset(0, info.date, info.noticesleft)
+            else:
+                self.disableBouncingMember(member, info, msg)
+        elif self.bounce_notify_owner_on_bounce_increment and first_today:
+            self.__sendAdminBounceNotice(member, msg,
+                                         did=_('bounce score incremented'))
+        # We've set/changed bounce info above.  We now need to tell the
+        # MemberAdaptor to set/update it.  We do it here in case the
+        # MemberAdaptor stores bounce info externally to the list object to
+        # be sure updated information is stored, but we have to be sure the
+        # member wasn't removed.
+        if self.isMember(member):
+            self.setBounceInfo(member, info)
+
+    def disableBouncingMember(self, member, info, msg):
+        # Initialize their confirmation cookie.  If we do it when we get the
+        # first bounce, it'll expire by the time we get the disabling bounce.
+        cookie = self.pend_new(Pending.RE_ENABLE, self.internal_name(), member)
+        info.cookie = cookie
+        # In case the MemberAdaptor stores bounce info externally to
+        # the list, we need to tell it to save the cookie
+        self.setBounceInfo(member, info)
+        # Disable them
+        if mm_cfg.VERP_PROBES:
+            syslog('bounce', '%s: %s disabling due to probe bounce received',
+                   self.internal_name(), member)
+        else:
+            syslog('bounce', '%s: %s disabling due to bounce score %s >= %s',
+                   self.internal_name(), member,
+                   info.score, self.bounce_score_threshold)
+        self.setDeliveryStatus(member, MemberAdaptor.BYBOUNCE)
+        self.sendNextNotification(member)
+        if self.bounce_notify_owner_on_disable:
+            self.__sendAdminBounceNotice(member, msg)
+
+    def __sendAdminBounceNotice(self, member, msg, did=None):
+        # BAW: This is a bit kludgey, but we're not providing as much
+        # information in the new admin bounce notices as we used to (some of
+        # it was of dubious value).  However, we'll provide empty, strange, or
+        # meaningless strings for the unused %()s fields so that the language
+        # translators don't have to provide new templates.
+        if did is None:
+            did = _('disabled')
+        siteowner = Utils.get_site_email(self.host_name)
+        text = Utils.maketext(
+            'bounce.txt',
+            {'listname' : self.real_name,
+             'addr'     : member,
+             'negative' : '',
+             'did'      : did,
+             'but'      : '',
+             'reenable' : '',
+             'owneraddr': siteowner,
+             }, mlist=self)
+        subject = _('Bounce action notification')
+        umsg = Message.UserNotification(self.GetOwnerEmail(),
+                                        siteowner, subject,
+                                        lang=self.preferred_language)
+        # BAW: Be sure you set the type before trying to attach, or you'll get
+        # a MultipartConversionError.
+        umsg.set_type('multipart/mixed')
+        umsg.attach(
+            MIMEText(text, _charset=Utils.GetCharSet(self.preferred_language)))
+        if isinstance(msg, StringType):
+            umsg.attach(MIMEText(msg))
+        else:
+            umsg.attach(MIMEMessage(msg))
+        umsg.send(self)
+
+    def sendNextNotification(self, member):
+        global _
+        info = self.getBounceInfo(member)
+        if info is None:
+            return
+        reason = self.getDeliveryStatus(member)
+        if info.noticesleft <= 0:
+            # BAW: Remove them now, with a notification message
+            _ = D_
+            self.ApprovedDeleteMember(
+                member, _('disabled address'),
+                admin_notif=self.bounce_notify_owner_on_removal,
+                userack=1)
+            _ = i18n._
+            # Expunge the pending cookie for the user.  We throw away the
+            # returned data.
+            self.pend_confirm(info.cookie)
+            if reason == MemberAdaptor.BYBOUNCE:
+                syslog('bounce', '%s: %s deleted after exhausting notices',
+                       self.internal_name(), member)
+            syslog('subscribe', '%s: %s auto-unsubscribed [reason: %s]',
+                   self.internal_name(), member,
+                   {MemberAdaptor.BYBOUNCE: 'BYBOUNCE',
+                    MemberAdaptor.BYUSER: 'BYUSER',
+                    MemberAdaptor.BYADMIN: 'BYADMIN',
+                    MemberAdaptor.UNKNOWN: 'UNKNOWN'}.get(
+                reason, 'invalid value'))
+            return
+        # Send the next notification
+        confirmurl = '%s/%s' % (self.GetScriptURL('confirm', absolute=1),
+                                info.cookie)
+        optionsurl = self.GetOptionsURL(member, absolute=1)
+        reqaddr = self.GetRequestEmail()
+        lang = self.getMemberLanguage(member)
+        txtreason = REASONS.get(reason)
+        if txtreason is None:
+            txtreason = _('for unknown reasons')
+        else:
+            txtreason = _(txtreason)
+        # Give a little bit more detail on bounce disables
+        if reason == MemberAdaptor.BYBOUNCE:
+            date = time.strftime('%d-%b-%Y',
+                                 time.localtime(Utils.midnight(info.date)))
+            extra = _(' The last bounce received from you was dated %(date)s')
+            txtreason += extra
+        text = Utils.maketext(
+            'disabled.txt',
+            {'listname'   : self.real_name,
+             'noticesleft': info.noticesleft,
+             'confirmurl' : confirmurl,
+             'optionsurl' : optionsurl,
+             'password'   : self.getMemberPassword(member),
+             'owneraddr'  : self.GetOwnerEmail(),
+             'reason'     : txtreason,
+             }, lang=lang, mlist=self)
+        msg = Message.UserNotification(member, reqaddr, text=text, lang=lang)
+        # BAW: See the comment in MailList.py ChangeMemberAddress() for why we
+        # set the Subject this way.
+        del msg['subject']
+        msg['Subject'] = 'confirm ' + info.cookie
+        # Send without Precedence: bulk.  Bug #808821.
+        msg.send(self, noprecedence=True)
+        info.noticesleft -= 1
+        info.lastnotice = time.localtime()[:3]
+        # In case the MemberAdaptor stores bounce info externally to
+        # the list, we need to tell it to update
+        self.setBounceInfo(member, info)
+
+    def BounceMessage(self, msg, msgdata, e=None):
+        # Bounce a message back to the sender, with an error message if
+        # provided in the exception argument.
+        sender = msg.get_sender()
+        subject = msg.get('subject', _('(no subject)'))
+        subject = Utils.oneline(subject,
+                                Utils.GetCharSet(self.preferred_language))
+        if e is None:
+            notice = _('[No bounce details are available]')
+        else:
+            notice = _(e.notice())
+        # Currently we always craft bounces as MIME messages.
+        bmsg = Message.UserNotification(msg.get_sender(),
+                                        self.GetOwnerEmail(),
+                                        subject,
+                                        lang=self.preferred_language)
+        # BAW: Be sure you set the type before trying to attach, or you'll get
+        # a MultipartConversionError.
+        bmsg.set_type('multipart/mixed')
+        txt = MIMEText(notice,
+                       _charset=Utils.GetCharSet(self.preferred_language))
+        bmsg.attach(txt)
+        bmsg.attach(MIMEMessage(msg))
+        bmsg.send(self)

=== added directory 'Mailman/Bouncers'
=== added file 'Mailman/Bouncers/AOL.py'
--- Mailman/Bouncers/AOL.py	1970-01-01 00:00:00 +0000
+++ Mailman/Bouncers/AOL.py	2025-02-01 12:21:11 +0000
@@ -0,0 +1,45 @@
+# Copyright (C) 2009-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301,
+# USA.
+
+"""Recognizes a class of messages from AOL that report only Screen Name."""
+
+import re
+from email.Utils import parseaddr
+
+scre = re.compile('mail to the following recipients could not be delivered')
+
+def process(msg):
+    if msg.get_content_type() <> 'text/plain':
+        return
+    if not parseaddr(msg.get('from', ''))[1].lower().endswith('@aol.com'):
+        return
+    addrs = []
+    found = False
+    for line in msg.get_payload(decode=True).splitlines():
+        if scre.search(line):
+            found = True
+            continue
+        if found:
+            local = line.strip()
+            if local:
+                if re.search(r'\s', local):
+                    break
+                if re.search('@', local):
+                    addrs.append(local)
+                else:
+                    addrs.append('[e-mail]' % local)
+    return addrs

=== added file 'Mailman/Bouncers/BouncerAPI.py'
--- Mailman/Bouncers/BouncerAPI.py	1970-01-01 00:00:00 +0000
+++ Mailman/Bouncers/BouncerAPI.py	2025-02-01 12:21:11 +0000
@@ -0,0 +1,70 @@
+# Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301,
+# USA.
+
+"""Contains all the common functionality for msg bounce scanning API.
+
+This module can also be used as the basis for a bounce detection testing
+framework.  When run as a script, it expects two arguments, the listname and
+the filename containing the bounce message.
+
+"""
+
+import sys
+
+from Mailman.Logging.Syslog import syslog
+
+# If a bounce detector returns Stop, that means to just discard the message.
+# An example is warning messages for temporary delivery problems.  These
+# shouldn't trigger a bounce notification, but we also don't want to send them
+# on to the list administrator.
+class _Stop:
+    pass
+Stop = _Stop()
+
+
+BOUNCE_PIPELINE = [
+    'DSN',
+    'Qmail',
+    'Postfix',
+    'Yahoo',
+    'Caiwireless',
+    'Exchange',
+    'Exim',
+    'Netscape',
+    'Compuserve',
+    'Microsoft',
+    'GroupWise',
+    'SMTP32',
+    'SimpleMatch',
+    'SimpleWarning',
+    'Yale',
+    'LLNL',
+    'AOL',
+    ]
+
+
+

+# msg must be a mimetools.Message
+def ScanMessages(mlist, msg):
+    for module in BOUNCE_PIPELINE:
+        modname = 'Mailman.Bouncers.' + module
+        __import__(modname)
+        addrs = sys.modules[modname].process(msg)
+        if addrs:
+            # Return addrs even if it is Stop. BounceRunner needs this info.
+            return addrs
+    return []

=== added file 'Mailman/Bouncers/Caiwireless.py'
--- Mailman/Bouncers/Caiwireless.py	1970-01-01 00:00:00 +0000
+++ Mailman/Bouncers/Caiwireless.py	2025-02-01 12:21:11 +0000
@@ -0,0 +1,45 @@
+# Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software 
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+
+"""Parse mystery style generated by MTA at caiwireless.net."""
+
+import re
+import email
+from cStringIO import StringIO
+
+tcre = re.compile(r'the following recipients did not receive this message:',
+                  re.IGNORECASE)
+acre = re.compile(r'<(?P<addr>[^>]*)>')
+
+
+

+def process(msg):
+    if msg.get_content_type() <> 'multipart/mixed':
+        return None
+    # simple state machine
+    #     0 == nothing seen
+    #     1 == tag line seen
+    state = 0
+    # This format thinks it's a MIME, but it really isn't
+    for line in email.Iterators.body_line_iterator(msg):
+        line = line.strip()
+        if state == 0 and tcre.match(line):
+            state = 1
+        elif state == 1 and line:
+            mo = acre.match(line)
+            if not mo:
+                return None
+            return [mo.group('addr')]

=== added file 'Mailman/Bouncers/Compuserve.py'
--- Mailman/Bouncers/Compuserve.py	1970-01-01 00:00:00 +0000
+++ Mailman/Bouncers/Compuserve.py	2025-02-01 12:21:11 +0000
@@ -0,0 +1,45 @@
+# Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software 
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+
+"""Compuserve has its own weird format for bounces."""
+
+import re
+import email
+
+dcre = re.compile(r'your message could not be delivered', re.IGNORECASE)
+acre = re.compile(r'Invalid receiver address: (?P<addr>.*)')
+
+
+

+def process(msg):
+    # simple state machine
+    #    0 = nothing seen yet
+    #    1 = intro line seen
+    state = 0
+    addrs = []
+    for line in email.Iterators.body_line_iterator(msg):
+        if state == 0:
+            mo = dcre.search(line)
+            if mo:
+                state = 1
+        elif state == 1:
+            mo = dcre.search(line)
+            if mo:
+                break
+            mo = acre.search(line)
+            if mo:
+                addrs.append(mo.group('addr'))
+    return addrs

=== added file 'Mailman/Bouncers/DSN.py'
--- Mailman/Bouncers/DSN.py	1970-01-01 00:00:00 +0000
+++ Mailman/Bouncers/DSN.py	2025-02-01 12:21:11 +0000
@@ -0,0 +1,88 @@
+# Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301,
+# USA.
+
+"""Parse RFC 3464 (i.e. DSN) bounce formats.
+
+RFC 3464 obsoletes 1894 which was the old DSN standard.  This module has not
+been audited for differences between the two.
+"""
+
+from email.Iterators import typed_subpart_iterator
+from email.Utils import parseaddr
+from cStringIO import StringIO
+
+from Mailman.Bouncers.BouncerAPI import Stop
+
+try:
+    True, False
+except NameError:
+    True = 1
+    False = 0
+
+
+

+def process(msg):
+    # Iterate over each message/delivery-status subpart
+    addrs = []
+    for part in typed_subpart_iterator(msg, 'message', 'delivery-status'):
+        if not part.is_multipart():
+            # Huh?
+            continue
+        # Each message/delivery-status contains a list of Message objects
+        # which are the header blocks.  Iterate over those too.
+        for msgblock in part.get_payload():
+            # We try to dig out the Original-Recipient (which is optional) and
+            # Final-Recipient (which is mandatory, but may not exactly match
+            # an address on our list).  Some MTA's also use X-Actual-Recipient
+            # as a synonym for Original-Recipient, but some apparently use
+            # that for other purposes :(
+            #
+            # Also grok out Action so we can do something with that too.
+            action = msgblock.get('action', '').lower()
+            # Some MTAs have been observed that put comments on the action.
+            if action.startswith('delayed'):
+                return Stop
+            # opensmtpd uses non-compliant Action: error.
+            if not (action.startswith('fail') or action.startswith('error')):
+                # Some non-permanent failure, so ignore this block
+                continue
+            params = []
+            foundp = False
+            for header in ('original-recipient', 'final-recipient'):
+                for k, v in msgblock.get_params([], header):
+                    if k.lower() == 'rfc822':
+                        foundp = True
+                    else:
+                        params.append(k)
+                if foundp:
+                    # Note that params should already be unquoted.
+                    addrs.extend(params)
+                    break
+                else:
+                    # MAS: This is a kludge, but SMTP-GATEWAY01.intra.home.dk
+                    # has a final-recipient with an angle-addr and no
+                    # address-type parameter at all. Non-compliant, but ...
+                    for param in params:
+                        if param.startswith('<') and param.endswith('>'):
+                            addrs.append(param[1:-1])
+    # Uniquify
+    rtnaddrs = {}
+    for a in addrs:
+        if a is not None:
+            realname, a = parseaddr(a)
+            rtnaddrs[a] = True
+    return rtnaddrs.keys()

=== added file 'Mailman/Bouncers/Exchange.py'
--- Mailman/Bouncers/Exchange.py	1970-01-01 00:00:00 +0000
+++ Mailman/Bouncers/Exchange.py	2025-02-01 12:21:11 +0000
@@ -0,0 +1,47 @@
+# Copyright (C) 2002-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+
+"""Recognizes (some) Microsoft Exchange formats."""
+
+import re
+import email.Iterators
+
+scre = re.compile('did not reach the following recipient')
+ecre = re.compile('MSEXCH:')
+a1cre = re.compile('SMTP=(?P<addr>[^;]+); on ')
+a2cre = re.compile('(?P<addr>[^ ]+) on ')
+
+
+

+def process(msg):
+    addrs = {}
+    it = email.Iterators.body_line_iterator(msg)
+    # Find the start line
+    for line in it:
+        if scre.search(line):
+            break
+    else:
+        return []
+    # Search each line until we hit the end line
+    for line in it:
+        if ecre.search(line):
+            break
+        mo = a1cre.search(line)
+        if not mo:
+            mo = a2cre.search(line)
+        if mo:
+            addrs[mo.group('addr')] = 1
+    return addrs.keys()

=== added file 'Mailman/Bouncers/Exim.py'
--- Mailman/Bouncers/Exim.py	1970-01-01 00:00:00 +0000
+++ Mailman/Bouncers/Exim.py	2025-02-01 12:21:11 +0000
@@ -0,0 +1,30 @@
+# Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software 
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+
+"""Parse bounce messages generated by Exim.
+
+Exim adds an X-Failed-Recipients: header to bounce messages containing
+an `addresslist' of failed addresses.
+
+"""
+
+from email.Utils import getaddresses
+
+
+

+def process(msg):
+    all = msg.get_all('x-failed-recipients', [])
+    return [a for n, a in getaddresses(all)]

=== added file 'Mailman/Bouncers/GroupWise.py'
--- Mailman/Bouncers/GroupWise.py	1970-01-01 00:00:00 +0000
+++ Mailman/Bouncers/GroupWise.py	2025-02-01 12:21:11 +0000
@@ -0,0 +1,72 @@
+# Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software 
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+
+"""This appears to be the format for Novell GroupWise and NTMail
+
+X-Mailer: Novell GroupWise Internet Agent 5.5.3.1
+X-Mailer: NTMail v4.30.0012
+X-Mailer: Internet Mail Service (5.5.2653.19)
+"""
+
+import re
+from email.Message import Message
+from cStringIO import StringIO
+
+acre = re.compile(r'<(?P<addr>[^>]*)>')
+
+
+

+def find_textplain(msg):
+    if msg.get_content_type() == 'text/plain':
+        return msg
+    if msg.is_multipart:
+        for part in msg.get_payload():
+            if not isinstance(part, Message):
+                continue
+            ret = find_textplain(part)
+            if ret:
+                return ret
+    return None
+
+
+

+def process(msg):
+    if msg.get_content_type() <> 'multipart/mixed' or not msg['x-mailer']:
+        return None
+    if msg['x-mailer'][:3].lower() not in ('nov', 'ntm', 'int'):
+        return None
+    addrs = {}
+    # find the first text/plain part in the message
+    textplain = find_textplain(msg)
+    if not textplain:
+        return None
+    body = StringIO(textplain.get_payload())
+    while 1:
+        line = body.readline()
+        if not line:
+            break
+        mo = acre.search(line)
+        if mo:
+            addrs[mo.group('addr')] = 1
+        elif '@' in line:
+            i = line.find(' ')
+            if i == 0:
+                continue
+            if i < 0:
+                addrs[line] = 1
+            else:
+                addrs[line[:i]] = 1
+    return addrs.keys()

=== added file 'Mailman/Bouncers/LLNL.py'
--- Mailman/Bouncers/LLNL.py	1970-01-01 00:00:00 +0000
+++ Mailman/Bouncers/LLNL.py	2025-02-01 12:21:11 +0000
@@ -0,0 +1,31 @@
+# Copyright (C) 2001-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software 
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+
+"""LLNL's custom Sendmail bounce message."""
+
+import re
+import email
+
+acre = re.compile(r',\s*(?P<addr>\S+@[^,]+),', re.IGNORECASE)
+
+
+

+def process(msg):
+    for line in email.Iterators.body_line_iterator(msg):
+        mo = acre.search(line)
+        if mo:
+            return [mo.group('addr')]
+    return []

=== added file 'Mailman/Bouncers/Makefile.in'
--- Mailman/Bouncers/Makefile.in	1970-01-01 00:00:00 +0000
+++ Mailman/Bouncers/Makefile.in	2025-02-01 12:21:11 +0000
@@ -0,0 +1,75 @@
+# Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software 
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+
+# NOTE: Makefile.in is converted into Makefile by the configure script
+# in the parent directory.  Once configure has run, you can recreate
+# the Makefile by running just config.status.
+
+# Variables set by configure
+
+VPATH=		@srcdir@
+srcdir= 	@srcdir@
+bindir= 	@bindir@
+prefix=   	@prefix@
+exec_prefix=	@exec_prefix@
+DESTDIR=
+
+CC=		@CC@
+CHMOD=  	@CHMOD@
+INSTALL=	@INSTALL@
+
+DEFS=   	@DEFS@
+
+# Customizable but not set by configure
+
+OPT=		@OPT@
+CFLAGS=		$(OPT) $(DEFS)
+PACKAGEDIR= 	$(prefix)/Mailman/Bouncers
+SHELL=		/bin/sh
+
+MODULES=	*.py
+
+# Modes for directories and executables created by the install
+# process.  Default to group-writable directories but
+# user-only-writable for executables.
+DIRMODE=	775
+EXEMODE=	755
+FILEMODE=	644
+INSTALL_PROGRAM=$(INSTALL) -m $(EXEMODE)
+
+
+# Rules
+
+all:
+
+install: 
+	for f in $(MODULES); \
+	do \
+	    $(INSTALL) -m $(FILEMODE) $(srcdir)/$$f $(DESTDIR)$(PACKAGEDIR); \
+	done
+
+finish:
+
+clean:
+
+distclean:
+	-rm *.pyc
+	-rm Makefile
+
+
+# Local Variables:
+# indent-tabs-mode: t
+# End:

=== added file 'Mailman/Bouncers/Microsoft.py'
--- Mailman/Bouncers/Microsoft.py	1970-01-01 00:00:00 +0000
+++ Mailman/Bouncers/Microsoft.py	2025-02-01 12:21:11 +0000
@@ -0,0 +1,53 @@
+# Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+
+"""Microsoft's `SMTPSVC' nears I kin tell."""
+
+import re
+from cStringIO import StringIO
+from types import ListType
+
+scre = re.compile(r'transcript of session follows', re.IGNORECASE)
+
+
+

+def process(msg):
+    if msg.get_content_type() <> 'multipart/mixed':
+        return None
+    # Find the first subpart, which has no MIME type
+    try:
+        subpart = msg.get_payload(0)
+    except IndexError:
+        # The message *looked* like a multipart but wasn't
+        return None
+    data = subpart.get_payload()
+    if isinstance(data, ListType):
+        # The message is a multi-multipart, so not a matching bounce
+        return None
+    body = StringIO(data)
+    state = 0
+    addrs = []
+    while 1:
+        line = body.readline()
+        if not line:
+            break
+        if state == 0:
+            if scre.search(line):
+                state = 1
+        if state == 1:
+            if '@' in line:
+                addrs.append(line)
+    return addrs

=== added file 'Mailman/Bouncers/Netscape.py'
--- Mailman/Bouncers/Netscape.py	1970-01-01 00:00:00 +0000
+++ Mailman/Bouncers/Netscape.py	2025-02-01 12:21:11 +0000
@@ -0,0 +1,88 @@
+# Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software 
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+
+"""Netscape Messaging Server bounce formats.
+
+I've seen at least one NMS server version 3.6 (envy.gmp.usyd.edu.au) bounce
+messages of this format.  Bounces come in DSN MIME format, but don't include
+any -Recipient: headers.  Gotta just parse the text :(
+
+NMS 4.1 (dfw-smtpin1.email.verio.net) seems even worse, but we'll try to
+decipher the format here too.
+
+"""
+
+import re
+from cStringIO import StringIO
+
+pcre = re.compile(
+    r'This Message was undeliverable due to the following reason:',
+    re.IGNORECASE)
+
+acre = re.compile(
+    r'(?P<reply>please reply to)?.*<(?P<addr>[^>]*)>',
+    re.IGNORECASE)
+
+
+

+def flatten(msg, leaves):
+    # give us all the leaf (non-multipart) subparts
+    if msg.is_multipart():
+        for part in msg.get_payload():
+            flatten(part, leaves)
+    else:
+        leaves.append(msg)
+
+
+

+def process(msg):
+    # Sigh.  Some show NMS 3.6's show
+    #     multipart/report; report-type=delivery-status
+    # and some show
+    #     multipart/mixed;
+    if not msg.is_multipart():
+        return None
+    # We're looking for a text/plain subpart occuring before a
+    # message/delivery-status subpart.
+    plainmsg = None
+    leaves = []
+    flatten(msg, leaves)
+    for i, subpart in zip(range(len(leaves)-1), leaves):
+        if subpart.get_content_type() == 'text/plain':
+            plainmsg = subpart
+            break
+    if not plainmsg:
+        return None
+    # Total guesswork, based on captured examples...
+    body = StringIO(plainmsg.get_payload())
+    addrs = []
+    while 1:
+        line = body.readline()
+        if not line:
+            break
+        mo = pcre.search(line)
+        if mo:
+            # We found a bounce section, but I have no idea what the official
+            # format inside here is.  :(  We'll just search for <addr>
+            # strings.
+            while 1:
+                line = body.readline()
+                if not line:
+                    break
+                mo = acre.search(line)
+                if mo and not mo.group('reply'):
+                    addrs.append(mo.group('addr'))
+    return addrs

=== added file 'Mailman/Bouncers/Postfix.py'
--- Mailman/Bouncers/Postfix.py	1970-01-01 00:00:00 +0000
+++ Mailman/Bouncers/Postfix.py	2025-02-01 12:21:11 +0000
@@ -0,0 +1,85 @@
+# Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+
+"""Parse bounce messages generated by Postfix.
+
+This also matches something called `Keftamail' which looks just like Postfix
+bounces with the word Postfix scratched out and the word `Keftamail' written
+in in crayon.
+
+It also matches something claiming to be `The BNS Postfix program', and
+`SMTP_Gateway'.  Everybody's gotta be different, huh?
+"""
+
+import re
+from cStringIO import StringIO
+
+
+

+def flatten(msg, leaves):
+    # give us all the leaf (non-multipart) subparts
+    if msg.is_multipart():
+        for part in msg.get_payload():
+            flatten(part, leaves)
+    else:
+        leaves.append(msg)
+
+
+

+# are these heuristics correct or guaranteed?
+pcre = re.compile(r'[ \t]*the\s*(bns)?\s*(postfix|keftamail|smtp_gateway)',
+                  re.IGNORECASE)
+rcre = re.compile(r'failure reason:$', re.IGNORECASE)
+acre = re.compile(r'<(?P<addr>[^>]*)>:')
+
+def findaddr(msg):
+    addrs = []
+    body = StringIO(msg.get_payload())
+    # simple state machine
+    #     0 == nothing found
+    #     1 == salutation found
+    state = 0
+    while 1:
+        line = body.readline()
+        if not line:
+            break
+        # preserve leading whitespace
+        line = line.rstrip()
+        # yes use match to match at beginning of string
+        if state == 0 and (pcre.match(line) or rcre.match(line)):
+            state = 1
+        elif state == 1 and line:
+            mo = acre.search(line)
+            if mo:
+                addrs.append(mo.group('addr'))
+            # probably a continuation line
+    return addrs
+
+
+

+def process(msg):
+    if msg.get_content_type() not in ('multipart/mixed', 'multipart/report'):
+        return None
+    # We're looking for the plain/text subpart with a Content-Description: of
+    # `notification'.
+    leaves = []
+    flatten(msg, leaves)
+    for subpart in leaves:
+        if subpart.get_content_type() == 'text/plain' and \
+           subpart.get('content-description', '').lower() == 'notification':
+            # then...
+            return findaddr(subpart)
+    return None

=== added file 'Mailman/Bouncers/Qmail.py'
--- Mailman/Bouncers/Qmail.py	1970-01-01 00:00:00 +0000
+++ Mailman/Bouncers/Qmail.py	2025-02-01 12:21:11 +0000
@@ -0,0 +1,74 @@
+# Copyright (C) 1998-2018 by the Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301,
+# USA.
+
+"""Parse bounce messages generated by qmail.
+
+Qmail actually has a standard, called QSBMF (qmail-send bounce message
+format), as described in
+
+    http://cr.yp.to/proto/qsbmf.txt
+
+This module should be conformant.
+
+"""
+
+import re
+import email.Iterators
+
+# Other (non-standard?) intros have been observed in the wild.
+introtags = [
+    'Hi. This is the',
+    'Hi. The MTA program at',
+    "We're sorry. There's a problem",
+    'Check your send e-mail address.',
+    'This is the mail delivery agent at',
+    'Unfortunately, your mail was not delivered',
+    'Your mail message to the following',
+    ]
+acre = re.compile(r'<(?P<addr>[^>]*)>:')
+