Dovecot panics when sieve filter outputs much data

Status changed to 'Confirmed' because the bug affects multiple users.

Thank you for filing this bug report! We will add it to our queue and look to provide a fix soon.

Ok, this is broken in Yakkety as well (added a bug task) as it uses the same pigeonhole version.
The latest Debian release upgraded pigeonhole and by that can be considered fixed.
dovecot (1:2.2.25-1) unstable; urgency=medium
  * [cc29a81] Imported Upstream version 2.2.25
  * [d19bcca] Updated pigeonhole patch to 0.4.14
  * [16db179] Merged in some features of the Ubuntu dovecot package.
    + dovecot-core: added lsb-base dependency.
    + dovecot-core: Added apport hook.
    + dovecot-imapd,dovecot-pop3d: Added ufw profiles.
    Thanks to Christian Erhardt <email address hidden>
    (Closes: #828864)

Here an outlined plan of action to follow SRU policy to be available in the Dev release first:

1. merge latest dovecot to fix in Zesty (also a chance to drop more delta)
2. SRU for Xenial
3. SRU for Yakkety?

@M97N - for Yakkety SRU consideration, is that a likely case to happen (more than 60k)?
Also since this requires the special setup - would you be willing to verify this in Xenial and Yakkety once made available?

@ChristianEhrhardt - Yes, it is sure that dovecot panics for e-mails of this size.
We encountered the issue with e-mails which are a few kilobytes smaller, but the exact threshold
might vary (e.g. depending on 'server speed').
All in all 60K seems to be the right magnitude.
I would also like to mention that the bug occurs independent of the executed filter.
A simple filter that would read the mail from stdin and pass it to stdout without any modification would be sufficient to trigger the described behavior.

We could verify a fix / a newer version against our setup in Xenial and Yakkety.

FYI - I realized that the latest dovecot FTBFS on Ubuntu.
So some extra work, I bed a pardon this takes a few extra days depending how much I get to it.

The related FTBFS is fixed in 1636781, now waiting for a few things to settle on zesty release that just opened to be able to go on.

It would be great to have some early verification for what I plan to bring to zesty, so I build the same for yakkety and made it available at a ppa: https://launchpad.net/~paelzer/+archive/ubuntu/bug-1633220-dovecot

If with your setup that triggers the issue, you could confirm that this fixes it it would help me to go on with that and then also prep the backport work for the older releases.

This bug was fixed in the package dovecot - 1:2.2.25-1ubuntu1

---------------
dovecot (1:2.2.25-1ubuntu1) zesty; urgency=medium

  * Merge with Debian (LP: #1633220); Remaining Changes:
    + Add updated autopkgtest to debian/tests/*.
    + Drop build dependency on libstemmer-dev (universe)
    + Use Snakeoil SSL certificates by default
      - d/control: Depend on ssl-cert
    + Add mail-stack-delivery
      - add package in d/rules, d/control
      - add d/*mail-stack-delivery* maintainer scripts and default conf
      - d/mail-stack-delivery.preinst: Move previously installed backups and
        config files to a new package namespace.
      - d/mail-stack-delivery.README.Debian clarified use of configuration files
    + Disable dovecot-lucene plugin as it had various issues, has universe
      dependencies and is deprecated in favor of solr anyway.
    + handle conffile removal of /etc/init/dovecot.conf (due to dropping
      upstart). Can be removed once no upgrade path from <yakkety is left.
  * Added changes:
    + Fix FTBFS of dovecot 2.2.25 in Ubuntu due to being incompatible with
      -Bsymbolic-functions linker flag (LP: #1636781).
  * Dropped Changes as they got accepted in Debian:
    + add lsb base dependency to ensure debian/dovecot-core.dovecot.init is
      working correctly
    + Add ufw integration:
      - d/dovecot-core.ufw.profile: new ufw profile.
      - d/rules: install profile in dovecot-core.
      - d/control: dovecot-core - suggest ufw.
    + Add apport hook:
      - d/rules, d/source_dovecot.py
    + Remove lintian override for drac

 -- Christian Ehrhardt <email address hidden> Tue, 25 Oct 2016 13:12:40 +0200

Now that this is available in latest Dev Release I prepared debdiffs and SRU template for Xenial and Yakkety.

I added Debdiffs here as reference.

And I Uploaded to unapproved queue for the SRU Team to consider.

@M97N - The SRU Team will post here once it (hopefully) gets to -proposed so you can then verify the fix

Hello M97N, or anyone else affected,

Accepted dovecot into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dovecot/1:2.2.24-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Just saw the Xenial reject based on other changes that went in and I missed - re-basing ...

actually I think it was correct, need to discuss that

Hello M97N, or anyone else affected,

Accepted dovecot into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dovecot/2.2.22-1ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hi M97N,
you were very responsive before so this might just have been dropped accidentially.
I wanted to ping you again if you could do a verification on the proposed package as oulined in the former post.
Only after that it will be made available to everybody.

@ChristianEhrhardt - Thanks for the ping and sorry for delay. We scheduled the verification of the proposed fix for friday evening (CET) now.

We did the verification for xenial and yakkety. Everything works fine.
Find the version information for both systems below.

--- xenial
$ uname -a
  Linux ubuntest 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ doveconf | head -n 2
  # 2.2.22 (fe789d2): /etc/dovecot/dovecot.conf
  # Pigeonhole version 0.4.13 (7b14904)

$ dpkg -s dovecot-core
  Package: dovecot-core
  Version: 1:2.2.22-1ubuntu2.2

--- yakkety
$ uname -a
  Linux yakketest 4.8.0-27-generic #29-Ubuntu SMP Thu Oct 20 21:03:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$ doveconf | head -n 2
  # 2.2.24 (a82c823): /etc/dovecot/dovecot.conf
  # Pigeonhole version 0.4.13 (7b14904)
$ dpkg -s dovecot-core
  Package: dovecot-core
  Version: 1:2.2.24-1ubuntu1.1

This bug was fixed in the package dovecot - 1:2.2.22-1ubuntu2.2

---------------
dovecot (1:2.2.22-1ubuntu2.2) xenial; urgency=medium

  * d/p/fix-sieve-pigeonhole-crash-on-huge-mails.patch: Fix sieve-pigeonhole
    crash when filtering too much data (LP: #1633220)

 -- Christian Ehrhardt <email address hidden> Wed, 09 Nov 2016 13:13:08 +0100

The verification of the Stable Release Update for dovecot has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package dovecot - 1:2.2.24-1ubuntu1.1

---------------
dovecot (1:2.2.24-1ubuntu1.1) yakkety; urgency=medium

  * d/p/fix-sieve-pigeonhole-crash-on-huge-mails.patch: Fix sieve-pigeonhole
    crash when filtering too much data (LP: #1633220)

 -- Christian Ehrhardt <email address hidden> Wed, 09 Nov 2016 13:13:28 +0100