Merge lp:~therp-nl/server-env-tools/6.1-fetchmail_attach_from_folder into lp:~server-env-tools-core-editors/server-env-tools/6.1

[Holger Brunn (Therp)]

Found a concurrency problem, causing long running fetching threads to be called multiple times, resulting in duplicated mails

[Stefan Rijnhart (Opener) (stefan-opener)]

Thanks for taking my comments from https://code.launchpad.net/~therp-nl/server-env-tools/fetchmail_attach_from_folder/+merge/155448 into account

review: Approve

[Alexandre Fayolle - camptocamp (alexandre-fayolle-c2c)]

Suggested enhancements:

formatting : would have been nice to get the proper spaces added (after commas, before and after assignment equal signs...)

the fetch_mail method (line 416 of the MP) could be split. I would personnally add one method for the body of the "for folder" loop and another for the body of the for message loop. This gains 8 columns, with allows for less line wrapping and clearer code, within the 80 col convention.

line 497: prefer "key in dictionary" to dictionary.has_key(key).

line 632: you are passing a cmp func to sorted to compare by the first element of a tuple. There are several small issues in there:

1. it is more efficient to use the key named argument (passing itemgetter(0) as key function in this case)
2. but it looks to me that the default sorting of Python does exactly what you want
3. you have a local list : using tuple(sorted(algorithms)) eats up unnecessary memory. Just use algorithms.sort() then return algorithms

in the column definitions (line 634 of the MP and following), using double quotes as delimiter removes the need to escape the single quotes inside the various strings, which makes the code easier to read.

review: Needs Fixing (code review, no test)

[Ronald Portier (Therp) (rportier1962)]

Hello Holger,

I made a slightly modified branch:
https://code.launchpad.net/~therp-nl/server-env-tools/6.1-fetchmail_attach_mail_from_folder_rpo

- solved a problem when trying to match on either, from, or to, or cc, or bcc: code was chocking on fields not present in mail.
- enhanced change of matching by converting address to lowercase. To complement this, really should be a good idea to have only lower case e-mail adresses in for instance res.partner.address as well. Or have a view that always converts to lowercase...
- some minor code reorganisations to make it easer to step debug and see what is going on.

Ronald

[Holger Brunn (Therp) (hbrunn)]

Thanks Ronald, would you issue a merge request on this branch as a reminder? I see that assumingly your IDE already took care of a few formatting issues that were pointed out by Alexandre, but there's more to do for me

[Stefan Rijnhart (Opener) (stefan-opener)]

Thanks everyone, great to see such collaboration!

review: Approve

[Ronald Portier (Therp) (rportier1962)]

Holger,

I just noticed that any mail retrieved gets a status 'received'.

However, it is possible to retrieve both received and send-mail.

So if the customer-email address is found in a sender field (most likely from), the mail is really received.

If it found in a receiver field (to, cc or bcc), the mail should get a status sent.

Kind regard, Ronald

[Holger Brunn (Therp) (hbrunn)]

Ronald: That's a very good point, thanks. But I think it depends too much on the configured object/field what the state of a message should be to put it into the code. That's why I made it a configuration option for a folder in rev30, with 'received' as default.

Alexandre: Today I did the refactoring you proposed, now I think all your points are covered. Could you have another look?

[Alexandre Fayolle - camptocamp (alexandre-fayolle-c2c)]

LGTM.

Thanks for taking the time to implement my suggestions.

review: Approve (code review, no test)

[Ronald Portier (Therp) (rportier1962)]

Holger,

Great improvements! Thanx.

Sorry to bother you, but I have one more concern: when we are searching for matches, I think there will be a danger that we match on our own e-mail, instead of the partner e-mail.

Example, suppose I have a folder for all my customer mail. Matching will be done on email in res.partner.address.

As I have ingoing and outgoing mail in the same folder, I create two configurations/folders. One having to, cc and bcc as Field (email), and linking it to status send. The other to from and linking it to status received.

Now when I send a mail to a customer, it will find my own e-mail in from, an ingoing mail will be registered, with myself as the customer.

So I think there should be a way to exclude our own e-mail addresses from the search. Maybe by having all mail addresses with our own domain (might be multiple!) excluded from the search results.

What do you think?

Regards, Ronald

[Holger Brunn (Therp) (hbrunn)]

I think you can exclude such cases by setting a domain in your configuration. So for matching customers only, you could use [('is_customer','=',True)], causing only customers to be possible candidates for matching in the first place. Or filter out employees: [('is_employee','=',False)]. The domain expression is passed directly to the model's search function, see currently lines 261ff in the MP.

In the end you get an expression like
((model_field=='address1') || (model_field=='address2') || ...) && your_domain_expression

Does this help?

[Holger Brunn (Therp) (hbrunn)]

now I realize btw that today's refactoring would be a lot more useful if the functions passed down the ids of the mail.message(s) created. I'll fix that tomorrow

[Ronald Portier (Therp) (rportier1962)]

Approve.

I tested the following use-case: single imap folder contains mail sent to and received from customers.

Did the following configuration;
- Created an incoming server document, with all the usual options
- Linked two folder configurations to this document:

1. - Folder 'Klanten' (customers)
   - match algorithm: Exact mailaddress
   - Model 'relatieadressen' (res.partner.address)
   - Field (emal): from
   - Field (model): email
   - Message state: Received
   - Domain: [('partner_id.customer', '=', True)]
   - Delete matches: False
   - Flag nonmatching: True
   - Use first match: True
2. The same as 1., except for the following:
   - Field (email): to, cc, bcc
   - Message state: Sent

I then used my mail client to send a mail wit an attachment (here our offer....) to a test customer-email. Then I used the test customer email account to sent back a reply, (I agree to the offer, find enclosed...), with another attachment.

Then I placed both the e-mail sent, and the reply received in the folder Klanten (customers) and ran the receive mail action in the incoming mail server document.

Outcome was as expected:
- I could find both mails, with the right status, on the tab 'Email' within the partner record
- I could see and open both the outgoing and the incoming attachment from within OpenERP.

Kind regards,

Ronald

review: Approve

[Ronald Portier (Therp) (rportier1962)]

Saw the problem. Solution looks good to me.

review: Approve

[lp:~therp-nl/server-env-tools/6.1-fetchmail_attach_from_folder]

34. By Ronald Portier (Therp) on 2013-04-26

[MRG] [ADD] mail_client_view: gives normal users access to the mail views.

The only other possibilities users have of looking at e-mails, is through the
partner object, or other objects mails are directly linked to. This makes it
impossible to get an overall view, or to search through mails.

Furthermore the module makes newly received mails stand out with a red circle
in the mail view. These mails are in need of action. Via a simple click a user
can either mark a message as also in need of action, or the other way around,
as having been handled.

[Alexandre Fayolle - camptocamp (alexandre-fayolle-c2c)]

review: Approve (code review, no test)

[lp:~therp-nl/server-env-tools/6.1-fetchmail_attach_from_folder]

35. By Holger Brunn (Therp) on 2013-04-26

[MRG][ADD] fetchmail_attach_from_folder

Adds the possibility to attach emails from a certain IMAP folder to objects,
ie partners. Matching is done via several algorithms, ie email address.

This gives a simple possibility to archive emails in OpenERP without a mail
client integration.

=== added directory 'fetchmail_attach_from_folder'
=== added file 'fetchmail_attach_from_folder/__init__.py'
--- fetchmail_attach_from_folder/__init__.py	1970-01-01 00:00:00 +0000
+++ fetchmail_attach_from_folder/__init__.py	2013-04-18 14:54:31 +0000
@@ -0,0 +1,25 @@
+# -*- encoding: utf-8 -*-
+##############################################################################
+#
+#    OpenERP, Open Source Management Solution
+#    This module copyright (C) 2013 Therp BV (<http://therp.nl>)
+#    All Rights Reserved
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU Affero General Public License as
+#    published by the Free Software Foundation, either version 3 of the
+#    License, or (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU Affero General Public License for more details.
+#
+#    You should have received a copy of the GNU Affero General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+##############################################################################
+
+import match_algorithm
+import model
+import wizard
 
=== added file 'fetchmail_attach_from_folder/__openerp__.py'
--- fetchmail_attach_from_folder/__openerp__.py	1970-01-01 00:00:00 +0000
+++ fetchmail_attach_from_folder/__openerp__.py	2013-04-18 14:54:31 +0000
@@ -0,0 +1,45 @@
+# -*- encoding: utf-8 -*-
+##############################################################################
+#
+#    OpenERP, Open Source Management Solution
+#    This module copyright (C) 2013 Therp BV (<http://therp.nl>)
+#    All Rights Reserved
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU Affero General Public License as
+#    published by the Free Software Foundation, either version 3 of the
+#    License, or (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU Affero General Public License for more details.
+#
+#    You should have received a copy of the GNU Affero General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+##############################################################################
+
+{
+    'name': 'Attach mails in an IMAP folder to existing objects',
+    'version': '1.0',
+    'description': """
+    Adds the possibility to attach emails from a certain IMAP folder to objects,
+    ie partners. Matching is done via several algorithms, ie email address.
+
+    This gives a simple possibility to archive emails in OpenERP without a mail
+    client integration.
+    """,
+    'author': 'Therp BV',
+    'website': 'http://www.therp.nl',
+    "category": "Tools",
+    "depends": ['fetchmail'],
+    'data': [
+        'view/fetchmail_server.xml',
+        'wizard/attach_mail_manually.xml',
+        ],
+    'js': [],
+    'installable': True,
+    'active': False,
+    'certificate': '',
+}
 
=== added directory 'fetchmail_attach_from_folder/match_algorithm'
=== added file 'fetchmail_attach_from_folder/match_algorithm/__init__.py'
--- fetchmail_attach_from_folder/match_algorithm/__init__.py	1970-01-01 00:00:00 +0000
+++ fetchmail_attach_from_folder/match_algorithm/__init__.py	2013-04-18 14:54:31 +0000
@@ -0,0 +1,26 @@
+# -*- encoding: utf-8 -*-
+##############################################################################
+#
+#    OpenERP, Open Source Management Solution
+#    This module copyright (C) 2013 Therp BV (<http://therp.nl>)
+#    All Rights Reserved
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU Affero General Public License as
+#    published by the Free Software Foundation, either version 3 of the
+#    License, or (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU Affero General Public License for more details.
+#
+#    You should have received a copy of the GNU Affero General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+##############################################################################
+
+import base
+import email_exact
+import email_domain
+import openerp_standard
 
=== added file 'fetchmail_attach_from_folder/match_algorithm/base.py'
--- fetchmail_attach_from_folder/match_algorithm/base.py	1970-01-01 00:00:00 +0000
+++ fetchmail_attach_from_folder/match_algorithm/base.py	2013-04-18 14:54:31 +0000
@@ -0,0 +1,43 @@
+# -*- encoding: utf-8 -*-
+##############################################################################
+#
+#    OpenERP, Open Source Management Solution
+#    This module copyright (C) 2013 Therp BV (<http://therp.nl>)
+#    All Rights Reserved
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU Affero General Public License as
+#    published by the Free Software Foundation, either version 3 of the
+#    License, or (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU Affero General Public License for more details.
+#
+#    You should have received a copy of the GNU Affero General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+##############################################################################
+
+class base(object):
+    name = None
+    '''Name shown to the user'''
+
+    required_fields = []
+    '''Fields on fetchmail_server folder that are required for this algorithm'''
+
+    readonly_fields = []
+    '''Fields on fetchmail_server folder that are readonly for this algorithm'''
+
+
+    def search_matches(self, cr, uid, conf, mail_message, mail_message_org):
+        '''Returns ids found for model with mail_message'''
+        return []
+
+    def handle_match(
+            self, cr, uid, connection, object_id, folder,
+            mail_message, mail_message_org, msgid, context=None):
+        '''Do whatever it takes to handle a match'''
+        return folder.server_id.attach_mail(connection, object_id, folder,
+                mail_message, msgid)
 
=== added file 'fetchmail_attach_from_folder/match_algorithm/email_domain.py'
--- fetchmail_attach_from_folder/match_algorithm/email_domain.py	1970-01-01 00:00:00 +0000
+++ fetchmail_attach_from_folder/match_algorithm/email_domain.py	2013-04-18 14:54:31 +0000
@@ -0,0 +1,44 @@
+# -*- encoding: utf-8 -*-
+##############################################################################
+#
+#    OpenERP, Open Source Management Solution
+#    This module copyright (C) 2013 Therp BV (<http://therp.nl>)
+#    All Rights Reserved
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU Affero General Public License as
+#    published by the Free Software Foundation, either version 3 of the
+#    License, or (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU Affero General Public License for more details.
+#
+#    You should have received a copy of the GNU Affero General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+##############################################################################
+
+from email_exact import email_exact
+
+class email_domain(email_exact):
+    '''Search objects by domain name of email address.
+    Beware of match_first here, this is most likely to get it wrong (gmail)'''
+    name = 'Domain of email address'
+
+    def search_matches(self, cr, uid, conf, mail_message, mail_message_org):
+        ids = super(email_domain, self).search_matches(
+                cr, uid, conf, mail_message, mail_message_org)
+        if not ids:
+            domains = []
+            for addr in self._get_mailaddresses(conf, mail_message):
+                domains.append(addr.split('@')[-1])
+            ids = conf.pool.get(conf.model_id.model).search(
+                    cr, uid,
+                    self._get_mailaddress_search_domain(
+                        conf, mail_message,
+                        operator='like',
+                        values=['%@'+domain for domain in set(domains)]),
+                    order=conf.model_order)
+        return ids
 
=== added file 'fetchmail_attach_from_folder/match_algorithm/email_exact.py'
--- fetchmail_attach_from_folder/match_algorithm/email_exact.py	1970-01-01 00:00:00 +0000
+++ fetchmail_attach_from_folder/match_algorithm/email_exact.py	2013-04-18 14:54:31 +0000
@@ -0,0 +1,56 @@
+# -*- encoding: utf-8 -*-
+##############################################################################
+#
+#    OpenERP, Open Source Management Solution
+#    This module copyright (C) 2013 Therp BV (<http://therp.nl>)
+#    All Rights Reserved
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU Affero General Public License as
+#    published by the Free Software Foundation, either version 3 of the
+#    License, or (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU Affero General Public License for more details.
+#
+#    You should have received a copy of the GNU Affero General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+##############################################################################
+
+from base import base
+from openerp.tools.safe_eval import safe_eval
+from openerp.addons.mail.mail_message import to_email
+
+class email_exact(base):
+    '''Search for exactly the mailadress as noted in the email'''
+
+    name = 'Exact mailadress'
+    required_fields = ['model_field', 'mail_field']
+
+    def _get_mailaddresses(self, conf, mail_message):
+        mailaddresses = []
+        fields = conf.mail_field.split(',')
+        for field in fields:
+            if field in mail_message:
+                mailaddresses += to_email(mail_message[field])
+        return [ addr.lower() for addr in mailaddresses ]
+
+    def _get_mailaddress_search_domain(
+            self, conf, mail_message, operator='=', values=None):
+        mailaddresses = values or self._get_mailaddresses(
+                conf, mail_message)
+        if not mailaddresses:
+            return [(0, '=', 1)]
+        search_domain = ((['|'] * (len(mailaddresses) - 1)) + [
+                (conf.model_field, operator, addr) for addr in mailaddresses] +
+                safe_eval(conf.domain or '[]'))
+        return search_domain
+
+    def search_matches(self, cr, uid, conf, mail_message, mail_message_org):
+        conf_model = conf.pool.get(conf.model_id.model)
+        search_domain = self._get_mailaddress_search_domain(conf, mail_message)
+        return conf_model.search(
+            cr, uid, search_domain, order=conf.model_order)
 
=== added file 'fetchmail_attach_from_folder/match_algorithm/openerp_standard.py'
--- fetchmail_attach_from_folder/match_algorithm/openerp_standard.py	1970-01-01 00:00:00 +0000
+++ fetchmail_attach_from_folder/match_algorithm/openerp_standard.py	2013-04-18 14:54:31 +0000
@@ -0,0 +1,51 @@
+# -*- encoding: utf-8 -*-
+##############################################################################
+#
+#    OpenERP, Open Source Management Solution
+#    This module copyright (C) 2013 Therp BV (<http://therp.nl>)
+#    All Rights Reserved
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU Affero General Public License as
+#    published by the Free Software Foundation, either version 3 of the
+#    License, or (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU Affero General Public License for more details.
+#
+#    You should have received a copy of the GNU Affero General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+##############################################################################
+
+from base import base
+from openerp.tools.safe_eval import safe_eval
+
+class openerp_standard(base):
+    '''No search at all. Use OpenERP's standard mechanism to attach mails to
+    mail.thread objects. Note that this algorithm always matches.'''
+
+    name = 'OpenERP standard'
+    readonly_fields = ['model_field', 'mail_field', 'match_first', 'domain',
+            'model_order', 'flag_nonmatching']
+
+    def search_matches(self, cr, uid, conf, mail_message, mail_message_org):
+        '''Always match. Duplicates will be fished out by message_id'''
+        return [True]
+
+    def handle_match(
+            self, cr, uid, connection, object_id, folder,
+            mail_message, mail_message_org, msgid, context):
+        result = folder.pool.get('mail.thread').message_process(
+                cr, uid, 
+                folder.model_id.model, mail_message_org,
+                save_original=folder.server_id.original,
+                strip_attachments=(not folder.server_id.attach),
+                context=context)
+
+        if folder.delete_matching:
+            connection.store(msgid, '+FLAGS', '\\DELETED')
+
+        return [result]
 
=== added directory 'fetchmail_attach_from_folder/model'
=== added file 'fetchmail_attach_from_folder/model/__init__.py'
--- fetchmail_attach_from_folder/model/__init__.py	1970-01-01 00:00:00 +0000
+++ fetchmail_attach_from_folder/model/__init__.py	2013-04-18 14:54:31 +0000
@@ -0,0 +1,24 @@
+# -*- encoding: utf-8 -*-
+##############################################################################
+#
+#    OpenERP, Open Source Management Solution
+#    This module copyright (C) 2013 Therp BV (<http://therp.nl>)
+#    All Rights Reserved
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU Affero General Public License as
+#    published by the Free Software Foundation, either version 3 of the
+#    License, or (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU Affero General Public License for more details.
+#
+#    You should have received a copy of the GNU Affero General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+##############################################################################
+
+import fetchmail_server
+import fetchmail_server_folder
 
=== added file 'fetchmail_attach_from_folder/model/fetchmail_server.py'
--- fetchmail_attach_from_folder/model/fetchmail_server.py	1970-01-01 00:00:00 +0000
+++ fetchmail_attach_from_folder/model/fetchmail_server.py	2013-04-18 14:54:31 +0000
@@ -0,0 +1,287 @@
+# -*- encoding: utf-8 -*-
+##############################################################################
+#
+#    OpenERP, Open Source Management Solution
+#    This module copyright (C) 2013 Therp BV (<http://therp.nl>)
+#    All Rights Reserved
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU Affero General Public License as
+#    published by the Free Software Foundation, either version 3 of the
+#    License, or (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU Affero General Public License for more details.
+#
+#    You should have received a copy of the GNU Affero General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+##############################################################################
+
+import base64
+import simplejson
+from lxml import etree
+from openerp.osv.orm import Model, except_orm, browse_null
+from openerp.tools.translate import _
+from openerp.osv import fields
+from openerp.addons.fetchmail.fetchmail import logger
+from openerp.tools.misc import UnquoteEvalContext
+from openerp.tools.safe_eval import safe_eval
+
+
+class fetchmail_server(Model):
+    _inherit = 'fetchmail.server'
+
+    _columns = {
+        'folder_ids': fields.one2many(
+        'fetchmail.server.folder', 'server_id', 'Folders'),
+    }
+
+    _defaults = {
+        'type': 'imap',
+    }
+
+    def __init__(self, pool, cr):
+        self._columns['object_id'].required = False
+        return super(fetchmail_server, self).__init__(pool, cr)
+
+    def onchange_server_type(
+            self, cr, uid, ids, server_type=False, ssl=False,
+            object_id=False):
+        retval = super(
+            fetchmail_server, self).onchange_server_type(cr, uid,
+                                                         ids, server_type, ssl,
+                                                         object_id)
+        retval['value']['state'] = 'draft'
+        return retval
+
+    def fetch_mail(self, cr, uid, ids, context=None):
+        if context is None:
+            context = {}
+
+        check_original = []
+
+        for this in self.browse(cr, uid, ids, context):
+            if this.object_id:
+                check_original.append(this.id)
+
+            context.update(
+                {
+                    'fetchmail_server_id': this.id,
+                    'server_type': this.type
+                })
+
+            connection = this.connect()
+            for folder in this.folder_ids:
+                this.handle_folder(connection, folder)
+
+            connection.close()
+
+        return super(fetchmail_server, self).fetch_mail(
+            cr, uid, check_original, context)
+
+    def handle_folder(self, cr, uid, ids, connection, folder, context=None):
+        '''Return ids of objects matched'''
+
+        matched_object_ids = []
+
+        for this in self.browse(cr, uid, ids, context=context):
+            logger.info('start checking for emails in %s server %s',
+                        folder.path, this.name)
+
+            match_algorithm = folder.get_algorithm()
+
+            if connection.select(folder.path)[0] != 'OK':
+                logger.error(
+                    'Could not open mailbox %s on %s' % (
+                    folder.path, this.server))
+                connection.select()
+                continue
+            result, msgids = this.get_msgids(connection)
+            if result != 'OK':
+                logger.error(
+                    'Could not search mailbox %s on %s' % (
+                    folder.path, this.server))
+                continue
+
+            for msgid in msgids[0].split():
+                matched_object_ids += this.apply_matching(
+                        connection, folder, msgid, match_algorithm)
+
+            logger.info('finished checking for emails in %s server %s',
+                        folder.path, this.name)
+
+        return matched_object_ids
+
+    def get_msgids(self, cr, uid, ids, connection, context=None):
+        '''Return imap ids of messages to process'''
+        return connection.search(None, 'UNDELETED')
+
+    def apply_matching(self, cr, uid, ids, connection, folder, msgid,
+                       match_algorithm, context=None):
+        '''Return ids of objects matched'''
+
+        matched_object_ids = []
+
+        for this in self.browse(cr, uid, ids, context=context):
+            result, msgdata = connection.fetch(msgid, '(RFC822)')
+
+            if result != 'OK':
+                logger.error(
+                    'Could not fetch %s in %s on %s' % (
+                    msgid, folder.path, this.server))
+                continue
+
+            mail_message = self.pool.get('mail.message').parse_message(
+                msgdata[0][1], this.original)
+
+            if self.pool.get('mail.message').search(cr, uid, [
+                ('message_id', '=', mail_message['message-id'])]):
+                continue
+
+            found_ids = match_algorithm.search_matches(
+                cr, uid, folder,
+                mail_message, msgdata[0][1])
+
+            if found_ids and (len(found_ids) == 1 or
+                              folder.match_first):
+                try:
+                    cr.execute('savepoint apply_matching')
+                    match_algorithm.handle_match(
+                        cr, uid, connection,
+                        found_ids[0], folder, mail_message,
+                        msgdata[0][1], msgid, context)
+                    cr.execute('release savepoint apply_matching')
+                    matched_object_ids += found_ids[:1]
+                except Exception, e:
+                    cr.execute('rollback to savepoint apply_matching')
+                    logger.exception(
+                        "Failed to fetch mail %s from %s",
+                        msgid, this.name)
+            elif folder.flag_nonmatching:
+                connection.store(msgid, '+FLAGS', '\\FLAGGED')
+
+        return matched_object_ids
+
+    def attach_mail(
+            self, cr, uid, ids, connection, object_id, folder,
+            mail_message, msgid, context=None):
+        '''Return ids of messages created'''
+
+        mail_message_ids = []
+
+        for this in self.browse(cr, uid, ids, context):
+            partner_id = None
+            if folder.model_id.model == 'res.partner':
+                partner_id = object_id
+            if 'partner_id' in self.pool.get(folder.model_id.model)._columns:
+                partner_id = self.pool.get(
+                    folder.model_id.model).browse(
+                        cr, uid, object_id, context
+                    ).partner_id.id
+
+            attachments=[]
+            if this.attach and mail_message.get('attachments'):
+                for attachment in mail_message['attachments']:
+                    fname, fcontent = attachment
+                    if isinstance(fcontent, unicode):
+                        fcontent = fcontent.encode('utf-8')
+                    data_attach = {
+                            'name': fname,
+                            'datas': base64.b64encode(str(fcontent)),
+                            'datas_fname': fname,
+                            'description': _('Mail attachment'),
+                            'res_model': folder.model_id.model,
+                            'res_id': object_id,
+                            }
+                    attachments.append(
+                        self.pool.get('ir.attachment').create(
+                            cr, uid, data_attach, context=context))
+
+            mail_message_ids.append(
+                    self.pool.get('mail.message').create(
+                        cr, uid,
+                        {
+                            'partner_id': partner_id,
+                            'model': folder.model_id.model,
+                            'res_id': object_id,
+                            'body_text': mail_message.get('body'),
+                            'body_html': mail_message.get('body_html'),
+                            'subject': mail_message.get('subject'),
+                            'email_to': mail_message.get('to'),
+                            'email_from': mail_message.get('from'),
+                            'email_cc': mail_message.get('cc'),
+                            'reply_to': mail_message.get('reply'),
+                            'date': mail_message.get('date'),
+                            'message_id': mail_message.get('message-id'),
+                            'subtype': mail_message.get('subtype'),
+                            'headers': mail_message.get('headers'),
+                            'state': folder.msg_state,
+                            'attachment_ids': [(6, 0, attachments)],
+                        },
+                        context))
+
+            if folder.delete_matching:
+                connection.store(msgid, '+FLAGS', '\\DELETED')
+        return mail_message_ids
+
+    def button_confirm_login(self, cr, uid, ids, context=None):
+        retval = super(fetchmail_server, self).button_confirm_login(cr, uid,
+                                                                    ids,
+                                                                    context)
+
+        for this in self.browse(cr, uid, ids, context):
+            this.write({'state': 'draft'})
+            connection = this.connect()
+            connection.select()
+            for folder in this.folder_ids:
+                if connection.select(folder.path)[0] != 'OK':
+                    raise except_orm(
+                        _('Error'), _('Mailbox %s not found!') %
+                        folder.path)
+                folder.get_algorithm().search_matches(
+                    cr, uid, folder, browse_null(), '')
+            connection.close()
+            this.write({'state': 'done'})
+
+        return retval
+
+    def fields_view_get(self, cr, user, view_id=None, view_type='form',
+                        context=None, toolbar=False, submenu=False):
+        result = super(fetchmail_server, self).fields_view_get(
+            cr, user, view_id, view_type, context, toolbar, submenu)
+
+        if view_type == 'form':
+            view = etree.fromstring(
+                result['fields']['folder_ids']['views']['form']['arch'])
+            modifiers = {}
+            docstr = ''
+            for algorithm in self.pool.get('fetchmail.server.folder')\
+                    ._get_match_algorithms().itervalues():
+                for modifier in ['required', 'readonly']:
+                    for field in getattr(algorithm, modifier + '_fields'):
+                        modifiers.setdefault(field, {})
+                        modifiers[field].setdefault(modifier, [])
+                        if modifiers[field][modifier]:
+                            modifiers[field][modifier].insert(0, '|')
+                        modifiers[field][modifier].append(
+                            ("match_algorithm", "==", algorithm.__name__))
+                docstr += _(algorithm.name) + '\n' + _(algorithm.__doc__) + \
+                    '\n\n'
+
+            for field in view:
+                if field.tag == 'field' and field.get('name') in modifiers:
+                    field.set('modifiers', simplejson.dumps(
+                        dict(
+                            eval(field.attrib['modifiers'],
+                                 UnquoteEvalContext({})),
+                            **modifiers[field.attrib['name']])))
+                if (field.tag == 'field' and
+                        field.get('name') == 'match_algorithm'):
+                    field.set('help', docstr)
+            result['fields']['folder_ids']['views']['form']['arch'] = \
+                etree.tostring(view)
+
+        return result
 
=== added file 'fetchmail_attach_from_folder/model/fetchmail_server_folder.py'
--- fetchmail_attach_from_folder/model/fetchmail_server_folder.py	1970-01-01 00:00:00 +0000
+++ fetchmail_attach_from_folder/model/fetchmail_server_folder.py	2013-04-18 14:54:31 +0000
@@ -0,0 +1,120 @@
+# -*- encoding: utf-8 -*-
+##############################################################################
+#
+#    OpenERP, Open Source Management Solution
+#    This module copyright (C) 2013 Therp BV (<http://therp.nl>)
+#    All Rights Reserved
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU Affero General Public License as
+#    published by the Free Software Foundation, either version 3 of the
+#    License, or (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU Affero General Public License for more details.
+#
+#    You should have received a copy of the GNU Affero General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+########################################################################
+
+from openerp.osv import fields
+from openerp.osv.orm import Model
+from .. import match_algorithm
+
+
+class fetchmail_server_folder(Model):
+    _name = 'fetchmail.server.folder'
+    _rec_name = 'path'
+
+    def _get_match_algorithms(self):
+        def get_all_subclasses(cls):
+            return cls.__subclasses__() + [subsub
+                                           for sub in cls.__subclasses__()
+                                           for subsub in get_all_subclasses(sub)]
+        return dict([(cls.__name__, cls) for cls in get_all_subclasses(
+            match_algorithm.base.base)])
+
+    def _get_match_algorithms_sel(self, cr, uid, context=None):
+        algorithms = []
+        for cls in self._get_match_algorithms().itervalues():
+            algorithms.append((cls.__name__, cls.name))
+        algorithms.sort()
+        return algorithms
+
+    _columns = {
+        'sequence': fields.integer('Sequence'),
+        'path': fields.char(
+        'Path', size=256, help='The path to your mail '
+        "folder. Typically would be something like 'INBOX.myfolder'",
+        required=True),
+        'model_id': fields.many2one(
+        'ir.model', 'Model', required=True,
+        help='The model to attach emails to'),
+        'model_field': fields.char(
+        'Field (model)', size=128,
+        help='The field in your model that contains the field to match '
+        'against.\n'
+        'Examples:\n'
+        "'email' if your model is res.partner, or "
+        "'partner_id.email' if you're matching sale orders"),
+        'model_order': fields.char(
+        'Order (model)', size=128,
+        help='Fields to order by, this mostly useful in conjunction '
+        "with 'Use 1st match'"),
+        'match_algorithm': fields.selection(
+        _get_match_algorithms_sel,
+        'Match algorithm', required=True, translate=True,
+        help='The algorithm used to determine which object an email '
+        'matches.'),
+        'mail_field': fields.char(
+        'Field (email)', size=128,
+        help='The field in the email used for matching. Typically '
+        "this is 'to' or 'from'"),
+        'server_id': fields.many2one('fetchmail.server', 'Server'),
+        'delete_matching': fields.boolean(
+        'Delete matches',
+        help='Delete matched emails from server'),
+        'flag_nonmatching': fields.boolean(
+        'Flag nonmatching',
+        help="Flag emails in the server that don't match any object "
+        'in OpenERP'),
+        'match_first': fields.boolean(
+        'Use 1st match',
+        help='If there are multiple matches, use the first one. If '
+        'not checked, multiple matches count as no match at all'),
+        'domain': fields.char(
+        'Domain', size=128, help='Fill in a search '
+        'filter to narrow down objects to match'),
+        'msg_state': fields.selection(
+        [
+            ('sent', 'Sent'),
+            ('received', 'Received'),
+        ],
+        'Message state',
+        help='The state messages fetched from this folder should be '
+        'assigned in OpenERP'),
+    }
+
+    _defaults = {
+        'flag_nonmatching': True,
+        'msg_state': 'received',
+    }
+
+    def get_algorithm(self, cr, uid, ids, context=None):
+        for this in self.browse(cr, uid, ids, context):
+            return self._get_match_algorithms()[this.match_algorithm]()
+
+    def button_attach_mail_manually(self, cr, uid, ids, context=None):
+        for this in self.browse(cr, uid, ids, context):
+            context.update({'default_folder_id': this.id})
+        return {
+            'type': 'ir.actions.act_window',
+            'res_model': 'fetchmail.attach.mail.manually',
+            'target': 'new',
+            'context': context,
+            'view_type': 'form',
+            'view_mode': 'form',
+        }
 
=== added directory 'fetchmail_attach_from_folder/view'
=== added file 'fetchmail_attach_from_folder/view/fetchmail_server.xml'
--- fetchmail_attach_from_folder/view/fetchmail_server.xml	1970-01-01 00:00:00 +0000
+++ fetchmail_attach_from_folder/view/fetchmail_server.xml	2013-04-18 14:54:31 +0000
@@ -0,0 +1,58 @@
+<?xml version="1.0" encoding="utf-8"?>
+<openerp>
+    <data>
+        <record model="ir.ui.view" id="view_email_server_form">
+            <field name="name">fetchmail.server.form</field>
+            <field name="model">fetchmail.server</field>
+            <field name="type">form</field>
+            <field name="inherit_id" ref="fetchmail.view_email_server_form" />
+            <field name="arch" type="xml">
+                <data>
+                    <field name="object_id" position="attributes">
+                        <attribute name="attrs">{'required': [('type', '!=', 'imap')]}</attribute>
+                    </field>
+                    <xpath expr="//page[@string='Server &amp; Login']/group[3]" position="after">
+                        <group col="2" colspan="2" attrs="{'invisible': [('type','!=','imap')]}">
+                            <separator string="Folders to monitor" colspan="2"/>
+                            <field
+                                name="folder_ids"
+                                nolabel="1"
+                                colspan="2"
+                                on_change="onchange_server_type(type, is_ssl, object_id)">
+                                <tree>
+                                    <field name="sequence" invisible="1" />
+                                    <field name="path" />
+                                    <field name="model_id" />
+                                    <field name="model_field" />
+                                    <field name="match_algorithm" />
+                                    <field name="mail_field" />
+                                </tree>
+                                <form col="6">
+                                    <field name="path" />
+                                    <field name="model_id" />
+                                    <field name="model_field" />
+                                    <field name="match_algorithm" />
+                                    <field name="mail_field" />
+                                    <newline />
+                                    <field name="delete_matching" />
+                                    <field name="flag_nonmatching" />
+                                    <field name="match_first" />
+                                    <field name="msg_state" />
+                                    <field name="model_order" attrs="{'readonly': [('match_first','==',False)], 'required': [('match_first','==',True)]}" />
+                                    <field name="domain" />
+                                    <newline />
+                                    <label />
+                                    <label />
+                                    <label />
+                                    <label />
+                                    <label />
+                                    <button type="object" name="button_attach_mail_manually" string="Attach mail manually" icon="gtk-redo" />
+                                </form>
+                            </field>
+                        </group>
+                    </xpath>
+                </data>
+            </field>
+        </record>
+    </data>
+</openerp>
 
=== added directory 'fetchmail_attach_from_folder/wizard'
=== added file 'fetchmail_attach_from_folder/wizard/__init__.py'
--- fetchmail_attach_from_folder/wizard/__init__.py	1970-01-01 00:00:00 +0000
+++ fetchmail_attach_from_folder/wizard/__init__.py	2013-04-18 14:54:31 +0000
@@ -0,0 +1,23 @@
+# -*- encoding: utf-8 -*-
+##############################################################################
+#
+#    OpenERP, Open Source Management Solution
+#    This module copyright (C) 2013 Therp BV (<http://therp.nl>)
+#    All Rights Reserved
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU Affero General Public License as
+#    published by the Free Software Foundation, either version 3 of the
+#    License, or (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU Affero General Public License for more details.
+#
+#    You should have received a copy of the GNU Affero General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+##############################################################################
+
+import attach_mail_manually
 
=== added file 'fetchmail_attach_from_folder/wizard/attach_mail_manually.py'
--- fetchmail_attach_from_folder/wizard/attach_mail_manually.py	1970-01-01 00:00:00 +0000
+++ fetchmail_attach_from_folder/wizard/attach_mail_manually.py	2013-04-18 14:54:31 +0000
@@ -0,0 +1,110 @@
+# -*- encoding: utf-8 -*-
+##############################################################################
+#
+#    OpenERP, Open Source Management Solution
+#    This module copyright (C) 2013 Therp BV (<http://therp.nl>)
+#    All Rights Reserved
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU Affero General Public License as
+#    published by the Free Software Foundation, either version 3 of the
+#    License, or (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU Affero General Public License for more details.
+#
+#    You should have received a copy of the GNU Affero General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+##############################################################################
+
+from openerp.osv import fields
+from openerp.osv.orm import TransientModel
+
+
+class attach_mail_manually(TransientModel):
+    _name = 'fetchmail.attach.mail.manually'
+
+    _columns = {
+            'folder_id': fields.many2one('fetchmail.server.folder', 'Folder',
+                readonly=True),
+            'mail_ids': fields.one2many(
+                'fetchmail.attach.mail.manually.mail', 'wizard_id', 'Emails'),
+            }
+
+    def default_get(self, cr, uid, fields_list, context=None):
+        if context is None:
+            context = {}
+
+        defaults = super(attach_mail_manually, self).default_get(cr, uid, 
+                                fields_list, context)
+
+        for folder in self.pool.get('fetchmail.server.folder').browse(cr, uid,
+                [context.get('default_folder_id')], context):
+            defaults['mail_ids']=[]
+            connection = folder.server_id.connect()
+            connection.select(folder.path)
+            result, msgids = connection.search(None, 
+                    'FLAGGED' if folder.flag_nonmatching else 'UNDELETED')
+            if result != 'OK':
+                logger.error('Could not search mailbox %s on %s' % (
+                    folder.path, this.server))
+                continue
+            attach_mail_manually_mail._columns['object_id'].selection=[
+                    (folder.model_id.model, folder.model_id.name)]
+            for msgid in msgids[0].split():
+                result, msgdata = connection.fetch(msgid, '(RFC822)')
+                if result != 'OK':
+                    logger.error('Could not fetch %s in %s on %s' % (
+                        msgid, folder.path, this.server))
+                    continue
+                mail_message = self.pool.get('mail.message').parse_message(
+                        msgdata[0][1])
+                defaults['mail_ids'].append((0, 0, {
+                    'msgid': msgid,
+                    'subject': mail_message.get('subject', ''),
+                    'date': mail_message.get('date', ''),
+                    'object_id': folder.model_id.model+',False'
+                    }))
+            connection.close()
+
+        return defaults
+
+    def attach_mails(self, cr, uid, ids, context=None):
+        for this in self.browse(cr, uid, ids, context):
+            for mail in this.mail_ids:
+                connection = this.folder_id.server_id.connect()
+                connection.select(this.folder_id.path)
+                result, msgdata = connection.fetch(mail.msgid, '(RFC822)') 
+                if result != 'OK': 
+                    logger.error('Could not fetch %s in %s on %s' % ( 
+                        msgid, folder.path, this.server)) 
+                    continue 
+                
+                mail_message = self.pool.get('mail.message').parse_message( 
+                        msgdata[0][1], this.folder_id.server_id.original)
+
+                this.folder_id.server_id.attach_mail(connection, 
+                        mail.object_id.id, this.folder_id, mail_message, 
+                        mail.msgid)
+                connection.close()
+        return {'type': 'ir.actions.act_window_close'}
+
+class attach_mail_manually_mail(TransientModel):
+    _name = 'fetchmail.attach.mail.manually.mail'
+
+    _columns = {
+            'wizard_id': fields.many2one('fetchmail.attach.mail.manually', 
+                readonly=True),
+            'msgid': fields.char('Message id', size=16, readonly=True),
+            'subject': fields.char('Subject', size=128, readonly=True),
+            'date': fields.datetime('Date', readonly=True),
+            'object_id': fields.reference('Object', 
+                selection=lambda self, cr, uid, context: 
+                    [(m.model, m.name) for m in 
+                        self.pool.get('ir.model').browse(cr, uid,
+                            self.pool.get('ir.model').search(cr, uid, []),
+                            context)], size=128),
+            }
 
=== added file 'fetchmail_attach_from_folder/wizard/attach_mail_manually.xml'
--- fetchmail_attach_from_folder/wizard/attach_mail_manually.xml	1970-01-01 00:00:00 +0000
+++ fetchmail_attach_from_folder/wizard/attach_mail_manually.xml	2013-04-18 14:54:31 +0000
@@ -0,0 +1,26 @@
+<?xml version="1.0" encoding="utf-8"?>
+<openerp>
+    <data>
+        <record model="ir.ui.view" id="view_attach_mail_manually">
+            <field name="name">fetchmail.attach.mail.manually</field>
+            <field name="model">fetchmail.attach.mail.manually</field>
+            <field name="type">form</field>
+            <field name="arch" type="xml">
+                <form col="4">
+                    <field name="folder_id" colspan="4" />
+                    <field name="mail_ids" nolabel="1" colspan="4">
+                        <tree editable="top">
+                            <field name="subject" />
+                            <field name="date" />
+                            <field name="object_id" />
+                        </tree>
+                    </field>
+                    <label string="" />
+                    <label string="" />
+                    <button special="cancel" string="Cancel" icon="gtk-cancel" />
+                    <button string="Save" type="object" name="attach_mails" icon="gtk-ok" />
+                </form>
+            </field>
+        </record>
+    </data>
+</openerp>