Merge lp:~therp-nl/banking-addons/6.1-account_banking_nl_ing_mt940 into lp:banking-addons/6.1

Proposed by Holger Brunn (Therp) on 2014-02-26
Status: Merged
Merged at revision: 195
Proposed branch: lp:~therp-nl/banking-addons/6.1-account_banking_nl_ing_mt940
Merge into: lp:banking-addons/6.1
Diff against target: 502 lines (+460/-0)
6 files modified
account_banking_mt940/__init__.py (+21/-0)
account_banking_mt940/__openerp__.py (+53/-0)
account_banking_mt940/mt940.py (+217/-0)
account_banking_nl_ing_mt940/__init__.py (+21/-0)
account_banking_nl_ing_mt940/__openerp__.py (+48/-0)
account_banking_nl_ing_mt940/account_banking_nl_ing_mt940.py (+100/-0)
To merge this branch: bzr merge lp:~therp-nl/banking-addons/6.1-account_banking_nl_ing_mt940
Reviewer Review Type Date Requested Status
Guewen Baconnier @ Camptocamp Approve on 2014-02-28
Stefan Rijnhart (Opener) 2014-02-26 Approve on 2014-02-27
Review via email: mp+208430@code.launchpad.net

Description of the change

This adds a generic mt940 parser (account_banking_mt940) and on top of that a parser for ING's version of MT940 in the Netherlands.

The hope would be that account_banking_mt940 makes it very simple to support other MT940 dialects.

To post a comment you must log in.
197. By Holger Brunn (Therp) on 2014-02-26

[FIX] typos

198. By Holger Brunn (Therp) on 2014-02-26

[IMP] code formatting, forgotten comment

Awesome, finally a generic MT940 parser!

Just two comments from code review, I'll do some testing later on:
- l.122 You might want to add a comment that the exception handling here allows you to execute the file to test the parser outside the context of the OpenERP server (I think).
- l.165 Quite a fashion statement ;-) I know this runs, but you might want to either add two single quotes at the start or remove two at the end so that it looks like a regular docstring.

review: Needs Fixing
199. By Holger Brunn (Therp) on 2014-02-26

[FIX] typos, manifest formatting, comments
[RFR] allow using custom mem_bank_{statement,transaction}s in a
convenient way

Holger Brunn (Therp) (hbrunn) wrote :

thanks for the points, I fixed a typo and added a minor refactoring

Thanks. I have not been able to find a sample statement file for this particular bank, so I won't be able to test. But the code looks excellent so it will have to do. Maybe there are other reviewers who have access to ING structured MT940 statement files?

review: Approve
200. By Holger Brunn (Therp) on 2014-02-28

[FIX] pass cr to refactored method

201. By Holger Brunn (Therp) on 2014-02-28

[ADD] allow transaction without remote_account
[ADD] get some information for unstructured mt940

> This addon provides a generic parser for MT940 files. Given that MT940 is a
> non-open non-standard of pure evil in the way that every bank cooks up its own
> interpretation of it, this addon alone won't help you much. It is rather
> intended to be used by other addons to implement the dialect specific to a
> certain bank.

hmmm I trust you and will try to stay away of this evil dialect.

review: Approve
Holger Brunn (Therp) (hbrunn) wrote :

well, fortunately you have account_banking_mt940 now to use as a pitchfork if you ever face a situation where you can't avoid it

I will remember of that, thanks ;-)

Merged into 6.1, and cherrypicked unmodified in 7.0.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added directory 'account_banking_mt940'
2=== added file 'account_banking_mt940/__init__.py'
3--- account_banking_mt940/__init__.py 1970-01-01 00:00:00 +0000
4+++ account_banking_mt940/__init__.py 2014-02-28 13:34:37 +0000
5@@ -0,0 +1,21 @@
6+# -*- coding: utf-8 -*-
7+##############################################################################
8+#
9+# OpenERP, Open Source Management Solution
10+# This module copyright (C) 2014 Therp BV (<http://therp.nl>).
11+#
12+# This program is free software: you can redistribute it and/or modify
13+# it under the terms of the GNU Affero General Public License as
14+# published by the Free Software Foundation, either version 3 of the
15+# License, or (at your option) any later version.
16+#
17+# This program is distributed in the hope that it will be useful,
18+# but WITHOUT ANY WARRANTY; without even the implied warranty of
19+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
20+# GNU Affero General Public License for more details.
21+#
22+# You should have received a copy of the GNU Affero General Public License
23+# along with this program. If not, see <http://www.gnu.org/licenses/>.
24+#
25+##############################################################################
26+from . import mt940
27
28=== added file 'account_banking_mt940/__openerp__.py'
29--- account_banking_mt940/__openerp__.py 1970-01-01 00:00:00 +0000
30+++ account_banking_mt940/__openerp__.py 2014-02-28 13:34:37 +0000
31@@ -0,0 +1,53 @@
32+# -*- coding: utf-8 -*-
33+##############################################################################
34+#
35+# OpenERP, Open Source Management Solution
36+# This module copyright (C) 2014 Therp BV (<http://therp.nl>).
37+#
38+# This program is free software: you can redistribute it and/or modify
39+# it under the terms of the GNU Affero General Public License as
40+# published by the Free Software Foundation, either version 3 of the
41+# License, or (at your option) any later version.
42+#
43+# This program is distributed in the hope that it will be useful,
44+# but WITHOUT ANY WARRANTY; without even the implied warranty of
45+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
46+# GNU Affero General Public License for more details.
47+#
48+# You should have received a copy of the GNU Affero General Public License
49+# along with this program. If not, see <http://www.gnu.org/licenses/>.
50+#
51+##############################################################################
52+{
53+ "name" : "MT940",
54+ "version" : "1.0",
55+ "author" : "Therp BV",
56+ "complexity": "expert",
57+ "description": """
58+This addon provides a generic parser for MT940 files. Given that MT940 is a
59+non-open non-standard of pure evil in the way that every bank cooks up its own
60+interpretation of it, this addon alone won't help you much. It is rather
61+intended to be used by other addons to implement the dialect specific to a
62+certain bank.
63+
64+See account_banking_nl_ing_mt940 for an example on how to use it.
65+ """,
66+ "category" : "Dependency",
67+ "depends" : [
68+ 'account_banking',
69+ ],
70+ "data" : [
71+ ],
72+ "js": [
73+ ],
74+ "css": [
75+ ],
76+ "qweb": [
77+ ],
78+ "auto_install": False,
79+ "installable": True,
80+ "application": False,
81+ "external_dependencies" : {
82+ 'python' : [],
83+ },
84+}
85
86=== added file 'account_banking_mt940/mt940.py'
87--- account_banking_mt940/mt940.py 1970-01-01 00:00:00 +0000
88+++ account_banking_mt940/mt940.py 2014-02-28 13:34:37 +0000
89@@ -0,0 +1,217 @@
90+#!/usr/bin/env python2
91+# -*- coding: utf-8 -*-
92+##############################################################################
93+#
94+# OpenERP, Open Source Management Solution
95+# This module copyright (C) 2014 Therp BV (<http://therp.nl>).
96+#
97+# This program is free software: you can redistribute it and/or modify
98+# it under the terms of the GNU Affero General Public License as
99+# published by the Free Software Foundation, either version 3 of the
100+# License, or (at your option) any later version.
101+#
102+# This program is distributed in the hope that it will be useful,
103+# but WITHOUT ANY WARRANTY; without even the implied warranty of
104+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
105+# GNU Affero General Public License for more details.
106+#
107+# You should have received a copy of the GNU Affero General Public License
108+# along with this program. If not, see <http://www.gnu.org/licenses/>.
109+#
110+##############################################################################
111+
112+"""
113+Parser for MT940 format files
114+"""
115+import re
116+import datetime
117+import logging
118+try:
119+ from openerp.addons.account_banking.parsers.models import\
120+ mem_bank_statement, mem_bank_transaction
121+ from openerp.tools.misc import DEFAULT_SERVER_DATE_FORMAT
122+except ImportError:
123+ #this allows us to run this file standalone, see __main__ at the end
124+ class mem_bank_statement:
125+ def __init__(self):
126+ self.transactions = []
127+ class mem_bank_transaction:
128+ pass
129+ DEFAULT_SERVER_DATE_FORMAT = "%Y-%m-%d"
130+
131+class MT940(object):
132+ '''Inherit this class in your account_banking.parsers.models.parser,
133+ define functions to handle the tags you need to handle and adjust static
134+ variables as needed.
135+
136+ Note that order matters: You need to do your_parser(MT940, parser), not the
137+ other way around!
138+
139+ At least, you should override handle_tag_61 and handle_tag_86. Don't forget
140+ to call super.
141+ handle_tag_* functions receive the remainder of the the line (that is,
142+ without ':XX:') and are supposed to write into self.current_transaction'''
143+
144+ header_lines = 3
145+ '''One file can contain multiple statements, each with its own poorly
146+ documented header. For now, the best thing to do seems to skip that'''
147+
148+ footer_regex = '^-}$'
149+ footer_regex = '^-XXX$'
150+ 'The line that denotes end of message, we need to create a new statement'
151+
152+ tag_regex = '^:[0-9]{2}[A-Z]*:'
153+ 'The beginning of a record, should be anchored to beginning of the line'
154+
155+ def __init__(self, *args, **kwargs):
156+ super(MT940, self).__init__(*args, **kwargs)
157+ 'state variables'
158+ self.current_statement = None
159+ 'type account_banking.parsers.models.mem_bank_statement'
160+ self.current_transaction = None
161+ 'type account_banking.parsers.models.mem_bank_transaction'
162+ self.statements = []
163+ 'parsed statements up to now'
164+
165+ def parse(self, cr, data):
166+ 'implements account_banking.parsers.models.parser.parse()'
167+ iterator = data.split('\r\n').__iter__()
168+ line = None
169+ record_line = ''
170+ try:
171+ while True:
172+ if not self.current_statement:
173+ self.handle_header(cr, line, iterator)
174+ line = iterator.next()
175+ if not self.is_tag(cr, line) and not self.is_footer(cr, line):
176+ record_line = self.append_continuation_line(
177+ cr, record_line, line)
178+ continue
179+ if record_line:
180+ self.handle_record(cr, record_line)
181+ if self.is_footer(cr, line):
182+ self.handle_footer(cr, line, iterator)
183+ record_line = ''
184+ continue
185+ record_line = line
186+ except StopIteration:
187+ pass
188+ return self.statements
189+
190+ def append_continuation_line(self, cr, line, continuation_line):
191+ '''append a continuation line for a multiline record.
192+ Override and do data cleanups as necessary.'''
193+ return line + continuation_line
194+
195+ def create_statement(self, cr):
196+ '''create a mem_bank_statement - override if you need a custom
197+ implementation'''
198+ return mem_bank_statement()
199+
200+ def create_transaction(self, cr):
201+ '''create a mem_bank_transaction - override if you need a custom
202+ implementation'''
203+ return mem_bank_transaction()
204+
205+ def is_footer(self, cr, line):
206+ '''determine if a line is the footer of a statement'''
207+ return line and bool(re.match(self.footer_regex, line))
208+
209+ def is_tag(self, cr, line):
210+ '''determine if a line has a tag'''
211+ return line and bool(re.match(self.tag_regex, line))
212+
213+ def handle_header(self, cr, line, iterator):
214+ '''skip header lines, create current statement'''
215+ for i in range(self.header_lines):
216+ iterator.next()
217+ self.current_statement = self.create_statement(cr)
218+
219+ def handle_footer(self, cr, line, iterator):
220+ '''add current statement to list, reset state'''
221+ self.statements.append(self.current_statement)
222+ self.current_statement = None
223+
224+ def handle_record(self, cr, line):
225+ '''find a function to handle the record represented by line'''
226+ tag_match = re.match(self.tag_regex, line)
227+ tag = tag_match.group(0).strip(':')
228+ if not hasattr(self, 'handle_tag_%s' % tag):
229+ logging.error('Unknown tag %s', tag)
230+ logging.error(line)
231+ return
232+ handler = getattr(self, 'handle_tag_%s' % tag)
233+ handler(cr, line[tag_match.end():])
234+
235+ def handle_tag_20(self, cr, data):
236+ '''ignore reference number'''
237+ pass
238+
239+ def handle_tag_25(self, cr, data):
240+ '''get account owner information'''
241+ self.current_statement.local_account = data
242+
243+ def handle_tag_28C(self, cr, data):
244+ '''get sequence number _within_this_batch_ - this alone
245+ doesn't provide a unique id!'''
246+ self.current_statement.id = data
247+
248+ def handle_tag_60F(self, cr, data):
249+ '''get start balance and currency'''
250+ self.current_statement.local_currency = data[7:10]
251+ self.current_statement.date = str2date(data[1:7])
252+ self.current_statement.start_balance = \
253+ (1 if data[0] == 'C' else -1) * str2float(data[10:])
254+ self.current_statement.id = '%s/%s' % (
255+ self.current_statement.date.strftime('%Y'),
256+ self.current_statement.id)
257+
258+ def handle_tag_62F(self, cr, data):
259+ '''get ending balance'''
260+ self.current_statement.end_balance = \
261+ (1 if data[0] == 'C' else -1) * str2float(data[10:])
262+
263+ def handle_tag_64(self, cr, data):
264+ '''get current balance in currency'''
265+ pass
266+
267+ def handle_tag_65(self, cr, data):
268+ '''get future balance in currency'''
269+ pass
270+
271+ def handle_tag_61(self, cr, data):
272+ '''get transaction values'''
273+ transaction = self.create_transaction(cr)
274+ self.current_statement.transactions.append(transaction)
275+ self.current_transaction = transaction
276+ transaction.execution_date = str2date(data[:6])
277+ transaction.effective_date = str2date(data[:6])
278+ '...and the rest already is highly bank dependent'
279+
280+ def handle_tag_86(self, cr, data):
281+ '''details for previous transaction, here most differences between
282+ banks occur'''
283+ pass
284+
285+'utility functions'
286+def str2date(string, fmt='%y%m%d'):
287+ return datetime.datetime.strptime(string, fmt)
288+
289+def str2float(string):
290+ return float(string.replace(',', '.'))
291+
292+'testing'
293+def main(filename):
294+ parser = MT940()
295+ parser.parse(None, open(filename, 'r').read())
296+ for statement in parser.statements:
297+ print '''statement found for %(local_account)s at %(date)s
298+ with %(local_currency)s%(start_balance)s to %(end_balance)s
299+ ''' % statement.__dict__
300+ for transaction in statement.transactions:
301+ print '''
302+ transaction on %(execution_date)s''' % transaction.__dict__
303+
304+if __name__ == '__main__':
305+ import sys
306+ main(*sys.argv[1:])
307
308=== added directory 'account_banking_mt940/static'
309=== added directory 'account_banking_mt940/static/src'
310=== added directory 'account_banking_mt940/static/src/img'
311=== added file 'account_banking_mt940/static/src/img/icon.png'
312Binary files account_banking_mt940/static/src/img/icon.png 1970-01-01 00:00:00 +0000 and account_banking_mt940/static/src/img/icon.png 2014-02-28 13:34:37 +0000 differ
313=== added directory 'account_banking_nl_ing_mt940'
314=== added file 'account_banking_nl_ing_mt940/__init__.py'
315--- account_banking_nl_ing_mt940/__init__.py 1970-01-01 00:00:00 +0000
316+++ account_banking_nl_ing_mt940/__init__.py 2014-02-28 13:34:37 +0000
317@@ -0,0 +1,21 @@
318+# -*- coding: utf-8 -*-
319+##############################################################################
320+#
321+# OpenERP, Open Source Management Solution
322+# This module copyright (C) 2014 Therp BV (<http://therp.nl>).
323+#
324+# This program is free software: you can redistribute it and/or modify
325+# it under the terms of the GNU Affero General Public License as
326+# published by the Free Software Foundation, either version 3 of the
327+# License, or (at your option) any later version.
328+#
329+# This program is distributed in the hope that it will be useful,
330+# but WITHOUT ANY WARRANTY; without even the implied warranty of
331+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
332+# GNU Affero General Public License for more details.
333+#
334+# You should have received a copy of the GNU Affero General Public License
335+# along with this program. If not, see <http://www.gnu.org/licenses/>.
336+#
337+##############################################################################
338+from . import account_banking_nl_ing_mt940
339
340=== added file 'account_banking_nl_ing_mt940/__openerp__.py'
341--- account_banking_nl_ing_mt940/__openerp__.py 1970-01-01 00:00:00 +0000
342+++ account_banking_nl_ing_mt940/__openerp__.py 2014-02-28 13:34:37 +0000
343@@ -0,0 +1,48 @@
344+# -*- coding: utf-8 -*-
345+##############################################################################
346+#
347+# OpenERP, Open Source Management Solution
348+# This module copyright (C) 2014 Therp BV (<http://therp.nl>).
349+#
350+# This program is free software: you can redistribute it and/or modify
351+# it under the terms of the GNU Affero General Public License as
352+# published by the Free Software Foundation, either version 3 of the
353+# License, or (at your option) any later version.
354+#
355+# This program is distributed in the hope that it will be useful,
356+# but WITHOUT ANY WARRANTY; without even the implied warranty of
357+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
358+# GNU Affero General Public License for more details.
359+#
360+# You should have received a copy of the GNU Affero General Public License
361+# along with this program. If not, see <http://www.gnu.org/licenses/>.
362+#
363+##############################################################################
364+{
365+ "name" : "MT940 import for Dutch ING",
366+ "version" : "1.0",
367+ "author" : "Therp BV",
368+ "complexity": "normal",
369+ "description": """
370+This addon imports the structured MT940 format as offered by the Dutch ING
371+bank.
372+ """,
373+ "category" : "Account Banking",
374+ "depends" : [
375+ 'account_banking_mt940',
376+ ],
377+ "data" : [
378+ ],
379+ "js": [
380+ ],
381+ "css": [
382+ ],
383+ "qweb": [
384+ ],
385+ "auto_install": False,
386+ "installable": True,
387+ "application": False,
388+ "external_dependencies" : {
389+ 'python' : [],
390+ },
391+}
392
393=== added file 'account_banking_nl_ing_mt940/account_banking_nl_ing_mt940.py'
394--- account_banking_nl_ing_mt940/account_banking_nl_ing_mt940.py 1970-01-01 00:00:00 +0000
395+++ account_banking_nl_ing_mt940/account_banking_nl_ing_mt940.py 2014-02-28 13:34:37 +0000
396@@ -0,0 +1,100 @@
397+# -*- coding: utf-8 -*-
398+##############################################################################
399+#
400+# OpenERP, Open Source Management Solution
401+# This module copyright (C) 2014 Therp BV (<http://therp.nl>).
402+#
403+# This program is free software: you can redistribute it and/or modify
404+# it under the terms of the GNU Affero General Public License as
405+# published by the Free Software Foundation, either version 3 of the
406+# License, or (at your option) any later version.
407+#
408+# This program is distributed in the hope that it will be useful,
409+# but WITHOUT ANY WARRANTY; without even the implied warranty of
410+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
411+# GNU Affero General Public License for more details.
412+#
413+# You should have received a copy of the GNU Affero General Public License
414+# along with this program. If not, see <http://www.gnu.org/licenses/>.
415+#
416+##############################################################################
417+import re
418+from openerp.tools.translate import _
419+from openerp.addons.account_banking.parsers.models import parser,\
420+ mem_bank_transaction
421+from openerp.addons.account_banking_mt940.mt940 import MT940, str2float
422+
423+
424+class transaction(mem_bank_transaction):
425+ def is_valid(self):
426+ '''allow transactions without remote account'''
427+ return bool(self.execution_date) and bool(self.transferred_amount)
428+
429+class IngMT940Parser(MT940, parser):
430+ name = _('ING MT940 (structured)')
431+ country_code = 'NL'
432+ code = 'INT_MT940_STRUC'
433+
434+ tag_61_regex = re.compile(
435+ '^(?P<date>\d{6})(?P<sign>[CD])(?P<amount>\d+,\d{2})N(?P<type>\d{3})'
436+ '(?P<reference>\w{1,16})')
437+
438+ def create_transaction(self, cr):
439+ return transaction()
440+
441+ def handle_tag_60F(self, cr, data):
442+ super(IngMT940Parser, self).handle_tag_60F(cr, data)
443+ self.current_statement.id = '%s-%s' % (
444+ self.get_unique_account_identifier(
445+ cr, self.current_statement.local_account),
446+ self.current_statement.id)
447+
448+ def handle_tag_61(self, cr, data):
449+ super(IngMT940Parser, self).handle_tag_61(cr, data)
450+ parsed_data = self.tag_61_regex.match(data).groupdict()
451+ self.current_transaction.transferred_amount = \
452+ (-1 if parsed_data['sign'] == 'D' else 1) * str2float(
453+ parsed_data['amount'])
454+ self.current_transaction.reference = parsed_data['reference']
455+
456+ def handle_tag_86(self, cr, data):
457+ if not self.current_transaction:
458+ return
459+ super(IngMT940Parser, self).handle_tag_86(cr, data)
460+ codewords = ['RTRN', 'BENM', 'ORDP', 'CSID', 'BUSP', 'MARF', 'EREF',
461+ 'PREF', 'REMI', 'ID', 'PURP', 'ULTB', 'ULTD']
462+ subfields = {}
463+ current_codeword = None
464+ for word in data.split('/'):
465+ if not word and not current_codeword:
466+ continue
467+ if word in codewords:
468+ current_codeword = word
469+ subfields[current_codeword] = []
470+ continue
471+ subfields[current_codeword].append(word)
472+
473+ if 'BENM' in subfields:
474+ self.current_transaction.remote_account = subfields['BENM'][0]
475+ self.current_transaction.remote_bank_bic = subfields['BENM'][1]
476+ self.current_transaction.remote_owner = subfields['BENM'][2]
477+ self.current_transaction.remote_owner_city = subfields['BENM'][3]
478+
479+ if 'ORDP' in subfields:
480+ self.current_transaction.remote_account = subfields['ORDP'][0]
481+ self.current_transaction.remote_bank_bic = subfields['ORDP'][1]
482+ self.current_transaction.remote_owner = subfields['ORDP'][2]
483+ self.current_transaction.remote_owner_city = subfields['ORDP'][3]
484+
485+ if 'REMI' in subfields:
486+ self.current_transaction.message = '/'.join(
487+ filter(lambda x: bool(x), subfields['REMI']))
488+
489+ if self.current_transaction.reference in subfields:
490+ self.current_transaction.reference = ''.join(
491+ subfields[self.current_transaction.reference])
492+
493+ if not subfields:
494+ self.current_transaction.message = data
495+
496+ self.current_transaction = None
497
498=== added directory 'account_banking_nl_ing_mt940/static'
499=== added directory 'account_banking_nl_ing_mt940/static/src'
500=== added directory 'account_banking_nl_ing_mt940/static/src/img'
501=== added file 'account_banking_nl_ing_mt940/static/src/img/icon.png'
502Binary files account_banking_nl_ing_mt940/static/src/img/icon.png 1970-01-01 00:00:00 +0000 and account_banking_nl_ing_mt940/static/src/img/icon.png 2014-02-28 13:34:37 +0000 differ

Subscribers

People subscribed via source and target branches