Merge lp:~openerp-connector-core-editors/openerp-connector/7.0-connector-mapper-refactor into lp:~openerp-connector-core-editors/openerp-connector/7.0

Proposed by Guewen Baconnier @ Camptocamp on 2013-11-08
Status: Superseded
Proposed branch: lp:~openerp-connector-core-editors/openerp-connector/7.0-connector-mapper-refactor
Merge into: lp:~openerp-connector-core-editors/openerp-connector/7.0
Diff against target: 1344 lines (+902/-187)
8 files modified
connector/CHANGES.rst (+3/-0)
connector/backend.py (+7/-4)
connector/connector.py (+5/-1)
connector/doc/guides/concepts.rst (+4/-8)
connector/exception.py (+4/-0)
connector/tests/test_backend.py (+3/-2)
connector/tests/test_mapper.py (+333/-40)
connector/unit/mapper.py (+543/-132)
To merge this branch: bzr merge lp:~openerp-connector-core-editors/openerp-connector/7.0-connector-mapper-refactor
Reviewer Review Type Date Requested Status
OpenERP Connector Core Editors 2013-11-08 Pending
Review via email: mp+194484@code.launchpad.net

This proposal has been superseded by a proposal from 2013-11-08.

Description of the change

Refactoring of the Mappers.
===========================

Work in progress.

To post a comment you must log in.
584. By Guewen Baconnier @ Camptocamp on 2013-11-08

[CHG] rename map_record.values(only_create=True) to map_record.values(for_create=True) -> less ambiguous

585. By Guewen Baconnier @ Camptocamp on 2013-11-08

[CHG] move deprecated methods, set some private methods to public (the ones that are really intended to be inherited)

586. By Guewen Baconnier @ Camptocamp on 2013-11-08

[CHG] updated the docstring according to the last developments

587. By Guewen Baconnier @ Camptocamp on 2013-11-08

[FIX] simplify the MapRecord.update() method

588. By Guewen Baconnier @ Camptocamp on 2013-11-08

[IMP] docstring

589. By Guewen Baconnier @ Camptocamp on 2013-11-09

[FIX] mapping options: use an object whose items can be accessed like attributes.

It offers a far better interface as well as for the the call to MapRecord.values() as for the usage in mapping methods

590. By Guewen Baconnier @ Camptocamp on 2013-11-09

[FIX] the backward compatibility code was outdated: binding is the name of the binding to use, to provide only when the relation is not a binding

591. By Guewen Baconnier @ Camptocamp on 2013-11-12

[CHG] MapOptions now returns None when a key is undefined, allowing a better handling in mapping methods when an option is not used

592. By Guewen Baconnier @ Camptocamp on 2013-12-09

[MRG] sync with master branch

593. By Guewen Baconnier @ Camptocamp on 2013-12-09

[MRG] sync with master branch

594. By Guewen Baconnier @ Camptocamp on 2013-12-20

[IMP] extract the build of the MapChild unit instance to allow better reusability

595. By Guewen Baconnier @ Camptocamp on 2013-12-24

[MRG] from lp:openerp-connector

Unmerged revisions

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'connector/CHANGES.rst'
2--- connector/CHANGES.rst 2013-10-09 18:52:36 +0000
3+++ connector/CHANGES.rst 2013-11-08 11:46:46 +0000
4@@ -4,6 +4,9 @@
5 2.0.1.dev0
6 ~~~~~~~~~~
7
8+* Add the possibility to use 'Modifiers' functions in the 'direct
9+ mappings' (details in the documentation of the Mapper class)
10+
11 2.0.1 (2013-09-12)
12 ~~~~~~~~~~~~~~~~~~
13
14
15=== modified file 'connector/backend.py'
16--- connector/backend.py 2013-08-26 20:05:18 +0000
17+++ connector/backend.py 2013-11-08 11:46:46 +0000
18@@ -20,6 +20,7 @@
19 ##############################################################################
20 from functools import partial
21 from collections import namedtuple
22+from .exception import NoConnectorUnitError
23
24 __all__ = ['Backend']
25
26@@ -257,10 +258,12 @@
27 """
28 matching_classes = self._get_classes(base_class, session,
29 model_name)
30- assert matching_classes, ('No matching class found for %s '
31- 'with session: %s, '
32- 'model name: %s' %
33- (base_class, session, model_name))
34+ if not matching_classes:
35+ raise NoConnectorUnitError('No matching class found for %s '
36+ 'with session: %s, '
37+ 'model name: %s' %
38+ (base_class, session, model_name))
39+
40 assert len(matching_classes) == 1, (
41 'Several classes found for %s '
42 'with session %s, model name: %s. Found: %s' %
43
44=== modified file 'connector/connector.py'
45--- connector/connector.py 2013-09-13 13:22:49 +0000
46+++ connector/connector.py 2013-11-08 11:46:46 +0000
47@@ -283,11 +283,15 @@
48 """
49 raise NotImplementedError
50
51- def to_backend(self, binding_id):
52+ def to_backend(self, binding_id, wrap=False):
53 """ Give the external ID for an OpenERP binding ID
54 (ID in a model magento.*)
55
56 :param binding_id: OpenERP binding ID for which we want the backend id
57+ :param wrap: if False, binding_id is the ID of the binding,
58+ if True, binding_id is the ID of the normal record, the
59+ method will search the corresponding binding and returns
60+ the backend id of the binding
61 :return: external ID of the record
62 """
63 raise NotImplementedError
64
65=== modified file 'connector/doc/guides/concepts.rst'
66--- connector/doc/guides/concepts.rst 2013-11-01 09:10:27 +0000
67+++ connector/doc/guides/concepts.rst 2013-11-08 11:46:46 +0000
68@@ -146,8 +146,7 @@
69 systems.
70
71 The connector defines some base classes, which you can find below.
72-Note that you can define your own ConnectorUnits as well without
73-reusing them.
74+Note that you can define your own ConnectorUnits as well.
75
76 Mappings
77 ========
78@@ -159,19 +158,16 @@
79
80 It supports:
81
82-* direct mappings
83-
84+direct mappings
85 Fields *a* is written in field *b*.
86
87-* method mappings
88-
89+method mappings
90 A method is used to convert one or many fields to one or many
91 fields, with transformation.
92 It can be filtered, for example only applied when the record is
93 created or when the source fields are modified.
94
95-* submapping
96-
97+submapping
98 a sub-record (lines of a sale order) is converted using another
99 Mapper
100
101
102=== modified file 'connector/exception.py'
103--- connector/exception.py 2013-06-19 14:47:17 +0000
104+++ connector/exception.py 2013-11-08 11:46:46 +0000
105@@ -24,6 +24,10 @@
106 """ Base Exception for the connectors """
107
108
109+class NoConnectorUnitError(ConnectorException):
110+ """ No ConnectorUnit has been found """
111+
112+
113 class InvalidDataError(ConnectorException):
114 """ Data Invalid """
115
116
117=== modified file 'connector/tests/test_backend.py'
118--- connector/tests/test_backend.py 2013-08-23 20:34:05 +0000
119+++ connector/tests/test_backend.py 2013-11-08 11:46:46 +0000
120@@ -6,6 +6,7 @@
121 from openerp.addons.connector.backend import (Backend,
122 get_backend,
123 BACKENDS)
124+from openerp.addons.connector.exception import NoConnectorUnitError
125 from openerp.addons.connector.connector import (Binder,
126 ConnectorUnit)
127 from openerp.addons.connector.unit.mapper import (Mapper,
128@@ -73,7 +74,7 @@
129 self.uid)
130
131 def tearDown(self):
132- super(test_backend_register, self).setUp()
133+ super(test_backend_register, self).tearDown()
134 BACKENDS.backends.clear()
135 del self.backend._class_entries[:]
136
137@@ -110,7 +111,7 @@
138
139 def test_no_register_error(self):
140 """ Error when asking for a class and none is found"""
141- with self.assertRaises(AssertionError):
142+ with self.assertRaises(NoConnectorUnitError):
143 ref = self.backend.get_class(BackendAdapter,
144 self.session,
145 'res.users')
146
147=== modified file 'connector/tests/test_mapper.py'
148--- connector/tests/test_mapper.py 2013-04-08 11:48:59 +0000
149+++ connector/tests/test_mapper.py 2013-11-08 11:46:46 +0000
150@@ -2,15 +2,26 @@
151
152 import unittest2
153 import mock
154+import openerp.tests.common as common
155
156 from openerp.addons.connector.unit.mapper import (
157 Mapper,
158 ImportMapper,
159+ ImportMapChild,
160 MappingDefinition,
161 changed_by,
162 only_create,
163+ convert,
164+ m2o_to_backend,
165+ backend_to_m2o,
166+ none,
167 mapping)
168
169+from openerp.addons.connector.exception import NoConnectorUnitError
170+from openerp.addons.connector.backend import Backend
171+from openerp.addons.connector.connector import Environment
172+from openerp.addons.connector.session import ConnectorSession
173+
174
175 class test_mapper(unittest2.TestCase):
176 """ Test Mapper """
177@@ -151,11 +162,11 @@
178 record = {'name': 'Guewen',
179 'street': 'street'}
180 mapper = MyMapper(env)
181- mapper.convert(record)
182+ map_record = mapper.map_record(record)
183 expected = {'out_name': 'Guewen',
184 'out_street': 'STREET'}
185- self.assertEqual(mapper.data, expected)
186- self.assertEqual(mapper.data_for_create, expected)
187+ self.assertEqual(map_record.values(), expected)
188+ self.assertEqual(map_record.values(only_create=True), expected)
189
190 def test_mapping_record_on_create(self):
191 """ Map a record and check the result for creation of record """
192@@ -176,61 +187,343 @@
193 record = {'name': 'Guewen',
194 'street': 'street'}
195 mapper = MyMapper(env)
196- mapper.convert(record)
197+ map_record = mapper.map_record(record)
198 expected = {'out_name': 'Guewen',
199 'out_street': 'STREET'}
200- self.assertEqual(mapper.data, expected)
201+ self.assertEqual(map_record.values(), expected)
202 expected = {'out_name': 'Guewen',
203 'out_street': 'STREET',
204 'out_city': 'city'}
205- self.assertEqual(mapper.data_for_create, expected)
206+ self.assertEqual(map_record.values(only_create=True), expected)
207+
208+ def test_mapping_update(self):
209+ """ Force values on a map record """
210+ class MyMapper(ImportMapper):
211+
212+ direct = [('name', 'out_name')]
213+
214+ @mapping
215+ def street(self, record):
216+ return {'out_street': record['street'].upper()}
217+
218+ @only_create
219+ @mapping
220+ def city(self, record):
221+ return {'out_city': 'city'}
222+
223+ env = mock.MagicMock()
224+ record = {'name': 'Guewen',
225+ 'street': 'street'}
226+ mapper = MyMapper(env)
227+ map_record = mapper.map_record(record)
228+ map_record.update(out_city='forced')
229+ map_record.update({'test': 1})
230+ expected = {'out_name': 'Guewen',
231+ 'out_street': 'STREET',
232+ 'out_city': 'forced',
233+ 'test': 1}
234+ self.assertEqual(map_record.values(), expected)
235+ expected = {'out_name': 'Guewen',
236+ 'out_street': 'STREET',
237+ 'out_city': 'forced',
238+ 'test': 1}
239+ self.assertEqual(map_record.values(only_create=True), expected)
240+
241+ def test_finalize(self):
242+ """ Inherit _finalize to modify values """
243+ class MyMapper(ImportMapper):
244+
245+ direct = [('name', 'out_name')]
246+
247+ def _finalize(self, record, values):
248+ result = super(MyMapper, self)._finalize(record, values)
249+ result['test'] = 'abc'
250+ return result
251+
252+ env = mock.MagicMock()
253+ record = {'name': 'Guewen',
254+ 'street': 'street'}
255+ mapper = MyMapper(env)
256+ map_record = mapper.map_record(record)
257+ expected = {'out_name': 'Guewen',
258+ 'test': 'abc'}
259+ self.assertEqual(map_record.values(), expected)
260+ expected = {'out_name': 'Guewen',
261+ 'test': 'abc'}
262+ self.assertEqual(map_record.values(only_create=True), expected)
263+
264+ def test_some_fields(self):
265+ """ Map only a selection of fields """
266+ class MyMapper(ImportMapper):
267+
268+ direct = [('name', 'out_name'),
269+ ('street', 'out_street'),
270+ ]
271+
272+ @changed_by('country')
273+ @mapping
274+ def country(self, record):
275+ return {'country': 'country'}
276+
277+ env = mock.MagicMock()
278+ record = {'name': 'Guewen',
279+ 'street': 'street',
280+ 'country': 'country'}
281+ mapper = MyMapper(env)
282+ map_record = mapper.map_record(record)
283+ expected = {'out_name': 'Guewen',
284+ 'country': 'country'}
285+ self.assertEqual(map_record.values(fields=['name', 'country']),
286+ expected)
287+ expected = {'out_name': 'Guewen',
288+ 'country': 'country'}
289+ self.assertEqual(map_record.values(only_create=True,
290+ fields=['name', 'country']),
291+ expected)
292+
293+ def test_mapping_modifier(self):
294+ """ Map a direct record with a modifier function """
295+
296+ def do_nothing(field):
297+ def transform(self, record, to_attr):
298+ return record[field]
299+ return transform
300+
301+ class MyMapper(ImportMapper):
302+ direct = [(do_nothing('name'), 'out_name')]
303+
304+ env = mock.MagicMock()
305+ record = {'name': 'Guewen'}
306+ mapper = MyMapper(env)
307+ map_record = mapper.map_record(record)
308+ expected = {'out_name': 'Guewen'}
309+ self.assertEqual(map_record.values(), expected)
310+ self.assertEqual(map_record.values(only_create=True), expected)
311+
312+ def test_mapping_convert(self):
313+ """ Map a direct record with the convert modifier function """
314+ class MyMapper(ImportMapper):
315+ direct = [(convert('name', int), 'out_name')]
316+
317+ env = mock.MagicMock()
318+ record = {'name': '300'}
319+ mapper = MyMapper(env)
320+ map_record = mapper.map_record(record)
321+ expected = {'out_name': 300}
322+ self.assertEqual(map_record.values(), expected)
323+ self.assertEqual(map_record.values(only_create=True), expected)
324+
325+ def test_mapping_modifier_none(self):
326+ """ Pipeline of modifiers """
327+ class MyMapper(ImportMapper):
328+ direct = [(none('in_f'), 'out_f'),
329+ (none('in_t'), 'out_t')]
330+
331+ env = mock.MagicMock()
332+ record = {'in_f': False, 'in_t': True}
333+ mapper = MyMapper(env)
334+ map_record = mapper.map_record(record)
335+ expected = {'out_f': None, 'out_t': True}
336+ self.assertEqual(map_record.values(), expected)
337+ self.assertEqual(map_record.values(only_create=True), expected)
338+
339+ def test_mapping_modifier_pipeline(self):
340+ """ Pipeline of modifiers """
341+ class MyMapper(ImportMapper):
342+ direct = [(none(convert('in_f', bool)), 'out_f'),
343+ (none(convert('in_t', bool)), 'out_t')]
344+
345+ env = mock.MagicMock()
346+ record = {'in_f': 0, 'in_t': 1}
347+ mapper = MyMapper(env)
348+ map_record = mapper.map_record(record)
349+ expected = {'out_f': None, 'out_t': True}
350+ self.assertEqual(map_record.values(), expected)
351+ self.assertEqual(map_record.values(only_create=True), expected)
352+
353+ def test_mapping_custom_option(self):
354+ """ Usage of custom options in mappings """
355+ class MyMapper(ImportMapper):
356+ @mapping
357+ def any(self, record):
358+ if self.options.get('custom'):
359+ res = True
360+ else:
361+ res = False
362+ return {'res': res}
363+
364+ env = mock.MagicMock()
365+ record = {}
366+ mapper = MyMapper(env)
367+ map_record = mapper.map_record(record)
368+ expected = {'res': True}
369+ self.assertEqual(map_record.values(options=dict(custom=True)),
370+ expected)
371+
372+
373+class test_mapper_binding(common.TransactionCase):
374+ """ Test Mapper with Bindings"""
375+
376+ def setUp(self):
377+ super(test_mapper_binding, self).setUp()
378+ self.session = ConnectorSession(self.cr, self.uid)
379+ self.Partner = self.registry('res.partner')
380+ self.backend = mock.Mock(wraps=Backend('x', version='y'),
381+ name='backend')
382+ backend_record = mock.Mock()
383+ backend_record.get_backend.return_value = self.backend
384+ self.env = Environment(backend_record, self.session, 'res.partner')
385+ self.country_binder = mock.Mock(name='country_binder')
386+ self.country_binder.return_value = self.country_binder
387+ self.backend.get_class.return_value = self.country_binder
388+
389+ def test_mapping_m2o_to_backend(self):
390+ """ Map a direct record with the m2o_to_backend modifier function """
391+ class MyMapper(ImportMapper):
392+ _model_name = 'res.partner'
393+ direct = [(m2o_to_backend('country_id'), 'country')]
394+
395+ partner_id = self.ref('base.main_partner')
396+ self.Partner.write(self.cr, self.uid, partner_id,
397+ {'country_id': self.ref('base.ch')})
398+ partner = self.Partner.browse(self.cr, self.uid, partner_id)
399+ self.country_binder.to_backend.return_value = 10
400+
401+ mapper = MyMapper(self.env)
402+ map_record = mapper.map_record(partner)
403+ self.assertEqual(map_record.values(), {'country': 10})
404+ self.country_binder.to_backend.assert_called_once_with(
405+ partner.country_id.id, wrap=False)
406+
407+ def test_mapping_backend_to_m2o(self):
408+ """ Map a direct record with the backend_to_m2o modifier function """
409+ class MyMapper(ImportMapper):
410+ _model_name = 'res.partner'
411+ direct = [(backend_to_m2o('country'), 'country_id')]
412+
413+ record = {'country': 10}
414+ self.country_binder.to_openerp.return_value = 44
415+ mapper = MyMapper(self.env)
416+ map_record = mapper.map_record(record)
417+ self.assertEqual(map_record.values(), {'country_id': 44})
418+ self.country_binder.to_openerp.assert_called_once_with(
419+ 10, unwrap=False)
420+
421+ def test_mapping_record_children_no_map_child(self):
422+ """ Map a record with children, using default MapChild """
423+
424+ backend = Backend('backend', '42')
425+
426+ @backend
427+ class LineMapper(ImportMapper):
428+ _model_name = 'res.currency.rate'
429+ direct = [('name', 'name')]
430+
431+ @mapping
432+ def price(self, record):
433+ return {'rate': record['rate'] * 2}
434+
435+ @only_create
436+ @mapping
437+ def discount(self, record):
438+ return {'test': .5}
439+
440+ @backend
441+ class ObjectMapper(ImportMapper):
442+ _model_name = 'res.currency'
443+
444+ direct = [('name', 'name')]
445+
446+ children = [('lines', 'line_ids', 'res.currency.rate')]
447+
448+
449+ backend_record = mock.Mock()
450+ backend_record.get_backend.side_effect = lambda *a: backend
451+ env = Environment(backend_record, self.session, 'res.currency')
452+
453+ record = {'name': 'SO1',
454+ 'lines': [{'name': '2013-11-07',
455+ 'rate': 10},
456+ {'name': '2013-11-08',
457+ 'rate': 20}]}
458+ mapper = ObjectMapper(env)
459+ map_record = mapper.map_record(record)
460+ expected = {'name': 'SO1',
461+ 'line_ids': [(0, 0, {'name': '2013-11-07',
462+ 'rate': 20}),
463+ (0, 0, {'name': '2013-11-08',
464+ 'rate': 40})]
465+ }
466+ self.assertEqual(map_record.values(), expected)
467+ expected = {'name': 'SO1',
468+ 'line_ids': [(0, 0, {'name': '2013-11-07',
469+ 'rate': 20,
470+ 'test': .5}),
471+ (0, 0, {'name': '2013-11-08',
472+ 'rate': 40,
473+ 'test': .5})]
474+ }
475+ self.assertEqual(map_record.values(only_create=True), expected)
476
477 def test_mapping_record_children(self):
478- """ Map a record with children and check the result """
479-
480+ """ Map a record with children, using defined MapChild """
481+
482+ backend = Backend('backend', '42')
483+
484+ @backend
485 class LineMapper(ImportMapper):
486+ _model_name = 'res.currency.rate'
487 direct = [('name', 'name')]
488
489 @mapping
490 def price(self, record):
491- return {'price': record['price'] * 2}
492+ return {'rate': record['rate'] * 2}
493
494 @only_create
495 @mapping
496 def discount(self, record):
497- return {'discount': .5}
498-
499- class SaleMapper(ImportMapper):
500+ return {'test': .5}
501+
502+ @backend
503+ class SaleLineImportMapChild(ImportMapChild):
504+ _model_name = 'res.currency.rate'
505+ def _format_items(self, items_values):
506+ return [('ABC', values) for values in items_values]
507+
508+
509+ @backend
510+ class ObjectMapper(ImportMapper):
511+ _model_name = 'res.currency'
512
513 direct = [('name', 'name')]
514
515- children = [('lines', 'line_ids', 'sale.order.line')]
516-
517- def _init_child_mapper(self, model_name):
518- return LineMapper(self.environment)
519-
520- env = mock.MagicMock()
521+ children = [('lines', 'line_ids', 'res.currency.rate')]
522+
523+
524+ backend_record = mock.Mock()
525+ backend_record.get_backend.side_effect = lambda *a: backend
526+ env = Environment(backend_record, self.session, 'res.currency')
527
528 record = {'name': 'SO1',
529- 'lines': [{'name': 'Mango',
530- 'price': 10},
531- {'name': 'Pumpkin',
532- 'price': 20}]}
533- mapper = SaleMapper(env)
534- mapper.convert(record)
535- expected = {'name': 'SO1',
536- 'line_ids': [(0, 0, {'name': 'Mango',
537- 'price': 20}),
538- (0, 0, {'name': 'Pumpkin',
539- 'price': 40})]
540- }
541- self.assertEqual(mapper.data, expected)
542- expected = {'name': 'SO1',
543- 'line_ids': [(0, 0, {'name': 'Mango',
544- 'price': 20,
545- 'discount': .5}),
546- (0, 0, {'name': 'Pumpkin',
547- 'price': 40,
548- 'discount': .5})]
549- }
550- self.assertEqual(mapper.data_for_create, expected)
551+ 'lines': [{'name': '2013-11-07',
552+ 'rate': 10},
553+ {'name': '2013-11-08',
554+ 'rate': 20}]}
555+ mapper = ObjectMapper(env)
556+ map_record = mapper.map_record(record)
557+ expected = {'name': 'SO1',
558+ 'line_ids': [('ABC', {'name': '2013-11-07',
559+ 'rate': 20}),
560+ ('ABC', {'name': '2013-11-08',
561+ 'rate': 40})]
562+ }
563+ self.assertEqual(map_record.values(), expected)
564+ expected = {'name': 'SO1',
565+ 'line_ids': [('ABC', {'name': '2013-11-07',
566+ 'rate': 20,
567+ 'test': .5}),
568+ ('ABC', {'name': '2013-11-08',
569+ 'rate': 40,
570+ 'test': .5})]
571+ }
572+ self.assertEqual(map_record.values(only_create=True), expected)
573
574=== modified file 'connector/unit/mapper.py'
575--- connector/unit/mapper.py 2013-10-28 15:17:17 +0000
576+++ connector/unit/mapper.py 2013-11-08 11:46:46 +0000
577@@ -21,9 +21,10 @@
578
579 import logging
580 from collections import namedtuple
581+from contextlib import contextmanager
582
583 from ..connector import ConnectorUnit, MetaConnectorUnit, Environment
584-from ..exception import MappingError
585+from ..exception import MappingError, NoConnectorUnitError
586
587 _logger = logging.getLogger(__name__)
588
589@@ -60,6 +61,142 @@
590 return func
591
592
593+def none(field):
594+ """ A modifier intended to be used on the ``direct`` mappings.
595+
596+ Replace the False-ish values by None.
597+ It can be used in a pipeline of modifiers when .
598+
599+ Example::
600+
601+ direct = [(none('source'), 'target'),
602+ (none(m2o_to_backend('rel_id'), 'rel_id')]
603+
604+ :param field: name of the source field in the record
605+ :param binding: True if the relation is a binding record
606+ """
607+ def modifier(self, record, to_attr):
608+ if callable(field):
609+ result = field(self, record, to_attr)
610+ else:
611+ result = record[field]
612+ if not result:
613+ return None
614+ return result
615+ return modifier
616+
617+
618+def convert(field, conv_type):
619+ """ A modifier intended to be used on the ``direct`` mappings.
620+
621+ Convert a field's value to a given type.
622+
623+ Example::
624+
625+ direct = [(convert('source', str), 'target')]
626+
627+ :param field: name of the source field in the record
628+ :param binding: True if the relation is a binding record
629+ """
630+ def modifier(self, record, to_attr):
631+ value = record[field]
632+ if not value:
633+ return False
634+ return conv_type(value)
635+ return modifier
636+
637+
638+def m2o_to_backend(field, binding=None):
639+ """ A modifier intended to be used on the ``direct`` mappings.
640+
641+ For a many2one, get the ID on the backend and returns it.
642+
643+ When the field's relation is not a binding (i.e. it does not point to
644+ something like ``magento.*``), the binding model needs to be provided
645+ in the ``binding`` keyword argument.
646+
647+ Example::
648+
649+ direct = [(m2o_to_backend('country_id', binding='magento.res.country'),
650+ 'country'),
651+ (m2o_to_backend('magento_country_id'), 'country')]
652+
653+ :param field: name of the source field in the record
654+ :param binding: name of the binding model is the relation is not a binding
655+ """
656+ def modifier(self, record, to_attr):
657+ if not record[field]:
658+ return False
659+ column = self.model._all_columns[field].column
660+ if column._type != 'many2one':
661+ raise ValueError('The column %s should be a many2one, got %s' %
662+ field, column._type)
663+ rel_id = record[field].id
664+ if binding is None:
665+ binding_model = column._obj
666+ else:
667+ binding_model = binding
668+ binder = self.get_binder_for_model(binding_model)
669+ # if a relation is not a binding, we wrap the record in the
670+ # binding, we'll return the id of the binding
671+ wrap = bool(binding)
672+ value = binder.to_backend(rel_id, wrap=wrap)
673+ if not value:
674+ raise MappingError("Can not find an external id for record "
675+ "%s in model %s %s wrapping" %
676+ (rel_id, binding_model,
677+ 'with' if wrap else 'without'))
678+ return value
679+ return modifier
680+
681+
682+def backend_to_m2o(field, binding=None, with_inactive=False):
683+ """ A modifier intended to be used on the ``direct`` mappings.
684+
685+ For a field from a backend which is an ID, search the corresponding
686+ binding in OpenERP and returns its ID.
687+
688+ When the field's relation is not a binding (i.e. it does not point to
689+ something like ``magento.*``), the binding model needs to be provided
690+ in the ``binding`` keyword argument.
691+
692+ Example::
693+
694+ direct = [(backend_to_m2o('country', binding='magento.res.country'),
695+ 'country_id'),
696+ (backend_to_m2o('country'), 'magento_country_id')]
697+
698+ :param field: name of the source field in the record
699+ :param binding: name of the binding model is the relation is not a binding
700+ :param with_inactive: include the inactive records in OpenERP in the search
701+ """
702+ def modifier(self, record, to_attr):
703+ if not record[field]:
704+ return False
705+ column = self.model._all_columns[to_attr].column
706+ if column._type != 'many2one':
707+ raise ValueError('The column %s should be a many2one, got %s' %
708+ to_attr, column._type)
709+ rel_id = record[field]
710+ if binding is None:
711+ binding_model = column._obj
712+ else:
713+ binding_model = binding
714+ binder = self.get_binder_for_model(binding_model)
715+ # if we want the ID of a normal record, not a binding,
716+ # we ask the unwrapped id to the binder
717+ unwrap = bool(binding)
718+ with self.session.change_context({'active_test': False}):
719+ value = binder.to_openerp(rel_id, unwrap=unwrap)
720+ if not value:
721+ raise MappingError("Can not find an existing %s for external "
722+ "record %s %s unwrapping" %
723+ (binding_model, rel_id,
724+ 'with' if unwrap else 'without'))
725+ return value
726+ return modifier
727+
728+
729 MappingDefinition = namedtuple('MappingDefinition',
730 ['changed_by',
731 'only_create'])
732@@ -105,8 +242,189 @@
733 return cls
734
735
736+class MapChild(ConnectorUnit):
737+ """
738+
739+ """
740+ _model_name = None
741+
742+ def _child_mapper(self):
743+ raise NotImplementedError
744+
745+ def _skip_convert_child(self, map_record):
746+ """ Hook to implement in sub-classes when some child
747+ records should be skipped.
748+
749+ The parent record is accessible in ``map_record``.
750+ If it returns True, the current child record is skipped.
751+
752+ :param map_record: record that we are converting
753+ :type map_record: :py:class:`MapRecord`
754+ """
755+ return False
756+
757+ def get_items(self, items, parent, to_attr, options):
758+ mapper = self._child_mapper()
759+ mapped = []
760+ for item in items:
761+ map_record = mapper.map_record(item, parent=parent)
762+ if self._skip_convert_child(map_record):
763+ continue
764+ mapped.append(self._get_item_values(map_record, to_attr, options))
765+ return self._format_items(mapped)
766+
767+ def _get_item_values(self, map_record, to_attr, options):
768+ """ Get the values from the child Mappers for the items.
769+
770+ It can be overridden for instance to:
771+
772+ * Change options
773+ * Use a Binder to know if an item already exists to modify an
774+ existing item, rather than to add it
775+
776+ :param map_record: record that we are converting
777+ :type map_record: :py:class:`MapRecord`
778+ :param to_attr: destination field (can be used for introspecting
779+ the relation)
780+ :type to_attr: str
781+ :param options: dict of options, herited from the main mapper
782+ """
783+ return map_record.values(options=options)
784+
785+ def _format_items(self, items_values):
786+ """ Format the values of the items mapped from the child Mappers.
787+
788+ It can be overridden for instance to add the OpenERP
789+ relationships commands ``(6, 0, [IDs])``, ...
790+
791+ As instance, it can be modified to handle update of existing
792+ items: check if 'id' existings in values, use the ``(1, ID,
793+ {values}``) command
794+
795+ :param items_values: list of values for the items to create
796+ """
797+ return items_values
798+
799+
800+class ImportMapChild(MapChild):
801+
802+ def _child_mapper(self):
803+ return self.get_connector_unit_for_model(ImportMapper, self.model._name)
804+
805+ def _format_items(self, items_values):
806+ """ Format the values of the items mapped from the child Mappers.
807+
808+ It can be overridden for instance to add the OpenERP
809+ relationships commands ``(6, 0, [IDs])``, ...
810+
811+ As instance, it can be modified to handle update of existing
812+ items: check if 'id' existings in values, use the ``(1, ID,
813+ {values}``) command
814+
815+ :param items_values: list of values for the items to create
816+ """
817+ return [(0, 0, values) for values in items_values]
818+
819+
820+class ExportMapChild(MapChild):
821+
822+ def _child_mapper(self):
823+ return self.get_connector_unit_for_model(ExportMapper, self.model._name)
824+
825+
826 class Mapper(ConnectorUnit):
827- """ Transform a record to a defined output """
828+ """ A Mapper translates an external record to an OpenERP record and
829+ conversely. The output of a Mapper is a ``dict``.
830+
831+ 3 types of mappings are supported:
832+
833+ Direct Mappings
834+ Example::
835+
836+ direct = [('source', 'target')]
837+
838+ Here, the ``source`` field will be copied in the ``target`` field.
839+
840+ A modifier can be used in the source item.
841+ The modifier will be applied to the source field before being
842+ copied in the target field.
843+ It should be a closure function respecting this idiom::
844+
845+ def a_function(field):
846+ ''' ``field`` is the name of the source field '''
847+ def modifier(self, record, to_attr):
848+ ''' self is the current Mapper,
849+ record is the current record to map,
850+ to_attr is the target field'''
851+ return record[field]
852+ return modifier
853+
854+ And used like that::
855+
856+ direct = [
857+ (a_function('source'), 'target'),
858+ ]
859+
860+ A more concrete example of modifier::
861+
862+ def convert(field, conv_type):
863+ ''' Convert the source field to a defined ``conv_type``
864+ (ex. str) before returning it'''
865+ def modifier(self, record, to_attr):
866+ value = record[field]
867+ if not value:
868+ return None
869+ return conv_type(value)
870+ return modifier
871+
872+ And used like that::
873+
874+ direct = [
875+ (convert('myfield', float), 'target_field'),
876+ ]
877+
878+ More examples of modifiers:
879+
880+ * :py:func:`convert`
881+ * :py:func:`m2o_to_backend`
882+ * :py:func:`backend_to_m2o`
883+
884+ Method Mappings
885+ A mapping method allows to execute arbitrary code and return one
886+ or many fields::
887+
888+ @mapping
889+ def compute_state(self, record):
890+ # compute some state, using the ``record`` or not
891+ state = 'pending'
892+ return {'state': state}
893+
894+ We can also specify that a mapping methods should be applied
895+ only when an object is created, and never applied on further
896+ updates::
897+
898+ @only_create
899+ @mapping
900+ def default_warehouse(self, record):
901+ # get default warehouse
902+ warehouse_id = ...
903+ return {'warehouse_id': warehouse_id}
904+
905+ Submappings
906+ When a record contains sub-items, like the lines of a sales order,
907+ we can convert the children using another Mapper::
908+
909+ children = [('items', 'line_ids', LineMapper)]
910+
911+ Usage of a Mapper::
912+
913+ mapper = Mapper(env)
914+ map_record = mapper.map_record()
915+ values = map_record.values()
916+ values = map_record.values(only_create=True)
917+ values = map_record.values(fields=['name', 'street'])
918+
919+ """
920
921 __metaclass__ = MetaMapper
922
923@@ -118,6 +436,8 @@
924
925 _map_methods = None
926
927+ _map_child_class = None
928+
929 def __init__(self, environment):
930 """
931
932@@ -125,191 +445,282 @@
933 :type environment: :py:class:`connector.connector.Environment`
934 """
935 super(Mapper, self).__init__(environment)
936- self._data = None
937- self._data_for_create = None
938- self._data_children = None
939-
940- def _init_child_mapper(self, model_name):
941- raise NotImplementedError
942+ self._options = None
943
944 def _after_mapping(self, result):
945- return result
946+ """ Mapper.convert_child() has been deprecated """
947+ raise DeprecationWarning('Mapper._after_mapping() has been deprecated, '
948+ 'use Mapper._finalize()')
949
950 def _map_direct(self, record, from_attr, to_attr):
951+ """ Apply the ``direct`` mappings.
952+
953+ :param record: record to convert from a source to a target
954+ :param from_attr: name of the source attribute or a callable
955+ :type from_attr: callable | str
956+ :param to_attr: name of the target attribute
957+ :type to_attr: str
958+ """
959 raise NotImplementedError
960
961 def _map_children(self, record, attr, model):
962 raise NotImplementedError
963
964+ def skip_convert_child(self, record, parent_values=None):
965+ """ Hook to implement in sub-classes when some child
966+ records should be skipped.
967+
968+ This method is only relevable for child mappers.
969+
970+ If it returns True, the current child record is skipped."""
971+ return False
972+
973 @property
974 def map_methods(self):
975 for meth, definition in self._map_methods.iteritems():
976 yield getattr(self, meth), definition
977
978- def _convert(self, record, fields=None, parent_values=None):
979- if fields is None:
980- fields = {}
981-
982- self._data = {}
983- self._data_for_create = {}
984- self._data_children = {}
985-
986- _logger.debug('converting record %s to model %s', record, self._model_name)
987+ def convert_child(self, record, parent_values=None):
988+ """ Mapper.convert_child() has been deprecated """
989+ raise DeprecationWarning('Mapper.convert_child() has been deprecated, '
990+ 'use Mapper.map_record() then '
991+ 'map_record.values() ')
992+
993+ def convert(self, record, fields=None):
994+ """ Mapper.convert() has been deprecated
995+
996+ The usage of a Mapper is now::
997+
998+ map_record = Mapper(env).map_record(record)
999+ values = map_record.values()
1000+
1001+ See :py:meth:`Mapper.map_record`, :py:meth:`MapRecord.values`
1002+
1003+ """
1004+ raise DeprecationWarning('Mapper.convert() has been deprecated, '
1005+ 'use Mapper.map_record() then '
1006+ 'map_record.values() ')
1007+
1008+ @property
1009+ def data(self):
1010+ """ Mapper.data has been deprecated
1011+
1012+ See :py:meth:`Mapper.map_record`, :py:meth:`MapRecord.values`
1013+
1014+ """
1015+ raise DeprecationWarning('Mapper.data has been deprecated, '
1016+ 'use Mapper.map_record() then '
1017+ 'map_record.values() ')
1018+
1019+ @property
1020+ def data_for_create(self):
1021+ """ Mapper.data has been deprecated
1022+
1023+ See :py:meth:`Mapper.map_record`, :py:meth:`MapRecord.values`
1024+ """
1025+ raise DeprecationWarning('Mapper.data_for_create has been deprecated, '
1026+ 'use Mapper.map_record() then '
1027+ 'map_record.values() ')
1028+
1029+ def _map_child(self, map_record, from_attr, to_attr, model_name):
1030+ assert self._map_child_class is not None, "_map_child_class required"
1031+ child_records = map_record.source[from_attr]
1032+ try:
1033+ mapper_child = self.get_connector_unit_for_model(
1034+ self._map_child_class, model_name)
1035+ except NoConnectorUnitError:
1036+ # does not force developers to use a MapChild ->
1037+ # will use the default one if not explicitely defined
1038+ env = Environment(self.backend_record,
1039+ self.session,
1040+ model_name)
1041+ mapper_child = self._map_child_class(env)
1042+ items = mapper_child.get_items(child_records, map_record,
1043+ to_attr, options=self.options)
1044+ return items
1045+
1046+ @contextmanager
1047+ def _mapping_options(self, options):
1048+ """ Change the mapping options for the Mapper.
1049+
1050+ Context Manager to use in order to alter the behavior
1051+ of the mapping, when using ``_apply`` or ``finalize``.
1052+
1053+ """
1054+ current = self._options
1055+ self._options = options
1056+ yield
1057+ self._options = current
1058+
1059+ @property
1060+ def options(self):
1061+ """ Should not be modified manually, only with _mapping_options.
1062+
1063+ Options can be accessed in the mapping methods.
1064+
1065+ """
1066+ return self._options
1067+
1068+ def map_record(self, record, parent=None):
1069+ """ Get a MapRecord with record, ready to be converted using the
1070+ current Mapper.
1071+
1072+ :param record: record to transform
1073+ """
1074+ return MapRecord(self, record, parent=parent)
1075+
1076+ def _apply(self, map_record, options=None):
1077+ if options is None:
1078+ options = {}
1079+ with self._mapping_options(options):
1080+ return self._apply_with_options(map_record)
1081+
1082+ def _apply_with_options(self, map_record):
1083+ assert self.options is not None, (
1084+ "options should be defined with '_mapping_options'")
1085+ _logger.debug('converting record %s to model %s',
1086+ map_record.source, self.model._name)
1087+
1088+ fields = self.options.get('fields')
1089+ only_create = self.options.get('only_create')
1090+ result = {}
1091 for from_attr, to_attr in self.direct:
1092 if (not fields or from_attr in fields):
1093- # XXX not compatible with all
1094- # record type (wrap
1095- # records in a standard class representation?)
1096- value = self._map_direct(record,
1097+ value = self._map_direct(map_record.source,
1098 from_attr,
1099 to_attr)
1100- self._data[to_attr] = value
1101+ result[to_attr] = value
1102
1103 for meth, definition in self.map_methods:
1104 changed_by = definition.changed_by
1105 if (not fields or not changed_by or
1106 changed_by.intersection(fields)):
1107- values = meth(record)
1108+ values = meth(map_record.source)
1109 if not values:
1110 continue
1111 if not isinstance(values, dict):
1112 raise ValueError('%s: invalid return value for the '
1113 'mapping method %s' % (values, meth))
1114- if definition.only_create:
1115- self._data_for_create.update(values)
1116- else:
1117- self._data.update(values)
1118+ if not definition.only_create or only_create:
1119+ result.update(values)
1120
1121 for from_attr, to_attr, model_name in self.children:
1122 if (not fields or from_attr in fields):
1123- self._map_child(record, from_attr, to_attr, model_name)
1124-
1125- def skip_convert_child(self, record, parent_values=None):
1126- """ Hook to implement in sub-classes when some child
1127- records should be skipped.
1128-
1129- This method is only relevable for child mappers.
1130-
1131- If it returns True, the current child record is skipped."""
1132- return False
1133-
1134- def convert_child(self, record, parent_values=None):
1135- """ Transform child row contained in a main record, only
1136- called from another Mapper.
1137-
1138- :param parent_values: openerp record of the containing object
1139- (e.g. sale_order for a sale_order_line)
1140- """
1141- self._convert(record, parent_values=parent_values)
1142-
1143- def convert(self, record, fields=None):
1144- """ Transform an external record to an OpenERP record or the opposite
1145-
1146- Sometimes we want to map values only when we create the records.
1147- The mapping methods have to be decorated with ``only_create`` to
1148- map values only for creation of records.
1149-
1150- :param record: record to transform
1151- :param fields: list of fields to convert, if empty, all fields
1152- are converted
1153- """
1154- self._convert(record, fields=fields)
1155-
1156- @property
1157- def data(self):
1158- """ Returns a dict for a record processed by
1159- :py:meth:`~_convert` """
1160- if self._data is None:
1161- raise ValueError('Mapper.convert should be called before '
1162- 'accessing the data')
1163- result = self._data.copy()
1164- for attr, mappers in self._data_children.iteritems():
1165- child_data = [mapper.data for mapper in mappers]
1166- if child_data:
1167- result[attr] = self._format_child_rows(child_data)
1168- return self._after_mapping(result)
1169-
1170- @property
1171- def data_for_create(self):
1172- """ Returns a dict for a record processed by
1173- :py:meth:`~_convert` to use only for creation of the record. """
1174- if self._data is None:
1175- raise ValueError('Mapper.convert should be called before '
1176- 'accessing the data')
1177- result = self._data.copy()
1178- result.update(self._data_for_create)
1179- for attr, mappers in self._data_children.iteritems():
1180- child_data = [mapper.data_for_create for mapper in mappers]
1181- if child_data:
1182- result[attr] = self._format_child_rows(child_data)
1183- return self._after_mapping(result)
1184-
1185- def _format_child_rows(self, child_records):
1186- return child_records
1187-
1188- def _map_child(self, record, from_attr, to_attr, model_name):
1189- child_records = record[from_attr]
1190- self._data_children[to_attr] = []
1191- for child_record in child_records:
1192- mapper = self._init_child_mapper(model_name)
1193- if mapper.skip_convert_child(child_record, parent_values=record):
1194- continue
1195- mapper.convert_child(child_record, parent_values=record)
1196- self._data_children[to_attr].append(mapper)
1197+ result[to_attr] = self._map_child(map_record, from_attr,
1198+ to_attr, model_name)
1199+
1200+ return self._finalize(map_record, result)
1201+
1202+ def _finalize(self, map_record, values):
1203+ """ Called at the end of the mapping. Can be used to
1204+ modify the values before returning them.
1205+
1206+ :param map_record: source map_record
1207+ :type map_record: :py:class:`MapRecord`
1208+ :param values: mapped values
1209+ :returns: mapped values
1210+ :rtype: dict
1211+ """
1212+ return values
1213
1214
1215 class ImportMapper(Mapper):
1216 """ Transform a record from a backend to an OpenERP record """
1217
1218+ _map_child_class = ImportMapChild
1219+
1220 def _map_direct(self, record, from_attr, to_attr):
1221+ """ Apply the ``direct`` mappings.
1222+
1223+ :param record: record to convert from a source to a target
1224+ :param from_attr: name of the source attribute or a callable
1225+ :type from_attr: callable | str
1226+ :param to_attr: name of the target attribute
1227+ :type to_attr: str
1228+ """
1229+ if callable(from_attr):
1230+ return from_attr(self, record, to_attr)
1231+
1232 value = record.get(from_attr)
1233 if not value:
1234 return False
1235
1236+ # may be replaced by the explicit backend_to_m2o
1237 column = self.model._all_columns[to_attr].column
1238 if column._type == 'many2one':
1239- rel_id = record[from_attr]
1240- model_name = column._obj
1241- binder = self.get_binder_for_model(model_name)
1242- value = binder.to_openerp(rel_id)
1243-
1244- if not value:
1245- raise MappingError("Can not find an existing %s for external "
1246- "record %s" % (model_name, rel_id))
1247+ mapping = backend_to_m2o(from_attr, binding=True)
1248+ value = mapping(self, record, to_attr)
1249 return value
1250
1251- def _init_child_mapper(self, model_name):
1252- env = Environment(self.backend_record,
1253- self.session,
1254- model_name)
1255- return env.get_connector_unit(ImportMapper)
1256-
1257- def _format_child_rows(self, child_records):
1258- return [(0, 0, data) for data in child_records]
1259-
1260
1261 class ExportMapper(Mapper):
1262 """ Transform a record from OpenERP to a backend record """
1263
1264+ _map_child_class = ExportMapChild
1265+
1266 def _map_direct(self, record, from_attr, to_attr):
1267+ """ Apply the ``direct`` mappings.
1268+
1269+ :param record: record to convert from a source to a target
1270+ :param from_attr: name of the source attribute or a callable
1271+ :type from_attr: callable | str
1272+ :param to_attr: name of the target attribute
1273+ :type to_attr: str
1274+ """
1275+ if callable(from_attr):
1276+ return from_attr(self, record, to_attr)
1277+
1278 value = record[from_attr]
1279 if not value:
1280 return False
1281
1282+ # may be replaced by the explicit m2o_to_backend
1283 column = self.model._all_columns[from_attr].column
1284 if column._type == 'many2one':
1285- rel_id = record[from_attr].id
1286- model_name = column._obj
1287- binder = self.get_binder_for_model(model_name)
1288- value = binder.to_backend(rel_id)
1289-
1290- if not value:
1291- raise MappingError("Can not find an external id for record "
1292- "%s in model %s" % (rel_id, model_name))
1293+ mapping = m2o_to_backend(from_attr, binding=True)
1294+ value = mapping(self, record, to_attr)
1295 return value
1296
1297- def _init_child_mapper(self, model_name):
1298- env = Environment(self.backend_record,
1299- self.session,
1300- model_name)
1301- return env.get_connector_unit(ExportMapper)
1302+
1303+class MapRecord(object):
1304+
1305+ def __init__(self, mapper, source, parent=None):
1306+ self._source = source
1307+ self._mapper = mapper
1308+ self._parent = parent
1309+ self._forced_values = {}
1310+
1311+ @property
1312+ def source(self):
1313+ return self._source
1314+
1315+ @property
1316+ def parent(self):
1317+ return self._parent
1318+
1319+ def values(self, only_create=None, fields=None, options=None):
1320+ """
1321+
1322+ Sometimes we want to map values only for creation the records.
1323+ The mapping methods have to be decorated with ``only_create`` to
1324+ map values only for creation of records. Then, we must give the
1325+ ``only_create=True`` argument to ``values()``.
1326+
1327+ """
1328+ if options is None:
1329+ options = {}
1330+ if only_create is not None:
1331+ options['only_create'] = only_create
1332+ if fields is not None:
1333+ options['fields'] = fields
1334+ values = self._mapper._apply(self, options=options)
1335+ values.update(self._forced_values)
1336+ return values
1337+
1338+ def update(self, *args, **kwargs):
1339+ if args:
1340+ assert len(args) == 1, 'dict expected, got: %s' % args
1341+ assert isinstance(args[0], dict), 'dict expected, got %s' % args
1342+ self._forced_values.update(args[0])
1343+ if kwargs:
1344+ self._forced_values.update(kwargs)