Merge lp:~camptocamp/magentoerpconnect/oerp61-oldstable-import-partners-pagination-2013-02-12 into lp:magentoerpconnect/oerp6.1-oldstable

Proposed by Yannick Vaucher @ Camptocamp on 2013-02-12
Status: Merged
Merged at revision: 666
Proposed branch: lp:~camptocamp/magentoerpconnect/oerp61-oldstable-import-partners-pagination-2013-02-12
Merge into: lp:magentoerpconnect/oerp6.1-oldstable
Diff against target: 62 lines (+24/-5)
1 file modified
magentoerpconnect/sale.py (+24/-5)
To merge this branch: bzr merge lp:~camptocamp/magentoerpconnect/oerp61-oldstable-import-partners-pagination-2013-02-12
Reviewer Review Type Date Requested Status
Guewen Baconnier @ Camptocamp Approve on 2013-02-18
Alexandre Fayolle - camptocamp code review, no test 2013-02-12 Approve on 2013-02-18
Review via email: mp+147986@code.launchpad.net

Description of the change

Improve partner import using ol_customer.search before using customer.list in order to paginate the request.

To post a comment you must log in.

l. 28: use xrange instead of range, and you can probably avoid the creation of this list by iterating directly on the xrange object in the for loop line 31.

review: Needs Fixing (code review, no test)

BTW the local variable name customer_groups is a little bit misleading because Magento has a notion of 'customer groups'. customer_chunks would maybe be better. (I propose that just because the MP is already in Needs Fixing).

Ok tanks for the reviews. I'll improve that

Ok this is now improved

review: Approve (code review, no test)
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'magentoerpconnect/sale.py'
2--- magentoerpconnect/sale.py 2013-01-10 06:48:02 +0000
3+++ magentoerpconnect/sale.py 2013-02-14 13:02:20 +0000
4@@ -1,4 +1,4 @@
5-# -*- encoding: utf-8 -*-
6+# -*- coding: utf-8 -*-
7 #########################################################################
8 # #
9 #########################################################################
10@@ -49,13 +49,33 @@
11 'cancel': 'canceled',
12 'waiting_date': 'holded'}
13 SALE_ORDER_IMPORT_STEP = 200
14+PARTNER_IMPORT_STEP = 400
15
16
17 class external_shop_group(magerp_osv.magerp_osv):
18 _inherit = 'external.shop.group'
19
20 @staticmethod
21- def _get_magento_partners(connection, from_date=False, website_id=False):
22+ def _get_magento_partners(connection, filters, delta=PARTNER_IMPORT_STEP):
23+ """
24+ Get customer using pagination, we get all ids
25+ then ask customer info per chunk of ids
26+ this to avoid memory and timing issues on Magento server
27+
28+ :param connection: connection data to magento
29+ :param filters: filter terms for partner search
30+ :param delta: size of groups to import
31+ """
32+ customer_ids = connection.call('ol_customer.search', filters)
33+
34+ data = []
35+ for i in xrange(0, len(customer_ids), delta):
36+ filters = [{'customer_id': {'in': customer_ids[i:i+delta]}}]
37+ data += connection.call('customer.list', filters)
38+ return data
39+
40+ @staticmethod
41+ def _get_magento_partners_update(connection, from_date=False, website_id=False):
42 """
43 Get data from magento of new and updated customers since a specific_date
44
45@@ -75,8 +95,7 @@
46 else:
47 filters = [{'website_id': {'eq': website_id}}]
48
49- data = connection.call('customer.list', filters)
50- return data
51+ return external_shop_group._get_magento_partners(connection, filters)
52
53 def _import_partners(self, cr, uid, group, context=None):
54 """
55@@ -103,7 +122,7 @@
56
57 # Get partners from magento which where created or updated
58 # since last import
59- data = self._get_magento_partners(connection, from_date, website_id)
60+ data = self._get_magento_partners_update(connection, from_date, website_id)
61
62 data.sort(key=lambda customer: customer['updated_at'] or customer['created_at'])
63