Merge lp:~camptocamp/magentoerpconnect/oerp61-oldstable-import-partners-pagination-2013-02-12 into lp:magentoerpconnect/oerp6.1-oldstable

[Yannick Vaucher @ Camptocamp]

Improve partner import using ol_customer.search before using customer.list in order to paginate the request.

[Alexandre Fayolle - camptocamp (alexandre-fayolle-c2c)]

l. 28: use xrange instead of range, and you can probably avoid the creation of this list by iterating directly on the xrange object in the for loop line 31.

review: Needs Fixing (code review, no test)

[Guewen Baconnier @ Camptocamp (gbaconnier-c2c)]

BTW the local variable name customer_groups is a little bit misleading because Magento has a notion of 'customer groups'. customer_chunks would maybe be better. (I propose that just because the MP is already in Needs Fixing).

[Yannick Vaucher @ Camptocamp (yvaucher-c2c)]

Ok tanks for the reviews. I'll improve that

[Yannick Vaucher @ Camptocamp (yvaucher-c2c)]

Ok this is now improved

[Alexandre Fayolle - camptocamp (alexandre-fayolle-c2c)]

review: Approve (code review, no test)

[Guewen Baconnier @ Camptocamp (gbaconnier-c2c)]

review: Approve

=== modified file 'magentoerpconnect/sale.py'
--- magentoerpconnect/sale.py	2013-01-10 06:48:02 +0000
+++ magentoerpconnect/sale.py	2013-02-14 13:02:20 +0000
@@ -1,4 +1,4 @@
-# -*- encoding: utf-8 -*-
+# -*- coding: utf-8 -*-
 #########################################################################
 #                                                                       #
 #########################################################################
@@ -49,13 +49,33 @@
     'cancel': 'canceled',
     'waiting_date': 'holded'}
 SALE_ORDER_IMPORT_STEP = 200
+PARTNER_IMPORT_STEP = 400
 
 
 class external_shop_group(magerp_osv.magerp_osv):
     _inherit = 'external.shop.group'
 
     @staticmethod
-    def _get_magento_partners(connection, from_date=False, website_id=False):
+    def _get_magento_partners(connection, filters, delta=PARTNER_IMPORT_STEP):
+        """
+        Get customer using pagination, we get all ids
+        then ask customer info per chunk of ids
+        this to avoid memory and timing issues on Magento server
+
+        :param connection: connection data to magento
+        :param filters: filter terms for partner search
+        :param delta: size of groups to import
+        """
+        customer_ids = connection.call('ol_customer.search', filters)
+
+        data = []
+        for i in xrange(0, len(customer_ids), delta):
+            filters = [{'customer_id': {'in': customer_ids[i:i+delta]}}]
+            data += connection.call('customer.list', filters)
+        return data
+
+    @staticmethod
+    def _get_magento_partners_update(connection, from_date=False, website_id=False):
         """
         Get data from magento of new and updated customers since a specific_date
 
@@ -75,8 +95,7 @@
             else:
                 filters = [{'website_id': {'eq': website_id}}]
 
-        data = connection.call('customer.list', filters)
-        return data
+        return external_shop_group._get_magento_partners(connection, filters)
 
     def _import_partners(self, cr, uid, group, context=None):
         """
@@ -103,7 +122,7 @@
 
         # Get partners from magento which where created or updated
         # since last import
-        data = self._get_magento_partners(connection, from_date, website_id)
+        data = self._get_magento_partners_update(connection, from_date, website_id)
 
         data.sort(key=lambda customer: customer['updated_at'] or customer['created_at'])