Merge lp:~openerp-commiter/openobject-server/trunk-import-binary-file into lp:openobject-server

[Nicolas DS]

To import binary data in a binary field for now you can only does this:
- Use XML-RPC, NET-RPC.
- Using direct python code. (Not nice)
- Convert manually all binary file in Base64 and put it in XML, CSV or YAML file.

This merge allows to import binary file directly into binary field with module by using XML, CSV or YAML.

Usage:
======
* Put a binary file in your module path
ie: my_module/my_picture.png and my_module/pict/robot.jpg

In CSV ("my_model.csv):
-----------------------
id, name, image (binary field)
pict1,My pict,@my_picture.png
pict2,Robot pict,@pict/robot.jpg

In XML:
-------
<record model="my_model" id="pict1">
    <field name="name">My pict</field>
    <field name="image" filename="my_picture.png" />
</record>
<record model="my_model" id="pict2">
    <field name="name">Robot pict</field/>
    <field name="image" filename="pict/robot.jpg" />
</record>

In YAML:
-------------
-
  !record {model: my_model, id: pict1}:
    name: My pict
    image: !filename my_picture.png
-
  !record {model: my_model, id: pict2}:
    name: Robot pict
    image: !filename pict/robot.jpg

Test module available at:
http://zyphos.be/openobject/import_binary_test.zip

Forum topic:
http://openobject.com/forum/topic6577.html

[Christophe Simonis (OpenERP) (kangol)]

The "@" is a hack.

Why not just add a new attribute "filename" to the xml ?
and a "!! filename" directive to the yaml parser ?

the csv parser could possibly keep the "@" hack.

review: Disapprove

[lp:~openerp-commiter/openobject-server/trunk-import-binary-file]

2387. By Nicolas DS on 2010-06-17

[FIX] Use filename attribute for XML and !filename for YAML instead of @

[Nicolas DS (zyphos)]

> The "@" is a hack.
>
> Why not just add a new attribute "filename" to the xml ?
> and a "!! filename" directive to the yaml parser ?
>
> the csv parser could possibly keep the "@" hack.

It's corrected.
See this merge description for the new usage.
Test module has been updated according to those changes.

[lp:~openerp-commiter/openobject-server/trunk-import-binary-file]

2388. By Nicolas DS on 2010-06-17

[FIX] Typos in function name

[Christophe Simonis (OpenERP) (kangol)]

Seems ok to me

review: Approve

[lp:~openerp-commiter/openobject-server/trunk-import-binary-file]

2389. By Nicolas DS on 2010-07-05

[MERGE] with official trunk

2390. By Nicolas DS on 2010-09-14

[MERGE] from official trunk

2391. By Nicolas DS on 2010-09-14

[FIX] Small fix for last merge

[Nicolas DS (zyphos)]

Ready again to be merge into official trunk.

review: Needs Resubmitting

[Nicolas DS (zyphos)]

review: Approve

[lp:~openerp-commiter/openobject-server/trunk-import-binary-file]

2392. By Nicolas DS on 2010-09-20

[FIX] Def beginning with "_" is local only in module

[Fabien (Open ERP) (fp-tinyerp)]

Hello,

The idea is very good.

I don't think you can accept "@..." in import_data which can be called through web-services.
This is a very big security hole. Think about using "@/etc/passwd".

The processing of the @ must be in the XML or YAML processing.

review: Disapprove

[Fabien (Open ERP) (fp-tinyerp)]

You can resubmit when this is fixed, as it's a good contribution.

Unmerged revisions

2392. By Nicolas DS on 2010-09-20

[FIX] Def beginning with "_" is local only in module

2391. By Nicolas DS on 2010-09-14

[FIX] Small fix for last merge

2390. By Nicolas DS on 2010-09-14

[MERGE] from official trunk

2389. By Nicolas DS on 2010-07-05

[MERGE] with official trunk

2388. By Nicolas DS on 2010-06-17

[FIX] Typos in function name

2387. By Nicolas DS on 2010-06-17

[FIX] Use filename attribute for XML and !filename for YAML instead of @

2386. By Nicolas DS on 2010-06-17

[IMP] Import @ targeted file in binary field, for CSV, XML and YAML
Usage:
- Put a file (picture, data,...) in your module path. Ie: mymodule/my_picture.png
- In XML, YAML, or CSV put the relative path preceded by a @. Ie: @my_picture.png

=== modified file 'bin/import_xml.rng'
--- bin/import_xml.rng	2010-06-09 13:40:13 +0000
+++ bin/import_xml.rng	2010-09-20 09:13:46 +0000
@@ -127,6 +127,7 @@
             <rng:optional><rng:attribute name="search"/></rng:optional>
             <rng:optional><rng:attribute name="model"/></rng:optional>
             <rng:optional><rng:attribute name="use"/></rng:optional>
+            <rng:optional><rng:attribute name="filename"/></rng:optional>
             <rng:oneOrMore>
                 <rng:choice>
                     <rng:ref name="any"/>

=== modified file 'bin/osv/orm.py'
--- bin/osv/orm.py	2010-09-20 09:00:56 +0000
+++ bin/osv/orm.py	2010-09-20 09:13:46 +0000
@@ -849,6 +849,10 @@
                         res = line[i].lower() not in ('0', 'false', 'off')
                     elif fields_def[field[len(prefix)]]['type'] == 'float':
                         res = line[i] and float(line[i])
+                    elif fields_def[field[len(prefix)]]['type'] == 'binary':
+                        if line[i] and line[i][0] == '@':
+                            # if binary field and first char is '@', import targeted file
+                            res = tools.get_file(current_module, line[i][1:])
                     elif fields_def[field[len(prefix)]]['type'] == 'selection':
                         res = False
                         if isinstance(fields_def[field[len(prefix)]]['selection'],

=== modified file 'bin/tools/convert.py'
--- bin/tools/convert.py	2010-09-16 11:47:26 +0000
+++ bin/tools/convert.py	2010-09-20 09:13:46 +0000
@@ -25,6 +25,7 @@
 import os.path
 import pickle
 import re
+import base64
 
 # for eval context:
 import time
@@ -68,6 +69,7 @@
     model = pool.get(model_str)
     return lambda x: model.browse(cr, uid, x, context=context)
 
+<<<<<<< TREE
 def _fix_multiple_roots(node):
     """
     Surround the children of the ``node`` element of an XML field with a
@@ -86,6 +88,13 @@
             data_node.append(child)
         node.append(data_node)
 
+=======
+def get_file(module, filename):
+    path = os.path.join(module, filename)
+    file_data = misc.file_open(path,'rb').read()
+    return base64.encodestring(file_data)
+                
+>>>>>>> MERGE-SOURCE
 def _eval_xml(self, node, pool, cr, uid, idref, context=None):
     if context is None:
         context = {}
@@ -769,6 +778,7 @@
             f_name = field.get("name",'').encode('utf-8')
             f_ref = field.get("ref",'').encode('utf-8')
             f_search = field.get("search",'').encode('utf-8')
+            f_filename = field.get("filename",'').encode('utf-8')
             f_model = field.get("model",'').encode('utf-8')
             if not f_model and model._columns.get(f_name,False):
                 f_model = model._columns[f_name]._obj
@@ -801,6 +811,8 @@
                         f_val = val[0] + ',' + str(val[1])
                     else:
                         f_val = self.id_get(cr, f_model, f_ref)
+            elif f_filename:
+                f_val = get_file(self.module, f_filename)
             else:
                 f_val = _eval_xml(self,field, self.pool, cr, self.uid, self.idref)
                 if model._columns.has_key(f_name):

=== modified file 'bin/tools/yaml_import.py'
--- bin/tools/yaml_import.py	2010-09-17 09:41:00 +0000
+++ bin/tools/yaml_import.py	2010-09-20 09:13:46 +0000
@@ -713,7 +713,7 @@
         """
         Processes a Yaml string. Custom tags are interpreted by 'process_' instance methods.
         """
-        yaml_tag.add_constructors()
+        yaml_tag.add_constructors(self.module)
 
         is_preceded_by_comment = False
         for node in yaml.load(yaml_string):

=== modified file 'bin/tools/yaml_tag.py'
--- bin/tools/yaml_tag.py	2010-04-30 14:18:10 +0000
+++ bin/tools/yaml_tag.py	2010-09-20 09:13:46 +0000
@@ -1,6 +1,8 @@
 import yaml
 import logging
 
+import convert
+
 class YamlTag(object):
     """
     Superclass for constructors of custom tags defined in yaml file.
@@ -143,6 +145,12 @@
     else:
         kwargs = loader.construct_mapping(node)
     return Ref(**kwargs)
+
+def get_filename_constructor(module):
+    def filename_constructor(loader, node):
+        filename = loader.construct_scalar(node)
+        return convert.get_file(module,filename)
+    return filename_constructor
     
 def ir_set_constructor(loader, node):
     kwargs = loader.construct_mapping(node)
@@ -151,7 +159,7 @@
 # Registers constructors for custom tags.
 # Constructors are actually defined globally: do not redefined them in another
 # class/file/package.  This means that module recorder need import this file.
-def add_constructors():
+def add_constructors(module = ''):
     yaml.add_constructor(u"!assert", assert_constructor)
     yaml.add_constructor(u"!record", record_constructor)
     yaml.add_constructor(u"!python", python_constructor)
@@ -166,4 +174,5 @@
     yaml.add_constructor(u"!eval", eval_constructor)
     yaml.add_multi_constructor(u"!ref", ref_constructor)
     yaml.add_constructor(u"!ir_set", ir_set_constructor)
+    yaml.add_constructor(u"!filename", get_filename_constructor(module))
 add_constructors()
\ No newline at end of file