Banking Addons

bank-statement-reconcile: import of bank statement fails to sniff the csv

Bug #1285075 reported by Guewen Baconnier @ Camptocamp on 2014-02-26

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Banking Addons	Fix Released	Undecided	Unassigned

Bug Description

When importing a CSV file with account_statement_base_import, it tries to auto-detect the dialect of the CSV file to import.
I have some CSV files that cannot be detected, ending with an error: Error: Could not determine delimiter

Example of CSV that can't be sniffed (with many more lines):

transaction_id,date,amount,commission_amount,label
1524624,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524635,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524637,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524643,2014-02-18 00:00:00,276.000000000,4.14,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524660,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524662,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524693,2014-02-18 00:00:00,187.000000000,2.81,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524704,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524712,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524730,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524732,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524735,2014-02-18 00:00:00,187.000000000,2.81,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524737,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524740,2014-02-18 00:00:00,187.000000000,2.81,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524751,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524762,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524766,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524773,2014-02-18 00:00:00,187.000000000,2.81,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889
1524775,2014-02-18 00:00:00,98.000000000,1.47,Sept Payment Batch#x5457631.PEUPUSE.EVP.MRX.209381.100002.20140217.544889

Currently the code is:

    sniffer = csv.Sniffer()
    pos = utf8_data.tell()
    sample_data = utf8_data.read(1024)
    utf8_data.seek(pos) dialect = sniffer.sniff(sample_data, delimiters=',;\t')

From my analysis, it seems that the sample of 1024 bytes is insufficient to guess the delimiter for this type of files. Raising it to 2048 bytes allowed it to guess the delimiter correctly.