Merge lp:~paul-lucas/zorba/pjl-misc into lp:zorba

[Paul J. Lucas]

Discovered a bug where strings like street addresses, e.g., "870 Market Street", will cause an exception to be thrown due to the mis-guessing about what kind of JSON token a string really is. Fixed.

Discovered a bug where strings like street addresses, e.g., "870 Market Street", will cause an exception to be thrown due to the mis-guessing about what kind of JSON token a string really is. Fixed.

[lp:~paul-lucas/zorba/pjl-misc]

11463. By Paul J. Lucas on 2014-03-01

Fixed typo in comment.

[Paul J. Lucas (paul-lucas)]

review: Approve

[Matthias Brantner (matthias-brantner)]

review: Approve

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for the following merge proposals:
https://code.launchpad.net/~paul-lucas/zorba/pjl-misc/+merge/208907

Progress dashboard at http://jenkins.zorba.io:8180/view/ValidationQueue

[Zorba Build Bot (zorba-buildbot)]

Validation queue succeeded - proposal merged!

=== modified file 'src/runtime/csv/csv_impl.cpp'
--- src/runtime/csv/csv_impl.cpp	2014-02-28 01:46:12 +0000
+++ src/runtime/csv/csv_impl.cpp	2014-03-01 01:57:20 +0000
@@ -162,8 +162,31 @@
   istringstream iss;
   iss.ios::rdbuf( &buf );
   json::lexer lex( iss );
-  return lex.next( ptoken, false ) ?
-    json::map_type( ptoken->get_type() ) : json::none;
+  if ( !lex.next( ptoken, false ) )
+    return json::none;
+  json::token::type const tt = ptoken->get_type();
+  if ( tt == json::token::number ) {
+    //
+    // The JSON lexer will stop lex'ing as soon as it finds a valid token.  For
+    // strings that start with numbers, e.g., "870 Market St", the lexer will
+    // return "870" as a number token but we need to know the type of the whole
+    // string 's'; hence we need to do an extra check for number tokens.
+    //
+    // If the length(t) < length(s), remove trailing whitespace from 's' and
+    // check again.  If length(t) < length(s2), it means that that the token is
+    // really a string; if length(t) == length(s2), then the length difference
+    // was caused only by whitespace that can be ignored and the token is still
+    // a number.
+    //
+    json::token::value_type const &value = ptoken->get_value();
+    if ( value.size() < s.size() ) {
+      zstring s2;
+      ascii::trim_end_space( s, &s2 );
+      if ( value.size() < s2.size() )
+        return json::string;
+    }
+  }
+  return json::map_type( tt );
 }
 
 static void set_keys( store::Item_t const &item, vector<store::Item_t> *keys,
 
=== added file 'test/rbkt/ExpQueryResults/zorba/csv/csv-parse-address.xml.res'
--- test/rbkt/ExpQueryResults/zorba/csv/csv-parse-address.xml.res	1970-01-01 00:00:00 +0000
+++ test/rbkt/ExpQueryResults/zorba/csv/csv-parse-address.xml.res	2014-03-01 01:57:20 +0000
@@ -0,0 +1,3 @@
+{
+  "Street" : "870 Market Street"
+}
 
=== added file 'test/rbkt/Queries/zorba/csv/csv-parse-address.jq'
--- test/rbkt/Queries/zorba/csv/csv-parse-address.jq	1970-01-01 00:00:00 +0000
+++ test/rbkt/Queries/zorba/csv/csv-parse-address.jq	2014-03-01 01:57:20 +0000
@@ -0,0 +1,7 @@
+import module namespace csv = "http://zorba.io/modules/json-csv";
+
+let $csv := "870 Market Street"
+let $options := { "field-names" : [ "Street" ] }
+return csv:parse( $csv, $options )
+
+(: vim:set syntax=xquery et sw=2 ts=2 :)
 
=== added file 'test/rbkt/Queries/zorba/csv/csv-parse-address.spec'
--- test/rbkt/Queries/zorba/csv/csv-parse-address.spec	1970-01-01 00:00:00 +0000
+++ test/rbkt/Queries/zorba/csv/csv-parse-address.spec	2014-03-01 01:57:20 +0000
@@ -0,0 +1,1 @@
+Serialization: indent=yes