Merge lp:~zorba-coders/zorba/tokenize into lp:zorba

Proposed by Matthias Brantner on 2011-12-23
Status: Superseded
Proposed branch: lp:~zorba-coders/zorba/tokenize
Merge into: lp:zorba
Diff against target: 605 lines (+368/-2)
25 files modified
ChangeLog (+2/-0)
modules/com/zorba-xquery/www/modules/CMakeLists.txt (+1/-1)
modules/com/zorba-xquery/www/modules/string.xq (+21/-1)
src/functions/pregenerated/func_strings.cpp (+23/-0)
src/functions/pregenerated/func_strings.h (+15/-0)
src/functions/pregenerated/function_enum.h (+1/-0)
src/runtime/spec/strings/strings.xml (+31/-0)
src/runtime/strings/pregenerated/strings.cpp (+42/-0)
src/runtime/strings/pregenerated/strings.h (+52/-0)
src/runtime/strings/strings_impl.cpp (+130/-0)
src/runtime/visitors/pregenerated/planiter_visitor.h (+5/-0)
src/runtime/visitors/pregenerated/printer_visitor.cpp (+14/-0)
src/runtime/visitors/pregenerated/printer_visitor.h (+3/-0)
test/rbkt/ExpQueryResults/zorba/string/tokenize01.xml.res (+1/-0)
test/rbkt/ExpQueryResults/zorba/string/tokenize02.xml.res (+1/-0)
test/rbkt/ExpQueryResults/zorba/string/tokenize03.xml.res (+1/-0)
test/rbkt/ExpQueryResults/zorba/string/tokenize04.xml.res (+1/-0)
test/rbkt/Queries/zorba/string/token01.txt (+1/-0)
test/rbkt/Queries/zorba/string/token02.txt (+1/-0)
test/rbkt/Queries/zorba/string/token03.txt (+1/-0)
test/rbkt/Queries/zorba/string/token04.txt (+1/-0)
test/rbkt/Queries/zorba/string/tokenize01.xq (+5/-0)
test/rbkt/Queries/zorba/string/tokenize02.xq (+5/-0)
test/rbkt/Queries/zorba/string/tokenize03.xq (+5/-0)
test/rbkt/Queries/zorba/string/tokenize04.xq (+5/-0)
To merge this branch: bzr merge lp:~zorba-coders/zorba/tokenize
Reviewer Review Type Date Requested Status
William Candillon 2011-12-23 Approve on 2011-12-24
Matthias Brantner Approve on 2011-12-23
Paul J. Lucas 2011-12-23 Pending
Review via email: mp+86835@code.launchpad.net

This proposal supersedes a proposal from 2011-12-23.

This proposal has been superseded by a proposal from 2011-12-23.

Commit message

implementation of string:split function that doesn't accept regular expressions but allows for streamable processing of the input (resolves bug #898074)

Description of the change

implementation of string:split function that doesn't accept regular expressions but allows for streamable processing of the input (resolves bug #898074)

To post a comment you must log in.
Zorba Build Bot (zorba-buildbot) wrote : Posted in a previous version of this proposal
Zorba Build Bot (zorba-buildbot) wrote : Posted in a previous version of this proposal

Validation queue job tokenize-2011-12-21T21-46-05.289Z is finished. The final status was:

All tests succeeded!

Zorba Build Bot (zorba-buildbot) wrote : Posted in a previous version of this proposal

Voting does not meet specified criteria. Required: Approve > 1, Disapprove < 1. Got: 2 Pending.

Paul J. Lucas (paul-lucas) wrote : Posted in a previous version of this proposal

On line 357, why do you call assert()? An invalid byte should throw an exception, not assert and dump core.

Paul J. Lucas (paul-lucas) wrote : Posted in a previous version of this proposal

> On line 357, why do you call assert()? An invalid byte should throw an
> exception, not assert and dump core.

I meant line 367.

Paul J. Lucas (paul-lucas) : Posted in a previous version of this proposal
review: Needs Fixing
Matthias Brantner (matthias-brantner) wrote : Posted in a previous version of this proposal

Once you finished the implementation of the transcoding stream buffer, I don't even want to do this check anymore. This must not happen with the stream buffer.

Paul J. Lucas (paul-lucas) wrote : Posted in a previous version of this proposal

> Once you finished the implementation of the transcoding stream buffer, I don't
> even want to do this check anymore. This must not happen with the stream
> buffer.

I don't understand how it "must not happen." It can always happen. However, I think you're saying that you assume the check will happen in the transcoder. While it will be doing checks, bad input can still happen.

In the mean time, using an assert() is still too Draconian.

Matthias Brantner (matthias-brantner) wrote : Posted in a previous version of this proposal

I have replaced the assertion with a graceful error.

Paul J. Lucas (paul-lucas) : Posted in a previous version of this proposal
review: Approve
William Candillon (wcandillon) wrote : Posted in a previous version of this proposal

Is there an example that works with streaming?
I wasn't able to make the following work:
import module namespace http = "http://www.zorba-xquery.com/modules/http-client";

declare namespace h = "http://expath.org/ns/http-client";

let $item := http:send-request(<h:request href="https://stream.twitter.com/1/statuses/sample.json?delimited=length"
                                          method="GET"
                                          username="wcandillon"
                                          password="wnvbb86g"
                                          override-media-type="text/plain"
                               />
                               ,
                               "https://stream.twitter.com/1/statuses/sample.json?delimited=length",
                               ()
                                )[2]
for $tweet in tokenize($item,"a")
return $tweet

Where:
import module namespace http = "http://www.zorba-xquery.com/modules/http-client";

declare namespace h = "http://expath.org/ns/http-client";

let $item := http:send-request(<h:request href="https://stream.twitter.com/1/statuses/sample.json?delimited=length"
                                          method="GET"
                                          username="wcandillon"
                                          password="wnvbb86g"
                                          override-media-type="text/plain"
                               />
                               ,
                               "https://stream.twitter.com/1/statuses/sample.json?delimited=length",
                               ()
                                )[2]
return $item

streams fine.
What am I missing?

Matthias Brantner (matthias-brantner) wrote : Posted in a previous version of this proposal

As discussed in this thread, only the new tokenize function of the string module streams.
Use the following instead

import module namespace s = "http://www.zorba-xquery.com/modules/string";

s:tokenize($item, "a")

William Candillon (wcandillon) wrote : Posted in a previous version of this proposal

Works like a charm.

review: Approve
Zorba Build Bot (zorba-buildbot) wrote : Posted in a previous version of this proposal

Attempt to merge into lp:zorba failed due to conflicts:

text conflict in ChangeLog

Matthias Brantner (matthias-brantner) : Posted in a previous version of this proposal
review: Approve
William Candillon (wcandillon) wrote : Posted in a previous version of this proposal

Works great.

review: Approve
Zorba Build Bot (zorba-buildbot) wrote : Posted in a previous version of this proposal
Zorba Build Bot (zorba-buildbot) wrote : Posted in a previous version of this proposal

The attempt to merge lp:~zorba-coders/zorba/tokenize into lp:zorba failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:273 (message):
  Validation queue job tokenize-2011-12-23T20-55-01.864Z is finished. The
  final status was:

  1 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

Zorba Build Bot (zorba-buildbot) wrote : Posted in a previous version of this proposal
Zorba Build Bot (zorba-buildbot) wrote : Posted in a previous version of this proposal

The attempt to merge lp:~zorba-coders/zorba/tokenize into lp:zorba failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:273 (message):
  Validation queue job tokenize-2011-12-23T21-25-57.422Z is finished. The
  final status was:

  1 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

review: Approve
Zorba Build Bot (zorba-buildbot) wrote :

There are additional revisions which have not been approved in review. Please seek review and approval of these new revisions.

review: Approve

Unmerged revisions

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'ChangeLog'
2--- ChangeLog 2011-12-23 19:38:53 +0000
3+++ ChangeLog 2011-12-23 22:04:28 +0000
4@@ -12,6 +12,8 @@
5 set multiple times via the c++ api).
6 * Fixed bug #905050 (setting and getting the context item type via the c++ api)
7 * Added createDayTimeDuration, createYearMonthDuration, createDocumentNode, createCommentNode, createPiNode to api's ItemFactory.
8+ * Added split function to the string module that allows for streamable tokenization but doesn't have regular expression
9+ support.
10 * zerr is not predeclared anymore to be http://www.zorba-xquery.com/errors
11
12 version 2.1
13
14=== modified file 'modules/com/zorba-xquery/www/modules/CMakeLists.txt'
15--- modules/com/zorba-xquery/www/modules/CMakeLists.txt 2011-12-21 14:40:33 +0000
16+++ modules/com/zorba-xquery/www/modules/CMakeLists.txt 2011-12-23 22:04:28 +0000
17@@ -58,7 +58,7 @@
18 URI "http://www.zorba-xquery.com/modules/reflection")
19 DECLARE_ZORBA_MODULE(FILE schema.xq VERSION 2.0
20 URI "http://www.zorba-xquery.com/modules/schema")
21-DECLARE_ZORBA_MODULE(FILE string.xq VERSION 2.0
22+DECLARE_ZORBA_MODULE(FILE string.xq VERSION 2.1
23 URI "http://www.zorba-xquery.com/modules/string")
24 DECLARE_ZORBA_MODULE(FILE xml.xq VERSION 2.0
25 URI "http://www.zorba-xquery.com/modules/xml")
26
27=== modified file 'modules/com/zorba-xquery/www/modules/string.xq'
28--- modules/com/zorba-xquery/www/modules/string.xq 2011-08-03 15:12:40 +0000
29+++ modules/com/zorba-xquery/www/modules/string.xq 2011-12-23 22:04:28 +0000
30@@ -25,7 +25,7 @@
31 :)
32 module namespace string = "http://www.zorba-xquery.com/modules/string";
33 declare namespace ver = "http://www.zorba-xquery.com/options/versioning";
34-declare option ver:module-version "2.0";
35+declare option ver:module-version "2.1";
36
37 (:~
38 : This function materializes a streamable string.
39@@ -63,3 +63,23 @@
40 :
41 :)
42 declare function string:is-streamable($s as xs:string) as xs:boolean external;
43+
44+(:~
45+ : Returns a sequence of strings constructed by splitting the input wherever the given
46+ : separator is found.
47+ :
48+ : The function is different from fn:tokenize. It doesn't allow
49+ : the separator to be a regular expression. This restriction allows for more
50+ : performant implementation. Specifically, the function processes
51+ : streamable strings as input in a streamable way which is particularly useful
52+ : to tokenize huge strings (e.g. if returned by the file module's read-text
53+ : function).
54+ :
55+ : @param $s the input string to split
56+ : @param $separator the separator used for splitting the input string $s
57+ :
58+ : @return a sequence of strings constructed by splitting the input
59+ :)
60+declare function string:split(
61+ $s as xs:string,
62+ $separator as xs:string) as xs:string* external;
63
64=== modified file 'src/functions/pregenerated/func_strings.cpp'
65--- src/functions/pregenerated/func_strings.cpp 2011-12-21 14:40:33 +0000
66+++ src/functions/pregenerated/func_strings.cpp 2011-12-23 22:04:28 +0000
67@@ -320,6 +320,16 @@
68 return new StringIsStreamableIterator(sctx, loc, argv);
69 }
70
71+PlanIter_t fn_zorba_string_split::codegen(
72+ CompilerCB*,
73+ static_context* sctx,
74+ const QueryLoc& loc,
75+ std::vector<PlanIter_t>& argv,
76+ AnnotationHolder& ann) const
77+{
78+ return new StringSplitIterator(sctx, loc, argv);
79+}
80+
81 void populate_context_strings(static_context* sctx)
82 {
83 {
84@@ -890,6 +900,19 @@
85
86 }
87
88+
89+ {
90+
91+
92+ DECL_WITH_KIND(sctx, fn_zorba_string_split,
93+ (createQName("http://www.zorba-xquery.com/modules/string","","split"),
94+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
95+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
96+ GENV_TYPESYSTEM.STRING_TYPE_STAR),
97+ FunctionConsts::FN_ZORBA_STRING_SPLIT_2);
98+
99+ }
100+
101 }
102
103
104
105=== modified file 'src/functions/pregenerated/func_strings.h'
106--- src/functions/pregenerated/func_strings.h 2011-12-22 14:14:53 +0000
107+++ src/functions/pregenerated/func_strings.h 2011-12-23 22:04:28 +0000
108@@ -481,6 +481,21 @@
109 };
110
111
112+//fn-zorba-string:split
113+class fn_zorba_string_split : public function
114+{
115+public:
116+ fn_zorba_string_split(const signature& sig, FunctionConsts::FunctionKind kind)
117+ :
118+ function(sig, kind)
119+ {
120+
121+ }
122+
123+ CODEGEN_DECL();
124+};
125+
126+
127 } //namespace zorba
128
129
130
131=== modified file 'src/functions/pregenerated/function_enum.h'
132--- src/functions/pregenerated/function_enum.h 2011-12-21 14:40:33 +0000
133+++ src/functions/pregenerated/function_enum.h 2011-12-23 22:04:28 +0000
134@@ -371,6 +371,7 @@
135 FN_ANALYZE_STRING_3,
136 FN_ZORBA_STRING_MATERIALIZE_1,
137 FN_ZORBA_STRING_IS_STREAMABLE_1,
138+ FN_ZORBA_STRING_SPLIT_2,
139 FN_ZORBA_XQDOC_XQDOC_1,
140 FN_ZORBA_XQDOC_XQDOC_CONTENT_1,
141
142
143=== modified file 'src/runtime/spec/strings/strings.xml'
144--- src/runtime/spec/strings/strings.xml 2011-12-21 14:40:33 +0000
145+++ src/runtime/spec/strings/strings.xml 2011-12-23 22:04:28 +0000
146@@ -729,4 +729,35 @@
147
148 </zorba:iterator>
149
150+<!--
151+/*******************************************************************************
152+ * string:tokenize
153+********************************************************************************/
154+-->
155+<zorba:iterator name="StringSplitIterator">
156+
157+ <zorba:description author="Matthias Brantner">
158+ string:split
159+ </zorba:description>
160+
161+ <zorba:function>
162+ <zorba:signature localname="split" prefix="fn-zorba-string">
163+ <zorba:param>xs:string</zorba:param>
164+ <zorba:param>xs:string</zorba:param>
165+ <zorba:output>xs:string*</zorba:output>
166+ </zorba:signature>
167+ </zorba:function>
168+
169+ <zorba:state>
170+ <zorba:member type="zstring" name="theSeparator"
171+ brief="separator for the tokenization"/>
172+ <zorba:member type="std::istream*" name="theIStream"
173+ brief="the remaining string (if the input is streamable)"/>
174+ <zorba:member type="zstring" name="theInput"
175+ brief="the string to tokenize (if the input is not streamable)"/>
176+ <zorba:member type="size_t" name="theNextStartPos" defaultValue="0"/>
177+ </zorba:state>
178+
179+</zorba:iterator>
180+
181 </zorba:iterators>
182
183=== modified file 'src/runtime/strings/pregenerated/strings.cpp'
184--- src/runtime/strings/pregenerated/strings.cpp 2011-12-21 14:40:33 +0000
185+++ src/runtime/strings/pregenerated/strings.cpp 2011-12-23 22:04:28 +0000
186@@ -830,6 +830,48 @@
187 // </StringIsStreamableIterator>
188
189
190+// <StringSplitIterator>
191+const char* StringSplitIterator::class_name_str = "StringSplitIterator";
192+StringSplitIterator::class_factory<StringSplitIterator>
193+StringSplitIterator::g_class_factory;
194+
195+const serialization::ClassVersion
196+StringSplitIterator::class_versions[] ={{ 1, 0x000905, false}};
197+
198+const int StringSplitIterator::class_versions_count =
199+sizeof(StringSplitIterator::class_versions)/sizeof(struct serialization::ClassVersion);
200+
201+void StringSplitIterator::accept(PlanIterVisitor& v) const {
202+ v.beginVisit(*this);
203+
204+ std::vector<PlanIter_t>::const_iterator lIter = theChildren.begin();
205+ std::vector<PlanIter_t>::const_iterator lEnd = theChildren.end();
206+ for ( ; lIter != lEnd; ++lIter ){
207+ (*lIter)->accept(v);
208+ }
209+
210+ v.endVisit(*this);
211+}
212+
213+StringSplitIterator::~StringSplitIterator() {}
214+
215+StringSplitIteratorState::StringSplitIteratorState() {}
216+
217+StringSplitIteratorState::~StringSplitIteratorState() {}
218+
219+
220+void StringSplitIteratorState::init(PlanState& planState) {
221+ PlanIteratorState::init(planState);
222+ theNextStartPos = 0;
223+}
224+
225+void StringSplitIteratorState::reset(PlanState& planState) {
226+ PlanIteratorState::reset(planState);
227+ theNextStartPos = 0;
228+}
229+// </StringSplitIterator>
230+
231+
232
233 }
234
235
236=== modified file 'src/runtime/strings/pregenerated/strings.h'
237--- src/runtime/strings/pregenerated/strings.h 2011-12-21 14:40:33 +0000
238+++ src/runtime/strings/pregenerated/strings.h 2011-12-23 22:04:28 +0000
239@@ -1075,6 +1075,58 @@
240 };
241
242
243+/**
244+ *
245+ * string:split
246+ *
247+ * Author: Matthias Brantner
248+ */
249+class StringSplitIteratorState : public PlanIteratorState
250+{
251+public:
252+ zstring theSeparator; //separator for the tokenization
253+ std::istream* theIStream; //the remaining string (if the input is streamable)
254+ zstring theInput; //the string to tokenize (if the input is not streamable)
255+ size_t theNextStartPos; //
256+
257+ StringSplitIteratorState();
258+
259+ ~StringSplitIteratorState();
260+
261+ void init(PlanState&);
262+ void reset(PlanState&);
263+};
264+
265+class StringSplitIterator : public NaryBaseIterator<StringSplitIterator, StringSplitIteratorState>
266+{
267+public:
268+ SERIALIZABLE_CLASS(StringSplitIterator);
269+
270+ SERIALIZABLE_CLASS_CONSTRUCTOR2T(StringSplitIterator,
271+ NaryBaseIterator<StringSplitIterator, StringSplitIteratorState>);
272+
273+ void serialize( ::zorba::serialization::Archiver& ar)
274+ {
275+ serialize_baseclass(ar,
276+ (NaryBaseIterator<StringSplitIterator, StringSplitIteratorState>*)this);
277+ }
278+
279+ StringSplitIterator(
280+ static_context* sctx,
281+ const QueryLoc& loc,
282+ std::vector<PlanIter_t>& children)
283+ :
284+ NaryBaseIterator<StringSplitIterator, StringSplitIteratorState>(sctx, loc, children)
285+ {}
286+
287+ virtual ~StringSplitIterator();
288+
289+ void accept(PlanIterVisitor& v) const;
290+
291+ bool nextImpl(store::Item_t& result, PlanState& aPlanState) const;
292+};
293+
294+
295 }
296 #endif
297 /*
298
299=== modified file 'src/runtime/strings/strings_impl.cpp'
300--- src/runtime/strings/strings_impl.cpp 2011-12-23 06:41:43 +0000
301+++ src/runtime/strings/strings_impl.cpp 2011-12-23 22:04:28 +0000
302@@ -140,6 +140,7 @@
303 p = ec;
304
305 if ( utf8::read( *state->theStream, ec ) == utf8::npos )
306+ {
307 if ( state->theStream->good() ) {
308 //
309 // If read() failed but the stream state is good, it means that an
310@@ -165,6 +166,7 @@
311 zerr::ZOSE0003_STREAM_READ_FAILURE, ERROR_LOC( loc )
312 );
313 }
314+ }
315 state->theResult.clear();
316 state->theResult.push_back( utf8::next_char( p ) );
317
318@@ -2284,5 +2286,133 @@
319 STACK_END(state);
320 }
321
322+/**
323+ *______________________________________________________________________
324+ *
325+ * http://www.zorba-xquery.com/modules/string
326+ * string:split
327+ */
328+bool StringSplitIterator::nextImpl(
329+ store::Item_t& result,
330+ PlanState& planState) const
331+{
332+ store::Item_t item;
333+ size_t lNewPos = 0;
334+ zstring lToken;
335+ zstring lPartialMatch;
336+
337+ StringSplitIteratorState* state;
338+ DEFAULT_STACK_INIT(StringSplitIteratorState, state, planState);
339+
340+ // init phase, get input string and tokens
341+ consumeNext(item, theChildren[0].getp(), planState);
342+
343+ if (item->isStreamable())
344+ {
345+ state->theIStream = &item->getStream();
346+ }
347+ else
348+ {
349+ state->theIStream = 0;
350+ item->getStringValue2(state->theInput);
351+ }
352+
353+ consumeNext(item, theChildren[1].getp(), planState);
354+
355+ item->getStringValue2(state->theSeparator);
356+
357+ // working phase, do the tokenization
358+ if (state->theIStream)
359+ {
360+ while ( !state->theIStream->eof() )
361+ {
362+ utf8::encoded_char_type ec;
363+ memset( ec, '\0' , sizeof(ec) );
364+ utf8::storage_type *p;
365+ p = ec;
366+
367+ if ( utf8::read( *state->theIStream, ec ) != utf8::npos )
368+ {
369+ if (state->theSeparator.compare(lNewPos, 1, ec) == 0)
370+ {
371+ if (++lNewPos == state->theSeparator.length())
372+ {
373+ GENV_ITEMFACTORY->createString(result, lToken);
374+ STACK_PUSH(true, state);
375+ }
376+ else
377+ {
378+ lPartialMatch.append(ec);
379+ }
380+ }
381+ else
382+ {
383+ lToken.append(lPartialMatch);
384+ lToken.append(ec);
385+ }
386+ }
387+ else
388+ {
389+ if (state->theIStream->good())
390+ {
391+ char buf[ 6 /* bytes at most */ * 5 /* chars per byte */ ], *b = buf;
392+ bool first = true;
393+ for ( ; *p; ++p ) {
394+ if ( first )
395+ first = false;
396+ else
397+ *b++ = ',';
398+ ::strcpy( b, "0x" ); b += 2;
399+ ::sprintf( b, "%0hhX", *p ); b += 2;
400+ }
401+ throw XQUERY_EXCEPTION(
402+ zerr::ZXQD0006_INVALID_UTF8_BYTE_SEQUENCE,
403+ ERROR_PARAMS( buf ),
404+ ERROR_LOC( loc )
405+ );
406+ }
407+ if (!lToken.empty())
408+ {
409+ GENV_ITEMFACTORY->createString(result, lToken);
410+ STACK_PUSH(true, state);
411+ }
412+ break;
413+ }
414+ }
415+ }
416+ else
417+ {
418+ while (true)
419+ {
420+ if (state->theNextStartPos == zstring::npos)
421+ {
422+ break;
423+ }
424+
425+ lNewPos =
426+ state->theInput.find(state->theSeparator, state->theNextStartPos);
427+ if (lNewPos != zstring::npos)
428+ {
429+ zstring lSubStr = state->theInput.substr(
430+ state->theNextStartPos,
431+ lNewPos - state->theNextStartPos);
432+ GENV_ITEMFACTORY->createString(result, lSubStr);
433+ state->theNextStartPos =
434+ lNewPos==state->theInput.length() - state->theSeparator.length()
435+ ? zstring::npos
436+ : lNewPos + state->theSeparator.length();
437+ }
438+ else
439+ {
440+ zstring lSubStr = state->theInput.substr(state->theNextStartPos);
441+ GENV_ITEMFACTORY->createString(result, lSubStr);
442+ state->theNextStartPos = zstring::npos;
443+ }
444+ STACK_PUSH(true, state);
445+ }
446+ }
447+
448+ STACK_END(state);
449+}
450 } // namespace zorba
451 /* vim:set et sw=2 ts=2: */
452
453=== modified file 'src/runtime/visitors/pregenerated/planiter_visitor.h'
454--- src/runtime/visitors/pregenerated/planiter_visitor.h 2011-12-21 14:40:33 +0000
455+++ src/runtime/visitors/pregenerated/planiter_visitor.h 2011-12-23 22:04:28 +0000
456@@ -582,6 +582,8 @@
457
458 class StringIsStreamableIterator;
459
460+ class StringSplitIterator;
461+
462 class XQDocIterator;
463
464 class XQDocContentIterator;
465@@ -1423,6 +1425,9 @@
466 virtual void beginVisit ( const StringIsStreamableIterator& ) = 0;
467 virtual void endVisit ( const StringIsStreamableIterator& ) = 0;
468
469+ virtual void beginVisit ( const StringSplitIterator& ) = 0;
470+ virtual void endVisit ( const StringSplitIterator& ) = 0;
471+
472 virtual void beginVisit ( const XQDocIterator& ) = 0;
473 virtual void endVisit ( const XQDocIterator& ) = 0;
474
475
476=== modified file 'src/runtime/visitors/pregenerated/printer_visitor.cpp'
477--- src/runtime/visitors/pregenerated/printer_visitor.cpp 2011-12-21 14:40:33 +0000
478+++ src/runtime/visitors/pregenerated/printer_visitor.cpp 2011-12-23 22:04:28 +0000
479@@ -3961,6 +3961,20 @@
480 // </StringIsStreamableIterator>
481
482
483+// <StringSplitIterator>
484+void PrinterVisitor::beginVisit ( const StringSplitIterator& a) {
485+ thePrinter.startBeginVisit("StringSplitIterator", ++theId);
486+ printCommons( &a, theId );
487+ thePrinter.endBeginVisit( theId );
488+}
489+
490+void PrinterVisitor::endVisit ( const StringSplitIterator& ) {
491+ thePrinter.startEndVisit();
492+ thePrinter.endEndVisit();
493+}
494+// </StringSplitIterator>
495+
496+
497 // <XQDocIterator>
498 void PrinterVisitor::beginVisit ( const XQDocIterator& a) {
499 thePrinter.startBeginVisit("XQDocIterator", ++theId);
500
501=== modified file 'src/runtime/visitors/pregenerated/printer_visitor.h'
502--- src/runtime/visitors/pregenerated/printer_visitor.h 2011-12-21 14:40:33 +0000
503+++ src/runtime/visitors/pregenerated/printer_visitor.h 2011-12-23 22:04:28 +0000
504@@ -876,6 +876,9 @@
505 void beginVisit( const StringIsStreamableIterator& );
506 void endVisit ( const StringIsStreamableIterator& );
507
508+ void beginVisit( const StringSplitIterator& );
509+ void endVisit ( const StringSplitIterator& );
510+
511 void beginVisit( const XQDocIterator& );
512 void endVisit ( const XQDocIterator& );
513
514
515=== added file 'test/rbkt/ExpQueryResults/zorba/string/tokenize01.xml.res'
516--- test/rbkt/ExpQueryResults/zorba/string/tokenize01.xml.res 1970-01-01 00:00:00 +0000
517+++ test/rbkt/ExpQueryResults/zorba/string/tokenize01.xml.res 2011-12-23 22:04:28 +0000
518@@ -0,0 +1,1 @@
519+a d a d
520
521=== added file 'test/rbkt/ExpQueryResults/zorba/string/tokenize02.xml.res'
522--- test/rbkt/ExpQueryResults/zorba/string/tokenize02.xml.res 1970-01-01 00:00:00 +0000
523+++ test/rbkt/ExpQueryResults/zorba/string/tokenize02.xml.res 2011-12-23 22:04:28 +0000
524@@ -0,0 +1,1 @@
525+a a
526
527=== added file 'test/rbkt/ExpQueryResults/zorba/string/tokenize03.xml.res'
528--- test/rbkt/ExpQueryResults/zorba/string/tokenize03.xml.res 1970-01-01 00:00:00 +0000
529+++ test/rbkt/ExpQueryResults/zorba/string/tokenize03.xml.res 2011-12-23 22:04:28 +0000
530@@ -0,0 +1,1 @@
531+ d d
532
533=== added file 'test/rbkt/ExpQueryResults/zorba/string/tokenize04.xml.res'
534--- test/rbkt/ExpQueryResults/zorba/string/tokenize04.xml.res 1970-01-01 00:00:00 +0000
535+++ test/rbkt/ExpQueryResults/zorba/string/tokenize04.xml.res 2011-12-23 22:04:28 +0000
536@@ -0,0 +1,1 @@
537+abcd abcd
538
539=== added file 'test/rbkt/Queries/zorba/string/token01.txt'
540--- test/rbkt/Queries/zorba/string/token01.txt 1970-01-01 00:00:00 +0000
541+++ test/rbkt/Queries/zorba/string/token01.txt 2011-12-23 22:04:28 +0000
542@@ -0,0 +1,1 @@
543+abcd
544\ No newline at end of file
545
546=== added file 'test/rbkt/Queries/zorba/string/token02.txt'
547--- test/rbkt/Queries/zorba/string/token02.txt 1970-01-01 00:00:00 +0000
548+++ test/rbkt/Queries/zorba/string/token02.txt 2011-12-23 22:04:28 +0000
549@@ -0,0 +1,1 @@
550+abc
551\ No newline at end of file
552
553=== added file 'test/rbkt/Queries/zorba/string/token03.txt'
554--- test/rbkt/Queries/zorba/string/token03.txt 1970-01-01 00:00:00 +0000
555+++ test/rbkt/Queries/zorba/string/token03.txt 2011-12-23 22:04:28 +0000
556@@ -0,0 +1,1 @@
557+bcd
558\ No newline at end of file
559
560=== added file 'test/rbkt/Queries/zorba/string/token04.txt'
561--- test/rbkt/Queries/zorba/string/token04.txt 1970-01-01 00:00:00 +0000
562+++ test/rbkt/Queries/zorba/string/token04.txt 2011-12-23 22:04:28 +0000
563@@ -0,0 +1,1 @@
564+abcd
565\ No newline at end of file
566
567=== added file 'test/rbkt/Queries/zorba/string/tokenize01.xq'
568--- test/rbkt/Queries/zorba/string/tokenize01.xq 1970-01-01 00:00:00 +0000
569+++ test/rbkt/Queries/zorba/string/tokenize01.xq 2011-12-23 22:04:28 +0000
570@@ -0,0 +1,5 @@
571+import module namespace f = "http://expath.org/ns/file";
572+import module namespace s = "http://www.zorba-xquery.com/modules/string";
573+
574+s:split(f:read-text(fn:resolve-uri("token01.txt")), "bc"),
575+s:split(s:materialize(f:read-text(fn:resolve-uri("token01.txt"))), "bc")
576
577=== added file 'test/rbkt/Queries/zorba/string/tokenize02.xq'
578--- test/rbkt/Queries/zorba/string/tokenize02.xq 1970-01-01 00:00:00 +0000
579+++ test/rbkt/Queries/zorba/string/tokenize02.xq 2011-12-23 22:04:28 +0000
580@@ -0,0 +1,5 @@
581+import module namespace f = "http://expath.org/ns/file";
582+import module namespace s = "http://www.zorba-xquery.com/modules/string";
583+
584+s:split(f:read-text(fn:resolve-uri("token02.txt")), "bc"),
585+s:split(s:materialize(f:read-text(fn:resolve-uri("token02.txt"))), "bc")
586
587=== added file 'test/rbkt/Queries/zorba/string/tokenize03.xq'
588--- test/rbkt/Queries/zorba/string/tokenize03.xq 1970-01-01 00:00:00 +0000
589+++ test/rbkt/Queries/zorba/string/tokenize03.xq 2011-12-23 22:04:28 +0000
590@@ -0,0 +1,5 @@
591+import module namespace f = "http://expath.org/ns/file";
592+import module namespace s = "http://www.zorba-xquery.com/modules/string";
593+
594+s:split(f:read-text(fn:resolve-uri("token03.txt")), "bc"),
595+s:split(s:materialize(f:read-text(fn:resolve-uri("token03.txt"))), "bc")
596
597=== added file 'test/rbkt/Queries/zorba/string/tokenize04.xq'
598--- test/rbkt/Queries/zorba/string/tokenize04.xq 1970-01-01 00:00:00 +0000
599+++ test/rbkt/Queries/zorba/string/tokenize04.xq 2011-12-23 22:04:28 +0000
600@@ -0,0 +1,5 @@
601+import module namespace f = "http://expath.org/ns/file";
602+import module namespace s = "http://www.zorba-xquery.com/modules/string";
603+
604+s:split(f:read-text(fn:resolve-uri("token04.txt")), "f"),
605+s:split(s:materialize(f:read-text(fn:resolve-uri("token04.txt"))), "f")

Subscribers

People subscribed via source and target branches