Merge lp:~zorba-coders/zorba/tokenize into lp:zorba

Proposed by Matthias Brantner on 2011-12-23
Status: Superseded
Proposed branch: lp:~zorba-coders/zorba/tokenize
Merge into: lp:zorba
Diff against target: 603 lines (+366/-2)
25 files modified
ChangeLog (+2/-0)
modules/com/zorba-xquery/www/modules/CMakeLists.txt (+1/-1)
modules/com/zorba-xquery/www/modules/string.xq (+21/-1)
src/functions/pregenerated/func_strings.cpp (+23/-0)
src/functions/pregenerated/func_strings.h (+13/-0)
src/functions/pregenerated/function_enum.h (+1/-0)
src/runtime/spec/strings/strings.xml (+31/-0)
src/runtime/strings/pregenerated/strings.cpp (+42/-0)
src/runtime/strings/pregenerated/strings.h (+52/-0)
src/runtime/strings/strings_impl.cpp (+130/-0)
src/runtime/visitors/pregenerated/planiter_visitor.h (+5/-0)
src/runtime/visitors/pregenerated/printer_visitor.cpp (+14/-0)
src/runtime/visitors/pregenerated/printer_visitor.h (+3/-0)
test/rbkt/ExpQueryResults/zorba/string/tokenize01.xml.res (+1/-0)
test/rbkt/ExpQueryResults/zorba/string/tokenize02.xml.res (+1/-0)
test/rbkt/ExpQueryResults/zorba/string/tokenize03.xml.res (+1/-0)
test/rbkt/ExpQueryResults/zorba/string/tokenize04.xml.res (+1/-0)
test/rbkt/Queries/zorba/string/token01.txt (+1/-0)
test/rbkt/Queries/zorba/string/token02.txt (+1/-0)
test/rbkt/Queries/zorba/string/token03.txt (+1/-0)
test/rbkt/Queries/zorba/string/token04.txt (+1/-0)
test/rbkt/Queries/zorba/string/tokenize01.xq (+5/-0)
test/rbkt/Queries/zorba/string/tokenize02.xq (+5/-0)
test/rbkt/Queries/zorba/string/tokenize03.xq (+5/-0)
test/rbkt/Queries/zorba/string/tokenize04.xq (+5/-0)
To merge this branch: bzr merge lp:~zorba-coders/zorba/tokenize
Reviewer Review Type Date Requested Status
William Candillon 2011-12-23 Approve on 2011-12-23
Matthias Brantner Approve on 2011-12-23
Paul J. Lucas 2011-12-23 Pending
Review via email: mp+86829@code.launchpad.net

This proposal supersedes a proposal from 2011-12-22.

This proposal has been superseded by a proposal from 2011-12-23.

Commit message

implementation of string:split function that doesn't accept regular expressions but allows for streamable processing of the input (resolves bug #898074)

Description of the change

implementation of string:split function that doesn't accept regular expressions but allows for streamable processing of the input (resolves bug #898074)

To post a comment you must log in.
Zorba Build Bot (zorba-buildbot) wrote : Posted in a previous version of this proposal
Zorba Build Bot (zorba-buildbot) wrote : Posted in a previous version of this proposal

Validation queue job tokenize-2011-12-21T21-46-05.289Z is finished. The final status was:

All tests succeeded!

Zorba Build Bot (zorba-buildbot) wrote : Posted in a previous version of this proposal

Voting does not meet specified criteria. Required: Approve > 1, Disapprove < 1. Got: 2 Pending.

Paul J. Lucas (paul-lucas) wrote : Posted in a previous version of this proposal

On line 357, why do you call assert()? An invalid byte should throw an exception, not assert and dump core.

Paul J. Lucas (paul-lucas) wrote : Posted in a previous version of this proposal

> On line 357, why do you call assert()? An invalid byte should throw an
> exception, not assert and dump core.

I meant line 367.

Paul J. Lucas (paul-lucas) : Posted in a previous version of this proposal
review: Needs Fixing
Matthias Brantner (matthias-brantner) wrote : Posted in a previous version of this proposal

Once you finished the implementation of the transcoding stream buffer, I don't even want to do this check anymore. This must not happen with the stream buffer.

Paul J. Lucas (paul-lucas) wrote : Posted in a previous version of this proposal

> Once you finished the implementation of the transcoding stream buffer, I don't
> even want to do this check anymore. This must not happen with the stream
> buffer.

I don't understand how it "must not happen." It can always happen. However, I think you're saying that you assume the check will happen in the transcoder. While it will be doing checks, bad input can still happen.

In the mean time, using an assert() is still too Draconian.

Matthias Brantner (matthias-brantner) wrote : Posted in a previous version of this proposal

I have replaced the assertion with a graceful error.

Paul J. Lucas (paul-lucas) : Posted in a previous version of this proposal
review: Approve
William Candillon (wcandillon) wrote : Posted in a previous version of this proposal

Is there an example that works with streaming?
I wasn't able to make the following work:
import module namespace http = "http://www.zorba-xquery.com/modules/http-client";

declare namespace h = "http://expath.org/ns/http-client";

let $item := http:send-request(<h:request href="https://stream.twitter.com/1/statuses/sample.json?delimited=length"
                                          method="GET"
                                          username="wcandillon"
                                          password="wnvbb86g"
                                          override-media-type="text/plain"
                               />
                               ,
                               "https://stream.twitter.com/1/statuses/sample.json?delimited=length",
                               ()
                                )[2]
for $tweet in tokenize($item,"a")
return $tweet

Where:
import module namespace http = "http://www.zorba-xquery.com/modules/http-client";

declare namespace h = "http://expath.org/ns/http-client";

let $item := http:send-request(<h:request href="https://stream.twitter.com/1/statuses/sample.json?delimited=length"
                                          method="GET"
                                          username="wcandillon"
                                          password="wnvbb86g"
                                          override-media-type="text/plain"
                               />
                               ,
                               "https://stream.twitter.com/1/statuses/sample.json?delimited=length",
                               ()
                                )[2]
return $item

streams fine.
What am I missing?

Matthias Brantner (matthias-brantner) wrote : Posted in a previous version of this proposal

As discussed in this thread, only the new tokenize function of the string module streams.
Use the following instead

import module namespace s = "http://www.zorba-xquery.com/modules/string";

s:tokenize($item, "a")

William Candillon (wcandillon) wrote : Posted in a previous version of this proposal

Works like a charm.

review: Approve
Zorba Build Bot (zorba-buildbot) wrote : Posted in a previous version of this proposal

Attempt to merge into lp:zorba failed due to conflicts:

text conflict in ChangeLog

review: Approve
William Candillon (wcandillon) wrote :

Works great.

review: Approve
Zorba Build Bot (zorba-buildbot) wrote :

The attempt to merge lp:~zorba-coders/zorba/tokenize into lp:zorba failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:273 (message):
  Validation queue job tokenize-2011-12-23T20-55-01.864Z is finished. The
  final status was:

  1 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

Zorba Build Bot (zorba-buildbot) wrote :

The attempt to merge lp:~zorba-coders/zorba/tokenize into lp:zorba failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:273 (message):
  Validation queue job tokenize-2011-12-23T21-25-57.422Z is finished. The
  final status was:

  1 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

lp:~zorba-coders/zorba/tokenize updated on 2011-12-23
10589. By Matthias Brantner on 2011-12-23

forgot to commit pregenerated file

Unmerged revisions

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'ChangeLog'
2--- ChangeLog 2011-12-23 19:38:53 +0000
3+++ ChangeLog 2011-12-23 20:33:38 +0000
4@@ -12,6 +12,8 @@
5 set multiple times via the c++ api).
6 * Fixed bug #905050 (setting and getting the context item type via the c++ api)
7 * Added createDayTimeDuration, createYearMonthDuration, createDocumentNode, createCommentNode, createPiNode to api's ItemFactory.
8+ * Added split function to the string module that allows for streamable tokenization but doesn't have regular expression
9+ support.
10 * zerr is not predeclared anymore to be http://www.zorba-xquery.com/errors
11
12 version 2.1
13
14=== modified file 'modules/com/zorba-xquery/www/modules/CMakeLists.txt'
15--- modules/com/zorba-xquery/www/modules/CMakeLists.txt 2011-12-21 14:40:33 +0000
16+++ modules/com/zorba-xquery/www/modules/CMakeLists.txt 2011-12-23 20:33:38 +0000
17@@ -58,7 +58,7 @@
18 URI "http://www.zorba-xquery.com/modules/reflection")
19 DECLARE_ZORBA_MODULE(FILE schema.xq VERSION 2.0
20 URI "http://www.zorba-xquery.com/modules/schema")
21-DECLARE_ZORBA_MODULE(FILE string.xq VERSION 2.0
22+DECLARE_ZORBA_MODULE(FILE string.xq VERSION 2.1
23 URI "http://www.zorba-xquery.com/modules/string")
24 DECLARE_ZORBA_MODULE(FILE xml.xq VERSION 2.0
25 URI "http://www.zorba-xquery.com/modules/xml")
26
27=== modified file 'modules/com/zorba-xquery/www/modules/string.xq'
28--- modules/com/zorba-xquery/www/modules/string.xq 2011-08-03 15:12:40 +0000
29+++ modules/com/zorba-xquery/www/modules/string.xq 2011-12-23 20:33:38 +0000
30@@ -25,7 +25,7 @@
31 :)
32 module namespace string = "http://www.zorba-xquery.com/modules/string";
33 declare namespace ver = "http://www.zorba-xquery.com/options/versioning";
34-declare option ver:module-version "2.0";
35+declare option ver:module-version "2.1";
36
37 (:~
38 : This function materializes a streamable string.
39@@ -63,3 +63,23 @@
40 :
41 :)
42 declare function string:is-streamable($s as xs:string) as xs:boolean external;
43+
44+(:~
45+ : Returns a sequence of strings constructed by splitting the input wherever the given
46+ : separator is found.
47+ :
48+ : The function is different from fn:tokenize. It doesn't allow
49+ : the separator to be a regular expression. This restriction allows for more
50+ : performant implementation. Specifically, the function processes
51+ : streamable strings as input in a streamable way which is particularly useful
52+ : to tokenize huge strings (e.g. if returned by the file module's read-text
53+ : function).
54+ :
55+ : @param $s the input string to split
56+ : @param $separator the separator used for splitting the input string $s
57+ :
58+ : @return a sequence of strings constructed by splitting the input
59+ :)
60+declare function string:split(
61+ $s as xs:string,
62+ $separator as xs:string) as xs:string* external;
63
64=== modified file 'src/functions/pregenerated/func_strings.cpp'
65--- src/functions/pregenerated/func_strings.cpp 2011-12-21 14:40:33 +0000
66+++ src/functions/pregenerated/func_strings.cpp 2011-12-23 20:33:38 +0000
67@@ -320,6 +320,16 @@
68 return new StringIsStreamableIterator(sctx, loc, argv);
69 }
70
71+PlanIter_t fn_zorba_string_split::codegen(
72+ CompilerCB*,
73+ static_context* sctx,
74+ const QueryLoc& loc,
75+ std::vector<PlanIter_t>& argv,
76+ AnnotationHolder& ann) const
77+{
78+ return new StringSplitIterator(sctx, loc, argv);
79+}
80+
81 void populate_context_strings(static_context* sctx)
82 {
83 {
84@@ -890,6 +900,19 @@
85
86 }
87
88+
89+ {
90+
91+
92+ DECL_WITH_KIND(sctx, fn_zorba_string_split,
93+ (createQName("http://www.zorba-xquery.com/modules/string","","split"),
94+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
95+ GENV_TYPESYSTEM.STRING_TYPE_ONE,
96+ GENV_TYPESYSTEM.STRING_TYPE_STAR),
97+ FunctionConsts::FN_ZORBA_STRING_SPLIT_2);
98+
99+ }
100+
101 }
102
103
104
105=== modified file 'src/functions/pregenerated/func_strings.h'
106--- src/functions/pregenerated/func_strings.h 2011-12-22 14:14:53 +0000
107+++ src/functions/pregenerated/func_strings.h 2011-12-23 20:33:38 +0000
108@@ -481,6 +481,19 @@
109 };
110
111
112+//fn-zorba-string:split
113+class fn_zorba_string_split : public function
114+{
115+public:
116+ fn_zorba_string_split(const signature& sig, FunctionConsts::FunctionKind kind)
117+ : function(sig, kind) {
118+
119+}
120+
121+ CODEGEN_DECL();
122+};
123+
124+
125 } //namespace zorba
126
127
128
129=== modified file 'src/functions/pregenerated/function_enum.h'
130--- src/functions/pregenerated/function_enum.h 2011-12-21 14:40:33 +0000
131+++ src/functions/pregenerated/function_enum.h 2011-12-23 20:33:38 +0000
132@@ -371,6 +371,7 @@
133 FN_ANALYZE_STRING_3,
134 FN_ZORBA_STRING_MATERIALIZE_1,
135 FN_ZORBA_STRING_IS_STREAMABLE_1,
136+ FN_ZORBA_STRING_SPLIT_2,
137 FN_ZORBA_XQDOC_XQDOC_1,
138 FN_ZORBA_XQDOC_XQDOC_CONTENT_1,
139
140
141=== modified file 'src/runtime/spec/strings/strings.xml'
142--- src/runtime/spec/strings/strings.xml 2011-12-21 14:40:33 +0000
143+++ src/runtime/spec/strings/strings.xml 2011-12-23 20:33:38 +0000
144@@ -729,4 +729,35 @@
145
146 </zorba:iterator>
147
148+<!--
149+/*******************************************************************************
150+ * string:tokenize
151+********************************************************************************/
152+-->
153+<zorba:iterator name="StringSplitIterator">
154+
155+ <zorba:description author="Matthias Brantner">
156+ string:split
157+ </zorba:description>
158+
159+ <zorba:function>
160+ <zorba:signature localname="split" prefix="fn-zorba-string">
161+ <zorba:param>xs:string</zorba:param>
162+ <zorba:param>xs:string</zorba:param>
163+ <zorba:output>xs:string*</zorba:output>
164+ </zorba:signature>
165+ </zorba:function>
166+
167+ <zorba:state>
168+ <zorba:member type="zstring" name="theSeparator"
169+ brief="separator for the tokenization"/>
170+ <zorba:member type="std::istream*" name="theIStream"
171+ brief="the remaining string (if the input is streamable)"/>
172+ <zorba:member type="zstring" name="theInput"
173+ brief="the string to tokenize (if the input is not streamable)"/>
174+ <zorba:member type="size_t" name="theNextStartPos" defaultValue="0"/>
175+ </zorba:state>
176+
177+</zorba:iterator>
178+
179 </zorba:iterators>
180
181=== modified file 'src/runtime/strings/pregenerated/strings.cpp'
182--- src/runtime/strings/pregenerated/strings.cpp 2011-12-21 14:40:33 +0000
183+++ src/runtime/strings/pregenerated/strings.cpp 2011-12-23 20:33:38 +0000
184@@ -830,6 +830,48 @@
185 // </StringIsStreamableIterator>
186
187
188+// <StringSplitIterator>
189+const char* StringSplitIterator::class_name_str = "StringSplitIterator";
190+StringSplitIterator::class_factory<StringSplitIterator>
191+StringSplitIterator::g_class_factory;
192+
193+const serialization::ClassVersion
194+StringSplitIterator::class_versions[] ={{ 1, 0x000905, false}};
195+
196+const int StringSplitIterator::class_versions_count =
197+sizeof(StringSplitIterator::class_versions)/sizeof(struct serialization::ClassVersion);
198+
199+void StringSplitIterator::accept(PlanIterVisitor& v) const {
200+ v.beginVisit(*this);
201+
202+ std::vector<PlanIter_t>::const_iterator lIter = theChildren.begin();
203+ std::vector<PlanIter_t>::const_iterator lEnd = theChildren.end();
204+ for ( ; lIter != lEnd; ++lIter ){
205+ (*lIter)->accept(v);
206+ }
207+
208+ v.endVisit(*this);
209+}
210+
211+StringSplitIterator::~StringSplitIterator() {}
212+
213+StringSplitIteratorState::StringSplitIteratorState() {}
214+
215+StringSplitIteratorState::~StringSplitIteratorState() {}
216+
217+
218+void StringSplitIteratorState::init(PlanState& planState) {
219+ PlanIteratorState::init(planState);
220+ theNextStartPos = 0;
221+}
222+
223+void StringSplitIteratorState::reset(PlanState& planState) {
224+ PlanIteratorState::reset(planState);
225+ theNextStartPos = 0;
226+}
227+// </StringSplitIterator>
228+
229+
230
231 }
232
233
234=== modified file 'src/runtime/strings/pregenerated/strings.h'
235--- src/runtime/strings/pregenerated/strings.h 2011-12-21 14:40:33 +0000
236+++ src/runtime/strings/pregenerated/strings.h 2011-12-23 20:33:38 +0000
237@@ -1075,6 +1075,58 @@
238 };
239
240
241+/**
242+ *
243+ * string:split
244+ *
245+ * Author: Matthias Brantner
246+ */
247+class StringSplitIteratorState : public PlanIteratorState
248+{
249+public:
250+ zstring theSeparator; //separator for the tokenization
251+ std::istream* theIStream; //the remaining string (if the input is streamable)
252+ zstring theInput; //the string to tokenize (if the input is not streamable)
253+ size_t theNextStartPos; //
254+
255+ StringSplitIteratorState();
256+
257+ ~StringSplitIteratorState();
258+
259+ void init(PlanState&);
260+ void reset(PlanState&);
261+};
262+
263+class StringSplitIterator : public NaryBaseIterator<StringSplitIterator, StringSplitIteratorState>
264+{
265+public:
266+ SERIALIZABLE_CLASS(StringSplitIterator);
267+
268+ SERIALIZABLE_CLASS_CONSTRUCTOR2T(StringSplitIterator,
269+ NaryBaseIterator<StringSplitIterator, StringSplitIteratorState>);
270+
271+ void serialize( ::zorba::serialization::Archiver& ar)
272+ {
273+ serialize_baseclass(ar,
274+ (NaryBaseIterator<StringSplitIterator, StringSplitIteratorState>*)this);
275+ }
276+
277+ StringSplitIterator(
278+ static_context* sctx,
279+ const QueryLoc& loc,
280+ std::vector<PlanIter_t>& children)
281+ :
282+ NaryBaseIterator<StringSplitIterator, StringSplitIteratorState>(sctx, loc, children)
283+ {}
284+
285+ virtual ~StringSplitIterator();
286+
287+ void accept(PlanIterVisitor& v) const;
288+
289+ bool nextImpl(store::Item_t& result, PlanState& aPlanState) const;
290+};
291+
292+
293 }
294 #endif
295 /*
296
297=== modified file 'src/runtime/strings/strings_impl.cpp'
298--- src/runtime/strings/strings_impl.cpp 2011-12-23 06:41:43 +0000
299+++ src/runtime/strings/strings_impl.cpp 2011-12-23 20:33:38 +0000
300@@ -140,6 +140,7 @@
301 p = ec;
302
303 if ( utf8::read( *state->theStream, ec ) == utf8::npos )
304+ {
305 if ( state->theStream->good() ) {
306 //
307 // If read() failed but the stream state is good, it means that an
308@@ -165,6 +166,7 @@
309 zerr::ZOSE0003_STREAM_READ_FAILURE, ERROR_LOC( loc )
310 );
311 }
312+ }
313 state->theResult.clear();
314 state->theResult.push_back( utf8::next_char( p ) );
315
316@@ -2284,5 +2286,133 @@
317 STACK_END(state);
318 }
319
320+/**
321+ *______________________________________________________________________
322+ *
323+ * http://www.zorba-xquery.com/modules/string
324+ * string:split
325+ */
326+bool StringSplitIterator::nextImpl(
327+ store::Item_t& result,
328+ PlanState& planState) const
329+{
330+ store::Item_t item;
331+ size_t lNewPos = 0;
332+ zstring lToken;
333+ zstring lPartialMatch;
334+
335+ StringSplitIteratorState* state;
336+ DEFAULT_STACK_INIT(StringSplitIteratorState, state, planState);
337+
338+ // init phase, get input string and tokens
339+ consumeNext(item, theChildren[0].getp(), planState);
340+
341+ if (item->isStreamable())
342+ {
343+ state->theIStream = &item->getStream();
344+ }
345+ else
346+ {
347+ state->theIStream = 0;
348+ item->getStringValue2(state->theInput);
349+ }
350+
351+ consumeNext(item, theChildren[1].getp(), planState);
352+
353+ item->getStringValue2(state->theSeparator);
354+
355+ // working phase, do the tokenization
356+ if (state->theIStream)
357+ {
358+ while ( !state->theIStream->eof() )
359+ {
360+ utf8::encoded_char_type ec;
361+ memset( ec, '\0' , sizeof(ec) );
362+ utf8::storage_type *p;
363+ p = ec;
364+
365+ if ( utf8::read( *state->theIStream, ec ) != utf8::npos )
366+ {
367+ if (state->theSeparator.compare(lNewPos, 1, ec) == 0)
368+ {
369+ if (++lNewPos == state->theSeparator.length())
370+ {
371+ GENV_ITEMFACTORY->createString(result, lToken);
372+ STACK_PUSH(true, state);
373+ }
374+ else
375+ {
376+ lPartialMatch.append(ec);
377+ }
378+ }
379+ else
380+ {
381+ lToken.append(lPartialMatch);
382+ lToken.append(ec);
383+ }
384+ }
385+ else
386+ {
387+ if (state->theIStream->good())
388+ {
389+ char buf[ 6 /* bytes at most */ * 5 /* chars per byte */ ], *b = buf;
390+ bool first = true;
391+ for ( ; *p; ++p ) {
392+ if ( first )
393+ first = false;
394+ else
395+ *b++ = ',';
396+ ::strcpy( b, "0x" ); b += 2;
397+ ::sprintf( b, "%0hhX", *p ); b += 2;
398+ }
399+ throw XQUERY_EXCEPTION(
400+ zerr::ZXQD0006_INVALID_UTF8_BYTE_SEQUENCE,
401+ ERROR_PARAMS( buf ),
402+ ERROR_LOC( loc )
403+ );
404+ }
405+ if (!lToken.empty())
406+ {
407+ GENV_ITEMFACTORY->createString(result, lToken);
408+ STACK_PUSH(true, state);
409+ }
410+ break;
411+ }
412+ }
413+ }
414+ else
415+ {
416+ while (true)
417+ {
418+ if (state->theNextStartPos == zstring::npos)
419+ {
420+ break;
421+ }
422+
423+ lNewPos =
424+ state->theInput.find(state->theSeparator, state->theNextStartPos);
425+ if (lNewPos != zstring::npos)
426+ {
427+ zstring lSubStr = state->theInput.substr(
428+ state->theNextStartPos,
429+ lNewPos - state->theNextStartPos);
430+ GENV_ITEMFACTORY->createString(result, lSubStr);
431+ state->theNextStartPos =
432+ lNewPos==state->theInput.length() - state->theSeparator.length()
433+ ? zstring::npos
434+ : lNewPos + state->theSeparator.length();
435+ }
436+ else
437+ {
438+ zstring lSubStr = state->theInput.substr(state->theNextStartPos);
439+ GENV_ITEMFACTORY->createString(result, lSubStr);
440+ state->theNextStartPos = zstring::npos;
441+ }
442+ STACK_PUSH(true, state);
443+ }
444+ }
445+
446+ STACK_END(state);
447+}
448 } // namespace zorba
449 /* vim:set et sw=2 ts=2: */
450
451=== modified file 'src/runtime/visitors/pregenerated/planiter_visitor.h'
452--- src/runtime/visitors/pregenerated/planiter_visitor.h 2011-12-21 14:40:33 +0000
453+++ src/runtime/visitors/pregenerated/planiter_visitor.h 2011-12-23 20:33:38 +0000
454@@ -582,6 +582,8 @@
455
456 class StringIsStreamableIterator;
457
458+ class StringSplitIterator;
459+
460 class XQDocIterator;
461
462 class XQDocContentIterator;
463@@ -1423,6 +1425,9 @@
464 virtual void beginVisit ( const StringIsStreamableIterator& ) = 0;
465 virtual void endVisit ( const StringIsStreamableIterator& ) = 0;
466
467+ virtual void beginVisit ( const StringSplitIterator& ) = 0;
468+ virtual void endVisit ( const StringSplitIterator& ) = 0;
469+
470 virtual void beginVisit ( const XQDocIterator& ) = 0;
471 virtual void endVisit ( const XQDocIterator& ) = 0;
472
473
474=== modified file 'src/runtime/visitors/pregenerated/printer_visitor.cpp'
475--- src/runtime/visitors/pregenerated/printer_visitor.cpp 2011-12-21 14:40:33 +0000
476+++ src/runtime/visitors/pregenerated/printer_visitor.cpp 2011-12-23 20:33:38 +0000
477@@ -3961,6 +3961,20 @@
478 // </StringIsStreamableIterator>
479
480
481+// <StringSplitIterator>
482+void PrinterVisitor::beginVisit ( const StringSplitIterator& a) {
483+ thePrinter.startBeginVisit("StringSplitIterator", ++theId);
484+ printCommons( &a, theId );
485+ thePrinter.endBeginVisit( theId );
486+}
487+
488+void PrinterVisitor::endVisit ( const StringSplitIterator& ) {
489+ thePrinter.startEndVisit();
490+ thePrinter.endEndVisit();
491+}
492+// </StringSplitIterator>
493+
494+
495 // <XQDocIterator>
496 void PrinterVisitor::beginVisit ( const XQDocIterator& a) {
497 thePrinter.startBeginVisit("XQDocIterator", ++theId);
498
499=== modified file 'src/runtime/visitors/pregenerated/printer_visitor.h'
500--- src/runtime/visitors/pregenerated/printer_visitor.h 2011-12-21 14:40:33 +0000
501+++ src/runtime/visitors/pregenerated/printer_visitor.h 2011-12-23 20:33:38 +0000
502@@ -876,6 +876,9 @@
503 void beginVisit( const StringIsStreamableIterator& );
504 void endVisit ( const StringIsStreamableIterator& );
505
506+ void beginVisit( const StringSplitIterator& );
507+ void endVisit ( const StringSplitIterator& );
508+
509 void beginVisit( const XQDocIterator& );
510 void endVisit ( const XQDocIterator& );
511
512
513=== added file 'test/rbkt/ExpQueryResults/zorba/string/tokenize01.xml.res'
514--- test/rbkt/ExpQueryResults/zorba/string/tokenize01.xml.res 1970-01-01 00:00:00 +0000
515+++ test/rbkt/ExpQueryResults/zorba/string/tokenize01.xml.res 2011-12-23 20:33:38 +0000
516@@ -0,0 +1,1 @@
517+a d a d
518
519=== added file 'test/rbkt/ExpQueryResults/zorba/string/tokenize02.xml.res'
520--- test/rbkt/ExpQueryResults/zorba/string/tokenize02.xml.res 1970-01-01 00:00:00 +0000
521+++ test/rbkt/ExpQueryResults/zorba/string/tokenize02.xml.res 2011-12-23 20:33:38 +0000
522@@ -0,0 +1,1 @@
523+a a
524
525=== added file 'test/rbkt/ExpQueryResults/zorba/string/tokenize03.xml.res'
526--- test/rbkt/ExpQueryResults/zorba/string/tokenize03.xml.res 1970-01-01 00:00:00 +0000
527+++ test/rbkt/ExpQueryResults/zorba/string/tokenize03.xml.res 2011-12-23 20:33:38 +0000
528@@ -0,0 +1,1 @@
529+ d d
530
531=== added file 'test/rbkt/ExpQueryResults/zorba/string/tokenize04.xml.res'
532--- test/rbkt/ExpQueryResults/zorba/string/tokenize04.xml.res 1970-01-01 00:00:00 +0000
533+++ test/rbkt/ExpQueryResults/zorba/string/tokenize04.xml.res 2011-12-23 20:33:38 +0000
534@@ -0,0 +1,1 @@
535+abcd abcd
536
537=== added file 'test/rbkt/Queries/zorba/string/token01.txt'
538--- test/rbkt/Queries/zorba/string/token01.txt 1970-01-01 00:00:00 +0000
539+++ test/rbkt/Queries/zorba/string/token01.txt 2011-12-23 20:33:38 +0000
540@@ -0,0 +1,1 @@
541+abcd
542\ No newline at end of file
543
544=== added file 'test/rbkt/Queries/zorba/string/token02.txt'
545--- test/rbkt/Queries/zorba/string/token02.txt 1970-01-01 00:00:00 +0000
546+++ test/rbkt/Queries/zorba/string/token02.txt 2011-12-23 20:33:38 +0000
547@@ -0,0 +1,1 @@
548+abc
549\ No newline at end of file
550
551=== added file 'test/rbkt/Queries/zorba/string/token03.txt'
552--- test/rbkt/Queries/zorba/string/token03.txt 1970-01-01 00:00:00 +0000
553+++ test/rbkt/Queries/zorba/string/token03.txt 2011-12-23 20:33:38 +0000
554@@ -0,0 +1,1 @@
555+bcd
556\ No newline at end of file
557
558=== added file 'test/rbkt/Queries/zorba/string/token04.txt'
559--- test/rbkt/Queries/zorba/string/token04.txt 1970-01-01 00:00:00 +0000
560+++ test/rbkt/Queries/zorba/string/token04.txt 2011-12-23 20:33:38 +0000
561@@ -0,0 +1,1 @@
562+abcd
563\ No newline at end of file
564
565=== added file 'test/rbkt/Queries/zorba/string/tokenize01.xq'
566--- test/rbkt/Queries/zorba/string/tokenize01.xq 1970-01-01 00:00:00 +0000
567+++ test/rbkt/Queries/zorba/string/tokenize01.xq 2011-12-23 20:33:38 +0000
568@@ -0,0 +1,5 @@
569+import module namespace f = "http://expath.org/ns/file";
570+import module namespace s = "http://www.zorba-xquery.com/modules/string";
571+
572+s:split(f:read-text(fn:resolve-uri("token01.txt")), "bc"),
573+s:split(s:materialize(f:read-text(fn:resolve-uri("token01.txt"))), "bc")
574
575=== added file 'test/rbkt/Queries/zorba/string/tokenize02.xq'
576--- test/rbkt/Queries/zorba/string/tokenize02.xq 1970-01-01 00:00:00 +0000
577+++ test/rbkt/Queries/zorba/string/tokenize02.xq 2011-12-23 20:33:38 +0000
578@@ -0,0 +1,5 @@
579+import module namespace f = "http://expath.org/ns/file";
580+import module namespace s = "http://www.zorba-xquery.com/modules/string";
581+
582+s:split(f:read-text(fn:resolve-uri("token02.txt")), "bc"),
583+s:split(s:materialize(f:read-text(fn:resolve-uri("token02.txt"))), "bc")
584
585=== added file 'test/rbkt/Queries/zorba/string/tokenize03.xq'
586--- test/rbkt/Queries/zorba/string/tokenize03.xq 1970-01-01 00:00:00 +0000
587+++ test/rbkt/Queries/zorba/string/tokenize03.xq 2011-12-23 20:33:38 +0000
588@@ -0,0 +1,5 @@
589+import module namespace f = "http://expath.org/ns/file";
590+import module namespace s = "http://www.zorba-xquery.com/modules/string";
591+
592+s:split(f:read-text(fn:resolve-uri("token03.txt")), "bc"),
593+s:split(s:materialize(f:read-text(fn:resolve-uri("token03.txt"))), "bc")
594
595=== added file 'test/rbkt/Queries/zorba/string/tokenize04.xq'
596--- test/rbkt/Queries/zorba/string/tokenize04.xq 1970-01-01 00:00:00 +0000
597+++ test/rbkt/Queries/zorba/string/tokenize04.xq 2011-12-23 20:33:38 +0000
598@@ -0,0 +1,5 @@
599+import module namespace f = "http://expath.org/ns/file";
600+import module namespace s = "http://www.zorba-xquery.com/modules/string";
601+
602+s:split(f:read-text(fn:resolve-uri("token04.txt")), "f"),
603+s:split(s:materialize(f:read-text(fn:resolve-uri("token04.txt"))), "f")

Subscribers

People subscribed via source and target branches