Zorba

Merge lp:~zorba-coders/zorba/feature-ft_module into lp:zorba

  1. feature-ft_module
  2. Merge into trunk
Proposed by Paul J. Lucas
Status: Merged
Approved by: Matthias Brantner
Approved revision: no longer in the source branch.
Merged at revision: 10852
Proposed branch: lp:~zorba-coders/zorba/feature-ft_module
Merge into: lp:zorba
Diff against target: 455 lines (+76/-76)
16 files modified
doc/zorba/ft_tokenizer.dox (+1/-1)
modules/com/zorba-xquery/www/modules/full-text.xq (+8/-8)
src/functions/func_ft_module_impl.cpp (+8/-8)
src/functions/func_ft_module_impl.h (+3/-3)
src/functions/function_consts.h (+2/-2)
src/runtime/full_text/ft_module_impl.cpp (+15/-14)
src/runtime/full_text/pregenerated/ft_module.cpp (+15/-15)
src/runtime/full_text/pregenerated/ft_module.h (+9/-9)
src/runtime/spec/full_text/ft_module.xml (+1/-1)
src/runtime/visitors/pregenerated/planiter_visitor.h (+3/-3)
src/runtime/visitors/pregenerated/printer_visitor.cpp (+5/-5)
src/runtime/visitors/pregenerated/printer_visitor.h (+2/-2)
test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-node-1.xq (+1/-1)
test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-node-2.xq (+1/-2)
test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-node-3.xq (+1/-1)
test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-node-4.xq (+1/-1)
To merge this branch: bzr merge lp:~zorba-coders/zorba/feature-ft_module
Reviewer Review Type Date Requested Status
Matthias Brantner Approve
Paul J. Lucas Approve
Review via email: mp+106235@code.launchpad.net

Commit message

Getting in another public API change for 2.5 for the full-text module since now's the time to do it. Renamed tokenize() to tokenize-node() for 2 reasons:

1. There already exists tokenize-string() and therefore tokenize-node() is a better name than just plain tokenize().

2. The forthcoming addition of the black & white tokenization function will most likely be called tokenize-nodes() -- plural.

Description of the change

Getting in another public API change for 2.5 for the full-text module since now's the time to do it. Renamed tokenize() to tokenize-node() for 2 reasons:

1. There already exists tokenize-string() and therefore tokenize-node() is a better name than just plain tokenize().

2. The forthcoming addition of the black & white tokenization function will most likely be called tokenize-nodes() -- plural.

To post a comment you must log in.
Revision history for this message
Paul J. Lucas (paul-lucas) :
review: Approve
Revision history for this message
Zorba Build Bot (zorba-buildbot) wrote :

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-ft_module-2012-05-17T21-57-06.963Z/log.html

Revision history for this message
Zorba Build Bot (zorba-buildbot) wrote :

The attempt to merge lp:~zorba-coders/zorba/feature-ft_module into lp:zorba failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:274 (message):
  Validation queue job feature-ft_module-2012-05-17T21-57-06.963Z is
  finished. The final status was:

  1 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

Revision history for this message
Matthias Brantner (matthias-brantner) :
review: Approve
Revision history for this message
Zorba Build Bot (zorba-buildbot) wrote :

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-ft_module-2012-05-18T03-45-08.316Z/log.html

Revision history for this message
Zorba Build Bot (zorba-buildbot) wrote :

Validation queue job feature-ft_module-2012-05-18T03-45-08.316Z is finished. The final status was:

All tests succeeded!

lp:~zorba-coders/zorba/feature-ft_module updated
10852. By Paul J. Lucas

Getting in another public API change for 2.5 for the full-text module since now's the time to do it. Renamed tokenize() to tokenize-node() for 2 reasons:

1. There already exists tokenize-string() and therefore tokenize-node() is a better name than just plain tokenize().

2. The forthcoming addition of the black & white tokenization function will most likely be called tokenize-nodes() -- plural. Approved: Matthias Brantner, Paul J. Lucas

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'doc/zorba/ft_tokenizer.dox'
2--- doc/zorba/ft_tokenizer.dox 2012-05-16 01:01:06 +0000
3+++ doc/zorba/ft_tokenizer.dox 2012-05-17 22:48:19 +0000
4@@ -152,7 +152,7 @@
5 </tr>
6 </table>
7
8-A complete implementation of \c %tokenize() is non-trivial
9+A complete implementation of \c %tokenize_string() is non-trivial
10 and therefore an example is beyond the scope of this API documentation.
11 However,
12 the things a tokenizer should take into consideration include:
13
14=== modified file 'modules/com/zorba-xquery/www/modules/full-text.xq'
15--- modules/com/zorba-xquery/www/modules/full-text.xq 2012-05-08 23:49:22 +0000
16+++ modules/com/zorba-xquery/www/modules/full-text.xq 2012-05-17 22:48:19 +0000
17@@ -762,7 +762,7 @@
18 as xs:string+ external;
19
20 (:~
21- : Tokenizes the given document.
22+ : Tokenizes the given node and all of its descendants.
23 :
24 : @param $node The node to tokenize.
25 : @param $lang The default
26@@ -770,13 +770,13 @@
27 : of <code>$node</code>.
28 : @return a (possibly empty) sequence of tokens.
29 : @error err:FTST0009 if <code>$lang</code> is not supported in general.
30- : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-1.xq
31+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-node-1.xq
32 :)
33-declare function ft:tokenize( $node as node(), $lang as xs:language )
34+declare function ft:tokenize-node( $node as node(), $lang as xs:language )
35 as element(ft-schema:token)* external;
36
37 (:~
38- : Tokenizes the given document.
39+ : Tokenizes the given node and all of its descendants.
40 :
41 : @param $node The node to tokenize.
42 : The document's default
43@@ -785,11 +785,11 @@
44 : @return a (possibly empty) sequence of tokens.
45 : @error err:FTST0009 if <code>ft:current-lang()</code> is not supported in
46 : general.
47- : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-2.xq
48- : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-3.xq
49- : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-4.xq
50+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-node-2.xq
51+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-node-3.xq
52+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-node-4.xq
53 :)
54-declare function ft:tokenize( $node as node() )
55+declare function ft:tokenize-node( $node as node() )
56 as element(ft-schema:token)* external;
57
58 (:~
59
60=== modified file 'src/functions/func_ft_module_impl.cpp'
61--- src/functions/func_ft_module_impl.cpp 2012-05-15 21:13:21 +0000
62+++ src/functions/func_ft_module_impl.cpp 2012-05-17 22:48:19 +0000
63@@ -25,14 +25,14 @@
64
65 #ifndef ZORBA_NO_FULL_TEXT
66
67-PlanIter_t full_text_tokenize::codegen(
68+PlanIter_t full_text_tokenize_node::codegen(
69 CompilerCB*,
70 static_context* sctx,
71 const QueryLoc& loc,
72 std::vector<PlanIter_t>& argv,
73 expr& ann) const
74 {
75- return new TokenizeIterator(sctx, loc, argv);
76+ return new TokenizeNodeIterator(sctx, loc, argv);
77 }
78
79
80@@ -90,20 +90,20 @@
81 false);
82 {
83 DECL_WITH_KIND(sctx,
84- full_text_tokenize,
85- (createQName(FT_MODULE_NS, "", "tokenize"),
86+ full_text_tokenize_node,
87+ (createQName(FT_MODULE_NS, "", "tokenize-node"),
88 GENV_TYPESYSTEM.ANY_NODE_TYPE_ONE,
89 tokenize_return_type),
90- FunctionConsts::FULL_TEXT_TOKENIZE_1);
91+ FunctionConsts::FULL_TEXT_TOKENIZE_NODE_1);
92 }
93 {
94 DECL_WITH_KIND(sctx,
95- full_text_tokenize,
96- (createQName( FT_MODULE_NS, "", "tokenize"),
97+ full_text_tokenize_node,
98+ (createQName( FT_MODULE_NS, "", "tokenize-node"),
99 GENV_TYPESYSTEM.ANY_NODE_TYPE_ONE,
100 GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE,
101 tokenize_return_type),
102- FunctionConsts::FULL_TEXT_TOKENIZE_2);
103+ FunctionConsts::FULL_TEXT_TOKENIZE_NODE_2);
104 }
105
106 xqtref_t tokenizer_properties_return_type =
107
108=== modified file 'src/functions/func_ft_module_impl.h'
109--- src/functions/func_ft_module_impl.h 2012-05-09 20:40:03 +0000
110+++ src/functions/func_ft_module_impl.h 2012-05-17 22:48:19 +0000
111@@ -30,11 +30,11 @@
112 ///////////////////////////////////////////////////////////////////////////////
113
114 //full-text:tokenize
115-class full_text_tokenize : public function
116+class full_text_tokenize_node : public function
117 {
118 public:
119- full_text_tokenize(const signature& sig, FunctionConsts::FunctionKind kind)
120- :
121+ full_text_tokenize_node(const signature& sig,
122+ FunctionConsts::FunctionKind kind) :
123 function(sig, kind)
124 {
125
126
127=== modified file 'src/functions/function_consts.h'
128--- src/functions/function_consts.h 2012-05-08 23:49:22 +0000
129+++ src/functions/function_consts.h 2012-05-17 22:48:19 +0000
130@@ -229,8 +229,8 @@
131 FULL_TEXT_CURRENT_COMPARE_OPTIONS_0,
132 FULL_TEXT_TOKENIZER_PROPERTIES_1,
133 FULL_TEXT_TOKENIZER_PROPERTIES_0,
134- FULL_TEXT_TOKENIZE_2,
135- FULL_TEXT_TOKENIZE_1,
136+ FULL_TEXT_TOKENIZE_NODE_2,
137+ FULL_TEXT_TOKENIZE_NODE_1,
138 #endif
139
140 #include "functions/function_enum.h"
141
142=== modified file 'src/runtime/full_text/ft_module_impl.cpp'
143--- src/runtime/full_text/ft_module_impl.cpp 2012-05-17 15:21:43 +0000
144+++ src/runtime/full_text/ft_module_impl.cpp 2012-05-17 22:48:19 +0000
145@@ -528,14 +528,15 @@
146
147 ///////////////////////////////////////////////////////////////////////////////
148
149-TokenizeIterator::TokenizeIterator( static_context *sctx, QueryLoc const &loc,
150- std::vector<PlanIter_t>& children ) :
151- NaryBaseIterator<TokenizeIterator,TokenizeIteratorState>(sctx, loc, children)
152+TokenizeNodeIterator::TokenizeNodeIterator( static_context *sctx,
153+ QueryLoc const &loc,
154+ std::vector<PlanIter_t>& children ):
155+ NaryBaseIterator<TokenizeNodeIterator,TokenizeNodeIteratorState>(sctx, loc, children)
156 {
157 initMembers();
158 }
159
160-void TokenizeIterator::initMembers() {
161+void TokenizeNodeIterator::initMembers() {
162 GENV_ITEMFACTORY->createQName(
163 token_qname_, static_context::ZORBA_FULL_TEXT_FN_NS, "", "token" );
164
165@@ -555,8 +556,8 @@
166 ref_qname_, "", "", "node-ref" );
167 }
168
169-bool TokenizeIterator::nextImpl( store::Item_t &result,
170- PlanState &plan_state ) const {
171+bool TokenizeNodeIterator::nextImpl( store::Item_t &result,
172+ PlanState &plan_state ) const {
173 store::Item_t node_name, attr_node;
174 zstring base_uri;
175 store::Item_t item;
176@@ -567,8 +568,8 @@
177 store::Item_t type_name;
178 zstring value_string;
179
180- TokenizeIteratorState *state;
181- DEFAULT_STACK_INIT( TokenizeIteratorState, state, plan_state );
182+ TokenizeNodeIteratorState *state;
183+ DEFAULT_STACK_INIT( TokenizeNodeIteratorState, state, plan_state );
184
185 if ( consumeNext( state->doc_item_, theChildren[0], plan_state ) ) {
186 if ( theChildren.size() > 1 ) {
187@@ -651,19 +652,19 @@
188 STACK_END( state );
189 }
190
191-void TokenizeIterator::resetImpl( PlanState &plan_state ) const {
192- NaryBaseIterator<TokenizeIterator,TokenizeIteratorState>::
193+void TokenizeNodeIterator::resetImpl( PlanState &plan_state ) const {
194+ NaryBaseIterator<TokenizeNodeIterator,TokenizeNodeIteratorState>::
195 resetImpl( plan_state );
196- TokenizeIteratorState *const state =
197- StateTraitsImpl<TokenizeIteratorState>::getState(
198+ TokenizeNodeIteratorState *const state =
199+ StateTraitsImpl<TokenizeNodeIteratorState>::getState(
200 plan_state, this->theStateOffset
201 );
202 state->doc_tokens_->reset();
203 }
204
205-void TokenizeIterator::serialize( serialization::Archiver &ar ) {
206+void TokenizeNodeIterator::serialize( serialization::Archiver &ar ) {
207 serialize_baseclass(
208- ar, (NaryBaseIterator<TokenizeIterator,TokenizeIteratorState>*)this
209+ ar, (NaryBaseIterator<TokenizeNodeIterator,TokenizeNodeIteratorState>*)this
210 );
211 if ( !ar.is_serializing_out() )
212 initMembers();
213
214=== modified file 'src/runtime/full_text/pregenerated/ft_module.cpp'
215--- src/runtime/full_text/pregenerated/ft_module.cpp 2012-05-08 23:49:22 +0000
216+++ src/runtime/full_text/pregenerated/ft_module.cpp 2012-05-17 22:48:19 +0000
217@@ -295,12 +295,12 @@
218
219 #endif
220 #ifndef ZORBA_NO_FULL_TEXT
221-// <TokenizeIterator>
222-TokenizeIterator::class_factory<TokenizeIterator>
223-TokenizeIterator::g_class_factory;
224-
225-
226-void TokenizeIterator::accept(PlanIterVisitor& v) const {
227+// <TokenizeNodeIterator>
228+TokenizeNodeIterator::class_factory<TokenizeNodeIterator>
229+TokenizeNodeIterator::g_class_factory;
230+
231+
232+void TokenizeNodeIterator::accept(PlanIterVisitor& v) const {
233 v.beginVisit(*this);
234
235 std::vector<PlanIter_t>::const_iterator lIter = theChildren.begin();
236@@ -312,17 +312,17 @@
237 v.endVisit(*this);
238 }
239
240-TokenizeIterator::~TokenizeIterator() {}
241-
242-TokenizeIteratorState::TokenizeIteratorState() {}
243-
244-TokenizeIteratorState::~TokenizeIteratorState() {}
245-
246-
247-void TokenizeIteratorState::reset(PlanState& planState) {
248+TokenizeNodeIterator::~TokenizeNodeIterator() {}
249+
250+TokenizeNodeIteratorState::TokenizeNodeIteratorState() {}
251+
252+TokenizeNodeIteratorState::~TokenizeNodeIteratorState() {}
253+
254+
255+void TokenizeNodeIteratorState::reset(PlanState& planState) {
256 PlanIteratorState::reset(planState);
257 }
258-// </TokenizeIterator>
259+// </TokenizeNodeIterator>
260
261 #endif
262 #ifndef ZORBA_NO_FULL_TEXT
263
264=== modified file 'src/runtime/full_text/pregenerated/ft_module.h'
265--- src/runtime/full_text/pregenerated/ft_module.h 2012-05-08 23:49:22 +0000
266+++ src/runtime/full_text/pregenerated/ft_module.h 2012-05-17 22:48:19 +0000
267@@ -455,20 +455,20 @@
268 *
269 * Author:
270 */
271-class TokenizeIteratorState : public PlanIteratorState
272+class TokenizeNodeIteratorState : public PlanIteratorState
273 {
274 public:
275 store::Item_t doc_item_; //
276 FTTokenIterator_t doc_tokens_; //
277
278- TokenizeIteratorState();
279+ TokenizeNodeIteratorState();
280
281- ~TokenizeIteratorState();
282+ ~TokenizeNodeIteratorState();
283
284 void reset(PlanState&);
285 };
286
287-class TokenizeIterator : public NaryBaseIterator<TokenizeIterator, TokenizeIteratorState>
288+class TokenizeNodeIterator : public NaryBaseIterator<TokenizeNodeIterator, TokenizeNodeIteratorState>
289 {
290 protected:
291 store::Item_t token_qname_; //
292@@ -478,20 +478,20 @@
293 store::Item_t value_qname_; //
294 store::Item_t ref_qname_; //
295 public:
296- SERIALIZABLE_CLASS(TokenizeIterator);
297+ SERIALIZABLE_CLASS(TokenizeNodeIterator);
298
299- SERIALIZABLE_CLASS_CONSTRUCTOR2T(TokenizeIterator,
300- NaryBaseIterator<TokenizeIterator, TokenizeIteratorState>);
301+ SERIALIZABLE_CLASS_CONSTRUCTOR2T(TokenizeNodeIterator,
302+ NaryBaseIterator<TokenizeNodeIterator, TokenizeNodeIteratorState>);
303
304 void serialize( ::zorba::serialization::Archiver& ar);
305
306- TokenizeIterator(
307+ TokenizeNodeIterator(
308 static_context* sctx,
309 const QueryLoc& loc,
310 std::vector<PlanIter_t>& children)
311 ;
312
313- virtual ~TokenizeIterator();
314+ virtual ~TokenizeNodeIterator();
315
316 public:
317 void initMembers();
318
319=== modified file 'src/runtime/spec/full_text/ft_module.xml'
320--- src/runtime/spec/full_text/ft_module.xml 2012-05-08 23:49:22 +0000
321+++ src/runtime/spec/full_text/ft_module.xml 2012-05-17 22:48:19 +0000
322@@ -167,7 +167,7 @@
323 </zorba:state>
324 </zorba:iterator>
325
326-<zorba:iterator name="TokenizeIterator"
327+<zorba:iterator name="TokenizeNodeIterator"
328 generateResetImpl="true"
329 generateSerialize="false"
330 generateConstructor="false"
331
332=== modified file 'src/runtime/visitors/pregenerated/planiter_visitor.h'
333--- src/runtime/visitors/pregenerated/planiter_visitor.h 2012-05-08 23:49:22 +0000
334+++ src/runtime/visitors/pregenerated/planiter_visitor.h 2012-05-17 22:48:19 +0000
335@@ -227,7 +227,7 @@
336 class ThesaurusLookupIterator;
337 #endif
338 #ifndef ZORBA_NO_FULL_TEXT
339- class TokenizeIterator;
340+ class TokenizeNodeIterator;
341 #endif
342 #ifndef ZORBA_NO_FULL_TEXT
343 class TokenizerPropertiesIterator;
344@@ -951,8 +951,8 @@
345 virtual void endVisit ( const ThesaurusLookupIterator& ) = 0;
346 #endif
347 #ifndef ZORBA_NO_FULL_TEXT
348- virtual void beginVisit ( const TokenizeIterator& ) = 0;
349- virtual void endVisit ( const TokenizeIterator& ) = 0;
350+ virtual void beginVisit ( const TokenizeNodeIterator& ) = 0;
351+ virtual void endVisit ( const TokenizeNodeIterator& ) = 0;
352 #endif
353 #ifndef ZORBA_NO_FULL_TEXT
354 virtual void beginVisit ( const TokenizerPropertiesIterator& ) = 0;
355
356=== modified file 'src/runtime/visitors/pregenerated/printer_visitor.cpp'
357--- src/runtime/visitors/pregenerated/printer_visitor.cpp 2012-05-08 23:49:22 +0000
358+++ src/runtime/visitors/pregenerated/printer_visitor.cpp 2012-05-17 22:48:19 +0000
359@@ -1412,18 +1412,18 @@
360
361 #endif
362 #ifndef ZORBA_NO_FULL_TEXT
363-// <TokenizeIterator>
364-void PrinterVisitor::beginVisit ( const TokenizeIterator& a) {
365- thePrinter.startBeginVisit("TokenizeIterator", ++theId);
366+// <TokenizeNodeIterator>
367+void PrinterVisitor::beginVisit ( const TokenizeNodeIterator& a) {
368+ thePrinter.startBeginVisit("TokenizeNodeIterator", ++theId);
369 printCommons( &a, theId );
370 thePrinter.endBeginVisit( theId );
371 }
372
373-void PrinterVisitor::endVisit ( const TokenizeIterator& ) {
374+void PrinterVisitor::endVisit ( const TokenizeNodeIterator& ) {
375 thePrinter.startEndVisit();
376 thePrinter.endEndVisit();
377 }
378-// </TokenizeIterator>
379+// </TokenizeNodeIterator>
380
381 #endif
382 #ifndef ZORBA_NO_FULL_TEXT
383
384=== modified file 'src/runtime/visitors/pregenerated/printer_visitor.h'
385--- src/runtime/visitors/pregenerated/printer_visitor.h 2012-05-08 23:49:22 +0000
386+++ src/runtime/visitors/pregenerated/printer_visitor.h 2012-05-17 22:48:19 +0000
387@@ -348,8 +348,8 @@
388 #endif
389
390 #ifndef ZORBA_NO_FULL_TEXT
391- void beginVisit( const TokenizeIterator& );
392- void endVisit ( const TokenizeIterator& );
393+ void beginVisit( const TokenizeNodeIterator& );
394+ void endVisit ( const TokenizeNodeIterator& );
395 #endif
396
397 #ifndef ZORBA_NO_FULL_TEXT
398
399=== renamed file 'test/rbkt/ExpQueryResults/zorba/fulltext/ft-module-tokenize-1.xml.res' => 'test/rbkt/ExpQueryResults/zorba/fulltext/ft-module-tokenize-node-1.xml.res'
400=== renamed file 'test/rbkt/ExpQueryResults/zorba/fulltext/ft-module-tokenize-2.xml.res' => 'test/rbkt/ExpQueryResults/zorba/fulltext/ft-module-tokenize-node-2.xml.res'
401=== renamed file 'test/rbkt/ExpQueryResults/zorba/fulltext/ft-module-tokenize-3.xml.res' => 'test/rbkt/ExpQueryResults/zorba/fulltext/ft-module-tokenize-node-3.xml.res'
402=== renamed file 'test/rbkt/ExpQueryResults/zorba/fulltext/ft-module-tokenize-4.xml.res' => 'test/rbkt/ExpQueryResults/zorba/fulltext/ft-module-tokenize-node-4.xml.res'
403=== renamed file 'test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-1.xq' => 'test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-node-1.xq'
404--- test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-1.xq 2012-05-08 17:24:54 +0000
405+++ test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-node-1.xq 2012-05-17 22:48:19 +0000
406@@ -2,7 +2,7 @@
407 import schema namespace fts = "http://www.zorba-xquery.com/modules/full-text";
408
409 let $doc := <msg>hello, world</msg>
410-let $tokens := ft:tokenize( $doc, xs:language("en") )
411+let $tokens := ft:tokenize-node( $doc, xs:language("en") )
412 let $t1 := validate { $tokens[1] }
413 let $t2 := validate { $tokens[2] }
414
415
416=== renamed file 'test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-2.xq' => 'test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-node-2.xq'
417--- test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-2.xq 2012-05-05 11:37:42 +0000
418+++ test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-node-2.xq 2012-05-17 22:48:19 +0000
419@@ -1,9 +1,8 @@
420 import module namespace ft = "http://www.zorba-xquery.com/modules/full-text";
421-
422 import schema namespace fts = "http://www.zorba-xquery.com/modules/full-text";
423
424 let $doc := <msg xml:lang="es">hola, mundo</msg>
425-let $tokens := ft:tokenize( $doc )
426+let $tokens := ft:tokenize-node( $doc )
427 let $t1 := validate { $tokens[1] }
428 let $t2 := validate { $tokens[2] }
429
430
431=== renamed file 'test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-3.xq' => 'test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-node-3.xq'
432--- test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-3.xq 2012-05-05 16:28:22 +0000
433+++ test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-node-3.xq 2012-05-17 22:48:19 +0000
434@@ -4,7 +4,7 @@
435 import schema namespace fts = "http://www.zorba-xquery.com/modules/full-text";
436
437 let $x := <p xml:lang="en">Houston, we have a <em>problem</em>!</p>
438-let $tokens := ft:tokenize( $x )
439+let $tokens := ft:tokenize-node( $x )
440 let $node-ref := (validate { $tokens[5] })/@node-ref
441 let $node := ref:node-by-reference( $node-ref )
442 return $node instance of text()
443
444=== renamed file 'test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-4.xq' => 'test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-node-4.xq'
445--- test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-4.xq 2012-05-05 16:28:22 +0000
446+++ test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-node-4.xq 2012-05-17 22:48:19 +0000
447@@ -4,7 +4,7 @@
448 import schema namespace fts = "http://www.zorba-xquery.com/modules/full-text";
449
450 let $x := <msg xml:lang="en" content="Houston, we have a problem!"/>
451-let $tokens := ft:tokenize( $x/@content )
452+let $tokens := ft:tokenize-node( $x/@content )
453 let $node-ref := (validate { $tokens[5] }) /@node-ref
454 let $node := ref:node-by-reference( $node-ref )
455 return $node instance of attribute(content)

Subscribers

People subscribed via source and target branches