Merge lp:~oontvoo/akiban-server/bug_drop_fulltext_index into lp:~akiban-technologies/akiban-server/trunk

Proposed by Vy Nguyen
Status: Superseded
Proposed branch: lp:~oontvoo/akiban-server/bug_drop_fulltext_index
Merge into: lp:~akiban-technologies/akiban-server/trunk
Diff against target: 2442 lines (+1511/-117)
29 files modified
src/main/java/com/akiban/rest/resources/FullTextResource.java (+0/-19)
src/main/java/com/akiban/server/rowdata/RowDefCache.java (+3/-1)
src/main/java/com/akiban/server/service/dxl/BasicDDLFunctions.java (+8/-1)
src/main/java/com/akiban/server/service/restdml/RestDMLService.java (+0/-2)
src/main/java/com/akiban/server/service/restdml/RestDMLServiceImpl.java (+0/-10)
src/main/java/com/akiban/server/service/text/FullTextCursor.java (+5/-4)
src/main/java/com/akiban/server/service/text/FullTextIndexInfo.java (+79/-2)
src/main/java/com/akiban/server/service/text/FullTextIndexInfosImpl.java (+16/-0)
src/main/java/com/akiban/server/service/text/FullTextIndexService.java (+15/-1)
src/main/java/com/akiban/server/service/text/FullTextIndexServiceImpl.java (+343/-12)
src/main/java/com/akiban/server/service/text/FullTextIndexShared.java (+5/-0)
src/main/java/com/akiban/server/service/text/Indexer.java (+8/-1)
src/main/java/com/akiban/server/service/text/RowIndexer.java (+35/-4)
src/main/java/com/akiban/server/store/PersistitStore.java (+395/-2)
src/main/java/com/akiban/server/types3/mcompat/mfuncs/MSleep.java (+71/-0)
src/main/java/com/akiban/sql/aisddl/AISDDL.java (+1/-1)
src/main/java/com/akiban/sql/aisddl/AlterTableDDL.java (+1/-1)
src/main/java/com/akiban/sql/aisddl/IndexDDL.java (+13/-7)
src/main/java/com/akiban/sql/aisddl/TableDDL.java (+6/-5)
src/main/resources/com/akiban/server/service/config/configuration-defaults.properties (+3/-0)
src/test/java/com/akiban/server/rowdata/SchemaFactory.java (+18/-3)
src/test/java/com/akiban/server/service/text/FullTextIndexServiceIT.java (+317/-28)
src/test/java/com/akiban/server/test/ApiTestBase.java (+10/-10)
src/test/java/com/akiban/server/test/it/bugs/bug701614/MissingColumnsIT.java (+1/-1)
src/test/java/com/akiban/sql/optimizer/OptimizerTestBase.java (+51/-2)
src/test/resources/com/akiban/sql/pg/yaml/functional/test-drop-fulltext.yaml (+8/-0)
src/test/resources/com/akiban/sql/pg/yaml/functional/test-ft-with-inherited-key.yaml (+28/-0)
src/test/resources/com/akiban/sql/pg/yaml/functional/test-ft-with-pk.yaml (+22/-0)
src/test/resources/com/akiban/sql/pg/yaml/functional/test-fulltext-maintenance.yaml (+49/-0)
To merge this branch: bzr merge lp:~oontvoo/akiban-server/bug_drop_fulltext_index
Reviewer Review Type Date Requested Status
Nathan Williams Needs Fixing
Vy Nguyen (community) Needs Resubmitting
Review via email: mp+158707@code.launchpad.net

This proposal has been superseded by a proposal from 2013-04-15.

Description of the change

fix drop fulltext index bug (bug 1163026)

To post a comment you must log in.
Revision history for this message
Nathan Williams (nwilliams) wrote :

What is the IndexDef needed for?

I'm guessing bug1154336 is quite similar too. Perhaps dropTableInternal() needs to be in on the game.

review: Needs Information
Revision history for this message
Vy Nguyen (oontvoo) wrote :

PersisitStore.java - line 1554 (+/- a few lines)

It looks like the index needs to be 'attached' to an IndexDef.

Revision history for this message
Nathan Williams (nwilliams) wrote :

Only to delete the tree, which full texts don't have.

DDLFunctions has separate methods for dropping table and group indexes. Something different needs to happen for full texts (i.e. FullTextIndexService#dropIndex), so another entry point would be consistent and make sense.

Revision history for this message
Nathan Williams (nwilliams) wrote :

Actually, handling the distinction in PS#deleteIndexes() would let dropTableInternal() use it as well, so I'm fine with that too.

2621. By Vy Nguyen

skip fulltext index in persistitStore.deleteIndexes()

Revision history for this message
Vy Nguyen (oontvoo) wrote :

skip fulltext index in persistitStore.deleteIndexes()

review: Needs Resubmitting
Revision history for this message
Nathan Williams (nwilliams) wrote :

Since we're here, can you please fix the other bug I mentioned. We just need to call FTIS#dropIndex() for them.

review: Needs Fixing
2622. By Vy Nguyen

pass FullTextIndexService to PersistitStore for deleteIndexes() to use

2623. By Vy Nguyen

bind FTS to FTSImpl

Revision history for this message
Vy Nguyen (oontvoo) wrote :

I am having a compilation-error in

src/test/java/com/akiban/server/test/FailureOnStartupIT.java:[85,12] error: constructor OperatorStore in class OperatorStore cannot be applied to given types;

(I declared FTS as an argument in PersistStore's ctor, and other places where it's called.)

What did I miss?

review: Needs Information
Revision history for this message
Vy Nguyen (oontvoo) wrote :

Hm, never mind, found it!

Revision history for this message
Nathan Williams (nwilliams) wrote :

Look at FailureOnStartupIT.java, line 85. It extends OperatorStore so its constructor needs adjusted too.

2624. By Vy Nguyen

have FullTextIndexService attache itself to PersistStore rather than passing it to the constructor because that'd be circular dependency

2625. By Vy Nguyen

revert back to 2621

Revision history for this message
Vy Nguyen (oontvoo) wrote :

Actually, it's more complicated than just calling FTS.dropIndex()

+ By the time deleteIndexes() is called, the index is no longer in AIS for FullTextIndexShared.forAIS() to look up. But that's not the 'complicated' part.

+ In order for dropIndex() to work, the index needs to be in 'indexes' map, which none of the current fulltext index is because no one puts it in there. The createIndex() method, which would do this, needs to be called manually.
This problem is solved in the do-maintenance branch. And I can certainly have it fix the bug you mentioned. (In fact, I tried merging this branch into it and the delete works correctly, which it wouldn't in this branch).

review: Needs Resubmitting
Revision history for this message
Nathan Williams (nwilliams) wrote :

> By the time deleteIndexes() is called, the index is no
> longer in AIS for FullTextIndexShared.forAIS() to look up

It does not need to be in the current AIS and probably shouldn't be, given that it's being removed. Look at the flow of the other drop methods (e.g. table, index). The AIS changes are performed through the SchemaManager before tree deletion is done. You can either hang onto the 'old' AIS or get it through the Index reference.

> In order for dropIndex() to work, the index needs to be in 'indexes' map,

It should be, and sounds like will be once your other branch is in, but doesn't need to be. The getIndex() method will populate the indexes map on demand. It creates a value cached on the AIS, which is going away in this case, but will work just fine.

review: Needs Fixing
2626. By Vy Nguyen

fix bug1154336

2627. By Vy Nguyen

remove formatting changes

2628. By Vy Nguyen

clean up

2629. By Vy Nguyen
2630. By Vy Nguyen

more tests

2631. By Vy Nguyen

require FulLTetIndexService for testing

2632. By Vy Nguyen

basicddl

2633. By Vy Nguyen

add tests on droping table with ftindex. removeTrees should also delete fulltext index, even if these don't really have a tree

2634. By Vy Nguyen

add the original testcase from the bug report

2635. By Vy Nguyen
2636. By Vy Nguyen

merge trunk

2637. By Vy Nguyen

use wait() in test instead of blindly sleep for a long time

2638. By Vy Nguyen

merge trunk, move configs to TestConfigService

2639. By Vy Nguyen

edge cases in removeTrees()

2640. By Vy Nguyen

remove unused imports in FullTextIndexServiceIT

2641. By Vy Nguyen

use getownFullTextIndex

2642. By Vy Nguyen

clean up

2643. By Vy Nguyen

clean up

2644. By Vy Nguyen

pass Index as an argument to dropIndex to correctly determine the treename

2645. By Vy Nguyen

delete fulltext index

2646. By Vy Nguyen

use default settings in OptimizerTestBase

2647. By Vy Nguyen

use the old AIS when neccessary

2648. By Vy Nguyen

remove unused import accidentally added

Unmerged revisions

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'src/main/java/com/akiban/rest/resources/FullTextResource.java'
2--- src/main/java/com/akiban/rest/resources/FullTextResource.java 2013-03-22 20:05:57 +0000
3+++ src/main/java/com/akiban/rest/resources/FullTextResource.java 2013-04-15 17:23:26 +0000
4@@ -30,7 +30,6 @@
5 import javax.ws.rs.QueryParam;
6 import javax.ws.rs.core.Context;
7 import javax.ws.rs.core.Response;
8-import javax.ws.rs.core.UriInfo;
9 import java.io.PrintWriter;
10
11 import static com.akiban.rest.resources.ResourceHelper.MEDIATYPE_JSON_JAVASCRIPT;
12@@ -66,22 +65,4 @@
13 })
14 .build();
15 }
16-
17- @POST
18- @Produces(MEDIATYPE_JSON_JAVASCRIPT)
19- public Response textSearch(@Context final HttpServletRequest request,
20- @PathParam("table") String table,
21- @PathParam("index") String index) throws Exception {
22- final IndexName indexName = new IndexName(ResourceHelper.parseTableName(request, table), index);
23- ResourceHelper.checkSchemaAccessible(reqs.securityService, request, indexName.getSchemaName());
24- return RestResponseBuilder
25- .forRequest(request)
26- .body(new RestResponseBuilder.BodyGenerator() {
27- @Override
28- public void write(PrintWriter writer) throws Exception {
29- reqs.restDMLService.refreshFullTextIndex(writer, indexName);
30- }
31- })
32- .build();
33- }
34 }
35
36=== modified file 'src/main/java/com/akiban/server/rowdata/RowDefCache.java'
37--- src/main/java/com/akiban/server/rowdata/RowDefCache.java 2013-03-22 20:05:57 +0000
38+++ src/main/java/com/akiban/server/rowdata/RowDefCache.java 2013-04-15 17:23:26 +0000
39@@ -38,6 +38,7 @@
40
41 import com.akiban.ais.model.AkibanInformationSchema;
42 import com.akiban.ais.model.Column;
43+import com.akiban.ais.model.FullTextIndex;
44 import com.akiban.ais.model.IndexColumn;
45 import com.akiban.ais.model.Join;
46 import com.akiban.ais.model.JoinColumn;
47@@ -184,6 +185,7 @@
48 for (TableIndex index : table.getIndexesIncludingInternal()) {
49 List<IndexColumn> indexColumns = index.getKeyColumns();
50 if(!indexColumns.isEmpty()) {
51+
52 new IndexDef(rowDef, index);
53 if (index.isPrimaryKey()) {
54 indexList.add(0, index);
55@@ -193,7 +195,7 @@
56 }
57 //else Don't create IndexDef for empty, autogenerated indexes
58 }
59-
60+
61 // Group indexes
62 final List<GroupIndex> groupIndexList = new ArrayList<>();
63 for (GroupIndex index : table.getGroupIndexes()) {
64
65=== modified file 'src/main/java/com/akiban/server/service/dxl/BasicDDLFunctions.java'
66--- src/main/java/com/akiban/server/service/dxl/BasicDDLFunctions.java 2013-04-11 18:35:20 +0000
67+++ src/main/java/com/akiban/server/service/dxl/BasicDDLFunctions.java 2013-04-15 17:23:26 +0000
68@@ -1031,7 +1031,14 @@
69 for(String indexName : indexNamesToDrop) {
70 Index index = table.getIndex(indexName);
71 if(index == null) {
72- throw new NoSuchIndexException (indexName);
73+ if (table instanceof UserTable)
74+ {
75+ index = ((UserTable)table).getFullTextIndex(indexName);
76+ if (index == null)
77+ throw new NoSuchIndexException (indexName);
78+ }
79+ else
80+ throw new NoSuchIndexException (indexName);
81 }
82 if(index.isPrimaryKey()) {
83 throw new ProtectedIndexException(indexName, table.getName());
84
85=== modified file 'src/main/java/com/akiban/server/service/restdml/RestDMLService.java'
86--- src/main/java/com/akiban/server/service/restdml/RestDMLService.java 2013-04-15 14:38:06 +0000
87+++ src/main/java/com/akiban/server/service/restdml/RestDMLService.java 2013-04-15 17:23:26 +0000
88@@ -51,8 +51,6 @@
89 TableName procName, Map<String,List<String>> queryParams, String content) throws SQLException;
90
91 public void fullTextSearch(PrintWriter writer, IndexName indexName, Integer depth, String query, Integer limit);
92- // TODO: Temporary.
93- public void refreshFullTextIndex(PrintWriter writer, IndexName indexName);
94
95 public String jonquilToSQL(TableName tableName, String jonquil) throws IOException;
96 }
97
98=== modified file 'src/main/java/com/akiban/server/service/restdml/RestDMLServiceImpl.java'
99--- src/main/java/com/akiban/server/service/restdml/RestDMLServiceImpl.java 2013-04-15 14:38:06 +0000
100+++ src/main/java/com/akiban/server/service/restdml/RestDMLServiceImpl.java 2013-04-15 17:23:26 +0000
101@@ -649,14 +649,4 @@
102 true);
103 }
104 }
105-
106- // TODO: Temporary.
107- public void refreshFullTextIndex(PrintWriter writer, IndexName indexName) {
108- long count;
109- try (Session session = sessionService.createSession()) {
110- count = fullTextService.createIndex(session, indexName);
111- }
112- writer.write(String.format("{\"count\":%d}", count));
113- }
114-
115 }
116
117=== modified file 'src/main/java/com/akiban/server/service/text/FullTextCursor.java'
118--- src/main/java/com/akiban/server/service/text/FullTextCursor.java 2013-03-22 20:05:57 +0000
119+++ src/main/java/com/akiban/server/service/text/FullTextCursor.java 2013-04-15 17:23:26 +0000
120@@ -103,7 +103,7 @@
121 catch (IOException ex) {
122 throw new AkibanInternalException("Error reading document", ex);
123 }
124- HKey hkey = hkey(doc.getBinaryValue(IndexedField.KEY_FIELD));
125+ HKey hkey = hkey(doc.get(IndexedField.KEY_FIELD));
126 Row row = new HKeyRow(rowType, hkey, hKeyCache);
127 logger.debug("FullTextCursor: yield {}", row);
128 return row;
129@@ -152,11 +152,12 @@
130
131 /* Allocate a new <code>PersistitHKey</code> and copy the given
132 * key bytes into it. */
133- protected HKey hkey(BytesRef keyBytes) {
134+ protected HKey hkey(String encoded) {
135 PersistitHKey hkey = (PersistitHKey)context.getStore().newHKey(rowType.hKey());
136 Key key = hkey.key();
137- key.setEncodedSize(keyBytes.length);
138- System.arraycopy(keyBytes.bytes, keyBytes.offset, key.getEncodedBytes(), 0, keyBytes.length);
139+ byte decodedBytes[] = RowIndexer.decodeString(encoded);
140+ key.setEncodedSize(decodedBytes.length);
141+ System.arraycopy(decodedBytes, 0, key.getEncodedBytes(), 0, decodedBytes.length);
142 return hkey;
143 }
144
145
146=== modified file 'src/main/java/com/akiban/server/service/text/FullTextIndexInfo.java'
147--- src/main/java/com/akiban/server/service/text/FullTextIndexInfo.java 2013-03-22 20:05:57 +0000
148+++ src/main/java/com/akiban/server/service/text/FullTextIndexInfo.java 2013-04-15 17:23:26 +0000
149@@ -20,12 +20,16 @@
150 import com.akiban.ais.model.AkibanInformationSchema;
151 import com.akiban.ais.model.Column;
152 import com.akiban.ais.model.FullTextIndex;
153+import com.akiban.ais.model.Group;
154 import com.akiban.ais.model.IndexColumn;
155 import com.akiban.ais.model.IndexName;
156 import com.akiban.ais.model.UserTable;
157+import com.akiban.qp.expression.IndexKeyRange;
158 import com.akiban.qp.operator.API;
159 import com.akiban.qp.operator.Operator;
160+import com.akiban.qp.row.HKeyRow;
161 import com.akiban.qp.rowtype.HKeyRowType;
162+import com.akiban.qp.rowtype.IndexRowType;
163 import com.akiban.qp.rowtype.RowType;
164 import com.akiban.qp.rowtype.Schema;
165 import com.akiban.qp.rowtype.UserTableRowType;
166@@ -50,7 +54,8 @@
167 private Map<Column,IndexedField> fieldsByColumn;
168 private Map<RowType,List<IndexedField>> fieldsByRowType;
169 private String defaultFieldName;
170-
171+ private Operator plan;
172+
173 public FullTextIndexInfo(FullTextIndexShared shared) {
174 this.shared = shared;
175 }
176@@ -63,7 +68,9 @@
177 }
178 index = table.getFullTextIndex(name.getName());
179 if (index == null) {
180- throw new NoSuchIndexException(name.getName());
181+ NoSuchIndexException ret = new NoSuchIndexException(name.getName());
182+ ret.printStackTrace();
183+ throw ret;
184 }
185 schema = SchemaCache.globalSchema(ais);
186 indexedRowType = schema.userTableRowType(table);
187@@ -87,6 +94,8 @@
188 }
189 fields.add(entry.getValue());
190 }
191+
192+ plan = computePlan();
193 }
194
195 public FullTextIndex getIndex() {
196@@ -139,6 +148,74 @@
197 plan = API.filter_Default(plan, rowTypes);
198 return plan;
199 }
200+
201+
202+
203+ private Operator computePlan()
204+ {
205+ Operator ret = null;
206+
207+ Group group = indexedRowType.userTable().getGroup();
208+ Set<UserTableRowType> ancestors = new HashSet<>();
209+ boolean hasDesc = false;
210+
211+ for (IndexColumn ic : index.getKeyColumns())
212+ {
213+ UserTable colUserTable = ic.getColumn().getUserTable();
214+
215+ if (!hasDesc && !colUserTable.equals(indexedRowType.userTable()))
216+ // if any column in the index def belongs to a table
217+ // that is a descendant of this indexed row's table
218+ // (meaning this indexed row has descendant(s))
219+ hasDesc = colUserTable.isDescendantOf(indexedRowType.userTable());
220+
221+ // if the indexed table is a child of this column's table
222+ // (meaning this indexed row has parent(s))
223+ // collect all ancestor's rowtype
224+ if (indexedRowType.userTable().isDescendantOf(colUserTable))
225+ ancestors.add(schema.userTableRowType(colUserTable));
226+ }
227+
228+ if (hasDesc)
229+ {
230+ ancestors.remove(indexedRowType);
231+
232+ ret = API.branchLookup_Nested(group,
233+ hKeyRowType,
234+ indexedRowType,
235+ API.InputPreservationOption.DISCARD_INPUT,
236+ 0);
237+ if (!ancestors.isEmpty())
238+ {
239+
240+ ret = API.ancestorLookup_Default(ret,
241+ group,
242+ indexedRowType,
243+ ancestors,
244+ API.InputPreservationOption.KEEP_INPUT);
245+ }
246+ }
247+ else
248+ {
249+ // has at least one ancestor (which is itself)
250+ ancestors.add(indexedRowType);
251+ ret = API.ancestorLookup_Nested(group,
252+ hKeyRowType,
253+ ancestors,
254+ 0);
255+ }
256+
257+ return ret;
258+ }
259+ /**
260+ *
261+ * @param row
262+ * @return the operator plan to get to every row related to this index row
263+ */
264+ public Operator getOperator()
265+ {
266+ return plan;
267+ }
268
269 public Analyzer getAnalyzer() {
270 Analyzer analyzer;
271
272=== modified file 'src/main/java/com/akiban/server/service/text/FullTextIndexInfosImpl.java'
273--- src/main/java/com/akiban/server/service/text/FullTextIndexInfosImpl.java 2013-03-22 20:05:57 +0000
274+++ src/main/java/com/akiban/server/service/text/FullTextIndexInfosImpl.java 2013-04-15 17:23:26 +0000
275@@ -23,6 +23,7 @@
276 import com.akiban.qp.rowtype.RowType;
277 import com.akiban.qp.rowtype.Schema;
278 import com.akiban.server.error.FullTextQueryParseException;
279+import com.akiban.server.error.NoSuchIndexException;
280 import com.akiban.server.service.session.Session;
281
282 import org.apache.lucene.queryparser.flexible.core.QueryNodeException;
283@@ -65,6 +66,21 @@
284 protected abstract AkibanInformationSchema getAIS(Session session);
285 protected abstract File getIndexPath();
286
287+ protected FullTextIndexInfo getIndexToDrop(IndexName name)
288+ {
289+ synchronized(indexes)
290+ {
291+ FullTextIndexShared shared = indexes.get(name);
292+ if (shared != null)
293+ {
294+ return shared.valueFor();
295+ }
296+ else
297+ {
298+ throw new NoSuchIndexException(name.toString());
299+ }
300+ }
301+ }
302 protected FullTextIndexInfo getIndex(Session session, IndexName name) {
303 AkibanInformationSchema ais = getAIS(session);
304 FullTextIndexInfo info;
305
306=== modified file 'src/main/java/com/akiban/server/service/text/FullTextIndexService.java'
307--- src/main/java/com/akiban/server/service/text/FullTextIndexService.java 2013-03-22 20:05:57 +0000
308+++ src/main/java/com/akiban/server/service/text/FullTextIndexService.java 2013-04-15 17:23:26 +0000
309@@ -27,7 +27,21 @@
310
311 /** Full service that does index maintenance and querying. */
312 public interface FullTextIndexService extends FullTextIndexInfos {
313- public long createIndex(Session session, IndexName name);
314+
315+ /**
316+ * This is a 'promise' to populate the given index, which is *being*
317+ * created.
318+ * @param name
319+ */
320+ public void schedulePopulate(String schema, String table, String index);
321+
322+ /**
323+ * Update the given index based on the changedRow
324+ *
325+ * @param name
326+ * @param changedRow
327+ */
328+ public void updateIndex(Session session, IndexName name, Iterable<byte[]> rows);
329 public void dropIndex(Session session, IndexName name);
330 public Cursor searchIndex(QueryContext context, IndexName name,
331 Query query, int limit);
332
333=== modified file 'src/main/java/com/akiban/server/service/text/FullTextIndexServiceImpl.java'
334--- src/main/java/com/akiban/server/service/text/FullTextIndexServiceImpl.java 2013-03-22 20:05:57 +0000
335+++ src/main/java/com/akiban/server/service/text/FullTextIndexServiceImpl.java 2013-04-15 17:23:26 +0000
336@@ -18,39 +18,59 @@
337 package com.akiban.server.service.text;
338
339 import com.akiban.ais.model.AkibanInformationSchema;
340+import com.akiban.ais.model.Index;
341 import com.akiban.ais.model.IndexName;
342+import com.akiban.ais.model.TableName;
343+import com.akiban.ais.model.UserTable;
344 import com.akiban.qp.operator.API;
345 import com.akiban.qp.operator.Cursor;
346 import com.akiban.qp.operator.Operator;
347 import com.akiban.qp.operator.QueryContext;
348 import com.akiban.qp.operator.SimpleQueryContext;
349 import com.akiban.qp.operator.StoreAdapter;
350-import com.akiban.qp.rowtype.RowType;
351-import com.akiban.qp.rowtype.Schema;
352 import com.akiban.qp.persistitadapter.PersistitAdapter;
353+import com.akiban.qp.persistitadapter.PersistitHKey;
354+import com.akiban.qp.row.HKeyRow;
355+import com.akiban.qp.rowtype.HKeyRowType;
356+import com.akiban.qp.util.HKeyCache;
357 import com.akiban.server.error.AkibanInternalException;
358-import com.akiban.server.error.DuplicateIndexException;
359 import com.akiban.server.service.Service;
360 import com.akiban.server.service.config.ConfigurationService;
361 import com.akiban.server.service.dxl.DXLService;
362 import com.akiban.server.service.session.Session;
363+import com.akiban.server.service.session.SessionService;
364+import com.akiban.server.service.session.SessionServiceImpl;
365 import com.akiban.server.service.transaction.TransactionService;
366 import com.akiban.server.service.tree.TreeService;
367+import com.akiban.server.store.PersistitStore;
368+import com.akiban.server.store.PersistitStore.HKeyBytesStream;
369 import com.akiban.server.store.Store;
370
371-import org.apache.lucene.analysis.Analyzer;
372 import org.apache.lucene.index.IndexWriter;
373 import org.apache.lucene.search.Query;
374
375 import com.google.inject.Inject;
376+import com.persistit.Exchange;
377+import com.persistit.exception.PersistitException;
378+import com.persistit.Key;
379 import org.slf4j.Logger;
380 import org.slf4j.LoggerFactory;
381
382 import java.io.*;
383-import java.util.*;
384+import java.util.Timer;
385+import java.util.TimerTask;
386+
387
388 public class FullTextIndexServiceImpl extends FullTextIndexInfosImpl implements FullTextIndexService, Service {
389 public static final String INDEX_PATH_PROPERTY = "akserver.text.indexpath";
390+ public static final String UPDATE_INTERVAL = "akserver.text.maintenanceInterval";
391+ public static final String POPULATE_DELAY_INTERVAL = "akserver.text.populateDelayInterval";
392+
393+ private static final String POPULATE_SCHEMA = "populate";
394+ private static final String FULL_TEXT_TABLE = "full_text_populate";
395+
396+ private final SessionService sessionService = new SessionServiceImpl();
397+
398
399 private final ConfigurationService configService;
400 private final DXLService dxlService;
401@@ -59,7 +79,15 @@
402 private final TreeService treeService;
403
404 private File indexPath;
405-
406+
407+ private volatile Timer maintenanceTimer;
408+ private long maintenanceInterval;
409+ private TimerTask updateWorker;
410+ private PersistitStore persistitStore;
411+
412+ private volatile Timer populateTimer;
413+ private long populateDelayInterval;
414+
415 private static final Logger logger = LoggerFactory.getLogger(FullTextIndexServiceImpl.class);
416
417 @Inject
418@@ -72,12 +100,14 @@
419 this.store = store;
420 this.transactionService = transactionService;
421 this.treeService = treeService;
422+
423 }
424
425 /* FullTextIndexService */
426
427- @Override
428- public long createIndex(Session session, IndexName name) {
429+ private long createIndex(Session session, IndexName name) {
430+ if (session == null)
431+ session = sessionService.createSession();
432 FullTextIndexInfo index = getIndex(session, name);
433 try {
434 return populateIndex(session, index);
435@@ -89,10 +119,27 @@
436
437 @Override
438 public void dropIndex(Session session, IndexName name) {
439- FullTextIndexInfo index = getIndex(session, name);
440- index.deletePath();
441- synchronized (indexes) {
442- indexes.remove(name);
443+
444+ try
445+ {
446+ // see if there exists a promise for populating this index
447+ Exchange ex = getPopulateExchange(session);
448+ ex.clear().append(name.getSchemaName())
449+ .append(name.getTableName())
450+ .append(name.getName());
451+
452+ if (ex.traverse(Key.Direction.EQ, true, 0))
453+ ex.fetchAndRemove();
454+
455+ FullTextIndexInfo index = getIndexToDrop(name);
456+ index.deletePath();
457+ synchronized (indexes) {
458+ indexes.remove(name);
459+ }
460+ }
461+ catch (PersistitException e)
462+ {
463+ throw new AkibanInternalException("Error while removing index", e);
464 }
465 }
466
467@@ -112,6 +159,10 @@
468
469 @Override
470 public void start() {
471+ persistitStore = store.getPersistitStore();
472+ persistitStore.setFullTextService(this);
473+ enableUpdateWorker();
474+ enablePopulateWorker();
475 }
476
477 @Override
478@@ -120,6 +171,8 @@
479 for (FullTextIndexShared index : indexes.values()) {
480 index.close();
481 }
482+ disableUpdateWorker();
483+ disablePopulateWorker();
484 }
485 catch (IOException ex) {
486 throw new AkibanInternalException("Error closing index", ex);
487@@ -193,4 +246,282 @@
488 }
489 }
490
491+ @Override
492+ public void updateIndex(Session session, IndexName name, Iterable<byte[]> rows)
493+ {
494+ try
495+ {
496+ FullTextIndexInfo indexInfo = getIndex(session, name);
497+ StoreAdapter adapter = session.get(StoreAdapter.STORE_ADAPTER_KEY);
498+ if (adapter == null)
499+ adapter = new PersistitAdapter(indexInfo.getSchema(),
500+ store, treeService,
501+ session, configService);
502+ QueryContext queryContext = new SimpleQueryContext(adapter);
503+ HKeyCache<com.akiban.qp.row.HKey> cache = new HKeyCache<>(adapter);
504+ IndexWriter writer = indexInfo.getIndexer().getWriter();
505+ RowIndexer rowIndexer = new RowIndexer(indexInfo, writer, true);
506+ Cursor cursor = null;
507+ boolean success = false;
508+ try
509+ {
510+ Operator operator = indexInfo.getOperator();
511+ int n = 0;
512+ for (byte row[] : rows)
513+ {
514+ HKeyRow hkeyRow = toHKeyRow(row, indexInfo.getHKeyRowType(),
515+ adapter, cache);
516+ queryContext.setRow(0, hkeyRow);
517+ cursor = API.cursor(operator, queryContext);
518+ rowIndexer.updateDocument(cursor, row);
519+ }
520+ success = true;
521+ }
522+ finally
523+ {
524+ rowIndexer.close();
525+ if (cursor != null)
526+ cursor.destroy();
527+
528+ try
529+ {
530+ if (success)
531+ writer.commit();
532+ else
533+ writer.rollback();
534+ }
535+ catch (IOException ex)
536+ {
537+ throw new AkibanInternalException("Error commit writer", ex);
538+ }
539+ }
540+ }
541+ catch (IOException e)
542+ {
543+ throw new AkibanInternalException("Error updating index", e);
544+ }
545+ }
546+
547+ private class DefaultUpdateWorker extends TimerTask
548+ {
549+ @Override
550+ // 'sync' because only one worker can work at a time?
551+ public synchronized void run()
552+ {
553+ Session session = sessionService.createSession();
554+ boolean transaction = true;
555+ try
556+ {
557+ transactionService.beginTransaction(session);
558+ HKeyBytesStream rows = persistitStore.getChangedRows(session);
559+ if (rows != null) // if tree is not empty
560+ {
561+ do
562+ {
563+ updateIndex(session,
564+ rows.getIndexName(),
565+ rows);
566+ }
567+ while (rows.nextIndex());
568+ rows.removeAll(); // done updating. remove all entries
569+ }
570+ transaction= false;
571+ }
572+ catch(PersistitException e)
573+ {
574+ throw new AkibanInternalException("Error while maintaning full_text indices");
575+ }
576+ finally
577+ {
578+ if (transaction)
579+ transactionService.rollbackTransaction(session);
580+ else
581+ transactionService.commitTransaction(session);
582+ session.close();
583+ }
584+ }
585+ };
586+
587+
588+ @Override
589+ public void schedulePopulate(String schema, String table, String index)
590+ {
591+ Session session = sessionService.createSession();
592+ boolean success = false;
593+ try
594+ {
595+ transactionService.beginTransaction(session);
596+ if(addPopulate(session, schema, table, index) && !hasScheduled && populateEnabled)
597+ {
598+ populateTimer.schedule(populateWorker(), populateDelayInterval);
599+ hasScheduled = true;
600+ }
601+
602+ success = true;
603+ }
604+ catch (PersistitException ex)
605+ {
606+ throw new AkibanInternalException("Error while scheduling index population", ex);
607+ }
608+ finally
609+ {
610+ if (success)
611+ transactionService.commitTransaction(session);
612+ else
613+ transactionService.rollbackTransaction(session);
614+ session.close();
615+ }
616+ }
617+
618+ private class DefaultPopulateWorker extends TimerTask
619+ {
620+ @Override
621+ public synchronized void run()
622+ {
623+ Session session = sessionService.createSession();
624+ try
625+ {
626+ Exchange ex = getPopulateExchange(session);
627+ IndexName toPopulate;
628+ while ((toPopulate = nextInQueue(ex)) != null)
629+ {
630+ if (stillExists(session, toPopulate))
631+ createIndex(session, toPopulate);
632+ else
633+ logger.debug("FullTextIndex " + toPopulate + " deleted before population");
634+ }
635+ ex.removeAll();
636+ hasScheduled = false;
637+ }
638+ catch (PersistitException ex1)
639+ {
640+ throw new AkibanInternalException("Error while populating full_text indices", ex1);
641+ }
642+ finally
643+ {
644+ session.close();
645+ }
646+ }
647+
648+ private boolean stillExists(Session session, IndexName indexName)
649+ {
650+ AkibanInformationSchema ais = getAIS(session);
651+ UserTable table = ais.getUserTable(indexName.getFullTableName());
652+ return !(table == null || table.getFullTextIndex(indexName.getName()) == null);
653+ }
654+
655+ };
656+
657+ // ---------- for testing ---------------
658+ void disableUpdateWorker()
659+ {
660+ if (maintenanceTimer == null)
661+ {
662+ logger.debug("maintenance worker ALREADY disabled");
663+ return;
664+ }
665+ updateWorker.cancel();
666+ maintenanceTimer.cancel();
667+ maintenanceTimer.purge();
668+ maintenanceTimer = null;
669+ updateWorker = null;
670+ }
671+
672+ protected void enableUpdateWorker()
673+ {
674+ maintenanceInterval = Long.parseLong(configService.getProperty(UPDATE_INTERVAL));
675+ maintenanceTimer = new Timer();
676+ updateWorker = new DefaultUpdateWorker();
677+ maintenanceTimer.scheduleAtFixedRate(updateWorker, maintenanceInterval, maintenanceInterval);
678+ }
679+
680+ protected void enablePopulateWorker()
681+ {
682+ populateDelayInterval = Long.parseLong(configService.getProperty(POPULATE_DELAY_INTERVAL));
683+ populateTimer = new Timer();
684+ populateTimer.schedule(populateWorker(), populateDelayInterval);
685+ populateEnabled = true;
686+ hasScheduled = true;
687+ }
688+
689+ void disablePopulateWorker()
690+ {
691+ if (populateTimer == null)
692+ {
693+ logger.debug("populate worker already disabled");
694+ return;
695+ }
696+ populateTimer.cancel();
697+ populateTimer.purge();
698+ hasScheduled = false;
699+ populateEnabled = false;
700+ populateTimer = null;
701+ }
702+
703+ private TimerTask populateWorker()
704+ {
705+ return new DefaultPopulateWorker();
706+ }
707+
708+ //----------- private helpers -----------
709+ protected IndexName nextInQueue(Exchange ex) throws PersistitException
710+ {
711+ Key key = ex.getKey();
712+
713+ if (ex.next(true)) // empty tree?
714+ {
715+ key.reset();
716+ IndexName ret = new IndexName(new TableName(key.decodeString(),
717+ key.decodeString()),
718+ key.decodeString());
719+ return ret;
720+ }
721+ else
722+ return null;
723+ }
724+
725+ protected Exchange getPopulateExchange(Session session) throws PersistitException
726+ {
727+ Exchange ret = treeService.getExchange(session,
728+ treeService.treeLink(POPULATE_SCHEMA,
729+ FULL_TEXT_TABLE));
730+ // start at the first entry
731+ ret.append(Key.BEFORE);
732+ return ret;
733+ }
734+
735+ private volatile boolean hasScheduled = false;
736+ private volatile boolean populateEnabled = false;
737+
738+ private synchronized boolean addPopulate(Session session,
739+ String schema,
740+ String table,
741+ String index) throws PersistitException
742+ {
743+ Exchange ex = getPopulateExchange(session);
744+
745+ // Assumption: There is not any existing entry about this index
746+ // (Because they should have been removed in dropIndex())
747+ // KEY: schema | table | indexName
748+ ex.getKey().clear()
749+ .append(schema)
750+ .append(table)
751+ .append(index);
752+
753+ // VALUE: <empty>
754+
755+ ex.store();
756+ return true;
757+ }
758+
759+ private HKeyRow toHKeyRow(byte rowBytes[], HKeyRowType hKeyRowType,
760+ StoreAdapter store, HKeyCache<com.akiban.qp.row.HKey> cache)
761+ {
762+ PersistitHKey hkey = (PersistitHKey)store.newHKey(hKeyRowType.hKey());
763+ Key key = hkey.key();
764+ key.setEncodedSize(rowBytes.length);
765+ System.arraycopy(rowBytes, 0, key.getEncodedBytes(), 0, rowBytes.length);
766+ return new HKeyRow(hKeyRowType, hkey, cache);
767+ }
768+
769 }
770
771=== modified file 'src/main/java/com/akiban/server/service/text/FullTextIndexShared.java'
772--- src/main/java/com/akiban/server/service/text/FullTextIndexShared.java 2013-03-22 20:05:57 +0000
773+++ src/main/java/com/akiban/server/service/text/FullTextIndexShared.java 2013-04-15 17:23:26 +0000
774@@ -103,6 +103,11 @@
775 return ais.getCachedValue(this, this);
776 }
777
778+ public FullTextIndexInfo valueFor()
779+ {
780+ return new FullTextIndexInfo(this);
781+ }
782+
783 @Override
784 public FullTextIndexInfo valueFor(AkibanInformationSchema ais) {
785 FullTextIndexInfo result = new FullTextIndexInfo(this);
786
787=== modified file 'src/main/java/com/akiban/server/service/text/Indexer.java'
788--- src/main/java/com/akiban/server/service/text/Indexer.java 2013-03-22 20:05:57 +0000
789+++ src/main/java/com/akiban/server/service/text/Indexer.java 2013-04-15 17:23:26 +0000
790@@ -26,6 +26,7 @@
791
792 import java.io.Closeable;
793 import java.io.IOException;
794+import org.apache.lucene.index.IndexReader;
795
796 public class Indexer implements Closeable
797 {
798@@ -34,8 +35,14 @@
799
800 public Indexer(FullTextIndexShared index, Analyzer analyzer) throws IOException {
801 this.index = index;
802+
803+
804 IndexWriterConfig iwc = new IndexWriterConfig(Version.LUCENE_40, analyzer);
805- this.writer = new IndexWriter(index.open(), iwc);
806+ iwc.setMaxBufferedDeleteTerms(1); // The deletion needs to be reflected immediately (on disk)
807+
808+ this.writer = new IndexWriter(index.open(), iwc);
809+ IndexReader r;
810+
811 }
812
813 public FullTextIndexShared getIndex() {
814
815=== modified file 'src/main/java/com/akiban/server/service/text/RowIndexer.java'
816--- src/main/java/com/akiban/server/service/text/RowIndexer.java 2013-03-22 20:05:57 +0000
817+++ src/main/java/com/akiban/server/service/text/RowIndexer.java 2013-04-15 17:23:26 +0000
818@@ -20,9 +20,11 @@
819 import com.akiban.ais.model.AkibanInformationSchema;
820 import com.akiban.qp.operator.Cursor;
821 import com.akiban.qp.persistitadapter.PersistitHKey;
822+import com.akiban.qp.row.HKey;
823 import com.akiban.qp.row.Row;
824 import com.akiban.qp.rowtype.RowType;
825 import com.akiban.qp.rowtype.UserTableRowType;
826+import com.akiban.qp.util.PersistitKey;
827 import com.akiban.server.types3.pvalue.PValueSource;
828 import com.akiban.util.ShareHolder;
829 import com.persistit.Key;
830@@ -39,6 +41,11 @@
831
832 import java.io.*;
833 import java.util.*;
834+import org.apache.commons.codec.binary.Base64;
835+import org.apache.lucene.document.Field.Store;
836+import org.apache.lucene.document.StringField;
837+import org.apache.lucene.index.DirectoryReader;
838+import org.apache.lucene.index.IndexReader;
839
840 /** Given <code>Row</code>s in hkey order, create <code>Document</code>s. */
841 public class RowIndexer implements Closeable
842@@ -50,7 +57,8 @@
843 private IndexWriter writer;
844 private Document currentDocument;
845 private long documentCount;
846- private BytesRef keyBytes;
847+ //private BytesRef keyBytes;
848+ private String keyEncodedString;
849 private boolean updating;
850
851 private static final Logger logger = LoggerFactory.getLogger(RowIndexer.class);
852@@ -134,10 +142,21 @@
853 return documentCount;
854 }
855
856+ protected void updateDocument(Cursor cursor, byte hkeyBytes[]) throws IOException
857+ {
858+ if (indexRows(cursor) == 0)
859+ {
860+ String encoded = encodeBytes(hkeyBytes, 0, hkeyBytes.length);
861+ writer.deleteDocuments(new Term(IndexedField.KEY_FIELD, encoded));
862+ logger.debug("Deleted documents with encoded byptes: " + encoded);
863+ }
864+ }
865+
866 protected void addDocument() throws IOException {
867 if (currentDocument != null) {
868 if (updating) {
869- writer.updateDocument(new Term(IndexedField.KEY_FIELD, keyBytes),
870+
871+ writer.updateDocument(new Term(IndexedField.KEY_FIELD, keyEncodedString),
872 currentDocument);
873 logger.debug("Updated {}", currentDocument);
874 }
875@@ -152,8 +171,8 @@
876
877 protected void getKeyBytes(Row row) {
878 Key key = ((PersistitHKey)row.hKey()).key();
879- keyBytes = new BytesRef(key.getEncodedBytes(), 0, key.getEncodedSize());
880- Field field = new StoredField(IndexedField.KEY_FIELD, keyBytes);
881+ keyEncodedString = encodeBytes(key.getEncodedBytes(), 0, key.getEncodedSize());
882+ Field field = new StringField(IndexedField.KEY_FIELD, keyEncodedString, Store.YES);
883 currentDocument.add(field);
884 }
885
886@@ -166,6 +185,18 @@
887 }
888 }
889
890+ static String encodeBytes(byte bytes[], int offset, int length)
891+ {
892+ // TODO: needs to be more efficient?
893+ String ret = Base64.encodeBase64String(Arrays.copyOfRange(bytes, offset, length));
894+ return ret;
895+ }
896+
897+ static byte[] decodeString(String st)
898+ {
899+ return Base64.decodeBase64(st);
900+ }
901+
902 @Override
903 public void close() {
904 for (ShareHolder<Row> holder : ancestors) {
905
906=== modified file 'src/main/java/com/akiban/server/store/PersistitStore.java'
907--- src/main/java/com/akiban/server/store/PersistitStore.java 2013-04-11 06:15:45 +0000
908+++ src/main/java/com/akiban/server/store/PersistitStore.java 2013-04-15 17:23:26 +0000
909@@ -18,6 +18,7 @@
910 package com.akiban.server.store;
911
912 import com.akiban.ais.model.*;
913+import com.akiban.ais.model.Index.IndexType;
914 import com.akiban.qp.operator.StoreAdapter;
915 import com.akiban.qp.persistitadapter.OperatorBasedRowCollector;
916 import com.akiban.qp.persistitadapter.PersistitAdapter;
917@@ -41,6 +42,7 @@
918 import com.akiban.server.service.config.ConfigurationService;
919 import com.akiban.server.service.lock.LockService;
920 import com.akiban.server.service.session.Session;
921+import com.akiban.server.service.text.FullTextIndexService;
922 import com.akiban.server.service.transaction.TransactionService;
923 import com.akiban.server.service.tree.TreeLink;
924 import com.akiban.server.service.tree.TreeService;
925@@ -95,7 +97,9 @@
926 private static final String BULKLOAD_PK_BUFFER_ALLOC = "akserver.bulkload.bufferalloc.pk";
927 private static final String BULKLOAD_GROUP_BUFFER_ALLOC = "akserver.bulkload.bufferalloc.group";
928 private static final String WRITE_LOCK_ENABLED_CONFIG = "akserver.write_lock_enabled";
929-
930+ private static final String MAINTENANCE_SCHEMA = "maintenance";
931+ private static final String FULL_TEXT_TABLE = "full_text";
932+
933 private final static int MAX_ROW_SIZE = 5000000;
934
935 private final static int MAX_INDEX_TRANCHE_SIZE = 10 * 1024 * 1024;
936@@ -128,8 +132,14 @@
937
938 private int deferredIndexKeyLimit = MAX_INDEX_TRANCHE_SIZE;
939
940+ private FullTextIndexService fullTextService;
941+
942 private RowDataValueCoder valueCoder;
943
944+ // Each row change has a 'uniqueChangeId', stored in the 'maintenance.full_text' table
945+ // The number would be reset after all maintenance is done
946+ private volatile AtomicLong uniqueChangeId = new AtomicLong(0); // TODO: change this back to MIN_VAL
947+
948 private static final StorageAction DIRECT_STORAGE = new StorageAction() {
949 @Override
950 public void doStore(Exchange exchange) throws PersistitException {
951@@ -151,6 +161,368 @@
952 this.transactionService = transactionService;
953 }
954
955+ public void setFullTextService(FullTextIndexService service)
956+ {
957+ fullTextService = service;
958+ }
959+
960+ // --- for tracking changes
961+ private Exchange getChangeExchange(Session session) throws PersistitException
962+ {
963+ return treeService.getExchange(session,
964+ treeService.treeLink(MAINTENANCE_SCHEMA,
965+ FULL_TEXT_TABLE));
966+ }
967+
968+ private void addChange(Session session,
969+ String schema,
970+ String table,
971+ String index,
972+ Integer indexId,
973+ Key rowHKey) throws PersistitException
974+ {
975+ Exchange ex = getChangeExchange(session);
976+
977+ // KEY: schema | table | indexName | indexId | unique_num
978+ ex.clear().append(schema).append(table).append(index).append(indexId).append(uniqueChangeId.getAndIncrement());
979+
980+ // VALUE: rowHKey's bytes | indexId
981+ ex.getValue().clear().putByteArray(rowHKey.getEncodedBytes(),
982+ 0,
983+ rowHKey.getEncodedSize());
984+ ex.store();
985+ }
986+
987+ private void addChangeFor(UserTable tb, Session session, Key hKey) throws PersistitException
988+ {
989+ for (Index index : tb.getFullTextIndexes())
990+ {
991+ IndexName idxName = index.getIndexName();
992+ addChange(session,
993+ idxName.getSchemaName(),
994+ idxName.getTableName(),
995+ idxName.getName(),
996+ index.getIndexId(),
997+ hKey);
998+ }
999+ }
1000+ // --- for reporting changes
1001+ /**
1002+ * Collect all 'HKeyRow's from the list of key-value pairs that have
1003+ * the same schema.table.indexName,
1004+ * also fill schema[0],
1005+ * table[0],
1006+ * and indexName[0]
1007+ * with correct values
1008+ *
1009+ * @param schema
1010+ * @param table
1011+ * @param indexName
1012+ * @param ex
1013+ * @return
1014+ */
1015+ public HKeyBytesStream getChangedRows(Session session) throws PersistitException
1016+ {
1017+ Exchange ex = getChangeExchange(session);
1018+ ex.append(Key.BEFORE);
1019+ HKeyBytesStream ret;
1020+
1021+ if (ex.hasNext(true) // if the tree is not empty
1022+ && (ret = new HKeyBytesStream(ex, session)).hasNext()) // and does not contain only invalid indices
1023+ {
1024+ return ret;
1025+ }
1026+ else
1027+ {
1028+ return null;
1029+ }
1030+ }
1031+
1032+
1033+ /**
1034+ *
1035+ * This is supposed to behave similar* to Scheme's stream,
1036+ * (ie., each element is not computed until it absolutely has to be.)
1037+ *
1038+ * In other words, the byte arrays are not decoded before hand
1039+ */
1040+ public class HKeyBytesStream implements Iterable<byte[]>
1041+ {
1042+
1043+ private final Exchange ex;
1044+ private IndexName indexName;
1045+ private int modCount = 0;
1046+ private boolean moreRows; // more rows of this index
1047+ private final Session session;
1048+ private boolean eot; // end of tree;
1049+ private HKeyBytesStream()
1050+ {
1051+ ex = null;
1052+ session = null;
1053+ }
1054+
1055+ // 'private' because this should not be constructed
1056+ // anywhere other than PersistitStore.getChangedRows()
1057+ // The class itself, however, is public, as it can be used anywhere
1058+ private HKeyBytesStream (Exchange ex, Session session) throws PersistitException
1059+ {
1060+ this.ex = ex;
1061+ this.session = session;
1062+ eot = false;
1063+ moreRows = true;
1064+ nextIndex(); // at the beginning of the tree now. look for the first legit index
1065+ }
1066+
1067+ public final boolean nextIndex() throws PersistitException
1068+ {
1069+ if (eot)
1070+ return false;
1071+
1072+ boolean nextIndex = ex.next(true);
1073+ if (nextIndex)
1074+ {
1075+ modCount = 0;
1076+ indexName = buildName(ex);
1077+ if (eot = !ignoreDeleted(ex, indexName, true)) // reach the end after ignoring all deleted indexId
1078+ return moreRows = false; // hence no more rows (or indices)
1079+ }
1080+ else
1081+ {
1082+ eot = true;
1083+ moreRows = false;
1084+ }
1085+
1086+ return nextIndex;
1087+ }
1088+
1089+ public boolean hasNext()
1090+ {
1091+ return !eot;
1092+ }
1093+
1094+ public IndexName getIndexName()
1095+ {
1096+ return indexName;
1097+ }
1098+
1099+ @Override
1100+ public Iterator<byte[]> iterator()
1101+ {
1102+ return new StreamIterator(moreRows);
1103+ }
1104+
1105+ /**
1106+ * remove all change-entries
1107+ *
1108+ *
1109+ */
1110+ public void removeAll() throws PersistitException
1111+ {
1112+ ex.removeAll();
1113+ ++modCount;
1114+ }
1115+
1116+ private class StreamIterator implements Iterator<byte[]>
1117+ {
1118+
1119+ private boolean hasNext;
1120+ private int innerModCount = modCount;
1121+
1122+ private StreamIterator (boolean hasNext)
1123+ {
1124+ this.hasNext = hasNext;
1125+ }
1126+
1127+ @Override
1128+ public boolean hasNext()
1129+ {
1130+ return hasNext;
1131+ }
1132+
1133+ @Override
1134+ public byte[] next()
1135+ {
1136+ try
1137+ {
1138+ if (innerModCount != modCount)
1139+ throw new ConcurrentModificationException();
1140+
1141+ byte ret[] = ex.getValue().getByteArray();
1142+ boolean seeNewIndex = false;
1143+ boolean hasMore;
1144+ hasNext = (hasMore = ex.next(true))
1145+ && !(seeNewIndex = seeNewIndex(indexName.getSchemaName(),
1146+ indexName.getTableName(),
1147+ indexName.getName(),
1148+ ex.getKey())
1149+ // and following this row is at least one row of
1150+ // this index whose index-id is valid
1151+ && ignoreDeleted(ex, indexName, false));
1152+
1153+ // if there are no more entries,
1154+ // set eot so the 'outer loop', which takes care of each index
1155+ // does not attempt to do 'next(true)'
1156+ // (because surpringly, after the end has been reached
1157+ // the key will be set to BEFORE, therefore next(true)
1158+ /// will return 'true'. ==> infinite loop!)
1159+ eot = !hasMore;
1160+
1161+ // Take caution in order NOT to go to the next index (ie., different name)
1162+ if (seeNewIndex)
1163+ // back up one entry, because we have read past the last
1164+ // entry that has the same schema.table.indexName
1165+ ex.previous();
1166+ ++innerModCount;
1167+ ++modCount;
1168+
1169+ return ret;
1170+ }
1171+ catch (PersistitException ex)
1172+ {
1173+ throw new AkibanInternalException("Error retrieving rows from Exchange", ex);
1174+ }
1175+ }
1176+
1177+ @Override
1178+ public void remove()
1179+ {
1180+ throw new UnsupportedOperationException("Not supported yet.");
1181+ }
1182+ }
1183+
1184+ /**
1185+ * Skip all entries whose indexName is that of a non-existing index,
1186+ * or whose indexId is not the same as that of the the current index (with the same name)
1187+ *
1188+ * @param ex
1189+ * @param indexName
1190+ * @param lookPastNewIndex whether to go beyond the current index
1191+ * @return <FALSE> iff after ignoring all deleted indices, there's nothing left
1192+ * <TRUE> otherwise
1193+ * @throws PersistitException
1194+ */
1195+ private boolean ignoreDeleted(Exchange ex,
1196+ IndexName indexName,
1197+ boolean lookPastNewIndex)
1198+ throws PersistitException
1199+ {
1200+ // KEY: Schema | Table | indexName | indexID | ...
1201+ Key key = ex.getKey();
1202+ key.reset();
1203+
1204+ key.reset();
1205+ assert indexName.getSchemaName().equals(key.decodeString())
1206+ : "Unexpected schema" ;
1207+ assert indexName.getTableName().equals(key.decodeString())
1208+ : "Unexpected table";
1209+ assert indexName.getName().equals(key.decodeString())
1210+ : "Unexpected name";
1211+
1212+ Integer indexId = key.decodeInt();
1213+ while (sameIdAsCurrent(session, indexName, indexId) != NOCHANGE)
1214+ // there was a DELETE (and possibly RECREATE)
1215+ // ignore all changes in the old id(s)
1216+ {
1217+ boolean hasMore = true;
1218+ boolean seeNewIndex = false;
1219+
1220+ do
1221+ { /*do nothing (skipping deleted 'index')*/}
1222+ while ((hasMore = ex.next(true))
1223+ && !(seeNewIndex = seeNewIndex(indexName.getSchemaName(),
1224+ indexName.getTableName(),
1225+ indexName.getName(),
1226+ ex.getKey()))
1227+ && !seeNewIndexId(ex, indexId));
1228+
1229+ if (eot = !hasMore) // reach the end! (no more rows or indices!)
1230+ return false; // set 'eot' for the same reason in StreamIterator.next()
1231+
1232+ else if (seeNewIndex) // back up one entry in order not to
1233+ { // go past the last pair
1234+ ex.previous(true);
1235+
1236+ // Saw new index,
1237+ // hence do the checking again,
1238+ // if we're allowed to look past new index
1239+ // (otherwise, meaning no more rows with good ids of this
1240+ // index, return false)
1241+ return lookPastNewIndex ? nextIndex() : false;
1242+ }
1243+ else // ie., see new IndexId
1244+ {
1245+ Key k = ex.getKey();
1246+ k.reset();
1247+ k.decodeString();
1248+ k.decodeString();
1249+ k.decodeString();
1250+
1251+ indexId = k.decodeInt();
1252+ }
1253+ }
1254+ return true;
1255+ }
1256+
1257+ private boolean seeNewIndexId(Exchange ex, Integer oldId)
1258+ {
1259+ Key key = ex.getKey();
1260+ key.reset();
1261+ // skip uninteresting parts
1262+ key.decodeString();
1263+ key.decodeString();
1264+ key.decodeString();
1265+
1266+ return !oldId.equals(key.decodeInt());
1267+
1268+ }
1269+
1270+ /**
1271+ * Check if the given id is the same as the current index's id.
1272+ *
1273+ * (If the index with the given name no longer exists, return false)
1274+ * @param schema
1275+ * @param table
1276+ * @param indexName
1277+ * @param id
1278+ * @return
1279+ */
1280+ private int sameIdAsCurrent(Session session, IndexName indexName, Integer id)
1281+ {
1282+ AkibanInformationSchema ais = getAIS(session);
1283+ UserTable table = ais.getUserTable(indexName.getFullTableName());
1284+ Index index;
1285+ if (table == null || (index = table.getFullTextIndex(indexName.getName())) == null)
1286+ return DELETED;
1287+
1288+ return index.getIndexId() != id ? RECREATED : NOCHANGE;
1289+ }
1290+
1291+ private boolean seeNewIndex(String schema, String table, String indexName, Key k)
1292+ {
1293+ k.reset();
1294+ return !k.decodeString().equals(schema)
1295+ || !k.decodeString().equals(table)
1296+ || !k.decodeString().equals(indexName);
1297+ }
1298+
1299+ private IndexName buildName(Exchange e)
1300+ {
1301+ Key key = e.getKey();
1302+ key.reset();
1303+ return new IndexName(new TableName(key.decodeString(),
1304+ key.decodeString()),
1305+ key.decodeString());
1306+ }
1307+
1308+ // status of index
1309+ private static final int NOCHANGE = 0;
1310+ private static final int DELETED = 1;
1311+ private static final int RECREATED = 2;
1312+
1313+ }
1314+
1315+ //------ end fulltext index maintenance services ------
1316+
1317 @Override
1318 public synchronized void start() {
1319 try {
1320@@ -444,6 +816,7 @@
1321 }
1322
1323 PersistitIndexRowBuffer indexRow = new PersistitIndexRowBuffer(adapter(session));
1324+ addChangeFor(rowDef.userTable(), session, hEx.getKey());
1325 for (Index index : rowDef.getIndexes()) {
1326 insertIntoIndex(session, index, rowData, hEx.getKey(), indexRow, deferIndexes);
1327 }
1328@@ -668,6 +1041,7 @@
1329 {
1330 RowDef rowDef = writeCheck(session, rowData, false);
1331 Exchange hEx = null;
1332+
1333 DELETE_ROW_TAP.in();
1334 try {
1335 hEx = getExchange(session, rowDef);
1336@@ -681,12 +1055,16 @@
1337 if (!hEx.getValue().isDefined()) {
1338 throw new NoSuchRowException(hEx.getKey());
1339 }
1340+ // record the deletion of the old index row
1341+ if (deleteIndexes)
1342+ addChangeFor(rowDef.userTable(), session, hEx.getKey());
1343
1344 // Remove the h-row
1345 hEx.remove();
1346 rowDef.getTableStatus().rowDeleted();
1347
1348 // Remove the indexes, including the PK index
1349+
1350 PersistitIndexRowBuffer indexRow = new PersistitIndexRowBuffer(adapter(session));
1351 if(deleteIndexes) {
1352 for (Index index : rowDef.getIndexes()) {
1353@@ -738,6 +1116,7 @@
1354 RowDef newRowDef = rowDefFromExplicitOrId(session, newRowData);
1355 PersistitAdapter adapter = adapter(session);
1356 Exchange hEx = null;
1357+
1358 UPDATE_ROW_TAP.in();
1359 try {
1360 hEx = getExchange(session, rowDef);
1361@@ -751,6 +1130,7 @@
1362 if (!hEx.getValue().isDefined()) {
1363 throw new NoSuchRowException (hEx.getKey());
1364 }
1365+
1366 // Combine current version of row with the version coming in
1367 // on the update request.
1368 // This is done by taking only the values of columns listed
1369@@ -770,7 +1150,9 @@
1370 packRowData(hEx, newRowDef, mergedRowData);
1371 // Store the h-row
1372 hEx.store();
1373- // Update the indexes
1374+ // Update the indexes (new row)
1375+ addChangeFor(newRowDef.userTable(), session, hEx.getKey());
1376+
1377 PersistitIndexRowBuffer indexRowBuffer = new PersistitIndexRowBuffer(adapter);
1378 Index[] indexes = (indexesToMaintain == null) ? rowDef.getIndexes() : indexesToMaintain;
1379 for (Index index : indexes) {
1380@@ -874,6 +1256,10 @@
1381 int descendentOrdinal = descendentRowDef.getOrdinal();
1382 if ((tablesRequiringHKeyMaintenance == null || tablesRequiringHKeyMaintenance.get(descendentOrdinal))) {
1383 PROPAGATE_HKEY_CHANGE_ROW_REPLACE_TAP.hit();
1384+
1385+ // record the change
1386+ addChangeFor(descendentRowDef.userTable(), session, exchange.getKey());
1387+
1388 // Delete the current row from the tree. Don't call deleteRow, because we don't need to recompute
1389 // the hkey.
1390 exchange.remove();
1391@@ -1554,6 +1940,13 @@
1392 public void deleteIndexes(final Session session, final Collection<? extends Index> indexes) {
1393 List<TreeLink> links = new ArrayList<>(indexes.size());
1394 for(Index index : indexes) {
1395+ // no trees to drop
1396+ if (index.getIndexType() == IndexType.FULL_TEXT)
1397+ {
1398+ fullTextService.dropIndex(session, index.getIndexName());
1399+ indexes.remove(index);
1400+ continue;
1401+ }
1402 final IndexDef indexDef = index.indexDef();
1403 if(indexDef == null) {
1404 throw new IllegalStateException("indexDef is null for index: " + index);
1405
1406=== added file 'src/main/java/com/akiban/server/types3/mcompat/mfuncs/MSleep.java'
1407--- src/main/java/com/akiban/server/types3/mcompat/mfuncs/MSleep.java 1970-01-01 00:00:00 +0000
1408+++ src/main/java/com/akiban/server/types3/mcompat/mfuncs/MSleep.java 2013-04-15 17:23:26 +0000
1409@@ -0,0 +1,71 @@
1410+/**
1411+ * Copyright (C) 2009-2013 Akiban Technologies, Inc.
1412+ *
1413+ * This program is free software: you can redistribute it and/or modify
1414+ * it under the terms of the GNU Affero General Public License as published by
1415+ * the Free Software Foundation, either version 3 of the License, or
1416+ * (at your option) any later version.
1417+ *
1418+ * This program is distributed in the hope that it will be useful,
1419+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
1420+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1421+ * GNU Affero General Public License for more details.
1422+ *
1423+ * You should have received a copy of the GNU Affero General Public License
1424+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
1425+ */
1426+
1427+package com.akiban.server.types3.mcompat.mfuncs;
1428+
1429+import com.akiban.server.error.AkibanInternalException;
1430+import com.akiban.server.types3.LazyList;
1431+import com.akiban.server.types3.TExecutionContext;
1432+import com.akiban.server.types3.TOverloadResult;
1433+import com.akiban.server.types3.TScalar;
1434+import com.akiban.server.types3.mcompat.mtypes.MNumeric;
1435+import com.akiban.server.types3.pvalue.PValueSource;
1436+import com.akiban.server.types3.pvalue.PValueTarget;
1437+import com.akiban.server.types3.texpressions.TInputSetBuilder;
1438+import com.akiban.server.types3.texpressions.TScalarBase;
1439+
1440+public class MSleep extends TScalarBase
1441+{
1442+
1443+ public static final TScalar SLEEP = new MSleep();
1444+
1445+ private MSleep() {}
1446+ @Override
1447+ protected void buildInputSets(TInputSetBuilder builder)
1448+ {
1449+ builder.covers(MNumeric.INT, 0);
1450+ }
1451+
1452+ @Override
1453+ protected void doEvaluate(TExecutionContext context, LazyList<? extends PValueSource> inputs, PValueTarget output)
1454+ {
1455+ try
1456+ {
1457+ int value = inputs.get(0).getInt32();
1458+ Thread.sleep(value);
1459+ output.putInt32(value);
1460+ }
1461+ catch (InterruptedException ex)
1462+ {
1463+ output.putNull();
1464+ throw new AkibanInternalException("InteruptedException " + ex);
1465+ }
1466+ }
1467+
1468+ @Override
1469+ public String displayName()
1470+ {
1471+ return "sleep";
1472+ }
1473+
1474+ @Override
1475+ public TOverloadResult resultType()
1476+ {
1477+ return TOverloadResult.fixed(MNumeric.INT);
1478+ }
1479+
1480+}
1481
1482=== modified file 'src/main/java/com/akiban/sql/aisddl/AISDDL.java'
1483--- src/main/java/com/akiban/sql/aisddl/AISDDL.java 2013-03-22 20:05:57 +0000
1484+++ src/main/java/com/akiban/sql/aisddl/AISDDL.java 2013-04-15 17:23:26 +0000
1485@@ -84,7 +84,7 @@
1486 server.getBinderContext(), context);
1487 return;
1488 case NodeTypes.CREATE_INDEX_NODE:
1489- IndexDDL.createIndex(ddlFunctions, session, schema, (CreateIndexNode)ddl);
1490+ IndexDDL.createIndex(ddlFunctions, session, schema, (CreateIndexNode)ddl, server.getServiceManager());
1491 return;
1492 case NodeTypes.DROP_INDEX_NODE:
1493 IndexDDL.dropIndex(ddlFunctions, session, schema, (DropIndexNode)ddl, context);
1494
1495=== modified file 'src/main/java/com/akiban/sql/aisddl/AlterTableDDL.java'
1496--- src/main/java/com/akiban/sql/aisddl/AlterTableDDL.java 2013-03-22 20:05:57 +0000
1497+++ src/main/java/com/akiban/sql/aisddl/AlterTableDDL.java 2013-04-15 17:23:26 +0000
1498@@ -254,7 +254,7 @@
1499 TableName newName = tableCopy.getName();
1500 for(ConstraintDefinitionNode cdn : conDefNodes) {
1501 assert cdn.getConstraintType() != ConstraintType.DROP : cdn;
1502- String name = TableDDL.addIndex(indexNamer, builder, cdn, newName.getSchemaName(), newName.getTableName());
1503+ String name = TableDDL.addIndex(indexNamer, builder, cdn, newName.getSchemaName(), newName.getTableName(), context);
1504 indexChanges.add(TableChange.createAdd(name));
1505 }
1506
1507
1508=== modified file 'src/main/java/com/akiban/sql/aisddl/IndexDDL.java'
1509--- src/main/java/com/akiban/sql/aisddl/IndexDDL.java 2013-03-22 20:05:57 +0000
1510+++ src/main/java/com/akiban/sql/aisddl/IndexDDL.java 2013-04-15 17:23:26 +0000
1511@@ -36,6 +36,8 @@
1512 import com.akiban.server.service.session.Session;
1513
1514 import com.akiban.qp.operator.QueryContext;
1515+import com.akiban.server.service.ServiceManager;
1516+import com.akiban.server.service.text.FullTextIndexService;
1517
1518 /** DDL operations on Indices */
1519 public class IndexDDL
1520@@ -87,7 +89,7 @@
1521 }
1522 // if we can't find the index, set tableName to null
1523 // to flag not a user table index.
1524- if (table.getIndex(indexName) == null) {
1525+ if (table.getIndex(indexName) == null && table.getFullTextIndex(indexName) == null) {
1526 tableName = null;
1527 }
1528 // Check the group associated to the table for the
1529@@ -148,17 +150,18 @@
1530 public static void createIndex(DDLFunctions ddlFunctions,
1531 Session session,
1532 String defaultSchemaName,
1533- CreateIndexNode createIndex) {
1534+ CreateIndexNode createIndex,
1535+ ServiceManager sm) {
1536 AkibanInformationSchema ais = ddlFunctions.getAIS(session);
1537
1538 Collection<Index> indexesToAdd = new LinkedList<>();
1539
1540- indexesToAdd.add(buildIndex(ais, defaultSchemaName, createIndex));
1541+ indexesToAdd.add(buildIndex(ais, defaultSchemaName, createIndex, sm));
1542
1543 ddlFunctions.createIndexes(session, indexesToAdd);
1544 }
1545
1546- protected static Index buildIndex (AkibanInformationSchema ais, String defaultSchemaName, CreateIndexNode index) {
1547+ protected static Index buildIndex (AkibanInformationSchema ais, String defaultSchemaName, CreateIndexNode index, ServiceManager sm) {
1548 final String schemaName = index.getObjectName().getSchemaName() != null ? index.getObjectName().getSchemaName() : defaultSchemaName;
1549 final String indexName = index.getObjectName().getTableName();
1550
1551@@ -173,7 +176,7 @@
1552
1553 if (index.getColumnList().functionType() == IndexColumnList.FunctionType.FULL_TEXT) {
1554 logger.debug ("Building Full text index on table {}", tableName) ;
1555- tableIndex = buildFullTextIndex (builder, tableName, indexName, index);
1556+ tableIndex = buildFullTextIndex (builder, tableName, indexName, index, sm);
1557 } else if (checkIndexType (index, tableName) == Index.IndexType.TABLE) {
1558 logger.debug ("Building Table index on table {}", tableName) ;
1559 tableIndex = buildTableIndex (builder, tableName, indexName, index);
1560@@ -328,9 +331,10 @@
1561 return builder.akibanInformationSchema().getGroup(groupName).getIndex(indexName);
1562 }
1563
1564- protected static Index buildFullTextIndex (AISBuilder builder, TableName tableName, String indexName, IndexDefinition index) {
1565+ protected static Index buildFullTextIndex (AISBuilder builder, TableName tableName, String indexName, IndexDefinition index, ServiceManager sm) {
1566 UserTable table = builder.akibanInformationSchema().getUserTable(tableName);
1567-
1568+ FullTextIndexService fullTextService = sm.getServiceByClass(FullTextIndexService.class);
1569+
1570 if (index.getJoinType() != null) {
1571 throw new TableIndexJoinTypeException();
1572 }
1573@@ -363,6 +367,8 @@
1574 }
1575
1576 builder.fullTextIndexColumn(tableName, indexName, schemaName, columnTable.getTableName(), columnName, i);
1577+ // populate index
1578+ fullTextService.schedulePopulate(schemaName, tableName.getTableName(), indexName);
1579 i++;
1580 }
1581 return builder.akibanInformationSchema().getUserTable(tableName).getFullTextIndex(indexName);
1582
1583=== modified file 'src/main/java/com/akiban/sql/aisddl/TableDDL.java'
1584--- src/main/java/com/akiban/sql/aisddl/TableDDL.java 2013-04-11 05:21:42 +0000
1585+++ src/main/java/com/akiban/sql/aisddl/TableDDL.java 2013-04-15 17:23:26 +0000
1586@@ -195,7 +195,7 @@
1587 }
1588 }
1589 else if (tableElement instanceof ConstraintDefinitionNode) {
1590- addIndex (namer, builder, (ConstraintDefinitionNode)tableElement, schemaName, tableName);
1591+ addIndex (namer, builder, (ConstraintDefinitionNode)tableElement, schemaName, tableName, context);
1592 }
1593 }
1594 builder.basicSchemaIsComplete();
1595@@ -312,7 +312,7 @@
1596
1597
1598 public static String addIndex(IndexNameGenerator namer, AISBuilder builder, ConstraintDefinitionNode cdn,
1599- String schemaName, String tableName) {
1600+ String schemaName, String tableName, QueryContext context) {
1601 // We don't (yet) have a constraint representation so override any provided
1602 UserTable table = builder.akibanInformationSchema().getUserTable(schemaName, tableName);
1603 final String constraint;
1604@@ -330,7 +330,7 @@
1605 }
1606 // Indexes do things a little differently because they need to support Group indexes, Full Text and Geospacial
1607 else if (cdn.getConstraintType() == ConstraintDefinitionNode.ConstraintType.INDEX) {
1608- return generateTableIndex(namer, builder, cdn, table);
1609+ return generateTableIndex(namer, builder, cdn, table, context);
1610 } else {
1611 throw new UnsupportedCheckConstraintException ();
1612 }
1613@@ -474,7 +474,8 @@
1614 private static String generateTableIndex(IndexNameGenerator namer,
1615 AISBuilder builder,
1616 ConstraintDefinitionNode cdn,
1617- Table table) {
1618+ Table table,
1619+ QueryContext context) {
1620 IndexDefinition id = ((IndexConstraintDefinitionNode)cdn);
1621 IndexColumnList columnList = id.getIndexColumnList();
1622 Index tableIndex;
1623@@ -485,7 +486,7 @@
1624
1625 if (columnList.functionType() == IndexColumnList.FunctionType.FULL_TEXT) {
1626 logger.debug ("Building Full text index on table {}", table.getName()) ;
1627- tableIndex = IndexDDL.buildFullTextIndex (builder, table.getName(), indexName, id);
1628+ tableIndex = IndexDDL.buildFullTextIndex (builder, table.getName(), indexName, id, context.getServiceManager());
1629 } else if (IndexDDL.checkIndexType (id, table.getName()) == Index.IndexType.TABLE) {
1630 logger.debug ("Building Table index on table {}", table.getName()) ;
1631 tableIndex = IndexDDL.buildTableIndex (builder, table.getName(), indexName, id);
1632
1633=== modified file 'src/main/resources/com/akiban/server/service/config/configuration-defaults.properties'
1634--- src/main/resources/com/akiban/server/service/config/configuration-defaults.properties 2013-04-06 05:07:23 +0000
1635+++ src/main/resources/com/akiban/server/service/config/configuration-defaults.properties 2013-04-15 17:23:26 +0000
1636@@ -36,6 +36,9 @@
1637 akserver.bulkload.bufferalloc.pk=0.17
1638 akserver.bulkload.bufferalloc.group=0.33
1639 akserver.restrict_user_schema=false
1640+akserver.text.indexpath=/tmp/aktext
1641+akserver.text.maintenanceInterval=3000
1642+akserver.text.populateDelayInterval=1000
1643 # EXPERIMENTAL
1644 akserver.indexRowPooling = false
1645 akserver.pt.osc.hook=disabled
1646
1647=== modified file 'src/test/java/com/akiban/server/rowdata/SchemaFactory.java'
1648--- src/test/java/com/akiban/server/rowdata/SchemaFactory.java 2013-04-11 18:35:20 +0000
1649+++ src/test/java/com/akiban/server/rowdata/SchemaFactory.java 2013-04-15 17:23:26 +0000
1650@@ -31,6 +31,7 @@
1651 import com.akiban.server.api.DDLFunctions;
1652 import com.akiban.server.api.ddl.DDLFunctionsMockBase;
1653 import com.akiban.server.error.PersistitAdapterException;
1654+import com.akiban.server.service.ServiceManager;
1655 import com.akiban.server.service.routines.MockRoutineLoader;
1656 import com.akiban.server.service.session.Session;
1657 import com.akiban.sql.StandardException;
1658@@ -76,7 +77,12 @@
1659 }
1660
1661 public AkibanInformationSchema ais(String... ddl) {
1662- return ais(new AkibanInformationSchema(), ddl);
1663+ return ais((ServiceManager)null, ddl);
1664+ }
1665+
1666+ public AkibanInformationSchema ais(ServiceManager sm, String... ddl)
1667+ {
1668+ return ais(sm, new AkibanInformationSchema(), ddl);
1669 }
1670
1671 public static AkibanInformationSchema loadAIS(File fromFile, String defaultSchema) {
1672@@ -89,11 +95,20 @@
1673 }
1674 }
1675
1676+ public AkibanInformationSchema ais(ServiceManager sm, AkibanInformationSchema baseAIS, String... ddl)
1677+ {
1678+ return ais(sm, new CreateOnlyDDLMock(baseAIS), null, ddl);
1679+ }
1680 public AkibanInformationSchema ais(AkibanInformationSchema baseAIS, String... ddl) {
1681 return ais(new CreateOnlyDDLMock(baseAIS), null, ddl);
1682 }
1683
1684- public AkibanInformationSchema ais(DDLFunctions ddlFunctions, Session session, String... ddl) {
1685+ public AkibanInformationSchema ais(DDLFunctions ddlFunctions, Session session, String... ddl)
1686+ {
1687+ return ais(null, ddlFunctions, session, ddl);
1688+ }
1689+
1690+ public AkibanInformationSchema ais(ServiceManager sm, DDLFunctions ddlFunctions, Session session, String... ddl) {
1691 StringBuilder buffer = new StringBuilder();
1692 for (String line : ddl) {
1693 buffer.append(line);
1694@@ -110,7 +125,7 @@
1695 if (stmt instanceof CreateTableNode) {
1696 TableDDL.createTable(ddlFunctions , session , defaultSchema, (CreateTableNode) stmt, null);
1697 } else if (stmt instanceof CreateIndexNode) {
1698- IndexDDL.createIndex(ddlFunctions, session, defaultSchema, (CreateIndexNode) stmt);
1699+ IndexDDL.createIndex(ddlFunctions, session, defaultSchema, (CreateIndexNode) stmt, sm);
1700 } else if (stmt instanceof CreateViewNode) {
1701 ViewDDL.createView(ddlFunctions, session, defaultSchema, (CreateViewNode) stmt,
1702 new AISBinderContext(ddlFunctions.getAIS(session), defaultSchema), null);
1703
1704=== modified file 'src/test/java/com/akiban/server/service/text/FullTextIndexServiceIT.java'
1705--- src/test/java/com/akiban/server/service/text/FullTextIndexServiceIT.java 2013-03-22 20:05:57 +0000
1706+++ src/test/java/com/akiban/server/service/text/FullTextIndexServiceIT.java 2013-04-15 17:23:26 +0000
1707@@ -18,6 +18,7 @@
1708 package com.akiban.server.service.text;
1709
1710 import com.akiban.ais.model.FullTextIndex;
1711+import com.akiban.ais.model.IndexName;
1712 import com.akiban.qp.operator.Operator;
1713 import com.akiban.qp.operator.QueryContext;
1714 import static com.akiban.qp.operator.API.cursor;
1715@@ -27,9 +28,13 @@
1716 import com.akiban.qp.rowtype.Schema;
1717 import com.akiban.qp.util.SchemaCache;
1718 import com.akiban.server.service.servicemanager.GuicedServiceManager;
1719+import com.akiban.server.service.session.Session;
1720+import com.akiban.server.service.session.SessionServiceImpl;
1721 import com.akiban.server.test.it.ITBase;
1722 import com.akiban.server.test.it.qp.TestRow;
1723
1724+import com.persistit.Exchange;
1725+import com.persistit.exception.PersistitException;
1726 import org.junit.Before;
1727 import org.junit.Test;
1728 import static org.junit.Assert.*;
1729@@ -39,12 +44,18 @@
1730 public class FullTextIndexServiceIT extends ITBase
1731 {
1732 public static final String SCHEMA = "test";
1733-
1734+ private static final long updateInterval = 1000L; // for testing
1735+ private static final long populateDelayInterval = 1000L;
1736 protected FullTextIndexService fullText;
1737 protected Schema schema;
1738 protected PersistitAdapter adapter;
1739 protected QueryContext queryContext;
1740
1741+ private int c;
1742+ private int o;
1743+ private int i;
1744+ private int a;
1745+
1746 @Override
1747 protected GuicedServiceManager.BindingsConfigurationProvider serviceBindingsProvider() {
1748 return super.serviceBindingsProvider()
1749@@ -55,29 +66,31 @@
1750 protected Map<String, String> startupConfigProperties() {
1751 Map<String, String> properties = new HashMap<>();
1752 properties.put("akserver.text.indexpath", "/tmp/aktext");
1753+ properties.put(FullTextIndexServiceImpl.UPDATE_INTERVAL, Long.toString(updateInterval));
1754+ properties.put(FullTextIndexServiceImpl.POPULATE_DELAY_INTERVAL, Long.toString(populateDelayInterval));
1755 return properties;
1756 }
1757
1758 @Before
1759 public void createData() {
1760- int c = createTable(SCHEMA, "c",
1761- "cid INT PRIMARY KEY NOT NULL",
1762- "name VARCHAR(128) COLLATE en_us_ci");
1763- int o = createTable(SCHEMA, "o",
1764- "oid INT PRIMARY KEY NOT NULL",
1765- "cid INT NOT NULL",
1766- "GROUPING FOREIGN KEY(cid) REFERENCES c(cid)",
1767- "order_date DATE");
1768- int i = createTable(SCHEMA, "i",
1769- "iid INT PRIMARY KEY NOT NULL",
1770- "oid INT NOT NULL",
1771- "GROUPING FOREIGN KEY(oid) REFERENCES o(oid)",
1772- "sku VARCHAR(10) NOT NULL");
1773- int a = createTable(SCHEMA, "a",
1774- "aid INT PRIMARY KEY NOT NULL",
1775- "cid INT NOT NULL",
1776- "GROUPING FOREIGN KEY(cid) REFERENCES c(cid)",
1777- "state CHAR(2)");
1778+ c = createTable(SCHEMA, "c",
1779+ "cid INT PRIMARY KEY NOT NULL",
1780+ "name VARCHAR(128) COLLATE en_us_ci");
1781+ o = createTable(SCHEMA, "o",
1782+ "oid INT PRIMARY KEY NOT NULL",
1783+ "cid INT NOT NULL",
1784+ "GROUPING FOREIGN KEY(cid) REFERENCES c(cid)",
1785+ "order_date DATE");
1786+ i = createTable(SCHEMA, "i",
1787+ "iid INT PRIMARY KEY NOT NULL",
1788+ "oid INT NOT NULL",
1789+ "GROUPING FOREIGN KEY(oid) REFERENCES o(oid)",
1790+ "sku VARCHAR(10) NOT NULL");
1791+ a = createTable(SCHEMA, "a",
1792+ "aid INT PRIMARY KEY NOT NULL",
1793+ "cid INT NOT NULL",
1794+ "GROUPING FOREIGN KEY(cid) REFERENCES c(cid)",
1795+ "state CHAR(2)");
1796 writeRow(c, 1, "Fred Flintstone");
1797 writeRow(o, 101, 1, "2012-12-12");
1798 writeRow(i, 10101, 101, "ABCD");
1799@@ -99,11 +112,286 @@
1800 }
1801
1802 @Test
1803- public void cDown() {
1804- FullTextIndex index = createFullTextIndex(SCHEMA, "c", "idx_c",
1805- "name", "i.sku", "a.state");
1806- fullText.createIndex(session(), index.getIndexName());
1807-
1808+ public void testPopulateScheduling() throws InterruptedException, PersistitException
1809+ {
1810+ // This test is specifically for FullTextIndexServiceImpl.java
1811+ assertEquals(FullTextIndexServiceImpl.class, fullText.getClass());
1812+ FullTextIndexServiceImpl fullTextImpl = (FullTextIndexServiceImpl)fullText;
1813+
1814+ // disable the populate worker (so it doesn't read all the entries
1815+ // out before we get a chance to look at the tree.
1816+ fullTextImpl.disablePopulateWorker();
1817+
1818+ // create 3 indices
1819+ final FullTextIndex expecteds[] = new FullTextIndex[]
1820+ {
1821+ createFullTextIndex(serviceManager(),
1822+ SCHEMA, "c", "idx1_c",
1823+ "name"),
1824+
1825+ createFullTextIndex(serviceManager(),
1826+ SCHEMA, "c", "idx2_c",
1827+ "i.sku"),
1828+ createFullTextIndex(serviceManager(),
1829+ SCHEMA, "c", "idx3_c",
1830+ "name", "i.sku")
1831+ };
1832+
1833+
1834+ // read the entries out
1835+ traverse(fullTextImpl,
1836+ new Visitor()
1837+ {
1838+ int n = 0;
1839+ public void visit(IndexName idx)
1840+ {
1841+ assertEquals("entry[" + n + "]", expecteds[n++].getIndexName(),
1842+ idx);
1843+ }
1844+
1845+ public void endOfTree()
1846+ {
1847+ assertEquals(expecteds.length, n);
1848+ }
1849+ });
1850+
1851+
1852+ // let the worker do its job.
1853+ // (After it is done, the tree had better be empty)
1854+ fullTextImpl.enablePopulateWorker();
1855+ Thread.sleep(populateDelayInterval * 5);
1856+
1857+ traverse(fullTextImpl,
1858+ new Visitor()
1859+ {
1860+ int n = 0;
1861+
1862+ @Override
1863+ public void visit(IndexName idx)
1864+ {
1865+ ++n;
1866+ }
1867+
1868+ @Override
1869+ public void endOfTree()
1870+ {
1871+ assertEquals (0, n);
1872+ }
1873+ });
1874+
1875+ // drop all the indices
1876+ Session session = new SessionServiceImpl().createSession();
1877+ for (FullTextIndex idx : expecteds)
1878+ fullTextImpl.dropIndex(session, idx.getIndexName());
1879+ session.close();
1880+ }
1881+
1882+
1883+ @Test
1884+ public void testDeleteIndex() throws PersistitException, InterruptedException
1885+ {
1886+
1887+ // This test is specifically for FullTextIndexServiceImpl.java
1888+ assertEquals(FullTextIndexServiceImpl.class, fullText.getClass());
1889+ FullTextIndexServiceImpl fullTextImpl = (FullTextIndexServiceImpl)fullText;
1890+
1891+ // <1> disable worker
1892+ fullTextImpl.disablePopulateWorker();
1893+
1894+ // <2> create 3 new indices
1895+ // create 3 indices
1896+ final FullTextIndex expecteds[] = new FullTextIndex[]
1897+ {
1898+ createFullTextIndex(serviceManager(),
1899+ SCHEMA, "c", "idx4_c",
1900+ "a.state"),
1901+
1902+ createFullTextIndex(serviceManager(),
1903+ SCHEMA, "c", "idx5_c",
1904+ "i.sku", "a.state"),
1905+ createFullTextIndex(serviceManager(),
1906+ SCHEMA, "c", "idx6_c",
1907+ "name", "i.sku")
1908+ };
1909+
1910+ // <3> delete 2 of them
1911+ Session session = new SessionServiceImpl().createSession();
1912+ boolean sawNPE = false;
1913+ try
1914+ {
1915+ fullTextImpl.dropIndex(session, expecteds[0].getIndexName());
1916+ }
1917+ catch (NullPointerException e) // NPE is expected because, the index hasn't been
1918+ { // populated yet. But we're not testing that!
1919+ sawNPE = true;
1920+ }
1921+ assertTrue("NPE should have happened", sawNPE);
1922+
1923+
1924+ sawNPE = false;
1925+ try
1926+ {
1927+ fullTextImpl.dropIndex(session, expecteds[1].getIndexName());
1928+ }
1929+ catch (NullPointerException e) // NPE is expected because, the index hasn't been
1930+ { // populated yet. But we're not testing that!
1931+ sawNPE = true;
1932+ }
1933+ assertTrue("NPE should have happened", sawNPE);
1934+
1935+ // <4> check that the tree only has one entry now (ie., epxecteds2[2]
1936+ traverse(fullTextImpl,
1937+ new Visitor()
1938+ {
1939+ int n = 0;
1940+
1941+ @Override
1942+ public void visit(IndexName idx)
1943+ {
1944+ assertEquals("entry[" + n++ + "]", expecteds[2].getIndexName(),
1945+ idx);
1946+ }
1947+
1948+ @Override
1949+ public void endOfTree()
1950+ {
1951+ assertEquals (1, n);
1952+ }
1953+ });
1954+
1955+ // wake the worker up to do its job
1956+ fullTextImpl.enablePopulateWorker();
1957+ Thread.sleep(populateDelayInterval * 5);
1958+
1959+ // drop the remaining index
1960+ fullTextImpl.dropIndex(session, expecteds[2].getIndexName());
1961+ session.close();
1962+ }
1963+
1964+ private static interface Visitor
1965+ {
1966+ void visit(IndexName idx);
1967+ void endOfTree();
1968+ }
1969+
1970+
1971+ private static void traverse(FullTextIndexServiceImpl serv,
1972+ Visitor visitor) throws PersistitException
1973+ {
1974+ Session session = new SessionServiceImpl().createSession();
1975+
1976+ try
1977+ {
1978+ Exchange ex = serv.getPopulateExchange(session);
1979+ IndexName toPopulate;
1980+ while ((toPopulate = serv.nextInQueue(ex)) != null)
1981+ visitor.visit(toPopulate);
1982+ visitor.endOfTree();
1983+ }
1984+ finally
1985+ {
1986+ session.close();
1987+ }
1988+ }
1989+
1990+ @Test
1991+ public void testUpdate() throws InterruptedException
1992+ {
1993+
1994+ /*
1995+ Test plans:
1996+ 1 - do a search, confirm that the rows come back as expected
1997+ 2 - (With update worker enabled)
1998+ + insert new rows
1999+ + sleep for sometime
2000+ + do the search again and confirm that the new rows are found
2001+ 3 - (With update worker NOT enable)
2002+ + disable update worker
2003+ + insert new rows
2004+ + do the search again and confirm that the new rows are NOT found
2005+
2006+ */
2007+
2008+ //CREATE INDEX cust_ft ON customers(FULL_TEXT(name, addresses.state, items.sku))
2009+
2010+ // part 1
2011+ FullTextIndex index = createFullTextIndex(serviceManager(),
2012+ SCHEMA, "c", "idx_c",
2013+ "name", "i.sku", "a.state");
2014+ //fullText.createIndex(session(), index.getIndexName());
2015+ Thread.sleep(populateDelayInterval * 5);
2016+ RowType rowType = rowType("c");
2017+ RowBase[] expected1 = new RowBase[]
2018+ {
2019+ row(rowType, 1L),
2020+ row(rowType, 3L)
2021+ };
2022+ FullTextQueryBuilder builder = new FullTextQueryBuilder(index, ais(), queryContext);
2023+ Operator plan = builder.scanOperator("flintstone", 15);
2024+ compareRows(expected1, cursor(plan, queryContext));
2025+
2026+ plan = builder.scanOperator("state:MA", 15);
2027+ compareRows(expected1, cursor(plan, queryContext));
2028+
2029+
2030+ // part 2
2031+ // write new rows
2032+ writeRow(c, 4, "John Watson");
2033+ writeRow(c, 5, "Sherlock Flintstone");
2034+ writeRow(c, 6, "Mycroft Holmes");
2035+ writeRow(c, 7, "Flintstone Lestrade");
2036+
2037+ // sleep a bit (wait for the worker to do the update)
2038+ Thread.sleep(updateInterval * 5);
2039+ RowBase expected2[] = new RowBase[]
2040+ {
2041+ row(rowType, 1L),
2042+ row(rowType, 3L),
2043+ row(rowType, 5L),
2044+ row(rowType, 7L)
2045+ };
2046+
2047+ // confirm new changes
2048+ plan = builder.scanOperator("flintstone", 15);
2049+ compareRows(expected2, cursor(plan, queryContext));
2050+
2051+ // part 3
2052+ ((FullTextIndexServiceImpl)fullText).disableUpdateWorker();
2053+
2054+ writeRow(c, 8, "Flintstone Hudson");
2055+ writeRow(c, 9, "Jim Flintstone");
2056+
2057+ Thread.sleep(updateInterval * 2);
2058+
2059+ // confirm that new rows are not found (ie., expected2 still works)
2060+ plan = builder.scanOperator("flintstone", 15);
2061+ compareRows(expected2, cursor(plan, queryContext));
2062+
2063+ ((FullTextIndexServiceImpl)fullText).enableUpdateWorker();
2064+ Thread.sleep(updateInterval * 5);
2065+
2066+ // now the rows should be seen.
2067+ // (Because disabling the worker does not stop the changes fron being recorded)
2068+ RowBase expected3[] = new RowBase[]
2069+ {
2070+ row(rowType, 1L),
2071+ row(rowType, 3L),
2072+ row(rowType, 5L),
2073+ row(rowType, 7L),
2074+ row(rowType, 8L),
2075+ row(rowType, 9L)
2076+ };
2077+
2078+ plan = builder.scanOperator("flintstone", 15);
2079+ compareRows(expected3, cursor(plan, queryContext));
2080+ }
2081+
2082+ @Test
2083+ public void cDown() throws InterruptedException {
2084+ FullTextIndex index = createFullTextIndex(serviceManager(),
2085+ SCHEMA, "c", "idx_c",
2086+ "name", "i.sku", "a.state");
2087+ Thread.sleep(populateDelayInterval * 5);
2088 RowType rowType = rowType("c");
2089 RowBase[] expected = new RowBase[] {
2090 row(rowType, 1L),
2091@@ -115,14 +403,15 @@
2092
2093 plan = builder.scanOperator("state:MA", 10);
2094 compareRows(expected, cursor(plan, queryContext));
2095+
2096 }
2097
2098 @Test
2099- public void oUpDown() {
2100- FullTextIndex index = createFullTextIndex(SCHEMA, "o", "idx_o",
2101+ public void oUpDown() throws InterruptedException {
2102+ FullTextIndex index = createFullTextIndex(serviceManager(),
2103+ SCHEMA, "o", "idx_o",
2104 "c.name", "i.sku");
2105- fullText.createIndex(session(), index.getIndexName());
2106-
2107+ Thread.sleep(populateDelayInterval * 5);
2108 RowType rowType = rowType("o");
2109 RowBase[] expected = new RowBase[] {
2110 row(rowType, 1L, 101L)
2111
2112=== modified file 'src/test/java/com/akiban/server/test/ApiTestBase.java'
2113--- src/test/java/com/akiban/server/test/ApiTestBase.java 2013-03-22 20:05:57 +0000
2114+++ src/test/java/com/akiban/server/test/ApiTestBase.java 2013-04-15 17:23:26 +0000
2115@@ -520,16 +520,16 @@
2116 * overriding the {@link #startupConfigProperties()} and/or
2117 * {@link #serviceBindingsProvider()} methods.
2118 */
2119- protected static Map<String, String> uniqueStartupConfigProperties(Class clazz) {
2120+ protected Map<String, String> uniqueStartupConfigProperties(Class clazz) {
2121 return Collections.singletonMap(
2122 "test.services",
2123 clazz.getName()
2124 );
2125 }
2126
2127- protected AkibanInformationSchema createFromDDL(String schema, String ddl) {
2128+ protected AkibanInformationSchema createFromDDL(ServiceManager sm, String schema, String ddl) {
2129 SchemaFactory schemaFactory = new SchemaFactory(schema);
2130- return schemaFactory.ais(ddl().getAIS(session()), ddl);
2131+ return schemaFactory.ais(sm, ddl().getAIS(session()), ddl);
2132 }
2133
2134 protected static final class SimpleColumn {
2135@@ -602,7 +602,7 @@
2136
2137 protected final int createTable(String schema, String table, String definition) throws InvalidOperationException {
2138 String ddl = String.format("CREATE TABLE \"%s\" (%s)", table, definition);
2139- AkibanInformationSchema tempAIS = createFromDDL(schema, ddl);
2140+ AkibanInformationSchema tempAIS = createFromDDL(serviceManager(), schema, ddl);
2141 UserTable tempTable = tempAIS.getUserTable(schema, table);
2142 ddl().createTable(session(), tempTable);
2143 updateAISGeneration();
2144@@ -621,7 +621,7 @@
2145
2146 protected final void createSequence (String schema, String name, String definition) {
2147 String ddl = String.format("CREATE SEQUENCE %s %s", name, definition);
2148- AkibanInformationSchema tempAIS = createFromDDL(schema, ddl);
2149+ AkibanInformationSchema tempAIS = createFromDDL(serviceManager(), schema, ddl);
2150 Sequence sequence = tempAIS.getSequence(new TableName(schema, name));
2151 ddl().createSequence(session(), sequence);
2152 updateAISGeneration();
2153@@ -629,7 +629,7 @@
2154
2155 protected final void createView(String schema, String name, String definition) {
2156 String ddl = String.format("CREATE VIEW %s AS %s", name, definition);
2157- AkibanInformationSchema tempAIS = createFromDDL(schema, ddl);
2158+ AkibanInformationSchema tempAIS = createFromDDL(serviceManager(), schema, ddl);
2159 View view = tempAIS.getView(new TableName(schema, name));
2160 ddl().createView(session(), view);
2161 updateAISGeneration();
2162@@ -664,7 +664,7 @@
2163 schema,
2164 table,
2165 Strings.join(Arrays.asList(indexCols), ","));
2166- return createFromDDL(schema, ddl);
2167+ return createFromDDL(serviceManager(), schema, ddl);
2168 }
2169
2170 protected final TableIndex createIndex(String schema, String table, String indexName, String... indexCols) {
2171@@ -767,7 +767,7 @@
2172 return ddl().getAIS(session()).getGroup(groupName).getIndex(indexName);
2173 }
2174
2175- protected final FullTextIndex createFullTextIndex(String schema, String table, String indexName, String... indexCols) {
2176+ protected final FullTextIndex createFullTextIndex(ServiceManager sm, String schema, String table, String indexName, String... indexCols) {
2177 AkibanInformationSchema tempAIS = createIndexInternal(schema, table, indexName, "FULL_TEXT(" + Strings.join(Arrays.asList(indexCols), ",") + ")");
2178 Index tempIndex = tempAIS.getUserTable(schema, table).getFullTextIndex(indexName);
2179 ddl().createIndexes(session(), Collections.singleton(tempIndex));
2180@@ -779,10 +779,10 @@
2181 SchemaFactory schemaFactory = new SchemaFactory(schema);
2182
2183 // Insert DDL into the System
2184- AkibanInformationSchema ais = schemaFactory.ais(ddl(), session(), ddl);
2185+ AkibanInformationSchema ais = schemaFactory.ais(serviceManager(), ddl(), session(), ddl);
2186
2187 // sort DDL to find first root table of the user schema
2188- ais = schemaFactory.ais (ddl);
2189+ ais = schemaFactory.ais (serviceManager(), ddl);
2190 List<UserTable> tables = new ArrayList<>(ais.getUserTables().values());
2191 Collections.sort(tables, new Comparator<UserTable>() {
2192 @Override
2193
2194=== modified file 'src/test/java/com/akiban/server/test/it/bugs/bug701614/MissingColumnsIT.java'
2195--- src/test/java/com/akiban/server/test/it/bugs/bug701614/MissingColumnsIT.java 2013-03-22 20:05:57 +0000
2196+++ src/test/java/com/akiban/server/test/it/bugs/bug701614/MissingColumnsIT.java 2013-04-15 17:23:26 +0000
2197@@ -38,7 +38,7 @@
2198
2199 private int loadBlocksTable() throws InvalidOperationException, IOException {
2200 final String blocksDDL = Strings.join(Strings.dumpResource(getClass(), "blocks-table.ddl"));
2201- AkibanInformationSchema tempAIS = createFromDDL("drupal", blocksDDL);
2202+ AkibanInformationSchema tempAIS = createFromDDL(serviceManager(), "drupal", blocksDDL);
2203 ddl().createTable(session(), tempAIS.getUserTable("drupal", "blocks"));
2204 updateAISGeneration();
2205 AkibanInformationSchema ais = ddl().getAIS(session());
2206
2207=== modified file 'src/test/java/com/akiban/sql/optimizer/OptimizerTestBase.java'
2208--- src/test/java/com/akiban/sql/optimizer/OptimizerTestBase.java 2013-03-22 20:05:57 +0000
2209+++ src/test/java/com/akiban/sql/optimizer/OptimizerTestBase.java 2013-04-15 17:23:26 +0000
2210@@ -25,18 +25,26 @@
2211 import com.akiban.sql.views.ViewDefinition;
2212
2213 import com.akiban.ais.model.AkibanInformationSchema;
2214+import com.akiban.server.service.ServiceManager;
2215+import com.akiban.server.service.config.TestConfigService;
2216 import com.akiban.server.service.functions.FunctionsRegistryImpl;
2217
2218+import com.akiban.server.service.servicemanager.GuicedServiceManager;
2219+import com.akiban.server.service.text.FullTextIndexService;
2220+import com.akiban.server.service.text.FullTextIndexServiceImpl;
2221 import org.junit.Before;
2222 import org.junit.Ignore;
2223
2224 import java.io.File;
2225 import java.util.Collections;
2226+import java.util.HashMap;
2227 import java.util.List;
2228+import java.util.Map;
2229
2230 @Ignore
2231 public class OptimizerTestBase extends ASTTransformTestBase
2232 {
2233+
2234 protected OptimizerTestBase(String caseName, String sql,
2235 String expected, String error) {
2236 super(caseName, sql, expected, error);
2237@@ -47,13 +55,17 @@
2238 + OptimizerTestBase.class.getPackage().getName().replace('.', '/'));
2239 public static final String DEFAULT_SCHEMA = "test";
2240
2241+ protected static ServiceManager sm = prepareServices();
2242+ protected static int populateDelayInterval = 1000;
2243+ protected static int updateInterval = 3000;
2244+
2245 // Base class has all possible transformers for convenience.
2246 protected AISBinder binder;
2247 protected TypeComputer typeComputer;
2248 protected BooleanNormalizer booleanNormalizer;
2249 protected SubqueryFlattener subqueryFlattener;
2250 protected DistinctEliminator distinctEliminator;
2251-
2252+
2253 @Before
2254 public void makeTransformers() throws Exception {
2255 parser = new SQLParser();
2256@@ -65,13 +77,19 @@
2257 distinctEliminator = new DistinctEliminator(parser);
2258 }
2259
2260+ protected static ServiceManager serviceManager()
2261+ {
2262+ return sm;
2263+ }
2264+
2265 public static AkibanInformationSchema parseSchema(List<File> ddls) throws Exception {
2266 StringBuilder ddlBuilder = new StringBuilder();
2267 for (File ddl : ddls)
2268 ddlBuilder.append(fileContents(ddl));
2269 String sql = ddlBuilder.toString();
2270 SchemaFactory schemaFactory = new SchemaFactory(DEFAULT_SCHEMA);
2271- return schemaFactory.ais(sql);
2272+ AkibanInformationSchema ret = schemaFactory.ais(serviceManager(), sql);
2273+ return ret;
2274 }
2275
2276 public static AkibanInformationSchema parseSchema(File ddl) throws Exception {
2277@@ -98,4 +116,35 @@
2278 }
2279 }
2280
2281+
2282+ protected static ServiceManager createServiceManager(Map<String, String> startupConfigProperties)
2283+ {
2284+ return new GuicedServiceManager(serviceBindingsProvider());
2285+ }
2286+
2287+ protected static GuicedServiceManager.BindingsConfigurationProvider serviceBindingsProvider()
2288+ {
2289+ return GuicedServiceManager
2290+ .testUrls()
2291+ .bindAndRequire(FullTextIndexService.class,
2292+ FullTextIndexServiceImpl.class);
2293+ }
2294+
2295+
2296+ protected static Map<String, String> startupConfigProperties() {
2297+ Map<String, String> properties = new HashMap<>();
2298+ properties.put("akserver.text.indexpath", "/tmp/aktext");
2299+ properties.put(FullTextIndexServiceImpl.UPDATE_INTERVAL, Long.toString(updateInterval));
2300+ properties.put(FullTextIndexServiceImpl.POPULATE_DELAY_INTERVAL, Long.toString(populateDelayInterval));
2301+ return properties;
2302+ }
2303+
2304+ private static ServiceManager prepareServices()
2305+ {
2306+ System.setProperty("akiban.home", System.getProperty("user.home"));
2307+ ServiceManager ret = createServiceManager(startupConfigProperties());
2308+ ret.startServices();
2309+
2310+ return ret;
2311+ }
2312 }
2313
2314=== added file 'src/test/resources/com/akiban/sql/pg/yaml/functional/test-drop-fulltext.yaml'
2315--- src/test/resources/com/akiban/sql/pg/yaml/functional/test-drop-fulltext.yaml 1970-01-01 00:00:00 +0000
2316+++ src/test/resources/com/akiban/sql/pg/yaml/functional/test-drop-fulltext.yaml 2013-04-15 17:23:26 +0000
2317@@ -0,0 +1,8 @@
2318+# test drop full_text index
2319+---
2320+- CreateTable: t(id int, name varchar(224));
2321+---
2322+- Statement: CREATE INDEX idx on t(full_text(name));
2323+---
2324+- Statement: DROP INDEX t.idx;
2325+...
2326\ No newline at end of file
2327
2328=== added file 'src/test/resources/com/akiban/sql/pg/yaml/functional/test-ft-with-inherited-key.yaml'
2329--- src/test/resources/com/akiban/sql/pg/yaml/functional/test-ft-with-inherited-key.yaml 1970-01-01 00:00:00 +0000
2330+++ src/test/resources/com/akiban/sql/pg/yaml/functional/test-ft-with-inherited-key.yaml 2013-04-15 17:23:26 +0000
2331@@ -0,0 +1,28 @@
2332+# test full-text updates with inherited keys
2333+---
2334+- CreateTable: t1(id INT NOT NULL PRIMARY KEY);
2335+---
2336+- CreateTable: t2(id INT NOT NULL PRIMARY KEY, pid INT NOT NULL, GROUPING FOREIGN KEY(pid) REFERENCES t1(id));
2337+---
2338+- CreateTable: t3(id INT NOT NULL PRIMARY KEY, pid INT NOT NULL, GROUPING FOREIGN KEY(pid) REFERENCES t2(id), name VARCHAR(16) COLLATE en_ci);
2339+---
2340+- Statement: INSERT INTO t1 VALUES(1),(2);
2341+---
2342+- Statement: INSERT INTO t2 VALUES(11, 1);
2343+---
2344+- Statement: INSERT INTO t3 VALUES(111, 11, 'Fred'),(112, 11, 'Barney');
2345+---
2346+- Statement: CREATE INDEX t3_ft ON t3(FULL_TEXT(name));
2347+---
2348+- Statement: SELECT SLEEP(7000);
2349+---
2350+- Statement: SELECT * FROM t3 WHERE FULL_TEXT_SEARCH(name, 'fred');
2351+- output: [[111, 11, 'Fred']]
2352+---
2353+- Statement: UPDATE t2 SET pid = 2 WHERE pid = 1;
2354+---
2355+- Statement: SELECT SLEEP(7000);
2356+---
2357+- Statement: SELECT * FROM t3 WHERE FULL_TEXT_SEARCH(name, 'fred');
2358+- output: [[111, 11, 'Fred']]
2359+...
2360\ No newline at end of file
2361
2362=== added file 'src/test/resources/com/akiban/sql/pg/yaml/functional/test-ft-with-pk.yaml'
2363--- src/test/resources/com/akiban/sql/pg/yaml/functional/test-ft-with-pk.yaml 1970-01-01 00:00:00 +0000
2364+++ src/test/resources/com/akiban/sql/pg/yaml/functional/test-ft-with-pk.yaml 2013-04-15 17:23:26 +0000
2365@@ -0,0 +1,22 @@
2366+# Tests with changed primary key
2367+---
2368+- CreateTable: t (id int primary key not null, name varchar(224));
2369+---
2370+- Statement: INSERT INTO t values (1, 'foo1');
2371+---
2372+- Statement: INSERT INTO t values (2, 'foo2');
2373+---
2374+- Statement: create index idx1 on t (full_text(name));
2375+---
2376+- Statement: SELECT SLEEP(7000);
2377+---
2378+- Statement: SELECT id FROM t where full_text_search(name = 'foo1');
2379+- output: [[1]]
2380+---
2381+- Statement: UPDATE t SET id = 3 where id = 1;
2382+---
2383+- Statement: SELECT SLEEP(7000);
2384+---
2385+- Statement: SELECT id FROM t where full_text_search(name = 'foo1');
2386+- output: [[3]]
2387+...
2388\ No newline at end of file
2389
2390=== added file 'src/test/resources/com/akiban/sql/pg/yaml/functional/test-fulltext-maintenance.yaml'
2391--- src/test/resources/com/akiban/sql/pg/yaml/functional/test-fulltext-maintenance.yaml 1970-01-01 00:00:00 +0000
2392+++ src/test/resources/com/akiban/sql/pg/yaml/functional/test-fulltext-maintenance.yaml 2013-04-15 17:23:26 +0000
2393@@ -0,0 +1,49 @@
2394+# test that full text indices are updated (and populated) properly
2395+---
2396+- CreateTable: t (id int, name varchar(224));
2397+---
2398+- Statement: INSERT INTO t VALUES (1, 'bar1');
2399+---
2400+- Statement: INSERT INTO t VALUES (2, 'bar1');
2401+---
2402+- Statement: INSERT INTO t VALUES (3, 'bar3');
2403+---
2404+- Statement: INSERT INTO t VALUES (4, 'bar4');
2405+---
2406+- Statement: CREATE INDEX idx ON t (full_text(name));
2407+--- # should be long enough! (Could query the actual interval from AIS, but we'd need to parse the string!)
2408+- Statement: SELECT SLEEP (7000);
2409+---
2410+- Statement: select id from t where full_text_search(name = 'bar1');
2411+- output: [[1], [2]]
2412+---
2413+- Statement: UPDATE t set name='foo' WHERE id = 1;
2414+---
2415+- Statement: SELECT SLEEP(7000);
2416+---
2417+- Statement: select id from t where full_text_search(name = 'bar1');
2418+- output: [[2]]
2419+---
2420+- Statement: SELECT id from t where full_text_search(name = 'bar4');
2421+- output: [[4]]
2422+---
2423+- Statement: DELETE FROM t where id = 4;
2424+---
2425+- Statement: SELECT SLEEP(7000);
2426+---
2427+- Statement: SELECT id from t where full_text_search(name = 'bar4');
2428+- row_count: 0
2429+---
2430+- Create
2431+...
2432+
2433+CREATE TABLE t1(id INT NOT NULL PRIMARY KEY);
2434+CREATE TABLE t2(id INT NOT NULL PRIMARY KEY, pid INT NOT NULL, GROUPING FOREIGN KEY(pid) REFERENCES t1(id));
2435+CREATE TABLE t3(id INT NOT NULL PRIMARY KEY, pid INT NOT NULL, GROUPING FOREIGN KEY(pid) REFERENCES t2(id), name VARCHAR(16) COLLATE en_ci);
2436+INSERT INTO t1 VALUES(1),(2);
2437+INSERT INTO t2 VALUES(11, 1);
2438+INSERT INTO t3 VALUES(111, 11, 'Fred'),(112, 11, 'Barney');
2439+CREATE INDEX t3_ft ON t3(FULL_TEXT(name));
2440+SELECT * FROM t3 WHERE FULL_TEXT_SEARCH(name, 'fred');
2441+UPDATE t2 SET pid = 2 WHERE pid = 1;
2442+SELECT * FROM t3 WHERE FULL_TEXT_SEARCH(name, 'fred');

Subscribers

People subscribed via source and target branches

to all changes: