Merge lp:~oontvoo/akiban-server/bug_drop_fulltext_index into lp:~akiban-technologies/akiban-server/trunk
- bug_drop_fulltext_index
- Merge into trunk
Status: | Superseded |
---|---|
Proposed branch: | lp:~oontvoo/akiban-server/bug_drop_fulltext_index |
Merge into: | lp:~akiban-technologies/akiban-server/trunk |
Diff against target: |
2442 lines (+1511/-117) 29 files modified
src/main/java/com/akiban/rest/resources/FullTextResource.java (+0/-19) src/main/java/com/akiban/server/rowdata/RowDefCache.java (+3/-1) src/main/java/com/akiban/server/service/dxl/BasicDDLFunctions.java (+8/-1) src/main/java/com/akiban/server/service/restdml/RestDMLService.java (+0/-2) src/main/java/com/akiban/server/service/restdml/RestDMLServiceImpl.java (+0/-10) src/main/java/com/akiban/server/service/text/FullTextCursor.java (+5/-4) src/main/java/com/akiban/server/service/text/FullTextIndexInfo.java (+79/-2) src/main/java/com/akiban/server/service/text/FullTextIndexInfosImpl.java (+16/-0) src/main/java/com/akiban/server/service/text/FullTextIndexService.java (+15/-1) src/main/java/com/akiban/server/service/text/FullTextIndexServiceImpl.java (+343/-12) src/main/java/com/akiban/server/service/text/FullTextIndexShared.java (+5/-0) src/main/java/com/akiban/server/service/text/Indexer.java (+8/-1) src/main/java/com/akiban/server/service/text/RowIndexer.java (+35/-4) src/main/java/com/akiban/server/store/PersistitStore.java (+395/-2) src/main/java/com/akiban/server/types3/mcompat/mfuncs/MSleep.java (+71/-0) src/main/java/com/akiban/sql/aisddl/AISDDL.java (+1/-1) src/main/java/com/akiban/sql/aisddl/AlterTableDDL.java (+1/-1) src/main/java/com/akiban/sql/aisddl/IndexDDL.java (+13/-7) src/main/java/com/akiban/sql/aisddl/TableDDL.java (+6/-5) src/main/resources/com/akiban/server/service/config/configuration-defaults.properties (+3/-0) src/test/java/com/akiban/server/rowdata/SchemaFactory.java (+18/-3) src/test/java/com/akiban/server/service/text/FullTextIndexServiceIT.java (+317/-28) src/test/java/com/akiban/server/test/ApiTestBase.java (+10/-10) src/test/java/com/akiban/server/test/it/bugs/bug701614/MissingColumnsIT.java (+1/-1) src/test/java/com/akiban/sql/optimizer/OptimizerTestBase.java (+51/-2) src/test/resources/com/akiban/sql/pg/yaml/functional/test-drop-fulltext.yaml (+8/-0) src/test/resources/com/akiban/sql/pg/yaml/functional/test-ft-with-inherited-key.yaml (+28/-0) src/test/resources/com/akiban/sql/pg/yaml/functional/test-ft-with-pk.yaml (+22/-0) src/test/resources/com/akiban/sql/pg/yaml/functional/test-fulltext-maintenance.yaml (+49/-0) |
To merge this branch: | bzr merge lp:~oontvoo/akiban-server/bug_drop_fulltext_index |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Nathan Williams | Needs Fixing | ||
Vy Nguyen (community) | Needs Resubmitting | ||
Review via email: mp+158707@code.launchpad.net |
This proposal has been superseded by a proposal from 2013-04-15.
Commit message
Description of the change
fix drop fulltext index bug (bug 1163026)
Vy Nguyen (oontvoo) wrote : | # |
PersisitStore.java - line 1554 (+/- a few lines)
It looks like the index needs to be 'attached' to an IndexDef.
Nathan Williams (nwilliams) wrote : | # |
Only to delete the tree, which full texts don't have.
DDLFunctions has separate methods for dropping table and group indexes. Something different needs to happen for full texts (i.e. FullTextIndexSe
Nathan Williams (nwilliams) wrote : | # |
Actually, handling the distinction in PS#deleteIndexes() would let dropTableInternal() use it as well, so I'm fine with that too.
Vy Nguyen (oontvoo) wrote : | # |
skip fulltext index in persistitStore.
Nathan Williams (nwilliams) wrote : | # |
Since we're here, can you please fix the other bug I mentioned. We just need to call FTIS#dropIndex() for them.
Vy Nguyen (oontvoo) wrote : | # |
I am having a compilation-error in
src/test/
(I declared FTS as an argument in PersistStore's ctor, and other places where it's called.)
What did I miss?
Vy Nguyen (oontvoo) wrote : | # |
Hm, never mind, found it!
Nathan Williams (nwilliams) wrote : | # |
Look at FailureOnStartu
Vy Nguyen (oontvoo) wrote : | # |
Actually, it's more complicated than just calling FTS.dropIndex()
+ By the time deleteIndexes() is called, the index is no longer in AIS for FullTextIndexSh
+ In order for dropIndex() to work, the index needs to be in 'indexes' map, which none of the current fulltext index is because no one puts it in there. The createIndex() method, which would do this, needs to be called manually.
This problem is solved in the do-maintenance branch. And I can certainly have it fix the bug you mentioned. (In fact, I tried merging this branch into it and the delete works correctly, which it wouldn't in this branch).
Nathan Williams (nwilliams) wrote : | # |
> By the time deleteIndexes() is called, the index is no
> longer in AIS for FullTextIndexSh
It does not need to be in the current AIS and probably shouldn't be, given that it's being removed. Look at the flow of the other drop methods (e.g. table, index). The AIS changes are performed through the SchemaManager before tree deletion is done. You can either hang onto the 'old' AIS or get it through the Index reference.
> In order for dropIndex() to work, the index needs to be in 'indexes' map,
It should be, and sounds like will be once your other branch is in, but doesn't need to be. The getIndex() method will populate the indexes map on demand. It creates a value cached on the AIS, which is going away in this case, but will work just fine.
- 2626. By Vy Nguyen
-
fix bug1154336
- 2627. By Vy Nguyen
-
remove formatting changes
- 2628. By Vy Nguyen
-
clean up
- 2629. By Vy Nguyen
- 2630. By Vy Nguyen
-
more tests
- 2631. By Vy Nguyen
-
require FulLTetIndexService for testing
- 2632. By Vy Nguyen
-
basicddl
- 2633. By Vy Nguyen
-
add tests on droping table with ftindex. removeTrees should also delete fulltext index, even if these don't really have a tree
- 2634. By Vy Nguyen
-
add the original testcase from the bug report
- 2635. By Vy Nguyen
- 2636. By Vy Nguyen
-
merge trunk
- 2637. By Vy Nguyen
-
use wait() in test instead of blindly sleep for a long time
- 2638. By Vy Nguyen
-
merge trunk, move configs to TestConfigService
- 2639. By Vy Nguyen
-
edge cases in removeTrees()
- 2640. By Vy Nguyen
-
remove unused imports in FullTextIndexSe
rviceIT - 2641. By Vy Nguyen
-
use getownFullTextIndex
- 2642. By Vy Nguyen
-
clean up
- 2643. By Vy Nguyen
-
clean up
- 2644. By Vy Nguyen
-
pass Index as an argument to dropIndex to correctly determine the treename
- 2645. By Vy Nguyen
-
delete fulltext index
- 2646. By Vy Nguyen
-
use default settings in OptimizerTestBase
- 2647. By Vy Nguyen
-
use the old AIS when neccessary
- 2648. By Vy Nguyen
-
remove unused import accidentally added
Unmerged revisions
Preview Diff
1 | === modified file 'src/main/java/com/akiban/rest/resources/FullTextResource.java' |
2 | --- src/main/java/com/akiban/rest/resources/FullTextResource.java 2013-03-22 20:05:57 +0000 |
3 | +++ src/main/java/com/akiban/rest/resources/FullTextResource.java 2013-04-15 17:23:26 +0000 |
4 | @@ -30,7 +30,6 @@ |
5 | import javax.ws.rs.QueryParam; |
6 | import javax.ws.rs.core.Context; |
7 | import javax.ws.rs.core.Response; |
8 | -import javax.ws.rs.core.UriInfo; |
9 | import java.io.PrintWriter; |
10 | |
11 | import static com.akiban.rest.resources.ResourceHelper.MEDIATYPE_JSON_JAVASCRIPT; |
12 | @@ -66,22 +65,4 @@ |
13 | }) |
14 | .build(); |
15 | } |
16 | - |
17 | - @POST |
18 | - @Produces(MEDIATYPE_JSON_JAVASCRIPT) |
19 | - public Response textSearch(@Context final HttpServletRequest request, |
20 | - @PathParam("table") String table, |
21 | - @PathParam("index") String index) throws Exception { |
22 | - final IndexName indexName = new IndexName(ResourceHelper.parseTableName(request, table), index); |
23 | - ResourceHelper.checkSchemaAccessible(reqs.securityService, request, indexName.getSchemaName()); |
24 | - return RestResponseBuilder |
25 | - .forRequest(request) |
26 | - .body(new RestResponseBuilder.BodyGenerator() { |
27 | - @Override |
28 | - public void write(PrintWriter writer) throws Exception { |
29 | - reqs.restDMLService.refreshFullTextIndex(writer, indexName); |
30 | - } |
31 | - }) |
32 | - .build(); |
33 | - } |
34 | } |
35 | |
36 | === modified file 'src/main/java/com/akiban/server/rowdata/RowDefCache.java' |
37 | --- src/main/java/com/akiban/server/rowdata/RowDefCache.java 2013-03-22 20:05:57 +0000 |
38 | +++ src/main/java/com/akiban/server/rowdata/RowDefCache.java 2013-04-15 17:23:26 +0000 |
39 | @@ -38,6 +38,7 @@ |
40 | |
41 | import com.akiban.ais.model.AkibanInformationSchema; |
42 | import com.akiban.ais.model.Column; |
43 | +import com.akiban.ais.model.FullTextIndex; |
44 | import com.akiban.ais.model.IndexColumn; |
45 | import com.akiban.ais.model.Join; |
46 | import com.akiban.ais.model.JoinColumn; |
47 | @@ -184,6 +185,7 @@ |
48 | for (TableIndex index : table.getIndexesIncludingInternal()) { |
49 | List<IndexColumn> indexColumns = index.getKeyColumns(); |
50 | if(!indexColumns.isEmpty()) { |
51 | + |
52 | new IndexDef(rowDef, index); |
53 | if (index.isPrimaryKey()) { |
54 | indexList.add(0, index); |
55 | @@ -193,7 +195,7 @@ |
56 | } |
57 | //else Don't create IndexDef for empty, autogenerated indexes |
58 | } |
59 | - |
60 | + |
61 | // Group indexes |
62 | final List<GroupIndex> groupIndexList = new ArrayList<>(); |
63 | for (GroupIndex index : table.getGroupIndexes()) { |
64 | |
65 | === modified file 'src/main/java/com/akiban/server/service/dxl/BasicDDLFunctions.java' |
66 | --- src/main/java/com/akiban/server/service/dxl/BasicDDLFunctions.java 2013-04-11 18:35:20 +0000 |
67 | +++ src/main/java/com/akiban/server/service/dxl/BasicDDLFunctions.java 2013-04-15 17:23:26 +0000 |
68 | @@ -1031,7 +1031,14 @@ |
69 | for(String indexName : indexNamesToDrop) { |
70 | Index index = table.getIndex(indexName); |
71 | if(index == null) { |
72 | - throw new NoSuchIndexException (indexName); |
73 | + if (table instanceof UserTable) |
74 | + { |
75 | + index = ((UserTable)table).getFullTextIndex(indexName); |
76 | + if (index == null) |
77 | + throw new NoSuchIndexException (indexName); |
78 | + } |
79 | + else |
80 | + throw new NoSuchIndexException (indexName); |
81 | } |
82 | if(index.isPrimaryKey()) { |
83 | throw new ProtectedIndexException(indexName, table.getName()); |
84 | |
85 | === modified file 'src/main/java/com/akiban/server/service/restdml/RestDMLService.java' |
86 | --- src/main/java/com/akiban/server/service/restdml/RestDMLService.java 2013-04-15 14:38:06 +0000 |
87 | +++ src/main/java/com/akiban/server/service/restdml/RestDMLService.java 2013-04-15 17:23:26 +0000 |
88 | @@ -51,8 +51,6 @@ |
89 | TableName procName, Map<String,List<String>> queryParams, String content) throws SQLException; |
90 | |
91 | public void fullTextSearch(PrintWriter writer, IndexName indexName, Integer depth, String query, Integer limit); |
92 | - // TODO: Temporary. |
93 | - public void refreshFullTextIndex(PrintWriter writer, IndexName indexName); |
94 | |
95 | public String jonquilToSQL(TableName tableName, String jonquil) throws IOException; |
96 | } |
97 | |
98 | === modified file 'src/main/java/com/akiban/server/service/restdml/RestDMLServiceImpl.java' |
99 | --- src/main/java/com/akiban/server/service/restdml/RestDMLServiceImpl.java 2013-04-15 14:38:06 +0000 |
100 | +++ src/main/java/com/akiban/server/service/restdml/RestDMLServiceImpl.java 2013-04-15 17:23:26 +0000 |
101 | @@ -649,14 +649,4 @@ |
102 | true); |
103 | } |
104 | } |
105 | - |
106 | - // TODO: Temporary. |
107 | - public void refreshFullTextIndex(PrintWriter writer, IndexName indexName) { |
108 | - long count; |
109 | - try (Session session = sessionService.createSession()) { |
110 | - count = fullTextService.createIndex(session, indexName); |
111 | - } |
112 | - writer.write(String.format("{\"count\":%d}", count)); |
113 | - } |
114 | - |
115 | } |
116 | |
117 | === modified file 'src/main/java/com/akiban/server/service/text/FullTextCursor.java' |
118 | --- src/main/java/com/akiban/server/service/text/FullTextCursor.java 2013-03-22 20:05:57 +0000 |
119 | +++ src/main/java/com/akiban/server/service/text/FullTextCursor.java 2013-04-15 17:23:26 +0000 |
120 | @@ -103,7 +103,7 @@ |
121 | catch (IOException ex) { |
122 | throw new AkibanInternalException("Error reading document", ex); |
123 | } |
124 | - HKey hkey = hkey(doc.getBinaryValue(IndexedField.KEY_FIELD)); |
125 | + HKey hkey = hkey(doc.get(IndexedField.KEY_FIELD)); |
126 | Row row = new HKeyRow(rowType, hkey, hKeyCache); |
127 | logger.debug("FullTextCursor: yield {}", row); |
128 | return row; |
129 | @@ -152,11 +152,12 @@ |
130 | |
131 | /* Allocate a new <code>PersistitHKey</code> and copy the given |
132 | * key bytes into it. */ |
133 | - protected HKey hkey(BytesRef keyBytes) { |
134 | + protected HKey hkey(String encoded) { |
135 | PersistitHKey hkey = (PersistitHKey)context.getStore().newHKey(rowType.hKey()); |
136 | Key key = hkey.key(); |
137 | - key.setEncodedSize(keyBytes.length); |
138 | - System.arraycopy(keyBytes.bytes, keyBytes.offset, key.getEncodedBytes(), 0, keyBytes.length); |
139 | + byte decodedBytes[] = RowIndexer.decodeString(encoded); |
140 | + key.setEncodedSize(decodedBytes.length); |
141 | + System.arraycopy(decodedBytes, 0, key.getEncodedBytes(), 0, decodedBytes.length); |
142 | return hkey; |
143 | } |
144 | |
145 | |
146 | === modified file 'src/main/java/com/akiban/server/service/text/FullTextIndexInfo.java' |
147 | --- src/main/java/com/akiban/server/service/text/FullTextIndexInfo.java 2013-03-22 20:05:57 +0000 |
148 | +++ src/main/java/com/akiban/server/service/text/FullTextIndexInfo.java 2013-04-15 17:23:26 +0000 |
149 | @@ -20,12 +20,16 @@ |
150 | import com.akiban.ais.model.AkibanInformationSchema; |
151 | import com.akiban.ais.model.Column; |
152 | import com.akiban.ais.model.FullTextIndex; |
153 | +import com.akiban.ais.model.Group; |
154 | import com.akiban.ais.model.IndexColumn; |
155 | import com.akiban.ais.model.IndexName; |
156 | import com.akiban.ais.model.UserTable; |
157 | +import com.akiban.qp.expression.IndexKeyRange; |
158 | import com.akiban.qp.operator.API; |
159 | import com.akiban.qp.operator.Operator; |
160 | +import com.akiban.qp.row.HKeyRow; |
161 | import com.akiban.qp.rowtype.HKeyRowType; |
162 | +import com.akiban.qp.rowtype.IndexRowType; |
163 | import com.akiban.qp.rowtype.RowType; |
164 | import com.akiban.qp.rowtype.Schema; |
165 | import com.akiban.qp.rowtype.UserTableRowType; |
166 | @@ -50,7 +54,8 @@ |
167 | private Map<Column,IndexedField> fieldsByColumn; |
168 | private Map<RowType,List<IndexedField>> fieldsByRowType; |
169 | private String defaultFieldName; |
170 | - |
171 | + private Operator plan; |
172 | + |
173 | public FullTextIndexInfo(FullTextIndexShared shared) { |
174 | this.shared = shared; |
175 | } |
176 | @@ -63,7 +68,9 @@ |
177 | } |
178 | index = table.getFullTextIndex(name.getName()); |
179 | if (index == null) { |
180 | - throw new NoSuchIndexException(name.getName()); |
181 | + NoSuchIndexException ret = new NoSuchIndexException(name.getName()); |
182 | + ret.printStackTrace(); |
183 | + throw ret; |
184 | } |
185 | schema = SchemaCache.globalSchema(ais); |
186 | indexedRowType = schema.userTableRowType(table); |
187 | @@ -87,6 +94,8 @@ |
188 | } |
189 | fields.add(entry.getValue()); |
190 | } |
191 | + |
192 | + plan = computePlan(); |
193 | } |
194 | |
195 | public FullTextIndex getIndex() { |
196 | @@ -139,6 +148,74 @@ |
197 | plan = API.filter_Default(plan, rowTypes); |
198 | return plan; |
199 | } |
200 | + |
201 | + |
202 | + |
203 | + private Operator computePlan() |
204 | + { |
205 | + Operator ret = null; |
206 | + |
207 | + Group group = indexedRowType.userTable().getGroup(); |
208 | + Set<UserTableRowType> ancestors = new HashSet<>(); |
209 | + boolean hasDesc = false; |
210 | + |
211 | + for (IndexColumn ic : index.getKeyColumns()) |
212 | + { |
213 | + UserTable colUserTable = ic.getColumn().getUserTable(); |
214 | + |
215 | + if (!hasDesc && !colUserTable.equals(indexedRowType.userTable())) |
216 | + // if any column in the index def belongs to a table |
217 | + // that is a descendant of this indexed row's table |
218 | + // (meaning this indexed row has descendant(s)) |
219 | + hasDesc = colUserTable.isDescendantOf(indexedRowType.userTable()); |
220 | + |
221 | + // if the indexed table is a child of this column's table |
222 | + // (meaning this indexed row has parent(s)) |
223 | + // collect all ancestor's rowtype |
224 | + if (indexedRowType.userTable().isDescendantOf(colUserTable)) |
225 | + ancestors.add(schema.userTableRowType(colUserTable)); |
226 | + } |
227 | + |
228 | + if (hasDesc) |
229 | + { |
230 | + ancestors.remove(indexedRowType); |
231 | + |
232 | + ret = API.branchLookup_Nested(group, |
233 | + hKeyRowType, |
234 | + indexedRowType, |
235 | + API.InputPreservationOption.DISCARD_INPUT, |
236 | + 0); |
237 | + if (!ancestors.isEmpty()) |
238 | + { |
239 | + |
240 | + ret = API.ancestorLookup_Default(ret, |
241 | + group, |
242 | + indexedRowType, |
243 | + ancestors, |
244 | + API.InputPreservationOption.KEEP_INPUT); |
245 | + } |
246 | + } |
247 | + else |
248 | + { |
249 | + // has at least one ancestor (which is itself) |
250 | + ancestors.add(indexedRowType); |
251 | + ret = API.ancestorLookup_Nested(group, |
252 | + hKeyRowType, |
253 | + ancestors, |
254 | + 0); |
255 | + } |
256 | + |
257 | + return ret; |
258 | + } |
259 | + /** |
260 | + * |
261 | + * @param row |
262 | + * @return the operator plan to get to every row related to this index row |
263 | + */ |
264 | + public Operator getOperator() |
265 | + { |
266 | + return plan; |
267 | + } |
268 | |
269 | public Analyzer getAnalyzer() { |
270 | Analyzer analyzer; |
271 | |
272 | === modified file 'src/main/java/com/akiban/server/service/text/FullTextIndexInfosImpl.java' |
273 | --- src/main/java/com/akiban/server/service/text/FullTextIndexInfosImpl.java 2013-03-22 20:05:57 +0000 |
274 | +++ src/main/java/com/akiban/server/service/text/FullTextIndexInfosImpl.java 2013-04-15 17:23:26 +0000 |
275 | @@ -23,6 +23,7 @@ |
276 | import com.akiban.qp.rowtype.RowType; |
277 | import com.akiban.qp.rowtype.Schema; |
278 | import com.akiban.server.error.FullTextQueryParseException; |
279 | +import com.akiban.server.error.NoSuchIndexException; |
280 | import com.akiban.server.service.session.Session; |
281 | |
282 | import org.apache.lucene.queryparser.flexible.core.QueryNodeException; |
283 | @@ -65,6 +66,21 @@ |
284 | protected abstract AkibanInformationSchema getAIS(Session session); |
285 | protected abstract File getIndexPath(); |
286 | |
287 | + protected FullTextIndexInfo getIndexToDrop(IndexName name) |
288 | + { |
289 | + synchronized(indexes) |
290 | + { |
291 | + FullTextIndexShared shared = indexes.get(name); |
292 | + if (shared != null) |
293 | + { |
294 | + return shared.valueFor(); |
295 | + } |
296 | + else |
297 | + { |
298 | + throw new NoSuchIndexException(name.toString()); |
299 | + } |
300 | + } |
301 | + } |
302 | protected FullTextIndexInfo getIndex(Session session, IndexName name) { |
303 | AkibanInformationSchema ais = getAIS(session); |
304 | FullTextIndexInfo info; |
305 | |
306 | === modified file 'src/main/java/com/akiban/server/service/text/FullTextIndexService.java' |
307 | --- src/main/java/com/akiban/server/service/text/FullTextIndexService.java 2013-03-22 20:05:57 +0000 |
308 | +++ src/main/java/com/akiban/server/service/text/FullTextIndexService.java 2013-04-15 17:23:26 +0000 |
309 | @@ -27,7 +27,21 @@ |
310 | |
311 | /** Full service that does index maintenance and querying. */ |
312 | public interface FullTextIndexService extends FullTextIndexInfos { |
313 | - public long createIndex(Session session, IndexName name); |
314 | + |
315 | + /** |
316 | + * This is a 'promise' to populate the given index, which is *being* |
317 | + * created. |
318 | + * @param name |
319 | + */ |
320 | + public void schedulePopulate(String schema, String table, String index); |
321 | + |
322 | + /** |
323 | + * Update the given index based on the changedRow |
324 | + * |
325 | + * @param name |
326 | + * @param changedRow |
327 | + */ |
328 | + public void updateIndex(Session session, IndexName name, Iterable<byte[]> rows); |
329 | public void dropIndex(Session session, IndexName name); |
330 | public Cursor searchIndex(QueryContext context, IndexName name, |
331 | Query query, int limit); |
332 | |
333 | === modified file 'src/main/java/com/akiban/server/service/text/FullTextIndexServiceImpl.java' |
334 | --- src/main/java/com/akiban/server/service/text/FullTextIndexServiceImpl.java 2013-03-22 20:05:57 +0000 |
335 | +++ src/main/java/com/akiban/server/service/text/FullTextIndexServiceImpl.java 2013-04-15 17:23:26 +0000 |
336 | @@ -18,39 +18,59 @@ |
337 | package com.akiban.server.service.text; |
338 | |
339 | import com.akiban.ais.model.AkibanInformationSchema; |
340 | +import com.akiban.ais.model.Index; |
341 | import com.akiban.ais.model.IndexName; |
342 | +import com.akiban.ais.model.TableName; |
343 | +import com.akiban.ais.model.UserTable; |
344 | import com.akiban.qp.operator.API; |
345 | import com.akiban.qp.operator.Cursor; |
346 | import com.akiban.qp.operator.Operator; |
347 | import com.akiban.qp.operator.QueryContext; |
348 | import com.akiban.qp.operator.SimpleQueryContext; |
349 | import com.akiban.qp.operator.StoreAdapter; |
350 | -import com.akiban.qp.rowtype.RowType; |
351 | -import com.akiban.qp.rowtype.Schema; |
352 | import com.akiban.qp.persistitadapter.PersistitAdapter; |
353 | +import com.akiban.qp.persistitadapter.PersistitHKey; |
354 | +import com.akiban.qp.row.HKeyRow; |
355 | +import com.akiban.qp.rowtype.HKeyRowType; |
356 | +import com.akiban.qp.util.HKeyCache; |
357 | import com.akiban.server.error.AkibanInternalException; |
358 | -import com.akiban.server.error.DuplicateIndexException; |
359 | import com.akiban.server.service.Service; |
360 | import com.akiban.server.service.config.ConfigurationService; |
361 | import com.akiban.server.service.dxl.DXLService; |
362 | import com.akiban.server.service.session.Session; |
363 | +import com.akiban.server.service.session.SessionService; |
364 | +import com.akiban.server.service.session.SessionServiceImpl; |
365 | import com.akiban.server.service.transaction.TransactionService; |
366 | import com.akiban.server.service.tree.TreeService; |
367 | +import com.akiban.server.store.PersistitStore; |
368 | +import com.akiban.server.store.PersistitStore.HKeyBytesStream; |
369 | import com.akiban.server.store.Store; |
370 | |
371 | -import org.apache.lucene.analysis.Analyzer; |
372 | import org.apache.lucene.index.IndexWriter; |
373 | import org.apache.lucene.search.Query; |
374 | |
375 | import com.google.inject.Inject; |
376 | +import com.persistit.Exchange; |
377 | +import com.persistit.exception.PersistitException; |
378 | +import com.persistit.Key; |
379 | import org.slf4j.Logger; |
380 | import org.slf4j.LoggerFactory; |
381 | |
382 | import java.io.*; |
383 | -import java.util.*; |
384 | +import java.util.Timer; |
385 | +import java.util.TimerTask; |
386 | + |
387 | |
388 | public class FullTextIndexServiceImpl extends FullTextIndexInfosImpl implements FullTextIndexService, Service { |
389 | public static final String INDEX_PATH_PROPERTY = "akserver.text.indexpath"; |
390 | + public static final String UPDATE_INTERVAL = "akserver.text.maintenanceInterval"; |
391 | + public static final String POPULATE_DELAY_INTERVAL = "akserver.text.populateDelayInterval"; |
392 | + |
393 | + private static final String POPULATE_SCHEMA = "populate"; |
394 | + private static final String FULL_TEXT_TABLE = "full_text_populate"; |
395 | + |
396 | + private final SessionService sessionService = new SessionServiceImpl(); |
397 | + |
398 | |
399 | private final ConfigurationService configService; |
400 | private final DXLService dxlService; |
401 | @@ -59,7 +79,15 @@ |
402 | private final TreeService treeService; |
403 | |
404 | private File indexPath; |
405 | - |
406 | + |
407 | + private volatile Timer maintenanceTimer; |
408 | + private long maintenanceInterval; |
409 | + private TimerTask updateWorker; |
410 | + private PersistitStore persistitStore; |
411 | + |
412 | + private volatile Timer populateTimer; |
413 | + private long populateDelayInterval; |
414 | + |
415 | private static final Logger logger = LoggerFactory.getLogger(FullTextIndexServiceImpl.class); |
416 | |
417 | @Inject |
418 | @@ -72,12 +100,14 @@ |
419 | this.store = store; |
420 | this.transactionService = transactionService; |
421 | this.treeService = treeService; |
422 | + |
423 | } |
424 | |
425 | /* FullTextIndexService */ |
426 | |
427 | - @Override |
428 | - public long createIndex(Session session, IndexName name) { |
429 | + private long createIndex(Session session, IndexName name) { |
430 | + if (session == null) |
431 | + session = sessionService.createSession(); |
432 | FullTextIndexInfo index = getIndex(session, name); |
433 | try { |
434 | return populateIndex(session, index); |
435 | @@ -89,10 +119,27 @@ |
436 | |
437 | @Override |
438 | public void dropIndex(Session session, IndexName name) { |
439 | - FullTextIndexInfo index = getIndex(session, name); |
440 | - index.deletePath(); |
441 | - synchronized (indexes) { |
442 | - indexes.remove(name); |
443 | + |
444 | + try |
445 | + { |
446 | + // see if there exists a promise for populating this index |
447 | + Exchange ex = getPopulateExchange(session); |
448 | + ex.clear().append(name.getSchemaName()) |
449 | + .append(name.getTableName()) |
450 | + .append(name.getName()); |
451 | + |
452 | + if (ex.traverse(Key.Direction.EQ, true, 0)) |
453 | + ex.fetchAndRemove(); |
454 | + |
455 | + FullTextIndexInfo index = getIndexToDrop(name); |
456 | + index.deletePath(); |
457 | + synchronized (indexes) { |
458 | + indexes.remove(name); |
459 | + } |
460 | + } |
461 | + catch (PersistitException e) |
462 | + { |
463 | + throw new AkibanInternalException("Error while removing index", e); |
464 | } |
465 | } |
466 | |
467 | @@ -112,6 +159,10 @@ |
468 | |
469 | @Override |
470 | public void start() { |
471 | + persistitStore = store.getPersistitStore(); |
472 | + persistitStore.setFullTextService(this); |
473 | + enableUpdateWorker(); |
474 | + enablePopulateWorker(); |
475 | } |
476 | |
477 | @Override |
478 | @@ -120,6 +171,8 @@ |
479 | for (FullTextIndexShared index : indexes.values()) { |
480 | index.close(); |
481 | } |
482 | + disableUpdateWorker(); |
483 | + disablePopulateWorker(); |
484 | } |
485 | catch (IOException ex) { |
486 | throw new AkibanInternalException("Error closing index", ex); |
487 | @@ -193,4 +246,282 @@ |
488 | } |
489 | } |
490 | |
491 | + @Override |
492 | + public void updateIndex(Session session, IndexName name, Iterable<byte[]> rows) |
493 | + { |
494 | + try |
495 | + { |
496 | + FullTextIndexInfo indexInfo = getIndex(session, name); |
497 | + StoreAdapter adapter = session.get(StoreAdapter.STORE_ADAPTER_KEY); |
498 | + if (adapter == null) |
499 | + adapter = new PersistitAdapter(indexInfo.getSchema(), |
500 | + store, treeService, |
501 | + session, configService); |
502 | + QueryContext queryContext = new SimpleQueryContext(adapter); |
503 | + HKeyCache<com.akiban.qp.row.HKey> cache = new HKeyCache<>(adapter); |
504 | + IndexWriter writer = indexInfo.getIndexer().getWriter(); |
505 | + RowIndexer rowIndexer = new RowIndexer(indexInfo, writer, true); |
506 | + Cursor cursor = null; |
507 | + boolean success = false; |
508 | + try |
509 | + { |
510 | + Operator operator = indexInfo.getOperator(); |
511 | + int n = 0; |
512 | + for (byte row[] : rows) |
513 | + { |
514 | + HKeyRow hkeyRow = toHKeyRow(row, indexInfo.getHKeyRowType(), |
515 | + adapter, cache); |
516 | + queryContext.setRow(0, hkeyRow); |
517 | + cursor = API.cursor(operator, queryContext); |
518 | + rowIndexer.updateDocument(cursor, row); |
519 | + } |
520 | + success = true; |
521 | + } |
522 | + finally |
523 | + { |
524 | + rowIndexer.close(); |
525 | + if (cursor != null) |
526 | + cursor.destroy(); |
527 | + |
528 | + try |
529 | + { |
530 | + if (success) |
531 | + writer.commit(); |
532 | + else |
533 | + writer.rollback(); |
534 | + } |
535 | + catch (IOException ex) |
536 | + { |
537 | + throw new AkibanInternalException("Error commit writer", ex); |
538 | + } |
539 | + } |
540 | + } |
541 | + catch (IOException e) |
542 | + { |
543 | + throw new AkibanInternalException("Error updating index", e); |
544 | + } |
545 | + } |
546 | + |
547 | + private class DefaultUpdateWorker extends TimerTask |
548 | + { |
549 | + @Override |
550 | + // 'sync' because only one worker can work at a time? |
551 | + public synchronized void run() |
552 | + { |
553 | + Session session = sessionService.createSession(); |
554 | + boolean transaction = true; |
555 | + try |
556 | + { |
557 | + transactionService.beginTransaction(session); |
558 | + HKeyBytesStream rows = persistitStore.getChangedRows(session); |
559 | + if (rows != null) // if tree is not empty |
560 | + { |
561 | + do |
562 | + { |
563 | + updateIndex(session, |
564 | + rows.getIndexName(), |
565 | + rows); |
566 | + } |
567 | + while (rows.nextIndex()); |
568 | + rows.removeAll(); // done updating. remove all entries |
569 | + } |
570 | + transaction= false; |
571 | + } |
572 | + catch(PersistitException e) |
573 | + { |
574 | + throw new AkibanInternalException("Error while maintaning full_text indices"); |
575 | + } |
576 | + finally |
577 | + { |
578 | + if (transaction) |
579 | + transactionService.rollbackTransaction(session); |
580 | + else |
581 | + transactionService.commitTransaction(session); |
582 | + session.close(); |
583 | + } |
584 | + } |
585 | + }; |
586 | + |
587 | + |
588 | + @Override |
589 | + public void schedulePopulate(String schema, String table, String index) |
590 | + { |
591 | + Session session = sessionService.createSession(); |
592 | + boolean success = false; |
593 | + try |
594 | + { |
595 | + transactionService.beginTransaction(session); |
596 | + if(addPopulate(session, schema, table, index) && !hasScheduled && populateEnabled) |
597 | + { |
598 | + populateTimer.schedule(populateWorker(), populateDelayInterval); |
599 | + hasScheduled = true; |
600 | + } |
601 | + |
602 | + success = true; |
603 | + } |
604 | + catch (PersistitException ex) |
605 | + { |
606 | + throw new AkibanInternalException("Error while scheduling index population", ex); |
607 | + } |
608 | + finally |
609 | + { |
610 | + if (success) |
611 | + transactionService.commitTransaction(session); |
612 | + else |
613 | + transactionService.rollbackTransaction(session); |
614 | + session.close(); |
615 | + } |
616 | + } |
617 | + |
618 | + private class DefaultPopulateWorker extends TimerTask |
619 | + { |
620 | + @Override |
621 | + public synchronized void run() |
622 | + { |
623 | + Session session = sessionService.createSession(); |
624 | + try |
625 | + { |
626 | + Exchange ex = getPopulateExchange(session); |
627 | + IndexName toPopulate; |
628 | + while ((toPopulate = nextInQueue(ex)) != null) |
629 | + { |
630 | + if (stillExists(session, toPopulate)) |
631 | + createIndex(session, toPopulate); |
632 | + else |
633 | + logger.debug("FullTextIndex " + toPopulate + " deleted before population"); |
634 | + } |
635 | + ex.removeAll(); |
636 | + hasScheduled = false; |
637 | + } |
638 | + catch (PersistitException ex1) |
639 | + { |
640 | + throw new AkibanInternalException("Error while populating full_text indices", ex1); |
641 | + } |
642 | + finally |
643 | + { |
644 | + session.close(); |
645 | + } |
646 | + } |
647 | + |
648 | + private boolean stillExists(Session session, IndexName indexName) |
649 | + { |
650 | + AkibanInformationSchema ais = getAIS(session); |
651 | + UserTable table = ais.getUserTable(indexName.getFullTableName()); |
652 | + return !(table == null || table.getFullTextIndex(indexName.getName()) == null); |
653 | + } |
654 | + |
655 | + }; |
656 | + |
657 | + // ---------- for testing --------------- |
658 | + void disableUpdateWorker() |
659 | + { |
660 | + if (maintenanceTimer == null) |
661 | + { |
662 | + logger.debug("maintenance worker ALREADY disabled"); |
663 | + return; |
664 | + } |
665 | + updateWorker.cancel(); |
666 | + maintenanceTimer.cancel(); |
667 | + maintenanceTimer.purge(); |
668 | + maintenanceTimer = null; |
669 | + updateWorker = null; |
670 | + } |
671 | + |
672 | + protected void enableUpdateWorker() |
673 | + { |
674 | + maintenanceInterval = Long.parseLong(configService.getProperty(UPDATE_INTERVAL)); |
675 | + maintenanceTimer = new Timer(); |
676 | + updateWorker = new DefaultUpdateWorker(); |
677 | + maintenanceTimer.scheduleAtFixedRate(updateWorker, maintenanceInterval, maintenanceInterval); |
678 | + } |
679 | + |
680 | + protected void enablePopulateWorker() |
681 | + { |
682 | + populateDelayInterval = Long.parseLong(configService.getProperty(POPULATE_DELAY_INTERVAL)); |
683 | + populateTimer = new Timer(); |
684 | + populateTimer.schedule(populateWorker(), populateDelayInterval); |
685 | + populateEnabled = true; |
686 | + hasScheduled = true; |
687 | + } |
688 | + |
689 | + void disablePopulateWorker() |
690 | + { |
691 | + if (populateTimer == null) |
692 | + { |
693 | + logger.debug("populate worker already disabled"); |
694 | + return; |
695 | + } |
696 | + populateTimer.cancel(); |
697 | + populateTimer.purge(); |
698 | + hasScheduled = false; |
699 | + populateEnabled = false; |
700 | + populateTimer = null; |
701 | + } |
702 | + |
703 | + private TimerTask populateWorker() |
704 | + { |
705 | + return new DefaultPopulateWorker(); |
706 | + } |
707 | + |
708 | + //----------- private helpers ----------- |
709 | + protected IndexName nextInQueue(Exchange ex) throws PersistitException |
710 | + { |
711 | + Key key = ex.getKey(); |
712 | + |
713 | + if (ex.next(true)) // empty tree? |
714 | + { |
715 | + key.reset(); |
716 | + IndexName ret = new IndexName(new TableName(key.decodeString(), |
717 | + key.decodeString()), |
718 | + key.decodeString()); |
719 | + return ret; |
720 | + } |
721 | + else |
722 | + return null; |
723 | + } |
724 | + |
725 | + protected Exchange getPopulateExchange(Session session) throws PersistitException |
726 | + { |
727 | + Exchange ret = treeService.getExchange(session, |
728 | + treeService.treeLink(POPULATE_SCHEMA, |
729 | + FULL_TEXT_TABLE)); |
730 | + // start at the first entry |
731 | + ret.append(Key.BEFORE); |
732 | + return ret; |
733 | + } |
734 | + |
735 | + private volatile boolean hasScheduled = false; |
736 | + private volatile boolean populateEnabled = false; |
737 | + |
738 | + private synchronized boolean addPopulate(Session session, |
739 | + String schema, |
740 | + String table, |
741 | + String index) throws PersistitException |
742 | + { |
743 | + Exchange ex = getPopulateExchange(session); |
744 | + |
745 | + // Assumption: There is not any existing entry about this index |
746 | + // (Because they should have been removed in dropIndex()) |
747 | + // KEY: schema | table | indexName |
748 | + ex.getKey().clear() |
749 | + .append(schema) |
750 | + .append(table) |
751 | + .append(index); |
752 | + |
753 | + // VALUE: <empty> |
754 | + |
755 | + ex.store(); |
756 | + return true; |
757 | + } |
758 | + |
759 | + private HKeyRow toHKeyRow(byte rowBytes[], HKeyRowType hKeyRowType, |
760 | + StoreAdapter store, HKeyCache<com.akiban.qp.row.HKey> cache) |
761 | + { |
762 | + PersistitHKey hkey = (PersistitHKey)store.newHKey(hKeyRowType.hKey()); |
763 | + Key key = hkey.key(); |
764 | + key.setEncodedSize(rowBytes.length); |
765 | + System.arraycopy(rowBytes, 0, key.getEncodedBytes(), 0, rowBytes.length); |
766 | + return new HKeyRow(hKeyRowType, hkey, cache); |
767 | + } |
768 | + |
769 | } |
770 | |
771 | === modified file 'src/main/java/com/akiban/server/service/text/FullTextIndexShared.java' |
772 | --- src/main/java/com/akiban/server/service/text/FullTextIndexShared.java 2013-03-22 20:05:57 +0000 |
773 | +++ src/main/java/com/akiban/server/service/text/FullTextIndexShared.java 2013-04-15 17:23:26 +0000 |
774 | @@ -103,6 +103,11 @@ |
775 | return ais.getCachedValue(this, this); |
776 | } |
777 | |
778 | + public FullTextIndexInfo valueFor() |
779 | + { |
780 | + return new FullTextIndexInfo(this); |
781 | + } |
782 | + |
783 | @Override |
784 | public FullTextIndexInfo valueFor(AkibanInformationSchema ais) { |
785 | FullTextIndexInfo result = new FullTextIndexInfo(this); |
786 | |
787 | === modified file 'src/main/java/com/akiban/server/service/text/Indexer.java' |
788 | --- src/main/java/com/akiban/server/service/text/Indexer.java 2013-03-22 20:05:57 +0000 |
789 | +++ src/main/java/com/akiban/server/service/text/Indexer.java 2013-04-15 17:23:26 +0000 |
790 | @@ -26,6 +26,7 @@ |
791 | |
792 | import java.io.Closeable; |
793 | import java.io.IOException; |
794 | +import org.apache.lucene.index.IndexReader; |
795 | |
796 | public class Indexer implements Closeable |
797 | { |
798 | @@ -34,8 +35,14 @@ |
799 | |
800 | public Indexer(FullTextIndexShared index, Analyzer analyzer) throws IOException { |
801 | this.index = index; |
802 | + |
803 | + |
804 | IndexWriterConfig iwc = new IndexWriterConfig(Version.LUCENE_40, analyzer); |
805 | - this.writer = new IndexWriter(index.open(), iwc); |
806 | + iwc.setMaxBufferedDeleteTerms(1); // The deletion needs to be reflected immediately (on disk) |
807 | + |
808 | + this.writer = new IndexWriter(index.open(), iwc); |
809 | + IndexReader r; |
810 | + |
811 | } |
812 | |
813 | public FullTextIndexShared getIndex() { |
814 | |
815 | === modified file 'src/main/java/com/akiban/server/service/text/RowIndexer.java' |
816 | --- src/main/java/com/akiban/server/service/text/RowIndexer.java 2013-03-22 20:05:57 +0000 |
817 | +++ src/main/java/com/akiban/server/service/text/RowIndexer.java 2013-04-15 17:23:26 +0000 |
818 | @@ -20,9 +20,11 @@ |
819 | import com.akiban.ais.model.AkibanInformationSchema; |
820 | import com.akiban.qp.operator.Cursor; |
821 | import com.akiban.qp.persistitadapter.PersistitHKey; |
822 | +import com.akiban.qp.row.HKey; |
823 | import com.akiban.qp.row.Row; |
824 | import com.akiban.qp.rowtype.RowType; |
825 | import com.akiban.qp.rowtype.UserTableRowType; |
826 | +import com.akiban.qp.util.PersistitKey; |
827 | import com.akiban.server.types3.pvalue.PValueSource; |
828 | import com.akiban.util.ShareHolder; |
829 | import com.persistit.Key; |
830 | @@ -39,6 +41,11 @@ |
831 | |
832 | import java.io.*; |
833 | import java.util.*; |
834 | +import org.apache.commons.codec.binary.Base64; |
835 | +import org.apache.lucene.document.Field.Store; |
836 | +import org.apache.lucene.document.StringField; |
837 | +import org.apache.lucene.index.DirectoryReader; |
838 | +import org.apache.lucene.index.IndexReader; |
839 | |
840 | /** Given <code>Row</code>s in hkey order, create <code>Document</code>s. */ |
841 | public class RowIndexer implements Closeable |
842 | @@ -50,7 +57,8 @@ |
843 | private IndexWriter writer; |
844 | private Document currentDocument; |
845 | private long documentCount; |
846 | - private BytesRef keyBytes; |
847 | + //private BytesRef keyBytes; |
848 | + private String keyEncodedString; |
849 | private boolean updating; |
850 | |
851 | private static final Logger logger = LoggerFactory.getLogger(RowIndexer.class); |
852 | @@ -134,10 +142,21 @@ |
853 | return documentCount; |
854 | } |
855 | |
856 | + protected void updateDocument(Cursor cursor, byte hkeyBytes[]) throws IOException |
857 | + { |
858 | + if (indexRows(cursor) == 0) |
859 | + { |
860 | + String encoded = encodeBytes(hkeyBytes, 0, hkeyBytes.length); |
861 | + writer.deleteDocuments(new Term(IndexedField.KEY_FIELD, encoded)); |
862 | + logger.debug("Deleted documents with encoded byptes: " + encoded); |
863 | + } |
864 | + } |
865 | + |
866 | protected void addDocument() throws IOException { |
867 | if (currentDocument != null) { |
868 | if (updating) { |
869 | - writer.updateDocument(new Term(IndexedField.KEY_FIELD, keyBytes), |
870 | + |
871 | + writer.updateDocument(new Term(IndexedField.KEY_FIELD, keyEncodedString), |
872 | currentDocument); |
873 | logger.debug("Updated {}", currentDocument); |
874 | } |
875 | @@ -152,8 +171,8 @@ |
876 | |
877 | protected void getKeyBytes(Row row) { |
878 | Key key = ((PersistitHKey)row.hKey()).key(); |
879 | - keyBytes = new BytesRef(key.getEncodedBytes(), 0, key.getEncodedSize()); |
880 | - Field field = new StoredField(IndexedField.KEY_FIELD, keyBytes); |
881 | + keyEncodedString = encodeBytes(key.getEncodedBytes(), 0, key.getEncodedSize()); |
882 | + Field field = new StringField(IndexedField.KEY_FIELD, keyEncodedString, Store.YES); |
883 | currentDocument.add(field); |
884 | } |
885 | |
886 | @@ -166,6 +185,18 @@ |
887 | } |
888 | } |
889 | |
890 | + static String encodeBytes(byte bytes[], int offset, int length) |
891 | + { |
892 | + // TODO: needs to be more efficient? |
893 | + String ret = Base64.encodeBase64String(Arrays.copyOfRange(bytes, offset, length)); |
894 | + return ret; |
895 | + } |
896 | + |
897 | + static byte[] decodeString(String st) |
898 | + { |
899 | + return Base64.decodeBase64(st); |
900 | + } |
901 | + |
902 | @Override |
903 | public void close() { |
904 | for (ShareHolder<Row> holder : ancestors) { |
905 | |
906 | === modified file 'src/main/java/com/akiban/server/store/PersistitStore.java' |
907 | --- src/main/java/com/akiban/server/store/PersistitStore.java 2013-04-11 06:15:45 +0000 |
908 | +++ src/main/java/com/akiban/server/store/PersistitStore.java 2013-04-15 17:23:26 +0000 |
909 | @@ -18,6 +18,7 @@ |
910 | package com.akiban.server.store; |
911 | |
912 | import com.akiban.ais.model.*; |
913 | +import com.akiban.ais.model.Index.IndexType; |
914 | import com.akiban.qp.operator.StoreAdapter; |
915 | import com.akiban.qp.persistitadapter.OperatorBasedRowCollector; |
916 | import com.akiban.qp.persistitadapter.PersistitAdapter; |
917 | @@ -41,6 +42,7 @@ |
918 | import com.akiban.server.service.config.ConfigurationService; |
919 | import com.akiban.server.service.lock.LockService; |
920 | import com.akiban.server.service.session.Session; |
921 | +import com.akiban.server.service.text.FullTextIndexService; |
922 | import com.akiban.server.service.transaction.TransactionService; |
923 | import com.akiban.server.service.tree.TreeLink; |
924 | import com.akiban.server.service.tree.TreeService; |
925 | @@ -95,7 +97,9 @@ |
926 | private static final String BULKLOAD_PK_BUFFER_ALLOC = "akserver.bulkload.bufferalloc.pk"; |
927 | private static final String BULKLOAD_GROUP_BUFFER_ALLOC = "akserver.bulkload.bufferalloc.group"; |
928 | private static final String WRITE_LOCK_ENABLED_CONFIG = "akserver.write_lock_enabled"; |
929 | - |
930 | + private static final String MAINTENANCE_SCHEMA = "maintenance"; |
931 | + private static final String FULL_TEXT_TABLE = "full_text"; |
932 | + |
933 | private final static int MAX_ROW_SIZE = 5000000; |
934 | |
935 | private final static int MAX_INDEX_TRANCHE_SIZE = 10 * 1024 * 1024; |
936 | @@ -128,8 +132,14 @@ |
937 | |
938 | private int deferredIndexKeyLimit = MAX_INDEX_TRANCHE_SIZE; |
939 | |
940 | + private FullTextIndexService fullTextService; |
941 | + |
942 | private RowDataValueCoder valueCoder; |
943 | |
944 | + // Each row change has a 'uniqueChangeId', stored in the 'maintenance.full_text' table |
945 | + // The number would be reset after all maintenance is done |
946 | + private volatile AtomicLong uniqueChangeId = new AtomicLong(0); // TODO: change this back to MIN_VAL |
947 | + |
948 | private static final StorageAction DIRECT_STORAGE = new StorageAction() { |
949 | @Override |
950 | public void doStore(Exchange exchange) throws PersistitException { |
951 | @@ -151,6 +161,368 @@ |
952 | this.transactionService = transactionService; |
953 | } |
954 | |
955 | + public void setFullTextService(FullTextIndexService service) |
956 | + { |
957 | + fullTextService = service; |
958 | + } |
959 | + |
960 | + // --- for tracking changes |
961 | + private Exchange getChangeExchange(Session session) throws PersistitException |
962 | + { |
963 | + return treeService.getExchange(session, |
964 | + treeService.treeLink(MAINTENANCE_SCHEMA, |
965 | + FULL_TEXT_TABLE)); |
966 | + } |
967 | + |
968 | + private void addChange(Session session, |
969 | + String schema, |
970 | + String table, |
971 | + String index, |
972 | + Integer indexId, |
973 | + Key rowHKey) throws PersistitException |
974 | + { |
975 | + Exchange ex = getChangeExchange(session); |
976 | + |
977 | + // KEY: schema | table | indexName | indexId | unique_num |
978 | + ex.clear().append(schema).append(table).append(index).append(indexId).append(uniqueChangeId.getAndIncrement()); |
979 | + |
980 | + // VALUE: rowHKey's bytes | indexId |
981 | + ex.getValue().clear().putByteArray(rowHKey.getEncodedBytes(), |
982 | + 0, |
983 | + rowHKey.getEncodedSize()); |
984 | + ex.store(); |
985 | + } |
986 | + |
987 | + private void addChangeFor(UserTable tb, Session session, Key hKey) throws PersistitException |
988 | + { |
989 | + for (Index index : tb.getFullTextIndexes()) |
990 | + { |
991 | + IndexName idxName = index.getIndexName(); |
992 | + addChange(session, |
993 | + idxName.getSchemaName(), |
994 | + idxName.getTableName(), |
995 | + idxName.getName(), |
996 | + index.getIndexId(), |
997 | + hKey); |
998 | + } |
999 | + } |
1000 | + // --- for reporting changes |
1001 | + /** |
1002 | + * Collect all 'HKeyRow's from the list of key-value pairs that have |
1003 | + * the same schema.table.indexName, |
1004 | + * also fill schema[0], |
1005 | + * table[0], |
1006 | + * and indexName[0] |
1007 | + * with correct values |
1008 | + * |
1009 | + * @param schema |
1010 | + * @param table |
1011 | + * @param indexName |
1012 | + * @param ex |
1013 | + * @return |
1014 | + */ |
1015 | + public HKeyBytesStream getChangedRows(Session session) throws PersistitException |
1016 | + { |
1017 | + Exchange ex = getChangeExchange(session); |
1018 | + ex.append(Key.BEFORE); |
1019 | + HKeyBytesStream ret; |
1020 | + |
1021 | + if (ex.hasNext(true) // if the tree is not empty |
1022 | + && (ret = new HKeyBytesStream(ex, session)).hasNext()) // and does not contain only invalid indices |
1023 | + { |
1024 | + return ret; |
1025 | + } |
1026 | + else |
1027 | + { |
1028 | + return null; |
1029 | + } |
1030 | + } |
1031 | + |
1032 | + |
1033 | + /** |
1034 | + * |
1035 | + * This is supposed to behave similar* to Scheme's stream, |
1036 | + * (ie., each element is not computed until it absolutely has to be.) |
1037 | + * |
1038 | + * In other words, the byte arrays are not decoded before hand |
1039 | + */ |
1040 | + public class HKeyBytesStream implements Iterable<byte[]> |
1041 | + { |
1042 | + |
1043 | + private final Exchange ex; |
1044 | + private IndexName indexName; |
1045 | + private int modCount = 0; |
1046 | + private boolean moreRows; // more rows of this index |
1047 | + private final Session session; |
1048 | + private boolean eot; // end of tree; |
1049 | + private HKeyBytesStream() |
1050 | + { |
1051 | + ex = null; |
1052 | + session = null; |
1053 | + } |
1054 | + |
1055 | + // 'private' because this should not be constructed |
1056 | + // anywhere other than PersistitStore.getChangedRows() |
1057 | + // The class itself, however, is public, as it can be used anywhere |
1058 | + private HKeyBytesStream (Exchange ex, Session session) throws PersistitException |
1059 | + { |
1060 | + this.ex = ex; |
1061 | + this.session = session; |
1062 | + eot = false; |
1063 | + moreRows = true; |
1064 | + nextIndex(); // at the beginning of the tree now. look for the first legit index |
1065 | + } |
1066 | + |
1067 | + public final boolean nextIndex() throws PersistitException |
1068 | + { |
1069 | + if (eot) |
1070 | + return false; |
1071 | + |
1072 | + boolean nextIndex = ex.next(true); |
1073 | + if (nextIndex) |
1074 | + { |
1075 | + modCount = 0; |
1076 | + indexName = buildName(ex); |
1077 | + if (eot = !ignoreDeleted(ex, indexName, true)) // reach the end after ignoring all deleted indexId |
1078 | + return moreRows = false; // hence no more rows (or indices) |
1079 | + } |
1080 | + else |
1081 | + { |
1082 | + eot = true; |
1083 | + moreRows = false; |
1084 | + } |
1085 | + |
1086 | + return nextIndex; |
1087 | + } |
1088 | + |
1089 | + public boolean hasNext() |
1090 | + { |
1091 | + return !eot; |
1092 | + } |
1093 | + |
1094 | + public IndexName getIndexName() |
1095 | + { |
1096 | + return indexName; |
1097 | + } |
1098 | + |
1099 | + @Override |
1100 | + public Iterator<byte[]> iterator() |
1101 | + { |
1102 | + return new StreamIterator(moreRows); |
1103 | + } |
1104 | + |
1105 | + /** |
1106 | + * remove all change-entries |
1107 | + * |
1108 | + * |
1109 | + */ |
1110 | + public void removeAll() throws PersistitException |
1111 | + { |
1112 | + ex.removeAll(); |
1113 | + ++modCount; |
1114 | + } |
1115 | + |
1116 | + private class StreamIterator implements Iterator<byte[]> |
1117 | + { |
1118 | + |
1119 | + private boolean hasNext; |
1120 | + private int innerModCount = modCount; |
1121 | + |
1122 | + private StreamIterator (boolean hasNext) |
1123 | + { |
1124 | + this.hasNext = hasNext; |
1125 | + } |
1126 | + |
1127 | + @Override |
1128 | + public boolean hasNext() |
1129 | + { |
1130 | + return hasNext; |
1131 | + } |
1132 | + |
1133 | + @Override |
1134 | + public byte[] next() |
1135 | + { |
1136 | + try |
1137 | + { |
1138 | + if (innerModCount != modCount) |
1139 | + throw new ConcurrentModificationException(); |
1140 | + |
1141 | + byte ret[] = ex.getValue().getByteArray(); |
1142 | + boolean seeNewIndex = false; |
1143 | + boolean hasMore; |
1144 | + hasNext = (hasMore = ex.next(true)) |
1145 | + && !(seeNewIndex = seeNewIndex(indexName.getSchemaName(), |
1146 | + indexName.getTableName(), |
1147 | + indexName.getName(), |
1148 | + ex.getKey()) |
1149 | + // and following this row is at least one row of |
1150 | + // this index whose index-id is valid |
1151 | + && ignoreDeleted(ex, indexName, false)); |
1152 | + |
1153 | + // if there are no more entries, |
1154 | + // set eot so the 'outer loop', which takes care of each index |
1155 | + // does not attempt to do 'next(true)' |
1156 | + // (because surpringly, after the end has been reached |
1157 | + // the key will be set to BEFORE, therefore next(true) |
1158 | + /// will return 'true'. ==> infinite loop!) |
1159 | + eot = !hasMore; |
1160 | + |
1161 | + // Take caution in order NOT to go to the next index (ie., different name) |
1162 | + if (seeNewIndex) |
1163 | + // back up one entry, because we have read past the last |
1164 | + // entry that has the same schema.table.indexName |
1165 | + ex.previous(); |
1166 | + ++innerModCount; |
1167 | + ++modCount; |
1168 | + |
1169 | + return ret; |
1170 | + } |
1171 | + catch (PersistitException ex) |
1172 | + { |
1173 | + throw new AkibanInternalException("Error retrieving rows from Exchange", ex); |
1174 | + } |
1175 | + } |
1176 | + |
1177 | + @Override |
1178 | + public void remove() |
1179 | + { |
1180 | + throw new UnsupportedOperationException("Not supported yet."); |
1181 | + } |
1182 | + } |
1183 | + |
1184 | + /** |
1185 | + * Skip all entries whose indexName is that of a non-existing index, |
1186 | + * or whose indexId is not the same as that of the the current index (with the same name) |
1187 | + * |
1188 | + * @param ex |
1189 | + * @param indexName |
1190 | + * @param lookPastNewIndex whether to go beyond the current index |
1191 | + * @return <FALSE> iff after ignoring all deleted indices, there's nothing left |
1192 | + * <TRUE> otherwise |
1193 | + * @throws PersistitException |
1194 | + */ |
1195 | + private boolean ignoreDeleted(Exchange ex, |
1196 | + IndexName indexName, |
1197 | + boolean lookPastNewIndex) |
1198 | + throws PersistitException |
1199 | + { |
1200 | + // KEY: Schema | Table | indexName | indexID | ... |
1201 | + Key key = ex.getKey(); |
1202 | + key.reset(); |
1203 | + |
1204 | + key.reset(); |
1205 | + assert indexName.getSchemaName().equals(key.decodeString()) |
1206 | + : "Unexpected schema" ; |
1207 | + assert indexName.getTableName().equals(key.decodeString()) |
1208 | + : "Unexpected table"; |
1209 | + assert indexName.getName().equals(key.decodeString()) |
1210 | + : "Unexpected name"; |
1211 | + |
1212 | + Integer indexId = key.decodeInt(); |
1213 | + while (sameIdAsCurrent(session, indexName, indexId) != NOCHANGE) |
1214 | + // there was a DELETE (and possibly RECREATE) |
1215 | + // ignore all changes in the old id(s) |
1216 | + { |
1217 | + boolean hasMore = true; |
1218 | + boolean seeNewIndex = false; |
1219 | + |
1220 | + do |
1221 | + { /*do nothing (skipping deleted 'index')*/} |
1222 | + while ((hasMore = ex.next(true)) |
1223 | + && !(seeNewIndex = seeNewIndex(indexName.getSchemaName(), |
1224 | + indexName.getTableName(), |
1225 | + indexName.getName(), |
1226 | + ex.getKey())) |
1227 | + && !seeNewIndexId(ex, indexId)); |
1228 | + |
1229 | + if (eot = !hasMore) // reach the end! (no more rows or indices!) |
1230 | + return false; // set 'eot' for the same reason in StreamIterator.next() |
1231 | + |
1232 | + else if (seeNewIndex) // back up one entry in order not to |
1233 | + { // go past the last pair |
1234 | + ex.previous(true); |
1235 | + |
1236 | + // Saw new index, |
1237 | + // hence do the checking again, |
1238 | + // if we're allowed to look past new index |
1239 | + // (otherwise, meaning no more rows with good ids of this |
1240 | + // index, return false) |
1241 | + return lookPastNewIndex ? nextIndex() : false; |
1242 | + } |
1243 | + else // ie., see new IndexId |
1244 | + { |
1245 | + Key k = ex.getKey(); |
1246 | + k.reset(); |
1247 | + k.decodeString(); |
1248 | + k.decodeString(); |
1249 | + k.decodeString(); |
1250 | + |
1251 | + indexId = k.decodeInt(); |
1252 | + } |
1253 | + } |
1254 | + return true; |
1255 | + } |
1256 | + |
1257 | + private boolean seeNewIndexId(Exchange ex, Integer oldId) |
1258 | + { |
1259 | + Key key = ex.getKey(); |
1260 | + key.reset(); |
1261 | + // skip uninteresting parts |
1262 | + key.decodeString(); |
1263 | + key.decodeString(); |
1264 | + key.decodeString(); |
1265 | + |
1266 | + return !oldId.equals(key.decodeInt()); |
1267 | + |
1268 | + } |
1269 | + |
1270 | + /** |
1271 | + * Check if the given id is the same as the current index's id. |
1272 | + * |
1273 | + * (If the index with the given name no longer exists, return false) |
1274 | + * @param schema |
1275 | + * @param table |
1276 | + * @param indexName |
1277 | + * @param id |
1278 | + * @return |
1279 | + */ |
1280 | + private int sameIdAsCurrent(Session session, IndexName indexName, Integer id) |
1281 | + { |
1282 | + AkibanInformationSchema ais = getAIS(session); |
1283 | + UserTable table = ais.getUserTable(indexName.getFullTableName()); |
1284 | + Index index; |
1285 | + if (table == null || (index = table.getFullTextIndex(indexName.getName())) == null) |
1286 | + return DELETED; |
1287 | + |
1288 | + return index.getIndexId() != id ? RECREATED : NOCHANGE; |
1289 | + } |
1290 | + |
1291 | + private boolean seeNewIndex(String schema, String table, String indexName, Key k) |
1292 | + { |
1293 | + k.reset(); |
1294 | + return !k.decodeString().equals(schema) |
1295 | + || !k.decodeString().equals(table) |
1296 | + || !k.decodeString().equals(indexName); |
1297 | + } |
1298 | + |
1299 | + private IndexName buildName(Exchange e) |
1300 | + { |
1301 | + Key key = e.getKey(); |
1302 | + key.reset(); |
1303 | + return new IndexName(new TableName(key.decodeString(), |
1304 | + key.decodeString()), |
1305 | + key.decodeString()); |
1306 | + } |
1307 | + |
1308 | + // status of index |
1309 | + private static final int NOCHANGE = 0; |
1310 | + private static final int DELETED = 1; |
1311 | + private static final int RECREATED = 2; |
1312 | + |
1313 | + } |
1314 | + |
1315 | + //------ end fulltext index maintenance services ------ |
1316 | + |
1317 | @Override |
1318 | public synchronized void start() { |
1319 | try { |
1320 | @@ -444,6 +816,7 @@ |
1321 | } |
1322 | |
1323 | PersistitIndexRowBuffer indexRow = new PersistitIndexRowBuffer(adapter(session)); |
1324 | + addChangeFor(rowDef.userTable(), session, hEx.getKey()); |
1325 | for (Index index : rowDef.getIndexes()) { |
1326 | insertIntoIndex(session, index, rowData, hEx.getKey(), indexRow, deferIndexes); |
1327 | } |
1328 | @@ -668,6 +1041,7 @@ |
1329 | { |
1330 | RowDef rowDef = writeCheck(session, rowData, false); |
1331 | Exchange hEx = null; |
1332 | + |
1333 | DELETE_ROW_TAP.in(); |
1334 | try { |
1335 | hEx = getExchange(session, rowDef); |
1336 | @@ -681,12 +1055,16 @@ |
1337 | if (!hEx.getValue().isDefined()) { |
1338 | throw new NoSuchRowException(hEx.getKey()); |
1339 | } |
1340 | + // record the deletion of the old index row |
1341 | + if (deleteIndexes) |
1342 | + addChangeFor(rowDef.userTable(), session, hEx.getKey()); |
1343 | |
1344 | // Remove the h-row |
1345 | hEx.remove(); |
1346 | rowDef.getTableStatus().rowDeleted(); |
1347 | |
1348 | // Remove the indexes, including the PK index |
1349 | + |
1350 | PersistitIndexRowBuffer indexRow = new PersistitIndexRowBuffer(adapter(session)); |
1351 | if(deleteIndexes) { |
1352 | for (Index index : rowDef.getIndexes()) { |
1353 | @@ -738,6 +1116,7 @@ |
1354 | RowDef newRowDef = rowDefFromExplicitOrId(session, newRowData); |
1355 | PersistitAdapter adapter = adapter(session); |
1356 | Exchange hEx = null; |
1357 | + |
1358 | UPDATE_ROW_TAP.in(); |
1359 | try { |
1360 | hEx = getExchange(session, rowDef); |
1361 | @@ -751,6 +1130,7 @@ |
1362 | if (!hEx.getValue().isDefined()) { |
1363 | throw new NoSuchRowException (hEx.getKey()); |
1364 | } |
1365 | + |
1366 | // Combine current version of row with the version coming in |
1367 | // on the update request. |
1368 | // This is done by taking only the values of columns listed |
1369 | @@ -770,7 +1150,9 @@ |
1370 | packRowData(hEx, newRowDef, mergedRowData); |
1371 | // Store the h-row |
1372 | hEx.store(); |
1373 | - // Update the indexes |
1374 | + // Update the indexes (new row) |
1375 | + addChangeFor(newRowDef.userTable(), session, hEx.getKey()); |
1376 | + |
1377 | PersistitIndexRowBuffer indexRowBuffer = new PersistitIndexRowBuffer(adapter); |
1378 | Index[] indexes = (indexesToMaintain == null) ? rowDef.getIndexes() : indexesToMaintain; |
1379 | for (Index index : indexes) { |
1380 | @@ -874,6 +1256,10 @@ |
1381 | int descendentOrdinal = descendentRowDef.getOrdinal(); |
1382 | if ((tablesRequiringHKeyMaintenance == null || tablesRequiringHKeyMaintenance.get(descendentOrdinal))) { |
1383 | PROPAGATE_HKEY_CHANGE_ROW_REPLACE_TAP.hit(); |
1384 | + |
1385 | + // record the change |
1386 | + addChangeFor(descendentRowDef.userTable(), session, exchange.getKey()); |
1387 | + |
1388 | // Delete the current row from the tree. Don't call deleteRow, because we don't need to recompute |
1389 | // the hkey. |
1390 | exchange.remove(); |
1391 | @@ -1554,6 +1940,13 @@ |
1392 | public void deleteIndexes(final Session session, final Collection<? extends Index> indexes) { |
1393 | List<TreeLink> links = new ArrayList<>(indexes.size()); |
1394 | for(Index index : indexes) { |
1395 | + // no trees to drop |
1396 | + if (index.getIndexType() == IndexType.FULL_TEXT) |
1397 | + { |
1398 | + fullTextService.dropIndex(session, index.getIndexName()); |
1399 | + indexes.remove(index); |
1400 | + continue; |
1401 | + } |
1402 | final IndexDef indexDef = index.indexDef(); |
1403 | if(indexDef == null) { |
1404 | throw new IllegalStateException("indexDef is null for index: " + index); |
1405 | |
1406 | === added file 'src/main/java/com/akiban/server/types3/mcompat/mfuncs/MSleep.java' |
1407 | --- src/main/java/com/akiban/server/types3/mcompat/mfuncs/MSleep.java 1970-01-01 00:00:00 +0000 |
1408 | +++ src/main/java/com/akiban/server/types3/mcompat/mfuncs/MSleep.java 2013-04-15 17:23:26 +0000 |
1409 | @@ -0,0 +1,71 @@ |
1410 | +/** |
1411 | + * Copyright (C) 2009-2013 Akiban Technologies, Inc. |
1412 | + * |
1413 | + * This program is free software: you can redistribute it and/or modify |
1414 | + * it under the terms of the GNU Affero General Public License as published by |
1415 | + * the Free Software Foundation, either version 3 of the License, or |
1416 | + * (at your option) any later version. |
1417 | + * |
1418 | + * This program is distributed in the hope that it will be useful, |
1419 | + * but WITHOUT ANY WARRANTY; without even the implied warranty of |
1420 | + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
1421 | + * GNU Affero General Public License for more details. |
1422 | + * |
1423 | + * You should have received a copy of the GNU Affero General Public License |
1424 | + * along with this program. If not, see <http://www.gnu.org/licenses/>. |
1425 | + */ |
1426 | + |
1427 | +package com.akiban.server.types3.mcompat.mfuncs; |
1428 | + |
1429 | +import com.akiban.server.error.AkibanInternalException; |
1430 | +import com.akiban.server.types3.LazyList; |
1431 | +import com.akiban.server.types3.TExecutionContext; |
1432 | +import com.akiban.server.types3.TOverloadResult; |
1433 | +import com.akiban.server.types3.TScalar; |
1434 | +import com.akiban.server.types3.mcompat.mtypes.MNumeric; |
1435 | +import com.akiban.server.types3.pvalue.PValueSource; |
1436 | +import com.akiban.server.types3.pvalue.PValueTarget; |
1437 | +import com.akiban.server.types3.texpressions.TInputSetBuilder; |
1438 | +import com.akiban.server.types3.texpressions.TScalarBase; |
1439 | + |
1440 | +public class MSleep extends TScalarBase |
1441 | +{ |
1442 | + |
1443 | + public static final TScalar SLEEP = new MSleep(); |
1444 | + |
1445 | + private MSleep() {} |
1446 | + @Override |
1447 | + protected void buildInputSets(TInputSetBuilder builder) |
1448 | + { |
1449 | + builder.covers(MNumeric.INT, 0); |
1450 | + } |
1451 | + |
1452 | + @Override |
1453 | + protected void doEvaluate(TExecutionContext context, LazyList<? extends PValueSource> inputs, PValueTarget output) |
1454 | + { |
1455 | + try |
1456 | + { |
1457 | + int value = inputs.get(0).getInt32(); |
1458 | + Thread.sleep(value); |
1459 | + output.putInt32(value); |
1460 | + } |
1461 | + catch (InterruptedException ex) |
1462 | + { |
1463 | + output.putNull(); |
1464 | + throw new AkibanInternalException("InteruptedException " + ex); |
1465 | + } |
1466 | + } |
1467 | + |
1468 | + @Override |
1469 | + public String displayName() |
1470 | + { |
1471 | + return "sleep"; |
1472 | + } |
1473 | + |
1474 | + @Override |
1475 | + public TOverloadResult resultType() |
1476 | + { |
1477 | + return TOverloadResult.fixed(MNumeric.INT); |
1478 | + } |
1479 | + |
1480 | +} |
1481 | |
1482 | === modified file 'src/main/java/com/akiban/sql/aisddl/AISDDL.java' |
1483 | --- src/main/java/com/akiban/sql/aisddl/AISDDL.java 2013-03-22 20:05:57 +0000 |
1484 | +++ src/main/java/com/akiban/sql/aisddl/AISDDL.java 2013-04-15 17:23:26 +0000 |
1485 | @@ -84,7 +84,7 @@ |
1486 | server.getBinderContext(), context); |
1487 | return; |
1488 | case NodeTypes.CREATE_INDEX_NODE: |
1489 | - IndexDDL.createIndex(ddlFunctions, session, schema, (CreateIndexNode)ddl); |
1490 | + IndexDDL.createIndex(ddlFunctions, session, schema, (CreateIndexNode)ddl, server.getServiceManager()); |
1491 | return; |
1492 | case NodeTypes.DROP_INDEX_NODE: |
1493 | IndexDDL.dropIndex(ddlFunctions, session, schema, (DropIndexNode)ddl, context); |
1494 | |
1495 | === modified file 'src/main/java/com/akiban/sql/aisddl/AlterTableDDL.java' |
1496 | --- src/main/java/com/akiban/sql/aisddl/AlterTableDDL.java 2013-03-22 20:05:57 +0000 |
1497 | +++ src/main/java/com/akiban/sql/aisddl/AlterTableDDL.java 2013-04-15 17:23:26 +0000 |
1498 | @@ -254,7 +254,7 @@ |
1499 | TableName newName = tableCopy.getName(); |
1500 | for(ConstraintDefinitionNode cdn : conDefNodes) { |
1501 | assert cdn.getConstraintType() != ConstraintType.DROP : cdn; |
1502 | - String name = TableDDL.addIndex(indexNamer, builder, cdn, newName.getSchemaName(), newName.getTableName()); |
1503 | + String name = TableDDL.addIndex(indexNamer, builder, cdn, newName.getSchemaName(), newName.getTableName(), context); |
1504 | indexChanges.add(TableChange.createAdd(name)); |
1505 | } |
1506 | |
1507 | |
1508 | === modified file 'src/main/java/com/akiban/sql/aisddl/IndexDDL.java' |
1509 | --- src/main/java/com/akiban/sql/aisddl/IndexDDL.java 2013-03-22 20:05:57 +0000 |
1510 | +++ src/main/java/com/akiban/sql/aisddl/IndexDDL.java 2013-04-15 17:23:26 +0000 |
1511 | @@ -36,6 +36,8 @@ |
1512 | import com.akiban.server.service.session.Session; |
1513 | |
1514 | import com.akiban.qp.operator.QueryContext; |
1515 | +import com.akiban.server.service.ServiceManager; |
1516 | +import com.akiban.server.service.text.FullTextIndexService; |
1517 | |
1518 | /** DDL operations on Indices */ |
1519 | public class IndexDDL |
1520 | @@ -87,7 +89,7 @@ |
1521 | } |
1522 | // if we can't find the index, set tableName to null |
1523 | // to flag not a user table index. |
1524 | - if (table.getIndex(indexName) == null) { |
1525 | + if (table.getIndex(indexName) == null && table.getFullTextIndex(indexName) == null) { |
1526 | tableName = null; |
1527 | } |
1528 | // Check the group associated to the table for the |
1529 | @@ -148,17 +150,18 @@ |
1530 | public static void createIndex(DDLFunctions ddlFunctions, |
1531 | Session session, |
1532 | String defaultSchemaName, |
1533 | - CreateIndexNode createIndex) { |
1534 | + CreateIndexNode createIndex, |
1535 | + ServiceManager sm) { |
1536 | AkibanInformationSchema ais = ddlFunctions.getAIS(session); |
1537 | |
1538 | Collection<Index> indexesToAdd = new LinkedList<>(); |
1539 | |
1540 | - indexesToAdd.add(buildIndex(ais, defaultSchemaName, createIndex)); |
1541 | + indexesToAdd.add(buildIndex(ais, defaultSchemaName, createIndex, sm)); |
1542 | |
1543 | ddlFunctions.createIndexes(session, indexesToAdd); |
1544 | } |
1545 | |
1546 | - protected static Index buildIndex (AkibanInformationSchema ais, String defaultSchemaName, CreateIndexNode index) { |
1547 | + protected static Index buildIndex (AkibanInformationSchema ais, String defaultSchemaName, CreateIndexNode index, ServiceManager sm) { |
1548 | final String schemaName = index.getObjectName().getSchemaName() != null ? index.getObjectName().getSchemaName() : defaultSchemaName; |
1549 | final String indexName = index.getObjectName().getTableName(); |
1550 | |
1551 | @@ -173,7 +176,7 @@ |
1552 | |
1553 | if (index.getColumnList().functionType() == IndexColumnList.FunctionType.FULL_TEXT) { |
1554 | logger.debug ("Building Full text index on table {}", tableName) ; |
1555 | - tableIndex = buildFullTextIndex (builder, tableName, indexName, index); |
1556 | + tableIndex = buildFullTextIndex (builder, tableName, indexName, index, sm); |
1557 | } else if (checkIndexType (index, tableName) == Index.IndexType.TABLE) { |
1558 | logger.debug ("Building Table index on table {}", tableName) ; |
1559 | tableIndex = buildTableIndex (builder, tableName, indexName, index); |
1560 | @@ -328,9 +331,10 @@ |
1561 | return builder.akibanInformationSchema().getGroup(groupName).getIndex(indexName); |
1562 | } |
1563 | |
1564 | - protected static Index buildFullTextIndex (AISBuilder builder, TableName tableName, String indexName, IndexDefinition index) { |
1565 | + protected static Index buildFullTextIndex (AISBuilder builder, TableName tableName, String indexName, IndexDefinition index, ServiceManager sm) { |
1566 | UserTable table = builder.akibanInformationSchema().getUserTable(tableName); |
1567 | - |
1568 | + FullTextIndexService fullTextService = sm.getServiceByClass(FullTextIndexService.class); |
1569 | + |
1570 | if (index.getJoinType() != null) { |
1571 | throw new TableIndexJoinTypeException(); |
1572 | } |
1573 | @@ -363,6 +367,8 @@ |
1574 | } |
1575 | |
1576 | builder.fullTextIndexColumn(tableName, indexName, schemaName, columnTable.getTableName(), columnName, i); |
1577 | + // populate index |
1578 | + fullTextService.schedulePopulate(schemaName, tableName.getTableName(), indexName); |
1579 | i++; |
1580 | } |
1581 | return builder.akibanInformationSchema().getUserTable(tableName).getFullTextIndex(indexName); |
1582 | |
1583 | === modified file 'src/main/java/com/akiban/sql/aisddl/TableDDL.java' |
1584 | --- src/main/java/com/akiban/sql/aisddl/TableDDL.java 2013-04-11 05:21:42 +0000 |
1585 | +++ src/main/java/com/akiban/sql/aisddl/TableDDL.java 2013-04-15 17:23:26 +0000 |
1586 | @@ -195,7 +195,7 @@ |
1587 | } |
1588 | } |
1589 | else if (tableElement instanceof ConstraintDefinitionNode) { |
1590 | - addIndex (namer, builder, (ConstraintDefinitionNode)tableElement, schemaName, tableName); |
1591 | + addIndex (namer, builder, (ConstraintDefinitionNode)tableElement, schemaName, tableName, context); |
1592 | } |
1593 | } |
1594 | builder.basicSchemaIsComplete(); |
1595 | @@ -312,7 +312,7 @@ |
1596 | |
1597 | |
1598 | public static String addIndex(IndexNameGenerator namer, AISBuilder builder, ConstraintDefinitionNode cdn, |
1599 | - String schemaName, String tableName) { |
1600 | + String schemaName, String tableName, QueryContext context) { |
1601 | // We don't (yet) have a constraint representation so override any provided |
1602 | UserTable table = builder.akibanInformationSchema().getUserTable(schemaName, tableName); |
1603 | final String constraint; |
1604 | @@ -330,7 +330,7 @@ |
1605 | } |
1606 | // Indexes do things a little differently because they need to support Group indexes, Full Text and Geospacial |
1607 | else if (cdn.getConstraintType() == ConstraintDefinitionNode.ConstraintType.INDEX) { |
1608 | - return generateTableIndex(namer, builder, cdn, table); |
1609 | + return generateTableIndex(namer, builder, cdn, table, context); |
1610 | } else { |
1611 | throw new UnsupportedCheckConstraintException (); |
1612 | } |
1613 | @@ -474,7 +474,8 @@ |
1614 | private static String generateTableIndex(IndexNameGenerator namer, |
1615 | AISBuilder builder, |
1616 | ConstraintDefinitionNode cdn, |
1617 | - Table table) { |
1618 | + Table table, |
1619 | + QueryContext context) { |
1620 | IndexDefinition id = ((IndexConstraintDefinitionNode)cdn); |
1621 | IndexColumnList columnList = id.getIndexColumnList(); |
1622 | Index tableIndex; |
1623 | @@ -485,7 +486,7 @@ |
1624 | |
1625 | if (columnList.functionType() == IndexColumnList.FunctionType.FULL_TEXT) { |
1626 | logger.debug ("Building Full text index on table {}", table.getName()) ; |
1627 | - tableIndex = IndexDDL.buildFullTextIndex (builder, table.getName(), indexName, id); |
1628 | + tableIndex = IndexDDL.buildFullTextIndex (builder, table.getName(), indexName, id, context.getServiceManager()); |
1629 | } else if (IndexDDL.checkIndexType (id, table.getName()) == Index.IndexType.TABLE) { |
1630 | logger.debug ("Building Table index on table {}", table.getName()) ; |
1631 | tableIndex = IndexDDL.buildTableIndex (builder, table.getName(), indexName, id); |
1632 | |
1633 | === modified file 'src/main/resources/com/akiban/server/service/config/configuration-defaults.properties' |
1634 | --- src/main/resources/com/akiban/server/service/config/configuration-defaults.properties 2013-04-06 05:07:23 +0000 |
1635 | +++ src/main/resources/com/akiban/server/service/config/configuration-defaults.properties 2013-04-15 17:23:26 +0000 |
1636 | @@ -36,6 +36,9 @@ |
1637 | akserver.bulkload.bufferalloc.pk=0.17 |
1638 | akserver.bulkload.bufferalloc.group=0.33 |
1639 | akserver.restrict_user_schema=false |
1640 | +akserver.text.indexpath=/tmp/aktext |
1641 | +akserver.text.maintenanceInterval=3000 |
1642 | +akserver.text.populateDelayInterval=1000 |
1643 | # EXPERIMENTAL |
1644 | akserver.indexRowPooling = false |
1645 | akserver.pt.osc.hook=disabled |
1646 | |
1647 | === modified file 'src/test/java/com/akiban/server/rowdata/SchemaFactory.java' |
1648 | --- src/test/java/com/akiban/server/rowdata/SchemaFactory.java 2013-04-11 18:35:20 +0000 |
1649 | +++ src/test/java/com/akiban/server/rowdata/SchemaFactory.java 2013-04-15 17:23:26 +0000 |
1650 | @@ -31,6 +31,7 @@ |
1651 | import com.akiban.server.api.DDLFunctions; |
1652 | import com.akiban.server.api.ddl.DDLFunctionsMockBase; |
1653 | import com.akiban.server.error.PersistitAdapterException; |
1654 | +import com.akiban.server.service.ServiceManager; |
1655 | import com.akiban.server.service.routines.MockRoutineLoader; |
1656 | import com.akiban.server.service.session.Session; |
1657 | import com.akiban.sql.StandardException; |
1658 | @@ -76,7 +77,12 @@ |
1659 | } |
1660 | |
1661 | public AkibanInformationSchema ais(String... ddl) { |
1662 | - return ais(new AkibanInformationSchema(), ddl); |
1663 | + return ais((ServiceManager)null, ddl); |
1664 | + } |
1665 | + |
1666 | + public AkibanInformationSchema ais(ServiceManager sm, String... ddl) |
1667 | + { |
1668 | + return ais(sm, new AkibanInformationSchema(), ddl); |
1669 | } |
1670 | |
1671 | public static AkibanInformationSchema loadAIS(File fromFile, String defaultSchema) { |
1672 | @@ -89,11 +95,20 @@ |
1673 | } |
1674 | } |
1675 | |
1676 | + public AkibanInformationSchema ais(ServiceManager sm, AkibanInformationSchema baseAIS, String... ddl) |
1677 | + { |
1678 | + return ais(sm, new CreateOnlyDDLMock(baseAIS), null, ddl); |
1679 | + } |
1680 | public AkibanInformationSchema ais(AkibanInformationSchema baseAIS, String... ddl) { |
1681 | return ais(new CreateOnlyDDLMock(baseAIS), null, ddl); |
1682 | } |
1683 | |
1684 | - public AkibanInformationSchema ais(DDLFunctions ddlFunctions, Session session, String... ddl) { |
1685 | + public AkibanInformationSchema ais(DDLFunctions ddlFunctions, Session session, String... ddl) |
1686 | + { |
1687 | + return ais(null, ddlFunctions, session, ddl); |
1688 | + } |
1689 | + |
1690 | + public AkibanInformationSchema ais(ServiceManager sm, DDLFunctions ddlFunctions, Session session, String... ddl) { |
1691 | StringBuilder buffer = new StringBuilder(); |
1692 | for (String line : ddl) { |
1693 | buffer.append(line); |
1694 | @@ -110,7 +125,7 @@ |
1695 | if (stmt instanceof CreateTableNode) { |
1696 | TableDDL.createTable(ddlFunctions , session , defaultSchema, (CreateTableNode) stmt, null); |
1697 | } else if (stmt instanceof CreateIndexNode) { |
1698 | - IndexDDL.createIndex(ddlFunctions, session, defaultSchema, (CreateIndexNode) stmt); |
1699 | + IndexDDL.createIndex(ddlFunctions, session, defaultSchema, (CreateIndexNode) stmt, sm); |
1700 | } else if (stmt instanceof CreateViewNode) { |
1701 | ViewDDL.createView(ddlFunctions, session, defaultSchema, (CreateViewNode) stmt, |
1702 | new AISBinderContext(ddlFunctions.getAIS(session), defaultSchema), null); |
1703 | |
1704 | === modified file 'src/test/java/com/akiban/server/service/text/FullTextIndexServiceIT.java' |
1705 | --- src/test/java/com/akiban/server/service/text/FullTextIndexServiceIT.java 2013-03-22 20:05:57 +0000 |
1706 | +++ src/test/java/com/akiban/server/service/text/FullTextIndexServiceIT.java 2013-04-15 17:23:26 +0000 |
1707 | @@ -18,6 +18,7 @@ |
1708 | package com.akiban.server.service.text; |
1709 | |
1710 | import com.akiban.ais.model.FullTextIndex; |
1711 | +import com.akiban.ais.model.IndexName; |
1712 | import com.akiban.qp.operator.Operator; |
1713 | import com.akiban.qp.operator.QueryContext; |
1714 | import static com.akiban.qp.operator.API.cursor; |
1715 | @@ -27,9 +28,13 @@ |
1716 | import com.akiban.qp.rowtype.Schema; |
1717 | import com.akiban.qp.util.SchemaCache; |
1718 | import com.akiban.server.service.servicemanager.GuicedServiceManager; |
1719 | +import com.akiban.server.service.session.Session; |
1720 | +import com.akiban.server.service.session.SessionServiceImpl; |
1721 | import com.akiban.server.test.it.ITBase; |
1722 | import com.akiban.server.test.it.qp.TestRow; |
1723 | |
1724 | +import com.persistit.Exchange; |
1725 | +import com.persistit.exception.PersistitException; |
1726 | import org.junit.Before; |
1727 | import org.junit.Test; |
1728 | import static org.junit.Assert.*; |
1729 | @@ -39,12 +44,18 @@ |
1730 | public class FullTextIndexServiceIT extends ITBase |
1731 | { |
1732 | public static final String SCHEMA = "test"; |
1733 | - |
1734 | + private static final long updateInterval = 1000L; // for testing |
1735 | + private static final long populateDelayInterval = 1000L; |
1736 | protected FullTextIndexService fullText; |
1737 | protected Schema schema; |
1738 | protected PersistitAdapter adapter; |
1739 | protected QueryContext queryContext; |
1740 | |
1741 | + private int c; |
1742 | + private int o; |
1743 | + private int i; |
1744 | + private int a; |
1745 | + |
1746 | @Override |
1747 | protected GuicedServiceManager.BindingsConfigurationProvider serviceBindingsProvider() { |
1748 | return super.serviceBindingsProvider() |
1749 | @@ -55,29 +66,31 @@ |
1750 | protected Map<String, String> startupConfigProperties() { |
1751 | Map<String, String> properties = new HashMap<>(); |
1752 | properties.put("akserver.text.indexpath", "/tmp/aktext"); |
1753 | + properties.put(FullTextIndexServiceImpl.UPDATE_INTERVAL, Long.toString(updateInterval)); |
1754 | + properties.put(FullTextIndexServiceImpl.POPULATE_DELAY_INTERVAL, Long.toString(populateDelayInterval)); |
1755 | return properties; |
1756 | } |
1757 | |
1758 | @Before |
1759 | public void createData() { |
1760 | - int c = createTable(SCHEMA, "c", |
1761 | - "cid INT PRIMARY KEY NOT NULL", |
1762 | - "name VARCHAR(128) COLLATE en_us_ci"); |
1763 | - int o = createTable(SCHEMA, "o", |
1764 | - "oid INT PRIMARY KEY NOT NULL", |
1765 | - "cid INT NOT NULL", |
1766 | - "GROUPING FOREIGN KEY(cid) REFERENCES c(cid)", |
1767 | - "order_date DATE"); |
1768 | - int i = createTable(SCHEMA, "i", |
1769 | - "iid INT PRIMARY KEY NOT NULL", |
1770 | - "oid INT NOT NULL", |
1771 | - "GROUPING FOREIGN KEY(oid) REFERENCES o(oid)", |
1772 | - "sku VARCHAR(10) NOT NULL"); |
1773 | - int a = createTable(SCHEMA, "a", |
1774 | - "aid INT PRIMARY KEY NOT NULL", |
1775 | - "cid INT NOT NULL", |
1776 | - "GROUPING FOREIGN KEY(cid) REFERENCES c(cid)", |
1777 | - "state CHAR(2)"); |
1778 | + c = createTable(SCHEMA, "c", |
1779 | + "cid INT PRIMARY KEY NOT NULL", |
1780 | + "name VARCHAR(128) COLLATE en_us_ci"); |
1781 | + o = createTable(SCHEMA, "o", |
1782 | + "oid INT PRIMARY KEY NOT NULL", |
1783 | + "cid INT NOT NULL", |
1784 | + "GROUPING FOREIGN KEY(cid) REFERENCES c(cid)", |
1785 | + "order_date DATE"); |
1786 | + i = createTable(SCHEMA, "i", |
1787 | + "iid INT PRIMARY KEY NOT NULL", |
1788 | + "oid INT NOT NULL", |
1789 | + "GROUPING FOREIGN KEY(oid) REFERENCES o(oid)", |
1790 | + "sku VARCHAR(10) NOT NULL"); |
1791 | + a = createTable(SCHEMA, "a", |
1792 | + "aid INT PRIMARY KEY NOT NULL", |
1793 | + "cid INT NOT NULL", |
1794 | + "GROUPING FOREIGN KEY(cid) REFERENCES c(cid)", |
1795 | + "state CHAR(2)"); |
1796 | writeRow(c, 1, "Fred Flintstone"); |
1797 | writeRow(o, 101, 1, "2012-12-12"); |
1798 | writeRow(i, 10101, 101, "ABCD"); |
1799 | @@ -99,11 +112,286 @@ |
1800 | } |
1801 | |
1802 | @Test |
1803 | - public void cDown() { |
1804 | - FullTextIndex index = createFullTextIndex(SCHEMA, "c", "idx_c", |
1805 | - "name", "i.sku", "a.state"); |
1806 | - fullText.createIndex(session(), index.getIndexName()); |
1807 | - |
1808 | + public void testPopulateScheduling() throws InterruptedException, PersistitException |
1809 | + { |
1810 | + // This test is specifically for FullTextIndexServiceImpl.java |
1811 | + assertEquals(FullTextIndexServiceImpl.class, fullText.getClass()); |
1812 | + FullTextIndexServiceImpl fullTextImpl = (FullTextIndexServiceImpl)fullText; |
1813 | + |
1814 | + // disable the populate worker (so it doesn't read all the entries |
1815 | + // out before we get a chance to look at the tree. |
1816 | + fullTextImpl.disablePopulateWorker(); |
1817 | + |
1818 | + // create 3 indices |
1819 | + final FullTextIndex expecteds[] = new FullTextIndex[] |
1820 | + { |
1821 | + createFullTextIndex(serviceManager(), |
1822 | + SCHEMA, "c", "idx1_c", |
1823 | + "name"), |
1824 | + |
1825 | + createFullTextIndex(serviceManager(), |
1826 | + SCHEMA, "c", "idx2_c", |
1827 | + "i.sku"), |
1828 | + createFullTextIndex(serviceManager(), |
1829 | + SCHEMA, "c", "idx3_c", |
1830 | + "name", "i.sku") |
1831 | + }; |
1832 | + |
1833 | + |
1834 | + // read the entries out |
1835 | + traverse(fullTextImpl, |
1836 | + new Visitor() |
1837 | + { |
1838 | + int n = 0; |
1839 | + public void visit(IndexName idx) |
1840 | + { |
1841 | + assertEquals("entry[" + n + "]", expecteds[n++].getIndexName(), |
1842 | + idx); |
1843 | + } |
1844 | + |
1845 | + public void endOfTree() |
1846 | + { |
1847 | + assertEquals(expecteds.length, n); |
1848 | + } |
1849 | + }); |
1850 | + |
1851 | + |
1852 | + // let the worker do its job. |
1853 | + // (After it is done, the tree had better be empty) |
1854 | + fullTextImpl.enablePopulateWorker(); |
1855 | + Thread.sleep(populateDelayInterval * 5); |
1856 | + |
1857 | + traverse(fullTextImpl, |
1858 | + new Visitor() |
1859 | + { |
1860 | + int n = 0; |
1861 | + |
1862 | + @Override |
1863 | + public void visit(IndexName idx) |
1864 | + { |
1865 | + ++n; |
1866 | + } |
1867 | + |
1868 | + @Override |
1869 | + public void endOfTree() |
1870 | + { |
1871 | + assertEquals (0, n); |
1872 | + } |
1873 | + }); |
1874 | + |
1875 | + // drop all the indices |
1876 | + Session session = new SessionServiceImpl().createSession(); |
1877 | + for (FullTextIndex idx : expecteds) |
1878 | + fullTextImpl.dropIndex(session, idx.getIndexName()); |
1879 | + session.close(); |
1880 | + } |
1881 | + |
1882 | + |
1883 | + @Test |
1884 | + public void testDeleteIndex() throws PersistitException, InterruptedException |
1885 | + { |
1886 | + |
1887 | + // This test is specifically for FullTextIndexServiceImpl.java |
1888 | + assertEquals(FullTextIndexServiceImpl.class, fullText.getClass()); |
1889 | + FullTextIndexServiceImpl fullTextImpl = (FullTextIndexServiceImpl)fullText; |
1890 | + |
1891 | + // <1> disable worker |
1892 | + fullTextImpl.disablePopulateWorker(); |
1893 | + |
1894 | + // <2> create 3 new indices |
1895 | + // create 3 indices |
1896 | + final FullTextIndex expecteds[] = new FullTextIndex[] |
1897 | + { |
1898 | + createFullTextIndex(serviceManager(), |
1899 | + SCHEMA, "c", "idx4_c", |
1900 | + "a.state"), |
1901 | + |
1902 | + createFullTextIndex(serviceManager(), |
1903 | + SCHEMA, "c", "idx5_c", |
1904 | + "i.sku", "a.state"), |
1905 | + createFullTextIndex(serviceManager(), |
1906 | + SCHEMA, "c", "idx6_c", |
1907 | + "name", "i.sku") |
1908 | + }; |
1909 | + |
1910 | + // <3> delete 2 of them |
1911 | + Session session = new SessionServiceImpl().createSession(); |
1912 | + boolean sawNPE = false; |
1913 | + try |
1914 | + { |
1915 | + fullTextImpl.dropIndex(session, expecteds[0].getIndexName()); |
1916 | + } |
1917 | + catch (NullPointerException e) // NPE is expected because, the index hasn't been |
1918 | + { // populated yet. But we're not testing that! |
1919 | + sawNPE = true; |
1920 | + } |
1921 | + assertTrue("NPE should have happened", sawNPE); |
1922 | + |
1923 | + |
1924 | + sawNPE = false; |
1925 | + try |
1926 | + { |
1927 | + fullTextImpl.dropIndex(session, expecteds[1].getIndexName()); |
1928 | + } |
1929 | + catch (NullPointerException e) // NPE is expected because, the index hasn't been |
1930 | + { // populated yet. But we're not testing that! |
1931 | + sawNPE = true; |
1932 | + } |
1933 | + assertTrue("NPE should have happened", sawNPE); |
1934 | + |
1935 | + // <4> check that the tree only has one entry now (ie., epxecteds2[2] |
1936 | + traverse(fullTextImpl, |
1937 | + new Visitor() |
1938 | + { |
1939 | + int n = 0; |
1940 | + |
1941 | + @Override |
1942 | + public void visit(IndexName idx) |
1943 | + { |
1944 | + assertEquals("entry[" + n++ + "]", expecteds[2].getIndexName(), |
1945 | + idx); |
1946 | + } |
1947 | + |
1948 | + @Override |
1949 | + public void endOfTree() |
1950 | + { |
1951 | + assertEquals (1, n); |
1952 | + } |
1953 | + }); |
1954 | + |
1955 | + // wake the worker up to do its job |
1956 | + fullTextImpl.enablePopulateWorker(); |
1957 | + Thread.sleep(populateDelayInterval * 5); |
1958 | + |
1959 | + // drop the remaining index |
1960 | + fullTextImpl.dropIndex(session, expecteds[2].getIndexName()); |
1961 | + session.close(); |
1962 | + } |
1963 | + |
1964 | + private static interface Visitor |
1965 | + { |
1966 | + void visit(IndexName idx); |
1967 | + void endOfTree(); |
1968 | + } |
1969 | + |
1970 | + |
1971 | + private static void traverse(FullTextIndexServiceImpl serv, |
1972 | + Visitor visitor) throws PersistitException |
1973 | + { |
1974 | + Session session = new SessionServiceImpl().createSession(); |
1975 | + |
1976 | + try |
1977 | + { |
1978 | + Exchange ex = serv.getPopulateExchange(session); |
1979 | + IndexName toPopulate; |
1980 | + while ((toPopulate = serv.nextInQueue(ex)) != null) |
1981 | + visitor.visit(toPopulate); |
1982 | + visitor.endOfTree(); |
1983 | + } |
1984 | + finally |
1985 | + { |
1986 | + session.close(); |
1987 | + } |
1988 | + } |
1989 | + |
1990 | + @Test |
1991 | + public void testUpdate() throws InterruptedException |
1992 | + { |
1993 | + |
1994 | + /* |
1995 | + Test plans: |
1996 | + 1 - do a search, confirm that the rows come back as expected |
1997 | + 2 - (With update worker enabled) |
1998 | + + insert new rows |
1999 | + + sleep for sometime |
2000 | + + do the search again and confirm that the new rows are found |
2001 | + 3 - (With update worker NOT enable) |
2002 | + + disable update worker |
2003 | + + insert new rows |
2004 | + + do the search again and confirm that the new rows are NOT found |
2005 | + |
2006 | + */ |
2007 | + |
2008 | + //CREATE INDEX cust_ft ON customers(FULL_TEXT(name, addresses.state, items.sku)) |
2009 | + |
2010 | + // part 1 |
2011 | + FullTextIndex index = createFullTextIndex(serviceManager(), |
2012 | + SCHEMA, "c", "idx_c", |
2013 | + "name", "i.sku", "a.state"); |
2014 | + //fullText.createIndex(session(), index.getIndexName()); |
2015 | + Thread.sleep(populateDelayInterval * 5); |
2016 | + RowType rowType = rowType("c"); |
2017 | + RowBase[] expected1 = new RowBase[] |
2018 | + { |
2019 | + row(rowType, 1L), |
2020 | + row(rowType, 3L) |
2021 | + }; |
2022 | + FullTextQueryBuilder builder = new FullTextQueryBuilder(index, ais(), queryContext); |
2023 | + Operator plan = builder.scanOperator("flintstone", 15); |
2024 | + compareRows(expected1, cursor(plan, queryContext)); |
2025 | + |
2026 | + plan = builder.scanOperator("state:MA", 15); |
2027 | + compareRows(expected1, cursor(plan, queryContext)); |
2028 | + |
2029 | + |
2030 | + // part 2 |
2031 | + // write new rows |
2032 | + writeRow(c, 4, "John Watson"); |
2033 | + writeRow(c, 5, "Sherlock Flintstone"); |
2034 | + writeRow(c, 6, "Mycroft Holmes"); |
2035 | + writeRow(c, 7, "Flintstone Lestrade"); |
2036 | + |
2037 | + // sleep a bit (wait for the worker to do the update) |
2038 | + Thread.sleep(updateInterval * 5); |
2039 | + RowBase expected2[] = new RowBase[] |
2040 | + { |
2041 | + row(rowType, 1L), |
2042 | + row(rowType, 3L), |
2043 | + row(rowType, 5L), |
2044 | + row(rowType, 7L) |
2045 | + }; |
2046 | + |
2047 | + // confirm new changes |
2048 | + plan = builder.scanOperator("flintstone", 15); |
2049 | + compareRows(expected2, cursor(plan, queryContext)); |
2050 | + |
2051 | + // part 3 |
2052 | + ((FullTextIndexServiceImpl)fullText).disableUpdateWorker(); |
2053 | + |
2054 | + writeRow(c, 8, "Flintstone Hudson"); |
2055 | + writeRow(c, 9, "Jim Flintstone"); |
2056 | + |
2057 | + Thread.sleep(updateInterval * 2); |
2058 | + |
2059 | + // confirm that new rows are not found (ie., expected2 still works) |
2060 | + plan = builder.scanOperator("flintstone", 15); |
2061 | + compareRows(expected2, cursor(plan, queryContext)); |
2062 | + |
2063 | + ((FullTextIndexServiceImpl)fullText).enableUpdateWorker(); |
2064 | + Thread.sleep(updateInterval * 5); |
2065 | + |
2066 | + // now the rows should be seen. |
2067 | + // (Because disabling the worker does not stop the changes fron being recorded) |
2068 | + RowBase expected3[] = new RowBase[] |
2069 | + { |
2070 | + row(rowType, 1L), |
2071 | + row(rowType, 3L), |
2072 | + row(rowType, 5L), |
2073 | + row(rowType, 7L), |
2074 | + row(rowType, 8L), |
2075 | + row(rowType, 9L) |
2076 | + }; |
2077 | + |
2078 | + plan = builder.scanOperator("flintstone", 15); |
2079 | + compareRows(expected3, cursor(plan, queryContext)); |
2080 | + } |
2081 | + |
2082 | + @Test |
2083 | + public void cDown() throws InterruptedException { |
2084 | + FullTextIndex index = createFullTextIndex(serviceManager(), |
2085 | + SCHEMA, "c", "idx_c", |
2086 | + "name", "i.sku", "a.state"); |
2087 | + Thread.sleep(populateDelayInterval * 5); |
2088 | RowType rowType = rowType("c"); |
2089 | RowBase[] expected = new RowBase[] { |
2090 | row(rowType, 1L), |
2091 | @@ -115,14 +403,15 @@ |
2092 | |
2093 | plan = builder.scanOperator("state:MA", 10); |
2094 | compareRows(expected, cursor(plan, queryContext)); |
2095 | + |
2096 | } |
2097 | |
2098 | @Test |
2099 | - public void oUpDown() { |
2100 | - FullTextIndex index = createFullTextIndex(SCHEMA, "o", "idx_o", |
2101 | + public void oUpDown() throws InterruptedException { |
2102 | + FullTextIndex index = createFullTextIndex(serviceManager(), |
2103 | + SCHEMA, "o", "idx_o", |
2104 | "c.name", "i.sku"); |
2105 | - fullText.createIndex(session(), index.getIndexName()); |
2106 | - |
2107 | + Thread.sleep(populateDelayInterval * 5); |
2108 | RowType rowType = rowType("o"); |
2109 | RowBase[] expected = new RowBase[] { |
2110 | row(rowType, 1L, 101L) |
2111 | |
2112 | === modified file 'src/test/java/com/akiban/server/test/ApiTestBase.java' |
2113 | --- src/test/java/com/akiban/server/test/ApiTestBase.java 2013-03-22 20:05:57 +0000 |
2114 | +++ src/test/java/com/akiban/server/test/ApiTestBase.java 2013-04-15 17:23:26 +0000 |
2115 | @@ -520,16 +520,16 @@ |
2116 | * overriding the {@link #startupConfigProperties()} and/or |
2117 | * {@link #serviceBindingsProvider()} methods. |
2118 | */ |
2119 | - protected static Map<String, String> uniqueStartupConfigProperties(Class clazz) { |
2120 | + protected Map<String, String> uniqueStartupConfigProperties(Class clazz) { |
2121 | return Collections.singletonMap( |
2122 | "test.services", |
2123 | clazz.getName() |
2124 | ); |
2125 | } |
2126 | |
2127 | - protected AkibanInformationSchema createFromDDL(String schema, String ddl) { |
2128 | + protected AkibanInformationSchema createFromDDL(ServiceManager sm, String schema, String ddl) { |
2129 | SchemaFactory schemaFactory = new SchemaFactory(schema); |
2130 | - return schemaFactory.ais(ddl().getAIS(session()), ddl); |
2131 | + return schemaFactory.ais(sm, ddl().getAIS(session()), ddl); |
2132 | } |
2133 | |
2134 | protected static final class SimpleColumn { |
2135 | @@ -602,7 +602,7 @@ |
2136 | |
2137 | protected final int createTable(String schema, String table, String definition) throws InvalidOperationException { |
2138 | String ddl = String.format("CREATE TABLE \"%s\" (%s)", table, definition); |
2139 | - AkibanInformationSchema tempAIS = createFromDDL(schema, ddl); |
2140 | + AkibanInformationSchema tempAIS = createFromDDL(serviceManager(), schema, ddl); |
2141 | UserTable tempTable = tempAIS.getUserTable(schema, table); |
2142 | ddl().createTable(session(), tempTable); |
2143 | updateAISGeneration(); |
2144 | @@ -621,7 +621,7 @@ |
2145 | |
2146 | protected final void createSequence (String schema, String name, String definition) { |
2147 | String ddl = String.format("CREATE SEQUENCE %s %s", name, definition); |
2148 | - AkibanInformationSchema tempAIS = createFromDDL(schema, ddl); |
2149 | + AkibanInformationSchema tempAIS = createFromDDL(serviceManager(), schema, ddl); |
2150 | Sequence sequence = tempAIS.getSequence(new TableName(schema, name)); |
2151 | ddl().createSequence(session(), sequence); |
2152 | updateAISGeneration(); |
2153 | @@ -629,7 +629,7 @@ |
2154 | |
2155 | protected final void createView(String schema, String name, String definition) { |
2156 | String ddl = String.format("CREATE VIEW %s AS %s", name, definition); |
2157 | - AkibanInformationSchema tempAIS = createFromDDL(schema, ddl); |
2158 | + AkibanInformationSchema tempAIS = createFromDDL(serviceManager(), schema, ddl); |
2159 | View view = tempAIS.getView(new TableName(schema, name)); |
2160 | ddl().createView(session(), view); |
2161 | updateAISGeneration(); |
2162 | @@ -664,7 +664,7 @@ |
2163 | schema, |
2164 | table, |
2165 | Strings.join(Arrays.asList(indexCols), ",")); |
2166 | - return createFromDDL(schema, ddl); |
2167 | + return createFromDDL(serviceManager(), schema, ddl); |
2168 | } |
2169 | |
2170 | protected final TableIndex createIndex(String schema, String table, String indexName, String... indexCols) { |
2171 | @@ -767,7 +767,7 @@ |
2172 | return ddl().getAIS(session()).getGroup(groupName).getIndex(indexName); |
2173 | } |
2174 | |
2175 | - protected final FullTextIndex createFullTextIndex(String schema, String table, String indexName, String... indexCols) { |
2176 | + protected final FullTextIndex createFullTextIndex(ServiceManager sm, String schema, String table, String indexName, String... indexCols) { |
2177 | AkibanInformationSchema tempAIS = createIndexInternal(schema, table, indexName, "FULL_TEXT(" + Strings.join(Arrays.asList(indexCols), ",") + ")"); |
2178 | Index tempIndex = tempAIS.getUserTable(schema, table).getFullTextIndex(indexName); |
2179 | ddl().createIndexes(session(), Collections.singleton(tempIndex)); |
2180 | @@ -779,10 +779,10 @@ |
2181 | SchemaFactory schemaFactory = new SchemaFactory(schema); |
2182 | |
2183 | // Insert DDL into the System |
2184 | - AkibanInformationSchema ais = schemaFactory.ais(ddl(), session(), ddl); |
2185 | + AkibanInformationSchema ais = schemaFactory.ais(serviceManager(), ddl(), session(), ddl); |
2186 | |
2187 | // sort DDL to find first root table of the user schema |
2188 | - ais = schemaFactory.ais (ddl); |
2189 | + ais = schemaFactory.ais (serviceManager(), ddl); |
2190 | List<UserTable> tables = new ArrayList<>(ais.getUserTables().values()); |
2191 | Collections.sort(tables, new Comparator<UserTable>() { |
2192 | @Override |
2193 | |
2194 | === modified file 'src/test/java/com/akiban/server/test/it/bugs/bug701614/MissingColumnsIT.java' |
2195 | --- src/test/java/com/akiban/server/test/it/bugs/bug701614/MissingColumnsIT.java 2013-03-22 20:05:57 +0000 |
2196 | +++ src/test/java/com/akiban/server/test/it/bugs/bug701614/MissingColumnsIT.java 2013-04-15 17:23:26 +0000 |
2197 | @@ -38,7 +38,7 @@ |
2198 | |
2199 | private int loadBlocksTable() throws InvalidOperationException, IOException { |
2200 | final String blocksDDL = Strings.join(Strings.dumpResource(getClass(), "blocks-table.ddl")); |
2201 | - AkibanInformationSchema tempAIS = createFromDDL("drupal", blocksDDL); |
2202 | + AkibanInformationSchema tempAIS = createFromDDL(serviceManager(), "drupal", blocksDDL); |
2203 | ddl().createTable(session(), tempAIS.getUserTable("drupal", "blocks")); |
2204 | updateAISGeneration(); |
2205 | AkibanInformationSchema ais = ddl().getAIS(session()); |
2206 | |
2207 | === modified file 'src/test/java/com/akiban/sql/optimizer/OptimizerTestBase.java' |
2208 | --- src/test/java/com/akiban/sql/optimizer/OptimizerTestBase.java 2013-03-22 20:05:57 +0000 |
2209 | +++ src/test/java/com/akiban/sql/optimizer/OptimizerTestBase.java 2013-04-15 17:23:26 +0000 |
2210 | @@ -25,18 +25,26 @@ |
2211 | import com.akiban.sql.views.ViewDefinition; |
2212 | |
2213 | import com.akiban.ais.model.AkibanInformationSchema; |
2214 | +import com.akiban.server.service.ServiceManager; |
2215 | +import com.akiban.server.service.config.TestConfigService; |
2216 | import com.akiban.server.service.functions.FunctionsRegistryImpl; |
2217 | |
2218 | +import com.akiban.server.service.servicemanager.GuicedServiceManager; |
2219 | +import com.akiban.server.service.text.FullTextIndexService; |
2220 | +import com.akiban.server.service.text.FullTextIndexServiceImpl; |
2221 | import org.junit.Before; |
2222 | import org.junit.Ignore; |
2223 | |
2224 | import java.io.File; |
2225 | import java.util.Collections; |
2226 | +import java.util.HashMap; |
2227 | import java.util.List; |
2228 | +import java.util.Map; |
2229 | |
2230 | @Ignore |
2231 | public class OptimizerTestBase extends ASTTransformTestBase |
2232 | { |
2233 | + |
2234 | protected OptimizerTestBase(String caseName, String sql, |
2235 | String expected, String error) { |
2236 | super(caseName, sql, expected, error); |
2237 | @@ -47,13 +55,17 @@ |
2238 | + OptimizerTestBase.class.getPackage().getName().replace('.', '/')); |
2239 | public static final String DEFAULT_SCHEMA = "test"; |
2240 | |
2241 | + protected static ServiceManager sm = prepareServices(); |
2242 | + protected static int populateDelayInterval = 1000; |
2243 | + protected static int updateInterval = 3000; |
2244 | + |
2245 | // Base class has all possible transformers for convenience. |
2246 | protected AISBinder binder; |
2247 | protected TypeComputer typeComputer; |
2248 | protected BooleanNormalizer booleanNormalizer; |
2249 | protected SubqueryFlattener subqueryFlattener; |
2250 | protected DistinctEliminator distinctEliminator; |
2251 | - |
2252 | + |
2253 | @Before |
2254 | public void makeTransformers() throws Exception { |
2255 | parser = new SQLParser(); |
2256 | @@ -65,13 +77,19 @@ |
2257 | distinctEliminator = new DistinctEliminator(parser); |
2258 | } |
2259 | |
2260 | + protected static ServiceManager serviceManager() |
2261 | + { |
2262 | + return sm; |
2263 | + } |
2264 | + |
2265 | public static AkibanInformationSchema parseSchema(List<File> ddls) throws Exception { |
2266 | StringBuilder ddlBuilder = new StringBuilder(); |
2267 | for (File ddl : ddls) |
2268 | ddlBuilder.append(fileContents(ddl)); |
2269 | String sql = ddlBuilder.toString(); |
2270 | SchemaFactory schemaFactory = new SchemaFactory(DEFAULT_SCHEMA); |
2271 | - return schemaFactory.ais(sql); |
2272 | + AkibanInformationSchema ret = schemaFactory.ais(serviceManager(), sql); |
2273 | + return ret; |
2274 | } |
2275 | |
2276 | public static AkibanInformationSchema parseSchema(File ddl) throws Exception { |
2277 | @@ -98,4 +116,35 @@ |
2278 | } |
2279 | } |
2280 | |
2281 | + |
2282 | + protected static ServiceManager createServiceManager(Map<String, String> startupConfigProperties) |
2283 | + { |
2284 | + return new GuicedServiceManager(serviceBindingsProvider()); |
2285 | + } |
2286 | + |
2287 | + protected static GuicedServiceManager.BindingsConfigurationProvider serviceBindingsProvider() |
2288 | + { |
2289 | + return GuicedServiceManager |
2290 | + .testUrls() |
2291 | + .bindAndRequire(FullTextIndexService.class, |
2292 | + FullTextIndexServiceImpl.class); |
2293 | + } |
2294 | + |
2295 | + |
2296 | + protected static Map<String, String> startupConfigProperties() { |
2297 | + Map<String, String> properties = new HashMap<>(); |
2298 | + properties.put("akserver.text.indexpath", "/tmp/aktext"); |
2299 | + properties.put(FullTextIndexServiceImpl.UPDATE_INTERVAL, Long.toString(updateInterval)); |
2300 | + properties.put(FullTextIndexServiceImpl.POPULATE_DELAY_INTERVAL, Long.toString(populateDelayInterval)); |
2301 | + return properties; |
2302 | + } |
2303 | + |
2304 | + private static ServiceManager prepareServices() |
2305 | + { |
2306 | + System.setProperty("akiban.home", System.getProperty("user.home")); |
2307 | + ServiceManager ret = createServiceManager(startupConfigProperties()); |
2308 | + ret.startServices(); |
2309 | + |
2310 | + return ret; |
2311 | + } |
2312 | } |
2313 | |
2314 | === added file 'src/test/resources/com/akiban/sql/pg/yaml/functional/test-drop-fulltext.yaml' |
2315 | --- src/test/resources/com/akiban/sql/pg/yaml/functional/test-drop-fulltext.yaml 1970-01-01 00:00:00 +0000 |
2316 | +++ src/test/resources/com/akiban/sql/pg/yaml/functional/test-drop-fulltext.yaml 2013-04-15 17:23:26 +0000 |
2317 | @@ -0,0 +1,8 @@ |
2318 | +# test drop full_text index |
2319 | +--- |
2320 | +- CreateTable: t(id int, name varchar(224)); |
2321 | +--- |
2322 | +- Statement: CREATE INDEX idx on t(full_text(name)); |
2323 | +--- |
2324 | +- Statement: DROP INDEX t.idx; |
2325 | +... |
2326 | \ No newline at end of file |
2327 | |
2328 | === added file 'src/test/resources/com/akiban/sql/pg/yaml/functional/test-ft-with-inherited-key.yaml' |
2329 | --- src/test/resources/com/akiban/sql/pg/yaml/functional/test-ft-with-inherited-key.yaml 1970-01-01 00:00:00 +0000 |
2330 | +++ src/test/resources/com/akiban/sql/pg/yaml/functional/test-ft-with-inherited-key.yaml 2013-04-15 17:23:26 +0000 |
2331 | @@ -0,0 +1,28 @@ |
2332 | +# test full-text updates with inherited keys |
2333 | +--- |
2334 | +- CreateTable: t1(id INT NOT NULL PRIMARY KEY); |
2335 | +--- |
2336 | +- CreateTable: t2(id INT NOT NULL PRIMARY KEY, pid INT NOT NULL, GROUPING FOREIGN KEY(pid) REFERENCES t1(id)); |
2337 | +--- |
2338 | +- CreateTable: t3(id INT NOT NULL PRIMARY KEY, pid INT NOT NULL, GROUPING FOREIGN KEY(pid) REFERENCES t2(id), name VARCHAR(16) COLLATE en_ci); |
2339 | +--- |
2340 | +- Statement: INSERT INTO t1 VALUES(1),(2); |
2341 | +--- |
2342 | +- Statement: INSERT INTO t2 VALUES(11, 1); |
2343 | +--- |
2344 | +- Statement: INSERT INTO t3 VALUES(111, 11, 'Fred'),(112, 11, 'Barney'); |
2345 | +--- |
2346 | +- Statement: CREATE INDEX t3_ft ON t3(FULL_TEXT(name)); |
2347 | +--- |
2348 | +- Statement: SELECT SLEEP(7000); |
2349 | +--- |
2350 | +- Statement: SELECT * FROM t3 WHERE FULL_TEXT_SEARCH(name, 'fred'); |
2351 | +- output: [[111, 11, 'Fred']] |
2352 | +--- |
2353 | +- Statement: UPDATE t2 SET pid = 2 WHERE pid = 1; |
2354 | +--- |
2355 | +- Statement: SELECT SLEEP(7000); |
2356 | +--- |
2357 | +- Statement: SELECT * FROM t3 WHERE FULL_TEXT_SEARCH(name, 'fred'); |
2358 | +- output: [[111, 11, 'Fred']] |
2359 | +... |
2360 | \ No newline at end of file |
2361 | |
2362 | === added file 'src/test/resources/com/akiban/sql/pg/yaml/functional/test-ft-with-pk.yaml' |
2363 | --- src/test/resources/com/akiban/sql/pg/yaml/functional/test-ft-with-pk.yaml 1970-01-01 00:00:00 +0000 |
2364 | +++ src/test/resources/com/akiban/sql/pg/yaml/functional/test-ft-with-pk.yaml 2013-04-15 17:23:26 +0000 |
2365 | @@ -0,0 +1,22 @@ |
2366 | +# Tests with changed primary key |
2367 | +--- |
2368 | +- CreateTable: t (id int primary key not null, name varchar(224)); |
2369 | +--- |
2370 | +- Statement: INSERT INTO t values (1, 'foo1'); |
2371 | +--- |
2372 | +- Statement: INSERT INTO t values (2, 'foo2'); |
2373 | +--- |
2374 | +- Statement: create index idx1 on t (full_text(name)); |
2375 | +--- |
2376 | +- Statement: SELECT SLEEP(7000); |
2377 | +--- |
2378 | +- Statement: SELECT id FROM t where full_text_search(name = 'foo1'); |
2379 | +- output: [[1]] |
2380 | +--- |
2381 | +- Statement: UPDATE t SET id = 3 where id = 1; |
2382 | +--- |
2383 | +- Statement: SELECT SLEEP(7000); |
2384 | +--- |
2385 | +- Statement: SELECT id FROM t where full_text_search(name = 'foo1'); |
2386 | +- output: [[3]] |
2387 | +... |
2388 | \ No newline at end of file |
2389 | |
2390 | === added file 'src/test/resources/com/akiban/sql/pg/yaml/functional/test-fulltext-maintenance.yaml' |
2391 | --- src/test/resources/com/akiban/sql/pg/yaml/functional/test-fulltext-maintenance.yaml 1970-01-01 00:00:00 +0000 |
2392 | +++ src/test/resources/com/akiban/sql/pg/yaml/functional/test-fulltext-maintenance.yaml 2013-04-15 17:23:26 +0000 |
2393 | @@ -0,0 +1,49 @@ |
2394 | +# test that full text indices are updated (and populated) properly |
2395 | +--- |
2396 | +- CreateTable: t (id int, name varchar(224)); |
2397 | +--- |
2398 | +- Statement: INSERT INTO t VALUES (1, 'bar1'); |
2399 | +--- |
2400 | +- Statement: INSERT INTO t VALUES (2, 'bar1'); |
2401 | +--- |
2402 | +- Statement: INSERT INTO t VALUES (3, 'bar3'); |
2403 | +--- |
2404 | +- Statement: INSERT INTO t VALUES (4, 'bar4'); |
2405 | +--- |
2406 | +- Statement: CREATE INDEX idx ON t (full_text(name)); |
2407 | +--- # should be long enough! (Could query the actual interval from AIS, but we'd need to parse the string!) |
2408 | +- Statement: SELECT SLEEP (7000); |
2409 | +--- |
2410 | +- Statement: select id from t where full_text_search(name = 'bar1'); |
2411 | +- output: [[1], [2]] |
2412 | +--- |
2413 | +- Statement: UPDATE t set name='foo' WHERE id = 1; |
2414 | +--- |
2415 | +- Statement: SELECT SLEEP(7000); |
2416 | +--- |
2417 | +- Statement: select id from t where full_text_search(name = 'bar1'); |
2418 | +- output: [[2]] |
2419 | +--- |
2420 | +- Statement: SELECT id from t where full_text_search(name = 'bar4'); |
2421 | +- output: [[4]] |
2422 | +--- |
2423 | +- Statement: DELETE FROM t where id = 4; |
2424 | +--- |
2425 | +- Statement: SELECT SLEEP(7000); |
2426 | +--- |
2427 | +- Statement: SELECT id from t where full_text_search(name = 'bar4'); |
2428 | +- row_count: 0 |
2429 | +--- |
2430 | +- Create |
2431 | +... |
2432 | + |
2433 | +CREATE TABLE t1(id INT NOT NULL PRIMARY KEY); |
2434 | +CREATE TABLE t2(id INT NOT NULL PRIMARY KEY, pid INT NOT NULL, GROUPING FOREIGN KEY(pid) REFERENCES t1(id)); |
2435 | +CREATE TABLE t3(id INT NOT NULL PRIMARY KEY, pid INT NOT NULL, GROUPING FOREIGN KEY(pid) REFERENCES t2(id), name VARCHAR(16) COLLATE en_ci); |
2436 | +INSERT INTO t1 VALUES(1),(2); |
2437 | +INSERT INTO t2 VALUES(11, 1); |
2438 | +INSERT INTO t3 VALUES(111, 11, 'Fred'),(112, 11, 'Barney'); |
2439 | +CREATE INDEX t3_ft ON t3(FULL_TEXT(name)); |
2440 | +SELECT * FROM t3 WHERE FULL_TEXT_SEARCH(name, 'fred'); |
2441 | +UPDATE t2 SET pid = 2 WHERE pid = 1; |
2442 | +SELECT * FROM t3 WHERE FULL_TEXT_SEARCH(name, 'fred'); |
What is the IndexDef needed for?
I'm guessing bug1154336 is quite similar too. Perhaps dropTableInternal() needs to be in on the game.