Merge lp:~tjoneslo/akiban-server/fix-bug-1172013 into lp:~akiban-technologies/akiban-server/trunk

[Thomas Jones-Low]

Fix the w-w conflict in the testing system for the full text indexes. The testing has uncovered a problem which can (if circumstances are right) the end user too.

The core problem is the interaction of the delete index (which tries to delete the full text index) and the background thread populating the index. In order to get this to work correctly, I did two things. One, moved the point at which the index creation process calls the FT service to schedule the background process. Second, update the FT service to make the delete and populate aware of each other, so they are not stepping on each other.

BackgroundWorkBase - Update the internal set of background listeners from a HahsSet to a ConcurrentHashMap. Adding or removing listeners from different threads causes concurrent access exceptions at the worst possible time.

PersistitStore - This is where the FT service schedulePopulate now gets called. This in in the transaction to create the index itself, so the populate process gets flagged at the same time as the index creation, and doesn't if the index creation fails.

PersistitStoreSchemaManager - Part of the process of managing the index creation means the FT indexes need to be passed to the PeristitStore for processing.

IndexDDL - This is where previously the FT service sechedulePopulate was called. Since the only reason for the ServiceManager to be passed through to here was to get the FT Service, remove the now unused parameter. This has a cascade on the callers to remove the unused parameter as well.

FullTextIndexServiceBug1172013IT - This is the test for the bug, with three variations.

[Nathan Williams (nwilliams)]

There seems to be unrelated changes related to REST and session handling. Are those ready to go or shall we revert for now?

Otherwise looks as described, still thinking through though.

[Mike McMahon (mmcm)]

There is no semantic difference, but lines 81-87 of the diff could be just one switch on getIndexType(), I believe.

Unmerged revisions

2658. By tjoneslo on 2013-05-03

Update the FullTextIndexServiceImpl code to handle the background task populating the full text indexes and managing the process of deleteing the index while the populate is going on. Plus test.

=== modified file 'src/main/java/com/akiban/http/HttpConductorImpl.java'
--- src/main/java/com/akiban/http/HttpConductorImpl.java	2013-04-30 21:19:31 +0000
+++ src/main/java/com/akiban/http/HttpConductorImpl.java	2013-05-09 13:22:31 +0000
@@ -21,6 +21,7 @@
 import com.akiban.server.service.config.ConfigurationService;
 import com.akiban.server.service.monitor.MonitorService;
 import com.akiban.server.service.monitor.ServerMonitor;
+import com.akiban.server.service.monitor.SessionMonitor;
 import com.akiban.server.service.security.SecurityService;
 import com.akiban.server.service.session.Session;
 import com.akiban.server.service.session.SessionService;
@@ -387,9 +388,9 @@
             ServerSessionMonitor sessionMonitor = new ServerSessionMonitor(SERVER_TYPE,
                     monitorService.allocateSessionId());
 
-            conn.setAssociatedObject(sessionMonitor);
             this.session = sessionService.createSession();
             monitorService.registerSessionMonitor(sessionMonitor, session);
+            conn.setAssociatedObject(session);
             return conn;
         }
         
@@ -397,10 +398,12 @@
         protected void connectionClosed(Connection connection) {
             if (connection instanceof AsyncHttpConnection) {
                 AsyncHttpConnection conn = (AsyncHttpConnection)connection;
-                ServerSessionMonitor monitor = (ServerSessionMonitor)conn.getAssociatedObject();
+                
+                SessionMonitor monitor = monitorService.getSessionMonitor(session);
                 if (monitor != null) {
                     monitorService.deregisterSessionMonitor(monitor, session);
                     conn.setAssociatedObject(null);
+                    session.close();
                 }
             }
             super.connectionClosed(connection);
@@ -420,9 +423,9 @@
             ServerSessionMonitor sessionMonitor = new ServerSessionMonitor(SERVER_TYPE,
                     monitorService.allocateSessionId());
 
-            conn.setAssociatedObject(sessionMonitor);
             this.session = sessionService.createSession();
             monitorService.registerSessionMonitor(sessionMonitor, session);
+            conn.setAssociatedObject(session);
             return conn;
         }
         
@@ -430,10 +433,11 @@
         protected void connectionClosed (Connection connection) {
             if (connection instanceof SslConnection) {
                 AsyncHttpConnection conn = (AsyncHttpConnection)((SslConnection) connection).getSslEndPoint().getConnection();
-                ServerSessionMonitor monitor = (ServerSessionMonitor)conn.getAssociatedObject();
+                SessionMonitor monitor = monitorService.getSessionMonitor(session);
                 if (monitor != null) {
                     monitorService.deregisterSessionMonitor(monitor, session);
                     conn.setAssociatedObject(null);
+                    session.close();
                 }
             }
             super.connectionClosed(connection);

=== modified file 'src/main/java/com/akiban/qp/persistitadapter/OperatorStore.java'
--- src/main/java/com/akiban/qp/persistitadapter/OperatorStore.java	2013-04-30 22:35:43 +0000
+++ src/main/java/com/akiban/qp/persistitadapter/OperatorStore.java	2013-05-09 13:22:31 +0000
@@ -18,6 +18,7 @@
 package com.akiban.qp.persistitadapter;
 
 import com.akiban.ais.model.*;
+import com.akiban.ais.model.Index.IndexType;
 import com.akiban.qp.exec.UpdatePlannable;
 import com.akiban.qp.exec.UpdateResult;
 import com.akiban.qp.expression.IndexBound;
@@ -222,11 +223,11 @@
 
     @Override
     public void buildIndexes(Session session, Collection<? extends Index> indexes, boolean defer) {
-        List<TableIndex> tableIndexes = new ArrayList<>();
+        List<Index> tableIndexes = new ArrayList<>();
         List<GroupIndex> groupIndexes = new ArrayList<>();
         for(Index index : indexes) {
-            if(index.isTableIndex()) {
-                tableIndexes.add((TableIndex)index);
+            if(index.isTableIndex() || index.getIndexType() == IndexType.FULL_TEXT) {
+                tableIndexes.add(index);
             }
             else if(index.isGroupIndex()) {
                 groupIndexes.add((GroupIndex)index);

=== modified file 'src/main/java/com/akiban/rest/RestResponseBuilder.java'
--- src/main/java/com/akiban/rest/RestResponseBuilder.java	2013-04-26 15:25:38 +0000
+++ src/main/java/com/akiban/rest/RestResponseBuilder.java	2013-05-09 13:22:31 +0000
@@ -128,7 +128,6 @@
         builder.append("\"}");
     }
 
-
     private String formatErrorWithJsonp(String code, String message) {
         StringBuilder builder = new StringBuilder();
         if(isJsonp) {

=== modified file 'src/main/java/com/akiban/rest/resources/EntityResource.java'
--- src/main/java/com/akiban/rest/resources/EntityResource.java	2013-04-23 14:20:47 +0000
+++ src/main/java/com/akiban/rest/resources/EntityResource.java	2013-05-09 13:22:31 +0000
@@ -36,6 +36,9 @@
 import javax.ws.rs.core.MediaType;
 import javax.ws.rs.core.Response;
 import javax.ws.rs.core.UriInfo;
+
+import org.eclipse.jetty.server.Request;
+
 import java.io.PrintWriter;
 
 import static com.akiban.rest.resources.ResourceHelper.IDENTIFIERS_MULTI;

=== modified file 'src/main/java/com/akiban/rest/resources/ModelResource.java'
--- src/main/java/com/akiban/rest/resources/ModelResource.java	2013-04-23 11:04:53 +0000
+++ src/main/java/com/akiban/rest/resources/ModelResource.java	2013-05-09 13:22:31 +0000
@@ -35,6 +35,8 @@
 import com.akiban.util.tap.InOutTap;
 import com.akiban.util.tap.Tap;
 import com.fasterxml.jackson.databind.JsonNode;
+
+import org.eclipse.jetty.server.Request;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -88,14 +90,14 @@
                               @PathParam("schema") String schemaParam) {
         final String schema = getSchemaName(request, schemaParam);
         checkSchemaAccessible(reqs.securityService, request, schema);
+        final Session session = (Session)request.getAttribute(Session.class.getName());
         return RestResponseBuilder
                 .forRequest(request)
                 .body(new RestResponseBuilder.BodyGenerator() {
                     @Override
                     public void write(PrintWriter writer) throws Exception {
                         MODEL_VIEW.in();
-                        try (Session session = reqs.sessionService.createSession();
-                             CloseableTransaction txn = reqs.transactionService.beginCloseableTransaction(session)) {
+                        try ( CloseableTransaction txn = reqs.transactionService.beginCloseableTransaction(session)) {
                             Space space = spaceForAIS(session, schema);
                             String json = space.toJson();
                             writer.write(json);

=== modified file 'src/main/java/com/akiban/rest/resources/ResourceHelper.java'
--- src/main/java/com/akiban/rest/resources/ResourceHelper.java	2013-03-26 18:08:04 +0000
+++ src/main/java/com/akiban/rest/resources/ResourceHelper.java	2013-05-09 13:22:31 +0000
@@ -19,12 +19,16 @@
 
 import com.akiban.ais.model.TableName;
 import com.akiban.server.service.security.SecurityService;
+import com.akiban.server.service.session.Session;
 
 import javax.servlet.http.HttpServletRequest;
 import javax.ws.rs.WebApplicationException;
 import javax.ws.rs.core.MediaType;
 import javax.ws.rs.core.Response;
 import javax.ws.rs.core.UriInfo;
+
+import org.eclipse.jetty.server.Request;
+
 import java.security.Principal;
 
 public class ResourceHelper {
@@ -61,6 +65,11 @@
         }
     }
 
+    public static Session getSession (Request request) {
+        Object obj = request.getConnection().getAssociatedObject();
+        assert obj instanceof Session;
+        return (Session)obj;
+    }
     /** Expected to be used along with {@link #IDENTIFIERS_MULTI} */
     public static String getPKString(UriInfo uri) {
         String pks[] = uri.getPath(false).split("/");

=== modified file 'src/main/java/com/akiban/server/service/BackgroundWorkBase.java'
--- src/main/java/com/akiban/server/service/BackgroundWorkBase.java	2013-04-08 19:34:41 +0000
+++ src/main/java/com/akiban/server/service/BackgroundWorkBase.java	2013-05-09 13:22:31 +0000
@@ -18,12 +18,11 @@
 package com.akiban.server.service;
 
 import java.util.Collection;
-import java.util.HashSet;
-import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
 
 public abstract class BackgroundWorkBase implements BackgroundWork<BackgroundObserver, BackgroundWork>
 {
-    private final Set<BackgroundObserver> observers = new HashSet<>();
+    private final ConcurrentHashMap<BackgroundObserver,Boolean> observers = new ConcurrentHashMap<>();
     
     public BackgroundWorkBase()
     {
@@ -34,7 +33,7 @@
     public void addObserver(BackgroundObserver observer)
     {
         // two observers are equal IFF they are the same object.
-        observers.add(observer);
+        observers.put(observer, true);
     }
 
     @Override
@@ -45,19 +44,21 @@
 
     public void removeObsevers(Collection<BackgroundObserver> os)
     {
-        observers.removeAll(os);
+        for (BackgroundObserver key : os) {
+            observers.remove(key);
+        }
     }
 
     @Override
     public void removeAllObservers()
     {
-        observers.removeAll(observers);
+        observers.clear();
     }
 
     @Override
     public void notifyObservers()
     {
-        for (BackgroundObserver o : observers)
+        for (BackgroundObserver o : observers.keySet())
             o.update(this);
     }
 }

=== modified file 'src/main/java/com/akiban/server/service/dxl/BasicDDLFunctions.java'
--- src/main/java/com/akiban/server/service/dxl/BasicDDLFunctions.java	2013-04-22 02:00:55 +0000
+++ src/main/java/com/akiban/server/service/dxl/BasicDDLFunctions.java	2013-05-09 13:22:31 +0000
@@ -959,8 +959,8 @@
             // Cannot use Index.getAllTableIDs(), stub AIS only has to be name-correct
             for(Index index : indexesToAdd) {
                 switch(index.getIndexType()) {
+                    case FULL_TEXT: // TODO: More IDs?
                     case TABLE:
-                    case FULL_TEXT: // TODO: More IDs?
                         UserTable table = ais.getUserTable(index.getIndexName().getFullTableName());
                         if(table != null) {
                             tableIDs.add(table.getTableId());

=== modified file 'src/main/java/com/akiban/server/service/dxl/HookableDDLFunctions.java'
--- src/main/java/com/akiban/server/service/dxl/HookableDDLFunctions.java	2013-04-11 00:05:29 +0000
+++ src/main/java/com/akiban/server/service/dxl/HookableDDLFunctions.java	2013-05-09 13:22:31 +0000
@@ -39,7 +39,6 @@
 
 import static com.akiban.ais.util.TableChangeValidator.ChangeLevel;
 import static com.akiban.util.Exceptions.throwAlways;
-import static com.akiban.util.Exceptions.throwIfInstanceOf;
 
 public final class HookableDDLFunctions implements DDLFunctions {
 

=== modified file 'src/main/java/com/akiban/server/service/text/FullTextIndexService.java'
--- src/main/java/com/akiban/server/service/text/FullTextIndexService.java	2013-04-22 18:37:21 +0000
+++ src/main/java/com/akiban/server/service/text/FullTextIndexService.java	2013-05-09 13:22:31 +0000
@@ -35,7 +35,7 @@
      * created.
      * @param name 
      */
-    public void schedulePopulate(String schema, String table, String index);
+    public void schedulePopulate(Session session, IndexName name);
 
     /**
      * Update the given index based on the changedRow 

=== modified file 'src/main/java/com/akiban/server/service/text/FullTextIndexServiceImpl.java'
--- src/main/java/com/akiban/server/service/text/FullTextIndexServiceImpl.java	2013-04-29 15:59:33 +0000
+++ src/main/java/com/akiban/server/service/text/FullTextIndexServiceImpl.java	2013-05-09 13:22:31 +0000
@@ -34,6 +34,7 @@
 import com.akiban.qp.rowtype.HKeyRowType;
 import com.akiban.qp.util.HKeyCache;
 import com.akiban.server.error.AkibanInternalException;
+import com.akiban.server.error.QueryCanceledException;
 import com.akiban.server.service.BackgroundWork;
 import com.akiban.server.service.BackgroundWorkBase;
 import com.akiban.server.service.Service;
@@ -47,6 +48,7 @@
 import com.akiban.server.store.PersistitStore;
 import com.akiban.server.store.PersistitStore.HKeyBytesStream;
 import com.akiban.server.store.Store;
+import com.akiban.server.types3.mcompat.mfuncs.WaitFunctionHelpers;
 
 import org.apache.lucene.index.IndexWriter;
 import org.apache.lucene.search.Query;
@@ -64,6 +66,7 @@
 import java.util.List;
 import java.util.Timer;
 import java.util.TimerTask;
+import java.util.concurrent.ConcurrentHashMap;
 
 
 public class FullTextIndexServiceImpl extends FullTextIndexInfosImpl implements FullTextIndexService, Service {
@@ -92,7 +95,8 @@
 
     private volatile Timer populateTimer;
     private long populateDelayInterval;
-
+    private ConcurrentHashMap<IndexName,Session> populating;
+    
     private static final Logger logger = LoggerFactory.getLogger(FullTextIndexServiceImpl.class);
 
     @Inject
@@ -105,6 +109,7 @@
         this.store = store;
         this.transactionService = transactionService;
         this.treeService = treeService;
+        this.populating = new ConcurrentHashMap<> ();
 
     }
 
@@ -140,8 +145,24 @@
     
     @Override
     public void dropIndex(Session session, FullTextIndex idx) {
-        // delete 'promise' for population, if any
-        deleteFromTree(session, idx.getIndexName());
+
+        logger.error("Delete {}", idx.getIndexName());
+        
+        Session populatingSession = populating.putIfAbsent(idx.getIndexName(), session);
+        if (populatingSession != null) {
+            // if the population process is running, cancel it
+            populatingSession.cancelCurrentQuery(true);
+            // wait for the thread to complete 
+            try {
+                WaitFunctionHelpers.waitOn(getBackgroundWorks());
+            } catch (InterruptedException e) {
+                ;// TODO: do nothing
+            }
+        } else {
+            // delete 'promise' for population, if any
+            deleteFromTree(session, idx.getIndexName());
+            populating.remove(idx.getIndexName());
+        }
         
         // delete documents
         FullTextIndexInfo idxInfo = getIndex(session, idx.getIndexName(), idx.getIndexedTable().getAIS());
@@ -149,6 +170,7 @@
         synchronized (indexes) {    
             indexes.remove(idx.getIndexName());
         }
+        
     }
 
     @Override
@@ -321,33 +343,26 @@
     
  
     @Override
-    public void schedulePopulate(String schema, String table, String index)
+    public void schedulePopulate(Session session, IndexName name)
     {   
-        Session session = sessionService.createSession();
-        boolean success = false;
+        logger.debug("Scheduled populate {}", name.toString());
+        
         try
         {
-            transactionService.beginTransaction(session);
-            if(addPopulate(session, schema, table, index) && !hasScheduled && populateEnabled)
-            {
-                populateTimer.schedule(populateWorker(), populateDelayInterval);
-                hasScheduled = true;
+            // Add the index to the list (in persisit) of indexes to build
+            addPopulate(session, name);
+
+            // if there are no scheduled populate workers running, 
+            // add one to run shortly. 
+            if (populateEnabled && !hasScheduled) {
+                 populateTimer.schedule(populateWorker(), populateDelayInterval);
+                 hasScheduled = true;
             }
-
-            success = true;            
         }
         catch (PersistitException ex)
         {
             throw new AkibanInternalException("Error while scheduling index population", ex);
         }
-        finally
-        {
-            if (success)
-                transactionService.commitTransaction(session);
-            else
-                transactionService.rollbackTransaction(session);
-            session.close();
-        }
     }
     
     private final List<? extends BackgroundWork> backgroundWorks
@@ -457,26 +472,45 @@
 
 
     //----------- private helpers -----------
-    private synchronized void runPopulate()
+    protected boolean populateNextIndex(Session session) throws PersistitException 
+    {
+        IndexName toPopulate = null;
+        transactionService.beginTransaction(session);
+        Exchange ex = null; 
+        try {
+            ex = getPopulateExchange(session);
+            
+            toPopulate = nextInQueue(session, ex, false);
+            if (toPopulate != null && stillExists (session, toPopulate)) {
+                createIndex(session, toPopulate);
+                ex.remove();
+                transactionService.commitTransaction(session);
+                populating.remove(toPopulate);
+                return true;
+            }
+        } catch (QueryCanceledException e) {
+            // The query could be canceled if the user drops the index  
+            // while this thread is populating the index
+            // Clean up after ourselves. 
+            if (ex != null) ex.remove();
+            populating.remove(toPopulate);
+            transactionService.commitTransaction(session);
+            // start another thread to make sure we're not missing anything
+            populateTimer.schedule(populateWorker(), populateDelayInterval);
+            hasScheduled = true;
+        } finally {
+            transactionService.rollbackTransactionIfOpen(session);
+        }
+        return false;
+    }
+    
+    protected synchronized void runPopulate()
     {
         populateRunning = true;
         Session session = sessionService.createSession();
-        boolean transaction = true;
         try
         {
-            transactionService.beginTransaction(session);
-            Exchange ex = getPopulateExchange(session);
-            IndexName toPopulate;
-            while ((toPopulate = nextInQueue(ex)) != null)
-            {
-                if (stillExists(session, toPopulate))
-                    createIndex(session, toPopulate);
-                else
-                    logger.debug("FullTextIndex " + toPopulate + " deleted before population");
-            }
-            ex.removeAll();
-            hasScheduled = false;
-            transaction = false;
+            while (populateNextIndex(session)) {}
         }
         catch (PersistitException ex1)
         {
@@ -484,11 +518,7 @@
         }
         finally
         {
-            if (transaction)
-                transactionService.rollbackTransaction(session);
-            else
-                transactionService.commitTransaction(session);
-            session.close();
+            hasScheduled = false;
             backgroundWorks.get(populateWork).notifyObservers();
             populateRunning = false;
         }
@@ -595,20 +625,28 @@
         return new DefaultPopulateWorker();
     }
 
-    protected IndexName nextInQueue(Exchange ex) throws PersistitException
+    protected IndexName nextInQueue(Session session, Exchange ex, boolean traversing) throws PersistitException
     {
         Key key = ex.getKey();
 
-        if (ex.next(true)) // empty tree?
+        while (ex.next(true))
         {
             key.reset();
             IndexName ret = new IndexName(new TableName(key.decodeString(),
                                                         key.decodeString()),
                                           key.decodeString());
+            // The populating map contains the indexes currently being built
+            // if this name is already in the tree, skip this one, and try
+            // the next.
+            if (!traversing) {
+                if (populating.putIfAbsent(ret, session) != null) {
+                    continue;
+                }
+            }
+            
             return ret;
         }
-        else
-            return null;
+        return null;
     }
     
     protected Exchange getPopulateExchange(Session session) throws PersistitException
@@ -624,10 +662,7 @@
     private volatile boolean hasScheduled = false;
     private volatile boolean populateEnabled = false;
 
-    private synchronized boolean addPopulate(Session session,
-                           String schema,
-                           String table,
-                           String index) throws PersistitException
+    private synchronized boolean addPopulate(Session session, IndexName name) throws PersistitException
     {
         Exchange ex = getPopulateExchange(session);
 
@@ -635,9 +670,9 @@
         // (Because they should have been removed in dropIndex())
         // KEY: schema | table | indexName
         ex.getKey().clear()
-                   .append(schema)
-                   .append(table)
-                   .append(index);
+                   .append(name.getSchemaName())
+                   .append(name.getTableName())
+                   .append(name.getName());
 
         // VALUE: <empty>
 

=== modified file 'src/main/java/com/akiban/server/store/PersistitStore.java'
--- src/main/java/com/akiban/server/store/PersistitStore.java	2013-05-03 00:59:49 +0000
+++ src/main/java/com/akiban/server/store/PersistitStore.java	2013-05-09 13:22:31 +0000
@@ -1824,14 +1824,20 @@
         Map<Integer,RowDef> userRowDefs = new HashMap<>();
         Set<Index> indexesToBuild = new HashSet<>();
         for(Index index : indexes) {
-            IndexDef indexDef = index.indexDef();
-            if(indexDef == null) {
-                throw new IllegalArgumentException("indexDef was null for index: " + index);
+            if (index.getIndexType() == IndexType.FULL_TEXT) {
+                // This schedules a deferred process to populate the
+                // full text index at a later date (starting in a few seconds).
+                fullTextService.schedulePopulate(session, index.getIndexName());
+            } else {
+                IndexDef indexDef = index.indexDef();
+                if(indexDef == null) {
+                    throw new IllegalArgumentException("indexDef was null for index: " + index);
+                }
+                indexesToBuild.add(index);
+                RowDef rowDef = indexDef.getRowDef();
+                userRowDefs.put(rowDef.getRowDefId(), rowDef);
+                groups.add(rowDef.table().getGroup());
             }
-            indexesToBuild.add(index);
-            RowDef rowDef = indexDef.getRowDef();
-            userRowDefs.put(rowDef.getRowDefId(), rowDef);
-            groups.add(rowDef.table().getGroup());
         }
         PersistitIndexRowBuffer indexRow = new PersistitIndexRowBuffer(adapter(session));
         for (Group group : groups) {

=== modified file 'src/main/java/com/akiban/server/store/PersistitStoreSchemaManager.java'
--- src/main/java/com/akiban/server/store/PersistitStoreSchemaManager.java	2013-04-11 18:35:20 +0000
+++ src/main/java/com/akiban/server/store/PersistitStoreSchemaManager.java	2013-05-09 13:22:31 +0000
@@ -389,10 +389,7 @@
             if(keepTree) {
                 newIndex.setTreeName(proposed.getTreeName());
             }
-            if (newIndex.getIndexType() != Index.IndexType.FULL_TEXT) {
-                // TODO: For now, do not build.
-                newIndexes.add(newIndex);
-            }
+            newIndexes.add(newIndex);
             tableIDs.addAll(newIndex.getAllTableIDs());
             schemas.add(DefaultNameGenerator.schemaNameForIndex(newIndex));
         }

=== modified file 'src/main/java/com/akiban/sql/aisddl/AISDDL.java'
--- src/main/java/com/akiban/sql/aisddl/AISDDL.java	2013-03-27 04:42:10 +0000
+++ src/main/java/com/akiban/sql/aisddl/AISDDL.java	2013-05-09 13:22:31 +0000
@@ -84,7 +84,7 @@
                              server.getBinderContext(), context);
             return;
         case NodeTypes.CREATE_INDEX_NODE:
-            IndexDDL.createIndex(ddlFunctions, session, schema, (CreateIndexNode)ddl, server.getServiceManager());
+            IndexDDL.createIndex(ddlFunctions, session, schema, (CreateIndexNode)ddl);
             return;
         case NodeTypes.DROP_INDEX_NODE:
             IndexDDL.dropIndex(ddlFunctions, session, schema, (DropIndexNode)ddl, context);

=== modified file 'src/main/java/com/akiban/sql/aisddl/IndexDDL.java'
--- src/main/java/com/akiban/sql/aisddl/IndexDDL.java	2013-04-15 17:21:44 +0000
+++ src/main/java/com/akiban/sql/aisddl/IndexDDL.java	2013-05-09 13:22:31 +0000
@@ -36,8 +36,6 @@
 import com.akiban.server.service.session.Session;
 
 import com.akiban.qp.operator.QueryContext;
-import com.akiban.server.service.ServiceManager;
-import com.akiban.server.service.text.FullTextIndexService;
 
 /** DDL operations on Indices */
 public class IndexDDL
@@ -150,18 +148,17 @@
     public static void createIndex(DDLFunctions ddlFunctions,
                                    Session session,
                                    String defaultSchemaName,
-                                   CreateIndexNode createIndex,
-                                   ServiceManager sm)  {
+                                   CreateIndexNode createIndex)  {
         AkibanInformationSchema ais = ddlFunctions.getAIS(session);
         
         Collection<Index> indexesToAdd = new LinkedList<>();
 
-        indexesToAdd.add(buildIndex(ais, defaultSchemaName, createIndex, sm));
+        indexesToAdd.add(buildIndex(ais, defaultSchemaName, createIndex));
         
         ddlFunctions.createIndexes(session, indexesToAdd);
     }
     
-    protected static Index buildIndex (AkibanInformationSchema ais, String defaultSchemaName, CreateIndexNode index, ServiceManager sm) {
+    protected static Index buildIndex (AkibanInformationSchema ais, String defaultSchemaName, CreateIndexNode index){
         final String schemaName = index.getObjectName().getSchemaName() != null ? index.getObjectName().getSchemaName() : defaultSchemaName;
         final String indexName = index.getObjectName().getTableName();
 
@@ -176,7 +173,7 @@
         
         if (index.getColumnList().functionType() == IndexColumnList.FunctionType.FULL_TEXT) {
             logger.debug ("Building Full text index on table {}", tableName) ;
-            tableIndex = buildFullTextIndex (builder, tableName, indexName, index, sm);
+            tableIndex = buildFullTextIndex (builder, tableName, indexName, index);
         } else if (checkIndexType (index, tableName) == Index.IndexType.TABLE) {
             logger.debug ("Building Table index on table {}", tableName) ;
             tableIndex = buildTableIndex (builder, tableName, indexName, index);
@@ -331,9 +328,8 @@
         return builder.akibanInformationSchema().getGroup(groupName).getIndex(indexName);
     }
 
-    protected static Index buildFullTextIndex (AISBuilder builder, TableName tableName, String indexName, IndexDefinition index, ServiceManager sm) {
+    protected static Index buildFullTextIndex (AISBuilder builder, TableName tableName, String indexName, IndexDefinition index) {
         UserTable table = builder.akibanInformationSchema().getUserTable(tableName);
-        FullTextIndexService fullTextService = sm.getServiceByClass(FullTextIndexService.class);
         
         if (index.getJoinType() != null) {
             throw new TableIndexJoinTypeException();
@@ -367,8 +363,6 @@
             }
             
             builder.fullTextIndexColumn(tableName, indexName, schemaName, columnTable.getTableName(), columnName, i);
-            // populate index
-            fullTextService.schedulePopulate(schemaName, tableName.getTableName(), indexName);
             i++;
         }
         return builder.akibanInformationSchema().getUserTable(tableName).getFullTextIndex(indexName);

=== modified file 'src/main/java/com/akiban/sql/aisddl/TableDDL.java'
--- src/main/java/com/akiban/sql/aisddl/TableDDL.java	2013-04-28 04:20:16 +0000
+++ src/main/java/com/akiban/sql/aisddl/TableDDL.java	2013-05-09 13:22:31 +0000
@@ -48,7 +48,6 @@
 import com.akiban.sql.parser.ConstraintDefinitionNode;
 import com.akiban.sql.parser.CreateTableNode;
 import com.akiban.sql.parser.CurrentDatetimeOperatorNode;
-import com.akiban.sql.parser.DefaultNode;
 import com.akiban.sql.parser.DropGroupNode;
 import com.akiban.sql.parser.DropTableNode;
 import com.akiban.sql.parser.ExistenceCheck;
@@ -487,7 +486,7 @@
 
         if (columnList.functionType() == IndexColumnList.FunctionType.FULL_TEXT) {
             logger.debug ("Building Full text index on table {}", table.getName()) ;
-            tableIndex = IndexDDL.buildFullTextIndex (builder, table.getName(), indexName, id, context.getServiceManager());
+            tableIndex = IndexDDL.buildFullTextIndex (builder, table.getName(), indexName, id);
         } else if (IndexDDL.checkIndexType (id, table.getName()) == Index.IndexType.TABLE) {
             logger.debug ("Building Table index on table {}", table.getName()) ;
             tableIndex = IndexDDL.buildTableIndex (builder, table.getName(), indexName, id);

=== modified file 'src/test/java/com/akiban/server/rowdata/SchemaFactory.java'
--- src/test/java/com/akiban/server/rowdata/SchemaFactory.java	2013-04-15 16:20:25 +0000
+++ src/test/java/com/akiban/server/rowdata/SchemaFactory.java	2013-05-09 13:22:31 +0000
@@ -125,7 +125,7 @@
             if (stmt instanceof CreateTableNode) {
                 TableDDL.createTable(ddlFunctions , session , defaultSchema, (CreateTableNode) stmt, null);
             } else if (stmt instanceof CreateIndexNode) {
-                IndexDDL.createIndex(ddlFunctions, session, defaultSchema, (CreateIndexNode) stmt, sm);
+                IndexDDL.createIndex(ddlFunctions, session, defaultSchema, (CreateIndexNode) stmt);
             } else if (stmt instanceof CreateViewNode) {
                 ViewDDL.createView(ddlFunctions, session, defaultSchema, (CreateViewNode) stmt,
                                    new AISBinderContext(ddlFunctions.getAIS(session), defaultSchema), null);

=== added file 'src/test/java/com/akiban/server/service/text/FullTextIndexServiceBug1172013IT.java'
--- src/test/java/com/akiban/server/service/text/FullTextIndexServiceBug1172013IT.java	1970-01-01 00:00:00 +0000
+++ src/test/java/com/akiban/server/service/text/FullTextIndexServiceBug1172013IT.java	2013-05-09 13:22:31 +0000
@@ -0,0 +1,296 @@
+/**
+ * Copyright (C) 2009-2013 Akiban Technologies, Inc.
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+package com.akiban.server.service.text;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.akiban.ais.model.IndexName;
+import com.akiban.ais.model.TableName;
+import com.akiban.qp.operator.QueryContext;
+import com.akiban.qp.persistitadapter.PersistitAdapter;
+import com.akiban.qp.rowtype.Schema;
+import com.akiban.qp.util.SchemaCache;
+import com.akiban.server.error.DuplicateIndexColumnException;
+import com.akiban.server.error.DuplicateIndexException;
+import com.akiban.server.service.servicemanager.GuicedServiceManager;
+import com.akiban.server.service.session.Session;
+import com.akiban.server.service.session.SessionServiceImpl;
+import com.akiban.server.test.it.ITBase;
+import com.akiban.server.types3.mcompat.mfuncs.WaitFunctionHelpers;
+import com.akiban.sql.embedded.EmbeddedJDBCService;
+import com.akiban.sql.embedded.EmbeddedJDBCServiceImpl;
+import com.persistit.Exchange;
+import com.persistit.exception.PersistitException;
+
+public class FullTextIndexServiceBug1172013IT extends ITBase {
+    public static final String SCHEMA = "test";
+    protected FullTextIndexService fullText;
+    protected Schema schema;
+    protected PersistitAdapter adapter;
+    protected QueryContext queryContext;
+    private int c;
+    private int o;
+    private int i;
+    private int a;
+    private static final Logger logger = LoggerFactory.getLogger(FullTextIndexServiceBug1172013IT.class);
+
+    @Override
+    protected GuicedServiceManager.BindingsConfigurationProvider serviceBindingsProvider() {
+        return super.serviceBindingsProvider()
+                .bindAndRequire(FullTextIndexService.class, FullTextIndexServiceImpl.class)
+                .bindAndRequire(EmbeddedJDBCService.class, EmbeddedJDBCServiceImpl.class);
+    }
+
+    @Before
+    public void createData() {
+        c = createTable(SCHEMA, "c",
+                        "cid INT PRIMARY KEY NOT NULL",
+                        "name VARCHAR(128) COLLATE en_us_ci");
+        o = createTable(SCHEMA, "o",
+                        "oid INT PRIMARY KEY NOT NULL",
+                        "cid INT NOT NULL",
+                        "c1  VARCHAR(128) COLLATE en_us_ci",
+                        "c2  VARCHAR(128) COLLATE en_us_ci",
+                        "c3  VARCHAR(128) COLLATE en_us_ci",
+                        "c4  VARCHAR(128) COLLATE en_us_ci",
+                        "GROUPING FOREIGN KEY(cid) REFERENCES c(cid)",
+                        "order_date DATE");
+        i = createTable(SCHEMA, "i",
+                        "iid INT PRIMARY KEY NOT NULL",
+                        "oid INT NOT NULL",
+                        "GROUPING FOREIGN KEY(oid) REFERENCES o(oid)",
+                        "sku VARCHAR(10) NOT NULL");
+        a = createTable(SCHEMA, "a",
+                        "aid INT PRIMARY KEY NOT NULL",
+                        "cid INT NOT NULL",
+                        "GROUPING FOREIGN KEY(cid) REFERENCES c(cid)",
+                        "state CHAR(2)");
+        writeRow(c, 1, "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam ultrices justo in sapien ullamcorper eu mattis massa pretium.");
+        writeRow(o, 101, 1, "c1", "c2", "c3", "c4", "2012-12-12");
+        writeRow(i, 10101, 101, "ABCD");
+        writeRow(i, 10102, 101, "1234");
+        writeRow(o, 102, 1, "c1", "c2", "c3", "c4","2013-01-01");
+        writeRow(a, 101, 1, "MA");
+        writeRow(c, 2, "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam ultrices justo in sapien ullamcorper eu mattis massa pretium.");
+        writeRow(a, 201, 2, "NY");
+        writeRow(c, 3, "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam ultrices justo in sapien ullamcorper eu mattis massa pretium.");
+        writeRow(o, 301, 3, "c1", "c2", "c3", "c4", "2010-04-01");
+        writeRow(a, 301, 3, "MA");
+        writeRow(a, 302, 3, "ME");
+
+        fullText = serviceManager().getServiceByClass(FullTextIndexService.class);
+
+        schema = SchemaCache.globalSchema(ais());
+        adapter = persistitAdapter(schema);
+        queryContext = queryContext(adapter);
+    }
+
+    @Test
+    public void testDelete1 () throws InterruptedException, PersistitException {
+        logger.error("Running test delete 1");
+        // This test is specifically for FullTextIndexServiceImpl.java
+        assertEquals(FullTextIndexServiceImpl.class, fullText.getClass());
+        FullTextIndexServiceImpl fullTextImpl = (FullTextIndexServiceImpl)fullText;
+
+        // disable the populate worker (so it doesn't read all the entries
+        // out before we get a chance to look at the tree.
+        createFullTextIndex(serviceManager(),
+                SCHEMA, "o", "idx3_o",
+                "oid", "c1", "c2", "c3", "c4");
+        //fullTextImpl.getBackgroundWorks().get(0).forceExecution();
+        //WaitFunctionHelpers.waitOn(fullText.getBackgroundWorks());
+        
+        new Thread(new DropIndex()).start();
+        
+        try {
+            createFullTextIndex(serviceManager(),
+                    SCHEMA, "o", "idx3_o",
+                    "oid", "c1", "c2", "c3", "c4");
+        } catch (DuplicateIndexException ex) {
+            logger.error("Got Duplicate Index");
+            ; // an expected possible outcome. 
+        } catch (DuplicateIndexColumnException ex) {
+            logger.error("Got Duplicate Column");
+            ; // an expected possible outcome
+        }
+
+         WaitFunctionHelpers.waitOn(fullText.getBackgroundWorks());
+
+         traverse(fullTextImpl,
+                 new Visitor()
+                 {
+                     int n = 0;
+                    
+                     @Override
+                     public void visit(IndexName idx)
+                     {
+                         ++n;
+                     }
+
+                     @Override
+                     public void endOfTree()
+                     {
+                         assertEquals (0, n);
+                     } 
+                 });
+
+    }
+    
+    @Test
+    public void testDelete2() throws InterruptedException, PersistitException {
+        logger.error("Running test delete 2");
+        // This test is specifically for FullTextIndexServiceImpl.java
+        assertEquals(FullTextIndexServiceImpl.class, fullText.getClass());
+        FullTextIndexServiceImpl fullTextImpl = (FullTextIndexServiceImpl)fullText;
+
+        createFullTextIndex(serviceManager(),
+                SCHEMA, "o", "idx3_o",
+                "oid", "c1", "c2", "c3", "c4");
+        fullTextImpl.getBackgroundWorks().get(0).forceExecution();
+        WaitFunctionHelpers.waitOn(fullText.getBackgroundWorks());
+        
+        new Thread(new DropIndex()).start();
+        
+        try {
+            createFullTextIndex(serviceManager(),
+                    SCHEMA, "o", "idx3_o",
+                    "oid", "c1", "c2", "c3", "c4");
+        } catch (DuplicateIndexException ex) {
+            logger.error("Got Duplicate Index");
+            ; // an expected possible outcome. 
+        } catch (DuplicateIndexColumnException ex) {
+            logger.error("Got Duplicate Column");
+            ; // an expected possible outcome
+        }
+
+         WaitFunctionHelpers.waitOn(fullText.getBackgroundWorks());
+
+         traverse(fullTextImpl,
+                 new Visitor()
+                 {
+                     int n = 0;
+                    
+                     @Override
+                     public void visit(IndexName idx)
+                     {
+                         ++n;
+                     }
+
+                     @Override
+                     public void endOfTree()
+                     {
+                         assertEquals (0, n);
+                     } 
+                 });
+
+
+    }
+    
+    @Test
+    public void testDelete3() throws InterruptedException, PersistitException {
+        logger.error("Running test delete 3");
+        // This test is specifically for FullTextIndexServiceImpl.java
+        assertEquals(FullTextIndexServiceImpl.class, fullText.getClass());
+        FullTextIndexServiceImpl fullTextImpl = (FullTextIndexServiceImpl)fullText;
+
+        createFullTextIndex(serviceManager(),
+                SCHEMA, "o", "idx3_o",
+                "oid", "c1", "c2", "c3", "c4");
+        fullTextImpl.getBackgroundWorks().get(0).forceExecution();
+        //WaitFunctionHelpers.waitOn(fullText.getBackgroundWorks());
+        
+        deleteFullTextIndex(serviceManager(), new IndexName(new TableName(SCHEMA, "o"), "idx3_o"));
+        
+         WaitFunctionHelpers.waitOn(fullText.getBackgroundWorks());
+
+         traverse(fullTextImpl,
+                 new Visitor()
+                 {
+                     int n = 0;
+                    
+                     @Override
+                     public void visit(IndexName idx)
+                     {
+                         ++n;
+                     }
+
+                     @Override
+                     public void endOfTree()
+                     {
+                         assertEquals (0, n);
+                     } 
+                 });
+
+
+    }
+    
+    
+    private static interface Visitor
+    {
+        void visit(IndexName idx);
+        void endOfTree();
+    }
+
+    private class DropIndex implements Runnable
+    {
+        @Override
+        public void run()
+        {
+            Connection conn;
+            try {
+                conn = DriverManager.getConnection("jdbc:default:connection", "test", "");
+                conn.createStatement().execute("DROP INDEX test.o.idx3_o");
+            } catch (SQLException e) {
+                logger.error("drop index failed; {}", e.getMessage());
+                assertTrue ("Drop index failed", false);
+             }
+            //IndexName name = new IndexName (new TableName(SCHEMA, "o"), "idx3_o");
+            //deleteFullTextIndex(serviceManager(), name);
+        }
+    }; 
+
+    
+    
+    private static void traverse(FullTextIndexServiceImpl serv,
+            Visitor visitor) throws PersistitException
+    {
+        Session session = new SessionServiceImpl().createSession();
+
+        try
+        {
+            Exchange ex = serv.getPopulateExchange(session);
+            IndexName toPopulate;
+            while ((toPopulate = serv.nextInQueue(session, ex, true)) != null)
+            visitor.visit(toPopulate);
+            visitor.endOfTree();
+        }
+        finally
+        {
+            session.close();
+        }
+    }
+}

=== modified file 'src/test/java/com/akiban/server/service/text/FullTextIndexServiceIT.java'
--- src/test/java/com/akiban/server/service/text/FullTextIndexServiceIT.java	2013-04-22 19:01:34 +0000
+++ src/test/java/com/akiban/server/service/text/FullTextIndexServiceIT.java	2013-05-09 13:22:31 +0000
@@ -38,10 +38,11 @@
 import com.persistit.exception.PersistitException;
 import org.junit.Before;
 import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
 import static org.junit.Assert.*;
 
-import java.util.*;
-
 public class FullTextIndexServiceIT extends ITBase
 {
     public static final String SCHEMA = "test";
@@ -49,6 +50,8 @@
     protected Schema schema;
     protected PersistitAdapter adapter;
     protected QueryContext queryContext;
+    private static final Logger logger = LoggerFactory.getLogger(FullTextIndexServiceIT.class);
+
 
     private int c;
     private int o;
@@ -108,6 +111,7 @@
         assertEquals(FullTextIndexServiceImpl.class, fullText.getClass());
         FullTextIndexServiceImpl fullTextImpl = (FullTextIndexServiceImpl)fullText;
         
+        logger.error ("DDL Service: {}", ddl().getClass().getCanonicalName());
         // disable the populate worker (so it doesn't read all the entries
         // out before we get a chance to look at the tree.
         fullTextImpl.disablePopulateWorker();
@@ -246,7 +250,7 @@
          {
              Exchange ex = serv.getPopulateExchange(session);
              IndexName toPopulate;
-             while ((toPopulate = serv.nextInQueue(ex)) != null)
+             while ((toPopulate = serv.nextInQueue(session, ex, true)) != null)
                  visitor.visit(toPopulate);
              visitor.endOfTree();
          }