Merge lp:~posulliv/drizzle/simple-tc-engine into lp:~drizzle-trunk/drizzle/development

[Padraig O'Sullivan]

[Padraig O'Sullivan (posulliv)]

This merge proposal consists of a simple in-memory storage engine that uses the Tokyo Cabinet in-memory hash API. It also includes a very lacking test suite with 2 small test files for the storage engine. Since this proposal is a little bit out of nowhere and you are probably wondering why the heck I did this, let me explain why I did it...

I'm taking a distributed systems class this semester for fun which will involve a project. I've managed to get a project involving Drizzle approved. What I plan on doing is investigating how to use cloud storage services (such as S3) as the storage backend for a SQL engine (which will be drizzle) and how to implement various consistency protocols over an eventually consistent data store such as S3. Since S3 is basically a key/value store (as described in the Dynamo paper), I wanted to get some practice creating a storage engine using a key/value store interface. Thus, I found Brian's posting on a Tokyo Cabinet based storage engine from a while back and went with that. I ported it to Drizzle and modified it to use the in-memory API calls from Tokyo Cabinet for the heck of it. This will help me in getting my basic S3 storage engine working in the next few weeks as I now have a good idea of how to create a storage engine using a key/value store interface.

I wasn't sure what to do with this engine, so I figured I'd propose it for merging. I also remember Brian's post about a new memory engine so I thought this might make a good start. I'm open to giving this over to someone. I was also under the impression this doesn't really conflict with BlitzDB. From what I understand about BlitzDB, its aimed at being a more general purpose storage engine but then again, maybe I should give this to Toru and he can do what he likes with it.

Anyway, any comments are more than welcome. If no-one is interested in this, or it doesn't prove very useful, well then this proposal can be ignored :) It was good practice for working with a key/value store for me! Alternatively, if there is interest, I can work on it more in my spare time to make it a better in-memory engine if that is desirable.

-Padraig

[Brian Aker (brianaker)]

Hi!

Yep, we need a new in-memory engine. I can't really judge whether TC would work well for in-memory or not (we need to study up the analytics around this... aka benchmark).

My personal thought? No harm in doing it, and I suspect it will be no worse then the current engine (might even be an improvement...). Is it the final engine? I have my doubts mainly because of concurrency issues. You can certainly learn the interface this way, and... most likely this would be useful for others.

Cheers,
   -Brian

[Stewart Smith (stewart)]

On Fri, Sep 25, 2009 at 04:45:48AM -0000, Brian Aker wrote:
> Hi!
>
> Yep, we need a new in-memory engine. I can't really judge whether TC
> would work well for in-memory or not (we need to study up the analytics
> around this... aka benchmark).
>
> My personal thought? No harm in doing it, and I suspect it will be no
> worse then the current engine (might even be an improvement...). Is it
> the final engine? I have my doubts mainly because of concurrency issues.
> You can certainly learn the interface this way, and... most likely this
> would be useful for others.

I also really like the idea of maintaining an engine with a clear
boundary between engine API and storage engine code as all of our
existing engines are a bit muddy there and/or rather complex.

--
Stewart Smith

[Stewart Smith (stewart)]

the delete table paths seem to be a bit strange. it looks like you're deleting it on the close of the last tableshare, although the proper way would be to do it when explicitly called to remove the table (and check exists/not exists).

If you wanted to be fancy and demo the data dictionary APIs, you could have a (on disk) tc db of table names with the data being the table proto.

review: Needs Information

Unmerged revisions

1155. By Padraig O'Sullivan on 2009-09-24

Merge from trunk.

1154. By Padraig O'Sullivan on 2009-09-23

Added some more doxygen comments.

1153. By Padraig O'Sullivan on 2009-09-23

Updated plugin main file to have the authors reflected accurately. Included
Brian and Stewart since I based this engine off the tokyo cabinet engine
they did.

1152. By Padraig O'Sullivan on 2009-09-23

Converted the engine to an in-memory engine. It would be quite trivial to
create an option which lets the engine operate in memory or on disk. We
could even support having tables of both types at the same time. However, I
don't see why you would need a disk-based storage engine when you have
InnoDB?

1151. By Padraig O'Sullivan on 2009-09-23

Added some doxygen comments and updated the plugin.ini file to indicate that
a test suite is available for this plugin. Thus, the test suite will get run
when 'make test' issued.

1150. By Padraig O'Sullivan on 2009-09-23

Tiny modification based on valgrind output.

1149. By Padraig O'Sullivan on 2009-09-23

Updating the flags based to the tokyo cabinet based storage engine so that
altering of tables is supported.

1148. By Padraig O'Sullivan on 2009-09-23

Corrected the way tables are updated. Update statements now work correctly.

1147. By Padraig O'Sullivan on 2009-09-23

Added support for deleting a row in the storage engine.

1146. By Padraig O'Sullivan on 2009-09-23

Added support for storing rows of non-fixed length to the storage engine.
Updates to rows which store blobs still don't work correctly but we are
making some progress.

=== added directory 'plugin/tokyo_cabinet_engine'
=== added file 'plugin/tokyo_cabinet_engine/open_tables.cc'
--- plugin/tokyo_cabinet_engine/open_tables.cc	1970-01-01 00:00:00 +0000
+++ plugin/tokyo_cabinet_engine/open_tables.cc	2009-09-24 03:50:23 +0000
@@ -0,0 +1,76 @@
+/*
+ *  Copyright (C) 2009 Sun Microsystems
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; if not, write to the Free Software
+ *  Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ */
+
+#include <drizzled/server_includes.h>
+#include "tokyo_cabinet_handler.h"
+#include "open_tables.h"
+
+#include <string>
+#include <map>
+
+using namespace std;
+using namespace drizzled;
+
+TokyoCabinetShare *OpenTables::getShare(const string &name)
+{
+  map<const string, TokyoCabinetShare *>::iterator it;
+  pthread_mutex_lock(&mutex);
+
+  it= open_tables.find(name);
+  if (it != open_tables.end())
+  {
+    share= (*it).second;
+  }
+  else
+  {
+    share= NULL;
+  }
+
+  if (! share)
+  {
+    share= new(std::nothrow) TokyoCabinetShare(name);
+    if (! share)
+    {
+      pthread_mutex_unlock(&mutex);
+      return NULL;
+    }
+    if (! share->openTable())
+    {
+      delete share;
+      return NULL;
+    }
+    open_tables[name]= share;
+    share->initThreadLock();
+  }
+  share->incUseCount();
+  pthread_mutex_unlock(&mutex);
+
+  return share;
+}
+
+void OpenTables::freeShare()
+{
+  pthread_mutex_lock(&mutex);
+  share->decUseCount();
+  if (share->getUseCount() == 0)
+  {
+    share->freeDatabaseHandler();
+    open_tables.erase(share->getName());
+  }
+  pthread_mutex_unlock(&mutex);
+}
 
=== added file 'plugin/tokyo_cabinet_engine/open_tables.h'
--- plugin/tokyo_cabinet_engine/open_tables.h	1970-01-01 00:00:00 +0000
+++ plugin/tokyo_cabinet_engine/open_tables.h	2009-09-24 03:50:23 +0000
@@ -0,0 +1,92 @@
+/* 
+ *  Copyright (C) 2009 Sun Microsystems
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; version 2 of the License.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; if not, write to the Free Software
+ *  Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ */
+
+/**
+ * @file 
+ *
+ * Defines a class for how open tables will be tracked for this storage
+ * engine.
+ */
+
+#ifndef DRIZZLE_PLUGIN_TOKYO_ENGINE_OPEN_TABLES_H
+#define DRIZZLE_PLUGIN_TOKYO_ENGINE_OPEN_TABLES_H
+
+#include "tokyo_cabinet_handler.h"
+
+#include <string>
+#include <vector>
+#include <map>
+
+/**
+ * @class OpenTables
+ *
+ * Class which tracks all the open tables for the storage engine.
+ * Encapsulating this functionality in a class will make it easier for us to
+ * change things such as whether a std::map or HASH is used to lookup the
+ * open tables.
+ */
+class OpenTables
+{
+public:
+
+  static OpenTables &singleton()
+  {
+    static OpenTables open_tabs;
+    return open_tabs;
+  }
+
+  /**
+   * Retrieve a share from the structure used to main the list of open
+   * shares. We do a lookup by name.
+   *
+   * @param[in] name name to lookup
+   * @return pointer to the appropriate table share
+   */
+  TokyoCabinetShare *getShare(const std::string &name);
+
+  /**
+   * Erase a share from the structure used to maintain the list of shares.
+   */
+  void freeShare();
+
+private:
+
+  pthread_mutex_t mutex;
+
+  std::map<const std::string, TokyoCabinetShare *>
+    open_tables;
+
+  TokyoCabinetShare *share;
+
+  OpenTables()
+    :
+      mutex(),
+      open_tables(),
+      share(NULL)
+  {
+    pthread_mutex_init(&mutex, MY_MUTEX_INIT_FAST);
+  }
+
+  ~OpenTables()
+  {
+    pthread_mutex_destroy(&mutex);
+  }
+
+  OpenTables(const OpenTables&);
+};
+
+#endif /* DRIZZLE_PLUGIN_TOKYO_ENGINE_OPEN_TABLES_H */
 
=== added file 'plugin/tokyo_cabinet_engine/plugin.ac'
--- plugin/tokyo_cabinet_engine/plugin.ac	1970-01-01 00:00:00 +0000
+++ plugin/tokyo_cabinet_engine/plugin.ac	2009-09-24 03:50:23 +0000
@@ -0,0 +1,8 @@
+AC_LIB_HAVE_LINKFLAGS(tokyocabinet,,
+  [#include <tchdb.h>],
+  [
+     TCHDB hdb;
+  ])
+  AS_IF([test "x$ac_cv_libtokyocabinet" = "xno"],
+        AC_MSG_WARN([tokyocabinet not found: not building TokyoCabinet engine.]))
+DRIZZLED_PLUGIN_DEP_LIBS="${DRIZZLED_PLUGIN_DEP_LIBS} ${LTLIBTOKYOCABINET}"
 
=== added file 'plugin/tokyo_cabinet_engine/plugin.ini'
--- plugin/tokyo_cabinet_engine/plugin.ini	1970-01-01 00:00:00 +0000
+++ plugin/tokyo_cabinet_engine/plugin.ini	2009-09-24 03:50:23 +0000
@@ -0,0 +1,12 @@
+[plugin]
+name=tokyo_cabinet_engine
+title=Tokyo Cabinet Storage Engine
+description=A Tokyo Cabinet Based Storage Engine 
+load_by_default= yes
+sources=tokyo_cabinet_engine.cc tokyo_cabinet_handler.cc open_tables.cc
+        tokyo_cabinet_share.cc
+headers=tokyo_cabinet_engine.h tokyo_cabinet_handler.h open_tables.h
+        tokyo_cabinet_share.h
+build_conditional="${ac_cv_libtokyocabinet}" = "yes"
+ldflags=${LTLIBTOKYOCABINET}
+testsuite=tc_engine
 
=== added file 'plugin/tokyo_cabinet_engine/tokyo_cabinet_engine.cc'
--- plugin/tokyo_cabinet_engine/tokyo_cabinet_engine.cc	1970-01-01 00:00:00 +0000
+++ plugin/tokyo_cabinet_engine/tokyo_cabinet_engine.cc	2009-09-24 03:50:23 +0000
@@ -0,0 +1,71 @@
+/*
+ *  Copyright (C) 2009 Sun Microsystems
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; if not, write to the Free Software
+ *  Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ */
+
+#include <drizzled/server_includes.h>
+#include <drizzled/show.h>
+#include "tokyo_cabinet_engine.h"
+#include "tokyo_cabinet_handler.h"
+#include "open_tables.h"
+
+#include <string>
+#include <map>
+
+using namespace std;
+using namespace drizzled;
+
+static const string engine_name("Tokyo");
+static StorageEngine *tokyo_cabinet_engine= NULL;
+
+static int init(plugin::Registry &registry)
+{
+  tokyo_cabinet_engine= new(std::nothrow) TokyoCabinetEngine(engine_name);
+  if (! tokyo_cabinet_engine)
+  {
+    return true;
+  }
+
+  /* we are good to go */
+  registry.add(tokyo_cabinet_engine);
+
+  return false;
+}
+
+static int deinit(plugin::Registry &registry)
+{
+  if (tokyo_cabinet_engine)
+  {
+    registry.remove(tokyo_cabinet_engine);
+    delete tokyo_cabinet_engine;
+  }
+  return false;
+}
+
+drizzle_declare_plugin(tokyo_cabinet_engine)
+{
+  "tokyo_cabinet_engine",
+  "0.1",
+  "Padraig O'Sullivan, Brian Aker, Stewart Smith",
+  "Simple In-Memory Tokyo Cabinet Storage Engine",
+  PLUGIN_LICENSE_GPL,
+  init,   /* Plugin Init      */
+  deinit, /* Plugin Deinit    */
+  NULL,   /* status variables */
+  NULL,   /* system variables */
+  NULL    /* config options   */
+}
+drizzle_declare_plugin_end;
 
=== added file 'plugin/tokyo_cabinet_engine/tokyo_cabinet_engine.h'
--- plugin/tokyo_cabinet_engine/tokyo_cabinet_engine.h	1970-01-01 00:00:00 +0000
+++ plugin/tokyo_cabinet_engine/tokyo_cabinet_engine.h	2009-09-24 03:50:23 +0000
@@ -0,0 +1,73 @@
+/*
+ *  Copyright (C) 2009 Sun Microsystems
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; if not, write to the Free Software
+ *  Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ */
+
+/**
+ * @file
+ *
+ * Declares the class that will represent this storage engine.
+ */
+
+#ifndef DRIZZLE_PLUGIN_TOKYO_CABINET_ENGINE_H
+#define DRIZZLE_PLUGIN_TOKYO_CABINET_ENGINE_H
+
+#include <drizzled/error.h>
+#include "tokyo_cabinet_handler.h"
+
+#include <string>
+
+class TokyoCabinetEngine : public StorageEngine
+{
+public:
+
+  TokyoCabinetEngine(const std::string &in_name)
+    :
+      StorageEngine(in_name,
+                    HTON_CAN_RECREATE)
+  {}
+
+  ~TokyoCabinetEngine() {}
+
+  virtual handler *create(TableShare *table,
+                          MEM_ROOT *mem_root)
+  {
+    return new(mem_root) TokyoCabinetHandler(this, table);
+  }
+
+  const char **bas_ext() const
+  {
+    return NULL;
+  }
+
+  int createTableImplementation(Session *,
+                                const char *,
+                                Table *,
+                                HA_CREATE_INFO *,
+                                drizzled::message::Table *)
+  {
+    return 0;
+  }
+
+  int deleteTableImplementation(Session *,
+                                const string)
+  {
+    return 0;
+  }
+
+};
+
+#endif /* DRIZZLE_PLUGIN_TOKYO_CABINET_ENGINE_H */
 
=== added file 'plugin/tokyo_cabinet_engine/tokyo_cabinet_handler.cc'
--- plugin/tokyo_cabinet_engine/tokyo_cabinet_handler.cc	1970-01-01 00:00:00 +0000
+++ plugin/tokyo_cabinet_engine/tokyo_cabinet_handler.cc	2009-09-24 03:50:23 +0000
@@ -0,0 +1,221 @@
+/*
+ *  Copyright (C) 2009 Sun Microsystems
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; if not, write to the Free Software
+ *  Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ */
+
+#include "drizzled/server_includes.h"
+#include "drizzled/join_table.h"
+#include "drizzled/handler.h"
+#include "drizzled/field/blob.h"
+#include "drizzled/field/timestamp.h"
+#include "drizzled/session.h"
+
+#include "tokyo_cabinet_handler.h"
+#include "tokyo_cabinet_share.h"
+#include "open_tables.h"
+
+#include <string>
+#include <map>
+
+using namespace std;
+using namespace drizzled;
+
+int TokyoCabinetHandler::open(const char *name, int, uint32_t)
+{
+  OpenTables &open_tables= OpenTables::singleton();
+  if (! (share= open_tables.getShare(name)))
+  {
+    return HA_ERR_OUT_OF_MEM;
+  }
+
+  assert(share);
+
+  thr_lock_data_init(share->getThreadLock(), &data_lock, NULL);
+
+  bool pack_table= ! (table->s->db_create_options &
+                      HA_OPTION_PACK_RECORD);
+  share->setFixedLengthTable(pack_table);
+  share->setRowLength(getMaxRowLength());
+  share->setTableFields(table->field);
+  share->setNullBytes(table->s->null_bytes);
+  share->setTableRecord(table->record[0]);
+
+  return 0;
+}
+
+int TokyoCabinetHandler::close()
+{
+  OpenTables &open_tables= OpenTables::singleton();
+  open_tables.freeShare();
+  return 0;
+}
+
+int TokyoCabinetHandler::info(uint32_t flags)
+{
+  /*
+   * If this flag is set, then we know that the time of the last
+   * modification needs to be modified. 
+   */
+  if (flags & HA_STATUS_TIME)
+  {
+  }
+
+  if (flags & HA_STATUS_VARIABLE)
+  {
+    stats.records= share->getNumOfRecords();
+  }
+
+  /*
+   * If this flag is set, then we know that the auto_increment value needs
+   * to be updated.
+   */
+  if (flags & HA_STATUS_AUTO)
+  {
+    stats.auto_increment_value= 1;
+  }
+
+  return 0;
+}
+
+int TokyoCabinetHandler::rnd_init(bool)
+{
+  share->initIterators();
+  return 0;
+}
+
+int TokyoCabinetHandler::rnd_next(unsigned char *buf)
+{
+  return (share->retrieveRow(buf));
+}
+
+int TokyoCabinetHandler::rnd_pos(unsigned char *buf, unsigned char *pos)
+{
+  Field *field_key= table->key_info[table->s->primary_key].key_part->field;
+  uint16_t len;
+  memcpy(&len, pos, sizeof(uint16_t));
+  int ret= share->findRow(field_key, buf, pos + 2, len);
+
+  return ret;
+}
+
+void TokyoCabinetHandler::position(const unsigned char *)
+{
+  return;
+}
+
+THR_LOCK_DATA **TokyoCabinetHandler::store_lock(Session *session,
+                                                THR_LOCK_DATA **to,
+                                                enum thr_lock_type lock_type)
+{
+  /* borrowed from archive and BDB engines. TODO: understand this! */
+  if (lock_type != TL_IGNORE && data_lock.type == TL_UNLOCK)
+  {
+    if ((lock_type >= TL_WRITE_CONCURRENT_INSERT && lock_type <= TL_WRITE) &&
+        ! session_tablespace_op(session))
+    {
+      lock_type= TL_WRITE_ALLOW_WRITE;
+    }
+    if (lock_type == TL_READ_NO_INSERT)
+    {
+      lock_type= TL_READ;
+    }
+    data_lock.type= lock_type;
+  }
+  *to++= &data_lock;
+  return to;
+}
+
+uint32_t TokyoCabinetHandler::index_flags(uint32_t,
+                                          uint32_t,
+                                          bool) const
+{
+  return 0;
+}
+
+int TokyoCabinetHandler::index_read(unsigned char *buf,
+                                    const unsigned char *key,
+                                    uint32_t key_len,
+                                    enum ha_rkey_function)
+{
+  Field *key_field= table->key_info[active_index].key_part->field;
+  ha_statistic_increment(&SSV::ha_read_key_count);
+  int ret= share->findRow(key_field, buf, key, key_len);
+  return ret;
+}
+
+int TokyoCabinetHandler::index_next(unsigned char *)
+{
+  return HA_ERR_END_OF_FILE;
+}
+
+int TokyoCabinetHandler::write_row(unsigned char *buf)
+{
+  Field *key_field= table->key_info[table->s->primary_key].key_part->field;
+
+  ha_statistic_increment(&SSV::ha_write_count);
+
+  if (table->next_number_field && buf == table->record[0])
+  {
+    update_auto_increment();
+  }
+
+  /* push the data to the key/value store */
+  return (share->storeRow(key_field, buf));
+}
+
+int TokyoCabinetHandler::update_row(const unsigned char *,
+                                    unsigned char *new_data)
+{
+  Field *key_field= table->key_info[table->s->primary_key].key_part->field;
+
+  ha_statistic_increment(&SSV::ha_update_count);
+
+  if (table->timestamp_field_type & TIMESTAMP_AUTO_SET_ON_UPDATE)
+  {
+    table->timestamp_field->set_time();
+  }
+
+  return (share->updateRow(key_field, new_data));
+}
+
+int TokyoCabinetHandler::delete_row(const unsigned char *)
+{
+  Field *key_field= table->key_info[table->s->primary_key].key_part->field;
+  ha_statistic_increment(&SSV::ha_delete_count);
+  return (share->deleteRow(key_field));
+}
+
+int TokyoCabinetHandler::delete_all_rows()
+{
+  share->deleteAllRows();
+  return 0;
+}
+
+uint32_t TokyoCabinetHandler::getMaxRowLength()
+{
+  uint32_t len= table->s->reclength + table->s->fields * 2;
+  uint32_t *pos= table->getBlobField();
+  uint32_t *end= pos + table->sizeBlobFields();
+
+  while (pos != end)
+  {
+    Field_blob *blob_field= static_cast<Field_blob *>(table->field[*pos]);
+    len+= 2 + blob_field->get_length();
+    pos++;
+  }
+
+  return len;
+}
 
=== added file 'plugin/tokyo_cabinet_engine/tokyo_cabinet_handler.h'
--- plugin/tokyo_cabinet_engine/tokyo_cabinet_handler.h	1970-01-01 00:00:00 +0000
+++ plugin/tokyo_cabinet_engine/tokyo_cabinet_handler.h	2009-09-24 03:50:23 +0000
@@ -0,0 +1,140 @@
+/* 
+ *  Copyright (C) 2009 Sun Microsystems
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; if not, write to the Free Software
+ *  Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ */
+
+/**
+ * @file
+ *
+ * Declares the handler for the Tokyo Cabinet storage engine.
+ */
+
+#ifndef DRIZZLE_PLUGIN_TOKYO_CABINET_HANDLER_H
+#define DRIZZLE_PLUGIN_TOKYO_CABINET_HANDLER_H
+
+#include <drizzled/handler.h>
+#include <drizzled/show.h>
+#include <mysys/thr_lock.h>
+
+#include "tokyo_cabinet_share.h"
+
+class TokyoCabinetHandler : public handler
+{
+public:
+
+  /* determines how many indexes are allowed per table */
+  static const uint32_t MAX_KEYS= 1;
+  static const uint32_t MAX_KEY_PARTS= 1;
+  static const uint32_t MAX_KEY_LEN= 1024;
+
+  TokyoCabinetHandler(StorageEngine *engine_arg, TableShare *table_arg)
+    :
+      handler(engine_arg, table_arg)
+  {}
+
+  ~TokyoCabinetHandler() {}
+
+  const char *table_type() const
+  {
+    return "TOKYO";
+  }
+
+  /*
+    The name of the index type that will be used for display
+    don't implement this method unless you really have indexes
+  */
+  const char *index_type(uint32_t)
+  {
+    return "HASH";
+  }
+
+  uint64_t table_flags() const
+  {
+    return (HA_NO_TRANSACTIONS |
+            HA_REQUIRE_PRIMARY_KEY |
+            HA_PRIMARY_KEY_REQUIRED_FOR_POSITION |
+            HA_PRIMARY_KEY_REQUIRED_FOR_DELETE);
+  }
+
+  /* open a table */
+  int open(const char *name, int mode, uint32_t test_if_locked);
+
+  /* close a table */
+  int close(void);
+
+  /* provide info to the optimizer */
+  int info(uint32_t flag);
+
+  /* preparation for table scan (is this true?) */
+  int rnd_init(bool scan);
+
+  /* get the next row and copy it into buf */
+  int rnd_next(unsigned char *buf);
+
+  /* locate row pointed to by pos, copy it into buf */
+  int rnd_pos(unsigned char *buf, unsigned char *pos);
+
+  /* record position of a record for reordering */
+  void position(const unsigned char *record);
+
+  THR_LOCK_DATA **store_lock(Session *session,
+                             THR_LOCK_DATA **to,
+                             enum thr_lock_type lock_type);
+
+  uint32_t index_flags(uint32_t idx, uint32_t part, bool all_parts) const;
+  int index_read(unsigned char *buf,
+                 const unsigned char *key,
+                 uint32_t key_len,
+                 enum ha_rkey_function find_flag);
+  int index_next(unsigned char *buf);
+
+  int write_row(unsigned char *buf);
+  int update_row(const unsigned char *,
+                 unsigned char *);
+  int delete_row(const unsigned char *);
+  int delete_all_rows();
+
+  uint32_t max_supported_keys() const
+  {
+    return MAX_KEYS;
+  }
+
+  uint32_t max_supported_key_parts() const
+  {
+    return MAX_KEY_PARTS;
+  }
+
+  uint32_t max_supported_key_length() const
+  {
+    return MAX_KEY_LEN;
+  }
+
+private:
+
+  /**
+   * Determine the maximum length of a row in the table we are working with.
+   * This function is "borrowed" from the archive storage engine.
+   *
+   * @return the max row length for a table
+   */
+  uint32_t getMaxRowLength();
+
+  THR_LOCK_DATA data_lock;
+  TokyoCabinetShare *share;
+
+};
+
+#endif /* DRIZZLE_PLUGIN_TOKYO_CABINET_HANDLER_H */
 
=== added file 'plugin/tokyo_cabinet_engine/tokyo_cabinet_share.cc'
--- plugin/tokyo_cabinet_engine/tokyo_cabinet_share.cc	1970-01-01 00:00:00 +0000
+++ plugin/tokyo_cabinet_engine/tokyo_cabinet_share.cc	2009-09-24 03:50:23 +0000
@@ -0,0 +1,240 @@
+/* 
+ *  Copyright (C) 2009 Sun Microsystems
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; if not, write to the Free Software
+ *  Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ */
+
+#include "drizzled/server_includes.h"
+#include "drizzled/handler.h"
+#include "drizzled/show.h"
+#include "mysys/thr_lock.h"
+
+#include <tcutil.h>
+#include <tchdb.h>
+#include <stdlib.h>
+#include <stdbool.h>
+#include <stdint.h>
+
+#include "tokyo_cabinet_share.h"
+
+using namespace std;
+
+bool TokyoCabinetShare::createDatabaseHandler()
+{
+  data_handler= tcmdbnew();
+  if (! data_handler)
+  {
+    return false;
+  }
+  return true;
+}
+
+bool TokyoCabinetShare::openTable()
+{
+  return true;
+}
+
+void TokyoCabinetShare::freeDatabaseHandler()
+{
+  tcmdbdel(data_handler);
+}
+
+int TokyoCabinetShare::storeRow(Field *key_field,
+                                unsigned char *row)
+{
+  createKey(key_field);
+
+  unsigned char *buf= new(std::nothrow) unsigned char[row_len];
+  if (! buf)
+  {
+    return HA_ERR_OUT_OF_MEM;
+  }
+
+  size_t len= packRow(buf, row);
+
+  bool ret= tcmdbputkeep(data_handler,
+                         primary_key,
+                         primary_key_len,
+                         buf,
+                         len);
+  if (! ret)
+  {
+    delete buf;
+    return HA_ERR_GENERIC;
+  }
+
+  delete buf;
+  return 0;
+}
+
+int TokyoCabinetShare::updateRow(Field *key_field,
+                                 unsigned char *row)
+{
+  createKey(key_field);
+
+  unsigned char *buf= new(std::nothrow) unsigned char[row_len];
+  if (! buf)
+  {
+    return HA_ERR_OUT_OF_MEM;
+  }
+
+  size_t len= packRow(buf, row);
+
+  /* TODO: somehow, check if this operation succeeded */
+  tcmdbput(data_handler,
+           primary_key,
+           primary_key_len,
+           buf,
+           len);
+
+  delete buf;
+  return 0;
+}
+
+size_t TokyoCabinetShare::packRow(unsigned char *buf,
+                                  unsigned char *row)
+{
+  size_t len= 0;
+
+  if (fixed_length_table)
+  {
+    memcpy(buf, row, row_len);
+    len= row_len;
+  }
+  else
+  {
+    /* copy NULL bits */
+    memcpy(buf, row, null_bytes);
+    unsigned char *pos= buf + null_bytes;
+
+    for (Field **field= fields; *field; field++)
+    {
+      if (! ((*field)->is_null()))
+      {
+        pos= (*field)->pack(pos, row + (*field)->offset(row));
+      }
+    }
+    len= (size_t) (pos - buf);
+  }
+  return len;
+}
+
+void TokyoCabinetShare::unpackRow(unsigned char *buf,
+                                  const char *row)
+{
+  if (fixed_length_table)
+  {
+    memcpy(buf, row, row_len);
+  }
+  else
+  {
+    const unsigned char *pos= (const unsigned char *) row;
+    memcpy(buf, pos, null_bytes);
+    pos+= null_bytes;
+
+    for (Field **field= fields; *field; field++)
+    {
+      if (! (*field)->is_null())
+      {
+        pos= (*field)->unpack(buf + (*field)->offset(table_record), pos);
+      }
+    }
+  }
+}
+
+void TokyoCabinetShare::createKey(Field *key)
+{
+  char *pos= primary_key_buff;
+  char attr_buff[1024];
+
+  String attribute(attr_buff, 
+                   sizeof(attr_buff),
+                   &my_charset_bin);
+
+  key->setReadSet();
+  key->val_str(&attribute, &attribute);
+
+  size_t len= strlen(key->field_name);
+  memcpy(pos, key->field_name, len);
+  pos+= len;
+
+  memcpy(pos, attribute.ptr(), attribute.length());
+  pos+= attribute.length();
+
+  primary_key_len= (size_t) (pos - primary_key_buff);
+  primary_key= primary_key_buff;
+}
+
+void TokyoCabinetShare::initIterators()
+{
+  tcmdbiterinit(data_handler);
+}
+
+int TokyoCabinetShare::retrieveRow(unsigned char *buf)
+{
+  char *key= NULL;
+
+  while ((key= tcmdbiternext2(data_handler)) != NULL)
+  {
+    char *value= tcmdbget2(data_handler, key);
+    free(key);
+    if (value)
+    {
+      unpackRow(buf, value);
+      free(value);
+      return 0;
+    }
+  }
+
+  return HA_ERR_END_OF_FILE;
+}
+
+int TokyoCabinetShare::deleteRow(Field *key_field)
+{
+  createKey(key_field);
+
+  bool ret= tcmdbout(data_handler,
+                     primary_key,
+                     primary_key_len);
+  if (! ret)
+  {
+    return -1;
+  }
+  return 0;
+}
+
+int TokyoCabinetShare::findRow(Field *key_field,
+                               unsigned char *buf,
+                               const unsigned char *key,
+                               size_t key_len)
+{
+  key_field->set_key_image(key, key_len);
+  key_field->set_notnull();
+  createKey(key_field);
+
+  int size;
+  char *tc_row= (char *) tcmdbget(data_handler,
+                                  primary_key,
+                                  primary_key_len,
+                                  &size);
+  if (tc_row == NULL)
+  {
+    return HA_ERR_KEY_NOT_FOUND;
+  }
+
+  unpackRow(buf, tc_row);
+  free(tc_row);
+  return 0;
+}
 
=== added file 'plugin/tokyo_cabinet_engine/tokyo_cabinet_share.h'
--- plugin/tokyo_cabinet_engine/tokyo_cabinet_share.h	1970-01-01 00:00:00 +0000
+++ plugin/tokyo_cabinet_engine/tokyo_cabinet_share.h	2009-09-24 03:50:23 +0000
@@ -0,0 +1,219 @@
+/* 
+ *  Copyright (C) 2009 Sun Microsystems
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program; if not, write to the Free Software
+ *  Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ */
+
+/**
+ * @file
+ *
+ * Contains a declaration of the share class for the Tokyo Cabinet storage
+ * engine. 
+ *
+ * @todo
+ * would it not be better to create a class that encapsulates all the calls
+ * to the Tokyo Cabinet API that we can just use from within the table
+ * share?
+ */
+
+#ifndef DRIZZLE_PLUGIN_TOKYO_CABINET_SHARE_H
+#define DRIZZLE_PLUGIN_TOKYO_CABINET_SHARE_H
+
+#include <drizzled/handler.h>
+#include <drizzled/show.h>
+#include <mysys/thr_lock.h>
+
+#include <tcutil.h>
+#include <tchdb.h>
+#include <stdlib.h>
+#include <stdbool.h>
+#include <stdint.h>
+
+class TokyoCabinetShare
+{
+public:
+
+  static const uint32_t MAX_KEY_LEN= 1024;
+
+  TokyoCabinetShare()
+    :
+      use_count(0),
+      lock(),
+      name(),
+      data_handler(NULL),
+      fixed_length_table(false),
+      row_len(0),
+      fields(NULL),
+      null_bytes(0),
+      table_record(NULL),
+      primary_key(NULL),
+      primary_key_len(0)
+  {
+    data_handler= tcmdbnew();
+  }
+
+  TokyoCabinetShare(const std::string &in_name)
+    :
+      use_count(0),
+      lock(),
+      name(in_name),
+      data_handler(NULL),
+      fixed_length_table(false),
+      row_len(0),
+      fields(NULL),
+      null_bytes(0),
+      table_record(NULL),
+      primary_key(NULL),
+      primary_key_len(0)
+  {
+    data_handler= tcmdbnew();
+  }
+
+  ~TokyoCabinetShare()
+  {
+    thr_lock_delete(&lock);
+    tcmdbdel(data_handler);
+  }
+
+  uint32_t getUseCount() const
+  {
+    return use_count;
+  }
+
+  void incUseCount()
+  {
+    use_count++;
+  }
+
+  void decUseCount()
+  {
+    use_count--;
+  }
+
+  const std::string &getName() const
+  {
+    return name;
+  }
+
+  void initThreadLock()
+  {
+    thr_lock_init(&lock);
+  }
+
+  THR_LOCK *getThreadLock()
+  {
+    return &lock;
+  }
+
+  void setFixedLengthTable(bool input)
+  {
+    fixed_length_table= input;
+  }
+
+  bool isFixedLengthTable() const
+  {
+    return fixed_length_table;
+  }
+
+  void setRowLength(uint32_t in_len)
+  {
+    row_len= in_len;
+  }
+
+  void setTableFields(Field **in_fields)
+  {
+    fields= in_fields;
+  }
+
+  void setNullBytes(size_t in_null_bytes)
+  {
+    null_bytes= in_null_bytes;
+  }
+
+  void setTableRecord(unsigned char *in_record)
+  {
+    table_record= in_record;
+  }
+
+  bool createDatabaseHandler();
+
+  bool openTable();
+
+  void freeDatabaseHandler();
+
+  uint64_t getNumOfRecords() 
+  {
+    return tcmdbrnum(data_handler);
+  }
+
+  int storeRow(Field *key_field,
+               unsigned char *row);
+
+  int updateRow(Field *key_field,
+                unsigned char *row);
+
+  /**
+   * Create a key for this row to be stored in the Tokyo Cabinet store. The
+   * key is created based on the primary key value for the row.
+   *
+   * @param[in] the primary key field 
+   */
+  void createKey(Field *key);
+
+  int deleteRow(Field *key_field);
+
+  void deleteAllRows()
+  {
+    tcmdbvanish(data_handler);
+  }
+
+  void initIterators();
+
+  int retrieveRow(unsigned char *buf);
+
+  int findRow(Field *key_field, 
+              unsigned char *buf,
+              const unsigned char *key,
+              size_t key_len);
+
+private:
+
+  /**
+   * Store a variable sized row in Tokyo Cabinet.
+   */
+  size_t packRow(unsigned char *buf, unsigned char *row);
+
+  /**
+   * Retrieve a variable sized row from Tokyo Cabinet.
+   */
+  void unpackRow(unsigned char *buf, const char *row);
+
+  uint32_t use_count;
+  THR_LOCK lock;
+  std::string name;
+  TCMDB *data_handler;
+  bool fixed_length_table;
+  uint32_t row_len;
+  Field **fields;
+  size_t null_bytes;
+  unsigned char *table_record;
+
+  char primary_key_buff[MAX_KEY_LEN];
+  char *primary_key;
+  size_t primary_key_len;
+
+};
+
+#endif /* DRIZZLE_PLUGIN_TOKYO_CABINET_SHARE_H */
 
=== added directory 'tests/suite/tc_engine'
=== added directory 'tests/suite/tc_engine/r'
=== added file 'tests/suite/tc_engine/r/auto_increment.result'
--- tests/suite/tc_engine/r/auto_increment.result	1970-01-01 00:00:00 +0000
+++ tests/suite/tc_engine/r/auto_increment.result	2009-09-24 03:50:24 +0000
@@ -0,0 +1,32 @@
+DROP TABLE IF EXISTS t2;
+Warnings:
+Note	1051	Unknown table 't2'
+CREATE TABLE t2 (a int primary key auto_increment, b bigint) ENGINE=TOKYO;
+insert into t2 (b) values (42),(74),(108),(256);
+SELECT * from t2 ORDER BY a;
+a	b
+1	42
+2	74
+3	108
+4	256
+SELECT COUNT(*) from t2;
+COUNT(*)
+4
+EXPLAIN SELECT * FROM t2 WHERE a=1;
+id	select_type	table	type	possible_keys	key	key_len	ref	rows	Extra
+1	SIMPLE	t2	const	PRIMARY	PRIMARY	4	const	1	
+SELECT * FROM t2 WHERE a=1;
+a	b
+1	42
+SELECT * FROM t2 WHERE a=2;
+a	b
+2	74
+SELECT * FROM t2 WHERE a=3;
+a	b
+3	108
+SELECT * FROM t2 WHERE a=4;
+a	b
+4	256
+SELECT * FROM t2 WHERE a=5;
+a	b
+DROP TABLE t2;
 
=== added file 'tests/suite/tc_engine/r/basic.result'
--- tests/suite/tc_engine/r/basic.result	1970-01-01 00:00:00 +0000
+++ tests/suite/tc_engine/r/basic.result	2009-09-24 03:50:24 +0000
@@ -0,0 +1,138 @@
+DROP TABLE IF EXISTS t1;
+Warnings:
+Note	1051	Unknown table 't1'
+CREATE TABLE t1 (a int primary key, b varchar(200)) ENGINE=TOKYO;
+INSERT INTO t1 (a,b) VALUES (1, 'FOO');
+INSERT INTO t1 (a,b) VALUES (2, 'BAR');
+INSERT INTO t1 (a,b) VALUES (3, 'LCA ROCKS');
+INSERT INTO t1 (a,b) VALUES (4, 'freeze ray');
+INSERT INTO t1 (a,b) VALUES (5, 'this row appears because of the kind donations of....');
+SELECT COUNT(*) FROM t1;
+COUNT(*)
+5
+SELECT * from t1;
+a	b
+4	freeze ray
+5	this row appears because of the kind donations of....
+1	FOO
+2	BAR
+3	LCA ROCKS
+DROP TABLE t1;
+CREATE TABLE t2 (a int primary key,b bigint) ENGINE=TOKYO;
+insert into t2 (a,b) values (1,42),(2,74),(3,108),(4,256);
+SELECT COUNT(*) FROM t2;
+COUNT(*)
+4
+SELECT * from t2;
+a	b
+4	256
+1	42
+2	74
+3	108
+SELECT * FROM t2 WHERE a > 2;
+a	b
+4	256
+3	108
+SELECT * FROM t2 WHERE a > 4;
+a	b
+DROP TABLE t2;
+CREATE TABLE t3 (a int primary key, b int unique) ENGINE=TOKYO;
+ERROR 42000: Too many keys specified; max 1 keys allowed
+CREATE TABLE t4 (a int, b int, PRIMARY KEY (a,b)) ENGINE=TOKYO;
+ERROR 42000: Too many key parts specified; max 1 parts allowed
+CREATE TABLE t5 (a int, index(a)) ENGINE=TOKYO;
+ERROR 42000: Table handler doesn't support NULL in given index. Please change column 'a' to be NOT NULL or use another handler
+drop table if exists t1,t2;
+create table t1 (id int primary key, num int) engine=tokyo;
+insert into t1 values (1, 100);
+insert into t1 values (2, 101);
+insert into t1 values (3, 102);
+insert into t1 values (4, 103);
+insert into t1 values (5, 104);
+select * from t1;
+id	num
+1	100
+2	101
+3	102
+4	103
+5	104
+select count(*) from t1;
+count(*)
+5
+select * from t1 where id = 1;
+id	num
+1	100
+select * from t1 where id = 2;
+id	num
+2	101
+select * from t1 where id = 3;
+id	num
+3	102
+select * from t1 where id = 4;
+id	num
+4	103
+select * from t1 where id = 5;
+id	num
+5	104
+update t1 set num = 1111 where id = 1;
+update t1 set num = 2222 where id = 2;
+update t1 set num = 3333 where id = 3;
+select * from t1 where id < 4;
+id	num
+1	1111
+2	2222
+3	3333
+delete from t1 where id = 1;
+delete from t1 where id = 2;
+delete from t1 where id = 3;
+delete from t1;
+drop table t1;
+create table t2 (id int primary key, b blob) engine=tokyo;
+insert into t2 values (1, "aaa");
+insert into t2 values (2, "bbb");
+insert into t2 values (3, "aaaaaa");
+insert into t2 values (4, "bbbbbb");
+select * from t2;
+id	b
+1	aaa
+2	bbb
+3	aaaaaa
+4	bbbbbb
+update t2 set b = NULL where id = 1;
+update t2 set b = "updated" where id = 2;
+update t2 set b = NULL where id = 3;
+update t2 set b = "updated" where id = 4;
+select * from t2;
+id	b
+1	NULL
+2	updated
+3	NULL
+4	updated
+drop table t2;
+create table t1(id int primary key, num1 int, num2 int) engine=tokyo;
+create table t2 like t1;
+insert into t2 values (1,1,1), (2,2,2), (3,3,3), (4,4,4), (5,5,5), (6,6,6);
+insert into t2 values (7,7,7), (8,8,8), (9,9,9), (10,10,10), (11,11,11);
+insert into t1 select * from t2;
+select * from t1;
+id	num1	num2
+8	8	8
+1	1	1
+9	9	9
+10	10	10
+2	2	2
+11	11	11
+3	3	3
+4	4	4
+5	5	5
+6	6	6
+7	7	7
+delete from t1;
+insert into t1 select * from t2 where id < 5;
+select * from t1;
+id	num1	num2
+1	1	1
+2	2	2
+3	3	3
+4	4	4
+drop table t1, t2;
 
=== added directory 'tests/suite/tc_engine/t'
=== added file 'tests/suite/tc_engine/t/auto_increment.test'
--- tests/suite/tc_engine/t/auto_increment.test	1970-01-01 00:00:00 +0000
+++ tests/suite/tc_engine/t/auto_increment.test	2009-09-24 03:50:24 +0000
@@ -0,0 +1,13 @@
+DROP TABLE IF EXISTS t2;
+CREATE TABLE t2 (a int primary key auto_increment, b bigint) ENGINE=TOKYO;
+insert into t2 (b) values (42),(74),(108),(256);
+SELECT * from t2 ORDER BY a;
+SELECT COUNT(*) from t2;
+EXPLAIN SELECT * FROM t2 WHERE a=1;
+SELECT * FROM t2 WHERE a=1;
+SELECT * FROM t2 WHERE a=2;
+SELECT * FROM t2 WHERE a=3;
+SELECT * FROM t2 WHERE a=4;
+SELECT * FROM t2 WHERE a=5;
+DROP TABLE t2;
+
 
=== added file 'tests/suite/tc_engine/t/basic.test'
--- tests/suite/tc_engine/t/basic.test	1970-01-01 00:00:00 +0000
+++ tests/suite/tc_engine/t/basic.test	2009-09-24 03:50:24 +0000
@@ -0,0 +1,107 @@
+--disable-warnings
+DROP TABLE IF EXISTS t1;
+--enable-warnings
+
+CREATE TABLE t1 (a int primary key, b varchar(200)) ENGINE=TOKYO;
+
+INSERT INTO t1 (a,b) VALUES (1, 'FOO');
+INSERT INTO t1 (a,b) VALUES (2, 'BAR');
+INSERT INTO t1 (a,b) VALUES (3, 'LCA ROCKS');
+INSERT INTO t1 (a,b) VALUES (4, 'freeze ray');
+INSERT INTO t1 (a,b) VALUES (5, 'this row appears because of the kind donations of....');
+SELECT COUNT(*) FROM t1;
+SELECT * from t1;
+
+DROP TABLE t1;
+
+CREATE TABLE t2 (a int primary key,b bigint) ENGINE=TOKYO;
+insert into t2 (a,b) values (1,42),(2,74),(3,108),(4,256);
+SELECT COUNT(*) FROM t2;
+SELECT * from t2;
+SELECT * FROM t2 WHERE a > 2;
+SELECT * FROM t2 WHERE a > 4;
+DROP TABLE t2;
+
+--error ER_TOO_MANY_KEYS
+CREATE TABLE t3 (a int primary key, b int unique) ENGINE=TOKYO;
+
+--error ER_TOO_MANY_KEY_PARTS
+CREATE TABLE t4 (a int, b int, PRIMARY KEY (a,b)) ENGINE=TOKYO;
+
+--error ER_NULL_COLUMN_IN_INDEX
+CREATE TABLE t5 (a int, index(a)) ENGINE=TOKYO;
+
+--disable_warnings
+drop table if exists t1,t2;
+--enable_warnings
+
+## Easy warm up test
+create table t1 (id int primary key, num int) engine=tokyo;
+
+insert into t1 values (1, 100);
+insert into t1 values (2, 101);
+insert into t1 values (3, 102);
+insert into t1 values (4, 103);
+insert into t1 values (5, 104);
+
+select * from t1;
+select count(*) from t1;
+
+# Test Primary Key
+select * from t1 where id = 1;
+select * from t1 where id = 2;
+select * from t1 where id = 3;
+select * from t1 where id = 4;
+select * from t1 where id = 5;
+
+# Update rows based on Primary Key
+update t1 set num = 1111 where id = 1;
+update t1 set num = 2222 where id = 2;
+update t1 set num = 3333 where id = 3;
+
+select * from t1 where id < 4;
+
+# Delete rows based on Primary Key
+delete from t1 where id = 1;
+delete from t1 where id = 2;
+delete from t1 where id = 3;
+
+# Delete remaining rows
+delete from t1;
+
+drop table t1;
+
+# Test BLOB type
+create table t2 (id int primary key, b blob) engine=tokyo;
+
+insert into t2 values (1, "aaa");
+insert into t2 values (2, "bbb");
+insert into t2 values (3, "aaaaaa");
+insert into t2 values (4, "bbbbbb");
+
+select * from t2;
+
+update t2 set b = NULL where id = 1;
+update t2 set b = "updated" where id = 2;
+update t2 set b = NULL where id = 3;
+update t2 set b = "updated" where id = 4;
+
+select * from t2;
+
+drop table t2;
+
+# Test INSERT-SELECT statements
+create table t1(id int primary key, num1 int, num2 int) engine=tokyo;
+create table t2 like t1;
+
+insert into t2 values (1,1,1), (2,2,2), (3,3,3), (4,4,4), (5,5,5), (6,6,6);
+insert into t2 values (7,7,7), (8,8,8), (9,9,9), (10,10,10), (11,11,11);
+
+insert into t1 select * from t2;
+select * from t1;
+delete from t1;
+
+insert into t1 select * from t2 where id < 5;
+select * from t1;
+
+drop table t1, t2;