Merge lp:~sinzui/charmworld/drain-ingest-queue into lp:~juju-jitsu/charmworld/trunk

[Curtis Hovey]

Add a dequeue script so that the ingest queue can be drained during deploys.

Add a dequeue script so that the ingest queue can be drained during deploys.

RULES

    This branch, pre-implementation: abentley
    * Add a dequeue script that a user or charm could call to empty
      the ingest queue.

    Next Branches
    * Update the charmworld charm to call dequeue when the revno changes
    * Update ingest to only enqueue when the code's revno is => queue's
      revno.
    * Update the code to not pull from the queue when its revno is lower
      the queue's revno.

QA

    Using two terminals
    * T1: ssh to the staging.charmworld.com's charmworld app and get ready:
      juju ssh -e staging charmworld/6
      cd /home/webops_deploy/charmworld
      ls ./bin/dequeue
    * T2 ssh to the staging.charmworld.com's mongdb and check the queue size:
      juju ssh -e staging mongodb/0
      mongo juju --eval 'db["charm-queue"].count()'
    * T1: Run dequeue
      sudo su charmworld -c 'HOME=./ INI=./production.ini ./bin/dequeue'
    * T2: Verify the the queue was drained (the count is zero):
      mongo juju --eval 'db["charm-queue"].count()'

[Richard Harding (rharding)]

Thanks, looks good to me.

review: Approve

=== modified file 'charmworld/jobs/lp.py'
--- charmworld/jobs/lp.py	2013-06-11 18:45:50 +0000
+++ charmworld/jobs/lp.py	2013-07-01 15:42:49 +0000
@@ -98,6 +98,29 @@
             log.info("Queued %s", charm)
 
 
+def dequeue():
+    """Dequeue all the charm data in the CHARM_QUEUE.
+
+    This is a script entry point to dequeue all charms. This might be done when
+    when a running instance will get a code update that is incompatible with
+    charm data in the queue.
+    """
+    configure_logging()
+    log = logging.getLogger("charm.launchpad")
+    settings = get_ini()
+    connection = getconnection(settings)
+    db = getdb(connection, settings.get('mongo.database'))
+    try:
+        lease_time = int(settings['script_lease_time']) * 60
+        with lock('ingest-queue', lease_time, db, log):
+            queue = get_queue(CHARM_QUEUE)
+            queue_size = queue.size()
+            queue.clear()
+            log.info("Dequeued %s charms", queue_size)
+    except LockHeld, error:
+        log.warn(str(error))
+
+
 def main():
     configure_logging()
     parser = argparse.ArgumentParser()

=== modified file 'charmworld/jobs/tests/test_lp.py'
--- charmworld/jobs/tests/test_lp.py	2013-06-10 11:35:18 +0000
+++ charmworld/jobs/tests/test_lp.py	2013-07-01 15:42:49 +0000
@@ -4,17 +4,23 @@
 import json
 
 from httmock import HTTMock
-from mock import patch
+from mock import (
+    ANY,
+    patch,
+)
 
 from charmworld.testing import (
     factory,
     JobTestBase,
+    MONGO_DATABASE,
 )
 
 from charmworld.jobs.lp import (
     all_charms,
+    available_charms,
+    CHARM_QUEUE,
     db_charms,
-    available_charms,
+    dequeue,
     queue_charms,
 )
 from charmworld.jobs.utils import get_queue
@@ -231,3 +237,17 @@
         item = out_queue.next()
         self.assertEqual(
             {'branch_dir': 'foo', 'branch_spec': 'bar'}, item.payload)
+
+    def test_dequeue(self):
+        queue = get_queue('test_queue')
+        self.addCleanup(queue.clear)
+        charm_data = factory.get_charm_json()
+        queue.put(charm_data)
+        self.assertEqual(1, queue.size())
+        with patch('charmworld.jobs.lp.getdb', return_value=self.db) as getdb:
+            lp_get_queue = 'charmworld.jobs.lp.get_queue'
+            with patch(lp_get_queue, return_value=queue) as gq:
+                dequeue()
+        getdb.assert_called_with(ANY, MONGO_DATABASE)
+        gq.assert_called_with(CHARM_QUEUE)
+        self.assertEqual(0, queue.size())

=== modified file 'setup.py'
--- setup.py	2013-05-03 13:13:14 +0000
+++ setup.py	2013-07-01 15:42:49 +0000
@@ -34,6 +34,7 @@
       main = charmworld:main
       [console_scripts]
       enqueue = charmworld.jobs.lp:main
+      dequeue = charmworld.jobs.lp:dequeue
       ingest-queued = charmworld.jobs.worker:main
       migrations = migrations.migrate:main
       es-update = charmworld.search:update_main