Merge lp:~adeuring/charmworld/more-heartbeat-info-3 into lp:charmworld

[Abel Deuring]

heartbeat page:
  - show a warning if charmworld's crontab file is missing
  - show the time when the last ingest job finished
  - show the time when the charmworld app server(s) were started

This branch adds three more details to the heartbeat page:

- if a crontrab file for charmworld exists. (The crontab file was
  missing in the past...)
- the time when the server was started (suggestion by rick_h)
- the time when the last ingest job finished.

The crontab check is trivial. To store the timestamps "server started"
and "last ingest job", I added a collection "status" to the mongo DB.
I am not sure if this is the right collection name -- "misc" might
be better just in case that some container for some other data is
needed.

The time of the last ingest job is added by calling the new function
set_status_timestamp() in ingest.run_job(); this required a modification
in the unit tests of run_job() and in run_job() because
set_status_timestamp() obviously needs a MongoDB instance.

[Benji York (benji)]

> This branch adds three more details to the heartbeat page:

Very nice. The branch looks good once you have addressed my comments to
your satisfaction.

> To store the timestamps "server started" and "last ingest job", I
> added a collection "status" to the mongo DB. I am not sure if this is
> the right collection name -- "misc" might be better just in case that
> some container for some other data is needed.

How about "server_status"? That will keep "status" available for
application data.

We happen to run a single server at the moment, but we should assume
that in as few places as possible. How about a mapping from hostname to
last start time, that way we can grow into multiple servers.

> The time of the last ingest job is added by calling the new function
> set_status_timestamp() in ingest.run_job(); this required a modification
> in the unit tests of run_job() and in run_job() because
> set_status_timestamp() obviously needs a MongoDB instance.

From line 106 of the diff:

+ # Some tests do not set the 'db' attribute.
+ if getattr(job, 'db', None) is not None:
+ set_status_timestamp(job.db, LAST_INGEST_JOB_FINISHED)

It seems that there is only one test that was faking the DB; I wonder if
introducing this code path is worth it for one test. It seems to me
that it would be preferable to mutate the test rather than the code.

Other thoughts:

The first sentence of the docstring on line 277 spans more than one
line. I would just make it a comment.

Should we check the mode bits of the crontab file? I don't know that it
has ever caused us a problem, but it is a common problem people have
with crontabs.

In formatted_status_timestamp(), it would be nice to have the time zone
on the formatted time (i.e., tack on a "Z").

I would like to see the already too big test_all_checks_pass() and
test_checks_fail() broken into one test for each passing/failing
component, but that's not a blocker.

review: Approve

[lp:~adeuring/charmworld/more-heartbeat-info-3]

459. By Abel Deuring on 2013-11-13

MongoDB collection name 'status' changed to 'server_status'

460. By Abel Deuring on 2013-11-13

timezone added in formatted_status_timestamp()

461. By Abel Deuring on 2013-11-13

call set_status_timestamp() in IngestJob.run(), not in run_job()

462. By Abel Deuring on 2013-11-14

record a server's start time together with the hostname

463. By Abel Deuring on 2013-11-14

script added to remove stale entries in the "server_started" dict.

[Abel Deuring (adeuring)]

On 13.11.2013 15:24, Benji York wrote:
> Review: Approve
>
>> This branch adds three more details to the heartbeat page:
>
> Very nice. The branch looks good once you have addressed my comments to
> your satisfaction.
>
>> To store the timestamps "server started" and "last ingest job", I
>> added a collection "status" to the mongo DB. I am not sure if this is
>> the right collection name -- "misc" might be better just in case that
>> some container for some other data is needed.
>
> How about "server_status"? That will keep "status" available for
> application data.

Fixed.

>
> We happen to run a single server at the moment, but we should assume
> that in as few places as possible. How about a mapping from hostname to
> last start time, that way we can grow into multiple servers.

Changed. We discussed this on IRC; ISTM that for now the best key to
identify a server is it's hostname. This is not ideal in conjunction
with juju, but it allows at least to look up a machine as seen on the
heartbeat page in "juju status".

Since hostnames may become stale, I also added a script that allows to
manually remove stale entries from the database.

>
>> The time of the last ingest job is added by calling the new function
>> set_status_timestamp() in ingest.run_job(); this required a modification
>> in the unit tests of run_job() and in run_job() because
>> set_status_timestamp() obviously needs a MongoDB instance.
>
>>From line 106 of the diff:
>
> + # Some tests do not set the 'db' attribute.
> + if getattr(job, 'db', None) is not None:
> + set_status_timestamp(job.db, LAST_INGEST_JOB_FINISHED)
>
> It seems that there is only one test that was faking the DB; I wonder if
> introducing this code path is worth it for one test. It seems to me
> that it would be preferable to mutate the test rather than the code.

Right, that's a bit ugly. More than one test is affected though:
test_run_job_defaults(), test_run_job_no_setup(),
test_run_job_db_specified() (here, the second run_job() call fails),
test_run_job_general_exception().

I moved the call of set_status_timestamp() into UpdateCharmJob.run() and
UpdateBundleJob.run(); this makes the "if getattr(...)" unnecessary.

>
> Other thoughts:
>
> The first sentence of the docstring on line 277 spans more than one
> line. I would just make it a comment.

changed.

>
> Should we check the mode bits of the crontab file? I don't know that it
> has ever caused us a problem, but it is a common problem people have
> with crontabs.
>
> In formatted_status_timestamp(), it would be nice to have the time zone
> on the formatted time (i.e., tack on a "Z").

Changed.

>
> I would like to see the already too big test_all_checks_pass() and
> test_checks_fail() broken into one test for each passing/failing
> component, but that's not a blocker.
>

[Benji York (benji)]

The changes look great.

Since we understand tests the best at the moment we write them, I'm a big fan of starting tests with short comments that describe what the test is about. Here's a diff that will apply some to this branch:

--- charmworld/tests/test_utils.py 2013-11-14 10:17:12.880649945 -0600
+++ charmworld/tests/test_utils.py 2013-11-14 10:42:51.832685061 -0600
@@ -1,4 +1,5 @@
     def test_remove_server_start_time_entries(self):
+ # Each time a server starts it stores the current time in the DB.
         self.db.server_status.insert({
             '_id': SERVER_STARTED,
             'foo': 123.456,
@@ -17,6 +18,8 @@
         self.assertEqual('', output.getvalue())

     def test_remove_server_start_time_entries_no_args(self):
+ # If the server start time script is run without arguments it generates
+ # an error message.
         output = StringIO()
         self.assertEqual(
             1, remove_server_start_time_entries([''], output))
@@ -25,6 +28,8 @@
             output.getvalue())

     def test_remove_server_start_time_entries_no_db_record(self):
+ # If there are no server start times yet and the script is run, it
+ # informs the user.
         output = StringIO()
         self.assertEqual(
             1, remove_server_start_time_entries(['', 'foo'], output))
@@ -32,6 +37,8 @@
             'No server start times are yet recorded.\n', output.getvalue())

     def test_remove_server_start_time_entries_invalid_server_name(self):
+ # If the script is asked to remove a server start time that does not
+ # exist, it generates an informative message.
         self.db.server_status.insert({
             '_id': SERVER_STARTED,
             'foo': 123.456,

review: Approve

[lp:~adeuring/charmworld/more-heartbeat-info-3]

464. By Abel Deuring on 2013-11-14

comments added to the new tests.

465. By Abel Deuring on 2013-11-14

trunk merged; conflicts resolved.

[Abel Deuring (adeuring)]

review: Approve

[Juju Gui Bot (juju-gui-bot)]

review: Approve (continuous-integration)

=== modified file 'charmworld/__init__.py'
--- charmworld/__init__.py	2013-09-03 19:57:54 +0000
+++ charmworld/__init__.py	2013-11-14 17:14:59 +0000
@@ -19,7 +19,10 @@
 from charmworld.models import UserMgr
 from charmworld.routes import build_routes
 from charmworld.globals import default_globals_factory
-from charmworld.utils import get_session_url
+from charmworld.utils import (
+    get_session_url,
+    record_server_start_time,
+)
 
 
 class cached_view_config(view_config):
@@ -142,5 +145,6 @@
     config.add_static_view('static', 'charmworld:static', cache_max_age=3600)
     config = build_routes(config)
     config.scan("charmworld.views")
+    record_server_start_time(database)
 
     return config.make_wsgi_app()
 
=== modified file 'charmworld/health.py'
--- charmworld/health.py	2013-10-31 16:00:29 +0000
+++ charmworld/health.py	2013-11-14 17:14:59 +0000
@@ -4,7 +4,14 @@
 from bzrlib.branch import Branch
 from bzrlib.transport import get_transport
 from collections import namedtuple
+import os
 from os.path import dirname
+import time
+
+from charmworld.utils import (
+    LAST_INGEST_JOB_FINISHED,
+    SERVER_STARTED,
+)
 
 
 Check = namedtuple('check', 'name status remark')
@@ -185,3 +192,47 @@
     finally:
         check = Check('ElasticSearch server', status, remark)
     return check
+
+
+def check_crontab(request):
+    """Check that a crontab file for charmworld exists."""
+    if 'charmworld' in os.listdir('/etc/cron.d'):
+        status = PASS
+        remark = 'Crontab file for charmworld exists.'
+    else:
+        status = FAIL
+        remark = 'Crontab file for charmworld not found.'
+    return Check('Crontab', status, remark)
+
+
+def formatted_timestamp(timestamp):
+    return time.strftime('%Y-%m-%d %H:%M:%SZ', time.gmtime(timestamp))
+
+
+def check_server_start_time(request):
+    """Return the time when the server was last started."""
+    try:
+        start_times = request.db.server_status.find_one(SERVER_STARTED)
+        del start_times['_id']
+        start_times = [
+            '%s: %s' % (hostname, formatted_timestamp(timestamp))
+            for hostname, timestamp in sorted(start_times.items())]
+        remark = ', '.join(start_times)
+        status = PASS
+    except Exception:
+        remark = "Can't retrieve the server's start time."
+        status = FAIL
+    return Check('Server started', status, remark)
+
+
+def check_last_ingest_job(request):
+    """Return the time when the ingest job finished."""
+    try:
+        timestamp = request.db.server_status.find_one(
+            LAST_INGEST_JOB_FINISHED)['time']
+        remark = 'finished at %s' % formatted_timestamp(timestamp)
+        status = PASS
+    except Exception:
+        remark = "Can't retrieve the time when the last ingest job finished."
+        status = FAIL
+    return Check('Last ingest job', status, remark)
 
=== modified file 'charmworld/jobs/ingest.py'
--- charmworld/jobs/ingest.py	2013-11-06 22:54:45 +0000
+++ charmworld/jobs/ingest.py	2013-11-14 17:14:59 +0000
@@ -51,9 +51,11 @@
     SearchServiceNotAvailable,
 )
 from charmworld.utils import (
+    LAST_INGEST_JOB_FINISHED,
     quote_key,
     quote_yaml,
     read_locked,
+    set_status_timestamp,
     timestamp,
     unquote_yaml,
 )
@@ -95,6 +97,9 @@
     def run(self, charm_data):
         raise NotImplementedError
 
+    def update_ingest_status(self):
+        set_status_timestamp(self.db, LAST_INGEST_JOB_FINISHED)
+
 
 class DBIngestJob(IngestJob):
 
@@ -303,6 +308,7 @@
             CharmSource(self.db, index_client).save(charm_data)
         else:
             self.log.info('Skipping %s' % charm_data['_id'])
+        self.update_ingest_status()
 
 
 class UpdateBundleJob(DBIngestJob):
@@ -374,6 +380,7 @@
                 basket_data['name_revno'],
                 basket_data['first_change'], basket_data['last_change'],
                 basket_data['changes'], basket_data['branch_spec'])
+        self.update_ingest_status()
 
 
 def _rev_info(r, branch):
 
=== modified file 'charmworld/jobs/tests/test_ingest.py'
--- charmworld/jobs/tests/test_ingest.py	2013-11-06 23:04:32 +0000
+++ charmworld/jobs/tests/test_ingest.py	2013-11-14 17:14:59 +0000
@@ -12,6 +12,7 @@
 import shutil
 import subprocess
 from textwrap import dedent
+import time
 
 import bzrlib
 from bzrlib.bzrdir import BzrDir
@@ -53,6 +54,7 @@
     TestCase,
 )
 from charmworld.utils import (
+    LAST_INGEST_JOB_FINISHED,
     quote_key,
 )
 
@@ -688,9 +690,11 @@
         if self.exc is not None:
             raise self.exc('test error')
         self.run_called = True
-
-
-class TestRunJob(TestCase):
+        if getattr(self, 'db', None) is not None:
+            self.update_ingest_status()
+
+
+class TestRunJob(MongoTestBase):
 
     def test_run_job_defaults(self):
         job = TestJob()
@@ -708,10 +712,15 @@
 
     def test_run_job_db_specified(self):
         job = TestJob()
-        run_job(job, 'foo', db='db')
-        self.assertEqual('db', job.db)
+        before = time.time()
+        run_job(job, 'foo', db=self.db)
+        after = time.time()
+        self.assertEqual(self.db, job.db)
+        job_finished = self.db.server_status.find_one(LAST_INGEST_JOB_FINISHED)
+        job_finished = job_finished['time']
+        self.assertTrue(before <= job_finished <= after)
         job = TestJob()
-        run_job(job, 'foo', needs_setup=False, db='db')
+        run_job(job, 'foo', needs_setup=False, db=self.db)
         self.assertIs(None, getattr(job, 'db', None))
 
     def test_run_job_general_exception(self):
 
=== modified file 'charmworld/tests/test_health.py'
--- charmworld/tests/test_health.py	2013-10-31 16:00:29 +0000
+++ charmworld/tests/test_health.py	2013-11-14 17:14:59 +0000
@@ -2,6 +2,8 @@
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 from mock import patch
+import os
+import re
 
 from charmworld.health import (
     check_api2,
@@ -11,14 +13,22 @@
     check_charms_collection,
     check_collections,
     check_charm_index,
+    check_crontab,
     check_elasticsearch_status,
     check_ingest_queues,
+    check_last_ingest_job,
+    check_server_start_time,
 )
 from charmworld.models import FeaturedSource
 from charmworld.testing import (
     factory,
     ViewTestBase,
 )
+from charmworld.utils import (
+    LAST_INGEST_JOB_FINISHED,
+    record_server_start_time,
+    set_status_timestamp,
+)
 
 
 class TestCheckMixin:
@@ -261,3 +271,61 @@
             check = check_elasticsearch_status(request)
             remark = "Can't retrieve the ES server's health status."
         self.assertCheck(check, 'ElasticSearch server', 'Fail', remark)
+
+    def test_check_crontab_pass(self):
+        def make_fake_listdir():
+            def fake_listdir(path):
+                return ('foo', 'charmworld')
+            return fake_listdir
+
+        with patch.object(os, 'listdir', new_callable=make_fake_listdir):
+            request = self.getRequest()
+            check = check_crontab(request)
+        remark = 'Crontab file for charmworld exists.'
+        self.assertCheck(check, 'Crontab', 'Pass', remark)
+
+    def test_check_crontab_fail(self):
+        def make_fake_listdir():
+            def fake_listdir(path):
+                return ('foo', )
+            return fake_listdir
+
+        with patch.object(os, 'listdir', new_callable=make_fake_listdir):
+            request = self.getRequest()
+            check = check_crontab(request)
+        remark = 'Crontab file for charmworld not found.'
+        self.assertCheck(check, 'Crontab', 'Fail', remark)
+
+    def test_check_server_start_time_pass(self):
+        def make_uname():
+            def uname():
+                return (None, 'foo')
+            return uname
+
+        with patch.object(os, 'uname', new_callable=make_uname):
+            record_server_start_time(self.db)
+        check = check_server_start_time(self.getRequest())
+        self.assertCheck(check, 'Server started', 'Pass')
+        mo = re.search(
+            '^foo: \d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\dZ$', check.remark)
+        self.assertIsNot(None, mo)
+
+    def test_check_server_start_time_fail(self):
+        check = check_server_start_time(self.getRequest())
+        self.assertCheck(
+            check, 'Server started', 'Fail',
+            "Can't retrieve the server's start time.")
+
+    def test_check_last_ingest_job_pass(self):
+        set_status_timestamp(self.db, LAST_INGEST_JOB_FINISHED)
+        check = check_last_ingest_job(self.getRequest())
+        self.assertCheck(check, 'Last ingest job', 'Pass')
+        mo = re.search(
+            '^finished at \d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\dZ$', check.remark)
+        self.assertIsNot(None, mo)
+
+    def test_check_last_ingest_job_fail(self):
+        check = check_last_ingest_job(self.getRequest())
+        self.assertCheck(
+            check, 'Last ingest job', 'Fail',
+            "Can't retrieve the time when the last ingest job finished.")
 
=== modified file 'charmworld/tests/test_utils.py'
--- charmworld/tests/test_utils.py	2013-11-13 17:40:11 +0000
+++ charmworld/tests/test_utils.py	2013-11-14 17:14:59 +0000
@@ -13,6 +13,7 @@
 )
 import tempfile
 from textwrap import dedent
+import time
 from unittest import TestCase
 import urllib
 
@@ -29,6 +30,10 @@
     pretty_timedelta,
     quote_key,
     quote_yaml,
+    record_server_start_time,
+    remove_server_start_time_entries,
+    SERVER_STARTED,
+    set_status_timestamp,
     unquote_yaml,
     _process_dict_keys,
 )
@@ -364,3 +369,109 @@
         self.assertEqual(
             build_metric_key('metric', 'foo/bar/42'),
             'metric/foo/bar/42')
+
+
+class TestSetStatusTimeStamp(MongoTestBase):
+
+    def test_set_status_timestamp(self):
+        # set_status_timestamp() records the current time in a MongoDB
+        # record.
+        before = time.time()
+        set_status_timestamp(self.db, 'foo')
+        after = time.time()
+        recorded = self.db.server_status.find_one('foo')['time']
+        self.assertTrue(before <= recorded <= after)
+
+
+class TestServerStartTime(MongoTestBase):
+
+    def test_record_server_start_time(self):
+        # record_server_start_time() records the hostname and the current
+        # time in a MongoDB record.
+        import os
+
+        def make_uname(hostname):
+            def uname():
+                return (None, hostname)
+            return uname
+
+        with patch.object(
+                os, 'uname', new_callable=make_uname,
+                hostname='foo'):
+            before = time.time()
+            record_server_start_time(self.db)
+            after = time.time()
+        foo_start_time = self.db.server_status.find_one(SERVER_STARTED)['foo']
+        self.assertTrue(before <= foo_start_time <= after)
+
+        # Updating a server's start time retains already recorded data.
+        with patch.object(
+                os, 'uname', new_callable=make_uname,
+                hostname='bar'):
+            record_server_start_time(self.db)
+        recorded = self.db.server_status.find_one(SERVER_STARTED)
+        self.assertEqual(set(('_id', 'foo', 'bar')), set(recorded))
+
+        # Existing records are updated.
+        with patch.object(
+                os, 'uname', new_callable=make_uname,
+                hostname='foo'):
+            before = time.time()
+            record_server_start_time(self.db)
+            after = time.time()
+        recorded = self.db.server_status.find_one(SERVER_STARTED)
+        self.assertTrue(recorded['foo'] > foo_start_time)
+
+    def test_remove_server_start_time_entries(self):
+        # Each time a server starts it stores the current time in the DB.
+        # Stale entries can be removed by calling
+        # remove_server_start_time_entries() (used in the script
+        # remove-start-time-entry).
+        self.db.server_status.insert({
+            '_id': SERVER_STARTED,
+            'foo': 123.456,
+            'bar': 234.567,
+        })
+        output = StringIO()
+        self.assertEqual(
+            0, remove_server_start_time_entries(['', 'foo'], output))
+        start_times = self.db.server_status.find_one({'_id': SERVER_STARTED})
+        self.assertEqual(
+            {
+                '_id': SERVER_STARTED,
+                'bar': 234.567,
+            },
+            start_times)
+        self.assertEqual('', output.getvalue())
+
+    def test_remove_server_start_time_entries_no_args(self):
+        # If the server start time script is run without arguments it
+        # generates an error message.
+        output = StringIO()
+        self.assertEqual(
+            1, remove_server_start_time_entries([''], output))
+        self.assertEqual(
+            'At least one server hostname must be specified.\n',
+            output.getvalue())
+
+    def test_remove_server_start_time_entries_no_db_record(self):
+        # If there are no server start times yet and the script is run, it
+        # informs the user.
+        output = StringIO()
+        self.assertEqual(
+            1, remove_server_start_time_entries(['', 'foo'], output))
+        self.assertEqual(
+            'No server start times are yet recorded.\n', output.getvalue())
+
+    def test_remove_server_start_time_entries_invalid_server_name(self):
+        # If the script is asked to remove a server start time that does not
+        # exist, it generates an informative message.
+        self.db.server_status.insert({
+            '_id': SERVER_STARTED,
+            'foo': 123.456,
+        })
+        output = StringIO()
+        self.assertEqual(
+            1, remove_server_start_time_entries(['', 'bar'], output))
+        self.assertEqual(
+            'hostname bar not found.\n', output.getvalue())
 
=== modified file 'charmworld/utils.py'
--- charmworld/utils.py	2013-11-13 19:37:19 +0000
+++ charmworld/utils.py	2013-11-14 17:14:59 +0000
@@ -11,6 +11,7 @@
 import logging.config
 import os
 import re
+import sys
 import time
 from urllib import unquote
 import urlparse
@@ -28,6 +29,12 @@
     (60, '%d minute')
 )
 
+# The MongoDB ID of the time the last ingest job finished in the 'status'
+# collection.
+LAST_INGEST_JOB_FINISHED = 'last_ingest_job_finished'
+# The MongoDB ID of the time the server was started in the 'status' collection.
+SERVER_STARTED = 'server_started'
+
 
 def prettify_timedelta(diff):
     """Return a prettified time delta, suitable for pretty printing."""
@@ -227,3 +234,62 @@
     # Metrics are stored for the versionless ID.
     versionless_id = id_revision_regex.sub('', item_id)
     return '/'.join([metric_name, versionless_id])
+
+
+def set_status_timestamp(database, id):
+    database.server_status.save({
+        '_id': id,
+        'time': time.time(),
+    })
+
+
+def record_server_start_time(database):
+    hostname = os.uname()[1]
+    database.server_status.update(
+        {
+            '_id': SERVER_STARTED,
+        },
+        {
+            '$set': {
+                hostname: time.time(),
+            },
+        },
+        upsert=True)
+
+
+def remove_server_start_time_entries(argv, output, db=None):
+    from charmworld.models import getconnection
+    from charmworld.models import getdb
+
+    if db is None:
+        settings = get_ini()
+        connection = getconnection(settings)
+        db = getdb(connection, settings.get('mongo.database'))
+
+    if len(argv) < 2:
+        print >>output, "At least one server hostname must be specified."
+        return 1
+
+    server_start_times = db.server_status.find_one({'_id': SERVER_STARTED})
+    if server_start_times is None:
+        print >>output, "No server start times are yet recorded."
+        return 1
+
+    for name in argv[1:]:
+        if name not in server_start_times:
+            print >>output, "hostname %s not found." % name
+            return 1
+
+    for name in argv[1:]:
+        db.server_status.update(
+            {
+                '_id': SERVER_STARTED,
+            },
+            {
+                '$unset': {name: ''},
+            })
+    return 0
+
+
+def remove_server_start_time():
+    sys.exit(remove_server_start_time_entries(sys.argv, sys.stderr))
 
=== modified file 'charmworld/views/misc.py'
--- charmworld/views/misc.py	2013-10-31 14:10:57 +0000
+++ charmworld/views/misc.py	2013-11-14 17:14:59 +0000
@@ -14,8 +14,11 @@
     check_charm_index,
     check_charms_collection,
     check_collections,
+    check_crontab,
     check_elasticsearch_status,
     check_ingest_queues,
+    check_last_ingest_job,
+    check_server_start_time,
 )
 
 
@@ -62,4 +65,7 @@
     checks.append(check_ingest_queues(request))
     checks.append(check_bzr_revision(request))
     checks.append(check_elasticsearch_status(request))
+    checks.append(check_crontab(request))
+    checks.append(check_server_start_time(request))
+    checks.append(check_last_ingest_job(request))
     return {'checks': checks}
 
=== modified file 'charmworld/views/tests/test_misc.py'
--- charmworld/views/tests/test_misc.py	2013-11-11 10:44:53 +0000
+++ charmworld/views/tests/test_misc.py	2013-11-14 17:14:59 +0000
@@ -17,6 +17,11 @@
     WebTestBase,
 )
 from charmworld.tests.test_health import TestCheckMixin
+from charmworld.utils import (
+    LAST_INGEST_JOB_FINISHED,
+    record_server_start_time,
+    set_status_timestamp,
+)
 from charmworld.views.misc import heartbeat
 
 
@@ -119,6 +124,8 @@
         self.use_index_client()
         self.index_client.index_charm(charm_data)
         FeaturedSource.from_db(self.db).set_featured(charm_data, 'charm')
+        record_server_start_time(self.db)
+        set_status_timestamp(self.db, LAST_INGEST_JOB_FINISHED)
 
         def make_all_collection_names():
             def all_collection_names():
@@ -127,13 +134,19 @@
                     'bundles', 'charm-queue', 'charms']
             return all_collection_names
 
+        def make_fake_listdir():
+            def fake_listdir(path):
+                return ('foo', 'charmworld')
+            return fake_listdir
+
         request = self.getRequest()
         with patch.object(request.db, 'collection_names',
                           new_callable=make_all_collection_names):
-            response = heartbeat(request)
+            with patch.object(os, 'listdir', new_callable=make_fake_listdir):
+                response = heartbeat(request)
 
         checks = response['checks']
-        self.assertEqual(9, len(checks))
+        self.assertEqual(12, len(checks))
         remark = '1 charms found'
         self.assertCheck(checks[0], 'charms collection', 'Pass', remark)
         remark = '1 bundles found'
@@ -149,17 +162,28 @@
         self.assertCheck(checks[6], 'Ingest queue sizes', 'Pass', remark)
         self.assertCheck(checks[7], 'BZR revision', 'Pass')
         self.assertCheck(checks[8], 'ElasticSearch server', 'Pass')
+        self.assertCheck(checks[9], 'Crontab', 'Pass')
+        self.assertCheck(checks[10], 'Server started', 'Pass')
+        self.assertCheck(checks[11], 'Last ingest job', 'Pass')
 
     def test_checks_fail(self):
         # When services or data are not available, the checks fail.
         # No setup is performed, creating a case where services and data
         # are not available
         from charmworld import health
+
+        def make_fake_listdir():
+            def fake_listdir(path):
+                return ('foo', )
+            return fake_listdir
+
         with patch.object(health, 'get_transport',
                           new_callable=self.makeFailCallable):
-            response = heartbeat(self.getRequest())
+            with patch.object(os, 'listdir', new_callable=make_fake_listdir):
+                response = heartbeat(self.getRequest())
+
         checks = response['checks']
-        self.assertEqual(9, len(checks))
+        self.assertEqual(12, len(checks))
         remark = 'There are no charms. Is ingest running?'
         self.assertCheck(checks[0], 'charms collection', 'Fail', remark)
         remark = 'There are no bundles. Is ingest running?'
@@ -180,3 +204,9 @@
         self.assertCheck(checks[7], 'BZR revision', 'Fail', remark)
         remark = "Can't retrieve the ES server's health status."
         self.assertCheck(checks[8], 'ElasticSearch server', 'Fail', remark)
+        remark = 'Crontab file for charmworld not found.'
+        self.assertCheck(checks[9], 'Crontab', 'Fail', remark)
+        remark = "Can't retrieve the server's start time."
+        self.assertCheck(checks[10], 'Server started', 'Fail', remark)
+        remark = "Can't retrieve the time when the last ingest job finished."
+        self.assertCheck(checks[11], 'Last ingest job', 'Fail', remark)
 
=== modified file 'setup.py'
--- setup.py	2013-08-05 20:16:49 +0000
+++ setup.py	2013-11-14 17:14:59 +0000
@@ -39,6 +39,7 @@
       migrations = charmworld.migrations.migrate:main
       es-update = charmworld.search:update_main
       sync-index = charmworld.models:sync_index
+      remove-start-time-entry = charmworld.utils:remove_server_start_time
       [beaker.backends]
       mongodb = mongodb_beaker:MongoDBNamespaceManager
       """,