Merge lp:~soren/surveilr/make-it-slightly-useful into lp:surveilr

[Soren Hansen]

Add basic data model and API server

Add basic data model and API server

[Linux2Go Jenkins (linux2go-jenkins)]

The attempt to merge lp:~soren/surveilr/make-it-slightly-useful into lp:surveilr failed. Below is the output from the failed tests.

running nosetests
running egg_info
creating surveilr.egg-info
writing requirements to surveilr.egg-info/requires.txt
writing surveilr.egg-info/PKG-INFO
writing top-level names to surveilr.egg-info/top_level.txt
writing dependency_links to surveilr.egg-info/dependency_links.txt
writing manifest file 'surveilr.egg-info/SOURCES.txt'
reading manifest file 'surveilr.egg-info/SOURCES.txt'
writing manifest file 'surveilr.egg-info/SOURCES.txt'
running build_ext
ok

E.E
======================================================================
ERROR: Failure: ImportError (No module named eventlet)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/nose/loader.py", line 390, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/pymodules/python2.7/nose/importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/pymodules/python2.7/nose/importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/tmp/tmpBkJp3S/surveilr/api/server.py", line 24, in <module>
    import eventlet
ImportError: No module named eventlet

======================================================================
ERROR: Failure: ImportError (No module named mock)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/nose/loader.py", line 390, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/pymodules/python2.7/nose/importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/pymodules/python2.7/nose/importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/tmp/tmpBkJp3S/surveilr/tests/test_api_server.py", line 24, in <module>
    import mock
ImportError: No module named mock

----------------------------------------------------------------------
Ran 3 tests in 0.418s

FAILED (errors=2)

[Soren Hansen (soren)]

Installed python-{mock,eventlet} on the Jenkis box.

[Linux2Go Jenkins (linux2go-jenkins)]

The attempt to merge lp:~soren/surveilr/make-it-slightly-useful into lp:surveilr failed. Below is the output from the failed tests.

running nosetests
running egg_info
creating surveilr.egg-info
writing requirements to surveilr.egg-info/requires.txt
writing surveilr.egg-info/PKG-INFO
writing top-level names to surveilr.egg-info/top_level.txt
writing dependency_links to surveilr.egg-info/dependency_links.txt
writing manifest file 'surveilr.egg-info/SOURCES.txt'
reading manifest file 'surveilr.egg-info/SOURCES.txt'
writing manifest file 'surveilr.egg-info/SOURCES.txt'
running build_ext
ok

E.E
======================================================================
ERROR: Failure: ImportError (No module named routes)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/nose/loader.py", line 390, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/pymodules/python2.7/nose/importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/pymodules/python2.7/nose/importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/tmp/tmpXx8lxx/surveilr/api/server.py", line 30, in <module>
    from routes import Mapper
ImportError: No module named routes

======================================================================
ERROR: Failure: ImportError (No module named webob)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/nose/loader.py", line 390, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/pymodules/python2.7/nose/importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/pymodules/python2.7/nose/importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/tmp/tmpXx8lxx/surveilr/tests/test_api_server.py", line 26, in <module>
    from webob import Request
ImportError: No module named webob

----------------------------------------------------------------------
Ran 3 tests in 0.811s

FAILED (errors=2)

[Soren Hansen (soren)]

...and python-{routes,webob}.

[Linux2Go Jenkins (linux2go-jenkins)]

The attempt to merge lp:~soren/surveilr/make-it-slightly-useful into lp:surveilr failed. Below is the output from the failed tests.

running nosetests
running egg_info
creating surveilr.egg-info
writing requirements to surveilr.egg-info/requires.txt
writing surveilr.egg-info/PKG-INFO
writing top-level names to surveilr.egg-info/top_level.txt
writing dependency_links to surveilr.egg-info/dependency_links.txt
writing manifest file 'surveilr.egg-info/SOURCES.txt'
reading manifest file 'surveilr.egg-info/SOURCES.txt'
writing manifest file 'surveilr.egg-info/SOURCES.txt'
running build_ext
ok

.EE..
======================================================================
ERROR: test_create_retrieve_metric (test_api_server.APIServerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmpt9Dlnm/surveilr/tests/test_api_server.py", line 61, in test_create_retrieve_metric
    resp = application(req)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 161, in __call__
    return self.func(req, *args, **kw)
  File "/tmp/tmpt9Dlnm/surveilr/api/server.py", line 125, in __call__
    return getattr(controller, method)(req, **kwargs)
  File "/tmp/tmpt9Dlnm/surveilr/api/server.py", line 51, in create
    service.save()
  File "/var/lib/jenkins/riakalchemy/riakalchemy/__init__.py", line 182, in save
    self._riak_obj = bucket.new(self.key, data=data_dict)
  File "/usr/lib/python2.7/dist-packages/riak/bucket.py", line 218, in new
    raise TypeError('Unicode data values are not supported.')
TypeError: Unicode data values are not supported.

======================================================================
ERROR: Create, retrieve, delete, attempt to retrieve again
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmpt9Dlnm/surveilr/tests/test_api_server.py", line 42, in test_create_retrieve_service
    resp = application(req)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 161, in __call__
    return self.func(req, *args, **kw)
  File "/tmp/tmpt9Dlnm/surveilr/api/server.py", line 125, in __call__
    return getattr(controller, method)(req, **kwargs)
  File "/tmp/tmpt9Dlnm/surveilr/api/server.py", line 51, in create
    service.save()
  File "/var/lib/jenkins/riakalchemy/riakalchemy/__init__.py", line 182, in save
    self._riak_obj = bucket.new(self.key, data=data_dict)
  File "/usr/lib/python2.7/dist-packages/riak/bucket.py", line 218, in new
    raise TypeError('Unicode data values are not supported.')
TypeError: Unicode data values are not supported.

----------------------------------------------------------------------
Ran 5 tests in 1.128s

FAILED (errors=2)

[Linux2Go Jenkins (linux2go-jenkins)]

The attempt to merge lp:~soren/surveilr/make-it-slightly-useful into lp:surveilr failed. Below is the output from the failed tests.

running nosetests
running egg_info
creating surveilr.egg-info
writing requirements to surveilr.egg-info/requires.txt
writing surveilr.egg-info/PKG-INFO
writing top-level names to surveilr.egg-info/top_level.txt
writing dependency_links to surveilr.egg-info/dependency_links.txt
writing manifest file 'surveilr.egg-info/SOURCES.txt'
reading manifest file 'surveilr.egg-info/SOURCES.txt'
writing manifest file 'surveilr.egg-info/SOURCES.txt'
running build_ext
ok

.EE..
======================================================================
ERROR: test_create_retrieve_metric (test_api_server.APIServerTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmpfUlS7f/surveilr/tests/test_api_server.py", line 61, in test_create_retrieve_metric
    resp = application(req)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 161, in __call__
    return self.func(req, *args, **kw)
  File "/tmp/tmpfUlS7f/surveilr/api/server.py", line 125, in __call__
    return getattr(controller, method)(req, **kwargs)
  File "/tmp/tmpfUlS7f/surveilr/api/server.py", line 51, in create
    service.save()
  File "/var/lib/jenkins/riakalchemy/riakalchemy/model.py", line 182, in save
    self._riak_obj = bucket.new(self.key, data=data_dict)
  File "/usr/lib/python2.7/dist-packages/riak/bucket.py", line 218, in new
    raise TypeError('Unicode data values are not supported.')
TypeError: Unicode data values are not supported.

======================================================================
ERROR: Create, retrieve, delete, attempt to retrieve again
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmpfUlS7f/surveilr/tests/test_api_server.py", line 42, in test_create_retrieve_service
    resp = application(req)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 161, in __call__
    return self.func(req, *args, **kw)
  File "/tmp/tmpfUlS7f/surveilr/api/server.py", line 125, in __call__
    return getattr(controller, method)(req, **kwargs)
  File "/tmp/tmpfUlS7f/surveilr/api/server.py", line 51, in create
    service.save()
  File "/var/lib/jenkins/riakalchemy/riakalchemy/model.py", line 182, in save
    self._riak_obj = bucket.new(self.key, data=data_dict)
  File "/usr/lib/python2.7/dist-packages/riak/bucket.py", line 218, in new
    raise TypeError('Unicode data values are not supported.')
TypeError: Unicode data values are not supported.

----------------------------------------------------------------------
Ran 5 tests in 0.382s

FAILED (errors=2)

=== added file 'setup.cfg'
--- setup.cfg	1970-01-01 00:00:00 +0000
+++ setup.cfg	2011-11-19 22:08:22 +0000
@@ -0,0 +1,2 @@
+[nosetests]
+with-doctest=1

=== added directory 'surveilr'
=== added file 'surveilr/__init__.py'
--- surveilr/__init__.py	1970-01-01 00:00:00 +0000
+++ surveilr/__init__.py	2011-11-19 22:08:22 +0000
@@ -0,0 +1,19 @@
+"""
+    Surveilr - Log aggregation, analysis and visualisation
+
+    Copyright (C) 2011  Linux2Go
+
+    This program is free software: you can redistribute it and/or
+    modify it under the terms of the GNU Affero General Public License
+    as published by the Free Software Foundation, either version 3 of
+    the License, or (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU Affero General Public License for more details.
+
+    You should have received a copy of the GNU Affero General Public
+    License along with this program.  If not, see
+    <http://www.gnu.org/licenses/>.
+"""

=== added directory 'surveilr/api'
=== added file 'surveilr/api/__init__.py'
=== added file 'surveilr/api/server.py'
--- surveilr/api/server.py	1970-01-01 00:00:00 +0000
+++ surveilr/api/server.py	2011-11-19 22:08:22 +0000
@@ -0,0 +1,134 @@
+#!/usr/bin/python
+"""
+    Surveilr - Log aggregation, analysis and visualisation
+
+    Copyright (C) 2011  Linux2Go
+
+    This program is free software: you can redistribute it and/or
+    modify it under the terms of the GNU Affero General Public License
+    as published by the Free Software Foundation, either version 3 of
+    the License, or (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU Affero General Public License for more details.
+
+    You should have received a copy of the GNU Affero General Public
+    License along with this program.  If not, see
+    <http://www.gnu.org/licenses/>.
+
+    Log collection server implementation
+"""
+
+import eventlet
+import eventlet.wsgi
+import json
+import time
+
+import riakalchemy
+from routes import Mapper
+from routes.util import URLGenerator
+
+from webob import exc
+from webob import Response
+from webob.dec import wsgify
+from webob.exc import HTTPNotFound
+
+from surveilr import models
+from surveilr import utils
+
+class ServiceController(object):
+    """Routes style controller for actions related to services"""
+
+    def create(self, req):
+        """Called for POST requests to /services
+
+        Creates the service, returns a JSON object with the ID assigned
+        to the service"""
+        data = json.loads(req.body)
+        service = models.Service(**data)
+        service.save()
+        response = {'id':service.key}
+        return Response(json.dumps(response))
+
+    def show(self, req, id):
+        """Called for GET requests to /services/{id}
+
+        Returns information for the given service"""
+        try:
+            service = models.Service.get(id)
+            return Response({'id': service.key})
+        except riakalchemy.NoSuchObjectError:
+            return HTTPNotFound()
+
+    def delete(self, req, id):
+        """Called for DELETE requests to /services/{id}
+
+        Delete the given service"""
+        models.Service.get(id).delete()
+        return Response('')
+
+
+class MetricController(object):
+    """Routes style controller for actions related to log entries"""
+    def create(self, req, service_name):
+        """Called for POST requests to /services/{id}/metrics
+
+        Logs a measurement against the service identified by {id}.
+        Returns an empty response"""
+        data = json.loads(req.body)
+        service = models.Service.get(service_name)
+        data['service'] = [service]
+        data['timestamp'] = utils.truncate(time.time(), 60)
+        models.LogEntry(**data).save()
+        return Response('')
+
+    def index(self, req, service_name):
+        """Called for GET requests to /services/{id}/metrics
+
+        Returns a list of metrics logged against the service identified
+        by {id}."""
+        service = models.Service.get(service_name)
+        retval = []
+        for x in models.LogEntry.get(service=service).all():
+            retval += [{'metrics': x.metrics, 'timestamp': x.timestamp}]
+        return Response(json.dumps(retval))
+
+class SurveilrApplication(object):
+    """The core Surveilr Monitoring WSGI application"""
+    controllers = {}
+
+    map = Mapper()
+    map.resource("metric", "metrics", controller='MetricController',
+                 path_prefix='/services/{service_name}')
+    map.resource("service", "services", controller='ServiceController')
+
+    @wsgify
+    def __call__(self, req):
+        """Where it all happens
+
+        Using the Mapper object, it finds the relevant controller
+        based on the URL and delegates the call to that."""
+        results = self.map.routematch(environ=req.environ)
+        if not results:
+            return exc.HTTPNotFound()
+        match, route = results
+        link = URLGenerator(self.map, req.environ)
+        req.urlvars = ((), match)
+        kwargs = match.copy()
+        controller = globals()[kwargs.pop('controller')]()
+        method = kwargs.pop('action')
+        req.link = link
+        req.route_name = route.name
+
+        return getattr(controller, method)(req, **kwargs)
+
+application = SurveilrApplication()
+
+def main():
+    socket = eventlet.listen(('', 9877))
+    eventlet.wsgi.server(socket, application)
+
+if __name__ == '__main__': # pragma: nocover
+    main()

=== added file 'surveilr/models.py'
--- surveilr/models.py	1970-01-01 00:00:00 +0000
+++ surveilr/models.py	2011-11-19 22:08:22 +0000
@@ -0,0 +1,54 @@
+"""
+    Surveilr - Log aggregation, analysis and visualisation
+
+    Copyright (C) 2011  Linux2Go
+
+    This program is free software: you can redistribute it and/or
+    modify it under the terms of the GNU Affero General Public License
+    as published by the Free Software Foundation, either version 3 of
+    the License, or (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU Affero General Public License for more details.
+
+    You should have received a copy of the GNU Affero General Public
+    License along with this program.  If not, see
+    <http://www.gnu.org/licenses/>.
+
+    Model definitions
+"""
+from riakalchemy import RiakObject
+from riakalchemy.types import Integer, String, Dict, RelatedObjects
+
+class Service(RiakObject):
+    """A service that is referenced by many LogEntry's
+
+    If the same logical service gets information from multiple sources,
+    it needs separate Service objects. However, if several metrics are
+    provided for a single service in each log entry, only one Service
+    is needed"""
+    bucket_name = 'services'
+    searchable = True
+
+    name = String()
+    most_recent_log_entry = RelatedObjects()
+
+class LogEntry(RiakObject):
+    """A log entry holding one or more metrics
+
+    A log entry "belongs" to a single Service. Each LogEntry
+    can hold multiple metrics, but all LogEntry's must
+    have the same set of metrics."""
+    bucket_name = 'log_entries'
+
+    timestamp = Integer()
+    metrics = Dict()
+    service = RelatedObjects(backref=True)
+
+    def post_save(self):
+        super(LogEntry, self).post_save()
+        service = self.service[0]
+        service.most_recent_log_entry = [self]
+        service.save()

=== added directory 'surveilr/tests'
=== added file 'surveilr/tests/test_api_server.py'
--- surveilr/tests/test_api_server.py	1970-01-01 00:00:00 +0000
+++ surveilr/tests/test_api_server.py	2011-11-19 22:08:22 +0000
@@ -0,0 +1,90 @@
+"""
+    Surveilr - Log aggregation, analysis and visualisation
+
+    Copyright (C) 2011  Linux2Go
+
+    This program is free software: you can redistribute it and/or
+    modify it under the terms of the GNU Affero General Public License
+    as published by the Free Software Foundation, either version 3 of
+    the License, or (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU Affero General Public License for more details.
+
+    You should have received a copy of the GNU Affero General Public
+    License along with this program.  If not, see
+    <http://www.gnu.org/licenses/>.
+
+    Tests for API server
+"""
+
+import json
+import mock
+import unittest
+from webob import Request
+
+from surveilr.api import server
+from surveilr.api.server import application
+from surveilr.api.server import main as server_main
+
+class APIServerTests(unittest.TestCase):
+    def setUp(self):
+        import riakalchemy
+        riakalchemy.connect()
+
+    def test_create_retrieve_service(self):
+        """Create, retrieve, delete, attempt to retrieve again"""
+        req = Request.blank('/services',
+                            method='POST',
+                            POST=json.dumps({'name': 'this_or_the_other'}))
+        resp = application(req)
+        self.assertEquals(resp.status_int, 200)
+
+        service_id = json.loads(resp.body)['id']
+
+        req = Request.blank('/services/%s' % service_id)
+        resp = application(req)
+        self.assertEquals(resp.status_int, 200)
+
+        req = Request.blank('/services/%s' % service_id, method='DELETE')
+        resp = application(req)
+        self.assertEquals(resp.status_int, 200)
+
+        req = Request.blank('/services/%s' % service_id)
+        resp = application(req)
+        self.assertEquals(resp.status_int, 404)
+
+    def test_create_retrieve_metric(self):
+        req = Request.blank('/services', method='POST', POST='{"name": "this_or_the_other"}')
+        resp = application(req)
+        self.assertEquals(resp.status_int, 200)
+
+        service_id = json.loads(resp.body)['id']
+        req = Request.blank('/services/%s/metrics' % service_id,
+                            method='POST',
+                            POST=json.dumps({
+                                         'timestamp': 13217362355575,
+                                         'metrics': {'duration': 85000,
+                                                     'response_size': 12435}}))
+        resp = application(req)
+        self.assertEquals(resp.status_int, 200)
+
+        req = Request.blank('/services/%s/metrics' % service_id)
+        resp = application(req)
+
+    def test_invalid_url(self):
+        req = Request.blank('/stuff')
+        resp = application(req)
+        self.assertEquals(resp.status_int, 404)
+
+    def test_main(self):
+        with mock.patch('surveilr.api.server.eventlet',
+                        spec=['listen', 'wsgi']) as eventlet:
+            socket_sentinel = mock.sentinel.return_value
+            eventlet.listen.return_value = socket_sentinel
+            server.main()
+            
+            eventlet.listen.assert_called_with(('', 9877))
+            eventlet.wsgi.server.assert_called_with(socket_sentinel, application)

=== added file 'surveilr/utils.py'
--- surveilr/utils.py	1970-01-01 00:00:00 +0000
+++ surveilr/utils.py	2011-11-19 22:08:22 +0000
@@ -0,0 +1,35 @@
+"""
+    Surveilr - Log aggregation, analysis and visualisation
+
+    Copyright (C) 2011  Linux2Go
+
+    This program is free software: you can redistribute it and/or
+    modify it under the terms of the GNU Affero General Public License
+    as published by the Free Software Foundation, either version 3 of
+    the License, or (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU Affero General Public License for more details.
+
+    You should have received a copy of the GNU Affero General Public
+    License along with this program.  If not, see
+    <http://www.gnu.org/licenses/>.
+
+    Utility functions
+"""
+
+def truncate(number, rounding_factor):
+    """Truncate to nearest arbitrary multiple
+
+    >>> truncate(1000, 100)
+    1000
+    >>> truncate(1099, 100)
+    1000
+    >>> truncate(1099, 50)
+    1050
+    >>> truncate(1099, 50)
+    1050
+    """
+    return (int(number) / int(rounding_factor)) * int(rounding_factor)