Merge lp:~pwlars/core-selftest-agent/agent-config into lp:core-selftest-agent

[Paul Larson]

Add config bits

Add config bits

[Francis Ginther (fginther)]

A have a couple of inline comments. I think we're now safe to remove the modified hostname methods.

review: Needs Fixing

[Paul Larson (pwlars)]

Actually, I don't know that I'll even need a retry queue so I took it out for now. It may be necessary to retry internally for a failed manifest download or something, but I don't think this service has a concept of an infrastructure failure outside of its control that would require requeueing. I also replaced the hostname bit with socket.gethostname()

review: Needs Resubmitting

[Francis Ginther (fginther)]

I'm happy with this. As mentioned over IRC, there really is no reason for the agent to retry messages. The input is a fan-out exchange so every consumer will already have a copy, no need to throw it back to see if another consumer can make use of it. I also doubt we'll need deadletter, but am happy to wait and see on this.

I'm curious as to why you kept ".replace('-machine', '')" on the hostname. I don't think this is hurting anything. But I'm not going to block on this either.

review: Approve

[Ubuntu CI Bot (uci-bot)]

The attempt to merge lp:~pwlars/core-selftest-agent/agent-config into lp:core-selftest-agent failed. Below is the output from the failed tests.

Using base prefix '/usr'
New python executable in /tmp/venv-core-selftest-agentwtd8oky_/bin/python3
Also creating executable in /tmp/venv-core-selftest-agentwtd8oky_/bin/python
Installing setuptools, pip...done.
Running virtualenv with interpreter /usr/bin/python3
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking python-logstash==0.4.2 (from -r requirements.txt (line 1))
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/tmp/venv-core-selftest-agentwtd8oky_/lib/python3.4/site-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/tmp/venv-core-selftest-agentwtd8oky_/lib/python3.4/site-packages/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/tmp/venv-core-selftest-agentwtd8oky_/lib/python3.4/site-packages/pip/req.py", line 1177, in prepare_files
    url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
  File "/tmp/venv-core-selftest-agentwtd8oky_/lib/python3.4/site-packages/pip/index.py", line 252, in find_requirement
    for page in self._get_pages(locations, req):
  File "/tmp/venv-core-selftest-agentwtd8oky_/lib/python3.4/site-packages/pip/index.py", line 387, in _get_pages
    page = self._get_page(location, req)
  File "/tmp/venv-core-selftest-agentwtd8oky_/lib/python3.4/site-packages/pip/index.py", line 568, in _get_page
    session=self.session,
  File "/tmp/venv-core-selftest-agentwtd8oky_/lib/python3.4/site-packages/pip/index.py", line 670, in get_page
    resp = session.get(url, headers={"Accept": "text/html"})
  File "/tmp/venv-core-selftest-agentwtd8oky_/lib/python3.4/site-packages/pip/_vendor/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/tmp/venv-core-selftest-agentwtd8oky_/lib/python3.4/site-packages/pip/download.py", line 237, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/tmp/venv-core-selftest-agentwtd8oky_/lib/python3.4/site-packages/pip/_vendor/requests/sessions.py", line 349, in request
    prep = self.prepare_request(req)
  File "/tmp/venv-core-selftest-agentwtd8oky_/lib/python3.4/site-packages/pip/_vendor/requests/sessions.py", line 287, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/tmp/venv-core-selftest-agentwtd8oky_/lib/python3.4/site-packages/pip/_vendor/requests/models.py", line 287, in prepare
    self.prepare_url(url, params)
  File "/tmp/venv-core-selftest-agentwtd8oky_/lib/python3.4/site-packages/pip/_vendor/requests/models.py", line 338, in prepare_url
    "Perhaps you meant http://{0}?".format(url))
pip._vendor.requests.exceptions.MissingSchema: Invalid URL '/tmp/tmpgxqm1lttpip-cache-/cache': No schema supplied. Perhaps you meant http:///tmp/tmpgxqm1lttpip-cache-/cache?

Storing debug log for failure in /home/tarmac/.pip/pip.log
INFO: Executing: virtualenv -p python3 /tmp/venv-core-selftest-agentwtd8oky_
INFO: Executing: /tmp/venv-core-selftest-agentwtd8...

[Ubuntu CI Bot (uci-bot)]

There are additional revisions which have not been approved in review. Please seek review and approval of these new revisions.

[Paul Larson (pwlars)]

review: Needs Resubmitting

[Para Siva (psivaa)]

I see core-service.conf and SOLUTION_NAME=core-image-testing as potentially confusing but could notice some other services in the same solution are using this.

review: Approve

[Paul Larson (pwlars)]

Yeah, that's just a placeholder for now. A lot of that is going to need to change.

=== modified file 'called-by-tarmac.py'
--- called-by-tarmac.py	2015-05-20 22:06:09 +0000
+++ called-by-tarmac.py	2015-05-22 16:26:21 +0000
@@ -59,6 +59,12 @@
         pip_cache = os.path.join(pip_dir, 'cache')
         all_cmds = (
             [
+                'bzr',
+                'branch',
+                PIP_CACHE_BRANCH,
+                pip_cache,
+            ],
+            [
                 'virtualenv',
                 '-p',
                 'python3',

=== modified file 'core_selftest_agent/__init__.py'
--- core_selftest_agent/__init__.py	2015-05-20 17:30:07 +0000
+++ core_selftest_agent/__init__.py	2015-05-22 16:26:21 +0000
@@ -15,6 +15,50 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
+import argparse
+import configparser
+import logging
+import os
+
+from uservice_utils.logging import (
+    configure_service_logging,
+    ExtraLogger,
+)
+
+from core_selftest_agent import constants
+
+
+def read_config():
+    parser = argparse.ArgumentParser(description='Core Selftest Agent.')
+    parser.add_argument(
+        '-c',
+        '--conf',
+        default='core-service.conf',
+        help='Configuration file path'
+    )
+    args = parser.parse_args()
+
+    # Load configuration options.
+    config = configparser.ConfigParser()
+    config.read(args.conf)
+    return config
+
+
+def get_logger(name, extra={}):
+    if logging.getLoggerClass() != ExtraLogger:
+        logging.setLoggerClass(ExtraLogger)
+    logger = logging.getLogger(name)
+    e = constants.LOGGING_EXTRA.copy()
+    e.update(extra)
+    logger.set_extra_args(e)
+    return logger
+
 
 def main():
-    pass
+    config = read_config()
+    log_path = os.path.abspath(
+        os.path.join(__file__, '../../logs/core-selftest-agent.log'))
+    configure_service_logging(
+        log_path,
+        config['logstash'] if 'logstash' in config else None
+    )

=== added file 'core_selftest_agent/constants.py'
--- core_selftest_agent/constants.py	1970-01-01 00:00:00 +0000
+++ core_selftest_agent/constants.py	2015-05-22 16:26:21 +0000
@@ -0,0 +1,42 @@
+# core-selftest-agent
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+"""Constants for this service."""
+import socket
+
+# Queue names are constant, rather than being defined in the config.
+
+# The queue we listen to for new payloads to check:
+INPUT_QUEUE = "pm.candidates"
+
+# The queue we send events on to request image build and test
+OUTPUT_QUEUE = "core.package.v1"
+
+# The queue we put fatally error'd payloads into:
+DEAD_LETTER_QUEUE = "core.deadletters.v1"
+
+SOLUTION_NAME = "core-image-testing"
+
+SERVICE_NAME = "core-selftest-agent"
+
+HOSTNAME = socket.gethostname().replace('-machine', '')
+
+LOGGING_EXTRA = {
+    'solution': SOLUTION_NAME,
+    'service': SERVICE_NAME,
+    'hostname': HOSTNAME,
+}

=== modified file 'requirements.txt'
--- requirements.txt	2015-05-20 17:10:57 +0000
+++ requirements.txt	2015-05-22 16:26:21 +0000
@@ -0,0 +1,2 @@
+python-logstash==0.4.2
+uservice-utils==1.0.2.1