Merge lp:~cjwatson/launchpad/private-loggerhead into lp:launchpad

Proposed by Colin Watson on 2018-05-16
Status: Merged
Merged at revision: 18691
Proposed branch: lp:~cjwatson/launchpad/private-loggerhead
Merge into: lp:launchpad
Diff against target: 118 lines (+50/-8)
4 files modified
lib/launchpad_loggerhead/app.py (+19/-8)
lib/launchpad_loggerhead/tests.py (+25/-0)
lib/launchpad_loggerhead/wsgi.py (+1/-0)
lib/lp/services/config/schema-lazr.conf (+5/-0)
To merge this branch: bzr merge lp:~cjwatson/launchpad/private-loggerhead
Reviewer Review Type Date Requested Status
William Grant code 2018-05-16 Approve on 2018-06-21
Review via email: mp+345680@code.launchpad.net

Commit message

Make loggerhead listen on an additional private port without authentication.

Description of the change

The idea of this is to be able to use loggerhead as an API from Launchpad, mostly its JSON endpoints.

paste.httpserver couldn't listen on multiple ports without having to bring up entire new thread pools for each one, so I ported this to gunicorn. I've tried to keep the worker configuration as close to what was there before as possible, but of course we should keep an eye on poor old babaco's memory usage after deployment.

The one thing I couldn't figure out how to port was the NoLockingFileHandler business. In principle it would be possible to do this by going through all the existing loggers in LoggerheadLogger.setup and hacking out the acquire and release methods of their handlers. However, gunicorn's different thread management may avoid the problem, and the whole thing is extremely dubious anyway; bearing in mind that locks are used for more than just writing to log files, it wouldn't surprise me if it caused as many weird hangs as it prevented. My plan is to ignore this until we see a problem that can be ascribed to it.

To post a comment you must log in.
William Grant (wgrant) wrote :

Can you split out the gunicorn port into an earlier MP? We can get that landed and deployed now and separately work out the security around the private port.

review: Needs Fixing
18660. By Colin Watson on 2018-06-06

Merge devel.

Colin Watson (cjwatson) wrote :

loggerhead is now on gunicorn on production. I believe I understand the necessary port security changes now after my work on this last week, so I can take care of that if this is approved.

William Grant (wgrant) wrote :

We'll need to get production/qastaging/staging host firewalls sorted out before this can land.

review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/launchpad_loggerhead/app.py'
2--- lib/launchpad_loggerhead/app.py 2017-01-14 15:16:36 +0000
3+++ lib/launchpad_loggerhead/app.py 2018-06-06 12:47:56 +0000
4@@ -1,4 +1,4 @@
5-# Copyright 2009-2017 Canonical Ltd. This software is licensed under the
6+# Copyright 2009-2018 Canonical Ltd. This software is licensed under the
7 # GNU Affero General Public License version 3 (see the file LICENSE).
8
9 import logging
10@@ -45,6 +45,7 @@
11 from lp.code.interfaces.codehosting import (
12 BRANCH_TRANSPORT,
13 LAUNCHPAD_ANONYMOUS,
14+ LAUNCHPAD_SERVICES,
15 )
16 from lp.codehosting.safe_open import safe_open
17 from lp.codehosting.vfs import get_lp_server
18@@ -185,6 +186,8 @@
19 raise HTTPMovedPermanently(next_url)
20
21 def __call__(self, environ, start_response):
22+ request_is_private = (
23+ environ['SERVER_PORT'] == str(config.codebrowse.private_port))
24 environ['loggerhead.static.url'] = environ['SCRIPT_NAME']
25 if environ['PATH_INFO'].startswith('/static/'):
26 path_info_pop(environ)
27@@ -193,15 +196,23 @@
28 return favicon_app(environ, start_response)
29 elif environ['PATH_INFO'] == '/robots.txt':
30 return robots_app(environ, start_response)
31- elif environ['PATH_INFO'].startswith('/+login'):
32- return self._complete_login(environ, start_response)
33- elif environ['PATH_INFO'].startswith('/+logout'):
34- return self._logout(environ, start_response)
35+ elif not request_is_private:
36+ if environ['PATH_INFO'].startswith('/+login'):
37+ return self._complete_login(environ, start_response)
38+ elif environ['PATH_INFO'].startswith('/+logout'):
39+ return self._logout(environ, start_response)
40 path = environ['PATH_INFO']
41 trailingSlashCount = len(path) - len(path.rstrip('/'))
42- identity_url = environ[self.session_var].get(
43- 'identity_url', LAUNCHPAD_ANONYMOUS)
44- user = environ[self.session_var].get('user', LAUNCHPAD_ANONYMOUS)
45+ if request_is_private:
46+ # Requests on the private port are internal API requests from
47+ # something that has already performed security checks. As
48+ # such, they get read-only access to everything.
49+ identity_url = LAUNCHPAD_SERVICES
50+ user = LAUNCHPAD_SERVICES
51+ else:
52+ identity_url = environ[self.session_var].get(
53+ 'identity_url', LAUNCHPAD_ANONYMOUS)
54+ user = environ[self.session_var].get('user', LAUNCHPAD_ANONYMOUS)
55 lp_server = get_lp_server(
56 identity_url, branch_transport=self.get_transport())
57 lp_server.start_server()
58
59=== modified file 'lib/launchpad_loggerhead/tests.py'
60--- lib/launchpad_loggerhead/tests.py 2018-06-06 08:57:53 +0000
61+++ lib/launchpad_loggerhead/tests.py 2018-06-06 12:47:56 +0000
62@@ -204,3 +204,28 @@
63 self.assertEqual(
64 "testopenid.dev:8085",
65 urlsplit(response.headers["Location"]).netloc)
66+
67+ def test_private_port_public_branch(self):
68+ # Requests for public branches on the private port are allowed.
69+ db_branch, _ = self.create_branch_and_tree()
70+ branch_url = "http://127.0.0.1:%d/%s" % (
71+ config.codebrowse.private_port, db_branch.unique_name)
72+ response = requests.get(branch_url)
73+ self.assertEqual(200, response.status_code)
74+ title_tag = soupmatchers.Tag(
75+ "page title", "title", text="%s : changes" % db_branch.unique_name)
76+ self.assertThat(response.text, soupmatchers.HTMLContains(title_tag))
77+
78+ def test_private_port_private_branch(self):
79+ # Requests for private branches on the private port are allowed.
80+ db_branch, _ = self.create_branch_and_tree(
81+ information_type=InformationType.USERDATA)
82+ naked_branch = removeSecurityProxy(db_branch)
83+ branch_url = "http://127.0.0.1:%d/%s" % (
84+ config.codebrowse.private_port, naked_branch.unique_name)
85+ response = requests.get(branch_url)
86+ self.assertEqual(200, response.status_code)
87+ title_tag = soupmatchers.Tag(
88+ "page title", "title",
89+ text="%s : changes" % naked_branch.unique_name)
90+ self.assertThat(response.text, soupmatchers.HTMLContains(title_tag))
91
92=== modified file 'lib/launchpad_loggerhead/wsgi.py'
93--- lib/launchpad_loggerhead/wsgi.py 2018-06-04 14:02:57 +0000
94+++ lib/launchpad_loggerhead/wsgi.py 2018-06-06 12:47:56 +0000
95@@ -105,6 +105,7 @@
96 "accesslog": os.path.join(log_folder, "access.log"),
97 "bind": [
98 "%s:%s" % (listen_host, config.codebrowse.port),
99+ "%s:%s" % (listen_host, config.codebrowse.private_port),
100 ],
101 "errorlog": os.path.join(log_folder, "debug.log"),
102 # Trust that firewalls only permit sending requests to
103
104=== modified file 'lib/lp/services/config/schema-lazr.conf'
105--- lib/lp/services/config/schema-lazr.conf 2018-05-31 13:14:12 +0000
106+++ lib/lp/services/config/schema-lazr.conf 2018-06-06 12:47:56 +0000
107@@ -199,6 +199,11 @@
108 # datatype: int
109 port: 8080
110
111+# The port to listen on for private API requests.
112+#
113+# datatype: int
114+private_port: 8090
115+
116 # A file path that contains a secret used for verifying cookies sent
117 # to codebrowse. This file should ideally contain 64 bytes of random
118 # data, and should be considered private -- if it is disclosed, we