OpenStack Object Storage (swift)

Merge lp:~notmyname/swift/fix_query_parsing into lp:~hudson-openstack/swift/trunk

  1. fix_query_parsing
  2. Merge into trunk
Proposed by John Dickinson
Status: Merged
Approved by: gholt
Approved revision: 252
Merged at revision: 251
Proposed branch: lp:~notmyname/swift/fix_query_parsing
Merge into: lp:~hudson-openstack/swift/trunk
Diff against target: 69 lines (+28/-2)
3 files modified
swift/stats/access_processor.py (+4/-1)
test/unit/stats/test_access_processor.py (+24/-0)
test/unit/stats/test_log_processor.py (+0/-1)
To merge this branch: bzr merge lp:~notmyname/swift/fix_query_parsing
Reviewer Review Type Date Requested Status
Swift Core security contacts Pending
Review via email: mp+53917@code.launchpad.net

Description of the change

makes the query parameter parsing explicit in the access log parser

To post a comment you must log in.
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :
Download full text (72.6 KiB)

The attempt to merge lp:~notmyname/swift/fix_query_parsing into lp:swift failed. Below is the output from the failed tests.

running test
running egg_info
creating swift.egg-info
writing swift.egg-info/PKG-INFO
writing top-level names to swift.egg-info/top_level.txt
writing dependency_links to swift.egg-info/dependency_links.txt
writing entry points to swift.egg-info/entry_points.txt
writing manifest file 'swift.egg-info/SOURCES.txt'
reading manifest file 'swift.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'ChangeLog'
writing manifest file 'swift.egg-info/SOURCES.txt'
running build_ext

UNABLE TO READ FUNCTIONAL TESTS CONFIG FILE
UNABLE TO READ FUNCTIONAL TESTS CONFIG FILE
SKIPPING FUNCTIONAL TESTS DUE TO NO CONFIG
testAccountHead (test.functional.tests.TestAccount) ... SKIP
testContainerListing (test.functional.tests.TestAccount) ... SKIP
testContainerSerializedInfo (test.functional.tests.TestAccount) ... SKIP
testContainersOrderedByName (test.functional.tests.TestAccount) ... SKIP
testInvalidAuthToken (test.functional.tests.TestAccount) ... SKIP
testInvalidPath (test.functional.tests.TestAccount) ... SKIP
testInvalidUTF8Path (test.functional.tests.TestAccount) ... SKIP
testLastContainerMarker (test.functional.tests.TestAccount) ... SKIP
testListingLimit (test.functional.tests.TestAccount) ... SKIP
testMarkerLimitContainerList (test.functional.tests.TestAccount) ... SKIP
testNoAuthToken (test.functional.tests.TestAccount) ... SKIP
testPUT (test.functional.tests.TestAccount) ... SKIP
testVersionOnlyPath (test.functional.tests.TestAccount) ... SKIP
testGetRequest (test.functional.tests.TestAccountNoContainers) ... SKIP
testGetRequest (test.functional.tests.TestAccountNoContainersUTF8) ... SKIP
testAccountHead (test.functional.tests.TestAccountUTF8) ... SKIP
testContainerListing (test.functional.tests.TestAccountUTF8) ... SKIP
testContainerSerializedInfo (test.functional.tests.TestAccountUTF8) ... SKIP
testContainersOrderedByName (test.functional.tests.TestAccountUTF8) ... SKIP
testInvalidAuthToken (test.functional.tests.TestAccountUTF8) ... SKIP
testInvalidPath (test.functional.tests.TestAccountUTF8) ... SKIP
testInvalidUTF8Path (test.functional.tests.TestAccountUTF8) ... SKIP
testLastContainerMarker (test.functional.tests.TestAccountUTF8) ... SKIP
testListingLimit (test.functional.tests.TestAccountUTF8) ... SKIP
testMarkerLimitContainerList (test.functional.tests.TestAccountUTF8) ... SKIP
testNoAuthToken (test.functional.tests.TestAccountUTF8) ... SKIP
testPUT (test.functional.tests.TestAccountUTF8) ... SKIP
testVersionOnlyPath (test.functional.tests.TestAccountUTF8) ... SKIP
testContainerExistenceCachingProblem (test.functional.tests.TestContainer) ... SKIP
testContainerFileList (test.functional.tests.TestContainer) ... SKIP
testContainerFileListOnContainerThatDoesNotExist (test.functional.tests.TestContainer) ... SKIP
testContainerFileListWithLimit (test.functional.tests.TestContainer) ... SKIP
testContainerInfo (test.functional.tests.TestContainer) ... SKIP
testContainerInfoOnContainerThatDoesNotExist (test.functional.tests.TestContainer) ... SKIP
testContainerNameLimit (test.functional.tes...

lp:~notmyname/swift/fix_query_parsing updated
252. By John Dickinson

fixed tests

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'swift/stats/access_processor.py'
2--- swift/stats/access_processor.py 2011-02-02 21:39:08 +0000
3+++ swift/stats/access_processor.py 2011-03-17 22:24:35 +0000
4@@ -20,6 +20,8 @@
5 from swift.common.utils import split_path, get_logger
6
7 month_map = '_ Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec'.split()
8+LISTING_PARAMS = set(
9+ 'path limit format delimiter marker end_marker prefix'.split())
10
11
12 class AccessLogProcessor(object):
13@@ -95,7 +97,8 @@
14 # (format, path, delimiter, etc.). Save a "1" here
15 # to indicate that this request is 1 request for
16 # its respective key.
17- d[k] = 1
18+ if k in LISTING_PARAMS:
19+ d[k] = 1
20 d['client_ip'] = client_ip
21 d['lb_ip'] = lb_ip
22 d['method'] = method
23
24=== modified file 'test/unit/stats/test_access_processor.py'
25--- test/unit/stats/test_access_processor.py 2011-01-17 17:07:58 +0000
26+++ test/unit/stats/test_access_processor.py 2011-03-17 22:24:35 +0000
27@@ -21,6 +21,30 @@
28
29 class TestAccessProcessor(unittest.TestCase):
30
31+ def test_log_line_parser_query_args(self):
32+ p = access_processor.AccessLogProcessor({})
33+ log_line = [str(x) for x in range(18)]
34+ log_line[1] = 'proxy-server'
35+ log_line[4] = '1/Jan/3/4/5/6'
36+ query = 'foo'
37+ for param in access_processor.LISTING_PARAMS:
38+ query += '&%s=blah' % param
39+ log_line[6] = '/v1/a/c/o?%s' % query
40+ log_line = 'x'*16 + ' '.join(log_line)
41+ res = p.log_line_parser(log_line)
42+ expected = {'code': 8, 'processing_time': '17', 'auth_token': '11',
43+ 'month': '01', 'second': '6', 'year': '3', 'tz': '+0000',
44+ 'http_version': '7', 'object_name': 'o', 'etag': '14',
45+ 'method': '5', 'trans_id': '15', 'client_ip': '2',
46+ 'bytes_out': 13, 'container_name': 'c', 'day': '1',
47+ 'minute': '5', 'account': 'a', 'hour': '4',
48+ 'referrer': '9', 'request': '/v1/a/c/o',
49+ 'user_agent': '10', 'bytes_in': 12, 'lb_ip': '3'}
50+ for param in access_processor.LISTING_PARAMS:
51+ expected[param] = 1
52+ expected['query'] = query
53+ self.assertEquals(res, expected)
54+
55 def test_log_line_parser_field_count(self):
56 p = access_processor.AccessLogProcessor({})
57 # too few fields
58
59=== modified file 'test/unit/stats/test_log_processor.py'
60--- test/unit/stats/test_log_processor.py 2011-02-16 22:29:59 +0000
61+++ test/unit/stats/test_log_processor.py 2011-03-17 22:24:35 +0000
62@@ -131,7 +131,6 @@
63 'http_version': 'HTTP/1.0',
64 'object_name': 'bar',
65 'etag': '-',
66- 'foo': 1,
67 'method': 'GET',
68 'trans_id': 'txfa431231-7f07-42fd-8fc7-7da9d8cc1f90',
69 'client_ip': '1.2.3.4',