cloud-init

Merge lp:~harlowja/cloud-init/tox-venvs into lp:~cloud-init-dev/cloud-init/trunk

  1. tox-venvs
  2. Merge into trunk
Proposed by Joshua Harlow
Status: Merged
Merge reported by: Scott Moser
Merged at revision: not available
Proposed branch: lp:~harlowja/cloud-init/tox-venvs
Merge into: lp:~cloud-init-dev/cloud-init/trunk
Diff against target: 431 lines (+64/-284)
8 files modified
HACKING.rst (+2/-2)
Makefile (+12/-26)
pylintrc (+0/-19)
test-requirements.txt (+1/-3)
tools/hacking.py (+0/-170)
tools/run-pep8 (+0/-38)
tools/run-pylint (+0/-26)
tox.ini (+49/-0)
To merge this branch: bzr merge lp:~harlowja/cloud-init/tox-venvs
Reviewer Review Type Date Requested Status
Server Team CI bot continuous-integration Needs Fixing
cloud-init Commiters Pending
Review via email: mp+203225@code.launchpad.net

Description of the change

Use a venv for testing and for lint checking

To post a comment you must log in.
Revision history for this message
Scott Moser (smoser) wrote :

Thanks. this is really nice.

need to be able to run 'make test' without tox. Ie, similar to in openstack using only packaged versions. This is important to be able to run tests on build (which won't have access to stuff other than the archive).

you replaced pep8 and pylint with 'flake8'. I think I'm ok with that, but we have lots of '# pylint:' things throughout the code. if we're dropping that we might as well clean them out. I think thats fine as a separate commit though.

Revision history for this message
Joshua Harlow (harlowja) wrote :

Makes sense, I understand the need for both, will make a few tweaks to do that.

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:934
https://server-team-jenkins.canonical.com/job/cloud-init-ci/57/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/lp-vote-on-merge/31/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/cloud-init-ci/57/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote :

marking this merged.
we use tox in trunk now.
if you think there is something missing, please propose a merge to cloud-init git.
thanks.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'HACKING.rst'
--- HACKING.rst 2013-02-05 07:10:36 +0000
+++ HACKING.rst 2014-01-26 01:03:26 +0000
@@ -19,9 +19,9 @@
1919
20 - ``bzr commit``20 - ``bzr commit``
2121
22* Check pylint and pep8 and test, and address any issues:22* Check flake8 and test, and address any issues:
2323
24 - ``make test pylint pep8``24 - ``make test flake8``
2525
26* Push to launchpad to a personal branch:26* Push to launchpad to a personal branch:
2727
2828
=== modified file 'Makefile'
--- Makefile 2014-01-20 18:50:04 +0000
+++ Makefile 2014-01-26 01:03:26 +0000
@@ -2,40 +2,27 @@
2PY_FILES=$(shell find cloudinit bin tests tools -name "*.py" -type f )2PY_FILES=$(shell find cloudinit bin tests tools -name "*.py" -type f )
3PY_FILES+="bin/cloud-init"3PY_FILES+="bin/cloud-init"
44
5PY_ENV=$(shell python -c 'import sys; print("py%s%s" % (sys.version_info[0:2]))')
6
5YAML_FILES=$(shell find cloudinit bin tests tools -name "*.yaml" -type f )7YAML_FILES=$(shell find cloudinit bin tests tools -name "*.yaml" -type f )
6YAML_FILES+=$(shell find doc/examples -name "cloud-config*.txt" -type f )8YAML_FILES+=$(shell find doc/examples -name "cloud-config*.txt" -type f )
79
8CHANGELOG_VERSION=$(shell $(CWD)/tools/read-version)10CHANGELOG_VERSION=$(shell $(CWD)/tools/read-version)
9CODE_VERSION=$(shell python -c "from cloudinit import version; print version.version_string()")11CODE_VERSION=$(shell python -c "from cloudinit import version; print version.version_string()")
1012
11PIP_INSTALL := pip install
12
13ifeq ($(distro),)13ifeq ($(distro),)
14 distro = redhat14 distro = redhat
15endif15endif
1616
17all: test check_version17all: test check_version
1818
19pep8:19flake8:
20 @$(CWD)/tools/run-pep8 $(PY_FILES)20 @echo "Running flake8 using tox..."
2121 @tox -e flake8 $(PY_FILES)
22pylint:22
23 @$(CWD)/tools/run-pylint $(PY_FILES)23test:
2424 @echo "Running tests for env=$(PY_ENV) using tox..."
25pyflakes:25 @tox -e $(PY_ENV)
26 pyflakes $(PY_FILES)
27
28pip-requirements:
29 @echo "Installing cloud-init dependencies..."
30 $(PIP_INSTALL) -r "$@.txt" -q
31
32pip-test-requirements:
33 @echo "Installing cloud-init test dependencies..."
34 $(PIP_INSTALL) -r "$@.txt" -q
35
36test: clean_pyc
37 @echo "Running tests..."
38 @nosetests $(noseopts) tests/
3926
40check_version:27check_version:
41 @if [ "$(CHANGELOG_VERSION)" != "$(CODE_VERSION)" ]; then \28 @if [ "$(CHANGELOG_VERSION)" != "$(CODE_VERSION)" ]; then \
@@ -44,13 +31,13 @@
44 else true; fi31 else true; fi
4532
46clean_pyc:33clean_pyc:
47 @find . -type f -name "*.pyc" -delete34 find . -type f -name "*.pyc" -delete
4835
492to3:362to3:
50 2to3 $(PY_FILES)37 2to3 $(PY_FILES)
5138
52clean: clean_pyc39clean: clean_pyc
53 rm -rf /var/log/cloud-init.log /var/lib/cloud/40 rm -rf /var/log/cloud-init.log /var/lib/cloud/ $(CWD)/*.egg-info $(CWD)/.tox
5441
55yaml:42yaml:
56 @$(CWD)/tools/validate-yaml.py $(YAML_FILES)43 @$(CWD)/tools/validate-yaml.py $(YAML_FILES)
@@ -61,5 +48,4 @@
61deb:48deb:
62 ./packages/bddeb49 ./packages/bddeb
6350
64.PHONY: test pylint pyflakes 2to3 clean pep8 rpm deb yaml check_version51.PHONY: test flake8 2to3 clean rpm deb yaml check_version clean_pyc
65.PHONY: pip-test-requirements pip-requirements clean_pyc
6652
=== removed file 'pylintrc'
--- pylintrc 2012-10-28 02:25:48 +0000
+++ pylintrc 1970-01-01 00:00:00 +0000
@@ -1,19 +0,0 @@
1[General]
2init-hook='import sys; sys.path.append("tests/")'
3
4[MESSAGES CONTROL]
5# See: http://pylint-messages.wikidot.com/all-codes
6# W0142: *args and **kwargs are fine.
7# W0511: TODOs in code comments are fine.
8# W0702: No exception type(s) specified
9# W0703: Catch "Exception"
10# C0103: Invalid name
11# C0111: Missing docstring
12disable=W0142,W0511,W0702,W0703,C0103,C0111
13
14[REPORTS]
15reports=no
16include-ids=yes
17
18[FORMAT]
19max-line-length=79
200
=== modified file 'test-requirements.txt'
--- test-requirements.txt 2014-01-18 07:46:19 +0000
+++ test-requirements.txt 2014-01-26 01:03:26 +0000
@@ -1,6 +1,4 @@
1httpretty>=0.7.11httpretty>=0.7.1
2mocker2mocker
3nose3nose
4pep84hacking>=0.8.0,<0.9
5pyflakes
6pylint
75
=== removed file 'tools/hacking.py'
--- tools/hacking.py 2012-11-11 03:32:49 +0000
+++ tools/hacking.py 1970-01-01 00:00:00 +0000
@@ -1,170 +0,0 @@
1#!/usr/bin/env python
2# vim: tabstop=4 shiftwidth=4 softtabstop=4
3
4# Copyright (c) 2012, Cloudscaling
5# All Rights Reserved.
6#
7# Licensed under the Apache License, Version 2.0 (the "License"); you may
8# not use this file except in compliance with the License. You may obtain
9# a copy of the License at
10#
11# http://www.apache.org/licenses/LICENSE-2.0
12#
13# Unless required by applicable law or agreed to in writing, software
14# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
15# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
16# License for the specific language governing permissions and limitations
17# under the License.
18
19"""cloudinit HACKING file compliance testing (based off of nova hacking.py)
20
21built on top of pep8.py
22"""
23
24import inspect
25import logging
26import re
27import sys
28
29import pep8
30
31# Don't need this for testing
32logging.disable('LOG')
33
34# N1xx comments
35# N2xx except
36# N3xx imports
37# N4xx docstrings
38# N[5-9]XX (future use)
39
40DOCSTRING_TRIPLE = ['"""', "'''"]
41VERBOSE_MISSING_IMPORT = False
42_missingImport = set([])
43
44
45def import_normalize(line):
46 # convert "from x import y" to "import x.y"
47 # handle "from x import y as z" to "import x.y as z"
48 split_line = line.split()
49 if (line.startswith("from ") and "," not in line and
50 split_line[2] == "import" and split_line[3] != "*" and
51 split_line[1] != "__future__" and
52 (len(split_line) == 4 or
53 (len(split_line) == 6 and split_line[4] == "as"))):
54 return "import %s.%s" % (split_line[1], split_line[3])
55 else:
56 return line
57
58
59def cloud_import_alphabetical(physical_line, line_number, lines):
60 """Check for imports in alphabetical order.
61
62 HACKING guide recommendation for imports:
63 imports in human alphabetical order
64 N306
65 """
66 # handle import x
67 # use .lower since capitalization shouldn't dictate order
68 split_line = import_normalize(physical_line.strip()).lower().split()
69 split_previous = import_normalize(lines[line_number - 2])
70 split_previous = split_previous.strip().lower().split()
71 # with or without "as y"
72 length = [2, 4]
73 if (len(split_line) in length and len(split_previous) in length and
74 split_line[0] == "import" and split_previous[0] == "import"):
75 if split_line[1] < split_previous[1]:
76 return (0, "N306: imports not in alphabetical order (%s, %s)"
77 % (split_previous[1], split_line[1]))
78
79
80def cloud_docstring_start_space(physical_line):
81 """Check for docstring not start with space.
82
83 HACKING guide recommendation for docstring:
84 Docstring should not start with space
85 N401
86 """
87 pos = max([physical_line.find(i) for i in DOCSTRING_TRIPLE]) # start
88 if (pos != -1 and len(physical_line) > pos + 1):
89 if (physical_line[pos + 3] == ' '):
90 return (pos, "N401: one line docstring should not start with"
91 " a space")
92
93
94def cloud_todo_format(physical_line):
95 """Check for 'TODO()'.
96
97 HACKING guide recommendation for TODO:
98 Include your name with TODOs as in "#TODO(termie)"
99 N101
100 """
101 pos = physical_line.find('TODO')
102 pos1 = physical_line.find('TODO(')
103 pos2 = physical_line.find('#') # make sure it's a comment
104 if (pos != pos1 and pos2 >= 0 and pos2 < pos):
105 return pos, "N101: Use TODO(NAME)"
106
107
108def cloud_docstring_one_line(physical_line):
109 """Check one line docstring end.
110
111 HACKING guide recommendation for one line docstring:
112 A one line docstring looks like this and ends in a period.
113 N402
114 """
115 pos = max([physical_line.find(i) for i in DOCSTRING_TRIPLE]) # start
116 end = max([physical_line[-4:-1] == i for i in DOCSTRING_TRIPLE]) # end
117 if (pos != -1 and end and len(physical_line) > pos + 4):
118 if (physical_line[-5] != '.'):
119 return pos, "N402: one line docstring needs a period"
120
121
122def cloud_docstring_multiline_end(physical_line):
123 """Check multi line docstring end.
124
125 HACKING guide recommendation for docstring:
126 Docstring should end on a new line
127 N403
128 """
129 pos = max([physical_line.find(i) for i in DOCSTRING_TRIPLE]) # start
130 if (pos != -1 and len(physical_line) == pos):
131 print physical_line
132 if (physical_line[pos + 3] == ' '):
133 return (pos, "N403: multi line docstring end on new line")
134
135
136current_file = ""
137
138
139def readlines(filename):
140 """Record the current file being tested."""
141 pep8.current_file = filename
142 return open(filename).readlines()
143
144
145def add_cloud():
146 """Monkey patch pep8 for cloud-init guidelines.
147
148 Look for functions that start with cloud_
149 and add them to pep8 module.
150
151 Assumes you know how to write pep8.py checks
152 """
153 for name, function in globals().items():
154 if not inspect.isfunction(function):
155 continue
156 if name.startswith("cloud_"):
157 exec("pep8.%s = %s" % (name, name)) # pylint: disable=W0122
158
159if __name__ == "__main__":
160 # NOVA based 'hacking.py' error codes start with an N
161 pep8.ERRORCODE_REGEX = re.compile(r'[EWN]\d{3}')
162 add_cloud()
163 pep8.current_file = current_file
164 pep8.readlines = readlines
165 try:
166 pep8._main() # pylint: disable=W0212
167 finally:
168 if len(_missingImport) > 0:
169 print >> sys.stderr, ("%i imports missing in this test environment"
170 % len(_missingImport))
1710
=== removed file 'tools/run-pep8'
--- tools/run-pep8 2014-01-24 19:47:28 +0000
+++ tools/run-pep8 1970-01-01 00:00:00 +0000
@@ -1,38 +0,0 @@
1#!/bin/bash
2
3if [ $# -eq 0 ]; then
4 files=( bin/cloud-init $(find * -name "*.py" -type f) )
5else
6 files=( "$@" );
7fi
8
9if [ -f 'hacking.py' ]
10then
11 base=`pwd`
12else
13 base=`pwd`/tools/
14fi
15
16IGNORE="E501" # Line too long (these are caught by pylint)
17
18# King Arthur: Be quiet! ... Be Quiet! I Order You to Be Quiet.
19IGNORE="$IGNORE,E121" # Continuation line indentation is not a multiple of four
20IGNORE="$IGNORE,E123" # Closing bracket does not match indentation of opening bracket's line
21IGNORE="$IGNORE,E124" # Closing bracket missing visual indentation
22IGNORE="$IGNORE,E125" # Continuation line does not distinguish itself from next logical line
23IGNORE="$IGNORE,E126" # Continuation line over-indented for hanging indent
24IGNORE="$IGNORE,E127" # Continuation line over-indented for visual indent
25IGNORE="$IGNORE,E128" # Continuation line under-indented for visual indent
26IGNORE="$IGNORE,E502" # The backslash is redundant between brackets
27
28cmd=(
29 ${base}/hacking.py
30
31 --ignore="$IGNORE"
32
33 "${files[@]}"
34)
35
36echo -e "\nRunning 'cloudinit' pep8:"
37echo "${cmd[@]}"
38"${cmd[@]}"
390
=== removed file 'tools/run-pylint'
--- tools/run-pylint 2014-01-24 20:28:06 +0000
+++ tools/run-pylint 1970-01-01 00:00:00 +0000
@@ -1,26 +0,0 @@
1#!/bin/bash
2
3if [ $# -eq 0 ]; then
4 files=( bin/cloud-init $(find * -name "*.py" -type f) )
5else
6 files=( "$@" );
7fi
8
9RC_FILE="pylintrc"
10if [ ! -f $RC_FILE ]; then
11 RC_FILE="../pylintrc"
12fi
13
14cmd=(
15 pylint
16 --rcfile=$RC_FILE
17 --disable=R
18 --disable=I
19 --dummy-variables-rgx="_"
20 "${files[@]}"
21)
22
23echo -e "\nRunning pylint:"
24echo "${cmd[@]}"
25"${cmd[@]}"
26
270
=== added file 'tox.ini'
--- tox.ini 1970-01-01 00:00:00 +0000
+++ tox.ini 2014-01-26 01:03:26 +0000
@@ -0,0 +1,49 @@
1[tox]
2minversion = 1.6
3skipsdist = True
4envlist = py26,py27,py33
5
6[testenv]
7usedevelop = True
8setenv = VIRTUAL_ENV={envdir}
9 LANG=en_US.UTF-8
10 LANGUAGE=en_US:en
11 LC_ALL=C
12deps = -r{toxinidir}/requirements.txt
13 -r{toxinidir}/test-requirements.txt
14commands = nosetests {posargs}
15install_command = pip install {opts} {packages}
16
17[testenv:venv]
18commands = {posargs}
19
20[testenv:flake8]
21commands = flake8 {posargs}
22
23[flake8]
24# See: http://pylint-messages.wikidot.com/all-codes
25# W0142: *args and **kwargs are fine.
26# W0511: TODOs in code comments are fine.
27# W0702: No exception type(s) specified
28# W0703: Catch "Exception"
29# C0103: Invalid name
30# C0111: Missing docstring
31# F841: local variable ABC is assigned to but never used
32# E121: Continuation line indentation is not a multiple of four
33# E123: Closing bracket does not match indentation of opening bracket's line
34# E124: Closing bracket missing visual indentation
35# E125: Continuation line does not distinguish itself from next logical line
36# E126: Continuation line over-indented for hanging indent
37# E127: Continuation line over-indented for visual indent
38# E128: Continuation line under-indented for visual indent
39# E502: The backslash is redundant between brackets
40# H201: no 'except:' at least use 'except Exception:'
41# H232: Python 3.x incompatible octal X should be written as Y
42# H233: Python 3.x incompatible use of print operator
43# H234: assertEquals is deprecated, use assertEqual
44# H301: one import per line
45# H302: import only modules.
46# H404: multi line docstring should start without a leading new line
47ignore = F841,E121,E123,E124,E125,E126,E127,E128,E502,H234,H201,H301,H404,H302,H232,H233,W0142,W0511,W0702,W0703,C0103,C0111
48exclude = .venv,.tox,dist,doc,*egg,.bzr,build,tools
49max-line-length = 79