Merge lp:~harlowja/cloud-init/tox-venvs into lp:~cloud-init-dev/cloud-init/trunk

Proposed by Joshua Harlow on 2014-01-26
Status: Merged
Merge reported by: Scott Moser
Merged at revision: not available
Proposed branch: lp:~harlowja/cloud-init/tox-venvs
Merge into: lp:~cloud-init-dev/cloud-init/trunk
Diff against target: 431 lines (+64/-284)
8 files modified
HACKING.rst (+2/-2)
Makefile (+12/-26)
pylintrc (+0/-19)
test-requirements.txt (+1/-3)
tools/hacking.py (+0/-170)
tools/run-pep8 (+0/-38)
tools/run-pylint (+0/-26)
tox.ini (+49/-0)
To merge this branch: bzr merge lp:~harlowja/cloud-init/tox-venvs
Reviewer Review Type Date Requested Status
Server Team CI bot continuous-integration Needs Fixing on 2016-07-22
cloud-init Commiters 2014-01-26 Pending
Review via email: mp+203225@code.launchpad.net

Description of the change

Use a venv for testing and for lint checking

To post a comment you must log in.
Scott Moser (smoser) wrote :

Thanks. this is really nice.

need to be able to run 'make test' without tox. Ie, similar to in openstack using only packaged versions. This is important to be able to run tests on build (which won't have access to stuff other than the archive).

you replaced pep8 and pylint with 'flake8'. I think I'm ok with that, but we have lots of '# pylint:' things throughout the code. if we're dropping that we might as well clean them out. I think thats fine as a separate commit though.

Joshua Harlow (harlowja) wrote :

Makes sense, I understand the need for both, will make a few tweaks to do that.

review: Needs Fixing (continuous-integration)
Scott Moser (smoser) wrote :

marking this merged.
we use tox in trunk now.
if you think there is something missing, please propose a merge to cloud-init git.
thanks.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'HACKING.rst'
2--- HACKING.rst 2013-02-05 07:10:36 +0000
3+++ HACKING.rst 2014-01-26 01:03:26 +0000
4@@ -19,9 +19,9 @@
5
6 - ``bzr commit``
7
8-* Check pylint and pep8 and test, and address any issues:
9+* Check flake8 and test, and address any issues:
10
11- - ``make test pylint pep8``
12+ - ``make test flake8``
13
14 * Push to launchpad to a personal branch:
15
16
17=== modified file 'Makefile'
18--- Makefile 2014-01-20 18:50:04 +0000
19+++ Makefile 2014-01-26 01:03:26 +0000
20@@ -2,40 +2,27 @@
21 PY_FILES=$(shell find cloudinit bin tests tools -name "*.py" -type f )
22 PY_FILES+="bin/cloud-init"
23
24+PY_ENV=$(shell python -c 'import sys; print("py%s%s" % (sys.version_info[0:2]))')
25+
26 YAML_FILES=$(shell find cloudinit bin tests tools -name "*.yaml" -type f )
27 YAML_FILES+=$(shell find doc/examples -name "cloud-config*.txt" -type f )
28
29 CHANGELOG_VERSION=$(shell $(CWD)/tools/read-version)
30 CODE_VERSION=$(shell python -c "from cloudinit import version; print version.version_string()")
31
32-PIP_INSTALL := pip install
33-
34 ifeq ($(distro),)
35 distro = redhat
36 endif
37
38 all: test check_version
39
40-pep8:
41- @$(CWD)/tools/run-pep8 $(PY_FILES)
42-
43-pylint:
44- @$(CWD)/tools/run-pylint $(PY_FILES)
45-
46-pyflakes:
47- pyflakes $(PY_FILES)
48-
49-pip-requirements:
50- @echo "Installing cloud-init dependencies..."
51- $(PIP_INSTALL) -r "$@.txt" -q
52-
53-pip-test-requirements:
54- @echo "Installing cloud-init test dependencies..."
55- $(PIP_INSTALL) -r "$@.txt" -q
56-
57-test: clean_pyc
58- @echo "Running tests..."
59- @nosetests $(noseopts) tests/
60+flake8:
61+ @echo "Running flake8 using tox..."
62+ @tox -e flake8 $(PY_FILES)
63+
64+test:
65+ @echo "Running tests for env=$(PY_ENV) using tox..."
66+ @tox -e $(PY_ENV)
67
68 check_version:
69 @if [ "$(CHANGELOG_VERSION)" != "$(CODE_VERSION)" ]; then \
70@@ -44,13 +31,13 @@
71 else true; fi
72
73 clean_pyc:
74- @find . -type f -name "*.pyc" -delete
75+ find . -type f -name "*.pyc" -delete
76
77 2to3:
78 2to3 $(PY_FILES)
79
80 clean: clean_pyc
81- rm -rf /var/log/cloud-init.log /var/lib/cloud/
82+ rm -rf /var/log/cloud-init.log /var/lib/cloud/ $(CWD)/*.egg-info $(CWD)/.tox
83
84 yaml:
85 @$(CWD)/tools/validate-yaml.py $(YAML_FILES)
86@@ -61,5 +48,4 @@
87 deb:
88 ./packages/bddeb
89
90-.PHONY: test pylint pyflakes 2to3 clean pep8 rpm deb yaml check_version
91-.PHONY: pip-test-requirements pip-requirements clean_pyc
92+.PHONY: test flake8 2to3 clean rpm deb yaml check_version clean_pyc
93
94=== removed file 'pylintrc'
95--- pylintrc 2012-10-28 02:25:48 +0000
96+++ pylintrc 1970-01-01 00:00:00 +0000
97@@ -1,19 +0,0 @@
98-[General]
99-init-hook='import sys; sys.path.append("tests/")'
100-
101-[MESSAGES CONTROL]
102-# See: http://pylint-messages.wikidot.com/all-codes
103-# W0142: *args and **kwargs are fine.
104-# W0511: TODOs in code comments are fine.
105-# W0702: No exception type(s) specified
106-# W0703: Catch "Exception"
107-# C0103: Invalid name
108-# C0111: Missing docstring
109-disable=W0142,W0511,W0702,W0703,C0103,C0111
110-
111-[REPORTS]
112-reports=no
113-include-ids=yes
114-
115-[FORMAT]
116-max-line-length=79
117
118=== modified file 'test-requirements.txt'
119--- test-requirements.txt 2014-01-18 07:46:19 +0000
120+++ test-requirements.txt 2014-01-26 01:03:26 +0000
121@@ -1,6 +1,4 @@
122 httpretty>=0.7.1
123 mocker
124 nose
125-pep8
126-pyflakes
127-pylint
128+hacking>=0.8.0,<0.9
129
130=== removed file 'tools/hacking.py'
131--- tools/hacking.py 2012-11-11 03:32:49 +0000
132+++ tools/hacking.py 1970-01-01 00:00:00 +0000
133@@ -1,170 +0,0 @@
134-#!/usr/bin/env python
135-# vim: tabstop=4 shiftwidth=4 softtabstop=4
136-
137-# Copyright (c) 2012, Cloudscaling
138-# All Rights Reserved.
139-#
140-# Licensed under the Apache License, Version 2.0 (the "License"); you may
141-# not use this file except in compliance with the License. You may obtain
142-# a copy of the License at
143-#
144-# http://www.apache.org/licenses/LICENSE-2.0
145-#
146-# Unless required by applicable law or agreed to in writing, software
147-# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
148-# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
149-# License for the specific language governing permissions and limitations
150-# under the License.
151-
152-"""cloudinit HACKING file compliance testing (based off of nova hacking.py)
153-
154-built on top of pep8.py
155-"""
156-
157-import inspect
158-import logging
159-import re
160-import sys
161-
162-import pep8
163-
164-# Don't need this for testing
165-logging.disable('LOG')
166-
167-# N1xx comments
168-# N2xx except
169-# N3xx imports
170-# N4xx docstrings
171-# N[5-9]XX (future use)
172-
173-DOCSTRING_TRIPLE = ['"""', "'''"]
174-VERBOSE_MISSING_IMPORT = False
175-_missingImport = set([])
176-
177-
178-def import_normalize(line):
179- # convert "from x import y" to "import x.y"
180- # handle "from x import y as z" to "import x.y as z"
181- split_line = line.split()
182- if (line.startswith("from ") and "," not in line and
183- split_line[2] == "import" and split_line[3] != "*" and
184- split_line[1] != "__future__" and
185- (len(split_line) == 4 or
186- (len(split_line) == 6 and split_line[4] == "as"))):
187- return "import %s.%s" % (split_line[1], split_line[3])
188- else:
189- return line
190-
191-
192-def cloud_import_alphabetical(physical_line, line_number, lines):
193- """Check for imports in alphabetical order.
194-
195- HACKING guide recommendation for imports:
196- imports in human alphabetical order
197- N306
198- """
199- # handle import x
200- # use .lower since capitalization shouldn't dictate order
201- split_line = import_normalize(physical_line.strip()).lower().split()
202- split_previous = import_normalize(lines[line_number - 2])
203- split_previous = split_previous.strip().lower().split()
204- # with or without "as y"
205- length = [2, 4]
206- if (len(split_line) in length and len(split_previous) in length and
207- split_line[0] == "import" and split_previous[0] == "import"):
208- if split_line[1] < split_previous[1]:
209- return (0, "N306: imports not in alphabetical order (%s, %s)"
210- % (split_previous[1], split_line[1]))
211-
212-
213-def cloud_docstring_start_space(physical_line):
214- """Check for docstring not start with space.
215-
216- HACKING guide recommendation for docstring:
217- Docstring should not start with space
218- N401
219- """
220- pos = max([physical_line.find(i) for i in DOCSTRING_TRIPLE]) # start
221- if (pos != -1 and len(physical_line) > pos + 1):
222- if (physical_line[pos + 3] == ' '):
223- return (pos, "N401: one line docstring should not start with"
224- " a space")
225-
226-
227-def cloud_todo_format(physical_line):
228- """Check for 'TODO()'.
229-
230- HACKING guide recommendation for TODO:
231- Include your name with TODOs as in "#TODO(termie)"
232- N101
233- """
234- pos = physical_line.find('TODO')
235- pos1 = physical_line.find('TODO(')
236- pos2 = physical_line.find('#') # make sure it's a comment
237- if (pos != pos1 and pos2 >= 0 and pos2 < pos):
238- return pos, "N101: Use TODO(NAME)"
239-
240-
241-def cloud_docstring_one_line(physical_line):
242- """Check one line docstring end.
243-
244- HACKING guide recommendation for one line docstring:
245- A one line docstring looks like this and ends in a period.
246- N402
247- """
248- pos = max([physical_line.find(i) for i in DOCSTRING_TRIPLE]) # start
249- end = max([physical_line[-4:-1] == i for i in DOCSTRING_TRIPLE]) # end
250- if (pos != -1 and end and len(physical_line) > pos + 4):
251- if (physical_line[-5] != '.'):
252- return pos, "N402: one line docstring needs a period"
253-
254-
255-def cloud_docstring_multiline_end(physical_line):
256- """Check multi line docstring end.
257-
258- HACKING guide recommendation for docstring:
259- Docstring should end on a new line
260- N403
261- """
262- pos = max([physical_line.find(i) for i in DOCSTRING_TRIPLE]) # start
263- if (pos != -1 and len(physical_line) == pos):
264- print physical_line
265- if (physical_line[pos + 3] == ' '):
266- return (pos, "N403: multi line docstring end on new line")
267-
268-
269-current_file = ""
270-
271-
272-def readlines(filename):
273- """Record the current file being tested."""
274- pep8.current_file = filename
275- return open(filename).readlines()
276-
277-
278-def add_cloud():
279- """Monkey patch pep8 for cloud-init guidelines.
280-
281- Look for functions that start with cloud_
282- and add them to pep8 module.
283-
284- Assumes you know how to write pep8.py checks
285- """
286- for name, function in globals().items():
287- if not inspect.isfunction(function):
288- continue
289- if name.startswith("cloud_"):
290- exec("pep8.%s = %s" % (name, name)) # pylint: disable=W0122
291-
292-if __name__ == "__main__":
293- # NOVA based 'hacking.py' error codes start with an N
294- pep8.ERRORCODE_REGEX = re.compile(r'[EWN]\d{3}')
295- add_cloud()
296- pep8.current_file = current_file
297- pep8.readlines = readlines
298- try:
299- pep8._main() # pylint: disable=W0212
300- finally:
301- if len(_missingImport) > 0:
302- print >> sys.stderr, ("%i imports missing in this test environment"
303- % len(_missingImport))
304
305=== removed file 'tools/run-pep8'
306--- tools/run-pep8 2014-01-24 19:47:28 +0000
307+++ tools/run-pep8 1970-01-01 00:00:00 +0000
308@@ -1,38 +0,0 @@
309-#!/bin/bash
310-
311-if [ $# -eq 0 ]; then
312- files=( bin/cloud-init $(find * -name "*.py" -type f) )
313-else
314- files=( "$@" );
315-fi
316-
317-if [ -f 'hacking.py' ]
318-then
319- base=`pwd`
320-else
321- base=`pwd`/tools/
322-fi
323-
324-IGNORE="E501" # Line too long (these are caught by pylint)
325-
326-# King Arthur: Be quiet! ... Be Quiet! I Order You to Be Quiet.
327-IGNORE="$IGNORE,E121" # Continuation line indentation is not a multiple of four
328-IGNORE="$IGNORE,E123" # Closing bracket does not match indentation of opening bracket's line
329-IGNORE="$IGNORE,E124" # Closing bracket missing visual indentation
330-IGNORE="$IGNORE,E125" # Continuation line does not distinguish itself from next logical line
331-IGNORE="$IGNORE,E126" # Continuation line over-indented for hanging indent
332-IGNORE="$IGNORE,E127" # Continuation line over-indented for visual indent
333-IGNORE="$IGNORE,E128" # Continuation line under-indented for visual indent
334-IGNORE="$IGNORE,E502" # The backslash is redundant between brackets
335-
336-cmd=(
337- ${base}/hacking.py
338-
339- --ignore="$IGNORE"
340-
341- "${files[@]}"
342-)
343-
344-echo -e "\nRunning 'cloudinit' pep8:"
345-echo "${cmd[@]}"
346-"${cmd[@]}"
347
348=== removed file 'tools/run-pylint'
349--- tools/run-pylint 2014-01-24 20:28:06 +0000
350+++ tools/run-pylint 1970-01-01 00:00:00 +0000
351@@ -1,26 +0,0 @@
352-#!/bin/bash
353-
354-if [ $# -eq 0 ]; then
355- files=( bin/cloud-init $(find * -name "*.py" -type f) )
356-else
357- files=( "$@" );
358-fi
359-
360-RC_FILE="pylintrc"
361-if [ ! -f $RC_FILE ]; then
362- RC_FILE="../pylintrc"
363-fi
364-
365-cmd=(
366- pylint
367- --rcfile=$RC_FILE
368- --disable=R
369- --disable=I
370- --dummy-variables-rgx="_"
371- "${files[@]}"
372-)
373-
374-echo -e "\nRunning pylint:"
375-echo "${cmd[@]}"
376-"${cmd[@]}"
377-
378
379=== added file 'tox.ini'
380--- tox.ini 1970-01-01 00:00:00 +0000
381+++ tox.ini 2014-01-26 01:03:26 +0000
382@@ -0,0 +1,49 @@
383+[tox]
384+minversion = 1.6
385+skipsdist = True
386+envlist = py26,py27,py33
387+
388+[testenv]
389+usedevelop = True
390+setenv = VIRTUAL_ENV={envdir}
391+ LANG=en_US.UTF-8
392+ LANGUAGE=en_US:en
393+ LC_ALL=C
394+deps = -r{toxinidir}/requirements.txt
395+ -r{toxinidir}/test-requirements.txt
396+commands = nosetests {posargs}
397+install_command = pip install {opts} {packages}
398+
399+[testenv:venv]
400+commands = {posargs}
401+
402+[testenv:flake8]
403+commands = flake8 {posargs}
404+
405+[flake8]
406+# See: http://pylint-messages.wikidot.com/all-codes
407+# W0142: *args and **kwargs are fine.
408+# W0511: TODOs in code comments are fine.
409+# W0702: No exception type(s) specified
410+# W0703: Catch "Exception"
411+# C0103: Invalid name
412+# C0111: Missing docstring
413+# F841: local variable ABC is assigned to but never used
414+# E121: Continuation line indentation is not a multiple of four
415+# E123: Closing bracket does not match indentation of opening bracket's line
416+# E124: Closing bracket missing visual indentation
417+# E125: Continuation line does not distinguish itself from next logical line
418+# E126: Continuation line over-indented for hanging indent
419+# E127: Continuation line over-indented for visual indent
420+# E128: Continuation line under-indented for visual indent
421+# E502: The backslash is redundant between brackets
422+# H201: no 'except:' at least use 'except Exception:'
423+# H232: Python 3.x incompatible octal X should be written as Y
424+# H233: Python 3.x incompatible use of print operator
425+# H234: assertEquals is deprecated, use assertEqual
426+# H301: one import per line
427+# H302: import only modules.
428+# H404: multi line docstring should start without a leading new line
429+ignore = F841,E121,E123,E124,E125,E126,E127,E128,E502,H234,H201,H301,H404,H302,H232,H233,W0142,W0511,W0702,W0703,C0103,C0111
430+exclude = .venv,.tox,dist,doc,*egg,.bzr,build,tools
431+max-line-length = 79