Python PGBouncer

Merge lp:~cjwatson/python-pgbouncer/tox into lp:python-pgbouncer

  1. tox
  2. Merge into trunk
Proposed by Colin Watson
Status: Merged
Merged at revision: 14
Proposed branch: lp:~cjwatson/python-pgbouncer/tox
Merge into: lp:python-pgbouncer
Diff against target: 538 lines (+54/-373)
9 files modified
.bzrignore (+2/-7)
NEWS.txt (+6/-0)
README (+13/-20)
bootstrap.py (+0/-259)
buildout.cfg (+0/-35)
pgbouncer/tests.py (+20/-34)
setup.py (+5/-3)
tox.ini (+8/-0)
versions.cfg (+0/-15)
To merge this branch: bzr merge lp:~cjwatson/python-pgbouncer/tox
Reviewer Review Type Date Requested Status
William Grant code Approve
Review via email: mp+368549@code.launchpad.net

Commit message

Use postgresfixture to test against multiple PostgreSQL versions, and switch to tox.

Description of the change

Regarding postgresfixture, I sacrificed the clever inter-test resource management as part of this work. It might be possible to make it work again, but the test suite isn't all that slow, and this at least seems pretty robust.

Regarding tox, I generally can't get buildout working these days, not least because of bad interactions with pbr, and so have been progressively converting packages to tox as I touch them. This also makes it easier to set up test matrixes that test against things like multiple Python versions.

These should really be two separate MPs, but I couldn't get tox working without doing something about the PostgreSQL test setup, and I couldn't get tests running at all without switching to tox, so they ended up a bit intertwined. Hopefully this isn't too hard to follow.

To post a comment you must log in.
Revision history for this message
William Grant (wgrant) :
review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file '.bzrignore'
--- .bzrignore 2011-10-25 20:03:51 +0000
+++ .bzrignore 2019-06-07 15:00:45 +0000
@@ -1,10 +1,5 @@
1./eggs/*1./.tox
2./.installed.cfg2./MANIFEST
3./develop-eggs
4./bin
5./pgbouncer.egg-info3./pgbouncer.egg-info
6./parts
7./eggs
8./download-cache
9TAGS4TAGS
10tags5tags
116
=== modified file 'NEWS.txt'
--- NEWS.txt 2012-08-20 16:39:41 +0000
+++ NEWS.txt 2019-06-07 15:00:45 +0000
@@ -1,3 +1,9 @@
10.0.9
2=====
3
4- Use postgresfixture to test against multiple PostgreSQL versions.
5- Switch to tox.
6
10.0.8 (2012-08-20)70.0.8 (2012-08-20)
2==================8==================
39
410
=== modified file 'README'
--- README 2011-10-25 20:03:29 +0000
+++ README 2019-06-07 15:00:45 +0000
@@ -28,22 +28,22 @@
28* pgbouncer28* pgbouncer
2929
30* python-fixtures (https://launchpad.net/python-fixtures or30* python-fixtures (https://launchpad.net/python-fixtures or
31 http://pypi.python.org/pypi/fixtures)31 https://pypi.org/project/fixtures)
3232
33* testtools (http://pypi.python.org/pypi/testtools)33* testtools (https://pypi.org/project/testtools)
3434
35Testing Dependencies35Testing Dependencies
36====================36====================
3737
38In addition to the above, the tests also depend on:38In addition to the above, the tests also depend on:
3939
40* psycopg2 (http://pypi.python.org/pypi/psycopg2)40* postgresfixture (https://pypi.org/project/postgresfixture)
4141
42* subunit (http://pypi.python.org/pypi/python-subunit) (optional)42* psycopg2 (https://pypi.org/project/psycopg2)
4343
44* testresources (http://pypi.python.org/pypi/testresources)44* subunit (https://pypi.org/project/python-subunit) (optional)
4545
46* van.pg (http://pypi.python.org/pypi/van.pg)46* testscenarios (https://pypi.org/project/testscenarios)
4747
48Usage48Usage
49=====49=====
@@ -77,14 +77,7 @@
77===========77===========
7878
79Upstream development takes place at https://launchpad.net/python-pgbouncer.79Upstream development takes place at https://launchpad.net/python-pgbouncer.
80To setup a working area for development, if the dependencies are not80
81immediately available, you can use ./bootstrap.py to create bin/buildout, then81To run the tests, run:
82bin/py to get a python interpreter with the dependencies available.82
8383 $ tox
84To run the tests, run either:
85
86 $ bin/py pgbouncer/tests.py
87
88or:
89
90 $ bin/py -m testtools.run pgbouncer.tests.test_suite
9184
=== removed file 'bootstrap.py'
--- bootstrap.py 2011-07-18 03:31:27 +0000
+++ bootstrap.py 1970-01-01 00:00:00 +0000
@@ -1,259 +0,0 @@
1#!/usr/bin/env python
2##############################################################################
3#
4# Copyright (c) 2006 Zope Foundation and Contributors.
5# All Rights Reserved.
6#
7# This software is subject to the provisions of the Zope Public License,
8# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
9# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
10# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
11# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
12# FOR A PARTICULAR PURPOSE.
13#
14##############################################################################
15"""Bootstrap a buildout-based project
16
17Simply run this script in a directory containing a buildout.cfg.
18The script accepts buildout command-line options, so you can
19use the -c option to specify an alternate configuration file.
20"""
21
22import os, shutil, sys, tempfile, textwrap, urllib, urllib2, subprocess
23from optparse import OptionParser
24
25if sys.platform == 'win32':
26 def quote(c):
27 if ' ' in c:
28 return '"%s"' % c # work around spawn lamosity on windows
29 else:
30 return c
31else:
32 quote = str
33
34# See zc.buildout.easy_install._has_broken_dash_S for motivation and comments.
35stdout, stderr = subprocess.Popen(
36 [sys.executable, '-Sc',
37 'try:\n'
38 ' import ConfigParser\n'
39 'except ImportError:\n'
40 ' print 1\n'
41 'else:\n'
42 ' print 0\n'],
43 stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
44has_broken_dash_S = bool(int(stdout.strip()))
45
46# In order to be more robust in the face of system Pythons, we want to
47# run without site-packages loaded. This is somewhat tricky, in
48# particular because Python 2.6's distutils imports site, so starting
49# with the -S flag is not sufficient. However, we'll start with that:
50if not has_broken_dash_S and 'site' in sys.modules:
51 # We will restart with python -S.
52 args = sys.argv[:]
53 args[0:0] = [sys.executable, '-S']
54 args = map(quote, args)
55 os.execv(sys.executable, args)
56# Now we are running with -S. We'll get the clean sys.path, import site
57# because distutils will do it later, and then reset the path and clean
58# out any namespace packages from site-packages that might have been
59# loaded by .pth files.
60clean_path = sys.path[:]
61import site
62sys.path[:] = clean_path
63for k, v in sys.modules.items():
64 if (hasattr(v, '__path__') and
65 len(v.__path__)==1 and
66 not os.path.exists(os.path.join(v.__path__[0],'__init__.py'))):
67 # This is a namespace package. Remove it.
68 sys.modules.pop(k)
69
70is_jython = sys.platform.startswith('java')
71
72setuptools_source = 'http://peak.telecommunity.com/dist/ez_setup.py'
73distribute_source = 'http://python-distribute.org/distribute_setup.py'
74
75# parsing arguments
76def normalize_to_url(option, opt_str, value, parser):
77 if value:
78 if '://' not in value: # It doesn't smell like a URL.
79 value = 'file://%s' % (
80 urllib.pathname2url(
81 os.path.abspath(os.path.expanduser(value))),)
82 if opt_str == '--download-base' and not value.endswith('/'):
83 # Download base needs a trailing slash to make the world happy.
84 value += '/'
85 else:
86 value = None
87 name = opt_str[2:].replace('-', '_')
88 setattr(parser.values, name, value)
89
90usage = '''\
91[DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options]
92
93Bootstraps a buildout-based project.
94
95Simply run this script in a directory containing a buildout.cfg, using the
96Python that you want bin/buildout to use.
97
98Note that by using --setup-source and --download-base to point to
99local resources, you can keep this script from going over the network.
100'''
101
102parser = OptionParser(usage=usage)
103parser.add_option("-v", "--version", dest="version",
104 help="use a specific zc.buildout version")
105parser.add_option("-d", "--distribute",
106 action="store_true", dest="use_distribute", default=False,
107 help="Use Distribute rather than Setuptools.")
108parser.add_option("--setup-source", action="callback", dest="setup_source",
109 callback=normalize_to_url, nargs=1, type="string",
110 help=("Specify a URL or file location for the setup file. "
111 "If you use Setuptools, this will default to " +
112 setuptools_source + "; if you use Distribute, this "
113 "will default to " + distribute_source +"."))
114parser.add_option("--download-base", action="callback", dest="download_base",
115 callback=normalize_to_url, nargs=1, type="string",
116 help=("Specify a URL or directory for downloading "
117 "zc.buildout and either Setuptools or Distribute. "
118 "Defaults to PyPI."))
119parser.add_option("--eggs",
120 help=("Specify a directory for storing eggs. Defaults to "
121 "a temporary directory that is deleted when the "
122 "bootstrap script completes."))
123parser.add_option("-t", "--accept-buildout-test-releases",
124 dest='accept_buildout_test_releases',
125 action="store_true", default=False,
126 help=("Normally, if you do not specify a --version, the "
127 "bootstrap script and buildout gets the newest "
128 "*final* versions of zc.buildout and its recipes and "
129 "extensions for you. If you use this flag, "
130 "bootstrap and buildout will get the newest releases "
131 "even if they are alphas or betas."))
132parser.add_option("-c", None, action="store", dest="config_file",
133 help=("Specify the path to the buildout configuration "
134 "file to be used."))
135
136options, args = parser.parse_args()
137
138# if -c was provided, we push it back into args for buildout's main function
139if options.config_file is not None:
140 args += ['-c', options.config_file]
141
142if options.eggs:
143 eggs_dir = os.path.abspath(os.path.expanduser(options.eggs))
144else:
145 eggs_dir = tempfile.mkdtemp()
146
147if options.setup_source is None:
148 if options.use_distribute:
149 options.setup_source = distribute_source
150 else:
151 options.setup_source = setuptools_source
152
153if options.accept_buildout_test_releases:
154 args.append('buildout:accept-buildout-test-releases=true')
155args.append('bootstrap')
156
157try:
158 import pkg_resources
159 import setuptools # A flag. Sometimes pkg_resources is installed alone.
160 if not hasattr(pkg_resources, '_distribute'):
161 raise ImportError
162except ImportError:
163 ez_code = urllib2.urlopen(
164 options.setup_source).read().replace('\r\n', '\n')
165 ez = {}
166 exec ez_code in ez
167 setup_args = dict(to_dir=eggs_dir, download_delay=0)
168 if options.download_base:
169 setup_args['download_base'] = options.download_base
170 if options.use_distribute:
171 setup_args['no_fake'] = True
172 ez['use_setuptools'](**setup_args)
173 reload(sys.modules['pkg_resources'])
174 import pkg_resources
175 # This does not (always?) update the default working set. We will
176 # do it.
177 for path in sys.path:
178 if path not in pkg_resources.working_set.entries:
179 pkg_resources.working_set.add_entry(path)
180
181cmd = [quote(sys.executable),
182 '-c',
183 quote('from setuptools.command.easy_install import main; main()'),
184 '-mqNxd',
185 quote(eggs_dir)]
186
187if not has_broken_dash_S:
188 cmd.insert(1, '-S')
189
190find_links = options.download_base
191if not find_links:
192 find_links = os.environ.get('bootstrap-testing-find-links')
193if find_links:
194 cmd.extend(['-f', quote(find_links)])
195
196if options.use_distribute:
197 setup_requirement = 'distribute'
198else:
199 setup_requirement = 'setuptools'
200ws = pkg_resources.working_set
201setup_requirement_path = ws.find(
202 pkg_resources.Requirement.parse(setup_requirement)).location
203env = dict(
204 os.environ,
205 PYTHONPATH=setup_requirement_path)
206
207requirement = 'zc.buildout'
208version = options.version
209if version is None and not options.accept_buildout_test_releases:
210 # Figure out the most recent final version of zc.buildout.
211 import setuptools.package_index
212 _final_parts = '*final-', '*final'
213 def _final_version(parsed_version):
214 for part in parsed_version:
215 if (part[:1] == '*') and (part not in _final_parts):
216 return False
217 return True
218 index = setuptools.package_index.PackageIndex(
219 search_path=[setup_requirement_path])
220 if find_links:
221 index.add_find_links((find_links,))
222 req = pkg_resources.Requirement.parse(requirement)
223 if index.obtain(req) is not None:
224 best = []
225 bestv = None
226 for dist in index[req.project_name]:
227 distv = dist.parsed_version
228 if _final_version(distv):
229 if bestv is None or distv > bestv:
230 best = [dist]
231 bestv = distv
232 elif distv == bestv:
233 best.append(dist)
234 if best:
235 best.sort()
236 version = best[-1].version
237if version:
238 requirement = '=='.join((requirement, version))
239cmd.append(requirement)
240
241if is_jython:
242 import subprocess
243 exitcode = subprocess.Popen(cmd, env=env).wait()
244else: # Windows prefers this, apparently; otherwise we would prefer subprocess
245 exitcode = os.spawnle(*([os.P_WAIT, sys.executable] + cmd + [env]))
246if exitcode != 0:
247 sys.stdout.flush()
248 sys.stderr.flush()
249 print ("An error occurred when trying to install zc.buildout. "
250 "Look above this message for any errors that "
251 "were output by easy_install.")
252 sys.exit(exitcode)
253
254ws.add_entry(eggs_dir)
255ws.require(requirement)
256import zc.buildout.buildout
257zc.buildout.buildout.main(args)
258if not options.eggs: # clean up temporary egg directory
259 shutil.rmtree(eggs_dir)
2600
=== removed file 'buildout.cfg'
--- buildout.cfg 2012-08-20 16:27:22 +0000
+++ buildout.cfg 1970-01-01 00:00:00 +0000
@@ -1,35 +0,0 @@
1# Copyright 2011 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4[buildout]
5parts =
6 scripts
7unzip = true
8eggs-directory = eggs
9download-cache = download-cache
10relative-paths = true
11
12# Disable this option temporarily if you want buildout to find software
13# dependencies *other* than those in our download-cache. Once you have the
14# desired software, reenable this option (and check in the new software to
15# lp:lp-source-dependencies if this is going to be reviewed/merged/deployed.)
16install-from-cache = true
17
18# This also will need to be temporarily disabled or changed for package
19# upgrades. Newly-added packages should also add their desired version number
20# to versions.cfg.
21extends = versions.cfg
22
23allow-picked-versions = false
24
25prefer-final = true
26
27develop = .
28
29[scripts]
30recipe = z3c.recipe.scripts
31eggs = pgbouncer [test]
32include-site-packages = true
33allowed-eggs-from-site-packages =
34 subunit
35interpreter = py
360
=== modified file 'pgbouncer/tests.py'
--- pgbouncer/tests.py 2011-10-28 11:45:32 +0000
+++ pgbouncer/tests.py 2019-06-07 15:00:45 +0000
@@ -14,51 +14,42 @@
14# You should have received a copy of the GNU Affero General Public License14# You should have received a copy of the GNU Affero General Public License
15# along with this program. If not, see <http://www.gnu.org/licenses/>.15# along with this program. If not, see <http://www.gnu.org/licenses/>.
1616
17import os17from contextlib import closing
18import unittest
1918
20import fixtures19import fixtures
20from postgresfixture import ClusterFixture
21from postgresfixture.cluster import PG_VERSIONS
21import psycopg222import psycopg2
22from van.pg import DatabaseManager23import testscenarios
23import testresources
24import testtools24import testtools
2525
26from pgbouncer.fixture import PGBouncerFixture26from pgbouncer.fixture import PGBouncerFixture
2727
28# A 'just-works' workaround for Ubuntu not exposing initdb to the main PATH.28
29os.environ['PATH'] = os.environ['PATH'] + ':/usr/lib/postgresql/8.4/bin'29class TestFixture(testscenarios.WithScenarios, testtools.TestCase):
3030
3131 scenarios = sorted(
32def test_suite():32 (version, {'version': version}) for version in PG_VERSIONS)
33 loader = testresources.TestLoader()
34 return loader.loadTestsFromName(__name__)
35
36
37class ResourcedTestCase(testtools.TestCase, testresources.ResourcedTestCase):
38 """Mix together testtools and testresources."""
39
40
41def setup_user(db):
42 conn = psycopg2.connect(host=db.host, database=db.database)
43 conn.cursor().execute('CREATE USER user1')
44 conn.commit()
45 conn.close()
46
47
48class TestFixture(ResourcedTestCase):
49
50 resources = [('db', DatabaseManager(initialize_sql=setup_user))]
5133
52 def setUp(self):34 def setUp(self):
53 super(TestFixture, self).setUp()35 super(TestFixture, self).setUp()
36 datadir = self.useFixture(fixtures.TempDir()).path
37 self.dbname = 'test_pgbouncer'
38 self.cluster = self.useFixture(
39 ClusterFixture(datadir, version=self.version))
40 self.cluster.createdb(self.dbname)
41 with closing(self.cluster.connect()) as conn:
42 with closing(conn.cursor()) as cur:
43 cur.execute('DROP USER IF EXISTS user1')
44 cur.execute('CREATE USER user1')
54 self.bouncer = PGBouncerFixture()45 self.bouncer = PGBouncerFixture()
55 self.bouncer.databases[self.db.database] = 'host=' + self.db.host46 self.bouncer.databases[self.dbname] = 'host=' + datadir
56 self.bouncer.users['user1'] = ''47 self.bouncer.users['user1'] = ''
5748
58 def connect(self, host=None):49 def connect(self, host=None):
59 return psycopg2.connect(50 return psycopg2.connect(
60 host=(self.bouncer.host if host is None else host),51 host=(self.bouncer.host if host is None else host),
61 port=self.bouncer.port, database=self.db.database,52 port=self.bouncer.port, database=self.dbname,
62 user='user1')53 user='user1')
6354
64 def test_dynamic_port_allocation(self):55 def test_dynamic_port_allocation(self):
@@ -99,8 +90,3 @@
99 bouncer_pid = self.bouncer.process.pid90 bouncer_pid = self.bouncer.process.pid
100 self.bouncer.start()91 self.bouncer.start()
101 self.assertEqual(bouncer_pid, self.bouncer.process.pid)92 self.assertEqual(bouncer_pid, self.bouncer.process.pid)
102
103
104if __name__ == "__main__":
105 loader = testresources.TestLoader()
106 unittest.main(testLoader=loader)
10793
=== modified file 'setup.py'
--- setup.py 2012-08-20 16:39:41 +0000
+++ setup.py 2019-06-07 15:00:45 +0000
@@ -14,12 +14,13 @@
14#14#
15# You should have received a copy of the GNU Affero General Public License15# You should have received a copy of the GNU Affero General Public License
16# along with this program. If not, see <http://www.gnu.org/licenses/>.16# along with this program. If not, see <http://www.gnu.org/licenses/>.
17 17
1818
19from distutils.core import setup19from distutils.core import setup
20import os.path20import os.path
2121
22description = file(os.path.join(os.path.dirname(__file__), 'README'), 'rb').read()22with open(os.path.join(os.path.dirname(__file__), 'README')) as f:
23 description = f.read()
2324
24setup(name="pgbouncer",25setup(name="pgbouncer",
25 version="0.0.8",26 version="0.0.8",
@@ -43,8 +44,9 @@
43 ],44 ],
44 extras_require = dict(45 extras_require = dict(
45 test=[46 test=[
47 'postgresfixture',
48 'testscenarios',
46 'testtools',49 'testtools',
47 'van.pg',
48 ]50 ]
49 ),51 ),
50 )52 )
5153
=== added file 'tox.ini'
--- tox.ini 1970-01-01 00:00:00 +0000
+++ tox.ini 2019-06-07 15:00:45 +0000
@@ -0,0 +1,8 @@
1[tox]
2envlist =
3 py27
4
5[testenv]
6deps =
7 .[test]
8commands = python -m testtools.run discover -s pgbouncer {posargs}
09
=== removed file 'versions.cfg'
--- versions.cfg 2012-08-20 16:27:22 +0000
+++ versions.cfg 1970-01-01 00:00:00 +0000
@@ -1,15 +0,0 @@
1[buildout]
2versions = versions
3
4[versions]
5fixtures = 0.3.6
6psycopg2 = 2.4.5
7pytz = 2012d
8setuptools = 0.6c11
9testresources = 0.2.4
10testtools = 0.9.11
11van.pg = 1.4
12z3c.recipe.filetemplate = 2.1.0
13z3c.recipe.scripts = 1.0.1
14zc.buildout = 1.5.1
15zc.recipe.egg = 1.3.2

Subscribers

People subscribed via source and target branches

to status/vote changes: