Merge lp:~allenap/storm/oneiric-admin-shutdown-bug-871596 into lp:storm

[Gavin Panella]

This branch adds three tests for disconnection errors:

- When the connection is terminated with pg_terminate_backend.

- When the connection is terminated with pg_terminate_backend and the
  connection goes via pgbouncer.

- When the connection is terminated because pgbouncer is terminated.

It then updates is_disconnection_error to match those situations.

Additionally, under Oneiric, some disconnection tests were failing
anyway, so is_disconnection_error has been updated to catch those too.

This branch adds two optional test dependencies: python-fixtures and
python-pgbouncer. If these are not available the tests will not be
attempted.

[Stuart Bishop (stub)]

Thanks for working on this!

So the only time you ever saw the 57P01 error was with pgbouncer? That is interesting as in the past we often never saw the error codes at all (hence the string matching).

The code seems good. Just a few suggestions and points for discussion:

- Rather than check just for 57P01, I think we might as well check for the following list:

pg_connection_failure_codes = frozenset([
    '08006', # CONNECTION FAILURE
    '08001', # SQLCLIENT UNABLE TO ESTABLISH SQLCONNECTION
    '08004', # SQLSERVER REJECTED ESTABLISHMENT OF SQLCONNECTION
    '53300', # TOO MANY CONNECTIONS
    '57000', # OPERATOR INTERVENTION
    '57014', # QUERY CANCELED
    '57P01', # ADMIN SHUTDOWN
    '57P02', # CRASH SHUTDOWN
    '57P03', # CANNOT CONNECT NOW
    ])

- Should we document installing python-pgbouncer using the Python egg from pypi rather than checking out the Bazaar branch and using a symlink hack?

- Should python-fixtures be an optional dependency? Maybe if it gets used elsewhere.

- Do we want python-pgbouncer and fixtures as optional dependencies for running the PostgreSQL tests? I understand not needing the environment setup for testing every database backend, but there seems little point for allowing people to run a subset of the tests for a particular DB backend.

review: Approve

[lp:~allenap/storm/oneiric-admin-shutdown-bug-871596]

427. By Gavin Panella on 2011-10-18

Match many more pgcodes, as suggested by stub.

428. By Gavin Panella on 2011-10-18

Fix lint in setup.py.

429. By Gavin Panella on 2011-10-18

Use the test_suite and tests_require setuptools extensions to declare dependencies on pgbouncer and fixtures.

430. By Gavin Panella on 2011-10-18

Define all test dependencies and use setuptools for make check.

431. By Gavin Panella on 2011-10-18

Delete eggs in make clean.

432. By Gavin Panella on 2011-10-18

make check no longer depends on build.

433. By Gavin Panella on 2011-10-18

Also depend on timeline.

434. By Gavin Panella on 2011-10-18

Go completely over to the Dark, ahem, setuptools side.

435. By Gavin Panella on 2011-10-18

Make test use eggs installed by python setup.py test.

436. By Gavin Panella on 2011-10-18

New develop target to dry-run the tests so that dependencies are installed.

[Gavin Panella (allenap)]

> Thanks for working on this!
>
> So the only time you ever saw the 57P01 error was with pgbouncer? That is
> interesting as in the past we often never saw the error codes at all (hence
> the string matching).

Yeah, I don't fully understand what was happening there.

>
> The code seems good. Just a few suggestions and points for discussion:
>
> - Rather than check just for 57P01, I think we might as well check for the
> following list:
>
> pg_connection_failure_codes = frozenset([
> '08006', # CONNECTION FAILURE
> '08001', # SQLCLIENT UNABLE TO ESTABLISH SQLCONNECTION
> '08004', # SQLSERVER REJECTED ESTABLISHMENT OF SQLCONNECTION
> '53300', # TOO MANY CONNECTIONS
> '57000', # OPERATOR INTERVENTION
> '57014', # QUERY CANCELED
> '57P01', # ADMIN SHUTDOWN
> '57P02', # CRASH SHUTDOWN
> '57P03', # CANNOT CONNECT NOW
> ])

Tip top, done.

>
> - Should we document installing python-pgbouncer using the Python egg from
> pypi rather than checking out the Bazaar branch and using a symlink hack?
>
> - Should python-fixtures be an optional dependency? Maybe if it gets used
> elsewhere.
>
> - Do we want python-pgbouncer and fixtures as optional dependencies for
> running the PostgreSQL tests? I understand not needing the environment setup
> for testing every database backend, but there seems little point for allowing
> people to run a subset of the tests for a particular DB backend.

I've switched everything to use setuptools, and have defined all the
test dependencies in setup.py. I've switched make check over to use
setuptools' testing support, and I've also made the ./test script use
the eggs that setuptools downloads. There's also an additional develop
Makefile target that downloads the test dependencies without running
the tests.

This means you can do make check from a fresh branch of Storm and run
*all* the tests without further intervention (other than doing the
one-off package installations and database set-up). This has already
shown me that there's another Django disconnection test failure that I
need to address.

[lp:~allenap/storm/oneiric-admin-shutdown-bug-871596]

437. By Gavin Panella on 2011-10-18

develop is PHONY.

438. By Gavin Panella on 2011-10-18

If we're relying on setuptools we don't need our own find_packages().

439. By Gavin Panella on 2011-10-18

Remove _trial_temp in make clean.

440. By Gavin Panella on 2011-10-18

Document make develop in README.

[Robert Collins (lifeless)]

van.pg can do postgresql cluster setup as a test fixture, if you want :)

[Gavin Panella (allenap)]

> van.pg can do postgresql cluster setup as a test fixture, if you
> want :)

Indeed, I do want! I did some work on van.pg last week or the week
before, but it was a little more complicated than I had imagined... or
I was misunderstanding. I won't attempt it again in this branch, but I
feel like I might have a much better chance of success with the things
I've learnt doing this.

[Stuart Bishop (stub)]

On Wed, Oct 19, 2011 at 3:42 PM, Gavin Panella
<email address hidden> wrote:
>> van.pg can do postgresql cluster setup as a test fixture, if you
>> want :)
>
> Indeed, I do want! I did some work on van.pg last week or the week
> before, but it was a little more complicated than I had imagined... or
> I was misunderstanding. I won't attempt it again in this branch, but I
> feel like I might have a much better chance of success with the things
> I've learnt doing this.

Yer, separate branch for that or this will never land :)

--
Stuart Bishop <email address hidden>
http://www.stuartbishop.net/

[Gavin Panella (allenap)]

I'm going to split this branch into two, to help review.

[lp:~allenap/storm/oneiric-admin-shutdown-bug-871596]

441. By Gavin Panella on 2011-10-19

Merged go-setuptools into oneiric-admin-shutdown-bug-871596.

442. By Gavin Panella on 2011-10-19

Revert previous revision.

443. By Gavin Panella on 2011-10-25

Include ez_setup.py in tarballs.

444. By Gavin Panella on 2011-10-26

Rely upon pgbouncer 0.0.6 or later, owing to unreliable shutdown problems in earlier versions.

445. By Gavin Panella on 2011-10-26

Deal with a weird ProgrammingError when connecting via pgbouncer.

446. By Gavin Panella on 2011-10-26

QUERY CANCELED is not a disconnection.

447. By Gavin Panella on 2011-10-27

Merged go-setuptools into oneiric-admin-shutdown-bug-871596, resolving conflicts.

448. By Gavin Panella on 2011-10-28

Merged go-setuptools into oneiric-admin-shutdown-bug-871596, resolving conflicts.

449. By Gavin Panella on 2011-10-28

Merged go-setuptools into oneiric-admin-shutdown-bug-871596.

450. By Gavin Panella on 2011-10-28

Use has_fixtures from tests/__init__.py.

451. By Gavin Panella on 2011-10-28

Update NEWS.

452. By Gavin Panella on 2011-10-28

pgbouncer 0.0.7 is needed.

453. By Gavin Panella on 2011-10-28

Add eggs to path before importing tests.

454. By Gavin Panella on 2011-10-28

Depend on testtools >= 0.9.8; newer versions of python-fixtures needs it.

Unmerged revisions

=== modified file '.bzrignore'
--- .bzrignore	2009-10-08 07:38:11 +0000
+++ .bzrignore	2011-10-19 22:12:25 +0000
@@ -11,3 +11,6 @@
 debian/python-storm.substvars
 apidoc
 _trial_temp
+*.egg
+TAGS
+tags

=== modified file 'Makefile'
--- Makefile	2009-11-24 18:34:34 +0000
+++ Makefile	2011-10-19 22:12:25 +0000
@@ -1,7 +1,7 @@
 PYTHON ?= python
 PYDOCTOR ?= pydoctor
 
-TEST_COMMAND = $(PYTHON) test
+TEST_COMMAND = $(PYTHON) setup.py test
 
 STORM_POSTGRES_URI = postgres:storm_test
 STORM_POSTGRES_HOST_URI = postgres://localhost/storm_test
@@ -18,8 +18,11 @@
 build:
 	$(PYTHON) setup.py build_ext -i
 
-check: build
-	# Run the tests once with C extensions and once without them.
+develop:
+	$(TEST_COMMAND) --quiet --dry-run
+
+check:
+	@ # Run the tests once with C extensions and once without them.
 	$(TEST_COMMAND) && STORM_CEXTENSIONS=0 $(TEST_COMMAND)
 
 doc:
@@ -36,8 +39,10 @@
 	rm -rf debian/files
 	rm -rf debian/python-storm
 	rm -rf debian/python-storm.*
+	rm -rf *.egg
+	rm -rf _trial_temp
 	find . -name "*.so" -type f -exec rm -f {} \;
 	find . -name "*.pyc" -type f -exec rm -f {} \;
 	find . -name "*~" -type f -exec rm -f {} \;
 
-.PHONY: all build test
+.PHONY: all build check clean develop doc release

=== modified file 'README'
--- README	2009-10-08 07:23:43 +0000
+++ README	2011-10-19 22:12:25 +0000
@@ -98,8 +98,10 @@
 you will need to install MySQL and PostgreSQL, along with the
 related Python database drivers:
 
- $ sudo apt-get install python-mysqldb python-psycopg2 mysql-server \
-                        postgresql build-essential
+ $ sudo apt-get install \
+       python-mysqldb mysql-server \
+       postgresql pgbouncer \
+       build-essential
 
 These will take a few minutes to download (its a bit under 200MB all
 together).  Once the download is complete, a screen called
@@ -110,6 +112,28 @@
 asked to enter a password multiple times.  Leave it blank in each
 case.
 
+The Python dependencies for running tests can mostly be installed with
+apt-get:
+
+ $ apt-get install \
+       python-django python-fixtures python-psycopg2 \
+       python-testresources python-transaction python-twisted \
+       python-zope.component python-zope.security
+
+Two modules - pgbouncer and timeline - are not yet packaged in
+Ubuntu. These can be installed from PyPI:
+
+  http://pypi.python.org/pypi/pgbouncer
+  http://pypi.python.org/pypi/timeline
+
+Alternatively, dependencies can be downloaded as eggs into the current
+directory with:
+
+ $ make develop
+
+This ensures that all dependencies are available, downloading from
+PyPI as appropriate.
+
 Setting up database users and access security
 ---------------------------------------------
 

=== added file 'ez_setup.py'
--- ez_setup.py	1970-01-01 00:00:00 +0000
+++ ez_setup.py	2011-10-19 22:12:25 +0000
@@ -0,0 +1,284 @@
+#!python
+"""Bootstrap setuptools installation
+
+If you want to use setuptools in your package's setup.py, just include this
+file in the same directory with it, and add this to the top of your setup.py::
+
+    from ez_setup import use_setuptools
+    use_setuptools()
+
+If you want to require a specific version of setuptools, set a download
+mirror, or use an alternate download directory, you can do so by supplying
+the appropriate options to ``use_setuptools()``.
+
+This file can also be run as a script to install or upgrade setuptools.
+"""
+import sys
+DEFAULT_VERSION = "0.6c11"
+DEFAULT_URL     = "http://pypi.python.org/packages/%s/s/setuptools/" % sys.version[:3]
+
+md5_data = {
+    'setuptools-0.6b1-py2.3.egg': '8822caf901250d848b996b7f25c6e6ca',
+    'setuptools-0.6b1-py2.4.egg': 'b79a8a403e4502fbb85ee3f1941735cb',
+    'setuptools-0.6b2-py2.3.egg': '5657759d8a6d8fc44070a9d07272d99b',
+    'setuptools-0.6b2-py2.4.egg': '4996a8d169d2be661fa32a6e52e4f82a',
+    'setuptools-0.6b3-py2.3.egg': 'bb31c0fc7399a63579975cad9f5a0618',
+    'setuptools-0.6b3-py2.4.egg': '38a8c6b3d6ecd22247f179f7da669fac',
+    'setuptools-0.6b4-py2.3.egg': '62045a24ed4e1ebc77fe039aa4e6f7e5',
+    'setuptools-0.6b4-py2.4.egg': '4cb2a185d228dacffb2d17f103b3b1c4',
+    'setuptools-0.6c1-py2.3.egg': 'b3f2b5539d65cb7f74ad79127f1a908c',
+    'setuptools-0.6c1-py2.4.egg': 'b45adeda0667d2d2ffe14009364f2a4b',
+    'setuptools-0.6c10-py2.3.egg': 'ce1e2ab5d3a0256456d9fc13800a7090',
+    'setuptools-0.6c10-py2.4.egg': '57d6d9d6e9b80772c59a53a8433a5dd4',
+    'setuptools-0.6c10-py2.5.egg': 'de46ac8b1c97c895572e5e8596aeb8c7',
+    'setuptools-0.6c10-py2.6.egg': '58ea40aef06da02ce641495523a0b7f5',
+    'setuptools-0.6c11-py2.3.egg': '2baeac6e13d414a9d28e7ba5b5a596de',
+    'setuptools-0.6c11-py2.4.egg': 'bd639f9b0eac4c42497034dec2ec0c2b',
+    'setuptools-0.6c11-py2.5.egg': '64c94f3bf7a72a13ec83e0b24f2749b2',
+    'setuptools-0.6c11-py2.6.egg': 'bfa92100bd772d5a213eedd356d64086',
+    'setuptools-0.6c2-py2.3.egg': 'f0064bf6aa2b7d0f3ba0b43f20817c27',
+    'setuptools-0.6c2-py2.4.egg': '616192eec35f47e8ea16cd6a122b7277',
+    'setuptools-0.6c3-py2.3.egg': 'f181fa125dfe85a259c9cd6f1d7b78fa',
+    'setuptools-0.6c3-py2.4.egg': 'e0ed74682c998bfb73bf803a50e7b71e',
+    'setuptools-0.6c3-py2.5.egg': 'abef16fdd61955514841c7c6bd98965e',
+    'setuptools-0.6c4-py2.3.egg': 'b0b9131acab32022bfac7f44c5d7971f',
+    'setuptools-0.6c4-py2.4.egg': '2a1f9656d4fbf3c97bf946c0a124e6e2',
+    'setuptools-0.6c4-py2.5.egg': '8f5a052e32cdb9c72bcf4b5526f28afc',
+    'setuptools-0.6c5-py2.3.egg': 'ee9fd80965da04f2f3e6b3576e9d8167',
+    'setuptools-0.6c5-py2.4.egg': 'afe2adf1c01701ee841761f5bcd8aa64',
+    'setuptools-0.6c5-py2.5.egg': 'a8d3f61494ccaa8714dfed37bccd3d5d',
+    'setuptools-0.6c6-py2.3.egg': '35686b78116a668847237b69d549ec20',
+    'setuptools-0.6c6-py2.4.egg': '3c56af57be3225019260a644430065ab',
+    'setuptools-0.6c6-py2.5.egg': 'b2f8a7520709a5b34f80946de5f02f53',
+    'setuptools-0.6c7-py2.3.egg': '209fdf9adc3a615e5115b725658e13e2',
+    'setuptools-0.6c7-py2.4.egg': '5a8f954807d46a0fb67cf1f26c55a82e',
+    'setuptools-0.6c7-py2.5.egg': '45d2ad28f9750e7434111fde831e8372',
+    'setuptools-0.6c8-py2.3.egg': '50759d29b349db8cfd807ba8303f1902',
+    'setuptools-0.6c8-py2.4.egg': 'cba38d74f7d483c06e9daa6070cce6de',
+    'setuptools-0.6c8-py2.5.egg': '1721747ee329dc150590a58b3e1ac95b',
+    'setuptools-0.6c9-py2.3.egg': 'a83c4020414807b496e4cfbe08507c03',
+    'setuptools-0.6c9-py2.4.egg': '260a2be2e5388d66bdaee06abec6342a',
+    'setuptools-0.6c9-py2.5.egg': 'fe67c3e5a17b12c0e7c541b7ea43a8e6',
+    'setuptools-0.6c9-py2.6.egg': 'ca37b1ff16fa2ede6e19383e7b59245a',
+}
+
+import sys, os
+try: from hashlib import md5
+except ImportError: from md5 import md5
+
+def _validate_md5(egg_name, data):
+    if egg_name in md5_data:
+        digest = md5(data).hexdigest()
+        if digest != md5_data[egg_name]:
+            print >>sys.stderr, (
+                "md5 validation of %s failed!  (Possible download problem?)"
+                % egg_name
+            )
+            sys.exit(2)
+    return data
+
+def use_setuptools(
+    version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir,
+    download_delay=15
+):
+    """Automatically find/download setuptools and make it available on sys.path
+
+    `version` should be a valid setuptools version number that is available
+    as an egg for download under the `download_base` URL (which should end with
+    a '/').  `to_dir` is the directory where setuptools will be downloaded, if
+    it is not already available.  If `download_delay` is specified, it should
+    be the number of seconds that will be paused before initiating a download,
+    should one be required.  If an older version of setuptools is installed,
+    this routine will print a message to ``sys.stderr`` and raise SystemExit in
+    an attempt to abort the calling script.
+    """
+    was_imported = 'pkg_resources' in sys.modules or 'setuptools' in sys.modules
+    def do_download():
+        egg = download_setuptools(version, download_base, to_dir, download_delay)
+        sys.path.insert(0, egg)
+        import setuptools; setuptools.bootstrap_install_from = egg
+    try:
+        import pkg_resources
+    except ImportError:
+        return do_download()       
+    try:
+        pkg_resources.require("setuptools>="+version); return
+    except pkg_resources.VersionConflict, e:
+        if was_imported:
+            print >>sys.stderr, (
+            "The required version of setuptools (>=%s) is not available, and\n"
+            "can't be installed while this script is running. Please install\n"
+            " a more recent version first, using 'easy_install -U setuptools'."
+            "\n\n(Currently using %r)"
+            ) % (version, e.args[0])
+            sys.exit(2)
+    except pkg_resources.DistributionNotFound:
+        pass
+
+    del pkg_resources, sys.modules['pkg_resources']    # reload ok
+    return do_download()
+
+def download_setuptools(
+    version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir,
+    delay = 15
+):
+    """Download setuptools from a specified location and return its filename
+
+    `version` should be a valid setuptools version number that is available
+    as an egg for download under the `download_base` URL (which should end
+    with a '/'). `to_dir` is the directory where the egg will be downloaded.
+    `delay` is the number of seconds to pause before an actual download attempt.
+    """
+    import urllib2, shutil
+    egg_name = "setuptools-%s-py%s.egg" % (version,sys.version[:3])
+    url = download_base + egg_name
+    saveto = os.path.join(to_dir, egg_name)
+    src = dst = None
+    if not os.path.exists(saveto):  # Avoid repeated downloads
+        try:
+            from distutils import log
+            if delay:
+                log.warn("""
+---------------------------------------------------------------------------
+This script requires setuptools version %s to run (even to display
+help).  I will attempt to download it for you (from
+%s), but
+you may need to enable firewall access for this script first.
+I will start the download in %d seconds.
+
+(Note: if this machine does not have network access, please obtain the file
+
+   %s
+
+and place it in this directory before rerunning this script.)
+---------------------------------------------------------------------------""",
+                    version, download_base, delay, url
+                ); from time import sleep; sleep(delay)
+            log.warn("Downloading %s", url)
+            src = urllib2.urlopen(url)
+            # Read/write all in one block, so we don't create a corrupt file
+            # if the download is interrupted.
+            data = _validate_md5(egg_name, src.read())
+            dst = open(saveto,"wb"); dst.write(data)
+        finally:
+            if src: src.close()
+            if dst: dst.close()
+    return os.path.realpath(saveto)
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+def main(argv, version=DEFAULT_VERSION):
+    """Install or upgrade setuptools and EasyInstall"""
+    try:
+        import setuptools
+    except ImportError:
+        egg = None
+        try:
+            egg = download_setuptools(version, delay=0)
+            sys.path.insert(0,egg)
+            from setuptools.command.easy_install import main
+            return main(list(argv)+[egg])   # we're done here
+        finally:
+            if egg and os.path.exists(egg):
+                os.unlink(egg)
+    else:
+        if setuptools.__version__ == '0.0.1':
+            print >>sys.stderr, (
+            "You have an obsolete version of setuptools installed.  Please\n"
+            "remove it from your system entirely before rerunning this script."
+            )
+            sys.exit(2)
+
+    req = "setuptools>="+version
+    import pkg_resources
+    try:
+        pkg_resources.require(req)
+    except pkg_resources.VersionConflict:
+        try:
+            from setuptools.command.easy_install import main
+        except ImportError:
+            from easy_install import main
+        main(list(argv)+[download_setuptools(delay=0)])
+        sys.exit(0) # try to force an exit
+    else:
+        if argv:
+            from setuptools.command.easy_install import main
+            main(argv)
+        else:
+            print "Setuptools version",version,"or greater has been installed."
+            print '(Run "ez_setup.py -U setuptools" to reinstall or upgrade.)'
+
+def update_md5(filenames):
+    """Update our built-in md5 registry"""
+
+    import re
+
+    for name in filenames:
+        base = os.path.basename(name)
+        f = open(name,'rb')
+        md5_data[base] = md5(f.read()).hexdigest()
+        f.close()
+
+    data = ["    %r: %r,\n" % it for it in md5_data.items()]
+    data.sort()
+    repl = "".join(data)
+
+    import inspect
+    srcfile = inspect.getsourcefile(sys.modules[__name__])
+    f = open(srcfile, 'rb'); src = f.read(); f.close()
+
+    match = re.search("\nmd5_data = {\n([^}]+)}", src)
+    if not match:
+        print >>sys.stderr, "Internal error!"
+        sys.exit(2)
+
+    src = src[:match.start(1)] + repl + src[match.end(1):]
+    f = open(srcfile,'w')
+    f.write(src)
+    f.close()
+
+
+if __name__=='__main__':
+    if len(sys.argv)>2 and sys.argv[1]=='--md5update':
+        update_md5(sys.argv[2:])
+    else:
+        main(sys.argv[1:])
+
+
+
+
+
+

=== modified file 'setup.py'
--- setup.py	2011-08-31 17:04:09 +0000
+++ setup.py	2011-10-19 22:12:25 +0000
@@ -2,10 +2,10 @@
 import os
 import re
 
-try:
-    from setuptools import setup, Extension
-except ImportError:
-    from distutils.core import setup, Extension
+import ez_setup
+ez_setup.use_setuptools()
+
+from setuptools import setup, Extension, find_packages
 
 
 if os.path.isfile("MANIFEST"):
@@ -19,20 +19,12 @@
                     open("storm/__init__.py").read()).group(1)
 
 
-def find_packages():
-    # implement a simple find_packages so we don't have to depend on
-    # setuptools
-    packages = []
-    for directory, subdirectories, files in os.walk("storm"):
-        if '__init__.py' in files:
-            packages.append(directory.replace(os.sep, '.'))
-    return packages
-
-
 setup(
     name="storm",
     version=VERSION,
-    description="Storm is an object-relational mapper (ORM) for Python developed at Canonical.",
+    description=(
+        "Storm is an object-relational mapper (ORM) for Python "
+        "developed at Canonical."),
     author="Gustavo Niemeyer",
     author_email="[e-mail]",
     maintainer="Storm Developers",
@@ -41,18 +33,42 @@
     url="https://storm.canonical.com",
     download_url="https://launchpad.net/storm/+download",
     packages=find_packages(),
-    zip_safe=False,
-    include_package_data=True,
     package_data={"": ["*.zcml"]},
     classifiers=[
         "Development Status :: 5 - Production/Stable",
         "Intended Audience :: Developers",
-        "License :: OSI Approved :: GNU Library or Lesser General Public License (LGPL)",
+        ("License :: OSI Approved :: GNU Library or "
+         "Lesser General Public License (LGPL)"),
         "Programming Language :: Python",
         "Topic :: Database",
         "Topic :: Database :: Front-Ends",
         "Topic :: Software Development :: Libraries :: Python Modules",
     ],
     ext_modules=(BUILD_CEXTENSIONS and
-                 [Extension("storm.cextensions", ["storm/cextensions.c"])])
-)
+                 [Extension("storm.cextensions", ["storm/cextensions.c"])]),
+    # The following options are specific to setuptools but ignored (with a
+    # warning) by distutils.
+    include_package_data=True,
+    zip_safe=False,
+    test_suite = "tests.find_tests",
+    tests_require=[
+        # Versions based on Lucid, where packaged.
+        "django >= 1.1.1",
+        "fixtures >= 0.3.5",
+        # pgbouncer (the Python module) is not yet packaged in Ubuntu.
+        "pgbouncer >= 0.0.5",
+        "psycopg2 >= 2.0.13",
+        "testresources >= 0.2.4",
+        # timeline is not yet packaged in Ubuntu.
+        "timeline >= 0.0.2",
+        "transaction >= 1.0.0",
+        "twisted >= 10.0.0",
+        "zope.component >= 3.8.0",
+        # zope.component 3.11.0 requires a version of zope.interface that no
+        # version of Ubuntu yet packages. The following rule exists for the
+        # sake of convenience rather than necessity, for the situation where
+        # zope.interface is installed via a package but zope.component is not.
+        "zope.component < 3.11.0",
+        "zope.security >= 3.7.2",
+        ],
+    )

=== modified file 'storm/databases/postgres.py'
--- storm/databases/postgres.py	2011-09-22 09:16:49 +0000
+++ storm/databases/postgres.py	2011-10-19 22:12:25 +0000
@@ -47,7 +47,7 @@
 from storm.database import Database, Connection, Result
 from storm.exceptions import (
     install_exceptions, DatabaseError, DatabaseModuleError, InterfaceError,
-    OperationalError, ProgrammingError, TimeoutError)
+    OperationalError, ProgrammingError, TimeoutError, Error)
 from storm.tracer import TimeoutTracer
 
 
@@ -217,6 +217,19 @@
         return And(*equals)
 
 
+pg_connection_failure_codes = frozenset([
+    '08006',  # CONNECTION FAILURE
+    '08001',  # SQLCLIENT UNABLE TO ESTABLISH SQLCONNECTION
+    '08004',  # SQLSERVER REJECTED ESTABLISHMENT OF SQLCONNECTION
+    '53300',  # TOO MANY CONNECTIONS
+    '57000',  # OPERATOR INTERVENTION
+    '57014',  # QUERY CANCELED
+    '57P01',  # ADMIN SHUTDOWN
+    '57P02',  # CRASH SHUTDOWN
+    '57P03',  # CANNOT CONNECT NOW
+    ])
+
+
 class PostgresConnection(Connection):
 
     result_factory = PostgresResult
@@ -277,22 +290,35 @@
                 yield param
 
     def is_disconnection_error(self, exc, extra_disconnection_errors=()):
-        if not isinstance(exc, (InterfaceError, OperationalError,
-                                ProgrammingError, extra_disconnection_errors)):
-            return False
-
-        # XXX: 2007-09-17 jamesh
-        # The last message is for the benefit of old versions of
-        # psycopg2 (prior to 2.0.7) which have a bug related to
-        # stripping the error severity from the message.
-        msg = str(exc)
-        return ("server closed the connection unexpectedly" in msg or
+        # Attempt to use pgcode to determine the nature of the error. This is
+        # more reliable than string matching because it is not affected by
+        # locale settings. Fall through if pgcode is not available.
+        if isinstance(exc, Error):
+            pgcode = getattr(exc, "pgcode", None)
+            if pgcode in pg_connection_failure_codes:
+                return True
+
+        disconnection_errors = (
+            InterfaceError, OperationalError, ProgrammingError,
+            extra_disconnection_errors)
+
+        if isinstance(exc, disconnection_errors):
+            msg = str(exc)
+            return (
+                "connection already closed" in msg or
+                "connection not open" in msg or
                 "could not connect to server" in msg or
+                "could not receive data from server" in msg or
+                "losed the connection unexpectedly" in msg or
                 "no connection to the server" in msg or
-                "connection not open" in msg or
-                "connection already closed" in msg or
-                "losed the connection unexpectedly" in msg or
-                "could not receive data from server" in msg)
+                "terminating connection due to administrator" in msg)
+        elif isinstance(exc, DatabaseError):
+            msg = str(exc)
+            return (
+                "EOF detected" in msg or
+                "server closed the connection unexpectedly" in msg)
+        else:
+            return False
 
 
 class Postgres(Database):

=== modified file 'test'
--- test	2011-09-13 10:43:40 +0000
+++ test	2011-10-19 22:12:25 +0000
@@ -19,6 +19,7 @@
 # You should have received a copy of the GNU Lesser General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
+import glob
 import optparse
 import unittest
 import doctest
@@ -28,73 +29,11 @@
 import tests
 
 
-def find_tests(testpaths=()):
-    """Find all test paths, or test paths contained in the provided sequence.
-
-    @param testpaths: If provided, only tests in the given sequence will
-                      be considered.  If not provided, all tests are
-                      considered.
-    @return: a test suite containing the requested tests.
-    """
-    suite = unittest.TestSuite()
-    topdir = os.path.abspath(os.path.dirname(__file__))
-    testdir = os.path.dirname(tests.__file__)
-    testpaths = set(testpaths)
-    for root, dirnames, filenames in os.walk(testdir):
-        for filename in filenames:
-            filepath = os.path.join(root, filename)
-            relpath = filepath[len(topdir)+1:]
-
-            if (filename == "__init__.py" or filename.endswith(".pyc") or
-                relpath == os.path.join("tests", "conftest.py")):
-                # Skip non-tests.
-                continue
-
-            if testpaths:
-                # Skip any tests not in testpaths.
-                for testpath in testpaths:
-                    if relpath.startswith(testpath):
-                        break
-                else:
-                    continue
-
-            if filename.endswith(".py"):
-                modpath = relpath.replace(os.path.sep, ".")[:-3]
-                module = __import__(modpath, None, None, [""])
-                suite.addTest(
-                    unittest.defaultTestLoader.loadTestsFromModule(module))
-            elif filename.endswith(".txt"):
-                load_test = True
-                if relpath == os.path.join("tests", "zope", "README.txt"):
-                    # Special case the inclusion of the Zope-dependent
-                    # ZStorm doctest.
-                    import tests.zope as ztest
-                    load_test = (
-                        ztest.has_transaction and
-                        ztest.has_zope_component and
-                        ztest.has_zope_security)
-                if load_test:
-                    parent_path = os.path.dirname(relpath).replace(
-                        os.path.sep, ".")
-                    parent_module = __import__(parent_path, None, None, [""])
-                    suite.addTest(doctest.DocFileSuite(
-                            os.path.basename(relpath),
-                            module_relative=True,
-                            package=parent_module,
-                            optionflags=doctest.ELLIPSIS))
-
-    return suite
-
-
-def parse_sys_argv():
-    """Extract any arguments not starting with '-' from sys.argv."""
-    testpaths = []
-    for i in range(len(sys.argv)-1,0,-1):
-        arg = sys.argv[i]
-        if not arg.startswith("-"):
-            testpaths.append(arg)
-            del sys.argv[i]
-    return testpaths
+def add_eggs_to_path():
+    here = os.path.dirname(__file__)
+    egg_paths = glob.glob(os.path.join(here, "*.egg"))
+    sys.path[:0] = map(os.path.abspath, egg_paths)
+
 
 def test_with_runner(runner):
     usage = "test.py [options] [<test filename>, ...]"
@@ -107,7 +46,11 @@
     if opts.verbose:
         runner.verbosity = 2
 
-    suite = find_tests(args)
+    # python setup.py test [--dry-run] puts $package.egg directories in the
+    # top directory, so we add them to sys.path here for convenience.
+    add_eggs_to_path()
+
+    suite = tests.find_tests(args)
     result = runner.run(suite)
     return not result.wasSuccessful()
 

=== modified file 'tests/__init__.py'
--- tests/__init__.py	2008-06-20 14:08:30 +0000
+++ tests/__init__.py	2011-10-19 22:12:25 +0000
@@ -0,0 +1,96 @@
+#
+# Copyright (c) 2011 Canonical
+#
+# Written by Gustavo Niemeyer <[e-mail]>
+#
+# This file is part of Storm Object Relational Mapper.
+#
+# Storm is free software; you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License as
+# published by the Free Software Foundation; either version 2.1 of
+# the License, or (at your option) any later version.
+#
+# Storm is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+__all__ = [
+    'find_tests',
+    'has_fixtures',
+    ]
+
+import doctest
+import os
+import unittest
+
+try:
+    import fixtures
+    fixtures  # Silence lint.
+except ImportError:
+    has_fixtures = False
+else:
+    has_fixtures = True
+
+
+def find_tests(testpaths=()):
+    """Find all test paths, or test paths contained in the provided sequence.
+
+    @param testpaths: If provided, only tests in the given sequence will
+                      be considered.  If not provided, all tests are
+                      considered.
+    @return: a test suite containing the requested tests.
+    """
+    suite = unittest.TestSuite()
+    topdir = os.path.abspath(
+        os.path.join(os.path.dirname(__file__), os.pardir))
+    testdir = os.path.dirname(__file__)
+    testpaths = set(testpaths)
+    for root, dirnames, filenames in os.walk(testdir):
+        for filename in filenames:
+            filepath = os.path.join(root, filename)
+            relpath = filepath[len(topdir)+1:]
+
+            if (filename == "__init__.py" or filename.endswith(".pyc") or
+                relpath == os.path.join("tests", "conftest.py")):
+                # Skip non-tests.
+                continue
+
+            if testpaths:
+                # Skip any tests not in testpaths.
+                for testpath in testpaths:
+                    if relpath.startswith(testpath):
+                        break
+                else:
+                    continue
+
+            if filename.endswith(".py"):
+                modpath = relpath.replace(os.path.sep, ".")[:-3]
+                module = __import__(modpath, None, None, [""])
+                suite.addTest(
+                    unittest.defaultTestLoader.loadTestsFromModule(module))
+            elif filename.endswith(".txt"):
+                load_test = True
+                if relpath == os.path.join("tests", "zope", "README.txt"):
+                    # Special case the inclusion of the Zope-dependent
+                    # ZStorm doctest.
+                    import tests.zope as ztest
+                    load_test = (
+                        ztest.has_transaction and
+                        ztest.has_zope_component and
+                        ztest.has_zope_security)
+                if load_test:
+                    parent_path = os.path.dirname(relpath).replace(
+                        os.path.sep, ".")
+                    parent_module = __import__(parent_path, None, None, [""])
+                    suite.addTest(doctest.DocFileSuite(
+                            os.path.basename(relpath),
+                            module_relative=True,
+                            package=parent_module,
+                            optionflags=doctest.ELLIPSIS))
+
+    return suite

=== modified file 'tests/databases/postgres.py'
--- tests/databases/postgres.py	2011-02-25 10:26:26 +0000
+++ tests/databases/postgres.py	2011-10-19 22:12:25 +0000
@@ -22,26 +22,53 @@
 import os
 
 from storm.databases.postgres import (
-    Postgres, compile, currval, Returning, PostgresTimeoutTracer)
+    Postgres, compile, currval, Returning, PostgresTimeoutTracer, make_dsn)
 from storm.database import create_database
 from storm.exceptions import InterfaceError, ProgrammingError
 from storm.variables import DateTimeVariable, RawStrVariable
 from storm.variables import ListVariable, IntVariable, Variable
 from storm.properties import Int
+from storm.exceptions import DisconnectionError
 from storm.expr import (Union, Select, Insert, Alias, SQLRaw, State,
                         Sequence, Like, Column, COLUMN)
 from storm.tracer import install_tracer, TimeoutError
+from storm.uri import URI
 
 # We need the info to register the 'type' compiler.  In normal
 # circumstances this is naturally imported.
 import storm.info
+storm  # Silence lint.
 
+from tests import has_fixtures
 from tests.databases.base import (
     DatabaseTest, DatabaseDisconnectionTest, UnsupportedDatabaseTest)
 from tests.expr import column1, column2, column3, elem1, table1, TrackContext
 from tests.tracer import TimeoutTracerTestBase
 from tests.helper import TestHelper
 
+try:
+    import pgbouncer
+except ImportError:
+    has_pgbouncer = False
+else:
+    has_pgbouncer = True
+
+
+def terminate_other_backends(connection):
+    """Terminate all connections to the database except the one given."""
+    connection.execute(
+        "SELECT pg_terminate_backend(procpid)"
+        "  FROM pg_stat_activity"
+        " WHERE datname = current_database()"
+        "   AND procpid != pg_backend_pid()")
+
+
+def terminate_all_backends(database):
+    """Terminate all connections to the given database."""
+    connection = database.connect()
+    terminate_other_backends(connection)
+    connection.close()
+
 
 class PostgresTest(DatabaseTest, TestHelper):
 
@@ -542,6 +569,98 @@
             self.fail('Exception should have been swallowed: %s' % repr(exc))
 
 
+class PostgresDisconnectionTestWithoutProxyBase(object):
+    # DatabaseDisconnectionTest uses a socket proxy to simulate broken
+    # connections. This class tests some other causes of disconnection.
+
+    database_uri = None
+
+    def is_supported(self):
+        return bool(self.database_uri)
+
+    def setUp(self):
+        super(PostgresDisconnectionTestWithoutProxyBase, self).setUp()
+        self.database = create_database(self.database_uri)
+
+    def test_terminated_backend(self):
+        # The error raised when trying to use a connection that has been
+        # terminated at the server is considered a disconnection error.
+        connection = self.database.connect()
+        terminate_all_backends(self.database)
+        self.assertRaises(
+            DisconnectionError, connection.execute,
+            "SELECT current_database()")
+
+
+class PostgresDisconnectionTestWithoutProxyUnixSockets(
+    PostgresDisconnectionTestWithoutProxyBase, TestHelper):
+    """Disconnection tests using Unix sockets."""
+
+    database_uri = os.environ.get("STORM_POSTGRES_URI")
+
+
+class PostgresDisconnectionTestWithoutProxyTCPSockets(
+    PostgresDisconnectionTestWithoutProxyBase, TestHelper):
+    """Disconnection tests using TCP sockets."""
+
+    database_uri = os.environ.get("STORM_POSTGRES_HOST_URI")
+
+
+class PostgresDisconnectionTestWithPGBouncerBase(object):
+    # Connecting via pgbouncer <http://pgfoundry.org/projects/pgbouncer>
+    # introduces new possible causes of disconnections.
+
+    def is_supported(self):
+        return (
+            has_fixtures and has_pgbouncer and
+            bool(os.environ.get("STORM_POSTGRES_HOST_URI")))
+
+    def setUp(self):
+        super(PostgresDisconnectionTestWithPGBouncerBase, self).setUp()
+        database_uri = URI(os.environ["STORM_POSTGRES_HOST_URI"])
+        database_user = database_uri.username or os.environ['USER']
+        database_dsn = make_dsn(database_uri)
+        # Create a pgbouncer fixture.
+        self.pgbouncer = pgbouncer.fixture.PGBouncerFixture()
+        self.pgbouncer.databases[database_uri.database] = database_dsn
+        self.pgbouncer.users[database_user] = "trusted"
+        self.pgbouncer.admin_users = [database_user]
+        self.useFixture(self.pgbouncer)
+        # Create a Database that uses pgbouncer.
+        pgbouncer_uri = database_uri.copy()
+        pgbouncer_uri.host = self.pgbouncer.host
+        pgbouncer_uri.port = self.pgbouncer.port
+        self.database = create_database(pgbouncer_uri)
+
+    def test_terminated_backend(self):
+        # The error raised when trying to use a connection through pgbouncer
+        # that has been terminated at the server is considered a disconnection
+        # error.
+        connection = self.database.connect()
+        terminate_all_backends(self.database)
+        self.assertRaises(
+            DisconnectionError, connection.execute,
+            "SELECT current_database()")
+
+    def test_pgbouncer_stopped(self):
+        # The error raised from a connection that is no longer connected
+        # because pgbouncer has been immediately shutdown (via SIGTERM; see
+        # man 1 pgbouncer) is considered a disconnection error.
+        connection = self.database.connect()
+        self.pgbouncer.stop()
+        self.assertRaises(
+            DisconnectionError, connection.execute,
+            "SELECT current_database()")
+
+
+if has_fixtures:
+    # Upgrade to full test case class with fixtures.
+    from fixtures import TestWithFixtures
+    class PostgresDisconnectionTestWithPGBouncer(
+        PostgresDisconnectionTestWithPGBouncerBase,
+        TestWithFixtures, TestHelper): pass
+
+
 class PostgresTimeoutTracerTest(TimeoutTracerTestBase):
 
     tracer_class = PostgresTimeoutTracer