django.db.utils.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections

Bug #1640300 reported by Nathaniel W. Turner
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Blake Rouse
2.1
Fix Released
Critical
Blake Rouse

Bug Description

With no obvious preceding event, am now seeing this error in the logs (and MAAS UI is showing Internal server error for all requests):

Nov 8 07:24:56 maas1 sh[17635]: 2016-11-08 07:24:56 [maas.websocket.listener] Listening for database notifications.
Nov 8 07:24:56 maas1 sh[17635]: 2016-11-08 07:24:56 [HTTPChannel,14914,::1] 500 Error - /MAAS/rpc/
Nov 8 07:24:56 maas1 sh[17635]: #011Traceback (most recent call last):
Nov 8 07:24:56 maas1 sh[17635]: #011 File "/usr/lib/python3/dist-packages/twisted/web/wsgi.py", line 496, in run
Nov 8 07:24:56 maas1 sh[17635]: #011 appIterator = self.application(self.environ, self.startResponse)
Nov 8 07:24:56 maas1 sh[17635]: #011 File "/usr/lib/python3/dist-packages/django/core/handlers/wsgi.py", line 189, in __call__
Nov 8 07:24:56 maas1 sh[17635]: #011 response = self.get_response(request)
Nov 8 07:24:56 maas1 sh[17635]: #011 File "/usr/lib/python3/dist-packages/maasserver/utils/views.py", line 227, in get_response
Nov 8 07:24:56 maas1 sh[17635]: #011 response = get_response(request)
Nov 8 07:24:56 maas1 sh[17635]: #011 File "/usr/lib/python3/dist-packages/maasserver/utils/views.py", line 216, in get_response
Nov 8 07:24:56 maas1 sh[17635]: #011 request, get_resolver(None), sys.exc_info())
Nov 8 07:24:56 maas1 sh[17635]: #011--- <exception caught here> ---
Nov 8 07:24:56 maas1 sh[17635]: #011 File "/usr/lib/python3/dist-packages/maasserver/utils/views.py", line 200, in get_response
Nov 8 07:24:56 maas1 sh[17635]: #011 with transaction.atomic():
Nov 8 07:24:56 maas1 sh[17635]: #011 File "/usr/lib/python3/dist-packages/django/db/transaction.py", line 150, in __enter__
Nov 8 07:24:56 maas1 sh[17635]: #011 if not connection.get_autocommit():
Nov 8 07:24:56 maas1 sh[17635]: #011 File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 286, in get_autocommit
Nov 8 07:24:56 maas1 sh[17635]: #011 self.ensure_connection()
Nov 8 07:24:56 maas1 sh[17635]: #011 File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
Nov 8 07:24:56 maas1 sh[17635]: #011 self.connect()
Nov 8 07:24:56 maas1 sh[17635]: #011 File "/usr/lib/python3/dist-packages/django/db/utils.py", line 98, in __exit__
Nov 8 07:24:56 maas1 sh[17635]: #011 six.reraise(dj_exc_type, dj_exc_value, traceback)
Nov 8 07:24:56 maas1 sh[17635]: #011 File "/usr/lib/python3/dist-packages/django/utils/six.py", line 685, in reraise
Nov 8 07:24:56 maas1 sh[17635]: #011 raise value.with_traceback(tb)
Nov 8 07:24:56 maas1 sh[17635]: #011 File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
Nov 8 07:24:56 maas1 sh[17635]: #011 self.connect()
Nov 8 07:24:56 maas1 sh[17635]: #011 File "/usr/lib/python3/dist-packages/django/db/backends/base/base.py", line 119, in connect
Nov 8 07:24:56 maas1 sh[17635]: #011 self.connection = self.get_new_connection(conn_params)
Nov 8 07:24:56 maas1 sh[17635]: #011 File "/usr/lib/python3/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 176, in get_new_connection
Nov 8 07:24:56 maas1 sh[17635]: #011 connection = Database.connect(**conn_params)
Nov 8 07:24:56 maas1 sh[17635]: #011 File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 164, in connect
Nov 8 07:24:56 maas1 sh[17635]: #011 conn = _connect(dsn, connection_factory=connection_factory, async=async)
Nov 8 07:24:56 maas1 sh[17635]: #011django.db.utils.OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections
Nov 8 07:24:56 maas1 sh[17635]: #011
Nov 8 07:24:56 maas1 sh[17635]: #011

Some kind of leak, maybe?

Versions:
maas 2.1.0+bzr5480-0ubuntu

Related branches

description: updated
Revision history for this message
Nathaniel W. Turner (nturner) wrote :
Revision history for this message
Nathaniel W. Turner (nturner) wrote :

Interestingly, the presenting symptom is gone now. I did not restart services.

Revision history for this message
Nathaniel W. Turner (nturner) wrote :
Revision history for this message
Blake Rouse (blake-rouse) wrote :

Looks to be an issue in the PostgreSQL listener. This error will continue and fill up the number of connections to Postgres.

2016-11-07 13:09:48 [maas.websocket.listener] Connection lost.
 Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/twisted/python/log.py", line 84, in callWithContext
     return context.call({ILogContext: newCtx}, func, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
     return self.currentContext().callWithContext(ctx, func, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
     return func(*args,**kw)
   File "/usr/lib/python3/dist-packages/twisted/internet/posixbase.py", line 610, in _doReadOrWrite
     self._disconnectSelectable(selectable, why, inRead)
 --- <exception caught here> ---
   File "/usr/lib/python3/dist-packages/twisted/internet/posixbase.py", line 597, in _doReadOrWrite
     why = selectable.doRead()
   File "/usr/lib/python3/dist-packages/maasserver/listener.py", line 168, in doRead
     handler = self.listeners[notify.channel][0]
 builtins.IndexError: list index out of range

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.