OpenStack Object Storage (swift)

Memcached connections growing beyond max_memcache_connections parameter

Bug #1235027 reported by Peter Portante on 2013-10-04

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Object Storage (swift)	Fix Released	High	Unassigned	OpenStack Object Storage (swift) 1.10.0 "havana"

Bug Description

As of https://review.openstack.org/45134, pooled memcache connections is supposed to be working. But after running a number of tests, the number of memcache connections is much more than the default of 2:

[pportant@dhcp31-22 ~]$ lsof COMMAND PID USER FD TYPE swift-pro 7959 pportant 1u CHR swift-pro 7959 pportant 2u CHR swift-pro 7959 pportant swift-pro 7959 pportant 4u IPv4 swift-pro 7959 pportant swift-pro 7959 pportant swift-pro 7959 pportant swift-pro 7959 pportant swift-pro 7959 pportant 10u swift-pro 7959 pportant 11u swift-pro 7959 pportant 12u IPv4 swift-pro 7959 pportant 14u IPv4 swift-pro 7959 pportant 15u IPv4 swift-pro 7959 pportant 16u IPv4 swift-pro 7959 pportant 17u IPv4 swift-pro 7959 pportant 18u IPv4 swift-pro 7959 pportant 19u IPv4 swift-pro 7959 pportant 20u IPv4 swift-pro 7959 pportant 21u IPv4 swift-pro 7959 pportant 22u IPv4 swift-pro 7959 pportant 23u IPv4 swift-pro 7959 pportant 24u IPv4 swift-pro 7959 pportant 26u IPv4 swift-pro 7959 pportant 325u IPv4 swift-pro 7959 pportant 326u IPv4 swift-pro 7959 pportant 327u IPv4 swift-pro 7959 pportant 328u IPv4 swift-pro 7959 pportant 329u IPv4 swift-pro 7959 pportant 330u IPv4 swift-pro 7959 pportant 331u IPv4 swift-pro 7959 pportant 332u IPv4 swift-pro 7959 pportant 333u IPv4 swift-pro 7959 pportant 334u IPv4 swift-pro 7959 pportant 335u IPv4 swift-pro 7959 pportant 685u IPv4 swift-pro 7959 pportant 731u IPv4 swift-pro 7959 pportant 732u IPv4 swift-pro 7959 pportant 734u IPv4 swift-pro 7959 pportant 735u IPv4 swift-pro 7959 pportant 736u IPv4 swift-pro 7959 pportant 737u IPv4 swift-pro 7959 pportant 744u IPv4 swift-pro 7959 pportant 745u IPv4 swift-pro 7959 pportant 746u IPv4 swift-pro 7959 pportant 748u IPv4 swift-pro 7959 pportant 749u IPv4 swift-pro 7959 pportant 750u IPv4 swift-pro 7959 pportant 753u IPv4 swift-pro 7959 pportant 755u IPv4 swift-pro 7959 pportant 756u IPv4 swift-pro 7959 pportant 759u IPv4 swift-pro 7959 pportant 760u IPv4 swift-pro 7959 pportant 762u IPv4 swift-pro 7959 pportant 764u IPv4 swift-pro 7959 pportant 767u IPv4 swift-pro 7959 pportant 770u IPv4 swift-pro 7959 pportant 772u IPv4 swift-pro 7959 pportant 774u IPv4 swift-pro 7959 pportant 775u IPv4 swift-pro 7959 pportant 777u IPv4 swift-pro 7959 pportant 778u IPv4 swift-pro 7959 pportant 780u IPv4 swift-pro 7959 pportant 781u IPv4 -a -d 1-99999 -p 7959
DEVICE SIZE/OFF NODE NAME
1,3 0t0 4942 /dev/null
1,3 0t0 4942 /dev/null
3u unix 0xffff8800058eee00 0t0 523001 socket
523002 0t0 TCP *:webcache (LISTEN)
6u unix 0xffff880025cd4380 0t0 523067 socket
7u unix 0xffff88003c1dae00 0t0 523072 socket
8u unix 0xffff88003c1db880 0t0 523073 socket
9u unix 0xffff88003c1dbc00 0t0 523080 socket
unix 0xffff88003c1db500 0t0 523081 socket
unix 0xffff880034e86a80 0t0 523071 socket
523626 0t0 TCP localhost:41859->localhost:memcache (ESTABLISHED)
545103 0t0 TCP localhost:48649->localhost:memcache (ESTABLISHED)
545105 0t0 TCP localhost:48650->localhost:memcache (ESTABLISHED)
545112 0t0 TCP localhost:48653->localhost:memcache (ESTABLISHED)
545110 0t0 TCP localhost:48652->localhost:memcache (ESTABLISHED)
545132 0t0 TCP localhost:48660->localhost:memcache (ESTABLISHED)
545134 0t0 TCP localhost:48661->localhost:memcache (ESTABLISHED)
545136 0t0 TCP localhost:48662->localhost:memcache (ESTABLISHED)
545138 0t0 TCP localhost:48663->localhost:memcache (ESTABLISHED)
545173 0t0 TCP localhost:48672->localhost:memcache (ESTABLISHED)
545143 0t0 TCP localhost:48665->localhost:memcache (ESTABLISHED)
545180 0t0 TCP localhost:48673->localhost:memcache (ESTABLISHED)
545185 0t0 TCP localhost:48675->localhost:memcache (ESTABLISHED)
546078 0t0 TCP localhost:48976->localhost:memcache (ESTABLISHED)
546080 0t0 TCP localhost:48977->localhost:memcache (ESTABLISHED)
546082 0t0 TCP localhost:48978->localhost:memcache (ESTABLISHED)
546084 0t0 TCP localhost:48979->localhost:memcache (ESTABLISHED)
546086 0t0 TCP localhost:48980->localhost:memcache (ESTABLISHED)
546088 0t0 TCP localhost:48981->localhost:memcache (ESTABLISHED)
546090 0t0 TCP localhost:48982->localhost:memcache (ESTABLISHED)
546092 0t0 TCP localhost:48983->localhost:memcache (ESTABLISHED)
546094 0t0 TCP localhost:48984->localhost:memcache (ESTABLISHED)
546096 0t0 TCP localhost:48985->localhost:memcache (ESTABLISHED)
546098 0t0 TCP localhost:48986->localhost:memcache (ESTABLISHED)
537750 0t0 TCP localhost:46264->localhost:memcache (ESTABLISHED)
543256 0t0 TCP localhost:48110->localhost:memcache (ESTABLISHED)
543258 0t0 TCP localhost:48111->localhost:memcache (ESTABLISHED)
543262 0t0 TCP localhost:48113->localhost:memcache (ESTABLISHED)
543264 0t0 TCP localhost:48114->localhost:memcache (ESTABLISHED)
543266 0t0 TCP localhost:48115->localhost:memcache (ESTABLISHED)
543268 0t0 TCP localhost:48116->localhost:memcache (ESTABLISHED)
543283 0t0 TCP localhost:48123->localhost:memcache (ESTABLISHED)
543285 0t0 TCP localhost:48124->localhost:memcache (ESTABLISHED)
543287 0t0 TCP localhost:48125->localhost:memcache (ESTABLISHED)
543291 0t0 TCP localhost:48127->localhost:memcache (ESTABLISHED)
543293 0t0 TCP localhost:isnetserv->localhost:memcache (ESTABLISHED)
543295 0t0 TCP localhost:blp5->localhost:memcache (ESTABLISHED)
543301 0t0 TCP localhost:48132->localhost:memcache (ESTABLISHED)
543305 0t0 TCP localhost:48134->localhost:memcache (ESTABLISHED)
543307 0t0 TCP localhost:48135->localhost:memcache (ESTABLISHED)
543313 0t0 TCP localhost:48138->localhost:memcache (ESTABLISHED)
543315 0t0 TCP localhost:48139->localhost:memcache (ESTABLISHED)
543319 0t0 TCP localhost:48141->localhost:memcache (ESTABLISHED)
543323 0t0 TCP localhost:48143->localhost:memcache (ESTABLISHED)
543329 0t0 TCP localhost:48146->localhost:memcache (ESTABLISHED)
543336 0t0 TCP localhost:48149->localhost:memcache (ESTABLISHED)
543340 0t0 TCP localhost:48151->localhost:memcache (ESTABLISHED)
543344 0t0 TCP localhost:48153->localhost:memcache (ESTABLISHED)
543346 0t0 TCP localhost:48154->localhost:memcache (ESTABLISHED)
543350 0t0 TCP localhost:48156->localhost:memcache (ESTABLISHED)
543352 0t0 TCP localhost:48157->localhost:memcache (ESTABLISHED)
543356 0t0 TCP localhost:48159->localhost:memcache (ESTABLISHED)
543358 0t0 TCP localhost:48160->localhost:memcache (ESTABLISHED)

Run on a Fedora 19 SAIO using the patched code for https://review.openstack.org/48538, Sam Merrit's client disconnect fix.

John Dickinson (notmyname) on 2013-10-07

Changed in swift:
milestone:	none → 1.9.3-rc1
importance:	Undecided → High

John Dickinson (notmyname) on 2013-10-08

Changed in swift:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-10-09: Fix merged to swift (master)

Reviewed: https://review.openstack.org/49739
Committed: http://github.com/openstack/swift/commit/6607beab0dc8043251b490471761fa2dd85f2816
Submitter: Jenkins
Branch: master

commit 6607beab0dc8043251b490471761fa2dd85f2816
Author: Peter Portante <email address hidden>
Date: Fri Oct 4 08:04:42 2013 -0400

Don't apply timeout to Pool.get operation (leaks)

    The connection timeout to a memcache server is performed by using the
    "with Timeout()" construct over the sock.connect() call in the
    .create() method. In addition, the same construct was being applied to
    the Pool.get() call in ._get_conns().

    If the maximum number of connections was already created, and the
    Pool.get() called took longer than the connect timeout, then the error
    handling path would add a place holder to the connection
    pool. Eventlet's Pool class allows for additional items to be added to
    the pool, over and above the max_size setting. This additional place
    holder will eventually be pulled and a new connection created to take
    its place.

The fix is to remove the timeout construct in the _get_conns() method.

    In addition, we also apply the unit test patch mentioned in the review
    comments for Patch Set 6 of https://review.openstack.org/45134,
    located at http://paste.openstack.org/show/47288/.

Fixes bug 1235027

Change-Id: I786cabefe3e8ddf7d92feb7ebc7cfb613d60a1da
Signed-off-by: Peter Portante <email address hidden>

Changed in swift:
status:	In Progress → Fix Committed

Thierry Carrez (ttx) on 2013-10-09

Changed in swift:
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2013-10-17

Changed in swift:
milestone:	1.10.0-rc1 → 1.10.0

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-10-27: Fix proposed to swift (feature/ec)

Fix proposed to branch: feature/ec
Review: https://review.openstack.org/54029

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-10-29: Fix merged to swift (feature/ec)

Download full text (24.9 KiB)

Reviewed: https://review.openstack.org/54029
Committed: http://github.com/openstack/swift/commit/94d3671b0bbf87fdbff845643963f3f9a97c58b5
Submitter: Jenkins
Branch: feature/ec

commit abcecd26a7b5871f75f0fbddf0d08bbac95bb089
Author: Kun Huang <email address hidden>
Date: Wed Oct 23 21:19:01 2013 +0800

utf8 encode tempurl key

    In tempurl middleware, hmac uses the value of account metadata to
    generate HMAC-SHA1 signature and hmac must accept a str-type string, not
    a unicode string. The meta dict returned from get_info stroges special
    chars as unicode however. So just encode it for tempurl using.

Closes-Bug: #1242644
Change-Id: I4be62eea014a573efc4748470de57dccf00e431d

commit cd2e7df0b69bbd269cd3c4170e0fee8186a07c95
Author: Pete Zaitcev <email address hidden>
Date: Tue Oct 22 17:18:04 2013 -0600

Add an __str__ method to brokers

    A few uses of broker.db_file are in printouts where we do need
    them, so the administrator may know what's up. Seems like an easy
    way to get rid of those is to make brokers identify themselves
    with common __str__. Alternative back-end implementations may
    supply something other than a filename here, for example a cluster
    name and a volume name.

Note that I'm not sure if correct coercion would occur when
brokers are bounced through dictionaries, hence explicit str().

Change-Id: I329788ebd1fbe39ffadcf9f9d5194a74a88dde58

commit 9807a358c6d1314d25e3a41da75be5851fa0ac27
Author: Donagh McCabe <email address hidden>
Date: Fri Aug 23 15:03:08 2013 +0100

Add WWW-Authenticate to 401 responses

    Per http://www.ietf.org/rfc/rfc2616.txt, when a 401 error is returned, the
    Www-Authenticate response header MUST also be returned. The format is
    described in http://www.ietf.org/rfc/rfc2617.txt.

    Swift supports and/or implements a number of authentication schemes
    including tempauth, Keystone, tempurl, formpost and container sync. In
    this fix, we use a catch-all, "Swift". The realm is the account (where
    known) or "unknown" (bad path or where the 401 is returned from code
    that does not have the request). Examples:

Www-Authenticate: Swift realm="AUTH_1234567889"
Www-Authenticate: Swift realm="unknown"

Fixes bug #1215491

Change-Id: I03362789318dfa156d3733ef9348795062a9cfc4

commit ed5101b2002b877518466ae5f9a6d652581238f2
Author: Yuan Zhou <email address hidden>
Date: Sat Oct 19 11:40:35 2013 +0800

Adding more unit tests for audit_location_generator

Change-Id: I40410fbbb79cea8647074f703e4675364c69d930

commit 5202b0e58613738cc81ec63e7c6da14ce5429526
Author: Peter Portante <email address hidden>
Date: Thu Sep 12 19:51:18 2013 -0400

DiskFile API, with reference implementation

    Refactor on-disk knowledge out of the object server by pushing the
    async update pickle creation to the new DiskFileManager class (name is
    not the best, so suggestions welcome), along with the REPLICATOR
    method logic. We also move the mount checking and thread pool storage
    to the new ondisk.Devices object, which then also becomes th...

Reviewed:  https://review.openstack.org/54029
Committed: http://github.com/openstack/swift/commit/94d3671b0bbf87fdbff845643963f3f9a97c58b5
Submitter: Jenkins
Branch:    feature/ec

commit abcecd26a7b5871f75f0fbddf0d08bbac95bb089
Author: Kun Huang <academicgareth@gmail.com>
Date:   Wed Oct 23 21:19:01 2013 +0800

utf8 encode tempurl key
    
    In tempurl middleware, hmac uses the value of account metadata to
    generate HMAC-SHA1 signature and hmac must accept a str-type string, not
    a unicode string. The meta dict returned from get_info stroges special
    chars as unicode however. So just encode it for tempurl using.
    
    Closes-Bug: #1242644
    Change-Id: I4be62eea014a573efc4748470de57dccf00e431d

commit cd2e7df0b69bbd269cd3c4170e0fee8186a07c95
Author: Pete Zaitcev <zaitcev@kotori.zaitcev.us>
Date:   Tue Oct 22 17:18:04 2013 -0600

Add an __str__ method to brokers
    
    A few uses of broker.db_file are in printouts where we do need
    them, so the administrator may know what's up. Seems like an easy
    way to get rid of those is to make brokers identify themselves
    with common __str__. Alternative back-end implementations may
    supply something other than a filename here, for example a cluster
    name and a volume name.
    
    Note that I'm not sure if correct coercion would occur when
    brokers are bounced through dictionaries, hence explicit str().
    
    Change-Id: I329788ebd1fbe39ffadcf9f9d5194a74a88dde58

commit 9807a358c6d1314d25e3a41da75be5851fa0ac27
Author: Donagh McCabe <donagh.mccabe@hp.com>
Date:   Fri Aug 23 15:03:08 2013 +0100

Add WWW-Authenticate to 401 responses
    
    Per http://www.ietf.org/rfc/rfc2616.txt, when a 401 error is returned, the
    Www-Authenticate response header MUST also be returned. The format is
    described in http://www.ietf.org/rfc/rfc2617.txt.
    
    Swift supports and/or implements a number of authentication schemes
    including tempauth, Keystone, tempurl, formpost and container sync. In
    this fix, we use a catch-all, "Swift". The realm is the account (where
    known) or "unknown" (bad path or where the 401 is returned from code
    that does not have the request). Examples:
    
         Www-Authenticate: Swift realm="AUTH_1234567889"
         Www-Authenticate: Swift realm="unknown"
    
    Fixes bug #1215491
    
    Change-Id: I03362789318dfa156d3733ef9348795062a9cfc4

commit ed5101b2002b877518466ae5f9a6d652581238f2
Author: Yuan Zhou <yuan.zhou@intel.com>
Date:   Sat Oct 19 11:40:35 2013 +0800

Adding more unit tests for audit_location_generator
    
    Change-Id: I40410fbbb79cea8647074f703e4675364c69d930

commit 5202b0e58613738cc81ec63e7c6da14ce5429526
Author: Peter Portante <peter.portante@redhat.com>
Date:   Thu Sep 12 19:51:18 2013 -0400

DiskFile API, with reference implementation
    
    Refactor on-disk knowledge out of the object server by pushing the
    async update pickle creation to the new DiskFileManager class (name is
    not the best, so suggestions welcome), along with the REPLICATOR
    method logic. We also move the mount checking and thread pool storage
    to the new ondisk.Devices object, which then also becomes the new home
    of the audit_location_generator method.
    
    For the object server, a new setup() method is now called at the end
    of the controller's construction, and the _diskfile() method has been
    renamed to get_diskfile(), to allow implementation specific behavior.
    
    We then hide the need for the REST API layer to know how and where
    quarantining needs to be performed. There are now two places it is
    checked internally, on open() where we verify the content-length,
    name, and x-timestamp metadata, and in the reader on close where the
    etag metadata is checked if the entire file was read.
    
    We add a reader class to allow implementations to isolate the WSGI
    handling code for that specific environment (it is used no-where else
    in the REST APIs). This simplifies the caller's code to just use a
    "with" statement once open to avoid multiple points where close needs
    to be called.
    
    For a full historical comparison, including the usage patterns see:
    https://gist.github.com/portante/5488238
    
    (as of master, 2b639f5, Merge
     "Fix 500 from account-quota     This Commit
     middleware")
    --------------------------------+------------------------------------
                                     DiskFileManager(conf)
    
                                       Methods:
                                         .pickle_async_update()
                                         .get_diskfile()
                                         .get_hashes()
    
                                       Attributes:
                                         .devices
                                         .logger
                                         .disk_chunk_size
                                         .keep_cache_size
                                         .bytes_per_sync
    
    DiskFile(a,c,o,keep_data_fp=)    DiskFile(a,c,o)
    
      Methods:                         Methods:
       *.__iter__()
        .close(verify_file=)
        .is_deleted()
        .is_expired()
        .quarantine()
        .get_data_file_size()
                                         .open()
                                         .read_metadata()
        .create()                        .create()
                                         .write_metadata()
        .delete()                        .delete()
    
      Attributes:                      Attributes:
        .quarantined_dir
        .keep_cache
        .metadata
                                    *DiskFileReader()
    
                                       Methods:
                                         .__iter__()
                                         .close()
    
                                       Attributes:
                                        +.was_quarantined
    
    DiskWriter()                     DiskFileWriter()
    
      Methods:                         Methods:
        .write()                         .write()
        .put()                           .put()
    
    * Note that the DiskFile class   * Note that the DiskReader() object
      implements all the methods       returned by the
      necessary for a WSGI app         DiskFileOpened.reader() method
      iterator                         implements all the methods
                                       necessary for a WSGI app iterator
    
                                     + Note that if the auditor is
                                       refactored to not use the DiskFile
                                       class, see
                                       https://review.openstack.org/44787
                                       then we don't need the
                                       was_quarantined attribute
    
    A reference "in-memory" object server implementation of a backend
    DiskFile class in swift/obj/mem_server.py and
    swift/obj/mem_diskfile.py.
    
    One can also reference
    https://github.com/portante/gluster-swift/commits/diskfile for the
    proposed integration with the gluster-swift code based on these
    changes.
    
    Change-Id: I44e153fdb405a5743e9c05349008f94136764916
    Signed-off-by: Peter Portante <peter.portante@redhat.com>

commit a4e25d6da8d007051b3b0575de47338e210d4315
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Mon Oct 14 13:12:22 2013 -0700

Proxy: use just one greenthread per client
    
    Instead of using two greenthreads and an eventlet.queue.Queue of size
    1 to stream response bodies from backends to clients, just do it with
    a single greenthread that reads and writes in a loop. This should
    lower the amount of CPU used by the proxy in its response streaming.
    
    Client fairness used to be provided implicitly; since the queue only
    held 1 item, the read-from-backend greenthread would block after each
    chunk until the write-to-client greenthread got a chance to run. Now
    fairness is provided by an explicit eventlet.sleep() call, just as it
    is in the object server.
    
    Change-Id: Iae27109f5a3d109ad21ec9a972e39f22150f6dbb

commit a0017e053734ea5e72bfcc3fdc630c8a7a25431a
Author: Greg Lange <greglange@gmail.com>
Date:   Tue Oct 15 16:23:00 2013 +0000

Removes vim comment
    
    Change-Id: Ibc930638ee2d839e8499a13018c205b615f8b854
    Partial-Bug: 1229324

commit d4c658750562f2baa978cc28d341680c6b470070
Author: Greg Lange <greglange@gmail.com>
Date:   Tue Oct 15 15:38:02 2013 +0000

return value on memcache client decr call
    
    Closes-Bug: 1238204
    Change-Id: I7d2ece118159637e5c5bf4bf5dd4ce4d2f6e4523

commit 9c226e3a1530a21d754ac7f1702c35f4a8e5927c
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Thu Oct 10 13:33:05 2013 -0700

Small optimization to container_quotas
    
    When handling an object PUT with X-Copy-From, we only need to fetch
    the source object's size if the container has a byte quota, but we
    were always fetching it. Now we only fetch the size when we need it.
    
    Change-Id: I0c2c6820cdf0022ef57df5fe7dcb2dd466665a4f

commit b61add12e4910208ec115966c259553c94671202
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Wed Oct 9 12:31:26 2013 -0700

Small efficiency improvement in account quotas
    
    If you were performing an object PUT with X-Copy-From and you had
    account_quotas in your pipeline but your account did not have a quota
    set, then we will now avoid an unnecessary call to get_object_info().
    
    Change-Id: I8592d4ad65e45ecf9e0557bc10ce2c2db19732e8

commit 2419df2730c5879092f0058e63a7dfbae6519fa0
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Wed Oct 9 12:25:42 2013 -0700

Fix 500 from account-quota middleware
    
    If a user had set X-Account-Meta-Quota-Bytes to something non-integer
    prior to the installation of the account-quota middleware, then the
    quota check would choke on it. Now a non-integer value is treated as
    "no quota".
    
    Change-Id: I5c38911be1f66fa293aea9c78590d4ce7d184113

commit 9b521fe800ed3984169d37e9772674867745fede
Author: John Dickinson <me@not.mn>
Date:   Tue Oct 8 07:35:43 2013 -0700

CHANGELOG and AUTHORS updates for 1.10.0 release
    
    Change-Id: Iaf8d289c5ddb6ac778243a5e1fc27600bade6feb

commit 221aa9edd33fab41c696431261cb3414f3c02355
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Tue Oct 8 13:34:50 2013 -0700

allow container create even if over account quota
    
    Change-Id: I07278bc3314c26426ff0557fe2f5c69c02d6550c

commit 841b8082cee923eb34f2b1bc89f0c5f4e91ce9b6
Author: Aaron Rosen <arosen@nicira.com>
Date:   Mon Oct 7 22:24:10 2013 -0700

Correct URL in readme
    
    doc.openstack.org -> docs.openstack.org
    
    Change-Id: Iacadb8f70cb94adc002c562a1068ff38be19a9fc

commit 0b370269111957fec7521d284fcbd742ff8b8c13
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Mon Oct 7 21:28:48 2013 -0700

Add a Timeout when getting Memcache connections
    
    The old Timeout behavior when pulling connections of the MemcacheConnPool left
    ambiguity around what timed out and could put more placeholders in the queue
    than the configured max_connections.
    
    To avoid waiting indefinitely on slow severs we raise a custom Timeout when we
    fail get a connection from the pool.  We still error limit the slow server,
    and move onto the next, but we still don't allow more than max_connections.
    
    Change-Id: I9e2409896423d52da69e35c038e5f457c71f705d

commit 6607beab0dc8043251b490471761fa2dd85f2816
Author: Peter Portante <peter.portante@redhat.com>
Date:   Fri Oct 4 08:04:42 2013 -0400

Don't apply timeout to Pool.get operation (leaks)
    
    The connection timeout to a memcache server is performed by using the
    "with Timeout()" construct over the sock.connect() call in the
    .create() method. In addition, the same construct was being applied to
    the Pool.get() call in ._get_conns().
    
    If the maximum number of connections was already created, and the
    Pool.get() called took longer than the connect timeout, then the error
    handling path would add a place holder to the connection
    pool. Eventlet's Pool class allows for additional items to be added to
    the pool, over and above the max_size setting. This additional place
    holder will eventually be pulled and a new connection created to take
    its place.
    
    The fix is to remove the timeout construct in the _get_conns() method.
    
    In addition, we also apply the unit test patch mentioned in the review
    comments for Patch Set 6 of https://review.openstack.org/45134,
    located at http://paste.openstack.org/show/47288/.
    
    Fixes bug 1235027
    
    Change-Id: I786cabefe3e8ddf7d92feb7ebc7cfb613d60a1da
    Signed-off-by: Peter Portante <peter.portante@redhat.com>

commit 9411a24ba7b5a1380ec8a2aa13d4aa92988d2ff9
Author: Peter Portante <peter.a.portante@gmail.com>
Date:   Mon Oct 7 12:10:31 2013 +0000

Revert "Refactor common/utils methods to common/ondisk"
    
    This reverts commit 7760f41c3ce436cb23b4b8425db3749a3da33d32
    
    Change-Id: I95e57a2563784a8cd5e995cc826afeac0eadbe62
    Signed-off-by: Peter Portante <peter.portante@redhat.com>

commit 61e135e211822840731d3153e9b249b6b81a1c21
Author: Peter Portante <peter.portante@redhat.com>
Date:   Sat Oct 5 11:27:35 2013 -0400

Fix bad hash_path reference
    
    Mea culpa: these two scripts were missed in commit:
    
      https://review.openstack.org/46956
    
    Fixes bug 1235441
    
    Change-Id: I4303bc808448a79bddbb991526b0cca26150b392
    Signed-off-by: Peter Portante <peter.portante@redhat.com>

commit c7e65db27da59ddf221d1720362434581ef30311
Author: Pete Zaitcev <zaitcev@kotori.zaitcev.us>
Date:   Fri Oct 4 23:18:07 2013 -0600

Make test_translations test our tree
    
    In order to run the correct classes, Python test framework adjusts
    sys.path. However, these changes are not propagated to subprocesses.
    Therefore, the test actually tries to test installed Swift, not
    the one in which it is running.
    
    The usual suggestion is to run "python setup.py develop" before
    testing, but it's annoying and error-prone. If you forget it,
    you may test the code in /usr very easily, and never know.
    
    Let's just pass the correct path to subprocess. Much safer.
    
    Change-Id: Ic71314e8462cf6e0579d704ffe9fbbfac7e6ba24

commit 3d8f0f18059482412e9b60909de69df90416dfdd
Author: Peter Portante <peter.portante@redhat.com>
Date:   Wed Oct 2 23:58:13 2013 -0400

Fedora 19: need to use /etc/rc.d/rc.local
    
    Change-Id: I80e9a4c40ff99ec09a8eeef935447c6393ea78ec
    Signed-off-by: Peter Portante <peter.portante@redhat.com>

commit db4547d01dab84142da40fdeb1ca5fbb34ef56ba
Author: Peter Portante <peter.portante@redhat.com>
Date:   Wed Oct 2 23:52:30 2013 -0400

Remove sphinx build warnings
    
    Change-Id: Ic34bbd9cc65d96ea9b8434be7b54e5bcfae28b63
    Signed-off-by: Peter Portante <peter.portante@redhat.com>

commit def37fb56aea7b9fe4254621e10667712052d3ac
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Wed Sep 25 10:41:41 2013 -0700

Stop reading from object server when client disconnects.
    
    If a client were in the middle of an object GET request and then
    disconnected, the proxy would wait a while (default 60s) and then time
    out the connection. As part of the teardown for this, the proxy would
    attempt to close the connection to the object server, then drain any
    associated buffers. However, this didn't work particularly well,
    resulting in the proxy reading the entire remainder of the object for
    no gain.
    
    Now, the proxy closes the connection hard, by calling .close() on the
    underlying socket._socket object. This is different from calling
    .close() on a socket._socketobject object, which is what you get back
    from socket.socket() and similar methods. Calling .close() on a
    socket._socketobject simply decrements a reference counter on the
    socket._socket, which has been observed in the past to result in
    socket leaks when something holds onto a reference. However, calling
    .close() on a socket._socket actually closes the socket regardless of
    who else has a reference to it.
    
    I had to delete a test assertion that said the object server never got
    SIGPIPE after a GET w/X-Newest. Well, you get a SIGPIPE when you write
    to a closed socket, and now the proxy is actually closing the sockets
    early, so now you *do* get a SIGPIPE.
    
    closes-bug: 1174660
    
    Note that this will cause a regression on bug 1037337; unfortunately,
    the cure is worse than the disease, so out it goes.
    
    Change-Id: I9c7a2e7fdb8b4232e53ea96f86b50e8d34c27221

commit 6b67952442429cff306ceefc83dd206a4a5c16ed
Author: Dirk Mueller <dirk@dmllr.de>
Date:   Wed Oct 2 21:27:40 2013 +0200

assertEquals is deprecated, use assertEqual
    
    The very same functionality for fewer keystrokes.
    
    Change-Id: I8c9513412f398132db5636fdfb7bf8301e8443cf

commit 02e247c1b3d9637c6d6ecfd47ce82d250c576dbb
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Wed Oct 2 11:39:35 2013 -0700

Add "note" box callouts to SAIO for user changes.
    
    The SAIO is purpously cut into two parts, so that you don't have to switch
    back and forth between root and your unprivledged user.  Add some "note" box
    callouts to highlight this changeover.
    
    Change-Id: I8b1a8f0539eac60d4121bdd4dab01df75ecca207

commit 062a67b07d9a16fd4970bb90798ea67d1f99b5cd
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Wed Oct 2 09:57:34 2013 -0700

Allow slightly older dnspython (>= 1.9.4).
    
    This has a couple benefits.
    
    First, it means Ubuntu Precise users can just install python-dnspython
    from packages instead of having to pull one in from source. This
    should also fix an install error with new SAIO boxes where running
    "python setup.py develop" fails unless a satisfactory dnspython is
    already installed.
    
    Second, it matches the dnspython dependency in the global
    requirements. This means that the gates are already running with this
    dnspython dependency, and it means our requirements.txt is one step
    closer to being a subset of the global requirements.txt, which is
    important to some people.
    
    Change-Id: I5d58f488e1e4c4139c9fb20d89f386cab1537e98

commit ae8470131ead095e3bf1c290bac866a5e6e29e79
Author: Chuck Thier <cthier@gmail.com>
Date:   Wed Sep 4 22:20:44 2013 +0000

Pool memcache connections
    
    This creates a pool to each memcache server so that connections will not
    grow without bound.  This also adds a proxy config
    "max_memcache_connections" which can control how many connections are
    available in the pool.
    
    A side effect of the change is that we had to change the memcache calls
    that used noreply, and instead wait for the result of the request.
    Leaving with noreply could cause a race condition (specifically in
    account auto create), due to one request calling `memcache.del(key)` and
    then `memcache.get(key)` with a different pooled connection.  If the
    delete didn't complete fast enough, the get would return the old value
    before it was deleted, and thus believe that the account was not
    autocreated.
    
    ClaysMindExploded
    DocImpact
    Change-Id: I350720b7bba29e1453894d3d4105ac1ea232595b

commit e8a07c4ca784f958e33044c267212925d6452979
Author: Peter Portante <peter.portante@redhat.com>
Date:   Thu Sep 26 15:47:36 2013 -0400

Fedora 19 updates
    
    Change-Id: I95138852e45aa7632218a7107e0e7ba1f6ef373c
    Signed-off-by: Peter Portante <peter.portante@redhat.com>

commit f9c4eb806f14ce095554d70261504781dbb7aa3e
Author: Brian D. Burns <iosctr@gmail.com>
Date:   Sun Sep 29 15:22:05 2013 -0400

Simplify object path when reporting SLO put errors
    
    When reporting errors for SLO PUT requests, use the object path from the
    manifest without the '/vrs/account' prefix. This is a continuation of
    the same changes made for SLO/bulk delete requests in 6768d5b.
    
    Change-Id: I40c90cccc1b7b5303d9f2b084dccb3be4f4448d8

commit 8ab06eb8cc08a6f5905efde5b07cee3e71805a77
Author: anc <alistair.coles@hp.com>
Date:   Fri Sep 27 10:10:55 2013 +0100

Set path_info in sub_slo manifest GET requests
    
    The new_req Request object created for fetching
    nested sub_slo manifests does not have its
    path_info set to the sub_slo path before being
    passed to GETorHEAD_base.
    
    The sub manifest GET works fine because
    GETorHEAD_base uses its path argument
    (not req.path) to set up its connection,
    BUT if an exception is thrown,
    GETorHEAD_base uses req.path to construct
    the log message - so the exception log would
    have the wrong path.
    
    Fixes bug 1231872
    
    Change-Id: I0f0f969958f814d8a7053440d9de4e50796169e4

commit 180cd18da64b59c39ec1696e4316b18711d1fb82
Author: Brian Curtin <brian.curtin@rackspace.com>
Date:   Tue Sep 24 15:07:59 2013 -0500

Move location of HTTPException in swift_testing
    
    While doing some work around swiftclient Python 3 support, httplib
    imports were changed to use six.moves in order to access httplib in Py2
    and http.client in Py3. Tests related to those changes are failing
    because these functional tests are depending on HTTPException being
    a name in swiftclient.client. These tests should acces that name from
    its canonical location in httplib (or, eventually, a Py3 friendly
    location like six.moves).
    
    Change-Id: I6335d465574045daedab47e5fe23c415d171a83a

commit fa308d60bded8b730dda451321bb7baf5ded2aa8
Author: anc <alistair.coles@hp.com>
Date:   Fri Sep 20 15:08:12 2013 +0100

Fix utf-8 handling in object versions.
    
    Fixes object versioning when object name and/or version
    container name contain multibyte utf-8 characters.
    
    When object names containing non-ASCII characters
    are PUT multiple times into a container with an
    x-versions-location set, subsequent DELETE of the
    object results in a 500 response status.
    
    When the versions container name contains
    non-ASCII characters the first delete of an object
    succeeds but fails to restore previous version of
    object, so second delete incorrectly returns 404.
    
    Fixes bug 1229142
    
    Change-Id: I425440f76b8328f8e119d390bfa4c7022181e89e

commit 4e8b2ffc2e2b97f8bd223c90848ecaaf777bebc9
Author: Peter Portante <peter.portante@redhat.com>
Date:   Mon Sep 23 22:16:59 2013 -0400

Use created container in unit test
    
    Change-Id: I2573be1ac14f65b8008611edf940363b31c8d86e
    Signed-off-by: Peter Portante <peter.portante@redhat.com>

commit d5fcc9aaf3b6707b1ddbc577e9f8c93fa90d6d89
Author: Samuel Merritt <sam@swiftstack.com>
Date:   Mon Sep 23 10:05:34 2013 -0700

Two small account-quota fixes
    
    First: even if a user has exceeded their account quota, they should be
    able to make object POST requests. Updating an object's metadata isn't
    going to make them any more over quota, so should be allowed.
    
    Second: don't bother with the reseller_admin check for container or
    object requests. If I send the header X-Account-Meta-Quota-Bytes: 100
    on e.g. an object PUT request, the proxy will (rightly) ignore it. Now
    account-quotas does too.
    
    Change-Id: I970a76349659acdd8229a324bd33bfe7fe7261a4

commit 56440eb95da79506cc27d92e07f0f5969cc683ce
Author: Christian Schwede <info@cschwede.de>
Date:   Wed Sep 18 19:49:32 2013 +0200

Handle X-Copy-From header in container_quota middleware
    
    Content length of the copied object is checked before
    allowing the copy request according to the container quota.
    
    Closes-Bug: #1201875
    Change-Id: If44b916791e94ac6c66eee04a5727186ce0e56ae

commit f72704fc82803254a6f335fa2a7d2459f86f877d
Author: ZhiQiang Fan <aji.zqfan@gmail.com>
Date:   Fri Sep 20 01:00:54 2013 +0800

Change OpenStack LLC to Foundation
    
    Change-Id: I7c3df47c31759dbeb3105f8883e2688ada848d58
    Closes-bug: #1214176

commit b0aeed1ec7fa2fd315f6293b0fd3d8118e20c7f9
Author: Clay Gerrard <clay.gerrard@gmail.com>
Date:   Mon Sep 16 17:06:09 2013 -0700

Fix default replication options for ring-builder add
    
    Change-Id: I957deeb0e711bfe7cd9d852726c77179a4613ee0

commit 42f4b150e3fcd8c49ecac2e42839449d00190053
Author: Florian Hines <syn@ronin.io>
Date:   Thu Sep 5 18:12:15 2013 -0500

Faster swift-dispersion-populate
    
    - Makes swift-dispersion-populate a bit faster when using a larger
      dispersion_coverage with a larger part_power.
    - Adds option to only run population for container OR objects
    - Adds option to let you resume population at given point (useful if you
      need to resume population after a previous run error'd out or the
      like) by specifying which suffix to start at.
    
    The original populate just randomly used uuid4().hex as a suffix on the
    container/object names until all the partition's required where covered.
    This isn't a big deal if you're only doing 1% coverage on a ring with a
    small part power but takes ages if you're doing 100% on a larger ring.
    
    Change-Id: I52f890a774412c1d6179f12db9081aedc58b6bc2

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.