[SRU] Memory leaks in rados.py
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| ceph (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
| Trusty |
Fix Released
|
High
|
Billy Olsen | ||
| Utopic |
Fix Released
|
High
|
Billy Olsen | ||
| Vivid |
Fix Released
|
High
|
Unassigned | ||
Bug Description
[Impact]
Using Ceph with Cinder causes memory to grow due to memory leaks in librados.py which holds on to native C callback function references. These references cause a circular dependency which prevents the python garbage collector from freeing the memory used by the Ioctx object. Over time with repeated volume and snapshot creation/deletion, this can cause the memory used by the Cinder process to grow continuously without retraction.
The fix is to not store a reference to the native c callback functions in the Ioctx object and instead pass the references around so they can live only as long as necessary to fulfill the request. This avoids the circular dependency issue.
[Test Case]
1. Deploy OpenStack using Ceph as a block storage for Cinder.
2. Repeatedly create and delete volumes.
3. Observe memory usage by the cinder-volume backend services.
[Regression Potential]
From a regression standpoint, there exists the possibility of a small performance regression due to creating multiple C native callback references and passing those around rather than keeping a single class level reference to a native C callback function. This may be more noticeable when using the librados library to actually perform I/O to a volume rather than using it to manage objects, which is done within the Cinder library.
[Other Info]
This has been filed and fix in upstream bug http://
| tags: | added: cts |
| Changed in ceph (Ubuntu Vivid): | |
| status: | New → Fix Released |
| Changed in ceph (Ubuntu Trusty): | |
| importance: | Undecided → High |
| Changed in ceph (Ubuntu Utopic): | |
| importance: | Undecided → High |
| Changed in ceph (Ubuntu Vivid): | |
| importance: | Undecided → High |
| Changed in ceph (Ubuntu Utopic): | |
| status: | New → Triaged |
| Changed in ceph (Ubuntu Trusty): | |
| status: | New → Triaged |
| summary: |
- Memory leaks in librados.py + [SRU] Memory leaks in librados.py |
| Billy Olsen (billy-olsen) wrote Re: [SRU] Memory leaks in librados.py | #1 |
| Billy Olsen (billy-olsen) wrote | #2 |
| summary: |
- [SRU] Memory leaks in librados.py + [SRU] Memory leaks in rados.py |
| Changed in ceph (Ubuntu Trusty): | |
| status: | Triaged → In Progress |
| Changed in ceph (Ubuntu Utopic): | |
| status: | Triaged → In Progress |
| Changed in ceph (Ubuntu Trusty): | |
| assignee: | nobody → Billy Olsen (billy-olsen) |
| Changed in ceph (Ubuntu Utopic): | |
| assignee: | nobody → Billy Olsen (billy-olsen) |
| Chris J Arges (arges) wrote Please test proposed package | #3 |
| Changed in ceph (Ubuntu Trusty): | |
| status: | In Progress → Fix Committed |
| tags: | added: verification-needed |
| Changed in ceph (Ubuntu Utopic): | |
| status: | In Progress → Fix Committed |
| Chris J Arges (arges) wrote | #4 |
| Takashi Aosawa (t-aosawa) wrote | #5 |
| tags: | added: verification-done-trusty |
| Billy Olsen (billy-olsen) wrote | #6 |
| tags: |
added: verification-done removed: cts verification-done-trusty verification-needed |
| tags: | added: cts |
| Launchpad Janitor (janitor) wrote | #7 |
| Changed in ceph (Ubuntu Trusty): | |
| status: | Fix Committed → Fix Released |
| Chris J Arges (arges) wrote Update Released | #8 |
| Launchpad Janitor (janitor) wrote | #9 |
| Changed in ceph (Ubuntu Utopic): | |
| status: | Fix Committed → Fix Released |
Hello Billy, or anyone else affected,
Accepted ceph into trusty-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-
Further information regarding the verification process can be found at https:/