Comment 2 for bug 2012596

Revision history for this message
German Mazurenko (german-m) wrote (last edit ):

Hi Anton,
> 1. Does this memory leak happens if you make a clean 3.2.6 install?
I added freshly installed region and rack controller to my test setup, but they leaking too.
> 2. If you make a clean 2.8 install and upgrade it to 3.2.6 will it leak?
Yes
> 3. Any noticeable errors in MAAS logs?
Unfortunately, no
> 4. What did tracemalloc show? Would be great if you can provide top 10 from tracemalloc

I didn't save tracemalloc output because it wasn't helpful(and there is strange behavior: when I include tracemalloc leaking... gone)
Instead, there is pympler output which is also not so helpful but proves my problem(leackage is slower because it's test installation for debug with only two Virsh KVM and 11 VM's inside):

region, 03:46:15
                        types | # objects | total size
 ============================ | =========== | ============
                          str | 178920 | 19.41 MB
                         list | 77308 | 18.18 MB
                         dict | 56909 | 16.56 MB
                         type | 6794 | 6.51 MB
                         code | 33649 | 5.80 MB
                        tuple | 48106 | 3.10 MB
                          set | 2498 | 954.92 KB
                      weakref | 11256 | 791.44 KB
                          int | 25271 | 699.57 KB
          function (__init__) | 2695 | 357.93 KB
                         cell | 9127 | 356.52 KB
            getset_descriptor | 5594 | 349.62 KB
                    frozenset | 1022 | 324.08 KB
                  abc.ABCMeta | 314 | 321.24 KB
   builtin_function_or_method | 4046 | 284.48 KB

region, 09:49:15

                     types | # objects | total size
 ========================= | =========== | ============
                      list | 197328 | 60.93 MB
                       str | 298641 | 28.07 MB
                      dict | 60274 | 17.41 MB
                      type | 6794 | 6.51 MB
                      code | 33649 | 5.84 MB
                     tuple | 48836 | 3.15 MB
                       int | 50384 | 1.35 MB
                       set | 2786 | 1016.17 KB
                   weakref | 11259 | 791.65 KB
                      cell | 9681 | 378.16 KB
       function (__init__) | 2695 | 357.93 KB
         getset_descriptor | 5594 | 349.62 KB
   collections.OrderedDict | 724 | 335.00 KB
                 frozenset | 1022 | 324.08 KB
               abc.ABCMeta | 314 | 321.24 KB

rack, 03:45:24:

                        types | # objects | total size
 ============================ | =========== | ============
                          str | 107517 | 11.20 MB
                         list | 44329 | 10.19 MB
                         dict | 28532 | 8.87 MB
                         type | 4264 | 3.93 MB
                         code | 21193 | 3.68 MB
                        tuple | 24374 | 1.76 MB
                          set | 1876 | 688.72 KB
                      weakref | 7420 | 521.72 KB
                          int | 16852 | 468.88 KB
                  abc.ABCMeta | 281 | 288.73 KB
                    frozenset | 821 | 271.93 KB
                  uvloop.Loop | 1 | 250.52 KB
            getset_descriptor | 3850 | 240.62 KB
          function (__init__) | 1806 | 239.86 KB
   builtin_function_or_method | 3403 | 239.27 KB

rack, 09:46:20:
                        types | # objects | total size
 ============================ | =========== | ============
                         list | 119093 | 35.61 MB
                          str | 186742 | 16.80 MB
                         dict | 31835 | 9.88 MB
                         type | 4264 | 3.93 MB
                         code | 21203 | 3.72 MB
                        tuple | 25640 | 1.83 MB
                          int | 34078 | 941.95 KB
                          set | 2309 | 780.05 KB
                      weakref | 7420 | 521.72 KB
                  abc.ABCMeta | 281 | 288.73 KB
                    frozenset | 821 | 271.93 KB
                  uvloop.Loop | 1 | 250.52 KB
            getset_descriptor | 3850 | 240.62 KB
          function (__init__) | 1806 | 239.86 KB
   builtin_function_or_method | 3405 | 239.41 KB