NetApp tests fail with hash randomization

Bug #1715915 reported by Eric Harney
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
In Progress
Undecided
Eric Harney

Bug Description

NetApp driver unit tests fail with hash randomization enabled.

It is unclear whether this is a bug in the driver or just the tests, but it's possible that this indicates a real bug in the driver.

master @ 4d5f502e841

See a test run and some related fixes here: https://review.openstack.org/#/c/465636/

$ sed -i s/PYTHONHASHSEED=0/PYTHONHASHSEED=100/ tox.ini

$ tox -e py27 -- --regex cinder.tests.unit.volume.drivers.netapp

==============================
Failed 4 tests - output below:
==============================

cinder.tests.unit.volume.drivers.netapp.dataontap.performance.test_perf_cmode.PerformanceCmodeLibraryTestCase.test_update_performance_cache_first_pass
------------------------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "cinder/tests/unit/volume/drivers/netapp/dataontap/performance/test_perf_cmode.py", line 196, in test_update_performance_cache_first_pass
        self.perf_library.performance_counters)
      File "/home/eharney/src/cinder-review5/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
        self.assertThat(observed, matcher, message)
      File "/home/eharney/src/cinder-review5/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
        raise mismatch_error
    testtools.matchers._impl.MismatchError: {'node2': [21], 'node1': [11]} != {'node2': [11], 'node1': [21]}

cinder.tests.unit.volume.drivers.netapp.dataontap.test_block_base.NetAppBlockStorageLibraryTestCase.test_build_initiator_target_map_with_lookup_service
-------------------------------------------------------------------------------------------------------------------------------------------------------

Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
    2017-09-08 10:10:30,519 WARNING [cinder.volume.drivers.netapp.utils] It is not the recommended way to use drivers by NetApp. Please use NetAppDriver to achieve the functionality.

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/home/eharney/src/cinder-review5/.tox/py27/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
        return func(*args, **keywargs)
      File "cinder/tests/unit/volume/drivers/netapp/dataontap/test_block_base.py", line 425, in test_build_initiator_target_map_with_lookup_service
        self.assertDictEqual(fake.FC_I_T_MAP, init_targ_map)
      File "/home/eharney/src/cinder-review5/.tox/py27/lib/python2.7/site-packages/unittest2/case.py", line 1128, in assertDictEqual
        self.fail(self._formatMessage(msg, standardMsg))
      File "/home/eharney/src/cinder-review5/.tox/py27/lib/python2.7/site-packages/unittest2/case.py", line 690, in fail
        raise self.failureException(msg)
    AssertionError: {'210[21 chars]a098280feeba5', '500a098290feeba5'], '21000024[47 chars]a5']} != {'210[21 chars]a098290feeba5', '500a098280feeba5'], '21000024[47 chars]a5']}
    - {'21000024ff406cc2': ['500a098190feeba5', '500a098180feeba5'],
    - '21000024ff406cc3': ['500a098280feeba5', '500a098290feeba5']}
    ? ^ ^ ^ ^ ^

    + {'21000024ff406cc2': ['500a098180feeba5', '500a098190feeba5'],
    ? ^ ^ ^ ^ ^

    + '21000024ff406cc3': ['500a098290feeba5', '500a098280feeba5']}

cinder.tests.unit.volume.drivers.netapp.dataontap.performance.test_perf_cmode.PerformanceCmodeLibraryTestCase.test_update_performance_cache
-------------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "cinder/tests/unit/volume/drivers/netapp/dataontap/performance/test_perf_cmode.py", line 164, in test_update_performance_cache
        self.perf_library.performance_counters)
      File "/home/eharney/src/cinder-review5/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
        self.assertThat(observed, matcher, message)
      File "/home/eharney/src/cinder-review5/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
        raise mismatch_error
    testtools.matchers._impl.MismatchError: !=:
    reference = {'node1': [12, 13, 14, 15, 16, 17, 18, 19, 20, 21],
     'node2': [22, 23, 24, 25, 26, 27, 28, 29, 30, 31]}
    actual = {'node1': [12, 13, 14, 15, 16, 17, 18, 19, 20, 31],
     'node2': [22, 23, 24, 25, 26, 27, 28, 29, 30, 21]}

cinder.tests.unit.volume.drivers.netapp.eseries.test_library.NetAppEseriesLibraryTestCase.test_build_initiator_target_map_fc_with_lookup_service
------------------------------------------------------------------------------------------------------------------------------------------------

Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
    2017-09-08 10:10:41,321 WARNING [cinder.volume.drivers.netapp.eseries.library] Production use of "NetApp_ESeries" backend requires the Cinder controller to have multipathing properly set up and the configuration option "use_multipath_for_image_xfer" to be set to "True".
    2017-09-08 10:10:41,321 INFO [cinder.volume.drivers.netapp.eseries.library] System 1fa6efb5-f07b-4de4-9f0e-52e5f7ff5d1b has needsattn status.
    2017-09-08 10:10:41,322 WARNING [oslo_config.cfg] Config option None.netapp_storage_pools is deprecated. Use option backend1.netapp_pool_name_search_pattern instead.

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "cinder/tests/unit/volume/drivers/netapp/eseries/test_library.py", line 987, in test_build_initiator_target_map_fc_with_lookup_service
        self.assertDictEqual(eseries_fake.FC_I_T_MAP, initiator_target_map)
      File "/home/eharney/src/cinder-review5/.tox/py27/lib/python2.7/site-packages/unittest2/case.py", line 1128, in assertDictEqual
        self.fail(self._formatMessage(msg, standardMsg))
      File "/home/eharney/src/cinder-review5/.tox/py27/lib/python2.7/site-packages/unittest2/case.py", line 690, in fail
        raise self.failureException(msg)
    AssertionError: {'201[21 chars]a098190feeba5', '500a098180feeba5'], '20230080[47 chars]a5']} != {'201[21 chars]a098180feeba5', '500a098190feeba5'], '20230080[47 chars]a5']}
    - {'20130080E5322230': ['500a098190feeba5', '500a098180feeba5'],
    - '20230080E5322230': ['500a098280feeba5', '500a098290feeba5']}
    ? ^ ^ ^ ^ ^

    + {'20130080E5322230': ['500a098180feeba5', '500a098190feeba5'],
    ? ^ ^ ^ ^ ^

    + '20230080E5322230': ['500a098290feeba5', '500a098280feeba5']}

======
Totals
======
Ran: 1564 tests in 39.0000 sec.
 - Passed: 1560
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 4
Sum of execute time for each test: 75.5792 sec.

Tags: drivers netapp
Eric Harney (eharney)
Changed in cinder:
assignee: nobody → NetApp (netapp)
Changed in cinder:
assignee: NetApp (netapp) → Goutham Pacha Ravi (gouthamr)
Revision history for this message
Jay Bryant (jsbryant) wrote :

Goutham,

Are you working on this? Any chance this might get some attention soon?

Thanks!

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/545949

Changed in cinder:
assignee: Goutham Pacha Ravi (gouthamr) → Eric Harney (eharney)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to cinder (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/545950

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/545949
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=3b13d77a263d787fce7d6e627ceda9e38fde2236
Submitter: Zuul
Branch: master

commit 3b13d77a263d787fce7d6e627ceda9e38fde2236
Author: Eric Harney <email address hidden>
Date: Mon Feb 19 10:15:57 2018 -0500

    Tests: Fix NetApp tests with hash randomization

    Values in initiator target maps are ordered randomly depending
    on the hash seed used -- account for this in unit tests.

    Partial-Bug: #1715915

    Change-Id: I599b585444cbd51182d9f17fddd97bd91637e1e3

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to cinder (master)

Reviewed: https://review.openstack.org/545950
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=9eeda1db928bab477cd7fb8435fa217ad1e49d1c
Submitter: Zuul
Branch: master

commit 9eeda1db928bab477cd7fb8435fa217ad1e49d1c
Author: Eric Harney <email address hidden>
Date: Mon Feb 19 12:47:52 2018 -0500

    Tests: Disable 2 netapp unit tests

    These tests do not pass with python
    hash randomization enabled, on py2 or py3.

    Disable them so we can enable hash randomization
    for all other tests.

    Change-Id: I338a55d7df298c00c365be6e523ca63a815d8250
    Related-Bug: #1715915

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/465636
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=52965d8974ca62bfaa4201d3057a0b5742e5c818
Submitter: Zuul
Branch: master

commit 52965d8974ca62bfaa4201d3057a0b5742e5c818
Author: Eric Harney <email address hidden>
Date: Wed May 17 11:29:58 2017 -0400

    Enable hash randomization in unit tests

    This is needed to ensure that unit tests match how
    deployed environments run.

    Related-Bug: #1348818
    Related-Bug: #1658725
    Related-Bug: #1715915

    Depends-On: I91a53fdef98c9a14ccc536403063e228e99d6971
    Depends-On: Ie548c90f5657e5270fd966207f453d5211e21867
    Depends-On: I754d48d8bc4081b96fdf90465c4897d0fadd6262
    Depends-On: Ia8762aff6fd9954d350310b4c68ee4fe94de68bf
    Depends-On: I6fe0ae5da270d9c77cb248551dcb32a5c063977a
    Depends-On: I96b9b413387f2ef02ea7ccfb6e676cad0b87c6f1
    Depends-On: Icdde5e3f213adfe4893d1b9e8f2bfe4f61704037
    Depends-On: I29a874ebe0ebaeca58b428faa0bab50482baf6f1
    Depends-On: I30817ae767e08c45f9540b19e922864fe2f7a9c2
    Depends-On: I281ad59fa518f936aa8d1b518acf61800ba949f5
    Depends-On: I0abd0415306c0c0544eb5d9a6ed08640ff6bb0cf
    Depends-On: I599b585444cbd51182d9f17fddd97bd91637e1e3
    Depends-On: I6e4093ca8e79128ed23a4d5068e5d2870e16332c
    Depends-On: I574dca83a971e746a54895a87e394befda0ccdcf
    Depends-On: Id5a961c66a62e289a96daa7437374768ef407bb6
    Depends-On: Ibc6c20e6ffce58c6bbef3155f73e54909d76e132
    Change-Id: I4c7396f9d8e58ac8420754503f8ed60cd7473426

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.