rabbitmq erlang cookie (and other secrets) changes between minor updates

I forgot to post the config download config link: http://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b91/707695/29/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/b913f14/logs/undercloud/var/lib/mistral/config-download-latest/Controller/config_settings.yaml

So the first thing that we need to understand IMO is the following:
https://eed4e0e2e2fdeb21eb4f-a68d87ea4094260bf037e32b34176a60.ssl.cf1.rackcdn.com/359060/58/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/dbe49f1/logs/subnode-1/var/log/containers/stdouts/rabbitmq_bootstrap.log

The cookie seems to change during subsequent runs in the rabbitmq_bootstrap container.

2020-03-25T04:11:56.961337819+00:00 stderr F ++ echo tVJ1oCq5BczpxH7EJzCt

2020-03-25T04:37:11.905973470+00:00 stderr F ++ echo H6FcJlGgFDcEibgWqggL

Now from the inspect on podman (which is taken after the failure): https://eed4e0e2e2fdeb21eb4f-a68d87ea4094260bf037e32b34176a60.ssl.cf1.rackcdn.com/359060/58/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/dbe49f1/logs/subnode-1/var/log/extra/podman/containers/rabbitmq_bootstrap/podman_info.log we see:
                "RABBITMQ_CLUSTER_COOKIE=H6FcJlGgFDcEibgWqggL",

Which makes sense so far.

So one of the questions becomes why would the RABBITMQ_CLUSTER_COOKIE change at all between those two runs?

In hiera we have:
    "rabbitmq::erlang_cookie": "H6FcJlGgFDcEibgWqggL",

We can see that the cookie has changed over time via puppet also in the journal:
Mar 25 04:10:53 centos-8-rax-iad-0015448418 puppet-user[22779]: Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/File[/var/lib/rabbitmq/.erlang.cookie]/content: content changed '{md5}0f117457d3c5c59b6f1439623b7c78b8' to '{md5}f6e79c1383666880d3636409ebc260a3'
Mar 25 04:36:31 centos-8-rax-iad-0015448418 puppet-user[123010]: Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Rabbitmq_bundle/File[/var/lib/rabbitmq/.erlang.cookie]/content: content changed '{md5}f6e79c1383666880d3636409ebc260a3' to '{md5}4f94dd5fd87baa33c55a228fe5927376'

So if the same puppet code changed the cookie, once at 04:10 and then at 04:36 I am inclined to think that the hiera key 'rabbitmq::erlang_cookie' changed over time?

Might be fixing it https://review.opendev.org/359060

interestingly, the same can be observed for

mysql::server::root_password: LWOLbgZxfy
vs
"DB_ROOT_PASSWORD": "0Ye18BSoEJ"

http://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b91/707695/29/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/b913f14/logs/undercloud/var/lib/mistral/config-download-latest/Controller/config_settings.yaml

Fix proposed to branch: master
Review: https://review.opendev.org/715251

Reviewed: https://review.opendev.org/715251
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=a56157f64b7455dbae4118f669e476c1cbd929a5
Submitter: Zuul
Branch: master

commit a56157f64b7455dbae4118f669e476c1cbd929a5
Author: Rabi Mishra <email address hidden>
Date: Thu Mar 26 21:10:16 2020 +0530

    Don't rotate passwords when updating plan

    If the passwods exist we should not update those when updating
    plan.

    Change-Id: Iaa7b3d3e24f05bdb66a0aba42ff3be897500735c
    Closes-Bug: #1869010

This issue was fixed in the openstack/tripleo-ansible 1.3.0 release.