Redeployment failed on tasks[system_provision]

Another failure
Scenario:
            1. Check mcollective version on bootstrap
            2. Create cluster
            3. Add one node to cluster
            4. Provision nodes
            5. Check mcollective version on node

Another failure, scenario
            1. Check mcollective version on bootstrap
            2. Create cluster
            3. Add one node to cluster
            4. Provision nodes <<< failed here
Version: 10.0 ISO #1472

So the root cause is that when we run a transaction we do not filter nodes per-graph, but rather per-transaction.

See

4344 2017-03-14 22:11:48.516 INFO [7fa84a51f880] (manager) Start new transaction: cluster=1 graphs=[{u'type': u'net-verification'}, {u'type': u'deletion'}, {u'type': u'prov ision'}, {u'type': u'default'}] dry_run=0 noop_run=False force=0
4345 2017-03-14 22:11:48.535 DEBUG [7fa84a51f880] (task) Mark task as deleted: 52547e48-8e1a-4e4c-84ec-bfddc3d3359b
4346 2017-03-14 22:11:48.536 DEBUG [7fa84a51f880] (task) Updating task: 52547e48-8e1a-4e4c-84ec-bfddc3d3359b
4347 2017-03-14 22:11:48.578 DEBUG [7fa84a51f880] (manager) Transaction 10 starts assembling.
4348 2017-03-14 22:11:48.579 DEBUG [7fa84a51f880] (manager) Transaction 10 finish assembling.
4349 2017-03-14 22:11:48.579 DEBUG [7fa84a51f880] (mule) MULE STARTING for TransactionsManager._execute_async
4350 2017-03-14 22:11:48.585 DEBUG [7fa84a51f880] (manager) Transaction 10 starts assembling.
4351 [pid: 18806|app: 0|req: 91/456] 10.109.0.1 () {40 vars in 601 bytes} [Tue Mar 14 22:11:48 2017] PUT /api/clusters/1/changes/ => generated 276 bytes in 217 msecs (HTTP/ 1.1 202) 4 headers in 191 bytes (2 switches on core 0)
4352 2017-03-14 22:11:48.612 WARNING [7fa84a51f880] (deployment_graph) Graph association with type 'net-verification' was requested for the unappropriated model instance <n ailgun.db.sqlalchemy.models.cluster.Cluster object at 0x7fa82c035c50> with ID=1
4353 2017-03-14 22:11:48.614 DEBUG [7fa84a51f880] (manager) applying nodes filter: $.pending_addition

We filter nodes only once here: for net-verification, but not for other graphs. Thus we run provision graph on already provisioned nodes.

Okay, the RC is different:

we have Node.reset_to_discover call in stop_deployment_resp method of nailgun/rpc/receiver.py code. When astute sends the report for stop deployment, it sends these nodes in rpc message body. And this method marks nodes as 'discover' and pending_addition. That's why provision graph picks them for reprovisioning.

Raising to critical - it affects big amount of swarm threads

Fix proposed to branch: master
Review: https://review.openstack.org/447083

Reviewed: https://review.openstack.org/447083
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=496212798efa9f167de20a3ea0c2146658b2b466
Submitter: Jenkins
Branch: master

commit 496212798efa9f167de20a3ea0c2146658b2b466
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Fri Mar 17 20:48:52 2017 +0300

    Fix wrong ready status instead of stopped for stop deployment

    Report ready status for node means successful node status
    which can be get if all tasks was passed with ready and skipped
    statuses.

    Same effect can be get if Astute mark node as skipped. In this
    case we also get equal status 'successful'.

    So we need ask node about skipped statuses before ask it about
    successful status to prevent losing context about stop
    deployment operation.

    Change-Id: I3c042425cab800de0bfc4e03f29414b145f44983
    Closes-Bug: #1672964

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/447481

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/447482

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/447483

Reviewed: https://review.openstack.org/447481
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=56ca061009d419a0866d58d5d1aecef45349bdc5
Submitter: Jenkins
Branch: stable/ocata

commit 56ca061009d419a0866d58d5d1aecef45349bdc5
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Fri Mar 17 20:48:52 2017 +0300

    Fix wrong ready status instead of stopped for stop deployment

    Report ready status for node means successful node status
    which can be get if all tasks was passed with ready and skipped
    statuses.

    Same effect can be get if Astute mark node as skipped. In this
    case we also get equal status 'successful'.

    So we need ask node about skipped statuses before ask it about
    successful status to prevent losing context about stop
    deployment operation.

    Change-Id: I3c042425cab800de0bfc4e03f29414b145f44983
    Closes-Bug: #1672964
    (cherry picked from commit 496212798efa9f167de20a3ea0c2146658b2b466)

Reviewed: https://review.openstack.org/447482
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=8213652e49f0f2c28721dfb9da81ecb69d07f9c9
Submitter: Jenkins
Branch: stable/newton

commit 8213652e49f0f2c28721dfb9da81ecb69d07f9c9
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Fri Mar 17 20:48:52 2017 +0300

    Fix wrong ready status instead of stopped for stop deployment

    Report ready status for node means successful node status
    which can be get if all tasks was passed with ready and skipped
    statuses.

    Same effect can be get if Astute mark node as skipped. In this
    case we also get equal status 'successful'.

    So we need ask node about skipped statuses before ask it about
    successful status to prevent losing context about stop
    deployment operation.

    Change-Id: I3c042425cab800de0bfc4e03f29414b145f44983
    Closes-Bug: #1672964
    (cherry picked from commit 496212798efa9f167de20a3ea0c2146658b2b466)

Fix didn't help:
https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.bvt_ubuntu_bootstrap/216/testReport/(root)/deploy_stop_on_deploying_ubuntu_bootstrap/deploy_stop_on_deploying_ubuntu_bootstrap/
Fuel 10 build #1501

Fix proposed to branch: master
Review: https://review.openstack.org/449247

We reverted this feature. So the bug should disappear.

https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.error_node_reinstallation/224/

2017-03-28 04:57:52 +0000 Puppet (err): /etc/puppet/shell_manifests/system_provision_command.sh: line 5: /usr/bin/provision: No such file or directory

Fix proposed to branch: master
Review: https://review.openstack.org/450788

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/450875

Reviewed: https://review.openstack.org/450788
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=039ce9e0b82a9e24b0f598fbe528b80feb3fdf3c
Submitter: Jenkins
Branch: master

commit 039ce9e0b82a9e24b0f598fbe528b80feb3fdf3c
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Tue Mar 28 16:39:03 2017 +0300

    Do not send data about nodes in case of task deployment

    Nailgun use data about nodes in stop deployment respond
    to reset it to discovory state which is unexpected behavior
    for already provisioned nodes in case of task deployment

    Change-Id: I39de8a8afd627b0bf209d9a7f6ad6e19abd99016
    Partial-Bug: #1672964

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/450882

Reviewed: https://review.openstack.org/450875
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=ee2d95de33842187bf99929af321207011c3284f
Submitter: Jenkins
Branch: stable/newton

commit ee2d95de33842187bf99929af321207011c3284f
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Tue Mar 28 16:39:03 2017 +0300

    Do not send data about nodes in case of task deployment

    Nailgun use data about nodes in stop deployment respond
    to reset it to discovory state which is unexpected behavior
    for already provisioned nodes in case of task deployment

    Change-Id: I39de8a8afd627b0bf209d9a7f6ad6e19abd99016
    Partial-Bug: #1672964
    (cherry picked from commit 039ce9e0b82a9e24b0f598fbe528b80feb3fdf3c)

Passed on Newton:
https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.bvt_ubuntu_bootstrap/224/
https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.error_node_reinstallation/225/

Reviewed: https://review.openstack.org/450882
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=445a16aac208c7b65b82ba66a5a083a882568c2e
Submitter: Jenkins
Branch: stable/ocata

commit 445a16aac208c7b65b82ba66a5a083a882568c2e
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Tue Mar 28 16:39:03 2017 +0300

    Do not send data about nodes in case of task deployment

    Nailgun use data about nodes in stop deployment respond
    to reset it to discovory state which is unexpected behavior
    for already provisioned nodes in case of task deployment

    Change-Id: I39de8a8afd627b0bf209d9a7f6ad6e19abd99016
    Partial-Bug: #1672964
    (cherry picked from commit 039ce9e0b82a9e24b0f598fbe528b80feb3fdf3c)

Reviewed: https://review.openstack.org/449247
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=3fb3b83f4ad1254a9a54d6d81f5c58f33c92c601
Submitter: Jenkins
Branch: master

commit 3fb3b83f4ad1254a9a54d6d81f5c58f33c92c601
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Thu Mar 23 20:33:52 2017 +0300

    Excluding number of nodes in stop operation notification

    Nailgun use block of nodes in stop operation to reset
    such nodes in discovery state. Also Nailgun used such
    data to calculate count of nodes for notifications.
    But Astute will not send info about nodes in case
    of task deployment.

    This patch exclude count of nodes in stop notification
    to prevent misslining message about successful operation
    for 0 nodes

    Change-Id: I32da2ccce11b22378f58759703fc4a56e31fd993
    Closes-Bug: #1672964

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/460132

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/460134

Reviewed: https://review.openstack.org/460134
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=d60e2ac26753ef3130ee8b68199016d7c7b8dec6
Submitter: Jenkins
Branch: stable/newton

commit d60e2ac26753ef3130ee8b68199016d7c7b8dec6
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Thu Mar 23 20:33:52 2017 +0300

    Excluding number of nodes in stop operation notification

    Nailgun use block of nodes in stop operation to reset
    such nodes in discovery state. Also Nailgun used such
    data to calculate count of nodes for notifications.
    But Astute will not send info about nodes in case
    of task deployment.

    This patch exclude count of nodes in stop notification
    to prevent misslining message about successful operation
    for 0 nodes

    Change-Id: I32da2ccce11b22378f58759703fc4a56e31fd993
    Closes-Bug: #1672964
    (cherry picked from commit 3fb3b83f4ad1254a9a54d6d81f5c58f33c92c601)

Change abandoned by Joshua Hesketh (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/447483
Reason: This branch (stable/mitaka) is at End Of Life

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/ocata
Review: https://review.opendev.org/460132
Reason: This repo is retired now, no further work will get merged.

This issue was fixed in the openstack/fuel-astute ocata-eol release.