tripleo

docker-puppet tasks can only run on the primary controller role

Bug #1792613 reported by Steven Hardy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Steven Hardy
tripleo stein-3

Bug Description

We calculate a global bootstrap_node_id for the docker-puppet tasks:

https://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps.j2#L283

This is a problem because if you move e.g Keystone to some role that is not the role tagged primary/controller we fail to run the docker-puppet tasks.

In the past we added per-service bootstrap node IDs for composable HA to avoid similar problems, so we need the docker-puppet bootstrapping/tasks to consider the same data - that should ensure compatibility with composable roles/services.

More information in this downstream bz: https://bugzilla.redhat.com/show_bug.cgi?id=1626140

Steven Hardy (shardy)
Changed in tripleo:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Steven Hardy (shardy)
milestone: none → stein-2
Bogdan Dobrelya (bogdando)
tags: added: tech-debt
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (master)

Fix proposed to branch: master
Review: https://review.openstack.org/605046

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/605430

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/605491

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/605492

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/605682

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (master)

Fix proposed to branch: master
Review: https://review.openstack.org/605728

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (master)

Reviewed: https://review.openstack.org/605046
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=1317799e84eae26f21b0d5c5e446db0b144768f1
Submitter: Zuul
Branch: master

commit 1317799e84eae26f21b0d5c5e446db0b144768f1
Author: Steven Hardy <email address hidden>
Date: Fri Oct 12 09:58:16 2018 +0100

    Add AllNodesConfig to config-download group vars

    Since I7b865bf82520006eef3ac2f36df34b1f3c34e642 added an AllNodesConfig
    output to the overcloud, we can consume that in the group_vars
    and add it to the main overcloud group which means the data will be
    available to all overcloud nodes.

    This may be helpful to solve issues resolving per-service bootstrap
    nodes, and also the data there may be more generally useful in
    ansible tasks.

    Partial-Bug: #1792613
    Depends-On: I7b865bf82520006eef3ac2f36df34b1f3c34e642
    Change-Id: Idcee177b21e85cff9e0bf10f4c43c71eff9364ec

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (master)

Reviewed: https://review.openstack.org/605728
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=9cde9139c4f2a59e6fca765e87ffce42cf7da696
Submitter: Zuul
Branch: master

commit 9cde9139c4f2a59e6fca765e87ffce42cf7da696
Author: Steven Hardy <email address hidden>
Date: Thu Sep 27 14:57:23 2018 +0100

    Replace bootstrap_nodeid with SERVICE_short_bootstrap_node_name

    This solves the problem that bootstrap_nodeid, which is set to the
    first node in each role via t-h-t, can match potentially more than
    one node - e.g in the event that a service is deployed such that it
    spans more than one role.

    The SERVICE_short_bootstrap_node_name is automatically generated
    based on the composable service template service_name, and this
    considers all roles where the service is enabled, e.g it should
    only evaluate true once regardles off the roles where the service
    is enabled.

    Change-Id: I48ec4549552910f3cb8db960b0ff10a6c61b4bb9
    Partial-Bug: #1792613

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/605430
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=52c1641e2c3ad5caeb70fc8a09f29eba6fe5b53d
Submitter: Zuul
Branch: master

commit 52c1641e2c3ad5caeb70fc8a09f29eba6fe5b53d
Author: Steven Hardy <email address hidden>
Date: Wed Sep 26 14:52:14 2018 +0100

    Convert *tasks from bootstrap_nodeid to short_bootstrap_node_name

    The current approach has several disadvantages:
    - Requires shelling out to the hiera CLI, and is coupled to the puppet hieradata
    - The bootstrap_nodeid is only unique per Role, not per service, so if you
      deploy a service spanning more than one role it will evaluate true for
      every role, not only once.

    Instead lets use the per-service short_bootstrap_node_name, which is
    available directly via the ansible inventory now ref
    https://review.openstack.org/#/c/605046/

    This is the first part of a cleanup for inconsistent handling of
    bootstrap node evaluation, triggered by bug #1792613

    Change-Id: Iefe4a37e8ced6f4e9018ae0da00e2349390d4927
    Partial-Bug: #1792613
    Depends-On: Idcee177b21e85cff9e0bf10f4c43c71eff9364ec

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/605491
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b278f6c4760e7803fd243e946233921399b403f5
Submitter: Zuul
Branch: master

commit b278f6c4760e7803fd243e946233921399b403f5
Author: Steven Hardy <email address hidden>
Date: Wed Sep 26 15:24:55 2018 +0100

    Remove unused tls-cert-inject.yaml template

    This is no longer handled as the TLS handling tasks were converted
    to ansible, and in the context of this series we need to remove it
    because it references bootstrap_nodeid

    Partial-Bug: #1792613
    Change-Id: Ib32177b116f148f007574847320566e32240cf96

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/605492
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a77d045663d01aaeaf1f49d2ada2a342ad6289c3
Submitter: Zuul
Branch: master

commit a77d045663d01aaeaf1f49d2ada2a342ad6289c3
Author: Steven Hardy <email address hidden>
Date: Wed Sep 26 16:34:33 2018 +0100

    Add SERVICE_bootstrap_node_ip values to allNodesConfig

    This can be used to replace the per-role bootstrap_nodeid_ip,
    and the redis-base template is updated to use the new hiera
    key.

    The old bootstrap_nodeid_ip appears to only be used for redis,
    so the old key is removed, with an upgrade release note added
    should any out-of-tree services reference this value.

    Partial-Bug: #1792613
    Change-Id: I830d5b9bae3e9d65c2c393e3dcdf70bffdb1ac7b

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/605682
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=21905f75886aa516e0952668f49178d3612a1e0f
Submitter: Zuul
Branch: master

commit 21905f75886aa516e0952668f49178d3612a1e0f
Author: Steven Hardy <email address hidden>
Date: Thu Sep 27 10:55:00 2018 +0100

    Remove common bootstrap_nodeid from deploy_steps/tripleo-packages.yaml

    There was also a special flag for FFU that triggered repo setup only
    on the bootstrap node, so switch this to use the per-service bootstrap
    name instead.

    Change-Id: I32f963a002399af4911acbf507312f378aac3599
    Partial-Bug: #1792613

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/625078

Emilien Macchi (emilienm)
Changed in tripleo:
milestone: stein-2 → stein-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/632711

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/632714

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/rocky)

Reviewed: https://review.openstack.org/632711
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=5defaf300fb0f59231062066ae1f0360748ab156
Submitter: Zuul
Branch: stable/rocky

commit 5defaf300fb0f59231062066ae1f0360748ab156
Author: Steven Hardy <email address hidden>
Date: Fri Oct 12 09:58:16 2018 +0100

    Add AllNodesConfig to config-download group vars

    Since I7b865bf82520006eef3ac2f36df34b1f3c34e642 added an AllNodesConfig
    output to the overcloud, we can consume that in the group_vars
    and add it to the main overcloud group which means the data will be
    available to all overcloud nodes.

    This may be helpful to solve issues resolving per-service bootstrap
    nodes, and also the data there may be more generally useful in
    ansible tasks.

    Partial-Bug: #1792613
    Depends-On: I7b865bf82520006eef3ac2f36df34b1f3c34e642
    Change-Id: Idcee177b21e85cff9e0bf10f4c43c71eff9364ec

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/625078
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a0a09d29aac1ad44fa48dae5d1db72e3905aeb69
Submitter: Zuul
Branch: master

commit a0a09d29aac1ad44fa48dae5d1db72e3905aeb69
Author: Steven Hardy <email address hidden>
Date: Thu Dec 13 17:56:44 2018 +0000

    run docker_puppet_tasks on any role

    Currently this assumes all tasks will run on the primary controller
    but because of composable roles, that may not be the case.

    An example is if you deploy keystone on any role other than the
    role tagged primary e.g Controller by default, we don't create
    any of the users/endpoints because the tasks aren't written to
    the role unless keystone actually runs there.

    Closes-Bug: #1792613
    Change-Id: Ib6efd03584c95ed4ab997f614aa3178b01877b8c

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.openstack.org/632714
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=87b16ddbfc19263e81bf49a5c643855c98ffb240
Submitter: Zuul
Branch: stable/rocky

commit 87b16ddbfc19263e81bf49a5c643855c98ffb240
Author: Steven Hardy <email address hidden>
Date: Thu Dec 13 17:56:44 2018 +0000

    run docker_puppet_tasks on any role

    Currently this assumes all tasks will run on the primary controller
    but because of composable roles, that may not be the case.

    An example is if you deploy keystone on any role other than the
    role tagged primary e.g Controller by default, we don't create
    any of the users/endpoints because the tasks aren't written to
    the role unless keystone actually runs there.

    Conflicts:
     common/deploy-steps-tasks.yaml

    Closes-Bug: #1792613
    Change-Id: Ib6efd03584c95ed4ab997f614aa3178b01877b8c
    (cherry picked from commit a0a09d29aac1ad44fa48dae5d1db72e3905aeb69)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 10.4.0

This issue was fixed in the openstack/tripleo-heat-templates 10.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 9.3.0

This issue was fixed in the openstack/tripleo-heat-templates 9.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/666660

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (master)

Fix proposed to branch: master
Review: https://review.opendev.org/666661

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/666667

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/666672

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/666688

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (stable/queens)

Reviewed: https://review.opendev.org/666667
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=8a30b6a9e10983476e26ff6d99670fb20293547d
Submitter: Zuul
Branch: stable/queens

commit 8a30b6a9e10983476e26ff6d99670fb20293547d
Author: Steven Hardy <email address hidden>
Date: Thu Sep 27 14:57:23 2018 +0100

    Replace bootstrap_nodeid with SERVICE_short_bootstrap_node_name

    This solves the problem that bootstrap_nodeid, which is set to the
    first node in each role via t-h-t, can match potentially more than
    one node - e.g in the event that a service is deployed such that it
    spans more than one role.

    The SERVICE_short_bootstrap_node_name is automatically generated
    based on the composable service template service_name, and this
    considers all roles where the service is enabled, e.g it should
    only evaluate true once regardles off the roles where the service
    is enabled.

    Change-Id: I48ec4549552910f3cb8db960b0ff10a6c61b4bb9
    Partial-Bug: #1792613
    (cherry picked from commit 9cde9139c4f2a59e6fca765e87ffce42cf7da696)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (stable/rocky)

Reviewed: https://review.opendev.org/666660
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=f83d957d91081357c5e59ad05840bb162157625d
Submitter: Zuul
Branch: stable/rocky

commit f83d957d91081357c5e59ad05840bb162157625d
Author: Steven Hardy <email address hidden>
Date: Thu Sep 27 14:57:23 2018 +0100

    Replace bootstrap_nodeid with SERVICE_short_bootstrap_node_name

    This solves the problem that bootstrap_nodeid, which is set to the
    first node in each role via t-h-t, can match potentially more than
    one node - e.g in the event that a service is deployed such that it
    spans more than one role.

    The SERVICE_short_bootstrap_node_name is automatically generated
    based on the composable service template service_name, and this
    considers all roles where the service is enabled, e.g it should
    only evaluate true once regardles off the roles where the service
    is enabled.

    Change-Id: I48ec4549552910f3cb8db960b0ff10a6c61b4bb9
    Partial-Bug: #1792613
    (cherry picked from commit 9cde9139c4f2a59e6fca765e87ffce42cf7da696)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (master)

Reviewed: https://review.opendev.org/666661
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=31e5cc36fdb91a83a76ed0c7cf32949e464222cb
Submitter: Zuul
Branch: master

commit 31e5cc36fdb91a83a76ed0c7cf32949e464222cb
Author: Emilien Macchi <email address hidden>
Date: Thu Jun 20 15:23:39 2019 -0400

    Fix bootstrap node for nova conductor

    The doc was updated but the actual default value still the one.
    See context in I48ec4549552910f3cb8db960b0ff10a6c61b4bb9

    Partial-Bug: #1792613
    Change-Id: I8e0f4bd90d22617a6451fa591f9ff50821d9c33b

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (stable/rocky)

Reviewed: https://review.opendev.org/666672
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=c590802947eea9ddf4421590d6aa7ee9a8a757f9
Submitter: Zuul
Branch: stable/rocky

commit c590802947eea9ddf4421590d6aa7ee9a8a757f9
Author: Emilien Macchi <email address hidden>
Date: Thu Jun 20 15:23:39 2019 -0400

    Fix bootstrap node for nova conductor

    The doc was updated but the actual default value still the one.
    See context in I48ec4549552910f3cb8db960b0ff10a6c61b4bb9

    Partial-Bug: #1792613
    Change-Id: I8e0f4bd90d22617a6451fa591f9ff50821d9c33b
    (cherry picked from commit 31e5cc36fdb91a83a76ed0c7cf32949e464222cb)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (stable/queens)

Reviewed: https://review.opendev.org/666688
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=02660341f3c7389fb740a6a844858ea73d2f34d3
Submitter: Zuul
Branch: stable/queens

commit 02660341f3c7389fb740a6a844858ea73d2f34d3
Author: Emilien Macchi <email address hidden>
Date: Thu Jun 20 15:23:39 2019 -0400

    Fix bootstrap node for nova conductor

    The doc was updated but the actual default value still the one.
    See context in I48ec4549552910f3cb8db960b0ff10a6c61b4bb9

    Partial-Bug: #1792613
    Change-Id: I8e0f4bd90d22617a6451fa591f9ff50821d9c33b
    (cherry picked from commit 31e5cc36fdb91a83a76ed0c7cf32949e464222cb)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.