Fuel for OpenStack

Nailgun marks node as offline after the node reboot

Bug #1655595 reported by Sergey Novikov on 2017-01-11

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Committed	High	Dmitry Ilyin	Fuel for OpenStack 11.0
Nominated for Ocata by Stanislaw Bogatkin
	Mitaka	Fix Committed	High	Vladimir Sharshov	Fuel for OpenStack 9.x-updates
	Newton	Fix Released	High	Vladimir Sharshov	Fuel for OpenStack 10.1

Bug Description

Detailed bug description:
The issue was found by https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.support_dpdk/168/testReport/(root)/deploy_cluster_with_dpdk_tun/deploy_cluster_with_dpdk_tun/

Steps to reproduce:
            1. Create new environment with VXLAN segmentation for Neutron
            2. Set KVM as Hypervisor
            3. Add controller and compute nodes
            4. Configure HugePages for compute nodes
            5. Configure private network in DPDK mode
            6. Run network verification
            7. Deploy environment
            8. Run network verification
            9. Run OSTF
            10. Reboot compute
            11. Run OSTF
            12. Run instance on compute with DPDK and check its availability
                via floating IP
Expected results: all is fine
Actual result: step #10 fails, node is offline for nailgun, but node is available by ssh/ping. bellow is part of nailgun-agent.log from this node:

I, [2016-12-27T09:53:39.328070 #2914] INFO -- : API URL is https://10.109.36.2:8443/api
/usr/bin/nailgun-agent:338:in `_network': undefined method `gsub' for nil:NilClass (NoMethodError)
        from /usr/bin/nailgun-agent:168:in `initialize'
        from /usr/bin/nailgun-agent:1363:in `new'
        from /usr/bin/nailgun-agent:1363:in `<main>'
at depth 0 - 18: self signed certificate

@Georgy Kibardin found out the reason of this exception is absence of default route:

root@node-2:~# ip route
10.109.5.0/24 dev br-fw-admin proto kernel scope link src 10.109.5.4

https://bugs.launchpad.net/fuel/mitaka/+bug/1612670/comments/11
https://bugs.launchpad.net/fuel/mitaka/+bug/1612670/comments/12

Tags:

Stanislaw Bogatkin (sbogatkin) on 2017-01-11

Changed in fuel:
status:	New → Confirmed
assignee:	nobody → Fuel Sustaining (fuel-sustaining-team)
milestone:	9.2 → 11.0

Dmitry Ilyin (idv1985) on 2017-01-11

Changed in fuel:
assignee:	Fuel Sustaining (fuel-sustaining-team) → Dmitry Ilyin (idv1985)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-11: Fix proposed to fuel-nailgun-agent (master)

Fix proposed to branch: master
Review: https://review.openstack.org/418940

Changed in fuel:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-17: Fix merged to fuel-nailgun-agent (master)

Reviewed: https://review.openstack.org/418940
Committed: https://git.openstack.org/cgit/openstack/fuel-nailgun-agent/commit/?id=5ea2e3cc7504eb8386b266ad1227b6d1b4707e42
Submitter: Jenkins
Branch: master

commit 5ea2e3cc7504eb8386b266ad1227b6d1b4707e42
Author: Dmitry Ilyin <email address hidden>
Date: Wed Jan 11 16:43:29 2017 +0300

Add check if there is no default gateway

Do not try to return the default gateway data
if there is no default gateway.

Change-Id: I68b62b12cce2730ece968cede98b8f1597281fc5
Closes-Bug: 1655595

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-17: Fix proposed to fuel-nailgun-agent (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/421237

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-17: Fix proposed to fuel-nailgun-agent (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/421238

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-17: Fix merged to fuel-nailgun-agent (stable/mitaka)

Reviewed: https://review.openstack.org/421237
Committed: https://git.openstack.org/cgit/openstack/fuel-nailgun-agent/commit/?id=26a7eb39a5105878930502fabd3659b8b4aa66ca
Submitter: Jenkins
Branch: stable/mitaka

commit 26a7eb39a5105878930502fabd3659b8b4aa66ca
Author: Dmitry Ilyin <email address hidden>
Date: Wed Jan 11 16:43:29 2017 +0300

Add check if there is no default gateway

Do not try to return the default gateway data
if there is no default gateway.

    Change-Id: I68b62b12cce2730ece968cede98b8f1597281fc5
    Closes-Bug: 1655595
    (cherry picked from commit 5ea2e3cc7504eb8386b266ad1227b6d1b4707e42)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-17: Fix merged to fuel-nailgun-agent (stable/newton)

Reviewed: https://review.openstack.org/421238
Committed: https://git.openstack.org/cgit/openstack/fuel-nailgun-agent/commit/?id=db0caef9c4db36d8537d0381e83169bf66938d54
Submitter: Jenkins
Branch: stable/newton

commit db0caef9c4db36d8537d0381e83169bf66938d54
Author: Dmitry Ilyin <email address hidden>
Date: Wed Jan 11 16:43:29 2017 +0300

Add check if there is no default gateway

Do not try to return the default gateway data
if there is no default gateway.

    Change-Id: I68b62b12cce2730ece968cede98b8f1597281fc5
    Closes-Bug: 1655595
    (cherry picked from commit 5ea2e3cc7504eb8386b266ad1227b6d1b4707e42)