[10.0] [Tempest] Failed to deploy env because ironic_upload_images task is failed

Bug #1653204 reported by Yury Tregubov
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Vasyl Saienko

Bug Description

Tempest environment with ironic nodes on LVM storage is failed to deploy with below errors in astute logs.

The problem is reproduced on mos 10.0 iso #1135 and #1163. Have no information about previous builds on 10.0. However this problem is not seen on deploy of the similar env with Ceph storage.
At least deploy with Ceph is not stopped due to failed ironic_upload_images task.

Diagnostic snapshot is: https://drive.google.com/open?id=0B1Crk-sAvGanV0dMVk5JaEthQzQ

Environment is available on 172.18.162.74

Astute logs in case of LVM:

2016-12-29 14:12:55 DEBUG [17330] Node 1, task ironic_upload_images, manifest /etc/puppet/shell_manifests/ironic_upload_images_manifest.pp, status: running
2016-12-29 14:12:55 ERROR [17330] Puppet agent took too long to run puppet task. Mark task as failed. Node 1, task ironic_upload_images, manifest /etc/puppet/shell_manifests/ironic_upload_images_manifest.pp
2016-12-29 14:12:55 INFO [17330] Node[1]: Finished task: Task[ironic_upload_images/1] with status: busy
2016-12-29 14:12:55 DEBUG [17330] Node[1]: Decreasing node concurrency to: 0
2016-12-29 14:12:55 DEBUG [17330] Cluster[]: Count failed node 1 for group primary-controller
2016-12-29 14:12:55 INFO [17330] Casting message to Nailgun:
{"method"=>"deploy_resp",
 "args"=>
  {"task_uuid"=>"3f42f677-5866-46ee-be4a-b607e8d54d9d",
   "nodes"=>
    [{"uid"=>"1",
      "progress"=>100,
      "status"=>"error",
      "error_type"=>"deploy",
      "deployment_graph_task_name"=>"ironic_upload_images",
      "task_status"=>"error",
      "summary"=>
       {:time=>{"last_run"=>0},
        :resources=>
         {"changed_resources"=>"",
          "failed_resources"=>"",
          "failed"=>1,
          "changed"=>0,
          "total"=>0,
          "restarted"=>0,
          "out_of_sync"=>0},
        :changes=>nil,
        :events=>nil,
        :version=>nil,
        :raw_report=>nil,
        :status=>"running",
        :running=>1,
        :enabled=>1,
        :idling=>0,
        :stopped=>0,
        :lastrun=>1483020594,
        :runtime=>181,
        :output=>"Currently running; last completed run 181 seconds ago"},
      "error_msg"=>"Task ironic_upload_images failed on node 1"}]}}

Changed in fuel:
milestone: none → 10.1
importance: Undecided → High
Revision history for this message
Yury Tregubov (ytregubov) wrote :

Looks like it is the same as https://bugs.launchpad.net/mos/+bug/1652484

description: updated
Changed in fuel:
assignee: nobody → Fuel Sustaining (fuel-sustaining-team)
status: New → Confirmed
Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :
Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

It seems to be some kind of problems with environment

2016-12-29T13:22:59.288847+00:00 node-1 haproxy[9397]: 10.109.4.3:40034 [29/Dec/2016:13:22:11.085] neutron neutron/node-4 0/0/1/-1/48202 -1 0 - - SD-- 76/0/0/0/0 0/0 "GET /v2.0/networks.json?fields=id&name=baremetal HTTP/1.1"

Actually nothing reached neutron/node-4

Revision history for this message
Yury Tregubov (ytregubov) wrote :

Actually this problem is not reproduced any more on fresh runs like 10.0 mos #1328.
I propose to move it to Invalid so far.

Changed in fuel:
status: Confirmed → Invalid
Revision history for this message
Yury Tregubov (ytregubov) wrote :

Got the fresh reproduction on mos10 #1468 build. Diagnostic snapshot is attached.

Changed in fuel:
status: Invalid → Confirmed
tags: added: blocker-for-qa
Revision history for this message
Yury Tregubov (ytregubov) wrote :

Filed to add the snapshot here.
Uploaded to google drive https://drive.google.com/open?id=0B1Crk-sAvGanREtfOTVKeUxJbUU

Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

After investigating of latest logs it seems that nailgun doesn't trigger creation of ironic images.

https://github.com/openstack/fuel-web/blob/7940d3361badf18a8ac6eae27ad682f8a1d5a613/nailgun/nailgun/orchestrator/tasks_templates.py#L254

tags: added: area-python
tags: added: swarm-blocker
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/447112

Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Vasyl Saienko (vsaienko)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/447122

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/447123

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/447112
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=218e53584eb5ac7906f09a1f15fe7fcc11973fb8
Submitter: Jenkins
Branch: master

commit 218e53584eb5ac7906f09a1f15fe7fcc11973fb8
Author: Vasyl Saienko <email address hidden>
Date: Fri Mar 17 21:38:31 2017 +0200

    Fix building ironic bootstrap

    This patch fixes issues with building ironic bootstrap:
      * fix if condition that is a trigger to build ironic bootstrap
      * build fuel image always, ironic bootstrap only when ironic is enabled
      * Drop incorrect condition check for fuel image

    Change-Id: Ib7ee0c656ffa71ce913e1924bb80260744319a3a
    Closes-Bug: #1653204

Changed in fuel:
status: In Progress → Fix Committed
tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/ocata)

Reviewed: https://review.openstack.org/447122
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=66b19041c941f9f05598bc12cc54bac0a0221a6d
Submitter: Jenkins
Branch: stable/ocata

commit 66b19041c941f9f05598bc12cc54bac0a0221a6d
Author: Vasyl Saienko <email address hidden>
Date: Fri Mar 17 21:38:31 2017 +0200

    Fix building ironic bootstrap

    This patch fixes issues with building ironic bootstrap:
      * fix if condition that is a trigger to build ironic bootstrap
      * build fuel image always, ironic bootstrap only when ironic is enabled
      * Drop incorrect condition check for fuel image

    Change-Id: Ib7ee0c656ffa71ce913e1924bb80260744319a3a
    Closes-Bug: #1653204

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/newton)

Reviewed: https://review.openstack.org/447123
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=c90b8a2fab0b8deab06d75b93394f32c60963be7
Submitter: Jenkins
Branch: stable/newton

commit c90b8a2fab0b8deab06d75b93394f32c60963be7
Author: Vasyl Saienko <email address hidden>
Date: Fri Mar 17 21:38:31 2017 +0200

    Fix building ironic bootstrap

    This patch fixes issues with building ironic bootstrap:
      * fix if condition that is a trigger to build ironic bootstrap
      * build fuel image always, ironic bootstrap only when ironic is enabled
      * Drop incorrect condition check for fuel image

    Change-Id: Ib7ee0c656ffa71ce913e1924bb80260744319a3a
    Closes-Bug: #1653204

tags: added: in-stable-newton
Revision history for this message
Yury Tregubov (ytregubov) wrote :

Verified on mos 10.0 build #1509.

Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library ocata-eol

This issue was fixed in the openstack/fuel-library ocata-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.