Fuel for OpenStack

[9.2] community ISO fails BVT due to problems on the master node

Bug #1655922 reported by Denis Meltsaykin on 2017-01-12

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Committed	High	Vladimir Kozhukalov	Fuel for OpenStack 9.2
	Mitaka	Confirmed	High	Vladimir Kozhukalov	Fuel for OpenStack 9.x-updates

Bug Description

Since the build #802 of https://ci.fuel-infra.org/job/9.0-community.main.ubuntu.bvt_2/ the tests are failed with the following error:

======================================================================
ERROR: Create environment and set up master node
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/9.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 120, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0-community.main.ubuntu.bvt_2/fuelweb_test/tests/base_test_case.py", line 399, in setup_master
    self.env.setup_environment()
  File "/home/jenkins/workspace/9.0-community.main.ubuntu.bvt_2/fuelweb_test/models/environment.py", line 441, in setup_environment
    provision_admin(admin)
  File "/home/jenkins/workspace/9.0-community.main.ubuntu.bvt_2/fuelweb_test/models/environment.py", line 433, in provision_admin
    self.wait_ssh_available()
  File "/home/jenkins/workspace/9.0-community.main.ubuntu.bvt_2/core/helpers/log_helpers.py", line 215, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0-community.main.ubuntu.bvt_2/fuelweb_test/models/environment.py", line 503, in wait_ssh_available
    ('admin'), 22), timeout=timeout)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 126, in wait_pass
    return raising_predicate()
  File "/home/jenkins/workspace/9.0-community.main.ubuntu.bvt_2/fuelweb_test/models/environment.py", line 503, in <lambda>
    ('admin'), 22), timeout=timeout)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 67, in tcp_ping_
    s.connect((str(host), int(port)))
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 113] No route to host

----------------------------------------------------------------------

It occurs early on the master node setup stage, so there is no diagnostic snapshot available.

Tags:

Revision history for this message

Dmitry Teselkin (teselkin-d) wrote on 2017-01-13:

It's not a duplicate of https://bugs.launchpad.net/fuel/+bug/1587411

Jobs #802 - #815 [1] failed because of different reason - master node couldn't be deployed from ISO because package logrotate couldn't be installed. The package exists on ISO in 'extra/proposed' repo, but installed didn't see it.

The last ISO that contains logrotate in 'Packages' directory is [2], here is a part of its build log
-----
...
08:01:10 2017-01-10 08:01:10 URL:http://mirror.seed-cz1.fuel-infra.org/pkgs/snapshots/centos-7.2.1511-2016-12-12-030000/updates/x86_64//Packages/logrotate-3.8.6-7.el7_2.x86_64.rpm [67276/67276] -> "/home/jenkins/workspace/tmp/9.0-community.all/local_mirror/centos/os/x86_64/Packages/logrotate-3.8.6-7.el7_2.x86_64.rpm" [1]
...
08:02:09 [proposed: 32 of 169 ] Downloading Packages/logrotate-3.8.6-7.el7~mos1.x86_64.rpm
08:02:09
logrotate-3.8.6-7.el7~mos1.x86_64.rpm | 66 kB 00:00
...
-----

Next ISO [3] doesn't contain such string. It contains only the following
-----
...
14:02:09 [proposed: 32 of 169 ] Downloading Packages/logrotate-3.8.6-7.el7~mos2.x86_64.rpm
14:02:09
logrotate-3.8.6-7.el7~mos2.x86_64.rpm | 66 kB 00:00
...
-----

It seems that our ISO build procedure is broken.

[1] https://ci.fuel-infra.org/job/9.0-community.main.ubuntu.bvt_2/
[2] https://ci.fuel-infra.org/job/9.0-community.all/5153/
[3] https://ci.fuel-infra.org/job/9.0-community.all/5154/

Revision history for this message

Alexander Kurenyshev (akurenyshev) wrote on 2017-01-13:

If CI is broken this in not a problem of fuel qa

Changed in fuel:
assignee:	Fuel QA Team (fuel-qa) → Fuel CI (fuel-ci)

Revision history for this message

Roman Vyalov (r0mikiam) wrote on 2017-01-13:

the problem not related to CI, because this job did not changed during half an year .
and we found the problem 3 days ago

Changed in fuel:
assignee:	Fuel CI (fuel-ci) → MOS Linux (mos-linux)

Revision history for this message

Dmitry Teselkin (teselkin-d) wrote on 2017-01-13:

May I know the reason it's a task for MOS Linux please?

Changed in fuel:
assignee:	MOS Linux (mos-linux) → Roman Vyalov (r0mikiam)

Revision history for this message

Roman Vyalov (r0mikiam) wrote on 2017-01-13:

I dont understand why our build process is broken ? and how the new version of logrotate was broke the master node?

Changed in fuel:
assignee:	Roman Vyalov (r0mikiam) → Dmitry Teselkin (teselkin-d)

Revision history for this message

Dmitry Teselkin (teselkin-d) wrote on 2017-01-13:

Screenshot1.png Edit (35.5 KiB, image/png)

Revision history for this message

Dmitry Teselkin (teselkin-d) wrote on 2017-01-13:

Screenshot2.png Edit (29.3 KiB, image/png)

Revision history for this message

Dmitry Teselkin (teselkin-d) wrote on 2017-01-13:

The root cause is the following - logrotate can't be installed because kickstart file doesn't use additional repos available on ISO (extra-repos/proposed, mos-centos) where the package is located.

Changed in fuel:
assignee:	Dmitry Teselkin (teselkin-d) → Fuel Sustaining (fuel-sustaining-team)

Vladimir Kozhukalov (kozhukalov) on 2017-01-13

Changed in fuel:
assignee:	Fuel Sustaining (fuel-sustaining-team) → Vladimir Kozhukalov (kozhukalov)

Revision history for this message

Vladimir Kozhukalov (kozhukalov) wrote on 2017-01-13:

Looks like the isssue is that version of keystone that we have on jenkins slaves on community CI is incompatible with current fuel-qa. Please compare `pip freeze` output in /home/jenkins/venv-nailgun-tests-2.9 virtualenv on ci-slave01.fuel-infra.org (community-ci where we have problems) and on srv114-bud.infra.mirantis.net (product-ci where everything is ok).

Changed in fuel:
assignee:	Vladimir Kozhukalov (kozhukalov) → Roman Vyalov (r0mikiam)

Dmitry Kaigarodеsev (dkaiharodsev) on 2017-01-16

Changed in fuel:
assignee:	Roman Vyalov (r0mikiam) → Dmitry Kaigarodеsev (dkaiharodsev)

Revision history for this message

Dmitry Kaigarodеsev (dkaiharodsev) wrote on 2017-01-17:

#10

@Vladimir, regarding keystone here's situation on mentioned slaves:

slave with sucessful bvt:
(venv-nailgun-tests-2.9)root@srv114-bud:~# pip freeze | grep keystone
keystoneauth1==2.14.0
python-keystoneclient==3.6.0

slave on fuel-ci:
(venv-nailgun-tests-2.9)root@ci-slave01:~# pip freeze | grep keystone
keystoneauth1==2.15.0
python-keystoneclient==3.7.0

more detailed 'pip freeze'output may be found below:
srv114-bud(green bvt) http://paste.openstack.org/show/595149/
ci-slave01(red bvt) http://paste.openstack.org/show/595150/

Revision history for this message

Dmitry Kaigarodеsev (dkaiharodsev) wrote on 2017-01-17:

#11

test run of bvt2 with adjusted versions of keystone and python-keystoneclient:
https://ci.fuel-infra.org/job/9.0-community.main.ubuntu.bvt_2/832/console

Changed in fuel:
importance:	Undecided → High
status:	New → Confirmed

Revision history for this message

Dmitry Kaigarodеsev (dkaiharodsev) wrote on 2017-01-17:

#12

run with same venv (http://paste.openstack.org/show/595166/)
https://ci.fuel-infra.org/job/9.0-community.main.ubuntu.bvt_2/835/console

Dmitry Kaigarodеsev (dkaiharodsev) on 2017-01-17

Changed in fuel:
assignee:	Dmitry Kaigarodеsev (dkaiharodsev) → Vladimir Kozhukalov (kozhukalov)

Alexey Shtokolov (ashtokolov) on 2017-01-17

tags:

added: non-release

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-17: Fix proposed to fuel-main (stable/mitaka)

#13

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/421289

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-17: Fix merged to fuel-main (stable/mitaka)

#14

Reviewed: https://review.openstack.org/421289
Committed: https://git.openstack.org/cgit/openstack/fuel-main/commit/?id=642e1c09938c23e349d3d301ae985ba793d91322
Submitter: Jenkins
Branch: stable/mitaka

commit 642e1c09938c23e349d3d301ae985ba793d91322
Author: Vladimir Kozhukalov <email address hidden>
Date: Tue Jan 17 17:19:53 2017 +0300

Use mos and proposed repos during centos install

    Base OS repository that we place on iso is not consistent
    due to some packages that depend on other packages that we
    put into mos end propsed repos.

Change-Id: I7d57afde6f87906e5b22f925595d7f941d72c66a
Closes-Bug: #1655922

tags:

added: in-stable-mitaka

Vladimir Kozhukalov (kozhukalov) on 2017-01-17

Changed in fuel:
status:	Confirmed → Fix Committed
status:	Fix Committed → Confirmed

Roman Vyalov (r0mikiam) on 2017-02-03

Changed in fuel:
status:	Confirmed → Won't Fix

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-14: Fix proposed to fuel-main (stable/newton)

#15

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/433580

Changed in fuel:
status:	Won't Fix → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-14: Fix proposed to fuel-main (master)

#16

Fix proposed to branch: master
Review: https://review.openstack.org/433581

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-14: Fix merged to fuel-main (master)

#17

Reviewed: https://review.openstack.org/433581
Committed: https://git.openstack.org/cgit/openstack/fuel-main/commit/?id=3aaacb211d6726700ca18a4b9bc43309ceeb5f2d
Submitter: Jenkins
Branch: master

commit 3aaacb211d6726700ca18a4b9bc43309ceeb5f2d
Author: Vladimir Kozhukalov <email address hidden>
Date: Tue Jan 17 17:19:53 2017 +0300

Use mos and proposed repos during centos install

    Base OS repository that we place on iso is not consistent
    due to some packages that depend on other packages that we
    put into mos end propsed repos.

    Change-Id: I7d57afde6f87906e5b22f925595d7f941d72c66a
    Closes-Bug: #1655922
    (cherry picked from commit 642e1c09938c23e349d3d301ae985ba793d91322)

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-14: Fix merged to fuel-main (stable/newton)

#18

Reviewed: https://review.openstack.org/433580
Committed: https://git.openstack.org/cgit/openstack/fuel-main/commit/?id=18cc389f57b371a2f57925d2d3dd317eb725930e
Submitter: Jenkins
Branch: stable/newton

commit 18cc389f57b371a2f57925d2d3dd317eb725930e
Author: Vladimir Kozhukalov <email address hidden>
Date: Tue Jan 17 17:19:53 2017 +0300

Use mos and proposed repos during centos install

    Base OS repository that we place on iso is not consistent
    due to some packages that depend on other packages that we
    put into mos end propsed repos.

    Change-Id: I7d57afde6f87906e5b22f925595d7f941d72c66a
    Closes-Bug: #1655922
    (cherry picked from commit 642e1c09938c23e349d3d301ae985ba793d91322)