CentOS-7.2 based ISO often fails with '[Errno 113] No route to - host' when e1000 driver is used

Bug #1526544 reported by Alexander Ignatov
36
This bug affects 7 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Ivan Suzdal
8.0.x
Won't Fix
High
MOS Linux
Mitaka
Fix Released
High
Ivan Suzdal

Bug Description

During the executing 8.0-pkg-systest-ubuntu job for https://review.fuel-infra.org/#/c/14261/ there was deployment error occured

======================================================================
ERROR: Deploy ceph HA with RadosGW for objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
...
  File "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/http.py", line 116, in _get_response
    return self.opener.open(req)
  File "/usr/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 113] No route to host>

in job https://packaging-ci.infra.mirantis.net/job/8.0-pkg-systest-ubuntu/1353/console

UPDATE (2016-03-01)
===================

The root cause here is broken driver for e1000 interface. This issue is not CentOS related, it happened (and still happening) on almost every platform. There are only workarounds for this issue and no *real* solution, like 'fix the driver'.

There is an interesting commit that looks like a solution, it was merged Sep 8, 2015 - https://github.com/torvalds/linux/commit/9eab46b7cb8d0b0dcf014bf7b25e0e72b9e4d929
Unfortunately, it doesn't help.

Workarounds
-----------
* Disable TSO offloading
* Use 'virtio' driver

Related Bugs
------------
kernel - https://bugzilla.kernel.org/show_bug.cgi?id=47331
novell - http://www.novell.com/support/kb/doc.php?id=7000925
redhat - https://access.redhat.com/solutions/25528
redhat - https://bugzilla.redhat.com/show_bug.cgi?id=398921
centos - https://bugs.centos.org/view.php?id=6168

Revision history for this message
Vitaly Gusev (vgusev) wrote :
Changed in fuel:
status: New → Confirmed
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

CI team, could you please take a look?

Thank you!

Changed in fuel:
assignee: MOS QA Team (mos-qa) → Fuel CI (fuel-ci)
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :
Download full text (4.0 KiB)

Comments from the duplicated bug:

https://packaging-ci.infra.mirantis.net/job/8.0-pkg-systest-ubuntu/1384/console this one is failed.

the same ISO https://packaging-ci.infra.mirantis.net/job/8.0-pkg-systest-ubuntu/1387/ - success

Failed tests:

File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
   return self.do_open(httplib.HTTPConnection, req)
 File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
   raise URLError(err)
URLError: <urlopen error [Errno 113] No route to host>

rabbitmq-server is unreachable but still alive according to logs.

Master node has eth stucked.

2015-12-16T07:00:34.358636+00:00 err: [ 2045.553869] e1000 0000:00:03.0 enp0s3: Detected Tx Unit Hang
2015-12-16T07:00:34.358648+00:00 err: [ 2045.553869] Tx Queue <0>
2015-12-16T07:00:34.358650+00:00 err: [ 2045.553869] TDH <3>
2015-12-16T07:00:34.358651+00:00 err: [ 2045.553869] TDT <3>
2015-12-16T07:00:34.358664+00:00 err: [ 2045.553869] next_to_use <de>
2015-12-16T07:00:34.358665+00:00 err: [ 2045.553869] next_to_clean <3>
2015-12-16T07:00:34.358665+00:00 err: [ 2045.553869] buffer_info[next_to_clean]
2015-12-16T07:00:34.358666+00:00 err: [ 2045.553869] time_stamp <1001a8c59>
2015-12-16T07:00:34.358667+00:00 err: [ 2045.553869] next_to_watch <16>
2015-12-16T07:00:34.358667+00:00 err: [ 2045.553869] jiffies <1001aa291>
2015-12-16T07:00:34.358668+00:00 err: [ 2045.553869] next_to_watch.status <0>
2015-12-16T07:00:35.524042+00:00 warning: [ 2046.720054] ------------[ cut here ]------------
2015-12-16T07:00:35.524055+00:00 warning: [ 2046.720084] WARNING: at net/sched/sch_generic.c:297 dev_watchdog+0x270/0x280()
2015-12-16T07:00:35.524058+00:00 info: [ 2046.720100] NETDEV WATCHDOG: enp0s3 (e1000): transmit queue 0 timed out
2015-12-16T07:00:35.526521+00:00 alert: [ 2046.720104] Modules linked in: loop xt_CHECKSUM iptable_mangle nf_log_ipv4 nf_log_common xt_LOG xt_limit ipt_MASQUERADE nf_nat_masque
2015-12-16T07:00:35.556044+00:00 alert: [ 2046.721693] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.0-327.3.1.el7.x86_64 #1
2015-12-16T07:00:35.556054+00:00 alert: [ 2046.721693] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20151012_155810-obs-1 04/01/2014
2015-12-16T07:00:35.556057+00:00 alert: [ 2046.721693] ffff88007fc03d88 76f0075ca754d7b4 ffff88007fc03d40 ffffffff8163516c
2015-12-16T07:00:35.556058+00:00 alert: [ 2046.721693] ffff88007fc03d78 ffffffff8107b200 0000000000000000 ffff88007a352000
2015-12-16T07:00:35.556059+00:00 alert: [ 2046.721693] ffff88007878ac80 0000000000000001 0000000000000000 ffff88007fc03de0
2015-12-16T07:00:35.556060+00:00 alert: [ 2046.721693] Call Trace:
2015-12-16T07:00:35.556061+00:00 alert: [ 2046.721693] <IRQ> [<ffffffff8163516c>] dump_stack+0x19/0x1b
2015-12-16T07:00:35.556062+00:00 alert: [ 2046.721693] [<ffffffff8107b200>] warn_slowpath_common+0x70/0xb0
2015-12-16T07:00:35.556063+00:00 alert: [ 2046.721693] [<ffffffff8107b29c>] warn_slowpath_fmt+0x5c/0x80
2015-12-16T07:00:35.556063+00:00 alert: [ 2046.721693] [<ffffffff8154ca50>] dev_watchdog+0x270/0x280
2015-12-16T07:00:35.556064+00:00 alert: [ 2046.721693] [<ffffffff8154c7e0>] ? dev_graft_qdisc+0x80/0x80
2015-12-16T07:00:35.556065+00:00 alert: [ 2046....

Read more...

Changed in fuel:
importance: High → Critical
tags: added: swarm-blocker
tags: added: blocker-for-qa
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Comments from the duplicated bug

Can't reproduce locally.
Has the same software and tried different dd operations but failed to trigger it.

Changed in fuel:
assignee: Fuel CI (fuel-ci) → Fuel QA Team (fuel-qa)
status: Confirmed → New
Changed in fuel:
status: New → Confirmed
status: Confirmed → New
Ilya Kutukov (ikutukov)
Changed in fuel:
status: New → Confirmed
tags: added: area-qa
Revision history for this message
Nastya Urlapova (aurlapova) wrote :

How it can be swarm-blocker if you use tests on packaging ci?

Changed in fuel:
importance: Critical → Undecided
status: Confirmed → Incomplete
tags: removed: swarm-blocker
Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → nobody
Revision history for this message
Alexei Sheplyakov (asheplyakov) wrote :

@Timur,

tl;dr: the machine running the tests is overloaded, please buy more RAM.

> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20151012_155810-obs-1 04/01/2014

The host powering the VMs got overloaded, and its kernel have been heavily swapping.
As a result the qemu process got blocked for a very long time. This looks like an Ethernet
card lockup from the point of the view of the guest kernel.

Possible solutions:

1) Add more RAM to the host,
2) Use SSD for the host OS and swap partition (logical volume),
3) Use a different hard drive to store VMs images, so intensive guests IO does not interfere with host IO (in particular paging/swapping),
4) Restrict the load on the host

Changed in fuel:
assignee: nobody → Artem Silenkov (asilenkov)
importance: Undecided → Critical
status: Incomplete → Triaged
Revision history for this message
Artem Silenkov (asilenkov) wrote :

@asheplyakov

             total used free shared buffers cached
Mem: 65943276 65362908 580368 32300 386524 56175536
-/+ buffers/cache: 8800848 57142428
Swap: 15998972 0 15998972

Sadly we could reproduce it on zero-loaded server.
It could be possibly related to RAM lack for containers.
Gathering data still.

Revision history for this message
Igor Marnat (imarnat) wrote :

Custom BVT with the same settings and ISO as in #431 successfully passed when amount of RAM was increased up to 4 GB for master node and slave nodes:
http://jenkins-product.srt.mirantis.net:8080/job/8.0.custom.ubuntu.bvt_2/437/

Job which failed with smaller amount of RAM:
http://jenkins-product.srt.mirantis.net:8080/job/8.0.custom.ubuntu.bvt_2/431

Revision history for this message
Artem Silenkov (asilenkov) wrote :

Error is still presented but not fatal in case of increased RAM.

Revision history for this message
Igor Marnat (imarnat) wrote :

#438 with 3 GB per node also passed successfully (same ISO and same settings besides RAM as for #431):
http://jenkins-product.srt.mirantis.net:8080/job/8.0.custom.ubuntu.bvt_2/438/console

Revision history for this message
Igor Marnat (imarnat) wrote :
Revision history for this message
Artem Silenkov (asilenkov) wrote :

Yet another solution green http://jenkins-product.srt.mirantis.net:8080/job/8.0.custom.ubuntu.bvt_2/441/console

TCP offload is disabled, RAM is default.

Need deep testing as it could lead to unpredictable slow down

Revision history for this message
Artem Silenkov (asilenkov) wrote :

We recommend to increase RAM for every node to 3Gb as we have 3 successful test run with this.
It doesn't mean that bug is fixed doing this because error is still presented but not fatal in this case.

We also recommend to play with tcp offload switch and find out the impact while switching it off.

Production users are affected also so it's not enough to tune fuel-qa but fix it proper way.

Changed in fuel:
assignee: Artem Silenkov (asilenkov) → Fuel CI (fuel-ci)
Revision history for this message
Alexander Tsamutali (astsmtl) wrote :

I would recommend at least 4 GB for master node.

Revision history for this message
Aleksandra Fedorova (bookwar) wrote :

On agreement with QA we temporarily switched Packaging CI tests to use stable ISO 289.

QA team, please consider updating default ADMIN_NODE_MEMORY value in fuel-qa

Changed in fuel:
assignee: Fuel CI (fuel-ci) → Fuel QA Team (fuel-qa)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/258662

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/258662
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=6b77e014248c23c6540630777be1058637f3cbfa
Submitter: Jenkins
Branch: master

commit 6b77e014248c23c6540630777be1058637f3cbfa
Author: NastyaUrlapova <email address hidden>
Date: Thu Dec 17 00:24:30 2015 +0300

    HotFix for ADMIN RAM size

    Change-Id: I89c0d36c74fa3e3252e0dc00742ace274c7a69ec
    Related-Bug: #1526544

tags: removed: area-qa
summary: - Systest failed with URLError: <urlopen error [Errno 113] No route to
- host>
+ We have to pin Centos 7 version by default
Revision history for this message
Nastya Urlapova (aurlapova) wrote : Re: We have to pin Centos 7 version by default

After investigation was found, that we don't use pinning for Centos 7 version. Update to Centos 7.2(http://mirror.centos.org/centos-7/7.2.1511/) has brought regression during the operation of master node, network driver for e1000 doesn't work correctly. We have to investigate in more details why it happens and how RAM size and/or CPU count are correlated with issue.
For avoiding such regressions we must do pinning of Centos version and don't update major version of OS after FF.

Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → MOS Linux (mos-linux)
Revision history for this message
Igor Marnat (imarnat) wrote :

Nastya,
do you have data indicating that "network driver for e1000 doesn't work correctly" ?

Revision history for this message
Igor Marnat (imarnat) wrote :

BTW, old summary [0] showed something about the symptoms of the problem, new summary [1] is very generic and doesn't describe symptoms or anything about the problem

[0] Systest failed with URLError: <urlopen error [Errno 113] No route to - host>
[1] We have to pin Centos 7 version by default

Revision history for this message
Aleksander Mogylchenko (amogylchenko) wrote :

We should not pin versions, but move forward with the rest of the community and fix problems that appear.

Revision history for this message
Aleksander Mogylchenko (amogylchenko) wrote :
Revision history for this message
Artem Silenkov (asilenkov) wrote : Re: [Bug 1526544] Re: We have to pin Centos 7 version by default
Download full text (5.6 KiB)

Hello!
Every pin/unpin operation would require recompiling of all centos packages
we maintain.
Regards
Artem Silenkov
MOS-PAckaging

On Thu, Dec 17, 2015, 11:45 Aleksander Mogylchenko <
<email address hidden>> wrote:

> This might be related:
> https://wiki.centos.org/FAQ/CentOS7#ethernet
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1526695).
> https://bugs.launchpad.net/bugs/1526544
>
> Title:
> We have to pin Centos 7 version by default
>
> Status in Fuel for OpenStack:
> Triaged
>
> Bug description:
> During the executing 8.0-pkg-systest-ubuntu job for https://review
> .fuel-infra.org/#/c/14261/ there was deployment error occured
>
> ======================================================================
> ERROR: Deploy ceph HA with RadosGW for objects
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File
> "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py",
> line 296, in testng_method_mistake_capture_func
> compatability.capture_type_error(s_func)
> File
> "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py",
> line 27, in capture_type_error
> func()
> File
> "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py",
> line 350, in func
> func(test_case.state.get_state())
> File
> "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py",
> line 80, in wrapper
> result = func(*args, **kwargs)
> File
> "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/tests/test_ceph.py",
> line 504, in ceph_rados_gw
> self.fuel_web.deploy_cluster_wait(cluster_id)
> File
> "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py",
> line 428, in wrapper
> result = func(*args, **kwargs)
> File
> "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py",
> line 413, in wrapper
> result = func(*args, **kwargs)
> File
> "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py",
> line 464, in wrapper
> return func(*args, **kwargs)
> File
> "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py",
> line 474, in wrapper
> result = func(*args, **kwargs)
> File
> "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py",
> line 355, in wrapper
> return func(*args, **kwargs)
> File
> "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/fuel_web_client.py",
> line 695, in deploy_cluster_wait
> self.assert_task_success(task, interval=interval, timeout=timeout)
> File
> "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/__init__.py",
> line 57, in wrapped
> result = func(*args, **kwargs)
> File
> "/home/jenkins/workspace/8.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/fuel_web_client.py",
> line 312, in assert_task...

Read more...

Igor Marnat (imarnat)
summary: - We have to pin Centos 7 version by default
+ After update of master node to CentOS 7.2 Systests often fail with
+ URLError: <urlopen error [Errno 113] No route to - host>
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-devops (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/258949

summary: - After update of master node to CentOS 7.2 Systests often fail with
- URLError: <urlopen error [Errno 113] No route to - host>
+ After update of master node from Centos 7.1.1503 to CentOS 7.2 Systests
+ often fail with URLError: <urlopen error [Errno 113] No route to - host>
Revision history for this message
Nastya Urlapova (aurlapova) wrote : Re: After update of master node from Centos 7.1.1503 to CentOS 7.2 Systests often fail with URLError: <urlopen error [Errno 113] No route to - host>

Folks, looks like you lost the FF meaning. If you always want use upstream code, you should improve gates first of all.

@Igor,
>>>BTW, old summary [0] showed something about the symptoms of the problem, new summary [1] is very generic and doesn't describe symptoms or anything about the problem

if you consider my summary as very generic, please don't hesitate do real investigation.

Revision history for this message
Igor Marnat (imarnat) wrote :

@Nastya:
we work on it.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-devops (master)

Reviewed: https://review.openstack.org/258949
Committed: https://git.openstack.org/cgit/openstack/fuel-devops/commit/?id=dfb711f84d0dfd748b64a3a3e2dd1cd46e30759b
Submitter: Jenkins
Branch: master

commit dfb711f84d0dfd748b64a3a3e2dd1cd46e30759b
Author: NastyaUrlapova <email address hidden>
Date: Thu Dec 17 14:01:05 2015 +0300

    Updating ADMIN_NODE_MEMORY to 3Gb

    Change-Id: Ia8c1ecd90a74c4fbdcb4bae06c227b0c21159e76
    Related-Bug: #1526544

Dmitry Pyzhov (dpyzhov)
tags: added: area-linux
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to fuel-infra/jenkins-jobs (master)

Related fix proposed to branch: master
Change author: Aleksandra Fedorova <email address hidden>
Review: https://review.fuel-infra.org/14813

Changed in fuel:
status: Triaged → In Progress
Revision history for this message
Alexei Sheplyakov (asheplyakov) wrote : Re: After update of master node from Centos 7.1.1503 to CentOS 7.2 Systests often fail with URLError: <urlopen error [Errno 113] No route to - host>

> Sadly we could reproduce it on zero-loaded server.

6+ VMs heavily writing to their virtual hard drives backed by the same physical hard drive is not
even nearly a "zero loaded server".

Revision history for this message
Alexei Sheplyakov (asheplyakov) wrote :

> network driver for e1000 doesn't work correctly

Why Fuel CI is the only user experiencing this problem? Intel NICs and RHEL/CentOS 7 are quite popular, there should
have been quite a lot of similar bug reports.

Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Tag "blocker-for-qa" removed because we fixed it with CentOS 7.1

tags: removed: blocker-for-qa
Revision history for this message
Ivan Suzdal (isuzdal) wrote :

It would be nice to have root/sudo at least on one of this machine (srv83-bud.infra, srv84-bud.infra)

Revision history for this message
Igor Shishkin (teran) wrote :

Granted on srv83-bud.

the server requires redeployment before getting it back to CI.

Changed in fuel:
assignee: MOS Linux (mos-linux) → Ivan Suzdal (isuzdal)
Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

I believe, this is no longer a blocker, downgrading to High.

Changed in fuel:
importance: Critical → High
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

Not a QA blocker anymore, but does block progress with bug #1528535.

Revision history for this message
Dmitry Teselkin (teselkin-d) wrote :

There is a bug with similar root cause (e1000 NIC in virtual machine) on Ubuntu - https://bugs.launchpad.net/mos/+bug/1533165
I'm attaching the part of messages logfile from that bug with the same stacktrace.
It doen't look like CentOS-7 related issue.

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on fuel-infra/jenkins-jobs (master)

Change abandoned by Aleksandra Fedorova <email address hidden> on branch: master
Review: https://review.fuel-infra.org/14813
Reason: Not needed anymore

Revision history for this message
Roman Podoliaka (rpodolyaka) wrote : Re: After update of master node from Centos 7.1.1503 to CentOS 7.2 Systests often fail with URLError: <urlopen error [Errno 113] No route to - host>

Are we still working on this in 8.0? AFAIU, we are not going to use the newer CentOS version in 8.0, so maybe let's just close this as Won't fix for 8.0 and give it another try in 9.0?

Revision history for this message
Dmitry Teselkin (teselkin-d) wrote :

MOS-8 will not get CentOS-7.2, moving to won't fix

summary: - After update of master node from Centos 7.1.1503 to CentOS 7.2 Systests
- often fail with URLError: <urlopen error [Errno 113] No route to - host>
+ CentOS-7.2 based ISO often fails with '[Errno 113] No route to - host'
+ when e1000 driver is used
description: updated
tags: added: area-docs
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/285306
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=6e44e96f1f249bd2021544b2c48ef9dce4ac17fe
Submitter: Jenkins
Branch: master

commit 6e44e96f1f249bd2021544b2c48ef9dce4ac17fe
Author: Dmitry Teselkin <email address hidden>
Date: Fri Feb 26 16:40:53 2016 +0300

    Disable TSO offloading on master node

    Disabling TSO offloading helps to get rid of
    e1000's 'Tx Unit Hung' issues.

    Related-bug: #1526544

    Change-Id: I4dea660b616661f40a3555be642d0a5020f379d4

Revision history for this message
Michele Fagan (michelefagan) wrote :

Created JIRA https://mirantis.jira.com/browse/PROD-4029 to capture doc workaround by Dmitry Teselkin. For publication in 9.0 RNs.

tags: added: release-notes
Revision history for this message
Sergey Yudin (tsipa740) wrote :

We observed this issue in Rocket CI with disabled TSO offloading on literally zero loaded node.

Maybe we should use virtio for a while, or another nic?

Revision history for this message
Sergey Yudin (tsipa740) wrote :

Guys, this is gonna affect AT&T prod since they want to install FM on VMs.

Moreover, we have tried to play with INTERFACE_MODEL. With virtio FM is not working at all - i've not checked why but FM is not bringing up ifaces.
With rtl8139 we still observing the same problem. The trace is kinda different but the problem is still the same.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-docs (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/304612

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-docs (master)

Reviewed: https://review.openstack.org/304612
Committed: https://git.openstack.org/cgit/openstack/fuel-docs/commit/?id=e9d1be7df3a46bfa5d40229dcdf73c64864f1be6
Submitter: Jenkins
Branch: master

commit e9d1be7df3a46bfa5d40229dcdf73c64864f1be6
Author: Evgeny Konstantinov <email address hidden>
Date: Tue Apr 12 16:15:08 2016 +0300

    Add Fuel Mitaka known issues to relnotes
    Related-Bug: #1439776
    Related-Bug: #1450100
    Related-Bug: #1460169
    Related-Bug: #1490597
    Related-Bug: #1526544
    Related-Bug: #1556854
    Related-Bug: #1446704

    Change-Id: I3df16c163d82af7d0db8a64643b915909cabd8f1

Dmitry Pyzhov (dpyzhov)
Changed in fuel:
milestone: 9.0 → 10.0
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to fuel-infra/zuul-layouts (master)

Related fix proposed to branch: master
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/19945

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix merged to fuel-infra/zuul-layouts (master)

Reviewed: https://review.fuel-infra.org/19945
Submitter: Aleksandra Fedorova <email address hidden>
Branch: master

Commit: 71f3258521688be4459d10e2eb0bd68e8cba7150
Author: Ivan Suzdal <email address hidden>
Date: Thu Apr 21 13:59:45 2016

Add processing for e1000 kmod package

 -- packages/centos7/e1000

Change-Id: I7cbc212d8cd3a6536761f6056ac9f631575b1086
Related-Bug: #1526544

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to fuel-infra/jeepyb-config (master)

Related fix proposed to branch: master
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/19957

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix merged to fuel-infra/jeepyb-config (master)

Reviewed: https://review.fuel-infra.org/19957
Submitter: Andrey Nikitin <email address hidden>
Branch: master

Commit: 9024d03f61238e098f20f94554f23719e2b3a7a1
Author: Ivan Suzdal <email address hidden>
Date: Thu Apr 21 21:33:19 2016

Add new project packages/centos7/e1000

e1000 kmod driver for Centos 7

Change-Id: I911372f4f581fe5634c5432d8010a9dcbdf4993b
Related-Bug: #1526544

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to packages/centos7/e1000 (master)

Fix proposed to branch: master
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/19985

tags: added: centos-72-target
tags: added: centos72-target
removed: centos-72-target
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to packages/centos7/e1000 (master)

Reviewed: https://review.fuel-infra.org/19985
Submitter: Pkgs Jenkins <email address hidden>
Branch: master

Commit: bf9b0d9ded90ee10bda3bb42f86d77980d0fbd38
Author: Ivan Suzdal <email address hidden>
Date: Mon May 23 16:48:41 2016

e1000 kmod driver for Centos 7

This kmod contains original e1000 driver code from Centos kernel
with additional patches which resolve e1000 Hangs issue.

Change-Id: I20416cf2288e8dcd188c7a408057724a44dad4ea
Closes-Bug: #1526544

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to packages/centos7/e1000 (9.0)

Fix proposed to branch: 9.0
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/21202

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to packages/centos7/e1000 (9.0)

Reviewed: https://review.fuel-infra.org/21202
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0

Commit: 0d5a904ef95ad373907de0d159a191a34b075e2c
Author: Ivan Suzdal <email address hidden>
Date: Tue May 24 16:48:16 2016

e1000 kmod driver for Centos 7

This kmod contains original e1000 driver code from Centos kernel
with additional patches which resolve e1000 Hangs issue.

Change-Id: I20416cf2288e8dcd188c7a408057724a44dad4ea
Closes-Bug: #1526544
(cherry picked from commit bf9b0d9ded90ee10bda3bb42f86d77980d0fbd38)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/323419

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-main (master)

Fix proposed to branch: master
Review: https://review.openstack.org/325342

Changed in fuel:
assignee: Ivan Suzdal (isuzdal) → Dmitry Teselkin (teselkin-d)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/325347

Changed in fuel:
assignee: Dmitry Teselkin (teselkin-d) → Ivan Suzdal (isuzdal)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-main (master)

Change abandoned by Dmitry Teselkin (<email address hidden>) on branch: master
Review: https://review.openstack.org/325342
Reason: https://review.openstack.org/#/c/325347/

Revision history for this message
Alexei Sheplyakov (asheplyakov) wrote :

> The root cause here is broken driver for e1000 interface.

Any proofs please? e1000 is quite a common hardware, yet I haven't seen any reports regarding
broken e1000 support in RHEL/CentOS 7.x

Revision history for this message
Ivan Suzdal (isuzdal) wrote :

Alexei, it's a really long story.

Related to this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1330516
(But only if you have RH account with sufficient permissions - it's private bug)

Also:
https://access.redhat.com/solutions/25528
https://bugzilla.kernel.org/show_bug.cgi?id=47331

Also:
https://github.com/torvalds/linux/commit/a4605fef7132f19afded76ee025c957558271a7d
https://github.com/torvalds/linux/commit/847a1d6796c767f8b697ead60997b847a84b897b

If you will try ask google "e1000 tx unit hang" - you, probably, will be surprised

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-main (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/326368

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-main (master)

Reviewed: https://review.openstack.org/325977
Committed: https://git.openstack.org/cgit/openstack/fuel-main/commit/?id=722bcb67df6fbcb7e7b07ccdab085c4c43932571
Submitter: Jenkins
Branch: master

commit 722bcb67df6fbcb7e7b07ccdab085c4c43932571
Author: Ivan Suzdal <email address hidden>
Date: Mon Jun 6 18:19:56 2016 +0300

    Add MASTER_NODE_EXTRA_PACKAGES processing

    Each package from MASTER_NODE_EXTRA_PACKAGES
    shall be installed from kickstart.

    Related-Bug: #1526544

    Change-Id: I569c15177ca8cd5f990a4ee1ae986654975919a1

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/326000
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=72830315e5eb36f0fbb255786726c2f52ebd3d7f
Submitter: Jenkins
Branch: master

commit 72830315e5eb36f0fbb255786726c2f52ebd3d7f
Author: Ivan Suzdal <email address hidden>
Date: Mon Jun 6 18:43:52 2016 +0300

    Add MASTER_NODE_EXTRA_PACKAGES env variable

    This change allow to install any
    additional packages from any additional
    repository during install master-node.

    Related-Bug: #1526544

    Change-Id: I4b10ff432b385824595c0ac65b5282a2b81c859c

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to fuel-infra/jeepyb-config (master)

Related fix proposed to branch: master
Change author: Dmitry Teselkin <email address hidden>
Review: https://review.fuel-infra.org/21772

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix proposed to fuel-infra/zuul-layouts (master)

Related fix proposed to branch: master
Change author: Dmitry Teselkin <email address hidden>
Review: https://review.fuel-infra.org/21774

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Related fix merged to fuel-infra/jeepyb-config (master)

Reviewed: https://review.fuel-infra.org/21772
Submitter: Andrey Nikitin <email address hidden>
Branch: master

Commit: 43b2b2994649f972492d9d6c5034ff8e2653c9f0
Author: Dmitry Teselkin <email address hidden>
Date: Tue Jun 7 14:29:20 2016

New project added

-- packages/centos7/dkms

Change-Id: I54d429de3d5694a134e986c2ff6e299285d446e7
Related-Bug: #1526544

Revision history for this message
Maksim Malchuk (mmalchuk) wrote :

Alexei, this screenshot JFYI, this is default installation in the VirtualBox for example

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-main (master)

Reviewed: https://review.openstack.org/326368
Committed: https://git.openstack.org/cgit/openstack/fuel-main/commit/?id=8b4910d5a4b2b62cf4b0f23c3cb6e56d1bd52fca
Submitter: Jenkins
Branch: master

commit 8b4910d5a4b2b62cf4b0f23c3cb6e56d1bd52fca
Author: Ivan Suzdal <email address hidden>
Date: Tue Jun 7 13:33:05 2016 +0300

    Add kmod-e1000 package

    Add kmod-e1000 rpm which sould be installed
    later in our tests.

    Change-Id: I08bfd85b66797aab5e7e5e83f7366ea7a2f553d5
    Related-Bug: #1526544

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-main (stable/mitaka)

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/326932

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (stable/mitaka)

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/326938

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-qa (stable/mitaka)

Reviewed: https://review.openstack.org/326938
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=4a2fdcde81b31e274bc4d79d150beb8c8462d578
Submitter: Jenkins
Branch: stable/mitaka

commit 4a2fdcde81b31e274bc4d79d150beb8c8462d578
Author: Ivan Suzdal <email address hidden>
Date: Mon Jun 6 18:43:52 2016 +0300

    Add MASTER_NODE_EXTRA_PACKAGES env variable

    This change allow to install any
    additional packages from any additional
    repository during install master-node.

    Related-Bug: #1526544

    Change-Id: I4b10ff432b385824595c0ac65b5282a2b81c859c
    (cherry picked from commit 72830315e5eb36f0fbb255786726c2f52ebd3d7f)

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-main (stable/mitaka)

Reviewed: https://review.openstack.org/326932
Committed: https://git.openstack.org/cgit/openstack/fuel-main/commit/?id=f33d2b90194ccee5a71b7fb5c8fc876917385e57
Submitter: Jenkins
Branch: stable/mitaka

commit f33d2b90194ccee5a71b7fb5c8fc876917385e57
Author: Ivan Suzdal <email address hidden>
Date: Tue Jun 7 13:33:05 2016 +0300

    Add kmod-e1000 package

    Add kmod-e1000 rpm which sould be installed
    later in our tests.

    Change-Id: I08bfd85b66797aab5e7e5e83f7366ea7a2f553d5
    Related-Bug: #1526544
    (cherry picked from commit 8b4910d5a4b2b62cf4b0f23c3cb6e56d1bd52fca)

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (master)

Fix proposed to branch: master
Review: https://review.openstack.org/327093

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-main (stable/mitaka)

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/327106

Changed in fuel:
status: Fix Committed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-main (stable/mitaka)

Reviewed: https://review.openstack.org/327106
Committed: https://git.openstack.org/cgit/openstack/fuel-main/commit/?id=d6a22557d132c592b18c6bac90f5f4b8d1aa3ad7
Submitter: Jenkins
Branch: stable/mitaka

commit d6a22557d132c592b18c6bac90f5f4b8d1aa3ad7
Author: Ivan Suzdal <email address hidden>
Date: Mon Jun 6 18:19:56 2016 +0300

    Add MASTER_NODE_EXTRA_PACKAGES processing

    Each package from MASTER_NODE_EXTRA_PACKAGES
    shall be installed from kickstart.

    Related-Bug: #1526544

    Change-Id: I569c15177ca8cd5f990a4ee1ae986654975919a1
    (cherry picked from commit 722bcb67df6fbcb7e7b07ccdab085c4c43932571)

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to fuel-infra/jenkins-jobs (master)

Reviewed: https://review.fuel-infra.org/21830
Submitter: Alexander Evseev <email address hidden>
Branch: master

Commit: c9c1ef7b9b347ba289772788baa54b757fd214b7
Author: Dmitry Teselkin <email address hidden>
Date: Wed Jun 8 13:25:00 2016

Use custom e1000 in pkg-systest

Closes-bug: #1526544

Change-Id: I7a28c394aa3a10d42b570c480e05a70d0996f5a5

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-qa (stable/mitaka)

Change abandoned by Ivan Suzdal (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/327061
Reason: https://review.fuel-infra.org/#/c/21830/

Changed in fuel:
status: In Progress → Fix Committed
tags: added: release-notes-done
removed: release-notes
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Ok now everything is done, marked as Fix Released for MOS 9.0 #495.

Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-qa (master)

Change abandoned by Ivan Suzdal (<email address hidden>) on branch: master
Review: https://review.openstack.org/323419

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: master
Review: https://review.openstack.org/327093
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-main (master)

Change abandoned by Ivan Suzdal (<email address hidden>) on branch: master
Review: https://review.openstack.org/325347

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on fuel-infra/zuul-layouts (master)

Change abandoned by Dmitry Teselkin <email address hidden> on branch: master
Review: https://review.fuel-infra.org/21774
Reason: outdated

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.