AttributeError: 'NoneType' object has no attribute 'port_security_enabled

Bug #1694420 reported by Eduardo Gonzalez
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Ihar Hrachyshka
neutron (Ubuntu)
Fix Released
Undecided
Unassigned

Bug Description

Hi, I'm seeing in kolla gates failing with this error:
AttributeError: 'NoneType' object has no attribute 'port_security_enabled'

Instances fail to deploy while retrieving the port in openvswitch_agent.

I think may be related to this recent change https://review.openstack.org/#/c/466158/

2017-05-30 10:05:23.865 7 ERROR neutron.agent.rpc [req-3d80325e-9430-46d2-ace7-b5a6ad358d77 - - - - -] Failed to get details for device c22edc09-6451-4b78-9160-399fd549314e
2017-05-30 10:05:23.865 7 ERROR neutron.agent.rpc Traceback (most recent call last):
2017-05-30 10:05:23.865 7 ERROR neutron.agent.rpc File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/rpc.py", line 219, in get_devices_details_list_and_failed_devices
2017-05-30 10:05:23.865 7 ERROR neutron.agent.rpc self.get_device_details(context, device, agent_id, host))
2017-05-30 10:05:23.865 7 ERROR neutron.agent.rpc File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/rpc.py", line 257, in get_device_details
2017-05-30 10:05:23.865 7 ERROR neutron.agent.rpc 'port_security_enabled': port_obj.security.port_security_enabled,
2017-05-30 10:05:23.865 7 ERROR neutron.agent.rpc AttributeError: 'NoneType' object has no attribute 'port_security_enabled'
2017-05-30 10:05:23.865 7 ERROR neutron.agent.rpc

http://logs.openstack.org/93/463593/17/check/gate-kolla-dsvm-deploy-centos-source-centos-7-nv/f61667d/logs/kolla/neutron/neutron-openvswitch-agent.txt.gz#_2017-05-30_10_05_23_865

Environment:

Source code from master
Distributions affected: centos, ubuntu, oraclelinux

Regards

Tags: sg-fw
Revision history for this message
Eduardo Gonzalez (egonzalez90) wrote :

Tested locally out of the gates and fails with the same error

description: updated
Revision history for this message
Hirofumi Ichihara (ichihara-hirofumi) wrote :

I reproduced the issue. When "port_security" isn't set to extension_drivers and then we boot VM, the issue occurred.

tags: added: sg-fw
Changed in neutron:
status: New → Confirmed
importance: Undecided → Critical
Revision history for this message
Steven Dake (sdake) wrote :

Hey folks,

Hate to rush upstream, however, kolla's upstream gates are completely blocked because of this regression. Anything you can do to prioritize a revert or a resolution would be appreciated.

Thanks
-steve

Revision history for this message
Hirofumi Ichihara (ichihara-hirofumi) wrote :

I'm not sure the cause of issue. I tried to revert https://review.openstack.org/#/c/466158/ is referred on Bug Description but the issue occurs again. I put importance Critical so I believe neutron team can solve the issue soon.

Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

As pointed out in #2, a kolla-side fix would be to add the 'port_security' to the list of extension_drivers list for ML2 like in [1]. It's good practice to have this extension enabled.

[1] http://logs.openstack.org/58/466158/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/a2c03a6/logs/etc/neutron/plugins/ml2/ml2_conf.ini.txt.gz

Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

Any reason why the Kolla project does not have this extension enabled by default?

Changed in neutron:
assignee: nobody → Armando Migliaccio (armando-migliaccio)
Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

How is the kolla gate blocked? I see failures only on non-voting jobs. Where they flipped because of this issue?

Changed in neutron:
importance: Critical → Low
importance: Low → High
Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/469327

Changed in neutron:
assignee: Armando Migliaccio (armando-migliaccio) → Kevin Benton (kevinbenton)
status: Confirmed → In Progress
Revision history for this message
Eduardo Gonzalez (egonzalez90) wrote :

Armando, asked in neutron IRC, but didn't get an answer. Is supported enabling port_security in active deployments with existing networks? As far I know was not possible to do that in the past (around kilo). Not sure if in current master is supported.
If port_security can be enabled without manually touching the database as in the past, enabling port_security by default is an option (tested and fix the issue).
But if is not possible, that would be a blocker for enabling it by default as will break current deployments and upgrades.

At this moment we only enable port_security when designate or tacker are deployed as they need that feature.
Enable for all cases by default is just one liner change https://review.openstack.org/#/c/469373/ , but I'm concerned on the upgrade procedure or issue it may cause as commented before.

Regards

Changed in neutron:
assignee: Kevin Benton (kevinbenton) → Ihar Hrachyshka (ihar-hrachyshka)
Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

Eduardo, yes, enabling it post installation is possible now with https://review.openstack.org/#/q/I8607cdecdc16c5f94635c94e2f02700c732806eb,n,z (fixed since Liberty).

Revision history for this message
Eduardo Gonzalez (egonzalez90) wrote :

Ihar, opened another bug for port_security on existing networks https://bugs.launchpad.net/neutron/+bug/1694965

DHCP does not work in old networks until neutron subnet-update --disable-dhcp and another --enable-dhcp is done.

Regards

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.openstack.org/470648
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=94a882babe7210cdec0029f4796f2e1b83df116b
Submitter: Jenkins
Branch: master

commit 94a882babe7210cdec0029f4796f2e1b83df116b
Author: Jeffrey Zhang <email address hidden>
Date: Sun Jun 4 09:01:49 2017 +0800

    Enable port_security in gate to fix the neutron broken

    Revert this when neutron bug[0] is fixed.

    [0] https://bugs.launchpad.net/neutron/+bug/1694420

    Change-Id: Id9f84608826351b9675cd6a6f2a183e91ce33bf6
    Partial-Bug: #1694420

Changed in kolla-ansible:
importance: Undecided → Critical
Revision history for this message
Surya Prakash Singh (confisurya) wrote :

As only kolla-ansible is affected by this issue, so removing non affecting deliverable kolla from the project affecting list for this bug.

no longer affects: kolla
Changed in kolla-ansible:
importance: Critical → High
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/469327
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=817f39e49599c3308b5d619163a2426269633067
Submitter: Jenkins
Branch: master

commit 817f39e49599c3308b5d619163a2426269633067
Author: Kevin Benton <email address hidden>
Date: Tue May 30 21:38:45 2017 -0700

    Provide fallback for disabled port security extension

    The push notification logic always assumed the port security object
    would exist but it is not present on the port when the extension is
    disabled. This defaults it to true like the server side code.[1]

    1.
    https://github.com/openstack/neutron/blob/c430e9b8d41c139284e840be37629afcdbc96b37/neutron/plugins/ml2/rpc.py#L142

    Change-Id: Ice89ad9dd486ad5fcac534ef5f7d8aae3b6b0f97
    Closes-Bug: #1694420

Changed in neutron:
status: In Progress → Fix Released
Changed in kolla-ansible:
assignee: nobody → Eduardo Gonzalez (egonzalez90)
status: Confirmed → In Progress
milestone: none → pike-2
tags: added: neutron-proactive-backport-potential
Changed in kolla-ansible:
milestone: pike-2 → pike-3
Revision history for this message
Ihar Hrachyshka (ihar-hrachyshka) wrote :

It's Pike only, not a backport material.

tags: removed: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on kolla-ansible (master)

Change abandoned by Eduardo Gonzalez (<email address hidden>) on branch: master
Review: https://review.openstack.org/473321

no longer affects: kolla-ansible
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package neutron - 2:11.0.0~b2-0ubuntu2

---------------
neutron (2:11.0.0~b2-0ubuntu2) artful; urgency=medium

  * d/p/bug1694420.patch: Cherry pick fix to resolve issues in deployments
    where the port-security driver is not enabled (LP: #1694420).

 -- James Page <email address hidden> Wed, 21 Jun 2017 14:23:01 +0100

Changed in neutron (Ubuntu):
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 5.0.0.0b3

This issue was fixed in the openstack/kolla-ansible 5.0.0.0b3 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 11.0.0.0b3

This issue was fixed in the openstack/neutron 11.0.0.0b3 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.