Ubuntu
openvswitch package

[SRU] ovsdb-server.service needs a depedency on local-fs.target

Bug #1887177 reported by Brian Turek
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
openvswitch (Ubuntu)
Fix Released
Undecided
Chris MacNaughton
Focal
Fix Released
Undecided
Unassigned
Groovy
Fix Released
Undecided
Chris MacNaughton

Bug Description

[Impact]
When /var is on a separate filesystem (ZFS), ovsdb-server crashes on start if it is triggered before that filesystem is ready.

I recently just did a from-scratch install of Ubuntu 20.04 server edition and ran into issues with Open vSwitch and ZFS. I attempted to use ZFS for all of /var only to find that ovsdb-server pre-empted my ZFS /var mount which caused it to crash when trying to read its configuration DB at/var/lib/openvswitch/conf.db After much troubleshooting, the problem basically boils down to ovsdb-server.service needing a requirement on local-fs.target

I then found a blog post on Open Cloud Blog (https://www.opencloudblog.com/?p=240) that contained a fix:

The "After" line /lib/systemd/system/ovsdb-server.service needs the following changes:

    [Unit]
    Description=Open vSwitch Database Unit
    After=syslog.target network-pre.target dpdk.service local-fs.target
    Before=network.target networking.service
    PartOf=openvswitch-switch.service
    DefaultDependencies=no

    [Service]
    LimitNOFILE=1048576
    Type=forking
    Restart=on-failure
    EnvironmentFile=-/etc/default/openvswitch
    ExecStart=/usr/share/openvswitch/scripts/ovs-ctl \
              --no-ovs-vswitchd --no-monitor --system-id=random \
              start $OPTIONS
    ExecStop=/usr/share/openvswitch/scripts/ovs-ctl --no-ovs-vswitchd stop
    ExecReload=/usr/share/openvswitch/scripts/ovs-ctl --no-ovs-vswitchd \
               --no-monitor restart $OPTIONS
    RuntimeDirectory=openvswitch
    RuntimeDirectoryMode=0755

[Test Case]

Install ZFS on a machine, configure /var to be mounted on ZFS, install Open vSwitch, restart the server. The OpenvSwitch process should wait on the ZFS mount to start.

[Regression Potential]

Low. The only change in this is to defer the ovsdb-server startup until after the local-fs Systemd target has started. The only risk I can forsee is if the local-fs target didn't come up.

[racb] Service dependency and thus ordering is being adjusted, so if there is a regression it might manifest in users with unusual or different service installations from the norm, or in users with customised service configurations. There might also be unrelated latent issues or race conditions revealed as a result of changing the order of service startups.

[Discussion]

See original description

Related branches

~chris.macnaughton/ubuntu/+source/openvswitch:bug/focal/1887177
James Page: Approve
Diff: 28 lines (+9/-1)
2 files modified
debian/changelog (+8/-0)
debian/openvswitch-switch.ovsdb-server.service (+1/-1)
~chris.macnaughton/ubuntu/+source/openvswitch:bug/1887177
James Page: Pending requested
Diff: 32 lines (+8/-2)
2 files modified
debian/changelog (+7/-1)
debian/openvswitch-switch.ovsdb-server.service (+1/-1)
Chris MacNaughton (chris.macnaughton)
Changed in openvswitch (Ubuntu):
status: New → In Progress
assignee: nobody → Chris MacNaughton (chris.macnaughton)
Revision history for this message
Sebastian Marsching (sebastian-marsching) wrote :

We are affected by this bug, too.

A more stable workaround than modifying /lib/systemd/system/ovsdb-server.service might be adding an overrides file (so that the fix will survive package upgrades).

On our affected systems, we added a file /etc/systemd/system/ovsdb-server.service.d/after-local-fs.conf with the following content:

[Unit]
After=local-fs.target

Revision history for this message
Corey Bryant (corey.bryant) wrote :

This is fixed in the groovy openvswitch packaging branch.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openvswitch - 2.13.1-0ubuntu1

---------------
openvswitch (2.13.1-0ubuntu1) groovy; urgency=medium

  [ Chris MacNaughton ]
  * d/openvswitch-switch.ovsdb-server.service: Add local-fs.target to
    systemd service file to ensure that local filesystems are ready
    before the ovsdb service tries to start (LP: #1887177).
  * d/control: Remove Breaks/Replaces that are older than Focal (LP: #1878419).

  [ James Page ]
  * New upstream point release.
  * d/p/py3-compat.patch: Refresh.

 -- James Page <email address hidden> Wed, 05 Aug 2020 12:17:06 +0100

Changed in openvswitch (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Brian Turek (brian-turek) wrote :

Is this something that can be released to Focal as a SRU?

Chris MacNaughton (chris.macnaughton)
summary: - ovsdb-server.service needs a depedency on local-fs.target
+ [SRU] ovsdb-server.service needs a depedency on local-fs.target
description: updated
Robie Basak (racb)
description: updated
Revision history for this message
Robie Basak (racb) wrote : Please test proposed package

Hello Brian, or anyone else affected,

Accepted openvswitch into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvswitch/2.13.1-0ubuntu0.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in openvswitch (Ubuntu Focal):
status: New → Fix Committed
tags: added: verification-needed verification-needed-focal
Revision history for this message
Brian Turek (brian-turek) wrote :

I'm not entirely sure what happened but it appears the build completely broke on amd64 with a majority of the tests failing. I fear the unrelated minor version bump (2.13.0 to 2.13.1) may have really gotten things in a bad state.

Revision history for this message
Brian Turek (brian-turek) wrote :

I checked proposed and the amd64 versions finally built. I installed v2.13.1-0ubuntu0.20.04.1, removed my temporary fix, rebooted, and am pleased to report that ovsdb-server is now waiting for ZFS to mount. Manual inspection using "systemctl list-dependencies ovsdb-server.service --after" shows dependencies on the relevant ZFS mounts as well.

tags: added: verification-done-focal
removed: verification-needed-focal
James Page (james-page)
tags: added: verification-done
removed: verification-needed
Revision history for this message
Chris Halse Rogers (raof) wrote : Update Released

The verification of the Stable Release Update for openvswitch has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openvswitch - 2.13.1-0ubuntu0.20.04.1

---------------
openvswitch (2.13.1-0ubuntu0.20.04.1) focal; urgency=medium

  [ Chris MacNaughton ]
  * d/openvswitch-switch.ovsdb-server.service: Add local-fs.target to systemd
    service file to ensure that local filesystems are ready before the ovsdb
    service tries to start (LP: #1887177).

  [ James Page ]
  * New upstream point release (LP: #1895101).
  * d/p/py3-compat.patch: Refresh.

 -- Chris MacNaughton <email address hidden> Thu, 10 Sep 2020 10:30:24 +0100

Changed in openvswitch (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.