syslog logging does not work

Bug #1958162 reported by Andreas Hasenack
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
frr (Ubuntu)
Fix Released
High
Andreas Hasenack
Focal
Fix Released
Medium
Andreas Hasenack
Impish
Won't Fix
Medium
Unassigned
Jammy
Fix Released
High
Andreas Hasenack
Kinetic
Fix Released
High
Andreas Hasenack

Bug Description

[Impact]
frr is configured out of the box to use rsyslog logging into a /var/log/frr/frr.log file. This is not working because the permissions in /var/log/frr prevent the unprivileged rsyslog daemon from writing there.

In Debian this works because rsyslog there runs as root.

[Test Plan]

For each $RELEASE under testing, launch a container:

lxc launch ubuntu:$RELEASE $RELEASE-frr-logging

1) upgrade test
Then shell into it, and install frr:

lxc shell $RELEASE-frr-logging

apt update; apt install frr -y

Notice /var/log/frr is empty even after a restart, and that the frr directory is owned by frr:frr:

root@focal-frr-logging:~# systemctl restart frr
root@focal-frr-logging:~# ls -la /var/log/frr
total 0
drwxr-x--- 1 frr frr 0 Jan 20 2020 .
drwxrwxr-x 1 root syslog 274 Jul 20 14:08 ..

And /var/log/syslog should have this error:
root@focal-frr-logging:~# grep frr\\.log /var/log/syslog
Jul 20 14:09:23 focal-frr-logging rsyslogd: file '/var/log/frr/frr.log': open error: Permission denied [v8.2001.0 try https://www.rsyslog.com/e/2433 ]

Now install the package from proposed, and:
a) /var/log/frr ownership is changed to syslog:adm
root@focal-frr-logging:~# ls -lad /var/log/frr
drwxr-x--- 1 syslog adm 0 Jan 20 2020 /var/log/frr

b) after restarting frr, the log file is created as syslog:adm:
root@focal-frr-logging:~# ls -la /var/log/frr
total 4
drwxr-x--- 1 syslog adm 14 Jul 20 14:12 .
drwxrwxr-x 1 root syslog 274 Jul 20 14:08 ..
-rw-r----- 1 syslog adm 1175 Jul 20 14:12 frr.log

c) logrotate maintains these permissions:
root@focal-frr-logging:~# ls -la /var/log/frr
total 4
drwxr-x--- 1 syslog adm 14 Jul 20 14:12 .
drwxrwxr-x 1 root syslog 274 Jul 20 14:08 ..
-rw-r----- 1 syslog adm 1175 Jul 20 14:12 frr.log
root@focal-frr-logging:~# logrotate -f /etc/logrotate.conf
root@focal-frr-logging:~# ls -la /var/log/frr
total 4
drwxr-x--- 1 syslog adm 38 Jul 20 14:12 .
drwxrwxr-x 1 root syslog 354 Jul 20 14:12 ..
-rw-r----- 1 syslog adm 0 Jul 20 14:12 frr.log
-rw-r----- 1 syslog adm 406 Jul 20 14:12 frr.log.1.gz

2) Fresh install test

In the container, install the frr package from proposed directly. The ownership of /var/log/frr should be correct from the start (syslog:adm):

root@focal-frr-logging:~# ls -lad /var/log/frr
drwxr-x--- 1 syslog adm 0 Jul 19 20:40 /var/log/frr

And upon restarting frr, the frr.log file should appear:

root@focal-frr-logging:~# systemctl restart frr
root@focal-frr-logging:~# ls -la /var/log/frr/
total 4
drwxr-x--- 1 syslog adm 14 Jul 20 14:30 .
drwxrwxr-x 1 root syslog 326 Jul 20 14:28 ..
-rw-r----- 1 syslog adm 1175 Jul 20 14:30 frr.log

Finally, logrotate should also work and preserve the ownership:
root@focal-frr-logging:~# logrotate -f /etc/logrotate.conf
root@focal-frr-logging:~# ls -la /var/log/frr/
total 4
drwxr-x--- 1 syslog adm 38 Jul 20 14:30 .
drwxrwxr-x 1 root syslog 462 Jul 20 14:30 ..
-rw-r----- 1 syslog adm 0 Jul 20 14:30 frr.log
-rw-r----- 1 syslog adm 409 Jul 20 14:30 frr.log.1.gz

[Where problems could occur]

Some users might have fixed this problem on their systems manually, either following the same approach here, or doing something else. An important part of the fix from this SRU is in the frr.postinst maintainer script, which is something administrators cannot easily override. Therefore it's possible that this update might break such existing setups where users fixed the problem themselves.

It's also possible that users elected to switch to frr direct logging, bypassing rsyslog. In such case, since /var/log/frr is now owned by syslog:adm instead of frr:frr, the frr daemon won't be able to create new files in /var/log/frr anymore, nor update the existing /var/log/frr/frr.log one. The original debian packaging works both ways, since rsyslog is privileged and can chown or switch to the frr user, but that's not the case for ubuntu.

In general:
a) users who have broken logging (potentially without realizing it) will be fixed by this update, and the fix here is the intended default behavior in ubuntu as it should have been from the start
b) users who fixed their own logging in some way might have issues with this update, mainly depending if they kept using the package-provided /var/log/frr directory, which is what we are manipulating here.

Trying to cope with all possible scenarios can quickly become complex, and introduce new bugs on its own.

I *could* perhaps gate on "log syslog" being in /etc/frr/frr.conf, and do nothing if it's not there, but it can be a slippery slope: I fear introducing more complexity.

[Other Info]
Original MP that landed this fix in Kinetic, with some discussion:
https://code.launchpad.net/~ahasenack/ubuntu/+source/frr/+git/frr/+merge/424952

[Original Description]

Out of the box, the frr package is set to use syslog for logging:
# cat frr.conf
# default to using syslog. /etc/rsyslog.d/45-frr.conf places the log in
# /var/log/frr/frr.log
...
log syslog informational

The packaging creates /var/log/frr owned by frr:
drwxr-x--- 2 frr frr 3 Jan 17 17:43 /var/log/frr

In Ubuntu, rsyslog runs as the unprivileged syslog user, not root (like in debian), which means it cannot write to this logging directory:

Jan 17 17:57:25 j-frr-mir rsyslogd: file '/var/log/frr/frr.log': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ]

Related branches

summary: - syslog lgoging does not work
+ syslog logging does not work
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

And note the logrotate file sets ownership and mode to frr:frrvty 0640, respectively, which makes rsyslog again unable to write to the log.

tags: added: server-todo
Changed in frr (Ubuntu):
status: New → Triaged
importance: Undecided → High
Changed in frr (Ubuntu Jammy):
status: New → Triaged
importance: Undecided → High
Changed in frr (Ubuntu):
assignee: nobody → Andreas Hasenack (ahasenack)
Changed in frr (Ubuntu Jammy):
assignee: nobody → Andreas Hasenack (ahasenack)
Changed in frr (Ubuntu):
status: Triaged → In Progress
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Interesting remark on the upstream docs[1]:

"(...) generic permission issues (this is often a problem under Ubuntu where permissions are dropped by default)"

1. https://www.rsyslog.com/doc/v8-stable/configuration/modules/omfile.html#caveats-known-bugs

Changed in frr (Ubuntu Impish):
status: New → Triaged
Changed in frr (Ubuntu Focal):
status: New → Triaged
Changed in frr (Ubuntu Impish):
importance: Undecided → Medium
Changed in frr (Ubuntu Focal):
importance: Undecided → Medium
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

My current approach for frr is to change the syslog config to log to /var/log/frr.log, which our rsyslog can write to. If the user wants to use file logging instead of syslog, then that can be done by changing the default frr configuration to log to /var/log/frr/frr.log and it will also just work.

I added a /var/log/frr.log to the existing logrotate snippet, so that takes care of the (default) syslog case.

To make rsyslog log to /var/log/frr/frr.log requires a bigger delta with debian, also involving maintainer scripts. I might look into that too, check how it feels.

But I hit a small snag in the current approach, because apparently watchfrr *always* logs to syslog, independent of the configuration in frr.conf. I contacted upstream[1], and will check in one or two days if that's really on purpose or a bug.

1. https://frrouting.slack.com/archives/C4T714TAQ/p1654782531144459 (sorry, login required)

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

To my knowledge, the only reasons one would have to use a logging subdirectory are:
- privileges: the daemon does its own logging, and does not run as root, so it can't write to /var/log directly and needs its own subdir, owned by it. This is the case when frr is configured to use file logging
- many log files: when there is more than one log file, it also makes sense to use a subdirectory to avoid polluting the main /var/log dir with too many individual files. That is not the case with frr. Both when file logging is used, and syslog, only one log file is produced. With syslog, one could create filters and have each routing daemon log to its own file, but that is not being done here. In fact, the rsyslog config as shipped in debian (and ubuntu) explicitly groups all logging from all daemons into one frr.log file.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Testing another approach.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package frr - 8.1-1ubuntu2

---------------
frr (8.1-1ubuntu2) kinetic; urgency=medium

  * Fix logging with Ubuntu's unprivileged rsyslog (LP: #1958162):
    - d/frr.postinst: change log files ownership
    - d/frr.logrotate: change rotated log file ownership

 -- Andreas Hasenack <email address hidden> Thu, 09 Jun 2022 12:35:58 -0300

Changed in frr (Ubuntu Kinetic):
status: In Progress → Fix Released
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Impish is EOL

Changed in frr (Ubuntu Impish):
status: Triaged → Won't Fix
Changed in frr (Ubuntu Jammy):
status: Triaged → In Progress
Changed in frr (Ubuntu Focal):
status: Triaged → In Progress
assignee: nobody → Andreas Hasenack (ahasenack)
description: updated
description: updated
description: updated
Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello Andreas, or anyone else affected,

Accepted frr into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/frr/7.2.1-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in frr (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-focal
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Focal verification

1) upgrade test

Starting with:
frr:
  Installed: 7.2.1-1
  Candidate: 7.2.1-1
  Version table:
 *** 7.2.1-1 500
        500 http://br.archive.ubuntu.com/ubuntu focal/universe amd64 Packages

/var/log/frr is empty even after a restart:
root@focal-frr-logging:~# systemctl restart frr
root@focal-frr-logging:~# ls -la /var/log/frr
total 0
drwxr-x--- 1 frr frr 0 Jan 20 2020 .
drwxrwxr-x 1 root syslog 274 Aug 9 14:50 ..

rsyslogd complains about a permission denied:
root@focal-frr-logging:~# grep frr\\.log /var/log/syslog
Aug 9 14:51:34 focal-frr-logging rsyslogd: file '/var/log/frr/frr.log': open error: Permission denied [v8.2001.0 try https://www.rsyslog.com/e/2433 ]

Upgrading to the package in proposed:
frr:
  Installed: 7.2.1-1ubuntu0.1
  Candidate: 7.2.1-1ubuntu0.1
  Version table:
 *** 7.2.1-1ubuntu0.1 500
        500 http://br.archive.ubuntu.com/ubuntu focal-proposed/universe amd64 Packages

a) /var/log/frr ownership is fixed:
root@focal-frr-logging:~# ls -lad /var/log/frr
drwxr-x--- 1 syslog adm 0 Jan 20 2020 /var/log/frr

b) after restarting frr, the log file is created as syslog:adm:
root@focal-frr-logging:~# ls -la /var/log/frr
total 0
drwxr-x--- 1 syslog adm 0 Jan 20 2020 .
drwxrwxr-x 1 root syslog 274 Aug 9 14:50 ..
root@focal-frr-logging:~# systemctl restart frr
root@focal-frr-logging:~# ls -la /var/log/frr
total 4
drwxr-x--- 1 syslog adm 14 Aug 9 14:55 .
drwxrwxr-x 1 root syslog 274 Aug 9 14:50 ..
-rw-r----- 1 syslog adm 1297 Aug 9 14:55 frr.log

c) logrotate maintains these permissions:
root@focal-frr-logging:~# logrotate -f /etc/logrotate.conf
root@focal-frr-logging:~# ls -la /var/log/frr
total 4
drwxr-x--- 1 syslog adm 38 Aug 9 14:56 .
drwxrwxr-x 1 root syslog 354 Aug 9 14:56 ..
-rw-r----- 1 syslog adm 0 Aug 9 14:56 frr.log
-rw-r----- 1 syslog adm 446 Aug 9 14:55 frr.log.1.gz

2) Fresh install test

Package from focal-proposed:
root@focal-frr-logging:~# apt-cache policy frr
frr:
  Installed: 7.2.1-1ubuntu0.1
  Candidate: 7.2.1-1ubuntu0.1
  Version table:
 *** 7.2.1-1ubuntu0.1 500
        500 http://br.archive.ubuntu.com/ubuntu focal-proposed/universe amd64 Packages

The ownership of /var/log/frr should be correct from the start (syslog:adm):

root@focal-frr-logging:~# ls -lad /var/log/frr
drwxr-x--- 1 syslog adm 0 Jul 19 20:40 /var/log/frr

Upon restarting frr, the frr.log file appears:

root@focal-frr-logging:~# systemctl restart frr
root@focal-frr-logging:~# ls -la /var/log/frr/
total 4
drwxr-x--- 1 syslog adm 14 Aug 9 17:34 .
drwxrwxr-x 1 root syslog 274 Aug 9 17:34 ..
-rw-r----- 1 syslog adm 1175 Aug 9 17:34 frr.log

logrotate is working and preserving the ownership:

root@focal-frr-logging:~# logrotate -f /etc/logrotate.conf
root@focal-frr-logging:~# ls -la /var/log/frr/
total 4
drwxr-x--- 1 syslog adm 38 Aug 9 17:35 .
drwxrwxr-x 1 root syslog 354 Aug 9 17:35 ..
-rw-r----- 1 syslog adm 0 Aug 9 17:35 frr.log
-rw-r----- 1 syslog adm 399 Aug 9 17:34 frr.log.1.gz

Focal verification succeeded.

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Steve Langasek (vorlon) wrote :

Hello Andreas, or anyone else affected,

Accepted frr into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/frr/8.1-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in frr (Ubuntu Jammy):
status: In Progress → Fix Committed
tags: added: verification-needed-jammy
Revision history for this message
Andreas Hasenack (ahasenack) wrote :
Download full text (4.0 KiB)

Jammy verification

Reproducing the problem with the release package:
frr:
  Installed: 8.1-1ubuntu1
  Candidate: 8.1-1ubuntu1
  Version table:
 *** 8.1-1ubuntu1 500
        500 http://br.archive.ubuntu.com/ubuntu jammy/main amd64 Packages

Right after install, /var/log/frr is empty even after a restart, and ownership is frr:frr:
root@jammy-frr-logging:~# systemctl restart frr
root@jammy-frr-logging:~# ls -la /var/log/frr
total 0
drwxr-x--- 1 frr frr 0 Mar 11 12:33 .
drwxrwxr-x 1 root syslog 256 Aug 15 12:58 ..
root@jammy-frr-logging:~#

And there is a permission denied error in syslog:
root@jammy-frr-logging:~# grep frr\\.log /var/log/syslog
Aug 15 12:59:49 jammy-frr-logging rsyslogd: file '/var/log/frr/frr.log': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ]

1) upgrade test

Updating to proposed:
root@jammy-frr-logging:~# apt-cache policy frr
frr:
  Installed: 8.1-1ubuntu1.1
  Candidate: 8.1-1ubuntu1.1
  Version table:
 *** 8.1-1ubuntu1.1 500
        500 http://br.archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages

a) right after the upgrade, /var/log/frr is now correctly owned by syslog:adm:
root@jammy-frr-logging:~# ls -lad /var/log/frr
drwxr-x--- 1 syslog adm 0 Mar 11 12:33 /var/log/frr

b) an frr restart creates the log file finally, owned by syslog:adm:
root@jammy-frr-logging:~# ls -la /var/log/frr
total 0
drwxr-x--- 1 syslog adm 0 Mar 11 12:33 .
drwxrwxr-x 1 root syslog 256 Aug 15 12:58 ..
root@jammy-frr-logging:~# systemctl restart frr
root@jammy-frr-logging:~# ls -la /var/log/frr
total 4
drwxr-x--- 1 syslog adm 14 Aug 15 13:03 .
drwxrwxr-x 1 root syslog 256 Aug 15 12:58 ..
-rw-r----- 1 syslog adm 1379 Aug 15 13:03 frr.log

c) logrotate works, and preserves permissions:
root@jammy-frr-logging:~# ls -la /var/log/frr
total 4
drwxr-x--- 1 syslog adm 14 Aug 15 13:03 .
drwxrwxr-x 1 root syslog 256 Aug 15 12:58 ..
-rw-r----- 1 syslog adm 1379 Aug 15 13:03 frr.log
root@jammy-frr-logging:~#
root@jammy-frr-logging:~#
root@jammy-frr-logging:~#
root@jammy-frr-logging:~# ls -la /var/log/frr
total 4
drwxr-x--- 1 syslog adm 14 Aug 15 13:03 .
drwxrwxr-x 1 root syslog 256 Aug 15 12:58 ..
-rw-r----- 1 syslog adm 1379 Aug 15 13:03 frr.log
root@jammy-frr-logging:~# logrotate -f /etc/logrotate.conf
root@jammy-frr-logging:~# ls -la /var/log/frr
total 4
drwxr-x--- 1 syslog adm 38 Aug 15 13:03 .
drwxrwxr-x 1 root syslog 336 Aug 15 13:03 ..
-rw-r----- 1 syslog adm 0 Aug 15 13:03 frr.log
-rw-r----- 1 syslog adm 507 Aug 15 13:03 frr.log.1.gz

Upgrade test succeeded.

2) Fresh install test
root@jammy-frr-logging-fresh-install:~# apt-cache policy frr
frr:
  Installed: 8.1-1ubuntu1.1
  Candidate: 8.1-1ubuntu1.1
  Version table:
 *** 8.1-1ubuntu1.1 500
        500 http://br.archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages

Ownership of /var/log/frr is correct from the start (syslog:adm):
root@jammy-frr-logging-fresh-install:~# ls -lad /var/log/frr
drwxr-x--- 1 syslog adm 0 Jul 19 20:36 /var/log/frr

After a restart, frr is ably to produce its log file via rsyslogd with correct ownership:
root@jammy-frr-logging-fresh-install:~# systemctl rest...

Read more...

tags: added: verification-done verification-done-jammy
removed: verification-needed verification-needed-jammy
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package frr - 8.1-1ubuntu1.1

---------------
frr (8.1-1ubuntu1.1) jammy; urgency=medium

  * Fix logging with Ubuntu's unprivileged rsyslog (LP: #1958162):
    - d/frr.postinst: change log files ownership
    - d/frr.logrotate: change rotated log file ownership

 -- Andreas Hasenack <email address hidden> Tue, 19 Jul 2022 17:36:23 -0300

Changed in frr (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for frr has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package frr - 7.2.1-1ubuntu0.1

---------------
frr (7.2.1-1ubuntu0.1) focal; urgency=medium

  * Fix logging with Ubuntu's unprivileged rsyslog (LP: #1958162):
    - d/frr.postinst: change log files ownership
    - d/frr.logrotate: change rotated log file ownership

 -- Andreas Hasenack <email address hidden> Tue, 19 Jul 2022 17:40:11 -0300

Changed in frr (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.