[FFe] /var/log/dmesg No Longer Being Updated

Right, /var/log/dmesg was an upstart specific file. There is little point in writing yet another log file, given that all the information is already available in the systemd journal, /var/log/kern.log, and /var/log/syslog.

Unfortunately, systemd journal, /var/log/kern.log, and /var/log/syslog are all log rotated so on long running systems, the initial boot logs are lost.

Any chance we can reconsider the decision to stop writing out /var/log/dmesg?

No.

Adjust your journal config for longer retention, if so desired.
http://manpages.ubuntu.com/manpages/xenial/man5/journald.conf.5.html

Sorry but that argument is illogical. The only retention period that would ensure that you always have the boot time kernel messages is 'infinity' and that's not a reasonable option for obvious reasons (c.f. LP #1618188).

./debian/rsyslog.dmesg.upstart was shipped by rsyslog package in Trusty and/or other releases.

It was:
start on runlevel [2345]
task
script
    savelog -q -p -c 5 /var/log/dmesg
    dmesg -s 524288 > /var/log/dmesg
    chgrp adm /var/log/dmesg
end script

Thus if such functionality is desired to be preserved, imho rsyslog should be shipping a similar configuration or unit. Note that journald pushes all entries to rsyslog, thus i would have thought it could have generated /var/log/dmesg using normal rsyslog logging configuration facilities.

I'm not sure how to configure rsyslog correctly to retain these. But I can write an equivalent systemd unit:

$ systemctl cat dmesg
# /etc/systemd/system/dmesg.service
[Unit]
Description=Save kernel messages

[Service]
Type=idle
ExecStartPre=/usr/bin/savelog -q -p -c 5 /var/log/dmesg
ExecStart=/bin/sh -c '/bin/dmesg -s 524288 > /var/log/dmesg'
ExecStartPost=/bin/chgrp adm /var/log/dmesg

[Install]
WantedBy=multi-user.target

The dmesg command can be approximated with the following journalctl command, which may have more data than dmesg ring-buffer:

$ journalctl --boot 0 --dmesg --output short-monotonic --quiet --no-pager --no-hostname

however, that is slightly incompatible format due to the extra `kernel:` stanza after the timestamp

[1195009.811669] kernel: aufs au_plink_put:436:umount[15188]: pseudo-link is not flushed

--boot 0 -> means to show output from current boot only
--dmesg -> means to show dmesg logs only
--output short-monotonic -> means to prefix things with regular [monotonic] time in dmesg like format
--quiet -> disables the headers with when the logs/boot started finished
--no-pager -> dumps to stdout, instead of less
--no-hostname -> supresses the hostname entry which is injected between time and kernel: normally

E.g. on my system journalctl command above recovers things from 0.000000 time microcode updated, instead of from very late messages.

Imho journalctl should suppress the `kernel:` as that should be implied by --dmesg flag.

Depending on how much things happen during boot, i'm not sure if dmesg should be used, or journalctl to retrieve the boot log.

Marking as invalid for systemd package. As neither it, nor upstart shipped the functionality in question. It is a regression in the rsyslog package.

Or possibly makes sense to ship this in like util-linux which ships /bin/dmesg.

Status changed to 'Confirmed' because the bug affects multiple users.

Thanks Dimitri for all the pre-work and the suggested systemd unit matching the old upstart job.
We really like that, especially as it avoids the concern of "just another ongoing log" due to being an initial one-shot task that does not continue to log the ongoing messages.
And yes - having especially the initial boot messages can be quite handy way later.

There also is not only the journal but also at /etc/rsyslog.d/50-default.conf:
 kern.* -/var/log/kern.log
But it is on logrotation (4) so it would as well over time loose the early boot messages.

In our discussion we even consider to SRU this as it is essentially a loss of functionality post-Trusty.

The journal style invocation is clearly better to not exceed ring buffers so fast on huge systems.
And especially on such it might be interesting later on what was going on.

Being lost post-trusty we might consider the more reliable way to get all messages (journalctl) more important than keeping the format absolutely the same.
There is nothing there people can rely upon yet, so the new format won't break them.

This isn't releas breaking, so as usual we'd have to wait until DD opens, then implement there and potentially SRU it back then.

As soon as something isn't urgent it falls off everyones lists :-/
I have found this old bug (again) in my notes, felt embarrassed and converted the suggestion of xnox into an MP that we can ack and push to early 19.10 then (hopefully not forgetting it again).

We probably will have to rebase then, but lets agree on an MP now by reviewing and testing functionality in Disco.

After some experiments I use systemd'd output redirection which will make it not rely on sh, nor internal redirection and nicely leaves stderr on the job/journal but the output we want in the file.

A PPA to try is at [1] and an MP to package that change is at [2].

The change actually isn't that bg even thou it is a feature. We might convert this to an FFe if review and tests are fine.

[1]: https://launchpad.net/~paelzer/+archive/ubuntu/bug-1450588-rsyslog-to-get-dmesg-log-again/
[2]: https://code.launchpad.net/~paelzer/ubuntu/+source/rsyslog/+git/rsyslog/+merge/364133

I'm not sure why you think an FFe is needed for this. It is a simple regression since move from upstart->systemd in the rsyslog package, and we must SRU this back to all the releases back.

@xnox - thanks for your ping, I agree if we would have worked on this in Vivid or at least Xenial.
But being so late I'd think to ask for an ack by the Release team is not wrong.
I thought here "better safe than sorry" somewhat applies.
OTOH fortunately the change itself isn't very invasive to an existing thing (as it is a new service), so it seems rather safe anyway.

But I can be convinced - let me ask in the Team if everyone else also thinks this wouldn't be requiring an FFe. If so we can just upload.

It would be the readdition of a feature that we lost as a feature regression nine releases ago. I don't see why the release team would object, but I can't speak for the release team. It feels like a feature to me. In the general case, ignoring feature freeze for features being readded that were dropped multiple releases ago seems disingenuous to me.

Per former discussion, better safe than sorry - added a FFe request to the description and subscribed the release-team.

From a feature freeze POV, this is ok with me. I don't know that I agree it makes sense to reintroduce this log file now after so long if the information is available elsewhere, but I don't think I need to block this w/ release team hat.

Thanks Steve,
as people have explained above the problem with the "if the information is available elsewhere" argument is, that very long living systems have rotated out the logs that contained the initial boot. This is the extra value this wants to give back to the users.

This bug was fixed in the package rsyslog - 8.32.0-1ubuntu7

---------------
rsyslog (8.32.0-1ubuntu7) disco; urgency=medium

  * Install dmesg.service (LP: #1450588) to provide /var/log/dmesg.log
    as non log-rotated log for boot time kernel messages.
    - debian/dmesg.service: new service to write /var/log/dmesg
    - debian/rsyslog.install: install dmesg.service with rsyslog

 -- Christian Ehrhardt <email address hidden> Thu, 07 Mar 2019 14:00:30 +0100