Code review comment for lp:~allenap/maas/normalise-logging--bug-1639182

Revision history for this message
Gavin Panella (allenap) wrote :

> We need to have a serious discussion how we can overall improve the
> logging system.

Let me kick it off.

There has been pain in all parts of the stack: Django, Twisted, and the
standard library all do stupid things that we need to work around. I've
been able to contain the ugliness to the p.logger package.

The logs that Django and `logging` produce and those that Twisted
produce are integrated only when they hit the log file. Logs from one
subsystem are not shipped from `logging` to Twisted or vice-versa. There
is an adapter that can pipe Twisted's logs over to the standard library
but it's too lossy right now, and it's not clear that it'll help much
anyway.

Twisted has just one idea of what a log should look like. It doesn't
care about customising the textual output, though it does provide the
means to write your own log event[1] observer and so emit rot-13 logs to
a thermal printer on a pidgeon. It's neither right nor wrong, just a
different approach to that of the stdlib's `logging` package.

This bug in particular has been painful for a couple of reasons:

1. I'm trying to do it the "wrong" way with Twisted. When invoking a
   plugin via twistd the "right" way to customise logging output is to
   specify an observer via `--logger=fully.qualified.callable`.

2. As I've mentioned before, Twisted is in a transitional state between
   its "legacy" and "modern" logging subsystems. There's a lot of
   complexity behind the scenes to make this work, and significant
   change between Twisted in Xenial and Yakkety.

I've been trying to avoid #1 so far, to keep the details hidden, but no
longer. The changes between Twisted 16.0 (Xenial) and 16.4 (Yakkety)
means this is likely to be the only sane way to achieve what we want,
and this is what I'll work on now.

[1] A log *event* is a dict containing some standard keys and any number
of other pieces of information, passed between log observers. One such
observer might filter events, one might forward events to multiple
observers, one might format the event and write it out to a log file.

« Back to merge proposal