rsyslog should accumulate on all state machines

Bug #1310268 reported by John A Meinel
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
juju-core
Fix Released
High
Wayne Witzel III

Bug Description

I'm not sure what triggers what machine to forward to, but in HA mode I only see (26-juju-unit-ubuntu-31.conf):

:syslogtag, startswith, "juju-" @@ip-10-72-129-222.eu-west-1.compute.internal:6514;LongTagForwardFormat
& ~

Note there are 2 problems.

1) machine-1/2 in HA mode only copy their data to all-machines.log on the local machine, they should *also* forward that data on to the other API servers

2) machine-N only forwards its log to 1 API server. I'm not sure how it figures out which one to write to (it might just be the specific API server that it manages to connect to, or it might be the one that is considered 'master').

Because 'juju debug-log' is tailing /var/log/all-machines.log, you need to have the contents on all machines.

Tags: debug-log ha

Related branches

Revision history for this message
John A Meinel (jameinel) wrote :

Now, digging around this page:
http://www.rsyslog.com/sending-messages-to-a-remote-syslog-server/

Seems to say we could just add another line like either:
:syslogtag, startswith, "juju-" @@ip-10-72-129-222.eu-west-1.compute.internal:6514;LongTagForwardFormat
& @@ip-10-33-183-137.eu-west-1.compute.internal:6514;LongTagForwardFormat

Or
:syslogtag, startswith, "juju-" @@ip-10-72-129-222.eu-west-1.compute.internal:6514;LongTagForwardFormat
:syslogtag, startswith, "juju-" @@ip-10-33-183-137.eu-west-1.compute.internal:6514;LongTagForwardFormat

or something like that. However, I couldn't actually see it logging to 2 machines.

Note that I *did* see it logging to machine-0, so the data is still at least slightly correct.

Revision history for this message
John A Meinel (jameinel) wrote :

Note that rsyslog doesn't grab the addresses from the API, either. Instead it just always uses agentConfig.APIAddresses()[0]

Because we updated the code so that config will get updates by polling APIHostPorts and that returns addresses which might even be machine private, we risk having agents end up recording an address that ends up not routable, and we'll lose all the logs for that machine.

I think that is reasonably unlikely, since everytime I've seen it machine-local addresses end up after public addresses. Though it does mean we'll probably forward to the Public address of the API server, rather than the private one.

Changed in juju-core:
assignee: nobody → Wayne Witzel III (wwitzel3)
Changed in juju-core:
status: Triaged → In Progress
Curtis Hovey (sinzui)
Changed in juju-core:
milestone: 1.19.1 → 2.0
Curtis Hovey (sinzui)
Changed in juju-core:
milestone: 2.0 → 1.20.0
Changed in juju-core:
milestone: 1.20.0 → next-stable
Changed in juju-core:
status: In Progress → Fix Committed
Curtis Hovey (sinzui)
Changed in juju-core:
milestone: next-stable → 1.19.3
Curtis Hovey (sinzui)
Changed in juju-core:
milestone: 1.19.3 → 1.19.4
Curtis Hovey (sinzui)
Changed in juju-core:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.