Merge lp:~stub/charms/precise/postgresql/syslog into lp:charms/postgresql
- Precise Pangolin (12.04)
- syslog
- Merge into trunk
Status: | Merged |
---|---|
Merged at revision: | 88 |
Proposed branch: | lp:~stub/charms/precise/postgresql/syslog |
Merge into: | lp:charms/postgresql |
Prerequisite: | lp:~stub/charms/precise/postgresql/bug-1278731-hot-standby-allowed-units |
Diff against target: |
271 lines (+147/-1) 7 files modified
README.md (+10/-0) config.yaml (+1/-1) hooks/hooks.py (+73/-0) metadata.yaml (+2/-0) templates/postgresql.conf.tmpl (+2/-0) templates/rsyslog_forward.conf (+19/-0) test.py (+40/-0) |
To merge this branch: | bzr merge lp:~stub/charms/precise/postgresql/syslog |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Marco Ceppi (community) | Approve | ||
Charles Butler (community) | deployment + audit | Approve | |
Review via email: mp+206176@code.launchpad.net |
Commit message
Description of the change
Ship PostgreSQL log files via syslog.
I'm not too familiar with other charms using syslog, in particular if I should be sending on port 10514 or 514, so I'd like a review from someone who has done work in this area.
The goal of this work is to connect PostgreSQL to reporting programs, and I'll be creating a pgBadger charm shortly that will consume the logs. For this use case, I also need to advertise the log_line_prefix that PostgreSQL uses to consumers, so they can decode the logs. I also added the programname required to tell logs from different units apart (IP address isn't reliable).
- 175. By Stuart Bishop
-
Basic docs for syslog
Charles Butler (lazypower) wrote : | # |
Stuart Bishop (stub) wrote : | # |
On 25 February 2014 02:27, Charles Butler <email address hidden> wrote:
> These modifications look fine to me. I've pulled them and deployed, setup a rather basic deployment test using rsyslog charm and the branch+prereq on HP-Cloud, AWS, and Local providers. All of which were properly forwarding the syslog. I haven't deployed the PGBadger service, however I see it as being ready in the queue, and I'll get to that next with additional commentary.
>
> >From what I've experienced in the past, forwarding over port 514 to your logging-nexus is perfectly acceptable. If you want to get into TLS wrapping your rsyslog forwarding, that's typically when you go to port 1999, so I'm a bit confused on the port 10514 dialogue above. Can you shed some light on why that was of a concern?
Its a concern as I'm cargo culting code from elsewhere, rather than
follow a specification. If there actually is a specification for the
syslog interface somewhere please let me know ;)
From what I can tell with the rsyslog and rsyslog-forarder charms, the
'open' port is 514 (TCP+UDP) but the 'closed' one used for syslog
forwarding is 10514 (TCP only). I also see nothing about TLS in those
charms, and nothing on port 1999.
--
Stuart Bishop <email address hidden>
Charles Butler (lazypower) wrote : | # |
Ah, i reached out to my prior co-workers about the TLS security wrappers
and port 1999, that was a convention we coined. its not part of the spec.
Apparently getting transport security on our syslog forwarding constituted
partying like its 1999, hence the port chosen.
On Tue, Feb 25, 2014 at 12:42 AM, Stuart Bishop <<email address hidden>
> wrote:
> On 25 February 2014 02:27, Charles Butler <email address hidden>
> wrote:
>
> > These modifications look fine to me. I've pulled them and deployed,
> setup a rather basic deployment test using rsyslog charm and the
> branch+prereq on HP-Cloud, AWS, and Local providers. All of which were
> properly forwarding the syslog. I haven't deployed the PGBadger service,
> however I see it as being ready in the queue, and I'll get to that next
> with additional commentary.
> >
> > >From what I've experienced in the past, forwarding over port 514 to
> your logging-nexus is perfectly acceptable. If you want to get into TLS
> wrapping your rsyslog forwarding, that's typically when you go to port
> 1999, so I'm a bit confused on the port 10514 dialogue above. Can you shed
> some light on why that was of a concern?
>
> Its a concern as I'm cargo culting code from elsewhere, rather than
> follow a specification. If there actually is a specification for the
> syslog interface somewhere please let me know ;)
>
> >From what I can tell with the rsyslog and rsyslog-forarder charms, the
> 'open' port is 514 (TCP+UDP) but the 'closed' one used for syslog
> forwarding is 10514 (TCP only). I also see nothing about TLS in those
> charms, and nothing on port 1999.
>
>
> --
> Stuart Bishop <email address hidden>
>
>
> https:/
> You are reviewing the proposed merge of
> lp:~stub/charms/precise/postgresql/syslog into lp:charms/postgresql.
>
- 176. By Stuart Bishop
-
Merged bug-1278731-hot-standby-
allowed- units into syslog. - 177. By Stuart Bishop
-
Merged bug-1278731-hot-standby-
allowed- units into syslog.
Preview Diff
1 | === modified file 'README.md' | |||
2 | --- README.md 2014-02-05 12:29:09 +0000 | |||
3 | +++ README.md 2014-02-13 13:49:35 +0000 | |||
4 | @@ -130,6 +130,16 @@ | |||
5 | 130 | - `allowed-units`: space separated list of allowed clients (unit name). You | 130 | - `allowed-units`: space separated list of allowed clients (unit name). You |
6 | 131 | should check this to determine if you can connect to the database yet. | 131 | should check this to determine if you can connect to the database yet. |
7 | 132 | 132 | ||
8 | 133 | ## During syslog-relation-changed | ||
9 | 134 | |||
10 | 135 | ### the postgresql service provides: | ||
11 | 136 | |||
12 | 137 | - `programname`: the syslog 'programname' identifying this unit's | ||
13 | 138 | PostgreSQL logs. | ||
14 | 139 | - `log_line_prefix`: the 'log_line_prefix' setting for the PostgreSQL | ||
15 | 140 | service. | ||
16 | 141 | |||
17 | 142 | |||
18 | 133 | ## For replicated database support | 143 | ## For replicated database support |
19 | 134 | 144 | ||
20 | 135 | A PostgreSQL service may contain multiple units (a single master, and | 145 | A PostgreSQL service may contain multiple units (a single master, and |
21 | 136 | 146 | ||
22 | === modified file 'config.yaml' | |||
23 | --- config.yaml 2014-01-23 11:41:03 +0000 | |||
24 | +++ config.yaml 2014-02-13 13:49:35 +0000 | |||
25 | @@ -90,7 +90,7 @@ | |||
26 | 90 | type: boolean | 90 | type: boolean |
27 | 91 | description: Log disconnections | 91 | description: Log disconnections |
28 | 92 | log_line_prefix: | 92 | log_line_prefix: |
30 | 93 | default: "%t " | 93 | default: "%t [%p]: [%l-1] db=%d,user=%u " |
31 | 94 | type: string | 94 | type: string |
32 | 95 | description: | | 95 | description: | |
33 | 96 | special values: | 96 | special values: |
34 | 97 | 97 | ||
35 | === modified file 'hooks/hooks.py' | |||
36 | --- hooks/hooks.py 2014-02-13 13:49:35 +0000 | |||
37 | +++ hooks/hooks.py 2014-02-13 13:49:35 +0000 | |||
38 | @@ -14,6 +14,7 @@ | |||
39 | 14 | import subprocess | 14 | import subprocess |
40 | 15 | import sys | 15 | import sys |
41 | 16 | from tempfile import NamedTemporaryFile | 16 | from tempfile import NamedTemporaryFile |
42 | 17 | from textwrap import dedent | ||
43 | 17 | import time | 18 | import time |
44 | 18 | import yaml | 19 | import yaml |
45 | 19 | from yaml.constructor import ConstructorError | 20 | from yaml.constructor import ConstructorError |
46 | @@ -482,6 +483,9 @@ | |||
47 | 482 | config_file, pg_config, | 483 | config_file, pg_config, |
48 | 483 | owner="postgres", group="postgres", perms=0600) | 484 | owner="postgres", group="postgres", perms=0600) |
49 | 484 | 485 | ||
50 | 486 | # Create or update files included from postgresql.conf. | ||
51 | 487 | configure_log_destination(os.path.dirname(config_file)) | ||
52 | 488 | |||
53 | 485 | local_state['saved_config'] = config_data | 489 | local_state['saved_config'] = config_data |
54 | 486 | local_state.save() | 490 | local_state.save() |
55 | 487 | 491 | ||
56 | @@ -999,6 +1003,11 @@ | |||
57 | 999 | postgresql_restart() | 1003 | postgresql_restart() |
58 | 1000 | postgresql_reload_or_restart() | 1004 | postgresql_reload_or_restart() |
59 | 1001 | 1005 | ||
60 | 1006 | # In case the log_line_prefix has changed, inform syslog consumers. | ||
61 | 1007 | for relid in hookenv.relation_ids('syslog'): | ||
62 | 1008 | hookenv.relation_set( | ||
63 | 1009 | relid, log_line_prefix=hookenv.config('log_line_prefix')) | ||
64 | 1010 | |||
65 | 1002 | 1011 | ||
66 | 1003 | @hooks.hook() | 1012 | @hooks.hook() |
67 | 1004 | def install(run_pre=True): | 1013 | def install(run_pre=True): |
68 | @@ -2194,6 +2203,69 @@ | |||
69 | 2194 | host.service_reload('nagios-nrpe-server') | 2203 | host.service_reload('nagios-nrpe-server') |
70 | 2195 | 2204 | ||
71 | 2196 | 2205 | ||
72 | 2206 | @hooks.hook() | ||
73 | 2207 | def syslog_relation_changed(): | ||
74 | 2208 | configure_log_destination(_get_postgresql_config_dir()) | ||
75 | 2209 | postgresql_reload() | ||
76 | 2210 | |||
77 | 2211 | # We extend the syslog interface by exposing the log_line_prefix. | ||
78 | 2212 | # This is required so consumers of the PostgreSQL logs can decode | ||
79 | 2213 | # them. Consumers not smart enough to cope with arbitrary prefixes | ||
80 | 2214 | # can at a minimum abort if they detect it is set to something they | ||
81 | 2215 | # cannot support. Similarly, inform the consumer of the programname | ||
82 | 2216 | # we are using so they can tell one units log messages from another. | ||
83 | 2217 | hookenv.relation_set( | ||
84 | 2218 | log_line_prefix=hookenv.config('log_line_prefix'), | ||
85 | 2219 | programname=sanitize(hookenv.local_unit())) | ||
86 | 2220 | |||
87 | 2221 | template_path = "{0}/templates/rsyslog_forward.conf".format( | ||
88 | 2222 | hookenv.charm_dir()) | ||
89 | 2223 | rsyslog_conf = Template(open(template_path).read()).render( | ||
90 | 2224 | local_unit=sanitize(hookenv.local_unit()), | ||
91 | 2225 | raw_local_unit=hookenv.local_unit(), | ||
92 | 2226 | raw_remote_unit=hookenv.remote_unit(), | ||
93 | 2227 | remote_addr=hookenv.relation_get('private-address')) | ||
94 | 2228 | host.write_file(rsyslog_conf_path(hookenv.remote_unit()), rsyslog_conf) | ||
95 | 2229 | run(['service', 'rsyslog', 'restart']) | ||
96 | 2230 | |||
97 | 2231 | |||
98 | 2232 | @hooks.hook() | ||
99 | 2233 | def syslog_relation_departed(): | ||
100 | 2234 | configure_log_destination(_get_postgresql_config_dir()) | ||
101 | 2235 | postgresql_reload() | ||
102 | 2236 | os.unlink(rsyslog_conf_path(hookenv.remote_unit())) | ||
103 | 2237 | run(['service', 'rsyslog', 'restart']) | ||
104 | 2238 | |||
105 | 2239 | |||
106 | 2240 | def configure_log_destination(config_dir): | ||
107 | 2241 | """Set the log_destination PostgreSQL config flag appropriately""" | ||
108 | 2242 | # We currently support either 'standard' logs (the files in | ||
109 | 2243 | # /var/log/postgresql), or syslog + 'standard' logs. This should | ||
110 | 2244 | # grow more complex in the future, as the local logs will be | ||
111 | 2245 | # redundant if you are using syslog for log aggregation, and we | ||
112 | 2246 | # probably want to add csvlog in the future. Note that csvlog | ||
113 | 2247 | # requires switching from 'Debian' log redirection and rotation to | ||
114 | 2248 | # the PostgreSQL builtin facilities. | ||
115 | 2249 | logdest_conf_path = os.path.join(config_dir, 'juju_logdest.conf') | ||
116 | 2250 | logdest_conf = open(logdest_conf_path, 'w') | ||
117 | 2251 | if hookenv.relation_ids('syslog'): | ||
118 | 2252 | # For syslog, we change the ident from the default of 'postgres' | ||
119 | 2253 | # to the unit name to allow remote services to easily identify | ||
120 | 2254 | # and filter which unit messages are from. We don't use IP | ||
121 | 2255 | # address for this as it is not necessarily unique. | ||
122 | 2256 | logdest_conf.write(dedent("""\ | ||
123 | 2257 | log_destination='stderr,syslog' | ||
124 | 2258 | syslog_ident={0} | ||
125 | 2259 | """).format(sanitize(hookenv.local_unit()))) | ||
126 | 2260 | else: | ||
127 | 2261 | open(logdest_conf_path, 'w').write("log_destination='stderr'") | ||
128 | 2262 | |||
129 | 2263 | |||
130 | 2264 | def rsyslog_conf_path(remote_unit): | ||
131 | 2265 | return '/etc/rsyslog.d/juju-{0}-{1}.conf'.format( | ||
132 | 2266 | sanitize(hookenv.local_unit()), sanitize(remote_unit)) | ||
133 | 2267 | |||
134 | 2268 | |||
135 | 2197 | def _get_postgresql_config_dir(config_data=None): | 2269 | def _get_postgresql_config_dir(config_data=None): |
136 | 2198 | """ Return the directory path of the postgresql configuration files. """ | 2270 | """ Return the directory path of the postgresql configuration files. """ |
137 | 2199 | if config_data is None: | 2271 | if config_data is None: |
138 | @@ -2202,6 +2274,7 @@ | |||
139 | 2202 | cluster_name = config_data['cluster_name'] | 2274 | cluster_name = config_data['cluster_name'] |
140 | 2203 | return os.path.join("/etc/postgresql", version, cluster_name) | 2275 | return os.path.join("/etc/postgresql", version, cluster_name) |
141 | 2204 | 2276 | ||
142 | 2277 | |||
143 | 2205 | ############################################################################### | 2278 | ############################################################################### |
144 | 2206 | # Global variables | 2279 | # Global variables |
145 | 2207 | ############################################################################### | 2280 | ############################################################################### |
146 | 2208 | 2281 | ||
147 | === added symlink 'hooks/syslog-relation-changed' | |||
148 | === target is u'hooks.py' | |||
149 | === added symlink 'hooks/syslog-relation-departed' | |||
150 | === target is u'hooks.py' | |||
151 | === modified file 'metadata.yaml' | |||
152 | --- metadata.yaml 2013-10-08 22:44:28 +0000 | |||
153 | +++ metadata.yaml 2014-02-13 13:49:35 +0000 | |||
154 | @@ -26,6 +26,8 @@ | |||
155 | 26 | requires: | 26 | requires: |
156 | 27 | persistent-storage: | 27 | persistent-storage: |
157 | 28 | interface: directory-path | 28 | interface: directory-path |
158 | 29 | syslog: | ||
159 | 30 | interface: syslog | ||
160 | 29 | peers: | 31 | peers: |
161 | 30 | replication: | 32 | replication: |
162 | 31 | interface: pgreplication | 33 | interface: pgreplication |
163 | 32 | 34 | ||
164 | === modified file 'templates/postgresql.conf.tmpl' | |||
165 | --- templates/postgresql.conf.tmpl 2014-01-30 16:44:33 +0000 | |||
166 | +++ templates/postgresql.conf.tmpl 2014-02-13 13:49:35 +0000 | |||
167 | @@ -88,6 +88,8 @@ | |||
168 | 88 | # ERROR REPORTING AND LOGGING | 88 | # ERROR REPORTING AND LOGGING |
169 | 89 | #------------------------------------------------------------------------------ | 89 | #------------------------------------------------------------------------------ |
170 | 90 | 90 | ||
171 | 91 | include 'juju_logdest.conf' | ||
172 | 92 | |||
173 | 91 | {% if log_min_duration_statement != "" -%} | 93 | {% if log_min_duration_statement != "" -%} |
174 | 92 | log_min_duration_statement = {{log_min_duration_statement}} | 94 | log_min_duration_statement = {{log_min_duration_statement}} |
175 | 93 | {% endif -%} | 95 | {% endif -%} |
176 | 94 | 96 | ||
177 | === added file 'templates/rsyslog_forward.conf' | |||
178 | --- templates/rsyslog_forward.conf 1970-01-01 00:00:00 +0000 | |||
179 | +++ templates/rsyslog_forward.conf 2014-02-13 13:49:35 +0000 | |||
180 | @@ -0,0 +1,19 @@ | |||
181 | 1 | ## | ||
182 | 2 | ## Managed by juju | ||
183 | 3 | ## | ||
184 | 4 | |||
185 | 5 | # This file is installed into /etc/rsyslog.d to forward the unit's | ||
186 | 6 | # PostgreSQL logs via syslog to another service. | ||
187 | 7 | |||
188 | 8 | # This configuration is pretty dumb, and was cargo culted from the | ||
189 | 9 | # rsyslog-forwarder charm. The client really needs to inform the | ||
190 | 10 | # server what sort of service it requires (eg. reliability), but that | ||
191 | 11 | # involves extending and documenting the juju common syslog interface. | ||
192 | 12 | |||
193 | 13 | $ActionQueueType LinkedList # use asynchronous processing | ||
194 | 14 | $ActionQueueFileName @@service@@ # set file name, also enables disk mode | ||
195 | 15 | $ActionResumeRetryCount -1 # infinite retries on insert failure | ||
196 | 16 | $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down | ||
197 | 17 | |||
198 | 18 | # Forward all PostgreSQL unit {{raw_local_unit}} logs to {{raw_remote_unit}} | ||
199 | 19 | if $programname == '{{local_unit}}' then @@{{remote_addr}}:10514 | ||
200 | 0 | 20 | ||
201 | === modified file 'test.py' | |||
202 | --- test.py 2014-02-13 13:49:35 +0000 | |||
203 | +++ test.py 2014-02-13 13:49:35 +0000 | |||
204 | @@ -14,6 +14,7 @@ | |||
205 | 14 | import subprocess | 14 | import subprocess |
206 | 15 | import time | 15 | import time |
207 | 16 | import unittest | 16 | import unittest |
208 | 17 | import uuid | ||
209 | 17 | 18 | ||
210 | 18 | import fixtures | 19 | import fixtures |
211 | 19 | import psycopg2 | 20 | import psycopg2 |
212 | @@ -137,6 +138,10 @@ | |||
213 | 137 | result = self.sql('SELECT TRUE') | 138 | result = self.sql('SELECT TRUE') |
214 | 138 | self.assertEqual(result, [['t']]) | 139 | self.assertEqual(result, [['t']]) |
215 | 139 | 140 | ||
216 | 141 | # Confirm that the relation tears down without errors. | ||
217 | 142 | self.juju.do(['destroy-relation', 'postgresql:db', 'psql:db']) | ||
218 | 143 | self.juju.wait_until_ready() | ||
219 | 144 | |||
220 | 140 | def test_streaming_replication(self): | 145 | def test_streaming_replication(self): |
221 | 141 | self.juju.deploy( | 146 | self.juju.deploy( |
222 | 142 | TEST_CHARM, 'postgresql', num_units=2, config=self.pg_config) | 147 | TEST_CHARM, 'postgresql', num_units=2, config=self.pg_config) |
223 | @@ -163,6 +168,11 @@ | |||
224 | 163 | result = self.sql('SELECT TRUE', dbname='postgres') | 168 | result = self.sql('SELECT TRUE', dbname='postgres') |
225 | 164 | self.assertEqual(result, [['t']]) | 169 | self.assertEqual(result, [['t']]) |
226 | 165 | 170 | ||
227 | 171 | # Confirm that the relation tears down without errors. | ||
228 | 172 | self.juju.do([ | ||
229 | 173 | 'destroy-relation', 'postgresql:db-admin', 'psql:db-admin']) | ||
230 | 174 | self.juju.wait_until_ready() | ||
231 | 175 | |||
232 | 166 | def is_master(self, postgres_unit, dbname=None): | 176 | def is_master(self, postgres_unit, dbname=None): |
233 | 167 | is_master = self.sql( | 177 | is_master = self.sql( |
234 | 168 | 'SELECT NOT pg_is_in_recovery()', | 178 | 'SELECT NOT pg_is_in_recovery()', |
235 | @@ -425,6 +435,36 @@ | |||
236 | 425 | ''') | 435 | ''') |
237 | 426 | self.assertEqual(result, [['f', 'f', 'f']]) | 436 | self.assertEqual(result, [['f', 'f', 'f']]) |
238 | 427 | 437 | ||
239 | 438 | def test_syslog(self): | ||
240 | 439 | # Deploy 2 PostgreSQL units and 2 rsyslog units to ensure that | ||
241 | 440 | # log messages from every source reach every sink. | ||
242 | 441 | self.pg_config['log_min_duration_statement'] = 0 # Log all statements | ||
243 | 442 | self.juju.deploy( | ||
244 | 443 | TEST_CHARM, 'postgresql', num_units=2, config=self.pg_config) | ||
245 | 444 | self.juju.deploy(PSQL_CHARM, 'psql') | ||
246 | 445 | self.juju.do(['add-relation', 'postgresql:db', 'psql:db']) | ||
247 | 446 | self.juju.deploy('cs:rsyslog', 'rsyslog', num_units=2) | ||
248 | 447 | self.juju.do([ | ||
249 | 448 | 'add-relation', 'postgresql:syslog', 'rsyslog:aggregator']) | ||
250 | 449 | self.juju.wait_until_ready() | ||
251 | 450 | |||
252 | 451 | token = str(uuid.uuid1()) | ||
253 | 452 | |||
254 | 453 | self.sql("SELECT 'master {}'".format(token), 'master') | ||
255 | 454 | self.sql("SELECT 'hot standby {}'".format(token), 'hot standby') | ||
256 | 455 | time.sleep(2) | ||
257 | 456 | |||
258 | 457 | for runit in ['rsyslog/0', 'rsyslog/1']: | ||
259 | 458 | cmd = ['juju', 'ssh', runit, 'tail -100 /var/log/syslog'] | ||
260 | 459 | out = run(self, cmd) | ||
261 | 460 | self.failUnless('master {}'.format(token) in out) | ||
262 | 461 | self.failUnless('hot standby {}'.format(token) in out) | ||
263 | 462 | |||
264 | 463 | # Confirm that the relation tears down correctly. | ||
265 | 464 | self.juju.do([ | ||
266 | 465 | 'destroy-relation', 'postgresql:syslog', 'rsyslog:aggregator']) | ||
267 | 466 | self.juju.wait_until_ready() | ||
268 | 467 | |||
269 | 428 | 468 | ||
270 | 429 | class PG91Tests( | 469 | class PG91Tests( |
271 | 430 | PostgreSQLCharmBaseTestCase, | 470 | PostgreSQLCharmBaseTestCase, |
Stuart,
These modifications look fine to me. I've pulled them and deployed, setup a rather basic deployment test using rsyslog charm and the branch+prereq on HP-Cloud, AWS, and Local providers. All of which were properly forwarding the syslog. I haven't deployed the PGBadger service, however I see it as being ready in the queue, and I'll get to that next with additional commentary.
From what I've experienced in the past, forwarding over port 514 to your logging-nexus is perfectly acceptable. If you want to get into TLS wrapping your rsyslog forwarding, that's typically when you go to port 1999, so I'm a bit confused on the port 10514 dialogue above. Can you shed some light on why that was of a concern?
I'll be pushing this review to another charmer for an additional +1 review.
Thank you for the submission, keep up the excellent work!