Merge lp:~chad.smith/charms/precise/postgresql/postgresql-manual-tune-kernel-params-fix into lp:charms/postgresql

Proposed by Chad Smith
Status: Merged
Merged at revision: 77
Proposed branch: lp:~chad.smith/charms/precise/postgresql/postgresql-manual-tune-kernel-params-fix
Merge into: lp:charms/postgresql
Diff against target: 1001 lines (+545/-86)
7 files modified
Makefile (+9/-0)
README.md (+14/-0)
config.yaml (+23/-15)
hooks/hooks.py (+143/-70)
hooks/test_hooks.py (+354/-0)
revision (+1/-0)
templates/postgresql.conf.tmpl (+1/-1)
To merge this branch: bzr merge lp:~chad.smith/charms/precise/postgresql/postgresql-manual-tune-kernel-params-fix
Reviewer Review Type Date Requested Status
Stuart Bishop Approve
David Britton (community) Approve
Review via email: mp+200730@code.launchpad.net

Description of the change

Work in progress branch for initial review. Please let me know what you guys think before I go too far down this route. I didn't want to cause any issues by using local "trial" tests instead of juju integration tests, but it seems with a timeout of 15 minutes, the 5 existing integration tests were taking 1hr 15 mins to complete (during error conditions)

This branch does the following:
  - adds basic framework for functional unit tests of hooks.
         - these tests don't involve time-consuming juju unit deployment like the existing integration tests and provide
           local testing of hook functions
  - fixes some of the kernel_shmall, and kernel_shmmax config parameters, so ensure they properly written when
    performance-tuning == "manual"
  - Some cleanup to reduce use of globals and reduce the use of local variable names that match global variable names to avoid confusion. Removing globals also better enabled the local unit testing framework by avoiding a global call to
    hookenv.config() during module import

If we think this is a good initial approach, it should be straight forward for us to add more unit tests to get coverage of most hook functions defined within. All thoughts appreciated.

To post a comment you must log in.
87. By Chad Smith on 2014-01-07

uncomment integration tests

88. By Chad Smith on 2014-01-07

linting

89. By Chad Smith on 2014-01-07

revision bump

90. By Chad Smith on 2014-01-07

include python-psutil in install dependencies

David Britton (dpb) wrote :

Makefile:

[0] make two targets, test and integration-test
[1] remove ls-lint, as that is a landscape specific bzr plugin

91. By Chad Smith on 2014-01-07

silly comma, I've got you this time

David Britton (dpb) wrote :

[3] put in Makefile target for "devel" or "build-devel", where it will install appropriate packages

like: sudo apt-get install -y python-mocker python-twisted-core

[4] get rid of the revision file

I agree with the general approach. It's very light, but I think can be expanded as you go forward.

+1

review: Approve
Stuart Bishop (stub) wrote :

This all looks good. The cleanups are much needed, as are the retrofitted unittests. And those globals... thanks for killing some more of them.

The checks for "wal_buffers = -1" are unnecessary, as this setting is not related to replication.

review: Approve
Stuart Bishop (stub) wrote :

The file 'revision' needs to be removed before landing

Stuart Bishop (stub) wrote :

Integration tests pass with the local provider (but still flaky due to the usual juju race conditions).

Merged.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added file 'Makefile'
2--- Makefile 1970-01-01 00:00:00 +0000
3+++ Makefile 2014-01-07 21:59:36 +0000
4@@ -0,0 +1,9 @@
5+CHARM_DIR := $(shell pwd)
6+
7+test:
8+ cd hooks && CHARM_DIR=$(CHARM_DIR) trial test_hooks.py
9+ echo "Integration tests using Juju deployed units"
10+ TEST_TIMEOUT=900 ./test.py -v
11+
12+lint:
13+ bzr ls-lint
14
15=== modified file 'README.md'
16--- README.md 2013-12-23 10:35:30 +0000
17+++ README.md 2014-01-07 21:59:36 +0000
18@@ -134,3 +134,17 @@
19 - `state`: 'standalone', 'master' or 'hot standby'
20 - `allowed-units`: space separated list of allowed clients (unit name).
21 You should check this to determine if you can connect to the database yet.
22+
23+### For clustered support
24+In order for client charms to support replication the client will need to be
25+aware when relation-list reports > 1 unit of postgresql related:
26+ - When > 1 postgresql units are related:
27+ - if the client charm needs database write access, they will ignore
28+ all "standalone", "hot standby" and "failover" states as those will
29+ likely come from a standby unit (read-only) during standby install,
30+ setup or teardown
31+ - If read-only access is needed for a client, acting on
32+ db-admin-relation-changed "hot standby" state will provide you with a
33+ readonly replicated copy of the db
34+ - When 1 postgresql unit is related:
35+ - watch for updates to the db-admin-relation-changed with "standalone" state
36
37=== modified file 'config.yaml'
38--- config.yaml 2013-11-13 10:07:41 +0000
39+++ config.yaml 2014-01-07 21:59:36 +0000
40@@ -160,7 +160,7 @@
41 type: boolean
42 description: |
43 Hot standby or warm standby. When True, queries can be run against
44- the database is in recovery or standby mode (ie. replicated).
45+ the database when in recovery or standby mode (ie. replicated).
46 Overridden by juju when master/slave relations are used.
47 hot_standby_feedback:
48 default: False
49@@ -230,28 +230,40 @@
50 type: string
51 description: |
52 Possible values here are "auto" or "manual". If we set "auto" then the
53- charm will attempt to automatically tune all the performance paramaters
54- as below. If manual, then it will use the defaults below unless
55- overridden. "auto" gathers information about the node you're deployed
56- on and tries to make intelligent guesses about what tuning parameters
57- to set based on available RAM and CPU under the assumption that it's
58- the only significant service running on this node.
59+ charm will attempt to automatically tune all the performance parameters
60+ for kernel_shmall, kernel_shmmax, shared_buffers and
61+ effective_cache_size below, unless those config values are explicitly
62+ set. If manual, then it will use the defaults below unless set.
63+ "auto" gathers information about the node on which you are deployed and
64+ tries to make intelligent guesses about what tuning parameters to set
65+ based on available RAM and CPU under the assumption that it's the only
66+ significant service running on this node.
67 kernel_shmall:
68 default: 0
69 type: int
70- description: Kernel/shmall
71+ description: Total amount of shared memory available, in bytes.
72 kernel_shmmax:
73 default: 0
74 type: int
75- description: Kernel/shmmax
76+ description: The maximum size, in bytes, of a shared memory segment.
77 shared_buffers:
78 default: ""
79 type: string
80- description: Shared buffers
81+ description: |
82+ The amount of memory the database server uses for shared memory
83+ buffers. This string should be of the format '###MB'.
84+ effective_cache_size:
85+ default: ""
86+ type: string
87+ description: |
88+ Effective cache size is an estimate of how much memory is available for
89+ disk caching within the database. (50% to 75% of system memory). This
90+ string should be of the format '###MB'.
91 temp_buffers:
92 default: "1MB"
93 type: string
94- description: Temp buffers
95+ description: |
96+ The maximum number of temporary buffers used by each database session.
97 wal_buffers:
98 default: "-1"
99 type: string
100@@ -265,10 +277,6 @@
101 default: 4.0
102 type: float
103 description: Random page cost
104- effective_cache_size:
105- default: ""
106- type: string
107- description: Effective cache size
108 #------------------------------------------------------------------------
109 # Volume management
110 # volume-map, volume-dev_regexp are only used
111
112=== modified file 'hooks/hooks.py'
113--- hooks/hooks.py 2013-12-23 10:35:30 +0000
114+++ hooks/hooks.py 2014-01-07 21:59:36 +0000
115@@ -48,7 +48,7 @@
116 '''
117 myname = hookenv.local_unit().replace('/', '-')
118 ts = time.strftime('%Y-%m-%d %H:%M:%S', time.gmtime())
119- with open('/var/log/juju/{}-debug.log'.format(myname), 'a') as f:
120+ with open('{}/{}-debug.log'.format(juju_log_dir, myname), 'a') as f:
121 f.write('{} {}: {}\n'.format(ts, lvl, msg))
122 hookenv.log(msg, lvl)
123
124@@ -121,7 +121,7 @@
125 def volume_get_volid_from_volume_map():
126 volume_map = {}
127 try:
128- volume_map = yaml.load(config_data['volume-map'].strip())
129+ volume_map = yaml.load(hookenv.config('volume-map').strip())
130 if volume_map:
131 return volume_map.get(os.environ['JUJU_UNIT_NAME'])
132 except ConstructorError as e:
133@@ -154,7 +154,7 @@
134 # @returns volid
135 # None config state is invalid - we should not serve
136 def volume_get_volume_id():
137- ephemeral_storage = config_data['volume-ephemeral-storage']
138+ ephemeral_storage = hookenv.config('volume-ephemeral-storage')
139 volid = volume_get_volid_from_volume_map()
140 juju_unit_name = hookenv.local_unit()
141 if ephemeral_storage in [True, 'yes', 'Yes', 'true', 'True']:
142@@ -195,6 +195,7 @@
143
144
145 def postgresql_autostart(enabled):
146+ postgresql_config_dir = _get_postgresql_config_dir()
147 startup_file = os.path.join(postgresql_config_dir, 'start.conf')
148 if enabled:
149 log("Enabling PostgreSQL startup in {}".format(startup_file))
150@@ -202,8 +203,8 @@
151 else:
152 log("Disabling PostgreSQL startup in {}".format(startup_file))
153 mode = 'manual'
154- contents = Template(open("templates/start_conf.tmpl").read()).render(
155- {'mode': mode})
156+ template_file = "{}/templates/start_conf.tmpl".format(hookenv.charm_dir())
157+ contents = Template(open(template_file).read()).render({'mode': mode})
158 host.write_file(
159 startup_file, contents, 'postgres', 'postgres', perms=0o644)
160
161@@ -229,7 +230,7 @@
162 if status != 0:
163 return False
164 # e.g. output: "Running clusters: 9.1/main"
165- vc = "%s/%s" % (config_data["version"], config_data["cluster_name"])
166+ vc = "%s/%s" % (hookenv.config("version"), hookenv.config("cluster_name"))
167 return vc in output.decode('utf8').split()
168
169
170@@ -253,7 +254,7 @@
171 # success = host.service_restart('postgresql')
172 try:
173 run('pg_ctlcluster -force {version} {cluster_name} '
174- 'restart'.format(**config_data))
175+ 'restart'.format(**hookenv.config()))
176 success = True
177 except subprocess.CalledProcessError:
178 success = False
179@@ -327,11 +328,11 @@
180 return success
181
182
183-def get_service_port(postgresql_config):
184+def get_service_port(config_file):
185 '''Return the port PostgreSQL is listening on.'''
186- if not os.path.exists(postgresql_config):
187+ if not os.path.exists(config_file):
188 return None
189- postgresql_config = open(postgresql_config, 'r').read()
190+ postgresql_config = open(config_file, 'r').read()
191 port = re.search("port.*=(.*)", postgresql_config).group(1).strip()
192 try:
193 return int(port)
194@@ -339,14 +340,26 @@
195 return None
196
197
198-def create_postgresql_config(postgresql_config):
199+def _get_system_ram():
200+ """ Return the system ram in Megabytes """
201+ import psutil
202+ return psutil.phymem_usage()[0] / (1024 ** 2)
203+
204+
205+def _get_page_size():
206+ """ Return the operating system's configured PAGE_SIZE """
207+ return int(run("getconf PAGE_SIZE")) # frequently 4096
208+
209+
210+def create_postgresql_config(config_file):
211 '''Create the postgresql.conf file'''
212+ config_data = hookenv.config()
213 if config_data["performance_tuning"] == "auto":
214 # Taken from:
215 # http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
216 # num_cpus is not being used ... commenting it out ... negronjl
217 #num_cpus = run("cat /proc/cpuinfo | grep processor | wc -l")
218- total_ram = run("free -m | grep Mem | awk '{print $2}'")
219+ total_ram = _get_system_ram()
220 if not config_data["effective_cache_size"]:
221 config_data["effective_cache_size"] = \
222 "%sMB" % (int(int(total_ram) * 0.75),)
223@@ -357,15 +370,22 @@
224 else:
225 config_data["shared_buffers"] = \
226 "%sMB" % (int(int(total_ram) * 0.15),)
227- # XXX: This is very messy - should probably be a subordinate charm
228- conf_file = open("/etc/sysctl.d/50-postgresql.conf", "w")
229- conf_file.write("kernel.sem = 250 32000 100 1024\n")
230- conf_file.write("kernel.shmall = %s\n" %
231- ((int(total_ram) * 1024 * 1024) + 1024),)
232- conf_file.write("kernel.shmmax = %s\n" %
233- ((int(total_ram) * 1024 * 1024) + 1024),)
234- conf_file.close()
235- run("sysctl -p /etc/sysctl.d/50-postgresql.conf")
236+ config_data["kernel_shmmax"] = (int(total_ram) * 1024 * 1024) + 1024
237+ config_data["kernel_shmall"] = config_data["kernel_shmmax"]
238+
239+ # XXX: This is very messy - should probably be a subordinate charm
240+ lines = ["kernel.sem = 250 32000 100 1024\n"]
241+ if config_data["kernel_shmall"] > 0:
242+ # Convert config kernel_shmall (bytes) to pages
243+ page_size = _get_page_size()
244+ num_pages = config_data["kernel_shmall"] / page_size
245+ if (config_data["kernel_shmall"] % page_size) > 0:
246+ num_pages += 1
247+ lines.append("kernel.shmall = %s\n" % num_pages)
248+ if config_data["kernel_shmmax"] > 0:
249+ lines.append("kernel.shmmax = %s\n" % config_data["kernel_shmmax"])
250+ host.write_file(postgresql_sysctl, ''.join(lines), perms=0600)
251+ run("sysctl -p {}".format(postgresql_sysctl))
252
253 # If we are replicating, some settings may need to be overridden to
254 # certain minimum levels.
255@@ -383,28 +403,31 @@
256
257 # Send config data to the template
258 # Return it as pg_config
259+ charm_dir = hookenv.charm_dir()
260+ template_file = "{}/templates/postgresql.conf.tmpl".format(charm_dir)
261 pg_config = Template(
262- open("templates/postgresql.conf.tmpl").read()).render(config_data)
263+ open(template_file).read()).render(config_data)
264 host.write_file(
265- postgresql_config, pg_config,
266+ config_file, pg_config,
267 owner="postgres", group="postgres", perms=0600)
268
269 local_state['saved_config'] = config_data
270 local_state.save()
271
272
273-def create_postgresql_ident(postgresql_ident):
274+def create_postgresql_ident(output_file):
275 '''Create the pg_ident.conf file.'''
276 ident_data = {}
277- pg_ident_template = Template(
278- open("templates/pg_ident.conf.tmpl").read())
279+ charm_dir = hookenv.charm_dir()
280+ template_file = "{}/templates/pg_ident.conf.tmpl".format(charm_dir)
281+ pg_ident_template = Template(open(template_file).read())
282 host.write_file(
283- postgresql_ident, pg_ident_template.render(ident_data),
284+ output_file, pg_ident_template.render(ident_data),
285 owner="postgres", group="postgres", perms=0600)
286
287
288 def generate_postgresql_hba(
289- postgresql_hba, user=None, schema_user=None, database=None):
290+ output_file, user=None, schema_user=None, database=None):
291 '''Create the pg_hba.conf file.'''
292
293 # Per Bug #1117542, when generating the postgresql_hba file we
294@@ -429,6 +452,7 @@
295 return output.rstrip(".") # trailing dot
296 return addr
297
298+ config_data = hookenv.config()
299 allowed_units = set()
300 relation_data = []
301 relids = hookenv.relation_ids('db') + hookenv.relation_ids('db-admin')
302@@ -525,9 +549,10 @@
303 'private-address': munge_address(admin_ip)}
304 relation_data.append(admin_host)
305
306- pg_hba_template = Template(open("templates/pg_hba.conf.tmpl").read())
307+ template_file = "{}/templates/pg_hba.conf.tmpl".format(hookenv.charm_dir())
308+ pg_hba_template = Template(open(template_file).read())
309 host.write_file(
310- postgresql_hba, pg_hba_template.render(access_list=relation_data),
311+ output_file, pg_hba_template.render(access_list=relation_data),
312 owner="postgres", group="postgres", perms=0600)
313 postgresql_reload()
314
315@@ -539,27 +564,36 @@
316 relid, {"allowed-units": " ".join(unit_sorted(allowed_units))})
317
318
319-def install_postgresql_crontab(postgresql_ident):
320+def install_postgresql_crontab(output_file):
321 '''Create the postgres user's crontab'''
322+ config_data = hookenv.config()
323 crontab_data = {
324 'backup_schedule': config_data["backup_schedule"],
325 'scripts_dir': postgresql_scripts_dir,
326 'backup_days': config_data["backup_retention_count"],
327 }
328+ charm_dir = hookenv.charm_dir()
329+ template_file = "{}/templates/postgres.cron.tmpl".format(charm_dir)
330 crontab_template = Template(
331- open("templates/postgres.cron.tmpl").read()).render(crontab_data)
332- host.write_file('/etc/cron.d/postgres', crontab_template, perms=0600)
333+ open(template_file).read()).render(crontab_data)
334+ host.write_file(output_file, crontab_template, perms=0600)
335
336
337 def create_recovery_conf(master_host, restart_on_change=False):
338+ version = hookenv.config('version')
339+ cluster_name = hookenv.config('cluster_name')
340+ postgresql_cluster_dir = os.path.join(
341+ postgresql_data_dir, version, cluster_name)
342+
343 recovery_conf_path = os.path.join(postgresql_cluster_dir, 'recovery.conf')
344 if os.path.exists(recovery_conf_path):
345 old_recovery_conf = open(recovery_conf_path, 'r').read()
346 else:
347 old_recovery_conf = None
348
349- recovery_conf = Template(
350- open("templates/recovery.conf.tmpl").read()).render({
351+ charm_dir = hookenv.charm_dir()
352+ template_file = "{}/templates/recovery.conf.tmpl".format(charm_dir)
353+ recovery_conf = Template(open(template_file).read()).render({
354 'host': master_host,
355 'password': local_state['replication_password']})
356 log(recovery_conf, DEBUG)
357@@ -578,9 +612,9 @@
358 # Returns a string containing the postgresql config or
359 # None
360 #------------------------------------------------------------------------------
361-def load_postgresql_config(postgresql_config):
362- if os.path.isfile(postgresql_config):
363- return(open(postgresql_config).read())
364+def load_postgresql_config(config_file):
365+ if os.path.isfile(config_file):
366+ return(open(config_file).read())
367 else:
368 return(None)
369
370@@ -677,7 +711,11 @@
371 # - manipulate /var/lib/postgresql/VERSION/CLUSTER symlink
372 #------------------------------------------------------------------------------
373 def config_changed_volume_apply():
374- data_directory_path = postgresql_cluster_dir
375+ version = hookenv.config('version')
376+ cluster_name = hookenv.config('cluster_name')
377+ data_directory_path = os.path.join(
378+ postgresql_data_dir, version, cluster_name)
379+
380 assert(data_directory_path)
381 volid = volume_get_volume_id()
382 if volid:
383@@ -698,7 +736,7 @@
384 mount_point = volume_mount_point_from_volid(volid)
385 new_pg_dir = os.path.join(mount_point, "postgresql")
386 new_pg_version_cluster_dir = os.path.join(
387- new_pg_dir, config_data["version"], config_data["cluster_name"])
388+ new_pg_dir, version, cluster_name)
389 if not mount_point:
390 log(
391 "invalid mount point from volid = {}, "
392@@ -724,7 +762,7 @@
393 # /var/lib/postgresql/9.1/main
394 curr_dir_stat = os.stat(data_directory_path)
395 for new_dir in [new_pg_dir,
396- os.path.join(new_pg_dir, config_data["version"]),
397+ os.path.join(new_pg_dir, version),
398 new_pg_version_cluster_dir]:
399 if not os.path.isdir(new_dir):
400 log("mkdir %s".format(new_dir))
401@@ -781,7 +819,8 @@
402
403 @hooks.hook()
404 def config_changed(force_restart=False):
405- update_repos_and_packages()
406+ config_data = hookenv.config()
407+ update_repos_and_packages(config_data["version"])
408
409 # Trigger volume initialization logic for permanent storage
410 volid = volume_get_volume_id()
411@@ -813,10 +852,17 @@
412 "Disabled and stopped postgresql service "
413 "(config_changed_volume_apply failure)", ERROR)
414 sys.exit(1)
415+
416+ postgresql_config_dir = _get_postgresql_config_dir(config_data)
417+ postgresql_config = os.path.join(postgresql_config_dir, "postgresql.conf")
418+ postgresql_hba = os.path.join(postgresql_config_dir, "pg_hba.conf")
419+ postgresql_ident = os.path.join(postgresql_config_dir, "pg_ident.conf")
420+
421 current_service_port = get_service_port(postgresql_config)
422 create_postgresql_config(postgresql_config)
423 generate_postgresql_hba(postgresql_hba)
424 create_postgresql_ident(postgresql_ident)
425+
426 updated_service_port = config_data["listen_port"]
427 update_service_port(current_service_port, updated_service_port)
428 update_nrpe_checks()
429@@ -832,8 +878,8 @@
430 if os.path.isfile(f) and os.access(f, os.X_OK):
431 subprocess.check_call(['sh', '-c', f])
432
433- update_repos_and_packages()
434-
435+ config_data = hookenv.config()
436+ update_repos_and_packages(config_data["version"])
437 if not 'state' in local_state:
438 # Fresh installation. Because this function is invoked by both
439 # the install hook and the upgrade-charm hook, we need to guard
440@@ -848,6 +894,10 @@
441 run("pg_createcluster --locale='{}' --encoding='{}' 9.1 main".format(
442 config_data['locale'], config_data['encoding']))
443
444+ postgresql_backups_dir = (
445+ config_data['backup_dir'].strip() or
446+ os.path.join(postgresql_data_dir, 'backups'))
447+
448 host.mkdir(postgresql_backups_dir, owner="postgres", perms=0o755)
449 host.mkdir(postgresql_scripts_dir, owner="postgres", perms=0o755)
450 host.mkdir(postgresql_logs_dir, owner="postgres", perms=0o755)
451@@ -857,10 +907,11 @@
452 'scripts_dir': postgresql_scripts_dir,
453 'logs_dir': postgresql_logs_dir,
454 }
455- dump_script = Template(
456- open("templates/dump-pg-db.tmpl").read()).render(paths)
457- backup_job = Template(
458- open("templates/pg_backup_job.tmpl").read()).render(paths)
459+ charm_dir = hookenv.charm_dir()
460+ template_file = "{}/templates/dump-pg-db.tmpl".format(charm_dir)
461+ dump_script = Template(open(template_file).read()).render(paths)
462+ template_file = "{}/templates/pg_backup_job.tmpl".format(charm_dir)
463+ backup_job = Template(open(template_file).read()).render(paths)
464 host.write_file(
465 '{}/dump-pg-db'.format(postgresql_scripts_dir),
466 dump_script, perms=0755)
467@@ -1176,6 +1227,7 @@
468 log("Client relations {}".format(local_state['client_relations']))
469 local_state.publish()
470
471+ postgresql_hba = os.path.join(_get_postgresql_config_dir(), "pg_hba.conf")
472 generate_postgresql_hba(postgresql_hba, user=user,
473 schema_user=schema_user,
474 database=database)
475@@ -1213,6 +1265,7 @@
476 log("Client relations {}".format(local_state['client_relations']))
477 local_state.publish()
478
479+ postgresql_hba = os.path.join(_get_postgresql_config_dir(), "pg_hba.conf")
480 generate_postgresql_hba(postgresql_hba)
481
482 snapshot_relations()
483@@ -1245,6 +1298,7 @@
484 run_sql_as_postgres(sql, AsIs(quote_identifier(database)),
485 AsIs(quote_identifier(user + "_schema")))
486
487+ postgresql_hba = os.path.join(_get_postgresql_config_dir(), "pg_hba.conf")
488 generate_postgresql_hba(postgresql_hba)
489
490 # Cleanup our local state.
491@@ -1261,13 +1315,14 @@
492 sql = "ALTER USER %s NOSUPERUSER"
493 run_sql_as_postgres(sql, AsIs(quote_identifier(user)))
494
495+ postgresql_hba = os.path.join(_get_postgresql_config_dir(), "pg_hba.conf")
496 generate_postgresql_hba(postgresql_hba)
497
498 # Cleanup our local state.
499 snapshot_relations()
500
501
502-def update_repos_and_packages():
503+def update_repos_and_packages(version):
504 extra_repos = hookenv.config('extra_archives')
505 extra_repos_added = local_state.setdefault('extra_repos_added', set())
506 if extra_repos:
507@@ -1284,10 +1339,12 @@
508 # It might have been better for debversion and plpython to only get
509 # installed if they were listed in the extra-packages config item,
510 # but they predate this feature.
511- packages = ["postgresql-%s" % config_data["version"],
512- "postgresql-contrib-%s" % config_data["version"],
513- "postgresql-plpython-%s" % config_data["version"],
514- "postgresql-%s-debversion" % config_data["version"],
515+ packages = ["python-psutil", # to obtain system RAM from python
516+ "libc-bin", # for getconf
517+ "postgresql-%s" % version,
518+ "postgresql-contrib-%s" % version,
519+ "postgresql-plpython-%s" % version,
520+ "postgresql-%s-debversion" % version,
521 "python-jinja2", "syslinux", "python-psycopg2"]
522 packages.extend((hookenv.config('extra-packages') or '').split())
523 packages = fetch.filter_installed_packages(packages)
524@@ -1351,6 +1408,11 @@
525
526 def promote_database():
527 '''Take the database out of recovery mode.'''
528+ config_data = hookenv.config()
529+ version = config_data['version']
530+ cluster_name = config_data['cluster_name']
531+ postgresql_cluster_dir = os.path.join(
532+ postgresql_data_dir, version, cluster_name)
533 recovery_conf = os.path.join(postgresql_cluster_dir, 'recovery.conf')
534 if os.path.exists(recovery_conf):
535 # Rather than using 'pg_ctl promote', we do the promotion
536@@ -1554,6 +1616,8 @@
537 del local_state['paused_at_failover']
538
539 publish_hot_standby_credentials()
540+ postgresql_hba = os.path.join(
541+ _get_postgresql_config_dir(), "pg_hba.conf")
542 generate_postgresql_hba(postgresql_hba)
543
544 local_state.publish()
545@@ -1706,7 +1770,7 @@
546 lock.
547 '''
548 import psycopg2
549- key = long(config_data['advisory_lock_restart_key'])
550+ key = long(hookenv.config('advisory_lock_restart_key'))
551 if exclusive:
552 lock_function = 'pg_advisory_lock'
553 else:
554@@ -1749,6 +1813,12 @@
555 postgresql_stop()
556 log("Cloning master {}".format(master_unit))
557
558+ config_data = hookenv.config()
559+ version = config_data['version']
560+ cluster_name = config_data['cluster_name']
561+ postgresql_cluster_dir = os.path.join(
562+ postgresql_data_dir, version, cluster_name)
563+ postgresql_config_dir = _get_postgresql_config_dir(config_data)
564 cmd = [
565 'sudo', '-E', # -E needed to locate pgpass file.
566 '-u', 'postgres', 'pg_basebackup', '-D', postgresql_cluster_dir,
567@@ -1802,6 +1872,11 @@
568
569
570 def postgresql_is_in_backup_mode():
571+ version = hookenv.config('version')
572+ cluster_name = hookenv.config('cluster_name')
573+ postgresql_cluster_dir = os.path.join(
574+ postgresql_data_dir, version, cluster_name)
575+
576 return os.path.exists(
577 os.path.join(postgresql_cluster_dir, 'backup_label'))
578
579@@ -1919,30 +1994,28 @@
580 host.service_reload('nagios-nrpe-server')
581
582
583+def _get_postgresql_config_dir(config_data=None):
584+ """ Return the directory path of the postgresql configuration files. """
585+ if config_data == None:
586+ config_data = hookenv.config()
587+ version = config_data['version']
588+ cluster_name = config_data['cluster_name']
589+ return os.path.join("/etc/postgresql", version, cluster_name)
590+
591 ###############################################################################
592 # Global variables
593 ###############################################################################
594-config_data = hookenv.config()
595-version = config_data['version']
596-cluster_name = config_data['cluster_name']
597 postgresql_data_dir = "/var/lib/postgresql"
598-postgresql_cluster_dir = os.path.join(
599- postgresql_data_dir, version, cluster_name)
600-postgresql_bin_dir = os.path.join('/usr/lib/postgresql', version, 'bin')
601-postgresql_config_dir = os.path.join("/etc/postgresql", version, cluster_name)
602-postgresql_config = os.path.join(postgresql_config_dir, "postgresql.conf")
603-postgresql_ident = os.path.join(postgresql_config_dir, "pg_ident.conf")
604-postgresql_hba = os.path.join(postgresql_config_dir, "pg_hba.conf")
605+postgresql_scripts_dir = os.path.join(postgresql_data_dir, 'scripts')
606+postgresql_logs_dir = os.path.join(postgresql_data_dir, 'logs')
607+
608+postgresql_sysctl = "/etc/sysctl.d/50-postgresql.conf"
609 postgresql_crontab = "/etc/cron.d/postgresql"
610 postgresql_service_config_dir = "/var/run/postgresql"
611-postgresql_scripts_dir = os.path.join(postgresql_data_dir, 'scripts')
612-postgresql_backups_dir = (
613- config_data['backup_dir'].strip() or
614- os.path.join(postgresql_data_dir, 'backups'))
615-postgresql_logs_dir = os.path.join(postgresql_data_dir, 'logs')
616-hook_name = os.path.basename(sys.argv[0])
617 replication_relation_types = ['master', 'slave', 'replication']
618 local_state = State('local_state.pickle')
619+hook_name = os.path.basename(sys.argv[0])
620+juju_log_dir = "/var/log/juju"
621
622
623 if __name__ == '__main__':
624
625=== added file 'hooks/test_hooks.py'
626--- hooks/test_hooks.py 1970-01-01 00:00:00 +0000
627+++ hooks/test_hooks.py 2014-01-07 21:59:36 +0000
628@@ -0,0 +1,354 @@
629+import mocker
630+import hooks
631+
632+
633+class TestJujuHost(object):
634+ """
635+ Testing object to intercept charmhelper calls and inject data, or make sure
636+ certain data is set.
637+ """
638+ def write_file(self, file_path, contents, owner=None, group=None,
639+ perms=None):
640+ """
641+ Only write the file as requested. owner, group and perms untested.
642+ """
643+ with open(file_path, 'w') as target:
644+ target.write(contents)
645+
646+ def mkdir(self, dir_path, owner, group, perms):
647+ """Not yet tested"""
648+ pass
649+
650+ def service_start(self, service_name):
651+ """Not yet tested"""
652+ pass
653+
654+ def service_reload(self, service_name):
655+ """Not yet tested"""
656+ pass
657+
658+ def service_pwgen(self, service_name):
659+ """Not yet tested"""
660+ return ""
661+
662+ def service_stop(self, service_name):
663+ """Not yet tested"""
664+ pass
665+
666+
667+class TestJuju(object):
668+ """
669+ Testing object to intercept juju calls and inject data, or make sure
670+ certain data is set.
671+ """
672+
673+ _relation_data = {}
674+ _relation_ids = {}
675+ _relation_list = ("postgres/0",)
676+
677+ def __init__(self):
678+ self._config = {
679+ "admin_addresses": "",
680+ "locale": "C",
681+ "encoding": "UTF-8",
682+ "extra_packages": "",
683+ "dumpfile_location": "None",
684+ "config_change_command": "reload",
685+ "version": "9.1",
686+ "cluster_name": "main",
687+ "listen_ip": "*",
688+ "listen_port": "5432",
689+ "max_connections": "100",
690+ "ssl": "True",
691+ "log_min_duration_statement": -1,
692+ "log_checkpoints": False,
693+ "log_connections": False,
694+ "log_disconnections": False,
695+ "log_line_prefix": "%t ",
696+ "log_lock_waits": False,
697+ "log_timezone": "UTC",
698+ "autovacuum": True,
699+ "log_autovacuum_min_duration": -1,
700+ "autovacuum_analyze_threshold": 50,
701+ "autovacuum_vacuum_scale_factor": 0.2,
702+ "autovacuum_analyze_scale_factor": 0.1,
703+ "autovacuum_vacuum_cost_delay": "20ms",
704+ "search_path": "\"$user\",public",
705+ "standard_conforming_strings": True,
706+ "hot_standby": False,
707+ "hot_standby_feedback": False,
708+ "wal_level": "minimal",
709+ "max_wal_senders": 0,
710+ "wal_keep_segments": 0,
711+ "replicated_wal_keep_segments": 5000,
712+ "archive_mode": False,
713+ "archive_command": "",
714+ "work_mem": "1MB",
715+ "maintenance_work_mem": "1MB",
716+ "performance_tuning": "auto",
717+ "kernel_shmall": 0,
718+ "kernel_shmmax": 0,
719+ "shared_buffers": "",
720+ "effective_cache_size": "",
721+ "temp_buffers": "1MB",
722+ "wal_buffers": "-1",
723+ "checkpoint_segments": 3,
724+ "random_page_cost": 4.0,
725+ "volume_ephemeral_storage": True,
726+ "volume_map": "",
727+ "volume_dev_regexp": "/dev/db[b-z]",
728+ "backup_dir": "/var/lib/postgresql/backups",
729+ "backup_schedule": "13 4 * * *",
730+ "backup_retention_count": 7,
731+ "nagios_context": "juju",
732+ "extra_archives": "",
733+ "advisory_lock_restart_key": 765}
734+
735+ def relation_set(self, *args, **kwargs):
736+ """
737+ Capture result of relation_set into _relation_data, which
738+ can then be checked later.
739+ """
740+ if "relation_id" in kwargs:
741+ del kwargs["relation_id"]
742+ self._relation_data = dict(self._relation_data, **kwargs)
743+ for arg in args:
744+ (key, value) = arg.split("=")
745+ self._relation_data[key] = value
746+
747+ def relation_ids(self, relation_name="db-admin"):
748+ """
749+ Return expected relation_ids for tests. Feel free to expand
750+ as more tests are added.
751+ """
752+ return [self._relation_ids[name] for name in self._relation_ids.keys()
753+ if name.find(relation_name) == 0]
754+
755+ def related_units(self, relid="db-admin:5"):
756+ """
757+ Return expected relation_ids for tests. Feel free to expand
758+ as more tests are added.
759+ """
760+ return [name for name, value in self._relation_ids.iteritems()
761+ if value == relid]
762+
763+ def relation_list(self):
764+ """
765+ Hardcode expected relation_list for tests. Feel free to expand
766+ as more tests are added.
767+ """
768+ return list(self._relation_list)
769+
770+ def unit_get(self, *args):
771+ """
772+ for now the only thing this is called for is "public-address",
773+ so it's a simplistic return.
774+ """
775+ return "localhost"
776+
777+ def local_unit(self):
778+ return hooks.os.environ["JUJU_UNIT_NAME"]
779+
780+ def charm_dir(self):
781+ return hooks.os.environ["CHARM_DIR"]
782+
783+ def juju_log(self, *args, **kwargs):
784+ pass
785+
786+ def log(self, *args, **kwargs):
787+ pass
788+
789+ def config_get(self, scope=None):
790+ if scope is None:
791+ return self.config
792+ else:
793+ return self.config[scope]
794+
795+ def relation_get(self, scope=None, unit_name=None, relation_id=None):
796+ pass
797+
798+
799+class TestHooks(mocker.MockerTestCase):
800+
801+ def setUp(self):
802+ hooks.hookenv = TestJuju()
803+ hooks.host = TestJujuHost()
804+ hooks.juju_log_dir = self.makeDir()
805+ hooks.hookenv.config = lambda: hooks.hookenv._config
806+ #hooks.hookenv.localunit = lambda: "localhost"
807+ hooks.os.environ["JUJU_UNIT_NAME"] = "landscape/1"
808+ hooks.postgresql_sysctl = self.makeFile()
809+ hooks._get_system_ram = lambda: 1024 # MB
810+ hooks._get_page_size = lambda: 1024 * 1024 # bytes
811+ self.maxDiff = None
812+
813+ def assertFileContains(self, filename, lines):
814+ """Make sure strings exist in a file."""
815+ with open(filename, "r") as fp:
816+ contents = fp.read()
817+ for line in lines:
818+ self.assertIn(line, contents)
819+
820+ def assertNotFileContains(self, filename, lines):
821+ """Make sure strings do not exist in a file."""
822+ with open(filename, "r") as fp:
823+ contents = fp.read()
824+ for line in lines:
825+ self.assertNotIn(line, contents)
826+
827+ def assertFilesEqual(self, file1, file2):
828+ """Given two filenames, compare them."""
829+ with open(file1, "r") as fp1:
830+ contents1 = fp1.read()
831+ with open(file2, "r") as fp2:
832+ contents2 = fp2.read()
833+ self.assertEqual(contents1, contents2)
834+
835+
836+class TestHooksService(TestHooks):
837+
838+ def test_create_postgresql_config_wal_no_replication(self):
839+ """
840+ When postgresql is in C{standalone} mode, and participates in no
841+ C{replication} relations, default wal settings will be present.
842+ """
843+ config_outfile = self.makeFile()
844+ run = self.mocker.replace(hooks.run)
845+ run("sysctl -p %s" % hooks.postgresql_sysctl)
846+ self.mocker.result(True)
847+ self.mocker.replay()
848+ hooks.create_postgresql_config(config_outfile)
849+ self.assertFileContains(
850+ config_outfile,
851+ ["wal_buffers = -1", "wal_level = minimal", "max_wal_senders = 0",
852+ "wal_keep_segments = 0"])
853+
854+ def test_create_postgresql_config_wal_with_replication(self):
855+ """
856+ When postgresql is in C{replicated} mode, and participates in a
857+ C{replication} relation, C{hot_standby} will be set to C{on},
858+ C{wal_level} will be enabled as C{hot_standby} and the
859+ C{max_wall_senders} will match the count of replication relations.
860+ The value of C{wal_keep_segments} will be the maximum of the configured
861+ C{wal_keep_segments} and C{replicated_wal_keep_segments}.
862+ """
863+ self.addCleanup(
864+ setattr, hooks.hookenv, "_relation_ids", {})
865+ hooks.hookenv._relation_ids = {
866+ "replication/0": "db-admin:5", "replication/1": "db-admin:6"}
867+ config_outfile = self.makeFile()
868+ run = self.mocker.replace(hooks.run)
869+ run("sysctl -p %s" % hooks.postgresql_sysctl)
870+ self.mocker.result(True)
871+ self.mocker.replay()
872+ hooks.create_postgresql_config(config_outfile)
873+ self.assertFileContains(
874+ config_outfile,
875+ ["hot_standby = on", "wal_buffers = -1", "wal_level = hot_standby",
876+ "max_wal_senders = 2", "wal_keep_segments = 5000"])
877+
878+ def test_create_postgresql_config_wal_with_replication_max_override(self):
879+ """
880+ When postgresql is in C{replicated} mode, and participates in a
881+ C{replication} relation, C{hot_standby} will be set to C{on},
882+ C{wal_level} will be enabled as C{hot_standby}. The written value for
883+ C{max_wal_senders} will be the maximum of replication slave count and
884+ the configuration value for C{max_wal_senders}.
885+ The written value of C{wal_keep_segments} will be
886+ the maximum of the configuration C{wal_keep_segments} and
887+ C{replicated_wal_keep_segments}.
888+ """
889+ self.addCleanup(
890+ setattr, hooks.hookenv, "_relation_ids", ())
891+ hooks.hookenv._relation_ids = {
892+ "replication/0": "db-admin:5", "replication/1": "db-admin:6"}
893+ hooks.hookenv._config["max_wal_senders"] = "3"
894+ hooks.hookenv._config["wal_keep_segments"] = 1000
895+ hooks.hookenv._config["replicated_wal_keep_segments"] = 999
896+ config_outfile = self.makeFile()
897+ run = self.mocker.replace(hooks.run)
898+ run("sysctl -p %s" % hooks.postgresql_sysctl)
899+ self.mocker.result(True)
900+ self.mocker.replay()
901+ hooks.create_postgresql_config(config_outfile)
902+ self.assertFileContains(
903+ config_outfile,
904+ ["hot_standby = on", "wal_buffers = -1", "wal_level = hot_standby",
905+ "max_wal_senders = 3", "wal_keep_segments = 1000"])
906+
907+ def test_create_postgresql_config_performance_tune_auto_large_ram(self):
908+ """
909+ When configuration attribute C{performance_tune} is set to C{auto} and
910+ total RAM on a system is > 1023MB. It will automatically calculate
911+ values for the following attributes if these attributes were left as
912+ default values:
913+ - C{effective_cache_size} set to 75% of total RAM in MegaBytes
914+ - C{shared_buffers} set to 25% of total RAM in MegaBytes
915+ - C{kernel_shmmax} set to total RAM in bytes
916+ - C{kernel_shmall} equal to kernel_shmmax in pages
917+ """
918+ config_outfile = self.makeFile()
919+ run = self.mocker.replace(hooks.run)
920+ run("sysctl -p %s" % hooks.postgresql_sysctl)
921+ self.mocker.result(True)
922+ self.mocker.replay()
923+ hooks.create_postgresql_config(config_outfile)
924+ self.assertFileContains(
925+ config_outfile,
926+ ["shared_buffers = 256MB", "effective_cache_size = 768MB"])
927+ self.assertFileContains(
928+ hooks.postgresql_sysctl,
929+ ["kernel.shmall = 1025\nkernel.shmmax = 1073742848"])
930+
931+ def test_create_postgresql_config_performance_tune_auto_small_ram(self):
932+ """
933+ When configuration attribute C{performance_tune} is set to C{auto} and
934+ total RAM on a system is <= 1023MB. It will automatically calculate
935+ values for the following attributes if these attributes were left as
936+ default values:
937+ - C{effective_cache_size} set to 75% of total RAM in MegaBytes
938+ - C{shared_buffers} set to 15% of total RAM in MegaBytes
939+ - C{kernel_shmmax} set to total RAM in bytes
940+ - C{kernel_shmall} equal to kernel_shmmax in pages
941+ """
942+ hooks._get_system_ram = lambda: 1023 # MB
943+ config_outfile = self.makeFile()
944+ run = self.mocker.replace(hooks.run)
945+ run("sysctl -p %s" % hooks.postgresql_sysctl)
946+ self.mocker.result(True)
947+ self.mocker.replay()
948+ hooks.create_postgresql_config(config_outfile)
949+ self.assertFileContains(
950+ config_outfile,
951+ ["shared_buffers = 153MB", "effective_cache_size = 767MB"])
952+ self.assertFileContains(
953+ hooks.postgresql_sysctl,
954+ ["kernel.shmall = 1024\nkernel.shmmax = 1072694272"])
955+
956+ def test_create_postgresql_config_performance_tune_auto_overridden(self):
957+ """
958+ When configuration attribute C{performance_tune} is set to C{auto} any
959+ non-default values for the configuration parameters below will be used
960+ instead of the automatically calculated values.
961+ - C{effective_cache_size}
962+ - C{shared_buffers}
963+ - C{kernel_shmmax}
964+ - C{kernel_shmall}
965+ """
966+ hooks.hookenv._config["effective_cache_size"] = "999MB"
967+ hooks.hookenv._config["shared_buffers"] = "101MB"
968+ hooks.hookenv._config["kernel_shmmax"] = 50000
969+ hooks.hookenv._config["kernel_shmall"] = 500
970+ hooks._get_system_ram = lambda: 1023 # MB
971+ config_outfile = self.makeFile()
972+ run = self.mocker.replace(hooks.run)
973+ run("sysctl -p %s" % hooks.postgresql_sysctl)
974+ self.mocker.result(True)
975+ self.mocker.replay()
976+ hooks.create_postgresql_config(config_outfile)
977+ self.assertFileContains(
978+ config_outfile,
979+ ["shared_buffers = 101MB", "effective_cache_size = 999MB"])
980+ self.assertFileContains(
981+ hooks.postgresql_sysctl,
982+ ["kernel.shmall = 1024\nkernel.shmmax = 1072694272"])
983
984=== added file 'revision'
985--- revision 1970-01-01 00:00:00 +0000
986+++ revision 2014-01-07 21:59:36 +0000
987@@ -0,0 +1,1 @@
988+2
989
990=== modified file 'templates/postgresql.conf.tmpl'
991--- templates/postgresql.conf.tmpl 2013-01-24 11:28:39 +0000
992+++ templates/postgresql.conf.tmpl 2014-01-07 21:59:36 +0000
993@@ -40,7 +40,7 @@
994 {% if shared_buffers != "" -%}
995 shared_buffers = {{shared_buffers}}
996 {% endif -%}
997-{% if temp_buffers!= "" -%}
998+{% if temp_buffers != "" -%}
999 temp_buffers = {{temp_buffers}}
1000 {% endif -%}
1001 {% if work_mem != "" -%}

Subscribers

People subscribed via source and target branches