Merge lp:~ballot/turku/turku-agent into lp:turku

[Benjamin Allot]

Unmerged revisions

58. By Benjamin Allot on 2022-06-22

Add debian directory for packaging.

Lintian errors are still there

57. By Ryan Finnie on 2020-06-23

Mega-noop cleanup

Reviewed-on: https://code.launchpad.net/~fo0bar/turku/turku-agent-cleanup/+merge/386145
Reviewed-by: Barry Price <email address hidden>
Reviewed-by: Stuart Bishop <email address hidden>

56. By Ryan Finnie on 2020-04-07

Switch timers from OnCalendar (+ dependency hacks) to monotonic OnUnitActiveSec/OnStartupSec

Reviewed-on: https://code.launchpad.net/~fo0bar/turku/turku-agent-timers/+merge/381281
Reviewed-by: Joel Sing <email address hidden>

55. By Ryan Finnie on 2020-03-30

Revert subprocess portion of revno 53

Reviewed-on: https://code.launchpad.net/~fo0bar/turku/turku-agent-subprocess-encoding/+merge/381217
Reviewed-by: Joel Sing <email address hidden>

54. By Ryan Finnie on 2020-03-24

Run timers after network-online.target / time-sync.target

Reviewed-on: https://code.launchpad.net/~fo0bar/turku/turku-agent-timers-network/+merge/381073
Reviewed-by: Haw Loeung <email address hidden>
Reviewed-by: Stuart Bishop <email address hidden>

53. By Ryan Finnie on 2020-03-24

Move encoding from writes to opens

Reviewed-on: https://code.launchpad.net/~fo0bar/turku/turku-agent-encoding/+merge/381075
Reviewed-by: Haw Loeung <email address hidden>
Reviewed-by: Stuart Bishop <email address hidden>

52. By Ryan Finnie on 2020-03-24

Allow gonogo_program to be stored in config, allow --gonogo-program to be shlex-split

Reviewed-on: https://code.launchpad.net/~fo0bar/turku/turku-agent-gonogo/+merge/368854
Reviewed-by: Tom Haddon <email address hidden>

51. By Colin Watson on 2019-05-23

On Upstart, check if turku-agent-rsyncd is already running before starting it.

Reviewed-on: https://code.launchpad.net/~cjwatson/turku/turku-agent-fix-rsyncd-interaction/+merge/367847
Reviewed-by: Haw Loeung <email address hidden>

50. By Colin Watson on 2019-05-23

Don't restart rsyncd when updating its configuration.

Reviewed-on: https://code.launchpad.net/~cjwatson/turku/turku-agent-fix-rsyncd-interaction/+merge/367418
Reviewed-by: Haw Loeung <email address hidden>

49. By Ryan Finnie on 2019-04-22

Fix Python 3 .values conversion with --restore

=== added file '.bzrignore'
--- .bzrignore	1970-01-01 00:00:00 +0000
+++ .bzrignore	2022-06-22 14:48:04 +0000
@@ -0,0 +1,61 @@
+MANIFEST
+.pybuild/
+.pytest_cache/
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*,cover
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/

=== renamed file '.bzrignore' => '.bzrignore.moved'
=== added file 'MANIFEST.in'
--- MANIFEST.in	1970-01-01 00:00:00 +0000
+++ MANIFEST.in	2022-06-22 14:48:04 +0000
@@ -0,0 +1,16 @@
+include Makefile
+include README
+include requirements.txt
+include tests/*.py
+include tox.ini
+include turku-agent.cron
+include turku-agent-ping
+include turku-agent-ping.service
+include turku-agent-ping.timer
+include turku-agent-rsyncd.conf
+include turku-agent-rsyncd.init-debian
+include turku-agent-rsyncd.service
+include turku-agent-rsyncd-wrapper
+include turku-update-config
+include turku-update-config.service
+include turku-update-config.timer

=== renamed file 'MANIFEST.in' => 'MANIFEST.in.moved'
=== added file 'Makefile'
--- Makefile	1970-01-01 00:00:00 +0000
+++ Makefile	2022-06-22 14:48:04 +0000
@@ -0,0 +1,28 @@
+PYTHON := python3
+
+all: build
+
+build:
+	$(PYTHON) setup.py build
+
+lint:
+	$(PYTHON) -mtox -e flake8
+
+test:
+	$(PYTHON) -mtox
+
+test-quick:
+	$(PYTHON) -mtox -e black,flake8,pytest-quick
+
+black-check:
+	$(PYTHON) -mtox -e black
+
+black:
+	$(PYTHON) -mblack $(CURDIR)
+
+install: build
+	$(PYTHON) setup.py install
+
+clean:
+	$(PYTHON) setup.py clean
+	$(RM) -r build MANIFEST

=== renamed file 'Makefile' => 'Makefile.moved'
=== added file 'README'
--- README	1970-01-01 00:00:00 +0000
+++ README	2022-06-22 14:48:04 +0000
@@ -0,0 +1,4 @@
+Turku backups - client agent
+Copyright 2015 Canonical Ltd.
+
+https://launchpad.net/turku

=== renamed file 'README' => 'README.moved'
=== added directory 'debian'
=== added file 'debian/changelog'
--- debian/changelog	1970-01-01 00:00:00 +0000
+++ debian/changelog	2022-06-22 14:48:04 +0000
@@ -0,0 +1,31 @@
+turku-agent (0.1.0~bzr43) precise-cat; urgency=medium
+
+  * turku-agent r43
+
+ -- Laurent Sesques <[e-mail]>  Mon, 22 May 2017 11:33:17 +0200
+
+turku-agent (0.1.0~bzr40-1~0.IS.12.04) precise-cat; urgency=low
+
+  * precise-cat rebuild (no changes)
+
+ -- Ryan Finnie <[e-mail]>  Sun, 29 Mar 2015 17:25:09 +0000
+
+turku-agent (0.1.0~bzr40-1) lucid; urgency=low
+
+  * turku-agent r40
+  * upstart file is not (yet) updated to turku-agent-rsyncd-wrapper, to
+    assist in a mass migration.
+
+ -- Ryan Finnie <[e-mail]>  Sun, 29 Mar 2015 08:19:25 +0000
+
+turku-agent (0.0.20150318-1) lucid; urgency=low
+
+  * turku-agent r26
+
+ -- Ryan Finnie <[e-mail]>  Wed, 18 Mar 2015 05:48:26 +0000
+
+turku-agent (0.0.20150225-1) trusty; urgency=medium
+
+  * Initial packaging.
+
+ -- Ryan Finnie <[e-mail]>  Fri, 06 Mar 2015 05:52:34 +0000

=== added file 'debian/compat'
--- debian/compat	1970-01-01 00:00:00 +0000
+++ debian/compat	2022-06-22 14:48:04 +0000
@@ -0,0 +1,1 @@
+7

=== added file 'debian/control'
--- debian/control	1970-01-01 00:00:00 +0000
+++ debian/control	2022-06-22 14:48:04 +0000
@@ -0,0 +1,14 @@
+Source: turku-agent
+Priority: extra
+Maintainer: Ryan Finnie <[e-mail]>
+Build-Depends: debhelper (>= 7.0.50~), python3-all, dh-python
+Standards-Version: 3.9.2
+Section: admin
+
+Package: turku-agent
+Section: admin
+Architecture: all
+Depends: ${shlibs:Depends}, ${misc:Depends}, ${python3:Depends}, rsync
+Recommends: upstart
+Description: Turku backups (agent)
+ This package contains the Turku backups agent.

=== added file 'debian/copyright'
--- debian/copyright	1970-01-01 00:00:00 +0000
+++ debian/copyright	2022-06-22 14:48:04 +0000
@@ -0,0 +1,13 @@
+Copyright 2015 Canonical Ltd.
+
+This program is free software: you can redistribute it and/or modify it
+under the terms of the GNU General Public License version 3, as published by
+the Free Software Foundation.
+
+This program is distributed in the hope that it will be useful, but WITHOUT
+ANY WARRANTY; without even the implied warranties of MERCHANTABILITY,
+SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+General Public License for more details.
+
+You should have received a copy of the GNU General Public License along with
+this program.  If not, see /usr/share/common-licenses/GPL-3

=== added file 'debian/dirs'
--- debian/dirs	1970-01-01 00:00:00 +0000
+++ debian/dirs	2022-06-22 14:48:04 +0000
@@ -0,0 +1,2 @@
+etc/turku-agent/config.d
+etc/turku-agent/sources.d

=== added file 'debian/install'
--- debian/install	1970-01-01 00:00:00 +0000
+++ debian/install	2022-06-22 14:48:04 +0000
@@ -0,0 +1,2 @@
+debian/turku-agent-rsyncd.conf etc/init
+debian/turku-agent-rsyncd.service lib/systemd/system

=== added file 'debian/rules'
--- debian/rules	1970-01-01 00:00:00 +0000
+++ debian/rules	2022-06-22 14:48:04 +0000
@@ -0,0 +1,14 @@
+#!/usr/bin/make -f
+# -*- makefile -*-
+# Sample debian/rules that uses debhelper.
+# This file was originally written by Joey Hess and Craig Small.
+# As a special exception, when this file is copied by dh-make into a
+# dh-make output file, you may use that output file without restriction.
+# This special exception was added by Craig Small in version 0.37 of dh-make.
+
+# Uncomment this to turn on verbose mode.
+export DH_VERBOSE=1
+export PYBUILD_NAME=turku-agent
+
+%:
+	dh $@ --with python3 --buildsystem=pybuild

=== added directory 'debian/source'
=== added file 'debian/source/format'
--- debian/source/format	1970-01-01 00:00:00 +0000
+++ debian/source/format	2022-06-22 14:48:04 +0000
@@ -0,0 +1,1 @@
+1.0

=== added file 'debian/turku-agent-rsyncd.conf'
--- debian/turku-agent-rsyncd.conf	1970-01-01 00:00:00 +0000
+++ debian/turku-agent-rsyncd.conf	2022-06-22 14:48:04 +0000
@@ -0,0 +1,8 @@
+description "turku rsync daemon"
+
+start on runlevel [2345]
+stop on runlevel [!2345]
+
+respawn
+
+exec /usr/bin/rsync --no-detach --daemon --config=/var/lib/turku-agent/rsyncd.conf

=== added file 'debian/turku-agent-rsyncd.service'
--- debian/turku-agent-rsyncd.service	1970-01-01 00:00:00 +0000
+++ debian/turku-agent-rsyncd.service	2022-06-22 14:48:04 +0000
@@ -0,0 +1,10 @@
+[Unit]
+Description=turku rsyncd daemon
+ConditionPathExists=/var/lib/turku-agent/rsyncd.conf
+
+[Service]
+ExecStart=/usr/bin/env turku-agent-rsyncd-wrapper
+Restart=always
+
+[Install]
+WantedBy=multi-user.target

=== added file 'debian/turku-agent.cron.d'
--- debian/turku-agent.cron.d	1970-01-01 00:00:00 +0000
+++ debian/turku-agent.cron.d	2022-06-22 14:48:04 +0000
@@ -0,0 +1,3 @@
+PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
+*/5 * * * * root turku-agent-ping --wait=300 >/dev/null 2>/dev/null
+0 0,12 * * * root turku-update-config --wait=7200 >/dev/null 2>/dev/null

=== added file 'requirements.txt'
=== renamed file 'requirements.txt' => 'requirements.txt.moved'
=== added file 'setup.py'
--- setup.py	1970-01-01 00:00:00 +0000
+++ setup.py	2022-06-22 14:48:04 +0000
@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+
+# Turku backups - client agent
+# Copyright 2015 Canonical Ltd.
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published by
+# the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranties of MERCHANTABILITY,
+# SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import sys
+from setuptools import setup
+
+assert sys.version_info > (3, 4)
+
+
+setup(
+    name="turku_agent",
+    description="Turku backups - client agent",
+    version="0.2.0",
+    author="Ryan Finnie",
+    author_email="[e-mail]",
+    url="https://launchpad.net/turku",
+    packages=["turku_agent"],
+    entry_points={
+        "console_scripts": [
+            "turku-agent-ping = turku_agent.ping:main",
+            "turku-agent-rsyncd-wrapper = turku_agent.rsyncd_wrapper:main",
+            "turku-update-config = turku_agent.update_config:main",
+        ]
+    },
+)

=== renamed file 'setup.py' => 'setup.py.moved'
=== added directory 'tests'
=== renamed directory 'tests' => 'tests.moved'
=== added file 'tests/__init__.py'
=== added file 'tests/test_stub.py'
--- tests/test_stub.py	1970-01-01 00:00:00 +0000
+++ tests/test_stub.py	2022-06-22 14:48:04 +0000
@@ -0,0 +1,8 @@
+import unittest
+import warnings
+
+
+class TestStub(unittest.TestCase):
+    def test_stub(self):
+        # pytest doesn't like a tests/ with no tests
+        warnings.warn("Remove this file once unit tests are added")

=== added file 'tox.ini'
--- tox.ini	1970-01-01 00:00:00 +0000
+++ tox.ini	2022-06-22 14:48:04 +0000
@@ -0,0 +1,38 @@
+[tox]
+envlist = black, flake8, pytest
+
+[testenv]
+basepython = python
+
+[testenv:black]
+commands = python -mblack --check .
+deps = black
+
+[testenv:flake8]
+commands = python -mflake8
+deps = flake8
+
+[testenv:pytest]
+commands = python -mpytest --cov=turku_agent --cov-report=term-missing
+deps = pytest
+       pytest-cov
+       -r{toxinidir}/requirements.txt
+
+[testenv:pytest-quick]
+commands = python -mpytest -m "not slow"
+deps = pytest
+       -r{toxinidir}/requirements.txt
+
+[flake8]
+exclude =
+    .git,
+    __pycache__,
+    .tox,
+# TODO: remove C901 once complexity is reduced
+ignore = C901,E203,E231,W503
+max-line-length = 120
+max-complexity = 10
+
+[pytest]
+markers =
+    slow

=== renamed file 'tox.ini' => 'tox.ini.moved'
=== added file 'turku-agent-ping'
--- turku-agent-ping	1970-01-01 00:00:00 +0000
+++ turku-agent-ping	2022-06-22 14:48:04 +0000
@@ -0,0 +1,20 @@
+#!/usr/bin/env python3
+
+# Turku backups - client agent
+# Copyright 2015 Canonical Ltd.
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published by
+# the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranties of MERCHANTABILITY,
+# SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import sys
+from turku_agent import ping
+sys.exit(ping.main())

=== added file 'turku-agent-ping.service'
--- turku-agent-ping.service	1970-01-01 00:00:00 +0000
+++ turku-agent-ping.service	2022-06-22 14:48:04 +0000
@@ -0,0 +1,6 @@
+[Unit]
+Description=turku-agent-ping
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/env turku-agent-ping

=== added file 'turku-agent-ping.timer'
--- turku-agent-ping.timer	1970-01-01 00:00:00 +0000
+++ turku-agent-ping.timer	2022-06-22 14:48:04 +0000
@@ -0,0 +1,10 @@
+[Unit]
+Description=turku-agent-ping
+
+[Timer]
+OnUnitActiveSec=5m
+RandomizedDelaySec=5m
+OnStartupSec=15m
+
+[Install]
+WantedBy=timers.target

=== added file 'turku-agent-rsyncd-wrapper'
--- turku-agent-rsyncd-wrapper	1970-01-01 00:00:00 +0000
+++ turku-agent-rsyncd-wrapper	2022-06-22 14:48:04 +0000
@@ -0,0 +1,20 @@
+#!/usr/bin/env python3
+
+# Turku backups - client agent
+# Copyright 2015 Canonical Ltd.
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published by
+# the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranties of MERCHANTABILITY,
+# SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import sys
+from turku_agent import rsyncd_wrapper
+sys.exit(rsyncd_wrapper.main())

=== added file 'turku-agent-rsyncd.conf'
--- turku-agent-rsyncd.conf	1970-01-01 00:00:00 +0000
+++ turku-agent-rsyncd.conf	2022-06-22 14:48:04 +0000
@@ -0,0 +1,8 @@
+description "turku rsync daemon"
+
+start on runlevel [2345]
+stop on runlevel [!2345]
+
+respawn
+
+exec /usr/bin/env turku-agent-rsyncd-wrapper

=== added file 'turku-agent-rsyncd.init-debian'
--- turku-agent-rsyncd.init-debian	1970-01-01 00:00:00 +0000
+++ turku-agent-rsyncd.init-debian	2022-06-22 14:48:04 +0000
@@ -0,0 +1,55 @@
+#! /bin/sh
+
+### BEGIN INIT INFO
+# Provides:          turku-agent-rsyncd
+# Required-Start:    $remote_fs $syslog
+# Required-Stop:     $remote_fs $syslog
+# Should-Start:      $named
+# Default-Start:     2 3 4 5
+# Default-Stop:      
+# Short-Description: turku rsync daemon
+# Description:       turku rsync daemon
+### END INIT INFO
+
+set -e
+
+PID_FILE=/var/run/turku-agent-rsyncd.pid
+
+. /lib/lsb/init-functions
+
+export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
+
+case "$1" in
+  start)
+    log_daemon_msg "Starting turku rsync daemon" "turku-agent-rsyncd"
+    start-stop-daemon --start --quiet --background --make-pidfile \
+      --pidfile $PID_FILE --startas /usr/bin/env -- turku-agent-rsyncd-wrapper
+    log_end_msg $?
+    ;;
+  stop)
+    log_daemon_msg "Stopping turku rsync daemon" "turku-agent-rsyncd"
+    start-stop-daemon --stop --quiet --oknodo --pidfile $PID_FILE --retry 5
+    log_end_msg $?
+    rm -f $PID_FILE
+    ;;
+
+  reload|force-reload)
+    log_daemon_msg "Reloading turku rsync daemon" "turku-agent-rsyncd"
+    log_end_msg 0
+    ;;
+
+  restart)
+    "$0" stop
+    "$0" start
+    ;;
+
+  status)
+    status_of_proc -p $PID_FILE rsync turku-agent-rsyncd
+    exit $? # notreached due to set -e
+    ;;
+  *)
+    echo "Usage: /etc/init.d/turku-agent-rsyncd {start|stop|reload|force-reload|restart|status}"
+    exit 1
+esac
+
+exit 0

=== added file 'turku-agent-rsyncd.service'
--- turku-agent-rsyncd.service	1970-01-01 00:00:00 +0000
+++ turku-agent-rsyncd.service	2022-06-22 14:48:04 +0000
@@ -0,0 +1,10 @@
+[Unit]
+Description=turku rsyncd daemon
+ConditionPathExists=/var/lib/turku-agent/rsyncd.conf
+
+[Service]
+ExecStart=/usr/bin/env turku-agent-rsyncd-wrapper
+Restart=always
+
+[Install]
+WantedBy=multi-user.target

=== added file 'turku-agent.cron'
--- turku-agent.cron	1970-01-01 00:00:00 +0000
+++ turku-agent.cron	2022-06-22 14:48:04 +0000
@@ -0,0 +1,3 @@
+PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
+*/5 * * * * root sh -c 'systemctl is-active basic.target 2>/dev/null >/dev/null || turku-agent-ping --wait=300 >/dev/null 2>/dev/null'
+0 0,12 * * * root sh -c 'systemctl is-active basic.target 2>/dev/null >/dev/null || turku-update-config --wait=7200 >/dev/null 2>/dev/null'

=== added file 'turku-update-config'
--- turku-update-config	1970-01-01 00:00:00 +0000
+++ turku-update-config	2022-06-22 14:48:04 +0000
@@ -0,0 +1,20 @@
+#!/usr/bin/env python3
+
+# Turku backups - client agent
+# Copyright 2015 Canonical Ltd.
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published by
+# the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranties of MERCHANTABILITY,
+# SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import sys
+from turku_agent import update_config
+sys.exit(update_config.main())

=== added file 'turku-update-config.service'
--- turku-update-config.service	1970-01-01 00:00:00 +0000
+++ turku-update-config.service	2022-06-22 14:48:04 +0000
@@ -0,0 +1,6 @@
+[Unit]
+Description=turku-update-config
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/env turku-update-config

=== added file 'turku-update-config.timer'
--- turku-update-config.timer	1970-01-01 00:00:00 +0000
+++ turku-update-config.timer	2022-06-22 14:48:04 +0000
@@ -0,0 +1,10 @@
+[Unit]
+Description=turku-update-config
+
+[Timer]
+OnUnitActiveSec=12h
+RandomizedDelaySec=12h
+OnStartupSec=1h
+
+[Install]
+WantedBy=timers.target

=== added directory 'turku_agent'
=== added file 'turku_agent/__init__.py'
=== added file 'turku_agent/ping.py'
--- turku_agent/ping.py	1970-01-01 00:00:00 +0000
+++ turku_agent/ping.py	2022-06-22 14:48:04 +0000
@@ -0,0 +1,242 @@
+# Turku backups - client agent
+# Copyright 2015 Canonical Ltd.
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published by
+# the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranties of MERCHANTABILITY,
+# SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+import json
+import os
+import random
+import shlex
+import subprocess
+import tempfile
+import time
+
+from .utils import load_config, acquire_lock, api_call
+
+
+def parse_args():
+    import argparse
+
+    parser = argparse.ArgumentParser(
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter
+    )
+    parser.add_argument("--config-dir", "-c", type=str, default="/etc/turku-agent")
+    parser.add_argument("--wait", "-w", type=float)
+    parser.add_argument("--restore", action="store_true")
+    parser.add_argument("--restore-storage", type=str, default=None)
+    parser.add_argument(
+        "--gonogo-program",
+        type=str,
+        default=None,
+        help="Go/no-go program run each time to determine whether to ping",
+    )
+    return parser.parse_args()
+
+
+def call_ssh(config, storage, ssh_req):
+    # Write the server host public key
+    t = tempfile.NamedTemporaryFile(mode="w+", encoding="UTF-8")
+    for key in storage["ssh_ping_host_keys"]:
+        t.write("%s %s\n" % (storage["ssh_ping_host"], key))
+    t.flush()
+
+    # Call ssh
+    ssh_command = config["ssh_command"]
+    ssh_command += [
+        "-T",
+        "-o",
+        "BatchMode=yes",
+        "-o",
+        "UserKnownHostsFile=%s" % t.name,
+        "-o",
+        "StrictHostKeyChecking=yes",
+        "-o",
+        "CheckHostIP=no",
+        "-i",
+        config["ssh_private_key_file"],
+        "-R",
+        "%d:%s:%d"
+        % (
+            ssh_req["port"],
+            config["rsyncd_local_address"],
+            config["rsyncd_local_port"],
+        ),
+        "-p",
+        str(storage["ssh_ping_port"]),
+        "-l",
+        storage["ssh_ping_user"],
+        storage["ssh_ping_host"],
+        "turku-ping-remote",
+    ]
+    p = subprocess.Popen(ssh_command, stdin=subprocess.PIPE)
+
+    # Write the ssh request
+    p.stdin.write((json.dumps(ssh_req) + "\n.\n").encode("UTF-8"))
+    p.stdin.flush()
+
+    # Wait for the server to close the SSH connection
+    try:
+        p.wait()
+    except KeyboardInterrupt:
+        pass
+
+    # Cleanup
+    t.close()
+
+
+def main():
+    args = parse_args()
+
+    # Sleep a random amount of time if requested
+    if args.wait:
+        time.sleep(random.uniform(0, args.wait))
+
+    config = load_config(args.config_dir)
+
+    # Basic checks
+    for i in ("ssh_private_key_file", "machine_uuid", "machine_secret", "api_url"):
+        if i not in config:
+            return
+    if not os.path.isfile(config["ssh_private_key_file"]):
+        return
+
+    # If a go/no-go program is defined, run it and only go if it exits 0.
+    # Example: prevent backups during high-load for sensitive systems:
+    #   ['check_load', '-c', '1,5,15']
+    gonogo_program = (
+        args.gonogo_program if args.gonogo_program else config["gonogo_program"]
+    )
+    if isinstance(gonogo_program, (list, tuple)):
+        # List, program name first, optional arguments after
+        gonogo_program_and_args = list(gonogo_program)
+    elif isinstance(gonogo_program, str):
+        # String, shlex split it
+        gonogo_program_and_args = shlex.split(gonogo_program)
+    else:
+        # None
+        gonogo_program_and_args = []
+    if gonogo_program_and_args:
+        try:
+            subprocess.check_call(gonogo_program_and_args)
+        except (subprocess.CalledProcessError, OSError):
+            return
+
+    lock = acquire_lock(os.path.join(config["lock_dir"], "turku-agent-ping.lock"))
+
+    restore_mode = args.restore
+
+    # Check with the API server
+    api_out = {}
+
+    machine_merge_map = (("machine_uuid", "uuid"), ("machine_secret", "secret"))
+    api_out["machine"] = {}
+    for a, b in machine_merge_map:
+        if a in config:
+            api_out["machine"][b] = config[a]
+
+    if restore_mode:
+        print("Entering restore mode.")
+        print()
+        api_reply = api_call(config["api_url"], "agent_ping_restore", api_out)
+
+        sources_by_storage = {}
+        for source_name in api_reply["machine"]["sources"]:
+            source = api_reply["machine"]["sources"][source_name]
+            if source_name not in config["sources"]:
+                continue
+            if "storage" not in source:
+                continue
+            if source["storage"]["name"] not in sources_by_storage:
+                sources_by_storage[source["storage"]["name"]] = {}
+            sources_by_storage[source["storage"]["name"]][source_name] = source
+
+        if len(sources_by_storage) == 0:
+            print("Cannot find any appropraite sources.")
+            return
+        print("This machine's sources are on the following storage units:")
+        for storage_name in sources_by_storage:
+            print("    %s" % storage_name)
+            for source_name in sources_by_storage[storage_name]:
+                print("        %s" % source_name)
+        print()
+        if len(sources_by_storage) == 1:
+            storage = list(list(sources_by_storage.values())[0].values())[0]["storage"]
+        elif args.restore_storage:
+            if args.restore_storage in sources_by_storage:
+                storage = sources_by_storage[args.restore_storage]["storage"]
+            else:
+                print('Cannot find appropriate storage "%s"' % args.restore_storage)
+                return
+        else:
+            print(
+                "Multiple storages found.  Please use --restore-storage to specify one."
+            )
+            return
+
+        ssh_req = {
+            "verbose": True,
+            "action": "restore",
+            "port": random.randint(49152, 65535),
+        }
+        print("Storage unit: %s" % storage["name"])
+        if "restore_path" in config:
+            print("Local destination path: %s" % config["restore_path"])
+            print("Sample restore usage from storage unit:")
+            print(
+                "    RSYNC_PASSWORD=%s rsync -avzP --numeric-ids ${P?}/ rsync://%s@127.0.0.1:%s/%s/"
+                % (
+                    config["restore_password"],
+                    config["restore_username"],
+                    ssh_req["port"],
+                    config["restore_module"],
+                )
+            )
+            print()
+        call_ssh(config, storage, ssh_req)
+    else:
+        api_reply = api_call(config["api_url"], "agent_ping_checkin", api_out)
+
+        if "scheduled_sources" not in api_reply:
+            return
+        sources_by_storage = {}
+        for source_name in api_reply["machine"]["scheduled_sources"]:
+            source = api_reply["machine"]["scheduled_sources"][source_name]
+            if source_name not in config["sources"]:
+                continue
+            if "storage" not in source:
+                continue
+            if source["storage"]["name"] not in sources_by_storage:
+                sources_by_storage[source["storage"]["name"]] = {}
+            sources_by_storage[source["storage"]["name"]][source_name] = source
+
+        for storage_name in sources_by_storage:
+            ssh_req = {
+                "verbose": True,
+                "action": "checkin",
+                "port": random.randint(49152, 65535),
+                "sources": {},
+            }
+            for source in sources_by_storage[storage_name]:
+                ssh_req["sources"][source] = {
+                    "username": config["sources"][source]["username"],
+                    "password": config["sources"][source]["password"],
+                }
+            call_ssh(
+                config,
+                list(sources_by_storage[storage_name].values())[0]["storage"],
+                ssh_req,
+            )
+
+    # Cleanup
+    lock.close()

=== added file 'turku_agent/rsyncd_wrapper.py'
--- turku_agent/rsyncd_wrapper.py	1970-01-01 00:00:00 +0000
+++ turku_agent/rsyncd_wrapper.py	2022-06-22 14:48:04 +0000
@@ -0,0 +1,46 @@
+#!/usr/bin/env python3
+
+# Turku backups - client agent
+# Copyright 2015 Canonical Ltd.
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published by
+# the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranties of MERCHANTABILITY,
+# SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+
+from .utils import load_config
+
+
+def parse_args():
+    import argparse
+
+    parser = argparse.ArgumentParser(
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter
+    )
+    parser.add_argument("--config-dir", "-c", type=str, default="/etc/turku-agent")
+    parser.add_argument("--detach", action="store_true")
+    return parser.parse_known_args()
+
+
+def main():
+    args, rest = parse_args()
+
+    config = load_config(args.config_dir)
+    rsyncd_command = config["rsyncd_command"]
+    if not args.detach:
+        rsyncd_command.append("--no-detach")
+    rsyncd_command.append("--daemon")
+    rsyncd_command.append(
+        "--config=%s" % os.path.join(config["var_dir"], "rsyncd.conf")
+    )
+    rsyncd_command += rest
+    os.execvp(rsyncd_command[0], rsyncd_command)

=== added file 'turku_agent/update_config.py'
--- turku_agent/update_config.py	1970-01-01 00:00:00 +0000
+++ turku_agent/update_config.py	2022-06-22 14:48:04 +0000
@@ -0,0 +1,181 @@
+# Turku backups - client agent
+# Copyright 2015 Canonical Ltd.
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published by
+# the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranties of MERCHANTABILITY,
+# SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import logging
+import os
+import random
+import subprocess
+import time
+
+from .utils import json_dumps_p, load_config, fill_config, acquire_lock, api_call
+
+
+class IncompleteConfigError(Exception):
+    pass
+
+
+def parse_args():
+    import argparse
+
+    parser = argparse.ArgumentParser(
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter
+    )
+    parser.add_argument("--config-dir", "-c", type=str, default="/etc/turku-agent")
+    parser.add_argument("--wait", "-w", type=float)
+    parser.add_argument("--debug", action="store_true")
+    return parser.parse_args()
+
+
+def write_conf_files(config):
+    # Build rsyncd.conf
+    built_rsyncd_conf = (
+        "address = %s\n" % config["rsyncd_local_address"]
+        + "port = %d\n" % config["rsyncd_local_port"]
+        + "log file = /dev/stdout\n"
+        + "uid = root\n"
+        + "gid = root\n"
+        + "list = false\n\n"
+    )
+    rsyncd_secrets = []
+    rsyncd_secrets.append((config["restore_username"], config["restore_password"]))
+    built_rsyncd_conf += (
+        "[%s]\n"
+        + "    path = %s\n"
+        + "    auth users = %s\n"
+        + "    secrets file = %s\n"
+        + "    read only = false\n\n"
+    ) % (
+        config["restore_module"],
+        config["restore_path"],
+        config["restore_username"],
+        os.path.join(config["var_dir"], "rsyncd.secrets"),
+    )
+    for s in config["sources"]:
+        sd = config["sources"][s]
+        rsyncd_secrets.append((sd["username"], sd["password"]))
+        built_rsyncd_conf += (
+            "[%s]\n"
+            + "    path = %s\n"
+            + "    auth users = %s\n"
+            + "    secrets file = %s\n"
+            + "    read only = true\n\n"
+        ) % (
+            s,
+            sd["path"],
+            sd["username"],
+            os.path.join(config["var_dir"], "rsyncd.secrets"),
+        )
+    with open(os.path.join(config["var_dir"], "rsyncd.conf"), "w") as f:
+        f.write(built_rsyncd_conf)
+
+    # Build rsyncd.secrets
+    built_rsyncd_secrets = ""
+    for (username, password) in rsyncd_secrets:
+        built_rsyncd_secrets += username + ":" + password + "\n"
+    with open(os.path.join(config["var_dir"], "rsyncd.secrets"), "w") as f:
+        os.fchmod(f.fileno(), 0o600)
+        f.write(built_rsyncd_secrets)
+
+
+def init_is_upstart():
+    try:
+        return "upstart" in subprocess.check_output(
+            ["initctl", "version"], stderr=subprocess.DEVNULL, universal_newlines=True
+        )
+    except (FileNotFoundError, subprocess.CalledProcessError):
+        return False
+
+
+def start_services():
+    # Start rsyncd if it isn't already running.
+    # Note that we do *not* need to reload rsyncd when changing rsyncd.conf,
+    # as it rereads it on every client connection; but we may need to start
+    # it as it won't start if its configuration file doesn't exist.
+    if init_is_upstart():
+        # With Upstart, start will fail if the service is already running,
+        # so we need to check for that first.
+        try:
+            if "start/running" in subprocess.check_output(
+                ["status", "turku-agent-rsyncd"],
+                stderr=subprocess.STDOUT,
+                universal_newlines=True,
+            ):
+                return
+        except subprocess.CalledProcessError:
+            pass
+    subprocess.check_call(["service", "turku-agent-rsyncd", "start"])
+
+
+def send_config(config):
+    required_keys = ["api_url"]
+    if "api_auth" not in config:
+        required_keys += ["api_auth_name", "api_auth_secret"]
+    for k in required_keys:
+        if k not in config:
+            raise IncompleteConfigError('Required config "%s" not found.' % k)
+
+    api_out = {}
+    if ("api_auth_name" in config) and ("api_auth_secret" in config):
+        # name/secret style
+        api_out["auth"] = {
+            "name": config["api_auth_name"],
+            "secret": config["api_auth_secret"],
+        }
+    else:
+        # nameless secret style
+        api_out["auth"] = config["api_auth"]
+
+    # Merge the following options into the machine section
+    machine_merge_map = (
+        ("machine_uuid", "uuid"),
+        ("machine_secret", "secret"),
+        ("environment_name", "environment_name"),
+        ("service_name", "service_name"),
+        ("unit_name", "unit_name"),
+        ("ssh_public_key", "ssh_public_key"),
+        ("published", "published"),
+    )
+    api_out["machine"] = {}
+    for a, b in machine_merge_map:
+        if a in config:
+            api_out["machine"][b] = config[a]
+
+    api_out["machine"]["sources"] = config["sources"]
+
+    api_call(config["api_url"], "update_config", api_out)
+
+
+def main():
+    args = parse_args()
+    # Sleep a random amount of time if requested
+    if args.wait:
+        time.sleep(random.uniform(0, args.wait))
+
+    config = load_config(args.config_dir)
+    lock = acquire_lock(os.path.join(config["lock_dir"], "turku-update-config.lock"))
+    fill_config(config)
+    if args.debug:
+        print(json_dumps_p(config))
+    write_conf_files(config)
+    try:
+        send_config(config)
+    except Exception as e:
+        if args.debug:
+            raise
+        logging.exception(e)
+        return 1
+    start_services()
+
+    lock.close()

=== added file 'turku_agent/utils.py'
--- turku_agent/utils.py	1970-01-01 00:00:00 +0000
+++ turku_agent/utils.py	2022-06-22 14:48:04 +0000
@@ -0,0 +1,350 @@
+# Turku backups - client agent
+# Copyright 2015 Canonical Ltd.
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published by
+# the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranties of MERCHANTABILITY,
+# SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import copy
+import http.client
+import json
+import os
+import platform
+import random
+import string
+import subprocess
+import urllib.parse
+import uuid
+
+
+class RuntimeLock:
+    name = None
+    file = None
+
+    def __init__(self, name):
+        import fcntl
+
+        file = open(name, "w")
+        try:
+            fcntl.lockf(file, fcntl.LOCK_EX | fcntl.LOCK_NB)
+        except IOError as e:
+            import errno
+
+            if e.errno in (errno.EACCES, errno.EAGAIN):
+                raise
+        file.write("%10s\n" % os.getpid())
+        file.flush()
+        file.seek(0)
+        self.name = name
+        self.file = file
+
+    def close(self):
+        if self.file:
+            self.file.close()
+            self.file = None
+            os.unlink(self.name)
+
+    def __del__(self):
+        self.close()
+
+    def __enter__(self):
+        self.file.__enter__()
+        return self
+
+    def __exit__(self, exc, value, tb):
+        result = self.file.__exit__(exc, value, tb)
+        self.close()
+        return result
+
+
+def acquire_lock(name):
+    return RuntimeLock(name)
+
+
+def json_dump_p(obj, f):
+    """Calls json.dump with standard (pretty) formatting"""
+    return json.dump(obj, f, sort_keys=True, indent=4, separators=(",", ": "))
+
+
+def json_dumps_p(obj):
+    """Calls json.dumps with standard (pretty) formatting"""
+    return json.dumps(obj, sort_keys=True, indent=4, separators=(",", ": "))
+
+
+def json_load_file(file):
+    with open(file) as f:
+        try:
+            return json.load(f)
+        except ValueError as e:
+            e.args += (file,)
+            raise
+
+
+def dict_merge(s, m):
+    """Recursively merge one dict into another."""
+    if not isinstance(m, dict):
+        return m
+    out = copy.deepcopy(s)
+    for k, v in list(m.items()):
+        if k in out and isinstance(out[k], dict):
+            out[k] = dict_merge(out[k], v)
+        else:
+            out[k] = copy.deepcopy(v)
+    return out
+
+
+def load_config(config_dir):
+    config = {}
+    config["config_dir"] = config_dir
+
+    config_d = os.path.join(config["config_dir"], "config.d")
+    sources_d = os.path.join(config["config_dir"], "sources.d")
+
+    # Merge in config.d/*.json to the root level
+    config_files = []
+    if os.path.isdir(config_d):
+        config_files = [
+            os.path.join(config_d, fn)
+            for fn in os.listdir(config_d)
+            if fn.endswith(".json")
+            and os.path.isfile(os.path.join(config_d, fn))
+            and os.access(os.path.join(config_d, fn), os.R_OK)
+        ]
+    config_files.sort()
+    for file in config_files:
+        config = dict_merge(config, json_load_file(file))
+
+    if "var_dir" not in config:
+        config["var_dir"] = "/var/lib/turku-agent"
+
+    var_config_d = os.path.join(config["var_dir"], "config.d")
+
+    # Load /var config.d files
+    var_config = {}
+    var_config_files = []
+    if os.path.isdir(var_config_d):
+        var_config_files = [
+            os.path.join(var_config_d, fn)
+            for fn in os.listdir(var_config_d)
+            if fn.endswith(".json")
+            and os.path.isfile(os.path.join(var_config_d, fn))
+            and os.access(os.path.join(var_config_d, fn), os.R_OK)
+        ]
+    var_config_files.sort()
+    for file in var_config_files:
+        var_config = dict_merge(var_config, json_load_file(file))
+    # /etc gets priority over /var
+    var_config = dict_merge(var_config, config)
+    config = var_config
+
+    if "lock_dir" not in config:
+        config["lock_dir"] = "/var/lock"
+
+    if "rsyncd_command" not in config:
+        config["rsyncd_command"] = ["rsync"]
+
+    if "rsyncd_local_address" not in config:
+        config["rsyncd_local_address"] = "127.0.0.1"
+
+    if "rsyncd_local_port" not in config:
+        config["rsyncd_local_port"] = 27873
+
+    if "ssh_command" not in config:
+        config["ssh_command"] = ["ssh"]
+
+    # If a go/no-go program is defined, run it and only go if it exits 0.
+    # Type: String (program with no args) or list (program first, optional arguments after)
+    if "gonogo_program" not in config:
+        config["gonogo_program"] = None
+
+    var_sources_d = os.path.join(config["var_dir"], "sources.d")
+
+    # Validate the unit name
+    if "unit_name" not in config:
+        config["unit_name"] = platform.node()
+        # If this isn't in the on-disk config, don't write it; just
+        # generate it every time
+
+    # Pull the SSH public key
+    if os.path.isfile(os.path.join(config["var_dir"], "ssh_key.pub")):
+        with open(os.path.join(config["var_dir"], "ssh_key.pub")) as f:
+            config["ssh_public_key"] = f.read().rstrip()
+        config["ssh_public_key_file"] = os.path.join(config["var_dir"], "ssh_key.pub")
+        config["ssh_private_key_file"] = os.path.join(config["var_dir"], "ssh_key")
+
+    sources_config = {}
+    # Merge in sources.d/*.json to the sources dict
+    sources_files = []
+    if os.path.isdir(sources_d):
+        sources_files = [
+            os.path.join(sources_d, fn)
+            for fn in os.listdir(sources_d)
+            if fn.endswith(".json")
+            and os.path.isfile(os.path.join(sources_d, fn))
+            and os.access(os.path.join(sources_d, fn), os.R_OK)
+        ]
+    sources_files.sort()
+    var_sources_files = []
+    if os.path.isdir(var_sources_d):
+        var_sources_files = [
+            os.path.join(var_sources_d, fn)
+            for fn in os.listdir(var_sources_d)
+            if fn.endswith(".json")
+            and os.path.isfile(os.path.join(var_sources_d, fn))
+            and os.access(os.path.join(var_sources_d, fn), os.R_OK)
+        ]
+    var_sources_files.sort()
+    sources_files += var_sources_files
+    for file in sources_files:
+        sources_config = dict_merge(sources_config, json_load_file(file))
+
+    # Check for required sources options
+    for s in list(sources_config.keys()):
+        if "path" not in sources_config[s]:
+            del sources_config[s]
+
+    config["sources"] = sources_config
+
+    return config
+
+
+def fill_config(config):
+    config_d = os.path.join(config["config_dir"], "config.d")
+    sources_d = os.path.join(config["config_dir"], "sources.d")
+    var_config_d = os.path.join(config["var_dir"], "config.d")
+    var_sources_d = os.path.join(config["var_dir"], "sources.d")
+
+    # Create required directories
+    for d in (config_d, sources_d, var_config_d, var_sources_d):
+        if not os.path.isdir(d):
+            os.makedirs(d)
+
+    # Validate the machine UUID/secret
+    write_uuid_data = False
+    if "machine_uuid" not in config:
+        config["machine_uuid"] = str(uuid.uuid4())
+        write_uuid_data = True
+    if "machine_secret" not in config:
+        config["machine_secret"] = "".join(
+            random.choice(string.ascii_letters + string.digits) for i in range(30)
+        )
+        write_uuid_data = True
+    # Write out the machine UUID/secret if needed
+    if write_uuid_data:
+        with open(os.path.join(var_config_d, "10-machine_uuid.json"), "w") as f:
+            os.fchmod(f.fileno(), 0o600)
+            json_dump_p(
+                {
+                    "machine_uuid": config["machine_uuid"],
+                    "machine_secret": config["machine_secret"],
+                },
+                f,
+            )
+
+    # Restoration configuration
+    write_restore_data = False
+    if "restore_path" not in config:
+        config["restore_path"] = "/var/backups/turku-agent/restore"
+        write_restore_data = True
+    if "restore_module" not in config:
+        config["restore_module"] = "turku-restore"
+        write_restore_data = True
+    if "restore_username" not in config:
+        config["restore_username"] = str(uuid.uuid4())
+        write_restore_data = True
+    if "restore_password" not in config:
+        config["restore_password"] = "".join(
+            random.choice(string.ascii_letters + string.digits) for i in range(30)
+        )
+        write_restore_data = True
+    if write_restore_data:
+        with open(os.path.join(var_config_d, "10-restore.json"), "w") as f:
+            os.fchmod(f.fileno(), 0o600)
+            restore_out = {
+                "restore_path": config["restore_path"],
+                "restore_module": config["restore_module"],
+                "restore_username": config["restore_username"],
+                "restore_password": config["restore_password"],
+            }
+            json_dump_p(restore_out, f)
+    if not os.path.isdir(config["restore_path"]):
+        os.makedirs(config["restore_path"])
+
+    # Generate the SSH keypair if it doesn't exist
+    if "ssh_private_key_file" not in config:
+        subprocess.check_call(
+            [
+                "ssh-keygen",
+                "-t",
+                "rsa",
+                "-N",
+                "",
+                "-C",
+                "turku",
+                "-f",
+                os.path.join(config["var_dir"], "ssh_key"),
+            ]
+        )
+        with open(os.path.join(config["var_dir"], "ssh_key.pub")) as f:
+            config["ssh_public_key"] = f.read().rstrip()
+        config["ssh_public_key_file"] = os.path.join(config["var_dir"], "ssh_key.pub")
+        config["ssh_private_key_file"] = os.path.join(config["var_dir"], "ssh_key")
+
+    for s in config["sources"]:
+        # Check for missing usernames/passwords
+        if not (
+            "username" in config["sources"][s] or "password" in config["sources"][s]
+        ):
+            if "username" not in config["sources"][s]:
+                config["sources"][s]["username"] = str(uuid.uuid4())
+            if "password" not in config["sources"][s]:
+                config["sources"][s]["password"] = "".join(
+                    random.choice(string.ascii_letters + string.digits)
+                    for i in range(30)
+                )
+            with open(os.path.join(var_sources_d, "10-" + s + ".json"), "w") as f:
+                os.fchmod(f.fileno(), 0o600)
+                json_dump_p(
+                    {
+                        s: {
+                            "username": config["sources"][s]["username"],
+                            "password": config["sources"][s]["password"],
+                        }
+                    },
+                    f,
+                )
+
+
+def api_call(api_url, cmd, post_data, timeout=5):
+    url = urllib.parse.urlparse(api_url)
+    if url.scheme == "https":
+        h = http.client.HTTPSConnection(url.netloc, timeout=timeout)
+    else:
+        h = http.client.HTTPConnection(url.netloc, timeout=timeout)
+    out = json.dumps(post_data)
+    h.putrequest("POST", "%s/%s" % (url.path, cmd))
+    h.putheader("Content-Type", "application/json")
+    h.putheader("Content-Length", len(out))
+    h.putheader("Accept", "application/json")
+    h.endheaders()
+    h.send(out.encode("UTF-8"))
+
+    res = h.getresponse()
+    if not res.status == http.client.OK:
+        raise Exception(
+            "Received error %d (%s) from API server" % (res.status, res.reason)
+        )
+    if not res.getheader("content-type") == "application/json":
+        raise Exception("Received invalid reply from API server")
+    try:
+        return json.loads(res.read().decode("UTF-8"))
+    except ValueError:
+        raise Exception("Received invalid reply from API server")