Merge lp:~adeuring/juju-ci-tools/backup-to-s3 into lp:juju-ci-tools

[Abel Deuring]

A new script to backup Jenkins data to S3.

The main idea is to archive the whole home directory to S3, except those files and directories where we know that they are not important or "archived" as Bazaar branches.

Workspace and build data of jobs are also excluded: Most build information is already stored on S3.

There is a special rule for "jobs/disabled-repository" -- As I understand it, this is not a real job.

The first rule, "exclude hidden files/dirs in the home directory", may omit some important files. On the other hand, archiving for example .ssh/ may be questionable since it contains sensitive data.

[Curtis Hovey (sinzui)]

Thank you Abel.

I don't want .ssh backed up. The keys in .ssh came from cloud-city. Maybe we want to move .ssh/config to cloud-city too.

review: Approve (code)

=== added file 'backup-to-s3.py'
--- backup-to-s3.py	1970-01-01 00:00:00 +0000
+++ backup-to-s3.py	2014-07-25 13:02:30 +0000
@@ -0,0 +1,72 @@
+#!/usr/bin/env python
+"""Backup Jenkins data to S3 and remove old backups."""
+
+from datetime import datetime
+import os
+import re
+import subprocess
+
+
+MAX_BACKUPS = 10
+BACKUP_URL = 's3://juju-qa-data/juju-ci/backups/'
+# Exclude hidden files in the home directory, workspace and build data
+# of jobs, caches and Bazaar repositories.
+BACKUP_PARAMS = [
+    r'--rexclude=^\.',
+    '--rexclude=^jobs/.*?/workspace/',
+    '--rexclude=^jobs/.*?/builds/',
+    '--rexclude=^jobs/disabled-repository',
+    '--rexclude=^local-tools-cache/',
+    '--rexclude=^ci-director/',
+    '--rexclude=^cloud-city/',
+    '--rexclude=^failure-emails/',
+    '--rexclude=^juju-ci-tools/',
+    '--rexclude=^juju-release-tools/',
+    '--rexclude=^repository',
+    ]
+
+
+def s3_cmd(params, drop_output=False):
+    s3cfg_path = os.path.join(
+        os.environ['HOME'], 'cloud-city/juju-qa.s3cfg')
+    if drop_output:
+        return subprocess.check_call(
+            ['s3cmd', '-c', s3cfg_path] + params, stdout=open('/dev/null', 'w'))
+    else:
+        return subprocess.check_output(
+            ['s3cmd', '-c', s3cfg_path] + params)
+
+
+def current_backups():
+    """Return a list of S3 URLs of existing backups."""
+    # We expect lines like
+    # "         DIR   s3://juju-qa-data/juju-ci/backups/2014-07-25/"
+    result = []
+    for line in s3_cmd(['ls', BACKUP_URL]).split('\n'):
+        mo = re.search(r'^\s+DIR\s+(%s\d\d\d\d-\d\d-\d\d/)$' % BACKUP_URL, line)
+        if mo is None:
+            continue
+        url = mo.group(1)
+        result.append(url)
+    return sorted(result)
+
+
+def run_backup(url):
+    s3_cmd(['sync', '.', url] + BACKUP_PARAMS, drop_output=True)
+
+
+def remove_backups(urls):
+    if urls:
+        s3_cmd(['del', '-r'] + urls, drop_output=True)
+
+
+if __name__ == '__main__':
+    all_backups = current_backups()
+    today = datetime.now().strftime('%Y-%m-%d')
+    todays_url = '%s%s/' % (BACKUP_URL, today)
+    if todays_url in all_backups:
+        print "backup for %s already exists." % today
+    else:
+        run_backup(todays_url)
+        all_backups.append(todays_url)
+    remove_backups(all_backups[:-MAX_BACKUPS])