Merge lp:~ec0/charms/trusty/glance-sync-slave/layer-metadata-fixes into lp:~canonical-bootstack/charms/trusty/glance-sync-slave/layer

[James Hebden]

Add changes to exclude glance and additional custom metadata from sync, as well as readonly attributes.

[Alvaro Uria (aluria)]

1) My layer/branch (lp:~aluria/canonical-bootstack/bootstack-tele2-glance-sync-slave-layer) had issues on "juju set" a few params (see cRT#100292). This is the reason it was not pushed via ~canonical-bootstack user (but ~aluria).

2) glance-sync-slave python script on such layer (in ~aluria) has some modifications thought for tele2 (ie tenant mapping by replacing REGION_MASTER by REGION_SLAVE). I think we should keep a fork of the main "glance-sync-slave/layer"

Unmerged revisions

52. By James Hebden on 2017-08-03

[jhebden,r=jillr] Added sajoupa's changes to ignore syncing of glance and extra properties in metadata

51. By Alvaro Uria on 2017-02-22

added extra params to config.yaml and reactive logic (config-changed not calling @when)

50. By Alvaro Uria on 2017-02-20

fixes on project mapping and errors capture. First real tests ok

49. By Alvaro Uria on 2017-02-17

minor fixes: hardcoded glance master endpoint, temp_image format

48. By Alvaro Uria on 2017-02-09

glance-sync-slave redesign

=== modified file 'config.yaml'
--- config.yaml	2017-01-11 03:55:07 +0000
+++ config.yaml	2017-08-03 07:40:40 +0000
@@ -24,6 +24,22 @@
         default: /srv/glance_sync_slave/logs
         description: directory to store sync logfiles
         type: string
+    master_password:
+        default: ''
+        description: auth password against glance master service
+        type: string
+    master_auth_url:
+        default: ''
+        description: master keystone endpoint
+        type: string
+    master_glance_url:
+        default: ''
+        description: master glance service endpoint
+        type: string
+    master_region:
+        default: 'RegionOne'
+        description: name of the master region
+        type: string
     nagios_context:
         default: "juju"
         type: string

=== modified file 'files/glance_sync_slave.py'
--- files/glance_sync_slave.py	2016-05-17 19:43:27 +0000
+++ files/glance_sync_slave.py	2017-08-03 07:40:40 +0000
@@ -7,189 +7,28 @@
 import argparse
 import json
 import datetime
+import dateutil.parser
 import atexit
-import re
+#import re
 import shlex
 from glanceclient import Client
-from glanceclient.exc import HTTPConflict
+from glanceclient.exc import (
+    HTTPConflict,
+    HTTPForbidden,
+)
 from keystoneclient.v2_0 import client
 import subprocess
 
+class OSProjectNotFound(Exception):
+    """This indicates no sync is possible
+    (not defaulting to admin project)
+    """
+    pass
+
 
 class ImageSyncSlave:
-
-    def __init__(self):
-        self.glance = self.glance_connect()
-
-        # remove readonly properties (from glanceclient/v2/image_schema.py)
-        readonly_properties = ['file', 'size', 'self', 'direct_url', 'schema',
-                               'status', 'updated_at', 'virtual_size',
-                               'checksum',
-                               'created_at']
-
-        reserved_properties = ['owner', 'locations', 'deleted',
-                               'deleted_at', 'direct_url', 'self', 'file',
-                               'schema']
-        self.glance_properties = readonly_properties + reserved_properties
-
-    # rsync image and metadata files to data_dir
-    def download_data(self, source, data_dir):
-
-        if not data_dir.endswith('/'):
-            data_dir = data_dir + '/'
-
-        if not source.endswith('/'):
-                source = source + '/'
-
-        command = "/usr/bin/rsync -az --delete -e " \
-                  "'ssh -o StrictHostKeyChecking=no' {} {}".format(source,
-                                                                   data_dir)
-        proc = subprocess.Popen(shlex.split(command),
-                                stdout=subprocess.PIPE,
-                                stderr=subprocess.PIPE)
-        (stdout, stderr) = proc.communicate()
-        if proc.returncode:
-            print("{} ERROR: problem while getting data from "
-                  "master ({}).".format(self.timestamp_now(), command))
-            print(stderr)
-            self.release_lock()
-            sys.exit(2)
-        else:
-            return True
-
-    def glance_upload(self, data_dir):
-
-        meta_files = [file for file in self.list_files(data_dir)
-                      if file.endswith('.json')]
-
-        glance_ids = [image.id for image in self.glance.images.list()]
-
-        for file in meta_files:
-            metadata = self.read_metadata(os.path.join(data_dir, file))
-
-            # compare image checksums with metadata checksums
-            if self.check_md5(metadata['checksum'],
-                              os.path.join(data_dir, metadata['id'])):
-                # import image into glance
-
-                # clean_meta contains the image id from the master metadata file
-                clean_meta = {key: value for key, value in metadata.items() if
-                              key not in self.glance_properties}
-
-                if clean_meta['id'] not in glance_ids:
-                    # create glance image
-                    print("{} image-id {}: creating image".format(
-                        self.timestamp_now(), metadata['id']))
-                    try:
-                        self.glance.images.create(**clean_meta)
-                    # catch conflicts with existing image ids (should already be
-                    # caught by glance_ids)
-                    except HTTPConflict as e:
-                        print("{} image-id {}: ERROR: Image already exists in "
-                              "glance.".format(self.timestamp_now(),
-                                               clean_meta['id']))
-                        print(e)
-                        try:
-                            self.update_metadata(clean_meta)
-                            continue
-                        except:
-                            print("{} image-id {}: ERROR: both image create "
-                                  "and image metadata update failed. This can "
-                                  "happen if the image was deleted through the "
-                                  "API but still exists in the glance "
-                                  "database.".format(
-                                      self.timestamp_now(), clean_meta['id']))
-                            continue
-                    try:
-                        meta_file = os.path.join(data_dir, clean_meta['id'])
-                        self.glance.images.upload(clean_meta['id'],
-                                                  open(meta_file))
-
-                    except Exception as e:
-                        print(e)
-                        print("{} image-id {}: Error: problem uploading data "
-                              "to glance.".format(self.timestamp_now(),
-                                                  clean_meta['id']))
-                        continue
-                else:
-                    # only update metadata
-                    self.update_metadata(clean_meta)
-            else:
-                print("{} image-id {}: checksum mismatch, not "
-                      "importing.".format(self.timestamp_now(), metadata['id']))
-
-    def glance_removal(self, data_dir):
-        meta_files = [file for file in self.list_files(data_dir)
-                      if file.endswith('.json')]
-
-        local_image_files = []
-        for meta_file in meta_files:
-            metadata = self.read_metadata(os.path.join(data_dir, meta_file))
-            local_image_files.append(metadata['id'])
-
-        deleted_images = [image.id for image in self.glance.images.list()
-                          if image.id not in local_image_files]
-        for image_id in deleted_images:
-
-            # remove image from glance
-            print("{} image-id {}: removing image".format(self.timestamp_now(),
-                                                          image_id))
-            try:
-                self.glance.images.delete(image_id)
-            except HTTPConflict as e:
-                print("{} image-id {}: ERROR: {}".format(self.timestamp_now(),
-                                                         image_id,
-                                                         e))
-
-    # generate checksum on disk
-    def check_md5(self, control_md5, image_file):
-        return self.get_md5(image_file) == control_md5
-
-    def get_md5(self, image_file):
-        hash_md5 = hashlib.md5()
-        with open(image_file, "rb") as f:
-                for chunk in iter(lambda: f.read(4096), b""):
-                    hash_md5.update(chunk)
-        return hash_md5.hexdigest()
-
-    def read_metadata(self, metadata_file):
-        with open(metadata_file) as meta_file:
-            data = json.load(meta_file)
-        return data
-
-    def glance_connect(self):
-        try:
-            username = os.environ['OS_USERNAME']
-            password = os.environ['OS_PASSWORD']
-            tenant_name = os.environ['OS_TENANT_NAME']
-            auth_url = os.environ['OS_AUTH_URL']
-        except Exception as e:
-            print("ERROR: unable to load environment variables, please source "
-                  "novarc")
-            print(e)
-            self.release_lock()
-            sys.exit(2)
-        keystone = client.Client(username=username, password=password,
-                                 tenant_name=tenant_name, auth_url=auth_url)
-
-        token = keystone.auth_token
-
-        service = keystone.services.find(name='glance')
-        endpoint = keystone.endpoints.find(service_id=service.id)
-        glance_url = endpoint.internalurl
-
-        return Client('2', endpoint=glance_url, token=token)
-
-    def timestamp_now(self):
-        return datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
-
-    def update_metadata(self, metadata):
-        print("{} image-id {}: updating metadata".format(self.timestamp_now(),
-                                                         metadata['id']))
-
-        # list of glance reserved properties
-        # (from glanceclient/v2/image_schema.py)
-        glance_properties = ["architecture",
+    extra_properties = set(['bs_owner'])
+    glance_properties = set(["architecture",
                              "checksum",
                              "container_format",
                              "created_at",
@@ -217,92 +56,557 @@
                              "tags",
                              "updated_at",
                              "virtual_size",
-                             "visibility"]
-
-        image = self.glance.images.get(metadata['id'])
-        removed_props = [key for key in image.keys() if
-                         key not in metadata.keys()+glance_properties]
-
-        try:
-            self.glance.images.update(metadata['id'],
-                                      remove_props=removed_props,
-                                      **metadata)
-        except Exception as e:
-            print("{} image-id {}: ERROR: Problem updating metadata".format(
-                self.timestamp_now(), metadata['id']))
-            print(e)
+                             "visibility"])
+    # egrep -B2 readOnly glanceclient/v2/image_schema.py | \
+    #       awk '/\{/ {print $1}' | tr -d \":
+    readonly_properties = set(['file',
+                               'size',
+                               'status',
+                               'self',
+                               'direct_url',
+                               'schema',
+                               'updated_at',
+                               'locations',
+                               'virtual_size',
+                               'checksum',
+                               'created_at'])
+
+    def __init__(self, data_dir, source):
+        self.projects_slave = {}
+        self.DATA_DIR = data_dir
+        self.SOURCE = source
+        self.valid_properties = self.glance_properties.difference(
+            self.readonly_properties.union(
+                self.extra_properties))
+        self.set_filelock()
+        self.glance_connect_slave()
+        #self.glance_connect_master()
+
+    def download_metadata_from_master(self):
+        """rsync metadata files from source to data_dir"""
+
+        if not self.SOURCE.endswith('/'):
+                self.SOURCE += '/'
+        if not self.DATA_DIR.endswith('/'):
+            self.DATA_DIR += '/'
+
+        command = '/usr/bin/rsync -az --delete -e ' \
+                  "'ssh -o StrictHostKeyChecking=no' " \
+                  '{0} {1}'.format(self.SOURCE, self.DATA_DIR)
+        proc = subprocess.Popen(shlex.split(command),
+                                stdout=subprocess.PIPE,
+                                stderr=subprocess.PIPE)
+        (stdout, stderr) = proc.communicate()
+        if proc.returncode:
+            self.log('ERROR: problem while getting data from master '
+                     '({0})'.format(command))
+            self.log('ERROR: {0}'.format(stderr))
+            self.release_lock()
+            sys.exit(2)
+        else:
+            return True
+
+    def parse_glance_slave_images(self):
+        """
+        image.id => [0-9a-z]*
+            len(image.id) < 2 : DATA_DIR/<image-id>.json
+            len(image.id) >= 2: DATA_DIR/XX/XXZZZ.json
+
+        @returns processed (aka. parsed) images
+        """
+        self.log('getting image list from slave')
+        processed_images_ids = set()
+        to_delete_images_ids = set()
+        for image in self.glance_slave.images.list():
+            if len(image.id) < 2:
+                basename = '{0}.json'.format(image.id)
+            else:
+                basename = '{0}/{1}.json'.format(str(image.id)[:2],
+                                                 image.id)
+            filename = os.path.join(self.DATA_DIR, basename)
+            if not os.path.isfile(filename):
+                to_delete_images_ids.add(image.id)
+                continue
+
+            metadata_local = self.read_metadata(filename)
+            if not metadata_local:
+                self.log('ERROR: read_metadata did not retrieve anything '
+                         '({0})'.format(filename))
+                continue
+
+            if metadata_local['checksum'] == image.checksum:
+                if not self.is_latest_metadata(metadata_local['id'],
+                                               metadata_local['updated_at'],
+                                               image.updated_at):
+                    # checksum ok, metadata outdated
+                    self.update_metadata(metadata_local, image)
+                processed_images_ids.add(image.id)
+            # checksum mismatch, re-upload
+            elif self.download_from_master(metadata_local):
+                self.upload_to_slave(metadata_local)
+                processed_images_ids.add(image.id)
+
+        self.log('DEBUG: images pending to be deleted: '
+                 '{0}'.format(to_delete_images_ids))
+        self.delete_images_from_slave(to_delete_images_ids)
+        self.log('DEBUG: processed images (to skip while parsing metadata '
+                 'files): {0}'.format(processed_images_ids))
+        return processed_images_ids
+
+    def is_latest_metadata(self, image_id, master_updated_at, slave_updated_at):
+        """Compares filename content (JSON metadata) and glance service info
+        @return
+            True: no need to update
+            False: need to update metadata on local copy
+        """
+        master_dup = dateutil.parser.parse(master_updated_at)
+        slave_dup = dateutil.parser.parse(slave_updated_at)
+
+        if master_dup <= slave_dup:
+            self.log('INFO: is_latest_metadata :: {0} up to '
+                     'date'.format(image_id))
+            return True
+        else:
+            self.log('INFO: is_latest_metadata :: {0} outdated. Needs '
+                     'update_metadata.'.format(image_id))
+            return False
+
+    def upload_to_slave(self, metadata_local):
+        """upload image to glance slave service
+        """
+        tmp_image_basename = '{0}.img'.format(metadata_local['id'])
+        tmp_image = os.path.join(self.DATA_DIR, tmp_image_basename)
+        try:
+            clean_metadata, removed_props = self.mangle_metadata(metadata_local)
+        except OSProjectNotFound as e:
+            self.log('EXCEPTION: upload_to_slave :: image-id {0} :: '
+                     'problem uploading data to glance '
+                     'slave (image could not be removed) :: '
+                     '{1}'.format(metadata_local['id'], e))
+            return False
+
+        for k in removed_props:
+            del clean_metadata[k]
+
+        self.log('INFO: creating image {0}'.format(clean_metadata['id']))
+        try:
+            # FIXME
+            self.glance_slave.images.create(**clean_metadata)
+            self.log('DEBUG: create image: {0}'.format(clean_metadata))
+        except HTTPConflict as e:
+            self.log('EXCEPTION: upload_to_slave :: {0}'.format(e))
+            try:
+                # update metadata
+                # FIXME
+                self.glance_slave.images.update(clean_metadata['id'],
+                                                remove_props=removed_props,
+                                                **clean_metadata)
+                self.log('DEBUG: update_to_slave :: update metadata '
+                         '{0}'.format(clean_metadata))
+            except Exception as e:
+                self.log('ERROR: update_to_slave (both image create/update '
+                         'failed :: {0} - this can happen if the image was '
+                         'deleted through the API but still exists in the glance '
+                         'database :: {1}'.format(clean_metadata['id'], e))
+                return False
+        try:
+            # upload
+            # FIXME
+            self.glance_slave.images.upload(clean_metadata['id'],
+                                            open(tmp_image))
+            os.remove(tmp_image)
+            self.log('DEBUG: update_to_slave :: upload {0}'.format(tmp_image))
+        except Exception as e:
+            self.log('ERROR: upload_to_slave :: image-id {0} :: '
+                     'problem uploading data to glance '
+                     'slave (image could not be removed) :: '
+                     '{1}'.format(clean_metadata['id'], e))
+            return False
+
+    def download_from_master(self, metadata_local):
+        """downloads images from glance master service
+
+        @return True: downloaded or already on local storage
+        @return False: error
+        """
+        tmp_image_basename = '{0}.img'.format(metadata_local['id'])
+        tmp_image = os.path.join(self.DATA_DIR, tmp_image_basename)
+        if os.path.isfile(tmp_image):
+            if self.check_md5(metadata_local['checksum'], tmp_image):
+                return True
+            else:
+                try:
+                    os.remove(tmp_image)
+                except Exception as e:
+                    self.log('ERROR: download_from_master :: {0}'.format(e))
+                    return False
+        self.glance_connect_master()
+        downloaded = False
+        retries = 3
+        for i in xrange(0, retries):
+            try:
+                bin_image = self.glance_master.images.data(
+                    image_id=metadata_local['id'])
+
+                hash_md5 = hashlib.md5()
+                with open(tmp_image, 'wb') as fd:
+                    for chunk in bin_image:
+                        fd.write(chunk)
+                        hash_md5.update(chunk)
+                bin_image_checksum = hash_md5.hexdigest()
+                if metadata_local['checksum'] == bin_image_checksum:
+                    downloaded = True
+                    self.log('INFO: download_from_master ({0} - {1}):: '
+                             'checksum OK'.format(metadata_local['id'],
+                                                  metadata_local['checksum']))
+                    break
+                elif os.path.exists(tmp_image):
+                    self.log('INFO: download_from_master ({0}/{1}; {2}):: '
+                             'invalid checksum '
+                             '{3}'.format(metadata_local['id'], i, retries,
+                                          bin_image_checksum))
+                    os.remove(tmp_image)
+            except Exception as e:
+                self.log('EXCEPTION: download_from_master ({0}/{1}; {2}):: '
+                         '{3}'.format(i, retries, metadata_local['id'], e))
+                if os.path.exists(tmp_image):
+                    os.remove(tmp_image)
+                self.glance_connect_master()
+
+        return downloaded
+
+    def delete_images_from_slave(self, to_delete_images_ids):
+        """walks through local metadata files
+        deletes images not found in local storage
+        """
+        if not to_delete_images_ids:
+            self.log('WARNING: precautionary halt. No glance images found '
+                     'to be deleted. noop.')
+            return
+
+        for image_id in to_delete_images_ids:
+            self.log('INFO: removing image {0}'.format(image_id))
+            try:
+                # FIXME
+                self.glance_slave.images.delete(image_id)
+                self.log('DEBUG: image {0} removed'.format(image_id))
+            except (HTTPConflict,HTTPForbidden) as e:
+                self.log('ERROR: could not delete {0} :: '
+                         '{1}'.format(image_id, e))
+
+    def create_missing_slave_images(self, processed_images_ids):
+
+        for dirpath, dirnames, filenames in os.walk(self.DATA_DIR):
+            if dirpath != self.DATA_DIR and \
+               len(dirnames) == 0 and \
+               len(filenames) == 0:
+                os.rmdir(dirpath)
+                continue
+
+            for filename in filenames:
+                full_path = os.path.join(dirpath, filename)
+                if filename.endswith('.json'):
+                    image_id = filename[:-5]
+                    if image_id in processed_images_ids:
+                        continue
+
+                    metadata_local = self.read_metadata(full_path)
+                    if not metadata_local:
+                        self.log('ERROR: read_metadata did not '
+                                 'retrieve anything '
+                                 '({0})'.format(full_path))
+                        continue
+
+                    slave_project_id = self.project_mapping(metadata_local)
+                    if not slave_project_id:
+                        self.log('DEBUG: could not map image into any '
+                                 'slave project :: {0}'.format(metadata_local))
+                        continue
+
+                    metadata_local['owner'] = slave_project_id
+                    if self.download_from_master(metadata_local):
+                        self.upload_to_slave(metadata_local)
+                    else:
+                        self.log('ERROR: image {0} could not be downloaded '
+                                 'from master'.format(metadata_local['id']))
+
+    def project_mapping(self, metadata_local):
+        """can master/slave projects be mapped, no matter project_ids are not
+        the same?
+        """
+        # master/slave match can't be done (no project_id)
+        if 'owner' not in metadata_local:
+            return False
+
+        # master/slave project_ids match
+        if metadata_local['owner'] in self.projects_slave:
+            return metadata_local['owner']
+
+        # no extra project_name passed -- can't check match
+        if 'bs_owner' not in metadata_local:
+            return False
+
+        master_project_name = metadata_local['bs_owner']
+
+        # XXX(aluria): no image on slave service
+        # XXX(aluria): look for similar project on slave
+        for slave_project_id, slave_project_name in \
+          self.projects_slave.items():
+            slave_to_master = slave_project_name.replace(self.REGION_SLAVE,
+                                                         self.REGION_MASTER)
+            # XXX(aluria): pitfall, if on master service there are
+            # XXX(aluria): 2 projects:
+            # XXX(auria):   REGION_SLAVE-restofprojectname
+            # XXX(auria):   REGION_MASTER-restofprojectname
+            # XXX(auria): first found gets image assigned
+            if master_project_name in (slave_project_name,
+                                       slave_to_master):
+                return slave_project_id
+        return False
+
+    # generate checksum on disk
+    def check_md5(self, control_md5, image_file):
+        return self.get_md5(image_file) == control_md5
+
+    def get_md5(self, image_file):
+        hash_md5 = hashlib.md5()
+        with open(image_file, "rb") as f:
+                for chunk in iter(lambda: f.read(4096), b""):
+                    hash_md5.update(chunk)
+        return hash_md5.hexdigest()
+
+    def read_metadata(self, metadata_file):
+        """copy full json file into memory
+        """
+        if os.path.isfile(metadata_file):
+            with open(metadata_file) as meta_file:
+                try:
+                    data = json.load(meta_file)
+                    return data
+                except Exception, e:
+                    self.log('EXCEPTION: {0}'.format(e))
+                    return False
+        else:
+            self.log('INFO: {0} not found.'.format(metadata_file))
+            return False
+
+    def glance_connect_slave(self):
+        try:
+            username = os.environ['OS_USERNAME']
+            password = os.environ['OS_PASSWORD']
+            tenant_name = os.environ['OS_TENANT_NAME']
+            auth_url = os.environ['OS_AUTH_URL']
+            self.REGION_MASTER = os.environ['OS_REGION_NAME_MASTER'].upper()
+        except Exception as e:
+            self.log('EXCEPTION: {0}'.format(e))
+            self.log('ERROR: unable to load environment variables, please '
+                     'source novarc')
+            self.release_lock()
+            sys.exit(2)
+        self.keystone = client.Client(username=username, password=password,
+                                      tenant_name=tenant_name,
+                                      auth_url=auth_url)
+
+        if not self.projects_slave:
+            self.projects_slave = dict([(tenant.id, tenant.name) for tenant in
+                                       self.keystone.tenants.list() if
+                                       tenant.enabled])
+        token = self.keystone.auth_token
+        service = self.keystone.services.find(name='glance')
+        endpoint = self.keystone.endpoints.find(service_id=service.id)
+        glance_url = endpoint.internalurl
+        self.REGION_SLAVE = endpoint.region.upper()
+        self.glance_slave = Client('2', endpoint=glance_url, token=token)
+        return self.glance_slave
+
+    def glance_connect_master(self):
+        try:
+            username = 'glancesync'
+            password = os.environ['OS_PASSWORD_MASTER']
+            tenant_name = os.environ['OS_TENANT_NAME']
+            auth_url = os.environ['OS_AUTH_URL_MASTER']
+            glance_url_master = os.environ['OS_GLANCE_URL_MASTER']
+        except Exception as e:
+            self.log('EXCEPTION: glance_connect_master :: {0}'.format(e))
+            self.log('ERROR: unable to load environment variables, please '
+                     'source novarc')
+            self.release_lock()
+            sys.exit(2)
+        self.keystone_master = client.Client(username=username,
+                                             password=password,
+                                             tenant_name=tenant_name,
+                                             auth_url=auth_url)
+        token = self.keystone_master.auth_token
+        # XXX: uses adminURL, which is OS-MGMT
+        # XXX: publicURL should be used
+        #service = self.keystone_master.services.find(name='glance')
+        #endpoint = self.keystone_master.endpoints.find(service_id=service.id)
+        #glance_url = endpoint.publicurl
+        #self.REGION_MASTER = endpoint.region.upper()
+        self.glance_master = Client('2', endpoint=glance_url_master,
+                                    token=token)
+        return self.glance_master
+
+    def timestamp_now(self):
+        return datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+
+    def log(self, msg):
+        print('{0} {1}'.format(self.timestamp_now(), msg))
+
+    def mangle_metadata(self, metadata_local, metadata_slave=None):
+        """Maps projects in MASTER region with projects in SLAVE region
+        metadata_local: metadata from master images
+        metatada_slave: metadata from slave images
+
+        (from glanceclient/v2/image_schema.py)
+        [
+        "image_state": "available",
+        "container_format": "bare",
+        "min_ram": 0,
+        "ramdisk_id": null,
+        "updated_at": "2017-02-06T12:01:31Z",
+        "file": "/v2/images/1019f307-0c1a-4b23-bf9f-725150ab80ab/file",
+        "owner": "ae75c492635143f79fa9161d73e0cb73",
+        "id": "1019f307-0c1a-4b23-bf9f-725150ab80ab",
+        "size": 34965880832,
+        "user_id": "7ac9feb8b7b04761a780f6aea1ee8a02",
+        "image_type": "snapshot",
+        "disk_format": "qcow2",
+        "base_image_ref": "6c738559-fb83-4fb4-98dd-6c28b100e417",
+        "schema": "/v2/schemas/image",
+        "status": "active",
+        "image_location": "snapshot",
+        "tags": [],
+        "kernel_id": null,
+        "visibility": "private",
+        "locations": [
+            {
+                "url": "rbd://75f1cd1b-1ef4-4a90-9033-5c026959f679/
+                  glance/1019f307-0c1a-4b23-bf9f-725150ab80ab/snap",
+                "metadata": {}
+            }
+        ],
+        "min_disk": 40,
+        "virtual_size": null,
+        "instance_uuid": "db494977-27be-4875-939b-d2af0f7035a4",
+        "name": "ams-samdb-snap",
+        "checksum": "f82eb5d18273b28f74175c2ce3c8ff18",
+        "created_at": "2017-02-06T11:54:09Z",
+        "protected": false,
+        "bs_owner": "AMS1-VEPC-MGMT-CENTRAL",
+        "owner_id": "ae75c492635143f79fa9161d73e0cb73"
+        ]
+        > pprint.pprint([key for key in d.keys() if key not in
+        > glance_properties+extra_properties])
+        ['user_id',
+         'image_type',
+         'image_state',
+         'image_location',
+         'base_image_ref',
+         'owner_id']
+        """
+        if 'owner' not in metadata_local:
+            raise (OSProjectNotFound, 'no owner :: {0}'.format(metadata_local))
+
+        if metadata_local['owner'] not in self.projects_slave:
+            if 'bs_owner' in metadata_local:
+                master_project_name = metadata_local['bs_owner']
+            else:
+                raise (OSProjectNotFound, 'no bs_owner :: '
+                       '{0}'.format(metadata_local))
+
+            # XXX(aluria): image does not exist on slave service
+            if not metadata_slave:
+                slave_project_id = self.project_mapping(metadata_local)
+                if not slave_project_id:
+                    raise (OSProjectNotFound, 'no project_mapping :: '
+                           '{0}'.format(metadata_local))
+                elif metadata_local['owner'] != slave_project_id:
+                    metadata_local['owner'] = slave_project_id
+            # XXX(aluria): image exists on slave service
+            # XXX(aluria): keep all metadata and mangle project_id (owner)
+            else:
+                # ie. admin, services, SLAVE-CENTRAL
+                slave_project_name = \
+                        self.projects_slave[metadata_slave['owner']]
+                # ie. admin, services, MASTER-CENTRAL
+                slave_to_master = slave_project_name.replace(self.REGION_SLAVE,
+                                                            self.REGION_MASTER)
+                # ie. admin, services, MASTER-CENTRAL
+                if master_project_name in (slave_project_name, slave_to_master):
+                    metadata_local['owner'] = metadata_slave['owner']
+                else:
+                    raise (OSProjectNotFound, 'project not found: '
+                          '{0}'.format(metadata_local))
+
+        # do not remove properties defined in extra_properties and 
+        # glance_properties
+        removed_props = [k for k in metadata_slave.keys() if k not in
+                         metadata_local.keys() and k not in
+                         self.extra_properties.union(
+                             self.glance_properties)]
+        return (metadata_local, removed_props)
+
+
+    def update_metadata(self, metadata_local, metadata_slave):
+        self.log('INFO: image-id {0}: updating '
+                 'metadata'.format(metadata_local['id']))
+
+        metadata, removed_props = self.mangle_metadata(metadata_local,
+                                                       metadata_slave)
+        # do not update read only and extra properties
+        for k in metadata_local.keys():
+            if k in extra_properties.union(readonly_properties):
+                del metadata_local[k]
+        try:
+            self.glance_slave.images.update(metadata['id'],
+                                            remove_props=removed_props,
+                                            **metadata)
+        except Exception as e:
+            self.log('EXCEPTION: update_metadata :: {0} - '
+                     '{1}'.format(metadata['id'], e))
             raise e
 
-    # lists filenames relative to path, not full path to match with glance names
-    # so with a directory/subdirectories like the following
-    # /tmp/foo
-    # /tmp/bar/baz
-    #
-    # list_files('/tmp') would return:
-    # ['foo','bar/baz']
-    #
-    def list_files(self, path):
-        file_list = []
-        for root, dirs, files in os.walk(path):
-            for file in files:
-                file_list.append(os.path.join(root, file))
-        relative = [re.sub(path+'/*', '', file) for file in file_list]
-        return relative
-
     def create_lock(self, lockfile):
         try:
             with open(lockfile, 'w') as lock:
                 lock.write(str(os.getpid()))
         except OSError:
-            print("{} ERROR: could not create lockfile {}".format(
-                self.timestamp_now(), lockfile))
+            self.log('ERROR: could not create lockfile {0}'.format(lockfile))
 
-    def file_locked(self, lockfile):
+    def file_locked(self, lockfile='/tmp/glance_sync_slave.lock'):
         if os.path.isfile(lockfile):
             return True
         else:
             return False
 
-    def release_lock(self):
-        lockfile = '/tmp/glance_sync_slave.lock'
+    def release_lock(self, lockfile='/tmp/glance_sync_slave.lock'):
         if os.path.isfile(lockfile):
             try:
                 os.remove(lockfile)
             except OSError as e:
-                print(e)
-                os.system("rm -f {}".format(lockfile))
-
-    def main(self, data_dir, source, lockfile):
-
+                self.log(e)
+
+    def set_filelock(self, lockfile='/tmp/glance_sync_slave.lock'):
         if self.file_locked(lockfile):
-            message = "{} WARNING: sync already in progress, exiting".format(
-                self.timestamp_now())
-            print(message)
-            sys.exit(1)
+            self.log('WARNING: sync already in progress, exiting')
+            sys.exit(2)
 
         self.create_lock(lockfile)
         atexit.register(self.release_lock)
 
-        print("{} starting glance image sync run".format(self.timestamp_now()))
-
-        # get images and metadata from master source
-        print("{} getting images and metadata from master".format(
-            self.timestamp_now()))
-        self.download_data(source, data_dir)
-
-        print("{} starting import into glance".format(self.timestamp_now()))
-        self.glance_upload(data_dir)
-
-        print("{} cleaning up files that were removed from master".format(
-            self.timestamp_now()))
-        self.glance_removal(data_dir)
-        print("{} ending glance image sync slave run".format(
-            self.timestamp_now()))
+    def main(self):
+        self.log('starting glance sync')
+        self.log('getting metadata from master')
+        self.download_metadata_from_master()
+        processed_images_ids = self.parse_glance_slave_images()
+        self.create_missing_slave_images(processed_images_ids)
+        self.log('ending glance image sync slave run')
         self.release_lock()
 
 if __name__ == '__main__':
-    parser = argparse.ArgumentParser(description='Synchronize remote images to '
-                                                 'disk and import into glance ')
+    parser = argparse.ArgumentParser(description='Synchronize remote images '
+                                     'metadata to disk and import into glance')
     parser.add_argument("-d", "--datadir", help="directory to write images to")
     parser.add_argument("-s", "--source", help="full path to master rsync "
                                                "source. Format: "
@@ -310,7 +614,6 @@
                                                "<directory>")
     args = parser.parse_args()
 
-    data_dir = ""
     if args.datadir:
         data_dir = args.datadir
     else:
@@ -323,5 +626,5 @@
         parser.print_help()
         sys.exit('ERROR: please specify an image source to sync from')
 
-    slave = ImageSyncSlave()
-    slave.main(data_dir, source, lockfile='/tmp/glance_sync_slave.lock')
+    slave = ImageSyncSlave(data_dir, source)
+    slave.main()

=== modified file 'reactive/glance-sync-slave.py'
--- reactive/glance-sync-slave.py	2016-05-19 17:30:30 +0000
+++ reactive/glance-sync-slave.py	2017-08-03 07:40:40 +0000
@@ -17,6 +17,7 @@
     fetch.apt_update()
     fetch.apt_install('python-glanceclient')
     fetch.apt_install('python-nose')
+    fetch.apt_install('python-dateutil')
 
     # configure the variables that have a default value
     configure_config_dir()
@@ -151,6 +152,26 @@
 def configure_custom_novarc():
     configure_novarc()
 
+@when('config.changed.master_{password,auth_url,glance_url,region}')
+def configure_master_novarc():
+    context = {}
+    for i in ('master_password', 'master_auth_url',
+              'master_glance_url', 'master_region'):
+        value = hookenv.config(i)
+        if not value:
+            hookenv.status_set('blocked', 'master service credentials '
+                               'missing')
+            return
+        context[i] = value
+
+    glancesync_novarc_file = os.path.join(config_dir, 'glancesync.novarc')
+
+    templating.render(source='glancesync.novarc.j2',
+                      target=glancesync_novarc_file,
+                      owner='ubuntu',
+                      perms=0o600,
+                      context=relation)
+    hookenv.status_set('active', 'glancesync.novarc configured')
 
 @hook('{requires:keystone-admin}-relation-{joined,changed}')
 def configure_relation_novarc(relation=None):

=== modified file 'templates/glance_sync_slave_cron.j2'
--- templates/glance_sync_slave_cron.j2	2017-01-11 03:55:07 +0000
+++ templates/glance_sync_slave_cron.j2	2017-08-03 07:40:40 +0000
@@ -6,7 +6,7 @@
 MAILTO={{ admin_email }}
 SHELL=/bin/bash
 
-{{ cron_frequency }} ubuntu bash -c 'source {{ config_dir}}/novarc && python {{ script_dir }}/glance_sync_slave.py -d {{ data_dir }} -s {{ sync_source }} >> {{ log_dir }}/glance_sync_slave.log 2>&1'
+{{ cron_frequency }} ubuntu bash -c 'source {{ config_dir }}/glancesync.novarc && source {{ config_dir}}/novarc && python {{ script_dir }}/glance_sync_slave.py -d {{ data_dir }} -s {{ sync_source }} >> {{ log_dir }}/glance_sync_slave.log 2>&1'
 
 {% endif %}
 

=== added file 'templates/glancesync.novarc.j2'
--- templates/glancesync.novarc.j2	1970-01-01 00:00:00 +0000
+++ templates/glancesync.novarc.j2	2017-08-03 07:40:40 +0000
@@ -0,0 +1,5 @@
+export OS_PASSWORD_MASTER={{ master_password }}
+export OS_AUTH_URL_MASTER={{ master_auth_url }}
+export OS_GLANCE_URL_MASTER={{ master_glance_url }}
+export OS_REGION_NAME_MASTER={{ master_region }}
+

=== modified file 'templates/novarc.j2'
--- templates/novarc.j2	2016-04-26 17:54:16 +0000
+++ templates/novarc.j2	2017-08-03 07:40:40 +0000
@@ -1,4 +1,5 @@
-export OS_AUTH_URL=http://{{ service_hostname }}:{{ service_port }}/v2.0
+#export OS_AUTH_URL=http://{{ service_hostname }}:{{ service_port }}/v2.0
+export OS_AUTH_URL=https://{{ service_hostname }}:{{ service_port }}/v2.0
 export OS_USERNAME={{ service_username }}
 export OS_PASSWORD={{ service_password }}
 export OS_TENANT_NAME={{ service_tenant_name }}