Merge lp:~user3942934/duplicity/pydrive into lp:~duplicity-team/duplicity/0.7-series

[westerngateguard]

Hello,

Currently duplicity uses gdocs backend for Google Drive backups.
gdocs uses deprecated API and don't allow backups for managed Google accounts.
(see https://bugs.launchpad.net/duplicity/+bug/1315684)

I added pydrive backend that solves both of those problems. I published it also on GitHub: https://github.com/westerngateguard/duplicity-pydrive-backend.

It works fine on my Debian (Wheezy) machine.

[Michael Terry (mterry)]

I think you accidentally deleted some azure documentation from the man page?

Also, you use a keyfile specified by the URL. But all of the other backends tend to use environment variables for that sort of thing. Is there a reason you went with a file?

[Kenneth Loafman (kenneth-loafman)]

I agree with Michael on this. We should stay with environment variables
until we can go to a backup level configuration file.

On Tue, Jan 13, 2015 at 7:54 AM, Michael Terry <email address hidden>
wrote:

> I think you accidentally deleted some azure documentation from the man
> page?
>
> Also, you use a keyfile specified by the URL. But all of the other
> backends tend to use environment variables for that sort of thing. Is
> there a reason you went with a file?
> --
> https://code.launchpad.net/~user3942934/duplicity/duplicity/+merge/246258
> You are subscribed to branch lp:duplicity.
>

[lp:~user3942934/duplicity/pydrive]

1052. By Yigal Asnis <email address hidden> on 2015-01-14

Changed passing keyfile via query in URL to passing the key by environment variable. Fixed manpage accordingly. Restored Azure info in manpage that was somehow mistakenly deleted.

[westerngateguard (user3942934)]

Do I need resubmit the merging proposal or the commit message is enough?

=== modified file 'bin/duplicity.1'
--- bin/duplicity.1	2015-01-10 19:38:59 +0000
+++ bin/duplicity.1	2015-01-14 11:12:41 +0000
@@ -149,7 +149,15 @@
 .br
 (also see
 .BR "A NOTE ON SSL CERTIFICATE VERIFICATION" ).
-
+.TP
+.BR "pydrive backend"
+.B PyDrive -- a wrapper library of google-api-python-client
+- https://pypi.python.org/pypi/PyDrive
+.br
+(also see
+.BR "A NOTE ON PYDRIVE BACKEND"
+) below.
+.br
 .SH DESCRIPTION
 Duplicity incrementally backs up files and folders into
 tar-format volumes encrypted with GnuPG and places them to a
@@ -1277,6 +1285,16 @@
 .B alternatively
 try lftp+webdav[s]://
 .RE
+.PP
+.BR "pydrive"
+.PP
+.RS
+pydrive://<service account' email address>@developer.gserviceaccount.com/some_dir
+.PP
+See also
+.B "A NOTE ON PYDRIVE BACKEND"
+below.
+.RE
 
 .SH TIME FORMATS
 duplicity uses time strings in two places.  Firstly, many of the files
@@ -1859,6 +1877,27 @@
 .B SWIFT_AUTHVERSION
 is unspecified, it will default to version 1.
 
+.SH A NOTE ON PYDRIVE BACKEND
+The pydrive backend requires Python PyDrive package to be installed on the system. See
+.B REQUIREMENTS
+above.
+ 
+You need to create a service account in "Google developers console" at https://console.developers.google.com
+
+Make sure Drive API is enabled. 
+
+The email address of the account will be used as part of URL. See
+.B URL FORMAT
+above.
+
+Download the .p12 key file of the account and convert it to the .pem format:
+.br
+openssl pkcs12 -in XXX.p12  -nodes -nocerts > pydriveprivatekey.pem
+
+The content of .pem file should be passed to 
+.BR GOOGLE_DRIVE_ACCOUNT_KEY
+environment variable for authentification.
+
 .SH A NOTE ON SYMMETRIC ENCRYPTION AND SIGNING
 Signing and symmetrically encrypt at the same time with the gpg binary on the
 command line, as used within duplicity, is a specifically challenging issue.

=== modified file 'duplicity/backend.py'
--- duplicity/backend.py	2014-12-17 10:35:11 +0000
+++ duplicity/backend.py	2015-01-14 11:12:41 +0000
@@ -251,7 +251,7 @@
             pu = urlparse.urlparse(url_string)
         except Exception:
             raise InvalidBackendURL("Syntax error in: %s" % url_string)
-
+                
         try:
             self.scheme = pu.scheme
         except Exception:

=== added file 'duplicity/backends/pydrivebackend.py'
--- duplicity/backends/pydrivebackend.py	1970-01-01 00:00:00 +0000
+++ duplicity/backends/pydrivebackend.py	2015-01-14 11:12:41 +0000
@@ -0,0 +1,107 @@
+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
+#
+# Copyright 2015 Yigal Asnis 
+#
+# This file is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# It is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with duplicity; if not, write to the Free Software Foundation,
+# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import string
+import os
+
+import duplicity.backend
+from duplicity.errors import BackendException
+
+class PyDriveBackend(duplicity.backend.Backend):
+    """Connect to remote store using PyDrive API"""
+
+    def __init__(self, parsed_url):
+        duplicity.backend.Backend.__init__(self, parsed_url)
+        try:
+            global pydrive
+            import httplib2
+            from apiclient.discovery import build
+            from oauth2client.client import SignedJwtAssertionCredentials
+            from pydrive.auth import GoogleAuth
+            from pydrive.drive import GoogleDrive
+        except ImportError:
+            raise BackendException('PyDrive backend requires PyDrive installation'
+                                   'Please read the manpage to fix.')
+        
+        if 'GOOGLE_DRIVE_ACCOUNT_KEY' not in os.environ:
+            raise BackendException('GOOGLE_DRIVE_ACCOUNT_KEY environment variable not set. Please read the manpage to fix.')
+        account_key = os.environ['GOOGLE_DRIVE_ACCOUNT_KEY']
+
+        credentials = SignedJwtAssertionCredentials(parsed_url.username + '@' + parsed_url.hostname, account_key, scope='https://www.googleapis.com/auth/drive')
+        credentials.authorize(httplib2.Http())
+        gauth = GoogleAuth()
+        gauth.credentials = credentials
+        self.drive = GoogleDrive(gauth)
+        
+        # Dirty way to find root folder id
+        file_list = self.drive.ListFile({'q': "'Root' in parents"}).GetList()
+        if file_list:
+            parent_folder_id = file_list[0]['parents'][0]['id']
+        else:
+            file_in_root = self.drive.CreateFile({'title': 'i_am_in_root'})
+            file_in_root.Upload()
+            parent_folder_id = file_in_root['parents'][0]['id']
+                         
+        # Fetch destination folder entry and create hierarchy if required.
+        folder_names = string.split(parsed_url.path, '/')
+        for folder_name in folder_names:
+            if not folder_name:
+                continue
+            file_list = self.drive.ListFile({'q': "'" + parent_folder_id + "' in parents"}).GetList()
+            folder =  next((item for item in file_list if item['title'] == folder_name and item['mimeType'] == 'application/vnd.google-apps.folder'), None)
+            if folder is None:
+                folder = self.drive.CreateFile({'title': folder_name, 'mimeType': "application/vnd.google-apps.folder", 'parents': [{'id': parent_folder_id}]})
+                folder.Upload()
+            parent_folder_id = folder['id']
+        self.folder = parent_folder_id
+        
+    def FilesList(self):
+        return self.drive.ListFile({'q': "'" + self.folder + "' in parents"}).GetList()
+    
+    def id_by_name(self, filename):
+        try:
+            return next(item for item in self.FilesList() if item['title'] == filename)['id']
+        except:
+            return ''
+            
+    def _put(self, source_path, remote_filename):
+        drive_file = self.drive.CreateFile({'title': remote_filename, 'parents': [{"kind": "drive#fileLink", "id": self.folder}]})
+        drive_file.SetContentFile(source_path.name)
+        drive_file.Upload()
+
+    def _get(self, remote_filename, local_path):
+        drive_file = self.drive.CreateFile({'id': self.id_by_name(remote_filename)})
+        drive_file.GetContentFile(local_path.name)
+        
+    def _list(self):
+        return [item['title'] for item in self.FilesList()]
+       
+    def _delete(self, filename):
+        file_id = self.id_by_name(filename)
+        drive_file = self.drive.CreateFile({'id': file_id})
+        drive_file.auth.service.files().delete(fileId=drive_file['id']).execute()
+        
+    def _query(self, filename):
+        try:
+            size = int((item for item in self.FilesList() if item['title'] == filename).next()['fileSize'])
+        except:
+            size = -1
+        return {'size': size}
+                   
+duplicity.backend.register_backend('pydrive', PyDriveBackend)
+duplicity.backend.uses_netloc.extend([ 'pydrive' ])

=== modified file 'duplicity/commandline.py'
--- duplicity/commandline.py	2015-01-10 19:38:59 +0000
+++ duplicity/commandline.py	2015-01-14 11:12:41 +0000
@@ -856,6 +856,7 @@
   webdav://%(user)s[:%(password)s]@%(other_host)s/%(some_dir)s
   webdavs://%(user)s[:%(password)s]@%(other_host)s/%(some_dir)s
   gdocs://%(user)s[:%(password)s]@%(other_host)s/%(some_dir)s
+  pydrive://%(user)s@%(other_host)s/%(some_dir)s
   mega://%(user)s[:%(password)s]@%(other_host)s/%(some_dir)s
   copy://%(user)s[:%(password)s]@%(other_host)s/%(some_dir)s
   dpbx:///%(some_dir)s