Merge lp:~adrien-delhorme/duplicity/hubic into lp:~duplicity-team/duplicity/0.7-series

[Adrien Delhorme]

Add Hubic support through pyrax and a custom pyrax_identity module.

[edso (ed.so)]

hi,

1. please update the manpage bin/duplicity.1, sections Requirements, URL Format as needed

2. please use the prefix cf+hubic to make clear that it depends on cf support, pyrax will become cf soon as the latter is deprecated

.e.de/duply.net

review: Needs Fixing

[lp:~adrien-delhorme/duplicity/hubic]

1015. By Adrien Delhorme on 2014-11-25

Update man page
Add hubic identity module

[Adrien Delhorme (adrien-delhorme)]

Hi,

I've pushed some fixes yesterday. Let me know if I can improve something.

[edso (ed.so)]

please recheck.. at least in manpage you have things like

<<<<<<< TREE
 and
======

which are leftovers from a conflict.

also you can join hubic & cloudfiles under REQUIREMENTS, as they both need pyrax.

is "hubiC" a typo?

thx ..ede

[edso (ed.so)]

ok, sorry.. no leftovers. actually your manpage file conflicts with the branch. there were changes in between.

update your local branch and readd/merge your changes to the manpage again. then repush your branch to this location.

..ede

[lp:~adrien-delhorme/duplicity/hubic]

1016. By Adrien Delhorme on 2014-11-27

Update man page

1017. By Adrien Delhorme on 2014-11-27

Merge lp:duplicity onto lp:~adrien-delhorme/duplicity/hubic

[Adrien Delhorme (adrien-delhorme)]

I did the merge.

hubiC, with a capital C is the real name: it stands for "hub in the Cloud".

[edso (ed.so)]

thanks.. ede

review: Approve

=== modified file 'bin/duplicity.1'
--- bin/duplicity.1	2014-10-30 17:22:39 +0000
+++ bin/duplicity.1	2014-11-28 06:24:34 +0000
@@ -64,7 +64,7 @@
 .B Cloud Files Python API (deprecated)
 - http://www.rackspace.com/knowledge_center/article/python-api-installation-for-cloud-files
 .TP
-.BR "cfpyrax backend" " (Rackspace Cloud)"
+.BR "cfpyrax backend" " (Rackspace Cloud) and " "hubic backend" " (hubic.com)"
 .B Rackspace CloudFiles Pyrax API
 - http://docs.rackspace.com/sdks/guide/content/python.html
 .TP
@@ -1143,6 +1143,15 @@
 hsi://user[:password]@other.host/some_dir
 .RE
 .PP
+.B "hubiC"
+.PP
+.RS
+cf+hubic://container_name
+.PP
+See also
+.B "A NOTE ON HUBIC"
+.RE
+.PP
 .B "IMAP email storage"
 .PP
 .RS
@@ -1635,6 +1644,25 @@
 Create Access Keys:
 https://code.google.com/apis/console#:storage:legacy
 
+.SH A NOTE ON HUBIC
+The hubic backend requires the pyrax library to be installed on the system. See REQUIREMENTS above. 
+You will need to set your credentials for hubiC in a file called ~/.hubic_credentials, following this
+pattern:
+.PP
+.RS
+[hubic]
+.br
+email = your_email
+.br
+password = your_password
+.br
+client_id = api_client_id
+.br
+client_secret = api_secret_key
+.br
+redirect_uri = http://localhost/
+.RE
+
 .SH A NOTE ON IMAP
 An IMAP account can be used as a target for the upload.  The userid may
 be specified and the password will be requested.

=== added file 'duplicity/backends/hubicbackend.py'
--- duplicity/backends/hubicbackend.py	1970-01-01 00:00:00 +0000
+++ duplicity/backends/hubicbackend.py	2014-11-28 06:24:34 +0000
@@ -0,0 +1,62 @@
+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
+#
+# Copyright 2013 J.P. Krauss <[e-mail]>
+#
+# This file is part of duplicity.
+#
+# Duplicity is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# Duplicity is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with duplicity; if not, write to the Free Software Foundation,
+# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import os
+
+import duplicity.backend
+from duplicity import log
+from duplicity import util
+from duplicity.errors import BackendException
+from ._cf_pyrax import PyraxBackend
+
+
+class HubicBackend(PyraxBackend):
+    """
+    Backend for Hubic using Pyrax
+    """
+    def __init__(self, parsed_url):
+        try:
+            import pyrax
+        except ImportError:
+            raise BackendException("This backend requires the pyrax "
+                                   "library available from Rackspace.")
+
+        # Inform Pyrax that we're talking to Hubic
+        pyrax.set_setting("identity_type", "duplicity.backends.pyrax_identity.hubic.HubicIdentity")
+
+        CREDENTIALS_FILE = os.path.expanduser("~/.hubic_credentials")
+        if os.path.exists(CREDENTIALS_FILE):
+            try:
+                pyrax.set_credential_file(CREDENTIALS_FILE)
+            except Exception as e:
+                log.FatalError("Connection failed, please check your credentials: %s %s"
+                               % (e.__class__.__name__, util.uexc(e)),
+                               log.ErrorCode.connection_failed)
+
+        else:
+            raise BackendException("No ~/.hubic_credentials file found.")
+
+        container = parsed_url.path.lstrip('/')
+
+        self.client_exc = pyrax.exceptions.ClientException
+        self.nso_exc = pyrax.exceptions.NoSuchObject
+        self.container = pyrax.cloudfiles.create_container(container)
+
+duplicity.backend.register_backend("cf+hubic", HubicBackend)

=== added directory 'duplicity/backends/pyrax_identity'
=== added file 'duplicity/backends/pyrax_identity/__init__.py'
--- duplicity/backends/pyrax_identity/__init__.py	1970-01-01 00:00:00 +0000
+++ duplicity/backends/pyrax_identity/__init__.py	2014-11-28 06:24:34 +0000
@@ -0,0 +1,20 @@
+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
+#
+# Copyright 2002 Ben Escoto <[e-mail]>
+# Copyright 2007 Kenneth Loafman <[e-mail]>
+#
+# This file is part of duplicity.
+#
+# Duplicity is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# Duplicity is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with duplicity; if not, write to the Free Software Foundation,
+# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
\ No newline at end of file

=== added file 'duplicity/backends/pyrax_identity/hubic.py'
--- duplicity/backends/pyrax_identity/hubic.py	1970-01-01 00:00:00 +0000
+++ duplicity/backends/pyrax_identity/hubic.py	2014-11-28 06:24:34 +0000
@@ -0,0 +1,256 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# Copyright (c) 2014 Gu1
+# Licensed under the MIT license
+
+import os
+import pyrax
+import pyrax.exceptions as exc
+import requests
+import re
+import urlparse
+import ConfigParser
+import time
+from pyrax.base_identity import BaseIdentity, Service
+from requests.compat import quote, quote_plus
+
+
+OAUTH_ENDPOINT = "https://api.hubic.com/oauth/"
+API_ENDPOINT = "https://api.hubic.com/1.0/"
+TOKENS_FILE = os.path.expanduser("~/.hubic_tokens")
+
+
+class BearerTokenAuth(requests.auth.AuthBase):
+    def __init__(self, token):
+        self.token = token
+
+    def __call__(self, req):
+        req.headers['Authorization'] = 'Bearer '+self.token
+        return req
+
+
+class HubicIdentity(BaseIdentity):
+
+    def _get_auth_endpoint(self):
+        return ""
+
+    def set_credentials(self, email, password, client_id,
+                        client_secret, redirect_uri,
+                        authenticate=False):
+        """Sets the username and password directly."""
+        self._email = email
+        self._password = password
+        self._client_id = client_id
+        self.tenant_id = client_id
+        self._client_secret = client_secret
+        self._redirect_uri = redirect_uri
+        if authenticate:
+            self.authenticate()
+
+    def _read_credential_file(self, cfg):
+        """
+        Parses the credential file with Rackspace-specific labels.
+        """
+        self._email = cfg.get("hubic", "email")
+        self._password = cfg.get("hubic", "password")
+        self._client_id = cfg.get("hubic", "client_id")
+        self.tenant_id = self._client_id
+        self._client_secret = cfg.get("hubic", "client_secret")
+        self._redirect_uri = cfg.get("hubic", "redirect_uri")
+
+    def _parse_error(self, resp):
+        if not 'location' in resp.headers:
+            return None
+        query = urlparse.urlsplit(resp.headers['location']).query
+        qs = dict(urlparse.parse_qsl(query))
+        return {'error': qs['error'], 'error_description': qs['error_description']}
+
+    def _get_access_token(self, code):
+        r = requests.post(
+            OAUTH_ENDPOINT+'token/',
+            data={
+                'code': code,
+                'redirect_uri': self._redirect_uri,
+                'grant_type': 'authorization_code',
+            },
+            auth=(self._client_id, self._client_secret)
+        )
+        if r.status_code != 200:
+            try:
+                err = r.json()
+                err['code'] = r.status_code
+            except:
+                err = {}
+
+            raise exc.AuthenticationFailed("Unable to get oauth access token, "
+                                           "wrong client_id or client_secret ? (%s)" % str(err))
+
+        oauth_token = r.json()
+
+        config = ConfigParser.ConfigParser()
+        config.read(TOKENS_FILE)
+
+        if not config.has_section("hubic"):
+            config.add_section("hubic")
+
+        if oauth_token['access_token'] is not None:
+            config.set("hubic", "access_token", oauth_token['access_token'])
+            with open(TOKENS_FILE, 'wb') as configfile:
+                config.write(configfile)
+        else:
+            raise exc.AuthenticationFailed("Unable to get oauth access token, wrong client_id or client_secret ? (%s)" % str(err))
+
+        if oauth_token['refresh_token'] is not None:
+            config.set("hubic", "refresh_token", oauth_token['refresh_token'])
+            with open(TOKENS_FILE, 'wb') as configfile:
+                config.write(configfile)
+        else:
+            raise exc.AuthenticationFailed("Unable to get the refresh token.")
+
+        # removing username and password from .hubic_tokens
+        if config.has_option("hubic", "email"):
+            config.remove_option("hubic", "email")
+            with open(TOKENS_FILE, 'wb') as configfile:
+                config.write(configfile)
+            print "username has been removed from the .hubic_tokens file sent to the CE."
+        if config.has_option("hubic", "password"):
+            config.remove_option("hubic", "password")
+            with open(TOKENS_FILE, 'wb') as configfile:
+                config.write(configfile)
+            print "password has been removed from the .hubic_tokens file sent to the CE."
+
+        return oauth_token
+
+    def _refresh_access_token(self):
+
+        config = ConfigParser.ConfigParser()
+        config.read(TOKENS_FILE)
+        refresh_token = config.get("hubic", "refresh_token")
+
+        if refresh_token is None:
+            raise exc.AuthenticationFailed("refresh_token is null. Not acquiered before ?")
+
+        success = False
+        max_retries = 20
+        retries = 0
+        sleep_time = 30
+        max_sleep_time = 3600
+
+        while retries < max_retries and not success:
+            r = requests.post(
+                OAUTH_ENDPOINT+'token/',
+                data={
+                    'refresh_token': refresh_token,
+                    'grant_type': 'refresh_token',
+                },
+                auth=(self._client_id, self._client_secret)
+            )
+            if r.status_code != 200:
+                if r.status_code == 509:
+                    print "status_code 509: attempt #", retries, " failed"
+                    retries += 1
+                    time.sleep(sleep_time)
+                    sleep_time = sleep_time * 2
+                    if sleep_time > max_sleep_time:
+                        sleep_time = max_sleep_time
+                else:
+                    try:
+                        err = r.json()
+                        err['code'] = r.status_code
+                    except:
+                        err = {}
+
+                    raise exc.AuthenticationFailed("Unable to get oauth access token, wrong client_id or client_secret ? (%s)" % str(err))
+            else:
+                success = True
+
+        if not success:
+            raise exc.AuthenticationFailed("All the attempts failed to get the refresh token: status_code = 509: Bandwidth Limit Exceeded")
+
+        oauth_token = r.json()
+
+        if oauth_token['access_token'] is not None:
+            return oauth_token
+        else:
+            raise exc.AuthenticationFailed("Unable to get oauth access token from json")
+
+    def authenticate(self):
+        config = ConfigParser.ConfigParser()
+        config.read(TOKENS_FILE)
+
+        if config.has_option("hubic", "refresh_token"):
+            oauth_token = self._refresh_access_token()
+        else:
+            r = requests.get(
+                OAUTH_ENDPOINT+'auth/?client_id={0}&redirect_uri={1}'
+                '&scope=credentials.r,account.r&response_type=code&state={2}'.format(
+                    quote(self._client_id),
+                    quote_plus(self._redirect_uri),
+                    pyrax.utils.random_ascii()  # csrf ? wut ?..
+                ),
+                allow_redirects=False
+            )
+            if r.status_code != 200:
+                raise exc.AuthenticationFailed("Incorrect/unauthorized "
+                                               "client_id (%s)" % str(self._parse_error(r)))
+
+            try:
+                from lxml import html as lxml_html
+            except ImportError:
+                lxml_html = None
+
+            if lxml_html:
+                oauth = lxml_html.document_fromstring(r.content).xpath('//input[@name="oauth"]')
+                oauth = oauth[0].value if oauth else None
+            else:
+                oauth = re.search(r'<input\s+[^>]*name=[\'"]?oauth[\'"]?\s+[^>]*value=[\'"]?(\d+)[\'"]?>', r.content)
+                oauth = oauth.group(1) if oauth else None
+
+            if not oauth:
+                raise exc.AuthenticationFailed("Unable to get oauth_id from authorization page")
+
+            if self._email is None or self._password is None:
+                raise exc.AuthenticationFailed("Cannot retrieve email and/or password. Please run expresslane-hubic-setup.sh")
+
+            r = requests.post(
+                OAUTH_ENDPOINT+'auth/',
+                data={
+                    'action': 'accepted',
+                    'oauth': oauth,
+                    'login': self._email,
+                    'user_pwd': self._password,
+                    'account': 'r',
+                    'credentials': 'r',
+
+                },
+                allow_redirects=False
+            )
+
+            try:
+                query = urlparse.urlsplit(r.headers['location']).query
+                code = dict(urlparse.parse_qsl(query))['code']
+            except:
+                raise exc.AuthenticationFailed("Unable to authorize client_id, invalid login/password ?")
+
+            oauth_token = self._get_access_token(code)
+
+        if oauth_token['token_type'].lower() != 'bearer':
+            raise exc.AuthenticationFailed("Unsupported access token type")
+
+        r = requests.get(
+            API_ENDPOINT+'account/credentials',
+            auth=BearerTokenAuth(oauth_token['access_token']),
+        )
+
+        swift_token = r.json()
+        self.authenticated = True
+        self.token = swift_token['token']
+        self.expires = swift_token['expires']
+        self.services['object_store'] = Service(self, {
+            'name': 'HubiC',
+            'type': 'cloudfiles',
+            'endpoints': [
+                {'public_url': swift_token['endpoint']}
+            ]
+        })
+        self.username = self.password = None

=== modified file 'setup.py'
--- setup.py	2014-10-27 02:27:36 +0000
+++ setup.py	2014-11-28 06:24:34 +0000
@@ -132,6 +132,7 @@
       url="http://duplicity.nongnu.org/index.html",
       packages = ['duplicity',
                   'duplicity.backends',
+                  'duplicity.backends.pyrax_identity',
                   'testing',
                   'testing.functional',
                   'testing.overrides',