[enhancement] please consider using rclone as backend

rclone looks good. Will have to investigate further.

Just curious... What makes 'pip install pydrive' an ugly experience?

On Fri, May 4, 2018 at 3:55 PM, c sights <email address hidden> wrote:

> Public bug reported:
>
> Using rclone as a backend might help simplify the setup of the many
> targets duplicity supports.
> someone has written a rclone adapter for duplicity already:
> https://github.com/GilGalaad/duplicity-rclone
>
> I'm planning to use it with Google Drive (as the author above did) and
> not have to install pydrive, which is an ugly pip experience
>
> rclone supports many other targets also (below).
> Thanks for your work!
>
> https://rclone.org/
>
> Amazon Drive
> Amazon S3
> Backblaze B2
> Box
> Ceph
> DigitalOcean Spaces
> Dreamhost
> Dropbox
> FTP
> Google Cloud Storage
> Google Drive
> HTTP
> Hubic
> IBM COS S3
> Memset Memstore
> Mega
> Microsoft Azure Blob Storage
> Microsoft OneDrive
> Minio
> Nextloud
> OVH
> Openstack Swift
> Oracle Cloud Storage
> ownCloud
> pCloud
> put.io
> QingStor
> Rackspace Cloud Files
> SFTP
> Wasabi
> WebDAV
> Yandex Disk
> The local filesystem
>
> ** Affects: duplicity
> Importance: Undecided
> Status: New
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/1769267
>
> Title:
> [enhancement] please consider using rclone as backend
>
> Status in Duplicity:
> New
>
> Bug description:
> Using rclone as a backend might help simplify the setup of the many
> targets duplicity supports.
> someone has written a rclone adapter for duplicity already:
> https://github.com/GilGalaad/duplicity-rclone
>
> I'm planning to use it with Google Drive (as the author above did) and
> not have to install pydrive, which is an ugly pip experience
>
> rclone supports many other targets also (below).
> Thanks for your work!
>
> https://rclone.org/
>
> Amazon Drive
> Amazon S3
> Backblaze B2
> Box
> Ceph
> DigitalOcean Spaces
> Dreamhost
> Dropbox
> FTP
> Google Cloud Storage
> Google Drive
> HTTP
> Hubic
> IBM COS S3
> Memset Memstore
> Mega
> Microsoft Azure Blob Storage
> Microsoft OneDrive
> Minio
> Nextloud
> OVH
> Openstack Swift
> Oracle Cloud Storage
> ownCloud
> pCloud
> put.io
> QingStor
> Rackspace Cloud Files
> SFTP
> Wasabi
> WebDAV
> Yandex Disk
> The local filesystem
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/duplicity/+bug/1769267/+subscriptions
>

Hi Ken,

> Just curious... What makes 'pip install pydrive' an ugly experience?

It's been a couple years. I seem to remember pip splattering files
throughout the local file hierarchy (/usr/share/python ?) and not
tracking them so that uninstall didn't reverse the process.

I prefer rclone's one file approach. (Probably pip is very lovable once
you get to know it. :) )

Also, I think back then pydrive or google-python-api-wrapper was
stagnant and depended on old other pip packages. My old config shows
that I was specifically using oauth2client==1.5.2 . Not sure why
anymore. :/

Thanks for your work!
C.

Could we simply support rclone as backend by integrating https://github.com/GilGalaad/duplicity-rclone into duplicity upstream?

https://github.com/GilGalaad/duplicity-rclone

has some clutter and should be cleaned up first.
- method subprocess should be removed
- local renaming of temp files seems superfluous
- error reporting just shows first messag line

also the url scheme should be streamlined to current duplicity standard eg.
rclone://gdrive:/ => rclone+gdrive://

and agn. manpage doc is missing

..ede

Yes, it needs a bit of work, especially conversion to Python 3. I am
thinking that, with everything in the backends, here is a product built for
talking to all the remotes we have, and then some.

It will need a man page entry, but that will probably just refer to the
rclone docs (much better than ours). We will normalize the process and put
in some tests as well.

On Mon, Nov 18, 2019 at 12:01 PM edso <email address hidden> wrote:

> https://github.com/GilGalaad/duplicity-rclone
>
> has some clutter and should be cleaned up first.
> - method subprocess should be removed
> - local renaming of temp files seems superfluous
> - error reporting just shows first messag line
>
> also the url scheme should be streamlined to current duplicity standard eg.
> rclone://gdrive:/ => rclone+gdrive://
>
> and agn. manpage doc is missing
>
> ..ede
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1769267
>
> Title:
> [enhancement] please consider using rclone as backend
>
> Status in Duplicity:
> In Progress
>
> Bug description:
> Using rclone as a backend might help simplify the setup of the many
> targets duplicity supports.
> someone has written a rclone adapter for duplicity already:
> https://github.com/GilGalaad/duplicity-rclone
>
> I'm planning to use it with Google Drive (as the author above did) and
> not have to install pydrive, which is an ugly pip experience
>
> rclone supports many other targets also (below).
> Thanks for your work!
>
> https://rclone.org/
>
> Amazon Drive
> Amazon S3
> Backblaze B2
> Box
> Ceph
> DigitalOcean Spaces
> Dreamhost
> Dropbox
> FTP
> Google Cloud Storage
> Google Drive
> HTTP
> Hubic
> IBM COS S3
> Memset Memstore
> Mega
> Microsoft Azure Blob Storage
> Microsoft OneDrive
> Minio
> Nextloud
> OVH
> Openstack Swift
> Oracle Cloud Storage
> ownCloud
> pCloud
> put.io
> QingStor
> Rackspace Cloud Files
> SFTP
> Wasabi
> WebDAV
> Yandex Disk
> The local filesystem
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/duplicity/+bug/1769267/+subscriptions
>

One possible show-stopper, duplicity is under GPL v2, rclonebackend.py is under GPL v3. I don't think we can arbitrarily mix the two licenses. Thoughts?

The license issue has been resolved.

https://github.com/GilGalaad/duplicity-rclone/issues/6#issuecomment-562692643

I've released a working version of rclonebackend.py, but am not happy yet.

1) The renames are a real sticking point. You can't 'rclone copy dir/a dir/b'. Instead of copying to dir/b, it will create dir/b/a. The renames are a point of confusion when debugging, especially with our Ctrl-C happy set of users. I am investigating the use of 'rclone cat' at the moment. That could be the solution.

2) No man page yet. Another sticking point.

3) There is a test case, so at least it works.

wrt. the license issue

as duplicity is licensed
"
# Duplicity is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the
# Free Software Foundation; either version 2 of the License, or (at your
# option) any later version.
"
note the _any later version_ option, we can easily include GPL3 code, which will make the whole then a GPL2 and GPL3 mixture. totally legal ;)
the other way around won't work though, basing GPL2'd code on GPL3 code.

..ede/duply.net

> 1) The renames are a real sticking point. You can't 'rclone copy dir/a
> dir/b'. Instead of copying to dir/b, it will create dir/b/a. The
> renames are a point of confusion when debugging, especially with our
> Ctrl-C happy set of users. I am investigating the use of 'rclone cat'
> at the moment. That could be the solution.

rclone moveto ?
https://rclone.org/commands/rclone_moveto/

Congratulations! Very exciting.
C.

On 3/19/20 2:47 PM, Kenneth Loafman wrote:
> ** Changed in: duplicity
> Status: Fix Committed => Fix Released
>