lp:rclone-dev

Author: Nick Craig-Wood
Author Date: 2021-09-30 10:11:46 UTC

union: fix rename not working with union of local disk and bucket based remote

Before this change the union's feature flags were a strict AND of the
underlying remotes. This means that a union of a local disk (which can
Move but not Copy) and a bucket based remote (which can Copy but not
Move) could neither Move nor Copy.

This fix advertises Move in the union if all the remotes can Move or
Copy. It also implements Move as Copy+Delete (like rclone does
normally) if the underlying union does not support Move.

This enables renames to work with unions of local disk and bucket
based remotes expected.

Fixes #5632

Author: Ivan Andreev
Author Date: 2021-05-16 17:26:43 UTC

bisync: documentation - #5164

Co-authored-by: Chris Nelson <stuff@cjnaz.com>

Author: Nick Craig-Wood
Author Date: 2021-09-19 12:09:32 UTC

Start v1.56.2-DEV development

Author: albertony
Author Date: 2021-09-16 20:37:51 UTC

config: allow dot in remote names

Author: albertony
Author Date: 2021-09-16 18:02:16 UTC

docs: extend documentation on valid remote names

Author: Nick Craig-Wood
Author Date: 2020-09-23 16:20:28 UTC

lsjson: add --stat flag and operations/stat api FIXME WIP

Needs tests, docs. Is there an issue?

Not sure it is 100% working!

This was rebased with fixed conflicts which may have broken stuff too

Author: Nick Craig-Wood
Author Date: 2021-09-04 11:47:18 UTC

serve ftp: ensure modtime is passed as UTC always to fix timezone oddities

See: https://forum.rclone.org/t/ftp-server-reports-file-timestamps-in-utc/26274/4

Author: Nick Craig-Wood
Author Date: 2021-08-23 14:57:06 UTC

box: retry operation_blocked_temporary errors #5545

Author: Cnly
Author Date: 2021-08-22 05:09:08 UTC

Don't replace go build with make

Author: Nick Craig-Wood
Author Date: 2021-08-14 12:22:54 UTC

dropbox: add --dropbox-batch-commit-timeout to control batch timeout

This also adds an Debug message log showing how long each batch took

See: #5491

Author: Nick Craig-Wood
Author Date: 2021-08-13 16:27:49 UTC

s3: Use a combination of SDK retries and rclone retries - fixes #5509

This reverts commit

dc06973796392edf Revert "s3: use rclone's low level retries instead of AWS SDK to fix listing retries"

Which in turn reverted

5470d34740d03e15 "backend/s3: use low-level-retries as the number of SDK retries"

So we are back where we started.

It then modifies it to set the AWS SDK to `--low-level-retries`
retries, but set the rclone retries to 2 so that directory listings
can be retried.

Author: Nick Craig-Wood
Author Date: 2021-08-11 10:47:24 UTC

s3: Attempt to fix auth problems #5468

Revert "s3: remove WebIdentityRoleProvider to fix crash on auth #5255"

This reverts commit e618ea83dd3e539b53d2e60b49336e27347b56f8.

Author: Nick Craig-Wood
Author Date: 2021-08-01 13:24:05 UTC

crypt: fix uploads with --crypt-no-data-encryption

Before this change, when uploading to a crypt, the ObjectInfo
accidentally used the encrypted size, not the unencrypted size when
--crypt-no-data-encryption was set.

Fixes #5498

Author: Nick Craig-Wood
Author Date: 2021-07-31 10:52:34 UTC

sftp: add extra debuging if --dump headers is used

Author: Cnly
Author Date: 2021-07-27 09:23:13 UTC

fix code quality problems

Author: Cnly
Author Date: 2021-06-21 16:33:46 UTC

fs/config: make --dump imply -vv

Author: Nick Craig-Wood
Author Date: 2021-06-16 11:02:45 UTC

operations: don't remove destination files on errored transfers

See: https://forum.rclone.org/t/transfer-on-mega-in-ftp-mode-is-not-working/24642/7

Author: Nick Craig-Wood
Author Date: 2021-06-15 17:41:56 UTC

fixup! fichier: implement DirMove

Author: Nick Craig-Wood
Author Date: 2021-02-07 17:18:52 UTC

s3: don't check to see if remote is object if it ends with /

Before this change, rclone would always check the root to see if it
was an object.

This change doesn't check to see if the root is an object if the path
ends with a /

This avoids a transaction where rclone HEADs the path to see if it
exists.

See #4990

Author: Nick Craig-Wood
Author Date: 2021-04-29 13:38:39 UTC

local: add debugging for link problem

See: https://forum.rclone.org/t/problem-with-symlinks-and-links/23840/13

Author: Nick Craig-Wood
Author Date: 2021-04-12 16:36:59 UTC

config: remove log.Fatal from loading config file #5234

Author: Nick Craig-Wood
Author Date: 2021-02-23 20:23:21 UTC

webdav: chunked uploading - WIP DO NOT MERGE

Author: Nick Craig-Wood
Author Date: 2021-04-26 07:32:21 UTC

Version v1.55.1

Author: Nick Craig-Wood
Author Date: 2021-04-11 10:56:57 UTC

b2: fix decoding of 404 errors producing "Couldn't decode error response: EOF"

See: https://forum.rclone.org/t/move-to-b2-creates-empty-file-versions/23420/

Author: Nick Craig-Wood
Author Date: 2021-04-10 11:30:34 UTC

gcs: allow uncompressed downloads of Content-Type encoding gzip #2658

Author: Nick Craig-Wood
Author Date: 2020-06-28 16:36:50 UTC

gzip: new backend for wrapping Content-Encoding: gzip backends FIXME WIP

See: #2658

Author: Nick Craig-Wood
Author Date: 2021-04-08 11:39:03 UTC

fs: dump config for an Fs before creation - FIXME TESTING DO NOT MERGE

See: https://forum.rclone.org/t/authenticating-to-box-using-connection-strings-and-config-file-fails/23333/7

Author: Nick Craig-Wood
Author Date: 2021-03-31 08:52:47 UTC

onedrive: work around duplicated directory listing entries

When paging big directories onedrive sometimes duplicates the last
item on one page as the first item of the next page.

This patch detects that and skips the duplicated item with an error
message.

See: https://forum.rclone.org/t/unexpected-duplicates-on-onedrive-with-0s-in-filename/23164

Author: Nick Craig-Wood
Author Date: 2021-03-29 18:32:09 UTC

webdav: fix DirMove for bitrix - FIXME DO NOT MERGE

This implements a workaround to enable DirMove for bitrix to work.

This is an apparently non RFC compliant change.

See: https://forum.rclone.org/t/cant-manage-folders-using-bitrix-webdav-under-windows/23073

Author: Nick Craig-Wood
Author Date: 2021-03-25 17:24:46 UTC

build: add version to android builds and fix upload

Author: Nick Craig-Wood
Author Date: 2021-03-15 16:55:56 UTC

build: make arm .deb packages be armel architecture

This is as recommended by debian.

Fixes #5107

Author: Lucas Messenger
Author Date: 2021-03-11 16:50:59 UTC

hdfs: fix permissions for when directory is created

Author: buengese
Author Date: 2021-03-11 15:04:29 UTC

zoho: replace client id

Author: Nick Craig-Wood
Author Date: 2019-05-29 09:24:49 UTC

s3: when copying from s3 to s3, preserve metadata #111

See: https://forum.rclone.org/t/s3-bucket-migration-with-metadata-issues/10262

Author: Nick Craig-Wood
Author Date: 2021-02-07 12:57:39 UTC

cmount: support --mount-tpslimit flag

See: https://forum.rclone.org/t/mac-os-big-sur-mount-error/21726

Author: Nick Craig-Wood
Author Date: 2021-02-05 19:09:46 UTC

cache: set timeout to 24 hour - FIXME DO NOT MERGE

Author: Nick Craig-Wood
Author Date: 2019-06-14 14:39:33 UTC

webdav: don't delete files on error - fixes #3263

Previous to this change rclone deleted a file if the upload failed
half way through.

After this change we only delete the upload if the
--webdav-delete-on-error flag is set.

We set this in the serve webdav tests to make them pass.

Author: Nick Craig-Wood
Author Date: 2021-01-12 17:09:44 UTC

jottacloud: fix token refresh failed: is not a regular file error

Before this change the jottacloud token renewer would run and give the
error:

    Token refresh failed: is not a regular file

This is because the refresh runs on the root and it isn't a file.

This was fixed by ignoring that specific error.

See: https://forum.rclone.org/t/jottacloud-crypt-3-gb-copy-runs-for-a-week-without-completing/21173

Author: Cnly
Author Date: 2021-01-05 18:57:51 UTC

onedrive: fix server-side copy completely disabled on OneDrive for Business

This fixes a little problem in PR #4903, which is a fix for #4342

Author: buengese
Author Date: 2020-09-28 21:43:47 UTC

press: new name format

Author: Nick Craig-Wood
Author Date: 2020-10-27 17:37:31 UTC

s3: stop the S3 library doing path cleaning to use keys called "." #4704

Author: Nick Craig-Wood
Author Date: 2020-06-11 11:50:33 UTC

s3: fix v2 auth for multipart server side copy

Before this change the v2 signer sorted the headers for signing as
joined key:value pairs. However this put these two headers in the
wrong order.

    x-amz-copy-source-range:
    x-amz-copy-source:

This changes sorts on the keys before joining the values producing the
correct sort order.

    x-amz-copy-source:
    x-amz-copy-source-range:

This commit also adds some missing query parameters for signing that I
spotted in the s3cmd source.

Author: albertony
Author Date: 2020-10-20 10:43:28 UTC

jottacloud: avoid double url escaping of device/mountpoint - fixes #4697

Author: Stephen Harris
Author Date: 2020-09-24 18:51:35 UTC

sftp: allow cert based auth via optional pubkey

Author: Nick Craig-Wood
Author Date: 2020-09-13 10:07:44 UTC

ftp: Update ftp library to support PRET for drftpd

See: https://forum.rclone.org/t/ftp-client-error/18983

Author: Nick Craig-Wood
Author Date: 2019-08-13 16:48:39 UTC

local: when overwriting a symlink with a file delete the link first

If the destination file was a symlink, rclone would update the pointed
to file instead of replacing the symlink with a file.

rsync replaces files with symlinks and symlinks with files silently so
rclone should do the same.

Fixes #3400

Author: Nick Craig-Wood
Author Date: 2020-09-08 14:00:45 UTC

dropbox: make sure we wait for previous batch to finish before starting a new one

Author: Anagh Kumar Baranwal
Author Date: 2020-09-03 10:48:56 UTC

Fix docs formatting

Author: Nick Craig-Wood
Author Date: 2020-09-01 12:49:56 UTC

cmount: make work under OpenBSD - fixes #1727

Author: Nick Craig-Wood
Author Date: 2020-08-07 15:26:55 UTC

Version v1.52.3

Author: Nick Craig-Wood
Author Date: 2020-07-22 12:10:16 UTC

vfs: check destination on rename and delete if it exists first FIXME needs tests

Some backends (eg onedrive) will throw an error if we rename over an
existing file and some (drive) will create a duplicate.

This patch checks to see if the destination exists and removes it or
returns an error as appropriate.

Fixes #4293

Author: Nick Craig-Wood
Author Date: 2020-07-22 09:18:52 UTC

vfs: fix saving from chrome without --vfs-cache-mode writes

Due to Chrome's rather complicated use of file handles when saving
files from the download windows, rclone was attempting to truncate a
closed file.

The file appeared closed due to the handling of 0 length files.

This patch removes the check for the file being closed in the
WriteFileHandle.Truncate call. This is safe because the only action
this method takes is to emit an error message if the file is the wrong
size.

See: https://forum.rclone.org/t/google-drive-cannot-save-files-directly-from-browser-to-gdrive-mounted-path/17992/

Author: Nick Craig-Wood
Author Date: 2020-07-15 13:54:09 UTC

vfs: remove virtual directory entries on backends which can have empty dirs

Before this change we only removed virtual directory entries when they
appeared in the listing.

This works fine except for when virtual directory entries are deleted
outside rclone.

This change deletes directory virtual directory entries for backends
which can have empty directories before reading the directory.

See: https://forum.rclone.org/t/google-drive-rclone-rc-operations-mkdir-fails-on-repeats/17787

Author: Nick Craig-Wood
Author Date: 2020-07-10 16:11:30 UTC

lib/terminal: DEBUG which kind of terminal we are using FIXME DO NOT MERGE

Author: Nick Craig-Wood
Author Date: 2020-06-28 17:05:03 UTC

fs: Debug the names of env vars we read values from as part of the config

See: https://forum.rclone.org/t/how-can-i-check-if-environment-variables-are-being-used/17490

Author: Nick Craig-Wood
Author Date: 2018-10-16 20:09:51 UTC

gcs: Allow compressed files to be downloaded - fixes #2658

Before this change, the go runtime would automatically decompress
compressed objects leading to length mismatches.

After this change rclone will download the compressed object which
will match in length and checksum.

Author: Nick Craig-Wood
Author Date: 2020-06-11 14:29:19 UTC

s3: Make a new "unset" state for acl to work around bug in b2 s3 interface

B2 has started rejecting uploads from rclone with the error message

    Backblaze does not support the 'x-amz-acl' header

This introduces a new "unset" value. If `acl` or `bucket_acl` is set
to `unset` then rclone will omit the ACL header from the request.

See: https://forum.rclone.org/t/s3-acl-issues-on-backblaze-with-multi-part-upload/16850

Author: Nick Craig-Wood
Author Date: 2020-05-15 08:46:42 UTC

vendor patch: allow 2048 bit DSA keys DO NOT MERGE

Author: Nick Craig-Wood
Author Date: 2020-04-15 17:13:11 UTC

cmd: make rclone work as rclone.mount #992 - UNFINISHED

Author: Nick Craig-Wood
Author Date: 2020-03-10 15:59:42 UTC

union: fix slash behaviour on Windows

Author: buengese
Author Date: 2020-03-03 16:32:28 UTC

press: add xz compression algorithm

Author: Nick Craig-Wood
Author Date: 2020-02-28 14:44:15 UTC

vfs: factor the vfs cache into its own package

Author: Anagh Kumar Baranwal
Author Date: 2020-02-10 04:57:22 UTC

Added support for Max Pages

Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>

Author: Nick Craig-Wood
Author Date: 2020-01-11 11:57:42 UTC

mega: fix key decoding - FIXME VENDOR PATCH DO NOT MERGE

See: https://forum.rclone.org/t/problem-to-login-with-mega/12276

Author: Nick Craig-Wood
Author Date: 2019-12-29 21:08:08 UTC

vendor: update github.com/aws/aws-sdk-go to v1.26.8 from v1.25.31

To fix s3 memory usage problems

See: https://forum.rclone.org/t/memory-usage-s3-alike-to-glacier-without-big-directories/13563
See: https://forum.rclone.org/t/copy-from-local-to-s3-has-high-memory-usage/13405

Author: Cnly
Author Date: 2019-12-20 06:37:21 UTC

accounting: fix error count shown as checks

Author: Nick Craig-Wood
Author Date: 2019-11-13 11:42:37 UTC

vendor: patch fuse to try to fix #3697 - FIXME DO NOT MERGE VENDOR PATCH

Author: Nick Craig-Wood
Author Date: 2019-11-17 16:43:53 UTC

mega: skip undecodable FSNodes - #3740 - FIXME vendor patch DO NOT MERGE

Also test whether we have a decryption key for shared nodes

Author: Nick Craig-Wood
Author Date: 2019-11-19 16:03:01 UTC

Version v1.50.2

Author: Nick Craig-Wood
Author Date: 2019-10-14 10:24:03 UTC

vendor: Update github.com/t3rm1n4l/go-mega to fix base64 decoding issue

See: https://forum.rclone.org/t/problem-to-login-with-mega/12276

Author: Nick Craig-Wood
Author Date: 2019-10-05 11:07:20 UTC

Version v1.49.5

Author: Nick Craig-Wood
Author Date: 2019-09-25 09:12:59 UTC

config: experiment with a text UI for config

Author: Nick Craig-Wood
Author Date: 2019-09-12 15:30:25 UTC

build: fix circleci build to use billziss/xgo-cgofuse container

Use built rclone to do upload

Author: Nick Craig-Wood
Author Date: 2019-08-06 10:40:26 UTC

build: unify uploading in CI and add artifacts to Azure Pipelines

- remove compile_all step as we compile everything in the release step
- rename travis_beta to beta
- split into upload_beta and make depend on rclone
- use latest OSes in pipelines build
- Add workaround for cmd/mount tests lockup from #3154

Author: Nick Craig-Wood
Author Date: 2019-08-05 11:13:24 UTC

add build caching

Author: Nick Craig-Wood
Author Date: 2019-08-04 16:03:27 UTC

build: bring pipelines in to line with travis

Author: Nick Craig-Wood
Author Date: 2019-06-06 16:04:51 UTC

drive: add --drive-server-side-across-configs

In #2728 and 55b9a4e we decided to allow server side operations
between google drives with different configurations.

This works in some cases (eg between teamdrives) but does not work in
the general case, and this caused breakage in quite a number of
people's workflows.

This change makes the feature conditional on the
--drive-server-side-across-configs flag which defaults to off.

See: https://forum.rclone.org/t/gdrive-to-gdrive-error-404-file-not-found/9621/10

Fixes #3119

Author: Nick Craig-Wood
Author Date: 2019-05-01 16:37:16 UTC

mount test: die after 60 seconds with traceback to debug #3154

Author: Cnly
Author Date: 2019-05-23 13:10:17 UTC

fstests: add FsRootCollapse test - #3164

Author: buengese
Author Date: 2019-05-17 15:39:11 UTC

jottacloud: update docs regarding devices and mountpoints

Author: Nick Craig-Wood
Author Date: 2018-12-28 18:07:09 UTC

mount,cmount: run Release asynchronously

Before this change the mount and cmount would run Release
synchronously. This would mean that it would wait for files to be
closed (eg uploaded) before returning to the kernel.

However Release is already running asynchronously from userspace so
this commit changes it to do the functionality of Release
asynchronously too.

This should fix libfuse blocking when Release is active and it is
asked to do something else with a file.

Forum: https://forum.rclone.org/t/vfs-cache-mode-writes-upload-expected-behaviour/8014

Author: Nick Craig-Wood
Author Date: 2019-04-16 11:08:01 UTC

drive: add experimental --drive-fast-list-grouping flag #3114

Author: Nick Craig-Wood
Author Date: 2019-03-25 11:21:26 UTC

mount: make directories show with non zero size

See: https://forum.rclone.org/t/empty-folder-when-rclone-mount-used-as-external-storage-of-nextcloud/9251

Author: Cnly
Author Date: 2019-03-11 12:54:14 UTC

onedrive: Implement graceful cancel

Author: Nick Craig-Wood
Author Date: 2019-03-18 22:26:10 UTC

mega VENDOR PATCH FIXME FIXME

Fix crash caused by compkey too short

Fixes #3055

Author: Nick Craig-Wood
Author Date: 2019-03-10 17:05:15 UTC

mega: fix unmarshal error on download - FIXME need to upstream this fix

    Failed to copy: failed to open source object: open download file
    failed: json: cannot unmarshal number -18 into Go struct field
    DownloadResp.e of type uint32

See: https://forum.rclone.org/t/failed-message-what-does-it-mean/9065

Author: Nick Craig-Wood
Author Date: 2019-03-05 14:37:46 UTC

drive: set the IncludeCorpusRemovals flag so changes returns deletes

Without this flag changes does not include files which were hard
deleted.

Fixes #3020

Author: Nick Craig-Wood
Author Date: 2019-02-28 11:09:05 UTC

Revert "drive: supply correct scopes to when using --drive-impersonate"

This reverts commit b8280521a5cc4bce22442b3b17f67ec9980eb3c9.

Author: =?utf-8?q?Fabian_M=C3=B6ller?=
Author Date: 2019-02-15 10:59:11 UTC

adb: wait for correct size after upload

Author: Nick Craig-Wood
Author Date: 2018-05-07 16:16:54 UTC

crypt: add --crypt-pass-corrupted-blocks flag

Author: Nick Craig-Wood
Author Date: 2019-01-21 17:08:27 UTC

local: put debug statements in on Read FIXME WIP

Author: Nick Craig-Wood
Author Date: 2019-01-08 17:22:23 UTC

fs: Remove the charset from the Mimetype -FIXME EXPERIMENT

Author: Cnly
Author Date: 2018-12-26 08:27:58 UTC

onedrive: fix driveID of src being moved not determined correctly

Author: Nick Craig-Wood
Author Date: 2018-11-26 08:32:58 UTC

azureblob: enable on freebsd, netbsd, openbsd

At some point the SDK was fixed on these architectures, so re-enable
building the azure blob backend for them.

Author: Nick Craig-Wood
Author Date: 2018-11-04 20:44:47 UTC

rc: update docs with new methods

Author: Nick Craig-Wood
Author Date: 2018-10-25 16:23:16 UTC

onedrive: add --onedrive-list-chunk to control listing size

On very large directories this may help with timeouts.

See: https://forum.rclone.org/t/error-reading-destination-directory/7316

Author: Nick Craig-Wood
Author Date: 2018-10-18 12:18:24 UTC

webdav: add Content-Type to PUT requests - fixes #2664

Author: Henning Surmeier
Author Date: 2018-09-17 20:49:02 UTC

webdav/sharepoint: renew cookies after 12hrs

Author: Nick Craig-Wood
Author Date: 2018-09-07 15:10:18 UTC

Version v1.43.1

Author: Cnly
Author Date: 2018-08-21 02:55:35 UTC

onedrive: graph: Remove unnecessary error checks