missing ssh on rsyncd url - rsync: Failed to exec ssh: No such file or directory (2)

oops - my mistake: the problem applies only to 0.6.09, not to 0.6.08.

I will have a look at this. I probably introduced it with the changes in 0.6.09

http://bazaar.launchpad.net/~duplicity-team/duplicity/0.6-series/revision/654

..ede/duply.net

just checked with python2.6 on a suse linux 32, seems to work.

Please run your commandline with -v9 for debugging verbosity and post the output here or on pastebin. Include the full commandline at the beginning and obfuscate private strings first.

..ede/duply.net

here comes the fallout of one such run.

the cmdline is:
/usr/bin/duplicity -v 9 --no-encryption full /tmp rsync://luser@otherbox::/module/somedir/

please note the remote user specification: duplicity loses that, and of course, things subsequently don't work.

i've also attached the strace output (filtered; only execve's) of said run: you can see that the remote user name is
never passed to rsync.

the same happens if you go for authenticated rsync: rsync://luser:lame@otherbox::/module/somedir/
results in the same dud arguments passed to rsync.

Something else is wrong here. If rsyncbackend really would misinterpret the url the failing command shown would be like

Running "rsync -e 'ssh -oBatchMode=yes' luser@otherbox:module/somedir/" failed

For some reason the rsync wants to use ssh as remote shell. Can it be that the env var RSYNC_RSH=ssh is set in your environment?

At least that's what the error says

Running 'rsync rsync://otherbox::/module/somedir/' failed (attempt #1)
Error is:
rsync: Failed to exec ssh: No such file or directory (2)

Strangely though in you strace a ssh is found an executed.

However I can't reproduce the error until now. Even exporting RSYNC_RSH and excuting rsync on commandline does not sport similar issues. You should try yourself, does

rsync rsync://otherbox::/module/somedir/

also spit out the 'rsync: Failed to exec ssh' error?
Can you connect to you rsyncd with authentification? What commandline do you use for that?

--

Regarding your missing credentials. They are set in the backend as env vars USER, RSYNC_PASSWORD if rsync does not use ssh for security reasons. Rsync over ssh currently only works with keyauth and ssh is too stupid to read the USER var hence it is located in the url.
I cannot see an error that mentions a failed login. Did you occur one? What was the error message?

..so far
ede/duply.net

On Fri, 12 Nov 2010 13:08:40 GMT, edso writes:
>For some reason the rsync wants to use ssh as remote shell. Can it be
>that the env var RSYNC_RSH=ssh is set in your environment?

it's set. with python 2.5 the same version of duplicity
has worked fine for years, with the same environment vars, and
using rsh-module-urls (and /definitely/ does not use ssh as transport).

regards
az

--
+ Alexander Zangerl + DSA 42BD645D + (RSA 5B586291)
Come to think of it, there are already a million monkeys on a million
typewriters, and Usenet is NOTHING like Shakespeare.

On 13.11.2010 15:30, az wrote:
> On Fri, 12 Nov 2010 13:08:40 GMT, edso writes:
>> For some reason the rsync wants to use ssh as remote shell. Can it be
>> that the env var RSYNC_RSH=ssh is set in your environment?
>
> it's set. with python 2.5 the same version of duplicity
> has worked fine for years, with the same environment vars, and
> using rsh-module-urls (and /definitely/ does not use ssh as transport).
>

Did you try whether unsetting it solves the issue?
Did you anything I asked for in the last post? What were the results?

My rsync for example ignores RSYNC_RSH if an url with rsync:// prefix or module notation (::) is given.

ede/duply.net

i could reproduce the behaviour with rsync 3.0.7 ..

check your machines for different rsync versions. For me rsync 3.0.4 and 3.0.7 behave differently with an existing RSYNC_RSH entry.

3.0.4 ignores on rsync://url
3.0.7 tries to use it on the same url

i committed a fix to launchpad.

for now you can:

a) unset the variable
or
b) patch rsyncbackend.py as don here
http://bazaar.launchpad.net/~ed.so/duplicity/RSYNC_RSH-fix/revision/675/duplicity/backends/rsyncbackend.py

regards ede/duply.net

fix on lp,
http://bazaar.launchpad.net/~ed.so/duplicity/RSYNC_RSH-fix/revision/675/duplicity/backends/rsyncbackend.py
awaiting merge

On Sat, 13 Nov 2010 16:41:32 GMT, edso writes:
>check your machines for different rsync versions. For me rsync 3.0.4 and
>3.0.7 behave differently with an existing RSYNC_RSH entry.

thanks for the hunting down of this issue.

as far as i can see from the rsync docs, the src/dest args that
duplicity feeds it was never expected to work: for rsync-daemon access
it's either ':: after hostname but no rsync://' or
'rsync:// but then a / after hostname'. i assume that they've tightened
their uri handling between 3.0.4 and .7.

>3.0.4 ignores on rsync://url
>3.0.7 tries to use it on the same url

i've just experimented a bit further, and rsync-daemon with 3.0.7
works fine (regardless of RSYNC_RSH) iff you run it like
 rsync otheruser@othermachine::modulename/furtherdirs/
or
 rsync rsync://otheruser@othermachine/modulename/furtherdirs/

>a) unset the variable

this doesn't work: duplicity still fails with rsync 3.0.7 when an rsync-daemon
url is used.

----
$ usr/bin/duplicity -v 9 --no-encryption full /tmp rsync://luser@otherbox::/module/somedir/
Using archive dir: /root/.cache/duplicity/daec6b842453957f4d631c706d7fa883
Using backup name: daec6b842453957f4d631c706d7fa883
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.giobackend Failed: No module named gio
Import of duplicity.backends.ftpbackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.cloudfilesbackend Succeeded
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.sshbackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Main action: full
================================================================================
duplicity 0.6.09 (July 25, 2010)
Args: usr/bin/duplicity -v 9 --no-encryption full /tmp rsync://luser@otherbox::/module/somedir/
Linux CFT 2.6.31.9 #1 SMP Thu Mar 11 15:50:06 EST 2010 i686
/usr/bin/python 2.6.6 (r266:84292, Aug 29 2010, 19:11:12)
[GCC 4.4.5 20100824 (prerelease)]
================================================================================
Using temporary directory /tmp/duplicity-yOxGl9-tempdir
Registering (mkstemp) temporary file /tmp/duplicity-yOxGl9-tempdir/mkstemp-cMbFUG-1
Temp has 3652853760 available, backup will use approx 34078720.
Reading results of 'rsync rsync://otherbox::/module/somedir/'
Running 'rsync rsync://otherbox::/module/somedir/' failed (attempt #1)
----

rsync 3.0.7 doesn't accept this as a valid rsync-daemon dest, and falls
back to rsh-based access (and the debian one is compiled with ssh as default
rsh).

regards
az

--
+ Alexander Zangerl + DSA 42BD645D + (RSA 5B586291)
My computer NEVER cras

On 16.11.2010 00:40, az wrote:
> On Sat, 13 Nov 2010 16:41:32 GMT, edso writes:
>> check your machines for different rsync versions. For me rsync 3.0.4 and
>> 3.0.7 behave differently with an existing RSYNC_RSH entry.
>
> thanks for the hunting down of this issue.
>
> as far as i can see from the rsync docs, the src/dest args that
> duplicity feeds it was never expected to work: for rsync-daemon access
> it's either ':: after hostname but no rsync://' or
> 'rsync:// but then a / after hostname'. i assume that they've tightened
> their uri handling between 3.0.4 and .7.

You are right man page states
"you either use a double colon :: instead of a single colon to separate the hostname from the path, or you use an rsync:// URL."

Who would have guessed that.

Just for my backlog please, what are your rsync versions?

>
>> 3.0.4 ignores on rsync://url
>> 3.0.7 tries to use it on the same url
>
> i've just experimented a bit further, and rsync-daemon with 3.0.7
> works fine (regardless of RSYNC_RSH) iff you run it like
> rsync otheruser@othermachine::modulename/furtherdirs/
> or
> rsync rsync://otheruser@othermachine/modulename/furtherdirs/

that's obviously because rsyncbackend uses the "erroneous" syntax
rsync://otherbox::/module/somedir/

>
>> a) unset the variable
>
> this doesn't work: duplicity still fails with rsync 3.0.7 when an rsync-daemon
> url is used.
>

ould you please try to replace the line

self.url_string = "rsync://%s::/%s" % (host, path.lstrip('/:'))

in rsyncbackend.py with

self.url_string = "%s::%s" % (host, path.lstrip(':'))

and check if that works?

thanks ede/duply.net

On Tue, 16 Nov 2010 10:10:36 GMT, edso writes:
>Just for my backlog please, what are your rsync versions?

debian stable has 3.0.2 and testing/squeeze has 3.0.7

>ould you please try to replace the line
>
>self.url_string = "rsync://%s::/%s" % (host, path.lstrip('/:'))
>
>in rsyncbackend.py with
>
>self.url_string = "%s::%s" % (host, path.lstrip(':'))
>
>and check if that works?

that works perfectly, thanks.

one other item re the rsync backend: is there a good reason for
feeding the user part of an rsync uri via the env for rsyncd-module uris?

rsync can deal fine with the username on the commandline for both ssh and rsyncd-modules,
so the code to handle usernames separately seems unnecessary.

regards
az

--
+ Alexander Zangerl + DSA 42BD645D + (RSA 5B586291)
"So instead, I toil through fields of various OSes, planting applications,
spraying patches to catch the bugs, and watching the databases grow."
 -- D. Joseph Creighton

On Sat, 20 Nov 2010 15:08:26 +1000, Alexander Zangerl writes:
>>self.url_string = "rsync://%s::/%s" % (host, path.lstrip('/:'))
>>
>>in rsyncbackend.py with
>>
>>self.url_string = "%s::%s" % (host, path.lstrip(':'))
>>
>>and check if that works?
>
>that works perfectly, thanks.

with one minor change that i forgot to mention: lstrip('/:') is required,
because rsync with rsyncd requires that the module name starts without a /.

regards
az

--
+ Alexander Zangerl + DSA 42BD645D + (RSA 5B586291)
"There are only two industries that refer to their customers
as 'users'." -- Edward Tufte

On 20.11.2010 06:08, az wrote:
> one other item re the rsync backend: is there a good reason for
> feeding the user part of an rsync uri via the env for rsyncd-module uris?
>
> rsync can deal fine with the username on the commandline for both ssh and rsyncd-modules,
> so the code to handle usernames separately seems unnecessary.

These strings show up in os process listings e.g. ps, which poses a security risk as other users might catch it. Hence we prefer to deliver them as env var if possible.

..ede/duply.net

fix is committed to

lp:~ed.so/duplicity/RSYNC_RSH-fix

thanks alexander

..ede/duply.net