Repeated ssh connections from tbzrcache on paramiko auth failure

Bug #922395 reported by Martin Packman
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Bazaar Windows Installers
Fix Released
Medium
Martin Packman
TortoiseBZR
Fix Released
Medium
IWATA Hidetaka

Bug Description

The tbzrcache process can get into a state where it makes a huge number of ssh connections in succession, which is likely to upset remote servers. The exact cause isn't clear, but a particular user machine that caused issues with the launchpad codehosting has this in its log:

[ 3472] 2012-01-25 11:44:03.417 ERROR: Another instance of this server is running - please close that instance and try again
[ 4076] 2012-01-25 11:44:03.967 INFO: tbzrcache running...
6.545 falling back to default implementation
6.545 failed to load system host keys: [Errno 2] No such file or directory: 'C:\\Users\\username/.ssh/known_hosts'
[ 4076] 2012-01-25 11:44:08.961 INFO: Connected (version 2.0, client Twisted)
[ 4076] 2012-01-25 11:44:09.230 ERROR: Error processing remote request
Traceback (most recent call last):
  File "tbzrlib\pipe\server.pyo", line 134, in run_command_loop
  File "tbzrlib\dispatcher.pyo", line 259, in dispatch
  File "tbzrlib\dispatcher.pyo", line 62, in get_file_status
  File "tbzrlib\wtcache.pyo", line 829, in get_status_for_path
  File "tbzrlib\wtcache.pyo", line 760, in _create_cache_for_path
  File "bzrlib\workingtree.pyo", line 309, in open_containing
  File "bzrlib\bzrdir.pyo", line 1318, in open_workingtree
  File "bzrlib\workingtree_4.pyo", line 1590, in open
  File "bzrlib\workingtree_4.pyo", line 1600, in _open
  File "bzrlib\bzrdir.pyo", line 1302, in open_branch
  File "bzrlib\branch.pyo", line 2304, in open
  File "bzrlib\bzrdir.pyo", line 828, in open
  File "bzrlib\bzrdir.pyo", line 863, in open_from_transport
  File "bzrlib\bzrdir.pyo", line 1590, in open
  File "bzrlib\remote.pyo", line 293, in _open
  File "bzrlib\remote.pyo", line 356, in __init__
  File "bzrlib\remote.pyo", line 368, in _probe_bzrdir
  File "bzrlib\remote.pyo", line 375, in _rpc_open_2_1
  File "bzrlib\remote.pyo", line 56, in _call
  File "bzrlib\smart\client.pyo", line 132, in call
  File "bzrlib\smart\client.pyo", line 145, in call_expecting_body
  File "bzrlib\smart\client.pyo", line 90, in _call_and_read_response
  File "bzrlib\smart\client.pyo", line 63, in _send_request
  File "bzrlib\smart\protocol.pyo", line 1309, in call
  File "bzrlib\smart\protocol.pyo", line 1133, in _write_end
  File "bzrlib\smart\protocol.pyo", line 1099, in flush
  File "bzrlib\smart\medium.pyo", line 395, in accept_bytes
  File "bzrlib\smart\medium.pyo", line 977, in _accept_bytes
  File "bzrlib\smart\medium.pyo", line 793, in _accept_bytes
  File "bzrlib\smart\medium.pyo", line 817, in _ensure_connection
  File "bzrlib\transport\ssh.pyo", line 332, in connect_ssh
  File "bzrlib\transport\ssh.pyo", line 320, in _connect
  File "bzrlib\transport\ssh.pyo", line 544, in _paramiko_auth
ConnectionError: Connection error: Unable to authenticate to SSH host as
  <email address hidden>
supported auth types: ['publickey']
7.247 failed to load system host keys: [Errno 2] No such file or directory: 'C:\\Users\\username/.ssh/known_hosts'
[ 4076] 2012-01-25 11:44:09.638 INFO: Connected (version 2.0, client Twisted)
[ 4076] 2012-01-25 11:44:09.900 ERROR: Error processing remote request
...

And it continues with a new connection at less than one second intervals until launchpad starts just refusing the connection.

The bzr version is 2.4.2 from the all-in-one windows installer. There are some earlier checkout operations, but no obvious cause for this connection retry loop.

Revision history for this message
methane (songofacandy) wrote :

Do you use lightweight checkout from bzr+ssh or branch stacking to bzr+ssh?
It may cause this problem.
Bazaar Explorer's "Checkout" dialog implicitly use lightweight checkout.

Next TortoiseBZR avoid showing status for lightweight checkout.

Revision history for this message
Martin Packman (gz) wrote :

Checkouts do seem to be involved from the log, but I suspect there must be something else as well or more people would be DDOSing launchpad codehosting.

What is the change to not show status for checkouts exactly? Is it in a version of tortoisebzr we could ship with the 2.4.3 windows installer release?

Revision history for this message
IWATA Hidetaka (hid-iwata) wrote : Re: [Bug 922395] Re: Repeated ssh connections from tbzrcache on paramiko auth failure
Download full text (4.4 KiB)

Now, TortoiseBzr 0.6.6b1 is available. And I'll release stable version soon.
This version is still compatible with bzr2.3 and 2.4.
 2012/01/27 20:30 "Martin Packman" <email address hidden>:

> Checkouts do seem to be involved from the log, but I suspect there must
> be something else as well or more people would be DDOSing launchpad
> codehosting.
>
> What is the change to not show status for checkouts exactly? Is it in a
> version of tortoisebzr we could ship with the 2.4.3 windows installer
> release?
>
> --
> You received this bug notification because you are a member of
> TortoiseBZR Developers, which is subscribed to TortoiseBZR.
> https://bugs.launchpad.net/bugs/922395
>
> Title:
> Repeated ssh connections from tbzrcache on paramiko auth failure
>
> Status in TortoiseBZR:
> New
>
> Bug description:
> The tbzrcache process can get into a state where it makes a huge
> number of ssh connections in succession, which is likely to upset
> remote servers. The exact cause isn't clear, but a particular user
> machine that caused issues with the launchpad codehosting has this in
> its log:
>
> [ 3472] 2012-01-25 11:44:03.417 ERROR: Another instance of this server is
> running - please close that instance and try again
> [ 4076] 2012-01-25 11:44:03.967 INFO: tbzrcache running...
> 6.545 falling back to default implementation
> 6.545 failed to load system host keys: [Errno 2] No such file or
> directory: 'C:\\Users\\username/.ssh/known_hosts'
> [ 4076] 2012-01-25 11:44:08.961 INFO: Connected (version 2.0, client
> Twisted)
> [ 4076] 2012-01-25 11:44:09.230 ERROR: Error processing remote request
> Traceback (most recent call last):
> File "tbzrlib\pipe\server.pyo", line 134, in run_command_loop
> File "tbzrlib\dispatcher.pyo", line 259, in dispatch
> File "tbzrlib\dispatcher.pyo", line 62, in get_file_status
> File "tbzrlib\wtcache.pyo", line 829, in get_status_for_path
> File "tbzrlib\wtcache.pyo", line 760, in _create_cache_for_path
> File "bzrlib\workingtree.pyo", line 309, in open_containing
> File "bzrlib\bzrdir.pyo", line 1318, in open_workingtree
> File "bzrlib\workingtree_4.pyo", line 1590, in open
> File "bzrlib\workingtree_4.pyo", line 1600, in _open
> File "bzrlib\bzrdir.pyo", line 1302, in open_branch
> File "bzrlib\branch.pyo", line 2304, in open
> File "bzrlib\bzrdir.pyo", line 828, in open
> File "bzrlib\bzrdir.pyo", line 863, in open_from_transport
> File "bzrlib\bzrdir.pyo", line 1590, in open
> File "bzrlib\remote.pyo", line 293, in _open
> File "bzrlib\remote.pyo", line 356, in __init__
> File "bzrlib\remote.pyo", line 368, in _probe_bzrdir
> File "bzrlib\remote.pyo", line 375, in _rpc_open_2_1
> File "bzrlib\remote.pyo", line 56, in _call
> File "bzrlib\smart\client.pyo", line 132, in call
> File "bzrlib\smart\client.pyo", line 145, in call_expecting_body
> File "bzrlib\smart\client.pyo", line 90, in _call_and_read_response
> File "bzrlib\smart\client.pyo", line 63, in _send_request
> File "bzrlib\smart\protocol.pyo", line 1309, in call
> File "bzrlib\smart\protocol.pyo", line 1133, in _write_end
> File "bzrlib\smart\protocol.pyo...

Read more...

Martin Packman (gz)
Changed in bzr-windows-installers:
assignee: nobody → Martin Packman (gz)
importance: Undecided → Medium
status: New → In Progress
Martin Packman (gz)
Changed in bzr-windows-installers:
status: In Progress → Fix Released
Changed in tortoisebzr:
status: New → Fix Released
assignee: nobody → IWATA Hidetaka (hid-iwata)
importance: Undecided → Medium
Changed in tortoisebzr:
milestone: none → 0.6.6
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.