[FFE] Proxy "tunnel" for syncdaemon

This seems a very intrusive change at this point, so I have some questions:

 * Does this change the code path/structure for situations where users use no proxies? I. e. will the "no proxy" case also have a new process and use the same code path? In other words, what's the regression potential for the "no proxy" case?

 * Why do we need a separate process for this tunneling? It sounds rather heavy (memory usage, wakeups, startup time, etc.) to have yet another long-running Python process in the system for protocol translation, these can usually happen in-process.

 * Where does it get the proxy configuration from? Do you use libproxy? There are quite a few places where proxies are configured (/etc/environment, GNOME control center, Firefox etc.).

Hi Martin, thanks a lot for your input.

> * Does this change the code path/structure for situations where users use no proxies? I. e. will the "no proxy" case also have a new process and use the same code path? In other words, what's the regression potential for the "no proxy" case?

When the user has not configured a proxy then the tunnel process will not be started, so it will still use the same code path as before. (The only new code that will be run in that case is the checking for the config)

> * Why do we need a separate process for this tunneling? It sounds rather heavy (memory usage, wakeups, startup time, etc.) to have yet another long-running Python process in the system for protocol translation, these can usually happen in-process.

We have settled on QtNetwork to provide consistent proxy support for every Ubuntu One process, since we ran a small survey[1] with our users, and we found out that they wanted to use a number of proxy protocols and proxy authentication schemes, and in our research this was the library that supported the best variety.

The Ubuntu One syncdaemon uses a twisted reactor as the mainloop, and to integrate with QtNetwork it needs a specialized reactor that can be integrated with QtCore. ATM, the only twisted reactor that integrates with the Qt mainloop is the qt4reactor which is currently available in universe. We could file a MIR to use it from ubuntuone-client, but we prefer not to do that at this point since there is no clear intent from upstream to maintain it, and the U1 team does not have (yet) the expertise nor are able to commit to maintain it.

[1] http://blog.protocultura.net/post/12660278531/u1-proxy-support

> * Where does it get the proxy configuration from? Do you use libproxy? There are quite a few places where proxies are configured (/etc/environment, GNOME control center, Firefox etc.).

The proxy configuration is taken from gsettings, so the way to set them is with the GNOME control center.

Hello Alejandro,

Alejandro J. Cura [2012-02-28 17:29 -0000]:
> When the user has not configured a proxy then the tunnel process will
> not be started, so it will still use the same code path as before. (The
> only new code that will be run in that case is the checking for the
> config)

That seems fine then, thanks.

> We have settled on QtNetwork to provide consistent proxy support for
> every Ubuntu One process

Is the proxy tunnel helper written in C++ or in Python? In the later
case, it's again blocked by not having PyQt in the default install. If
it's in C++, it's fine.

> The proxy configuration is taken from gsettings, so the way to set them
> is with the GNOME control center.

OK. Using libproxy might be a bit more generic, but that should do for
now.

Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

On Wed, 2012-02-29 at 06:36 +0000, Martin Pitt wrote:
> Is the proxy tunnel helper written in C++ or in Python? In the later
> case, it's again blocked by not having PyQt in the default install. If
> it's in C++, it's fine.

It's Python, but is a separate set of code (and will be a new binary
package), and not required for people who don't have proxies. We'll also
add it to the list of packages which the ubuntuone-installer installs.

Thanks for the explanations! Please go ahead.

Also affecting ubuntu-sso-client, since the common webclient code in ubuntu-sso-client needs to use the tunnel process to allow proxy support for the rest webcalls done by syncdaemon.

This should not cause regressions in ubuntu-sso-client, since the twisted webclient backend that will be modified is only currently used for unit/integration tests.

This bug was fixed in the package ubuntu-sso-client - 2.99.91-0ubuntu1

---------------
ubuntu-sso-client (2.99.91-0ubuntu1) precise; urgency=low

  * New upstream release:
    - Do not allow ssl errors to be ignored (LP: #959390).
    - Handle wrong credentials properly in qtnetwork webclient
      (LP: #957317).
    - Use HTTPClientFactory to allow replacing the reactor used to connect
      (LP: #929207).
    - Decode the content of help_text (LP: #951371).
    - Adding missing file for translation (LP: #951376).
    - Adding a general error message when the argument received by
      build_general_error_message is not a dict (LP: #865176).
    - Adding some checks to setup_page (LP: #951461).
    - Adding a padding to the right margin of the reset layout, and align
      one of its layout to the left (LP: #945065).
    - Executing hide_error when the user click the refresh captcha link,
      not inside of the _refresh_captcha method, because this is executed
      automatically when a captcha error is generated, so we will always
      miss the error message (LP: #955010).
    - Removing the align property from the label that wasn't necessary
      and was breakint the work wrap. Also adjust the height of the widget
      depending if it has more than one line (LP: #940392).
    - Improve logging operations (LP: #934500).
    - Making LINK_STYLE to be unicode (LP: #950953).
    - Setting the window title equal to the app_name (LP: #949744).
    - The _move_to_email_verification_page wasn't receiving the params
      that the signal emits (LP: #945066).
    - Improve the grammar for the CLOSE_AND_SETUP_LATER button text
      (LP: #949978).
    - Changed the way in which we deal with proxies to work around bugs
      found in the QNetworkAccessManager (LP: #957313).
    - Stopped listening to the proxyAuthenticationRequired to avoid the
      dialog showing more than once (LP: #957170)
    - Made changes in the way the webclient is selected to ensure that qt
      is used when possible (LP: #957169).
    - Connected the WebKit browser correctly so that the tc page gets
      loaded (LP: #933081).
    - Added code to check if the browser with the t&c was already loaded.
      If it is just show the t&c page, otherwise perform the request
      (LP: #956185).
    - Added a translatable string to give more context of the ssl cert
      info to the user (LP: #948119).
    - Provided the logic required for the Qt webclient implementation to
      detect ssl errors and spawn the ssl dialog to allow the user accept
      the ssl cert exceptions (LP: #948134).
    - Changed the qt webclient implementation to use a proxy factory so
      that the correct proxy is chosen according to the request
      (LP: #950088).
    - Added the required code to allow the webclient use authenticated
      proxies and request the creds when needed (LP: #933727).
    - The tooltip should not be shown for titles and subtitles when
      no cut off was needed (LP: #949741).
    - Enable platform-specific styling (LP: #953318).
    - Only import DBus on Linux (LP: #956304).
    - Don't hard-code font sizes.
    - Remove usage of weight property as a numeric; just use...

This bug was fixed in the package ubuntuone-client - 2.99.91-0ubuntu1

---------------
ubuntuone-client (2.99.91-0ubuntu1) precise; urgency=low

  * New upstream release.
    - Tunnel web service calls if proxy is enabled. (LP: #929212)
    - Proxy tunnel service for syncdaemon. (LP: #929207)
    - Tunnel storage protocol if proxy enabled in settings. (LP: #929208)
    - Use transient hint for notifications. (LP: #887369)
    - Apport recipe attaches old/useless log files. (LP: #956407)
    - Typo in man page for u1sdtool. (LP: #682954)
  * debian/watch:
    - Update watch file for new release.
  * debian/control:
    - Update ubuntu-sso-client dependencies for proxy support.
    - Create new binary package for proxy support which requires qt.
  * debian/python-ubuntuone-client.install,
    debian/ubuntuone-client-proxy.install:
    - Update for the new binary package for proxy support.
 -- Rodney Dawes <email address hidden> Wed, 21 Mar 2012 13:58:04 -0400