LP API broken in oneiric with python-httplib2 0.7.0-1

Converted from https://answers.launchpad.net/ubuntu/+source/python-httplib2/+question/161415. I don't have yet oneiric installed. Fabien Tassin, who asked the question may be able to provide more information.

Confirmed in Oneiric and reproducible with the code below:

"""
#!/usr/bin/python

from launchpadlib.launchpad import Launchpad

launchpad = Launchpad.login_anonymously('this is a test','production')
u = launchpad.projects['ubuntu']
"""

fta pointed out in <https://answers.launchpad.net/ubuntu/+source/python-httplib2/+question/161415> that this is fixed by going back to 0.6.0-5

Barry, could you have a look at this, please?

Confirmed with the upstream Mercurial clone.

One odd thing about our Oneiric package. It doesn't look like a proper merge of upstream. In particular the CHANGELOG in the package does not match the CHANGELOG in the upstream clone. I haven't check to see if that might have gotten committed after the 0.7.0 release though.

This is interesting because the 0.7.0 news (in the upstream branch, not in the packaging branch) says the attached. disable_ssl_certificate_validation might be a temporary workaround, though I'm not yet sure how to plumb that through launchpadlib, if it's even possible.

0.7.0
    The two major changes in this release are SSL Certificate
    checking and App Engine support. By default the certificates
    of an HTTPS connection are checked, but that can be disabled
    via disable_ssl_certificate_validation. The second change
    is that on App Engine there is a new connection object
    that utilizes the urlfetch capabilities on App Engine, including
    setting timeouts and validating certificates.

    The following issues have been addressed:

    Fixes issue 72. Always lowercase authorization header.
    Fix issue 47. Redirects that become a GET should not have a body.
    Fixes issue 19. Set Content-location on redirected HEAD requests
    Fixes issue 139. Redirect with a GET on 302 regardless of the originating method.
    Fixes issue 138. Handle unicode in headers when writing and retrieving cache entries. Who says headers have to be ASCII!
    Add certificate validation. Work initially started by Christoph Kern.
    Set a version number. Fixes issue # 135.
    Sync to latest version of socks.py
    Add gzip to the user-agent, in case we are making a request to an app engine project: http://code.google.com/appengine/kb/general.html#compression
    Uses a custom httplib shim on App Engine to wrap urlfetch, as opposed
    Add default support for optimistic concurrency on PATCH requests
    Fixes issue 126. IPv6 under various conditions would fail.
    Fixes issue 131. Handle socket.timeout's that occur during send.
    proxy support: degrade gracefully when socket.socket is unavailable

@Jean-Baptiste: Can you try something for me please?

After you get the traceback from the sample code in comment #2, try it exactly the same way again. For me the second time works perfectly (no traceback). I wonder if something's getting cached but not accessed correctly the first time, whereas the second time you run it, the cached file exists and it succeeds.

I think the _ssl traceback is masking a an ENOENT IOError. See this Python issue for why I think that:

http://bugs.python.org/issue9706

and especially this comment:

http://bugs.python.org/issue9706#msg115182

If that's correct, then the bug is probably in launchpadlib.

I've tried your suggestion, but no matter how many times I try, I always get the traceback.
I also tried removing ~/.launchpadlib and with a freshly created profile, it doesn't make any difference.

---
Ubuntu Bug Squad volunteer triager
http://wiki.ubuntu.com/BugSquad

@Jean-Baptiste: thanks, I think that was pebkac on my part, so ignore that. I can reproduce it every time too.

Interestingly, when stepping through httplib2, after the call to _ssl_wrap_socket() I get this error:

SSLError(8, '_ssl.c:503: EOF occurred in violation of protocol')

Hmm... Possibly a problem with Launchpad?

Okay, I think lazr.restfulclient() is simply not prepared to handle certificate validation, which is the new default in httplib2. I have a workaround, which is to pass disable_ssl_certificate_validation=True to the underlying Http object. With this, the example code works. I'll upload a temporary fix to lazr.restfulclient and create a bug task on that project.

Seems like there should be a follow on bug that it's vulnerable to mitm.

Perhaps. The thing is, it wasn't doing cert validation before, and with this change it's still not. I think validation would be a new feature for lazr.restfulclient (albeit a useful one!).

I think a new bug is essential; can you propose your existing
workaround to trunk though ?

LP: 798405 is the bug.

Merge proposal for workaround submitted on trunk.

ohai launchpad, please make that LP: #798405

Thanks. I know you're not introducing a new bug, only discovering an
existing one, but it's worth having.

Unassigning myself from the lazr.restfulclient task since I cannot push to trunk.

Barrry, I added you to ~lazr-developers, you should be able to commit changes directly to lazr projects again. (You lost that once you were removed from the ~launchpad team.)

Graham has merged your branch in the mean time though.

On Jun 21, 2011, at 03:58 PM, Francis J. Lacoste wrote:

>Barrry, I added you to ~lazr-developers, you should be able to commit
>changes directly to lazr projects again. (You lost that once you were
>removed from the ~launchpad team.)
>
>Graham has merged your branch in the mean time though.

Thanks, and thanks!
-Barry

This also affects ubuntuone-couch, which uses httplib2 to talk to one.ubuntu.com. I'll work on that.

Barry, you said bug 798405 was the follow-on bug, but I don't see it. I get a 404 from LP going to that bug. Was there a typo?

This bug was fixed in the package ubuntuone-couch - 0.2.0-0ubuntu4

---------------
ubuntuone-couch (0.2.0-0ubuntu4) oneiric; urgency=low

  * debian/patches/no-ssl-validation.patch:
    - Disable SSL validation because it isn't currently working with
      one.ubuntu.com (LP: #797281)
 -- Michael Terry <email address hidden> Tue, 21 Jun 2011 13:52:46 -0400

@Michael, nope it's there, but the bug is private.

We need to get this http://code.google.com/p/httplib2/source/detail?r=adfecbabd3f9481f059aaa20e69005ff9ea20952 instead of disabling ssl validation altogether.

I created a new bug report for current SSL DNS name check here - LP:839826 - patch has migrated there.

Do not disable SSL checks in your packages due to DNS checks, this is httplib2 bug.