OAUTH server ignores ignores first element in header (rather than realm key)

How did you send the request?

Diogo,

I am experimenting with sending the requests in Java, using the oauth library at [1]. Getting the request token and delegating it works fine, the token is the one I got from step 3 at [2].

As said, in this example:

Authorization: OAuth realm="https://api.launchpad.net/",
                oauth_consumer_key="just+testing",
                oauth_token="PsK9cpbll1KwehhRDckr",
                oauth_signature_method="PLAINTEXT",
[...]

It doesn't matter what is sent as first value (realm= in this case), it seems to be ignored. Just re-arranging the key-value pairs shows the bug.

[1] http://code.google.com/p/oauth/
[2] https://help.launchpad.net/API/SigningRequests

i can update my pc

realm is required by the OAuth spec. We ignore it because checking for the token presence in our database is enough.

Actually, the spec seems to imply that realm is OPTIONAL, so it's a bug in the underlying oauth library.

The problem is not particular the "realm" element, the problem is that the FIRST element is ignored. Whatever I pass as first argument (even if it is oauth_consumer_key ) is ignored. So the workaround is to add any dummy element.

Hmmm. Max, can you confirm this with a different library, just to verify it's really a problem on our end?

Finally managed to connect to SSL via command line:

Please note that in the first example the consumer is "None", wheres when a dummy key is added, the consumer is successfully recognized.

Max

openssl s_client -connect api.launchpad.net:443
[...]
GET /beta/ HTTP/1.1
Host: api.launchpad.net
Authorization: OAuth oauth_consumer_key="just+testing"

HTTP/1.1 401 Unauthorized
Date: Tue, 27 Jan 2009 09:49:08 GMT
Server: zope.server.http (HTTP)
X-Powered-By: Zope (www.zope.org), Python (www.python.org)
X-Lazr-Oopsid: OOPS-1123H722
Content-Type: text/plain
Content-Length: 24
Via: 1.1 wildcard.launchpad.net

Unknown consumer (None).
DONE

---

openssl s_client -connect api.launchpad.net:443
[...]
GET /beta/ HTTP/1.1
Host: api.launchpad.net
Authorization: OAuth dummy="bla", oauth_consumer_key="just+testing"

HTTP/1.1 401 Unauthorized
Date: Tue, 27 Jan 2009 09:50:25 GMT
Server: zope.server.http (HTTP)
X-Powered-By: Zope (www.zope.org), Python (www.python.org)
X-Lazr-Oopsid: OOPS-1123C781
Content-Type: text/plain
Content-Length: 32
Via: 1.1 wildcard.launchpad.net

Unknown consumer (just+testing).
DONE

Sure looks like a bug to me. Francis?

This is devastating for non-Ubuntu developers. launchpadlib is only available for Ubuntu, since the documented build procedure for launchpadlib does not work, and there is no help from the launchpadlib team. So us non-Ubuntu folk are left with the "API/Hacking" method, even if we want to work with Python.

The problem reported by Max, above, makes LaunchPad integration impossible for non-launchpadlib developers like myself. All authentication attempts fail with the fallacious message about "Unknown consumer". If I ask for JSON back from my request token request, it clearly shows that it is indeed using my supplied consumer key . Is it not clear that this bug should be high priority?

As this ticket has sat for a year, I guess I'll dig into the launchpadlib code and see what it is in there that causes it to work despite the procedure in the API/Hacking page not working (as proven by Max with openssl and myself with Java).

Blaine, can you please make sure there is a bug report in <https://bugs.launchpad.net/launchpadlib> for the problem with building launchpadlib?

This exception seems to be raised from lib/canonical/launchpad/webapp/servers.py.

After bug this was filed, Launchpad started accepting unauthenticated readonly requests and it seems to me that in that case, you don't need to provide oauth data at all, only a user-agent. This, at least, does seem to work for me (tested with "curl --verbose https://api.launchpad.net/beta/").

It does look like what Max describes in comment #8 is still true, that if oauth_consumer_key is the first value in the list, it's just not seen by the server. The difference is that this now makes the request be treated as unauthenticated rather than failing.

There is some pretty dodgy code in contrib/oauth.py that seems to be assuming the 'realm' will always be first:

    # util function: turn Authorization: header into parameters, has to do some unescaping
    @staticmethod
    def _split_header(header):
        params = {}
        parts = header.split(',')
        for param in parts:
            # ignore realm parameter
            if param.find('OAuth realm') > -1:
                continue
            # remove whitespace
            param = param.strip()
            # split key-value
            param_parts = param.split('=', 1)
            # remove quotes and unescape the value
            params[param_parts[0]] = urllib.unquote(param_parts[1].strip('\"'))
        return params

This is clearly a bug, and the impact on 3rd party API client writers clearly significant.

I'd look for comments about 'realm' in the launchpad source tree to find where to start poking at this. Its probably pretty shallow to fix.

@Blaine have you filed a bug about launchpadlib not building? I know folk using it on windows and other places, so its really not 'Just Ubuntu' folk that can use it.

that explains it: if the realm parameter is not first, we'll get a dictionary key of "OAuth oauth_consumer_key" which then won't be observed later.

So as a client-side workaround:
 * pass a realm first
 * or just do anonymous readonly requests

and the fix should be to make _split_header more correct; the only real question is where to test that.

I think I can trivially fix this.

didn't mean to actually change the importance; just due to non-transactional form submissions

Proposed fix: https://code.launchpad.net/~mbp/launchpad/314507-oauth/+merge/44188

Re #13

I did not file a bug, as I only wanted to use launchpadlib as a convenience, and used up all the time that I could justify towards that end. I like to get confirmation that my problems are not due to documentation being out of date or me missing something, before registering bugs. But I found that launchpadlib Answers is only supported in the ubuntu sub-project, not the top-level launchpadlib project; I found that nobody "answers" Answers for the ubuntu python-launchpadlib project.

I came to the conclusion that only Ubuntu is supported based on the installation instructions for launchpadlib and other factors. https://help.launchpad.net/API/launchpadlib#Installation . Under the "Installation" section for the launchpadlib product, there are only two alternatives: a: Instructions for Ubuntu with apt-get. b: Instructions for building from trunk code. As I am not running Ubuntu there was no alternative for me other than "b", which failed spectacularly, even after installing undocumented third party Python dependencies, as reported at https://answers.launchpad.net/ubuntu/+source/python-launchpadlib/+question/137470 . As I said, maybe it's just a mistake or obsolescence in the instructions, or something I goofed, but nobody "Answered" me. The downloads page for launchpadlib at https://launchpad.net/launchpadlib/+download has only *.tar.gz files. Looks to me like they are intended only for Linux and UNIX at the most. I suppose it's possible that people could get either apt-get or the tar.gz distros working on Windows with Cygwin or MinGW or similar, but there's nothing about that in the installation instructions either.

Re #14

(If the imminent fix will be applied soon, and will fix both the ignoring-first-param problem and the realm-position/rejected-consumer-key problem, then please just ignore this comment).

Re. client-side work-arounds

By "pass a realm first", do you mean pass as the first parameter, or pass in an earlier parameter, of pass in an earlier transmission?

If you mean pass as the first parameter, that can't be tried as long as LaunchPad server is discarding the first parameter.

I have tried passing the passing realm as an earlier parameter and the consumer key is still rejected:

                    "dummy", "blah",
                    "realm", "https://api.launchpad.net",
                    "oauth_consumer_key", consumerKey,
                    "oauth_token", accessToken,
                    "oauth_signature_method", "PLAINTEXT",
                    "oauth_signature", accessSignature,
                    "oauth_timestamp",
                            Long.toString(new java.util.Date().getTime()/1000),
                    "oauth_nonce", "1", // just unique for reqs. for this session
                    "oauth_version", "1.0"

If you mean I should pass the realm in an earlier transmittal, the only earlier transmittal in the handshaking would be the token request transmittal. Should I add the realm param to that or insert some new transmittal?

Fixed in stable r12127 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/12127>.

qastaging works fine with launchpadlib and manual requests which break on production. Looks good to me.

POSTNOTE Re. my own #13

What I said about "maybe... I goofed" was right! The 2nd documented Installation tactic for launchpadlib did succeed after Matin pointed out my mistake. Therefore, I see how Python non-Ubuntu UNIX/Linux developers have a way to integrate with LaunchPad. For the reasons stated though, I still don't know how Windows and non-Python developers could perform successful POST operations, since it seems only launchpadlib could get the realm and the consumer key to LaunchPad such that authentication would be honored.

On 22 December 2010 10:42, William Grant <email address hidden> wrote:
> qastaging works fine with launchpadlib and manual requests which break
> on production. Looks good to me.

Thanks, I also tested hand-crafted requests and it looks good to me too.

Hooray, I think this is the fastest turnaround yet for me to fix
something in lp. (I guess it doesn't count until it's live.)

--
Martin

On 22 December 2010 11:04, Blaine Simpson <email address hidden> wrote:

> For the reasons
> stated though, I still don't know how Windows and non-Python developers
> could perform successful POST operations, since it seems only
> launchpadlib could get the realm and the consumer key to LaunchPad such
> that authentication would be honored.

If launchpadlib can't be installed on Windows, please file a specific bug.

Bug 314507 should be fixed on production within a matter of days and
at that point any oauth rest client ought to work. If not, please
file a new bug for that too.

--
Martin

@Blaine, Bugs about documentation are welcome too. You question was slightly misfiled; I've moved it and answered it. Winzip and Python can open tgz files. Perhaps for non-Ubuntu operating systems it would be best if easy_install worked.

The fix should be live soon. What I meant by passing the realm first was, in a single http request, to put the 'realm' parameter at the start of the Authentication header. Your client library might not let you control that though. In that case you just have to wait.

Re. #24
Based on what you've taught me, the Windows issue is probably just insufficient documentation as opposed to missing functionality, but as I won't be running Windows for some weeks, I won't be following up. As I pointed out, the Install instructions for launchpadlib cover only apt-get on Ubuntu and UNIX shell commands, i.e. no Windows installation instructions. Reasonable (but apparently incorrect) to conclude from that that launchpadlib is not supported on Windows.

R3. #25
Regarding the question move, the relationships between some of the projects, like the semi-present "python-launchpadlib in ubuntu" baffles me, but it looks like you moved my question to an orphaned location. I intended to post the question originally to the launchpadlib project, but as I explained in #13 above, Answers is disabled for that project. Is seems that you've moved the question to https://answers.launchpad.net/launchpadlib/+question/137470 , but if you click on Answers from there, you see that that new parent project has no Answers, so I think the question is now not reachable through site navigation. Fine with me to leave it orphaned though.

Re. Windows tools, it's not that one couldn't attempt it, it is that installation instructions are glaringly missing for Windows, and supplied artifacts give the impression of being designed for UNIX/Windows, and this all gives one little confidence that it would work on Windows. Neither "$ sudo apt-get..." nor "$sudo ./setup.py..." will work on a typical Windows installation, and those are the alternatives offered.

I can put my headers in any order I want to, but whereas changing param sequences fixes the consumer key recognition problem, it doesn't fix my problem that the correctly "recognized" consumer key is incorrectly being rejected. I'll register a new bug on that if it persists after the fix here goes live. I just hope that that turn-around doesn't eat up too many days.

CORRECTION: I can put my parameters in any order... (not headers, as it's a single http header under discussion)

launchpadlib is pretty ubuntu-oriented by custom, but not by intention.

I see what you mean about
<https://answers.launchpad.net/launchpadlib>. I don't know where the
launchpadlib maintainers (I'm not one) expect you to ask for help.

> I can put my headers in any order I want to, but whereas changing param sequences fixes the consumer key recognition problem, it doesn't fix my problem that the correctly "recognized" consumer key is incorrectly being rejected. I'll register a new bug on that if it persists after
the fix here goes live.

It's probably unlikely that I accidentally fixed that, so please just
file a bug now. We can always close it later.

Just for curiousity, can I ask what client program you're planning to write?

--
Martin

This probably also affects upstream OAuth, and Ubuntu's python-oauth, in which case the patch here should fix it. Or it may already be fixed upstream.

See Francis's comment in https://code.launchpad.net/~mbp/launchpad/314507-oauth/+merge/44188

There is already an equivalent fix in Maverick's python-oauth.

follow on bug 701545.