latest git-ubuntu tries to create a new wallet

For the time being I went to refresh from latest/stable and that still works.

That's a weird error. I don't hit this, is it reproducible by any chance? For me, when I do a git ubuntu submit, all it prompts me is the password to connect to the web interface (which wasn't the case before, that is, when I used the latest/stable one, I guess), which leads me to think if this is by any chance related?

  latest/edge: 1.0-32-gd11cf50 2021-04-13 (481) 109MB classic
100% reprodocuible

  latest/stable: 1.0 2020-09-24 (474) 118MB classic
not affected

If switching to edge made you see a password request now that still seems to be related.
Some "let us authenticate with LP" mechanism kicks in and the details might depend on the other bits of your host config. But TBH - I'd just want to have it working as before which is that it works without extra config/walltes/PWs for git-ubuntu and just using e.g. the auth/agents on the host.

I agree. Back to latest/stable makes submission easier for me as I don't have to keep re-typing passwords for connection to the web UI. So indeed something that needs fixing in latest/edge.

I agree this is really annoying.

Here are some reduced steps to reproduce if you need to test various configurations:

snap run --shell git-ubuntu
python3
import gitubuntu.source_information
gitubuntu.source_information.launchpad_login_auth()

Then you'll be prompted, or not, depending on what the keyring module does. When that function returns with a Launchpad object, auth is successful. You can just "exit"/Ctrl-D twice (once for Python; once for the snap environment shell) to get out from there when done.

git-ubuntu uses launchpadlib which uses keyring (https://pypi.org/project/keyring/) to store its API token. keyring can use a variety of backends. kwallet is one of them. There's also keyrings.alt (https://pypi.org/project/keyrings.alt/) which includes a PlaintextKeyring which just uses a file (no encryption). The git-ubuntu snap includes keyrings.alt, so can use PlaintextKeyring if that is configured.

You can configure the keyring module to just use PlaintextKeyring. Then it'll store the API key in a file, and you won't otherwise be bothered. To do this, put this in ~/.local/share/python_keyring/keyringrc.cfg:

[backend]
default-keyring=keyrings.alt.file.PlaintextKeyring

However, this will cause _all_ uses of the keyring module to store credentials in plain text. This may not be suitable for ordinary desktop users.

According to the keyring documentation, "The library will attempt to automatically choose the most suitable backend for the current environment". In Christian's case, is this because he happens to have kwallet installed? Maybe keyring is running through the backends to see what is available in some priority order? The version of keyring being used by git-ubuntu has probably changed - it is now using the version that ships with Focal (python3-keyring). Christian - does "lp-shell" do the same thing for you now, or different?

There is an option to explicitly set the keyring backend to use by API. git-ubuntu could do this. But should it be arbitrarily overriding the keyring default?

I hope the above explains what is going on. I agree it's annoying, but I'm also reluctant to have git-ubuntu behave differently from launchpadlib (and unfortunately I think we'll be stuck with Focal's default behaviour since the snap is based on Focal, in case there are any differences in other releases).

What do you think git-ubuntu should do?

> According to the keyring documentation, "The library will attempt to
> automatically choose the most suitable backend for the current
> environment". In Christian's case, is this because he happens to have
> kwallet installed?

ii kwalletmanager 4:19.12.3-0ubuntu1
...

I have no explicit use for it, maybe by recommends from some kde tools
I still use (krdc, konversation).
I removed all I could and all that is left is now

ii libkwalletbackend5-5:amd64 5.68.0-0ubuntu2

But with just that installed the behavior still is the same.

> Maybe keyring is running through the backends to see
> what is available in some priority order?

> The version of keyring being
> used by git-ubuntu has probably changed - it is now using the version
> that ships with Focal (python3-keyring). Christian - does "lp-shell" do
> the same thing for you now, or different?

Different - It doesn't ask for anything

$ lp-shell
E: ipython not available. Using normal python shell.
Connected to LP service "production" with API version "devel":
Note: LP can be accessed through the "lp" object.
>>>

It feels like normal programs can resolve to maybe an existing instance.
While the snap program misses that and tries to create a new one.

> There is an option to explicitly set the keyring backend to use by API.
...
> What do you think git-ubuntu should do?

Yeah, I don't think we should override it in the API :-/

But I feel there is something left to resolve ...
You know the report and what happens when I run it snapped, but when I
run the very same git ubuntu command from current git head but WITHOUT
the snap it works just fine.

$ /home/paelzer/work/usd-importer/usd-importer/bin/git-ubuntu submit
--reviewer canonical-server-packageset-reviewers --target-branch
ubuntu/hirsute-devel
04/15/2021 07:40:07 - ERROR:Unable to create merge proposal: HTTP
Error 400: Bad Request
...
b'There is already a branch merge proposal registered for branch
~paelzer/ubuntu/+source/open-iscsi:lp-1922976-finalrd-logout-HIRSUTE
to land on ubuntu/+source/open-iscsi:ubuntu/hirsute-devel that is
still active.'
---

I have indeed submitted this MP already, so the follow on error is fine.
But there was no popup about kwallet or any other authentication.
So for your question "what should git-ubuntu do" my answer is "work as
smoothly as if I run it without the snap"

I have straced this good call of the non-snapped git-ubuntu.
I see it reads the config you have expected (but it doesn't exist).

37311 4165256 0.000320
stat("/home/paelzer/.local/share/python_keyring/keyringrc.cfg",
0x7fff94243570) = -1 ENOENT (No such file or directory) <0.000024>

So if I haven't configured anything explicitly we might want to
understand which keyring implementation the non-snap execution will
select.
And from there derive what might be going wrong in the snapped execution.

I see it iterate over the keyring backends but there is no obvious "I
stop at keyring foo"

4165256 0.000073 openat(AT_FDCWD,
"/usr/lib/python3/dist-packages/keyring/backends/__pycache__/__init__.cpython-38.pyc",
O_RDONLY|O_CLOEXEC) = 3 <0.000020>
4165256 0.000060 openat(AT_FDCWD,
"/usr/lib/python3/dist-packages/keyring/backends",
O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3 <0.000018>
4165256 0.000072 openat(AT_FDCWD,
"/usr/lib/python3/dist-packages/keyring/backends/__pycache__/fail.cpython-38.pyc",
O_RDONLY|O_CLOEXEC) = 3 <0.000019>
4165256 0.000077 openat(AT_FDCWD,
"/usr/lib/python3/dist-packages/keyring/backends/__pycache__/kwallet.cpython-38.pyc",
O_RDONLY|O_CLOEXEC) = 3 <0.000021>
4165256 0.000074 openat(AT_FDCWD,
"/usr/lib/python3/dist-packages/keyring/backends/__pycache__/SecretService.cpython-38.pyc",
O_RDONLY|O_CLOEXEC) = 3 <0.000021>
4165256 0.000138 openat(AT_FDCWD,
"/usr/lib/python3/dist-packages/keyring/backends/__pycache__/Windows.cpython-38.pyc",
O_RDONLY|O_CLOEXEC) = 3 <0.000030>
4165256 0.000074 openat(AT_FDCWD,
"/usr/lib/python3/dist-packages/keyring/backends/__pycache__/chainer.cpython-38.pyc",
O_RDONLY|O_CLOEXEC) = 3 <0.000020>
4165256 0.000073 openat(AT_FDCWD,
"/usr/lib/python3/dist-packages/keyring/backends/__pycache__/OS_X.cpython-38.pyc",
O_RDONLY|O_CLOEXEC) = 3 <0.000020>
4165256 0.000073 openat(AT_FDCWD,
"/usr/lib/python3/dist-packages/keyring/backends/__pycache__/_OS_X_API.cpython-38.pyc",
O_RDONLY|O_CLOEXEC) = 3 <0.000020>

I inspected the different options that could be used and found that
the local execution reaches the gnome keystore.
Therein I have a password entry for service "lauchpadlib" username
"... something constricted out of my sytems ID and a string".
And the content is a B64 encoded token.

That perfectly matches what is stored in
src/launchpadlib/credentials.py by the function
KeyringCredentialStore.

If I delete that entry in the gnome keyring, then the next non-snapped
execution presents me the very same "do you want to create a kde
wallet" popup
Any other launchpadlib based python app on my system now behaves the same.

So we can assume that the snapped-git-ubuntu fails to reach the gnome keyrings.
I know we are a classic snap, but might there be something needed to
reach the dbus based gnome keyring?
(https://specifications.freedesktop.org/secret-service/latest/)

So we have actually two problems here:
1. with an LP-auth entry existing in the gnome-keyring non-snapped
git-ubuntu finds and uses it, but snapped git-ubuntu fails and
therefore falls back to #2
2. (my local setups problem) if a key isn't found it wants to create a
new kdewallet based storage...

A random thought:

> git-ubuntu uses launchpadlib which uses keyring
> (https://pypi.org/project/keyring/) to store its API token.

I remember, initially, when I wanted to use the web UI for any tasks via CLI (for instance, running ustriage -o, maybe?), it prompted me once about allowing the access and it has options like "Never allow", "Allow for 15 minutes", "Allow for 1 hour", "Always allow this user/system" (something along these lines, not these exactly, but you get the point). And once I check off "Always allow this user/system", I have never felt the need to either enter password or grant some sort of access again.

Now, can git-ubuntu do something along the same lines as well?

(NOTE: in case I am mixing up two entirely different things here, please spare & ignore me :P)

> Now, can git-ubuntu do something along the same lines as well?

This "IS" the data that is stored in those keyrings :-)
The result of this "auth until I disable" is a token and that is then stored in the keyrings

So we already "do it along the same lines" :-)

For issue #2 - I've fixed my local problem

I indeed had kwalletd (the daemon) installed along the default gnome-keyring-dameon.
Both implement a compatible API.

When python.keyring wants a key it iterates all backends - if it finds it in any it uses that data.
## ^^ here the snapped version fails ^^ ##

But if it didn't find any it will try to create a new keystore.
That this happens to prefer kdewallet is a bug that I'll file upstream.
Internally there is a priority system and gnome has 5 which is above the 4.9 of kde.

With the help of our Desktop Team I've found that as workaround one can set this.

$ cat /home/paelzer/.local/share/python_keyring/keyringrc.cfg
[backend]
default-keyring=keyring.backends.SecretService.Keyring

This is because both still use different endpoints, that might there be version dependent as IIRC they want both to implement this.
That will make sure the gnome keyring is used, and not kwallet.

Now with my local setup sorted out and via config hard-set to prefer the gnome-keyring the git-ubuntu side of this gets even more interesting -- see next bug update ...

With the config in place to hard-prefer gnome-keyring-daemon

git ubuntu throws this error:

$ git ubuntu submit --reviewer canonical-server-packageset-reviewers --target-branch ubuntu/hirsute-devel
Traceback (most recent call last):
  File "/snap/git-ubuntu/483/usr/bin/git-ubuntu", line 11, in <module>
    load_entry_point('gitubuntu==1.0', 'console_scripts', 'git-ubuntu')()
  File "/snap/git-ubuntu/483/usr/lib/python3/dist-packages/gitubuntu/__main__.py", line 269, in main
    sys.exit(args.func(args))
  File "/snap/git-ubuntu/483/usr/lib/python3/dist-packages/gitubuntu/submit.py", line 75, in cli_main
    return main(
  File "/snap/git-ubuntu/483/usr/lib/python3/dist-packages/gitubuntu/submit.py", line 178, in main
    lp = launchpad_login_auth()
  File "/snap/git-ubuntu/483/usr/lib/python3/dist-packages/gitubuntu/source_information.py", line 82, in launchpad_login_auth
    _LP_LOGIN_AUTH = LP.login_with('git-ubuntu-importer', _lp_service,
  File "/snap/git-ubuntu/483/usr/lib/python3/dist-packages/launchpadlib/launchpad.py", line 564, in login_with
    return cls._authorize_token_and_login(
  File "/snap/git-ubuntu/483/usr/lib/python3/dist-packages/launchpadlib/launchpad.py", line 370, in _authorize_token_and_login
    cached_credentials = credential_store.load(
  File "/snap/git-ubuntu/483/usr/lib/python3/dist-packages/launchpadlib/credentials.py", line 336, in load
    return self.do_load(unique_key)
  File "/snap/git-ubuntu/483/usr/lib/python3/dist-packages/launchpadlib/credentials.py", line 411, in do_load
    self._ensure_keyring_imported()
  File "/snap/git-ubuntu/483/usr/lib/python3/dist-packages/launchpadlib/credentials.py", line 379, in _ensure_keyring_imported
    import keyring
  File "/snap/git-ubuntu/483/usr/lib/python3/dist-packages/keyring/__init__.py", line 3, in <module>
    from .core import (
  File "/snap/git-ubuntu/483/usr/lib/python3/dist-packages/keyring/core.py", line 189, in <module>
    init_backend()
  File "/snap/git-ubuntu/483/usr/lib/python3/dist-packages/keyring/core.py", line 97, in init_backend
    or load_config()
  File "/snap/git-ubuntu/483/usr/lib/python3/dist-packages/keyring/core.py", line 176, in load_config
    return load_keyring(keyring_name)
  File "/snap/git-ubuntu/483/usr/lib/python3/dist-packages/keyring/core.py", line 137, in load_keyring
    class_.priority
  File "/snap/git-ubuntu/483/usr/lib/python3/dist-packages/keyring/util/properties.py", line 26, in __get__
    return self.fget.__get__(None, owner)()
  File "/snap/git-ubuntu/483/usr/lib/python3/dist-packages/keyring/backends/SecretService.py", line 30, in priority
    raise RuntimeError("SecretStorage required")
RuntimeError: SecretStorage required

That is very interesting!
Could it be that with 20.04 things were split and we just need to make sure that the snap has "python3-secretstorage"?
I don't see it in snap/snapcraft.yaml nor in setup.py

I tried to modify and test, but snapcraft fails my build of git-ubuntu :-/
Could you provide me a version with python3-secretstorage for a try?

BTW - that would also explain why it failed to read the existing entry from there in the first place and then fell back to create a kdewallet.

FYI the follow on symptoms of "if not exist trigger creation" is a known upstream case at https://github.com/jaraco/keyring/issues/496

> I tried to modify and test, but snapcraft fails my build of git-ubuntu :-/

A build is in progress via Jenkins, but FWIW, you can run snap.sh in an isolated Focal environment (don't use your own environment as it'll clobber it). It's suboptimal but an explanation of why is needed starts in comments in that script :-/

installed: 1.0-43-ge5fbc6d (491) 113MB classic

Test (as before)
$ git ubuntu submit --reviewer canonical-server-packageset-reviewers --target-branch ubuntu/groovy-devel

1. as-is (with workaround in place) => still working
2. drop the config I put into ~/.local/share/python_keyring/keyringrc.cfg => now working

I can confirm that the fix works for my scenario.
Thanks!

Hello,

No keyring problems for me either so far! \o/

Fix released in git-ubuntu version 1.1