git: no support for gpgsig tags

Do you have an estimate of the complexity of fixing this? It's semi-blocking something that I need to work on, so if it's likely to be doable by someone with no prior bzr-git experience I'd be able to hack on it.

Basically what needs to happen is:

 * in dulwich, extend Commit to serialize and deserialize gpgsig data (in dulwich/objects.py)
 * in bzr-git:
  - have all old mappings raise an exception if a gpgsigs are encountered (mapping.py)
  - have new mapping convert gpg data into a bzr revision property and back

I think it'd probably a day or two if you're familiar with the git and bzr internals.

Hm. Given I'm familiar with neither the bzr nor the git internals, that
might be a bit longer for me then.

Don't let me block you from working on it ☺.

> Don't let me block you from working on it ☺.
FWIW I don't think bzr-git is actively maintained at the moment.

Hi Artur,

Please stop marking these bugs as fixed - they're not.

(I reverted the Launchpad bug state to New, since I can't set it to Triaged anymore)

Jelmer

Making this non-fatal (on launchpad) would be rather helpful since as of a couple of days ago this is blocking KDE Frameworks 5 building [1]. Upstream is less than thrilled.

[1] https://code.launchpad.net/~neon/kde-workspace/master

On Sun, Nov 03, 2013 at 10:07:36AM -0000, Harald Sitter wrote:
> Making this non-fatal (on launchpad) would be rather helpful since as of
> a couple of days ago this is blocking KDE Frameworks 5 building [1].
> Upstream is less than thrilled.
>
> [1] https://code.launchpad.net/~neon/kde-workspace/master
>
> ** Attachment added: "raw commit data of affected commit in kde-workspace"
> https://bugs.launchpad.net/ubuntu/+source/bzr-git/+bug/1084403/+attachment/3899038/+files/commit
Unfortunately just ignoring these tags is not possible without
fundamentally breaking various assumptions in bzr-git.

bzr-git does a lossless conversion from git to bzr, so it can later
regenerate the exact same git objects. This is necessary since objects
sent by the server can be sent as deltas against fulltexts that the
server knows the client has.

If these tags are ignored, then bzr-git will be unable to reproduce
git commit objects with the exact same contents, and break.

Jelmer

More practically speaking, I think the most achievable solution is probably to run a manual fast-export/fast-import import.

Is there a way that can work around this bug and set up auto-import for a git repository that has some historical commits contain gpgsig? It seems that I'm not able to push to the branch set up by vcs-import, so fast-export/fast-import does not work for this use case.

On Tue, Nov 12, 2013 at 06:55:17PM -0000, Aron Xu wrote:
> Is there a way that can work around this bug and set up auto-import for
> a git repository that has some historical commits contain gpgsig? It
> seems that I'm not able to push to the branch set up by vcs-import, so
> fast-export/fast-import does not work for this use case.
Unfortunately that's not possible.

You should be able to sue fast-export/fast-import with a new (non-vcs-imports)
branch.

Cheers,

Jelmer

same problem here: https://code.launchpad.net/~thopiekar/emgd/x11proto-core

See bug 1251682 for the relevant dulwich bug.

Just ran into this issue a few days ago when a project I contribute to failed to sync because of a gpg signed commit:

You can find the log from launchpad in the attachment as well as in the following pastebin:
http://paste.ubuntu.com/6788623/

This is now preventing me from importing the matplotlib codebase to do nightly builds:

https://code.launchpad.net/~takluyver/matplotlib/trunk

I'm unable to import openstack/tempest because of this

Thanks for the replies.

At this point the issue is well documented; please don't add more examples of broken repositories.

I do not want to add another example, because there are already too many.

Just FYI, Debian is discussing right now the defaulting of gpg signed commits, so I guess this bug will be more and more important from now.

Major git hosting service GitHub has just announced support for verifying signed git commits and tags [1].
Just noting, as this will most probably increase the usage of signed git commits.

[1] : https://github.com/blog/2144-gpg-signature-verification

Just found this patched version: https://code.launchpad.net/~simon-marchi/bzr-git/devel
Could this get merged please? This bug breaks SuperTuxKart daily builds, see https://github.com/supertuxkart/stk-code/issues/1415 towards end.

That fix is incomplete and possibly corrupts future pulls. See the bug report for an explanation.

On 13 June 2016 03:45:52 BST, Qwerty Chouskie <email address hidden> wrote:
>Just found this patched version:
>https://code.launchpad.net/~simon-marchi/bzr-git/devel
>Could this get merged please? This bug breaks SuperTuxKart daily
>builds, see https://github.com/supertuxkart/stk-code/issues/1415
>towards end.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

I saw comment #8, but it does not seem like ignoring gpgsig tags should break deltas, as the code changes in the commits are the same...

Also, the code change will defiantly not make anything worse, and might make things better, so it would make sense to merge it as a temporary fix... And, I tested the first section of the diff (https://i264933589.restricted.launchpadlibrarian.net/264933589/8e33a154-3111-11e6-9da8-002481e91f22.txt?token=17TdjWddN9v0cBJKTTTsGgMmsb8NjTg8) and it worked for branching SuperTuxKart (https://github.com/supertuxkart/stk-code.git).

Our direction for this in Launchpad is to get git-to-git imports working (hopefully soonish), and then you can just use git-based recipes instead. This will be much simpler and more robust against all kinds of strangenesses in git repositories; for example, a fix for gpgsig is no help for the also-common problem that bzr-git doesn't handle submodules.

It will make things worse. People will get confusing errors (sha1 checksum or delta invalid exceptions), and bzr-git loses the ability to properly support the gpgsig tag.

On 13 June 2016 22:16:19 BST, Qwerty Chouskie <email address hidden> wrote:
>Also, the code change will defiantly not make anything worse, and might
>make things better, so it would make sense to merge it as a temporary
>fix... And, I tested the first section of the diff
>(https://i264933589.restricted.launchpadlibrarian.net/264933589/8e33a154-3111-11e6-9da8-002481e91f22.txt?token=17TdjWddN9v0cBJKTTTsGgMmsb8NjTg8)
>and it worked for branching SuperTuxKart
>(https://github.com/supertuxkart/stk-code.git).

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

I tried:

=== modified file 'mapping.py'
--- mapping.py 2013-09-02 00:54:58 +0000
+++ mapping.py 2016-06-13 04:07:10 +0000
@@ -333,7 +333,11 @@
         if commit is None:
             raise AssertionError("Commit object can't be None")
         if commit.extra:
- raise UnknownCommitExtra(commit, [item[0] for item in commit.extra])
+ for (extra_key, extra_value) in commit.extra:
+ if extra_key == "gpgsig":
+ pass
+ sys.stderr.write("WARNING: GPGSIG TAGS ARE NOT SUPPOTED AND MAY CAUSE ISSUES!!!!!!!\n")
+ else:
+ raise UnknownCommitExtra(commit, [item[0] for item in commit.extra])
         rev = ForeignRevision(commit.id, self,
                 self.revision_id_foreign_to_bzr(commit.id))
         rev.git_metadata = None

But there was no output on the terminal, what am I doing wrong?

It will break deltas, as bzr-git won't be able to create the original git object anymore from the bzr data. The object it creates will lack the gpgsig tag and thus result in a checksum mismatch.

On 13 June 2016 21:29:38 BST, Qwerty Chouskie <email address hidden> wrote:
>I saw comment #8, but it does not seem like ignoring gpgsig tags should
>break deltas, as the code changes in the commits are the same...

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

I made this patch: http://jacobspctuneup.tk/bzr-git%20patches/patchtry1.txt
It looks good, but nothing actually appears in the terminal or "~/.bzr.log". Maybe it only shows up if there is an error?

Are you fetching into a new repository and have you removed the bzr-git cache?

On 15 June 2016 01:24:10 BST, Qwerty Chouskie <email address hidden> wrote:
>I made this patch:
>http://jacobspctuneup.tk/bzr-git%20patches/patchtry1.txt
>It looks good, but nothing actually appears in the terminal or
>"~/.bzr.log". Maybe it only shows up if there is an error?

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Where is the cache?

It depends on your system. Usually $XDG_CACHE_HOME/bazaar/git I think.

On 16 June 2016 17:50:44 BST, Qwerty Chouskie <email address hidden> wrote:
>Where is the cache?

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

I checked ~/.cache, but there was nothing bzr-related there. I also checked ~/.bazzar/, but there was nothing cache-related there.

The cache could also be under the bzr repository somewhere.

On 16 June 2016 19:06:43 BST, Qwerty Chouskie <email address hidden> wrote:
>I checked ~/.cache, but there was nothing bzr-related there. I also
>checked ~/.bazzar/, but there was nothing cache-related there.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

I deleted the whole repo before re-import, so that should not be the problem.

That patch won't do anything because you included the new check inside the "if commit is not None" block, which is an assertion that is meant to never fail.

Even if it was in the right place, however, it would try to raise None as an exception and crash.

If you fixed that, it would do as Jelmer says: create a corrupt branch that is unable to reproduce the commits that it says it can, because it won't store the gpgsig field that is needed for reconstitution.

This is fundamentally a very difficult dilemma, because you can't ignore the field: you either have to store it verbatim in a revision property, giving up the ability to implement it properly in future; or implement it properly, which it isn't at all clear how to do. Due to the way that git and bzr-git operate you can't just ignore it and hope it will work.

This is not a "trivial bug", which papercuts seems to be about.

This also has affected libretro-snes9x:
http://launchpadlibrarian.net/272633750/libretro-libretro-snes9x-libretro.log

bzrlib.plugins.git.errors.UnknownCommitExtra: Unknown extra fields in <Commit 03c8bad484ff5210c9a374d687d26126b6c1f71c>: ['gpgsig'].

https://github.com/libretro/snes9x/issues/22

Built ontop of wgrant's feedback, a new patch:
http://files.robloach.net/bzr-git--gpg.diff

While I understand the best way around this would be to implement the GPG signatures in bzr, but this is affecting many projects brought into Launchpad through bzr-git. Ignoring it may be the best option until we can implement it correctly in the bzr space.

The attachment "bzr-git--gpg.diff" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Ignoring it will cause confusing import errors in incremental pulls for Launchpad imports as well.

Incremental pulls rely on deltas against full texts recreated from bzr data including PGP signatures. If even one byte in the full text is incorrect, that causes incremental pulls to fail.

On 19 July 2016 02:08:31 GMT+02:00, Rob Loach <email address hidden> wrote:
>Built ontop of wgrant's feedback, a new patch:
>http://files.robloach.net/bzr-git--gpg.diff
>
>While I understand the best way around this would be to implement the
>GPG signatures in bzr, but this is affecting many projects brought into
>Launchpad through bzr-git. Ignoring it may be the best option until we
>can implement it correctly in the bzr space.

Thanks for taking a look, Jelmer. Do you have an idea of a way forward with it? William mentioned he was unsure of how to do it properly.

On Tue, Jul 19, 2016 at 05:56:28PM -0000, Rob Loach wrote:
> Thanks for taking a look, Jelmer. Do you have an idea of a way forward
> with it? William mentioned he was unsure of how to do it properly.
They are the equivalent of testaments, so ideally they should be
converted to those. I don't recall enough about testaments to know
whether that would actually work though.

Alternatively, they could just be stowed in bzr revision properties
and then reimported whenever bzr revisions are converted to git
commits.

Jelmer

Adding a note to mention that this bug makes is quite difficult to use the PPA with Github repositories. I wanted to try to build my own .deb for Jabref, so I started by trying to import the code from the Jabref github pages. Here is where I got to:
https://code.launchpad.net/~jdpipe/jabref/debian

However, when importing from Github, there is an error from Launchpad, specifically from "/srv/importd.launchpad.net/production/launchpad-rev-18169/bzrplugins/git/mapping.py" line 334, where a 'gpgsig' commit field seems to have triggered some kind of error.

A little surprised by this... surely other people are importing code from Github?

Cheers
JP

2016-08-03 02:07:49 INFO Starting job.
2016-08-03 02:07:49 INFO Getting exising bzr branch from central store.
2016-08-03 02:07:50 INFO [chan bzr SocketAsChannelAdapter] Opened sftp connection (server version 3)
2016-08-03 02:07:50 INFO [chan bzr SocketAsChannelAdapter] Opened sftp connection (server version 3)
2016-08-03 02:07:50 INFO 285 bytes transferred
2016-08-03 02:07:51 INFO Importing branch.
2016-08-03 02:07:53 INFO Counting objects: 118432, done. 0
2016-08-03 02:08:53 INFO 68240614 bytes transferred | Compressing objects 3/3
2016-08-03 02:09:05 INFO finding revisions to fetch:generating index 0/118432
2016-08-03 02:09:26 INFO finding revisions to fetch:generating index 0/118432
2016-08-03 02:09:38 INFO finding revisions to fetch 1/3
2016-08-03 02:09:38 INFO
Traceback (most recent call last):
  File "/srv/importd.launchpad.net/production/launchpad-rev-18169/scripts/code-import-worker.py", line 96, in <module>
    sys.exit(script.main())
  File "/srv/importd.launchpad.net/production/launchpad-rev-18169/scripts/code-import-worker.py", line 91, in main
    return import_worker.run()
  File "/srv/importd.launchpad.net/production/launchpad-rev-18169/lib/lp/codehosting/codeimport/worker.py", line 583, in run
    return self._doImport()
  File "/srv/importd.launchpad.net/production/launchpad-rev-18169/lib/lp/codehosting/codeimport/worker.py", line 737, in _doImport
    inter_branch.fetch(limit=revision_limit)
  File "/srv/importd.launchpad.net/production/launchpad-rev-18169/bzrplugins/git/branch.py", line 722, in fetch
    self.fetch_objects(stop_revision, fetch_tags=fetch_tags, limit=limit)
  File "/srv/importd.launchpad.net/production/launchpad-rev-18169/bzrplugins/git/branch.py", line 745, in fetch_objects
    determine_wants, self.source.mapping, limit=limit)
  File "/srv/importd.launchpad.net/production/launchpad-rev-18169/bzrplugins/git/fetch.py", line 718, in fetch_objects
    limit)
  File "/srv/importd.launchpad.net/production/launchpad-rev-18169/bzrplugins/git/fetch.py", line 484, in import_git_objects
    mapping.revision_id_foreign_to_bzr)
  File "/srv/importd.launchpad.net/production/launchpad-rev-18169/bzrplugins/git/mapping.py", line 334, in import_commit
    raise UnknownCommitExtra(commit, [item[0] for item in commit.extra])
bzrlib.plugins.git.errors.UnknownCommitExtra: Unknown extra fields in <Commit e0380b795f649afaf6cd708abc5bf2680c8a01c0>: ['gpgsig'].
Import failed:
Traceback (most recent call last):
Failure: twisted.internet.error.ProcessTerm...

John, it has nothing to do with whether you're importing from GitHub; it depends on whether the repository's history contains commits with gpgsig tags.

Our plan for this remains as I outlined in comment 24, although some other urgent bits of work have jumped the queue and delayed us implementing it. A workaround for the time being might be to occasionally push the git repository to Launchpad's own git hosting manually.

lp:emacs is bit by this bug since April 2016. I'm eagerly awaiting those git-to-git imports ;-)

Launchpad git-to-git imports are available now: http://blog.launchpad.net/code/git-to-git-imports

Great news! Is there also a issue report here on Launchpad for the nest-part implementation for git recipes? Would like to get notified when it's there.

Thomas: I implemented that a while back (bug 1537579), but just forgot to update the documentation. I've done the latter as well now.

Awesome! Many thanks, Colin!

Same error while trying to import GitHub repo -- https://github.com/voria/samsung-tools
The latest commit is marked as "Verified" by GitHub and import fails.

https://code.launchpad.net/~voria/samsung-tools/master
http://launchpadlibrarian.net/351369453/voria-samsung-tools-master.log

brz-git can now import and preserve signatures (but won't create them on git commits you create from within breezy).

It seems Launchpad code import is still using bzr/bzr-git (Python 2 based, and unmaintained as far as I can tell) rather than brz (which has the fix for this issue). What are the plans (if any) to update the Launchpad importer to use brz?