Merge lp:~smoser/vmbuilder/mfdiff-apt-key-transition into lp:~ubuntu-on-ec2/vmbuilder/mfdiff

[Scott Moser]

[Dimitri John Ledkov (xnox)]

You can choose to trust /usr/share/keyrings/ubuntu-archive-removed-keys.gpg and /usr/share/keyrings/ubuntu-archive-keyring.gpg, however everything should be dual-signed, and just one valid trusted signature should be sufficient (aka just the /usr/share/keyrings/ubuntu-archive-keyring.gpg).

[Scott Moser (smoser)]

@Dimitri,
I'm confused.

"everything should be dual-signed, and just one valid trusted signature should be sufficient (aka just the /usr/share/keyrings/ubuntu-archive-keyring.gpg)"

I didn't make up this problem. I'm pretty sure that everything is *not* dual-signed... thats the point of the transition, otherwise it wouldn't be a 'transition'.

The problem is specifically that if you run mfdiff on zesty, then the python-apt code (or something in that path) downloads apt data, and checks it, but the apt data it downloads is signed with a key that is not in /usr/share/keyrings/ubuntu-archive-keyring.gpg . It is downloading xenial archive data on a zesty system and the zesty key ring does not have the xenial signer.

I wonder, what is the plan for a compromise on an archive signing key? Would that key then get moved into ubuntu-archive-removed-keys.gpg ? or would it be added to a ubuntu-archive-COMPROMISED-keys.gpg ? The reason I ask is that if we added it to -removed-keys.gpg, then anyone using that (as you suggest above) would be open to MIM attack.

I do admit that checking these keys in directly here as I've suggested in this MP *does* open me up to a MIM attack on someone who has gained access to the old archive key, and it does seem like there should be some path forward that would allow me to trust "old keys that are not known to be compromised".

[Dimitri John Ledkov (xnox)]

On 9 January 2017 at 18:52, Scott Moser <email address hidden> wrote:
> @Dimitri,
> I'm confused.
>
> "everything should be dual-signed, and just one valid trusted signature should be sufficient (aka just the /usr/share/keyrings/ubuntu-archive-keyring.gpg)"
>
> I didn't make up this problem. I'm pretty sure that everything is *not* dual-signed... thats the point of the transition, otherwise it wouldn't be a 'transition'.
>
> The problem is specifically that if you run mfdiff on zesty, then the python-apt code (or something in that path) downloads apt data, and checks it, but the apt data it downloads is signed with a key that is not in /usr/share/keyrings/ubuntu-archive-keyring.gpg . It is downloading xenial archive data on a zesty system and the zesty key ring does not have the xenial signer.
>

python-apt nor apt itself use
/usr/share/keyrings/ubuntu-archive-kerying.gpg by default

by default they use /etc/apt/trusted.gpg.d/* fragments and legacy
/etc/apt/trusted.gpg only

looking at the code, i'm struggling to see what exactly errors out.
What keys are installed in the cacheroot? doesn't everything read the
trusted keys not from the host, but from the cacheroot... which has no
keys at all? shouldn't trusted-keys be pointed to the host, rather
than cacheroot?

note python-apt has changed in zesty. Can you try to install e.g.
ubuntu-kerying from zesty on a xenial system; and check if this script
works or not?

> I wonder, what is the plan for a compromise on an archive signing key? Would that key then get moved into ubuntu-archive-removed-keys.gpg ? or would it be added to a ubuntu-archive-COMPROMISED-keys.gpg ? The reason I ask is that if we added it to -removed-keys.gpg, then anyone using that (as you suggest above) would be open to MIM attack.
>
> I do admit that checking these keys in directly here as I've suggested in this MP *does* open me up to a MIM attack on someone who has gained access to the old archive key, and it does seem like there should be some path forward that would allow me to trust "old keys that are not known to be compromised".
>

Even when the key is compromised we will continue to sign the archives
with the compromised key in addition to the newly generated key, as
otherwise it will break fresh installs with last point release media.
(I'm guessing if key is ever compromised we will be doing installation
media respin trusting new keys only).

There should be valid signatures for one or more keys one trusts, so a
valid signature with the 4k key and a signature for a missing key
should be still ok. The signature with a missing key might be a newly
generated key; or an obsoleted key; one can never know.

--
Regards,

Dimitri.

[Dan Watkins (oddbloke)]

We've just added mfdiff to our review tracker, so I've just seen this. Scott, is this still something we need?

[Scott Moser (smoser)]

I'm pretty sure that commit b2544cf3955c7d907ce412187bc902910cae9f4a was
commited to fix this basic issue. However, the fix provided there does
entirely fix it.

Below is a demonstration of failure that is still present. Essentially
if you are running on bionic you cannot verify precise data.

In a fresh bionic lxc container as non-root user:

$ git clone -q git://git.launchpad.net/~cloud-images-release-managers/cloud-images/+git/mfdiff
$ cd mfdiff
$ r=precise; for s in 20170424 20170502; do:
   wget http://cloud-images.ubuntu.com/$r/$s/$r-server-cloudimg-amd64.manifest -O precise-$s.manifest; done

$ python3 ./mfdiff amd64 precise precise-20170424.manifest precise-20170502.manifest 2>&1 | tee out.log
$ pastebinit out.log
http://paste.ubuntu.com/p/9449xgH2bs/

$ dpkg-query --show ubuntu-keyring
ubuntu-keyring 2018.09.18.1~18.04.0
$ ls -l /etc/apt/trusted.gpg.d/
total 10
-rw-r--r-- 1 root root 2796 Sep 17 23:15 ubuntu-keyring-2012-archive.gpg
-rw-r--r-- 1 root root 2794 Sep 17 23:15 ubuntu-keyring-2012-cdimage.gpg
-rw-r--r-- 1 root root 1733 Sep 17 23:15 ubuntu-keyring-2018-archive.gpg

[Dan Watkins (oddbloke)]

So is the underlying issue with precise that we are no longer shipping the 2004 key as a trusted key in (zesty/)bionic and later?

Should we be concerned that we'll start seeing breakage in the opposite direction once the 2012 key is no longer being used for signing new releases?

[Scott Moser (smoser)]

probably/possibly.
I guess that precise's available keys would not be able to verify bionic data.
thats much less of a problem though, as its easy enough to just use bionic.
On Tue, Dec 11, 2018 at 2:28 PM Dan Watkins
<email address hidden> wrote:
>
> So is the underlying issue with precise that we are no longer shipping the 2004 key as a trusted key in (zesty/)bionic and later?
>
> Should we be concerned that we'll start seeing breakage in the opposite direction once the 2012 key is no longer being used for signing new releases?
> --
> https://code.launchpad.net/~smoser/vmbuilder/mfdiff-apt-key-transition/+merge/313797
> You are the owner of lp:~smoser/vmbuilder/mfdiff-apt-key-transition.

[Dan Watkins (oddbloke)]

My concern is that, unless I'm misunderstanding something, this change effectively freezes the keyring we use to xenial's keyring. So at some point in the future, this keyring will become outdated regardless of the underlying release (so I won't be able to just use bionic).

Is my understanding off?

[Scott Moser (smoser)]

of course. At said later date, you can just add a key to the keyring
and commit it.

The change that is in place currently relies on the host to provide
the keys, and quite simply no host will *ever* provide historical and
future keys in /etc/apt/trusted.gpg.d .

So yes, this change does add maintenance on the order of once per N
years (N=6?) that is in some cases provided "for free" by using the
content from ubuntu-keyring.

On Tue, Dec 11, 2018 at 2:40 PM Dan Watkins
<email address hidden> wrote:
>
> My concern is that, unless I'm misunderstanding something, this change effectively freezes the keyring we use to xenial's keyring. So at some point in the future, this keyring will become outdated regardless of the underlying release (so I won't be able to just use bionic).
>
> Is my understanding off?
> --
> https://code.launchpad.net/~smoser/vmbuilder/mfdiff-apt-key-transition/+merge/313797
> You are the owner of lp:~smoser/vmbuilder/mfdiff-apt-key-transition.

[Dan Watkins (oddbloke)]

Would copying /etc/apt/trusted.gpg* in to the cache and then adding this keyring.gpg to cache_dir/etc/apt/trusted.gpg.d save us from having to do the maintenance in future?

[Scott Moser (smoser)]

well, no.
because eventually there will be a host that does not have keys in
/etc/apt/trusted.gpg/* that can verify bionic (when the 2018-archive
signing key is retired).

essentially all you'd be doing there is changing this the keyring
maintained here to be 'obsolete-signing-keys'.

Possibly the right thing to do would be to have a package called
'ubuntu-keyring-retired'. Then if that package got updated correctly
mfdiff and other things could just rely on the fact that the
combination of keyrings provided by 'ubuntu-keyring' and
'ubuntu-keyring-retired' would be able to verify all things.

On Tue, Dec 11, 2018 at 3:15 PM Dan Watkins
<email address hidden> wrote:
>
> Would copying /etc/apt/trusted.gpg* in to the cache and then adding this keyring.gpg to cache_dir/etc/apt/trusted.gpg.d save us from having to do the maintenance in future?
> --
> https://code.launchpad.net/~smoser/vmbuilder/mfdiff-apt-key-transition/+merge/313797
> You are the owner of lp:~smoser/vmbuilder/mfdiff-apt-key-transition.

[Dan Watkins (oddbloke)]

I've filed https://bugs.launchpad.net/ubuntu/+source/ubuntu-keyring/+bug/1809027 to capture a way of fixing this at a higher-level. Until we get some resolution there (whether that be fixing it, or declining to make it a priority), I don't think we should merge this change.

Even if we _should_, it should be resubmitted against the git trunk as a separate MP.

(I'm also going to mark it WIP so it doesn't show up in our review board until the new one is submitted.)

review: Needs Resubmitting

Unmerged revisions

19. By Scott Moser on 2016-12-22

Add and use local keyring for verification of apt data.

A transition is being made from signing with the dsa1024 'Archive'
signing key below to the newer rsa4096 Archive key.
In zesty, the package installed keyring no longer contains the
older keys.

Without this, if running on zesty, then mfdiff will fail to download
changelogs (using apt.cache), as it will not trust the data.

The solution here is to check in the gpg keyring that was provided
by the Ubuntu 16.04 (xenial) package ubuntu-keyring, and to use that
for verification.

$ gpg keyring.gpg

pub dsa1024 2004-09-12 [SC]
      630239CC130E1A7FD81A27B140976EAF437D05B5
uid Ubuntu Archive Automatic Signing Key <email address hidden>
sub elg2048 2004-09-12 [E]
pub dsa1024 2004-12-30 [SC]
      C5986B4F1257FFA86632CBA746181433FBB75451
uid Ubuntu CD Image Automatic Signing Key <email address hidden>
pub rsa4096 2012-05-11 [SC]
      790BC7277767219C42C86F933B4FE6ACC0B21F32
uid Ubuntu Archive Automatic Signing Key (2012) <email address hidden>
pub rsa4096 2012-05-11 [SC]
      843938DF228D22F7B3742BC0D94AA3F0EFE21092
uid Ubuntu CD Image Automatic Signing Key (2012) <email address hidden>

=== added file 'keyring.gpg'
Binary files keyring.gpg	1970-01-01 00:00:00 +0000 and keyring.gpg	2016-12-22 19:15:46 +0000 differ
=== modified file 'mfdiff'
--- mfdiff	2014-10-03 11:27:32 +0000
+++ mfdiff	2016-12-22 19:15:46 +0000
@@ -29,6 +29,10 @@
 import apt
 import re
 
+# use the locally provided 'keyring.gpg' which will accept
+# keys that sign releases newer than 16.04 and older.
+KEYRING = os.path.join(os.path.dirname(__file__), "keyring.gpg")
+
 Usage="""
 Usage: %s --apt-sources F arch manifest1 manifest2
    Compare two manifest files, and show changelog differences.
@@ -125,6 +129,11 @@
 		os.makedirs( "%s/etc/apt" % cache_d )
 	except:
 		pass
+
+	with open(KEYRING, "rb") as fpr:
+		with open("%s/etc/apt/trusted.gpg" % cache_d, "wb") as fpw:
+			fpw.write(fpr.read())
+
 	asl = open( "%s/etc/apt/sources.list" % cache_d, "w" )
 	asl.write('\n'.join(srclines))
 	asl.close()