Cannot restore when passphrase has changed

My use case:

Done incremental backups for a long time:

export PASSPHRASE=one
duplicity full $args $src $dst
duplicity $args $src $dst
... etc

Now, I need to rotate the passphrase, so I create a new chain with the new passphrase and keep on doing incrementals with that new passphrase:

export PASSPHRASE=two
duplicity full $args $src $dst
duplicity $args $src $dst
... etc

Now, I need to restore the last backup in another machine. It fails to fetch the metadata:

export PASSPHRASE=two
duplicity restore $args --force $dst $src
Synchronizing remote metadata to local cache...
Copying manifest-backup.example.com-duplicity-full.20181230T010029Z.manifest.gpg to local cache.
GPGError: GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: AES encrypted data
gpg: encrypted with 1 passphrase
gpg: decryption failed: Bad session key
===== End GnuPG log =====

According to the date of the archive it's restoring, it must have been encrypted using the old PASSPHRASE. However, this is expected. I don't really need to restore that file until I need to restore an older backup. Is there no other way to do this? Can't I just tell Duplicity to sync signatures and manifests only for the needed chains (the last one actually)?