[SRU] dpkg zstd support

Bug #1764220 reported by Balint Reczey
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
dpkg
Fix Released
Unknown
dpkg (Ubuntu)
Fix Released
Undecided
Unassigned
Xenial
Fix Released
Undecided
Balint Reczey

Bug Description

[Impact]

* Xenial's dpkg can't decompress zstd-compressed binary packages preventing some systems of Launchpad from processing packages with such compression. This blocks publishing zstd-compressed binary packages through Launchpad for later Ubuntu releases as well.

[Test Plan]

* https://people.canonical.com/~rbalint/zstd-debs/ contains a .deb built on Hirsute having both data and control members of the .deb being compressed with zstd.
* Download and unpack it. With unfixed dpkg an error should be shown.

$ wget https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb

# unfixed:
$ dpkg-deb -R glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb glibc-doc-extracted
dpkg-deb: error: archive 'glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb' uses unknown compression for member 'control.tar.zst', giving up

# fixed
$ time dpkg-deb -R glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb glibc-doc-extracted

real 0m0.148s
user 0m0.041s
sys 0m0.124s

* Also install the package:

root@x-zstd:~# dpkg -i glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
Selecting previously unselected package glibc-doc-reference.
(Reading database ... 25816 files and directories currently installed.)
Preparing to unpack glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb ...
Unpacking glibc-doc-reference (2.33-0ubuntu2~zstd1) ...
Setting up glibc-doc-reference (2.33-0ubuntu2~zstd1) ...
Processing triggers for install-info (6.1.0.dfsg.1-5) ...
root@x-zstd:~#

* Build the hello package, it should work
* Build the hello package overriding the compression to zstd, this should fail:

$ cat debian/rules
...

override_dh_builddeb:
 dh_builddeb -- -Zzstd

...
make[1]: Entering directory '/root/hello-2.10'
dh_builddeb -- -Zzstd
dpkg-deb: error: only decompression is supported for 'zstd'!

Type dpkg-deb --help for help about manipulating *.deb files;
Type dpkg --help for help about installing and deinstalling packages.
dh_builddeb: dpkg-deb -Zzstd --build debian/hello .. returned exit code 2
debian/rules:12: recipe for target 'override_dh_builddeb' failed

[Where problems could occur]

* The fix is isolated and is a backport from Bionic with the compression part omitted. Crashes could happen due to coding errors should they exist.
Only decompression should be supported and this is verified in the test plan.

* Incompabilities between libzstd present in Xenial and present in later releases could prevent dpkg running on Xenial from successfully processing packages built on later releases. The package for Xenial is built with libzstd1 (build-depending on libzstd1-dev) which is at version 1.3.1+dfsg-1~ubuntu0.16.04.1. Bionic's version is 1.3.3+dfsg-2ubuntu1.2 and there were no format breaking changes between those versions, not in any later version. See https://github.com/facebook/zstd/issues/999#issuecomment-359538229 .

[Original Bug Text]

As discussed previously, we want to have zstd support in 18.04 to evaluate and potentially enable it in later releases.

The zstd support adds a dependency on libzstd1 to dpkg. This should not have any effect on live images, since libzstd1 is part of the various live tasks, as btrfs-progs need it. For installed systems, this might be a new dependency (if they do not use btrfs, tor, or some other tools), so an increase of ~520 KB, as that's the size of the library and the library only depends on libc6.

The change is isolated, it adds the compressor and decompressor to dpkg, please see the attached patch for the details.

The change is being discussed here:
https://lists.ubuntu.com/archives/ubuntu-devel/2018-March/040211.html
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892664

Revision history for this message
Balint Reczey (rbalint) wrote :
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Since it's a non-invasive feature (+ we're already +1 on it for apt), FFe approved. Please proceed!

Changed in dpkg (Ubuntu):
status: New → Triaged
tags: added: id-5a9f9b96fff3bd2877ffa9de
Changed in dpkg:
status: Unknown → New
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dpkg - 1.19.0.5ubuntu2

---------------
dpkg (1.19.0.5ubuntu2) bionic; urgency=medium

  * Add Zstandard compression and decompression support for binary packages
    (LP: #1764220)

 -- Balint Reczey <email address hidden> Mon, 16 Apr 2018 04:25:21 +0200

Changed in dpkg (Ubuntu):
status: Triaged → Fix Released
Changed in dpkg:
status: New → Incomplete
Balint Reczey (rbalint)
summary: - [FFe] dpkg zstd support
+ [SRU] dpkg zstd support
Balint Reczey (rbalint)
Changed in dpkg (Ubuntu Xenial):
assignee: nobody → Balint Reczey (rbalint)
Balint Reczey (rbalint)
description: updated
Balint Reczey (rbalint)
description: updated
Balint Reczey (rbalint)
description: updated
Balint Reczey (rbalint)
description: updated
description: updated
Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello Balint, or anyone else affected,

Accepted dpkg into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dpkg/1.18.4ubuntu1.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in dpkg (Ubuntu Xenial):
status: New → Fix Committed
tags: added: verification-needed verification-needed-xenial
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (dpkg/1.18.4ubuntu1.7)

All autopkgtests for the newly accepted dpkg (1.18.4ubuntu1.7) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

sbuild/0.67.0-2ubuntu7.1 (arm64, ppc64el, amd64)
systemd/229-4ubuntu21.31 (amd64, i386)
glib2.0/2.48.2-0ubuntu4.8 (armhf)
cmake/3.5.1-1ubuntu3 (arm64, armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#dpkg

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Balint Reczey (rbalint) wrote :
Download full text (6.5 KiB)

Verified 1.18.4ubuntu1.7 on Xenial:

root@x-zstd:~# wget https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
--2021-04-21 15:37:36-- https://people.canonical.com/~rbalint/zstd-debs/glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
Resolving people.canonical.com (people.canonical.com)... 91.189.89.62
Connecting to people.canonical.com (people.canonical.com)|91.189.89.62|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5618446 (5.4M) [application/x-debian-package]
Saving to: ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’

glibc-doc-reference_2.33-0ubuntu2~zstd1_all 100%[=========================================================================================>] 5.36M 4.76MB/s in 1.1s

2021-04-21 15:37:38 (4.76 MB/s) - ‘glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb’ saved [5618446/5618446]

root@x-zstd:~# dpkg-deb -R glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb glibc-doc-extracted
dpkg-deb: error: archive 'glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb' uses unknown compression for member 'control.tar.zst', giving up
root@x-zstd:~# sed -i s/backports/proposed/ /etc/apt/sources.list
root@x-zstd:~# apt update -qq
5 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@x-zstd:~# apt install -qq dpkg
The following package was automatically installed and is no longer required:
  libfreetype6
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
  libzstd1
The following NEW packages will be installed:
  libzstd1
The following packages will be upgraded:
  dpkg
1 upgraded, 1 newly installed, 0 to remove and 4 not upgraded.
Need to get 2,237 kB of archives.
After this operation, 399 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Selecting previously unselected package libzstd1.
(Reading database ... 25816 files and directories currently installed.)
Preparing to unpack .../libzstd1_1.3.1+dfsg-1~ubuntu0.16.04.1_amd64.deb ...
Unpacking libzstd1 (1.3.1+dfsg-1~ubuntu0.16.04.1) ...
Processing triggers for libc-bin (2.23-0ubuntu11.2) ...
Setting up libzstd1 (1.3.1+dfsg-1~ubuntu0.16.04.1) ...
Processing triggers for libc-bin (2.23-0ubuntu11.2) ...
(Reading database ... 25821 files and directories currently installed.)
Preparing to unpack .../dpkg_1.18.4ubuntu1.7_amd64.deb ...
Unpacking dpkg (1.18.4ubuntu1.7) over (1.18.4ubuntu1.6) ...
Setting up dpkg (1.18.4ubuntu1.7) ...
Processing triggers for man-db (2.7.5-1) ...
root@x-zstd:~# time dpkg-deb -R glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb glibc-doc-extracted

real 0m0.031s
user 0m0.009s
sys 0m0.037s
root@x-zstd:~# dpkg -i glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb
Selecting previously unselected package glibc-doc-reference.
(Reading database ... 25821 files and directories currently installed.)
Preparing to unpack glibc-doc-reference_2.33-0ubuntu2~zstd1_all.deb ...
Unpacking glibc-doc-reference (2.33-0ubuntu2~zstd1) ...
Setting up glibc-doc-reference (2.33-0ubuntu2~zstd1) ...
Processing triggers for install-info (6.1.0.dfsg.1-5) ...
root@x-zstd:~# dpkg -l glibc-doc-reference
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/C...

Read more...

tags: added: verification-done verification-done-xenial
removed: verification-needed verification-needed-xenial
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for dpkg has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dpkg - 1.18.4ubuntu1.7

---------------
dpkg (1.18.4ubuntu1.7) xenial; urgency=medium

  * Add Zstandard (zstd) decompression support for binary packages
    (LP: #1764220)
    - The changes to ./configure and *.in files were cherry-picked from a
      manual autoreconf run because the package build does not run
      autoreconf automatically and the full delta from autoreconf
      was very big.

 -- Balint Reczey <email address hidden> Mon, 12 Apr 2021 17:41:27 +0200

Changed in dpkg (Ubuntu Xenial):
status: Fix Committed → Fix Released
Changed in dpkg:
status: Incomplete → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.