Switching to to xz -0 from pxz increases compressed raspi2 image size

Bug #1732681 reported by Balint Reczey
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
livecd-rootfs (Ubuntu)
Fix Released
Undecided
Balint Reczey
Xenial
Invalid
Undecided
Balint Reczey

Bug Description

[Impact]

 * Switching from xz -0 to xz (-6) for compression raspi2 image.

[Test Case]

 * To test the change install updated livecd-rootfs on armhf and create images setting the following environment variables: ARCH=armhf SUBARCH=raspi2 PROJECT=ubuntu-cpc.
   Notice that the xz-compressed rapi2 image is still created with xz's default (-6) compression.

[Regression Potential]

 * In case of this change causes regression it would manifest itself as failing build for raspi2, failing build for ubuntu-cpc images or using other compression for raspi2 images than xz -6.

[Original Bug Report Text]

Switching from pxz to xz -0 in LP: #1701132 made the downloadable raspi2 image bigger, ~360MB which is a notable size increase from the original ~250MB while decreasing time the needed for compressing it.
The bigger compressed image is currently downloadable for Artful:
http://cdimage.ubuntu.com/ubuntu/releases/17.10/release/

Zesty's image is 232MB:
http://cdimage.ubuntu.com/ubuntu/releases/17.04/release/

To find the proper trade-off I ran compression tests on Xenial (xz ignores -T4 there):
https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/1701132/comments/5 :
...
rbalint@xenial-test:~/Downloads$ ls -al ubuntu-16.04.3-preinstalled-server-armhf+raspi2.img.p.xz
-rw-rw-r-- 1 rbalint rbalint 254621780 Nov 16 00:03 ubuntu-16.04.3-preinstalled-server-armhf+raspi2.img.p.xz
...
-0 -T4 81.56 73.64 1.46 4820 29.27 25.82 0.09 2252 329984
-1 -T4 103.68 95.14 1.67 10800 24.41 22.86 0.12 3180 298060
-2 -T4 134.05 126.22 1.76 18480 22.06 21.65 0.13 4232 286276
-3 -T4 170.57 162.72 1.72 33812 21.62 20.94 0.12 6188 275036
-4 -T4 322.15 317.83 2.10 50284 21.55 21.05 0.07 6136 258684
-5 -T4 407.15 399.93 1.92 97656 21.85 20.50 0.12 10412 247320
-6 -T4 478.64 473.32 1.95 97660 21.92 20.34 0.09 10348 245772
-7 -T4 493.97 485.52 1.99 191692 20.66 20.01 0.08 18548 240192
-8 -T4 512.28 506.08 2.20 380096 22.03 19.58 0.11 34888 229740
-9 -T4 652.71 559.20 4.78 691076 20.57 19.15 0.08 67700 224496
...

The tests on Bionic are still running.

Generating the raspi2 image is not part of autopkgtests at the moment, but may be periodically tested internally often and I can't tell the number of downloads either thus I'm asking for help to pick the proper compression setting for xz.

Looking at the Xenial numbers I believe -3 would result reasonably fast builds with a tolerable increase in size compared to xz -6's results.

Related branches

Balint Reczey (rbalint)
Changed in livecd-rootfs (Ubuntu):
status: New → Triaged
assignee: nobody → Balint Reczey (rbalint)
Revision history for this message
Balint Reczey (rbalint) wrote :

Compr. Threads Wall.t User.t Sys.t Res.t Wall.t User.t Sys.t Res.t Size(kB)
-0 -T1 77.68 68.78 1.10 4916 25.04 23.07 0.14 2516 365040
-1 -T1 100.05 89.72 1.24 11020 21.79 20.07 0.09 3360 322660
-2 -T1 127.23 118.40 1.31 18760 20.88 19.12 0.08 4292 309644
-3 -T1 165.56 160.31 1.30 34060 20.28 18.62 0.10 6388 302344
-4 -T1 257.81 252.85 1.26 50688 19.54 18.20 0.07 6364 284368
-5 -T1 322.24 318.51 1.41 97724 19.51 17.95 0.09 10576 272192
-6 -T1 374.25 366.92 1.39 97568 19.28 17.92 0.11 10664 270496
-7 -T1 389.19 385.40 1.44 191940 18.11 17.61 0.14 18664 266972
-8 -T1 405.00 400.37 1.73 380276 17.41 17.01 0.10 35028 258024
-9 -T1 486.82 436.84 3.08 690076 16.81 16.24 0.13 67888 251436
-0 -T4 37.83 93.87 1.32 25000 24.91 23.35 0.10 2652 368880
-1 -T4 45.74 138.40 1.59 72108 22.46 20.58 0.10 3304 330744
-2 -T4 56.26 175.05 1.86 126880 20.49 19.52 0.10 4440 315876
-3 -T4 68.22 240.02 1.93 235104 20.29 18.84 0.16 6360 307180
-4 -T4 94.85 348.37 2.52 300212 20.71 19.62 0.11 6488 289376
-5 -T4 118.08 435.77 2.88 556820 19.67 18.34 0.13 10580 276576
-6 -T4 133.11 495.87 2.37 556876 18.85 18.30 0.12 10468 274912
...

With -T4 -7 the Bionic VM with 1GB RAM which I tested the combinations in started swapping but probably we would not want to pick a better compression. If so I'll add the missing results.

Revision history for this message
Adam Conrad (adconrad) wrote :

-6 is the default, why would we be using -0?

Revision history for this message
Steve Langasek (vorlon) wrote :

> Looking at the Xenial numbers I believe -3 would result reasonably fast
> builds with a tolerable increase in size compared to xz -6's results.

As you've seen, the rpi2 server image is the only place we're using this code today. We are not overly concerned with the build speed as these are highly-parallelizable builds that are not part of a critical pipeline. I believe we should go with xz -6; certainly for bionic and forward where we have actual parallelization, the build time penalty appears to be negligible.

Revision history for this message
Balint Reczey (rbalint) wrote :

In development branch xz -0 is used for compressing the Ubuntu Core image as well but xz's compression setting have little impact on its compressed size thus -0 can be kept there:

(ubuntu-core-16-amd64.img)
ubuntu@bb:~/test-core$ cat time-size.txt
Compr. Threads Wall.t User.t Sys.t Res.t Wall.t User.t Sys.t Res.t Size(kB)
-0 -T1 79.97 76.04 1.54 5068 10.37 6.82 0.08 2528 381472
-1 -T1 119.13 115.42 1.76 10952 11.65 6.90 0.10 3376 379540
-2 -T1 148.43 142.16 1.74 18612 11.83 6.92 0.11 4440 379504
-3 -T1 164.89 158.90 1.93 34076 10.05 6.88 0.14 6416 379496
-4 -T1 178.44 172.44 1.85 50616 11.58 6.91 0.13 6296 379464
-5 -T1 187.43 181.44 2.13 97644 17.77 7.21 0.10 10484 379444
-6 -T1 201.22 194.96 2.03 97716 14.27 8.59 0.15 10572 379444
-7 -T1 201.56 194.90 1.97 191952 14.57 8.53 0.15 18660 379444
-8 -T1 200.86 195.67 2.10 380228 13.11 8.61 0.10 35112 379440
-9 -T1 283.20 238.30 4.75 689920 16.62 8.59 0.13 68004 379440
-0 -T4 33.40 109.76 2.43 24920 12.12 8.47 0.09 2744 381844
-1 -T4 56.35 174.24 2.91 73592 11.99 8.51 0.10 3296 379668
-2 -T4 55.58 185.15 3.82 141184 12.63 8.55 0.10 4376 379572
-3 -T4 55.92 194.41 3.15 276356 11.82 8.67 0.10 6348 379528
-4 -T4 63.59 218.39 3.29 342144 11.74 8.57 0.09 6416 379496
-5 -T4 64.24 226.16 5.84 675728 12.31 8.56 0.12 10488 379460
-6 -T4 67.20 239.20 5.55 675512 12.88 8.54 0.15 10652 379456
-7 -T4 3539.73 306.44 98.61 740100 12.55 8.60 0.12 18632 379448
Command terminated by signal 2
-8 -T4 34404.61 126.70 410.87 720576

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package livecd-rootfs - 2.485

---------------
livecd-rootfs (2.485) bionic; urgency=medium

  [ Michael Hudson-Doyle ]
  * Automount a partition of fixed UUID for giving subiquity canned answers.

  [ Balint Reczey ]
  * Use default xz compression (-6) for raspi2 image.
    Xz with -0 generated ~35% bigger compressed image. (LP: #1732681)
  * Zero fill space in UEFI partitions, too
  * Set LB_ISO_PREPARER to livecd-rootfs to avoid unbound variable in
    default string
  * Build minimized images in a 1GiB image

 -- Michael Hudson-Doyle <email address hidden> Mon, 04 Dec 2017 10:51:13 +1300

Changed in livecd-rootfs (Ubuntu):
status: Triaged → Fix Released
Balint Reczey (rbalint)
Changed in livecd-rootfs (Ubuntu Xenial):
status: New → In Progress
assignee: nobody → Balint Reczey (rbalint)
Balint Reczey (rbalint)
description: updated
Revision history for this message
Balint Reczey (rbalint) wrote :

Since the change passing -0 to xz never entered Xenial on its own this bug does not have to be tracked for Xenial.

Changed in livecd-rootfs (Ubuntu Xenial):
status: In Progress → Fix Committed
status: Fix Committed → Invalid
tags: added: id-59ff10e7eddaf8d676d43eb3
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.