Ubuntu
simplestreams package

Signature verification is too slow

Bug #1580534 reported by Robie Basak
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
simplestreams
Fix Released
Medium
Unassigned
simplestreams (Ubuntu)
Fix Released
Medium
Unassigned
Xenial
Fix Released
Medium
Unassigned

Bug Description

[Impact]

When running against http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:download.sjson, read_signed() in simplestreams/utils.py seems to take minutes (402 seconds in my instrumented timing) running 100% CPU. At the time of my test, download.sjson is 451737 bytes in 1870 lines.

This makes a uvtool sync virtually unusable on Xenial.

[Development Fix]

The culprit is the constant += of a string as it is read in. This is inefficient in Python as each += results in a new string that has to be allocated. Keeping the lines in a list and joining them at the end reduces the time to 0.127 seconds.

[Stable Fix]

No change cherry-pick of fix committed upstream.

[Test Case]

On a machine with reasonable connectivity, run: uvt-simplestreams-libvirt sync release=xenial arch=amd64

Expected result: takes a few seconds with minimal CPU use.

Actual result: takes more than five minutes with CPU pegged at 100%.

Scott also has a more direct test case here: http://paste.ubuntu.com/16377613/

[Original Notes]

Merge proposal to follow. I'm filing this bug to track a Xenial SRU.

See original description

Related branches

lp:~racb/simplestreams/optimise-read_signed
simplestreams-dev: Pending requested
Diff: 27 lines (+5/-4)
1 file modified
simplestreams/util.py (+5/-4)
lp:~simplestreams-dev/simplestreams/artful
Robie Basak (racb)
description: updated
Scott Moser (smoser)
Changed in simplestreams:
status: New → Fix Committed
importance: Undecided → Medium
Changed in simplestreams (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
Changed in simplestreams (Ubuntu Xenial):
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Robie Basak (racb) wrote :
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package simplestreams - 0.1.0~bzr430-0ubuntu1

---------------
simplestreams (0.1.0~bzr430-0ubuntu1) yakkety; urgency=medium

  * New upstream snapshot.
    - set a default user agent (LP: #1578622)
    - glance mirror: do not require that hypervisor_config be
      present (LP: #1578624)
    - Optimise read_signed method for better performance (LP: #1580534)

 -- Scott Moser <email address hidden> Thu, 12 May 2016 12:55:30 -0400

Changed in simplestreams (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Martin Pitt (pitti) wrote : Please test proposed package

Hello Robie, or anyone else affected,

Accepted simplestreams into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/simplestreams/0.1.0~bzr426-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in simplestreams (Ubuntu Xenial):
status: Confirmed → Fix Committed
tags: added: verification-needed
Revision history for this message
Mike Pontillo (mpontillo) wrote :

Looks much better.

$ time uvt-simplestreams-libvirt sync

real 6m24.381s
user 1m7.168s
sys 1m27.080s

$ sudo apt-get install python-simplestreams/xenial-proposed
...
Unpacking python-simplestreams (0.1.0~bzr426-0ubuntu1.1) over (0.1.0~bzr426-0ubuntu1) ...
Setting up python-simplestreams (0.1.0~bzr426-0ubuntu1.1) ...

$ time uvt-simplestreams-libvirt sync

real 0m2.743s
user 0m0.356s
sys 0m0.032s

tags: added: verification-done
removed: verification-needed
Revision history for this message
Mike Pontillo (mpontillo) wrote :

Just realized my test case was flawed. I did the first command on a system without all the images synced. I repeated my test on a fresh system:

Initial sync:
$ time uvt-simplestreams-libvirt sync 'release~(xenial|trusty)' arch=amd64

real 5m59.505s
user 0m35.128s
sys 0m54.144s

Subsequent sync:
$ time uvt-simplestreams-libvirt sync 'release~(xenial|trusty)' arch=amd64

real 0m42.292s
user 0m18.360s
sys 0m21.176s

Sync after installing python-simplestreams from xenial-proposed:
$ time uvt-simplestreams-libvirt sync 'release~(xenial|trusty)' arch=amd64

real 0m3.211s
user 0m0.396s
sys 0m0.120s

Verdict: looks good to me.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package simplestreams - 0.1.0~bzr426-0ubuntu1.1

---------------
simplestreams (0.1.0~bzr426-0ubuntu1.1) xenial; urgency=medium

  * Fix signature verification speed (LP: #1580534).

 -- Robie Basak <email address hidden> Thu, 12 May 2016 17:43:07 +0000

Changed in simplestreams (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Chris J Arges (arges) wrote : Update Released

The verification of the Stable Release Update for simplestreams has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Scott Moser (smoser) wrote : Fixed in simplestreams version 0.1.0.

This bug is believed to be fixed in simplestreams in version 0.1.0. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in simplestreams:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.