Merge ~cjwatson/launchpad:built-using-model into launchpad:master

Proposed by Colin Watson
Status: Merged
Approved by: Colin Watson
Approved revision: 4f4c8809bc92c7ef495d8281b0fbac9cd2d7bc92
Merge reported by: Otto Co-Pilot
Merged at revision: not available
Proposed branch: ~cjwatson/launchpad:built-using-model
Merge into: launchpad:master
Diff against target: 1100 lines (+678/-24)
18 files modified
database/schema/security.cfg (+2/-0)
lib/lp/archiveuploader/nascentuploadfile.py (+8/-1)
lib/lp/archiveuploader/tests/test_nascentuploadfile.py (+29/-1)
lib/lp/registry/model/distroseries.py (+21/-1)
lib/lp/soyuz/configure.zcml (+14/-0)
lib/lp/soyuz/doc/distroarchseriesbinarypackage.txt (+2/-0)
lib/lp/soyuz/enums.py (+14/-1)
lib/lp/soyuz/interfaces/binarypackagebuild.py (+3/-3)
lib/lp/soyuz/interfaces/binarypackagerelease.py (+6/-1)
lib/lp/soyuz/interfaces/binarysourcereference.py (+86/-0)
lib/lp/soyuz/model/binarypackagebuild.py (+13/-4)
lib/lp/soyuz/model/binarypackagerelease.py (+15/-1)
lib/lp/soyuz/model/binarysourcereference.py (+149/-0)
lib/lp/soyuz/scripts/gina/handlers.py (+16/-1)
lib/lp/soyuz/scripts/gina/packages.py (+10/-1)
lib/lp/soyuz/scripts/tests/test_gina.py (+73/-4)
lib/lp/soyuz/tests/test_binarysourcereference.py (+208/-0)
lib/lp/soyuz/tests/test_publishing.py (+9/-5)
Reviewer Review Type Date Requested Status
Tom Wardill (community) Approve
William Grant direction Approve
Review via email: mp+381234@code.launchpad.net

Commit message

Parse Built-Using fields from uploaded binaries

Description of the change

These are parsed into a form that we'll later be able to use in the dominator, with the goal of keeping source publications that are referenced by Built-Using fields in active binary publications.

The most complicated bit is working out which source package releases a given Built-Using field refers to, since there are edge cases where just the name and version can be ambiguous. We do the best we can by following the archive dependencies of the build and finding sources that it could plausibly have used.

Unresolvable entries in Built-Using, including ones that refer to deleted source publications, will result in binary uploads being rejected.

Database MP: https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/381036

To post a comment you must log in.
Revision history for this message
William Grant (wgrant) :
review: Approve (direction)
de9e46b... by Colin Watson

Explicitly check "build.current_component is None"

This can happen in tests, especially of gina, and the resulting crash is
a bit confusing, so let's raise a clearer exception in this case.

fde0f10... by Colin Watson

Preserve original Built-Using fields for publication

We aren't sure whether apt will get confused if the exact serialisation
(e.g. ordering) of Built-Using in Packages differs from that in the
package's control file, so let's preserve it rather than trying to
reconstruct it.

0692e86... by Colin Watson

Don't consider SPPH.status when creating BSRs

We don't normally treat SUPERSEDED differently from DELETED in this sort
of situation, and at that point it isn't really worth checking the
status at all.

a4163e3... by Colin Watson

Add more vertical whitespace

7ca1f32... by Colin Watson

Limit Built-Using references to the same archive and series

Allowing cross-archive references makes the dominator's job harder, as
well as permitting non-consensual pinning of sources to the Published
state in foreign archives. Forbidding such references may be confusing
in some cases, but seems likely to be less bad than the alternative.

8983670... by Colin Watson

Refactor createFromRelationship using Archive.getPublishedSources

Now that we're only considering a single archive, this is equivalent and
much clearer.

e9bb481... by Colin Watson

Add BinarySourceReference.createFromSourcePackageReleases

This makes some tests (especially in branches further up this stack)
more convenient.

Revision history for this message
Tom Wardill (twom) wrote :

What is the possible 'unfortunate apt behaviour'?

review: Needs Information
6fd4090... by Colin Watson

Clarify "unfortunate apt behaviour"

4f4c880... by Colin Watson

Give IBinaryPackageRelease.built_using_references a value_type

Revision history for this message
Colin Watson (cjwatson) wrote :

I've clarified this.

Revision history for this message
Tom Wardill (twom) wrote :

The model and implementation for this seem to make sense. I'm not entirely fluent in apt behaviours, but this LGTM :).

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/database/schema/security.cfg b/database/schema/security.cfg
index a0cf410..5ead0b8 100644
--- a/database/schema/security.cfg
+++ b/database/schema/security.cfg
@@ -1213,6 +1213,7 @@ public.binarypackagefile = SELECT, INSERT, UPDATE
1213public.binarypackagename = SELECT, INSERT, UPDATE1213public.binarypackagename = SELECT, INSERT, UPDATE
1214public.binarypackagepublishinghistory = SELECT, INSERT, UPDATE, DELETE1214public.binarypackagepublishinghistory = SELECT, INSERT, UPDATE, DELETE
1215public.binarypackagerelease = SELECT, INSERT, UPDATE1215public.binarypackagerelease = SELECT, INSERT, UPDATE
1216public.binarysourcereference = SELECT, INSERT
1216public.branch = SELECT, INSERT, UPDATE1217public.branch = SELECT, INSERT, UPDATE
1217public.bug = SELECT, INSERT, UPDATE1218public.bug = SELECT, INSERT, UPDATE
1218public.bugactivity = SELECT, INSERT, UPDATE1219public.bugactivity = SELECT, INSERT, UPDATE
@@ -1385,6 +1386,7 @@ public.binarypackagefile = SELECT, INSERT
1385public.binarypackagename = SELECT, INSERT1386public.binarypackagename = SELECT, INSERT
1386public.binarypackagepublishinghistory = SELECT1387public.binarypackagepublishinghistory = SELECT
1387public.binarypackagerelease = SELECT, INSERT1388public.binarypackagerelease = SELECT, INSERT
1389public.binarysourcereference = SELECT, INSERT
1388public.bug = SELECT, UPDATE1390public.bug = SELECT, UPDATE
1389public.bugactivity = SELECT, INSERT1391public.bugactivity = SELECT, INSERT
1390public.bugaffectsperson = SELECT, INSERT, UPDATE, DELETE1392public.bugaffectsperson = SELECT, INSERT, UPDATE, DELETE
diff --git a/lib/lp/archiveuploader/nascentuploadfile.py b/lib/lp/archiveuploader/nascentuploadfile.py
index aa0ec95..f28e43e 100644
--- a/lib/lp/archiveuploader/nascentuploadfile.py
+++ b/lib/lp/archiveuploader/nascentuploadfile.py
@@ -1,4 +1,4 @@
1# Copyright 2009-2019 Canonical Ltd. This software is licensed under the1# Copyright 2009-2020 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4"""Specific models for uploaded files"""4"""Specific models for uploaded files"""
@@ -445,6 +445,12 @@ class BaseBinaryUploadFile(PackageUploadFile):
445 "Provides",445 "Provides",
446 "Pre-Depends",446 "Pre-Depends",
447 "Enhances",447 "Enhances",
448 # Note that we intentionally don't include Built-Using here;
449 # although we parse it, we want to preserve its original form to
450 # make sure apt doesn't decide that it needs to keep re-upgrading
451 # the package to the same version because the metadata looks
452 # slightly out of sync. This is most easily done by adding it to
453 # user_defined_fields.
448 "Essential",454 "Essential",
449 "Description",455 "Description",
450 "Installed-Size",456 "Installed-Size",
@@ -920,6 +926,7 @@ class BaseBinaryUploadFile(PackageUploadFile):
920 pre_depends=encoded.get('Pre-Depends', ''),926 pre_depends=encoded.get('Pre-Depends', ''),
921 enhances=encoded.get('Enhances', ''),927 enhances=encoded.get('Enhances', ''),
922 breaks=encoded.get('Breaks', ''),928 breaks=encoded.get('Breaks', ''),
929 built_using=encoded.get('Built-Using', ''),
923 homepage=encoded.get('Homepage'),930 homepage=encoded.get('Homepage'),
924 essential=is_essential,931 essential=is_essential,
925 installedsize=installedsize,932 installedsize=installedsize,
diff --git a/lib/lp/archiveuploader/tests/test_nascentuploadfile.py b/lib/lp/archiveuploader/tests/test_nascentuploadfile.py
index 54841a9..ff63636 100644
--- a/lib/lp/archiveuploader/tests/test_nascentuploadfile.py
+++ b/lib/lp/archiveuploader/tests/test_nascentuploadfile.py
@@ -1,4 +1,4 @@
1# Copyright 2010-2019 Canonical Ltd. This software is licensed under the1# Copyright 2010-2020 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4"""Test NascentUploadFile functionality."""4"""Test NascentUploadFile functionality."""
@@ -26,6 +26,8 @@ from testtools.matchers import (
26 MatchesAny,26 MatchesAny,
27 MatchesListwise,27 MatchesListwise,
28 MatchesRegex,28 MatchesRegex,
29 MatchesSetwise,
30 MatchesStructure,
29 )31 )
3032
31from lp.archiveuploader.changesfile import ChangesFile33from lp.archiveuploader.changesfile import ChangesFile
@@ -43,6 +45,7 @@ from lp.services.compat import lzma
43from lp.services.log.logger import BufferLogger45from lp.services.log.logger import BufferLogger
44from lp.services.osutils import write_file46from lp.services.osutils import write_file
45from lp.soyuz.enums import (47from lp.soyuz.enums import (
48 BinarySourceReferenceType,
46 PackagePublishingStatus,49 PackagePublishingStatus,
47 PackageUploadCustomFormat,50 PackageUploadCustomFormat,
48 )51 )
@@ -573,6 +576,31 @@ class DebBinaryUploadFileTests(PackageUploadFileTestCase):
573 [u"RandomData", u"Foo\nbar\nbla\n"],576 [u"RandomData", u"Foo\nbar\nbla\n"],
574 ], bpr.user_defined_fields)577 ], bpr.user_defined_fields)
575578
579 def test_built_using(self):
580 # storeInDatabase parses Built-Using into BinarySourceReference
581 # rows, and also adds the unparsed contents to user_defined_fields.
582 uploadfile = self.createDebBinaryUploadFile(
583 "foo_0.42_i386.deb", "main/python", "unknown", "mypkg", "0.42",
584 None)
585 control = self.getBaseControl()
586 control["Built-Using"] = b"bar (= 0.1)"
587 uploadfile.parseControl(control)
588 build = self.factory.makeBinaryPackageBuild()
589 spph = self.factory.makeSourcePackagePublishingHistory(
590 archive=build.archive, distroseries=build.distro_series,
591 pocket=build.pocket, sourcepackagename="bar", version="0.1")
592 bpr = uploadfile.storeInDatabase(build)
593 self.assertThat(
594 bpr.built_using_references,
595 MatchesSetwise(
596 MatchesStructure.byEquality(
597 binary_package_release=bpr,
598 source_package_release=spph.sourcepackagerelease,
599 reference_type=BinarySourceReferenceType.BUILT_USING,
600 )))
601 self.assertEqual(
602 [[u"Built-Using", u"bar (= 0.1)"]], bpr.user_defined_fields)
603
576 def test_homepage(self):604 def test_homepage(self):
577 # storeInDatabase stores homepage field.605 # storeInDatabase stores homepage field.
578 uploadfile = self.createDebBinaryUploadFile(606 uploadfile = self.createDebBinaryUploadFile(
diff --git a/lib/lp/registry/model/distroseries.py b/lib/lp/registry/model/distroseries.py
index 51dbeea..fefc8cd 100644
--- a/lib/lp/registry/model/distroseries.py
+++ b/lib/lp/registry/model/distroseries.py
@@ -1,4 +1,4 @@
1# Copyright 2009-2019 Canonical Ltd. This software is licensed under the1# Copyright 2009-2020 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4"""Database classes for a distribution series."""4"""Database classes for a distribution series."""
@@ -124,6 +124,7 @@ from lp.services.propertycache import (
124from lp.services.worlddata.model.language import Language124from lp.services.worlddata.model.language import Language
125from lp.soyuz.enums import (125from lp.soyuz.enums import (
126 ArchivePurpose,126 ArchivePurpose,
127 BinarySourceReferenceType,
127 IndexCompressionType,128 IndexCompressionType,
128 PackagePublishingStatus,129 PackagePublishingStatus,
129 PackageUploadStatus,130 PackageUploadStatus,
@@ -1138,6 +1139,9 @@ class DistroSeries(SQLBase, BugTargetBase, HasSpecificationsMixin,
11381139
1139 def getBinaryPackagePublishing(self, archtag, pocket, component, archive):1140 def getBinaryPackagePublishing(self, archtag, pocket, component, archive):
1140 """See `IDistroSeries`."""1141 """See `IDistroSeries`."""
1142 # Circular import.
1143 from lp.soyuz.model.binarysourcereference import BinarySourceReference
1144
1141 bpphs = Store.of(self).find(1145 bpphs = Store.of(self).find(
1142 BinaryPackagePublishingHistory,1146 BinaryPackagePublishingHistory,
1143 DistroArchSeries.distroseries == self,1147 DistroArchSeries.distroseries == self,
@@ -1161,6 +1165,22 @@ class DistroSeries(SQLBase, BugTargetBase, HasSpecificationsMixin,
1161 bpbs = load_related(BinaryPackageBuild, bprs, ["buildID"])1165 bpbs = load_related(BinaryPackageBuild, bprs, ["buildID"])
1162 sprs = load_related(1166 sprs = load_related(
1163 SourcePackageRelease, bpbs, ["source_package_release_id"])1167 SourcePackageRelease, bpbs, ["source_package_release_id"])
1168
1169 built_using_bsrs = load_referencing(
1170 BinarySourceReference, bprs, ["binary_package_release_id"],
1171 extra_conditions=[
1172 BinarySourceReference.reference_type ==
1173 BinarySourceReferenceType.BUILT_USING,
1174 ])
1175 # Make sure this is initialised for all BPRs, as some may not
1176 # have any BinarySourceReferences.
1177 built_using_bsr_map = {bpr: [] for bpr in bprs}
1178 for bsr in built_using_bsrs:
1179 built_using_bsr_map[bsr.binary_package_release].append(bsr)
1180 for bpr, bsrs in built_using_bsr_map.items():
1181 get_property_cache(bpr).built_using_references = sorted(
1182 bsrs, key=attrgetter("id"))
1183
1164 bpfs = load_referencing(1184 bpfs = load_referencing(
1165 BinaryPackageFile, bprs, ["binarypackagereleaseID"])1185 BinaryPackageFile, bprs, ["binarypackagereleaseID"])
1166 file_map = collections.defaultdict(list)1186 file_map = collections.defaultdict(list)
diff --git a/lib/lp/soyuz/configure.zcml b/lib/lp/soyuz/configure.zcml
index 89c9d83..2e97958 100644
--- a/lib/lp/soyuz/configure.zcml
+++ b/lib/lp/soyuz/configure.zcml
@@ -70,6 +70,20 @@
70 set_attributes="changelog"/>70 set_attributes="changelog"/>
71 </class>71 </class>
7272
73 <!-- BinarySourceReference -->
74
75 <class
76 class="lp.soyuz.model.binarysourcereference.BinarySourceReference">
77 <allow
78 interface="lp.soyuz.interfaces.binarysourcereference.IBinarySourceReference"/>
79 </class>
80 <securedutility
81 class="lp.soyuz.model.binarysourcereference.BinarySourceReferenceSet"
82 provides="lp.soyuz.interfaces.binarysourcereference.IBinarySourceReferenceSet">
83 <allow
84 interface="lp.soyuz.interfaces.binarysourcereference.IBinarySourceReferenceSet"/>
85 </securedutility>
86
73 <!-- SourcePackagePublishingHistory -->87 <!-- SourcePackagePublishingHistory -->
7488
75 <class89 <class
diff --git a/lib/lp/soyuz/doc/distroarchseriesbinarypackage.txt b/lib/lp/soyuz/doc/distroarchseriesbinarypackage.txt
index 03d690a..c759bbf 100644
--- a/lib/lp/soyuz/doc/distroarchseriesbinarypackage.txt
+++ b/lib/lp/soyuz/doc/distroarchseriesbinarypackage.txt
@@ -62,6 +62,7 @@ needs to be removed.
62 ... pre_depends=None,62 ... pre_depends=None,
63 ... enhances=None,63 ... enhances=None,
64 ... breaks=None,64 ... breaks=None,
65 ... built_using=None,
65 ... essential=False,66 ... essential=False,
66 ... installedsize=0,67 ... installedsize=0,
67 ... architecturespecific=False,68 ... architecturespecific=False,
@@ -106,6 +107,7 @@ needs to be removed.
106 ... pre_depends=None,107 ... pre_depends=None,
107 ... enhances=None,108 ... enhances=None,
108 ... breaks=None,109 ... breaks=None,
110 ... built_using=None,
109 ... essential=False,111 ... essential=False,
110 ... installedsize=0,112 ... installedsize=0,
111 ... architecturespecific=False,113 ... architecturespecific=False,
diff --git a/lib/lp/soyuz/enums.py b/lib/lp/soyuz/enums.py
index b8dafa8..641df6b 100644
--- a/lib/lp/soyuz/enums.py
+++ b/lib/lp/soyuz/enums.py
@@ -1,4 +1,4 @@
1# Copyright 2010-2019 Canonical Ltd. This software is licensed under the1# Copyright 2010-2020 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4"""Enumerations used in the lp/soyuz modules."""4"""Enumerations used in the lp/soyuz modules."""
@@ -13,6 +13,7 @@ __all__ = [
13 'archive_suffixes',13 'archive_suffixes',
14 'BinaryPackageFileType',14 'BinaryPackageFileType',
15 'BinaryPackageFormat',15 'BinaryPackageFormat',
16 'BinarySourceReferenceType',
16 'DistroArchSeriesFilterSense',17 'DistroArchSeriesFilterSense',
17 'IndexCompressionType',18 'IndexCompressionType',
18 'PackageCopyPolicy',19 'PackageCopyPolicy',
@@ -613,3 +614,15 @@ class DistroArchSeriesFilterSense(DBEnumeratedType):
613614
614 Packages in this package set are excluded from the distro arch series.615 Packages in this package set are excluded from the distro arch series.
615 """)616 """)
617
618
619class BinarySourceReferenceType(DBEnumeratedType):
620 """The type of a reference from a binary package to a source package."""
621
622 BUILT_USING = DBItem(1, """
623 Built-Using
624
625 The referencing binary package incorporates part of the referenced
626 source package, and so the referenced source package needs to remain
627 in the archive for as long as the referencing binary package does.
628 """)
diff --git a/lib/lp/soyuz/interfaces/binarypackagebuild.py b/lib/lp/soyuz/interfaces/binarypackagebuild.py
index 8546a85..d811c6c 100644
--- a/lib/lp/soyuz/interfaces/binarypackagebuild.py
+++ b/lib/lp/soyuz/interfaces/binarypackagebuild.py
@@ -1,4 +1,4 @@
1# Copyright 2009-2017 Canonical Ltd. This software is licensed under the1# Copyright 2009-2020 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4"""BinaryPackageBuild interfaces."""4"""BinaryPackageBuild interfaces."""
@@ -179,8 +179,8 @@ class IBinaryPackageBuildView(IPackageBuild):
179 component, section, priority, installedsize, architecturespecific,179 component, section, priority, installedsize, architecturespecific,
180 shlibdeps=None, depends=None, recommends=None, suggests=None,180 shlibdeps=None, depends=None, recommends=None, suggests=None,
181 conflicts=None, replaces=None, provides=None, pre_depends=None,181 conflicts=None, replaces=None, provides=None, pre_depends=None,
182 enhances=None, breaks=None, essential=False, debug_package=None,182 enhances=None, breaks=None, built_using=None, essential=False,
183 user_defined_fields=None, homepage=None):183 debug_package=None, user_defined_fields=None, homepage=None):
184 """Create and return a `BinaryPackageRelease`.184 """Create and return a `BinaryPackageRelease`.
185185
186 The binarypackagerelease will be attached to this specific build.186 The binarypackagerelease will be attached to this specific build.
diff --git a/lib/lp/soyuz/interfaces/binarypackagerelease.py b/lib/lp/soyuz/interfaces/binarypackagerelease.py
index 32fe93c..c69a23d 100644
--- a/lib/lp/soyuz/interfaces/binarypackagerelease.py
+++ b/lib/lp/soyuz/interfaces/binarypackagerelease.py
@@ -1,4 +1,4 @@
1# Copyright 2009-2012 Canonical Ltd. This software is licensed under the1# Copyright 2009-2020 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4"""Binary package release interfaces."""4"""Binary package release interfaces."""
@@ -61,6 +61,11 @@ class IBinaryPackageRelease(Interface):
61 pre_depends = TextLine(required=False)61 pre_depends = TextLine(required=False)
62 enhances = TextLine(required=False)62 enhances = TextLine(required=False)
63 breaks = TextLine(required=False)63 breaks = TextLine(required=False)
64 built_using_references = List(
65 title=_("Sequence of Built-Using references."),
66 # Really IBinarySourceReference.
67 value_type=Reference(schema=Interface),
68 required=True)
64 essential = Bool(required=False)69 essential = Bool(required=False)
65 installedsize = Int(required=False)70 installedsize = Int(required=False)
66 architecturespecific = Bool(required=True)71 architecturespecific = Bool(required=True)
diff --git a/lib/lp/soyuz/interfaces/binarysourcereference.py b/lib/lp/soyuz/interfaces/binarysourcereference.py
67new file mode 10064472new file mode 100644
index 0000000..dc21765
--- /dev/null
+++ b/lib/lp/soyuz/interfaces/binarysourcereference.py
@@ -0,0 +1,86 @@
1# Copyright 2020 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""Interface for references from binary packages to source packages."""
5
6from __future__ import absolute_import, print_function, unicode_literals
7
8__metaclass__ = type
9__all__ = [
10 'IBinarySourceReference',
11 'IBinarySourceReferenceSet',
12 'UnparsableBuiltUsing',
13 ]
14
15from lazr.restful.fields import Reference
16from zope.interface import Interface
17from zope.schema import (
18 Choice,
19 Int,
20 )
21
22from lp import _
23from lp.soyuz.enums import BinarySourceReferenceType
24from lp.soyuz.interfaces.binarypackagerelease import IBinaryPackageRelease
25from lp.soyuz.interfaces.sourcepackagerelease import ISourcePackageRelease
26
27
28class UnparsableBuiltUsing(Exception):
29 """A Built-Using field could not be parsed."""
30
31
32class IBinarySourceReference(Interface):
33 """A reference from a binary package to a source package."""
34
35 id = Int(title=_("ID"))
36
37 binary_package_release = Reference(
38 IBinaryPackageRelease,
39 title=_("The referencing binary package release."),
40 required=True, readonly=True)
41 source_package_release = Reference(
42 ISourcePackageRelease,
43 title=_("The referenced source package release."),
44 required=True, readonly=True)
45 reference_type = Choice(
46 title=_("The type of the reference."),
47 vocabulary=BinarySourceReferenceType,
48 required=True, readonly=True)
49
50
51class IBinarySourceReferenceSet(Interface):
52 """A set of references from binary packages to source packages."""
53
54 def createFromRelationship(bpr, relationship, reference_type):
55 """Create references from a text relationship field.
56
57 :param bpr: The `IBinaryPackageRelease` from which new references
58 should be created.
59 :param relationship: A text relationship field containing one or
60 more source package relations in the usual Debian encoding (e.g.
61 "source1 (= 1.0), source2 (= 2.0)").
62 :param reference_type: The `BinarySourceReferenceType` of references
63 to create.
64 :return: A list of new `IBinarySourceReference`s.
65 """
66
67 def createFromSourcePackageReleases(bpr, sprs, reference_type):
68 """Create references from a sequence of source package releases.
69
70 This is a convenience method for use in tests.
71
72 :param bpr: The `IBinaryPackageRelease` from which new references
73 should be created.
74 :param sprs: A sequence of `ISourcePackageRelease`s.
75 :param reference_type: The `BinarySourceReferenceType` of references
76 to create.
77 :return: A list of new `IBinarySourceReference`s.
78 """
79
80 def findByBinaryPackageRelease(bpr, reference_type):
81 """Find references from a given binary package release.
82
83 :param bpr: An `IBinaryPackageRelease` to search for.
84 :param reference_type: A `BinarySourceReferenceType` to search for.
85 :return: A `ResultSet` of matching `IBinarySourceReference`s.
86 """
diff --git a/lib/lp/soyuz/model/binarypackagebuild.py b/lib/lp/soyuz/model/binarypackagebuild.py
index ae4f1de..470a8f2 100644
--- a/lib/lp/soyuz/model/binarypackagebuild.py
+++ b/lib/lp/soyuz/model/binarypackagebuild.py
@@ -1,4 +1,4 @@
1# Copyright 2009-2019 Canonical Ltd. This software is licensed under the1# Copyright 2009-2020 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4__metaclass__ = type4__metaclass__ = type
@@ -91,6 +91,7 @@ from lp.services.macaroons.model import MacaroonIssuerBase
91from lp.soyuz.adapters.buildarch import determine_architectures_to_build91from lp.soyuz.adapters.buildarch import determine_architectures_to_build
92from lp.soyuz.enums import (92from lp.soyuz.enums import (
93 ArchivePurpose,93 ArchivePurpose,
94 BinarySourceReferenceType,
94 PackagePublishingStatus,95 PackagePublishingStatus,
95 )96 )
96from lp.soyuz.interfaces.archive import (97from lp.soyuz.interfaces.archive import (
@@ -104,6 +105,9 @@ from lp.soyuz.interfaces.binarypackagebuild import (
104 IBinaryPackageBuildSet,105 IBinaryPackageBuildSet,
105 UnparsableDependencies,106 UnparsableDependencies,
106 )107 )
108from lp.soyuz.interfaces.binarysourcereference import (
109 IBinarySourceReferenceSet,
110 )
107from lp.soyuz.interfaces.distroarchseries import IDistroArchSeries111from lp.soyuz.interfaces.distroarchseries import IDistroArchSeries
108from lp.soyuz.interfaces.packageset import IPackagesetSet112from lp.soyuz.interfaces.packageset import IPackagesetSet
109from lp.soyuz.mail.binarypackagebuild import BinaryPackageBuildMailer113from lp.soyuz.mail.binarypackagebuild import BinaryPackageBuildMailer
@@ -662,10 +666,11 @@ class BinaryPackageBuild(PackageBuildMixin, SQLBase):
662 binpackageformat, component, section, priority, installedsize,666 binpackageformat, component, section, priority, installedsize,
663 architecturespecific, shlibdeps=None, depends=None, recommends=None,667 architecturespecific, shlibdeps=None, depends=None, recommends=None,
664 suggests=None, conflicts=None, replaces=None, provides=None,668 suggests=None, conflicts=None, replaces=None, provides=None,
665 pre_depends=None, enhances=None, breaks=None, essential=False,669 pre_depends=None, enhances=None, breaks=None, built_using=None,
666 debug_package=None, user_defined_fields=None, homepage=None):670 essential=False, debug_package=None, user_defined_fields=None,
671 homepage=None):
667 """See IBuild."""672 """See IBuild."""
668 return BinaryPackageRelease(673 bpr = BinaryPackageRelease(
669 build=self, binarypackagename=binarypackagename, version=version,674 build=self, binarypackagename=binarypackagename, version=version,
670 summary=summary, description=description,675 summary=summary, description=description,
671 binpackageformat=binpackageformat,676 binpackageformat=binpackageformat,
@@ -677,6 +682,10 @@ class BinaryPackageBuild(PackageBuildMixin, SQLBase):
677 architecturespecific=architecturespecific,682 architecturespecific=architecturespecific,
678 debug_package=debug_package,683 debug_package=debug_package,
679 user_defined_fields=user_defined_fields, homepage=homepage)684 user_defined_fields=user_defined_fields, homepage=homepage)
685 if built_using:
686 getUtility(IBinarySourceReferenceSet).createFromRelationship(
687 bpr, built_using, BinarySourceReferenceType.BUILT_USING)
688 return bpr
680689
681 def estimateDuration(self):690 def estimateDuration(self):
682 """See `IPackageBuild`."""691 """See `IPackageBuild`."""
diff --git a/lib/lp/soyuz/model/binarypackagerelease.py b/lib/lp/soyuz/model/binarypackagerelease.py
index ad4f514..3458b70 100644
--- a/lib/lp/soyuz/model/binarypackagerelease.py
+++ b/lib/lp/soyuz/model/binarypackagerelease.py
@@ -1,4 +1,4 @@
1# Copyright 2009-2012 Canonical Ltd. This software is licensed under the1# Copyright 2009-2020 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4__metaclass__ = type4__metaclass__ = type
@@ -7,6 +7,7 @@ __all__ = [
7 'BinaryPackageReleaseDownloadCount',7 'BinaryPackageReleaseDownloadCount',
8 ]8 ]
99
10from operator import attrgetter
1011
11import simplejson12import simplejson
12from sqlobject import (13from sqlobject import (
@@ -22,6 +23,7 @@ from storm.locals import (
22 Store,23 Store,
23 Storm,24 Storm,
24 )25 )
26from zope.component import getUtility
25from zope.interface import implementer27from zope.interface import implementer
2628
27from lp.services.database.constants import UTC_NOW29from lp.services.database.constants import UTC_NOW
@@ -35,12 +37,16 @@ from lp.services.propertycache import (
35from lp.soyuz.enums import (37from lp.soyuz.enums import (
36 BinaryPackageFileType,38 BinaryPackageFileType,
37 BinaryPackageFormat,39 BinaryPackageFormat,
40 BinarySourceReferenceType,
38 PackagePublishingPriority,41 PackagePublishingPriority,
39 )42 )
40from lp.soyuz.interfaces.binarypackagerelease import (43from lp.soyuz.interfaces.binarypackagerelease import (
41 IBinaryPackageRelease,44 IBinaryPackageRelease,
42 IBinaryPackageReleaseDownloadCount,45 IBinaryPackageReleaseDownloadCount,
43 )46 )
47from lp.soyuz.interfaces.binarysourcereference import (
48 IBinarySourceReferenceSet,
49 )
44from lp.soyuz.model.files import BinaryPackageFile50from lp.soyuz.model.files import BinaryPackageFile
4551
4652
@@ -89,6 +95,14 @@ class BinaryPackageRelease(SQLBase):
89 del kwargs['user_defined_fields']95 del kwargs['user_defined_fields']
90 super(BinaryPackageRelease, self).__init__(*args, **kwargs)96 super(BinaryPackageRelease, self).__init__(*args, **kwargs)
9197
98 @cachedproperty
99 def built_using_references(self):
100 reference_set = getUtility(IBinarySourceReferenceSet)
101 references = reference_set.findByBinaryPackageRelease(
102 self, BinarySourceReferenceType.BUILT_USING)
103 # Preserving insertion order is good enough.
104 return sorted(references, key=attrgetter('id'))
105
92 @property106 @property
93 def user_defined_fields(self):107 def user_defined_fields(self):
94 """See `IBinaryPackageRelease`."""108 """See `IBinaryPackageRelease`."""
diff --git a/lib/lp/soyuz/model/binarysourcereference.py b/lib/lp/soyuz/model/binarysourcereference.py
95new file mode 100644109new file mode 100644
index 0000000..adcfb6f
--- /dev/null
+++ b/lib/lp/soyuz/model/binarysourcereference.py
@@ -0,0 +1,149 @@
1# Copyright 2020 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""References from binary packages to source packages."""
5
6from __future__ import absolute_import, print_function, unicode_literals
7
8__metaclass__ = type
9__all__ = [
10 'BinarySourceReference',
11 'BinarySourceReferenceSet',
12 ]
13
14import warnings
15
16from debian.deb822 import PkgRelation
17from storm.locals import (
18 Int,
19 Reference,
20 )
21from zope.interface import implementer
22
23from lp.services.database.bulk import create
24from lp.services.database.enumcol import DBEnum
25from lp.services.database.interfaces import IStore
26from lp.services.database.stormbase import StormBase
27from lp.soyuz.adapters.archivedependencies import pocket_dependencies
28from lp.soyuz.enums import BinarySourceReferenceType
29from lp.soyuz.interfaces.binarysourcereference import (
30 IBinarySourceReference,
31 IBinarySourceReferenceSet,
32 UnparsableBuiltUsing,
33 )
34
35
36@implementer(IBinarySourceReference)
37class BinarySourceReference(StormBase):
38 """See `IBinarySourceReference`."""
39
40 __storm_table__ = "BinarySourceReference"
41
42 id = Int(primary=True)
43
44 binary_package_release_id = Int(
45 name="binary_package_release", allow_none=False)
46 binary_package_release = Reference(
47 binary_package_release_id, "BinaryPackageRelease.id")
48
49 source_package_release_id = Int(
50 name="source_package_release", allow_none=False)
51 source_package_release = Reference(
52 source_package_release_id, "SourcePackageRelease.id")
53
54 reference_type = DBEnum(enum=BinarySourceReferenceType, allow_none=False)
55
56 def __init__(self, binary_package_release, source_package_release,
57 reference_type):
58 """Construct a `BinarySourceReference`."""
59 super(BinarySourceReference, self).__init__()
60 self.binary_package_release = binary_package_release
61 self.source_package_release = source_package_release
62 self.reference_type = reference_type
63
64
65@implementer(IBinarySourceReferenceSet)
66class BinarySourceReferenceSet:
67 """See `IBinarySourceReferenceSet`."""
68
69 @classmethod
70 def createFromRelationship(cls, bpr, relationship, reference_type):
71 """See `IBinarySourceReferenceSet`."""
72 if not relationship:
73 return []
74
75 try:
76 with warnings.catch_warnings():
77 warnings.simplefilter("error")
78 parsed_rel = PkgRelation.parse_relations(relationship)
79 except Warning as error:
80 raise UnparsableBuiltUsing(
81 "Invalid Built-Using field; cannot be parsed by deb822: %s"
82 % (error,))
83
84 build = bpr.build
85 values = []
86 for or_rel in parsed_rel:
87 if len(or_rel) != 1:
88 raise UnparsableBuiltUsing(
89 "Alternatives are not allowed in Built-Using field: %s"
90 % (PkgRelation.str([or_rel]),))
91 rel = or_rel[0]
92 if rel["version"] is None or rel["version"][0] != "=":
93 raise UnparsableBuiltUsing(
94 "Built-Using must contain strict dependencies: %s"
95 % (PkgRelation.str([or_rel]),))
96
97 # "source-package-name (= version)" might refer to any of
98 # several SPRs, for example if the same source package was
99 # uploaded to a PPA and then uploaded separately (not copied -
100 # copies reuse the same SPR) to the distribution's primary
101 # archive. We need to disambiguate this and find an actual SPR
102 # so that we can efficiently look up references for a given
103 # source publication.
104 #
105 # However, allowing cross-archive references would make the
106 # dominator's job much harder and have other undesirable
107 # properties, such as being able to pin a source in Published in
108 # a foreign archive just by adding it as a dependency and
109 # declaring a Built-Using relationship on it.
110 #
111 # Therefore, as a safe approximation, try this build's pocket
112 # dependencies within its archive and series. Within this
113 # constraint, a name and version should uniquely identify an
114 # SPR, although we pick the latest by ID just in case that
115 # somehow ends up not being true.
116 closest_spph = build.archive.getPublishedSources(
117 name=rel["name"], version=rel["version"][1],
118 distroseries=build.distro_series,
119 pocket=pocket_dependencies[build.pocket],
120 exact_match=True).first()
121 if closest_spph is None:
122 raise UnparsableBuiltUsing(
123 "Built-Using refers to source package %s (= %s), which is "
124 "not known in %s in %s" %
125 (rel["name"], rel["version"][1],
126 build.distro_series.name, build.archive.reference))
127 values.append(
128 (bpr.id, closest_spph.sourcepackagereleaseID, reference_type))
129
130 return create(
131 (BinarySourceReference.binary_package_release_id,
132 BinarySourceReference.source_package_release_id,
133 BinarySourceReference.reference_type),
134 values, get_objects=True)
135
136 @classmethod
137 def createFromSourcePackageReleases(cls, bpr, sprs, reference_type):
138 """See `IBinarySourceReferenceSet`."""
139 relationship = ", ".join(
140 ["%s (= %s)" % (spr.name, spr.version) for spr in sprs])
141 return cls.createFromRelationship(bpr, relationship, reference_type)
142
143 @classmethod
144 def findByBinaryPackageRelease(cls, bpr, reference_type):
145 """See `IBinarySourceReferenceSet`."""
146 return IStore(BinarySourceReference).find(
147 BinarySourceReference,
148 BinarySourceReference.binary_package_release == bpr,
149 BinarySourceReference.reference_type == reference_type)
diff --git a/lib/lp/soyuz/scripts/gina/handlers.py b/lib/lp/soyuz/scripts/gina/handlers.py
index 66d2aeb..54539a4 100644
--- a/lib/lp/soyuz/scripts/gina/handlers.py
+++ b/lib/lp/soyuz/scripts/gina/handlers.py
@@ -1,4 +1,4 @@
1# Copyright 2009-2016 Canonical Ltd. This software is licensed under the1# Copyright 2009-2020 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4"""Gina db handlers.4"""Gina db handlers.
@@ -52,10 +52,15 @@ from lp.services.librarian.interfaces import ILibraryFileAliasSet
52from lp.services.scripts import log52from lp.services.scripts import log
53from lp.soyuz.enums import (53from lp.soyuz.enums import (
54 BinaryPackageFormat,54 BinaryPackageFormat,
55 BinarySourceReferenceType,
55 PackagePublishingStatus,56 PackagePublishingStatus,
56 )57 )
57from lp.soyuz.interfaces.binarypackagebuild import IBinaryPackageBuildSet58from lp.soyuz.interfaces.binarypackagebuild import IBinaryPackageBuildSet
58from lp.soyuz.interfaces.binarypackagename import IBinaryPackageNameSet59from lp.soyuz.interfaces.binarypackagename import IBinaryPackageNameSet
60from lp.soyuz.interfaces.binarysourcereference import (
61 IBinarySourceReferenceSet,
62 UnparsableBuiltUsing,
63 )
59from lp.soyuz.interfaces.publishing import (64from lp.soyuz.interfaces.publishing import (
60 active_publishing_status,65 active_publishing_status,
61 IPublishingSet,66 IPublishingSet,
@@ -820,6 +825,16 @@ class BinaryPackageHandler:
820 installedsize=bin.installed_size,825 installedsize=bin.installed_size,
821 architecturespecific=architecturespecific,826 architecturespecific=architecturespecific,
822 **kwargs)827 **kwargs)
828 try:
829 getUtility(IBinarySourceReferenceSet).createFromRelationship(
830 binpkg, bin.built_using, BinarySourceReferenceType.BUILT_USING)
831 except UnparsableBuiltUsing:
832 # XXX cjwatson 2020-02-03: It might be nice if we created
833 # BinarySourceReference rows at least for those relations that
834 # can be parsed and resolved to SourcePackageReleases. It's not
835 # worth spending much time on given that we don't use binary
836 # imports much, though.
837 pass
823 log.info('Binary Package Release %s (%s) created' %838 log.info('Binary Package Release %s (%s) created' %
824 (bin_name.name, bin.version))839 (bin_name.name, bin.version))
825840
diff --git a/lib/lp/soyuz/scripts/gina/packages.py b/lib/lp/soyuz/scripts/gina/packages.py
index 4126b80..6e2f090 100644
--- a/lib/lp/soyuz/scripts/gina/packages.py
+++ b/lib/lp/soyuz/scripts/gina/packages.py
@@ -1,4 +1,4 @@
1# Copyright 2009-2019 Canonical Ltd. This software is licensed under the1# Copyright 2009-2020 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4"""Package information classes.4"""Package information classes.
@@ -466,6 +466,7 @@ class BinaryPackageData(AbstractPackageData):
466 pre_depends = ""466 pre_depends = ""
467 enhances = ""467 enhances = ""
468 breaks = ""468 breaks = ""
469 built_using = ""
469 essential = False470 essential = False
470471
471 # Overwritten in do_package, optionally472 # Overwritten in do_package, optionally
@@ -493,6 +494,14 @@ class BinaryPackageData(AbstractPackageData):
493 except ValueError:494 except ValueError:
494 raise MissingRequiredArguments("Installed-Size is "495 raise MissingRequiredArguments("Installed-Size is "
495 "not a valid integer: %r" % v)496 "not a valid integer: %r" % v)
497 elif k == "Built-Using":
498 self.built_using = v
499 # Preserve the original form of Built-Using to avoid
500 # possible unfortunate apt behaviour. This is most easily
501 # done by adding it to _user_defined as well.
502 if self._user_defined is None:
503 self._user_defined = []
504 self._user_defined.append([k, v])
496 else:505 else:
497 self.set_field(k, v)506 self.set_field(k, v)
498507
diff --git a/lib/lp/soyuz/scripts/tests/test_gina.py b/lib/lp/soyuz/scripts/tests/test_gina.py
index 2653b4f..9574c38 100644
--- a/lib/lp/soyuz/scripts/tests/test_gina.py
+++ b/lib/lp/soyuz/scripts/tests/test_gina.py
@@ -1,4 +1,4 @@
1# Copyright 2009-2019 Canonical Ltd. This software is licensed under the1# Copyright 2009-2020 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).2# GNU Affero General Public License version 3 (see the file LICENSE).
33
4from doctest import DocTestSuite4from doctest import DocTestSuite
@@ -12,6 +12,10 @@ from unittest import TestLoader
1212
13import apt_pkg13import apt_pkg
14from fixtures import EnvironmentVariableFixture14from fixtures import EnvironmentVariableFixture
15from testtools.matchers import (
16 MatchesSetwise,
17 MatchesStructure,
18 )
15import transaction19import transaction
1620
17from lp.archiveuploader.tagfiles import parse_tagfile21from lp.archiveuploader.tagfiles import parse_tagfile
@@ -22,7 +26,10 @@ from lp.services.features.testing import FeatureFixture
22from lp.services.log.logger import DevNullLogger26from lp.services.log.logger import DevNullLogger
23from lp.services.osutils import write_file27from lp.services.osutils import write_file
24from lp.services.tarfile_helpers import LaunchpadWriteTarFile28from lp.services.tarfile_helpers import LaunchpadWriteTarFile
25from lp.soyuz.enums import PackagePublishingStatus29from lp.soyuz.enums import (
30 BinarySourceReferenceType,
31 PackagePublishingStatus,
32 )
26from lp.soyuz.scripts.gina import ExecutionError33from lp.soyuz.scripts.gina import ExecutionError
27from lp.soyuz.scripts.gina.archive import (34from lp.soyuz.scripts.gina.archive import (
28 ArchiveComponentItems,35 ArchiveComponentItems,
@@ -483,9 +490,62 @@ class TestBinaryPackageHandler(TestCaseWithFactory):
483 "Summary": "",490 "Summary": "",
484 "Priority": "extra",491 "Priority": "extra",
485 "Python-Version": "2.7",492 "Python-Version": "2.7",
493 "Built-Using": "nonexistent (= 0.1)",
494 }
495 bp_data = BinaryPackageData(**deb_contents)
496 self.assertContentEqual(
497 [["Python-Version", "2.7"],
498 ["Built-Using", "nonexistent (= 0.1)"]],
499 bp_data._user_defined)
500 bp_data.archive_root = archive_root
501 # We don't need a real .deb here.
502 write_file(
503 os.path.join(archive_root, "pool/main/f/foo/foo_1.0-1_amd64.deb"),
504 b"x")
505 bpr = bphandler.createBinaryPackage(bp_data, spr, das, "amd64")
506 self.assertIsNotNone(bpr)
507 self.assertEqual([], bpr.built_using_references)
508 self.assertContentEqual(
509 [["Python-Version", "2.7"],
510 ["Built-Using", "nonexistent (= 0.1)"]],
511 bpr.user_defined_fields)
512
513 def test_resolvable_built_using(self):
514 das = self.factory.makeDistroArchSeries()
515 archive_root = self.useTempDir()
516 sphandler = SourcePackageHandler(
517 das.distroseries.distribution.name, archive_root,
518 PackagePublishingPocket.RELEASE, None)
519 bphandler = BinaryPackageHandler(
520 sphandler, archive_root, PackagePublishingPocket.RELEASE)
521 spr = self.factory.makeSourcePackagePublishingHistory(
522 distroseries=das.distroseries,
523 component="main").sourcepackagerelease
524 built_using_spph = self.factory.makeSourcePackagePublishingHistory(
525 archive=das.main_archive, distroseries=das.distroseries,
526 pocket=PackagePublishingPocket.RELEASE)
527 built_using_spr = built_using_spph.sourcepackagerelease
528 built_using_relationship = "%s (= %s)" % (
529 built_using_spr.name, built_using_spr.version)
530 deb_contents = {
531 "Package": "foo",
532 "Installed-Size": "0",
533 "Maintainer": "Foo Bar <foo@canonical.com>",
534 "Section": "misc",
535 "Architecture": "amd64",
536 "Version": "1.0-1",
537 "Filename": "pool/main/f/foo/foo_1.0-1_amd64.deb",
538 "Component": "main",
539 "Size": "0",
540 "MD5sum": "0" * 32,
541 "Description": "",
542 "Summary": "",
543 "Priority": "extra",
544 "Built-Using": built_using_relationship,
486 }545 }
487 bp_data = BinaryPackageData(**deb_contents)546 bp_data = BinaryPackageData(**deb_contents)
488 self.assertEqual([["Python-Version", "2.7"]], bp_data._user_defined)547 self.assertContentEqual(
548 [["Built-Using", built_using_relationship]], bp_data._user_defined)
489 bp_data.archive_root = archive_root549 bp_data.archive_root = archive_root
490 # We don't need a real .deb here.550 # We don't need a real .deb here.
491 write_file(551 write_file(
@@ -493,7 +553,16 @@ class TestBinaryPackageHandler(TestCaseWithFactory):
493 b"x")553 b"x")
494 bpr = bphandler.createBinaryPackage(bp_data, spr, das, "amd64")554 bpr = bphandler.createBinaryPackage(bp_data, spr, das, "amd64")
495 self.assertIsNotNone(bpr)555 self.assertIsNotNone(bpr)
496 self.assertEqual([["Python-Version", "2.7"]], bpr.user_defined_fields)556 self.assertThat(
557 bpr.built_using_references,
558 MatchesSetwise(
559 MatchesStructure.byEquality(
560 binary_package_release=bpr,
561 source_package_release=built_using_spr,
562 reference_type=BinarySourceReferenceType.BUILT_USING)))
563 self.assertContentEqual(
564 [["Built-Using", built_using_relationship]],
565 bpr.user_defined_fields)
497566
498567
499class TestBinaryPackagePublisher(TestCaseWithFactory):568class TestBinaryPackagePublisher(TestCaseWithFactory):
diff --git a/lib/lp/soyuz/tests/test_binarysourcereference.py b/lib/lp/soyuz/tests/test_binarysourcereference.py
500new file mode 100644569new file mode 100644
index 0000000..f682de8
--- /dev/null
+++ b/lib/lp/soyuz/tests/test_binarysourcereference.py
@@ -0,0 +1,208 @@
1# Copyright 2020 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""Test references from binary packages to source packages."""
5
6from __future__ import absolute_import, print_function, unicode_literals
7
8__metaclass__ = type
9
10import re
11
12from testtools.matchers import (
13 MatchesSetwise,
14 MatchesStructure,
15 )
16from testtools.testcase import ExpectedException
17from zope.component import getUtility
18
19from lp.registry.interfaces.pocket import PackagePublishingPocket
20from lp.soyuz.enums import (
21 ArchivePurpose,
22 BinarySourceReferenceType,
23 )
24from lp.soyuz.interfaces.binarysourcereference import (
25 IBinarySourceReferenceSet,
26 UnparsableBuiltUsing,
27 )
28from lp.testing import TestCaseWithFactory
29from lp.testing.layers import DatabaseFunctionalLayer
30
31
32class TestBinarySourceReference(TestCaseWithFactory):
33
34 layer = DatabaseFunctionalLayer
35
36 def setUp(self):
37 super(TestBinarySourceReference, self).setUp()
38 self.reference_set = getUtility(IBinarySourceReferenceSet)
39
40 def test_createFromRelationship_empty(self):
41 bpr = self.factory.makeBinaryPackageRelease()
42 self.assertEqual(
43 [],
44 self.reference_set.createFromRelationship(
45 bpr, "", BinarySourceReferenceType.BUILT_USING))
46
47 def test_createFromRelationship_nonsense(self):
48 bpr = self.factory.makeBinaryPackageRelease()
49 expected_message = (
50 r"Invalid Built-Using field; cannot be parsed by deb822: .*")
51 with ExpectedException(UnparsableBuiltUsing, expected_message):
52 self.reference_set.createFromRelationship(
53 bpr, "nonsense (", BinarySourceReferenceType.BUILT_USING)
54
55 def test_createFromRelationship_alternatives(self):
56 bpr = self.factory.makeBinaryPackageRelease()
57 expected_message = (
58 r"Alternatives are not allowed in Built-Using field: "
59 r"foo \(= 1\) \| bar \(= 2\)")
60 with ExpectedException(UnparsableBuiltUsing, expected_message):
61 self.reference_set.createFromRelationship(
62 bpr, "foo (= 1) | bar (= 2)",
63 BinarySourceReferenceType.BUILT_USING)
64
65 def test_createFromRelationship_no_version(self):
66 bpr = self.factory.makeBinaryPackageRelease()
67 expected_message = r"Built-Using must contain strict dependencies: foo"
68 with ExpectedException(UnparsableBuiltUsing, expected_message):
69 self.reference_set.createFromRelationship(
70 bpr, "foo", BinarySourceReferenceType.BUILT_USING)
71
72 def test_createFromRelationship_inequality(self):
73 bpr = self.factory.makeBinaryPackageRelease()
74 expected_message = (
75 r"Built-Using must contain strict dependencies: foo \(>= 1\)")
76 with ExpectedException(UnparsableBuiltUsing, expected_message):
77 self.reference_set.createFromRelationship(
78 bpr, "foo (>= 1)", BinarySourceReferenceType.BUILT_USING)
79
80 def test_createFromRelationship_unknown_source_package_name(self):
81 bpr = self.factory.makeBinaryPackageRelease()
82 relationship = "nonexistent (= 1)"
83 expected_message = (
84 r"Built-Using refers to source package %s, which is not known in "
85 r"%s in %s" % (
86 re.escape(relationship), bpr.build.distro_series.name,
87 bpr.build.archive.reference))
88 with ExpectedException(UnparsableBuiltUsing, expected_message):
89 self.reference_set.createFromRelationship(
90 bpr, relationship, BinarySourceReferenceType.BUILT_USING)
91
92 def test_createFromRelationship_unknown_source_package_version(self):
93 bpr = self.factory.makeBinaryPackageRelease()
94 spph = self.factory.makeSourcePackagePublishingHistory(
95 archive=bpr.build.archive,
96 distroseries=bpr.build.distro_series,
97 component=bpr.build.current_component)
98 spr = spph.sourcepackagerelease
99 relationship = "%s (= %s.1)" % (spr.name, spr.version)
100 expected_message = (
101 r"Built-Using refers to source package %s, which is not known in "
102 r"%s in %s" % (
103 re.escape(relationship), bpr.build.distro_series.name,
104 bpr.build.archive.reference))
105 with ExpectedException(UnparsableBuiltUsing, expected_message):
106 self.reference_set.createFromRelationship(
107 bpr, relationship, BinarySourceReferenceType.BUILT_USING)
108
109 def test_createFromRelationship_simple(self):
110 bpr = self.factory.makeBinaryPackageRelease()
111 spphs = [
112 self.factory.makeSourcePackagePublishingHistory(
113 archive=bpr.build.archive,
114 distroseries=bpr.build.distro_series, pocket=bpr.build.pocket)
115 for _ in range(3)]
116 sprs = [spph.sourcepackagerelease for spph in spphs]
117 # Create a few more SPPHs with slight mismatches to ensure that
118 # createFromRelationship matches correctly.
119 self.factory.makeSourcePackagePublishingHistory(
120 archive=bpr.build.archive, pocket=bpr.build.pocket,
121 sourcepackagename=sprs[0].name, version=sprs[0].version)
122 self.factory.makeSourcePackagePublishingHistory(
123 archive=bpr.build.archive, distroseries=bpr.build.distro_series,
124 pocket=PackagePublishingPocket.BACKPORTS,
125 sourcepackagename=sprs[0].name, version=sprs[0].version)
126 self.factory.makeSourcePackagePublishingHistory(
127 archive=bpr.build.archive, distroseries=bpr.build.distro_series,
128 pocket=bpr.build.pocket, sourcepackagename=sprs[0].name)
129 self.factory.makeSourcePackagePublishingHistory(
130 archive=bpr.build.archive, distroseries=bpr.build.distro_series,
131 pocket=bpr.build.pocket, version=sprs[0].version)
132 self.factory.makeSourcePackagePublishingHistory()
133 relationship = (
134 "%s (= %s), %s (= %s)" %
135 (sprs[0].name, sprs[0].version, sprs[1].name, sprs[1].version))
136 bsrs = self.reference_set.createFromRelationship(
137 bpr, relationship, BinarySourceReferenceType.BUILT_USING)
138 self.assertThat(bsrs, MatchesSetwise(*(
139 MatchesStructure.byEquality(
140 binary_package_release=bpr,
141 source_package_release=spr,
142 reference_type=BinarySourceReferenceType.BUILT_USING)
143 for spr in sprs[:2])))
144
145 def test_createFromRelationship_foreign_archive(self):
146 # createFromRelationship only considers SPRs found in the same
147 # archive as the build.
148 archive = self.factory.makeArchive(purpose=ArchivePurpose.PPA)
149 build = self.factory.makeBinaryPackageBuild(archive=archive)
150 bpr = self.factory.makeBinaryPackageRelease(build=build)
151 spph = self.factory.makeSourcePackagePublishingHistory(
152 archive=build.distro_series.main_archive,
153 distroseries=build.distro_series, pocket=build.pocket)
154 spr = spph.sourcepackagerelease
155 relationship = "%s (= %s)" % (spr.name, spr.version)
156 expected_message = (
157 r"Built-Using refers to source package %s, which is not known in "
158 r"%s in %s" % (
159 re.escape(relationship), build.distro_series.name,
160 build.archive.reference))
161 with ExpectedException(UnparsableBuiltUsing, expected_message):
162 self.reference_set.createFromRelationship(
163 bpr, relationship, BinarySourceReferenceType.BUILT_USING)
164
165 def test_findByBinaryPackageRelease_empty(self):
166 bpr = self.factory.makeBinaryPackageRelease()
167 self.assertContentEqual(
168 [],
169 self.reference_set.findByBinaryPackageRelease(
170 bpr, BinarySourceReferenceType.BUILT_USING))
171
172 def test_findByBinaryPackageRelease(self):
173 bprs = [self.factory.makeBinaryPackageRelease() for _ in range(2)]
174 all_sprs = []
175 for bpr in bprs:
176 spphs = [
177 self.factory.makeSourcePackagePublishingHistory(
178 archive=bpr.build.archive,
179 distroseries=bpr.build.distro_series,
180 pocket=bpr.build.pocket)
181 for _ in range(2)]
182 sprs = [spph.sourcepackagerelease for spph in spphs]
183 all_sprs.extend(sprs)
184 self.reference_set.createFromSourcePackageReleases(
185 bpr, sprs, BinarySourceReferenceType.BUILT_USING)
186 other_bpr = self.factory.makeBinaryPackageRelease()
187 self.assertThat(
188 self.reference_set.findByBinaryPackageRelease(
189 bprs[0], BinarySourceReferenceType.BUILT_USING),
190 MatchesSetwise(*(
191 MatchesStructure.byEquality(
192 binary_package_release=bprs[0],
193 source_package_release=spr,
194 reference_type=BinarySourceReferenceType.BUILT_USING)
195 for spr in all_sprs[:2])))
196 self.assertThat(
197 self.reference_set.findByBinaryPackageRelease(
198 bprs[1], BinarySourceReferenceType.BUILT_USING),
199 MatchesSetwise(*(
200 MatchesStructure.byEquality(
201 binary_package_release=bprs[1],
202 source_package_release=spr,
203 reference_type=BinarySourceReferenceType.BUILT_USING)
204 for spr in all_sprs[2:])))
205 self.assertContentEqual(
206 [],
207 self.reference_set.findByBinaryPackageRelease(
208 other_bpr, BinarySourceReferenceType.BUILT_USING))
diff --git a/lib/lp/soyuz/tests/test_publishing.py b/lib/lp/soyuz/tests/test_publishing.py
index e65003d..cd7d1ea 100644
--- a/lib/lp/soyuz/tests/test_publishing.py
+++ b/lib/lp/soyuz/tests/test_publishing.py
@@ -306,7 +306,8 @@ class SoyuzTestPublisher:
306 shlibdep=None, depends=None, recommends=None,306 shlibdep=None, depends=None, recommends=None,
307 suggests=None, conflicts=None, replaces=None,307 suggests=None, conflicts=None, replaces=None,
308 provides=None, pre_depends=None, enhances=None,308 provides=None, pre_depends=None, enhances=None,
309 breaks=None, filecontent=b'bbbiiinnnaaarrryyy',309 breaks=None, built_using=None,
310 filecontent=b'bbbiiinnnaaarrryyy',
310 changes_file_content=b"Fake: fake changes file",311 changes_file_content=b"Fake: fake changes file",
311 status=PackagePublishingStatus.PENDING,312 status=PackagePublishingStatus.PENDING,
312 pocket=PackagePublishingPocket.RELEASE,313 pocket=PackagePublishingPocket.RELEASE,
@@ -353,7 +354,8 @@ class SoyuzTestPublisher:
353 build, binaryname + '-dbgsym', filecontent, summary,354 build, binaryname + '-dbgsym', filecontent, summary,
354 description, shlibdep, depends, recommends, suggests,355 description, shlibdep, depends, recommends, suggests,
355 conflicts, replaces, provides, pre_depends, enhances,356 conflicts, replaces, provides, pre_depends, enhances,
356 breaks, BinaryPackageFormat.DDEB, version=version)357 breaks, built_using, BinaryPackageFormat.DDEB,
358 version=version)
357 pub_binaries += self.publishBinaryInArchive(359 pub_binaries += self.publishBinaryInArchive(
358 binarypackagerelease_ddeb, archive, status,360 binarypackagerelease_ddeb, archive, status,
359 pocket, scheduleddeletiondate, dateremoved,361 pocket, scheduleddeletiondate, dateremoved,
@@ -364,7 +366,7 @@ class SoyuzTestPublisher:
364 binarypackagerelease = self.uploadBinaryForBuild(366 binarypackagerelease = self.uploadBinaryForBuild(
365 build, binaryname, filecontent, summary, description,367 build, binaryname, filecontent, summary, description,
366 shlibdep, depends, recommends, suggests, conflicts, replaces,368 shlibdep, depends, recommends, suggests, conflicts, replaces,
367 provides, pre_depends, enhances, breaks, format,369 provides, pre_depends, enhances, breaks, built_using, format,
368 binarypackagerelease_ddeb, version=version,370 binarypackagerelease_ddeb, version=version,
369 user_defined_fields=user_defined_fields)371 user_defined_fields=user_defined_fields)
370 pub_binaries += self.publishBinaryInArchive(372 pub_binaries += self.publishBinaryInArchive(
@@ -387,8 +389,9 @@ class SoyuzTestPublisher:
387 summary="summary", description="description", shlibdep=None,389 summary="summary", description="description", shlibdep=None,
388 depends=None, recommends=None, suggests=None, conflicts=None,390 depends=None, recommends=None, suggests=None, conflicts=None,
389 replaces=None, provides=None, pre_depends=None, enhances=None,391 replaces=None, provides=None, pre_depends=None, enhances=None,
390 breaks=None, format=BinaryPackageFormat.DEB, debug_package=None,392 breaks=None, built_using=None, format=BinaryPackageFormat.DEB,
391 user_defined_fields=None, homepage=None, version=None):393 debug_package=None, user_defined_fields=None, homepage=None,
394 version=None):
392 """Return the corresponding `BinaryPackageRelease`."""395 """Return the corresponding `BinaryPackageRelease`."""
393 sourcepackagerelease = build.source_package_release396 sourcepackagerelease = build.source_package_release
394 distroarchseries = build.distro_arch_series397 distroarchseries = build.distro_arch_series
@@ -418,6 +421,7 @@ class SoyuzTestPublisher:
418 pre_depends=pre_depends,421 pre_depends=pre_depends,
419 enhances=enhances,422 enhances=enhances,
420 breaks=breaks,423 breaks=breaks,
424 built_using=built_using,
421 essential=False,425 essential=False,
422 installedsize=100,426 installedsize=100,
423 architecturespecific=architecturespecific,427 architecturespecific=architecturespecific,

Subscribers

People subscribed via source and target branches

to status/vote changes: