Merge ~cjwatson/launchpad:ci-build-upload-make-spr into launchpad:master

Proposed by Colin Watson
Status: Merged
Approved by: Colin Watson
Approved revision: 25428c91bf957de899995044925e87d9a654c618
Merge reported by: Otto Co-Pilot
Merged at revision: not available
Proposed branch: ~cjwatson/launchpad:ci-build-upload-make-spr
Merge into: launchpad:master
Prerequisite: ~cjwatson/launchpad:refactor-ci-build-upload-job-scan-file
Diff against target: 1058 lines (+522/-135)
9 files modified
lib/lp/_schema_circular_imports.py (+1/-0)
lib/lp/archivepublisher/tests/test_artifactory.py (+3/-3)
lib/lp/code/interfaces/cibuild.py (+12/-0)
lib/lp/code/model/cibuild.py (+30/-0)
lib/lp/code/model/tests/test_cibuild.py (+22/-0)
lib/lp/registry/interfaces/sourcepackage.py (+3/-3)
lib/lp/soyuz/interfaces/sourcepackagerelease.py (+12/-3)
lib/lp/soyuz/model/archivejob.py (+154/-46)
lib/lp/soyuz/tests/test_archivejob.py (+285/-80)
Reviewer Review Type Date Requested Status
Andrey Fedoseev (community) Approve
Review via email: mp+426502@code.launchpad.net

Commit message

Create a source publication when uploading a CI build

Description of the change

This has some advantages and disadvantages, but on the whole I think the advantages outweigh the disadvantages.

The downsides are:

 * if the CI build produces nothing that looks like source code at all, then we end up with a slightly odd "source package" with no files attached to it;

 * if builds do produce something that looks like source code (e.g. a Python sdist), then they're on the honour system to either produce it from only one architecture or to produce something equivalent from all architectures.

On the other hand:

 * Launchpad's data model for packages makes the assumption in many places that binary packages have an associated source package, and my attempts to make it tolerate isolated binary packages have only been marginally successful;

 * treating source-like artifacts (Python sdists, zip files of Go source code, etc.) as source packages rather than binary packages makes more sense to humans;

 * source packages provide a useful grouping for collections of binary packages from builds of the same version;

 * creating source publications means that the PPA web UI works for PPAs populated from CI builds with only a few small adjustments (not in this branch);

 * creating source publications means that it should be straightforward to get the existing package copying mechanism to work.

I've slightly cheekily renamed `SourcePackageType.SDIST` to `SourcePackageType.CI_BUILD`. The type of an individual source file is represented separately using `SourcePackageFileType`, and since the data model's assumptions generally involve a single source with zero or more binaries, it's simpler for `SourcePackageRelease.format` not to need to change depending on the types of the individual files it contains. There was previously no way to create a source package with `format=SDIST` outside the test suite, and a query on staging returns zero rows.

As a concrete example of a source package file type and in order to allow more useful testing, I've added support for uploading Python sdists from CI builds as part of this branch.

To post a comment you must log in.
Revision history for this message
Andrey Fedoseev (andrey-fedoseev) wrote :

This looks good.

I'd like the artifact metadata to be a typed structure (NamedTuple) rather than a dict.

review: Approve
Revision history for this message
Colin Watson (cjwatson) wrote :

I find namedtuples awkwardly inflexible in general and usually regret using them; in this case, they aren't really suitable since they don't support optional arguments on construction. I get the general idea, though, so I looked for alternatives. A dataclass would be ideal, but those don't exist until Python 3.7. Type annotations on class attributes don't work in Python 3.6, not even the old comment style unless the attribute has an initializer.

However, `attrs` (https://www.attrs.org/) *does* work, and although Launchpad doesn't currently use it directly, it's already in our virtualenv as an indirect dependency. It's a pretty well-regarded package, it can be integrated with `mypy` (https://www.attrs.org/en/stable/types.html#mypy), it gives us an automatic `__init__` method and such, and it would be easy to convert from a simple use of it to a dataclass once we're on Python >= 3.7.

I've reworked this branch using that. Would you mind having another look?

Revision history for this message
Jürgen Gmach (jugmac00) wrote (last edit ):

I am with Andrey in that we should not use a dictionary.

While attrs is a great library, I am -0.5 on introducing it.

Introducing a new dependency
- which it is only used for two classes
- only for a short interim period
- with a new "syntax"
- which is even dated, ie it is not reflected in the main documentation of the package - you need to dive into its history: https://www.attrs.org/en/stable/names.html#a-short-history-lesson

I am strongly in favor of using plain classes - with that usual boilerplate - and convert them to dataclasses once we are on a suitable Python version.

If you go forward using it, please leave a comment to remove it asap.

Revision history for this message
Andrey Fedoseev (andrey-fedoseev) wrote :

I'm not in favor of using `attrs` either.

I suggested to use named tuples because they can be automatically converted to 3.6 syntax with `pyupgrade` (https://github.com/asottile/pyupgrade#typingnamedtuple--typingtypeddict-py36-syntax) and left as is, or converted to `dataclass` with minimal manual work.

Revision history for this message
Colin Watson (cjwatson) wrote :

I really don't think namedtuples are suitable here because of the lack of support for optional items in their constructors.

For plain classes, how do you propose spelling them? This doesn't work for obvious reasons:

    class SourceArtifactMetadata:
        format # type: SourcePackageFileType
        name # type: str
        version # type: str

And if I use something like `format = None` there, then they all have to be `Optional[...]` when they really shouldn't be optional.

Revision history for this message
Andrey Fedoseev (andrey-fedoseev) wrote :

> I really don't think namedtuples are suitable here because of the lack of support for optional items in their constructors.

You're right, the optional values are available only since 3.7

> For plain classes, how do you propose spelling them?

I think you could use the constructor for that:

    class SourceArtifactMetadata:

        def __init__(self, format: SourcePackageFileType, name: str, version: str):
            self.format = format
            self.name = name
            self.version = version

Revision history for this message
Colin Watson (cjwatson) wrote :

OK, I've converted this to a plain class with `__init__` boilerplate as you suggest. How's this?

Revision history for this message
Andrey Fedoseev (andrey-fedoseev) wrote :

> OK, I've converted this to a plain class with `__init__` boilerplate as you suggest. How's this?

This looks great

PyCharm reports two issues:

1. The signature of `CIBuildUploadJob.create()` doesn't match the base`ArchiveJobDerived.create()` - this may be ok
2. `scanned` argument in `_uploadSources` should probably be a `Sequence` rather than an `Iterable` because you're doing `scanned[0]` there

Revision history for this message
Colin Watson (cjwatson) wrote :

The `create` method thing is at least intentional for now, if possibly not ideal (the whole derived jobs business is a bit unnecessarily complicated, I think).

I've fixed the `scanned` argument type.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/lib/lp/_schema_circular_imports.py b/lib/lp/_schema_circular_imports.py
index e0bd33a..677163b 100644
--- a/lib/lp/_schema_circular_imports.py
+++ b/lib/lp/_schema_circular_imports.py
@@ -335,6 +335,7 @@ patch_reference_property(IPackageUpload, 'copy_source_archive', IArchive)
335patch_reference_property(335patch_reference_property(
336 ISourcePackageRelease, 'source_package_recipe_build',336 ISourcePackageRelease, 'source_package_recipe_build',
337 ISourcePackageRecipeBuild)337 ISourcePackageRecipeBuild)
338patch_reference_property(ISourcePackageRelease, 'ci_build', ICIBuild)
338339
339# IIndexedMessage340# IIndexedMessage
340patch_reference_property(IIndexedMessage, 'inside', IBugTask)341patch_reference_property(IIndexedMessage, 'inside', IBugTask)
diff --git a/lib/lp/archivepublisher/tests/test_artifactory.py b/lib/lp/archivepublisher/tests/test_artifactory.py
index c9feb58..18af683 100644
--- a/lib/lp/archivepublisher/tests/test_artifactory.py
+++ b/lib/lp/archivepublisher/tests/test_artifactory.py
@@ -576,7 +576,7 @@ class TestArtifactoryPoolFromLibrarian(TestCaseWithFactory):
576 sourcepackagename="foo",576 sourcepackagename="foo",
577 version="1.0",577 version="1.0",
578 channel="edge",578 channel="edge",
579 format=SourcePackageType.SDIST,579 format=SourcePackageType.CI_BUILD,
580 )580 )
581 spr = spph.sourcepackagerelease581 spr = spph.sourcepackagerelease
582 sprf = self.factory.makeSourcePackageReleaseFile(582 sprf = self.factory.makeSourcePackageReleaseFile(
@@ -635,7 +635,7 @@ class TestArtifactoryPoolFromLibrarian(TestCaseWithFactory):
635 archive=pool.archive,635 archive=pool.archive,
636 sourcepackagename="foo",636 sourcepackagename="foo",
637 version="1.0",637 version="1.0",
638 format=SourcePackageType.SDIST,638 format=SourcePackageType.CI_BUILD,
639 )639 )
640 bpph = self.factory.makeBinaryPackagePublishingHistory(640 bpph = self.factory.makeBinaryPackagePublishingHistory(
641 archive=pool.archive,641 archive=pool.archive,
@@ -940,7 +940,7 @@ class TestArtifactoryPoolFromLibrarian(TestCaseWithFactory):
940 archive=pool.archive,940 archive=pool.archive,
941 sourcepackagename="foo",941 sourcepackagename="foo",
942 version="1.0",942 version="1.0",
943 format=SourcePackageType.SDIST,943 format=SourcePackageType.CI_BUILD,
944 )944 )
945 bpph = self.factory.makeBinaryPackagePublishingHistory(945 bpph = self.factory.makeBinaryPackagePublishingHistory(
946 archive=pool.archive,946 archive=pool.archive,
diff --git a/lib/lp/code/interfaces/cibuild.py b/lib/lp/code/interfaces/cibuild.py
index 69c765b..d929283 100644
--- a/lib/lp/code/interfaces/cibuild.py
+++ b/lib/lp/code/interfaces/cibuild.py
@@ -133,6 +133,10 @@ class ICIBuildView(IPackageBuildView, IPrivacy):
133 "A mapping from job IDs to result tokens, retrieved from the "133 "A mapping from job IDs to result tokens, retrieved from the "
134 "builder.")))134 "builder.")))
135135
136 sourcepackages = Attribute(
137 "A list of source packages that resulted from this build, ordered by "
138 "name.")
139
136 binarypackages = Attribute(140 binarypackages = Attribute(
137 "A list of binary packages that resulted from this build, ordered by "141 "A list of binary packages that resulted from this build, ordered by "
138 "name.")142 "name.")
@@ -182,6 +186,14 @@ class ICIBuildView(IPackageBuildView, IPrivacy):
182 :return: A collection of URLs for this build.186 :return: A collection of URLs for this build.
183 """187 """
184188
189 def createSourcePackageRelease(
190 distroseries, sourcepackagename, version, creator=None,
191 archive=None):
192 """Create and return a `SourcePackageRelease` for this CI build.
193
194 The new source package release will be linked to this build.
195 """
196
185 def createBinaryPackageRelease(197 def createBinaryPackageRelease(
186 binarypackagename, version, summary, description, binpackageformat,198 binarypackagename, version, summary, description, binpackageformat,
187 architecturespecific, installedsize=None, homepage=None,199 architecturespecific, installedsize=None, homepage=None,
diff --git a/lib/lp/code/model/cibuild.py b/lib/lp/code/model/cibuild.py
index 76ccfc3..281c3a1 100644
--- a/lib/lp/code/model/cibuild.py
+++ b/lib/lp/code/model/cibuild.py
@@ -65,8 +65,10 @@ from lp.code.model.gitref import GitRef
65from lp.code.model.lpcraft import load_configuration65from lp.code.model.lpcraft import load_configuration
66from lp.registry.interfaces.pocket import PackagePublishingPocket66from lp.registry.interfaces.pocket import PackagePublishingPocket
67from lp.registry.interfaces.series import SeriesStatus67from lp.registry.interfaces.series import SeriesStatus
68from lp.registry.interfaces.sourcepackage import SourcePackageType
68from lp.registry.model.distribution import Distribution69from lp.registry.model.distribution import Distribution
69from lp.registry.model.distroseries import DistroSeries70from lp.registry.model.distroseries import DistroSeries
71from lp.registry.model.sourcepackagename import SourcePackageName
70from lp.services.database.bulk import load_related72from lp.services.database.bulk import load_related
71from lp.services.database.constants import DEFAULT73from lp.services.database.constants import DEFAULT
72from lp.services.database.decoratedresultset import DecoratedResultSet74from lp.services.database.decoratedresultset import DecoratedResultSet
@@ -91,6 +93,7 @@ from lp.services.propertycache import cachedproperty
91from lp.soyuz.model.binarypackagename import BinaryPackageName93from lp.soyuz.model.binarypackagename import BinaryPackageName
92from lp.soyuz.model.binarypackagerelease import BinaryPackageRelease94from lp.soyuz.model.binarypackagerelease import BinaryPackageRelease
93from lp.soyuz.model.distroarchseries import DistroArchSeries95from lp.soyuz.model.distroarchseries import DistroArchSeries
96from lp.soyuz.model.sourcepackagerelease import SourcePackageRelease
9497
9598
96def get_stages(configuration):99def get_stages(configuration):
@@ -471,6 +474,17 @@ class CIBuild(PackageBuildMixin, StormBase):
471 # We don't currently send any notifications.474 # We don't currently send any notifications.
472475
473 @property476 @property
477 def sourcepackages(self):
478 """See `ICIBuild`."""
479 releases = IStore(SourcePackageRelease).find(
480 (SourcePackageRelease, SourcePackageName),
481 SourcePackageRelease.ci_build == self,
482 SourcePackageRelease.sourcepackagename == SourcePackageName.id)
483 releases = releases.order_by(
484 SourcePackageName.name, SourcePackageRelease.id)
485 return DecoratedResultSet(releases, result_decorator=itemgetter(0))
486
487 @property
474 def binarypackages(self):488 def binarypackages(self):
475 """See `ICIBuild`."""489 """See `ICIBuild`."""
476 releases = IStore(BinaryPackageRelease).find(490 releases = IStore(BinaryPackageRelease).find(
@@ -481,6 +495,22 @@ class CIBuild(PackageBuildMixin, StormBase):
481 BinaryPackageName.name, BinaryPackageRelease.id)495 BinaryPackageName.name, BinaryPackageRelease.id)
482 return DecoratedResultSet(releases, result_decorator=itemgetter(0))496 return DecoratedResultSet(releases, result_decorator=itemgetter(0))
483497
498 def createSourcePackageRelease(
499 self, distroseries, sourcepackagename, version, creator=None,
500 archive=None):
501 """See `ICIBuild`."""
502 return distroseries.createUploadedSourcePackageRelease(
503 sourcepackagename=sourcepackagename,
504 version=version,
505 format=SourcePackageType.CI_BUILD,
506 # This doesn't really make sense for SPRs created for CI builds,
507 # but the column is NOT NULL. The empty string will do though,
508 # since nothing will use this.
509 architecturehintlist="",
510 creator=creator,
511 archive=archive,
512 ci_build=self)
513
484 def createBinaryPackageRelease(514 def createBinaryPackageRelease(
485 self, binarypackagename, version, summary, description,515 self, binarypackagename, version, summary, description,
486 binpackageformat, architecturespecific, installedsize=None,516 binpackageformat, architecturespecific, installedsize=None,
diff --git a/lib/lp/code/model/tests/test_cibuild.py b/lib/lp/code/model/tests/test_cibuild.py
index 89a432c..ca0192b 100644
--- a/lib/lp/code/model/tests/test_cibuild.py
+++ b/lib/lp/code/model/tests/test_cibuild.py
@@ -62,6 +62,7 @@ from lp.code.model.cibuild import (
62from lp.code.model.lpcraft import load_configuration62from lp.code.model.lpcraft import load_configuration
63from lp.code.tests.helpers import GitHostingFixture63from lp.code.tests.helpers import GitHostingFixture
64from lp.registry.interfaces.series import SeriesStatus64from lp.registry.interfaces.series import SeriesStatus
65from lp.registry.interfaces.sourcepackage import SourcePackageType
65from lp.services.authserver.xmlrpc import AuthServerAPIView66from lp.services.authserver.xmlrpc import AuthServerAPIView
66from lp.services.config import config67from lp.services.config import config
67from lp.services.librarian.browser import ProxiedLibraryFileAlias68from lp.services.librarian.browser import ProxiedLibraryFileAlias
@@ -400,6 +401,27 @@ class TestCIBuild(TestCaseWithFactory):
400 commit_sha1=build.commit_sha1,401 commit_sha1=build.commit_sha1,
401 ci_build=build))402 ci_build=build))
402403
404 def test_createSourcePackageRelease(self):
405 distroseries = self.factory.makeDistroSeries()
406 archive = self.factory.makeArchive(
407 distribution=distroseries.distribution)
408 build = self.factory.makeCIBuild()
409 spn = self.factory.makeSourcePackageName()
410 spr = build.createSourcePackageRelease(
411 distroseries, spn, "1.0", creator=build.git_repository.owner,
412 archive=archive)
413 self.assertThat(spr, MatchesStructure(
414 upload_distroseries=Equals(distroseries),
415 sourcepackagename=Equals(spn),
416 version=Equals("1.0"),
417 format=Equals(SourcePackageType.CI_BUILD),
418 architecturehintlist=Equals(""),
419 creator=Equals(build.git_repository.owner),
420 upload_archive=Equals(archive),
421 ci_build=Equals(build),
422 ))
423 self.assertContentEqual([spr], build.sourcepackages)
424
403 def test_createBinaryPackageRelease(self):425 def test_createBinaryPackageRelease(self):
404 build = self.factory.makeCIBuild()426 build = self.factory.makeCIBuild()
405 bpn = self.factory.makeBinaryPackageName()427 bpn = self.factory.makeBinaryPackageName()
diff --git a/lib/lp/registry/interfaces/sourcepackage.py b/lib/lp/registry/interfaces/sourcepackage.py
index fcde488..6a7e707 100644
--- a/lib/lp/registry/interfaces/sourcepackage.py
+++ b/lib/lp/registry/interfaces/sourcepackage.py
@@ -456,10 +456,10 @@ class SourcePackageType(DBEnumeratedType):
456 This is the source package format used by Gentoo.456 This is the source package format used by Gentoo.
457 """)457 """)
458458
459 SDIST = DBItem(4, """459 CI_BUILD = DBItem(4, """
460 The Python Format460 CI Build
461461
462 This is the source package format used by Python packages.462 An ad-hoc source package generated by a CI build.
463 """)463 """)
464464
465465
diff --git a/lib/lp/soyuz/interfaces/sourcepackagerelease.py b/lib/lp/soyuz/interfaces/sourcepackagerelease.py
index 8e6cee9..aac4f39 100644
--- a/lib/lp/soyuz/interfaces/sourcepackagerelease.py
+++ b/lib/lp/soyuz/interfaces/sourcepackagerelease.py
@@ -145,11 +145,20 @@ class ISourcePackageRelease(Interface):
145 # Really ISourcePackageRecipeBuild -- see _schema_circular_imports.145 # Really ISourcePackageRecipeBuild -- see _schema_circular_imports.
146 source_package_recipe_build = Reference(146 source_package_recipe_build = Reference(
147 schema=Interface,147 schema=Interface,
148 description=_("The `SourcePackageRecipeBuild` which produced this "148 description=_(
149 "source package release, or None if it was created from a "149 "The `SourcePackageRecipeBuild` which produced this source "
150 "traditional upload."),150 "package release, or None if it was not created from a source "
151 "package recipe."),
151 title=_("Source package recipe build"),152 title=_("Source package recipe build"),
152 required=False, readonly=True)153 required=False, readonly=True)
154 # Really ICIBuild, patched in _schema_circular_imports.
155 ci_build = Reference(
156 schema=Interface,
157 description=_(
158 "The `CIBuild` which produced this source package release, or "
159 "None if it was not created from a CI build."),
160 title=_("CI build"),
161 required=False, readonly=True)
153162
154 def getUserDefinedField(name):163 def getUserDefinedField(name):
155 """Case-insensitively get a user-defined field."""164 """Case-insensitively get a user-defined field."""
diff --git a/lib/lp/soyuz/model/archivejob.py b/lib/lp/soyuz/model/archivejob.py
index d62f199..ae038a3 100644
--- a/lib/lp/soyuz/model/archivejob.py
+++ b/lib/lp/soyuz/model/archivejob.py
@@ -13,11 +13,18 @@ from typing import (
13 Dict,13 Dict,
14 Iterable,14 Iterable,
15 List,15 List,
16 Optional,
17 Sequence,
18 Tuple,
19 Union,
16 )20 )
17import zipfile21import zipfile
1822
19from lazr.delegates import delegate_to23from lazr.delegates import delegate_to
20from pkginfo import Wheel24from pkginfo import (
25 SDist,
26 Wheel,
27 )
21from storm.expr import And28from storm.expr import And
22from storm.locals import (29from storm.locals import (
23 Int,30 Int,
@@ -43,6 +50,7 @@ from lp.registry.interfaces.distributionsourcepackage import (
43 )50 )
44from lp.registry.interfaces.distroseries import IDistroSeriesSet51from lp.registry.interfaces.distroseries import IDistroSeriesSet
45from lp.registry.interfaces.pocket import PackagePublishingPocket52from lp.registry.interfaces.pocket import PackagePublishingPocket
53from lp.registry.interfaces.sourcepackage import SourcePackageFileType
46from lp.services.config import config54from lp.services.config import config
47from lp.services.database.enumcol import DBEnum55from lp.services.database.enumcol import DBEnum
48from lp.services.database.interfaces import IMasterStore56from lp.services.database.interfaces import IMasterStore
@@ -212,10 +220,39 @@ class ScanException(Exception):
212 """A CI build upload job failed to scan a file."""220 """A CI build upload job failed to scan a file."""
213221
214222
223class SourceArtifactMetadata:
224 """Metadata extracted from a source package."""
225
226 def __init__(self, format: SourcePackageFileType, name: str, version: str):
227 self.format = format
228 self.name = name
229 self.version = version
230
231
232class BinaryArtifactMetadata:
233 """Metadata extracted from a binary package."""
234
235 def __init__(self, format: BinaryPackageFormat, name: str, version: str,
236 summary: str, description: str, architecturespecific: bool,
237 homepage: str,
238 user_defined_fields: Optional[List[Tuple[str, str]]] = None):
239 self.format = format
240 self.name = name
241 self.version = version
242 self.summary = summary
243 self.description = description
244 self.architecturespecific = architecturespecific
245 self.homepage = homepage
246 self.user_defined_fields = user_defined_fields
247
248
249ArtifactMetadata = Union[SourceArtifactMetadata, BinaryArtifactMetadata]
250
251
215class ScannedArtifact:252class ScannedArtifact:
216253
217 def __init__(254 def __init__(
218 self, *, artifact: IRevisionStatusArtifact, metadata: Dict[str, Any]255 self, *, artifact: IRevisionStatusArtifact, metadata: ArtifactMetadata
219 ):256 ):
220 self.artifact = artifact257 self.artifact = artifact
221 self.metadata = metadata258 self.metadata = metadata
@@ -247,13 +284,16 @@ class CIBuildUploadJob(ArchiveJobDerived):
247284
248 # We're only interested in uploading certain kinds of packages to285 # We're only interested in uploading certain kinds of packages to
249 # certain kinds of archives.286 # certain kinds of archives.
250 binary_format_by_repository_format = {287 format_by_repository_format = {
251 ArchiveRepositoryFormat.DEBIAN: {288 ArchiveRepositoryFormat.DEBIAN: {
252 BinaryPackageFormat.DEB,289 BinaryPackageFormat.DEB,
253 BinaryPackageFormat.UDEB,290 BinaryPackageFormat.UDEB,
254 BinaryPackageFormat.DDEB,291 BinaryPackageFormat.DDEB,
255 },292 },
256 ArchiveRepositoryFormat.PYTHON: {BinaryPackageFormat.WHL},293 ArchiveRepositoryFormat.PYTHON: {
294 SourcePackageFileType.SDIST,
295 BinaryPackageFormat.WHL,
296 },
257 ArchiveRepositoryFormat.CONDA: {297 ArchiveRepositoryFormat.CONDA: {
258 BinaryPackageFormat.CONDA_V1,298 BinaryPackageFormat.CONDA_V1,
259 BinaryPackageFormat.CONDA_V2,299 BinaryPackageFormat.CONDA_V2,
@@ -326,7 +366,7 @@ class CIBuildUploadJob(ArchiveJobDerived):
326 def target_channel(self):366 def target_channel(self):
327 return self.metadata["target_channel"]367 return self.metadata["target_channel"]
328368
329 def _scanWheel(self, path: str) -> Dict[str, Any]:369 def _scanWheel(self, path: str) -> Optional[BinaryArtifactMetadata]:
330 try:370 try:
331 parsed_path = parse_wheel_filename(path)371 parsed_path = parse_wheel_filename(path)
332 wheel = Wheel(path)372 wheel = Wheel(path)
@@ -335,34 +375,50 @@ class CIBuildUploadJob(ArchiveJobDerived):
335 "Failed to scan %s as a Python wheel: %s",375 "Failed to scan %s as a Python wheel: %s",
336 os.path.basename(path), e)376 os.path.basename(path), e)
337 return None377 return None
338 return {378 return BinaryArtifactMetadata(
339 "name": wheel.name,379 format=BinaryPackageFormat.WHL,
340 "version": wheel.version,380 name=wheel.name,
341 "summary": wheel.summary or "",381 version=wheel.version,
342 "description": wheel.description,382 summary=wheel.summary or "",
343 "binpackageformat": BinaryPackageFormat.WHL,383 description=wheel.description,
344 "architecturespecific": "any" not in parsed_path.platform_tags,384 architecturespecific="any" not in parsed_path.platform_tags,
345 "homepage": wheel.home_page or "",385 homepage=wheel.home_page or "",
346 }386 )
387
388 def _scanSDist(self, path: str) -> Optional[SourceArtifactMetadata]:
389 try:
390 sdist = SDist(path)
391 except Exception as e:
392 logger.warning(
393 "Failed to scan %s as a Python sdist: %s",
394 os.path.basename(path), e)
395 return None
396 return SourceArtifactMetadata(
397 format=SourcePackageFileType.SDIST,
398 name=sdist.name,
399 version=sdist.version,
400 )
347401
348 def _scanCondaMetadata(402 def _scanCondaMetadata(
349 self, index: Dict[Any, Any], about: Dict[Any, Any]403 self, format: BinaryPackageFormat, index: Dict[Any, Any],
350 ) -> Dict[str, Any]:404 about: Dict[Any, Any]
351 return {405 ) -> Optional[BinaryArtifactMetadata]:
352 "name": index["name"],406 return BinaryArtifactMetadata(
353 "version": index["version"],407 format=format,
354 "summary": about.get("summary", ""),408 name=index["name"],
355 "description": about.get("description", ""),409 version=index["version"],
356 "architecturespecific": index["platform"] is not None,410 summary=about.get("summary", ""),
357 "homepage": about.get("home", ""),411 description=about.get("description", ""),
412 architecturespecific=index["platform"] is not None,
413 homepage=about.get("home", ""),
358 # We should perhaps model this explicitly since it's used by the414 # We should perhaps model this explicitly since it's used by the
359 # publisher, but this gives us an easy way to pass this through415 # publisher, but this gives us an easy way to pass this through
360 # without needing to add a column to a large table that's only416 # without needing to add a column to a large table that's only
361 # relevant to a tiny minority of rows.417 # relevant to a tiny minority of rows.
362 "user_defined_fields": [("subdir", index["subdir"])],418 user_defined_fields=[("subdir", index["subdir"])],
363 }419 )
364420
365 def _scanCondaV1(self, path: str) -> Dict[str, Any]:421 def _scanCondaV1(self, path: str) -> Optional[BinaryArtifactMetadata]:
366 try:422 try:
367 with tarfile.open(path) as tar:423 with tarfile.open(path) as tar:
368 index = json.loads(424 index = json.loads(
@@ -374,11 +430,10 @@ class CIBuildUploadJob(ArchiveJobDerived):
374 "Failed to scan %s as a Conda v1 package: %s",430 "Failed to scan %s as a Conda v1 package: %s",
375 os.path.basename(path), e)431 os.path.basename(path), e)
376 return None432 return None
377 scanned = {"binpackageformat": BinaryPackageFormat.CONDA_V1}433 return self._scanCondaMetadata(
378 scanned.update(self._scanCondaMetadata(index, about))434 BinaryPackageFormat.CONDA_V1, index, about)
379 return scanned
380435
381 def _scanCondaV2(self, path: str) -> Dict[str, Any]:436 def _scanCondaV2(self, path: str) -> Optional[BinaryArtifactMetadata]:
382 try:437 try:
383 with zipfile.ZipFile(path) as zipf:438 with zipfile.ZipFile(path) as zipf:
384 base_name = os.path.basename(path)[:-len(".conda")]439 base_name = os.path.basename(path)[:-len(".conda")]
@@ -396,13 +451,14 @@ class CIBuildUploadJob(ArchiveJobDerived):
396 "Failed to scan %s as a Conda v2 package: %s",451 "Failed to scan %s as a Conda v2 package: %s",
397 os.path.basename(path), e)452 os.path.basename(path), e)
398 return None453 return None
399 scanned = {"binpackageformat": BinaryPackageFormat.CONDA_V2}454 return self._scanCondaMetadata(
400 scanned.update(self._scanCondaMetadata(index, about))455 BinaryPackageFormat.CONDA_V2, index, about)
401 return scanned
402456
403 def _scanFile(self, path: str) -> Dict[str, Any]:457 def _scanFile(self, path: str) -> Optional[ArtifactMetadata]:
404 _scanners = (458 _scanners = (
405 (".whl", self._scanWheel),459 (".whl", self._scanWheel),
460 (".tar.gz", self._scanSDist),
461 (".zip", self._scanSDist),
406 (".tar.bz2", self._scanCondaV1),462 (".tar.bz2", self._scanCondaV1),
407 (".conda", self._scanCondaV2),463 (".conda", self._scanCondaV2),
408 )464 )
@@ -429,8 +485,8 @@ class CIBuildUploadJob(ArchiveJobDerived):
429 Returns a list of `ScannedArtifact`s containing metadata for485 Returns a list of `ScannedArtifact`s containing metadata for
430 relevant artifacts.486 relevant artifacts.
431 """487 """
432 allowed_binary_formats = (488 allowed_formats = (
433 self.binary_format_by_repository_format.get(489 self.format_by_repository_format.get(
434 self.archive.repository_format, set()))490 self.archive.repository_format, set()))
435 scanned = []491 scanned = []
436 with tempfile.TemporaryDirectory(prefix="ci-build-copy-job") as tmpdir:492 with tempfile.TemporaryDirectory(prefix="ci-build-copy-job") as tmpdir:
@@ -444,7 +500,7 @@ class CIBuildUploadJob(ArchiveJobDerived):
444 metadata = self._scanFile(contents)500 metadata = self._scanFile(contents)
445 if metadata is None:501 if metadata is None:
446 continue502 continue
447 if metadata["binpackageformat"] not in allowed_binary_formats:503 if metadata.format not in allowed_formats:
448 logger.info(504 logger.info(
449 "Skipping %s (not relevant to %s archives)",505 "Skipping %s (not relevant to %s archives)",
450 name, self.archive.repository_format)506 name, self.archive.repository_format)
@@ -453,28 +509,79 @@ class CIBuildUploadJob(ArchiveJobDerived):
453 ScannedArtifact(artifact=artifact, metadata=metadata))509 ScannedArtifact(artifact=artifact, metadata=metadata))
454 return scanned510 return scanned
455511
512 def _uploadSources(self, scanned: Sequence[ScannedArtifact]) -> None:
513 """Upload sources from an sequence of `ScannedArtifact`s."""
514 # Launchpad's data model generally assumes that a single source is
515 # associated with multiple binaries. However, a source package
516 # release can have multiple (or indeed no) files attached to it, so
517 # we make use of that if necessary.
518 releases = {
519 release.sourcepackagename: release
520 for release in self.ci_build.sourcepackages}
521 distroseries = self.ci_build.distro_arch_series.distroseries
522 build_target = self.ci_build.git_repository.target
523 spr = releases.get(build_target.sourcepackagename)
524 if spr is None:
525 spr = self.ci_build.createSourcePackageRelease(
526 distroseries=distroseries,
527 sourcepackagename=build_target.sourcepackagename,
528 # We don't have a good concept of source version here, but
529 # the data model demands one. Arbitrarily pick the version
530 # of the first scanned artifact.
531 version=scanned[0].metadata.version,
532 creator=self.requester,
533 archive=self.archive)
534 for scanned_artifact in scanned:
535 metadata = scanned_artifact.metadata
536 if not isinstance(metadata, SourceArtifactMetadata):
537 continue
538 library_file = scanned_artifact.artifact.library_file
539 logger.info(
540 "Uploading %s to %s %s (%s)",
541 library_file.filename, self.archive.reference,
542 self.target_distroseries.getSuite(self.target_pocket),
543 self.target_channel)
544 for sprf in spr.files:
545 if (sprf.libraryfile == library_file and
546 sprf.filetype == metadata.format):
547 break
548 else:
549 spr.addFile(library_file, filetype=metadata.format)
550 getUtility(IPublishingSet).newSourcePublication(
551 archive=self.archive, sourcepackagerelease=spr,
552 distroseries=self.target_distroseries, pocket=self.target_pocket,
553 creator=self.requester, channel=self.target_channel)
554
456 def _uploadBinaries(self, scanned: Iterable[ScannedArtifact]) -> None:555 def _uploadBinaries(self, scanned: Iterable[ScannedArtifact]) -> None:
457 """Upload an iterable of `ScannedArtifact`s to an archive."""556 """Upload binaries from an iterable of `ScannedArtifact`s."""
458 releases = {557 releases = {
459 (release.binarypackagename, release.binpackageformat): release558 (release.binarypackagename, release.binpackageformat): release
460 for release in self.ci_build.binarypackages}559 for release in self.ci_build.binarypackages}
461 binaries = OrderedDict()560 binaries = OrderedDict()
462 for scanned_artifact in scanned:561 for scanned_artifact in scanned:
562 metadata = scanned_artifact.metadata
563 if not isinstance(metadata, BinaryArtifactMetadata):
564 continue
463 library_file = scanned_artifact.artifact.library_file565 library_file = scanned_artifact.artifact.library_file
464 metadata = dict(scanned_artifact.metadata)
465 binpackageformat = metadata["binpackageformat"]
466 logger.info(566 logger.info(
467 "Uploading %s to %s %s (%s)",567 "Uploading %s to %s %s (%s)",
468 library_file.filename, self.archive.reference,568 library_file.filename, self.archive.reference,
469 self.target_distroseries.getSuite(self.target_pocket),569 self.target_distroseries.getSuite(self.target_pocket),
470 self.target_channel)570 self.target_channel)
471 metadata["binarypackagename"] = bpn = (571 binarypackagename = getUtility(IBinaryPackageNameSet).ensure(
472 getUtility(IBinaryPackageNameSet).ensure(metadata["name"]))572 metadata.name)
473 del metadata["name"]573 filetype = self.filetype_by_format[metadata.format]
474 filetype = self.filetype_by_format[binpackageformat]574 bpr = releases.get((binarypackagename, metadata.format))
475 bpr = releases.get((bpn, binpackageformat))
476 if bpr is None:575 if bpr is None:
477 bpr = self.ci_build.createBinaryPackageRelease(**metadata)576 bpr = self.ci_build.createBinaryPackageRelease(
577 binarypackagename=binarypackagename,
578 version=metadata.version,
579 summary=metadata.summary,
580 description=metadata.description,
581 binpackageformat=metadata.format,
582 architecturespecific=metadata.architecturespecific,
583 homepage=metadata.homepage,
584 user_defined_fields=metadata.user_defined_fields)
478 for bpf in bpr.files:585 for bpf in bpr.files:
479 if (bpf.libraryfile == library_file and586 if (bpf.libraryfile == library_file and
480 bpf.filetype == filetype):587 bpf.filetype == filetype):
@@ -503,6 +610,7 @@ class CIBuildUploadJob(ArchiveJobDerived):
503 self.ci_build)610 self.ci_build)
504 scanned = self._scanArtifacts(artifacts)611 scanned = self._scanArtifacts(artifacts)
505 if scanned:612 if scanned:
613 self._uploadSources(scanned)
506 self._uploadBinaries(scanned)614 self._uploadBinaries(scanned)
507 else:615 else:
508 names = [artifact.library_file.filename for artifact in artifacts]616 names = [artifact.library_file.filename for artifact in artifacts]
diff --git a/lib/lp/soyuz/tests/test_archivejob.py b/lib/lp/soyuz/tests/test_archivejob.py
index b744bc8..813b423 100644
--- a/lib/lp/soyuz/tests/test_archivejob.py
+++ b/lib/lp/soyuz/tests/test_archivejob.py
@@ -7,6 +7,7 @@ from debian.deb822 import Changes
7from fixtures import (7from fixtures import (
8 FakeLogger,8 FakeLogger,
9 MockPatch,9 MockPatch,
10 MockPatchObject,
10 )11 )
11from testtools.matchers import (12from testtools.matchers import (
12 ContainsDict,13 ContainsDict,
@@ -20,6 +21,10 @@ import transaction
20from lp.code.enums import RevisionStatusArtifactType21from lp.code.enums import RevisionStatusArtifactType
21from lp.code.model.revisionstatus import RevisionStatusArtifact22from lp.code.model.revisionstatus import RevisionStatusArtifact
22from lp.registry.interfaces.pocket import PackagePublishingPocket23from lp.registry.interfaces.pocket import PackagePublishingPocket
24from lp.registry.interfaces.sourcepackage import (
25 SourcePackageFileType,
26 SourcePackageType,
27 )
23from lp.services.config import config28from lp.services.config import config
24from lp.services.database.interfaces import IStore29from lp.services.database.interfaces import IStore
25from lp.services.features.testing import FeatureFixture30from lp.services.features.testing import FeatureFixture
@@ -41,8 +46,10 @@ from lp.soyuz.enums import (
41from lp.soyuz.model.archivejob import (46from lp.soyuz.model.archivejob import (
42 ArchiveJob,47 ArchiveJob,
43 ArchiveJobDerived,48 ArchiveJobDerived,
49 BinaryArtifactMetadata,
44 CIBuildUploadJob,50 CIBuildUploadJob,
45 PackageUploadNotificationJob,51 PackageUploadNotificationJob,
52 SourceArtifactMetadata,
46 )53 )
47from lp.soyuz.tests import datadir54from lp.soyuz.tests import datadir
48from lp.testing import (55from lp.testing import (
@@ -242,16 +249,19 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
242 build, build.git_repository.owner, archive, distroseries,249 build, build.git_repository.owner, archive, distroseries,
243 PackagePublishingPocket.RELEASE, target_channel="edge")250 PackagePublishingPocket.RELEASE, target_channel="edge")
244 path = "wheel-indep/dist/wheel_indep-0.0.1-py3-none-any.whl"251 path = "wheel-indep/dist/wheel_indep-0.0.1-py3-none-any.whl"
245 expected = {252 metadata = job._scanFile(datadir(path))
246 "name": "wheel-indep",253 self.assertIsInstance(metadata, BinaryArtifactMetadata)
247 "version": "0.0.1",254 self.assertThat(
248 "summary": "Example description",255 metadata,
249 "description": "Example long description\n",256 MatchesStructure(
250 "binpackageformat": BinaryPackageFormat.WHL,257 format=Equals(BinaryPackageFormat.WHL),
251 "architecturespecific": False,258 name=Equals("wheel-indep"),
252 "homepage": "",259 version=Equals("0.0.1"),
253 }260 summary=Equals("Example description"),
254 self.assertEqual(expected, job._scanFile(datadir(path)))261 description=Equals("Example long description\n"),
262 architecturespecific=Is(False),
263 homepage=Equals(""),
264 ))
255265
256 def test__scanFile_wheel_arch(self):266 def test__scanFile_wheel_arch(self):
257 archive = self.factory.makeArchive()267 archive = self.factory.makeArchive()
@@ -262,16 +272,38 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
262 build, build.git_repository.owner, archive, distroseries,272 build, build.git_repository.owner, archive, distroseries,
263 PackagePublishingPocket.RELEASE, target_channel="edge")273 PackagePublishingPocket.RELEASE, target_channel="edge")
264 path = "wheel-arch/dist/wheel_arch-0.0.1-cp310-cp310-linux_x86_64.whl"274 path = "wheel-arch/dist/wheel_arch-0.0.1-cp310-cp310-linux_x86_64.whl"
265 expected = {275 metadata = job._scanFile(datadir(path))
266 "name": "wheel-arch",276 self.assertIsInstance(metadata, BinaryArtifactMetadata)
267 "version": "0.0.1",277 self.assertThat(
268 "summary": "Example description",278 metadata,
269 "description": "Example long description\n",279 MatchesStructure(
270 "binpackageformat": BinaryPackageFormat.WHL,280 format=Equals(BinaryPackageFormat.WHL),
271 "architecturespecific": True,281 name=Equals("wheel-arch"),
272 "homepage": "http://example.com/",282 version=Equals("0.0.1"),
273 }283 summary=Equals("Example description"),
274 self.assertEqual(expected, job._scanFile(datadir(path)))284 description=Equals("Example long description\n"),
285 architecturespecific=Is(True),
286 homepage=Equals("http://example.com/"),
287 ))
288
289 def test__scanFile_sdist(self):
290 archive = self.factory.makeArchive()
291 distroseries = self.factory.makeDistroSeries(
292 distribution=archive.distribution)
293 build = self.makeCIBuild(archive.distribution)
294 job = CIBuildUploadJob.create(
295 build, build.git_repository.owner, archive, distroseries,
296 PackagePublishingPocket.RELEASE, target_channel="edge")
297 path = "wheel-arch/dist/wheel-arch-0.0.1.tar.gz"
298 metadata = job._scanFile(datadir(path))
299 self.assertIsInstance(metadata, SourceArtifactMetadata)
300 self.assertThat(
301 metadata,
302 MatchesStructure.byEquality(
303 format=SourcePackageFileType.SDIST,
304 name="wheel-arch",
305 version="0.0.1",
306 ))
275307
276 def test__scanFile_conda_indep(self):308 def test__scanFile_conda_indep(self):
277 archive = self.factory.makeArchive()309 archive = self.factory.makeArchive()
@@ -282,17 +314,20 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
282 build, build.git_repository.owner, archive, distroseries,314 build, build.git_repository.owner, archive, distroseries,
283 PackagePublishingPocket.RELEASE, target_channel="edge")315 PackagePublishingPocket.RELEASE, target_channel="edge")
284 path = "conda-indep/dist/noarch/conda-indep-0.1-0.tar.bz2"316 path = "conda-indep/dist/noarch/conda-indep-0.1-0.tar.bz2"
285 expected = {317 metadata = job._scanFile(datadir(path))
286 "name": "conda-indep",318 self.assertIsInstance(metadata, BinaryArtifactMetadata)
287 "version": "0.1",319 self.assertThat(
288 "summary": "Example summary",320 metadata,
289 "description": "Example description",321 MatchesStructure(
290 "binpackageformat": BinaryPackageFormat.CONDA_V1,322 format=Equals(BinaryPackageFormat.CONDA_V1),
291 "architecturespecific": False,323 name=Equals("conda-indep"),
292 "homepage": "",324 version=Equals("0.1"),
293 "user_defined_fields": [("subdir", "noarch")],325 summary=Equals("Example summary"),
294 }326 description=Equals("Example description"),
295 self.assertEqual(expected, job._scanFile(datadir(path)))327 architecturespecific=Is(False),
328 homepage=Equals(""),
329 user_defined_fields=Equals([("subdir", "noarch")]),
330 ))
296331
297 def test__scanFile_conda_arch(self):332 def test__scanFile_conda_arch(self):
298 archive = self.factory.makeArchive()333 archive = self.factory.makeArchive()
@@ -303,17 +338,20 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
303 build, build.git_repository.owner, archive, distroseries,338 build, build.git_repository.owner, archive, distroseries,
304 PackagePublishingPocket.RELEASE, target_channel="edge")339 PackagePublishingPocket.RELEASE, target_channel="edge")
305 path = "conda-arch/dist/linux-64/conda-arch-0.1-0.tar.bz2"340 path = "conda-arch/dist/linux-64/conda-arch-0.1-0.tar.bz2"
306 expected = {341 metadata = job._scanFile(datadir(path))
307 "name": "conda-arch",342 self.assertIsInstance(metadata, BinaryArtifactMetadata)
308 "version": "0.1",343 self.assertThat(
309 "summary": "Example summary",344 metadata,
310 "description": "Example description",345 MatchesStructure(
311 "binpackageformat": BinaryPackageFormat.CONDA_V1,346 format=Equals(BinaryPackageFormat.CONDA_V1),
312 "architecturespecific": True,347 name=Equals("conda-arch"),
313 "homepage": "http://example.com/",348 version=Equals("0.1"),
314 "user_defined_fields": [("subdir", "linux-64")],349 summary=Equals("Example summary"),
315 }350 description=Equals("Example description"),
316 self.assertEqual(expected, job._scanFile(datadir(path)))351 architecturespecific=Is(True),
352 homepage=Equals("http://example.com/"),
353 user_defined_fields=Equals([("subdir", "linux-64")]),
354 ))
317355
318 def test__scanFile_conda_v2_indep(self):356 def test__scanFile_conda_v2_indep(self):
319 archive = self.factory.makeArchive()357 archive = self.factory.makeArchive()
@@ -324,17 +362,20 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
324 build, build.git_repository.owner, archive, distroseries,362 build, build.git_repository.owner, archive, distroseries,
325 PackagePublishingPocket.RELEASE, target_channel="edge")363 PackagePublishingPocket.RELEASE, target_channel="edge")
326 path = "conda-v2-indep/dist/noarch/conda-v2-indep-0.1-0.conda"364 path = "conda-v2-indep/dist/noarch/conda-v2-indep-0.1-0.conda"
327 expected = {365 metadata = job._scanFile(datadir(path))
328 "name": "conda-v2-indep",366 self.assertIsInstance(metadata, BinaryArtifactMetadata)
329 "version": "0.1",367 self.assertThat(
330 "summary": "Example summary",368 metadata,
331 "description": "Example description",369 MatchesStructure(
332 "binpackageformat": BinaryPackageFormat.CONDA_V2,370 format=Equals(BinaryPackageFormat.CONDA_V2),
333 "architecturespecific": False,371 name=Equals("conda-v2-indep"),
334 "homepage": "",372 version=Equals("0.1"),
335 "user_defined_fields": [("subdir", "noarch")],373 summary=Equals("Example summary"),
336 }374 description=Equals("Example description"),
337 self.assertEqual(expected, job._scanFile(datadir(path)))375 architecturespecific=Is(False),
376 homepage=Equals(""),
377 user_defined_fields=Equals([("subdir", "noarch")]),
378 ))
338379
339 def test__scanFile_conda_v2_arch(self):380 def test__scanFile_conda_v2_arch(self):
340 archive = self.factory.makeArchive()381 archive = self.factory.makeArchive()
@@ -345,17 +386,20 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
345 build, build.git_repository.owner, archive, distroseries,386 build, build.git_repository.owner, archive, distroseries,
346 PackagePublishingPocket.RELEASE, target_channel="edge")387 PackagePublishingPocket.RELEASE, target_channel="edge")
347 path = "conda-v2-arch/dist/linux-64/conda-v2-arch-0.1-0.conda"388 path = "conda-v2-arch/dist/linux-64/conda-v2-arch-0.1-0.conda"
348 expected = {389 metadata = job._scanFile(datadir(path))
349 "name": "conda-v2-arch",390 self.assertIsInstance(metadata, BinaryArtifactMetadata)
350 "version": "0.1",391 self.assertThat(
351 "summary": "Example summary",392 metadata,
352 "description": "Example description",393 MatchesStructure(
353 "binpackageformat": BinaryPackageFormat.CONDA_V2,394 format=Equals(BinaryPackageFormat.CONDA_V2),
354 "architecturespecific": True,395 name=Equals("conda-v2-arch"),
355 "homepage": "http://example.com/",396 version=Equals("0.1"),
356 "user_defined_fields": [("subdir", "linux-64")],397 summary=Equals("Example summary"),
357 }398 description=Equals("Example description"),
358 self.assertEqual(expected, job._scanFile(datadir(path)))399 architecturespecific=Is(True),
400 homepage=Equals("http://example.com/"),
401 user_defined_fields=Equals([("subdir", "linux-64")]),
402 ))
359403
360 def test_run_indep(self):404 def test_run_indep(self):
361 archive = self.factory.makeArchive(405 archive = self.factory.makeArchive(
@@ -384,6 +428,21 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
384 with dbuser(job.config.dbuser):428 with dbuser(job.config.dbuser):
385 JobRunner([job]).runAll()429 JobRunner([job]).runAll()
386430
431 self.assertThat(archive.getPublishedSources(), MatchesSetwise(
432 MatchesStructure(
433 sourcepackagename=MatchesStructure.byEquality(
434 name=build.git_repository.target.name),
435 sourcepackagerelease=MatchesStructure(
436 ci_build=Equals(build),
437 sourcepackagename=MatchesStructure.byEquality(
438 name=build.git_repository.target.name),
439 version=Equals("0.0.1"),
440 format=Equals(SourcePackageType.CI_BUILD),
441 architecturehintlist=Equals(""),
442 creator=Equals(build.git_repository.owner),
443 files=Equals([])),
444 format=Equals(SourcePackageType.CI_BUILD),
445 distroseries=Equals(distroseries))))
387 self.assertThat(archive.getAllPublishedBinaries(), MatchesSetwise(*(446 self.assertThat(archive.getAllPublishedBinaries(), MatchesSetwise(*(
388 MatchesStructure(447 MatchesStructure(
389 binarypackagename=MatchesStructure.byEquality(448 binarypackagename=MatchesStructure.byEquality(
@@ -418,13 +477,17 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
418 archive.distribution, distro_arch_series=dases[0])477 archive.distribution, distro_arch_series=dases[0])
419 report = build.getOrCreateRevisionStatusReport("build:0")478 report = build.getOrCreateRevisionStatusReport("build:0")
420 report.setLog(b"log data")479 report.setLog(b"log data")
421 path = "wheel-arch/dist/wheel_arch-0.0.1-cp310-cp310-linux_x86_64.whl"480 sdist_path = "wheel-arch/dist/wheel-arch-0.0.1.tar.gz"
422 with open(datadir(path), mode="rb") as f:481 wheel_path = (
423 report.attach(name=os.path.basename(path), data=f.read())482 "wheel-arch/dist/wheel_arch-0.0.1-cp310-cp310-linux_x86_64.whl")
424 artifact = IStore(RevisionStatusArtifact).find(483 with open(datadir(sdist_path), mode="rb") as f:
484 report.attach(name=os.path.basename(sdist_path), data=f.read())
485 with open(datadir(wheel_path), mode="rb") as f:
486 report.attach(name=os.path.basename(wheel_path), data=f.read())
487 artifacts = IStore(RevisionStatusArtifact).find(
425 RevisionStatusArtifact,488 RevisionStatusArtifact,
426 report=report,489 report=report,
427 artifact_type=RevisionStatusArtifactType.BINARY).one()490 artifact_type=RevisionStatusArtifactType.BINARY).order_by("id")
428 job = CIBuildUploadJob.create(491 job = CIBuildUploadJob.create(
429 build, build.git_repository.owner, archive, distroseries,492 build, build.git_repository.owner, archive, distroseries,
430 PackagePublishingPocket.RELEASE, target_channel="edge")493 PackagePublishingPocket.RELEASE, target_channel="edge")
@@ -433,6 +496,24 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
433 with dbuser(job.config.dbuser):496 with dbuser(job.config.dbuser):
434 JobRunner([job]).runAll()497 JobRunner([job]).runAll()
435498
499 self.assertThat(archive.getPublishedSources(), MatchesSetwise(
500 MatchesStructure(
501 sourcepackagename=MatchesStructure.byEquality(
502 name=build.git_repository.target.name),
503 sourcepackagerelease=MatchesStructure(
504 ci_build=Equals(build),
505 sourcepackagename=MatchesStructure.byEquality(
506 name=build.git_repository.target.name),
507 version=Equals("0.0.1"),
508 format=Equals(SourcePackageType.CI_BUILD),
509 architecturehintlist=Equals(""),
510 creator=Equals(build.git_repository.owner),
511 files=MatchesSetwise(
512 MatchesStructure.byEquality(
513 libraryfile=artifacts[0].library_file,
514 filetype=SourcePackageFileType.SDIST))),
515 format=Equals(SourcePackageType.CI_BUILD),
516 distroseries=Equals(distroseries))))
436 self.assertThat(archive.getAllPublishedBinaries(), MatchesSetwise(517 self.assertThat(archive.getAllPublishedBinaries(), MatchesSetwise(
437 MatchesStructure(518 MatchesStructure(
438 binarypackagename=MatchesStructure.byEquality(519 binarypackagename=MatchesStructure.byEquality(
@@ -449,7 +530,7 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
449 homepage=Equals("http://example.com/"),530 homepage=Equals("http://example.com/"),
450 files=MatchesSetwise(531 files=MatchesSetwise(
451 MatchesStructure.byEquality(532 MatchesStructure.byEquality(
452 libraryfile=artifact.library_file,533 libraryfile=artifacts[1].library_file,
453 filetype=BinaryPackageFileType.WHL))),534 filetype=BinaryPackageFileType.WHL))),
454 binarypackageformat=Equals(BinaryPackageFormat.WHL),535 binarypackageformat=Equals(BinaryPackageFormat.WHL),
455 distroarchseries=Equals(dases[0]))))536 distroarchseries=Equals(dases[0]))))
@@ -481,6 +562,21 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
481 with dbuser(job.config.dbuser):562 with dbuser(job.config.dbuser):
482 JobRunner([job]).runAll()563 JobRunner([job]).runAll()
483564
565 self.assertThat(archive.getPublishedSources(), MatchesSetwise(
566 MatchesStructure(
567 sourcepackagename=MatchesStructure.byEquality(
568 name=build.git_repository.target.name),
569 sourcepackagerelease=MatchesStructure(
570 ci_build=Equals(build),
571 sourcepackagename=MatchesStructure.byEquality(
572 name=build.git_repository.target.name),
573 version=Equals("0.1"),
574 format=Equals(SourcePackageType.CI_BUILD),
575 architecturehintlist=Equals(""),
576 creator=Equals(build.git_repository.owner),
577 files=Equals([])),
578 format=Equals(SourcePackageType.CI_BUILD),
579 distroseries=Equals(distroseries))))
484 self.assertThat(archive.getAllPublishedBinaries(), MatchesSetwise(580 self.assertThat(archive.getAllPublishedBinaries(), MatchesSetwise(
485 MatchesStructure(581 MatchesStructure(
486 binarypackagename=MatchesStructure.byEquality(582 binarypackagename=MatchesStructure.byEquality(
@@ -529,6 +625,21 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
529 with dbuser(job.config.dbuser):625 with dbuser(job.config.dbuser):
530 JobRunner([job]).runAll()626 JobRunner([job]).runAll()
531627
628 self.assertThat(archive.getPublishedSources(), MatchesSetwise(
629 MatchesStructure(
630 sourcepackagename=MatchesStructure.byEquality(
631 name=build.git_repository.target.name),
632 sourcepackagerelease=MatchesStructure(
633 ci_build=Equals(build),
634 sourcepackagename=MatchesStructure.byEquality(
635 name=build.git_repository.target.name),
636 version=Equals("0.1"),
637 format=Equals(SourcePackageType.CI_BUILD),
638 architecturehintlist=Equals(""),
639 creator=Equals(build.git_repository.owner),
640 files=Equals([])),
641 format=Equals(SourcePackageType.CI_BUILD),
642 distroseries=Equals(distroseries))))
532 self.assertThat(archive.getAllPublishedBinaries(), MatchesSetwise(643 self.assertThat(archive.getAllPublishedBinaries(), MatchesSetwise(
533 MatchesStructure(644 MatchesStructure(
534 binarypackagename=MatchesStructure.byEquality(645 binarypackagename=MatchesStructure.byEquality(
@@ -550,7 +661,7 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
550 binarypackageformat=Equals(BinaryPackageFormat.CONDA_V2),661 binarypackageformat=Equals(BinaryPackageFormat.CONDA_V2),
551 distroarchseries=Equals(dases[0]))))662 distroarchseries=Equals(dases[0]))))
552663
553 def test_existing_release(self):664 def test_existing_source_and_binary_releases(self):
554 # A `CIBuildUploadJob` can be run even if the build in question was665 # A `CIBuildUploadJob` can be run even if the build in question was
555 # already uploaded somewhere, and in that case may add publications666 # already uploaded somewhere, and in that case may add publications
556 # in other locations for the same package.667 # in other locations for the same package.
@@ -561,13 +672,16 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
561 das = self.factory.makeDistroArchSeries(distroseries=distroseries)672 das = self.factory.makeDistroArchSeries(distroseries=distroseries)
562 build = self.makeCIBuild(archive.distribution, distro_arch_series=das)673 build = self.makeCIBuild(archive.distribution, distro_arch_series=das)
563 report = build.getOrCreateRevisionStatusReport("build:0")674 report = build.getOrCreateRevisionStatusReport("build:0")
564 path = "wheel-indep/dist/wheel_indep-0.0.1-py3-none-any.whl"675 sdist_path = "wheel-indep/dist/wheel-indep-0.0.1.tar.gz"
565 with open(datadir(path), mode="rb") as f:676 with open(datadir(sdist_path), mode="rb") as f:
566 report.attach(name=os.path.basename(path), data=f.read())677 report.attach(name=os.path.basename(sdist_path), data=f.read())
567 artifact = IStore(RevisionStatusArtifact).find(678 wheel_path = "wheel-indep/dist/wheel_indep-0.0.1-py3-none-any.whl"
679 with open(datadir(wheel_path), mode="rb") as f:
680 report.attach(name=os.path.basename(wheel_path), data=f.read())
681 artifacts = IStore(RevisionStatusArtifact).find(
568 RevisionStatusArtifact,682 RevisionStatusArtifact,
569 report=report,683 report=report,
570 artifact_type=RevisionStatusArtifactType.BINARY).one()684 artifact_type=RevisionStatusArtifactType.BINARY).order_by("id")
571 job = CIBuildUploadJob.create(685 job = CIBuildUploadJob.create(
572 build, build.git_repository.owner, archive, distroseries,686 build, build.git_repository.owner, archive, distroseries,
573 PackagePublishingPocket.RELEASE, target_channel="edge")687 PackagePublishingPocket.RELEASE, target_channel="edge")
@@ -582,12 +696,31 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
582 with dbuser(job.config.dbuser):696 with dbuser(job.config.dbuser):
583 JobRunner([job]).runAll()697 JobRunner([job]).runAll()
584698
699 spphs = archive.getPublishedSources()
700 # The source publications are for the same source package release,
701 # which has a single file attached to it.
702 self.assertEqual(1, len({spph.sourcepackagename for spph in spphs}))
703 self.assertEqual(1, len({spph.sourcepackagerelease for spph in spphs}))
704 self.assertThat(spphs, MatchesSetwise(*(
705 MatchesStructure(
706 sourcepackagename=MatchesStructure.byEquality(
707 name=build.git_repository.target.name),
708 sourcepackagerelease=MatchesStructure(
709 ci_build=Equals(build),
710 files=MatchesSetwise(
711 MatchesStructure.byEquality(
712 libraryfile=artifacts[0].library_file,
713 filetype=SourcePackageFileType.SDIST))),
714 format=Equals(SourcePackageType.CI_BUILD),
715 distroseries=Equals(distroseries),
716 channel=Equals(channel))
717 for channel in ("edge", "0.0.1/edge"))))
585 bpphs = archive.getAllPublishedBinaries()718 bpphs = archive.getAllPublishedBinaries()
586 # The publications are for the same binary package release, which719 # The binary publications are for the same binary package release,
587 # has a single file attached to it.720 # which has a single file attached to it.
588 self.assertEqual(1, len({bpph.binarypackagename for bpph in bpphs}))721 self.assertEqual(1, len({bpph.binarypackagename for bpph in bpphs}))
589 self.assertEqual(1, len({bpph.binarypackagerelease for bpph in bpphs}))722 self.assertEqual(1, len({bpph.binarypackagerelease for bpph in bpphs}))
590 self.assertThat(archive.getAllPublishedBinaries(), MatchesSetwise(*(723 self.assertThat(bpphs, MatchesSetwise(*(
591 MatchesStructure(724 MatchesStructure(
592 binarypackagename=MatchesStructure.byEquality(725 binarypackagename=MatchesStructure.byEquality(
593 name="wheel-indep"),726 name="wheel-indep"),
@@ -595,7 +728,79 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
595 ci_build=Equals(build),728 ci_build=Equals(build),
596 files=MatchesSetwise(729 files=MatchesSetwise(
597 MatchesStructure.byEquality(730 MatchesStructure.byEquality(
598 libraryfile=artifact.library_file,731 libraryfile=artifacts[1].library_file,
732 filetype=BinaryPackageFileType.WHL))),
733 binarypackageformat=Equals(BinaryPackageFormat.WHL),
734 distroarchseries=Equals(das),
735 channel=Equals(channel))
736 for channel in ("edge", "0.0.1/edge"))))
737
738 def test_existing_binary_release_no_existing_source_release(self):
739 # A `CIBuildUploadJob` can be run even if the build in question was
740 # already uploaded somewhere, and in that case may add publications
741 # in other locations for the same package. This works even if there
742 # was no existing source package release (because
743 # `CIBuildUploadJob`s didn't always create one).
744 archive = self.factory.makeArchive(
745 repository_format=ArchiveRepositoryFormat.PYTHON)
746 distroseries = self.factory.makeDistroSeries(
747 distribution=archive.distribution)
748 das = self.factory.makeDistroArchSeries(distroseries=distroseries)
749 build = self.makeCIBuild(archive.distribution, distro_arch_series=das)
750 report = build.getOrCreateRevisionStatusReport("build:0")
751 sdist_path = "wheel-indep/dist/wheel-indep-0.0.1.tar.gz"
752 with open(datadir(sdist_path), mode="rb") as f:
753 report.attach(name=os.path.basename(sdist_path), data=f.read())
754 wheel_path = "wheel-indep/dist/wheel_indep-0.0.1-py3-none-any.whl"
755 with open(datadir(wheel_path), mode="rb") as f:
756 report.attach(name=os.path.basename(wheel_path), data=f.read())
757 artifacts = IStore(RevisionStatusArtifact).find(
758 RevisionStatusArtifact,
759 report=report,
760 artifact_type=RevisionStatusArtifactType.BINARY).order_by("id")
761 job = CIBuildUploadJob.create(
762 build, build.git_repository.owner, archive, distroseries,
763 PackagePublishingPocket.RELEASE, target_channel="edge")
764 transaction.commit()
765 with MockPatchObject(CIBuildUploadJob, "_uploadSources"):
766 with dbuser(job.config.dbuser):
767 JobRunner([job]).runAll()
768 job = CIBuildUploadJob.create(
769 build, build.git_repository.owner, archive, distroseries,
770 PackagePublishingPocket.RELEASE, target_channel="0.0.1/edge")
771 transaction.commit()
772
773 with dbuser(job.config.dbuser):
774 JobRunner([job]).runAll()
775
776 # There is a source publication for a new source package release.
777 self.assertThat(archive.getPublishedSources(), MatchesSetwise(
778 MatchesStructure(
779 sourcepackagename=MatchesStructure.byEquality(
780 name=build.git_repository.target.name),
781 sourcepackagerelease=MatchesStructure(
782 ci_build=Equals(build),
783 files=MatchesSetwise(
784 MatchesStructure.byEquality(
785 libraryfile=artifacts[0].library_file,
786 filetype=SourcePackageFileType.SDIST))),
787 format=Equals(SourcePackageType.CI_BUILD),
788 distroseries=Equals(distroseries),
789 channel=Equals("0.0.1/edge"))))
790 bpphs = archive.getAllPublishedBinaries()
791 # The binary publications are for the same binary package release,
792 # which has a single file attached to it.
793 self.assertEqual(1, len({bpph.binarypackagename for bpph in bpphs}))
794 self.assertEqual(1, len({bpph.binarypackagerelease for bpph in bpphs}))
795 self.assertThat(bpphs, MatchesSetwise(*(
796 MatchesStructure(
797 binarypackagename=MatchesStructure.byEquality(
798 name="wheel-indep"),
799 binarypackagerelease=MatchesStructure(
800 ci_build=Equals(build),
801 files=MatchesSetwise(
802 MatchesStructure.byEquality(
803 libraryfile=artifacts[1].library_file,
599 filetype=BinaryPackageFileType.WHL))),804 filetype=BinaryPackageFileType.WHL))),
600 binarypackageformat=Equals(BinaryPackageFormat.WHL),805 binarypackageformat=Equals(BinaryPackageFormat.WHL),
601 distroarchseries=Equals(das),806 distroarchseries=Equals(das),

Subscribers

People subscribed via source and target branches

to status/vote changes: