Merge ~cjwatson/launchpad:ci-build-upload-make-spr into launchpad:master

Proposed by Colin Watson
Status: Merged
Approved by: Colin Watson
Approved revision: 25428c91bf957de899995044925e87d9a654c618
Merge reported by: Otto Co-Pilot
Merged at revision: not available
Proposed branch: ~cjwatson/launchpad:ci-build-upload-make-spr
Merge into: launchpad:master
Prerequisite: ~cjwatson/launchpad:refactor-ci-build-upload-job-scan-file
Diff against target: 1058 lines (+522/-135)
9 files modified
lib/lp/_schema_circular_imports.py (+1/-0)
lib/lp/archivepublisher/tests/test_artifactory.py (+3/-3)
lib/lp/code/interfaces/cibuild.py (+12/-0)
lib/lp/code/model/cibuild.py (+30/-0)
lib/lp/code/model/tests/test_cibuild.py (+22/-0)
lib/lp/registry/interfaces/sourcepackage.py (+3/-3)
lib/lp/soyuz/interfaces/sourcepackagerelease.py (+12/-3)
lib/lp/soyuz/model/archivejob.py (+154/-46)
lib/lp/soyuz/tests/test_archivejob.py (+285/-80)
Reviewer Review Type Date Requested Status
Andrey Fedoseev (community) Approve
Review via email: mp+426502@code.launchpad.net

Commit message

Create a source publication when uploading a CI build

Description of the change

This has some advantages and disadvantages, but on the whole I think the advantages outweigh the disadvantages.

The downsides are:

 * if the CI build produces nothing that looks like source code at all, then we end up with a slightly odd "source package" with no files attached to it;

 * if builds do produce something that looks like source code (e.g. a Python sdist), then they're on the honour system to either produce it from only one architecture or to produce something equivalent from all architectures.

On the other hand:

 * Launchpad's data model for packages makes the assumption in many places that binary packages have an associated source package, and my attempts to make it tolerate isolated binary packages have only been marginally successful;

 * treating source-like artifacts (Python sdists, zip files of Go source code, etc.) as source packages rather than binary packages makes more sense to humans;

 * source packages provide a useful grouping for collections of binary packages from builds of the same version;

 * creating source publications means that the PPA web UI works for PPAs populated from CI builds with only a few small adjustments (not in this branch);

 * creating source publications means that it should be straightforward to get the existing package copying mechanism to work.

I've slightly cheekily renamed `SourcePackageType.SDIST` to `SourcePackageType.CI_BUILD`. The type of an individual source file is represented separately using `SourcePackageFileType`, and since the data model's assumptions generally involve a single source with zero or more binaries, it's simpler for `SourcePackageRelease.format` not to need to change depending on the types of the individual files it contains. There was previously no way to create a source package with `format=SDIST` outside the test suite, and a query on staging returns zero rows.

As a concrete example of a source package file type and in order to allow more useful testing, I've added support for uploading Python sdists from CI builds as part of this branch.

To post a comment you must log in.
Revision history for this message
Andrey Fedoseev (andrey-fedoseev) wrote :

This looks good.

I'd like the artifact metadata to be a typed structure (NamedTuple) rather than a dict.

review: Approve
Revision history for this message
Colin Watson (cjwatson) wrote :

I find namedtuples awkwardly inflexible in general and usually regret using them; in this case, they aren't really suitable since they don't support optional arguments on construction. I get the general idea, though, so I looked for alternatives. A dataclass would be ideal, but those don't exist until Python 3.7. Type annotations on class attributes don't work in Python 3.6, not even the old comment style unless the attribute has an initializer.

However, `attrs` (https://www.attrs.org/) *does* work, and although Launchpad doesn't currently use it directly, it's already in our virtualenv as an indirect dependency. It's a pretty well-regarded package, it can be integrated with `mypy` (https://www.attrs.org/en/stable/types.html#mypy), it gives us an automatic `__init__` method and such, and it would be easy to convert from a simple use of it to a dataclass once we're on Python >= 3.7.

I've reworked this branch using that. Would you mind having another look?

Revision history for this message
Jürgen Gmach (jugmac00) wrote (last edit ):

I am with Andrey in that we should not use a dictionary.

While attrs is a great library, I am -0.5 on introducing it.

Introducing a new dependency
- which it is only used for two classes
- only for a short interim period
- with a new "syntax"
- which is even dated, ie it is not reflected in the main documentation of the package - you need to dive into its history: https://www.attrs.org/en/stable/names.html#a-short-history-lesson

I am strongly in favor of using plain classes - with that usual boilerplate - and convert them to dataclasses once we are on a suitable Python version.

If you go forward using it, please leave a comment to remove it asap.

Revision history for this message
Andrey Fedoseev (andrey-fedoseev) wrote :

I'm not in favor of using `attrs` either.

I suggested to use named tuples because they can be automatically converted to 3.6 syntax with `pyupgrade` (https://github.com/asottile/pyupgrade#typingnamedtuple--typingtypeddict-py36-syntax) and left as is, or converted to `dataclass` with minimal manual work.

Revision history for this message
Colin Watson (cjwatson) wrote :

I really don't think namedtuples are suitable here because of the lack of support for optional items in their constructors.

For plain classes, how do you propose spelling them? This doesn't work for obvious reasons:

    class SourceArtifactMetadata:
        format # type: SourcePackageFileType
        name # type: str
        version # type: str

And if I use something like `format = None` there, then they all have to be `Optional[...]` when they really shouldn't be optional.

Revision history for this message
Andrey Fedoseev (andrey-fedoseev) wrote :

> I really don't think namedtuples are suitable here because of the lack of support for optional items in their constructors.

You're right, the optional values are available only since 3.7

> For plain classes, how do you propose spelling them?

I think you could use the constructor for that:

    class SourceArtifactMetadata:

        def __init__(self, format: SourcePackageFileType, name: str, version: str):
            self.format = format
            self.name = name
            self.version = version

Revision history for this message
Colin Watson (cjwatson) wrote :

OK, I've converted this to a plain class with `__init__` boilerplate as you suggest. How's this?

Revision history for this message
Andrey Fedoseev (andrey-fedoseev) wrote :

> OK, I've converted this to a plain class with `__init__` boilerplate as you suggest. How's this?

This looks great

PyCharm reports two issues:

1. The signature of `CIBuildUploadJob.create()` doesn't match the base`ArchiveJobDerived.create()` - this may be ok
2. `scanned` argument in `_uploadSources` should probably be a `Sequence` rather than an `Iterable` because you're doing `scanned[0]` there

Revision history for this message
Colin Watson (cjwatson) wrote :

The `create` method thing is at least intentional for now, if possibly not ideal (the whole derived jobs business is a bit unnecessarily complicated, I think).

I've fixed the `scanned` argument type.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/lib/lp/_schema_circular_imports.py b/lib/lp/_schema_circular_imports.py
2index e0bd33a..677163b 100644
3--- a/lib/lp/_schema_circular_imports.py
4+++ b/lib/lp/_schema_circular_imports.py
5@@ -335,6 +335,7 @@ patch_reference_property(IPackageUpload, 'copy_source_archive', IArchive)
6 patch_reference_property(
7 ISourcePackageRelease, 'source_package_recipe_build',
8 ISourcePackageRecipeBuild)
9+patch_reference_property(ISourcePackageRelease, 'ci_build', ICIBuild)
10
11 # IIndexedMessage
12 patch_reference_property(IIndexedMessage, 'inside', IBugTask)
13diff --git a/lib/lp/archivepublisher/tests/test_artifactory.py b/lib/lp/archivepublisher/tests/test_artifactory.py
14index c9feb58..18af683 100644
15--- a/lib/lp/archivepublisher/tests/test_artifactory.py
16+++ b/lib/lp/archivepublisher/tests/test_artifactory.py
17@@ -576,7 +576,7 @@ class TestArtifactoryPoolFromLibrarian(TestCaseWithFactory):
18 sourcepackagename="foo",
19 version="1.0",
20 channel="edge",
21- format=SourcePackageType.SDIST,
22+ format=SourcePackageType.CI_BUILD,
23 )
24 spr = spph.sourcepackagerelease
25 sprf = self.factory.makeSourcePackageReleaseFile(
26@@ -635,7 +635,7 @@ class TestArtifactoryPoolFromLibrarian(TestCaseWithFactory):
27 archive=pool.archive,
28 sourcepackagename="foo",
29 version="1.0",
30- format=SourcePackageType.SDIST,
31+ format=SourcePackageType.CI_BUILD,
32 )
33 bpph = self.factory.makeBinaryPackagePublishingHistory(
34 archive=pool.archive,
35@@ -940,7 +940,7 @@ class TestArtifactoryPoolFromLibrarian(TestCaseWithFactory):
36 archive=pool.archive,
37 sourcepackagename="foo",
38 version="1.0",
39- format=SourcePackageType.SDIST,
40+ format=SourcePackageType.CI_BUILD,
41 )
42 bpph = self.factory.makeBinaryPackagePublishingHistory(
43 archive=pool.archive,
44diff --git a/lib/lp/code/interfaces/cibuild.py b/lib/lp/code/interfaces/cibuild.py
45index 69c765b..d929283 100644
46--- a/lib/lp/code/interfaces/cibuild.py
47+++ b/lib/lp/code/interfaces/cibuild.py
48@@ -133,6 +133,10 @@ class ICIBuildView(IPackageBuildView, IPrivacy):
49 "A mapping from job IDs to result tokens, retrieved from the "
50 "builder.")))
51
52+ sourcepackages = Attribute(
53+ "A list of source packages that resulted from this build, ordered by "
54+ "name.")
55+
56 binarypackages = Attribute(
57 "A list of binary packages that resulted from this build, ordered by "
58 "name.")
59@@ -182,6 +186,14 @@ class ICIBuildView(IPackageBuildView, IPrivacy):
60 :return: A collection of URLs for this build.
61 """
62
63+ def createSourcePackageRelease(
64+ distroseries, sourcepackagename, version, creator=None,
65+ archive=None):
66+ """Create and return a `SourcePackageRelease` for this CI build.
67+
68+ The new source package release will be linked to this build.
69+ """
70+
71 def createBinaryPackageRelease(
72 binarypackagename, version, summary, description, binpackageformat,
73 architecturespecific, installedsize=None, homepage=None,
74diff --git a/lib/lp/code/model/cibuild.py b/lib/lp/code/model/cibuild.py
75index 76ccfc3..281c3a1 100644
76--- a/lib/lp/code/model/cibuild.py
77+++ b/lib/lp/code/model/cibuild.py
78@@ -65,8 +65,10 @@ from lp.code.model.gitref import GitRef
79 from lp.code.model.lpcraft import load_configuration
80 from lp.registry.interfaces.pocket import PackagePublishingPocket
81 from lp.registry.interfaces.series import SeriesStatus
82+from lp.registry.interfaces.sourcepackage import SourcePackageType
83 from lp.registry.model.distribution import Distribution
84 from lp.registry.model.distroseries import DistroSeries
85+from lp.registry.model.sourcepackagename import SourcePackageName
86 from lp.services.database.bulk import load_related
87 from lp.services.database.constants import DEFAULT
88 from lp.services.database.decoratedresultset import DecoratedResultSet
89@@ -91,6 +93,7 @@ from lp.services.propertycache import cachedproperty
90 from lp.soyuz.model.binarypackagename import BinaryPackageName
91 from lp.soyuz.model.binarypackagerelease import BinaryPackageRelease
92 from lp.soyuz.model.distroarchseries import DistroArchSeries
93+from lp.soyuz.model.sourcepackagerelease import SourcePackageRelease
94
95
96 def get_stages(configuration):
97@@ -471,6 +474,17 @@ class CIBuild(PackageBuildMixin, StormBase):
98 # We don't currently send any notifications.
99
100 @property
101+ def sourcepackages(self):
102+ """See `ICIBuild`."""
103+ releases = IStore(SourcePackageRelease).find(
104+ (SourcePackageRelease, SourcePackageName),
105+ SourcePackageRelease.ci_build == self,
106+ SourcePackageRelease.sourcepackagename == SourcePackageName.id)
107+ releases = releases.order_by(
108+ SourcePackageName.name, SourcePackageRelease.id)
109+ return DecoratedResultSet(releases, result_decorator=itemgetter(0))
110+
111+ @property
112 def binarypackages(self):
113 """See `ICIBuild`."""
114 releases = IStore(BinaryPackageRelease).find(
115@@ -481,6 +495,22 @@ class CIBuild(PackageBuildMixin, StormBase):
116 BinaryPackageName.name, BinaryPackageRelease.id)
117 return DecoratedResultSet(releases, result_decorator=itemgetter(0))
118
119+ def createSourcePackageRelease(
120+ self, distroseries, sourcepackagename, version, creator=None,
121+ archive=None):
122+ """See `ICIBuild`."""
123+ return distroseries.createUploadedSourcePackageRelease(
124+ sourcepackagename=sourcepackagename,
125+ version=version,
126+ format=SourcePackageType.CI_BUILD,
127+ # This doesn't really make sense for SPRs created for CI builds,
128+ # but the column is NOT NULL. The empty string will do though,
129+ # since nothing will use this.
130+ architecturehintlist="",
131+ creator=creator,
132+ archive=archive,
133+ ci_build=self)
134+
135 def createBinaryPackageRelease(
136 self, binarypackagename, version, summary, description,
137 binpackageformat, architecturespecific, installedsize=None,
138diff --git a/lib/lp/code/model/tests/test_cibuild.py b/lib/lp/code/model/tests/test_cibuild.py
139index 89a432c..ca0192b 100644
140--- a/lib/lp/code/model/tests/test_cibuild.py
141+++ b/lib/lp/code/model/tests/test_cibuild.py
142@@ -62,6 +62,7 @@ from lp.code.model.cibuild import (
143 from lp.code.model.lpcraft import load_configuration
144 from lp.code.tests.helpers import GitHostingFixture
145 from lp.registry.interfaces.series import SeriesStatus
146+from lp.registry.interfaces.sourcepackage import SourcePackageType
147 from lp.services.authserver.xmlrpc import AuthServerAPIView
148 from lp.services.config import config
149 from lp.services.librarian.browser import ProxiedLibraryFileAlias
150@@ -400,6 +401,27 @@ class TestCIBuild(TestCaseWithFactory):
151 commit_sha1=build.commit_sha1,
152 ci_build=build))
153
154+ def test_createSourcePackageRelease(self):
155+ distroseries = self.factory.makeDistroSeries()
156+ archive = self.factory.makeArchive(
157+ distribution=distroseries.distribution)
158+ build = self.factory.makeCIBuild()
159+ spn = self.factory.makeSourcePackageName()
160+ spr = build.createSourcePackageRelease(
161+ distroseries, spn, "1.0", creator=build.git_repository.owner,
162+ archive=archive)
163+ self.assertThat(spr, MatchesStructure(
164+ upload_distroseries=Equals(distroseries),
165+ sourcepackagename=Equals(spn),
166+ version=Equals("1.0"),
167+ format=Equals(SourcePackageType.CI_BUILD),
168+ architecturehintlist=Equals(""),
169+ creator=Equals(build.git_repository.owner),
170+ upload_archive=Equals(archive),
171+ ci_build=Equals(build),
172+ ))
173+ self.assertContentEqual([spr], build.sourcepackages)
174+
175 def test_createBinaryPackageRelease(self):
176 build = self.factory.makeCIBuild()
177 bpn = self.factory.makeBinaryPackageName()
178diff --git a/lib/lp/registry/interfaces/sourcepackage.py b/lib/lp/registry/interfaces/sourcepackage.py
179index fcde488..6a7e707 100644
180--- a/lib/lp/registry/interfaces/sourcepackage.py
181+++ b/lib/lp/registry/interfaces/sourcepackage.py
182@@ -456,10 +456,10 @@ class SourcePackageType(DBEnumeratedType):
183 This is the source package format used by Gentoo.
184 """)
185
186- SDIST = DBItem(4, """
187- The Python Format
188+ CI_BUILD = DBItem(4, """
189+ CI Build
190
191- This is the source package format used by Python packages.
192+ An ad-hoc source package generated by a CI build.
193 """)
194
195
196diff --git a/lib/lp/soyuz/interfaces/sourcepackagerelease.py b/lib/lp/soyuz/interfaces/sourcepackagerelease.py
197index 8e6cee9..aac4f39 100644
198--- a/lib/lp/soyuz/interfaces/sourcepackagerelease.py
199+++ b/lib/lp/soyuz/interfaces/sourcepackagerelease.py
200@@ -145,11 +145,20 @@ class ISourcePackageRelease(Interface):
201 # Really ISourcePackageRecipeBuild -- see _schema_circular_imports.
202 source_package_recipe_build = Reference(
203 schema=Interface,
204- description=_("The `SourcePackageRecipeBuild` which produced this "
205- "source package release, or None if it was created from a "
206- "traditional upload."),
207+ description=_(
208+ "The `SourcePackageRecipeBuild` which produced this source "
209+ "package release, or None if it was not created from a source "
210+ "package recipe."),
211 title=_("Source package recipe build"),
212 required=False, readonly=True)
213+ # Really ICIBuild, patched in _schema_circular_imports.
214+ ci_build = Reference(
215+ schema=Interface,
216+ description=_(
217+ "The `CIBuild` which produced this source package release, or "
218+ "None if it was not created from a CI build."),
219+ title=_("CI build"),
220+ required=False, readonly=True)
221
222 def getUserDefinedField(name):
223 """Case-insensitively get a user-defined field."""
224diff --git a/lib/lp/soyuz/model/archivejob.py b/lib/lp/soyuz/model/archivejob.py
225index d62f199..ae038a3 100644
226--- a/lib/lp/soyuz/model/archivejob.py
227+++ b/lib/lp/soyuz/model/archivejob.py
228@@ -13,11 +13,18 @@ from typing import (
229 Dict,
230 Iterable,
231 List,
232+ Optional,
233+ Sequence,
234+ Tuple,
235+ Union,
236 )
237 import zipfile
238
239 from lazr.delegates import delegate_to
240-from pkginfo import Wheel
241+from pkginfo import (
242+ SDist,
243+ Wheel,
244+ )
245 from storm.expr import And
246 from storm.locals import (
247 Int,
248@@ -43,6 +50,7 @@ from lp.registry.interfaces.distributionsourcepackage import (
249 )
250 from lp.registry.interfaces.distroseries import IDistroSeriesSet
251 from lp.registry.interfaces.pocket import PackagePublishingPocket
252+from lp.registry.interfaces.sourcepackage import SourcePackageFileType
253 from lp.services.config import config
254 from lp.services.database.enumcol import DBEnum
255 from lp.services.database.interfaces import IMasterStore
256@@ -212,10 +220,39 @@ class ScanException(Exception):
257 """A CI build upload job failed to scan a file."""
258
259
260+class SourceArtifactMetadata:
261+ """Metadata extracted from a source package."""
262+
263+ def __init__(self, format: SourcePackageFileType, name: str, version: str):
264+ self.format = format
265+ self.name = name
266+ self.version = version
267+
268+
269+class BinaryArtifactMetadata:
270+ """Metadata extracted from a binary package."""
271+
272+ def __init__(self, format: BinaryPackageFormat, name: str, version: str,
273+ summary: str, description: str, architecturespecific: bool,
274+ homepage: str,
275+ user_defined_fields: Optional[List[Tuple[str, str]]] = None):
276+ self.format = format
277+ self.name = name
278+ self.version = version
279+ self.summary = summary
280+ self.description = description
281+ self.architecturespecific = architecturespecific
282+ self.homepage = homepage
283+ self.user_defined_fields = user_defined_fields
284+
285+
286+ArtifactMetadata = Union[SourceArtifactMetadata, BinaryArtifactMetadata]
287+
288+
289 class ScannedArtifact:
290
291 def __init__(
292- self, *, artifact: IRevisionStatusArtifact, metadata: Dict[str, Any]
293+ self, *, artifact: IRevisionStatusArtifact, metadata: ArtifactMetadata
294 ):
295 self.artifact = artifact
296 self.metadata = metadata
297@@ -247,13 +284,16 @@ class CIBuildUploadJob(ArchiveJobDerived):
298
299 # We're only interested in uploading certain kinds of packages to
300 # certain kinds of archives.
301- binary_format_by_repository_format = {
302+ format_by_repository_format = {
303 ArchiveRepositoryFormat.DEBIAN: {
304 BinaryPackageFormat.DEB,
305 BinaryPackageFormat.UDEB,
306 BinaryPackageFormat.DDEB,
307 },
308- ArchiveRepositoryFormat.PYTHON: {BinaryPackageFormat.WHL},
309+ ArchiveRepositoryFormat.PYTHON: {
310+ SourcePackageFileType.SDIST,
311+ BinaryPackageFormat.WHL,
312+ },
313 ArchiveRepositoryFormat.CONDA: {
314 BinaryPackageFormat.CONDA_V1,
315 BinaryPackageFormat.CONDA_V2,
316@@ -326,7 +366,7 @@ class CIBuildUploadJob(ArchiveJobDerived):
317 def target_channel(self):
318 return self.metadata["target_channel"]
319
320- def _scanWheel(self, path: str) -> Dict[str, Any]:
321+ def _scanWheel(self, path: str) -> Optional[BinaryArtifactMetadata]:
322 try:
323 parsed_path = parse_wheel_filename(path)
324 wheel = Wheel(path)
325@@ -335,34 +375,50 @@ class CIBuildUploadJob(ArchiveJobDerived):
326 "Failed to scan %s as a Python wheel: %s",
327 os.path.basename(path), e)
328 return None
329- return {
330- "name": wheel.name,
331- "version": wheel.version,
332- "summary": wheel.summary or "",
333- "description": wheel.description,
334- "binpackageformat": BinaryPackageFormat.WHL,
335- "architecturespecific": "any" not in parsed_path.platform_tags,
336- "homepage": wheel.home_page or "",
337- }
338+ return BinaryArtifactMetadata(
339+ format=BinaryPackageFormat.WHL,
340+ name=wheel.name,
341+ version=wheel.version,
342+ summary=wheel.summary or "",
343+ description=wheel.description,
344+ architecturespecific="any" not in parsed_path.platform_tags,
345+ homepage=wheel.home_page or "",
346+ )
347+
348+ def _scanSDist(self, path: str) -> Optional[SourceArtifactMetadata]:
349+ try:
350+ sdist = SDist(path)
351+ except Exception as e:
352+ logger.warning(
353+ "Failed to scan %s as a Python sdist: %s",
354+ os.path.basename(path), e)
355+ return None
356+ return SourceArtifactMetadata(
357+ format=SourcePackageFileType.SDIST,
358+ name=sdist.name,
359+ version=sdist.version,
360+ )
361
362 def _scanCondaMetadata(
363- self, index: Dict[Any, Any], about: Dict[Any, Any]
364- ) -> Dict[str, Any]:
365- return {
366- "name": index["name"],
367- "version": index["version"],
368- "summary": about.get("summary", ""),
369- "description": about.get("description", ""),
370- "architecturespecific": index["platform"] is not None,
371- "homepage": about.get("home", ""),
372+ self, format: BinaryPackageFormat, index: Dict[Any, Any],
373+ about: Dict[Any, Any]
374+ ) -> Optional[BinaryArtifactMetadata]:
375+ return BinaryArtifactMetadata(
376+ format=format,
377+ name=index["name"],
378+ version=index["version"],
379+ summary=about.get("summary", ""),
380+ description=about.get("description", ""),
381+ architecturespecific=index["platform"] is not None,
382+ homepage=about.get("home", ""),
383 # We should perhaps model this explicitly since it's used by the
384 # publisher, but this gives us an easy way to pass this through
385 # without needing to add a column to a large table that's only
386 # relevant to a tiny minority of rows.
387- "user_defined_fields": [("subdir", index["subdir"])],
388- }
389+ user_defined_fields=[("subdir", index["subdir"])],
390+ )
391
392- def _scanCondaV1(self, path: str) -> Dict[str, Any]:
393+ def _scanCondaV1(self, path: str) -> Optional[BinaryArtifactMetadata]:
394 try:
395 with tarfile.open(path) as tar:
396 index = json.loads(
397@@ -374,11 +430,10 @@ class CIBuildUploadJob(ArchiveJobDerived):
398 "Failed to scan %s as a Conda v1 package: %s",
399 os.path.basename(path), e)
400 return None
401- scanned = {"binpackageformat": BinaryPackageFormat.CONDA_V1}
402- scanned.update(self._scanCondaMetadata(index, about))
403- return scanned
404+ return self._scanCondaMetadata(
405+ BinaryPackageFormat.CONDA_V1, index, about)
406
407- def _scanCondaV2(self, path: str) -> Dict[str, Any]:
408+ def _scanCondaV2(self, path: str) -> Optional[BinaryArtifactMetadata]:
409 try:
410 with zipfile.ZipFile(path) as zipf:
411 base_name = os.path.basename(path)[:-len(".conda")]
412@@ -396,13 +451,14 @@ class CIBuildUploadJob(ArchiveJobDerived):
413 "Failed to scan %s as a Conda v2 package: %s",
414 os.path.basename(path), e)
415 return None
416- scanned = {"binpackageformat": BinaryPackageFormat.CONDA_V2}
417- scanned.update(self._scanCondaMetadata(index, about))
418- return scanned
419+ return self._scanCondaMetadata(
420+ BinaryPackageFormat.CONDA_V2, index, about)
421
422- def _scanFile(self, path: str) -> Dict[str, Any]:
423+ def _scanFile(self, path: str) -> Optional[ArtifactMetadata]:
424 _scanners = (
425 (".whl", self._scanWheel),
426+ (".tar.gz", self._scanSDist),
427+ (".zip", self._scanSDist),
428 (".tar.bz2", self._scanCondaV1),
429 (".conda", self._scanCondaV2),
430 )
431@@ -429,8 +485,8 @@ class CIBuildUploadJob(ArchiveJobDerived):
432 Returns a list of `ScannedArtifact`s containing metadata for
433 relevant artifacts.
434 """
435- allowed_binary_formats = (
436- self.binary_format_by_repository_format.get(
437+ allowed_formats = (
438+ self.format_by_repository_format.get(
439 self.archive.repository_format, set()))
440 scanned = []
441 with tempfile.TemporaryDirectory(prefix="ci-build-copy-job") as tmpdir:
442@@ -444,7 +500,7 @@ class CIBuildUploadJob(ArchiveJobDerived):
443 metadata = self._scanFile(contents)
444 if metadata is None:
445 continue
446- if metadata["binpackageformat"] not in allowed_binary_formats:
447+ if metadata.format not in allowed_formats:
448 logger.info(
449 "Skipping %s (not relevant to %s archives)",
450 name, self.archive.repository_format)
451@@ -453,28 +509,79 @@ class CIBuildUploadJob(ArchiveJobDerived):
452 ScannedArtifact(artifact=artifact, metadata=metadata))
453 return scanned
454
455+ def _uploadSources(self, scanned: Sequence[ScannedArtifact]) -> None:
456+ """Upload sources from an sequence of `ScannedArtifact`s."""
457+ # Launchpad's data model generally assumes that a single source is
458+ # associated with multiple binaries. However, a source package
459+ # release can have multiple (or indeed no) files attached to it, so
460+ # we make use of that if necessary.
461+ releases = {
462+ release.sourcepackagename: release
463+ for release in self.ci_build.sourcepackages}
464+ distroseries = self.ci_build.distro_arch_series.distroseries
465+ build_target = self.ci_build.git_repository.target
466+ spr = releases.get(build_target.sourcepackagename)
467+ if spr is None:
468+ spr = self.ci_build.createSourcePackageRelease(
469+ distroseries=distroseries,
470+ sourcepackagename=build_target.sourcepackagename,
471+ # We don't have a good concept of source version here, but
472+ # the data model demands one. Arbitrarily pick the version
473+ # of the first scanned artifact.
474+ version=scanned[0].metadata.version,
475+ creator=self.requester,
476+ archive=self.archive)
477+ for scanned_artifact in scanned:
478+ metadata = scanned_artifact.metadata
479+ if not isinstance(metadata, SourceArtifactMetadata):
480+ continue
481+ library_file = scanned_artifact.artifact.library_file
482+ logger.info(
483+ "Uploading %s to %s %s (%s)",
484+ library_file.filename, self.archive.reference,
485+ self.target_distroseries.getSuite(self.target_pocket),
486+ self.target_channel)
487+ for sprf in spr.files:
488+ if (sprf.libraryfile == library_file and
489+ sprf.filetype == metadata.format):
490+ break
491+ else:
492+ spr.addFile(library_file, filetype=metadata.format)
493+ getUtility(IPublishingSet).newSourcePublication(
494+ archive=self.archive, sourcepackagerelease=spr,
495+ distroseries=self.target_distroseries, pocket=self.target_pocket,
496+ creator=self.requester, channel=self.target_channel)
497+
498 def _uploadBinaries(self, scanned: Iterable[ScannedArtifact]) -> None:
499- """Upload an iterable of `ScannedArtifact`s to an archive."""
500+ """Upload binaries from an iterable of `ScannedArtifact`s."""
501 releases = {
502 (release.binarypackagename, release.binpackageformat): release
503 for release in self.ci_build.binarypackages}
504 binaries = OrderedDict()
505 for scanned_artifact in scanned:
506+ metadata = scanned_artifact.metadata
507+ if not isinstance(metadata, BinaryArtifactMetadata):
508+ continue
509 library_file = scanned_artifact.artifact.library_file
510- metadata = dict(scanned_artifact.metadata)
511- binpackageformat = metadata["binpackageformat"]
512 logger.info(
513 "Uploading %s to %s %s (%s)",
514 library_file.filename, self.archive.reference,
515 self.target_distroseries.getSuite(self.target_pocket),
516 self.target_channel)
517- metadata["binarypackagename"] = bpn = (
518- getUtility(IBinaryPackageNameSet).ensure(metadata["name"]))
519- del metadata["name"]
520- filetype = self.filetype_by_format[binpackageformat]
521- bpr = releases.get((bpn, binpackageformat))
522+ binarypackagename = getUtility(IBinaryPackageNameSet).ensure(
523+ metadata.name)
524+ filetype = self.filetype_by_format[metadata.format]
525+ bpr = releases.get((binarypackagename, metadata.format))
526 if bpr is None:
527- bpr = self.ci_build.createBinaryPackageRelease(**metadata)
528+ bpr = self.ci_build.createBinaryPackageRelease(
529+ binarypackagename=binarypackagename,
530+ version=metadata.version,
531+ summary=metadata.summary,
532+ description=metadata.description,
533+ binpackageformat=metadata.format,
534+ architecturespecific=metadata.architecturespecific,
535+ homepage=metadata.homepage,
536+ user_defined_fields=metadata.user_defined_fields)
537 for bpf in bpr.files:
538 if (bpf.libraryfile == library_file and
539 bpf.filetype == filetype):
540@@ -503,6 +610,7 @@ class CIBuildUploadJob(ArchiveJobDerived):
541 self.ci_build)
542 scanned = self._scanArtifacts(artifacts)
543 if scanned:
544+ self._uploadSources(scanned)
545 self._uploadBinaries(scanned)
546 else:
547 names = [artifact.library_file.filename for artifact in artifacts]
548diff --git a/lib/lp/soyuz/tests/test_archivejob.py b/lib/lp/soyuz/tests/test_archivejob.py
549index b744bc8..813b423 100644
550--- a/lib/lp/soyuz/tests/test_archivejob.py
551+++ b/lib/lp/soyuz/tests/test_archivejob.py
552@@ -7,6 +7,7 @@ from debian.deb822 import Changes
553 from fixtures import (
554 FakeLogger,
555 MockPatch,
556+ MockPatchObject,
557 )
558 from testtools.matchers import (
559 ContainsDict,
560@@ -20,6 +21,10 @@ import transaction
561 from lp.code.enums import RevisionStatusArtifactType
562 from lp.code.model.revisionstatus import RevisionStatusArtifact
563 from lp.registry.interfaces.pocket import PackagePublishingPocket
564+from lp.registry.interfaces.sourcepackage import (
565+ SourcePackageFileType,
566+ SourcePackageType,
567+ )
568 from lp.services.config import config
569 from lp.services.database.interfaces import IStore
570 from lp.services.features.testing import FeatureFixture
571@@ -41,8 +46,10 @@ from lp.soyuz.enums import (
572 from lp.soyuz.model.archivejob import (
573 ArchiveJob,
574 ArchiveJobDerived,
575+ BinaryArtifactMetadata,
576 CIBuildUploadJob,
577 PackageUploadNotificationJob,
578+ SourceArtifactMetadata,
579 )
580 from lp.soyuz.tests import datadir
581 from lp.testing import (
582@@ -242,16 +249,19 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
583 build, build.git_repository.owner, archive, distroseries,
584 PackagePublishingPocket.RELEASE, target_channel="edge")
585 path = "wheel-indep/dist/wheel_indep-0.0.1-py3-none-any.whl"
586- expected = {
587- "name": "wheel-indep",
588- "version": "0.0.1",
589- "summary": "Example description",
590- "description": "Example long description\n",
591- "binpackageformat": BinaryPackageFormat.WHL,
592- "architecturespecific": False,
593- "homepage": "",
594- }
595- self.assertEqual(expected, job._scanFile(datadir(path)))
596+ metadata = job._scanFile(datadir(path))
597+ self.assertIsInstance(metadata, BinaryArtifactMetadata)
598+ self.assertThat(
599+ metadata,
600+ MatchesStructure(
601+ format=Equals(BinaryPackageFormat.WHL),
602+ name=Equals("wheel-indep"),
603+ version=Equals("0.0.1"),
604+ summary=Equals("Example description"),
605+ description=Equals("Example long description\n"),
606+ architecturespecific=Is(False),
607+ homepage=Equals(""),
608+ ))
609
610 def test__scanFile_wheel_arch(self):
611 archive = self.factory.makeArchive()
612@@ -262,16 +272,38 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
613 build, build.git_repository.owner, archive, distroseries,
614 PackagePublishingPocket.RELEASE, target_channel="edge")
615 path = "wheel-arch/dist/wheel_arch-0.0.1-cp310-cp310-linux_x86_64.whl"
616- expected = {
617- "name": "wheel-arch",
618- "version": "0.0.1",
619- "summary": "Example description",
620- "description": "Example long description\n",
621- "binpackageformat": BinaryPackageFormat.WHL,
622- "architecturespecific": True,
623- "homepage": "http://example.com/",
624- }
625- self.assertEqual(expected, job._scanFile(datadir(path)))
626+ metadata = job._scanFile(datadir(path))
627+ self.assertIsInstance(metadata, BinaryArtifactMetadata)
628+ self.assertThat(
629+ metadata,
630+ MatchesStructure(
631+ format=Equals(BinaryPackageFormat.WHL),
632+ name=Equals("wheel-arch"),
633+ version=Equals("0.0.1"),
634+ summary=Equals("Example description"),
635+ description=Equals("Example long description\n"),
636+ architecturespecific=Is(True),
637+ homepage=Equals("http://example.com/"),
638+ ))
639+
640+ def test__scanFile_sdist(self):
641+ archive = self.factory.makeArchive()
642+ distroseries = self.factory.makeDistroSeries(
643+ distribution=archive.distribution)
644+ build = self.makeCIBuild(archive.distribution)
645+ job = CIBuildUploadJob.create(
646+ build, build.git_repository.owner, archive, distroseries,
647+ PackagePublishingPocket.RELEASE, target_channel="edge")
648+ path = "wheel-arch/dist/wheel-arch-0.0.1.tar.gz"
649+ metadata = job._scanFile(datadir(path))
650+ self.assertIsInstance(metadata, SourceArtifactMetadata)
651+ self.assertThat(
652+ metadata,
653+ MatchesStructure.byEquality(
654+ format=SourcePackageFileType.SDIST,
655+ name="wheel-arch",
656+ version="0.0.1",
657+ ))
658
659 def test__scanFile_conda_indep(self):
660 archive = self.factory.makeArchive()
661@@ -282,17 +314,20 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
662 build, build.git_repository.owner, archive, distroseries,
663 PackagePublishingPocket.RELEASE, target_channel="edge")
664 path = "conda-indep/dist/noarch/conda-indep-0.1-0.tar.bz2"
665- expected = {
666- "name": "conda-indep",
667- "version": "0.1",
668- "summary": "Example summary",
669- "description": "Example description",
670- "binpackageformat": BinaryPackageFormat.CONDA_V1,
671- "architecturespecific": False,
672- "homepage": "",
673- "user_defined_fields": [("subdir", "noarch")],
674- }
675- self.assertEqual(expected, job._scanFile(datadir(path)))
676+ metadata = job._scanFile(datadir(path))
677+ self.assertIsInstance(metadata, BinaryArtifactMetadata)
678+ self.assertThat(
679+ metadata,
680+ MatchesStructure(
681+ format=Equals(BinaryPackageFormat.CONDA_V1),
682+ name=Equals("conda-indep"),
683+ version=Equals("0.1"),
684+ summary=Equals("Example summary"),
685+ description=Equals("Example description"),
686+ architecturespecific=Is(False),
687+ homepage=Equals(""),
688+ user_defined_fields=Equals([("subdir", "noarch")]),
689+ ))
690
691 def test__scanFile_conda_arch(self):
692 archive = self.factory.makeArchive()
693@@ -303,17 +338,20 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
694 build, build.git_repository.owner, archive, distroseries,
695 PackagePublishingPocket.RELEASE, target_channel="edge")
696 path = "conda-arch/dist/linux-64/conda-arch-0.1-0.tar.bz2"
697- expected = {
698- "name": "conda-arch",
699- "version": "0.1",
700- "summary": "Example summary",
701- "description": "Example description",
702- "binpackageformat": BinaryPackageFormat.CONDA_V1,
703- "architecturespecific": True,
704- "homepage": "http://example.com/",
705- "user_defined_fields": [("subdir", "linux-64")],
706- }
707- self.assertEqual(expected, job._scanFile(datadir(path)))
708+ metadata = job._scanFile(datadir(path))
709+ self.assertIsInstance(metadata, BinaryArtifactMetadata)
710+ self.assertThat(
711+ metadata,
712+ MatchesStructure(
713+ format=Equals(BinaryPackageFormat.CONDA_V1),
714+ name=Equals("conda-arch"),
715+ version=Equals("0.1"),
716+ summary=Equals("Example summary"),
717+ description=Equals("Example description"),
718+ architecturespecific=Is(True),
719+ homepage=Equals("http://example.com/"),
720+ user_defined_fields=Equals([("subdir", "linux-64")]),
721+ ))
722
723 def test__scanFile_conda_v2_indep(self):
724 archive = self.factory.makeArchive()
725@@ -324,17 +362,20 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
726 build, build.git_repository.owner, archive, distroseries,
727 PackagePublishingPocket.RELEASE, target_channel="edge")
728 path = "conda-v2-indep/dist/noarch/conda-v2-indep-0.1-0.conda"
729- expected = {
730- "name": "conda-v2-indep",
731- "version": "0.1",
732- "summary": "Example summary",
733- "description": "Example description",
734- "binpackageformat": BinaryPackageFormat.CONDA_V2,
735- "architecturespecific": False,
736- "homepage": "",
737- "user_defined_fields": [("subdir", "noarch")],
738- }
739- self.assertEqual(expected, job._scanFile(datadir(path)))
740+ metadata = job._scanFile(datadir(path))
741+ self.assertIsInstance(metadata, BinaryArtifactMetadata)
742+ self.assertThat(
743+ metadata,
744+ MatchesStructure(
745+ format=Equals(BinaryPackageFormat.CONDA_V2),
746+ name=Equals("conda-v2-indep"),
747+ version=Equals("0.1"),
748+ summary=Equals("Example summary"),
749+ description=Equals("Example description"),
750+ architecturespecific=Is(False),
751+ homepage=Equals(""),
752+ user_defined_fields=Equals([("subdir", "noarch")]),
753+ ))
754
755 def test__scanFile_conda_v2_arch(self):
756 archive = self.factory.makeArchive()
757@@ -345,17 +386,20 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
758 build, build.git_repository.owner, archive, distroseries,
759 PackagePublishingPocket.RELEASE, target_channel="edge")
760 path = "conda-v2-arch/dist/linux-64/conda-v2-arch-0.1-0.conda"
761- expected = {
762- "name": "conda-v2-arch",
763- "version": "0.1",
764- "summary": "Example summary",
765- "description": "Example description",
766- "binpackageformat": BinaryPackageFormat.CONDA_V2,
767- "architecturespecific": True,
768- "homepage": "http://example.com/",
769- "user_defined_fields": [("subdir", "linux-64")],
770- }
771- self.assertEqual(expected, job._scanFile(datadir(path)))
772+ metadata = job._scanFile(datadir(path))
773+ self.assertIsInstance(metadata, BinaryArtifactMetadata)
774+ self.assertThat(
775+ metadata,
776+ MatchesStructure(
777+ format=Equals(BinaryPackageFormat.CONDA_V2),
778+ name=Equals("conda-v2-arch"),
779+ version=Equals("0.1"),
780+ summary=Equals("Example summary"),
781+ description=Equals("Example description"),
782+ architecturespecific=Is(True),
783+ homepage=Equals("http://example.com/"),
784+ user_defined_fields=Equals([("subdir", "linux-64")]),
785+ ))
786
787 def test_run_indep(self):
788 archive = self.factory.makeArchive(
789@@ -384,6 +428,21 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
790 with dbuser(job.config.dbuser):
791 JobRunner([job]).runAll()
792
793+ self.assertThat(archive.getPublishedSources(), MatchesSetwise(
794+ MatchesStructure(
795+ sourcepackagename=MatchesStructure.byEquality(
796+ name=build.git_repository.target.name),
797+ sourcepackagerelease=MatchesStructure(
798+ ci_build=Equals(build),
799+ sourcepackagename=MatchesStructure.byEquality(
800+ name=build.git_repository.target.name),
801+ version=Equals("0.0.1"),
802+ format=Equals(SourcePackageType.CI_BUILD),
803+ architecturehintlist=Equals(""),
804+ creator=Equals(build.git_repository.owner),
805+ files=Equals([])),
806+ format=Equals(SourcePackageType.CI_BUILD),
807+ distroseries=Equals(distroseries))))
808 self.assertThat(archive.getAllPublishedBinaries(), MatchesSetwise(*(
809 MatchesStructure(
810 binarypackagename=MatchesStructure.byEquality(
811@@ -418,13 +477,17 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
812 archive.distribution, distro_arch_series=dases[0])
813 report = build.getOrCreateRevisionStatusReport("build:0")
814 report.setLog(b"log data")
815- path = "wheel-arch/dist/wheel_arch-0.0.1-cp310-cp310-linux_x86_64.whl"
816- with open(datadir(path), mode="rb") as f:
817- report.attach(name=os.path.basename(path), data=f.read())
818- artifact = IStore(RevisionStatusArtifact).find(
819+ sdist_path = "wheel-arch/dist/wheel-arch-0.0.1.tar.gz"
820+ wheel_path = (
821+ "wheel-arch/dist/wheel_arch-0.0.1-cp310-cp310-linux_x86_64.whl")
822+ with open(datadir(sdist_path), mode="rb") as f:
823+ report.attach(name=os.path.basename(sdist_path), data=f.read())
824+ with open(datadir(wheel_path), mode="rb") as f:
825+ report.attach(name=os.path.basename(wheel_path), data=f.read())
826+ artifacts = IStore(RevisionStatusArtifact).find(
827 RevisionStatusArtifact,
828 report=report,
829- artifact_type=RevisionStatusArtifactType.BINARY).one()
830+ artifact_type=RevisionStatusArtifactType.BINARY).order_by("id")
831 job = CIBuildUploadJob.create(
832 build, build.git_repository.owner, archive, distroseries,
833 PackagePublishingPocket.RELEASE, target_channel="edge")
834@@ -433,6 +496,24 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
835 with dbuser(job.config.dbuser):
836 JobRunner([job]).runAll()
837
838+ self.assertThat(archive.getPublishedSources(), MatchesSetwise(
839+ MatchesStructure(
840+ sourcepackagename=MatchesStructure.byEquality(
841+ name=build.git_repository.target.name),
842+ sourcepackagerelease=MatchesStructure(
843+ ci_build=Equals(build),
844+ sourcepackagename=MatchesStructure.byEquality(
845+ name=build.git_repository.target.name),
846+ version=Equals("0.0.1"),
847+ format=Equals(SourcePackageType.CI_BUILD),
848+ architecturehintlist=Equals(""),
849+ creator=Equals(build.git_repository.owner),
850+ files=MatchesSetwise(
851+ MatchesStructure.byEquality(
852+ libraryfile=artifacts[0].library_file,
853+ filetype=SourcePackageFileType.SDIST))),
854+ format=Equals(SourcePackageType.CI_BUILD),
855+ distroseries=Equals(distroseries))))
856 self.assertThat(archive.getAllPublishedBinaries(), MatchesSetwise(
857 MatchesStructure(
858 binarypackagename=MatchesStructure.byEquality(
859@@ -449,7 +530,7 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
860 homepage=Equals("http://example.com/"),
861 files=MatchesSetwise(
862 MatchesStructure.byEquality(
863- libraryfile=artifact.library_file,
864+ libraryfile=artifacts[1].library_file,
865 filetype=BinaryPackageFileType.WHL))),
866 binarypackageformat=Equals(BinaryPackageFormat.WHL),
867 distroarchseries=Equals(dases[0]))))
868@@ -481,6 +562,21 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
869 with dbuser(job.config.dbuser):
870 JobRunner([job]).runAll()
871
872+ self.assertThat(archive.getPublishedSources(), MatchesSetwise(
873+ MatchesStructure(
874+ sourcepackagename=MatchesStructure.byEquality(
875+ name=build.git_repository.target.name),
876+ sourcepackagerelease=MatchesStructure(
877+ ci_build=Equals(build),
878+ sourcepackagename=MatchesStructure.byEquality(
879+ name=build.git_repository.target.name),
880+ version=Equals("0.1"),
881+ format=Equals(SourcePackageType.CI_BUILD),
882+ architecturehintlist=Equals(""),
883+ creator=Equals(build.git_repository.owner),
884+ files=Equals([])),
885+ format=Equals(SourcePackageType.CI_BUILD),
886+ distroseries=Equals(distroseries))))
887 self.assertThat(archive.getAllPublishedBinaries(), MatchesSetwise(
888 MatchesStructure(
889 binarypackagename=MatchesStructure.byEquality(
890@@ -529,6 +625,21 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
891 with dbuser(job.config.dbuser):
892 JobRunner([job]).runAll()
893
894+ self.assertThat(archive.getPublishedSources(), MatchesSetwise(
895+ MatchesStructure(
896+ sourcepackagename=MatchesStructure.byEquality(
897+ name=build.git_repository.target.name),
898+ sourcepackagerelease=MatchesStructure(
899+ ci_build=Equals(build),
900+ sourcepackagename=MatchesStructure.byEquality(
901+ name=build.git_repository.target.name),
902+ version=Equals("0.1"),
903+ format=Equals(SourcePackageType.CI_BUILD),
904+ architecturehintlist=Equals(""),
905+ creator=Equals(build.git_repository.owner),
906+ files=Equals([])),
907+ format=Equals(SourcePackageType.CI_BUILD),
908+ distroseries=Equals(distroseries))))
909 self.assertThat(archive.getAllPublishedBinaries(), MatchesSetwise(
910 MatchesStructure(
911 binarypackagename=MatchesStructure.byEquality(
912@@ -550,7 +661,7 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
913 binarypackageformat=Equals(BinaryPackageFormat.CONDA_V2),
914 distroarchseries=Equals(dases[0]))))
915
916- def test_existing_release(self):
917+ def test_existing_source_and_binary_releases(self):
918 # A `CIBuildUploadJob` can be run even if the build in question was
919 # already uploaded somewhere, and in that case may add publications
920 # in other locations for the same package.
921@@ -561,13 +672,16 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
922 das = self.factory.makeDistroArchSeries(distroseries=distroseries)
923 build = self.makeCIBuild(archive.distribution, distro_arch_series=das)
924 report = build.getOrCreateRevisionStatusReport("build:0")
925- path = "wheel-indep/dist/wheel_indep-0.0.1-py3-none-any.whl"
926- with open(datadir(path), mode="rb") as f:
927- report.attach(name=os.path.basename(path), data=f.read())
928- artifact = IStore(RevisionStatusArtifact).find(
929+ sdist_path = "wheel-indep/dist/wheel-indep-0.0.1.tar.gz"
930+ with open(datadir(sdist_path), mode="rb") as f:
931+ report.attach(name=os.path.basename(sdist_path), data=f.read())
932+ wheel_path = "wheel-indep/dist/wheel_indep-0.0.1-py3-none-any.whl"
933+ with open(datadir(wheel_path), mode="rb") as f:
934+ report.attach(name=os.path.basename(wheel_path), data=f.read())
935+ artifacts = IStore(RevisionStatusArtifact).find(
936 RevisionStatusArtifact,
937 report=report,
938- artifact_type=RevisionStatusArtifactType.BINARY).one()
939+ artifact_type=RevisionStatusArtifactType.BINARY).order_by("id")
940 job = CIBuildUploadJob.create(
941 build, build.git_repository.owner, archive, distroseries,
942 PackagePublishingPocket.RELEASE, target_channel="edge")
943@@ -582,12 +696,31 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
944 with dbuser(job.config.dbuser):
945 JobRunner([job]).runAll()
946
947+ spphs = archive.getPublishedSources()
948+ # The source publications are for the same source package release,
949+ # which has a single file attached to it.
950+ self.assertEqual(1, len({spph.sourcepackagename for spph in spphs}))
951+ self.assertEqual(1, len({spph.sourcepackagerelease for spph in spphs}))
952+ self.assertThat(spphs, MatchesSetwise(*(
953+ MatchesStructure(
954+ sourcepackagename=MatchesStructure.byEquality(
955+ name=build.git_repository.target.name),
956+ sourcepackagerelease=MatchesStructure(
957+ ci_build=Equals(build),
958+ files=MatchesSetwise(
959+ MatchesStructure.byEquality(
960+ libraryfile=artifacts[0].library_file,
961+ filetype=SourcePackageFileType.SDIST))),
962+ format=Equals(SourcePackageType.CI_BUILD),
963+ distroseries=Equals(distroseries),
964+ channel=Equals(channel))
965+ for channel in ("edge", "0.0.1/edge"))))
966 bpphs = archive.getAllPublishedBinaries()
967- # The publications are for the same binary package release, which
968- # has a single file attached to it.
969+ # The binary publications are for the same binary package release,
970+ # which has a single file attached to it.
971 self.assertEqual(1, len({bpph.binarypackagename for bpph in bpphs}))
972 self.assertEqual(1, len({bpph.binarypackagerelease for bpph in bpphs}))
973- self.assertThat(archive.getAllPublishedBinaries(), MatchesSetwise(*(
974+ self.assertThat(bpphs, MatchesSetwise(*(
975 MatchesStructure(
976 binarypackagename=MatchesStructure.byEquality(
977 name="wheel-indep"),
978@@ -595,7 +728,79 @@ class TestCIBuildUploadJob(TestCaseWithFactory):
979 ci_build=Equals(build),
980 files=MatchesSetwise(
981 MatchesStructure.byEquality(
982- libraryfile=artifact.library_file,
983+ libraryfile=artifacts[1].library_file,
984+ filetype=BinaryPackageFileType.WHL))),
985+ binarypackageformat=Equals(BinaryPackageFormat.WHL),
986+ distroarchseries=Equals(das),
987+ channel=Equals(channel))
988+ for channel in ("edge", "0.0.1/edge"))))
989+
990+ def test_existing_binary_release_no_existing_source_release(self):
991+ # A `CIBuildUploadJob` can be run even if the build in question was
992+ # already uploaded somewhere, and in that case may add publications
993+ # in other locations for the same package. This works even if there
994+ # was no existing source package release (because
995+ # `CIBuildUploadJob`s didn't always create one).
996+ archive = self.factory.makeArchive(
997+ repository_format=ArchiveRepositoryFormat.PYTHON)
998+ distroseries = self.factory.makeDistroSeries(
999+ distribution=archive.distribution)
1000+ das = self.factory.makeDistroArchSeries(distroseries=distroseries)
1001+ build = self.makeCIBuild(archive.distribution, distro_arch_series=das)
1002+ report = build.getOrCreateRevisionStatusReport("build:0")
1003+ sdist_path = "wheel-indep/dist/wheel-indep-0.0.1.tar.gz"
1004+ with open(datadir(sdist_path), mode="rb") as f:
1005+ report.attach(name=os.path.basename(sdist_path), data=f.read())
1006+ wheel_path = "wheel-indep/dist/wheel_indep-0.0.1-py3-none-any.whl"
1007+ with open(datadir(wheel_path), mode="rb") as f:
1008+ report.attach(name=os.path.basename(wheel_path), data=f.read())
1009+ artifacts = IStore(RevisionStatusArtifact).find(
1010+ RevisionStatusArtifact,
1011+ report=report,
1012+ artifact_type=RevisionStatusArtifactType.BINARY).order_by("id")
1013+ job = CIBuildUploadJob.create(
1014+ build, build.git_repository.owner, archive, distroseries,
1015+ PackagePublishingPocket.RELEASE, target_channel="edge")
1016+ transaction.commit()
1017+ with MockPatchObject(CIBuildUploadJob, "_uploadSources"):
1018+ with dbuser(job.config.dbuser):
1019+ JobRunner([job]).runAll()
1020+ job = CIBuildUploadJob.create(
1021+ build, build.git_repository.owner, archive, distroseries,
1022+ PackagePublishingPocket.RELEASE, target_channel="0.0.1/edge")
1023+ transaction.commit()
1024+
1025+ with dbuser(job.config.dbuser):
1026+ JobRunner([job]).runAll()
1027+
1028+ # There is a source publication for a new source package release.
1029+ self.assertThat(archive.getPublishedSources(), MatchesSetwise(
1030+ MatchesStructure(
1031+ sourcepackagename=MatchesStructure.byEquality(
1032+ name=build.git_repository.target.name),
1033+ sourcepackagerelease=MatchesStructure(
1034+ ci_build=Equals(build),
1035+ files=MatchesSetwise(
1036+ MatchesStructure.byEquality(
1037+ libraryfile=artifacts[0].library_file,
1038+ filetype=SourcePackageFileType.SDIST))),
1039+ format=Equals(SourcePackageType.CI_BUILD),
1040+ distroseries=Equals(distroseries),
1041+ channel=Equals("0.0.1/edge"))))
1042+ bpphs = archive.getAllPublishedBinaries()
1043+ # The binary publications are for the same binary package release,
1044+ # which has a single file attached to it.
1045+ self.assertEqual(1, len({bpph.binarypackagename for bpph in bpphs}))
1046+ self.assertEqual(1, len({bpph.binarypackagerelease for bpph in bpphs}))
1047+ self.assertThat(bpphs, MatchesSetwise(*(
1048+ MatchesStructure(
1049+ binarypackagename=MatchesStructure.byEquality(
1050+ name="wheel-indep"),
1051+ binarypackagerelease=MatchesStructure(
1052+ ci_build=Equals(build),
1053+ files=MatchesSetwise(
1054+ MatchesStructure.byEquality(
1055+ libraryfile=artifacts[1].library_file,
1056 filetype=BinaryPackageFileType.WHL))),
1057 binarypackageformat=Equals(BinaryPackageFormat.WHL),
1058 distroarchseries=Equals(das),

Subscribers

People subscribed via source and target branches

to status/vote changes: