Merge ~cjwatson/launchpad:ci-build-upload-make-spr into launchpad:master
- Git
- lp:~cjwatson/launchpad
- ci-build-upload-make-spr
- Merge into master
Status: | Merged |
---|---|
Approved by: | Colin Watson |
Approved revision: | 25428c91bf957de899995044925e87d9a654c618 |
Merge reported by: | Otto Co-Pilot |
Merged at revision: | not available |
Proposed branch: | ~cjwatson/launchpad:ci-build-upload-make-spr |
Merge into: | launchpad:master |
Prerequisite: | ~cjwatson/launchpad:refactor-ci-build-upload-job-scan-file |
Diff against target: |
1058 lines (+522/-135) 9 files modified
lib/lp/_schema_circular_imports.py (+1/-0) lib/lp/archivepublisher/tests/test_artifactory.py (+3/-3) lib/lp/code/interfaces/cibuild.py (+12/-0) lib/lp/code/model/cibuild.py (+30/-0) lib/lp/code/model/tests/test_cibuild.py (+22/-0) lib/lp/registry/interfaces/sourcepackage.py (+3/-3) lib/lp/soyuz/interfaces/sourcepackagerelease.py (+12/-3) lib/lp/soyuz/model/archivejob.py (+154/-46) lib/lp/soyuz/tests/test_archivejob.py (+285/-80) |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Andrey Fedoseev (community) | Approve | ||
Review via email: mp+426502@code.launchpad.net |
Commit message
Create a source publication when uploading a CI build
Description of the change
This has some advantages and disadvantages, but on the whole I think the advantages outweigh the disadvantages.
The downsides are:
* if the CI build produces nothing that looks like source code at all, then we end up with a slightly odd "source package" with no files attached to it;
* if builds do produce something that looks like source code (e.g. a Python sdist), then they're on the honour system to either produce it from only one architecture or to produce something equivalent from all architectures.
On the other hand:
* Launchpad's data model for packages makes the assumption in many places that binary packages have an associated source package, and my attempts to make it tolerate isolated binary packages have only been marginally successful;
* treating source-like artifacts (Python sdists, zip files of Go source code, etc.) as source packages rather than binary packages makes more sense to humans;
* source packages provide a useful grouping for collections of binary packages from builds of the same version;
* creating source publications means that the PPA web UI works for PPAs populated from CI builds with only a few small adjustments (not in this branch);
* creating source publications means that it should be straightforward to get the existing package copying mechanism to work.
I've slightly cheekily renamed `SourcePackageT
As a concrete example of a source package file type and in order to allow more useful testing, I've added support for uploading Python sdists from CI builds as part of this branch.
Colin Watson (cjwatson) wrote : | # |
I find namedtuples awkwardly inflexible in general and usually regret using them; in this case, they aren't really suitable since they don't support optional arguments on construction. I get the general idea, though, so I looked for alternatives. A dataclass would be ideal, but those don't exist until Python 3.7. Type annotations on class attributes don't work in Python 3.6, not even the old comment style unless the attribute has an initializer.
However, `attrs` (https:/
I've reworked this branch using that. Would you mind having another look?
Jürgen Gmach (jugmac00) wrote (last edit ): | # |
I am with Andrey in that we should not use a dictionary.
While attrs is a great library, I am -0.5 on introducing it.
Introducing a new dependency
- which it is only used for two classes
- only for a short interim period
- with a new "syntax"
- which is even dated, ie it is not reflected in the main documentation of the package - you need to dive into its history: https:/
I am strongly in favor of using plain classes - with that usual boilerplate - and convert them to dataclasses once we are on a suitable Python version.
If you go forward using it, please leave a comment to remove it asap.
Andrey Fedoseev (andrey-fedoseev) wrote : | # |
I'm not in favor of using `attrs` either.
I suggested to use named tuples because they can be automatically converted to 3.6 syntax with `pyupgrade` (https:/
Colin Watson (cjwatson) wrote : | # |
I really don't think namedtuples are suitable here because of the lack of support for optional items in their constructors.
For plain classes, how do you propose spelling them? This doesn't work for obvious reasons:
class SourceArtifactM
format # type: SourcePackageFi
name # type: str
version # type: str
And if I use something like `format = None` there, then they all have to be `Optional[...]` when they really shouldn't be optional.
Andrey Fedoseev (andrey-fedoseev) wrote : | # |
> I really don't think namedtuples are suitable here because of the lack of support for optional items in their constructors.
You're right, the optional values are available only since 3.7
> For plain classes, how do you propose spelling them?
I think you could use the constructor for that:
class SourceArtifactM
def __init__(self, format: SourcePackageFi
Colin Watson (cjwatson) wrote : | # |
OK, I've converted this to a plain class with `__init__` boilerplate as you suggest. How's this?
Andrey Fedoseev (andrey-fedoseev) wrote : | # |
> OK, I've converted this to a plain class with `__init__` boilerplate as you suggest. How's this?
This looks great
PyCharm reports two issues:
1. The signature of `CIBuildUploadJ
2. `scanned` argument in `_uploadSources` should probably be a `Sequence` rather than an `Iterable` because you're doing `scanned[0]` there
Colin Watson (cjwatson) wrote : | # |
The `create` method thing is at least intentional for now, if possibly not ideal (the whole derived jobs business is a bit unnecessarily complicated, I think).
I've fixed the `scanned` argument type.
Preview Diff
1 | diff --git a/lib/lp/_schema_circular_imports.py b/lib/lp/_schema_circular_imports.py |
2 | index e0bd33a..677163b 100644 |
3 | --- a/lib/lp/_schema_circular_imports.py |
4 | +++ b/lib/lp/_schema_circular_imports.py |
5 | @@ -335,6 +335,7 @@ patch_reference_property(IPackageUpload, 'copy_source_archive', IArchive) |
6 | patch_reference_property( |
7 | ISourcePackageRelease, 'source_package_recipe_build', |
8 | ISourcePackageRecipeBuild) |
9 | +patch_reference_property(ISourcePackageRelease, 'ci_build', ICIBuild) |
10 | |
11 | # IIndexedMessage |
12 | patch_reference_property(IIndexedMessage, 'inside', IBugTask) |
13 | diff --git a/lib/lp/archivepublisher/tests/test_artifactory.py b/lib/lp/archivepublisher/tests/test_artifactory.py |
14 | index c9feb58..18af683 100644 |
15 | --- a/lib/lp/archivepublisher/tests/test_artifactory.py |
16 | +++ b/lib/lp/archivepublisher/tests/test_artifactory.py |
17 | @@ -576,7 +576,7 @@ class TestArtifactoryPoolFromLibrarian(TestCaseWithFactory): |
18 | sourcepackagename="foo", |
19 | version="1.0", |
20 | channel="edge", |
21 | - format=SourcePackageType.SDIST, |
22 | + format=SourcePackageType.CI_BUILD, |
23 | ) |
24 | spr = spph.sourcepackagerelease |
25 | sprf = self.factory.makeSourcePackageReleaseFile( |
26 | @@ -635,7 +635,7 @@ class TestArtifactoryPoolFromLibrarian(TestCaseWithFactory): |
27 | archive=pool.archive, |
28 | sourcepackagename="foo", |
29 | version="1.0", |
30 | - format=SourcePackageType.SDIST, |
31 | + format=SourcePackageType.CI_BUILD, |
32 | ) |
33 | bpph = self.factory.makeBinaryPackagePublishingHistory( |
34 | archive=pool.archive, |
35 | @@ -940,7 +940,7 @@ class TestArtifactoryPoolFromLibrarian(TestCaseWithFactory): |
36 | archive=pool.archive, |
37 | sourcepackagename="foo", |
38 | version="1.0", |
39 | - format=SourcePackageType.SDIST, |
40 | + format=SourcePackageType.CI_BUILD, |
41 | ) |
42 | bpph = self.factory.makeBinaryPackagePublishingHistory( |
43 | archive=pool.archive, |
44 | diff --git a/lib/lp/code/interfaces/cibuild.py b/lib/lp/code/interfaces/cibuild.py |
45 | index 69c765b..d929283 100644 |
46 | --- a/lib/lp/code/interfaces/cibuild.py |
47 | +++ b/lib/lp/code/interfaces/cibuild.py |
48 | @@ -133,6 +133,10 @@ class ICIBuildView(IPackageBuildView, IPrivacy): |
49 | "A mapping from job IDs to result tokens, retrieved from the " |
50 | "builder."))) |
51 | |
52 | + sourcepackages = Attribute( |
53 | + "A list of source packages that resulted from this build, ordered by " |
54 | + "name.") |
55 | + |
56 | binarypackages = Attribute( |
57 | "A list of binary packages that resulted from this build, ordered by " |
58 | "name.") |
59 | @@ -182,6 +186,14 @@ class ICIBuildView(IPackageBuildView, IPrivacy): |
60 | :return: A collection of URLs for this build. |
61 | """ |
62 | |
63 | + def createSourcePackageRelease( |
64 | + distroseries, sourcepackagename, version, creator=None, |
65 | + archive=None): |
66 | + """Create and return a `SourcePackageRelease` for this CI build. |
67 | + |
68 | + The new source package release will be linked to this build. |
69 | + """ |
70 | + |
71 | def createBinaryPackageRelease( |
72 | binarypackagename, version, summary, description, binpackageformat, |
73 | architecturespecific, installedsize=None, homepage=None, |
74 | diff --git a/lib/lp/code/model/cibuild.py b/lib/lp/code/model/cibuild.py |
75 | index 76ccfc3..281c3a1 100644 |
76 | --- a/lib/lp/code/model/cibuild.py |
77 | +++ b/lib/lp/code/model/cibuild.py |
78 | @@ -65,8 +65,10 @@ from lp.code.model.gitref import GitRef |
79 | from lp.code.model.lpcraft import load_configuration |
80 | from lp.registry.interfaces.pocket import PackagePublishingPocket |
81 | from lp.registry.interfaces.series import SeriesStatus |
82 | +from lp.registry.interfaces.sourcepackage import SourcePackageType |
83 | from lp.registry.model.distribution import Distribution |
84 | from lp.registry.model.distroseries import DistroSeries |
85 | +from lp.registry.model.sourcepackagename import SourcePackageName |
86 | from lp.services.database.bulk import load_related |
87 | from lp.services.database.constants import DEFAULT |
88 | from lp.services.database.decoratedresultset import DecoratedResultSet |
89 | @@ -91,6 +93,7 @@ from lp.services.propertycache import cachedproperty |
90 | from lp.soyuz.model.binarypackagename import BinaryPackageName |
91 | from lp.soyuz.model.binarypackagerelease import BinaryPackageRelease |
92 | from lp.soyuz.model.distroarchseries import DistroArchSeries |
93 | +from lp.soyuz.model.sourcepackagerelease import SourcePackageRelease |
94 | |
95 | |
96 | def get_stages(configuration): |
97 | @@ -471,6 +474,17 @@ class CIBuild(PackageBuildMixin, StormBase): |
98 | # We don't currently send any notifications. |
99 | |
100 | @property |
101 | + def sourcepackages(self): |
102 | + """See `ICIBuild`.""" |
103 | + releases = IStore(SourcePackageRelease).find( |
104 | + (SourcePackageRelease, SourcePackageName), |
105 | + SourcePackageRelease.ci_build == self, |
106 | + SourcePackageRelease.sourcepackagename == SourcePackageName.id) |
107 | + releases = releases.order_by( |
108 | + SourcePackageName.name, SourcePackageRelease.id) |
109 | + return DecoratedResultSet(releases, result_decorator=itemgetter(0)) |
110 | + |
111 | + @property |
112 | def binarypackages(self): |
113 | """See `ICIBuild`.""" |
114 | releases = IStore(BinaryPackageRelease).find( |
115 | @@ -481,6 +495,22 @@ class CIBuild(PackageBuildMixin, StormBase): |
116 | BinaryPackageName.name, BinaryPackageRelease.id) |
117 | return DecoratedResultSet(releases, result_decorator=itemgetter(0)) |
118 | |
119 | + def createSourcePackageRelease( |
120 | + self, distroseries, sourcepackagename, version, creator=None, |
121 | + archive=None): |
122 | + """See `ICIBuild`.""" |
123 | + return distroseries.createUploadedSourcePackageRelease( |
124 | + sourcepackagename=sourcepackagename, |
125 | + version=version, |
126 | + format=SourcePackageType.CI_BUILD, |
127 | + # This doesn't really make sense for SPRs created for CI builds, |
128 | + # but the column is NOT NULL. The empty string will do though, |
129 | + # since nothing will use this. |
130 | + architecturehintlist="", |
131 | + creator=creator, |
132 | + archive=archive, |
133 | + ci_build=self) |
134 | + |
135 | def createBinaryPackageRelease( |
136 | self, binarypackagename, version, summary, description, |
137 | binpackageformat, architecturespecific, installedsize=None, |
138 | diff --git a/lib/lp/code/model/tests/test_cibuild.py b/lib/lp/code/model/tests/test_cibuild.py |
139 | index 89a432c..ca0192b 100644 |
140 | --- a/lib/lp/code/model/tests/test_cibuild.py |
141 | +++ b/lib/lp/code/model/tests/test_cibuild.py |
142 | @@ -62,6 +62,7 @@ from lp.code.model.cibuild import ( |
143 | from lp.code.model.lpcraft import load_configuration |
144 | from lp.code.tests.helpers import GitHostingFixture |
145 | from lp.registry.interfaces.series import SeriesStatus |
146 | +from lp.registry.interfaces.sourcepackage import SourcePackageType |
147 | from lp.services.authserver.xmlrpc import AuthServerAPIView |
148 | from lp.services.config import config |
149 | from lp.services.librarian.browser import ProxiedLibraryFileAlias |
150 | @@ -400,6 +401,27 @@ class TestCIBuild(TestCaseWithFactory): |
151 | commit_sha1=build.commit_sha1, |
152 | ci_build=build)) |
153 | |
154 | + def test_createSourcePackageRelease(self): |
155 | + distroseries = self.factory.makeDistroSeries() |
156 | + archive = self.factory.makeArchive( |
157 | + distribution=distroseries.distribution) |
158 | + build = self.factory.makeCIBuild() |
159 | + spn = self.factory.makeSourcePackageName() |
160 | + spr = build.createSourcePackageRelease( |
161 | + distroseries, spn, "1.0", creator=build.git_repository.owner, |
162 | + archive=archive) |
163 | + self.assertThat(spr, MatchesStructure( |
164 | + upload_distroseries=Equals(distroseries), |
165 | + sourcepackagename=Equals(spn), |
166 | + version=Equals("1.0"), |
167 | + format=Equals(SourcePackageType.CI_BUILD), |
168 | + architecturehintlist=Equals(""), |
169 | + creator=Equals(build.git_repository.owner), |
170 | + upload_archive=Equals(archive), |
171 | + ci_build=Equals(build), |
172 | + )) |
173 | + self.assertContentEqual([spr], build.sourcepackages) |
174 | + |
175 | def test_createBinaryPackageRelease(self): |
176 | build = self.factory.makeCIBuild() |
177 | bpn = self.factory.makeBinaryPackageName() |
178 | diff --git a/lib/lp/registry/interfaces/sourcepackage.py b/lib/lp/registry/interfaces/sourcepackage.py |
179 | index fcde488..6a7e707 100644 |
180 | --- a/lib/lp/registry/interfaces/sourcepackage.py |
181 | +++ b/lib/lp/registry/interfaces/sourcepackage.py |
182 | @@ -456,10 +456,10 @@ class SourcePackageType(DBEnumeratedType): |
183 | This is the source package format used by Gentoo. |
184 | """) |
185 | |
186 | - SDIST = DBItem(4, """ |
187 | - The Python Format |
188 | + CI_BUILD = DBItem(4, """ |
189 | + CI Build |
190 | |
191 | - This is the source package format used by Python packages. |
192 | + An ad-hoc source package generated by a CI build. |
193 | """) |
194 | |
195 | |
196 | diff --git a/lib/lp/soyuz/interfaces/sourcepackagerelease.py b/lib/lp/soyuz/interfaces/sourcepackagerelease.py |
197 | index 8e6cee9..aac4f39 100644 |
198 | --- a/lib/lp/soyuz/interfaces/sourcepackagerelease.py |
199 | +++ b/lib/lp/soyuz/interfaces/sourcepackagerelease.py |
200 | @@ -145,11 +145,20 @@ class ISourcePackageRelease(Interface): |
201 | # Really ISourcePackageRecipeBuild -- see _schema_circular_imports. |
202 | source_package_recipe_build = Reference( |
203 | schema=Interface, |
204 | - description=_("The `SourcePackageRecipeBuild` which produced this " |
205 | - "source package release, or None if it was created from a " |
206 | - "traditional upload."), |
207 | + description=_( |
208 | + "The `SourcePackageRecipeBuild` which produced this source " |
209 | + "package release, or None if it was not created from a source " |
210 | + "package recipe."), |
211 | title=_("Source package recipe build"), |
212 | required=False, readonly=True) |
213 | + # Really ICIBuild, patched in _schema_circular_imports. |
214 | + ci_build = Reference( |
215 | + schema=Interface, |
216 | + description=_( |
217 | + "The `CIBuild` which produced this source package release, or " |
218 | + "None if it was not created from a CI build."), |
219 | + title=_("CI build"), |
220 | + required=False, readonly=True) |
221 | |
222 | def getUserDefinedField(name): |
223 | """Case-insensitively get a user-defined field.""" |
224 | diff --git a/lib/lp/soyuz/model/archivejob.py b/lib/lp/soyuz/model/archivejob.py |
225 | index d62f199..ae038a3 100644 |
226 | --- a/lib/lp/soyuz/model/archivejob.py |
227 | +++ b/lib/lp/soyuz/model/archivejob.py |
228 | @@ -13,11 +13,18 @@ from typing import ( |
229 | Dict, |
230 | Iterable, |
231 | List, |
232 | + Optional, |
233 | + Sequence, |
234 | + Tuple, |
235 | + Union, |
236 | ) |
237 | import zipfile |
238 | |
239 | from lazr.delegates import delegate_to |
240 | -from pkginfo import Wheel |
241 | +from pkginfo import ( |
242 | + SDist, |
243 | + Wheel, |
244 | + ) |
245 | from storm.expr import And |
246 | from storm.locals import ( |
247 | Int, |
248 | @@ -43,6 +50,7 @@ from lp.registry.interfaces.distributionsourcepackage import ( |
249 | ) |
250 | from lp.registry.interfaces.distroseries import IDistroSeriesSet |
251 | from lp.registry.interfaces.pocket import PackagePublishingPocket |
252 | +from lp.registry.interfaces.sourcepackage import SourcePackageFileType |
253 | from lp.services.config import config |
254 | from lp.services.database.enumcol import DBEnum |
255 | from lp.services.database.interfaces import IMasterStore |
256 | @@ -212,10 +220,39 @@ class ScanException(Exception): |
257 | """A CI build upload job failed to scan a file.""" |
258 | |
259 | |
260 | +class SourceArtifactMetadata: |
261 | + """Metadata extracted from a source package.""" |
262 | + |
263 | + def __init__(self, format: SourcePackageFileType, name: str, version: str): |
264 | + self.format = format |
265 | + self.name = name |
266 | + self.version = version |
267 | + |
268 | + |
269 | +class BinaryArtifactMetadata: |
270 | + """Metadata extracted from a binary package.""" |
271 | + |
272 | + def __init__(self, format: BinaryPackageFormat, name: str, version: str, |
273 | + summary: str, description: str, architecturespecific: bool, |
274 | + homepage: str, |
275 | + user_defined_fields: Optional[List[Tuple[str, str]]] = None): |
276 | + self.format = format |
277 | + self.name = name |
278 | + self.version = version |
279 | + self.summary = summary |
280 | + self.description = description |
281 | + self.architecturespecific = architecturespecific |
282 | + self.homepage = homepage |
283 | + self.user_defined_fields = user_defined_fields |
284 | + |
285 | + |
286 | +ArtifactMetadata = Union[SourceArtifactMetadata, BinaryArtifactMetadata] |
287 | + |
288 | + |
289 | class ScannedArtifact: |
290 | |
291 | def __init__( |
292 | - self, *, artifact: IRevisionStatusArtifact, metadata: Dict[str, Any] |
293 | + self, *, artifact: IRevisionStatusArtifact, metadata: ArtifactMetadata |
294 | ): |
295 | self.artifact = artifact |
296 | self.metadata = metadata |
297 | @@ -247,13 +284,16 @@ class CIBuildUploadJob(ArchiveJobDerived): |
298 | |
299 | # We're only interested in uploading certain kinds of packages to |
300 | # certain kinds of archives. |
301 | - binary_format_by_repository_format = { |
302 | + format_by_repository_format = { |
303 | ArchiveRepositoryFormat.DEBIAN: { |
304 | BinaryPackageFormat.DEB, |
305 | BinaryPackageFormat.UDEB, |
306 | BinaryPackageFormat.DDEB, |
307 | }, |
308 | - ArchiveRepositoryFormat.PYTHON: {BinaryPackageFormat.WHL}, |
309 | + ArchiveRepositoryFormat.PYTHON: { |
310 | + SourcePackageFileType.SDIST, |
311 | + BinaryPackageFormat.WHL, |
312 | + }, |
313 | ArchiveRepositoryFormat.CONDA: { |
314 | BinaryPackageFormat.CONDA_V1, |
315 | BinaryPackageFormat.CONDA_V2, |
316 | @@ -326,7 +366,7 @@ class CIBuildUploadJob(ArchiveJobDerived): |
317 | def target_channel(self): |
318 | return self.metadata["target_channel"] |
319 | |
320 | - def _scanWheel(self, path: str) -> Dict[str, Any]: |
321 | + def _scanWheel(self, path: str) -> Optional[BinaryArtifactMetadata]: |
322 | try: |
323 | parsed_path = parse_wheel_filename(path) |
324 | wheel = Wheel(path) |
325 | @@ -335,34 +375,50 @@ class CIBuildUploadJob(ArchiveJobDerived): |
326 | "Failed to scan %s as a Python wheel: %s", |
327 | os.path.basename(path), e) |
328 | return None |
329 | - return { |
330 | - "name": wheel.name, |
331 | - "version": wheel.version, |
332 | - "summary": wheel.summary or "", |
333 | - "description": wheel.description, |
334 | - "binpackageformat": BinaryPackageFormat.WHL, |
335 | - "architecturespecific": "any" not in parsed_path.platform_tags, |
336 | - "homepage": wheel.home_page or "", |
337 | - } |
338 | + return BinaryArtifactMetadata( |
339 | + format=BinaryPackageFormat.WHL, |
340 | + name=wheel.name, |
341 | + version=wheel.version, |
342 | + summary=wheel.summary or "", |
343 | + description=wheel.description, |
344 | + architecturespecific="any" not in parsed_path.platform_tags, |
345 | + homepage=wheel.home_page or "", |
346 | + ) |
347 | + |
348 | + def _scanSDist(self, path: str) -> Optional[SourceArtifactMetadata]: |
349 | + try: |
350 | + sdist = SDist(path) |
351 | + except Exception as e: |
352 | + logger.warning( |
353 | + "Failed to scan %s as a Python sdist: %s", |
354 | + os.path.basename(path), e) |
355 | + return None |
356 | + return SourceArtifactMetadata( |
357 | + format=SourcePackageFileType.SDIST, |
358 | + name=sdist.name, |
359 | + version=sdist.version, |
360 | + ) |
361 | |
362 | def _scanCondaMetadata( |
363 | - self, index: Dict[Any, Any], about: Dict[Any, Any] |
364 | - ) -> Dict[str, Any]: |
365 | - return { |
366 | - "name": index["name"], |
367 | - "version": index["version"], |
368 | - "summary": about.get("summary", ""), |
369 | - "description": about.get("description", ""), |
370 | - "architecturespecific": index["platform"] is not None, |
371 | - "homepage": about.get("home", ""), |
372 | + self, format: BinaryPackageFormat, index: Dict[Any, Any], |
373 | + about: Dict[Any, Any] |
374 | + ) -> Optional[BinaryArtifactMetadata]: |
375 | + return BinaryArtifactMetadata( |
376 | + format=format, |
377 | + name=index["name"], |
378 | + version=index["version"], |
379 | + summary=about.get("summary", ""), |
380 | + description=about.get("description", ""), |
381 | + architecturespecific=index["platform"] is not None, |
382 | + homepage=about.get("home", ""), |
383 | # We should perhaps model this explicitly since it's used by the |
384 | # publisher, but this gives us an easy way to pass this through |
385 | # without needing to add a column to a large table that's only |
386 | # relevant to a tiny minority of rows. |
387 | - "user_defined_fields": [("subdir", index["subdir"])], |
388 | - } |
389 | + user_defined_fields=[("subdir", index["subdir"])], |
390 | + ) |
391 | |
392 | - def _scanCondaV1(self, path: str) -> Dict[str, Any]: |
393 | + def _scanCondaV1(self, path: str) -> Optional[BinaryArtifactMetadata]: |
394 | try: |
395 | with tarfile.open(path) as tar: |
396 | index = json.loads( |
397 | @@ -374,11 +430,10 @@ class CIBuildUploadJob(ArchiveJobDerived): |
398 | "Failed to scan %s as a Conda v1 package: %s", |
399 | os.path.basename(path), e) |
400 | return None |
401 | - scanned = {"binpackageformat": BinaryPackageFormat.CONDA_V1} |
402 | - scanned.update(self._scanCondaMetadata(index, about)) |
403 | - return scanned |
404 | + return self._scanCondaMetadata( |
405 | + BinaryPackageFormat.CONDA_V1, index, about) |
406 | |
407 | - def _scanCondaV2(self, path: str) -> Dict[str, Any]: |
408 | + def _scanCondaV2(self, path: str) -> Optional[BinaryArtifactMetadata]: |
409 | try: |
410 | with zipfile.ZipFile(path) as zipf: |
411 | base_name = os.path.basename(path)[:-len(".conda")] |
412 | @@ -396,13 +451,14 @@ class CIBuildUploadJob(ArchiveJobDerived): |
413 | "Failed to scan %s as a Conda v2 package: %s", |
414 | os.path.basename(path), e) |
415 | return None |
416 | - scanned = {"binpackageformat": BinaryPackageFormat.CONDA_V2} |
417 | - scanned.update(self._scanCondaMetadata(index, about)) |
418 | - return scanned |
419 | + return self._scanCondaMetadata( |
420 | + BinaryPackageFormat.CONDA_V2, index, about) |
421 | |
422 | - def _scanFile(self, path: str) -> Dict[str, Any]: |
423 | + def _scanFile(self, path: str) -> Optional[ArtifactMetadata]: |
424 | _scanners = ( |
425 | (".whl", self._scanWheel), |
426 | + (".tar.gz", self._scanSDist), |
427 | + (".zip", self._scanSDist), |
428 | (".tar.bz2", self._scanCondaV1), |
429 | (".conda", self._scanCondaV2), |
430 | ) |
431 | @@ -429,8 +485,8 @@ class CIBuildUploadJob(ArchiveJobDerived): |
432 | Returns a list of `ScannedArtifact`s containing metadata for |
433 | relevant artifacts. |
434 | """ |
435 | - allowed_binary_formats = ( |
436 | - self.binary_format_by_repository_format.get( |
437 | + allowed_formats = ( |
438 | + self.format_by_repository_format.get( |
439 | self.archive.repository_format, set())) |
440 | scanned = [] |
441 | with tempfile.TemporaryDirectory(prefix="ci-build-copy-job") as tmpdir: |
442 | @@ -444,7 +500,7 @@ class CIBuildUploadJob(ArchiveJobDerived): |
443 | metadata = self._scanFile(contents) |
444 | if metadata is None: |
445 | continue |
446 | - if metadata["binpackageformat"] not in allowed_binary_formats: |
447 | + if metadata.format not in allowed_formats: |
448 | logger.info( |
449 | "Skipping %s (not relevant to %s archives)", |
450 | name, self.archive.repository_format) |
451 | @@ -453,28 +509,79 @@ class CIBuildUploadJob(ArchiveJobDerived): |
452 | ScannedArtifact(artifact=artifact, metadata=metadata)) |
453 | return scanned |
454 | |
455 | + def _uploadSources(self, scanned: Sequence[ScannedArtifact]) -> None: |
456 | + """Upload sources from an sequence of `ScannedArtifact`s.""" |
457 | + # Launchpad's data model generally assumes that a single source is |
458 | + # associated with multiple binaries. However, a source package |
459 | + # release can have multiple (or indeed no) files attached to it, so |
460 | + # we make use of that if necessary. |
461 | + releases = { |
462 | + release.sourcepackagename: release |
463 | + for release in self.ci_build.sourcepackages} |
464 | + distroseries = self.ci_build.distro_arch_series.distroseries |
465 | + build_target = self.ci_build.git_repository.target |
466 | + spr = releases.get(build_target.sourcepackagename) |
467 | + if spr is None: |
468 | + spr = self.ci_build.createSourcePackageRelease( |
469 | + distroseries=distroseries, |
470 | + sourcepackagename=build_target.sourcepackagename, |
471 | + # We don't have a good concept of source version here, but |
472 | + # the data model demands one. Arbitrarily pick the version |
473 | + # of the first scanned artifact. |
474 | + version=scanned[0].metadata.version, |
475 | + creator=self.requester, |
476 | + archive=self.archive) |
477 | + for scanned_artifact in scanned: |
478 | + metadata = scanned_artifact.metadata |
479 | + if not isinstance(metadata, SourceArtifactMetadata): |
480 | + continue |
481 | + library_file = scanned_artifact.artifact.library_file |
482 | + logger.info( |
483 | + "Uploading %s to %s %s (%s)", |
484 | + library_file.filename, self.archive.reference, |
485 | + self.target_distroseries.getSuite(self.target_pocket), |
486 | + self.target_channel) |
487 | + for sprf in spr.files: |
488 | + if (sprf.libraryfile == library_file and |
489 | + sprf.filetype == metadata.format): |
490 | + break |
491 | + else: |
492 | + spr.addFile(library_file, filetype=metadata.format) |
493 | + getUtility(IPublishingSet).newSourcePublication( |
494 | + archive=self.archive, sourcepackagerelease=spr, |
495 | + distroseries=self.target_distroseries, pocket=self.target_pocket, |
496 | + creator=self.requester, channel=self.target_channel) |
497 | + |
498 | def _uploadBinaries(self, scanned: Iterable[ScannedArtifact]) -> None: |
499 | - """Upload an iterable of `ScannedArtifact`s to an archive.""" |
500 | + """Upload binaries from an iterable of `ScannedArtifact`s.""" |
501 | releases = { |
502 | (release.binarypackagename, release.binpackageformat): release |
503 | for release in self.ci_build.binarypackages} |
504 | binaries = OrderedDict() |
505 | for scanned_artifact in scanned: |
506 | + metadata = scanned_artifact.metadata |
507 | + if not isinstance(metadata, BinaryArtifactMetadata): |
508 | + continue |
509 | library_file = scanned_artifact.artifact.library_file |
510 | - metadata = dict(scanned_artifact.metadata) |
511 | - binpackageformat = metadata["binpackageformat"] |
512 | logger.info( |
513 | "Uploading %s to %s %s (%s)", |
514 | library_file.filename, self.archive.reference, |
515 | self.target_distroseries.getSuite(self.target_pocket), |
516 | self.target_channel) |
517 | - metadata["binarypackagename"] = bpn = ( |
518 | - getUtility(IBinaryPackageNameSet).ensure(metadata["name"])) |
519 | - del metadata["name"] |
520 | - filetype = self.filetype_by_format[binpackageformat] |
521 | - bpr = releases.get((bpn, binpackageformat)) |
522 | + binarypackagename = getUtility(IBinaryPackageNameSet).ensure( |
523 | + metadata.name) |
524 | + filetype = self.filetype_by_format[metadata.format] |
525 | + bpr = releases.get((binarypackagename, metadata.format)) |
526 | if bpr is None: |
527 | - bpr = self.ci_build.createBinaryPackageRelease(**metadata) |
528 | + bpr = self.ci_build.createBinaryPackageRelease( |
529 | + binarypackagename=binarypackagename, |
530 | + version=metadata.version, |
531 | + summary=metadata.summary, |
532 | + description=metadata.description, |
533 | + binpackageformat=metadata.format, |
534 | + architecturespecific=metadata.architecturespecific, |
535 | + homepage=metadata.homepage, |
536 | + user_defined_fields=metadata.user_defined_fields) |
537 | for bpf in bpr.files: |
538 | if (bpf.libraryfile == library_file and |
539 | bpf.filetype == filetype): |
540 | @@ -503,6 +610,7 @@ class CIBuildUploadJob(ArchiveJobDerived): |
541 | self.ci_build) |
542 | scanned = self._scanArtifacts(artifacts) |
543 | if scanned: |
544 | + self._uploadSources(scanned) |
545 | self._uploadBinaries(scanned) |
546 | else: |
547 | names = [artifact.library_file.filename for artifact in artifacts] |
548 | diff --git a/lib/lp/soyuz/tests/test_archivejob.py b/lib/lp/soyuz/tests/test_archivejob.py |
549 | index b744bc8..813b423 100644 |
550 | --- a/lib/lp/soyuz/tests/test_archivejob.py |
551 | +++ b/lib/lp/soyuz/tests/test_archivejob.py |
552 | @@ -7,6 +7,7 @@ from debian.deb822 import Changes |
553 | from fixtures import ( |
554 | FakeLogger, |
555 | MockPatch, |
556 | + MockPatchObject, |
557 | ) |
558 | from testtools.matchers import ( |
559 | ContainsDict, |
560 | @@ -20,6 +21,10 @@ import transaction |
561 | from lp.code.enums import RevisionStatusArtifactType |
562 | from lp.code.model.revisionstatus import RevisionStatusArtifact |
563 | from lp.registry.interfaces.pocket import PackagePublishingPocket |
564 | +from lp.registry.interfaces.sourcepackage import ( |
565 | + SourcePackageFileType, |
566 | + SourcePackageType, |
567 | + ) |
568 | from lp.services.config import config |
569 | from lp.services.database.interfaces import IStore |
570 | from lp.services.features.testing import FeatureFixture |
571 | @@ -41,8 +46,10 @@ from lp.soyuz.enums import ( |
572 | from lp.soyuz.model.archivejob import ( |
573 | ArchiveJob, |
574 | ArchiveJobDerived, |
575 | + BinaryArtifactMetadata, |
576 | CIBuildUploadJob, |
577 | PackageUploadNotificationJob, |
578 | + SourceArtifactMetadata, |
579 | ) |
580 | from lp.soyuz.tests import datadir |
581 | from lp.testing import ( |
582 | @@ -242,16 +249,19 @@ class TestCIBuildUploadJob(TestCaseWithFactory): |
583 | build, build.git_repository.owner, archive, distroseries, |
584 | PackagePublishingPocket.RELEASE, target_channel="edge") |
585 | path = "wheel-indep/dist/wheel_indep-0.0.1-py3-none-any.whl" |
586 | - expected = { |
587 | - "name": "wheel-indep", |
588 | - "version": "0.0.1", |
589 | - "summary": "Example description", |
590 | - "description": "Example long description\n", |
591 | - "binpackageformat": BinaryPackageFormat.WHL, |
592 | - "architecturespecific": False, |
593 | - "homepage": "", |
594 | - } |
595 | - self.assertEqual(expected, job._scanFile(datadir(path))) |
596 | + metadata = job._scanFile(datadir(path)) |
597 | + self.assertIsInstance(metadata, BinaryArtifactMetadata) |
598 | + self.assertThat( |
599 | + metadata, |
600 | + MatchesStructure( |
601 | + format=Equals(BinaryPackageFormat.WHL), |
602 | + name=Equals("wheel-indep"), |
603 | + version=Equals("0.0.1"), |
604 | + summary=Equals("Example description"), |
605 | + description=Equals("Example long description\n"), |
606 | + architecturespecific=Is(False), |
607 | + homepage=Equals(""), |
608 | + )) |
609 | |
610 | def test__scanFile_wheel_arch(self): |
611 | archive = self.factory.makeArchive() |
612 | @@ -262,16 +272,38 @@ class TestCIBuildUploadJob(TestCaseWithFactory): |
613 | build, build.git_repository.owner, archive, distroseries, |
614 | PackagePublishingPocket.RELEASE, target_channel="edge") |
615 | path = "wheel-arch/dist/wheel_arch-0.0.1-cp310-cp310-linux_x86_64.whl" |
616 | - expected = { |
617 | - "name": "wheel-arch", |
618 | - "version": "0.0.1", |
619 | - "summary": "Example description", |
620 | - "description": "Example long description\n", |
621 | - "binpackageformat": BinaryPackageFormat.WHL, |
622 | - "architecturespecific": True, |
623 | - "homepage": "http://example.com/", |
624 | - } |
625 | - self.assertEqual(expected, job._scanFile(datadir(path))) |
626 | + metadata = job._scanFile(datadir(path)) |
627 | + self.assertIsInstance(metadata, BinaryArtifactMetadata) |
628 | + self.assertThat( |
629 | + metadata, |
630 | + MatchesStructure( |
631 | + format=Equals(BinaryPackageFormat.WHL), |
632 | + name=Equals("wheel-arch"), |
633 | + version=Equals("0.0.1"), |
634 | + summary=Equals("Example description"), |
635 | + description=Equals("Example long description\n"), |
636 | + architecturespecific=Is(True), |
637 | + homepage=Equals("http://example.com/"), |
638 | + )) |
639 | + |
640 | + def test__scanFile_sdist(self): |
641 | + archive = self.factory.makeArchive() |
642 | + distroseries = self.factory.makeDistroSeries( |
643 | + distribution=archive.distribution) |
644 | + build = self.makeCIBuild(archive.distribution) |
645 | + job = CIBuildUploadJob.create( |
646 | + build, build.git_repository.owner, archive, distroseries, |
647 | + PackagePublishingPocket.RELEASE, target_channel="edge") |
648 | + path = "wheel-arch/dist/wheel-arch-0.0.1.tar.gz" |
649 | + metadata = job._scanFile(datadir(path)) |
650 | + self.assertIsInstance(metadata, SourceArtifactMetadata) |
651 | + self.assertThat( |
652 | + metadata, |
653 | + MatchesStructure.byEquality( |
654 | + format=SourcePackageFileType.SDIST, |
655 | + name="wheel-arch", |
656 | + version="0.0.1", |
657 | + )) |
658 | |
659 | def test__scanFile_conda_indep(self): |
660 | archive = self.factory.makeArchive() |
661 | @@ -282,17 +314,20 @@ class TestCIBuildUploadJob(TestCaseWithFactory): |
662 | build, build.git_repository.owner, archive, distroseries, |
663 | PackagePublishingPocket.RELEASE, target_channel="edge") |
664 | path = "conda-indep/dist/noarch/conda-indep-0.1-0.tar.bz2" |
665 | - expected = { |
666 | - "name": "conda-indep", |
667 | - "version": "0.1", |
668 | - "summary": "Example summary", |
669 | - "description": "Example description", |
670 | - "binpackageformat": BinaryPackageFormat.CONDA_V1, |
671 | - "architecturespecific": False, |
672 | - "homepage": "", |
673 | - "user_defined_fields": [("subdir", "noarch")], |
674 | - } |
675 | - self.assertEqual(expected, job._scanFile(datadir(path))) |
676 | + metadata = job._scanFile(datadir(path)) |
677 | + self.assertIsInstance(metadata, BinaryArtifactMetadata) |
678 | + self.assertThat( |
679 | + metadata, |
680 | + MatchesStructure( |
681 | + format=Equals(BinaryPackageFormat.CONDA_V1), |
682 | + name=Equals("conda-indep"), |
683 | + version=Equals("0.1"), |
684 | + summary=Equals("Example summary"), |
685 | + description=Equals("Example description"), |
686 | + architecturespecific=Is(False), |
687 | + homepage=Equals(""), |
688 | + user_defined_fields=Equals([("subdir", "noarch")]), |
689 | + )) |
690 | |
691 | def test__scanFile_conda_arch(self): |
692 | archive = self.factory.makeArchive() |
693 | @@ -303,17 +338,20 @@ class TestCIBuildUploadJob(TestCaseWithFactory): |
694 | build, build.git_repository.owner, archive, distroseries, |
695 | PackagePublishingPocket.RELEASE, target_channel="edge") |
696 | path = "conda-arch/dist/linux-64/conda-arch-0.1-0.tar.bz2" |
697 | - expected = { |
698 | - "name": "conda-arch", |
699 | - "version": "0.1", |
700 | - "summary": "Example summary", |
701 | - "description": "Example description", |
702 | - "binpackageformat": BinaryPackageFormat.CONDA_V1, |
703 | - "architecturespecific": True, |
704 | - "homepage": "http://example.com/", |
705 | - "user_defined_fields": [("subdir", "linux-64")], |
706 | - } |
707 | - self.assertEqual(expected, job._scanFile(datadir(path))) |
708 | + metadata = job._scanFile(datadir(path)) |
709 | + self.assertIsInstance(metadata, BinaryArtifactMetadata) |
710 | + self.assertThat( |
711 | + metadata, |
712 | + MatchesStructure( |
713 | + format=Equals(BinaryPackageFormat.CONDA_V1), |
714 | + name=Equals("conda-arch"), |
715 | + version=Equals("0.1"), |
716 | + summary=Equals("Example summary"), |
717 | + description=Equals("Example description"), |
718 | + architecturespecific=Is(True), |
719 | + homepage=Equals("http://example.com/"), |
720 | + user_defined_fields=Equals([("subdir", "linux-64")]), |
721 | + )) |
722 | |
723 | def test__scanFile_conda_v2_indep(self): |
724 | archive = self.factory.makeArchive() |
725 | @@ -324,17 +362,20 @@ class TestCIBuildUploadJob(TestCaseWithFactory): |
726 | build, build.git_repository.owner, archive, distroseries, |
727 | PackagePublishingPocket.RELEASE, target_channel="edge") |
728 | path = "conda-v2-indep/dist/noarch/conda-v2-indep-0.1-0.conda" |
729 | - expected = { |
730 | - "name": "conda-v2-indep", |
731 | - "version": "0.1", |
732 | - "summary": "Example summary", |
733 | - "description": "Example description", |
734 | - "binpackageformat": BinaryPackageFormat.CONDA_V2, |
735 | - "architecturespecific": False, |
736 | - "homepage": "", |
737 | - "user_defined_fields": [("subdir", "noarch")], |
738 | - } |
739 | - self.assertEqual(expected, job._scanFile(datadir(path))) |
740 | + metadata = job._scanFile(datadir(path)) |
741 | + self.assertIsInstance(metadata, BinaryArtifactMetadata) |
742 | + self.assertThat( |
743 | + metadata, |
744 | + MatchesStructure( |
745 | + format=Equals(BinaryPackageFormat.CONDA_V2), |
746 | + name=Equals("conda-v2-indep"), |
747 | + version=Equals("0.1"), |
748 | + summary=Equals("Example summary"), |
749 | + description=Equals("Example description"), |
750 | + architecturespecific=Is(False), |
751 | + homepage=Equals(""), |
752 | + user_defined_fields=Equals([("subdir", "noarch")]), |
753 | + )) |
754 | |
755 | def test__scanFile_conda_v2_arch(self): |
756 | archive = self.factory.makeArchive() |
757 | @@ -345,17 +386,20 @@ class TestCIBuildUploadJob(TestCaseWithFactory): |
758 | build, build.git_repository.owner, archive, distroseries, |
759 | PackagePublishingPocket.RELEASE, target_channel="edge") |
760 | path = "conda-v2-arch/dist/linux-64/conda-v2-arch-0.1-0.conda" |
761 | - expected = { |
762 | - "name": "conda-v2-arch", |
763 | - "version": "0.1", |
764 | - "summary": "Example summary", |
765 | - "description": "Example description", |
766 | - "binpackageformat": BinaryPackageFormat.CONDA_V2, |
767 | - "architecturespecific": True, |
768 | - "homepage": "http://example.com/", |
769 | - "user_defined_fields": [("subdir", "linux-64")], |
770 | - } |
771 | - self.assertEqual(expected, job._scanFile(datadir(path))) |
772 | + metadata = job._scanFile(datadir(path)) |
773 | + self.assertIsInstance(metadata, BinaryArtifactMetadata) |
774 | + self.assertThat( |
775 | + metadata, |
776 | + MatchesStructure( |
777 | + format=Equals(BinaryPackageFormat.CONDA_V2), |
778 | + name=Equals("conda-v2-arch"), |
779 | + version=Equals("0.1"), |
780 | + summary=Equals("Example summary"), |
781 | + description=Equals("Example description"), |
782 | + architecturespecific=Is(True), |
783 | + homepage=Equals("http://example.com/"), |
784 | + user_defined_fields=Equals([("subdir", "linux-64")]), |
785 | + )) |
786 | |
787 | def test_run_indep(self): |
788 | archive = self.factory.makeArchive( |
789 | @@ -384,6 +428,21 @@ class TestCIBuildUploadJob(TestCaseWithFactory): |
790 | with dbuser(job.config.dbuser): |
791 | JobRunner([job]).runAll() |
792 | |
793 | + self.assertThat(archive.getPublishedSources(), MatchesSetwise( |
794 | + MatchesStructure( |
795 | + sourcepackagename=MatchesStructure.byEquality( |
796 | + name=build.git_repository.target.name), |
797 | + sourcepackagerelease=MatchesStructure( |
798 | + ci_build=Equals(build), |
799 | + sourcepackagename=MatchesStructure.byEquality( |
800 | + name=build.git_repository.target.name), |
801 | + version=Equals("0.0.1"), |
802 | + format=Equals(SourcePackageType.CI_BUILD), |
803 | + architecturehintlist=Equals(""), |
804 | + creator=Equals(build.git_repository.owner), |
805 | + files=Equals([])), |
806 | + format=Equals(SourcePackageType.CI_BUILD), |
807 | + distroseries=Equals(distroseries)))) |
808 | self.assertThat(archive.getAllPublishedBinaries(), MatchesSetwise(*( |
809 | MatchesStructure( |
810 | binarypackagename=MatchesStructure.byEquality( |
811 | @@ -418,13 +477,17 @@ class TestCIBuildUploadJob(TestCaseWithFactory): |
812 | archive.distribution, distro_arch_series=dases[0]) |
813 | report = build.getOrCreateRevisionStatusReport("build:0") |
814 | report.setLog(b"log data") |
815 | - path = "wheel-arch/dist/wheel_arch-0.0.1-cp310-cp310-linux_x86_64.whl" |
816 | - with open(datadir(path), mode="rb") as f: |
817 | - report.attach(name=os.path.basename(path), data=f.read()) |
818 | - artifact = IStore(RevisionStatusArtifact).find( |
819 | + sdist_path = "wheel-arch/dist/wheel-arch-0.0.1.tar.gz" |
820 | + wheel_path = ( |
821 | + "wheel-arch/dist/wheel_arch-0.0.1-cp310-cp310-linux_x86_64.whl") |
822 | + with open(datadir(sdist_path), mode="rb") as f: |
823 | + report.attach(name=os.path.basename(sdist_path), data=f.read()) |
824 | + with open(datadir(wheel_path), mode="rb") as f: |
825 | + report.attach(name=os.path.basename(wheel_path), data=f.read()) |
826 | + artifacts = IStore(RevisionStatusArtifact).find( |
827 | RevisionStatusArtifact, |
828 | report=report, |
829 | - artifact_type=RevisionStatusArtifactType.BINARY).one() |
830 | + artifact_type=RevisionStatusArtifactType.BINARY).order_by("id") |
831 | job = CIBuildUploadJob.create( |
832 | build, build.git_repository.owner, archive, distroseries, |
833 | PackagePublishingPocket.RELEASE, target_channel="edge") |
834 | @@ -433,6 +496,24 @@ class TestCIBuildUploadJob(TestCaseWithFactory): |
835 | with dbuser(job.config.dbuser): |
836 | JobRunner([job]).runAll() |
837 | |
838 | + self.assertThat(archive.getPublishedSources(), MatchesSetwise( |
839 | + MatchesStructure( |
840 | + sourcepackagename=MatchesStructure.byEquality( |
841 | + name=build.git_repository.target.name), |
842 | + sourcepackagerelease=MatchesStructure( |
843 | + ci_build=Equals(build), |
844 | + sourcepackagename=MatchesStructure.byEquality( |
845 | + name=build.git_repository.target.name), |
846 | + version=Equals("0.0.1"), |
847 | + format=Equals(SourcePackageType.CI_BUILD), |
848 | + architecturehintlist=Equals(""), |
849 | + creator=Equals(build.git_repository.owner), |
850 | + files=MatchesSetwise( |
851 | + MatchesStructure.byEquality( |
852 | + libraryfile=artifacts[0].library_file, |
853 | + filetype=SourcePackageFileType.SDIST))), |
854 | + format=Equals(SourcePackageType.CI_BUILD), |
855 | + distroseries=Equals(distroseries)))) |
856 | self.assertThat(archive.getAllPublishedBinaries(), MatchesSetwise( |
857 | MatchesStructure( |
858 | binarypackagename=MatchesStructure.byEquality( |
859 | @@ -449,7 +530,7 @@ class TestCIBuildUploadJob(TestCaseWithFactory): |
860 | homepage=Equals("http://example.com/"), |
861 | files=MatchesSetwise( |
862 | MatchesStructure.byEquality( |
863 | - libraryfile=artifact.library_file, |
864 | + libraryfile=artifacts[1].library_file, |
865 | filetype=BinaryPackageFileType.WHL))), |
866 | binarypackageformat=Equals(BinaryPackageFormat.WHL), |
867 | distroarchseries=Equals(dases[0])))) |
868 | @@ -481,6 +562,21 @@ class TestCIBuildUploadJob(TestCaseWithFactory): |
869 | with dbuser(job.config.dbuser): |
870 | JobRunner([job]).runAll() |
871 | |
872 | + self.assertThat(archive.getPublishedSources(), MatchesSetwise( |
873 | + MatchesStructure( |
874 | + sourcepackagename=MatchesStructure.byEquality( |
875 | + name=build.git_repository.target.name), |
876 | + sourcepackagerelease=MatchesStructure( |
877 | + ci_build=Equals(build), |
878 | + sourcepackagename=MatchesStructure.byEquality( |
879 | + name=build.git_repository.target.name), |
880 | + version=Equals("0.1"), |
881 | + format=Equals(SourcePackageType.CI_BUILD), |
882 | + architecturehintlist=Equals(""), |
883 | + creator=Equals(build.git_repository.owner), |
884 | + files=Equals([])), |
885 | + format=Equals(SourcePackageType.CI_BUILD), |
886 | + distroseries=Equals(distroseries)))) |
887 | self.assertThat(archive.getAllPublishedBinaries(), MatchesSetwise( |
888 | MatchesStructure( |
889 | binarypackagename=MatchesStructure.byEquality( |
890 | @@ -529,6 +625,21 @@ class TestCIBuildUploadJob(TestCaseWithFactory): |
891 | with dbuser(job.config.dbuser): |
892 | JobRunner([job]).runAll() |
893 | |
894 | + self.assertThat(archive.getPublishedSources(), MatchesSetwise( |
895 | + MatchesStructure( |
896 | + sourcepackagename=MatchesStructure.byEquality( |
897 | + name=build.git_repository.target.name), |
898 | + sourcepackagerelease=MatchesStructure( |
899 | + ci_build=Equals(build), |
900 | + sourcepackagename=MatchesStructure.byEquality( |
901 | + name=build.git_repository.target.name), |
902 | + version=Equals("0.1"), |
903 | + format=Equals(SourcePackageType.CI_BUILD), |
904 | + architecturehintlist=Equals(""), |
905 | + creator=Equals(build.git_repository.owner), |
906 | + files=Equals([])), |
907 | + format=Equals(SourcePackageType.CI_BUILD), |
908 | + distroseries=Equals(distroseries)))) |
909 | self.assertThat(archive.getAllPublishedBinaries(), MatchesSetwise( |
910 | MatchesStructure( |
911 | binarypackagename=MatchesStructure.byEquality( |
912 | @@ -550,7 +661,7 @@ class TestCIBuildUploadJob(TestCaseWithFactory): |
913 | binarypackageformat=Equals(BinaryPackageFormat.CONDA_V2), |
914 | distroarchseries=Equals(dases[0])))) |
915 | |
916 | - def test_existing_release(self): |
917 | + def test_existing_source_and_binary_releases(self): |
918 | # A `CIBuildUploadJob` can be run even if the build in question was |
919 | # already uploaded somewhere, and in that case may add publications |
920 | # in other locations for the same package. |
921 | @@ -561,13 +672,16 @@ class TestCIBuildUploadJob(TestCaseWithFactory): |
922 | das = self.factory.makeDistroArchSeries(distroseries=distroseries) |
923 | build = self.makeCIBuild(archive.distribution, distro_arch_series=das) |
924 | report = build.getOrCreateRevisionStatusReport("build:0") |
925 | - path = "wheel-indep/dist/wheel_indep-0.0.1-py3-none-any.whl" |
926 | - with open(datadir(path), mode="rb") as f: |
927 | - report.attach(name=os.path.basename(path), data=f.read()) |
928 | - artifact = IStore(RevisionStatusArtifact).find( |
929 | + sdist_path = "wheel-indep/dist/wheel-indep-0.0.1.tar.gz" |
930 | + with open(datadir(sdist_path), mode="rb") as f: |
931 | + report.attach(name=os.path.basename(sdist_path), data=f.read()) |
932 | + wheel_path = "wheel-indep/dist/wheel_indep-0.0.1-py3-none-any.whl" |
933 | + with open(datadir(wheel_path), mode="rb") as f: |
934 | + report.attach(name=os.path.basename(wheel_path), data=f.read()) |
935 | + artifacts = IStore(RevisionStatusArtifact).find( |
936 | RevisionStatusArtifact, |
937 | report=report, |
938 | - artifact_type=RevisionStatusArtifactType.BINARY).one() |
939 | + artifact_type=RevisionStatusArtifactType.BINARY).order_by("id") |
940 | job = CIBuildUploadJob.create( |
941 | build, build.git_repository.owner, archive, distroseries, |
942 | PackagePublishingPocket.RELEASE, target_channel="edge") |
943 | @@ -582,12 +696,31 @@ class TestCIBuildUploadJob(TestCaseWithFactory): |
944 | with dbuser(job.config.dbuser): |
945 | JobRunner([job]).runAll() |
946 | |
947 | + spphs = archive.getPublishedSources() |
948 | + # The source publications are for the same source package release, |
949 | + # which has a single file attached to it. |
950 | + self.assertEqual(1, len({spph.sourcepackagename for spph in spphs})) |
951 | + self.assertEqual(1, len({spph.sourcepackagerelease for spph in spphs})) |
952 | + self.assertThat(spphs, MatchesSetwise(*( |
953 | + MatchesStructure( |
954 | + sourcepackagename=MatchesStructure.byEquality( |
955 | + name=build.git_repository.target.name), |
956 | + sourcepackagerelease=MatchesStructure( |
957 | + ci_build=Equals(build), |
958 | + files=MatchesSetwise( |
959 | + MatchesStructure.byEquality( |
960 | + libraryfile=artifacts[0].library_file, |
961 | + filetype=SourcePackageFileType.SDIST))), |
962 | + format=Equals(SourcePackageType.CI_BUILD), |
963 | + distroseries=Equals(distroseries), |
964 | + channel=Equals(channel)) |
965 | + for channel in ("edge", "0.0.1/edge")))) |
966 | bpphs = archive.getAllPublishedBinaries() |
967 | - # The publications are for the same binary package release, which |
968 | - # has a single file attached to it. |
969 | + # The binary publications are for the same binary package release, |
970 | + # which has a single file attached to it. |
971 | self.assertEqual(1, len({bpph.binarypackagename for bpph in bpphs})) |
972 | self.assertEqual(1, len({bpph.binarypackagerelease for bpph in bpphs})) |
973 | - self.assertThat(archive.getAllPublishedBinaries(), MatchesSetwise(*( |
974 | + self.assertThat(bpphs, MatchesSetwise(*( |
975 | MatchesStructure( |
976 | binarypackagename=MatchesStructure.byEquality( |
977 | name="wheel-indep"), |
978 | @@ -595,7 +728,79 @@ class TestCIBuildUploadJob(TestCaseWithFactory): |
979 | ci_build=Equals(build), |
980 | files=MatchesSetwise( |
981 | MatchesStructure.byEquality( |
982 | - libraryfile=artifact.library_file, |
983 | + libraryfile=artifacts[1].library_file, |
984 | + filetype=BinaryPackageFileType.WHL))), |
985 | + binarypackageformat=Equals(BinaryPackageFormat.WHL), |
986 | + distroarchseries=Equals(das), |
987 | + channel=Equals(channel)) |
988 | + for channel in ("edge", "0.0.1/edge")))) |
989 | + |
990 | + def test_existing_binary_release_no_existing_source_release(self): |
991 | + # A `CIBuildUploadJob` can be run even if the build in question was |
992 | + # already uploaded somewhere, and in that case may add publications |
993 | + # in other locations for the same package. This works even if there |
994 | + # was no existing source package release (because |
995 | + # `CIBuildUploadJob`s didn't always create one). |
996 | + archive = self.factory.makeArchive( |
997 | + repository_format=ArchiveRepositoryFormat.PYTHON) |
998 | + distroseries = self.factory.makeDistroSeries( |
999 | + distribution=archive.distribution) |
1000 | + das = self.factory.makeDistroArchSeries(distroseries=distroseries) |
1001 | + build = self.makeCIBuild(archive.distribution, distro_arch_series=das) |
1002 | + report = build.getOrCreateRevisionStatusReport("build:0") |
1003 | + sdist_path = "wheel-indep/dist/wheel-indep-0.0.1.tar.gz" |
1004 | + with open(datadir(sdist_path), mode="rb") as f: |
1005 | + report.attach(name=os.path.basename(sdist_path), data=f.read()) |
1006 | + wheel_path = "wheel-indep/dist/wheel_indep-0.0.1-py3-none-any.whl" |
1007 | + with open(datadir(wheel_path), mode="rb") as f: |
1008 | + report.attach(name=os.path.basename(wheel_path), data=f.read()) |
1009 | + artifacts = IStore(RevisionStatusArtifact).find( |
1010 | + RevisionStatusArtifact, |
1011 | + report=report, |
1012 | + artifact_type=RevisionStatusArtifactType.BINARY).order_by("id") |
1013 | + job = CIBuildUploadJob.create( |
1014 | + build, build.git_repository.owner, archive, distroseries, |
1015 | + PackagePublishingPocket.RELEASE, target_channel="edge") |
1016 | + transaction.commit() |
1017 | + with MockPatchObject(CIBuildUploadJob, "_uploadSources"): |
1018 | + with dbuser(job.config.dbuser): |
1019 | + JobRunner([job]).runAll() |
1020 | + job = CIBuildUploadJob.create( |
1021 | + build, build.git_repository.owner, archive, distroseries, |
1022 | + PackagePublishingPocket.RELEASE, target_channel="0.0.1/edge") |
1023 | + transaction.commit() |
1024 | + |
1025 | + with dbuser(job.config.dbuser): |
1026 | + JobRunner([job]).runAll() |
1027 | + |
1028 | + # There is a source publication for a new source package release. |
1029 | + self.assertThat(archive.getPublishedSources(), MatchesSetwise( |
1030 | + MatchesStructure( |
1031 | + sourcepackagename=MatchesStructure.byEquality( |
1032 | + name=build.git_repository.target.name), |
1033 | + sourcepackagerelease=MatchesStructure( |
1034 | + ci_build=Equals(build), |
1035 | + files=MatchesSetwise( |
1036 | + MatchesStructure.byEquality( |
1037 | + libraryfile=artifacts[0].library_file, |
1038 | + filetype=SourcePackageFileType.SDIST))), |
1039 | + format=Equals(SourcePackageType.CI_BUILD), |
1040 | + distroseries=Equals(distroseries), |
1041 | + channel=Equals("0.0.1/edge")))) |
1042 | + bpphs = archive.getAllPublishedBinaries() |
1043 | + # The binary publications are for the same binary package release, |
1044 | + # which has a single file attached to it. |
1045 | + self.assertEqual(1, len({bpph.binarypackagename for bpph in bpphs})) |
1046 | + self.assertEqual(1, len({bpph.binarypackagerelease for bpph in bpphs})) |
1047 | + self.assertThat(bpphs, MatchesSetwise(*( |
1048 | + MatchesStructure( |
1049 | + binarypackagename=MatchesStructure.byEquality( |
1050 | + name="wheel-indep"), |
1051 | + binarypackagerelease=MatchesStructure( |
1052 | + ci_build=Equals(build), |
1053 | + files=MatchesSetwise( |
1054 | + MatchesStructure.byEquality( |
1055 | + libraryfile=artifacts[1].library_file, |
1056 | filetype=BinaryPackageFileType.WHL))), |
1057 | binarypackageformat=Equals(BinaryPackageFormat.WHL), |
1058 | distroarchseries=Equals(das), |
This looks good.
I'd like the artifact metadata to be a typed structure (NamedTuple) rather than a dict.