Merge lp:~cjwatson/launchpad/snap-channels-ui into lp:launchpad

Proposed by Colin Watson on 2016-06-30
Status: Merged
Merged at revision: 18144
Proposed branch: lp:~cjwatson/launchpad/snap-channels-ui
Merge into: lp:launchpad
Prerequisite: lp:~cjwatson/launchpad/snap-channels-job
Diff against target: 351 lines (+122/-8)
10 files modified
lib/lp/scripts/utilities/importfascist.py (+3/-1)
lib/lp/snappy/browser/snap.py (+15/-3)
lib/lp/snappy/browser/snapbuild.py (+5/-0)
lib/lp/snappy/browser/tests/test_snap.py (+33/-3)
lib/lp/snappy/browser/tests/test_snapbuild.py (+19/-0)
lib/lp/snappy/templates/snap-edit.pt (+4/-0)
lib/lp/snappy/templates/snap-index.pt (+10/-0)
lib/lp/snappy/templates/snap-new.pt (+4/-0)
lib/lp/snappy/vocabularies.py (+18/-1)
lib/lp/snappy/vocabularies.zcml (+11/-0)
To merge this branch: bzr merge lp:~cjwatson/launchpad/snap-channels-ui
Reviewer Review Type Date Requested Status
Thomi Richards (community) Approve on 2016-07-06
Launchpad code reviewers 2016-06-30 Pending
Review via email: mp+298811@code.launchpad.net

Commit message

Add UI for automatically releasing snap packages.

Description of the change

Add UI for automatically releasing snap packages.

This will require firewall changes first to let the appservers talk to search.apps.ubuntu.com (although there's a feature flag in the snap-channels-store-client branch so that can be disabled if something goes wrong).

To post a comment you must log in.

Hi Colin,

The code looks fine, but when running './bin/test -cvvt snap' I get this message printed out while running many of the tests:

Traceback (most recent call last):
  File "/home/thomi/launchpad/lp-branches/snap-channels-ui/lib/lp/snappy/tests/test_snap.py", line 422, in test_getBuildSummariesForSnapBuildIds_log_size_field
    snap = self.factory.makeSnap()
  File "/home/thomi/launchpad/lp-branches/snap-channels-ui/lib/lp/testing/factory.py", line 390, in with_default_master_store
    return func(*args, **kw)
  File "/home/thomi/launchpad/lp-branches/snap-channels-ui/lib/lp/testing/factory.py", line 4664, in makeSnap
    store_channels=store_channels)
  File "/home/thomi/launchpad/lp-branches/snap-channels-ui/lib/lp/snappy/model/snap.py", line 531, in new
    if self.exists(owner, name):
  File "/home/thomi/launchpad/lp-branches/snap-channels-ui/lib/lp/snappy/model/snap.py", line 583, in exists
    return self._getByName(owner, name) is not None
  File "/home/thomi/launchpad/lp-branches/snap-channels-ui/lib/lp/snappy/model/snap.py", line 579, in _getByName
    Snap, Snap.owner == owner, Snap.name == name).one()
  File "/home/thomi/launchpad/lp-sourcedeps/eggs/storm-0.19.0.99_lpwithnodatetime_r408-py2.7-linux-i686.egg/storm/store.py", line 1162, in one
    result = self._store._connection.execute(select)
  File "/home/thomi/launchpad/lp-sourcedeps/eggs/storm-0.19.0.99_lpwithnodatetime_r408-py2.7-linux-i686.egg/storm/databases/postgres.py", line 266, in execute
    return Connection.execute(self, statement, params, noresult)
  File "/home/thomi/launchpad/lp-sourcedeps/eggs/storm-0.19.0.99_lpwithnodatetime_r408-py2.7-linux-i686.egg/storm/database.py", line 238, in execute
    raw_cursor = self.raw_execute(statement, params)
  File "/home/thomi/launchpad/lp-sourcedeps/eggs/storm-0.19.0.99_lpwithnodatetime_r408-py2.7-linux-i686.egg/storm/databases/postgres.py", line 276, in raw_execute
    return Connection.raw_execute(self, statement, params)
  File "/home/thomi/launchpad/lp-sourcedeps/eggs/storm-0.19.0.99_lpwithnodatetime_r408-py2.7-linux-i686.egg/storm/database.py", line 322, in raw_execute
    self._check_disconnect(raw_cursor.execute, *args)
  File "/home/thomi/launchpad/lp-sourcedeps/eggs/storm-0.19.0.99_lpwithnodatetime_r408-py2.7-linux-i686.egg/storm/database.py", line 371, in _check_disconnect
    return function(*args, **kwargs)
  File "/home/thomi/launchpad/lp-branches/snap-channels-ui/lib/lp/testing/pgsql.py", line 118, in execute
    return self.real_cursor.execute(*args, **kwargs)
ProgrammingError: column snap.auto_build does not exist
LINE 1: SELECT Snap.auto_build, Snap.auto_build_archive, Snap.auto_b...

I'm not convinced this has anything to do with your branch.... any ideas? I've run the clean, build and schema targets. Not sure what else could be causing this?

I'll leave a provisional +1 here in any case.

review: Approve
Colin Watson (cjwatson) wrote :

That implies that you haven't upgraded the database schema to its latest
revision. You can do "make schema" to burn your current database down
and start afresh, or use upgrade.py and security.py from
database/schema/ to apply patches in place. I'm not quite sure why the
"make schema" that you said you ran didn't apply
database/schema/patch-2209-69-4.sql which is in this branch, but perhaps
the database that you're running tests against is somewhere else?
upgrade.py gives you a bit more control and it should be easier to see
what's going on there, anyway.

After that, you'll run into Snap.store_channels similarly not existing.
The top three items in this branch stack (of which this branch is
currently the deepest) also depend on
https://code.launchpad.net/~cjwatson/launchpad/db-snap-channels/+merge/298803.
It's not listed as a prerequisite because database patches are targeted
at a separate trunk (lp:launchpad/db-devel), but db-snap-channels will
need to be merged, deployed in a fastdowntime, and then db-devel merged
into devel before it's possible to land snap-channels-store-client or
above.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/lp/scripts/utilities/importfascist.py'
2--- lib/lp/scripts/utilities/importfascist.py 2016-05-11 10:45:12 +0000
3+++ lib/lp/scripts/utilities/importfascist.py 2016-07-16 07:54:57 +0000
4@@ -1,4 +1,4 @@
5-# Copyright 2009-2012 Canonical Ltd. This software is licensed under the
6+# Copyright 2009-2016 Canonical Ltd. This software is licensed under the
7 # GNU Affero General Public License version 3 (see the file LICENSE).
8
9 import __builtin__
10@@ -27,6 +27,8 @@
11 valid_imports_not_in_all = {
12 'bzrlib.lsprof': set(['BzrProfiler']),
13 'cookielib': set(['domain_match']),
14+ # Exported in Python 3, but missing and so not exported in Python 2.
15+ 'json.decoder': set(['JSONDecodeError']),
16 'openid.fetchers': set(['Urllib2Fetcher']),
17 'openid.message': set(['NamespaceAliasRegistrationError']),
18 'storm.database': set(['STATE_DISCONNECTED']),
19
20=== modified file 'lib/lp/snappy/browser/snap.py'
21--- lib/lp/snappy/browser/snap.py 2016-07-12 13:38:58 +0000
22+++ lib/lp/snappy/browser/snap.py 2016-07-16 07:54:57 +0000
23@@ -331,6 +331,9 @@
24 # This is only required if store_upload is True. Later validation takes
25 # care of adjusting the required attribute.
26 store_name = copy_field(ISnap['store_name'], required=True)
27+ store_channels = copy_field(
28+ ISnap['store_channels'],
29+ value_type=Choice(vocabulary='SnapStoreChannel'), required=True)
30
31
32 def log_oops(error, request):
33@@ -368,9 +371,11 @@
34 'auto_build_pocket',
35 'store_upload',
36 'store_name',
37+ 'store_channels',
38 ]
39 custom_widget('store_distro_series', LaunchpadRadioWidget)
40 custom_widget('auto_build_archive', SnapArchiveWidget)
41+ custom_widget('store_channels', LabeledMultiCheckBoxWidget)
42
43 def initialize(self):
44 """See `LaunchpadView`."""
45@@ -457,6 +462,7 @@
46 super(SnapAddView, self).validate_widgets(data, ['store_upload'])
47 store_upload = data.get('store_upload', False)
48 self.widgets['store_name'].context.required = store_upload
49+ self.widgets['store_channels'].context.required = store_upload
50 super(SnapAddView, self).validate_widgets(data, names=names)
51
52 @action('Create snap package', name='create')
53@@ -476,10 +482,11 @@
54 auto_build=data['auto_build'],
55 auto_build_archive=data['auto_build_archive'],
56 auto_build_pocket=data['auto_build_pocket'],
57- private=private, store_upload=data['store_upload'],
58+ processors=data['processors'], private=private,
59+ store_upload=data['store_upload'],
60 store_series=data['store_distro_series'].snappy_series,
61- store_name=data['store_name'], processors=data['processors'],
62- **kwargs)
63+ store_name=data['store_name'],
64+ store_channels=data.get('store_channels'), **kwargs)
65 if data['store_upload']:
66 self.requestAuthorization(snap)
67 else:
68@@ -549,6 +556,7 @@
69 data, ['store_upload'])
70 store_upload = data.get('store_upload', False)
71 self.widgets['store_name'].context.required = store_upload
72+ self.widgets['store_channels'].context.required = store_upload
73 super(BaseSnapEditView, self).validate_widgets(data, names=names)
74
75 def _needStoreReauth(self, data):
76@@ -587,6 +595,8 @@
77 if not store_upload:
78 if 'store_name' in data:
79 del data['store_name']
80+ if 'store_channels' in data:
81+ del data['store_channels']
82 need_store_reauth = self._needStoreReauth(data)
83 self.updateContextFromData(data)
84 if need_store_reauth:
85@@ -646,8 +656,10 @@
86 'auto_build_pocket',
87 'store_upload',
88 'store_name',
89+ 'store_channels',
90 ]
91 custom_widget('store_distro_series', LaunchpadRadioWidget)
92+ custom_widget('store_channels', LabeledMultiCheckBoxWidget)
93 custom_widget('vcs', LaunchpadRadioWidget)
94 custom_widget('git_ref', GitRefWidget)
95 custom_widget('auto_build_archive', SnapArchiveWidget)
96
97=== modified file 'lib/lp/snappy/browser/snapbuild.py'
98--- lib/lp/snappy/browser/snapbuild.py 2016-07-14 15:19:03 +0000
99+++ lib/lp/snappy/browser/snapbuild.py 2016-07-16 07:54:57 +0000
100@@ -95,6 +95,11 @@
101 return structured(
102 '<a href="%s">Manage this package in the store</a>',
103 job.store_url).escapedtext
104+ elif job.store_url:
105+ return structured(
106+ '<a href="%s">Manage this package in the store</a><br />'
107+ 'Releasing package to channels failed: %s',
108+ job.store_url, job.error_message).escapedtext
109 else:
110 return structured(
111 "Store upload failed: %s", job.error_message).escapedtext
112
113=== modified file 'lib/lp/snappy/browser/tests/test_snap.py'
114--- lib/lp/snappy/browser/tests/test_snap.py 2016-07-12 13:38:58 +0000
115+++ lib/lp/snappy/browser/tests/test_snap.py 2016-07-16 07:54:57 +0000
116@@ -63,6 +63,7 @@
117 SNAP_TESTING_FLAGS,
118 SnapPrivateFeatureDisabled,
119 )
120+from lp.snappy.interfaces.snapstoreclient import ISnapStoreClient
121 from lp.testing import (
122 admin_logged_in,
123 BrowserTestCase,
124@@ -125,6 +126,10 @@
125 def test_private_feature_flag_disabled(self):
126 # Without a private_snap feature flag, we will not create Snaps for
127 # private contexts.
128+ self.snap_store_client = FakeMethod()
129+ self.snap_store_client.listChannels = FakeMethod(result=[])
130+ self.useFixture(
131+ ZopeUtilityFixture(self.snap_store_client, ISnapStoreClient))
132 owner = self.factory.makePerson()
133 branch = self.factory.makeAnyBranch(
134 owner=owner, information_type=InformationType.USERDATA)
135@@ -149,6 +154,15 @@
136 with admin_logged_in():
137 self.snappyseries = self.factory.makeSnappySeries(
138 usable_distro_series=[self.distroseries])
139+ self.snap_store_client = FakeMethod()
140+ self.snap_store_client.listChannels = FakeMethod(result=[
141+ {"name": "stable", "display_name": "Stable"},
142+ {"name": "edge", "display_name": "Edge"},
143+ ])
144+ self.snap_store_client.requestPackageUploadPermission = (
145+ getUtility(ISnapStoreClient).requestPackageUploadPermission)
146+ self.useFixture(
147+ ZopeUtilityFixture(self.snap_store_client, ISnapStoreClient))
148
149 def setUpDistroSeries(self):
150 """Set up a distroseries with some available processors."""
151@@ -371,6 +385,8 @@
152 browser.getControl("Automatically upload to store").selected = True
153 browser.getControl("Registered store package name").value = (
154 "store-name")
155+ self.assertFalse(browser.getControl("Stable").selected)
156+ browser.getControl("Edge").selected = True
157 root_macaroon = Macaroon()
158 root_macaroon.add_third_party_caveat(
159 urlsplit(config.launchpad.openid_provider_root).netloc, "",
160@@ -395,7 +411,8 @@
161 owner=self.person, distro_series=self.distroseries,
162 name=u"snap-name", source=branch, store_upload=True,
163 store_series=self.snappyseries, store_name=u"store-name",
164- store_secrets={"root": root_macaroon_raw}))
165+ store_secrets={"root": root_macaroon_raw},
166+ store_channels=["edge"]))
167 self.assertThat(self.request, MatchesStructure.byEquality(
168 url="http://sca.example/dev/api/acl/", method="POST"))
169 expected_body = {
170@@ -590,6 +607,15 @@
171 with admin_logged_in():
172 self.snappyseries = self.factory.makeSnappySeries(
173 usable_distro_series=[self.distroseries])
174+ self.snap_store_client = FakeMethod()
175+ self.snap_store_client.listChannels = FakeMethod(result=[
176+ {"name": "stable", "display_name": "Stable"},
177+ {"name": "edge", "display_name": "Edge"},
178+ ])
179+ self.snap_store_client.requestPackageUploadPermission = (
180+ getUtility(ISnapStoreClient).requestPackageUploadPermission)
181+ self.useFixture(
182+ ZopeUtilityFixture(self.snap_store_client, ISnapStoreClient))
183
184 def test_initial_store_series(self):
185 # The initial store_series is the newest that is usable for the
186@@ -839,10 +865,13 @@
187 snap = self.factory.makeSnap(
188 registrant=self.person, owner=self.person,
189 distroseries=self.distroseries, store_upload=True,
190- store_series=self.snappyseries, store_name=u"one")
191+ store_series=self.snappyseries, store_name=u"one",
192+ store_channels=["edge"])
193 view_url = canonical_url(snap, view_name="+edit")
194 browser = self.getNonRedirectingBrowser(url=view_url, user=self.person)
195 browser.getControl("Registered store package name").value = "two"
196+ browser.getControl("Stable").selected = True
197+ self.assertTrue(browser.getControl("Edge").selected)
198 root_macaroon = Macaroon()
199 root_macaroon.add_third_party_caveat(
200 urlsplit(config.launchpad.openid_provider_root).netloc, "",
201@@ -863,7 +892,8 @@
202 HTTPError, browser.getControl("Update snap package").click)
203 login_person(self.person)
204 self.assertThat(snap, MatchesStructure.byEquality(
205- store_name=u"two", store_secrets={"root": root_macaroon_raw}))
206+ store_name=u"two", store_secrets={"root": root_macaroon_raw},
207+ store_channels=["stable", "edge"]))
208 self.assertThat(self.request, MatchesStructure.byEquality(
209 url="http://sca.example/dev/api/acl/", method="POST"))
210 expected_body = {
211
212=== modified file 'lib/lp/snappy/browser/tests/test_snapbuild.py'
213--- lib/lp/snappy/browser/tests/test_snapbuild.py 2016-07-14 15:19:03 +0000
214+++ lib/lp/snappy/browser/tests/test_snapbuild.py 2016-07-16 07:54:57 +0000
215@@ -113,6 +113,25 @@
216 attrs={"id": "store-upload-status"},
217 text="Store upload failed: Scan failed.")))
218
219+ def test_store_upload_status_release_failed(self):
220+ build = self.factory.makeSnapBuild(status=BuildStatus.FULLYBUILT)
221+ job = getUtility(ISnapStoreUploadJobSource).create(build)
222+ naked_job = removeSecurityProxy(job)
223+ naked_job.job._status = JobStatus.FAILED
224+ naked_job.store_url = "http://sca.example/dev/click-apps/1/rev/1/"
225+ naked_job.error_message = "Failed to publish"
226+ build_view = create_initialized_view(build, "+index")
227+ self.assertThat(build_view(), soupmatchers.HTMLContains(
228+ soupmatchers.Within(
229+ soupmatchers.Tag(
230+ "store upload status", "li",
231+ attrs={"id": "store-upload-status"},
232+ text=(
233+ "Releasing package to channels failed: "
234+ "Failed to publish")),
235+ soupmatchers.Tag(
236+ "store link", "a", attrs={"href": job.store_url}))))
237+
238
239 class TestSnapBuildOperations(BrowserTestCase):
240
241
242=== modified file 'lib/lp/snappy/templates/snap-edit.pt'
243--- lib/lp/snappy/templates/snap-edit.pt 2016-06-20 21:17:58 +0000
244+++ lib/lp/snappy/templates/snap-edit.pt 2016-07-16 07:54:57 +0000
245@@ -87,6 +87,10 @@
246 <metal:block use-macro="context/@@launchpad_form/widget_row" />
247 </tal:widget>
248 </table>
249+ <tal:widget define="widget nocall:view/widgets/store_channels"
250+ condition="widget/context/value_type/vocabulary">
251+ <metal:block use-macro="context/@@launchpad_form/widget_row" />
252+ </tal:widget>
253 <p class="formHelp">
254 If you change any settings related to automatically uploading
255 builds of this snap to the store, then the login service will
256
257=== modified file 'lib/lp/snappy/templates/snap-index.pt'
258--- lib/lp/snappy/templates/snap-index.pt 2016-06-20 21:17:58 +0000
259+++ lib/lp/snappy/templates/snap-index.pt 2016-07-16 07:54:57 +0000
260@@ -104,6 +104,16 @@
261 <a tal:replace="structure view/menu:overview/edit/fmt:icon"/>
262 </dd>
263 </dl>
264+ <dl id="store_channels" tal:condition="context/store_channels">
265+ <dt>Store channels:</dt>
266+ <dd>
267+ <span tal:content="context/store_channels"/>
268+ <a tal:replace="structure view/menu:overview/edit/fmt:icon"/>
269+ </dd>
270+ </dl>
271+ <p id="store_channels" tal:condition="not: context/store_channels">
272+ This snap package will not be released to any channels on the store.
273+ </p>
274 </div>
275 <p id="store_upload" tal:condition="not: context/store_upload">
276 Builds of this snap package are not automatically uploaded to the store.
277
278=== modified file 'lib/lp/snappy/templates/snap-new.pt'
279--- lib/lp/snappy/templates/snap-new.pt 2016-07-12 13:38:58 +0000
280+++ lib/lp/snappy/templates/snap-new.pt 2016-07-16 07:54:57 +0000
281@@ -61,6 +61,10 @@
282 <tal:widget define="widget nocall:view/widgets/store_name">
283 <metal:block use-macro="context/@@launchpad_form/widget_row" />
284 </tal:widget>
285+ <tal:widget define="widget nocall:view/widgets/store_channels"
286+ condition="widget/context/value_type/vocabulary">
287+ <metal:block use-macro="context/@@launchpad_form/widget_row" />
288+ </tal:widget>
289 </table>
290 <p class="formHelp">
291 If you ask Launchpad to automatically upload builds of this
292
293=== modified file 'lib/lp/snappy/vocabularies.py'
294--- lib/lp/snappy/vocabularies.py 2016-05-09 09:28:58 +0000
295+++ lib/lp/snappy/vocabularies.py 2016-07-16 07:54:57 +0000
296@@ -11,13 +11,18 @@
297 ]
298
299 from storm.locals import Desc
300-from zope.schema.vocabulary import SimpleTerm
301+from zope.component import getUtility
302+from zope.schema.vocabulary import (
303+ SimpleTerm,
304+ SimpleVocabulary,
305+ )
306
307 from lp.registry.model.distribution import Distribution
308 from lp.registry.model.distroseries import DistroSeries
309 from lp.registry.model.series import ACTIVE_STATUSES
310 from lp.services.database.interfaces import IStore
311 from lp.services.webapp.vocabulary import StormVocabularyBase
312+from lp.snappy.interfaces.snapstoreclient import ISnapStoreClient
313 from lp.snappy.model.snappyseries import (
314 SnappyDistroSeries,
315 SnappySeries,
316@@ -109,3 +114,15 @@
317 _clauses = SnappyDistroSeriesVocabulary._clauses + [
318 SnappySeries.status.is_in(ACTIVE_STATUSES),
319 ]
320+
321+
322+class SnapStoreChannelVocabulary(SimpleVocabulary):
323+ """A vocabulary for searching store channels."""
324+
325+ def __init__(self, context=None):
326+ channels = getUtility(ISnapStoreClient).listChannels()
327+ terms = [
328+ self.createTerm(
329+ channel["name"], channel["name"], channel["display_name"])
330+ for channel in channels]
331+ super(SnapStoreChannelVocabulary, self).__init__(terms)
332
333=== modified file 'lib/lp/snappy/vocabularies.zcml'
334--- lib/lp/snappy/vocabularies.zcml 2016-05-05 20:02:01 +0000
335+++ lib/lp/snappy/vocabularies.zcml 2016-07-16 07:54:57 +0000
336@@ -48,4 +48,15 @@
337 <allow interface="lp.services.webapp.vocabulary.IHugeVocabulary" />
338 </class>
339
340+ <securedutility
341+ name="SnapStoreChannel"
342+ component="lp.snappy.vocabularies.SnapStoreChannelVocabulary"
343+ provides="zope.schema.interfaces.IVocabularyFactory">
344+ <allow interface="zope.schema.interfaces.IVocabularyFactory" />
345+ </securedutility>
346+
347+ <class class="lp.snappy.vocabularies.SnapStoreChannelVocabulary">
348+ <allow interface="zope.schema.interfaces.IVocabularyTokenized" />
349+ </class>
350+
351 </configure>