Merge lp:~cjwatson/launchpad/snap-channels-ui into lp:launchpad

[Colin Watson]

Add UI for automatically releasing snap packages.

Add UI for automatically releasing snap packages.

This will require firewall changes first to let the appservers talk to search.apps.ubuntu.com (although there's a feature flag in the snap-channels-store-client branch so that can be disabled if something goes wrong).

[Thomi Richards (thomir-deactivatedaccount)]

Hi Colin,

The code looks fine, but when running './bin/test -cvvt snap' I get this message printed out while running many of the tests:

Traceback (most recent call last):
  File "/home/thomi/launchpad/lp-branches/snap-channels-ui/lib/lp/snappy/tests/test_snap.py", line 422, in test_getBuildSummariesForSnapBuildIds_log_size_field
    snap = self.factory.makeSnap()
  File "/home/thomi/launchpad/lp-branches/snap-channels-ui/lib/lp/testing/factory.py", line 390, in with_default_master_store
    return func(*args, **kw)
  File "/home/thomi/launchpad/lp-branches/snap-channels-ui/lib/lp/testing/factory.py", line 4664, in makeSnap
    store_channels=store_channels)
  File "/home/thomi/launchpad/lp-branches/snap-channels-ui/lib/lp/snappy/model/snap.py", line 531, in new
    if self.exists(owner, name):
  File "/home/thomi/launchpad/lp-branches/snap-channels-ui/lib/lp/snappy/model/snap.py", line 583, in exists
    return self._getByName(owner, name) is not None
  File "/home/thomi/launchpad/lp-branches/snap-channels-ui/lib/lp/snappy/model/snap.py", line 579, in _getByName
    Snap, Snap.owner == owner, Snap.name == name).one()
  File "/home/thomi/launchpad/lp-sourcedeps/eggs/storm-0.19.0.99_lpwithnodatetime_r408-py2.7-linux-i686.egg/storm/store.py", line 1162, in one
    result = self._store._connection.execute(select)
  File "/home/thomi/launchpad/lp-sourcedeps/eggs/storm-0.19.0.99_lpwithnodatetime_r408-py2.7-linux-i686.egg/storm/databases/postgres.py", line 266, in execute
    return Connection.execute(self, statement, params, noresult)
  File "/home/thomi/launchpad/lp-sourcedeps/eggs/storm-0.19.0.99_lpwithnodatetime_r408-py2.7-linux-i686.egg/storm/database.py", line 238, in execute
    raw_cursor = self.raw_execute(statement, params)
  File "/home/thomi/launchpad/lp-sourcedeps/eggs/storm-0.19.0.99_lpwithnodatetime_r408-py2.7-linux-i686.egg/storm/databases/postgres.py", line 276, in raw_execute
    return Connection.raw_execute(self, statement, params)
  File "/home/thomi/launchpad/lp-sourcedeps/eggs/storm-0.19.0.99_lpwithnodatetime_r408-py2.7-linux-i686.egg/storm/database.py", line 322, in raw_execute
    self._check_disconnect(raw_cursor.execute, *args)
  File "/home/thomi/launchpad/lp-sourcedeps/eggs/storm-0.19.0.99_lpwithnodatetime_r408-py2.7-linux-i686.egg/storm/database.py", line 371, in _check_disconnect
    return function(*args, **kwargs)
  File "/home/thomi/launchpad/lp-branches/snap-channels-ui/lib/lp/testing/pgsql.py", line 118, in execute
    return self.real_cursor.execute(*args, **kwargs)
ProgrammingError: column snap.auto_build does not exist
LINE 1: SELECT Snap.auto_build, Snap.auto_build_archive, Snap.auto_b...

I'm not convinced this has anything to do with your branch.... any ideas? I've run the clean, build and schema targets. Not sure what else could be causing this?

I'll leave a provisional +1 here in any case.

review: Approve

[Colin Watson (cjwatson)]

That implies that you haven't upgraded the database schema to its latest
revision. You can do "make schema" to burn your current database down
and start afresh, or use upgrade.py and security.py from
database/schema/ to apply patches in place. I'm not quite sure why the
"make schema" that you said you ran didn't apply
database/schema/patch-2209-69-4.sql which is in this branch, but perhaps
the database that you're running tests against is somewhere else?
upgrade.py gives you a bit more control and it should be easier to see
what's going on there, anyway.

After that, you'll run into Snap.store_channels similarly not existing.
The top three items in this branch stack (of which this branch is
currently the deepest) also depend on
https://code.launchpad.net/~cjwatson/launchpad/db-snap-channels/+merge/298803.
It's not listed as a prerequisite because database patches are targeted
at a separate trunk (lp:launchpad/db-devel), but db-snap-channels will
need to be merged, deployed in a fastdowntime, and then db-devel merged
into devel before it's possible to land snap-channels-store-client or
above.

=== modified file 'lib/lp/scripts/utilities/importfascist.py'
--- lib/lp/scripts/utilities/importfascist.py	2016-05-11 10:45:12 +0000
+++ lib/lp/scripts/utilities/importfascist.py	2016-07-16 07:54:57 +0000
@@ -1,4 +1,4 @@
-# Copyright 2009-2012 Canonical Ltd.  This software is licensed under the
+# Copyright 2009-2016 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 import __builtin__
@@ -27,6 +27,8 @@
 valid_imports_not_in_all = {
     'bzrlib.lsprof': set(['BzrProfiler']),
     'cookielib': set(['domain_match']),
+    # Exported in Python 3, but missing and so not exported in Python 2.
+    'json.decoder': set(['JSONDecodeError']),
     'openid.fetchers': set(['Urllib2Fetcher']),
     'openid.message': set(['NamespaceAliasRegistrationError']),
     'storm.database': set(['STATE_DISCONNECTED']),

=== modified file 'lib/lp/snappy/browser/snap.py'
--- lib/lp/snappy/browser/snap.py	2016-07-12 13:38:58 +0000
+++ lib/lp/snappy/browser/snap.py	2016-07-16 07:54:57 +0000
@@ -331,6 +331,9 @@
     # This is only required if store_upload is True.  Later validation takes
     # care of adjusting the required attribute.
     store_name = copy_field(ISnap['store_name'], required=True)
+    store_channels = copy_field(
+        ISnap['store_channels'],
+        value_type=Choice(vocabulary='SnapStoreChannel'), required=True)
 
 
 def log_oops(error, request):
@@ -368,9 +371,11 @@
         'auto_build_pocket',
         'store_upload',
         'store_name',
+        'store_channels',
         ]
     custom_widget('store_distro_series', LaunchpadRadioWidget)
     custom_widget('auto_build_archive', SnapArchiveWidget)
+    custom_widget('store_channels', LabeledMultiCheckBoxWidget)
 
     def initialize(self):
         """See `LaunchpadView`."""
@@ -457,6 +462,7 @@
             super(SnapAddView, self).validate_widgets(data, ['store_upload'])
             store_upload = data.get('store_upload', False)
             self.widgets['store_name'].context.required = store_upload
+            self.widgets['store_channels'].context.required = store_upload
         super(SnapAddView, self).validate_widgets(data, names=names)
 
     @action('Create snap package', name='create')
@@ -476,10 +482,11 @@
             auto_build=data['auto_build'],
             auto_build_archive=data['auto_build_archive'],
             auto_build_pocket=data['auto_build_pocket'],
-            private=private, store_upload=data['store_upload'],
+            processors=data['processors'], private=private,
+            store_upload=data['store_upload'],
             store_series=data['store_distro_series'].snappy_series,
-            store_name=data['store_name'], processors=data['processors'],
-            **kwargs)
+            store_name=data['store_name'],
+            store_channels=data.get('store_channels'), **kwargs)
         if data['store_upload']:
             self.requestAuthorization(snap)
         else:
@@ -549,6 +556,7 @@
                 data, ['store_upload'])
             store_upload = data.get('store_upload', False)
             self.widgets['store_name'].context.required = store_upload
+            self.widgets['store_channels'].context.required = store_upload
         super(BaseSnapEditView, self).validate_widgets(data, names=names)
 
     def _needStoreReauth(self, data):
@@ -587,6 +595,8 @@
         if not store_upload:
             if 'store_name' in data:
                 del data['store_name']
+            if 'store_channels' in data:
+                del data['store_channels']
         need_store_reauth = self._needStoreReauth(data)
         self.updateContextFromData(data)
         if need_store_reauth:
@@ -646,8 +656,10 @@
         'auto_build_pocket',
         'store_upload',
         'store_name',
+        'store_channels',
         ]
     custom_widget('store_distro_series', LaunchpadRadioWidget)
+    custom_widget('store_channels', LabeledMultiCheckBoxWidget)
     custom_widget('vcs', LaunchpadRadioWidget)
     custom_widget('git_ref', GitRefWidget)
     custom_widget('auto_build_archive', SnapArchiveWidget)

=== modified file 'lib/lp/snappy/browser/snapbuild.py'
--- lib/lp/snappy/browser/snapbuild.py	2016-07-14 15:19:03 +0000
+++ lib/lp/snappy/browser/snapbuild.py	2016-07-16 07:54:57 +0000
@@ -95,6 +95,11 @@
             return structured(
                 '<a href="%s">Manage this package in the store</a>',
                 job.store_url).escapedtext
+        elif job.store_url:
+            return structured(
+                '<a href="%s">Manage this package in the store</a><br />'
+                'Releasing package to channels failed: %s',
+                job.store_url, job.error_message).escapedtext
         else:
             return structured(
                 "Store upload failed: %s", job.error_message).escapedtext

=== modified file 'lib/lp/snappy/browser/tests/test_snap.py'
--- lib/lp/snappy/browser/tests/test_snap.py	2016-07-12 13:38:58 +0000
+++ lib/lp/snappy/browser/tests/test_snap.py	2016-07-16 07:54:57 +0000
@@ -63,6 +63,7 @@
     SNAP_TESTING_FLAGS,
     SnapPrivateFeatureDisabled,
     )
+from lp.snappy.interfaces.snapstoreclient import ISnapStoreClient
 from lp.testing import (
     admin_logged_in,
     BrowserTestCase,
@@ -125,6 +126,10 @@
     def test_private_feature_flag_disabled(self):
         # Without a private_snap feature flag, we will not create Snaps for
         # private contexts.
+        self.snap_store_client = FakeMethod()
+        self.snap_store_client.listChannels = FakeMethod(result=[])
+        self.useFixture(
+            ZopeUtilityFixture(self.snap_store_client, ISnapStoreClient))
         owner = self.factory.makePerson()
         branch = self.factory.makeAnyBranch(
             owner=owner, information_type=InformationType.USERDATA)
@@ -149,6 +154,15 @@
         with admin_logged_in():
             self.snappyseries = self.factory.makeSnappySeries(
                 usable_distro_series=[self.distroseries])
+        self.snap_store_client = FakeMethod()
+        self.snap_store_client.listChannels = FakeMethod(result=[
+            {"name": "stable", "display_name": "Stable"},
+            {"name": "edge", "display_name": "Edge"},
+            ])
+        self.snap_store_client.requestPackageUploadPermission = (
+            getUtility(ISnapStoreClient).requestPackageUploadPermission)
+        self.useFixture(
+            ZopeUtilityFixture(self.snap_store_client, ISnapStoreClient))
 
     def setUpDistroSeries(self):
         """Set up a distroseries with some available processors."""
@@ -371,6 +385,8 @@
         browser.getControl("Automatically upload to store").selected = True
         browser.getControl("Registered store package name").value = (
             "store-name")
+        self.assertFalse(browser.getControl("Stable").selected)
+        browser.getControl("Edge").selected = True
         root_macaroon = Macaroon()
         root_macaroon.add_third_party_caveat(
             urlsplit(config.launchpad.openid_provider_root).netloc, "",
@@ -395,7 +411,8 @@
             owner=self.person, distro_series=self.distroseries,
             name=u"snap-name", source=branch, store_upload=True,
             store_series=self.snappyseries, store_name=u"store-name",
-            store_secrets={"root": root_macaroon_raw}))
+            store_secrets={"root": root_macaroon_raw},
+            store_channels=["edge"]))
         self.assertThat(self.request, MatchesStructure.byEquality(
             url="http://sca.example/dev/api/acl/", method="POST"))
         expected_body = {
@@ -590,6 +607,15 @@
         with admin_logged_in():
             self.snappyseries = self.factory.makeSnappySeries(
                 usable_distro_series=[self.distroseries])
+        self.snap_store_client = FakeMethod()
+        self.snap_store_client.listChannels = FakeMethod(result=[
+            {"name": "stable", "display_name": "Stable"},
+            {"name": "edge", "display_name": "Edge"},
+            ])
+        self.snap_store_client.requestPackageUploadPermission = (
+            getUtility(ISnapStoreClient).requestPackageUploadPermission)
+        self.useFixture(
+            ZopeUtilityFixture(self.snap_store_client, ISnapStoreClient))
 
     def test_initial_store_series(self):
         # The initial store_series is the newest that is usable for the
@@ -839,10 +865,13 @@
         snap = self.factory.makeSnap(
             registrant=self.person, owner=self.person,
             distroseries=self.distroseries, store_upload=True,
-            store_series=self.snappyseries, store_name=u"one")
+            store_series=self.snappyseries, store_name=u"one",
+            store_channels=["edge"])
         view_url = canonical_url(snap, view_name="+edit")
         browser = self.getNonRedirectingBrowser(url=view_url, user=self.person)
         browser.getControl("Registered store package name").value = "two"
+        browser.getControl("Stable").selected = True
+        self.assertTrue(browser.getControl("Edge").selected)
         root_macaroon = Macaroon()
         root_macaroon.add_third_party_caveat(
             urlsplit(config.launchpad.openid_provider_root).netloc, "",
@@ -863,7 +892,8 @@
                 HTTPError, browser.getControl("Update snap package").click)
         login_person(self.person)
         self.assertThat(snap, MatchesStructure.byEquality(
-            store_name=u"two", store_secrets={"root": root_macaroon_raw}))
+            store_name=u"two", store_secrets={"root": root_macaroon_raw},
+            store_channels=["stable", "edge"]))
         self.assertThat(self.request, MatchesStructure.byEquality(
             url="http://sca.example/dev/api/acl/", method="POST"))
         expected_body = {

=== modified file 'lib/lp/snappy/browser/tests/test_snapbuild.py'
--- lib/lp/snappy/browser/tests/test_snapbuild.py	2016-07-14 15:19:03 +0000
+++ lib/lp/snappy/browser/tests/test_snapbuild.py	2016-07-16 07:54:57 +0000
@@ -113,6 +113,25 @@
                 attrs={"id": "store-upload-status"},
                 text="Store upload failed: Scan failed.")))
 
+    def test_store_upload_status_release_failed(self):
+        build = self.factory.makeSnapBuild(status=BuildStatus.FULLYBUILT)
+        job = getUtility(ISnapStoreUploadJobSource).create(build)
+        naked_job = removeSecurityProxy(job)
+        naked_job.job._status = JobStatus.FAILED
+        naked_job.store_url = "http://sca.example/dev/click-apps/1/rev/1/"
+        naked_job.error_message = "Failed to publish"
+        build_view = create_initialized_view(build, "+index")
+        self.assertThat(build_view(), soupmatchers.HTMLContains(
+            soupmatchers.Within(
+                soupmatchers.Tag(
+                    "store upload status", "li",
+                    attrs={"id": "store-upload-status"},
+                    text=(
+                        "Releasing package to channels failed: "
+                        "Failed to publish")),
+                soupmatchers.Tag(
+                    "store link", "a", attrs={"href": job.store_url}))))
+
 
 class TestSnapBuildOperations(BrowserTestCase):
 

=== modified file 'lib/lp/snappy/templates/snap-edit.pt'
--- lib/lp/snappy/templates/snap-edit.pt	2016-06-20 21:17:58 +0000
+++ lib/lp/snappy/templates/snap-edit.pt	2016-07-16 07:54:57 +0000
@@ -87,6 +87,10 @@
                   <metal:block use-macro="context/@@launchpad_form/widget_row" />
                 </tal:widget>
               </table>
+              <tal:widget define="widget nocall:view/widgets/store_channels"
+                          condition="widget/context/value_type/vocabulary">
+                <metal:block use-macro="context/@@launchpad_form/widget_row" />
+              </tal:widget>
               <p class="formHelp">
                 If you change any settings related to automatically uploading
                 builds of this snap to the store, then the login service will

=== modified file 'lib/lp/snappy/templates/snap-index.pt'
--- lib/lp/snappy/templates/snap-index.pt	2016-06-20 21:17:58 +0000
+++ lib/lp/snappy/templates/snap-index.pt	2016-07-16 07:54:57 +0000
@@ -104,6 +104,16 @@
           <a tal:replace="structure view/menu:overview/edit/fmt:icon"/>
         </dd>
       </dl>
+      <dl id="store_channels" tal:condition="context/store_channels">
+        <dt>Store channels:</dt>
+        <dd>
+          <span tal:content="context/store_channels"/>
+          <a tal:replace="structure view/menu:overview/edit/fmt:icon"/>
+        </dd>
+      </dl>
+      <p id="store_channels" tal:condition="not: context/store_channels">
+        This snap package will not be released to any channels on the store.
+      </p>
     </div>
     <p id="store_upload" tal:condition="not: context/store_upload">
       Builds of this snap package are not automatically uploaded to the store.

=== modified file 'lib/lp/snappy/templates/snap-new.pt'
--- lib/lp/snappy/templates/snap-new.pt	2016-07-12 13:38:58 +0000
+++ lib/lp/snappy/templates/snap-new.pt	2016-07-16 07:54:57 +0000
@@ -61,6 +61,10 @@
                 <tal:widget define="widget nocall:view/widgets/store_name">
                   <metal:block use-macro="context/@@launchpad_form/widget_row" />
                 </tal:widget>
+                <tal:widget define="widget nocall:view/widgets/store_channels"
+                            condition="widget/context/value_type/vocabulary">
+                  <metal:block use-macro="context/@@launchpad_form/widget_row" />
+                </tal:widget>
               </table>
               <p class="formHelp">
                 If you ask Launchpad to automatically upload builds of this

=== modified file 'lib/lp/snappy/vocabularies.py'
--- lib/lp/snappy/vocabularies.py	2016-05-09 09:28:58 +0000
+++ lib/lp/snappy/vocabularies.py	2016-07-16 07:54:57 +0000
@@ -11,13 +11,18 @@
     ]
 
 from storm.locals import Desc
-from zope.schema.vocabulary import SimpleTerm
+from zope.component import getUtility
+from zope.schema.vocabulary import (
+    SimpleTerm,
+    SimpleVocabulary,
+    )
 
 from lp.registry.model.distribution import Distribution
 from lp.registry.model.distroseries import DistroSeries
 from lp.registry.model.series import ACTIVE_STATUSES
 from lp.services.database.interfaces import IStore
 from lp.services.webapp.vocabulary import StormVocabularyBase
+from lp.snappy.interfaces.snapstoreclient import ISnapStoreClient
 from lp.snappy.model.snappyseries import (
     SnappyDistroSeries,
     SnappySeries,
@@ -109,3 +114,15 @@
     _clauses = SnappyDistroSeriesVocabulary._clauses + [
         SnappySeries.status.is_in(ACTIVE_STATUSES),
         ]
+
+
+class SnapStoreChannelVocabulary(SimpleVocabulary):
+    """A vocabulary for searching store channels."""
+
+    def __init__(self, context=None):
+        channels = getUtility(ISnapStoreClient).listChannels()
+        terms = [
+            self.createTerm(
+                channel["name"], channel["name"], channel["display_name"])
+            for channel in channels]
+        super(SnapStoreChannelVocabulary, self).__init__(terms)

=== modified file 'lib/lp/snappy/vocabularies.zcml'
--- lib/lp/snappy/vocabularies.zcml	2016-05-05 20:02:01 +0000
+++ lib/lp/snappy/vocabularies.zcml	2016-07-16 07:54:57 +0000
@@ -48,4 +48,15 @@
         <allow interface="lp.services.webapp.vocabulary.IHugeVocabulary" />
     </class>
 
+    <securedutility
+        name="SnapStoreChannel"
+        component="lp.snappy.vocabularies.SnapStoreChannelVocabulary"
+        provides="zope.schema.interfaces.IVocabularyFactory">
+        <allow interface="zope.schema.interfaces.IVocabularyFactory" />
+    </securedutility>
+
+    <class class="lp.snappy.vocabularies.SnapStoreChannelVocabulary">
+        <allow interface="zope.schema.interfaces.IVocabularyTokenized" />
+    </class>
+
 </configure>