Merge lp:~sergiusens/ubuntu-touch-session/install_clicks into lp:ubuntu-touch-session

[Sergio Schvezov]

Upfront I'm not sure about the default path (/usr/share is good, but what comes after I don't know).

Sync with unity8, not sure we want that

[lp:~sergiusens/ubuntu-touch-session/install_clicks]

97. By Sergio Schvezov on 2013-08-22

Unity8 sync.

[Ricardo Salveti (rsalveti)]

Just curious on why do we need this to run at every boot.

review: Needs Fixing

[Sergio Schvezov (sergiusens)]

> Just curious on why do we need this to run at every boot.

Some discussion on possibility of removing a click package that we don't wan't removed from the system.

[Ted Gould (ted)]

It seems like auto-upgrading defeats the point of having different versions per user.

Also, this doesn't seem like something that unity8 needs to block on? It has to handle adding click packages at any time anyway.

Lastly, if this is needed, it belongs in the click package not the touch session package.

[Sergio Schvezov (sergiusens)]

> It seems like auto-upgrading defeats the point of having different versions
> per user.

This only affects system provided packages and not regular packages you would install from the store. User installed packages would allow this.

> Also, this doesn't seem like something that unity8 needs to block on? It has
> to handle adding click packages at any time anyway.

I only added it to avoid a slow unity8 because some click package is being installed in the background, I'll remove it and we can come back to this later.

> Lastly, if this is needed, it belongs in the click package not the touch
> session package.

this is more of a touch specific thing, I don't want desktop users to be hit by this, yet at least.

[Ted Gould (ted)]

On Thu, 2013-08-22 at 21:50 +0000, Sergio Schvezov wrote:

> > It seems like auto-upgrading defeats the point of having different versions
> > per user.
>
>
> This only affects system provided packages and not regular packages
> you would install from the store. User installed packages would allow
> this.

System provided packages are likely to be set by the carrier, and will
probably include things like Facebook and your favorite shopping app.
It seems like those shouldn't autoupdate.

> > Lastly, if this is needed, it belongs in the click package not the touch
> > session package.
>
>
> this is more of a touch specific thing, I don't want desktop users to
> be hit by this, yet at least.

It seems like if you're using click packages on the touch image or the
desktop image you'd want the same behavior.

Could you perhaps post about this on the mailing list for Click
packages? Seems like there'll be a lot of folks interested.

Ubuntu Appstore Developers
<email address hidden>

[Loïc Minier (lool)]

I guess there are a number of different use cases:
a) package is in app store but we want to preinstall it
b) package is not in app store but we want to preinstall it

and another subcase:
1) package may be removed by end-user
2) package may not be removed by end-user

and what we want to decide is whether we're upgrading/reinstalling packages and whether we do this on every boot.

Now some conclusions:
- it seems clear to me that for a), it doesn't matter whether we update it or not as updates can be delivered from appstore, so we could optionally skip this on boot
- it seems natural that things that are in appstore and hence can be installed after the fact should be uninstallable, so that a) should imply 1)
- it is obvious that in the case of b), we must update it on each boot
- it seems natural that in the case of b), we should not allow removal -- otherwise lets just put it in the appstore

So what I'd propose is followings:
* /usr/share/preinstalled/click and /custom/preinstalled/click contain clicks that aren't in the appstore and must be installed or upgraded automatically and can't be removed (no flag to allow this for now, but should add one)
* /opt/click.u.c is prepopulated in the rootfs with appstore packages that we want to install by default but may remove, but is never upgraded (which I think is what happens with writable pathes?)

We should add a flag for packages that can't be removed (both for appstore and preinstalled packages).

[Loïc Minier (lool)]

Minor points: I don't care about which package this lives in (click package is meant to be generic too, but that's ok) and boot time is IMO ok, but we could also do it on session startup since that's per-user, so nicer I guess.

[Sergio Schvezov (sergiusens)]

> I guess there are a number of different use cases:
> a) package is in app store but we want to preinstall it
> b) package is not in app store but we want to preinstall it
>
> and another subcase:
> 1) package may be removed by end-user
> 2) package may not be removed by end-user
>
> and what we want to decide is whether we're upgrading/reinstalling packages
> and whether we do this on every boot.
>
>
> Now some conclusions:
> - it seems clear to me that for a), it doesn't matter whether we update it or
> not as updates can be delivered from appstore, so we could optionally skip
> this on boot
> - it seems natural that things that are in appstore and hence can be installed
> after the fact should be uninstallable, so that a) should imply 1)
> - it is obvious that in the case of b), we must update it on each boot
> - it seems natural that in the case of b), we should not allow removal --
> otherwise lets just put it in the appstore
>
> So what I'd propose is followings:
> * /usr/share/preinstalled/click and /custom/preinstalled/click contain clicks
> that aren't in the appstore and must be installed or upgraded automatically
> and can't be removed (no flag to allow this for now, but should add one)
> * /opt/click.u.c is prepopulated in the rootfs with appstore packages that we
> want to install by default but may remove, but is never upgraded (which I
> think is what happens with writable pathes?)

Makes sense. Let's do this.

I will add one more thing which is to check the image upgrade version (as discussed with rsalveti) so we can only run this on first boot after an upgrade.

> We should add a flag for packages that can't be removed (both for appstore and
> preinstalled packages).

Yes, this would allow for doing the only run the package install on upgrade

[Ted Gould (ted)]

On Thu, 2013-08-22 at 22:05 +0000, Loïc Minier wrote:

> I guess there are a number of different use cases:
> a) package is in app store but we want to preinstall it
> b) package is not in app store but we want to preinstall it
>
> and another subcase:
> 1) package may be removed by end-user
> 2) package may not be removed by end-user
>
>
>
> and what we want to decide is whether we're upgrading/reinstalling
> packages and whether we do this on every boot.

To be clear, we can't do it on boot because versions are per-user and so
are hooks that need to be run by the click system. Also, at boot the
user's home directory may not be decrypted.

We probably do need something on the user session init to handle the
case of clicks that are on the image (not writable) and the version has
been upgraded. This way the per-user click hooks can be run.

> Now some conclusions:
> - it seems clear to me that for a), it doesn't matter whether we
> update it or not as updates can be delivered from appstore, so we
> could optionally skip this on boot
> - it seems natural that things that are in appstore and hence can be
> installed after the fact should be uninstallable, so that a) should
> imply 1)
> - it is obvious that in the case of b), we must update it on each boot
> - it seems natural that in the case of b), we should not allow removal
> -- otherwise lets just put it in the appstore

I think that we can't simplify things like this with the per-user
installation. There's no reason both users on a system would have the
same version of Facebook whether it was pre-installed or not. It's a
disk space issue, not an automatic update one. And, as a user, I could
not want to update something.

> So what I'd propose is followings:
> * /usr/share/preinstalled/click and /custom/preinstalled/click contain
> clicks that aren't in the appstore and must be installed or upgraded
> automatically and can't be removed (no flag to allow this for now, but
> should add one)
> * /opt/click.u.c is prepopulated in the rootfs with appstore packages
> that we want to install by default but may remove, but is never
> upgraded (which I think is what happens with writable pathes?)

Hmm, I don't think that we should have paths hard coded anywhere but in
the click package. So this is, in my opinion a strong vote for putting
any upstart job in the click package directly.

> We should add a flag for packages that can't be removed (both for
> appstore and preinstalled packages).

Wouldn't they be on a read-only partition? Isn't that "can't be
removed" enough? :-)

Ted

[Loïc Minier (lool)]

On Fri, Aug 23, 2013, Ted Gould wrote:
> > and what we want to decide is whether we're upgrading/reinstalling
> > packages and whether we do this on every boot.
>
> To be clear, we can't do it on boot because versions are per-user and so
> are hooks that need to be run by the click system. Also, at boot the
> user's home directory may not be decrypted.
>
> We probably do need something on the user session init to handle the
> case of clicks that are on the image (not writable) and the version has
> been upgraded. This way the per-user click hooks can be run.

This was a fairly broad "on boot"; on session startup works too, and I
agree it's a better choice. +1 on doing this in the session startup.

> I think that we can't simplify things like this with the per-user
> installation. There's no reason both users on a system would have the
> same version of Facebook whether it was pre-installed or not. It's a
> disk space issue, not an automatic update one. And, as a user, I could
> not want to update something.

I tried to address these cases; I think it's good enough as a first
iteration. Consider we don't even have click package updates or
removals yet! Let's implement this for now, and iterate on it.

> Hmm, I don't think that we should have paths hard coded anywhere but in
> the click package. So this is, in my opinion a strong vote for putting
> any upstart job in the click package directly.

Not sure where click.u.c is currently hardcoded; it seems a good idea to
keep things together I agree.

> > We should add a flag for packages that can't be removed (both for
> > appstore and preinstalled packages).
>
> Wouldn't they be on a read-only partition? Isn't that "can't be
> removed" enough? :-)

Dont think this relates at all; first, /opt/click.u.c is not r/o but rw
(as you need to be able to install/upgrade/remove packages); second, we
want package updatable by the appstore that can't be user-removed. E.g.
OEM customizations. But that's kind of a longer term problem.

--
Loïc Minier

[lp:~sergiusens/ubuntu-touch-session/install_clicks]

98. By Sergio Schvezov on 2013-08-23

Removing sync for installation.

99. By Sergio Schvezov on 2013-08-23

Install from /custom/preinstalled/click as well

100. By Sergio Schvezov on 2013-08-23

Stamping the last run with /etc/ubuntu-build

[Colin Watson (cjwatson)]

I understand that the world couldn't stop while I was away :-), but it's
unfortunate that this branch was merged when I was on holiday and didn't
have the opportunity to comment on the fundamental design issues around
preinstalled click packages, which is an item on my near-term to-do list
to improve (it wasn't a priority for the IoM demo and then I was at
DebConf and then on vacation).

I'm afraid I think this branch takes quite the wrong approach and should
be reverted for reconsideration, preferably as soon as possible before
its behaviour becomes entrenched.

On Thu, Aug 22, 2013 at 10:29:49PM -0500, Ted Gould wrote:
> On Thu, 2013-08-22 at 22:05 +0000, Loïc Minier wrote:
> > I guess there are a number of different use cases:
> > a) package is in app store but we want to preinstall it
> > b) package is not in app store but we want to preinstall it
> >
> > and another subcase:
> > 1) package may be removed by end-user
> > 2) package may not be removed by end-user
> >
> >
> >
> > and what we want to decide is whether we're upgrading/reinstalling
> > packages and whether we do this on every boot.
>
> To be clear, we can't do it on boot because versions are per-user and so
> are hooks that need to be run by the click system. Also, at boot the
> user's home directory may not be decrypted.

Quite so.

> > Now some conclusions:
> > - it seems clear to me that for a), it doesn't matter whether we
> > update it or not as updates can be delivered from appstore, so we
> > could optionally skip this on boot
> > - it seems natural that things that are in appstore and hence can be
> > installed after the fact should be uninstallable, so that a) should
> > imply 1)
> > - it is obvious that in the case of b), we must update it on each boot
> > - it seems natural that in the case of b), we should not allow removal
> > -- otherwise lets just put it in the appstore
>
> I think that we can't simplify things like this with the per-user
> installation. There's no reason both users on a system would have the
> same version of Facebook whether it was pre-installed or not. It's a
> disk space issue, not an automatic update one. And, as a user, I could
> not want to update something.

There's really only one decision to take here, and it's whether a
preinstalled package should be upgraded after a system image update. My
take on this is that, if the user has installed a newer version, then
that should be preserved, but otherwise the preinstalled version should
win. The solution implemented in this branch has the unfortunate
property that an older version delivered by a system image update will
take precedence over a newer one installed by the user.

There is a straightforward way to implement this which does not involve
doing any work at boot time and not much at session init time, but does
involve some (mostly already planned) extensions to click:

 * change click to support multiple base unpack directories rather than
   just /opt/click.ubuntu.com/ (it's already architected to support
   this, with just a few missing pieces), with system configuration for
   the priority order

 * add a mechanism in click to register an app for all users in a
   parti...

[Sergio Schvezov (sergiusens)]

On Tue, Aug 27, 2013 at 7:37 AM, Colin Watson <email address hidden> wrote:

> I understand that the world couldn't stop while I was away :-), but it's
> unfortunate that this branch was merged when I was on holiday and didn't
> have the opportunity to comment on the fundamental design issues around
> preinstalled click packages, which is an item on my near-term to-do list
> to improve (it wasn't a priority for the IoM demo and then I was at
> DebConf and then on vacation).
>
> I'm afraid I think this branch takes quite the wrong approach and should
> be reverted for reconsideration, preferably as soon as possible before
> its behaviour becomes entrenched.
>

Reverting it now now would prevent people getting updates for the click
packages when image based upgrading, but I get your point after reading all
of this.

>
> On Thu, Aug 22, 2013 at 10:29:49PM -0500, Ted Gould wrote:
> > On Thu, 2013-08-22 at 22:05 +0000, Loïc Minier wrote:
> > > I guess there are a number of different use cases:
> > > a) package is in app store but we want to preinstall it
> > > b) package is not in app store but we want to preinstall it
> > >
> > > and another subcase:
> > > 1) package may be removed by end-user
> > > 2) package may not be removed by end-user
> > >
> > >
> > >
> > > and what we want to decide is whether we're upgrading/reinstalling
> > > packages and whether we do this on every boot.
> >
> > To be clear, we can't do it on boot because versions are per-user and so
> > are hooks that need to be run by the click system. Also, at boot the
> > user's home directory may not be decrypted.
>
> Quite so.
>

I'm doing everything at session init.

> > > Now some conclusions:
> > > - it seems clear to me that for a), it doesn't matter whether we
> > > update it or not as updates can be delivered from appstore, so we
> > > could optionally skip this on boot
> > > - it seems natural that things that are in appstore and hence can be
> > > installed after the fact should be uninstallable, so that a) should
> > > imply 1)
> > > - it is obvious that in the case of b), we must update it on each boot
> > > - it seems natural that in the case of b), we should not allow removal
> > > -- otherwise lets just put it in the appstore
> >
> > I think that we can't simplify things like this with the per-user
> > installation. There's no reason both users on a system would have the
> > same version of Facebook whether it was pre-installed or not. It's a
> > disk space issue, not an automatic update one. And, as a user, I could
> > not want to update something.
>
> There's really only one decision to take here, and it's whether a
> preinstalled package should be upgraded after a system image update. My
> take on this is that, if the user has installed a newer version, then
> that should be preserved, but otherwise the preinstalled version should
> win. The solution implemented in this branch has the unfortunate
> property that an older version delivered by a system image update will
> take precedence over a newer one installed by the user.
>

I am doing version checking of what's installed against what is provided
though.

> There is a straightforward way to implement th...

=== modified file 'debian/changelog'
--- debian/changelog	2013-08-21 22:50:21 +0000
+++ debian/changelog	2013-08-23 15:48:25 +0000
@@ -1,3 +1,9 @@
+ubuntu-touch-session (0.65) UNRELEASED; urgency=low
+
+  * Adding upstart session job that installs click packages for user.
+
+ -- Sergio Schvezov <[e-mail]>  Thu, 22 Aug 2013 11:30:13 -0300
+
 ubuntu-touch-session (0.64) saucy; urgency=low
 
   * Providing an override for application-click to use the desktop file hint.

=== added file 'startup-scripts/preinstall-click'
--- startup-scripts/preinstall-click	1970-01-01 00:00:00 +0000
+++ startup-scripts/preinstall-click	2013-08-23 15:48:25 +0000
@@ -0,0 +1,88 @@
+#!/usr/bin/python3
+# Copyright (C) 2013 Canonical Ltd.
+# Author: Sergio Schvezov <[e-mail]>
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 3 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import json
+import os
+import os.path
+import subprocess
+
+preinstalled_path = '/usr/share/preinstalled/click'
+customization_path = '/custom/preinstalled/click'
+
+
+def install_required(pkg, manifest):
+    pkg_manifest = json.loads(
+        subprocess.check_output(['click', 'info', pkg]).decode('utf-8'))
+    installed_pkg_version = ''
+    for item in manifest:
+        if item['name'] == pkg_manifest['name']:
+            installed_pkg_version = item['version']
+            break
+    if installed_pkg_version < pkg_manifest['version']:
+        return True
+    return False
+
+
+def install():
+    install_manifest = subprocess.check_output(
+        ['click', 'list', '--manifest'])
+    install_manifest = json.loads(install_manifest.decode('utf-8'))
+
+    for click_path in (preinstalled_path, customization_path):
+        click_pkg_path = []
+        try:
+            click_pkg = filter((lambda x: x.endswith('.click')),
+                               os.listdir(click_path))
+            click_pkg_path = [os.path.join(click_path, x) for x in click_pkg]
+        except OSError as e:
+            if e.errno == 2:
+                print('Directory %s does not exist' % click_path)
+                pass
+            else:
+                raise e
+        for pkg in click_pkg_path:
+            if install_required(pkg, install_manifest):
+                subprocess.call(['/usr/bin/pkcon', 'install-local', pkg])
+            else:
+                print('Installation of %s not required' % pkg)
+
+
+def run_required():
+    version_file = '/etc/ubuntu-build'
+    last_version_run = os.path.expanduser('~/.ubuntu-build')
+    if not os.path.exists(version_file):
+        print('Not on image based upgrade system, forcing install')
+        return True
+    with open(version_file, 'r') as f:
+        build = f.read().strip()
+    if not os.path.exists(last_version_run):
+        print('No previous run for %s found, installing' % build)
+        with open(last_version_run, 'w') as f:
+            f.write(build)
+        return True
+    with open(last_version_run, 'r') as f:
+        last_build = f.read().strip()
+    if build == last_build:
+        return False
+    else:
+        print('Previous run was for %s but we are now on %s' %
+              (last_version_run, build))
+        return True
+
+
+if __name__ == '__main__':
+    if run_required():
+        install()

=== added file 'upstart-session/click-install.conf'
--- upstart-session/click-install.conf	1970-01-01 00:00:00 +0000
+++ upstart-session/click-install.conf	2013-08-23 15:48:25 +0000
@@ -0,0 +1,8 @@
+description "Click app installer"
+author "Sergio Schvezov <[e-mail]>"
+
+console log
+
+start on started dbus
+
+exec preinstall-click