Merge lp:~robru/phablet-tools/make-useful-again into lp:phablet-tools

[Robert Bruce Park]

Make citrain tool able to install silo packages with deps & without extras.

Hopefully we can restore the usefulness of citrain tool in the face of overlay PPA and "new deps in distro" challenges.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:347
http://jenkins.qa.ubuntu.com/job/phablet-tools-ci/453/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-amd64-ci/30/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-armhf-ci/30/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-i386-ci/30/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/phablet-tools-ci/453/rebuild

review: Needs Fixing (continuous-integration)

[lp:~robru/phablet-tools/make-useful-again]

348. By Robert Bruce Park on 2015-10-14

Bring back autoremove.

[Robert Bruce Park (robru)]

Ok, I tested this on silo 14 (webbrowser-app), and got the following results:

+ adb shell SUDO_ASKPASS=/tmp/askpass.sh sudo -A apt-get install ubuntu-touch qtdeclarative5-ubuntu-ui-extras-browser-plugin qtdeclarative5-ubuntu-web-plugin webapp-container webbrowser-app --yes
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  qtdeclarative5-ubuntu-web-plugin-assets
Use 'apt-get autoremove' to remove it.
The following extra packages will be installed:
  gstreamer1.0-plugins-ugly-amr indicator-transfer
  indicator-transfer-download-manager libindicator-transfer0
  libopencore-amrnb0 libopencore-amrwb0 liboxideqt-qmlplugin liboxideqtcore0
  liboxideqtquick0 oxideqt-codecs-extra
The following NEW packages will be installed:
  gstreamer1.0-plugins-ugly-amr indicator-transfer-download-manager
  libindicator-transfer0 libopencore-amrnb0 libopencore-amrwb0
The following packages will be upgraded:
  indicator-transfer liboxideqt-qmlplugin liboxideqtcore0 liboxideqtquick0
  oxideqt-codecs-extra qtdeclarative5-ubuntu-ui-extras-browser-plugin
  qtdeclarative5-ubuntu-web-plugin ubuntu-touch webapp-container
  webbrowser-app
10 upgraded, 5 newly installed, 0 to remove and 278 not upgraded.
Need to get 24.4 MB of archives.
After this operation, 909 kB of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports/ vivid/universe libopencore-amrnb0 armhf 0.1.3-2.1 [81.8 kB]
Get:2 http://ppa.launchpad.net/ci-train-ppa-service/stable-snapshot/ubuntu/ vivid/main liboxideqtcore0 armhf 1.9.5-0ubuntu0.15.04.1 [22.1 MB]
Get:3 http://ports.ubuntu.com/ubuntu-ports/ vivid/universe libopencore-amrwb0 armhf 0.1.3-2.1 [41.5 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports/ vivid-updates/main liboxideqt-qmlplugin armhf 1.9.5-0ubuntu0.15.04.1 [168 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports/ vivid-updates/main liboxideqtquick0 armhf 1.9.5-0ubuntu0.15.04.1 [233 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports/ vivid-updates/main oxideqt-codecs-extra armhf 1.9.5-0ubuntu0.15.04.1 [769 kB]
Get:7 http://ports.ubuntu.com/ubuntu-ports/ vivid/universe gstreamer1.0-plugins-ugly-amr armhf 1.4.5-1ubuntu2 [16.5 kB]
Get:8 http://ppa.launchpad.net/ci-train-ppa-service/landing-014/ubuntu/ vivid/main qtdeclarative5-ubuntu-ui-extras-browser-plugin armhf 0.23+15.04.20151013.1-0ubuntu1 [10.7 kB]
Get:9 http://ppa.launchpad.net/ci-train-ppa-service/landing-014/ubuntu/ vivid/main webapp-container armhf 0.23+15.04.20151013.1-0ubuntu1 [142 kB]
Get:10 http://ppa.launchpad.net/ci-train-ppa-service/landing-014/ubuntu/ vivid/main webbrowser-app armhf 0.23+15.04.20151013.1-0ubuntu1 [722 kB]
Get:11 http://ppa.launchpad.net/ci-train-ppa-service/landing-014/ubuntu/ vivid/main qtdeclarative5-ubuntu-web-plugin armhf 0.23+15.04.20151013.1-0ubuntu1 [17.5 kB]
Get:12 http://ppa.launchpad.net/ci-train-ppa-service/stable-snapshot/ubuntu/ vivid/main libindicator-transfer0 armhf 0.2+15.04.20150929.is.0.2+15.10.20150819-0ubuntu1 [43.2 kB]
Get:13 http://ppa.launchpad.net/ci-train-ppa-service/stable-snapshot/ubuntu/ vivid/main indicator-transfer armhf 0.2+15.04.20150929.is.0.2+1...

[lp:~robru/phablet-tools/make-useful-again]

349. By Robert Bruce Park on 2015-10-14

Stop forcing ubuntu-touch.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:348
http://jenkins.qa.ubuntu.com/job/phablet-tools-ci/454/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-amd64-ci/31/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-armhf-ci/31/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-i386-ci/31/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/phablet-tools-ci/454/rebuild

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:349
http://jenkins.qa.ubuntu.com/job/phablet-tools-ci/455/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-amd64-ci/32/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-armhf-ci/32/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-i386-ci/32/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/phablet-tools-ci/455/rebuild

review: Needs Fixing (continuous-integration)

[lp:~robru/phablet-tools/make-useful-again]

350. By Robert Bruce Park on 2015-10-14

Appease static code checker.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:350
http://jenkins.qa.ubuntu.com/job/phablet-tools-ci/456/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-amd64-ci/33
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-armhf-ci/33
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-i386-ci/33

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/phablet-tools-ci/456/rebuild

review: Approve (continuous-integration)

[Olivier Tilloy (osomon)]

> And, liboxideqt-qmlplugin, liboxideqtquick0, oxideqt-codecs-extra,
> gstreamer1.0-plugins-ugly-amr were installed from vivid archive, but
> it's not clear to me if those are needed by the silo or not. oxide is
> related to webbrowser-app.

Silo 14 doesn’t require an oxide version bump, so the version that was already installed on the device you tested on was sufficient. Therefore I think that the upgrade shouldn’t have happened, if the tool was meant to install only packages from the silo.

[Robert Bruce Park (robru)]

Ugh.

Thanks for clarifying.

[Olivier Tilloy (osomon)]

> The autoremove is seemingly correct in the sense that it didn't
> uninstall the entire world, but I'm not sure if this removal is
> correct or not.

The removal is expected, that package doesn’t exist anymore (it was removed in the ota7 dev cycle).

[lp:~robru/phablet-tools/make-useful-again]

351. By Robert Bruce Park on 2015-10-14

Use dist-upgrade again with clever pinning.

[Robert Bruce Park (robru)]

Ok, I think I've got this, here's the results on silo 56:

The following packages will be upgraded:
  libaccounts-qt5-1 libonline-accounts-client1 libsignon-extension1
  libsignon-plugins-common1 libsignon-qt5-1 qml-module-ubuntu-onlineaccounts
  qml-module-ubuntu-onlineaccounts-client qtdeclarative5-accounts-plugin
  qtdeclarative5-online-accounts-client0.1 signon-plugin-oauth2
  signon-plugin-password signond
12 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 407 kB of archives.
After this operation, 2,048 B of additional disk space will be used.
Get:1 http://ppa.launchpad.net/ci-train-ppa-service/landing-056/ubuntu/ vivid/main libaccounts-qt5-1 armhf 1.14+15.04.20150930-0ubuntu1 [28.7 kB]
Get:2 http://ppa.launchpad.net/ci-train-ppa-service/landing-056/ubuntu/ vivid/main libonline-accounts-client1 armhf 0.7+15.04.20151009.2-0ubuntu1 [13.7 kB]
Get:3 http://ppa.launchpad.net/ci-train-ppa-service/landing-056/ubuntu/ vivid/main libsignon-extension1 armhf 8.58+15.04.20151001-0ubuntu1 [18.8 kB]
Get:4 http://ppa.launchpad.net/ci-train-ppa-service/landing-056/ubuntu/ vivid/main libsignon-plugins-common1 armhf 8.58+15.04.20151001-0ubuntu1 [16.2 kB]
Get:5 http://ppa.launchpad.net/ci-train-ppa-service/landing-056/ubuntu/ vivid/main libsignon-qt5-1 armhf 8.58+15.04.20151001-0ubuntu1 [54.3 kB]
Get:6 http://ppa.launchpad.net/ci-train-ppa-service/landing-056/ubuntu/ vivid/main qml-module-ubuntu-onlineaccounts armhf 0.6+15.04.20151007-0ubuntu1 [47.7 kB]
Get:7 http://ppa.launchpad.net/ci-train-ppa-service/landing-056/ubuntu/ vivid/main qml-module-ubuntu-onlineaccounts-client armhf 0.7+15.04.20151009.2-0ubuntu1 [6,988 B]
Get:8 http://ppa.launchpad.net/ci-train-ppa-service/landing-056/ubuntu/ vivid/main qtdeclarative5-accounts-plugin armhf 0.6+15.04.20151007-0ubuntu1 [1,390 B]
Get:9 http://ppa.launchpad.net/ci-train-ppa-service/landing-056/ubuntu/ vivid/main qtdeclarative5-online-accounts-client0.1 armhf 0.7+15.04.20151009.2-0ubuntu1 [1,462 B]
Get:10 http://ppa.launchpad.net/ci-train-ppa-service/landing-056/ubuntu/ vivid/main signon-plugin-oauth2 armhf 0.23+15.04.20151001.1-0ubuntu1 [52.4 kB]
Get:11 http://ppa.launchpad.net/ci-train-ppa-service/landing-056/ubuntu/ vivid/main signond armhf 8.58+15.04.20151001-0ubuntu1 [153 kB]
Get:12 http://ppa.launchpad.net/ci-train-ppa-service/landing-056/ubuntu/ vivid/main signon-plugin-password armhf 8.58+15.04.20151001-0ubuntu1 [11.9 kB]
Fetched 407 kB in 4s (99.6 kB/s)

So it's only installed packages from silo 56, nothing from vivid and nothing from stable snapshot ppa.

$ adb shell apt-cache policy signond
signond:
  Installed: 8.58+15.04.20151001-0ubuntu1
  Candidate: 8.58+15.04.20151001-0ubuntu1
  Version table:
 *** 8.58+15.04.20151001-0ubuntu1 0
       1100 http://ppa.launchpad.net/ci-train-ppa-service/landing-056/ubuntu/ vivid/main armhf Packages
        100 /var/lib/dpkg/status
     8.57+15.04.20150616-0ubuntu1 0
         50 http://ppa.launchpad.net/ci-train-ppa-service/stable-snapshot/ubuntu/ vivid/main armhf Packages
     8.57+15.04.20150204.1-0ubuntu1 0
         50 http://ports.ubuntu.com/ubuntu-ports/ vivid/main armhf Packages

this shows the correct silo pinned at 1100...

review: Approve

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:351
http://jenkins.qa.ubuntu.com/job/phablet-tools-ci/457/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-amd64-ci/34
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-armhf-ci/34
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-i386-ci/34

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/phablet-tools-ci/457/rebuild

review: Approve (continuous-integration)

[lp:~robru/phablet-tools/make-useful-again]

352. By Robert Bruce Park on 2015-10-14

Drop unused var.

353. By Robert Bruce Park on 2015-10-14

Appease shellcheck & check-bashisms simultaneously.

354. By Robert Bruce Park on 2015-10-14

Simplify lsb-release parsing.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:353
http://jenkins.qa.ubuntu.com/job/phablet-tools-ci/458/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-amd64-ci/35
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-armhf-ci/35
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-i386-ci/35

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/phablet-tools-ci/458/rebuild

review: Approve (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:354
http://jenkins.qa.ubuntu.com/job/phablet-tools-ci/459/
Executed test runs:
    None: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-amd64-ci/36/console
    None: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-armhf-ci/36/console
    None: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-i386-ci/36/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/phablet-tools-ci/459/rebuild

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:354
http://jenkins.qa.ubuntu.com/job/phablet-tools-ci/460/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-amd64-ci/37
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-armhf-ci/37
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-vivid-i386-ci/37

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/phablet-tools-ci/460/rebuild

review: Approve (continuous-integration)

[Robert Bruce Park (robru)]

Saviq reports this is working for him also.

[lp:~robru/phablet-tools/make-useful-again]

355. By Robert Bruce Park on 2015-10-14

Use --force-yes for maximum yesness.

=== modified file 'citrain'
--- citrain	2015-09-25 18:10:53 +0000
+++ citrain	2015-10-14 22:35:53 +0000
@@ -8,7 +8,7 @@
 # MERCHANTABILITY, SATISFACTORY QUALITY or FITNESS FOR A PARTICULAR
 # PURPOSE.  See the applicable version of the GNU General Public
 # License for more details.
-#.
+#
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
@@ -16,7 +16,7 @@
 
 usage () {
 cat <<EOF
-usage: $0 COMMAND SILO-NUMBER [DEVICE-PASSWORD] [DISTRIBUTION]
+usage: $0 COMMAND SILO-NUMBER [DEVICE-PASSWORD]
 $1
 
 COMMANDS:
@@ -26,10 +26,6 @@
     device-upgrade      Upgrades your connected device with packages from the silo.
     host-purge          Uses ppa-purge to uninstall the silo contents from the host machine.
     device-purge        Not implemented.
-
-DISTRIBUTION: (optional)
-    ubuntu
-    ubuntu-rtm
 EOF
 exit 1
 }
@@ -37,8 +33,9 @@
 [ -x /usr/bin/sudo               ] || { echo "Please install 'sudo'"; exit 1; }
 [ -x /usr/bin/add-apt-repository ] || { echo "Please install 'software-properties-common'"; exit 1; }
 
-if [ -f "$(dirname $0)/shell-adb-common.sh" ]; then
-    . "$(dirname $0)/shell-adb-common.sh"
+DIR=$(dirname "$0")
+if [ -f "$DIR/shell-adb-common.sh" ]; then
+    . "$DIR/shell-adb-common.sh"
 else
     . "/usr/share/phabletutils/shell-adb-common.sh"
 fi
@@ -60,7 +57,7 @@
 case "$COMMAND" in
     host-upgrade)
         set -x
-        sudo add-apt-repository $PPA/$SILO
+        sudo add-apt-repository "$PPA/$SILO"
         sudo apt-get update -qq
         sudo apt-get dist-upgrade --yes
         ;;
@@ -72,36 +69,29 @@
         [ $# -gt 0 ] && shift || usage "Missing DEVICE-PASSWORD"
 
         set -x
+        SERIES=$(adb shell lsb_release -cs | tr -d '\r')
+        DISTRO=$(adb shell lsb_release -is | tr -d '\r' | tr '[:upper:]' '[:lower:]' | tr ' ' '-')
+
         adb shell egrep ^deb /etc/apt/sources.list.d/\*.list
-        DISTRO="$1"
-        if [ -z $DISTRO ]; then
-            # In case no distro specified, guess
-            if adb shell system-image-cli -i | grep -q ubuntu-rtm; then
-                DISTRO="ubuntu-rtm"
-            else
-                DISTRO="ubuntu"
-            fi
-        fi
         adb shell "echo -e '#\x21/bin/sh\necho $PASSWORD' >/tmp/askpass.sh"
         adb shell chmod +x /tmp/askpass.sh
         adb shell SUDO_ASKPASS=/tmp/askpass.sh sudo -A touch /userdata/.adb_onlock
         adb shell SUDO_ASKPASS=/tmp/askpass.sh sudo -A mount -o remount,rw /
-        adb shell SUDO_ASKPASS=/tmp/askpass.sh sudo -A add-apt-repository -y $PPA/$DISTRO/$SILO
+        adb shell SUDO_ASKPASS=/tmp/askpass.sh sudo -A add-apt-repository -y "$PPA/$DISTRO/$SILO"
 
         adb shell "test -e /usr/sbin/policy-rc.d && cp /usr/sbin/policy-rc.d /tmp/policy-rc.d"
         adb shell "echo 'exit 101' | SUDO_ASKPASS=/tmp/askpass.sh sudo -A tee /usr/sbin/policy-rc.d"
         adb shell SUDO_ASKPASS=/tmp/askpass.sh sudo -A chmod +x /usr/sbin/policy-rc.d
-        adb shell "echo -e 'Package: *\nPin: release o=LP-PPA-ci-train-ppa-service-$SILO\nPin-Priority: 1100' | SUDO_ASKPASS=/tmp/askpass.sh sudo -A tee /etc/apt/preferences.d/silo-$SILO.pref"
-        adb shell SUDO_ASKPASS=/tmp/askpass.sh sudo -A apt-get -o Dir::Etc::SourceList=/dev/null update
+        adb shell "echo -e 'Package: *\nPin: release o=*$SILO*\nPin-Priority: 1100\n\nPackage: *\nPin: release a=$SERIES*\nPin-Priority: 50' | SUDO_ASKPASS=/tmp/askpass.sh sudo -A tee /etc/apt/preferences.d/extra-ppas.pref"
+        adb shell SUDO_ASKPASS=/tmp/askpass.sh sudo -A apt-get update
 
         # We need to put lxc-android-config on hold here as it's the only
         # package which must be upgrade by hand through the recovery. See
         # https://wiki.ubuntu.com/Touch/Testing/lxc-android-config for details.
         adb shell SUDO_ASKPASS=/tmp/askpass.sh sudo -A apt-mark hold lxc-android-config
 
-        adb shell SUDO_ASKPASS=/tmp/askpass.sh sudo -A apt-get dist-upgrade --yes
-        adb shell SUDO_ASKPASS=/tmp/askpass.sh sudo -A apt-get update
-        adb shell SUDO_ASKPASS=/tmp/askpass.sh sudo -A apt-get autoremove --yes
+        adb shell SUDO_ASKPASS=/tmp/askpass.sh sudo -A apt-get dist-upgrade --yes --force-yes
+        adb shell SUDO_ASKPASS=/tmp/askpass.sh sudo -A apt-get autoremove --yes --force-yes
 
         adb shell "test ! -e /tmp/policy-rc.d && SUDO_ASKPASS=/tmp/askpass.sh sudo -A rm /usr/sbin/policy-rc.d"
         adb shell "test -e /tmp/policy-rc.d && SUDO_ASKPASS=/tmp/askpass.sh sudo -A mv /tmp/policy-rc.d /usr/sbin/policy-rc.d"
@@ -113,7 +103,7 @@
 
     host-purge)
         set -x
-        sudo ppa-purge $PPA/$SILO
+        sudo ppa-purge "$PPA/$SILO"
         ;;
 
     device-purge)