Merge lp:~craig.magina/charms/trusty/ilp32-rebase/trunk into lp:~dannf/charms/trusty/ilp32-rebase/trunk

[Craig Magina]

[dann frazier (dannf)]

I think it makes sense for you to own this code now. If you agree, I can just mark my branch is abandoned. But, I'm happy to provide review comments - I'll try to add them inline.

One overall comment - in general I find it easier to review things when there is 1 commit per change. I think you've done a good job at splitting out the individual functional changes in the bullets, but with one commit, it isn't always obvious which bullet a specific line change is addressing.

[lp:~craig.magina/charms/trusty/ilp32-rebase/trunk]

4. By Craig Magina on 2015-03-05

Fixed an issue with the package versioning that was making the new packages older then the ones they were supposed to replace

[Craig Magina (craig.magina)]

I'll keep that in mind for all future work, thanks for the tip. I am fine taking ownership of it from here on out. Another set of eyes is always useful with code, so I would appreciate your review.

[dann frazier (dannf)]

Huh.. I did do an inline review.. but now I'm not seeing those
comments on the MP.. wtf?

[dann frazier (dannf)]

Ah - ok, your last push hides those review comments since it changes the diff. To see those comments, look under "Preview Diff" below, and select "r3 into r2 on 2015-03-02"

[lp:~craig.magina/charms/trusty/ilp32-rebase/trunk]

5. By Craig Magina on 2015-03-16

Added support for specifying the release to use as the base kernel (updates|proposed)

[lp:~craig.magina/charms/trusty/ilp32-rebase/trunk]

6. By Craig Magina on 2015-03-16

Moved the apt clean back to its original location as suggested by dann

7. By Craig Magina on 2015-03-16

Revert manual debian changelog entry generation to debain/rules startnewrelease target as suggested by dann

8. By Craig Magina on 2015-03-17

Single quotes are not expanded in bash, need double quotes to cause $RELEASE to be expanded properly

9. By Craig Magina on 2015-04-01

Fix for the package version startnewrelease sets, since we want to build on the current package version, not incremement to the next version.

10. By Craig Magina on 2015-04-14

Removed the script generation from do-ilp32-rebase
Added configuration output to do-ilp32-rebase
Created script to download the results from the juju unit
Created a script to upload the package to the ilp32 ppa

11. By Craig Magina on 2015-04-15

Removed un-used variable

Unmerged revisions

11. By Craig Magina on 2015-04-15

Removed un-used variable

10. By Craig Magina on 2015-04-14

Removed the script generation from do-ilp32-rebase
Added configuration output to do-ilp32-rebase
Created script to download the results from the juju unit
Created a script to upload the package to the ilp32 ppa

9. By Craig Magina on 2015-04-01

Fix for the package version startnewrelease sets, since we want to build on the current package version, not incremement to the next version.

8. By Craig Magina on 2015-03-17

Single quotes are not expanded in bash, need double quotes to cause $RELEASE to be expanded properly

7. By Craig Magina on 2015-03-16

Revert manual debian changelog entry generation to debain/rules startnewrelease target as suggested by dann

6. By Craig Magina on 2015-03-16

Moved the apt clean back to its original location as suggested by dann

5. By Craig Magina on 2015-03-16

Added support for specifying the release to use as the base kernel (updates|proposed)

4. By Craig Magina on 2015-03-05

Fixed an issue with the package versioning that was making the new packages older then the ones they were supposed to replace

3. By Craig Magina on 2015-03-02

Config changes:
  * The URI for the ILP32 patches git tree is setable
  * The ILP32 start and end pointers are setable
The linux-meta package is now built to match the kernel abi
do-ilp32-rebase no longer needs to be run under root, running it as root will actually break on the linux-meta package build
Scripts are generate at the end of each build phase (linux and linux-meta):
  * Downloads all parts required for upload
  * Signs package sources for upload
  * Uploads package sources to the ilp32-proposed ppa
  * Suggests an optimized means of downloading the git trees to be saved for history

=== modified file 'README'
--- README	2014-12-17 01:14:12 +0000
+++ README	2015-04-15 13:14:49 +0000
@@ -1,19 +1,23 @@
 # Overview
 
-This charm is used to help maintain the ilp32-enabled arm64 kernel in ppa:ce-hyperscale/ilp32. As the Ubuntu 14.04 kernel is updated, the ilp32 patches need to be rebased and packaged. On deployment, this charm will install all the necessary packages to do the job and seed content from various git locations. A user can then login and run a single command that will:
+This charm is used to help maintain the ilp32-enabled arm64 kernel in ppa:ce-hyperscale/ilp32-proposed. As the Ubuntu 14.04 kernel is updated, the ilp32 patches need to be rebased and packaged. On deployment, this charm will install all the necessary packages to do the job and seed content from various git locations. A user can then login and run a single command that will:
  1. Query launchpad to find the version of the current kernel package in trusty-proposed
  2. Pull updates for all relevant git branches
  3. Create a new git branch based on the tag of the kernel currently in trusty-proposed
  4. Rebase ILP32 patches from a reference tree onto this tag
  5. Prepare a source package suitable for upload to a test PPA
+ 6. Repeat for the linux-meta package
 
 # Usage
 
 Step by step instructions on using the charm:
 
     juju deploy ilp32-rebase
+    juju set user-email="[e-mail]" user-fullname="Example User"
     juju ssh ilp32-rebase/0
-    sudo /usr/bin/do-ilp32-rebase
+    /usr/bin/do-ilp32-rebase
+
+The config allows a user to set the URI where the ILP32 patches are located and pointers to the start and end of the patch set.
     
 ## Scale out Usage
 
 
=== modified file 'config.yaml'
--- config.yaml	2014-12-17 01:14:12 +0000
+++ config.yaml	2015-04-15 13:14:49 +0000
@@ -7,3 +7,19 @@
     type: string
     default: "Example User"
     description: "Full name to use in commits and changelogs"
+  release:
+    type: string
+    default: "proposed"
+    description: "The trusty release to use as the base for the ilp32 kernel"
+  ilp32-ref-tree:
+    type: string
+    default: "git://kernel.ubuntu.com/dannf/trusty-xgene.git"
+    description: "GIT reference tree containing the ilp32 support patches"
+  ilp32-start-ref:
+    type: string
+    default: "ilp32/ilp32-start"
+    description: "GIT reference to the first ilp32 support commit"
+  ilp32-end-ref:
+    type: string
+    default: "ilp32/ilp32"
+    description: "GIT reference to the end ilp32 support commit"
 
=== added directory 'files'
=== renamed file 'do-ilp32-rebase' => 'files/do-ilp32-rebase'
--- do-ilp32-rebase	2014-12-17 01:14:12 +0000
+++ files/do-ilp32-rebase	2015-04-15 13:14:49 +0000
@@ -12,15 +12,25 @@
 export GIT_COMMITTER_EMAIL="$USER_EMAIL"
 export GIT_COMMITTER_NAME="$USER_FULLNAME"
 
-cd $GIT_TREE
+ILP32_REBASE_DIR=`mktemp -d /tmp/ilp32-rebase-XXXX`
+
+cd $ILP32_REBASE_DIR
+
+# Build linux
+# Fetch the ubuntu and reference trees.
+git clone /usr/src/ilp32-rebase/linux
+cd linux
+
+git remote add ilp32 $ILP32_REF_TREE
+git fetch ilp32
+git remote add trusty git://kernel.ubuntu.com/ubuntu/ubuntu-trusty.git
 git fetch trusty
-git fetch ilp32
 
 prover=$(get-src-package-version \
-    --archive 'http://ports.ubuntu.com/ubuntu-ports' \
-    --release 'trusty-proposed' \
-    --section 'main' \
-      linux)
+    --archive "http://ports.ubuntu.com/ubuntu-ports" \
+    --release "trusty-$RELEASE" \
+    --section "main" \
+    linux)
 
 tag="Ubuntu-${prover}"
 ilp32_branch="${prover}+ilp32"
@@ -29,7 +39,7 @@
 git reset --hard HEAD
 # get off of ilp32 branch if we are on it
 # (git won't let us delete the branch we're sitting on)
-git checkout origin/master
+git checkout master
 git branch -D "$ilp32_branch" || /bin/true
 git branch "$ilp32_branch" "$tag"
 git checkout "$ilp32_branch"
@@ -40,30 +50,116 @@
 ./debian/scripts/misc/getabis ${upstream_ver} ${old_package_rev}
 
 # clean causes debian/control to be generated, needed by startnewrelease
-make -f debian/rules clean
-make -f debian/rules startnewrelease
+fakeroot make -f debian/rules clean
+fakeroot make -f debian/rules startnewrelease
+
 # This adds a middle ABI number and appends '+ilp32.1'
-sed -i -r '0,/(.*\(.+)-(.+)\.(.+)\)(.*)/s//\1\-\2\.\2\.\3\+ilp32\.1\)\4/' \
+sed -i -r "0,/(.*\(.+)-(.+)\.(.+)\)(.*)/s//\1\-${old_package_rev%.*}\.${old_package_rev#*.}\.${old_package_rev#*.}\+ilp32\.1\)\4/" \
     debian.master/changelog
 git add debian.master/changelog
-git commit -s -m 'UBUNTU: Start new release
-
-Ignore: yes'
-
-# FIXME: the start/end references should be user configurable
-git log --reverse ilp32/ilp32-start..ilp32/ilp32 | grep ^commit | cut -d' ' -f2 | \
+git commit -s -F debian/commit-templates/newrelease
+
+git log --reverse $ILP32_START_REF..$ILP32_END_REF | grep ^commit | cut -d' ' -f2 | \
     while read c; do
-	git cherry-pick $c
-    done
+    git cherry-pick $c
+done
 
-make -f debian/rules insertchanges
+fakeroot make -f debian/rules insertchanges
 dch -c debian.master/changelog -r ""
 git add debian.master/changelog
 newver="$(dpkg-parsechangelog --show-field Version -ldebian.master/changelog)"
 git commit -s -m "UBUNTU: Ubuntu-$newver"
+
+cp /usr/src/ilp32-rebase/linux_${upstream_ver}.orig.tar.gz $ILP32_REBASE_DIR
 rm -rf ../linux-${upstream_ver}
 git clone . ../linux-${upstream_ver}
 cd ../linux-${upstream_ver}
 rm -rf .git
-make -f debian/rules clean
+fakeroot make -f debian/rules clean
 dpkg-buildpackage -rfakeroot -uc -us -S
+
+# Generate ilp32-rebase-results file
+cd $ILP32_REBASE_DIR
+changes=`ls -1 linux_${upstream_ver}-*+ilp32.1_source.changes`
+dsc=`ls -1 linux_${upstream_ver}-*+ilp32.1.dsc`
+diff=`ls -1 linux_${upstream_ver}-*+ilp32.1.diff.gz`
+
+cat > $HOME/ilp32-rebase-results <<EOF
+JUJU_UNIT_NAME=$JUJU_UNIT_NAME
+ILP32_REBASE_DIR=$ILP32_REBASE_DIR
+LINUX_CHANGES=$changes
+LINUX_DSC=$dsc
+LINUX_DIFF=$diff
+LINUX_GIT=$ILP32_REBASE_DIR/linux
+EOF
+# End of linux build
+
+# Build linux-meta
+cd $ILP32_REBASE_DIR
+
+git clone /usr/src/ilp32-rebase/linux-meta
+cd linux-meta
+
+prover=$(get-src-package-version \
+    --archive "http://ports.ubuntu.com/ubuntu-ports" \
+    --release "trusty-$RELEASE" \
+    --section "main" \
+    linux-meta)
+
+tag="Ubuntu-${prover}"
+ilp32_branch="${prover}+ilp32"
+
+git clean -x -f -d .
+git reset --hard HEAD
+# get off of ilp32 branch if we are on it
+# (git won't let us delete the branch we're sitting on)
+git checkout master
+git branch -D "$ilp32_branch" || /bin/true
+git branch "$ilp32_branch" "$tag"
+git checkout "$ilp32_branch"
+
+linux_upstream_ver="${upstream_ver}"
+linux_package_rev="${old_package_rev}"
+
+oldver="$(dpkg-parsechangelog --show-field Version -lmeta-source/debian/changelog)"
+upstream_ver="${oldver%.*.*}"
+old_package_rev="${oldver#${upstream_ver}.}"
+
+now="$(date -R)"
+cat > meta-source/debian/changelog.new <<EOF
+linux-meta (${upstream_ver}.${linux_package_rev}.${old_package_rev#*.}+ilp32.1) UNRELEASED; urgency=low
+
+  * linux ABI ${upstream_ver}-${linux_package_rev}
+
+ -- $DEBFULLNAME <$DEBEMAIL>  $now
+EOF
+cat meta-source/debian/changelog >> meta-source/debian/changelog.new
+mv meta-source/debian/changelog.new meta-source/debian/changelog
+dch -c meta-source/debian/changelog -r ""
+git add meta-source/debian/changelog
+
+# Update the KERNEL_VERSION and KERNEL_ABI in the debian/rules file
+sed "/KERNEL_ABI_VERSION=.*/i \
+KERNEL_VERSION=${linux_upstream_ver}\n\
+KERNEL_ABI=${linux_package_rev}" -i meta-source/debian/rules
+git add meta-source/debian/rules
+git commit -s -m "UBUNTU: Ubuntu-${upstream_ver}.${linux_package_rev}.${old_package_rev#*.}+ilp32.1"
+
+make
+
+# Append linux-meta to il32-rebase-results config
+changes=`ls -1 linux-meta_*source.changes`
+dsc=`ls -1 linux-meta_*.dsc`
+tarball=`ls -1 linux-meta_*.tar.gz`
+
+echo >> $HOME/ilp32-rebase-results <<EOF
+LINUX_META_CHANGES=$changes
+LINUX_META_DSC=$dsc
+LINUX_META_TARBALL=$tarball
+LINUX_META_GIT=$ILP32_REBASE_DIR/linux-meta
+EOF
+# End of linux-meta build
+
+echo "To retrieve results, download the ilp32-rebase-results file and run the get-results script"
+echo "juju scp $JUJU_UNIT_NAME:$HOME/ilp32-rebase-results ."
+echo "get-results"
 
=== renamed file 'get-src-package-version' => 'files/get-src-package-version'
=== modified file 'gen-config'
--- gen-config	2014-12-17 01:14:12 +0000
+++ gen-config	2015-04-15 13:14:49 +0000
@@ -3,15 +3,23 @@
 set -e
 set -x
 
-git_tree="$CHARM_DIR/linux"
+juju_unit_name="$JUJU_UNIT_NAME"
 user_email="$(config-get user-email)"
 user_fullname="$(config-get user-fullname)"
+release="$(config-get release)"
+ilp32_ref_tree="$(config-get ilp32-ref-tree)"
+ilp32_start_ref="$(config-get ilp32-start-ref)"
+ilp32_end_ref="$(config-get ilp32-end-ref)"
 
 mkdir -p /etc/ilp32-rebase
 cat > /etc/ilp32-rebase/config.new <<EOF
-GIT_TREE="$git_tree"
+JUJU_UNIT_NAME="$juju_unit_name"
 USER_EMAIL="$user_email"
 USER_FULLNAME="$user_fullname"
+RELEASE="$release"
+ILP32_REF_TREE="$ilp32_ref_tree"
+ILP32_START_REF="$ilp32_start_ref"
+ILP32_END_REF="$ilp32_end_ref"
 EOF
 
 # Atomically replace to avoid using half-written configs
 
=== added file 'get-results'
--- get-results	1970-01-01 00:00:00 +0000
+++ get-results	2015-04-15 13:14:49 +0000
@@ -0,0 +1,17 @@
+#!/bin/sh
+#
+# Downloads the results generated by the do-ilp32-rebase script from the juju unit.
+#
+
+source ilp32-rebase-results
+
+ILP32_REBASE_IP=$(juju-deployer -f ilp32-rebase)
+
+juju scp $JUJU_UNIT_NAME:$ILP32_REBASE_DIR/$LINUX_CHANGES
+juju scp $JUJU_UNIT_NAME:$ILP32_REBASE_DIR/$LINUX_DSC
+juju scp $JUJU_UNIT_NAME:$ILP32_REBASE_DIR/$LINUX_DIFF
+juju scp $JUJU_UNIT_NAME:$ILP32_REBASE_DIR/$LINUX_META_CHANGES
+juju scp $JUJU_UNIT_NAME:$ILP32_REBASE_DIR/$LINUX_DSC
+
+git clone --single-branch ssh://$ILP32_REBASE_IP/$LINUX_GIT linux
+git clone --single-branch ssh://$ILP32_REBASE_IP/$LINUX_META_GIT linux-meta
 
=== modified file 'hooks/install'
--- hooks/install	2014-12-17 01:14:12 +0000
+++ hooks/install	2015-04-15 13:14:49 +0000
@@ -5,12 +5,17 @@
 
 apt-get update
 apt-get build-dep -y linux
+apt-get install -y devscripts dpkg-dev fakeroot git python3-debian
+
+mkdir -m 0755 -p /usr/src/ilp32-rebase
+cd /usr/src/ilp32-rebase
+
 # For the upstream tarball, since this is a non-native package
 apt-get source --download-only linux
-apt-get install -y devscripts dpkg-dev fakeroot git python3-debian
 apt-get clean
 
-rm -rf linux # for indempotency (are install hooks retriable?)
+# for indempotency (are install hooks retriable?)
+rm -rf linux
 
 #
 # HACK
@@ -25,19 +30,13 @@
 #git clone git://kernel.ubuntu.com/ubuntu/ubuntu-trusty.git linux
 git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
 
-# Fetch the ubuntu and reference trees. The do-ilp32-rebase script will
-# fetch new changes, this is just to seed it.
-cd linux
-git remote add ilp32 git://kernel.ubuntu.com/dannf/trusty-xgene.git
-git fetch ilp32
-git remote add trusty git://kernel.ubuntu.com/ubuntu/ubuntu-trusty.git
-git fetch trusty
-cd -
+# Clone the linux-meta sources
+git clone git://kernel.ubuntu.com/ubuntu/ubuntu-trusty-meta.git linux-meta
 
 # do-ilp32-rebase currently runs outside of juju context, so it can't
 # get-config charm settings itself. Use gen-config to pass those on.
 $CHARM_DIR/gen-config
 
 # These should be replaced by juju actions when they exist
-install -m 755 get-src-package-version /usr/bin
-install -m 755 do-ilp32-rebase /usr/bin
+install -m 755 $CHARM_DIR/files/get-src-package-version /usr/bin
+install -m 755 $CHARM_DIR/files/do-ilp32-rebase /usr/bin
 
=== added file 'upload-results'
--- upload-results	1970-01-01 00:00:00 +0000
+++ upload-results	2015-04-15 13:14:49 +0000
@@ -0,0 +1,20 @@
+#!/bin/sh
+#
+# Sign and upload linux and linux-meta to the ilp32 ppa
+# By default, it will upload to the proposed ppa unless
+# "updates" is passed in as the first argument.
+#
+
+source ilp32-rebase-results
+
+debsign $LINUX_CHANGES
+debsign $LINUX_META_CHANGES
+
+if [ "$1" = "updates" ]; then
+   ppa="ppa:ce-hyperscale/ilp32"
+else
+   ppa="ppa:ce-hyperscale/ilp32-proposed"
+fi
+
+dput $ppa $LINUX_CHANGES
+dput $ppa $LINUX_META_CHANGES