Merge lp:~craig.magina/charms/trusty/ilp32-rebase/trunk into lp:~dannf/charms/trusty/ilp32-rebase/trunk

Proposed by Craig Magina
Status: Needs review
Proposed branch: lp:~craig.magina/charms/trusty/ilp32-rebase/trunk
Merge into: lp:~dannf/charms/trusty/ilp32-rebase/trunk
Diff against target: 355 lines (+196/-36)
7 files modified
README (+6/-2)
config.yaml (+16/-0)
files/do-ilp32-rebase (+116/-20)
gen-config (+10/-2)
get-results (+17/-0)
hooks/install (+11/-12)
upload-results (+20/-0)
To merge this branch: bzr merge lp:~craig.magina/charms/trusty/ilp32-rebase/trunk
Reviewer Review Type Date Requested Status
dann frazier Pending
Review via email: mp+251473@code.launchpad.net
To post a comment you must log in.
Revision history for this message
dann frazier (dannf) wrote :

I think it makes sense for you to own this code now. If you agree, I can just mark my branch is abandoned. But, I'm happy to provide review comments - I'll try to add them inline.

One overall comment - in general I find it easier to review things when there is 1 commit per change. I think you've done a good job at splitting out the individual functional changes in the bullets, but with one commit, it isn't always obvious which bullet a specific line change is addressing.

4. By Craig Magina

Fixed an issue with the package versioning that was making the new packages older then the ones they were supposed to replace

Revision history for this message
Craig Magina (craig.magina) wrote :

I'll keep that in mind for all future work, thanks for the tip. I am fine taking ownership of it from here on out. Another set of eyes is always useful with code, so I would appreciate your review.

Revision history for this message
dann frazier (dannf) wrote :

Huh.. I did do an inline review.. but now I'm not seeing those
comments on the MP.. wtf?

Revision history for this message
dann frazier (dannf) wrote :

Ah - ok, your last push hides those review comments since it changes the diff. To see those comments, look under "Preview Diff" below, and select "r3 into r2 on 2015-03-02"

5. By Craig Magina

Added support for specifying the release to use as the base kernel (updates|proposed)

Revision history for this message
Craig Magina (craig.magina) :
6. By Craig Magina

Moved the apt clean back to its original location as suggested by dann

7. By Craig Magina

Revert manual debian changelog entry generation to debain/rules startnewrelease target as suggested by dann

8. By Craig Magina

Single quotes are not expanded in bash, need double quotes to cause $RELEASE to be expanded properly

9. By Craig Magina

Fix for the package version startnewrelease sets, since we want to build on the current package version, not incremement to the next version.

10. By Craig Magina

Removed the script generation from do-ilp32-rebase
Added configuration output to do-ilp32-rebase
Created script to download the results from the juju unit
Created a script to upload the package to the ilp32 ppa

11. By Craig Magina

Removed un-used variable

Unmerged revisions

11. By Craig Magina

Removed un-used variable

10. By Craig Magina

Removed the script generation from do-ilp32-rebase
Added configuration output to do-ilp32-rebase
Created script to download the results from the juju unit
Created a script to upload the package to the ilp32 ppa

9. By Craig Magina

Fix for the package version startnewrelease sets, since we want to build on the current package version, not incremement to the next version.

8. By Craig Magina

Single quotes are not expanded in bash, need double quotes to cause $RELEASE to be expanded properly

7. By Craig Magina

Revert manual debian changelog entry generation to debain/rules startnewrelease target as suggested by dann

6. By Craig Magina

Moved the apt clean back to its original location as suggested by dann

5. By Craig Magina

Added support for specifying the release to use as the base kernel (updates|proposed)

4. By Craig Magina

Fixed an issue with the package versioning that was making the new packages older then the ones they were supposed to replace

3. By Craig Magina

Config changes:
  * The URI for the ILP32 patches git tree is setable
  * The ILP32 start and end pointers are setable
The linux-meta package is now built to match the kernel abi
do-ilp32-rebase no longer needs to be run under root, running it as root will actually break on the linux-meta package build
Scripts are generate at the end of each build phase (linux and linux-meta):
  * Downloads all parts required for upload
  * Signs package sources for upload
  * Uploads package sources to the ilp32-proposed ppa
  * Suggests an optimized means of downloading the git trees to be saved for history

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'README'
2--- README 2014-12-17 01:14:12 +0000
3+++ README 2015-04-15 13:14:49 +0000
4@@ -1,19 +1,23 @@
5 # Overview
6
7-This charm is used to help maintain the ilp32-enabled arm64 kernel in ppa:ce-hyperscale/ilp32. As the Ubuntu 14.04 kernel is updated, the ilp32 patches need to be rebased and packaged. On deployment, this charm will install all the necessary packages to do the job and seed content from various git locations. A user can then login and run a single command that will:
8+This charm is used to help maintain the ilp32-enabled arm64 kernel in ppa:ce-hyperscale/ilp32-proposed. As the Ubuntu 14.04 kernel is updated, the ilp32 patches need to be rebased and packaged. On deployment, this charm will install all the necessary packages to do the job and seed content from various git locations. A user can then login and run a single command that will:
9 1. Query launchpad to find the version of the current kernel package in trusty-proposed
10 2. Pull updates for all relevant git branches
11 3. Create a new git branch based on the tag of the kernel currently in trusty-proposed
12 4. Rebase ILP32 patches from a reference tree onto this tag
13 5. Prepare a source package suitable for upload to a test PPA
14+ 6. Repeat for the linux-meta package
15
16 # Usage
17
18 Step by step instructions on using the charm:
19
20 juju deploy ilp32-rebase
21+ juju set user-email="user@example.org" user-fullname="Example User"
22 juju ssh ilp32-rebase/0
23- sudo /usr/bin/do-ilp32-rebase
24+ /usr/bin/do-ilp32-rebase
25+
26+The config allows a user to set the URI where the ILP32 patches are located and pointers to the start and end of the patch set.
27
28 ## Scale out Usage
29
30
31=== modified file 'config.yaml'
32--- config.yaml 2014-12-17 01:14:12 +0000
33+++ config.yaml 2015-04-15 13:14:49 +0000
34@@ -7,3 +7,19 @@
35 type: string
36 default: "Example User"
37 description: "Full name to use in commits and changelogs"
38+ release:
39+ type: string
40+ default: "proposed"
41+ description: "The trusty release to use as the base for the ilp32 kernel"
42+ ilp32-ref-tree:
43+ type: string
44+ default: "git://kernel.ubuntu.com/dannf/trusty-xgene.git"
45+ description: "GIT reference tree containing the ilp32 support patches"
46+ ilp32-start-ref:
47+ type: string
48+ default: "ilp32/ilp32-start"
49+ description: "GIT reference to the first ilp32 support commit"
50+ ilp32-end-ref:
51+ type: string
52+ default: "ilp32/ilp32"
53+ description: "GIT reference to the end ilp32 support commit"
54
55=== added directory 'files'
56=== renamed file 'do-ilp32-rebase' => 'files/do-ilp32-rebase'
57--- do-ilp32-rebase 2014-12-17 01:14:12 +0000
58+++ files/do-ilp32-rebase 2015-04-15 13:14:49 +0000
59@@ -12,15 +12,25 @@
60 export GIT_COMMITTER_EMAIL="$USER_EMAIL"
61 export GIT_COMMITTER_NAME="$USER_FULLNAME"
62
63-cd $GIT_TREE
64+ILP32_REBASE_DIR=`mktemp -d /tmp/ilp32-rebase-XXXX`
65+
66+cd $ILP32_REBASE_DIR
67+
68+# Build linux
69+# Fetch the ubuntu and reference trees.
70+git clone /usr/src/ilp32-rebase/linux
71+cd linux
72+
73+git remote add ilp32 $ILP32_REF_TREE
74+git fetch ilp32
75+git remote add trusty git://kernel.ubuntu.com/ubuntu/ubuntu-trusty.git
76 git fetch trusty
77-git fetch ilp32
78
79 prover=$(get-src-package-version \
80- --archive 'http://ports.ubuntu.com/ubuntu-ports' \
81- --release 'trusty-proposed' \
82- --section 'main' \
83- linux)
84+ --archive "http://ports.ubuntu.com/ubuntu-ports" \
85+ --release "trusty-$RELEASE" \
86+ --section "main" \
87+ linux)
88
89 tag="Ubuntu-${prover}"
90 ilp32_branch="${prover}+ilp32"
91@@ -29,7 +39,7 @@
92 git reset --hard HEAD
93 # get off of ilp32 branch if we are on it
94 # (git won't let us delete the branch we're sitting on)
95-git checkout origin/master
96+git checkout master
97 git branch -D "$ilp32_branch" || /bin/true
98 git branch "$ilp32_branch" "$tag"
99 git checkout "$ilp32_branch"
100@@ -40,30 +50,116 @@
101 ./debian/scripts/misc/getabis ${upstream_ver} ${old_package_rev}
102
103 # clean causes debian/control to be generated, needed by startnewrelease
104-make -f debian/rules clean
105-make -f debian/rules startnewrelease
106+fakeroot make -f debian/rules clean
107+fakeroot make -f debian/rules startnewrelease
108+
109 # This adds a middle ABI number and appends '+ilp32.1'
110-sed -i -r '0,/(.*\(.+)-(.+)\.(.+)\)(.*)/s//\1\-\2\.\2\.\3\+ilp32\.1\)\4/' \
111+sed -i -r "0,/(.*\(.+)-(.+)\.(.+)\)(.*)/s//\1\-${old_package_rev%.*}\.${old_package_rev#*.}\.${old_package_rev#*.}\+ilp32\.1\)\4/" \
112 debian.master/changelog
113 git add debian.master/changelog
114-git commit -s -m 'UBUNTU: Start new release
115-
116-Ignore: yes'
117-
118-# FIXME: the start/end references should be user configurable
119-git log --reverse ilp32/ilp32-start..ilp32/ilp32 | grep ^commit | cut -d' ' -f2 | \
120+git commit -s -F debian/commit-templates/newrelease
121+
122+git log --reverse $ILP32_START_REF..$ILP32_END_REF | grep ^commit | cut -d' ' -f2 | \
123 while read c; do
124- git cherry-pick $c
125- done
126+ git cherry-pick $c
127+done
128
129-make -f debian/rules insertchanges
130+fakeroot make -f debian/rules insertchanges
131 dch -c debian.master/changelog -r ""
132 git add debian.master/changelog
133 newver="$(dpkg-parsechangelog --show-field Version -ldebian.master/changelog)"
134 git commit -s -m "UBUNTU: Ubuntu-$newver"
135+
136+cp /usr/src/ilp32-rebase/linux_${upstream_ver}.orig.tar.gz $ILP32_REBASE_DIR
137 rm -rf ../linux-${upstream_ver}
138 git clone . ../linux-${upstream_ver}
139 cd ../linux-${upstream_ver}
140 rm -rf .git
141-make -f debian/rules clean
142+fakeroot make -f debian/rules clean
143 dpkg-buildpackage -rfakeroot -uc -us -S
144+
145+# Generate ilp32-rebase-results file
146+cd $ILP32_REBASE_DIR
147+changes=`ls -1 linux_${upstream_ver}-*+ilp32.1_source.changes`
148+dsc=`ls -1 linux_${upstream_ver}-*+ilp32.1.dsc`
149+diff=`ls -1 linux_${upstream_ver}-*+ilp32.1.diff.gz`
150+
151+cat > $HOME/ilp32-rebase-results <<EOF
152+JUJU_UNIT_NAME=$JUJU_UNIT_NAME
153+ILP32_REBASE_DIR=$ILP32_REBASE_DIR
154+LINUX_CHANGES=$changes
155+LINUX_DSC=$dsc
156+LINUX_DIFF=$diff
157+LINUX_GIT=$ILP32_REBASE_DIR/linux
158+EOF
159+# End of linux build
160+
161+# Build linux-meta
162+cd $ILP32_REBASE_DIR
163+
164+git clone /usr/src/ilp32-rebase/linux-meta
165+cd linux-meta
166+
167+prover=$(get-src-package-version \
168+ --archive "http://ports.ubuntu.com/ubuntu-ports" \
169+ --release "trusty-$RELEASE" \
170+ --section "main" \
171+ linux-meta)
172+
173+tag="Ubuntu-${prover}"
174+ilp32_branch="${prover}+ilp32"
175+
176+git clean -x -f -d .
177+git reset --hard HEAD
178+# get off of ilp32 branch if we are on it
179+# (git won't let us delete the branch we're sitting on)
180+git checkout master
181+git branch -D "$ilp32_branch" || /bin/true
182+git branch "$ilp32_branch" "$tag"
183+git checkout "$ilp32_branch"
184+
185+linux_upstream_ver="${upstream_ver}"
186+linux_package_rev="${old_package_rev}"
187+
188+oldver="$(dpkg-parsechangelog --show-field Version -lmeta-source/debian/changelog)"
189+upstream_ver="${oldver%.*.*}"
190+old_package_rev="${oldver#${upstream_ver}.}"
191+
192+now="$(date -R)"
193+cat > meta-source/debian/changelog.new <<EOF
194+linux-meta (${upstream_ver}.${linux_package_rev}.${old_package_rev#*.}+ilp32.1) UNRELEASED; urgency=low
195+
196+ * linux ABI ${upstream_ver}-${linux_package_rev}
197+
198+ -- $DEBFULLNAME <$DEBEMAIL> $now
199+EOF
200+cat meta-source/debian/changelog >> meta-source/debian/changelog.new
201+mv meta-source/debian/changelog.new meta-source/debian/changelog
202+dch -c meta-source/debian/changelog -r ""
203+git add meta-source/debian/changelog
204+
205+# Update the KERNEL_VERSION and KERNEL_ABI in the debian/rules file
206+sed "/KERNEL_ABI_VERSION=.*/i \
207+KERNEL_VERSION=${linux_upstream_ver}\n\
208+KERNEL_ABI=${linux_package_rev}" -i meta-source/debian/rules
209+git add meta-source/debian/rules
210+git commit -s -m "UBUNTU: Ubuntu-${upstream_ver}.${linux_package_rev}.${old_package_rev#*.}+ilp32.1"
211+
212+make
213+
214+# Append linux-meta to il32-rebase-results config
215+changes=`ls -1 linux-meta_*source.changes`
216+dsc=`ls -1 linux-meta_*.dsc`
217+tarball=`ls -1 linux-meta_*.tar.gz`
218+
219+echo >> $HOME/ilp32-rebase-results <<EOF
220+LINUX_META_CHANGES=$changes
221+LINUX_META_DSC=$dsc
222+LINUX_META_TARBALL=$tarball
223+LINUX_META_GIT=$ILP32_REBASE_DIR/linux-meta
224+EOF
225+# End of linux-meta build
226+
227+echo "To retrieve results, download the ilp32-rebase-results file and run the get-results script"
228+echo "juju scp $JUJU_UNIT_NAME:$HOME/ilp32-rebase-results ."
229+echo "get-results"
230
231=== renamed file 'get-src-package-version' => 'files/get-src-package-version'
232=== modified file 'gen-config'
233--- gen-config 2014-12-17 01:14:12 +0000
234+++ gen-config 2015-04-15 13:14:49 +0000
235@@ -3,15 +3,23 @@
236 set -e
237 set -x
238
239-git_tree="$CHARM_DIR/linux"
240+juju_unit_name="$JUJU_UNIT_NAME"
241 user_email="$(config-get user-email)"
242 user_fullname="$(config-get user-fullname)"
243+release="$(config-get release)"
244+ilp32_ref_tree="$(config-get ilp32-ref-tree)"
245+ilp32_start_ref="$(config-get ilp32-start-ref)"
246+ilp32_end_ref="$(config-get ilp32-end-ref)"
247
248 mkdir -p /etc/ilp32-rebase
249 cat > /etc/ilp32-rebase/config.new <<EOF
250-GIT_TREE="$git_tree"
251+JUJU_UNIT_NAME="$juju_unit_name"
252 USER_EMAIL="$user_email"
253 USER_FULLNAME="$user_fullname"
254+RELEASE="$release"
255+ILP32_REF_TREE="$ilp32_ref_tree"
256+ILP32_START_REF="$ilp32_start_ref"
257+ILP32_END_REF="$ilp32_end_ref"
258 EOF
259
260 # Atomically replace to avoid using half-written configs
261
262=== added file 'get-results'
263--- get-results 1970-01-01 00:00:00 +0000
264+++ get-results 2015-04-15 13:14:49 +0000
265@@ -0,0 +1,17 @@
266+#!/bin/sh
267+#
268+# Downloads the results generated by the do-ilp32-rebase script from the juju unit.
269+#
270+
271+source ilp32-rebase-results
272+
273+ILP32_REBASE_IP=$(juju-deployer -f ilp32-rebase)
274+
275+juju scp $JUJU_UNIT_NAME:$ILP32_REBASE_DIR/$LINUX_CHANGES
276+juju scp $JUJU_UNIT_NAME:$ILP32_REBASE_DIR/$LINUX_DSC
277+juju scp $JUJU_UNIT_NAME:$ILP32_REBASE_DIR/$LINUX_DIFF
278+juju scp $JUJU_UNIT_NAME:$ILP32_REBASE_DIR/$LINUX_META_CHANGES
279+juju scp $JUJU_UNIT_NAME:$ILP32_REBASE_DIR/$LINUX_DSC
280+
281+git clone --single-branch ssh://$ILP32_REBASE_IP/$LINUX_GIT linux
282+git clone --single-branch ssh://$ILP32_REBASE_IP/$LINUX_META_GIT linux-meta
283
284=== modified file 'hooks/install'
285--- hooks/install 2014-12-17 01:14:12 +0000
286+++ hooks/install 2015-04-15 13:14:49 +0000
287@@ -5,12 +5,17 @@
288
289 apt-get update
290 apt-get build-dep -y linux
291+apt-get install -y devscripts dpkg-dev fakeroot git python3-debian
292+
293+mkdir -m 0755 -p /usr/src/ilp32-rebase
294+cd /usr/src/ilp32-rebase
295+
296 # For the upstream tarball, since this is a non-native package
297 apt-get source --download-only linux
298-apt-get install -y devscripts dpkg-dev fakeroot git python3-debian
299 apt-get clean
300
301-rm -rf linux # for indempotency (are install hooks retriable?)
302+# for indempotency (are install hooks retriable?)
303+rm -rf linux
304
305 #
306 # HACK
307@@ -25,19 +30,13 @@
308 #git clone git://kernel.ubuntu.com/ubuntu/ubuntu-trusty.git linux
309 git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
310
311-# Fetch the ubuntu and reference trees. The do-ilp32-rebase script will
312-# fetch new changes, this is just to seed it.
313-cd linux
314-git remote add ilp32 git://kernel.ubuntu.com/dannf/trusty-xgene.git
315-git fetch ilp32
316-git remote add trusty git://kernel.ubuntu.com/ubuntu/ubuntu-trusty.git
317-git fetch trusty
318-cd -
319+# Clone the linux-meta sources
320+git clone git://kernel.ubuntu.com/ubuntu/ubuntu-trusty-meta.git linux-meta
321
322 # do-ilp32-rebase currently runs outside of juju context, so it can't
323 # get-config charm settings itself. Use gen-config to pass those on.
324 $CHARM_DIR/gen-config
325
326 # These should be replaced by juju actions when they exist
327-install -m 755 get-src-package-version /usr/bin
328-install -m 755 do-ilp32-rebase /usr/bin
329+install -m 755 $CHARM_DIR/files/get-src-package-version /usr/bin
330+install -m 755 $CHARM_DIR/files/do-ilp32-rebase /usr/bin
331
332=== added file 'upload-results'
333--- upload-results 1970-01-01 00:00:00 +0000
334+++ upload-results 2015-04-15 13:14:49 +0000
335@@ -0,0 +1,20 @@
336+#!/bin/sh
337+#
338+# Sign and upload linux and linux-meta to the ilp32 ppa
339+# By default, it will upload to the proposed ppa unless
340+# "updates" is passed in as the first argument.
341+#
342+
343+source ilp32-rebase-results
344+
345+debsign $LINUX_CHANGES
346+debsign $LINUX_META_CHANGES
347+
348+if [ "$1" = "updates" ]; then
349+ ppa="ppa:ce-hyperscale/ilp32"
350+else
351+ ppa="ppa:ce-hyperscale/ilp32-proposed"
352+fi
353+
354+dput $ppa $LINUX_CHANGES
355+dput $ppa $LINUX_META_CHANGES

Subscribers

People subscribed via source and target branches

to all changes: