Merge lp:~nskaggs/phablet-tools/add-lp-branch-support into lp:phablet-tools

Proposed by Nicholas Skaggs on 2014-01-14
Status: Work in progress
Proposed branch: lp:~nskaggs/phablet-tools/add-lp-branch-support
Merge into: lp:phablet-tools
Diff against target: 392 lines (+316/-18)
3 files modified
click-build.py (+231/-0)
phablet-click-test-setup (+61/-18)
po2mo.sh (+24/-0)
To merge this branch: bzr merge lp:~nskaggs/phablet-tools/add-lp-branch-support
Reviewer Review Type Date Requested Status
Ubuntu Phablet Team 2014-01-14 Pending
Review via email: mp+201685@code.launchpad.net

Commit message

Make some tweaks to phablet-click-test-setup to allow an easier workflow for app developers.

Add option to not bootstrap device everytime (we shouldn't need to constantly pull unity8 tests)
Add option to build click app and extract tests to target device for running
Add -c option (same as --click)

Description of the change

Make some tweaks to phablet-click-test-setup to allow an easier workflow for app developers. Desired workflow.

Hack on tests
Push to launchpad
Run tests on device / emulator via
phablet-click-test-setup lp:mybranch
phablet-test-run my_apptest

Add option to not bootstrap device everytime (we shouldn't need to constantly pull unity8 tests)
Add option to build click app and extract tests to target device for running
Add -c option (same as --click)

To post a comment you must log in.
239. By Nicholas Skaggs on 2014-01-14

remove old shell script

Nicholas Skaggs (nskaggs) wrote :

Note click_build.py is a tweaked version of Sergio's script. Need to migrate po2mo.sh functionality into click_build.py.

Sergio Schvezov (sergiusens) wrote :

Hm, I did this as an intermediate step until we got the debian packaging out of the way and all the path hardcoding was gone, that hasn't happened though; I would much rather have a simple cmake rules (or plain Makefile) instead of this.

This also doesn't work for compiled apps (such as notes, gallery, camera, mediaplayer, etc).

Unmerged revisions

239. By Nicholas Skaggs on 2014-01-14

remove old shell script

238. By Nicholas Skaggs on 2014-01-14

removed hacky globals, tweaked variable names

237. By Nicholas Skaggs on 2014-01-14

streamlined to make program more logical

236. By Nicholas Skaggs on 2014-01-14

fixed some bugs, hacky, but runs as expected

235. By Nicholas Skaggs on 2014-01-14

wip commit to move install_from_branch to click-test-setup

234. By Nicholas Skaggs on 2014-01-14

working version, nice and hacky

233. By Nicholas Skaggs on 2014-01-14

initial version of installing from an arbitrary lp branch to device

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added file 'click-build.py'
2--- click-build.py 1970-01-01 00:00:00 +0000
3+++ click-build.py 2014-01-14 22:09:32 +0000
4@@ -0,0 +1,231 @@
5+#!/usr/bin/python3
6+
7+import argparse
8+import atexit
9+import configparser
10+import json
11+import os
12+import shutil
13+import sys
14+import tempfile
15+
16+from subprocess import check_call, check_output
17+
18+
19+tool_path = os.path.dirname(os.path.abspath(sys.modules['__main__'].__file__))
20+
21+
22+def get_parser():
23+ parser = argparse.ArgumentParser(
24+ description='''Parses sources and stages them to build click
25+ packages out of them.''')
26+ parser.add_argument('--source-directory', dest='path',
27+ default=os.path.curdir)
28+ parser.add_argument('--bzr-source', required=True)
29+ parser.add_argument('--bzr-revno', default=None)
30+ parser.add_argument('--no-clean', action='store_true')
31+ return parser
32+
33+
34+def fix_exec_inline(exec_line, path):
35+ print('Fixing exec inline %s with %s' % (exec_line, path))
36+ exec_line_len = len(exec_line)
37+ if exec_line_len == 3:
38+ exec_line = '%s %s %s' % (
39+ exec_line[0], strip_path(exec_line[1], path),
40+ strip_path(exec_line[2], path))
41+ elif exec_line_len == 2:
42+ exec_line = '%s %s' % (exec_line[0], strip_path(exec_line[1], path))
43+ elif exec_line_len == 1:
44+ exec_line = strip_path(exec_line[0], path)
45+ else:
46+ raise NotImplementedError('Cannot handle Exec of len %s' %
47+ exec_line_len)
48+ print('New exec line is %s' % exec_line)
49+ return exec_line
50+
51+
52+def add_import(exec_line):
53+ print('Adding import %s' % exec_line)
54+ if 'qmlscene' in exec_line:
55+ exec_line = '%s -I ./plugins' % exec_line
56+ return exec_line
57+
58+
59+def strip_path(item, path):
60+ """Returns unstripped path for item."""
61+ basename = os.path.basename(item)
62+ if item == basename:
63+ print('%s is good' % item)
64+ else:
65+ for root, sub, files in os.walk(path):
66+ if basename in files:
67+ basename = os.path.join(os.path.relpath(root, path), basename)
68+ break
69+ print('Replacing %s with %s' % (item, basename))
70+ return basename
71+
72+
73+def remove_cruft(path):
74+ print('Checking for unneeded directories')
75+ for cruft_dir in ('debian', 'tests', 'po'):
76+ cruft_dir = os.path.join(path, cruft_dir)
77+ if os.path.exists(cruft_dir):
78+ print('Removing %s' % cruft_dir)
79+ shutil.rmtree(cruft_dir)
80+
81+
82+def get_plugins(path):
83+ print('Looking for needed plugins')
84+ plugin_manifest = os.path.join(path, 'plugins.json')
85+ if not os.path.exists(plugin_manifest):
86+ print('No extra plugins required')
87+ return
88+ plugin_target_path = os.path.join(path, 'plugins')
89+ os.makedirs(plugin_target_path)
90+ with open(plugin_manifest) as f:
91+ manifest = json.loads(f.read())
92+ staging_path = tempfile.mkdtemp()
93+ atexit.register(lambda x: shutil.rmtree(x), staging_path)
94+ for item in manifest:
95+ exec_line = [os.path.join(tool_path, 'pull-lp-bin.py'),
96+ '-o', staging_path, item['package']]
97+ if 'ppa' in item:
98+ exec_line += ['--ppa', item['ppa']]
99+ check_call(exec_line, cwd=tool_path)
100+ debs = [os.path.join(staging_path, x) for x in os.listdir(staging_path)
101+ if x.endswith('.deb')]
102+ for deb in debs:
103+ extract_path = deb.strip('.deb')
104+ check_call(['dpkg-deb', '-x', deb, extract_path])
105+ plugin_path = os.path.join(extract_path,
106+ 'usr/lib/arm-linux-gnueabihf/qt5/qml')
107+ for plugin in os.listdir(plugin_path):
108+ print('Moving %s to %s' % (plugin, plugin_target_path))
109+ shutil.move(os.path.join(plugin_path, plugin),
110+ plugin_target_path)
111+
112+
113+def run_translations(path, appname):
114+ #trans_arg = [os.path.join(tool_path, 'po2mo.sh'), path, appname]
115+ trans_arg = [os.path.join(tool_path, 'po2mo.sh'), path, appname, source_path]
116+ print('Checking for translations with %s' % trans_arg)
117+ check_call(trans_arg)
118+
119+
120+class Click(object):
121+
122+ @property
123+ def desktop(self):
124+ for i in self.manifest['hooks']:
125+ desktop = self.manifest['hooks'][i]['desktop']
126+ print(desktop)
127+ return desktop
128+
129+ @property
130+ def name(self):
131+ return self.manifest['name']
132+
133+ @property
134+ def architecture(self):
135+ return self.manifest['architecture'] \
136+ if 'architecture' in self.manifest else None
137+
138+ def __init__(self, path):
139+ self.manifest_path = os.path.join(path, 'manifest.json')
140+ self.manifest_read()
141+
142+ def manifest_read(self):
143+ print('Reading manifest.json')
144+ with open(self.manifest_path) as f:
145+ self.manifest = json.loads(f.read())
146+
147+ def manifest_write(self):
148+ print('Writing manifest.json')
149+ with open(self.manifest_path, 'w') as f:
150+ for line in json.dumps(self.manifest, sort_keys=True, indent=4):
151+ f.write(line)
152+
153+ def add_x_source(self, bzr_source, bzr_revno):
154+ print('Adding source to manifest %s, at rev %s' % (bzr_source, bzr_revno))
155+ self.manifest['x-source'] = {
156+ 'vcs-bzr': bzr_source,
157+ 'vcs-bzr-revno': bzr_revno
158+ }
159+
160+ def append_version(self, bzr_revno):
161+ print('Adding version to manifest %s' % (bzr_revno))
162+ self.manifest['version'] += '.%s' % bzr_revno
163+
164+
165+class Desktop(object):
166+
167+ def __init__(self, desktop_path):
168+ self.desktop_path = desktop_path
169+ self.read()
170+
171+ def read(self):
172+ self.desktop = configparser.RawConfigParser()
173+ self.desktop.optionxform = str
174+ self.desktop.read(self.desktop_path)
175+
176+ def fix_icon(self, path):
177+ print('Fixing icon %s' % path)
178+ icon = self.desktop.get('Desktop Entry', 'Icon')
179+ self.desktop.set('Desktop Entry', 'Icon', strip_path(icon, path))
180+
181+ def fix_exec(self, path, arch=None):
182+ print('Fixing exec %s' % path)
183+ exec_line = self.desktop.get('Desktop Entry', 'Exec').split()
184+ exec_line = fix_exec_inline(exec_line, path)
185+ if arch:
186+ print('Creating custom exec')
187+ exec_line = add_import(exec_line)
188+ self.desktop.set('Desktop Entry', 'Exec', exec_line)
189+
190+ def write(self):
191+ with open(self.desktop_path, 'w') as f:
192+ self.desktop.write(f, space_around_delimiters=False)
193+
194+
195+def setup_workdir(path, no_clean):
196+ print('Setting up workdir %s' % path)
197+ tempdir = tempfile.mkdtemp()
198+ if not no_clean:
199+ print('Cleaning up after work %s' % tempdir)
200+ atexit.register(lambda x: shutil.rmtree(x), tempdir)
201+ if not os.path.exists(path):
202+ raise RuntimeError('Directory %s does not exist' % path)
203+ source_path = os.path.join(tempdir, 'source')
204+ check_call(['bzr', 'export', source_path], cwd=path)
205+ print('Exported source to %s' % source_path)
206+ return source_path
207+
208+
209+def revno(path):
210+ revno = check_output(['bzr', 'revno'], cwd=path).decode("utf-8").strip()
211+ return revno
212+
213+
214+if __name__ == '__main__':
215+ parser = get_parser()
216+ args = parser.parse_args()
217+ path = os.path.abspath(args.path)
218+ source_path = setup_workdir(path, args.no_clean)
219+ click = Click(source_path)
220+ bzr_revno = args.bzr_revno if args.bzr_revno else revno(path)
221+ print('Building package from %s with revno %s' % (path, bzr_revno))
222+ click.add_x_source(args.bzr_source, bzr_revno)
223+ click.append_version(bzr_revno)
224+ get_plugins(source_path)
225+ desktop = Desktop(os.path.join(source_path, click.desktop))
226+ desktop.fix_icon(source_path)
227+ desktop.fix_exec(source_path, arch=click.architecture)
228+ desktop.write()
229+ click.manifest_write()
230+ run_translations(source_path, click.name)
231+ remove_cruft(source_path)
232+ click_arg = ['click', 'build', source_path]
233+ print('Building click package with %s' % click_arg)
234+ check_call(click_arg)
235+ print('Finished building click package from %s' % path)
236
237=== modified file 'phablet-click-test-setup'
238--- phablet-click-test-setup 2013-10-21 18:14:51 +0000
239+++ phablet-click-test-setup 2014-01-14 22:09:32 +0000
240@@ -26,6 +26,7 @@
241 import atexit
242 import json
243 import os
244+import sys
245 import shutil
246 import tempfile
247 import urllib2
248@@ -92,14 +93,29 @@
249 help='''Device serial. Use when more than
250 one device is connected.''',
251 )
252- parser.add_argument('--click', default=None,
253+ parser.add_argument('-c', '--click', default=None,
254 help='Specific click package to setup for.'
255 )
256+ parser.add_argument('-b', '--branch', default=None,
257+ help='Setup click package from lp branch. Requires \
258+ you specify click package --click'
259+ )
260+ parser.add_argument('--no-bootstrap', action="store_false",
261+ dest="bootstrap",
262+ help='Do not setup basic packages '
263+ )
264 parser.add_argument('--user', default='phablet',
265 help='User on device to use')
266 parser.add_argument('--wipe', action='store_true',
267 help='Clean up previous setup on device')
268- return parser.parse_args()
269+
270+ args = parser.parse_args()
271+
272+ #check dependency
273+ if args.branch and not args.click:
274+ parser.print_help()
275+ sys.exit()
276+ return args
277
278
279 def cleanup(directory):
280@@ -176,7 +192,7 @@
281 test_dir = path.join(test_base_dir, test_dir)
282 print('Moving %s to %s' % (test_dir, target_dir))
283 shutil.move(test_dir, target_dir)
284-
285+ return tmp_dir
286
287 def fetch_test_base(adb, test_dir):
288 for package in python_packages:
289@@ -185,32 +201,59 @@
290 version = adb.get_package_version(package['binary'])
291 get_source_package_tests(package['source'], version, test_dir)
292
293-
294-def fetch_click_tests(adb, test_dir, user, click=None):
295- manifest = adb.get_click_manifest(user)
296- if click:
297+def fetch_click_tests(adb, test_dir, user, click=None, branch=None):
298+ if branch:
299+ #do we need to build and install the click package to the device?
300 print('Only setting up for %s' % click)
301- manifest = [entry for entry in manifest if click == entry['name']]
302- print('Only keeping entries with x-source')
303- manifest = [entry for entry in manifest if 'x-source' in entry]
304- for entry in manifest:
305- get_bzr_tests(entry['x-source']['vcs-bzr'],
306- entry['x-source']['vcs-bzr-revno'],
307- test_dir)
308-
309+ print("Building and installing click package for branch %s" % branch)
310+ #force checkout the latest version
311+ revision = "revno:-1"
312+ tmp_dir=get_bzr_tests(branch, revision, test_dir)
313+ install_click_from_branch(branch,
314+ path.join(tmp_dir, 'work'), click, adb)
315+ else:
316+ manifest = adb.get_click_manifest(user)
317+ if click:
318+ print('Only setting up for %s' % click)
319+ manifest = [entry for entry in manifest if click == entry['name']]
320+ print('Only keeping entries with x-source')
321+ manifest = [entry for entry in manifest if 'x-source' in entry]
322+ for entry in manifest:
323+ get_bzr_tests(entry['x-source']['vcs-bzr'],
324+ entry['x-source']['vcs-bzr-revno'],
325+ test_dir)
326+
327+def install_click_from_branch(branch, branch_path, click, adb):
328+ #grab clickbuild tool
329+ print("Calling click build with %s, source %s" % (branch, branch_path))
330+ click_arg = "python3 click-build.py --bzr-source %s --source-directory %s" \
331+ % (branch, branch_path)
332+ os.system(click_arg)
333+
334+ click_arg = "ls | grep %s*.click" % click
335+ click_name = check_output(click_arg, shell=True).strip()
336+ click_path = path.join(branch_path, 'click_name')
337+ shutil.move(click_name, click_path)
338+ print("Pushing %s to device" % click_name)
339+ adb.push(click_path, '/tmp')
340+
341+ print("Installing %s on device" % click_name)
342+ installcmd="sudo -u phablet pkcon install-local /tmp/%s" % click_name
343+ adb.shell(installcmd)
344
345 def main():
346 global lp
347 args = parse_arguments()
348 adb = UbuntuDevice(args.serial)
349 adb.start()
350+ test_dir = tempfile.mkdtemp()
351 arch = adb.shell('dpkg --print-architecture').strip()
352 series = adb.shell('lsb_release -c -s').strip()
353 lp = LP(series, arch)
354- test_dir = tempfile.mkdtemp()
355 atexit.register(cleanup, test_dir)
356- fetch_test_base(adb, test_dir)
357- fetch_click_tests(adb, test_dir, args.user, args.click)
358+ if args.bootstrap:
359+ fetch_test_base(adb, test_dir)
360+ fetch_click_tests(adb, test_dir, args.user, args.click, args.branch)
361 destination = path.join('/home', args.user, 'autopilot')
362 if args.wipe:
363 print('Clearing previous test setup in %s' % destination)
364
365=== added file 'po2mo.sh'
366--- po2mo.sh 1970-01-01 00:00:00 +0000
367+++ po2mo.sh 2014-01-14 22:09:32 +0000
368@@ -0,0 +1,24 @@
369+#!/bin/sh
370+
371+target=$1
372+appname=$2
373+path=$3
374+
375+processed=0
376+for pofile in $(find $3 -name "*.po"); do
377+ processed=1
378+ pofilename="${pofile##*/}"
379+ #echo "Processing $pofilename with $langcode"
380+ langcode="${pofilename%.*}"
381+ localedir="$target/locale/$langcode/LC_MESSAGES"
382+ #echo "Making $localedir"
383+ mkdir -p $localedir
384+ mofile="$localedir/$appname.mo"
385+ msgfmt -o "$mofile" "$pofile"
386+done
387+
388+if [ processed ]; then
389+ echo 'Found and processed po translations to locale folder'
390+else
391+ echo 'No translations found to process'
392+fi

Subscribers

People subscribed via source and target branches