Ubuntu File Manager App

Merge lp:~popey/ubuntu-filemanager-app/add-click-deps into lp:ubuntu-filemanager-app

  1. add-click-deps
  2. Merge into trunk
Status: Merged
Approved by: Alan Pope 🍺🐧🐱 πŸ¦„ on 2015-10-19
Approved revision: 469
Merged at revision: 481
Proposed branch: lp:~popey/ubuntu-filemanager-app/add-click-deps
Merge into: lp:ubuntu-filemanager-app
Diff against target: 538 lines (+518/-0)
3 files modified
CMakeLists.txt (+17/-0)
filemanager-libs.json (+50/-0)
get-click-deps (+451/-0)
To merge this branch: bzr merge lp:~popey/ubuntu-filemanager-app/add-click-deps
Reviewer Review Type Date Requested Status
Carlos Jose Mazieri 2015-09-07 Approve on 2015-10-19
Ubuntu Phone Apps Jenkins Bot continuous-integration Approve on 2015-10-14
Review via email: mp+270287@code.launchpad.net

Commit Message

Add basics to cmake file and necessary parts to pull in samba libs.

Description of the Change

First pass at adding get-click-deps from Stefano Verzegnassi to pull samba libs in during build process. Not final.

To post a comment you must log in.
Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot) wrote :
review: Approve (continuous-integration)
lp:~popey/ubuntu-filemanager-app/add-click-deps updated on 2015-09-07
467. By Alan Pope 🍺🐧🐱 πŸ¦„ on 2015-09-07

New get-click-deps from Stefano and updated deployment process

Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot) wrote :
review: Approve (continuous-integration)
Alan Pope 🍺🐧🐱 πŸ¦„ (popey) wrote :

Ok, built a click based on this merge, which builds successfully, and bundles in all the samba libs.
http://people.canonical.com/~alan/filemanager/com.ubuntu.filemanager_0.4.467_armhf.click

Tried goto -> smb://nas.local or some other local IP which serves up over SMB and it failed. Here's a log extract:- http://paste.ubuntu.com/12307671/

Pretty sure this worked when I manually put all the samba libs in an already installed file manager build from the past. Am I doing something wrong? (probably)

Stefano Verzegnassi (verzegnassi-stefano) wrote :

Feel like I'm having the same issue with loviewer dependencies (libraries not found by the application).

May the links to the shared libraries, provided in the click package, be somehow broken? (probably when cp'ing files from temp folder to their destination)

Carlos Jose Mazieri (carlos-mazieri) wrote :

> Ok, built a click based on this merge, which builds successfully, and bundles
> in all the samba libs.
> http://people.canonical.com/~alan/filemanager/com.ubuntu.filemanager_0.4.467_a
> rmhf.click
>
> Tried goto -> smb://nas.local or some other local IP which serves up over SMB
> and it failed. Here's a log extract:- http://paste.ubuntu.com/12307671/
>
> Pretty sure this worked when I manually put all the samba libs in an already
> installed file manager build from the past. Am I doing something wrong?
> (probably)

Try "unlock full access" first, it blocks remote browsing.
Next MPs will correct this as we discussed on File Manager meeting.

Alan Pope 🍺🐧🐱 πŸ¦„ (popey) wrote :

@stefano, I think it must be finding the libs, because the app crashes (won't even start) if the libs aren't there, or are in the wrong directory usually.

@carlos, yes, I tried that, but it still wouldn't connect. Will try on a different device as this one has lots of other things installed which may compromise it.

Alan Pope 🍺🐧🐱 πŸ¦„ (popey) wrote :

Just tried this click on my "retail" bq e4.5 (so it's still read only - not had any debian packages installed - is as a customer would find it). Not able to browse to my NAS.

http://termbin.com/vdt4

Is there a "known good" config that this should connect to?

Carlos Jose Mazieri (carlos-mazieri) wrote :

Looking at the log you posted, it looks like there is an extra space
after "smb":
void DirModel::setPath(const QString&, const QString&, const QString&,
bool) DirModel_QML_137(0x12ff010) path or url may not exist or cannot
be read: "smb ://192.168.1.3/"

The correct url must be "smb://192.168.1.3".

BTW, I have had some hard time trying to type "smb" urls on the device.

On 9/8/15, Alan Pope ξƒΏ <email address hidden> wrote:
> Just tried this click on my "retail" bq e4.5 (so it's still read only - not
> had any debian packages installed - is as a customer would find it). Not
> able to browse to my NAS.
>
> http://termbin.com/vdt4
>
> Is there a "known good" config that this should connect to?
> --
> https://code.launchpad.net/~popey/ubuntu-filemanager-app/add-click-deps/+merge/270287
> Your team Ubuntu File Manager Developers is requested to review the proposed
> merge of lp:~popey/ubuntu-filemanager-app/add-click-deps into
> lp:ubuntu-filemanager-app.
>

Alan Pope 🍺🐧🐱 πŸ¦„ (popey) wrote :

Well spotted. We should disable the keyboard helpers (which try to autocorrect when you type in a location) and that will stop the space getting added, and also stop smb being 'corrected' to sob.

I tried again and this time the file manager crashes.

http://termbin.com/d8x0

The crash dump was uploaded, and processed. It's a secured page so I printed it as a PDF, I hope that's readable.

http://people.canonical.com/~alan/filemanager_crash.pdf

Carlos Jose Mazieri (carlos-mazieri) wrote :

It is hard to say something about the crash, if you have the
'libsmbclient' installed in your phone you can try this click package
which contains 'Network' in Places to start navigating in Samba.
https://www.dropbox.com/s/f6gxiqbav3th8bx/com.ubuntu.filemanager_0.4.latest_armhf.click?dl=0

Have you tested on Desktop?

On 9/8/15, Alan Pope ξƒΏ <email address hidden> wrote:
> Well spotted. We should disable the keyboard helpers (which try to
> autocorrect when you type in a location) and that will stop the space
> getting added, and also stop smb being 'corrected' to sob.
>
> I tried again and this time the file manager crashes.
>
> http://termbin.com/d8x0
>
> The crash dump was uploaded, and processed. It's a secured page so I printed
> it as a PDF, I hope that's readable.
>
> http://people.canonical.com/~alan/filemanager_crash.pdf
> --
> https://code.launchpad.net/~popey/ubuntu-filemanager-app/add-click-deps/+merge/270287
> Your team Ubuntu File Manager Developers is requested to review the proposed
> merge of lp:~popey/ubuntu-filemanager-app/add-click-deps into
> lp:ubuntu-filemanager-app.
>

Carlos Jose Mazieri (carlos-mazieri) wrote :

> It is hard to say something about the crash, if you have the
> 'libsmbclient' installed in your phone you can try this click package
> which contains 'Network' in Places to start navigating in Samba.
> https://www.dropbox.com/s/f6gxiqbav3th8bx/com.ubuntu.filemanager_0.4.latest_ar
> mhf.click?dl=0
>
> Have you tested on Desktop?
>
> On 9/8/15, Alan Pope ξƒΏ <email address hidden> wrote:
> > Well spotted. We should disable the keyboard helpers (which try to
> > autocorrect when you type in a location) and that will stop the space
> > getting added, and also stop smb being 'corrected' to sob.
> >
> > I tried again and this time the file manager crashes.
> >
> > http://termbin.com/d8x0
> >
> > The crash dump was uploaded, and processed. It's a secured page so I printed
> > it as a PDF, I hope that's readable.
> >
> > http://people.canonical.com/~alan/filemanager_crash.pdf
> > --
> > https://code.launchpad.net/~popey/ubuntu-filemanager-app/add-click-
> deps/+merge/270287
> > Your team Ubuntu File Manager Developers is requested to review the proposed
> > merge of lp:~popey/ubuntu-filemanager-app/add-click-deps into
> > lp:ubuntu-filemanager-app.
> >

@popey, I do confirm the crash on Samba browsing, it happens on the phone, works on Desktop and works on the phone using the plugin/model as console application.

I am sorry for that, I did not test it properly, I worked on this on weekend and cannot solve it, I need the debug to work in order to find out the problem, I am able to use the debug from qtCreator on device, but it does not work properly, it behaves as a non debug invocation, when the crash happens there is not information about it.

Also tried using "gdb" directly on the device without success, in this case I could generate a core file with no enough information either.

Also added some qDebug() lines but it did not give me the reason for the crash.

If you have any idea about the crash or about other way of debugging please share with me.

Carlos Jose Mazieri (carlos-mazieri) wrote :

Hello,

I did a first test without removing my Samba libraries.

On desktop it downloaded the libraries but on armhf it did not.

See some output:
-- Setting BZR_SOURCE to lp:ubuntu-filemanager-app
Grabbing upstream libs to /home/carlos/ubuntu_work/build-add-click-deps-UbuntuSDK_for_armhf_GCC_ubuntu_sdk_15_04_vivid-Default/upstream-libs
Installing upstream libs from /home/carlos/ubuntu_work/build-add-click-deps-UbuntuSDK_for_armhf_GCC_ubuntu_sdk_15_04_vivid-Default/upstream-libs/usr/lib/arm-linux-gnueabihf/ to /lib/arm-linux-gnueabihf
Following files to install:-

-- Found PkgConfig: arm-linux-gnueabihf-pkg-config (found version "0.28")
-- samba include=/usr/include/samba-4.0
-- samba lib=/usr/lib/arm-linux-gnueabihf/libsmbclient.so=/usr/lib/arm-linux-gnueabihf/libsmbclient.so
-- Found samba: include=/usr/include/samba-4.0 library=/usr/lib/arm-linux-gnueabihf/libsmbclient.so

Questions:
   1. Did I do something wrong?
   2. Cmake still found my installed samba libraries, should not it be changed to find downloaded stuff including libsmbclient.h which I think is missed from download script?
   3. I am wondering how is the mechanism used to load local libraries in the click package, can anyone tell how it works?

review: Needs Information
Alan Pope 🍺🐧🐱 πŸ¦„ (popey) wrote :

Ok, I just checked building this branch on a nexus 7. Here's what I did to prove it builds:-

# grab this branch
bzr branch lp:~popey/ubuntu-filemanager-app/add-click-deps
# install build dependencies and cmake and phablet tools (to get click-buddy)
sudo apt-get install cmake debhelper intltool qtbase5-dev qtdeclarative5-dev qt5-default pkg-kde-tools python3-all libtag1-dev libpam0g-dev libsmbclient-dev cmake phablet-tools
# Build it:-
click-buddy --dir add-click-deps/

This grabbed the binary-built stuff which gets used at runtime, it then compiles and built filemanager.

I guess maybe when you un-installed samba libs you removed the headers needed for building?

Carlos Jose Mazieri (carlos-mazieri) wrote :

Hello Alan,

My attempt was done using the SDK armhf 15-04.

I see that you did install 'libsmbclient-dev' (it installs all the samba libraries) which I thought it was going to be downloaded by the proposed solution.

I thought that solution would bring all the necessary resources to compile and deploy on device. Any new developer will try to create the click package using SDK.

Am I wrong?

Can you try it again after removing all the samba libraries?

If so, I would suggest a sequence of commands:
sudo apt-get remove libsmbclient-dev
sudo apt-get autoremove # I hope this will uninstall all other samba libraries
click-buddy --dir add-click-deps/

If you want, can you also try it using armhf SDK?

Thanks,
Carlos

lp:~popey/ubuntu-filemanager-app/add-click-deps updated on 2015-10-13
468. By Alan Pope 🍺🐧🐱 πŸ¦„ on 2015-10-13

Merge trunk

Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot) wrote :
review: Approve (continuous-integration)
lp:~popey/ubuntu-filemanager-app/add-click-deps updated on 2015-10-14
469. By Alan Pope 🍺🐧🐱 πŸ¦„ on 2015-10-14

update get-click-deps to python3

Ubuntu Phone Apps Jenkins Bot (ubuntu-phone-apps-jenkins-bot) wrote :
review: Approve (continuous-integration)
Carlos Jose Mazieri (carlos-mazieri) wrote :

Thanks,

I am looking forward to getting the entire Samba stuff in the next release.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'CMakeLists.txt'
2--- CMakeLists.txt 2015-09-30 23:39:47 +0000
3+++ CMakeLists.txt 2015-10-14 11:21:34 +0000
4@@ -65,6 +65,23 @@
5 add_custom_target(com_ubuntu_calendar_CLICKFiles ALL SOURCES ${CLICK_FILES})
6
7 install(FILES ${CMAKE_CURRENT_BINARY_DIR}/manifest.json filemanager.apparmor ${CONTENT_HUB_EXPORTER} DESTINATION ${CMAKE_INSTALL_PREFIX})
8+
9+ MESSAGE("Grabbing upstream libs to ${CMAKE_CURRENT_BINARY_DIR}/upstream-libs")
10+ execute_process(
11+ COMMAND mkdir ${CMAKE_CURRENT_BINARY_DIR}/upstream-libs
12+ COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/get-click-deps -d ${CMAKE_CURRENT_SOURCE_DIR}/filemanager-libs.json ${CLICK_ARCH} ${CMAKE_CURRENT_BINARY_DIR}/upstream-libs
13+ )
14+ MESSAGE("Installing upstream libs from ${CMAKE_CURRENT_BINARY_DIR}/upstream-libs/usr/lib/${ARCH_TRIPLET}/ to ${DATA_DIR}lib/${ARCH_TRIPLET}")
15+ file(GLOB_RECURSE UPSTREAM_LIBS "${CMAKE_CURRENT_BINARY_DIR}/upstream-libs/usr/lib/${ARCH_TRIPLET}/*")
16+ foreach(ITEM ${UPSTREAM_LIBS})
17+ IF( IS_DIRECTORY "${ITEM}" )
18+ LIST( APPEND DIRS_TO_DEPLOY "${ITEM}" )
19+ ELSE()
20+ LIST( APPEND FILES_TO_DEPLOY "${ITEM}" )
21+ ENDIF()
22+ endforeach()
23+ MESSAGE("Following files to install:- ${FILES_TO_DEPLOY}")
24+ INSTALL( FILES ${FILES_TO_DEPLOY} DESTINATION ${DATA_DIR}lib/${ARCH_TRIPLET} )
25 else(CLICK_MODE)
26 execute_process(
27 COMMAND qmake -query QT_INSTALL_QML
28
29=== added file 'filemanager-libs.json'
30--- filemanager-libs.json 1970-01-01 00:00:00 +0000
31+++ filemanager-libs.json 2015-10-14 11:21:34 +0000
32@@ -0,0 +1,50 @@
33+{
34+ "armhf": [
35+ {
36+ "url": "http://ports.ubuntu.com/ubuntu-ports",
37+ "dist": "vivid",
38+ "component": "main",
39+ "packages": [
40+ "libsmbclient",
41+ "samba-libs",
42+ "libtalloc2",
43+ "libtevent0",
44+ "libwbclient0",
45+ "libldb1",
46+ "libntdb1"
47+ ]
48+ }
49+ ],
50+ "amd64": [
51+ {
52+ "url": "http://archive.ubuntu.com/ubuntu",
53+ "dist": "vivid",
54+ "component": "main",
55+ "packages": [
56+ "libsmbclient",
57+ "samba-libs",
58+ "libtalloc2",
59+ "libtevent0",
60+ "libwbclient0",
61+ "libldb1",
62+ "libntdb1"
63+ ]
64+ }
65+ ],
66+ "i386": [
67+ {
68+ "url": "http://archive.ubuntu.com/ubuntu",
69+ "dist": "vivid",
70+ "component": "main",
71+ "packages": [
72+ "libsmbclient",
73+ "samba-libs",
74+ "libtalloc2",
75+ "libtevent0",
76+ "libwbclient0",
77+ "libldb1",
78+ "libntdb1"
79+ ]
80+ }
81+ ]
82+}
83
84=== added file 'get-click-deps'
85--- get-click-deps 1970-01-01 00:00:00 +0000
86+++ get-click-deps 2015-10-14 11:21:34 +0000
87@@ -0,0 +1,451 @@
88+#!/usr/bin/python3
89+
90+# Copyright (C) 2015 Stefano Verzegnassi <verzegnassi.stefano@gmail.com>
91+# Copyright (C) 2015 Didier Roche <didrocks@ubuntu.com>
92+#
93+# This program is free software: you can redistribute it and/or modify it
94+# under the terms of the GNU General Public License version 3, as published
95+# by the Free Software Foundation.
96+#
97+# This program is distributed in the hope that it will be useful, but
98+# WITHOUT ANY WARRANTY; without even the implied warranties of
99+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
100+# PURPOSE. See the GNU General Public License for more details.
101+#
102+# You should have received a copy of the GNU General Public License along
103+# with this program. If not, see <http://www.gnu.org/licenses/>.
104+
105+#
106+# Source code at:
107+# https://code.launchpad.net/~verzegnassi-stefano/+junk/get-click-deps
108+#
109+# A script to automate the fetching all the external dependencies of a
110+# Click packaged application from internet, and copying them in a given
111+# folder, so that the click packaging tool can easily build them in the
112+# package.
113+#
114+# usage: get-click-deps [-h] [-f] [-d] [-e] [-c SCRIPT_PATH]
115+# manifest_path {amd64,i386,armhf} target_path
116+#
117+# A tool for adding external libraries to a Ubuntu SDK application or scope.
118+#
119+# positional arguments:
120+# manifest_path path of json file containing the list of packages to
121+# be downloaded.
122+# {amd64,i386,armhf} CPU architecture ("amd64", "i386" or "armhf")
123+# target_path path to the target (a click package or a folder)
124+# where this tool will include the downloaded binaries.
125+# If the folder does not exist, it will be created.
126+#
127+# optional arguments:
128+# -h, --help show this help message and exit
129+# -f, --force-download force a new download of the packages
130+# -d, --delete-temp delete temp files at the end of the process
131+# -e, --extract-only only create temp directory and extract the content of
132+# downloaded packages
133+# -c SCRIPT_PATH, --custom-script SCRIPT_PATH
134+# run a custom script after the extraction of Debian
135+# packages and before copying their content to the
136+# target destination. The tool will execute the script
137+# with the path to the packages dump as argument. If
138+# the '-e' flag has been specified, the script will be
139+# anyway executed. This option is useful when you need
140+# to automatically modify the content of temp folder
141+# (e.g. when you need to fix some path before including
142+# the files in a click package).
143+#
144+#
145+# USAGE EXAMPLE:
146+# get-click-deps packages.json armhf <path/to/package.click>
147+#
148+# package.json is the package manifest, and contains all the references to the
149+# .deb packages to be included into the Ubuntu SDK project.
150+# If you're familiar to the Debian/Ubuntu world, you'll see that it's pretty
151+# similar to therepository management of these distros:
152+# see https://wiki.debian.org/SourcesList for further informations.
153+#
154+# An example of packages.json file is:
155+# {
156+# "armhf": [
157+# {
158+# "url": "http://ports.ubuntu.com/ubuntu-ports/",
159+# "dist": "vivid",
160+# "component": "main",
161+# "packages": [
162+# "libgl1-mesa-glx",
163+# "libxslt1.1",
164+# "libxcb-glx0",
165+# "libxcb-dri3-0",
166+# "libxcb-present0",
167+# "libxshmfence1",
168+# "libxxf86vm1"
169+# ]
170+# },
171+#
172+# {
173+# "url": "http://ppa.launchpad.net/canonical-community/ppa/ubuntu",
174+# "dist": "vivid",
175+# "component": "main",
176+# "packages": [
177+# "libreoffice-vanilla"
178+# ]
179+# }
180+# ],
181+#
182+# "amd64": [
183+# "url": "http://ppa.launchpad.net/canonical-community/ppa/ubuntu",
184+# "dist": "vivid",
185+# "component": "main",
186+# "packages": [
187+# "libreoffice-vanilla"
188+# ]
189+# }
190+# ]
191+# }
192+#
193+# Instead of a click package, you can specify the build folder used by Ubuntu
194+# SDK for compiling the sources of your project. This way the binaries
195+# downloaded by this tool will be automatically included in the .click package
196+# the next time you ask Ubuntu SDK to create a new package.
197+#
198+
199+# TODO: Complete error handling
200+# TODO: Make target_path optional if the '-e' flag has been specified.
201+
202+import sys
203+import os
204+import stat
205+import time
206+import argparse
207+import json
208+import gzip
209+import subprocess
210+import urllib.request, urllib.error, urllib.parse
211+
212+def get_timestamp():
213+ return time.time()
214+
215+
216+def check_internet_connection():
217+ try:
218+ urllib.request.urlopen('http://www.google.com', timeout=20)
219+ return True
220+ except urllib2.error.URLError as err:
221+ pass
222+ return False
223+
224+
225+def get_arch_triplet(arch):
226+ return subprocess.check_output([
227+ 'dpkg-architecture',
228+ '-A',
229+ arch,
230+ '-qDEB_TARGET_MULTIARCH'])
231+
232+
233+def download_file(url, dest, verbose=True):
234+ if verbose:
235+ print ("\nDownloading:\n{}".format(url))
236+ # TODO: Switch to subprocess
237+ os.system('cd %s && { curl -# -O %s ; cd - ; }' % (dest, url))
238+
239+
240+def download_file_and_rename(url, dest, new_filename, verbose=True):
241+ if verbose:
242+ print ("\nDownloading:\n{}".format(url))
243+ new_path = os.path.join(dest, new_filename)
244+ subprocess.call(['curl', url, '-#', '-o', new_path])
245+
246+ return new_path
247+
248+
249+def get_package_download_url(package_name, packages_list, base_url):
250+ pkgs_list = packages_list.decode('utf-8').split('\n')
251+ index = pkgs_list.index('Package: %s' % package_name)
252+
253+ for i in range(index, len(pkgs_list)):
254+ if pkgs_list[i].find('Filename:') > -1:
255+ return "%s/%s" % (base_url, pkgs_list[i].replace('Filename: ', ''))
256+
257+
258+def get_URLs_for_arch(manifest_path, arch, destpath):
259+ urls = []
260+
261+ f = open(manifest_path)
262+ content = json.load(f)
263+ f.close()
264+
265+ # Download repository index for each repository in the JSON package
266+ # manifest.
267+ try:
268+ repo_index = 0
269+ for repo in content[arch]:
270+ repo_index_url = '%s/dists/%s/%s/binary-%s/Packages.gz' % (
271+ repo['url'],
272+ repo['dist'],
273+ repo['component'],
274+ arch)
275+
276+ print ("\nDownloading repository index at:\n".format(repo_index_url))
277+
278+ repo_index_zip = download_file_and_rename(
279+ repo_index_url,
280+ destpath,
281+ 'repo-index-%s-%s.gz' % (arch, repo_index),
282+ False)
283+
284+ with gzip.open(repo_index_zip, 'r') as f:
285+ repo_index_content = f.read()
286+ f.close()
287+
288+ # Get the download URL of each package of the repository.
289+ packages = repo['packages']
290+ for package in packages:
291+ urls.append(get_package_download_url(
292+ package,
293+ repo_index_content,
294+ repo['url']))
295+
296+ repo_index += 1
297+ except KeyError:
298+ # Arch not found in the manifest. Exit with no error, since there's no
299+ # need to run the script for this arch.
300+ print ("\n\nRequested arch has been not specified in the manifest. \
301+ Exiting...")
302+ sys.exit(0)
303+ print ("\nObtained packages informations")
304+ return urls
305+
306+
307+def check_if_temp_folder_already_exists(path):
308+ return os.path.isdir(path)
309+
310+
311+def copy_directory_content(sourcepath, destpath):
312+ subprocess.call(['cp', '-r', '%s/.' % sourcepath, destpath])
313+
314+
315+def delete_folder(path, recursive=False):
316+ if recursive:
317+ flag = '-rf'
318+ else:
319+ flag = '-f'
320+ subprocess.call(['rm', flag, path])
321+
322+
323+def extract_deb_package(deb_path, destpath):
324+ subprocess.call(['dpkg-deb', '-x', deb_path, destpath])
325+
326+
327+def extract_click_package(click_path, destpath):
328+ extract_deb_package(click_path, destpath)
329+
330+ manifest = subprocess.check_output(['click', 'info', click_path])
331+
332+ # The manifest we get has an 'installed-size' key with the value of the
333+ # previous package. Anyway this value will be replaced when we'll run
334+ # 'click build <pkg>', so there's no reason for removing it here.
335+ f = open(os.path.join(destpath, 'manifest.json'), 'w')
336+ f.write(manifest)
337+ f.close()
338+
339+ return destpath
340+
341+
342+def build_click_package(source_dirpath):
343+ output = subprocess.check_output(['click', 'build', source_dirpath])
344+
345+ for line in output.split(os.linesep):
346+ if line.find('Successfully built package in ') > -1:
347+ # FIXME: Very ugly.
348+ path = line.replace('Successfully built package in \'', '')
349+ path = path.replace('\'.', '')
350+ return path
351+
352+
353+def copy_file(sourcepath, destpath, overwrite=False):
354+ flag = ''
355+
356+ if not overwrite:
357+ flag = '-n'
358+
359+ subprocess.call(['cp', flag, sourcepath, destpath])
360+
361+
362+# Argument parser
363+parser = argparse.ArgumentParser(
364+ description="A tool for adding external libraries to a Ubuntu SDK \
365+ application or scope.")
366+
367+parser.add_argument(
368+ '-f',
369+ '--force-download',
370+ dest='force_download',
371+ action='store_true',
372+ help='force a new download of the packages')
373+
374+parser.add_argument(
375+ '-d',
376+ '--delete-temp',
377+ dest='delete_temp',
378+ action='store_true',
379+ help='delete temp files at the end of the process')
380+
381+parser.add_argument(
382+ '-e',
383+ '--extract-only',
384+ dest='extract_only',
385+ action='store_true',
386+ help='only create temp directory and extract the content of downloaded \
387+ packages')
388+
389+parser.add_argument(
390+ '-c',
391+ '--custom-script',
392+ dest='script_path',
393+ type=str,
394+ help='run a custom script after the extraction of Debian packages and \
395+ before copying their content to the target destination. The tool will \
396+ execute the script with the path to the packages dump as argument. If \
397+ the \'-e\' flag has been specified, the script will be anyway \
398+ executed. This option is useful when you need to automatically modify \
399+ the content of temp folder (e.g. when you need to fix some path \
400+ before including the files in a click package).')
401+
402+parser.add_argument(
403+ 'manifest_path',
404+ type=str,
405+ help='path of json file containing the list of packages to be downloaded.')
406+
407+parser.add_argument(
408+ 'arch',
409+ type=str,
410+ choices=['amd64', 'i386', 'armhf'],
411+ help='CPU architecture ("amd64", "i386" or "armhf")')
412+
413+parser.add_argument(
414+ 'target_path',
415+ type=str,
416+ help='path to the target (a click package or a folder) where this \
417+ tool will include the downloaded binaries. If the folder does not \
418+ exist, it will be created.')
419+
420+args = parser.parse_args()
421+
422+# Variables
423+manifest_path = args.manifest_path
424+target_path = args.target_path
425+manifest_stat = os.stat(manifest_path)
426+temp_folder = os.path.join(
427+ '/tmp/',
428+ 'tmp-click-deps-%s-%s-%s-%s' % (
429+ manifest_stat.st_dev,
430+ manifest_stat.st_ino,
431+ manifest_stat.st_size,
432+ manifest_stat.st_mtime))
433+temp_arch_folder = os.path.join(temp_folder, args.arch)
434+is_click_target = os.path.isfile(args.target_path)
435+
436+# Check command line arguments
437+if not os.path.isfile(manifest_path):
438+ sys.exit("\n\nERROR: Package manifest is not a valid file. Exit...")
439+
440+if not os.path.isfile(target_path) and target_path.endswith('.click'):
441+ sys.exit("\n\nERROR: The specified target .click does not exists.")
442+elif not os.path.isdir(target_path):
443+ print ("\n\nCreating dest folder\n{}".format(target_path))
444+ os.mkdir(target_path)
445+
446+# Check if the script exist, if specified any.
447+if args.script_path and not os.path.exists(args.script_path):
448+ sys.exit("\n\nERROR: The specified script does not exists.")
449+
450+# If -f argument has been specified, remove all the existing data before
451+# running this script.
452+if args.force_download:
453+ if os.path.isdir(temp_folder):
454+ print ("\nRemoving temp data of the previous run, as requested")
455+ delete_folder(temp_folder, True)
456+
457+# Check if we already have run this script for the same target.
458+if not check_if_temp_folder_already_exists(temp_arch_folder):
459+ # Check internet connection
460+ if not check_internet_connection():
461+ sys.exit("\n\nERROR: An internet connection is required in order to \
462+ download packages from repositories.")
463+
464+ # Create temp folder in /tmp
465+ print ("\nCreating temp folder in {}".format(temp_folder))
466+ os.mkdir(temp_folder)
467+ os.mkdir(temp_arch_folder)
468+
469+ # Parse the JSON package list and get the download URL of the packages.
470+ debs_url_list = get_URLs_for_arch(
471+ manifest_path,
472+ args.arch,
473+ temp_folder)
474+
475+ # Download packages from web
476+ for url in debs_url_list:
477+ download_file(url, temp_folder)
478+
479+ # Extract DEBs packages
480+ print ("\nExtracting .deb packages to {}".format(temp_arch_folder))
481+ deb_pkgs_list = []
482+ for file in os.listdir(temp_folder):
483+ if file.endswith('.deb'):
484+ deb_pkgs_list.append(file)
485+
486+ for deb_pkg in deb_pkgs_list:
487+ extract_deb_package(
488+ os.path.join(temp_folder, deb_pkg),
489+ temp_arch_folder)
490+
491+ # If a script has been specified, run it.
492+ if args.script_path:
493+ print ("\nRunning the script at: {}\n".format(args.script_path))
494+
495+ # Ensure that we can run the script, otherwise we don't have the
496+ # permission
497+ subprocess.call(['chmod', '+x', args.script_path])
498+ subprocess.call([
499+ args.script_path,
500+ temp_arch_folder,
501+ get_arch_triplet(args.arch)])
502+else:
503+ print ("\nPackages are already downloaded. Use them...")
504+
505+# If -e or --extract-only flags have been specified, we have completed our work
506+if args.extract_only:
507+ print ("\n\nCompleted successfully.")
508+ sys.exit(0)
509+
510+# Copy temp_arch folder content to its destination
511+if is_click_target:
512+ print ("\nExtracting target .click package")
513+ temp_click_folder = extract_click_package(
514+ args.target_path,
515+ os.path.join('/tmp/', str(get_timestamp())))
516+
517+ print ("\nAdding extracted binaries to the package")
518+ copy_directory_content(temp_arch_folder, temp_click_folder)
519+
520+ new_click_package_path = build_click_package(temp_click_folder)
521+ print ("\nCreated new .click package at:\n{}".format(new_click_package_path))
522+
523+ print ("\nReplacing older .click package")
524+ copy_file(new_click_package_path, args.target_path, True)
525+
526+ delete_folder(temp_click_folder, True)
527+
528+else:
529+ print ("\nCopying extracted binaries to their destination")
530+ copy_directory_content(temp_arch_folder, args.target_path)
531+
532+# Delete temp files
533+if args.delete_temp:
534+ print ("\)nRemoving temp files and directory, as requested")
535+ delete_folder(temp_folder, True)
536+
537+# Exit
538+print ("\n\nCompleted successfully.")

Subscribers

People subscribed via source and target branches