Merge lp:~longsleep/snapcraft/snapcraft-debs-plugin into lp:~snappy-dev/snapcraft/core

[Simon Eisenmann]

Adds a debs plugin which allows to directly use .deb files either from local file or URL.

[lp:~longsleep/snapcraft/snapcraft-debs-plugin]

116. By Leo Arias on 2015-08-03

Send error logs to stderr and message logs to stdout. by elopio approved by mterry

[Michael Terry (mterry)]

Neat, thanks! But what's the use case for this? Pointing at a PPA's deb file? Grabbing debs from Debian directly?

It seems fragile to point to a single deb on a server since when the repository gets updated, the url will be dead.

Maybe it would be better to add PPA support to the ubuntu plugin?

[Simon Eisenmann (longsleep)]

The use case is to have reproduceable builds of defined and we'll known deb files, independent of local system arch, dist an version. I guess one could go so far to even comitt deb files along with the snapcraft file. In my case this makes it simple to build armhf snaps on amd64 from defined builds done by pbuilder. For reference please also see https://bugs.launchpad.net/bugs/1480144

review: Needs Fixing

[lp:~longsleep/snapcraft/snapcraft-debs-plugin]

117. By Michael Terry on 2015-08-04

Include the multiarch triplet to LD_LIBRARY_PATH and set the 'architecture' key in package.yaml if not present for the user. by mterry approved by rsalveti

118. By Michael Terry on 2015-08-04

Add intro and tutorial docs.

The intro file explains Snappy and the general Snapcraft concepts a bit. It covers the same material as flow.md, so I dropped that file.

The tutorial file goes through crafting a webcam demo snap [1].

I want to add an installing/testing/uploading-to-store section too, but I want to get some experience using the webcam bit and dealing with apparmor for it (seeing if I can get kvm to use my laptop webcam or if I need to have a real USB webcam with a beaglebone). But that addition can happen in a separate branch.

This is taking over from lp:~mvo/snapcraft/docs-1 since mvo is on vacation.

[1] Very similar to https://developer.ubuntu.com/en/snappy/guides/appliance-builder-guide-webcam/ by snappy-dev approved by elopio

119. By Leo Arias on 2015-08-04

Added unit tests for the base plugin.
Added python3-testscenarios and wget as a dependency. by elopio approved by mterry

120. By Michael Terry on 2015-08-04

Add python2 and python2-project plugins.

I did this branch when testing the python3 branch (to make sure there was no python bits on the system interfering with the bundled python).

It's basically a straight copy of python3, with appropriate string changes.

I also dropped the configflags option from python3-project. I think that was a copy/paste error. by mterry approved by elopio,sergiusens

121. By Daniel Holbach on 2015-08-05

Add some next steps. by dholbach approved by sergiusens,longsleep

122. By Daniel Holbach on 2015-08-05

Small doc markup fixes. by dholbach approved by sergiusens,longsleep

123. By Daniel Holbach on 2015-08-05

Rename docs/tutorial.md to be less generic. by dholbach approved by elopio

124. By Leo Arias on 2015-08-05

Refactored the plugin makedirs, with tests. Added python3 >= 3.2 as a dependency, needed by makedirs exist_ok argument. by elopio approved by mterry

125. By Leo Arias on 2015-08-05

Fixed the call to the log from the ubuntu plugin, with a test. by elopio approved by mterry

[Michael Terry (mterry)]

Some comments

[lp:~longsleep/snapcraft/snapcraft-debs-plugin]

126. By Ted Gould on 2015-08-05

QML Plugin by ted approved by mterry

127. By Michael Terry on 2015-08-05

Tiny packaging fixes

128. By Sergio Schvezov on 2015-08-06

Adding support for mercurial sources (includes debian/control dependency change). by sergiusens approved by elopio

129. By Sergio Schvezov on 2015-08-06

Recommends defaulting to None when not part of options by sergiusens approved by elopio

[Simon Eisenmann (longsleep)]

Rebased on latest upstream and updates to match comments from mterry.

review: Resubmit

[Michael Terry (mterry)]

This feels better, thanks. :) Less code duplication. Some further comments though.

[lp:~longsleep/snapcraft/snapcraft-debs-plugin]

130. By Sergio Schvezov on 2015-08-08

debian/control: adding devscripts as a dep as dget is needed by the ubuntu plugin by sergiusens approved by ogra

131. By Simon Eisenmann on 2015-08-10

Fix symlinks for directories as well. by longsleep approved by elopio,mterry

132. By Michael Terry on 2015-08-14

Support the Go compiler on all arches, not just amd64.

This switches from using Google's pre-built binaries to using our archive builds of Google's compiler, thus giving us all platforms we support. And matching what we do for Python and Java.

Since this means we use whichever version of Go the host platform has, I've dropped the "1.4" from the plugin names.

I've tested this on wily and trusty. by mterry approved by sergiusens

133. By Sergio Schvezov on 2015-08-15

go as build-deps instead of plugged by sergiusens approved by chipaca

134. By Loïc Minier on 2015-08-20

Add initial and trivial mvn plugin; runs "mvn package" and copies resulting *.jar and *.war files. by lool approved by sergiusens

135. By Zygmunt Krynicki on 2015-08-26

Correct logging setup by zyga approved by mvo

136. By Zygmunt Krynicki on 2015-08-26

Ensure that snapcraft run kills qemu by zyga approved by mvo

137. By Zygmunt Krynicki on 2015-08-26

Correct all the logging calls to pass arguments directly by zyga approved by mvo

138. By Zygmunt Krynicki on 2015-08-26

Fix numerous issues in snapcraft run by zyga approved by mvo

139. By Zygmunt Krynicki on 2015-08-26

Add support for SNAPCRAFT_RUN_QEMU_ARGS by zyga approved by mvo

140. By Sergio Schvezov on 2015-08-27

Generate meta (package.yaml and readme.md) from snapcraft.yaml by sergiusens approved by chipaca,elopio

141. By Sergio Schvezov on 2015-08-27

Initial json schema support by sergiusens approved by chipaca,elopio

142. By Sergio Schvezov on 2015-08-28

Multiple testing additions by sergiusens approved by elopio

143. By Sergio Schvezov on 2015-08-28

icon support for snapcraft.yaml by sergiusens approved by chipaca

144. By Sergio Schvezov on 2015-08-28

Reducing mccabe complexity from 10 to 3 for PluginHandler.collect_snap_files by sergiusens approved by elopio

145. By Daniel Holbach on 2015-09-01

Open 0.2 for development, so the daily build (https://code.launchpad.net/~snappy-dev/+recipe/snapcraft-daily) can trump what's in the archive. by dholbach approved by mvo

146. By Sergio Schvezov on 2015-09-01

Moving source management complexity out of the base plugin by sergiusens approved by sergiusens

147. By Sergio Schvezov on 2015-09-01

Unit tests for sources by sergiusens approved by sergiusens,mvo

148. By Sergio Schvezov on 2015-09-01

Remove embedded functions from snapcraft.cmd.run by sergiusens approved by sergiusens,mvo

149. By Daniel Holbach on 2015-09-02

Fix syntax error in debian/control by dholbach approved by mvo

150. By Daniel Holbach on 2015-09-02

To use the ant-project plugin, you need to have ant installed as a build-tool. by dholbach approved by mvo

151. By Michael Nelson on 2015-09-07

Fix snapcraft.yaml's reference to go-package plugin to match changes in r132. by michael.nelson approved by sergiusens

152. By Ted Gould on 2015-09-07

Add demo.qml back to the snapcraft.yaml for the QML Demo by ted approved by sergiusens,dholbach

153. By Daniel Holbach on 2015-09-08

Update tomcat upstream URL, fixes example again. by dholbach approved by sergiusens

154. By Sergio Schvezov on 2015-09-09

Replace Ubuntu plugin with a builtin solution by sergiusens approved by mvo

155. By Sergio Schvezov on 2015-09-09

Simplifying snapcraft.yaml.Config.__init__ by sergiusens approved by elopio

156. By Sergio Schvezov on 2015-09-09

Provide a nice error when tabs found in snapcraft.yaml by sergiusens approved by ted

157. By Sergio Schvezov on 2015-09-09

Plugins are of a type by sergiusens approved by ted

158. By Björn Tillenius on 2015-09-14

Add support for setuptools based python projects. by bjornt approved by sergiusens,ted

159. By Sergio Schvezov on 2015-09-14

Filesets based filtering for stage and snap by sergiusens approved by sergiusens,mvo

160. By Sergio Schvezov on 2015-09-14

snapcraft init with templated values by sergiusens approved by mvo

161. By Sergio Schvezov on 2015-09-14

regex for binary and service names by sergiusens approved by mvo

162. By Sergio Schvezov on 2015-09-14

docs refresh by sergiusens approved by chipaca

163. By Sergio Schvezov on 2015-09-14

Collision logic updates with an introduction of _BUILTIN_OPTIONS so all parts have a certain set of options by default even if not declared. by sergiusens approved by mvo

164. By Daniel Holbach on 2015-09-15

fix crash when stage_packages is defined in the yaml for the plugin, but empty (getattr does not use [] as default return value in that case) by dholbach approved by sergiusens

165. By Daniel Holbach on 2015-09-15

Fix small typo. by dholbach approved by sergiusens

166. By Sergio Schvezov on 2015-09-15

Support config by sergiusens approved by sergiusens,chipaca

167. By Sergio Schvezov on 2015-09-16

Try to use wiki when after in part is not local by sergiusens approved by sergiusens,chipaca,dholbach

168. By Daniel Holbach on 2015-09-16

Depend and Build-Depend on python3-requests. by dholbach approved by sergiusens

169. By Sergio Schvezov on 2015-09-16

apparmor and seccomp by sergiusens approved by chipaca

170. By Sergio Schvezov on 2015-09-17

Fetch from the wiki only once by sergiusens approved by chipaca,dholbach

171. By Sergio Schvezov on 2015-09-17

Robustness for stage and snap which follow the spec more closely now by sergiusens approved by chipaca

172. By Sergio Schvezov on 2015-09-17

Conflicting files that are essentially the same file should not cause problems by sergiusens approved by chipaca

173. By Daniel Holbach on 2015-09-17

Add autopkgtest, reshuffle build-deps accordingly and run wrap-and-sort. by dholbach approved by elopio,sergiusens

174. By Sergio Schvezov on 2015-09-17

Fixing pep issues with bin/snapcraft by sergiusens approved by elopio

175. By Sergio Schvezov on 2015-09-17

Add a clean command by sergiusens approved by chipaca

176. By Sergio Schvezov on 2015-09-17

setup.py using pypy to setup a config integration test by sergiusens approved by elopio

177. By Daniel Holbach on 2015-09-18

add libgudev-1.0-dev as build-tools for godd example by dholbach approved by sergiusens

178. By Ted Gould on 2015-09-18

Add in support for PIP requirements.txt files in the python plugins. by ted approved by sergiusens

179. By Zygmunt Krynicki on 2015-09-18

Ignore __pycache__ by zyga approved by sergiusens,dholbach

180. By Sergio Schvezov on 2015-09-18

build-packages instead of build-tools to follow the spec by sergiusens approved by mvo

181. By Leo Arias on 2015-09-18

On clean, check the contents of the parts dir only if it exists. by elopio approved by sergiusens

182. By Sergio Schvezov on 2015-09-18

Implementation of the organize keyword from the spec by sergiusens approved by sergiusens,ted

183. By Ted Gould on 2015-09-18

Wrap setup.py calls to configure the shebang writer by ted approved by sergiusens

184. By Sergio Schvezov on 2015-09-21

Config hook examples update by sergiusens approved by sergiusens,dholbach

185. By Sergio Schvezov on 2015-09-21

Using a map instead of list for binaries and services as defined by the spec by sergiusens approved by sergiusens

186. By Ted Gould on 2015-09-21

Improving stage-package handling by snappy-dev approved by mvo

187. By Sergio Schvezov on 2015-09-21

Used builddir as a base for setup.py overrides by sergiusens approved by ted

188. By Sergio Schvezov on 2015-09-22

Use relative paths for image creation by sergiusens approved by elopio

189. By Sergio Schvezov on 2015-09-22

Notify user why the password is being requested by sergiusens approved by elopio

190. By Sergio Schvezov on 2015-09-22

Load the config before trying to run by sergiusens approved by elopio

191. By Sergio Schvezov on 2015-09-22

Remove modules from plugins and infer it from the file name to follow the spec by sergiusens approved by elopio

192. By Sergio Schvezov on 2015-09-22

Enable ports for architectures that are not amd64 or i386 by sergiusens approved by ogra

193. By Sergio Schvezov on 2015-09-22

Local plugins as fallback instead of special casing to follow the spec by sergiusens approved by sergiusens,elopio

194. By Michael Vogt on 2015-09-22

Fetch all packages in a single download run with proper progress by mvo approved by dholbach

195. By Michael Vogt on 2015-09-22

stage package libgudev-1.0-0, which makes the godd snap actually work by dholbach approved by sergiusens

196. By Sergio Schvezov on 2015-09-22

Update docs to reflect list to dictionary change for services example by sergiusens approved by elopio

197. By Federico Gimenez on 2015-09-22

plainbox test for building the examples by fgimenez approved by dholbach

198. By Leo Arias on 2015-09-22

Fixed the python depends on the snapcraft-examples package. by elopio approved by sergiusens

199. By Ted Gould on 2015-09-22

Choose the python path based on the version installed by ted approved by sergiusens

200. By Zygmunt Krynicki on 2015-09-23

Add 'simple' and 'has-leftovers' flags to all integration tests job units. by zyga approved by dholbach,elopio

201. By Michael Vogt on 2015-09-23

Add quotes around $TEST_PLAN and pass the $1 directly to run_test_plan instead of indirectly in runtests.sh by mvo approved by sergiusens,fgimenez,dholbach

202. By Sergio Schvezov on 2015-09-23

Make the py 2 and 3 projects smarter and have dependencies pulled during the pull phase by sergiusens approved by chipaca

203. By Sergio Schvezov on 2015-09-23

do NOT print in tests by sergiusens approved by fgimenez

204. By Sergio Schvezov on 2015-09-23

Show skipped packages only once per filter by sergiusens approved by chipaca

205. By Ted Gould on 2015-09-23

Ensure C library configuration tools don't use system paths by ted approved by sergiusens

206. By Michael Vogt on 2015-09-23

The go plugin should only download in the pull phase by mvo approved by sergiusens

207. By Sergio Schvezov on 2015-09-23

preparing a changelog by sergiusens approved by sergiusens,dholbach

208. By Sergio Schvezov on 2015-09-24

Allow using local sources through an env var (mostly useful for launchpad building) by sergiusens approved by mvo,dholbach,cjwatson

209. By Daniel Holbach on 2015-09-24

Open 0.3 for development. by dholbach approved by sergiusens

210. By Daniel Holbach on 2015-09-25

Brown paperbag release (install examples into the right directory). by dholbach approved by mvo

211. By Daniel Holbach on 2015-09-25

Reopen 0.3 for development (after 0.2.1 brown paperbag release). by dholbach approved by sergiusens

212. By Leo Arias on 2015-09-28

Fixed the heading on the build section of the docs. by elopio approved by dholbach

213. By Sergio Schvezov on 2015-09-29

Mask tar files so they are writable and set an extraction parameter to start with a clean build dir by sergiusens approved by sergiusens,mvo

214. By Sergio Schvezov on 2015-09-29

Auto detect tar sources by sergiusens approved by sergiusens,zyga

215. By Sergio Schvezov on 2015-09-29

Adding a simple scons plugin by sergiusens approved by chipaca

216. By Zygmunt Krynicki on 2015-09-29

Make integration tests easier to write by zyga approved by sergiusens

217. By Sergio Schvezov on 2015-09-29

Revert type to plugin by sergiusens approved by sergiusens,zyga

218. By Daniel Holbach on 2015-09-29

add a tour which details and explains relevant snapcraft features in the examples directory by dholbach approved by elopio

219. By Sergio Schvezov on 2015-09-29

Allow parameters/options for scons by sergiusens approved by elopio

220. By Colin Watson on 2015-09-29

Apply better logging and progress behaviour if stdout is not a tty. by cjwatson approved by sergiusens,mvo

221. By Sergio Schvezov on 2015-09-30

Support framework policies by sergiusens approved by mvo

222. By Sergio Schvezov on 2015-09-30

Use the series of the host for stage-packages. by sergiusens approved by chipaca

223. By Sergio Schvezov on 2015-09-30

Do not infer a plugin from the part name by sergiusens approved by sergiusens,chipaca

224. By Sergio Schvezov on 2015-09-30

Backwards compatibility for using type by sergiusens approved by mvo

225. By Sergio Schvezov on 2015-10-01

Remove the -project from plugin names that have been released and add a deprecation message by sergiusens approved by chipaca

226. By Leo Arias on 2015-10-02

Added the make plugin name in the py2-project example. by elopio approved by sergiusens

227. By Sergio Schvezov on 2015-10-02

Add a buildpackages for libpython2.7-dev by sergiusens approved by elopio

228. By Leo Arias on 2015-10-02

Return to the branch root to make the test coverage report. by elopio approved by dholbach

229. By Sergio Schvezov on 2015-10-03

Improving error message when missing the plugin statement by sergiusens approved by elopio

230. By Yasushi SHOJI on 2015-10-03

Support multiple supported-versions in py3versions by sergiusens approved by chipaca

231. By Sergio Schvezov on 2015-10-04

Build packages failing to install on the host or not found should be the cause of of a failure by sergiusens approved by chipaca

232. By Daniel Holbach on 2015-10-05

Fix markdown indentation, make pep8 and pyflakes3 happy. by elopio approved by elopio

233. By Sergio Schvezov on 2015-10-08

Updating tomcat pull version in examples by sergiusens approved by dholbach

234. By Sergio Schvezov on 2015-10-08

Copy links to packages we did not download and also untarring not world writable. by sergiusens approved by chipaca

[Simon Eisenmann (longsleep)]

Synced and rebased with latest snapcraft trunk. Going to add an example next.

[Simon Eisenmann (longsleep)]

This plugin needs an example.

review: Needs Fixing

[lp:~longsleep/snapcraft/snapcraft-debs-plugin]

235. By Sergio Schvezov on 2015-10-09

Removing suid/guid bits from extracted debs by sergiusens approved by chipaca

236. By Sergio Schvezov on 2015-10-13

Plugin yaml's are obsolete by sergiusens approved by elopio,chipaca

237. By Sergio Schvezov on 2015-10-13

debian/changelog for 0.3 by sergiusens approved by elopio

238. By Daniel Holbach on 2015-10-14

Open 0.4 for development. by dholbach approved by sergiusens

239. By Sergio Schvezov on 2015-10-14

Run unit tests when building the deb and make a failing test work by sergiusens approved by fgimenez,chipaca

240. By Sergio Schvezov on 2015-10-14

Exit cleanly with an error when using the source keyword with a local file by sergiusens approved by elopio

241. By Sergio Schvezov on 2015-10-15

Search the wiki if the plugin keyword is missing for a part. Tests related to wiki searches are improved by sergiusens approved by elopio

242. By Sergio Schvezov on 2015-10-16

debian/control: adding python3-requests to Build-Depends by sergiusens approved by chipaca

243. By Ted Gould on 2015-10-16

Adding a catkin plugin by ted approved by sergiusens

244. By Ted Gould on 2015-10-16

Make sure there is a directory before trying to copy to it. by ted approved by sergiusens

245. By Ted Gould on 2015-10-19

Fixes needed for some autotools builds by ted approved by sergiusens

246. By Daniel Holbach on 2015-10-19

add python3-autopilot as build-depends by dholbach approved by sergiusens

247. By Sergio Schvezov on 2015-10-19

Document and cleanup the plugin base class

This cleans up the base class to only expose things plugins will need, since the basic schema includes 'source' options it makes sense for pull to implement that as well automatically if supported.

This also moves everything from the base plugin the plugin.py (which should be renamed to lifecycle.py in a future MP, this one is already too big) and any other thing to its own supporting class.

This branch also includes all the plugin cleanup required for things to keep on working. by sergiusens approved by elopio,chipaca

248. By Sergio Schvezov on 2015-10-20

Rename snapcraft.plugin to snapcraft.lifecycle by sergiusens approved by elopio

249. By Sergio Schvezov on 2015-10-20

Add support for maven options by sergiusens approved by chipaca

250. By Sergio Schvezov on 2015-10-20

Query the apt cache to verify if a package is installed or not by sergiusens approved by mvo

251. By Daniel Holbach on 2015-10-22

Subtle first-sight readability changes on intro doc. by dholbach approved by elopio

252. By Sergio Schvezov on 2015-10-26

Plugins raise exception instead of true/false by sergiusens approved by elopio

253. By Sergio Schvezov on 2015-10-26

Add list-plugins by sergiusens approved by elopio,chipaca

254. By Simon Eisenmann on 2015-10-27

Implemented debs plugin to directly use debs files without local apt.

255. By Simon Eisenmann on 2015-10-27

Added debs plugin to test.

256. By Simon Eisenmann on 2015-10-27

Updated to latest snapcraft.

Unmerged revisions

256. By Simon Eisenmann on 2015-10-27

Updated to latest snapcraft.

255. By Simon Eisenmann on 2015-10-27

Added debs plugin to test.

254. By Simon Eisenmann on 2015-10-27

Implemented debs plugin to directly use debs files without local apt.

=== added file 'snapcraft/plugins/debs.py'
--- snapcraft/plugins/debs.py	1970-01-01 00:00:00 +0000
+++ snapcraft/plugins/debs.py	2015-10-27 14:24:07 +0000
@@ -0,0 +1,74 @@
+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
+#
+# Copyright (C) 2015 struktur AG
+# Copyright (C) 2015 Canonical Ltd
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import urllib.parse
+import snapcraft
+
+from snapcraft.repo import _fix_contents
+from snapcraft.common import isurl
+
+
+class DebsPlugin(snapcraft.BasePlugin):
+
+    @classmethod
+    def schema(cls):
+        return {
+            'properties': {
+                'debs': {
+                    'type': 'array',
+                },
+            },
+            'required': [
+                'debs',
+            ]
+        }
+
+    def __init__(self, name, options):
+        super().__init__(name, options)
+        packages = set()
+        debs = set()
+        for source in sorted(self.options.debs):
+            if isurl(source):
+                filename = os.path.split(urllib.parse.urlparse(source).path)[1]
+            else:
+                # Local file.
+                filename = source
+            packages.add(filename)
+            debs.add(source)
+        self.packages = sorted(packages)
+        self.debs = sorted(debs)
+
+    def pull(self):
+        for source in self.debs:
+            self.pull_file(source, destdir=self.sourcedir)
+
+    def build(self):
+        self.unpack_debs(self.packages, self.installdir, debdir=self.sourcedir)
+        _fix_contents(debdir=self.installdir)
+
+    def pull_file(self, source, destdir=None):
+        if isurl(source):
+            self.run(['wget', '-q', '-c', source], cwd=destdir)
+        else:
+            self.run(['cp', os.path.join(os.getcwd(), source), destdir])
+
+    def unpack_debs(self, pkgs, targetDir, debdir=None):
+        debdir = debdir or self.builddir
+
+        for p in pkgs:
+            self.run(['dpkg-deb', '--extract', p, targetDir], cwd=debdir)

=== modified file 'snapcraft/tests/test_cmds.py'
--- snapcraft/tests/test_cmds.py	2015-10-26 15:54:30 +0000
+++ snapcraft/tests/test_cmds.py	2015-10-27 14:24:07 +0000
@@ -87,6 +87,7 @@
 catkin
 cmake
 copy
+debs
 go
 jdk
 make