usd-importer

Merge ~nacc/usd-importer:lp1717965-lint-applied-branches into usd-importer:master

  1. Git
  2. lp:~nacc/usd-importer
  3. lp1717965-lint-applied-branches
  4. Merge into master
Proposed by Nish Aravamudan on 2017-10-11
Status: Merged
Merged at revision: 262a939c67cd2913867966b7f5b368b0a45b5c47
Proposed branch: ~nacc/usd-importer:lp1717965-lint-applied-branches
Merge into: usd-importer:master
Diff against target: 2730 lines (+1245/-463)
23 files modified
doc/README.md (+1/-1)
doc/gitubuntu-completion.sh (+3/-0)
gitubuntu/__main__.py (+1/-1)
gitubuntu/build.py (+278/-192)
gitubuntu/buildsource.py (+25/-20)
gitubuntu/git_repository.py (+546/-41)
gitubuntu/importer.py (+98/-77)
gitubuntu/importppa.py (+8/-0)
gitubuntu/lint.py (+29/-4)
gitubuntu/merge.py (+11/-12)
gitubuntu/remote.py (+128/-31)
gitubuntu/review.py (+4/-0)
gitubuntu/run.py (+31/-14)
gitubuntu/versioning.py (+14/-5)
man/man1/git-ubuntu-build-source.1 (+8/-1)
man/man1/git-ubuntu-build.1 (+8/-1)
man/man1/git-ubuntu-import-ppa.1 (+16/-1)
man/man1/git-ubuntu-import.1 (+16/-1)
man/man1/git-ubuntu.1 (+1/-1)
setup.py (+1/-0)
snap/snapcraft.yaml (+4/-60)
snap/wrappers/git-merge-changelogs (+7/-0)
snap/wrappers/git-reconstruct-changelog (+7/-0)
Reviewer Review Type Date Requested Status
Server Team CI bot continuous-integration Approve on 2017-10-26
Robie Basak 2017-10-11 Pending
Review via email: mp+332160@code.launchpad.net

Description of the Change

Make Jenkins happy.

This resolves several first-time user issues with the build/buildsource code, and allows us to toggle the default branch for clone to be patches-applied.

To post a comment you must log in.
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:2b64fa48d7e149c40179065753d520224572f394
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/142/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    FAILED: Integration Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/142/rebuild

review: Needs Fixing (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:79700c896319f16864b9b30cb7ca6002fd4f1b45
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/144/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    FAILED: Integration Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/144/rebuild

review: Needs Fixing (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:79700c896319f16864b9b30cb7ca6002fd4f1b45
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/145/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    FAILED: Integration Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/145/rebuild

review: Needs Fixing (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:41bd8600400ebf1ad1ab1cc8f5dfc571c95c2ecc
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/146/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    FAILED: Integration Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/146/rebuild

review: Needs Fixing (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:ece83566572b7c8f9bf89071ce63db30d403a6b7
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/147/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    FAILED: Integration Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/147/rebuild

review: Needs Fixing (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:98c5c3c522fbd91894ffaa98b9752093bc407636
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/149/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    FAILED: Integration Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/149/rebuild

review: Needs Fixing (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:9657a342abe8f8d21cb594197d62e16368f7401e
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/150/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    FAILED: Integration Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/150/rebuild

review: Needs Fixing (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:e7e4018d118d0ae57b70fa1f0650189d6d21c620
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/151/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    FAILED: Integration Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/151/rebuild

review: Needs Fixing (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:9fed1c0078c04a87a31e993625291961ab9ff5e5
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/153/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    FAILED: Integration Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/153/rebuild

review: Needs Fixing (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:0ea422567c9180f1b46cb67ec6988216baf8a2c8
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/154/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/154/rebuild

review: Approve (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:c216f0968a57217a096ac00fa5511891c356b951
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/155/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/155/rebuild

review: Approve (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:72c44168170b9b4860c599b2e7a22ed44874c963
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/157/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    FAILED: Integration Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/157/rebuild

review: Needs Fixing (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:72c44168170b9b4860c599b2e7a22ed44874c963
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/158/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/158/rebuild

review: Approve (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:a8da6ba278c6614213f05f6a0cda77b4849afcb6
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/171/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Style Check

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/171/rebuild

review: Needs Fixing (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:5ffa1cee5eadd53e37e7cf177f4ee3282936955e
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/176/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    FAILED: Integration Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/176/rebuild

review: Needs Fixing (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:5ffa1cee5eadd53e37e7cf177f4ee3282936955e
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/178/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/178/rebuild

review: Approve (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:60ffc3cc1beb95337fb8a07ff06c7dd49b0ec9ef
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/179/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    FAILED: Integration Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/179/rebuild

review: Needs Fixing (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:faee1277ab42214121b76f53bda4c2b285bc6983
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/181/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/181/rebuild

review: Approve (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:0ff756c254236558c6c0e70eda264e068a6c199d
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/182/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/182/rebuild

review: Approve (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:ce1f0689665e555eddd4d7300fee63761a3c3764
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/183/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    FAILED: Integration Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/183/rebuild

review: Needs Fixing (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:0036188e37d392efb14febeeb9ebdb46efccab04
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/184/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/184/rebuild

review: Approve (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:ce1f0689665e555eddd4d7300fee63761a3c3764
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/185/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/185/rebuild

review: Approve (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:0036188e37d392efb14febeeb9ebdb46efccab04
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/186/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/186/rebuild

review: Approve (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:262a939c67cd2913867966b7f5b368b0a45b5c47
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/187/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/187/rebuild

review: Approve (continuous-integration)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/doc/README.md b/doc/README.md
2index f525112..0e33343 100644
3--- a/doc/README.md
4+++ b/doc/README.md
5@@ -21,7 +21,7 @@ or
6 3. Get necessary dependencies
7
8 $ sudo apt update -qy
9- $ deps="dpkg-dev git-buildpackage python3-argcomplete python3-lazr.restfulclient python3-debian python3-distro-info python3-launchpadlib python3-pygit2 python3-ubuntutools python3-pkg-resources python3-pytest quilt"
10+ $ deps="dpkg-dev git-buildpackage python3-argcomplete python3-lazr.restfulclient python3-debian python3-distro-info python3-launchpadlib python3-pygit2 python3-ubuntutools python3-pkg-resources python3-pytest python3-petname quilt"
11 $ sudo apt install -qy $deps
12
13 ## Getting via snap ##
14diff --git a/doc/gitubuntu-completion.sh b/doc/gitubuntu-completion.sh
15index 5449b54..4f1d9a2 100644
16--- a/doc/gitubuntu-completion.sh
17+++ b/doc/gitubuntu-completion.sh
18@@ -43,6 +43,7 @@ _git_ubuntu ()
19 case "$subcommand,$cur" in
20 build,--*)
21 __gitcomp "
22+ --commitish
23 --dl-cache
24 --for-merge
25 --keep-build-env
26@@ -58,6 +59,7 @@ _git_ubuntu ()
27 ;;
28 build-source,--*)
29 __gitcomp "
30+ --commitish
31 --dl-cache
32 --for-merge
33 --keep-build-env
34@@ -81,6 +83,7 @@ _git_ubuntu ()
35 import,--*)
36 __gitcomp "
37 --active-series-only
38+ --allow-applied-failures
39 -d --directory
40 --dl-cache
41 --fixup-devel
42diff --git a/gitubuntu/__main__.py b/gitubuntu/__main__.py
43index ea3bcba..867b581 100644
44--- a/gitubuntu/__main__.py
45+++ b/gitubuntu/__main__.py
46@@ -24,7 +24,7 @@ TopLevelDefaults = namedtuple(
47 )
48 top_level_defaults = TopLevelDefaults(
49 verbose=False,
50- retries=3,
51+ retries=5,
52 retry_backoffs = [2 ** i for i in range(3)],
53 proto='https',
54 parentfile=pkg_resources.resource_filename(
55diff --git a/gitubuntu/build.py b/gitubuntu/build.py
56index 1eb8b5f..6c86374 100644
57--- a/gitubuntu/build.py
58+++ b/gitubuntu/build.py
59@@ -30,17 +30,11 @@ import sys
60 import tempfile
61 import time
62 import traceback
63-import uuid
64 from gitubuntu.__main__ import top_level_defaults
65 from gitubuntu.cache import CACHE_PATH
66 from gitubuntu.dsc import GitUbuntuDsc
67-from gitubuntu.git_repository import (
68- Changelog,
69- GitUbuntuRepository,
70- MultiplePristineTarFoundError,
71- PristineTarNotFoundError,
72-)
73 from gitubuntu.lint import derive_target_branch
74+import gitubuntu.git_repository
75 from gitubuntu.run import decode_binary, run, runq
76 from gitubuntu.source_information import (
77 GitUbuntuSourceInformation,
78@@ -51,6 +45,10 @@ from gitubuntu.source_information import (
79 try:
80 pkg = 'python3-debian'
81 from debian.debfile import PART_EXTS
82+ pkg = 'python3-petname'
83+ import petname
84+ pkg = 'python3-pygit2'
85+ import pygit2
86 pkg = 'python3-pytest'
87 import pytest
88 except ImportError:
89@@ -265,7 +263,7 @@ def fetch_orig_from_pristine_tar(changelog, source, repo):
90 changelog.srcpkg,
91 changelog.upstream_version,
92 )
93- except MultiplePristineTarFoundError as e:
94+ except gitubuntu.git_repository.MultiplePristineTarFoundError as e:
95 logging.warning("%s. This is often because the orig "
96 "tarball compression changed and it is not possible "
97 "to determine which tarball to use automatically.", e
98@@ -387,12 +385,12 @@ def fetch_orig_from_launchpad(changelog, source, pullfile, retries,
99
100 def check_repository():
101 try:
102- cp = run(['git', 'status', '--porcelain'])
103+ stdout, _ = run(['git', 'status', '--porcelain'])
104 except CalledProcessError:
105 logging.error('Is the current directory a git repository?')
106 return False
107
108- if len(cp.stdout) > 0:
109+ if len(stdout) > 0:
110 logging.warning(
111 "Working tree is not clean. `git status` output follows."
112 )
113@@ -410,9 +408,11 @@ def check_repository():
114
115
116 def main(
117+ repo,
118+ commitish,
119 search_list,
120- changelog,
121 rem_args,
122+ for_merge,
123 sign,
124 use_lxd=True,
125 keep_build_env=False,
126@@ -423,9 +423,11 @@ def main(
127 ):
128 """main entry point for build
129
130+ @param repo: a gitubuntu.git_repository.GitUbuntuRepository object
131+ @param commitish: string commitish to build
132 @param search_list: list of OrigSearchListEntry namedtuples
133- @param changelog: gitubuntu.git_repository.Changelog object for source package to build
134 @param rem_args: namespace object of remaining arguments to pass onto dpkg-buildpackage
135+ @param for_merge: if True, the build is an Ubuntu merge
136 @param sign: if True, sign the resulting changes file
137 @param use_lxd: if True, use a LXD container to build the package
138 @param keep_build_env: if True, do not purge the build environment
139@@ -443,13 +445,10 @@ def main(
140 return []
141
142 try:
143- repo = GitUbuntuRepository('.')
144- pkg_remote_branch_string = derive_target_branch(repo, 'HEAD')
145- if len(pkg_remote_branch_string) == 0:
146- sys.exit(1)
147+ pkg_remote_branch_string = derive_target_branch(repo, commitish)
148
149 # Implies a merge
150- if 'debian' in pkg_remote_branch_string:
151+ if 'debian' in pkg_remote_branch_string or for_merge:
152 oldubuntu_version, _ = repo.get_changelog_versions_from_treeish(
153 'pkg/ubuntu/devel',
154 )
155@@ -462,8 +461,9 @@ def main(
156 )
157
158 files = fetch_orig_and_build(
159+ repo,
160+ commitish,
161 search_list,
162- changelog,
163 rem_args,
164 use_lxd,
165 keep_build_env,
166@@ -482,7 +482,7 @@ def main(
167 changes_file = changes_files.pop()
168 try:
169 logging.info("Signing changes file %s", changes_file)
170- run(['debsign', changes_file,])
171+ run(['debsign', '-p%s' % _gpg_abspath(), changes_file,])
172 except CalledProcessError as e:
173 raise RuntimeError(
174 "Failed to sign changes file %s" % changes_file
175@@ -556,6 +556,12 @@ def parse_args(subparsers=None, base_subparsers=None):
176 help="LXD image to use. If not specified, the image will be "
177 "derived from distribution value of HEAD:debian/changelog.",
178 )
179+ parser.add_argument(
180+ '--commitish',
181+ type=str,
182+ help="Git commitish to build.",
183+ default='HEAD',
184+ )
185 parser.add_argument('rem_args', help=argparse.SUPPRESS,
186 nargs=argparse.REMAINDER)
187 if not subparsers:
188@@ -563,8 +569,31 @@ def parse_args(subparsers=None, base_subparsers=None):
189 return 'build - %s' % kwargs['description']
190
191
192-def derive_orig_search_list_from_args(args):
193- source = 'debian' if args.for_merge else 'ubuntu'
194+def derive_orig_search_list_from_args(
195+ repo,
196+ commitish,
197+ for_merge,
198+ no_pristine_tar,
199+ pullfile=top_level_defaults.pullfile,
200+ retries=top_level_defaults.retries,
201+ retry_backoffs=top_level_defaults.retry_backoffs,
202+ dl_cache=None,
203+):
204+ native = is_native_package(
205+ repo.get_changelog_from_treeish(commitish)
206+ )
207+
208+ if native:
209+ # No orig tarball required
210+ return [
211+ OrigSearchListEntry(
212+ mechanism=fetch_orig_noop,
213+ source=[],
214+ must_build=True,
215+ ),
216+ ]
217+
218+ source = 'debian' if for_merge else 'ubuntu'
219 orig_search_list = [
220 OrigSearchListEntry(
221 mechanism=fetch_orig_from_parent_dir,
222@@ -574,34 +603,29 @@ def derive_orig_search_list_from_args(args):
223 OrigSearchListEntry(
224 mechanism=fetch_orig_from_cache,
225 source=source,
226- must_build=True,
227+ must_build=False,
228 ),
229 ]
230- if not args.no_pristine_tar:
231+ if not no_pristine_tar:
232 orig_search_list.append(
233 OrigSearchListEntry(
234 mechanism=functools.partial(fetch_orig_from_pristine_tar,
235- repo=GitUbuntuRepository('.'),
236+ repo=repo,
237 ),
238 source=source,
239- must_build=True,
240+ must_build=False,
241 )
242 )
243 orig_search_list.extend([
244 OrigSearchListEntry(
245 mechanism=functools.partial(fetch_orig_from_launchpad,
246- pullfile=args.pullfile,
247- retries=args.retries,
248- retry_backoffs=args.retry_backoffs,
249- dl_cache=args.dl_cache,
250+ pullfile=pullfile,
251+ retries=retries,
252+ retry_backoffs=retry_backoffs,
253+ dl_cache=dl_cache,
254 ),
255 source=source,
256- must_build=True,
257- ),
258- OrigSearchListEntry(
259- mechanism=fetch_orig_noop,
260- source=[],
261- must_build=True,
262+ must_build=False,
263 ),
264 ])
265
266@@ -615,32 +639,31 @@ def cli_main(args):
267 else:
268 rem_args = default_rem_args
269
270- changelog = Changelog.from_path('debian/changelog')
271+ repo = gitubuntu.git_repository.GitUbuntuRepository('.')
272
273 try:
274- native = is_native_package(changelog)
275+ orig_search_list = derive_orig_search_list_from_args(
276+ repo,
277+ args.commitish,
278+ args.for_merge,
279+ args.no_pristine_tar,
280+ args.pullfile,
281+ args.retries,
282+ args.retry_backoffs,
283+ args.dl_cache,
284+ )
285 except NativenessMismatchError as e:
286 logging.error("%s" % e)
287 return 1
288
289- if native:
290- # No orig tarball required
291- orig_search_list = [
292- OrigSearchListEntry(
293- mechanism=fetch_orig_noop,
294- source=[],
295- must_build=True,
296- ),
297- ]
298- else:
299- orig_search_list = derive_orig_search_list_from_args(args)
300-
301 # See http://pad.ubuntu.com/KKB1kMR0JH for logic
302 if len(
303 main(
304+ repo,
305+ args.commitish,
306 orig_search_list,
307- changelog,
308 rem_args,
309+ for_merge=args.for_merge,
310 sign=args.sign,
311 use_lxd=not args.no_lxd,
312 keep_build_env=args.keep_build_env,
313@@ -667,7 +690,7 @@ def derive_source_from_changelog(changelog):
314 ])
315 def test_derive_source_from_changelog(changelog_path, expected):
316 assert derive_source_from_changelog(
317- Changelog.from_path(changelog_path)
318+ gitubuntu.git_repository.Changelog.from_path(changelog_path)
319 ) == expected
320
321 def derive_codename_from_changelog(changelog):
322@@ -681,11 +704,11 @@ def derive_codename_from_changelog(changelog):
323 ])
324 def test_derive_codename_from_changelog(changelog_path, expected):
325 assert derive_codename_from_changelog(
326- Changelog.from_path(changelog_path)
327+ gitubuntu.git_repository.Changelog.from_path(changelog_path)
328 ) == expected
329 with pytest.raises(ValueError):
330 derive_codename_from_changelog(
331- Changelog.from_path(
332+ gitubuntu.git_repository.Changelog.from_path(
333 'tests/changelogs/test_distribution_source_4',
334 )
335 )
336@@ -706,7 +729,8 @@ def expand_changelog_source_aliases(orig_search_list, changelog):
337
338
339 def do_build(
340- changelog,
341+ repo,
342+ commitish,
343 tarballs,
344 rem_args,
345 use_lxd,
346@@ -716,66 +740,89 @@ def do_build(
347 retries,
348 retry_backoffs,
349 ):
350- if use_lxd:
351- return do_build_lxd(
352- changelog,
353- tarballs,
354- rem_args,
355- keep_build_env,
356- lxd_profile,
357- lxd_image,
358- retries,
359- retry_backoffs,
360+ with ExitStack() as stack:
361+ changelog = repo.get_changelog_from_treeish(commitish)
362+
363+ commitish_tree_hash = str(repo.get_commitish(commitish).peel(pygit2.Tree).id)
364+
365+ if gitubuntu.git_repository.is_3_0_quilt(repo, commitish):
366+ tree_hash = repo.quiltify_and_changelogify_tree_hash(commitish)
367+ else:
368+ tree_hash = commitish_tree_hash
369+
370+ archive_tarball = stack.enter_context(
371+ tempfile.NamedTemporaryFile(
372+ suffix='.tar.gz',
373+ )
374 )
375- else:
376- return do_build_host(
377- changelog,
378- tarballs,
379- rem_args,
380- keep_build_env,
381+ archive_prefix = '%s-%s' % (
382+ changelog.srcpkg,
383+ changelog.version,
384 )
385+ cmd = [
386+ 'git',
387+ 'archive',
388+ '-o', archive_tarball.name,
389+ '--prefix=%s/' % archive_prefix,
390+ tree_hash,
391+ ]
392+ try:
393+ run(cmd)
394+ except Exception as e:
395+ raise RuntimeError("Failed to archive the repository") from e
396+
397+ if use_lxd:
398+ built_files = do_build_lxd_exitstack(
399+ changelog,
400+ archive_tarball.name,
401+ tarballs,
402+ rem_args,
403+ keep_build_env,
404+ lxd_profile,
405+ lxd_image,
406+ retries,
407+ retry_backoffs,
408+ stack,
409+ )
410+ else:
411+ built_files = do_build_host_exitstack(
412+ changelog,
413+ archive_prefix,
414+ archive_tarball.name,
415+ tarballs,
416+ rem_args,
417+ keep_build_env,
418+ stack,
419+ )
420
421+ # Did we do a quiltify/changelogify fixup?
422+ if commitish_tree_hash != tree_hash:
423+ fixup_commit_hash = repo.commit_tree_hash(
424+ tree_hash=tree_hash,
425+ parents=[commitish],
426+ log_message=b'Automatically generated git-ubuntu fixup.',
427+ )
428+ logging.info(
429+"""We automatically generated fixup changes relative to
430+%s as commit %s.
431+If you would like to create a branch at that commit, run:
432+\tgit checkout -b <branch name> %s""",
433+ commitish,
434+ fixup_commit_hash,
435+ fixup_commit_hash,
436+ )
437
438-def do_build_host(changelog, tarballs, rem_args, keep_build_env):
439- with ExitStack() as stack:
440- return _do_build_host_exitstack(
441- changelog,
442- tarballs,
443- rem_args,
444- keep_build_env,
445- stack,
446- )
447+ return built_files
448
449-def _do_build_host_exitstack(
450+def do_build_host_exitstack(
451 changelog,
452+ archive_prefix,
453+ archive_tarball_name,
454 tarballs,
455 rem_args,
456 keep_build_env,
457 stack,
458 ):
459- commitish = 'HEAD'
460-
461- archive_tarball = stack.enter_context(
462- tempfile.NamedTemporaryFile(
463- suffix='.tar.gz',
464- )
465- )
466- archive_prefix = '%s-%s' % (
467- changelog.srcpkg,
468- changelog.version,
469- )
470- cmd = [
471- 'git',
472- 'archive',
473- '-o', archive_tarball.name,
474- '--prefix=%s/' % archive_prefix,
475- commitish
476- ]
477- try:
478- run(cmd)
479- except Exception as e:
480- raise RuntimeError("Failed to archive the repository") from e
481-
482 tempdir = tempfile.mkdtemp()
483 if keep_build_env:
484 logging.info(
485@@ -790,7 +837,7 @@ def _do_build_host_exitstack(
486 shutil.copy(tarball, tempdir)
487
488 try:
489- run(['tar', 'zxCf', tempdir, archive_tarball.name,])
490+ run(['tar', 'zxCf', tempdir, archive_tarball_name,])
491 except Exception as e:
492 raise RuntimeError(
493 "Failed to untar archive tarball in temporary directory"
494@@ -814,29 +861,6 @@ def _do_build_host_exitstack(
495
496 return built_pardir_contents
497
498-def do_build_lxd(
499- changelog,
500- tarballs,
501- rem_args,
502- keep_build_env,
503- profile,
504- image,
505- retries,
506- retry_backoffs,
507-):
508- with ExitStack() as stack:
509- return _do_build_lxd_exitstack(
510- changelog,
511- tarballs,
512- rem_args,
513- keep_build_env,
514- profile,
515- image,
516- retries,
517- retry_backoffs,
518- stack,
519- )
520-
521 def get_cmd_in_origpath(cmd):
522 return shutil.which(
523 cmd,
524@@ -847,20 +871,58 @@ def get_cmd_in_origpath(cmd):
525 )
526
527 @functools.lru_cache()
528+def _gpg_abspath():
529+ orig_gpg = get_cmd_in_origpath('gpg')
530+ if orig_gpg:
531+ return orig_gpg
532+ raise RuntimeError("Unable to find a gpg binary, is it installed?")
533+
534+@functools.lru_cache()
535 def _lxc_abspath():
536 orig_lxc = get_cmd_in_origpath('lxc')
537- return orig_lxc if orig_lxc else 'lxc'
538+ if orig_lxc:
539+ return orig_lxc
540+ raise RuntimeError("Unable to find a lxc binary, is it installed?")
541
542 def _cleanup_lxd(container_name):
543 run([_lxc_abspath(), 'stop', '--force', container_name])
544
545-def _run_in_lxd(container_name, args):
546+def _run_in_lxd(container_name, args, user=None, **kwargs):
547 cmd = [_lxc_abspath(), 'exec', container_name, '--',]
548+ # user == None means run as root
549+ if user:
550+ try:
551+ _run_in_lxd(
552+ container_name,
553+ [
554+ 'grep',
555+ '-q',
556+ user,
557+ '/etc/passwd',
558+ ],
559+ user=None,
560+ verbose_on_failure=False,
561+ )
562+ except CalledProcessError as e:
563+ if e.returncode == 1:
564+ _run_in_lxd(
565+ container_name,
566+ [
567+ 'adduser',
568+ user,
569+ '--disabled-password',
570+ '--gecos=""',
571+ ],
572+ )
573+ else:
574+ raise
575+ cmd.extend(['sudo', '-s', '-H', '-u', user,])
576 cmd.extend(args)
577- return run(cmd)
578+ return run(cmd, **kwargs)
579
580-def _do_build_lxd_exitstack(
581+def do_build_lxd_exitstack(
582 changelog,
583+ archive_tarball_name,
584 tarballs,
585 rem_args,
586 keep_build_env,
587@@ -876,26 +938,7 @@ def _do_build_lxd_exitstack(
588 except Exception as e:
589 raise RuntimeError("LXD running?") from e
590
591- commitish = 'HEAD'
592-
593- archive_tarball = stack.enter_context(
594- tempfile.NamedTemporaryFile(
595- suffix='.tar.gz',
596- )
597- )
598- cmd = [
599- 'git',
600- 'archive',
601- '-o', archive_tarball.name,
602- '--prefix=%s-%s/' % (changelog.srcpkg, changelog.version),
603- commitish
604- ]
605- try:
606- run(cmd)
607- except Exception as e:
608- raise RuntimeError("Failed to archive the repository") from e
609-
610- container_name = 'gu-build-%s' % uuid.uuid4()
611+ container_name = petname.Generate(2, '-')
612
613 cmd = [lxc, 'launch', '-e']
614 if profile:
615@@ -937,9 +980,6 @@ def _do_build_lxd_exitstack(
616 "manually stopped with:\n\tlxc stop --force %s",
617 container_name,
618 )
619- # wait for the LXD container to have networking, which always takes
620- # a bit of time
621- time.sleep(10)
622
623 for i in range(retries+1):
624 try:
625@@ -950,6 +990,7 @@ def _do_build_lxd_exitstack(
626 '-y',
627 'devscripts',
628 'equivs',
629+ 'sudo',
630 ])
631 break
632 except Exception as e:
633@@ -970,7 +1011,7 @@ def _do_build_lxd_exitstack(
634 lxc,
635 'file',
636 'push',
637- archive_tarball.name,
638+ archive_tarball_name,
639 '%s/tmp/' % container_name,
640 ])
641 except Exception as e:
642@@ -978,17 +1019,9 @@ def _do_build_lxd_exitstack(
643 "Failed to push archive tarball to ephemeral build container"
644 ) from e
645
646- try:
647- cp = _run_in_lxd(container_name, ['mktemp' , '-d',])
648- containertemp = decode_binary(cp.stdout).strip()
649- except Exception as e:
650- raise RuntimeError(
651- "Failed to create temporary build directory in container"
652- ) from e
653-
654 cmd = [lxc, 'file', 'push']
655 cmd.extend(tarballs)
656- cmd.extend(['%s%s/' % (container_name, containertemp),])
657+ cmd.extend(['%s/tmp/' % container_name,])
658 try:
659 run(cmd)
660 except Exception as e:
661@@ -997,29 +1030,36 @@ def _do_build_lxd_exitstack(
662 logging.info("Copied build files to %s", container_name)
663
664 try:
665- _run_in_lxd(container_name, [
666- 'tar',
667- 'zxCf',
668- containertemp,
669- '/tmp/%s' % os.path.basename(archive_tarball.name),
670- ])
671+ _run_in_lxd(
672+ container_name,
673+ [
674+ 'tar',
675+ 'zxCf',
676+ '/tmp',
677+ '/tmp/%s' % os.path.basename(archive_tarball_name),
678+ ],
679+ user='ubuntu',
680+ )
681 except Exception as e:
682 raise RuntimeError(
683 "Failed to untar archive tarball in container"
684 ) from e
685
686- cp = _run_in_lxd(container_name, ['ls', containertemp,])
687+ stdout, _ = _run_in_lxd(
688+ container_name,
689+ ['ls', '/tmp',],
690+ user='ubuntu',
691+ )
692 orig_temp_contents = set(
693 filename_stripped
694 for filename_stripped in (
695 filename.strip() for filename in
696- decode_binary(cp.stdout).splitlines()
697+ stdout.splitlines()
698 )
699 if filename_stripped
700 )
701
702- container_path = '%s/%s-%s' % (
703- containertemp,
704+ container_path = '/tmp/%s-%s' % (
705 changelog.srcpkg,
706 changelog.version,
707 )
708@@ -1033,11 +1073,14 @@ def _do_build_lxd_exitstack(
709 # ),
710 # except that Trusty does not support apt-get build-dep with a
711 # local dir.
712- _run_in_lxd(container_name, [
713- 'sh',
714- '-c',
715- 'cd %s && (export DEBIAN_FRONTEND=noninteractive; mk-build-deps -i -t "apt-get -y -o DPkg::options::=\'--force-confdef\' -o DPkg::options::=\'--force-confold\' -o Debug::pkgProblemResolver=yes --no-install-recommends" -r debian/control)' % container_path,
716- ])
717+ _run_in_lxd(
718+ container_name,
719+ [
720+ 'sh',
721+ '-c',
722+ 'cd %s && (export DEBIAN_FRONTEND=noninteractive; mk-build-deps -i -t "apt-get -y -o DPkg::options::=\'--force-confdef\' -o DPkg::options::=\'--force-confold\' -o Debug::pkgProblemResolver=yes --no-install-recommends" -r debian/control)' % container_path,
723+ ],
724+ )
725 except Exception as e:
726 raise RuntimeError(
727 "Failed to install build dependencies in container"
728@@ -1046,23 +1089,31 @@ def _do_build_lxd_exitstack(
729 logging.info("Running dpkg-buildpackage in %s", container_name)
730
731 try:
732- _run_in_lxd(container_name, [
733- 'sh',
734- '-c',
735- 'cd %s && dpkg-buildpackage %s' % (
736- container_path,
737- ' '.join(rem_args),
738- ),
739- ])
740+ _run_in_lxd(
741+ container_name,
742+ [
743+ 'sh',
744+ '-c',
745+ 'cd %s && dpkg-buildpackage %s' % (
746+ container_path,
747+ ' '.join(rem_args),
748+ ),
749+ ],
750+ user='ubuntu',
751+ )
752 except Exception as e:
753 raise RuntimeError("dpkg-buildpackage failed in the container") from e
754
755- cp = _run_in_lxd(container_name, ['ls', containertemp])
756+ stdout, _ = _run_in_lxd(
757+ container_name,
758+ ['ls', '/tmp',],
759+ user='ubuntu',
760+ )
761 new_temp_contents = set(
762 filename_stripped
763 for filename_stripped in (
764 filename.strip() for filename in
765- decode_binary(cp.stdout).splitlines()
766+ stdout.splitlines()
767 )
768 if filename_stripped
769 )
770@@ -1070,7 +1121,7 @@ def _do_build_lxd_exitstack(
771 built_temp_contents = new_temp_contents - orig_temp_contents
772 cmd = [lxc, 'file', 'pull']
773 cmd.extend([
774- '%s%s' % (container_name, os.path.join(containertemp, f))
775+ '%s%s' % (container_name, os.path.join('/tmp', f))
776 for f in built_temp_contents
777 ]
778 )
779@@ -1080,9 +1131,39 @@ def _do_build_lxd_exitstack(
780 return [os.path.join(os.pardir, f) for f in built_temp_contents]
781
782
783-def fetch_orig_and_build(
784+def fetch_orig(
785 orig_search_list,
786 changelog,
787+):
788+ unaliased_orig_search_list = expand_changelog_source_aliases(
789+ orig_search_list,
790+ changelog,
791+ )
792+ # Follow searches already attempted as a 'changelog' source alias
793+ # may expand to a duplicate.
794+ for entry in unique_everseen(unaliased_orig_search_list):
795+ try:
796+ mechanism_name = entry.mechanism.__name__
797+ except AttributeError:
798+ mechanism_name = entry.mechanism.func.__name__
799+ tarballs = entry.mechanism(changelog, entry.source)
800+ if tarballs is None:
801+ logging.debug('%s(source=%s) failed',
802+ mechanism_name,
803+ entry.source,
804+ )
805+ continue # search returned negative; try next search entry
806+ logging.info('Successfully fetched%susing %s(source=%s)',
807+ ':\n' + '\n'.join(tarballs) + '\n' if tarballs else ' ',
808+ mechanism_name,
809+ entry.source,
810+ )
811+ return tarballs
812+
813+def fetch_orig_and_build(
814+ repo,
815+ commitish,
816+ orig_search_list,
817 rem_args=[],
818 use_lxd=True,
819 keep_build_env=False,
820@@ -1091,6 +1172,7 @@ def fetch_orig_and_build(
821 retries=top_level_defaults.retries,
822 retry_backoffs=top_level_defaults.retry_backoffs,
823 ):
824+ changelog = repo.get_changelog_from_treeish(commitish)
825 unaliased_orig_search_list = expand_changelog_source_aliases(
826 orig_search_list,
827 changelog,
828@@ -1111,7 +1193,8 @@ def fetch_orig_and_build(
829 continue # search returned negative; try next search entry
830 try:
831 built_files = do_build(
832- changelog,
833+ repo,
834+ commitish,
835 tarballs,
836 rem_args,
837 use_lxd,
838@@ -1133,6 +1216,9 @@ def fetch_orig_and_build(
839 # Build failed, but the search list entry declares that
840 # this should not be fatal and so we try the next search
841 # entry
842+ # First, remove any tarballs fetched with this mechanism
843+ for tarball in tarballs:
844+ os.remove(tarball)
845 logging.debug(
846 "Ignoring failure to build using %s(source=%s)",
847 mechanism_name,
848diff --git a/gitubuntu/buildsource.py b/gitubuntu/buildsource.py
849index 833ae74..dfbef8c 100644
850--- a/gitubuntu/buildsource.py
851+++ b/gitubuntu/buildsource.py
852@@ -8,7 +8,7 @@ from subprocess import CalledProcessError
853 import sys
854 import gitubuntu.build
855 from gitubuntu.cache import CACHE_PATH
856-from gitubuntu.git_repository import Changelog
857+import gitubuntu.git_repository
858 from gitubuntu.run import run
859
860 def parse_args(subparsers=None, base_subparsers=None):
861@@ -74,8 +74,16 @@ def parse_args(subparsers=None, base_subparsers=None):
862 help="LXD image to use. If not specified, the image will be "
863 "derived from distribution value of HEAD:debian/changelog.",
864 )
865- parser.add_argument('rem_args', help=argparse.SUPPRESS,
866- nargs=argparse.REMAINDER
867+ parser.add_argument(
868+ '--commitish',
869+ type=str,
870+ help="Git commitish to build.",
871+ default='HEAD',
872+ )
873+ parser.add_argument(
874+ 'rem_args',
875+ help=argparse.SUPPRESS,
876+ nargs=argparse.REMAINDER,
877 )
878 if not subparsers:
879 return parser.parse_args()
880@@ -96,33 +104,30 @@ def cli_main(args):
881 if args.for_merge:
882 args.rem_args += ['-sa']
883
884- changelog = Changelog.from_path('debian/changelog')
885+ repo = gitubuntu.git_repository.GitUbuntuRepository('.')
886
887 try:
888- native = gitubuntu.build.is_native_package(changelog)
889+ orig_search_list = gitubuntu.build.derive_orig_search_list_from_args(
890+ repo,
891+ args.commitish,
892+ args.for_merge,
893+ args.no_pristine_tar,
894+ args.pullfile,
895+ args.retries,
896+ args.retry_backoffs,
897+ args.dl_cache,
898+ )
899 except gitubuntu.build.NativenessMismatchError as e:
900 logging.error("%s" % e)
901 return 1
902
903- if native:
904- # No orig tarball required
905- orig_search_list = [
906- gitubuntu.build.OrigSearchListEntry(
907- mechanism=gitubuntu.build.fetch_orig_noop,
908- source=[],
909- must_build=True,
910- ),
911- ]
912- else:
913- orig_search_list = gitubuntu.build.derive_orig_search_list_from_args(
914- args,
915- )
916-
917 if len(
918 gitubuntu.build.main(
919+ repo,
920+ args.commitish,
921 orig_search_list,
922- changelog,
923 rem_args,
924+ for_merge=args.for_merge,
925 sign=args.sign,
926 use_lxd=not args.no_lxd,
927 keep_build_env=args.keep_build_env,
928diff --git a/gitubuntu/git_repository.py b/gitubuntu/git_repository.py
929index 58d5a09..dffeaae 100644
930--- a/gitubuntu/git_repository.py
931+++ b/gitubuntu/git_repository.py
932@@ -16,7 +16,9 @@ from subprocess import CalledProcessError
933 import sys
934 import tempfile
935 import time
936+import gitubuntu.lint
937 from gitubuntu.__main__ import top_level_defaults
938+import gitubuntu.build
939 from gitubuntu.dsc import component_tarball_matches
940 from gitubuntu.run import run, runq, decode_binary
941 try:
942@@ -149,23 +151,23 @@ class Changelog:
943
944 @lru_cache()
945 def _dpkg_parsechangelog(self, parse_params):
946- cp = run(
947+ stdout, _ = run(
948 'dpkg-parsechangelog -l- %s' % parse_params,
949 input=self._contents,
950 shell=True,
951 verbose_on_failure=False,
952 )
953- return decode_binary(cp.stdout).strip()
954+ return stdout
955
956 @lru_cache()
957 def _shell(self, cmd):
958- cp = run(
959+ stdout, _ = run(
960 cmd,
961 input=self._contents,
962 shell=True,
963 verbose_on_failure=False,
964 )
965- return decode_binary(cp.stdout).strip()
966+ return stdout
967
968 @property
969 def _shell_version(self):
970@@ -408,6 +410,27 @@ def upstream_tag(version, namespace):
971 def orphan_tag(version, namespace):
972 return '%s/orphan/%s' % (namespace, git_dep14_tag(version))
973
974+def is_dir_3_0_quilt(_dir=None):
975+ _dir = _dir if _dir else '.'
976+ try:
977+ fmt, _ = run(['dpkg-source', '--print-format', _dir])
978+ if '3.0 (quilt)' in fmt:
979+ return True
980+ except CalledProcessError as e:
981+ try:
982+ with open(os.path.join(_dir, 'debian/source/format'), 'r') as f:
983+ for line in f:
984+ if re.match(r'3.0 (.*)', line):
985+ return True
986+ # `man dpkg-source` indicates no d/s/format implies 1.0
987+ except OSError:
988+ pass
989+
990+ return False
991+
992+def is_3_0_quilt(repo, commitish='HEAD'):
993+ with repo.temporary_worktree(commitish):
994+ return is_dir_3_0_quilt()
995
996 class GitUbuntuRepositoryFetchError(Exception):
997 pass
998@@ -465,11 +488,10 @@ class GitUbuntuRepository:
999 self._lp_user = lp_user
1000 else:
1001 try:
1002- cp = self.git_run(
1003+ self._lp_user, _ = self.git_run(
1004 ['config', 'gitubuntu.lpuser'],
1005 verbose_on_failure=False,
1006 )
1007- self._lp_user = decode_binary(cp.stdout).strip()
1008 except CalledProcessError:
1009 logging.error("Unable to determine Launchpad user")
1010 sys.exit(1)
1011@@ -554,8 +576,8 @@ class GitUbuntuRepository:
1012
1013 def pristine_tar_list(self, dist, namespace='pkg'):
1014 with self.pristine_tar_branches(dist, namespace):
1015- cp = run(['pristine-tar', 'list'])
1016- return decode_binary(cp.stdout).strip().splitlines()
1017+ stdout, _ = run(['pristine-tar', 'list'])
1018+ return stdout.splitlines()
1019
1020 def pristine_tar_extract(self, pkgname, version, dist=None, namespace='pkg'):
1021 '''Extract orig tarballs for a given package and upstream version
1022@@ -737,6 +759,26 @@ class GitUbuntuRepository:
1023 # https://github.com/libgit2/pygit2/issues/671
1024 return any(remote.name == remote_name for remote in self.raw_repo.remotes)
1025
1026+ def _add_remote_by_fetch_url(self, remote_name, fetch_url):
1027+ if not self._fetch_proto:
1028+ raise Exception('Cannot fetch using an object without a protocol')
1029+
1030+ logging.debug('Adding %s as remote %s', fetch_url, remote_name)
1031+
1032+ if not self.remote_exists(remote_name):
1033+ self.raw_repo.remotes.create(remote_name, fetch_url,
1034+ 'refs/heads/*:refs/remotes/%s/*' % remote_name)
1035+ # grab unreachable tags (orphans)
1036+ self.raw_repo.remotes.add_fetch(remote_name,
1037+ 'refs/tags/*:refs/tags/%s/*' % remote_name)
1038+ self.git_run(
1039+ [
1040+ 'config',
1041+ 'remote.%s.tagOpt' % remote_name,
1042+ '--no-tags',
1043+ ]
1044+ )
1045+
1046 def _add_remote(self, remote_name, remote_url):
1047 if not self._fetch_proto:
1048 raise Exception('Cannot fetch using an object without a protocol')
1049@@ -770,6 +812,12 @@ class GitUbuntuRepository:
1050
1051 self._add_remote(remote_name, remote_url)
1052
1053+ def add_remote_by_url(self, remote_name, fetch_url):
1054+ if not self._fetch_proto:
1055+ raise Exception('Cannot fetch using an object without a protocol')
1056+
1057+ self._add_remote_by_fetch_url(remote_name, fetch_url)
1058+
1059 def add_base_remotes(self, pkgname, repo_owner='usd-import-team'):
1060 self.add_remote(pkgname, repo_owner, 'pkg')
1061
1062@@ -798,7 +846,7 @@ class GitUbuntuRepository:
1063 kwargs['stderr'] = None
1064 try:
1065 logging.debug("Fetching remote %s", remote_name)
1066- cp = self.git_run(
1067+ self.git_run(
1068 args=['fetch', remote_name],
1069 env={'GIT_TERMINAL_PROMPT': '0',},
1070 **kwargs
1071@@ -950,7 +998,8 @@ class GitUbuntuRepository:
1072
1073 Note that the hash may still become ambiguous in the future.
1074 """
1075- return self.git_run(['rev-parse', '--short', hash]).stdout.decode().strip()
1076+ stdout, _ = self.git_run(['rev-parse', '--short', hash])
1077+ return stdout
1078
1079 def git_run(self, args, env=None, **kwargs):
1080 # Explicitly take a copy of self._env so the following update doesn't
1081@@ -1140,6 +1189,29 @@ class GitUbuntuRepository:
1082 return set()
1083
1084
1085+ def nearest_tag(
1086+ self,
1087+ commitish_string,
1088+ prefix,
1089+ max_commits=100,
1090+ ):
1091+ # 1) cache all patterned tag names by commit
1092+ pattern_tags_by_commit = collections.defaultdict(set)
1093+ for t in self.tags:
1094+ if t.name.startswith('refs/tags/' + prefix):
1095+ pattern_tags_by_commit[t.peel(pygit2.Commit).id].add(t)
1096+
1097+ commits = self.raw_repo.walk(
1098+ self.get_commitish(commitish_string).id,
1099+ pygit2.GIT_SORT_TOPOLOGICAL,
1100+ )
1101+ for commit in itertools.islice(commits, max_commits):
1102+ if commit.id not in pattern_tags_by_commit:
1103+ continue
1104+
1105+ return pattern_tags_by_commit[commit.id].pop()
1106+
1107+ return None
1108
1109 @staticmethod
1110 def tag_to_pretty_name(tag):
1111@@ -1349,7 +1421,8 @@ class GitUbuntuRepository:
1112 fp.flush()
1113 commit_tree += ['-F', fp.name]
1114 try:
1115- cp = run(commit_tree, env=commit_env)
1116+ stdout, _ = run(commit_tree, env=commit_env)
1117+ return stdout
1118 except CalledProcessError:
1119 _, t = tempfile.mkstemp()
1120 shutil.copy(fp.name, t)
1121@@ -1357,8 +1430,6 @@ class GitUbuntuRepository:
1122 "temp commit message as %s", t)
1123 sys.exit(1)
1124
1125- return decode_binary(cp.stdout).strip()
1126-
1127 @classmethod
1128 def _create_replacement_tree_builder(cls, repo, treeish, sub_path):
1129 '''Create a replacement TreeBuilder
1130@@ -1467,40 +1538,51 @@ class GitUbuntuRepository:
1131 with tempfile.TemporaryDirectory() as index_dir:
1132 index_path = os.path.join(index_dir, 'index')
1133 index_env = {'GIT_INDEX_FILE': index_path}
1134- git_dir = os.path.join(path, '.git')
1135- if os.path.exists(git_dir):
1136- logging.warning('.git directory found in source '
1137- 'package. Will remove it.')
1138- shutil.rmtree(git_dir)
1139 self.git_run(
1140 ['--work-tree', path, 'add', '-f', '-A'],
1141 env=index_env,
1142 )
1143- cp = self.git_run(
1144+ self.git_run(
1145+ ['--work-tree', path, 'reset', 'HEAD', '--', '.git',],
1146+ env=index_env,
1147+ )
1148+ self.git_run(
1149+ ['--work-tree', path, 'reset', 'HEAD', '--', '.pc',],
1150+ env=index_env,
1151+ )
1152+ tree_hash_str, _ = self.git_run(
1153 ['--work-tree', path, 'write-tree'],
1154 env=index_env,
1155 )
1156- tree_hash_str = decode_binary(cp.stdout).strip()
1157- tree = self.raw_repo.get(tree_hash_str)
1158-
1159- # Add any empty directories that git did not import. Workaround for LP:
1160- # #1687057.
1161- replacement_oid = self._add_missing_tree_dirs(
1162- repo=self.raw_repo,
1163- top_path=path,
1164- top_tree_object=tree,
1165- )
1166- if replacement_oid:
1167- # Empty directories had to be added
1168- return str(replacement_oid) # return the replacement instead
1169- else:
1170- # No empty directories were added
1171- return tree_hash_str # no replacement was needed
1172+ tree = self.raw_repo.get(tree_hash_str)
1173+
1174+ # Add any empty directories that git did not import. Workaround for LP:
1175+ # #1687057.
1176+ replacement_oid = self._add_missing_tree_dirs(
1177+ repo=self.raw_repo,
1178+ top_path=path,
1179+ top_tree_object=tree,
1180+ )
1181+ if replacement_oid:
1182+ # Empty directories had to be added
1183+ return str(replacement_oid) # return the replacement instead
1184+ else:
1185+ # No empty directories were added
1186+ return tree_hash_str # no replacement was needed
1187
1188 @contextmanager
1189- def temporary_worktree(self, commitish):
1190- with tempfile.TemporaryDirectory() as tempdir:
1191- self.git_run(['worktree', 'add', '--force', tempdir, commitish])
1192+ def temporary_worktree(self, commitish, prefix=None):
1193+ with tempfile.TemporaryDirectory(prefix=prefix) as tempdir:
1194+ self.git_run(
1195+ [
1196+ 'worktree',
1197+ 'add',
1198+ '--detach',
1199+ '--force',
1200+ tempdir,
1201+ commitish,
1202+ ]
1203+ )
1204
1205 oldcwd = os.getcwd()
1206 os.chdir(tempdir)
1207@@ -1523,6 +1605,429 @@ class GitUbuntuRepository:
1208 raise
1209
1210 run(["git", "add", "-f", ".",])
1211- cp = run(["git", "write-tree"])
1212- tree_hash = decode_binary(cp.stdout).strip()
1213- return tree_hash
1214+ tree_hash, _ = run(["git", "write-tree"])
1215+ return tree_hash
1216+
1217+ def tree_hash_subpath(self, treeish_string, path):
1218+ """Get the tree hash for path at a given treeish
1219+
1220+ Arguments:
1221+ @treeish_string: a string Git treeish
1222+ @path: a string path present in @treeish_string
1223+
1224+ Returns:
1225+ String hash of Git tree corresponding to @path in @treeish_string
1226+ """
1227+ tree_obj = self.raw_repo.revparse_single(treeish_string).peel(
1228+ pygit2.Tree
1229+ )
1230+ return str(tree_obj[path].id)
1231+
1232+ def paths_are_identical(self, treeish1_string, treeish2_string, path):
1233+ """Determine if a given path is the same in two treeishs
1234+
1235+ Arguments:
1236+ @treeish1_string: a string Git treeish
1237+ @treeish2_string: a string Git treeish
1238+ @path: a string path present in @treeish1_string and @treeish2_string
1239+
1240+ Returns:
1241+ True, if @path is the same in @treeish1_string and @treeish2_string
1242+ False, otherwise
1243+ """
1244+ return self.tree_hash_subpath(
1245+ treeish1_string,
1246+ path,
1247+ ) == self.tree_hash_subpath(
1248+ treeish2_string,
1249+ path,
1250+ )
1251+
1252+ @lru_cache()
1253+ def quilt_env(self, commit_hash):
1254+ with self.temporary_worktree(commit_hash):
1255+ combined_env = self.env.copy()
1256+ combined_env.update(
1257+ {
1258+ 'QUILT_PATCHES': 'debian/patches',
1259+ 'QUILT_NO_DIFF_INDEX': '1',
1260+ 'QUILT_NO_DIFF_TIMESTAMPS': '1',
1261+ 'EDITOR': 'true',
1262+ }
1263+ )
1264+ for path in [
1265+ os.path.join('debian', 'patches', 'debian.series'),
1266+ os.path.join('debian', 'patches', 'series'),
1267+ ]:
1268+ if os.path.exists(path):
1269+ combined_env.update({'QUILT_SERIES': path,})
1270+ break
1271+ else:
1272+ logging.debug(
1273+ "Unable to find a series file in %s",
1274+ commit_hash
1275+ )
1276+
1277+ return combined_env
1278+
1279+ def is_patches_applied(self, commit_hash, regenerated_pc_path):
1280+ # first see if quilt push -a would do anything to
1281+ # differentiate between applied an unapplied
1282+ with self.temporary_worktree(commit_hash):
1283+ try:
1284+ run(['quilt', 'push', '-a'], env=self.quilt_env(commit_hash))
1285+ # False if in an unapplied state, which is signified by
1286+ # successful push (rc=0)
1287+ return False
1288+ except CalledProcessError as e:
1289+ # non-zero return might be an error or it might mean no
1290+ # patches exist
1291+ if e.returncode == 1:
1292+ # an error may occur if we need to recreate the .pc
1293+ # first
1294+ try:
1295+ # the first quilt push may have created a .pc/
1296+ shutil.rmtree('.pc')
1297+ shutil.copytree(
1298+ regenerated_pc_path,
1299+ '.pc',
1300+ )
1301+ except FileNotFoundError:
1302+ # if there was no .pc directory, then the first
1303+ # quilt push failure was a real error
1304+ raise e
1305+
1306+ try:
1307+ run(['quilt', 'push', '-a'], env=self.quilt_env(commit_hash))
1308+ # False if in an unapplied state
1309+ return False
1310+ except CalledProcessError as e:
1311+ # True if in a patches-applied state or
1312+ # there are no patches to apply
1313+ if e.returncode == 2:
1314+ return True
1315+ else:
1316+ raise
1317+ # True if in a patches-applied state or there are
1318+ # no patches to apply
1319+ elif e.returncode == 2:
1320+ return True
1321+ else:
1322+ raise
1323+
1324+ def _maybe_quiltify_tree_hash(self, commit_hash):
1325+ """Determine if quiltify is needed and yield the quiltify'd tree hash
1326+
1327+ The imported patches-applied trees do not contain .pc
1328+ directories. To determine if an additional quilt patch is
1329+ necessary, we have to first regenerate the .pc directory, then
1330+ see if dpkg-source --commit generates a new quilt patch.
1331+
1332+ In order for dpkg-source --commit to function, we need to know
1333+ if the commit we are building is patches-unapplied or
1334+ patches-applied. In the latter case, we can build the commit
1335+ directly after copying the regenerated .pc directory. In the
1336+ former case, we do not want to copy the regenerated .pc
1337+ directory, as dpkg-source will do this for us, as it applies the
1338+ current patches. We determine if patches are applied or
1339+ unapplied by relying `quilt push -a`'s exit status at
1340+ @commit_hash.
1341+
1342+ This is a common method used by multiple callers.
1343+
1344+ Arguments:
1345+ @commit_hash: a string Git commit hash
1346+
1347+ Returns:
1348+ String tree hash of quiltify'ing @commit_hash.
1349+ If no quiltify is needed, the return value is @commit_hash's
1350+ tree hash
1351+ """
1352+ # we need the orig tarballs for quilt and dpkg-source
1353+ # but suppress any logging
1354+ logger = logging.getLogger()
1355+ oldLevel = logger.getEffectiveLevel()
1356+ logger.setLevel(logging.WARNING)
1357+ tarballs = gitubuntu.build.fetch_orig(
1358+ orig_search_list=gitubuntu.build.derive_orig_search_list_from_args(
1359+ self,
1360+ commitish=commit_hash,
1361+ for_merge=False,
1362+ no_pristine_tar=False,
1363+ ),
1364+ changelog=Changelog.from_path('debian/changelog'),
1365+ )
1366+ logger.setLevel(oldLevel)
1367+ commit_tree_hash = str(
1368+ self.raw_repo.get(commit_hash).peel(pygit2.Tree).id
1369+ )
1370+ # the tarballs need to be in the parent directory from where we
1371+ # run dpkg-source
1372+ with tempfile.TemporaryDirectory() as tempdir:
1373+ # copy the generated tarballs
1374+ new_tarballs = []
1375+ for tarball in tarballs:
1376+ new_tarballs.append(shutil.copy(tarball, tempdir))
1377+ tarballs = new_tarballs
1378+
1379+ # create a nested temporary directory where we will recreate
1380+ # the .pc directory
1381+ with tempfile.TemporaryDirectory(prefix=tempdir+'/') as ttempdir:
1382+ oldcwd = os.getcwd()
1383+ os.chdir(ttempdir)
1384+
1385+ for tarball in tarballs:
1386+ run(['tar', '-x', '--strip-components=1', '-f', tarball,])
1387+
1388+ # need the debia/patches
1389+ shutil.copytree(
1390+ os.path.join(self.local_dir, 'debian',),
1391+ 'debian',
1392+ )
1393+
1394+ # generate the equivalent .pc directory
1395+ run(
1396+ ['quilt', 'push', '-a'],
1397+ env=self.quilt_env(commit_hash),
1398+ rcs=[2],
1399+ )
1400+
1401+ regenerated_pc_path = os.path.join(tempdir, '.pc')
1402+ shutil.copytree(
1403+ '.pc',
1404+ regenerated_pc_path,
1405+ )
1406+
1407+ os.chdir(oldcwd)
1408+
1409+ patches_applied = self.is_patches_applied(
1410+ commit_hash,
1411+ regenerated_pc_path,
1412+ )
1413+
1414+ with self.temporary_worktree(commit_hash, prefix=tempdir+'/'):
1415+ # we only need to copy the generated .pc directory
1416+ # if we are building a patches-applied tree, which
1417+ # we determine by comparing our current tree hash to
1418+ # the generated tree hash.
1419+ if patches_applied:
1420+ try:
1421+ shutil.copytree(
1422+ regenerated_pc_path,
1423+ '.pc',
1424+ )
1425+ except FileNotFoundError:
1426+ # it is possible no quilt patches exist yet
1427+ pass
1428+
1429+ fixup_patch_path = os.path.join(
1430+ 'debian',
1431+ 'patches',
1432+ 'git-ubuntu-fixup.patch'
1433+ )
1434+
1435+ if os.path.exists(fixup_patch_path):
1436+ raise ValueError(
1437+ "A quilt patch with the name git-ubuntu-fixup.patch "
1438+ "already exists in %s" % commit_hash
1439+ )
1440+
1441+ run(
1442+ [
1443+ 'dpkg-source',
1444+ '--commit',
1445+ '.',
1446+ 'git-ubuntu-fixup.patch',
1447+ ],
1448+ env=self.quilt_env(commit_hash),
1449+ )
1450+
1451+ # do not want the .pc directory in the resulting
1452+ # treeish
1453+ shutil.rmtree('.pc')
1454+
1455+ if os.path.exists(fixup_patch_path):
1456+ # dpkg-source uses debian/changelog to generate some
1457+ # fields. We do not know yet if the changelog has
1458+ # been updated, so elide that section of comments.
1459+ with open(fixup_patch_path, 'r+') as f:
1460+ for line in f:
1461+ if '---' in line:
1462+ break
1463+ text = """Description: git-ubuntu generated quilt fixup patch
1464+TODO: Put a short summary on the line above and replace this paragraph
1465+with a longer explanation of this change. Complete the meta-information
1466+with other relevant fields (see below for details).
1467+---\n"""
1468+ for line in f:
1469+ text += line
1470+ f.seek(0)
1471+ f.write(text)
1472+ f.truncate()
1473+
1474+ # If we are on a patches-unapplied tree, then we
1475+ # need to reset ourselves back to @commit_hash with
1476+ # our new patch.
1477+ # In order for this to be buildable, we have to
1478+ # reverse-apply our patch, to undo the git-commited
1479+ # upstream changes.
1480+ if not patches_applied:
1481+ run(['git', 'add', '-f', 'debian/patches',])
1482+ # if any patches add files that are untracked,
1483+ # remove them
1484+ run(['git', 'clean', '-f', '-d',])
1485+ # reset all the other files to their status in
1486+ # HEAD
1487+ run(['git', 'checkout', commit_hash, '--', '*',])
1488+ with open(fixup_patch_path, 'rb') as f:
1489+ run(['patch', '-Rp1',], input=f.read())
1490+
1491+ return self.dir_to_tree('.')
1492+ else:
1493+ return commit_tree_hash
1494+
1495+ def maybe_quiltify_tree_hash(self, commitish_string):
1496+ """Determine if quiltify is needed and return the quiltify'd tree hash
1497+
1498+ See _maybe_quiltify_tree_hash for details.
1499+
1500+ Arguments:
1501+ @commitish_string: a string Git commitish
1502+
1503+ Returns:
1504+ String tree hash of quiltify'ing @commitish_string.
1505+ If no quiltify is needed, the return value is the tree hash of
1506+ @commitish_string.
1507+ """
1508+ commit_hash = str(
1509+ self.get_commitish(commitish_string).peel(pygit2.Commit).id
1510+ )
1511+ return self._maybe_quiltify_tree_hash(commit_hash)
1512+
1513+ def maybe_changelogify_tree_hash(self, commit_hash):
1514+ """Determine if changelogify is needed and yield the changelogify'd tree hash
1515+
1516+ Given a commit, we need to detect if the user has inserted a
1517+ changelog entry relative to a published version for the purpose
1518+ of test builds.
1519+
1520+ Arguments:
1521+ @commit_hash: a string Git commit hash
1522+
1523+ Returns:
1524+ String tree hash of changelogify'ing @commit_hash.
1525+ If no changelogify is needed, the return value is the tree hash of
1526+ @commit_hash.
1527+ """
1528+ commit_tree_hash = str(
1529+ self.raw_repo.get(commit_hash).peel(pygit2.Tree).id
1530+ )
1531+
1532+ # one of these are the "base" pkg that @commit_hash's changes
1533+ # are based on
1534+ remote_tag = self.nearest_tag(
1535+ commit_hash,
1536+ prefix='pkg/',
1537+ )
1538+ remote_branch = gitubuntu.lint.derive_target_branch(
1539+ self,
1540+ commit_hash,
1541+ )
1542+
1543+ assert remote_tag or remote_branch
1544+
1545+ if remote_tag:
1546+ if remote_branch:
1547+ try:
1548+ self.git_run(
1549+ [
1550+ 'merge-base',
1551+ '--is-ancestor',
1552+ remote_tag.name,
1553+ remote_branch,
1554+ ],
1555+ verbose_on_failure=False,
1556+ )
1557+ parent_ref = remote_branch
1558+ except CalledProcessError as e:
1559+ if e.returncode == 1:
1560+ parent_ref = remote_tag.name
1561+ else:
1562+ raise
1563+ else:
1564+ parent_ref = remote_tag.name
1565+ else:
1566+ parent_ref = remote_branch
1567+
1568+ # If there are any changes relative to parent_ref but there are
1569+ # not any changelog changes, insert a snapshot changelog entry,
1570+ # starting from parent_ref, and return the resulting tree hash.
1571+ if str(self.raw_repo.revparse_single(parent_ref).peel(
1572+ pygit2.Tree
1573+ ).id) != commit_tree_hash and self.paths_are_identical(
1574+ parent_ref,
1575+ commit_hash,
1576+ 'debian/changelog',
1577+ ):
1578+ with self.temporary_worktree(commit_hash):
1579+ run(
1580+ [
1581+ 'gbp',
1582+ 'dch',
1583+ '--snapshot',
1584+ '--ignore-branch',
1585+ '--since=%s' % str(parent_ref),
1586+ ]
1587+ )
1588+ return self.dir_to_tree('.')
1589+
1590+ # otherwise, return @commit_hash's tree hash
1591+ return commit_tree_hash
1592+
1593+ def quiltify_and_changelogify_tree_hash(self, commitish_string):
1594+ """Given a commitish, possibly quiltify and changelogify its tree
1595+
1596+ Definitions:
1597+ quiltify: generate a quilt patch from untracked upstream
1598+ changes
1599+ changelogify: generate a snapshot changelog entry if any
1600+ changes exist, and no new changelog entry yet exists
1601+
1602+ Arguments:
1603+ @commitish_string: string Git commitish
1604+
1605+ Returns:
1606+ string Git tree hash of quiltify-ing and changelogify-ing
1607+ @commitish_string, if needed
1608+ if neither quiltify or changelogify are needed, return
1609+ @commitish_string's tree hash
1610+ """
1611+ commit_hash = str(
1612+ self.get_commitish(commitish_string).peel(pygit2.Commit).id
1613+ )
1614+ quiltify_tree_hash = self._maybe_quiltify_tree_hash(commit_hash)
1615+ changelogify_tree_hash = self.maybe_changelogify_tree_hash(commit_hash)
1616+
1617+ quiltify_tree_obj = self.raw_repo.get(quiltify_tree_hash)
1618+ changelogify_tree_obj = self.raw_repo.get(changelogify_tree_hash)
1619+
1620+ # There are multiple ways to solve this problem, but the
1621+ # simplest is to use a TreeBuilder to merge the quiltify tree
1622+ # with the changelog from the changelogify tree
1623+ # top-level TreeBuilder
1624+ tb = self.raw_repo.TreeBuilder(quiltify_tree_obj)
1625+ te = tb.get('debian')
1626+ # TreeBuilder for debian/
1627+ dtb = self.raw_repo.TreeBuilder(self.raw_repo.get(te.id))
1628+ dtb.insert( # does not take kwargs
1629+ 'changelog', # name
1630+ changelogify_tree_obj['debian/changelog'].oid, # oid
1631+ pygit2.GIT_FILEMODE_BLOB, # attr
1632+ )
1633+ # insert can replace
1634+ tb.insert( # does not take kwargs
1635+ 'debian', # name
1636+ dtb.write(), # oid
1637+ pygit2.GIT_FILEMODE_TREE, # attr
1638+ )
1639+ return str(tb.write())
1640diff --git a/gitubuntu/importer.py b/gitubuntu/importer.py
1641index 7248ee6..f438739 100644
1642--- a/gitubuntu/importer.py
1643+++ b/gitubuntu/importer.py
1644@@ -50,6 +50,7 @@ from gitubuntu.git_repository import (
1645 import_tag,
1646 upstream_tag,
1647 PristineTarError,
1648+ is_dir_3_0_quilt,
1649 )
1650 from gitubuntu.run import decode_binary, run, runq
1651 from gitubuntu.source_information import (
1652@@ -141,6 +142,7 @@ def main(
1653 skip_orig,
1654 skip_applied,
1655 reimport,
1656+ allow_applied_failures,
1657 directory=None,
1658 dl_cache=None,
1659 user=None,
1660@@ -167,6 +169,9 @@ def main(
1661 publishes
1662 @reimport: if True, import the source package from scratch and
1663 delete/recreate the target repository.
1664+ @allow_import_failures: if True, and patches fail to apply for any
1665+ publish, that patches-applied import will be skipped rather than an
1666+ error
1667 @directory: string path for local repository
1668 @user: string user to authenticate to Launchpad as
1669 @proto: string protocol to use (one of 'http', 'https', 'git')
1670@@ -329,6 +334,7 @@ def main(
1671 workdir=workdir,
1672 fixup_devel=fixup_devel,
1673 skip_orig=skip_orig,
1674+ allow_applied_failures=allow_applied_failures,
1675 )
1676
1677 if not history_found:
1678@@ -350,6 +356,7 @@ def main(
1679 workdir=workdir,
1680 fixup_devel=fixup_devel,
1681 skip_orig=skip_orig,
1682+ allow_applied_failures=allow_applied_failures,
1683 )
1684
1685 os.chdir(oldcwd)
1686@@ -424,11 +431,11 @@ def get_changelog_for_commit(
1687 if changelog_parent_commit is not None:
1688 cmd = ['diff-tree', '-p', changelog_parent_commit,
1689 tree_hash, '--', 'debian/changelog']
1690- raw_clog_entry = repo.git_run(cmd).stdout
1691+ raw_clog_entry, _ = repo.git_run(cmd, decode=False)
1692 elif publish_parent_commit is not None:
1693 cmd = ['diff-tree', '-p', publish_parent_commit,
1694 tree_hash, '--', 'debian/changelog']
1695- raw_clog_entry = repo.git_run(cmd).stdout
1696+ raw_clog_entry, _ = repo.git_run(cmd, decode=False)
1697
1698 changelog_entry = b''
1699 changelog_entry_found = False
1700@@ -754,10 +761,9 @@ def import_patches_applied_tree(repo, dsc_pathname):
1701 repo.git_run(
1702 ['--work-tree', extracted_dir, 'rm', '-r', '-f', '.pc']
1703 )
1704- cp = repo.git_run(
1705+ import_tree_hash, _ = repo.git_run(
1706 ['--work-tree', extracted_dir, 'write-tree']
1707 )
1708- import_tree_hash = decode_binary(cp.stdout).strip()
1709 yield (
1710 import_tree_hash,
1711 None,
1712@@ -765,31 +771,15 @@ def import_patches_applied_tree(repo, dsc_pathname):
1713 )
1714
1715 try:
1716- try:
1717- cp = run(['dpkg-source', '--print-format', extracted_dir])
1718- fmt = decode_binary(cp.stdout).strip()
1719- if '3.0 (quilt)' not in fmt:
1720- raise StopIteration()
1721- except CalledProcessError as e:
1722- try:
1723- with open('debian/source/format', 'r') as f:
1724- for line in f:
1725- if re.match(r'3.0 (.*)', line):
1726- break
1727- else:
1728- raise StopIteration()
1729- # `man dpkg-source` indicates no d/s/format implies 1.0
1730- except OSError:
1731- raise StopIteration()
1732+ if not is_dir_3_0_quilt(extracted_dir):
1733+ raise StopIteration()
1734
1735 while True:
1736 try:
1737 os.chdir(extracted_dir)
1738- run(['quilt', 'push'], rcs=[2])
1739- cp = run(['quilt', 'top'])
1740- patch_name = decode_binary(cp.stdout).strip()
1741- cp = run(['quilt', 'header'])
1742- header = decode_binary(cp.stdout).strip()
1743+ run(['quilt', 'push'], verbose_on_failure=False,)
1744+ patch_name, _ = run(['quilt', 'top'])
1745+ header, _ = run(['quilt', 'header'])
1746 patch_desc = None
1747 for regex in (r'Subject:\s*(.*?)$',
1748 r'Description:\s*(.*?)$'
1749@@ -818,8 +808,9 @@ def import_patches_applied_tree(repo, dsc_pathname):
1750 '--',
1751 '.pc',
1752 ])
1753- cp = repo.git_run(['--work-tree', extracted_dir, 'write-tree'])
1754- import_tree_hash = decode_binary(cp.stdout).strip()
1755+ import_tree_hash, _ = repo.git_run(
1756+ ['--work-tree', extracted_dir, 'write-tree']
1757+ )
1758
1759 yield (import_tree_hash, patch_name, patch_desc)
1760 except CalledProcessError as e:
1761@@ -1195,14 +1186,6 @@ def import_unapplied_spi(repo, spi, namespace, skip_orig, ubuntu_sinfo):
1762 unapplied_publish_parent_commit = str(unapplied_publish_parent_head.peel().id)
1763 except AttributeError:
1764 pass
1765- finally:
1766- try:
1767- unapplied_publish_parent_tag = repo.nearest_import_tag(unapplied_publish_parent_commit, namespace)
1768- logging.debug('Publishing parent (tag) is %s',
1769- repo.tag_to_pretty_name(unapplied_publish_parent_tag)
1770- )
1771- except AttributeError:
1772- pass
1773
1774 if version_compare(str(spi.version), unapplied_tip_version) <= 0:
1775 logging.warn('Version to import (%s) is not after %s tip (%s)',
1776@@ -1265,11 +1248,15 @@ def import_unapplied_spi(repo, spi, namespace, skip_orig, ubuntu_sinfo):
1777 upload_parent_commit = str(upload_tag.peel().id)
1778 if unapplied_publish_parent_commit is not None:
1779 try:
1780- repo.git_run(['merge-base', '--is-ancestor',
1781- unapplied_publish_parent_commit,
1782- upload_parent_commit
1783- ]
1784- )
1785+ repo.git_run(
1786+ [
1787+ 'merge-base',
1788+ '--is-ancestor',
1789+ unapplied_publish_parent_commit,
1790+ upload_parent_commit,
1791+ ],
1792+ verbose_on_failure=False,
1793+ )
1794 unapplied_publish_parent_commit = None
1795 except CalledProcessError as e:
1796 if e.returncode != 1:
1797@@ -1277,11 +1264,15 @@ def import_unapplied_spi(repo, spi, namespace, skip_orig, ubuntu_sinfo):
1798
1799 if unapplied_changelog_parent_commit is not None:
1800 try:
1801- repo.git_run(['merge-base', '--is-ancestor',
1802- unapplied_changelog_parent_commit,
1803- upload_parent_commit
1804- ]
1805- )
1806+ repo.git_run(
1807+ [
1808+ 'merge-base',
1809+ '--is-ancestor',
1810+ unapplied_changelog_parent_commit,
1811+ upload_parent_commit,
1812+ ],
1813+ verbose_on_failure=False,
1814+ )
1815 unapplied_changelog_parent_commit = None
1816 except CalledProcessError as e:
1817 if e.returncode != 1:
1818@@ -1305,7 +1296,13 @@ def import_unapplied_spi(repo, spi, namespace, skip_orig, ubuntu_sinfo):
1819 spi=spi,
1820 )
1821
1822-def import_applied_spi(repo, spi, namespace, ubuntu_sinfo):
1823+def import_applied_spi(
1824+ repo,
1825+ spi,
1826+ namespace,
1827+ ubuntu_sinfo,
1828+ allow_applied_failures
1829+):
1830 """Imports a source package from Launchpad into the git
1831 repository
1832
1833@@ -1424,35 +1421,40 @@ def import_applied_spi(repo, spi, namespace, ubuntu_sinfo):
1834 # Assume no patches to apply
1835 applied_import_tree_hash = unapplied_import_tree_hash
1836 # get tree id from above commit
1837- for (
1838- applied_import_tree_hash,
1839- patch_name,
1840- patch_desc
1841- ) in import_patches_applied_tree(repo, spi.dsc_pathname):
1842- # special case for .pc removal
1843- if patch_name is None:
1844- msg = b'%b.' % (patch_desc.encode())
1845- else:
1846- if patch_desc is None:
1847- patch_desc = (
1848- "%s\n\nNo DEP3 Subject or Description header found" %
1849- patch_name
1850+ try:
1851+ for (
1852+ applied_import_tree_hash,
1853+ patch_name,
1854+ patch_desc
1855+ ) in import_patches_applied_tree(repo, spi.dsc_pathname):
1856+ # special case for .pc removal
1857+ if patch_name is None:
1858+ msg = b'%b.' % (patch_desc.encode())
1859+ else:
1860+ if patch_desc is None:
1861+ patch_desc = (
1862+ "%s\n\nNo DEP3 Subject or Description header found" %
1863+ patch_name
1864+ )
1865+ msg = b'%b\n\nGbp-Pq: %b.' % (
1866+ patch_desc.encode(),
1867+ patch_name.encode()
1868 )
1869- msg = b'%b\n\nGbp-Pq: %b.' % (
1870- patch_desc.encode(),
1871- patch_name.encode()
1872+ unapplied_parent_commit = repo.commit_tree_hash(
1873+ applied_import_tree_hash,
1874+ [unapplied_parent_commit],
1875+ msg,
1876+ spi
1877 )
1878- unapplied_parent_commit = repo.commit_tree_hash(
1879- applied_import_tree_hash,
1880- [unapplied_parent_commit],
1881- msg,
1882- spi
1883- )
1884
1885- logging.debug(
1886- "Committed patch-application of %s as %s",
1887- patch_name, unapplied_parent_commit
1888- )
1889+ logging.debug(
1890+ "Committed patch-application of %s as %s",
1891+ patch_name, unapplied_parent_commit
1892+ )
1893+ except CalledProcessError as e:
1894+ if allow_applied_failures:
1895+ return
1896+ raise
1897
1898 commit_applied_patches_import(
1899 repo,
1900@@ -1472,10 +1474,20 @@ def import_applied_spi(repo, spi, namespace, ubuntu_sinfo):
1901 spi=spi,
1902 )
1903
1904-def import_publishes(repo, pkgname, namespace, patches_applied,
1905- debian_head_versions, ubuntu_head_versions, debian_sinfo,
1906- ubuntu_sinfo, active_series_only, workdir, fixup_devel,
1907+def import_publishes(
1908+ repo,
1909+ pkgname,
1910+ namespace,
1911+ patches_applied,
1912+ debian_head_versions,
1913+ ubuntu_head_versions,
1914+ debian_sinfo,
1915+ ubuntu_sinfo,
1916+ active_series_only,
1917+ workdir,
1918+ fixup_devel,
1919 skip_orig,
1920+ allow_applied_failures,
1921 ):
1922 history_found = False
1923 only_debian = False
1924@@ -1484,13 +1496,16 @@ def import_publishes(repo, pkgname, namespace, patches_applied,
1925 _namespace = namespace
1926 namespace = '%s/applied' % namespace
1927 import_type = 'patches-applied'
1928- import_func = import_applied_spi
1929+ import_func = functools.partial(
1930+ import_applied_spi,
1931+ allow_applied_failures=allow_applied_failures,
1932+ )
1933 else:
1934 _namespace = namespace
1935 import_type = 'patches-unapplied'
1936 import_func = functools.partial(
1937 import_unapplied_spi,
1938- skip_orig=skip_orig
1939+ skip_orig=skip_orig,
1940 )
1941 for distname, versions, dist_sinfo in (
1942 ("debian", debian_head_versions, debian_sinfo),
1943@@ -1592,6 +1607,11 @@ def parse_args(subparsers=None, base_subparsers=None):
1944 help=argparse.SUPPRESS)
1945 parser.add_argument('--reimport', action='store_true',
1946 help=argparse.SUPPRESS)
1947+ parser.add_argument(
1948+ '--allow-applied-failures',
1949+ action='store_true',
1950+ help=argparse.SUPPRESS,
1951+ )
1952 if not subparsers:
1953 return parser.parse_args()
1954 return 'import - %s' % kwargs['description']
1955@@ -1631,6 +1651,7 @@ def cli_main(args):
1956 skip_orig=args.skip_orig,
1957 skip_applied=args.skip_applied,
1958 reimport=args.reimport,
1959+ allow_applied_failures=args.allow_applied_failures,
1960 directory=directory,
1961 user=user,
1962 proto=args.proto,
1963diff --git a/gitubuntu/importppa.py b/gitubuntu/importppa.py
1964index 3d5ba65..1f3283c 100644
1965--- a/gitubuntu/importppa.py
1966+++ b/gitubuntu/importppa.py
1967@@ -52,6 +52,7 @@ def main(
1968 retries,
1969 retry_backoffs,
1970 skip_orig,
1971+ allow_applied_failures,
1972 ):
1973 if re.match(r'ppa:\w+', ppa) is None:
1974 logging.error(
1975@@ -159,6 +160,7 @@ def main(
1976 spi=srcpkg_information,
1977 namespace=namespace,
1978 ubuntu_sinfo=source_information,
1979+ allow_applied_failures=allow_applied_failures,
1980 )
1981 except DownloadError:
1982 # it is non-fatal for a PPA to not have files for older
1983@@ -215,6 +217,11 @@ def parse_args(subparsers=None, base_subparsers=None):
1984 action='store_true',
1985 help=argparse.SUPPRESS,
1986 )
1987+ parser.add_argument(
1988+ '--allow-applied-failures',
1989+ action='store_true',
1990+ help=argparse.SUPPRESS,
1991+ )
1992 if not subparsers:
1993 return parser.parse_args()
1994 return 'import-ppa - %s' % kwargs['description']
1995@@ -241,4 +248,5 @@ def cli_main(args):
1996 args.retries,
1997 args.retry_backoffs,
1998 args.skip_orig,
1999+ args.allow_applied_failures,
2000 )
2001diff --git a/gitubuntu/lint.py b/gitubuntu/lint.py
2002index 0897eef..dda55e8 100644
2003--- a/gitubuntu/lint.py
2004+++ b/gitubuntu/lint.py
2005@@ -625,14 +625,39 @@ def do_change_lint(repo, commitish_string, pkg_remote_branch_string):
2006 # Don"t need most arguments here as we"re only grabbing some
2007 # data from launchpad about the active series
2008 ubuntu_source_information = GitUbuntuSourceInformation("ubuntu")
2009- # 1) does changelog in new branch only have additions at the top
2010+ # 1) Are there upstream changes not stored in a quilt patch
2011+ try:
2012+ quiltify_needed = not repo.paths_are_identical(
2013+ commitish_string,
2014+ repo.maybe_quiltify_tree_hash(commitish_string),
2015+ 'debian/patches',
2016+ )
2017+ except ValueError:
2018+ quiltify_needed = False
2019+ if quiltify_needed:
2020+ # 1a) if so, emit an error
2021+ error(
2022+ "Upstream changes exist that have not yet "
2023+ "been converted into a quilt patch. Consider using "
2024+ "`git ubuntu build` to generate a quilt patch."
2025+ )
2026+ ret = False
2027+ elif 'applied' in pkg_remote_branch_string:
2028+ # 1b) if not, emit a warning recommending if they are targetting
2029+ # a patches-applied branch
2030+ warning(
2031+ "All upstream changes are correctly tracked in quilt "
2032+ "patches, but changes are targetting a "
2033+ "patches-applied tree."
2034+ )
2035+ # 2) does changelog in new branch only have additions at the top
2036 # relative to target
2037 ret = _check_changelog_addition(
2038 repo,
2039 pkg_remote_branch_string,
2040 commitish_string,
2041 ) and ret
2042- # 2) does versioning make sense?
2043+ # 3) does versioning make sense?
2044 if dist in ["devel", ubuntu_source_information.active_series_name_list[0]]:
2045 ret = _check_versioning(
2046 repo,
2047@@ -655,9 +680,9 @@ def do_change_lint(repo, commitish_string, pkg_remote_branch_string):
2048 else:
2049 error("Targetted distribution (%s) is not active", dist)
2050 ret = False
2051- # 3) has update-maintainer been run?
2052+ # 4) has update-maintainer been run?
2053 ret = _check_update_maintainer(repo, commitish_string) and ret
2054- # 4) does target branch match changelog?
2055+ # 5) does target branch match changelog?
2056 pkg_branch_series = pkg_remote_branch_string[len('pkg/'):].split('/')[1].split('-')[0]
2057 if dist != pkg_branch_series:
2058 error(
2059diff --git a/gitubuntu/merge.py b/gitubuntu/merge.py
2060index 3baa748..c16fe4b 100644
2061--- a/gitubuntu/merge.py
2062+++ b/gitubuntu/merge.py
2063@@ -129,15 +129,16 @@ def do_tag(repo, tag_prefix, commitish, merge_base_id, onto, force):
2064 def do_reconstruct(repo, tag_prefix, commitish, merge_base_id, force):
2065 versions=()
2066 repo.checkout_commitish(merge_base_id)
2067- cp = repo.git_run(
2068+ stdout, _ = repo.git_run(
2069 [
2070 'rev-list',
2071 '--ancestry-path',
2072 '--reverse',
2073 '%s..%s' % (merge_base_id, commitish),
2074- ]
2075+ ],
2076+ decode=False,
2077 )
2078- for commit in cp.stdout.split(b'\n'):
2079+ for commit in stdout.split(b'\n'):
2080 commit = decode_binary(commit).strip()
2081 # empty newline
2082 if len(commit) == 0:
2083@@ -153,14 +154,13 @@ def do_reconstruct(repo, tag_prefix, commitish, merge_base_id, force):
2084 args += [str(obj.id)]
2085 repo.git_run(args)
2086 try:
2087- cp = repo.git_run(['diff', '--exit-code', commitish])
2088+ repo.git_run(['diff', '--exit-code', commitish])
2089 except:
2090 logging.error(
2091 "Resulting cleaned-up commit is not "
2092 "source-identical to %s",
2093 commitish
2094 )
2095- logging.error(decode_binary(cp.stdout))
2096 raise ReconstructException("Failed to reconstruct commit.")
2097
2098 old_head_version, _ = repo.get_changelog_versions_from_treeish(commitish)
2099@@ -237,14 +237,13 @@ def do_reconstruct_changelog(repo, onto, release, bug):
2100 fp.write(' (LP: #%s)' % bug)
2101 fp.write('. Remaining changes:\n')
2102 # merge-changelogs is HEAD
2103- cp = repo.git_run(
2104+ stdout, _ = repo.git_run(
2105 ['rev-list', '--reverse', '%s..HEAD^' % onto,]
2106 )
2107- for rev in decode_binary(cp.stdout).splitlines():
2108- cp = repo.git_run(
2109+ for rev in stdout.splitlines():
2110+ out, _ = repo.git_run(
2111 ['log', '--pretty=%B', '-n', '1', rev,]
2112 )
2113- out = decode_binary(cp.stdout)
2114 look_for_marker = False
2115 if '--CL--' in out:
2116 look_for_marker = True
2117@@ -432,10 +431,10 @@ def main(
2118 return 1
2119 onto_obj = repo.get_commitish(onto)
2120
2121- cp = run(['git', 'status', '--porcelain'])
2122- if len(cp.stdout) > 0:
2123+ stdout, _ = run(['git', 'status', '--porcelain'])
2124+ if len(stdout) > 0:
2125 logging.error('Working tree must be clean to continue:')
2126- logging.error(decode_binary(cp.stdout))
2127+ logging.error(stdout)
2128 return 1
2129
2130 merge_base_id = repo.raw_repo.merge_base(
2131diff --git a/gitubuntu/remote.py b/gitubuntu/remote.py
2132index 4526e3f..6e21b5f 100644
2133--- a/gitubuntu/remote.py
2134+++ b/gitubuntu/remote.py
2135@@ -25,37 +25,66 @@ def parse_args(subparsers=None, base_subparsers=None):
2136 parser.set_defaults(func=cli_main)
2137 else:
2138 parser = argparse.ArgumentParser(**kwargs)
2139- parser.add_argument('subsubcommand',
2140- help='add - Add a launchpad user\'s repository as a remote',
2141- metavar='add',
2142- choices=['add'])
2143- parser.add_argument('user', type=str,
2144- help='Launchpad user to add as a remote')
2145- parser.add_argument('url', type=str,
2146- help='Specify URL to add as a remote. If '
2147- 'not specified, a URL is derived from '
2148- 'USER and the repository source package.',
2149- default=None,
2150- nargs='?')
2151- parser.add_argument('--directory', type=str,
2152- help='Local git directory to modify. If not '
2153- 'specified, the current directory is '
2154- 'assumed')
2155- parser.add_argument('-l', '--lp-user', type=str,
2156- help=argparse.SUPPRESS)
2157- parser.add_argument('-r', '--remote-name', type=str,
2158- help='Specify remote name. If not '
2159- 'specified the remote will be named '
2160- 'after the user argument.')
2161- parser.add_argument('--no-fetch', action='store_true',
2162- help='Do not fetch the remote after adding')
2163- parser.add_argument('--package', type=str,
2164- help='Specify the source package rather than '
2165- 'automatically determining it from '
2166- 'debian/changelog')
2167+ known_subcommands = {
2168+ 'add': 'Add a launchpad user\'s repository as a remote',
2169+ }
2170+ subsubparsers = parser.add_subparsers(
2171+ dest='subsubcommand',
2172+ help='',
2173+ # this will look off until more than one subcommand exists
2174+ metavar='%s' % '|'.join(sorted(known_subcommands.keys())),
2175+ )
2176+ add_parser = subsubparsers.add_parser(
2177+ 'add',
2178+ help=known_subcommands['add']
2179+ )
2180+ add_parser.add_argument(
2181+ 'user',
2182+ type=str,
2183+ help="Launchpad user to add as a remote. If 'debian', parse "
2184+ "pkg/ubuntu/devel:debian/control for either "
2185+ "X-Debian-Vcs-Git or Vcs-Git. If found, add it as a remote "
2186+ "named 'debian'.",
2187+ )
2188+ add_parser.add_argument(
2189+ 'url',
2190+ type=str,
2191+ help="Specify URL to add as a remote. If not specified, a URL is "
2192+ "derived from USER and the repository source package.",
2193+ default=None,
2194+ nargs='?',
2195+ )
2196+ add_parser.add_argument(
2197+ '--directory',
2198+ type=str,
2199+ help="Local git directory to modify. If not specified, the "
2200+ "current directory is assumed.",
2201+ )
2202+ add_parser.add_argument(
2203+ '-l', '--lp-user',
2204+ type=str,
2205+ help=argparse.SUPPRESS
2206+ )
2207+ add_parser.add_argument(
2208+ '-r', '--remote-name',
2209+ type=str,
2210+ help="Specify remote name. If not specified the remote will be "
2211+ "named after the user argument.",
2212+ )
2213+ add_parser.add_argument(
2214+ '--no-fetch',
2215+ action='store_true',
2216+ help="Do not fetch the remote after adding",
2217+ )
2218+ add_parser.add_argument(
2219+ '--package',
2220+ type=str,
2221+ help="Specify the source package rather than automatically "
2222+ "determining it from debian/changelog.",
2223+ )
2224 if not subparsers:
2225 return parser.parse_args()
2226- return 'remote - %s' % kwargs['description']
2227+ return "remote - %s" % kwargs['description']
2228
2229 def cli_main(args):
2230 if args.directory is not None:
2231@@ -75,6 +104,68 @@ def cli_main(args):
2232 args.proto,
2233 )
2234
2235+def do_add_debian(repo, package, no_fetch=False):
2236+ """add a Debian remote to a repository
2237+
2238+ Parses debian/control in pkg/ubuntu/devel, and uses the value of:
2239+ 1) X-Debian-Vcs-Git
2240+ or
2241+ 2) Vcs-Git, only if there is no X-Debian-Vcs-Git
2242+
2243+ as the URL for a remote named Debian.
2244+
2245+ @repo: GitUbuntuRepository to modify
2246+ @package: source package name (XXX: is this derive-able from @repo?)
2247+ @no_fetch: if True, do not fetch the remote after adding it
2248+ """
2249+ remote_name = 'debian'
2250+
2251+ control_file = None
2252+ try:
2253+ control_file = repo.extract_file_from_treeish(
2254+ treeish_string='pkg/ubuntu/devel',
2255+ filename='debian/control',
2256+ )
2257+ control_file.seek(0)
2258+ x_debian_vcs_git = None
2259+ vcs_git = None
2260+ for line in control_file:
2261+ line = decode_binary(line)
2262+ if line.startswith('X-Debian-Vcs-Git:'):
2263+ _, x_debian_vcs_git = line.split(':', 1)
2264+ x_debian_vcs_git = x_debian_vcs_git.strip()
2265+ break
2266+ if line.startswith('Vcs-Git:'):
2267+ _, vcs_git = line.split(':', 1)
2268+ vcs_git = vcs_git.strip()
2269+
2270+ if not x_debian_vcs_git and not vcs_git:
2271+ logging.error(
2272+ "Unable to find any Vcs metadata in "
2273+ "pkg/ubuntu/devel:debian/control."
2274+ )
2275+ return 1
2276+
2277+ url = x_debian_vcs_git if x_debian_vcs_git else vcs_git
2278+ repo.add_remote_by_url(remote_name, url)
2279+ finally:
2280+ if control_file:
2281+ control_file.close()
2282+
2283+ if not no_fetch:
2284+ try:
2285+ repo.fetch_remote(remote_name, verbose=True)
2286+ except GitUbuntuRepositoryFetchError:
2287+ pass
2288+
2289+ logging.debug(
2290+ "added remote '%s' -> %s",
2291+ remote_name,
2292+ repo.raw_repo.remotes[remote_name].url,
2293+ )
2294+
2295+ return 0
2296+
2297 def do_add(repo, package, user, url=None, remote_name=None, no_fetch=False):
2298 """add a remote to a repository
2299
2300@@ -94,7 +185,7 @@ def do_add(repo, package, user, url=None, remote_name=None, no_fetch=False):
2301 remote_name = user
2302
2303 if url:
2304- repo.git_run(['remote', 'add', remote_name, url])
2305+ repo.add_remote_by_url(remote_name, url)
2306 else:
2307 repo.add_remote(
2308 pkgname=package,
2309@@ -133,6 +224,9 @@ def main(
2310 @package: string name of source package
2311 @url: the string URL of the remote to add
2312 @lp_user: string user to authenticate to Launchpad as
2313+ @directory: directory containing git repository to modify
2314+ @remote_name: name of remote
2315+ @no_fetch: do not fetch remote after modifying it
2316 @proto: string protocol to use (one of 'http', 'https', 'git')
2317
2318 If package is None, it will be derived from HEAD's debian/changelog.
2319@@ -181,7 +275,10 @@ def main(
2320 return 1
2321
2322 if subcommand == 'add':
2323- return do_add(repo, package, user, url, remote_name, no_fetch)
2324+ if user == 'debian':
2325+ return do_add_debian(repo, package, no_fetch)
2326+ else:
2327+ return do_add(repo, package, user, url, remote_name, no_fetch)
2328
2329 return 1
2330
2331diff --git a/gitubuntu/review.py b/gitubuntu/review.py
2332index 8e252d5..3a02c62 100644
2333--- a/gitubuntu/review.py
2334+++ b/gitubuntu/review.py
2335@@ -91,6 +91,10 @@ def main(clone, url, add_comment):
2336 target_user
2337 )
2338 return 1
2339+ if 'applied' in target_branch:
2340+ logging.error(
2341+ "Upload tagging a patches-applied upload is not supported."
2342+ )
2343
2344 source_url = mp.source_git_repository.git_https_url
2345 source_branch = mp.source_git_path[len('refs/heads/'):]
2346diff --git a/gitubuntu/run.py b/gitubuntu/run.py
2347index 098438c..2f6fa53 100644
2348--- a/gitubuntu/run.py
2349+++ b/gitubuntu/run.py
2350@@ -54,34 +54,51 @@ def run(
2351 try:
2352 logging.debug("Executing: %s", pcmd)
2353 if input:
2354- return subprocess.run(
2355+ cp = subprocess.run(
2356 args, env=env, check=check, shell=shell,
2357 input=input,
2358 stdout=stdout, stderr=stderr)
2359 else:
2360- return subprocess.run(
2361+ cp = subprocess.run(
2362 args, env=env, check=check, shell=shell,
2363 stdout=stdout, stderr=stderr, stdin=stdin)
2364+ ret = cp
2365 except subprocess.CalledProcessError as e:
2366 not_captured = "[Not captured]\n"
2367- err = not_captured
2368 out = not_captured
2369- if stderr is subprocess.PIPE:
2370- err = e.stderr.decode(errors="replace")
2371+ err = not_captured
2372 if stdout is subprocess.PIPE:
2373- out = e.stdout.decode(errors="replace")
2374- if verbose_on_failure and e.returncode not in rcs:
2375- logging.error("Command exited %d: %s", e.returncode, pcmd)
2376- logging.error("stdout: %s",
2377- out.rstrip().replace("\n", "\n "))
2378- logging.error("stderr: %s",
2379- err.rstrip().replace("\n", "\n "))
2380- raise e
2381+ out = e.stdout.decode(errors='replace')
2382+ if stderr is subprocess.PIPE:
2383+ err = e.stderr.decode(errors='replace')
2384+ if e.returncode not in rcs:
2385+ if verbose_on_failure:
2386+ logging.error("Command exited %d: %s", e.returncode, pcmd)
2387+ logging.error(
2388+ "stdout: %s",
2389+ out.replace("\n", "\n "),
2390+ )
2391+ logging.error(
2392+ "stderr: %s",
2393+ err.replace("\n", "\n "),
2394+ )
2395+ raise e
2396+ ret = e
2397+ pass
2398
2399+ if decode:
2400+ out = decode_binary(ret.stdout)
2401+ err = decode_binary(ret.stderr)
2402+ return (
2403+ out if out is None else out.strip(),
2404+ err if err is None else err.strip(),
2405+ )
2406+ else:
2407+ return (ret.stdout, ret.stderr)
2408
2409 def decode_binary(binary, verbose=True):
2410 try:
2411- return binary.decode('utf-8')
2412+ return binary if binary is None else binary.decode('utf-8')
2413 except UnicodeDecodeError as e:
2414 if verbose:
2415 logging.warning("Failed to decode blob: %s", e)
2416diff --git a/gitubuntu/versioning.py b/gitubuntu/versioning.py
2417index 3a5dc53..7b1eb7b 100644
2418--- a/gitubuntu/versioning.py
2419+++ b/gitubuntu/versioning.py
2420@@ -4,7 +4,10 @@ import functools
2421 import logging
2422 import re
2423 import sys
2424-from gitubuntu.source_information import GitUbuntuSourceInformation
2425+from gitubuntu.source_information import (
2426+ GitUbuntuSourceInformation,
2427+ NoPublicationHistoryException,
2428+)
2429
2430 try:
2431 pkg = 'python3-debian'
2432@@ -218,10 +221,13 @@ def max_version(lp_series_object, pkgname):
2433 and series
2434 """
2435 ubuntu_source_information = GitUbuntuSourceInformation('ubuntu', pkgname)
2436- for spi in ubuntu_source_information.launchpad_versions_published(
2437- sorted_by_version=True, series=lp_series_object
2438- ):
2439- return Version(spi.version)
2440+ try:
2441+ for spi in ubuntu_source_information.launchpad_versions_published(
2442+ sorted_by_version=True, series=lp_series_object
2443+ ):
2444+ return Version(spi.version)
2445+ except NoPublicationHistoryException:
2446+ return None
2447
2448
2449 def next_sru_version_string(lp_series_object, pkgname):
2450@@ -301,6 +307,9 @@ def test_next_development_version_string(test_input, expected):
2451 (['1.0-1'], '16.04', '1.0-1ubuntu1', ['1.0-1ubuntu11'], '1.0-1ubuntu1.1'),
2452 (['1.0-1'], '16.04', '1.0-2ubuntu1', ['1.0-2ubuntu11'], '1.0-2ubuntu1.1'),
2453 ([], '16.04', '1.0-2ubuntu1', ['2.0-2ubuntu1'], '1.0-2ubuntu1.1'),
2454+ (['1.0.1-0ubuntu0.16.04.1', None, None,], '17.04', '1.0.1-0ubuntu1', ['1.0.1-1ubuntu2', '1.0.1-1ubuntu2',], '1.0.1-0ubuntu1.1'),
2455+ ([None, None,], '17.04', '1.0.1-0ubuntu1', ['1.0.1-1ubuntu2', '1.0.1-1ubuntu2',], '1.0.1-0ubuntu1.1'),
2456+ ([None, None,], '17.04', '1.0.1-0ubuntu1', [None, None,], '1.0.1-0ubuntu1.1'),
2457 ])
2458 def test__next_sru_version(befores, series_string, current, afters, expected):
2459 assert _next_sru_version(
2460diff --git a/man/man1/git-ubuntu-build-source.1 b/man/man1/git-ubuntu-build-source.1
2461index a9131d8..6ee03ec 100644
2462--- a/man/man1/git-ubuntu-build-source.1
2463+++ b/man/man1/git-ubuntu-build-source.1
2464@@ -7,7 +7,8 @@ git-ubuntu build-source \- Build source packages
2465 .NF
2466 \fIgit ubuntu build-source\fR [\-\-dl-cache <dl_cache>] [\-\-sign]
2467 [\-\-for-merge] [\-\-no-pristine-tar] [\-\-no-lxd] [\-\-keep-build-env]
2468-[\-\-lxd-profile <profile>] [\-\-lxd-image <image>] [-- \&.\&.\&.]
2469+[\-\-lxd-profile <profile>] [\-\-lxd-image <image>]
2470+[\-\-commitish <commitish>] [-- \&.\&.\&.]
2471 .FI
2472 .SP
2473 .SH "DESCRIPTION"
2474@@ -83,6 +84,12 @@ By default, the image is derived from the distribution and series
2475 targetted by debian/changelog of the source package to build\&.
2476 .RE
2477 .PP
2478+\-\-commitish <commitish>
2479+.RS 4
2480+The commitish to build in the Git repository\&.
2481+If not specified, HEAD will be built\&.
2482+.RE
2483+.PP
2484 \-\- \&.\&.\&.
2485 .RS 4
2486 Any arguments after "--" are passed on verbatim to
2487diff --git a/man/man1/git-ubuntu-build.1 b/man/man1/git-ubuntu-build.1
2488index 3a24ccf..9b719cb 100644
2489--- a/man/man1/git-ubuntu-build.1
2490+++ b/man/man1/git-ubuntu-build.1
2491@@ -7,7 +7,8 @@ git-ubuntu build \- Build packages
2492 .NF
2493 \fIgit ubuntu build\fR [\-\-dl-cache <dl_cache>] [\-\-sign]
2494 [\-\-for-merge] [\-\-no-pristine-tar] [\-\-no-lxd] [\-\-keep-build-env]
2495-[\-\-lxd-profile <profile>] [\-\-lxd-image <image>] [-- \&.\&.\&.]
2496+[\-\-lxd-profile <profile>] [\-\-lxd-image <image>]
2497+[\-\-commitish <commitish>] [-- \&.\&.\&.]
2498 .FI
2499 .SP
2500 .SH "DESCRIPTION"
2501@@ -81,6 +82,12 @@ By default, the image is derived from the distribution and series
2502 targetted by debian/changelog of the source package to build\&.
2503 .RE
2504 .PP
2505+\-\-commitish <commitish>
2506+.RS 4
2507+The commitish to build in the Git repository\&.
2508+If not specified, HEAD will be built\&.
2509+.RE
2510+.PP
2511 \-\- \&.\&.\&.
2512 .RS 4
2513 Any arguments after "--" are passed on verbatim to
2514diff --git a/man/man1/git-ubuntu-import-ppa.1 b/man/man1/git-ubuntu-import-ppa.1
2515index d302464..318cd5a 100644
2516--- a/man/man1/git-ubuntu-import-ppa.1
2517+++ b/man/man1/git-ubuntu-import-ppa.1
2518@@ -7,7 +7,7 @@ git-ubuntu import-ppa\- Import a DSC file to Git
2519 .NF
2520 \fIgit ubuntu import-ppa\fR [\-l | \-\-lp-user <user>]
2521 [\-d | \-\-directory <directory>] [\-\-skip-orig]
2522-[\-\-dl-cache <dl_cache>] <ppa> <package>
2523+[\-\-allow-applied-failures] [\-\-dl-cache <dl_cache>] <ppa> <package>
2524 .FI
2525 .SP
2526 .SH "DESCRIPTION"
2527@@ -48,6 +48,21 @@ This option can be useful for \fBgit-ubuntu\fR(1) developers when
2528 debugging issues with "gbp" or "pristine-tar"\&.
2529 .RE
2530 .PP
2531+\-\-allow-applied-failures
2532+.RS 4
2533+Do not treat a patch application as a hard failure when importing the
2534+patches-applied history\&.
2535+As \fBdpkg-source\fR(1)'s (or other tool's) behavior has evolved over
2536+time, it is possible that the version used by \fBgit ubuntu import\fR
2537+is not able to reproduce the patch application of a historical
2538+publication\&.
2539+By default, such a failure will stop the historical import for the
2540+patches-applied branch, to allow a \fBgit ubuntu\fR developer to
2541+investigate\&.
2542+After investigation, this flag can be used to indicate the importer is
2543+allowed to ignore such a failure\&.
2544+.RE
2545+.PP
2546 <ppa>
2547 .RS 4
2548 The name of the PPA to import from\&.
2549diff --git a/man/man1/git-ubuntu-import.1 b/man/man1/git-ubuntu-import.1
2550index 129ecd9..c73f7ce 100644
2551--- a/man/man1/git-ubuntu-import.1
2552+++ b/man/man1/git-ubuntu-import.1
2553@@ -9,7 +9,7 @@ git-ubuntu import \- Import Launchpad publishing history to Git
2554 <user>] [\-\-dl-cache <dl_cache>] [\-\-no-fetch] [\-\-no-push]
2555 [\-\-no-clean] [\-d | \-\-directory <directory>] [\-\-fixup-devel]
2556 [\-\-active-series-only] [\-\-skip-applied] [\-\-skip-orig]
2557-[\-\-reimport] <package>
2558+[\-\-reimport] [\-\-allow-applied-failures] <package>
2559 .FI
2560 .SP
2561 .SH "DESCRIPTION"
2562@@ -198,6 +198,21 @@ occurred) and forcibly \fBgit-push\fR the resulting repository to
2563 .RE
2564 .RE
2565 .PP
2566+\-\-allow-applied-failures
2567+.RS 4
2568+Do not treat a patch application as a hard failure when importing the
2569+patches-applied history\&.
2570+As \fBdpkg-source\fR(1)'s (or other tool's) behavior has evolved over
2571+time, it is possible that the version used by \fBgit ubuntu import\fR
2572+is not able to reproduce the patch application of a historical
2573+publication\&.
2574+By default, such a failure will stop the historical import for the
2575+patches-applied branch, to allow a \fBgit ubuntu\fR developer to
2576+investigate\&.
2577+After investigation, this flag can be used to indicate the importer is
2578+allowed to ignore such a failure\&.
2579+.RE
2580+.PP
2581 <package>
2582 .RS 4
2583 The name of the source package to import\&.
2584diff --git a/man/man1/git-ubuntu.1 b/man/man1/git-ubuntu.1
2585index 09230bb..9817e87 100644
2586--- a/man/man1/git-ubuntu.1
2587+++ b/man/man1/git-ubuntu.1
2588@@ -43,7 +43,7 @@ Useful for debugging and seeing what underlying commands are run\&.
2589 \-\-retries <num_retries>
2590 .RS 4
2591 Number of retries to attempt for network interactions with Launchpad\&.
2592-The default is 3 retry attempts\&.
2593+The default is 5 retry attempts\&.
2594 .RE
2595 .PP
2596 \-\-retry-backoffs <backoff,\&.\&.\&.>
2597diff --git a/setup.py b/setup.py
2598index 75222db..ab91ab9 100644
2599--- a/setup.py
2600+++ b/setup.py
2601@@ -15,6 +15,7 @@ setup(name='gitubuntu',
2602 'python-debian==0.1.31',
2603 'pygit2==0.24.2',
2604 'launchpadlib==1.10.5',
2605+ 'petname',
2606 'setuptools',
2607 'keyrings.alt==2.2',
2608 'lazr.restfulclient==0.13.5',
2609diff --git a/snap/snapcraft.yaml b/snap/snapcraft.yaml
2610index b18d430..6ddda16 100644
2611--- a/snap/snapcraft.yaml
2612+++ b/snap/snapcraft.yaml
2613@@ -25,6 +25,10 @@ apps:
2614 environment:
2615 LD_LIBRARY_PATH: "$SNAP/usr/lib/man-db:$SNAP/lib/x86_64-linux-gnu:$SNAP/usr/lib/x86_64-linux-gnu"
2616 PATH: "$SNAP/usr/local/sbin:$SNAP/usr/local/bin:$SNAP/usr/sbin:$SNAP/usr/bin:$SNAP/sbin:$SNAP/bin:$CORE_SNAP_PATH"
2617+ reconstruct-changelog:
2618+ command: wrappers/git-reconstruct-changelog
2619+ merge-changelogs:
2620+ command: wrappers/git-merge-changelogs
2621
2622 parts:
2623 gbp:
2624@@ -206,57 +210,6 @@ parts:
2625 stage-packages:
2626 - distro-info
2627 after: [python3]
2628- libgpg-error:
2629- plugin: autotools
2630- source: https://gnupg.org/ftp/gcrypt/libgpg-error/libgpg-error-1.27.tar.bz2
2631- source-type: tar
2632- configflags: [--prefix=/usr]
2633- stage:
2634- - -usr/share/info
2635- libgcrypt:
2636- plugin: autotools
2637- source: https://gnupg.org/ftp/gcrypt/libgcrypt/libgcrypt-1.8.1.tar.bz2
2638- source-type: tar
2639- configflags: [--prefix=/usr]
2640- stage:
2641- - -usr/share/info
2642- libassuan:
2643- plugin: autotools
2644- source: https://gnupg.org/ftp/gcrypt/libassuan/libassuan-2.4.3.tar.bz2
2645- source-type: tar
2646- configflags: [--prefix=/usr]
2647- after: [libgpg-error]
2648- stage:
2649- - -usr/share/info
2650- libksba:
2651- plugin: autotools
2652- source: https://gnupg.org/ftp/gcrypt/libksba/libksba-1.3.5.tar.bz2
2653- source-type: tar
2654- configflags: [--prefix=/usr]
2655- stage:
2656- - -usr/share/info
2657- npth:
2658- plugin: autotools
2659- source: https://gnupg.org/ftp/gcrypt/npth/npth-1.5.tar.bz2
2660- source-type: tar
2661- configflags: [--prefix=/usr]
2662- stage:
2663- - -usr/share/info
2664- gnupg2:
2665- plugin: autotools
2666- source: https://gnupg.org/ftp/gcrypt/gnupg/gnupg-2.2.1.tar.bz2
2667- source-type: tar
2668- configflags:
2669- - --prefix=/usr
2670- - --enable-gpg2-is-gpg
2671- after:
2672- - libgpg-error
2673- - libgcrypt
2674- - libassuan
2675- - libksba
2676- - npth
2677- stage:
2678- - -usr/share/info
2679 devscripts:
2680 plugin: nil
2681 stage-packages:
2682@@ -276,7 +229,6 @@ parts:
2683 - -usr/bin/xzgrep
2684 - -usr/bin/xzless
2685 - -usr/bin/xzmore
2686- after: [gnupg2]
2687 git-ubuntu:
2688 plugin: python
2689 python-version: python3
2690@@ -299,14 +251,6 @@ parts:
2691 - -lib/python3.6/site-packages/oauth*
2692 - -lib/python3.6/site-packages/launchpadlib*
2693 - -usr/lib/python3.6
2694- - -usr/bin/gpg-error*
2695- - -usr/share/aclocal/gpg-error.m4
2696- - -usr/bin/dumpsexp
2697- - -usr/bin/hmac256
2698- - -usr/bin/libgcrypt-config
2699- - -usr/bin/mpicalc
2700- - -usr/include/gcrypt.h
2701- - -usr/share/aclocal/libgcrypt.m4
2702 prime:
2703 - -usr/share/doc
2704 install: |
2705diff --git a/snap/wrappers/git-merge-changelogs b/snap/wrappers/git-merge-changelogs
2706new file mode 100755
2707index 0000000..89af38d
2708--- /dev/null
2709+++ b/snap/wrappers/git-merge-changelogs
2710@@ -0,0 +1,7 @@
2711+#!/snap/core/current/bin/bash
2712+
2713+CORE_SNAP="/snap/core/current"
2714+CORE_SNAP_PATH="$CORE_SNAP/usr/local/sbin:$CORE_SNAP/usr/local/bin:$CORE_SNAP/usr/sbin:$CORE_SNAP/usr/bin:$CORE_SNAP/sbin:$CORE_SNAP/bin"
2715+export PATH="$SNAP/bin/snap:$SNAP/usr/local/sbin:$SNAP/usr/local/bin:$SNAP/usr/sbin:$SNAP/usr/bin:$SNAP/sbin:$SNAP/bin:$CORE_SNAP_PATH"
2716+export LD_LIBRARY_PATH="$SNAP/lib/x86_64-linux-gnu:$SNAP/usr/lib/x86_64-linux-gnu:$CORE_SNAP/lib/x86_64-linux-gnu:$CORE_SNAP/usr/lib/x86_64-linux-gnu"
2717+exec "$SNAP/bin/git-merge-changelogs" "$@"
2718diff --git a/snap/wrappers/git-reconstruct-changelog b/snap/wrappers/git-reconstruct-changelog
2719new file mode 100755
2720index 0000000..356b1eb
2721--- /dev/null
2722+++ b/snap/wrappers/git-reconstruct-changelog
2723@@ -0,0 +1,7 @@
2724+#!/snap/core/current/bin/bash
2725+
2726+CORE_SNAP="/snap/core/current"
2727+CORE_SNAP_PATH="$CORE_SNAP/usr/local/sbin:$CORE_SNAP/usr/local/bin:$CORE_SNAP/usr/sbin:$CORE_SNAP/usr/bin:$CORE_SNAP/sbin:$CORE_SNAP/bin"
2728+export PATH="$SNAP/bin/snap:$SNAP/usr/local/sbin:$SNAP/usr/local/bin:$SNAP/usr/sbin:$SNAP/usr/bin:$SNAP/sbin:$SNAP/bin:$CORE_SNAP_PATH"
2729+export LD_LIBRARY_PATH="$SNAP/lib/x86_64-linux-gnu:$SNAP/usr/lib/x86_64-linux-gnu:$CORE_SNAP/lib/x86_64-linux-gnu:$CORE_SNAP/usr/lib/x86_64-linux-gnu"
2730+exec "$SNAP/bin/git-reconstruct-changelog" "$@"

Subscribers

People subscribed via source and target branches