Merge ~nacc/git-ubuntu:refactor-main-v2 into git-ubuntu:master

Proposed by Nish Aravamudan
Status: Merged
Approved by: Nish Aravamudan
Approved revision: 6f347d12bf6d133b05e5729f2fa09d5132662d99
Merged at revision: b57968371be53c39948ea6d5c5e46ac5d9738930
Proposed branch: ~nacc/git-ubuntu:refactor-main-v2
Merge into: git-ubuntu:master
Prerequisite: ~nacc/git-ubuntu:bugfixes-lint-clean
Diff against target: 8236 lines (+4156/-3550)
13 files modified
gitubuntu/__main__.py (+139/-99)
gitubuntu/build.py (+127/-117)
gitubuntu/buildsource.py (+55/-59)
gitubuntu/clone.py (+136/-105)
gitubuntu/importer.py (+1389/-1178)
gitubuntu/importlocal.py (+312/-260)
gitubuntu/importppa.py (+227/-175)
gitubuntu/lint.py (+570/-532)
gitubuntu/merge.py (+423/-355)
gitubuntu/queue.py (+299/-281)
gitubuntu/remote.py (+158/-100)
gitubuntu/submit.py (+189/-176)
gitubuntu/tag.py (+132/-113)
Reviewer Review Type Date Requested Status
Server Team CI bot continuous-integration Approve
Andreas Hasenack Approve
Robie Basak Pending
Review via email: mp+330468@code.launchpad.net

This proposal supersedes a proposal from 2017-09-08.

Description of the change

This is admittedly a messy patchset. I am tired of rebasing it, though, and think we can land it in roughly this state (even if only in the edge channel of the snap) and then decide what to do from there after testing.

A ton of change, but we are now pylint3 clean for errors except for pygit2 imports:

$ pylint3 -E gitubuntu --ignored-modules=pygit2
No config file found, using default configuration

All unit tests still pass and I've run a from-scratch import, a clone and a build using the resulting git repository.

To post a comment you must log in.
Revision history for this message
Server Team CI bot (server-team-bot) wrote : Posted in a previous version of this proposal

FAILED: Continuous integration, rev:bd7e040556cd86c24859dea6af7d56fbb92954ba
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/24/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Tests
    FAILED: Build

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/24/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote : Posted in a previous version of this proposal

FAILED: Continuous integration, rev:a54e9ee31944d9af5820b98bf66e12329408c788
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/25/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Tests
    FAILED: Build

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/25/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Nish Aravamudan (nacc) wrote : Posted in a previous version of this proposal

The overall structure of this is good, I'm just still fine-tuning.

Revision history for this message
Andreas Hasenack (ahasenack) wrote : Posted in a previous version of this proposal

I see help_text being built several times in gitubuntu/__main__.py, but not being used:

        help_text = '\n'.join( ...
        help_text += '\n'
        help_text += '<pkgname> ...
        help_text += ...
        parser.add_argument(
            '-P', '--parentfile',
            type=str,
            help=argparse.SUPPRESS,
            default=pkg_resources.resource_filename(
                'gitubuntu',
                'parent_overrides.txt',
            ),
        )
        help_text = '\n'.join(... <--- starts over

Maybe you meant to use it in parser.add_argument()?
        parser.add_argument(
            '-P', '--parentfile',
            type=str,
            help=help_text, <---------
            default=pkg_resources.resource_filename(
                'gitubuntu',
                'parent_overrides.txt',
            ),
        )

Revision history for this message
Andreas Hasenack (ahasenack) wrote : Posted in a previous version of this proposal

Ok, first pass on gitubuntu/__main__.py.

These are notes to myself regarding the changes that I reviewed and saw:
- switch to use import_module
- code reformatting (probably as a result of lint)
- switch from calling the module's class instance method to calling the module's parse_args() method, which also sets args.func to be m.cli_main() via parser.set_defaults() (to be verified in each module still in this review)
- cleaner setting of debug logging

The whole buildup of the parser is still a bit confusing to me, but I can see that in this branch it was must moved around for better formatting and not really changed. Some of this could probably me moved to methods/functions at some point to make it easier to read. Not something for this branch at this time.

The review continues! :)

Revision history for this message
Nish Aravamudan (nacc) wrote : Posted in a previous version of this proposal

On Fri, Sep 8, 2017 at 1:00 PM, Andreas Hasenack <email address hidden> wrote:
> Ok, first pass on gitubuntu/__main__.py.
>
> These are notes to myself regarding the changes that I reviewed and saw:
> - switch to use import_module
> - code reformatting (probably as a result of lint)

One of our goals for 1.0 is to cleanup style to follow doc/STYLE.md as
we touch code. That's why there are some stylistic changes
intereleaved.

> - switch from calling the module's class instance method to calling the module's parse_args() method, which also sets args.func to be m.cli_main() via parser.set_defaults() (to be verified in each module still in this review)

We always called instance parse_args then the implicit func via the default.

Now, we set call a module's parse_args and then the implicit func via
the default.

> - cleaner setting of debug logging
>
> The whole buildup of the parser is still a bit confusing to me, but I can see that in this branch it was must moved around for better formatting and not really changed. Some of this could probably me moved to methods/functions at some point to make it easier to read. Not something for this branch at this time.
>
> The review continues! :)
>
>
> Diff comments:
>
>> diff --git a/gitubuntu/__main__.py b/gitubuntu/__main__.py
>> index eb6078a..49373a7 100644
>> --- a/gitubuntu/__main__.py
>> +++ b/gitubuntu/__main__.py
>> @@ -188,26 +201,30 @@ def main():
>> )
>> except CalledProcessError:
>> if isatty(sys.stdin.fileno()):
>> - user = input("gitubuntu.lpuser is not set. What is your "
>> + user = input(
>> + "gitubuntu.lpuser is not set. What is your "
>> "Launchpad username? We will set this in your "
>> "personally global git configuration for you. "
>> "To abort, press Enter.\nUsername: "
>> )
>> if len(user) == 0:
>> - logging.error('git-ubuntu requires a launchpad user')
>> + logging.error("git-ubuntu requires a launchpad user")
>> sys.exit(1)
>> else:
>> - logging.warning('Setting the Launchpad user for '
>> - 'git-ubuntu to %s with '
>> - '`git config --global gitubuntu.lpuser %s`.',
>> - user, user)
>> + logging.warning(
>> + "Setting the Launchpad user for git-ubuntu to %s "
>> + "with `git config --global gitubuntu.lpuser %s`.",
>> + user,
>> + user,
>
> Both "user" can be in the same line, and no need for the trailing comma.

Yeah, I wasn't 100% with how we wanted to handle this specific case of
style. Quoting STYLE.md:

 * If a method call goes to multiple lines, use one parameter per line, each
   ending in a trailing comma including the last parameter, and the closing
   bracket on a separate final line.

Technically, these are parameters to logging.warning, not
substitutions to the parameter string.

Revision history for this message
Andreas Hasenack (ahasenack) wrote : Posted in a previous version of this proposal

Notes for gitubuntu/build.py:
- not a class anymore
- class.parse_args -> module.parse_args:
  - parser.set_defaults points at cli_main now
- class.check_repository -> module.check_repository:
  - no change
- class.derive_orig_search_list_from_args -> module.derive_orig_search_list_from_args:
  - one change:
     if not args.no_pristine_tar:
         orig_search_list.append(
             OrigSearchListEntry(
                 mechanism=functools.partial(fetch_orig_from_pristine_tar,
+ repo=self.local_repo,
- repo=GitUbuntuRepository('.'),
  - local_repo was being set to repo=GitUbuntuRepository('.') before, so it's ok
- class.main -> split into module.cli_main which calls module.main

Other minor changes:
- derive_orig_search_list_from_args()
  - Use of GitUbuntuRepository('.') insgtead of self.local_repo
- fetch_orig_from_launchpad:
  - build: fix use of undefined variables
- fetch_orig_from_cache: tiny logging change
- derive_orig_search_list_from_args: local_repo to GitUbuntuRepository('.')
- git_repository: move derive_source_from_series from build

+1 for gitubuntu/build.py. Next up is gitubuntu/buildsource.py

Revision history for this message
Server Team CI bot (server-team-bot) wrote : Posted in a previous version of this proposal

FAILED: Continuous integration, rev:6a15a640423df39fc68adc657c7f32aef117749d
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/40/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Style Check

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/40/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote : Posted in a previous version of this proposal

FAILED: Continuous integration, rev:6a15a640423df39fc68adc657c7f32aef117749d
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/44/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    FAILED: Integration Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/44/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote : Posted in a previous version of this proposal

PASSED: Continuous integration, rev:5d0cb19e1fcbc2711d478eec9df40809d5cd8953
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/57/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/57/rebuild

review: Approve (continuous-integration)
~nacc/git-ubuntu:refactor-main-v2 updated
9cb168b... by Nish Aravamudan

lint: rework so that main is a more useful function

This will be folded up with the other changes before merging.

6f347d1... by Nish Aravamudan

remote: rework so that main is a more useful function

This will be folded up with the other changes before merging.

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:9ba1036b3937a1bdcb3b9151497bbb181cc45023
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/58/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/58/rebuild

review: Approve (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:960a19c34771f455df02253617749317749e299d
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/61/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/61/rebuild

review: Approve (continuous-integration)
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

build_source.py: OK

gitubuntu/clone.py:
cli_main() calls main() and discards its return value. main() however returns "local_repo", is that correct? It's not documented in main's docstrings, and in the previous gitubuntu/clone.py module there was no such return value.

def main(package, directory=None, lp_user=None, proto=None):
    """Entry point to clone subcommand

    @package: Name of source package
    @directory: directory to clone the repository into
    @lp_user: user to authenticate to Launchpad as
    @proto: protocol to use (one of 'http', 'https', 'git')

    If directory is None, a relative directory with the same name as
    package will be used.

    If lp_user is None, value of `git config gitubuntu.lpuser` will be
    used.

    If proto is None, the default value from gitubuntu.__main__ will be
    used (top_level_defaults.proto).
    """
(...)
    if os.path.isfile(os.path.join(directory, '.gitignore')):
        logging.warning('A .gitignore file exists in the source '
            'package. This will affect the behavior of git. Consider '
            'backing up the gitignore while working on this package '
            'to ensure all changes are tracked or passing appropriate '
            'flags to git commands (e.g., git status --ignored).'
        )

    return local_repo <-----

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

gitubuntu/importer.py:
- docstrings overall. A new branch with just docstrings fixes should address these
- import_publishes(): commented code, should either be removed (new change) or uncommented (so it's like the previous import_publishes):

            #if not patches_applied:
            raise GitUbuntuImportError(msg) from e
            #else:
            # logging.error(msg)

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

importlocal.py: ok
importppa.py
- note: parse_args() got a new option: --skip-orig
- main(): has commented code. Ideally it should be removed. It can always be brought back if needed.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

lint.py: OK, I just see a few extra checks in the diff

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

merge.py: ok
queue.py: ok
remote.py: ok
submit.py: ok
tag.py: ok

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Ah, I see that at least the new review command uses the return value of clone's main():

        repo = gitubuntu.clone.main(package=srcpkg)

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

+1

review: Approve
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:6f347d12bf6d133b05e5729f2fa09d5132662d99
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/63/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/63/rebuild

review: Approve (continuous-integration)
Revision history for this message
Nish Aravamudan (nacc) wrote :

This looks good to land now. I'm going to try and get to some more commits tomorrow that, at least, add docstring for main() in each subcommand.

That should be a noop relative to the CI pass and code review, though.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/gitubuntu/__main__.py b/gitubuntu/__main__.py
2index 165fc59..edeee43 100644
3--- a/gitubuntu/__main__.py
4+++ b/gitubuntu/__main__.py
5@@ -1,26 +1,29 @@
6 #!/usr/bin/env python3
7
8+from collections import namedtuple
9+TopLevelDefaults = namedtuple(
10+ 'TopLevelDefaults',
11+ [
12+ 'verbose',
13+ 'retries',
14+ 'retry_backoffs',
15+ 'proto',
16+ 'parentfile',
17+ 'pullfile',
18+ ],
19+)
20+top_level_defaults = object()
21+
22 def main():
23 try:
24 import argparse
25+ from importlib import import_module
26 import logging
27 from os import isatty
28 import shutil
29 from subprocess import CalledProcessError
30 import sys
31 import textwrap
32- from gitubuntu.importer import GitUbuntuImport
33- from gitubuntu.build import GitUbuntuBuild
34- from gitubuntu.buildsource import GitUbuntuBuildSource
35- from gitubuntu.merge import GitUbuntuMerge
36- from gitubuntu.clone import GitUbuntuClone
37- from gitubuntu.tag import GitUbuntuTag
38- from gitubuntu.queue import GitUbuntuQueue
39- from gitubuntu.importlocal import GitUbuntuImportLocal
40- from gitubuntu.importppa import GitUbuntuImportPPA
41- from gitubuntu.remote import GitUbuntuRemote
42- from gitubuntu.submit import GitUbuntuSubmit
43- from gitubuntu.lint import GitUbuntuLint
44 from gitubuntu.run import run
45
46 try:
47@@ -32,95 +35,128 @@ def main():
48 logging.error('Is %s installed?', pkg)
49 sys.exit(1)
50
51+ global top_level_defaults
52+ top_level_defaults = TopLevelDefaults(
53+ verbose=False,
54+ retries=3,
55+ retry_backoffs = [2 ** i for i in range(3)],
56+ proto='https',
57+ parentfile=pkg_resources.resource_filename(
58+ 'gitubuntu',
59+ 'parent_overrides.txt',
60+ ),
61+ pullfile=pkg_resources.resource_filename(
62+ 'gitubuntu',
63+ 'pull_overrides.txt',
64+ ),
65+ )
66+
67 logging.getLogger('keyring').setLevel(logging.WARNING)
68
69- known_subcommands = {'import':GitUbuntuImport,
70- 'build':GitUbuntuBuild,
71- 'build-source':GitUbuntuBuildSource,
72- 'merge':GitUbuntuMerge,
73- 'clone':GitUbuntuClone,
74- 'tag':GitUbuntuTag,
75- 'queue':GitUbuntuQueue,
76- 'import-local':GitUbuntuImportLocal,
77- 'import-ppa':GitUbuntuImportPPA,
78- 'remote':GitUbuntuRemote,
79- 'submit':GitUbuntuSubmit,
80- 'lint':GitUbuntuLint,
81- }
82+ known_subcommands = {
83+ 'import': 'gitubuntu.importer',
84+ 'build': 'gitubuntu.build',
85+ 'build-source': 'gitubuntu.buildsource',
86+ 'merge': 'gitubuntu.merge',
87+ 'clone': 'gitubuntu.clone',
88+ 'tag': 'gitubuntu.tag',
89+ 'queue': 'gitubuntu.queue',
90+ 'import-local': 'gitubuntu.importlocal',
91+ 'import-ppa': 'gitubuntu.importppa',
92+ 'remote': 'gitubuntu.remote',
93+ 'submit': 'gitubuntu.submit',
94+ 'lint': 'gitubuntu.lint',
95+ }
96
97- known_network_subcommands = {'import', 'import-local',
98- 'import-ppa', 'clone', 'build',
99- 'build-source', 'queue',
100- 'remote', 'submit'}
101+ known_network_subcommands = {
102+ 'import',
103+ 'import-local',
104+ 'import-ppa',
105+ 'clone',
106+ 'build',
107+ 'build-source',
108+ 'queue',
109+ 'remote',
110+ 'submit',
111+ }
112
113 parser = argparse.ArgumentParser(
114 description='Ubuntu git development tool',
115 formatter_class=argparse.RawTextHelpFormatter,
116- epilog='More information is available at https://wiki.ubuntu.com/UbuntuDevelopment/Merging/GitWorkflow.'
117- )
118- subparsers = parser.add_subparsers(dest='subcommand',
119- help='',
120- metavar='%s' % '|'.join(sorted(known_subcommands))
121- )
122+ epilog='More information is available at https://wiki.ubuntu.com/UbuntuDevelopment/Merging/GitWorkflow.',
123+ )
124+ subparsers = parser.add_subparsers(
125+ dest='subcommand',
126+ help='',
127+ metavar='%s' % '|'.join(sorted(known_subcommands)),
128+ )
129 # This help ends up not being very useful to the end user
130 # subparsers.required = True
131
132 # common flags to all subcommands
133 base_subparser = argparse.ArgumentParser(add_help=False)
134- base_subparser.add_argument('-v', '--verbose', action='store_true',
135- help='Increase verbosity')
136+ base_subparser.add_argument(
137+ '-v', '--verbose',
138+ action='store_true',
139+ help="Increase verbosity",
140+ default=top_level_defaults.verbose,
141+ )
142 network_base_subparser = argparse.ArgumentParser(add_help=False)
143 network_base_subparser.add_argument(
144- '--retries', type=int,
145- help='Number of times to attempt to retry '
146- 'downloading from Launchpad.',
147- default=3
148- )
149+ '--retries',
150+ type=int,
151+ help="Number of times to attempt to retry downloading from "
152+ "Launchpad.",
153+ default=top_level_defaults.retries,
154+ )
155 network_base_subparser.add_argument(
156- '--retry-backoffs',
157- type=lambda s : [int(item) for item in s.split(',')],
158- help='Comma-separated list of backoffs in '
159- 'seconds to use between each retry '
160- 'attempt. Default is exponential backoff.',
161- default=argparse.SUPPRESS
162- )
163+ '--retry-backoffs',
164+ type=lambda s : [int(item) for item in s.split(',')],
165+ help="Comma-separated list of backoffs in seconds to use "
166+ "between each retry attempt. Default is exponential "
167+ "backoff.",
168+ default=argparse.SUPPRESS,
169+ )
170 known_protos = ['git', 'http', 'https']
171 network_base_subparser.add_argument(
172- '--proto',
173- metavar='[%s]' % '|'.join(known_protos),
174- choices=known_protos,
175- help='Specify protocol to use for fetch. Default: %(default)s',
176- default='https'
177- )
178+ '--proto',
179+ metavar='[%s]' % '|'.join(known_protos),
180+ choices=known_protos,
181+ help="Specify protocol to use for fetch. Default: %(default)s",
182+ default=top_level_defaults.proto,
183+ )
184
185 width, _ = shutil.get_terminal_size()
186 subhelp_width = width - 30
187 for sub in sorted(known_subcommands):
188- s = known_subcommands[sub]()
189+ m = import_module(known_subcommands[sub])
190 if sub in known_network_subcommands:
191- help_text = s.parse_args(subparsers, [base_subparser, network_base_subparser])
192+ help_text = m.parse_args(
193+ subparsers,
194+ [base_subparser, network_base_subparser],
195+ )
196 else:
197- help_text = s.parse_args(subparsers, [base_subparser])
198+ help_text = m.parse_args(subparsers, [base_subparser])
199 subparsers.help += '\n'
200- subparsers.help += '\n'.join(textwrap.wrap(help_text,
201- subhelp_width,
202- break_on_hyphens=False
203- )
204- )
205- parser.add_argument('-P', '--parentfile', type=str,
206- help=argparse.SUPPRESS,
207- default=pkg_resources.resource_filename(
208- 'gitubuntu',
209- 'parent_overrides.txt'
210- )
211- )
212- parser.add_argument('-L', '--pullfile', type=str,
213- help=argparse.SUPPRESS,
214- default=pkg_resources.resource_filename(
215- 'gitubuntu',
216- 'pull_overrides.txt'
217- )
218- )
219+ subparsers.help += '\n'.join(
220+ textwrap.wrap(
221+ help_text,
222+ subhelp_width,
223+ break_on_hyphens=False,
224+ )
225+ )
226+ parser.add_argument(
227+ '-P', '--parentfile',
228+ type=str,
229+ help=argparse.SUPPRESS,
230+ default=top_level_defaults.parentfile,
231+ )
232+ parser.add_argument(
233+ '-L', '--pullfile',
234+ type=str,
235+ help=argparse.SUPPRESS,
236+ default=top_level_defaults.pullfile,
237+ )
238
239 argcomplete.autocomplete(parser)
240 args = parser.parse_args()
241@@ -131,22 +167,22 @@ def main():
242 # fragile, assumes developers will pass the base_subparser above
243 # should make this structural in the classes
244 if args.verbose:
245- logging.basicConfig(level=logging.DEBUG,
246- format='%(asctime)s - %(levelname)s:%(message)s',
247- datefmt='%m/%d/%Y %H:%M:%S',
248- )
249+ level=logging.DEBUG
250 else:
251- logging.basicConfig(level=logging.INFO,
252- format='%(asctime)s - %(levelname)s:%(message)s',
253- datefmt='%m/%d/%Y %H:%M:%S',
254- )
255+ level=logging.INFO
256+ logging.basicConfig(
257+ level=level,
258+ format='%(asctime)s - %(levelname)s:%(message)s',
259+ datefmt='%m/%d/%Y %H:%M:%S',
260+ )
261 if args.subcommand in known_network_subcommands:
262 try:
263 retry_backoffs = args.retry_backoffs
264 if len(retry_backoffs) != args.retries:
265- logging.error('Number of backoffs specified in '
266- '--retry-backoffs must match --retries.'
267- )
268+ logging.error(
269+ "Number of backoffs specified in "
270+ "--retry-backoffs must match --retries."
271+ )
272 sys.exit(1)
273 except AttributeError:
274 args.retry_backoffs = [2 ** i for i in range(args.retries)]
275@@ -158,26 +194,30 @@ def main():
276 )
277 except CalledProcessError:
278 if isatty(sys.stdin.fileno()):
279- user = input("gitubuntu.lpuser is not set. What is your "
280+ user = input(
281+ "gitubuntu.lpuser is not set. What is your "
282 "Launchpad username? We will set this in your "
283 "personally global git configuration for you. "
284 "To abort, press Enter.\nUsername: "
285 )
286 if len(user) == 0:
287- logging.error('git-ubuntu requires a launchpad user')
288+ logging.error("git-ubuntu requires a launchpad user")
289 sys.exit(1)
290 else:
291- logging.warning('Setting the Launchpad user for '
292- 'git-ubuntu to %s with '
293- '`git config --global gitubuntu.lpuser %s`.',
294- user, user)
295+ logging.warning(
296+ "Setting the Launchpad user for git-ubuntu to %s "
297+ "with `git config --global gitubuntu.lpuser %s`.",
298+ user,
299+ user,
300+ )
301 run(['git', 'config', '--global', 'gitubuntu.lpuser', user])
302 else:
303- logging.error('Not connected to a TTY and no '
304- 'git-ubuntu lpuser defined. Please run '
305- '`git config --global gitubuntu.lpuser '
306- '<Launchpad username>` and re-run this '
307- 'command.')
308+ logging.error(
309+ "Not connected to a TTY and no git-ubuntu lpuser "
310+ "defined. Please run `git config --global "
311+ "gitubuntu.lpuser <Launchpad username>` and re-run "
312+ "this command."
313+ )
314 sys.exit(1)
315
316 args.func(args)
317diff --git a/gitubuntu/build.py b/gitubuntu/build.py
318index a8f9cf3..d3d292e 100644
319--- a/gitubuntu/build.py
320+++ b/gitubuntu/build.py
321@@ -372,142 +372,152 @@ def fetch_orig_from_launchpad(changelog, source, pullfile, retries,
322 return None
323
324
325-class GitUbuntuBuild:
326- def parse_args(self, subparsers=None, base_subparsers=None):
327- kwargs = dict(
328- description='Build a git-ubuntu-cloned tree with dpkg-buildpackage',
329- formatter_class=argparse.ArgumentDefaultsHelpFormatter,
330- usage='%(prog)s [options] -- ...',
331- epilog='Any arguments after -- are passed to dpkg-buildpackage.'
332- )
333- if base_subparsers:
334- kwargs['parents'] = base_subparsers
335- if subparsers:
336- parser = subparsers.add_parser('build', **kwargs)
337- parser.set_defaults(func=self.main)
338- else:
339- parser = argparse.ArgumentParser(**kwargs)
340- parser.add_argument('--dl-cache', type=str,
341- help='Cache directory for downloads.',
342- default=os.path.join(os.getcwd(), '.git', CACHE_PATH),
343- )
344- parser.add_argument(
345- '--for-merge',
346- action='store_true',
347- help="Build an Ubuntu merge. Implies searching Debian for "
348- "published versions first.",
349+def check_repository():
350+ try:
351+ cp = run(['git', 'status', '--porcelain'])
352+ except CalledProcessError:
353+ logging.error('Is the current directory a git repository?')
354+ sys.exit(1)
355+
356+ if len(cp.stdout) > 0:
357+ logging.warning(
358+ "Working tree is not clean. `git status` output follows."
359 )
360- parser.add_argument(
361- '--no-pristine-tar',
362- action='store_true',
363- help="Do not use pristine-tar branches to generate orig "
364- "tarballs. This can be necessary as we stabilize the pristine-tar "
365- "data generation, particularly if multiple component tarballs are "
366- "present.",
367+ run(['git', 'status'], stdout=None)
368+ sys.exit(1)
369
370+ if not os.path.isfile('debian/changelog'):
371+ logging.error(
372+ "No debian/changelog found, is this a source "
373+ "package repository? See `git ubuntu clone`."
374 )
375- parser.add_argument('rem_args', help=argparse.SUPPRESS,
376- nargs=argparse.REMAINDER)
377- if not subparsers:
378- return parser.parse_args()
379- return 'build - %s' % kwargs['description']
380+ sys.exit(1)
381
382- def check_repository(self):
383- try:
384- cp = run(['git', 'status', '--porcelain'])
385- except CalledProcessError:
386- logging.error('Is the current directory a git repository?')
387- sys.exit(1)
388
389- if len(cp.stdout) > 0:
390- logging.warning(
391- "Working tree is not clean. `git status` output follows."
392- )
393- run(['git', 'status'], stdout=None)
394- sys.exit(1)
395+def main(search_list, changelog, rem_args):
396+ """main entry point for build
397
398- if not os.path.isfile('debian/changelog'):
399- logging.error(
400- "No debian/changelog found, is this a source "
401- "package repository? See `git ubuntu clone`."
402- )
403- sys.exit(1)
404+ @param search_list: list of OrigSearchListEntry namedtuples
405+ @param changelog: gitubuntu.git_repository.Changelog object for source package to build
406+ @param rem_args: namespace object of remaining arguments to pass onto dpkg-buildpackage
407+ """
408+ check_repository()
409+
410+ fetch_orig_and_build(search_list, changelog, rem_args)
411+
412+
413+def parse_args(subparsers=None, base_subparsers=None):
414+ kwargs = dict(
415+ description='Build a git-ubuntu-cloned tree with dpkg-buildpackage',
416+ formatter_class=argparse.ArgumentDefaultsHelpFormatter,
417+ usage='%(prog)s [options] -- ...',
418+ epilog='Any arguments after -- are passed to dpkg-buildpackage.'
419+ )
420+ if base_subparsers:
421+ kwargs['parents'] = base_subparsers
422+ if subparsers:
423+ parser = subparsers.add_parser('build', **kwargs)
424+ parser.set_defaults(func=cli_main)
425+ else:
426+ parser = argparse.ArgumentParser(**kwargs)
427+ parser.add_argument('--dl-cache', type=str,
428+ help='Cache directory for downloads.',
429+ default=os.path.join(os.getcwd(), '.git', CACHE_PATH),
430+ )
431+ parser.add_argument(
432+ '--for-merge',
433+ action='store_true',
434+ help="Build an Ubuntu merge. Implies searching Debian for "
435+ "published versions first.",
436+ )
437+ parser.add_argument(
438+ '--no-pristine-tar',
439+ action='store_true',
440+ help="Do not use pristine-tar branches to generate orig "
441+ "tarballs. This can be necessary as we stabilize the pristine-tar "
442+ "data generation, particularly if multiple component tarballs are "
443+ "present.",
444
445- def derive_orig_search_list_from_args(self, args):
446- source = 'debian' if args.for_merge else 'ubuntu'
447- orig_search_list = [
448- OrigSearchListEntry(
449- mechanism=fetch_orig_from_parent_dir,
450- source=None,
451- must_build=True,
452- ),
453- OrigSearchListEntry(
454- mechanism=fetch_orig_from_cache,
455- source=source,
456- must_build=True,
457- ),
458- ]
459- if not args.no_pristine_tar:
460- orig_search_list.append(
461- OrigSearchListEntry(
462- mechanism=functools.partial(fetch_orig_from_pristine_tar,
463- repo=self.local_repo,
464- ),
465- source=source,
466- must_build=True,
467- )
468- )
469- orig_search_list.extend([
470+ )
471+ parser.add_argument('rem_args', help=argparse.SUPPRESS,
472+ nargs=argparse.REMAINDER)
473+ if not subparsers:
474+ return parser.parse_args()
475+ return 'build - %s' % kwargs['description']
476+
477+
478+def derive_orig_search_list_from_args(args):
479+ source = 'debian' if args.for_merge else 'ubuntu'
480+ orig_search_list = [
481+ OrigSearchListEntry(
482+ mechanism=fetch_orig_from_parent_dir,
483+ source=None,
484+ must_build=True,
485+ ),
486+ OrigSearchListEntry(
487+ mechanism=fetch_orig_from_cache,
488+ source=source,
489+ must_build=True,
490+ ),
491+ ]
492+ if not args.no_pristine_tar:
493+ orig_search_list.append(
494 OrigSearchListEntry(
495- mechanism=functools.partial(fetch_orig_from_launchpad,
496- pullfile=args.pullfile,
497- retries=args.retries,
498- retry_backoffs=args.retry_backoffs,
499- dl_cache=args.dl_cache,
500+ mechanism=functools.partial(fetch_orig_from_pristine_tar,
501+ repo=GitUbuntuRepository('.'),
502 ),
503 source=source,
504 must_build=True,
505+ )
506+ )
507+ orig_search_list.extend([
508+ OrigSearchListEntry(
509+ mechanism=functools.partial(fetch_orig_from_launchpad,
510+ pullfile=args.pullfile,
511+ retries=args.retries,
512+ retry_backoffs=args.retry_backoffs,
513+ dl_cache=args.dl_cache,
514 ),
515- OrigSearchListEntry(
516- mechanism=fetch_orig_noop,
517- source=[],
518- must_build=True,
519- ),
520- ])
521+ source=source,
522+ must_build=True,
523+ ),
524+ OrigSearchListEntry(
525+ mechanism=fetch_orig_noop,
526+ source=[],
527+ must_build=True,
528+ ),
529+ ])
530
531- return orig_search_list
532+ return orig_search_list
533
534
535- def main(self, args):
536- rem_args = args.rem_args
537- if len(rem_args) > 0:
538- rem_args = rem_args[1:]
539+def cli_main(args):
540+ rem_args = args.rem_args
541+ if len(rem_args) > 0:
542+ rem_args = rem_args[1:]
543
544- self.check_repository()
545- self.local_repo = GitUbuntuRepository('.')
546+ changelog = Changelog.from_path('debian/changelog')
547
548- changelog = Changelog.from_path('debian/changelog')
549+ try:
550+ native = is_native_package(changelog)
551+ except NativenessMismatchError as e:
552+ logging.error("%s" % e)
553+ sys.exit(1)
554
555- try:
556- native = is_native_package(changelog)
557- except NativenessMismatchError as e:
558- logging.error("%s" % e)
559- sys.exit(1)
560-
561- if native:
562- # No orig tarball required
563- orig_search_list = [
564- OrigSearchListEntry(
565- mechanism=fetch_orig_noop,
566- source=[],
567- must_build=True,
568- ),
569- ]
570- else:
571- orig_search_list = self.derive_orig_search_list_from_args(args)
572+ if native:
573+ # No orig tarball required
574+ orig_search_list = [
575+ OrigSearchListEntry(
576+ mechanism=fetch_orig_noop,
577+ source=[],
578+ must_build=True,
579+ ),
580+ ]
581+ else:
582+ orig_search_list = derive_orig_search_list_from_args(args)
583
584- # See http://pad.ubuntu.com/KKB1kMR0JH for logic
585- fetch_orig_and_build(orig_search_list, changelog, rem_args)
586+ # See http://pad.ubuntu.com/KKB1kMR0JH for logic
587+ main(orig_search_list, changelog, rem_args)
588
589
590 def derive_source_from_changelog(changelog):
591diff --git a/gitubuntu/buildsource.py b/gitubuntu/buildsource.py
592index 4e9b69b..1735129 100644
593--- a/gitubuntu/buildsource.py
594+++ b/gitubuntu/buildsource.py
595@@ -5,66 +5,62 @@ import argparse
596 import logging
597 import os
598 import sys
599-from gitubuntu.build import GitUbuntuBuild
600+from gitubuntu.build import cli_main as build_cli_main
601 from gitubuntu.cache import CACHE_PATH
602
603-class GitUbuntuBuildSource(GitUbuntuBuild):
604- def __init__(self):
605- super().__init__()
606+def parse_args(subparsers=None, base_subparsers=None):
607+ kwargs = dict(
608+ description="Build a source package and changes file",
609+ formatter_class=argparse.ArgumentDefaultsHelpFormatter
610+ )
611+ if base_subparsers:
612+ kwargs['parents'] = base_subparsers
613+ if subparsers:
614+ parser = subparsers.add_parser('build-source', **kwargs)
615+ help_text = "build-source - Build a source package"
616+ parser.set_defaults(func=cli_main)
617+ else:
618+ parser = argparse.ArgumentParser(**kwargs)
619+ parser.add_argument('--dl-cache', type=str,
620+ help=("Cache directory for downloads."),
621+ default=os.path.join(os.getcwd(), '.git', CACHE_PATH),
622+ )
623+ parser.add_argument('--sign', action='store_true',
624+ help="Sign the source package.",
625+ )
626+ parser.add_argument('--for-merge', action='store_true',
627+ help=("Build an Ubuntu merge. Implies searching Debian for "
628+ "published versions first and passing -sa to "
629+ "dpkg-buildpackage."
630+ ),
631+ )
632+ parser.add_argument(
633+ '--no-pristine-tar',
634+ action='store_true',
635+ help="Do not use pristine-tar branches to generate orig "
636+ "tarballs. This can be necessary as we stabilize the pristine-tar "
637+ "data generation, particularly if multiple component tarballs are "
638+ "present.",
639
640- def parse_args(self, subparsers=None, base_subparsers=None):
641- kwargs = dict(
642- description="Build a source package and changes file",
643- formatter_class=argparse.ArgumentDefaultsHelpFormatter
644- )
645- if base_subparsers:
646- kwargs['parents'] = base_subparsers
647- if subparsers:
648- parser = subparsers.add_parser('build-source', **kwargs)
649- help_text = "build-source - Build a source package"
650- parser.set_defaults(func=self.main)
651- else:
652- parser = argparse.ArgumentParser(**kwargs)
653- parser.add_argument('--dl-cache', type=str,
654- help=("Cache directory for downloads."),
655- default=os.path.join(os.getcwd(), '.git', CACHE_PATH),
656- )
657- parser.add_argument('--sign', action='store_true',
658- help="Sign the source package.",
659- )
660- parser.add_argument('--for-merge', action='store_true',
661- help=("Build an Ubuntu merge. Implies searching Debian for "
662- "published versions first and passing -sa to "
663- "dpkg-buildpackage."
664- ),
665- )
666- parser.add_argument(
667- '--no-pristine-tar',
668- action='store_true',
669- help="Do not use pristine-tar branches to generate orig "
670- "tarballs. This can be necessary as we stabilize the pristine-tar "
671- "data generation, particularly if multiple component tarballs are "
672- "present.",
673+ )
674+ parser.add_argument('rem_args', help=argparse.SUPPRESS,
675+ nargs=argparse.REMAINDER
676+ )
677+ if not subparsers:
678+ return parser.parse_args()
679+ return "build-source - %s" % kwargs['description']
680
681- )
682- parser.add_argument('rem_args', help=argparse.SUPPRESS,
683- nargs=argparse.REMAINDER
684- )
685- if not subparsers:
686- return parser.parse_args()
687- return "build-source - %s" % kwargs['description']
688-
689- def main(self, args):
690- if len(args.rem_args) != 0:
691- logging.warning('Passing specified flags (%s) on to '
692- 'dpkg-buildpackage. Did you mean to '
693- 'call `git ubuntu build` instead?' %
694- ' '.join(args.rem_args)
695- )
696- else:
697- args.rem_args = ['--', '-S', '-nc', '-d', '-i', '-I']
698- if args.for_merge:
699- args.rem_args += ['-sa']
700- if not args.sign:
701- args.rem_args += ['-us', '-uc']
702- super().main(args)
703+def cli_main(args):
704+ if len(args.rem_args) != 0:
705+ logging.warning('Passing specified flags (%s) on to '
706+ 'dpkg-buildpackage. Did you mean to '
707+ 'call `git ubuntu build` instead?' %
708+ ' '.join(args.rem_args)
709+ )
710+ else:
711+ args.rem_args = ['--', '-S', '-nc', '-d', '-i', '-I']
712+ if args.for_merge:
713+ args.rem_args += ['-sa']
714+ if not args.sign:
715+ args.rem_args += ['-us', '-uc']
716+ build_cli_main(args)
717diff --git a/gitubuntu/clone.py b/gitubuntu/clone.py
718index 9a15c9c..a24a1a5 100644
719--- a/gitubuntu/clone.py
720+++ b/gitubuntu/clone.py
721@@ -5,6 +5,7 @@ import re
722 import shutil
723 from subprocess import CalledProcessError
724 import sys
725+from gitubuntu.__main__ import top_level_defaults
726 from gitubuntu.git_repository import (
727 GitUbuntuRepository,
728 GitUbuntuRepositoryFetchError,
729@@ -18,12 +19,108 @@ except ImportError:
730 logging.error('Is %s installed?', pkg)
731 sys.exit(1)
732
733-class GitUbuntuClone:
734- def parse_args(self, subparsers=None, base_subparsers=None):
735- kwargs = dict(
736- description='Clone a source package git repository to a directory',
737- formatter_class=argparse.RawTextHelpFormatter,
738- epilog='''
739+def copy_hooks(src, dst):
740+ try:
741+ os.mkdir(dst)
742+ except FileExistsError:
743+ pass
744+
745+ for hook in os.listdir(src):
746+ shutil.copy2(
747+ os.path.join(src, hook),
748+ dst,
749+ )
750+
751+def main(package, directory=None, lp_user=None, proto=None):
752+ """Entry point to clone subcommand
753+
754+ @package: Name of source package
755+ @directory: directory to clone the repository into
756+ @lp_user: user to authenticate to Launchpad as
757+ @proto: protocol to use (one of 'http', 'https', 'git')
758+
759+ If directory is None, a relative directory with the same name as
760+ package will be used.
761+
762+ If lp_user is None, value of `git config gitubuntu.lpuser` will be
763+ used.
764+
765+ If proto is None, the default value from gitubuntu.__main__ will be
766+ used (top_level_defaults.proto).
767+
768+ Returns the resulting GitUbuntuRepository object
769+ """
770+ directory = (
771+ os.path.abspath(directory)
772+ if directory
773+ else os.path.join(os.path.abspath(os.getcwd()), package)
774+ )
775+ if os.path.isdir(directory):
776+ logging.error('directory %s exists' % directory)
777+ sys.exit(1)
778+
779+ if proto is None:
780+ proto = top_level_defaults.proto
781+
782+ local_repo = GitUbuntuRepository(
783+ local_dir=directory,
784+ lp_user=lp_user,
785+ fetch_proto=proto,
786+ )
787+
788+ copy_hooks(
789+ os.path.join(os.path.dirname(__file__), os.path.pardir, 'hooks'),
790+ os.path.join(directory, '.git', 'hooks')
791+ )
792+
793+ try:
794+ local_repo.add_base_remotes(package)
795+ local_repo.fetch_base_remotes(verbose=True)
796+ except GitUbuntuRepositoryFetchError:
797+ logging.error("Unable to find an imported repository for %s. "
798+ "Please request an import by e-mailing "
799+ "usd-import-team@lists.launchpad.net.",
800+ package
801+ )
802+ shutil.rmtree(local_repo.local_dir)
803+ raise
804+
805+ try:
806+ local_repo.add_lpuser_remote(pkgname=package)
807+ local_repo.fetch_lpuser_remote(verbose=True)
808+
809+ logging.debug("added remote '%s' -> %s", local_repo.lp_user,
810+ local_repo.raw_repo.remotes[local_repo.lp_user].url
811+ )
812+ except GitUbuntuRepositoryFetchError:
813+ pass
814+
815+ try:
816+ local_repo.create_tracking_branch(
817+ 'ubuntu/devel',
818+ 'pkg/ubuntu/devel'
819+ )
820+ local_repo.checkout_commitish('ubuntu/devel')
821+ except:
822+ logging.error('Unable to checkout ubuntu/devel, does '
823+ 'pkg/ubuntu/devel branch exist?'
824+ )
825+
826+ if os.path.isfile(os.path.join(directory, '.gitignore')):
827+ logging.warning('A .gitignore file exists in the source '
828+ 'package. This will affect the behavior of git. Consider '
829+ 'backing up the gitignore while working on this package '
830+ 'to ensure all changes are tracked or passing appropriate '
831+ 'flags to git commands (e.g., git status --ignored).'
832+ )
833+
834+ return local_repo
835+
836+def parse_args(subparsers=None, base_subparsers=None):
837+ kwargs = dict(
838+ description='Clone a source package git repository to a directory',
839+ formatter_class=argparse.RawTextHelpFormatter,
840+ epilog='''
841 Example:
842 * clone to open-iscsi/
843 %(prog)s open-iscsi
844@@ -32,107 +129,41 @@ Example:
845 * use https rather than git protocol for read-only remotes:
846 %(prog)s --https open-iscsi
847 '''
848- )
849- if base_subparsers:
850- kwargs['parents'] = base_subparsers
851- if subparsers:
852- parser = subparsers.add_parser('clone', **kwargs)
853- parser.set_defaults(func=self.main)
854- else:
855- parser = argparse.ArgumentParser(**kwargs)
856- parser.add_argument('package', type=str,
857- help='Name of source package to clone'
858- )
859- parser.add_argument('directory', type=str,
860- help='Local directory to clone to. If not specified, a '
861- 'directory with the same name as PACKAGE will be '
862- 'used',
863- default=None,
864- nargs='?'
865- )
866- parser.add_argument('-l', '--lp-user', type=str, help=argparse.SUPPRESS)
867- if not subparsers:
868- return parser.parse_args()
869- return 'clone - %s' % kwargs['description']
870-
871- @staticmethod
872- def copy_hooks(src, dst):
873- try:
874- os.mkdir(dst)
875- except FileExistsError:
876- pass
877-
878- for hook in os.listdir(src):
879- shutil.copy2(
880- os.path.join(src, hook),
881- dst,
882- )
883-
884- def main(self, args):
885- try:
886- lp_user = args.lp_user
887- except AttributeError:
888- lp_user = None
889- directory = (
890- os.path.abspath(args.directory)
891- if args.directory
892- else os.path.join(os.path.abspath(os.getcwd()), args.package)
893- )
894+ )
895+ if base_subparsers:
896+ kwargs['parents'] = base_subparsers
897+ if subparsers:
898+ parser = subparsers.add_parser('clone', **kwargs)
899+ parser.set_defaults(func=cli_main)
900+ else:
901+ parser = argparse.ArgumentParser(**kwargs)
902+ parser.add_argument('package', type=str,
903+ help='Name of source package to clone'
904+ )
905+ parser.add_argument('directory', type=str,
906+ help='Local directory to clone to. If not specified, a '
907+ ' directory with the same name as PACKAGE will be '
908+ 'used',
909+ default=None,
910+ nargs='?'
911+ )
912+ parser.add_argument('-l', '--lp-user', type=str, help=argparse.SUPPRESS)
913+ if not subparsers:
914+ return parser.parse_args()
915+ return 'clone - %s' % kwargs['description']
916
917- if os.path.isdir(directory):
918- logging.error('directory %s exists' % directory)
919- sys.exit(1)
920+def cli_main(args):
921+ try:
922+ lp_user = args.lp_user
923+ except AttributeError:
924+ lp_user = None
925
926- local_repo = GitUbuntuRepository(
927- local_dir=directory,
928- lp_user=lp_user,
929- fetch_proto=args.proto,
930- )
931-
932- self.copy_hooks(
933- os.path.join(os.path.dirname(__file__), os.path.pardir, 'hooks'),
934- os.path.join(directory, '.git', 'hooks')
935- )
936+ main(
937+ package=args.package,
938+ directory=args.directory,
939+ lp_user=lp_user,
940+ proto=args.proto,
941+ )
942
943- try:
944- local_repo.add_base_remotes(args.package)
945- local_repo.fetch_base_remotes(verbose=True)
946- except GitUbuntuRepositoryFetchError:
947- logging.error("Unable to find an imported repository for %s. "
948- "Please request an import by e-mailing "
949- "usd-import-team@lists.launchpad.net.",
950- args.package
951- )
952- shutil.rmtree(local_repo.local_dir)
953- raise
954-
955- try:
956- local_repo.add_lpuser_remote(pkgname=args.package)
957- local_repo.fetch_lpuser_remote(verbose=True)
958-
959- logging.debug("added remote '%s' -> %s", local_repo.lp_user,
960- local_repo.raw_repo.remotes[local_repo.lp_user].url
961- )
962- except GitUbuntuRepositoryFetchError:
963- pass
964-
965- try:
966- local_repo.create_tracking_branch(
967- 'ubuntu/devel',
968- 'pkg/ubuntu/devel'
969- )
970- local_repo.checkout_commitish('ubuntu/devel')
971- except:
972- logging.error('Unable to checkout ubuntu/devel, does '
973- 'pkg/ubuntu/devel branch exist?'
974- )
975-
976- if os.path.isfile(os.path.join(directory, '.gitignore')):
977- logging.warning('A .gitignore file exists in the source '
978- 'package. This will affect the behavior of git. Consider '
979- 'backing up the gitignore while working on this package '
980- 'to ensure all changes are tracked or passing appropriate '
981- 'flags to git commands (e.g., git status --ignored).'
982- )
983
984 # vi: ts=4 expandtab
985diff --git a/gitubuntu/importer.py b/gitubuntu/importer.py
986index cce1999..db08f46 100644
987--- a/gitubuntu/importer.py
988+++ b/gitubuntu/importer.py
989@@ -26,6 +26,7 @@
990
991 import argparse
992 import atexit
993+import functools
994 import getpass
995 import logging
996 import os
997@@ -50,7 +51,12 @@ from gitubuntu.git_repository import (
998 PristineTarError,
999 )
1000 from gitubuntu.run import decode_binary, run, runq
1001-from gitubuntu.source_information import GitUbuntuSourceInformation, NoPublicationHistoryException, SourceExtractionException, launchpad_login_auth
1002+from gitubuntu.source_information import (
1003+ GitUbuntuSourceInformation,
1004+ NoPublicationHistoryException,
1005+ SourceExtractionException,
1006+ launchpad_login_auth,
1007+)
1008 from gitubuntu.version import VERSION
1009 from gitubuntu.versioning import version_compare
1010 try:
1011@@ -86,16 +92,18 @@ def dsc_to_tree_hash(repo, dsc_path):
1012 # extracted_dir.
1013
1014 extracted_dir = os.path.join(temp_dir, 'x')
1015- run(['dpkg-source',
1016- '-x',
1017- '--skip-patches',
1018- dsc_path,
1019- extracted_dir, # this is <temp_dir>/x
1020+ run(
1021+ [
1022+ 'dpkg-source',
1023+ '-x',
1024+ '--skip-patches',
1025+ dsc_path,
1026+ extracted_dir, # this is <temp_dir>/x
1027 ]
1028- )
1029+ )
1030
1031 if not os.path.exists(extracted_dir):
1032- logging.exception('No source extracted?')
1033+ logging.error("No source extracted?")
1034 raise SourceExtractionException(
1035 "Failed to find an extracted directory from "
1036 "dpkg-source -x"
1037@@ -104,1259 +112,1462 @@ def dsc_to_tree_hash(repo, dsc_path):
1038 return repo.dir_to_tree(extracted_dir)
1039
1040
1041-class GitUbuntuImport:
1042- def __init__(self):
1043- self.parent_overrides = None
1044-
1045- def parse_args(self, subparsers=None, base_subparsers=None):
1046- kwargs = dict(description='Update a launchpad git tree based upon '
1047- 'the state of the Ubuntu and Debian archives',
1048- formatter_class=argparse.ArgumentDefaultsHelpFormatter
1049- )
1050- if base_subparsers:
1051- kwargs['parents'] = base_subparsers
1052- if subparsers:
1053- parser = subparsers.add_parser('import', **kwargs)
1054- parser.set_defaults(func=self.main)
1055- else:
1056- parser = argparse.ArgumentParser(**kwargs)
1057- parser.add_argument('package', type=str,
1058- help='Which package to update in the git tree')
1059- parser.add_argument('-o', '--lp-owner', type=str,
1060- help=argparse.SUPPRESS,
1061- default='usd-import-team')
1062- parser.add_argument('-l', '--lp-user', type=str,
1063- help=argparse.SUPPRESS)
1064- parser.add_argument('--dl-cache', type=str,
1065- help=('Cache directory for downloads. Default is to '
1066- 'put downloads in <directory>/.git/%s' %
1067- CACHE_PATH
1068- ),
1069- default=argparse.SUPPRESS)
1070- parser.add_argument('--no-fetch', action='store_true',
1071- help='Do not fetch from the remote (DANGEROUS; '
1072- 'only useful for debugging); implies '
1073- '--no-push')
1074- parser.add_argument('--no-push', action='store_true',
1075- help='Do not push to the remote')
1076- parser.add_argument('--no-clean', action='store_true',
1077- help='Do not clean the temporary directory')
1078- parser.add_argument('-d', '--directory', type=str,
1079- help='Use git repository at specified location rather '
1080- 'than a temporary directory (implies --no-clean). '
1081- 'Will create the local repository if needed.',
1082- default=argparse.SUPPRESS
1083- )
1084- parser.add_argument('--fixup-devel', action='store_true',
1085- help=argparse.SUPPRESS)
1086- parser.add_argument('--active-series-only', action='store_true',
1087- help='Do an import of only active Ubuntu series '
1088- 'history, implies --no-push.')
1089- parser.add_argument('--skip-applied', action='store_true',
1090- help=argparse.SUPPRESS)
1091- parser.add_argument('--skip-orig', action='store_true',
1092- help=argparse.SUPPRESS)
1093- parser.add_argument('--reimport', action='store_true',
1094- help=argparse.SUPPRESS)
1095- if not subparsers:
1096- return parser.parse_args()
1097- return 'import - %s' % kwargs['description']
1098-
1099- def cleanup(self, no_clean, local_dir):
1100- """Recursively remove local_dir if no_clean is not True"""
1101- if not no_clean:
1102- shutil.rmtree(local_dir)
1103- else:
1104- logging.info('Leaving %s as directed' % local_dir)
1105-
1106- def get_changelog_for_commit(self, tree_hash, publish_parent_commit, changelog_parent_commit):
1107- """Extract changes to debian/changelog in this publish
1108-
1109- LP: #1633114 -- favor the changelog parent's diff
1110- """
1111- raw_clog_entry = b''
1112- if changelog_parent_commit is not None:
1113- cmd = ['diff-tree', '-p', changelog_parent_commit,
1114- tree_hash, '--', 'debian/changelog']
1115- raw_clog_entry = self.local_repo.git_run(cmd).stdout
1116- elif publish_parent_commit is not None:
1117- cmd = ['diff-tree', '-p', publish_parent_commit,
1118- tree_hash, '--', 'debian/changelog']
1119- raw_clog_entry = self.local_repo.git_run(cmd).stdout
1120-
1121- changelog_entry = b''
1122- changelog_entry_found = False
1123- i = 0
1124- for line in raw_clog_entry.split(b'\n'):
1125- # skip the header lines
1126- if i < 4:
1127- i += 1
1128- continue
1129- if not line.startswith(b'+'):
1130- # in case there are stray changelog changes, just take
1131- # the first complete section of changelog additions
1132- if changelog_entry_found:
1133- break
1134- continue
1135- line = re.sub(b'^[+]', b'', line)
1136- if not line.startswith(b' '):
1137- continue
1138- if not changelog_entry_found:
1139- changelog_entry = b'\n\nNew changelog entries:\n'
1140- changelog_entry += line + b'\n'
1141- changelog_entry_found = True
1142- return changelog_entry
1143-
1144- def _commit_import(self, spi, tree_hash, publish_parent_commit, changelog_parent_commit, upload_parent_commit, unapplied_parent_commit):
1145- """Commit a tree object into the repository with the specified
1146- parents
1147-
1148- The importer algorithm works by 'staging' the import of a
1149- publication as a tree, then using that tree to determine the
1150- appropriate parents from the publishing history and
1151- debian/changelog.
1152-
1153- Arguments:
1154- spi - A SourcePackageInformation instance for this publish
1155- tree_hash - SHA1 from git-dsc-commit --tree-only of this publish
1156- publish_parent_commit = SHA1 of publishing parent
1157- changelog_parent_commit = SHA1 of changelog parent
1158- upload_parent_commit = SHA1 of upload parent
1159- unapplied_parent_commit = SHA1 of unapplied-patches import parent
1160- """
1161- tag = None
1162-
1163- if unapplied_parent_commit:
1164- import_type = 'patches-applied'
1165- target_head_name = spi.applied_head_name(self.namespace)
1166- if self.local_repo.get_applied_tag(spi.version, self.namespace) is None:
1167- # Not imported before
1168- tag = applied_tag(spi.version, self.namespace)
1169- else:
1170- import_type = 'patches-unapplied'
1171- target_head_name = spi.head_name(self.namespace)
1172- if self.local_repo.get_import_tag(spi.version, self.namespace) is None:
1173- # Not imported before
1174- tag = import_tag(spi.version, self.namespace)
1175-
1176- # Do not show importer/ namespace to user
1177- _, _, pretty_head_name = target_head_name.partition('%s/' % self.namespace)
1178-
1179- changelog_entry = self.get_changelog_for_commit(
1180- tree_hash,
1181- publish_parent_commit,
1182- changelog_parent_commit
1183- )
1184- msg = (b'Import %s version %b to %b\n\nImported using git-ubuntu import.' %
1185- (import_type.encode(), spi.version.encode(), pretty_head_name.encode())
1186- )
1187-
1188- parents = []
1189-
1190- if publish_parent_commit is None and \
1191- changelog_parent_commit is None and \
1192- upload_parent_commit is None and \
1193- unapplied_parent_commit is None and \
1194- target_head_name in self.local_repo.local_branch_names:
1195- # No parents and not creating a new branch
1196- tag = orphan_tag(spi.version, self.namespace)
1197- else:
1198- msg += b'\n'
1199-
1200- if publish_parent_commit is not None:
1201- parents.append(publish_parent_commit)
1202- msg += b'\nPublish parent: %b' % publish_parent_commit.encode()
1203- if changelog_parent_commit is not None:
1204- parents.append(changelog_parent_commit)
1205- msg += b'\nChangelog parent: %b' % changelog_parent_commit.encode()
1206- if upload_parent_commit is not None:
1207- parents.append(upload_parent_commit)
1208- msg += b'\nUpload parent: %b' % upload_parent_commit.encode()
1209- if unapplied_parent_commit is not None:
1210- parents.append(unapplied_parent_commit)
1211- msg += b'\nUnapplied parent: %b' % unapplied_parent_commit.encode()
1212-
1213- msg += b'%b' % changelog_entry
1214-
1215- commit_hash = self.local_repo.commit_tree_hash(
1216- tree_hash,
1217- parents,
1218- msg,
1219- environment_spi=spi
1220- )
1221+def cleanup(no_clean, local_dir):
1222+ '''Remove a local directory, conditionally
1223
1224- self.local_repo.update_head_to_commit(target_head_name, commit_hash)
1225+ no_clean: if True, do not remove @local_dir and print a message instead
1226+ local_dir: directory to remove
1227+ '''
1228+ if not no_clean:
1229+ shutil.rmtree(local_dir)
1230+ else:
1231+ logging.info('Leaving %s as directed' % local_dir)
1232
1233- logging.debug('Committed %s import of %s as %s in %s',
1234- import_type, spi.version, commit_hash, pretty_head_name
1235- )
1236
1237- if tag is not None:
1238- # should be annotated to use create_tag API
1239- logging.debug('Creating tag %s pointing to %s', tag, commit_hash)
1240- self.local_repo.git_run(['tag', '-a', '-m', 'git-ubuntu import v%s' % VERSION,
1241- tag, commit_hash
1242- ]
1243- )
1244+# XXX: need a namedtuple to hold common arguments
1245+def main(pkgname, owner, no_clean, directory, user, proto, no_fetch,
1246+ no_push, dl_cache, fixup_devel, active_series_only, skip_orig,
1247+ pullfile, parentfile, retries, retry_backoffs, skip_applied,
1248+ reimport,
1249+):
1250+ if owner == 'usd-import-team':
1251+ namespace = 'importer'
1252+ else:
1253+ namespace = owner
1254
1255- def commit_unapplied_patches_import(self, spi, import_tree_hash, publish_parent_commit, changelog_parent_commit, upload_parent_commit):
1256- self._commit_import(
1257- spi,
1258- import_tree_hash,
1259- publish_parent_commit,
1260- changelog_parent_commit,
1261- upload_parent_commit,
1262- # unapplied trees do not have a distinct unapplied parent
1263- None
1264- )
1265+ logging.info('Ubuntu Server Team importer v%s' % VERSION)
1266
1267- def commit_applied_patches_import(self, spi, import_tree_hash, publish_parent_commit, changelog_parent_commit, unapplied_parent_commit):
1268- self._commit_import(
1269- spi,
1270- import_tree_hash,
1271- publish_parent_commit,
1272- changelog_parent_commit,
1273- # uploads will be unapplied trees currently, so applied trees
1274- # can not have them as direct parents
1275- None,
1276- unapplied_parent_commit
1277- )
1278+ repo = GitUbuntuRepository(directory, user, proto)
1279+ if not directory:
1280+ logging.info('Using git repository at %s', repo.local_dir)
1281+
1282+ atexit.register(cleanup, no_clean, repo.local_dir)
1283
1284- def import_dsc(self, dsc, version, dist):
1285- """Imports the dsc file to importer/{debian,ubuntu}/dsc
1286-
1287- Arguments:
1288- spi - A GitUbuntuSourcePackageInformation instance
1289- """
1290-
1291- # Add DSC file to the appropriate branch
1292- runq(['git', 'checkout', '%s/importer/%s/dsc' % (self.namespace, dist)], env=self.local_repo.env)
1293- # It is possible the same-named DSC is published multiple times
1294- # with different contents, e.g. on epoch bumps
1295- local_dir = os.path.join(self.local_repo.local_dir, version)
1296- os.makedirs(local_dir, exist_ok=True)
1297- shutil.copy(dsc.dsc_path,
1298- local_dir)
1299- self.local_repo.git_run(['add', self.local_repo.local_dir])
1300+ repo.add_base_remotes(pkgname, repo_owner=owner)
1301+ if not no_fetch and not reimport:
1302 try:
1303- # Anything to commit? (this will be 0 if, for instance, the
1304- # same dsc is being imported again)
1305- runq(['git', 'diff', '--exit-code', 'HEAD'], env=self.local_repo.env)
1306- except:
1307- self.local_repo.git_run(['commit', '-m', 'DSC file for %s' % version])
1308+ repo.fetch_base_remotes()
1309+ except GitUbuntuRepositoryFetchError:
1310+ pass
1311
1312- self.local_repo.clean_repository_state()
1313+ if not no_fetch:
1314+ repo.delete_branches_in_namespace(namespace)
1315+ repo.delete_tags_in_namespace(namespace)
1316+ repo.copy_base_references(namespace)
1317
1318- def orig_imported(self, orig_tarball_paths, dist):
1319- """Determines if a list of paths to tarballs have already been imported to @dist
1320+ if not no_fetch and not reimport:
1321+ try:
1322+ repo.fetch_remote_refspecs('pkg',
1323+ refspecs=['refs/tags/*:refs/tags/%s/*' % namespace],
1324+ )
1325+ except GitUbuntuRepositoryFetchError:
1326+ pass
1327
1328- Assumes that the pristine-tar branch exists
1329- """
1330+ if reimport:
1331+ if no_fetch:
1332+ logging.warning(
1333+ "--reimport specified with --no-fetch. Upload "
1334+ "tags would not be incorporated into the new import "
1335+ "and would be lost forever. This is not currently "
1336+ "supported and --no-fetch will be ignored for upload "
1337+ "tags."
1338+ )
1339 try:
1340- return self.local_repo.verify_pristine_tar(
1341- orig_tarball_paths,
1342- dist,
1343- self.namespace
1344+ repo.fetch_remote_refspecs('pkg',
1345+ refspecs=['refs/tags/upload/*:refs/tags/%s/upload/*' % namespace],
1346 )
1347- except PristineTarError as e:
1348- raise GitUbuntuImportOrigError from e
1349+ except GitUbuntuRepositoryFetchError:
1350+ pass
1351
1352- def import_orig(self, dsc, version, dist):
1353- """Imports the orig-tarball using gbp import-org --pristine-tar
1354+ repo.ensure_importer_branches_exist(namespace)
1355+
1356+ debian_sinfo = GitUbuntuSourceInformation(
1357+ 'debian',
1358+ pkgname,
1359+ os.path.abspath(pullfile),
1360+ retries,
1361+ retry_backoffs,
1362+ )
1363+
1364+ ubuntu_sinfo = GitUbuntuSourceInformation(
1365+ 'ubuntu',
1366+ pkgname,
1367+ os.path.abspath(pullfile),
1368+ retries,
1369+ retry_backoffs,
1370+ )
1371+
1372+ debian_head_versions = (
1373+ repo.get_heads_and_versions('debian', namespace)
1374+ )
1375+ for head_name in sorted(debian_head_versions):
1376+ _, _, pretty_head_name = head_name.partition('%s/' % namespace)
1377+ logging.debug('Last imported %s version is %s',
1378+ pretty_head_name,
1379+ debian_head_versions[head_name]['version']
1380+ )
1381
1382- Arguments:
1383- dsc - A GitUbuntuRepository object
1384- version - the string package version
1385- dist - the string distribution, either 'ubuntu' or 'debian'
1386- """
1387- try:
1388- orig_tarball_path = dsc.orig_tarball_path
1389- except GitUbuntuDscError as e:
1390- raise GitUbuntuImportOrigError(
1391- 'Unable to import orig tarball in DSC for version %s' % version
1392- ) from e
1393-
1394- if orig_tarball_path is None:
1395- # native packages only have a .tar.*
1396- native_re = re.compile(r'.*\.tar\.[^.]+$')
1397- for entry in dsc['Files']:
1398- if native_re.match(entry['name']):
1399- return
1400- raise GitUbuntuImportOrigError('No orig tarball in '
1401- 'DSC for version %s.' % version)
1402+ ubuntu_head_versions = (
1403+ repo.get_heads_and_versions('ubuntu', namespace)
1404+ )
1405+ for head_name in sorted(ubuntu_head_versions):
1406+ _, _, pretty_head_name = head_name.partition('%s/' % namespace)
1407+ logging.debug('Last imported %s version is %s',
1408+ pretty_head_name,
1409+ ubuntu_head_versions[head_name]['version']
1410+ )
1411
1412- try:
1413- orig_component_paths = dsc.component_tarball_paths
1414- except GitUbuntuDscError as e:
1415- raise GitUbuntuImportOrigError(
1416- 'Unable to import component tarballs in DSC for version '
1417- '%s' % version
1418- ) from e
1419-
1420- orig_paths = [orig_tarball_path] + list(orig_component_paths.values())
1421- if (not orig_tarball_path or
1422- self.orig_imported(orig_paths, dist)):
1423- return
1424- if not all(map(os.path.exists, orig_paths)):
1425- raise GitUbuntuImportOrigError(
1426- 'Unable to find tarball: %s' %
1427- [p for p in orig_paths if not os.path.exists(p)]
1428+ if not skip_applied:
1429+ applied_debian_head_versions = (
1430+ repo.get_heads_and_versions(
1431+ 'applied/debian', namespace
1432+ )
1433+ )
1434+ for head_name in sorted(applied_debian_head_versions):
1435+ _, _, pretty_head_name = head_name.partition(
1436+ '%s/' % namespace
1437+ )
1438+ logging.debug('Last applied %s version is %s',
1439+ pretty_head_name,
1440+ applied_debian_head_versions[head_name]['version']
1441 )
1442
1443- try:
1444- with self.local_repo.pristine_tar_branches(dist, self.namespace):
1445- oldcwd = os.getcwd()
1446- # gbp does not support running from arbitrary git trees or
1447- # working directories
1448- # https://github.com/agx/git-buildpackage/pull/16
1449- os.chdir(self.local_repo.local_dir)
1450-
1451- ext = os.path.splitext(dsc.orig_tarball_path)[1]
1452-
1453- # make this non-fatal if upstream already exist as tagged?
1454- cmd = ['gbp', 'import-orig', '--no-merge',
1455- '--upstream-branch', 'do-not-push',
1456- '--pristine-tar', '--no-interactive',
1457- '--no-symlink-orig',
1458- '--upstream-tag=%s/upstream/%s/%%(version)s%s' %
1459- (self.namespace, dist, ext)]
1460- cmd.extend(map(lambda x: '--component=%s' % x,
1461- list(orig_component_paths.keys()))
1462- )
1463- cmd.extend([orig_tarball_path,])
1464- run(cmd, env=self.local_repo.env)
1465- except CalledProcessError as e:
1466- raise GitUbuntuImportOrigError(
1467- 'Unable to import tarballs: %s' % orig_paths
1468- ) from e
1469- finally:
1470- os.chdir(oldcwd)
1471+ applied_ubuntu_head_versions = (
1472+ repo.get_heads_and_versions(
1473+ 'applied/ubuntu', namespace
1474+ )
1475+ )
1476+ for head_name in sorted(applied_ubuntu_head_versions):
1477+ _, _, pretty_head_name = head_name.partition(
1478+ '%s/' % namespace
1479+ )
1480+ logging.debug('Last applied %s version is %s',
1481+ pretty_head_name,
1482+ applied_ubuntu_head_versions[head_name]['version']
1483+ )
1484
1485- self.local_repo.clean_repository_state()
1486+ oldcwd = os.getcwd()
1487+ os.chdir(repo.local_dir)
1488+
1489+ if dl_cache is None:
1490+ workdir = os.path.join(repo.git_dir, CACHE_PATH)
1491+ else:
1492+ workdir = dl_cache
1493+
1494+ os.makedirs(workdir, exist_ok=True)
1495+
1496+ # now sets a global _PARENT_OVERRIDES
1497+ parse_parentfile(parentfile, pkgname)
1498+
1499+ only_debian, history_found = import_publishes(
1500+ repo=repo,
1501+ pkgname=pkgname,
1502+ namespace=namespace,
1503+ patches_applied=False,
1504+ debian_head_versions=debian_head_versions,
1505+ ubuntu_head_versions=ubuntu_head_versions,
1506+ debian_sinfo=debian_sinfo,
1507+ ubuntu_sinfo=ubuntu_sinfo,
1508+ active_series_only=active_series_only,
1509+ workdir=workdir,
1510+ fixup_devel=fixup_devel,
1511+ skip_orig=skip_orig,
1512+ )
1513+
1514+ if not history_found:
1515+ logging.error("No publication history for '%s' in debian or ubuntu. "
1516+ "Wrong source package name?", pkgname)
1517+ sys.exit(1)
1518+
1519+ if not skip_applied:
1520+ import_publishes(
1521+ repo=repo,
1522+ pkgname=pkgname,
1523+ namespace=namespace,
1524+ patches_applied=True,
1525+ debian_head_versions=applied_debian_head_versions,
1526+ ubuntu_head_versions=applied_ubuntu_head_versions,
1527+ debian_sinfo=debian_sinfo,
1528+ ubuntu_sinfo=ubuntu_sinfo,
1529+ active_series_only=active_series_only,
1530+ workdir=workdir,
1531+ fixup_devel=fixup_devel,
1532+ skip_orig=skip_orig,
1533+ )
1534
1535- def import_patches_unapplied_tree(self, dsc_pathname):
1536- """Imports the patches-unapplied source package and writes the
1537- corresponding working tree for a given srcpkg
1538+ os.chdir(oldcwd)
1539
1540- Arguments:
1541- dsc_pathname - A string path to a DSC file
1542- """
1543+ repo.garbage_collect()
1544
1545- import_tree_hash = dsc_to_tree_hash(self.local_repo, dsc_pathname)
1546- self.local_repo.clean_repository_state()
1547+ if no_push:
1548+ logging.info('Not pushing to remote as specified')
1549+ else:
1550+ lp = launchpad_login_auth()
1551+ repo_path = '~%s/ubuntu/+source/%s/+git/%s' % (
1552+ owner,
1553+ pkgname,
1554+ pkgname,
1555+ )
1556+ lp_git_repo = lp.git_repositories.getByPath(path=repo_path)
1557+ if reimport:
1558+ if not lp_git_repo.canBeDeleted():
1559+ logging.warning(
1560+ "There are linked MPs to the pkg repository, "
1561+ "which will be deleted:"
1562+ )
1563+ for mp in lp_git_repo.landing_candidates:
1564+ logging.warning(
1565+ "Adding comment to %s indicating reimport",
1566+ mp.web_link
1567+ )
1568+ mp.createComment(
1569+ content="%s has been reimported and this MP "
1570+ "must be manually resubmitted." %
1571+ pkgname,
1572+ subject="%s has been reimported" % pkgname,
1573+ )
1574+ # repo deletion will delete the MP, but hopefully
1575+ # the above comment will be sufficient to explain
1576+ # why
1577+ lp_git_repo.lp_delete()
1578+
1579+ repo.git_run([
1580+ 'push', '--atomic', 'pkg',
1581+ 'refs/heads/%s/*:refs/heads/*' % namespace,
1582+ 'refs/tags/%s/*:refs/tags/*' % namespace,
1583+ ])
1584+ # update our reference
1585+ lp_git_repo = lp.git_repositories.getByPath(path=repo_path)
1586+ for i in range(retries):
1587+ try:
1588+ if only_debian:
1589+ lp_git_repo.default_branch = 'refs/heads/debian/sid'
1590+ else:
1591+ lp_git_repo.default_branch = 'refs/heads/ubuntu/devel'
1592+ lp_git_repo.lp_save()
1593+ break
1594+ except (NotFound, PreconditionFailed) as e:
1595+ time.sleep(retry_backoffs[i])
1596+ lp_git_repo.lp_refresh()
1597+
1598+
1599+def get_changelog_for_commit(
1600+ repo,
1601+ tree_hash,
1602+ publish_parent_commit,
1603+ changelog_parent_commit,
1604+):
1605+ """Extract changes to debian/changelog in this publish
1606+
1607+ LP: #1633114 -- favor the changelog parent's diff
1608+ """
1609+ raw_clog_entry = b''
1610+ if changelog_parent_commit is not None:
1611+ cmd = ['diff-tree', '-p', changelog_parent_commit,
1612+ tree_hash, '--', 'debian/changelog']
1613+ raw_clog_entry = repo.git_run(cmd).stdout
1614+ elif publish_parent_commit is not None:
1615+ cmd = ['diff-tree', '-p', publish_parent_commit,
1616+ tree_hash, '--', 'debian/changelog']
1617+ raw_clog_entry = repo.git_run(cmd).stdout
1618+
1619+ changelog_entry = b''
1620+ changelog_entry_found = False
1621+ i = 0
1622+ for line in raw_clog_entry.split(b'\n'):
1623+ # skip the header lines
1624+ if i < 4:
1625+ i += 1
1626+ continue
1627+ if not line.startswith(b'+'):
1628+ # in case there are stray changelog changes, just take
1629+ # the first complete section of changelog additions
1630+ if changelog_entry_found:
1631+ break
1632+ continue
1633+ line = re.sub(b'^[+]', b'', line)
1634+ if not line.startswith(b' '):
1635+ continue
1636+ if not changelog_entry_found:
1637+ changelog_entry = b'\n\nNew changelog entries:\n'
1638+ changelog_entry += line + b'\n'
1639+ changelog_entry_found = True
1640+ return changelog_entry
1641+
1642+def _commit_import(repo, spi, tree_hash, namespace,
1643+ publish_parent_commit,
1644+ changelog_parent_commit, upload_parent_commit,
1645+ unapplied_parent_commit
1646+):
1647+ """Commit a tree object into the repository with the specified
1648+ parents
1649+
1650+ The importer algorithm works by 'staging' the import of a
1651+ publication as a tree, then using that tree to determine the
1652+ appropriate parents from the publishing history and
1653+ debian/changelog.
1654+
1655+ Arguments:
1656+ spi - A SourcePackageInformation instance for this publish
1657+ tree_hash - SHA1 from git-dsc-commit --tree-only of this publish
1658+ publish_parent_commit = SHA1 of publishing parent
1659+ changelog_parent_commit = SHA1 of changelog parent
1660+ upload_parent_commit = SHA1 of upload parent
1661+ unapplied_parent_commit = SHA1 of unapplied-patches import parent
1662+ """
1663+ tag = None
1664+
1665+ if unapplied_parent_commit:
1666+ import_type = 'patches-applied'
1667+ target_head_name = spi.applied_head_name(namespace)
1668+ if repo.get_applied_tag(spi.version, namespace) is None:
1669+ # Not imported before
1670+ tag = applied_tag(spi.version, namespace)
1671+ else:
1672+ import_type = 'patches-unapplied'
1673+ target_head_name = spi.head_name(namespace)
1674+ if repo.get_import_tag(spi.version, namespace) is None:
1675+ # Not imported before
1676+ tag = import_tag(spi.version, namespace)
1677+
1678+ # Do not show importer/ namespace to user
1679+ _, _, pretty_head_name = target_head_name.partition('%s/' % namespace)
1680+
1681+ changelog_entry = get_changelog_for_commit(
1682+ repo,
1683+ tree_hash,
1684+ publish_parent_commit,
1685+ changelog_parent_commit
1686+ )
1687+ msg = (b'Import %s version %b to %b\n\nImported using git-ubuntu import.' %
1688+ (import_type.encode(), spi.version.encode(), pretty_head_name.encode())
1689+ )
1690+
1691+ parents = []
1692+
1693+ if publish_parent_commit is None and \
1694+ changelog_parent_commit is None and \
1695+ upload_parent_commit is None and \
1696+ unapplied_parent_commit is None and \
1697+ target_head_name in repo.local_branch_names:
1698+ # No parents and not creating a new branch
1699+ tag = orphan_tag(spi.version, namespace)
1700+ else:
1701+ msg += b'\n'
1702+
1703+ if publish_parent_commit is not None:
1704+ parents.append(publish_parent_commit)
1705+ msg += b'\nPublish parent: %b' % publish_parent_commit.encode()
1706+ if changelog_parent_commit is not None:
1707+ parents.append(changelog_parent_commit)
1708+ msg += b'\nChangelog parent: %b' % changelog_parent_commit.encode()
1709+ if upload_parent_commit is not None:
1710+ parents.append(upload_parent_commit)
1711+ msg += b'\nUpload parent: %b' % upload_parent_commit.encode()
1712+ if unapplied_parent_commit is not None:
1713+ parents.append(unapplied_parent_commit)
1714+ msg += b'\nUnapplied parent: %b' % unapplied_parent_commit.encode()
1715+
1716+ msg += b'%b' % changelog_entry
1717+
1718+ commit_hash = repo.commit_tree_hash(
1719+ tree_hash,
1720+ parents,
1721+ msg,
1722+ environment_spi=spi
1723+ )
1724+
1725+ repo.update_head_to_commit(target_head_name, commit_hash)
1726+
1727+ logging.debug('Committed %s import of %s as %s in %s',
1728+ import_type, spi.version, commit_hash, pretty_head_name
1729+ )
1730
1731- return import_tree_hash
1732+ if tag is not None:
1733+ # should be annotated to use create_tag API
1734+ logging.debug('Creating tag %s pointing to %s', tag, commit_hash)
1735+ repo.git_run(['tag', '-a', '-m', 'git-ubuntu import v%s' % VERSION,
1736+ tag, commit_hash
1737+ ]
1738+ )
1739
1740- def import_patches_applied_tree(self, dsc_pathname):
1741- """Imports the patches-applied source package and writes the
1742- corresponding working tree for a given srcpkg, patch by patch
1743+def commit_unapplied_patches_import(
1744+ repo,
1745+ spi,
1746+ import_tree_hash,
1747+ namespace,
1748+ publish_parent_commit,
1749+ changelog_parent_commit,
1750+ upload_parent_commit,
1751+):
1752+ _commit_import(
1753+ repo,
1754+ spi,
1755+ import_tree_hash,
1756+ namespace,
1757+ publish_parent_commit,
1758+ changelog_parent_commit,
1759+ upload_parent_commit,
1760+ # unapplied trees do not have a distinct unapplied parent
1761+ None,
1762+ )
1763+
1764+def commit_applied_patches_import(
1765+ repo,
1766+ spi,
1767+ import_tree_hash,
1768+ namespace,
1769+ publish_parent_commit,
1770+ changelog_parent_commit,
1771+ unapplied_parent_commit
1772+):
1773+ _commit_import(
1774+ repo,
1775+ spi,
1776+ import_tree_hash,
1777+ namespace,
1778+ publish_parent_commit,
1779+ changelog_parent_commit,
1780+ # uploads will be unapplied trees currently, so applied trees
1781+ # can not have them as direct parents
1782+ None,
1783+ unapplied_parent_commit,
1784+ )
1785+
1786+def import_dsc(repo, dsc, namespace, version, dist):
1787+ """Imports the dsc file to importer/{debian,ubuntu}/dsc
1788+
1789+ Arguments:
1790+ spi - A GitUbuntuSourcePackageInformation instance
1791+ """
1792+
1793+ # Add DSC file to the appropriate branch
1794+ runq(['git', 'checkout', '%s/importer/%s/dsc' % (namespace, dist)], env=repo.env)
1795+ # It is possible the same-named DSC is published multiple times
1796+ # with different contents, e.g. on epoch bumps
1797+ local_dir = os.path.join(repo.local_dir, version)
1798+ os.makedirs(local_dir, exist_ok=True)
1799+ shutil.copy(dsc.dsc_path, local_dir)
1800+ repo.git_run(['add', repo.local_dir])
1801+ try:
1802+ # Anything to commit? (this will be 0 if, for instance, the
1803+ # same dsc is being imported again)
1804+ runq(['git', 'diff', '--exit-code', 'HEAD'], env=repo.env)
1805+ except:
1806+ repo.git_run(['commit', '-m', 'DSC file for %s' % version])
1807+
1808+ repo.clean_repository_state()
1809+
1810+def orig_imported(repo, orig_tarball_paths, namespace, dist):
1811+ """Determines if a list of paths to tarballs have already been imported to @dist
1812+
1813+ Assumes that the pristine-tar branch exists
1814+ """
1815+ try:
1816+ return repo.verify_pristine_tar(
1817+ orig_tarball_paths,
1818+ dist,
1819+ namespace,
1820+ )
1821+ except PristineTarError as e:
1822+ raise GitUbuntuImportOrigError from e
1823+
1824+def import_orig(repo, dsc, namespace, version, dist):
1825+ """Imports the orig-tarball using gbp import-org --pristine-tar
1826+
1827+ Arguments:
1828+ repo - A GitUbuntuRepository object
1829+ dsc - a GitUbuntuDsc object
1830+ namespace - string namespace to search for tags, etc.
1831+ version - the string package version
1832+ dist - the string distribution, either 'ubuntu' or 'debian'
1833+ """
1834+ try:
1835+ orig_tarball_path = dsc.orig_tarball_path
1836+ except GitUbuntuDscError as e:
1837+ raise GitUbuntuImportOrigError(
1838+ 'Unable to import orig tarball in DSC for version %s' % version
1839+ ) from e
1840+
1841+ if orig_tarball_path is None:
1842+ # native packages only have a .tar.*
1843+ native_re = re.compile(r'.*\.tar\.[^.]+$')
1844+ for entry in dsc['Files']:
1845+ if native_re.match(entry['name']):
1846+ return
1847+ raise GitUbuntuImportOrigError('No orig tarball in '
1848+ 'DSC for version %s.' % version
1849+ )
1850
1851- Arguments:
1852- dsc_pathname - A string path to a DSC file
1853- """
1854- oldcwd = os.getcwd()
1855+ try:
1856+ orig_component_paths = dsc.component_tarball_paths
1857+ except GitUbuntuDscError as e:
1858+ raise GitUbuntuImportOrigError(
1859+ 'Unable to import component tarballs in DSC for version '
1860+ '%s' % version
1861+ ) from e
1862+
1863+ orig_paths = [orig_tarball_path] + list(orig_component_paths.values())
1864+ if (not orig_tarball_path or
1865+ orig_imported(repo, orig_paths, namespace, dist)
1866+ ):
1867+ return
1868+ if not all(map(os.path.exists, orig_paths)):
1869+ raise GitUbuntuImportOrigError('Unable to find tarball: '
1870+ '%s' % [p for p in orig_paths if not os.path.exists(p)])
1871+
1872+ try:
1873+ with repo.pristine_tar_branches(dist, namespace):
1874+ oldcwd = os.getcwd()
1875+ # gbp does not support running from arbitrary git trees or
1876+ # working directories
1877+ # https://github.com/agx/git-buildpackage/pull/16
1878+ os.chdir(repo.local_dir)
1879+
1880+ ext = os.path.splitext(dsc.orig_tarball_path)[1]
1881+
1882+ # make this non-fatal if upstream already exist as tagged?
1883+ cmd = ['gbp', 'import-orig', '--no-merge',
1884+ '--upstream-branch', 'do-not-push',
1885+ '--pristine-tar', '--no-interactive',
1886+ '--no-symlink-orig',
1887+ '--upstream-tag=%s/upstream/%s/%%(version)s%s' %
1888+ (namespace, dist, ext)]
1889+ cmd.extend(map(lambda x: '--component=%s' % x,
1890+ list(orig_component_paths.keys()))
1891+ )
1892+ cmd.extend([orig_tarball_path,])
1893+ run(cmd, env=repo.env)
1894+ except CalledProcessError as e:
1895+ raise GitUbuntuImportOrigError(
1896+ 'Unable to import tarballs: %s' % orig_paths
1897+ ) from e
1898+ finally:
1899+ os.chdir(oldcwd)
1900
1901- extract_dir = tempfile.mkdtemp()
1902- os.chdir(extract_dir)
1903+ repo.clean_repository_state()
1904
1905- run(['dpkg-source', '-x',
1906- '--skip-patches',
1907- os.path.join(oldcwd, dsc_pathname)
1908- ]
1909- )
1910+def import_patches_unapplied_tree(repo, dsc_pathname):
1911+ """Imports the patches-unapplied source package and writes the
1912+ corresponding working tree for a given srcpkg
1913
1914- extracted_dir = None
1915- for path in os.listdir(extract_dir):
1916- if os.path.isdir(path):
1917- extracted_dir = os.path.join(extract_dir, path)
1918- break
1919- if extracted_dir is None:
1920- logging.exception('No source extracted?')
1921- raise SourceExtractionException("Failed to find an extracted "
1922- "directory from dpkg-source -x")
1923-
1924- if os.path.isdir(os.path.join(extracted_dir, '.pc')):
1925- self.local_repo.git_run(
1926- ['--work-tree', extracted_dir, 'add', '-f', '-A']
1927- )
1928- self.local_repo.git_run(
1929- ['--work-tree', extracted_dir, 'rm', '-r', '-f', '.pc']
1930- )
1931- cp = self.local_repo.git_run(
1932- ['--work-tree', extracted_dir, 'write-tree']
1933- )
1934- import_tree_hash = decode_binary(cp.stdout).strip()
1935- yield (
1936- import_tree_hash,
1937- None,
1938- 'Remove .pc directory from source package',
1939- )
1940+ Arguments:
1941+ dsc_pathname - A string path to a DSC file
1942+ """
1943
1944- try:
1945- try:
1946- cp = run(['dpkg-source', '--print-format', extracted_dir])
1947- fmt = decode_binary(cp.stdout).strip()
1948- if '3.0 (quilt)' not in fmt:
1949- raise StopIteration()
1950- except CalledProcessError as e:
1951- try:
1952- with open('debian/source/format', 'r') as f:
1953- for line in f:
1954- if re.match(r'3.0 (.*)', line):
1955- break
1956- else:
1957- raise StopIteration()
1958- # `man dpkg-source` indicates no d/s/format implies 1.0
1959- except OSError:
1960- raise StopIteration()
1961-
1962- while True:
1963- try:
1964- os.chdir(extracted_dir)
1965- run(['quilt', 'push'], rcs=[2])
1966- cp = run(['quilt', 'top'])
1967- patch_name = decode_binary(cp.stdout).strip()
1968- cp = run(['quilt', 'header'])
1969- header = decode_binary(cp.stdout).strip()
1970- patch_desc = None
1971- for regex in (r'Subject:\s*(.*?)$',
1972- r'Description:\s*(.*?)$'
1973- ):
1974- try:
1975- m = re.search(regex, header, re.MULTILINE)
1976- patch_desc = m.group(1)
1977- except AttributeError:
1978- pass
1979- if patch_desc is None:
1980- logging.debug('Unable to find Subject or Description in %s' % patch_name)
1981- self.local_repo.git_run(['--work-tree', extracted_dir, 'add', '-f', '-A'])
1982- self.local_repo.git_run(['--work-tree', extracted_dir, 'reset', 'HEAD', '--', '.pc'])
1983- cp = self.local_repo.git_run(['--work-tree', extracted_dir, 'write-tree'])
1984- import_tree_hash = decode_binary(cp.stdout).strip()
1985-
1986- yield (import_tree_hash, patch_name, patch_desc)
1987- except CalledProcessError as e:
1988- # quilt returns 2 when done pushing
1989- if e.returncode != 2:
1990- raise
1991- raise StopIteration()
1992- finally:
1993- os.chdir(oldcwd)
1994- shutil.rmtree(extract_dir)
1995- self.local_repo.clean_repository_state()
1996+ import_tree_hash = dsc_to_tree_hash(repo, dsc_pathname)
1997+ repo.clean_repository_state()
1998
1999- def parse_parentfile(self, parentfile):
2000- """Extract parent overrides from a file
2001+ return import_tree_hash
2002
2003- The parent overrides file specifies parent-child relationship(s),
2004- where the importer may not be able to determine them automatically.
2005- The format of the file is:
2006- <pkgname> <child version> <publish parent version> <changelog parent version>
2007- with one override per line.
2008+def import_patches_applied_tree(repo, dsc_pathname):
2009+ """Imports the patches-applied source package and writes the
2010+ corresponding working tree for a given srcpkg, patch by patch
2011
2012- <child version> is the published version to which this override
2013- applies.
2014+ Arguments:
2015+ dsc_pathname - A string path to a DSC file
2016+ """
2017+ oldcwd = os.getcwd()
2018
2019- The <publish parent version> is the prior version published in the
2020- same series/pocket, or the immediately preceding series/pocket, if
2021- this is the first publish in a series/pocket.
2022+ extract_dir = tempfile.mkdtemp()
2023+ os.chdir(extract_dir)
2024
2025- The <changelog parent version> is the prior version in the
2026- child publish's debian/changelog.
2027+ run(['dpkg-source', '-x',
2028+ '--skip-patches',
2029+ os.path.join(oldcwd, dsc_pathname)
2030+ ]
2031+ )
2032
2033- Keyword Arguments:
2034- parentfile -- Path to parent overrides file
2035- """
2036- if self.parent_overrides:
2037- return
2038- parent_overrides = dict()
2039+ extracted_dir = None
2040+ for path in os.listdir(extract_dir):
2041+ if os.path.isdir(path):
2042+ extracted_dir = os.path.join(extract_dir, path)
2043+ break
2044+ if extracted_dir is None:
2045+ logging.exception('No source extracted?')
2046+ raise SourceExtractionException("Failed to find an extracted "
2047+ "directory from dpkg-source -x")
2048+
2049+ if os.path.isdir(os.path.join(extracted_dir, '.pc')):
2050+ repo.git_run(
2051+ ['--work-tree', extracted_dir, 'add', '-f', '-A']
2052+ )
2053+ repo.git_run(
2054+ ['--work-tree', extracted_dir, 'rm', '-r', '-f', '.pc']
2055+ )
2056+ cp = repo.git_run(
2057+ ['--work-tree', extracted_dir, 'write-tree']
2058+ )
2059+ import_tree_hash = decode_binary(cp.stdout).strip()
2060+ yield (
2061+ import_tree_hash,
2062+ None,
2063+ 'Remove .pc directory from source package',
2064+ )
2065+
2066+ try:
2067 try:
2068- with open(parentfile) as f:
2069- for line in f:
2070- if line.startswith('#'):
2071- continue
2072- m = re.match(
2073- r'(?P<pkgname>\S*)\s*(?P<childversion>\S*)\s*(?P<parent1version>\S*)\s*(?P<parent2version>\S*)',
2074- line
2075- )
2076- if m is None:
2077- continue
2078- if m.group('pkgname') != self.pkgname:
2079- continue
2080- parent_overrides[m.group('childversion')] = {
2081- 'publish_parent':m.group('parent1version'),
2082- 'changelog_parent':m.group('parent2version')
2083- }
2084- except FileNotFoundError:
2085- pass
2086- self.parent_overrides = parent_overrides
2087+ cp = run(['dpkg-source', '--print-format', extracted_dir])
2088+ fmt = decode_binary(cp.stdout).strip()
2089+ if '3.0 (quilt)' not in fmt:
2090+ raise StopIteration()
2091+ except CalledProcessError as e:
2092+ try:
2093+ with open('debian/source/format', 'r') as f:
2094+ for line in f:
2095+ if re.match(r'3.0 (.*)', line):
2096+ break
2097+ else:
2098+ raise StopIteration()
2099+ # `man dpkg-source` indicates no d/s/format implies 1.0
2100+ except OSError:
2101+ raise StopIteration()
2102
2103- def override_parents(self, spi):
2104- unapplied_publish_parent_commit = None
2105- unapplied_changelog_parent_commit = None
2106- applied_publish_parent_commit = None
2107- applied_changelog_parent_commit = None
2108+ while True:
2109+ try:
2110+ os.chdir(extracted_dir)
2111+ run(['quilt', 'push'], rcs=[2])
2112+ cp = run(['quilt', 'top'])
2113+ patch_name = decode_binary(cp.stdout).strip()
2114+ cp = run(['quilt', 'header'])
2115+ header = decode_binary(cp.stdout).strip()
2116+ patch_desc = None
2117+ for regex in (r'Subject:\s*(.*?)$',
2118+ r'Description:\s*(.*?)$'
2119+ ):
2120+ try:
2121+ m = re.search(regex, header, re.MULTILINE)
2122+ patch_desc = m.group(1)
2123+ except AttributeError:
2124+ pass
2125+ if patch_desc is None:
2126+ logging.debug('Unable to find Subject or Description '
2127+ 'in %s' % patch_name
2128+ )
2129+ repo.git_run([
2130+ '--work-tree',
2131+ extracted_dir,
2132+ 'add',
2133+ '-f',
2134+ '-A',
2135+ ])
2136+ repo.git_run([
2137+ '--work-tree',
2138+ extracted_dir,
2139+ 'reset',
2140+ 'HEAD',
2141+ '--',
2142+ '.pc',
2143+ ])
2144+ cp = repo.git_run(['--work-tree', extracted_dir, 'write-tree'])
2145+ import_tree_hash = decode_binary(cp.stdout).strip()
2146+
2147+ yield (import_tree_hash, patch_name, patch_desc)
2148+ except CalledProcessError as e:
2149+ # quilt returns 2 when done pushing
2150+ if e.returncode != 2:
2151+ raise
2152+ raise StopIteration()
2153+ finally:
2154+ os.chdir(oldcwd)
2155+ shutil.rmtree(extract_dir)
2156+ repo.clean_repository_state()
2157+
2158+_PARENT_OVERRIDES = None
2159+def parse_parentfile(parentfile, pkgname):
2160+ """Extract parent overrides from a file
2161+
2162+ The parent overrides file specifies parent-child relationship(s),
2163+ where the importer may not be able to determine them automatically.
2164+ The format of the file is:
2165+ <pkgname> <child version> <publish parent version> <changelog parent version>
2166+ with one override per line.
2167+
2168+ <child version> is the published version to which this override
2169+ applies.
2170+
2171+ The <publish parent version> is the prior version published in the
2172+ same series/pocket, or the immediately preceding series/pocket, if
2173+ this is the first publish in a series/pocket.
2174+
2175+ The <changelog parent version> is the prior version in the
2176+ child publish's debian/changelog.
2177+
2178+ Keyword Arguments:
2179+ parentfile -- Path to parent overrides file
2180+ """
2181+ global _PARENT_OVERRIDES
2182+ if _PARENT_OVERRIDES:
2183+ return
2184+ _PARENT_OVERRIDES = dict()
2185+ try:
2186+ with open(parentfile) as f:
2187+ for line in f:
2188+ if line.startswith('#'):
2189+ continue
2190+ m = re.match(
2191+ r'(?P<pkgname>\S*)\s*(?P<childversion>\S*)\s*(?P<parent1version>\S*)\s*(?P<parent2version>\S*)',
2192+ line
2193+ )
2194+ if m is None:
2195+ continue
2196+ if m.group('pkgname') != pkgname:
2197+ continue
2198+ _PARENT_OVERRIDES[m.group('childversion')] = {
2199+ 'publish_parent':m.group('parent1version'),
2200+ 'changelog_parent':m.group('parent2version')
2201+ }
2202+ except FileNotFoundError:
2203+ pass
2204+
2205+def override_parents(repo, spi, namespace):
2206+ unapplied_publish_parent_commit = None
2207+ unapplied_changelog_parent_commit = None
2208+ applied_publish_parent_commit = None
2209+ applied_changelog_parent_commit = None
2210+
2211+ unapplied_publish_parent_tag = repo.get_import_tag(
2212+ _PARENT_OVERRIDES[spi.version]['publish_parent'],
2213+ namespace
2214+ )
2215+ applied_publish_parent_tag = repo.get_applied_tag(
2216+ _PARENT_OVERRIDES[spi.version]['publish_parent'],
2217+ namespace
2218+ )
2219+ if unapplied_publish_parent_tag is not None:
2220+ # sanity check that version from d/changelog of the
2221+ # tagged commit matches ours
2222+ parent_publish_version, _ = \
2223+ repo.get_changelog_versions_from_treeish(
2224+ str(unapplied_publish_parent_tag.peel().id),
2225+ )
2226+ if parent_publish_version != _PARENT_OVERRIDES[spi.version]['publish_parent']:
2227+ logging.error('Found a tag corresponding to publish parent '
2228+ 'override version %s, but d/changelog version (%s) '
2229+ 'differs. Will orphan commit.',
2230+ _PARENT_OVERRIDES[spi.version]['publish_parent'],
2231+ parent_publish_version
2232+ )
2233+ else:
2234+ unapplied_publish_parent_commit = str(unapplied_publish_parent_tag.peel().id)
2235+ if applied_publish_parent_tag is not None:
2236+ applied_publish_parent_commit = str(applied_publish_parent_tag.peel().id)
2237+ logging.debug('Overriding publish parent (tag) to %s',
2238+ repo.tag_to_pretty_name(unapplied_publish_parent_tag)
2239+ )
2240+ else:
2241+ if _PARENT_OVERRIDES[spi.version]['publish_parent'] == '-':
2242+ logging.debug('Not setting publish parent as '
2243+ 'specified in override file.'
2244+ )
2245+ else:
2246+ logging.error('Specified publish parent override '
2247+ '(%s) for version (%s) not found in tags. '
2248+ 'Unable to proceed.' % (
2249+ _PARENT_OVERRIDES[spi.version]['publish_parent'],
2250+ spi.version
2251+ )
2252+ )
2253+ sys.exit(1)
2254
2255- unapplied_publish_parent_tag = self.local_repo.get_import_tag(
2256- self.parent_overrides[spi.version]['publish_parent'],
2257- self.namespace)
2258- applied_publish_parent_tag = self.local_repo.get_applied_tag(
2259- self.parent_overrides[spi.version]['publish_parent'],
2260- self.namespace)
2261- if unapplied_publish_parent_tag is not None:
2262- # sanity check that version from d/changelog of the
2263- # tagged commit matches ours
2264- parent_publish_version, _ = \
2265- self.local_repo.get_changelog_versions_from_treeish(
2266- str(unapplied_publish_parent_tag.peel().id),
2267- )
2268- if parent_publish_version != self.parent_overrides[spi.version]['publish_parent']:
2269- logging.error('Found a tag corresponding to '
2270- 'publish parent override '
2271- 'version %s, but d/changelog '
2272- 'version (%s) differs. Will '
2273- 'orphan commit.' % (
2274- self.parent_overrides[spi.version]['publish_parent'],
2275- parent_publish_version
2276- )
2277- )
2278- else:
2279- unapplied_publish_parent_commit = str(unapplied_publish_parent_tag.peel().id)
2280- if applied_publish_parent_tag is not None:
2281- applied_publish_parent_commit = str(applied_publish_parent_tag.peel().id)
2282- logging.debug('Overriding publish parent (tag) to %s',
2283- self.local_repo.tag_to_pretty_name(unapplied_publish_parent_tag)
2284- )
2285+ unapplied_changelog_parent_tag = repo.get_import_tag(
2286+ _PARENT_OVERRIDES[spi.version]['changelog_parent'],
2287+ namespace
2288+ )
2289+ applied_changelog_parent_tag = repo.get_applied_tag(
2290+ _PARENT_OVERRIDES[spi.version]['changelog_parent'],
2291+ namespace
2292+ )
2293+ if unapplied_changelog_parent_tag is not None:
2294+ # sanity check that version from d/changelog of the
2295+ # tagged commit matches ours
2296+ parent_changelog_version, _ = repo.get_changelog_versions_from_treeish(
2297+ str(unapplied_changelog_parent_tag.peel().id),
2298+ )
2299+ if parent_changelog_version != _PARENT_OVERRIDES[spi.version]['changelog_parent']:
2300+ logging.error('Found a tag corresponding to '
2301+ 'changelog parent override '
2302+ 'version %s, but d/changelog '
2303+ 'version (%s) differs. Will '
2304+ 'orphan commit.' % (
2305+ _PARENT_OVERRIDES[spi.version]['changelog_parent'],
2306+ parent_changelog_version
2307+ )
2308+ )
2309 else:
2310- if self.parent_overrides[spi.version]['publish_parent'] == '-':
2311- logging.debug('Not setting publish parent as '
2312- 'specified in override file.'
2313- )
2314- else:
2315- logging.error('Specified publish parent override '
2316- '(%s) for version (%s) not found in tags. '
2317- 'Unable to proceed.' % (
2318- self.parent_overrides[spi.version]['publish_parent'],
2319- spi.version
2320- )
2321- )
2322- sys.exit(1)
2323-
2324- unapplied_changelog_parent_tag = self.local_repo.get_import_tag(
2325- self.parent_overrides[spi.version]['changelog_parent'],
2326- self.namespace)
2327- applied_changelog_parent_tag = self.local_repo.get_applied_tag(
2328- self.parent_overrides[spi.version]['changelog_parent'],
2329- self.namespace)
2330- if unapplied_changelog_parent_tag is not None:
2331- # sanity check that version from d/changelog of the
2332- # tagged commit matches ours
2333- parent_changelog_version, _ = \
2334- self.local_repo.get_changelog_versions_from_treeish(
2335- str(unapplied_changelog_parent_tag.peel().id),
2336- )
2337- if parent_changelog_version != self.parent_overrides[spi.version]['changelog_parent']:
2338- logging.error('Found a tag corresponding to '
2339- 'changelog parent override '
2340- 'version %s, but d/changelog '
2341- 'version (%s) differs. Will '
2342- 'orphan commit.' % (
2343- self.parent_overrides[spi.version]['changelog_parent'],
2344- parent_changelog_version
2345- )
2346- )
2347- else:
2348- unapplied_changelog_parent_commit = str(unapplied_changelog_parent_tag.peel().id)
2349- if applied_changelog_parent_tag is not None:
2350- applied_changelog_parent_commit = str(applied_changelog_parent_tag.peel().id)
2351- logging.debug('Overriding changelog parent (tag) to %s',
2352- self.local_repo.tag_to_pretty_name(unapplied_changelog_parent_tag)
2353- )
2354+ unapplied_changelog_parent_commit = str(unapplied_changelog_parent_tag.peel().id)
2355+ if applied_changelog_parent_tag is not None:
2356+ applied_changelog_parent_commit = str(applied_changelog_parent_tag.peel().id)
2357+ logging.debug('Overriding changelog parent (tag) to %s',
2358+ repo.tag_to_pretty_name(unapplied_changelog_parent_tag)
2359+ )
2360+ else:
2361+ if _PARENT_OVERRIDES[spi.version]['changelog_parent'] == '-':
2362+ logging.debug('Not setting changelog parent as specified '
2363+ 'in override file.'
2364+ )
2365 else:
2366- if self.parent_overrides[spi.version]['changelog_parent'] == '-':
2367- logging.debug('Not setting changelog parent as '
2368- 'specified in override file.'
2369- )
2370- else:
2371- logging.error('Specified changelog parent override '
2372- '(%s) for version (%s) not found in tags. '
2373- 'Unable to proceed.' % (
2374- self.parent_overrides[spi.version]['changelog_parent'],
2375- spi.version
2376- )
2377- )
2378- sys.exit(1)
2379- return (unapplied_publish_parent_commit,
2380- unapplied_changelog_parent_commit,
2381- applied_publish_parent_commit,
2382- applied_changelog_parent_commit)
2383-
2384- def _update_devel_branches(self, namespace, spi=None):
2385- if spi:
2386- # we do not maintain -devel pointers for debian
2387- if str(spi.distribution.name.lower()) != "ubuntu":
2388+ logging.error('Specified changelog parent override (%s) '
2389+ 'for version (%s) not found in tags. Unable to proceed.',
2390+ _PARENT_OVERRIDES[spi.version]['changelog_parent'],
2391+ spi.version
2392+ )
2393+ sys.exit(1)
2394+ return (
2395+ unapplied_publish_parent_commit,
2396+ unapplied_changelog_parent_commit,
2397+ applied_publish_parent_commit,
2398+ applied_changelog_parent_commit,
2399+ )
2400+
2401+def _update_devel_branches(
2402+ repo,
2403+ namespace,
2404+ pkgname,
2405+ ubuntu_sinfo,
2406+ spi=None,
2407+):
2408+ if spi:
2409+ # we do not maintain -devel pointers for debian
2410+ if str(spi.distribution.name.lower()) != "ubuntu":
2411+ return
2412+ for devel_head in (
2413+ '%s/ubuntu/%s-devel' % (namespace, spi.series.name.lower()),
2414+ '%s/ubuntu/devel' % namespace
2415+ ):
2416+ try:
2417+ devel_head_hash = repo.head_to_commit(devel_head)
2418+ devel_head_version, _ = repo.get_changelog_versions_from_treeish(devel_head_hash)
2419+ except AttributeError:
2420+ devel_head_version = None
2421+ if version_compare(spi.version, devel_head_version) <= 0:
2422 return
2423- for devel_head in (
2424- '%s/ubuntu/%s-devel' % (namespace, spi.series.name.lower()),
2425- '%s/ubuntu/devel' % namespace
2426- ):
2427- try:
2428- devel_head_hash = self.local_repo.head_to_commit(devel_head)
2429- devel_head_version, _ = self.local_repo.get_changelog_versions_from_treeish(devel_head_hash)
2430- except AttributeError:
2431- devel_head_version = None
2432- if version_compare(spi.version, devel_head_version) <= 0:
2433- return
2434- logging.debug("Updating %s to %s" % (devel_head,
2435- spi.head_name(namespace)))
2436- self.local_repo.merge_commit_to_devel_head(
2437- namespace, devel_head,
2438- self.local_repo.head_to_commit(spi.head_name(namespace)),
2439- environment_spi=spi)
2440+ logging.debug("Updating %s to %s" % (devel_head,
2441+ spi.head_name(namespace)))
2442+ repo.merge_commit_to_devel_head(
2443+ namespace,
2444+ devel_head,
2445+ repo.head_to_commit(spi.head_name(namespace)),
2446+ environment_spi=spi,
2447+ )
2448+ else:
2449+ ubuntu_head_versions = repo.get_heads_and_versions(
2450+ 'ubuntu',
2451+ namespace,
2452+ )
2453+ devel_hash = None
2454+ devel_name = None
2455+ for rel in ubuntu_sinfo.active_series_name_list:
2456+ series_devel_hash = None
2457+ series_devel_name = None
2458+ series_devel_version = None
2459+ # Find highest version publish in these pockets
2460+ for suff in ("-proposed", "-updates", "-security", ""):
2461+ name = "%s/ubuntu/%s%s" % (namespace, rel, suff)
2462+ if name in ubuntu_head_versions and version_compare(
2463+ ubuntu_head_versions[name]['version'],
2464+ series_devel_version
2465+ ) > 0:
2466+ series_devel_name = name
2467+ series_devel_version = ubuntu_head_versions[name]['version']
2468+ series_devel_hash = str(ubuntu_head_versions[name]['head'].peel().id)
2469+ if series_devel_hash:
2470+ if not devel_hash:
2471+ devel_hash = series_devel_hash
2472+ devel_name = series_devel_name
2473+ logging.debug("Updating %s/ubuntu/%s-devel branch to '%s' (%s)",
2474+ namespace, rel, series_devel_name, series_devel_hash)
2475+ repo.merge_commit_to_devel_head(namespace,
2476+ "%s/ubuntu/%s-devel" % (namespace, rel), series_devel_hash)
2477+ if devel_hash is None:
2478+ logging.warn("Package '%s' does not appear to have been published "
2479+ "in Ubuntu. No %s/ubuntu/devel branch created.",
2480+ pkgname, namespace)
2481 else:
2482- ubuntu_head_versions = self.local_repo.get_heads_and_versions(
2483- 'ubuntu', namespace
2484+ logging.debug("Setting %s/ubuntu/devel branch to '%s' (%s)",
2485+ namespace, devel_name, devel_hash)
2486+ repo.merge_commit_to_devel_head(
2487+ namespace, "%s/ubuntu/devel" % namespace, devel_hash
2488 )
2489- devel_hash = None
2490- devel_name = None
2491- for rel in self.ubuntu_sinfo.active_series_name_list:
2492- series_devel_hash = None
2493- series_devel_name = None
2494- series_devel_version = None
2495- # Find highest version publish in these pockets
2496- for suff in ("-proposed", "-updates", "-security", ""):
2497- name = "%s/ubuntu/%s%s" % (namespace, rel, suff)
2498- if name in ubuntu_head_versions and version_compare(
2499- ubuntu_head_versions[name]['version'],
2500- series_devel_version
2501- ) > 0:
2502- series_devel_name = name
2503- series_devel_version = ubuntu_head_versions[name]['version']
2504- series_devel_hash = str(ubuntu_head_versions[name]['head'].peel().id)
2505- if series_devel_hash:
2506- if not devel_hash:
2507- devel_hash = series_devel_hash
2508- devel_name = series_devel_name
2509- logging.debug("Updating %s/ubuntu/%s-devel branch to '%s' (%s)",
2510- namespace, rel, series_devel_name, series_devel_hash)
2511- self.local_repo.merge_commit_to_devel_head(namespace,
2512- "%s/ubuntu/%s-devel" % (namespace, rel), series_devel_hash)
2513- if devel_hash is None:
2514- logging.warn("Package '%s' does not appear to have been published "
2515- "in Ubuntu. No %s/ubuntu/devel branch created.",
2516- self.pkgname, namespace)
2517- else:
2518- logging.debug("Setting %s/ubuntu/devel branch to '%s' (%s)",
2519- namespace, devel_name, devel_hash)
2520- self.local_repo.merge_commit_to_devel_head(
2521- namespace, "%s/ubuntu/devel" % namespace, devel_hash
2522- )
2523-
2524- def update_devel_branches(self, spi=None):
2525- self._update_devel_branches(self.namespace, spi)
2526-
2527- def update_applied_devel_branches(self, spi=None):
2528- self._update_devel_branches('%s/applied' % self.namespace, spi)
2529-
2530- # imports a package based upon source package information
2531- def import_unapplied_spi(self, spi):
2532- """Imports a source package from Launchpad into the git
2533- repository
2534-
2535- Arguments:
2536- spi - a SourcePackageInformation instance
2537- """
2538- pretty_head_name = spi.pretty_head_name
2539
2540- logging.info('Importing patches-unapplied %s to %s' %
2541- (spi.version, pretty_head_name))
2542- unapplied_tip_head = self.local_repo.get_head_by_name(spi.head_name(self.namespace))
2543-
2544- spi.pull()
2545-
2546- self.local_repo.clean_repository_state()
2547-
2548- self.import_dsc(
2549+def update_devel_branches(
2550+ repo,
2551+ namespace,
2552+ pkgname,
2553+ ubuntu_sinfo,
2554+ spi=None,
2555+):
2556+ _update_devel_branches(
2557+ repo=repo,
2558+ namespace=namespace,
2559+ pkgname=pkgname,
2560+ ubuntu_sinfo=ubuntu_sinfo,
2561+ spi=spi,
2562+ )
2563+
2564+def update_applied_devel_branches(
2565+ repo,
2566+ namespace,
2567+ pkgname,
2568+ ubuntu_sinfo,
2569+ spi=None,
2570+):
2571+ _update_devel_branches(
2572+ repo=repo,
2573+ namespace='%s/applied' % namespace,
2574+ pkgname=pkgname,
2575+ ubuntu_sinfo=ubuntu_sinfo,
2576+ spi=spi,
2577+ )
2578+
2579+# imports a package based upon source package information
2580+def import_unapplied_spi(repo, spi, namespace, skip_orig, ubuntu_sinfo):
2581+ """Imports a source package from Launchpad into the git
2582+ repository
2583+
2584+ Arguments:
2585+ spi - a SourcePackageInformation instance
2586+ """
2587+ pretty_head_name = spi.pretty_head_name
2588+
2589+ logging.info('Importing patches-unapplied %s to %s',
2590+ spi.version, pretty_head_name
2591+ )
2592+ unapplied_tip_head = repo.get_head_by_name(spi.head_name(namespace))
2593+
2594+ spi.pull()
2595+
2596+ repo.clean_repository_state()
2597+
2598+ import_dsc(
2599+ repo,
2600+ GitUbuntuDsc(spi.dsc_pathname),
2601+ namespace,
2602+ str(spi.version),
2603+ spi.distribution.name.lower(),
2604+ )
2605+ if not skip_orig:
2606+ import_orig(
2607+ repo,
2608 GitUbuntuDsc(spi.dsc_pathname),
2609+ namespace,
2610 str(spi.version),
2611- spi.distribution.name.lower()
2612+ spi.distribution.name.lower(),
2613 )
2614- if not self.skip_orig:
2615- self.import_orig(
2616- GitUbuntuDsc(spi.dsc_pathname),
2617- version = str(spi.version),
2618- dist = spi.distribution.name.lower(),
2619- )
2620
2621- unapplied_import_tree_hash = self.import_patches_unapplied_tree(spi.dsc_pathname)
2622- logging.debug('Imported patches-unapplied version %s as tree %s',
2623- spi.version, unapplied_import_tree_hash
2624- )
2625+ unapplied_import_tree_hash = import_patches_unapplied_tree(
2626+ repo,
2627+ spi.dsc_pathname
2628+ )
2629+ logging.debug(
2630+ "Imported patches-unapplied version %s as tree %s",
2631+ spi.version,
2632+ unapplied_import_tree_hash
2633+ )
2634+
2635+ import_tree_versions = repo.get_all_changelog_versions_from_treeish(
2636+ unapplied_import_tree_hash
2637+ )
2638+ changelog_version = import_tree_versions[0]
2639+
2640+ logging.debug(
2641+ "Found changelog version %s in newly imported tree" %
2642+ changelog_version
2643+ )
2644+
2645+ # sanity check on spph and d/changelog agreeing on the latest version
2646+ if spi.version not in _PARENT_OVERRIDES:
2647+ assert spi.version == changelog_version, (
2648+ 'source pkg version: {} != changelog version: {}'.format(
2649+ spi.version, changelog_version))
2650+
2651+ unapplied_tip_version = None
2652+ # check if the version to import has already been imported to
2653+ # this head
2654+ if unapplied_tip_head is not None:
2655+ if repo.treeishs_identical(
2656+ unapplied_import_tree_hash, str(unapplied_tip_head.peel().id)
2657+ ):
2658+ logging.warn('%s is identical to %s',
2659+ pretty_head_name, spi.version
2660+ )
2661+ return
2662+ unapplied_tip_version, _ = repo.get_changelog_versions_from_treeish(
2663+ str(unapplied_tip_head.peel().id),
2664+ )
2665+
2666+ logging.debug('Tip version is %s', unapplied_tip_version)
2667
2668- import_tree_versions = self.local_repo.get_all_changelog_versions_from_treeish(
2669- unapplied_import_tree_hash
2670- )
2671- changelog_version = import_tree_versions[0]
2672+ unapplied_publish_parent_commit = None
2673+ unapplied_changelog_parent_commit = None
2674+ upload_parent_commit = None
2675+ unapplied_parent_commit = None
2676
2677+ if spi.version in _PARENT_OVERRIDES:
2678 logging.debug(
2679- 'Found changelog version %s in newly imported tree' %
2680- changelog_version
2681- )
2682+ '%s is specified in the parent override file.',
2683+ spi.version
2684+ )
2685
2686- # sanity check on spph and d/changelog agreeing on the latest version
2687- if spi.version not in self.parent_overrides:
2688- assert spi.version == changelog_version, (
2689- 'source pkg version: {} != changelog version: {}'.format(
2690- spi.version, changelog_version))
2691-
2692- unapplied_tip_version = None
2693- # check if the version to import has already been imported to
2694- # this head
2695- if unapplied_tip_head is not None:
2696- if self.local_repo.treeishs_identical(
2697- unapplied_import_tree_hash, str(unapplied_tip_head.peel().id)
2698- ):
2699- logging.warn('%s is identical to %s',
2700- pretty_head_name, spi.version
2701+ (
2702+ unapplied_publish_parent_commit,
2703+ unapplied_changelog_parent_commit,
2704+ _,
2705+ _
2706+ ) = override_parents(repo, spi, namespace)
2707+ else:
2708+ # Get parent from publishing history (which is the last
2709+ # published version in the corresponding series)
2710+ try:
2711+ unapplied_publish_parent_commit = str(unapplied_tip_head.peel().id)
2712+ except AttributeError:
2713+ try:
2714+ unapplied_publish_parent_head = repo.get_head_by_name(spi.parent_head_name(namespace))
2715+ unapplied_publish_parent_commit = str(unapplied_publish_parent_head.peel().id)
2716+ except AttributeError:
2717+ pass
2718+ finally:
2719+ try:
2720+ unapplied_publish_parent_tag = repo.nearest_import_tag(unapplied_publish_parent_commit, namespace)
2721+ logging.debug('Publishing parent (tag) is %s',
2722+ repo.tag_to_pretty_name(unapplied_publish_parent_tag)
2723 )
2724- return
2725- unapplied_tip_version, _ = self.local_repo.get_changelog_versions_from_treeish(
2726- str(unapplied_tip_head.peel().id),
2727- )
2728+ except AttributeError:
2729+ pass
2730+
2731+ if version_compare(str(spi.version), unapplied_tip_version) <= 0:
2732+ logging.warn('Version to import (%s) is not after %s tip (%s)',
2733+ spi.version, pretty_head_name, unapplied_tip_version
2734+ )
2735
2736- logging.debug('Tip version is %s', unapplied_tip_version)
2737+ # Walk changelog backwards until we find an imported version
2738+ for version in import_tree_versions:
2739+ unapplied_changelog_parent_tag = repo.get_import_tag(version, namespace)
2740+ if unapplied_changelog_parent_tag is not None:
2741+ # sanity check that version from d/changelog of the
2742+ # tagged parent matches ours
2743+ parent_changelog_version, _ = \
2744+ repo.get_changelog_versions_from_treeish(
2745+ str(unapplied_changelog_parent_tag.peel(pygit2.Tree).id),
2746+ )
2747+ # if the parent was imported via a parent override
2748+ # itself, assume it is valid without checking its
2749+ # tree's debian/changelog, as it wasn't used
2750+ if (version not in _PARENT_OVERRIDES and
2751+ parent_changelog_version != version
2752+ ):
2753+ logging.error('Found a tag corresponding to '
2754+ 'parent version %s, but d/changelog '
2755+ 'version (%s) differs. Will '
2756+ 'orphan commit.' % (
2757+ version,
2758+ parent_changelog_version
2759+ )
2760+ )
2761+ else:
2762+ unapplied_changelog_parent_commit = str(unapplied_changelog_parent_tag.peel().id)
2763+ logging.debug('Changelog parent (tag) is %s',
2764+ repo.tag_to_pretty_name(unapplied_changelog_parent_tag)
2765+ )
2766+ break
2767
2768- unapplied_publish_parent_commit = None
2769+ # If the two parents are tree-identical, then favor publication
2770+ # history. This deals with copy-forwards between series and with
2771+ # syncs.
2772+ if repo.treeishs_identical(unapplied_publish_parent_commit, unapplied_changelog_parent_commit):
2773 unapplied_changelog_parent_commit = None
2774- upload_parent_commit = None
2775- unapplied_parent_commit = None
2776
2777- if spi.version in self.parent_overrides:
2778- logging.debug('%s is specified in the parent override file.' %
2779+ # check if the version to import has already been uploaded to
2780+ # this head -- we leverage the above code to perform a 'dry-run'
2781+ # of the import tree, to obtain the (up to) two parents
2782+ upload_tag = repo.get_upload_tag(spi.version, namespace)
2783+ import_tag = repo.get_import_tag(spi.version, namespace)
2784+ if upload_tag and not import_tag:
2785+ if str(upload_tag.peel().tree.id) != unapplied_import_tree_hash:
2786+ logging.warn('Found upload tag %s, '
2787+ 'but the corresponding tree '
2788+ 'does not match the published '
2789+ 'version. Will import %s as '
2790+ 'normal, ignoring the upload tag.',
2791+ repo.tag_to_pretty_name(upload_tag),
2792 spi.version
2793 )
2794-
2795- (unapplied_publish_parent_commit,
2796- unapplied_changelog_parent_commit,
2797- _, _) = self.override_parents(spi)
2798 else:
2799- # Get parent from publishing history (which is the last
2800- # published version in the corresponding series)
2801- try:
2802- unapplied_publish_parent_commit = str(unapplied_tip_head.peel().id)
2803- except AttributeError:
2804+ upload_parent_commit = str(upload_tag.peel().id)
2805+ if unapplied_publish_parent_commit is not None:
2806 try:
2807- unapplied_publish_parent_head = self.local_repo.get_head_by_name(spi.parent_head_name(self.namespace))
2808- unapplied_publish_parent_commit = str(unapplied_publish_parent_head.peel().id)
2809- except AttributeError:
2810- pass
2811- finally:
2812- try:
2813- unapplied_publish_parent_tag = self.local_repo.nearest_import_tag(unapplied_publish_parent_commit, self.namespace)
2814- logging.debug('Publishing parent (tag) is %s',
2815- self.local_repo.tag_to_pretty_name(unapplied_publish_parent_tag)
2816+ repo.git_run(['merge-base', '--is-ancestor',
2817+ unapplied_publish_parent_commit,
2818+ upload_parent_commit
2819+ ]
2820 )
2821- except AttributeError:
2822- pass
2823-
2824- if version_compare(str(spi.version), unapplied_tip_version) <= 0:
2825- logging.warn('Version to import (%s) is not after %s tip (%s)',
2826- spi.version, pretty_head_name, unapplied_tip_version
2827- )
2828-
2829- # Walk changelog backwards until we find an imported version
2830- for version in import_tree_versions:
2831- unapplied_changelog_parent_tag = self.local_repo.get_import_tag(version, self.namespace)
2832- if unapplied_changelog_parent_tag is not None:
2833- # sanity check that version from d/changelog of the
2834- # tagged parent matches ours
2835- parent_changelog_version, _ = \
2836- self.local_repo.get_changelog_versions_from_treeish(
2837- str(unapplied_changelog_parent_tag.peel(pygit2.Tree).id),
2838- )
2839- # if the parent was imported via a parent override
2840- # itself, assume it is valid without checking its
2841- # tree's debian/changelog, as it wasn't used
2842- if (version not in self.parent_overrides and
2843- parent_changelog_version != version
2844- ):
2845- logging.error('Found a tag corresponding to '
2846- 'parent version %s, but d/changelog '
2847- 'version (%s) differs. Will '
2848- 'orphan commit.' % (
2849- version,
2850- parent_changelog_version
2851- )
2852- )
2853- else:
2854- unapplied_changelog_parent_commit = str(unapplied_changelog_parent_tag.peel().id)
2855- logging.debug('Changelog parent (tag) is %s',
2856- self.local_repo.tag_to_pretty_name(unapplied_changelog_parent_tag)
2857- )
2858- break
2859-
2860- # If the two parents are tree-identical, then favor publication
2861- # history. This deals with copy-forwards between series and with
2862- # syncs.
2863- if self.local_repo.treeishs_identical(unapplied_publish_parent_commit, unapplied_changelog_parent_commit):
2864- unapplied_changelog_parent_commit = None
2865-
2866- # check if the version to import has already been uploaded to
2867- # this head -- we leverage the above code to perform a 'dry-run'
2868- # of the import tree, to obtain the (up to) two parents
2869- upload_tag = self.local_repo.get_upload_tag(spi.version, self.namespace)
2870- import_tag = self.local_repo.get_import_tag(spi.version, self.namespace)
2871- if upload_tag and not import_tag:
2872- if str(upload_tag.peel().tree.id) != unapplied_import_tree_hash:
2873- logging.warn('Found upload tag %s, '
2874- 'but the corresponding tree '
2875- 'does not match the published '
2876- 'version. Will import %s as '
2877- 'normal, ignoring the upload tag.',
2878- self.local_repo.tag_to_pretty_name(upload_tag),
2879- spi.version
2880- )
2881- else:
2882- upload_parent_commit = str(upload_tag.peel().id)
2883- if unapplied_publish_parent_commit is not None:
2884- try:
2885- self.local_repo.git_run(['merge-base', '--is-ancestor',
2886- unapplied_publish_parent_commit,
2887- upload_parent_commit
2888- ]
2889- )
2890- unapplied_publish_parent_commit = None
2891- except CalledProcessError as e:
2892- if e.returncode != 1:
2893- raise
2894-
2895- if unapplied_changelog_parent_commit is not None:
2896- try:
2897- self.local_repo.git_run(['merge-base', '--is-ancestor',
2898- unapplied_changelog_parent_commit,
2899- upload_parent_commit
2900- ]
2901- )
2902- unapplied_changelog_parent_commit = None
2903- except CalledProcessError as e:
2904- if e.returncode != 1:
2905- raise
2906-
2907- self.commit_unapplied_patches_import(
2908- spi,
2909- unapplied_import_tree_hash,
2910- unapplied_publish_parent_commit,
2911- unapplied_changelog_parent_commit,
2912- upload_parent_commit,
2913- )
2914-
2915- self.update_devel_branches(spi)
2916-
2917- def import_applied_spi(self, spi):
2918- """Imports a source package from Launchpad into the git
2919- repository
2920-
2921- Arguments:
2922- spi - a SourcePackageInformation instance
2923- """
2924- pretty_head_name = spi.pretty_head_name
2925-
2926- logging.info('Importing patches-applied %s to %s' % (spi.version, pretty_head_name))
2927- applied_tip_head = self.local_repo.get_head_by_name(spi.applied_head_name(self.namespace))
2928-
2929- spi.pull()
2930-
2931- self.local_repo.clean_repository_state()
2932-
2933- unapplied_parent_tag = self.local_repo.get_import_tag(spi.version, self.namespace)
2934- unapplied_parent_commit = str(unapplied_parent_tag.peel().id)
2935- unapplied_import_tree_hash = str(unapplied_parent_tag.peel(pygit2.Tree).id)
2936- logging.debug('Found patches-unapplied version %s as commit %s',
2937- str(spi.version), unapplied_parent_commit)
2938-
2939- import_tree_versions = self.local_repo.get_all_changelog_versions_from_treeish(
2940- unapplied_import_tree_hash
2941- )
2942- changelog_version = import_tree_versions[0]
2943-
2944- applied_tip_version = None
2945- # check if the version to import has already been imported to
2946- # this head
2947- if applied_tip_head is not None:
2948- if self.local_repo.treeishs_identical(
2949- unapplied_import_tree_hash, str(applied_tip_head.peel().id)
2950- ):
2951- logging.warn('%s is identical to %s',
2952- pretty_head_name, spi.version
2953- )
2954- return
2955- applied_tip_version, _ = self.local_repo.get_changelog_versions_from_treeish(
2956- str(applied_tip_head.peel().id),
2957- )
2958-
2959- logging.debug('Tip version is %s', applied_tip_version)
2960+ unapplied_publish_parent_commit = None
2961+ except CalledProcessError as e:
2962+ if e.returncode != 1:
2963+ raise
2964
2965- applied_publish_parent_commit = None
2966- applied_changelog_parent_commit = None
2967+ if unapplied_changelog_parent_commit is not None:
2968+ try:
2969+ repo.git_run(['merge-base', '--is-ancestor',
2970+ unapplied_changelog_parent_commit,
2971+ upload_parent_commit
2972+ ]
2973+ )
2974+ unapplied_changelog_parent_commit = None
2975+ except CalledProcessError as e:
2976+ if e.returncode != 1:
2977+ raise
2978
2979- if spi.version in self.parent_overrides:
2980- logging.debug('%s is specified in the parent override file.' %
2981- spi.version
2982+ commit_unapplied_patches_import(
2983+ repo,
2984+ spi,
2985+ unapplied_import_tree_hash,
2986+ namespace,
2987+ unapplied_publish_parent_commit,
2988+ unapplied_changelog_parent_commit,
2989+ upload_parent_commit,
2990+ )
2991+
2992+ update_devel_branches(
2993+ repo=repo,
2994+ namespace=namespace,
2995+ pkgname=spi.name,
2996+ ubuntu_sinfo=ubuntu_sinfo,
2997+ spi=spi,
2998+ )
2999+
3000+def import_applied_spi(repo, spi, namespace, ubuntu_sinfo):
3001+ """Imports a source package from Launchpad into the git
3002+ repository
3003+
3004+ Arguments:
3005+ spi - a SourcePackageInformation instance
3006+ """
3007+ pretty_head_name = spi.pretty_head_name
3008+
3009+ logging.info('Importing patches-applied %s to %s' % (spi.version, pretty_head_name))
3010+ applied_tip_head = repo.get_head_by_name(spi.applied_head_name(namespace))
3011+
3012+ spi.pull()
3013+
3014+ repo.clean_repository_state()
3015+
3016+ unapplied_parent_tag = repo.get_import_tag(spi.version, namespace)
3017+ unapplied_parent_commit = str(unapplied_parent_tag.peel().id)
3018+ unapplied_import_tree_hash = str(unapplied_parent_tag.peel(pygit2.Tree).id)
3019+ logging.debug('Found patches-unapplied version %s as commit %s',
3020+ str(spi.version), unapplied_parent_commit)
3021+
3022+ import_tree_versions = repo.get_all_changelog_versions_from_treeish(
3023+ unapplied_import_tree_hash
3024+ )
3025+ changelog_version = import_tree_versions[0]
3026+
3027+ applied_tip_version = None
3028+ # check if the version to import has already been imported to
3029+ # this head
3030+ if applied_tip_head is not None:
3031+ if repo.treeishs_identical(
3032+ unapplied_import_tree_hash, str(applied_tip_head.peel().id)
3033+ ):
3034+ logging.warn('%s is identical to %s',
3035+ pretty_head_name, spi.version
3036 )
3037+ return
3038+ applied_tip_version, _ = repo.get_changelog_versions_from_treeish(
3039+ str(applied_tip_head.peel().id),
3040+ )
3041
3042- (_, _,
3043- applied_publish_parent_commit,
3044- applied_changelog_parent_commit) = self.override_parents(spi)
3045- else:
3046- # Get parent from publishing history (which is the last
3047- # published version in the corresponding series)
3048- try:
3049- applied_publish_parent_commit = str(applied_tip_head.peel().id)
3050- except AttributeError:
3051- try:
3052- applied_publish_parent_head = self.local_repo.get_head_by_name(spi.applied_parent_head_name(self.namespace))
3053- applied_publish_parent_head_commit = str(applied_publish_parent_head.peel().id)
3054- except AttributeError:
3055- pass
3056-
3057- if version_compare(str(spi.version), applied_tip_version) <= 0:
3058- logging.warn('Version to import (%s) is not after %s tip (%s)',
3059- spi.version, pretty_head_name, applied_tip_version
3060- )
3061+ logging.debug('Tip version is %s', applied_tip_version)
3062
3063- # Walk changelog backwards until we find an imported version
3064- for version in import_tree_versions:
3065- applied_changelog_parent_tag = self.local_repo.get_applied_tag(version, self.namespace)
3066- if applied_changelog_parent_tag is not None:
3067- # sanity check that version from d/changelog of the
3068- # tagged parent matches ours
3069- parent_changelog_version, _ = \
3070- self.local_repo.get_changelog_versions_from_treeish(
3071- str(applied_changelog_parent_tag.peel(pygit2.Tree).id),
3072- )
3073- # if the parent was imported via a parent override
3074- # itself, assume it is valid without checking its
3075- # tree's debian/changelog, as it wasn't used
3076- if (version not in self.parent_overrides and
3077- parent_changelog_version != version
3078- ):
3079- logging.error('Found a tag corresponding to '
3080- 'parent version %s, but d/changelog '
3081- 'version (%s) differs. Will '
3082- 'orphan commit.' % (
3083- version,
3084- parent_changelog_version
3085- )
3086- )
3087- else:
3088- applied_changelog_parent_commit = str(applied_changelog_parent_tag.peel().id)
3089- logging.debug('Changelog parent (tag) is %s',
3090- self.local_repo.tag_to_pretty_name(applied_changelog_parent_tag)
3091- )
3092- break
3093-
3094- # If the two parents are tree-identical, then favor publication
3095- # history. This deals with copy-forwards between series and with
3096- # syncs.
3097- if self.local_repo.treeishs_identical(applied_publish_parent_commit, applied_changelog_parent_commit):
3098- applied_changelog_parent_commit = None
3099-
3100- # Assume no patches to apply
3101- applied_import_tree_hash = unapplied_import_tree_hash
3102- # get tree id from above commit
3103- for (applied_import_tree_hash, patch_name, patch_desc) in self.import_patches_applied_tree(spi.dsc_pathname):
3104- # special case for .pc removal
3105- if patch_name is None:
3106- msg = b'%b.' % (patch_desc.encode())
3107- else:
3108- if patch_desc is None:
3109- patch_desc = '%s\n\nNo DEP3 Subject or Description header found' % patch_name
3110- msg = b'%b\n\nGbp-Pq: %b.' % (patch_desc.encode(), patch_name.encode())
3111- unapplied_parent_commit = self.local_repo.commit_tree_hash(
3112- applied_import_tree_hash,
3113- [unapplied_parent_commit],
3114- msg,
3115- spi)
3116-
3117- logging.debug('Committed patch-application of %s as %s',
3118- patch_name, unapplied_parent_commit
3119- )
3120+ applied_publish_parent_commit = None
3121+ applied_changelog_parent_commit = None
3122
3123- self.commit_applied_patches_import(
3124- spi,
3125- applied_import_tree_hash,
3126+ if spi.version in _PARENT_OVERRIDES:
3127+ logging.debug('%s is specified in the parent override file.' %
3128+ spi.version
3129+ )
3130+
3131+ (
3132+ _,
3133+ _,
3134 applied_publish_parent_commit,
3135 applied_changelog_parent_commit,
3136- unapplied_parent_commit
3137- )
3138-
3139- self.update_applied_devel_branches(spi)
3140-
3141- def import_publishes(self, patches_applied,
3142- debian_head_versions, ubuntu_head_versions):
3143- history_found = False
3144- only_debian = False
3145- srcpkg_information = None
3146- if patches_applied:
3147- namespace = '%s/applied' % self.namespace
3148- import_type = 'patches-applied'
3149- import_func = self.import_applied_spi
3150- else:
3151- namespace = self.namespace
3152- import_type = 'patches-unapplied'
3153- import_func = self.import_unapplied_spi
3154- for distname, versions, dist_sinfo in (
3155- ("debian", debian_head_versions, self.debian_sinfo),
3156- ("ubuntu", ubuntu_head_versions, self.ubuntu_sinfo)):
3157- if self.active_series_only and distname == "debian":
3158- continue
3159- try:
3160- for srcpkg_information in dist_sinfo.launchpad_versions_published_after(
3161- versions,
3162- namespace,
3163- workdir=self.workdir,
3164- active_series_only=self.active_series_only
3165- ):
3166- history_found = True
3167- import_func(srcpkg_information)
3168- # Do not push upstream
3169- if self.fixup_devel and distname == "ubuntu":
3170- self.update_devel_branches(spi=None)
3171- except NoPublicationHistoryException:
3172- logging.warning("No publication history found for %s in %s. ",
3173- self.pkgname, distname)
3174- if distname == 'ubuntu':
3175- only_debian = True
3176- except Exception as e:
3177- if srcpkg_information is None:
3178- msg = 'Unable to import %s to %s' % (import_type, distname)
3179- else:
3180- msg = 'Unable to import %s %s to %s' % (import_type,
3181- str(srcpkg_information.version), distname)
3182- if not patches_applied:
3183- raise GitUbuntuImportError(msg) from e
3184- else:
3185- logging.error(msg)
3186- else:
3187- history_found = True
3188-
3189- return (only_debian, history_found)
3190-
3191- def main(self, args):
3192- self.pkgname = args.package
3193- owner = args.lp_owner
3194- no_clean = args.no_clean
3195- try:
3196- directory = args.directory
3197- no_clean = True
3198- except AttributeError:
3199- directory = None
3200- try:
3201- user = args.lp_user
3202- except AttributeError:
3203- user = None
3204- # --active-series-only/--no-fetch to a tmpdir/--skip-orig
3205- # implies --no-push
3206- no_push = (
3207- args.no_push or
3208- (args.no_fetch and not directory) or
3209- args.active_series_only or
3210- args.skip_orig
3211- )
3212+ ) = override_parents(
3213+ repo,
3214+ spi,
3215+ namespace,
3216+ )
3217+ else:
3218+ # Get parent from publishing history (which is the last
3219+ # published version in the corresponding series)
3220 try:
3221- dl_cache = args.dl_cache
3222+ applied_publish_parent_commit = str(applied_tip_head.peel().id)
3223 except AttributeError:
3224- dl_cache = None
3225- self.fixup_devel = args.fixup_devel
3226- self.active_series_only = args.active_series_only
3227- self.skip_orig = args.skip_orig
3228-
3229- if owner == 'usd-import-team':
3230- self.namespace = 'importer'
3231- else:
3232- self.namespace = owner
3233-
3234- logging.info('Ubuntu Server Team importer v%s' % VERSION)
3235-
3236- self.local_repo = GitUbuntuRepository(directory, user, args.proto)
3237- if not directory:
3238- logging.info('Using git repository at %s', self.local_repo.local_dir)
3239-
3240- atexit.register(self.cleanup, no_clean, self.local_repo.local_dir)
3241-
3242- self.local_repo.add_base_remotes(self.pkgname, repo_owner=owner)
3243- if not args.no_fetch and not args.reimport:
3244- try:
3245- self.local_repo.fetch_base_remotes()
3246- except GitUbuntuRepositoryFetchError:
3247- pass
3248-
3249- if not args.no_fetch:
3250- self.local_repo.delete_branches_in_namespace(self.namespace)
3251- self.local_repo.delete_tags_in_namespace(self.namespace)
3252- self.local_repo.copy_base_references(self.namespace)
3253-
3254- if not args.no_fetch and not args.reimport:
3255- try:
3256- self.local_repo.fetch_remote_refspecs('pkg',
3257- refspecs=['refs/tags/*:refs/tags/%s/*' % self.namespace],
3258- )
3259- except GitUbuntuRepositoryFetchError:
3260- pass
3261-
3262- if args.reimport:
3263- if args.no_fetch:
3264- logging.warning(
3265- "--reimport specified with --no-fetch. Upload "
3266- "tags would not be incorporated into the new import "
3267- "and would be lost forever. This is not currently "
3268- "supported and --no-fetch will be ignored for upload "
3269- "tags."
3270- )
3271 try:
3272- self.local_repo.fetch_remote_refspecs('pkg',
3273- refspecs=['refs/tags/upload/*:refs/tags/%s/upload/*' % self.namespace],
3274- )
3275- except GitUbuntuRepositoryFetchError:
3276+ applied_publish_parent_head = repo.get_head_by_name(spi.applied_parent_head_name(namespace))
3277+ applied_publish_parent_head_commit = str(applied_publish_parent_head.peel().id)
3278+ except AttributeError:
3279 pass
3280
3281- self.local_repo.ensure_importer_branches_exist(self.namespace)
3282+ if version_compare(str(spi.version), applied_tip_version) <= 0:
3283+ logging.warn('Version to import (%s) is not after %s tip (%s)',
3284+ spi.version, pretty_head_name, applied_tip_version
3285+ )
3286
3287- self.debian_sinfo = GitUbuntuSourceInformation('debian',
3288- self.pkgname, os.path.abspath(args.pullfile),
3289- args.retries, args.retry_backoffs)
3290+ # Walk changelog backwards until we find an imported version
3291+ for version in import_tree_versions:
3292+ applied_changelog_parent_tag = repo.get_applied_tag(version, namespace)
3293+ if applied_changelog_parent_tag is not None:
3294+ # sanity check that version from d/changelog of the
3295+ # tagged parent matches ours
3296+ parent_changelog_version, _ = \
3297+ repo.get_changelog_versions_from_treeish(
3298+ str(applied_changelog_parent_tag.peel(pygit2.Tree).id),
3299+ )
3300+ # if the parent was imported via a parent override
3301+ # itself, assume it is valid without checking its
3302+ # tree's debian/changelog, as it wasn't used
3303+ if (version not in _PARENT_OVERRIDES and
3304+ parent_changelog_version != version
3305+ ):
3306+ logging.error('Found a tag corresponding to '
3307+ 'parent version %s, but d/changelog '
3308+ 'version (%s) differs. Will '
3309+ 'orphan commit.' % (
3310+ version,
3311+ parent_changelog_version
3312+ )
3313+ )
3314+ else:
3315+ applied_changelog_parent_commit = str(applied_changelog_parent_tag.peel().id)
3316+ logging.debug('Changelog parent (tag) is %s',
3317+ repo.tag_to_pretty_name(applied_changelog_parent_tag)
3318+ )
3319+ break
3320
3321- self.ubuntu_sinfo = GitUbuntuSourceInformation('ubuntu',
3322- self.pkgname, os.path.abspath(args.pullfile),
3323- args.retries, args.retry_backoffs)
3324+ # If the two parents are tree-identical, then favor publication
3325+ # history. This deals with copy-forwards between series and with
3326+ # syncs.
3327+ if repo.treeishs_identical(applied_publish_parent_commit, applied_changelog_parent_commit):
3328+ applied_changelog_parent_commit = None
3329
3330- debian_head_versions = (
3331- self.local_repo.get_heads_and_versions('debian', self.namespace)
3332- )
3333- for head_name in sorted(debian_head_versions):
3334- _, _, pretty_head_name = head_name.partition('%s/' % self.namespace)
3335- logging.debug('Last imported %s version is %s',
3336- pretty_head_name,
3337- debian_head_versions[head_name]['version']
3338+ # Assume no patches to apply
3339+ applied_import_tree_hash = unapplied_import_tree_hash
3340+ # get tree id from above commit
3341+ for (
3342+ applied_import_tree_hash,
3343+ patch_name,
3344+ patch_desc
3345+ ) in import_patches_applied_tree(repo, spi.dsc_pathname):
3346+ # special case for .pc removal
3347+ if patch_name is None:
3348+ msg = b'%b.' % (patch_desc.encode())
3349+ else:
3350+ if patch_desc is None:
3351+ patch_desc = (
3352+ "%s\n\nNo DEP3 Subject or Description header found" %
3353+ patch_name
3354+ )
3355+ msg = b'%b\n\nGbp-Pq: %b.' % (
3356+ patch_desc.encode(),
3357+ patch_name.encode()
3358 )
3359+ unapplied_parent_commit = repo.commit_tree_hash(
3360+ applied_import_tree_hash,
3361+ [unapplied_parent_commit],
3362+ msg,
3363+ spi
3364+ )
3365
3366- ubuntu_head_versions = (
3367- self.local_repo.get_heads_and_versions('ubuntu', self.namespace)
3368+ logging.debug(
3369+ "Committed patch-application of %s as %s",
3370+ patch_name, unapplied_parent_commit
3371 )
3372- for head_name in sorted(ubuntu_head_versions):
3373- _, _, pretty_head_name = head_name.partition('%s/' % self.namespace)
3374- logging.debug('Last imported %s version is %s',
3375- pretty_head_name,
3376- ubuntu_head_versions[head_name]['version']
3377- )
3378
3379- if not args.skip_applied:
3380- applied_debian_head_versions = (
3381- self.local_repo.get_heads_and_versions(
3382- 'applied/debian', self.namespace
3383- )
3384- )
3385- for head_name in sorted(applied_debian_head_versions):
3386- _, _, pretty_head_name = head_name.partition(
3387- '%s/' % self.namespace
3388- )
3389- logging.debug('Last applied %s version is %s',
3390- pretty_head_name,
3391- applied_debian_head_versions[head_name]['version']
3392+ commit_applied_patches_import(
3393+ repo,
3394+ spi,
3395+ applied_import_tree_hash,
3396+ namespace,
3397+ applied_publish_parent_commit,
3398+ applied_changelog_parent_commit,
3399+ unapplied_parent_commit,
3400+ )
3401+
3402+ update_applied_devel_branches(
3403+ repo=repo,
3404+ pkgname=spi.name,
3405+ namespace=namespace,
3406+ ubuntu_sinfo=ubuntu_sinfo,
3407+ spi=spi,
3408+ )
3409+
3410+def import_publishes(repo, pkgname, namespace, patches_applied,
3411+ debian_head_versions, ubuntu_head_versions, debian_sinfo,
3412+ ubuntu_sinfo, active_series_only, workdir, fixup_devel,
3413+ skip_orig,
3414+):
3415+ history_found = False
3416+ only_debian = False
3417+ srcpkg_information = None
3418+ if patches_applied:
3419+ _namespace = namespace
3420+ namespace = '%s/applied' % namespace
3421+ import_type = 'patches-applied'
3422+ import_func = import_applied_spi
3423+ else:
3424+ _namespace = namespace
3425+ import_type = 'patches-unapplied'
3426+ import_func = functools.partial(
3427+ import_unapplied_spi,
3428+ skip_orig=skip_orig
3429+ )
3430+ for distname, versions, dist_sinfo in (
3431+ ("debian", debian_head_versions, debian_sinfo),
3432+ ("ubuntu", ubuntu_head_versions, ubuntu_sinfo)):
3433+ if active_series_only and distname == "debian":
3434+ continue
3435+ try:
3436+ for srcpkg_information in dist_sinfo.launchpad_versions_published_after(
3437+ versions,
3438+ namespace,
3439+ workdir=workdir,
3440+ active_series_only=active_series_only
3441+ ):
3442+ history_found = True
3443+ import_func(
3444+ repo=repo,
3445+ spi=srcpkg_information,
3446+ namespace=_namespace,
3447+ ubuntu_sinfo=ubuntu_sinfo,
3448 )
3449-
3450- applied_ubuntu_head_versions = (
3451- self.local_repo.get_heads_and_versions(
3452- 'applied/ubuntu', self.namespace
3453+ # Do not push upstream
3454+ if fixup_devel and distname == "ubuntu":
3455+ update_devel_branches(
3456+ repo=repo,
3457+ namespace=namespace,
3458+ pkgname=pkgname,
3459+ ubuntu_sinfo=ubuntu_sinfo,
3460+ spi=None,
3461 )
3462+ except NoPublicationHistoryException:
3463+ logging.warning("No publication history found for %s in %s.",
3464+ pkgname, distname
3465 )
3466- for head_name in sorted(applied_ubuntu_head_versions):
3467- _, _, pretty_head_name = head_name.partition(
3468- '%s/' % self.namespace
3469- )
3470- logging.debug('Last applied %s version is %s',
3471- pretty_head_name,
3472- applied_ubuntu_head_versions[head_name]['version']
3473- )
3474-
3475- oldcwd = os.getcwd()
3476- os.chdir(self.local_repo.local_dir)
3477-
3478- if dl_cache is None:
3479- self.workdir = os.path.join(self.local_repo.git_dir, CACHE_PATH)
3480- else:
3481- self.workdir = dl_cache
3482-
3483- os.makedirs(self.workdir, exist_ok=True)
3484-
3485- self.parse_parentfile(args.parentfile)
3486-
3487- only_debian, history_found = self.import_publishes(
3488- patches_applied=False,
3489- debian_head_versions=debian_head_versions,
3490- ubuntu_head_versions=ubuntu_head_versions)
3491-
3492- if not history_found:
3493- logging.error("No publication history for '%s' in debian or ubuntu."
3494- " Wrong source package name?", self.pkgname)
3495- sys.exit(1)
3496-
3497- if not args.skip_applied:
3498- self.import_publishes(patches_applied=True,
3499- debian_head_versions=applied_debian_head_versions,
3500- ubuntu_head_versions=applied_ubuntu_head_versions)
3501-
3502- os.chdir(oldcwd)
3503-
3504- self.local_repo.garbage_collect()
3505-
3506- if no_push:
3507- logging.info('Not pushing to remote as specified')
3508+ if distname == 'ubuntu':
3509+ only_debian = True
3510+ except Exception as e:
3511+ if srcpkg_information is None:
3512+ msg = 'Unable to import %s to %s' % (import_type, distname)
3513+ else:
3514+ msg = 'Unable to import %s %s to %s' % (import_type,
3515+ str(srcpkg_information.version), distname)
3516+ if not patches_applied:
3517+ raise GitUbuntuImportError(msg) from e
3518+ else:
3519+ logging.error(msg)
3520 else:
3521- lp = launchpad_login_auth()
3522- repo_path = '~%s/ubuntu/+source/%s/+git/%s' % (
3523- owner,
3524- self.pkgname,
3525- self.pkgname,
3526- )
3527- lp_git_repo = lp.git_repositories.getByPath(path=repo_path)
3528- if args.reimport:
3529- if not lp_git_repo.canBeDeleted():
3530- logging.warning(
3531- "There are linked MPs to the pkg repository, "
3532- "which will be deleted:"
3533- )
3534- for mp in lp_git_repo.landing_candidates:
3535- logging.warning(
3536- "Adding comment to %s indicating reimport",
3537- mp.web_link
3538- )
3539- mp.createComment(
3540- content="%s has been reimported and this MP "
3541- "must be manually resubmitted." %
3542- self.pkgname,
3543- subject="%s has been reimported" % self.pkgname,
3544+ history_found = True
3545+
3546+ return (only_debian, history_found)
3547+
3548+
3549+def parse_args(subparsers=None, base_subparsers=None):
3550+ kwargs = dict(description='Update a launchpad git tree based upon '
3551+ 'the state of the Ubuntu and Debian archives',
3552+ formatter_class=argparse.ArgumentDefaultsHelpFormatter
3553+ )
3554+ if base_subparsers:
3555+ kwargs['parents'] = base_subparsers
3556+ if subparsers:
3557+ parser = subparsers.add_parser('import', **kwargs)
3558+ parser.set_defaults(func=cli_main)
3559+ else:
3560+ parser = argparse.ArgumentParser(**kwargs)
3561+ parser.add_argument('package', type=str,
3562+ help='Which package to update in the git tree')
3563+ parser.add_argument('-o', '--lp-owner', type=str,
3564+ help=argparse.SUPPRESS,
3565+ default='usd-import-team')
3566+ parser.add_argument('-l', '--lp-user', type=str,
3567+ help=argparse.SUPPRESS)
3568+ parser.add_argument('--dl-cache', type=str,
3569+ help=('Cache directory for downloads. Default is to '
3570+ 'put downloads in <directory>/.git/%s' %
3571+ CACHE_PATH
3572+ ),
3573+ default=argparse.SUPPRESS)
3574+ parser.add_argument('--no-fetch', action='store_true',
3575+ help='Do not fetch from the remote (DANGEROUS; '
3576+ 'only useful for debugging); implies '
3577+ '--no-push')
3578+ parser.add_argument('--no-push', action='store_true',
3579+ help='Do not push to the remote')
3580+ parser.add_argument('--no-clean', action='store_true',
3581+ help='Do not clean the temporary directory')
3582+ parser.add_argument('-d', '--directory', type=str,
3583+ help='Use git repository at specified location rather '
3584+ 'than a temporary directory (implies --no-clean). '
3585+ 'Will create the local repository if needed.',
3586+ default=argparse.SUPPRESS
3587 )
3588- # repo deletion will delete the MP, but hopefully
3589- # the above comment will be sufficient to explain
3590- # why
3591- lp_git_repo.lp_delete()
3592-
3593- self.local_repo.git_run([
3594- 'push', '--atomic', 'pkg',
3595- 'refs/heads/%s/*:refs/heads/*' % self.namespace,
3596- 'refs/tags/%s/*:refs/tags/*' % self.namespace,
3597- ])
3598- # update our reference
3599- lp_git_repo = lp.git_repositories.getByPath(path=repo_path)
3600- for i in range(args.retries):
3601- try:
3602- if only_debian:
3603- lp_git_repo.default_branch = 'refs/heads/debian/sid'
3604- else:
3605- lp_git_repo.default_branch = 'refs/heads/ubuntu/devel'
3606- lp_git_repo.lp_save()
3607- break
3608- except (NotFound, PreconditionFailed) as e:
3609- time.sleep(args.retry_backoffs[i])
3610- lp_git_repo.lp_refresh()
3611+ parser.add_argument('--fixup-devel', action='store_true',
3612+ help=argparse.SUPPRESS)
3613+ parser.add_argument('--active-series-only', action='store_true',
3614+ help='Do an import of only active Ubuntu series '
3615+ 'history, implies --no-push.')
3616+ parser.add_argument('--skip-applied', action='store_true',
3617+ help=argparse.SUPPRESS)
3618+ parser.add_argument('--skip-orig', action='store_true',
3619+ help=argparse.SUPPRESS)
3620+ parser.add_argument('--reimport', action='store_true',
3621+ help=argparse.SUPPRESS)
3622+ if not subparsers:
3623+ return parser.parse_args()
3624+ return 'import - %s' % kwargs['description']
3625+
3626+def cli_main(args):
3627+ no_clean = args.no_clean
3628+ try:
3629+ directory = args.directory
3630+ no_clean = True
3631+ except AttributeError:
3632+ directory = None
3633+ try:
3634+ user = args.lp_user
3635+ except AttributeError:
3636+ user = None
3637+ # --active-series-only/--no-fetch to a tmpdir/--skip-orig
3638+ # implies --no-push
3639+ no_push = (
3640+ args.no_push or
3641+ (args.no_fetch and not directory) or
3642+ args.active_series_only or
3643+ args.skip_orig
3644+ )
3645+ try:
3646+ dl_cache = args.dl_cache
3647+ except AttributeError:
3648+ dl_cache = None
3649+
3650+ main(
3651+ args.package,
3652+ args.lp_owner,
3653+ no_clean,
3654+ directory,
3655+ user,
3656+ args.proto,
3657+ args.no_fetch,
3658+ no_push,
3659+ dl_cache,
3660+ args.fixup_devel,
3661+ args.active_series_only,
3662+ args.skip_orig,
3663+ args.pullfile,
3664+ args.parentfile,
3665+ args.retries,
3666+ args.retry_backoffs,
3667+ args.skip_applied,
3668+ args.reimport,
3669+ )
3670diff --git a/gitubuntu/importlocal.py b/gitubuntu/importlocal.py
3671index bf1700e..c237551 100644
3672--- a/gitubuntu/importlocal.py
3673+++ b/gitubuntu/importlocal.py
3674@@ -2,15 +2,22 @@ import argparse
3675 import glob
3676 import logging
3677 import os
3678-import re
3679-from subprocess import CalledProcessError
3680 import sys
3681 import tempfile
3682 from gitubuntu.dsc import GitUbuntuDsc
3683 from gitubuntu.git_repository import GitUbuntuRepository, git_dep14_tag
3684-from gitubuntu.run import decode_binary, run, runq
3685+from gitubuntu.run import run, runq
3686 from gitubuntu.version import VERSION
3687-from gitubuntu.importer import GitUbuntuImport
3688+from gitubuntu.importer import (
3689+ parse_parentfile,
3690+ _PARENT_OVERRIDES,
3691+ import_orig,
3692+ import_dsc,
3693+ get_changelog_for_commit,
3694+ import_patches_unapplied_tree,
3695+ import_patches_applied_tree,
3696+)
3697+from gitubuntu.source_information import derive_source_from_series
3698 from gitubuntu.versioning import version_compare
3699 try:
3700 pkg = 'python3-pygit2'
3701@@ -20,264 +27,309 @@ except ImportError:
3702 sys.exit(1)
3703
3704
3705-class GitUbuntuImportLocal(GitUbuntuImport):
3706- def __init__(self):
3707- self.parent_overrides = None
3708-
3709- def parse_args(self, subparsers=None, base_subparsers=None):
3710- kwargs = dict(description='Import a DSC file locally',
3711- #usage='%(prog)s [options] -- ...',
3712- epilog='Unlike Launchpad imports, local imports '
3713- 'require specifying a namespace to uniquely '
3714- 'identify the source of the DSC file. This '
3715- 'means that if you want to tie history '
3716- 'together for several DSCs from the same '
3717- 'source, you must use the same namespace value.',
3718- formatter_class=argparse.ArgumentDefaultsHelpFormatter
3719- )
3720- if base_subparsers:
3721- kwargs['parents'] = base_subparsers
3722- if subparsers:
3723- parser = subparsers.add_parser('import-local', **kwargs)
3724- parser.set_defaults(func=self.main)
3725- else:
3726- parser = argparse.ArgumentParser(**kwargs)
3727- parser.add_argument('namespace',
3728- help='String namespace to use for tagging the imports')
3729- parser.add_argument('-d', '--directory', type=str,
3730- help='Use git repository at specified location rather '
3731- 'than a temporary directory. '
3732- 'Will create the local repository if needed.',
3733- default=argparse.SUPPRESS
3734- )
3735- parser.add_argument('dsc', type=str,
3736- help='Path to or URL of DSC file')
3737- parser.add_argument('--skip-applied', action='store_true',
3738- help=argparse.SUPPRESS)
3739- parser.add_argument('--skip-orig', action='store_true',
3740- help=argparse.SUPPRESS)
3741- if not subparsers:
3742- return parser.parse_args()
3743- return 'import-local - %s' % kwargs['description']
3744-
3745- def main(self, args):
3746- try:
3747- directory = args.directory
3748- no_clean = True
3749- except AttributeError:
3750- directory = None
3751- self.namespace = args.namespace
3752- self.dsc_path = args.dsc
3753- self.skip_applied = args.skip_applied
3754- self.skip_orig = args.skip_orig
3755-
3756- self.local_repo = GitUbuntuRepository(directory)
3757- if not directory:
3758- logging.info('Using git repository at %s', self.local_repo.local_dir)
3759-
3760- self.local_repo.ensure_importer_branches_exist(self.namespace)
3761- if len(self.local_repo.raw_repo.status()) != 0:
3762- logging.error('Working tree must be clean to continue.')
3763- run(['git', 'status'], stdout=None)
3764- sys.exit(1)
3765-
3766- if os.path.exists(self.dsc_path):
3767- self.dsc_path = os.path.abspath(self.dsc_path)
3768- else:
3769- dsc_dir = tempfile.mkdtemp()
3770-
3771- oldcwd = os.getcwd()
3772- os.chdir(dsc_dir)
3773-
3774- runq(['dget', '--quiet', '--allow-unauthenticated',
3775- '--download-only', self.dsc_path])
3776- dscs = glob.glob(os.path.join(dsc_dir, '*.dsc'))
3777- if len(dscs) == 0:
3778- logging.error('Unable to obtain DSC file.')
3779- sys.exit(1)
3780- if len(dscs) > 1:
3781- logging.error('Multiple DSCs found in temporary directory.')
3782- sys.exit(1)
3783- self.dsc_path = os.path.join(dsc_dir, dscs[0])
3784- os.chdir(oldcwd)
3785- logging.debug('Importing DSC at local path %s' % self.dsc_path)
3786- self.dsc = GitUbuntuDsc(self.dsc_path)
3787- if not self.dsc.verify():
3788- logging.error('Unable to verify orig tarball specified by DSC %s',
3789- self.dsc_path)
3790- sys.exit(1)
3791-
3792- self.pkgname = self.dsc['Source']
3793+def main(
3794+ directory,
3795+ namespace,
3796+ dsc_path,
3797+ skip_applied,
3798+ skip_orig,
3799+ parentfile,
3800+):
3801+ repo = GitUbuntuRepository(directory)
3802+ if not directory:
3803+ logging.info('Using git repository at %s', repo.local_dir)
3804+
3805+ repo.ensure_importer_branches_exist(namespace)
3806+ if len(repo.raw_repo.status()) != 0:
3807+ logging.error('Working tree must be clean to continue.')
3808+ run(['git', 'status'], stdout=None)
3809+ sys.exit(1)
3810+
3811+ if os.path.exists(dsc_path):
3812+ dsc_path = os.path.abspath(dsc_path)
3813+ else:
3814+ dsc_dir = tempfile.mkdtemp()
3815
3816 oldcwd = os.getcwd()
3817- os.chdir(self.local_repo.local_dir)
3818-
3819- self.local_repo.clean_repository_state()
3820+ os.chdir(dsc_dir)
3821
3822- self.parse_parentfile(args.parentfile)
3823-
3824- unapplied_import_tree_hash = self.import_patches_unapplied_tree(self.dsc_path)
3825-
3826- import_tree_versions = self.local_repo.get_all_changelog_versions_from_treeish(
3827- unapplied_import_tree_hash
3828- )
3829-
3830- changelog_version = import_tree_versions[0]
3831-
3832- logging.debug(
3833- 'Found changelog version %s in newly imported tree' %
3834- changelog_version
3835- )
3836-
3837- head = None
3838- head_version = None
3839- if not self.local_repo.raw_repo.head_is_unborn:
3840- head = self.local_repo.raw_repo.head
3841- # check if the version to import has already been imported to
3842- # this head
3843- if head is not None:
3844- head_version, _ = self.local_repo.get_changelog_versions_from_treeish(
3845- str(head.peel().id))
3846-
3847- logging.debug('Head version is %s', head_version)
3848-
3849- unapplied_changelog_parent_commit = None
3850- applied_changelog_parent_commit = None
3851-
3852- if version_compare(changelog_version, head_version) <= 0:
3853- logging.warn('Version to import (%s) is not after HEAD '
3854- 'version (%s), history may not make sense.',
3855- changelog_version, head_version
3856- )
3857-
3858- # Walk changelog backwards until we find an imported version
3859- for version in import_tree_versions:
3860- unapplied_tag_ref = '%s/import/%s' % (self.namespace, git_dep14_tag(version))
3861- applied_tag_ref = '%s/applied/%s' % (self.namespace, git_dep14_tag(version))
3862- unapplied_changelog_parent_tag = self.local_repo.get_tag_reference(unapplied_tag_ref)
3863- applied_changelog_parent_tag = self.local_repo.get_tag_reference(applied_tag_ref)
3864- if unapplied_changelog_parent_tag is not None:
3865- # sanity check that version from d/changelog of the
3866- # tagged parent matches ours
3867- parent_changelog_version, _ = \
3868- self.local_repo.get_changelog_versions_from_treeish(
3869- str(unapplied_changelog_parent_tag.peel(pygit2.Tree).id),
3870- )
3871- # if the parent was imported via a parent override
3872- # itself, assume it is valid without checking its
3873- # tree's debian/changelog, as it wasn't used
3874- if (version not in self.parent_overrides and
3875- parent_changelog_version != version
3876- ):
3877- logging.error('Found a tag corresponding to '
3878- 'parent version %s, but d/changelog '
3879- 'version (%s) differs. Will '
3880- 'orphan commit.' % (
3881- version,
3882- parent_changelog_version
3883- )
3884- )
3885- else:
3886- unapplied_changelog_parent_commit = str(unapplied_changelog_parent_tag.peel().id)
3887- if applied_changelog_parent_tag:
3888- applied_changelog_parent_commit = str(applied_changelog_parent_tag.peel().id)
3889- logging.debug('Changelog parent (tag) is %s',
3890- self.local_repo.tag_to_pretty_name(unapplied_changelog_parent_tag)
3891- )
3892- break
3893-
3894- distribution = self.local_repo.get_changelog_distribution_from_treeish(unapplied_import_tree_hash)
3895- # do lookup of series for dist on lp?
3896- dist = 'ubuntu'
3897- if distribution in ("stable", "testing", "unstable", "experimental"):
3898- dist = 'debian'
3899- self.import_dsc(self.dsc, changelog_version, dist)
3900- try:
3901- if not self.skip_orig:
3902- self.import_orig(self.dsc, changelog_version, dist)
3903- except Exception as e:
3904- logging.error('Unable to import orig tarball for %s: %s',
3905- changelog_version, e)
3906+ runq(['dget', '--quiet', '--allow-unauthenticated',
3907+ '--download-only', dsc_path])
3908+ dscs = glob.glob(os.path.join(dsc_dir, '*.dsc'))
3909+ if len(dscs) == 0:
3910+ logging.error('Unable to obtain DSC file.')
3911 sys.exit(1)
3912-
3913- msg = (b'Import unapplied version %b\n\nImported using git-ubuntu-import.' %
3914- (changelog_version.encode())
3915- )
3916-
3917- parents = []
3918-
3919- tag = None
3920- tag_ref = '%s/import/%s' % (self.namespace,
3921- git_dep14_tag(changelog_version))
3922- if self.local_repo.get_tag_reference(tag_ref) is None:
3923- # Not imported before
3924- tag = tag_ref
3925-
3926- if unapplied_changelog_parent_commit is not None:
3927- msg += b'\n'
3928- parents.append(unapplied_changelog_parent_commit)
3929- msg += b'\nChangelog parent: %b' % unapplied_changelog_parent_commit.encode()
3930-
3931- changelog_entry = self.get_changelog_for_commit(
3932- unapplied_import_tree_hash,
3933- None,
3934- unapplied_changelog_parent_commit
3935- )
3936-
3937- msg += b'%b' % changelog_entry
3938- unapplied_parent_commit = self.local_repo.commit_tree_hash(
3939- unapplied_import_tree_hash, parents, msg)
3940- if tag is None:
3941- logging.info('Committed unapplied import of %s as %s',
3942- changelog_version, unapplied_parent_commit)
3943- else:
3944- logging.debug('Creating tag %s pointing to %s', tag,
3945- unapplied_parent_commit)
3946- self.local_repo.git_run(['tag', '-a', '-m', 'git-ubuntu import v%s' % VERSION,
3947- tag, unapplied_parent_commit
3948- ]
3949- )
3950-
3951- if not args.skip_applied:
3952- # Assume no patches to apply
3953- applied_import_tree_hash = unapplied_import_tree_hash
3954- for (applied_import_tree_hash, patch_name, patch_desc) in self.import_patches_applied_tree(self.dsc_path):
3955- # special case for .pc removal
3956- if patch_name is None:
3957- msg = b'%b.' % (patch_desc.encode())
3958- else:
3959- if patch_desc is None:
3960- patch_desc = '%s\n\nNo DEP3 Subject or Description header found' % patch_name
3961- msg = b'%b\n\nGbp-Pq: %b.' % (patch_desc.encode(), patch_name.encode())
3962- unapplied_parent_commit = self.local_repo.commit_tree_hash(
3963- applied_import_tree_hash,
3964- [unapplied_parent_commit],
3965- msg)
3966-
3967- logging.debug('Committed patch-application of %s as %s',
3968- patch_name, unapplied_parent_commit
3969- )
3970-
3971- tag = None
3972- tag_ref = '%s/applied/%s' % (self.namespace,
3973- git_dep14_tag(changelog_version))
3974- if self.local_repo.get_tag_reference(tag_ref) is None:
3975- # Not imported before
3976- tag= tag_ref
3977-
3978- if tag is None:
3979- logging.info('Committed applied import of %s as %s',
3980- changelog_version, unapplied_parent_commit)
3981- else:
3982- logging.debug('Creating tag %s pointing to %s', tag,
3983- unapplied_parent_commit)
3984- self.local_repo.git_run(['tag', '-a', '-m', 'git-ubuntu import v%s' % VERSION,
3985- tag, unapplied_parent_commit
3986- ]
3987- )
3988-
3989- self.local_repo.clean_repository_state()
3990-
3991+ if len(dscs) > 1:
3992+ logging.error('Multiple DSCs found in temporary directory.')
3993+ sys.exit(1)
3994+ dsc_path = os.path.join(dsc_dir, dscs[0])
3995 os.chdir(oldcwd)
3996-
3997- self.local_repo.garbage_collect()
3998+ logging.debug('Importing DSC at local path %s', dsc_path)
3999+ dsc = GitUbuntuDsc(dsc_path)
4000+ if not dsc.verify():
4001+ logging.error(
4002+ 'Unable to verify orig tarball specified by DSC %s',
4003+ dsc_path,
4004+ )
4005+ sys.exit(1)
4006+
4007+ pkgname = dsc['Source']
4008+
4009+ oldcwd = os.getcwd()
4010+ os.chdir(repo.local_dir)
4011+
4012+ repo.clean_repository_state()
4013+
4014+ parse_parentfile(parentfile, pkgname)
4015+
4016+ unapplied_import_tree_hash = import_patches_unapplied_tree(repo, dsc_path)
4017+
4018+ import_tree_versions = repo.get_all_changelog_versions_from_treeish(
4019+ unapplied_import_tree_hash
4020+ )
4021+
4022+ changelog_version = import_tree_versions[0]
4023+
4024+ logging.debug(
4025+ 'Found changelog version %s in newly imported tree',
4026+ changelog_version
4027+ )
4028+
4029+ head = None
4030+ head_version = None
4031+ if not repo.raw_repo.head_is_unborn:
4032+ head = repo.raw_repo.head
4033+ # check if the version to import has already been imported to
4034+ # this head
4035+ if head is not None:
4036+ head_version, _ = repo.get_changelog_versions_from_treeish(
4037+ str(head.peel().id)
4038+ )
4039+
4040+ logging.debug('Head version is %s', head_version)
4041+
4042+ unapplied_changelog_parent_commit = None
4043+ applied_changelog_parent_commit = None
4044+
4045+ if version_compare(changelog_version, head_version) <= 0:
4046+ logging.warn('Version to import (%s) is not after HEAD '
4047+ 'version (%s), history may not make sense.',
4048+ changelog_version, head_version
4049+ )
4050+
4051+ # Walk changelog backwards until we find an imported version
4052+ for version in import_tree_versions:
4053+ unapplied_tag_ref = '%s/import/%s' % (namespace, git_dep14_tag(version))
4054+ applied_tag_ref = '%s/applied/%s' % (namespace, git_dep14_tag(version))
4055+ unapplied_changelog_parent_tag = repo.get_tag_reference(unapplied_tag_ref)
4056+ applied_changelog_parent_tag = repo.get_tag_reference(applied_tag_ref)
4057+ if unapplied_changelog_parent_tag is not None:
4058+ # sanity check that version from d/changelog of the
4059+ # tagged parent matches ours
4060+ parent_changelog_version, _ = \
4061+ repo.get_changelog_versions_from_treeish(
4062+ str(unapplied_changelog_parent_tag.peel(pygit2.Tree).id),
4063+ )
4064+ # if the parent was imported via a parent override
4065+ # itself, assume it is valid without checking its
4066+ # tree's debian/changelog, as it wasn't used
4067+ if (version not in _PARENT_OVERRIDES and
4068+ parent_changelog_version != version
4069+ ):
4070+ logging.error('Found a tag corresponding to '
4071+ 'parent version %s, but d/changelog '
4072+ 'version (%s) differs. Will '
4073+ 'orphan commit.',
4074+ version,
4075+ parent_changelog_version
4076+ )
4077+ else:
4078+ unapplied_changelog_parent_commit = str(
4079+ unapplied_changelog_parent_tag.peel().id
4080+ )
4081+ if applied_changelog_parent_tag:
4082+ applied_changelog_parent_commit = str(
4083+ applied_changelog_parent_tag.peel().id
4084+ )
4085+ logging.debug('Changelog parent (tag) is %s',
4086+ repo.tag_to_pretty_name(unapplied_changelog_parent_tag)
4087+ )
4088+ break
4089+
4090+ distribution = repo.get_changelog_distribution_from_treeish(
4091+ unapplied_import_tree_hash
4092+ )
4093+ dist = derive_source_from_series(distribution)
4094+ import_dsc(
4095+ repo=repo,
4096+ dsc=dsc,
4097+ namespace=namespace,
4098+ version=changelog_version,
4099+ dist=dist,
4100+ )
4101+ try:
4102+ if not skip_orig:
4103+ import_orig(
4104+ repo=repo,
4105+ dsc=dsc,
4106+ namespace=namespace,
4107+ version=changelog_version,
4108+ dist=dist,
4109+ )
4110+ except Exception as e:
4111+ logging.error('Unable to import orig tarball for %s: %s',
4112+ changelog_version, e)
4113+ raise
4114+ sys.exit(1)
4115+
4116+ msg = (
4117+ b'Import unapplied version %b\n\nImported using git-ubuntu-import.' %
4118+ (changelog_version.encode())
4119+ )
4120+
4121+ parents = []
4122+
4123+ tag = None
4124+ tag_ref = '%s/import/%s' % (
4125+ namespace,
4126+ git_dep14_tag(changelog_version)
4127+ )
4128+ if repo.get_tag_reference(tag_ref) is None:
4129+ # Not imported before
4130+ tag = tag_ref
4131+
4132+ if unapplied_changelog_parent_commit is not None:
4133+ msg += b'\n'
4134+ parents.append(unapplied_changelog_parent_commit)
4135+ msg += b'\nChangelog parent: %b' % unapplied_changelog_parent_commit.encode()
4136+
4137+ changelog_entry = get_changelog_for_commit(
4138+ repo=repo,
4139+ tree_hash=unapplied_import_tree_hash,
4140+ publish_parent_commit=None,
4141+ changelog_parent_commit=unapplied_changelog_parent_commit,
4142+ )
4143+
4144+ msg += b'%b' % changelog_entry
4145+ unapplied_parent_commit = repo.commit_tree_hash(
4146+ unapplied_import_tree_hash, parents, msg
4147+ )
4148+ if tag is None:
4149+ logging.info('Committed unapplied import of %s as %s',
4150+ changelog_version, unapplied_parent_commit)
4151+ else:
4152+ logging.debug('Creating tag %s pointing to %s', tag,
4153+ unapplied_parent_commit
4154+ )
4155+ repo.git_run(['tag', '-a', '-m', 'git-ubuntu import v%s' % VERSION,
4156+ tag, unapplied_parent_commit
4157+ ]
4158+ )
4159+
4160+ if not skip_applied:
4161+ # Assume no patches to apply
4162+ applied_import_tree_hash = unapplied_import_tree_hash
4163+ for (
4164+ applied_import_tree_hash,
4165+ patch_name,
4166+ patch_desc,
4167+ ) in import_patches_applied_tree(repo, dsc_path):
4168+ # special case for .pc removal
4169+ if patch_name is None:
4170+ msg = b'%b.' % (patch_desc.encode())
4171+ else:
4172+ if patch_desc is None:
4173+ patch_desc = '%s\n\nNo DEP3 Subject or Description header found' % patch_name
4174+ msg = b'%b\n\nGbp-Pq: %b.' % (patch_desc.encode(), patch_name.encode())
4175+ unapplied_parent_commit = repo.commit_tree_hash(
4176+ applied_import_tree_hash,
4177+ [unapplied_parent_commit],
4178+ msg)
4179+
4180+ logging.debug('Committed patch-application of %s as %s',
4181+ patch_name, unapplied_parent_commit
4182+ )
4183+
4184+ tag = None
4185+ tag_ref = '%s/applied/%s' % (namespace,
4186+ git_dep14_tag(changelog_version))
4187+ if repo.get_tag_reference(tag_ref) is None:
4188+ # Not imported before
4189+ tag = tag_ref
4190+
4191+ if tag is None:
4192+ logging.info('Committed applied import of %s as %s',
4193+ changelog_version,
4194+ unapplied_parent_commit
4195+ )
4196+ else:
4197+ logging.debug('Creating tag %s pointing to %s', tag,
4198+ unapplied_parent_commit
4199+ )
4200+ repo.git_run(
4201+ [
4202+ 'tag',
4203+ '-a',
4204+ '-m',
4205+ 'git-ubuntu import v%s' % VERSION,
4206+ tag,
4207+ unapplied_parent_commit,
4208+ ]
4209+ )
4210+
4211+ repo.clean_repository_state()
4212+
4213+ os.chdir(oldcwd)
4214+
4215+ repo.garbage_collect()
4216+
4217+def parse_args(subparsers=None, base_subparsers=None):
4218+ kwargs = dict(description='Import a DSC file locally',
4219+ #usage='%(prog)s [options] -- ...',
4220+ epilog='Unlike Launchpad imports, local imports '
4221+ 'require specifying a namespace to uniquely '
4222+ 'identify the source of the DSC file. This '
4223+ 'means that if you want to tie history '
4224+ 'together for several DSCs from the same '
4225+ 'source, you must use the same namespace value.',
4226+ formatter_class=argparse.ArgumentDefaultsHelpFormatter
4227+ )
4228+ if base_subparsers:
4229+ kwargs['parents'] = base_subparsers
4230+ if subparsers:
4231+ parser = subparsers.add_parser('import-local', **kwargs)
4232+ parser.set_defaults(func=cli_main)
4233+ else:
4234+ parser = argparse.ArgumentParser(**kwargs)
4235+ parser.add_argument('namespace',
4236+ help='String namespace to use for tagging the imports')
4237+ parser.add_argument('-d', '--directory', type=str,
4238+ help='Use git repository at specified location rather '
4239+ 'than a temporary directory. '
4240+ 'Will create the local repository if needed.',
4241+ default=argparse.SUPPRESS
4242+ )
4243+ parser.add_argument('dsc', type=str,
4244+ help='Path to or URL of DSC file')
4245+ parser.add_argument('--skip-applied', action='store_true',
4246+ help=argparse.SUPPRESS)
4247+ parser.add_argument('--skip-orig', action='store_true',
4248+ help=argparse.SUPPRESS)
4249+ if not subparsers:
4250+ return parser.parse_args()
4251+ return 'import-local - %s' % kwargs['description']
4252+
4253+def cli_main(args):
4254+ try:
4255+ directory = args.directory
4256+ except AttributeError:
4257+ directory = None
4258+
4259+ main(
4260+ directory,
4261+ args.namespace,
4262+ args.dsc,
4263+ args.skip_applied,
4264+ args.skip_orig,
4265+ args.parentfile,
4266+ )
4267diff --git a/gitubuntu/importppa.py b/gitubuntu/importppa.py
4268index 2ae1f81..da57f53 100644
4269--- a/gitubuntu/importppa.py
4270+++ b/gitubuntu/importppa.py
4271@@ -10,8 +10,25 @@ from gitubuntu.git_repository import (
4272 GitUbuntuRepository,
4273 GitUbuntuRepositoryFetchError,
4274 )
4275-from gitubuntu.importer import GitUbuntuImport
4276-from gitubuntu.source_information import GitUbuntuSourceInformation, NoPublicationHistoryException, SourceExtractionException, launchpad_login_auth
4277+from gitubuntu.importer import (
4278+ cleanup,
4279+ parse_parentfile,
4280+ _PARENT_OVERRIDES,
4281+ import_orig,
4282+ import_dsc,
4283+ get_changelog_for_commit,
4284+ import_unapplied_spi,
4285+ import_applied_spi,
4286+ import_patches_unapplied_tree,
4287+ import_patches_applied_tree,
4288+)
4289+from gitubuntu.source_information import (
4290+ GitUbuntuSourceInformation,
4291+ NoPublicationHistoryException,
4292+ SourceExtractionException,
4293+ launchpad_login_auth,
4294+ GitUbuntuSourceInformation,
4295+)
4296 from gitubuntu.version import VERSION
4297 try:
4298 pkg = 'python3-pygit2'
4299@@ -23,181 +40,216 @@ except ImportError:
4300 sys.exit(1)
4301
4302
4303-class GitUbuntuImportPPA(GitUbuntuImport):
4304- def __init__(self):
4305- self.parent_overrides = None
4306-
4307- def parse_args(self, subparsers=None, base_subparsers=None):
4308- kwargs = dict(description='Update a launchpad git tree based upon '
4309- 'the state of a PPA archive',
4310- epilog='PPA must be specified as ppa:<name>',
4311- formatter_class=argparse.ArgumentDefaultsHelpFormatter
4312- )
4313- if base_subparsers:
4314- kwargs['parents'] = base_subparsers
4315- if subparsers:
4316- parser = subparsers.add_parser('import-ppa', **kwargs)
4317- parser.set_defaults(func=self.main)
4318- else:
4319- parser = argparse.ArgumentParser(**kwargs)
4320- parser.add_argument('ppa', type=str,
4321- help='Name of PPA to update in the git tree')
4322- parser.add_argument('package', type=str,
4323- help='Which package to update in the git tree')
4324- parser.add_argument('-o', '--lp-owner', type=str,
4325- help='Which launchpad user\'s tree to update',
4326- default='usd-import-team')
4327- parser.add_argument('-l', '--lp-user', type=str,
4328- help='Specify the Launchpad user to use',
4329- default=getpass.getuser())
4330- parser.add_argument('--dl-cache', type=str,
4331- help=('Cache directory for downloads. Default is to '
4332- 'put downloads in <directory>/.git/%s' %
4333- CACHE_PATH
4334- ),
4335- default=argparse.SUPPRESS)
4336- parser.add_argument('--no-fetch', action='store_true',
4337- help='Do not fetch from the remote (DANGEROUS: '
4338- 'only useful for debugging); implies '
4339- '--no-push')
4340- parser.add_argument('--no-push', action='store_true',
4341- help='Do not push to the remote (WARNING: '
4342- 'PPA imports are a work-in-progress and '
4343- 'pushing is currently disabled)',
4344- default=True)
4345- parser.add_argument('--no-clean', action='store_true',
4346- help='Do not clean the temporary directory')
4347- parser.add_argument('-d', '--directory', type=str,
4348- help='Use git repository at specified location rather '
4349- 'than a temporary directory (implies --no-clean). '
4350- 'Will create the local repository if needed.',
4351- default=argparse.SUPPRESS
4352- )
4353- if not subparsers:
4354- return parser.parse_args()
4355- return 'import-ppa - %s' % kwargs['description']
4356-
4357- def main(self, args):
4358- if re.match(r'ppa:\w+', args.ppa) is None:
4359- logging.error('Specified PPA (%s) is not in the format '
4360- 'ppa:<name>', args.ppa)
4361- sys.exit(1)
4362- ppa = args.ppa
4363- self.pkgname = args.package
4364- owner = args.lp_owner
4365- user = args.lp_user
4366- no_clean = args.no_clean
4367- try:
4368- directory = args.directory
4369- no_clean = True
4370- except AttributeError:
4371- directory = None
4372- no_push = True
4373- try:
4374- dl_cache = args.dl_cache
4375- except AttributeError:
4376- dl_cache = None
4377-
4378- # what other transformations?
4379- self.namespace = ppa.replace(':', '_')
4380-
4381- logging.info('Ubuntu Server Team importer v%s' % VERSION)
4382-
4383- self.local_repo = GitUbuntuRepository(directory, user, args.proto)
4384- if not directory:
4385- logging.info('Using git repository at %s', self.local_repo.local_dir)
4386- self.local_repo.ensure_importer_branches_exist(self.namespace)
4387-
4388- atexit.register(self.cleanup, no_clean, self.local_repo.local_dir)
4389-
4390- self.local_repo.add_remote(self.pkgname, owner, self.namespace)
4391- if not args.no_fetch:
4392+def main(
4393+ ppa,
4394+ pkgname,
4395+ directory,
4396+ owner,
4397+ user,
4398+ pullfile,
4399+ parentfile,
4400+ no_clean,
4401+ dl_cache,
4402+ proto,
4403+ retries,
4404+ retry_backoffs,
4405+ skip_orig,
4406+):
4407+ if re.match(r'ppa:\w+', ppa) is None:
4408+ logging.error(
4409+ "Specified PPA (%s) is not in the format ppa:<name>",
4410+ ppa,
4411+ )
4412+ sys.exit(1)
4413+ # what other transformations?
4414+ namespace = ppa.replace(':', '_')
4415+
4416+ logging.info('Ubuntu Server Team importer v%s' % VERSION)
4417+
4418+ repo = GitUbuntuRepository(directory, user, proto)
4419+ if not directory:
4420+ logging.info('Using git repository at %s', repo.local_dir)
4421+ repo.ensure_importer_branches_exist(namespace)
4422+
4423+ atexit.register(cleanup, no_clean, repo.local_dir)
4424+
4425+ source_information = GitUbuntuSourceInformation(
4426+ ppa,
4427+ pkgname,
4428+ os.path.abspath(pullfile),
4429+ retries,
4430+ retry_backoffs,
4431+ )
4432+
4433+ ubuntu_head_versions = repo.get_heads_and_versions(
4434+ 'ubuntu',
4435+ namespace,
4436+ )
4437+ for head_name in sorted(ubuntu_head_versions):
4438+ _, _, pretty_head_name = head_name.partition('%s/', namespace)
4439+ logging.debug('Last imported %s version is %s',
4440+ pretty_head_name,
4441+ ubuntu_head_versions[head_name]['version']
4442+ )
4443+
4444+ applied_ubuntu_head_versions = repo.get_heads_and_versions(
4445+ 'applied/ubuntu',
4446+ namespace,
4447+ )
4448+ for head_name in sorted(applied_ubuntu_head_versions):
4449+ _, _, pretty_head_name = head_name.partition('%s/', namespace)
4450+ logging.debug('Last applied %s version is %s',
4451+ pretty_head_name,
4452+ applied_ubuntu_head_versions[head_name]['version']
4453+ )
4454+
4455+ oldcwd = os.getcwd()
4456+ os.chdir(repo.local_dir)
4457+
4458+ if dl_cache is None:
4459+ workdir = os.path.join(repo.git_dir, CACHE_PATH)
4460+ else:
4461+ workdir = dl_cache
4462+
4463+ os.makedirs(workdir, exist_ok=True)
4464+
4465+ parse_parentfile(pkgname, parentfile)
4466+
4467+ history_found = False
4468+ try:
4469+ for srcpkg_information in \
4470+ source_information.launchpad_versions_published_after(
4471+ ubuntu_head_versions,
4472+ namespace,
4473+ workdir=workdir
4474+ ):
4475 try:
4476- self.local_repo.fetch_remote(self.namespace)
4477- except GitUbuntuRepositoryFetchError:
4478+ import_unapplied_spi(
4479+ repo=repo,
4480+ spi=srcpkg_information,
4481+ namespace=namespace,
4482+ ubuntu_sinfo=source_information,
4483+ skip_orig=skip_orig,
4484+ )
4485+ except DownloadError:
4486+ # it is non-fatal for a PPA to not have files for older
4487+ # publishes
4488 pass
4489+ history_found = True
4490+ except NoPublicationHistoryException:
4491+ logging.warning("No publication history found for %s in %s.",
4492+ pkgname,
4493+ ppa,
4494+ )
4495
4496- source_information = GitUbuntuSourceInformation(ppa, self.pkgname,
4497- os.path.abspath(args.pullfile),
4498- args.retries, args.retry_backoffs)
4499-
4500- ubuntu_head_versions = self.local_repo.get_heads_and_versions(
4501- 'ubuntu', self.namespace
4502+ if not history_found:
4503+ logging.error(
4504+ "No publication history for '%s' in %s. Wrong source package name?",
4505+ pkgname,
4506+ ppa,
4507 )
4508- for head_name in sorted(ubuntu_head_versions):
4509- _, _, pretty_head_name = head_name.partition('%s/', self.namespace)
4510- logging.debug('Last imported %s version is %s',
4511- pretty_head_name,
4512- ubuntu_head_versions[head_name]['version']
4513- )
4514-
4515- applied_ubuntu_head_versions = self.local_repo.get_heads_and_versions(
4516- 'applied/ubuntu', self.namespace
4517+ sys.exit(1)
4518+
4519+ try:
4520+ for srcpkg_information in \
4521+ source_information.launchpad_versions_published_after(
4522+ applied_ubuntu_head_versions,
4523+ '%s/applied' % namespace,
4524+ workdir=workdir
4525+ ):
4526+ try:
4527+ import_applied_spi(
4528+ repo=repo,
4529+ spi=srcpkg_information,
4530+ namespace=namespace,
4531+ ubuntu_sinfo=source_information,
4532+ )
4533+ except DownloadError:
4534+ # it is non-fatal for a PPA to not have files for older
4535+ # publishes
4536+ pass
4537+ except NoPublicationHistoryException:
4538+ logging.warning("No publication history found for %s in %s.",
4539+ pkgname, ppa
4540 )
4541- for head_name in sorted(applied_ubuntu_head_versions):
4542- _, _, pretty_head_name = head_name.partition('%s/', self.namespace)
4543- logging.debug('Last applied %s version is %s',
4544- pretty_head_name,
4545- applied_ubuntu_head_versions[head_name]['version']
4546- )
4547-
4548- oldcwd = os.getcwd()
4549- os.chdir(self.local_repo.local_dir)
4550-
4551- if dl_cache is None:
4552- workdir = os.path.join(self.local_repo.git_dir, CACHE_PATH)
4553- else:
4554- workdir = dl_cache
4555-
4556- os.makedirs(workdir, exist_ok=True)
4557-
4558- self.parse_parentfile(args.parentfile)
4559-
4560- history_found = False
4561- try:
4562- for srcpkg_information in \
4563- source_information.launchpad_versions_published_after(ubuntu_head_versions,
4564- self.namespace,
4565- workdir=workdir):
4566- try:
4567- self.import_unapplied_spi(srcpkg_information)
4568- except DownloadError:
4569- # it is non-fatal for a PPA to not have files for older
4570- # publishes
4571- pass
4572- history_found = True
4573- except NoPublicationHistoryException:
4574- logging.warning("No publication history found for %s in %s. ",
4575- self.pkgname, ppa)
4576-
4577- if not history_found:
4578- logging.error("No publication history for '%s' in %s. "
4579- "Wrong source package name?", self.pkgname, ppa)
4580- sys.exit(1)
4581-
4582- try:
4583- for srcpkg_information in \
4584- source_information.launchpad_versions_published_after(applied_ubuntu_head_versions,
4585- '%s/applied' % self.namespace,
4586- workdir=workdir):
4587- try:
4588- self.import_applied_spi(srcpkg_information)
4589- except DownloadError:
4590- # it is non-fatal for a PPA to not have files for older
4591- # publishes
4592- pass
4593- except NoPublicationHistoryException:
4594- logging.warning("No publication history found for %s in %s. ",
4595- self.pkgname, ppa)
4596- except:
4597- logging.error("Unable to import patches-applied to %s", ppa)
4598-
4599- os.chdir(oldcwd)
4600-
4601- self.local_repo.garbage_collect()
4602-
4603- if no_push:
4604- logging.info('Not pushing to remote as specified')
4605- else:
4606- # what are appropriate refspects to push and where?
4607- pass
4608+ except:
4609+ logging.error("Unable to import patches-applied to %s", ppa)
4610+
4611+ os.chdir(oldcwd)
4612+
4613+ repo.garbage_collect()
4614+
4615+
4616+def parse_args(subparsers=None, base_subparsers=None):
4617+ kwargs = dict(description='Update a launchpad git tree based upon '
4618+ 'the state of a PPA archive',
4619+ epilog='PPA must be specified as ppa:<name>',
4620+ formatter_class=argparse.ArgumentDefaultsHelpFormatter
4621+ )
4622+ if base_subparsers:
4623+ kwargs['parents'] = base_subparsers
4624+ if subparsers:
4625+ parser = subparsers.add_parser('import-ppa', **kwargs)
4626+ parser.set_defaults(func=cli_main)
4627+ else:
4628+ parser = argparse.ArgumentParser(**kwargs)
4629+ parser.add_argument('ppa', type=str,
4630+ help='Name of PPA to update in the git tree')
4631+ parser.add_argument('package', type=str,
4632+ help='Which package to update in the git tree')
4633+ parser.add_argument('-o', '--lp-owner', type=str,
4634+ help='Which launchpad user\'s tree to update',
4635+ default='usd-import-team')
4636+ parser.add_argument('-l', '--lp-user', type=str,
4637+ help='Specify the Launchpad user to use',
4638+ default=getpass.getuser())
4639+ parser.add_argument('--dl-cache', type=str,
4640+ help=('Cache directory for downloads. Default is to '
4641+ 'put downloads in <directory>/.git/%s' %
4642+ CACHE_PATH
4643+ ),
4644+ default=argparse.SUPPRESS)
4645+ parser.add_argument('--no-clean', action='store_true',
4646+ help='Do not clean the temporary directory')
4647+ parser.add_argument('-d', '--directory', type=str,
4648+ help='Use git repository at specified location rather '
4649+ 'than a temporary directory (implies --no-clean). '
4650+ 'Will create the local repository if needed.',
4651+ default=argparse.SUPPRESS
4652+ )
4653+ parser.add_argument(
4654+ '--skip-orig',
4655+ action='store_true',
4656+ help=argparse.SUPPRESS,
4657+ )
4658+ if not subparsers:
4659+ return parser.parse_args()
4660+ return 'import-ppa - %s' % kwargs['description']
4661+
4662+def cli_main(args):
4663+ no_clean = args.no_clean
4664+ try:
4665+ directory = args.directory
4666+ no_clean = True
4667+ except AttributeError:
4668+ directory = None
4669+ try:
4670+ dl_cache = args.dl_cache
4671+ except AttributeError:
4672+ dl_cache = None
4673+
4674+ main(
4675+ args.ppa,
4676+ args.package,
4677+ directory,
4678+ args.lp_owner,
4679+ args.lp_user,
4680+ args.pullfile,
4681+ args.parentfile,
4682+ no_clean,
4683+ dl_cache,
4684+ args.proto,
4685+ args.retries,
4686+ args.retry_backoffs,
4687+ args.skip_orig,
4688+ )
4689diff --git a/gitubuntu/lint.py b/gitubuntu/lint.py
4690index a67e0f3..ac503b9 100644
4691--- a/gitubuntu/lint.py
4692+++ b/gitubuntu/lint.py
4693@@ -23,10 +23,6 @@ except ImportError:
4694 logging.error("Is %s installed?", pkg)
4695 sys.exit(1)
4696
4697-__all__ = [
4698- 'GitUbuntuLint',
4699-]
4700-
4701 class LintNamespaceError(Exception):
4702 pass
4703
4704@@ -162,594 +158,636 @@ def test__derive_target_branch_string(same_remote_branch_names,
4705 )
4706 assert target_branch_string == expected
4707
4708-class GitUbuntuLint:
4709- def __init__(self):
4710- pass
4711
4712- def success(self, msg, *args):
4713- if self.verbose:
4714- print(msg % args)
4715+_verbose = False
4716+def success(msg, *args):
4717+ if _verbose:
4718+ print(msg % args)
4719
4720- def parse_args(self, subparsers=None, base_subparsers=None):
4721- kwargs = dict(
4722- description="Given a commitish, perform automated checks",
4723- formatter_class=argparse.RawTextHelpFormatter,
4724- epilog="An exit code of 0 indicates all checks passed",
4725- )
4726- if base_subparsers:
4727- kwargs["parents"] = base_subparsers
4728- if subparsers:
4729- parser = subparsers.add_parser("lint", **kwargs)
4730- parser.set_defaults(func=self.main)
4731- else:
4732- parser = argparse.ArgumentParser(**kwargs)
4733
4734- parser.add_argument("commitish", type=str, default="HEAD", nargs="?",
4735- help="Commitish to lint"
4736- )
4737- parser.add_argument("--lint-namespace", type=str,
4738- help="The git-ref prefix under which the branches and tags "
4739- "to lint can be found. If linting a remote-tracking "
4740- "branch, this will be default to the remote name. If "
4741- "linting a local branch, this will default to ''."
4742- )
4743- parser.add_argument("--target-branch", type=str,
4744- help="Target branch (typically a remote-tracking branch in pkg/) "
4745- "the current branch is against. This will be derived if "
4746- "possible."
4747- )
4748- parser.add_argument("-d", "--directory", type=str,
4749- help="Use git repository at specified location.",
4750- default=os.path.abspath(os.getcwd())
4751- )
4752- if not subparsers:
4753- return parser.parse_args()
4754- return "lint - %s" % kwargs["description"]
4755+def parse_args(subparsers=None, base_subparsers=None):
4756+ kwargs = dict(
4757+ description="Given a commitish, perform automated checks",
4758+ formatter_class=argparse.RawTextHelpFormatter,
4759+ epilog="An exit code of 0 indicates all checks passed",
4760+ )
4761+ if base_subparsers:
4762+ kwargs["parents"] = base_subparsers
4763+ if subparsers:
4764+ parser = subparsers.add_parser("lint", **kwargs)
4765+ parser.set_defaults(func=cli_main)
4766+ else:
4767+ parser = argparse.ArgumentParser(**kwargs)
4768
4769- def _check_commitish_exists(self, commitish):
4770- try:
4771- actual_commitish_obj = self.local_repo.get_commitish(
4772- commitish
4773- )
4774- self.success("Verified %s exists", commitish)
4775- return True
4776- except KeyError:
4777- warning("%s not found", commitish)
4778- return False
4779+ parser.add_argument("commitish", type=str, default="HEAD", nargs="?",
4780+ help="Commitish to lint"
4781+ )
4782+ parser.add_argument("--lint-namespace", type=str,
4783+ help="The git-ref prefix under which the branches and tags "
4784+ "to lint can be found. If linting a remote-tracking "
4785+ "branch, this will be default to the remote name. If "
4786+ "linting a local branch, this will default to ''."
4787+ )
4788+ parser.add_argument("--target-branch", type=str,
4789+ help="Target branch (typically a remote-tracking branch in pkg/) "
4790+ "the current branch is against. This will be derived if "
4791+ "possible."
4792+ )
4793+ parser.add_argument("-d", "--directory", type=str,
4794+ help="Use git repository at specified location.",
4795+ default=os.path.abspath(os.getcwd())
4796+ )
4797+ if not subparsers:
4798+ return parser.parse_args()
4799+ return "lint - %s" % kwargs["description"]
4800+
4801+def cli_main(args):
4802+ repo = GitUbuntuRepository(args.directory)
4803+ do_lint(
4804+ repo,
4805+ args.commitish,
4806+ args.lint_namespace,
4807+ args.target_branch,
4808+ args.verbose,
4809+ )
4810
4811- def _check_commitish_exists_and_same(self, expected_commitish_pretty_name,
4812- actual_commitish, expected_commitish, check_commit_id=False
4813- ):
4814- """Check @actual_commitish exists ad is the same tree as
4815- @expected_commitish
4816-
4817- Arguments:
4818- expected_commitish_pretty_name - a pretty name to print to refer to the
4819- expected value
4820- actual_commitish - commitish to check for
4821- expected_commitish - commitish to check against
4822- check_commit_id - if True, check the commit hashes match as well
4823- """
4824- ret = True
4825- if not self._check_commitish_exists(actual_commitish):
4826- return False
4827+def _check_commitish_exists(repo, commitish):
4828+ try:
4829+ actual_commitish_obj = repo.get_commitish(commitish)
4830+ success("Verified %s exists", commitish)
4831+ return True
4832+ except KeyError:
4833+ warning("%s not found", commitish)
4834+ return False
4835
4836- try:
4837- expected_commitish_obj = self.local_repo.get_commitish(
4838- expected_commitish
4839- )
4840- except KeyError:
4841- fatal("Unable to find expected commit: %s", expected_commitish)
4842+def _check_commitish_exists_and_same(repo, expected_commitish_pretty_name,
4843+ actual_commitish, expected_commitish, check_commit_id=False
4844+):
4845+ """Check @actual_commitish exists ad is the same tree as
4846+ @expected_commitish
4847+
4848+ Arguments:
4849+ expected_commitish_pretty_name - a pretty name to print to refer to the
4850+ expected value
4851+ actual_commitish - commitish to check for
4852+ expected_commitish - commitish to check against
4853+ check_commit_id - if True, check the commit hashes match as well
4854+ """
4855+ ret = True
4856+ if not _check_commitish_exists(repo, actual_commitish):
4857+ return False
4858
4859- actual_commitish_obj = self.local_repo.get_commitish(
4860- actual_commitish
4861- )
4862- if check_commit_id:
4863- expected_commit_id = str(expected_commitish_obj.id)
4864- actual_commit_id = str(actual_commitish_obj.id)
4865- if expected_commit_id == actual_commit_id:
4866- self.success("Verified %s is the same commit as %s",
4867- expected_commitish_pretty_name, actual_commitish
4868- )
4869- else:
4870- warning("Expected %s (%s) is not the same commit as %s (%s)",
4871- expected_commitish_pretty_name, expected_commit_id,
4872- actual_commitish, actual_commit_id
4873- )
4874- ret = False
4875- expected_commit_tree_id = str(
4876- expected_commitish_obj.peel(pygit2.Tree).id
4877- )
4878- actual_commit_tree_id = str(
4879- actual_commitish_obj.peel(pygit2.Tree).id
4880- )
4881- if expected_commit_tree_id == actual_commit_tree_id:
4882- self.success("Verified %s is the same tree as %s",
4883+ try:
4884+ expected_commitish_obj = repo.get_commitish(expected_commitish)
4885+ except KeyError:
4886+ fatal("Unable to find expected commit: %s", expected_commitish)
4887+
4888+ actual_commitish_obj = repo.get_commitish(actual_commitish)
4889+ if check_commit_id:
4890+ expected_commit_id = str(expected_commitish_obj.id)
4891+ actual_commit_id = str(actual_commitish_obj.id)
4892+ if expected_commit_id == actual_commit_id:
4893+ success("Verified %s is the same commit as %s",
4894 expected_commitish_pretty_name, actual_commitish
4895 )
4896 else:
4897- error("Expected %s (%s) is not the same tree as %s (%s)",
4898- expected_commitish_pretty_name, expected_commit_tree_id,
4899- actual_commitish, actual_commit_tree_id
4900+ warning("Expected %s (%s) is not the same commit as %s (%s)",
4901+ expected_commitish_pretty_name, expected_commit_id,
4902+ actual_commitish, actual_commit_id
4903 )
4904 ret = False
4905+ expected_commit_tree_id = str(
4906+ expected_commitish_obj.peel(pygit2.Tree).id
4907+ )
4908+ actual_commit_tree_id = str(
4909+ actual_commitish_obj.peel(pygit2.Tree).id
4910+ )
4911+ if expected_commit_tree_id == actual_commit_tree_id:
4912+ success("Verified %s is the same tree as %s",
4913+ expected_commitish_pretty_name, actual_commitish
4914+ )
4915+ else:
4916+ error("Expected %s (%s) is not the same tree as %s (%s)",
4917+ expected_commitish_pretty_name, expected_commit_tree_id,
4918+ actual_commitish, actual_commit_tree_id
4919+ )
4920+ ret = False
4921
4922- return ret
4923+ return ret
4924
4925- def print_hunk(self, hunk):
4926- for line in hunk.lines:
4927- print("%s%s" % (line.origin, line.content.rstrip()))
4928+def print_hunk(hunk):
4929+ for line in hunk.lines:
4930+ print("%s%s" % (line.origin, line.content.rstrip()))
4931
4932- def print_patch(self, patch):
4933- for hunk in patch.hunks:
4934- self.print_hunk(hunk)
4935+def print_patch(patch):
4936+ for hunk in patch.hunks:
4937+ print_hunk(hunk)
4938
4939- def _check_deconstruct_to_logical_patch(self, patch):
4940- if "debian/changelog" in patch.delta.new_file.path:
4941- self.success("Verified debian/changelog is in diff between "
4942- "deconstruct and logical"
4943- )
4944- return True
4945- if "debian/control" in patch.delta.new_file.path:
4946- ret = True
4947- for hunk in patch.hunks:
4948- _ret = True
4949- for line in hunk.lines:
4950- if line.origin == " ":
4951+def _check_deconstruct_to_logical_patch(patch):
4952+ if "debian/changelog" in patch.delta.new_file.path:
4953+ success("Verified debian/changelog is in diff between "
4954+ "deconstruct and logical"
4955+ )
4956+ return True
4957+ if "debian/control" in patch.delta.new_file.path:
4958+ ret = True
4959+ for hunk in patch.hunks:
4960+ _ret = True
4961+ for line in hunk.lines:
4962+ if line.origin == " ":
4963+ continue
4964+ content = line.content.strip()
4965+ if line.origin == "-":
4966+ if content.startswith("Maintainer:"):
4967+ continue
4968+ if line.origin == "+":
4969+ if (
4970+ content.startswith("Maintainer:") and
4971+ content.endswith("<ubuntu-devel-discuss@lists.ubuntu.com>")
4972+ ):
4973+ continue
4974+ if content.startswith("XSBC-Original-Maintainer:"):
4975 continue
4976- content = line.content.strip()
4977- if line.origin == "-":
4978- if content.startswith("Maintainer:"):
4979- continue
4980- if line.origin == "+":
4981- if (
4982- content.startswith("Maintainer:") and
4983- content.endswith("<ubuntu-devel-discuss@lists.ubuntu.com>")
4984- ):
4985- continue
4986- if content.startswith("XSBC-Original-Maintainer:"):
4987- continue
4988- _ret = False
4989- if not _ret:
4990- error("unexpected differences between logical "
4991- "and deconstruct tags"
4992- )
4993- self.print_hunk(hunk)
4994- ret = _ret and ret
4995- if ret:
4996- self.success("Verified only update-maintainer changes "
4997- "are in diff between deconstruct and logical to "
4998- "debian/control"
4999+ _ret = False
5000+ if not _ret:
The diff has been truncated for viewing.

Subscribers

People subscribed via source and target branches