Merge ~nacc/git-ubuntu:refactor-main-v2 into git-ubuntu:master
- Git
- lp:~nacc/git-ubuntu
- refactor-main-v2
- Merge into master
Status: | Merged | ||||
---|---|---|---|---|---|
Approved by: | Nish Aravamudan | ||||
Approved revision: | 6f347d12bf6d133b05e5729f2fa09d5132662d99 | ||||
Merged at revision: | b57968371be53c39948ea6d5c5e46ac5d9738930 | ||||
Proposed branch: | ~nacc/git-ubuntu:refactor-main-v2 | ||||
Merge into: | git-ubuntu:master | ||||
Prerequisite: | ~nacc/git-ubuntu:bugfixes-lint-clean | ||||
Diff against target: |
8236 lines (+4156/-3550) 13 files modified
gitubuntu/__main__.py (+139/-99) gitubuntu/build.py (+127/-117) gitubuntu/buildsource.py (+55/-59) gitubuntu/clone.py (+136/-105) gitubuntu/importer.py (+1389/-1178) gitubuntu/importlocal.py (+312/-260) gitubuntu/importppa.py (+227/-175) gitubuntu/lint.py (+570/-532) gitubuntu/merge.py (+423/-355) gitubuntu/queue.py (+299/-281) gitubuntu/remote.py (+158/-100) gitubuntu/submit.py (+189/-176) gitubuntu/tag.py (+132/-113) |
||||
Related bugs: |
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Server Team CI bot | continuous-integration | Approve | |
Andreas Hasenack | Approve | ||
Robie Basak | Pending | ||
Review via email:
|
This proposal supersedes a proposal from 2017-09-08.
Commit message
Description of the change
This is admittedly a messy patchset. I am tired of rebasing it, though, and think we can land it in roughly this state (even if only in the edge channel of the snap) and then decide what to do from there after testing.
A ton of change, but we are now pylint3 clean for errors except for pygit2 imports:
$ pylint3 -E gitubuntu --ignored-
No config file found, using default configuration
All unit tests still pass and I've run a from-scratch import, a clone and a build using the resulting git repository.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Server Team CI bot (server-team-bot) wrote : Posted in a previous version of this proposal | # |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Server Team CI bot (server-team-bot) wrote : Posted in a previous version of this proposal | # |
FAILED: Continuous integration, rev:a54e9ee3194
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Tests
FAILED: Build
Click here to trigger a rebuild:
https:/
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Nish Aravamudan (nacc) wrote : Posted in a previous version of this proposal | # |
The overall structure of this is good, I'm just still fine-tuning.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andreas Hasenack (ahasenack) wrote : Posted in a previous version of this proposal | # |
I see help_text being built several times in gitubuntu/
help_text = '\n'.join( ...
help_text += '\n'
help_text += '<pkgname> ...
help_text += ...
'-P', '--parentfile',
),
)
help_text = '\n'.join(... <--- starts over
Maybe you meant to use it in parser.
'-P', '--parentfile',
),
)
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andreas Hasenack (ahasenack) wrote : Posted in a previous version of this proposal | # |
Ok, first pass on gitubuntu/
These are notes to myself regarding the changes that I reviewed and saw:
- switch to use import_module
- code reformatting (probably as a result of lint)
- switch from calling the module's class instance method to calling the module's parse_args() method, which also sets args.func to be m.cli_main() via parser.
- cleaner setting of debug logging
The whole buildup of the parser is still a bit confusing to me, but I can see that in this branch it was must moved around for better formatting and not really changed. Some of this could probably me moved to methods/functions at some point to make it easier to read. Not something for this branch at this time.
The review continues! :)
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Nish Aravamudan (nacc) wrote : Posted in a previous version of this proposal | # |
On Fri, Sep 8, 2017 at 1:00 PM, Andreas Hasenack <email address hidden> wrote:
> Ok, first pass on gitubuntu/
>
> These are notes to myself regarding the changes that I reviewed and saw:
> - switch to use import_module
> - code reformatting (probably as a result of lint)
One of our goals for 1.0 is to cleanup style to follow doc/STYLE.md as
we touch code. That's why there are some stylistic changes
intereleaved.
> - switch from calling the module's class instance method to calling the module's parse_args() method, which also sets args.func to be m.cli_main() via parser.
We always called instance parse_args then the implicit func via the default.
Now, we set call a module's parse_args and then the implicit func via
the default.
> - cleaner setting of debug logging
>
> The whole buildup of the parser is still a bit confusing to me, but I can see that in this branch it was must moved around for better formatting and not really changed. Some of this could probably me moved to methods/functions at some point to make it easier to read. Not something for this branch at this time.
>
> The review continues! :)
>
>
> Diff comments:
>
>> diff --git a/gitubuntu/
>> index eb6078a..49373a7 100644
>> --- a/gitubuntu/
>> +++ b/gitubuntu/
>> @@ -188,26 +201,30 @@ def main():
>> )
>> except CalledProcessError:
>> if isatty(
>> - user = input("
>> + user = input(
>> + "gitubuntu.lpuser is not set. What is your "
>> "Launchpad username? We will set this in your "
>> "personally global git configuration for you. "
>> "To abort, press Enter.\nUsername: "
>> )
>> if len(user) == 0:
>> - logging.
>> + logging.
>> sys.exit(1)
>> else:
>> - logging.
>> - 'git-ubuntu to %s with '
>> - '`git config --global gitubuntu.lpuser %s`.',
>> - user, user)
>> + logging.warning(
>> + "Setting the Launchpad user for git-ubuntu to %s "
>> + "with `git config --global gitubuntu.lpuser %s`.",
>> + user,
>> + user,
>
> Both "user" can be in the same line, and no need for the trailing comma.
Yeah, I wasn't 100% with how we wanted to handle this specific case of
style. Quoting STYLE.md:
* If a method call goes to multiple lines, use one parameter per line, each
ending in a trailing comma including the last parameter, and the closing
bracket on a separate final line.
Technically, these are parameters to logging.warning, not
substitutions to the parameter string.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andreas Hasenack (ahasenack) wrote : Posted in a previous version of this proposal | # |
Notes for gitubuntu/build.py:
- not a class anymore
- class.parse_args -> module.parse_args:
- parser.set_defaults points at cli_main now
- class.check_
- no change
- class.derive_
- one change:
if not args.no_
+ repo=self.
- repo=GitUbuntuR
- local_repo was being set to repo=GitUbuntuR
- class.main -> split into module.cli_main which calls module.main
Other minor changes:
- derive_
- Use of GitUbuntuReposi
- fetch_orig_
- build: fix use of undefined variables
- fetch_orig_
- derive_
- git_repository: move derive_
+1 for gitubuntu/build.py. Next up is gitubuntu/
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Server Team CI bot (server-team-bot) wrote : Posted in a previous version of this proposal | # |
FAILED: Continuous integration, rev:6a15a640423
https:/
Executed test runs:
SUCCESS: Checkout
FAILED: Style Check
Click here to trigger a rebuild:
https:/
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Server Team CI bot (server-team-bot) wrote : Posted in a previous version of this proposal | # |
FAILED: Continuous integration, rev:6a15a640423
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Style Check
SUCCESS: Unit Tests
FAILED: Integration Tests
Click here to trigger a rebuild:
https:/
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Server Team CI bot (server-team-bot) wrote : Posted in a previous version of this proposal | # |
PASSED: Continuous integration, rev:5d0cb19e1fc
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Style Check
SUCCESS: Unit Tests
SUCCESS: Integration Tests
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
- 9cb168b... by Nish Aravamudan
-
lint: rework so that main is a more useful function
This will be folded up with the other changes before merging.
- 6f347d1... by Nish Aravamudan
-
remote: rework so that main is a more useful function
This will be folded up with the other changes before merging.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:9ba1036b393
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Style Check
SUCCESS: Unit Tests
SUCCESS: Integration Tests
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:960a19c3477
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Style Check
SUCCESS: Unit Tests
SUCCESS: Integration Tests
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andreas Hasenack (ahasenack) wrote : | # |
build_source.py: OK
gitubuntu/clone.py:
cli_main() calls main() and discards its return value. main() however returns "local_repo", is that correct? It's not documented in main's docstrings, and in the previous gitubuntu/clone.py module there was no such return value.
def main(package, directory=None, lp_user=None, proto=None):
"""Entry point to clone subcommand
@package: Name of source package
@directory: directory to clone the repository into
@lp_user: user to authenticate to Launchpad as
@proto: protocol to use (one of 'http', 'https', 'git')
If directory is None, a relative directory with the same name as
package will be used.
If lp_user is None, value of `git config gitubuntu.lpuser` will be
used.
If proto is None, the default value from gitubuntu.__main__ will be
used (top_level_
"""
(...)
if os.path.
'to ensure all changes are tracked or passing appropriate '
'flags to git commands (e.g., git status --ignored).'
)
return local_repo <-----
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andreas Hasenack (ahasenack) wrote : | # |
gitubuntu/
- docstrings overall. A new branch with just docstrings fixes should address these
- import_publishes(): commented code, should either be removed (new change) or uncommented (so it's like the previous import_publishes):
#if not patches_applied:
raise GitUbuntuImport
#else:
# logging.error(msg)
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andreas Hasenack (ahasenack) wrote : | # |
importlocal.py: ok
importppa.py
- note: parse_args() got a new option: --skip-orig
- main(): has commented code. Ideally it should be removed. It can always be brought back if needed.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andreas Hasenack (ahasenack) wrote : | # |
lint.py: OK, I just see a few extra checks in the diff
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andreas Hasenack (ahasenack) wrote : | # |
merge.py: ok
queue.py: ok
remote.py: ok
submit.py: ok
tag.py: ok
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Andreas Hasenack (ahasenack) wrote : | # |
Ah, I see that at least the new review command uses the return value of clone's main():
repo = gitubuntu.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Server Team CI bot (server-team-bot) wrote : | # |
PASSED: Continuous integration, rev:6f347d12bf6
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Style Check
SUCCESS: Unit Tests
SUCCESS: Integration Tests
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild:
https:/
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Nish Aravamudan (nacc) wrote : | # |
This looks good to land now. I'm going to try and get to some more commits tomorrow that, at least, add docstring for main() in each subcommand.
That should be a noop relative to the CI pass and code review, though.
Preview Diff
1 | diff --git a/gitubuntu/__main__.py b/gitubuntu/__main__.py |
2 | index 165fc59..edeee43 100644 |
3 | --- a/gitubuntu/__main__.py |
4 | +++ b/gitubuntu/__main__.py |
5 | @@ -1,26 +1,29 @@ |
6 | #!/usr/bin/env python3 |
7 | |
8 | +from collections import namedtuple |
9 | +TopLevelDefaults = namedtuple( |
10 | + 'TopLevelDefaults', |
11 | + [ |
12 | + 'verbose', |
13 | + 'retries', |
14 | + 'retry_backoffs', |
15 | + 'proto', |
16 | + 'parentfile', |
17 | + 'pullfile', |
18 | + ], |
19 | +) |
20 | +top_level_defaults = object() |
21 | + |
22 | def main(): |
23 | try: |
24 | import argparse |
25 | + from importlib import import_module |
26 | import logging |
27 | from os import isatty |
28 | import shutil |
29 | from subprocess import CalledProcessError |
30 | import sys |
31 | import textwrap |
32 | - from gitubuntu.importer import GitUbuntuImport |
33 | - from gitubuntu.build import GitUbuntuBuild |
34 | - from gitubuntu.buildsource import GitUbuntuBuildSource |
35 | - from gitubuntu.merge import GitUbuntuMerge |
36 | - from gitubuntu.clone import GitUbuntuClone |
37 | - from gitubuntu.tag import GitUbuntuTag |
38 | - from gitubuntu.queue import GitUbuntuQueue |
39 | - from gitubuntu.importlocal import GitUbuntuImportLocal |
40 | - from gitubuntu.importppa import GitUbuntuImportPPA |
41 | - from gitubuntu.remote import GitUbuntuRemote |
42 | - from gitubuntu.submit import GitUbuntuSubmit |
43 | - from gitubuntu.lint import GitUbuntuLint |
44 | from gitubuntu.run import run |
45 | |
46 | try: |
47 | @@ -32,95 +35,128 @@ def main(): |
48 | logging.error('Is %s installed?', pkg) |
49 | sys.exit(1) |
50 | |
51 | + global top_level_defaults |
52 | + top_level_defaults = TopLevelDefaults( |
53 | + verbose=False, |
54 | + retries=3, |
55 | + retry_backoffs = [2 ** i for i in range(3)], |
56 | + proto='https', |
57 | + parentfile=pkg_resources.resource_filename( |
58 | + 'gitubuntu', |
59 | + 'parent_overrides.txt', |
60 | + ), |
61 | + pullfile=pkg_resources.resource_filename( |
62 | + 'gitubuntu', |
63 | + 'pull_overrides.txt', |
64 | + ), |
65 | + ) |
66 | + |
67 | logging.getLogger('keyring').setLevel(logging.WARNING) |
68 | |
69 | - known_subcommands = {'import':GitUbuntuImport, |
70 | - 'build':GitUbuntuBuild, |
71 | - 'build-source':GitUbuntuBuildSource, |
72 | - 'merge':GitUbuntuMerge, |
73 | - 'clone':GitUbuntuClone, |
74 | - 'tag':GitUbuntuTag, |
75 | - 'queue':GitUbuntuQueue, |
76 | - 'import-local':GitUbuntuImportLocal, |
77 | - 'import-ppa':GitUbuntuImportPPA, |
78 | - 'remote':GitUbuntuRemote, |
79 | - 'submit':GitUbuntuSubmit, |
80 | - 'lint':GitUbuntuLint, |
81 | - } |
82 | + known_subcommands = { |
83 | + 'import': 'gitubuntu.importer', |
84 | + 'build': 'gitubuntu.build', |
85 | + 'build-source': 'gitubuntu.buildsource', |
86 | + 'merge': 'gitubuntu.merge', |
87 | + 'clone': 'gitubuntu.clone', |
88 | + 'tag': 'gitubuntu.tag', |
89 | + 'queue': 'gitubuntu.queue', |
90 | + 'import-local': 'gitubuntu.importlocal', |
91 | + 'import-ppa': 'gitubuntu.importppa', |
92 | + 'remote': 'gitubuntu.remote', |
93 | + 'submit': 'gitubuntu.submit', |
94 | + 'lint': 'gitubuntu.lint', |
95 | + } |
96 | |
97 | - known_network_subcommands = {'import', 'import-local', |
98 | - 'import-ppa', 'clone', 'build', |
99 | - 'build-source', 'queue', |
100 | - 'remote', 'submit'} |
101 | + known_network_subcommands = { |
102 | + 'import', |
103 | + 'import-local', |
104 | + 'import-ppa', |
105 | + 'clone', |
106 | + 'build', |
107 | + 'build-source', |
108 | + 'queue', |
109 | + 'remote', |
110 | + 'submit', |
111 | + } |
112 | |
113 | parser = argparse.ArgumentParser( |
114 | description='Ubuntu git development tool', |
115 | formatter_class=argparse.RawTextHelpFormatter, |
116 | - epilog='More information is available at https://wiki.ubuntu.com/UbuntuDevelopment/Merging/GitWorkflow.' |
117 | - ) |
118 | - subparsers = parser.add_subparsers(dest='subcommand', |
119 | - help='', |
120 | - metavar='%s' % '|'.join(sorted(known_subcommands)) |
121 | - ) |
122 | + epilog='More information is available at https://wiki.ubuntu.com/UbuntuDevelopment/Merging/GitWorkflow.', |
123 | + ) |
124 | + subparsers = parser.add_subparsers( |
125 | + dest='subcommand', |
126 | + help='', |
127 | + metavar='%s' % '|'.join(sorted(known_subcommands)), |
128 | + ) |
129 | # This help ends up not being very useful to the end user |
130 | # subparsers.required = True |
131 | |
132 | # common flags to all subcommands |
133 | base_subparser = argparse.ArgumentParser(add_help=False) |
134 | - base_subparser.add_argument('-v', '--verbose', action='store_true', |
135 | - help='Increase verbosity') |
136 | + base_subparser.add_argument( |
137 | + '-v', '--verbose', |
138 | + action='store_true', |
139 | + help="Increase verbosity", |
140 | + default=top_level_defaults.verbose, |
141 | + ) |
142 | network_base_subparser = argparse.ArgumentParser(add_help=False) |
143 | network_base_subparser.add_argument( |
144 | - '--retries', type=int, |
145 | - help='Number of times to attempt to retry ' |
146 | - 'downloading from Launchpad.', |
147 | - default=3 |
148 | - ) |
149 | + '--retries', |
150 | + type=int, |
151 | + help="Number of times to attempt to retry downloading from " |
152 | + "Launchpad.", |
153 | + default=top_level_defaults.retries, |
154 | + ) |
155 | network_base_subparser.add_argument( |
156 | - '--retry-backoffs', |
157 | - type=lambda s : [int(item) for item in s.split(',')], |
158 | - help='Comma-separated list of backoffs in ' |
159 | - 'seconds to use between each retry ' |
160 | - 'attempt. Default is exponential backoff.', |
161 | - default=argparse.SUPPRESS |
162 | - ) |
163 | + '--retry-backoffs', |
164 | + type=lambda s : [int(item) for item in s.split(',')], |
165 | + help="Comma-separated list of backoffs in seconds to use " |
166 | + "between each retry attempt. Default is exponential " |
167 | + "backoff.", |
168 | + default=argparse.SUPPRESS, |
169 | + ) |
170 | known_protos = ['git', 'http', 'https'] |
171 | network_base_subparser.add_argument( |
172 | - '--proto', |
173 | - metavar='[%s]' % '|'.join(known_protos), |
174 | - choices=known_protos, |
175 | - help='Specify protocol to use for fetch. Default: %(default)s', |
176 | - default='https' |
177 | - ) |
178 | + '--proto', |
179 | + metavar='[%s]' % '|'.join(known_protos), |
180 | + choices=known_protos, |
181 | + help="Specify protocol to use for fetch. Default: %(default)s", |
182 | + default=top_level_defaults.proto, |
183 | + ) |
184 | |
185 | width, _ = shutil.get_terminal_size() |
186 | subhelp_width = width - 30 |
187 | for sub in sorted(known_subcommands): |
188 | - s = known_subcommands[sub]() |
189 | + m = import_module(known_subcommands[sub]) |
190 | if sub in known_network_subcommands: |
191 | - help_text = s.parse_args(subparsers, [base_subparser, network_base_subparser]) |
192 | + help_text = m.parse_args( |
193 | + subparsers, |
194 | + [base_subparser, network_base_subparser], |
195 | + ) |
196 | else: |
197 | - help_text = s.parse_args(subparsers, [base_subparser]) |
198 | + help_text = m.parse_args(subparsers, [base_subparser]) |
199 | subparsers.help += '\n' |
200 | - subparsers.help += '\n'.join(textwrap.wrap(help_text, |
201 | - subhelp_width, |
202 | - break_on_hyphens=False |
203 | - ) |
204 | - ) |
205 | - parser.add_argument('-P', '--parentfile', type=str, |
206 | - help=argparse.SUPPRESS, |
207 | - default=pkg_resources.resource_filename( |
208 | - 'gitubuntu', |
209 | - 'parent_overrides.txt' |
210 | - ) |
211 | - ) |
212 | - parser.add_argument('-L', '--pullfile', type=str, |
213 | - help=argparse.SUPPRESS, |
214 | - default=pkg_resources.resource_filename( |
215 | - 'gitubuntu', |
216 | - 'pull_overrides.txt' |
217 | - ) |
218 | - ) |
219 | + subparsers.help += '\n'.join( |
220 | + textwrap.wrap( |
221 | + help_text, |
222 | + subhelp_width, |
223 | + break_on_hyphens=False, |
224 | + ) |
225 | + ) |
226 | + parser.add_argument( |
227 | + '-P', '--parentfile', |
228 | + type=str, |
229 | + help=argparse.SUPPRESS, |
230 | + default=top_level_defaults.parentfile, |
231 | + ) |
232 | + parser.add_argument( |
233 | + '-L', '--pullfile', |
234 | + type=str, |
235 | + help=argparse.SUPPRESS, |
236 | + default=top_level_defaults.pullfile, |
237 | + ) |
238 | |
239 | argcomplete.autocomplete(parser) |
240 | args = parser.parse_args() |
241 | @@ -131,22 +167,22 @@ def main(): |
242 | # fragile, assumes developers will pass the base_subparser above |
243 | # should make this structural in the classes |
244 | if args.verbose: |
245 | - logging.basicConfig(level=logging.DEBUG, |
246 | - format='%(asctime)s - %(levelname)s:%(message)s', |
247 | - datefmt='%m/%d/%Y %H:%M:%S', |
248 | - ) |
249 | + level=logging.DEBUG |
250 | else: |
251 | - logging.basicConfig(level=logging.INFO, |
252 | - format='%(asctime)s - %(levelname)s:%(message)s', |
253 | - datefmt='%m/%d/%Y %H:%M:%S', |
254 | - ) |
255 | + level=logging.INFO |
256 | + logging.basicConfig( |
257 | + level=level, |
258 | + format='%(asctime)s - %(levelname)s:%(message)s', |
259 | + datefmt='%m/%d/%Y %H:%M:%S', |
260 | + ) |
261 | if args.subcommand in known_network_subcommands: |
262 | try: |
263 | retry_backoffs = args.retry_backoffs |
264 | if len(retry_backoffs) != args.retries: |
265 | - logging.error('Number of backoffs specified in ' |
266 | - '--retry-backoffs must match --retries.' |
267 | - ) |
268 | + logging.error( |
269 | + "Number of backoffs specified in " |
270 | + "--retry-backoffs must match --retries." |
271 | + ) |
272 | sys.exit(1) |
273 | except AttributeError: |
274 | args.retry_backoffs = [2 ** i for i in range(args.retries)] |
275 | @@ -158,26 +194,30 @@ def main(): |
276 | ) |
277 | except CalledProcessError: |
278 | if isatty(sys.stdin.fileno()): |
279 | - user = input("gitubuntu.lpuser is not set. What is your " |
280 | + user = input( |
281 | + "gitubuntu.lpuser is not set. What is your " |
282 | "Launchpad username? We will set this in your " |
283 | "personally global git configuration for you. " |
284 | "To abort, press Enter.\nUsername: " |
285 | ) |
286 | if len(user) == 0: |
287 | - logging.error('git-ubuntu requires a launchpad user') |
288 | + logging.error("git-ubuntu requires a launchpad user") |
289 | sys.exit(1) |
290 | else: |
291 | - logging.warning('Setting the Launchpad user for ' |
292 | - 'git-ubuntu to %s with ' |
293 | - '`git config --global gitubuntu.lpuser %s`.', |
294 | - user, user) |
295 | + logging.warning( |
296 | + "Setting the Launchpad user for git-ubuntu to %s " |
297 | + "with `git config --global gitubuntu.lpuser %s`.", |
298 | + user, |
299 | + user, |
300 | + ) |
301 | run(['git', 'config', '--global', 'gitubuntu.lpuser', user]) |
302 | else: |
303 | - logging.error('Not connected to a TTY and no ' |
304 | - 'git-ubuntu lpuser defined. Please run ' |
305 | - '`git config --global gitubuntu.lpuser ' |
306 | - '<Launchpad username>` and re-run this ' |
307 | - 'command.') |
308 | + logging.error( |
309 | + "Not connected to a TTY and no git-ubuntu lpuser " |
310 | + "defined. Please run `git config --global " |
311 | + "gitubuntu.lpuser <Launchpad username>` and re-run " |
312 | + "this command." |
313 | + ) |
314 | sys.exit(1) |
315 | |
316 | args.func(args) |
317 | diff --git a/gitubuntu/build.py b/gitubuntu/build.py |
318 | index a8f9cf3..d3d292e 100644 |
319 | --- a/gitubuntu/build.py |
320 | +++ b/gitubuntu/build.py |
321 | @@ -372,142 +372,152 @@ def fetch_orig_from_launchpad(changelog, source, pullfile, retries, |
322 | return None |
323 | |
324 | |
325 | -class GitUbuntuBuild: |
326 | - def parse_args(self, subparsers=None, base_subparsers=None): |
327 | - kwargs = dict( |
328 | - description='Build a git-ubuntu-cloned tree with dpkg-buildpackage', |
329 | - formatter_class=argparse.ArgumentDefaultsHelpFormatter, |
330 | - usage='%(prog)s [options] -- ...', |
331 | - epilog='Any arguments after -- are passed to dpkg-buildpackage.' |
332 | - ) |
333 | - if base_subparsers: |
334 | - kwargs['parents'] = base_subparsers |
335 | - if subparsers: |
336 | - parser = subparsers.add_parser('build', **kwargs) |
337 | - parser.set_defaults(func=self.main) |
338 | - else: |
339 | - parser = argparse.ArgumentParser(**kwargs) |
340 | - parser.add_argument('--dl-cache', type=str, |
341 | - help='Cache directory for downloads.', |
342 | - default=os.path.join(os.getcwd(), '.git', CACHE_PATH), |
343 | - ) |
344 | - parser.add_argument( |
345 | - '--for-merge', |
346 | - action='store_true', |
347 | - help="Build an Ubuntu merge. Implies searching Debian for " |
348 | - "published versions first.", |
349 | +def check_repository(): |
350 | + try: |
351 | + cp = run(['git', 'status', '--porcelain']) |
352 | + except CalledProcessError: |
353 | + logging.error('Is the current directory a git repository?') |
354 | + sys.exit(1) |
355 | + |
356 | + if len(cp.stdout) > 0: |
357 | + logging.warning( |
358 | + "Working tree is not clean. `git status` output follows." |
359 | ) |
360 | - parser.add_argument( |
361 | - '--no-pristine-tar', |
362 | - action='store_true', |
363 | - help="Do not use pristine-tar branches to generate orig " |
364 | - "tarballs. This can be necessary as we stabilize the pristine-tar " |
365 | - "data generation, particularly if multiple component tarballs are " |
366 | - "present.", |
367 | + run(['git', 'status'], stdout=None) |
368 | + sys.exit(1) |
369 | |
370 | + if not os.path.isfile('debian/changelog'): |
371 | + logging.error( |
372 | + "No debian/changelog found, is this a source " |
373 | + "package repository? See `git ubuntu clone`." |
374 | ) |
375 | - parser.add_argument('rem_args', help=argparse.SUPPRESS, |
376 | - nargs=argparse.REMAINDER) |
377 | - if not subparsers: |
378 | - return parser.parse_args() |
379 | - return 'build - %s' % kwargs['description'] |
380 | + sys.exit(1) |
381 | |
382 | - def check_repository(self): |
383 | - try: |
384 | - cp = run(['git', 'status', '--porcelain']) |
385 | - except CalledProcessError: |
386 | - logging.error('Is the current directory a git repository?') |
387 | - sys.exit(1) |
388 | |
389 | - if len(cp.stdout) > 0: |
390 | - logging.warning( |
391 | - "Working tree is not clean. `git status` output follows." |
392 | - ) |
393 | - run(['git', 'status'], stdout=None) |
394 | - sys.exit(1) |
395 | +def main(search_list, changelog, rem_args): |
396 | + """main entry point for build |
397 | |
398 | - if not os.path.isfile('debian/changelog'): |
399 | - logging.error( |
400 | - "No debian/changelog found, is this a source " |
401 | - "package repository? See `git ubuntu clone`." |
402 | - ) |
403 | - sys.exit(1) |
404 | + @param search_list: list of OrigSearchListEntry namedtuples |
405 | + @param changelog: gitubuntu.git_repository.Changelog object for source package to build |
406 | + @param rem_args: namespace object of remaining arguments to pass onto dpkg-buildpackage |
407 | + """ |
408 | + check_repository() |
409 | + |
410 | + fetch_orig_and_build(search_list, changelog, rem_args) |
411 | + |
412 | + |
413 | +def parse_args(subparsers=None, base_subparsers=None): |
414 | + kwargs = dict( |
415 | + description='Build a git-ubuntu-cloned tree with dpkg-buildpackage', |
416 | + formatter_class=argparse.ArgumentDefaultsHelpFormatter, |
417 | + usage='%(prog)s [options] -- ...', |
418 | + epilog='Any arguments after -- are passed to dpkg-buildpackage.' |
419 | + ) |
420 | + if base_subparsers: |
421 | + kwargs['parents'] = base_subparsers |
422 | + if subparsers: |
423 | + parser = subparsers.add_parser('build', **kwargs) |
424 | + parser.set_defaults(func=cli_main) |
425 | + else: |
426 | + parser = argparse.ArgumentParser(**kwargs) |
427 | + parser.add_argument('--dl-cache', type=str, |
428 | + help='Cache directory for downloads.', |
429 | + default=os.path.join(os.getcwd(), '.git', CACHE_PATH), |
430 | + ) |
431 | + parser.add_argument( |
432 | + '--for-merge', |
433 | + action='store_true', |
434 | + help="Build an Ubuntu merge. Implies searching Debian for " |
435 | + "published versions first.", |
436 | + ) |
437 | + parser.add_argument( |
438 | + '--no-pristine-tar', |
439 | + action='store_true', |
440 | + help="Do not use pristine-tar branches to generate orig " |
441 | + "tarballs. This can be necessary as we stabilize the pristine-tar " |
442 | + "data generation, particularly if multiple component tarballs are " |
443 | + "present.", |
444 | |
445 | - def derive_orig_search_list_from_args(self, args): |
446 | - source = 'debian' if args.for_merge else 'ubuntu' |
447 | - orig_search_list = [ |
448 | - OrigSearchListEntry( |
449 | - mechanism=fetch_orig_from_parent_dir, |
450 | - source=None, |
451 | - must_build=True, |
452 | - ), |
453 | - OrigSearchListEntry( |
454 | - mechanism=fetch_orig_from_cache, |
455 | - source=source, |
456 | - must_build=True, |
457 | - ), |
458 | - ] |
459 | - if not args.no_pristine_tar: |
460 | - orig_search_list.append( |
461 | - OrigSearchListEntry( |
462 | - mechanism=functools.partial(fetch_orig_from_pristine_tar, |
463 | - repo=self.local_repo, |
464 | - ), |
465 | - source=source, |
466 | - must_build=True, |
467 | - ) |
468 | - ) |
469 | - orig_search_list.extend([ |
470 | + ) |
471 | + parser.add_argument('rem_args', help=argparse.SUPPRESS, |
472 | + nargs=argparse.REMAINDER) |
473 | + if not subparsers: |
474 | + return parser.parse_args() |
475 | + return 'build - %s' % kwargs['description'] |
476 | + |
477 | + |
478 | +def derive_orig_search_list_from_args(args): |
479 | + source = 'debian' if args.for_merge else 'ubuntu' |
480 | + orig_search_list = [ |
481 | + OrigSearchListEntry( |
482 | + mechanism=fetch_orig_from_parent_dir, |
483 | + source=None, |
484 | + must_build=True, |
485 | + ), |
486 | + OrigSearchListEntry( |
487 | + mechanism=fetch_orig_from_cache, |
488 | + source=source, |
489 | + must_build=True, |
490 | + ), |
491 | + ] |
492 | + if not args.no_pristine_tar: |
493 | + orig_search_list.append( |
494 | OrigSearchListEntry( |
495 | - mechanism=functools.partial(fetch_orig_from_launchpad, |
496 | - pullfile=args.pullfile, |
497 | - retries=args.retries, |
498 | - retry_backoffs=args.retry_backoffs, |
499 | - dl_cache=args.dl_cache, |
500 | + mechanism=functools.partial(fetch_orig_from_pristine_tar, |
501 | + repo=GitUbuntuRepository('.'), |
502 | ), |
503 | source=source, |
504 | must_build=True, |
505 | + ) |
506 | + ) |
507 | + orig_search_list.extend([ |
508 | + OrigSearchListEntry( |
509 | + mechanism=functools.partial(fetch_orig_from_launchpad, |
510 | + pullfile=args.pullfile, |
511 | + retries=args.retries, |
512 | + retry_backoffs=args.retry_backoffs, |
513 | + dl_cache=args.dl_cache, |
514 | ), |
515 | - OrigSearchListEntry( |
516 | - mechanism=fetch_orig_noop, |
517 | - source=[], |
518 | - must_build=True, |
519 | - ), |
520 | - ]) |
521 | + source=source, |
522 | + must_build=True, |
523 | + ), |
524 | + OrigSearchListEntry( |
525 | + mechanism=fetch_orig_noop, |
526 | + source=[], |
527 | + must_build=True, |
528 | + ), |
529 | + ]) |
530 | |
531 | - return orig_search_list |
532 | + return orig_search_list |
533 | |
534 | |
535 | - def main(self, args): |
536 | - rem_args = args.rem_args |
537 | - if len(rem_args) > 0: |
538 | - rem_args = rem_args[1:] |
539 | +def cli_main(args): |
540 | + rem_args = args.rem_args |
541 | + if len(rem_args) > 0: |
542 | + rem_args = rem_args[1:] |
543 | |
544 | - self.check_repository() |
545 | - self.local_repo = GitUbuntuRepository('.') |
546 | + changelog = Changelog.from_path('debian/changelog') |
547 | |
548 | - changelog = Changelog.from_path('debian/changelog') |
549 | + try: |
550 | + native = is_native_package(changelog) |
551 | + except NativenessMismatchError as e: |
552 | + logging.error("%s" % e) |
553 | + sys.exit(1) |
554 | |
555 | - try: |
556 | - native = is_native_package(changelog) |
557 | - except NativenessMismatchError as e: |
558 | - logging.error("%s" % e) |
559 | - sys.exit(1) |
560 | - |
561 | - if native: |
562 | - # No orig tarball required |
563 | - orig_search_list = [ |
564 | - OrigSearchListEntry( |
565 | - mechanism=fetch_orig_noop, |
566 | - source=[], |
567 | - must_build=True, |
568 | - ), |
569 | - ] |
570 | - else: |
571 | - orig_search_list = self.derive_orig_search_list_from_args(args) |
572 | + if native: |
573 | + # No orig tarball required |
574 | + orig_search_list = [ |
575 | + OrigSearchListEntry( |
576 | + mechanism=fetch_orig_noop, |
577 | + source=[], |
578 | + must_build=True, |
579 | + ), |
580 | + ] |
581 | + else: |
582 | + orig_search_list = derive_orig_search_list_from_args(args) |
583 | |
584 | - # See http://pad.ubuntu.com/KKB1kMR0JH for logic |
585 | - fetch_orig_and_build(orig_search_list, changelog, rem_args) |
586 | + # See http://pad.ubuntu.com/KKB1kMR0JH for logic |
587 | + main(orig_search_list, changelog, rem_args) |
588 | |
589 | |
590 | def derive_source_from_changelog(changelog): |
591 | diff --git a/gitubuntu/buildsource.py b/gitubuntu/buildsource.py |
592 | index 4e9b69b..1735129 100644 |
593 | --- a/gitubuntu/buildsource.py |
594 | +++ b/gitubuntu/buildsource.py |
595 | @@ -5,66 +5,62 @@ import argparse |
596 | import logging |
597 | import os |
598 | import sys |
599 | -from gitubuntu.build import GitUbuntuBuild |
600 | +from gitubuntu.build import cli_main as build_cli_main |
601 | from gitubuntu.cache import CACHE_PATH |
602 | |
603 | -class GitUbuntuBuildSource(GitUbuntuBuild): |
604 | - def __init__(self): |
605 | - super().__init__() |
606 | +def parse_args(subparsers=None, base_subparsers=None): |
607 | + kwargs = dict( |
608 | + description="Build a source package and changes file", |
609 | + formatter_class=argparse.ArgumentDefaultsHelpFormatter |
610 | + ) |
611 | + if base_subparsers: |
612 | + kwargs['parents'] = base_subparsers |
613 | + if subparsers: |
614 | + parser = subparsers.add_parser('build-source', **kwargs) |
615 | + help_text = "build-source - Build a source package" |
616 | + parser.set_defaults(func=cli_main) |
617 | + else: |
618 | + parser = argparse.ArgumentParser(**kwargs) |
619 | + parser.add_argument('--dl-cache', type=str, |
620 | + help=("Cache directory for downloads."), |
621 | + default=os.path.join(os.getcwd(), '.git', CACHE_PATH), |
622 | + ) |
623 | + parser.add_argument('--sign', action='store_true', |
624 | + help="Sign the source package.", |
625 | + ) |
626 | + parser.add_argument('--for-merge', action='store_true', |
627 | + help=("Build an Ubuntu merge. Implies searching Debian for " |
628 | + "published versions first and passing -sa to " |
629 | + "dpkg-buildpackage." |
630 | + ), |
631 | + ) |
632 | + parser.add_argument( |
633 | + '--no-pristine-tar', |
634 | + action='store_true', |
635 | + help="Do not use pristine-tar branches to generate orig " |
636 | + "tarballs. This can be necessary as we stabilize the pristine-tar " |
637 | + "data generation, particularly if multiple component tarballs are " |
638 | + "present.", |
639 | |
640 | - def parse_args(self, subparsers=None, base_subparsers=None): |
641 | - kwargs = dict( |
642 | - description="Build a source package and changes file", |
643 | - formatter_class=argparse.ArgumentDefaultsHelpFormatter |
644 | - ) |
645 | - if base_subparsers: |
646 | - kwargs['parents'] = base_subparsers |
647 | - if subparsers: |
648 | - parser = subparsers.add_parser('build-source', **kwargs) |
649 | - help_text = "build-source - Build a source package" |
650 | - parser.set_defaults(func=self.main) |
651 | - else: |
652 | - parser = argparse.ArgumentParser(**kwargs) |
653 | - parser.add_argument('--dl-cache', type=str, |
654 | - help=("Cache directory for downloads."), |
655 | - default=os.path.join(os.getcwd(), '.git', CACHE_PATH), |
656 | - ) |
657 | - parser.add_argument('--sign', action='store_true', |
658 | - help="Sign the source package.", |
659 | - ) |
660 | - parser.add_argument('--for-merge', action='store_true', |
661 | - help=("Build an Ubuntu merge. Implies searching Debian for " |
662 | - "published versions first and passing -sa to " |
663 | - "dpkg-buildpackage." |
664 | - ), |
665 | - ) |
666 | - parser.add_argument( |
667 | - '--no-pristine-tar', |
668 | - action='store_true', |
669 | - help="Do not use pristine-tar branches to generate orig " |
670 | - "tarballs. This can be necessary as we stabilize the pristine-tar " |
671 | - "data generation, particularly if multiple component tarballs are " |
672 | - "present.", |
673 | + ) |
674 | + parser.add_argument('rem_args', help=argparse.SUPPRESS, |
675 | + nargs=argparse.REMAINDER |
676 | + ) |
677 | + if not subparsers: |
678 | + return parser.parse_args() |
679 | + return "build-source - %s" % kwargs['description'] |
680 | |
681 | - ) |
682 | - parser.add_argument('rem_args', help=argparse.SUPPRESS, |
683 | - nargs=argparse.REMAINDER |
684 | - ) |
685 | - if not subparsers: |
686 | - return parser.parse_args() |
687 | - return "build-source - %s" % kwargs['description'] |
688 | - |
689 | - def main(self, args): |
690 | - if len(args.rem_args) != 0: |
691 | - logging.warning('Passing specified flags (%s) on to ' |
692 | - 'dpkg-buildpackage. Did you mean to ' |
693 | - 'call `git ubuntu build` instead?' % |
694 | - ' '.join(args.rem_args) |
695 | - ) |
696 | - else: |
697 | - args.rem_args = ['--', '-S', '-nc', '-d', '-i', '-I'] |
698 | - if args.for_merge: |
699 | - args.rem_args += ['-sa'] |
700 | - if not args.sign: |
701 | - args.rem_args += ['-us', '-uc'] |
702 | - super().main(args) |
703 | +def cli_main(args): |
704 | + if len(args.rem_args) != 0: |
705 | + logging.warning('Passing specified flags (%s) on to ' |
706 | + 'dpkg-buildpackage. Did you mean to ' |
707 | + 'call `git ubuntu build` instead?' % |
708 | + ' '.join(args.rem_args) |
709 | + ) |
710 | + else: |
711 | + args.rem_args = ['--', '-S', '-nc', '-d', '-i', '-I'] |
712 | + if args.for_merge: |
713 | + args.rem_args += ['-sa'] |
714 | + if not args.sign: |
715 | + args.rem_args += ['-us', '-uc'] |
716 | + build_cli_main(args) |
717 | diff --git a/gitubuntu/clone.py b/gitubuntu/clone.py |
718 | index 9a15c9c..a24a1a5 100644 |
719 | --- a/gitubuntu/clone.py |
720 | +++ b/gitubuntu/clone.py |
721 | @@ -5,6 +5,7 @@ import re |
722 | import shutil |
723 | from subprocess import CalledProcessError |
724 | import sys |
725 | +from gitubuntu.__main__ import top_level_defaults |
726 | from gitubuntu.git_repository import ( |
727 | GitUbuntuRepository, |
728 | GitUbuntuRepositoryFetchError, |
729 | @@ -18,12 +19,108 @@ except ImportError: |
730 | logging.error('Is %s installed?', pkg) |
731 | sys.exit(1) |
732 | |
733 | -class GitUbuntuClone: |
734 | - def parse_args(self, subparsers=None, base_subparsers=None): |
735 | - kwargs = dict( |
736 | - description='Clone a source package git repository to a directory', |
737 | - formatter_class=argparse.RawTextHelpFormatter, |
738 | - epilog=''' |
739 | +def copy_hooks(src, dst): |
740 | + try: |
741 | + os.mkdir(dst) |
742 | + except FileExistsError: |
743 | + pass |
744 | + |
745 | + for hook in os.listdir(src): |
746 | + shutil.copy2( |
747 | + os.path.join(src, hook), |
748 | + dst, |
749 | + ) |
750 | + |
751 | +def main(package, directory=None, lp_user=None, proto=None): |
752 | + """Entry point to clone subcommand |
753 | + |
754 | + @package: Name of source package |
755 | + @directory: directory to clone the repository into |
756 | + @lp_user: user to authenticate to Launchpad as |
757 | + @proto: protocol to use (one of 'http', 'https', 'git') |
758 | + |
759 | + If directory is None, a relative directory with the same name as |
760 | + package will be used. |
761 | + |
762 | + If lp_user is None, value of `git config gitubuntu.lpuser` will be |
763 | + used. |
764 | + |
765 | + If proto is None, the default value from gitubuntu.__main__ will be |
766 | + used (top_level_defaults.proto). |
767 | + |
768 | + Returns the resulting GitUbuntuRepository object |
769 | + """ |
770 | + directory = ( |
771 | + os.path.abspath(directory) |
772 | + if directory |
773 | + else os.path.join(os.path.abspath(os.getcwd()), package) |
774 | + ) |
775 | + if os.path.isdir(directory): |
776 | + logging.error('directory %s exists' % directory) |
777 | + sys.exit(1) |
778 | + |
779 | + if proto is None: |
780 | + proto = top_level_defaults.proto |
781 | + |
782 | + local_repo = GitUbuntuRepository( |
783 | + local_dir=directory, |
784 | + lp_user=lp_user, |
785 | + fetch_proto=proto, |
786 | + ) |
787 | + |
788 | + copy_hooks( |
789 | + os.path.join(os.path.dirname(__file__), os.path.pardir, 'hooks'), |
790 | + os.path.join(directory, '.git', 'hooks') |
791 | + ) |
792 | + |
793 | + try: |
794 | + local_repo.add_base_remotes(package) |
795 | + local_repo.fetch_base_remotes(verbose=True) |
796 | + except GitUbuntuRepositoryFetchError: |
797 | + logging.error("Unable to find an imported repository for %s. " |
798 | + "Please request an import by e-mailing " |
799 | + "usd-import-team@lists.launchpad.net.", |
800 | + package |
801 | + ) |
802 | + shutil.rmtree(local_repo.local_dir) |
803 | + raise |
804 | + |
805 | + try: |
806 | + local_repo.add_lpuser_remote(pkgname=package) |
807 | + local_repo.fetch_lpuser_remote(verbose=True) |
808 | + |
809 | + logging.debug("added remote '%s' -> %s", local_repo.lp_user, |
810 | + local_repo.raw_repo.remotes[local_repo.lp_user].url |
811 | + ) |
812 | + except GitUbuntuRepositoryFetchError: |
813 | + pass |
814 | + |
815 | + try: |
816 | + local_repo.create_tracking_branch( |
817 | + 'ubuntu/devel', |
818 | + 'pkg/ubuntu/devel' |
819 | + ) |
820 | + local_repo.checkout_commitish('ubuntu/devel') |
821 | + except: |
822 | + logging.error('Unable to checkout ubuntu/devel, does ' |
823 | + 'pkg/ubuntu/devel branch exist?' |
824 | + ) |
825 | + |
826 | + if os.path.isfile(os.path.join(directory, '.gitignore')): |
827 | + logging.warning('A .gitignore file exists in the source ' |
828 | + 'package. This will affect the behavior of git. Consider ' |
829 | + 'backing up the gitignore while working on this package ' |
830 | + 'to ensure all changes are tracked or passing appropriate ' |
831 | + 'flags to git commands (e.g., git status --ignored).' |
832 | + ) |
833 | + |
834 | + return local_repo |
835 | + |
836 | +def parse_args(subparsers=None, base_subparsers=None): |
837 | + kwargs = dict( |
838 | + description='Clone a source package git repository to a directory', |
839 | + formatter_class=argparse.RawTextHelpFormatter, |
840 | + epilog=''' |
841 | Example: |
842 | * clone to open-iscsi/ |
843 | %(prog)s open-iscsi |
844 | @@ -32,107 +129,41 @@ Example: |
845 | * use https rather than git protocol for read-only remotes: |
846 | %(prog)s --https open-iscsi |
847 | ''' |
848 | - ) |
849 | - if base_subparsers: |
850 | - kwargs['parents'] = base_subparsers |
851 | - if subparsers: |
852 | - parser = subparsers.add_parser('clone', **kwargs) |
853 | - parser.set_defaults(func=self.main) |
854 | - else: |
855 | - parser = argparse.ArgumentParser(**kwargs) |
856 | - parser.add_argument('package', type=str, |
857 | - help='Name of source package to clone' |
858 | - ) |
859 | - parser.add_argument('directory', type=str, |
860 | - help='Local directory to clone to. If not specified, a ' |
861 | - 'directory with the same name as PACKAGE will be ' |
862 | - 'used', |
863 | - default=None, |
864 | - nargs='?' |
865 | - ) |
866 | - parser.add_argument('-l', '--lp-user', type=str, help=argparse.SUPPRESS) |
867 | - if not subparsers: |
868 | - return parser.parse_args() |
869 | - return 'clone - %s' % kwargs['description'] |
870 | - |
871 | - @staticmethod |
872 | - def copy_hooks(src, dst): |
873 | - try: |
874 | - os.mkdir(dst) |
875 | - except FileExistsError: |
876 | - pass |
877 | - |
878 | - for hook in os.listdir(src): |
879 | - shutil.copy2( |
880 | - os.path.join(src, hook), |
881 | - dst, |
882 | - ) |
883 | - |
884 | - def main(self, args): |
885 | - try: |
886 | - lp_user = args.lp_user |
887 | - except AttributeError: |
888 | - lp_user = None |
889 | - directory = ( |
890 | - os.path.abspath(args.directory) |
891 | - if args.directory |
892 | - else os.path.join(os.path.abspath(os.getcwd()), args.package) |
893 | - ) |
894 | + ) |
895 | + if base_subparsers: |
896 | + kwargs['parents'] = base_subparsers |
897 | + if subparsers: |
898 | + parser = subparsers.add_parser('clone', **kwargs) |
899 | + parser.set_defaults(func=cli_main) |
900 | + else: |
901 | + parser = argparse.ArgumentParser(**kwargs) |
902 | + parser.add_argument('package', type=str, |
903 | + help='Name of source package to clone' |
904 | + ) |
905 | + parser.add_argument('directory', type=str, |
906 | + help='Local directory to clone to. If not specified, a ' |
907 | + ' directory with the same name as PACKAGE will be ' |
908 | + 'used', |
909 | + default=None, |
910 | + nargs='?' |
911 | + ) |
912 | + parser.add_argument('-l', '--lp-user', type=str, help=argparse.SUPPRESS) |
913 | + if not subparsers: |
914 | + return parser.parse_args() |
915 | + return 'clone - %s' % kwargs['description'] |
916 | |
917 | - if os.path.isdir(directory): |
918 | - logging.error('directory %s exists' % directory) |
919 | - sys.exit(1) |
920 | +def cli_main(args): |
921 | + try: |
922 | + lp_user = args.lp_user |
923 | + except AttributeError: |
924 | + lp_user = None |
925 | |
926 | - local_repo = GitUbuntuRepository( |
927 | - local_dir=directory, |
928 | - lp_user=lp_user, |
929 | - fetch_proto=args.proto, |
930 | - ) |
931 | - |
932 | - self.copy_hooks( |
933 | - os.path.join(os.path.dirname(__file__), os.path.pardir, 'hooks'), |
934 | - os.path.join(directory, '.git', 'hooks') |
935 | - ) |
936 | + main( |
937 | + package=args.package, |
938 | + directory=args.directory, |
939 | + lp_user=lp_user, |
940 | + proto=args.proto, |
941 | + ) |
942 | |
943 | - try: |
944 | - local_repo.add_base_remotes(args.package) |
945 | - local_repo.fetch_base_remotes(verbose=True) |
946 | - except GitUbuntuRepositoryFetchError: |
947 | - logging.error("Unable to find an imported repository for %s. " |
948 | - "Please request an import by e-mailing " |
949 | - "usd-import-team@lists.launchpad.net.", |
950 | - args.package |
951 | - ) |
952 | - shutil.rmtree(local_repo.local_dir) |
953 | - raise |
954 | - |
955 | - try: |
956 | - local_repo.add_lpuser_remote(pkgname=args.package) |
957 | - local_repo.fetch_lpuser_remote(verbose=True) |
958 | - |
959 | - logging.debug("added remote '%s' -> %s", local_repo.lp_user, |
960 | - local_repo.raw_repo.remotes[local_repo.lp_user].url |
961 | - ) |
962 | - except GitUbuntuRepositoryFetchError: |
963 | - pass |
964 | - |
965 | - try: |
966 | - local_repo.create_tracking_branch( |
967 | - 'ubuntu/devel', |
968 | - 'pkg/ubuntu/devel' |
969 | - ) |
970 | - local_repo.checkout_commitish('ubuntu/devel') |
971 | - except: |
972 | - logging.error('Unable to checkout ubuntu/devel, does ' |
973 | - 'pkg/ubuntu/devel branch exist?' |
974 | - ) |
975 | - |
976 | - if os.path.isfile(os.path.join(directory, '.gitignore')): |
977 | - logging.warning('A .gitignore file exists in the source ' |
978 | - 'package. This will affect the behavior of git. Consider ' |
979 | - 'backing up the gitignore while working on this package ' |
980 | - 'to ensure all changes are tracked or passing appropriate ' |
981 | - 'flags to git commands (e.g., git status --ignored).' |
982 | - ) |
983 | |
984 | # vi: ts=4 expandtab |
985 | diff --git a/gitubuntu/importer.py b/gitubuntu/importer.py |
986 | index cce1999..db08f46 100644 |
987 | --- a/gitubuntu/importer.py |
988 | +++ b/gitubuntu/importer.py |
989 | @@ -26,6 +26,7 @@ |
990 | |
991 | import argparse |
992 | import atexit |
993 | +import functools |
994 | import getpass |
995 | import logging |
996 | import os |
997 | @@ -50,7 +51,12 @@ from gitubuntu.git_repository import ( |
998 | PristineTarError, |
999 | ) |
1000 | from gitubuntu.run import decode_binary, run, runq |
1001 | -from gitubuntu.source_information import GitUbuntuSourceInformation, NoPublicationHistoryException, SourceExtractionException, launchpad_login_auth |
1002 | +from gitubuntu.source_information import ( |
1003 | + GitUbuntuSourceInformation, |
1004 | + NoPublicationHistoryException, |
1005 | + SourceExtractionException, |
1006 | + launchpad_login_auth, |
1007 | +) |
1008 | from gitubuntu.version import VERSION |
1009 | from gitubuntu.versioning import version_compare |
1010 | try: |
1011 | @@ -86,16 +92,18 @@ def dsc_to_tree_hash(repo, dsc_path): |
1012 | # extracted_dir. |
1013 | |
1014 | extracted_dir = os.path.join(temp_dir, 'x') |
1015 | - run(['dpkg-source', |
1016 | - '-x', |
1017 | - '--skip-patches', |
1018 | - dsc_path, |
1019 | - extracted_dir, # this is <temp_dir>/x |
1020 | + run( |
1021 | + [ |
1022 | + 'dpkg-source', |
1023 | + '-x', |
1024 | + '--skip-patches', |
1025 | + dsc_path, |
1026 | + extracted_dir, # this is <temp_dir>/x |
1027 | ] |
1028 | - ) |
1029 | + ) |
1030 | |
1031 | if not os.path.exists(extracted_dir): |
1032 | - logging.exception('No source extracted?') |
1033 | + logging.error("No source extracted?") |
1034 | raise SourceExtractionException( |
1035 | "Failed to find an extracted directory from " |
1036 | "dpkg-source -x" |
1037 | @@ -104,1259 +112,1462 @@ def dsc_to_tree_hash(repo, dsc_path): |
1038 | return repo.dir_to_tree(extracted_dir) |
1039 | |
1040 | |
1041 | -class GitUbuntuImport: |
1042 | - def __init__(self): |
1043 | - self.parent_overrides = None |
1044 | - |
1045 | - def parse_args(self, subparsers=None, base_subparsers=None): |
1046 | - kwargs = dict(description='Update a launchpad git tree based upon ' |
1047 | - 'the state of the Ubuntu and Debian archives', |
1048 | - formatter_class=argparse.ArgumentDefaultsHelpFormatter |
1049 | - ) |
1050 | - if base_subparsers: |
1051 | - kwargs['parents'] = base_subparsers |
1052 | - if subparsers: |
1053 | - parser = subparsers.add_parser('import', **kwargs) |
1054 | - parser.set_defaults(func=self.main) |
1055 | - else: |
1056 | - parser = argparse.ArgumentParser(**kwargs) |
1057 | - parser.add_argument('package', type=str, |
1058 | - help='Which package to update in the git tree') |
1059 | - parser.add_argument('-o', '--lp-owner', type=str, |
1060 | - help=argparse.SUPPRESS, |
1061 | - default='usd-import-team') |
1062 | - parser.add_argument('-l', '--lp-user', type=str, |
1063 | - help=argparse.SUPPRESS) |
1064 | - parser.add_argument('--dl-cache', type=str, |
1065 | - help=('Cache directory for downloads. Default is to ' |
1066 | - 'put downloads in <directory>/.git/%s' % |
1067 | - CACHE_PATH |
1068 | - ), |
1069 | - default=argparse.SUPPRESS) |
1070 | - parser.add_argument('--no-fetch', action='store_true', |
1071 | - help='Do not fetch from the remote (DANGEROUS; ' |
1072 | - 'only useful for debugging); implies ' |
1073 | - '--no-push') |
1074 | - parser.add_argument('--no-push', action='store_true', |
1075 | - help='Do not push to the remote') |
1076 | - parser.add_argument('--no-clean', action='store_true', |
1077 | - help='Do not clean the temporary directory') |
1078 | - parser.add_argument('-d', '--directory', type=str, |
1079 | - help='Use git repository at specified location rather ' |
1080 | - 'than a temporary directory (implies --no-clean). ' |
1081 | - 'Will create the local repository if needed.', |
1082 | - default=argparse.SUPPRESS |
1083 | - ) |
1084 | - parser.add_argument('--fixup-devel', action='store_true', |
1085 | - help=argparse.SUPPRESS) |
1086 | - parser.add_argument('--active-series-only', action='store_true', |
1087 | - help='Do an import of only active Ubuntu series ' |
1088 | - 'history, implies --no-push.') |
1089 | - parser.add_argument('--skip-applied', action='store_true', |
1090 | - help=argparse.SUPPRESS) |
1091 | - parser.add_argument('--skip-orig', action='store_true', |
1092 | - help=argparse.SUPPRESS) |
1093 | - parser.add_argument('--reimport', action='store_true', |
1094 | - help=argparse.SUPPRESS) |
1095 | - if not subparsers: |
1096 | - return parser.parse_args() |
1097 | - return 'import - %s' % kwargs['description'] |
1098 | - |
1099 | - def cleanup(self, no_clean, local_dir): |
1100 | - """Recursively remove local_dir if no_clean is not True""" |
1101 | - if not no_clean: |
1102 | - shutil.rmtree(local_dir) |
1103 | - else: |
1104 | - logging.info('Leaving %s as directed' % local_dir) |
1105 | - |
1106 | - def get_changelog_for_commit(self, tree_hash, publish_parent_commit, changelog_parent_commit): |
1107 | - """Extract changes to debian/changelog in this publish |
1108 | - |
1109 | - LP: #1633114 -- favor the changelog parent's diff |
1110 | - """ |
1111 | - raw_clog_entry = b'' |
1112 | - if changelog_parent_commit is not None: |
1113 | - cmd = ['diff-tree', '-p', changelog_parent_commit, |
1114 | - tree_hash, '--', 'debian/changelog'] |
1115 | - raw_clog_entry = self.local_repo.git_run(cmd).stdout |
1116 | - elif publish_parent_commit is not None: |
1117 | - cmd = ['diff-tree', '-p', publish_parent_commit, |
1118 | - tree_hash, '--', 'debian/changelog'] |
1119 | - raw_clog_entry = self.local_repo.git_run(cmd).stdout |
1120 | - |
1121 | - changelog_entry = b'' |
1122 | - changelog_entry_found = False |
1123 | - i = 0 |
1124 | - for line in raw_clog_entry.split(b'\n'): |
1125 | - # skip the header lines |
1126 | - if i < 4: |
1127 | - i += 1 |
1128 | - continue |
1129 | - if not line.startswith(b'+'): |
1130 | - # in case there are stray changelog changes, just take |
1131 | - # the first complete section of changelog additions |
1132 | - if changelog_entry_found: |
1133 | - break |
1134 | - continue |
1135 | - line = re.sub(b'^[+]', b'', line) |
1136 | - if not line.startswith(b' '): |
1137 | - continue |
1138 | - if not changelog_entry_found: |
1139 | - changelog_entry = b'\n\nNew changelog entries:\n' |
1140 | - changelog_entry += line + b'\n' |
1141 | - changelog_entry_found = True |
1142 | - return changelog_entry |
1143 | - |
1144 | - def _commit_import(self, spi, tree_hash, publish_parent_commit, changelog_parent_commit, upload_parent_commit, unapplied_parent_commit): |
1145 | - """Commit a tree object into the repository with the specified |
1146 | - parents |
1147 | - |
1148 | - The importer algorithm works by 'staging' the import of a |
1149 | - publication as a tree, then using that tree to determine the |
1150 | - appropriate parents from the publishing history and |
1151 | - debian/changelog. |
1152 | - |
1153 | - Arguments: |
1154 | - spi - A SourcePackageInformation instance for this publish |
1155 | - tree_hash - SHA1 from git-dsc-commit --tree-only of this publish |
1156 | - publish_parent_commit = SHA1 of publishing parent |
1157 | - changelog_parent_commit = SHA1 of changelog parent |
1158 | - upload_parent_commit = SHA1 of upload parent |
1159 | - unapplied_parent_commit = SHA1 of unapplied-patches import parent |
1160 | - """ |
1161 | - tag = None |
1162 | - |
1163 | - if unapplied_parent_commit: |
1164 | - import_type = 'patches-applied' |
1165 | - target_head_name = spi.applied_head_name(self.namespace) |
1166 | - if self.local_repo.get_applied_tag(spi.version, self.namespace) is None: |
1167 | - # Not imported before |
1168 | - tag = applied_tag(spi.version, self.namespace) |
1169 | - else: |
1170 | - import_type = 'patches-unapplied' |
1171 | - target_head_name = spi.head_name(self.namespace) |
1172 | - if self.local_repo.get_import_tag(spi.version, self.namespace) is None: |
1173 | - # Not imported before |
1174 | - tag = import_tag(spi.version, self.namespace) |
1175 | - |
1176 | - # Do not show importer/ namespace to user |
1177 | - _, _, pretty_head_name = target_head_name.partition('%s/' % self.namespace) |
1178 | - |
1179 | - changelog_entry = self.get_changelog_for_commit( |
1180 | - tree_hash, |
1181 | - publish_parent_commit, |
1182 | - changelog_parent_commit |
1183 | - ) |
1184 | - msg = (b'Import %s version %b to %b\n\nImported using git-ubuntu import.' % |
1185 | - (import_type.encode(), spi.version.encode(), pretty_head_name.encode()) |
1186 | - ) |
1187 | - |
1188 | - parents = [] |
1189 | - |
1190 | - if publish_parent_commit is None and \ |
1191 | - changelog_parent_commit is None and \ |
1192 | - upload_parent_commit is None and \ |
1193 | - unapplied_parent_commit is None and \ |
1194 | - target_head_name in self.local_repo.local_branch_names: |
1195 | - # No parents and not creating a new branch |
1196 | - tag = orphan_tag(spi.version, self.namespace) |
1197 | - else: |
1198 | - msg += b'\n' |
1199 | - |
1200 | - if publish_parent_commit is not None: |
1201 | - parents.append(publish_parent_commit) |
1202 | - msg += b'\nPublish parent: %b' % publish_parent_commit.encode() |
1203 | - if changelog_parent_commit is not None: |
1204 | - parents.append(changelog_parent_commit) |
1205 | - msg += b'\nChangelog parent: %b' % changelog_parent_commit.encode() |
1206 | - if upload_parent_commit is not None: |
1207 | - parents.append(upload_parent_commit) |
1208 | - msg += b'\nUpload parent: %b' % upload_parent_commit.encode() |
1209 | - if unapplied_parent_commit is not None: |
1210 | - parents.append(unapplied_parent_commit) |
1211 | - msg += b'\nUnapplied parent: %b' % unapplied_parent_commit.encode() |
1212 | - |
1213 | - msg += b'%b' % changelog_entry |
1214 | - |
1215 | - commit_hash = self.local_repo.commit_tree_hash( |
1216 | - tree_hash, |
1217 | - parents, |
1218 | - msg, |
1219 | - environment_spi=spi |
1220 | - ) |
1221 | +def cleanup(no_clean, local_dir): |
1222 | + '''Remove a local directory, conditionally |
1223 | |
1224 | - self.local_repo.update_head_to_commit(target_head_name, commit_hash) |
1225 | + no_clean: if True, do not remove @local_dir and print a message instead |
1226 | + local_dir: directory to remove |
1227 | + ''' |
1228 | + if not no_clean: |
1229 | + shutil.rmtree(local_dir) |
1230 | + else: |
1231 | + logging.info('Leaving %s as directed' % local_dir) |
1232 | |
1233 | - logging.debug('Committed %s import of %s as %s in %s', |
1234 | - import_type, spi.version, commit_hash, pretty_head_name |
1235 | - ) |
1236 | |
1237 | - if tag is not None: |
1238 | - # should be annotated to use create_tag API |
1239 | - logging.debug('Creating tag %s pointing to %s', tag, commit_hash) |
1240 | - self.local_repo.git_run(['tag', '-a', '-m', 'git-ubuntu import v%s' % VERSION, |
1241 | - tag, commit_hash |
1242 | - ] |
1243 | - ) |
1244 | +# XXX: need a namedtuple to hold common arguments |
1245 | +def main(pkgname, owner, no_clean, directory, user, proto, no_fetch, |
1246 | + no_push, dl_cache, fixup_devel, active_series_only, skip_orig, |
1247 | + pullfile, parentfile, retries, retry_backoffs, skip_applied, |
1248 | + reimport, |
1249 | +): |
1250 | + if owner == 'usd-import-team': |
1251 | + namespace = 'importer' |
1252 | + else: |
1253 | + namespace = owner |
1254 | |
1255 | - def commit_unapplied_patches_import(self, spi, import_tree_hash, publish_parent_commit, changelog_parent_commit, upload_parent_commit): |
1256 | - self._commit_import( |
1257 | - spi, |
1258 | - import_tree_hash, |
1259 | - publish_parent_commit, |
1260 | - changelog_parent_commit, |
1261 | - upload_parent_commit, |
1262 | - # unapplied trees do not have a distinct unapplied parent |
1263 | - None |
1264 | - ) |
1265 | + logging.info('Ubuntu Server Team importer v%s' % VERSION) |
1266 | |
1267 | - def commit_applied_patches_import(self, spi, import_tree_hash, publish_parent_commit, changelog_parent_commit, unapplied_parent_commit): |
1268 | - self._commit_import( |
1269 | - spi, |
1270 | - import_tree_hash, |
1271 | - publish_parent_commit, |
1272 | - changelog_parent_commit, |
1273 | - # uploads will be unapplied trees currently, so applied trees |
1274 | - # can not have them as direct parents |
1275 | - None, |
1276 | - unapplied_parent_commit |
1277 | - ) |
1278 | + repo = GitUbuntuRepository(directory, user, proto) |
1279 | + if not directory: |
1280 | + logging.info('Using git repository at %s', repo.local_dir) |
1281 | + |
1282 | + atexit.register(cleanup, no_clean, repo.local_dir) |
1283 | |
1284 | - def import_dsc(self, dsc, version, dist): |
1285 | - """Imports the dsc file to importer/{debian,ubuntu}/dsc |
1286 | - |
1287 | - Arguments: |
1288 | - spi - A GitUbuntuSourcePackageInformation instance |
1289 | - """ |
1290 | - |
1291 | - # Add DSC file to the appropriate branch |
1292 | - runq(['git', 'checkout', '%s/importer/%s/dsc' % (self.namespace, dist)], env=self.local_repo.env) |
1293 | - # It is possible the same-named DSC is published multiple times |
1294 | - # with different contents, e.g. on epoch bumps |
1295 | - local_dir = os.path.join(self.local_repo.local_dir, version) |
1296 | - os.makedirs(local_dir, exist_ok=True) |
1297 | - shutil.copy(dsc.dsc_path, |
1298 | - local_dir) |
1299 | - self.local_repo.git_run(['add', self.local_repo.local_dir]) |
1300 | + repo.add_base_remotes(pkgname, repo_owner=owner) |
1301 | + if not no_fetch and not reimport: |
1302 | try: |
1303 | - # Anything to commit? (this will be 0 if, for instance, the |
1304 | - # same dsc is being imported again) |
1305 | - runq(['git', 'diff', '--exit-code', 'HEAD'], env=self.local_repo.env) |
1306 | - except: |
1307 | - self.local_repo.git_run(['commit', '-m', 'DSC file for %s' % version]) |
1308 | + repo.fetch_base_remotes() |
1309 | + except GitUbuntuRepositoryFetchError: |
1310 | + pass |
1311 | |
1312 | - self.local_repo.clean_repository_state() |
1313 | + if not no_fetch: |
1314 | + repo.delete_branches_in_namespace(namespace) |
1315 | + repo.delete_tags_in_namespace(namespace) |
1316 | + repo.copy_base_references(namespace) |
1317 | |
1318 | - def orig_imported(self, orig_tarball_paths, dist): |
1319 | - """Determines if a list of paths to tarballs have already been imported to @dist |
1320 | + if not no_fetch and not reimport: |
1321 | + try: |
1322 | + repo.fetch_remote_refspecs('pkg', |
1323 | + refspecs=['refs/tags/*:refs/tags/%s/*' % namespace], |
1324 | + ) |
1325 | + except GitUbuntuRepositoryFetchError: |
1326 | + pass |
1327 | |
1328 | - Assumes that the pristine-tar branch exists |
1329 | - """ |
1330 | + if reimport: |
1331 | + if no_fetch: |
1332 | + logging.warning( |
1333 | + "--reimport specified with --no-fetch. Upload " |
1334 | + "tags would not be incorporated into the new import " |
1335 | + "and would be lost forever. This is not currently " |
1336 | + "supported and --no-fetch will be ignored for upload " |
1337 | + "tags." |
1338 | + ) |
1339 | try: |
1340 | - return self.local_repo.verify_pristine_tar( |
1341 | - orig_tarball_paths, |
1342 | - dist, |
1343 | - self.namespace |
1344 | + repo.fetch_remote_refspecs('pkg', |
1345 | + refspecs=['refs/tags/upload/*:refs/tags/%s/upload/*' % namespace], |
1346 | ) |
1347 | - except PristineTarError as e: |
1348 | - raise GitUbuntuImportOrigError from e |
1349 | + except GitUbuntuRepositoryFetchError: |
1350 | + pass |
1351 | |
1352 | - def import_orig(self, dsc, version, dist): |
1353 | - """Imports the orig-tarball using gbp import-org --pristine-tar |
1354 | + repo.ensure_importer_branches_exist(namespace) |
1355 | + |
1356 | + debian_sinfo = GitUbuntuSourceInformation( |
1357 | + 'debian', |
1358 | + pkgname, |
1359 | + os.path.abspath(pullfile), |
1360 | + retries, |
1361 | + retry_backoffs, |
1362 | + ) |
1363 | + |
1364 | + ubuntu_sinfo = GitUbuntuSourceInformation( |
1365 | + 'ubuntu', |
1366 | + pkgname, |
1367 | + os.path.abspath(pullfile), |
1368 | + retries, |
1369 | + retry_backoffs, |
1370 | + ) |
1371 | + |
1372 | + debian_head_versions = ( |
1373 | + repo.get_heads_and_versions('debian', namespace) |
1374 | + ) |
1375 | + for head_name in sorted(debian_head_versions): |
1376 | + _, _, pretty_head_name = head_name.partition('%s/' % namespace) |
1377 | + logging.debug('Last imported %s version is %s', |
1378 | + pretty_head_name, |
1379 | + debian_head_versions[head_name]['version'] |
1380 | + ) |
1381 | |
1382 | - Arguments: |
1383 | - dsc - A GitUbuntuRepository object |
1384 | - version - the string package version |
1385 | - dist - the string distribution, either 'ubuntu' or 'debian' |
1386 | - """ |
1387 | - try: |
1388 | - orig_tarball_path = dsc.orig_tarball_path |
1389 | - except GitUbuntuDscError as e: |
1390 | - raise GitUbuntuImportOrigError( |
1391 | - 'Unable to import orig tarball in DSC for version %s' % version |
1392 | - ) from e |
1393 | - |
1394 | - if orig_tarball_path is None: |
1395 | - # native packages only have a .tar.* |
1396 | - native_re = re.compile(r'.*\.tar\.[^.]+$') |
1397 | - for entry in dsc['Files']: |
1398 | - if native_re.match(entry['name']): |
1399 | - return |
1400 | - raise GitUbuntuImportOrigError('No orig tarball in ' |
1401 | - 'DSC for version %s.' % version) |
1402 | + ubuntu_head_versions = ( |
1403 | + repo.get_heads_and_versions('ubuntu', namespace) |
1404 | + ) |
1405 | + for head_name in sorted(ubuntu_head_versions): |
1406 | + _, _, pretty_head_name = head_name.partition('%s/' % namespace) |
1407 | + logging.debug('Last imported %s version is %s', |
1408 | + pretty_head_name, |
1409 | + ubuntu_head_versions[head_name]['version'] |
1410 | + ) |
1411 | |
1412 | - try: |
1413 | - orig_component_paths = dsc.component_tarball_paths |
1414 | - except GitUbuntuDscError as e: |
1415 | - raise GitUbuntuImportOrigError( |
1416 | - 'Unable to import component tarballs in DSC for version ' |
1417 | - '%s' % version |
1418 | - ) from e |
1419 | - |
1420 | - orig_paths = [orig_tarball_path] + list(orig_component_paths.values()) |
1421 | - if (not orig_tarball_path or |
1422 | - self.orig_imported(orig_paths, dist)): |
1423 | - return |
1424 | - if not all(map(os.path.exists, orig_paths)): |
1425 | - raise GitUbuntuImportOrigError( |
1426 | - 'Unable to find tarball: %s' % |
1427 | - [p for p in orig_paths if not os.path.exists(p)] |
1428 | + if not skip_applied: |
1429 | + applied_debian_head_versions = ( |
1430 | + repo.get_heads_and_versions( |
1431 | + 'applied/debian', namespace |
1432 | + ) |
1433 | + ) |
1434 | + for head_name in sorted(applied_debian_head_versions): |
1435 | + _, _, pretty_head_name = head_name.partition( |
1436 | + '%s/' % namespace |
1437 | + ) |
1438 | + logging.debug('Last applied %s version is %s', |
1439 | + pretty_head_name, |
1440 | + applied_debian_head_versions[head_name]['version'] |
1441 | ) |
1442 | |
1443 | - try: |
1444 | - with self.local_repo.pristine_tar_branches(dist, self.namespace): |
1445 | - oldcwd = os.getcwd() |
1446 | - # gbp does not support running from arbitrary git trees or |
1447 | - # working directories |
1448 | - # https://github.com/agx/git-buildpackage/pull/16 |
1449 | - os.chdir(self.local_repo.local_dir) |
1450 | - |
1451 | - ext = os.path.splitext(dsc.orig_tarball_path)[1] |
1452 | - |
1453 | - # make this non-fatal if upstream already exist as tagged? |
1454 | - cmd = ['gbp', 'import-orig', '--no-merge', |
1455 | - '--upstream-branch', 'do-not-push', |
1456 | - '--pristine-tar', '--no-interactive', |
1457 | - '--no-symlink-orig', |
1458 | - '--upstream-tag=%s/upstream/%s/%%(version)s%s' % |
1459 | - (self.namespace, dist, ext)] |
1460 | - cmd.extend(map(lambda x: '--component=%s' % x, |
1461 | - list(orig_component_paths.keys())) |
1462 | - ) |
1463 | - cmd.extend([orig_tarball_path,]) |
1464 | - run(cmd, env=self.local_repo.env) |
1465 | - except CalledProcessError as e: |
1466 | - raise GitUbuntuImportOrigError( |
1467 | - 'Unable to import tarballs: %s' % orig_paths |
1468 | - ) from e |
1469 | - finally: |
1470 | - os.chdir(oldcwd) |
1471 | + applied_ubuntu_head_versions = ( |
1472 | + repo.get_heads_and_versions( |
1473 | + 'applied/ubuntu', namespace |
1474 | + ) |
1475 | + ) |
1476 | + for head_name in sorted(applied_ubuntu_head_versions): |
1477 | + _, _, pretty_head_name = head_name.partition( |
1478 | + '%s/' % namespace |
1479 | + ) |
1480 | + logging.debug('Last applied %s version is %s', |
1481 | + pretty_head_name, |
1482 | + applied_ubuntu_head_versions[head_name]['version'] |
1483 | + ) |
1484 | |
1485 | - self.local_repo.clean_repository_state() |
1486 | + oldcwd = os.getcwd() |
1487 | + os.chdir(repo.local_dir) |
1488 | + |
1489 | + if dl_cache is None: |
1490 | + workdir = os.path.join(repo.git_dir, CACHE_PATH) |
1491 | + else: |
1492 | + workdir = dl_cache |
1493 | + |
1494 | + os.makedirs(workdir, exist_ok=True) |
1495 | + |
1496 | + # now sets a global _PARENT_OVERRIDES |
1497 | + parse_parentfile(parentfile, pkgname) |
1498 | + |
1499 | + only_debian, history_found = import_publishes( |
1500 | + repo=repo, |
1501 | + pkgname=pkgname, |
1502 | + namespace=namespace, |
1503 | + patches_applied=False, |
1504 | + debian_head_versions=debian_head_versions, |
1505 | + ubuntu_head_versions=ubuntu_head_versions, |
1506 | + debian_sinfo=debian_sinfo, |
1507 | + ubuntu_sinfo=ubuntu_sinfo, |
1508 | + active_series_only=active_series_only, |
1509 | + workdir=workdir, |
1510 | + fixup_devel=fixup_devel, |
1511 | + skip_orig=skip_orig, |
1512 | + ) |
1513 | + |
1514 | + if not history_found: |
1515 | + logging.error("No publication history for '%s' in debian or ubuntu. " |
1516 | + "Wrong source package name?", pkgname) |
1517 | + sys.exit(1) |
1518 | + |
1519 | + if not skip_applied: |
1520 | + import_publishes( |
1521 | + repo=repo, |
1522 | + pkgname=pkgname, |
1523 | + namespace=namespace, |
1524 | + patches_applied=True, |
1525 | + debian_head_versions=applied_debian_head_versions, |
1526 | + ubuntu_head_versions=applied_ubuntu_head_versions, |
1527 | + debian_sinfo=debian_sinfo, |
1528 | + ubuntu_sinfo=ubuntu_sinfo, |
1529 | + active_series_only=active_series_only, |
1530 | + workdir=workdir, |
1531 | + fixup_devel=fixup_devel, |
1532 | + skip_orig=skip_orig, |
1533 | + ) |
1534 | |
1535 | - def import_patches_unapplied_tree(self, dsc_pathname): |
1536 | - """Imports the patches-unapplied source package and writes the |
1537 | - corresponding working tree for a given srcpkg |
1538 | + os.chdir(oldcwd) |
1539 | |
1540 | - Arguments: |
1541 | - dsc_pathname - A string path to a DSC file |
1542 | - """ |
1543 | + repo.garbage_collect() |
1544 | |
1545 | - import_tree_hash = dsc_to_tree_hash(self.local_repo, dsc_pathname) |
1546 | - self.local_repo.clean_repository_state() |
1547 | + if no_push: |
1548 | + logging.info('Not pushing to remote as specified') |
1549 | + else: |
1550 | + lp = launchpad_login_auth() |
1551 | + repo_path = '~%s/ubuntu/+source/%s/+git/%s' % ( |
1552 | + owner, |
1553 | + pkgname, |
1554 | + pkgname, |
1555 | + ) |
1556 | + lp_git_repo = lp.git_repositories.getByPath(path=repo_path) |
1557 | + if reimport: |
1558 | + if not lp_git_repo.canBeDeleted(): |
1559 | + logging.warning( |
1560 | + "There are linked MPs to the pkg repository, " |
1561 | + "which will be deleted:" |
1562 | + ) |
1563 | + for mp in lp_git_repo.landing_candidates: |
1564 | + logging.warning( |
1565 | + "Adding comment to %s indicating reimport", |
1566 | + mp.web_link |
1567 | + ) |
1568 | + mp.createComment( |
1569 | + content="%s has been reimported and this MP " |
1570 | + "must be manually resubmitted." % |
1571 | + pkgname, |
1572 | + subject="%s has been reimported" % pkgname, |
1573 | + ) |
1574 | + # repo deletion will delete the MP, but hopefully |
1575 | + # the above comment will be sufficient to explain |
1576 | + # why |
1577 | + lp_git_repo.lp_delete() |
1578 | + |
1579 | + repo.git_run([ |
1580 | + 'push', '--atomic', 'pkg', |
1581 | + 'refs/heads/%s/*:refs/heads/*' % namespace, |
1582 | + 'refs/tags/%s/*:refs/tags/*' % namespace, |
1583 | + ]) |
1584 | + # update our reference |
1585 | + lp_git_repo = lp.git_repositories.getByPath(path=repo_path) |
1586 | + for i in range(retries): |
1587 | + try: |
1588 | + if only_debian: |
1589 | + lp_git_repo.default_branch = 'refs/heads/debian/sid' |
1590 | + else: |
1591 | + lp_git_repo.default_branch = 'refs/heads/ubuntu/devel' |
1592 | + lp_git_repo.lp_save() |
1593 | + break |
1594 | + except (NotFound, PreconditionFailed) as e: |
1595 | + time.sleep(retry_backoffs[i]) |
1596 | + lp_git_repo.lp_refresh() |
1597 | + |
1598 | + |
1599 | +def get_changelog_for_commit( |
1600 | + repo, |
1601 | + tree_hash, |
1602 | + publish_parent_commit, |
1603 | + changelog_parent_commit, |
1604 | +): |
1605 | + """Extract changes to debian/changelog in this publish |
1606 | + |
1607 | + LP: #1633114 -- favor the changelog parent's diff |
1608 | + """ |
1609 | + raw_clog_entry = b'' |
1610 | + if changelog_parent_commit is not None: |
1611 | + cmd = ['diff-tree', '-p', changelog_parent_commit, |
1612 | + tree_hash, '--', 'debian/changelog'] |
1613 | + raw_clog_entry = repo.git_run(cmd).stdout |
1614 | + elif publish_parent_commit is not None: |
1615 | + cmd = ['diff-tree', '-p', publish_parent_commit, |
1616 | + tree_hash, '--', 'debian/changelog'] |
1617 | + raw_clog_entry = repo.git_run(cmd).stdout |
1618 | + |
1619 | + changelog_entry = b'' |
1620 | + changelog_entry_found = False |
1621 | + i = 0 |
1622 | + for line in raw_clog_entry.split(b'\n'): |
1623 | + # skip the header lines |
1624 | + if i < 4: |
1625 | + i += 1 |
1626 | + continue |
1627 | + if not line.startswith(b'+'): |
1628 | + # in case there are stray changelog changes, just take |
1629 | + # the first complete section of changelog additions |
1630 | + if changelog_entry_found: |
1631 | + break |
1632 | + continue |
1633 | + line = re.sub(b'^[+]', b'', line) |
1634 | + if not line.startswith(b' '): |
1635 | + continue |
1636 | + if not changelog_entry_found: |
1637 | + changelog_entry = b'\n\nNew changelog entries:\n' |
1638 | + changelog_entry += line + b'\n' |
1639 | + changelog_entry_found = True |
1640 | + return changelog_entry |
1641 | + |
1642 | +def _commit_import(repo, spi, tree_hash, namespace, |
1643 | + publish_parent_commit, |
1644 | + changelog_parent_commit, upload_parent_commit, |
1645 | + unapplied_parent_commit |
1646 | +): |
1647 | + """Commit a tree object into the repository with the specified |
1648 | + parents |
1649 | + |
1650 | + The importer algorithm works by 'staging' the import of a |
1651 | + publication as a tree, then using that tree to determine the |
1652 | + appropriate parents from the publishing history and |
1653 | + debian/changelog. |
1654 | + |
1655 | + Arguments: |
1656 | + spi - A SourcePackageInformation instance for this publish |
1657 | + tree_hash - SHA1 from git-dsc-commit --tree-only of this publish |
1658 | + publish_parent_commit = SHA1 of publishing parent |
1659 | + changelog_parent_commit = SHA1 of changelog parent |
1660 | + upload_parent_commit = SHA1 of upload parent |
1661 | + unapplied_parent_commit = SHA1 of unapplied-patches import parent |
1662 | + """ |
1663 | + tag = None |
1664 | + |
1665 | + if unapplied_parent_commit: |
1666 | + import_type = 'patches-applied' |
1667 | + target_head_name = spi.applied_head_name(namespace) |
1668 | + if repo.get_applied_tag(spi.version, namespace) is None: |
1669 | + # Not imported before |
1670 | + tag = applied_tag(spi.version, namespace) |
1671 | + else: |
1672 | + import_type = 'patches-unapplied' |
1673 | + target_head_name = spi.head_name(namespace) |
1674 | + if repo.get_import_tag(spi.version, namespace) is None: |
1675 | + # Not imported before |
1676 | + tag = import_tag(spi.version, namespace) |
1677 | + |
1678 | + # Do not show importer/ namespace to user |
1679 | + _, _, pretty_head_name = target_head_name.partition('%s/' % namespace) |
1680 | + |
1681 | + changelog_entry = get_changelog_for_commit( |
1682 | + repo, |
1683 | + tree_hash, |
1684 | + publish_parent_commit, |
1685 | + changelog_parent_commit |
1686 | + ) |
1687 | + msg = (b'Import %s version %b to %b\n\nImported using git-ubuntu import.' % |
1688 | + (import_type.encode(), spi.version.encode(), pretty_head_name.encode()) |
1689 | + ) |
1690 | + |
1691 | + parents = [] |
1692 | + |
1693 | + if publish_parent_commit is None and \ |
1694 | + changelog_parent_commit is None and \ |
1695 | + upload_parent_commit is None and \ |
1696 | + unapplied_parent_commit is None and \ |
1697 | + target_head_name in repo.local_branch_names: |
1698 | + # No parents and not creating a new branch |
1699 | + tag = orphan_tag(spi.version, namespace) |
1700 | + else: |
1701 | + msg += b'\n' |
1702 | + |
1703 | + if publish_parent_commit is not None: |
1704 | + parents.append(publish_parent_commit) |
1705 | + msg += b'\nPublish parent: %b' % publish_parent_commit.encode() |
1706 | + if changelog_parent_commit is not None: |
1707 | + parents.append(changelog_parent_commit) |
1708 | + msg += b'\nChangelog parent: %b' % changelog_parent_commit.encode() |
1709 | + if upload_parent_commit is not None: |
1710 | + parents.append(upload_parent_commit) |
1711 | + msg += b'\nUpload parent: %b' % upload_parent_commit.encode() |
1712 | + if unapplied_parent_commit is not None: |
1713 | + parents.append(unapplied_parent_commit) |
1714 | + msg += b'\nUnapplied parent: %b' % unapplied_parent_commit.encode() |
1715 | + |
1716 | + msg += b'%b' % changelog_entry |
1717 | + |
1718 | + commit_hash = repo.commit_tree_hash( |
1719 | + tree_hash, |
1720 | + parents, |
1721 | + msg, |
1722 | + environment_spi=spi |
1723 | + ) |
1724 | + |
1725 | + repo.update_head_to_commit(target_head_name, commit_hash) |
1726 | + |
1727 | + logging.debug('Committed %s import of %s as %s in %s', |
1728 | + import_type, spi.version, commit_hash, pretty_head_name |
1729 | + ) |
1730 | |
1731 | - return import_tree_hash |
1732 | + if tag is not None: |
1733 | + # should be annotated to use create_tag API |
1734 | + logging.debug('Creating tag %s pointing to %s', tag, commit_hash) |
1735 | + repo.git_run(['tag', '-a', '-m', 'git-ubuntu import v%s' % VERSION, |
1736 | + tag, commit_hash |
1737 | + ] |
1738 | + ) |
1739 | |
1740 | - def import_patches_applied_tree(self, dsc_pathname): |
1741 | - """Imports the patches-applied source package and writes the |
1742 | - corresponding working tree for a given srcpkg, patch by patch |
1743 | +def commit_unapplied_patches_import( |
1744 | + repo, |
1745 | + spi, |
1746 | + import_tree_hash, |
1747 | + namespace, |
1748 | + publish_parent_commit, |
1749 | + changelog_parent_commit, |
1750 | + upload_parent_commit, |
1751 | +): |
1752 | + _commit_import( |
1753 | + repo, |
1754 | + spi, |
1755 | + import_tree_hash, |
1756 | + namespace, |
1757 | + publish_parent_commit, |
1758 | + changelog_parent_commit, |
1759 | + upload_parent_commit, |
1760 | + # unapplied trees do not have a distinct unapplied parent |
1761 | + None, |
1762 | + ) |
1763 | + |
1764 | +def commit_applied_patches_import( |
1765 | + repo, |
1766 | + spi, |
1767 | + import_tree_hash, |
1768 | + namespace, |
1769 | + publish_parent_commit, |
1770 | + changelog_parent_commit, |
1771 | + unapplied_parent_commit |
1772 | +): |
1773 | + _commit_import( |
1774 | + repo, |
1775 | + spi, |
1776 | + import_tree_hash, |
1777 | + namespace, |
1778 | + publish_parent_commit, |
1779 | + changelog_parent_commit, |
1780 | + # uploads will be unapplied trees currently, so applied trees |
1781 | + # can not have them as direct parents |
1782 | + None, |
1783 | + unapplied_parent_commit, |
1784 | + ) |
1785 | + |
1786 | +def import_dsc(repo, dsc, namespace, version, dist): |
1787 | + """Imports the dsc file to importer/{debian,ubuntu}/dsc |
1788 | + |
1789 | + Arguments: |
1790 | + spi - A GitUbuntuSourcePackageInformation instance |
1791 | + """ |
1792 | + |
1793 | + # Add DSC file to the appropriate branch |
1794 | + runq(['git', 'checkout', '%s/importer/%s/dsc' % (namespace, dist)], env=repo.env) |
1795 | + # It is possible the same-named DSC is published multiple times |
1796 | + # with different contents, e.g. on epoch bumps |
1797 | + local_dir = os.path.join(repo.local_dir, version) |
1798 | + os.makedirs(local_dir, exist_ok=True) |
1799 | + shutil.copy(dsc.dsc_path, local_dir) |
1800 | + repo.git_run(['add', repo.local_dir]) |
1801 | + try: |
1802 | + # Anything to commit? (this will be 0 if, for instance, the |
1803 | + # same dsc is being imported again) |
1804 | + runq(['git', 'diff', '--exit-code', 'HEAD'], env=repo.env) |
1805 | + except: |
1806 | + repo.git_run(['commit', '-m', 'DSC file for %s' % version]) |
1807 | + |
1808 | + repo.clean_repository_state() |
1809 | + |
1810 | +def orig_imported(repo, orig_tarball_paths, namespace, dist): |
1811 | + """Determines if a list of paths to tarballs have already been imported to @dist |
1812 | + |
1813 | + Assumes that the pristine-tar branch exists |
1814 | + """ |
1815 | + try: |
1816 | + return repo.verify_pristine_tar( |
1817 | + orig_tarball_paths, |
1818 | + dist, |
1819 | + namespace, |
1820 | + ) |
1821 | + except PristineTarError as e: |
1822 | + raise GitUbuntuImportOrigError from e |
1823 | + |
1824 | +def import_orig(repo, dsc, namespace, version, dist): |
1825 | + """Imports the orig-tarball using gbp import-org --pristine-tar |
1826 | + |
1827 | + Arguments: |
1828 | + repo - A GitUbuntuRepository object |
1829 | + dsc - a GitUbuntuDsc object |
1830 | + namespace - string namespace to search for tags, etc. |
1831 | + version - the string package version |
1832 | + dist - the string distribution, either 'ubuntu' or 'debian' |
1833 | + """ |
1834 | + try: |
1835 | + orig_tarball_path = dsc.orig_tarball_path |
1836 | + except GitUbuntuDscError as e: |
1837 | + raise GitUbuntuImportOrigError( |
1838 | + 'Unable to import orig tarball in DSC for version %s' % version |
1839 | + ) from e |
1840 | + |
1841 | + if orig_tarball_path is None: |
1842 | + # native packages only have a .tar.* |
1843 | + native_re = re.compile(r'.*\.tar\.[^.]+$') |
1844 | + for entry in dsc['Files']: |
1845 | + if native_re.match(entry['name']): |
1846 | + return |
1847 | + raise GitUbuntuImportOrigError('No orig tarball in ' |
1848 | + 'DSC for version %s.' % version |
1849 | + ) |
1850 | |
1851 | - Arguments: |
1852 | - dsc_pathname - A string path to a DSC file |
1853 | - """ |
1854 | - oldcwd = os.getcwd() |
1855 | + try: |
1856 | + orig_component_paths = dsc.component_tarball_paths |
1857 | + except GitUbuntuDscError as e: |
1858 | + raise GitUbuntuImportOrigError( |
1859 | + 'Unable to import component tarballs in DSC for version ' |
1860 | + '%s' % version |
1861 | + ) from e |
1862 | + |
1863 | + orig_paths = [orig_tarball_path] + list(orig_component_paths.values()) |
1864 | + if (not orig_tarball_path or |
1865 | + orig_imported(repo, orig_paths, namespace, dist) |
1866 | + ): |
1867 | + return |
1868 | + if not all(map(os.path.exists, orig_paths)): |
1869 | + raise GitUbuntuImportOrigError('Unable to find tarball: ' |
1870 | + '%s' % [p for p in orig_paths if not os.path.exists(p)]) |
1871 | + |
1872 | + try: |
1873 | + with repo.pristine_tar_branches(dist, namespace): |
1874 | + oldcwd = os.getcwd() |
1875 | + # gbp does not support running from arbitrary git trees or |
1876 | + # working directories |
1877 | + # https://github.com/agx/git-buildpackage/pull/16 |
1878 | + os.chdir(repo.local_dir) |
1879 | + |
1880 | + ext = os.path.splitext(dsc.orig_tarball_path)[1] |
1881 | + |
1882 | + # make this non-fatal if upstream already exist as tagged? |
1883 | + cmd = ['gbp', 'import-orig', '--no-merge', |
1884 | + '--upstream-branch', 'do-not-push', |
1885 | + '--pristine-tar', '--no-interactive', |
1886 | + '--no-symlink-orig', |
1887 | + '--upstream-tag=%s/upstream/%s/%%(version)s%s' % |
1888 | + (namespace, dist, ext)] |
1889 | + cmd.extend(map(lambda x: '--component=%s' % x, |
1890 | + list(orig_component_paths.keys())) |
1891 | + ) |
1892 | + cmd.extend([orig_tarball_path,]) |
1893 | + run(cmd, env=repo.env) |
1894 | + except CalledProcessError as e: |
1895 | + raise GitUbuntuImportOrigError( |
1896 | + 'Unable to import tarballs: %s' % orig_paths |
1897 | + ) from e |
1898 | + finally: |
1899 | + os.chdir(oldcwd) |
1900 | |
1901 | - extract_dir = tempfile.mkdtemp() |
1902 | - os.chdir(extract_dir) |
1903 | + repo.clean_repository_state() |
1904 | |
1905 | - run(['dpkg-source', '-x', |
1906 | - '--skip-patches', |
1907 | - os.path.join(oldcwd, dsc_pathname) |
1908 | - ] |
1909 | - ) |
1910 | +def import_patches_unapplied_tree(repo, dsc_pathname): |
1911 | + """Imports the patches-unapplied source package and writes the |
1912 | + corresponding working tree for a given srcpkg |
1913 | |
1914 | - extracted_dir = None |
1915 | - for path in os.listdir(extract_dir): |
1916 | - if os.path.isdir(path): |
1917 | - extracted_dir = os.path.join(extract_dir, path) |
1918 | - break |
1919 | - if extracted_dir is None: |
1920 | - logging.exception('No source extracted?') |
1921 | - raise SourceExtractionException("Failed to find an extracted " |
1922 | - "directory from dpkg-source -x") |
1923 | - |
1924 | - if os.path.isdir(os.path.join(extracted_dir, '.pc')): |
1925 | - self.local_repo.git_run( |
1926 | - ['--work-tree', extracted_dir, 'add', '-f', '-A'] |
1927 | - ) |
1928 | - self.local_repo.git_run( |
1929 | - ['--work-tree', extracted_dir, 'rm', '-r', '-f', '.pc'] |
1930 | - ) |
1931 | - cp = self.local_repo.git_run( |
1932 | - ['--work-tree', extracted_dir, 'write-tree'] |
1933 | - ) |
1934 | - import_tree_hash = decode_binary(cp.stdout).strip() |
1935 | - yield ( |
1936 | - import_tree_hash, |
1937 | - None, |
1938 | - 'Remove .pc directory from source package', |
1939 | - ) |
1940 | + Arguments: |
1941 | + dsc_pathname - A string path to a DSC file |
1942 | + """ |
1943 | |
1944 | - try: |
1945 | - try: |
1946 | - cp = run(['dpkg-source', '--print-format', extracted_dir]) |
1947 | - fmt = decode_binary(cp.stdout).strip() |
1948 | - if '3.0 (quilt)' not in fmt: |
1949 | - raise StopIteration() |
1950 | - except CalledProcessError as e: |
1951 | - try: |
1952 | - with open('debian/source/format', 'r') as f: |
1953 | - for line in f: |
1954 | - if re.match(r'3.0 (.*)', line): |
1955 | - break |
1956 | - else: |
1957 | - raise StopIteration() |
1958 | - # `man dpkg-source` indicates no d/s/format implies 1.0 |
1959 | - except OSError: |
1960 | - raise StopIteration() |
1961 | - |
1962 | - while True: |
1963 | - try: |
1964 | - os.chdir(extracted_dir) |
1965 | - run(['quilt', 'push'], rcs=[2]) |
1966 | - cp = run(['quilt', 'top']) |
1967 | - patch_name = decode_binary(cp.stdout).strip() |
1968 | - cp = run(['quilt', 'header']) |
1969 | - header = decode_binary(cp.stdout).strip() |
1970 | - patch_desc = None |
1971 | - for regex in (r'Subject:\s*(.*?)$', |
1972 | - r'Description:\s*(.*?)$' |
1973 | - ): |
1974 | - try: |
1975 | - m = re.search(regex, header, re.MULTILINE) |
1976 | - patch_desc = m.group(1) |
1977 | - except AttributeError: |
1978 | - pass |
1979 | - if patch_desc is None: |
1980 | - logging.debug('Unable to find Subject or Description in %s' % patch_name) |
1981 | - self.local_repo.git_run(['--work-tree', extracted_dir, 'add', '-f', '-A']) |
1982 | - self.local_repo.git_run(['--work-tree', extracted_dir, 'reset', 'HEAD', '--', '.pc']) |
1983 | - cp = self.local_repo.git_run(['--work-tree', extracted_dir, 'write-tree']) |
1984 | - import_tree_hash = decode_binary(cp.stdout).strip() |
1985 | - |
1986 | - yield (import_tree_hash, patch_name, patch_desc) |
1987 | - except CalledProcessError as e: |
1988 | - # quilt returns 2 when done pushing |
1989 | - if e.returncode != 2: |
1990 | - raise |
1991 | - raise StopIteration() |
1992 | - finally: |
1993 | - os.chdir(oldcwd) |
1994 | - shutil.rmtree(extract_dir) |
1995 | - self.local_repo.clean_repository_state() |
1996 | + import_tree_hash = dsc_to_tree_hash(repo, dsc_pathname) |
1997 | + repo.clean_repository_state() |
1998 | |
1999 | - def parse_parentfile(self, parentfile): |
2000 | - """Extract parent overrides from a file |
2001 | + return import_tree_hash |
2002 | |
2003 | - The parent overrides file specifies parent-child relationship(s), |
2004 | - where the importer may not be able to determine them automatically. |
2005 | - The format of the file is: |
2006 | - <pkgname> <child version> <publish parent version> <changelog parent version> |
2007 | - with one override per line. |
2008 | +def import_patches_applied_tree(repo, dsc_pathname): |
2009 | + """Imports the patches-applied source package and writes the |
2010 | + corresponding working tree for a given srcpkg, patch by patch |
2011 | |
2012 | - <child version> is the published version to which this override |
2013 | - applies. |
2014 | + Arguments: |
2015 | + dsc_pathname - A string path to a DSC file |
2016 | + """ |
2017 | + oldcwd = os.getcwd() |
2018 | |
2019 | - The <publish parent version> is the prior version published in the |
2020 | - same series/pocket, or the immediately preceding series/pocket, if |
2021 | - this is the first publish in a series/pocket. |
2022 | + extract_dir = tempfile.mkdtemp() |
2023 | + os.chdir(extract_dir) |
2024 | |
2025 | - The <changelog parent version> is the prior version in the |
2026 | - child publish's debian/changelog. |
2027 | + run(['dpkg-source', '-x', |
2028 | + '--skip-patches', |
2029 | + os.path.join(oldcwd, dsc_pathname) |
2030 | + ] |
2031 | + ) |
2032 | |
2033 | - Keyword Arguments: |
2034 | - parentfile -- Path to parent overrides file |
2035 | - """ |
2036 | - if self.parent_overrides: |
2037 | - return |
2038 | - parent_overrides = dict() |
2039 | + extracted_dir = None |
2040 | + for path in os.listdir(extract_dir): |
2041 | + if os.path.isdir(path): |
2042 | + extracted_dir = os.path.join(extract_dir, path) |
2043 | + break |
2044 | + if extracted_dir is None: |
2045 | + logging.exception('No source extracted?') |
2046 | + raise SourceExtractionException("Failed to find an extracted " |
2047 | + "directory from dpkg-source -x") |
2048 | + |
2049 | + if os.path.isdir(os.path.join(extracted_dir, '.pc')): |
2050 | + repo.git_run( |
2051 | + ['--work-tree', extracted_dir, 'add', '-f', '-A'] |
2052 | + ) |
2053 | + repo.git_run( |
2054 | + ['--work-tree', extracted_dir, 'rm', '-r', '-f', '.pc'] |
2055 | + ) |
2056 | + cp = repo.git_run( |
2057 | + ['--work-tree', extracted_dir, 'write-tree'] |
2058 | + ) |
2059 | + import_tree_hash = decode_binary(cp.stdout).strip() |
2060 | + yield ( |
2061 | + import_tree_hash, |
2062 | + None, |
2063 | + 'Remove .pc directory from source package', |
2064 | + ) |
2065 | + |
2066 | + try: |
2067 | try: |
2068 | - with open(parentfile) as f: |
2069 | - for line in f: |
2070 | - if line.startswith('#'): |
2071 | - continue |
2072 | - m = re.match( |
2073 | - r'(?P<pkgname>\S*)\s*(?P<childversion>\S*)\s*(?P<parent1version>\S*)\s*(?P<parent2version>\S*)', |
2074 | - line |
2075 | - ) |
2076 | - if m is None: |
2077 | - continue |
2078 | - if m.group('pkgname') != self.pkgname: |
2079 | - continue |
2080 | - parent_overrides[m.group('childversion')] = { |
2081 | - 'publish_parent':m.group('parent1version'), |
2082 | - 'changelog_parent':m.group('parent2version') |
2083 | - } |
2084 | - except FileNotFoundError: |
2085 | - pass |
2086 | - self.parent_overrides = parent_overrides |
2087 | + cp = run(['dpkg-source', '--print-format', extracted_dir]) |
2088 | + fmt = decode_binary(cp.stdout).strip() |
2089 | + if '3.0 (quilt)' not in fmt: |
2090 | + raise StopIteration() |
2091 | + except CalledProcessError as e: |
2092 | + try: |
2093 | + with open('debian/source/format', 'r') as f: |
2094 | + for line in f: |
2095 | + if re.match(r'3.0 (.*)', line): |
2096 | + break |
2097 | + else: |
2098 | + raise StopIteration() |
2099 | + # `man dpkg-source` indicates no d/s/format implies 1.0 |
2100 | + except OSError: |
2101 | + raise StopIteration() |
2102 | |
2103 | - def override_parents(self, spi): |
2104 | - unapplied_publish_parent_commit = None |
2105 | - unapplied_changelog_parent_commit = None |
2106 | - applied_publish_parent_commit = None |
2107 | - applied_changelog_parent_commit = None |
2108 | + while True: |
2109 | + try: |
2110 | + os.chdir(extracted_dir) |
2111 | + run(['quilt', 'push'], rcs=[2]) |
2112 | + cp = run(['quilt', 'top']) |
2113 | + patch_name = decode_binary(cp.stdout).strip() |
2114 | + cp = run(['quilt', 'header']) |
2115 | + header = decode_binary(cp.stdout).strip() |
2116 | + patch_desc = None |
2117 | + for regex in (r'Subject:\s*(.*?)$', |
2118 | + r'Description:\s*(.*?)$' |
2119 | + ): |
2120 | + try: |
2121 | + m = re.search(regex, header, re.MULTILINE) |
2122 | + patch_desc = m.group(1) |
2123 | + except AttributeError: |
2124 | + pass |
2125 | + if patch_desc is None: |
2126 | + logging.debug('Unable to find Subject or Description ' |
2127 | + 'in %s' % patch_name |
2128 | + ) |
2129 | + repo.git_run([ |
2130 | + '--work-tree', |
2131 | + extracted_dir, |
2132 | + 'add', |
2133 | + '-f', |
2134 | + '-A', |
2135 | + ]) |
2136 | + repo.git_run([ |
2137 | + '--work-tree', |
2138 | + extracted_dir, |
2139 | + 'reset', |
2140 | + 'HEAD', |
2141 | + '--', |
2142 | + '.pc', |
2143 | + ]) |
2144 | + cp = repo.git_run(['--work-tree', extracted_dir, 'write-tree']) |
2145 | + import_tree_hash = decode_binary(cp.stdout).strip() |
2146 | + |
2147 | + yield (import_tree_hash, patch_name, patch_desc) |
2148 | + except CalledProcessError as e: |
2149 | + # quilt returns 2 when done pushing |
2150 | + if e.returncode != 2: |
2151 | + raise |
2152 | + raise StopIteration() |
2153 | + finally: |
2154 | + os.chdir(oldcwd) |
2155 | + shutil.rmtree(extract_dir) |
2156 | + repo.clean_repository_state() |
2157 | + |
2158 | +_PARENT_OVERRIDES = None |
2159 | +def parse_parentfile(parentfile, pkgname): |
2160 | + """Extract parent overrides from a file |
2161 | + |
2162 | + The parent overrides file specifies parent-child relationship(s), |
2163 | + where the importer may not be able to determine them automatically. |
2164 | + The format of the file is: |
2165 | + <pkgname> <child version> <publish parent version> <changelog parent version> |
2166 | + with one override per line. |
2167 | + |
2168 | + <child version> is the published version to which this override |
2169 | + applies. |
2170 | + |
2171 | + The <publish parent version> is the prior version published in the |
2172 | + same series/pocket, or the immediately preceding series/pocket, if |
2173 | + this is the first publish in a series/pocket. |
2174 | + |
2175 | + The <changelog parent version> is the prior version in the |
2176 | + child publish's debian/changelog. |
2177 | + |
2178 | + Keyword Arguments: |
2179 | + parentfile -- Path to parent overrides file |
2180 | + """ |
2181 | + global _PARENT_OVERRIDES |
2182 | + if _PARENT_OVERRIDES: |
2183 | + return |
2184 | + _PARENT_OVERRIDES = dict() |
2185 | + try: |
2186 | + with open(parentfile) as f: |
2187 | + for line in f: |
2188 | + if line.startswith('#'): |
2189 | + continue |
2190 | + m = re.match( |
2191 | + r'(?P<pkgname>\S*)\s*(?P<childversion>\S*)\s*(?P<parent1version>\S*)\s*(?P<parent2version>\S*)', |
2192 | + line |
2193 | + ) |
2194 | + if m is None: |
2195 | + continue |
2196 | + if m.group('pkgname') != pkgname: |
2197 | + continue |
2198 | + _PARENT_OVERRIDES[m.group('childversion')] = { |
2199 | + 'publish_parent':m.group('parent1version'), |
2200 | + 'changelog_parent':m.group('parent2version') |
2201 | + } |
2202 | + except FileNotFoundError: |
2203 | + pass |
2204 | + |
2205 | +def override_parents(repo, spi, namespace): |
2206 | + unapplied_publish_parent_commit = None |
2207 | + unapplied_changelog_parent_commit = None |
2208 | + applied_publish_parent_commit = None |
2209 | + applied_changelog_parent_commit = None |
2210 | + |
2211 | + unapplied_publish_parent_tag = repo.get_import_tag( |
2212 | + _PARENT_OVERRIDES[spi.version]['publish_parent'], |
2213 | + namespace |
2214 | + ) |
2215 | + applied_publish_parent_tag = repo.get_applied_tag( |
2216 | + _PARENT_OVERRIDES[spi.version]['publish_parent'], |
2217 | + namespace |
2218 | + ) |
2219 | + if unapplied_publish_parent_tag is not None: |
2220 | + # sanity check that version from d/changelog of the |
2221 | + # tagged commit matches ours |
2222 | + parent_publish_version, _ = \ |
2223 | + repo.get_changelog_versions_from_treeish( |
2224 | + str(unapplied_publish_parent_tag.peel().id), |
2225 | + ) |
2226 | + if parent_publish_version != _PARENT_OVERRIDES[spi.version]['publish_parent']: |
2227 | + logging.error('Found a tag corresponding to publish parent ' |
2228 | + 'override version %s, but d/changelog version (%s) ' |
2229 | + 'differs. Will orphan commit.', |
2230 | + _PARENT_OVERRIDES[spi.version]['publish_parent'], |
2231 | + parent_publish_version |
2232 | + ) |
2233 | + else: |
2234 | + unapplied_publish_parent_commit = str(unapplied_publish_parent_tag.peel().id) |
2235 | + if applied_publish_parent_tag is not None: |
2236 | + applied_publish_parent_commit = str(applied_publish_parent_tag.peel().id) |
2237 | + logging.debug('Overriding publish parent (tag) to %s', |
2238 | + repo.tag_to_pretty_name(unapplied_publish_parent_tag) |
2239 | + ) |
2240 | + else: |
2241 | + if _PARENT_OVERRIDES[spi.version]['publish_parent'] == '-': |
2242 | + logging.debug('Not setting publish parent as ' |
2243 | + 'specified in override file.' |
2244 | + ) |
2245 | + else: |
2246 | + logging.error('Specified publish parent override ' |
2247 | + '(%s) for version (%s) not found in tags. ' |
2248 | + 'Unable to proceed.' % ( |
2249 | + _PARENT_OVERRIDES[spi.version]['publish_parent'], |
2250 | + spi.version |
2251 | + ) |
2252 | + ) |
2253 | + sys.exit(1) |
2254 | |
2255 | - unapplied_publish_parent_tag = self.local_repo.get_import_tag( |
2256 | - self.parent_overrides[spi.version]['publish_parent'], |
2257 | - self.namespace) |
2258 | - applied_publish_parent_tag = self.local_repo.get_applied_tag( |
2259 | - self.parent_overrides[spi.version]['publish_parent'], |
2260 | - self.namespace) |
2261 | - if unapplied_publish_parent_tag is not None: |
2262 | - # sanity check that version from d/changelog of the |
2263 | - # tagged commit matches ours |
2264 | - parent_publish_version, _ = \ |
2265 | - self.local_repo.get_changelog_versions_from_treeish( |
2266 | - str(unapplied_publish_parent_tag.peel().id), |
2267 | - ) |
2268 | - if parent_publish_version != self.parent_overrides[spi.version]['publish_parent']: |
2269 | - logging.error('Found a tag corresponding to ' |
2270 | - 'publish parent override ' |
2271 | - 'version %s, but d/changelog ' |
2272 | - 'version (%s) differs. Will ' |
2273 | - 'orphan commit.' % ( |
2274 | - self.parent_overrides[spi.version]['publish_parent'], |
2275 | - parent_publish_version |
2276 | - ) |
2277 | - ) |
2278 | - else: |
2279 | - unapplied_publish_parent_commit = str(unapplied_publish_parent_tag.peel().id) |
2280 | - if applied_publish_parent_tag is not None: |
2281 | - applied_publish_parent_commit = str(applied_publish_parent_tag.peel().id) |
2282 | - logging.debug('Overriding publish parent (tag) to %s', |
2283 | - self.local_repo.tag_to_pretty_name(unapplied_publish_parent_tag) |
2284 | - ) |
2285 | + unapplied_changelog_parent_tag = repo.get_import_tag( |
2286 | + _PARENT_OVERRIDES[spi.version]['changelog_parent'], |
2287 | + namespace |
2288 | + ) |
2289 | + applied_changelog_parent_tag = repo.get_applied_tag( |
2290 | + _PARENT_OVERRIDES[spi.version]['changelog_parent'], |
2291 | + namespace |
2292 | + ) |
2293 | + if unapplied_changelog_parent_tag is not None: |
2294 | + # sanity check that version from d/changelog of the |
2295 | + # tagged commit matches ours |
2296 | + parent_changelog_version, _ = repo.get_changelog_versions_from_treeish( |
2297 | + str(unapplied_changelog_parent_tag.peel().id), |
2298 | + ) |
2299 | + if parent_changelog_version != _PARENT_OVERRIDES[spi.version]['changelog_parent']: |
2300 | + logging.error('Found a tag corresponding to ' |
2301 | + 'changelog parent override ' |
2302 | + 'version %s, but d/changelog ' |
2303 | + 'version (%s) differs. Will ' |
2304 | + 'orphan commit.' % ( |
2305 | + _PARENT_OVERRIDES[spi.version]['changelog_parent'], |
2306 | + parent_changelog_version |
2307 | + ) |
2308 | + ) |
2309 | else: |
2310 | - if self.parent_overrides[spi.version]['publish_parent'] == '-': |
2311 | - logging.debug('Not setting publish parent as ' |
2312 | - 'specified in override file.' |
2313 | - ) |
2314 | - else: |
2315 | - logging.error('Specified publish parent override ' |
2316 | - '(%s) for version (%s) not found in tags. ' |
2317 | - 'Unable to proceed.' % ( |
2318 | - self.parent_overrides[spi.version]['publish_parent'], |
2319 | - spi.version |
2320 | - ) |
2321 | - ) |
2322 | - sys.exit(1) |
2323 | - |
2324 | - unapplied_changelog_parent_tag = self.local_repo.get_import_tag( |
2325 | - self.parent_overrides[spi.version]['changelog_parent'], |
2326 | - self.namespace) |
2327 | - applied_changelog_parent_tag = self.local_repo.get_applied_tag( |
2328 | - self.parent_overrides[spi.version]['changelog_parent'], |
2329 | - self.namespace) |
2330 | - if unapplied_changelog_parent_tag is not None: |
2331 | - # sanity check that version from d/changelog of the |
2332 | - # tagged commit matches ours |
2333 | - parent_changelog_version, _ = \ |
2334 | - self.local_repo.get_changelog_versions_from_treeish( |
2335 | - str(unapplied_changelog_parent_tag.peel().id), |
2336 | - ) |
2337 | - if parent_changelog_version != self.parent_overrides[spi.version]['changelog_parent']: |
2338 | - logging.error('Found a tag corresponding to ' |
2339 | - 'changelog parent override ' |
2340 | - 'version %s, but d/changelog ' |
2341 | - 'version (%s) differs. Will ' |
2342 | - 'orphan commit.' % ( |
2343 | - self.parent_overrides[spi.version]['changelog_parent'], |
2344 | - parent_changelog_version |
2345 | - ) |
2346 | - ) |
2347 | - else: |
2348 | - unapplied_changelog_parent_commit = str(unapplied_changelog_parent_tag.peel().id) |
2349 | - if applied_changelog_parent_tag is not None: |
2350 | - applied_changelog_parent_commit = str(applied_changelog_parent_tag.peel().id) |
2351 | - logging.debug('Overriding changelog parent (tag) to %s', |
2352 | - self.local_repo.tag_to_pretty_name(unapplied_changelog_parent_tag) |
2353 | - ) |
2354 | + unapplied_changelog_parent_commit = str(unapplied_changelog_parent_tag.peel().id) |
2355 | + if applied_changelog_parent_tag is not None: |
2356 | + applied_changelog_parent_commit = str(applied_changelog_parent_tag.peel().id) |
2357 | + logging.debug('Overriding changelog parent (tag) to %s', |
2358 | + repo.tag_to_pretty_name(unapplied_changelog_parent_tag) |
2359 | + ) |
2360 | + else: |
2361 | + if _PARENT_OVERRIDES[spi.version]['changelog_parent'] == '-': |
2362 | + logging.debug('Not setting changelog parent as specified ' |
2363 | + 'in override file.' |
2364 | + ) |
2365 | else: |
2366 | - if self.parent_overrides[spi.version]['changelog_parent'] == '-': |
2367 | - logging.debug('Not setting changelog parent as ' |
2368 | - 'specified in override file.' |
2369 | - ) |
2370 | - else: |
2371 | - logging.error('Specified changelog parent override ' |
2372 | - '(%s) for version (%s) not found in tags. ' |
2373 | - 'Unable to proceed.' % ( |
2374 | - self.parent_overrides[spi.version]['changelog_parent'], |
2375 | - spi.version |
2376 | - ) |
2377 | - ) |
2378 | - sys.exit(1) |
2379 | - return (unapplied_publish_parent_commit, |
2380 | - unapplied_changelog_parent_commit, |
2381 | - applied_publish_parent_commit, |
2382 | - applied_changelog_parent_commit) |
2383 | - |
2384 | - def _update_devel_branches(self, namespace, spi=None): |
2385 | - if spi: |
2386 | - # we do not maintain -devel pointers for debian |
2387 | - if str(spi.distribution.name.lower()) != "ubuntu": |
2388 | + logging.error('Specified changelog parent override (%s) ' |
2389 | + 'for version (%s) not found in tags. Unable to proceed.', |
2390 | + _PARENT_OVERRIDES[spi.version]['changelog_parent'], |
2391 | + spi.version |
2392 | + ) |
2393 | + sys.exit(1) |
2394 | + return ( |
2395 | + unapplied_publish_parent_commit, |
2396 | + unapplied_changelog_parent_commit, |
2397 | + applied_publish_parent_commit, |
2398 | + applied_changelog_parent_commit, |
2399 | + ) |
2400 | + |
2401 | +def _update_devel_branches( |
2402 | + repo, |
2403 | + namespace, |
2404 | + pkgname, |
2405 | + ubuntu_sinfo, |
2406 | + spi=None, |
2407 | +): |
2408 | + if spi: |
2409 | + # we do not maintain -devel pointers for debian |
2410 | + if str(spi.distribution.name.lower()) != "ubuntu": |
2411 | + return |
2412 | + for devel_head in ( |
2413 | + '%s/ubuntu/%s-devel' % (namespace, spi.series.name.lower()), |
2414 | + '%s/ubuntu/devel' % namespace |
2415 | + ): |
2416 | + try: |
2417 | + devel_head_hash = repo.head_to_commit(devel_head) |
2418 | + devel_head_version, _ = repo.get_changelog_versions_from_treeish(devel_head_hash) |
2419 | + except AttributeError: |
2420 | + devel_head_version = None |
2421 | + if version_compare(spi.version, devel_head_version) <= 0: |
2422 | return |
2423 | - for devel_head in ( |
2424 | - '%s/ubuntu/%s-devel' % (namespace, spi.series.name.lower()), |
2425 | - '%s/ubuntu/devel' % namespace |
2426 | - ): |
2427 | - try: |
2428 | - devel_head_hash = self.local_repo.head_to_commit(devel_head) |
2429 | - devel_head_version, _ = self.local_repo.get_changelog_versions_from_treeish(devel_head_hash) |
2430 | - except AttributeError: |
2431 | - devel_head_version = None |
2432 | - if version_compare(spi.version, devel_head_version) <= 0: |
2433 | - return |
2434 | - logging.debug("Updating %s to %s" % (devel_head, |
2435 | - spi.head_name(namespace))) |
2436 | - self.local_repo.merge_commit_to_devel_head( |
2437 | - namespace, devel_head, |
2438 | - self.local_repo.head_to_commit(spi.head_name(namespace)), |
2439 | - environment_spi=spi) |
2440 | + logging.debug("Updating %s to %s" % (devel_head, |
2441 | + spi.head_name(namespace))) |
2442 | + repo.merge_commit_to_devel_head( |
2443 | + namespace, |
2444 | + devel_head, |
2445 | + repo.head_to_commit(spi.head_name(namespace)), |
2446 | + environment_spi=spi, |
2447 | + ) |
2448 | + else: |
2449 | + ubuntu_head_versions = repo.get_heads_and_versions( |
2450 | + 'ubuntu', |
2451 | + namespace, |
2452 | + ) |
2453 | + devel_hash = None |
2454 | + devel_name = None |
2455 | + for rel in ubuntu_sinfo.active_series_name_list: |
2456 | + series_devel_hash = None |
2457 | + series_devel_name = None |
2458 | + series_devel_version = None |
2459 | + # Find highest version publish in these pockets |
2460 | + for suff in ("-proposed", "-updates", "-security", ""): |
2461 | + name = "%s/ubuntu/%s%s" % (namespace, rel, suff) |
2462 | + if name in ubuntu_head_versions and version_compare( |
2463 | + ubuntu_head_versions[name]['version'], |
2464 | + series_devel_version |
2465 | + ) > 0: |
2466 | + series_devel_name = name |
2467 | + series_devel_version = ubuntu_head_versions[name]['version'] |
2468 | + series_devel_hash = str(ubuntu_head_versions[name]['head'].peel().id) |
2469 | + if series_devel_hash: |
2470 | + if not devel_hash: |
2471 | + devel_hash = series_devel_hash |
2472 | + devel_name = series_devel_name |
2473 | + logging.debug("Updating %s/ubuntu/%s-devel branch to '%s' (%s)", |
2474 | + namespace, rel, series_devel_name, series_devel_hash) |
2475 | + repo.merge_commit_to_devel_head(namespace, |
2476 | + "%s/ubuntu/%s-devel" % (namespace, rel), series_devel_hash) |
2477 | + if devel_hash is None: |
2478 | + logging.warn("Package '%s' does not appear to have been published " |
2479 | + "in Ubuntu. No %s/ubuntu/devel branch created.", |
2480 | + pkgname, namespace) |
2481 | else: |
2482 | - ubuntu_head_versions = self.local_repo.get_heads_and_versions( |
2483 | - 'ubuntu', namespace |
2484 | + logging.debug("Setting %s/ubuntu/devel branch to '%s' (%s)", |
2485 | + namespace, devel_name, devel_hash) |
2486 | + repo.merge_commit_to_devel_head( |
2487 | + namespace, "%s/ubuntu/devel" % namespace, devel_hash |
2488 | ) |
2489 | - devel_hash = None |
2490 | - devel_name = None |
2491 | - for rel in self.ubuntu_sinfo.active_series_name_list: |
2492 | - series_devel_hash = None |
2493 | - series_devel_name = None |
2494 | - series_devel_version = None |
2495 | - # Find highest version publish in these pockets |
2496 | - for suff in ("-proposed", "-updates", "-security", ""): |
2497 | - name = "%s/ubuntu/%s%s" % (namespace, rel, suff) |
2498 | - if name in ubuntu_head_versions and version_compare( |
2499 | - ubuntu_head_versions[name]['version'], |
2500 | - series_devel_version |
2501 | - ) > 0: |
2502 | - series_devel_name = name |
2503 | - series_devel_version = ubuntu_head_versions[name]['version'] |
2504 | - series_devel_hash = str(ubuntu_head_versions[name]['head'].peel().id) |
2505 | - if series_devel_hash: |
2506 | - if not devel_hash: |
2507 | - devel_hash = series_devel_hash |
2508 | - devel_name = series_devel_name |
2509 | - logging.debug("Updating %s/ubuntu/%s-devel branch to '%s' (%s)", |
2510 | - namespace, rel, series_devel_name, series_devel_hash) |
2511 | - self.local_repo.merge_commit_to_devel_head(namespace, |
2512 | - "%s/ubuntu/%s-devel" % (namespace, rel), series_devel_hash) |
2513 | - if devel_hash is None: |
2514 | - logging.warn("Package '%s' does not appear to have been published " |
2515 | - "in Ubuntu. No %s/ubuntu/devel branch created.", |
2516 | - self.pkgname, namespace) |
2517 | - else: |
2518 | - logging.debug("Setting %s/ubuntu/devel branch to '%s' (%s)", |
2519 | - namespace, devel_name, devel_hash) |
2520 | - self.local_repo.merge_commit_to_devel_head( |
2521 | - namespace, "%s/ubuntu/devel" % namespace, devel_hash |
2522 | - ) |
2523 | - |
2524 | - def update_devel_branches(self, spi=None): |
2525 | - self._update_devel_branches(self.namespace, spi) |
2526 | - |
2527 | - def update_applied_devel_branches(self, spi=None): |
2528 | - self._update_devel_branches('%s/applied' % self.namespace, spi) |
2529 | - |
2530 | - # imports a package based upon source package information |
2531 | - def import_unapplied_spi(self, spi): |
2532 | - """Imports a source package from Launchpad into the git |
2533 | - repository |
2534 | - |
2535 | - Arguments: |
2536 | - spi - a SourcePackageInformation instance |
2537 | - """ |
2538 | - pretty_head_name = spi.pretty_head_name |
2539 | |
2540 | - logging.info('Importing patches-unapplied %s to %s' % |
2541 | - (spi.version, pretty_head_name)) |
2542 | - unapplied_tip_head = self.local_repo.get_head_by_name(spi.head_name(self.namespace)) |
2543 | - |
2544 | - spi.pull() |
2545 | - |
2546 | - self.local_repo.clean_repository_state() |
2547 | - |
2548 | - self.import_dsc( |
2549 | +def update_devel_branches( |
2550 | + repo, |
2551 | + namespace, |
2552 | + pkgname, |
2553 | + ubuntu_sinfo, |
2554 | + spi=None, |
2555 | +): |
2556 | + _update_devel_branches( |
2557 | + repo=repo, |
2558 | + namespace=namespace, |
2559 | + pkgname=pkgname, |
2560 | + ubuntu_sinfo=ubuntu_sinfo, |
2561 | + spi=spi, |
2562 | + ) |
2563 | + |
2564 | +def update_applied_devel_branches( |
2565 | + repo, |
2566 | + namespace, |
2567 | + pkgname, |
2568 | + ubuntu_sinfo, |
2569 | + spi=None, |
2570 | +): |
2571 | + _update_devel_branches( |
2572 | + repo=repo, |
2573 | + namespace='%s/applied' % namespace, |
2574 | + pkgname=pkgname, |
2575 | + ubuntu_sinfo=ubuntu_sinfo, |
2576 | + spi=spi, |
2577 | + ) |
2578 | + |
2579 | +# imports a package based upon source package information |
2580 | +def import_unapplied_spi(repo, spi, namespace, skip_orig, ubuntu_sinfo): |
2581 | + """Imports a source package from Launchpad into the git |
2582 | + repository |
2583 | + |
2584 | + Arguments: |
2585 | + spi - a SourcePackageInformation instance |
2586 | + """ |
2587 | + pretty_head_name = spi.pretty_head_name |
2588 | + |
2589 | + logging.info('Importing patches-unapplied %s to %s', |
2590 | + spi.version, pretty_head_name |
2591 | + ) |
2592 | + unapplied_tip_head = repo.get_head_by_name(spi.head_name(namespace)) |
2593 | + |
2594 | + spi.pull() |
2595 | + |
2596 | + repo.clean_repository_state() |
2597 | + |
2598 | + import_dsc( |
2599 | + repo, |
2600 | + GitUbuntuDsc(spi.dsc_pathname), |
2601 | + namespace, |
2602 | + str(spi.version), |
2603 | + spi.distribution.name.lower(), |
2604 | + ) |
2605 | + if not skip_orig: |
2606 | + import_orig( |
2607 | + repo, |
2608 | GitUbuntuDsc(spi.dsc_pathname), |
2609 | + namespace, |
2610 | str(spi.version), |
2611 | - spi.distribution.name.lower() |
2612 | + spi.distribution.name.lower(), |
2613 | ) |
2614 | - if not self.skip_orig: |
2615 | - self.import_orig( |
2616 | - GitUbuntuDsc(spi.dsc_pathname), |
2617 | - version = str(spi.version), |
2618 | - dist = spi.distribution.name.lower(), |
2619 | - ) |
2620 | |
2621 | - unapplied_import_tree_hash = self.import_patches_unapplied_tree(spi.dsc_pathname) |
2622 | - logging.debug('Imported patches-unapplied version %s as tree %s', |
2623 | - spi.version, unapplied_import_tree_hash |
2624 | - ) |
2625 | + unapplied_import_tree_hash = import_patches_unapplied_tree( |
2626 | + repo, |
2627 | + spi.dsc_pathname |
2628 | + ) |
2629 | + logging.debug( |
2630 | + "Imported patches-unapplied version %s as tree %s", |
2631 | + spi.version, |
2632 | + unapplied_import_tree_hash |
2633 | + ) |
2634 | + |
2635 | + import_tree_versions = repo.get_all_changelog_versions_from_treeish( |
2636 | + unapplied_import_tree_hash |
2637 | + ) |
2638 | + changelog_version = import_tree_versions[0] |
2639 | + |
2640 | + logging.debug( |
2641 | + "Found changelog version %s in newly imported tree" % |
2642 | + changelog_version |
2643 | + ) |
2644 | + |
2645 | + # sanity check on spph and d/changelog agreeing on the latest version |
2646 | + if spi.version not in _PARENT_OVERRIDES: |
2647 | + assert spi.version == changelog_version, ( |
2648 | + 'source pkg version: {} != changelog version: {}'.format( |
2649 | + spi.version, changelog_version)) |
2650 | + |
2651 | + unapplied_tip_version = None |
2652 | + # check if the version to import has already been imported to |
2653 | + # this head |
2654 | + if unapplied_tip_head is not None: |
2655 | + if repo.treeishs_identical( |
2656 | + unapplied_import_tree_hash, str(unapplied_tip_head.peel().id) |
2657 | + ): |
2658 | + logging.warn('%s is identical to %s', |
2659 | + pretty_head_name, spi.version |
2660 | + ) |
2661 | + return |
2662 | + unapplied_tip_version, _ = repo.get_changelog_versions_from_treeish( |
2663 | + str(unapplied_tip_head.peel().id), |
2664 | + ) |
2665 | + |
2666 | + logging.debug('Tip version is %s', unapplied_tip_version) |
2667 | |
2668 | - import_tree_versions = self.local_repo.get_all_changelog_versions_from_treeish( |
2669 | - unapplied_import_tree_hash |
2670 | - ) |
2671 | - changelog_version = import_tree_versions[0] |
2672 | + unapplied_publish_parent_commit = None |
2673 | + unapplied_changelog_parent_commit = None |
2674 | + upload_parent_commit = None |
2675 | + unapplied_parent_commit = None |
2676 | |
2677 | + if spi.version in _PARENT_OVERRIDES: |
2678 | logging.debug( |
2679 | - 'Found changelog version %s in newly imported tree' % |
2680 | - changelog_version |
2681 | - ) |
2682 | + '%s is specified in the parent override file.', |
2683 | + spi.version |
2684 | + ) |
2685 | |
2686 | - # sanity check on spph and d/changelog agreeing on the latest version |
2687 | - if spi.version not in self.parent_overrides: |
2688 | - assert spi.version == changelog_version, ( |
2689 | - 'source pkg version: {} != changelog version: {}'.format( |
2690 | - spi.version, changelog_version)) |
2691 | - |
2692 | - unapplied_tip_version = None |
2693 | - # check if the version to import has already been imported to |
2694 | - # this head |
2695 | - if unapplied_tip_head is not None: |
2696 | - if self.local_repo.treeishs_identical( |
2697 | - unapplied_import_tree_hash, str(unapplied_tip_head.peel().id) |
2698 | - ): |
2699 | - logging.warn('%s is identical to %s', |
2700 | - pretty_head_name, spi.version |
2701 | + ( |
2702 | + unapplied_publish_parent_commit, |
2703 | + unapplied_changelog_parent_commit, |
2704 | + _, |
2705 | + _ |
2706 | + ) = override_parents(repo, spi, namespace) |
2707 | + else: |
2708 | + # Get parent from publishing history (which is the last |
2709 | + # published version in the corresponding series) |
2710 | + try: |
2711 | + unapplied_publish_parent_commit = str(unapplied_tip_head.peel().id) |
2712 | + except AttributeError: |
2713 | + try: |
2714 | + unapplied_publish_parent_head = repo.get_head_by_name(spi.parent_head_name(namespace)) |
2715 | + unapplied_publish_parent_commit = str(unapplied_publish_parent_head.peel().id) |
2716 | + except AttributeError: |
2717 | + pass |
2718 | + finally: |
2719 | + try: |
2720 | + unapplied_publish_parent_tag = repo.nearest_import_tag(unapplied_publish_parent_commit, namespace) |
2721 | + logging.debug('Publishing parent (tag) is %s', |
2722 | + repo.tag_to_pretty_name(unapplied_publish_parent_tag) |
2723 | ) |
2724 | - return |
2725 | - unapplied_tip_version, _ = self.local_repo.get_changelog_versions_from_treeish( |
2726 | - str(unapplied_tip_head.peel().id), |
2727 | - ) |
2728 | + except AttributeError: |
2729 | + pass |
2730 | + |
2731 | + if version_compare(str(spi.version), unapplied_tip_version) <= 0: |
2732 | + logging.warn('Version to import (%s) is not after %s tip (%s)', |
2733 | + spi.version, pretty_head_name, unapplied_tip_version |
2734 | + ) |
2735 | |
2736 | - logging.debug('Tip version is %s', unapplied_tip_version) |
2737 | + # Walk changelog backwards until we find an imported version |
2738 | + for version in import_tree_versions: |
2739 | + unapplied_changelog_parent_tag = repo.get_import_tag(version, namespace) |
2740 | + if unapplied_changelog_parent_tag is not None: |
2741 | + # sanity check that version from d/changelog of the |
2742 | + # tagged parent matches ours |
2743 | + parent_changelog_version, _ = \ |
2744 | + repo.get_changelog_versions_from_treeish( |
2745 | + str(unapplied_changelog_parent_tag.peel(pygit2.Tree).id), |
2746 | + ) |
2747 | + # if the parent was imported via a parent override |
2748 | + # itself, assume it is valid without checking its |
2749 | + # tree's debian/changelog, as it wasn't used |
2750 | + if (version not in _PARENT_OVERRIDES and |
2751 | + parent_changelog_version != version |
2752 | + ): |
2753 | + logging.error('Found a tag corresponding to ' |
2754 | + 'parent version %s, but d/changelog ' |
2755 | + 'version (%s) differs. Will ' |
2756 | + 'orphan commit.' % ( |
2757 | + version, |
2758 | + parent_changelog_version |
2759 | + ) |
2760 | + ) |
2761 | + else: |
2762 | + unapplied_changelog_parent_commit = str(unapplied_changelog_parent_tag.peel().id) |
2763 | + logging.debug('Changelog parent (tag) is %s', |
2764 | + repo.tag_to_pretty_name(unapplied_changelog_parent_tag) |
2765 | + ) |
2766 | + break |
2767 | |
2768 | - unapplied_publish_parent_commit = None |
2769 | + # If the two parents are tree-identical, then favor publication |
2770 | + # history. This deals with copy-forwards between series and with |
2771 | + # syncs. |
2772 | + if repo.treeishs_identical(unapplied_publish_parent_commit, unapplied_changelog_parent_commit): |
2773 | unapplied_changelog_parent_commit = None |
2774 | - upload_parent_commit = None |
2775 | - unapplied_parent_commit = None |
2776 | |
2777 | - if spi.version in self.parent_overrides: |
2778 | - logging.debug('%s is specified in the parent override file.' % |
2779 | + # check if the version to import has already been uploaded to |
2780 | + # this head -- we leverage the above code to perform a 'dry-run' |
2781 | + # of the import tree, to obtain the (up to) two parents |
2782 | + upload_tag = repo.get_upload_tag(spi.version, namespace) |
2783 | + import_tag = repo.get_import_tag(spi.version, namespace) |
2784 | + if upload_tag and not import_tag: |
2785 | + if str(upload_tag.peel().tree.id) != unapplied_import_tree_hash: |
2786 | + logging.warn('Found upload tag %s, ' |
2787 | + 'but the corresponding tree ' |
2788 | + 'does not match the published ' |
2789 | + 'version. Will import %s as ' |
2790 | + 'normal, ignoring the upload tag.', |
2791 | + repo.tag_to_pretty_name(upload_tag), |
2792 | spi.version |
2793 | ) |
2794 | - |
2795 | - (unapplied_publish_parent_commit, |
2796 | - unapplied_changelog_parent_commit, |
2797 | - _, _) = self.override_parents(spi) |
2798 | else: |
2799 | - # Get parent from publishing history (which is the last |
2800 | - # published version in the corresponding series) |
2801 | - try: |
2802 | - unapplied_publish_parent_commit = str(unapplied_tip_head.peel().id) |
2803 | - except AttributeError: |
2804 | + upload_parent_commit = str(upload_tag.peel().id) |
2805 | + if unapplied_publish_parent_commit is not None: |
2806 | try: |
2807 | - unapplied_publish_parent_head = self.local_repo.get_head_by_name(spi.parent_head_name(self.namespace)) |
2808 | - unapplied_publish_parent_commit = str(unapplied_publish_parent_head.peel().id) |
2809 | - except AttributeError: |
2810 | - pass |
2811 | - finally: |
2812 | - try: |
2813 | - unapplied_publish_parent_tag = self.local_repo.nearest_import_tag(unapplied_publish_parent_commit, self.namespace) |
2814 | - logging.debug('Publishing parent (tag) is %s', |
2815 | - self.local_repo.tag_to_pretty_name(unapplied_publish_parent_tag) |
2816 | + repo.git_run(['merge-base', '--is-ancestor', |
2817 | + unapplied_publish_parent_commit, |
2818 | + upload_parent_commit |
2819 | + ] |
2820 | ) |
2821 | - except AttributeError: |
2822 | - pass |
2823 | - |
2824 | - if version_compare(str(spi.version), unapplied_tip_version) <= 0: |
2825 | - logging.warn('Version to import (%s) is not after %s tip (%s)', |
2826 | - spi.version, pretty_head_name, unapplied_tip_version |
2827 | - ) |
2828 | - |
2829 | - # Walk changelog backwards until we find an imported version |
2830 | - for version in import_tree_versions: |
2831 | - unapplied_changelog_parent_tag = self.local_repo.get_import_tag(version, self.namespace) |
2832 | - if unapplied_changelog_parent_tag is not None: |
2833 | - # sanity check that version from d/changelog of the |
2834 | - # tagged parent matches ours |
2835 | - parent_changelog_version, _ = \ |
2836 | - self.local_repo.get_changelog_versions_from_treeish( |
2837 | - str(unapplied_changelog_parent_tag.peel(pygit2.Tree).id), |
2838 | - ) |
2839 | - # if the parent was imported via a parent override |
2840 | - # itself, assume it is valid without checking its |
2841 | - # tree's debian/changelog, as it wasn't used |
2842 | - if (version not in self.parent_overrides and |
2843 | - parent_changelog_version != version |
2844 | - ): |
2845 | - logging.error('Found a tag corresponding to ' |
2846 | - 'parent version %s, but d/changelog ' |
2847 | - 'version (%s) differs. Will ' |
2848 | - 'orphan commit.' % ( |
2849 | - version, |
2850 | - parent_changelog_version |
2851 | - ) |
2852 | - ) |
2853 | - else: |
2854 | - unapplied_changelog_parent_commit = str(unapplied_changelog_parent_tag.peel().id) |
2855 | - logging.debug('Changelog parent (tag) is %s', |
2856 | - self.local_repo.tag_to_pretty_name(unapplied_changelog_parent_tag) |
2857 | - ) |
2858 | - break |
2859 | - |
2860 | - # If the two parents are tree-identical, then favor publication |
2861 | - # history. This deals with copy-forwards between series and with |
2862 | - # syncs. |
2863 | - if self.local_repo.treeishs_identical(unapplied_publish_parent_commit, unapplied_changelog_parent_commit): |
2864 | - unapplied_changelog_parent_commit = None |
2865 | - |
2866 | - # check if the version to import has already been uploaded to |
2867 | - # this head -- we leverage the above code to perform a 'dry-run' |
2868 | - # of the import tree, to obtain the (up to) two parents |
2869 | - upload_tag = self.local_repo.get_upload_tag(spi.version, self.namespace) |
2870 | - import_tag = self.local_repo.get_import_tag(spi.version, self.namespace) |
2871 | - if upload_tag and not import_tag: |
2872 | - if str(upload_tag.peel().tree.id) != unapplied_import_tree_hash: |
2873 | - logging.warn('Found upload tag %s, ' |
2874 | - 'but the corresponding tree ' |
2875 | - 'does not match the published ' |
2876 | - 'version. Will import %s as ' |
2877 | - 'normal, ignoring the upload tag.', |
2878 | - self.local_repo.tag_to_pretty_name(upload_tag), |
2879 | - spi.version |
2880 | - ) |
2881 | - else: |
2882 | - upload_parent_commit = str(upload_tag.peel().id) |
2883 | - if unapplied_publish_parent_commit is not None: |
2884 | - try: |
2885 | - self.local_repo.git_run(['merge-base', '--is-ancestor', |
2886 | - unapplied_publish_parent_commit, |
2887 | - upload_parent_commit |
2888 | - ] |
2889 | - ) |
2890 | - unapplied_publish_parent_commit = None |
2891 | - except CalledProcessError as e: |
2892 | - if e.returncode != 1: |
2893 | - raise |
2894 | - |
2895 | - if unapplied_changelog_parent_commit is not None: |
2896 | - try: |
2897 | - self.local_repo.git_run(['merge-base', '--is-ancestor', |
2898 | - unapplied_changelog_parent_commit, |
2899 | - upload_parent_commit |
2900 | - ] |
2901 | - ) |
2902 | - unapplied_changelog_parent_commit = None |
2903 | - except CalledProcessError as e: |
2904 | - if e.returncode != 1: |
2905 | - raise |
2906 | - |
2907 | - self.commit_unapplied_patches_import( |
2908 | - spi, |
2909 | - unapplied_import_tree_hash, |
2910 | - unapplied_publish_parent_commit, |
2911 | - unapplied_changelog_parent_commit, |
2912 | - upload_parent_commit, |
2913 | - ) |
2914 | - |
2915 | - self.update_devel_branches(spi) |
2916 | - |
2917 | - def import_applied_spi(self, spi): |
2918 | - """Imports a source package from Launchpad into the git |
2919 | - repository |
2920 | - |
2921 | - Arguments: |
2922 | - spi - a SourcePackageInformation instance |
2923 | - """ |
2924 | - pretty_head_name = spi.pretty_head_name |
2925 | - |
2926 | - logging.info('Importing patches-applied %s to %s' % (spi.version, pretty_head_name)) |
2927 | - applied_tip_head = self.local_repo.get_head_by_name(spi.applied_head_name(self.namespace)) |
2928 | - |
2929 | - spi.pull() |
2930 | - |
2931 | - self.local_repo.clean_repository_state() |
2932 | - |
2933 | - unapplied_parent_tag = self.local_repo.get_import_tag(spi.version, self.namespace) |
2934 | - unapplied_parent_commit = str(unapplied_parent_tag.peel().id) |
2935 | - unapplied_import_tree_hash = str(unapplied_parent_tag.peel(pygit2.Tree).id) |
2936 | - logging.debug('Found patches-unapplied version %s as commit %s', |
2937 | - str(spi.version), unapplied_parent_commit) |
2938 | - |
2939 | - import_tree_versions = self.local_repo.get_all_changelog_versions_from_treeish( |
2940 | - unapplied_import_tree_hash |
2941 | - ) |
2942 | - changelog_version = import_tree_versions[0] |
2943 | - |
2944 | - applied_tip_version = None |
2945 | - # check if the version to import has already been imported to |
2946 | - # this head |
2947 | - if applied_tip_head is not None: |
2948 | - if self.local_repo.treeishs_identical( |
2949 | - unapplied_import_tree_hash, str(applied_tip_head.peel().id) |
2950 | - ): |
2951 | - logging.warn('%s is identical to %s', |
2952 | - pretty_head_name, spi.version |
2953 | - ) |
2954 | - return |
2955 | - applied_tip_version, _ = self.local_repo.get_changelog_versions_from_treeish( |
2956 | - str(applied_tip_head.peel().id), |
2957 | - ) |
2958 | - |
2959 | - logging.debug('Tip version is %s', applied_tip_version) |
2960 | + unapplied_publish_parent_commit = None |
2961 | + except CalledProcessError as e: |
2962 | + if e.returncode != 1: |
2963 | + raise |
2964 | |
2965 | - applied_publish_parent_commit = None |
2966 | - applied_changelog_parent_commit = None |
2967 | + if unapplied_changelog_parent_commit is not None: |
2968 | + try: |
2969 | + repo.git_run(['merge-base', '--is-ancestor', |
2970 | + unapplied_changelog_parent_commit, |
2971 | + upload_parent_commit |
2972 | + ] |
2973 | + ) |
2974 | + unapplied_changelog_parent_commit = None |
2975 | + except CalledProcessError as e: |
2976 | + if e.returncode != 1: |
2977 | + raise |
2978 | |
2979 | - if spi.version in self.parent_overrides: |
2980 | - logging.debug('%s is specified in the parent override file.' % |
2981 | - spi.version |
2982 | + commit_unapplied_patches_import( |
2983 | + repo, |
2984 | + spi, |
2985 | + unapplied_import_tree_hash, |
2986 | + namespace, |
2987 | + unapplied_publish_parent_commit, |
2988 | + unapplied_changelog_parent_commit, |
2989 | + upload_parent_commit, |
2990 | + ) |
2991 | + |
2992 | + update_devel_branches( |
2993 | + repo=repo, |
2994 | + namespace=namespace, |
2995 | + pkgname=spi.name, |
2996 | + ubuntu_sinfo=ubuntu_sinfo, |
2997 | + spi=spi, |
2998 | + ) |
2999 | + |
3000 | +def import_applied_spi(repo, spi, namespace, ubuntu_sinfo): |
3001 | + """Imports a source package from Launchpad into the git |
3002 | + repository |
3003 | + |
3004 | + Arguments: |
3005 | + spi - a SourcePackageInformation instance |
3006 | + """ |
3007 | + pretty_head_name = spi.pretty_head_name |
3008 | + |
3009 | + logging.info('Importing patches-applied %s to %s' % (spi.version, pretty_head_name)) |
3010 | + applied_tip_head = repo.get_head_by_name(spi.applied_head_name(namespace)) |
3011 | + |
3012 | + spi.pull() |
3013 | + |
3014 | + repo.clean_repository_state() |
3015 | + |
3016 | + unapplied_parent_tag = repo.get_import_tag(spi.version, namespace) |
3017 | + unapplied_parent_commit = str(unapplied_parent_tag.peel().id) |
3018 | + unapplied_import_tree_hash = str(unapplied_parent_tag.peel(pygit2.Tree).id) |
3019 | + logging.debug('Found patches-unapplied version %s as commit %s', |
3020 | + str(spi.version), unapplied_parent_commit) |
3021 | + |
3022 | + import_tree_versions = repo.get_all_changelog_versions_from_treeish( |
3023 | + unapplied_import_tree_hash |
3024 | + ) |
3025 | + changelog_version = import_tree_versions[0] |
3026 | + |
3027 | + applied_tip_version = None |
3028 | + # check if the version to import has already been imported to |
3029 | + # this head |
3030 | + if applied_tip_head is not None: |
3031 | + if repo.treeishs_identical( |
3032 | + unapplied_import_tree_hash, str(applied_tip_head.peel().id) |
3033 | + ): |
3034 | + logging.warn('%s is identical to %s', |
3035 | + pretty_head_name, spi.version |
3036 | ) |
3037 | + return |
3038 | + applied_tip_version, _ = repo.get_changelog_versions_from_treeish( |
3039 | + str(applied_tip_head.peel().id), |
3040 | + ) |
3041 | |
3042 | - (_, _, |
3043 | - applied_publish_parent_commit, |
3044 | - applied_changelog_parent_commit) = self.override_parents(spi) |
3045 | - else: |
3046 | - # Get parent from publishing history (which is the last |
3047 | - # published version in the corresponding series) |
3048 | - try: |
3049 | - applied_publish_parent_commit = str(applied_tip_head.peel().id) |
3050 | - except AttributeError: |
3051 | - try: |
3052 | - applied_publish_parent_head = self.local_repo.get_head_by_name(spi.applied_parent_head_name(self.namespace)) |
3053 | - applied_publish_parent_head_commit = str(applied_publish_parent_head.peel().id) |
3054 | - except AttributeError: |
3055 | - pass |
3056 | - |
3057 | - if version_compare(str(spi.version), applied_tip_version) <= 0: |
3058 | - logging.warn('Version to import (%s) is not after %s tip (%s)', |
3059 | - spi.version, pretty_head_name, applied_tip_version |
3060 | - ) |
3061 | + logging.debug('Tip version is %s', applied_tip_version) |
3062 | |
3063 | - # Walk changelog backwards until we find an imported version |
3064 | - for version in import_tree_versions: |
3065 | - applied_changelog_parent_tag = self.local_repo.get_applied_tag(version, self.namespace) |
3066 | - if applied_changelog_parent_tag is not None: |
3067 | - # sanity check that version from d/changelog of the |
3068 | - # tagged parent matches ours |
3069 | - parent_changelog_version, _ = \ |
3070 | - self.local_repo.get_changelog_versions_from_treeish( |
3071 | - str(applied_changelog_parent_tag.peel(pygit2.Tree).id), |
3072 | - ) |
3073 | - # if the parent was imported via a parent override |
3074 | - # itself, assume it is valid without checking its |
3075 | - # tree's debian/changelog, as it wasn't used |
3076 | - if (version not in self.parent_overrides and |
3077 | - parent_changelog_version != version |
3078 | - ): |
3079 | - logging.error('Found a tag corresponding to ' |
3080 | - 'parent version %s, but d/changelog ' |
3081 | - 'version (%s) differs. Will ' |
3082 | - 'orphan commit.' % ( |
3083 | - version, |
3084 | - parent_changelog_version |
3085 | - ) |
3086 | - ) |
3087 | - else: |
3088 | - applied_changelog_parent_commit = str(applied_changelog_parent_tag.peel().id) |
3089 | - logging.debug('Changelog parent (tag) is %s', |
3090 | - self.local_repo.tag_to_pretty_name(applied_changelog_parent_tag) |
3091 | - ) |
3092 | - break |
3093 | - |
3094 | - # If the two parents are tree-identical, then favor publication |
3095 | - # history. This deals with copy-forwards between series and with |
3096 | - # syncs. |
3097 | - if self.local_repo.treeishs_identical(applied_publish_parent_commit, applied_changelog_parent_commit): |
3098 | - applied_changelog_parent_commit = None |
3099 | - |
3100 | - # Assume no patches to apply |
3101 | - applied_import_tree_hash = unapplied_import_tree_hash |
3102 | - # get tree id from above commit |
3103 | - for (applied_import_tree_hash, patch_name, patch_desc) in self.import_patches_applied_tree(spi.dsc_pathname): |
3104 | - # special case for .pc removal |
3105 | - if patch_name is None: |
3106 | - msg = b'%b.' % (patch_desc.encode()) |
3107 | - else: |
3108 | - if patch_desc is None: |
3109 | - patch_desc = '%s\n\nNo DEP3 Subject or Description header found' % patch_name |
3110 | - msg = b'%b\n\nGbp-Pq: %b.' % (patch_desc.encode(), patch_name.encode()) |
3111 | - unapplied_parent_commit = self.local_repo.commit_tree_hash( |
3112 | - applied_import_tree_hash, |
3113 | - [unapplied_parent_commit], |
3114 | - msg, |
3115 | - spi) |
3116 | - |
3117 | - logging.debug('Committed patch-application of %s as %s', |
3118 | - patch_name, unapplied_parent_commit |
3119 | - ) |
3120 | + applied_publish_parent_commit = None |
3121 | + applied_changelog_parent_commit = None |
3122 | |
3123 | - self.commit_applied_patches_import( |
3124 | - spi, |
3125 | - applied_import_tree_hash, |
3126 | + if spi.version in _PARENT_OVERRIDES: |
3127 | + logging.debug('%s is specified in the parent override file.' % |
3128 | + spi.version |
3129 | + ) |
3130 | + |
3131 | + ( |
3132 | + _, |
3133 | + _, |
3134 | applied_publish_parent_commit, |
3135 | applied_changelog_parent_commit, |
3136 | - unapplied_parent_commit |
3137 | - ) |
3138 | - |
3139 | - self.update_applied_devel_branches(spi) |
3140 | - |
3141 | - def import_publishes(self, patches_applied, |
3142 | - debian_head_versions, ubuntu_head_versions): |
3143 | - history_found = False |
3144 | - only_debian = False |
3145 | - srcpkg_information = None |
3146 | - if patches_applied: |
3147 | - namespace = '%s/applied' % self.namespace |
3148 | - import_type = 'patches-applied' |
3149 | - import_func = self.import_applied_spi |
3150 | - else: |
3151 | - namespace = self.namespace |
3152 | - import_type = 'patches-unapplied' |
3153 | - import_func = self.import_unapplied_spi |
3154 | - for distname, versions, dist_sinfo in ( |
3155 | - ("debian", debian_head_versions, self.debian_sinfo), |
3156 | - ("ubuntu", ubuntu_head_versions, self.ubuntu_sinfo)): |
3157 | - if self.active_series_only and distname == "debian": |
3158 | - continue |
3159 | - try: |
3160 | - for srcpkg_information in dist_sinfo.launchpad_versions_published_after( |
3161 | - versions, |
3162 | - namespace, |
3163 | - workdir=self.workdir, |
3164 | - active_series_only=self.active_series_only |
3165 | - ): |
3166 | - history_found = True |
3167 | - import_func(srcpkg_information) |
3168 | - # Do not push upstream |
3169 | - if self.fixup_devel and distname == "ubuntu": |
3170 | - self.update_devel_branches(spi=None) |
3171 | - except NoPublicationHistoryException: |
3172 | - logging.warning("No publication history found for %s in %s. ", |
3173 | - self.pkgname, distname) |
3174 | - if distname == 'ubuntu': |
3175 | - only_debian = True |
3176 | - except Exception as e: |
3177 | - if srcpkg_information is None: |
3178 | - msg = 'Unable to import %s to %s' % (import_type, distname) |
3179 | - else: |
3180 | - msg = 'Unable to import %s %s to %s' % (import_type, |
3181 | - str(srcpkg_information.version), distname) |
3182 | - if not patches_applied: |
3183 | - raise GitUbuntuImportError(msg) from e |
3184 | - else: |
3185 | - logging.error(msg) |
3186 | - else: |
3187 | - history_found = True |
3188 | - |
3189 | - return (only_debian, history_found) |
3190 | - |
3191 | - def main(self, args): |
3192 | - self.pkgname = args.package |
3193 | - owner = args.lp_owner |
3194 | - no_clean = args.no_clean |
3195 | - try: |
3196 | - directory = args.directory |
3197 | - no_clean = True |
3198 | - except AttributeError: |
3199 | - directory = None |
3200 | - try: |
3201 | - user = args.lp_user |
3202 | - except AttributeError: |
3203 | - user = None |
3204 | - # --active-series-only/--no-fetch to a tmpdir/--skip-orig |
3205 | - # implies --no-push |
3206 | - no_push = ( |
3207 | - args.no_push or |
3208 | - (args.no_fetch and not directory) or |
3209 | - args.active_series_only or |
3210 | - args.skip_orig |
3211 | - ) |
3212 | + ) = override_parents( |
3213 | + repo, |
3214 | + spi, |
3215 | + namespace, |
3216 | + ) |
3217 | + else: |
3218 | + # Get parent from publishing history (which is the last |
3219 | + # published version in the corresponding series) |
3220 | try: |
3221 | - dl_cache = args.dl_cache |
3222 | + applied_publish_parent_commit = str(applied_tip_head.peel().id) |
3223 | except AttributeError: |
3224 | - dl_cache = None |
3225 | - self.fixup_devel = args.fixup_devel |
3226 | - self.active_series_only = args.active_series_only |
3227 | - self.skip_orig = args.skip_orig |
3228 | - |
3229 | - if owner == 'usd-import-team': |
3230 | - self.namespace = 'importer' |
3231 | - else: |
3232 | - self.namespace = owner |
3233 | - |
3234 | - logging.info('Ubuntu Server Team importer v%s' % VERSION) |
3235 | - |
3236 | - self.local_repo = GitUbuntuRepository(directory, user, args.proto) |
3237 | - if not directory: |
3238 | - logging.info('Using git repository at %s', self.local_repo.local_dir) |
3239 | - |
3240 | - atexit.register(self.cleanup, no_clean, self.local_repo.local_dir) |
3241 | - |
3242 | - self.local_repo.add_base_remotes(self.pkgname, repo_owner=owner) |
3243 | - if not args.no_fetch and not args.reimport: |
3244 | - try: |
3245 | - self.local_repo.fetch_base_remotes() |
3246 | - except GitUbuntuRepositoryFetchError: |
3247 | - pass |
3248 | - |
3249 | - if not args.no_fetch: |
3250 | - self.local_repo.delete_branches_in_namespace(self.namespace) |
3251 | - self.local_repo.delete_tags_in_namespace(self.namespace) |
3252 | - self.local_repo.copy_base_references(self.namespace) |
3253 | - |
3254 | - if not args.no_fetch and not args.reimport: |
3255 | - try: |
3256 | - self.local_repo.fetch_remote_refspecs('pkg', |
3257 | - refspecs=['refs/tags/*:refs/tags/%s/*' % self.namespace], |
3258 | - ) |
3259 | - except GitUbuntuRepositoryFetchError: |
3260 | - pass |
3261 | - |
3262 | - if args.reimport: |
3263 | - if args.no_fetch: |
3264 | - logging.warning( |
3265 | - "--reimport specified with --no-fetch. Upload " |
3266 | - "tags would not be incorporated into the new import " |
3267 | - "and would be lost forever. This is not currently " |
3268 | - "supported and --no-fetch will be ignored for upload " |
3269 | - "tags." |
3270 | - ) |
3271 | try: |
3272 | - self.local_repo.fetch_remote_refspecs('pkg', |
3273 | - refspecs=['refs/tags/upload/*:refs/tags/%s/upload/*' % self.namespace], |
3274 | - ) |
3275 | - except GitUbuntuRepositoryFetchError: |
3276 | + applied_publish_parent_head = repo.get_head_by_name(spi.applied_parent_head_name(namespace)) |
3277 | + applied_publish_parent_head_commit = str(applied_publish_parent_head.peel().id) |
3278 | + except AttributeError: |
3279 | pass |
3280 | |
3281 | - self.local_repo.ensure_importer_branches_exist(self.namespace) |
3282 | + if version_compare(str(spi.version), applied_tip_version) <= 0: |
3283 | + logging.warn('Version to import (%s) is not after %s tip (%s)', |
3284 | + spi.version, pretty_head_name, applied_tip_version |
3285 | + ) |
3286 | |
3287 | - self.debian_sinfo = GitUbuntuSourceInformation('debian', |
3288 | - self.pkgname, os.path.abspath(args.pullfile), |
3289 | - args.retries, args.retry_backoffs) |
3290 | + # Walk changelog backwards until we find an imported version |
3291 | + for version in import_tree_versions: |
3292 | + applied_changelog_parent_tag = repo.get_applied_tag(version, namespace) |
3293 | + if applied_changelog_parent_tag is not None: |
3294 | + # sanity check that version from d/changelog of the |
3295 | + # tagged parent matches ours |
3296 | + parent_changelog_version, _ = \ |
3297 | + repo.get_changelog_versions_from_treeish( |
3298 | + str(applied_changelog_parent_tag.peel(pygit2.Tree).id), |
3299 | + ) |
3300 | + # if the parent was imported via a parent override |
3301 | + # itself, assume it is valid without checking its |
3302 | + # tree's debian/changelog, as it wasn't used |
3303 | + if (version not in _PARENT_OVERRIDES and |
3304 | + parent_changelog_version != version |
3305 | + ): |
3306 | + logging.error('Found a tag corresponding to ' |
3307 | + 'parent version %s, but d/changelog ' |
3308 | + 'version (%s) differs. Will ' |
3309 | + 'orphan commit.' % ( |
3310 | + version, |
3311 | + parent_changelog_version |
3312 | + ) |
3313 | + ) |
3314 | + else: |
3315 | + applied_changelog_parent_commit = str(applied_changelog_parent_tag.peel().id) |
3316 | + logging.debug('Changelog parent (tag) is %s', |
3317 | + repo.tag_to_pretty_name(applied_changelog_parent_tag) |
3318 | + ) |
3319 | + break |
3320 | |
3321 | - self.ubuntu_sinfo = GitUbuntuSourceInformation('ubuntu', |
3322 | - self.pkgname, os.path.abspath(args.pullfile), |
3323 | - args.retries, args.retry_backoffs) |
3324 | + # If the two parents are tree-identical, then favor publication |
3325 | + # history. This deals with copy-forwards between series and with |
3326 | + # syncs. |
3327 | + if repo.treeishs_identical(applied_publish_parent_commit, applied_changelog_parent_commit): |
3328 | + applied_changelog_parent_commit = None |
3329 | |
3330 | - debian_head_versions = ( |
3331 | - self.local_repo.get_heads_and_versions('debian', self.namespace) |
3332 | - ) |
3333 | - for head_name in sorted(debian_head_versions): |
3334 | - _, _, pretty_head_name = head_name.partition('%s/' % self.namespace) |
3335 | - logging.debug('Last imported %s version is %s', |
3336 | - pretty_head_name, |
3337 | - debian_head_versions[head_name]['version'] |
3338 | + # Assume no patches to apply |
3339 | + applied_import_tree_hash = unapplied_import_tree_hash |
3340 | + # get tree id from above commit |
3341 | + for ( |
3342 | + applied_import_tree_hash, |
3343 | + patch_name, |
3344 | + patch_desc |
3345 | + ) in import_patches_applied_tree(repo, spi.dsc_pathname): |
3346 | + # special case for .pc removal |
3347 | + if patch_name is None: |
3348 | + msg = b'%b.' % (patch_desc.encode()) |
3349 | + else: |
3350 | + if patch_desc is None: |
3351 | + patch_desc = ( |
3352 | + "%s\n\nNo DEP3 Subject or Description header found" % |
3353 | + patch_name |
3354 | + ) |
3355 | + msg = b'%b\n\nGbp-Pq: %b.' % ( |
3356 | + patch_desc.encode(), |
3357 | + patch_name.encode() |
3358 | ) |
3359 | + unapplied_parent_commit = repo.commit_tree_hash( |
3360 | + applied_import_tree_hash, |
3361 | + [unapplied_parent_commit], |
3362 | + msg, |
3363 | + spi |
3364 | + ) |
3365 | |
3366 | - ubuntu_head_versions = ( |
3367 | - self.local_repo.get_heads_and_versions('ubuntu', self.namespace) |
3368 | + logging.debug( |
3369 | + "Committed patch-application of %s as %s", |
3370 | + patch_name, unapplied_parent_commit |
3371 | ) |
3372 | - for head_name in sorted(ubuntu_head_versions): |
3373 | - _, _, pretty_head_name = head_name.partition('%s/' % self.namespace) |
3374 | - logging.debug('Last imported %s version is %s', |
3375 | - pretty_head_name, |
3376 | - ubuntu_head_versions[head_name]['version'] |
3377 | - ) |
3378 | |
3379 | - if not args.skip_applied: |
3380 | - applied_debian_head_versions = ( |
3381 | - self.local_repo.get_heads_and_versions( |
3382 | - 'applied/debian', self.namespace |
3383 | - ) |
3384 | - ) |
3385 | - for head_name in sorted(applied_debian_head_versions): |
3386 | - _, _, pretty_head_name = head_name.partition( |
3387 | - '%s/' % self.namespace |
3388 | - ) |
3389 | - logging.debug('Last applied %s version is %s', |
3390 | - pretty_head_name, |
3391 | - applied_debian_head_versions[head_name]['version'] |
3392 | + commit_applied_patches_import( |
3393 | + repo, |
3394 | + spi, |
3395 | + applied_import_tree_hash, |
3396 | + namespace, |
3397 | + applied_publish_parent_commit, |
3398 | + applied_changelog_parent_commit, |
3399 | + unapplied_parent_commit, |
3400 | + ) |
3401 | + |
3402 | + update_applied_devel_branches( |
3403 | + repo=repo, |
3404 | + pkgname=spi.name, |
3405 | + namespace=namespace, |
3406 | + ubuntu_sinfo=ubuntu_sinfo, |
3407 | + spi=spi, |
3408 | + ) |
3409 | + |
3410 | +def import_publishes(repo, pkgname, namespace, patches_applied, |
3411 | + debian_head_versions, ubuntu_head_versions, debian_sinfo, |
3412 | + ubuntu_sinfo, active_series_only, workdir, fixup_devel, |
3413 | + skip_orig, |
3414 | +): |
3415 | + history_found = False |
3416 | + only_debian = False |
3417 | + srcpkg_information = None |
3418 | + if patches_applied: |
3419 | + _namespace = namespace |
3420 | + namespace = '%s/applied' % namespace |
3421 | + import_type = 'patches-applied' |
3422 | + import_func = import_applied_spi |
3423 | + else: |
3424 | + _namespace = namespace |
3425 | + import_type = 'patches-unapplied' |
3426 | + import_func = functools.partial( |
3427 | + import_unapplied_spi, |
3428 | + skip_orig=skip_orig |
3429 | + ) |
3430 | + for distname, versions, dist_sinfo in ( |
3431 | + ("debian", debian_head_versions, debian_sinfo), |
3432 | + ("ubuntu", ubuntu_head_versions, ubuntu_sinfo)): |
3433 | + if active_series_only and distname == "debian": |
3434 | + continue |
3435 | + try: |
3436 | + for srcpkg_information in dist_sinfo.launchpad_versions_published_after( |
3437 | + versions, |
3438 | + namespace, |
3439 | + workdir=workdir, |
3440 | + active_series_only=active_series_only |
3441 | + ): |
3442 | + history_found = True |
3443 | + import_func( |
3444 | + repo=repo, |
3445 | + spi=srcpkg_information, |
3446 | + namespace=_namespace, |
3447 | + ubuntu_sinfo=ubuntu_sinfo, |
3448 | ) |
3449 | - |
3450 | - applied_ubuntu_head_versions = ( |
3451 | - self.local_repo.get_heads_and_versions( |
3452 | - 'applied/ubuntu', self.namespace |
3453 | + # Do not push upstream |
3454 | + if fixup_devel and distname == "ubuntu": |
3455 | + update_devel_branches( |
3456 | + repo=repo, |
3457 | + namespace=namespace, |
3458 | + pkgname=pkgname, |
3459 | + ubuntu_sinfo=ubuntu_sinfo, |
3460 | + spi=None, |
3461 | ) |
3462 | + except NoPublicationHistoryException: |
3463 | + logging.warning("No publication history found for %s in %s.", |
3464 | + pkgname, distname |
3465 | ) |
3466 | - for head_name in sorted(applied_ubuntu_head_versions): |
3467 | - _, _, pretty_head_name = head_name.partition( |
3468 | - '%s/' % self.namespace |
3469 | - ) |
3470 | - logging.debug('Last applied %s version is %s', |
3471 | - pretty_head_name, |
3472 | - applied_ubuntu_head_versions[head_name]['version'] |
3473 | - ) |
3474 | - |
3475 | - oldcwd = os.getcwd() |
3476 | - os.chdir(self.local_repo.local_dir) |
3477 | - |
3478 | - if dl_cache is None: |
3479 | - self.workdir = os.path.join(self.local_repo.git_dir, CACHE_PATH) |
3480 | - else: |
3481 | - self.workdir = dl_cache |
3482 | - |
3483 | - os.makedirs(self.workdir, exist_ok=True) |
3484 | - |
3485 | - self.parse_parentfile(args.parentfile) |
3486 | - |
3487 | - only_debian, history_found = self.import_publishes( |
3488 | - patches_applied=False, |
3489 | - debian_head_versions=debian_head_versions, |
3490 | - ubuntu_head_versions=ubuntu_head_versions) |
3491 | - |
3492 | - if not history_found: |
3493 | - logging.error("No publication history for '%s' in debian or ubuntu." |
3494 | - " Wrong source package name?", self.pkgname) |
3495 | - sys.exit(1) |
3496 | - |
3497 | - if not args.skip_applied: |
3498 | - self.import_publishes(patches_applied=True, |
3499 | - debian_head_versions=applied_debian_head_versions, |
3500 | - ubuntu_head_versions=applied_ubuntu_head_versions) |
3501 | - |
3502 | - os.chdir(oldcwd) |
3503 | - |
3504 | - self.local_repo.garbage_collect() |
3505 | - |
3506 | - if no_push: |
3507 | - logging.info('Not pushing to remote as specified') |
3508 | + if distname == 'ubuntu': |
3509 | + only_debian = True |
3510 | + except Exception as e: |
3511 | + if srcpkg_information is None: |
3512 | + msg = 'Unable to import %s to %s' % (import_type, distname) |
3513 | + else: |
3514 | + msg = 'Unable to import %s %s to %s' % (import_type, |
3515 | + str(srcpkg_information.version), distname) |
3516 | + if not patches_applied: |
3517 | + raise GitUbuntuImportError(msg) from e |
3518 | + else: |
3519 | + logging.error(msg) |
3520 | else: |
3521 | - lp = launchpad_login_auth() |
3522 | - repo_path = '~%s/ubuntu/+source/%s/+git/%s' % ( |
3523 | - owner, |
3524 | - self.pkgname, |
3525 | - self.pkgname, |
3526 | - ) |
3527 | - lp_git_repo = lp.git_repositories.getByPath(path=repo_path) |
3528 | - if args.reimport: |
3529 | - if not lp_git_repo.canBeDeleted(): |
3530 | - logging.warning( |
3531 | - "There are linked MPs to the pkg repository, " |
3532 | - "which will be deleted:" |
3533 | - ) |
3534 | - for mp in lp_git_repo.landing_candidates: |
3535 | - logging.warning( |
3536 | - "Adding comment to %s indicating reimport", |
3537 | - mp.web_link |
3538 | - ) |
3539 | - mp.createComment( |
3540 | - content="%s has been reimported and this MP " |
3541 | - "must be manually resubmitted." % |
3542 | - self.pkgname, |
3543 | - subject="%s has been reimported" % self.pkgname, |
3544 | + history_found = True |
3545 | + |
3546 | + return (only_debian, history_found) |
3547 | + |
3548 | + |
3549 | +def parse_args(subparsers=None, base_subparsers=None): |
3550 | + kwargs = dict(description='Update a launchpad git tree based upon ' |
3551 | + 'the state of the Ubuntu and Debian archives', |
3552 | + formatter_class=argparse.ArgumentDefaultsHelpFormatter |
3553 | + ) |
3554 | + if base_subparsers: |
3555 | + kwargs['parents'] = base_subparsers |
3556 | + if subparsers: |
3557 | + parser = subparsers.add_parser('import', **kwargs) |
3558 | + parser.set_defaults(func=cli_main) |
3559 | + else: |
3560 | + parser = argparse.ArgumentParser(**kwargs) |
3561 | + parser.add_argument('package', type=str, |
3562 | + help='Which package to update in the git tree') |
3563 | + parser.add_argument('-o', '--lp-owner', type=str, |
3564 | + help=argparse.SUPPRESS, |
3565 | + default='usd-import-team') |
3566 | + parser.add_argument('-l', '--lp-user', type=str, |
3567 | + help=argparse.SUPPRESS) |
3568 | + parser.add_argument('--dl-cache', type=str, |
3569 | + help=('Cache directory for downloads. Default is to ' |
3570 | + 'put downloads in <directory>/.git/%s' % |
3571 | + CACHE_PATH |
3572 | + ), |
3573 | + default=argparse.SUPPRESS) |
3574 | + parser.add_argument('--no-fetch', action='store_true', |
3575 | + help='Do not fetch from the remote (DANGEROUS; ' |
3576 | + 'only useful for debugging); implies ' |
3577 | + '--no-push') |
3578 | + parser.add_argument('--no-push', action='store_true', |
3579 | + help='Do not push to the remote') |
3580 | + parser.add_argument('--no-clean', action='store_true', |
3581 | + help='Do not clean the temporary directory') |
3582 | + parser.add_argument('-d', '--directory', type=str, |
3583 | + help='Use git repository at specified location rather ' |
3584 | + 'than a temporary directory (implies --no-clean). ' |
3585 | + 'Will create the local repository if needed.', |
3586 | + default=argparse.SUPPRESS |
3587 | ) |
3588 | - # repo deletion will delete the MP, but hopefully |
3589 | - # the above comment will be sufficient to explain |
3590 | - # why |
3591 | - lp_git_repo.lp_delete() |
3592 | - |
3593 | - self.local_repo.git_run([ |
3594 | - 'push', '--atomic', 'pkg', |
3595 | - 'refs/heads/%s/*:refs/heads/*' % self.namespace, |
3596 | - 'refs/tags/%s/*:refs/tags/*' % self.namespace, |
3597 | - ]) |
3598 | - # update our reference |
3599 | - lp_git_repo = lp.git_repositories.getByPath(path=repo_path) |
3600 | - for i in range(args.retries): |
3601 | - try: |
3602 | - if only_debian: |
3603 | - lp_git_repo.default_branch = 'refs/heads/debian/sid' |
3604 | - else: |
3605 | - lp_git_repo.default_branch = 'refs/heads/ubuntu/devel' |
3606 | - lp_git_repo.lp_save() |
3607 | - break |
3608 | - except (NotFound, PreconditionFailed) as e: |
3609 | - time.sleep(args.retry_backoffs[i]) |
3610 | - lp_git_repo.lp_refresh() |
3611 | + parser.add_argument('--fixup-devel', action='store_true', |
3612 | + help=argparse.SUPPRESS) |
3613 | + parser.add_argument('--active-series-only', action='store_true', |
3614 | + help='Do an import of only active Ubuntu series ' |
3615 | + 'history, implies --no-push.') |
3616 | + parser.add_argument('--skip-applied', action='store_true', |
3617 | + help=argparse.SUPPRESS) |
3618 | + parser.add_argument('--skip-orig', action='store_true', |
3619 | + help=argparse.SUPPRESS) |
3620 | + parser.add_argument('--reimport', action='store_true', |
3621 | + help=argparse.SUPPRESS) |
3622 | + if not subparsers: |
3623 | + return parser.parse_args() |
3624 | + return 'import - %s' % kwargs['description'] |
3625 | + |
3626 | +def cli_main(args): |
3627 | + no_clean = args.no_clean |
3628 | + try: |
3629 | + directory = args.directory |
3630 | + no_clean = True |
3631 | + except AttributeError: |
3632 | + directory = None |
3633 | + try: |
3634 | + user = args.lp_user |
3635 | + except AttributeError: |
3636 | + user = None |
3637 | + # --active-series-only/--no-fetch to a tmpdir/--skip-orig |
3638 | + # implies --no-push |
3639 | + no_push = ( |
3640 | + args.no_push or |
3641 | + (args.no_fetch and not directory) or |
3642 | + args.active_series_only or |
3643 | + args.skip_orig |
3644 | + ) |
3645 | + try: |
3646 | + dl_cache = args.dl_cache |
3647 | + except AttributeError: |
3648 | + dl_cache = None |
3649 | + |
3650 | + main( |
3651 | + args.package, |
3652 | + args.lp_owner, |
3653 | + no_clean, |
3654 | + directory, |
3655 | + user, |
3656 | + args.proto, |
3657 | + args.no_fetch, |
3658 | + no_push, |
3659 | + dl_cache, |
3660 | + args.fixup_devel, |
3661 | + args.active_series_only, |
3662 | + args.skip_orig, |
3663 | + args.pullfile, |
3664 | + args.parentfile, |
3665 | + args.retries, |
3666 | + args.retry_backoffs, |
3667 | + args.skip_applied, |
3668 | + args.reimport, |
3669 | + ) |
3670 | diff --git a/gitubuntu/importlocal.py b/gitubuntu/importlocal.py |
3671 | index bf1700e..c237551 100644 |
3672 | --- a/gitubuntu/importlocal.py |
3673 | +++ b/gitubuntu/importlocal.py |
3674 | @@ -2,15 +2,22 @@ import argparse |
3675 | import glob |
3676 | import logging |
3677 | import os |
3678 | -import re |
3679 | -from subprocess import CalledProcessError |
3680 | import sys |
3681 | import tempfile |
3682 | from gitubuntu.dsc import GitUbuntuDsc |
3683 | from gitubuntu.git_repository import GitUbuntuRepository, git_dep14_tag |
3684 | -from gitubuntu.run import decode_binary, run, runq |
3685 | +from gitubuntu.run import run, runq |
3686 | from gitubuntu.version import VERSION |
3687 | -from gitubuntu.importer import GitUbuntuImport |
3688 | +from gitubuntu.importer import ( |
3689 | + parse_parentfile, |
3690 | + _PARENT_OVERRIDES, |
3691 | + import_orig, |
3692 | + import_dsc, |
3693 | + get_changelog_for_commit, |
3694 | + import_patches_unapplied_tree, |
3695 | + import_patches_applied_tree, |
3696 | +) |
3697 | +from gitubuntu.source_information import derive_source_from_series |
3698 | from gitubuntu.versioning import version_compare |
3699 | try: |
3700 | pkg = 'python3-pygit2' |
3701 | @@ -20,264 +27,309 @@ except ImportError: |
3702 | sys.exit(1) |
3703 | |
3704 | |
3705 | -class GitUbuntuImportLocal(GitUbuntuImport): |
3706 | - def __init__(self): |
3707 | - self.parent_overrides = None |
3708 | - |
3709 | - def parse_args(self, subparsers=None, base_subparsers=None): |
3710 | - kwargs = dict(description='Import a DSC file locally', |
3711 | - #usage='%(prog)s [options] -- ...', |
3712 | - epilog='Unlike Launchpad imports, local imports ' |
3713 | - 'require specifying a namespace to uniquely ' |
3714 | - 'identify the source of the DSC file. This ' |
3715 | - 'means that if you want to tie history ' |
3716 | - 'together for several DSCs from the same ' |
3717 | - 'source, you must use the same namespace value.', |
3718 | - formatter_class=argparse.ArgumentDefaultsHelpFormatter |
3719 | - ) |
3720 | - if base_subparsers: |
3721 | - kwargs['parents'] = base_subparsers |
3722 | - if subparsers: |
3723 | - parser = subparsers.add_parser('import-local', **kwargs) |
3724 | - parser.set_defaults(func=self.main) |
3725 | - else: |
3726 | - parser = argparse.ArgumentParser(**kwargs) |
3727 | - parser.add_argument('namespace', |
3728 | - help='String namespace to use for tagging the imports') |
3729 | - parser.add_argument('-d', '--directory', type=str, |
3730 | - help='Use git repository at specified location rather ' |
3731 | - 'than a temporary directory. ' |
3732 | - 'Will create the local repository if needed.', |
3733 | - default=argparse.SUPPRESS |
3734 | - ) |
3735 | - parser.add_argument('dsc', type=str, |
3736 | - help='Path to or URL of DSC file') |
3737 | - parser.add_argument('--skip-applied', action='store_true', |
3738 | - help=argparse.SUPPRESS) |
3739 | - parser.add_argument('--skip-orig', action='store_true', |
3740 | - help=argparse.SUPPRESS) |
3741 | - if not subparsers: |
3742 | - return parser.parse_args() |
3743 | - return 'import-local - %s' % kwargs['description'] |
3744 | - |
3745 | - def main(self, args): |
3746 | - try: |
3747 | - directory = args.directory |
3748 | - no_clean = True |
3749 | - except AttributeError: |
3750 | - directory = None |
3751 | - self.namespace = args.namespace |
3752 | - self.dsc_path = args.dsc |
3753 | - self.skip_applied = args.skip_applied |
3754 | - self.skip_orig = args.skip_orig |
3755 | - |
3756 | - self.local_repo = GitUbuntuRepository(directory) |
3757 | - if not directory: |
3758 | - logging.info('Using git repository at %s', self.local_repo.local_dir) |
3759 | - |
3760 | - self.local_repo.ensure_importer_branches_exist(self.namespace) |
3761 | - if len(self.local_repo.raw_repo.status()) != 0: |
3762 | - logging.error('Working tree must be clean to continue.') |
3763 | - run(['git', 'status'], stdout=None) |
3764 | - sys.exit(1) |
3765 | - |
3766 | - if os.path.exists(self.dsc_path): |
3767 | - self.dsc_path = os.path.abspath(self.dsc_path) |
3768 | - else: |
3769 | - dsc_dir = tempfile.mkdtemp() |
3770 | - |
3771 | - oldcwd = os.getcwd() |
3772 | - os.chdir(dsc_dir) |
3773 | - |
3774 | - runq(['dget', '--quiet', '--allow-unauthenticated', |
3775 | - '--download-only', self.dsc_path]) |
3776 | - dscs = glob.glob(os.path.join(dsc_dir, '*.dsc')) |
3777 | - if len(dscs) == 0: |
3778 | - logging.error('Unable to obtain DSC file.') |
3779 | - sys.exit(1) |
3780 | - if len(dscs) > 1: |
3781 | - logging.error('Multiple DSCs found in temporary directory.') |
3782 | - sys.exit(1) |
3783 | - self.dsc_path = os.path.join(dsc_dir, dscs[0]) |
3784 | - os.chdir(oldcwd) |
3785 | - logging.debug('Importing DSC at local path %s' % self.dsc_path) |
3786 | - self.dsc = GitUbuntuDsc(self.dsc_path) |
3787 | - if not self.dsc.verify(): |
3788 | - logging.error('Unable to verify orig tarball specified by DSC %s', |
3789 | - self.dsc_path) |
3790 | - sys.exit(1) |
3791 | - |
3792 | - self.pkgname = self.dsc['Source'] |
3793 | +def main( |
3794 | + directory, |
3795 | + namespace, |
3796 | + dsc_path, |
3797 | + skip_applied, |
3798 | + skip_orig, |
3799 | + parentfile, |
3800 | +): |
3801 | + repo = GitUbuntuRepository(directory) |
3802 | + if not directory: |
3803 | + logging.info('Using git repository at %s', repo.local_dir) |
3804 | + |
3805 | + repo.ensure_importer_branches_exist(namespace) |
3806 | + if len(repo.raw_repo.status()) != 0: |
3807 | + logging.error('Working tree must be clean to continue.') |
3808 | + run(['git', 'status'], stdout=None) |
3809 | + sys.exit(1) |
3810 | + |
3811 | + if os.path.exists(dsc_path): |
3812 | + dsc_path = os.path.abspath(dsc_path) |
3813 | + else: |
3814 | + dsc_dir = tempfile.mkdtemp() |
3815 | |
3816 | oldcwd = os.getcwd() |
3817 | - os.chdir(self.local_repo.local_dir) |
3818 | - |
3819 | - self.local_repo.clean_repository_state() |
3820 | + os.chdir(dsc_dir) |
3821 | |
3822 | - self.parse_parentfile(args.parentfile) |
3823 | - |
3824 | - unapplied_import_tree_hash = self.import_patches_unapplied_tree(self.dsc_path) |
3825 | - |
3826 | - import_tree_versions = self.local_repo.get_all_changelog_versions_from_treeish( |
3827 | - unapplied_import_tree_hash |
3828 | - ) |
3829 | - |
3830 | - changelog_version = import_tree_versions[0] |
3831 | - |
3832 | - logging.debug( |
3833 | - 'Found changelog version %s in newly imported tree' % |
3834 | - changelog_version |
3835 | - ) |
3836 | - |
3837 | - head = None |
3838 | - head_version = None |
3839 | - if not self.local_repo.raw_repo.head_is_unborn: |
3840 | - head = self.local_repo.raw_repo.head |
3841 | - # check if the version to import has already been imported to |
3842 | - # this head |
3843 | - if head is not None: |
3844 | - head_version, _ = self.local_repo.get_changelog_versions_from_treeish( |
3845 | - str(head.peel().id)) |
3846 | - |
3847 | - logging.debug('Head version is %s', head_version) |
3848 | - |
3849 | - unapplied_changelog_parent_commit = None |
3850 | - applied_changelog_parent_commit = None |
3851 | - |
3852 | - if version_compare(changelog_version, head_version) <= 0: |
3853 | - logging.warn('Version to import (%s) is not after HEAD ' |
3854 | - 'version (%s), history may not make sense.', |
3855 | - changelog_version, head_version |
3856 | - ) |
3857 | - |
3858 | - # Walk changelog backwards until we find an imported version |
3859 | - for version in import_tree_versions: |
3860 | - unapplied_tag_ref = '%s/import/%s' % (self.namespace, git_dep14_tag(version)) |
3861 | - applied_tag_ref = '%s/applied/%s' % (self.namespace, git_dep14_tag(version)) |
3862 | - unapplied_changelog_parent_tag = self.local_repo.get_tag_reference(unapplied_tag_ref) |
3863 | - applied_changelog_parent_tag = self.local_repo.get_tag_reference(applied_tag_ref) |
3864 | - if unapplied_changelog_parent_tag is not None: |
3865 | - # sanity check that version from d/changelog of the |
3866 | - # tagged parent matches ours |
3867 | - parent_changelog_version, _ = \ |
3868 | - self.local_repo.get_changelog_versions_from_treeish( |
3869 | - str(unapplied_changelog_parent_tag.peel(pygit2.Tree).id), |
3870 | - ) |
3871 | - # if the parent was imported via a parent override |
3872 | - # itself, assume it is valid without checking its |
3873 | - # tree's debian/changelog, as it wasn't used |
3874 | - if (version not in self.parent_overrides and |
3875 | - parent_changelog_version != version |
3876 | - ): |
3877 | - logging.error('Found a tag corresponding to ' |
3878 | - 'parent version %s, but d/changelog ' |
3879 | - 'version (%s) differs. Will ' |
3880 | - 'orphan commit.' % ( |
3881 | - version, |
3882 | - parent_changelog_version |
3883 | - ) |
3884 | - ) |
3885 | - else: |
3886 | - unapplied_changelog_parent_commit = str(unapplied_changelog_parent_tag.peel().id) |
3887 | - if applied_changelog_parent_tag: |
3888 | - applied_changelog_parent_commit = str(applied_changelog_parent_tag.peel().id) |
3889 | - logging.debug('Changelog parent (tag) is %s', |
3890 | - self.local_repo.tag_to_pretty_name(unapplied_changelog_parent_tag) |
3891 | - ) |
3892 | - break |
3893 | - |
3894 | - distribution = self.local_repo.get_changelog_distribution_from_treeish(unapplied_import_tree_hash) |
3895 | - # do lookup of series for dist on lp? |
3896 | - dist = 'ubuntu' |
3897 | - if distribution in ("stable", "testing", "unstable", "experimental"): |
3898 | - dist = 'debian' |
3899 | - self.import_dsc(self.dsc, changelog_version, dist) |
3900 | - try: |
3901 | - if not self.skip_orig: |
3902 | - self.import_orig(self.dsc, changelog_version, dist) |
3903 | - except Exception as e: |
3904 | - logging.error('Unable to import orig tarball for %s: %s', |
3905 | - changelog_version, e) |
3906 | + runq(['dget', '--quiet', '--allow-unauthenticated', |
3907 | + '--download-only', dsc_path]) |
3908 | + dscs = glob.glob(os.path.join(dsc_dir, '*.dsc')) |
3909 | + if len(dscs) == 0: |
3910 | + logging.error('Unable to obtain DSC file.') |
3911 | sys.exit(1) |
3912 | - |
3913 | - msg = (b'Import unapplied version %b\n\nImported using git-ubuntu-import.' % |
3914 | - (changelog_version.encode()) |
3915 | - ) |
3916 | - |
3917 | - parents = [] |
3918 | - |
3919 | - tag = None |
3920 | - tag_ref = '%s/import/%s' % (self.namespace, |
3921 | - git_dep14_tag(changelog_version)) |
3922 | - if self.local_repo.get_tag_reference(tag_ref) is None: |
3923 | - # Not imported before |
3924 | - tag = tag_ref |
3925 | - |
3926 | - if unapplied_changelog_parent_commit is not None: |
3927 | - msg += b'\n' |
3928 | - parents.append(unapplied_changelog_parent_commit) |
3929 | - msg += b'\nChangelog parent: %b' % unapplied_changelog_parent_commit.encode() |
3930 | - |
3931 | - changelog_entry = self.get_changelog_for_commit( |
3932 | - unapplied_import_tree_hash, |
3933 | - None, |
3934 | - unapplied_changelog_parent_commit |
3935 | - ) |
3936 | - |
3937 | - msg += b'%b' % changelog_entry |
3938 | - unapplied_parent_commit = self.local_repo.commit_tree_hash( |
3939 | - unapplied_import_tree_hash, parents, msg) |
3940 | - if tag is None: |
3941 | - logging.info('Committed unapplied import of %s as %s', |
3942 | - changelog_version, unapplied_parent_commit) |
3943 | - else: |
3944 | - logging.debug('Creating tag %s pointing to %s', tag, |
3945 | - unapplied_parent_commit) |
3946 | - self.local_repo.git_run(['tag', '-a', '-m', 'git-ubuntu import v%s' % VERSION, |
3947 | - tag, unapplied_parent_commit |
3948 | - ] |
3949 | - ) |
3950 | - |
3951 | - if not args.skip_applied: |
3952 | - # Assume no patches to apply |
3953 | - applied_import_tree_hash = unapplied_import_tree_hash |
3954 | - for (applied_import_tree_hash, patch_name, patch_desc) in self.import_patches_applied_tree(self.dsc_path): |
3955 | - # special case for .pc removal |
3956 | - if patch_name is None: |
3957 | - msg = b'%b.' % (patch_desc.encode()) |
3958 | - else: |
3959 | - if patch_desc is None: |
3960 | - patch_desc = '%s\n\nNo DEP3 Subject or Description header found' % patch_name |
3961 | - msg = b'%b\n\nGbp-Pq: %b.' % (patch_desc.encode(), patch_name.encode()) |
3962 | - unapplied_parent_commit = self.local_repo.commit_tree_hash( |
3963 | - applied_import_tree_hash, |
3964 | - [unapplied_parent_commit], |
3965 | - msg) |
3966 | - |
3967 | - logging.debug('Committed patch-application of %s as %s', |
3968 | - patch_name, unapplied_parent_commit |
3969 | - ) |
3970 | - |
3971 | - tag = None |
3972 | - tag_ref = '%s/applied/%s' % (self.namespace, |
3973 | - git_dep14_tag(changelog_version)) |
3974 | - if self.local_repo.get_tag_reference(tag_ref) is None: |
3975 | - # Not imported before |
3976 | - tag= tag_ref |
3977 | - |
3978 | - if tag is None: |
3979 | - logging.info('Committed applied import of %s as %s', |
3980 | - changelog_version, unapplied_parent_commit) |
3981 | - else: |
3982 | - logging.debug('Creating tag %s pointing to %s', tag, |
3983 | - unapplied_parent_commit) |
3984 | - self.local_repo.git_run(['tag', '-a', '-m', 'git-ubuntu import v%s' % VERSION, |
3985 | - tag, unapplied_parent_commit |
3986 | - ] |
3987 | - ) |
3988 | - |
3989 | - self.local_repo.clean_repository_state() |
3990 | - |
3991 | + if len(dscs) > 1: |
3992 | + logging.error('Multiple DSCs found in temporary directory.') |
3993 | + sys.exit(1) |
3994 | + dsc_path = os.path.join(dsc_dir, dscs[0]) |
3995 | os.chdir(oldcwd) |
3996 | - |
3997 | - self.local_repo.garbage_collect() |
3998 | + logging.debug('Importing DSC at local path %s', dsc_path) |
3999 | + dsc = GitUbuntuDsc(dsc_path) |
4000 | + if not dsc.verify(): |
4001 | + logging.error( |
4002 | + 'Unable to verify orig tarball specified by DSC %s', |
4003 | + dsc_path, |
4004 | + ) |
4005 | + sys.exit(1) |
4006 | + |
4007 | + pkgname = dsc['Source'] |
4008 | + |
4009 | + oldcwd = os.getcwd() |
4010 | + os.chdir(repo.local_dir) |
4011 | + |
4012 | + repo.clean_repository_state() |
4013 | + |
4014 | + parse_parentfile(parentfile, pkgname) |
4015 | + |
4016 | + unapplied_import_tree_hash = import_patches_unapplied_tree(repo, dsc_path) |
4017 | + |
4018 | + import_tree_versions = repo.get_all_changelog_versions_from_treeish( |
4019 | + unapplied_import_tree_hash |
4020 | + ) |
4021 | + |
4022 | + changelog_version = import_tree_versions[0] |
4023 | + |
4024 | + logging.debug( |
4025 | + 'Found changelog version %s in newly imported tree', |
4026 | + changelog_version |
4027 | + ) |
4028 | + |
4029 | + head = None |
4030 | + head_version = None |
4031 | + if not repo.raw_repo.head_is_unborn: |
4032 | + head = repo.raw_repo.head |
4033 | + # check if the version to import has already been imported to |
4034 | + # this head |
4035 | + if head is not None: |
4036 | + head_version, _ = repo.get_changelog_versions_from_treeish( |
4037 | + str(head.peel().id) |
4038 | + ) |
4039 | + |
4040 | + logging.debug('Head version is %s', head_version) |
4041 | + |
4042 | + unapplied_changelog_parent_commit = None |
4043 | + applied_changelog_parent_commit = None |
4044 | + |
4045 | + if version_compare(changelog_version, head_version) <= 0: |
4046 | + logging.warn('Version to import (%s) is not after HEAD ' |
4047 | + 'version (%s), history may not make sense.', |
4048 | + changelog_version, head_version |
4049 | + ) |
4050 | + |
4051 | + # Walk changelog backwards until we find an imported version |
4052 | + for version in import_tree_versions: |
4053 | + unapplied_tag_ref = '%s/import/%s' % (namespace, git_dep14_tag(version)) |
4054 | + applied_tag_ref = '%s/applied/%s' % (namespace, git_dep14_tag(version)) |
4055 | + unapplied_changelog_parent_tag = repo.get_tag_reference(unapplied_tag_ref) |
4056 | + applied_changelog_parent_tag = repo.get_tag_reference(applied_tag_ref) |
4057 | + if unapplied_changelog_parent_tag is not None: |
4058 | + # sanity check that version from d/changelog of the |
4059 | + # tagged parent matches ours |
4060 | + parent_changelog_version, _ = \ |
4061 | + repo.get_changelog_versions_from_treeish( |
4062 | + str(unapplied_changelog_parent_tag.peel(pygit2.Tree).id), |
4063 | + ) |
4064 | + # if the parent was imported via a parent override |
4065 | + # itself, assume it is valid without checking its |
4066 | + # tree's debian/changelog, as it wasn't used |
4067 | + if (version not in _PARENT_OVERRIDES and |
4068 | + parent_changelog_version != version |
4069 | + ): |
4070 | + logging.error('Found a tag corresponding to ' |
4071 | + 'parent version %s, but d/changelog ' |
4072 | + 'version (%s) differs. Will ' |
4073 | + 'orphan commit.', |
4074 | + version, |
4075 | + parent_changelog_version |
4076 | + ) |
4077 | + else: |
4078 | + unapplied_changelog_parent_commit = str( |
4079 | + unapplied_changelog_parent_tag.peel().id |
4080 | + ) |
4081 | + if applied_changelog_parent_tag: |
4082 | + applied_changelog_parent_commit = str( |
4083 | + applied_changelog_parent_tag.peel().id |
4084 | + ) |
4085 | + logging.debug('Changelog parent (tag) is %s', |
4086 | + repo.tag_to_pretty_name(unapplied_changelog_parent_tag) |
4087 | + ) |
4088 | + break |
4089 | + |
4090 | + distribution = repo.get_changelog_distribution_from_treeish( |
4091 | + unapplied_import_tree_hash |
4092 | + ) |
4093 | + dist = derive_source_from_series(distribution) |
4094 | + import_dsc( |
4095 | + repo=repo, |
4096 | + dsc=dsc, |
4097 | + namespace=namespace, |
4098 | + version=changelog_version, |
4099 | + dist=dist, |
4100 | + ) |
4101 | + try: |
4102 | + if not skip_orig: |
4103 | + import_orig( |
4104 | + repo=repo, |
4105 | + dsc=dsc, |
4106 | + namespace=namespace, |
4107 | + version=changelog_version, |
4108 | + dist=dist, |
4109 | + ) |
4110 | + except Exception as e: |
4111 | + logging.error('Unable to import orig tarball for %s: %s', |
4112 | + changelog_version, e) |
4113 | + raise |
4114 | + sys.exit(1) |
4115 | + |
4116 | + msg = ( |
4117 | + b'Import unapplied version %b\n\nImported using git-ubuntu-import.' % |
4118 | + (changelog_version.encode()) |
4119 | + ) |
4120 | + |
4121 | + parents = [] |
4122 | + |
4123 | + tag = None |
4124 | + tag_ref = '%s/import/%s' % ( |
4125 | + namespace, |
4126 | + git_dep14_tag(changelog_version) |
4127 | + ) |
4128 | + if repo.get_tag_reference(tag_ref) is None: |
4129 | + # Not imported before |
4130 | + tag = tag_ref |
4131 | + |
4132 | + if unapplied_changelog_parent_commit is not None: |
4133 | + msg += b'\n' |
4134 | + parents.append(unapplied_changelog_parent_commit) |
4135 | + msg += b'\nChangelog parent: %b' % unapplied_changelog_parent_commit.encode() |
4136 | + |
4137 | + changelog_entry = get_changelog_for_commit( |
4138 | + repo=repo, |
4139 | + tree_hash=unapplied_import_tree_hash, |
4140 | + publish_parent_commit=None, |
4141 | + changelog_parent_commit=unapplied_changelog_parent_commit, |
4142 | + ) |
4143 | + |
4144 | + msg += b'%b' % changelog_entry |
4145 | + unapplied_parent_commit = repo.commit_tree_hash( |
4146 | + unapplied_import_tree_hash, parents, msg |
4147 | + ) |
4148 | + if tag is None: |
4149 | + logging.info('Committed unapplied import of %s as %s', |
4150 | + changelog_version, unapplied_parent_commit) |
4151 | + else: |
4152 | + logging.debug('Creating tag %s pointing to %s', tag, |
4153 | + unapplied_parent_commit |
4154 | + ) |
4155 | + repo.git_run(['tag', '-a', '-m', 'git-ubuntu import v%s' % VERSION, |
4156 | + tag, unapplied_parent_commit |
4157 | + ] |
4158 | + ) |
4159 | + |
4160 | + if not skip_applied: |
4161 | + # Assume no patches to apply |
4162 | + applied_import_tree_hash = unapplied_import_tree_hash |
4163 | + for ( |
4164 | + applied_import_tree_hash, |
4165 | + patch_name, |
4166 | + patch_desc, |
4167 | + ) in import_patches_applied_tree(repo, dsc_path): |
4168 | + # special case for .pc removal |
4169 | + if patch_name is None: |
4170 | + msg = b'%b.' % (patch_desc.encode()) |
4171 | + else: |
4172 | + if patch_desc is None: |
4173 | + patch_desc = '%s\n\nNo DEP3 Subject or Description header found' % patch_name |
4174 | + msg = b'%b\n\nGbp-Pq: %b.' % (patch_desc.encode(), patch_name.encode()) |
4175 | + unapplied_parent_commit = repo.commit_tree_hash( |
4176 | + applied_import_tree_hash, |
4177 | + [unapplied_parent_commit], |
4178 | + msg) |
4179 | + |
4180 | + logging.debug('Committed patch-application of %s as %s', |
4181 | + patch_name, unapplied_parent_commit |
4182 | + ) |
4183 | + |
4184 | + tag = None |
4185 | + tag_ref = '%s/applied/%s' % (namespace, |
4186 | + git_dep14_tag(changelog_version)) |
4187 | + if repo.get_tag_reference(tag_ref) is None: |
4188 | + # Not imported before |
4189 | + tag = tag_ref |
4190 | + |
4191 | + if tag is None: |
4192 | + logging.info('Committed applied import of %s as %s', |
4193 | + changelog_version, |
4194 | + unapplied_parent_commit |
4195 | + ) |
4196 | + else: |
4197 | + logging.debug('Creating tag %s pointing to %s', tag, |
4198 | + unapplied_parent_commit |
4199 | + ) |
4200 | + repo.git_run( |
4201 | + [ |
4202 | + 'tag', |
4203 | + '-a', |
4204 | + '-m', |
4205 | + 'git-ubuntu import v%s' % VERSION, |
4206 | + tag, |
4207 | + unapplied_parent_commit, |
4208 | + ] |
4209 | + ) |
4210 | + |
4211 | + repo.clean_repository_state() |
4212 | + |
4213 | + os.chdir(oldcwd) |
4214 | + |
4215 | + repo.garbage_collect() |
4216 | + |
4217 | +def parse_args(subparsers=None, base_subparsers=None): |
4218 | + kwargs = dict(description='Import a DSC file locally', |
4219 | + #usage='%(prog)s [options] -- ...', |
4220 | + epilog='Unlike Launchpad imports, local imports ' |
4221 | + 'require specifying a namespace to uniquely ' |
4222 | + 'identify the source of the DSC file. This ' |
4223 | + 'means that if you want to tie history ' |
4224 | + 'together for several DSCs from the same ' |
4225 | + 'source, you must use the same namespace value.', |
4226 | + formatter_class=argparse.ArgumentDefaultsHelpFormatter |
4227 | + ) |
4228 | + if base_subparsers: |
4229 | + kwargs['parents'] = base_subparsers |
4230 | + if subparsers: |
4231 | + parser = subparsers.add_parser('import-local', **kwargs) |
4232 | + parser.set_defaults(func=cli_main) |
4233 | + else: |
4234 | + parser = argparse.ArgumentParser(**kwargs) |
4235 | + parser.add_argument('namespace', |
4236 | + help='String namespace to use for tagging the imports') |
4237 | + parser.add_argument('-d', '--directory', type=str, |
4238 | + help='Use git repository at specified location rather ' |
4239 | + 'than a temporary directory. ' |
4240 | + 'Will create the local repository if needed.', |
4241 | + default=argparse.SUPPRESS |
4242 | + ) |
4243 | + parser.add_argument('dsc', type=str, |
4244 | + help='Path to or URL of DSC file') |
4245 | + parser.add_argument('--skip-applied', action='store_true', |
4246 | + help=argparse.SUPPRESS) |
4247 | + parser.add_argument('--skip-orig', action='store_true', |
4248 | + help=argparse.SUPPRESS) |
4249 | + if not subparsers: |
4250 | + return parser.parse_args() |
4251 | + return 'import-local - %s' % kwargs['description'] |
4252 | + |
4253 | +def cli_main(args): |
4254 | + try: |
4255 | + directory = args.directory |
4256 | + except AttributeError: |
4257 | + directory = None |
4258 | + |
4259 | + main( |
4260 | + directory, |
4261 | + args.namespace, |
4262 | + args.dsc, |
4263 | + args.skip_applied, |
4264 | + args.skip_orig, |
4265 | + args.parentfile, |
4266 | + ) |
4267 | diff --git a/gitubuntu/importppa.py b/gitubuntu/importppa.py |
4268 | index 2ae1f81..da57f53 100644 |
4269 | --- a/gitubuntu/importppa.py |
4270 | +++ b/gitubuntu/importppa.py |
4271 | @@ -10,8 +10,25 @@ from gitubuntu.git_repository import ( |
4272 | GitUbuntuRepository, |
4273 | GitUbuntuRepositoryFetchError, |
4274 | ) |
4275 | -from gitubuntu.importer import GitUbuntuImport |
4276 | -from gitubuntu.source_information import GitUbuntuSourceInformation, NoPublicationHistoryException, SourceExtractionException, launchpad_login_auth |
4277 | +from gitubuntu.importer import ( |
4278 | + cleanup, |
4279 | + parse_parentfile, |
4280 | + _PARENT_OVERRIDES, |
4281 | + import_orig, |
4282 | + import_dsc, |
4283 | + get_changelog_for_commit, |
4284 | + import_unapplied_spi, |
4285 | + import_applied_spi, |
4286 | + import_patches_unapplied_tree, |
4287 | + import_patches_applied_tree, |
4288 | +) |
4289 | +from gitubuntu.source_information import ( |
4290 | + GitUbuntuSourceInformation, |
4291 | + NoPublicationHistoryException, |
4292 | + SourceExtractionException, |
4293 | + launchpad_login_auth, |
4294 | + GitUbuntuSourceInformation, |
4295 | +) |
4296 | from gitubuntu.version import VERSION |
4297 | try: |
4298 | pkg = 'python3-pygit2' |
4299 | @@ -23,181 +40,216 @@ except ImportError: |
4300 | sys.exit(1) |
4301 | |
4302 | |
4303 | -class GitUbuntuImportPPA(GitUbuntuImport): |
4304 | - def __init__(self): |
4305 | - self.parent_overrides = None |
4306 | - |
4307 | - def parse_args(self, subparsers=None, base_subparsers=None): |
4308 | - kwargs = dict(description='Update a launchpad git tree based upon ' |
4309 | - 'the state of a PPA archive', |
4310 | - epilog='PPA must be specified as ppa:<name>', |
4311 | - formatter_class=argparse.ArgumentDefaultsHelpFormatter |
4312 | - ) |
4313 | - if base_subparsers: |
4314 | - kwargs['parents'] = base_subparsers |
4315 | - if subparsers: |
4316 | - parser = subparsers.add_parser('import-ppa', **kwargs) |
4317 | - parser.set_defaults(func=self.main) |
4318 | - else: |
4319 | - parser = argparse.ArgumentParser(**kwargs) |
4320 | - parser.add_argument('ppa', type=str, |
4321 | - help='Name of PPA to update in the git tree') |
4322 | - parser.add_argument('package', type=str, |
4323 | - help='Which package to update in the git tree') |
4324 | - parser.add_argument('-o', '--lp-owner', type=str, |
4325 | - help='Which launchpad user\'s tree to update', |
4326 | - default='usd-import-team') |
4327 | - parser.add_argument('-l', '--lp-user', type=str, |
4328 | - help='Specify the Launchpad user to use', |
4329 | - default=getpass.getuser()) |
4330 | - parser.add_argument('--dl-cache', type=str, |
4331 | - help=('Cache directory for downloads. Default is to ' |
4332 | - 'put downloads in <directory>/.git/%s' % |
4333 | - CACHE_PATH |
4334 | - ), |
4335 | - default=argparse.SUPPRESS) |
4336 | - parser.add_argument('--no-fetch', action='store_true', |
4337 | - help='Do not fetch from the remote (DANGEROUS: ' |
4338 | - 'only useful for debugging); implies ' |
4339 | - '--no-push') |
4340 | - parser.add_argument('--no-push', action='store_true', |
4341 | - help='Do not push to the remote (WARNING: ' |
4342 | - 'PPA imports are a work-in-progress and ' |
4343 | - 'pushing is currently disabled)', |
4344 | - default=True) |
4345 | - parser.add_argument('--no-clean', action='store_true', |
4346 | - help='Do not clean the temporary directory') |
4347 | - parser.add_argument('-d', '--directory', type=str, |
4348 | - help='Use git repository at specified location rather ' |
4349 | - 'than a temporary directory (implies --no-clean). ' |
4350 | - 'Will create the local repository if needed.', |
4351 | - default=argparse.SUPPRESS |
4352 | - ) |
4353 | - if not subparsers: |
4354 | - return parser.parse_args() |
4355 | - return 'import-ppa - %s' % kwargs['description'] |
4356 | - |
4357 | - def main(self, args): |
4358 | - if re.match(r'ppa:\w+', args.ppa) is None: |
4359 | - logging.error('Specified PPA (%s) is not in the format ' |
4360 | - 'ppa:<name>', args.ppa) |
4361 | - sys.exit(1) |
4362 | - ppa = args.ppa |
4363 | - self.pkgname = args.package |
4364 | - owner = args.lp_owner |
4365 | - user = args.lp_user |
4366 | - no_clean = args.no_clean |
4367 | - try: |
4368 | - directory = args.directory |
4369 | - no_clean = True |
4370 | - except AttributeError: |
4371 | - directory = None |
4372 | - no_push = True |
4373 | - try: |
4374 | - dl_cache = args.dl_cache |
4375 | - except AttributeError: |
4376 | - dl_cache = None |
4377 | - |
4378 | - # what other transformations? |
4379 | - self.namespace = ppa.replace(':', '_') |
4380 | - |
4381 | - logging.info('Ubuntu Server Team importer v%s' % VERSION) |
4382 | - |
4383 | - self.local_repo = GitUbuntuRepository(directory, user, args.proto) |
4384 | - if not directory: |
4385 | - logging.info('Using git repository at %s', self.local_repo.local_dir) |
4386 | - self.local_repo.ensure_importer_branches_exist(self.namespace) |
4387 | - |
4388 | - atexit.register(self.cleanup, no_clean, self.local_repo.local_dir) |
4389 | - |
4390 | - self.local_repo.add_remote(self.pkgname, owner, self.namespace) |
4391 | - if not args.no_fetch: |
4392 | +def main( |
4393 | + ppa, |
4394 | + pkgname, |
4395 | + directory, |
4396 | + owner, |
4397 | + user, |
4398 | + pullfile, |
4399 | + parentfile, |
4400 | + no_clean, |
4401 | + dl_cache, |
4402 | + proto, |
4403 | + retries, |
4404 | + retry_backoffs, |
4405 | + skip_orig, |
4406 | +): |
4407 | + if re.match(r'ppa:\w+', ppa) is None: |
4408 | + logging.error( |
4409 | + "Specified PPA (%s) is not in the format ppa:<name>", |
4410 | + ppa, |
4411 | + ) |
4412 | + sys.exit(1) |
4413 | + # what other transformations? |
4414 | + namespace = ppa.replace(':', '_') |
4415 | + |
4416 | + logging.info('Ubuntu Server Team importer v%s' % VERSION) |
4417 | + |
4418 | + repo = GitUbuntuRepository(directory, user, proto) |
4419 | + if not directory: |
4420 | + logging.info('Using git repository at %s', repo.local_dir) |
4421 | + repo.ensure_importer_branches_exist(namespace) |
4422 | + |
4423 | + atexit.register(cleanup, no_clean, repo.local_dir) |
4424 | + |
4425 | + source_information = GitUbuntuSourceInformation( |
4426 | + ppa, |
4427 | + pkgname, |
4428 | + os.path.abspath(pullfile), |
4429 | + retries, |
4430 | + retry_backoffs, |
4431 | + ) |
4432 | + |
4433 | + ubuntu_head_versions = repo.get_heads_and_versions( |
4434 | + 'ubuntu', |
4435 | + namespace, |
4436 | + ) |
4437 | + for head_name in sorted(ubuntu_head_versions): |
4438 | + _, _, pretty_head_name = head_name.partition('%s/', namespace) |
4439 | + logging.debug('Last imported %s version is %s', |
4440 | + pretty_head_name, |
4441 | + ubuntu_head_versions[head_name]['version'] |
4442 | + ) |
4443 | + |
4444 | + applied_ubuntu_head_versions = repo.get_heads_and_versions( |
4445 | + 'applied/ubuntu', |
4446 | + namespace, |
4447 | + ) |
4448 | + for head_name in sorted(applied_ubuntu_head_versions): |
4449 | + _, _, pretty_head_name = head_name.partition('%s/', namespace) |
4450 | + logging.debug('Last applied %s version is %s', |
4451 | + pretty_head_name, |
4452 | + applied_ubuntu_head_versions[head_name]['version'] |
4453 | + ) |
4454 | + |
4455 | + oldcwd = os.getcwd() |
4456 | + os.chdir(repo.local_dir) |
4457 | + |
4458 | + if dl_cache is None: |
4459 | + workdir = os.path.join(repo.git_dir, CACHE_PATH) |
4460 | + else: |
4461 | + workdir = dl_cache |
4462 | + |
4463 | + os.makedirs(workdir, exist_ok=True) |
4464 | + |
4465 | + parse_parentfile(pkgname, parentfile) |
4466 | + |
4467 | + history_found = False |
4468 | + try: |
4469 | + for srcpkg_information in \ |
4470 | + source_information.launchpad_versions_published_after( |
4471 | + ubuntu_head_versions, |
4472 | + namespace, |
4473 | + workdir=workdir |
4474 | + ): |
4475 | try: |
4476 | - self.local_repo.fetch_remote(self.namespace) |
4477 | - except GitUbuntuRepositoryFetchError: |
4478 | + import_unapplied_spi( |
4479 | + repo=repo, |
4480 | + spi=srcpkg_information, |
4481 | + namespace=namespace, |
4482 | + ubuntu_sinfo=source_information, |
4483 | + skip_orig=skip_orig, |
4484 | + ) |
4485 | + except DownloadError: |
4486 | + # it is non-fatal for a PPA to not have files for older |
4487 | + # publishes |
4488 | pass |
4489 | + history_found = True |
4490 | + except NoPublicationHistoryException: |
4491 | + logging.warning("No publication history found for %s in %s.", |
4492 | + pkgname, |
4493 | + ppa, |
4494 | + ) |
4495 | |
4496 | - source_information = GitUbuntuSourceInformation(ppa, self.pkgname, |
4497 | - os.path.abspath(args.pullfile), |
4498 | - args.retries, args.retry_backoffs) |
4499 | - |
4500 | - ubuntu_head_versions = self.local_repo.get_heads_and_versions( |
4501 | - 'ubuntu', self.namespace |
4502 | + if not history_found: |
4503 | + logging.error( |
4504 | + "No publication history for '%s' in %s. Wrong source package name?", |
4505 | + pkgname, |
4506 | + ppa, |
4507 | ) |
4508 | - for head_name in sorted(ubuntu_head_versions): |
4509 | - _, _, pretty_head_name = head_name.partition('%s/', self.namespace) |
4510 | - logging.debug('Last imported %s version is %s', |
4511 | - pretty_head_name, |
4512 | - ubuntu_head_versions[head_name]['version'] |
4513 | - ) |
4514 | - |
4515 | - applied_ubuntu_head_versions = self.local_repo.get_heads_and_versions( |
4516 | - 'applied/ubuntu', self.namespace |
4517 | + sys.exit(1) |
4518 | + |
4519 | + try: |
4520 | + for srcpkg_information in \ |
4521 | + source_information.launchpad_versions_published_after( |
4522 | + applied_ubuntu_head_versions, |
4523 | + '%s/applied' % namespace, |
4524 | + workdir=workdir |
4525 | + ): |
4526 | + try: |
4527 | + import_applied_spi( |
4528 | + repo=repo, |
4529 | + spi=srcpkg_information, |
4530 | + namespace=namespace, |
4531 | + ubuntu_sinfo=source_information, |
4532 | + ) |
4533 | + except DownloadError: |
4534 | + # it is non-fatal for a PPA to not have files for older |
4535 | + # publishes |
4536 | + pass |
4537 | + except NoPublicationHistoryException: |
4538 | + logging.warning("No publication history found for %s in %s.", |
4539 | + pkgname, ppa |
4540 | ) |
4541 | - for head_name in sorted(applied_ubuntu_head_versions): |
4542 | - _, _, pretty_head_name = head_name.partition('%s/', self.namespace) |
4543 | - logging.debug('Last applied %s version is %s', |
4544 | - pretty_head_name, |
4545 | - applied_ubuntu_head_versions[head_name]['version'] |
4546 | - ) |
4547 | - |
4548 | - oldcwd = os.getcwd() |
4549 | - os.chdir(self.local_repo.local_dir) |
4550 | - |
4551 | - if dl_cache is None: |
4552 | - workdir = os.path.join(self.local_repo.git_dir, CACHE_PATH) |
4553 | - else: |
4554 | - workdir = dl_cache |
4555 | - |
4556 | - os.makedirs(workdir, exist_ok=True) |
4557 | - |
4558 | - self.parse_parentfile(args.parentfile) |
4559 | - |
4560 | - history_found = False |
4561 | - try: |
4562 | - for srcpkg_information in \ |
4563 | - source_information.launchpad_versions_published_after(ubuntu_head_versions, |
4564 | - self.namespace, |
4565 | - workdir=workdir): |
4566 | - try: |
4567 | - self.import_unapplied_spi(srcpkg_information) |
4568 | - except DownloadError: |
4569 | - # it is non-fatal for a PPA to not have files for older |
4570 | - # publishes |
4571 | - pass |
4572 | - history_found = True |
4573 | - except NoPublicationHistoryException: |
4574 | - logging.warning("No publication history found for %s in %s. ", |
4575 | - self.pkgname, ppa) |
4576 | - |
4577 | - if not history_found: |
4578 | - logging.error("No publication history for '%s' in %s. " |
4579 | - "Wrong source package name?", self.pkgname, ppa) |
4580 | - sys.exit(1) |
4581 | - |
4582 | - try: |
4583 | - for srcpkg_information in \ |
4584 | - source_information.launchpad_versions_published_after(applied_ubuntu_head_versions, |
4585 | - '%s/applied' % self.namespace, |
4586 | - workdir=workdir): |
4587 | - try: |
4588 | - self.import_applied_spi(srcpkg_information) |
4589 | - except DownloadError: |
4590 | - # it is non-fatal for a PPA to not have files for older |
4591 | - # publishes |
4592 | - pass |
4593 | - except NoPublicationHistoryException: |
4594 | - logging.warning("No publication history found for %s in %s. ", |
4595 | - self.pkgname, ppa) |
4596 | - except: |
4597 | - logging.error("Unable to import patches-applied to %s", ppa) |
4598 | - |
4599 | - os.chdir(oldcwd) |
4600 | - |
4601 | - self.local_repo.garbage_collect() |
4602 | - |
4603 | - if no_push: |
4604 | - logging.info('Not pushing to remote as specified') |
4605 | - else: |
4606 | - # what are appropriate refspects to push and where? |
4607 | - pass |
4608 | + except: |
4609 | + logging.error("Unable to import patches-applied to %s", ppa) |
4610 | + |
4611 | + os.chdir(oldcwd) |
4612 | + |
4613 | + repo.garbage_collect() |
4614 | + |
4615 | + |
4616 | +def parse_args(subparsers=None, base_subparsers=None): |
4617 | + kwargs = dict(description='Update a launchpad git tree based upon ' |
4618 | + 'the state of a PPA archive', |
4619 | + epilog='PPA must be specified as ppa:<name>', |
4620 | + formatter_class=argparse.ArgumentDefaultsHelpFormatter |
4621 | + ) |
4622 | + if base_subparsers: |
4623 | + kwargs['parents'] = base_subparsers |
4624 | + if subparsers: |
4625 | + parser = subparsers.add_parser('import-ppa', **kwargs) |
4626 | + parser.set_defaults(func=cli_main) |
4627 | + else: |
4628 | + parser = argparse.ArgumentParser(**kwargs) |
4629 | + parser.add_argument('ppa', type=str, |
4630 | + help='Name of PPA to update in the git tree') |
4631 | + parser.add_argument('package', type=str, |
4632 | + help='Which package to update in the git tree') |
4633 | + parser.add_argument('-o', '--lp-owner', type=str, |
4634 | + help='Which launchpad user\'s tree to update', |
4635 | + default='usd-import-team') |
4636 | + parser.add_argument('-l', '--lp-user', type=str, |
4637 | + help='Specify the Launchpad user to use', |
4638 | + default=getpass.getuser()) |
4639 | + parser.add_argument('--dl-cache', type=str, |
4640 | + help=('Cache directory for downloads. Default is to ' |
4641 | + 'put downloads in <directory>/.git/%s' % |
4642 | + CACHE_PATH |
4643 | + ), |
4644 | + default=argparse.SUPPRESS) |
4645 | + parser.add_argument('--no-clean', action='store_true', |
4646 | + help='Do not clean the temporary directory') |
4647 | + parser.add_argument('-d', '--directory', type=str, |
4648 | + help='Use git repository at specified location rather ' |
4649 | + 'than a temporary directory (implies --no-clean). ' |
4650 | + 'Will create the local repository if needed.', |
4651 | + default=argparse.SUPPRESS |
4652 | + ) |
4653 | + parser.add_argument( |
4654 | + '--skip-orig', |
4655 | + action='store_true', |
4656 | + help=argparse.SUPPRESS, |
4657 | + ) |
4658 | + if not subparsers: |
4659 | + return parser.parse_args() |
4660 | + return 'import-ppa - %s' % kwargs['description'] |
4661 | + |
4662 | +def cli_main(args): |
4663 | + no_clean = args.no_clean |
4664 | + try: |
4665 | + directory = args.directory |
4666 | + no_clean = True |
4667 | + except AttributeError: |
4668 | + directory = None |
4669 | + try: |
4670 | + dl_cache = args.dl_cache |
4671 | + except AttributeError: |
4672 | + dl_cache = None |
4673 | + |
4674 | + main( |
4675 | + args.ppa, |
4676 | + args.package, |
4677 | + directory, |
4678 | + args.lp_owner, |
4679 | + args.lp_user, |
4680 | + args.pullfile, |
4681 | + args.parentfile, |
4682 | + no_clean, |
4683 | + dl_cache, |
4684 | + args.proto, |
4685 | + args.retries, |
4686 | + args.retry_backoffs, |
4687 | + args.skip_orig, |
4688 | + ) |
4689 | diff --git a/gitubuntu/lint.py b/gitubuntu/lint.py |
4690 | index a67e0f3..ac503b9 100644 |
4691 | --- a/gitubuntu/lint.py |
4692 | +++ b/gitubuntu/lint.py |
4693 | @@ -23,10 +23,6 @@ except ImportError: |
4694 | logging.error("Is %s installed?", pkg) |
4695 | sys.exit(1) |
4696 | |
4697 | -__all__ = [ |
4698 | - 'GitUbuntuLint', |
4699 | -] |
4700 | - |
4701 | class LintNamespaceError(Exception): |
4702 | pass |
4703 | |
4704 | @@ -162,594 +158,636 @@ def test__derive_target_branch_string(same_remote_branch_names, |
4705 | ) |
4706 | assert target_branch_string == expected |
4707 | |
4708 | -class GitUbuntuLint: |
4709 | - def __init__(self): |
4710 | - pass |
4711 | |
4712 | - def success(self, msg, *args): |
4713 | - if self.verbose: |
4714 | - print(msg % args) |
4715 | +_verbose = False |
4716 | +def success(msg, *args): |
4717 | + if _verbose: |
4718 | + print(msg % args) |
4719 | |
4720 | - def parse_args(self, subparsers=None, base_subparsers=None): |
4721 | - kwargs = dict( |
4722 | - description="Given a commitish, perform automated checks", |
4723 | - formatter_class=argparse.RawTextHelpFormatter, |
4724 | - epilog="An exit code of 0 indicates all checks passed", |
4725 | - ) |
4726 | - if base_subparsers: |
4727 | - kwargs["parents"] = base_subparsers |
4728 | - if subparsers: |
4729 | - parser = subparsers.add_parser("lint", **kwargs) |
4730 | - parser.set_defaults(func=self.main) |
4731 | - else: |
4732 | - parser = argparse.ArgumentParser(**kwargs) |
4733 | |
4734 | - parser.add_argument("commitish", type=str, default="HEAD", nargs="?", |
4735 | - help="Commitish to lint" |
4736 | - ) |
4737 | - parser.add_argument("--lint-namespace", type=str, |
4738 | - help="The git-ref prefix under which the branches and tags " |
4739 | - "to lint can be found. If linting a remote-tracking " |
4740 | - "branch, this will be default to the remote name. If " |
4741 | - "linting a local branch, this will default to ''." |
4742 | - ) |
4743 | - parser.add_argument("--target-branch", type=str, |
4744 | - help="Target branch (typically a remote-tracking branch in pkg/) " |
4745 | - "the current branch is against. This will be derived if " |
4746 | - "possible." |
4747 | - ) |
4748 | - parser.add_argument("-d", "--directory", type=str, |
4749 | - help="Use git repository at specified location.", |
4750 | - default=os.path.abspath(os.getcwd()) |
4751 | - ) |
4752 | - if not subparsers: |
4753 | - return parser.parse_args() |
4754 | - return "lint - %s" % kwargs["description"] |
4755 | +def parse_args(subparsers=None, base_subparsers=None): |
4756 | + kwargs = dict( |
4757 | + description="Given a commitish, perform automated checks", |
4758 | + formatter_class=argparse.RawTextHelpFormatter, |
4759 | + epilog="An exit code of 0 indicates all checks passed", |
4760 | + ) |
4761 | + if base_subparsers: |
4762 | + kwargs["parents"] = base_subparsers |
4763 | + if subparsers: |
4764 | + parser = subparsers.add_parser("lint", **kwargs) |
4765 | + parser.set_defaults(func=cli_main) |
4766 | + else: |
4767 | + parser = argparse.ArgumentParser(**kwargs) |
4768 | |
4769 | - def _check_commitish_exists(self, commitish): |
4770 | - try: |
4771 | - actual_commitish_obj = self.local_repo.get_commitish( |
4772 | - commitish |
4773 | - ) |
4774 | - self.success("Verified %s exists", commitish) |
4775 | - return True |
4776 | - except KeyError: |
4777 | - warning("%s not found", commitish) |
4778 | - return False |
4779 | + parser.add_argument("commitish", type=str, default="HEAD", nargs="?", |
4780 | + help="Commitish to lint" |
4781 | + ) |
4782 | + parser.add_argument("--lint-namespace", type=str, |
4783 | + help="The git-ref prefix under which the branches and tags " |
4784 | + "to lint can be found. If linting a remote-tracking " |
4785 | + "branch, this will be default to the remote name. If " |
4786 | + "linting a local branch, this will default to ''." |
4787 | + ) |
4788 | + parser.add_argument("--target-branch", type=str, |
4789 | + help="Target branch (typically a remote-tracking branch in pkg/) " |
4790 | + "the current branch is against. This will be derived if " |
4791 | + "possible." |
4792 | + ) |
4793 | + parser.add_argument("-d", "--directory", type=str, |
4794 | + help="Use git repository at specified location.", |
4795 | + default=os.path.abspath(os.getcwd()) |
4796 | + ) |
4797 | + if not subparsers: |
4798 | + return parser.parse_args() |
4799 | + return "lint - %s" % kwargs["description"] |
4800 | + |
4801 | +def cli_main(args): |
4802 | + repo = GitUbuntuRepository(args.directory) |
4803 | + do_lint( |
4804 | + repo, |
4805 | + args.commitish, |
4806 | + args.lint_namespace, |
4807 | + args.target_branch, |
4808 | + args.verbose, |
4809 | + ) |
4810 | |
4811 | - def _check_commitish_exists_and_same(self, expected_commitish_pretty_name, |
4812 | - actual_commitish, expected_commitish, check_commit_id=False |
4813 | - ): |
4814 | - """Check @actual_commitish exists ad is the same tree as |
4815 | - @expected_commitish |
4816 | - |
4817 | - Arguments: |
4818 | - expected_commitish_pretty_name - a pretty name to print to refer to the |
4819 | - expected value |
4820 | - actual_commitish - commitish to check for |
4821 | - expected_commitish - commitish to check against |
4822 | - check_commit_id - if True, check the commit hashes match as well |
4823 | - """ |
4824 | - ret = True |
4825 | - if not self._check_commitish_exists(actual_commitish): |
4826 | - return False |
4827 | +def _check_commitish_exists(repo, commitish): |
4828 | + try: |
4829 | + actual_commitish_obj = repo.get_commitish(commitish) |
4830 | + success("Verified %s exists", commitish) |
4831 | + return True |
4832 | + except KeyError: |
4833 | + warning("%s not found", commitish) |
4834 | + return False |
4835 | |
4836 | - try: |
4837 | - expected_commitish_obj = self.local_repo.get_commitish( |
4838 | - expected_commitish |
4839 | - ) |
4840 | - except KeyError: |
4841 | - fatal("Unable to find expected commit: %s", expected_commitish) |
4842 | +def _check_commitish_exists_and_same(repo, expected_commitish_pretty_name, |
4843 | + actual_commitish, expected_commitish, check_commit_id=False |
4844 | +): |
4845 | + """Check @actual_commitish exists ad is the same tree as |
4846 | + @expected_commitish |
4847 | + |
4848 | + Arguments: |
4849 | + expected_commitish_pretty_name - a pretty name to print to refer to the |
4850 | + expected value |
4851 | + actual_commitish - commitish to check for |
4852 | + expected_commitish - commitish to check against |
4853 | + check_commit_id - if True, check the commit hashes match as well |
4854 | + """ |
4855 | + ret = True |
4856 | + if not _check_commitish_exists(repo, actual_commitish): |
4857 | + return False |
4858 | |
4859 | - actual_commitish_obj = self.local_repo.get_commitish( |
4860 | - actual_commitish |
4861 | - ) |
4862 | - if check_commit_id: |
4863 | - expected_commit_id = str(expected_commitish_obj.id) |
4864 | - actual_commit_id = str(actual_commitish_obj.id) |
4865 | - if expected_commit_id == actual_commit_id: |
4866 | - self.success("Verified %s is the same commit as %s", |
4867 | - expected_commitish_pretty_name, actual_commitish |
4868 | - ) |
4869 | - else: |
4870 | - warning("Expected %s (%s) is not the same commit as %s (%s)", |
4871 | - expected_commitish_pretty_name, expected_commit_id, |
4872 | - actual_commitish, actual_commit_id |
4873 | - ) |
4874 | - ret = False |
4875 | - expected_commit_tree_id = str( |
4876 | - expected_commitish_obj.peel(pygit2.Tree).id |
4877 | - ) |
4878 | - actual_commit_tree_id = str( |
4879 | - actual_commitish_obj.peel(pygit2.Tree).id |
4880 | - ) |
4881 | - if expected_commit_tree_id == actual_commit_tree_id: |
4882 | - self.success("Verified %s is the same tree as %s", |
4883 | + try: |
4884 | + expected_commitish_obj = repo.get_commitish(expected_commitish) |
4885 | + except KeyError: |
4886 | + fatal("Unable to find expected commit: %s", expected_commitish) |
4887 | + |
4888 | + actual_commitish_obj = repo.get_commitish(actual_commitish) |
4889 | + if check_commit_id: |
4890 | + expected_commit_id = str(expected_commitish_obj.id) |
4891 | + actual_commit_id = str(actual_commitish_obj.id) |
4892 | + if expected_commit_id == actual_commit_id: |
4893 | + success("Verified %s is the same commit as %s", |
4894 | expected_commitish_pretty_name, actual_commitish |
4895 | ) |
4896 | else: |
4897 | - error("Expected %s (%s) is not the same tree as %s (%s)", |
4898 | - expected_commitish_pretty_name, expected_commit_tree_id, |
4899 | - actual_commitish, actual_commit_tree_id |
4900 | + warning("Expected %s (%s) is not the same commit as %s (%s)", |
4901 | + expected_commitish_pretty_name, expected_commit_id, |
4902 | + actual_commitish, actual_commit_id |
4903 | ) |
4904 | ret = False |
4905 | + expected_commit_tree_id = str( |
4906 | + expected_commitish_obj.peel(pygit2.Tree).id |
4907 | + ) |
4908 | + actual_commit_tree_id = str( |
4909 | + actual_commitish_obj.peel(pygit2.Tree).id |
4910 | + ) |
4911 | + if expected_commit_tree_id == actual_commit_tree_id: |
4912 | + success("Verified %s is the same tree as %s", |
4913 | + expected_commitish_pretty_name, actual_commitish |
4914 | + ) |
4915 | + else: |
4916 | + error("Expected %s (%s) is not the same tree as %s (%s)", |
4917 | + expected_commitish_pretty_name, expected_commit_tree_id, |
4918 | + actual_commitish, actual_commit_tree_id |
4919 | + ) |
4920 | + ret = False |
4921 | |
4922 | - return ret |
4923 | + return ret |
4924 | |
4925 | - def print_hunk(self, hunk): |
4926 | - for line in hunk.lines: |
4927 | - print("%s%s" % (line.origin, line.content.rstrip())) |
4928 | +def print_hunk(hunk): |
4929 | + for line in hunk.lines: |
4930 | + print("%s%s" % (line.origin, line.content.rstrip())) |
4931 | |
4932 | - def print_patch(self, patch): |
4933 | - for hunk in patch.hunks: |
4934 | - self.print_hunk(hunk) |
4935 | +def print_patch(patch): |
4936 | + for hunk in patch.hunks: |
4937 | + print_hunk(hunk) |
4938 | |
4939 | - def _check_deconstruct_to_logical_patch(self, patch): |
4940 | - if "debian/changelog" in patch.delta.new_file.path: |
4941 | - self.success("Verified debian/changelog is in diff between " |
4942 | - "deconstruct and logical" |
4943 | - ) |
4944 | - return True |
4945 | - if "debian/control" in patch.delta.new_file.path: |
4946 | - ret = True |
4947 | - for hunk in patch.hunks: |
4948 | - _ret = True |
4949 | - for line in hunk.lines: |
4950 | - if line.origin == " ": |
4951 | +def _check_deconstruct_to_logical_patch(patch): |
4952 | + if "debian/changelog" in patch.delta.new_file.path: |
4953 | + success("Verified debian/changelog is in diff between " |
4954 | + "deconstruct and logical" |
4955 | + ) |
4956 | + return True |
4957 | + if "debian/control" in patch.delta.new_file.path: |
4958 | + ret = True |
4959 | + for hunk in patch.hunks: |
4960 | + _ret = True |
4961 | + for line in hunk.lines: |
4962 | + if line.origin == " ": |
4963 | + continue |
4964 | + content = line.content.strip() |
4965 | + if line.origin == "-": |
4966 | + if content.startswith("Maintainer:"): |
4967 | + continue |
4968 | + if line.origin == "+": |
4969 | + if ( |
4970 | + content.startswith("Maintainer:") and |
4971 | + content.endswith("<ubuntu-devel-discuss@lists.ubuntu.com>") |
4972 | + ): |
4973 | + continue |
4974 | + if content.startswith("XSBC-Original-Maintainer:"): |
4975 | continue |
4976 | - content = line.content.strip() |
4977 | - if line.origin == "-": |
4978 | - if content.startswith("Maintainer:"): |
4979 | - continue |
4980 | - if line.origin == "+": |
4981 | - if ( |
4982 | - content.startswith("Maintainer:") and |
4983 | - content.endswith("<ubuntu-devel-discuss@lists.ubuntu.com>") |
4984 | - ): |
4985 | - continue |
4986 | - if content.startswith("XSBC-Original-Maintainer:"): |
4987 | - continue |
4988 | - _ret = False |
4989 | - if not _ret: |
4990 | - error("unexpected differences between logical " |
4991 | - "and deconstruct tags" |
4992 | - ) |
4993 | - self.print_hunk(hunk) |
4994 | - ret = _ret and ret |
4995 | - if ret: |
4996 | - self.success("Verified only update-maintainer changes " |
4997 | - "are in diff between deconstruct and logical to " |
4998 | - "debian/control" |
4999 | + _ret = False |
5000 | + if not _ret: |
FAILED: Continuous integration, rev:bd7e040556c d86c24859dea6af 7d56fbb92954ba /jenkins. ubuntu. com/server/ job/git- ubuntu- ci/24/
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Tests
FAILED: Build
Click here to trigger a rebuild: /jenkins. ubuntu. com/server/ job/git- ubuntu- ci/24/rebuild
https:/