lp:~racb/usd-importer

Owned by Robie Basak
Get this repository:
git clone https://git.launchpad.net/~racb/usd-importer
Only Robie Basak can upload to this repository. If you are Robie Basak please log in for upload directions.

Branches

Name Last Modified Last Commit
ipc-concurrency 2020-06-29 13:31:23 UTC 2020-06-29
wip: move IPC CLI endpoints to subcommands

Author: Robie Basak
Author Date: 2020-06-26 11:43:57 UTC

wip: move IPC CLI endpoints to subcommands

importer-notes-newline 2020-06-02 14:49:36 UTC 2020-06-02
Add terminating newline to importer notes

Author: Robie Basak
Author Date: 2020-06-02 14:46:57 UTC

Add terminating newline to importer notes

It's inconveninent for the importer note to not have a terminating
newline. This was an oversight.

From some quick research:

If I use "git notes add -F", a terminating newline is added whether the
file had a terminating newline before or not. This can be seen in a
checkout of refs/notes/commits. However a checkout of the importer notes
currently does not have terminating newlines, so evidently pygit2 is not
performing that same normalization. Therefore there cannot be any harm
in doing so.

The changelog notes already are being provided with a terminating
newline, so there is no change there and none is required.

rich-history-edge-cases 2020-05-28 16:30:19 UTC 2020-05-28
Allow fast-forwards when porting rich history

Author: Robie Basak
Author Date: 2020-05-28 13:07:18 UTC

Allow fast-forwards when porting rich history

Currently we unconditionally port forward rich history, without checking
if parent commits have mutated. This is usually fine for a one-off
reimport which we are only conducting because we are expecting the
commits will mutate anyway. However in the case that commits have not
mutated (such as an immediate reimport of a reimport), this
unnecessarily mutates rich history by bumping forward the committer
dates.

Using "git cherry-pick --ff" prevents this by doing exactly what we
want: a cherry-pick is instead "fast forwarded" in the case that the
parent commit has not mutated.

This makes a reimport of a reimport stable in terms of commit and tag
mutations.

LP: #1876329

reimport-service 2020-05-27 10:43:27 UTC 2020-05-27
Convert import request structure to namedtuple

Author: Robie Basak
Author Date: 2020-05-26 14:15:32 UTC

Convert import request structure to namedtuple

Converting the tuple of (str, bool) to an explicit namedtuple makes the
code more readable.

release-process-changes-2 2020-05-12 12:54:01 UTC 2020-05-12
Release process: add service restart instructions

Author: Robie Basak
Author Date: 2020-05-12 12:54:01 UTC

Release process: add service restart instructions

We probably shouldn't leave the importer service running in a state
where a restart or reboot could change behaviour, so we should restart
the service as part of an update.

fix-rich-history-unescaped-patch 2020-05-11 13:31:44 UTC 2020-05-11
Fix docstring

Author: Robie Basak
Author Date: 2020-05-07 14:33:59 UTC

Fix docstring

This docstring was incorrect. The function uses a repo fixture, not a
repo_factory fixture, so update the docstring to match.

fix-changelog-timezone-parsing 2020-05-06 16:36:30 UTC 2020-05-06
Fix changelog timezone parsing

Author: Robie Basak
Author Date: 2020-05-06 16:30:09 UTC

Fix changelog timezone parsing

I noticed that in the reimport of gpsd, cgit reported

    2020-03-18 16:52:59 +0100

...whereas the changelog timestamp said

    Wed, 18 Mar 2020 15:52:59 +0100

...so there appeared to be something wrong with the parsing.

It turns out that the timezone component in the changelog was being
ignored because Python's time.strptime() is defined to ignore it. We
must use datetime.strptime() instead. The parsing is therefore entirely
adjusted to use only a datetime object.

This wasn't being caught in tests because the expected test value was
wrong - presumably calculated with the same mistake.

I have verified that, based on the date string given in
gitubuntu/changelog_tests/test_date{1,2}, using:

    date +%s -d "Mon, 12 May 2016 08:14:34 -0700"

...does report the value of 1463066074, so all tests are adjusted
accordingly. I have also verified that the above command does mutate the
answer if the timezone is mutated, so it does appear to be correctly
timezone-aware.

While there still a risk that I have calculated the expected value
wrong, hopefully the above test migitates this sufficiently. We do
already have adequate test coverage that will prevent regressions in
Changelog.git_authorship(); the problem was that the expected values in
those tests were wrong and this is now fixed.

fix-remote-add-changelog-notes 2020-05-06 13:59:19 UTC 2020-05-06
Don't add changelog notes fetch refspec for !pkg

Author: Robie Basak
Author Date: 2020-05-06 13:56:45 UTC

Don't add changelog notes fetch refspec for !pkg

The changelog notes fetch respec should only be added for the main 'pkg'
remote (the "base remote" in the code), as other remotes are not
expected to have these.

This fixes a bug:

$ git ubuntu remote add <anything other than pkg>
fatal: Couldn't find remote ref refs/notes/commits

We don't currently have test coverage for the affected methods, but I
have tried to reduce tech debt somewhat at least by adding docstrings to
clarify intended behaviour.

Thanks to Andreas Hasenack for finding this in the edge channel early.

preserve-mps-on-reimport 2020-05-01 13:28:41 UTC 2020-05-01
Preserve MPs on reimport

Author: Robie Basak
Author Date: 2020-04-24 14:14:19 UTC

Preserve MPs on reimport

When reimporting, instead of deleting a git repository before repushing
it, we can force push all our new refs and ensure that refs not being
pushed get deleted using --prune. This creates the same effect but
allows us to preserve old merge proposals.

There is a possibility that some configuration in the git repository
will be left behind from a previous import. However all configuration
(eg. the HEAD pointer) is reset by the importer on every run, so
everything that the importer touches will get fixed. Anything else can
be considered out of scope for --reimport to achieve.

The default repository for the source package will therefore not be
reset; if it was set to the old repository, it will continue to point to
the "new" one. This is actually what we want, so is fine.

try 2020-05-01 13:24:33 UTC 2020-05-01
Merge branch 'spec-overrides' into try

Author: Robie Basak
Author Date: 2020-05-01 13:24:33 UTC

Merge branch 'spec-overrides' into try

fix-head-detection 2020-05-01 12:01:47 UTC 2020-05-01
Fix Debian/Ubuntu HEAD branch detection

Author: Robie Basak
Author Date: 2020-04-28 16:51:31 UTC

Fix Debian/Ubuntu HEAD branch detection

repo.references provides a list of pygit2.Reference objects, not strings
of their names. Just try the lookup method directly instead.

master 2020-04-30 13:09:49 UTC 2020-04-30
Merge branch 'document-testing-strategy'

Author: Robie Basak
Author Date: 2020-04-30 13:09:49 UTC

Merge branch 'document-testing-strategy'

document-testing-strategy 2020-04-30 08:38:55 UTC 2020-04-30
Document testing strategy

Author: Robie Basak
Author Date: 2020-04-27 14:36:57 UTC

Document testing strategy

clone-default-repository 2020-04-29 14:36:55 UTC 2020-04-29
Clone the default target repository directly

Author: Robie Basak
Author Date: 2020-04-29 12:08:51 UTC

Clone the default target repository directly

We intend to move the imported repositories to a separate Launchpad team
whose only member will be the importer bot. This will break existing
users' pkg remote URLs, since they currently have ~usd-import-team
hardcoded.

Instead, change behaviour so that "git clone" will configure the "pkg"
remote to the URL for the default repository for a package only. We will
ensure out-of-band that the default repository is pointed correctly for
all imported packages.

Then when the default repository changes as expected, users who used
"git ubuntu clone" after this change will not be affected; they will
seamlessly access the new repository location at the Launchpad end.

The GitUbuntuRepository.add_base_remotes() is currently only called from
two places: once from importer.py, and once from clone.py. We want the
importer to always use an explicit owner, but clone not to do so in
order that the default is used in that case. It just so happens that the
call from importer.py already always specifies repo_owner, and clone.py
already does not, so the callers are already doing what we need and do
not need to be adjusted.

clean-up-changelog-notes 2020-04-29 14:36:30 UTC 2020-04-29
Clean up changelog note output

Author: Robie Basak
Author Date: 2020-04-29 09:49:07 UTC

Clean up changelog note output

Now that the changelog entries go into a separate git note, we no longer
need the leading newlines. This tidies up the "git log" output for
better readabilty for humans.

adjust-name-and-email-to-spec 2020-04-29 14:36:06 UTC 2020-04-29
Adjust name and email to match new spec

Author: Robie Basak
Author Date: 2020-04-29 09:41:47 UTC

Adjust name and email to match new spec

Settling on this now is required for hash stability.

clear-notes-before-reimport 2020-04-29 14:35:13 UTC 2020-04-29
Clear notes before reimport

Author: Robie Basak
Author Date: 2020-04-28 16:51:03 UTC

Clear notes before reimport

Fixes bug that a reimport will create notes that already exist
otherwise.

import-notes 2020-04-24 09:19:22 UTC 2020-04-24
Make tagging hash-stable

Author: Robie Basak
Author Date: 2020-04-22 15:34:59 UTC

Make tagging hash-stable

Now that we have importer notes, adjust the created import tags so that
they are hash-stable.

This does not include applied tags, which will be made stable at a later
point.

rich-history-preservation 2020-04-23 16:48:38 UTC 2020-04-23
Add rich history preservation

Author: Robie Basak
Author Date: 2020-04-20 17:51:02 UTC

Add rich history preservation

Add a new facility to permit rich history to be preserved across
reimports (when using "git ubuntu import --reimport"). Previous rich
history is exported to a temporary directory prior to the reimport.
During import, the rich history importer is then given the opportunity
to reconstruct an upload tag "just in time" before it is used.

Testing the importer end-to-end for this is not currently practical
since we have no tests for the --reimport function at all. We have
pragmatically concluded that it is sufficient to functionally test the
new rich history code, since --reimport will generally only be used with
manual supervision and the likely bugs in the importer code itself are
that rich history preservation won't work at all, or the importer will
fail entirely, and so will be easily noticed.

I have also added some limited additional tests to bring coverage of the
new gitubuntu.rich_history up to 100%.

interleave-publication-dates 2020-04-21 02:25:27 UTC 2020-04-21
Interleave LP publications during consumption

Author: Robie Basak
Author Date: 2020-04-17 14:59:27 UTC

Interleave LP publications during consumption

The specification requires that we consider Debian and Ubuntu
publications in Launchpad simultaneously, ordering by date_created
across them both. This implements this requirement with
interleave_launchpad_versions_published_after() to wrap multiple calls
of launchpad_versions_published_after() and interleaving the results as
they are returned. The caller is adjusted to use this instead of
iterating over the two distributions separately.

LP: #1851361

commit-message-spec 2020-04-09 12:32:34 UTC 2020-04-09
Add explanatory comments on changelog notes

Author: Robie Basak
Author Date: 2020-04-09 12:31:45 UTC

Add explanatory comments on changelog notes

bump-ubuntu-dev-tools 2020-03-31 13:24:43 UTC 2020-03-31
Bump ubuntu-dev-tools Python 3 version

Author: Robie Basak
Author Date: 2020-03-31 13:24:43 UTC

Bump ubuntu-dev-tools Python 3 version

Bump ubuntu-dev-tools to 0.176, but only for Python 3. The Python 2
installation will remain as-is, since 0.176 doesn't appear to work with
Python 2.

This fixes an import bug where the necessary fix was in ubuntu-dev-tools.

The Python 2 installation is presumed needed by some other tooling
called by git-ubuntu via the shell. I expect this can go away when we
move the snap base to core20, which is currently deferred for the
future. In the meantime, we exclude the Python 2 version from installing
into some common directories to avoid conflicts. This does lead to a
somewhat frankenstein installation of ubuntu-dev-tools, but I don't
think it will create a problem in practice until we can move to core20.

LP: #1860456

fix-unapplied-commits-and-tags 2020-03-27 17:46:44 UTC 2020-03-27
Rewrite unapplied commit creation and tagging

Author: Robie Basak
Author Date: 2020-03-19 03:08:31 UTC

Rewrite unapplied commit creation and tagging

This is a major functional change in the behaviour of the importer
algorithm. The output of the importer will now change to match the new
spec.

The impacted areas are the ones that find any previous commit to use,
create a commit as necessary, and tag accordingly.

Accepted upload tags are now adopted directly rather than creating a new
commit with the upload tag as its parent.

Reimport tags are now fully incorporated, eliminating the bugs around
edge cases where Launchpad stores different uploads in Debian and Ubuntu
against the same package version string.

The following bugs are fixed by this change. I have verified that for
each bug an adequate existing test exists or a test would not be
applicable (for example because orphan tags no longer exist).

LP: #1734883
LP: #1753800
LP: #1754194
LP: #1754507
LP: #1754706
LP: #1755247
LP: #1761331
LP: #1761332
LP: #1772462

fix-importer-tests 2020-03-20 17:21:40 UTC 2020-03-20
Fix importer test validation repo commits

Author: Robie Basak
Author Date: 2020-03-19 03:42:19 UTC

Fix importer test validation repo commits

These tests expect new commits to be created, so based on "reused" being
False, the expected results should have "new" commit messages and not
original ones.

This test continues to xfail and will be fixed in a future commit. This
is why the problem wasn't noticed previously.

renameabledir-must-exist 2020-03-19 00:46:26 UTC 2020-03-19
RenamableDir: check that the supplied file exists

Author: Robie Basak
Author Date: 2020-03-19 00:46:26 UTC

RenamableDir: check that the supplied file exists

This was required before anyway; this makes it explicit that it is
checked for by the constructor, rather than just being an assertion.

nofollow-broken-symlinks 2020-03-17 14:17:07 UTC 2020-03-17
Fix RenameableDir instantiation on broken symlinks

Author: Robie Basak
Author Date: 2020-03-16 09:47:38 UTC

Fix RenameableDir instantiation on broken symlinks

RenameableDir may be instantiated with a path that exists on the
filesystem but that is a broken symlink. In this case the "exists"
assertion should not follow the symlink, so it should use
os.path.lexists(), not os.path.exists().

This fixes the corresponding test, so we drop the xfail.

LP: #1864255

changelog-parents 2020-02-12 15:35:17 UTC 2020-02-12
Add further docstrings and comments

Author: Robie Basak
Author Date: 2020-02-12 14:44:28 UTC

Add further docstrings and comments

Add docstrings to some functions which previously did not have any. One
related comment is also added.

Some of these docstrings expose some shortcomings in the implementation,
but in the interest of making progress the code is documented as-is, and
improvements to the code are left for the future.

xfail-style 2020-02-11 11:15:47 UTC 2020-02-11
Add style notes on the use of xfail

Author: Robie Basak
Author Date: 2020-02-10 17:14:07 UTC

Add style notes on the use of xfail

recursive-dot-git-escaping 2020-02-10 16:52:03 UTC 2020-02-10
.git escaping: fix failing case

Author: Robie Basak
Author Date: 2020-01-09 16:03:57 UTC

.git escaping: fix failing case

Traverse the entire directory for recursive escapes;
sorted_subpaths_to_rename is already filtered so is not complete itself.

This fixes the recursive traversal that looks for entries that require
escaping.

This causes the previous failing test to pass, so we must remove the
associated xfail marker.

applied-enum 2020-02-04 14:20:26 UTC 2020-02-04
Collapse dual (un)applied code using an enum

Author: Robie Basak
Author Date: 2020-02-04 11:29:10 UTC

Collapse dual (un)applied code using an enum

In various places we have two implementations of near-identical code
where the only difference is whether they act on unapplied or applied
branches, tags or structure. Ultimately they do need to be handled
differently at some point, but most of the code in the middle of the
call graph does not need to care.

To avoid code duplication we define a PatchState enum and can then drop
half of the middling code by simply passing through the "unapplied-ness"
or "applied-ness" as a parameter.

The following functions are removed from gitubuntu.git_repository.
Instead, the unapplied previous equivalent should be used with
patch_state=PatchState.APPLIED:

applied_reimport_tag_prefix
applied_reimport_tag
applied_tag

The following methods are removed from
gitubuntu.git_repository.GitUbuntuRepository. Instead, the unapplied
previous equivalent should be used with patch_state=PatchState.APPLIED:

get_applied_reimport_tag
get_all_applied_reimport_tags

A grep suggests that the only callers are importer.py and
importer_test.py; these are refactored as part of this change, dropping
duplication where there was any.

"git ubuntu tag" is the obvious CLI command that might have been
affected, but it doesn't appear to need any of these functions or
methods. Perhaps it should, but that's a job for a future refactoring.

This change does not cover every instance of code duplication or where
refactoring to use the new enum might be appropriate; we merely fix the
code that needed touching anyway, leaving other refactoring for the
future.

This change should not result in any functional changes.

self-test-diagnostics 2020-01-28 13:49:21 UTC 2020-01-28
self-test: be more explicit when failing

Author: Robie Basak
Author Date: 2020-01-28 11:14:05 UTC

self-test: be more explicit when failing

The script will exit non-zero if any of a set of tests fail, but it
isn't necessarily obvious from the output what was a warning and what
caused the failure. Let's make this obvious in the output. In all cases
where we set a future failure, explicitly tell the caller via stdout.

unpin-lazr-restfulclient 2020-01-28 11:05:19 UTC 2020-01-28
Drop pin on lazr.restfulclient

Author: Robie Basak
Author Date: 2020-01-28 11:05:19 UTC

Drop pin on lazr.restfulclient

lazr.restfulclient 0.14.3 has been released, causing the self-test
error:

  gitubuntu 0.9.0 has requirement lazr.restfulclient==0.14.2, but you have lazr.restfulclient 0.14.3.

There is no need to pin this, so drop the pin.

phasing-defaults 2020-01-16 11:56:31 UTC 2020-01-16
Change main phasing default to 100%

Author: Robie Basak
Author Date: 2020-01-16 11:56:31 UTC

Change main phasing default to 100%

The official importer instance (the only known instance) had been at
100% phasing of main for months, but accidentally reverted to the old
default of 1% as the configuration was lost when it was converted to a
proper service.

Let's move the default to match the production instance to reduce
the likelyhood of mistakes.

commit-authorship 2020-01-06 14:49:43 UTC 2020-01-06
Rewrite commit_tree_hash into commit_source_tree

Author: Robie Basak
Author Date: 2019-12-09 11:44:32 UTC

Rewrite commit_tree_hash into commit_source_tree

To fix LP: #1764814, the updated specification changes the definition of
how commit authorship is derived. This is a rewrite of
GitUbuntuRepository.commit_tree_hash() to meet the new specification.

Since more accurately we're committing a tree that must contain a Debian
package source, the method is renamed to commit_source_tree() in this
rewrite.

fallback_author and fallback_date are no longer necessary and are
removed. Instead we use a commit_date parameter which specifies the
required date of the commit. According to the new specification this
date must come from the underlying Launchpad object. Since this method
operates on a low level git tree instead of at the Launchpad data
structure level, the commit date is the caller's responsiblity to obtain
and provide; this method will simply use the date given.

git repository hashes are now passed and returned as pygit2.Oid objects
as we move to this type as the usual standard.

The committer name and email are defined in the specification via
gitubuntu/spec.py, which is added as a new file. This file forms part of
the specification itself, and its contents are separated in its own file
to avoid inadvertent changes to the specification.

The GitUbuntuRepository methods get_commit_environment() and
get_commit_authorship() were only used by the old commit_tree_hash()
method and are removed. Instead we use Changelog.git_commit_authorship()
as added in the previous commit.

Since we have changed the definition of
commit_tree_hash()/commit_source_tree() in the rewrite, all callers are
updated to the new method name, new parameter names and types, and the
calculation of fallback_ is removed.

In tests, it is further no longer necessary to patch the
get_commit_environment() method, which no longer exists. Instead,
repo_builder.Commit is adjusted to default to using the same metadata as
the specification defines. This results in the same behaviour as before
when the test suite is run, but is cleaner since less patching is
required.

importer-add-tests 2019-12-10 11:16:55 UTC 2019-12-10
Add importer tests

Author: Robie Basak
Author Date: 2019-11-04 14:29:38 UTC

Add importer tests

style-on-imports 2019-12-06 08:02:50 UTC 2019-12-06
Add style convention on import statements

Author: Robie Basak
Author Date: 2019-12-06 08:02:50 UTC

Add style convention on import statements

This is what we've been following anyway, but without explicit
documentation.

pylint-bug-workaround 2019-10-15 16:18:27 UTC 2019-10-15
Work around pylint/astroid bug

Author: Robie Basak
Author Date: 2019-10-15 16:18:27 UTC

Work around pylint/astroid bug

astroid appears to have a bug causing pylint to return a false positive:
https://github.com/PyCQA/pylint/issues/3137

This was apparently introduced in astroid 2.3, so hold back from using
it to work around the problem since this is causing our CI to fail.

pylint 2.4 requires astroid 2.3 without declaring it
(https://github.com/PyCQA/pylint/issues/3022) so additionally we also
need to hold back from using pylint 2.4 to avoid hitting this other
failure.

fix_derive_codename_from_series.3 2019-08-05 13:07:30 UTC 2019-08-05
fixup! source_information: Fix ubuntu series

Author: Robie Basak
Author Date: 2019-08-05 12:51:19 UTC

fixup! source_information: Fix ubuntu series

Drop xfail now that the bug is fixed

blacklist-wheezy 2019-04-25 16:42:23 UTC 2019-04-25
Blacklist wheezy as an active series

Author: Robie Basak
Author Date: 2019-04-25 12:25:28 UTC

Blacklist wheezy as an active series

optimise-api-calls 2019-04-25 09:16:36 UTC 2019-04-25
Cache some Launchpad object fetches

Author: Robie Basak
Author Date: 2019-04-23 11:45:33 UTC

Cache some Launchpad object fetches

As we iterate through Launchpad's source_package_publication_history
records, we keep looking up their distro_series objects via
GitUbuntuSourcePackageInformation.distribution.name and suchlike. This
results in a very large number of API calls to fetch the same
distro_series object from Launchpad every time, causing a severe
degradation in performance.

Since distribution_link is the same every time, we can instead key a
cache on this, eliminating the excessive number of API calls. The same
applies to some other similar calls.

As far as I can tell we've always made this excessive number of calls
here, but in practice something recently changed in the amount of time
these calls are taking, causing import performance to drop virtually to
a standstill. This optimisation restores reasonable performance again.

This introduces a new dependency on cachetools, which seems to be the
cleanest caching solution without having to reimplement it inside
git-ubuntu. python3-cachetools is available in Xenial and the module
seems to be stable upstream, so I hope this addition will not cause us
too much trouble.

unpin-pylint 2018-11-29 15:37:27 UTC 2018-11-29
Friendlier self-test output

Author: Robie Basak
Author Date: 2018-11-29 15:31:37 UTC

Friendlier self-test output

If we change directory to the location of the gitubuntu Python module
first, the commands to pylint and pytest are simpler and additionally
has the advantage that the output from pytest is nicer.

httplib2-re-slash 2018-11-08 11:51:47 UTC 2018-11-08
Merge remote-tracking branch 'smoser/fix/1801168-build-when-no-patches'

Author: Robie Basak
Author Date: 2018-11-08 11:51:47 UTC

Merge remote-tracking branch 'smoser/fix/1801168-build-when-no-patches'

scriptutils-dry-run 2018-11-06 11:34:48 UTC 2018-11-06
Drop use of dry_run that is no longer defined

Author: Robie Basak
Author Date: 2018-11-06 11:34:48 UTC

Drop use of dry_run that is no longer defined

This fixes a regression caused by
d17507e57d60f744540088d6a1d4f56a80b92970.

cosmic-keyring 2018-11-05 16:29:28 UTC 2018-11-05
Bump ubuntu-keyring keyrings used in the snap

Author: Robie Basak
Author Date: 2018-11-05 14:41:49 UTC

Bump ubuntu-keyring keyrings used in the snap

LP: #1801725

importer-service-db 2018-11-01 16:55:43 UTC 2018-11-01
Mark "running" imports as errored on restart

Author: Robie Basak
Author Date: 2018-10-30 16:56:18 UTC

Mark "running" imports as errored on restart

fix-keyring-pypi-breakage 2018-11-01 14:17:07 UTC 2018-11-01
Pin keyring version to work around regression

Author: Robie Basak
Author Date: 2018-11-01 13:02:28 UTC

Pin keyring version to work around regression

importer-service-datadir 2018-10-23 12:28:41 UTC 2018-10-23
Use persistent state directory

Author: Robie Basak
Author Date: 2018-10-19 12:58:44 UTC

Use persistent state directory

Move the directory in which we keep persistent state into the one
specified by --data-directory, defaulting to /var/local/git-ubuntu.

This directory must exist.

This moves the previously used directory /tmp into somewhere that can
be expected to persist. This impacts git-ubuntu importer operators,
since they now need to make sure that a persistent area is ready before
use.

This applies to the importer service only. Users of the "git-ubuntu" CLI
tool, as opposed to the operators of the importer, are not affected.

gawk-readline-fix 2018-10-12 09:44:19 UTC 2018-10-12
Fix call into core snap awk binary

Author: Robie Basak
Author Date: 2018-10-12 09:44:19 UTC

Fix call into core snap awk binary

The core snap's loader is tied to its libc. usr/bin/awk inside the core
snap refers to /lib64/ld-linux-x86-64.so.2, which is therefore incorrect
when called from a classic snap.

Add a wrapper for awk which calls awk in the core snap correctly,
through its own loader.

Separately, we also need to set LD_LIBRARY_PATH so that the loader can
find the shared libraries supplied by the core snap that satisfy its own
dependencies, rather than look at the host system or into our own snap
which may not have them. This was the original bug. Fixing this revealed
the problem above. Both issues are resolved by wrapping this call.

This does raise some questions such as what it means for a classic snap
to be making use of a core snap's binaries, and whether this is an
appropriate thing to do given that the core snap's binaries all specify
the "wrong" loader path in the classic case since the core snap won't be
mounted at the root of the filesystem. This might need some refactoring
of how the git-ubuntu snap works in the future, but for now we can fix
the awk case to fix the current regression in master.

With thanks to Michael Vogt.

LP: #1796017

gawk-readline-fix-wip 2018-10-03 12:19:32 UTC 2018-10-03
Add awk integration test

Author: Robie Basak
Author Date: 2018-10-02 16:58:36 UTC

Add awk integration test

keyring-backend-fix 2018-09-28 15:47:00 UTC 2018-09-28
Drop keyrings.alt version restriction

Author: Robie Basak
Author Date: 2018-09-27 15:06:44 UTC

Drop keyrings.alt version restriction

The launchpadlib part requires "keyring" with no version restriction. By
requiring a particular version here, we're relying that some newest
version of keyring will work with this particular keyrings.alt, which
fails with keyring 15.0.0 vs. keyrings.alt 2.3. Since launchpadlib
already has no version restriction, do the same for keyrings.alt. Then
at least if "pip3 install" works, so should the snap.

Alternatively we could have added a version restriction to the
launchpadlib part, but at the moment that pattern isn't used anywhere,
and the requirements are taken from launchpadlib's setup.py with no
patching. If we want to restrict versions some other way, then we'll
need a major reworking of how the snap is put together.

Now the test checking that keyring can find a functional backend should
always pass.

LP: #1794041

improve-tag-dirty-tree-detection 2018-09-20 16:19:56 UTC 2018-09-20
tag: improve dirty tree detection

Author: Robie Basak
Author Date: 2018-09-20 16:18:48 UTC

tag: improve dirty tree detection

If raw_repo.status() returns entries that are GIT_STATUS_IGNORED then
they don't represent the tree being dirty, so ignore this case. This
fixes a case where "git ubuntu tag" refuses to work even when a working
tree is clean.

Related bugs:

https://bugs.launchpad.net/usd-importer/+bug/1687940
https://bugs.launchpad.net/usd-importer/+bug/1739000

proposed 2018-07-10 12:08:46 UTC 2018-07-10
Merge branch 'changelog-assertions'

Author: Robie Basak
Author Date: 2018-07-10 12:01:49 UTC

Merge branch 'changelog-assertions'

Some fixups were needed for the merge. The tests move from
git_repository.py to git_repository_test.py, one import gets removed
from git_repository.py (no longer needed from git_repository_test.py),
and a couple of references in new test code get prepended by "target.".

test-structure-consistency 2018-06-14 12:52:26 UTC 2018-06-14
Rename test files

Author: Robie Basak
Author Date: 2018-06-14 12:52:26 UTC

Rename test files

This is a style change: we're moving to keeping tests in foo_test.py.

changelog-assertions 2018-06-04 14:06:07 UTC 2018-06-04
Fix edge case changelog assertion check

Author: Robie Basak
Author Date: 2018-06-01 15:57:01 UTC

Fix edge case changelog assertion check

We should also check the length since zip()'s output is the shortest of
its two inputs. This fixes the associated test.

I've also simplified the associated error path code, since I think it's
sufficient to just raise an exception and not swallow it higher up the
stack. Problems with changelog parsing can be reproduced later easily
enough.

importer-cleanup-tmpdir-always 2018-05-21 10:08:17 UTC 2018-05-21
Drop inferred log message

Author: Robie Basak
Author Date: 2018-05-15 16:48:07 UTC

Drop inferred log message

This message was left behind to avoid a functional change in a previous
refactoring.

Unfortunately it has now become quite disconnected from the
implementation of logic that it is logging. It's at risk of becoming
wrong in the future if the implementation elsewhere is intentionally
changed.

Since it doesn't seem that important, drop it to avoid this happening.

snap-lazr-path 2018-05-16 13:11:56 UTC 2018-05-16
snap: fix lazr.restfulclient stage path

Author: Robie Basak
Author Date: 2018-05-16 13:11:56 UTC

snap: fix lazr.restfulclient stage path

The old path was incorrect, but snapcraft somehow previously worked with
it, perhaps due to a regexp match or similar. The current snapcraft does
not. This change fixes the current snapcraft build failure.

Thanks to Nish for identifying the root cause and fix.

snap-vendoring-updates 2018-04-27 11:01:34 UTC 2018-04-27
Document external dependency management policy

Author: Robie Basak
Author Date: 2018-04-27 11:01:34 UTC

Document external dependency management policy

External dependencies make the build more painful for developers and CI
in different environments, so we've agreed to prefer avoiding external
dependencies where possible.

apt-repo 2018-02-27 20:26:22 UTC 2018-02-27
Add gitubuntu/apt_repo_test/ to the snap

Author: Robie Basak
Author Date: 2018-02-27 20:26:22 UTC

Add gitubuntu/apt_repo_test/ to the snap

This is needed for the in-snap self test.

test-fixes 2018-02-23 13:05:22 UTC 2018-02-23
tests: changelog test files should move to pkg_resources

Author: Nish Aravamudan
Author Date: 2018-02-22 03:08:33 UTC

tests: changelog test files should move to pkg_resources

In order to access the changelog files regardless of running from the
Git repository or the snap, we need to have the test files in the
package and available via pkg_resources.

This should be done for all future testing infrastructure as we add it,
as well.

source-builder-changelog-versions 2018-02-21 10:21:09 UTC 2018-02-21
SourceSpec/SourceFiles: add changelog_versions

Author: Robie Basak
Author Date: 2018-02-12 15:58:02 UTC

SourceSpec/SourceFiles: add changelog_versions

This allows a Source to be created with a predefined list of versions in
debian/changelog. This is useful, for example, for control of changelog
parents during testing.

decouple-gitubunturepository 2018-02-19 14:02:05 UTC 2018-02-19
Prevent modification of GitUbuntuRepository env

Author: Robie Basak
Author Date: 2018-02-19 14:02:05 UTC

Prevent modification of GitUbuntuRepository env

Since it is common to take a GitUbuntuRepository instance env property
and then adjust it, make sure that modifying the dictionary returned
from the property getter does not modify the dictionary owned by the
instance itself. This adds some robustness against this common use
pattern.

source-tree 2018-02-12 15:00:57 UTC 2018-02-12
Add SourceTree

Author: Robie Basak
Author Date: 2018-01-25 15:20:13 UTC

Add SourceTree

commit-graph-builder 2018-02-07 15:27:36 UTC 2018-02-07
Add CommitGraph

Author: Robie Basak
Author Date: 2018-01-25 14:55:37 UTC

Add CommitGraph

derive-env 2018-02-07 13:49:52 UTC 2018-02-07
Replace self._env with a derived property

Author: Robie Basak
Author Date: 2018-02-06 13:29:07 UTC

Replace self._env with a derived property

As part of the drive to reduce stored state to improve testability,
derive self.env when we need it instead of working out in advance in the
constructor.

This is intended to have no functional changes.

GIT_WORK_TREE will end up differently normalised now, such as in
trailing slashes, since in testing pygit2.Repository.workdir was not
normalised like this and so a normalised assertion was needed. This
should not result in any change to git's behaviour.

import-tests 2018-01-24 15:03:18 UTC 2018-01-24
Initial test for import_unapplied_spi

Author: Robie Basak
Author Date: 2017-12-22 15:48:04 UTC

Initial test for import_unapplied_spi

Not got the verification at the end though.

source-builder 2018-01-24 14:42:22 UTC 2018-01-24
Add new source_builder module

Author: Robie Basak
Author Date: 2017-12-08 11:09:47 UTC

Add new source_builder module

unglobal-parent-overrides 2018-01-23 15:20:10 UTC 2018-01-23
Remove global _PARENT_OVERRIDES

Author: Robie Basak
Author Date: 2018-01-23 11:15:39 UTC

Remove global _PARENT_OVERRIDES

Make parent_overrides a regular parameter instead of a magic global
variable. This allows for easier testing along the "less initialisation"
theme.

create-dsc-on-first-use 2018-01-23 14:07:39 UTC 2018-01-23
Clean up catch-all exception

Author: Robie Basak
Author Date: 2018-01-19 17:50:57 UTC

Clean up catch-all exception

The only thing we want to catch here is the command returning a non-zero
exit status, which is subprocess.CalledProcessError. We shouldn't
swallow any other exceptions.

startup-pythonpath 2018-01-23 14:03:50 UTC 2018-01-23
wrapper: fix module search path

Author: Robie Basak
Author Date: 2018-01-23 14:02:34 UTC

wrapper: fix module search path

Fix the case when running git-ubuntu from the working tree via a
relative symlink.

nacc-breakdown/apt-get-not-apt 2018-01-22 13:34:04 UTC 2018-01-22
build: use apt-get over apt

Author: Nish Aravamudan
Author Date: 2018-01-09 16:12:30 UTC

build: use apt-get over apt

apt does not have a stable CLI API, so use the older, but very fine,
apt-get.

LP: #1734366

drop-unused-parameter 2018-01-22 11:35:24 UTC 2018-01-22
Drop unused parameter ubuntu_sinfo

Author: Robie Basak
Author Date: 2017-12-21 15:01:32 UTC

Drop unused parameter ubuntu_sinfo

This is used neither by import_applied_spi nor by import_unapplied_spi.

nacc-breakdown/run-quilt-gbp-correctly 2018-01-18 15:43:17 UTC 2018-01-18
gitubuntu: add API to call quilt and gbp

Author: Nish Aravamudan
Author Date: 2018-01-10 23:10:16 UTC

gitubuntu: add API to call quilt and gbp

LP: #1734137

nacc-breakdown/gbp-pbuilder-qemubuilder 2018-01-18 14:31:01 UTC 2018-01-18
gbp: do not use pbuilder or qemubuilder

Author: Nish Aravamudan
Author Date: 2018-01-10 21:44:49 UTC

gbp: do not use pbuilder or qemubuilder

nacc-breakdown/ubuntu-package-merge-base 2018-01-18 14:16:35 UTC 2018-01-18
gitubuntu: handle finding Ubuntu merge base with new importer layout

Author: Nish Aravamudan
Author Date: 2018-01-10 21:44:30 UTC

gitubuntu: handle finding Ubuntu merge base with new importer layout

`git ubuntu merge` relies on the old importer algorithm setting a
publishing parent around, so we can find the common ancestor between
new/debian and old/ubuntu, via `git merge-base`.

This works even for cases where the branch pointers are forcibly moved,
as long as old/debian is an ancestor of new/debian. But when old/ubuntu
was based on an NMU or security update, there is no guarantee that
new/debian will have that base in its history.

Add a helper method to find the Ubuntu merge point, which will try and
verify that the merge-base found by Git has the correct version. If it
does not, it will fall back to just using the import tag for that
version. The first case corresponds to imports with the old algorithm
and the second to imports with the new algorithm.

LP: #1734364

nacc-breakdown/lxc-snap-env 2018-01-18 14:05:30 UTC 2018-01-18
build: unset SNAP in lxc's env

Author: Nish Aravamudan
Author Date: 2018-01-09 19:21:16 UTC

build: unset SNAP in lxc's env

Ever since lxc gained snap support, it assumed that if SNAP is set in
it's environment that lxc itself was running as a SNAP. This is not
necessarily the case when git-ubuntu itself is a classic snap (and thus
has SNAP set in its environment) but is calling lxc in the host. Unset
SNAP manually on lxc run() calls.

LP: #1741949

dsc-builder 2017-12-22 15:48:04 UTC 2017-12-22
Initial test for import_unapplied_spi

Author: Robie Basak
Author Date: 2017-12-22 15:48:04 UTC

Initial test for import_unapplied_spi

Not got the verification at the end though.

fix-applied-branch-updates-wip 2017-11-24 17:34:59 UTC 2017-11-24
Add quilt env handling tests

Author: Robie Basak
Author Date: 2017-11-24 17:34:59 UTC

Add quilt env handling tests

fix-applied-branch-updates 2017-11-24 17:34:59 UTC 2017-11-24
Add quilt env handling tests

Author: Robie Basak
Author Date: 2017-11-24 17:34:59 UTC

Add quilt env handling tests

escape-git 2017-11-21 14:30:14 UTC 2017-11-21
Add .git escaping

Author: Robie Basak
Author Date: 2017-11-17 15:52:31 UTC

Add .git escaping

If a source package unpacks to contain any files or directories named
'.git', it cannot be represented in a git repository as-is because the
name '.git' is reserved by git and git will not permit it to be an entry
in a git tree object.

Instead we losslessly escape these names by prepending a '.' recursively.

See discussion thread at
http://lists.alioth.debian.org/pipermail/vcs-pkg-discuss/2017-May/000976.html
(continuing into June).

LP: #1730778

test-collection-hang 2017-11-21 12:56:57 UTC 2017-11-21
Do not import local Launchpad object

Author: Robie Basak
Author Date: 2017-11-21 11:47:10 UTC

Do not import local Launchpad object

This is a workaround for LP 1733388. It stops py.test-3 poking the
object's __getattr__ during test discovery, which currently results in a
hang as it prompts the user to log in to Launchpad but the prompt gets
hidden. And we don't want network communication during test discovery
anyway.

fix-importer-devel-branches 2017-11-15 14:34:53 UTC 2017-11-15
Initial tests for _devel_branch_updates

Author: Robie Basak
Author Date: 2017-11-15 10:45:05 UTC

Initial tests for _devel_branch_updates

repo-builder 2017-11-13 09:44:27 UTC 2017-11-13
Add test_git_repository.py

Author: Robie Basak
Author Date: 2017-10-31 17:26:27 UTC

Add test_git_repository.py

As a starting point, this adds tests for follow_symlinks_to_blob using
repo_builder.py.

modernize-scripts-review 2017-11-08 11:28:21 UTC 2017-11-08
Drop spphr_timestamp

Author: Robie Basak
Author Date: 2017-11-08 11:28:21 UTC

Drop spphr_timestamp

This function is no longer used.

dependency-loop 2017-10-30 16:10:07 UTC 2017-10-30
Fix dependency loop: move derive_target_branch

Author: Robie Basak
Author Date: 2017-10-30 16:06:01 UTC

Fix dependency loop: move derive_target_branch

Move derive_target_branch to git_repository.py. It is self contained and
needed from multiple subcommand implemnetations, such as build.py and
lint.py, and git_repository.py also needs it. To keep it in lint.py
creates a dependency loop making various modules unloadable (this
regressed in ae196ab).

This resolves the dependency loops so individual tests are runnable
again. Even if there is some technical debt in this not being exactly in
the ideal place now, it is materially better than it was before since
the tests are runnable.

Next I'd like to get the tests into CI. Then we can have some safety in
refactoring to move it to a final destination if we think it should
belong somewhere else.

commit-graph-v2 2017-10-27 15:40:17 UTC 2017-10-27
Also reset devel heads

Author: Robie Basak
Author Date: 2017-10-26 16:44:55 UTC

Also reset devel heads

We have decided to reset the devel heads instead of generating merge
commits for them.

lp1699541-queue-local-import 2017-10-23 14:03:12 UTC 2017-10-23
Use the branch tip found previously

Author: Robie Basak
Author Date: 2017-10-23 14:03:12 UTC

Use the branch tip found previously

changelog-parsing-fix 2017-08-14 13:31:36 UTC 2017-08-14
Test Changelog.distribution

Author: Robie Basak
Author Date: 2017-08-14 13:31:36 UTC

Test Changelog.distribution

This fails before the previous commit and now passes.

merge-fixes 2017-08-14 11:42:49 UTC 2017-08-14
Refactor merge subcommand commitish handling

Author: Robie Basak
Author Date: 2017-08-14 11:40:21 UTC

Refactor merge subcommand commitish handling

As we're now fetching a commitish object, use that to get the
treeish_string directly instead of looking it up again. This is cleaner
as it avoids the dual lookup, but my main motivation is that it isolates
the "treeish string"-ness more for eventual removal.

lp1698402 2017-08-11 16:01:03 UTC 2017-08-11
wip

Author: Robie Basak
Author Date: 2017-08-11 16:01:03 UTC

wip

versioning-revert 2017-07-26 14:42:14 UTC 2017-07-26
versioning: revert series related changes

Author: Robie Basak
Author Date: 2017-07-26 14:37:32 UTC

versioning: revert series related changes

This regressed tests.

Now we're back where we started. Tests pass for versions calculations
that we had working. The test for the SRU "same version in multiple
series" versioning exception fails as expected, and we're in a position
to be able to implement it.

lint-fixes-2 2017-07-26 12:16:03 UTC 2017-07-26
GitUbuntuRepository: silence noise on __init__

Author: Robie Basak
Author Date: 2017-07-26 12:07:30 UTC

GitUbuntuRepository: silence noise on __init__

logging.info() writes to the terminal by default, so do not use it in
such a common place. Library code shouln't write to the terminal except
by request of the caller.

lint-fixes 2017-07-25 17:46:25 UTC 2017-07-25
lint: correctly report lint failures on failures

Author: Robie Basak
Author Date: 2017-07-25 17:41:13 UTC

lint: correctly report lint failures on failures

We haven't agreed the best pattern to use for this yet. But currently
the return value of GitUbuntuLint.do_change_lint() is outright wrong,
returning only the status of the final check instead of all the checks
processed.

Fix this, at least for now, by correctly combining the results.

atomic-push 2017-07-13 14:47:30 UTC 2017-07-13
importer: push all refs atomically

Author: Robie Basak
Author Date: 2017-07-13 14:43:34 UTC

importer: push all refs atomically

Push any newly imported branches simultaneously with any new import tags
and atomically. This prevents any partial update and restores importer
idempotency.

This requires support for --atomic. It does exist on Xenial and seems to
work with Launchpad. I think this should be sufficient for our purposes.

LP: #1704139

queue-parent-fix 2017-07-12 09:55:06 UTC 2017-07-12
queue: do not clobber parent in series iteration

Author: Robie Basak
Author Date: 2017-07-12 09:55:06 UTC

queue: do not clobber parent in series iteration

If parent gets set for one series, it ends up being set for all of them,
which is wrong. Deconflate the parent requested by the caller from the
parent selected for use for a particular series by calling the latter
series_parent instead.

queue-typos 2017-07-12 09:29:31 UTC 2017-07-12
queue: fix typos for tag deletion

Author: Robie Basak
Author Date: 2017-07-12 09:29:31 UTC

queue: fix typos for tag deletion

tag_uri, not tag_url. Fixes NameErrors.

tag-print-name-only 2017-07-11 14:42:52 UTC 2017-07-11
tag: add --print-name-only

Author: Robie Basak
Author Date: 2017-07-11 14:42:52 UTC

tag: add --print-name-only

This is useful to get the dep14 substitution for scripting.

dsc-decoding 2017-06-30 11:28:10 UTC 2017-06-30
Open dsc in binary mode for better decoding

Author: Robie Basak
Author Date: 2017-06-30 11:28:10 UTC

Open dsc in binary mode for better decoding

Prior to Debian policy 3.8.1.0 (around March 2009), dsc files could be
encoded in anything as no encoding was mandated.

ubuntutools.archive.Dsc already has some support for auto-detection. It
appears to work, but the dsc must be opened in binary mode for this to
work.

This may mutate hashes if any other dscs are decoded differently as a
consequence of this change.

LP: #1700846

fix-empty-dir-symlink-traversal 2017-06-21 16:32:18 UTC 2017-06-21
Fix empty directory handling symlink traversal

Author: Robie Basak
Author Date: 2017-06-21 16:28:55 UTC

Fix empty directory handling symlink traversal

Fix the recursion to recurse only down real directories, not symlinks.

A symlink to a directory that has under the target directory an empty
directory fails the assertion in _create_replacement_tree_builder
because the provided entry is not a tree (a symlink entry is a blob).
This is non-sensical; we don't want to follow symlinks while traversing
for empty directories anyway.

Nish discovered this when importing websockify 0.5.1+dfsg1-1.

I had assumed that os.path.isdir() would not match on symlinks to
directories, but it does. Instead, use os.lstat() to check st_mode
directly.

queue-fixes-3b 2017-06-15 13:15:08 UTC 2017-06-15
Refactor queue sync

Author: Robie Basak
Author Date: 2017-06-15 13:12:18 UTC

Refactor queue sync

This was done in a spike. I don't think it's worth breaking it down for
this early stage script with only one known user (me). The changes
overlay each other significantly and I'm not confident the intermediate
commits are easy to get working anyway.

This adds --no-fetch, --source, --series, --parent, --orphan, --no-trim,
--new and --unapproved to modify the sync subcommand's behaviour. One
can now use this to pull queue tags into any git repository, which is
useful when working with package renames and similar.

I'm not sure it's appropriate for all of this functionality makes sense
to be in a command called "sync" any more; suggestions welcome.

queue-fixes-3 2017-05-31 13:25:48 UTC 2017-05-31
queue: stop tracking srcpkg_name

Author: Robie Basak
Author Date: 2017-05-31 13:25:48 UTC

queue: stop tracking srcpkg_name

We don't need it. Every series import is independent of the other. This
is intentional as the branches may change over a package rename; yet the
queue sync operation is still well-defined in this case.

1100 of 113 results
This repository contains Public information 
Everyone can see this information.

Subscribers