lp:~racb/git-ubuntu

Author: Robie Basak
Author Date: 2018-10-30 16:56:18 UTC

Mark "running" imports as errored on restart

Author: Robie Basak
Author Date: 2018-11-01 13:02:28 UTC

Pin keyring version to work around regression

Author: Robie Basak
Author Date: 2018-10-19 12:58:44 UTC

Use persistent state directory

Move the directory in which we keep persistent state into the one
specified by --data-directory, defaulting to /var/local/git-ubuntu.

This directory must exist.

This moves the previously used directory /tmp into somewhere that can
be expected to persist. This impacts git-ubuntu importer operators,
since they now need to make sure that a persistent area is ready before
use.

This applies to the importer service only. Users of the "git-ubuntu" CLI
tool, as opposed to the operators of the importer, are not affected.

Author: Robie Basak
Author Date: 2018-10-12 09:44:19 UTC

Fix call into core snap awk binary

The core snap's loader is tied to its libc. usr/bin/awk inside the core
snap refers to /lib64/ld-linux-x86-64.so.2, which is therefore incorrect
when called from a classic snap.

Add a wrapper for awk which calls awk in the core snap correctly,
through its own loader.

Separately, we also need to set LD_LIBRARY_PATH so that the loader can
find the shared libraries supplied by the core snap that satisfy its own
dependencies, rather than look at the host system or into our own snap
which may not have them. This was the original bug. Fixing this revealed
the problem above. Both issues are resolved by wrapping this call.

This does raise some questions such as what it means for a classic snap
to be making use of a core snap's binaries, and whether this is an
appropriate thing to do given that the core snap's binaries all specify
the "wrong" loader path in the classic case since the core snap won't be
mounted at the root of the filesystem. This might need some refactoring
of how the git-ubuntu snap works in the future, but for now we can fix
the awk case to fix the current regression in master.

With thanks to Michael Vogt.

LP: #1796017

Author: Robie Basak
Author Date: 2018-10-02 16:58:36 UTC

Add awk integration test

Author: Robie Basak
Author Date: 2018-09-27 15:06:44 UTC

Drop keyrings.alt version restriction

The launchpadlib part requires "keyring" with no version restriction. By
requiring a particular version here, we're relying that some newest
version of keyring will work with this particular keyrings.alt, which
fails with keyring 15.0.0 vs. keyrings.alt 2.3. Since launchpadlib
already has no version restriction, do the same for keyrings.alt. Then
at least if "pip3 install" works, so should the snap.

Alternatively we could have added a version restriction to the
launchpadlib part, but at the moment that pattern isn't used anywhere,
and the requirements are taken from launchpadlib's setup.py with no
patching. If we want to restrict versions some other way, then we'll
need a major reworking of how the snap is put together.

Now the test checking that keyring can find a functional backend should
always pass.

LP: #1794041

Author: Robie Basak
Author Date: 2018-09-20 16:18:48 UTC

tag: improve dirty tree detection

If raw_repo.status() returns entries that are GIT_STATUS_IGNORED then
they don't represent the tree being dirty, so ignore this case. This
fixes a case where "git ubuntu tag" refuses to work even when a working
tree is clean.

Related bugs:

https://bugs.launchpad.net/usd-importer/+bug/1687940
https://bugs.launchpad.net/usd-importer/+bug/1739000

Author: Robie Basak
Author Date: 2018-07-10 12:01:49 UTC

Merge branch 'changelog-assertions'

Some fixups were needed for the merge. The tests move from
git_repository.py to git_repository_test.py, one import gets removed
from git_repository.py (no longer needed from git_repository_test.py),
and a couple of references in new test code get prepended by "target.".

Author: Robie Basak
Author Date: 2018-06-14 12:52:26 UTC

Rename test files

This is a style change: we're moving to keeping tests in foo_test.py.

Author: Robie Basak
Author Date: 2018-06-01 15:57:01 UTC

Fix edge case changelog assertion check

We should also check the length since zip()'s output is the shortest of
its two inputs. This fixes the associated test.

I've also simplified the associated error path code, since I think it's
sufficient to just raise an exception and not swallow it higher up the
stack. Problems with changelog parsing can be reproduced later easily
enough.

Author: Robie Basak
Author Date: 2018-05-15 16:48:07 UTC

Drop inferred log message

This message was left behind to avoid a functional change in a previous
refactoring.

Unfortunately it has now become quite disconnected from the
implementation of logic that it is logging. It's at risk of becoming
wrong in the future if the implementation elsewhere is intentionally
changed.

Since it doesn't seem that important, drop it to avoid this happening.

Author: Robie Basak
Author Date: 2018-05-16 13:11:56 UTC

snap: fix lazr.restfulclient stage path

The old path was incorrect, but snapcraft somehow previously worked with
it, perhaps due to a regexp match or similar. The current snapcraft does
not. This change fixes the current snapcraft build failure.

Thanks to Nish for identifying the root cause and fix.

Author: Robie Basak
Author Date: 2018-04-27 11:01:34 UTC

Document external dependency management policy

External dependencies make the build more painful for developers and CI
in different environments, so we've agreed to prefer avoiding external
dependencies where possible.

Author: Robie Basak
Author Date: 2018-02-27 20:26:22 UTC

Add gitubuntu/apt_repo_test/ to the snap

This is needed for the in-snap self test.

Author: Nish Aravamudan
Author Date: 2018-02-22 03:08:33 UTC

tests: changelog test files should move to pkg_resources

In order to access the changelog files regardless of running from the
Git repository or the snap, we need to have the test files in the
package and available via pkg_resources.

This should be done for all future testing infrastructure as we add it,
as well.

Author: Robie Basak
Author Date: 2018-02-12 15:58:02 UTC

SourceSpec/SourceFiles: add changelog_versions

This allows a Source to be created with a predefined list of versions in
debian/changelog. This is useful, for example, for control of changelog
parents during testing.

Author: Robie Basak
Author Date: 2018-02-19 14:02:05 UTC

Prevent modification of GitUbuntuRepository env

Since it is common to take a GitUbuntuRepository instance env property
and then adjust it, make sure that modifying the dictionary returned
from the property getter does not modify the dictionary owned by the
instance itself. This adds some robustness against this common use
pattern.

Author: Robie Basak
Author Date: 2018-01-25 15:20:13 UTC

Add SourceTree

Author: Robie Basak
Author Date: 2018-01-25 14:55:37 UTC

Add CommitGraph

Author: Robie Basak
Author Date: 2018-02-06 13:29:07 UTC

Replace self._env with a derived property

As part of the drive to reduce stored state to improve testability,
derive self.env when we need it instead of working out in advance in the
constructor.

This is intended to have no functional changes.

GIT_WORK_TREE will end up differently normalised now, such as in
trailing slashes, since in testing pygit2.Repository.workdir was not
normalised like this and so a normalised assertion was needed. This
should not result in any change to git's behaviour.

Author: Robie Basak
Author Date: 2017-12-22 15:48:04 UTC

Initial test for import_unapplied_spi

Not got the verification at the end though.

Author: Robie Basak
Author Date: 2017-12-08 11:09:47 UTC

Add new source_builder module

Author: Robie Basak
Author Date: 2018-01-23 11:15:39 UTC

Remove global _PARENT_OVERRIDES

Make parent_overrides a regular parameter instead of a magic global
variable. This allows for easier testing along the "less initialisation"
theme.

Author: Robie Basak
Author Date: 2018-01-19 17:50:57 UTC

Clean up catch-all exception

The only thing we want to catch here is the command returning a non-zero
exit status, which is subprocess.CalledProcessError. We shouldn't
swallow any other exceptions.

Author: Robie Basak
Author Date: 2018-01-23 14:02:34 UTC

wrapper: fix module search path

Fix the case when running git-ubuntu from the working tree via a
relative symlink.

Author: Nish Aravamudan
Author Date: 2018-01-09 16:12:30 UTC

build: use apt-get over apt

apt does not have a stable CLI API, so use the older, but very fine,
apt-get.

LP: #1734366

Author: Robie Basak
Author Date: 2017-12-21 15:01:32 UTC

Drop unused parameter ubuntu_sinfo

This is used neither by import_applied_spi nor by import_unapplied_spi.

Author: Nish Aravamudan
Author Date: 2018-01-10 23:10:16 UTC

gitubuntu: add API to call quilt and gbp

LP: #1734137

Author: Nish Aravamudan
Author Date: 2018-01-10 21:44:49 UTC

gbp: do not use pbuilder or qemubuilder

Author: Nish Aravamudan
Author Date: 2018-01-10 21:44:30 UTC

gitubuntu: handle finding Ubuntu merge base with new importer layout

`git ubuntu merge` relies on the old importer algorithm setting a
publishing parent around, so we can find the common ancestor between
new/debian and old/ubuntu, via `git merge-base`.

This works even for cases where the branch pointers are forcibly moved,
as long as old/debian is an ancestor of new/debian. But when old/ubuntu
was based on an NMU or security update, there is no guarantee that
new/debian will have that base in its history.

Add a helper method to find the Ubuntu merge point, which will try and
verify that the merge-base found by Git has the correct version. If it
does not, it will fall back to just using the import tag for that
version. The first case corresponds to imports with the old algorithm
and the second to imports with the new algorithm.

LP: #1734364

Author: Nish Aravamudan
Author Date: 2018-01-09 19:21:16 UTC

build: unset SNAP in lxc's env

Ever since lxc gained snap support, it assumed that if SNAP is set in
it's environment that lxc itself was running as a SNAP. This is not
necessarily the case when git-ubuntu itself is a classic snap (and thus
has SNAP set in its environment) but is calling lxc in the host. Unset
SNAP manually on lxc run() calls.

LP: #1741949

Author: Robie Basak
Author Date: 2017-12-22 15:48:04 UTC

Initial test for import_unapplied_spi

Not got the verification at the end though.

Author: Robie Basak
Author Date: 2017-11-24 17:34:59 UTC

Add quilt env handling tests

Author: Robie Basak
Author Date: 2017-11-24 17:34:59 UTC

Add quilt env handling tests

Author: Robie Basak
Author Date: 2017-11-17 15:52:31 UTC

Add .git escaping

If a source package unpacks to contain any files or directories named
'.git', it cannot be represented in a git repository as-is because the
name '.git' is reserved by git and git will not permit it to be an entry
in a git tree object.

Instead we losslessly escape these names by prepending a '.' recursively.

See discussion thread at
http://lists.alioth.debian.org/pipermail/vcs-pkg-discuss/2017-May/000976.html
(continuing into June).

LP: #1730778

Author: Robie Basak
Author Date: 2017-11-21 11:47:10 UTC

Do not import local Launchpad object

This is a workaround for LP 1733388. It stops py.test-3 poking the
object's __getattr__ during test discovery, which currently results in a
hang as it prompts the user to log in to Launchpad but the prompt gets
hidden. And we don't want network communication during test discovery
anyway.

Author: Robie Basak
Author Date: 2017-11-15 10:45:05 UTC

Initial tests for _devel_branch_updates

Author: Robie Basak
Author Date: 2017-10-31 17:26:27 UTC

Add test_git_repository.py

As a starting point, this adds tests for follow_symlinks_to_blob using
repo_builder.py.

Author: Robie Basak
Author Date: 2017-11-08 11:28:21 UTC

Drop spphr_timestamp

This function is no longer used.

Author: Robie Basak
Author Date: 2017-10-30 16:06:01 UTC

Fix dependency loop: move derive_target_branch

Move derive_target_branch to git_repository.py. It is self contained and
needed from multiple subcommand implemnetations, such as build.py and
lint.py, and git_repository.py also needs it. To keep it in lint.py
creates a dependency loop making various modules unloadable (this
regressed in ae196ab).

This resolves the dependency loops so individual tests are runnable
again. Even if there is some technical debt in this not being exactly in
the ideal place now, it is materially better than it was before since
the tests are runnable.

Next I'd like to get the tests into CI. Then we can have some safety in
refactoring to move it to a final destination if we think it should
belong somewhere else.

Author: Robie Basak
Author Date: 2017-10-26 16:44:55 UTC

Also reset devel heads

We have decided to reset the devel heads instead of generating merge
commits for them.

Author: Robie Basak
Author Date: 2017-10-23 14:03:12 UTC

Use the branch tip found previously

Author: Robie Basak
Author Date: 2017-08-14 13:31:36 UTC

Test Changelog.distribution

This fails before the previous commit and now passes.

Author: Robie Basak
Author Date: 2017-08-14 11:40:21 UTC

Refactor merge subcommand commitish handling

As we're now fetching a commitish object, use that to get the
treeish_string directly instead of looking it up again. This is cleaner
as it avoids the dual lookup, but my main motivation is that it isolates
the "treeish string"-ness more for eventual removal.

Author: Robie Basak
Author Date: 2017-08-11 16:01:03 UTC

wip

Author: Robie Basak
Author Date: 2017-07-26 14:37:32 UTC

versioning: revert series related changes

This regressed tests.

Now we're back where we started. Tests pass for versions calculations
that we had working. The test for the SRU "same version in multiple
series" versioning exception fails as expected, and we're in a position
to be able to implement it.

Author: Robie Basak
Author Date: 2017-07-26 12:07:30 UTC

GitUbuntuRepository: silence noise on __init__

logging.info() writes to the terminal by default, so do not use it in
such a common place. Library code shouln't write to the terminal except
by request of the caller.

Author: Robie Basak
Author Date: 2017-07-25 17:41:13 UTC

lint: correctly report lint failures on failures

We haven't agreed the best pattern to use for this yet. But currently
the return value of GitUbuntuLint.do_change_lint() is outright wrong,
returning only the status of the final check instead of all the checks
processed.

Fix this, at least for now, by correctly combining the results.

Author: Robie Basak
Author Date: 2017-07-13 14:43:34 UTC

importer: push all refs atomically

Push any newly imported branches simultaneously with any new import tags
and atomically. This prevents any partial update and restores importer
idempotency.

This requires support for --atomic. It does exist on Xenial and seems to
work with Launchpad. I think this should be sufficient for our purposes.

LP: #1704139

Author: Robie Basak
Author Date: 2017-07-12 09:55:06 UTC

queue: do not clobber parent in series iteration

If parent gets set for one series, it ends up being set for all of them,
which is wrong. Deconflate the parent requested by the caller from the
parent selected for use for a particular series by calling the latter
series_parent instead.

Author: Robie Basak
Author Date: 2017-07-12 09:29:31 UTC

queue: fix typos for tag deletion

tag_uri, not tag_url. Fixes NameErrors.

Author: Robie Basak
Author Date: 2017-07-11 14:42:52 UTC

tag: add --print-name-only

This is useful to get the dep14 substitution for scripting.

Author: Robie Basak
Author Date: 2017-06-30 11:28:10 UTC

Open dsc in binary mode for better decoding

Prior to Debian policy 3.8.1.0 (around March 2009), dsc files could be
encoded in anything as no encoding was mandated.

ubuntutools.archive.Dsc already has some support for auto-detection. It
appears to work, but the dsc must be opened in binary mode for this to
work.

This may mutate hashes if any other dscs are decoded differently as a
consequence of this change.

LP: #1700846

Author: Robie Basak
Author Date: 2017-06-21 16:28:55 UTC

Fix empty directory handling symlink traversal

Fix the recursion to recurse only down real directories, not symlinks.

A symlink to a directory that has under the target directory an empty
directory fails the assertion in _create_replacement_tree_builder
because the provided entry is not a tree (a symlink entry is a blob).
This is non-sensical; we don't want to follow symlinks while traversing
for empty directories anyway.

Nish discovered this when importing websockify 0.5.1+dfsg1-1.

I had assumed that os.path.isdir() would not match on symlinks to
directories, but it does. Instead, use os.lstat() to check st_mode
directly.

Author: Robie Basak
Author Date: 2017-06-15 13:12:18 UTC

Refactor queue sync

This was done in a spike. I don't think it's worth breaking it down for
this early stage script with only one known user (me). The changes
overlay each other significantly and I'm not confident the intermediate
commits are easy to get working anyway.

This adds --no-fetch, --source, --series, --parent, --orphan, --no-trim,
--new and --unapproved to modify the sync subcommand's behaviour. One
can now use this to pull queue tags into any git repository, which is
useful when working with package renames and similar.

I'm not sure it's appropriate for all of this functionality makes sense
to be in a command called "sync" any more; suggestions welcome.

Author: Robie Basak
Author Date: 2017-05-31 13:25:48 UTC

queue: stop tracking srcpkg_name

We don't need it. Every series import is independent of the other. This
is intentional as the branches may change over a package rename; yet the
queue sync operation is still well-defined in this case.

Author: Robie Basak
Author Date: 2017-05-24 15:01:03 UTC

Fix queue devel ref lookup

Since pkg is now in refs/remotes/pkg/, look it up by the full reference
as it is not a local branch and so is no longer found by
GitUbuntuRepository.get_head_by_name().

Author: Robie Basak
Author Date: 2017-05-23 11:25:16 UTC

Refactor changelog parsing

Replace the shell-based symlink workaround with a proper implementation
that uses pygit2 directly. This results in a significant speedup.

get_changelog_versions_from_treeish() previously returned (None, None)
if 'debian/changelog' was not in the supplied tree-ish. This was not
documented and happened implicitly due to the shell fallback behaviour
in parsing debian/changelog. get_heads_and_versions() appears to rely on
this behaviour, for example when asking for versions for the
'debian/pristine-tar' branch. This behaviour is made explicit in the
replacement function that uses the new changelog parsing.

Author: Robie Basak
Author Date: 2017-05-17 14:24:08 UTC

Add hooks to catch empty directory issue

Warn when finding trees affected by LP 1687057.

Author: Robie Basak
Author Date: 2017-04-20 13:48:24 UTC

queue: support import of pocket copies

In the case of a pocket copy, we must follow through to find a suitable
source_package_publishing_history object and use that to download the
source.

Author: Robie Basak
Author Date: 2017-04-19 15:55:57 UTC

usd/queue: fix queue handling

This broke in the refactoring in bcb7e8de.

Author: Robie Basak
Author Date: 2017-01-11 17:34:08 UTC

Add the "queue" command

Author: Robie Basak
Author Date: 2017-01-11 17:24:29 UTC

Add msg option to USDGitRepository:annotated_tag

This allows the caller to specify the annotation, instead of assuming a
blank one.

Author: Robie Basak
Author Date: 2017-02-02 11:01:10 UTC

Follow symlinks reading debian/changelog

LP: #1661092

Author: Robie Basak
Author Date: 2017-01-19 10:36:44 UTC

usd import: add --no-fetch option

Author: Robie Basak
Author Date: 2017-01-19 12:50:15 UTC

Correctly instantiate exceptions

We should be raising instances of exception classes, not the classes
themselves.

Author: Robie Basak
Author Date: 2017-01-12 00:41:04 UTC

fixup commit_tree_hash

Author: Robie Basak
Author Date: 2016-12-14 12:37:59 UTC

Disable applied patches imports

This is a crude attempt to work around LP: #1649832 by not converting
the unapplied tips to applied tips.

Author: Robie Basak
Author Date: 2016-11-18 12:03:38 UTC

Update "usd clone" to meet current namespace specs