lp:~racb/git-ubuntu

Author: Robie Basak
Author Date: 2023-09-21 20:37:26 UTC

doc: adjust note on closing MPs in patch-pilot.rst

Author: Robie Basak
Author Date: 2023-09-15 16:06:58 UTC

importer: maintain staging branches

Create and maintain staging branches after the devel branches are
updated. We already push refs/heads/<namespace>/*, so these will also
get pushed as part of the importer run.

ACL handling is not implemented. It remains to be determined how ACL
handling will eventually land in Launchpad. See LP: #1993290 and
LP: #1993303 for details. Depending on the implementation, we may be
able to manage ACLs in team-wide or distribution-wide configuration and
not require any implementation inside the importer at all.

See the specification for details:

    https://discourse.ubuntu.com/t/spec-git-ubuntu-staged-uploads/35409

LP: #1844782

Author: Robie Basak
Author Date: 2023-09-04 12:08:59 UTC

Correct configure tracking branches

"git clone" by default sets up branch.<name>.remote and
branch.<name>.merge so that various git tooling can follow and set
default accordingly, so we should do the same when we create equivalent
branches.

LP: #1995489

Author: Robie Basak
Author Date: 2023-09-01 11:21:37 UTC

Package experimental commands

Acting on specific feedback, this provides the scripts formerly known as
gu-build and emptydirfixup.py as git-ubuntu.experimental-build and
git-ubuntu.experimental-emptydirfixup, to make them easier to consume.

They aren't being adopted into git-ubuntu "proper" yet as that would
imply CLI stability at a level that doesn't currently exist for them.
Hopefully this is a middle ground that will make them more accessible,
but without implying stability.

Snaps require additional entry points to be prefixed by the main snap
name (ie. "git-ubuntu."). This means the names a bit cumbersome,
although they do tab-complete. I recommend that interested developers
arrange local aliases such as gu-build and gu-emptydirfixup.

Author: Robie Basak
Author Date: 2023-08-31 12:48:39 UTC

version: bump to 1.1

Author: Robie Basak
Author Date: 2023-08-21 10:38:52 UTC

Revert "snap.sh: run snapcraft with sudo"

This is fixed upstream now with snapcraft 7.5.2 so we can remove this
workaround. See: https://github.com/snapcore/snapcraft/issues/4225

This reverts commit fe21a9f5c6c65a9a63b526d60c8e7207de92adb1.

Author: Robie Basak
Author Date: 2023-07-18 01:19:42 UTC

sandbox/gu-build: remove duplication of CLI args

These now exist in prepare_upload.py, so we can call that function
instead. That API isn't stable, but Steve prefers that to this code
duplication in his review in the MP, and I don't have a strong opinion.

Author: Robie Basak
Author Date: 2023-07-11 15:55:30 UTC

importer: ignore rich history requiring auth

Rich history may be supplied in such a way that fetching it requires
authentication. For example, Launchpad does this when referring to a
repository that does not exist, since a private repository may become
visible following authentication.

For rich history purposes, we should ignore this case and ensure that
git does not prompt the user for authentication but instead treat the
case as "rich history not provided".

It does not seem worth arranging a test for this, since we would need to
fake a remote repository that requires authentication and we don't
currently have (and until now have not needed) a test fixture for that.
I have tested this implementation manually. It is unlikely to regress in
our code base. If it were to regress because a future version of git
changes behaviour, then behaviour would revert back to a soft failure,
and we would be able to detect and debug the problem from there.

LP: #1980982

Author: Robie Basak
Author Date: 2023-06-07 11:14:26 UTC

Rewrite prepare-upload to fetch before push

This is a major refactoring/rewrite of the prepare-upload code. So much
changed that it didn't seem worth breaking it down. It's probably easier
to review afresh.

Functionally, what has changed is that we now fetch from the remote
using its default configured refspecs first, assuming that this will
update the remote tracking branch following normal conventions. Then we
can determine if the remote branch already contains our rich history or
not. We then only push if it does not.

In the code, this involved considerable refactoring. There is now a
Parameters namedtuple that carries the various facts around, and the
determination of these facts are moved into a classmethod constructor
method within that class. Various common steps betweent the "args" and
"mangle" operations are broken out.

Tests are updated and added accordingly.

LP: #1999177

Author: Robie Basak
Author Date: 2023-06-20 12:55:29 UTC

snap.sh: run snapcraft with sudo

A regression in snapcraft causes it to start failing with:

    Sorry, an error occurred in Snapcraft:
    [Errno 13] Permission denied: '/etc/apt/keyrings'

We run snapcraft with --destructive-mode. Presumably it used to use
sudo.

A workaround is to just run the whole thing with sudo ourselves.

See also my upstream report:
https://github.com/snapcore/snapcraft/issues/4225

Author: Robie Basak
Author Date: 2022-08-23 16:16:51 UTC

prepare-upload: handle ssh:// rewrites

According to LP 1942985, this is another case where a URL rewrite is
expected.

Author: Robie Basak
Author Date: 2023-04-17 11:02:32 UTC

wip

Author: Robie Basak
Author Date: 2023-04-04 14:00:05 UTC

importer-service-worker: set as default repository

Use --set-as-default-repository when calling "git ubuntu import" unless
--no-set-as-default-repository is specified. This hooks everything up so
that we set the default repository when we push from the official
importer service instance.

Author: Robie Basak
Author Date: 2023-02-06 15:17:21 UTC

Handle missing refs on rich history fetch

If rich history has been indicated using the changes file, but the
remote ref specified does not exist, then "git fetch" fails with a
message starting "fatal: couldn't find remote ref".

We already have code that retries in case there's a network outage
making rich history fetch temporarily unavailable.

However, this causes the "ref doesn't exist" case to also result in a
soft failure, so an import of an upload that is erroneous in this way
never succeeds.

Instead, we should treat this as a hard RichHistoryNotFoundError, which
already has a handling path that falls back to a synthesized commit.

Unfortunately there doesn't seem to be any good way for "git fetch" to
differentiate these cases, and we're using "git fetch" instead of the
pygit2 API because of https://github.com/libgit2/pygit2/issues/1060. So
we resort to parsing the output. This requires LC_ALL=C.UTF-8 to avoid
translation string related issues, and we also add a test to ensure that
we'll notice if this assumption about git's behaviour changes.

Ideally I'd also have a test that verifies that LC_ALL=C.UTF-8 does do
what we need, by perhaps expecting a translated string with LC_ALL
changed to something else. But it doesn't look like git has any
translated strings in practice, so it wouldn't be straightforward to
cause it to output something different. I think we'll need to rely on
just specifying it our call.

LP: #1992499

Author: Robie Basak
Author Date: 2022-10-10 13:17:02 UTC

poller: remove allowlists and phasing

Now that we are importing all* packages, allowlists (including
"allowlist_team") and phasing are no longer required, so we can remove
this functionality and remove significant sections of code.

Previously we were augmenting the "user-specified" allowlist with an
allowlist generated by reading the apt repository for source packages,
placing them in main and universe buckets, and then applying "phasing"
to them to allow us to include proportional subsets of main and
universe. This made sense at the time, but is no longer necessary.

The downside of the previous approach is that it locked the set of
source packages imported to those present at the time the poller was
last restarted. This would miss new source packages until after the
poller was restarted and a further upload was made to them after the
restart.

Now that we've imported the majority of packages currently published in
any pocket of any release that is not EOL, it's simpler to import any
package that isn't otherwise denylisted. We can maintain the denylist
with the set of edge case imports that still need attention to avoid
unnecessary importer resources.

This requires just a denylist and nothing else.

Since the apt repository parsing was being done solely to determine the
list of source packages in the archive for the purposes of the poller,
the apt repository reading and validation code can also be removed,
together with corresponding tests.

Author: Robie Basak
Author Date: 2022-10-13 11:37:51 UTC

More edge case changelog date parsing

According to the import specification, it's fine to parse Thurs as
Thu, Tues as Tue, and Sept as Sep, as these are all unambiguous. This
fixes imports of the following:

apachetop 0.12.5-7 Thurs, 12 Jan 2006 12:09:58 +0000
easychem 0.6-0ubuntu1 Tues, 20 Dec 2005 16:57:16 -0500
gnome-shell-extension-tilix-shortcut 1.0.1-1 Tue, 19 Sept 2017 14:23:17 +0200
libapp-cache-perl 0.35-1 Wed, 24 Sept 2008 19:32:23 +0200
libcrypt-hcesha-perl 0.70-2 Wed, 24 Sept 2008 19:44:01 +0200

Applications of multiple regular expression replacements are factored
out into _apply_re_substitutions(). This function isn't explicitly
tested; the comprehensive test coverage of its only caller
Changelog._parse_changelog_date() is sufficient.

Author: Robie Basak
Author Date: 2022-08-03 10:12:26 UTC

Move import_srcpkg() to importer_service_worker.py

This function is only used from here, so there's no need for it to be in
a different module.

Author: Robie Basak
Author Date: 2022-08-09 14:04:52 UTC

Add changelog date override for gmsh

Author: Robie Basak
Author Date: 2022-08-04 14:43:17 UTC

wip: overhaul emptydir workaround hooks

Author: Robie Basak
Author Date: 2022-07-01 10:56:26 UTC

Add comment on missing observability

This should help locate the older emailing code should it be needed in
the future.

Author: Robie Basak
Author Date: 2022-06-24 13:20:23 UTC

submit: default to ~canonical-server-reporter

On the Canonical Server Team, we have been using ~canonical-server in
its own review slot for the sole purpose of gathering all reviews we're
interested in tracking together on this team's +activereviews page.

A problem with this is that we all belong to this team, so when a person
does a review, they sometimes accidentally "grab" that slot, so it
appears as their name rather than the team's, and thus disappears from
the report.

One way around this is to use a separate team that none of us actually
belong to. This way we can't "grab" that slot.

This changes the default team to this new ~canonical-server-reporter
team to help those who use the "git ubuntu submit" command.

Author: Robie Basak
Author Date: 2022-07-01 15:49:04 UTC

Improve documentation on validation constants

This should hopefully do a better job of signposting anyone who wants to
change the constants to better understand the implications of doing so.

Author: Robie Basak
Author Date: 2022-06-23 13:48:11 UTC

Refactor _head_version_is_equal

Add a docstring, refactor the code to make it more readable, and rename
the method to match its definition better.

This should not result in any functional change.

Author: Robie Basak
Author Date: 2022-05-20 14:12:15 UTC

Update email address default used in tests

Since usd-import-team@lists.launchpad.net no longer exists, we'll use
ubuntu-distributed-devel@lists.ubuntu.com instead. This shouldn't affect
production code since the repo_builder and source_builder modules are
only used in tests.

Author: Robie Basak
Author Date: 2021-10-14 21:54:11 UTC

Fix git-ubuntu.merge-changelogs name typo

Author: Robie Basak
Author Date: 2021-10-12 16:54:45 UTC

Change repository owner to ~git-ubuntu-import

I'm making this change in order to separate the ownership/management of
the repositories from the development of git-ubuntu itself. This should
help prevent accidental pushes to the official import repositories,
since only the importer should normally be doing that.

Most of the code changes are for documentation or example scripts not in
production.

In gitubuntu/importer.py, we change the default, and also the magic
handling difference that if the official destination is used then the
local refs used are all prefixed ("namespaced") with "importer/". This
is the one place that might affect production behaviour, so I'm changing
it to be on the safe side. This change means a flag day to land into the
importer service, but this seems easier than removing or parameterising
the magic.

In gitubuntu/submit.py, we change the default destination and
understanding that this magic team name corresponds to the "pkg" remote.

A grep through the source found no other usd-import-team references
except the list email address and the use of the team for the hosting of
git-ubuntu itself, which aren't relevant to changing the repository
owner.

Announcement:
https://lists.ubuntu.com/archives/ubuntu-devel/2021-October/041644.html

Author: Robie Basak
Author Date: 2021-10-12 11:16:52 UTC

prepare-upload: add mangle feature

prepare-upload is changed to provide two subcommands. "args" does what we
did previously. "mangle" pushes like "args" but then mangles an existing
changelog file to add the headers as required.

Author: Robie Basak
Author Date: 2021-09-28 16:27:26 UTC

Fix spelling

Author: Robie Basak
Author Date: 2021-08-17 12:03:11 UTC

Add comment to change date override file

Author: Robie Basak
Author Date: 2021-08-16 13:59:06 UTC

poller: add --extra-trusted-keyring option

Debian and Ubuntu occasionally add new keys, and start signing their apt
repositories with them. Since we build with Ubuntu Focal's shipped
debian-keyring and ubuntu-keyring packages, these don't get updated with
the keys used by these updates. Consequently signature validation fails
and the poller fails to start.

Previously we were adjusting our packaging to include the newer public
keys, but this is becoming cumbersome and would need to be reimplemented
against our new deb/snap packaging anyway, since we rebased the snap to
core20 and a Focal deb build.

Instead, it seems easier to allow the importer service operator to
simply add the keys and provide them to the poller on startup. This will
save round tripping through the packaging for these updates in the
future. In practice there is only one importer service running, so this
should reduce maintenance burden overall.

Author: Robie Basak
Author Date: 2021-08-16 10:24:49 UTC

snap: set grade to stable

We think the core20 based snap is good enough to be considered stable
now. Setting this will allow us to publish a snap beyond the beta store
channel.

Author: Robie Basak
Author Date: 2021-08-03 20:17:39 UTC

Add examples to accompany MP review

See the review at:
https://code.launchpad.net/~bryce/usd-importer/+git/usd-importer/+merge/406335

Author: Robie Basak
Author Date: 2021-02-16 13:10:43 UTC

Add experimental "push-for-upload" subcommand

In order to use the new rich history supply facility, developers need to
push their rich history to Launchpad and then provide a reference to
that location using headers in the changes file.

This subcommand makes this easier. For example:

  dpkg-buildpackage <usual opts> $(git ubuntu push-for-upload)

This will run dpkg-buildpackage as normal, but first push the current
branch to the default location. The default location is configured as
"gitubuntu.lpuser" and should be set for git-ubuntu users already. The
generated changes file will contain headers to find the pushed branch
for later rich history adoption.

This is an experimental subcommand, and we'll need to finalise the
details before release of git-ubuntu 1.1. I'm deferring writing a
manpage until we've settled on the CLI, as I expect some iteration will
be needed to make this most convenient for developers' common workflows.

Author: Robie Basak
Author Date: 2021-06-29 10:29:33 UTC

Include all dpkg decompressors in the snap

bzip2/bunzip2 was missing in the snap, which prevented "git ubuntu
import" from working in the case that bzip2 compression was used in
their source packages.

Add an integration test to ensure that all dpkg decompressors are
available, and fix the snap to include them explicitly.

LP: #1928775

Author: Robie Basak
Author Date: 2021-02-01 15:21:30 UTC

Add rich history import from changes files

Allow uploaders to specify where rich history associated with an upload
can be found directly from the changes file. This allows rich history to
be provided by any uploader without concern for things like access
control for pushing upload tags.

This partly deprecates upload tags. The generally recommended approach
to supply rich history will be through the changes file. Upload tags
remain to preserve the rich history preservation mechanism on reimport.
After a transition to changes file based rich history is complete, we
might refactor the rich history preservation mechanism and then drop
support for upload tags altogether. However it might be useful to retain
upload tag support for testing purposes, as they are easier to arrange
than this new changes file mechanism.

For the retry logic, we add a dependency on tenacity. This module seems
to be the most widely used for this purpose, and is already in main in
Ubuntu Focal.

Author: Robie Basak
Author Date: 2021-04-23 06:06:16 UTC

source_builder: use git directly in common case

It is common for tests to use source_builder with a
repo_builder.SourceTree. Before this change, this was generated by
round-tripping through a packed source package (dsc, debian.tar.gz, etc)
which takes considerable time. This isn't really necessary but was the
easiest way to implement it at the time because the components already
existed.

Instead, we can arrange to construct the git objects directly, bypassing
generating the source package. On my system this speeds up the tests
significantly, going from 98 seconds to 32 seconds.

In the case that SourceTree is being called in patches-applied mode,
this isn't sufficient as we don't currently have a way to apply patches
purely in git. So we fall back to the previous method in this case.

Skipping the tests that use SourceTree in patches-applied mode saves
another ten seconds, so this suggests that there is yet more significant
performance improvement possible. But I'll leave this for another time.

Author: Robie Basak
Author Date: 2021-04-15 11:06:24 UTC

snap: add python3-secretstorage for GNOME keyring

If the user is using GNOME, then it is expected that GNOME Keyring will
be used for Launchpad API token storage, which requires this module (via
keyring) to work.

python3-secretstorage depends on python3-cryptography, which must
explicitly be specified in snapcraft.yaml due to LP: #1918999.

I've been unable to write a test for this. Creating a collection in
secretservice (as implemented by gnome-keyring) doesn't work because a
prompt is required, and gnome-keyring apparently won't do that headless.
I found one reference to this at:
https://github.com/99designs/aws-vault/issues/304#issuecomment-497387162

Otherwise, a simple test that does "import keyring;
keyring.backends.SecretService.Keyring().get_preferred_collection()"
might have worked.

LP: #1923727

Author: Robie Basak
Author Date: 2021-01-07 13:43:23 UTC

Add smart whitelist support

We don't want imported repositories to languish; if imported once into
Launchpad, they should continue to be maintained.

Rather than relying on manual import into the whitelist, we can instead
just dynamically check to see if we have already imported a package, and
if so, treat it as if it is whitelisted.

This implements that, calling it a "team whitelist" and adding a
--team-whitelist option to the poller to specify which Launchpad team
should be consulted in order to determine if a package is already
imported.

As the logic gets more complex, we refactor it all into
a filter function created by
importer_service_poller.create_filter_func(). This neatly simplifies
importer_service.request_imports(). For this to work, we pull out the
Launchpad login into the caller, renaming the object to "lp" everywhere
to match convention elsewhere.

I chose to add has_import_repository() to importer_service.py as it's a
general function that feels like it isn't necessary poller specific. But
I placed create_filter_func() to importer_service_poller.py as the logic
of what to include and exclude is closely tied to the poller itself.
This nicely separates the decision logic and list management in the
poller from the search implementation in the main importer_service.py
service handling library.

As the blacklist and whitelist functionality is now pulled into
create_filter_func(), that functionality is removed from
srcpkgs_to_import_list() and should_import_srcpkg(). This invalidates
the admin tools import-source-packages.py, source-package-walker.py and
update-repository-alias.py, so I've removed them. These tools haven't
been used in a long time anyway. import-source-packages.py and
source-package-walker.py are redundant now that we have the queue driven
importer service implementation. update-repository-alias.py is currently
done by hand, but I intend to replace it with an
imported-repository-driven equivalent anyway.

Finally, this means that scriptutils.py becomes mostly redundant and it
might be worth removing the redundant parts and pulling what's left into
importer_service_poller.py, but this change seems to have gotten big
enough now that I'll leave that for another time.

Author: Robie Basak
Author Date: 2021-04-19 12:13:41 UTC

Instantiate HTTPError better in tests

If we don't provide an fp parameter, then pytest subsequently fails when
it tries to look up a __name__ attribute on the exception object because
Python returns a KeyError in that case instead of an AttributeError. See
the linked bug for full details.

As a workaround, we can just supply an fp that behaves closer to what
might be expected, but with empty data.

LP: #1924983

Author: Robie Basak
Author Date: 2021-02-12 14:59:23 UTC

Correctly compare package version strings

build.py currently has no tests, so it's difficult to add a test for
this fix. But this is a trivial and obvious improvement, so I think it's
worth landing without.

Author: Robie Basak
Author Date: 2021-04-13 16:41:08 UTC

Drop GitUbuntuRepository.descendant_of()

Now that we have unpinned pygit2, the newer version of pygit2 provides a
pygit2.Repository.descendant_of() method directly, so we can drop our
temporary implementation and adjust the single caller.

We retain the test to ensure that the implementation swap does not
change behaviour.

Author: Robie Basak
Author Date: 2021-04-08 14:37:08 UTC

Drop GPG key ID workaround

With LP: #1918967 fixed in snapcraft, we must match and change the
substring of the fingerprint used for the public key file supplied to
snapcraft; otherwise the build fails.

Author: Robie Basak
Author Date: 2021-04-08 14:37:08 UTC

Drop GPG key ID workaround

Author: Robie Basak
Author Date: 2021-04-08 14:37:08 UTC

Drop GPG key ID workaround

Author: Robie Basak
Author Date: 2021-02-22 17:33:49 UTC

Pin cryptography to avoid Rust

Currently CI is failing because cryptography from PyPI now requires Rust
support to successfully build.

For the time being, we can pin to an older version of cryptography that
still works. This is a workaround, so I've filed LP: #1916520 to track
this.

Author: Robie Basak
Author Date: 2020-09-16 11:55:54 UTC

Retry database reads on lock timeouts

The broker has been crashing when the system is under heavy I/O load.
This happens when we get a sqlite3 timeout ("Database is locked") when
executing a "SELECT", as opposed to the earlier "BEGIN DEFERRED".
Reading the docs, this is expected behaviour, and we need to be retrying
the "SELECT" itself rather than the "BEGIN DEFERRED".

A retry is appropriate because there is no appropriate amount of time
after which we should time out. We want to block until the system is
able to process our database query, just like any other command.
Deadlocks are prevented by use of the complemetary _write_lock()
decorator.

I've not added a matching test for this change. The likely area where
this could go wrong is in my understanding of sqlite3's behaviour in
various edge cases, but I can only test that my code meets my
expectation of sqlite3's behaviour rather than what it actually does.

Author: Robie Basak
Author Date: 2020-09-11 16:35:38 UTC

Add some missing docstrings

Author: Robie Basak
Author Date: 2020-09-10 14:08:54 UTC

Remove all patch overrides

This haven't been active since patch override support was removed in
git-ubuntu in commit 806eaea (from Nov 2016).

Author: Robie Basak
Author Date: 2020-08-17 13:09:57 UTC

Revert "Ensure the locale is set consistently"

This reverts commit 6ff151dc208809e1b8e3108c69c05090174e6221.

This commit broke when using the snap on Focal (but not on Bionic) with:

    locale.Error: unsupported locale setting

...when calling locale.setlocale().

Since this is a regression, I'm reverting it immediately.

However I've since learned that locale-specific behaviour is not
activated until locale.setlocale() is called explicitly anyway, so
perhaps ensuring consistent behaviour despite locale settings is not as
important or necessary as I had previously thought.

Author: Robie Basak
Author Date: 2020-08-03 12:47:43 UTC

Add changelog date overrides

To handle the edge case of changelog entry signoff line dates not being
parseable due to not being valid dates, the spec now requires that we
revert to using the timestamp of the source package data instance
instead (which is what we always use for the commit date). This change
implements this requirement and adds overrides for the initial set of
changelog dates to be ignored as defined by the spec.

Author: Robie Basak
Author Date: 2020-08-04 13:57:28 UTC

Fix extra angle bracket parsing

RFCs 5321/5322/6531 make it clear that an angle bracket is not a valid
part of an email address, so we can correctly parse extra ones as not
being part of the email address.

This requires making the '.*' capture of the email address non-greedy by
appending a '?', so that the greedy '<+' and '>+' can grab the angle
brackets during the regexp match.

Author: Robie Basak
Author Date: 2020-07-23 13:46:43 UTC

Handle empty changelog author names

git (or at least pygit2) won't accept an author name that is empty, so
we must use some placeholder instead. I have adjusted the spec for this
to be 'Unnamed', and behaviour and the existing test case is adjusted
here accordingly.

This shouldn't cause "hash-abi-break" because the importer was failing
on such cases previously. None of the known cases (detailed in the spec)
had been successfully imported previously.

Author: Robie Basak
Author Date: 2020-07-31 12:37:42 UTC

poller: log package count after startup

We had a case where it seemed that the poller wasn't picking up on the
complete list of packages to import. To assist debugging of this if it
happens again, log the number of packages marked importable after this
list is determined at startup.

Author: Robie Basak
Author Date: 2020-07-22 13:56:11 UTC

Ensure the locale is set consistently

Changelog date parsing uses strptime which relies on the locale setting,
so add an assertion to ensure this is true and set the locale correctly
at the main importer entry point.

This does mean that the importer will not be localizable, but that seems
unlikely to be an issue in practice as the importer service operators
can all happily operate in the C locale. If it does become an issue then
we'd need to carefully consider what behaviour might be locale-dependent
to maintain import reproducibility. It may not just be the changelog
date parsing.

Since we don't actually enable localisation except in setting it to C in
the importer entry point, users of other locales should not be affected
when running git-ubuntu normally or when running git-ubuntu.self-test.

Author: Robie Basak
Author Date: 2020-07-20 14:18:56 UTC

Handle non-UTF8 characters in changelog notes

Handle non-UTF8 characters in changelog entries by replacing them when
creating changelog notes.

Author: Robie Basak
Author Date: 2020-06-11 12:51:54 UTC

Add new IPC-based service components

This adds three new subcommands: importer-service-poller,
importer-service-broker and importer-service-worker.

I had a choice here: I could have added new snap endpoints as was done
previously with import-source-packages.py becoming
git-ubuntu.import-source-packages, but I think I prefer moving all
endpoints into git-ubuntu subcommands now that we have these, so I did
this instead.

Author: Robie Basak
Author Date: 2020-07-06 12:44:37 UTC

Use the host's ssh client

The user expectation is that the user's host ssh configuration be used.
We are therefore dependent on the host system having ssh and it
functioning correctly. So just use it.

This fixes version mismatches between the ssh shipped in the snap and
the host ssh configuration file format, as well as library dependency
mismatches.

LP: #1867389
LP: #1871827

Author: Robie Basak
Author Date: 2020-06-02 14:46:57 UTC

Add terminating newline to importer notes

It's inconveninent for the importer note to not have a terminating
newline. This was an oversight.

From some quick research:

If I use "git notes add -F", a terminating newline is added whether the
file had a terminating newline before or not. This can be seen in a
checkout of refs/notes/commits. However a checkout of the importer notes
currently does not have terminating newlines, so evidently pygit2 is not
performing that same normalization. Therefore there cannot be any harm
in doing so.

The changelog notes already are being provided with a terminating
newline, so there is no change there and none is required.

Author: Robie Basak
Author Date: 2020-05-28 13:07:18 UTC

Allow fast-forwards when porting rich history

Currently we unconditionally port forward rich history, without checking
if parent commits have mutated. This is usually fine for a one-off
reimport which we are only conducting because we are expecting the
commits will mutate anyway. However in the case that commits have not
mutated (such as an immediate reimport of a reimport), this
unnecessarily mutates rich history by bumping forward the committer
dates.

Using "git cherry-pick --ff" prevents this by doing exactly what we
want: a cherry-pick is instead "fast forwarded" in the case that the
parent commit has not mutated.

This makes a reimport of a reimport stable in terms of commit and tag
mutations.

LP: #1876329

Author: Robie Basak
Author Date: 2020-05-26 14:15:32 UTC

Convert import request structure to namedtuple

Converting the tuple of (str, bool) to an explicit namedtuple makes the
code more readable.

Author: Robie Basak
Author Date: 2020-05-12 12:54:01 UTC

Release process: add service restart instructions

We probably shouldn't leave the importer service running in a state
where a restart or reboot could change behaviour, so we should restart
the service as part of an update.

Author: Robie Basak
Author Date: 2020-05-07 14:33:59 UTC

Fix docstring

This docstring was incorrect. The function uses a repo fixture, not a
repo_factory fixture, so update the docstring to match.

Author: Robie Basak
Author Date: 2020-05-06 16:30:09 UTC

Fix changelog timezone parsing

I noticed that in the reimport of gpsd, cgit reported

    2020-03-18 16:52:59 +0100

...whereas the changelog timestamp said

    Wed, 18 Mar 2020 15:52:59 +0100

...so there appeared to be something wrong with the parsing.

It turns out that the timezone component in the changelog was being
ignored because Python's time.strptime() is defined to ignore it. We
must use datetime.strptime() instead. The parsing is therefore entirely
adjusted to use only a datetime object.

This wasn't being caught in tests because the expected test value was
wrong - presumably calculated with the same mistake.

I have verified that, based on the date string given in
gitubuntu/changelog_tests/test_date{1,2}, using:

    date +%s -d "Mon, 12 May 2016 08:14:34 -0700"

...does report the value of 1463066074, so all tests are adjusted
accordingly. I have also verified that the above command does mutate the
answer if the timezone is mutated, so it does appear to be correctly
timezone-aware.

While there still a risk that I have calculated the expected value
wrong, hopefully the above test migitates this sufficiently. We do
already have adequate test coverage that will prevent regressions in
Changelog.git_authorship(); the problem was that the expected values in
those tests were wrong and this is now fixed.

Author: Robie Basak
Author Date: 2020-05-06 13:56:45 UTC

Don't add changelog notes fetch refspec for !pkg

The changelog notes fetch respec should only be added for the main 'pkg'
remote (the "base remote" in the code), as other remotes are not
expected to have these.

This fixes a bug:

$ git ubuntu remote add <anything other than pkg>
fatal: Couldn't find remote ref refs/notes/commits

We don't currently have test coverage for the affected methods, but I
have tried to reduce tech debt somewhat at least by adding docstrings to
clarify intended behaviour.

Thanks to Andreas Hasenack for finding this in the edge channel early.

Author: Robie Basak
Author Date: 2020-04-24 14:14:19 UTC

Preserve MPs on reimport

When reimporting, instead of deleting a git repository before repushing
it, we can force push all our new refs and ensure that refs not being
pushed get deleted using --prune. This creates the same effect but
allows us to preserve old merge proposals.

There is a possibility that some configuration in the git repository
will be left behind from a previous import. However all configuration
(eg. the HEAD pointer) is reset by the importer on every run, so
everything that the importer touches will get fixed. Anything else can
be considered out of scope for --reimport to achieve.

The default repository for the source package will therefore not be
reset; if it was set to the old repository, it will continue to point to
the "new" one. This is actually what we want, so is fine.

Author: Robie Basak
Author Date: 2020-05-01 13:24:33 UTC

Merge branch 'spec-overrides' into try

Author: Robie Basak
Author Date: 2020-04-28 16:51:31 UTC

Fix Debian/Ubuntu HEAD branch detection

repo.references provides a list of pygit2.Reference objects, not strings
of their names. Just try the lookup method directly instead.

Author: Robie Basak
Author Date: 2020-04-30 13:09:49 UTC

Merge branch 'document-testing-strategy'

Author: Robie Basak
Author Date: 2020-04-27 14:36:57 UTC

Document testing strategy

Author: Robie Basak
Author Date: 2020-04-29 12:08:51 UTC

Clone the default target repository directly

We intend to move the imported repositories to a separate Launchpad team
whose only member will be the importer bot. This will break existing
users' pkg remote URLs, since they currently have ~usd-import-team
hardcoded.

Instead, change behaviour so that "git clone" will configure the "pkg"
remote to the URL for the default repository for a package only. We will
ensure out-of-band that the default repository is pointed correctly for
all imported packages.

Then when the default repository changes as expected, users who used
"git ubuntu clone" after this change will not be affected; they will
seamlessly access the new repository location at the Launchpad end.

The GitUbuntuRepository.add_base_remotes() is currently only called from
two places: once from importer.py, and once from clone.py. We want the
importer to always use an explicit owner, but clone not to do so in
order that the default is used in that case. It just so happens that the
call from importer.py already always specifies repo_owner, and clone.py
already does not, so the callers are already doing what we need and do
not need to be adjusted.

Author: Robie Basak
Author Date: 2020-04-29 09:49:07 UTC

Clean up changelog note output

Now that the changelog entries go into a separate git note, we no longer
need the leading newlines. This tidies up the "git log" output for
better readabilty for humans.

Author: Robie Basak
Author Date: 2020-04-29 09:41:47 UTC

Adjust name and email to match new spec

Settling on this now is required for hash stability.

Author: Robie Basak
Author Date: 2020-04-28 16:51:03 UTC

Clear notes before reimport

Fixes bug that a reimport will create notes that already exist
otherwise.

Author: Robie Basak
Author Date: 2020-04-22 15:34:59 UTC

Make tagging hash-stable

Now that we have importer notes, adjust the created import tags so that
they are hash-stable.

This does not include applied tags, which will be made stable at a later
point.

Author: Robie Basak
Author Date: 2020-04-20 17:51:02 UTC

Add rich history preservation

Add a new facility to permit rich history to be preserved across
reimports (when using "git ubuntu import --reimport"). Previous rich
history is exported to a temporary directory prior to the reimport.
During import, the rich history importer is then given the opportunity
to reconstruct an upload tag "just in time" before it is used.

Testing the importer end-to-end for this is not currently practical
since we have no tests for the --reimport function at all. We have
pragmatically concluded that it is sufficient to functionally test the
new rich history code, since --reimport will generally only be used with
manual supervision and the likely bugs in the importer code itself are
that rich history preservation won't work at all, or the importer will
fail entirely, and so will be easily noticed.

I have also added some limited additional tests to bring coverage of the
new gitubuntu.rich_history up to 100%.

Author: Robie Basak
Author Date: 2020-04-17 14:59:27 UTC

Interleave LP publications during consumption

The specification requires that we consider Debian and Ubuntu
publications in Launchpad simultaneously, ordering by date_created
across them both. This implements this requirement with
interleave_launchpad_versions_published_after() to wrap multiple calls
of launchpad_versions_published_after() and interleaving the results as
they are returned. The caller is adjusted to use this instead of
iterating over the two distributions separately.

LP: #1851361

Author: Robie Basak
Author Date: 2020-04-09 12:31:45 UTC

Add explanatory comments on changelog notes

Author: Robie Basak
Author Date: 2020-03-31 13:24:43 UTC

Bump ubuntu-dev-tools Python 3 version

Bump ubuntu-dev-tools to 0.176, but only for Python 3. The Python 2
installation will remain as-is, since 0.176 doesn't appear to work with
Python 2.

This fixes an import bug where the necessary fix was in ubuntu-dev-tools.

The Python 2 installation is presumed needed by some other tooling
called by git-ubuntu via the shell. I expect this can go away when we
move the snap base to core20, which is currently deferred for the
future. In the meantime, we exclude the Python 2 version from installing
into some common directories to avoid conflicts. This does lead to a
somewhat frankenstein installation of ubuntu-dev-tools, but I don't
think it will create a problem in practice until we can move to core20.

LP: #1860456

Author: Robie Basak
Author Date: 2020-03-19 03:08:31 UTC

Rewrite unapplied commit creation and tagging

This is a major functional change in the behaviour of the importer
algorithm. The output of the importer will now change to match the new
spec.

The impacted areas are the ones that find any previous commit to use,
create a commit as necessary, and tag accordingly.

Accepted upload tags are now adopted directly rather than creating a new
commit with the upload tag as its parent.

Reimport tags are now fully incorporated, eliminating the bugs around
edge cases where Launchpad stores different uploads in Debian and Ubuntu
against the same package version string.

The following bugs are fixed by this change. I have verified that for
each bug an adequate existing test exists or a test would not be
applicable (for example because orphan tags no longer exist).

LP: #1734883
LP: #1753800
LP: #1754194
LP: #1754507
LP: #1754706
LP: #1755247
LP: #1761331
LP: #1761332
LP: #1772462

Author: Robie Basak
Author Date: 2020-03-19 03:42:19 UTC

Fix importer test validation repo commits

These tests expect new commits to be created, so based on "reused" being
False, the expected results should have "new" commit messages and not
original ones.

This test continues to xfail and will be fixed in a future commit. This
is why the problem wasn't noticed previously.

Author: Robie Basak
Author Date: 2020-03-19 00:46:26 UTC

RenamableDir: check that the supplied file exists

This was required before anyway; this makes it explicit that it is
checked for by the constructor, rather than just being an assertion.

Author: Robie Basak
Author Date: 2020-03-16 09:47:38 UTC

Fix RenameableDir instantiation on broken symlinks

RenameableDir may be instantiated with a path that exists on the
filesystem but that is a broken symlink. In this case the "exists"
assertion should not follow the symlink, so it should use
os.path.lexists(), not os.path.exists().

This fixes the corresponding test, so we drop the xfail.

LP: #1864255

Author: Robie Basak
Author Date: 2020-02-12 14:44:28 UTC

Add further docstrings and comments

Add docstrings to some functions which previously did not have any. One
related comment is also added.

Some of these docstrings expose some shortcomings in the implementation,
but in the interest of making progress the code is documented as-is, and
improvements to the code are left for the future.

Author: Robie Basak
Author Date: 2020-02-10 17:14:07 UTC

Add style notes on the use of xfail

Author: Robie Basak
Author Date: 2020-01-09 16:03:57 UTC

.git escaping: fix failing case

Traverse the entire directory for recursive escapes;
sorted_subpaths_to_rename is already filtered so is not complete itself.

This fixes the recursive traversal that looks for entries that require
escaping.

This causes the previous failing test to pass, so we must remove the
associated xfail marker.

Author: Robie Basak
Author Date: 2020-02-04 11:29:10 UTC

Collapse dual (un)applied code using an enum

In various places we have two implementations of near-identical code
where the only difference is whether they act on unapplied or applied
branches, tags or structure. Ultimately they do need to be handled
differently at some point, but most of the code in the middle of the
call graph does not need to care.

To avoid code duplication we define a PatchState enum and can then drop
half of the middling code by simply passing through the "unapplied-ness"
or "applied-ness" as a parameter.

The following functions are removed from gitubuntu.git_repository.
Instead, the unapplied previous equivalent should be used with
patch_state=PatchState.APPLIED:

applied_reimport_tag_prefix
applied_reimport_tag
applied_tag

The following methods are removed from
gitubuntu.git_repository.GitUbuntuRepository. Instead, the unapplied
previous equivalent should be used with patch_state=PatchState.APPLIED:

get_applied_reimport_tag
get_all_applied_reimport_tags

A grep suggests that the only callers are importer.py and
importer_test.py; these are refactored as part of this change, dropping
duplication where there was any.

"git ubuntu tag" is the obvious CLI command that might have been
affected, but it doesn't appear to need any of these functions or
methods. Perhaps it should, but that's a job for a future refactoring.

This change does not cover every instance of code duplication or where
refactoring to use the new enum might be appropriate; we merely fix the
code that needed touching anyway, leaving other refactoring for the
future.

This change should not result in any functional changes.

Author: Robie Basak
Author Date: 2020-01-28 11:14:05 UTC

self-test: be more explicit when failing

The script will exit non-zero if any of a set of tests fail, but it
isn't necessarily obvious from the output what was a warning and what
caused the failure. Let's make this obvious in the output. In all cases
where we set a future failure, explicitly tell the caller via stdout.

Author: Robie Basak
Author Date: 2020-01-28 11:05:19 UTC

Drop pin on lazr.restfulclient

lazr.restfulclient 0.14.3 has been released, causing the self-test
error:

  gitubuntu 0.9.0 has requirement lazr.restfulclient==0.14.2, but you have lazr.restfulclient 0.14.3.

There is no need to pin this, so drop the pin.

Author: Robie Basak
Author Date: 2020-01-16 11:56:31 UTC

Change main phasing default to 100%

The official importer instance (the only known instance) had been at
100% phasing of main for months, but accidentally reverted to the old
default of 1% as the configuration was lost when it was converted to a
proper service.

Let's move the default to match the production instance to reduce
the likelyhood of mistakes.

Author: Robie Basak
Author Date: 2019-12-09 11:44:32 UTC

Rewrite commit_tree_hash into commit_source_tree

To fix LP: #1764814, the updated specification changes the definition of
how commit authorship is derived. This is a rewrite of
GitUbuntuRepository.commit_tree_hash() to meet the new specification.

Since more accurately we're committing a tree that must contain a Debian
package source, the method is renamed to commit_source_tree() in this
rewrite.

fallback_author and fallback_date are no longer necessary and are
removed. Instead we use a commit_date parameter which specifies the
required date of the commit. According to the new specification this
date must come from the underlying Launchpad object. Since this method
operates on a low level git tree instead of at the Launchpad data
structure level, the commit date is the caller's responsiblity to obtain
and provide; this method will simply use the date given.

git repository hashes are now passed and returned as pygit2.Oid objects
as we move to this type as the usual standard.

The committer name and email are defined in the specification via
gitubuntu/spec.py, which is added as a new file. This file forms part of
the specification itself, and its contents are separated in its own file
to avoid inadvertent changes to the specification.

The GitUbuntuRepository methods get_commit_environment() and
get_commit_authorship() were only used by the old commit_tree_hash()
method and are removed. Instead we use Changelog.git_commit_authorship()
as added in the previous commit.

Since we have changed the definition of
commit_tree_hash()/commit_source_tree() in the rewrite, all callers are
updated to the new method name, new parameter names and types, and the
calculation of fallback_ is removed.

In tests, it is further no longer necessary to patch the
get_commit_environment() method, which no longer exists. Instead,
repo_builder.Commit is adjusted to default to using the same metadata as
the specification defines. This results in the same behaviour as before
when the test suite is run, but is cleaner since less patching is
required.

Author: Robie Basak
Author Date: 2019-11-04 14:29:38 UTC

Add importer tests

Author: Robie Basak
Author Date: 2019-12-06 08:02:50 UTC

Add style convention on import statements

This is what we've been following anyway, but without explicit
documentation.

Author: Robie Basak
Author Date: 2019-10-15 16:18:27 UTC

Work around pylint/astroid bug

astroid appears to have a bug causing pylint to return a false positive:
https://github.com/PyCQA/pylint/issues/3137

This was apparently introduced in astroid 2.3, so hold back from using
it to work around the problem since this is causing our CI to fail.

pylint 2.4 requires astroid 2.3 without declaring it
(https://github.com/PyCQA/pylint/issues/3022) so additionally we also
need to hold back from using pylint 2.4 to avoid hitting this other
failure.

Author: Robie Basak
Author Date: 2019-08-05 12:51:19 UTC

fixup! source_information: Fix ubuntu series

Drop xfail now that the bug is fixed

Author: Robie Basak
Author Date: 2019-04-25 12:25:28 UTC

Blacklist wheezy as an active series

Author: Robie Basak
Author Date: 2019-04-23 11:45:33 UTC

Cache some Launchpad object fetches

As we iterate through Launchpad's source_package_publication_history
records, we keep looking up their distro_series objects via
GitUbuntuSourcePackageInformation.distribution.name and suchlike. This
results in a very large number of API calls to fetch the same
distro_series object from Launchpad every time, causing a severe
degradation in performance.

Since distribution_link is the same every time, we can instead key a
cache on this, eliminating the excessive number of API calls. The same
applies to some other similar calls.

As far as I can tell we've always made this excessive number of calls
here, but in practice something recently changed in the amount of time
these calls are taking, causing import performance to drop virtually to
a standstill. This optimisation restores reasonable performance again.

This introduces a new dependency on cachetools, which seems to be the
cleanest caching solution without having to reimplement it inside
git-ubuntu. python3-cachetools is available in Xenial and the module
seems to be stable upstream, so I hope this addition will not cause us
too much trouble.

Author: Robie Basak
Author Date: 2018-11-29 15:31:37 UTC

Friendlier self-test output

If we change directory to the location of the gitubuntu Python module
first, the commands to pylint and pytest are simpler and additionally
has the advantage that the output from pytest is nicer.

Author: Robie Basak
Author Date: 2018-11-08 11:51:47 UTC

Merge remote-tracking branch 'smoser/fix/1801168-build-when-no-patches'

Author: Robie Basak
Author Date: 2018-11-06 11:34:48 UTC

Drop use of dry_run that is no longer defined

This fixes a regression caused by
d17507e57d60f744540088d6a1d4f56a80b92970.

Author: Robie Basak
Author Date: 2018-11-05 14:41:49 UTC

Bump ubuntu-keyring keyrings used in the snap

LP: #1801725